Examenos

Text Decision Index 0.3.2 · Full score

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Public score

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private same-skill score

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private new-domain score

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-18 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-18 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-18 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-18 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published Median latency for the board setup; hosted network and local inference are separate.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Local board timing on one RTX PRO 6000; not hosted network latency. Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published p95 latency for the board setup; hosted network and local inference are separate.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Local board timing on one RTX PRO 6000; not hosted network latency. Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BFCL formulated as typed decisions; published case exact accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "case exact accuracy", "requests": 1694, "answered": 1694, "unsupported": 0, "errors": 0, "median_ms": 180.1}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1694

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ToolRet formulated as typed decisions; published nDCG@10 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "nDCG@10", "requests": 685, "answered": 685, "unsupported": 0, "errors": 0, "median_ms": 1808.9}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 685

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

API-Bank formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 508, "answered": 508, "unsupported": 0, "errors": 0, "median_ms": 603.3}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 508

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BANKING77 formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 3080, "answered": 3080, "unsupported": 0, "errors": 0, "median_ms": 122.2}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 3080

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLINC150+OOS formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0, "median_ms": 133.2}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RouterBench formulated as typed decisions; published selected quality (quality objective) for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "selected quality (quality objective)", "requests": 10000, "answered": 10000, "unsupported": 0, "errors": 0, "median_ms": 339.8}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 10000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Home appliance simulator formulated as typed decisions; published case exact accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "case exact accuracy", "requests": 88, "answered": 88, "unsupported": 0, "errors": 0, "median_ms": 2007.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 88

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SGD/SGD-X formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 2500, "answered": 2500, "unsupported": 0, "errors": 0, "median_ms": 126}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ContractNLI formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 123, "answered": 123, "unsupported": 0, "errors": 0, "median_ms": 3047.7}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 123

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ANLI formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 3200, "answered": 3200, "unsupported": 0, "errors": 0, "median_ms": 120.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 3200

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BPoMP formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0, "median_ms": 119.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Humicroedit formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 2628, "answered": 2628, "unsupported": 0, "errors": 0, "median_ms": 119.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2628

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

POP909-CL formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0, "median_ms": 515.3}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

cfcolor formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0, "median_ms": 119.7}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 14033, "answered": 14033, "unsupported": 0, "errors": 0, "median_ms": 118.9}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 14033

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GPQA Diamond formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 196, "answered": 196, "unsupported": 0, "errors": 0, "median_ms": 120.2}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 196

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Easy formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 2376, "answered": 2376, "unsupported": 0, "errors": 0, "median_ms": 119}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2376

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Challenge formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 1172, "answered": 1172, "unsupported": 0, "errors": 0, "median_ms": 118.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1172

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

WinoGrande formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 1267, "answered": 1267, "unsupported": 0, "errors": 0, "median_ms": 118.1}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1267

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HellaSwag formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 10042, "answered": 10042, "unsupported": 0, "errors": 0, "median_ms": 119.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 10042

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GSM8K formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": 0.7301, "metric": "accuracy (4- and 10-choice)", "requests": 2638, "answered": 2638, "unsupported": 0, "errors": 0, "median_ms": null}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2638

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ChessBench formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0, "median_ms": 123.3}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MuSR formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 752, "answered": 752, "unsupported": 0, "errors": 0, "median_ms": 125.4}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 752

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SATA-Bench formulated as typed decisions; published case exact accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "case exact accuracy", "requests": 1650, "answered": 1650, "unsupported": 0, "errors": 0, "median_ms": 277.8}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1650

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BRIGHT formulated as typed decisions; published nDCG@10 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "nDCG@10", "requests": 220, "answered": 220, "unsupported": 0, "errors": 0, "median_ms": 2126.9}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 220

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Amazon ESCI formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0, "median_ms": 122.1}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ACOS formulated as typed decisions; published per-review F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "per-review F1", "requests": 1565, "answered": 1565, "unsupported": 0, "errors": 0, "median_ms": 806}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1565

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

FinEntity formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 979, "answered": 979, "unsupported": 0, "errors": 0, "median_ms": 122.2}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 979

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

iSarcasmEval formulated as typed decisions; published Sarcasm F1 · track A, English for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "Sarcasm F1 · track A, English", "requests": 4600, "answered": 4600, "unsupported": 0, "errors": 0, "median_ms": 120.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 4600

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

VAST formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 3006, "answered": 3006, "unsupported": 0, "errors": 0, "median_ms": 119.1}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 3006

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

NLI4CT formulated as typed decisions; published macro-F1 for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "macro-F1", "requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0, "median_ms": 121.7}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CRUXEval formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 570, "answered": 570, "unsupported": 0, "errors": 0, "median_ms": 119}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 570

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLadder formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0, "median_ms": 118.9}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HLE formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 501, "answered": 501, "unsupported": 0, "errors": 0, "median_ms": 120.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 501

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Habermas Machine formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 1676, "answered": 1676, "unsupported": 0, "errors": 0, "median_ms": 123.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 1676

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

PhishNChips phishing decisions formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0, "median_ms": 196.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU-Pro formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 12032, "answered": 12032, "unsupported": 0, "errors": 0, "median_ms": 119.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 12032

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BBH formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 5507, "answered": 5507, "unsupported": 0, "errors": 0, "median_ms": 118.5}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 5507

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RAGTruth formulated as typed decisions; published F1 on hallucinated class for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "F1 on hallucinated class", "requests": 2700, "answered": 2700, "unsupported": 0, "errors": 0, "median_ms": 122.9}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 2700

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HoVer formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 4000, "answered": 4000, "unsupported": 0, "errors": 0, "median_ms": 120.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 4000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

When2Call formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 3652, "answered": 3652, "unsupported": 0, "errors": 0, "median_ms": 122.8}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 3652

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

New Yorker formulated as typed decisions; published accuracy for this board population.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Exact published per-benchmark observation; no composite reconstructed. Counts/errors and timing: {"jev": null, "metric": "accuracy", "requests": 528, "answered": 528, "unsupported": 0, "errors": 0, "median_ms": 118.6}

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-18 · Samples: 528

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 11

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 11

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 11

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 9

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 9

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 9

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 5

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 5

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 5

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Source-published category quantity; skill-adjusted, raw and combined values are distinct.

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too.

Checked / reported: 2026-10-18 · Samples: 6

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF text 0.3.2 · Raw balanced index

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Source-published alternative aggregate, separate from Full and Public scores; no Examenos recomputation.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF text 0.3.2 · Public breadth index

Board exact pplx-decider-v1.2. Public results submitted by Perplexity; private tests run by the board on one RTX PRO 6000. One question per request, author serving code. Exact execution SHA/precision not restated by this source.

Submitted directly as a results dataset (perplexity-ai/decision-index-results), no PR. Replaces the earlier board entry pplx-decider-v1.1-27b (newer version of the same model). Submitted public 0.3 run adopted (kit rescore 67.74, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.79% on 827 rows (bar 98.5%). One question per request, the submitter's chosen mode. Training includes official training splits of several Decision Index benchmarks, decontaminated against the test sets; the checkpoint is a weight average of several runs chosen partly on Decision Index (author-declared). Model repository is private. Current official HF API is public/ungated; the board’s historical private-repository note describes its source snapshot, not current access.

Source-published alternative aggregate, separate from Full and Public scores; no Examenos recomputation.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Multimodal typed-choice vision evaluation · Full

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Public and private panels are separate; Full is the board-published aggregate, not vendor card scores.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Public

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Public and private panels are separate; Full is the board-published aggregate, not vendor card scores.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Private

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Public and private panels are separate; Full is the board-published aggregate, not vendor card scores.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); private population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on R-Bench-M; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 665.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on MMMU-Pro vision; pub population.

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Native board score; private normalized values can be negative. No clipping or substituted subset.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1730.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HF vision 0.3.2 · Chosen-answer accuracy

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Source-defined vision calibration population and confidence convention; distinct from text calibration.

Checked / reported: 2026-10-18 · Samples: 13695

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF vision 0.3.2 · Mean chosen confidence

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Source-defined vision calibration population and confidence convention; distinct from text calibration.

Checked / reported: 2026-10-18 · Samples: 13695

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF vision 0.3.2 · Chosen-answer ECE

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Source-defined vision calibration population and confidence convention; distinct from text calibration.

Checked / reported: 2026-10-18 · Samples: 13695

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF vision 0.3.2 · Incorrect and >95% confident

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Source-defined vision calibration population and confidence convention; distinct from text calibration.

Checked / reported: 2026-10-18 · Samples: 13695

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF vision 0.3.2 · Time per row (source)

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Published resource observation for this suite and hardware; not API speed or billing. Detailed distribution statistic for time-per-row is not inferred.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

HF vision 0.3.2 · Evaluation GPU hours

Exact vision entrant pplx-decider-v1.2. Ran through the checkpoint's own pplx_decider engine (the submitted text adapter), which accepts images. Effective image budget 1.6 MP. author cap equals the board cap (decision_config.json max_pixels)

Published resource observation for this suite and hardware; not API speed or billing. Detailed distribution statistic for time-per-row is not inferred.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Knowledge

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Knowledge

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Language

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Language

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Retrieval

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Retrieval

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Tools

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Tools

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Arts

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Arts

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · joint · Overall Decision Index

Perplexity card; joint request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 public · single · Overall Decision Index

Perplexity card; single request mode; original published suite weights. Qwen3.8-27B backbone; joint marker readouts.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 · Vision public index

Vendor vision public run; image cap 1,638,400 pixels.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations. Vendor card values differ slightly from the board rerun and are preserved separately.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · SATA-Bench (select all that apply)

Joint requests versus one question per request; one per option.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · ACOS (aspect-category sentiment)

Joint requests versus one question per request; one per category.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · PhishNChips

Joint requests versus one question per request; facets of one message.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · BFCL (function calling)

Joint requests versus one question per request; one per candidate call.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · BRIGHT (reasoning-intensive retrieval)

Joint requests versus one question per request; one per candidate document.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity v1.2 joint minus single · ToolRet (tool retrieval)

Joint requests versus one question per request; one per candidate tool.

Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Signed task-score difference, not absolute task accuracy. Vendor-published measurements. Decision Index 0.3 public suite, not AA, evaluator-private Full, or a controlled cross-provider comparison. Official training splits were included with test decontamination; held-out claims are limited to their stated populations.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Reference (padded, length-sorted batches) · Decision Index requests

One GB200; Reference (padded, length-sorted batches); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Reference (padded, length-sorted batches) · Reranking requests

One GB200; Reference (padded, length-sorted batches); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Reference (padded, length-sorted batches) · Vision requests (1.6 MP)

One GB200; Reference (padded, length-sorted batches); Vision requests (1.6 MP).

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Packing, no padding (compile=None, cuda_graphs=False) · Decision Index requests

One GB200; Packing, no padding (compile=None, cuda_graphs=False); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · Packing, no padding (compile=None, cuda_graphs=False) · Decision Index requests

One GB200; Packing, no padding (compile=None, cuda_graphs=False); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Packing, no padding (compile=None, cuda_graphs=False) · Reranking requests

One GB200; Packing, no padding (compile=None, cuda_graphs=False); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · Packing, no padding (compile=None, cuda_graphs=False) · Reranking requests

One GB200; Packing, no padding (compile=None, cuda_graphs=False); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + compile norms/MLP/conv (compile="sub") · Decision Index requests

One GB200; + compile norms/MLP/conv (compile="sub"); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + compile norms/MLP/conv (compile="sub") · Decision Index requests

One GB200; + compile norms/MLP/conv (compile="sub"); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + compile norms/MLP/conv (compile="sub") · Reranking requests

One GB200; + compile norms/MLP/conv (compile="sub"); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + compile norms/MLP/conv (compile="sub") · Reranking requests

One GB200; + compile norms/MLP/conv (compile="sub"); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + whole-layer compile (compile="layer") · Decision Index requests

One GB200; + whole-layer compile (compile="layer"); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + whole-layer compile (compile="layer") · Decision Index requests

One GB200; + whole-layer compile (compile="layer"); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + whole-layer compile (compile="layer") · Reranking requests

One GB200; + whole-layer compile (compile="layer"); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + whole-layer compile (compile="layer") · Reranking requests

One GB200; + whole-layer compile (compile="layer"); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + whole-layer compile (compile="layer") · Vision requests (1.6 MP)

One GB200; + whole-layer compile (compile="layer"); Vision requests (1.6 MP).

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + whole-layer compile (compile="layer") · Vision requests (1.6 MP)

One GB200; + whole-layer compile (compile="layer"); Vision requests (1.6 MP).

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + static 32k packs, CUDA graphs (buckets=(32768,)) · Decision Index requests

One GB200; + static 32k packs, CUDA graphs (buckets=(32768,)); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + static 32k packs, CUDA graphs (buckets=(32768,)) · Decision Index requests

One GB200; + static 32k packs, CUDA graphs (buckets=(32768,)); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + static 32k packs, CUDA graphs (buckets=(32768,)) · Reranking requests

One GB200; + static 32k packs, CUDA graphs (buckets=(32768,)); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + static 32k packs, CUDA graphs (buckets=(32768,)) · Reranking requests

One GB200; + static 32k packs, CUDA graphs (buckets=(32768,)); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + last-layer pruning · Decision Index requests

One GB200; + last-layer pruning; Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + last-layer pruning · Decision Index requests

One GB200; + last-layer pruning; Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + last-layer pruning · Reranking requests

One GB200; + last-layer pruning; Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + last-layer pruning · Reranking requests

One GB200; + last-layer pruning; Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + last-layer pruning · Vision requests (1.6 MP)

One GB200; + last-layer pruning; Vision requests (1.6 MP).

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + last-layer pruning · Vision requests (1.6 MP)

One GB200; + last-layer pruning; Vision requests (1.6 MP).

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + FP8 matmuls (fp8=True, off by default) · Decision Index requests

One GB200; + FP8 matmuls (fp8=True, off by default); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + FP8 matmuls (fp8=True, off by default) · Decision Index requests

One GB200; + FP8 matmuls (fp8=True, off by default); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · + FP8 matmuls (fp8=True, off by default) · Reranking requests

One GB200; + FP8 matmuls (fp8=True, off by default); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · + FP8 matmuls (fp8=True, off by default) · Reranking requests

One GB200; + FP8 matmuls (fp8=True, off by default); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Default fast=True (buckets 2k to 32k, pruning) · Decision Index requests

One GB200; Default fast=True (buckets 2k to 32k, pruning); Decision Index requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · Default fast=True (buckets 2k to 32k, pruning) · Decision Index requests

One GB200; Default fast=True (buckets 2k to 32k, pruning); Decision Index requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Default fast=True (buckets 2k to 32k, pruning) · Reranking requests

One GB200; Default fast=True (buckets 2k to 32k, pruning); Reranking requests.

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · Default fast=True (buckets 2k to 32k, pruning) · Reranking requests

One GB200; Default fast=True (buckets 2k to 32k, pruning); Reranking requests.

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 · Default fast=True (buckets 2k to 32k, pruning) · Vision requests (1.6 MP)

One GB200; Default fast=True (buckets 2k to 32k, pruning); Vision requests (1.6 MP).

Task distribution: Decision Index average 1,050 tokens, 1–64 questions; reranking 437 tokens; vision 1.6 MP. Compilation, packing and CUDA-graph conditions are path-specific.

One GB200, BF16 except the explicitly FP8 path; source request populations and batching differ. No provider API throughput inference.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 speedup · Default fast=True (buckets 2k to 32k, pruning) · Vision requests (1.6 MP)

One GB200; Default fast=True (buckets 2k to 32k, pruning); Vision requests (1.6 MP).

Reported speedup against this workload’s padded reference; not reconstructed from rounded throughput.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity dev · calibrated · Accuracy

Joint request format; calibrated probabilities; fitted temperature 1.382.

1,497-question development sample, separate from temperature-fitting data. Native Brier and ECE pp remain distinct.

Checked / reported: 2026-10-18 · Samples: 1497

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity dev · calibrated · ECE

Joint request format; calibrated probabilities; fitted temperature 1.382.

1,497-question development sample, separate from temperature-fitting data. Native Brier and ECE pp remain distinct.

Checked / reported: 2026-10-18 · Samples: 1497

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity dev · uncalibrated · ECE

Joint request format; uncalibrated probabilities; fitted temperature 1.382.

1,497-question development sample, separate from temperature-fitting data. Native Brier and ECE pp remain distinct.

Checked / reported: 2026-10-18 · Samples: 1497

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity dev · calibrated · Brier

Joint request format; calibrated probabilities; fitted temperature 1.382.

1,497-question development sample, separate from temperature-fitting data. Native Brier and ECE pp remain distinct.

Checked / reported: 2026-10-18 · Samples: 1497

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity dev · uncalibrated · Brier

Joint request format; uncalibrated probabilities; fitted temperature 1.382.

1,497-question development sample, separate from temperature-fitting data. Native Brier and ECE pp remain distinct.

Checked / reported: 2026-10-18 · Samples: 1497

Snapshot: 2026-10-16 · Revision: Official v1.2 cards · 2026-10-16 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · BF16 fast versus reference · Argmax agreement

BF16 fast versus reference

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · BF16 fast versus reference · Mean absolute probability difference

BF16 fast versus reference

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · Reference batched versus single-call · Argmax agreement

Reference batched versus single-call

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · Reference batched versus single-call · Mean absolute probability difference

Reference batched versus single-call

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · FP8 versus reference · Argmax agreement

FP8 versus reference

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · FP8 versus reference · Mean absolute probability difference

FP8 versus reference

300 Decision Index requests, 2,324 questions. Numerical agreement with the implementation reference is not task accuracy.

Checked / reported: 2026-10-18 · Samples: 2324

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity reranking · BF16 fast versus reference · Spearman

BF16 fast versus reference

40 BEIR queries × 100 candidates. Per-query rank correlation with the reference, not retrieval accuracy.

Checked / reported: 2026-10-18 · Samples: 40

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity reranking · FP8 versus reference · Spearman

FP8 versus reference

40 BEIR queries × 100 candidates. Per-query rank correlation with the reference, not retrieval accuracy.

Checked / reported: 2026-10-18 · Samples: 40

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · Reference separate processes mean probability difference

Reference separate processes mean probability difference

Numerical difference against the source implementation reference; not task error or an independent calibration study.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity numerics · FP8 maximum probability difference

FP8 maximum probability difference

Numerical difference against the source implementation reference; not task error or an independent calibration study.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 single request · Reference single-call

One GB200; Reference single-call.

Median single Decision Index request timing; not throughput or hosted API latency.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity GB200 single request · BF16 bucketed fast

One GB200; BF16 bucketed fast.

Median single Decision Index request timing; not throughput or hosted API latency.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · HotpotQA · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · CQADupStack · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · Quora · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · ArguAna · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · TREC-COVID · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity BEIR · 13-dataset macro · 8 minus 1 candidate/request

Vendor BEIR reranking, eight candidates in one request versus one candidate/request.

Signed nDCG@10 points. Source gives only ranges for FiQA/NQ/MS MARCO/Climate-FEVER (−0.2 to −0.4) and NFCorpus/SciFact/FEVER/DBpedia-Entity (−0.7 to −1.2); these ranges are retained here without inventing per-dataset point values.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity 243K log spot check · reference · Approximate peak gpu memory

Vendor long-context spot check; reference path.

Published approximate workload observation; no inference of provider latency or generic scaling law.

One fact retrieval from a 243K-token log on one GB200. Vendor spot check only; not validated whole-context accuracy. P(correct)>0.999 is a bound, not an exact point result.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity 243K log spot check · reference · Approximate request duration

Vendor long-context spot check; reference path.

Published approximate workload observation; no inference of provider latency or generic scaling law.

One fact retrieval from a 243K-token log on one GB200. Vendor spot check only; not validated whole-context accuracy. P(correct)>0.999 is a bound, not an exact point result.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity 243K log spot check · fast · Approximate peak gpu memory

Vendor long-context spot check; fast path.

Published approximate workload observation; no inference of provider latency or generic scaling law.

One fact retrieval from a 243K-token log on one GB200. Vendor spot check only; not validated whole-context accuracy. P(correct)>0.999 is a bound, not an exact point result.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

Perplexity 243K log spot check · fast · Approximate request duration

Vendor long-context spot check; fast path.

Published approximate workload observation; no inference of provider latency or generic scaling law.

One fact retrieval from a 243K-token log on one GB200. Vendor spot check only; not validated whole-context accuracy. P(correct)>0.999 is a bound, not an exact point result.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 225c913ddc8761a8264f04f64548e1f97e443eb3 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public index

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public Knowledge

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public Language

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public Retrieval

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public Tools

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

OneJev peer figures · Public Arts

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; All-case accuracy.

Named source route perplexity/pplx-decider-v1.2; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route perplexity/pplx-decider-v1.2; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 19

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route perplexity/pplx-decider-v1.2; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 14

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route perplexity/pplx-decider-v1.2; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 42

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route perplexity/pplx-decider-v1.2; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aegis2 / test · binary_brier. Population 100 cases / 148 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1348566219441989,"accuracy":0.7972972972972973,"positive_recall":0.7323943661971831,"specificity":0.8571428571428571,"precision":0.8253968253968254,"balanced_accuracy":0.7947686116700201,"f1":0.7761194029850746,"true_positive":52,"false_positive":11,"false_negative":19,"true_negative":66,"positive_prevalence":0.4797297297297297,"majority_accuracy_baseline":0.5202702702702703,"constant_brier_baseline":0.24958911614317017,"brier_skill":0.4596854861778429,"baseline_adjusted_skill":0.5895372233400402,"baseline_adjusted_score":0.5895372233400402}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aqua_rat / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.75,"uniform_accuracy_baseline":0.2005,"fixed_answer_accuracy_baseline":0.29,"baseline_adjusted_skill":0.647887323943662,"baseline_adjusted_score":0.647887323943662}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arc / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.2533333333333333,"fixed_answer_accuracy_baseline":0.32,"baseline_adjusted_skill":0.9705882352941176,"baseline_adjusted_score":0.9705882352941176}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arct / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.93,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.53,"baseline_adjusted_skill":0.8510638297872342,"baseline_adjusted_score":0.8510638297872342}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.51,"baseline_adjusted_skill":0.9591836734693877,"baseline_adjusted_score":0.9591836734693877}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.08711651317904412,"accuracy":0.65290653412,"positive_recall":0.8675986916343154,"specificity":0.30845512432117356,"precision":0.6680877907375,"balanced_accuracy":0.5880269079777445,"f1":0.7548833694824132,"true_positive":53.447023259,"false_positive":26.552976741000002,"false_negative":8.156369847,"true_negative":11.843630153,"positive_prevalence":0.61603393106,"majority_accuracy_baseline":0.61603393106,"constant_brier_baseline":0.07908707655202402,"brier_skill":-0.10152653223612695,"baseline_adjusted_skill":0.17605381595548897,"baseline_adjusted_score":0.17605381595548897}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/babi_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0071169483033712375,"accuracy":0.99,"positive_recall":0.98,"specificity":1,"precision":1,"balanced_accuracy":0.99,"f1":0.98989898989899,"true_positive":49,"false_positive":0,"false_negative":1,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9715322067865151,"baseline_adjusted_skill":0.98,"baseline_adjusted_score":0.98}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/banking77 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.89,"uniform_accuracy_baseline":0.012987012987012986,"fixed_answer_accuracy_baseline":0.05,"baseline_adjusted_skill":0.8842105263157894,"baseline_adjusted_score":0.8842105263157894}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/bbq / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.8666666666666667,"baseline_adjusted_score":0.8666666666666667}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/boardgameqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.89,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.36,"baseline_adjusted_skill":0.828125,"baseline_adjusted_score":0.828125}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/canttalk / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.14073441096147254,"accuracy":0.79,"positive_recall":1,"specificity":0.58,"precision":0.704225352112676,"balanced_accuracy":0.79,"f1":0.8264462809917356,"true_positive":50,"false_positive":21,"false_negative":0,"true_negative":29,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.4370623561541098,"baseline_adjusted_skill":0.5800000000000001,"baseline_adjusted_score":0.5800000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/civil_comments / test · binary_brier. Population 100 cases / 700 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05341598541414392,"accuracy":0.7144778209279424,"positive_recall":0.5092954713954787,"specificity":0.810018255508844,"precision":0.55521164449644,"balanced_accuracy":0.6596568634521613,"f1":0.5312632898111752,"true_positive":113.26317547727376,"false_positive":90.73682452272624,"false_negative":109.12870082771406,"true_negative":386.87129917228594,"positive_prevalence":0.3177026804356969,"majority_accuracy_baseline":0.6822973195643032,"constant_brier_baseline":0.08872692397768642,"brier_skill":0.3979732079117575,"baseline_adjusted_skill":0.3193137269043227,"baseline_adjusted_score":0.3193137269043227}. Counts: {"cases": 100, "expected": 700, "succeeded": 700, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cladder / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06973144705022905,"accuracy":0.89,"positive_recall":0.88,"specificity":0.9,"precision":0.8979591836734694,"balanced_accuracy":0.89,"f1":0.8888888888888888,"true_positive":44,"false_positive":5,"false_negative":6,"true_negative":45,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7210742117990838,"baseline_adjusted_skill":0.78,"baseline_adjusted_score":0.78}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/clinc / test · target_mass_at_prediction. Population 151 cases / 151 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9801324503311258,"uniform_accuracy_baseline":0.006622516556291391,"fixed_answer_accuracy_baseline":0.039735099337748346,"baseline_adjusted_skill":0.9793103448275862,"baseline_adjusted_score":0.9793103448275862}. Counts: {"cases": 151, "expected": 151, "succeeded": 151, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 151

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/commonsensemoralja-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.054843993413991314,"accuracy":0.9266666666666666,"positive_recall":0.8733333333333333,"specificity":0.98,"precision":0.9776119402985075,"balanced_accuracy":0.9266666666666666,"f1":0.9225352112676056,"true_positive":131,"false_positive":3,"false_negative":19,"true_negative":147,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7806240263440347,"baseline_adjusted_skill":0.8533333333333333,"baseline_adjusted_score":0.8533333333333333}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/contract_nli / test · target_mass_at_prediction. Population 100 cases / 1700 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8829411764705882,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.4576470588235294,"baseline_adjusted_skill":0.7841648590021691,"baseline_adjusted_score":0.7841648590021691}. Counts: {"cases": 100, "expected": 1700, "succeeded": 1700, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/corr2cause / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.75,"baseline_adjusted_score":0.75}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/creak / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05615246486066876,"accuracy":0.91,"positive_recall":0.94,"specificity":0.88,"precision":0.8867924528301887,"balanced_accuracy":0.9099999999999999,"f1":0.912621359223301,"true_positive":47,"false_positive":6,"false_negative":3,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7753901405573249,"baseline_adjusted_skill":0.8199999999999998,"baseline_adjusted_score":0.8199999999999998}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/crows_pairs / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.52,"baseline_adjusted_skill":0.6874999999999999,"baseline_adjusted_score":0.6874999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-depth-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5,"baseline_adjusted_skill":0.8999999999999999,"baseline_adjusted_score":0.8999999999999999}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-distance-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8933333333333333,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.52,"baseline_adjusted_skill":0.7777777777777777,"baseline_adjusted_score":0.7777777777777777}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-relation-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5266666666666666,"baseline_adjusted_skill":0.9366197183098591,"baseline_adjusted_score":0.9366197183098591}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/dbpedia / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.07142857142857142,"fixed_answer_accuracy_baseline":0.11,"baseline_adjusted_skill":0.9887640449438202,"baseline_adjusted_score":0.9887640449438202}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/defeasible_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.8913043478260868,"baseline_adjusted_score":0.8913043478260868}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/esci / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.56,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.27,"baseline_adjusted_skill":0.3972602739726028,"baseline_adjusted_score":0.3972602739726028}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · target_mass_at_prediction. Population 18 cases / 18 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9444444444444444,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.6666666666666666,"baseline_adjusted_skill":0.8333333333333333,"baseline_adjusted_score":0.8333333333333333}. Counts: {"cases": 18, "expected": 18, "succeeded": 18, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 18

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · binary_brier. Population 82 cases / 82 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.09337919022068719,"accuracy":0.8902439024390244,"positive_recall":0.875,"specificity":0.9047619047619048,"precision":0.8974358974358975,"balanced_accuracy":0.8898809523809523,"f1":0.8860759493670886,"true_positive":35,"false_positive":4,"false_negative":5,"true_negative":38,"positive_prevalence":0.4878048780487805,"majority_accuracy_baseline":0.5121951219512195,"constant_brier_baseline":0.2498512790005949,"brier_skill":0.6262609077119639,"baseline_adjusted_skill":0.7797619047619047,"baseline_adjusted_score":0.7797619047619047}. Counts: {"cases": 82, "expected": 82, "succeeded": 82, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 82

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethos / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.12006962003139046,"accuracy":0.82,"positive_recall":0.76,"specificity":0.88,"precision":0.8636363636363636,"balanced_accuracy":0.8200000000000001,"f1":0.8085106382978723,"true_positive":38,"false_positive":6,"false_negative":12,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5197215198744382,"baseline_adjusted_skill":0.6400000000000001,"baseline_adjusted_score":0.6400000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/few_nerd / test · target_mass_at_prediction. Population 100 cases / 358 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8435754189944135,"uniform_accuracy_baseline":0.125,"fixed_answer_accuracy_baseline":0.2206703910614525,"baseline_adjusted_skill":0.7992831541218639,"baseline_adjusted_score":0.7992831541218639}. Counts: {"cases": 100, "expected": 358, "succeeded": 358, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/fol_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.84,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.7377049180327868,"baseline_adjusted_score":0.7377049180327868}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_core17 / test · binary_brier. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1573298229819752,"accuracy":0.8,"positive_recall":0.45454545454545453,"specificity":0.8974358974358975,"precision":0.5555555555555556,"balanced_accuracy":0.675990675990676,"f1":0.5,"true_positive":20,"false_positive":16,"false_negative":24,"true_negative":140,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.1716,"brier_skill":0.08315953973207924,"baseline_adjusted_skill":0.35198135198135194,"baseline_adjusted_score":0.35198135198135194}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_news21 / test · binary_brier. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1513081157655736,"accuracy":0.785,"positive_recall":0.34,"specificity":0.9333333333333333,"precision":0.6296296296296297,"balanced_accuracy":0.6366666666666667,"f1":0.44155844155844154,"true_positive":17,"false_positive":10,"false_negative":33,"true_negative":140,"positive_prevalence":0.25,"majority_accuracy_baseline":0.75,"constant_brier_baseline":0.1875,"brier_skill":0.19302338258360752,"baseline_adjusted_skill":0.27333333333333343,"baseline_adjusted_score":0.27333333333333343}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_robust04 / test · binary_brier. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05002541744671899,"accuracy":0.925,"positive_recall":0.5789473684210527,"specificity":0.9613259668508287,"precision":0.6111111111111112,"balanced_accuracy":0.7701366676359407,"f1":0.5945945945945946,"true_positive":11,"false_positive":7,"false_negative":8,"true_negative":174,"positive_prevalence":0.095,"majority_accuracy_baseline":0.905,"constant_brier_baseline":0.085975,"brier_skill":0.41813995409457405,"baseline_adjusted_skill":0.5402733352718814,"baseline_adjusted_score":0.5402733352718814}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9933333333333333,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.9899999999999998,"baseline_adjusted_score":0.9899999999999998}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.006584803601323534,"accuracy":0.99,"positive_recall":1,"specificity":0.98,"precision":0.9803921568627451,"balanced_accuracy":0.99,"f1":0.9900990099009901,"true_positive":150,"false_positive":3,"false_negative":0,"true_negative":147,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9736607855947058,"baseline_adjusted_skill":0.98,"baseline_adjusted_score":0.98}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9933333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.991111111111111,"baseline_adjusted_score":0.991111111111111}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.055737346685108546,"accuracy":0.9233333333333333,"positive_recall":0.9933333333333333,"specificity":0.8533333333333334,"precision":0.8713450292397661,"balanced_accuracy":0.9233333333333333,"f1":0.9283489096573209,"true_positive":149,"false_positive":22,"false_negative":1,"true_negative":128,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7770506132595658,"baseline_adjusted_skill":0.8466666666666667,"baseline_adjusted_score":0.8466666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.04030633909054851,"normalized_score_rmse":0.10857148519559394,"constant_score_baseline":0.5,"constant_mae_baseline":0.3333333333333333,"baseline_adjusted_skill":0.8790809827283544,"baseline_adjusted_score":0.8790809827283544}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.48,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.30666666666666664,"baseline_adjusted_score":0.30666666666666664}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.23622149709503196,"accuracy":0.6133333333333333,"positive_recall":0.46,"specificity":0.7666666666666667,"precision":0.6634615384615384,"balanced_accuracy":0.6133333333333334,"f1":0.5433070866141733,"true_positive":69,"false_positive":35,"false_negative":81,"true_negative":115,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.05511401161987217,"baseline_adjusted_skill":0.2266666666666668,"baseline_adjusted_score":0.2266666666666668}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.2817145609576762,"normalized_score_rmse":0.33642104665797884,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":0.060951463474412604,"baseline_adjusted_score":0.060951463474412604}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9933333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3,"baseline_adjusted_skill":0.9904761904761906,"baseline_adjusted_score":0.9904761904761906}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0053639198122508915,"accuracy":0.9966666666666667,"positive_recall":0.9932885906040269,"specificity":1,"precision":1,"balanced_accuracy":0.9966442953020134,"f1":0.9966329966329966,"true_positive":148,"false_positive":0,"false_negative":1,"true_negative":151,"positive_prevalence":0.49666666666666665,"majority_accuracy_baseline":0.5033333333333334,"constant_brier_baseline":0.2499888888888889,"brier_skill":0.9785433671228686,"baseline_adjusted_skill":0.9932885906040267,"baseline_adjusted_score":0.9932885906040267}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1090242405842801,"normalized_score_rmse":0.15206898772275068,"constant_score_baseline":0.5,"constant_mae_baseline":0.36666666666666664,"baseline_adjusted_skill":0.7026611620428724,"baseline_adjusted_score":0.7026611620428724}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8866666666666667,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.2866666666666667,"baseline_adjusted_skill":0.8411214953271029,"baseline_adjusted_score":0.8411214953271029}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06807986434858435,"accuracy":0.9233333333333333,"positive_recall":0.9617834394904459,"specificity":0.8811188811188811,"precision":0.8988095238095238,"balanced_accuracy":0.9214511603046636,"f1":0.9292307692307692,"true_positive":151,"false_positive":17,"false_negative":6,"true_negative":126,"positive_prevalence":0.5233333333333333,"majority_accuracy_baseline":0.5233333333333333,"constant_brier_baseline":0.24945555555555557,"brier_skill":0.7270861969902191,"baseline_adjusted_skill":0.8429023206093271,"baseline_adjusted_score":0.8429023206093271}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.17274585040944823,"normalized_score_rmse":0.2154569776976249,"constant_score_baseline":0.5,"constant_mae_baseline":0.3016666666666667,"baseline_adjusted_skill":0.427361821847133,"baseline_adjusted_score":0.427361821847133}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/go_emotions / test · binary_brier. Population 100 cases / 2800 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06087845000033597,"accuracy":0.8829702380952381,"positive_recall":0.634857653010614,"specificity":0.8988088830349735,"precision":0.28596961572832885,"balanced_accuracy":0.7668332680227938,"f1":0.39431933705061456,"true_positive":106.66666666666666,"false_positive":266.3333333333333,"false_negative":61.35,"true_negative":2365.65,"positive_prevalence":0.060005952380952375,"majority_accuracy_baseline":0.9399940476190476,"constant_brier_baseline":0.03220097218679138,"brier_skill":-0.8905780125889458,"baseline_adjusted_skill":0.5336665360455877,"baseline_adjusted_score":0.5336665360455877}. Counts: {"cases": 100, "expected": 2800, "succeeded": 2800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gpqa-ja-main-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.5733333333333334,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.43111111111111117,"baseline_adjusted_score":0.43111111111111117}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gretel_pii / test · target_mass_at_prediction. Population 100 cases / 353 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.5155807365439093,"uniform_accuracy_baseline":0.027777777777777776,"fixed_answer_accuracy_baseline":0.1274787535410765,"baseline_adjusted_skill":0.44480519480519476,"baseline_adjusted_score":0.44480519480519476}. Counts: {"cases": 100, "expected": 353, "succeeded": 353, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gsm8k / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.91,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3,"baseline_adjusted_skill":0.8714285714285717,"baseline_adjusted_score":0.8714285714285717}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hans / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03808510510132626,"accuracy":0.94,"positive_recall":1,"specificity":0.88,"precision":0.8928571428571429,"balanced_accuracy":0.94,"f1":0.9433962264150944,"true_positive":50,"false_positive":6,"false_negative":0,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.847659579594695,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hatecheck / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.004919692551069,"accuracy":0.99,"positive_recall":0.98,"specificity":1,"precision":1,"balanced_accuracy":0.99,"f1":0.98989898989899,"true_positive":49,"false_positive":0,"false_negative":1,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.980321229795724,"baseline_adjusted_skill":0.98,"baseline_adjusted_score":0.98}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hh_rlhf / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.72,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.61,"baseline_adjusted_skill":0.282051282051282,"baseline_adjusted_score":0.282051282051282}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hwu64 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.015625,"fixed_answer_accuracy_baseline":0.06,"baseline_adjusted_skill":0.9680851063829787,"baseline_adjusted_score":0.9680851063829787}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/impli / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04750955868800842,"accuracy":0.93,"positive_recall":0.94,"specificity":0.92,"precision":0.9215686274509803,"balanced_accuracy":0.9299999999999999,"f1":0.9306930693069307,"true_positive":47,"false_positive":4,"false_negative":3,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8099617652479663,"baseline_adjusted_skill":0.8599999999999999,"baseline_adjusted_score":0.8599999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamc-qa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.55,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.4000000000000001,"baseline_adjusted_score":0.4000000000000001}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamp-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.7233333333333334,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.5850000000000001,"baseline_adjusted_score":0.5850000000000001}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/janli-choice / test · target_mass_at_prediction. Population 285 cases / 285 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8526315789473684,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5263157894736842,"baseline_adjusted_skill":0.6888888888888889,"baseline_adjusted_score":0.6888888888888889}. Counts: {"cases": 285, "expected": 285, "succeeded": 285, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 285

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcola-noul / test · binary_brier. Population 299 cases / 299 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.2036080262927082,"accuracy":0.68561872909699,"positive_recall":0.8533333333333334,"specificity":0.5167785234899329,"precision":0.64,"balanced_accuracy":0.6850559284116331,"f1":0.7314285714285714,"true_positive":128,"false_positive":72,"false_negative":22,"true_negative":77,"positive_prevalence":0.5016722408026756,"majority_accuracy_baseline":0.5016722408026756,"constant_brier_baseline":0.24999720361069788,"brier_skill":0.18555878485036226,"baseline_adjusted_skill":0.3701118568232662,"baseline_adjusted_score":0.3701118568232662}. Counts: {"cases": 299, "expected": 299, "succeeded": 299, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 299

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcommonsenseqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9566666666666667,"uniform_accuracy_baseline":0.2,"fixed_answer_accuracy_baseline":0.2,"baseline_adjusted_skill":0.9458333333333332,"baseline_adjusted_score":0.9458333333333332}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jculture-mcq-noul / test · binary_brier. Population 283 cases / 283 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.23076362672664025,"accuracy":0.6819787985865724,"positive_recall":0.7819548872180451,"specificity":0.5933333333333334,"precision":0.6303030303030303,"balanced_accuracy":0.6876441102756892,"f1":0.697986577181208,"true_positive":104,"false_positive":61,"false_negative":29,"true_negative":89,"positive_prevalence":0.46996466431095407,"majority_accuracy_baseline":0.5300353356890459,"constant_brier_baseline":0.24909787861004634,"brier_skill":0.0736026014581509,"baseline_adjusted_skill":0.37528822055137834,"baseline_adjusted_score":0.37528822055137834}. Counts: {"cases": 283, "expected": 283, "succeeded": 283, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 283

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmmlu-japan-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8666666666666667,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.8222222222222223,"baseline_adjusted_score":0.8222222222222223}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmteb-retrieval / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9766666666666667,"uniform_accuracy_baseline":0.20616666666666666,"fixed_answer_accuracy_baseline":0.21333333333333335,"baseline_adjusted_skill":0.9703389830508474,"baseline_adjusted_score":0.9703389830508474}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jnli-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8833333333333333,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.825,"baseline_adjusted_score":0.825}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsem-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.78,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.37333333333333335,"baseline_adjusted_skill":0.648936170212766,"baseline_adjusted_score":0.648936170212766}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsick-sts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1342312839355505,"normalized_score_rmse":0.16584751488167446,"constant_score_baseline":0.5,"constant_mae_baseline":0.26208333333333333,"baseline_adjusted_skill":0.4878297592284242,"baseline_adjusted_score":0.4878297592284242}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.07815546250914786,"normalized_score_rmse":0.10236854217663348,"constant_score_baseline":0.52,"constant_mae_baseline":0.24733333333333335,"baseline_adjusted_skill":0.6840075639791866,"baseline_adjusted_score":0.6840075639791866}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/koniq-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15299839671645668,"normalized_score_rmse":0.16914636750503914,"constant_score_baseline":0.578319964349225,"constant_mae_baseline":0.11212962028602311,"baseline_adjusted_skill":-0.36447797046119,"baseline_adjusted_score":0}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/kuci-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9033333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.8711111111111111,"baseline_adjusted_score":0.8711111111111111}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__ag_news / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.94,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.33,"baseline_adjusted_skill":0.9104477611940298,"baseline_adjusted_score":0.9104477611940298}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__enron_spam / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.055423441156195234,"accuracy":0.94,"positive_recall":0.92,"specificity":0.96,"precision":0.9583333333333334,"balanced_accuracy":0.94,"f1":0.9387755102040817,"true_positive":46,"false_positive":2,"false_negative":4,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.778306235375219,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · target_mass_at_prediction. Population 100 cases / 148 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.480001222972973,"uniform_accuracy_baseline":0.23445945945945948,"fixed_answer_accuracy_baseline":0.2623439054054054,"baseline_adjusted_skill":0.2950661143621256,"baseline_adjusted_score":0.2950661143621256}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · binary_brier. Population 100 cases / 152 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06202742784917273,"accuracy":0.722456125,"positive_recall":0.8224529880313062,"specificity":0.6272369108876282,"precision":0.6775185222222222,"balanced_accuracy":0.7248449494594672,"f1":0.7429836223409843,"true_positive":60.976667,"false_positive":29.023333,"false_negative":13.163336,"true_negative":48.836664,"positive_prevalence":0.4877631776315789,"majority_accuracy_baseline":0.512236822368421,"constant_brier_baseline":0.09451910039768556,"brier_skill":0.34375774221089006,"baseline_adjusted_skill":0.4496898989189344,"baseline_adjusted_score":0.4496898989189344}. Counts: {"cases": 100, "expected": 152, "succeeded": 152, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · normalized_expected_score_mae. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.07914787870241542,"normalized_score_rmse":0.11385206742628579,"constant_score_baseline":0.5841668333333333,"constant_mae_baseline":0.21700552708333334,"baseline_adjusted_skill":0.6352725215518522,"baseline_adjusted_score":0.6352725215518522}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lexcomp / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1580705202163836,"accuracy":0.78,"positive_recall":0.84,"specificity":0.72,"precision":0.75,"balanced_accuracy":0.78,"f1":0.7924528301886793,"true_positive":42,"false_positive":14,"false_negative":8,"true_negative":36,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.3677179191344656,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/logical_entailment / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.07955627094368,"accuracy":0.89,"positive_recall":0.82,"specificity":0.96,"precision":0.9534883720930233,"balanced_accuracy":0.8899999999999999,"f1":0.8817204301075269,"true_positive":41,"false_positive":2,"false_negative":9,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.68177491622528,"baseline_adjusted_skill":0.7799999999999998,"baseline_adjusted_score":0.7799999999999998}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lonli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.9836065573770492,"baseline_adjusted_score":0.9836065573770492}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/massive / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.94,"uniform_accuracy_baseline":0.036111111111111115,"fixed_answer_accuracy_baseline":0.08833333333333333,"baseline_adjusted_skill":0.9341864716636197,"baseline_adjusted_score":0.9341864716636197}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mintaka-ja-boolean-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.12167587994831791,"accuracy":0.84,"positive_recall":0.8666666666666667,"specificity":0.8133333333333334,"precision":0.8227848101265823,"balanced_accuracy":0.8400000000000001,"f1":0.8441558441558441,"true_positive":130,"false_positive":28,"false_negative":20,"true_negative":122,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5132964802067284,"baseline_adjusted_skill":0.6800000000000002,"baseline_adjusted_score":0.6800000000000002}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/miqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.56,"baseline_adjusted_skill":0.909090909090909,"baseline_adjusted_score":0.909090909090909}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmlu-prox-ja-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.6833333333333333,"uniform_accuracy_baseline":0.11021428571428572,"fixed_answer_accuracy_baseline":0.12554761904761905,"baseline_adjusted_skill":0.6378685980341439,"baseline_adjusted_score":0.6378685980341439}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmmlu-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.7833333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.7111111111111111,"baseline_adjusted_score":0.7111111111111111}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mtop / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.049879324215607396,"fixed_answer_accuracy_baseline":0.12442477876106194,"baseline_adjusted_skill":0.9086314938346473,"baseline_adjusted_score":0.9086314938346473}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mvtec-ad-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.189112379287752,"accuracy":0.7133333333333334,"positive_recall":0.6157407407407407,"specificity":0.9642857142857143,"precision":0.9779411764705882,"balanced_accuracy":0.7900132275132274,"f1":0.7556818181818182,"true_positive":133,"false_positive":3,"false_negative":83,"true_negative":81,"positive_prevalence":0.72,"majority_accuracy_baseline":0.72,"constant_brier_baseline":0.2016,"brier_skill":0.06194256305678569,"baseline_adjusted_skill":0.5800264550264549,"baseline_adjusted_score":0.5800264550264549}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nights-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.94,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5033333333333333,"baseline_adjusted_skill":0.8791946308724832,"baseline_adjusted_score":0.8791946308724832}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nlsat / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.3058028762936743,"accuracy":0.5,"positive_recall":0.04,"specificity":0.96,"precision":0.5,"balanced_accuracy":0.5,"f1":0.07407407407407407,"true_positive":2,"false_positive":2,"false_negative":48,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.22321150517469723,"baseline_adjusted_skill":0,"baseline_adjusted_score":0}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nlvr2-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03420611729946568,"accuracy":0.95,"positive_recall":0.975,"specificity":0.9214285714285714,"precision":0.9341317365269461,"balanced_accuracy":0.9482142857142857,"f1":0.9541284403669725,"true_positive":156,"false_positive":11,"false_negative":4,"true_negative":129,"positive_prevalence":0.5333333333333333,"majority_accuracy_baseline":0.5333333333333333,"constant_brier_baseline":0.2488888888888889,"brier_skill":0.8625647072789325,"baseline_adjusted_skill":0.8964285714285714,"baseline_adjusted_score":0.8964285714285714}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · target_mass_at_prediction. Population 36 cases / 36 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8888888888888888,"uniform_accuracy_baseline":0.23888888888888887,"fixed_answer_accuracy_baseline":0.2916666666666667,"baseline_adjusted_skill":0.8431372549019607,"baseline_adjusted_score":0.8431372549019607}. Counts: {"cases": 36, "expected": 36, "succeeded": 36, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 36

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · binary_brier. Population 64 cases / 64 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03493231304373406,"accuracy":0.9375,"positive_recall":0.90625,"specificity":0.96875,"precision":0.9666666666666667,"balanced_accuracy":0.9375,"f1":0.9354838709677419,"true_positive":29,"false_positive":1,"false_negative":3,"true_negative":31,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8602707478250637,"baseline_adjusted_skill":0.875,"baseline_adjusted_score":0.875}. Counts: {"cases": 64, "expected": 64, "succeeded": 64, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 64

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__browser-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__citation-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.84,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.7575757575757576,"baseline_adjusted_score":0.7575757575757576}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__context-retention-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":2.757406120060519e-07,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.999998897037552,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8823529411764706,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3235294117647059,"baseline_adjusted_skill":0.8260869565217391,"baseline_adjusted_score":0.8260869565217391}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0007543802373552097,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":16,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.9969797055938242,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · normalized_expected_score_mae. Population 50 cases / 50 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1226706529674539,"normalized_score_rmse":0.26455110935645515,"constant_score_baseline":0.25,"constant_mae_baseline":0.33,"baseline_adjusted_skill":0.6282707485834731,"baseline_adjusted_score":0.6282707485834731}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · target_mass_at_prediction. Population 32 cases / 32 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.875,"uniform_accuracy_baseline":0.178125,"fixed_answer_accuracy_baseline":0.2875,"baseline_adjusted_skill":0.8245614035087719,"baseline_adjusted_score":0.8245614035087719}. Counts: {"cases": 32, "expected": 32, "succeeded": 32, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 32

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · binary_brier. Population 34 cases / 34 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":1.4411433884095679e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":17,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9999942354264464,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.0791397328900114,"normalized_score_rmse":0.11769298845126085,"constant_score_baseline":0.5,"constant_mae_baseline":0.14705882352941177,"baseline_adjusted_skill":0.46184981634792244,"baseline_adjusted_score":0.46184981634792244}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · target_mass_at_prediction. Population 48 cases / 48 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9791666666666666,"uniform_accuracy_baseline":0.13194444444444445,"fixed_answer_accuracy_baseline":0.6006944444444444,"baseline_adjusted_skill":0.9478260869565216,"baseline_adjusted_score":0.9478260869565216}. Counts: {"cases": 48, "expected": 48, "succeeded": 48, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 48

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · binary_brier. Population 52 cases / 52 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":8.02214233783269e-08,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":26,"false_positive":0,"false_negative":0,"true_negative":26,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9999996791143065,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 52, "expected": 52, "succeeded": 52, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 52

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":2.011588296018919e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":39,"false_positive":0,"false_negative":0,"true_negative":36,"positive_prevalence":0.52,"majority_accuracy_baseline":0.52,"constant_brier_baseline":0.24960000000000004,"brier_skill":0.9999919407520191,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.03263408259193828,"normalized_score_rmse":0.0701622206354808,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.8834497050287918,"baseline_adjusted_score":0.8834497050287918}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9117647058823529,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.2647058823529412,"baseline_adjusted_skill":0.8799999999999998,"baseline_adjusted_score":0.8799999999999998}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · binary_brier. Population 28 cases / 28 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.001254325742636735,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":14,"false_positive":0,"false_negative":0,"true_negative":14,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9949826970294531,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 28, "expected": 28, "succeeded": 28, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 28

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · normalized_expected_score_mae. Population 55 cases / 55 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.003133546330586173,"normalized_score_rmse":0.004217533075331826,"constant_score_baseline":0.3333333333333333,"constant_mae_baseline":0.1696969696969697,"baseline_adjusted_skill":0.9815344591233315,"baseline_adjusted_score":0.9815344591233315}. Counts: {"cases": 55, "expected": 55, "succeeded": 55, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 55

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · target_mass_at_prediction. Population 15 cases / 15 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.15555555555555553,"fixed_answer_accuracy_baseline":0.2,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 15, "expected": 15, "succeeded": 15, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 15

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · binary_brier. Population 85 cases / 85 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.01797495671746262,"accuracy":0.9764705882352941,"positive_recall":1,"specificity":0.95,"precision":0.9574468085106383,"balanced_accuracy":0.975,"f1":0.9782608695652174,"true_positive":45,"false_positive":2,"false_negative":0,"true_negative":38,"positive_prevalence":0.5294117647058824,"majority_accuracy_baseline":0.5294117647058824,"constant_brier_baseline":0.2491349480968858,"brier_skill":0.927850520953518,"baseline_adjusted_skill":0.95,"baseline_adjusted_score":0.95}. Counts: {"cases": 85, "expected": 85, "succeeded": 85, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 85

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.07805555555555556,"fixed_answer_accuracy_baseline":0.24,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · binary_brier. Population 50 cases / 50 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":9.919437877551573e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":35,"false_positive":0,"false_negative":0,"true_negative":15,"positive_prevalence":0.7,"majority_accuracy_baseline":0.7,"constant_brier_baseline":0.21,"brier_skill":0.9999527645815355,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.005535966335170954,"normalized_score_rmse":0.0072578322827118995,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.9802286916601037,"baseline_adjusted_score":0.9802286916601037}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · target_mass_at_prediction. Population 79 cases / 79 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8987341772151899,"uniform_accuracy_baseline":0.24240506329113923,"fixed_answer_accuracy_baseline":0.4177215189873418,"baseline_adjusted_skill":0.8260869565217391,"baseline_adjusted_score":0.8260869565217391}. Counts: {"cases": 79, "expected": 79, "succeeded": 79, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 79

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · binary_brier. Population 21 cases / 21 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":5.140042560143306e-07,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":11,"false_positive":0,"false_negative":0,"true_negative":10,"positive_prevalence":0.5238095238095238,"majority_accuracy_baseline":0.5238095238095238,"constant_brier_baseline":0.2494331065759637,"brier_skill":0.99999793931021,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 21, "expected": 21, "succeeded": 21, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 21

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · target_mass_at_prediction. Population 33 cases / 33 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9090909090909091,"uniform_accuracy_baseline":0.32373737373737377,"fixed_answer_accuracy_baseline":0.3939393939393939,"baseline_adjusted_skill":0.85,"baseline_adjusted_score":0.85}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.11036821105699655,"accuracy":0.8484848484848485,"positive_recall":0.8235294117647058,"specificity":0.875,"precision":0.875,"balanced_accuracy":0.8492647058823529,"f1":0.8484848484848485,"true_positive":14,"false_positive":2,"false_negative":3,"true_negative":14,"positive_prevalence":0.5151515151515151,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.5581213902901866,"baseline_adjusted_skill":0.6985294117647058,"baseline_adjusted_score":0.6985294117647058}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.0139703118872704,"normalized_score_rmse":0.04806969077323907,"constant_score_baseline":0.7083333333333333,"constant_mae_baseline":0.3431372549019608,"baseline_adjusted_skill":0.959286519642812,"baseline_adjusted_score":0.959286519642812}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__silent-failure-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0003641651984735879,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9985433392061056,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8133333333333332,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.35333333333333333,"baseline_adjusted_skill":0.7113402061855668,"baseline_adjusted_score":0.7113402061855668}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06214159309612637,"accuracy":0.92,"positive_recall":0.8648648648648649,"specificity":0.9736842105263158,"precision":0.9696969696969697,"balanced_accuracy":0.9192745376955904,"f1":0.9142857142857143,"true_positive":32,"false_positive":1,"false_negative":5,"true_negative":37,"positive_prevalence":0.49333333333333335,"majority_accuracy_baseline":0.5066666666666666,"constant_brier_baseline":0.24995555555555554,"brier_skill":0.75138943018086,"baseline_adjusted_skill":0.8385490753911808,"baseline_adjusted_score":0.8385490753911808}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__sponsor-segment-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.14285714285714285,"fixed_answer_accuracy_baseline":0.22,"baseline_adjusted_skill":0.9743589743589743,"baseline_adjusted_score":0.9743589743589743}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__tic_tac_toe-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.46,"uniform_accuracy_baseline":0.32416666666666666,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.18181818181818182,"baseline_adjusted_score":0.18181818181818182}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":2.0401249090706266e-05,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":16,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.9999183199990449,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · normalized_expected_score_mae. Population 33 cases / 33 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.003458478872279131,"normalized_score_rmse":0.006045903609988432,"constant_score_baseline":0.5,"constant_mae_baseline":0.2196969696969697,"baseline_adjusted_skill":0.9842579582365226,"baseline_adjusted_score":0.9842579582365226}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__agent_trace_observability / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.14399579103130544,"accuracy":0.82,"positive_recall":0.9090909090909091,"specificity":0.7948717948717948,"precision":0.5555555555555556,"balanced_accuracy":0.8519813519813519,"f1":0.6896551724137931,"true_positive":20,"false_positive":16,"false_negative":2,"true_negative":62,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.1716,"brier_skill":0.16086368862875622,"baseline_adjusted_skill":0.7039627039627039,"baseline_adjusted_score":0.7039627039627039}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__customer_service / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.08685281732478772,"accuracy":0.88,"positive_recall":0.6756756756756757,"specificity":1,"precision":1,"balanced_accuracy":0.8378378378378378,"f1":0.8064516129032258,"true_positive":25,"false_positive":0,"false_negative":12,"true_negative":63,"positive_prevalence":0.37,"majority_accuracy_baseline":0.63,"constant_brier_baseline":0.23310000000000003,"brier_skill":0.6274010410777018,"baseline_adjusted_skill":0.6756756756756757,"baseline_adjusted_score":0.6756756756756757}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__invoice_processing / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05900880637940797,"accuracy":0.93,"positive_recall":0.8205128205128205,"specificity":1,"precision":1,"balanced_accuracy":0.9102564102564102,"f1":0.9014084507042254,"true_positive":32,"false_positive":0,"false_negative":7,"true_negative":61,"positive_prevalence":0.39,"majority_accuracy_baseline":0.61,"constant_brier_baseline":0.2379,"brier_skill":0.7519596200949644,"baseline_adjusted_skill":0.8205128205128205,"baseline_adjusted_score":0.8205128205128205}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__security_incidents / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05307477369128292,"accuracy":0.92,"positive_recall":0.9032258064516129,"specificity":0.927536231884058,"precision":0.8484848484848485,"balanced_accuracy":0.9153810191678354,"f1":0.875,"true_positive":28,"false_positive":5,"false_negative":3,"true_negative":64,"positive_prevalence":0.31,"majority_accuracy_baseline":0.69,"constant_brier_baseline":0.21389999999999998,"brier_skill":0.7518710907373403,"baseline_adjusted_skill":0.8307620383356709,"baseline_adjusted_score":0.8307620383356709}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/openbookqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.28,"baseline_adjusted_skill":0.9583333333333333,"baseline_adjusted_score":0.9583333333333333}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/patent_similarity / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.10974196754523874,"normalized_score_rmse":0.14259951367031884,"constant_score_baseline":0.25,"constant_mae_baseline":0.2325,"baseline_adjusted_skill":0.5279915374398334,"baseline_adjusted_score":0.5279915374398334}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/paws / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.037373246544903455,"accuracy":0.96,"positive_recall":0.98,"specificity":0.94,"precision":0.9423076923076923,"balanced_accuracy":0.96,"f1":0.9607843137254902,"true_positive":49,"false_positive":3,"false_negative":1,"true_negative":47,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8505070138203862,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/plane / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.3161404520793532,"accuracy":0.56,"positive_recall":0.88,"specificity":0.24,"precision":0.5365853658536586,"balanced_accuracy":0.56,"f1":0.6666666666666666,"true_positive":44,"false_positive":38,"false_negative":6,"true_negative":12,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.2645618083174128,"baseline_adjusted_skill":0.1200000000000001,"baseline_adjusted_score":0.1200000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/poem_sentiment / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.78,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.65,"baseline_adjusted_skill":0.37142857142857144,"baseline_adjusted_score":0.37142857142857144}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/pope-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.08783151651222229,"accuracy":0.89,"positive_recall":0.82,"specificity":0.96,"precision":0.9534883720930233,"balanced_accuracy":0.8899999999999999,"f1":0.8817204301075269,"true_positive":123,"false_positive":6,"false_negative":27,"true_negative":144,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6486739339511108,"baseline_adjusted_skill":0.7799999999999998,"baseline_adjusted_score":0.7799999999999998}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/qasper / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06851608025153291,"accuracy":0.91,"positive_recall":0.94,"specificity":0.88,"precision":0.8867924528301887,"balanced_accuracy":0.9099999999999999,"f1":0.912621359223301,"true_positive":47,"false_positive":6,"false_negative":3,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7259356789938683,"baseline_adjusted_skill":0.8199999999999998,"baseline_adjusted_score":0.8199999999999998}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/quartz / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.58,"baseline_adjusted_skill":0.9047619047619047,"baseline_adjusted_score":0.9047619047619047}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/robust_lr / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.8333333333333334,"baseline_adjusted_score":0.8333333333333334}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ruletaker / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.008408899701913178,"accuracy":0.99,"positive_recall":1,"specificity":0.98,"precision":0.9803921568627451,"balanced_accuracy":0.99,"f1":0.9900990099009901,"true_positive":50,"false_positive":1,"false_negative":0,"true_negative":49,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9663644011923473,"baseline_adjusted_skill":0.98,"baseline_adjusted_score":0.98}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/rvlcdip-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.75,"uniform_accuracy_baseline":0.0625,"fixed_answer_accuracy_baseline":0.10666666666666667,"baseline_adjusted_skill":0.7201492537313433,"baseline_adjusted_score":0.7201492537313433}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.20851190476190473,"fixed_answer_accuracy_baseline":0.22333333333333333,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.00330116375082126,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.986795344996715,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.014762235824475141,"normalized_score_rmse":0.04755840615435259,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":0.9507925472517496,"baseline_adjusted_score":0.9507925472517496}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.22851190476190475,"fixed_answer_accuracy_baseline":0.24333333333333332,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0008518690589899931,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9965925237640401,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.015208103106630703,"normalized_score_rmse":0.033373297648818606,"constant_score_baseline":0.4222222222222222,"constant_mae_baseline":0.3042222222222222,"baseline_adjusted_skill":0.9500098875238582,"baseline_adjusted_score":0.9500098875238582}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scicite / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.74,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.5806451612903225,"baseline_adjusted_score":0.5806451612903225}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scienceqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9966666666666667,"uniform_accuracy_baseline":0.3467777777777778,"fixed_answer_accuracy_baseline":0.36,"baseline_adjusted_skill":0.9947916666666667,"baseline_adjusted_score":0.9947916666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scitail / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05372433692564779,"accuracy":0.93,"positive_recall":0.96,"specificity":0.9,"precision":0.9056603773584906,"balanced_accuracy":0.9299999999999999,"f1":0.9320388349514563,"true_positive":48,"false_positive":5,"false_negative":2,"true_negative":45,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7851026522974088,"baseline_adjusted_skill":0.8599999999999999,"baseline_adjusted_score":0.8599999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scone / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.5,"baseline_adjusted_skill":0.98,"baseline_adjusted_score":0.98}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sdoh_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.02856711970351792,"accuracy":0.96,"positive_recall":0.92,"specificity":1,"precision":1,"balanced_accuracy":0.96,"f1":0.9583333333333334,"true_positive":46,"false_positive":0,"false_negative":4,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8857315211859283,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sgd / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.63,"uniform_accuracy_baseline":0.3533333333333333,"fixed_answer_accuracy_baseline":0.41,"baseline_adjusted_skill":0.3728813559322034,"baseline_adjusted_score":0.3728813559322034}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/snli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9677419354838709,"baseline_adjusted_score":0.9677419354838709}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sobaco-culture-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8366666666666667,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.755,"baseline_adjusted_score":0.755}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/spartqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9193548387096774,"baseline_adjusted_score":0.9193548387096774}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/stepgame / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.75,"uniform_accuracy_baseline":0.1111111111111111,"fixed_answer_accuracy_baseline":0.16,"baseline_adjusted_skill":0.7023809523809523,"baseline_adjusted_score":0.7023809523809523}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sugarcrepe-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9633333333333334,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.58,"baseline_adjusted_skill":0.9126984126984128,"baseline_adjusted_score":0.9126984126984128}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04421234609295232,"accuracy":0.8278749999999999,"positive_recall":0.6657877389584708,"specificity":0.8658028690421101,"precision":0.5372340425531915,"balanced_accuracy":0.7657953040002905,"f1":0.5946423314689432,"true_positive":50.5,"false_positive":43.5,"false_negative":25.35,"true_negative":280.65,"positive_prevalence":0.189625,"majority_accuracy_baseline":0.8103750000000001,"constant_brier_baseline":0.07612360937500001,"brier_skill":0.4192032346344282,"baseline_adjusted_skill":0.531590608000581,"baseline_adjusted_score":0.531590608000581}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15808511692445507,"normalized_score_rmse":0.19250493918683484,"constant_score_baseline":0,"constant_mae_baseline":0.21375,"baseline_adjusted_skill":0.2604205056165845,"baseline_adjusted_score":0.2604205056165845}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05810924923632298,"accuracy":0.799625,"positive_recall":0.43834974214721045,"specificity":0.9309698312595875,"precision":0.6977611940298507,"balanced_accuracy":0.684659786703399,"f1":0.5384393895767348,"true_positive":46.75,"false_positive":20.25,"false_negative":59.9,"true_negative":273.1,"positive_prevalence":0.266625,"majority_accuracy_baseline":0.733375,"constant_brier_baseline":0.115654859375,"brier_skill":0.497563271008707,"baseline_adjusted_skill":0.369319573406798,"baseline_adjusted_score":0.369319573406798}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1566024426390451,"normalized_score_rmse":0.21387950423007426,"constant_score_baseline":0.25,"constant_mae_baseline":0.26875,"baseline_adjusted_skill":0.4172932366919252,"baseline_adjusted_score":0.4172932366919252}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.046282981344126964,"accuracy":0.807625,"positive_recall":0.7373601789709173,"specificity":0.8348655680832611,"precision":0.6338461538461538,"balanced_accuracy":0.7861128735270893,"f1":0.6816959669079629,"true_positive":82.4,"false_positive":47.6,"false_negative":29.35,"true_negative":240.65,"positive_prevalence":0.279375,"majority_accuracy_baseline":0.7206250000000001,"constant_brier_baseline":0.11315585937500001,"brier_skill":0.5909802497213639,"baseline_adjusted_skill":0.5722257470541785,"baseline_adjusted_score":0.5722257470541785}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1436553950039977,"normalized_score_rmse":0.18047196478730765,"constant_score_baseline":0.25,"constant_mae_baseline":0.264375,"baseline_adjusted_skill":0.4566226193702215,"baseline_adjusted_score":0.4566226193702215}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.026364232266840367,"accuracy":0.9071250000000001,"positive_recall":0.8371830985915493,"specificity":0.9270682730923695,"precision":0.7659793814432989,"balanced_accuracy":0.8821256858419594,"f1":0.8,"true_positive":74.3,"false_positive":22.7,"false_negative":14.45,"true_negative":288.55,"positive_prevalence":0.221875,"majority_accuracy_baseline":0.778125,"constant_brier_baseline":0.12674023437499998,"brier_skill":0.7919821405029623,"baseline_adjusted_skill":0.7642513716839188,"baseline_adjusted_score":0.7642513716839188}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.11857593281909555,"normalized_score_rmse":0.16258857525205073,"constant_score_baseline":0,"constant_mae_baseline":0.225625,"baseline_adjusted_skill":0.47445569941675103,"baseline_adjusted_score":0.47445569941675103}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03598072150194952,"accuracy":0.8364999999999999,"positive_recall":0.7391304347826086,"specificity":0.8715159755268524,"precision":0.6741379310344828,"balanced_accuracy":0.8053232051547305,"f1":0.7051397655545537,"true_positive":78.2,"false_positive":37.8,"false_negative":27.6,"true_negative":256.4,"positive_prevalence":0.2645,"majority_accuracy_baseline":0.7355,"constant_brier_baseline":0.11362725,"brier_skill":0.6833442549920946,"baseline_adjusted_skill":0.6106464103094611,"baseline_adjusted_score":0.6106464103094611}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1290143633583338,"normalized_score_rmse":0.1701072620331327,"constant_score_baseline":0.25,"constant_mae_baseline":0.25125,"baseline_adjusted_skill":0.48650999658374605,"baseline_adjusted_score":0.48650999658374605}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05524158928395561,"accuracy":0.8292499999999999,"positive_recall":0.41318537859007837,"specificity":0.9277983920841063,"precision":0.5754545454545454,"balanced_accuracy":0.6704918853370924,"f1":0.48100303951367773,"true_positive":31.65,"false_positive":23.35,"false_negative":44.95,"true_negative":300.05,"positive_prevalence":0.19149999999999998,"majority_accuracy_baseline":0.8085,"constant_brier_baseline":0.09190274999999999,"brier_skill":0.3989125539338527,"baseline_adjusted_skill":0.3409837706741847,"baseline_adjusted_score":0.3409837706741847}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1405947277949444,"normalized_score_rmse":0.20954337147070878,"constant_score_baseline":0,"constant_mae_baseline":0.206875,"baseline_adjusted_skill":0.3203880227434712,"baseline_adjusted_score":0.3203880227434712}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.030413781621165047,"accuracy":0.91125,"positive_recall":0.774390243902439,"specificity":0.9422133660331085,"precision":0.7519736842105263,"balanced_accuracy":0.8583018049677738,"f1":0.7630173564753003,"true_positive":57.15,"false_positive":18.849999999999998,"false_negative":16.65,"true_negative":307.35,"positive_prevalence":0.1845,"majority_accuracy_baseline":0.8155,"constant_brier_baseline":0.10922225,"brier_skill":0.721542253330571,"baseline_adjusted_skill":0.7166036099355475,"baseline_adjusted_score":0.7166036099355475}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.13897632075703295,"normalized_score_rmse":0.1926308381102775,"constant_score_baseline":0,"constant_mae_baseline":0.189375,"baseline_adjusted_skill":0.2661316395668226,"baseline_adjusted_score":0.2661316395668226}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · binary_brier. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05726705680636272,"accuracy":0.8467500000000001,"positive_recall":0.28152866242038216,"specificity":0.9847589424572318,"precision":0.8185185185185185,"balanced_accuracy":0.633143802438807,"f1":0.41895734597156403,"true_positive":22.1,"false_positive":4.8999999999999995,"false_negative":56.4,"true_negative":316.6,"positive_prevalence":0.19625,"majority_accuracy_baseline":0.80375,"constant_brier_baseline":0.09878593749999999,"brier_skill":0.4202914073031627,"baseline_adjusted_skill":0.26628760487761394,"baseline_adjusted_score":0.26628760487761394}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15475548440916756,"normalized_score_rmse":0.23491099899453785,"constant_score_baseline":0,"constant_mae_baseline":0.21125,"baseline_adjusted_skill":0.2674296596015736,"baseline_adjusted_score":0.2674296596015736}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · binary_brier. Population 392 cases / 392 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0863446676786046,"accuracy":0.7017857142857143,"positive_recall":0.9154804270462633,"specificity":0.5405102954341987,"precision":0.6005836575875486,"balanced_accuracy":0.727995361240231,"f1":0.725328947368421,"true_positive":154.35,"false_positive":102.65,"false_negative":14.25,"true_negative":120.75,"positive_prevalence":0.4301020408163265,"majority_accuracy_baseline":0.5698979591836735,"constant_brier_baseline":0.12359641815910039,"brier_skill":0.30139830130468015,"baseline_adjusted_skill":0.455990722480462,"baseline_adjusted_score":0.455990722480462}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · normalized_expected_score_mae. Population 392 cases / 392 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15629232532931114,"normalized_score_rmse":0.20336521526427828,"constant_score_baseline":0.5,"constant_mae_baseline":0.2755102040816326,"baseline_adjusted_skill":0.43271674510101876,"baseline_adjusted_score":0.43271674510101876}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · binary_brier. Population 800 cases / 800 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.027276068198263878,"accuracy":0.9216250000000001,"positive_recall":0.6624704956726987,"specificity":0.9705751226036559,"precision":0.8096153846153846,"balanced_accuracy":0.8165228091381773,"f1":0.7286888792730419,"true_positive":84.2,"false_positive":19.8,"false_negative":42.9,"true_negative":653.1,"positive_prevalence":0.158875,"majority_accuracy_baseline":0.841125,"constant_brier_baseline":0.09877748437499999,"brier_skill":0.7238635062347539,"baseline_adjusted_skill":0.6330456182763546,"baseline_adjusted_score":0.6330456182763546}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · normalized_expected_score_mae. Population 800 cases / 800 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.12006586679863394,"normalized_score_rmse":0.17113833315283686,"constant_score_baseline":0,"constant_mae_baseline":0.165,"baseline_adjusted_skill":0.2723280800082791,"baseline_adjusted_score":0.2723280800082791}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/tallyqa-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.03456971840709728,"normalized_score_rmse":0.07136444415063241,"constant_score_baseline":0.1111111111111111,"constant_mae_baseline":0.10629629629629629,"baseline_adjusted_skill":0.674779652616158,"baseline_adjusted_score":0.674779652616158}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/temporal_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9193548387096774,"baseline_adjusted_score":0.9193548387096774}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/tracie / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.15135541564467397,"accuracy":0.78,"positive_recall":0.74,"specificity":0.82,"precision":0.8043478260869565,"balanced_accuracy":0.78,"f1":0.7708333333333334,"true_positive":37,"false_positive":9,"false_negative":13,"true_negative":41,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.3945783374213041,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ud_ewt / test · target_mass_at_prediction. Population 100 cases / 1509 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.27899271040424123,"uniform_accuracy_baseline":0.058823529411764705,"fixed_answer_accuracy_baseline":0.14910536779324055,"baseline_adjusted_skill":0.15264797507788164,"baseline_adjusted_score":0.15264797507788164}. Counts: {"cases": 100, "expected": 1509, "succeeded": 1509, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/visa-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.2952410705460945,"accuracy":0.5166666666666667,"positive_recall":0.16666666666666666,"specificity":1,"precision":1,"balanced_accuracy":0.5833333333333334,"f1":0.2857142857142857,"true_positive":29,"false_positive":0,"false_negative":145,"true_negative":126,"positive_prevalence":0.58,"majority_accuracy_baseline":0.58,"constant_brier_baseline":0.24359999999999998,"brier_skill":-0.2119912583994028,"baseline_adjusted_skill":0.16666666666666674,"baseline_adjusted_score":0.16666666666666674}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/vsr-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.10921997170536253,"accuracy":0.8366666666666667,"positive_recall":0.910828025477707,"specificity":0.7552447552447552,"precision":0.8033707865168539,"balanced_accuracy":0.8330363903612311,"f1":0.8537313432835821,"true_positive":143,"false_positive":35,"false_negative":14,"true_negative":108,"positive_prevalence":0.5233333333333333,"majority_accuracy_baseline":0.5233333333333333,"constant_brier_baseline":0.24945555555555557,"brier_skill":0.5621666093500233,"baseline_adjusted_skill":0.6660727807224622,"baseline_adjusted_score":0.6660727807224622}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winobias / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.8913043478260868,"baseline_adjusted_score":0.8913043478260868}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winowhy / test · binary_brier. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.10241718218464087,"accuracy":0.84,"positive_recall":0.78,"specificity":0.9,"precision":0.8863636363636364,"balanced_accuracy":0.8400000000000001,"f1":0.8297872340425532,"true_positive":39,"false_positive":5,"false_negative":11,"true_negative":45,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5903312712614366,"baseline_adjusted_skill":0.6800000000000002,"baseline_adjusted_score":0.6800000000000002}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/wiqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact perplexity-ai/pplx-decider-v1.2; adapter pplx-decider; settings {"questions_per_call": 1}. Run perplexity-ai__pplx-decider-v1.2.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.56,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.37,"baseline_adjusted_skill":0.3015873015873017,"baseline_adjusted_score":0.3015873015873017}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified