Examenos

22 datasets, 3,210 items.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: 3210

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

22 datasets, 3,210 items; lower is better.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: 3210

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

400 × 5 questions.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: 2000

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

400 × 5 questions; lower is better.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: 2000

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

Client in AWS us-east-1.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

Client in AWS us-east-1.

Harness 0cfd276, one scorer, each provider through its own decision endpoint, October 2026.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-08 · Revision: Harness 0cfd276; read 2026-11-02 · Published observations; result reuse terms not specified

The knowledge and reasoning figure on the product page, which also says Jev leads this area.

Celeris decisions page; no peer value is printed on this line.

Printed in the vendor's own table.

Celeris's own run of decision-benching harness 0cfd276, one scorer for all providers, October 2026; every dataset pinned to a Hugging Face revision. Latency is from one client in AWS us-east-1. The Perplexity v1.1 27B and Jev 1.13.0 columns are unregistered systems and are not stored. Not ranked here.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-08 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified

Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest.

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17; 2026-11-01; 2026-11-02 · Published observations; result reuse terms not specified

AA Decisions v1 · AA-Omniscience-Decision accuracy

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · AA-Omniscience-Decision non-hallucination

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · HLE-MultiChoice

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · When2Call

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · WildGuardTest average F1

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · Median time per task

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

AA Decisions v1 · Cost per million tasks

Shown as Celeris-1 Decision, released October 2026; AA does not state a sub-version. AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished.

Manual browser observation of a source-rounded chart label on 2 November 2026; no automated AA adapter.

Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20; 2026-11-01; 2026-11-02 · Published summary observations; upstream task/result terms retained

Capability Score on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

Intelligence on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

Calibration on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

Cost Score on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

Modelled cost per 1,000 decisions on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

API latency, end to end on the rendered board.

Celeris-1 Decision (Celeris) · Closed API (weights not public). Board key celeris-1-decision; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-16; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://docs.celeris.ai.

Rendered board row.

Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained

Capability on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained

Composite on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained

Intelligence on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained

Calibration on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained

Speed axis on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained

Cost axis on the core image track.

Celeris-1 Decision (Celeris); board key celeris_1_decision; last measured 2026-10-16T20:29:42.417026+00:00.

Exact published core-track row.

Benchmark Heaven's image decision benchmark, core track of v0.3.0, built 2026-10-28T18:06:26.533607+00:00. Published core track only; dated carry and computer-use track are separate. Cite source_sha256 for the exact result bytes. Separate from JevBench text rows and the community vision index. H2O image rows are not stored here, because serving settings changed between releases and no row names the registered release. No Examenos recomputation.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-16 · Samples: Not published

Snapshot: 2026-10-28 · Revision: v0.3.0; source_sha256=90ec4137edf485c5f6927c14d22188a6cea26e3355f0fd0e93d68da251c5e5d8 · Published summary observations; upstream task/result terms retained