Examenos

67.9% · raw 67.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. February–May 2025; pass@1 selected as best of two full runs, not pass@2. Headline from the updated text/v1 figures.

66.4% · raw 66.4 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 1 of 2. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

67.9% · raw 67.9 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 2 of 2. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

78.7% · raw 78.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Greedy code completion pass@1; not HumanEval+. Headline from the updated text/v1 figures.

54% · raw 54 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Exactly 50 tasks through Blockway Codeway, identical limits, each model default sampling. Not full 500-task Verified. Headline from the updated text/v1 figures.

82.1% · raw 82.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Non-live irrelevance category, official harness, temperature0.001. Headline from the updated text/v1 figures.

76.7% · raw 76.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Airline/retail/telecom 120 tasks; 13 retail tasks requiring an LLM judge count failed for every model; agent temperature0, Qwen3.8-27B user simulator. Headline from the updated text/v1 figures.

94% · raw 94 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Non-live parallel category, official harness, temperature0.001. Headline from the updated text/v1 figures.

80% · raw 80 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Select best of two avg@4 runs; not pass@4 or an average across both runs. Headline from the updated text/v1 figures.

78.3% · raw 78.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 1 of 2. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

80% · raw 80 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 2 of 2. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

96.8% · raw 96.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Published accuracy; thinking enabled. Headline from the updated text/v1 figures.

87.2% · raw 87.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Prompt-level strict accuracy; not instruction-level or loose accuracy. Headline from the updated text/v1 figures.

64.3% · raw 64.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Prompt-level strict accuracy. Headline from the updated text/v1 figures.

80.3% · raw 80.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. 6K thinking budget then forced answer; best of four runs per model. This protocol differs from unrestricted GPQA. Headline from the updated text/v1 figures.

79.3% · raw 79.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 1 of 4. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

78.8% · raw 78.8 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 2 of 4. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

80.3% · raw 80.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 3 of 4. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

78.8% · raw 78.8 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Individual published repeat 4 of 4. The selected best run remains the headline; these repeats are overlapping evidence, not separate benchmarks.

80% · raw 80 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Fixed stratified 1,400-question subset; not full MMLU-Pro. Headline from the updated text/v1 figures.

85.9% · raw 85.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway vendor evaluation. Thinking on, temperature0.6 and common output cap unless the task note specifies otherwise; cap-truncated answers count wrong. Exact execution date and numeric output cap unpublished. Fixed stratified 2,010-question subset; not full CMMLU. Headline from the updated text/v1 figures.

0 tasks · raw 0 tasks

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Zero of 50 tasks looped; Preview36/50 is a different unapproved checkpoint. Local serving measurements never populate hosted API speed.

120 tokens/step · raw 120 tokens/step

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Median reasoning step120 tokens; longest12K, no step≥16K. Efficiency observation, not reasoning quality. Local serving measurements never populate hosted API speed.

12000 tokens/step · raw 12000 tokens/step

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Published approximate12K longest thinking step. Local serving measurements never populate hosted API speed.

82 min/task · raw 82 min/task

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Median Codeway SWE mini task duration, not provider latency. Local serving measurements never populate hosted API speed.

100% · raw 100 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Needle retrieval100/100 at107K; not validated full262K context. Local serving measurements never populate hosted API speed.

75.9 tok/s · raw 75.9 tok/s

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Two48GB GPUs, BF16, DFlash2, greedy400-token JSON replies, single user. Local serving measurements never populate hosted API speed.

3 × · raw 3 ×

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Same BF16 setup; JSON speedup against plain decode. Local serving measurements never populate hosted API speed.

2.2 × · raw 2.2 ×

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Same BF16 setup; code workload. Local serving measurements never populate hosted API speed.

1.5 × · raw 1.5 ×

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Same BF16 setup; replies with thinking. Local serving measurements never populate hosted API speed.

1.4 × · raw 1.4 ×

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Blockway source-specific behavior/resource study. Same BF16 setup; Cantonese chat. Local serving measurements never populate hosted API speed.