H2O-Lightning-4B Decision Model Results

H2O-Lightning-4B

Explore H2O-Lightning-4B: exact decision configuration, published specifications, source-linked benchmarks and API tariffs when available.

Published specifications

Provider: H2O.ai
Version: v1.4.2
System type: Trained checkpoint
Access: Open weights
License: Apache-2.0
Inputs: text, image
Decision types: Choice, Yes / no, Score, Box
Published size: 4.66B
Active parameters: Not published
Context / input limit: 40,960 tokens by default; 131,072 in its Decision Index recipe
Announced: 2026-10-02

Model card & announcement · Website

Model notes

Decision model from H2O.ai: one forward pass and one output token per decision, with a probability for every option. A fine-tune by its developer, not a third-party quantization. Announced date is the Hugging Face repository creation date; no separate announcement was found. Built on Qwen/Qwen3.5-4B. Tag v1.4.2 is ee4e0e624a; later commits change only the card. Repository created 2 October 2026; older omission surfaced by card commits on 1 November. Benchmark Heaven board rows for other releases and the community Decision Index row name earlier releases, so they are not stored here.

Checkpoint revision: 659fc412de8486dbc2477b0d1cb4121e97e72471

← All decision models · Compare this decision system →

11 recorded observations. Published results retain their source and setup; no Examenos evaluation was run. Missing results are not zero. How to read these comparisons →

Official / vendor · H2O.ai model cards (vendor measured)

H2O-Lightning-4B · Official / vendor · H2O.ai model cards (vendor measured)
Evaluation	Result	Evidence
H2O Lightning cards · JevBench 231 public items	89.2%	Setup & source Correct answers on the 231 public JevBench items, read through JevBench's own CLI in the setup each card names for JevBench. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; two-way default, read 26 October 2026 on v1.4.1 serving, which the card says gives the same text answers. 206 of 231 correct on the card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 231 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · Decision Index 0.3 public text index	63.21	Setup & source The kit's chance-corrected index on the public text suite (140,178 scored rows), in the setup each card names for the Decision Index. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; two-way default, yes/no floor off as shipped, read 28 October 2026. Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 140178 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · Decision Index 0.3 public vision index	69.87	Setup & source The kit's index on the public vision suite (12,210 rows), in the setup each card names for the Decision Index. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; two-way default at v1.4.2's image cap, read 28 October 2026. Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 12210 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · Calibration error, text rows	7.01 pp	Setup & source Calibration error on Decision Index text rows, published on a 0–1 scale; lower is better. Each result quotes its card's wording. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; card label: calibration error on the Decision Index text rows, 274,729 questions. Published as 0.0701 on a 0–1 scale. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 274729 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · One-way reading · JevBench 231 public items	87.4%	Setup & source The same JevBench items read one-way, the fallback on stock vLLM. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; one-way (TWO_WAY=0). 202 of 231 correct on the card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 231 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · One-way reading · Decision Index 0.3 public text index	60.86	Setup & source The public text suite read one-way, the fallback on stock vLLM. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; one-way (TWO_WAY=0). Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 140178 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · One-way reading · Decision Index 0.3 public vision index	67.56	Setup & source The public vision suite read one-way, the fallback on stock vLLM. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; one-way (TWO_WAY=0). Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 12210 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · One-way reading · Calibration error, text rows	8.01 pp	Setup & source Calibration error of the one-way reading, published on a 0–1 scale; lower is better. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; one-way (TWO_WAY=0). Published as 0.0801 on a 0–1 scale. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · RefCOCOg box Acc@0.5	85.7%	Setup & source Box accuracy on RefCOCOg val, 600 referring expressions, excluding board images. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; RefCOCOg val, 600 referring expressions, no board images. Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 600 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · RefCOCOg box mean IoU	79.80%	Setup & source Mean IoU on RefCOCOg val, 600 referring expressions, excluding board images; shown as a percent. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; RefCOCOg val, 600 referring expressions, no board images; printed as a percent. Printed on the model card. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 600 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source
H2O Lightning cards · RefCOCOg box confidence calibration error	6.95 pp	Setup & source Confidence calibration error against IoU ≥ 0.5, published on a 0–1 scale; lower is better. Release v1.4.2 (same weights as v1.4.1); vLLM 0.30.0 build, NVIDIA L40S; confidence against IoU ≥ 0.5, RefCOCOg val. Published as 0.0695 on a 0–1 scale. H2O.ai's own runs, not a leaderboard result: its Decision Index rows use the public 0.3 kit and its own scorer, and the JevBench rows use JevBench's own CLI on the 231 public items. Hardware and serving profile differ by release, as each result states. Never compare with Leanmcp JevBench's board rows, Benchmark Heaven or community Decision Index 0.4 rows. Not ranked here. Checked / reported: 2026-11-02 · Samples: 600 Snapshot: 2026-11-02 · Revision: Cards read 2026-11-02; each result names its release and card revision · Published observations; result reuse terms not specified Result source · Benchmark source