Benchmark protocol
Scope and eligibility
A release defines which domains, datasets, model implementations and configurations are eligible.
Evaluation modes
Replay scores a model against recorded exposures and their outcomes. Compliance checks a schedule against a stated rule set. Counterfactual analysis models an outcome that was not observed. Each mode is named on the result, and counterfactual results are marked as modelled.
Overlap between development and evaluation data
OpenDeco records where an evaluation dataset was also used to develop a model. Results measured on that overlapping data are labelled as such.
Metrics
Metrics are chosen for the scientific question and versioned with the protocol. Each is reported separately.
Failure regions
Aggregate performance is accompanied by inspection of exposure regions where errors or model divergence concentrate.
Sensitivity
Configuration sweeps and analytical choices are tested where they can materially change the conclusion. A finding that disappears under reasonable settings is reported as fragile.
Validation across separate studies
When data come from several independent studies, OpenDeco can hold out a whole study for testing. Records from one programme often share equipment and study population, so splitting a single programme's records at random can overstate how well a model carries to new data.
Release
Public releases are built from frozen inputs and immutable run manifests. Each is checked for provenance and for restricted-data leakage, re-run from a clean environment to confirm it reproduces, then given stable version metadata.