Independent evaluation of Corvid-1 AI model scores 6.7 points below the company's published benchmark result
Updates expected. Corrections are logged, never silent.
The event
Corvid Labs published results on July 2 reporting that its Corvid-1 model scored 91.4 percent on the OpenReason-Hard benchmark; on July 9, researchers at the Halvorsen Institute reported scoring the same released model at 84.7 percent on the same public test set.
The record
Confirmed3
Independently corroborated by multiple sources.
Corvid Labs published results on July 2 reporting a 91.4 percent score for Corvid-1 on the OpenReason-Hard benchmark, which the company described as a state-of-the-art result.
4 sourcesNational WireJul 2, 2026The Civic PostJul 2, 2026The Morning StandardJul 3, 2026Broadcast One NewsJul 3, 2026On July 9, a group at the Halvorsen Institute published an evaluation reporting 84.7 percent for the same released model weights on the benchmark's public test set, using the benchmark's reference evaluation harness.
Corvid Labs published its evaluation configuration, including its prompting template and sampling settings, on July 10.
Attributed1
Stated by a party involved. Reported, not verified.
The benchmark's maintainers are reviewing whether the template complies with submission rules and will state a position within two weeks.
Said by: OpenReason-Hard maintainers
1 sourceNational WireJul 10, 2026
Disputed1
Credible sources contradict one another. Both positions shown.
Whether Corvid-1's published score reflects standard evaluation conditions for the benchmark.
Corvid Labs
The score gap is fully explained by the company's prompting template and sampling settings, which it considers legitimate and has now published.
Halvorsen Institute evaluation group
The template references example categories drawn from evaluation items, which amounts to tuning on the test set and is not comparable to other submissions.
What is not known
Unknown2
An open question. Listed, not papered over.
Whether the maintainers will re-score, annotate, or delist the Corvid-1 result.
Whether other high-ranking submissions used comparable templates; the leaderboard does not currently require configuration disclosure.
Timeline
Entries are appended, never edited. A corrected entry stays in the record, struck through, pointing to the entry that superseded it.
- Attributed
Corvid Labs announced a 91.4 percent score on OpenReason-Hard, describing it as state of the art.
- Confirmed
Corvid Labs released Corvid-1 model weights and an inference kit, enabling third-party evaluation.
- Confirmed
The Halvorsen Institute group published its 84.7 percent replication result using the reference harness and public test set.
- Disputed
Corvid Labs published its evaluation configuration and attributed the gap to its template and sampling settings; the Halvorsen group said the template draws on evaluation items. The benchmark maintainers opened a review.
Coverage lens4 outlets
Headlines reproduced verbatim from each outlet, side by side. Same event; the contrast is the point. The Ledger adds no commentary.
- National WireJul 9, 2026
Independent researchers report lower score for Corvid-1 than company's published result
- The Morning StandardJul 9, 2026
Replication effort fails to match Corvid's headline AI benchmark score
- The Civic PostJul 9, 2026
Did Corvid juice the leaderboard? A replication comes up short
- Broadcast One NewsJul 10, 2026
AI startup accused of gaming its own hype