Review
Independent review of existing evaluation logs.
Independent evaluator assessment
Check the systems
that score AI.
We audit evaluators. We do not sell a judge.
Independent checks on the evaluators AI teams rely on.
AI teams score their models with automated evaluators — LLM judges, test harnesses, reward checks. The evaluators themselves are rarely checked. When one accepts wrong work, every number downstream inherits the error, and nobody notices, because the evaluator is the thing that would have noticed.
What we check
No answer key needed.
If two accepted programs contradict each other on a behaviour fixed by the task, at least one accepted behaviour is wrong. The argument does not assume the original program is correct. It depends on a clear requirement and an observed contradiction.
Profile
An exposure profile for each judge configuration.
Verify
Repair with verification.
Research / Preprint
Commit-first LLM judging inherits the judge's own errors
Asking a judge to solve a task first can help—but its own mistakes can still undermine the score.
Read the paper ↗Contact
Get in touch.
Running an LLM judge, test harness or reward verifier in production? Talk to us about an assessment.