Judge Calibration Audit
Measure how often your LLM judges agree with your experts, and where they miss real failures.
The problem
Your eval scores come from AI judges nobody has checked against people.
What we do
- Blind expert labels on 100 to 200 of your outputs
- Agreement, coverage and missed-failure rates for each criterion
- Checks for position, length and self-preference bias
What you receive
- A calibration report with recommendations
What you provide
- Sample outputs under a data agreement
- Expert time for labeling
Limits
- Starts only after our formal data policy is published and a data agreement is signed, because it uses your transcripts or outputs
- Results are only as good as the reference labels; labeler provenance is reported
- This is design work on your evaluation system, which affects later independent evaluation (see independence policy)
Results describe behavior within the tested scope and configuration. They give you independent evidence for security reviews, audits and compliance work; certification itself is issued by accredited bodies.
Availability and independence
Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms. Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Pilot deliveries are scoped in writing, tell you it is an early delivery, and follow the same review and evidence rules as any engagement.
Evaluation-design work. It is recorded, and any later TrendLab evaluation of the same agent discloses it.
What is covered
The test areas this service always includes, with their status at the access level it uses.
- Checklist rubrics, single-question judges, and their calibration against client labelsHow do we score quality, and can we trust the scores?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.