Eval Setup
We build your own evaluation system in your stack, with tools you own.
The problem
"It feels off" is not a metric, and every prompt fix might break something else.
What we do
- Failure discovery from real transcripts
- A golden dataset with development and test splits
- Checklist rubrics, single-question judges and their calibration against your labels
- Deterministic code checks
- A release gate in your CI pipeline that blocks critical failures
- Tracing hookup and a production-sampling plan (Standard)
What you receive
- The evaluation system in your repository
- A calibration report
- Runbook, training and 30 days of support (up to 8 hours)
- A demonstration that the gate blocks a seeded failing change and passes a clean one
What you provide
- Repository and CI access
- Transcripts under a data agreement
- Domain-expert time (about 8 hours for Starter, 16 for Standard)
Limits
- Starts only after our formal data policy is published and a data agreement is signed, because it uses your transcripts or outputs
- Not an independent evaluation of the agent
- A later TrendLab verdict on the same agent depends on the independence policy (being finalized) and always discloses this work
Results describe behavior within the tested scope and configuration. They give you independent evidence for security reviews, audits and compliance work; certification itself is issued by accredited bodies.
Availability and independence
Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms. Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Pilot deliveries are scoped in writing, tell you it is an early delivery, and follow the same review and evidence rules as any engagement.
Evaluation-design work. It is recorded, and any later TrendLab evaluation of the same agent discloses it.
Modules
Optional, quoted separately
- Rubric authoring by domain experts (spreadsheet)Needs a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Verified-outcome tasks for agents that actNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Future modules, not quoted yet
- TrendLab-hosted CI action, native production sampling, rubric editorRoadmap: Not deliverable yet. Never quoted or included in a statement of work.
What is covered
The test areas this service always includes, with their status at the access level it uses.
- Failure discovery from real transcriptsWhat's going wrong?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Golden dataset with a dev/test splitWhat's our ground truth?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Checklist rubrics, single-question judges, and their calibration against client labelsHow do we score quality, and can we trust the scores?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Code checksWhat can be checked for free?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- CI gate in the client repositoryCan bad changes be blocked automatically?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.