Model & Harness Comparison
The same tests on the old and new model, or on competing harnesses, with repeated trials, latency and cost.
The problem
A model or harness switch can quietly change what your agent promises, refuses or costs, and vendor benchmarks do not use your tasks.
What we do
- One change at a time on a fixed test suite built from your tasks (model A vs. B, or harness A vs. B)
- Harness comparisons run off-the-shelf harnesses (for example Claude Code, Pi or an agent SDK) and your current setup on the same tasks; we have no stake in which one wins
- Repeated trials (at least 3) on critical tests
- Latency and, with your usage data, cost per successful task
What you receive
- A comparison table with intervals and paired statistical tests
- A go / no-go recommendation within tested scope
What you provide
- Staging or API access to both configurations
- Usage and price data for cost figures
Limits
- Differences apply to the tested suite; untested interactions are stated
- If TrendLab built the harness, results are an internal evaluation
Results describe behavior within the tested scope and configuration. They give you independent evidence for security reviews, audits and compliance work; certification itself is issued by accredited bodies.
Availability and independence
Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms. Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Pilot deliveries are scoped in writing, tell you it is an early delivery, and follow the same review and evidence rules as any engagement.
Independent evaluation: we had no hand in building or advising on your agent, and our price is fixed before testing starts, so the results don't change what we're paid.
Modules
Optional, quoted separately
- Harness comparison: off-the-shelf harnesses vs. your current setup on your tasksNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Cost per successful taskNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Verified task outcomesNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Staging vs. production parity checkNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
What is covered
The test areas this service always includes, with their status at the access level it uses.
- Harness vs. model comparison, including off-the-shelf harnesses compared with the client's setupIs our setup helping or hurting, and which harness should we standardize on?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Model upgrade and provider migrationIs the new model ready to switch to?Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Consistency across repeated trialsDoes it pass every time?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- LatencyIs it fast enough?Available now: Built into the TrendLab evaluation engine and used in sealed runs.