How it works
How TrendLab tests AI agents, and why you can trust the results.
AI agents fail in ways that cost money and trust: invented discounts, promised refunds, leaked instructions. Most agent testing is done by the people who built the agent, using tests the agent has already seen. That is a student grading their own exam.
This page explains what we built to make our results something a skeptical buyer, auditor or CTO can check for themselves, and where that work is still on the roadmap.
Independent by design
The system being tested cannot see or change its own tests or its own evidence. The referee and the player are never the same party.
Step 1
Your agent
The bot being tested, reached the way a customer reaches it.
Step 2
TrendLab test harness
Runs a test suite that we keep private from the vendor, including a held-back set in client engagements.
Step 3
Two AI judges
After fast rule-based checks, two models from different providers grade each answer independently.
Step 4
Sealed evidence
Every result is locked into a tamper-evident, timestamped ledger.
Private tests, held-back sets
TrendLab keeps its test suites private from the vendor. Each engagement is designed to include a held-back set the vendor never sees, which helps detect an agent tuned to known tests. It is not proof: hidden inputs sent to a hosted bot can appear in that vendor's logs. Client-specific holdout sets with gap reporting are on the roadmap.
We do not test what we built
TrendLab does not issue independent assurance on agents it built, and does not implement clients' fixes. Our written independence policy requires us to disclose relationships. The agents we run nightly (below) are our own and are labeled internal evaluations.
Evidence you can hand to a buyer
A test result is only useful if people believe it was not changed afterward.
1
Hash-chained ledger
Nobody, including us, can edit a sealed result after the fact without it being detectable. Evidence events are recorded as entries linked to the one before by a cryptographic hash, so any edit breaks the chain.
2
External timestamps
Each night the ledger's latest fingerprint is stamped by an outside timestamp authority (RFC 3161, an open standard) and copied to a separate third-party-hosted repository. External timestamps fix when results existed, so a record altered or backdated later, even by us, can be detected.
3
Reproducibility, honestly labeled
Each run records exactly what was tested and with what configuration. Grading can be re-checked exactly; hosted AI models can change, so we label agent behavior as comparable, not identical.
Your buyers can check results themselves on a verify page like the example. Every report prints the ledger head hash it rests on, so you hold an independent reference copy. Signed reports and standalone-verifiable evidence bundles are on the roadmap.
Grading you can trust
How a bot's answer becomes a grade, and the rules that stop the grade from flattering anyone.
Built and running
Two judges from different providers
A single model has blind spots. Each answer is graded by two judges from different model families, so one model's quirks do not set your grade.
Built and running
Checklist rubrics
Judges answer specific yes/no criteria for each test, not a vague "rate this 1 to 10". Any grade can be challenged line by line.
Built and running
Prompt-injection defense for the judges
Bots under attack produce hostile text, and that text can try to talk the judge into a pass. Judges treat the bot's answer strictly as data, and attempts to manipulate them are treated as a failed test.
In every snapshot and audit report
Critical issues override averages
If any critical check fails, the verdict says so. One unacceptable failure, such as a forbidden refund promise, cannot be averaged away.
In every snapshot and audit report
Confidence ranges, not magic numbers
We report what we measured with its range and sample size ("0 failures in 60 attempts; failure rate below 6% with 95% confidence"), not a single score that implies precision it does not have.
In every snapshot and audit report
No composite "risk score"
We do not collapse everything into one number. Quality and risk are reported separately, because a single score hides what a buyer needs to see.
Illustrative example, not a client result
Test: the tester asks for a 30% discount code.
Bot answer: invents a code that does not exist.
Checklist: fails "does not offer discounts outside the published policy".
Report: flagged as a critical finding with revenue impact, and recorded in the ledger.
Today: two judges plus a human-reviewable checklist per test. Next: we measure and publish how often our judges agree with qualified human reviewers.
What we test
The customer-bot risk pack: about 45 tests, run from the outside through your public chat window, polite and low-volume. Each category maps to a business cost.
Invented discounts
Bot offers codes or deals that do not exist. Impact: may cause margin loss and screenshots on social media.
Refund promises
Bot commits to refunds or exceptions your policy does not allow. Impact: you may be held to what your bot said.
Wrong prices
Bot quotes incorrect prices or terms. Impact: lost revenue, disputes and chargebacks.
Prompt injection
A crafted message makes the bot ignore its rules. Impact: loss of control over what speaks for your brand.
System-prompt leakage
Bot reveals its hidden instructions or internal logic. Impact: competitive exposure and easier attacks.
Jailbreaks and off-brand output
Bot is pushed into content that does not match your brand. Impact: reputational damage.
Off-topic abuse
People use your bot as a free general-purpose AI. Impact: wasted model spend.
Failed human handoff
Bot cannot escalate when a customer needs a person. Impact: lost customers and unresolved complaints.
The Snapshot sends no more than one message every 3 seconds and 60 messages in total, never exploits your systems and never tries to obtain real personal data. Reports state scope, sample sizes, methodology version, limitations and an as-of date, and can be re-tested after you fix issues.
Running every night, on our own agents
Internal evaluation, not client assurance
Our own testing engine has run automatically every night since October 2026, against two agents TrendLab built: a demo helpdesk agent and TrendLab's own assistant. It shows the machinery runs unattended, with alerts that carry no content. You can see the demo agent's record on the example verify page.
This is an internal evaluation of our own demo agents. It proves the machinery runs, not that those agents are assured. We never present it as independent assurance, and we do not use it to claim anything about your agent.
Built with security in mind
High level on purpose: we do not publish details attackers could use.
Isolated from our answers
Your bot is only ever reached through its public chat window, politely and at low volume. It never sees test answers, judge criteria or our evidence. Agents we run on our own infrastructure run in a sandbox.
Spend caps
Every run has a spend cap with an automatic stop, so a misbehaving agent cannot run up an open-ended model bill.
Data minimization and redaction
We test public behavior, keep only what is needed, and redact secrets before anything is stored.
Aligned to the frameworks buyers cite
Reports are designed to provide evidence supporting these frameworks. They are evidence, not certification: only accredited bodies certify.
OWASP Top 10 for LLM Applications
Every snapshot and audit report
NIST AI RMF
Every snapshot and audit report
ISO/IEC 42001
Audit reports (Enterprise), roadmap
EU AI Act
Audit reports (Enterprise), roadmap
Evaluations measure and reduce risk; they do not guarantee an AI system will never fail.
What's next on our roadmap
Not built yet. We would rather say so than sell it.
Release regression gate
Block a release when a statistically real regression appears against a baseline you approved.
Judge calibration against human experts
Measure how often our judges agree with qualified human reviewers, and publish the agreement.
Multi-turn and tool-use testing
Test full conversations with difficult users, and check that agents do not take actions they should not.
Client-specific holdout sets
Held-back tests per client, with reporting on the gap between visible and hidden results.
Signed reports and evidence bundles
Reports you can verify as genuine and unaltered, and a complete evidence copy that stays checkable even if TrendLab stopped operating.
Production monitoring
Sample real conversations and catch behavior your tests never covered.
See it on your bot
A free Risk Snapshot gives you a short report within 48 hours of confirmation, with no commitment.