Monitoring (recurring re-evaluation)
Scheduled re-tests and re-tests after releases, reporting whether fixed failures stay fixed.
The problem
A bot that passed last month can fail today after a change nobody re-tested.
What we do
- Scheduled runs (monthly for Monitoring Lite, weekly for Monitoring) plus a re-test within 2 business days of a release you tell us about
- Every earlier finding re-tested and kept as a regression test
- Provider-reported model-version changes recorded for API targets
What you receive
- A report within 3 business days of each run
- Email alert within 1 business day of a run that shows a critical failure, confirmed by a person within 2 business days
- Monthly summary (Monitoring)
What you provide
- Release notices
- A named contact
Limits
- This is recurring re-evaluation, not live monitoring of production conversations
- Production sampling through your observability tool is a pilot module; native production monitoring is on the roadmap
Results describe behavior within the tested scope and configuration. They give you independent evidence for security reviews, audits and compliance work; certification itself is issued by accredited bodies.
Availability and independence
Pilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms. Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Pilot deliveries are scoped in writing, tell you it is an early delivery, and follow the same review and evidence rules as any engagement.
Independent evaluation: we had no hand in building or advising on your agent, and our price is fixed before testing starts, so the results don't change what we're paid.
Modules
Optional, quoted separately
- Production sampling through your observability toolNeeds a test environment or system accessPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
- Evidence validity states on a verify pageWorks through public chatPilot: Procedure designed and deliverable with existing tools, but not yet delivered to a client. Sold under pilot terms.
Future modules, not quoted yet
- Native production monitoringRoadmap: Not deliverable yet. Never quoted or included in a statement of work.
What is covered
The test areas this service always includes, with their status at the access level it uses.
- Unauthorized commitmentsCan my bot promise something we don't offer?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Invented discounts, coupons, price matchesCan it give away margin?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Contradicting published policyDoes it say the same thing as our Terms?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Wrong prices, plans or termsDoes it quote the right price?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Accepting forged "earlier agreements"Can a customer fake a promise?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Direct prompt injectionCan one message hijack it?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- System-prompt and configuration leakageCan people see our instructions?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Hand-off to a human : works, offered when needed, not overusedCan customers reach a person, and only when they need one?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Prompt-change regressionDid fixing one thing break another?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Fixed failures staying fixedDo bugs come back?Available now: Built into the TrendLab evaluation engine and used in sealed runs.
- Consistency across repeated trialsDoes it pass every time?Available now: Built into the TrendLab evaluation engine and used in sealed runs.