Test AI
AI agent testing and LLM evaluation: task success, tool calls, regressions
An agent can pick the right tool, pass the wrong argument and still write a confident summary. You get a task set with written success criteria, a measured baseline of how your agent performs today, and a suite that turns every model, prompt or tool change into scores you can compare.
- Rebook a flight after a cancellation Task completed; no booking created twice 0.92
- Hotel search with a budget limit Search tool called with the budget as an argument 0.40
- Cancel a trip, 5-step task Asks for confirmation before the irreversible step 1.00
- Tool returns an error on step 3 Retries once, then tells the user 0.55
- Founders of CodeceptJS and Codeception
- Founders of Testomat.io and Testomat.ai
- Since 2015
- 4.9 out of 5 on Clutch
At a glance
- What it is
- AI agent testing and LLM evaluation
- First step
- Agent evaluation baseline About one week
- You receive
- Task set with success criteria, in your repository +2 more
- After that
- Evaluation pilot Two to six weeks
What we test
Services in this area
-
Task success scoring
Does the agent finish the job the user asked for? Success is defined per task type with your product owner and checked on the end state, not on the wording of the reply.
- Final state checked directly, for example the record that should exist
- Several runs per case, because one run proves little
- Partial completion recorded as such, not rounded up to success
-
Tool calls and planning
The calls an agent makes matter as much as the text it returns. Each step is compared with what the task needs.
- Tool name, arguments and order
- Confirmation requested before actions that cannot be undone
- Behaviour when a tool fails, times out or returns nothing
-
Trajectory checks
A mistake in step two shapes steps three to eight, so the path is scored as well as the destination.
- Loops, repeated calls and abandoned tasks flagged
- Context carried correctly between steps
- Steps and tokens per task tracked, so a better score is not bought silently
-
Sandboxed tools and permissions
An agent under test must not refund a real customer. Tools with side effects are stubbed or pointed at test accounts, and permission boundaries are tested on purpose.
- Which actions may run for real, agreed before the first run
- Requests outside the agent's permissions tried deliberately
- Stubs that can return errors, empty results and slow responses
Why an agent needs more than a prompt test
A chatbot answers. An agent acts: it chooses a tool, fills in arguments, reads the result and decides what to do next. That adds failure modes a single-prompt test never sees.
- Right answer, wrong action. The reply reads well while the call behind it used the wrong customer id or skipped a required confirmation.
- Errors compound. A bad choice in step two changes everything after it. The final answer may look fine or may be wrong in a way that is hard to trace without the step log.
- Testing can cause damage. An agent that can send email, issue refunds or edit records will do so during a test unless the harness stops it.
So the unit of testing is the task: a goal, a starting state, the tools available, and a definition of done.
What one task case records
| Field | Example |
|---|---|
| Goal | Move the customer’s booking to the next available date |
| Starting state | Test account with one confirmed booking and one cancelled booking |
| Tools available | Booking search, booking update, email (stubbed) |
| Done when | One booking exists on the new date, the old one is released, one email is queued |
| Must not | Touch the cancelled booking; send more than one email; change the price without asking |
| Failure handling | If booking update returns an error, retry once and then tell the user |
Every run of the case is scored against those fields, and the step log shows where a failed run went wrong.
Where it runs
The task set, the stubs and the scoring code live in your repository next to the agent. A CI job runs them when the model, the prompt or a tool definition changes, and on a schedule. The tools themselves and their API contracts are deterministic and get ordinary automated tests. If your agent is customer-facing, AI chatbot testing covers the conversation side; for the wider area, see AI testing services.
How an engagement runs
Start small, with a fixed scope
Each step has a defined scope, duration and deliverables. You decide after each one whether to continue.
- Step 1 About one week
Agent evaluation baseline
One agent or one LLM workflow. We list its task types and tools, write success criteria with your product owner, set up a safe test harness and run a first task set several times.
You get
- Task set with success criteria, in your repository
- Failure log by step, showing where each failed task went wrong
- A plan for which tools to stub and which to run for real
- Step 2 Two to six weeks
Evaluation pilot
The task set becomes an automated suite that scores outcomes, tool calls and trajectories on every change to the model, the prompt or a tool definition.
You get
- Suite and CI job in your repository
- Side-by-side comparison of the current and a candidate model or prompt
- Handover session with the engineers who own the agent
- Step 3 Monthly
Ongoing evaluation
As the agent gains tools and users, Zapple.Tech engineers add task types, keep the stubs in step with the real tools and read the results before each release.
You get
- New task types added as tools are added
- Release notes stating which tasks moved
- Monthly report on task success by task type
FAQ
Questions buyers ask
What do you need from us to start the baseline?
Access to the agent in a test environment, the list of tools it can call, test accounts for those tools, and a person who can say what a successful task looks like. Examples of real tasks with personal data removed help, but a first set can be written with you if you have none.
Will the tests trigger real actions in our systems?
Only the ones you agree to. Tools with side effects, such as payments, emails or bookings, are stubbed or run against test accounts. The list of actions allowed to run for real is written down before the first run.
How do you test an agent whose behaviour changes on every run?
Each case runs several times. Where the task has a checkable end state, such as a record created or a message sent once, that state is checked directly. Free-text parts are scored against written criteria, and the scoring is checked against cases a person has labelled.
We already run evals in-house. What would you add?
The suite can be built in the evaluation tooling you already use. What this adds is a task set with written success criteria, step-level tool checks, side-effect isolation, and a reviewer who did not build the agent.
How long does it take?
The baseline takes about a week for one agent. The pilot runs two to six weeks; agents with many tools, or tools that are hard to stub, take the longer end.
Who does the work?
Zapple.Tech test automation engineers. Most of an agent harness is ordinary test engineering, such as stubs, fixtures, test accounts and CI jobs, with the scoring built on top.
What happens when the provider updates the model?
The suite runs on a schedule as well as on your own changes, so a provider-side update shows up as tasks that moved. You see which ones before you decide whether to accept the new version.
How much does it cost?
Prices are not published. The quote depends mostly on the number of task types and tools, and on how much of the tool layer has to be stubbed. Send the list and you get a scoped quote within one business day.
Start with a one-week agent evaluation baseline
List the tasks the agent should complete and the tools it can call. We reply within one business day with scope, timeline and a quote.
- Founded by the founders of CodeceptJS and Codeception, open-source testing frameworks
- Founded by the founders of Testomat.io and Testomat.ai
- On the market since 2015
- Rated 4.9 out of 5 on Clutch, from 7 client reviews
Prefer to talk first? Schedule a call