Search Request a baseline quote

Test AI

AI testing services for LLM features, chatbots and agents

When a prompt, a model version or a retrieval index changes, you need to know what got worse before your users do. You get an evaluation dataset for your AI feature, a measured baseline of how it behaves today, and checks that run in your CI on every change.

Request a baseline quote How an engagement runs
eval run, support-assistant, prompt v14 vs v13 2 passed 1 failed 1 review
  • Refund request, order older than 90 days Answer matches the refund policy 0.94
  • Instructions pasted inside a ticket Prompt injection ignored 0.31
  • Book a call across time zones Right tool, right arguments 1.00
  • Question outside the knowledge base Says it does not know 0.58
  • Founders of CodeceptJS and Codeception
  • Founders of Testomat.io and Testomat.ai
  • Since 2015
  • 4.9 out of 5 on Clutch
What it is
AI testing and LLM evaluation
First step
Evaluation baseline About one week
You receive
Report on what fails, how often and how much it matters +2 more
After that
Pilot Two to six weeks

What we test

Services in this area

  • AI agent testing

    For agents and LLM workflows that call tools and take actions. Whole tasks are scored, step by step and by end state.

    • Tool calls checked for name, arguments and order
    • Side effects kept in a sandbox

  • AI chatbot testing

    For support and sales assistants. Whole conversations are tested against your policies, your tone and the moment a person must take over.

    • Multi-turn scenarios, not single prompts
    • Prompt injection and misuse attempts

  • RAG evaluation

    Retrieval quality measured apart from answer quality, so you know whether to fix the index or the prompt.

    • Retrieval checked on a labelled question set
    • Answer claims traced back to retrieved passages
    • Behaviour when the answer is not in your documents

  • Regression baselines in CI

    Every prompt, model or index change runs against the same dataset, and the pull request shows what moved.

    • Versioned evaluation dataset in your repository
    • Thresholds agreed with your product owner
    • A scheduled run to catch model updates on the provider's side
Evaluation loop for an AI featureFive steps in a loop. One, a dataset of the cases that matter, with criteria agreed with your product owner. Two, the dataset runs in CI on every prompt, model or index change. Three, each case is scored against its criterion, and people read the borderline results. Four, the scores are compared with the baseline: which cases got better and which got worse. Five, the gate: the pull request shows whether an agreed threshold was crossed. The dataset is kept current as features and policies change, and the loop starts again.Product ownerDataset ofcasesThe cases thatmatter, withcriteria agreedwith your productownerCIRunOn every prompt,model or indexchangeCIScoreEach case againstits criterion;people read theborderlineresultsCICompare withthe baselineWhich cases gotbetter, which gotworseYour thresholdsGate in CIThe pull requestshows whether anagreed thresholdwas crossedDataset kept current as features and policies changeEvaluation loop for an AI featureFive steps in a loop. One, a dataset of the cases that matter, with criteria agreed with your product owner. Two, the dataset runs in CI on every prompt, model or index change. Three, each case is scored against its criterion, and people read the borderline results. Four, the scores are compared with the baseline: which cases got better and which got worse. Five, the gate: the pull request shows whether an agreed threshold was crossed. The dataset is kept current as features and policies change, and the loop starts again.Product ownerDataset of casesThe cases that matter, with criteria agreedwith your product ownerCIRunOn every prompt, model or index changeCIScoreEach case against its criterion; peopleread the borderline resultsCICompare with the baselineWhich cases got better, which got worseYour thresholdsGate in CIThe pull request shows whether an agreedthreshold was crossedDataset kept current as features and policieschange
How an AI feature is evaluated on every change, from a dataset of the cases that matter to a pass or fail gate in your CI.

Why an AI feature needs a different kind of testing

A conventional test sends an input and asserts one expected output. That breaks down on an LLM feature for three reasons.

  • The output varies. The same input can produce different wording, and sometimes a different answer. An exact-match assertion either flakes or checks nothing useful.
  • Quality moves without a code change. A new model version from the provider, an edited system prompt, a re-indexed knowledge base or a changed tool definition can each make the feature better on one task and worse on another.
  • Failures rarely look like errors. There is no stack trace. There is a confident wrong answer, a tool called with the wrong argument, or a policy quoted that does not exist.

So the work is measurement: a dataset of the cases that matter to your business, a criterion for each, and a score you can compare from one change to the next.

What a dataset covers

Area Example checks
Correctness The answer agrees with your source documents or a reference answer
Tool use The right tool, with the right arguments, in the right order
Abuse and leakage Prompt injection through user content; the system prompt or another user’s data showing up in an answer
Edges Questions outside the feature’s scope, empty retrieval results, ambiguous requests
Cost and speed Latency and token use per task, so a quality gain is not paid for silently

Where the suite lives

The evaluation suite sits in your repository next to the feature. It runs as a CI job on every change to a prompt, a model setting or the retrieval index, and on a schedule to catch provider-side model updates. The pull request shows which cases got better, which got worse and whether an agreed threshold was crossed. Your engineers can run it locally, extend it and keep it after the engagement ends.

The detail differs by product. AI agent testing covers tool calls, multi-step tasks and side effects. AI chatbot testing covers conversations, policies and hand-off to a person.

How an engagement runs

Start small, with a fixed scope

Each step has a defined scope, duration and deliverables. You decide after each one whether to continue.

  1. Step 1 About one week

    Evaluation baseline

    One AI feature. We agree with your product owner what a good result is, build a first dataset of the cases that matter and measure the feature as it is today.

    You get

    • Report on what fails, how often and how much it matters
    • First evaluation dataset, versioned in your repository
    • A recommendation on which checks to automate first
  2. Step 2 Two to six weeks

    Pilot

    The baseline cases become an automated suite for that feature, run in your CI on every prompt, model or index change.

    You get

    • Evaluation suite and CI job in your repository
    • Agreed pass thresholds, written down
    • Handover session with your engineers
  3. Step 3 Monthly

    Ongoing

    Zapple.Tech engineers extend the suite to your next AI features and keep the dataset current as the product changes, either inside your team or as a dedicated team.

    You get

    • Dataset updated as features and policies change
    • Evaluation results read before each release
    • Monthly report on score trends per feature

FAQ

Questions buyers ask

How is this different from ordinary test automation?

An ordinary test asserts one exact output. LLM output varies between runs, so an evaluation scores many runs against criteria, tracks the scores over time and flags a regression when a change makes them worse. The deterministic parts around the model, such as API contracts and tool integrations, still get ordinary automated tests.

What do you need from us to start?

Access to the AI feature in a test environment, a product owner who can say what a good answer looks like, and, if you have them, examples of real conversations or tasks with personal data removed. For the one-week baseline, that is the full list.

Which page fits our product?

If your feature calls tools or changes data, start with AI agent testing. If customers talk to it, start with AI chatbot testing. If it answers from your documents, the baseline measures retrieval as well as the answers. Not sure? Describe the feature in the form and we say which baseline fits.

Does our data leave our environment?

The evaluation calls your feature in your test environment. If a judge model is used to score answers, which provider it runs on and what it may see is agreed with you before the baseline starts.

How long until we have numbers?

The baseline report arrives at the end of the first week. An automated suite in your CI follows in the pilot, which runs two to six weeks depending on how many task types the feature has.

Who does the work?

Zapple.Tech QA engineers. An evaluation suite is test code, so it is built, reviewed and kept in CI the way the company builds test automation. The founders of the company are also the founders of Testomat.io and Testomat.ai.

How much does it cost?

Prices are not published. The quote depends mainly on how many distinct tasks the feature handles and whether labelled examples already exist. Describe the feature and you get a scoped quote within one business day.

Start with a one-week evaluation baseline

Name the AI feature that worries you most and what it is meant to do. We reply within one business day with scope, timeline and a quote.

  • Founded by the founders of CodeceptJS and Codeception, open-source testing frameworks
  • Founded by the founders of Testomat.io and Testomat.ai
  • On the market since 2015
  • Rated 4.9 out of 5 on Clutch, from 7 client reviews

Prefer to talk first? Schedule a call

    We reply within one business day.