CIGE is a test case format for agentic AI test automation, proposed by two authors from Amazon.com Services LLC. Each test has four fields: Context (where it runs), Intent (why it exists and what evidence proves a pass), Guardrails (what the agent must not do) and Execution (a first path the agent may repair while the intent holds).
In agentic AI test automation, an AI agent reads a test’s goal and acts on the product. It recovers from small changes and collects evidence before it reports a result. That is how the CIGE paper describes the shift in its abstract.
The shift changes what a test case has to say. Steps are no longer enough. The test case has to say what it proves, what the agent may change and what it may never do.
This post walks through the paper field by field. It then shows a TypeScript gate that enforces those rules, with its real test output. The table below places CIGE next to three formats most teams already use.
| Format | What it does well | What CIGE adds | What you keep |
|---|---|---|---|
| Keyword-driven testing | Reuses actions and data through keywords | Intent, guardrails, evidence rules and staged context around keyword execution | The keyword library, as execution steps |
| BDD / Gherkin | Gives stakeholders a readable behavior description | Separates behavior intent from a repairable execution path | The scenario's Then clauses, as the source of success criteria |
| Page Object Model | Hides UI locator changes behind page abstractions | Can support vision-assisted execution that depends less on brittle locators, while preventing a repair from changing the test's purpose | Page objects, as tools the agent may call |
What is CIGE?
CIGE is a test case structure with four top-level fields: { context, intent, guardrails, execution }. Vivek
Krishna Choppa and Anusha Kovi of Amazon.com Services LLC propose it in
“CIGE: An Agentic AI Test Case Standard”.
The fields can live in JSON, YAML or a test management system. In the authors’ words, “the storage format is less important than the separation of concerns.”
The paper is short, at 7 pages. Its acknowledgments thank the reviewers of the QRS 2026 Industry Track, so it is an industry experience paper. It reports use in one codeless agentic testing platform. The title says “standard”, but CIGE is a proposal in a paper. It is not an ISO or IEEE standard.
The authors also set its scope. CIGE “is not a replacement for existing testing practices. It is a layer for a different execution style.” Its main idea is to keep the stable meaning of a test apart from the parts that change.
Why do scripts and prompts fall short in agentic AI test automation?
Scripts encode one path, and prompts mix everything together. Neither tells an agent where its freedom ends.
A script fails when a button moves. It also fails when the validation signal moves from a page to a service response. Neither is necessarily a product defect. The paper says many such failures are test drift or environment drift.
A plain prompt has the opposite problem. The paper notes that prompts mix setup, purpose, rules and steps. In practice, an agent reading one can improvise. It may report a pass by a shortcut, or repair a test into checking something else.
Giving the runner room to reason helps with drift. The authors name the price: it “shifts more responsibility to the test case to define intent, limits, and acceptable evidence.”
They frame the change as a move from step fidelity to goal fidelity. Step fidelity asks whether the automation followed the expected path. Goal fidelity asks whether the intended behavior was validated safely, with enough evidence, even if the path changed.
What goes in each CIGE field?
Each field answers one question: where, why, what not and how first. The paper’s own example (Table 1) is a workspace streaming flow. It is summarized under each field below.
Context: where the test runs and what the agent may load
Context is the operating ground for the test: system, environment, actor and tools. In the paper’s example, the system is a workspace streaming service in staging. The actor is a standard enterprise user. The tools are a browser, screenshot capture and a network-log reader.
Context also sets a reveal policy: “load login notes first, load diagnostics only after connection failure.”
The authors describe two mechanisms behind it. A tool-search tool finds tools on demand instead of loading every tool definition. A skills file acts as a light capability index, while detailed tool contracts load only at execution time. The paper calls the result progressive context disclosure.
It cites Anthropic’s post on context engineering for the underlying point. In the paper’s summary, agents need the right information at the right time, not one large block of mixed context. Anthropic’s own wording is to “find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome.”
Intent: why the test exists and what proves a pass
Intent-based testing judges a test by the behavior it must prove, not by the steps it took. In CIGE, intent states the capability under test, the success criteria, the failure criteria and the evidence required for a pass.
In the example, the intent is to verify that the user can start a streaming session. A pass needs a connected state and a first frame within the expected latency window. It also needs no authentication or policy error. The evidence is a screenshot plus connection logs.
Intent is the stable part. The authors say it “rarely evolves” and that it uniquely identifies each test. It also works as metadata. Their analogy is a large video file whose metadata sits in a database, so systems can read the metadata without loading the file. Agents can filter, group and write tests by intent in the same way.
Guardrails: what the agent must not do
Guardrails are runtime rules the agent must obey. The paper’s example lists five:
- Do not delete resources.
- Do not change production configuration.
- Do not expose tokens or credentials in logs.
- Do not mark pass from UI text alone.
- Do not update intent without human approval.
Guardrails do more than prevent damage. They also reduce false positives “by stopping shortcuts that make the final state look correct without really validating the feature.”
For runtime enforcement of agent rules, the paper builds on AgentSpec, which proposes customizable runtime enforcement for LLM agents.
Execution: the first path to try
Execution is the path the agent tries first, and it limits the agent’s search space. In the example: open the client, sign in with the test account and start streaming. Then wait for the connected state, capture the screenshot and connection log, and validate the latency and error fields.
The difference from a script fits in two sentences of the paper: “In traditional automation, the steps often become the test. In CIGE, the steps are guidance.” The agent may repair the steps when the product changes, as long as the repair keeps the intent.
How does agentic AI work in testing with a CIGE test case?
The agent treats the four fields differently at run time. Here is the paper’s runtime workflow (section 4 and Figure 1), split into seven steps:
- Reveal minimal context. The agent loads only what the reveal policy allows at the start.
- Run execution with guardrails. It follows the execution path, and the guardrails apply to every action.
- Classify any failure. A product defect becomes a bug report. Environment drift leads to a context repair. Test drift leads to an execution repair.
- Check a pass for a false positive. A false pass leads to a stricter evidence rule or guardrail.
- Replay candidate repairs in an isolated environment.
- Send every suggested test change to human review. A person reviews and commits it.
- Store the evidence and mark the pass when there is no failure and no false positive.
One detail differs between figure and text. Figure 1 labels the false-positive branch as test drift, with “repair guardrails”. The text says “A false positive may add a stricter evidence rule or guardrail.” Step 4 follows the text.
What this does not solve: classification is itself a judgement, and an agent can get it wrong. A real defect filed as drift gets repaired away. Every repair also costs review time, so the human gate is a real cost.
The same control applies to whole user journeys, which is the subject of end-to-end testing with AI agents.
Want agent-run journeys where a person approves each repair? The Zapple.Tech service page sets two rules: “Every red run triaged as defect, test fault or environment fault” and “An engineer approves or rejects every repair”. Start with a one-week end-to-end audit.
What does a CIGE test case look like in code?
A CIGE test case is data, so code can check it before and after an agent acts. Below is a test case for a billing portal. It is modeled on the paper’s Table 1, but the flow is this post’s own. A TypeScript gate then enforces three rules:
- Structure. Exactly four top-level fields. Intent needs a capability, success and failure criteria and evidence. Guardrails and steps cannot be empty.
- Pass gate. A reported pass counts only with every evidence kind the intent lists. With
no-pass-from-ui-text, UI text alone never counts. A guardrail violation in the action log blocks the run. - Repair gate. A change to intent or guardrails is rejected without an approval record. A change to context or execution waits for review and is never committed on its own.
The test case, invoice-download.cige.yaml:
# A CIGE test case: four top-level fields (Choppa and Kovi, sec. 3).
# The sub-keys and guardrail ids below are this example's own, not part of CIGE.
context:
system: customer billing portal
environment: staging
actor: standard customer account with three past invoices
tools: [browser, screenshot, network-log-reader, file-reader]
production_hosts: [billing.example, api.billing.example]
reveal_policy: >-
Load the login notes first. Load the PDF service diagnostics
only after a failed download.
intent:
capability: A customer can download last month's invoice as a PDF.
success:
- The download response is 200 with content type application/pdf.
- The invoice number in the file matches the number in the invoice list.
- No authentication or permission error appears.
failure:
- The download fails, times out or returns another customer's invoice.
evidence: [screenshot, network-log, downloaded-file]
guardrails:
- id: no-delete
rule: Do not delete invoices, accounts or any other resource.
- id: no-prod-hosts
rule: Do not send requests to production hosts.
- id: no-secrets-in-logs
rule: Do not write tokens or credentials to logs.
- id: no-pass-from-ui-text
rule: Do not mark a pass from UI text alone.
- id: intent-change-needs-approval
rule: Do not change intent or guardrails without human approval.
execution:
steps:
- Open the billing portal and sign in with the test account.
- Open Invoices and select last month's invoice.
- Click Download PDF.
- Capture a screenshot and the network log of the download.
- Read the file and compare its invoice number with the list.
The gate, checker.ts. It uses the yaml package to parse the file:
// A gate for CIGE test cases: structure, pass, actions and repairs.
// The four fields come from Choppa and Kovi, "CIGE: An Agentic AI Test Case Standard";
// the sub-keys and guardrail ids are this example's own.
import { readFileSync } from 'node:fs';
import { isDeepStrictEqual } from 'node:util';
import { parse } from 'yaml';
export interface Guardrail { id: string; rule: string }
export interface TestCase {
context: Record<string, unknown> & { production_hosts?: string[] };
intent: { capability: string; success: string[]; failure: string[]; evidence: string[] };
guardrails: Guardrail[];
execution: { steps: string[] };
}
export type Action = { kind: 'http'; method: string; url: string } | { kind: 'log'; line: string };
export interface Run { reported: 'passed' | 'failed'; evidence: { kind: string; ref: string }[]; actions: Action[] }
export interface Approval { reviewer: string; reason: string }
const FIELDS = ['context', 'intent', 'guardrails', 'execution'];
const TOKEN = /Bearer\s+[A-Za-z0-9._-]{8,}|AKIA[A-Z0-9]{16}/;
const list = (v: unknown) => Array.isArray(v) && v.length > 0 && v.every((x) => typeof x === 'string' && x.trim() !== '');
export function load(path: string): TestCase {
const tc = parse(readFileSync(path, 'utf8'));
const keys = Object.keys(tc ?? {}).sort();
if (keys.join() !== [...FIELDS].sort().join()) throw new Error(`top-level fields must be exactly ${FIELDS.join(', ')}; got ${keys.join(', ')}`);
if (typeof tc.context !== 'object' || tc.context === null) throw new Error('context must be a map');
const i = tc.intent ?? {};
if (typeof i.capability !== 'string' || !list(i.success) || !list(i.failure) || !list(i.evidence))
throw new Error('intent needs capability, success, failure and evidence');
if (!Array.isArray(tc.guardrails) || tc.guardrails.length === 0 || !tc.guardrails.every((g: Guardrail) => g?.id && g?.rule))
throw new Error('guardrails must be a non-empty list of {id, rule}');
if (!list(tc.execution?.steps)) throw new Error('execution.steps must be a non-empty list');
return tc as TestCase;
}
const has = (tc: TestCase, id: string) => tc.guardrails.some((g) => g.id === id);
/** The action gate: guardrail violations found in a run's action log. */
export function checkActions(tc: TestCase, actions: Action[]): string[] {
const out: string[] = [];
for (const a of actions) {
if (a.kind === 'http' && a.method.toUpperCase() === 'DELETE' && has(tc, 'no-delete')) out.push(`no-delete: ${a.method} ${a.url}`);
if (a.kind === 'http' && has(tc, 'no-prod-hosts') && (tc.context.production_hosts ?? []).includes(new URL(a.url).hostname))
out.push(`no-prod-hosts: ${a.url}`);
if (a.kind === 'log' && TOKEN.test(a.line) && has(tc, 'no-secrets-in-logs')) out.push('no-secrets-in-logs: token in a log line');
}
return out;
}
/** The pass gate: an agent's "passed" becomes pass only with the evidence the intent asks for. */
export function decidePass(tc: TestCase, run: Run) {
const violations = checkActions(tc, run.actions);
if (violations.length) return { result: 'blocked', reasons: violations };
if (run.reported === 'failed') return { result: 'failed', reasons: ['the run reported a failure'] };
const kinds = new Set(run.evidence.map((e) => e.kind));
const missing = tc.intent.evidence.filter((k) => !kinds.has(k));
if (missing.length) return { result: 'insufficient-evidence', reasons: missing.map((k) => `missing evidence: ${k}`) };
if (has(tc, 'no-pass-from-ui-text') && [...kinds].every((k) => k === 'ui-text'))
return { result: 'insufficient-evidence', reasons: ['no-pass-from-ui-text: only UI text attached'] };
return { result: 'pass', reasons: [] };
}
/** The repair gate: intent and guardrails change only with an approval; nothing commits unreviewed. */
export function reviewRepair(current: TestCase, proposed: TestCase, approval?: Approval) {
const reasons: string[] = [];
if (!isDeepStrictEqual(current.intent, proposed.intent)) reasons.push('intent changed');
for (const g of current.guardrails) {
const next = proposed.guardrails.find((p) => p.id === g.id);
if (!next) reasons.push(`guardrail removed: ${g.id}`);
else if (next.rule !== g.rule) reasons.push(`guardrail changed: ${g.id}`);
}
const approved = !!approval?.reviewer?.trim() && !!approval?.reason?.trim();
if (approved) return { result: 'approved-for-commit', reasons: reasons.length ? reasons : ['reviewed'] };
if (reasons.length) return { result: 'rejected', reasons: [...reasons, 'no approval record'] };
return { result: 'awaiting-review', reasons: ['context or execution changed; a person reviews before commit'] };
}
A short script, run.ts, feeds the gate one passing run, two refused runs and two repairs. This is its output:
run with all evidence: pass
run with a screenshot only: insufficient-evidence (missing evidence: network-log; missing evidence: downloaded-file)
run that sent a DELETE: blocked (no-delete: DELETE https://staging.billing.example/invoices/42)
repair that drops evidence: rejected (intent changed; no approval record)
repair of one execution step: awaiting-review (context or execution changed; a person reviews before commit)
The test file, checker.test.ts, covers 11 cases. They include a token in a log line, a request to a production
host and an intent change with and without approval. The run, compiled with tsc and run with node --test:
✔ the invoice test case loads with four fields (1.108041ms)
✔ a file without guardrails is refused (1.253167ms)
✔ pass with all three evidence kinds (0.19325ms)
✔ pass refused when the downloaded file is missing (0.081417ms)
✔ UI text alone is refused because of the guardrail, not missing evidence (0.823084ms)
✔ an HTTP DELETE blocks the run (0.075375ms)
✔ a request to a production host blocks the run (0.065125ms)
✔ a token in a log line blocks the run (0.109875ms)
✔ an execution-only repair waits for review (0.178542ms)
✔ deleting the no-delete guardrail is rejected without approval (0.136333ms)
✔ an intent change is rejected without approval and accepted with one (0.096458ms)
ℹ tests 11
ℹ suites 0
ℹ pass 11
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 78.0535
Tested with Node.js 26.13.1, TypeScript 6.0.3 and yaml 2.9.1 on 2026-11-18.
What this shows, and what it does not. It shows that code can check a CIGE test case: no pass without the evidence, and no silent change of purpose. It does not run an agent, classify failures or replay repairs. It reproduces no figure from the paper. The paper defines only the four top-level fields, so the sub-keys and guardrail ids here are this post’s illustration, not part of CIGE. Nor is it the authors’ implementation.
The gate also has a gap. The production host list sits in context, so a context repair can empty it and switch
no-prod-hosts off. The repair gate returns awaiting-review for that change, not rejected, so only the reviewer
catches it.
How does CIGE compare with BDD, keyword-driven testing and the Page Object Model?
CIGE wraps these formats rather than replacing them. The table near the top of this post sets the three side by side.
Keyword-driven testing separates test cases from reusable action keywords and test data. ISO/IEC/IEEE 29119-5:2024 defines it.
BDD is usually written in Given-When-Then form. For the basics, see this blog’s explainer on behavior-driven development.
The paper’s table also covers model-based testing, requirements-based generation and combinatorial testing.
What results do the authors report?
The authors report these changes after adopting CIGE in one codeless agentic testing platform. Its test set covered UI flows, service dependencies, environment setup and business-rule validations.
| Metric | Observed change | The authors’ explanation |
|---|---|---|
| Maintenance effort | About 90% lower | Many changes stayed in execution or context; the test purpose did not need rewriting |
| False positives | About 38% lower | Evidence rules and guardrails reduced invalid pass paths |
| Average execution time | About 62% lower | Staged context reduced repeated prompt loading and unnecessary exploration |
| Self-healed updates | Validated before approval | Candidate repairs were replayed in isolated environments before acceptance |
Read these figures with care. They come from an experience report on one platform. The paper names its data sources: execution logs, repair records and reviewer-approved updates. It states no baseline, period, test count or calculation method. The figures are not general results, and this post did not reproduce them.
What are CIGE’s limitations?
The paper names four next steps itself, in section 7:
- Migration layers from BDD scenarios, keyword-driven tests, Page Object automation and test management records.
- Multi-agent roles: planner, executor, validator and guardrail monitor.
- Evaluation under drift: selector changes, flaky environments, missing context, API contract changes, unavailable tools, prompt injection and changing business rules.
- A portable artifact that moves between workflows, test infrastructure, agent runtimes and validation tools.
Our reading adds four more:
- The paper publishes no schema below the four top-level fields. Two teams’ CIGE files will not match without an agreement.
- Guardrails are only as strong as the runtime that enforces them. A rule in a YAML file stops nothing by itself.
- Good intent and evidence rules are the hard part. AI does not write them for you; a person who knows the business rule does.
- The reported figures come from one platform. Agentic AI test automation on another stack may behave differently.
Repairs still need a reviewer. That is the work behind test automation maintenance with reviewed repairs.
One more distinction matters. CIGE is about testing a product with agents. Checking that an AI agent itself behaves, from its tool calls to task success, is AI agent testing. That is a different job.
How can a team start agentic AI test automation with CIGE?
Start with one journey and a log, not a migration.
- Pick one critical journey whose tests often break for reasons that are not defects.
- Write the intent first: capability, success criteria, failure criteria and required evidence.
- List the guardrails: destructive actions, production systems, secrets, “no pass from UI text alone” and “no intent change without approval”.
- Move the current steps into execution, and the environment, account and tools into context. If the steps are Playwright code, they stay Playwright code; Playwright testing services cover that layer.
- Put a gate in front of the runner, like the checker above. It blocks a pass without evidence and an unapproved intent change.
- Log every failure as product defect, environment drift, test drift or false positive. Log every repair with its reviewer.
- Decide from the log after a few weeks: how many failures were drift, how many repairs were accepted and how long review took.
CIGE gives agentic AI test automation a test case that states what it proves and where the agent must stop. Treat it as a structure to try on one journey, not a finished standard.
Zapple.Tech builds and tunes the AI layer for QA: skills, harnesses and agents, infrastructure tuning and monitoring. To talk about agent-run tests on your own product, tell us about your suite in the form below.
Sources
- Choppa and Kovi, CIGE: An Agentic AI Test Case Standard (Amazon Science, QRS 2026): definitions sec. 1-3, Table 1, workflow sec. 4, Table 2, Table 3, limitations sec. 7
- ISO/IEC/IEEE 29119-5:2024, Software testing, Part 5: Keyword-Driven Testing
- Wang, Poskitt and Sun, AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents (arXiv, 2025)
- Anthropic, Effective context engineering for AI agents (2025)