# CIGE: a test case standard for agentic AI test automation, explained with code

> Agentic AI Test Automation: How CIGE Test Cases Work. Canonical page: <https://zapple.tech/blog/test-automation-methodologies/cige-agentic-test-case-standard/>

CIGE is a test case format for agentic AI test automation, proposed by two authors from Amazon.com Services LLC. Each test has four fields: Context (where it runs), Intent (why it exists and what evidence proves a pass), Guardrails (what the agent must not do) and Execution (a first path the agent may repair while the intent holds).

In **agentic AI test automation**, an AI agent reads a test's goal and acts on the product. It recovers from small
changes and collects evidence before it reports a result. That is how the CIGE paper describes the shift in its
abstract.

The shift changes what a test case has to say. Steps are no longer enough. The test case has to say what it proves,
what the agent may change and what it may never do.

This post walks through the paper field by field. It then shows a TypeScript gate that enforces those rules, with its
real test output. The table below places CIGE next to three formats most teams already use.

## What is CIGE?

**CIGE** is a test case structure with four top-level fields: `{ context, intent, guardrails, execution }`. Vivek
Krishna Choppa and Anusha Kovi of Amazon.com Services LLC propose it in
"[CIGE: An Agentic AI Test Case Standard](https://www.amazon.science/publications/cige-an-agentic-ai-test-case-standard)".

The fields can live in JSON, YAML or a test management system. In the authors' words, "the storage format is less
important than the separation of concerns."

The paper is short, at 7 pages. Its acknowledgments thank the reviewers of the QRS 2026 Industry Track, so it is
an industry experience paper. It reports use in one codeless agentic testing platform. The title says "standard", but
CIGE is a proposal in a paper. It is not an ISO or IEEE standard.

The authors also set its scope. CIGE "is not a replacement for existing testing practices. It is a layer for a
different execution style." Its main idea is to keep the stable meaning of a test apart from the parts that change.

## Why do scripts and prompts fall short in agentic AI test automation?

Scripts encode one path, and prompts mix everything together. Neither tells an agent where its freedom ends.

A script fails when a button moves. It also fails when the validation signal moves from a page to a service response.
Neither is necessarily a product defect. The paper says many such failures are **test drift or environment drift**.

A plain prompt has the opposite problem. The paper notes that prompts mix setup, purpose, rules and steps. In
practice, an agent reading one can improvise. It may report a pass by a shortcut, or repair a test into checking something else.

Giving the runner room to reason helps with drift. The authors name the price: it "shifts more responsibility to the
test case to define intent, limits, and acceptable evidence."

They frame the change as a move from **step fidelity** to **goal fidelity**. Step fidelity asks whether the automation
followed the expected path. Goal fidelity asks whether the intended behavior was validated safely, with enough
evidence, even if the path changed.

## What goes in each CIGE field?

Each field answers one question: where, why, what not and how first. The paper's own example (Table 1) is a workspace
streaming flow. It is summarized under each field below.

### Context: where the test runs and what the agent may load

**Context** is the operating ground for the test: system, environment, actor and tools. In the paper's example, the
system is a workspace streaming service in staging. The actor is a standard enterprise user. The tools are a browser,
screenshot capture and a network-log reader.

Context also sets a reveal policy: "load login notes first, load diagnostics only after connection failure."

The authors describe two mechanisms behind it. A tool-search tool finds tools on demand instead of loading every tool
definition. A skills file acts as a light capability index, while detailed tool contracts load only at execution time.
The paper calls the result **progressive context disclosure**.

It cites [Anthropic's post on context engineering](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
for the underlying point. In the paper's summary, agents need the right information at the right time, not one large
block of mixed context. Anthropic's own wording is to "find the smallest set of high-signal tokens that maximize the
likelihood of your desired outcome."

### Intent: why the test exists and what proves a pass

**Intent-based testing** judges a test by the behavior it must prove, not by the steps it took. In CIGE, intent states
the capability under test, the success criteria, the failure criteria and the evidence required for a pass.

In the example, the intent is to verify that the user can start a streaming session. A pass needs a connected state and
a first frame within the expected latency window. It also needs no authentication or policy error. The evidence is a
screenshot plus connection logs.

Intent is the stable part. The authors say it "rarely evolves" and that it uniquely identifies each test. It also works
as metadata. Their analogy is a large video file whose metadata sits in a database, so systems can read the metadata
without loading the file. Agents can filter, group and write tests by intent in the same way.

### Guardrails: what the agent must not do

**Guardrails** are runtime rules the agent must obey. The paper's example lists five:

- Do not delete resources.
- Do not change production configuration.
- Do not expose tokens or credentials in logs.
- Do not mark pass from UI text alone.
- Do not update intent without human approval.

Guardrails do more than prevent damage. They also reduce false positives "by stopping shortcuts that make the final
state look correct without really validating the feature."

For runtime enforcement of agent rules, the paper builds on
[AgentSpec](https://arxiv.org/abs/2503.18666), which proposes customizable runtime enforcement for LLM agents.

### Execution: the first path to try

**Execution** is the path the agent tries first, and it limits the agent's search space. In the example: open the
client, sign in with the test account and start streaming. Then wait for the connected state, capture the screenshot
and connection log, and validate the latency and error fields.

The difference from a script fits in two sentences of the paper: "In traditional automation, the steps often become the
test. In CIGE, the steps are guidance." The agent may repair the steps when the product changes, as long as the repair
keeps the intent.

## How does agentic AI work in testing with a CIGE test case?

The agent treats the four fields differently at run time. Here is the paper's runtime workflow (section 4 and Figure 1),
split into seven steps:

1. **Reveal minimal context.** The agent loads only what the reveal policy allows at the start.
2. **Run execution with guardrails.** It follows the execution path, and the guardrails apply to every action.
3. **Classify any failure.** A product defect becomes a bug report. Environment drift leads to a context repair. Test
   drift leads to an execution repair.
4. **Check a pass for a false positive.** A false pass leads to a stricter evidence rule or guardrail.
5. **Replay candidate repairs** in an isolated environment.
6. **Send every suggested test change to human review.** A person reviews and commits it.
7. **Store the evidence and mark the pass** when there is no failure and no false positive.

One detail differs between figure and text. Figure 1 labels the false-positive branch as test drift, with "repair
guardrails".
The text says "A false positive may add a stricter evidence rule or guardrail." Step 4 follows the text.

What this does not solve: classification is itself a judgement, and an agent can get it wrong. A real defect filed as
drift gets repaired away. Every repair also costs review time, so the human gate is a real cost.

The same control applies to whole user journeys, which is the subject of
[end-to-end testing with AI agents](/services/agentic-e2e-testing/).

Want agent-run journeys where a person approves each repair? The Zapple.Tech service page sets two rules: "Every red
run triaged as defect, test fault or environment fault" and "An engineer approves or rejects every repair".
[Start with a one-week end-to-end audit](/services/agentic-e2e-testing/).

## What does a CIGE test case look like in code?

A CIGE test case is data, so code can check it before and after an agent acts. Below is a test case for a billing
portal. It is modeled on the paper's Table 1, but the flow is this post's own. A TypeScript gate then enforces three
rules:

1. **Structure.** Exactly four top-level fields. Intent needs a capability, success and failure criteria and evidence.
   Guardrails and steps cannot be empty.
2. **Pass gate.** A reported pass counts only with every evidence kind the intent lists. With `no-pass-from-ui-text`,
   UI text alone never counts. A guardrail violation in the action log blocks the run.
3. **Repair gate.** A change to intent or guardrails is rejected without an approval record. A change to context or
   execution waits for review and is never committed on its own.

The test case, `invoice-download.cige.yaml`:

```yaml
# A CIGE test case: four top-level fields (Choppa and Kovi, sec. 3).
# The sub-keys and guardrail ids below are this example's own, not part of CIGE.
context:
  system: customer billing portal
  environment: staging
  actor: standard customer account with three past invoices
  tools: [browser, screenshot, network-log-reader, file-reader]
  production_hosts: [billing.example, api.billing.example]
  reveal_policy: >-
    Load the login notes first. Load the PDF service diagnostics
    only after a failed download.

intent:
  capability: A customer can download last month's invoice as a PDF.
  success:
    - The download response is 200 with content type application/pdf.
    - The invoice number in the file matches the number in the invoice list.
    - No authentication or permission error appears.
  failure:
    - The download fails, times out or returns another customer's invoice.
  evidence: [screenshot, network-log, downloaded-file]

guardrails:
  - id: no-delete
    rule: Do not delete invoices, accounts or any other resource.
  - id: no-prod-hosts
    rule: Do not send requests to production hosts.
  - id: no-secrets-in-logs
    rule: Do not write tokens or credentials to logs.
  - id: no-pass-from-ui-text
    rule: Do not mark a pass from UI text alone.
  - id: intent-change-needs-approval
    rule: Do not change intent or guardrails without human approval.

execution:
  steps:
    - Open the billing portal and sign in with the test account.
    - Open Invoices and select last month's invoice.
    - Click Download PDF.
    - Capture a screenshot and the network log of the download.
    - Read the file and compare its invoice number with the list.
```

The gate, `checker.ts`. It uses the `yaml` package to parse the file:

```ts
// A gate for CIGE test cases: structure, pass, actions and repairs.
// The four fields come from Choppa and Kovi, "CIGE: An Agentic AI Test Case Standard";
// the sub-keys and guardrail ids are this example's own.
import { readFileSync } from 'node:fs';
import { isDeepStrictEqual } from 'node:util';
import { parse } from 'yaml';

export interface Guardrail { id: string; rule: string }
export interface TestCase {
  context: Record<string, unknown> & { production_hosts?: string[] };
  intent: { capability: string; success: string[]; failure: string[]; evidence: string[] };
  guardrails: Guardrail[];
  execution: { steps: string[] };
}
export type Action = { kind: 'http'; method: string; url: string } | { kind: 'log'; line: string };
export interface Run { reported: 'passed' | 'failed'; evidence: { kind: string; ref: string }[]; actions: Action[] }
export interface Approval { reviewer: string; reason: string }

const FIELDS = ['context', 'intent', 'guardrails', 'execution'];
const TOKEN = /Bearer\s+[A-Za-z0-9._-]{8,}|AKIA[A-Z0-9]{16}/;
const list = (v: unknown) => Array.isArray(v) && v.length > 0 && v.every((x) => typeof x === 'string' && x.trim() !== '');

export function load(path: string): TestCase {
  const tc = parse(readFileSync(path, 'utf8'));
  const keys = Object.keys(tc ?? {}).sort();
  if (keys.join() !== [...FIELDS].sort().join()) throw new Error(`top-level fields must be exactly ${FIELDS.join(', ')}; got ${keys.join(', ')}`);
  if (typeof tc.context !== 'object' || tc.context === null) throw new Error('context must be a map');
  const i = tc.intent ?? {};
  if (typeof i.capability !== 'string' || !list(i.success) || !list(i.failure) || !list(i.evidence))
    throw new Error('intent needs capability, success, failure and evidence');
  if (!Array.isArray(tc.guardrails) || tc.guardrails.length === 0 || !tc.guardrails.every((g: Guardrail) => g?.id && g?.rule))
    throw new Error('guardrails must be a non-empty list of {id, rule}');
  if (!list(tc.execution?.steps)) throw new Error('execution.steps must be a non-empty list');
  return tc as TestCase;
}

const has = (tc: TestCase, id: string) => tc.guardrails.some((g) => g.id === id);

/** The action gate: guardrail violations found in a run's action log. */
export function checkActions(tc: TestCase, actions: Action[]): string[] {
  const out: string[] = [];
  for (const a of actions) {
    if (a.kind === 'http' && a.method.toUpperCase() === 'DELETE' && has(tc, 'no-delete')) out.push(`no-delete: ${a.method} ${a.url}`);
    if (a.kind === 'http' && has(tc, 'no-prod-hosts') && (tc.context.production_hosts ?? []).includes(new URL(a.url).hostname))
      out.push(`no-prod-hosts: ${a.url}`);
    if (a.kind === 'log' && TOKEN.test(a.line) && has(tc, 'no-secrets-in-logs')) out.push('no-secrets-in-logs: token in a log line');
  }
  return out;
}

/** The pass gate: an agent's "passed" becomes pass only with the evidence the intent asks for. */
export function decidePass(tc: TestCase, run: Run) {
  const violations = checkActions(tc, run.actions);
  if (violations.length) return { result: 'blocked', reasons: violations };
  if (run.reported === 'failed') return { result: 'failed', reasons: ['the run reported a failure'] };
  const kinds = new Set(run.evidence.map((e) => e.kind));
  const missing = tc.intent.evidence.filter((k) => !kinds.has(k));
  if (missing.length) return { result: 'insufficient-evidence', reasons: missing.map((k) => `missing evidence: ${k}`) };
  if (has(tc, 'no-pass-from-ui-text') && [...kinds].every((k) => k === 'ui-text'))
    return { result: 'insufficient-evidence', reasons: ['no-pass-from-ui-text: only UI text attached'] };
  return { result: 'pass', reasons: [] };
}

/** The repair gate: intent and guardrails change only with an approval; nothing commits unreviewed. */
export function reviewRepair(current: TestCase, proposed: TestCase, approval?: Approval) {
  const reasons: string[] = [];
  if (!isDeepStrictEqual(current.intent, proposed.intent)) reasons.push('intent changed');
  for (const g of current.guardrails) {
    const next = proposed.guardrails.find((p) => p.id === g.id);
    if (!next) reasons.push(`guardrail removed: ${g.id}`);
    else if (next.rule !== g.rule) reasons.push(`guardrail changed: ${g.id}`);
  }
  const approved = !!approval?.reviewer?.trim() && !!approval?.reason?.trim();
  if (approved) return { result: 'approved-for-commit', reasons: reasons.length ? reasons : ['reviewed'] };
  if (reasons.length) return { result: 'rejected', reasons: [...reasons, 'no approval record'] };
  return { result: 'awaiting-review', reasons: ['context or execution changed; a person reviews before commit'] };
}
```

A short script, `run.ts`, feeds the gate one passing run, two refused runs and two repairs. This is its output:

```text
run with all evidence:             pass
run with a screenshot only:        insufficient-evidence (missing evidence: network-log; missing evidence: downloaded-file)
run that sent a DELETE:            blocked (no-delete: DELETE https://staging.billing.example/invoices/42)
repair that drops evidence:        rejected (intent changed; no approval record)
repair of one execution step:      awaiting-review (context or execution changed; a person reviews before commit)
```

The test file, `checker.test.ts`, covers 11 cases. They include a token in a log line, a request to a production
host and an intent change with and without approval. The run, compiled with `tsc` and run with `node --test`:

```text
✔ the invoice test case loads with four fields (1.108041ms)
✔ a file without guardrails is refused (1.253167ms)
✔ pass with all three evidence kinds (0.19325ms)
✔ pass refused when the downloaded file is missing (0.081417ms)
✔ UI text alone is refused because of the guardrail, not missing evidence (0.823084ms)
✔ an HTTP DELETE blocks the run (0.075375ms)
✔ a request to a production host blocks the run (0.065125ms)
✔ a token in a log line blocks the run (0.109875ms)
✔ an execution-only repair waits for review (0.178542ms)
✔ deleting the no-delete guardrail is rejected without approval (0.136333ms)
✔ an intent change is rejected without approval and accepted with one (0.096458ms)
ℹ tests 11
ℹ suites 0
ℹ pass 11
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 78.0535
```

Tested with Node.js 26.13.1, TypeScript 6.0.3 and yaml 2.9.1 on 2026-11-18.

**What this shows, and what it does not.** It shows that code can check a CIGE test case: no pass without the evidence,
and no silent change of purpose. It does not run an agent, classify failures or replay repairs. It reproduces no figure
from the paper. The paper defines only the four top-level fields, so the sub-keys and guardrail ids here are this
post's illustration, not part of CIGE. Nor is it the authors' implementation.

The gate also has a gap. The production host list sits in `context`, so a context repair can empty it and switch
`no-prod-hosts` off. The repair gate returns `awaiting-review` for that change, not `rejected`, so only the reviewer
catches it.

## How does CIGE compare with BDD, keyword-driven testing and the Page Object Model?

CIGE wraps these formats rather than replacing them. The table near the top of this post sets the three side by side.

**Keyword-driven testing** separates test cases from reusable action keywords and test data.
[ISO/IEC/IEEE 29119-5:2024](https://doi.org/10.1109/IEEESTD.2024.10806546) defines it.

**BDD** is usually written in Given-When-Then form. For the basics, see this blog's explainer on
[behavior-driven development](/blog/types-of-automation-testing/what-is-behaviour-driven-development/).

The paper's table also covers model-based testing, requirements-based generation and combinatorial testing.

## What results do the authors report?

The authors report these changes after adopting CIGE in one codeless agentic testing platform. Its test set covered UI
flows, service dependencies, environment setup and business-rule validations.

| Metric | Observed change | The authors' explanation |
|:--|:--|:--|
| Maintenance effort | About 90% lower | Many changes stayed in execution or context; the test purpose did not need rewriting |
| False positives | About 38% lower | Evidence rules and guardrails reduced invalid pass paths |
| Average execution time | About 62% lower | Staged context reduced repeated prompt loading and unnecessary exploration |
| Self-healed updates | Validated before approval | Candidate repairs were replayed in isolated environments before acceptance |

Read these figures with care. They come from an experience report on one platform. The paper names its data
sources: execution logs, repair records and reviewer-approved updates. It states no baseline, period, test count or
calculation method. The figures are not general results, and this post did not reproduce them.

## What are CIGE's limitations?

The paper names four next steps itself, in section 7:

1. **Migration layers** from BDD scenarios, keyword-driven tests, Page Object automation and test management records.
2. **Multi-agent roles**: planner, executor, validator and guardrail monitor.
3. **Evaluation under drift**: selector changes, flaky environments, missing context, API contract changes, unavailable
   tools, prompt injection and changing business rules.
4. **A portable artifact** that moves between workflows, test infrastructure, agent runtimes and validation tools.

Our reading adds four more:

- **The paper publishes no schema** below the four top-level fields. Two teams' CIGE files will not match without an
  agreement.
- **Guardrails are only as strong as the runtime** that enforces them. A rule in a YAML file stops nothing by itself.
- **Good intent and evidence rules are the hard part.** AI does not write them for you; a person who knows the business
  rule does.
- **The reported figures come from one platform.** Agentic AI test automation on another stack may behave differently.

Repairs still need a reviewer. That is the work behind
[test automation maintenance with reviewed repairs](/services/automated-testing-maintenance/).

One more distinction matters. CIGE is about testing a product with agents. Checking that an AI agent itself behaves,
from its tool calls to task success, is [AI agent testing](/services/ai-agent-testing/). That is a different job.

## How can a team start agentic AI test automation with CIGE?

Start with one journey and a log, not a migration.

1. **Pick one critical journey** whose tests often break for reasons that are not defects.
2. **Write the intent first**: capability, success criteria, failure criteria and required evidence.
3. **List the guardrails**: destructive actions, production systems, secrets, "no pass from UI text alone" and "no
   intent change without approval".
4. **Move the current steps into execution**, and the environment, account and tools into context. If the steps are
   Playwright code, they stay Playwright code; [Playwright testing services](/services/playwright-testing/) cover that
   layer.
5. **Put a gate in front of the runner**, like the checker above. It blocks a pass without evidence and an unapproved
   intent change.
6. **Log every failure** as product defect, environment drift, test drift or false positive. Log every repair with its
   reviewer.
7. **Decide from the log after a few weeks**: how many failures were drift, how many repairs were accepted and how long
   review took.

CIGE gives agentic AI test automation a test case that states what it proves and where the agent must stop. Treat it
as a structure to try on one journey, not a finished standard.

Zapple.Tech builds and tunes the AI layer for QA: skills, harnesses and agents, infrastructure tuning and monitoring.
To talk about agent-run tests on your own product, [tell us about your suite in the form below](#start).

## FAQ

### Can agentic AI be used for QA automation?

Yes, for drafting, running and repairing tests, as long as each test says what it proves and what the agent may not do. CIGE puts that into four fields: context, intent, guardrails and execution. The agent may repair the path, but a person still approves any change to a test's intent, and a pass needs the evidence the test asks for.

### What does CIGE stand for?

CIGE stands for Context, Intent, Guardrails and Execution, the four fields of a test case in a paper by two authors from Amazon.com Services LLC. Context says where the test runs and which tools it may load. Intent says why the test exists and what proves a pass. Guardrails say what the agent must not do. Execution gives the first path to try.

### Is CIGE a replacement for BDD or the Page Object Model?

No. The authors describe CIGE as a layer for agent-run tests, not a replacement for existing practices. In practice, a BDD scenario can supply the intent, especially its success criteria. Page objects and keyword libraries can become execution steps or tools the agent may call. CIGE adds the guardrails and evidence rules around them.

### Do CIGE tests heal themselves?

Only under control. An agent may repair context after environment drift or execution after test drift. Candidate repairs are replayed in an isolated environment, and a person reviews every suggested change before it is committed. Intent does not change without approval, so a repair cannot quietly change what the test checks.

### Has CIGE been tested in practice?

The authors report lower maintenance effort, fewer false positives and shorter runs after adopting CIGE in one codeless agentic testing platform. It is an experience report from a single platform. The paper states no baseline, period or test count, so the results describe that platform and are not general findings.

## Sources

- [Choppa and Kovi, CIGE: An Agentic AI Test Case Standard (Amazon Science, QRS 2026): definitions sec. 1-3, Table 1, workflow sec. 4, Table 2, Table 3, limitations sec. 7](https://www.amazon.science/publications/cige-an-agentic-ai-test-case-standard)
- [ISO/IEC/IEEE 29119-5:2024, Software testing, Part 5: Keyword-Driven Testing](https://doi.org/10.1109/IEEESTD.2024.10806546)
- [Wang, Poskitt and Sun, AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents (arXiv, 2025)](https://arxiv.org/abs/2503.18666)
- [Anthropic, Effective context engineering for AI agents (2025)](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents)
