# Test AI security boundaries

> Replay reviewed, authenticated requests against a test deployment of your AI app, check tenant isolation and tool authorisation, and verify that a fix held.

Source: https://keryn.dev/docs/guides/ai-testing/

## Goal

Check that the security boundaries of your AI application hold when it runs: one tenant cannot retrieve another tenant's documents, a user cannot make the assistant call a tool they are not allowed to, and a retrieved document cannot trigger an unapproved action. Keep the result as a receipt you can compare before and after a fix.

This is regression testing of assertions **you write and review**. Keryn does not discover routes, generate attacks or judge model behaviour on its own.

> **Note: In the next release**
> `keryn ai-test` (all subcommands) and the `--ai-tests` options of `keryn ci` are not in Keryn 0.5.2. Run `keryn ai-test --help` on the version you have to see what it supports.

> **Caution: Only test what you own or may test**
> `keryn ai-test run` sends requests to the origin written in the suite, with credentials you supply, and the steps can change state in that application. Use a staging or test deployment with disposable fixtures. Do not point it at production data or at a system you are not authorised to test.

## Before you start

- A test deployment of your app that you control, with test users for at least one actor, and a way to reset it. Packs call test-only routes under `/__keryn_tests/` that **you** implement; creating a pack sends no requests.
- A token for each actor in an environment variable (never in the suite file).
- Optional but recommended: a test-only endpoint that returns what the app actually did (which tools ran, which documents were retrieved), see [Observed behaviour](#observed-behaviour).
- For code and dependency checks of the same app, see [static analysis](https://keryn.dev/docs/guides/sast/) and [dependencies](https://keryn.dev/docs/guides/sca/). For web-layer testing of the running app, see [test a running web app](https://keryn.dev/docs/guides/dast/).

## 1. Start from a pack

```sh
keryn ai-test packs
```

There are three packs:

| Pack | What it proves |
|---|---|
| `tenant-isolation` | A known owned document is retrievable, and another tenant's document is denied at the retrieval boundary. |
| `tool-authorization` | An allowed tool runs and a forbidden tool does not; the authorisation decisions match what actually executed. |
| `indirect-injection` | A synthetic adversarial document is retrieved and cannot cause an unapproved tool call or state change. |

Create a suite from one:

```sh
keryn ai-test init tenant-isolation \
  --target https://staging.example.com \
  --application support-assistant --revision 2026-11-08.1 \
  --actor alice --tenant tenant-a --auth-env STAGING_ALICE_TOKEN \
  --out ai-tests/tenant-isolation
```

This writes `suite.json` and a `README.md` into the new directory. `--target` is an origin only (no path, query or credentials). `--application` and `--revision` are labels **you** declare; Keryn does not verify that the deployed code matches the revision. If you omit `--auth-env`, the suite reads `KERYN_TEST_AUTHORIZATION`; the default actor is `tester` in `tenant-a`.

You can inspect the generated files before editing the suite:

```sh
ls ai-tests/tenant-isolation
```

```text
README.md
suite.json
```

Open `suite.json`. The pack is a runnable contract, not an automatic scan: it assumes your test app implements `/__keryn_tests/identity`, `/__keryn_tests/reset/...`, `/__keryn_tests/control/...`, `/__keryn_tests/exercise/...`, `/__keryn_tests/cleanup/...` and `/__keryn_tests/observations`. Adapt the routes, synthetic inputs and fixtures to your app before you run it.

## 2. Know what a suite says

A suite (`keryn.dev/ai-security-suite/v1`) is plain JSON with a fixed shape. Unknown fields, ambiguous paths and bare assertions are rejected before any request is sent.

- **`target.baseUrl`**: one origin. Every request goes to it; redirects are refused.
- **`actors`**: who is making requests. Each actor has headers, at least one of them (`authorization`, `cookie` or another) taken from an **environment variable** (`{"env": "NAME"}`), and a `verify` request with checks that must show a successful status **and** a positive identity assertion (for example `/actorId` equals `alice`).
- **`cases`**: each has a `kind` (`tenant-isolation`, `tool-authorization`, `indirect-injection` or `custom`), an actor, `trials` (1 to 10, default 1), `control` steps, `steps` (the security steps) and optional `setup` and `cleanup`.
- **Checks**: `status`, `json-equals` (a JSON pointer and a value), `body-contains`, `body-not-contains`. Every step needs a status check.
- **`context`**: optional reproducibility labels: `model`, `modelRevision`, `configurationSha256`, `fixtureRevision`.

A **control** step is the positive control: it must succeed first, which shows the thing you test for denial would work if the boundary were missing. A failed control makes the trial inconclusive, never a pass.

Bounds: at most 100 cases, 20 steps per list, 1000 planned HTTP requests, a 1 MiB suite file, and 64 KiB per JSON body.

Check a suite without sending anything:

```sh
keryn ai-test validate ai-tests/tenant-isolation/suite.json
```

## 3. Run it

```sh
export STAGING_ALICE_TOKEN='...'
keryn ai-test run ai-tests/tenant-isolation/suite.json --out out/ai-tests
```

For each trial Keryn:

1. Verifies the actor's identity (the `verify` request), before anything else.
2. Runs `setup`, then the `control` steps, then the security `steps`.
3. If the suite has an observation endpoint, collects what the app recorded and evaluates it.
4. Verifies the identity again, then runs `cleanup`.

Limits: `--timeout <seconds>` is per request (default 30, at most 300); the whole suite has a 300 second deadline; a response is read up to 1 MiB. Redirects are refused. Credentials are read from the environment before the first request; a missing or malformed variable stops the run before anything is sent.

`--suite-sha256 <digest>` pins the suite bytes: if the file changed since you computed the digest, the run stops and no requests are sent. `keryn ci` uses this between its preflight and its run.

### Observed behaviour

HTTP responses alone cannot show that a tool did not execute. If you add an `observations` path to the suite, Keryn sends five correlation headers with each request (`x-keryn-test-run-id`, `-case-id`, `-trial`, `-step-id`, `-phase`) and, after the security steps, fetches that path. Your app returns a batch (`keryn.dev/ai-observations/v1`) of events such as `tool.requested`, `tool.authorized`, `tool.denied`, `tool.executed`, `retrieval.requested`, `retrieval.allowed`, `retrieval.denied`, `approval.granted`, `state.changed` and `outbound.sent`.

A case can then assert: event counts, `tool-authorization` (nothing executes without an authorisation), `retrieval-isolation` (nothing is allowed that belongs to another tenant), `approval-before-state-change`, and `outbound-allowlist` (outbound sends only go to listed origins). Incomplete, dropped, out-of-order or wrong-identity telemetry is **inconclusive**, never a pass. The events are claims made by your own instrumentation; Keryn does not independently attest them.

## 4. Read the result

```sh
cat out/ai-tests/summary.md
```

```text
# AI security regression

Suite: pack-tenant-isolation
Application: support-assistant; declared revision: 2026-11-08.1
Exit: 0

Observed assertion violations: 0; inconclusive: 0; execution failures: 0; skipped: 0.

| Case | Actor | Trial | Result |
|---|---|---|---|
| tenant-isolation-boundary | alice | 1 | passed |
| tenant-isolation-boundary | alice | 2 | passed |

See receipt.json for per-step evidence digests and check results. Request headers and response bodies are not stored.
A passing scenario establishes only the recorded assertions for the tested configuration and trials.
```

The run directory holds `receipt.json` and `summary.md` (and, when observations were collected, `observations.json` plus numbered batch files).

The receipt records, per trial and step: the SHA-256 of the request declaration, the response status, size and SHA-256, and each check's result. It does not store request headers, response bodies, resolved credentials or error text. That is what lets you keep it as evidence without storing the conversations it tested.

| Exit | Outcome | Meaning |
|---|---|---|
| 0 | passed | every assertion held in every trial that ran |
| 1 | violation | a security assertion observed a mismatch |
| 2 | (usage) | invalid suite or options |
| 3 | inconclusive | a control failed, an identity expired, an assertion could not be observed, a trial was skipped, or telemetry was incomplete |
| 4 | failed | execution failure: timeout, redirect, response too large, missing credentials |

A violation is a mismatch with what you wrote, not proof that someone could exploit the app. A pass covers only the recorded assertions, the declared application metadata and the trials that ran. It is not proof that the model is safe, that tenants are isolated or that prompt injection is impossible. The receipt states this itself in its `limitations`.

## 5. Verify a fix

Keep the receipt from the run that failed, and its suite. After the fix, run the suite again (the revision label may change; the trial count may go up; nothing else may change), then compare:

```sh
keryn ai-test verify out/after/receipt.json --suite after-suite.json \
  --baseline out/before/receipt.json --baseline-suite before-suite.json \
  --out out/verification
```

Keryn re-checks both receipts offline against their suites (every step, check and count must be consistent; a receipt cannot be hand-edited into a pass) and writes `verification.json` and `summary.md`. The result is one of:

| Result | Exit | Meaning |
|---|---|---|
| `verified-for-recorded-trials` | 0 | the baseline reproduced the violation and the current run has none |
| `still-failing` | 1 | the current run still violates |
| `baseline-not-reproduced` | 3 | the "before" run had no violation, so there is nothing to compare |
| `not-comparable` | 3 | the suite changed beyond the revision label and trial count, or trials were reduced |
| `inconclusive` | 3 | a receipt is incomplete, inconsistent or has non-violation failures |

Receipts are supplied evidence, not attestations of what was deployed. `verified-for-recorded-trials` says what it says: the recorded assertions passed in the recorded trials.

## 6. Draft a suite from a HAR

```sh
keryn ai-test capture session.har \
  --target https://staging.example.com --application support-assistant --revision 2026-11-08.1 \
  --out ai-tests/draft
```

Check the draft status before reviewing it:

```sh
jq -r '.status, .executable' ai-tests/draft/capture.json
```

```text
draft
false
```

This reads a HAR 1.2 file locally (at most 16 MiB and 1000 entries) and writes `draft.json`, `diagnostics.json` and `capture.json`. The draft is **not executable** and is redacted (sensitive values and path segments are replaced by markers such as `__KERYN_REDACTED__`). You add the reviewed routes, synthetic inputs, secret references, identity checks, positive controls and security assertions before you have a suite. No request is replayed.

## 7. In CI

```sh
keryn ci . --out out/ci \
  --ai-tests ai-tests/tenant-isolation/suite.json \
  --ai-baseline out/before/receipt.json --ai-baseline-suite before-suite.json
```

`--ai-tests` runs the suite with its digest pinned at preflight; `--ai-baseline` and `--ai-baseline-suite` add the fix verification. `--application` and `--revision` bind the run to your own labels and must match the suite. The aggregate exit code puts execution failures first, then usage, incomplete, findings and warnings. See [Add Keryn to CI](https://keryn.dev/docs/guides/ci/).

## 8. Replay a detection rule over the observations

> **Caution: Ships in the next release**
> On main; Keryn 0.5.2 has no `keryn detection` command.

The batches a run saves can also answer a different question: would this detection rule have fired on what the app did? `keryn detection replay` runs one Sigma single-event rule over them offline and writes a sealed receipt. Here a rule flags an outbound send to an origin outside an allow-list:

```yaml
title: Outbound send to an unlisted origin
logsource: {product: keryn, service: ai-observations, category: outbound}
detection:
  sent:
    type: outbound.sent
  allowed:
    destinationOrigin|endswith: '.example.com'
  condition: sent and not allowed
```

A saved batch (one per case and trial, `observations-0001.json` and so on in the run directory) looks like this:

```json
{"schema": "keryn.dev/ai-observations/v1", "runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1,
 "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "complete": true, "droppedEvents": 0,
 "events": [
  {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
   "stepId": "exfiltrate", "phase": "security", "sequence": 1, "type": "tool.executed", "operationId": "op-1", "toolId": "http_fetch"},
  {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
   "stepId": "exfiltrate", "phase": "security", "sequence": 2, "type": "outbound.sent", "operationId": "op-2", "destinationOrigin": "https://collector.attacker.test"},
  {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
   "stepId": "exfiltrate", "phase": "security", "sequence": 3, "type": "outbound.sent", "operationId": "op-3", "destinationOrigin": "https://api.example.com"}
 ]}
```

Label the events you know the answer for (event ids are run/case/trial/sequence), so the receipt can count precision and recall:

```json
{"run-1/tenant-isolation-boundary/1/2": "match", "run-1/tenant-isolation-boundary/1/3": "no_match"}
```

```sh
keryn detection replay --rule detection/outbound.yml --observations detection/observations-0001.json \
  --labels detection/labels.json --save-input detection/input.json --receipt detection/receipt.json
```

```text
Detection replay (sigma-replay-spike/0 (sigma-rust 0.7.0)): rule admitted
Events: 3 · match 1 · no match 2 · incomplete 0 · refused 0 · provisional 0
Quality over labelled events: precision 1/1 = 1.0000, recall 1/1 = 1.0000; excluded: incomplete 0, refused 0, provisional 0; unlabelled 1
Receipt sha256 e15d512f8935d65978dcb9f23bc4c72c238b6b00775c5baa23bddc40defba6e3. These numbers speak for this cohort only: no production detection claim follows.
Source: 1 AI observation batch(es), all complete (mapping ai-observations-sigma/0)
```

Pass every batch of the run with one `--observations` each. The events keep the field names of the observation contract (`application.id` stays nested). Three rules decide what the numbers can claim:

- A field another event type carries is absent, so the rule simply does not match that event (the `tool.executed` event above).
- A rule about something the contract does not record (a command line, a payload, a timestamp) is **incomplete** for every event, and the command exits 3. Aggregation, correlation and time windows are refused.
- A batch that is not `complete` or reports `droppedEvents` makes the source incomplete: every no-match becomes **provisional**, the summary names the batch, and the command exits 3.

Anyone with the input and the receipt can check that one is exactly the other's result; an edited receipt fails its seal:

```sh
keryn detection validate detection/input.json detection/receipt.json
```

```text
valid: detection/receipt.json is the receipt detection/input.json gives
```

The replay runs in a worker that is stopped at `--timeout` (60 seconds by default). Nothing is sent anywhere. The same replay runs in the browser under **Import > Detection** on keryn.app. The numbers describe these batches only: a fixture or a few test runs support no production detection claim. Every flag: [`keryn detection replay`](https://keryn.dev/docs/reference/cli/detection-replay/).

## What leaves your machine

`keryn ai-test run` sends the suite's HTTP requests, with the credentials from your environment, to the one origin in the suite. Nothing goes to Keryn. `validate`, `packs`, `init`, `capture`, `verify` and `keryn detection` send no requests. The receipt and summary stay in the directory you chose; sharing them is your decision, and they hold hashes rather than payloads.

## Limits

- You write the assertions. A suite with weak checks passes weak tests; the case `kind` is a label of your intent and does not mean the vulnerability class is covered ([coverage](https://keryn.dev/docs/concepts/coverage/)).
- The runner has no browser, no cookie jar and no variable substitution between steps. Each request is declared in advance.
- Steps and hooks may change state in the target. Use disposable fixtures.
- Application revision and target are declared, not verified. Observations are your app's own report.
- Finite trials do not prove absence of a flaw, and a run that cannot observe an assertion says inconclusive, not passed.

## Next

- Probe the web layer of the same deployment: [test a running web app](https://keryn.dev/docs/guides/dast/).
- Keep the receipts for a review: [evidence for auditors](https://keryn.dev/docs/guides/auditors/).
- Decide what a pass or a Hold means for the release: [release decisions](https://keryn.dev/docs/guides/release-decisions/).
