Test AI security boundaries
Check that the security boundaries of your AI application hold when it runs: one tenant cannot retrieve another tenant’s documents, a user cannot make the assistant call a tool they are not allowed to, and a retrieved document cannot trigger an unapproved action. Keep the result as a receipt you can compare before and after a fix.
This is regression testing of assertions you write and review. Keryn does not discover routes, generate attacks or judge model behaviour on its own.
Before you start
Section titled “Before you start”- A test deployment of your app that you control, with test users for at least one actor, and a way to reset it. Packs call test-only routes under
/__keryn_tests/that you implement; creating a pack sends no requests. - A token for each actor in an environment variable (never in the suite file).
- Optional but recommended: a test-only endpoint that returns what the app actually did (which tools ran, which documents were retrieved), see Observed behaviour.
- For code and dependency checks of the same app, see static analysis and dependencies. For web-layer testing of the running app, see test a running web app.
1. Start from a pack
Section titled “1. Start from a pack”keryn ai-test packsThere are three packs:
| Pack | What it proves |
|---|---|
tenant-isolation |
A known owned document is retrievable, and another tenant’s document is denied at the retrieval boundary. |
tool-authorization |
An allowed tool runs and a forbidden tool does not; the authorisation decisions match what actually executed. |
indirect-injection |
A synthetic adversarial document is retrieved and cannot cause an unapproved tool call or state change. |
Create a suite from one:
keryn ai-test init tenant-isolation \ --target https://staging.example.com \ --application support-assistant --revision 2026-11-08.1 \ --actor alice --tenant tenant-a --auth-env STAGING_ALICE_TOKEN \ --out ai-tests/tenant-isolationThis writes suite.json and a README.md into the new directory. --target is an origin only (no path, query or credentials). --application and --revision are labels you declare; Keryn does not verify that the deployed code matches the revision. If you omit --auth-env, the suite reads KERYN_TEST_AUTHORIZATION; the default actor is tester in tenant-a.
You can inspect the generated files before editing the suite:
ls ai-tests/tenant-isolationREADME.mdsuite.jsonOpen suite.json. The pack is a runnable contract, not an automatic scan: it assumes your test app implements /__keryn_tests/identity, /__keryn_tests/reset/..., /__keryn_tests/control/..., /__keryn_tests/exercise/..., /__keryn_tests/cleanup/... and /__keryn_tests/observations. Adapt the routes, synthetic inputs and fixtures to your app before you run it.
2. Know what a suite says
Section titled “2. Know what a suite says”A suite (keryn.dev/ai-security-suite/v1) is plain JSON with a fixed shape. Unknown fields, ambiguous paths and bare assertions are rejected before any request is sent.
target.baseUrl: one origin. Every request goes to it; redirects are refused.actors: who is making requests. Each actor has headers, at least one of them (authorization,cookieor another) taken from an environment variable ({"env": "NAME"}), and averifyrequest with checks that must show a successful status and a positive identity assertion (for example/actorIdequalsalice).cases: each has akind(tenant-isolation,tool-authorization,indirect-injectionorcustom), an actor,trials(1 to 10, default 1),controlsteps,steps(the security steps) and optionalsetupandcleanup.- Checks:
status,json-equals(a JSON pointer and a value),body-contains,body-not-contains. Every step needs a status check. context: optional reproducibility labels:model,modelRevision,configurationSha256,fixtureRevision.
A control step is the positive control: it must succeed first, which shows the thing you test for denial would work if the boundary were missing. A failed control makes the trial inconclusive, never a pass.
Bounds: at most 100 cases, 20 steps per list, 1000 planned HTTP requests, a 1 MiB suite file, and 64 KiB per JSON body.
Check a suite without sending anything:
keryn ai-test validate ai-tests/tenant-isolation/suite.json3. Run it
Section titled “3. Run it”export STAGING_ALICE_TOKEN='...'keryn ai-test run ai-tests/tenant-isolation/suite.json --out out/ai-testsFor each trial Keryn:
- Verifies the actor’s identity (the
verifyrequest), before anything else. - Runs
setup, then thecontrolsteps, then the securitysteps. - If the suite has an observation endpoint, collects what the app recorded and evaluates it.
- Verifies the identity again, then runs
cleanup.
Limits: --timeout <seconds> is per request (default 30, at most 300); the whole suite has a 300 second deadline; a response is read up to 1 MiB. Redirects are refused. Credentials are read from the environment before the first request; a missing or malformed variable stops the run before anything is sent.
--suite-sha256 <digest> pins the suite bytes: if the file changed since you computed the digest, the run stops and no requests are sent. keryn ci uses this between its preflight and its run.
Observed behaviour
Section titled “Observed behaviour”HTTP responses alone cannot show that a tool did not execute. If you add an observations path to the suite, Keryn sends five correlation headers with each request (x-keryn-test-run-id, -case-id, -trial, -step-id, -phase) and, after the security steps, fetches that path. Your app returns a batch (keryn.dev/ai-observations/v1) of events such as tool.requested, tool.authorized, tool.denied, tool.executed, retrieval.requested, retrieval.allowed, retrieval.denied, approval.granted, state.changed and outbound.sent.
A case can then assert: event counts, tool-authorization (nothing executes without an authorisation), retrieval-isolation (nothing is allowed that belongs to another tenant), approval-before-state-change, and outbound-allowlist (outbound sends only go to listed origins). Incomplete, dropped, out-of-order or wrong-identity telemetry is inconclusive, never a pass. The events are claims made by your own instrumentation; Keryn does not independently attest them.
4. Read the result
Section titled “4. Read the result”cat out/ai-tests/summary.md# AI security regression
Suite: pack-tenant-isolationApplication: support-assistant; declared revision: 2026-11-08.1Exit: 0
Observed assertion violations: 0; inconclusive: 0; execution failures: 0; skipped: 0.
| Case | Actor | Trial | Result ||---|---|---|---|| tenant-isolation-boundary | alice | 1 | passed || tenant-isolation-boundary | alice | 2 | passed |
See receipt.json for per-step evidence digests and check results. Request headers and response bodies are not stored.A passing scenario establishes only the recorded assertions for the tested configuration and trials.The run directory holds receipt.json and summary.md (and, when observations were collected, observations.json plus numbered batch files).
The receipt records, per trial and step: the SHA-256 of the request declaration, the response status, size and SHA-256, and each check’s result. It does not store request headers, response bodies, resolved credentials or error text. That is what lets you keep it as evidence without storing the conversations it tested.
| Exit | Outcome | Meaning |
|---|---|---|
| 0 | passed | every assertion held in every trial that ran |
| 1 | violation | a security assertion observed a mismatch |
| 2 | (usage) | invalid suite or options |
| 3 | inconclusive | a control failed, an identity expired, an assertion could not be observed, a trial was skipped, or telemetry was incomplete |
| 4 | failed | execution failure: timeout, redirect, response too large, missing credentials |
A violation is a mismatch with what you wrote, not proof that someone could exploit the app. A pass covers only the recorded assertions, the declared application metadata and the trials that ran. It is not proof that the model is safe, that tenants are isolated or that prompt injection is impossible. The receipt states this itself in its limitations.
5. Verify a fix
Section titled “5. Verify a fix”Keep the receipt from the run that failed, and its suite. After the fix, run the suite again (the revision label may change; the trial count may go up; nothing else may change), then compare:
keryn ai-test verify out/after/receipt.json --suite after-suite.json \ --baseline out/before/receipt.json --baseline-suite before-suite.json \ --out out/verificationKeryn re-checks both receipts offline against their suites (every step, check and count must be consistent; a receipt cannot be hand-edited into a pass) and writes verification.json and summary.md. The result is one of:
| Result | Exit | Meaning |
|---|---|---|
verified-for-recorded-trials |
0 | the baseline reproduced the violation and the current run has none |
still-failing |
1 | the current run still violates |
baseline-not-reproduced |
3 | the “before” run had no violation, so there is nothing to compare |
not-comparable |
3 | the suite changed beyond the revision label and trial count, or trials were reduced |
inconclusive |
3 | a receipt is incomplete, inconsistent or has non-violation failures |
Receipts are supplied evidence, not attestations of what was deployed. verified-for-recorded-trials says what it says: the recorded assertions passed in the recorded trials.
6. Draft a suite from a HAR
Section titled “6. Draft a suite from a HAR”keryn ai-test capture session.har \ --target https://staging.example.com --application support-assistant --revision 2026-11-08.1 \ --out ai-tests/draftCheck the draft status before reviewing it:
jq -r '.status, .executable' ai-tests/draft/capture.jsondraftfalseThis reads a HAR 1.2 file locally (at most 16 MiB and 1000 entries) and writes draft.json, diagnostics.json and capture.json. The draft is not executable and is redacted (sensitive values and path segments are replaced by markers such as __KERYN_REDACTED__). You add the reviewed routes, synthetic inputs, secret references, identity checks, positive controls and security assertions before you have a suite. No request is replayed.
7. In CI
Section titled “7. In CI”keryn ci . --out out/ci \ --ai-tests ai-tests/tenant-isolation/suite.json \ --ai-baseline out/before/receipt.json --ai-baseline-suite before-suite.json--ai-tests runs the suite with its digest pinned at preflight; --ai-baseline and --ai-baseline-suite add the fix verification. --application and --revision bind the run to your own labels and must match the suite. The aggregate exit code puts execution failures first, then usage, incomplete, findings and warnings. See Add Keryn to CI.
8. Replay a detection rule over the observations
Section titled “8. Replay a detection rule over the observations”The batches a run saves can also answer a different question: would this detection rule have fired on what the app did? keryn detection replay runs one Sigma single-event rule over them offline and writes a sealed receipt. Here a rule flags an outbound send to an origin outside an allow-list:
title: Outbound send to an unlisted originlogsource: {product: keryn, service: ai-observations, category: outbound}detection: sent: type: outbound.sent allowed: destinationOrigin|endswith: '.example.com' condition: sent and not allowedA saved batch (one per case and trial, observations-0001.json and so on in the run directory) looks like this:
{"schema": "keryn.dev/ai-observations/v1", "runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "complete": true, "droppedEvents": 0, "events": [ {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "stepId": "exfiltrate", "phase": "security", "sequence": 1, "type": "tool.executed", "operationId": "op-1", "toolId": "http_fetch"}, {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "stepId": "exfiltrate", "phase": "security", "sequence": 2, "type": "outbound.sent", "operationId": "op-2", "destinationOrigin": "https://collector.attacker.test"}, {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "stepId": "exfiltrate", "phase": "security", "sequence": 3, "type": "outbound.sent", "operationId": "op-3", "destinationOrigin": "https://api.example.com"} ]}Label the events you know the answer for (event ids are run/case/trial/sequence), so the receipt can count precision and recall:
{"run-1/tenant-isolation-boundary/1/2": "match", "run-1/tenant-isolation-boundary/1/3": "no_match"}keryn detection replay --rule detection/outbound.yml --observations detection/observations-0001.json \ --labels detection/labels.json --save-input detection/input.json --receipt detection/receipt.jsonDetection replay (sigma-replay-spike/0 (sigma-rust 0.7.0)): rule admittedEvents: 3 · match 1 · no match 2 · incomplete 0 · refused 0 · provisional 0Quality over labelled events: precision 1/1 = 1.0000, recall 1/1 = 1.0000; excluded: incomplete 0, refused 0, provisional 0; unlabelled 1Receipt sha256 e15d512f8935d65978dcb9f23bc4c72c238b6b00775c5baa23bddc40defba6e3. These numbers speak for this cohort only: no production detection claim follows.Source: 1 AI observation batch(es), all complete (mapping ai-observations-sigma/0)Pass every batch of the run with one --observations each. The events keep the field names of the observation contract (application.id stays nested). Three rules decide what the numbers can claim:
- A field another event type carries is absent, so the rule simply does not match that event (the
tool.executedevent above). - A rule about something the contract does not record (a command line, a payload, a timestamp) is incomplete for every event, and the command exits 3. Aggregation, correlation and time windows are refused.
- A batch that is not
completeor reportsdroppedEventsmakes the source incomplete: every no-match becomes provisional, the summary names the batch, and the command exits 3.
Anyone with the input and the receipt can check that one is exactly the other’s result; an edited receipt fails its seal:
keryn detection validate detection/input.json detection/receipt.jsonvalid: detection/receipt.json is the receipt detection/input.json givesThe replay runs in a worker that is stopped at --timeout (60 seconds by default). Nothing is sent anywhere. The same replay runs in the browser under Import > Detection on keryn.app. The numbers describe these batches only: a fixture or a few test runs support no production detection claim. Every flag: keryn detection replay.
What leaves your machine
Section titled “What leaves your machine”keryn ai-test run sends the suite’s HTTP requests, with the credentials from your environment, to the one origin in the suite. Nothing goes to Keryn. validate, packs, init, capture, verify and keryn detection send no requests. The receipt and summary stay in the directory you chose; sharing them is your decision, and they hold hashes rather than payloads.
Limits
Section titled “Limits”- You write the assertions. A suite with weak checks passes weak tests; the case
kindis a label of your intent and does not mean the vulnerability class is covered (coverage). - The runner has no browser, no cookie jar and no variable substitution between steps. Each request is declared in advance.
- Steps and hooks may change state in the target. Use disposable fixtures.
- Application revision and target are declared, not verified. Observations are your app’s own report.
- Finite trials do not prove absence of a flaw, and a run that cannot observe an assertion says inconclusive, not passed.
- Probe the web layer of the same deployment: test a running web app.
- Keep the receipts for a review: evidence for auditors.
- Decide what a pass or a Hold means for the release: release decisions.