Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Test AI security boundaries

    View as Markdown

    Check that the security boundaries of your AI application hold when it runs: one tenant cannot retrieve another tenant’s documents, a user cannot make the assistant call a tool they are not allowed to, and a retrieved document cannot trigger an unapproved action. Keep the result as a receipt you can compare before and after a fix.

    This is regression testing of assertions you write and review. Keryn does not discover routes, generate attacks or judge model behaviour on its own.

    • A test deployment of your app that you control, with test users for at least one actor, and a way to reset it. Packs call test-only routes under /__keryn_tests/ that you implement; creating a pack sends no requests.
    • A token for each actor in an environment variable (never in the suite file).
    • Optional but recommended: a test-only endpoint that returns what the app actually did (which tools ran, which documents were retrieved), see Observed behaviour.
    • For code and dependency checks of the same app, see static analysis and dependencies. For web-layer testing of the running app, see test a running web app.
    Terminal window
    keryn ai-test packs

    There are three packs:

    Pack What it proves
    tenant-isolation A known owned document is retrievable, and another tenant’s document is denied at the retrieval boundary.
    tool-authorization An allowed tool runs and a forbidden tool does not; the authorisation decisions match what actually executed.
    indirect-injection A synthetic adversarial document is retrieved and cannot cause an unapproved tool call or state change.

    Create a suite from one:

    Terminal window
    keryn ai-test init tenant-isolation \
    --target https://staging.example.com \
    --application support-assistant --revision 2026-11-08.1 \
    --actor alice --tenant tenant-a --auth-env STAGING_ALICE_TOKEN \
    --out ai-tests/tenant-isolation

    This writes suite.json and a README.md into the new directory. --target is an origin only (no path, query or credentials). --application and --revision are labels you declare; Keryn does not verify that the deployed code matches the revision. If you omit --auth-env, the suite reads KERYN_TEST_AUTHORIZATION; the default actor is tester in tenant-a.

    You can inspect the generated files before editing the suite:

    Terminal window
    ls ai-tests/tenant-isolation
    README.md
    suite.json

    Open suite.json. The pack is a runnable contract, not an automatic scan: it assumes your test app implements /__keryn_tests/identity, /__keryn_tests/reset/..., /__keryn_tests/control/..., /__keryn_tests/exercise/..., /__keryn_tests/cleanup/... and /__keryn_tests/observations. Adapt the routes, synthetic inputs and fixtures to your app before you run it.

    A suite (keryn.dev/ai-security-suite/v1) is plain JSON with a fixed shape. Unknown fields, ambiguous paths and bare assertions are rejected before any request is sent.

    • target.baseUrl: one origin. Every request goes to it; redirects are refused.
    • actors: who is making requests. Each actor has headers, at least one of them (authorization, cookie or another) taken from an environment variable ({"env": "NAME"}), and a verify request with checks that must show a successful status and a positive identity assertion (for example /actorId equals alice).
    • cases: each has a kind (tenant-isolation, tool-authorization, indirect-injection or custom), an actor, trials (1 to 10, default 1), control steps, steps (the security steps) and optional setup and cleanup.
    • Checks: status, json-equals (a JSON pointer and a value), body-contains, body-not-contains. Every step needs a status check.
    • context: optional reproducibility labels: model, modelRevision, configurationSha256, fixtureRevision.

    A control step is the positive control: it must succeed first, which shows the thing you test for denial would work if the boundary were missing. A failed control makes the trial inconclusive, never a pass.

    Bounds: at most 100 cases, 20 steps per list, 1000 planned HTTP requests, a 1 MiB suite file, and 64 KiB per JSON body.

    Check a suite without sending anything:

    Terminal window
    keryn ai-test validate ai-tests/tenant-isolation/suite.json
    Terminal window
    export STAGING_ALICE_TOKEN='...'
    keryn ai-test run ai-tests/tenant-isolation/suite.json --out out/ai-tests

    For each trial Keryn:

    1. Verifies the actor’s identity (the verify request), before anything else.
    2. Runs setup, then the control steps, then the security steps.
    3. If the suite has an observation endpoint, collects what the app recorded and evaluates it.
    4. Verifies the identity again, then runs cleanup.

    Limits: --timeout <seconds> is per request (default 30, at most 300); the whole suite has a 300 second deadline; a response is read up to 1 MiB. Redirects are refused. Credentials are read from the environment before the first request; a missing or malformed variable stops the run before anything is sent.

    --suite-sha256 <digest> pins the suite bytes: if the file changed since you computed the digest, the run stops and no requests are sent. keryn ci uses this between its preflight and its run.

    HTTP responses alone cannot show that a tool did not execute. If you add an observations path to the suite, Keryn sends five correlation headers with each request (x-keryn-test-run-id, -case-id, -trial, -step-id, -phase) and, after the security steps, fetches that path. Your app returns a batch (keryn.dev/ai-observations/v1) of events such as tool.requested, tool.authorized, tool.denied, tool.executed, retrieval.requested, retrieval.allowed, retrieval.denied, approval.granted, state.changed and outbound.sent.

    A case can then assert: event counts, tool-authorization (nothing executes without an authorisation), retrieval-isolation (nothing is allowed that belongs to another tenant), approval-before-state-change, and outbound-allowlist (outbound sends only go to listed origins). Incomplete, dropped, out-of-order or wrong-identity telemetry is inconclusive, never a pass. The events are claims made by your own instrumentation; Keryn does not independently attest them.

    Terminal window
    cat out/ai-tests/summary.md
    # AI security regression
    Suite: pack-tenant-isolation
    Application: support-assistant; declared revision: 2026-11-08.1
    Exit: 0
    Observed assertion violations: 0; inconclusive: 0; execution failures: 0; skipped: 0.
    | Case | Actor | Trial | Result |
    |---|---|---|---|
    | tenant-isolation-boundary | alice | 1 | passed |
    | tenant-isolation-boundary | alice | 2 | passed |
    See receipt.json for per-step evidence digests and check results. Request headers and response bodies are not stored.
    A passing scenario establishes only the recorded assertions for the tested configuration and trials.

    The run directory holds receipt.json and summary.md (and, when observations were collected, observations.json plus numbered batch files).

    The receipt records, per trial and step: the SHA-256 of the request declaration, the response status, size and SHA-256, and each check’s result. It does not store request headers, response bodies, resolved credentials or error text. That is what lets you keep it as evidence without storing the conversations it tested.

    Exit Outcome Meaning
    0 passed every assertion held in every trial that ran
    1 violation a security assertion observed a mismatch
    2 (usage) invalid suite or options
    3 inconclusive a control failed, an identity expired, an assertion could not be observed, a trial was skipped, or telemetry was incomplete
    4 failed execution failure: timeout, redirect, response too large, missing credentials

    A violation is a mismatch with what you wrote, not proof that someone could exploit the app. A pass covers only the recorded assertions, the declared application metadata and the trials that ran. It is not proof that the model is safe, that tenants are isolated or that prompt injection is impossible. The receipt states this itself in its limitations.

    Keep the receipt from the run that failed, and its suite. After the fix, run the suite again (the revision label may change; the trial count may go up; nothing else may change), then compare:

    Terminal window
    keryn ai-test verify out/after/receipt.json --suite after-suite.json \
    --baseline out/before/receipt.json --baseline-suite before-suite.json \
    --out out/verification

    Keryn re-checks both receipts offline against their suites (every step, check and count must be consistent; a receipt cannot be hand-edited into a pass) and writes verification.json and summary.md. The result is one of:

    Result Exit Meaning
    verified-for-recorded-trials 0 the baseline reproduced the violation and the current run has none
    still-failing 1 the current run still violates
    baseline-not-reproduced 3 the “before” run had no violation, so there is nothing to compare
    not-comparable 3 the suite changed beyond the revision label and trial count, or trials were reduced
    inconclusive 3 a receipt is incomplete, inconsistent or has non-violation failures

    Receipts are supplied evidence, not attestations of what was deployed. verified-for-recorded-trials says what it says: the recorded assertions passed in the recorded trials.

    Terminal window
    keryn ai-test capture session.har \
    --target https://staging.example.com --application support-assistant --revision 2026-11-08.1 \
    --out ai-tests/draft

    Check the draft status before reviewing it:

    Terminal window
    jq -r '.status, .executable' ai-tests/draft/capture.json
    draft
    false

    This reads a HAR 1.2 file locally (at most 16 MiB and 1000 entries) and writes draft.json, diagnostics.json and capture.json. The draft is not executable and is redacted (sensitive values and path segments are replaced by markers such as __KERYN_REDACTED__). You add the reviewed routes, synthetic inputs, secret references, identity checks, positive controls and security assertions before you have a suite. No request is replayed.

    Terminal window
    keryn ci . --out out/ci \
    --ai-tests ai-tests/tenant-isolation/suite.json \
    --ai-baseline out/before/receipt.json --ai-baseline-suite before-suite.json

    --ai-tests runs the suite with its digest pinned at preflight; --ai-baseline and --ai-baseline-suite add the fix verification. --application and --revision bind the run to your own labels and must match the suite. The aggregate exit code puts execution failures first, then usage, incomplete, findings and warnings. See Add Keryn to CI.

    8. Replay a detection rule over the observations

    Section titled “8. Replay a detection rule over the observations”

    The batches a run saves can also answer a different question: would this detection rule have fired on what the app did? keryn detection replay runs one Sigma single-event rule over them offline and writes a sealed receipt. Here a rule flags an outbound send to an origin outside an allow-list:

    title: Outbound send to an unlisted origin
    logsource: {product: keryn, service: ai-observations, category: outbound}
    detection:
    sent:
    type: outbound.sent
    allowed:
    destinationOrigin|endswith: '.example.com'
    condition: sent and not allowed

    A saved batch (one per case and trial, observations-0001.json and so on in the run directory) looks like this:

    {"schema": "keryn.dev/ai-observations/v1", "runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1,
    "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice", "complete": true, "droppedEvents": 0,
    "events": [
    {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
    "stepId": "exfiltrate", "phase": "security", "sequence": 1, "type": "tool.executed", "operationId": "op-1", "toolId": "http_fetch"},
    {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
    "stepId": "exfiltrate", "phase": "security", "sequence": 2, "type": "outbound.sent", "operationId": "op-2", "destinationOrigin": "https://collector.attacker.test"},
    {"runId": "run-1", "caseId": "tenant-isolation-boundary", "trial": 1, "application": {"id": "support-assistant", "build": "2026-11-08.1"}, "actorId": "alice",
    "stepId": "exfiltrate", "phase": "security", "sequence": 3, "type": "outbound.sent", "operationId": "op-3", "destinationOrigin": "https://api.example.com"}
    ]}

    Label the events you know the answer for (event ids are run/case/trial/sequence), so the receipt can count precision and recall:

    {"run-1/tenant-isolation-boundary/1/2": "match", "run-1/tenant-isolation-boundary/1/3": "no_match"}
    Terminal window
    keryn detection replay --rule detection/outbound.yml --observations detection/observations-0001.json \
    --labels detection/labels.json --save-input detection/input.json --receipt detection/receipt.json
    Detection replay (sigma-replay-spike/0 (sigma-rust 0.7.0)): rule admitted
    Events: 3 · match 1 · no match 2 · incomplete 0 · refused 0 · provisional 0
    Quality over labelled events: precision 1/1 = 1.0000, recall 1/1 = 1.0000; excluded: incomplete 0, refused 0, provisional 0; unlabelled 1
    Receipt sha256 e15d512f8935d65978dcb9f23bc4c72c238b6b00775c5baa23bddc40defba6e3. These numbers speak for this cohort only: no production detection claim follows.
    Source: 1 AI observation batch(es), all complete (mapping ai-observations-sigma/0)

    Pass every batch of the run with one --observations each. The events keep the field names of the observation contract (application.id stays nested). Three rules decide what the numbers can claim:

    • A field another event type carries is absent, so the rule simply does not match that event (the tool.executed event above).
    • A rule about something the contract does not record (a command line, a payload, a timestamp) is incomplete for every event, and the command exits 3. Aggregation, correlation and time windows are refused.
    • A batch that is not complete or reports droppedEvents makes the source incomplete: every no-match becomes provisional, the summary names the batch, and the command exits 3.

    Anyone with the input and the receipt can check that one is exactly the other’s result; an edited receipt fails its seal:

    Terminal window
    keryn detection validate detection/input.json detection/receipt.json
    valid: detection/receipt.json is the receipt detection/input.json gives

    The replay runs in a worker that is stopped at --timeout (60 seconds by default). Nothing is sent anywhere. The same replay runs in the browser under Import > Detection on keryn.app. The numbers describe these batches only: a fixture or a few test runs support no production detection claim. Every flag: keryn detection replay.

    keryn ai-test run sends the suite’s HTTP requests, with the credentials from your environment, to the one origin in the suite. Nothing goes to Keryn. validate, packs, init, capture, verify and keryn detection send no requests. The receipt and summary stay in the directory you chose; sharing them is your decision, and they hold hashes rather than payloads.

    • You write the assertions. A suite with weak checks passes weak tests; the case kind is a label of your intent and does not mean the vulnerability class is covered (coverage).
    • The runner has no browser, no cookie jar and no variable substitution between steps. Each request is declared in advance.
    • Steps and hooks may change state in the target. Use disposable fixtures.
    • Application revision and target are declared, not verified. Observations are your app’s own report.
    • Finite trials do not prove absence of a flaw, and a run that cannot observe an assertion says inconclusive, not passed.