Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    The proof ladder

    View as Markdown

    Every finding carries a proof level. It says how much evidence stands behind the finding, not how severe it is. Severity (error, warning or info) is the rule’s rating of the impact if the finding is real. The two are shown side by side and neither replaces the other.

    Level Meaning
    proof 0inventory An AI component, capability or posture fact. Never gates as a vulnerability.
    proof 1pattern A code pattern matched.
    proof 2path-in-file A path from a source (user input, model output) to a sink inside one file.
    proof 3path-cross-file A path across files (Python, TypeScript, JavaScript, Go, Java, C#, C/C++). How far it reaches depends on the language: see what Keryn does not do.
    proof 4known-fix The code matches a block that a published advisory’s fix removed; cites the CVE or GHSA.
    proof 5reproduced Verify reproduced it in an isolated container; cites the evidence hash.

    This function asks a model for a shell command and runs it. The first rule only looks for shell=True. The second follows the model’s reply to the call that runs it. Compare the proof levels in the two recorded results.

    Rung 1: a pattern matchedExample result
    import subprocess
    from anthropic import Anthropic
    
    client = Anthropic()
    
    
    def fix_it(error_text):
        reply = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=300,
            messages=[{"role": "user", "content": "Give one shell command that fixes: " + error_text}],
        )
        command = reply.content[0].text
        return subprocess.run(command, shell=True, capture_output=True)

    Rule

    rules:
      - id: acme.shell-true
        languages: [python]
        severity: WARNING
        message: A command string is run through a shell.
        pattern: subprocess.run(..., shell=True, ...)

    1 finding · 1 of 1 rules ran on 1 file

    • warningacme.shell-trueline 14proof 1 pattern

      A command string is run through a shell.

      return subprocess.run(command, shell=True, capture_output=True)

    Result computed when this page was built. Run an example with the CLI

    Rung 2: a path from source to sink in the fileExample result
    import subprocess
    from anthropic import Anthropic
    
    client = Anthropic()
    
    
    def fix_it(error_text):
        reply = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=300,
            messages=[{"role": "user", "content": "Give one shell command that fixes: " + error_text}],
        )
        command = reply.content[0].text
        return subprocess.run(command, shell=True, capture_output=True)

    Rule

    rules:
      - id: acme.model-output-reaches-shell
        languages: [python]
        severity: ERROR
        message: Text a model produced is run as a shell command. Treat it as untrusted input.
        mode: taint
        pattern-sources:
          - pattern: $CLIENT.messages.create(...)
        pattern-sinks:
          - pattern: subprocess.run($CMD, ...)

    1 finding · 1 of 1 rules ran on 1 file

    • erroracme.model-output-reaches-shellline 14proof 2 path-in-file

      Text a model produced is run as a shell command. Treat it as untrusted input.

      Flow source (line 8) → propagation (line 8) → propagation (line 13)

      return subprocess.run(command, shell=True, capture_output=True)

    Result computed when this page was built. Run an example with the CLI

    The second finding lists its flow: the model call, the assignment that carries the reply, and the call that runs it. Removing shell=True removes the pattern match. The data-flow rule still reports the path if model output reaches the subprocess call.

    A single finding sits on a path. How findings combine into chains, and how a chain is graded, is on paths, chains and chokepoints.

    A pattern match is often still present after the code was fixed. On Keryn’s benchmark of real vulnerability fixes, rules that found the vulnerable version also fired on the fixed version for 85% of the pairs at rung 1, 63% at rung 2 and 9% at rung 4. Treat a rung-1 finding as a lead to check, and a rung-4 or rung-5 finding as evidence.

    The default gate counts high-confidence pattern rules and data-flow findings at rung 3 and above. Rung-2 findings gate only for the finding classes where precision was measured on clean code.

    Findings also carry a rule confidence (P1 high, P2 lower). That rates the rule, not this finding. --gate-confidence P2 widens the gate to the lower tier.