The proof ladder
Every finding carries a proof level. It says how much evidence stands behind the finding, not how severe it is. Severity (error, warning or info) is the rule’s rating of the impact if the finding is real. The two are shown side by side and neither replaces the other.
| Level | Meaning |
|---|---|
| proof 0inventory | An AI component, capability or posture fact. Never gates as a vulnerability. |
| proof 1pattern | A code pattern matched. |
| proof 2path-in-file | A path from a source (user input, model output) to a sink inside one file. |
| proof 3path-cross-file | A path across files (Python, TypeScript, JavaScript, Go, Java, C#, C/C++). How far it reaches depends on the language: see what Keryn does not do. |
| proof 4known-fix | The code matches a block that a published advisory’s fix removed; cites the CVE or GHSA. |
| proof 5reproduced | Verify reproduced it in an isolated container; cites the evidence hash. |
The same code at two rungs
Section titled “The same code at two rungs”This function asks a model for a shell command and runs it. The first rule only looks for shell=True. The second follows the model’s reply to the call that runs it. Compare the proof levels in the two recorded results.
import subprocess
from anthropic import Anthropic
client = Anthropic()
def fix_it(error_text):
reply = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=300,
messages=[{"role": "user", "content": "Give one shell command that fixes: " + error_text}],
)
command = reply.content[0].text
return subprocess.run(command, shell=True, capture_output=True)Rule
rules:
- id: acme.shell-true
languages: [python]
severity: WARNING
message: A command string is run through a shell.
pattern: subprocess.run(..., shell=True, ...)1 finding · 1 of 1 rules ran on 1 file
- warning
acme.shell-trueline 14proof 1 patternA command string is run through a shell.
return subprocess.run(command, shell=True, capture_output=True)
Result computed when this page was built. Run an example with the CLI
import subprocess
from anthropic import Anthropic
client = Anthropic()
def fix_it(error_text):
reply = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=300,
messages=[{"role": "user", "content": "Give one shell command that fixes: " + error_text}],
)
command = reply.content[0].text
return subprocess.run(command, shell=True, capture_output=True)Rule
rules:
- id: acme.model-output-reaches-shell
languages: [python]
severity: ERROR
message: Text a model produced is run as a shell command. Treat it as untrusted input.
mode: taint
pattern-sources:
- pattern: $CLIENT.messages.create(...)
pattern-sinks:
- pattern: subprocess.run($CMD, ...)1 finding · 1 of 1 rules ran on 1 file
- error
acme.model-output-reaches-shellline 14proof 2 path-in-fileText a model produced is run as a shell command. Treat it as untrusted input.
Flow source (line 8) → propagation (line 8) → propagation (line 13)
return subprocess.run(command, shell=True, capture_output=True)
Result computed when this page was built. Run an example with the CLI
The second finding lists its flow: the model call, the assignment that carries the reply, and the call that runs it. Removing shell=True removes the pattern match. The data-flow rule still reports the path if model output reaches the subprocess call.
A single finding sits on a path. How findings combine into chains, and how a chain is graded, is on paths, chains and chokepoints.
Why the ladder exists
Section titled “Why the ladder exists”A pattern match is often still present after the code was fixed. On Keryn’s benchmark of real vulnerability fixes, rules that found the vulnerable version also fired on the fixed version for 85% of the pairs at rung 1, 63% at rung 2 and 9% at rung 4. Treat a rung-1 finding as a lead to check, and a rung-4 or rung-5 finding as evidence.
The default gate counts high-confidence pattern rules and data-flow findings at rung 3 and above. Rung-2 findings gate only for the finding classes where precision was measured on clean code.
Findings also carry a rule confidence (P1 high, P2 lower). That rates the rule, not this finding. --gate-confidence P2 widens the gate to the lower tier.