# Python IR fixtures: Keryn Forge

> Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored Python IR fixtures?

Source: https://keryn.dev/evals/forge-python/

Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored Python IR fixtures?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-forge at `552099d5d1fa`: Committed IR inputs through graph construction, solver and shipped queries, with default analysis limits. |
| Reference | Authored fixture labels . Regression expectations, not an independent upstream analyzer. |
| Corpus | Authored Python flow fixtures |
| Comparison | One verdict per fixture and tier, using the existing end-to-end label harness. Identical means actual verdict equals the authored label; an unknown result never counts as a detected flow or as clean. The harness selects labelled sink/source locations when available and otherwise uses the best available pair. |
| Receipt | [`forge-python.json`](https://keryn.dev/evals/forge-python.json), sha256 `0c238645fc030089` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Fixture/tier verdicts matching their labels | **100%** | 37 / 37 | all fixture/tier pairs, including errors and unsupported inputs |
| T1 verdict agreement | **100%** | 37 / 37 | all T1 fixture cases |
| flow labels preserved | **100%** | 19 / 19 | all fixture/tier pairs labelled flow, including errors and unsupported inputs |
| no_flow labels preserved | **100%** | 12 / 12 | all fixture/tier pairs labelled no_flow, including errors and unsupported inputs |
| unknown labels preserved | **100%** | 6 / 6 | all fixture/tier pairs labelled unknown, including errors and unsupported inputs |

Cases by status: 37 identical.

## What this does and does not measure

- This is a curated regression suite, not a measurement of real-world vulnerability detection accuracy. Errors and unavailable inputs remain in the denominator.
- T1 consumes committed native-producer FactBundles. It exercises the current solver and query stack; it does not regenerate source IR during this run. Source-producer and Python input-coverage tests are separate gates.
- Case digests cover the committed input and observed verdict/query result. No source, query text or raw analysis output is published.
- Reproduce from the recorded engine commit using node tools/evals/forge-source.mjs in a clean checkout.
- No Pysa installation or Pysa comparison was run. Python syntax variants, notebooks and partial-input reporting are covered by separate producer/coverage tests, not by this receipt.

## All cases (37)

**Show every case**

| Case | Status | Tier | Expected verdict | Keryn verdict |
|---|---|---|---|---|
| `python/cwe22/patched-flask-secure-filename/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe22/vulnerable-flask-open/T1` | identical | T1 | flow | flow |
| `python/cwe78/ambiguous-guard-after-sink/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/ambiguous-guard-one-branch/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/ambiguous-guard-sibling-branch/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/ambiguous-override-fanout-over-cap/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/ambiguous-receiver/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/ambiguous-third-party-normalizer/T1` | identical | T1 | unknown | unknown |
| `python/cwe78/negative-flask-html-escape-before-os-system/T1` | identical | T1 | flow | flow |
| `python/cwe78/patched-allowlist-continue-in-loop/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-allowlist-in-try-body/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-allowlist-positive-branch/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-argv-no-shell/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-chain-allowlist-map/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-constant-argument/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-flask-allowlist/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-middle-drops-arg/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-not-imported/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/patched-sibling-override-unrelated/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe78/vulnerable-check-does-not-exit/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-constructor-method/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-direct/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-fastapi-subprocess/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-flask-os-system/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-flask-shlex-quote-option-injection/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-flask-subprocess/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-inherited-override/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-module-alias/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-override-fanout-at-cap/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-package-reexport-two-levels/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-package-reexport/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-param-three-hops/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-param-two-hops/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-return-chain/T1` | identical | T1 | flow | flow |
| `python/cwe78/vulnerable-self-dispatch-template-method/T1` | identical | T1 | flow | flow |
| `python/cwe89/patched-flask-parameterised/T1` | identical | T1 | no_flow | no_flow |
| `python/cwe89/vulnerable-flask-cursor-execute/T1` | identical | T1 | flow | flow |
