Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Python IR fixtures: Keryn Forge

    View as Markdown

    Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored Python IR fixtures?

    Evaluation detail Description
    Keryn engine keryn-forge at 552099d5d1fa: Committed IR inputs through graph construction, solver and shipped queries, with default analysis limits.
    Reference Authored fixture labels . Regression expectations, not an independent upstream analyzer.
    Corpus Authored Python flow fixtures
    Comparison One verdict per fixture and tier, using the existing end-to-end label harness. Identical means actual verdict equals the authored label; an unknown result never counts as a detected flow or as clean. The harness selects labelled sink/source locations when available and otherwise uses the best available pair.
    Receipt forge-python.json, sha256 0c238645fc030089
    Measure Value Count Denominator
    Fixture/tier verdicts matching their labels 100% 37 / 37 all fixture/tier pairs, including errors and unsupported inputs
    T1 verdict agreement 100% 37 / 37 all T1 fixture cases
    flow labels preserved 100% 19 / 19 all fixture/tier pairs labelled flow, including errors and unsupported inputs
    no_flow labels preserved 100% 12 / 12 all fixture/tier pairs labelled no_flow, including errors and unsupported inputs
    unknown labels preserved 100% 6 / 6 all fixture/tier pairs labelled unknown, including errors and unsupported inputs

    Cases by status: 37 identical.

    • This is a curated regression suite, not a measurement of real-world vulnerability detection accuracy. Errors and unavailable inputs remain in the denominator.
    • T1 consumes committed native-producer FactBundles. It exercises the current solver and query stack; it does not regenerate source IR during this run. Source-producer and Python input-coverage tests are separate gates.
    • Case digests cover the committed input and observed verdict/query result. No source, query text or raw analysis output is published.
    • Reproduce from the recorded engine commit using node tools/evals/forge-source.mjs in a clean checkout.
    • No Pysa installation or Pysa comparison was run. Python syntax variants, notebooks and partial-input reporting are covered by separate producer/coverage tests, not by this receipt.
    Show every case
    Case Status Tier Expected verdict Keryn verdict
    python/cwe22/patched-flask-secure-filename/T1 identical T1 no_flow no_flow
    python/cwe22/vulnerable-flask-open/T1 identical T1 flow flow
    python/cwe78/ambiguous-guard-after-sink/T1 identical T1 unknown unknown
    python/cwe78/ambiguous-guard-one-branch/T1 identical T1 unknown unknown
    python/cwe78/ambiguous-guard-sibling-branch/T1 identical T1 unknown unknown
    python/cwe78/ambiguous-override-fanout-over-cap/T1 identical T1 unknown unknown
    python/cwe78/ambiguous-receiver/T1 identical T1 unknown unknown
    python/cwe78/ambiguous-third-party-normalizer/T1 identical T1 unknown unknown
    python/cwe78/negative-flask-html-escape-before-os-system/T1 identical T1 flow flow
    python/cwe78/patched-allowlist-continue-in-loop/T1 identical T1 no_flow no_flow
    python/cwe78/patched-allowlist-in-try-body/T1 identical T1 no_flow no_flow
    python/cwe78/patched-allowlist-positive-branch/T1 identical T1 no_flow no_flow
    python/cwe78/patched-argv-no-shell/T1 identical T1 no_flow no_flow
    python/cwe78/patched-chain-allowlist-map/T1 identical T1 no_flow no_flow
    python/cwe78/patched-constant-argument/T1 identical T1 no_flow no_flow
    python/cwe78/patched-flask-allowlist/T1 identical T1 no_flow no_flow
    python/cwe78/patched-middle-drops-arg/T1 identical T1 no_flow no_flow
    python/cwe78/patched-not-imported/T1 identical T1 no_flow no_flow
    python/cwe78/patched-sibling-override-unrelated/T1 identical T1 no_flow no_flow
    python/cwe78/vulnerable-check-does-not-exit/T1 identical T1 flow flow
    python/cwe78/vulnerable-constructor-method/T1 identical T1 flow flow
    python/cwe78/vulnerable-direct/T1 identical T1 flow flow
    python/cwe78/vulnerable-fastapi-subprocess/T1 identical T1 flow flow
    python/cwe78/vulnerable-flask-os-system/T1 identical T1 flow flow
    python/cwe78/vulnerable-flask-shlex-quote-option-injection/T1 identical T1 flow flow
    python/cwe78/vulnerable-flask-subprocess/T1 identical T1 flow flow
    python/cwe78/vulnerable-inherited-override/T1 identical T1 flow flow
    python/cwe78/vulnerable-module-alias/T1 identical T1 flow flow
    python/cwe78/vulnerable-override-fanout-at-cap/T1 identical T1 flow flow
    python/cwe78/vulnerable-package-reexport-two-levels/T1 identical T1 flow flow
    python/cwe78/vulnerable-package-reexport/T1 identical T1 flow flow
    python/cwe78/vulnerable-param-three-hops/T1 identical T1 flow flow
    python/cwe78/vulnerable-param-two-hops/T1 identical T1 flow flow
    python/cwe78/vulnerable-return-chain/T1 identical T1 flow flow
    python/cwe78/vulnerable-self-dispatch-template-method/T1 identical T1 flow flow
    python/cwe89/patched-flask-parameterised/T1 identical T1 no_flow no_flow
    python/cwe89/vulnerable-flask-cursor-execute/T1 identical T1 flow flow