Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    TypeScript IR and compiler fixtures: Keryn Forge

    View as Markdown

    Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored TypeScript IR fixtures?

    Evaluation detail Description
    Keryn engine keryn-forge at 552099d5d1fa: Committed IR inputs through graph construction, solver and shipped queries, with default analysis limits.
    Reference Authored fixture labels . Regression expectations, not an independent upstream analyzer.
    Corpus Authored TypeScript flow fixtures
    Comparison One verdict per fixture and tier, using the existing end-to-end label harness. Identical means actual verdict equals the authored label; an unknown result never counts as a detected flow or as clean. The harness selects labelled sink/source locations when available and otherwise uses the best available pair.
    Receipt forge-typescript.json, sha256 6ffd1ef85fccbf00
    Measure Value Count Denominator
    Fixture/tier verdicts matching their labels 92.3% 48 / 52 all fixture/tier pairs, including errors and unsupported inputs
    T1 verdict agreement 100% 26 / 26 all T1 fixture cases
    T3 verdict agreement 84.6% 22 / 26 all T3 fixture cases
    flow labels preserved 100% 28 / 28 all fixture/tier pairs labelled flow, including errors and unsupported inputs
    no_flow labels preserved 100% 12 / 12 all fixture/tier pairs labelled no_flow, including errors and unsupported inputs
    unknown labels preserved 66.7% 8 / 12 all fixture/tier pairs labelled unknown, including errors and unsupported inputs

    Cases by status: 4 differs, 48 identical.

    • This is a curated regression suite, not a measurement of real-world vulnerability detection accuracy. Errors and unavailable inputs remain in the denominator.
    • T1 consumes committed native-producer FactBundles. It exercises the current solver and query stack; it does not regenerate source IR during this run. Source-producer and Python input-coverage tests are separate gates.
    • Case digests cover the committed input and observed verdict/query result. No source, query text or raw analysis output is published.
    • Reproduce from the recorded engine commit using node tools/evals/forge-source.mjs in a clean checkout.
    • T3 imports committed TypeScript compiler facts. The compiler is not rerun here; T1 and T3 are reported separately and each fixture/tier pair remains in the denominator, including missing compiler exports.
    Case Status Tier Expected verdict Keryn verdict
    ts/cwe78/ambiguous-guard-after-sink/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-falls-through/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-one-branch/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-sibling-branch/T3 differs T3 unknown flow
    Show every case
    Case Status Tier Expected verdict Keryn verdict
    ts/cwe22/negative-sql-escape/T1 identical T1 flow flow
    ts/cwe22/negative-sql-escape/T3 identical T3 flow flow
    ts/cwe22/patched-resolve-prefix/T1 identical T1 no_flow no_flow
    ts/cwe22/patched-resolve-prefix/T3 identical T3 no_flow no_flow
    ts/cwe22/vulnerable-vulnerable/T1 identical T1 flow flow
    ts/cwe22/vulnerable-vulnerable/T3 identical T3 flow flow
    ts/cwe78/ambiguous-guard-after-sink/T1 identical T1 unknown unknown
    ts/cwe78/ambiguous-guard-after-sink/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-falls-through/T1 identical T1 unknown unknown
    ts/cwe78/ambiguous-guard-falls-through/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-one-branch/T1 identical T1 unknown unknown
    ts/cwe78/ambiguous-guard-one-branch/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-guard-sibling-branch/T1 identical T1 unknown unknown
    ts/cwe78/ambiguous-guard-sibling-branch/T3 differs T3 unknown flow
    ts/cwe78/ambiguous-missing-dependency/T1 identical T1 unknown unknown
    ts/cwe78/ambiguous-missing-dependency/T3 identical T3 unknown unknown
    ts/cwe78/negative-html-escape/T1 identical T1 flow flow
    ts/cwe78/negative-html-escape/T3 identical T3 flow flow
    ts/cwe78/patched-allowlist-execfile/T1 identical T1 no_flow no_flow
    ts/cwe78/patched-allowlist-execfile/T3 identical T3 no_flow no_flow
    ts/cwe78/patched-allowlist-positive-branch/T1 identical T1 no_flow no_flow
    ts/cwe78/patched-allowlist-positive-branch/T3 identical T3 no_flow no_flow
    ts/cwe78/vulnerable-async-await/T1 identical T1 flow flow
    ts/cwe78/vulnerable-async-await/T3 identical T3 flow flow
    ts/cwe78/vulnerable-callback/T1 identical T1 flow flow
    ts/cwe78/vulnerable-callback/T3 identical T3 flow flow
    ts/cwe78/vulnerable-cross-file-wrapper/T1 identical T1 flow flow
    ts/cwe78/vulnerable-cross-file-wrapper/T3 identical T3 flow flow
    ts/cwe78/vulnerable-direct/T1 identical T1 flow flow
    ts/cwe78/vulnerable-direct/T3 identical T3 flow flow
    ts/cwe79/negative-sql-escape/T1 identical T1 flow flow
    ts/cwe79/negative-sql-escape/T3 identical T3 flow flow
    ts/cwe79/patched-escape-html/T1 identical T1 no_flow no_flow
    ts/cwe79/patched-escape-html/T3 identical T3 no_flow no_flow
    ts/cwe79/vulnerable-vulnerable/T1 identical T1 flow flow
    ts/cwe79/vulnerable-vulnerable/T3 identical T3 flow flow
    ts/cwe89/ambiguous-dynamic-import/T1 identical T1 unknown unknown
    ts/cwe89/ambiguous-dynamic-import/T3 identical T3 unknown unknown
    ts/cwe89/negative-url-encode/T1 identical T1 flow flow
    ts/cwe89/negative-url-encode/T3 identical T3 flow flow
    ts/cwe89/patched-parameterized/T1 identical T1 no_flow no_flow
    ts/cwe89/patched-parameterized/T3 identical T3 no_flow no_flow
    ts/cwe89/vulnerable-cross-file-wrapper/T1 identical T1 flow flow
    ts/cwe89/vulnerable-cross-file-wrapper/T3 identical T3 flow flow
    ts/cwe89/vulnerable-direct/T1 identical T1 flow flow
    ts/cwe89/vulnerable-direct/T3 identical T3 flow flow
    ts/cwe918/negative-html-escape/T1 identical T1 flow flow
    ts/cwe918/negative-html-escape/T3 identical T3 flow flow
    ts/cwe918/patched-host-allowlist/T1 identical T1 no_flow no_flow
    ts/cwe918/patched-host-allowlist/T3 identical T3 no_flow no_flow
    ts/cwe918/vulnerable-vulnerable/T1 identical T1 flow flow
    ts/cwe918/vulnerable-vulnerable/T3 identical T3 flow flow