# TypeScript IR and compiler fixtures: Keryn Forge

> Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored TypeScript IR fixtures?

Source: https://keryn.dev/evals/forge-typescript/

Does Forge preserve the labelled flow, no-flow and unknown verdicts on the authored TypeScript IR fixtures?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-forge at `552099d5d1fa`: Committed IR inputs through graph construction, solver and shipped queries, with default analysis limits. |
| Reference | Authored fixture labels . Regression expectations, not an independent upstream analyzer. |
| Corpus | Authored TypeScript flow fixtures |
| Comparison | One verdict per fixture and tier, using the existing end-to-end label harness. Identical means actual verdict equals the authored label; an unknown result never counts as a detected flow or as clean. The harness selects labelled sink/source locations when available and otherwise uses the best available pair. |
| Receipt | [`forge-typescript.json`](https://keryn.dev/evals/forge-typescript.json), sha256 `6ffd1ef85fccbf00` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Fixture/tier verdicts matching their labels | **92.3%** | 48 / 52 | all fixture/tier pairs, including errors and unsupported inputs |
| T1 verdict agreement | **100%** | 26 / 26 | all T1 fixture cases |
| T3 verdict agreement | **84.6%** | 22 / 26 | all T3 fixture cases |
| flow labels preserved | **100%** | 28 / 28 | all fixture/tier pairs labelled flow, including errors and unsupported inputs |
| no_flow labels preserved | **100%** | 12 / 12 | all fixture/tier pairs labelled no_flow, including errors and unsupported inputs |
| unknown labels preserved | **66.7%** | 8 / 12 | all fixture/tier pairs labelled unknown, including errors and unsupported inputs |

Cases by status: 4 differs, 48 identical.

## What this does and does not measure

- This is a curated regression suite, not a measurement of real-world vulnerability detection accuracy. Errors and unavailable inputs remain in the denominator.
- T1 consumes committed native-producer FactBundles. It exercises the current solver and query stack; it does not regenerate source IR during this run. Source-producer and Python input-coverage tests are separate gates.
- Case digests cover the committed input and observed verdict/query result. No source, query text or raw analysis output is published.
- Reproduce from the recorded engine commit using node tools/evals/forge-source.mjs in a clean checkout.
- T3 imports committed TypeScript compiler facts. The compiler is not rerun here; T1 and T3 are reported separately and each fixture/tier pair remains in the denominator, including missing compiler exports.

## Differences and unassessed cases (4)

| Case | Status | Tier | Expected verdict | Keryn verdict |
|---|---|---|---|---|
| `ts/cwe78/ambiguous-guard-after-sink/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-falls-through/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-one-branch/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-sibling-branch/T3` | differs | T3 | unknown | flow |

## All cases (52)

**Show every case**

| Case | Status | Tier | Expected verdict | Keryn verdict |
|---|---|---|---|---|
| `ts/cwe22/negative-sql-escape/T1` | identical | T1 | flow | flow |
| `ts/cwe22/negative-sql-escape/T3` | identical | T3 | flow | flow |
| `ts/cwe22/patched-resolve-prefix/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe22/patched-resolve-prefix/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe22/vulnerable-vulnerable/T1` | identical | T1 | flow | flow |
| `ts/cwe22/vulnerable-vulnerable/T3` | identical | T3 | flow | flow |
| `ts/cwe78/ambiguous-guard-after-sink/T1` | identical | T1 | unknown | unknown |
| `ts/cwe78/ambiguous-guard-after-sink/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-falls-through/T1` | identical | T1 | unknown | unknown |
| `ts/cwe78/ambiguous-guard-falls-through/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-one-branch/T1` | identical | T1 | unknown | unknown |
| `ts/cwe78/ambiguous-guard-one-branch/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-guard-sibling-branch/T1` | identical | T1 | unknown | unknown |
| `ts/cwe78/ambiguous-guard-sibling-branch/T3` | differs | T3 | unknown | flow |
| `ts/cwe78/ambiguous-missing-dependency/T1` | identical | T1 | unknown | unknown |
| `ts/cwe78/ambiguous-missing-dependency/T3` | identical | T3 | unknown | unknown |
| `ts/cwe78/negative-html-escape/T1` | identical | T1 | flow | flow |
| `ts/cwe78/negative-html-escape/T3` | identical | T3 | flow | flow |
| `ts/cwe78/patched-allowlist-execfile/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe78/patched-allowlist-execfile/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe78/patched-allowlist-positive-branch/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe78/patched-allowlist-positive-branch/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe78/vulnerable-async-await/T1` | identical | T1 | flow | flow |
| `ts/cwe78/vulnerable-async-await/T3` | identical | T3 | flow | flow |
| `ts/cwe78/vulnerable-callback/T1` | identical | T1 | flow | flow |
| `ts/cwe78/vulnerable-callback/T3` | identical | T3 | flow | flow |
| `ts/cwe78/vulnerable-cross-file-wrapper/T1` | identical | T1 | flow | flow |
| `ts/cwe78/vulnerable-cross-file-wrapper/T3` | identical | T3 | flow | flow |
| `ts/cwe78/vulnerable-direct/T1` | identical | T1 | flow | flow |
| `ts/cwe78/vulnerable-direct/T3` | identical | T3 | flow | flow |
| `ts/cwe79/negative-sql-escape/T1` | identical | T1 | flow | flow |
| `ts/cwe79/negative-sql-escape/T3` | identical | T3 | flow | flow |
| `ts/cwe79/patched-escape-html/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe79/patched-escape-html/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe79/vulnerable-vulnerable/T1` | identical | T1 | flow | flow |
| `ts/cwe79/vulnerable-vulnerable/T3` | identical | T3 | flow | flow |
| `ts/cwe89/ambiguous-dynamic-import/T1` | identical | T1 | unknown | unknown |
| `ts/cwe89/ambiguous-dynamic-import/T3` | identical | T3 | unknown | unknown |
| `ts/cwe89/negative-url-encode/T1` | identical | T1 | flow | flow |
| `ts/cwe89/negative-url-encode/T3` | identical | T3 | flow | flow |
| `ts/cwe89/patched-parameterized/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe89/patched-parameterized/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe89/vulnerable-cross-file-wrapper/T1` | identical | T1 | flow | flow |
| `ts/cwe89/vulnerable-cross-file-wrapper/T3` | identical | T3 | flow | flow |
| `ts/cwe89/vulnerable-direct/T1` | identical | T1 | flow | flow |
| `ts/cwe89/vulnerable-direct/T3` | identical | T3 | flow | flow |
| `ts/cwe918/negative-html-escape/T1` | identical | T1 | flow | flow |
| `ts/cwe918/negative-html-escape/T3` | identical | T3 | flow | flow |
| `ts/cwe918/patched-host-allowlist/T1` | identical | T1 | no_flow | no_flow |
| `ts/cwe918/patched-host-allowlist/T3` | identical | T3 | no_flow | no_flow |
| `ts/cwe918/vulnerable-vulnerable/T1` | identical | T1 | flow | flow |
| `ts/cwe918/vulnerable-vulnerable/T3` | identical | T3 | flow | flow |
