# Evals

> Keryn measured against upstream tools, labelled benchmarks and authored regression fixtures, with every number traceable to a receipt.

Source: https://keryn.dev/evals/

These evals compare Keryn with **upstream tools**, **labelled benchmarks** or **authored regression fixtures**. Each suite names its reference and what agreement means. The result is a **receipt**: a JSON file that names the engine commit, the reference version, the corpus commit and a digest of every input and output. The same commits give the same receipt byte for byte, so a change in a number is a change in behaviour. These pages are generated from the receipts and change only when a receipt does.

Every percentage gives its numerator and denominator. Incomplete cases, timeouts and errors stay in the case denominator. An unavailable agreement score is not 0% detection. Fixture agreement measures regression coverage; it does not estimate real-world detection accuracy. Each suite states which inputs it covers and which steps were actually run.

## Source analysis and Juliet

- [NIST Juliet Java and C/C++ (full corpora, sampled): Keryn Forge](https://keryn.dev/evals/forge-juliet/): Cases whose bad path Forge found **2,410 / 3,062** (78.7%); Cases with no good path flagged **3,028 / 3,062** (98.9%); CWE groups with a Forge rule **23 / 232** (9.9%).
- [Python IR fixtures: Keryn Forge](https://keryn.dev/evals/forge-python/): T1 verdict agreement **37 / 37** (100%).
- [TypeScript IR and compiler fixtures: Keryn Forge](https://keryn.dev/evals/forge-typescript/): T1 verdict agreement **26 / 26** (100%); T3 verdict agreement **22 / 26** (84.6%).

Python and TypeScript fixture suites use committed IR inputs. Their pages distinguish source-producer inputs (T1) from compiler-export inputs (T3), and list unavailable inputs and mismatches. Juliet measures sampled NIST cases; supported-CWE coverage is reported separately from detection within supported cases.

## Default-scan location comparisons

These suites compare observed locations across independently authored checks. Matching locations do not establish equivalent rule semantics or exploitability. A failed or incomplete scan cannot earn agreement. Older suites retain their original engine revisions; publishing this page does not rerun them.

| Suite | Engine revision | Comparison | Whole-check agreement | Incomplete cases | Error cases | Timeouts | No reference |
|---|---|---|---|---|---|---|---|
| [Python: how much of what Bandit 1.9.4 finds does Keryn's default scan flag?](https://keryn.dev/evals/bandit/) | `36d56edabe89` | Not recorded | 97.3% (73 / 75) | 0 | 0 | 0 | 0 |
| [Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn's default scan flag?](https://keryn.dev/evals/brakeman/) | `2e13028478f1` | Complete | 59.3% (51 / 86) | 0 | 0 | 0 | 8 |
| [JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn's default scan flag?](https://keryn.dev/evals/eslint-security/) | `1474fd84760a` | Complete | 80.0% (12 / 15) | 0 | 0 | 0 | 0 |

Incomplete cases are not execution failures: a successful scan can leave rule applicability or analysis coverage unresolved. Without a rule-to-rule mapping, incomplete coverage prevents whole-check certification across the fired reference checks. Partial location observations remain visible below.

See [capabilities and availability](https://keryn.dev/docs/concepts/capabilities/) for the current browser build and host-specific limits. Healthy deployment checks are separate from these detection results.

| Eval | Keryn engine | Reference | Case agreement | Reference overlap | Engine overlap |
|---|---|---|---|---|---|
| [Python: how much of what Bandit 1.9.4 finds does Keryn's default scan flag?](https://keryn.dev/evals/bandit/) | Keryn default scan | Bandit 1.9.4 | 97.3% (73 / 75) | 99.5% (591 / 594) | 88.8% (561 / 632) |
| [Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn's default scan flag?](https://keryn.dev/evals/brakeman/) | Keryn default scan | Brakeman 8.1.0 | 59.3% (51 / 86) | 91.6% (624 / 681) | 78.3% (545 / 696) |
| [CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI](https://keryn.dev/evals/codeql-actions/) | keryn-ql | CodeQL CLI 2.26.3 | 100% (42 / 42) | 100% (509 / 509) | 100% (509 / 509) |
| [DefectDojo independent finding-field comparison](https://keryn.dev/evals/defectdojo-fields/) | Keryn native report importers | Unmodified DefectDojo Django parsers 5.2.16 | 100% (2 / 2) |  |  |
| [DefectDojo report importer fixture coverage](https://keryn.dev/evals/defectdojo-passive-importers/) | Keryn native report importers | DefectDojo pinned parser inventory and fixture expectations  | 0% (0 / 3) |  |  |
| [JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn's default scan flag?](https://keryn.dev/evals/eslint-security/) | Keryn default scan | eslint-plugin-security 4.2.1 on ESLint 9.39.1 | 80.0% (12 / 15) | 91.3% (105 / 115) | 74.8% (104 / 139) |
| [Real CVE fixes as NuGet artifacts (dotnet): Keryn Forge](https://keryn.dev/evals/forge-cve-dotnet/) | keryn-forge | GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history  | 0% (0 / 30) |  |  |
| [Real CVE fixes as Maven Central artifacts (jvm): Keryn Forge](https://keryn.dev/evals/forge-cve-jvm/) | keryn-forge | GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history  | 0% (0 / 50) |  |  |
| [NIST Juliet starter cases (Java, C/C++): Keryn Forge](https://keryn.dev/evals/forge-juliet-smoke/) | keryn-forge | NIST Juliet per-case SARIF manifests (expected flaw locations)  | 57.1% (4 / 7) |  |  |
| [NIST Juliet Java and C/C++ (full corpora, sampled): Keryn Forge](https://keryn.dev/evals/forge-juliet/) | keryn-forge | NIST Juliet per-case SARIF manifests (expected flaw locations)  | 2.2% (5 / 232) |  |  |
| [OWASP Benchmark v1.2 (Java): Keryn Forge vs the expected results](https://keryn.dev/evals/forge-owasp-benchmark/) | keryn-forge | OWASP Benchmark expected results v1.2 (ground-truth labels, not a tool)  | 95.1% (2,607 / 2,740) |  |  |
| [Python IR fixtures: Keryn Forge](https://keryn.dev/evals/forge-python/) | keryn-forge | Authored fixture labels  | 100% (37 / 37) |  |  |
| [TypeScript IR and compiler fixtures: Keryn Forge](https://keryn.dev/evals/forge-typescript/) | keryn-forge | Authored fixture labels  | 92.3% (48 / 52) |  |  |
| [OWASP WebGoat lesson endpoints (Spring Boot, JVM bytecode): Keryn Forge](https://keryn.dev/evals/forge-webgoat/) | keryn-forge | Hand labels from the WebGoat source at the pinned commit (suites/forge-webgoat/reference.json)  | 83.7% (41 / 49) |  |  |
| [garak detectors: keryn-eval vs garak 0.17.2.pre1](https://keryn.dev/evals/garak/) | keryn-eval detectors | garak detectors 0.17.2.pre1 | 100% (34 / 34) |  |  |
| [LLM Guard InvisibleText Unicode filtering](https://keryn.dev/evals/llm-guard-invisible-text/) | Keryn prompt-injection-detector scan_invisible_text | LLM Guard InvisibleText.scan  | 100% (17 / 17) |  |  |
| [LLM Guard passive scanner parity](https://keryn.dev/evals/llm-guard-passive/) | Keryn native passive text policy scanners | LLM Guard read-only scanner methods  | 37.5% (3 / 8) |  |  |
| [Model-file inspection (pickle, PyTorch, joblib, safetensors, GGUF, ONNX)](https://keryn.dev/evals/model-files/) | keryn artifact analyze (model readers) | picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed) | 73.3% (33 / 45) |  |  |
| [Nuclei query mutation: keryn-nuclei differential profile vs nuclei v3.11.1](https://keryn.dev/evals/nuclei-differential/) | keryn-nuclei differential | nuclei v3.11.1 | 100% (24 / 24) |  |  |
| [Nuclei HTTP templates: keryn-nuclei vs nuclei v3.11.1](https://keryn.dev/evals/nuclei/) | keryn-nuclei | nuclei v3.11.1 | 96.1% (49 / 51) | 98.0% (48 / 49) | 98.0% (48 / 49) |
| [Promptfoo offline assertion parity](https://keryn.dev/evals/promptfoo-structured/) | Keryn Eval native config importer and replay engine | Promptfoo assertion API 0.125.1 | 53.8% (7 / 13) |  |  |
| [promptfoo configs: keryn-eval vs promptfoo 0.125.1](https://keryn.dev/evals/promptfoo/) | keryn-eval | promptfoo 0.125.1 | 69.7% (302 / 433) |  |  |
| [Dependency reachability from imports (JavaScript/TypeScript, Python, Java)](https://keryn.dev/evals/sca-reachability/) | keryn import reachability (ts/packages/sca/src/import-reachability.ts) | independently authored truth labels authored-reach-1 | 93.3% (28 / 30) |  |  |
| [Semgrep rules: Keryn vs Semgrep CE 1.157.0](https://keryn.dev/evals/semgrep-ce/) | Keryn pattern rules | Semgrep CE 1.157.0 | 100% (164 / 164) | 100% (13,475 / 13,475) | 100% (13,475 / 13,475) |
| [ZAP MCP detections: keryn-passive vs ZAP 2.17.0 (community script 100045, mcp add-on importer)](https://keryn.dev/evals/zap-mcp/) | keryn-passive | ZAP's MCP detections: community-scripts OpenModelContextProtocolServer.js (rule 100045) on GraalJS inside the scripts add-on's ActiveScriptScanRule, and the zap-extensions mcp add-on's McpImporter and VariantMcpJsonRpc, on ZAP core 2.17.0 | 92.3% (96 / 104) |  |  |
| [ZAP passive scan rules: keryn-passive vs ZAP 2.17.0](https://keryn.dev/evals/zap-passive/) | keryn-passive | OWASP ZAP passive scan rules (zap-extensions pscanrules, pscanrulesBeta, pscanrulesAlpha) on ZAP core 2.17.0 | 100% (63 / 63) |  |  |

**Reference overlap** is the share of reference results also reported by Keryn; **engine overlap** is the share of Keryn results also reported by the reference. These are agreement measures under each suite’s comparator, not confirmed vulnerabilities or real-world precision. Values marked **partial observations** retain evidence from incomplete scans and cannot establish complete coverage.

Receipts as of Keryn commit `dd5df0da79c5`. Each page links its receipt for download.
