Evals
These evals compare Keryn with upstream tools, labelled benchmarks or authored regression fixtures. Each suite names its reference and what agreement means. The result is a receipt: a JSON file that names the engine commit, the reference version, the corpus commit and a digest of every input and output. The same commits give the same receipt byte for byte, so a change in a number is a change in behaviour. These pages are generated from the receipts and change only when a receipt does.
Every percentage gives its numerator and denominator. Incomplete cases, timeouts and errors stay in the case denominator. An unavailable agreement score is not 0% detection. Fixture agreement measures regression coverage; it does not estimate real-world detection accuracy. Each suite states which inputs it covers and which steps were actually run.
Source analysis and Juliet
Section titled “Source analysis and Juliet”- NIST Juliet Java and C/C++ (full corpora, sampled): Keryn Forge: Cases whose bad path Forge found 2,410 / 3,062 (78.7%); Cases with no good path flagged 3,028 / 3,062 (98.9%); CWE groups with a Forge rule 23 / 232 (9.9%).
- Python IR fixtures: Keryn Forge: T1 verdict agreement 37 / 37 (100%).
- TypeScript IR and compiler fixtures: Keryn Forge: T1 verdict agreement 26 / 26 (100%); T3 verdict agreement 22 / 26 (84.6%).
Python and TypeScript fixture suites use committed IR inputs. Their pages distinguish source-producer inputs (T1) from compiler-export inputs (T3), and list unavailable inputs and mismatches. Juliet measures sampled NIST cases; supported-CWE coverage is reported separately from detection within supported cases.
Default-scan location comparisons
Section titled “Default-scan location comparisons”These suites compare observed locations across independently authored checks. Matching locations do not establish equivalent rule semantics or exploitability. A failed or incomplete scan cannot earn agreement. Older suites retain their original engine revisions; publishing this page does not rerun them.
| Suite | Engine revision | Comparison | Whole-check agreement | Incomplete cases | Error cases | Timeouts | No reference |
|---|---|---|---|---|---|---|---|
| Python: how much of what Bandit 1.9.4 finds does Keryn’s default scan flag? | 36d56edabe89 |
Not recorded | 97.3% (73 / 75) | 0 | 0 | 0 | 0 |
| Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn’s default scan flag? | 2e13028478f1 |
Complete | 59.3% (51 / 86) | 0 | 0 | 0 | 8 |
| JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn’s default scan flag? | 1474fd84760a |
Complete | 80.0% (12 / 15) | 0 | 0 | 0 | 0 |
Incomplete cases are not execution failures: a successful scan can leave rule applicability or analysis coverage unresolved. Without a rule-to-rule mapping, incomplete coverage prevents whole-check certification across the fired reference checks. Partial location observations remain visible below.
See capabilities and availability for the current browser build and host-specific limits. Healthy deployment checks are separate from these detection results.
| Eval | Keryn engine | Reference | Case agreement | Reference overlap | Engine overlap |
|---|---|---|---|---|---|
| Python: how much of what Bandit 1.9.4 finds does Keryn’s default scan flag? | Keryn default scan | Bandit 1.9.4 | 97.3% (73 / 75) | 99.5% (591 / 594) | 88.8% (561 / 632) |
| Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn’s default scan flag? | Keryn default scan | Brakeman 8.1.0 | 59.3% (51 / 86) | 91.6% (624 / 681) | 78.3% (545 / 696) |
| CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI | keryn-ql | CodeQL CLI 2.26.3 | 100% (42 / 42) | 100% (509 / 509) | 100% (509 / 509) |
| DefectDojo independent finding-field comparison | Keryn native report importers | Unmodified DefectDojo Django parsers 5.2.16 | 100% (2 / 2) | ||
| DefectDojo report importer fixture coverage | Keryn native report importers | DefectDojo pinned parser inventory and fixture expectations | 0% (0 / 3) | ||
| JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn’s default scan flag? | Keryn default scan | eslint-plugin-security 4.2.1 on ESLint 9.39.1 | 80.0% (12 / 15) | 91.3% (105 / 115) | 74.8% (104 / 139) |
| Real CVE fixes as NuGet artifacts (dotnet): Keryn Forge | keryn-forge | GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history | 0% (0 / 30) | ||
| Real CVE fixes as Maven Central artifacts (jvm): Keryn Forge | keryn-forge | GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history | 0% (0 / 50) | ||
| NIST Juliet starter cases (Java, C/C++): Keryn Forge | keryn-forge | NIST Juliet per-case SARIF manifests (expected flaw locations) | 57.1% (4 / 7) | ||
| NIST Juliet Java and C/C++ (full corpora, sampled): Keryn Forge | keryn-forge | NIST Juliet per-case SARIF manifests (expected flaw locations) | 2.2% (5 / 232) | ||
| OWASP Benchmark v1.2 (Java): Keryn Forge vs the expected results | keryn-forge | OWASP Benchmark expected results v1.2 (ground-truth labels, not a tool) | 95.1% (2,607 / 2,740) | ||
| Python IR fixtures: Keryn Forge | keryn-forge | Authored fixture labels | 100% (37 / 37) | ||
| TypeScript IR and compiler fixtures: Keryn Forge | keryn-forge | Authored fixture labels | 92.3% (48 / 52) | ||
| OWASP WebGoat lesson endpoints (Spring Boot, JVM bytecode): Keryn Forge | keryn-forge | Hand labels from the WebGoat source at the pinned commit (suites/forge-webgoat/reference.json) | 83.7% (41 / 49) | ||
| garak detectors: keryn-eval vs garak 0.17.2.pre1 | keryn-eval detectors | garak detectors 0.17.2.pre1 | 100% (34 / 34) | ||
| LLM Guard InvisibleText Unicode filtering | Keryn prompt-injection-detector scan_invisible_text | LLM Guard InvisibleText.scan | 100% (17 / 17) | ||
| LLM Guard passive scanner parity | Keryn native passive text policy scanners | LLM Guard read-only scanner methods | 37.5% (3 / 8) | ||
| Model-file inspection (pickle, PyTorch, joblib, safetensors, GGUF, ONNX) | keryn artifact analyze (model readers) | picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed) | 73.3% (33 / 45) | ||
| Nuclei query mutation: keryn-nuclei differential profile vs nuclei v3.11.1 | keryn-nuclei differential | nuclei v3.11.1 | 100% (24 / 24) | ||
| Nuclei HTTP templates: keryn-nuclei vs nuclei v3.11.1 | keryn-nuclei | nuclei v3.11.1 | 96.1% (49 / 51) | 98.0% (48 / 49) | 98.0% (48 / 49) |
| Promptfoo offline assertion parity | Keryn Eval native config importer and replay engine | Promptfoo assertion API 0.125.1 | 53.8% (7 / 13) | ||
| promptfoo configs: keryn-eval vs promptfoo 0.125.1 | keryn-eval | promptfoo 0.125.1 | 69.7% (302 / 433) | ||
| Dependency reachability from imports (JavaScript/TypeScript, Python, Java) | keryn import reachability (ts/packages/sca/src/import-reachability.ts) | independently authored truth labels authored-reach-1 | 93.3% (28 / 30) | ||
| Semgrep rules: Keryn vs Semgrep CE 1.157.0 | Keryn pattern rules | Semgrep CE 1.157.0 | 100% (164 / 164) | 100% (13,475 / 13,475) | 100% (13,475 / 13,475) |
| ZAP MCP detections: keryn-passive vs ZAP 2.17.0 (community script 100045, mcp add-on importer) | keryn-passive | ZAP’s MCP detections: community-scripts OpenModelContextProtocolServer.js (rule 100045) on GraalJS inside the scripts add-on’s ActiveScriptScanRule, and the zap-extensions mcp add-on’s McpImporter and VariantMcpJsonRpc, on ZAP core 2.17.0 | 92.3% (96 / 104) | ||
| ZAP passive scan rules: keryn-passive vs ZAP 2.17.0 | keryn-passive | OWASP ZAP passive scan rules (zap-extensions pscanrules, pscanrulesBeta, pscanrulesAlpha) on ZAP core 2.17.0 | 100% (63 / 63) |
Reference overlap is the share of reference results also reported by Keryn; engine overlap is the share of Keryn results also reported by the reference. These are agreement measures under each suite’s comparator, not confirmed vulnerabilities or real-world precision. Values marked partial observations retain evidence from incomplete scans and cannot establish complete coverage.
Receipts as of Keryn commit dd5df0da79c5. Each page links its receipt for download.