Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Evals

    View as Markdown

    These evals compare Keryn with upstream tools, labelled benchmarks or authored regression fixtures. Each suite names its reference and what agreement means. The result is a receipt: a JSON file that names the engine commit, the reference version, the corpus commit and a digest of every input and output. The same commits give the same receipt byte for byte, so a change in a number is a change in behaviour. These pages are generated from the receipts and change only when a receipt does.

    Every percentage gives its numerator and denominator. Incomplete cases, timeouts and errors stay in the case denominator. An unavailable agreement score is not 0% detection. Fixture agreement measures regression coverage; it does not estimate real-world detection accuracy. Each suite states which inputs it covers and which steps were actually run.

    Python and TypeScript fixture suites use committed IR inputs. Their pages distinguish source-producer inputs (T1) from compiler-export inputs (T3), and list unavailable inputs and mismatches. Juliet measures sampled NIST cases; supported-CWE coverage is reported separately from detection within supported cases.

    These suites compare observed locations across independently authored checks. Matching locations do not establish equivalent rule semantics or exploitability. A failed or incomplete scan cannot earn agreement. Older suites retain their original engine revisions; publishing this page does not rerun them.

    Suite Engine revision Comparison Whole-check agreement Incomplete cases Error cases Timeouts No reference
    Python: how much of what Bandit 1.9.4 finds does Keryn’s default scan flag? 36d56edabe89 Not recorded 97.3% (73 / 75) 0 0 0 0
    Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn’s default scan flag? 2e13028478f1 Complete 59.3% (51 / 86) 0 0 0 8
    JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn’s default scan flag? 1474fd84760a Complete 80.0% (12 / 15) 0 0 0 0

    Incomplete cases are not execution failures: a successful scan can leave rule applicability or analysis coverage unresolved. Without a rule-to-rule mapping, incomplete coverage prevents whole-check certification across the fired reference checks. Partial location observations remain visible below.

    See capabilities and availability for the current browser build and host-specific limits. Healthy deployment checks are separate from these detection results.

    Eval Keryn engine Reference Case agreement Reference overlap Engine overlap
    Python: how much of what Bandit 1.9.4 finds does Keryn’s default scan flag? Keryn default scan Bandit 1.9.4 97.3% (73 / 75) 99.5% (591 / 594) 88.8% (561 / 632)
    Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn’s default scan flag? Keryn default scan Brakeman 8.1.0 59.3% (51 / 86) 91.6% (624 / 681) 78.3% (545 / 696)
    CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI keryn-ql CodeQL CLI 2.26.3 100% (42 / 42) 100% (509 / 509) 100% (509 / 509)
    DefectDojo independent finding-field comparison Keryn native report importers Unmodified DefectDojo Django parsers 5.2.16 100% (2 / 2)
    DefectDojo report importer fixture coverage Keryn native report importers DefectDojo pinned parser inventory and fixture expectations 0% (0 / 3)
    JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn’s default scan flag? Keryn default scan eslint-plugin-security 4.2.1 on ESLint 9.39.1 80.0% (12 / 15) 91.3% (105 / 115) 74.8% (104 / 139)
    Real CVE fixes as NuGet artifacts (dotnet): Keryn Forge keryn-forge GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history 0% (0 / 30)
    Real CVE fixes as Maven Central artifacts (jvm): Keryn Forge keryn-forge GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history 0% (0 / 50)
    NIST Juliet starter cases (Java, C/C++): Keryn Forge keryn-forge NIST Juliet per-case SARIF manifests (expected flaw locations) 57.1% (4 / 7)
    NIST Juliet Java and C/C++ (full corpora, sampled): Keryn Forge keryn-forge NIST Juliet per-case SARIF manifests (expected flaw locations) 2.2% (5 / 232)
    OWASP Benchmark v1.2 (Java): Keryn Forge vs the expected results keryn-forge OWASP Benchmark expected results v1.2 (ground-truth labels, not a tool) 95.1% (2,607 / 2,740)
    Python IR fixtures: Keryn Forge keryn-forge Authored fixture labels 100% (37 / 37)
    TypeScript IR and compiler fixtures: Keryn Forge keryn-forge Authored fixture labels 92.3% (48 / 52)
    OWASP WebGoat lesson endpoints (Spring Boot, JVM bytecode): Keryn Forge keryn-forge Hand labels from the WebGoat source at the pinned commit (suites/forge-webgoat/reference.json) 83.7% (41 / 49)
    garak detectors: keryn-eval vs garak 0.17.2.pre1 keryn-eval detectors garak detectors 0.17.2.pre1 100% (34 / 34)
    LLM Guard InvisibleText Unicode filtering Keryn prompt-injection-detector scan_invisible_text LLM Guard InvisibleText.scan 100% (17 / 17)
    LLM Guard passive scanner parity Keryn native passive text policy scanners LLM Guard read-only scanner methods 37.5% (3 / 8)
    Model-file inspection (pickle, PyTorch, joblib, safetensors, GGUF, ONNX) keryn artifact analyze (model readers) picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed) 73.3% (33 / 45)
    Nuclei query mutation: keryn-nuclei differential profile vs nuclei v3.11.1 keryn-nuclei differential nuclei v3.11.1 100% (24 / 24)
    Nuclei HTTP templates: keryn-nuclei vs nuclei v3.11.1 keryn-nuclei nuclei v3.11.1 96.1% (49 / 51) 98.0% (48 / 49) 98.0% (48 / 49)
    Promptfoo offline assertion parity Keryn Eval native config importer and replay engine Promptfoo assertion API 0.125.1 53.8% (7 / 13)
    promptfoo configs: keryn-eval vs promptfoo 0.125.1 keryn-eval promptfoo 0.125.1 69.7% (302 / 433)
    Dependency reachability from imports (JavaScript/TypeScript, Python, Java) keryn import reachability (ts/packages/sca/src/import-reachability.ts) independently authored truth labels authored-reach-1 93.3% (28 / 30)
    Semgrep rules: Keryn vs Semgrep CE 1.157.0 Keryn pattern rules Semgrep CE 1.157.0 100% (164 / 164) 100% (13,475 / 13,475) 100% (13,475 / 13,475)
    ZAP MCP detections: keryn-passive vs ZAP 2.17.0 (community script 100045, mcp add-on importer) keryn-passive ZAP’s MCP detections: community-scripts OpenModelContextProtocolServer.js (rule 100045) on GraalJS inside the scripts add-on’s ActiveScriptScanRule, and the zap-extensions mcp add-on’s McpImporter and VariantMcpJsonRpc, on ZAP core 2.17.0 92.3% (96 / 104)
    ZAP passive scan rules: keryn-passive vs ZAP 2.17.0 keryn-passive OWASP ZAP passive scan rules (zap-extensions pscanrules, pscanrulesBeta, pscanrulesAlpha) on ZAP core 2.17.0 100% (63 / 63)

    Reference overlap is the share of reference results also reported by Keryn; engine overlap is the share of Keryn results also reported by the reference. These are agreement measures under each suite’s comparator, not confirmed vulnerabilities or real-world precision. Values marked partial observations retain evidence from incomplete scans and cannot establish complete coverage.

    Receipts as of Keryn commit dd5df0da79c5. Each page links its receipt for download.