Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn's default scan flag?

    View as Markdown

    On eslint-plugin-security’s own rule tests, at how many of the places the plugin reports a problem does Keryn’s default scan also report a finding?

    Complete comparison. Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.

    Evaluation detail Description
    Keryn engine Keryn default scan at 1474fd84760a: keryn scan with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison.
    Rules revision b56b75050c5118e85ab2c3d7528d0f6a9cdc0e1a
    Measured CLI sha256 d1e280f99e9109b177381e52d2fdc9a7fd9fc226265927d9182add5dd0d3a331
    Measured asset manifest sha256 42ce30db0ded632010aa93c1ff50cdd7bf219776194aae4d76e8dec853e1d69a
    Reference eslint-plugin-security 4.2.1 on ESLint 9.39.1. Reference: run locally from the pinned checkout. Never shipped.
    Corpus eslint-plugin-security rule tests, one file per case, eslint-community/eslint-plugin-security at eslint-plugin-security-v4.2.1 (7c5b9cd51a70)
    Comparison By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule.
    Comparison status Complete
    Scanner execution Exit 0; successful invocation: true
    Receipt eslint-security.json, sha256 d03fef364f494ad9
    Measure Value Count Denominator
    Reference rules whose every location Keryn also flags 80.0% 12 / 15 rules the reference ran, including rules that never fired on the corpus
    Reference rules that fired on the corpus 100% 15 / 15 rules the reference ran
    Fired rules whose every location Keryn also flags 80.0% 12 / 15 reference rules that fired on the corpus
    Reference findings Keryn also flags at the same file:line 91.3% 105 / 115 distinct (rule, file, line) findings the reference reported
    Keryn locations the reference also flags 74.8% 104 / 139 distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth)

    Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

    Cases by status: 3 differs, 12 identical.

    • Every RuleTester case in the plugin’s test/rules/*.js (15 test files) written out as one file each: 109 invalid and 114 valid snippets. Snippets are a few lines each, not applications.
    • A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
    • The reference is ESLint 9.39.1 with all 15 plugin rules at their default options; RuleTester options are not applied, so the reference is what ESLint reports on each file, not each test’s expected errors. 0 snippet(s) do not parse and produce no reference findings.
    Case Status Reference rows Keryn rows Matching Missing Extra
    detect-bidi-characters differs 16 10 10 6
    detect-invisible-characters differs 14 11 11 3
    detect-unsafe-regex differs 3 2 2 1
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    detect-bidi-characters differs 16 10 10 6
    detect-buffer-noassert identical 29 29 29 0
    detect-child-process identical 15 15 15 0
    detect-disable-mustache-escape identical 1 1 1 0
    detect-eval-with-expression identical 1 1 1 0
    detect-invisible-characters differs 14 11 11 3
    detect-new-buffer identical 1 1 1 0
    detect-no-csrf-before-method-override identical 1 1 1 0
    detect-non-literal-fs-filename identical 25 25 25 0
    detect-non-literal-regexp identical 2 2 2 0
    detect-non-literal-require identical 2 2 2 0
    detect-object-injection identical 2 2 2 0
    detect-possible-timing-attacks identical 2 2 2 0
    detect-pseudoRandomBytes identical 1 1 1 0
    detect-unsafe-regex differs 3 2 2 1