JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn's default scan flag?
On eslint-plugin-security’s own rule tests, at how many of the places the plugin reports a problem does Keryn’s default scan also report a finding?
Complete comparison. Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.
| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn default scan at 1474fd84760a: keryn scan with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison. |
| Rules revision | b56b75050c5118e85ab2c3d7528d0f6a9cdc0e1a |
| Measured CLI | sha256 d1e280f99e9109b177381e52d2fdc9a7fd9fc226265927d9182add5dd0d3a331 |
| Measured asset manifest | sha256 42ce30db0ded632010aa93c1ff50cdd7bf219776194aae4d76e8dec853e1d69a |
| Reference | eslint-plugin-security 4.2.1 on ESLint 9.39.1. Reference: run locally from the pinned checkout. Never shipped. |
| Corpus | eslint-plugin-security rule tests, one file per case, eslint-community/eslint-plugin-security at eslint-plugin-security-v4.2.1 (7c5b9cd51a70) |
| Comparison | By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status | Complete |
| Scanner execution | Exit 0; successful invocation: true |
| Receipt | eslint-security.json, sha256 d03fef364f494ad9 |
Results
Section titled “Results”| Measure | Value | Count | Denominator |
|---|---|---|---|
| Reference rules whose every location Keryn also flags | 80.0% | 12 / 15 | rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus | 100% | 15 / 15 | rules the reference ran |
| Fired rules whose every location Keryn also flags | 80.0% | 12 / 15 | reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line | 91.3% | 105 / 115 | distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags | 74.8% | 104 / 139 | distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |
Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.
Cases by status: 3 differs, 12 identical.
What this does and does not measure
Section titled “What this does and does not measure”- Every RuleTester case in the plugin’s test/rules/*.js (15 test files) written out as one file each: 109 invalid and 114 valid snippets. Snippets are a few lines each, not applications.
- A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
- The reference is ESLint 9.39.1 with all 15 plugin rules at their default options; RuleTester options are not applied, so the reference is what ESLint reports on each file, not each test’s expected errors. 0 snippet(s) do not parse and produce no reference findings.
Differences and unassessed cases (3)
Section titled “Differences and unassessed cases (3)”| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
detect-bidi-characters |
differs | 16 | 10 | 10 | 6 | |
detect-invisible-characters |
differs | 14 | 11 | 11 | 3 | |
detect-unsafe-regex |
differs | 3 | 2 | 2 | 1 |
All cases (15)
Section titled “All cases (15)”Show every case
| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
detect-bidi-characters |
differs | 16 | 10 | 10 | 6 | |
detect-buffer-noassert |
identical | 29 | 29 | 29 | 0 | |
detect-child-process |
identical | 15 | 15 | 15 | 0 | |
detect-disable-mustache-escape |
identical | 1 | 1 | 1 | 0 | |
detect-eval-with-expression |
identical | 1 | 1 | 1 | 0 | |
detect-invisible-characters |
differs | 14 | 11 | 11 | 3 | |
detect-new-buffer |
identical | 1 | 1 | 1 | 0 | |
detect-no-csrf-before-method-override |
identical | 1 | 1 | 1 | 0 | |
detect-non-literal-fs-filename |
identical | 25 | 25 | 25 | 0 | |
detect-non-literal-regexp |
identical | 2 | 2 | 2 | 0 | |
detect-non-literal-require |
identical | 2 | 2 | 2 | 0 | |
detect-object-injection |
identical | 2 | 2 | 2 | 0 | |
detect-possible-timing-attacks |
identical | 2 | 2 | 2 | 0 | |
detect-pseudoRandomBytes |
identical | 1 | 1 | 1 | 0 | |
detect-unsafe-regex |
differs | 3 | 2 | 2 | 1 |