# JavaScript: how much of what eslint-plugin-security 4.2.1 finds does Keryn's default scan flag?

> On eslint-plugin-security's own rule tests, at how many of the places the plugin reports a problem does Keryn's default scan also report a finding?

Source: https://keryn.dev/evals/eslint-security/

On eslint-plugin-security's own rule tests, at how many of the places the plugin reports a problem does Keryn's default scan also report a finding?

**Complete comparison.** Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.

| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn default scan at `1474fd84760a`: `keryn scan` with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison. |
| Rules revision | `b56b75050c5118e85ab2c3d7528d0f6a9cdc0e1a` |
| Measured CLI | sha256 `d1e280f99e9109b177381e52d2fdc9a7fd9fc226265927d9182add5dd0d3a331` |
| Measured asset manifest | sha256 `42ce30db0ded632010aa93c1ff50cdd7bf219776194aae4d76e8dec853e1d69a` |
| Reference | eslint-plugin-security 4.2.1 on ESLint 9.39.1. Reference: run locally from the pinned checkout. Never shipped. |
| Corpus | eslint-plugin-security rule tests, one file per case, [eslint-community/eslint-plugin-security](https://github.com/eslint-community/eslint-plugin-security/tree/7c5b9cd51a70862b598d6125e50aae8af885d963/test/rules) at `eslint-plugin-security-v4.2.1` (`7c5b9cd51a70`) |
| Comparison | By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means "Keryn flags this line", not "Keryn found the same issue". A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status | Complete |
| Scanner execution | Exit 0; successful invocation: true |
| Receipt | [`eslint-security.json`](https://keryn.dev/evals/eslint-security.json), sha256 `d03fef364f494ad9` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Reference rules whose every location Keryn also flags | **80.0%** | 12 / 15 | rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus | **100%** | 15 / 15 | rules the reference ran |
| Fired rules whose every location Keryn also flags | **80.0%** | 12 / 15 | reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line | **91.3%** | 105 / 115 | distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags | **74.8%** | 104 / 139 | distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |

Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

Cases by status: 3 differs, 12 identical.

## What this does and does not measure

- Every RuleTester case in the plugin's test/rules/*.js (15 test files) written out as one file each: 109 invalid and 114 valid snippets. Snippets are a few lines each, not applications.
- A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
- The reference is ESLint 9.39.1 with all 15 plugin rules at their default options; RuleTester options are not applied, so the reference is what ESLint reports on each file, not each test's expected errors. 0 snippet(s) do not parse and produce no reference findings.

## Differences and unassessed cases (3)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `detect-bidi-characters` | differs | 16 | 10 | 10 | 6 |  |
| `detect-invisible-characters` | differs | 14 | 11 | 11 | 3 |  |
| `detect-unsafe-regex` | differs | 3 | 2 | 2 | 1 |  |

## All cases (15)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `detect-bidi-characters` | differs | 16 | 10 | 10 | 6 |  |
| `detect-buffer-noassert` | identical | 29 | 29 | 29 | 0 |  |
| `detect-child-process` | identical | 15 | 15 | 15 | 0 |  |
| `detect-disable-mustache-escape` | identical | 1 | 1 | 1 | 0 |  |
| `detect-eval-with-expression` | identical | 1 | 1 | 1 | 0 |  |
| `detect-invisible-characters` | differs | 14 | 11 | 11 | 3 |  |
| `detect-new-buffer` | identical | 1 | 1 | 1 | 0 |  |
| `detect-no-csrf-before-method-override` | identical | 1 | 1 | 1 | 0 |  |
| `detect-non-literal-fs-filename` | identical | 25 | 25 | 25 | 0 |  |
| `detect-non-literal-regexp` | identical | 2 | 2 | 2 | 0 |  |
| `detect-non-literal-require` | identical | 2 | 2 | 2 | 0 |  |
| `detect-object-injection` | identical | 2 | 2 | 2 | 0 |  |
| `detect-possible-timing-attacks` | identical | 2 | 2 | 2 | 0 |  |
| `detect-pseudoRandomBytes` | identical | 1 | 1 | 1 | 0 |  |
| `detect-unsafe-regex` | differs | 3 | 2 | 2 | 1 |  |
