# Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn's default scan flag?

> On Brakeman's own test applications, at how many of the places Brakeman reports a warning does Keryn's default scan also report a finding?

Source: https://keryn.dev/evals/brakeman/

On Brakeman's own test applications, at how many of the places Brakeman reports a warning does Keryn's default scan also report a finding?

**Complete comparison.** Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.

| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn default scan at `2e13028478f1`: `keryn scan` with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison. |
| Rules revision | `39867503faea04523bad6890549d514203eb8e97` |
| Measured CLI | sha256 `8dbea4ca727d50b89aefbd82d6db089bdd0d9d6d481bdb5c026f58c0d5f69335` |
| Measured asset manifest | sha256 `ed1f20de887db9302d57070a1552eaf12d423cd922efd2606a6407c18063da56` |
| Reference | Brakeman 8.1.0. Reference: run locally, once per application. Never shipped. |
| Corpus | Brakeman test applications, [presidentbeef/brakeman](https://github.com/presidentbeef/brakeman/tree/c7781648630c789dd1a2f089e94b5d8c463169fb/test/apps) at `v8.1.0` (`c7781648630c`) |
| Comparison | By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means "Keryn flags this line", not "Keryn found the same issue". A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status | Complete |
| Scanner execution | Exit 0; successful invocation: true |
| Analysis coverage | 1566/1566 rules executed (0 unsupported, 0 partial, 0 timed out); 996 files scanned, 89 skipped; 796 absence findings collapsed into 13 repo-level finding(s); 216 finding(s) suppressed (0 inline nosemgrep, 0 by config, 216 by code role); 4 file(s) parsed partially, 0 not parsed. |
| Receipt | [`brakeman.json`](https://keryn.dev/evals/brakeman.json), sha256 `d588d291083a2caf` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Reference rules whose every location Keryn also flags | **59.3%** | 51 / 86 | rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus | **90.7%** | 78 / 86 | rules the reference ran |
| Fired rules whose every location Keryn also flags | **65.4%** | 51 / 78 | reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line | **91.6%** | 624 / 681 | distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags | **78.3%** | 545 / 696 | distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |

Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

Cases by status: 27 differs, 51 identical, 8 no reference.

## What this does and does not measure

- Brakeman's test/apps directory: 14 Rails applications (Rails 2 to 8), 1085 files, built to exercise Brakeman's checks; not real-world code.
- A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
- One case per Brakeman check (warning check_name) in its default set (86); optional checks are not enabled. 46 warnings carry no line (for example, a Gemfile version advisory) and can never be matched by location; they stay in the denominator.
- Comparison assessment: Successful execution and complete analysis coverage were recorded. Agreement remains location-based, not a vulnerability-equivalence or accuracy claim.

## Differences and unassessed cases (35)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `CSRFTokenForgeryCVE` | differs | 10 | 9 | 9 | 1 |  |
| `CrossSiteScripting` | differs | 121 | 110 | 110 | 11 |  |
| `DefaultRoutes` | differs | 10 | 3 | 3 | 7 |  |
| `Deserialize` | differs | 17 | 15 | 15 | 2 |  |
| `DetailedExceptions` | differs | 3 | 2 | 2 | 1 |  |
| `DivideByZero` | no reference | 0 | 0 | 0 | 0 |  |
| `EOLRails` | differs | 10 | 8 | 8 | 2 |  |
| `EscapeFunction` | no reference | 0 | 0 | 0 | 0 |  |
| `FilterSkipping` | no reference | 0 | 0 | 0 | 0 |  |
| `ForceSSL` | differs | 1 | 0 | 0 | 1 |  |
| `ForgerySetting` | differs | 6 | 4 | 4 | 2 |  |
| `I18nXSS` | differs | 2 | 1 | 1 | 1 |  |
| `JRubyXML` | no reference | 0 | 0 | 0 | 0 |  |
| `JSONEntityEscape` | differs | 5 | 0 | 0 | 5 |  |
| `JSONParsing` | differs | 6 | 5 | 5 | 1 |  |
| `ModelAttrAccessible` | differs | 5 | 0 | 0 | 5 |  |
| `NestedAttributes` | no reference | 0 | 0 | 0 | 0 |  |
| `NumberToCurrency` | differs | 8 | 7 | 7 | 1 |  |
| `QuoteTableName` | differs | 2 | 1 | 1 | 1 |  |
| `Ransack` | no reference | 0 | 0 | 0 | 0 |  |
| `Redirect` | differs | 17 | 16 | 16 | 1 |  |
| `RenderInline` | differs | 7 | 6 | 6 | 1 |  |
| `ResponseSplitting` | differs | 1 | 0 | 0 | 1 |  |
| `ReverseTabnabbing` | no reference | 0 | 0 | 0 | 0 |  |
| `SQL` | differs | 108 | 106 | 106 | 2 |  |
| `SQLCVEs` | differs | 8 | 7 | 7 | 1 |  |
| `Secrets` | no reference | 0 | 0 | 0 | 0 |  |
| `SingleQuotes` | differs | 3 | 2 | 2 | 1 |  |
| `StripTags` | differs | 5 | 4 | 4 | 1 |  |
| `SymbolDoS` | differs | 1 | 0 | 0 | 1 |  |
| `SymbolDoSCVE` | differs | 4 | 3 | 3 | 1 |  |
| `UnscopedFind` | differs | 3 | 0 | 0 | 3 |  |
| `VerbConfusion` | differs | 1 | 0 | 0 | 1 |  |
| `XMLDoS` | differs | 8 | 7 | 7 | 1 |  |
| `YAMLParsing` | differs | 3 | 2 | 2 | 1 |  |

## All cases (86)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `BasicAuth` | identical | 2 | 2 | 2 | 0 |  |
| `BasicAuthTimingAttack` | identical | 1 | 1 | 1 | 0 |  |
| `CSRFTokenForgeryCVE` | differs | 10 | 9 | 9 | 1 |  |
| `ContentTag` | identical | 22 | 22 | 22 | 0 |  |
| `CookieSerialization` | identical | 2 | 2 | 2 | 0 |  |
| `CreateWith` | identical | 5 | 5 | 5 | 0 |  |
| `CrossSiteScripting` | differs | 121 | 110 | 110 | 11 |  |
| `DefaultRoutes` | differs | 10 | 3 | 3 | 7 |  |
| `Deserialize` | differs | 17 | 15 | 15 | 2 |  |
| `DetailedExceptions` | differs | 3 | 2 | 2 | 1 |  |
| `DigestDoS` | identical | 2 | 2 | 2 | 0 |  |
| `DivideByZero` | no reference | 0 | 0 | 0 | 0 |  |
| `DynamicFinders` | identical | 2 | 2 | 2 | 0 |  |
| `EOLRails` | differs | 10 | 8 | 8 | 2 |  |
| `EOLRuby` | identical | 2 | 2 | 2 | 0 |  |
| `EscapeFunction` | no reference | 0 | 0 | 0 | 0 |  |
| `Evaluation` | identical | 16 | 16 | 16 | 0 |  |
| `Execute` | identical | 49 | 49 | 49 | 0 |  |
| `FileAccess` | identical | 14 | 14 | 14 | 0 |  |
| `FileDisclosure` | identical | 1 | 1 | 1 | 0 |  |
| `FilterSkipping` | no reference | 0 | 0 | 0 | 0 |  |
| `ForceSSL` | differs | 1 | 0 | 0 | 1 |  |
| `ForgerySetting` | differs | 6 | 4 | 4 | 2 |  |
| `HeaderDoS` | identical | 4 | 4 | 4 | 0 |  |
| `I18nXSS` | differs | 2 | 1 | 1 | 1 |  |
| `JRubyXML` | no reference | 0 | 0 | 0 | 0 |  |
| `JSONEncoding` | identical | 1 | 1 | 1 | 0 |  |
| `JSONEntityEscape` | differs | 5 | 0 | 0 | 5 |  |
| `JSONParsing` | differs | 6 | 5 | 5 | 1 |  |
| `LinkTo` | identical | 3 | 3 | 3 | 0 |  |
| `LinkToHref` | identical | 11 | 11 | 11 | 0 |  |
| `MailTo` | identical | 1 | 1 | 1 | 0 |  |
| `MassAssignment` | identical | 23 | 23 | 23 | 0 |  |
| `MimeTypeDoS` | identical | 5 | 5 | 5 | 0 |  |
| `ModelAttrAccessible` | differs | 5 | 0 | 0 | 5 |  |
| `ModelAttributes` | identical | 7 | 7 | 7 | 0 |  |
| `ModelSerialize` | identical | 4 | 4 | 4 | 0 |  |
| `NestedAttributes` | no reference | 0 | 0 | 0 | 0 |  |
| `NestedAttributesBypass` | identical | 1 | 1 | 1 | 0 |  |
| `NumberToCurrency` | differs | 8 | 7 | 7 | 1 |  |
| `PageCachingCVE` | identical | 1 | 1 | 1 | 0 |  |
| `Pathname` | identical | 2 | 2 | 2 | 0 |  |
| `PermitAttributes` | identical | 1 | 1 | 1 | 0 |  |
| `QuoteTableName` | differs | 2 | 1 | 1 | 1 |  |
| `Ransack` | no reference | 0 | 0 | 0 | 0 |  |
| `Redirect` | differs | 17 | 16 | 16 | 1 |  |
| `RegexDoS` | identical | 3 | 3 | 3 | 0 |  |
| `Render` | identical | 6 | 6 | 6 | 0 |  |
| `RenderDoS` | identical | 2 | 2 | 2 | 0 |  |
| `RenderInline` | differs | 7 | 6 | 6 | 1 |  |
| `RenderRCE` | identical | 3 | 3 | 3 | 0 |  |
| `ResponseSplitting` | differs | 1 | 0 | 0 | 1 |  |
| `ReverseTabnabbing` | no reference | 0 | 0 | 0 | 0 |  |
| `RouteDoS` | identical | 1 | 1 | 1 | 0 |  |
| `SQL` | differs | 108 | 106 | 106 | 2 |  |
| `SQLCVEs` | differs | 8 | 7 | 7 | 1 |  |
| `SSLVerify` | identical | 2 | 2 | 2 | 0 |  |
| `SafeBufferManipulation` | identical | 2 | 2 | 2 | 0 |  |
| `SanitizeConfigCve` | identical | 3 | 3 | 3 | 0 |  |
| `SanitizeMethods` | identical | 10 | 10 | 10 | 0 |  |
| `Secrets` | no reference | 0 | 0 | 0 | 0 |  |
| `SelectTag` | identical | 2 | 2 | 2 | 0 |  |
| `SelectVulnerability` | identical | 3 | 3 | 3 | 0 |  |
| `Send` | identical | 7 | 7 | 7 | 0 |  |
| `SendFile` | identical | 2 | 2 | 2 | 0 |  |
| `SessionManipulation` | identical | 2 | 2 | 2 | 0 |  |
| `SessionSettings` | identical | 9 | 9 | 9 | 0 |  |
| `SimpleFormat` | identical | 3 | 3 | 3 | 0 |  |
| `SingleQuotes` | differs | 3 | 2 | 2 | 1 |  |
| `SkipBeforeFilter` | identical | 5 | 5 | 5 | 0 |  |
| `SprocketsPathTraversal` | identical | 2 | 2 | 2 | 0 |  |
| `StripTags` | differs | 5 | 4 | 4 | 1 |  |
| `SymbolDoS` | differs | 1 | 0 | 0 | 1 |  |
| `SymbolDoSCVE` | differs | 4 | 3 | 3 | 1 |  |
| `TemplateInjection` | identical | 2 | 2 | 2 | 0 |  |
| `TranslateBug` | identical | 3 | 3 | 3 | 0 |  |
| `UnsafeReflection` | identical | 9 | 9 | 9 | 0 |  |
| `UnsafeReflectionMethods` | identical | 4 | 4 | 4 | 0 |  |
| `UnscopedFind` | differs | 3 | 0 | 0 | 3 |  |
| `ValidationRegex` | identical | 15 | 15 | 15 | 0 |  |
| `VerbConfusion` | differs | 1 | 0 | 0 | 1 |  |
| `WeakHash` | identical | 8 | 8 | 8 | 0 |  |
| `WeakRSAKey` | identical | 14 | 14 | 14 | 0 |  |
| `WithoutProtection` | identical | 2 | 2 | 2 | 0 |  |
| `XMLDoS` | differs | 8 | 7 | 7 | 1 |  |
| `YAMLParsing` | differs | 3 | 2 | 2 | 1 |  |
