Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn's default scan flag?
On Brakeman’s own test applications, at how many of the places Brakeman reports a warning does Keryn’s default scan also report a finding?
Complete comparison. Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.
| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn default scan at 2e13028478f1: keryn scan with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison. |
| Rules revision | 39867503faea04523bad6890549d514203eb8e97 |
| Measured CLI | sha256 8dbea4ca727d50b89aefbd82d6db089bdd0d9d6d481bdb5c026f58c0d5f69335 |
| Measured asset manifest | sha256 ed1f20de887db9302d57070a1552eaf12d423cd922efd2606a6407c18063da56 |
| Reference | Brakeman 8.1.0. Reference: run locally, once per application. Never shipped. |
| Corpus | Brakeman test applications, presidentbeef/brakeman at v8.1.0 (c7781648630c) |
| Comparison | By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status | Complete |
| Scanner execution | Exit 0; successful invocation: true |
| Analysis coverage | 1566/1566 rules executed (0 unsupported, 0 partial, 0 timed out); 996 files scanned, 89 skipped; 796 absence findings collapsed into 13 repo-level finding(s); 216 finding(s) suppressed (0 inline nosemgrep, 0 by config, 216 by code role); 4 file(s) parsed partially, 0 not parsed. |
| Receipt | brakeman.json, sha256 d588d291083a2caf |
Results
Section titled “Results”| Measure | Value | Count | Denominator |
|---|---|---|---|
| Reference rules whose every location Keryn also flags | 59.3% | 51 / 86 | rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus | 90.7% | 78 / 86 | rules the reference ran |
| Fired rules whose every location Keryn also flags | 65.4% | 51 / 78 | reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line | 91.6% | 624 / 681 | distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags | 78.3% | 545 / 696 | distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |
Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.
Cases by status: 27 differs, 51 identical, 8 no reference.
What this does and does not measure
Section titled “What this does and does not measure”- Brakeman’s test/apps directory: 14 Rails applications (Rails 2 to 8), 1085 files, built to exercise Brakeman’s checks; not real-world code.
- A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
- One case per Brakeman check (warning check_name) in its default set (86); optional checks are not enabled. 46 warnings carry no line (for example, a Gemfile version advisory) and can never be matched by location; they stay in the denominator.
- Comparison assessment: Successful execution and complete analysis coverage were recorded. Agreement remains location-based, not a vulnerability-equivalence or accuracy claim.
Differences and unassessed cases (35)
Section titled “Differences and unassessed cases (35)”| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
CSRFTokenForgeryCVE |
differs | 10 | 9 | 9 | 1 | |
CrossSiteScripting |
differs | 121 | 110 | 110 | 11 | |
DefaultRoutes |
differs | 10 | 3 | 3 | 7 | |
Deserialize |
differs | 17 | 15 | 15 | 2 | |
DetailedExceptions |
differs | 3 | 2 | 2 | 1 | |
DivideByZero |
no reference | 0 | 0 | 0 | 0 | |
EOLRails |
differs | 10 | 8 | 8 | 2 | |
EscapeFunction |
no reference | 0 | 0 | 0 | 0 | |
FilterSkipping |
no reference | 0 | 0 | 0 | 0 | |
ForceSSL |
differs | 1 | 0 | 0 | 1 | |
ForgerySetting |
differs | 6 | 4 | 4 | 2 | |
I18nXSS |
differs | 2 | 1 | 1 | 1 | |
JRubyXML |
no reference | 0 | 0 | 0 | 0 | |
JSONEntityEscape |
differs | 5 | 0 | 0 | 5 | |
JSONParsing |
differs | 6 | 5 | 5 | 1 | |
ModelAttrAccessible |
differs | 5 | 0 | 0 | 5 | |
NestedAttributes |
no reference | 0 | 0 | 0 | 0 | |
NumberToCurrency |
differs | 8 | 7 | 7 | 1 | |
QuoteTableName |
differs | 2 | 1 | 1 | 1 | |
Ransack |
no reference | 0 | 0 | 0 | 0 | |
Redirect |
differs | 17 | 16 | 16 | 1 | |
RenderInline |
differs | 7 | 6 | 6 | 1 | |
ResponseSplitting |
differs | 1 | 0 | 0 | 1 | |
ReverseTabnabbing |
no reference | 0 | 0 | 0 | 0 | |
SQL |
differs | 108 | 106 | 106 | 2 | |
SQLCVEs |
differs | 8 | 7 | 7 | 1 | |
Secrets |
no reference | 0 | 0 | 0 | 0 | |
SingleQuotes |
differs | 3 | 2 | 2 | 1 | |
StripTags |
differs | 5 | 4 | 4 | 1 | |
SymbolDoS |
differs | 1 | 0 | 0 | 1 | |
SymbolDoSCVE |
differs | 4 | 3 | 3 | 1 | |
UnscopedFind |
differs | 3 | 0 | 0 | 3 | |
VerbConfusion |
differs | 1 | 0 | 0 | 1 | |
XMLDoS |
differs | 8 | 7 | 7 | 1 | |
YAMLParsing |
differs | 3 | 2 | 2 | 1 |
All cases (86)
Section titled “All cases (86)”Show every case
| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
BasicAuth |
identical | 2 | 2 | 2 | 0 | |
BasicAuthTimingAttack |
identical | 1 | 1 | 1 | 0 | |
CSRFTokenForgeryCVE |
differs | 10 | 9 | 9 | 1 | |
ContentTag |
identical | 22 | 22 | 22 | 0 | |
CookieSerialization |
identical | 2 | 2 | 2 | 0 | |
CreateWith |
identical | 5 | 5 | 5 | 0 | |
CrossSiteScripting |
differs | 121 | 110 | 110 | 11 | |
DefaultRoutes |
differs | 10 | 3 | 3 | 7 | |
Deserialize |
differs | 17 | 15 | 15 | 2 | |
DetailedExceptions |
differs | 3 | 2 | 2 | 1 | |
DigestDoS |
identical | 2 | 2 | 2 | 0 | |
DivideByZero |
no reference | 0 | 0 | 0 | 0 | |
DynamicFinders |
identical | 2 | 2 | 2 | 0 | |
EOLRails |
differs | 10 | 8 | 8 | 2 | |
EOLRuby |
identical | 2 | 2 | 2 | 0 | |
EscapeFunction |
no reference | 0 | 0 | 0 | 0 | |
Evaluation |
identical | 16 | 16 | 16 | 0 | |
Execute |
identical | 49 | 49 | 49 | 0 | |
FileAccess |
identical | 14 | 14 | 14 | 0 | |
FileDisclosure |
identical | 1 | 1 | 1 | 0 | |
FilterSkipping |
no reference | 0 | 0 | 0 | 0 | |
ForceSSL |
differs | 1 | 0 | 0 | 1 | |
ForgerySetting |
differs | 6 | 4 | 4 | 2 | |
HeaderDoS |
identical | 4 | 4 | 4 | 0 | |
I18nXSS |
differs | 2 | 1 | 1 | 1 | |
JRubyXML |
no reference | 0 | 0 | 0 | 0 | |
JSONEncoding |
identical | 1 | 1 | 1 | 0 | |
JSONEntityEscape |
differs | 5 | 0 | 0 | 5 | |
JSONParsing |
differs | 6 | 5 | 5 | 1 | |
LinkTo |
identical | 3 | 3 | 3 | 0 | |
LinkToHref |
identical | 11 | 11 | 11 | 0 | |
MailTo |
identical | 1 | 1 | 1 | 0 | |
MassAssignment |
identical | 23 | 23 | 23 | 0 | |
MimeTypeDoS |
identical | 5 | 5 | 5 | 0 | |
ModelAttrAccessible |
differs | 5 | 0 | 0 | 5 | |
ModelAttributes |
identical | 7 | 7 | 7 | 0 | |
ModelSerialize |
identical | 4 | 4 | 4 | 0 | |
NestedAttributes |
no reference | 0 | 0 | 0 | 0 | |
NestedAttributesBypass |
identical | 1 | 1 | 1 | 0 | |
NumberToCurrency |
differs | 8 | 7 | 7 | 1 | |
PageCachingCVE |
identical | 1 | 1 | 1 | 0 | |
Pathname |
identical | 2 | 2 | 2 | 0 | |
PermitAttributes |
identical | 1 | 1 | 1 | 0 | |
QuoteTableName |
differs | 2 | 1 | 1 | 1 | |
Ransack |
no reference | 0 | 0 | 0 | 0 | |
Redirect |
differs | 17 | 16 | 16 | 1 | |
RegexDoS |
identical | 3 | 3 | 3 | 0 | |
Render |
identical | 6 | 6 | 6 | 0 | |
RenderDoS |
identical | 2 | 2 | 2 | 0 | |
RenderInline |
differs | 7 | 6 | 6 | 1 | |
RenderRCE |
identical | 3 | 3 | 3 | 0 | |
ResponseSplitting |
differs | 1 | 0 | 0 | 1 | |
ReverseTabnabbing |
no reference | 0 | 0 | 0 | 0 | |
RouteDoS |
identical | 1 | 1 | 1 | 0 | |
SQL |
differs | 108 | 106 | 106 | 2 | |
SQLCVEs |
differs | 8 | 7 | 7 | 1 | |
SSLVerify |
identical | 2 | 2 | 2 | 0 | |
SafeBufferManipulation |
identical | 2 | 2 | 2 | 0 | |
SanitizeConfigCve |
identical | 3 | 3 | 3 | 0 | |
SanitizeMethods |
identical | 10 | 10 | 10 | 0 | |
Secrets |
no reference | 0 | 0 | 0 | 0 | |
SelectTag |
identical | 2 | 2 | 2 | 0 | |
SelectVulnerability |
identical | 3 | 3 | 3 | 0 | |
Send |
identical | 7 | 7 | 7 | 0 | |
SendFile |
identical | 2 | 2 | 2 | 0 | |
SessionManipulation |
identical | 2 | 2 | 2 | 0 | |
SessionSettings |
identical | 9 | 9 | 9 | 0 | |
SimpleFormat |
identical | 3 | 3 | 3 | 0 | |
SingleQuotes |
differs | 3 | 2 | 2 | 1 | |
SkipBeforeFilter |
identical | 5 | 5 | 5 | 0 | |
SprocketsPathTraversal |
identical | 2 | 2 | 2 | 0 | |
StripTags |
differs | 5 | 4 | 4 | 1 | |
SymbolDoS |
differs | 1 | 0 | 0 | 1 | |
SymbolDoSCVE |
differs | 4 | 3 | 3 | 1 | |
TemplateInjection |
identical | 2 | 2 | 2 | 0 | |
TranslateBug |
identical | 3 | 3 | 3 | 0 | |
UnsafeReflection |
identical | 9 | 9 | 9 | 0 | |
UnsafeReflectionMethods |
identical | 4 | 4 | 4 | 0 | |
UnscopedFind |
differs | 3 | 0 | 0 | 3 | |
ValidationRegex |
identical | 15 | 15 | 15 | 0 | |
VerbConfusion |
differs | 1 | 0 | 0 | 1 | |
WeakHash |
identical | 8 | 8 | 8 | 0 | |
WeakRSAKey |
identical | 14 | 14 | 14 | 0 | |
WithoutProtection |
identical | 2 | 2 | 2 | 0 | |
XMLDoS |
differs | 8 | 7 | 7 | 1 | |
YAMLParsing |
differs | 3 | 2 | 2 | 1 |