Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Ruby on Rails: how much of what Brakeman 8.1.0 finds does Keryn's default scan flag?

    View as Markdown

    On Brakeman’s own test applications, at how many of the places Brakeman reports a warning does Keryn’s default scan also report a finding?

    Complete comparison. Requested rule execution completed for this corpus. File coverage limits and detection differences are listed below.

    Evaluation detail Description
    Keryn engine Keryn default scan at 2e13028478f1: keryn scan with every bundled rule pack and the Deep pass, offline; rule names are not equated with reference checks by this location-based comparison.
    Rules revision 39867503faea04523bad6890549d514203eb8e97
    Measured CLI sha256 8dbea4ca727d50b89aefbd82d6db089bdd0d9d6d481bdb5c026f58c0d5f69335
    Measured asset manifest sha256 ed1f20de887db9302d57070a1552eaf12d423cd922efd2606a6407c18063da56
    Reference Brakeman 8.1.0. Reference: run locally, once per application. Never shipped.
    Corpus Brakeman test applications, presidentbeef/brakeman at v8.1.0 (c7781648630c)
    Comparison By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule.
    Comparison status Complete
    Scanner execution Exit 0; successful invocation: true
    Analysis coverage 1566/1566 rules executed (0 unsupported, 0 partial, 0 timed out); 996 files scanned, 89 skipped; 796 absence findings collapsed into 13 repo-level finding(s); 216 finding(s) suppressed (0 inline nosemgrep, 0 by config, 216 by code role); 4 file(s) parsed partially, 0 not parsed.
    Receipt brakeman.json, sha256 d588d291083a2caf
    Measure Value Count Denominator
    Reference rules whose every location Keryn also flags 59.3% 51 / 86 rules the reference ran, including rules that never fired on the corpus
    Reference rules that fired on the corpus 90.7% 78 / 86 rules the reference ran
    Fired rules whose every location Keryn also flags 65.4% 51 / 78 reference rules that fired on the corpus
    Reference findings Keryn also flags at the same file:line 91.6% 624 / 681 distinct (rule, file, line) findings the reference reported
    Keryn locations the reference also flags 78.3% 545 / 696 distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth)

    Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

    Cases by status: 27 differs, 51 identical, 8 no reference.

    • Brakeman’s test/apps directory: 14 Rails applications (Rails 2 to 8), 1085 files, built to exercise Brakeman’s checks; not real-world code.
    • A location gap measurement across independently authored rules. Matching locations do not establish equivalent rule semantics or exploitability.
    • One case per Brakeman check (warning check_name) in its default set (86); optional checks are not enabled. 46 warnings carry no line (for example, a Gemfile version advisory) and can never be matched by location; they stay in the denominator.
    • Comparison assessment: Successful execution and complete analysis coverage were recorded. Agreement remains location-based, not a vulnerability-equivalence or accuracy claim.
    Case Status Reference rows Keryn rows Matching Missing Extra
    CSRFTokenForgeryCVE differs 10 9 9 1
    CrossSiteScripting differs 121 110 110 11
    DefaultRoutes differs 10 3 3 7
    Deserialize differs 17 15 15 2
    DetailedExceptions differs 3 2 2 1
    DivideByZero no reference 0 0 0 0
    EOLRails differs 10 8 8 2
    EscapeFunction no reference 0 0 0 0
    FilterSkipping no reference 0 0 0 0
    ForceSSL differs 1 0 0 1
    ForgerySetting differs 6 4 4 2
    I18nXSS differs 2 1 1 1
    JRubyXML no reference 0 0 0 0
    JSONEntityEscape differs 5 0 0 5
    JSONParsing differs 6 5 5 1
    ModelAttrAccessible differs 5 0 0 5
    NestedAttributes no reference 0 0 0 0
    NumberToCurrency differs 8 7 7 1
    QuoteTableName differs 2 1 1 1
    Ransack no reference 0 0 0 0
    Redirect differs 17 16 16 1
    RenderInline differs 7 6 6 1
    ResponseSplitting differs 1 0 0 1
    ReverseTabnabbing no reference 0 0 0 0
    SQL differs 108 106 106 2
    SQLCVEs differs 8 7 7 1
    Secrets no reference 0 0 0 0
    SingleQuotes differs 3 2 2 1
    StripTags differs 5 4 4 1
    SymbolDoS differs 1 0 0 1
    SymbolDoSCVE differs 4 3 3 1
    UnscopedFind differs 3 0 0 3
    VerbConfusion differs 1 0 0 1
    XMLDoS differs 8 7 7 1
    YAMLParsing differs 3 2 2 1
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    BasicAuth identical 2 2 2 0
    BasicAuthTimingAttack identical 1 1 1 0
    CSRFTokenForgeryCVE differs 10 9 9 1
    ContentTag identical 22 22 22 0
    CookieSerialization identical 2 2 2 0
    CreateWith identical 5 5 5 0
    CrossSiteScripting differs 121 110 110 11
    DefaultRoutes differs 10 3 3 7
    Deserialize differs 17 15 15 2
    DetailedExceptions differs 3 2 2 1
    DigestDoS identical 2 2 2 0
    DivideByZero no reference 0 0 0 0
    DynamicFinders identical 2 2 2 0
    EOLRails differs 10 8 8 2
    EOLRuby identical 2 2 2 0
    EscapeFunction no reference 0 0 0 0
    Evaluation identical 16 16 16 0
    Execute identical 49 49 49 0
    FileAccess identical 14 14 14 0
    FileDisclosure identical 1 1 1 0
    FilterSkipping no reference 0 0 0 0
    ForceSSL differs 1 0 0 1
    ForgerySetting differs 6 4 4 2
    HeaderDoS identical 4 4 4 0
    I18nXSS differs 2 1 1 1
    JRubyXML no reference 0 0 0 0
    JSONEncoding identical 1 1 1 0
    JSONEntityEscape differs 5 0 0 5
    JSONParsing differs 6 5 5 1
    LinkTo identical 3 3 3 0
    LinkToHref identical 11 11 11 0
    MailTo identical 1 1 1 0
    MassAssignment identical 23 23 23 0
    MimeTypeDoS identical 5 5 5 0
    ModelAttrAccessible differs 5 0 0 5
    ModelAttributes identical 7 7 7 0
    ModelSerialize identical 4 4 4 0
    NestedAttributes no reference 0 0 0 0
    NestedAttributesBypass identical 1 1 1 0
    NumberToCurrency differs 8 7 7 1
    PageCachingCVE identical 1 1 1 0
    Pathname identical 2 2 2 0
    PermitAttributes identical 1 1 1 0
    QuoteTableName differs 2 1 1 1
    Ransack no reference 0 0 0 0
    Redirect differs 17 16 16 1
    RegexDoS identical 3 3 3 0
    Render identical 6 6 6 0
    RenderDoS identical 2 2 2 0
    RenderInline differs 7 6 6 1
    RenderRCE identical 3 3 3 0
    ResponseSplitting differs 1 0 0 1
    ReverseTabnabbing no reference 0 0 0 0
    RouteDoS identical 1 1 1 0
    SQL differs 108 106 106 2
    SQLCVEs differs 8 7 7 1
    SSLVerify identical 2 2 2 0
    SafeBufferManipulation identical 2 2 2 0
    SanitizeConfigCve identical 3 3 3 0
    SanitizeMethods identical 10 10 10 0
    Secrets no reference 0 0 0 0
    SelectTag identical 2 2 2 0
    SelectVulnerability identical 3 3 3 0
    Send identical 7 7 7 0
    SendFile identical 2 2 2 0
    SessionManipulation identical 2 2 2 0
    SessionSettings identical 9 9 9 0
    SimpleFormat identical 3 3 3 0
    SingleQuotes differs 3 2 2 1
    SkipBeforeFilter identical 5 5 5 0
    SprocketsPathTraversal identical 2 2 2 0
    StripTags differs 5 4 4 1
    SymbolDoS differs 1 0 0 1
    SymbolDoSCVE differs 4 3 3 1
    TemplateInjection identical 2 2 2 0
    TranslateBug identical 3 3 3 0
    UnsafeReflection identical 9 9 9 0
    UnsafeReflectionMethods identical 4 4 4 0
    UnscopedFind differs 3 0 0 3
    ValidationRegex identical 15 15 15 0
    VerbConfusion differs 1 0 0 1
    WeakHash identical 8 8 8 0
    WeakRSAKey identical 14 14 14 0
    WithoutProtection identical 2 2 2 0
    XMLDoS differs 8 7 7 1
    YAMLParsing differs 3 2 2 1