Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Python: how much of what Bandit 1.9.4 finds does Keryn's default scan flag?

    View as Markdown

    On Bandit’s own example corpus, at how many of the places Bandit reports a finding does Keryn’s default scan also report one?

    Evaluation detail Description
    Keryn engine Keryn default scan at 36d56edabe89: keryn scan with no options: every bundled rule pack, the Deep pass on, offline. No Bandit, Brakeman or ESLint rule is ported into it.
    Reference Bandit 1.9.4. Reference: run locally from the pinned checkout. Never shipped.
    Corpus Bandit examples, PyCQA/bandit at 1.9.4 (92ae8b82fb42)
    Comparison By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule.
    Comparison status Not recorded
    Receipt bandit.json, sha256 192051b35fc4d024
    Measure Value Count Denominator
    Reference rules whose every location Keryn also flags 97.3% 73 / 75 rules the reference ran, including rules that never fired on the corpus
    Reference rules that fired on the corpus 100% 75 / 75 rules the reference ran
    Fired rules whose every location Keryn also flags 97.3% 73 / 75 reference rules that fired on the corpus
    Reference findings Keryn also flags at the same file:line 99.5% 591 / 594 distinct (rule, file, line) findings the reference reported
    Keryn locations the reference also flags 88.8% 561 / 632 distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth)

    Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

    Cases by status: 2 differs, 73 identical.

    • Bandit’s examples/ directory (94 files), the snippets its own tests assert on: deliberately vulnerable, not real-world code.
    • A gap measurement. Keryn does not implement Bandit’s checks; its default rules target AI, agent and supply-chain risks, so a low ratio is expected and is the point of the measurement.
    • Bandit runs every built-in test (75) with default severity and confidence filters. Files Bandit cannot parse (3) are listed by Bandit as errors and produce no reference findings.
    Case Status Reference rows Keryn rows Matching Missing Extra
    B308 differs 43 41 41 2
    B704 differs 7 6 6 1
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    B101 identical 1 1 1 0
    B102 identical 1 1 1 0
    B103 identical 11 11 11 0
    B104 identical 1 1 1 0
    B105 identical 13 13 13 0
    B106 identical 2 2 2 0
    B107 identical 2 2 2 0
    B108 identical 4 4 4 0
    B110 identical 2 2 2 0
    B112 identical 2 2 2 0
    B113 identical 25 25 25 0
    B201 identical 1 1 1 0
    B202 identical 5 5 5 0
    B301 identical 13 13 13 0
    B302 identical 2 2 2 0
    B303 identical 11 11 11 0
    B304 identical 16 16 16 0
    B305 identical 1 1 1 0
    B306 identical 4 4 4 0
    B307 identical 3 3 3 0
    B308 differs 43 41 41 2
    B310 identical 8 8 8 0
    B311 identical 12 12 12 0
    B312 identical 1 1 1 0
    B313 identical 4 4 4 0
    B314 identical 4 4 4 0
    B315 identical 1 1 1 0
    B316 identical 2 2 2 0
    B317 identical 7 7 7 0
    B318 identical 2 2 2 0
    B319 identical 2 2 2 0
    B321 identical 1 1 1 0
    B323 identical 1 1 1 0
    B324 identical 24 24 24 0
    B401 identical 1 1 1 0
    B402 identical 2 2 2 0
    B403 identical 9 9 9 0
    B404 identical 14 14 14 0
    B405 identical 2 2 2 0
    B406 identical 3 3 3 0
    B407 identical 1 1 1 0
    B408 identical 2 2 2 0
    B409 identical 2 2 2 0
    B411 identical 1 1 1 0
    B412 identical 3 3 3 0
    B413 identical 16 16 16 0
    B415 identical 1 1 1 0
    B501 identical 18 18 18 0
    B502 identical 18 18 18 0
    B503 identical 4 4 4 0
    B504 identical 1 1 1 0
    B505 identical 14 14 14 0
    B506 identical 3 3 3 0
    B507 identical 8 8 8 0
    B508 identical 1 1 1 0
    B509 identical 2 2 2 0
    B601 identical 1 1 1 0
    B602 identical 28 28 28 0
    B603 identical 29 29 29 0
    B604 identical 1 1 1 0
    B605 identical 22 22 22 0
    B606 identical 20 20 20 0
    B607 identical 4 4 4 0
    B608 identical 49 49 49 0
    B609 identical 4 4 4 0
    B610 identical 11 11 11 0
    B611 identical 6 6 6 0
    B612 identical 1 1 1 0
    B613 identical 1 1 1 0
    B614 identical 3 3 3 0
    B615 identical 15 15 15 0
    B701 identical 5 5 5 0
    B702 identical 3 3 3 0
    B703 identical 26 26 26 0
    B704 differs 7 6 6 1