# Python: how much of what Bandit 1.9.4 finds does Keryn's default scan flag?

> On Bandit's own example corpus, at how many of the places Bandit reports a finding does Keryn's default scan also report one?

Source: https://keryn.dev/evals/bandit/

On Bandit's own example corpus, at how many of the places Bandit reports a finding does Keryn's default scan also report one?

| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn default scan at `36d56edabe89`: `keryn scan` with no options: every bundled rule pack, the Deep pass on, offline. No Bandit, Brakeman or ESLint rule is ported into it. |
| Reference | Bandit 1.9.4. Reference: run locally from the pinned checkout. Never shipped. |
| Corpus | Bandit examples, [PyCQA/bandit](https://github.com/PyCQA/bandit/tree/92ae8b82fb422a639f0ed8d99e96cea769594e08/examples) at `1.9.4` (`92ae8b82fb42`) |
| Comparison | By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means "Keryn flags this line", not "Keryn found the same issue". A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status | Not recorded |
| Receipt | [`bandit.json`](https://keryn.dev/evals/bandit.json), sha256 `192051b35fc4d024` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Reference rules whose every location Keryn also flags | **97.3%** | 73 / 75 | rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus | **100%** | 75 / 75 | rules the reference ran |
| Fired rules whose every location Keryn also flags | **97.3%** | 73 / 75 | reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line | **99.5%** | 591 / 594 | distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags | **88.8%** | 561 / 632 | distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |

Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.

Cases by status: 2 differs, 73 identical.

## What this does and does not measure

- Bandit's examples/ directory (94 files), the snippets its own tests assert on: deliberately vulnerable, not real-world code.
- A gap measurement. Keryn does not implement Bandit's checks; its default rules target AI, agent and supply-chain risks, so a low ratio is expected and is the point of the measurement.
- Bandit runs every built-in test (75) with default severity and confidence filters. Files Bandit cannot parse (3) are listed by Bandit as errors and produce no reference findings.

## Differences and unassessed cases (2)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `B308` | differs | 43 | 41 | 41 | 2 |  |
| `B704` | differs | 7 | 6 | 6 | 1 |  |

## All cases (75)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `B101` | identical | 1 | 1 | 1 | 0 |  |
| `B102` | identical | 1 | 1 | 1 | 0 |  |
| `B103` | identical | 11 | 11 | 11 | 0 |  |
| `B104` | identical | 1 | 1 | 1 | 0 |  |
| `B105` | identical | 13 | 13 | 13 | 0 |  |
| `B106` | identical | 2 | 2 | 2 | 0 |  |
| `B107` | identical | 2 | 2 | 2 | 0 |  |
| `B108` | identical | 4 | 4 | 4 | 0 |  |
| `B110` | identical | 2 | 2 | 2 | 0 |  |
| `B112` | identical | 2 | 2 | 2 | 0 |  |
| `B113` | identical | 25 | 25 | 25 | 0 |  |
| `B201` | identical | 1 | 1 | 1 | 0 |  |
| `B202` | identical | 5 | 5 | 5 | 0 |  |
| `B301` | identical | 13 | 13 | 13 | 0 |  |
| `B302` | identical | 2 | 2 | 2 | 0 |  |
| `B303` | identical | 11 | 11 | 11 | 0 |  |
| `B304` | identical | 16 | 16 | 16 | 0 |  |
| `B305` | identical | 1 | 1 | 1 | 0 |  |
| `B306` | identical | 4 | 4 | 4 | 0 |  |
| `B307` | identical | 3 | 3 | 3 | 0 |  |
| `B308` | differs | 43 | 41 | 41 | 2 |  |
| `B310` | identical | 8 | 8 | 8 | 0 |  |
| `B311` | identical | 12 | 12 | 12 | 0 |  |
| `B312` | identical | 1 | 1 | 1 | 0 |  |
| `B313` | identical | 4 | 4 | 4 | 0 |  |
| `B314` | identical | 4 | 4 | 4 | 0 |  |
| `B315` | identical | 1 | 1 | 1 | 0 |  |
| `B316` | identical | 2 | 2 | 2 | 0 |  |
| `B317` | identical | 7 | 7 | 7 | 0 |  |
| `B318` | identical | 2 | 2 | 2 | 0 |  |
| `B319` | identical | 2 | 2 | 2 | 0 |  |
| `B321` | identical | 1 | 1 | 1 | 0 |  |
| `B323` | identical | 1 | 1 | 1 | 0 |  |
| `B324` | identical | 24 | 24 | 24 | 0 |  |
| `B401` | identical | 1 | 1 | 1 | 0 |  |
| `B402` | identical | 2 | 2 | 2 | 0 |  |
| `B403` | identical | 9 | 9 | 9 | 0 |  |
| `B404` | identical | 14 | 14 | 14 | 0 |  |
| `B405` | identical | 2 | 2 | 2 | 0 |  |
| `B406` | identical | 3 | 3 | 3 | 0 |  |
| `B407` | identical | 1 | 1 | 1 | 0 |  |
| `B408` | identical | 2 | 2 | 2 | 0 |  |
| `B409` | identical | 2 | 2 | 2 | 0 |  |
| `B411` | identical | 1 | 1 | 1 | 0 |  |
| `B412` | identical | 3 | 3 | 3 | 0 |  |
| `B413` | identical | 16 | 16 | 16 | 0 |  |
| `B415` | identical | 1 | 1 | 1 | 0 |  |
| `B501` | identical | 18 | 18 | 18 | 0 |  |
| `B502` | identical | 18 | 18 | 18 | 0 |  |
| `B503` | identical | 4 | 4 | 4 | 0 |  |
| `B504` | identical | 1 | 1 | 1 | 0 |  |
| `B505` | identical | 14 | 14 | 14 | 0 |  |
| `B506` | identical | 3 | 3 | 3 | 0 |  |
| `B507` | identical | 8 | 8 | 8 | 0 |  |
| `B508` | identical | 1 | 1 | 1 | 0 |  |
| `B509` | identical | 2 | 2 | 2 | 0 |  |
| `B601` | identical | 1 | 1 | 1 | 0 |  |
| `B602` | identical | 28 | 28 | 28 | 0 |  |
| `B603` | identical | 29 | 29 | 29 | 0 |  |
| `B604` | identical | 1 | 1 | 1 | 0 |  |
| `B605` | identical | 22 | 22 | 22 | 0 |  |
| `B606` | identical | 20 | 20 | 20 | 0 |  |
| `B607` | identical | 4 | 4 | 4 | 0 |  |
| `B608` | identical | 49 | 49 | 49 | 0 |  |
| `B609` | identical | 4 | 4 | 4 | 0 |  |
| `B610` | identical | 11 | 11 | 11 | 0 |  |
| `B611` | identical | 6 | 6 | 6 | 0 |  |
| `B612` | identical | 1 | 1 | 1 | 0 |  |
| `B613` | identical | 1 | 1 | 1 | 0 |  |
| `B614` | identical | 3 | 3 | 3 | 0 |  |
| `B615` | identical | 15 | 15 | 15 | 0 |  |
| `B701` | identical | 5 | 5 | 5 | 0 |  |
| `B702` | identical | 3 | 3 | 3 | 0 |  |
| `B703` | identical | 26 | 26 | 26 | 0 |  |
| `B704` | differs | 7 | 6 | 6 | 1 |  |
