On Bandit’s own example corpus, at how many of the places Bandit reports a finding does Keryn’s default scan also report one?
| Evaluation detail |
Description |
| Keryn engine |
Keryn default scan at 36d56edabe89: keryn scan with no options: every bundled rule pack, the Deep pass on, offline. No Bandit, Brakeman or ESLint rule is ported into it. |
| Reference |
Bandit 1.9.4. Reference: run locally from the pinned checkout. Never shipped. |
| Corpus |
Bandit examples, PyCQA/bandit at 1.9.4 (92ae8b82fb42) |
| Comparison |
By location only: a reference finding is matched when Keryn reports any finding on the same file (relative to the corpus root) and line. No rule-to-rule mapping is used, so a match means “Keryn flags this line”, not “Keryn found the same issue”. A rule is identical when every location it reports is matched; Keryn findings elsewhere are counted in the summary, not charged to a rule. |
| Comparison status |
Not recorded |
| Receipt |
bandit.json, sha256 192051b35fc4d024 |
| Measure |
Value |
Count |
Denominator |
| Reference rules whose every location Keryn also flags |
97.3% |
73 / 75 |
rules the reference ran, including rules that never fired on the corpus |
| Reference rules that fired on the corpus |
100% |
75 / 75 |
rules the reference ran |
| Fired rules whose every location Keryn also flags |
97.3% |
73 / 75 |
reference rules that fired on the corpus |
| Reference findings Keryn also flags at the same file:line |
99.5% |
591 / 594 |
distinct (rule, file, line) findings the reference reported |
| Keryn locations the reference also flags |
88.8% |
561 / 632 |
distinct file:line locations Keryn reported on the corpus (agreement with the reference, not ground truth) |
Reference overlap counts distinct (reference check, file, line) findings; engine overlap counts distinct (file, line) locations. A location can belong to several reference checks, so the two overlap numerators can differ. Neither measure confirms exploitability.
Cases by status: 2 differs, 73 identical.
- Bandit’s examples/ directory (94 files), the snippets its own tests assert on: deliberately vulnerable, not real-world code.
- A gap measurement. Keryn does not implement Bandit’s checks; its default rules target AI, agent and supply-chain risks, so a low ratio is expected and is the point of the measurement.
- Bandit runs every built-in test (75) with default severity and confidence filters. Files Bandit cannot parse (3) are listed by Bandit as errors and produce no reference findings.
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
B308 |
differs |
43 |
41 |
41 |
2 |
|
B704 |
differs |
7 |
6 |
6 |
1 |
|
Show every case
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
B101 |
identical |
1 |
1 |
1 |
0 |
|
B102 |
identical |
1 |
1 |
1 |
0 |
|
B103 |
identical |
11 |
11 |
11 |
0 |
|
B104 |
identical |
1 |
1 |
1 |
0 |
|
B105 |
identical |
13 |
13 |
13 |
0 |
|
B106 |
identical |
2 |
2 |
2 |
0 |
|
B107 |
identical |
2 |
2 |
2 |
0 |
|
B108 |
identical |
4 |
4 |
4 |
0 |
|
B110 |
identical |
2 |
2 |
2 |
0 |
|
B112 |
identical |
2 |
2 |
2 |
0 |
|
B113 |
identical |
25 |
25 |
25 |
0 |
|
B201 |
identical |
1 |
1 |
1 |
0 |
|
B202 |
identical |
5 |
5 |
5 |
0 |
|
B301 |
identical |
13 |
13 |
13 |
0 |
|
B302 |
identical |
2 |
2 |
2 |
0 |
|
B303 |
identical |
11 |
11 |
11 |
0 |
|
B304 |
identical |
16 |
16 |
16 |
0 |
|
B305 |
identical |
1 |
1 |
1 |
0 |
|
B306 |
identical |
4 |
4 |
4 |
0 |
|
B307 |
identical |
3 |
3 |
3 |
0 |
|
B308 |
differs |
43 |
41 |
41 |
2 |
|
B310 |
identical |
8 |
8 |
8 |
0 |
|
B311 |
identical |
12 |
12 |
12 |
0 |
|
B312 |
identical |
1 |
1 |
1 |
0 |
|
B313 |
identical |
4 |
4 |
4 |
0 |
|
B314 |
identical |
4 |
4 |
4 |
0 |
|
B315 |
identical |
1 |
1 |
1 |
0 |
|
B316 |
identical |
2 |
2 |
2 |
0 |
|
B317 |
identical |
7 |
7 |
7 |
0 |
|
B318 |
identical |
2 |
2 |
2 |
0 |
|
B319 |
identical |
2 |
2 |
2 |
0 |
|
B321 |
identical |
1 |
1 |
1 |
0 |
|
B323 |
identical |
1 |
1 |
1 |
0 |
|
B324 |
identical |
24 |
24 |
24 |
0 |
|
B401 |
identical |
1 |
1 |
1 |
0 |
|
B402 |
identical |
2 |
2 |
2 |
0 |
|
B403 |
identical |
9 |
9 |
9 |
0 |
|
B404 |
identical |
14 |
14 |
14 |
0 |
|
B405 |
identical |
2 |
2 |
2 |
0 |
|
B406 |
identical |
3 |
3 |
3 |
0 |
|
B407 |
identical |
1 |
1 |
1 |
0 |
|
B408 |
identical |
2 |
2 |
2 |
0 |
|
B409 |
identical |
2 |
2 |
2 |
0 |
|
B411 |
identical |
1 |
1 |
1 |
0 |
|
B412 |
identical |
3 |
3 |
3 |
0 |
|
B413 |
identical |
16 |
16 |
16 |
0 |
|
B415 |
identical |
1 |
1 |
1 |
0 |
|
B501 |
identical |
18 |
18 |
18 |
0 |
|
B502 |
identical |
18 |
18 |
18 |
0 |
|
B503 |
identical |
4 |
4 |
4 |
0 |
|
B504 |
identical |
1 |
1 |
1 |
0 |
|
B505 |
identical |
14 |
14 |
14 |
0 |
|
B506 |
identical |
3 |
3 |
3 |
0 |
|
B507 |
identical |
8 |
8 |
8 |
0 |
|
B508 |
identical |
1 |
1 |
1 |
0 |
|
B509 |
identical |
2 |
2 |
2 |
0 |
|
B601 |
identical |
1 |
1 |
1 |
0 |
|
B602 |
identical |
28 |
28 |
28 |
0 |
|
B603 |
identical |
29 |
29 |
29 |
0 |
|
B604 |
identical |
1 |
1 |
1 |
0 |
|
B605 |
identical |
22 |
22 |
22 |
0 |
|
B606 |
identical |
20 |
20 |
20 |
0 |
|
B607 |
identical |
4 |
4 |
4 |
0 |
|
B608 |
identical |
49 |
49 |
49 |
0 |
|
B609 |
identical |
4 |
4 |
4 |
0 |
|
B610 |
identical |
11 |
11 |
11 |
0 |
|
B611 |
identical |
6 |
6 |
6 |
0 |
|
B612 |
identical |
1 |
1 |
1 |
0 |
|
B613 |
identical |
1 |
1 |
1 |
0 |
|
B614 |
identical |
3 |
3 |
3 |
0 |
|
B615 |
identical |
15 |
15 |
15 |
0 |
|
B701 |
identical |
5 |
5 |
5 |
0 |
|
B702 |
identical |
3 |
3 |
3 |
0 |
|
B703 |
identical |
26 |
26 |
26 |
0 |
|
B704 |
differs |
7 |
6 |
6 |
1 |
|