Which authored policy checks match the pinned passive scanners on the same inputs?
| Evaluation detail |
Description |
| Keryn engine |
Keryn native passive text policy scanners at a7d96a424d07 |
| Reference |
LLM Guard read-only scanner methods |
| Corpus |
Authored fixed text and recorded policy settings, restated from evals/suites/llm-guard-passive/summary.json |
| Comparison |
Exact cleaned-text digest, validity and risk, with matched error classes counted separately. Native scope refusals remain in the denominator. |
| Receipt |
llm-guard-passive.json, sha256 1db0cc203388fb39 |
| Measure |
Value |
Count |
Denominator |
| Scanner groups with exact agreement on every row |
37.5% |
3 / 8 |
all scanner groups tested |
| Rows with exact outcome agreement |
95.0% |
133 / 140 |
all recorded rows including errors and native scope refusals |
| Rows with matching error outcomes |
7.1% |
10 / 140 |
all recorded rows |
| Rows refused by native scope |
5.0% |
7 / 140 |
all recorded rows |
Cases by status: 3 identical, 5 unsupported.
- Fixture agreement checks the measured scanner contracts and preserves explicit native scope refusals.
- Other scanner families and production accuracy remain outside this comparison. No live model or external target was called.
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
input.BanSubstrings |
unsupported |
|
|
|
|
|
input.Regex |
unsupported |
|
|
|
|
|
output.BanSubstrings |
unsupported |
|
|
|
|
|
output.JSON |
unsupported |
|
|
|
|
|
output.Regex |
unsupported |
|
|
|
|
|
Show every case
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
input.BanSubstrings |
unsupported |
|
|
|
|
|
input.Regex |
unsupported |
|
|
|
|
|
output.BanSubstrings |
unsupported |
|
|
|
|
|
output.Deanonymize |
identical |
|
|
|
|
|
output.JSON |
unsupported |
|
|
|
|
|
output.NoRefusalLight |
identical |
|
|
|
|
|
output.ReadingTime |
identical |
|
|
|
|
|
output.Regex |
unsupported |
|
|
|
|
|