# LLM Guard passive scanner parity

> Which authored policy checks match the pinned passive scanners on the same inputs?

Source: https://keryn.dev/evals/llm-guard-passive/

Which authored policy checks match the pinned passive scanners on the same inputs?

| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn native passive text policy scanners at `a7d96a424d07` |
| Reference | LLM Guard read-only scanner methods  |
| Corpus | Authored fixed text and recorded policy settings, restated from `evals/suites/llm-guard-passive/summary.json` |
| Comparison | Exact cleaned-text digest, validity and risk, with matched error classes counted separately. Native scope refusals remain in the denominator. |
| Receipt | [`llm-guard-passive.json`](https://keryn.dev/evals/llm-guard-passive.json), sha256 `1db0cc203388fb39` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Scanner groups with exact agreement on every row | **37.5%** | 3 / 8 | all scanner groups tested |
| Rows with exact outcome agreement | **95.0%** | 133 / 140 | all recorded rows including errors and native scope refusals |
| Rows with matching error outcomes | **7.1%** | 10 / 140 | all recorded rows |
| Rows refused by native scope | **5.0%** | 7 / 140 | all recorded rows |

Cases by status: 3 identical, 5 unsupported.

## What this does and does not measure

- Fixture agreement checks the measured scanner contracts and preserves explicit native scope refusals.
- Other scanner families and production accuracy remain outside this comparison. No live model or external target was called.

## Differences and unassessed cases (5)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `input.BanSubstrings` | unsupported |  |  |  |  |  |
| `input.Regex` | unsupported |  |  |  |  |  |
| `output.BanSubstrings` | unsupported |  |  |  |  |  |
| `output.JSON` | unsupported |  |  |  |  |  |
| `output.Regex` | unsupported |  |  |  |  |  |

## All cases (8)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `input.BanSubstrings` | unsupported |  |  |  |  |  |
| `input.Regex` | unsupported |  |  |  |  |  |
| `output.BanSubstrings` | unsupported |  |  |  |  |  |
| `output.Deanonymize` | identical |  |  |  |  |  |
| `output.JSON` | unsupported |  |  |  |  |  |
| `output.NoRefusalLight` | identical |  |  |  |  |  |
| `output.ReadingTime` | identical |  |  |  |  |  |
| `output.Regex` | unsupported |  |  |  |  |  |
