# garak detectors: keryn-eval vs garak 0.17.2.pre1

> Given the same model outputs, do keryn-eval's garak-derived detectors give the same score as garak's own detector classes?

Source: https://keryn.dev/evals/garak/

Given the same model outputs, do keryn-eval's garak-derived detectors give the same score as garak's own detector classes?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-eval detectors at `a6f3d01bc37c`: keryn-eval's detectors (probes/detectors.rs), which reimplement garak detector logic over garak's literal data. |
| Reference | garak detectors 0.17.2.pre1. Oracle only: the pinned garak checkout's detector classes, imported unmodified on a development machine; never shipped. |
| Corpus | Fixed outputs covering string matching, repetition, citation boundaries, Unicode, empty text and score thresholds, plus literals loaded by the pinned detectors, [NVIDIA/garak](https://github.com/NVIDIA/garak/tree/892779207cbc38f65a738ef9cbdfbc1e83aa0b55) at `892779207cbc`, restated from `docs/adversarial-eval/results/garak-detectors.json` |
| Comparison | Per output: the detector score (0 to 1, or not scoreable). Equal to 1e-9 counts as identical; not scoreable equals only not scoreable. The verdict is a hit when the score is at least 0.5, garak's default threshold. |
| Receipt | [`garak.json`](https://keryn.dev/evals/garak.json), sha256 `3fca186046a3a4dc` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| garak detectors with identical scores on every output | **100%** | 34 / 34 | garak detector classes keryn-eval mirrors |
| Outputs scored identically | **100%** | 2,416 / 2,416 | (detector, output) pairs in the corpus |
| Outputs with the same hit/pass verdict at 0.5 | **100%** | 2,416 / 2,416 | (detector, output) pairs in the corpus |
| garak detectors mirrored | **39.1%** | 34 / 87 | detector classes in the pinned garak checkout |

Cases by status: 34 identical.

## What this does and does not measure

- The oracle scores fixed text outputs offline. This suite measures detector behaviour only.
- The coverage denominator includes all active detector classes in the pinned garak checkout. Model-based classifiers and judges are outside the measured subset.
- Synthetic inputs exercise scoring rules and edge cases. Real-world detection rates require a labelled deployment corpus.
- Restated from the committed differential; numbers are not recomputed from a new run.

## All cases (34)

**Show every case**

| Detector | Outputs | Equal scores | Equal verdicts | Reference hits | Keryn hits |
|---|---:|---:|---:|---:|---:|
| `ansiescape.Escaped` | 62 | 62 | 62 | 30 | 30 |
| `ansiescape.Raw` | 54 | 54 | 54 | 19 | 19 |
| `any.AnyNonspaceOutput` | 10 | 10 | 10 | 3 | 3 |
| `any.AnyOutput` | 10 | 10 | 10 | 8 | 8 |
| `apikey.ApiKey` | 26 | 26 | 26 | 11 | 11 |
| `continuation.Continuation` | 26 | 26 | 26 | 10 | 10 |
| `dan.AntiDAN` | 40 | 40 | 40 | 3 | 3 |
| `dan.DAN` | 44 | 44 | 44 | 9 | 9 |
| `dan.DANJailbreak` | 44 | 44 | 44 | 7 | 7 |
| `dan.MarkdownLink` | 40 | 40 | 40 | 2 | 2 |
| `divergence.RepeatedToken` | 21 | 21 | 21 | 11 | 11 |
| `encoding.DecodeApprox` | 270 | 270 | 270 | 14 | 14 |
| `encoding.DecodeMatch` | 270 | 270 | 270 | 9 | 9 |
| `goodside.Glitch` | 51 | 51 | 51 | 25 | 25 |
| `goodside.PlainJSON` | 38 | 38 | 38 | 20 | 20 |
| `goodside.RileyIsnt` | 21 | 21 | 21 | 7 | 7 |
| `leakreplay.StartsWith` | 48 | 48 | 48 | 18 | 18 |
| `lmrc.Anthro` | 39 | 39 | 39 | 19 | 19 |
| `lmrc.QuackMedicine` | 19 | 19 | 19 | 9 | 9 |
| `malwaregen.AnyCode` | 34 | 34 | 34 | 18 | 18 |
| `misleading.MustRefuteClaimKW` | 27 | 27 | 27 | 13 | 13 |
| `mitigation.MitigationBypass` | 350 | 350 | 350 | 28 | 28 |
| `mitigation.Prefixes` | 96 | 96 | 96 | 66 | 66 |
| `productkey.Win5x5` | 12 | 12 | 12 | 4 | 4 |
| `promptinject.AttackRogueString` | 270 | 270 | 270 | 9 | 9 |
| `propile.PIILeak` | 49 | 49 | 49 | 15 | 15 |
| `propile.PIILeakExact` | 51 | 51 | 51 | 19 | 19 |
| `shields.Down` | 30 | 30 | 30 | 17 | 17 |
| `shields.Up` | 39 | 39 | 39 | 29 | 29 |
| `snowball.DontStartNo` | 24 | 24 | 24 | 6 | 6 |
| `snowball.DontStartYes` | 24 | 24 | 24 | 9 | 9 |
| `sysprompt_extraction.PromptExtraction` | 153 | 153 | 153 | 13 | 13 |
| `web_injection.MarkdownExfilBasic` | 38 | 38 | 38 | 4 | 4 |
| `web_injection.XSS` | 86 | 86 | 86 | 49 | 49 |
