Given the same model outputs, do keryn-eval’s garak-derived detectors give the same score as garak’s own detector classes?
| Evaluation detail |
Description |
| Keryn engine |
keryn-eval detectors at a6f3d01bc37c: keryn-eval’s detectors (probes/detectors.rs), which reimplement garak detector logic over garak’s literal data. |
| Reference |
garak detectors 0.17.2.pre1. Oracle only: the pinned garak checkout’s detector classes, imported unmodified on a development machine; never shipped. |
| Corpus |
Fixed outputs covering string matching, repetition, citation boundaries, Unicode, empty text and score thresholds, plus literals loaded by the pinned detectors, NVIDIA/garak at 892779207cbc, restated from docs/adversarial-eval/results/garak-detectors.json |
| Comparison |
Per output: the detector score (0 to 1, or not scoreable). Equal to 1e-9 counts as identical; not scoreable equals only not scoreable. The verdict is a hit when the score is at least 0.5, garak’s default threshold. |
| Receipt |
garak.json, sha256 3fca186046a3a4dc |
| Measure |
Value |
Count |
Denominator |
| garak detectors with identical scores on every output |
100% |
34 / 34 |
garak detector classes keryn-eval mirrors |
| Outputs scored identically |
100% |
2,416 / 2,416 |
(detector, output) pairs in the corpus |
| Outputs with the same hit/pass verdict at 0.5 |
100% |
2,416 / 2,416 |
(detector, output) pairs in the corpus |
| garak detectors mirrored |
39.1% |
34 / 87 |
detector classes in the pinned garak checkout |
Cases by status: 34 identical.
- The oracle scores fixed text outputs offline. This suite measures detector behaviour only.
- The coverage denominator includes all active detector classes in the pinned garak checkout. Model-based classifiers and judges are outside the measured subset.
- Synthetic inputs exercise scoring rules and edge cases. Real-world detection rates require a labelled deployment corpus.
- Restated from the committed differential; numbers are not recomputed from a new run.
Show every case
| Detector |
Outputs |
Equal scores |
Equal verdicts |
Reference hits |
Keryn hits |
ansiescape.Escaped |
62 |
62 |
62 |
30 |
30 |
ansiescape.Raw |
54 |
54 |
54 |
19 |
19 |
any.AnyNonspaceOutput |
10 |
10 |
10 |
3 |
3 |
any.AnyOutput |
10 |
10 |
10 |
8 |
8 |
apikey.ApiKey |
26 |
26 |
26 |
11 |
11 |
continuation.Continuation |
26 |
26 |
26 |
10 |
10 |
dan.AntiDAN |
40 |
40 |
40 |
3 |
3 |
dan.DAN |
44 |
44 |
44 |
9 |
9 |
dan.DANJailbreak |
44 |
44 |
44 |
7 |
7 |
dan.MarkdownLink |
40 |
40 |
40 |
2 |
2 |
divergence.RepeatedToken |
21 |
21 |
21 |
11 |
11 |
encoding.DecodeApprox |
270 |
270 |
270 |
14 |
14 |
encoding.DecodeMatch |
270 |
270 |
270 |
9 |
9 |
goodside.Glitch |
51 |
51 |
51 |
25 |
25 |
goodside.PlainJSON |
38 |
38 |
38 |
20 |
20 |
goodside.RileyIsnt |
21 |
21 |
21 |
7 |
7 |
leakreplay.StartsWith |
48 |
48 |
48 |
18 |
18 |
lmrc.Anthro |
39 |
39 |
39 |
19 |
19 |
lmrc.QuackMedicine |
19 |
19 |
19 |
9 |
9 |
malwaregen.AnyCode |
34 |
34 |
34 |
18 |
18 |
misleading.MustRefuteClaimKW |
27 |
27 |
27 |
13 |
13 |
mitigation.MitigationBypass |
350 |
350 |
350 |
28 |
28 |
mitigation.Prefixes |
96 |
96 |
96 |
66 |
66 |
productkey.Win5x5 |
12 |
12 |
12 |
4 |
4 |
promptinject.AttackRogueString |
270 |
270 |
270 |
9 |
9 |
propile.PIILeak |
49 |
49 |
49 |
15 |
15 |
propile.PIILeakExact |
51 |
51 |
51 |
19 |
19 |
shields.Down |
30 |
30 |
30 |
17 |
17 |
shields.Up |
39 |
39 |
39 |
29 |
29 |
snowball.DontStartNo |
24 |
24 |
24 |
6 |
6 |
snowball.DontStartYes |
24 |
24 |
24 |
9 |
9 |
sysprompt_extraction.PromptExtraction |
153 |
153 |
153 |
13 |
13 |
web_injection.MarkdownExfilBasic |
38 |
38 |
38 |
4 |
4 |
web_injection.XSS |
86 |
86 |
86 |
49 |
49 |