Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    garak detectors: keryn-eval vs garak 0.17.2.pre1

    View as Markdown

    Given the same model outputs, do keryn-eval’s garak-derived detectors give the same score as garak’s own detector classes?

    Evaluation detail Description
    Keryn engine keryn-eval detectors at a6f3d01bc37c: keryn-eval’s detectors (probes/detectors.rs), which reimplement garak detector logic over garak’s literal data.
    Reference garak detectors 0.17.2.pre1. Oracle only: the pinned garak checkout’s detector classes, imported unmodified on a development machine; never shipped.
    Corpus Fixed outputs covering string matching, repetition, citation boundaries, Unicode, empty text and score thresholds, plus literals loaded by the pinned detectors, NVIDIA/garak at 892779207cbc, restated from docs/adversarial-eval/results/garak-detectors.json
    Comparison Per output: the detector score (0 to 1, or not scoreable). Equal to 1e-9 counts as identical; not scoreable equals only not scoreable. The verdict is a hit when the score is at least 0.5, garak’s default threshold.
    Receipt garak.json, sha256 3fca186046a3a4dc
    Measure Value Count Denominator
    garak detectors with identical scores on every output 100% 34 / 34 garak detector classes keryn-eval mirrors
    Outputs scored identically 100% 2,416 / 2,416 (detector, output) pairs in the corpus
    Outputs with the same hit/pass verdict at 0.5 100% 2,416 / 2,416 (detector, output) pairs in the corpus
    garak detectors mirrored 39.1% 34 / 87 detector classes in the pinned garak checkout

    Cases by status: 34 identical.

    • The oracle scores fixed text outputs offline. This suite measures detector behaviour only.
    • The coverage denominator includes all active detector classes in the pinned garak checkout. Model-based classifiers and judges are outside the measured subset.
    • Synthetic inputs exercise scoring rules and edge cases. Real-world detection rates require a labelled deployment corpus.
    • Restated from the committed differential; numbers are not recomputed from a new run.
    Show every case
    Detector Outputs Equal scores Equal verdicts Reference hits Keryn hits
    ansiescape.Escaped 62 62 62 30 30
    ansiescape.Raw 54 54 54 19 19
    any.AnyNonspaceOutput 10 10 10 3 3
    any.AnyOutput 10 10 10 8 8
    apikey.ApiKey 26 26 26 11 11
    continuation.Continuation 26 26 26 10 10
    dan.AntiDAN 40 40 40 3 3
    dan.DAN 44 44 44 9 9
    dan.DANJailbreak 44 44 44 7 7
    dan.MarkdownLink 40 40 40 2 2
    divergence.RepeatedToken 21 21 21 11 11
    encoding.DecodeApprox 270 270 270 14 14
    encoding.DecodeMatch 270 270 270 9 9
    goodside.Glitch 51 51 51 25 25
    goodside.PlainJSON 38 38 38 20 20
    goodside.RileyIsnt 21 21 21 7 7
    leakreplay.StartsWith 48 48 48 18 18
    lmrc.Anthro 39 39 39 19 19
    lmrc.QuackMedicine 19 19 19 9 9
    malwaregen.AnyCode 34 34 34 18 18
    misleading.MustRefuteClaimKW 27 27 27 13 13
    mitigation.MitigationBypass 350 350 350 28 28
    mitigation.Prefixes 96 96 96 66 66
    productkey.Win5x5 12 12 12 4 4
    promptinject.AttackRogueString 270 270 270 9 9
    propile.PIILeak 49 49 49 15 15
    propile.PIILeakExact 51 51 51 19 19
    shields.Down 30 30 30 17 17
    shields.Up 39 39 39 29 29
    snowball.DontStartNo 24 24 24 6 6
    snowball.DontStartYes 24 24 24 9 9
    sysprompt_extraction.PromptExtraction 153 153 153 13 13
    web_injection.MarkdownExfilBasic 38 38 38 4 4
    web_injection.XSS 86 86 86 49 49