Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    LLM Guard passive scanner parity

    View as Markdown

    Which authored policy checks match the pinned passive scanners on the same inputs?

    Evaluation detail Description
    Keryn engine Keryn native passive text policy scanners at a7d96a424d07
    Reference LLM Guard read-only scanner methods
    Corpus Authored fixed text and recorded policy settings, restated from evals/suites/llm-guard-passive/summary.json
    Comparison Exact cleaned-text digest, validity and risk, with matched error classes counted separately. Native scope refusals remain in the denominator.
    Receipt llm-guard-passive.json, sha256 1db0cc203388fb39
    Measure Value Count Denominator
    Scanner groups with exact agreement on every row 37.5% 3 / 8 all scanner groups tested
    Rows with exact outcome agreement 95.0% 133 / 140 all recorded rows including errors and native scope refusals
    Rows with matching error outcomes 7.1% 10 / 140 all recorded rows
    Rows refused by native scope 5.0% 7 / 140 all recorded rows

    Cases by status: 3 identical, 5 unsupported.

    • Fixture agreement checks the measured scanner contracts and preserves explicit native scope refusals.
    • Other scanner families and production accuracy remain outside this comparison. No live model or external target was called.
    Case Status Reference rows Keryn rows Matching Missing Extra
    input.BanSubstrings unsupported
    input.Regex unsupported
    output.BanSubstrings unsupported
    output.JSON unsupported
    output.Regex unsupported
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    input.BanSubstrings unsupported
    input.Regex unsupported
    output.BanSubstrings unsupported
    output.Deanonymize identical
    output.JSON unsupported
    output.NoRefusalLight identical
    output.ReadingTime identical
    output.Regex unsupported