Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Promptfoo offline assertion parity

    View as Markdown

    Do the two graders return the same outcome and score on authored deterministic examples?

    Evaluation detail Description
    Keryn engine Keryn Eval native config importer and replay engine at 3034ba83fd4d
    Reference Promptfoo assertion API 0.125.1
    Corpus Authored fixed outputs and deterministic assertions, restated from docs/adversarial-eval/results/promptfoo-structured.json
    Comparison Row status, pass and recorded reasons match exactly. Grade scores must differ by at most 1e-12. Matching errors and expected scope refusals remain in the denominator and are counted separately from matching grades.
    Receipt promptfoo-structured.json, sha256 65b71d7233533be9
    Measure Value Count Denominator
    Assertion types with matching outcomes on every row 53.8% 7 / 13 all offline assertion types compared
    Rows with matching outcomes and scores 90.5% 95 / 105 all fixture rows, including errors
    Rows with matching pass/fail grades and scores 82.9% 87 / 105 all fixture rows, including errors
    Rows both graders reported as errors 7.6% 8 / 105 all fixture rows, including errors
    Rows with expected native scope refusals 9.5% 10 / 105 all fixture rows

    Cases by status: 7 identical, 6 unsupported.

    Both revisions matched 18 of 18 expected verdicts per batch on 18 fixed text pairs. The timing applies to this workload.

    Implementation Revision Median time per batch
    Baseline a6f3d01bc37c 0.332 ms
    Candidate 29a8087bcdc3 0.228 ms

    The baseline-to-candidate time ratio was 1.46× on Linux x86_64 with Rust 1.94.0. Each median uses 15 samples of 4000 batches. Download timing receipt.

    • Offline fixture comparison with the recorded Promptfoo package and the native Keryn config/replay path.
    • The historical 433-config Promptfoo differential is measured separately. This comparison makes no claim about live providers or broader corpus coverage.
    Case Status Reference rows Keryn rows Matching Missing Extra
    is-sql unsupported
    is-xml unsupported
    meteor unsupported
    rouge-l unsupported
    rouge-n unsupported
    rouge-s unsupported
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    bleu identical
    contains-html identical
    contains-sql identical
    contains-xml identical
    gleu identical
    guardrails identical
    is-html identical
    is-sql unsupported
    is-xml unsupported
    meteor unsupported
    rouge-l unsupported
    rouge-n unsupported
    rouge-s unsupported