# Promptfoo offline assertion parity

> Do the two graders return the same outcome and score on authored deterministic examples?

Source: https://keryn.dev/evals/promptfoo-structured/

Do the two graders return the same outcome and score on authored deterministic examples?

| Evaluation detail | Description |
|---|---|
| Keryn engine | Keryn Eval native config importer and replay engine at `3034ba83fd4d` |
| Reference | Promptfoo assertion API 0.125.1 |
| Corpus | Authored fixed outputs and deterministic assertions, restated from `docs/adversarial-eval/results/promptfoo-structured.json` |
| Comparison | Row status, pass and recorded reasons match exactly. Grade scores must differ by at most 1e-12. Matching errors and expected scope refusals remain in the denominator and are counted separately from matching grades. |
| Receipt | [`promptfoo-structured.json`](https://keryn.dev/evals/promptfoo-structured.json), sha256 `65b71d7233533be9` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Assertion types with matching outcomes on every row | **53.8%** | 7 / 13 | all offline assertion types compared |
| Rows with matching outcomes and scores | **90.5%** | 95 / 105 | all fixture rows, including errors |
| Rows with matching pass/fail grades and scores | **82.9%** | 87 / 105 | all fixture rows, including errors |
| Rows both graders reported as errors | **7.6%** | 8 / 105 | all fixture rows, including errors |
| Rows with expected native scope refusals | **9.5%** | 10 / 105 | all fixture rows |

Cases by status: 7 identical, 6 unsupported.

## Native BLEU and GLEU grading time

Both revisions matched 18 of 18 expected verdicts per batch on 18 fixed text pairs. The timing applies to this workload.

| Implementation | Revision | Median time per batch |
|---|---|---:|
| Baseline | `a6f3d01bc37c` | 0.332 ms |
| Candidate | `29a8087bcdc3` | 0.228 ms |

The baseline-to-candidate time ratio was **1.46×** on Linux x86_64 with Rust 1.94.0. Each median uses 15 samples of 4000 batches. [Download timing receipt](https://keryn.dev/evals/promptfoo-structured.perf.json).

## What this does and does not measure

- Offline fixture comparison with the recorded Promptfoo package and the native Keryn config/replay path.
- The historical 433-config Promptfoo differential is measured separately. This comparison makes no claim about live providers or broader corpus coverage.

## Differences and unassessed cases (6)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `is-sql` | unsupported |  |  |  |  |  |
| `is-xml` | unsupported |  |  |  |  |  |
| `meteor` | unsupported |  |  |  |  |  |
| `rouge-l` | unsupported |  |  |  |  |  |
| `rouge-n` | unsupported |  |  |  |  |  |
| `rouge-s` | unsupported |  |  |  |  |  |

## All cases (13)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `bleu` | identical |  |  |  |  |  |
| `contains-html` | identical |  |  |  |  |  |
| `contains-sql` | identical |  |  |  |  |  |
| `contains-xml` | identical |  |  |  |  |  |
| `gleu` | identical |  |  |  |  |  |
| `guardrails` | identical |  |  |  |  |  |
| `is-html` | identical |  |  |  |  |  |
| `is-sql` | unsupported |  |  |  |  |  |
| `is-xml` | unsupported |  |  |  |  |  |
| `meteor` | unsupported |  |  |  |  |  |
| `rouge-l` | unsupported |  |  |  |  |  |
| `rouge-n` | unsupported |  |  |  |  |  |
| `rouge-s` | unsupported |  |  |  |  |  |
