Do the two graders return the same outcome and score on authored deterministic examples?
| Evaluation detail |
Description |
| Keryn engine |
Keryn Eval native config importer and replay engine at 3034ba83fd4d |
| Reference |
Promptfoo assertion API 0.125.1 |
| Corpus |
Authored fixed outputs and deterministic assertions, restated from docs/adversarial-eval/results/promptfoo-structured.json |
| Comparison |
Row status, pass and recorded reasons match exactly. Grade scores must differ by at most 1e-12. Matching errors and expected scope refusals remain in the denominator and are counted separately from matching grades. |
| Receipt |
promptfoo-structured.json, sha256 65b71d7233533be9 |
| Measure |
Value |
Count |
Denominator |
| Assertion types with matching outcomes on every row |
53.8% |
7 / 13 |
all offline assertion types compared |
| Rows with matching outcomes and scores |
90.5% |
95 / 105 |
all fixture rows, including errors |
| Rows with matching pass/fail grades and scores |
82.9% |
87 / 105 |
all fixture rows, including errors |
| Rows both graders reported as errors |
7.6% |
8 / 105 |
all fixture rows, including errors |
| Rows with expected native scope refusals |
9.5% |
10 / 105 |
all fixture rows |
Cases by status: 7 identical, 6 unsupported.
Both revisions matched 18 of 18 expected verdicts per batch on 18 fixed text pairs. The timing applies to this workload.
| Implementation |
Revision |
Median time per batch |
| Baseline |
a6f3d01bc37c |
0.332 ms |
| Candidate |
29a8087bcdc3 |
0.228 ms |
The baseline-to-candidate time ratio was 1.46× on Linux x86_64 with Rust 1.94.0. Each median uses 15 samples of 4000 batches. Download timing receipt.
- Offline fixture comparison with the recorded Promptfoo package and the native Keryn config/replay path.
- The historical 433-config Promptfoo differential is measured separately. This comparison makes no claim about live providers or broader corpus coverage.
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
is-sql |
unsupported |
|
|
|
|
|
is-xml |
unsupported |
|
|
|
|
|
meteor |
unsupported |
|
|
|
|
|
rouge-l |
unsupported |
|
|
|
|
|
rouge-n |
unsupported |
|
|
|
|
|
rouge-s |
unsupported |
|
|
|
|
|
Show every case
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
bleu |
identical |
|
|
|
|
|
contains-html |
identical |
|
|
|
|
|
contains-sql |
identical |
|
|
|
|
|
contains-xml |
identical |
|
|
|
|
|
gleu |
identical |
|
|
|
|
|
guardrails |
identical |
|
|
|
|
|
is-html |
identical |
|
|
|
|
|
is-sql |
unsupported |
|
|
|
|
|
is-xml |
unsupported |
|
|
|
|
|
meteor |
unsupported |
|
|
|
|
|
rouge-l |
unsupported |
|
|
|
|
|
rouge-n |
unsupported |
|
|
|
|
|
rouge-s |
unsupported |
|
|
|
|
|