On authored benign and unsafe model files, which unsafe files does Keryn flag, which benign ones does it pass, and where does it differ from picklescan and ModelScan?
| Evaluation detail |
Description |
| Keryn engine |
keryn artifact analyze (model readers) at 04ad1d5c1cb0: The artifact analyzer’s model readers (crates/artifact/crates/artifact-core/src/models) through the CLI, WebAssembly build; nothing is loaded or deserialised. |
| Reference |
picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed). Behaviour references, not truth: labels decide right and wrong. Reference verdicts are in suites/model-files/reference.json (tools/evals/model-files-reference.mjs). |
| Corpus |
45 byte-level recipes: pickle protocols 0-5, PyTorch zip, joblib, NumPy npy/npz, safetensors, GGUF and ONNX (24 unsafe, 21 benign) |
| Comparison |
model-files/1: Keryn’s safe_to_load from keryn artifact analyze (no -> unsafe, yes -> benign, otherwise not-assessed) against the authored label; each reference’s normalised verdict beside it. A reference that is out of scope, did not scan the file or errored decides nothing and is counted apart. |
| Receipt |
model-files.json, sha256 4740435b49231724 |
| Measure |
Value |
Count |
Denominator |
| Cases where Keryn’s verdict equals the authored label |
73.3% |
33 / 45 |
all corpus cases, not-assessed included |
| Unsafe files Keryn flags (safe_to_load no) |
100% |
24 / 24 |
cases labelled unsafe, including those Keryn did not assess |
| Benign files Keryn flags (false alarms) |
57.1% |
12 / 21 |
cases labelled benign, including those Keryn did not assess |
| Benign files Keryn calls safe to load |
42.9% |
9 / 21 |
cases labelled benign |
| Unsafe files Keryn detects (an error-level finding that fails the gate) |
100% |
24 / 24 |
cases labelled unsafe |
| Benign files with an error-level finding |
4.8% |
1 / 21 |
cases labelled benign |
| Files Keryn did not assess |
0% |
0 / 45 |
all corpus cases |
| Keryn agrees with modelscan |
68.8% |
22 / 32 |
cases modelscan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in modelscan-decided |
| Cases modelscan decided |
71.1% |
32 / 45 |
all corpus cases |
| Unsafe files modelscan flags |
83.3% |
20 / 24 |
cases labelled unsafe, out-of-scope included |
| Keryn agrees with picklescan |
66.7% |
22 / 33 |
cases picklescan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in picklescan-decided |
| Cases picklescan decided |
73.3% |
33 / 45 |
all corpus cases |
| Unsafe files picklescan flags |
83.3% |
20 / 24 |
cases labelled unsafe, out-of-scope included |
Cases by status: 12 differs, 33 identical.
- Static header and opcode reads only; no file is loaded.
- Synthetic files: these are regression rates on a small authored corpus, not real-world detection rates.
- Unsafe means loading in the usual loader can execute code or read outside the model directory (pickle globals, Jinja chat-template gadgets, ONNX external data paths).
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
joblib-plain |
differs |
2 |
|
|
|
|
onnx-custom-domain |
differs |
2 |
|
|
|
|
pickle-p0-plain-dict |
differs |
2 |
|
|
|
|
pickle-p1-plain-dict |
differs |
2 |
|
|
|
|
pickle-p2-ordereddict |
differs |
2 |
|
|
|
|
pickle-p2-plain-dict |
differs |
2 |
|
|
|
|
pickle-p3-plain-dict |
differs |
2 |
|
|
|
|
pickle-p4-ordereddict |
differs |
2 |
|
|
|
|
pickle-p4-plain-dict |
differs |
2 |
|
|
|
|
pickle-p5-plain-dict |
differs |
2 |
|
|
|
|
pytorch-zip-ordereddict |
differs |
2 |
|
|
|
|
safetensors-offsets-past-end |
differs |
2 |
|
|
|
|
Show every case
| Case |
Status |
Reference rows |
Keryn rows |
Matching |
Missing |
Extra |
gguf-no-template |
identical |
2 |
|
|
|
|
gguf-plain-template |
identical |
2 |
|
|
|
|
gguf-ssti-template |
identical |
2 |
|
|
|
|
joblib-os-system |
identical |
2 |
|
|
|
|
joblib-plain |
differs |
2 |
|
|
|
|
joblib-zlib-os-system |
identical |
2 |
|
|
|
|
numpy-npy-numeric |
identical |
2 |
|
|
|
|
numpy-npy-object-os-system |
identical |
2 |
|
|
|
|
numpy-npz-compressed-numeric |
identical |
2 |
|
|
|
|
numpy-npz-stored-object |
identical |
2 |
|
|
|
|
onnx-custom-domain |
differs |
2 |
|
|
|
|
onnx-external-absolute |
identical |
2 |
|
|
|
|
onnx-external-sibling |
identical |
2 |
|
|
|
|
onnx-external-traversal |
identical |
2 |
|
|
|
|
onnx-plain |
identical |
2 |
|
|
|
|
pickle-p0-inst-os-system |
identical |
2 |
|
|
|
|
pickle-p0-os-system |
identical |
2 |
|
|
|
|
pickle-p0-plain-dict |
differs |
2 |
|
|
|
|
pickle-p1-os-system |
identical |
2 |
|
|
|
|
pickle-p1-plain-dict |
differs |
2 |
|
|
|
|
pickle-p2-getattr-chain |
identical |
2 |
|
|
|
|
pickle-p2-ordereddict |
differs |
2 |
|
|
|
|
pickle-p2-os-system |
identical |
2 |
|
|
|
|
pickle-p2-plain-dict |
differs |
2 |
|
|
|
|
pickle-p2-posix-system |
identical |
2 |
|
|
|
|
pickle-p2-webbrowser |
identical |
2 |
|
|
|
|
pickle-p3-builtins-eval |
identical |
2 |
|
|
|
|
pickle-p3-os-system |
identical |
2 |
|
|
|
|
pickle-p3-plain-dict |
differs |
2 |
|
|
|
|
pickle-p4-builtins-exec |
identical |
2 |
|
|
|
|
pickle-p4-ordereddict |
differs |
2 |
|
|
|
|
pickle-p4-os-system |
identical |
2 |
|
|
|
|
pickle-p4-plain-dict |
differs |
2 |
|
|
|
|
pickle-p4-runpy |
identical |
2 |
|
|
|
|
pickle-p5-os-system |
identical |
2 |
|
|
|
|
pickle-p5-plain-dict |
differs |
2 |
|
|
|
|
pickle-p5-subprocess |
identical |
2 |
|
|
|
|
pytorch-zip-deflated-system |
identical |
2 |
|
|
|
|
pytorch-zip-ordereddict |
differs |
2 |
|
|
|
|
pytorch-zip-os-system |
identical |
2 |
|
|
|
|
pytorch-zip-p4-exec |
identical |
2 |
|
|
|
|
safetensors-empty |
identical |
2 |
|
|
|
|
safetensors-metadata |
identical |
2 |
|
|
|
|
safetensors-offsets-past-end |
differs |
2 |
|
|
|
|
safetensors-plain |
identical |
2 |
|
|
|
|