# Model-file inspection (pickle, PyTorch, joblib, safetensors, GGUF, ONNX)

> On authored benign and unsafe model files, which unsafe files does Keryn flag, which benign ones does it pass, and where does it differ from picklescan and ModelScan?

Source: https://keryn.dev/evals/model-files/

On authored benign and unsafe model files, which unsafe files does Keryn flag, which benign ones does it pass, and where does it differ from picklescan and ModelScan?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn artifact analyze (model readers) at `04ad1d5c1cb0`: The artifact analyzer's model readers (crates/artifact/crates/artifact-core/src/models) through the CLI, WebAssembly build; nothing is loaded or deserialised. |
| Reference | picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed). Behaviour references, not truth: labels decide right and wrong. Reference verdicts are in suites/model-files/reference.json (tools/evals/model-files-reference.mjs). |
| Corpus | 45 byte-level recipes: pickle protocols 0-5, PyTorch zip, joblib, NumPy npy/npz, safetensors, GGUF and ONNX (24 unsafe, 21 benign) |
| Comparison | model-files/1: Keryn's safe_to_load from keryn artifact analyze (no -> unsafe, yes -> benign, otherwise not-assessed) against the authored label; each reference's normalised verdict beside it. A reference that is out of scope, did not scan the file or errored decides nothing and is counted apart. |
| Receipt | [`model-files.json`](https://keryn.dev/evals/model-files.json), sha256 `4740435b49231724` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Cases where Keryn's verdict equals the authored label | **73.3%** | 33 / 45 | all corpus cases, not-assessed included |
| Unsafe files Keryn flags (safe_to_load no) | **100%** | 24 / 24 | cases labelled unsafe, including those Keryn did not assess |
| Benign files Keryn flags (false alarms) | **57.1%** | 12 / 21 | cases labelled benign, including those Keryn did not assess |
| Benign files Keryn calls safe to load | **42.9%** | 9 / 21 | cases labelled benign |
| Unsafe files Keryn detects (an error-level finding that fails the gate) | **100%** | 24 / 24 | cases labelled unsafe |
| Benign files with an error-level finding | **4.8%** | 1 / 21 | cases labelled benign |
| Files Keryn did not assess | **0%** | 0 / 45 | all corpus cases |
| Keryn agrees with modelscan | **68.8%** | 22 / 32 | cases modelscan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in modelscan-decided |
| Cases modelscan decided | **71.1%** | 32 / 45 | all corpus cases |
| Unsafe files modelscan flags | **83.3%** | 20 / 24 | cases labelled unsafe, out-of-scope included |
| Keryn agrees with picklescan | **66.7%** | 22 / 33 | cases picklescan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in picklescan-decided |
| Cases picklescan decided | **73.3%** | 33 / 45 | all corpus cases |
| Unsafe files picklescan flags | **83.3%** | 20 / 24 | cases labelled unsafe, out-of-scope included |

Cases by status: 12 differs, 33 identical.

## What this does and does not measure

- Static header and opcode reads only; no file is loaded.
- Synthetic files: these are regression rates on a small authored corpus, not real-world detection rates.
- Unsafe means loading in the usual loader can execute code or read outside the model directory (pickle globals, Jinja chat-template gadgets, ONNX external data paths).

## Differences and unassessed cases (12)

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `joblib-plain` | differs | 2 |  |  |  |  |
| `onnx-custom-domain` | differs | 2 |  |  |  |  |
| `pickle-p0-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p1-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p2-ordereddict` | differs | 2 |  |  |  |  |
| `pickle-p2-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p3-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p4-ordereddict` | differs | 2 |  |  |  |  |
| `pickle-p4-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p5-plain-dict` | differs | 2 |  |  |  |  |
| `pytorch-zip-ordereddict` | differs | 2 |  |  |  |  |
| `safetensors-offsets-past-end` | differs | 2 |  |  |  |  |

## All cases (45)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
| `gguf-no-template` | identical | 2 |  |  |  |  |
| `gguf-plain-template` | identical | 2 |  |  |  |  |
| `gguf-ssti-template` | identical | 2 |  |  |  |  |
| `joblib-os-system` | identical | 2 |  |  |  |  |
| `joblib-plain` | differs | 2 |  |  |  |  |
| `joblib-zlib-os-system` | identical | 2 |  |  |  |  |
| `numpy-npy-numeric` | identical | 2 |  |  |  |  |
| `numpy-npy-object-os-system` | identical | 2 |  |  |  |  |
| `numpy-npz-compressed-numeric` | identical | 2 |  |  |  |  |
| `numpy-npz-stored-object` | identical | 2 |  |  |  |  |
| `onnx-custom-domain` | differs | 2 |  |  |  |  |
| `onnx-external-absolute` | identical | 2 |  |  |  |  |
| `onnx-external-sibling` | identical | 2 |  |  |  |  |
| `onnx-external-traversal` | identical | 2 |  |  |  |  |
| `onnx-plain` | identical | 2 |  |  |  |  |
| `pickle-p0-inst-os-system` | identical | 2 |  |  |  |  |
| `pickle-p0-os-system` | identical | 2 |  |  |  |  |
| `pickle-p0-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p1-os-system` | identical | 2 |  |  |  |  |
| `pickle-p1-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p2-getattr-chain` | identical | 2 |  |  |  |  |
| `pickle-p2-ordereddict` | differs | 2 |  |  |  |  |
| `pickle-p2-os-system` | identical | 2 |  |  |  |  |
| `pickle-p2-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p2-posix-system` | identical | 2 |  |  |  |  |
| `pickle-p2-webbrowser` | identical | 2 |  |  |  |  |
| `pickle-p3-builtins-eval` | identical | 2 |  |  |  |  |
| `pickle-p3-os-system` | identical | 2 |  |  |  |  |
| `pickle-p3-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p4-builtins-exec` | identical | 2 |  |  |  |  |
| `pickle-p4-ordereddict` | differs | 2 |  |  |  |  |
| `pickle-p4-os-system` | identical | 2 |  |  |  |  |
| `pickle-p4-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p4-runpy` | identical | 2 |  |  |  |  |
| `pickle-p5-os-system` | identical | 2 |  |  |  |  |
| `pickle-p5-plain-dict` | differs | 2 |  |  |  |  |
| `pickle-p5-subprocess` | identical | 2 |  |  |  |  |
| `pytorch-zip-deflated-system` | identical | 2 |  |  |  |  |
| `pytorch-zip-ordereddict` | differs | 2 |  |  |  |  |
| `pytorch-zip-os-system` | identical | 2 |  |  |  |  |
| `pytorch-zip-p4-exec` | identical | 2 |  |  |  |  |
| `safetensors-empty` | identical | 2 |  |  |  |  |
| `safetensors-metadata` | identical | 2 |  |  |  |  |
| `safetensors-offsets-past-end` | differs | 2 |  |  |  |  |
| `safetensors-plain` | identical | 2 |  |  |  |  |
