Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Model-file inspection (pickle, PyTorch, joblib, safetensors, GGUF, ONNX)

    View as Markdown

    On authored benign and unsafe model files, which unsafe files does Keryn flag, which benign ones does it pass, and where does it differ from picklescan and ModelScan?

    Evaluation detail Description
    Keryn engine keryn artifact analyze (model readers) at 04ad1d5c1cb0: The artifact analyzer’s model readers (crates/artifact/crates/artifact-core/src/models) through the CLI, WebAssembly build; nothing is loaded or deserialised.
    Reference picklescan 1.0.5 and ModelScan 0.8.8 with numpy 1.26.4 (pip, development machine only; never shipped) picklescan 1.0.5; modelscan 0.8.8 (numpy 1.26.4: ModelScan 0.8.8 calls numpy.lib.format._check_version, which numpy 2 removed). Behaviour references, not truth: labels decide right and wrong. Reference verdicts are in suites/model-files/reference.json (tools/evals/model-files-reference.mjs).
    Corpus 45 byte-level recipes: pickle protocols 0-5, PyTorch zip, joblib, NumPy npy/npz, safetensors, GGUF and ONNX (24 unsafe, 21 benign)
    Comparison model-files/1: Keryn’s safe_to_load from keryn artifact analyze (no -> unsafe, yes -> benign, otherwise not-assessed) against the authored label; each reference’s normalised verdict beside it. A reference that is out of scope, did not scan the file or errored decides nothing and is counted apart.
    Receipt model-files.json, sha256 4740435b49231724
    Measure Value Count Denominator
    Cases where Keryn’s verdict equals the authored label 73.3% 33 / 45 all corpus cases, not-assessed included
    Unsafe files Keryn flags (safe_to_load no) 100% 24 / 24 cases labelled unsafe, including those Keryn did not assess
    Benign files Keryn flags (false alarms) 57.1% 12 / 21 cases labelled benign, including those Keryn did not assess
    Benign files Keryn calls safe to load 42.9% 9 / 21 cases labelled benign
    Unsafe files Keryn detects (an error-level finding that fails the gate) 100% 24 / 24 cases labelled unsafe
    Benign files with an error-level finding 4.8% 1 / 21 cases labelled benign
    Files Keryn did not assess 0% 0 / 45 all corpus cases
    Keryn agrees with modelscan 68.8% 22 / 32 cases modelscan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in modelscan-decided
    Cases modelscan decided 71.1% 32 / 45 all corpus cases
    Unsafe files modelscan flags 83.3% 20 / 24 cases labelled unsafe, out-of-scope included
    Keryn agrees with picklescan 66.7% 22 / 33 cases picklescan decided (unsafe or benign); out-of-scope, unread and error cases excluded and counted in picklescan-decided
    Cases picklescan decided 73.3% 33 / 45 all corpus cases
    Unsafe files picklescan flags 83.3% 20 / 24 cases labelled unsafe, out-of-scope included

    Cases by status: 12 differs, 33 identical.

    • Static header and opcode reads only; no file is loaded.
    • Synthetic files: these are regression rates on a small authored corpus, not real-world detection rates.
    • Unsafe means loading in the usual loader can execute code or read outside the model directory (pickle globals, Jinja chat-template gadgets, ONNX external data paths).
    Case Status Reference rows Keryn rows Matching Missing Extra
    joblib-plain differs 2
    onnx-custom-domain differs 2
    pickle-p0-plain-dict differs 2
    pickle-p1-plain-dict differs 2
    pickle-p2-ordereddict differs 2
    pickle-p2-plain-dict differs 2
    pickle-p3-plain-dict differs 2
    pickle-p4-ordereddict differs 2
    pickle-p4-plain-dict differs 2
    pickle-p5-plain-dict differs 2
    pytorch-zip-ordereddict differs 2
    safetensors-offsets-past-end differs 2
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    gguf-no-template identical 2
    gguf-plain-template identical 2
    gguf-ssti-template identical 2
    joblib-os-system identical 2
    joblib-plain differs 2
    joblib-zlib-os-system identical 2
    numpy-npy-numeric identical 2
    numpy-npy-object-os-system identical 2
    numpy-npz-compressed-numeric identical 2
    numpy-npz-stored-object identical 2
    onnx-custom-domain differs 2
    onnx-external-absolute identical 2
    onnx-external-sibling identical 2
    onnx-external-traversal identical 2
    onnx-plain identical 2
    pickle-p0-inst-os-system identical 2
    pickle-p0-os-system identical 2
    pickle-p0-plain-dict differs 2
    pickle-p1-os-system identical 2
    pickle-p1-plain-dict differs 2
    pickle-p2-getattr-chain identical 2
    pickle-p2-ordereddict differs 2
    pickle-p2-os-system identical 2
    pickle-p2-plain-dict differs 2
    pickle-p2-posix-system identical 2
    pickle-p2-webbrowser identical 2
    pickle-p3-builtins-eval identical 2
    pickle-p3-os-system identical 2
    pickle-p3-plain-dict differs 2
    pickle-p4-builtins-exec identical 2
    pickle-p4-ordereddict differs 2
    pickle-p4-os-system identical 2
    pickle-p4-plain-dict differs 2
    pickle-p4-runpy identical 2
    pickle-p5-os-system identical 2
    pickle-p5-plain-dict differs 2
    pickle-p5-subprocess identical 2
    pytorch-zip-deflated-system identical 2
    pytorch-zip-ordereddict differs 2
    pytorch-zip-os-system identical 2
    pytorch-zip-p4-exec identical 2
    safetensors-empty identical 2
    safetensors-metadata identical 2
    safetensors-offsets-past-end differs 2
    safetensors-plain identical 2