Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    ZAP MCP detections: keryn-passive vs ZAP 2.17.0 (community script 100045, mcp add-on importer)

    View as Markdown

    Given the same captured exchanges and the same MCP servers, does keryn-passive raise the alerts, send the requests and name the input vectors that ZAP’s own MCP code does?

    Evaluation detail Description
    Keryn engine keryn-passive at cb67cb0a6705: Keryn’s passive web engine (crates/keryn-passive), its MCP modules: rule 100045 (passive twin and the active form under keryn dast scan --active) and ZAP’s MCP importer as an input-vector adapter (keryn mcp inventory --requests), native and WebAssembly.
    Reference ZAP’s MCP detections: community-scripts OpenModelContextProtocolServer.js (rule 100045) on GraalJS inside the scripts add-on’s ActiveScriptScanRule, and the zap-extensions mcp add-on’s McpImporter and VariantMcpJsonRpc, on ZAP core 2.17.0. Oracle only: the upstream script and classes run unmodified by a small Java harness on a development machine, with fixture responses instead of a network; never shipped.
    Corpus Keryn MCP fixture corpus (captured initialize exchanges, fixture servers, importer server answers), run against the community script and the zap-extensions mcp add-on at the pinned commits, zaproxy/zap-extensions at 1b34a3b77298, restated from crates/keryn-passive/results/mcp-conformance.json
    Comparison Per fixture. passive: the alerts of rule 100045’s passive twin vs the community script’s analyzeMcpResponse on the same captured exchange, on every ZAP alert field (id, ref, name, risk, confidence, param, attack, evidence, other info, CWE, WASC, URI, description, solution, reference, tags); a fixture outside the twin’s scope is identical only when upstream would not alert either, else unsupported. active: the script’s scanHost against a fixture server vs the Rust probe plan and analysis: every request in order (method, URI, Accept, Content-Type, body) and every alert field. import: McpImporter.importServer against a fixture server vs the Rust replay: requestCount and every request (method, URI, the five headers the importer sets, body) with VariantMcpJsonRpc’s paramList size, tree path and leaf name.
    Receipt zap-mcp.json, sha256 86185ab95e235225
    Measure Value Count Denominator
    Fixtures on which Keryn and ZAP agree exactly 92.3% 96 / 104 all fixtures of the three suites (passive exchanges, fixture servers, importer servers)
    Passive twin identical to the script’s matcher 98.2% 56 / 57 captured exchanges inside the twin’s scope (unauthenticated POST initialize)
    Probe sequence and alerts identical to the script’s scanHost 100% 13 / 13 fixture servers
    Request sequence identical to McpImporter with VariantMcpJsonRpc naming 100% 26 / 26 importer fixture servers
    Importer requests compared byte for byte 100% 300 / 300 requests McpImporter sent across the importer fixtures (all compared inside identical cases)
    WebAssembly build identical to native 100% 104 / 104 all fixtures of the three suites
    Upstream MCP detection components ported 100% 3 / 3 MCP detection code at the pinned commits: the 100045 script, McpImporter, VariantMcpJsonRpc

    Cases by status: 1 differs, 96 identical, 7 unsupported.

    • ZAP has no prompt-injection, LLM-output or model rules; its MCP code is this script and this add-on. The llm add-on and alertFilters’ LLM review are triage aids and out of scope.
    • The passive twin only reads a captured unauthenticated POST initialize (Keryn’s scope; the script itself checks no authentication). Fixtures outside that scope on which the upstream matcher alerts are unsupported cases, not hidden.
    • The active form sends only the probes inside the declared dast target subtree; the active cases compare the full upstream probe sequence, which the Rust plan reproduces before that filter.
    • Importer placeholder values are commonlib’s DefaultValueProvider (no form-handler values configured).
    • The fixtures are authored, not traffic captured from real servers.
    • 7 out-of-scope passive fixtures are alerts the upstream matcher raises and the twin does not (unsupported).
    • Restated from the committed differential; numbers are not recomputed from a new run.
    Case Status Reference rows Keryn rows Matching Missing Extra
    passive/json-surrogate-at-500 differs
    passive/scope-authorization-header unsupported
    passive/scope-authorization-lowercase unsupported
    passive/scope-batch-request unsupported
    passive/scope-get-request unsupported
    passive/scope-initialize-not-json unsupported
    passive/scope-method-case unsupported
    passive/scope-tools-list-request unsupported
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    active/auth-required-everywhere identical
    active/default-port-omitted identical
    active/explicit-port-path-seed identical
    active/first-hit-stops identical
    active/network-error-then-hit identical
    active/nothing-found identical
    active/root-json-at-mcp identical
    active/root-json-at-root identical
    active/root-sse-at-mcp-slash identical
    active/seed-path-is-endpoint identical
    active/sse-without-session-at-rpc identical
    active/subtree-seed-app identical
    active/subtree-seed-app-endpoint identical
    import/bad-url-scheme identical
    import/basic-tools identical
    import/blank-security-key identical
    import/cursor-object-stops identical
    import/empty-url identical
    import/initialize-299 identical
    import/initialize-401 identical
    import/initialize-jsonrpc-error identical
    import/initialize-network-error identical
    import/initialize-no-protocol identical
    import/initialize-protocol-null identical
    import/no-logging-capability identical
    import/notification-network-error identical
    import/pagination-and-everything identical
    import/pagination-cap identical
    import/ping-network-error identical
    import/security-key-and-logging identical
    import/sse-content-type-mixed-case identical
    import/sse-initialize identical
    import/tool-name-edge-cases identical
    import/tools-list-network-error identical
    import/tools-list-not-json identical
    import/tools-list-trailing-garbage identical
    import/url-root-no-path identical
    import/url-uppercase-scheme identical
    import/url-with-query identical
    passive/html-with-valid-json identical
    passive/json-capabilities-empty identical
    passive/json-charset identical
    passive/json-client-payload identical
    passive/json-confirmed identical
    passive/json-content-type-upper identical
    passive/json-duplicate-key-last-wins identical
    passive/json-duplicate-key-loses identical
    passive/json-empty-body identical
    passive/json-error-401 identical
    passive/json-escapes identical
    passive/json-id-missing identical
    passive/json-id-null identical
    passive/json-jsonrpc-2.1 identical
    passive/json-jsonrpc-number identical
    passive/json-latin1-body identical
    passive/json-leading-bom identical
    passive/json-leading-zero identical
    passive/json-lone-surrogate-escape identical
    passive/json-long-body identical
    passive/json-lowercase-header-name identical
    passive/json-nbsp-whitespace identical
    passive/json-no-server-info identical
    passive/json-non-ascii-body identical
    passive/json-protocol-array identical
    passive/json-protocol-empty identical
    passive/json-protocol-false identical
    passive/json-protocol-huge identical
    passive/json-protocol-neg-zero identical
    passive/json-protocol-zero identical
    passive/json-raw-control-char identical
    passive/json-result-array identical
    passive/json-result-string identical
    passive/json-server-info-string identical
    passive/json-surrogate-at-500 differs
    passive/json-top-array identical
    passive/json-top-null identical
    passive/json-top-string identical
    passive/json-trailing-comma identical
    passive/json-trailing-garbage identical
    passive/json-trailing-whitespace identical
    passive/json-valid-but-500 identical
    passive/json-with-session-and-server identical
    passive/no-content-type-no-body identical
    passive/no-content-type-with-session identical
    passive/scope-authorization-header unsupported
    passive/scope-authorization-lowercase unsupported
    passive/scope-batch-request unsupported
    passive/scope-cookie-only identical
    passive/scope-get-request unsupported
    passive/scope-initialize-not-json unsupported
    passive/scope-initialize-notification identical
    passive/scope-method-case unsupported
    passive/scope-no-response identical
    passive/scope-tools-list-request unsupported
    passive/sse-and-json-content-type identical
    passive/sse-content-type-upper identical
    passive/sse-garbage-body identical
    passive/sse-long-body identical
    passive/sse-no-session-empty-body identical
    passive/sse-session-empty-body identical
    passive/sse-status-202 identical
    passive/sse-with-session identical
    passive/sse-without-session identical
    passive/two-content-types identical