ZAP MCP detections: keryn-passive vs ZAP 2.17.0 (community script 100045, mcp add-on importer)
Given the same captured exchanges and the same MCP servers, does keryn-passive raise the alerts, send the requests and name the input vectors that ZAP’s own MCP code does?
| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-passive at cb67cb0a6705: Keryn’s passive web engine (crates/keryn-passive), its MCP modules: rule 100045 (passive twin and the active form under keryn dast scan --active) and ZAP’s MCP importer as an input-vector adapter (keryn mcp inventory --requests), native and WebAssembly. |
| Reference | ZAP’s MCP detections: community-scripts OpenModelContextProtocolServer.js (rule 100045) on GraalJS inside the scripts add-on’s ActiveScriptScanRule, and the zap-extensions mcp add-on’s McpImporter and VariantMcpJsonRpc, on ZAP core 2.17.0. Oracle only: the upstream script and classes run unmodified by a small Java harness on a development machine, with fixture responses instead of a network; never shipped. |
| Corpus | Keryn MCP fixture corpus (captured initialize exchanges, fixture servers, importer server answers), run against the community script and the zap-extensions mcp add-on at the pinned commits, zaproxy/zap-extensions at 1b34a3b77298, restated from crates/keryn-passive/results/mcp-conformance.json |
| Comparison | Per fixture. passive: the alerts of rule 100045’s passive twin vs the community script’s analyzeMcpResponse on the same captured exchange, on every ZAP alert field (id, ref, name, risk, confidence, param, attack, evidence, other info, CWE, WASC, URI, description, solution, reference, tags); a fixture outside the twin’s scope is identical only when upstream would not alert either, else unsupported. active: the script’s scanHost against a fixture server vs the Rust probe plan and analysis: every request in order (method, URI, Accept, Content-Type, body) and every alert field. import: McpImporter.importServer against a fixture server vs the Rust replay: requestCount and every request (method, URI, the five headers the importer sets, body) with VariantMcpJsonRpc’s paramList size, tree path and leaf name. |
| Receipt | zap-mcp.json, sha256 86185ab95e235225 |
Results
Section titled “Results”| Measure | Value | Count | Denominator |
|---|---|---|---|
| Fixtures on which Keryn and ZAP agree exactly | 92.3% | 96 / 104 | all fixtures of the three suites (passive exchanges, fixture servers, importer servers) |
| Passive twin identical to the script’s matcher | 98.2% | 56 / 57 | captured exchanges inside the twin’s scope (unauthenticated POST initialize) |
| Probe sequence and alerts identical to the script’s scanHost | 100% | 13 / 13 | fixture servers |
| Request sequence identical to McpImporter with VariantMcpJsonRpc naming | 100% | 26 / 26 | importer fixture servers |
| Importer requests compared byte for byte | 100% | 300 / 300 | requests McpImporter sent across the importer fixtures (all compared inside identical cases) |
| WebAssembly build identical to native | 100% | 104 / 104 | all fixtures of the three suites |
| Upstream MCP detection components ported | 100% | 3 / 3 | MCP detection code at the pinned commits: the 100045 script, McpImporter, VariantMcpJsonRpc |
Cases by status: 1 differs, 96 identical, 7 unsupported.
What this does and does not measure
Section titled “What this does and does not measure”- ZAP has no prompt-injection, LLM-output or model rules; its MCP code is this script and this add-on. The llm add-on and alertFilters’ LLM review are triage aids and out of scope.
- The passive twin only reads a captured unauthenticated POST initialize (Keryn’s scope; the script itself checks no authentication). Fixtures outside that scope on which the upstream matcher alerts are unsupported cases, not hidden.
- The active form sends only the probes inside the declared dast target subtree; the active cases compare the full upstream probe sequence, which the Rust plan reproduces before that filter.
- Importer placeholder values are commonlib’s DefaultValueProvider (no form-handler values configured).
- The fixtures are authored, not traffic captured from real servers.
- 7 out-of-scope passive fixtures are alerts the upstream matcher raises and the twin does not (unsupported).
- Restated from the committed differential; numbers are not recomputed from a new run.
Differences and unassessed cases (8)
Section titled “Differences and unassessed cases (8)”| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
passive/json-surrogate-at-500 |
differs | |||||
passive/scope-authorization-header |
unsupported | |||||
passive/scope-authorization-lowercase |
unsupported | |||||
passive/scope-batch-request |
unsupported | |||||
passive/scope-get-request |
unsupported | |||||
passive/scope-initialize-not-json |
unsupported | |||||
passive/scope-method-case |
unsupported | |||||
passive/scope-tools-list-request |
unsupported |
All cases (104)
Section titled “All cases (104)”Show every case
| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra |
|---|---|---|---|---|---|---|
active/auth-required-everywhere |
identical | |||||
active/default-port-omitted |
identical | |||||
active/explicit-port-path-seed |
identical | |||||
active/first-hit-stops |
identical | |||||
active/network-error-then-hit |
identical | |||||
active/nothing-found |
identical | |||||
active/root-json-at-mcp |
identical | |||||
active/root-json-at-root |
identical | |||||
active/root-sse-at-mcp-slash |
identical | |||||
active/seed-path-is-endpoint |
identical | |||||
active/sse-without-session-at-rpc |
identical | |||||
active/subtree-seed-app |
identical | |||||
active/subtree-seed-app-endpoint |
identical | |||||
import/bad-url-scheme |
identical | |||||
import/basic-tools |
identical | |||||
import/blank-security-key |
identical | |||||
import/cursor-object-stops |
identical | |||||
import/empty-url |
identical | |||||
import/initialize-299 |
identical | |||||
import/initialize-401 |
identical | |||||
import/initialize-jsonrpc-error |
identical | |||||
import/initialize-network-error |
identical | |||||
import/initialize-no-protocol |
identical | |||||
import/initialize-protocol-null |
identical | |||||
import/no-logging-capability |
identical | |||||
import/notification-network-error |
identical | |||||
import/pagination-and-everything |
identical | |||||
import/pagination-cap |
identical | |||||
import/ping-network-error |
identical | |||||
import/security-key-and-logging |
identical | |||||
import/sse-content-type-mixed-case |
identical | |||||
import/sse-initialize |
identical | |||||
import/tool-name-edge-cases |
identical | |||||
import/tools-list-network-error |
identical | |||||
import/tools-list-not-json |
identical | |||||
import/tools-list-trailing-garbage |
identical | |||||
import/url-root-no-path |
identical | |||||
import/url-uppercase-scheme |
identical | |||||
import/url-with-query |
identical | |||||
passive/html-with-valid-json |
identical | |||||
passive/json-capabilities-empty |
identical | |||||
passive/json-charset |
identical | |||||
passive/json-client-payload |
identical | |||||
passive/json-confirmed |
identical | |||||
passive/json-content-type-upper |
identical | |||||
passive/json-duplicate-key-last-wins |
identical | |||||
passive/json-duplicate-key-loses |
identical | |||||
passive/json-empty-body |
identical | |||||
passive/json-error-401 |
identical | |||||
passive/json-escapes |
identical | |||||
passive/json-id-missing |
identical | |||||
passive/json-id-null |
identical | |||||
passive/json-jsonrpc-2.1 |
identical | |||||
passive/json-jsonrpc-number |
identical | |||||
passive/json-latin1-body |
identical | |||||
passive/json-leading-bom |
identical | |||||
passive/json-leading-zero |
identical | |||||
passive/json-lone-surrogate-escape |
identical | |||||
passive/json-long-body |
identical | |||||
passive/json-lowercase-header-name |
identical | |||||
passive/json-nbsp-whitespace |
identical | |||||
passive/json-no-server-info |
identical | |||||
passive/json-non-ascii-body |
identical | |||||
passive/json-protocol-array |
identical | |||||
passive/json-protocol-empty |
identical | |||||
passive/json-protocol-false |
identical | |||||
passive/json-protocol-huge |
identical | |||||
passive/json-protocol-neg-zero |
identical | |||||
passive/json-protocol-zero |
identical | |||||
passive/json-raw-control-char |
identical | |||||
passive/json-result-array |
identical | |||||
passive/json-result-string |
identical | |||||
passive/json-server-info-string |
identical | |||||
passive/json-surrogate-at-500 |
differs | |||||
passive/json-top-array |
identical | |||||
passive/json-top-null |
identical | |||||
passive/json-top-string |
identical | |||||
passive/json-trailing-comma |
identical | |||||
passive/json-trailing-garbage |
identical | |||||
passive/json-trailing-whitespace |
identical | |||||
passive/json-valid-but-500 |
identical | |||||
passive/json-with-session-and-server |
identical | |||||
passive/no-content-type-no-body |
identical | |||||
passive/no-content-type-with-session |
identical | |||||
passive/scope-authorization-header |
unsupported | |||||
passive/scope-authorization-lowercase |
unsupported | |||||
passive/scope-batch-request |
unsupported | |||||
passive/scope-cookie-only |
identical | |||||
passive/scope-get-request |
unsupported | |||||
passive/scope-initialize-not-json |
unsupported | |||||
passive/scope-initialize-notification |
identical | |||||
passive/scope-method-case |
unsupported | |||||
passive/scope-no-response |
identical | |||||
passive/scope-tools-list-request |
unsupported | |||||
passive/sse-and-json-content-type |
identical | |||||
passive/sse-content-type-upper |
identical | |||||
passive/sse-garbage-body |
identical | |||||
passive/sse-long-body |
identical | |||||
passive/sse-no-session-empty-body |
identical | |||||
passive/sse-session-empty-body |
identical | |||||
passive/sse-status-202 |
identical | |||||
passive/sse-with-session |
identical | |||||
passive/sse-without-session |
identical | |||||
passive/two-content-types |
identical |