Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    Semgrep rules: Keryn vs Semgrep CE 1.157.0

    View as Markdown

    Over a public and authored corpus, do Keryn’s pattern rules report the same findings as native Semgrep CE for every rule in the Keryn library?

    Evaluation detail Description
    Keryn engine Keryn pattern rules at 2d7fca3dc059: Keryn’s Semgrep-compatible rule engine (tree-sitter, Rust/WASM core).
    Reference Semgrep CE 1.157.0. Oracle, installed from PyPI.
    Corpus Keryn parity corpus (public and authored files), restated from ts/results/parity-corpus-2026-11-08.json (run of 2026-11-08)
    Comparison Per finding: same rule, file and range. A rule is identical when no finding is missed or extra.
    Receipt semgrep-ce.json, sha256 4f6bc4230f460fa7
    Measure Value Count Denominator
    Rules with identical findings 100% 164 / 164 rules that reported a finding on either side
    Library rules the engine executed 100% 1,398 / 1,398 rules in the Keryn library
    Semgrep findings Keryn also reported 100% 13,475 / 13,475 findings native Semgrep reported
    Keryn findings Semgrep also reported 100% 13,475 / 13,475 findings Keryn reported

    Cases by status: 164 identical.

    • The 1398 rules of the Keryn library (132 packs) run by both engines on the same 1002 files.
    • Rules that report nothing on the corpus on either side are executed but do not appear as cases.
    • Restated from the committed weekly parity receipt; numbers are not recomputed here.
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra
    javascript.permissions.mcp-tool-param-to-shell identical 2 2 2 0 0
    javascript.prompt-injection.user-input-in-system-prompt identical 4 4 4 0 0
    javascript.secrets.llm-api-key-logged identical 4 4 4 0 0
    keryn.agents2026.pydantic-ai-tool-argument-to-exec identical 3 3 3 0 0
    keryn.egress.py.secret-to-side-channel identical 17 17 17 0 0
    keryn.egress.ts.secret-to-side-channel identical 3 3 3 0 0
    keryn.errdisc.py.exception-detail-to-caller identical 4 4 4 0 0
    keryn.fastapi.cors-wildcard-with-credentials identical 4 4 4 0 0
    keryn.fastapi.llm-route-without-auth identical 4 4 4 0 0
    keryn.fastapi.llm-route-without-rate-limit identical 4 4 4 0 0
    keryn.gaps.dynamic-import-from-variable identical 1 1 1 0 0
    keryn.gaps.instructions-concatenated-with-input identical 2 2 2 0 0
    keryn.gaps.joined-path-written-unconfined identical 6 6 6 0 0
    keryn.gaps.parameter-url-fetched identical 2 2 2 0 0
    keryn.gaps.unbounded-model-loop identical 7 7 7 0 0
    keryn.gaps.vector-query-without-tenant-filter identical 4 4 4 0 0
    keryn.inv.lib.ai-sdk-providers identical 9 9 9 0 0
    keryn.inv.lib.anthropic identical 33 33 33 0 0
    keryn.inv.lib.cpp-ml-runtimes identical 13 13 13 0 0
    keryn.inv.lib.crewai-autogen identical 31 31 31 0 0
    keryn.inv.lib.dotnet-ai-sdks identical 10 10 10 0 0
    keryn.inv.lib.go-ai-sdks identical 6 6 6 0 0
    keryn.inv.lib.google-genai identical 1 1 1 0 0
    keryn.inv.lib.jvm-ai-sdks identical 53 53 53 0 0
    keryn.inv.lib.langchain identical 72 72 72 0 0
    keryn.inv.lib.langgraph identical 1 1 1 0 0
    keryn.inv.lib.llamaindex identical 120 120 120 0 0
    keryn.inv.lib.ollama identical 1 1 1 0 0
    keryn.inv.lib.openai identical 56 56 56 0 0
    keryn.inv.lib.php-ai-sdks identical 90 90 90 0 0
    keryn.inv.lib.rust-ai-crates identical 24 24 24 0 0
    keryn.inv.lib.swift-ai-frameworks identical 17 17 17 0 0
    keryn.inv.lib.vercel-ai identical 28 28 28 0 0
    keryn.inv.lib.vertex-ai identical 2 2 2 0 0
    keryn.inv.mcp identical 59 59 59 0 0
    keryn.inv.mcp.manifest identical 3 3 3 0 0
    keryn.inv.model.anthropic identical 33 33 33 0 0
    keryn.inv.model.google identical 1 1 1 0 0
    keryn.inv.model.open-weights identical 1 1 1 0 0
    keryn.inv.model.openai identical 65 65 65 0 0
    keryn.inv.service.endpoint identical 3 3 3 0 0
    keryn.inv.vectordb identical 8 8 8 0 0
    keryn.keyliteral.openai identical 21 21 21 0 0
    keryn.statetx.ts.action-without-replay-guard identical 3 3 3 0 0
    keryn.vendor.anthropic-content-read-without-stop-reason-check-py identical 4 4 4 0 0
    keryn.vendor.anthropic-messages-without-metadata-user-id-js identical 5 5 5 0 0
    keryn.vendor.anthropic-messages-without-metadata-user-id-py identical 15 15 15 0 0
    keryn.vendor.openai-call-without-moderation-js identical 35 35 35 0 0
    keryn.vendor.openai-call-without-moderation-py identical 11 11 11 0 0
    keryn.vendor.openai-chat-without-user-identifier-js identical 33 33 33 0 0
    keryn.vendor.openai-chat-without-user-identifier-py identical 11 11 11 0 0
    keryn.vendor.openai-content-read-without-refusal-check-js identical 3 3 3 0 0
    keryn.vendor.openai-content-read-without-refusal-check-py identical 4 4 4 0 0
    keryn.vfgap.child-process-exec-sync-js identical 5 5 5 0 0
    keryn.vfgap.exec-command-shell-go identical 1 1 1 0 0
    keryn.vfgap.flask-debug-run-py identical 2 2 2 0 0
    keryn.vfgap.process-builder-non-literal-java identical 2 2 2 0 0
    keryn.vfgap.process-start-non-literal-cs identical 3 3 3 0 0
    keryn.vfgap.subprocess-check-shell-true-py identical 1 1 1 0 0
    python.deserialization.pickle-load-model-file identical 1 1 1 0 0
    python.mcp.fastmcp-http-transport-without-auth identical 3 3 3 0 0
    python.permissions.agent-tool-param-to-shell identical 1 1 1 0 0
    seceng.agentcfg.mcp-env-inline-secret-known-prefix identical 21 21 21 0 0
    seceng.agentcfg.mcp-npx-unpinned-package identical 28 28 28 0 0
    seceng.agentcfg.mcp-remote-no-authorization identical 1 1 1 0 0
    seceng.agentframework.autogen-shared-workdir-or-history identical 101 101 101 0 0
    seceng.agentframework.crewai-memory-without-tenant-scope identical 12 12 12 0 0
    seceng.agentframework.crewai-process-without-rate-or-step-limit identical 457 457 457 0 0
    seceng.agentrules.pipe-to-shell identical 21 21 21 0 0
    seceng.bundle-embedded-system-prompt identical 24 24 24 0 0
    seceng.bundle-tool-schema-exposed identical 103 103 103 0 0
    seceng.cloudai.azure-openai-public-network-access identical 974 974 974 0 0
    seceng.cloudai.bedrock-broad-invoke-model-policy identical 1 1 1 0 0
    seceng.cloudflare-ai-gateway-logs-prompts identical 1 1 1 0 0
    seceng.cloudflare.vectorize-query-without-namespace-filter identical 3 3 3 0 0
    seceng.container-agent-no-readonly-rootfs identical 4797 4797 4797 0 0
    seceng.container-ai-image-latest-or-unpinned identical 57 57 57 0 0
    seceng.container-ai-secret-env-direct identical 1 1 1 0 0
    seceng.deno.edge-secret-logged identical 8 8 8 0 0
    seceng.deploy-ai-service-no-rate-limit-hint identical 20 20 20 0 0
    seceng.deploy-tool-container-unrestricted-egress identical 4797 4797 4797 0 0
    seceng.edge-ai-route-no-rate-limit identical 34 34 34 0 0
    seceng.forensics-ai-error-swallowed identical 7 7 7 0 0
    seceng.forensics-model-version-missing identical 109 109 109 0 0
    seceng.forensics-no-user-tenant-on-ai-event identical 47 47 47 0 0
    seceng.forensics-prompt-hash-missing identical 25 25 25 0 0
    seceng.forensics.ai-action-missing-correlation-id identical 215 215 215 0 0
    seceng.forensics.tool-call-missing-audit-event identical 5 5 5 0 0
    seceng.gateway.log-contains-api-key identical 1 1 1 0 0
    seceng.github-actions-excessive-token-permissions-ai-job identical 1 1 1 0 0
    seceng.github-workflow-dispatch-input-to-shell-or-ai identical 1 1 1 0 0
    seceng.github.actions-unpinned-third-party-action identical 16 16 16 0 0
    seceng.gitlab-ci-user-input-to-shell-or-ai identical 5 5 5 0 0
    seceng.gitlab-unpinned-ai-container-image identical 3 3 3 0 0
    seceng.graphql-schema-exposes-prompt-or-tool-types identical 2 2 2 0 0
    seceng.graphql.ai-mutation-without-auth-context identical 3 3 3 0 0
    seceng.iacai.bedrock-agent-iam-wildcard-action-resource identical 1 1 1 0 0
    seceng.java.langchain4j-tool-dangerous-method identical 1 1 1 0 0
    seceng.kotlin-prompt-or-completion-logged identical 1 1 1 0 0
    seceng.langgraph-checkpoint-shared-or-unscoped identical 7 7 7 0 0
    seceng.langgraph-state-persists-sensitive-context identical 1 1 1 0 0
    seceng.laravel-ai-output-direct-db-write identical 8 8 8 0 0
    seceng.llm01.missing-safety-sidecar-openai-python identical 24 24 24 0 0
    seceng.llm01.taint.request-to-system-prompt-js identical 1 1 1 0 0
    seceng.llm02-system-prompt-in-client identical 2 2 2 0 0
    seceng.llm02.prompt-message-logging identical 16 16 16 0 0
    seceng.llm02.prompt-message-logging-rust identical 1 1 1 0 0
    seceng.llm04.python-model-output-to-system-shell identical 7 7 7 0 0
    seceng.llm05.node-child-process-exec identical 5 5 5 0 0
    seceng.llm05.python-sql-string-format identical 2 2 2 0 0
    seceng.llm05.python-subprocess-shell-true identical 7 7 7 0 0
    seceng.llm05.react-dangerously-set-inner-html identical 1 1 1 0 0
    seceng.llm05.react-dangerously-set-inner-html-unprotected identical 1 1 1 0 0
    seceng.llm05.taint.model-output-to-child-process-js identical 2 2 2 0 0
    seceng.llm05.taint.model-output-to-dynamic-dispatch-js identical 20 20 20 0 0
    seceng.llm05.taint.model-output-to-reflection-dispatch-python identical 4 4 4 0 0
    seceng.llm06.agent-broad-tools-without-approval identical 2 2 2 0 0
    seceng.llm06.browser-agent-action identical 2 2 2 0 0
    seceng.llm06.dangerous-tool-without-obvious-approval identical 12 12 12 0 0
    seceng.llm06.langchain-tool-missing-args-schema identical 6 6 6 0 0
    seceng.llm06.tool-call-without-approval-gate identical 6 6 6 0 0
    seceng.llm06.tool-call-without-approval-gate-js identical 21 21 21 0 0
    seceng.llm09.missing-hallucination-disclaimer identical 7 7 7 0 0
    seceng.llm09.python-rag-response-missing-citation identical 8 8 8 0 0
    seceng.llm09.python-rag-response-missing-confidence identical 8 8 8 0 0
    seceng.llm10.openai-call-without-max-tokens identical 34 34 34 0 0
    seceng.llm10.unbounded-retry-loop-js identical 4 4 4 0 0
    seceng.loop.langchain-agent-no-max-iterations identical 1 1 1 0 0
    seceng.loop.langgraph-recursion-limit-missing identical 41 41 41 0 0
    seceng.mcp.command-tool identical 60 60 60 0 0
    seceng.mcp.filesystem-read-tool identical 35 35 35 0 0
    seceng.mcp.http-sse-server identical 18 18 18 0 0
    seceng.mcp.java.tool-process-exec identical 2 2 2 0 0
    seceng.mcp.v3.public-server-bind-no-auth identical 2 2 2 0 0
    seceng.modelsc.mcp-stdio-npx-unpinned-ts identical 2 2 2 0 0
    seceng.postgres-ai-audit-log-raw-prompt identical 44 44 44 0 0
    seceng.promptsc-runtime-prompt-edit-route identical 7 7 7 0 0
    seceng.promptsc.env-controlled-system-prompt identical 4 4 4 0 0
    seceng.promptsc.prompt-template-from-database-without-version identical 19 19 19 0 0
    seceng.redaction-tool-arguments-logged identical 3 3 3 0 0
    seceng.redaction.prompt-logged-before-redaction-python identical 2 2 2 0 0
    seceng.redaction.prompt-logged-before-redaction-ts identical 14 14 14 0 0
    seceng.redaction.trace-attributes-raw-ai-context identical 6 6 6 0 0
    seceng.redis.ai-cache-without-ttl identical 1 1 1 0 0
    seceng.stream-sensitive-token-leak identical 6 6 6 0 0
    seceng.stream.sse-direct-llm-stream-no-filter-ts identical 4 4 4 0 0
    seceng.toolauth.audit-missing-for-dangerous-tool identical 47 47 47 0 0
    seceng.toolauth.dangerous-tool-no-approval-python identical 4 4 4 0 0
    seceng.toolauth.dangerous-tool-no-approval-ts identical 1 1 1 0 0
    seceng.toolauth.model-controlled-user-or-tenant identical 3 3 3 0 0
    seceng.vectordb.chroma-query-without-where identical 15 15 15 0 0
    seceng.vectordb.pinecone-query-without-namespace-or-filter identical 5 5 5 0 0
    seceng.vendor.anthropic-computer-use-without-domain-or-action-policy identical 1 1 1 0 0
    seceng.vendor.openai-assistants-thread-not-user-bound identical 1 1 1 0 0
    seceng.vendor.openai-file-search-vector-store-no-boundary identical 12 12 12 0 0
    seceng.vendor.openai-realtime-without-auth-or-rate-limit identical 1 1 1 0 0
    seceng.vercelai-dangerous-tool-without-approval identical 1 1 1 0 0
    seceng.vercelai.streamtext-tools-without-maxsteps identical 1 1 1 0 0
    vulnfix-ai-ai_cors_wildcard_creds identical 4 4 4 0 0
    vulnfix-ai-ai_js_dangerous_html identical 1 1 1 0 0
    vulnfix-ai-ai_mcp_bind_all identical 2 2 2 0 0
    vulnfix-ai-ai_mcp_js_tool_path identical 2 2 2 0 0
    vulnfix-ai-ai_mcp_sse_transport identical 2 2 2 0 0
    vulnfix-ai-ai_mcp_tool_command identical 3 3 3 0 0