# benchmark-gaps-v1

> benchmark-gaps-v1: 14 Keryn rules for python (14 warning).

Source: https://keryn.dev/rules/benchmark-gaps-v1/

14 rules · Keryn Software Licence · rules 0.8.0 · [all packs](https://keryn.dev/rules/)

**How to read a rule**

- **Severity**: How bad the finding is if it is real: `error` fails a default gate, `warning` and `info` do not.

- **Confidence**: **P0** exploitable as written; **P1** dangerous in context; **P2** a heuristic or a posture check: confirm by reading the code.

- **False-positive risk**: **low**: an exact API or flag, so hits are almost always real; **medium**: the pattern is legitimate in many programs, so check each hit; **high**: it finds a lead to confirm, so expect benign hits.

- **Edge**: The relationship the finding sits on. **reaches**: content or traffic gets to a thing; **calls**: one thing invokes another; **reads**: one thing reads from another; **writes**: one thing writes to another; **sends**: data leaves for another party; **grants**: authority is given or left unchecked; **depends_on**: something relies on a component it does not verify; **trusts**: one party believes another without evidence. The boundary it crosses, when there is one: **user->prompt** (untrusted input reaching a prompt); **model->code** (model output reaching code, a shell, a query or a page); **secret->outside** (a secret leaving the trusted side); **supply-chain** (code, models or prompts pulled in from outside).

- **Taxonomy chips**: References for readers who work in OWASP LLM 2025, OWASP Agentic, CWE or NIST AI RMF. They say where a rule belongs, not that a framework is satisfied.

## keryn.gaps.retrieved-text-in-system-role

`warning` python data-flow rule openai, pinecone, chromadb, langchain

Text that came back from a retrieval call (vector index query, retriever, similarity search, or a list of retrieved documents) is placed in the system role, so a document anyone can plant becomes the model's instructions. Keep retrieved text in a user or tool message, fenced and labelled as data.

Confidence **P1**: dangerous in context. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [reaches](https://keryn.dev/docs/edges/edge-kinds/#reaches) content or traffic gets to a thing; it crosses the [`user->prompt`](https://keryn.dev/docs/edges/trust-boundaries/#user-to-prompt) boundary (untrusted input reaching a prompt).

[CWE-77](https://keryn.dev/taxonomies/cwe/#cwe-77-command-injection) [LLM01](https://keryn.dev/taxonomies/owasp-llm/#llm01-prompt-injection) [ASI01](https://keryn.dev/taxonomies/owasp-agentic/#asi01-agent-goal-hijack)

**Fix:** Put retrieved passages in a user-role message (or a tool result) wrapped in explicit delimiters, and keep the system message to instructions you wrote. If a system message must reference documents, reference them by id and have the model fetch them as data.

**References:** [https://developers.openai.com/api/docs/guides/text#message-roles-and-instruction-following](https://developers.openai.com/api/docs/guides/text#message-roles-and-instruction-following) · [https://docs.pinecone.io/reference/api/2024-10/data-plane/query](https://docs.pinecone.io/reference/api/2024-10/data-plane/query)

## keryn.gaps.instructions-concatenated-with-input

`warning` python openai, python

A constant that holds the system instructions is concatenated (or formatted) with a variable into one prompt string, so the caller's text sits in the same instruction channel as yours and can override it. Send instructions as the system message and the variable text as a separate user message.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [reaches](https://keryn.dev/docs/edges/edge-kinds/#reaches) content or traffic gets to a thing; it crosses the [`user->prompt`](https://keryn.dev/docs/edges/trust-boundaries/#user-to-prompt) boundary (untrusted input reaching a prompt).

[CWE-77](https://keryn.dev/taxonomies/cwe/#cwe-77-command-injection) [LLM01](https://keryn.dev/taxonomies/owasp-llm/#llm01-prompt-injection)

**Fix:** Keep the instruction constant in the system role and pass user-supplied text as its own user-role message; never build one string from both.

**References:** [https://developers.openai.com/api/docs/guides/text#message-roles-and-instruction-following](https://developers.openai.com/api/docs/guides/text#message-roles-and-instruction-following)

## keryn.gaps.provider-secret-in-response

`warning` python flask, fastapi, python

A provider API key, token or secret read from the environment is placed in a response body returned to the caller. Keep provider credentials server-side; return a model name or an opaque session id instead.

Confidence **P1**: dangerous in context. False-positive risk **low**: an exact API or flag, so hits are almost always real.

Edge inspected: [sends](https://keryn.dev/docs/edges/edge-kinds/#sends) data leaves for another party; it crosses the [`secret->outside`](https://keryn.dev/docs/edges/trust-boundaries/#secret-to-outside) boundary (a secret leaving the trusted side).

[CWE-200](https://keryn.dev/taxonomies/cwe/#cwe-200-exposure-of-sensitive-information-to-an-unauthorized-actor) [LLM02](https://keryn.dev/taxonomies/owasp-llm/#llm02-sensitive-information-disclosure)

**Fix:** Never serialise provider credentials into an HTTP response; proxy provider calls server-side or mint short-lived, scoped tokens for clients.

**References:** [https://flask.palletsprojects.com/en/stable/api/#flask.json.jsonify](https://flask.palletsprojects.com/en/stable/api/#flask.json.jsonify) · [https://docs.python.org/3/library/os.html#os.environ](https://docs.python.org/3/library/os.html#os.environ)

## keryn.gaps.prompt-or-messages-logged-raw

`warning` python python, logging

The prompt, the message list or the transcript is written to a log or stdout as-is, so whatever a user typed (PII, secrets) is persisted verbatim. Log a redacted copy, a hash or counts; never the raw content.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [sends](https://keryn.dev/docs/edges/edge-kinds/#sends) data leaves for another party.

[CWE-532](https://keryn.dev/taxonomies/cwe/#cwe-532-insertion-of-sensitive-information-into-log-file) [LLM02](https://keryn.dev/taxonomies/owasp-llm/#llm02-sensitive-information-disclosure)

**Fix:** Redact before logging (`redact(messages)`), or log structural facts (message count, token count, a content hash) and keep the raw transcript in a store with its own access control and retention.

**References:** [https://docs.python.org/3/library/logging.html](https://docs.python.org/3/library/logging.html)

## keryn.gaps.bytes-from-network-deserialized

`warning` python data-flow rule requests, pickle, torch, joblib

Bytes fetched over HTTP are deserialised (pickle, torch.load, joblib, marshal) or written to the artifact store without a digest check, so whoever controls the URL or the network path controls what runs when the artifact is loaded. Verify a pinned digest or signature before deserialising or storing.

Confidence **P1**: dangerous in context. False-positive risk **low**: an exact API or flag, so hits are almost always real.

Edge inspected: [depends_on](https://keryn.dev/docs/edges/edge-kinds/#depends-on) something relies on a component it does not verify; it crosses the [`supply-chain`](https://keryn.dev/docs/edges/trust-boundaries/#supply-chain) boundary (code, models or prompts pulled in from outside).

[CWE-502](https://keryn.dev/taxonomies/cwe/#cwe-502-deserialization-of-untrusted-data) [CWE-494](https://keryn.dev/taxonomies/cwe/#cwe-494-download-of-code-without-integrity-check) [LLM03](https://keryn.dev/taxonomies/owasp-llm/#llm03-supply-chain) [LLM05](https://keryn.dev/taxonomies/owasp-llm/#llm05-improper-output-handling)

**Fix:** Compare the downloaded bytes against a pinned SHA-256 (or verify a signature) before `pickle.loads` / `torch.load` / writing them into the model cache; prefer safetensors for weights.

**References:** [https://docs.python.org/3/library/pickle.html](https://docs.python.org/3/library/pickle.html) · [https://docs.pytorch.org/docs/stable/generated/torch.load.html](https://docs.pytorch.org/docs/stable/generated/torch.load.html) · [https://requests.readthedocs.io/en/latest/api/#requests.get](https://requests.readthedocs.io/en/latest/api/#requests.get)

## keryn.gaps.tool-argument-opened-as-path

`warning` python data-flow rule mcp, python

A path taken from the tool-call arguments is opened, read or deleted without confining it to a root directory, so the model (or whoever steers it) chooses which file the tool touches. Resolve the path and reject it unless it is inside an allowed root.

Confidence **P1**: dangerous in context. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [reads](https://keryn.dev/docs/edges/edge-kinds/#reads) one thing reads from another.

[CWE-22](https://keryn.dev/taxonomies/cwe/#cwe-22-path-traversal) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation)

**Fix:** Build the path as `(ROOT / arguments["path"]).resolve()`, refuse it unless `is_relative_to(ROOT)`, and prefer an allowlist of files or ids over free-form paths.

**References:** [https://modelcontextprotocol.io/specification/2025-06-18/server/tools](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) · [https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to)

## keryn.gaps.vector-query-without-tenant-filter

`warning` python pinecone, chromadb, langchain

A vector index shared by several users or tenants is queried on similarity alone, with no metadata filter, so one caller's query can return another's documents. Pass a tenant or owner filter on every query.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [grants](https://keryn.dev/docs/edges/edge-kinds/#grants) authority is given or left unchecked.

[CWE-862](https://keryn.dev/taxonomies/cwe/#cwe-862-missing-authorization) [LLM08](https://keryn.dev/taxonomies/owasp-llm/#llm08-vector-and-embedding-weaknesses)

**Fix:** Add `filter=&#123;"tenant": tenant_id&#125;` (Pinecone), `where=&#123;"owner": user_id&#125;` (Chroma) or the store's equivalent to every query, or give each tenant its own namespace or collection.

**References:** [https://docs.pinecone.io/guides/indexes/index-data-overview#metadata](https://docs.pinecone.io/guides/indexes/index-data-overview#metadata) · [https://docs.trychroma.com/docs/querying-collections/metadata-filtering](https://docs.trychroma.com/docs/querying-collections/metadata-filtering)

## keryn.gaps.unbounded-model-loop

`warning` python openai, anthropic, google-genai

A `while True` loop calls the model on every iteration with no iteration, time or token budget, so the model (or an injected instruction) decides how long the loop runs and how much it spends. Bound the loop with a counter or a budget and stop when it is exhausted.

Confidence **P1**: dangerous in context. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [grants](https://keryn.dev/docs/edges/edge-kinds/#grants) authority is given or left unchecked.

[CWE-770](https://keryn.dev/taxonomies/cwe/#cwe-770-allocation-of-resources-without-limits-or-throttling) [LLM10](https://keryn.dev/taxonomies/owasp-llm/#llm10-unbounded-consumption) [ASI08](https://keryn.dev/taxonomies/owasp-agentic/#asi08-cascading-failures)

**Fix:** Use `for turn in range(MAX_TURNS):` or a budget check such as `if turns &gt;= MAX_TURNS: break` inside the loop, and cap total tokens; a model-emitted "DONE" is not a bound.

**References:** [https://developers.openai.com/api/docs/guides/function-calling#handling-function-calls](https://developers.openai.com/api/docs/guides/function-calling#handling-function-calls)

## keryn.gaps.dynamic-import-from-variable

`warning` python python

The module to import is built from a variable, so whoever controls that value (a plan step, a tool argument, model output) chooses what gets imported and therefore what runs at import time. Map the value to a fixed table of handlers instead of importing by name.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [depends_on](https://keryn.dev/docs/edges/edge-kinds/#depends-on) something relies on a component it does not verify; it crosses the [`supply-chain`](https://keryn.dev/docs/edges/trust-boundaries/#supply-chain) boundary (code, models or prompts pulled in from outside).

[CWE-470](https://keryn.dev/rules/?cwe=CWE-470) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation)

**Fix:** Keep a dict of allowed handler modules keyed by the step name (`HANDLERS[name]`) and raise on an unknown key; never pass a formatted or concatenated string to `import_module` / `__import__`.

**References:** [https://docs.python.org/3/library/importlib.html#importlib.import_module](https://docs.python.org/3/library/importlib.html#importlib.import_module)

## keryn.gaps.joined-path-written-unconfined

`warning` python python

A file path is built by joining a base directory with a value and then written, with no check that the result stays under the base, so a value such as `../../.ssh/authorized_keys` writes outside the directory. Resolve the joined path and refuse it unless it starts with the base.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [writes](https://keryn.dev/docs/edges/edge-kinds/#writes) one thing writes to another.

[CWE-22](https://keryn.dev/taxonomies/cwe/#cwe-22-path-traversal) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation)

**Fix:** After joining, `real = os.path.realpath(path)` and check `real.startswith(BASE + os.sep)` (or `Path(...).resolve().is_relative_to(BASE)`) before opening for write; reject or rename anything else.

**References:** [https://docs.python.org/3/library/os.path.html#os.path.realpath](https://docs.python.org/3/library/os.path.html#os.path.realpath) · [https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to)

## keryn.gaps.money-moving-call-without-approval

`warning` python requests, python

A call that moves money or changes a customer account (refund, payment, charge, transfer, payout) is made from a tool with no approval, confirmation or authorisation step in the same function, so the model's say-so is enough to execute it. Gate it behind an explicit human approval of the exact amount and target.

Confidence **P1**: dangerous in context. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [grants](https://keryn.dev/docs/edges/edge-kinds/#grants) authority is given or left unchecked.

[CWE-862](https://keryn.dev/taxonomies/cwe/#cwe-862-missing-authorization) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation) [ASI05](https://keryn.dev/taxonomies/owasp-agentic/#asi05-unexpected-code-execution)

**Fix:** Call an approval helper with the exact ticket, amount and reason before the request and return early when it is refused; log the approver with the action.

**References:** [https://requests.readthedocs.io/en/latest/api/#requests.post](https://requests.readthedocs.io/en/latest/api/#requests.post)

## keryn.gaps.parameter-url-fetched

`warning` python data-flow rule requests, httpx, python

A URL taken straight from a function parameter (typically a tool argument the model fills) is fetched from inside the service, so the model can point the service at internal endpoints or the instance metadata service. Resolve the host and check it against an allowlist before fetching; a well-formedness check is not a destination check.

Confidence **P1**: dangerous in context. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [reaches](https://keryn.dev/docs/edges/edge-kinds/#reaches) content or traffic gets to a thing; it crosses the [`user->prompt`](https://keryn.dev/docs/edges/trust-boundaries/#user-to-prompt) boundary (untrusted input reaching a prompt).

[CWE-918](https://keryn.dev/taxonomies/cwe/#cwe-918-server-side-request-forgery) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation)

**Fix:** Parse the URL, resolve the host, refuse private, loopback and link-local ranges, and allow only hosts on an explicit list (`if urlparse(url).hostname not in ALLOWED_HOSTS: return`); then fetch with redirects disabled.

**References:** [https://requests.readthedocs.io/en/latest/api/#requests.get](https://requests.readthedocs.io/en/latest/api/#requests.get) · [https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse)

## keryn.gaps.table-dispatch-from-model-decision

`warning` python python

Both the tool name and its keyword arguments come from a decision function fed with untrusted content, and they are dispatched through a table (`TOOLS[name](**kwargs)`) with no mediation, so an injected instruction picks the tool and its arguments. Validate the chosen tool against an allowlist and its arguments against a schema before dispatching.

Confidence **P2**: a heuristic or a posture check: confirm by reading the code. False-positive risk **medium**: the pattern is legitimate in many programs, so check each hit.

Edge inspected: [calls](https://keryn.dev/docs/edges/edge-kinds/#calls) one thing invokes another; it crosses the [`model->code`](https://keryn.dev/docs/edges/trust-boundaries/#model-to-code) boundary (model output reaching code, a shell, a query or a page).

[CWE-862](https://keryn.dev/taxonomies/cwe/#cwe-862-missing-authorization) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation) [ASI05](https://keryn.dev/taxonomies/owasp-agentic/#asi05-unexpected-code-execution)

**Fix:** Check `name in ALLOWED_TOOLS`, validate `kwargs` with the tool's schema, and require approval for tools that change state before calling `TOOLS[name](**kwargs)`.

**References:** [https://developers.openai.com/api/docs/guides/function-calling#handling-function-calls](https://developers.openai.com/api/docs/guides/function-calling#handling-function-calls)

## keryn.gaps.text-written-to-tempfile-then-executed

`warning` python python, mcp

Text received by the tool (or produced by the model) is written to a temporary file and the file is then run with the Python interpreter, so the "argv is a literal list" shape hides arbitrary code execution: the payload travels through the file, not the command line. Execute only vetted, committed scripts, or run untrusted code in a sandbox with no network and no credentials.

Confidence **P1**: dangerous in context. False-positive risk **low**: an exact API or flag, so hits are almost always real.

Edge inspected: [calls](https://keryn.dev/docs/edges/edge-kinds/#calls) one thing invokes another; it crosses the [`model->code`](https://keryn.dev/docs/edges/trust-boundaries/#model-to-code) boundary (model output reaching code, a shell, a query or a page).

[CWE-94](https://keryn.dev/taxonomies/cwe/#cwe-94-code-injection) [LLM06](https://keryn.dev/taxonomies/owasp-llm/#llm06-excessive-agency) [ASI02](https://keryn.dev/taxonomies/owasp-agentic/#asi02-tool-misuse-and-exploitation) [ASI05](https://keryn.dev/taxonomies/owasp-agentic/#asi05-unexpected-code-execution)

**Fix:** Do not execute text that arrived as a tool argument or model output. If code execution is the tool's purpose, run it in an isolated sandbox (separate process namespace, no network, no secrets, strict time and memory limits) and never on the host interpreter with the service's credentials.

**References:** [https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile](https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile) · [https://docs.python.org/3/library/subprocess.html#security-considerations](https://docs.python.org/3/library/subprocess.html#security-considerations)
