Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    llm-output-taint-python-v1

    View as Markdown

    26 rules · Keryn Software Licence · rules 0.8.0 · all packs

    How to read a rule
    Severity
    How bad the finding is if it is real: error fails a default gate, warning and info do not.
    Confidence
    P0 exploitable as written; P1 dangerous in context; P2 a heuristic or a posture check: confirm by reading the code.
    False-positive risk
    low: an exact API or flag, so hits are almost always real; medium: the pattern is legitimate in many programs, so check each hit; high: it finds a lead to confirm, so expect benign hits.
    Edge
    The relationship the finding sits on. reaches: content or traffic gets to a thing; calls: one thing invokes another; reads: one thing reads from another; writes: one thing writes to another; sends: data leaves for another party; grants: authority is given or left unchecked; depends_on: something relies on a component it does not verify; trusts: one party believes another without evidence. The boundary it crosses, when there is one: user->prompt (untrusted input reaching a prompt); model->code (model output reaching code, a shell, a query or a page); secret->outside (a secret leaving the trusted side); supply-chain (code, models or prompts pulled in from outside).
    Taxonomy chips
    References for readers who work in OWASP LLM 2025, OWASP Agentic, CWE or NIST AI RMF. They say where a rule belongs, not that a framework is satisfied.

    seceng.llm05.taint.model-output-to-eval-python

    Section titled “seceng.llm05.taint.model-output-to-eval-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output (completion text or tool-call arguments) reaches eval/exec/compile. A prompt-injected response becomes arbitrary Python on this host; parse it with json.loads/ast.literal_eval or map it through an allow-list instead of executing it.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-94 CWE-95 LLM05 ASI05

    Fix: Never execute model text. If the model must choose an operation, have it return a name that you look up in a fixed dict of callables, or a JSON object you validate with pydantic; if you need literal data use ast.literal_eval or json.loads. If generated code execution is the product, run it in a separate sandbox (container or microVM with no credentials and no network) rather than in-process.

    References: https://docs.python.org/3/builtins/functions.html#eval · https://docs.python.org/3/library/ast.html#ast.literal_eval · https://developers.openai.com/api/reference/resources/chat · https://platform.claude.com/docs/en/api/messages · https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as calc/formula.py and scan it with keryn scan calc --verbose; the finding below appears.

    calc/formula.py
    from openai import OpenAI
    client = OpenAI()
    def evaluate_formula(description: str, values: dict) -> float:
    completion = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
    {"role": "system", "content": "Write a Python expression using the given variable names."},
    {"role": "user", "content": description},
    ],
    )
    expr = completion.choices[0].message.content.strip()
    return float(eval(expr, {"__builtins__": {}}, values))

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-eval-python at line 15 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-subprocess-python

    Section titled “seceng.llm05.taint.model-output-to-subprocess-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output selects a process executable or a shell program. Keep the executable and program fixed; pass untrusted values only as data arguments, and separately constrain utility-specific options. Shape-only schema validation does not remove command taint.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-78 CWE-88 LLM05 ASI05 ASI02

    Fix: Use a fixed executable with shell=False and data-only argv positions. When a shell program is necessary, constrain the program through a fixed trusted mapping. Do not treat a generic string schema, function name or approval prompt as command sanitization. Utility options and interpreter code arguments require separate checks.

    References: https://docs.python.org/3/library/subprocess.html#security-considerations · https://docs.python.org/3/library/shlex.html#shlex.quote · https://developers.openai.com/api/docs/guides/function-calling · https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as ops/shell_helper.py and scan it with keryn scan ops --verbose; the finding below appears.

    ops/shell_helper.py
    import subprocess
    from openai import OpenAI
    client = OpenAI()
    def run_suggested_fix(error_log: str) -> str:
    """Ask the model for a one-line shell fix and run it for the on-call engineer."""
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
    {"role": "system", "content": "Reply with one shell command only."},
    {"role": "user", "content": error_log[-2000:]},
    ],
    )
    command = response.choices[0].message.content.strip()
    result = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=60)
    return result.stdout or result.stderr

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-subprocess-python at line 18 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-os-command-python

    Section titled “seceng.llm05.taint.model-output-to-os-command-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output reaches os.system/os.popen/os.exec*/os.spawn*. These run through a shell or replace the process, so a prompt-injected response becomes command execution; replace with subprocess and a fixed argv plus an allow-list.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-78 LLM05 ASI05 ASI02

    Fix: Remove os.system/os.popen. Use a fixed executable and shell=False with data-only argv positions. Constrain any execution-capable option separately. A shape-only schema or quoting an entire model-selected command does not authorize that command.

    References: https://docs.python.org/3/library/os.html#os.system · https://docs.python.org/3/library/shlex.html#shlex.quote · https://developers.openai.com/api/reference/resources/chat · https://platform.claude.com/docs/en/api/messages

    seceng.llm05.taint.model-output-to-async-subprocess-python

    Section titled “seceng.llm05.taint.model-output-to-async-subprocess-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, asyncio, anyio

    Model output reaches asyncio/anyio process creation. create_subprocess_shell interprets shell syntax and create_subprocess_exec runs whatever program the model names; select the program from an allow-list and quote or validate arguments.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-78 CWE-88 LLM05 ASI05 ASI02

    Fix: Use create_subprocess_exec with a constant program and documented data-only argument positions. Keep interpreter code arguments fixed; shape validation and shell quoting do not make an attacker-selected program safe. Bind any required approval to the actual action and final arguments.

    References: https://docs.python.org/3/library/asyncio-subprocess.html · https://anyio.readthedocs.io/en/stable/subprocesses.html · https://developers.openai.com/api/reference/resources/chat · https://platform.claude.com/docs/en/api/messages

    seceng.llm05.taint.model-output-to-remote-command-python

    Section titled “seceng.llm05.taint.model-output-to-remote-command-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, paramiko, fabric, asyncssh, docker

    Model output reaches a remote command runner (paramiko exec_command, fabric/asyncssh run, docker exec_run/containers.run). A prompt-injected response executes on the remote host or container; select commands from an allow-list.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-78 LLM05 ASI05 ASI02

    Fix: Expose fixed remote operations and validate parameters for their exact argument positions. Keep executable and shell program fixed; quote only data operands in a supported shell context. Quoting the whole model-selected command does not authorize it, and a shape-only schema is not command sanitization. Use a least-privilege remote account.

    References: https://docs.paramiko.org/en/stable/api/client.html · https://docker-py.readthedocs.io/en/stable/containers.html · https://docs.fabfile.org/en/latest/api/connection.html · https://developers.openai.com/api/reference/resources/chat · https://platform.claude.com/docs/en/api/messages

    seceng.llm05.taint.model-output-to-sql-python

    Section titled “seceng.llm05.taint.model-output-to-sql-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, sqlalchemy, psycopg, asyncpg, sqlite3, django, pandas

    Model output is built into SQL text (DB-API execute, SQLAlchemy text/exec_driver_sql, asyncpg, psycopg sql.SQL, Django raw/RawSQL, pandas read_sql). Prompt injection turns into SQL injection; use bound parameters and allow-list identifiers.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-89 LLM05 ASI05 ASI02

    Fix: Keep the SQL text constant and pass model-derived values as bound parameters (%s/:name/$1). Identifiers chosen by the model (table, column, sort order) must come from an allow-list or be composed with psycopg.sql.Identifier. For text-to-SQL features, run the generated query on a read-only role with row limits and a statement allow-list (SELECT only), never on the application’s write connection.

    References: https://docs.sqlalchemy.org/en/20/core/sqlelement.html#sqlalchemy.sql.expression.text · https://www.psycopg.org/psycopg3/docs/api/sql.html · https://magicstack.github.io/asyncpg/current/api/index.html · https://docs.python.org/3/library/sqlite3.html#sqlite3-placeholders · https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as app/reports.py and scan it with keryn scan app --verbose; the finding below appears.

    app/reports.py
    import sqlite3
    from anthropic import Anthropic
    client = Anthropic()
    db = sqlite3.connect("analytics.db")
    def answer_question(question: str):
    message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=400,
    system="Translate the question into a single SQLite SELECT statement. Output SQL only.",
    messages=[{"role": "user", "content": question}],
    )
    sql = message.content[0].text
    cur = db.cursor()
    cur.execute(sql)
    return cur.fetchall()

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-sql-python at line 18 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-nosql-query-python

    Section titled “seceng.llm05.taint.model-output-to-nosql-query-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, pymongo, motor

    Model output becomes a MongoDB filter, pipeline or $where expression. A model-produced dict can inject query operators ($ne, $regex, $where JavaScript) and widen or mutate the result set; build the filter yourself from validated scalar fields.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-943 LLM05 ASI02 ASI05

    Fix: Do not pass a model-produced dict to the driver. Validate the model output with a pydantic model whose fields are plain scalars (str/int/enum), then build the filter literal yourself ({“name”: args.name}); never accept $where or other operators from the model, and scope every query by the caller’s tenant.

    References: https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html · https://www.mongodb.com/docs/manual/reference/operator/query/where/ · https://pydantic.dev/docs/validation/latest/concepts/models/#validating-data

    seceng.llm05.taint.model-output-to-url-fetch-python

    Section titled “seceng.llm05.taint.model-output-to-url-fetch-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, requests, httpx, aiohttp, urllib

    Model output chooses the URL of a server-side HTTP request (requests, httpx, urllib, aiohttp). A prompt-injected response can make the app fetch internal services or cloud metadata, or exfiltrate data in the query string; check the parsed host against an allow-list first.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-918 LLM05 ASI02

    Fix: Parse the URL with urllib.parse.urlparse, require https, and compare the hostname against an explicit allow-list before fetching; resolve DNS and reject private, loopback and link-local addresses, disable redirects or re-check each hop, and run fetch tools from an egress-restricted network without cloud credentials.

    References: https://requests.readthedocs.io/en/latest/api/ · https://www.python-httpx.org/api/ · https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse · https://developers.openai.com/api/docs/guides/function-calling

    seceng.llm05.taint.model-output-to-browser-navigation-python

    Section titled “seceng.llm05.taint.model-output-to-browser-navigation-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, playwright, selenium

    Model output chooses the page a headless browser navigates to (Playwright page.goto, Selenium driver.get, webbrowser.open). A prompt-injected agent can browse to internal hosts, file:// URLs or attacker pages carrying further injections; allow-list destinations.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-918 CWE-601 LLM05 ASI02 ASI01

    Fix: Validate the target with urllib.parse.urlparse: allow only http(s) and hosts on an explicit allow-list, block private address ranges, and run the browser in an isolated profile and network without session cookies for internal systems. Treat every page the agent reads as untrusted input to the next model call.

    References: https://playwright.dev/python/docs/api/class-page#page-goto · https://www.selenium.dev/documentation/webdriver/interactions/navigation/ · https://docs.python.org/3/library/webbrowser.html

    seceng.llm05.taint.model-output-to-file-path-python

    Section titled “seceng.llm05.taint.model-output-to-file-path-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, pathlib

    Model output selects a filesystem path that is opened, read, written or listed. A prompt-injected response can use ../ or an absolute path to read secrets or overwrite code; resolve the path and require is_relative_to(base), or use secure_filename/an allow-list.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-22 CWE-73 LLM05 ASI02

    Fix: Join the model value onto a fixed base directory, call .resolve(), and reject the result unless resolved.is_relative_to(base.resolve()); alternatively reduce it to a bare name with werkzeug.utils.secure_filename or pick from an allow-list. Give file tools a dedicated workspace directory and no access to the application’s own source or credentials.

    References: https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to · https://docs.python.org/3/library/pathlib.html#pathlib.Path.resolve · https://werkzeug.palletsprojects.com/en/stable/utils/#werkzeug.utils.secure_filename

    seceng.llm05.taint.model-output-to-filesystem-operation-python

    Section titled “seceng.llm05.taint.model-output-to-filesystem-operation-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, shutil, pathlib

    Model output names a path that is deleted, moved, copied, re-permissioned or extracted (shutil/os/pathlib). A prompt-injected agent can destroy or expose arbitrary files; confine the path to a workspace and require approval for destructive operations.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-22 CWE-73 LLM05 ASI02

    Fix: Resolve every model-supplied path against a dedicated workspace and reject it unless is_relative_to(workspace); never let the model choose the destination of rmtree/move/chmod outside that workspace, and put destructive file operations behind an explicit human approval step.

    References: https://docs.python.org/3/library/shutil.html · https://docs.python.org/3/library/os.html#files-and-directories · https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.is_relative_to

    seceng.llm05.taint.model-output-to-send-file-python

    Section titled “seceng.llm05.taint.model-output-to-send-file-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, flask, fastapi, starlette

    Model output selects the file served in an HTTP response (Flask send_file/send_from_directory, Starlette/FastAPI FileResponse). A prompt-injected response can make the app return any readable file to the client; resolve and confine the path first.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-22 CWE-552 LLM05 ASI02

    Fix: Serve generated files from one export directory: take only a file name from the model, pass it as the filename argument of send_from_directory (which joins safely), or resolve it and require is_relative_to(export_dir). Check that the file belongs to the requesting user.

    References: https://flask.palletsprojects.com/en/stable/api/#flask.send_file · https://flask.palletsprojects.com/en/stable/api/#flask.send_from_directory · https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse

    seceng.llm05.taint.model-output-to-deserialization-python

    Section titled “seceng.llm05.taint.model-output-to-deserialization-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, pickle, pyyaml

    Model output reaches pickle/dill/marshal/jsonpickle or a non-safe yaml.load. These formats construct arbitrary objects, so a crafted completion becomes code execution; parse model output with json.loads or yaml.safe_load and validate it.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-502 LLM05 ASI05

    Fix: Never unpickle or object-load anything a model produced. Ask the model for JSON (structured outputs) and parse it with json.loads or a pydantic model_validate_json; if YAML is required use yaml.safe_load or Loader=yaml.SafeLoader, then validate the resulting data against a schema.

    References: https://docs.python.org/3/library/pickle.html · https://pyyaml.org/wiki/PyYAMLDocumentation · https://pydantic.dev/docs/validation/latest/concepts/json/

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as jobs/restore.py and scan it with keryn scan jobs --verbose; the finding below appears.

    jobs/restore.py
    import pickle
    import base64
    from openai import OpenAI
    client = OpenAI()
    def restore_state(snapshot_hint: str):
    reply = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Return the base64 state blob for: {snapshot_hint}"}],
    )
    blob = reply.choices[0].message.content.strip()
    return pickle.loads(base64.b64decode(blob))

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-deserialization-python at line 15 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-template-python

    Section titled “seceng.llm05.taint.model-output-to-template-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, jinja2, flask, mako, django

    Model output is compiled as a template (jinja2 Template/from_string, Flask render_template_string, Mako, Django Template) or used as a str.format format string. Template syntax in a prompt-injected response is server-side template injection; pass model text as a render variable instead.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-1336 CWE-94 CWE-134 LLM05 ASI05

    Fix: Keep template source constant and supply the model’s text as a variable ({{ answer }} with autoescaping on). If users or the model must author templates, compile them in jinja2.sandbox.ImmutableSandboxedEnvironment and choose them from an allow-list; never call .format on a model-produced string.

    References: https://jinja.palletsprojects.com/en/stable/sandbox/ · https://flask.palletsprojects.com/en/stable/api/#flask.render_template_string · https://docs.python.org/3/library/string.html#format-string-syntax

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as web/greeting.py and scan it with keryn scan web --verbose; the finding below appears.

    web/greeting.py
    from flask import render_template_string
    from openai import OpenAI
    client = OpenAI()
    def render_welcome(user_name: str, tone: str) -> str:
    resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Write a {tone} welcome banner for {user_name} in HTML."}],
    )
    banner = resp.choices[0].message.content
    return render_template_string(banner, user=user_name)

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-template-python at line 13 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-html-response-python

    Section titled “seceng.llm05.taint.model-output-to-html-response-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, flask, django, fastapi, starlette

    Model output is written into an HTML HTTP response (Flask route return/make_response/Response, Django HttpResponse, FastAPI/Starlette HTMLResponse) without escaping. Markup or script in a prompt-injected answer runs in the user’s browser; escape or sanitize it, or render it through an autoescaping template.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-79 LLM05 ASI09

    Fix: Return model text as JSON or render it through an autoescaping template. If HTML output is required (e.g. rendered Markdown), sanitize the final HTML with nh3.clean or bleach.clean using a strict tag/attribute allow-list, otherwise html.escape/markupsafe.escape it, and send a Content-Security-Policy that blocks inline script.

    References: https://flask.palletsprojects.com/en/stable/web-security/ · https://docs.djangoproject.com/en/stable/ref/request-response/#httpresponse-objects · https://fastapi.tiangolo.com/advanced/custom-response/#htmlresponse · https://docs.python.org/3/library/html.html#html.escape · https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

    seceng.llm05.taint.model-output-to-safe-markup-python

    Section titled “seceng.llm05.taint.model-output-to-safe-markup-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, flask, django, jinja2, markupsafe

    Model output is wrapped in Markup/mark_safe/SafeString/format_html’s format string, which disables template autoescaping for it. Script or event-handler markup in a prompt-injected answer will execute in the browser; escape or sanitize before marking safe.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-79 LLM05 ASI09

    Fix: Do not mark model output safe. Let the template autoescape it; if you render Markdown to HTML, run the HTML through nh3.clean/bleach.clean with a strict allow-list before Markup/mark_safe, and with format_html keep the format string constant and pass the model text as an argument.

    References: https://markupsafe.palletsprojects.com/en/stable/escaping/ · https://docs.djangoproject.com/en/stable/ref/utils/#django.utils.safestring.mark_safe · https://docs.djangoproject.com/en/stable/ref/utils/#django.utils.html.format_html · https://nh3.readthedocs.io/en/latest/

    seceng.llm05.taint.model-output-to-redirect-python

    Section titled “seceng.llm05.taint.model-output-to-redirect-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, flask, django, fastapi, starlette

    Model output chooses an HTTP redirect target (Flask redirect, Django redirect/HttpResponseRedirect, FastAPI/Starlette RedirectResponse). A prompt-injected answer can send users to a phishing or token-harvesting site; check the target with url_has_allowed_host_and_scheme or a host allow-list.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-601 LLM05 ASI09

    Fix: Let the model pick a destination key that maps to a URL you control, or accept only relative paths on your own site; for absolute URLs require django.utils.http.url_has_allowed_host_and_scheme (or an equivalent urlparse hostname allow-list plus https) before redirecting.

    References: https://flask.palletsprojects.com/en/stable/api/#flask.redirect · https://docs.djangoproject.com/en/stable/topics/http/shortcuts/#redirect · https://fastapi.tiangolo.com/advanced/custom-response/#redirectresponse · https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse

    seceng.llm05.taint.model-output-to-os-environ-python

    Section titled “seceng.llm05.taint.model-output-to-os-environ-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output sets a process environment variable (os.environ[…] =, putenv, update, setdefault, or a subprocess env=). Variables such as PATH, LD_PRELOAD, PYTHONPATH or proxy and credential settings let a prompt-injected response hijack later code or requests; allow-list the keys and values.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-15 CWE-454 LLM05 ASI05 ASI02

    Fix: Do not let the model write the environment. If a tool needs configurable settings, map an allow-listed set of setting names to validated values in a dict passed to that tool only; never write model output into os.environ or a child process env, and never expose PATH, LD_*, PYTHON*, *_PROXY or credential variables to model control.

    References: https://docs.python.org/3/library/os.html#os.environ · https://docs.python.org/3/library/subprocess.html#popen-constructor · https://developers.openai.com/api/docs/guides/function-calling

    seceng.llm05.taint.model-output-to-dynamic-import-python

    Section titled “seceng.llm05.taint.model-output-to-dynamic-import-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output names a module, script or native library to load (importlib.import_module, __import__, runpy, ctypes). Loading attacker-chosen code paths is code execution; select plugins from an allow-list keyed by name.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: depends_on something relies on a component it does not verify; it crosses the supply-chain boundary (code, models or prompts pulled in from outside).

    CWE-470 CWE-94 LLM05 ASI05

    Fix: Register plugins or tools in a dict at import time and look the model’s choice up in it; reject unknown names. Never derive a module path, file path or shared-library name from model output.

    References: https://docs.python.org/3/library/importlib.html#importlib.import_module · https://docs.python.org/3/library/runpy.html · https://docs.python.org/3/library/ctypes.html

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as plugins/loader.py and scan it with keryn scan plugins --verbose; the finding below appears.

    plugins/loader.py
    import importlib
    from openai import OpenAI
    client = OpenAI()
    def load_plugin_for(task: str):
    resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Name the plugin module that handles: {task}"}],
    )
    module_name = resp.choices[0].message.content.strip()
    module = importlib.import_module(module_name)
    return module.Plugin()

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm05.taint.model-output-to-dynamic-import-python at line 14 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm05.taint.model-output-to-reflection-dispatch-python

    Section titled “seceng.llm05.taint.model-output-to-reflection-dispatch-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output (often a tool-call function name) is used as an attribute or global name in getattr/setattr/globals()/operator.methodcaller. A prompt-injected name can reach private methods or dangerous module functions instead of the intended tool; dispatch through an explicit registry.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-470 LLM05 ASI02 ASI05

    Fix: Keep a TOOLS = {“name”: callable} registry built from the same definitions you send to the model, look the tool-call name up there and reject unknown names; validate arguments with the tool’s schema before calling. Never resolve a model-chosen name with getattr on a module, self or globals().

    References: https://docs.python.org/3/builtins/functions.html#getattr · https://developers.openai.com/api/docs/guides/function-calling · https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview

    seceng.llm05.taint.model-output-to-expression-eval-python

    Section titled “seceng.llm05.taint.model-output-to-expression-eval-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, pandas, numexpr, sympy

    Model output reaches an expression evaluator (pandas DataFrame.query/eval, pandas.eval, numexpr, sympy.sympify/parse_expr). These evaluate Python-like expressions and have known escapes to code execution; parse the model’s intent into a validated filter instead.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-94 CWE-95 LLM05 ASI05

    Fix: Have the model return structured filter terms (column, operator, value) validated by a pydantic model with an allow-listed column set and operator enum, then apply them with boolean indexing; never hand model text to DataFrame.query/eval, numexpr or sympify. Run data-analysis agents that need generated code in an isolated sandbox.

    References: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html · https://pandas.pydata.org/docs/reference/api/pandas.eval.html · https://docs.sympy.org/latest/modules/core.html#sympy.core.sympify.sympify

    seceng.llm05.taint.model-output-to-regex-python

    Section titled “seceng.llm05.taint.model-output-to-regex-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama

    Model output is compiled as a regular expression (re/regex module). A prompt-injected pattern can cause catastrophic backtracking (ReDoS) or match far more than intended; escape it with re.escape when it is meant literally.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-1333 CWE-625 LLM05 LLM10 ASI08

    Fix: Treat model text as a literal: re.escape it before building a pattern. If the model must supply a real pattern, choose from an allow-list of vetted patterns or use a linear-time engine (google-re2) with a timeout and input size cap.

    References: https://docs.python.org/3/library/re.html#re.escape · https://pypi.org/project/regex/ · https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

    seceng.llm05.taint.model-output-to-code-interpreter-tool-python

    Section titled “seceng.llm05.taint.model-output-to-code-interpreter-tool-python”

    error python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, langchain-experimental

    Model output is run through an in-process code or shell tool (LangChain PythonREPL/PythonREPLTool/PythonAstREPLTool, ShellTool, BashProcess). Any prompt injection that reaches the model becomes code execution with the app’s privileges; move execution into an isolated sandbox.

    Confidence P0: exploitable as written. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: calls one thing invokes another; it crosses the model->code boundary (model output reaching code, a shell, a query or a page).

    CWE-94 CWE-78 LLM05 ASI05 ASI02

    Fix: Do not run generated code in the application process. Use a remote sandbox (container or microVM per session, no credentials, no network egress, CPU/memory/time limits), pass only the code string across the boundary, and require human approval before running code that touches external systems.

    References: https://reference.langchain.com/python/langchain-experimental/utilities/python/PythonREPL · https://docs.langchain.com/oss/python/integrations/providers/overview · https://docs.langchain.com/oss/python/langchain/overview

    seceng.llm01.taint.request-to-system-prompt-python

    Section titled “seceng.llm01.taint.request-to-system-prompt-python”

    warning python data-flow rule python, openai, anthropic, google-genai, aws-bedrock, litellm, langchain, llamaindex, ollama, flask, django, fastapi

    HTTP request data (Flask/Django/FastAPI parameters, body, headers or uploads) flows into the system/developer prompt. The system slot carries the highest instruction priority, so a caller can rewrite the assistant’s rules; keep user text in a user message and allow-list any value that must parameterise the system prompt.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: reaches content or traffic gets to a thing; it crosses the user->prompt boundary (untrusted input reaching a prompt).

    CWE-74 CWE-1427 LLM01 ASI01

    Fix: Keep the system prompt constant and send request text only in user-role messages. When the system prompt must vary (language, persona, tenant), map the request value through an allow-list or enum to a fixed snippet, or cast it to int/float; never interpolate free text from the request into system, developer or instructions fields.

    References: https://developers.openai.com/api/reference/resources/responses/methods/create#responses-create-instructions · https://platform.claude.com/docs/en/api/messages#body-system · https://ai.google.dev/gemini-api/docs/text-generation#system-instructions · https://docs.langchain.com/oss/python/langchain/overview#systemmessage · https://reference.langchain.com/python/langchain-core · https://genai.owasp.org/llmrisk/llm01-prompt-injection/

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as api/chat.py and scan it with keryn scan api --verbose; the finding below appears.

    api/chat.py
    from flask import Flask, request, jsonify
    from openai import OpenAI
    app = Flask(__name__)
    client = OpenAI()
    @app.post("/chat")
    def chat():
    persona = request.json.get("persona", "friendly assistant")
    question = request.json["question"]
    result = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
    {"role": "system", "content": f"You are a {persona} for Example Shop customers."},
    {"role": "user", "content": question},
    ],
    )
    return jsonify(answer=result.choices[0].message.content)

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm01.taint.request-to-system-prompt-python at line 15 (warning, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    seceng.llm01.taint.retrieved-content-to-system-prompt-python

    Section titled “seceng.llm01.taint.retrieved-content-to-system-prompt-python”

    warning python data-flow rule python, requests, httpx, langchain, llamaindex, mcp

    Fetched web content, retrieved RAG documents, document-loader output or agent memory flows into the system/developer prompt. Text planted in those sources is promoted to top-priority instructions (indirect prompt injection); pass retrieved content as delimited user-role context, never as system instructions.

    Confidence P1: dangerous in context. False-positive risk low: an exact API or flag, so hits are almost always real.

    Edge inspected: reaches content or traffic gets to a thing; it crosses the user->prompt boundary (untrusted input reaching a prompt).

    CWE-74 CWE-1427 LLM01 ASI01 ASI06

    Fix: Keep the system prompt constant and place retrieved documents, web pages and memory in a user-role (or tool-role) message inside clear delimiters, with an instruction that the content is data. Only values reduced to an allow-listed set (document type, source name) may parameterise the system prompt. Record document provenance and filter retrieval by tenant and trust level.

    References: https://developers.openai.com/api/reference/resources/responses/methods/create#responses-create-instructions · https://platform.claude.com/docs/en/api/messages#body-system · https://ai.google.dev/gemini-api/docs/text-generation#system-instructions · https://docs.langchain.com/oss/python/langchain/overview#systemmessage · https://reference.langchain.com/python/langchain-core · https://docs.langchain.com/oss/python/langchain/overview · https://developers.llamaindex.ai/python/framework/module_guides/querying/retriever/ · https://genai.owasp.org/llmrisk/llm01-prompt-injection/

    seceng.llm01.taint.tool-result-to-tool-args-python

    Section titled “seceng.llm01.taint.tool-result-to-tool-args-python”

    error python data-flow rule python, mcp, langchain

    The result of one tool call (MCP call_tool, a LangChain tool, or a registry-dispatched function) is fed directly into another tool’s arguments or a subprocess without passing a validation boundary. Content returned by a tool is attacker-reachable data, so this chains an indirect prompt injection into tool misuse; validate the intermediate result against the next tool’s schema or an allow-list.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: reaches content or traffic gets to a thing; it crosses the user->prompt boundary (untrusted input reaching a prompt).

    CWE-74 CWE-78 LLM01 LLM06 ASI02 ASI01

    Fix: Treat every tool result as untrusted. Before it drives another tool or a command, validate it with the next tool’s pydantic/jsonschema input model (enums, patterns, length limits) or reduce it to an allow-listed value, shlex.quote anything that reaches a shell, and require human approval when the chained tool has side effects.

    References: https://modelcontextprotocol.io/specification/2025-06-18/server/tools · https://github.com/modelcontextprotocol/python-sdk · https://docs.langchain.com/oss/python/langchain/overview · https://pydantic.dev/docs/validation/latest/concepts/models/#validating-data · https://genai.owasp.org/llmrisk/llm01-prompt-injection/

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as agent/relay.py and scan it with keryn scan agent --verbose; the finding below appears.

    agent/relay.py
    from mcp import ClientSession
    async def forward_lookup(search: ClientSession, files: ClientSession, query: str):
    found = await search.call_tool("find_document", {"query": query})
    doc_ref = found.content[0].text
    return await files.call_tool("read_file", {"path": doc_ref})

    Checked with Keryn’s browser engine when this page was built: it reports seceng.llm01.taint.tool-result-to-tool-args-python at line 7 (error, proof 2). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.