Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    eval-trace-pii-exposure-v1

    View as Markdown

    5 rules · Keryn Software Licence · rules 0.8.0 · all packs

    How to read a rule
    Severity
    How bad the finding is if it is real: error fails a default gate, warning and info do not.
    Confidence
    P0 exploitable as written; P1 dangerous in context; P2 a heuristic or a posture check: confirm by reading the code.
    False-positive risk
    low: an exact API or flag, so hits are almost always real; medium: the pattern is legitimate in many programs, so check each hit; high: it finds a lead to confirm, so expect benign hits.
    Edge
    The relationship the finding sits on. reaches: content or traffic gets to a thing; calls: one thing invokes another; reads: one thing reads from another; writes: one thing writes to another; sends: data leaves for another party; grants: authority is given or left unchecked; depends_on: something relies on a component it does not verify; trusts: one party believes another without evidence. The boundary it crosses, when there is one: user->prompt (untrusted input reaching a prompt); model->code (model output reaching code, a shell, a query or a page); secret->outside (a secret leaving the trusted side); supply-chain (code, models or prompts pulled in from outside).
    Taxonomy chips
    References for readers who work in OWASP LLM 2025, OWASP Agentic, CWE or NIST AI RMF. They say where a rule belongs, not that a framework is satisfied.

    seceng.eval.langfuse-raw-input-output-logging

    Section titled “seceng.eval.langfuse-raw-input-output-logging”

    info typescript, javascript, python eval-trace-pii-exposure

    Langfuse trace or generation call captures raw input/output without a redaction step. User messages, prompts, and completions may contain PII that should not be stored unredacted.

    Confidence P1: dangerous in context.

    Edge inspected: sends data leaves for another party; it crosses the secret->outside boundary (a secret leaving the trusted side).

    CWE-312 CWE-532 LLM02 LLM07 ASI05

    warning python, typescript, javascript eval-trace-pii-exposure

    Arize Phoenix or OpenInference span/trace data is written with raw input/output. Ensure PII redaction is applied before trace export.

    Confidence P1: dangerous in context. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: sends data leaves for another party; it crosses the secret->outside boundary (a secret leaving the trusted side).

    CWE-312 CWE-532 LLM02 ASI06

    info python mlflow

    MLflow experiment logging captures raw prompt or prediction data. Apply PII masking before mlflow.log_text or mlflow.log_param calls on sensitive fields.

    Confidence P0: exploitable as written. False-positive risk medium: the pattern is legitimate in many programs, so check each hit.

    Edge inspected: sends data leaves for another party; it crosses the secret->outside boundary (a secret leaving the trusted side).

    CWE-312 CWE-532 LLM02 ASI06

    Try it: a snippet that triggers this rule

    Written for this page, not taken from any project. Save it as eval/run.py and scan it with keryn scan eval --verbose; the finding below appears.

    eval/run.py
    import mlflow
    def log_run(prompt: str, completion: str, score: float):
    with mlflow.start_run():
    mlflow.log_text(prompt, "prompt.txt")
    mlflow.log_text(completion, "completion.txt")
    mlflow.log_metric("score", score)

    Checked with Keryn’s browser engine when this page was built: it reports seceng.eval.mlflow-log-raw-prompt at line 6 (info, proof 1). The rule’s own logic is not published, so it cannot run in this tab; the pack list in the playground shows what does.

    info python eval-trace-pii-exposure

    Weights & Biases Weave trace call logs model inputs/outputs. Confirm PII redaction is applied before trace publication.

    Confidence P0: exploitable as written.

    Edge inspected: sends data leaves for another party; it crosses the secret->outside boundary (a secret leaving the trusted side).

    CWE-312 LLM02 ASI05 ASI10

    info python, typescript, javascript mlflow

    Eval or observability dashboard endpoint appears to be served without authentication. Langfuse, Phoenix, and MLflow UIs may expose raw prompts and user data publicly.

    Confidence P0: exploitable as written.

    Edge inspected: sends data leaves for another party; it crosses the secret->outside boundary (a secret leaving the trusted side).

    CWE-306 CWE-312 LLM07 LLM02 ASI05 ASI01 ASI03