Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    promptfoo configs: keryn-eval vs promptfoo 0.125.1

    View as Markdown

    Given the same promptfoo configs and the same model responses, does keryn-eval reach the same verdict as promptfoo on every test cell?

    Evaluation detail Description
    Keryn engine keryn-eval at 5d19970c3589: Keryn’s adversarial evaluation kernel: imports promptfoo configs and runs deterministic assertions, natively and in WebAssembly.
    Reference promptfoo 0.125.1. Oracle only: the pinned npm package run offline on a development machine; never shipped.
    Corpus Every promptfooconfig*.yaml under promptfoo’s examples/ (no selection), promptfoo/promptfoo at 8569e72ed6cf, restated from docs/adversarial-eval/results/compatibility-results.json
    Comparison Per cell (test x prompt x provider): pass/fail, score, failure reason and error class. Providers are replaced by deterministic variant providers on both sides, and the engine replays exactly the responses promptfoo recorded, so the comparison is of assertion and scoring semantics. A config is identical when every cell executed and agreed.
    Receipt promptfoo.json, sha256 804e4f630ac21456
    Measure Value Count Denominator
    Configs fully executed with every cell agreeing 69.7% 302 / 433 promptfooconfig files in the pinned upstream examples/
    Configs whose executed cells all agree 100% 317 / 317 configs where Keryn executed at least one cell
    Executed cells with the same verdict 100% 13,104 / 13,104 cells (test x prompt x provider) both sides executed
    Cells Keryn executes 60.3% 13,104 / 21,728 all cells in the corpus, including configs promptfoo ran but Keryn does not support
    Deterministic assertions with the same pass/fail 100% 9,740 / 9,740 assertions in executed cells

    Cases by status: 302 identical, 16 no reference, 115 unsupported.

    The benchmark grades 8 fixed text pairs through the native assertion API. Both revisions matched 24 of 24 expected verdicts per batch and checked scores and failure distances. The timing covers this assertion workload only.

    Implementation Revision Median time per batch
    Baseline 43d555bf59cc 8.454 ms
    Optimized eeb090f476d4 0.422 ms

    The optimized run was 20.02× faster on Linux x86_64 with Rust 1.94.0. Each median uses 9 samples of 10 batches after warm-up. Download timing receipt.

    • Deterministic assertions and inline JavaScript (assertions and transforms, in the subset the keryn-eval interpreter supports; anything outside it is unsupported, never pass or fail). Model-graded, Python, similarity and script-file JavaScript assertions are not executed and count as unsupported.
    • No live model is called on either side; this measures config import, templating, assertions and scoring, not attack success against a model.
    • 15 configs ran in part: every cell Keryn executed agreed, but some cells use features it does not run. They are counted as unsupported, not identical.
    • Restated from the committed differential; numbers are not recomputed from a new run.
    Config Status Cells Matching Differing Unsupported
    examples/amazon-bedrock/agents/promptfooconfig.multi-agent.yaml no reference — — — —
    examples/amazon-bedrock/models/promptfooconfig.converse-images.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.kb.yaml unsupported 24 20 0 4
    examples/amazon-bedrock/models/promptfooconfig.llama-vision.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.nova-sonic.yaml unsupported 8 0 0 8
    examples/amazon-bedrock/models/promptfooconfig.nova.multimodal.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.yaml unsupported 360 352 0 8
    examples/azure/llama/promptfooconfig.yaml unsupported 12 8 0 4
    examples/azure/openai/promptfooconfig.vision.yaml unsupported 4 0 0 4
    examples/azure/openai/promptfooconfig.yaml unsupported 8 4 0 4
    examples/claude-agent-sdk/advanced/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/cyber-espionage/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/plugins/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/skill-comparison/promptfooconfig.yaml unsupported 8 0 0 8
    examples/claude-agent-sdk/skills/promptfooconfig.yaml unsupported 4 0 0 4
    examples/compare-claude-vs-gpt-image/promptfooconfig.yaml unsupported 8 0 0 8
    examples/compare-deepseek-r1-vs-openai-o1/promptfooconfig.yaml unsupported 40 0 0 40
    examples/compare-gpt-model-tiers-mmlu-pro/promptfooconfig.yaml unsupported 400 0 0 400
    examples/config-custom-prompt-function/promptfooconfig.yaml no reference — — — —
    examples/config-dynamic-var/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-executable-prompts/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-extension-api/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-external-defaulttest/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-javascript-test-cases/promptfooconfig.yaml unsupported 48 0 0 48
    examples/config-multi-turn/promptfooconfig.yaml unsupported 28 0 0 28
    examples/config-multiple-translations/promptfooconfig-scenarios.yaml unsupported 72 0 0 72
    examples/config-nunjucks-custom-filters/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-pdf-files/promptfooconfig.yaml unsupported 4 0 0 4
    examples/config-pdf-variables/promptfooconfig.yaml no reference — — — —
    examples/config-prompt-labels/promptfooconfig.yaml unsupported 32 0 0 32
    examples/config-python-test-cases/promptfooconfig.yaml unsupported 40 0 0 40
    examples/config-result-hooks/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-separate-test-configs/promptfooconfig.yaml unsupported 276 264 0 12
    examples/config-stateful-session-management/promptfooconfig.yaml no reference — — — —
    examples/config-store-and-reuse-outputs/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-tests-per-provider/promptfooconfig.yaml no reference — — — —
    examples/config-transform-file/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-assertion-scoring-override/promptfooconfig.yaml unsupported 12 0 0 12
    examples/eval-assertions-generate/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-bert-score/promptfooconfig-advanced.yaml unsupported 4 0 0 4
    examples/eval-bert-score/promptfooconfig.yaml unsupported 8 0 0 8
    examples/eval-context-relevance-grader-output/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-conversation-relevance/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-custom-grader-csv/promptfooconfig.yaml unsupported 32 0 0 32
    examples/eval-image-classification/promptfooconfig.yaml no reference — — — —
    examples/eval-javascript-assert-external/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-javascript-assert-params/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-max-score-selection/promptfooconfig.yaml unsupported 24 0 0 24
    examples/eval-moderation/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-named-metrics/promptfooconfig.yaml unsupported 40 32 0 8
    examples/eval-python-assert/promptfooconfig-external.yaml unsupported 20 0 0 20
    examples/eval-python-assert/promptfooconfig-inline.yaml unsupported 8 0 0 8
    examples/eval-rag/promptfooconfig.yaml unsupported 8 0 0 8
    examples/eval-search-rubric/promptfooconfig.yaml unsupported 32 0 0 32
    examples/eval-select-best/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-sql-validation/promptfooconfig.yaml unsupported 12 0 0 12
    examples/google-aistudio-gemini/promptfooconfig.image.yaml unsupported 4 0 0 4
    examples/google-vertex-tools/promptfooconfig.yaml unsupported 8 0 0 8
    examples/google-vertex/promptfooconfig.image.yaml unsupported 4 0 0 4
    examples/google-vertex/promptfooconfig.yaml unsupported 148 144 0 4
    examples/huggingface/dataset-factuality/promptfooconfig.yaml no reference — — — —
    examples/huggingface/dataset/promptfooconfig.yaml unsupported 2800 0 0 2800
    examples/huggingface/hate-speech-detection/promptfooconfig.yaml unsupported 24 0 0 24
    examples/huggingface/hle/promptfooconfig.yaml no reference — — — —
    examples/huggingface/pii/promptfooconfig.yaml unsupported 8 0 0 8
    examples/huggingface/similarity/promptfooconfig.yaml unsupported 72 0 0 72
    examples/integration-crewai/promptfooconfig.yaml unsupported 12 0 0 12
    examples/integration-docker/basic/promptfooconfig.comparison.advanced.yaml unsupported 4 0 0 4
    examples/integration-docker/code-generation-sandbox/promptfooconfig.yaml unsupported 12 0 0 12
    examples/integration-e2b/promptfooconfig.yaml unsupported 24 0 0 24
    examples/integration-google-sheets/promptfooconfig.yaml unsupported 24 0 0 24
    examples/integration-inspect-osworld/promptfooconfig.full.yaml no reference — — — —
    examples/integration-inspect-osworld/promptfooconfig.yaml no reference — — — —
    examples/integration-langgraph/promptfooconfig.yaml unsupported 4 0 0 4
    examples/integration-opentelemetry/javascript/promptfooconfig.yaml unsupported 8 0 0 8
    examples/integration-pydantic-ai/promptfooconfig.yaml unsupported 12 8 0 4
    examples/integration-sharepoint/promptfooconfig.yaml no reference — — — —
    examples/integration-tau/promptfooconfig.yaml unsupported 132 0 0 132
    examples/ismalicious-gate/promptfooconfig.yaml unsupported 12 0 0 12
    examples/mistral/promptfooconfig.aime2024.yaml unsupported 120 0 0 120
    examples/mistral/promptfooconfig.tool-routing.yaml no reference — — — —
    examples/mistral/promptfooconfig.yaml unsupported 36 20 0 16
    examples/openai-agents/promptfooconfig.yaml no reference — — — —
    examples/openai-audio/promptfooconfig.yaml unsupported 8 0 0 8
    examples/openai-codex-sdk/skill-comparison/promptfooconfig.yaml unsupported 8 0 0 8
    examples/openai-codex-sdk/skills/promptfooconfig.tracing.yaml unsupported 4 0 0 4
    examples/openai-codex-sdk/skills/promptfooconfig.yaml unsupported 4 0 0 4
    examples/openai-decisions/promptfooconfig.grading.yaml unsupported 8 0 0 8
    examples/openai-function-call/promptfooconfig.yaml unsupported 16 12 0 4
    examples/openai-realtime/promptfooconfig-conversation.yaml unsupported 24 0 0 24
    examples/openai-tools-call/promptfooconfig.yaml unsupported 16 12 0 4
    examples/provider-amazon-sagemaker/promptfooconfig.embedding.yaml unsupported 36 0 0 36
    examples/provider-amazon-sagemaker/promptfooconfig.transform.yaml unsupported 36 0 0 36
    examples/provider-cohere/promptfooconfig.yaml unsupported 4 0 0 4
    examples/provider-custom/embeddings/promptfooconfig.yaml unsupported 4 0 0 4
    examples/provider-elevenlabs/stt/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-fireworks/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-flexai/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-litellm/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-llama-cpp/promptfooconfig.yaml unsupported 40 0 0 40
    examples/provider-model-armor/promptfooconfig.vertex.yaml no reference — — — —
    examples/provider-model-armor/promptfooconfig.yaml unsupported 28 0 0 28
    examples/provider-quiverai/promptfooconfig.pipeline.yaml unsupported 12 0 0 12
    examples/provider-quiverai/promptfooconfig.vectorize.yaml unsupported 8 0 0 8
    examples/provider-quiverai/promptfooconfig.yaml unsupported 16 0 0 16
    examples/provider-replicate/image-generation/promptfooconfig-with-download.yaml unsupported 12 0 0 12
    examples/provider-replicate/llama-guard-moderation/promptfooconfig.yaml unsupported 16 0 0 16
    examples/provider-ruby/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-transformers-local/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-typesafe/promptfooconfig.yaml unsupported 8 0 0 8
    examples/provider-voyage-embeddings/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-api-top-10/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-beavertails/promptfooconfig.yaml unsupported 2800 0 0 2800
    examples/redteam-chatbot/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-cyberseceval/promptfooconfig.yaml unsupported 1004 0 0 1004
    examples/redteam-deepseek-foundation/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-grok-4-political-bias/promptfooconfig.yaml no reference — — — —
    examples/redteam-indirect-web-pwn/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-langchain/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-layer-strategy/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-mcp-agent/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-mcp-auth/promptfooconfig.agent-oauth.yaml unsupported 4 0 0 4
    examples/redteam-mcp-auth/promptfooconfig.oauth.yaml unsupported 4 0 0 4
    examples/redteam-mcp/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-medical-agent/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-multi-input/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-multi-modal/promptfooconfig.static-image.yaml unsupported 4 0 0 4
    examples/redteam-multi-modal/promptfooconfig.unsafebench.yaml unsupported 4 0 0 4
    examples/redteam-tracing-example/promptfooconfig.advanced.yaml no reference — — — —
    examples/simple-csv/promptfooconfig.xlsx.yaml unsupported 32 0 0 32
    examples/simple-test/promptfooconfig.yaml unsupported 40 24 0 16
    Show every case
    Config Status Cells Matching Differing Unsupported
    examples/amazon-bedrock/agents/promptfooconfig.multi-agent.yaml no reference — — — —
    examples/amazon-bedrock/agents/promptfooconfig.yaml identical 48 48 0 0
    examples/amazon-bedrock/models/promptfooconfig.a21.yaml identical 60 60 0 0
    examples/amazon-bedrock/models/promptfooconfig.claude.yaml identical 60 60 0 0
    examples/amazon-bedrock/models/promptfooconfig.converse-comprehensive.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.converse-images.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.converse-mcp.yaml identical 8 8 0 0
    examples/amazon-bedrock/models/promptfooconfig.converse-system.yaml identical 16 16 0 0
    examples/amazon-bedrock/models/promptfooconfig.converse-tools.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.converse.yaml identical 20 20 0 0
    examples/amazon-bedrock/models/promptfooconfig.deepseek.yaml identical 60 60 0 0
    examples/amazon-bedrock/models/promptfooconfig.grok.yaml identical 8 8 0 0
    examples/amazon-bedrock/models/promptfooconfig.inference-profiles-simple.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.inference-profiles.yaml identical 48 48 0 0
    examples/amazon-bedrock/models/promptfooconfig.kb.yaml unsupported 24 20 0 4
    examples/amazon-bedrock/models/promptfooconfig.llama-vision.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.llama.yaml identical 8 8 0 0
    examples/amazon-bedrock/models/promptfooconfig.mantle.yaml identical 8 8 0 0
    examples/amazon-bedrock/models/promptfooconfig.mistral.yaml identical 16 16 0 0
    examples/amazon-bedrock/models/promptfooconfig.nova-2-converse.yaml identical 20 20 0 0
    examples/amazon-bedrock/models/promptfooconfig.nova-2-reasoning.yaml identical 20 20 0 0
    examples/amazon-bedrock/models/promptfooconfig.nova-sonic.yaml unsupported 8 0 0 8
    examples/amazon-bedrock/models/promptfooconfig.nova.multimodal.yaml unsupported 4 0 0 4
    examples/amazon-bedrock/models/promptfooconfig.nova.tool.yaml identical 16 16 0 0
    examples/amazon-bedrock/models/promptfooconfig.nova.yaml identical 60 60 0 0
    examples/amazon-bedrock/models/promptfooconfig.openai-compatible.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.openai-frontier.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.openai-responses.yaml identical 4 4 0 0
    examples/amazon-bedrock/models/promptfooconfig.openai.yaml identical 12 12 0 0
    examples/amazon-bedrock/models/promptfooconfig.qwen.yaml identical 48 48 0 0
    examples/amazon-bedrock/models/promptfooconfig.yaml unsupported 360 352 0 8
    examples/amazon-bedrock/video/promptfooconfig.luma-ray.yaml identical 12 12 0 0
    examples/amazon-bedrock/video/promptfooconfig.nova-reel.yaml identical 8 8 0 0
    examples/anthropic/claude-code-session/promptfooconfig.yaml identical 8 8 0 0
    examples/anthropic/fable-5-coding/promptfooconfig.yaml identical 12 12 0 0
    examples/anthropic/mcp/promptfooconfig.yaml identical 12 12 0 0
    examples/anthropic/memory-tool/promptfooconfig.yaml identical 4 4 0 0
    examples/anthropic/opus-5-coding/promptfooconfig.yaml identical 12 12 0 0
    examples/anthropic/sonnet-5-5/promptfooconfig.yaml identical 8 8 0 0
    examples/anthropic/sonnet-5/promptfooconfig.yaml identical 8 8 0 0
    examples/anthropic/structured-outputs/promptfooconfig.yaml identical 4 4 0 0
    examples/anthropic/web-tools/promptfooconfig.yaml identical 16 16 0 0
    examples/azure-mai/promptfooconfig.vision-judge.yaml identical 8 8 0 0
    examples/azure-mai/promptfooconfig.yaml identical 8 8 0 0
    examples/azure/assistant/promptfooconfig-file-search.yaml identical 20 20 0 0
    examples/azure/assistant/promptfooconfig-function.yaml identical 16 16 0 0
    examples/azure/assistant/promptfooconfig-multi-tool.yaml identical 16 16 0 0
    examples/azure/claude/promptfooconfig.yaml identical 12 12 0 0
    examples/azure/comparison/promptfooconfig.yaml identical 16 16 0 0
    examples/azure/deepseek/promptfooconfig.yaml identical 12 12 0 0
    examples/azure/foundry-agent/promptfooconfig.yaml identical 12 12 0 0
    examples/azure/llama/promptfooconfig.yaml unsupported 12 8 0 4
    examples/azure/mistral/promptfooconfig.yaml identical 12 12 0 0
    examples/azure/openai/promptfooconfig.vision.yaml unsupported 4 0 0 4
    examples/azure/openai/promptfooconfig.yaml unsupported 8 4 0 4
    examples/claude-agent-sdk/advanced-options/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-agent-sdk/advanced/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/ask-user-question/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-agent-sdk/basic/promptfooconfig.tracing.yaml identical 8 8 0 0
    examples/claude-agent-sdk/basic/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-agent-sdk/cyber-espionage/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/mcp/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-agent-sdk/plugins/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/skill-comparison/promptfooconfig.yaml unsupported 8 0 0 8
    examples/claude-agent-sdk/skills/promptfooconfig.yaml unsupported 4 0 0 4
    examples/claude-agent-sdk/structured-output/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-agent-sdk/working-dir/promptfooconfig.yaml identical 4 4 0 0
    examples/claude-thinking/promptfooconfig.yaml identical 16 16 0 0
    examples/claude-vision/promptfooconfig.yaml identical 4 4 0 0
    examples/compare-agentic-sdks/promptfooconfig.yaml identical 4 4 0 0
    examples/compare-claude-vs-gpt-image/promptfooconfig.yaml unsupported 8 0 0 8
    examples/compare-claude-vs-gpt/promptfooconfig.yaml identical 60 60 0 0
    examples/compare-deepseek-r1-vs-openai-o1/promptfooconfig.yaml unsupported 40 0 0 40
    examples/compare-gpt-model-tiers-mmlu-pro/promptfooconfig.yaml unsupported 400 0 0 400
    examples/compare-gpt-model-tiers/promptfooconfig.yaml identical 60 60 0 0
    examples/compare-gpt-reasoning-effort/promptfooconfig.yaml identical 16 16 0 0
    examples/compare-gpt-temperature/promptfooconfig.yaml identical 44 44 0 0
    examples/compare-gpt-vs-claude-vs-gemini/promptfooconfig.yaml identical 12 12 0 0
    examples/compare-llama-vs-gpt/promptfooconfig.yaml identical 32 32 0 0
    examples/compare-mistral-vs-llama/promptfooconfig.yaml identical 32 32 0 0
    examples/compare-open-source-models/promptfooconfig.yaml identical 32 32 0 0
    examples/compare-openai-models/promptfooconfig.yaml identical 60 60 0 0
    examples/compare-phi-vs-llama/promptfooconfig.yaml identical 64 64 0 0
    examples/config-csv-metadata/promptfooconfig.yaml identical 12 12 0 0
    examples/config-custom-prompt-function/promptfooconfig.yaml no reference — — — —
    examples/config-dynamic-var/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-executable-prompts/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-extension-api/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-external-defaulttest/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-external-provider-config/promptfooconfig.yaml identical 16 16 0 0
    examples/config-javascript-test-cases/promptfooconfig.yaml unsupported 48 0 0 48
    examples/config-jsonl-test-cases/promptfooconfig.yaml identical 8 8 0 0
    examples/config-multi-turn/promptfooconfig.yaml unsupported 28 0 0 28
    examples/config-multiple-translations/promptfooconfig-scenarios.yaml unsupported 72 0 0 72
    examples/config-multiple-translations/promptfooconfig.yaml identical 200 200 0 0
    examples/config-multishot/promptfooconfig.yaml identical 32 32 0 0
    examples/config-node-module-package/promptfooconfig.yaml identical 4 4 0 0
    examples/config-nunjucks-custom-filters/promptfooconfig.yaml unsupported 16 0 0 16
    examples/config-pdf-files/promptfooconfig.yaml unsupported 4 0 0 4
    examples/config-pdf-variables/promptfooconfig.yaml no reference — — — —
    examples/config-prompt-labels/promptfooconfig.yaml unsupported 32 0 0 32
    examples/config-prompts-per-model/promptfooconfig.yaml identical 128 128 0 0
    examples/config-python-test-cases/promptfooconfig.yaml unsupported 40 0 0 40
    examples/config-result-hooks/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-retry-testing/promptfooconfig.yaml identical 40 40 0 0
    examples/config-separate-test-configs/promptfooconfig.yaml unsupported 276 264 0 12
    examples/config-stateful-session-management/promptfooconfig.yaml no reference — — — —
    examples/config-store-and-reuse-outputs/promptfooconfig.yaml unsupported 12 0 0 12
    examples/config-structured-outputs/promptfooconfig.yaml identical 16 16 0 0
    examples/config-tests-per-prompt/promptfooconfig.yaml identical 40 40 0 0
    examples/config-tests-per-provider/promptfooconfig.yaml no reference — — — —
    examples/config-transform-file/promptfooconfig.yaml unsupported 4 0 0 4
    examples/config-vars-referencing-vars/promptfooconfig.yaml identical 12 12 0 0
    examples/config-websockets/basic/promptfooconfig.yaml identical 32 32 0 0
    examples/config-websockets/streaming/promptfooconfig.yaml identical 32 32 0 0
    examples/eval-assertion-scoring-override/promptfooconfig.yaml unsupported 12 0 0 12
    examples/eval-assertions-generate/promptfooconfig-minimal.yaml identical 4 4 0 0
    examples/eval-assertions-generate/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-bert-score/promptfooconfig-advanced.yaml unsupported 4 0 0 4
    examples/eval-bert-score/promptfooconfig.yaml unsupported 8 0 0 8
    examples/eval-context-relevance-grader-output/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-conversation-relevance/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-custom-grader-csv/promptfooconfig.yaml unsupported 32 0 0 32
    examples/eval-custom-grading-prompt/promptfooconfig.yaml identical 8 8 0 0
    examples/eval-customer-support-rubric/promptfooconfig.yaml identical 32 32 0 0
    examples/eval-errors-vs-failures/promptfooconfig.yaml identical 12 12 0 0
    examples/eval-f-score/promptfooconfig.yaml identical 400 400 0 0
    examples/eval-g-eval/promptfooconfig.yaml identical 20 20 0 0
    examples/eval-image-classification/promptfooconfig.yaml no reference — — — —
    examples/eval-javascript-assert-external/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-javascript-assert-params/promptfooconfig.test-vars.yaml identical 4 4 0 0
    examples/eval-javascript-assert-params/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-json-output/promptfooconfig.yaml identical 8 8 0 0
    examples/eval-markdown-rendering/promptfooconfig.yaml identical 12 12 0 0
    examples/eval-max-score-selection/promptfooconfig.yaml unsupported 24 0 0 24
    examples/eval-moderation/promptfooconfig.yaml unsupported 4 0 0 4
    examples/eval-named-metrics/promptfooconfig.yaml unsupported 40 32 0 8
    examples/eval-python-assert/promptfooconfig-external.yaml unsupported 20 0 0 20
    examples/eval-python-assert/promptfooconfig-inline.yaml unsupported 8 0 0 8
    examples/eval-rag-full/promptfooconfig.with-asserts.yaml identical 36 36 0 0
    examples/eval-rag-full/promptfooconfig.yaml identical 36 36 0 0
    examples/eval-rag/promptfooconfig.yaml unsupported 8 0 0 8
    examples/eval-search-rubric/promptfooconfig.yaml unsupported 32 0 0 32
    examples/eval-select-best/promptfooconfig.yaml unsupported 16 0 0 16
    examples/eval-self-grading/promptfooconfig.yaml identical 16 16 0 0
    examples/eval-sql-validation/promptfooconfig.yaml unsupported 12 0 0 12
    examples/eval-summarization/promptfooconfig.yaml identical 76 76 0 0
    examples/eval-tool-use/promptfooconfig.bedrock.yaml identical 16 16 0 0
    examples/eval-tool-use/promptfooconfig.yaml identical 16 16 0 0
    examples/getting-started/promptfooconfig.yaml identical 8 8 0 0
    examples/google-aistudio-gemini/promptfooconfig.image.yaml unsupported 4 0 0 4
    examples/google-aistudio-gemini/promptfooconfig.yaml identical 4 4 0 0
    examples/google-aistudio-tools/promptfooconfig.codeexecution.yaml identical 4 4 0 0
    examples/google-aistudio-tools/promptfooconfig.search.yaml identical 12 12 0 0
    examples/google-aistudio-tools/promptfooconfig.urlcontext.yaml identical 4 4 0 0
    examples/google-aistudio-tools/promptfooconfig.yaml identical 8 8 0 0
    examples/google-imagen/promptfooconfig-advanced.yaml identical 4 4 0 0
    examples/google-imagen/promptfooconfig-gemini-grounding.yaml identical 8 8 0 0
    examples/google-imagen/promptfooconfig-gemini.yaml identical 20 20 0 0
    examples/google-imagen/promptfooconfig.yaml identical 20 20 0 0
    examples/google-live-audio/promptfooconfig.yaml identical 8 8 0 0
    examples/google-live/promptfooconfig.yaml identical 8 8 0 0
    examples/google-tts/promptfooconfig.yaml identical 4 4 0 0
    examples/google-vertex-tools/promptfooconfig.yaml unsupported 8 0 0 8
    examples/google-vertex/promptfooconfig.claude.yaml identical 12 12 0 0
    examples/google-vertex/promptfooconfig.gemini.yaml identical 16 16 0 0
    examples/google-vertex/promptfooconfig.image.yaml unsupported 4 0 0 4
    examples/google-vertex/promptfooconfig.llama.yaml identical 16 16 0 0
    examples/google-vertex/promptfooconfig.response-schema.yaml identical 12 12 0 0
    examples/google-vertex/promptfooconfig.search.yaml identical 16 16 0 0
    examples/google-vertex/promptfooconfig.yaml unsupported 148 144 0 4
    examples/google-video/promptfooconfig-extension.yaml identical 8 8 0 0
    examples/google-video/promptfooconfig-image.yaml identical 8 8 0 0
    examples/google-video/promptfooconfig.yaml identical 12 12 0 0
    examples/huggingface/chat/promptfooconfig.yaml identical 4 4 0 0
    examples/huggingface/dataset-factuality/promptfooconfig.yaml no reference — — — —
    examples/huggingface/dataset/promptfooconfig.yaml unsupported 2800 0 0 2800
    examples/huggingface/hate-speech-detection/promptfooconfig.yaml unsupported 24 0 0 24
    examples/huggingface/hle/promptfooconfig.yaml no reference — — — —
    examples/huggingface/inference-endpoint/promptfooconfig.yaml identical 8 8 0 0
    examples/huggingface/pii/promptfooconfig.yaml unsupported 8 0 0 8
    examples/huggingface/similarity/promptfooconfig.yaml unsupported 72 0 0 72
    examples/integration-browser/existing-session/promptfooconfig.yaml identical 4 4 0 0
    examples/integration-browser/headless/promptfooconfig.yaml identical 12 12 0 0
    examples/integration-crewai/promptfooconfig.yaml unsupported 12 0 0 12
    examples/integration-docker/basic/promptfooconfig.comparison.advanced.yaml unsupported 4 0 0 4
    examples/integration-docker/basic/promptfooconfig.comparison.simple.yaml identical 4 4 0 0
    examples/integration-docker/code-generation-sandbox/promptfooconfig.yaml unsupported 12 0 0 12
    examples/integration-e2b/promptfooconfig.yaml unsupported 24 0 0 24
    examples/integration-google-adk/promptfooconfig.workflow.yaml identical 4 4 0 0
    examples/integration-google-adk/promptfooconfig.yaml identical 4 4 0 0
    examples/integration-google-sheets/promptfooconfig.yaml unsupported 24 0 0 24
    examples/integration-helicone/promptfooconfig.yaml identical 20 20 0 0
    examples/integration-inspect-osworld/promptfooconfig.full.yaml no reference — — — —
    examples/integration-inspect-osworld/promptfooconfig.yaml no reference — — — —
    examples/integration-langchain/promptfooconfig.yaml identical 28 28 0 0
    examples/integration-langfuse/promptfooconfig.yaml identical 64 64 0 0
    examples/integration-langgraph/promptfooconfig.yaml unsupported 4 0 0 4
    examples/integration-opentelemetry/built-in/promptfooconfig.yaml identical 8 8 0 0
    examples/integration-opentelemetry/javascript/promptfooconfig.trajectory.yaml identical 4 4 0 0
    examples/integration-opentelemetry/javascript/promptfooconfig.yaml unsupported 8 0 0 8
    examples/integration-opentelemetry/python/promptfooconfig.yaml identical 8 8 0 0
    examples/integration-pydantic-ai/promptfooconfig.yaml unsupported 12 8 0 4
    examples/integration-sharepoint/promptfooconfig.yaml no reference — — — —
    examples/integration-slack/promptfooconfig.yaml identical 32 32 0 0
    examples/integration-strands-agents/promptfooconfig.yaml identical 20 20 0 0
    examples/integration-tau/promptfooconfig.yaml unsupported 132 0 0 132
    examples/integration-vercel/ai-gateway/promptfooconfig.yaml identical 12 12 0 0
    examples/integration-vercel/ai-sdk/promptfooconfig.yaml identical 20 20 0 0
    examples/ismalicious-gate/promptfooconfig.yaml unsupported 12 0 0 12
    examples/mistral/promptfooconfig.aime2024.yaml unsupported 120 0 0 120
    examples/mistral/promptfooconfig.code-generation.yaml identical 16 16 0 0
    examples/mistral/promptfooconfig.comparison.yaml identical 8 8 0 0
    examples/mistral/promptfooconfig.json-mode.yaml identical 12 12 0 0
    examples/mistral/promptfooconfig.multimodal.yaml identical 12 12 0 0
    examples/mistral/promptfooconfig.reasoning.yaml identical 12 12 0 0
    examples/mistral/promptfooconfig.tool-routing.yaml no reference — — — —
    examples/mistral/promptfooconfig.tool-use.yaml identical 8 8 0 0
    examples/mistral/promptfooconfig.yaml unsupported 36 20 0 16
    examples/n8n-agent/promptfooconfig.yaml identical 12 12 0 0
    examples/ollama/promptfooconfig.function-calling.yaml identical 12 12 0 0
    examples/ollama/promptfooconfig.yaml identical 80 80 0 0
    examples/openai-agents-advanced/promptfooconfig.sandbox.yaml identical 4 4 0 0
    examples/openai-agents-advanced/promptfooconfig.yaml identical 12 12 0 0
    examples/openai-agents-api/promptfooconfig.yaml identical 8 8 0 0
    examples/openai-agents-basic/promptfooconfig.yaml identical 36 36 0 0
    examples/openai-agents/promptfooconfig.redteam.coding.yaml identical 4 4 0 0
    examples/openai-agents/promptfooconfig.redteam.yaml identical 4 4 0 0
    examples/openai-agents/promptfooconfig.yaml no reference — — — —
    examples/openai-audio-transcription/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-audio/promptfooconfig.yaml unsupported 8 0 0 8
    examples/openai-azure-comparison/promptfooconfig.yaml identical 12 12 0 0
    examples/openai-chat-history/promptfooconfig.yaml identical 12 12 0 0
    examples/openai-codex-app-server/approval-policy/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-app-server/promptfooconfig.tracing.yaml identical 4 4 0 0
    examples/openai-codex-app-server/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-app-server/review-diff/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-app-server/skills/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-sdk/basic/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-sdk/bedrock/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-sdk/sandbox/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-codex-sdk/skill-comparison/promptfooconfig.yaml unsupported 8 0 0 8
    examples/openai-codex-sdk/skills/promptfooconfig.tracing.yaml unsupported 4 0 0 4
    examples/openai-codex-sdk/skills/promptfooconfig.yaml unsupported 4 0 0 4
    examples/openai-codex-sdk/thread-persistence/promptfooconfig.yaml identical 8 8 0 0
    examples/openai-codex-security/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-compatible-gateway/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-decisions/promptfooconfig.grading.yaml unsupported 8 0 0 8
    examples/openai-decisions/promptfooconfig.yaml identical 8 8 0 0
    examples/openai-eval-factuality/promptfooconfig.yaml identical 8 8 0 0
    examples/openai-function-call/promptfooconfig.yaml unsupported 16 12 0 4
    examples/openai-images/promptfooconfig.yaml identical 20 20 0 0
    examples/openai-live/promptfooconfig.yaml identical 4 4 0 0
    examples/openai-mcp/promptfooconfig.approval.yaml identical 24 24 0 0
    examples/openai-mcp/promptfooconfig.authenticated.yaml identical 36 36 0 0
    examples/openai-mcp/promptfooconfig.yaml identical 48 48 0 0
    examples/openai-multiline-yaml/promptfooconfig.yaml identical 12 12 0 0
    examples/openai-realtime/promptfooconfig-conversation.yaml unsupported 24 0 0 24
    examples/openai-realtime/promptfooconfig-gpt-realtime.yaml identical 16 16 0 0
    examples/openai-realtime/promptfooconfig.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.codex.yaml identical 4 4 0 0
    examples/openai-responses/promptfooconfig.external-format.yaml identical 8 8 0 0
    examples/openai-responses/promptfooconfig.function-call.yaml identical 8 8 0 0
    examples/openai-responses/promptfooconfig.function-callback.yaml identical 36 36 0 0
    examples/openai-responses/promptfooconfig.gpt-5.1.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.gpt-5.2.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.gpt-5.5.yaml identical 8 8 0 0
    examples/openai-responses/promptfooconfig.gpt-5.6.yaml identical 4 4 0 0
    examples/openai-responses/promptfooconfig.gpt-6-astra.yaml identical 8 8 0 0
    examples/openai-responses/promptfooconfig.image.yaml identical 4 4 0 0
    examples/openai-responses/promptfooconfig.mcp.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.prompt-cache.yaml identical 4 4 0 0
    examples/openai-responses/promptfooconfig.reasoning.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.web-search.yaml identical 12 12 0 0
    examples/openai-responses/promptfooconfig.yaml identical 8 8 0 0
    examples/openai-structured-output/promptfooconfig.chat.yaml identical 8 8 0 0
    examples/openai-structured-output/promptfooconfig.responses.yaml identical 8 8 0 0
    examples/openai-tools-call/promptfooconfig.yaml unsupported 16 12 0 4
    examples/openai-video/promptfooconfig.yaml identical 16 16 0 0
    examples/openai-vision/promptfooconfig.yaml identical 4 4 0 0
    examples/openinterpreter/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-a2a/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-abliteration/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-aiml-api/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.embedding.yaml unsupported 36 0 0 36
    examples/provider-amazon-sagemaker/promptfooconfig.jumpstart.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.llama-vs-mistral.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.llama.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.mistral.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.multimodel.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.openai.yaml identical 36 36 0 0
    examples/provider-amazon-sagemaker/promptfooconfig.transform.yaml unsupported 36 0 0 36
    examples/provider-atlascloud/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-cerebras/promptfooconfig-structured.yaml identical 12 12 0 0
    examples/provider-cerebras/promptfooconfig-tools.yaml identical 12 12 0 0
    examples/provider-cerebras/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-cloudflare/gateway/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-cohere-benchmark/promptfooconfig.yaml identical 24 24 0 0
    examples/provider-cohere/promptfooconfig.yaml unsupported 4 0 0 4
    examples/provider-cometapi/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-custom/basic/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-custom/embeddings/promptfooconfig.yaml unsupported 4 0 0 4
    examples/provider-custom/mjs/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-custom/typescript/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-databricks/promptfooconfig.vision.yaml identical 4 4 0 0
    examples/provider-databricks/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-elevenlabs/agents/promptfooconfig.yaml identical 64 64 0 0
    examples/provider-elevenlabs/alignment/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-elevenlabs/isolation/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-elevenlabs/stt/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-elevenlabs/tts-advanced/promptfooconfig.yaml identical 24 24 0 0
    examples/provider-elevenlabs/tts/promptfooconfig.yaml identical 24 24 0 0
    examples/provider-fal/promptfooconfig.yaml identical 28 28 0 0
    examples/provider-fireworks/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-flexai/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-golang/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-groq/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-http/auth-signature-jks/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-http/auth-signature-pfx/promptfooconfig-crt-key.yaml identical 4 4 0 0
    examples/provider-http/auth-signature-pfx/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-http/auth-signature/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-http/basic/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-http/streaming/promptfooconfig.yaml identical 24 24 0 0
    examples/provider-http/tls/promptfooconfig-mock.yaml identical 12 12 0 0
    examples/provider-http/tls/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-hyperbolic/promptfooconfig.audio-generation.yaml identical 20 20 0 0
    examples/provider-hyperbolic/promptfooconfig.image-generation.yaml identical 24 24 0 0
    examples/provider-hyperbolic/promptfooconfig.multimodal.yaml identical 8 8 0 0
    examples/provider-hyperbolic/promptfooconfig.reasoning.yaml identical 20 20 0 0
    examples/provider-hyperbolic/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-litellm/promptfooconfig.yaml unsupported 8 4 0 4
    examples/provider-llama-cpp/promptfooconfig.yaml unsupported 40 0 0 40
    examples/provider-lm-studio/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-meta/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-minimax/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-mlflow-gateway/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-model-armor/promptfooconfig.vertex.yaml no reference — — — —
    examples/provider-model-armor/promptfooconfig.yaml unsupported 28 0 0 28
    examples/provider-modelslab/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-moonshot/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-novita/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-nscale/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-nvidia/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-openclaw/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-opencode-sdk/basic/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-opencode-sdk/permissions/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-opencode-sdk/structured-output/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-opencode-sdk/working-dir/promptfooconfig.yaml identical 4 4 0 0
    examples/provider-orcarouter/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-perplexity/promptfooconfig.research-reasoning.yaml identical 16 16 0 0
    examples/provider-perplexity/promptfooconfig.search-filters.yaml identical 24 24 0 0
    examples/provider-perplexity/promptfooconfig.structured-output.yaml identical 12 12 0 0
    examples/provider-perplexity/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-python/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-quiverai/promptfooconfig.pipeline.yaml unsupported 12 0 0 12
    examples/provider-quiverai/promptfooconfig.vectorize.yaml unsupported 8 0 0 8
    examples/provider-quiverai/promptfooconfig.yaml unsupported 16 0 0 16
    examples/provider-replicate/comprehensive/promptfooconfig.yaml identical 16 16 0 0
    examples/provider-replicate/image-generation/promptfooconfig-with-download.yaml unsupported 12 0 0 12
    examples/provider-replicate/image-generation/promptfooconfig.yaml identical 32 32 0 0
    examples/provider-replicate/lifeboat/promptfooconfig.yaml identical 32 32 0 0
    examples/provider-replicate/llama-guard-moderation/promptfooconfig.yaml unsupported 16 0 0 16
    examples/provider-replicate/llama4-scout/promptfooconfig.yaml identical 56 56 0 0
    examples/provider-replicate/quickstart/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-ruby/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-transformers-local/promptfooconfig.yaml unsupported 12 0 0 12
    examples/provider-truefoundry/promptfooconfig-mcp.yaml identical 12 12 0 0
    examples/provider-truefoundry/promptfooconfig.yaml identical 12 12 0 0
    examples/provider-typesafe/promptfooconfig.questions.yaml identical 12 12 0 0
    examples/provider-typesafe/promptfooconfig.yaml unsupported 8 0 0 8
    examples/provider-voyage-embeddings/promptfooconfig.yaml unsupported 4 0 0 4
    examples/provider-watsonx/promptfooconfig.yaml identical 8 8 0 0
    examples/provider-zai/promptfooconfig.yaml identical 8 8 0 0
    examples/redteam-api-top-10/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-auth/promptfooconfig.api_key.yaml identical 4 4 0 0
    examples/redteam-auth/promptfooconfig.digital_signature.yaml identical 4 4 0 0
    examples/redteam-auth/promptfooconfig.file.yaml identical 4 4 0 0
    examples/redteam-auth/promptfooconfig.oauth.yaml identical 4 4 0 0
    examples/redteam-azure-assistant/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-beavertails/promptfooconfig-subcategories.yaml identical 4 4 0 0
    examples/redteam-beavertails/promptfooconfig.yaml unsupported 2800 0 0 2800
    examples/redteam-bestOfN-strategy/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-chatbot/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-coding-agent/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-context-overrides/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-conversation-ended-signal/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-custom-strategy/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-cyberseceval/promptfooconfig.yaml unsupported 1004 0 0 1004
    examples/redteam-deepseek-foundation/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-deepseek/promptfooconfig.yaml identical 5440 5440 0 0
    examples/redteam-docx-document-upload/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-donotanswer/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-grok-4-political-bias/promptfooconfig.yaml no reference — — — —
    examples/redteam-guardrails/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-harmbench/promptfooconfig.yaml identical 1600 1600 0 0
    examples/redteam-indirect-web-pwn/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-intent-sequence/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-langchain/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-layer-strategy/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-mcp-agent/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-mcp-auth/promptfooconfig.agent-oauth.yaml unsupported 4 0 0 4
    examples/redteam-mcp-auth/promptfooconfig.oauth.yaml unsupported 4 0 0 4
    examples/redteam-mcp/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-medical-agent/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-minimal/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-multi-input/promptfooconfig.yaml unsupported 4 0 0 4
    examples/redteam-multi-modal/promptfooconfig.image-strategy.yaml identical 4 4 0 0
    examples/redteam-multi-modal/promptfooconfig.static-image.yaml unsupported 4 0 0 4
    examples/redteam-multi-modal/promptfooconfig.unsafebench.yaml unsupported 4 0 0 4
    examples/redteam-multi-modal/promptfooconfig.vlguard.yaml identical 4 4 0 0
    examples/redteam-multi-modal/promptfooconfig.vlsu.yaml identical 4 4 0 0
    examples/redteam-ollama/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-provider-override/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-rag/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-starter/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-tracing-example/promptfooconfig-simple.yaml identical 4 4 0 0
    examples/redteam-tracing-example/promptfooconfig.advanced.yaml no reference — — — —
    examples/redteam-tracing-example/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-travel-agent/promptfooconfig.yaml identical 4 4 0 0
    examples/redteam-xstest/promptfooconfig.yaml identical 4 4 0 0
    examples/simple-cli/promptfooconfig.yaml identical 32 32 0 0
    examples/simple-csv/promptfooconfig.xlsx.yaml unsupported 32 0 0 32
    examples/simple-csv/promptfooconfig.yaml identical 32 32 0 0
    examples/simple-mcp/promptfooconfig.yaml identical 56 56 0 0
    examples/simple-test/promptfooconfig.yaml unsupported 40 24 0 16
    examples/vertex-live/promptfooconfig.yaml identical 8 8 0 0
    examples/xai/chat/promptfooconfig.images.yaml identical 8 8 0 0
    examples/xai/chat/promptfooconfig.promptfoo-search.yaml identical 20 20 0 0
    examples/xai/chat/promptfooconfig.responses.yaml identical 8 8 0 0
    examples/xai/chat/promptfooconfig.search.yaml identical 8 8 0 0
    examples/xai/chat/promptfooconfig.yaml identical 8 8 0 0
    examples/xai/video/promptfooconfig.yaml identical 4 4 0 0
    examples/xai/voice/promptfooconfig.yaml identical 20 20 0 0