Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    NIST Juliet starter cases (Java, C/C++): Keryn Forge

    View as Markdown

    On the test range’s seven Juliet starter cases, does Forge find the bad path of each case’s CWE without flagging its good paths?

    Evaluation detail Description
    Keryn engine keryn-forge at 98492c26edc4: Keryn Forge (crates/keryn-forge) via examples/forge_scan: JVM producer on the javac output of the range’s Juliet build, LLVM producer on clang -S -emit-llvm of the range’s C/C++ compile commands.
    Reference NIST Juliet per-case SARIF manifests (expected flaw locations) . Ground truth shipped with the corpus (evals/range/receipts/juliet-smoke-ground-truth.json).
    Corpus NIST Juliet starter cases from the Keryn test range (evals/range): Java 1.3 (SARD suite 109) 3 cases, C/C++ 1.3.1 (SARD suite 116) 4 cases, https://samate.nist.gov/SARD
    Comparison Per NIST Juliet case (mixed good/bad): identical when Forge reports a flow finding of the case’s CWE (CWE-120 counts for CWE-121: both are buffer-copy overflows) whose sink lies in a bad* function or on a flaw line from the NIST manifest, and no such finding in a good* function. unsupported when Forge ships no rule for that CWE in that language. Otherwise differs. Raw finding counts are never used.
    Receipt forge-juliet-smoke.json, sha256 c2cf365de0a2a25c
    Measure Value Count Denominator
    Cases where the bad path is found and no good path is flagged 57.1% 4 / 7 Juliet starter cases in the range
    Of the cases whose CWE Forge has a rule for, those handled correctly 100% 4 / 4 starter cases with a Forge rule for their CWE and language

    Cases by status: 4 identical, 3 unsupported.

    • Seven starter cases: a build and integration check, not a representative accuracy benchmark. The full Juliet corpora are not measured yet.
    • No Juliet executable is run.
    Case Status Vulnerable path flagged Safe paths flagged Coverage note
    juliet-cpp/104778-v1.1.0 unsupported Forge ships no cpp rule for CWE-1164
    juliet-cpp/79386-v1.0.0 unsupported Forge ships no cpp rule for CWE-134
    juliet-java/139834-v1.0.0 unsupported Forge ships no java rule for CWE-476
    Show every case
    Case Status Vulnerable path flagged Safe paths flagged Coverage note
    juliet-cpp/104778-v1.1.0 unsupported Forge ships no cpp rule for CWE-1164
    juliet-cpp/240691-v2.0.0 identical Yes 0
    juliet-cpp/64472-v1.0.0 identical Yes 0
    juliet-cpp/79386-v1.0.0 unsupported Forge ships no cpp rule for CWE-134
    juliet-java/139834-v1.0.0 unsupported Forge ships no java rule for CWE-476
    juliet-java/144350-v1.0.0 identical Yes 0
    juliet-java/146140-v1.0.0 identical Yes 0