On the test range’s seven Juliet starter cases, does Forge find the bad path of each case’s CWE without flagging its good paths?
| Evaluation detail |
Description |
| Keryn engine |
keryn-forge at 98492c26edc4: Keryn Forge (crates/keryn-forge) via examples/forge_scan: JVM producer on the javac output of the range’s Juliet build, LLVM producer on clang -S -emit-llvm of the range’s C/C++ compile commands. |
| Reference |
NIST Juliet per-case SARIF manifests (expected flaw locations) . Ground truth shipped with the corpus (evals/range/receipts/juliet-smoke-ground-truth.json). |
| Corpus |
NIST Juliet starter cases from the Keryn test range (evals/range): Java 1.3 (SARD suite 109) 3 cases, C/C++ 1.3.1 (SARD suite 116) 4 cases, https://samate.nist.gov/SARD |
| Comparison |
Per NIST Juliet case (mixed good/bad): identical when Forge reports a flow finding of the case’s CWE (CWE-120 counts for CWE-121: both are buffer-copy overflows) whose sink lies in a bad* function or on a flaw line from the NIST manifest, and no such finding in a good* function. unsupported when Forge ships no rule for that CWE in that language. Otherwise differs. Raw finding counts are never used. |
| Receipt |
forge-juliet-smoke.json, sha256 c2cf365de0a2a25c |
| Measure |
Value |
Count |
Denominator |
| Cases where the bad path is found and no good path is flagged |
57.1% |
4 / 7 |
Juliet starter cases in the range |
| Of the cases whose CWE Forge has a rule for, those handled correctly |
100% |
4 / 4 |
starter cases with a Forge rule for their CWE and language |
Cases by status: 4 identical, 3 unsupported.
- Seven starter cases: a build and integration check, not a representative accuracy benchmark. The full Juliet corpora are not measured yet.
- No Juliet executable is run.
| Case |
Status |
Vulnerable path flagged |
Safe paths flagged |
Coverage note |
juliet-cpp/104778-v1.1.0 |
unsupported |
|
|
Forge ships no cpp rule for CWE-1164 |
juliet-cpp/79386-v1.0.0 |
unsupported |
|
|
Forge ships no cpp rule for CWE-134 |
juliet-java/139834-v1.0.0 |
unsupported |
|
|
Forge ships no java rule for CWE-476 |
Show every case
| Case |
Status |
Vulnerable path flagged |
Safe paths flagged |
Coverage note |
juliet-cpp/104778-v1.1.0 |
unsupported |
|
|
Forge ships no cpp rule for CWE-1164 |
juliet-cpp/240691-v2.0.0 |
identical |
Yes |
0 |
|
juliet-cpp/64472-v1.0.0 |
identical |
Yes |
0 |
|
juliet-cpp/79386-v1.0.0 |
unsupported |
|
|
Forge ships no cpp rule for CWE-134 |
juliet-java/139834-v1.0.0 |
unsupported |
|
|
Forge ships no java rule for CWE-476 |
juliet-java/144350-v1.0.0 |
identical |
Yes |
0 |
|
juliet-java/146140-v1.0.0 |
identical |
Yes |
0 |
|