# NIST Juliet starter cases (Java, C/C++): Keryn Forge

> On the test range's seven Juliet starter cases, does Forge find the bad path of each case's CWE without flagging its good paths?

Source: https://keryn.dev/evals/forge-juliet-smoke/

On the test range's seven Juliet starter cases, does Forge find the bad path of each case's CWE without flagging its good paths?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-forge at `98492c26edc4`: Keryn Forge (crates/keryn-forge) via examples/forge_scan: JVM producer on the javac output of the range's Juliet build, LLVM producer on `clang -S -emit-llvm` of the range's C/C++ compile commands. |
| Reference | NIST Juliet per-case SARIF manifests (expected flaw locations) . Ground truth shipped with the corpus (evals/range/receipts/juliet-smoke-ground-truth.json). |
| Corpus | NIST Juliet starter cases from the Keryn test range (evals/range): Java 1.3 (SARD suite 109) 3 cases, C/C++ 1.3.1 (SARD suite 116) 4 cases, [https://samate.nist.gov/SARD](https://samate.nist.gov/SARD) |
| Comparison | Per NIST Juliet case (mixed good/bad): `identical` when Forge reports a `flow` finding of the case's CWE (CWE-120 counts for CWE-121: both are buffer-copy overflows) whose sink lies in a `bad*` function or on a flaw line from the NIST manifest, and no such finding in a `good*` function. `unsupported` when Forge ships no rule for that CWE in that language. Otherwise `differs`. Raw finding counts are never used. |
| Receipt | [`forge-juliet-smoke.json`](https://keryn.dev/evals/forge-juliet-smoke.json), sha256 `c2cf365de0a2a25c` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Cases where the bad path is found and no good path is flagged | **57.1%** | 4 / 7 | Juliet starter cases in the range |
| Of the cases whose CWE Forge has a rule for, those handled correctly | **100%** | 4 / 4 | starter cases with a Forge rule for their CWE and language |

Cases by status: 4 identical, 3 unsupported.

## What this does and does not measure

- Seven starter cases: a build and integration check, not a representative accuracy benchmark. The full Juliet corpora are not measured yet.
- No Juliet executable is run.

## Differences and unassessed cases (3)

| Case | Status | Vulnerable path flagged | Safe paths flagged | Coverage note |
|---|---|---|---:|---|
| `juliet-cpp/104778-v1.1.0` | unsupported |  |  | Forge ships no cpp rule for CWE-1164 |
| `juliet-cpp/79386-v1.0.0` | unsupported |  |  | Forge ships no cpp rule for CWE-134 |
| `juliet-java/139834-v1.0.0` | unsupported |  |  | Forge ships no java rule for CWE-476 |

## All cases (7)

**Show every case**

| Case | Status | Vulnerable path flagged | Safe paths flagged | Coverage note |
|---|---|---|---:|---|
| `juliet-cpp/104778-v1.1.0` | unsupported |  |  | Forge ships no cpp rule for CWE-1164 |
| `juliet-cpp/240691-v2.0.0` | identical | Yes | 0 |  |
| `juliet-cpp/64472-v1.0.0` | identical | Yes | 0 |  |
| `juliet-cpp/79386-v1.0.0` | unsupported |  |  | Forge ships no cpp rule for CWE-134 |
| `juliet-java/139834-v1.0.0` | unsupported |  |  | Forge ships no java rule for CWE-476 |
| `juliet-java/144350-v1.0.0` | identical | Yes | 0 |  |
| `juliet-java/146140-v1.0.0` | identical | Yes | 0 |  |
