# Real CVE fixes as NuGet artifacts (dotnet): Keryn Forge

> For advisories of NuGet packages, does Forge report a flow of the advisory's CWE in the last vulnerable release and no longer report it in the first patched release?

Source: https://keryn.dev/evals/forge-cve-dotnet/

For advisories of NuGet packages, does Forge report a flow of the advisory's CWE in the last vulnerable release and no longer report it in the first patched release?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-forge at `6b98539c6472`: Keryn Forge (crates/keryn-forge) examples/forge_scan, models/cil.json; binary sha256 55b9846b5fdf5957510209747a7a65e170581eb58aee885b237a17412ab8f4fc. |
| Reference | GitHub Advisory Database (via vulnfix-mine evidence db) + registry release history . Ground truth: the advisory says the vulnerable release has the weakness of the listed CWE and the first patched release does not. |
| Corpus | dotnet bytecode pairs |
| Comparison | Per advisory package (one vulnerable release, one patched release, both scanned with forge_scan --accept-missing-deps all). `identical` (outcome `found-and-fixed`): the vulnerable release has a `flow` of the advisory CWE family whose key (rule, source function, sink function) is absent as a flow in the patched release. `differs`: `flow-persists` (flow in both), or a miss (outcome = the miss class: no-sink-model, sink-bound-no-source, bound-no-flow, unknown-only, producer-gap). `unsupported`: no shipped Forge rule judges the advisory CWE. `error`/`timeout`: a scan failed. Failures stay in the denominator. |
| Receipt | [`forge-cve-dotnet.json`](https://keryn.dev/evals/forge-cve-dotnet.json), sha256 `cc464f50f932afbf` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Advisories where the vulnerable release has the CWE flow and the patched release no longer does | **0%** | 0 / 30 | advisory packages with both releases scanned or failed |
| Of the advisories whose CWE Forge judges, those with a flow in the vulnerable release | **4.8%** | 1 / 21 | cases whose CWE some shipped Forge rule judges and whose scans completed |
| Of the advisories whose CWE Forge judges, those where the flow disappears in the patched release | **0%** | 0 / 21 | cases whose CWE some shipped Forge rule judges and whose scans completed |
| Of the flows found in vulnerable releases, those with none of the same flows in the patched release | **0%** | 0 / 1 | cases with a CWE flow in the vulnerable release |
| Of the judged advisories with a known fix commit, those whose flow touches a class the fix changed | **0%** | 0 / 21 | judged cases with a known fix commit |
| Advisories whose CWE some shipped Forge rule judges | **70.0%** | 21 / 30 | advisory packages in the batch |
| Judged advisories where a sink is bound but no source exists (library entry points unmodelled) | **0%** | 0 / 21 | cases whose CWE some shipped Forge rule judges and whose scans completed |

Cases by status: 21 differs, 9 unsupported.

## What this does and does not measure

- Advisories with a known fixed version whose vulnerable and fixed releases are both on the registry; Jenkins/other-registry packages, pom-only packages and artifacts over 25 MB are dropped before scoring (counts in the pairs dropped.json, not cases).
- Libraries have no HTTP request source: unless a framework entry model applies, a library flow can only start at a modelled source, so most misses are `sink-bound-no-source`. That is a measured gap, not a data error.
- Artifacts over 1.5 MB are scanned as a slice (packages of the classes the fix changed, same list for both releases); the case records `mode`.
- Pair selection is deterministic but biased to advisories with a known fix commit and a CWE label; at most 2 pairs per package.

## Differences and unassessed cases (30)

| Case | Status | Analysis outcome |
|---|---|---|
| `nuget:AjaxNetProfessional@21.12.21.1..21.12.22.1` | differs | Unknown only |
| `nuget:Autoupdater.NET.Official@1.5.7..1.5.8` | differs | Unknown only |
| `nuget:DotNetNuke.Core@9.13.7-ci0064..9.13.8` | differs | Unknown only |
| `nuget:DotNetNuke.Core@9.13.8..9.13.9` | differs | Unknown only |
| `nuget:DotNetNuke.Web@9.13.8..9.13.9` | differs | Unknown only |
| `nuget:DotNetZip@1.10.1..1.11.0` | differs | Bound, no flow |
| `nuget:DotVVM@4.2.9..4.2.10` | differs | Flow persists |
| `nuget:HtmlSanitizer@5.0.355..5.0.372` | differs | No sink model |
| `nuget:ICG.AspNetCore.Utilities.CloudStorage@6.0.0..8.0.0` | unsupported | Weakness not modelled |
| `nuget:LiteDB@5.0.12..5.0.13` | differs | Bound, no flow |
| `nuget:Microsoft.OpenApi.Kiota.Builder@1.29.0..1.29.1` | differs | Bound, no flow |
| `nuget:NuGet.CommandLine.XPlat@6.2.0..6.2.1` | unsupported | Weakness not modelled |
| `nuget:NuGet.Commands@5.0.0-rtm.5867..5.0.2` | unsupported | Weakness not modelled |
| `nuget:NuGet.Commands@6.2.0..6.2.1` | unsupported | Weakness not modelled |
| `nuget:OWASP.AntiSamy@1.1.1..1.2.0` | differs | No sink model |
| `nuget:OrchardCore@1.2.0..1.2.1` | differs | Unknown only |
| `nuget:OrchardCore@1.3.0..1.4.0` | differs | Unknown only |
| `nuget:Serenity.Net.Core@6.6.6..6.7.0` | differs | Bound, no flow |
| `nuget:Serilog.Enrichers.ClientInfo@2.0.3..2.1.0` | differs | No sink model |
| `nuget:SharpCompress@0.47.4..0.48.0` | differs | Bound, no flow |
| `nuget:SharpZipLib@1.3.2..1.3.3` | differs | Bound, no flow |
| `nuget:Steeltoe.Management.Endpoint@4.1.0..4.2.0` | unsupported | Weakness not modelled |
| `nuget:Steeltoe.Management.Endpoint@4.2.0..4.3.0` | unsupported | Weakness not modelled |
| `nuget:Steeltoe.Management.EndpointCore@3.3.0..3.4.0` | unsupported | Weakness not modelled |
| `nuget:TrueLayer.Client@1.5.0..1.6.0` | differs | No sink model |
| `nuget:Umbraco.Cms.Api.Delivery@16.1.0..16.1.1` | unsupported | Weakness not modelled |
| `nuget:Umbraco.Cms.Api.Management@14.3.2..14.3.3` | unsupported | Weakness not modelled |
| `nuget:Umbraco.Cms.Core@13.1.0..13.1.1` | differs | Unknown only |
| `nuget:Umbraco.Cms.Web.BackOffice@13.1.0..13.1.1` | differs | Unknown only |
| `nuget:YoutubeDLSharp@1.1.1..1.1.2` | differs | Bound, no flow |

## All cases (30)

**Show every case**

| Case | Status | Analysis outcome |
|---|---|---|
| `nuget:AjaxNetProfessional@21.12.21.1..21.12.22.1` | differs | Unknown only |
| `nuget:Autoupdater.NET.Official@1.5.7..1.5.8` | differs | Unknown only |
| `nuget:DotNetNuke.Core@9.13.7-ci0064..9.13.8` | differs | Unknown only |
| `nuget:DotNetNuke.Core@9.13.8..9.13.9` | differs | Unknown only |
| `nuget:DotNetNuke.Web@9.13.8..9.13.9` | differs | Unknown only |
| `nuget:DotNetZip@1.10.1..1.11.0` | differs | Bound, no flow |
| `nuget:DotVVM@4.2.9..4.2.10` | differs | Flow persists |
| `nuget:HtmlSanitizer@5.0.355..5.0.372` | differs | No sink model |
| `nuget:ICG.AspNetCore.Utilities.CloudStorage@6.0.0..8.0.0` | unsupported | Weakness not modelled |
| `nuget:LiteDB@5.0.12..5.0.13` | differs | Bound, no flow |
| `nuget:Microsoft.OpenApi.Kiota.Builder@1.29.0..1.29.1` | differs | Bound, no flow |
| `nuget:NuGet.CommandLine.XPlat@6.2.0..6.2.1` | unsupported | Weakness not modelled |
| `nuget:NuGet.Commands@5.0.0-rtm.5867..5.0.2` | unsupported | Weakness not modelled |
| `nuget:NuGet.Commands@6.2.0..6.2.1` | unsupported | Weakness not modelled |
| `nuget:OWASP.AntiSamy@1.1.1..1.2.0` | differs | No sink model |
| `nuget:OrchardCore@1.2.0..1.2.1` | differs | Unknown only |
| `nuget:OrchardCore@1.3.0..1.4.0` | differs | Unknown only |
| `nuget:Serenity.Net.Core@6.6.6..6.7.0` | differs | Bound, no flow |
| `nuget:Serilog.Enrichers.ClientInfo@2.0.3..2.1.0` | differs | No sink model |
| `nuget:SharpCompress@0.47.4..0.48.0` | differs | Bound, no flow |
| `nuget:SharpZipLib@1.3.2..1.3.3` | differs | Bound, no flow |
| `nuget:Steeltoe.Management.Endpoint@4.1.0..4.2.0` | unsupported | Weakness not modelled |
| `nuget:Steeltoe.Management.Endpoint@4.2.0..4.3.0` | unsupported | Weakness not modelled |
| `nuget:Steeltoe.Management.EndpointCore@3.3.0..3.4.0` | unsupported | Weakness not modelled |
| `nuget:TrueLayer.Client@1.5.0..1.6.0` | differs | No sink model |
| `nuget:Umbraco.Cms.Api.Delivery@16.1.0..16.1.1` | unsupported | Weakness not modelled |
| `nuget:Umbraco.Cms.Api.Management@14.3.2..14.3.3` | unsupported | Weakness not modelled |
| `nuget:Umbraco.Cms.Core@13.1.0..13.1.1` | differs | Unknown only |
| `nuget:Umbraco.Cms.Web.BackOffice@13.1.0..13.1.1` | differs | Unknown only |
| `nuget:YoutubeDLSharp@1.1.1..1.1.2` | differs | Bound, no flow |
