# CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI

> Given the same extracted facts, does keryn-ql return exactly the result table that CodeQL returns, for every upstream GitHub Actions query that has an upstream test?

Source: https://keryn.dev/evals/codeql-actions/

Given the same extracted facts, does keryn-ql return exactly the result table that CodeQL returns, for every upstream GitHub Actions query that has an upstream test?

| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-ql at `af6cf3921d24`: Keryn's QL runner: parses and evaluates unmodified upstream .ql/.qll source over the relational facts the Actions extractor writes. |
| Reference | CodeQL CLI 2.26.3. Oracle only. Run on a development machine to produce the reference tables; never shipped, linked or redistributed. |
| Corpus | CodeQL's own GitHub Actions query tests, [github/codeql](https://github.com/github/codeql/tree/44a68d3a47fcbcd6a6a76ec7d1c1b3a1a28b201e/actions/ql/test) at `codeql-cli/v2.26.3` (`44a68d3a47fc`) |
| Comparison | Order-insensitive multiset equality of the query's #select table. Entity columns compare by label and full location (file relative to the test directory, start and end line and column); other columns by value. A case is identical only when every row matches. |
| Budget | 300 CPU seconds per case (CPU time, so machine load does not change which cases time out) |
| Receipt | [`codeql-actions.json`](https://keryn.dev/evals/codeql-actions.json), sha256 `63001902e8be5199` |

## Results

| Measure | Value | Count | Denominator |
|---|---|---|---|
| Queries with an identical result table | **100%** | 42 / 42 | upstream Actions queries that have an upstream test |
| Upstream Actions queries with an identical result | **95.5%** | 42 / 44 | all .ql queries in actions/ql/src |
| Reference rows the engine also returned | **100%** | 509 / 509 | rows CodeQL returned, over cases that finished |
| Engine rows that CodeQL also returned | **100%** | 509 / 509 | rows keryn-ql returned, over cases that finished |

Cases by status: 42 identical.

## What this does and does not measure

- Both sides read the same TRAP, written by CodeQL's own Actions extractor; this measures query evaluation, not extraction.
- Only GitHub Actions workflows. No other CodeQL language is covered by this suite.
- A timeout means the case used its whole budget of CPU seconds (not wall time, so machine load does not change it); it is not a proof that the query cannot finish.

## All cases (42)

**Show every case**

| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra | Wall seconds |
|---|---|---|---|---|---|---|---|
| `query-tests/Models/CompositeActionsSinks` | identical | 2 | 2 | 2 | 0 | 0 | 5.6 |
| `query-tests/Models/CompositeActionsSources` | identical | 5 | 5 | 5 | 0 | 0 | 4.7 |
| `query-tests/Models/CompositeActionsSummaries` | identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
| `query-tests/Models/ReusableWorkflowsSinks` | identical | 1 | 1 | 1 | 0 | 0 | 5.6 |
| `query-tests/Models/ReusableWorkflowsSources` | identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
| `query-tests/Models/ReusableWorkflowsSummaries` | identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
| `query-tests/Security/CWE-074/OutputClobberingHigh` | identical | 19 | 19 | 19 | 0 | 0 | 5.5 |
| `query-tests/Security/CWE-077/EnvPathInjectionCritical` | identical | 5 | 5 | 5 | 0 | 0 | 9.9 |
| `query-tests/Security/CWE-077/EnvPathInjectionMedium` | identical | 0 | 0 | 0 | 0 | 0 | 11.9 |
| `query-tests/Security/CWE-077/EnvVarInjectionCritical` | identical | 36 | 36 | 36 | 0 | 0 | 12.4 |
| `query-tests/Security/CWE-077/EnvVarInjectionMedium` | identical | 0 | 0 | 0 | 0 | 0 | 11.9 |
| `query-tests/Security/CWE-078/CommandInjectionCritical` | identical | 1 | 1 | 1 | 0 | 0 | 7.9 |
| `query-tests/Security/CWE-078/CommandInjectionMedium` | identical | 0 | 0 | 0 | 0 | 0 | 7 |
| `query-tests/Security/CWE-088/ArgumentInjectionCritical` | identical | 12 | 12 | 12 | 0 | 0 | 5.9 |
| `query-tests/Security/CWE-088/ArgumentInjectionMedium` | identical | 0 | 0 | 0 | 0 | 0 | 5.9 |
| `query-tests/Security/CWE-094/CodeInjectionCritical` | identical | 162 | 162 | 162 | 0 | 0 | 399.9 |
| `query-tests/Security/CWE-094/CodeInjectionMedium` | identical | 60 | 60 | 60 | 0 | 0 | 402.7 |
| `query-tests/Security/CWE-1395/UseOfKnownVulnerableAction` | identical | 2 | 2 | 2 | 0 | 0 | 4.2 |
| `query-tests/Security/CWE-200/SecretExfiltration` | identical | 1 | 1 | 1 | 0 | 0 | 5.8 |
| `query-tests/Security/CWE-275/MissingActionsPermissions` | identical | 8 | 8 | 8 | 0 | 0 | 4.4 |
| `query-tests/Security/CWE-285/ImproperAccessControl` | identical | 1 | 1 | 1 | 0 | 0 | 4.5 |
| `query-tests/Security/CWE-312/ExcessiveSecretsExposure` | identical | 3 | 3 | 3 | 0 | 0 | 4.4 |
| `query-tests/Security/CWE-312/SecretsInArtifacts` | identical | 5 | 5 | 5 | 0 | 0 | 4.5 |
| `query-tests/Security/CWE-312/UnmaskedSecretExposure` | identical | 2 | 2 | 2 | 0 | 0 | 4.5 |
| `query-tests/Security/CWE-349/CachePoisoningViaCodeInjection` | identical | 2 | 2 | 2 | 0 | 0 | 7.1 |
| `query-tests/Security/CWE-349/CachePoisoningViaDirectCache` | identical | 3 | 3 | 3 | 0 | 0 | 8.8 |
| `query-tests/Security/CWE-349/CachePoisoningViaPoisonableStep` | identical | 3 | 3 | 3 | 0 | 0 | 8.5 |
| `query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUCritical` | identical | 12 | 12 | 12 | 0 | 0 | 13.3 |
| `query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUHigh` | identical | 2 | 2 | 2 | 0 | 0 | 14.3 |
| `query-tests/Security/CWE-571/ExpressionIsAlwaysTrueCritical` | identical | 11 | 11 | 11 | 0 | 0 | 5.3 |
| `query-tests/Security/CWE-571/ExpressionIsAlwaysTrueHigh` | identical | 11 | 11 | 11 | 0 | 0 | 4.7 |
| `query-tests/Security/CWE-829/ArtifactPoisoningCritical` | identical | 18 | 18 | 18 | 0 | 0 | 66 |
| `query-tests/Security/CWE-829/ArtifactPoisoningMedium` | identical | 0 | 0 | 0 | 0 | 0 | 69.3 |
| `query-tests/Security/CWE-829/ArtifactPoisoningPathTraversal` | identical | 1 | 1 | 1 | 0 | 0 | 44.1 |
| `query-tests/Security/CWE-829/UnpinnedActionsTag` | identical | 42 | 42 | 42 | 0 | 0 | 19 |
| `query-tests/Security/CWE-829/UntrustedCheckoutCritical` | identical | 40 | 40 | 40 | 0 | 0 | 402 |
| `query-tests/Security/CWE-829/UntrustedCheckoutHigh` | identical | 23 | 23 | 23 | 0 | 0 | 394.1 |
| `query-tests/Security/CWE-829/UntrustedCheckoutMedium` | identical | 10 | 10 | 10 | 0 | 0 | 250.8 |
| `query-tests/Security/CWE-829/UnversionedImmutableAction` | identical | 0 | 0 | 0 | 0 | 0 | 11.7 |
| `query-tests/Security/CWE-918/RequestForgery` | identical | 1 | 1 | 1 | 0 | 0 | 9.7 |
| `query-tests/SyntaxError/SyntaxError` | identical | 1 | 1 | 1 | 0 | 0 | 7.4 |
| `query-tests/Violations Of Best Practice/CodeQL/UnnecessaryUseOfAdvancedConfig` | identical | 1 | 1 | 1 | 0 | 0 | 8.4 |

Wall seconds are from one run on Apple M2, 8 cores, darwin 24.6.0, 4 case(s) at a time, on 2026-11-09 (674 s in total). They are not part of the receipt and vary between machines.
