CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI
Given the same extracted facts, does keryn-ql return exactly the result table that CodeQL returns, for every upstream GitHub Actions query that has an upstream test?
| Evaluation detail | Description |
|---|---|
| Keryn engine | keryn-ql at af6cf3921d24: Keryn’s QL runner: parses and evaluates unmodified upstream .ql/.qll source over the relational facts the Actions extractor writes. |
| Reference | CodeQL CLI 2.26.3. Oracle only. Run on a development machine to produce the reference tables; never shipped, linked or redistributed. |
| Corpus | CodeQL’s own GitHub Actions query tests, github/codeql at codeql-cli/v2.26.3 (44a68d3a47fc) |
| Comparison | Order-insensitive multiset equality of the query’s #select table. Entity columns compare by label and full location (file relative to the test directory, start and end line and column); other columns by value. A case is identical only when every row matches. |
| Budget | 300 CPU seconds per case (CPU time, so machine load does not change which cases time out) |
| Receipt | codeql-actions.json, sha256 63001902e8be5199 |
Results
Section titled “Results”| Measure | Value | Count | Denominator |
|---|---|---|---|
| Queries with an identical result table | 100% | 42 / 42 | upstream Actions queries that have an upstream test |
| Upstream Actions queries with an identical result | 95.5% | 42 / 44 | all .ql queries in actions/ql/src |
| Reference rows the engine also returned | 100% | 509 / 509 | rows CodeQL returned, over cases that finished |
| Engine rows that CodeQL also returned | 100% | 509 / 509 | rows keryn-ql returned, over cases that finished |
Cases by status: 42 identical.
What this does and does not measure
Section titled “What this does and does not measure”- Both sides read the same TRAP, written by CodeQL’s own Actions extractor; this measures query evaluation, not extraction.
- Only GitHub Actions workflows. No other CodeQL language is covered by this suite.
- A timeout means the case used its whole budget of CPU seconds (not wall time, so machine load does not change it); it is not a proof that the query cannot finish.
All cases (42)
Section titled “All cases (42)”Show every case
| Case | Status | Reference rows | Keryn rows | Matching | Missing | Extra | Wall seconds |
|---|---|---|---|---|---|---|---|
query-tests/Models/CompositeActionsSinks |
identical | 2 | 2 | 2 | 0 | 0 | 5.6 |
query-tests/Models/CompositeActionsSources |
identical | 5 | 5 | 5 | 0 | 0 | 4.7 |
query-tests/Models/CompositeActionsSummaries |
identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
query-tests/Models/ReusableWorkflowsSinks |
identical | 1 | 1 | 1 | 0 | 0 | 5.6 |
query-tests/Models/ReusableWorkflowsSources |
identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
query-tests/Models/ReusableWorkflowsSummaries |
identical | 1 | 1 | 1 | 0 | 0 | 4.7 |
query-tests/Security/CWE-074/OutputClobberingHigh |
identical | 19 | 19 | 19 | 0 | 0 | 5.5 |
query-tests/Security/CWE-077/EnvPathInjectionCritical |
identical | 5 | 5 | 5 | 0 | 0 | 9.9 |
query-tests/Security/CWE-077/EnvPathInjectionMedium |
identical | 0 | 0 | 0 | 0 | 0 | 11.9 |
query-tests/Security/CWE-077/EnvVarInjectionCritical |
identical | 36 | 36 | 36 | 0 | 0 | 12.4 |
query-tests/Security/CWE-077/EnvVarInjectionMedium |
identical | 0 | 0 | 0 | 0 | 0 | 11.9 |
query-tests/Security/CWE-078/CommandInjectionCritical |
identical | 1 | 1 | 1 | 0 | 0 | 7.9 |
query-tests/Security/CWE-078/CommandInjectionMedium |
identical | 0 | 0 | 0 | 0 | 0 | 7 |
query-tests/Security/CWE-088/ArgumentInjectionCritical |
identical | 12 | 12 | 12 | 0 | 0 | 5.9 |
query-tests/Security/CWE-088/ArgumentInjectionMedium |
identical | 0 | 0 | 0 | 0 | 0 | 5.9 |
query-tests/Security/CWE-094/CodeInjectionCritical |
identical | 162 | 162 | 162 | 0 | 0 | 399.9 |
query-tests/Security/CWE-094/CodeInjectionMedium |
identical | 60 | 60 | 60 | 0 | 0 | 402.7 |
query-tests/Security/CWE-1395/UseOfKnownVulnerableAction |
identical | 2 | 2 | 2 | 0 | 0 | 4.2 |
query-tests/Security/CWE-200/SecretExfiltration |
identical | 1 | 1 | 1 | 0 | 0 | 5.8 |
query-tests/Security/CWE-275/MissingActionsPermissions |
identical | 8 | 8 | 8 | 0 | 0 | 4.4 |
query-tests/Security/CWE-285/ImproperAccessControl |
identical | 1 | 1 | 1 | 0 | 0 | 4.5 |
query-tests/Security/CWE-312/ExcessiveSecretsExposure |
identical | 3 | 3 | 3 | 0 | 0 | 4.4 |
query-tests/Security/CWE-312/SecretsInArtifacts |
identical | 5 | 5 | 5 | 0 | 0 | 4.5 |
query-tests/Security/CWE-312/UnmaskedSecretExposure |
identical | 2 | 2 | 2 | 0 | 0 | 4.5 |
query-tests/Security/CWE-349/CachePoisoningViaCodeInjection |
identical | 2 | 2 | 2 | 0 | 0 | 7.1 |
query-tests/Security/CWE-349/CachePoisoningViaDirectCache |
identical | 3 | 3 | 3 | 0 | 0 | 8.8 |
query-tests/Security/CWE-349/CachePoisoningViaPoisonableStep |
identical | 3 | 3 | 3 | 0 | 0 | 8.5 |
query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUCritical |
identical | 12 | 12 | 12 | 0 | 0 | 13.3 |
query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUHigh |
identical | 2 | 2 | 2 | 0 | 0 | 14.3 |
query-tests/Security/CWE-571/ExpressionIsAlwaysTrueCritical |
identical | 11 | 11 | 11 | 0 | 0 | 5.3 |
query-tests/Security/CWE-571/ExpressionIsAlwaysTrueHigh |
identical | 11 | 11 | 11 | 0 | 0 | 4.7 |
query-tests/Security/CWE-829/ArtifactPoisoningCritical |
identical | 18 | 18 | 18 | 0 | 0 | 66 |
query-tests/Security/CWE-829/ArtifactPoisoningMedium |
identical | 0 | 0 | 0 | 0 | 0 | 69.3 |
query-tests/Security/CWE-829/ArtifactPoisoningPathTraversal |
identical | 1 | 1 | 1 | 0 | 0 | 44.1 |
query-tests/Security/CWE-829/UnpinnedActionsTag |
identical | 42 | 42 | 42 | 0 | 0 | 19 |
query-tests/Security/CWE-829/UntrustedCheckoutCritical |
identical | 40 | 40 | 40 | 0 | 0 | 402 |
query-tests/Security/CWE-829/UntrustedCheckoutHigh |
identical | 23 | 23 | 23 | 0 | 0 | 394.1 |
query-tests/Security/CWE-829/UntrustedCheckoutMedium |
identical | 10 | 10 | 10 | 0 | 0 | 250.8 |
query-tests/Security/CWE-829/UnversionedImmutableAction |
identical | 0 | 0 | 0 | 0 | 0 | 11.7 |
query-tests/Security/CWE-918/RequestForgery |
identical | 1 | 1 | 1 | 0 | 0 | 9.7 |
query-tests/SyntaxError/SyntaxError |
identical | 1 | 1 | 1 | 0 | 0 | 7.4 |
query-tests/Violations Of Best Practice/CodeQL/UnnecessaryUseOfAdvancedConfig |
identical | 1 | 1 | 1 | 0 | 0 | 8.4 |
Wall seconds are from one run on Apple M2, 8 cores, darwin 24.6.0, 4 case(s) at a time, on 2026-11-09 (674 s in total). They are not part of the receipt and vary between machines.