Skip to content

    Type a page, a command such as keryn scan, or a rule id.

    CodeQL for GitHub Actions: keryn-ql vs CodeQL CLI

    View as Markdown

    Given the same extracted facts, does keryn-ql return exactly the result table that CodeQL returns, for every upstream GitHub Actions query that has an upstream test?

    Evaluation detail Description
    Keryn engine keryn-ql at af6cf3921d24: Keryn’s QL runner: parses and evaluates unmodified upstream .ql/.qll source over the relational facts the Actions extractor writes.
    Reference CodeQL CLI 2.26.3. Oracle only. Run on a development machine to produce the reference tables; never shipped, linked or redistributed.
    Corpus CodeQL’s own GitHub Actions query tests, github/codeql at codeql-cli/v2.26.3 (44a68d3a47fc)
    Comparison Order-insensitive multiset equality of the query’s #select table. Entity columns compare by label and full location (file relative to the test directory, start and end line and column); other columns by value. A case is identical only when every row matches.
    Budget 300 CPU seconds per case (CPU time, so machine load does not change which cases time out)
    Receipt codeql-actions.json, sha256 63001902e8be5199
    Measure Value Count Denominator
    Queries with an identical result table 100% 42 / 42 upstream Actions queries that have an upstream test
    Upstream Actions queries with an identical result 95.5% 42 / 44 all .ql queries in actions/ql/src
    Reference rows the engine also returned 100% 509 / 509 rows CodeQL returned, over cases that finished
    Engine rows that CodeQL also returned 100% 509 / 509 rows keryn-ql returned, over cases that finished

    Cases by status: 42 identical.

    • Both sides read the same TRAP, written by CodeQL’s own Actions extractor; this measures query evaluation, not extraction.
    • Only GitHub Actions workflows. No other CodeQL language is covered by this suite.
    • A timeout means the case used its whole budget of CPU seconds (not wall time, so machine load does not change it); it is not a proof that the query cannot finish.
    Show every case
    Case Status Reference rows Keryn rows Matching Missing Extra Wall seconds
    query-tests/Models/CompositeActionsSinks identical 2 2 2 0 0 5.6
    query-tests/Models/CompositeActionsSources identical 5 5 5 0 0 4.7
    query-tests/Models/CompositeActionsSummaries identical 1 1 1 0 0 4.7
    query-tests/Models/ReusableWorkflowsSinks identical 1 1 1 0 0 5.6
    query-tests/Models/ReusableWorkflowsSources identical 1 1 1 0 0 4.7
    query-tests/Models/ReusableWorkflowsSummaries identical 1 1 1 0 0 4.7
    query-tests/Security/CWE-074/OutputClobberingHigh identical 19 19 19 0 0 5.5
    query-tests/Security/CWE-077/EnvPathInjectionCritical identical 5 5 5 0 0 9.9
    query-tests/Security/CWE-077/EnvPathInjectionMedium identical 0 0 0 0 0 11.9
    query-tests/Security/CWE-077/EnvVarInjectionCritical identical 36 36 36 0 0 12.4
    query-tests/Security/CWE-077/EnvVarInjectionMedium identical 0 0 0 0 0 11.9
    query-tests/Security/CWE-078/CommandInjectionCritical identical 1 1 1 0 0 7.9
    query-tests/Security/CWE-078/CommandInjectionMedium identical 0 0 0 0 0 7
    query-tests/Security/CWE-088/ArgumentInjectionCritical identical 12 12 12 0 0 5.9
    query-tests/Security/CWE-088/ArgumentInjectionMedium identical 0 0 0 0 0 5.9
    query-tests/Security/CWE-094/CodeInjectionCritical identical 162 162 162 0 0 399.9
    query-tests/Security/CWE-094/CodeInjectionMedium identical 60 60 60 0 0 402.7
    query-tests/Security/CWE-1395/UseOfKnownVulnerableAction identical 2 2 2 0 0 4.2
    query-tests/Security/CWE-200/SecretExfiltration identical 1 1 1 0 0 5.8
    query-tests/Security/CWE-275/MissingActionsPermissions identical 8 8 8 0 0 4.4
    query-tests/Security/CWE-285/ImproperAccessControl identical 1 1 1 0 0 4.5
    query-tests/Security/CWE-312/ExcessiveSecretsExposure identical 3 3 3 0 0 4.4
    query-tests/Security/CWE-312/SecretsInArtifacts identical 5 5 5 0 0 4.5
    query-tests/Security/CWE-312/UnmaskedSecretExposure identical 2 2 2 0 0 4.5
    query-tests/Security/CWE-349/CachePoisoningViaCodeInjection identical 2 2 2 0 0 7.1
    query-tests/Security/CWE-349/CachePoisoningViaDirectCache identical 3 3 3 0 0 8.8
    query-tests/Security/CWE-349/CachePoisoningViaPoisonableStep identical 3 3 3 0 0 8.5
    query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUCritical identical 12 12 12 0 0 13.3
    query-tests/Security/CWE-367/UntrustedCheckoutTOCTOUHigh identical 2 2 2 0 0 14.3
    query-tests/Security/CWE-571/ExpressionIsAlwaysTrueCritical identical 11 11 11 0 0 5.3
    query-tests/Security/CWE-571/ExpressionIsAlwaysTrueHigh identical 11 11 11 0 0 4.7
    query-tests/Security/CWE-829/ArtifactPoisoningCritical identical 18 18 18 0 0 66
    query-tests/Security/CWE-829/ArtifactPoisoningMedium identical 0 0 0 0 0 69.3
    query-tests/Security/CWE-829/ArtifactPoisoningPathTraversal identical 1 1 1 0 0 44.1
    query-tests/Security/CWE-829/UnpinnedActionsTag identical 42 42 42 0 0 19
    query-tests/Security/CWE-829/UntrustedCheckoutCritical identical 40 40 40 0 0 402
    query-tests/Security/CWE-829/UntrustedCheckoutHigh identical 23 23 23 0 0 394.1
    query-tests/Security/CWE-829/UntrustedCheckoutMedium identical 10 10 10 0 0 250.8
    query-tests/Security/CWE-829/UnversionedImmutableAction identical 0 0 0 0 0 11.7
    query-tests/Security/CWE-918/RequestForgery identical 1 1 1 0 0 9.7
    query-tests/SyntaxError/SyntaxError identical 1 1 1 0 0 7.4
    query-tests/Violations Of Best Practice/CodeQL/UnnecessaryUseOfAdvancedConfig identical 1 1 1 0 0 8.4

    Wall seconds are from one run on Apple M2, 8 cores, darwin 24.6.0, 4 case(s) at a time, on 2026-11-09 (674 s in total). They are not part of the receipt and vary between machines.