Baselines and deltas
Review a change by what it adds: the AI components it brings in, and the findings that are new, instead of the whole repository’s history. You will compare against a git branch, save a baseline file, and fail a job when a change adds an AI component.
Before you start
Section titled “Before you start”- The Keryn CLI (install).
- A git clone that contains the base commit. In CI that means a full fetch (
fetch-depth: 0); see CI. - Nothing is fetched:
--basereads a ref of the clone you have, or a file you saved.
1. Compare with a branch
Section titled “1. Compare with a branch”keryn diff runs the AI inventory on the working tree and on the base, and prints what differs.
keryn diff . --base origin/mainKeryn — Checks passed (Go); dependency advisories not checked (offline) dependency advisories not checked (offline): 1 lockfile(s); keryn scan --online checks themAnalysis: complete · Dependency advisories: not checked (offline) · Policy: checks passed · Approval: not recordedbase 69a8fd978be3df6646542bac4cd38507eae55067 (origin/main) → analyzed 69a8fd978be3df6646542bac4cd38507eae55067-dirtyComparison: recorded finding delta; baseline compatibility not independently verifiedChange classes (metric contract 1; comparable): 0 introduced, 0 newly detected, 0 continuing, 0 reopened, 0 cleared, 0 not comparableCases: 0 current = 0 new + 0 existing + 0 unclassified; 0 cleared from previous assessmentNewly blocking: 0 (blocking now, not blocking in the base run under this policy)Cases: 0 of 0 shown (limit 20); 0 omitted by display limitAccounting: 5 recorded rule hits; 5 non-case observations; 0 suppressed rule hits; 0 failed checksdiff: +2 / -0 components + library anthropic (llm-sdk) + model claude-sonnet-4-5 (model) recorded rule-hit changes: +2 / -0 (includes non-case observations)Exceptions: 0 now set aside that counted in the base; 0 new case(s) set aside on arrival; 0 no longer set aside; 0 expired entriesCoverage · source patterns: scanned 3 files; 29/29 requested rules; 0 skipped; 0 excludedCoverage · dataflow: not recordedCoverage · dependency advisories: not checked (offline); 1 lockfile(s) (keryn scan --online checks them, sending package names and versions only)Coverage · known-fix matching: not recorded Limitation: Semgrep-compatible SUBSET: no join, no metavariable-pattern/comparison, no cross-file or type inference, no language equivalences/constant propagation; taint is an approximation. Limitation: Structural matching uses the bundled language grammars, including Ruby. Unsupported rule/language combinations are listed in coverage; generic text also supports regex rules. Limitation: Parity is only claimed where proven by differential tests against native Semgrep CE (results: https://keryn.dev/evals/semgrep-ce/). Limitation: Deliberate divergence: each language is matched with its own grammar. Semgrep CE reuses the JavaScript parse of a pattern for every language of a rule that lists javascript, so its Python side never matches keyword arguments, dict literals or eval/require there (reported as a rule warning). Limitation: 1 package licence unknown: their lockfiles record none. keryn scan --online looks them up on deps.dev, sending package names and versions only. Limitation: 1 lockfile not checked against vulnerability advisories (offline). keryn scan --online checks them, sending package names and versions only. Limitation: 1 npm or PyPI package checked against the shipped malicious-package table (OSV MAL records) and the popularity list; additional ecosystem signals depend on the available data and package contents. Limitation: Malicious-package data: PyPI; source date <time>; shipped. Published MAL records are not proof that unlisted packages are safe.Exit code: 0counts: 5 rule matches · 5 findings · 0 excluding AI-BOM inventory · 0 blocking before answersrules: 29 rules available · 29 supported · 20 applied to the 3 scanned filesThe change in this repository adds a second provider (anthropic) and a model. +2 / -0 components is the inventory delta. Zero current cases means this inventory-only run reports observations rather than vulnerability cases. keryn diff uses the AI inventory rules only, so it is fast.
2. Fail the job when a change adds a component
Section titled “2. Fail the job when a change adds a component”keryn diff . --base origin/main --fail-on-addedKeryn — Policy blocked (Hold); dependency advisories not checked (offline) 2 component(s) added vs base dependency advisories not checked (offline): 1 lockfile(s); keryn scan --online checks themAnalysis: complete · Dependency advisories: not checked (offline) · Policy: blocked · Approval: not recordedbase 69a8fd978be3df6646542bac4cd38507eae55067 (origin/main) → analyzed 69a8fd978be3df6646542bac4cd38507eae55067-dirtyComparison: recorded finding delta; baseline compatibility not independently verifiedChange classes (metric contract 1; comparable): 0 introduced, 0 newly detected, 0 continuing, 0 reopened, 0 cleared, 0 not comparableCases: 0 current = 0 new + 0 existing + 0 unclassified; 0 cleared from previous assessmentNewly blocking: 0 (blocking now, not blocking in the base run under this policy)Cases: 0 of 0 shown (limit 20); 0 omitted by display limitAccounting: 5 recorded rule hits; 5 non-case observations; 0 suppressed rule hits; 0 failed checksdiff: +2 / -0 components + library anthropic (llm-sdk) + model claude-sonnet-4-5 (model) recorded rule-hit changes: +2 / -0 (includes non-case observations)Exceptions: 0 now set aside that counted in the base; 0 new case(s) set aside on arrival; 0 no longer set aside; 0 expired entriesCoverage · source patterns: scanned 3 files; 29/29 requested rules; 0 skipped; 0 excludedCoverage · dataflow: not recordedCoverage · dependency advisories: not checked (offline); 1 lockfile(s) (keryn scan --online checks them, sending package names and versions only)Coverage · known-fix matching: not recorded Limitation: Semgrep-compatible SUBSET: no join, no metavariable-pattern/comparison, no cross-file or type inference, no language equivalences/constant propagation; taint is an approximation. Limitation: Structural matching uses the bundled language grammars, including Ruby. Unsupported rule/language combinations are listed in coverage; generic text also supports regex rules. Limitation: Parity is only claimed where proven by differential tests against native Semgrep CE (results: https://keryn.dev/evals/semgrep-ce/). Limitation: Deliberate divergence: each language is matched with its own grammar. Semgrep CE reuses the JavaScript parse of a pattern for every language of a rule that lists javascript, so its Python side never matches keyword arguments, dict literals or eval/require there (reported as a rule warning). Limitation: 1 package licence unknown: their lockfiles record none. keryn scan --online looks them up on deps.dev, sending package names and versions only. Limitation: 1 lockfile not checked against vulnerability advisories (offline). keryn scan --online checks them, sending package names and versions only. Limitation: 1 npm or PyPI package checked against the shipped malicious-package table (OSV MAL records) and the popularity list; additional ecosystem signals depend on the available data and package contents. Limitation: Malicious-package data: PyPI; source date <time>; shipped. Published MAL records are not proof that unlisted packages are safe.Exit code: 1counts: 5 rule matches · 5 findings · 0 excluding AI-BOM inventory · 0 blocking before answersrules: 29 rules available · 29 supported · 20 applied to the 3 scanned files--fail-on-added turns an added component into exit 1. It looks at components only. There is no flag that fails on added findings alone: --fail-on still judges every finding, so on a repository that already has findings, gate on components and keep the findings visible as the delta (step 4), or let suppressions carry the accepted ones.
3. Save a baseline
Section titled “3. Save a baseline”A branch is not always available: a release job, a scheduled scan, a repository you only receive as a zip. Save what the last good state looked like.
keryn scan . --json keryn-out/baseline.json --aibom keryn-out/baseline.cdx.json --quietThe scan exits 1 because this fixture has findings; the two files are written either way. --base takes a CycloneDX AI-BOM or a Keryn result JSON. Keep the file where your pipeline keeps artifacts (a release asset, a CI cache). Keryn recognises baseline reports by their contents and records them as skipped in coverage. Keep the baseline outside the checkout if you want it excluded from coverage accounting.
Now make a change that adds a vector store:
import chromadb
store = chromadb.Client().create_collection("tickets")
def remember(ticket_id: str, text: str) -> None: store.add(ids=[ticket_id], documents=[text])keryn diff . --base keryn-out/baseline.cdx.json --fail-on-addedKeryn — Policy blocked (Hold); dependency advisories not checked (offline) 1 component(s) added vs base dependency advisories not checked (offline): 1 lockfile(s); keryn scan --online checks themAnalysis: complete · Dependency advisories: not checked (offline) · Policy: blocked · Approval: not recordedbase keryn-out/baseline.cdx.json → analyzed 69a8fd978be3df6646542bac4cd38507eae55067-dirtyComparison: not comparable: baseline has no findingsCases: 0 current = 0 new + 0 existing + 0 unclassified; unknown cleared from previous assessmentNewly blocking: unknown (no complete base run to compare with)Cases: 0 of 0 shown (limit 20); 0 omitted by display limitAccounting: 6 recorded rule hits; 6 non-case observations; 0 suppressed rule hits; 0 failed checksdiff: +1 / -0 components + library chromadb (vector-db)Exceptions: 0 expired suppression entries; changes since the base unknown (no base run with its exceptions)Coverage · source patterns: scanned 4 files; 29/29 requested rules; 0 skipped; 0 excludedCoverage · dataflow: not recordedCoverage · dependency advisories: not checked (offline); 1 lockfile(s) (keryn scan --online checks them, sending package names and versions only)Coverage · known-fix matching: not recorded Limitation: Semgrep-compatible SUBSET: no join, no metavariable-pattern/comparison, no cross-file or type inference, no language equivalences/constant propagation; taint is an approximation. Limitation: Structural matching uses the bundled language grammars, including Ruby. Unsupported rule/language combinations are listed in coverage; generic text also supports regex rules. Limitation: Parity is only claimed where proven by differential tests against native Semgrep CE (results: https://keryn.dev/evals/semgrep-ce/). Limitation: Deliberate divergence: each language is matched with its own grammar. Semgrep CE reuses the JavaScript parse of a pattern for every language of a rule that lists javascript, so its Python side never matches keyword arguments, dict literals or eval/require there (reported as a rule warning). Limitation: 1 package licence unknown: their lockfiles record none. keryn scan --online looks them up on deps.dev, sending package names and versions only. Limitation: 1 lockfile not checked against vulnerability advisories (offline). keryn scan --online checks them, sending package names and versions only. Limitation: Maintainer/install-hook transition not checked without --base evidence; a lockfile alone does not record authenticated maintainer history. Limitation: 1 npm or PyPI package checked against the shipped malicious-package table (OSV MAL records) and the popularity list; additional ecosystem signals depend on the available data and package contents. Limitation: Malicious-package data: PyPI; source date <time>; shipped. Published MAL records are not proof that unlisted packages are safe.Exit code: 1counts: 6 rule matches · 6 findings · 0 excluding AI-BOM inventory · 0 blocking before answersrules: 29 rules available · 29 supported · 20 applied to the 4 scanned filesOnly chromadb is new, and the job fails on it. Coverage counts files excluded from this analysis, including earlier Keryn outputs. A CycloneDX baseline compares components only; it cannot establish case changes.
4. Read the findings delta
Section titled “4. Read the findings delta”With a full scan and a result JSON as the base, the console separates case changes from recorded rule-hit changes (which include inventory observations), and the Markdown summary (--markdown) lists the changes for a pull request comment.
keryn scan . --base keryn-out/baseline.json --quiet --markdown keryn-out/summary.mdgrep -A3 "Changes vs base" keryn-out/summary.md### Changes vs base: +1 / -0
- added `chromadb` (library, vector-db)Findings are compared by their fingerprint (keryn-fp/v1: rule, file and the matched code), not by line number, so code that moves usually stays the same finding. Usually: a finding whose own text includes a location can show once as removed and once as added after an unrelated edit above it. Treat the findings delta as a pointer for the reviewer and the component delta as the exact one.
Scans that compare with a base are never answered from the rescan cache: the comparison is the point.
5. Make it the default
Section titled “5. Make it the default”Put the base in .keryn.yml so the CLI, the editors and CI agree. The language server then marks every finding that is new against it with [new]:
apiVersion: keryn.dev/config/v1base: origin/mainA policy can judge changes too: changes.model: block fails a model swap, changes.prompt: review warns on a prompt edit.
What you just proved
Section titled “What you just proved”The delta is the set of new edges into your system: a provider, a model, a vector store the code now reaches that the base did not. Reviewing that list is the cheapest control on a pull request, and --fail-on-added makes it a gate. It does not say the existing findings are fine, and it sees only what the code references (static references only, not runtime use).
- Wire it into a pipeline: CI.
- Accept a finding on purpose, with a reason and an expiry: suppressions.
- Decide what a change may add: policies.