Skill Tracer
Skills › Compound Engineering › ce-compound › grounding-validation.md

grounding-validation.md

89 lines · skills/ce-compound/references/grounding-validation.md at 3449230

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Grounding Validation (Phase 2.45)
2
3Read this when Phase 2.45 runs. The doc just written becomes permanent, trusted knowledge. Future agents will act on its claims without re-verifying them. This phase checks the claims against reality before they compound: a deterministic mechanical pass (bundled script) plus a semantic pass (one read-only validator subagent). Neither pass is a hard stop. You judge every flag, because solution docs legitimately cite deleted paths and pre-fix states.
4

Grounding Validation (Phase 2.45)

Read this when Phase 2.45 runs. The doc just written becomes permanent, trusted knowledge. Future agents will act on its claims without re-verifying them. This phase checks the claims against reality before they compound: a deterministic mechanical pass (bundled script) plus a semantic pass (one read-only validator subagent). Neither pass is a hard stop. You judge every flag, because solution docs legitimately cite deleted paths and pre-fix states.

Raw lines 1 to 4
Starts an agent subagent :3
5## Which tree is the ground truth
6
7Two claim categories verify against different trees:
8
9- **Code-behavior claims** (enum values, status semantics, limits, defaults) verify against the **local working tree**. They describe what this session's work produced and verified here.
10- **Merge-state claims** ("fixed in #1608", "landed", "shipped") verify against **remote truth**. The checkout may predate a merge, so `gh pr view` (or the tracker equivalent) is primary and local git reachability is only the fallback. The script's `INFO: worktree is N commits behind …` line tells you how much to distrust the local tree for this category.
12Before running the script, optionally run `git fetch --quiet` (best-effort; skip silently on failure or offline, because the network is never a correctness dependency). When remote state cannot be checked at all, keep the claim, add an as-of qualifier ("as of this writing"), and record degraded verification in the run report.

Which tree is the ground truth

Two claim categories verify against different trees:

  • Code-behavior claims (enum values, status semantics, limits, defaults) verify against the local working tree. They describe what this session's work produced and verified here.
  • Merge-state claims ("fixed in #1608", "landed", "shipped") verify against remote truth. The checkout may predate a merge, so gh pr view (or the tracker equivalent) is primary and local git reachability is only the fallback. The script's INFO: worktree is N commits behind … line tells you how much to distrust the local tree for this category.

Before running the script, optionally run git fetch --quiet (best-effort; skip silently on failure or offline, because the network is never a correctness dependency). When remote state cannot be checked at all, keep the claim, add an as-of qualifier ("as of this writing"), and record degraded verification in the run report.

Raw lines 5 to 13
14## Step 1: Adjudicate the mechanical flags
16The script reports in two tiers and you decide each item. A **FLAG** is a claim it can defend; a **NOTE** is an item it reported without being able to classify, and notes do not affect its exit code. There are three resolutions, **fix**, **annotate**, or **confirm intentional**, and never an automatic rewrite or an automatic pass:
18| Flag | Likely meaning | Resolution |
19|------|----------------|------------|
20| path not found under the search base | Typo, drafted from memory, or a real file in a repo or store the check did not search | Keep the citation if the file exists in another repo or store you can check; otherwise fix the citation or remove the claim |
21| path missing here, exists at upstream | Stale checkout | Verify the claim against upstream; annotate if the doc implies the file is present locally |
22| path deliberately gone (doc says removed/renamed) | Historical citation | Confirm the surrounding prose marks it as historical ("removed by this fix", "pre-fix state"); add that marker if absent |
23| SHA does not resolve, cited as a commit | Fabricated or from another repo | Replace with the PR number, or drop |
24| NOTE: unresolved hex, no commit context | The script cannot tell a session id or content hash from a commit; it is reporting, not accusing | Read the line: if it was never a commit citation, nothing to do; if it was, verify it |
25| SHA reachable from HEAD only | Local-only commit; SHA will change on rebase/squash merge | Replace with the PR number |
26| SHA reachable from upstream only | Checkout predates the merge | Keep, with a temporal qualifier; verify the landed claim via `gh` |
27| SHA exists but unreachable | Rebased-away commit | Replace with the PR number |
28| scaffold ("Learning 3", `{{…}}`) | Drafting-context leak | Always fix; rewrite as a real path or link |
29| relative link unresolved | Wrong target | Fix the path |
31If the script cannot be resolved on this platform, apply its checks manually at the same scope: scan the body for cited paths that do not exist, hex words presented as commits anywhere in the body (in commands and code blocks as much as in prose), `Learning(s) N` / `{{…}}` scaffold, and broken relative links. Note in the run output that the check was manual. Do not silently skip.
33After any body edit from this step or Step 2, re-run the script until it reports clean or every remaining flag is confirmed intentional.

Step 1: Adjudicate the mechanical flags

The script reports in two tiers and you decide each item. A FLAG is a claim it can defend; a NOTE is an item it reported without being able to classify, and notes do not affect its exit code. There are three resolutions, fix, annotate, or confirm intentional, and never an automatic rewrite or an automatic pass:

FlagLikely meaningResolution
path not found under the search baseTypo, drafted from memory, or a real file in a repo or store the check did not searchKeep the citation if the file exists in another repo or store you can check; otherwise fix the citation or remove the claim
path missing here, exists at upstreamStale checkoutVerify the claim against upstream; annotate if the doc implies the file is present locally
path deliberately gone (doc says removed/renamed)Historical citationConfirm the surrounding prose marks it as historical ("removed by this fix", "pre-fix state"); add that marker if absent
SHA does not resolve, cited as a commitFabricated or from another repoReplace with the PR number, or drop
NOTE: unresolved hex, no commit contextThe script cannot tell a session id or content hash from a commit; it is reporting, not accusingRead the line: if it was never a commit citation, nothing to do; if it was, verify it
SHA reachable from HEAD onlyLocal-only commit; SHA will change on rebase/squash mergeReplace with the PR number
SHA reachable from upstream onlyCheckout predates the mergeKeep, with a temporal qualifier; verify the landed claim via gh
SHA exists but unreachableRebased-away commitReplace with the PR number
scaffold ("Learning 3", {{…}})Drafting-context leakAlways fix; rewrite as a real path or link
relative link unresolvedWrong targetFix the path

If the script cannot be resolved on this platform, apply its checks manually at the same scope: scan the body for cited paths that do not exist, hex words presented as commits anywhere in the body (in commands and code blocks as much as in prose), Learning(s) N / {{…}} scaffold, and broken relative links. Note in the run output that the check was manual. Do not silently skip.

After any body edit from this step or Step 2, re-run the script until it reports clean or every remaining flag is confirmed intentional.

Raw lines 14 to 34
35## Step 2: Semantic validator subagent (Full and non-interactive; skipped in lightweight)
37Dispatch **one generic read-only subagent** covering the written solution doc plus any `CONCEPTS.md` entries added or edited this run. Phase 2.4's entries are claims too; a glossary entry written from a session-level summary is exactly how wrong semantics enter the vocabulary. Use the same mid-tier model class as other reviewer subagents when the platform exposes one.
39If the user's or project's active instructions or the live prompt name a model for this subagent, use that model instead of the default tier above. They can name it by the subagent's tier, by its prompt-asset name, or for the subagents being dispatched. Without such an instruction, a model configured for this subagent's role takes precedence over the default tier.
41Build its prompt from this template:
43```
44You are a grounding validator for documentation about to enter a permanent
45knowledge store. You are read-only: never edit files. Inspect with Read,
46Grep, Glob, git (non-mutating), and gh when available.
48Inputs: the doc content below, the CONCEPTS.md entries below (if any), and
49this staleness context: <INFO line from the mechanical script, or "none">.
51Check every factual claim in three categories:
531. CODE-BEHAVIOR CLAIMS — assertions about how code behaves: enum values,
54 status semantics, limits, defaults, ordering, state transitions. For
55 each, locate the defining source in the current tree and quote the
56 defining line(s) with file:line. Verdict: verified (with quote),
57 contradicted (with the quote showing otherwise), or unverifiable
58 (defining source not found).
602. MERGE-STATE CLAIMS — assertions that a change landed ("fixed in",
61 "merged", "shipped in", "resolved by #N"). Primary check: gh pr view
62 <n> --json state,mergedAt,baseRefName (remote truth). Fallback: git
63 reachability from the upstream default branch. Verdict: verified,
64 contradicted (e.g. PR open, not merged), or unverifiable (offline / no
65 gh) — mark unverifiable as "degraded", do not guess.
673. INTERNAL COMPLETENESS — countable assertions ("six PRs", "three root
68 causes", "all N consumers"). Count the substantiating items in the doc
69 itself. Verdict: complete, or short (found M of N).
71Ignore session narrative ("we first tried X") — that describes the
72conversation, not the tree. Ignore style.
74Return a structured list, one entry per claim checked:
75 claim (verbatim) | category | verdict | evidence (quote + file:line, or
76 command output) | suggested edit (only for non-verified claims)
77```
79**Orchestrator handling of verdicts:**
81- **contradicted** → fix the doc using the quoted evidence (the quote, not the conversation, is authoritative)
82- **unverifiable** (behavior) → soften or attribute ("per this session's conclusion…"), or drop the claim
83- **unverifiable/degraded** (merge-state) → keep with an as-of qualifier; record degraded verification in the report
84- **short** (completeness) → complete the enumeration or restate the count to match what the doc substantiates
85- **verified** → no change

Step 2: Semantic validator subagent (Full and non-interactive; skipped in lightweight)

Dispatch one generic read-only subagent covering the written solution doc plus any CONCEPTS.md entries added or edited this run. Phase 2.4's entries are claims too; a glossary entry written from a session-level summary is exactly how wrong semantics enter the vocabulary. Use the same mid-tier model class as other reviewer subagents when the platform exposes one.

If the user's or project's active instructions or the live prompt name a model for this subagent, use that model instead of the default tier above. They can name it by the subagent's tier, by its prompt-asset name, or for the subagents being dispatched. Without such an instruction, a model configured for this subagent's role takes precedence over the default tier.

Build its prompt from this template:

You are a grounding validator for documentation about to enter a permanent
knowledge store. You are read-only: never edit files. Inspect with Read,
Grep, Glob, git (non-mutating), and gh when available.

Inputs: the doc content below, the CONCEPTS.md entries below (if any), and
this staleness context: <INFO line from the mechanical script, or "none">.

Check every factual claim in three categories:

1. CODE-BEHAVIOR CLAIMS — assertions about how code behaves: enum values,
   status semantics, limits, defaults, ordering, state transitions. For
   each, locate the defining source in the current tree and quote the
   defining line(s) with file:line. Verdict: verified (with quote),
   contradicted (with the quote showing otherwise), or unverifiable
   (defining source not found).

2. MERGE-STATE CLAIMS — assertions that a change landed ("fixed in",
   "merged", "shipped in", "resolved by #N"). Primary check: gh pr view
   <n> --json state,mergedAt,baseRefName (remote truth). Fallback: git
   reachability from the upstream default branch. Verdict: verified,
   contradicted (e.g. PR open, not merged), or unverifiable (offline / no
   gh) — mark unverifiable as "degraded", do not guess.

3. INTERNAL COMPLETENESS — countable assertions ("six PRs", "three root
   causes", "all N consumers"). Count the substantiating items in the doc
   itself. Verdict: complete, or short (found M of N).

Ignore session narrative ("we first tried X") — that describes the
conversation, not the tree. Ignore style.

Return a structured list, one entry per claim checked:
  claim (verbatim) | category | verdict | evidence (quote + file:line, or
  command output) | suggested edit (only for non-verified claims)

Orchestrator handling of verdicts:

  • contradicted → fix the doc using the quoted evidence (the quote, not the conversation, is authoritative)
  • unverifiable (behavior) → soften or attribute ("per this session's conclusion…"), or drop the claim
  • unverifiable/degraded (merge-state) → keep with an as-of qualifier; record degraded verification in the report
  • short (completeness) → complete the enumeration or restate the count to match what the doc substantiates
  • verified → no change
Raw lines 35 to 86
Starts an agent subagent :35
Starts an agent Dispatch :37
Starts an agent subagents :37
Starts an agent dispatched :39
87## Reporting
89Summarize the phase in one line of the run output (non-interactive report's `Grounding:` line; interactive success output): flags adjudicated (fixed / annotated / confirmed), claims checked, claims softened or corrected, and `degraded — merge-state claims unverified offline` when applicable.

Reporting

Summarize the phase in one line of the run output (non-interactive report's Grounding: line; interactive success output): flags adjudicated (fixed / annotated / confirmed), claims checked, claims softened or corrected, and degraded — merge-state claims unverified offline when applicable.

Raw lines 87 to 89