generate-test-failure-triage.ts
116 lines · scripts/resolvers/preamble/generate-test-failure-triage.ts at 3f79d81
2import { FREE_TEXT_WRITE_RULE, freeTextFileBash, freeTextFileUse } from '../free-text-file';
4export function generateFailureRouting(): string {
5 return `**Failures: check the cause, then route.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
7- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, invoke \`/investigate\` before any further tracing or edit.
8- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause, scope or intended behavior is still unclear, invoke \`/investigate\` before any broader diagnosis or edit.`;
9}
11export function generateTestFailureTriage(): string {
12 return `## Test Failure Ownership Triage
14When tests fail, do NOT immediately stop. First, determine ownership:
16### Step T1: Classify each failure
18For each failing test:
201. **Get the files changed on this branch:**
21 \`\`\`bash
22 git diff origin/<base>...HEAD --name-only
23 \`\`\`
252. **Classify the failure:**
26 - **In-branch** if: the failing test file itself was modified on this branch, OR the test output references code that was changed on this branch, OR you can trace the failure to a change in the branch diff.
27 - **Likely pre-existing** if: neither the test file nor the code it tests was modified on this branch, AND the failure is unrelated to any branch change you can identify.
28 - **When ambiguous, default to in-branch.** It is safer to stop the developer than to let a broken test ship. Only classify as pre-existing when you are confident.
30 This classification is heuristic — use your judgment reading the diff and the test output. You do not have a programmatic dependency graph.
32### Step T2: Handle in-branch failures
34**STOP.** These are your failures. Show them and do not proceed. The developer must fix their own broken tests before shipping.
36### Step T3: Handle pre-existing failures
38Check \`REPO_MODE\` from the preamble output.
40**If REPO_MODE is \`solo\`:**
42Ask with AskUserQuestion in the AskUserQuestion Format. List each failure with file:line and a brief error description, say the failures appear pre-existing (not caused by this branch), and say that in a solo repo nobody else will fix them. Options, with A recommended because the context is fresh:
43- A) Investigate and fix now (human: ~2-4h / CC: ~15min) — Completeness 10/10
44- B) Add as P0 TODO — fix after this branch lands — Completeness 7/10
45- C) Skip — I know about this, ship anyway — Completeness 3/10
47**If REPO_MODE is \`collaborative\` or \`unknown\`:**
49Ask with AskUserQuestion in the AskUserQuestion Format. List each failure the same way, and say that in a collaborative repo these may be someone else's responsibility. Options, with B recommended so the person who broke it fixes it:
50- A) Investigate and fix now anyway — Completeness 10/10
51- B) Blame + assign GitHub issue to the author — Completeness 9/10
52- C) Add as P0 TODO — Completeness 7/10
53- D) Skip — ship anyway — Completeness 3/10
55### Step T4: Execute the chosen action
57**If "Investigate and fix now":**
58- One read-only root-cause pass, then \`/investigate\` if unclear.
59- Fix the pre-existing failure.
60- Commit the fix separately from the branch's changes: \`git commit -m "fix: pre-existing test failure in <test-file>"\`
61- Continue with the workflow.
63**If "Add as P0 TODO":**
64- If \`TODOS.md\` exists, add the entry following the format in \`review/TODOS-format.md\` (or \`.claude/skills/review/TODOS-format.md\`).
65- If \`TODOS.md\` does not exist, create it with the standard header and add the entry.
66- Entry should include: title, the error output, which branch it was noticed on, and priority P0.
67- Continue with the workflow — treat the pre-existing failure as non-blocking.
69**If "Blame + assign GitHub issue" (collaborative only):**
70- Find who likely broke it. Check BOTH the test file AND the production code it tests:
71 \`\`\`bash
72 # Who last touched the failing test?
73 git log --format="%an (%ae)" -1 -- "<failing-test-file>"
74 # Who last touched the production code the test covers? (often the actual breaker)
75 git log --format="%an (%ae)" -1 -- "<source-file-under-test>"
76 \`\`\`
77 If these are different people, prefer the production code author — they likely introduced the regression.
78- Create an issue assigned to that person. Its title and body carry test names and error output, so they travel as files, never inside a command:
80\`\`\`bash
81${freeTextFileBash([{ variable: 'TITLE_FILE', stem: 'issue-title' }, { variable: 'BODY_FILE', stem: 'issue-body' }])}
82\`\`\`
84${FREE_TEXT_WRITE_RULE} Title file: \`Pre-existing test failure: <test name>\`. Body file (Markdown; the error goes in a \`~~~\` fence so backticks in it stay literal):
86\`\`\`text
87Failing on <current branch>; pre-existing.
89**Error:**
90~~~
91<first 10 lines of the failure>
92~~~
94**Last modified by:** <author>
95**Noticed by:** gstack /ship on <date>
96\`\`\`
98Then post with your platform from Step 0 (\`github\` or \`gitlab\`). Substitute the two printed names, and an assignee only when it is a valid login for that platform (GitHub: letters, digits and single hyphens, at most 39 characters); otherwise drop the assignee flag and name the person in the body.
100\`\`\`bash
101${freeTextFileUse([{ variable: 'TITLE_FILE', placeholder: '<title-file-name>' }, { variable: 'BODY_FILE', placeholder: '<body-file-name>' }], 'gh issue create --title \\"\\$(cat $TITLE_FILE)\\" --body-file $BODY_FILE')}
102case "<platform>" in
103 github) gh issue create --title "$(cat "$TITLE_FILE")" --body-file "$BODY_FILE" --assignee "<github-username>" ;;
104 gitlab) glab issue create -t "$(cat "$TITLE_FILE")" -d "$(cat "$BODY_FILE")" -a "<gitlab-username>" ;;
105 *) echo "Not sent: no GitHub or GitLab remote. Files: $TITLE_FILE $BODY_FILE" >&2; false ;;
106esac && rm -f "$TITLE_FILE" "$BODY_FILE"
107\`\`\`
109- If neither CLI is available or \`--assignee\`/\`-a\` fails (user not in org, etc.), create the issue without assignee and note who should look at it in the body.
110- Continue with the workflow.
112**If "Skip":**
113- Continue with the workflow.
114- Note in output: "Pre-existing test failure skipped: <test-name>"`;
115}