Skills › gstack › investigate › eval-bisect.md
eval-bisect.md
25 lines · investigate/sections/eval-bisect.md at 3f79d81
1<!-- AUTO-GENERATED from eval-bisect.md.tmpl — do not edit directly -->
2<!-- Regenerate: bun run gen:skill-docs -->
<!-- AUTO-GENERATED from eval-bisect.md.tmpl — do not edit directly --> <!-- Regenerate: bun run gen:skill-docs -->
Raw lines 1 to 23## Eval bisect recipe
5A judged or scored metric regressed between two revisions. Bisect inputs, not
6commits, and spend the paid judge last.
81. **Distinct revisions only.** `git log --oneline <good>..<bad> -- <prompt and
9 resolver paths>`; collapse commits that leave the expanded prompt bytes
10 identical (build the prompt with the actual builder at each revision and
11 compare hashes; a template edit that renders the same bytes is not a
12 candidate).
132. **A cached cheap scorer first.** Score every distinct revision with the
14 deterministic or cheap check (the structural judge, a token count, the
15 quality judge at its lowest tier) and cache results per
16 `(revision, case)` under `.gstack/tmp/eval-bisect/`. Bisect on the cheap
17 score; spend the paid judge only on the two revisions where the cheap scorer
18 disagrees with the reported regression.
193. **Detached runs.** Launch paid cases through the documented detached runner
20 (`bun run eval:bg:pr`), poll the log for `### gstack-detach EXIT=<code> ###`,
21 and keep the eval lock. Never retry a paid case; a failed launch is a launch
22 failure in the report, not a second sample.
234. **Report.** The first bad revision with its prompt diff, the cheap and paid
24 scores per revision (cached hits marked), the pre-registered panel size and
25 every trial, and the cases that were not run.
Eval bisect recipe
A judged or scored metric regressed between two revisions. Bisect inputs, not commits, and spend the paid judge last.
- Distinct revisions only.
git log --oneline <good>..<bad> -- <prompt and resolver paths>; collapse commits that leave the expanded prompt bytes identical (build the prompt with the actual builder at each revision and compare hashes; a template edit that renders the same bytes is not a candidate). - A cached cheap scorer first. Score every distinct revision with the deterministic or cheap check (the structural judge, a token count, the quality judge at its lowest tier) and cache results per
(revision, case)under.gstack/tmp/eval-bisect/. Bisect on the cheap score; spend the paid judge only on the two revisions where the cheap scorer disagrees with the reported regression. - Detached runs. Launch paid cases through the documented detached runner (
bun run eval:bg:pr), poll the log for### gstack-detach EXIT=<code> ###, and keep the eval lock. Never retry a paid case; a failed launch is a launch failure in the report, not a second sample. - Report. The first bad revision with its prompt diff, the cheap and paid scores per revision (cached hits marked), the pre-registered panel size and every trial, and the cases that were not run.