Phase 1: mine the archive

Zero model cost. Everything here reads files that already exist on disk. Do not run the harness in this phase. Phase 2 is where the first paid runs happen (references/noise-floor.md).

An archive is usually an order of magnitude larger than the experiment you can afford this week. In the engagement this method came from, 745 archived runs were on disk and 458 carried usable traces. That is more evidence than any deliberate sample, sitting unread.

What the archive must contain

Per run, in whatever form the harness stores it (JSONL transcript, structured log, a directory per run): the ordered tool-call trace, token counts, timestamps, and the final assistant message. If it stores only a pass/fail verdict, the archive is not minable. Say so and go to Phase 2.

Find the archive by asking the harness where it writes run output, or by locating the directory it appends to after a run. Do not assume a schema. Read one run end to end and derive the field paths before writing any extractor.

Build the phase-marker map first

phase_trace, process_followed, and the terminal-phase cross-tab are all functions of one artifact that does not exist yet: a table mapping each phase of the corpus's own workflow to the observable side effect that proves it ran. Derive it before writing the extractor, by reading the corpus's spine. Do not infer phases from the logs; that is circular.

Rules: one marker per phase minimum, each a tool call or a file that exists, never a phrase in the assistant's prose. A phase with no observable marker cannot be scored. Record it as unobservable rather than assuming it ran, and carry that list forward. It is the same list Phase 2 needs for the probe's unentered phases and Phase 5 needs for the unmeasured paths. Keep the map in the same directory as the extractor. Every later re-run and every re-scored table depends on it.

Extract one row per run

Write a script, not a per-run read. Reading 458 transcripts by hand is the failure mode this phase is supposed to avoid.

Derive phase_trace from artifacts, not claims. A phase ran if the trace shows its side effect: the reference file it must read, the helper it must dispatch, the file it must write. A run asserting "Phase 3 complete" in prose is not evidence. The defect class you are hunting is exactly runs that report success without doing the work.

Derived metrics

Tokens per minute is the metric that separates working from stalling. output_tokens / wall_clock_min. Compute it for every run and look for bimodality before looking at anything else.

In the engagement, healthy cells ran 10-12k tokens/min; every failing cell sat at 1.5-3.8k. A run burning hours at 2k tok/min is not thinking hard. It is waiting, because the prose told it a second party would respond. Duration alone cannot show this. A long successful run and a long stalled run look identical until you divide.

Treat those bands as the shape to look for, not as thresholds to import. Find your own corpus's two clusters, then read the gap.

The two booleans the spine requires, derived independently:

task_done       = the run's expected deliverable exists (artifact on disk, commit, PR, file changed)
process_followed = every required phase appears in phase_trace, in spine order

Never let one imply the other, and never let marker_present stand in for either. task_done AND NOT process_followed is done-no-workflow below: the run did the job inline and skipped the corpus. Collapsed into one success column, it reads as a win and hides the most expensive defect in the set.

Outcome taxonomy

Apply in order. First match wins. broken is first so that a harness fault can never be scored as a model failure.

Report the broken count as its own number every time you report a rate. In the engagement 20% of usable-looking runs were broken, and excluding them falsified the first headline finding. That finding was an apparently clean monotonic relationship between reasoning effort and completion, which was really the broken runs clustering in one cell (see Confounds).

Halt language in the final message

Match against final_message. Starter set, case-insensitive:

\b(let me know|just let me know)\b
\b(would you like|shall I|should I)\b.*\?
\b(ready (to|for)|standing by|awaiting)\b
\b(waiting (for|on)) (your|the)\b
\b(next steps? (for|would be))\b
\bI('ll| will) (wait|pause|hold)\b
\bonce you (confirm|approve|reply|respond)\b
\b(remaining|the rest|still needs? to be) (work|done|completed)\b

These are evidence that the run stopped, not proof of why. A match tells you to open that transcript and find which phase boundary produced it. A workflow that is supposed to stop and ask will match every one of these and be behaving correctly. Phase 4 sorts stops by who is actually on the other side. Extend the set with the phrasings your own corpus's prose invites. Grep the corpus for its own hand-off wording and add those.

Confounds that fool careful analysts

Each of these was believed by someone competent before being ruled out. Run the rule-out before reporting the effect.

Stratify before believing anything

This is the highest-leverage work in the phase. A single pooled completion rate hides everything worth knowing.

Cross-tabulate outcome against, at minimum:

1. model, the identity axis
2. settings: reasoning effort, temperature, context budget, whatever the harness varies
3. terminal phase, the last entry in phase_trace, which is where the run died

Read the third one against the first two. In the engagement, model identity explained roughly 3x the variance that reasoning-effort settings did, and the failing models died at one specific phase boundary 38-58% of the time while succeeding models died there 0-8%. That single cross-tab localized the defect without reading a line of corpus prose, and it turned Phase 3's audit from a 31-skill sweep into a targeted read.

Watch cell counts. A 3-of-4 cell is not a rate. Label small cells rather than ranking them. If every cell is small, the stratification's output is a hypothesis list, which is still the correct output of this phase.

Hand off

Carry forward: the broken-excluded baseline rate for complete, the observed output_tokens range, the tokens/min clusters, and the ranked list of phase boundaries where runs die. Phase 2 needs the baseline rate to compute a bar (references/noise-floor.md). Phase 3 needs the phase list to know which files to read first.

The limit

Archive mining is observational. Nobody assigned the conditions, the arms are not balanced, and the confounds above are the ones you thought to check. It produces a baseline rate and a ranked hypothesis list. It cannot attribute a cause, and a rate computed from it is not a bar until Phase 2 shows how wide the noise is. Do not credit a change against an archive number alone.