Skills › Compound Engineering › ce-optimize › text-targets.md

text-targets.md

51 lines · skills/ce-optimize/references/text-targets.md at 3449230

1# Targets that are instruction text

3Read this when the files in `scope.mutable` are instructions a model runs under: a skill, an agent-instructions file, a persona or system prompt, a tool description. The loop is the same as for code. What differs is how the eval set is built, which hypotheses are moves rather than guesses, which model runs what, and how to read a perfect score. `references/example-text-target-spec.yaml` is a complete spec of this shape.

Targets that are instruction text

Read this when the files in scope.mutable are instructions a model runs under: a skill, an agent-instructions file, a persona or system prompt, a tool description. The loop is the same as for code. What differs is how the eval set is built, which hypotheses are moves rather than guesses, which model runs what, and how to read a perfect score. references/example-text-target-spec.yaml is a complete spec of this shape.

Raw lines 1 to 4

5## The eval set

7An instruction-text run is only as good as the cases it is scored on, and the cases are the artifact the run keeps; the text is regenerated by every experiment.

9- **Build the set from failures that happened.** Each case records four fields: the input, what you wanted, what went wrong, and the output that was produced. Five to ten recorded occurrences is the floor before the text is worth optimizing at all; below that, the fix is to collect more failures, not to run the loop.

10- **Coverage rule: every rule you want preserved needs a case that fails without it.** The optimizer keeps text that moves the score and drops text that does not. A rule with no failing case has no measured reason to exist, so expect the run to remove or weaken it; if that would be a defect, add the case before Phase 1, not after the rule is gone.

11- **Cover each failure mode several times.** A mode seen once can land in either the selection sample or the holdout but not both, so it can never be both learned from and confirmed. Synthetic variants of a real failure are acceptable for this; a synthetic mode nobody has observed is not.

12- **Fix the set for the run.** The cases live in `scope.immutable` alongside the harness. Growing or editing the set is a scope change: it ends this run and starts another with a new baseline.

The eval set

An instruction-text run is only as good as the cases it is scored on, and the cases are the artifact the run keeps; the text is regenerated by every experiment.

Build the set from failures that happened. Each case records four fields: the input, what you wanted, what went wrong, and the output that was produced. Five to ten recorded occurrences is the floor before the text is worth optimizing at all; below that, the fix is to collect more failures, not to run the loop.
Coverage rule: every rule you want preserved needs a case that fails without it. The optimizer keeps text that moves the score and drops text that does not. A rule with no failing case has no measured reason to exist, so expect the run to remove or weaken it; if that would be a defect, add the case before Phase 1, not after the rule is gone.
Cover each failure mode several times. A mode seen once can land in either the selection sample or the holdout but not both, so it can never be both learned from and confirmed. Synthetic variants of a real failure are acceptable for this; a synthetic mode nobody has observed is not.
Fix the set for the run. The cases live in scope.immutable alongside the harness. Growing or editing the set is a scope change: it ends this run and starts another with a new baseline.

Raw lines 5 to 13

14## The harness

16The measurement command runs every case against the candidate text under the task model and prints one JSON object: a pass rate or judge score for the primary, a `cases` map (case id to pass/fail) so `measurement.per_case` can report regressions, and `prompt_tokens` or an equivalent length diagnostic. A binary PASS/FAIL judge given the case's "what went wrong" as the tell is usually enough; a graded rubric earns its cost only when partial credit changes which candidate wins. The holdout is a second case file the selection command never reads.

The harness

The measurement command runs every case against the candidate text under the task model and prints one JSON object: a pass rate or judge score for the primary, a cases map (case id to pass/fail) so measurement.per_case can report regressions, and prompt_tokens or an equivalent length diagnostic. A binary PASS/FAIL judge given the case's "what went wrong" as the tell is usually enough; a graded rubric earns its cost only when partial credit changes which candidate wins. The holdout is a second case file the selection command never reads.

Raw lines 14 to 17

18## Model roles

20Two models are involved and the spec pins one of them. The **task model** is the one the text will run under in production; the harness runs every case under it, and it does not change during the run. The **proposer** is whatever writes each candidate: the experiment worker, or an external optimizer. The proposer may be stronger than the task model; a candidate written by a strong model and measured under a cheap one is the normal shape, and improvements found this way carry to stronger task models more often than the reverse. Never let the harness silently use the proposer's model to score; that measures the wrong thing.

Model roles

Two models are involved and the spec pins one of them. The task model is the one the text will run under in production; the harness runs every case under it, and it does not change during the run. The proposer is whatever writes each candidate: the experiment worker, or an external optimizer. The proposer may be stronger than the task model; a candidate written by a strong model and measured under a cheap one is the normal shape, and improvements found this way carry to stronger task models more often than the reverse. Never let the harness silently use the proposer's model to score; that measures the wrong thing.

Raw lines 18 to 21

22## Hypotheses are optimizer moves

24Each hypothesis is one of a small number of moves applied to the current text. Name the move in the hypothesis `category` so the digest can see which moves are exhausted.

26| Move | What the experiment does | When it is the cheapest next step |

27|---|---|---|

28| `add-examples` | Insert labeled input/output pairs drawn from cases the current text fails | Failure themes name a pattern the prose already describes but the model does not follow |

29| `bootstrap-examples` | Run the task model on passing cases and keep its own good outputs as examples | Hand-written examples read differently from what the model produces |

30| `rewrite-from-feedback` | Rewrite the failing section from the digest's failure themes, as a condition rather than a case list | Several cases fail for one stated reason |

31| `shorten` | Cut text while `prompt_tokens` is a required or secondary objective | The primary is flat and the text is long |

32| `restructure` | Move, merge, or split sections; change the order rules fire in | A rule fires in the wrong place, or two rules compete |

33| `decompose` | Split one instruction into a caller and a reference it loads at the point of use | The always-loaded text carries steps most runs never reach |

34| `vote` | Have the task model produce several outputs and pick by a stated rule | Single outputs are inconsistent on cases the model can solve |

35| `external-optimizer` | See below | An optimizer library is available and a metric-call budget is cheaper than a batch of hand-written candidates |

37A rule added for one failing case is not a move. It is the shape the failure themes exist to prevent: when the same section keeps absorbing a case per experiment, the next hypothesis is `rewrite-from-feedback` on that section, restating the condition the cases share.

Hypotheses are optimizer moves

Each hypothesis is one of a small number of moves applied to the current text. Name the move in the hypothesis category so the digest can see which moves are exhausted.

Move	What the experiment does	When it is the cheapest next step
`add-examples`	Insert labeled input/output pairs drawn from cases the current text fails	Failure themes name a pattern the prose already describes but the model does not follow
`bootstrap-examples`	Run the task model on passing cases and keep its own good outputs as examples	Hand-written examples read differently from what the model produces
`rewrite-from-feedback`	Rewrite the failing section from the digest's failure themes, as a condition rather than a case list	Several cases fail for one stated reason
`shorten`	Cut text while `prompt_tokens` is a required or secondary objective	The primary is flat and the text is long
`restructure`	Move, merge, or split sections; change the order rules fire in	A rule fires in the wrong place, or two rules compete
`decompose`	Split one instruction into a caller and a reference it loads at the point of use	The always-loaded text carries steps most runs never reach
`vote`	Have the task model produce several outputs and pick by a stated rule	Single outputs are inconsistent on cases the model can solve
`external-optimizer`	See below	An optimizer library is available and a metric-call budget is cheaper than a batch of hand-written candidates

A rule added for one failing case is not a move. It is the shape the failure themes exist to prevent: when the same section keeps absorbing a case per experiment, the next hypothesis is rewrite-from-feedback on that section, restating the condition the cases share.

Raw lines 22 to 38

39### An external optimizer as an experiment

41When the project has a prompt-optimizer library installed and the harness can be called as its metric, one hypothesis may be: run the optimizer against the seed text for a stated number of metric calls, and take its returned candidate as the experiment's diff. The candidate is then measured by the immutable harness exactly like any other diff; the optimizer's own reported score is not a measurement. Run it with the harness's selection cases only; the holdout stays unseen. The library counts as available when it can be imported or invoked from the project right now; a missing config entry or environment variable proves nothing either way. Name the library and its metric-call budget in the hypothesis so the digest can compare its cost to the hand-written moves. When no optimizer is available, this move is absent from the backlog.

An external optimizer as an experiment

When the project has a prompt-optimizer library installed and the harness can be called as its metric, one hypothesis may be: run the optimizer against the seed text for a stated number of metric calls, and take its returned candidate as the experiment's diff. The candidate is then measured by the immutable harness exactly like any other diff; the optimizer's own reported score is not a measurement. Run it with the harness's selection cases only; the holdout stays unseen. The library counts as available when it can be imported or invoked from the project right now; a missing config entry or environment variable proves nothing either way. Name the library and its metric-call budget in the hypothesis so the digest can compare its cost to the hand-written moves. When no optimizer is available, this move is absent from the backlog.

Raw lines 39 to 42

43## Reading the score

45- **A ceiling is a smell, not a finish.** When the current best scores at the top of the scale on the selection set, the eval is too easy for the text as it now stands. Stop on `target_reached`, report that the set no longer separates candidates, and recommend adding harder or adversarial cases for the next run. Do not tighten the rubric or grow the set inside the run.

46- **Real gains lost to noise** on a small case set are a reason to raise the sample or repeat count, not to lower the threshold.

47- **Length is a stated objective or it is not measured.** If shorter text matters, put `prompt_tokens` in `metric.objectives` (minimize, `role: secondary` to log it, `required` to make it a keep condition). Otherwise do not penalize length by hand.

Reading the score

A ceiling is a smell, not a finish. When the current best scores at the top of the scale on the selection set, the eval is too easy for the text as it now stands. Stop on target_reached, report that the set no longer separates candidates, and recommend adding harder or adversarial cases for the next run. Do not tighten the rubric or grow the set inside the run.
Real gains lost to noise on a small case set are a reason to raise the sample or repeat count, not to lower the threshold.
Length is a stated objective or it is not measured. If shorter text matters, put prompt_tokens in metric.objectives (minimize, role: secondary to log it, required to make it a keep condition). Otherwise do not penalize length by hand.

Raw lines 43 to 48

49## Where things live

51The case files, the calibration labels, and the harness are repository files in `scope.immutable`; they are reused across runs. The judge cache, per-item feedback, failure themes, and the harness-validation record are run state in the experiment log and its scratch directory. The winning text is the diff on `optimize/<spec-name>`, reviewed by a person at wrap-up like any other kept change.

Where things live

The case files, the calibration labels, and the harness are repository files in scope.immutable; they are reused across runs. The judge cache, per-item feedback, failure themes, and the harness-validation record are run state in the experiment log and its scratch directory. The winning text is the diff on optimize/<spec-name>, reviewed by a person at wrap-up like any other kept change.

Raw lines 49 to 51