optimize-spec-schema.yaml · Compound Engineering

1# Optimization Spec Schema

2# This is the canonical schema for optimization spec files created by users

3# to configure a /ce-optimize run. The orchestrating agent validates specs

4# against this schema before proceeding.

5#

6# Usage: Create a YAML file matching this schema and pass it to /ce-optimize.

7# The agent reads this spec, validates required fields, and uses it to

8# configure the entire optimization run.

9

10# ============================================================================

11# REQUIRED FIELDS

12# ============================================================================

13

14required_fields:

15

16 name:

17 type: string

18 pattern: "^[a-z0-9]+(?:-[a-z0-9]+)*$"

19 description: "Unique identifier for this optimization run (lowercase kebab-case, safe for git refs and worktree paths)"

20 example: "improve-issue-clustering"

21

22 description:

23 type: string

24 description: "Human-readable description of the optimization goal"

25 example: "Improve coherence and coverage of issue/PR clusters"

26

27 metric:

28 type: object

29 description: "Three-tier metric configuration"

30 required_children:

31

32 primary:

33 type: object

34 description: "The metric the loop optimizes against"

35 required_children:

36

37 type:

38 type: enum

39 values:

40 - hard # scalar metric from measurement command (e.g., build time, test pass rate)

41 - judge # LLM-as-judge quality score from sampled outputs

42 description: "Whether the primary metric comes from the measurement command directly or from LLM-as-judge evaluation"

43

44 name:

45 type: string

46 description: "Metric name: must match a key in the measurement command's JSON output (for hard type) or a scoring field (for judge type)"

47 example: "cluster_coherence"

48

49 direction:

50 type: enum

51 values:

52 - maximize

53 - minimize

54 description: "Whether higher or lower is better"

55

56 optional_children:

57

58 baseline:

59 type: number

60 default: null

61 description: "Filled automatically during Phase 1 baseline measurement. Do not set manually."

62

63 target:

64 type: number

65 default: null

66 description: "Optional target value. Loop stops when this is reached. When metric.objectives is set, target_reached instead requires every required objective that declares a target."

67 example: 4.2

68

69 degenerate_gates:

70 type: array

71 description: "Fast boolean checks that reject obviously broken solutions before expensive evaluation. Run first, before the primary metric or judge."

72 required: true

73 items:

74 type: object

75 required_children:

76 name:

77 type: string

78 description: "Metric name: must match a key in the measurement command's JSON output"

79 check:

80 type: string

81 description: "Comparison operator and threshold. Supported operators: >=, <=, >, <, ==, !="

82 example: "<= 0.10"

83 optional_children:

84 description:

85 type: string

86 description: "Human-readable explanation of what this gate catches"

87

88 optional_children:

89

90 objectives:

91 type: array

92 default: []

93 description: >

94 Additional required or secondary hard targets. The primary is always

95 a required comparison, even when this list omits it. An experiment is

96 eligible when it improves at least one required objective beyond the

97 comparison threshold and does not violate any other required

98 objective. Completion via target_reached requires every declared

99 required target. A spec without metric.objectives keeps single-primary

100 acceptance: the primary is the only required objective.

101 items:

102 type: object

103 required_children:

104 name:

105 type: string

106 description: "Metric name: must match a key in the measurement command's JSON output"

107 direction:

108 type: enum

109 values:

110 - maximize

111 - minimize

112 optional_children:

113 role:

114 type: enum

115 values:

116 - required

117 - secondary

118 default: required

119 description: "required objectives participate in eligibility and completion; secondary objectives are logged only"

120 type:

121 type: enum

122 values:

123 - hard

124 default: hard

125 description: "Always hard. Additional objectives come from the measurement command."

126 target:

127 type: number

128 default: null

129 description: "Declared success value for this objective. target_reached requires every required objective that sets this."

130 max_regression:

131 type: object

132 description: "Optional explicit violation bound. When set, violation uses this bound rather than the comparison threshold, including when the bound is looser. When unset, a comparison-threshold regression is a violation."

133 required_children:

134 type:

135 type: enum

136 values:

137 - relative

138 - absolute

139 value:

140 type: number

141

142 diagnostics:

143 type: array

144 default: []

145 description: "Metrics logged for understanding but never gated on. Useful for understanding WHY a primary metric changed."

146 items:

147 type: object

148 required_children:

149 name:

150 type: string

151 description: "Metric name: must match a key in the measurement command's JSON output"

152

153 judge:

154 type: object

155 description: "LLM-as-judge configuration. Required when metric.primary.type is 'judge'. Ignored when type is 'hard'."

156 required_when: "metric.primary.type == 'judge'"

157 required_children:

158 rubric:

159 type: string

160 description: "Multi-line rubric text sent to the judge model. Must instruct the judge to return JSON."

161 example: |

162 Rate this cluster 1-5:

163 - 5: All items clearly about the same issue/feature

164 - 4: Strong theme, minor outliers

165 - 3: Related but covers 2-3 sub-topics

166 - 2: Weak connection

167 - 1: Unrelated items grouped together

168 scoring:

169 type: object

170 required_children:

171 primary:

172 type: string

173 description: "Field name from judge JSON output to use as the primary optimization target"

174 example: "mean_score"

175 optional_children:

176 secondary:

177 type: array

178 default: []

179 description: "Additional scoring fields to log (not optimized against)"

180 optional_children:

181 model:

182 type: enum

183 values:

184 - cheap # the host's fastest, lowest-cost tier

185 - strong # the host's most capable tier

186 default: cheap

187 description: "Capability tier for the judge. The active harness resolves the tier to a concrete model; the spec never names one. Legacy values haiku and sonnet read as cheap and strong."

188 confirmation_seed:

189 type: integer

190 default: null

191 description: "Seed for a second, held-out judge sample scored only before a keep and at final confirmation. Must differ from sample_seed. A different seed alone does not make the sample held out: draw it only from items of that same output the selection sample did not draw, so the two share no item. An output with no item left outside the selection sample cannot be confirmed this way, so its candidate is not kept; a spec whose outputs are that small needs measurement.holdout.command instead. This is the judge-type holdout when measurement.holdout is unset; see the holdout validation rules."

192 calibration:

193 type: object

194 default: null

195 description: "Optional evidence that the judge scores the way a person would, checked in Phase 1 when labels are set. Without it the approval gate states that the judge is unchecked. Required (labels or an explicit waiver) only when the run waits between ticks through a wake after turn end."

196 optional_children:

197 labels:

198 type: string

199 default: null

200 description: "Path to a human-labeled sample: items with the user's rubric score and, when given, the reason. Phase 1 writes it for the user during the guided check (default <state-root>/judge-labels.yaml); a sample kept in the repo is reusable across runs. Around ten items shows a pattern; 20-50 makes the agreement figure dependable."

201 min_agreement:

202 type: number

203 default: 0.8

204 description: "Fraction of labeled items on which the judge's score must agree with the human label for the harness to be accepted."

205 waived:

206 type: boolean

207 default: false

208 description: "True only when the user explicitly waived calibration for a run that would otherwise require it. Record who waived and when in the experiment log's harness_validation."

209 sample_size:

210 type: integer

211 default: 10

212 description: "Total number of output items to sample for judge evaluation per experiment"

213 stratification:

214 type: array

215 default: null

216 description: "Stratified sampling buckets. If null, uses uniform random sampling."

217 items:

218 type: object

219 required_children:

220 bucket:

221 type: string

222 description: "Bucket name for this stratum"

223 count:

224 type: integer

225 description: "Number of items to sample from this bucket"

226 singleton_sample:

227 type: integer

228 default: 0

229 description: "Number of singleton items to sample for false-negative evaluation"

230 singleton_rubric:

231 type: string

232 default: null

233 description: "Rubric for evaluating sampled singletons. Required if singleton_sample > 0."

234 sample_seed:

235 type: integer

236 default: 42

237 description: "Fixed seed for reproducible sampling across experiments"

238 batch_size:

239 type: integer

240 default: 5

241 description: "Number of samples per judge sub-agent batch. Controls parallelism vs overhead."

242 minimum_improvement:

243 type: number

244 default: 0.3

245 description: "Minimum judge score improvement required to accept an experiment as 'better'. Accounts for sample-composition variance when output structure changes between experiments. Distinct from measurement.stability.noise_threshold which handles run-to-run flakiness."

246 max_total_cost_usd:

247 type: number

248 default: 5

249 description: "Do not start a judge pass whose estimated cost would take cumulative judge spend past this cap. Counts judge scoring only; stopping.max_total_cost_usd caps the whole run. This is a first-run safety default; raise it only after the rubric and harness are trustworthy. Set to null only with explicit user approval."

250

251 measurement:

252 type: object

253 description: "How to run the measurement harness"

254 required_children:

255 command:

256 type: string

257 description: "Shell command that runs the evaluation and outputs JSON to stdout. The JSON must contain keys matching all gate names and diagnostic names."

258 example: "python evaluate.py"

259 optional_children:

260 timeout_seconds:

261 type: integer

262 default: 600

263 description: "Maximum seconds for the measurement command to run before being killed"

264 output_format:

265 type: enum

266 values:

267 - json

268 default: json

269 description: "Format of the measurement command's stdout. Currently only JSON is supported."

270 working_directory:

271 type: string

272 default: "."

273 description: "Working directory for the measurement command, relative to the repo root"

274 holdout:

275 type: object

276 default: null

277 description: >

278 A held-out confirmation set the loop never selects or generates

279 hypotheses from. Scored only before a keep and at final

280 confirmation. When configured, a keep requires the holdout comparison

281 to agree with the selection comparison (decide.mjs takes the second

282 snapshot pair as `holdout` and returns `next_measurement: holdout`

283 when it is still missing). Required when metric.primary.type is

284 'judge' and when the run waits between ticks through a wake after

285 turn end; otherwise optional, and its absence is stated in the

286 Phase 1 approval presentation.

287 optional_children:

288 command:

289 type: string

290 default: null

291 description: "Shell command that runs the harness on the held-out set and prints the same JSON shape as measurement.command. For a judge primary, metric.judge.confirmation_seed may stand in for this command."

292 per_case:

293 type: boolean

294 default: false

295 description: "When true, the measurement JSON also carries a `cases` object mapping a stable case id to a boolean pass/fail (1/0 accepted). decide.mjs then reports `regressions`: cases the reference passed and the candidate fails. Report only; it does not change the decision."

296 stability:

297 type: object

298 default: { mode: "stable" }

299 description: "How to handle metric variance across runs"

300 required_children:

301 mode:

302 type: enum

303 values:

304 - stable # run once, trust the result

305 - repeat # run N times, aggregate

306 - ladder # smoke, one paired exploratory sample, adaptive extras, full protocol only before keep

307 default: stable

308 optional_children:

309 repeat_count:

310 type: integer

311 default: 5

312 description: "Number of times to run the harness when mode is 'repeat', and the confirmation sample count when mode is 'ladder'"

313 aggregation:

314 type: enum

315 values:

316 - median

317 - mean

318 - min

319 - max

320 default: median

321 description: "How to combine repeated measurements into a single value"

322 noise_threshold:

323 type: number

324 default: 0.02

325 description: "Absolute improvement that must be exceeded when comparison.method is absolute (the default). Applied to hard metrics only."

326 comparison:

327 type: object

328 default: { method: "absolute" }

329 description: "How to decide keep / revert / inconclusive. Unset method preserves the legacy absolute noise_threshold rule."

330 required_children:

331 method:

332 type: enum

333 values:

334 - absolute

335 - relative

336 - paired

337 default: absolute

338 description: "absolute uses noise_threshold; relative uses relative_threshold; paired uses sample ranges / paired diffs and may return inconclusive"

339 optional_children:

340 relative_threshold:

341 type: number

342 default: 0.05

343 description: "Fraction of the current-best value that counts as a real change when method is relative or paired"

344 ladder:

345 type: object

346 description: "Cost-aware measurement stages. Used when mode is ladder; ignored otherwise."

347 optional_children:

348 smoke_command:

349 type: string

350 default: null

351 description: "Cheap correctness check. Failure is degenerate and skips timed measurement."

352 exploratory_pairs:

353 type: integer

354 default: 1

355 description: "Paired exploratory samples before deciding whether to spend more. Must be a positive integer."

356 confirmation_repeats:

357 type: integer

358 default: null

359 description: "Full protocol sample count before keep. Defaults to repeat_count. Must be a positive integer at least as large as exploratory_pairs."

360 futility:

361 type: object

362 description: "Predeclared abort bound for a clearly noncompetitive live run"

363 optional_children:

364 worse_factor:

365 type: number

366 default: 1.2

367 description: "Censor after the first sample if the candidate is this multiple worse than the current best (minimize: candidate >= best * factor)"

368 after_elapsed_seconds:

369 type: number

370 default: null

371 description: "Censor a still-noncompetitive live run once elapsed time reaches this bound. When set, must be a positive number."

372

373 scope:

374 type: object

375 description: "What the experiment agent is allowed to modify"

376 required_children:

377 mutable:

378 type: array

379 description: "Files and directories the agent MAY modify during experiments"

380 items:

381 type: string

382 description: "File path or directory (relative to repo root). Directories match all files within."

383 example:

384 - "src/clustering/"

385 - "src/preprocessing/"

386 - "config/clustering.yaml"

387 immutable:

388 type: array

389 description: "Files and directories the agent MUST NOT modify. The measurement harness should always be listed here."

390 items:

391 type: string

392 example:

393 - "evaluate.py"

394 - "tests/fixtures/"

395 - "data/"

396

397# ============================================================================

398# OPTIONAL FIELDS

399# ============================================================================

400

401optional_fields:

402

403 execution:

404 type: object

405 default: { mode: "parallel", backend: "worktree", max_concurrent: 4 }

406 description: "How experiments are executed"

407 optional_children:

408 mode:

409 type: enum

410 values:

411 - parallel # run experiments simultaneously (default)

412 - serial # run one at a time

413 default: parallel

414 backend:

415 type: enum

416 values:

417 - worktree # git worktrees for isolation (default)

418 - codex # Codex sandboxes for isolation

419 - remote # detached workers with their own checkout; results arrive as a pushed ref or store file

420 default: worktree

421 description: "Where experiments run. remote needs a detached-worker capability (SKILL.md Execution Surface); without one the run falls back to worktree. remote workers measure paired on their own machine, so it requires a relative or paired comparison."

422 max_concurrent:

423 type: integer

424 default: 4

425 minimum: 1

426 description: "Maximum experiments to run in parallel. Capped at 6 for worktree backend. 8+ only valid for Codex or remote backends. For remote this caps dispatched workers, not local worktrees."

427 max_experiments_per_tick:

428 type: integer

429 default: null

430 minimum: 1

431 description: "Optional cap on experiments dispatched in one tick (one batch) of the loop. Bounds how much a single wake or session does before it checkpoints. Unset means max_concurrent alone bounds the batch."

432 codex_security:

433 type: enum

434 values:

435 - full-auto # --full-auto (workspace write)

436 - yolo # --dangerously-bypass-approvals-and-sandbox

437 default: null

438 description: "Codex security posture. If null, user is asked once per session."

439

440 parallel:

441 type: object

442 default: {}

443 description: "Parallelism configuration discovered or set during Phase 1"

444 optional_children:

445 port_strategy:

446 type: enum

447 values:

448 - parameterized # use env var for port

449 - none # no port parameterization needed

450 default: null

451 description: "If null, auto-detected during Phase 1 parallelism probe"

452 port_env_var:

453 type: string

454 default: null

455 description: "Environment variable name for port parameterization (e.g., EVAL_PORT)"

456 port_base:

457 type: integer

458 default: null

459 description: "Base port number. Each experiment gets port_base + experiment_index."

460 shared_files:

461 type: array

462 default: []

463 description: "Files that must be copied into each experiment worktree (e.g., SQLite databases)"

464 items:

465 type: string

466 exclusive_resources:

467 type: array

468 default: []

469 description: "Resources requiring exclusive access (e.g., 'gpu'). If non-empty, forces serial mode."

470 items:

471 type: string

472

473 dependencies:

474 type: object

475 default: { approved: [] }

476 description: "Dependency management for experiments"

477 optional_children:

478 approved:

479 type: array

480 default: []

481 description: "Pre-approved new dependencies that experiments may add"

482 items:

483 type: string

484

485 constraints:

486 type: array

487 default: []

488 description: "Free-text constraints that experiment agents must follow"

489 items:

490 type: string

491 example:

492 - "Do not change the output format of clusters"

493 - "Preserve backward compatibility with existing cluster consumers"

494

495 stopping:

496 type: object

497 default: { max_iterations: 100, max_hours: 8, max_wall_hours: 72, plateau_iterations: 10, target_reached: true }

498 description: "When the optimization loop should stop. Any criterion can trigger a stop."

499 optional_children:

500 max_iterations:

501 type: integer

502 default: 100

503 description: "Stop after this many total experiments"

504 max_hours:

505 type: number

506 default: 8

507 description: "Stop after this many active hours: time spent inside loop ticks since Phase 3 started, summed across ticks. Time spent waiting between ticks does not count. In one uninterrupted session this equals wall-clock time."

508 max_wall_hours:

509 type: number

510 default: 72

511 description: "Calendar backstop: stop once this many hours have passed since Phase 3 started, whether or not the run was active. No wake or resume extends it. Must be >= max_hours."

512 plateau_iterations:

513 type: integer

514 default: 10

515 description: "Stop if no improvement for this many consecutive experiments"

516 target_reached:

517 type: boolean

518 default: true

519 description: "Stop when the primary metric reaches the target value (if set)"

520 max_total_cost_usd:

521 type: number

522 default: null

523 description: "Do not start a dispatch whose estimated cost would take model spend for the whole run (experiment workers, judges, and research subagents) past this cap. Counted from host-reported usage where available and estimated otherwise."

524

525 max_runner_up_merges_per_batch:

526 type: integer

527 default: 1

528 description: "Maximum number of file-disjoint runner-up experiments to attempt merging per batch after keeping the best experiment"

529

530# ============================================================================

531# VALIDATION RULES

532# ============================================================================

533

534validation_rules:

535 - "All required fields must be present"

536 - "name must be lowercase kebab-case (`^[a-z0-9]+(?:-[a-z0-9]+)*$`)"

537 - "metric.primary.type must be 'hard' or 'judge'"

538 - "If metric.primary.type is 'judge', metric.judge must be present with rubric and scoring"

539 - "metric.degenerate_gates must have at least one entry"

540 - "measurement.command must be a non-empty string"

541 - "scope.mutable must have at least one entry"

542 - "scope.immutable must have at least one entry"

543 - "Gate check operators must be one of: >=, <=, >, <, ==, !="

544 - "execution.max_concurrent must be >= 1"

545 - "execution.max_concurrent must not exceed 6 when execution.backend is 'worktree'"

546 - "execution.backend must be one of: worktree, codex, remote"

547 - "If execution.backend is 'remote', measurement.stability.comparison.method must be 'paired' or 'relative'; absolute numbers from different machines are not comparable"

548 - "If parallel.exclusive_resources is non-empty, execution.mode should be 'serial'"

549 - "If metric.judge.singleton_sample > 0, metric.judge.singleton_rubric must be present"

550 - "If metric.primary.type is 'judge' and metric.judge.max_total_cost_usd is null, the user should explicitly approve uncapped spend"

551 - "stopping must have at least one non-default criterion or use defaults"

552 - "stopping.max_total_cost_usd, when set, must be a positive number"

553 - "If metric.objectives is set, each additional objective is a hard metric whose name is unique and matches a measurement JSON key"

554 - "The primary is always a required comparison, including when metric.objectives omits it"

555 - "If measurement.stability.mode is 'ladder', comparison.method should be relative or paired unless the user explicitly keeps absolute"

556 - "A spec without metric.objectives keeps single-primary acceptance"

557 - "comparison.method, when set, must be one of: absolute, relative, paired"

558 - "If measurement.stability.mode is 'ladder', exploratory_pairs and confirmation_repeats (or repeat_count) must be positive integers, and confirmation must be at least exploratory_pairs"

559 - "futility.after_elapsed_seconds, when set, must be a positive number"

560 - "stopping.max_wall_hours must be a positive number and at least stopping.max_hours"

561 - "execution.max_experiments_per_tick, when set, must be a positive integer"

562 - "If the run waits between ticks through a wake after turn end and metric.primary.type is 'judge', metric.judge.max_total_cost_usd must be set; unattended spend is never uncapped"

563 - "metric.judge.model, when set, must be cheap or strong (legacy haiku and sonnet read as cheap and strong)"

564 - "A holdout is configured when measurement.holdout.command is set, or when metric.primary.type is 'judge' and metric.judge.confirmation_seed is set and differs from sample_seed"

565 - "If metric.primary.type is 'judge', a holdout must be configured"

566 - "If the run waits between ticks through a wake after turn end, a holdout must be configured; an unattended keep is never confirmed on the selection sample alone"

567 - "When a holdout is configured, a keep requires the holdout comparison to agree with the selection comparison; the holdout is never used to select or generate hypotheses"

568 - "When metric.judge.calibration.labels is set, the harness is accepted only when the judge meets min_agreement against them"

569 - "If metric.primary.type is 'judge' and the run waits between ticks through a wake after turn end, calibration labels must meet min_agreement or metric.judge.calibration.waived must be true with the user's explicit waiver; an attended judge run may proceed uncalibrated, stated at approval"

570 - "If measurement.per_case is true, the measurement JSON must carry a `cases` object"