Skill Tracer
Skills › Compound Engineering › ce-optimize › experiment-log-schema.yaml

experiment-log-schema.yaml

563 lines · skills/ce-optimize/references/experiment-log-schema.yaml at 3449230

Open on GitHub
1# Experiment Log Schema
2# This is the canonical schema for the experiment log file that accumulates
3# across an optimization run.
4#
5# Location: <state-root>/experiment-log.yaml, where <state-root> is the durable
6# state root resolved in Phase 0 (references/persistence.md): the harness's
7# durable store when it names one, otherwise
8# .context/compound-engineering/ce-optimize/<spec-name>/
9#
10# PERSISTENCE MODEL:
11# The experiment log on disk is the SINGLE SOURCE OF TRUTH. The agent's
12# in-memory context is expendable and will be compacted during long runs.
14# Write discipline:
15# - Each experiment gets one log entry, appended on its first measurement
16# (SKILL.md step 3.3), before batch evaluation
17# - Later ladder samples for that experiment update the same entry in place
18# - Outcome fields may also be updated in-place after batch evaluation (step 3.5)
19# - The `best` section is updated after each batch if a new best is found
20# - The `hypothesis_backlog` is updated after each batch
21# - `approval` is written once, the moment the Phase 1 gate is cleared
22# - `run_state` is written at CP-4 and whenever a pending wait is registered
23# or cleared
24# - The agent re-reads this file from disk at every phase boundary
26# The orchestrator does NOT read the full log each iteration -- it uses a
27# rolling window (last 10 experiments) + a strategy digest file for
28# hypothesis generation. But the full log exists on disk for resume,
29# crash recovery, and post-run analysis.
31# ============================================================================
32# TOP-LEVEL STRUCTURE
33# ============================================================================
35structure:
37 spec:
38 type: string
39 required: true
40 description: "Name of the optimization spec this log belongs to"
42 run_id:
43 type: string
44 required: true
45 description: "Unique identifier for this optimization run (timestamp-based). Distinguishes resumed runs from fresh starts."
47 started_at:
48 type: string
49 format: "ISO 8601 timestamp"
50 required: true
52 baseline:
53 type: object
54 required: true
55 description: "Metrics measured on the original code before any optimization"
56 children:
57 timestamp:
58 type: string
59 format: "ISO 8601 timestamp"
60 gates:
61 type: object
62 description: "Key-value pairs of gate metric names to their baseline values"
63 metrics:
64 type: object
65 description: "Required hard-objective snapshots (aggregate plus samples) that decide.mjs loads for later comparisons"
66 diagnostics:
67 type: object
68 description: "Key-value pairs of diagnostic metric names to their baseline values"
69 worst_cases:
70 type: array
71 description: "The few worst per-case results (items, tests, queries) with what went wrong, from the measurement output or judge reasons. Explains the score; never changes it"
72 items:
73 type: object
74 children:
75 case_id:
76 type: string
77 score:
78 type: string
79 reason:
80 type: string
81 judge:
82 type: object
83 description: "Judge scores on the baseline (only when primary type is 'judge')"
84 children:
85 # All fields from the scoring config appear here
86 # Plus:
87 sample_seed:
88 type: integer
89 judge_cost_usd:
90 type: number
92 harness_validation:
93 type: object
94 required: false
95 description: "Result of the Phase 1 metric validity gate, written at CP-1 before the baseline. Absent in legacy logs means the gate was not run, not that it passed."
96 children:
97 timestamp:
98 type: string
99 format: "ISO 8601 timestamp"
100 separates_known_good_from_bad:
101 type: boolean
102 description: "The harness scored every known-good exemplar above every known-bad one in the metric's direction; null when no exemplars existed and the probe was not run"
103 rejects_trivial_shortcut:
104 type: boolean
105 description: "An empty, constant, or copied output did not score at or above the known-good exemplars"
106 judge_agreement:
107 type: number
108 description: "Fraction of calibration labels the judge agreed with (judge primary only; null when waived or not applicable)"
109 calibration:
110 type: enum
111 values: [passed, waived, not_run, not_applicable]
112 waiver:
113 type: string
114 description: "Who waived calibration and when, verbatim from the user's message (only when calibration is waived)"
115 holdout_configured:
116 type: boolean
117 description: "Whether a held-out confirmation set was configured for this run"
118 probes:
119 type: array
120 description: "One record per probe run: name, inputs used, observed values, and pass/fail"
122 experiments:
123 type: array
124 required: true
125 description: "Ordered list of all experiments, including kept, reverted, errored, and deferred"
126 items:
127 type: object
128 # See EXPERIMENT ENTRY below
130 best:
131 type: object
132 required: true
133 description: "Summary of the current best result"
134 children:
135 iteration:
136 type: integer
137 description: "Iteration number of the best experiment (use 0 for the baseline snapshot before any experiment is kept)"
138 metrics:
139 type: object
140 description: "All metric values from the current best state (seed with baseline metrics during CP-1)"
141 judge:
142 type: object
143 description: "Judge scores from the best experiment (only when primary type is 'judge')"
144 total_judge_cost_usd:
145 type: number
146 description: "Running total of all judge costs across all experiments"
147 total_run_cost_usd:
148 type: number
149 description: "Running estimate of whole-run model spend: experiment workers, judges, and research subagents"
150 run_cost_basis:
151 type: string
152 description: "Which parts of total_run_cost_usd come from host-reported usage and which are estimates"
154 approval:
155 type: object
156 required: false
157 description: >
158 The Phase 1 user approval, recorded so a resume (including an unattended
159 wake) does not re-ask it. Valid while spec_sha256 matches the SHA-256 of
160 the spec.yaml bytes on disk and every cap below matches the spec. Absent,
161 or no longer matching, means the gate is presented again. Missing in
162 legacy logs means the gate is presented again on resume, as before.
163 children:
164 approved_at:
165 type: string
166 format: "ISO 8601 timestamp"
167 spec_sha256:
168 type: string
169 description: "SHA-256 hex digest of the saved spec.yaml bytes at approval time"
170 caps:
171 type: object
172 description: "The limits the user approved: stopping.* as in force, metric.judge.max_total_cost_usd (null only with the explicit uncapped-spend yes), execution.max_concurrent"
174 run_state:
175 type: object
176 required: false
177 description: >
178 Tick-level loop state for resume across turns. Written at CP-4 and
179 whenever a pending wait is registered or cleared. Absent in a log that
180 has not reached Phase 3 or in a legacy log; treat absent clocks as zero
181 and absent pending_waits as none.
182 children:
183 status:
184 type: string
185 enum: [running, waiting, blocked, final]
186 description: "running inside a tick; waiting on recorded pending waits or on the user's wrap-up choice; blocked on something outside the loop (report it, do not spin); final after CP-5"
187 tick:
188 type: integer
189 description: "Completed ticks (batches) so far"
190 phase3_started_at:
191 type: string
192 format: "ISO 8601 timestamp"
193 description: "Wall-clock anchor for stopping.max_wall_hours"
194 active_seconds:
195 type: number
196 description: "Time spent inside ticks since Phase 3 started, summed across ticks; the clock stopping.max_hours reads"
197 pending_waits:
198 type: array
199 description: "Work outstanding across the turn boundary. Empty when nothing is outstanding. Turn-end conditions, including the deadline timer and manual-resume handback, are defined in persistence.md (The Wait Record and Ticks)."
200 items:
201 type: object
202 children:
203 item:
204 type: string
205 description: "What is outstanding: an experiment iteration, a judge batch, a timer"
206 arrives_as:
207 type: string
208 description: "The observable the wake delivers: a store file, a pushed ref, a host message naming the launch, a timer fire"
209 wake:
210 type: string
211 description: "The registered wake identifier or description, or unavailable for a manual-resume checkpoint"
212 on_arrival:
213 type: string
214 description: "The action to take: collect through 3.3, evaluate, re-check stopping"
216 hypothesis_backlog:
217 type: array
218 description: "Remaining hypotheses not yet tested"
219 items:
220 type: object
221 children:
222 description:
223 type: string
224 category:
225 type: string
226 priority:
227 type: string
228 enum: [high, medium, low]
229 dep_status:
230 type: string
231 enum: [approved, needs_approval, not_applicable]
232 required_deps:
233 type: array
234 items:
235 type: string
236 opportunity:
237 type: object
238 description: "Pre-implementation opportunity record; see opportunity_record below. Optional for legacy logs."
240opportunity_record:
241 children:
242 workload:
243 type: string
244 description: "Representative workload and input identity"
245 baseline:
246 type: string
247 description: "Revision or recorded snapshot against which the benefit is estimated"
248 evidence:
249 type: string
250 description: "Source location and observed cost with units or workload share; rubric evidence for qualitative work"
251 expected_benefit:
252 type: string
253 description: "Target metric, units, expected reduction/increase range or upper bound, and assumptions; unknown with the missing evidence and cheapest resolving measurement when not estimable"
254 confidence:
255 type: string
256 description: "Confidence and the evidence or uncertainty that justifies it"
257 cost_and_risk:
258 type: string
259 description: "Estimated implementation and measurement effort, behavioral risk, and required correctness checks"
261# ============================================================================
262# EXPERIMENT ENTRY
263# ============================================================================
265experiment_entry:
266 required_children:
268 iteration:
269 type: integer
270 description: "Sequential experiment number (1-indexed, monotonically increasing)"
272 batch:
273 type: integer
274 description: "Batch number this experiment was part of. Multiple experiments in the same batch ran in parallel."
276 hypothesis:
277 type: string
278 description: "Human-readable description of what this experiment tried"
280 category:
281 type: string
282 description: "Category for grouping and diversity selection (e.g., signal-extraction, graph-signals, embedding, algorithm, preprocessing)"
284 outcome:
285 type: enum
286 values:
287 - measured # measurement finished and metrics were persisted, awaiting batch evaluation / integration
288 - promising # eligible on current samples but the ladder still needs confirmation before keep
289 - kept # eligible, confirmed, and integrated onto the optimization branch
290 - not_selected # eligible after comparison but not integrated (not the winner, overlapping, or past the runner-up cap)
291 - reverted # compared and not eligible (regressed a required objective, or none improved), or eligible but its gain did not come from doing the work the metric stands for (loop 3.4)
292 - inconclusive # delta inside the comparison threshold; not a keep and not a demonstrated regression
293 - censored # aborted as noncompetitive under the predeclared futility bound
294 - degenerate # degenerate gate or smoke test failed -> immediately reverted, no judge evaluation
295 - error # measurement command crashed, timed out, or produced malformed output
296 - deferred_needs_approval # experiment needs an unapproved dependency -> set aside for batch approval
297 - timeout # measurement command exceeded timeout_seconds
298 - runner_up_kept # file-disjoint runner-up that was cherry-picked and re-measured successfully
299 - runner_up_reverted # file-disjoint runner-up that was cherry-picked but combined measurement was not better
300 description: >
301 The loop branches on this value.
302 'measured' and 'promising' are non-terminal: CP-3 persists raw metrics
303 before batch-level comparison, and 'promising' means the ladder still
304 needs confirmation samples. An eligible decide `keep` stays `measured`
305 until its diff is on the optimization branch. 'kept' and 'runner_up_kept'
306 mean that integration happened. 'not_selected' is terminal for an
307 eligible candidate that was not integrated. 'deferred_needs_approval'
308 items are re-presented at wrap-up. All other states are terminal for
309 that experiment.
311 optional_children:
313 opportunity:
314 type: object
315 description: "Copy of opportunity_record made before this experiment's implementation; never reconstructed from its result. Missing in legacy logs means unrecorded."
317 comparisons:
318 type: array
319 description: "One record per distinct reference/candidate/workload pairing used in a decision. Later in-place updates must not replace a previously persisted distinct pairing. Absent legacy evidence is unknown, not inferred from the current best."
320 items:
321 type: object
322 children:
323 kind:
324 type: string
325 enum: [standalone, integrated, holdout]
326 description: "holdout pairings are scored on the held-out set before a keep and at final confirmation; they never feed selection or hypothesis generation"
327 reference_revision:
328 type: string
329 description: "Identity that uniquely identifies the measured reference bytes. A revision is enough when it names those bytes; a HEAD shared by different uncommitted trees is not."
330 candidate_revision:
331 type: string
332 description: "Identity that uniquely identifies the measured candidate bytes. A revision is enough when it names those bytes; a HEAD shared by different uncommitted trees is not."
333 workload:
334 type: string
335 machine:
336 type: string
337 description: "Stable identifier of the machine or worker that produced both sides of this pairing, on every kind including holdout. Present for remote-backend results; absent means this checkout. Pairings from different machines are never compared to each other's absolute numbers."
338 measured_by:
339 type: string
340 enum: [orchestrator, worker, confirmation_worker]
341 description: "Who produced this pairing, on every kind including holdout. A remote winner needs one worker pairing and one pairing not produced by the candidate's author before keep; a remote holdout pairing is always orchestrator or confirmation_worker, never worker."
342 reference:
343 type: object
344 description: "Existing snapshot shape: metrics and judge, with aggregates and samples where available"
345 candidate:
346 type: object
347 description: "Existing snapshot shape: metrics and judge, with aggregates and samples where available"
348 uncertainty:
349 type: string
350 description: "Configured comparison method, observed variability, confirmation status, and any missing uncertainty evidence"
351 correctness:
352 type: string
353 description: "Checks performed and results or exact unverified constraints; passing a timing comparison alone is not correctness evidence"
355 changes:
356 type: array
357 description: "Files modified by this experiment"
358 items:
359 type: object
360 children:
361 file:
362 type: string
363 summary:
364 type: string
366 gates:
367 type: object
368 description: "Gate metric values from the measurement command"
370 gates_passed:
371 type: boolean
372 description: "Whether all degenerate gates passed"
374 diagnostics:
375 type: object
376 description: "Diagnostic metric values from the measurement command"
378 metrics:
379 type: object
380 description: "Required hard-objective snapshots (aggregate plus samples) that decide.mjs loads"
382 judge:
383 type: object
384 description: "Judge evaluation scores (only when primary type is 'judge' and gates passed)"
385 children:
386 # All fields from scoring.primary and scoring.secondary appear here
387 # Plus:
388 judge_cost_usd:
389 type: number
390 description: "Cost of judge calls for this experiment"
391 items:
392 type: array
393 description: "Per-item judge results. Feedback is what hypothesis generation reads; scores alone cannot say what to change."
394 items:
395 type: object
396 children:
397 item_id:
398 type: string
399 content_hash:
400 type: string
401 description: "Hash of the judged item bytes; the key for the run's judge cache, so identical output is not re-judged"
402 reason:
403 type: string
404 description: "The score explanation retained in worst_cases and cached with the judge result"
405 feedback:
406 type: string
407 description: "What is wrong with this item and what change would fix it, from the judge"
408 ambiguous:
409 type: boolean
410 # Plus every scoring field the rubric requests
412 cost:
413 type: object
414 description: "Per-experiment diagnostics recorded when the harness or judge reports them; absent otherwise. Never gated on."
415 children:
416 usd:
417 type: number
418 tokens:
419 type: integer
420 latency_seconds:
421 type: number
423 regressions:
424 type: array
425 description: "Case ids the reference passed and this candidate fails, from decide.mjs when measurement.per_case is true. Report only."
426 items:
427 type: string
429 worst_cases:
430 type: array
431 description: "Same shape as baseline.worst_cases, for this experiment's latest measurement"
433 primary_delta:
434 type: string
435 description: "Change in primary metric from current best (e.g., '+0.7', '-0.3')"
437 objective_results:
438 type: object
439 description: "Per-required-objective comparison from decide.mjs (verdict, delta, relative)"
441 sample_count:
442 type: integer
443 description: "How many harness samples were spent on this experiment"
445 next_measurement:
446 type: string
447 enum: [none, smoke, exploratory, add_sample, confirm, holdout]
448 description: "Ladder next step from decide.mjs; none when the decision is terminal. holdout means the selection comparison would keep, and the held-out comparison is still missing."
450 learnings:
451 type: string
452 description: "What was learned from this experiment. The agent reads these to avoid re-trying similar approaches and to inform new hypothesis generation. For a revert whose gain did not come from doing the work, this names the observation or constraint the harness lacks."
454 commit:
455 type: string
456 description: "Git commit SHA on the optimization branch (only for 'kept' and 'runner_up_kept' outcomes)"
458 deferred_reason:
459 type: string
460 description: "Why this experiment was deferred (only for 'deferred_needs_approval' outcome)"
462 error_message:
463 type: string
464 description: "Error details (only for 'error' and 'timeout' outcomes)"
466 merged_with:
467 type: integer
468 description: "Iteration number of the experiment this was merged with (only for 'runner_up_kept' and 'runner_up_reverted')"
470# ============================================================================
471# RESULT MARKER (result.yaml)
472# ============================================================================
474# Written by the measuring party immediately after measurement, before CP-3.
475# Local backends write it at the experiment worktree root with metrics only.
476# A remote worker writes it at its checkout root, commits it on the result ref,
477# and the orchestrator copies it to <state-root>/exp-<NNN>-result.yaml.
479result_marker:
480 children:
481 experiment:
482 type: integer
483 base_sha:
484 type: string
485 description: "Commit the worker started from and measured as baseline; must equal the dispatched base_sha or the result is an error"
486 head_sha:
487 type: string
488 description: "The candidate commit (mutable-scope changes only); result.yaml is committed alone on top of it, so the result ref's tip has head_sha as its parent (remote only)"
489 machine:
490 type: string
491 description: "Stable identifier of the measuring machine or worker (remote only)"
492 measured_at:
493 type: string
494 format: "ISO 8601 timestamp"
495 baseline:
496 type: object
497 description: "Paired snapshot of base_sha on the same machine: gates, metrics { name: { aggregate, samples } }, diagnostics (remote only)"
498 candidate:
499 type: object
500 description: "Snapshot of head_sha; local backends write only this side's raw metrics"
501 correctness:
502 type: string
503 description: "Checks the measuring party ran and their results, or none"
505# ============================================================================
506# OUTCOME STATE TRANSITIONS
507# ============================================================================
509# proposed (in hypothesis_backlog)
510# -> selected for batch
511# -> experiment dispatched
512# -> measurement completed
513# -> gates failed -> outcome: degenerate
514# -> measurement error -> outcome: error
515# -> measurement timeout -> outcome: timeout
516# -> smoke failed -> outcome: degenerate
517# -> futile / censored -> outcome: censored
518# -> gates passed
519# -> persist raw metrics -> outcome: measured or promising
520# -> judge evaluated (if type: judge)
521# -> decide.mjs eligible, next_measurement none -> stay measured until integration
522# -> diff on optimization branch -> outcome: kept
523# -> eligible leftover -> outcome: not_selected
524# -> runner-up, file-disjoint -> cherry-pick + re-measure
525# -> combined eligible and integrated -> outcome: runner_up_kept
526# -> combined not kept -> outcome: runner_up_reverted
527# -> inconclusive -> outcome: inconclusive
528# -> not eligible -> outcome: reverted
529# -> needs unapproved dep -> outcome: deferred_needs_approval
531# Only 'kept' and 'runner_up_kept' produce a commit on the optimization branch.
532# Only 'deferred_needs_approval' items are re-presented at wrap-up for approval.
534# ============================================================================
535# STRATEGY DIGEST (separate file)
536# ============================================================================
538# Written after each batch to:
539# <state-root>/strategy-digest.md
541# Contains a compressed summary of:
542# - What hypothesis categories have been tried
543# - Which approaches succeeded (kept) and which failed (reverted)
544# - The exploration frontier: what hasn't been tried yet
545# - Key learnings that should inform next hypotheses
546# - Failure themes: recurring judge `feedback` (or failing `cases`) on the
547# current best, grouped, with counts. Built from selection-sample items
548# only; holdout items never appear here.
550# The orchestrator reads the strategy digest (not the full experiment log)
551# when generating new hypotheses between batches.
553# ============================================================================
554# JUDGE CACHE (separate file)
555# ============================================================================
557# Written as judge results land, to:
558# <state-root>/judge-cache.yaml
560# Maps content_hash -> the judge's item result (scores, reason, feedback, ambiguous).
561# Scoped to one run and one rubric: identical item bytes are scored once and
562# reused, so a long run does not pay to re-judge output that did not change.
563# A rubric change is a new run and a new cache.