Sweep state schema (v1)

This is the canonical, versioned contract for the ce-sweep state file. The deterministic state engine (scripts/sweep-state.py) is the only program that writes the file. Every peer agent (source connectors, the analyzer, the orchestrator) reads the file and changes it exclusively through the engine's subcommands, so the rules below are enforced in one place. Read this before touching state.

Top-level shape

schema_version: 1
lease:
  writer: "sweep-2026-07-02-cron"
  timestamp: "2026-07-02T12:00:00+00:00"
  ttl_minutes: 60
sources:
  "slack:C42":
    cursor: "1699999999.000100"
    sensitive: true          # optional; config-derived
items:
  "slack:C42:1699999999.000100":
    source: "slack:C42"
    status: "acknowledged"
    # ...arbitrary connector fields...
last_run:
  timestamp: "2026-07-02T12:05:00+00:00"
  outcome: "completed"
  writer: "sweep-2026-07-02-cron"
  counts: {"ingested": 5, "closed": 1}

Compatibility rule (forward/backward)

The engine is deliberately additive-safe so a newer writer and an older reader can share a file:

Bump schema_version only for a change that a v1 reader could misinterpret. Purely additive fields do not require a bump.

Status enum

Closed set of known lifecycle states. Unknown values are preserved, never dropped, so a future state can roll out writer-first.

Evidence fields and the validate downgrade rule

A closed item is a claim that work shipped and was verified, so the engine holds it to proof. An item may only remain closed if it carries all three:

validate scans every item and downgrades any closed item missing (or with a falsy value for) any of these back to fix_pending. It then rewrites the file and returns the list of downgraded ids. This self-heals a state left inconsistent by a crashed run. validate is lease-agnostic. It is a repair, run at sweep start.

sensitive semantics

The primary sensitivity mechanism is per-item. The orchestrator reads each source's config sensitive flag and includes "sensitive": true in the item JSON on every upsert-item for that source (run.md phase 2d). A sensitive: true on a source entry in state is a defensive fallback the engine also honors, but nothing seeds it today. The per-item flag is what enforces R28, so sensitivity works even though source entries carry only a cursor. On any upsert-item where either the item or its source entry is sensitive, the engine **drops body and quote before writing**, so redacted content never reaches disk. All other fields (title, url, status, ids) are retained. Redaction happens at write time, so flipping a source to sensitive protects only items written after the flag is set. Re-ingest to redact prior items.

id-keyed merge rule

Each writer changes only the keys it sends, not the whole file. upsert-item performs an id-keyed merge. It loads the existing item, replaces only the keys present in the incoming JSON, and preserves every other field already on that item. source is always (re)set from --source. No subcommand semantically rewrites the whole file. Each one changes only the keys it was given and preserves the rest, even though the physical write re-emits the file atomically. This lets independent connectors and the analyzer touch the same item across passes without clobbering each other's fields.

Lease (single-writer mutex)

Rules the engine enforces:

• lease-acquire succeeds (OK) when the lease is free or already held by the same writer (re-entrant, re-stamps). It returns LOCKED when a live lease is held by another writer, or STALE-RECLAIMED (with previous_writer / previous_timestamp) when it takes over a lease older than its TTL.
• Every mutating call (upsert-item, cursor-advance) re-checks ownership before writing and returns LEASE-LOST (no write) if the caller is not the current holder. On success it re-stamps the lease timestamp so a long sweep keeps the lease alive.
• lease-release clears the caller's own lease (OK, also OK if none is held). Releasing another writer's lease returns LEASE-LOST and does not write.
• Staleness is only asserted when it can be proven from parseable timestamps. An unparseable lease timestamp is treated as live and is never stomped.

Topology scope

The lease's guarantee depends on where the state file lives:

TTL-based reclaim (STALE-RECLAIMED) is what lets a crashed or killed writer's lease be taken over after ttl_minutes without manual cleanup.

run-record

Records the outcome of a sweep run under last_run.

run-record is intentionally lease-agnostic. A run that aborted precisely because the lease was LOCKED (outcome: aborted-locked) must still be able to record that fact, but that write happens while the lease holder is mid-sweep. To keep it from clobbering the holder's concurrent upserts, every mutating subcommand holds an OS advisory lock (flock on <state>.lock) across its whole load-modify-write, so two concurrent invocations serialize their writes regardless of lease ownership. The lease decides which writer is running the sweep. The file lock decides which process is writing the file right now. The .lock file is ephemeral and never committed. The skill's commit step adds only the state file and the plan, never -A.

Engine status words

Every subcommand prints one status word on line 1, then an optional JSON payload on line 2. Operational conditions exit 0 (never a traceback). Only CLI misuse exits non-zero.

YAML subset

The state file is genuine YAML restricted to a small, deterministic subset so a stdlib serializer/parser round-trips it exactly. Any YAML parser can read it; only the engine writes it.

A file that fails to parse under these rules, or that parses without a schema_version, is CORRUPT. The engine reports it and refuses to overwrite, so a hand-mangled file is never silently clobbered.