Skill Tracer
Skills › Scientific Agent Skills › get-available-resources › snapshot_schema.md

snapshot_schema.md

185 lines · skills/get-available-resources/references/snapshot_schema.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Snapshot Schema 1.1
2
3The detector emits one JSON object with sorted keys. Values vary by observation,
4but field names and meanings are stable for schema `1.1`.
5

Snapshot Schema 1.1

The detector emits one JSON object with sorted keys. Values vary by observation, but field names and meanings are stable for schema 1.1.

Raw lines 1 to 5
6## Top-level contract
7
8- `schema_version`: `"1.1"`.
9- `snapshot_kind`: `"effective_resource_snapshot"`.
10- `observed_at`: UTC observation time.
11- `completeness`: `complete`, `complete_with_informational_notes`, or
12 `partial`.
13- `platform`: OS family, architecture, and Python version. Hostname is omitted.
14- `privacy`: explicit redaction flags.
15- `cpu`, `memory`, `disk`, `accelerators`: resource observations.
16- `cgroup_v2`, `container`, `scheduler`: execution-context observations.
17- `warnings`: bounded sorted warning records.
18- `provenance`: bounded sorted source/status records.
20Null means unavailable, not zero and not unlimited. Zero is used only when a
21source explicitly establishes zero (for example, a named accelerator visibility
22variable that hides all devices).

Top-level contract

  • schema_version: "1.1".
  • snapshot_kind: "effective_resource_snapshot".
  • observed_at: UTC observation time.
  • completeness: complete, complete_with_informational_notes, or partial.
  • platform: OS family, architecture, and Python version. Hostname is omitted.
  • privacy: explicit redaction flags.
  • cpu, memory, disk, accelerators: resource observations.
  • cgroup_v2, container, scheduler: execution-context observations.
  • warnings: bounded sorted warning records.
  • provenance: bounded sorted source/status records.

Null means unavailable, not zero and not unlimited. Zero is used only when a source explicitly establishes zero (for example, a named accelerator visibility variable that hides all devices).

Raw lines 6 to 23
24## CPU
26`cpu.host`:
28- `logical`: system-visible logical CPUs.
29- `physical`: system-visible physical cores, or null. This value is not
30 converted into an effective process count.
32`cpu.process`:
34- `affinity_logical`: size of the current affinity set when supported.
35- `python_available_logical`: `os.process_cpu_count()` when supported.
37`cpu.cgroup_v2`:
39- `cpuset_logical`: count from `cpuset.cpus.effective`.
40- `quota_cores`: most restrictive finite ancestor `cpu.max` ratio. This may be
41 fractional.
43`cpu.effective`:
45- `capacity_cores`: minimum positive host/process/cgroup/scheduler capacity.
46- `worker_ceiling`: conservative bounded floor for CPU process workers.
47- `limiting_sources`: sources tied at that minimum.
49The effective value is intentionally not called a physical-core count.

CPU

cpu.host:

  • logical: system-visible logical CPUs.
  • physical: system-visible physical cores, or null. This value is not converted into an effective process count.

cpu.process:

  • affinity_logical: size of the current affinity set when supported.
  • python_available_logical: os.process_cpu_count() when supported.

cpu.cgroup_v2:

  • cpuset_logical: count from cpuset.cpus.effective.
  • quota_cores: most restrictive finite ancestor cpu.max ratio. This may be fractional.

cpu.effective:

  • capacity_cores: minimum positive host/process/cgroup/scheduler capacity.
  • worker_ceiling: conservative bounded floor for CPU process workers.
  • limiting_sources: sources tied at that minimum.

The effective value is intentionally not called a physical-core count.

Raw lines 24 to 50
51## Memory
53`memory.host` preserves system-visible `total_bytes` and `available_bytes`.
55`memory.cgroup_v2` preserves current-cgroup usage and hierarchical effective
56limits:
58- `current_bytes`
59- `available_bytes`
60- `high_bytes`
61- `max_bytes`
63`memory.effective`:
65- `hard_limit_bytes`: minimum of finite host total, cgroup hard limit, and
66 interpretable scheduler allocation.
67- `available_bytes`: minimum of host available, hierarchical cgroup remaining,
68 and scheduler upper bound.
69- `pressure_threshold_bytes`: cgroup `memory.high`; it is not relabeled as a
70 hard limit.
71- `hard_limit_sources` and `available_limiting_sources`: tied minimum sources.
73`memory.model` is `unified_cpu_gpu` on Apple silicon and `system_ram`
74otherwise. Unified GPU memory is not added again as dedicated VRAM.

Memory

memory.host preserves system-visible total_bytes and available_bytes.

memory.cgroup_v2 preserves current-cgroup usage and hierarchical effective limits:

  • current_bytes
  • available_bytes
  • high_bytes
  • max_bytes

memory.effective:

  • hard_limit_bytes: minimum of finite host total, cgroup hard limit, and interpretable scheduler allocation.
  • available_bytes: minimum of host available, hierarchical cgroup remaining, and scheduler upper bound.
  • pressure_threshold_bytes: cgroup memory.high; it is not relabeled as a hard limit.
  • hard_limit_sources and available_limiting_sources: tied minimum sources.

memory.model is unified_cpu_gpu on Apple silicon and system_ram otherwise. Unified GPU memory is not added again as dedicated VRAM.

Raw lines 51 to 75
76## Disk
78- `capacity_bytes`: total working-filesystem capacity.
79- `free_bytes`: filesystem free blocks.
80- `user_available_bytes`: user-available blocks where the OS exposes them.
81- `writable`: result of a non-writing access check.
82- `writability_check`: makes clear that no write probe occurred.
83- `scope`: `working_filesystem_path_redacted`.
85None of these values proves that a filesystem or project quota permits a write
86of the same size.

Disk

  • capacity_bytes: total working-filesystem capacity.
  • free_bytes: filesystem free blocks.
  • user_available_bytes: user-available blocks where the OS exposes them.
  • writable: result of a non-writing access check.
  • writability_check: makes clear that no write probe occurred.
  • scope: working_filesystem_path_redacted.

None of these values proves that a filesystem or project quota permits a write of the same size.

Raw lines 76 to 87
88## Accelerators
90`accelerators.devices` contains management-visible or explicitly
91platform-inferred candidates:
93- `vendor`: hardware vendor (`nvidia`, `amd`, `apple`, `intel`, or `unknown`).
94 macOS display discovery retains the actual vendor while offering Metal as
95 a backend candidate.
96- `device_class`: keeps integrated and discrete GPU concepts distinct.
97- `backend_candidate`: `cuda`, `rocm`, or `metal`.
98- `management_query`: visibility evidence.
99- `device_permission`: `not_tested`; a query does not prove device-node access.
100- `runtime_compatibility`: `not_tested` in detector output.
101- `memory.model`: dedicated, unified, device-reported pool, or unknown.
102 AMD records use `reported_total_bytes`; `dedicated_total_bytes` stays null
103 because a management query does not resolve shared-memory topology.
104- `local_index`: local query index; stable UUIDs and PCI addresses are omitted.
106`candidate_counts` is a query count, not a usable-device count.
107`candidate_upper_bounds` conservatively intersects query count with parsed
108visibility/allocation counts when available. These bounds concern management
109records, not MIG/partition/runtime-device enumeration. `runtime_usable_devices` remains
110null because no framework runtime is loaded.
112`visibility_environment` includes only four allowlisted variable names. Raw
113values are never emitted.

Accelerators

accelerators.devices contains management-visible or explicitly platform-inferred candidates:

  • vendor: hardware vendor (nvidia, amd, apple, intel, or unknown). macOS display discovery retains the actual vendor while offering Metal as a backend candidate.
  • device_class: keeps integrated and discrete GPU concepts distinct.
  • backend_candidate: cuda, rocm, or metal.
  • management_query: visibility evidence.
  • device_permission: not_tested; a query does not prove device-node access.
  • runtime_compatibility: not_tested in detector output.
  • memory.model: dedicated, unified, device-reported pool, or unknown. AMD records use reported_total_bytes; dedicated_total_bytes stays null because a management query does not resolve shared-memory topology.
  • local_index: local query index; stable UUIDs and PCI addresses are omitted.

candidate_counts is a query count, not a usable-device count. candidate_upper_bounds conservatively intersects query count with parsed visibility/allocation counts when available. These bounds concern management records, not MIG/partition/runtime-device enumeration. runtime_usable_devices remains null because no framework runtime is loaded.

visibility_environment includes only four allowlisted variable names. Raw values are never emitted.

Raw lines 88 to 114
115## Scheduler and cgroup
117`scheduler.fields_read` lists allowlisted Slurm names that were present.
118`scheduler.allocation` contains parsed bounded values and scopes.
119`scheduler.enforcement` remains `unknown`; variables alone do not prove
120confinement.
122`cgroup_v2.scope` says only `root`, `non_root`, `unknown`, or `not_applicable`.
123The cgroup path is not emitted. Scope is relative to the visible cgroup2 mount;
124hidden ancestors and cgroup v1 constraints are outside this observation.
125Unreadable or invalid membership keeps scope unknown.
127`container.detected` requires a known marker. A `cgroup_limit` can appear as
128evidence without asserting that the process is in a container.

Scheduler and cgroup

scheduler.fields_read lists allowlisted Slurm names that were present. scheduler.allocation contains parsed bounded values and scopes. scheduler.enforcement remains unknown; variables alone do not prove confinement.

cgroup_v2.scope says only root, non_root, unknown, or not_applicable. The cgroup path is not emitted. Scope is relative to the visible cgroup2 mount; hidden ancestors and cgroup v1 constraints are outside this observation. Unreadable or invalid membership keeps scope unknown.

container.detected requires a known marker. A cgroup_limit can appear as evidence without asserting that the process is in a container.

Raw lines 115 to 129
130## Warnings and provenance
132Warnings use:
134```json
136 "code": "STABLE_MACHINE_CODE",
137 "component": "cpu",
138 "message": "Human-readable, sanitized explanation.",
139 "severity": "info"
141```
143Probe exception text, stderr, paths, hostnames, device UUIDs, and broad
144environment content are excluded.
146Provenance uses:
148```json
150 "component": "cpu.process.affinity_logical",
151 "source": "os.sched_getaffinity",
152 "status": "ok"
154```
156Possible status values include `ok`, `unavailable`, `absent`, `skipped`,
157`not_found`, `timeout`, `truncated`, `error`, and `parse_error`.

Warnings and provenance

Warnings use:

{
  "code": "STABLE_MACHINE_CODE",
  "component": "cpu",
  "message": "Human-readable, sanitized explanation.",
  "severity": "info"
}

Probe exception text, stderr, paths, hostnames, device UUIDs, and broad environment content are excluded.

Provenance uses:

{
  "component": "cpu.process.affinity_logical",
  "source": "os.sched_getaffinity",
  "status": "ok"
}

Possible status values include ok, unavailable, absent, skipped, not_found, timeout, truncated, error, and parse_error.

Raw lines 130 to 158
159## Validation and diff
161Validate:
163```bash
164python scripts/snapshot_tools.py validate resource-snapshot.json
165```
167Diff while ignoring `observed_at`:
169```bash
170python scripts/snapshot_tools.py diff before.json after.json
171```
173Use `--include-volatile` only when timestamp changes matter. Diff output is
174bounded to 512 changes.
176All helper inputs are regular, non-symlink JSON files no larger than 1 MiB.
177Output defaults to stdout. Explicit file output is restricted to a `.json`
178filename in the current directory, refuses overwrite unless `--force` is used,
179and is opened with mode 0600 on POSIX. Windows privacy also depends on directory
180ACLs. Forced output refuses non-regular files and hard links.
182The planner retains schema version 1.1 and adds `recommendation.status`. A
183known budget that cannot fit one worker yields `insufficient_memory`, zero
184workers, and zero threads; no job should be launched. `review_required` means
185CPU capacity or available RAM is unknown. `provisional` is still an estimate.

Validation and diff

Validate:

python scripts/snapshot_tools.py validate resource-snapshot.json

Diff while ignoring observed_at:

python scripts/snapshot_tools.py diff before.json after.json

Use --include-volatile only when timestamp changes matter. Diff output is bounded to 512 changes.

All helper inputs are regular, non-symlink JSON files no larger than 1 MiB. Output defaults to stdout. Explicit file output is restricted to a .json filename in the current directory, refuses overwrite unless --force is used, and is opened with mode 0600 on POSIX. Windows privacy also depends on directory ACLs. Forced output refuses non-regular files and hard links.

The planner retains schema version 1.1 and adds recommendation.status. A known budget that cannot fit one worker yields insufficient_memory, zero workers, and zero threads; no job should be launched. review_required means CPU capacity or available RAM is unknown. provisional is still an estimate.

Raw lines 159 to 185
Runs code python scripts/snapshot_tools.py :164