Skill Tracer
Skills › Scientific Agent Skills › scholar-evaluation › security_validation.md

security_validation.md

101 lines · skills/scholar-evaluation/references/security_validation.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Security Validation Record
2
3Historical validation date: **2026-07-23**
4
5The results below describe the earlier remediation, not a fresh scan. The
62026-10-01 refresh adds synthetic regression coverage for evaluation-round
7agreement, bounded pair aggregation, scale consistency, unrounded sensitivity,
8and companion membership. Collection-wide security scanning is recorded by
9the refresh coordinator; do not infer a current scan from this historical record.

Security Validation Record

Historical validation date: 2026-07-23

The results below describe the earlier remediation, not a fresh scan. The 2026-10-01 refresh adds synthetic regression coverage for evaluation-round agreement, bounded pair aggregation, scale consistency, unrounded sensitivity, and companion membership. Collection-wide security scanning is recorded by the refresh coordinator; do not infer a current scan from this historical record.

Raw lines 1 to 10
11## Baseline
13The repository `SECURITY.md` entry recorded **10 findings** with maximum
14severity **CRITICAL**:
16- four CRITICAL cross-file, environment, and network-exfiltration findings;
17- three MEDIUM credential, prompt, and environment-harvesting findings; and
18- three LOW cross-skill, command, and resource-use findings.
20The affected files were the two former schematic-generation scripts and the
21old `SKILL.md`.

Baseline

The repository SECURITY.md entry recorded 10 findings with maximum severity CRITICAL:

  • four CRITICAL cross-file, environment, and network-exfiltration findings;
  • three MEDIUM credential, prompt, and environment-harvesting findings; and
  • three LOW cross-skill, command, and resource-use findings.

The affected files were the two former schematic-generation scripts and the old SKILL.md.

Raw lines 11 to 22
23## Remediation
25- Deleted both schematic-generation scripts.
26- Removed all network requests, API-key handling, environment access,
27 environment-file loading, third-party model behavior, image handling,
28 child-process execution, cross-skill invocation, and mandatory figure
29 instructions.
30- Replaced the former recommendation-producing score calculator with bounded,
31 transparent descriptive rubric math.
32- Added a strict prohibition on automated or assisted hiring, promotion,
33 tenure, admissions, funding, awards, discipline, person ranking, and other
34 high-impact personnel decisions.
35- Added qualitative-first metric and prestige safeguards.
36- Added construct, provenance, content-evidence, rater, agreement, uncertainty,
37 missingness, not-applicable, traceability, sensitivity, subgroup, conflict,
38 appeal, accessibility, privacy, and accountable-human controls.
39- Added dependency-free, bounded local JSON/CSV tools with duplicate-key,
40 unknown-field, size, depth, non-finite-number, symbolic-link, and private
41 application-field rejection.
42- Added minimized reports that never copy source-document content or emit rater
43 identifiers.
44- Added static AST tests that prohibit network libraries, dynamic-code calls,
45 executable serialization, process launching, environment access, and the
46 deleted schematic files.

Remediation

  • Deleted both schematic-generation scripts.
  • Removed all network requests, API-key handling, environment access, environment-file loading, third-party model behavior, image handling, child-process execution, cross-skill invocation, and mandatory figure instructions.
  • Replaced the former recommendation-producing score calculator with bounded, transparent descriptive rubric math.
  • Added a strict prohibition on automated or assisted hiring, promotion, tenure, admissions, funding, awards, discipline, person ranking, and other high-impact personnel decisions.
  • Added qualitative-first metric and prestige safeguards.
  • Added construct, provenance, content-evidence, rater, agreement, uncertainty, missingness, not-applicable, traceability, sensitivity, subgroup, conflict, appeal, accessibility, privacy, and accountable-human controls.
  • Added dependency-free, bounded local JSON/CSV tools with duplicate-key, unknown-field, size, depth, non-finite-number, symbolic-link, and private application-field rejection.
  • Added minimized reports that never copy source-document content or emit rater identifiers.
  • Added static AST tests that prohibit network libraries, dynamic-code calls, executable serialization, process launching, environment access, and the deleted schematic files.
Raw lines 23 to 47
48## Validation results
50- Agent Skills reference validator: **PASS**
51- Dependency-free CLI help checks: **8 passed**
52- Synthetic standard-library tests: **29 passed**
53- Explicit AST parse with bytecode disabled: **8 scripts parsed**
54- Bytecode artifacts: **0**
55- IDE lints: **0**
56- Documented local-path check: **PASS**
57- External source links: **24 passed**
58- Direct behavioral security scan: **SAFE, 0 findings**
59- Pull-request gate with `--fail-on HIGH`: **PASS**
60 - CRITICAL: 0
61 - HIGH: 0
62 - LOW: 2 (latest final run; LOW-only LLM wording varied between runs)

Validation results

  • Agent Skills reference validator: PASS
  • Dependency-free CLI help checks: 8 passed
  • Synthetic standard-library tests: 29 passed
  • Explicit AST parse with bytecode disabled: 8 scripts parsed
  • Bytecode artifacts: 0
  • IDE lints: 0
  • Documented local-path check: PASS
  • External source links: 24 passed
  • Direct behavioral security scan: SAFE, 0 findings
  • Pull-request gate with --fail-on HIGH: PASS
  • CRITICAL: 0
  • HIGH: 0
  • LOW: 2 (latest final run; LOW-only LLM wording varied between runs)
Raw lines 48 to 63
64## Residual LOW findings
66The latest LLM-assisted pull-request scan reported:
681. **Bash is broad but constrained.** Informational: the manifest declares Bash
69 because the documented fixed `python3` commands are shell invocations. The
70 body limits Bash to those local commands, and the direct behavioral scan
71 confirms no process-launching code in the scripts.
722. **Invented missing-file aliases.** False positive: the scanner claimed
73 inconsistent alternate directories for bundled assets and references. A
74 direct text search found no alternate template-directory path; every
75 documented local path is under the actual `assets`, `references`, or
76 `scripts` directory; and the deterministic path-resolution test resolves
77 every backticked local path.
79The direct behavioral scan is clean. The residual findings neither permit data
80transmission nor create a missing-file fallback. The generated root
81`SECURITY.md` snapshot was intentionally not edited in this scoped refresh.

Residual LOW findings

The latest LLM-assisted pull-request scan reported:

  1. Bash is broad but constrained. Informational: the manifest declares Bash because the documented fixed python3 commands are shell invocations. The body limits Bash to those local commands, and the direct behavioral scan confirms no process-launching code in the scripts.
  2. Invented missing-file aliases. False positive: the scanner claimed inconsistent alternate directories for bundled assets and references. A direct text search found no alternate template-directory path; every documented local path is under the actual assets, references, or scripts directory; and the deterministic path-resolution test resolves every backticked local path.

The direct behavioral scan is clean. The residual findings neither permit data transmission nor create a missing-file fallback. The generated root SECURITY.md snapshot was intentionally not edited in this scoped refresh.

Raw lines 64 to 82
83## Reproduction
85```bash
86PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover \
87 -s tests/scholar-evaluation -p 'test_*.py' -v
89for script in skills/scholar-evaluation/scripts/*.py; do
90 PYTHONDONTWRITEBYTECODE=1 python3 "$script" --help >/dev/null || exit 1
91done
93uv run skills-ref validate skills/scholar-evaluation
95uv run skill-scanner scan skills/scholar-evaluation --use-behavioral
97uv run python scan_pr_skills.py \
98 --fail-on HIGH \
99 --output /tmp/scholar-evaluation-pr-scan.md \
100 skills/scholar-evaluation
101```

Reproduction

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover \
  -s tests/scholar-evaluation -p 'test_*.py' -v

for script in skills/scholar-evaluation/scripts/*.py; do
  PYTHONDONTWRITEBYTECODE=1 python3 "$script" --help >/dev/null || exit 1
done

uv run skills-ref validate skills/scholar-evaluation

uv run skill-scanner scan skills/scholar-evaluation --use-behavioral

uv run python scan_pr_skills.py \
  --fail-on HIGH \
  --output /tmp/scholar-evaluation-pr-scan.md \
  skills/scholar-evaluation
Raw lines 83 to 101
Runs code python scan_pr_skills.py :97