Skill Tracer
Skills › Evals Skills › eval-audit

eval-audit

skill · 183 lines · Apache-2.0 · skills/eval-audit/SKILL.md

The facts

Its description

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

The description the agent reads to decide when to use it.

From its settingsline 3

When it starts

The agent picks this skill when the user wants an existing LLM eval pipeline inspected and diagnosed — for example after inheriting an eval system, when unsure whether the evals can be trusted, or when no eval infrastructure exists yet and the user needs a starting point. It is explicitly not for building a new evaluator from scratch.

Things a person might say (the model's guesses)

  • “Can you audit our eval pipeline and tell me what's wrong with it?”
  • “We just inherited this eval setup from another team — is it trustworthy?”
  • “I don't have any evals yet, where should I start?”

A model's summary of when it is used.

A model's reading

What it leads to

Other skills this one hands off to, calls, requires or offers instead, as a model read them.

A model's reading

What leads to it

Other skills that hand off to this one, call it, require it or offer it instead, as a model read them.

A model's reading

Licence

Apache-2.0, from LICENSE

The terms its author shares it under, which let it be shown here.

From the repositorylicence text
Other files in this skill (1) Close

The text

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1---
2name: eval-audit
3description: >
4 Audit an LLM eval pipeline and surface problems: missing error analysis,
5 unvalidated judges, vanity metrics, etc. Use when
6 inheriting an eval system, when unsure whether evals are trustworthy, or as a
7 starting point when no eval infrastructure exists. Do NOT use when the goal
8 is to build a new evaluator from scratch (use error-discovery,
9 write-judge-prompt, or validate-evaluator instead).
10---
12# Eval Audit
14Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

Eval Audit

Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

Raw lines 1 to 15
16## Overview
181. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
192. Run diagnostic checks across six areas
203. Produce a findings report ordered by impact, with each finding linking to a fix

Overview

  1. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
  2. Run diagnostic checks across six areas
  3. Produce a findings report ordered by impact, with each finding linking to a fix
Raw lines 16 to 21
22## Prerequisites
24Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."

Prerequisites

Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."

Raw lines 22 to 25
Uses an outside tool MCP :24
26## Connecting to Eval Infrastructure
28Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.

Connecting to Eval Infrastructure

Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.

Raw lines 26 to 29
Uses an outside tool MCP :28
30## Diagnostic Checks
32Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.
34Prioritize findings by impact on the user's product. Present the most impactful findings first.

Diagnostic Checks

Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.

Prioritize findings by impact on the user's product. Present the most impactful findings first.

Raw lines 30 to 35
36### 1. Error Analysis
38**Check:** Has the user done systematic error analysis on real or synthetic traces?
40Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.
42**Finding if missing:** Evaluators built without error analysis measure generic qualities ("helpfulness", "coherence") instead of actual failure modes. Start with `error-discovery`, or `generate-synthetic-data` first if no traces exist.
44See: [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/index.html), [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)
46**Check:** Were failure categories brainstormed or observed?
48Generic labels borrowed from research ("hallucination score", "toxicity", "coherence") suggest brainstorming. Application-grounded categories ("missing query constraints", "wrong client tone", "fabricated property features") suggest observation.
50**Finding if brainstormed:** Generic categories miss application-specific failures and produce evaluators that score well on paper but miss real problems. Re-do with `error-discovery`, starting from traces.
52See: [Who Validates the Validators?](https://arxiv.org/abs/2404.12272)

1. Error Analysis

Check: Has the user done systematic error analysis on real or synthetic traces?

Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.

Finding if missing: Evaluators built without error analysis measure generic qualities ("helpfulness", "coherence") instead of actual failure modes. Start with error-discovery, or generate-synthetic-data first if no traces exist.

See: Your AI Product Needs Evals, LLM Evals FAQ

Check: Were failure categories brainstormed or observed?

Generic labels borrowed from research ("hallucination score", "toxicity", "coherence") suggest brainstorming. Application-grounded categories ("missing query constraints", "wrong client tone", "fabricated property features") suggest observation.

Finding if brainstormed: Generic categories miss application-specific failures and produce evaluators that score well on paper but miss real problems. Re-do with error-discovery, starting from traces.

See: Who Validates the Validators?

Raw lines 36 to 53
Names a skill error-discovery :42
Names a skill generate-synthetic-data :42
54### 2. Evaluator Design
56**Check:** Are evaluators binary pass/fail?
58Flag any that use Likert scales (1-5), letter grades (A-F), or numeric scores without a clear pass/fail threshold.
60**Finding if not binary:** Likert scales are difficult to calibrate. Annotators disagree on the difference between a 3 and a 4, and judges inherit that noise. Consider converting to binary pass/fail with explicit definitions using `write-judge-prompt`.
62See: [Creating an LLM Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/)
64**Check:** Do LLM judge prompts target specific failure modes?
66Flag any that evaluate holistically ("Is this response helpful?", "Rate the quality of this output").
68**Finding if vague:** Holistic judges produce unactionable verdicts. Each judge should check exactly one failure mode with explicit pass/fail definitions and few-shot examples. Use `write-judge-prompt`.
70**Check:** Are code-based checks used where possible?
72Flag LLM judges used for objectively checkable criteria: format validation, constraint satisfaction, keyword presence, schema conformance.
74**Finding if over-relying on judges:** Replace objective checks with code (regex, parsing, schema validation, execution tests). Reserve LLM judges for criteria requiring interpretation. Use `write-code-eval`.
76**Check:** Are similarity metrics used as primary evaluation?
78Flag ROUGE, BERTScore, cosine similarity, or embedding distance used as the main evaluator for generation quality.
80**Finding if present:** These metrics measure surface-level overlap, not correctness. They suit retrieval ranking but not generation evaluation. Replace with binary evaluators grounded in specific failure modes.
82See: [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)

2. Evaluator Design

Check: Are evaluators binary pass/fail?

Flag any that use Likert scales (1-5), letter grades (A-F), or numeric scores without a clear pass/fail threshold.

Finding if not binary: Likert scales are difficult to calibrate. Annotators disagree on the difference between a 3 and a 4, and judges inherit that noise. Consider converting to binary pass/fail with explicit definitions using write-judge-prompt.

See: Creating an LLM Judge That Drives Business Results

Check: Do LLM judge prompts target specific failure modes?

Flag any that evaluate holistically ("Is this response helpful?", "Rate the quality of this output").

Finding if vague: Holistic judges produce unactionable verdicts. Each judge should check exactly one failure mode with explicit pass/fail definitions and few-shot examples. Use write-judge-prompt.

Check: Are code-based checks used where possible?

Flag LLM judges used for objectively checkable criteria: format validation, constraint satisfaction, keyword presence, schema conformance.

Finding if over-relying on judges: Replace objective checks with code (regex, parsing, schema validation, execution tests). Reserve LLM judges for criteria requiring interpretation. Use write-code-eval.

Check: Are similarity metrics used as primary evaluation?

Flag ROUGE, BERTScore, cosine similarity, or embedding distance used as the main evaluator for generation quality.

Finding if present: These metrics measure surface-level overlap, not correctness. They suit retrieval ranking but not generation evaluation. Replace with binary evaluators grounded in specific failure modes.

See: LLM Evals FAQ

Raw lines 54 to 83
Names a skill write-judge-prompt :60
Names a skill write-code-eval :74
84### 3. Judge Validation
86**Check:** Are LLM judges validated against human labels?
88Look for: confusion matrices, TPR/TNR measurements, alignment scores. Judges in production with no validation data is a critical finding.
90**Finding if unvalidated:** An unvalidated judge may consistently miss failures or flag passing traces. Measure alignment using TPR and TNR on a held-out test set. Use `validate-evaluator`.
92See: [Creating an LLM Judge That Drives Business Results](https://hamel.dev/blog/posts/llm-judge/)
94**Check:** Is alignment measured with TPR/TNR or with raw accuracy?
96Flag "accuracy", "percent agreement", or Cohen's Kappa as the primary alignment metric.
98**Finding if using accuracy:** With class imbalance, raw accuracy is misleading: a judge that always says "Pass" gets 90% accuracy when 90% of traces pass but catches zero failures. Use TPR and TNR, which map directly to bias correction. Use `validate-evaluator`.
100**Check:** Is there a proper train/dev/test split?
102Check whether few-shot examples in judge prompts come from the same data used to measure judge performance.
104**Finding if leaking:** Using evaluation data as few-shot examples inflates alignment scores and hides real judge failures. Split into train (few-shot source), dev (iteration), and test (final measurement). Use `validate-evaluator`.

3. Judge Validation

Check: Are LLM judges validated against human labels?

Look for: confusion matrices, TPR/TNR measurements, alignment scores. Judges in production with no validation data is a critical finding.

Finding if unvalidated: An unvalidated judge may consistently miss failures or flag passing traces. Measure alignment using TPR and TNR on a held-out test set. Use validate-evaluator.

See: Creating an LLM Judge That Drives Business Results

Check: Is alignment measured with TPR/TNR or with raw accuracy?

Flag "accuracy", "percent agreement", or Cohen's Kappa as the primary alignment metric.

Finding if using accuracy: With class imbalance, raw accuracy is misleading: a judge that always says "Pass" gets 90% accuracy when 90% of traces pass but catches zero failures. Use TPR and TNR, which map directly to bias correction. Use validate-evaluator.

Check: Is there a proper train/dev/test split?

Check whether few-shot examples in judge prompts come from the same data used to measure judge performance.

Finding if leaking: Using evaluation data as few-shot examples inflates alignment scores and hides real judge failures. Split into train (few-shot source), dev (iteration), and test (final measurement). Use validate-evaluator.

Raw lines 84 to 105
Names a skill validate-evaluator :90
106### 4. Human Review Process
108**Check:** Who is reviewing traces?
110Determine whether domain experts or outsourced annotators are labeling data.
112**Finding if outsourced without domain expertise:** General annotators catch formatting errors but miss domain-specific failures (wrong medical dosage, incorrect legal citation, mismatched property features). Involve a domain expert.
114See: [A Field Guide to Improving AI Products](https://hamel.dev/blog/posts/field-guide/)
116**Check:** Are reviewers seeing full traces or just final outputs?
118**Finding if output-only:** Reviewing only the final output hides where the pipeline broke. Show the full trace: input, intermediate steps, tool calls, retrieved context, and final output.
120**Check:** How is data displayed to reviewers?
122Flag raw JSON, unformatted text, or spreadsheets with trace data in cells.
124**Finding if raw format:** Reviewers spend effort parsing data instead of judging quality. Format in natural representation: render markdown, syntax-highlight code, display tables as tables. Use `build-review-interface`.
126See: [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)

4. Human Review Process

Check: Who is reviewing traces?

Determine whether domain experts or outsourced annotators are labeling data.

Finding if outsourced without domain expertise: General annotators catch formatting errors but miss domain-specific failures (wrong medical dosage, incorrect legal citation, mismatched property features). Involve a domain expert.

See: A Field Guide to Improving AI Products

Check: Are reviewers seeing full traces or just final outputs?

Finding if output-only: Reviewing only the final output hides where the pipeline broke. Show the full trace: input, intermediate steps, tool calls, retrieved context, and final output.

Check: How is data displayed to reviewers?

Flag raw JSON, unformatted text, or spreadsheets with trace data in cells.

Finding if raw format: Reviewers spend effort parsing data instead of judging quality. Format in natural representation: render markdown, syntax-highlight code, display tables as tables. Use build-review-interface.

See: LLM Evals FAQ

Raw lines 106 to 127
Names a skill build-review-interface :124
128### 5. Labeled Data
130**Check:** Is there enough labeled data?
132For error analysis, ~100 traces is the rough target for saturation. For judge validation, ~50 Pass and ~50 Fail examples are needed for reliable TPR/TNR. If labeled data is sparse, collect more by sampling traces more effectively:
134- **Random:** Always include a random sample alongside other strategies to discover unknown issues.
135- **Clustering:** Group traces by semantic similarity and review representatives from each cluster.
136- **Data analysis:** Analyze statistics on latency, turns, tool calls, and tokens for outliers.
137- **Classification:** Use existing evals, a predictive model, or an LLM to surface problematic traces. Use with caution.
138- **Feedback:** Use explicit customer feedback (complaints, thumbs-down signals) to filter traces.
140**Finding if insufficient:** Small datasets produce unreliable failure rates and wide confidence intervals. Use the sampling strategies above to collect more labeled data, or supplement with `generate-synthetic-data`.

5. Labeled Data

Check: Is there enough labeled data?

For error analysis, ~100 traces is the rough target for saturation. For judge validation, ~50 Pass and ~50 Fail examples are needed for reliable TPR/TNR. If labeled data is sparse, collect more by sampling traces more effectively:

  • Random: Always include a random sample alongside other strategies to discover unknown issues.
  • Clustering: Group traces by semantic similarity and review representatives from each cluster.
  • Data analysis: Analyze statistics on latency, turns, tool calls, and tokens for outliers.
  • Classification: Use existing evals, a predictive model, or an LLM to surface problematic traces. Use with caution.
  • Feedback: Use explicit customer feedback (complaints, thumbs-down signals) to filter traces.

Finding if insufficient: Small datasets produce unreliable failure rates and wide confidence intervals. Use the sampling strategies above to collect more labeled data, or supplement with generate-synthetic-data.

Raw lines 128 to 141
Names a skill generate-synthetic-data :140
142### 6. Pipeline Hygiene
144**Check:** Is error analysis re-run after significant changes?
146Check when error analysis was last performed relative to model switches, prompt rewrites, new features, or production incidents.
148**Finding if stale:** Failure modes shift after pipeline changes, and evaluators built for the old pipeline miss new failure types. Re-run error analysis after every significant change.
150**Check:** Are evaluators maintained?
152Look for periodic re-validation of judges or refreshed evaluation datasets.
154**Finding if set-and-forget:** Evaluators degrade as the pipeline evolves. Re-validate judges against fresh human labels and update eval datasets to reflect current usage.

6. Pipeline Hygiene

Check: Is error analysis re-run after significant changes?

Check when error analysis was last performed relative to model switches, prompt rewrites, new features, or production incidents.

Finding if stale: Failure modes shift after pipeline changes, and evaluators built for the old pipeline miss new failure types. Re-run error analysis after every significant change.

Check: Are evaluators maintained?

Look for periodic re-validation of judges or refreshed evaluation datasets.

Finding if set-and-forget: Evaluators degrade as the pipeline evolves. Re-validate judges against fresh human labels and update eval datasets to reflect current usage.

Raw lines 142 to 155
156## No Eval Infrastructure
158If the user has no eval artifacts (no traces, no evaluators, no labeled data):
1601. Start with `error-discovery` on a sample of real traces.
1612. If no production data exists, use `generate-synthetic-data` to create test inputs, run them through the pipeline, then apply `error-discovery` to the resulting traces.
1623. Do not recommend building evaluators, judges, or dashboards before completing error analysis.

No Eval Infrastructure

If the user has no eval artifacts (no traces, no evaluators, no labeled data):

  1. Start with error-discovery on a sample of real traces.
  2. If no production data exists, use generate-synthetic-data to create test inputs, run them through the pipeline, then apply error-discovery to the resulting traces.
  3. Do not recommend building evaluators, judges, or dashboards before completing error analysis.
Raw lines 156 to 163
Names a skill error-discovery :160
Names a skill generate-synthetic-data :161
164## Report Format
166Present findings ordered by impact. For each:
168```
169### [Problem Title]
170**Status:** [Problem exists / OK / Cannot determine]
171[1-2 sentence explanation of the specific problem found]
172**Fix:** [Concrete action, referencing a skill or article]
173```
175Group under the six diagnostic areas. Omit areas where no problems were found.

Report Format

Present findings ordered by impact. For each:

### [Problem Title]
**Status:** [Problem exists / OK / Cannot determine]
[1-2 sentence explanation of the specific problem found]
**Fix:** [Concrete action, referencing a skill or article]

Group under the six diagnostic areas. Omit areas where no problems were found.

Raw lines 164 to 176
177## Anti-Patterns
179- Running the audit as a checklist without inspecting actual artifacts.
180- Reporting generic advice disconnected from what was found in the user's pipeline.
181- Recommending evaluators before error analysis is complete.
182- Suggesting LLM judges for failures that code-based checks can handle.
183- Treating this audit as a one-time event. Re-audit after significant pipeline changes.

Anti-Patterns

  • Running the audit as a checklist without inspecting actual artifacts.
  • Reporting generic advice disconnected from what was found in the user's pipeline.
  • Recommending evaluators before error analysis is complete.
  • Suggesting LLM judges for failures that code-based checks can handle.
  • Treating this audit as a one-time event. Re-audit after significant pipeline changes.
Raw lines 177 to 183

From ai-evals-course/evals-skills at 80d5f7b, 2026-09-24. Skills read by DeepSeek V4.1 Flash on 2026-11-04. Summary and techniques by Claude Opus 5.5 on 2026-11-06.