Eval Skills is a set of nine skills that guide AI coding agents in building evals for a specific product, as opposed to benchmarks for foundation models. It is meant for teams evaluating their own LLM pipelines. An entry skill sends the user to error discovery, audits, judge writing, validation, synthetic data or RAG evaluation.
Why read it
evals-start is a single table that maps the user's situation to a skill. It tells the agent to say which skill it is loading, to follow that skill's whole workflow instead of improvising, and to ask the user when no row clearly fits. Keeping the entry point this small means it never drifts out of step with the workflows it points to.
Descriptions say when not to use a skill and name the alternative. That keeps sibling skills from overlapping when the agent picks one.
eval-audit pairs each check with the artifacts to look for, the finding to record and the skill that fixes it, then sets a fixed report template. Its anti-patterns warn against running the audit as a checklist without inspecting real artifacts.
Rules come with the reason they exist, such as why Likert scales fail or why the critique comes before the verdict. Knowing the reason lets the agent apply the rule to new cases and spot real exceptions. For example, Likert scoring is allowed for filtering synthetic queries.
The judge skill includes full Pass, Fail and borderline examples, each with a critique, from one domain (real estate). It tells the agent that borderline cases teach the most, and it carries the same domain into the synthetic-data skill.
error-discovery spells out browser details an agent tends to get wrong. It covers keeping the text selection when a popover opens and how to compute margin-note offsets without counting the scroll offset twice. Each rule maps a dimension of the data to exactly one visual channel.
The interactive skill tells the agent what to do in a non-interactive run: build and smoke-test the app, stop the server, and never claim the review loop ran. It also asks for progress updates at every phase.
error-discovery covers phases 1–4 in SKILL.md and moves the ongoing review loop to review-loop.md. The agent reads the second file only once the app is running.
A model read each skill to find these. Top to bottom is the usual order. Select a box to read the skill, or a line to see the text behind it.
Every one is a skill. Started by: You or the agent.
| SKILL | DESCRIPTION | LINKS |
|---|---|---|
| build-review-interface | Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels. | 1 |
| error-discovery | Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes. | 1 |
| eval-audit | Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead). | 6 |
| evals-start | Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly. | 8 |
| evaluate-rag | Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies. | 1 |
| generate-synthetic-data | Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs. | 0 |
| validate-evaluator | Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`). | 2 |
| write-code-eval | Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation. | 3 |
| write-judge-prompt | Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`. | 3 |
From ai-evals-course/evals-skills at 80d5f7b, 2026-09-24. Skills read by DeepSeek V4.1 Flash on 2026-11-04. Summary and techniques by Claude Opus 5.5 on 2026-11-06.