evals-start
The facts
Its description
Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.
The description the agent reads to decide when to use it.
When it starts
This is the plugin's entry point for eval work: the agent picks it when someone asks for help with evals in general, says they don't know where to begin, or asks for something no other eval skill in the plugin matches. It is explicitly not meant to fire when a more specific eval skill already matches, in which case that skill is loaded directly.
Things a person might say (the model's guesses)
- “I want to start evaluating my LLM app but I have no idea where to begin.”
- “Can you help me with evals?”
- “I need to set up evals for my pipeline — where do I start?”
A model's summary of when it is used.
Stops for a person
1 line stops to ask a person
Lines that stop and wait for a person before going on. Found by matching words, so treat them as pointers.
What it leads to
- hands off to build-review-interface when a custom annotation interface is wanted for another labeling task line 22
- hands off to error-discovery when the user has traces and no established taxonomy line 16
- hands off to eval-audit when an existing eval pipeline's trustworthiness is in question line 17
- hands off to evaluate-rag when the pipeline to evaluate is a RAG pipeline line 23
- hands off to generate-synthetic-data when there are no traces to review yet line 21
- hands off to validate-evaluator when an existing judge or evaluator's quality needs checking line 20
- hands off to write-code-eval when the known failure mode can be checked by code line 18
- hands off to write-judge-prompt when the known failure mode needs an LLM judge line 19
Other skills this one hands off to, calls, requires or offers instead, as a model read them.
Licence
Apache-2.0, from LICENSE
The terms its author shares it under, which let it be shown here.
Other files in this skill (1) Close
The text
Most marks come from matching words in the file, so treat them as pointers.
Evals Start
This plugin splits eval work into targeted skills. Your job here is small: find the row below that matches the user's situation, tell the user which skill you are loading and why, then load that skill and follow its workflow from start to finish instead of improvising your own version of it.
| Situation | Skill to load |
|---|---|
| Has traces, wants to find failure modes, no established taxonomy yet | error-discovery |
| Has an existing eval pipeline and wants to know if it can be trusted | eval-audit |
| Has a known failure mode that code can check (e.g., format, schema, regex, execution) | write-code-eval |
| Has a known failure mode and wants an LLM judge for it | write-judge-prompt |
| Has an LLM judge or evaluator and wants to check its quality | validate-evaluator |
| Has no traces to review yet | generate-synthetic-data, then error-discovery |
| Wants a custom annotation interface for some other labeling task | build-review-interface |
| Wants to evaluate a RAG pipeline | evaluate-rag |
Most requests that mention error analysis with traces in hand mean error-discovery. New users with an existing pipeline usually need eval-audit first. This file holds only routing. When in doubt about which row fits, ask the user instead of guessing. The workflow lives in the targeted skill.
From ai-evals-course/evals-skills at 80d5f7b, 2026-09-24. Skills read by DeepSeek V4.1 Flash on 2026-11-04. Summary and techniques by Claude Opus 5.5 on 2026-11-06.