write-code-eval
The facts
Its description
Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.
The description the agent reads to decide when to use it.
When it starts
The agent picks this skill when someone wants to write a code-based evaluator for a failure mode whose rule is objective and checkable from a trace, such as output structure, string matching, or comparison against a reference. It is chosen over the judge-prompt skill when the rule does not need interpretation.
Things a person might say (the model's guesses)
- “I found a failure mode where the model returns JSON with missing fields — can you write a check for that?”
- “Help me write a code evaluator for bad tool calls.”
- “Can I check with code whether the cited documents were actually retrieved?”
A model's summary of when it is used.
What it leads to
- calls validate-evaluator when the rule uses a proxy for human judgment line 15
- requires error-discovery line 11
- an alternative: write-judge-prompt when the rule requires interpretation line 13
Other skills this one hands off to, calls, requires or offers instead, as a model read them.
What leads to it
- evals-start hands off to this when the known failure mode can be checked by code evals-start, line 18
- eval-audit calls this only for objectively checkable criteria eval-audit, line 74
- validate-evaluator is an alternative to this for code-based evaluators validate-evaluator, line 7
- write-judge-prompt is an alternative to this when the failure mode can be checked with code write-judge-prompt, line 8
Other skills that hand off to this one, call it, require it or offer it instead, as a model read them.
Licence
Apache-2.0, from LICENSE
The terms its author shares it under, which let it be shown here.
Other files in this skill (1) Close
The text
Most marks come from matching words in the file, so treat them as pointers.
Write a code evaluator
Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
- State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use
write-judge-prompt. - Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
- Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.
Examples
| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |
| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
| Wrong value | Compare the output with a reference value |
Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.
Raw lines 17 to 27From ai-evals-course/evals-skills at 80d5f7b, 2026-09-24. Skills read by DeepSeek V4.1 Flash on 2026-11-04. Summary and techniques by Claude Opus 5.5 on 2026-11-06.