Skill Tracer
Skills › Evals Skills › build-review-interface

build-review-interface

skill · 96 lines · Apache-2.0 · skills/build-review-interface/SKILL.md

The facts

Its description

Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.

The description the agent reads to decide when to use it.

From its settingsline 3

When it starts

The agent would pick this skill when someone wants a tool for humans to look at LLM traces and label them — building an annotation/review page, collecting pass/fail judgments and notes, or setting up a browser UI for human feedback on model outputs.

Things a person might say (the model's guesses)

  • “I need a way for my team to review our LLM traces and mark them pass or fail”
  • “Build me an annotation interface for my eval data so I can collect human labels”
  • “Can you make a simple web page where I can go through traces and leave notes?”

A model's summary of when it is used.

A model's reading

What it leads to

Other skills this one hands off to, calls, requires or offers instead, as a model read them.

A model's reading

What leads to it

Other skills that hand off to this one, call it, require it or offer it instead, as a model read them.

A model's reading

Licence

Apache-2.0, from LICENSE

The terms its author shares it under, which let it be shown here.

From the repositorylicence text
Other files in this skill (1) Close

The text

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1---
2name: build-review-interface
3description: >
4 Build a custom browser-based annotation interface tailored to your data for
5 reviewing LLM traces and collecting structured feedback. Use when you need to
6 build an annotation tool, review traces, or collect human labels.
7---
8
9# Build a Custom Annotation Interface

Build a Custom Annotation Interface

Raw lines 1 to 10
11## Overview
13Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.

Overview

Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.

Raw lines 11 to 14
15## Data Display
17Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.
19- **Collapse repetitive elements.** If every trace shares the same system prompt, put it in a `<details>` toggle.
20- **Extract and surface key metadata.** If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
21- **Color-code by role or status.** Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
22- **Group related elements visually.** Tool calls and their responses should be visually linked (indentation, shared border).
23- **Collapse what doesn't help judgment.** Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
24- **Highlight what matters most.** Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
25- **Show the full trace.** Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
26- **Sanitize rendered content.** Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.

Data Display

Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.

  • Collapse repetitive elements. If every trace shares the same system prompt, put it in a <details> toggle.
  • Extract and surface key metadata. If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
  • Color-code by role or status. Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
  • Group related elements visually. Tool calls and their responses should be visually linked (indentation, shared border).
  • Collapse what doesn't help judgment. Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
  • Highlight what matters most. Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
  • Show the full trace. Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
  • Sanitize rendered content. Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.
Raw lines 15 to 27
28## Feedback Collection
30Annotate at the trace level. The reviewer judges the whole trace, not individual spans.
32- Binary Pass/Fail buttons as the primary action.
33- Free-text notes field for the reviewer to describe what went wrong (or right).
34- Defer button for uncertain cases.
35- Auto-save on every action.
37Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes. But don't add these in the initial build.

Feedback Collection

Annotate at the trace level. The reviewer judges the whole trace, not individual spans.

  • Binary Pass/Fail buttons as the primary action.
  • Free-text notes field for the reviewer to describe what went wrong (or right).
  • Defer button for uncertain cases.
  • Auto-save on every action.

Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes. But don't add these in the initial build.

Raw lines 28 to 38
Names a skill error-discovery :37
39## Navigation and Status
41- Next/Previous buttons and keyboard arrow keys.
42- Trace counter showing position and progress ("12 of 87 remaining").
43- Jump to specific trace by ID.
44- Counts of labeled vs unlabeled traces.

Navigation and Status

  • Next/Previous buttons and keyboard arrow keys.
  • Trace counter showing position and progress ("12 of 87 remaining").
  • Jump to specific trace by ID.
  • Counts of labeled vs unlabeled traces.
Raw lines 39 to 45
46## Keyboard Shortcuts
48```
49Arrow keys = Navigate traces
501 = Pass 2 = Fail
51D = Defer U = Undo last action
52Cmd+S = Save Cmd+Enter = Save and next
53```

Keyboard Shortcuts

Arrow keys = Navigate traces
1 = Pass              2 = Fail
D = Defer             U = Undo last action
Cmd+S = Save          Cmd+Enter = Save and next
Raw lines 46 to 54
55## Selecting Traces to Load
57Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.

Selecting Traces to Load

Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.

Raw lines 55 to 58
59## Additional Features
61**Reference panel:** Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.
63**Filtering:** Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).
65**Clustering:** Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.

Additional Features

Reference panel: Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.

Filtering: Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).

Clustering: Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.

Raw lines 59 to 66
67## Design Checklist
69- [ ] Same layout, controls, and terminology on every trace
70- [ ] Pass and Fail buttons are visually distinct (color, size)
71- [ ] Keyboard shortcuts work for all primary actions
72- [ ] Full trace accessible even when sections are collapsed
73- [ ] Labels persist automatically without explicit save
74- [ ] Trace-level annotation (not span-level) as the default
75- [ ] All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)

Design Checklist

  • [ ] Same layout, controls, and terminology on every trace
  • [ ] Pass and Fail buttons are visually distinct (color, size)
  • [ ] Keyboard shortcuts work for all primary actions
  • [ ] Full trace accessible even when sections are collapsed
  • [ ] Labels persist automatically without explicit save
  • [ ] Trace-level annotation (not span-level) as the default
  • [ ] All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)
Raw lines 67 to 76
77## Testing
79After building the interface, verify it with Playwright.
81**Visual review:** Take screenshots of the interface with representative trace data loaded. Review each screenshot for:
82- Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
83- Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
84- Aesthetics: does the interface look professional and clean? Would a domain expert use this?
85- Responsiveness: does the layout hold at different window sizes?
87**Functional test:** Write a Playwright script that performs a full annotation workflow:
881. Load the app and verify traces are displayed
892. Click Pass on a trace, verify the label is saved
903. Click Fail on a trace, add a note, verify both are saved
914. Click Defer, verify it is recorded
925. Navigate forward and backward with buttons and keyboard shortcuts
936. Verify the trace counter updates correctly
947. Verify auto-save by reloading the page and checking labels persist
958. Expand collapsed sections (system prompts, tool calls) and verify content is accessible
969. Test that all keyboard shortcuts trigger the correct actions

Testing

After building the interface, verify it with Playwright.

Visual review: Take screenshots of the interface with representative trace data loaded. Review each screenshot for:

  • Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
  • Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
  • Aesthetics: does the interface look professional and clean? Would a domain expert use this?
  • Responsiveness: does the layout hold at different window sizes?

Functional test: Write a Playwright script that performs a full annotation workflow:

  1. Load the app and verify traces are displayed
  2. Click Pass on a trace, verify the label is saved
  3. Click Fail on a trace, add a note, verify both are saved
  4. Click Defer, verify it is recorded
  5. Navigate forward and backward with buttons and keyboard shortcuts
  6. Verify the trace counter updates correctly
  7. Verify auto-save by reloading the page and checking labels persist
  8. Expand collapsed sections (system prompts, tool calls) and verify content is accessible
  9. Test that all keyboard shortcuts trigger the correct actions
Raw lines 77 to 96

From ai-evals-course/evals-skills at 80d5f7b, 2026-09-24. Skills read by DeepSeek V4.1 Flash on 2026-11-04. Summary and techniques by Claude Opus 5.5 on 2026-11-06.