Skills › HyperFrames › remotion-to-hyperframes › eval.md

eval.md

95 lines · skills/remotion-to-hyperframes/references/eval.md at a595755

Runs code

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Eval: how to validate a translation end-to-end

3Every translation should be measured. The skill ships a source linter and

4a tiered test corpus; the CLI's `diff` command scores the result. Together

5they gate translation quality.

Eval: how to validate a translation end-to-end

Every translation should be measured. The skill ships a source linter and a tiered test corpus; the CLI's diff command scores the result. Together they gate translation quality.

Raw lines 1 to 6

7## The tools

9| Tool | Input | Output |

10| -------------------------- | --------------------------- | ----------------------------------------------------------------------------- |

11| `scripts/lint_source.py` | Remotion source dir or file | JSON findings + exit code (0 clean, 1 has blockers) |

12| `npx hyperframes diff A B` | original MP4, HF MP4 | per-frame SSIM, worst cell per frame, cuts, `worst.jpg` sheet; exit 1 on fail |

14Run them in this order: **lint → render → diff → (if fail) read the sheet**.

The tools

Tool	Input	Output
`scripts/lint_source.py`	Remotion source dir or file	JSON findings + exit code (0 clean, 1 has blockers)
`npx hyperframes diff A B`	original MP4, HF MP4	per-frame SSIM, worst cell per frame, cuts, `worst.jpg` sheet; exit 1 on fail

Run them in this order: lint → render → diff → (if fail) read the sheet.

Raw lines 7 to 15

Runs code scripts/lint_source.py :11

16## Per-fixture flow

18```bash

19# 1. Lint the source — blockers mean stop

20python3 ../../scripts/lint_source.py ./remotion-src/src/

22# 2. Generate any binary assets (T2+T3 only)

23[ -f setup.sh ] && ./setup.sh

25# 3. Render Remotion baseline

26cd remotion-src && npm install && npm run render

27# -> remotion-src/out/baseline.mp4

29# 4. Render HF translation

30cd .. && node ../../../packages/cli/dist/cli.js render hf-src/ --output hf.mp4

31# -> hf.mp4

33# 5. SSIM diff, gated on the tier's mean threshold only

34mkdir -p diff && node ../../../packages/cli/dist/cli.js diff ./remotion-src/out/baseline.mp4 ./hf.mp4 \

35 --min-mean 0.95 --min-p05 0 --min-cell 0 --out ./diff --json > diff/summary.json

36# -> diff/summary.json, diff/worst.jpg, diff/ssim.log

37```

39Before rendering, `npx hyperframes diff <baseline.mp4> hf-src --at 0.5,2,4.1`

40scores the project at chosen times with no render.

Per-fixture flow

# 1. Lint the source — blockers mean stop
python3 ../../scripts/lint_source.py ./remotion-src/src/

# 2. Generate any binary assets (T2+T3 only)
[ -f setup.sh ] && ./setup.sh

# 3. Render Remotion baseline
cd remotion-src && npm install && npm run render
# -> remotion-src/out/baseline.mp4

# 4. Render HF translation
cd .. && node ../../../packages/cli/dist/cli.js render hf-src/ --output hf.mp4
# -> hf.mp4

# 5. SSIM diff, gated on the tier's mean threshold only
mkdir -p diff && node ../../../packages/cli/dist/cli.js diff ./remotion-src/out/baseline.mp4 ./hf.mp4 \
  --min-mean 0.95 --min-p05 0 --min-cell 0 --out ./diff --json > diff/summary.json
# -> diff/summary.json, diff/worst.jpg, diff/ssim.log

Before rendering, npx hyperframes diff <baseline.mp4> hf-src --at 0.5,2,4.1 scores the project at chosen times with no render.

Raw lines 16 to 41

Runs code python3 ../../scripts/lint_source.py :20

42## Reading `diff/summary.json`

44```json

45{

46 "pass": true,

47 "ssim": { "frames": 90, "mean": 0.974, "p05": 0.972, "min": 0.971, "cell_p05": 0.93 },

48 "worst": [

49 { "t": 1.2, "frame": 36, "ssim": 0.971, "cell": { "roi": [480, 270, 480, 270], "ssim": 0.88 } }

50 ],

51 "cuts": { "reference": [1.5], "yours": [1.5], "missing": [], "extra": [], "pass": true },

52 "duration": { "reference": 3, "yours": 3, "pass": true },

53 "sheet": "diff/worst.jpg"

54}

55```

57| Field | What it tells you |

58| --------------- | -------------------------------------------------------------------------------- |

59| `ssim.mean` | average SSIM across all frames; the headline number |

60| `ssim.p05` | 5th percentile: most frames sit above it |

61| `ssim.cell_p05` | each frame's worst cell of a 4x4 grid; a wrong element shows here, not in `mean` |

62| `worst` | the lowest-scoring moments, each with the cell (in original pixels) that failed |

63| `cuts` | hard cuts on both sides; a missing or extra one is a sequencing problem |

64| `pass` | every gate held: `--min-mean`, `--min-p05`, `--min-cell`, cuts and duration |

Reading `diff/summary.json`

{
  "pass": true,
  "ssim": { "frames": 90, "mean": 0.974, "p05": 0.972, "min": 0.971, "cell_p05": 0.93 },
  "worst": [
    { "t": 1.2, "frame": 36, "ssim": 0.971, "cell": { "roi": [480, 270, 480, 270], "ssim": 0.88 } }
  ],
  "cuts": { "reference": [1.5], "yours": [1.5], "missing": [], "extra": [], "pass": true },
  "duration": { "reference": 3, "yours": 3, "pass": true },
  "sheet": "diff/worst.jpg"
}

Field	What it tells you
`ssim.mean`	average SSIM across all frames; the headline number
`ssim.p05`	5th percentile: most frames sit above it
`ssim.cell_p05`	each frame's worst cell of a 4x4 grid; a wrong element shows here, not in `mean`
`worst`	the lowest-scoring moments, each with the cell (in original pixels) that failed
`cuts`	hard cuts on both sides; a missing or extra one is a sequencing problem
`pass`	every gate held: `--min-mean`, `--min-p05`, `--min-cell`, cuts and duration

Raw lines 42 to 65

66## Threshold rule of thumb

68Set the threshold ~0.02 below measured `p05`:

70- Real translation regressions drop mean by 0.05+ — caught.

71- Encoder/font drift between CI runs is bounded at ~0.01 — not caught.

73If a calibration run's measured mean is far above your initial threshold

74guess, _don't_ tighten the threshold to fit. Leave headroom — fixtures

75re-rendered on different hardware will drift.

Threshold rule of thumb

Set the threshold ~0.02 below measured p05:

Real translation regressions drop mean by 0.05+ — caught.
Encoder/font drift between CI runs is bounded at ~0.01 — not caught.

If a calibration run's measured mean is far above your initial threshold guess, don't tighten the threshold to fit. Leave headroom — fixtures re-rendered on different hardware will drift.

Raw lines 66 to 76

77## When the diff fails

791. **Look at `diff/worst.jpg` first.** The worst moments side by side, each

80 with its failing cell boxed, show whether the failure is structural

81 (wrong scene durations, missing element) or cosmetic (different font

82 weight, slight timing skew).

832. **Check `diff/ssim.log`.** Per-frame SSIM tells you _which_ frames

84 failed. Cluster of bad frames in the middle of a scene = animation

85 problem; bad frames at scene boundaries = sequencing problem.

863. **Re-read the relevant reference.** [timing.md](timing.md) for

87 spring/easing issues, [sequencing.md](sequencing.md) for scene

88 boundary issues, [media.md](media.md) for asset loading issues.

When the diff fails

**Look at diff/worst.jpg first.** The worst moments side by side, each with its failing cell boxed, show whether the failure is structural (wrong scene durations, missing element) or cosmetic (different font weight, slight timing skew).
**Check diff/ssim.log.** Per-frame SSIM tells you which frames failed. Cluster of bad frames in the middle of a scene = animation problem; bad frames at scene boundaries = sequencing problem.
Re-read the relevant reference. timing.md for spring/easing issues, sequencing.md for scene boundary issues, media.md for asset loading issues.

Raw lines 77 to 89

90## CI integration

92The fixtures are not yet wired into CI (`packages/producer/tests/` runs

93inside Docker; the skill corpus needs the same). PR 7 of the stack adds

94the orchestrator that runs all four tiers and emits an aggregated pass

95report. For now, evaluate by hand per fixture.

CI integration

The fixtures are not yet wired into CI (packages/producer/tests/ runs inside Docker; the skill corpus needs the same). PR 7 of the stack adds the orchestrator that runs all four tiers and emits an aggregated pass report. For now, evaluate by hand per fixture.

Raw lines 90 to 95