Skill Tracer
Skills › HyperFrames › media-use › tts-to-captions.md

tts-to-captions.md

35 lines · skills/media-use/audio/references/tts-to-captions.md at a595755

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# TTS → Captions
2
3When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:
4

TTS → Captions

When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:

Raw lines 1 to 4
5## Path A — HeyGen (single call, no Whisper)
6
7HeyGen returns word timestamps in the same response as the audio. Use the
8bundled REST helper (the `hyperframes tts` command is Kokoro-only):
9
10```bash
11node skills/media-use/audio/scripts/heygen-tts.mjs \
12 script.txt --output narration.wav --words narration.words.json
13```
15`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.

Path A — HeyGen (single call, no Whisper)

HeyGen returns word timestamps in the same response as the audio. Use the bundled REST helper (the hyperframes tts command is Kokoro-only):

node skills/media-use/audio/scripts/heygen-tts.mjs \
  script.txt --output narration.wav --words narration.words.json

narration.words.json is already in the [{ id, text, start, end }] shape the captions pipeline consumes — no separate transcribe pass.

Raw lines 5 to 16
Runs code scripts/heygen-tts.mjs :11
17## Path B — Gemini / ElevenLabs / Kokoro (TTS → transcription)
19These adapters supply audio without word data. The shared audio engine runs
20transcription automatically when timings are absent. For Gemini, use the
21request in [Text to speech](tts.md#gemini-narration), then consume
22`audio_meta.json` → `voices[].words`.
24For a standalone local Kokoro generation, generate the audio, then transcribe:
26```bash
27npx hyperframes tts script.txt --voice af_heart --output narration.wav
28npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
29```
31Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
33For Gemini, verify that transcription preserved the script, especially names,
34numbers, and delivery pauses. If `words` is empty, resolve the transcription
35failure before captioning. Generate and align again after changing the read.

Path B — Gemini / ElevenLabs / Kokoro (TTS → transcription)

These adapters supply audio without word data. The shared audio engine runs transcription automatically when timings are absent. For Gemini, use the request in Text to speech, then consume audio_meta.json → voices[].words.

For a standalone local Kokoro generation, generate the audio, then transcribe:

npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav --model small.en   # voice af_heart is American English

Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match --model to the voice's language (use small.en for a/b prefixes, small --language <code> otherwise). Then consume transcript.json via the caption references in captions/.

For Gemini, verify that transcription preserved the script, especially names, numbers, and delivery pauses. If words is empty, resolve the transcription failure before captioning. Generate and align again after changing the read.

Raw lines 17 to 35