Music video

A music video where the words matter. The picture moves on the beat, the words light up on the voice, and every scene does something different with the lyric. It's one window.renderAt(t) page, so the make-video contract applies; its music reference covers helios analyze, the beat map and the sync rules. This skill adds what a lyric video needs on top of that.

karaoke.html and lyrics.example.json are a tested reference for the mechanics: reading word and syllable timings, lighting each glyph as it's sung, looking lyrics up by content, and making the scene react to a word. Reuse the mechanics; design the look and scenes from the treatment.

Ground rules

• The motion syncs to the beat; the word syncs to the voice. A word lights exactly at its sung start and is fully lit by its end. A line may appear dim up to about 0.4 s early. The highlight never runs ahead of the voice.
• Never fake the timing. A wipe whose speed is computed from the number of letters, or a beat grid from a guessed BPM, drifts off the song within seconds. Get real word times (step 1) and a real beat map (helios analyze).
• Look lyrics up by content, never by hard-coded time. lyric('through the night').words[0].start. Re-timing the lyrics then re-times the whole video.
• Lyrics are usually copyrighted. Use the user's own song and lyrics. Don't commit lyric text to public examples.

1. Get the timings

Tier	When	How	Accuracy
0	The user has an .srt/.lrc with line times, or nothing but the text	Line times from the file (or from tapping along); split each line's span into words weighted by syllable count. Mark the file "approx": true.	Lines ±100 ms, words rough
1	Python is available	Separate the vocal (demucs --two-stems=vocals), then align the known lyrics to it with stable-ts or WhisperX	Words ±50–100 ms
2	The words must land on consonants	Forced alignment with torchaudio's MMS_FA on the vocal stem, plus onset refinement and a Whisper cross-check	Word starts ±30–50 ms

Whatever the tier, decode the song to WAV once and use that same file for analysis, alignment and --audio. MP3 decoders disagree by 20–50 ms because of encoder delay, which is enough to make every word look late.

Convert the result to one lyrics.json that both the page and helios verify --cues read:

{ "approx": false,
  "lines": [{ "text": "and carried it through the night", "start": 3.6, "end": 6.6, "first": 5, "last": 10 }],
  "words": [{ "w": "carried", "start": 3.9, "end": 4.5, "syl": [[3.9, 4.2], [4.2, 4.5]] }] }

See references/lyrics.md for each tier's commands, the conversion script, and how to check timings without ears.

2. Treatment

Write TREATMENT.md before any code. Start it with a random seed, so the concept comes from the song and the seed rather than from whatever music video you'd make by default. If an apply-randomness skill is available, use it; otherwise run LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 256; echo. Write three different concepts and three motifs, then let verifiable patterns in the seed (a digit run, a repeat, a letter cluster) choose between them, and between the palette and the lyric treatment for each section. Keep the seed and a one-line pattern → choice map in the treatment. The song and lyrics are fixed; the seed only decides what's open.

The treatment, on one page:

• Concept in one paragraph, and one motif that recurs and changes role across the song.
• Palette with one accent owned by one moment or one element. Type voices: two or three, each with a job (the sung line, the typed aside, the title).
• An anti-slop list: what this video will not contain (glowing brains, code rain, particle nebulae, neon for its own sake, a stock "audio visualizer" bar row).
• Visual ideas, not illustrations. Puns, transformations and metaphors for each section. Never draw literally what each line says.
• Chorus plan: the chorus returns to the same set or template and escalates each time.
• How the lyric lives in each scene: riding a curve, typed as tokens, stamped on a form, engraved, written by a pen, unmasked block by block. Never the same plain subtitle bar all the way through.

3. Storyboard by section

Run helios analyze song.wav first, then storyboard against its sections and downbeats:

| Section  | Time        | Lyric (first words)     | Shot / what happens                         | Lyric treatment       | Out             |
| intro    | 0.0–9.4     | —                       | the motif alone, slow push                  | title writes on       | match cut       |
| verse 1  | 9.4–28.6    | "we built a…"           | city of paper windows lights one by one     | words stamped on sills | windows → stars |
| chorus 1 | 28.6–47.8   | "every window…"         | sky set; motif becomes a constellation      | words as star paths   | iris on the moon |

• Something happens in every shot. A camera move in every shot. Calm shots in low-energy passages.
• Intensity follows the song: roughly 40 % in the intro, 70 % in verses, 100 % in choruses.
• Rotate shot types on the beat; never the same pose or framing on adjacent cuts.
• The Out column names a motivated transition (a zoom through an eye, a shape that becomes the next scene, a cut on action). Keep brush wipes or plain cuts for section breaks.
• When the beat map is noisy. On synthesized or sparse tracks analyze can read bass as kicks, find few snares, give coarse sections and miss crashes. Keep only kicks that sit on a beat, and when sections is coarse, anchor the structure on lyric content (lyric('every window')) instead.
• No fake singing. A mouth that flaps to the lyric reads as jitter. Characters react to the song; they don't lip-sync it.

See references/treatment.md for the full templates, scene reuse with parameters, and a brief for splitting scenes across several agents.

4. Build

Derive the edit from the data, not from typed-in seconds:

// Cut on the last beat at or before a line's first word.
const cut = (phrase, nth = 0) => {
  const s = lyric(phrase, nth).words[0].start;
  return B.beats[lastAt(B.beats, s + 0.02)];
};
const SCENES = [
  { start: 0,                       draw: intro },
  { start: cut('we built a'),       draw: verse,  params: { n: 1 } },
  { start: cut('every window'),     draw: chorus, params: { n: 1 } },
  { start: cut('every window', 1),  draw: chorus, params: { n: 2 } },   // same scene, escalated
];

A hard cut swallows a held note that is still sounding on that beat. Where the previous word crosses the beat, cut at the word boundary instead, or carry the camera through the beat with no cut:

const seam = (phrase, nth = 0) => {
  const b = cut(phrase, nth), w = lyric(phrase, nth).words[0], prev = L.words[L.words.indexOf(w) - 1];
  return prev && prev.end > b ? Math.min(prev.end, w.start) : b;
};

lastAt and the beat map come from the make-video music reference. One scene function serves every chorus through params, and each pass adds something.

Karaoke recipes are in references/lyrics.md: the per-glyph wipe with kerning-safe positions, the done colour that snaps instead of blending, a small pop on each word start, held notes that stretch, line layout and safe areas, and pen-written lyrics. Every word drawn on a canvas must be declared with window.heliosDrawnText?.add(word).

5. Check

npx -y @helios-project/cli@latest verify video.html --duration 156 --cues lyrics.json   # every word on screen at its time
npx -y @helios-project/cli@latest sheet video.html --strip 28.4:29.2 --fps 30           # every frame into the first chorus
npx -y @helios-project/cli@latest sheet video.html --at 9.4,28.6,47.8,67.0 --cols 4      # each section's first frame

• At every section start, is the new idea clear? Does each chorus escalate?
• In strips around a few words, does the highlight start on the frame the word starts, never before?
• Are lyrics inside title-safe margins (96 px or more at 1080p) and clear of anything busy behind them? Is contrast high enough to read?
• Did a word get swallowed by a transition? verify --cues names it.
• Is the tempo right? If cuts feel off by a steady fraction, the analysis may have locked to half, double or 2/3 time: re-run analyze with --tempo-range.

Render with the WAV:

npx -y @helios-project/cli@latest render video.html -o video.mp4 --duration 156 --audio song.wav --preset medium
npx -y @helios-project/cli@latest check video.mp4

Credits

The alignment pipeline, karaoke rules and lyric-anchored timeline are adapted from pdoom-video by Giacomo Magnanini (MIT; its song and lyrics are not covered and nothing from them is used here). The section storyboard with a transition column draws on ideas from PDoomVideo by John Heibel. See THIRD_PARTY.md.