Kinetic Music Video

A song goes in. A 1080p60 music video comes out, landscape for YouTube and X or vertical for TikTok, Reels and Shorts: a motion designer's showreel cut to the track. Kinetic type, 2D and 3D graphics and transitions land on the beats and sung syllables. Generated footage of the subject is woven through it where the concept wants a performer. The approved master is the only soundtrack. Preserve the original; an authorized audio edit produces a new, versioned master.

The work splits by what each tool does best:

• Scenario supplies what code can't: a consistent subject with real physics, choreography, hair and lip-sync, in whatever art style each section needs.
• The JS engine supplies what video models can't: exact typography, frame-exact timing to onsets and beats, infographics, compositing around the subject, and endless re-renders at zero generation cost.
• Sub-agents give each section a dedicated designer. A shared style bible and a verification loop keep them coherent.

Connection and the core generation loop: see the scenario skill in this repo. For a song-to-video cut without the engine, see scenario-seedance-music-video. If a sibling skill named here is missing from your available skills, ask the user to install it (npx skills add scenario-labs/skills --skill <name>); unattended, proceed from tool schemas and flag the gap.

Read references/lessons.md before starting. Every item in it cost real time or money once. references/motion_library.md is the design menu.

Before you start (ask once, then run)

• Workspace: the Scenario team and project (teams_list; ask if there is more than one).
• Subject:
  • Who or what performs: a character from reference images, a mascot, a product, the artist's own likeness (their own photos and consent only), or nobody (a pure motion-graphics or lyric video).
  • Footage share: performer-led, a mix (often about 60/40), or graphics only.
• Look:
  • Ask for references: images, videos or links for the mood, the motion and the color. Vague color asks without references drift to safe palettes.
  • Decide the 2-4 looks and which sections own which.
  • Ask what must never appear.
  • Defaults: generated footage shown clean (no filters on it), graphics layered on and around it, a fast and upbeat showreel energy.
• Brand: any logo, wordmark or credit. Use official files in their official colors, only where the director wants them.
• Budget: lean (6-10 video generations), moderate (12-20) or big. Price each paid step with dry_run (one run per distinct payload, then total) before spending. Relative anchors at 720p: a 6 s clip costs about twenty style frames, a 10 s clip about thirty-five, and a full mixed video about five hundred. A 1080p clip measured about 2.5 times its 720p price, so settle the footage resolution before quoting.
• Format: one canvas per project, asked once: portrait 9:16 for TikTok, Reels or Shorts, landscape 16:9 for YouTube or X (unattended with no destination: landscape). It is CANVAS in engine/timeline.js, and style frames and footage are generated at that ratio, never cropped from the other. On portrait, check the destination's current length cap and plan the teaser as the platform cut when the song runs longer, or when you cannot check. Both ratios only on request: the code re-renders free with --canvas and most shots are windowed, but each shot that must fill the second frame (hook, lip-sync close-ups, finale) is paid again (references/pipeline.md §9). X caps standard accounts at 2:20, so plan a teaser.
• Lyrics: use them if supplied (they beat any transcript). Otherwise transcribe and flag the uncertain words at the end.

With no one around to answer, choose sensible defaults for the creative choices and write them into TREATMENT.md. Do not default the money: with no stated budget or no explicit team and project, stop before the first paid step and say what is missing.

Workflow

1. Scaffold and analyze the song

Run bash <skill>/scripts/scaffold.sh <song> inside a new project folder, then follow references/pipeline.md §2:

• audio_analysis.py writes stems and engine/data/audio.json: beats, downbeats, kick/snare/vocal onsets, and 60 fps envelopes. Measure the tempo; labels lie (a "130 BPM" song was 98 with double-time drums).
• lyrics_align.py transcribe, then hand-correct analysis/lyrics.txt, then lyrics_align.py align writes engine/data/lyrics.json with word onsets.
• Check timing by eye with plot_lyrics.py. Audition when playback is available; a spectrogram locates onsets but cannot prove pronunciation or musical continuity.

1a. Edit the soundtrack when requested

For pronunciation repairs, lyric changes, section replacements or structural cuts, read references/audio_editing.md. Use scenario-ace-step for its Cover/Repaint contracts. Keep the current cut and master until a candidate is selected; preserve audio outside the agreed window and rebuild affected timing data after applying it. A picture-only revision reuses the approved soundtrack.

2. Research (parallel, background)

Launch two sub-agents with web access while you work on other things:

• Culture: for the song's subject, find recognizable events, memes and visual tropes, each with a verified fact, its iconic visual form, an insert idea and a short on-screen string.
• Craft: study the user's motion references (download them, contact-sheet them, read the frames), plus genre direction techniques. Extract timing numbers, color hex values and implementable JS moves.

Save the condensed results as analysis/research_*.md. Scene agents read them alongside references/motion_library.md.

3. Concept, style bible and treatment

• Give the video a spine. The strongest concepts make the edit itself carry the song's meaning: a device that progresses through the whole video, for example a counter, clock or countdown that moves with the song while the cut rate or the looks escalate with it. The spine usually lives in the HUD counters.
• STYLE.md (the law):
  • a one-line concept and the spine
  • the footage rules (clean plate and the back/mid/front layers, or another policy)
  • the looks, each with ground, ink, accent, rendering rule and matching footage style
  • the palette with roles
  • 4-5 type roles with named fonts
  • lyric display rules
  • a motion grammar mapping beats to effects, with the cut density per section
  • the subject (identity, design details usable as graphic motifs)
  • brand rules, references and bans
• TREATMENT.md:
  • song facts and the energy map
  • the arc
  • the hook for the first 3 seconds (the strongest idea in the video, and it must work as a muted autoplay)
  • one row per section: time range on downbeats, lyrics, look, must-have ideas
  • a footage map (filled in as clips land)
• PLAN.md: a one-page version for the director to approve before you spend on footage (unattended, this is the stop point if the budget is unstated).

4. Subject, style frames and footage on Scenario

Skip this for a graphics-only video, apart from any stills or 3D props. Otherwise follow references/pipeline.md §3-5:

• Subject sheet: a turnaround, expressions and callouts from the user's references. Look at it, then crop identity references from it.
• Style frames: one still per shot at the canvas ratio (an image model that takes reference images), in its section's look. Compose with negative space for type (on portrait, the face and the type space inside the safe band), on a set that composites (black void with rim light, flat color cyc, or white high-key). No text. In a lip-sync frame nothing covers the mouth (no mic, hand or prop): a covered mouth fails the sync check and is paid again. Contact-sheet them and get the look approved cheaply (unattended: proceed only inside a stated budget).
• Footage routes:
  • Image-to-video from the frame for actions.
  • A video model with reference images and reference audio, given a synthesized beat track for dances that lock to the song, then beat-warp.
  • Audio-to-video from a frame plus a vocal-stem slice for lip-sync. The alternative is reference-audio video plus a lip-sync correction model.
  • Code instead of generation for networks, charts, UI and particles.
• Never send the song to a video model. Turn the model's native audio off when its schema offers it (the render takes sound only from the master either way), and launch in waves within the team's concurrency ceiling (jobs_wait re-called with pending_job_ids, never job_get). Relaunch only a launch refused with 429 parallel-custom-jobs: it created no job, and dropping it leaves a hole in the cut.
• Prep every clip with tools/prep_clip.sh: frames, a subject matte (Apple Vision, with Scenario background removal as the fallback; see references/pipeline.md §5) and tracking (MediaPipe pose plus the matte bbox). Inspect the mattes. Write each clip's content, key moments, slot and matte quality into the footage map.

5. Engine and scene agents

• Smoke-test first:
  • Set PAL in core.js, and in engine/timeline.js set CANVAS, the sections on downbeats, HUD (title, mark, section codes, spine counters) and POST (neutral by default).
  • Write a tiny _smoke.js: plate → type behind the subject → matte → tracked label.
  • Render one still and a 5 s clip, then confirm audio and flashes line up with av_sync_check.py.
• Brief the agents. Fill in AGENTS_BRIEF.md from the template, then launch one background agent per section in a single message. Each gets its file, time range, lyrics, must-have ideas and clips, and reads AGENTS_BRIEF, STYLE, TREATMENT, ENGINE_API and the research first. Log the agent ids. For a teaser or anything under about 30 s, the lead may write the scenes itself and skip the research agents, PLAN.md and the per-section agents; keep the smoke test.
• Deliver clips as they land. Message the agent that needs each one ("clip ready: content, key moments, slot").
• Review every report. Contact-sheet the best stills and Read them. Fix engine-level problems yourself (they affect everyone) and tell the agents what changed.

6. Assemble, watch, iterate

• Render the full cut (3 workers, about 5 min for 2:00 on an M-series Mac). Wait for it in the foreground (or poll until it prints wrote) and carry on to verification and delivery: never end the run while a render or encode is still going, because a headless session that stops kills it.
• Verify:
  • av_sync_check.py: expect 0.0 ms audio and 0 to +1 frame visual. Below a correlation of about 0.15 the visual lag means nothing (type leading by E.LEAD reads as up to -3 frames): check onset stills instead.
  • freezedetect: no frozen footage.
  • Contact sheets at 0.5 s steps, 20 s per sheet.
  • Every section boundary, frame by frame.
  • On portrait, still.mjs --safe sheets, 20 s each: no word, logo or face outside the outlined band.
• Global look pass: a still sheet across all sections, then tune POST, the palette and the HUD theme.
• Revisions: map each director note (they arrive as timestamps) to its section and send it back to the owning agent via SendMessage. Resuming keeps its context. Fix a weak shot by replacing it with a new generation in a different look, or with code, never by adding effects. Offer alternatives as timeline variants (render with --variant b) instead of guessing.

7. Deliver

• Videos: a master, an upload file (landscape: the master for YouTube, an X-ready 2-pass 21 Mb/s file under 512 MB for X; portrait: the vertical upload in references/pipeline.md §8), a teaser (verse plus first chorus), plus one file per variant, and one set per canvas, suffixed _16x9 and _9x16, when both were asked for. Use distinct filenames, because macOS is case-insensitive.
• Stills: full-res PNGs rendered by the engine, never grabbed from the compressed video.
• Style sheet / making-of: publish it as an artifact or doc: palette, type, subject sheet, source-vs-final footage pairs, a frame index, the pipeline and iteration counts.
• Report: tell the user what was verified, which lyrics you guessed, and the CU spent, summed from this run's successful jobs_wait rows (cuCost, logged in analysis/jobs.md; a failed job is refunded but its row keeps a cuCost); usage is project-wide and includes other work.

Quality bar

• Every word shows on time. Each sung word is visible at its onset (as a hero or in the subtitle track), and hero words peak on the onset frame.
• Integration, not decoration. Every footage moment has a graphic that reacts to it (behind via the matte, tracked to it, timed to its move, or continuing its shapes). The footage pixels are untouched.
• No static frames. Footage always plays live, and something always moves.
• One focal point per frame. Type frames the subject and never covers the face. Behind-the-subject type still reads. On portrait, every word, logo and face sits inside the safe band.
• Section variance. Mix the looks, 2D and 3D, poster slams and quiet subtitles. Never use the same type treatment on two consecutive lines. Cut density climbs with the song's energy, and showcase moments (logo, end card) get time.
• Photosensitivity. At most 3 full-frame luminance flips per second.
• Watch it yourself before calling it done, then take a second look at the first 3 seconds.

Files in this skill

• Setup and analysis: scaffold.sh, audio_analysis.py, lyrics_align.py, plot_lyrics.py
• Approved-master stem lookup for transcription, plots and vocal slices: audio_paths.py
• Footage generation helpers: synth_beat.py, beatwarp.py
• Footage prep: prep_clip.sh, prep_video.py, matte.swift, fixmatte.py, matte_keyed.py, matte_fallback.py, track.py
• Sync checks: lipsync_check.py, dance_sync_check.py, av_sync_check.py
• Rendering and review: render.mjs, still.mjs, serve.mjs, sheet.py
• Engine, copied into the project by scaffold.sh: index.html, main.js, canvas.js, core.js, plate.js, roto.js, typekit.js, hud.js, plus timeline.template.js and scene.template.js
• references/pipeline.md: every command and Scenario call, step by step
• references/engine_api.md: the scene contract and the full E API (canvas and safe band, plates, mattes, tracking, post, HUD, variants)
• references/motion_library.md: looks, type system, a 40-move technique library, integration ideas, transitions, timing, easing and springs, sync and cut density
• references/agent_brief_template.md: the brief each scene agent reads
• references/lessons.md: production gotchas and direction defaults that held