Audio Reactive Video Composition

Audio-reactive video composition

Use this skill to turn an approved audio source into an auditable cue map and then into bounded visual behavior. The central contract is:

$$ \text{source audio} \rightarrow \text{measured features} \rightarrow \text{confidence-bearing cues} \rightarrow \text{reviewed anchors} \rightarrow \text{deterministic visual timeline} $$

Do not equate a detector output with editorial meaning. An onset is not automatically a beat, cut, drop, or lyric accent; an acoustic cluster is not automatically a verse or chorus.

Evidence stance

• Documented fact: behavior stated by official analysis libraries, standards, or cited research.
• Production heuristic: a practical mapping that must be tested on the track and audience.
• Empirical observation: a measured result from the supplied audio, analyzer run, render, or playback.

Analysis algorithms, defaults, and model behavior are volatile. Facts were verified 2026-07-12. Pin the decoder, library, model, parameters, and random seeds used for each production.

Scope

This skill owns source custody, analysis policy, confidence interpretation, rhythmic/non-rhythmic routing, cue promotion, feature-to-visual mappings, rational frame alignment, accessibility, and render QA.

It does not own music-video narrative or artist branding, music generation, mastering, source separation, transcription, lyric writing, or a HyperFrames/Remotion/FFmpeg-specific implementation.

Freeze the audio contract

Record before analysis:

• source path/URI, SHA-256, acquisition source, rights basis, and restrictions;
• selected stream, codec, native sample rate, channels/layout, start timestamp, and duration;
• decoder/resampler and versions;
• canonical PCM format, channel/downmix policy, analysis sample rate, window/hop sizes, centering, and padding;
• analyzer/library/model versions, priors, thresholds, and seeds;
• trim offsets and source time origin;
• master output frame rate as a rational number;
• transcript/lyrics source, language, timing provenance, review status, and separate rights basis.

Use sample indices as the primary analysis clock where possible. Derived seconds should not replace exact source positions.

Interpret features conservatively

Documented facts:

• librosa.beat.beat_track estimates tempo from onset strength and selects beat positions consistent with it; it does not return calibrated beat confidence.
• librosa.onset.onset_detect peak-picks an onset-strength envelope. Onsets estimate event attacks, not semantic accents.
• Predominant local pulse can model changing tempo, but remains an estimate.
• Essentia confidence values are algorithm-specific; they are not universal probabilities, and some routes return an unusable zero confidence.
• Beat evaluation commonly accounts for half/double-tempo metrical ambiguity and uses tolerances rather than exact equality.
• Structural boundaries and structural labels are separate tasks. Cluster labels such as A/B/C do not establish verse/chorus meaning.
• RMS is an energy measure. EBU R128 loudness uses defined weighting and windows. Neither is a direct emotion score.
• Spectral centroid, bandwidth, and contrast describe spectrum distribution; they do not mean happiness, tension, or quality.
• Silence detection is relative to a declared reference and threshold.
• pYIN voicing means pitch periodicity, not proof that a person is speaking or singing.

Keep raw candidates distinct from human-promoted anchors.

Cue-map contract

Each cue should include:

{
  "id": "accent-014",
  "type": "onset",
  "time_samples": 417312,
  "time_seconds": 9.46399,
  "value": 0.82,
  "units": "normalized-onset-strength",
  "analyzer": "record exact library and algorithm",
  "confidence": null,
  "confidence_semantics": "not supplied by analyzer",
  "profile_hash": "sha256:...",
  "status": "promoted-anchor",
  "editorial_role": "single visual accent",
  "reason": "reviewed strong transient before section change"
}

For interval features, include exact end samples/times. Record rejected candidates rather than deleting evidence. Keep acoustic section names neutral until lyrics, metadata, or human review supports functional labels.

Choose a routing mode

Rhythmic

Use when the selected tracker has useful evidence, local tempo is stable enough, and beat/onset results agree. Periodic motion may follow beats; selected strong accents or reviewed section changes may motivate cuts.

Mixed confidence

Use reliable rhythmic windows locally. Elsewhere shift to onsets, transcript phrases, energy, silence, or manual anchors. Never extrapolate a grid through a failed interval.

Non-rhythmic

Use reviewed speech/lyric phrases, silence, energy contour, spectral change, and manually confirmed macro anchors. This suits rubato, ambient work, spoken word, sparse recordings, and free improvisation.

Map features to visual behavior

Production heuristics:

• Give each feature a limited visual responsibility.
• Normalize continuous values with robust per-track or per-section statistics, not one extreme maximum.
• Smooth noise and use hysteresis before switching visual states.
• Clamp every parameter and define neutral behavior for missing/invalid values.
• Use macro section anchors, medium phrase anchors, and sparse accents rather than cutting on every event.
• Reserve low-energy or silent spans for holds, reading, resets, and visual breathing room.
• Protect lyric lines and important vocal phrases from competing cuts and overlays.
• Preserve one cue map across aspect ratios; recompose layout, not timing.

Avoid quantitative-looking mappings that imply false measurement. If energy maps to scale, define the exact bounded range and do not call it emotion.

Deterministic frame timing

Keep exact rates such as $30000/1001$ rational. For each cue, define a rounding policy. A conservative visual-response policy assigns the cue to the first output frame whose presentation time is at or after the audio event:

$$ f = \left\lceil t_{event} \cdot \frac{fps_{num}}{fps_{den}} \right\rceil $$

Record both the original event time and mapped frame. Verify actual frame presentation timestamps after encoding because an encoder may duplicate or drop frames to satisfy constant-frame-rate output.

The visual state at frame $f$ must derive from cue data and frame time, not wall-clock playback. Seed procedural mappings and freeze analyzer output before distributed rendering.