@vosjs/cli 0.73.0 → 0.75.0

package/skills/vos-score/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: vos-score
+description: Compose an original score (background music, BGM, a soundtrack, a beat) for a vos program, written as code. The music is locked to the program's own beat grid (BPM, bars, beat 1 at t = 0) and arranged section by section to what is on screen (a build, stop-time hits on the frames where poses snap, fills under transitions, the biggest moment, an ending that lands), synthesized to a normalized MP3, verified by numbers because an agent cannot listen (a tempo and phase fit against the grid, per-section loudness against the intended shape), and put on the program's doc.json as its track. Use when asked for music, a soundtrack, a score or BGM for a vos, when a catalog track does not fit the picture or does not land on its beats, or when a film dances, cuts or animates on a beat grid.
+license: MIT
+---
+
+# Compose a score for a vos program
+
+A catalog track (`GET https://vos.so/api/music`: CC0, a slug, a mood, a
+length) carries no tempo and no downbeats, and you cannot listen to it. Under
+an ambient picture that is fine. Under a picture that moves on a beat, it is
+wrong in ways you will not hear: the kick lands between the poses, the drop
+arrives two bars early, the loud part sits under the quiet scene.
+
+So write the music instead. The score is a spec (tempo, chords, a riff, a
+hook, sections by bar) that `scripts/compose.py` synthesizes into a WAV at
+exactly the program's length, beat 1 of bar 0 at t = 0. Because the grid is
+the program's own, every hit lands on the frame it was written for, and
+because the arrangement is data, a fix is an edit and a re-render.
+
+The bundled instruments are a funk and new jack swing palette: an 808-style
+kick with a pitch sweep, a layered gated clap, swung 16th hats, a crash,
+snare rolls, noise risers, a two-filter funk pluck bass with a sub under it,
+an FM electric piano, detuned saw brass, orchestra hits, a square/saw lead
+with vibrato, a pad, a room reverb and a bus saturation. Swing is a number
+(0 is straight), kicks, claps and hats are per-section patterns, so the same
+kit covers straight funk, four-on-the-floor dance and hip hop pocket. A
+genre it cannot play (strings, guitar, a choir) is a new instrument written
+beside the others in the same file, never a sample.
+
+## Setup
+
+```bash
+python3 -m venv .score-venv && .score-venv/bin/pip install numpy scipy
+PY=.score-venv/bin/python
+SKILL=<the directory holding this SKILL.md>
+ffmpeg -version | head -1      # with libmp3lame
+npm i -D @vosjs/cli            # render the preview, push
+```
+
+Python 3.9 or newer. Work in the program's directory; the scripts take
+their inputs as paths.
+
+## The procedure
+
+### 1. Read the grid
+
+The music's grid is the program's grid. Find it in the program:
+
+- a tempo constant in a function string (`const P = 60 / 124`, `BAR = P * 4`)
+  or in `data` (`bpm`, `beat`);
+- the sections: a plan table by bar (`[startBar, endBar, move, ...]`), the
+  scenes from `tl.addLabel('<scene>', t)`, or the times the picture changes
+  (render stills with `vos still config.json s.png --time <t>` and look);
+- the length: `config.duration`, which is the score's length to the sample
+  (a program that ends mid-bar sets the spec's `length` to it).
+
+Beat 1 of bar 0 is t = 0. If the program has scenes but no beat (a product
+reveal timed by eye), pick the BPM whose bar lines fall nearest its scene
+times, and, when you own the program, retime its labels onto those bars: a
+picture and a score on one grid is the point. See
+`references/arranging.md`.
+
+### 2. Write the section map
+
+Before any notes, write down what the music does under each section, in the
+spec's `sections` with a `picture` line each (the composer ignores it; it is
+for you and the next reader):
+
+- where the energy rises, and where it drops back to breathe;
+- where it STOPS: stop-time hits on the frames where poses snap or the cut
+  lands, with silence between;
+- the fills under transitions (a spin, a wipe, a title): a roll and a riser
+  that carry into the next downbeat, never a hole;
+- the biggest moment, ONE section, louder than everything else;
+- the ending: a pickup, the last hit on the final pose, and the room ringing
+  out to the last frame.
+
+`references/arranging.md` holds the craft and the measured level targets.
+
+### 3. Pick a genre and a harmony
+
+Fit the picture's era and mood: a 1990 dance film wants new jack swing (an
+808, swung 16ths, brass stabs, a funk bass), a 1977 one disco (straight,
+four on the floor, open hats on the off-beats). Pick a mode and two to four
+chords that vamp (E dorian's Em9 to A13 does a whole groove), a darker pair
+for the breakdown, and a lift for the chorus.
+
+**Everything is original.** Write a new riff, a new hook, new changes. Never
+a recognisable melody, riff, bassline or sample from a real record. A film
+made after a famous record is the most tempting case: the score is in that
+record's genre and era, and quotes nothing of it.
+
+### 4. Compose
+
+Copy `$SKILL/references/new-jack-swing-124.json` to `score.json` (a 124
+BPM new jack swing dance film, 30 bars) and change everything that belongs
+to that piece: the tempo, bars, chords, the riff, the hook, the sections.
+`references/spec.md` is the spec's full contract and every texture's parts.
+Composed as it ships, the example is a reference: it renders the score it
+was written as, sample for sample.
+
+```bash
+$PY $SKILL/scripts/compose.py score.json score.wav
+```
+
+It prints the section map in seconds: hold it against the picture's times.
+A spec error is said in words (an undefined chord, a gap between sections,
+a drop with no riff) before anything renders.
+
+### 5. Verify by numbers
+
+You cannot listen, so measure:
+
+```bash
+$PY $SKILL/scripts/analyze.py score.wav --spec score.json
+```
+
+- **grid**: the kick-band (30 to 150 Hz) onset fit must read the spec's BPM
+  (within 0.05) with beat 1 within 0.02 s of t = 0. A fit at half or double
+  tempo, or a phase off by half a beat, means the kick pattern is not
+  saying the downbeat: put a kick on every beat 1.
+- **order**: a section with more `energy` must be at least 0.5 dB louder
+  than one with less.
+- **biggest**: the highest-energy section must clear every other by 1 dB.
+- **holes**: no bar may fall more than 10 dB under the bar before it. A fill
+  is not a hole.
+- **ring**: the bar after the last hit is quieter than the hit's bar.
+
+Read the per-section table too: it prints each section against the main
+groove, the numbers `references/arranging.md` gives targets for. A failed
+check is fixed with RELATIVE levels: raise or lower that section's `energy`
+and compose again. Never fix a section's level with the master gain, which
+moves every section together. Three faults these checks were built from,
+all in one real first draft: the drop was not louder than the groove, the
+chorus was quieter than the drop, and the spin fills were holes (-29 dB).
+
+### 6. Master
+
+```bash
+$PY $SKILL/scripts/master.py score.wav score.mp3                 # -16 LUFS, 192k
+$PY $SKILL/scripts/analyze.py score.mp3 --spec score.json --lufs -16
+```
+
+`master.py` reaches -16 LUFS (the catalog's loudness) with ONE constant
+gain. Do not run a single-pass `ffmpeg -af loudnorm` instead: it rides the
+gain over a few seconds and flattens the arrangement (measured on a real
+score: the build came out as loud as the groove, and the chorus only 0.5 dB
+over the drop). The MP3 must pass the same checks as the WAV.
+
+### 7. Put it on the program and preview
+
+The score is a track in `doc.json` beside `config.json`, keyed by the MP3's
+file name in the same directory (`vos push` uploads it and keys the hosted
+file):
+
+```json
+{
+  "docSchemaVersion": 6,
+  "audio": [
+    {
+      "id": "score",
+      "key": "score.mp3",
+      "name": "<title>, composed for this film",
+      "start": 0, "in": 0, "out": 58.0645, "duration": 58.0645,
+      "gain": 0.95, "fadeIn": 0, "fadeOut": 2.2
+    }
+  ],
+  "program": {}
+}
+```
+
+`start: 0` and `in: 0` keep beat 1 on frame 0; `out` and `duration` are the
+program's length. No fade in (it would soften the first downbeat); a fade out
+about one bar long under the ring-out. If the program already has a
+`doc.json`, keep its other fields and replace the old track in `audio`.
+
+```bash
+npx vos render config.json preview.mp4 --width 640 --height 360   # mixes doc.json's audio
+ffmpeg -ss <a second inside the biggest moment> -i preview.mp4 -frames:v 1 f.png
+$PY $SKILL/scripts/analyze.py preview.mp4 --bpm <bpm> --bars <bars>
+```
+
+Look at the frame. The preview's analysis confirms the score is in the
+render at the right tempo; its beat 1 reads about 0.05 s late because the
+MP4's AAC track keeps its encoder delay, so the score file, not the preview,
+is the grid check.
+
+### 8. Credit and push
+
+Credit it honestly, as music made for this film and by no one else: the
+track's `name` is `<title>, composed for this film`, and a description that
+credits music says `music: <title>, composed for this film`. Never an
+artist's name, never "by" anyone who did not write it.
+
+```bash
+vos push config.json --label "score: <title>" \
+  --note "an original score at <bpm> BPM, <bars> bars, <genre>, written to the sections: <one line>" --wait
+```
+
+In a fetched or tracked directory this versions the same vos (pass
+`--vos <id>` otherwise); a new program takes the usual `--title`, `--folder`
+and `--prompt` (the person's words, as they wrote them, which the watch page
+shows by default: say so). Keep `score.json` beside the program so the next
+round edits the spec, not the MP3: the same spec renders the same WAV,
+sample for sample.
+
+### 9. Hand it over, and ask them to listen
+
+Give the person the preview (and the vos's watch link), and ask them to
+LISTEN before it is promoted or published. Say it plainly: the numbers show
+the score is on the grid and shaped as intended; they cannot say whether it
+is good music. Ears decide. Their words go back into the spec (a section's
+energy, a chord, the hook) and steps 4 to 8 run again.
+
+## Rules
+
+- The grid is the program's. Never fit a picture to a track's tempo when
+  you can write a track at the picture's.
+- Original or nothing: no quoted melody, riff, bassline, chord hook or
+  sample from a real record, and no artist named in the credit.
+- One biggest moment. A score where everything is loud has no drop.
+- Fix shape with section levels, never the master.
+- Normalize with one gain (`master.py`), never a dynamic normalizer.
+- A score is not finished until a person has listened to it.

package/skills/vos-score/references/arranging.md

@@ -0,0 +1,92 @@
+# Arranging to picture
+
+A score for a program is written to the picture the way a choreographer
+counts a dance: by bar. The picture already has its sections; the music's
+job is to make each one feel like what it is.
+
+## The section map
+
+Write it before any notes, one line per section: the bars, what is on
+screen, and what the music does there. The questions it answers:
+
+1. **Where is the downbeat?** Bar 0 at t = 0. Everything else is counted
+   from it. If the picture's sections do not sit on bar lines, they are not
+   on a grid yet: fix that first (step 1 of the skill).
+2. **What does each section DO?** Introduce, groove, stop, push, release,
+   peak, end. Two neighbouring sections that do the same thing should
+   sound different anyway (a new texture, a new chord pair, the hook
+   entering).
+3. **Where does it stop?** A pose that snaps, a cut, a word slamming in:
+   these are stop-time hits. The whole band hits on that 16th and is silent
+   until the next. Read the picture's own timing for them (a pose table
+   keyed every half bar means beats 1 and 3), never guess.
+4. **What carries the transitions?** A spin, a wipe, a scene change gets a
+   fill: a snare roll rising in velocity, a riser, kicks keeping the grid,
+   and ideally one empty beat before the next downbeat so it lands.
+5. **What is the biggest moment?** One section, usually two thirds to
+   three quarters through. It gets the most parts, the hook an octave up,
+   the gang claps, and the most energy.
+6. **How does it end?** A pickup into the final pose, the last hit ON the
+   pose, and the room ringing out under the last frames. A score that is
+   still grooving when the picture stops was not written to it.
+
+## Energy contrast, measured
+
+These are the steps of a real score that was approved, per section, against
+its main groove (the first `groove` section), all bands. Use them as
+targets, not laws: what matters is the order and the gaps.
+
+| Section | Against the groove |
+| --- | --- |
+| intro (a hit and a riser) | about -7 dB |
+| build | -1.5 dB |
+| groove | 0 (the reference) |
+| stop-time | +0.5 to +1 dB (fewer events, each one bigger) |
+| a pumping section | +2.5 to +3 dB |
+| the drop | +4.5 to +5 dB |
+| a darker breakdown after the drop | +1.5 to +2 dB (under the drop, over the groove) |
+| the biggest moment | +6 to +7 dB, and at least +1 dB over the drop |
+| a fill bar | about -1.5 dB; never 10 dB or more under the bar before it |
+| the last hit's bar, then the ring-out | about the pumping section, then 5 dB or more under it |
+
+## The faults the numbers catch
+
+All three were in one real first draft, and each was invisible without
+listening, which an agent cannot do:
+
+- **The drop was not louder than the groove.** The release had nothing to
+  release. The `order` check catches it.
+- **The chorus was quieter than the drop.** The biggest moment was not the
+  biggest. The `biggest` check catches it.
+- **The fills were holes.** The spin bars read -29 dB: the music vanished
+  exactly where the picture moved most. The `holes` check catches it.
+
+The fix for all three was the same: relative levels per section (`energy`),
+never the master gain. And the shape must survive mastering: a single-pass
+dynamic loudness normalizer moved that same score's sections by up to 3.5 dB
+against each other (the build rose to the groove, the chorus fell to 0.5 dB
+over the drop), which is why `master.py` applies one constant gain.
+
+## Writing the riff and the hook
+
+- **The riff** is one bar of intervals over the chord's root, so it follows
+  the changes: the root on 1, the octave and the seventh, a chromatic
+  passing note into beat 3, a run up into the next bar. Leave a 16th or two
+  of air; a bass that never rests has no groove.
+- **The hook** is two one-bar phrases, a call and an answer, in the mode's
+  pentatonic, with long notes on chord tones and short pickups between. It
+  enters with the drop and goes up an octave, with a harmony, in the
+  biggest moment, so the peak is the same idea, bigger.
+- **Originality.** Hold what you wrote against every famous song in the
+  genre you know; if it is close to one of them, change the rhythm and the
+  contour until it is not. Never write the riff, hook or bassline of the
+  record a film is made after, and never a sample.
+
+## When the picture has no beat
+
+Pick the BPM whose bar lines fall nearest the scene times (a scene change
+every 3.9 s suggests 124 BPM, two bars each). Then either retime the
+program's labels onto those bars, which is better when you own it, or
+write fills across the gaps and accept that a change lands up to a 16th
+off the bar. Never stretch the score to the picture: a tempo change you
+cannot hear is still one you cannot verify.

package/skills/vos-score/references/new-jack-swing-124.json

@@ -0,0 +1,38 @@
+{
+  "title": "A 124 BPM new jack swing dance film",
+  "bpm": 124,
+  "bars": 30,
+  "swing": 0.24,
+  "seed": 1990,
+  "harmony": "E dorian: Em9 and A13 vamps, a descending stop-time line (Em9 D C B7), Am9 to B7 for the dark section, C D Em for the chorus",
+  "chords": {
+    "Em9": { "root": "E1", "voicing": ["G3", "B3", "D4", "F#4"] },
+    "A13": { "root": "A1", "voicing": ["G3", "C#4", "F#4"] },
+    "Cmaj7": { "root": "C2", "voicing": ["G3", "B3", "E4"] },
+    "D6": { "root": "D2", "voicing": ["A3", "D4", "F#4", "B4"] },
+    "Am9": { "root": "A1", "voicing": ["G3", "C4", "E4", "B4"] },
+    "B7": { "root": "B1", "voicing": ["F#3", "B3", "D#4", "A4"] },
+    "Dmaj": { "root": "D2", "voicing": ["F#3", "A3", "D4"] }
+  },
+  "riff": [[0, 12, 2], [3, 12, 1], [4, 0, 1], [6, 15, 2], [8, 17, 1], [10, 19, 2], [13, 22, 1], [14, 24, 1]],
+  "hook": [
+    [[0, "B4", 2], [2, "D5", 1], [3, "E5", 3], [6, "G5", 1], [7, "E5", 2], [10, "D5", 4], [14, "B4", 2]],
+    [[0, "A4", 2], [2, "B4", 2], [4, "D5", 3], [8, "E5", 2], [11, "D5", 1], [12, "B4", 4]]
+  ],
+  "sections": [
+    { "name": "intro", "bars": [0, 2], "texture": "intro", "chords": ["Em9"], "picture": "the title card" },
+    { "name": "build", "bars": [2, 4], "texture": "build", "energy": 0.65, "chords": ["Em9", "A13"], "picture": "a half-strength shuffle" },
+    { "name": "i. the shuffle", "bars": [4, 8], "texture": "groove", "energy": 0.72, "chords": ["Em9", "A13"], "crash": 0.25, "picture": "the side-to-side shuffle on the eighths" },
+    { "name": "ii. the palm stop", "bars": [8, 12], "texture": "stop", "energy": 0.75, "chords": ["Em9", "Dmaj", "Cmaj7", "B7"], "hits": [0, 8], "picture": "poses snap on beats 1 and 3" },
+    { "name": "iii. the arm pump", "bars": [12, 15], "texture": "pump", "energy": 1.0, "chords": ["A13", "A13", "Em9"], "crash": 0.22, "picture": "a fist driven up on every beat" },
+    { "name": "spin 1", "bars": [15, 16], "texture": "spin", "picture": "a full spin into the drop" },
+    { "name": "iv. the drop", "bars": [16, 20], "texture": "drop", "energy": 1.05, "chords": ["Em9", "A13"], "crash": 0.35, "hit": "Em9", "picture": "the shuffle again, bigger" },
+    { "name": "v. the split drop", "bars": [20, 24], "texture": "dark", "energy": 0.8, "chords": ["Am9", "Am9", "B7", "B7"], "crash": 0.28, "picture": "down into a split and up, on every beat, the night palette" },
+    { "name": "vi. the chorus line", "bars": [24, 27], "texture": "chorus", "energy": 1.25, "chords": ["Cmaj7", "D6", "Em9"], "crash": 0.38, "hit": "Cmaj7", "picture": "the biggest moment: the pump, everyone" },
+    { "name": "the final pose", "bars": [27, 30], "texture": "ending", "pickup": 0.5, "hits": [{ "at": 27.5, "chord": "B7" }, { "at": 28, "chord": "Em9" }], "picture": "a half-bar spin, then the pose held to the end" }
+  ],
+  "fills": [
+    { "bar": 11, "from": 12, "to": 16, "level": 0.75 },
+    { "bar": 23, "from": 8, "to": 16, "level": 0.6 }
+  ]
+}

package/skills/vos-score/references/spec.md

@@ -0,0 +1,79 @@
+# The score spec
+
+`scripts/compose.py spec.json score.wav` reads one JSON object. The worked
+example is `new-jack-swing-124.json` beside this file.
+
+## Positions
+
+- A **bar** is four beats; bar 0 starts at t = 0. `bars: [4, 8]` is bars 4,
+  5, 6 and 7.
+- A **16th** position inside a bar is 0 to 15: beat 1 is 0, beat 2 is 4,
+  beat 3 is 8, beat 4 is 12; the "and" of beat 1 is 2. Odd 16ths are swung
+  late by `swing` (a fraction of a 16th).
+- A **bar position** as a number is bar plus a fraction: `27.5` is bar 27,
+  16th 8 (beat 3). Used by ending hits.
+- **Notes** are names or MIDI numbers: `"E1"` (28), `"F#3"`, `"Bb2"`, `64`.
+  C4 is 60.
+
+## Top level
+
+| Field | Required | Meaning |
+| --- | --- | --- |
+| `bpm` | yes | The program's tempo. |
+| `bars` | yes | The piece's length in bars; the sections must cover exactly 0 to `bars`. |
+| `length` | no | The output length in seconds, when the program's duration is not a whole number of bars (default `bars x 4 x 60 / bpm`). |
+| `swing` | no | How late odd 16ths fall, as a fraction of a 16th: 0 is straight, 0.24 (the default) is about 62 % swing, the new jack and hip hop pocket. |
+| `seed` | no | The noise seed (drums, risers, the room). Same spec, same seed, same file. |
+| `chords` | yes | `{ "<name>": { "root": <bass note>, "voicing": [<comp notes>] } }`. Roots sit low (E1 to D2), voicings around G3 to B4 (three or four notes, close, the 7th and 9th rather than doubled roots). |
+| `riff` | for `drop`, `chorus` | The bass riff, one bar: `[[16th, semitones over the chord's root, length in 16ths], ...]`. It follows the chord changes because it is intervals. |
+| `hook` | for `drop`, `chorus` | The melody: a list of one-bar phrases, each `[[16th, note, length in 16ths], ...]`, cycled bar by bar through a section. Two phrases make a call and an answer. |
+| `sub` | no | The plain bass pattern the build, groove and pump play (default the root, an octave up, on 1, on 3 and on the "and" of 4). Same shape as `riff`. |
+| `held` | no | The held bass the `dark` texture plays (default long notes on 1 and 3, a fifth on 4). |
+| `sections` | yes | The arrangement, in order, no gaps (below). |
+| `fills` | no | Snare rolls over the end of a bar: `{ "bar": 11, "from": 12, "to": 16, "level": 0.75 }` (16ths, straight, rising). A fill is written into the section it sits in. |
+| `title`, `harmony` | no | Words for the next reader. |
+
+## Sections
+
+Every section has `name` (unique), `bars: [start, end]` and `texture`, plus
+`picture` (free text: what is on screen, ignored by the composer). Groove
+textures take `chords` (one per bar, cycled) and `energy`.
+
+`energy` scales every level in the texture by `energy / reference`, so the
+reference plays it exactly as tuned. Textures differ in density, so equal
+energy is not equal loudness: compose, read `analyze.py`, adjust.
+
+| Texture | Reference energy | What plays |
+| --- | --- | --- |
+| `intro` | none | An orchestra hit on the first chord (`ring`, default 1.8 s), a riser over the last bar, a hat pickup and one kick into the next section. |
+| `build` | 0.65 | Kick on 1 and 3, claps on 2 and 4, 8th hats, the plain sub bass, a pad. |
+| `groove` | 0.72 | The main groove: a syncopated kick (1, the "and" of 2, 3, the "a" of 3), claps on 2 and 4, 8th hats with two open, the funk pluck on the `sub` pattern, the electric piano comping off the beat. |
+| `stop` | 0.75 | Stop-time: on each 16th in `hits` (default `[0, 8]`, beats 1 and 3) the kick, an orchestra hit, a bass note and a clap together, silence between. |
+| `pump` | 1.0 | Four on the floor, open hats on the off-beats, the funk pluck, brass on every beat. |
+| `spin` | none | A fill section: a rising snare roll and a riser to one beat before the end, kicks on the beats, the last beat empty so the next downbeat lands. `level` (default 0.95) is the roll's. |
+| `drop` | 1.05 | The riff, the hook, brass stabs on the "and" of 1 and 3, 16th hats, the kick adding a pickup every other bar. |
+| `dark` | 0.8 | A breakdown: kick on 1, 3 and its "and", the `held` bass, a pad every two bars, a soft snare on every beat. |
+| `chorus` | 1.25 | Everything: the riff, the hook up `octave` (default 1) with a harmony a fourth under, gang claps, a pad, brass stabs, 16th tambourine. |
+| `ending` | none | `pickup` (bars, a roll and a riser from the section start), then `hits: [{ "at": <bar position>, "chord": "<name>" }, ...]`. Every hit but the last is a short orchestra hit with a kick; the last is the big one (`ring`, default 3.6 s), with a falling brass chord, a held bass note and a crash, and soft hats carry the room out through the bars after it. |
+| `silence` | none | Nothing. The `holes` check skips these bars: a silence is a choice you declare. |
+
+Per-section options on groove textures:
+
+| Option | Meaning |
+| --- | --- |
+| `kicks` | 16ths for the kick: `[0, 4, 8, 12]`, or a list of lists cycled bar by bar. |
+| `claps` | 16ths for the claps (default `[4, 12]`). |
+| `hats` | `"16"`, `"8"` or `""` for none. |
+| `open_hats` | 16ths where the hat opens. |
+| `crash` | A crash on the section's downbeat, at this level (0.2 to 0.4). |
+| `hit` | A chord name: an orchestra hit on the section's downbeat. |
+
+## Changing the sound
+
+The instruments are methods on `Score` (`kick`, `clap`, `bass_note`,
+`brass_chord`, ...), the textures are `tex_<name>` methods, and the mix
+(room reverb, bus saturation, the cut to length) is `render`. A new texture
+is a `tex_<name>` method plus its reference in `TEXTURE_ENERGY` (and in
+`analyze.py`'s copy when it is a groove texture the order check should
+rank). A new instrument is a method beside the others, synthesized, never
+a sample of a record.

package/skills/vos-score/scripts/analyze.py

@@ -0,0 +1,225 @@
+#!/usr/bin/env python3
+"""
+Verify a score by numbers, because an agent cannot listen.
+
+    python analyze.py score.mp3 --spec spec.json [--lufs -16]
+    python analyze.py score.mp3 --bpm 124 --bars 30
+
+Prints the tempo and first-beat fit (kick-band onset flux), the length, the
+integrated loudness, per-bar loudness (all bands and the 30 to 150 Hz low
+band), and, with a spec, per-section loudness against the reference section
+and the checks below. Exits 1 when a check fails. Needs ffmpeg, numpy, scipy.
+
+The checks (with --spec):
+  grid      the tempo fit is the spec's BPM within 0.05, beat 1 within 0.02 s of t = 0
+  length    the file is the program's length within 0.05 s
+  order     a section with more energy is at least 0.5 dB louder than one with less
+  biggest   the highest-energy section is at least 1 dB louder than every other
+  holes     no bar falls more than 10 dB under the bar before it (a fill is not a hole)
+  ring      the bar after the last hit is quieter than the hit's bar (the ending rings out)
+  lufs      integrated loudness within 1 LU of --lufs, when given
+"""
+import argparse
+import json
+import re
+import subprocess
+import sys
+
+import numpy as np
+from scipy import signal
+
+SR = 11025
+# The energy each groove texture plays at by default (compose.py's TEXTURE_ENERGY).
+TEXTURE_ENERGY = {'build': 0.65, 'groove': 0.72, 'stop': 0.75, 'pump': 1.0, 'drop': 1.05, 'dark': 0.8, 'chorus': 1.25}
+GROOVES = tuple(TEXTURE_ENERGY)
+ORDER_GAP = 0.03    # energies closer than this are not ordered
+ORDER_STEP = 0.5    # dB a higher-energy section must clear
+BIGGEST_STEP = 1.0  # dB the biggest moment must clear
+HOLE = 10.0         # dB a bar may fall under the bar before it
+
+
+def decode(path):
+    out = subprocess.run(
+        ['ffmpeg', '-v', 'error', '-i', path, '-ac', '1', '-ar', str(SR), '-f', 'f32le', '-'],
+        capture_output=True,
+    )
+    if out.returncode != 0:
+        sys.exit(f'ffmpeg could not read {path}: {out.stderr.decode().strip()}')
+    return np.frombuffer(out.stdout, dtype=np.float32).astype(np.float64)
+
+
+def integrated_lufs(path):
+    out = subprocess.run(
+        ['ffmpeg', '-v', 'info', '-i', path, '-af', 'loudnorm=print_format=json', '-f', 'null', '-'],
+        capture_output=True, text=True,
+    )
+    m = re.search(r'"input_i"\s*:\s*"(-?[\d.]+|-inf)"', out.stderr)
+    return float(m.group(1)) if m and m.group(1) != '-inf' else None
+
+
+def tempo_fit(flux, fps, lo, hi, step):
+    """The BPM and first-beat phase whose beat grid lands on the most kick onsets."""
+    dur = len(flux) / fps
+    best = (-1.0, None, None)
+    for bpm in np.arange(lo, hi + step / 2, step):
+        P = 60.0 / bpm
+        offs = np.arange(0, P, 0.002)
+        beats = np.arange(0, dur, P)
+        idx = ((offs[:, None] + beats[None, :]) * fps).astype(int)
+        valid = idx < len(flux)
+        scores = np.where(valid, flux[np.minimum(idx, len(flux) - 1)], 0).sum(axis=1)
+        k = int(np.argmax(scores))
+        if scores[k] > best[0]:
+            best = (scores[k], bpm, offs[k])
+    return best[1], best[2]
+
+
+def db(x):
+    return 20 * np.log10(np.sqrt(np.mean(x ** 2)) + 1e-9) if len(x) else -180.0
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument('audio')
+    ap.add_argument('--spec', help='the compose spec: BPM, bars and the sections to check')
+    ap.add_argument('--bpm', type=float, help='the grid tempo, without a spec')
+    ap.add_argument('--bars', type=int, help='the number of bars, without a spec')
+    ap.add_argument('--lufs', type=float, help='check integrated loudness against this target (the catalog is -16)')
+    a = ap.parse_args()
+
+    spec = None
+    if a.spec:
+        with open(a.spec) as f:
+            spec = json.load(f)
+    bpm = float(spec['bpm']) if spec else a.bpm
+    bars = int(spec['bars']) if spec else a.bars
+
+    x = decode(a.audio)
+    sos = signal.butter(4, [30, 150], 'bandpass', fs=SR, output='sos')
+    low = signal.sosfilt(sos, x)
+    hop = 64
+    fps = SR / hop
+    n = (len(low) - hop) // hop
+    env = np.sqrt(np.mean(low[: n * hop].reshape(n, hop) ** 2, axis=1))
+    flux = np.maximum(0, np.diff(env, prepend=env[0]))
+
+    if bpm:
+        fit_bpm, phase = tempo_fit(flux, fps, bpm - 3, bpm + 3, 0.01)
+    else:
+        coarse, _ = tempo_fit(flux, fps, 70, 180, 0.1)
+        fit_bpm, phase = tempo_fit(flux, fps, coarse - 0.3, coarse + 0.3, 0.01)
+    P = 60.0 / fit_bpm
+    off = min(phase, P - phase)  # distance of the beat grid from t = 0
+    print(f'tempo fit: {fit_bpm:.2f} BPM, first beat at {phase:.3f} s')
+    length = len(x) / SR
+    print(f'length: {length:.3f} s')
+    lufs = integrated_lufs(a.audio)
+    if lufs is not None:
+        print(f'integrated loudness: {lufs:.1f} LUFS')
+
+    checks = []
+
+    def check(name, ok, line):
+        checks.append(ok)
+        print(f'  {"pass" if ok else "FAIL"}  {name:<8} {line}')
+
+    if not bpm:
+        print('(no --spec or --bpm: the fit is a search from 70 to 180 BPM; half and double tempo are possible)')
+        return
+
+    BAR = 4 * 60.0 / bpm
+    bars = bars or int(round(length / BAR))
+    per_bar = []
+    for b in range(bars):
+        i0, i1 = int(b * BAR * SR), int((b + 1) * BAR * SR)
+        per_bar.append((db(x[i0:i1]), db(low[i0:i1])))
+    print('\nbar   start s  all dB  low dB')
+    for b, (al, lo) in enumerate(per_bar):
+        print(f'{b:3d}  {b * BAR:8.3f}  {al:6.1f}  {lo:6.1f}')
+
+    print('\nchecks:')
+    check('grid', abs(fit_bpm - bpm) <= 0.05 and off <= 0.02,
+          f'{fit_bpm:.2f} BPM against {bpm:g}, beat 1 {off:.3f} s from t = 0 (limits 0.05 BPM, 0.02 s)')
+    expect_len = float(spec.get('length', bars * BAR)) if spec else bars * BAR
+    check('length', abs(length - expect_len) <= 0.05, f'{length:.3f} s against {expect_len:.3f} s')
+    if a.lufs is not None:
+        ok = lufs is not None and abs(lufs - a.lufs) <= 1.0
+        check('lufs', ok, f'{lufs if lufs is not None else "unreadable"} against {a.lufs:g} (within 1 LU)')
+
+    if spec:
+        sections = spec['sections']
+        loud = {}
+        print('\nsection                 bars      texture  energy  all dB  low dB  vs ref')
+        ref = next((s for s in sections if s['texture'] == 'groove'), None) or \
+            next((s for s in sections if s['texture'] in GROOVES), None)
+        for s in sections:
+            b0, b1 = s['bars']
+            i0, i1 = int(b0 * BAR * SR), int(b1 * BAR * SR)
+            loud[s['name']] = (db(x[i0:i1]), db(low[i0:i1]))
+        ref_db = loud[ref['name']][0] if ref else 0.0
+        for s in sections:
+            al, lo = loud[s['name']]
+            if s['texture'] in GROOVES:
+                s.setdefault('energy', TEXTURE_ENERGY[s['texture']])
+            e = s.get('energy')
+            es = f'{e:6.2f}' if e is not None else '      '
+            mark = '  (ref)' if ref is s else ''
+            print(f'{s["name"][:22]:<22}  {s["bars"][0]:>4g}-{s["bars"][1]:<4g}  {s["texture"]:<7}  {es}  '
+                  f'{al:6.1f}  {lo:6.1f}  {al - ref_db:+5.1f}{mark}')
+
+        print('\nshape:')
+        grooves = [s for s in sections if s['texture'] in GROOVES]
+        pairs, closest, order_failed = 0, None, False
+        for i, s in enumerate(grooves):
+            for t in grooves[i + 1:]:
+                hi_s, lo_s = (s, t) if s['energy'] > t['energy'] else (t, s)
+                if abs(s['energy'] - t['energy']) < ORDER_GAP:
+                    continue
+                pairs += 1
+                step = loud[hi_s['name']][0] - loud[lo_s['name']][0]
+                line = (f'{hi_s["name"]} (energy {hi_s["energy"]:g}) is {step:+.1f} dB over {lo_s["name"]} '
+                        f'(energy {lo_s["energy"]:g}), needs +{ORDER_STEP:g}')
+                if step < ORDER_STEP:
+                    order_failed = True
+                    check('order', False, line)
+                elif closest is None or step < closest[0]:
+                    closest = (step, line)
+        if closest and not order_failed:
+            check('order', True, f'{pairs} pairs in energy order; the closest: {closest[1]}')
+        if grooves:
+            top = max(grooves, key=lambda s: s['energy'])
+            others = [s for s in sections if s is not top]
+            if others:
+                runner = max(others, key=lambda s: loud[s['name']][0])
+                step = loud[top['name']][0] - loud[runner['name']][0]
+                check('biggest', step >= BIGGEST_STEP,
+                      f'{top["name"]} is {step:+.1f} dB over the next loudest, {runner["name"]} (needs +{BIGGEST_STEP:g})')
+        silent = set()
+        for s in sections:
+            if s['texture'] == 'silence':
+                silent.update(range(int(s['bars'][0]), int(np.ceil(s['bars'][1]))))
+        worst = (0.0, None)
+        for b in range(1, bars):
+            if b in silent:
+                continue
+            fall = per_bar[b - 1][0] - per_bar[b][0]
+            if fall > worst[0]:
+                worst = (fall, b)
+        check('holes', worst[0] <= HOLE,
+              f'the deepest fall is {worst[0]:.1f} dB, into bar {worst[1]} (limit {HOLE:g} dB)' if worst[1] is not None
+              else 'no bar falls under the bar before it')
+        ending = next((s for s in sections if s['texture'] == 'ending'), None)
+        if ending and ending.get('hits'):
+            hb = int(np.floor(max(h['at'] for h in ending['hits'])))
+            if hb + 1 < bars:
+                check('ring', per_bar[hb + 1][0] < per_bar[hb][0],
+                      f'bar {hb + 1} is {per_bar[hb + 1][0] - per_bar[hb][0]:+.1f} dB against the last hit\'s bar {hb}')
+
+    failed = checks.count(False)
+    print(f'\n{len(checks) - failed} of {len(checks)} checks pass' + ('' if not failed else f', {failed} FAIL'))
+    if failed:
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()