@homie-rocks/studio 0.49.0 → 0.50.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/bin/homie-studio.mjs +5 -5
- package/functions/FUNCTIONS.md +1 -1
- package/guides/app/SKILL.md +107 -1
- package/guides/art/SKILL.md +154 -0
- package/guides/art/references/RIGHTS.md +20 -0
- package/guides/art/scripts/art.mjs +308 -0
- package/guides/models/SKILL.md +198 -0
- package/guides/models/references/RIGHTS.md +46 -0
- package/guides/models/references/models.json +175 -0
- package/guides/models/scripts/models.mjs +483 -0
- package/guides/music/SKILL.md +197 -0
- package/guides/music/references/COMPOSITION.md +59 -0
- package/guides/music/references/GAME-AUDIO.md +50 -0
- package/guides/music/references/RIGHTS.md +43 -0
- package/guides/music/scripts/lib/audio.mjs +258 -0
- package/guides/music/scripts/lib/eleven.mjs +260 -0
- package/guides/music/scripts/lib/studio.mjs +147 -0
- package/guides/music/scripts/music.mjs +644 -0
- package/guides/plan/SKILL.md +126 -0
- package/guides/plan/references/CODEX.md +105 -0
- package/guides/plan/references/INTERVIEW.md +86 -0
- package/guides/playtest/SKILL.md +205 -0
- package/guides/playtest/references/METHOD.md +124 -0
- package/guides/playtest/references/REVIEWER.md +78 -0
- package/guides/playtest/scripts/lib/exit.mjs +31 -0
- package/guides/playtest/scripts/lib/judge.mjs +502 -0
- package/guides/playtest/scripts/lib/pixels.mjs +128 -0
- package/guides/playtest/scripts/lib/preflight.mjs +69 -0
- package/guides/playtest/scripts/playtest.mjs +870 -0
- package/guides/shop/SKILL.md +4 -0
- package/guides/sound/SKILL.md +156 -0
- package/guides/sound/references/GAME-SOUND.md +100 -0
- package/guides/sound/references/SCORE.md +131 -0
- package/guides/sound/references/TRAPS.md +93 -0
- package/guides/sound/scripts/lib/measure.mjs +252 -0
- package/guides/sound/scripts/lib/score.mjs +415 -0
- package/guides/sound/scripts/lib/sfx.mjs +222 -0
- package/guides/sound/scripts/lib/synth.mjs +509 -0
- package/guides/sound/scripts/sound.mjs +520 -0
- package/guides/style/SKILL.md +105 -0
- package/lib/app-needs.mjs +63 -0
- package/lib/check.mjs +10 -4
- package/lib/codex.mjs +95 -18
- package/lib/decisions.mjs +9 -7
- package/lib/handoff.mjs +1 -1
- package/lib/mcp-tools.mjs +10 -9
- package/lib/mcp.mjs +1 -1
- package/lib/scaffold.mjs +13 -5
- package/lib/studio.mjs +3 -3
- package/lib/template-history.json +42 -0
- package/package.json +1 -1
- package/shop/SHOP.md +4 -0
- package/shop/shop.ts +4 -4
- package/worker/build-id.mjs +1 -1
- package/worker/functions-schema.mjs +15 -0
- package/worker/shop-page.mjs +1 -1
- package/worker/shop.mjs +6 -2
- package/worker/version.mjs +1 -1
package/guides/shop/SKILL.md
CHANGED
|
@@ -147,3 +147,7 @@ Keyless has no per-session expiry or provider-read reconciliation. The keyed pat
|
|
|
147
147
|
carts, cap reservations, local refunds. Managed Payments is supported on Payment Links when Stripe enables it for the account. The studio chooses its needs.
|
|
148
148
|
|
|
149
149
|
For a cause, use `shop init --giving --currency <organisation currency>`: one-time and monthly suggested gifts, no badge or game access. Read `giving.receiptText` from the organisation; leave it empty until provided. Local /shop/ previews before Stripe. Monthly gifts use the existing keyless Payment Links connection and Stripe handles renewals.
|
|
150
|
+
|
|
151
|
+
## App payment references
|
|
152
|
+
|
|
153
|
+
Create the booking/ticket/gift intention first. `shop.buy('full-tune', 1, undefined, { reference: booking.id })` or `shop.checkout({ reference: booking.id })` sends the app's own reference through checkout. Existing boolean returns are unchanged: checkout opened is not paid. The verified durable `order.paid` event carries `data.order`, `data.reference` (null if omitted) and `data.app`. Reopened Payment Links retain the reference. A reference is buyer-supplied correlation, not authorization: verify the record, app, amount and currency, deduplicate by event/order ID, then fulfil it. Keep personal data out of references. Before Stripe is connected, local prices and unpaid requests can be demonstrated; actual payment and recurring collection remain unverified. Test events must be labelled simulations.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: sound
|
|
3
|
+
description: Make a game's sound effects and synthesized music on the creator's own computer, free, with no provider or account — a set of effects (jump, coin, hit, explosion, win, lose, UI) rendered from presets, a short synthesized theme or score from chords and patterns with stems and seamless section loops, a mix of any audio files and expressions — measured for loudness, clipping, late starts and what a phone speaker loses, then wired into the game with a small player that starts on the first touch and switches music on bar lines. Use when someone asks for sound effects, SFX, game audio, a chiptune or synth theme, background music without a paid service, music that changes with the game, or says a game is silent.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Sound for a studio's game
|
|
7
|
+
|
|
8
|
+
Everything here runs on this computer with Node 22 and ffmpeg: nothing is sent anywhere and nothing
|
|
9
|
+
costs money. Sung songs and produced music from a model are the `music` skill (ElevenLabs, paid);
|
|
10
|
+
this skill is effects and synthesized music. They share the studio's `music/` folder and manifest, so
|
|
11
|
+
a synthesized theme gets a song page just like a rendered one.
|
|
12
|
+
|
|
13
|
+
One script: `scripts/sound.mjs` in this skill's folder (Claude Code:
|
|
14
|
+
`node "${CLAUDE_PLUGIN_ROOT}/skills/sound/scripts/sound.mjs" <command>`), run from inside the studio
|
|
15
|
+
(the folder with `studio.json`). Every command takes `--json`.
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
node <sound.mjs> check # node, ffmpeg and its encoders, the studio and its games
|
|
19
|
+
node <sound.mjs> presets # every effect preset, instrument, groove and pattern, one line each
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
No ffmpeg: offer `brew install ffmpeg` (macOS) or the system package; the person approves.
|
|
23
|
+
Finish in this turn: an effect set takes seconds, a one-minute score under a minute.
|
|
24
|
+
|
|
25
|
+
## 1. Effects
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
node <sound.mjs> sfx <set> --kit arcade --for-game <id> # a kit: arcade, platformer, shooter, party, ui
|
|
29
|
+
node <sound.mjs> sfx <set> --only jump,coin,hit,win --style retro # just these, crunchy 8-bit
|
|
30
|
+
node <sound.mjs> sfx <set> --spec music/<set>/effects.json # your own list (below)
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
It writes `music/<set>/sfx/<name>-<n>.wav` and an Ogg copy of each, `music/<set>/sfx.json` (the
|
|
34
|
+
spec and every file's measured level), a reel to listen to (`<set>.mp3`) and its picture
|
|
35
|
+
(`<set>-reel.png`, a spectrogram over the waveform). **Open the picture and look**: a sound you
|
|
36
|
+
cannot hear is still a sound you can see. Then read the warnings; each one is something a person will
|
|
37
|
+
hear wrong, with the fix.
|
|
38
|
+
|
|
39
|
+
A spec is `{ "style": "clean", "variants": 3, "effects": [ { "name": "stomp", "preset": "land", "pitch": -3, "length": 1.2, "level": "big" } ] }`:
|
|
40
|
+
`preset` is any name from `presets` (the effect's name is used when it is one), `pitch` in
|
|
41
|
+
semitones, `length` a stretch, `seed` another take, `style` clean, retro or soft, `level` ui,
|
|
42
|
+
small, normal, big or a number in dB. Name effects after what happens in the game (`stomp`,
|
|
43
|
+
`gem`, `ko`), not after the preset.
|
|
44
|
+
|
|
45
|
+
- **Variants** (three by default) are the same recipe a touch higher or lower, so a sound heard fifty
|
|
46
|
+
times a round does not machine-gun. Signals heard once a round (win, lose, go, countdown) get one take.
|
|
47
|
+
- **Levels** are set by the loudest 50 ms of each effect into four classes (ui -20, small -16,
|
|
48
|
+
normal -12, big -9 dB), then held under -1 dBFS. The set sits together in a mix without balancing.
|
|
49
|
+
- **Every effect must start at once** (a hit that starts 30 ms late feels laggy) and **must reach a
|
|
50
|
+
phone speaker**: under about 300 Hz a phone plays almost nothing, so a thud needs a knock on top.
|
|
51
|
+
The warnings measure both.
|
|
52
|
+
|
|
53
|
+
## 2. A synthesized theme or score
|
|
54
|
+
|
|
55
|
+
Write `music/<slug>/score.json` (the full language, with a worked example: `references/SCORE.md`):
|
|
56
|
+
tempo, key, instruments (presets with a level, pan, echo in beats, reverb send, a duck under the
|
|
57
|
+
kick), sections in whole bars with chords and one pattern per instrument, an arrangement, and the
|
|
58
|
+
sections to loop. Then:
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
node <sound.mjs> score <slug> --spec music/<slug>/score.json --loops main,chase
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
- It renders the mix (-14 LUFS for a page; `--lufs -18` for a bed under a game's effects), the page
|
|
65
|
+
mp3, a 24-bit master kept off the site, one stem per instrument plus the room (they sum to the
|
|
66
|
+
mix), and **seamless loops** of the sections you name: each is rendered three times in a row and
|
|
67
|
+
the middle copy kept, so reverb and echoes that ring over the loop point are already in its head.
|
|
68
|
+
The report gives each loop's seam (1 or less is seamless) and its exact bar length. A loop over 1
|
|
69
|
+
is in `warnings` and the summary says NOT SEAMLESS (`loopsSeamless: false`): an empty warnings
|
|
70
|
+
list means the loops were checked and pass. The master's limiter is applied across the three
|
|
71
|
+
copies, so a loud loop keeps its seam; nothing is faded at the loop point and no bar is shortened.
|
|
72
|
+
- `plan.json` holds the bar grid, so the `video` skill cuts a trailer on this music's bar lines.
|
|
73
|
+
- Look at `<slug>.png` and read the numbers: loudness, true peak, and what it measures through a phone
|
|
74
|
+
speaker. Say them in your report, not adjectives.
|
|
75
|
+
- Write music a person would hum: a hook in the first bars, a clear melody over simple chords, a
|
|
76
|
+
change every 4 or 8 bars. A score for a game is sections of the same tempo and key (calm, chase,
|
|
77
|
+
final seconds) so the game can move between them on a bar line.
|
|
78
|
+
|
|
79
|
+
## 3. A mix of anything
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
node <sound.mjs> mix <slug> --spec music/<slug>/mix.json
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Parts are audio files (a recording, a render, a loop), ffmpeg `aevalsrc` expressions (one, or two for
|
|
86
|
+
left and right), or effect presets, each placed (`at` seconds), levelled (`db`), panned, faded, repeated
|
|
87
|
+
and filtered with an ffmpeg `-af` chain. They are summed at exactly the levels written (no automatic
|
|
88
|
+
normalisation), limited, and brought to `master.lufs` if you give one. Expressions have traps that
|
|
89
|
+
fail silently: read `references/TRAPS.md` before writing one.
|
|
90
|
+
|
|
91
|
+
## 4. Into the game
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
node <sound.mjs> wire <set> <score slug> --game <id>
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
It copies the files into the game (`public/sound/`, or `sound/` for a static game), writes
|
|
98
|
+
`sound.json` (what exists) and `sound.js` (the player) and prints the two lines to add. Then, in the
|
|
99
|
+
game's code:
|
|
100
|
+
|
|
101
|
+
```js
|
|
102
|
+
import { createSound } from './sound.js';
|
|
103
|
+
const sound = createSound({ base: 'sound/' });
|
|
104
|
+
sound.music('<score slug>'); // once, at the top: it starts on the first touch
|
|
105
|
+
sound.play('coin'); // where the thing happens
|
|
106
|
+
sound.play('coin', { pitch: combo }); // a climbing combo; pan -1..1 for where it is
|
|
107
|
+
sound.section('chase'); // the next loop, exactly on the next bar line
|
|
108
|
+
sound.duck(0.4, 0.5); // music down for a big hit
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
- **Sound starts on the first touch or key, with no "tap for sound" screen**: the player resumes audio
|
|
112
|
+
inside the first gesture. A sound asked for before that is dropped, not queued.
|
|
113
|
+
- **Every browser plays its own sounds** for what it sees: call `play` where the each browser
|
|
114
|
+
draw the event (a pickup that shows up in a snapshot, a hit that lands on screen), not only in the
|
|
115
|
+
rules handlers. In a rules game emit a declared effect and play it from room.on in the view; the server has no speaker.
|
|
116
|
+
- Loops go out as Ogg with a WAV fallback (older Safari cannot decode Ogg): the player takes the first
|
|
117
|
+
one the browser decodes. `wire` prints the real download size; keep a game's sound under about 3 MB.
|
|
118
|
+
- Each variant of an effect, and each loop, is a list of files tried in order (`["jump.ogg", "jump.wav"]`);
|
|
119
|
+
a list of one file is fine.
|
|
120
|
+
- A trailer filmed frame by frame gets its sound from this player's own log of what it scheduled
|
|
121
|
+
(off unless a recorder asks; `references/GAME-SOUND.md`, "A log for a recorder"): sound that goes
|
|
122
|
+
through `sound.play` and `sound.music` is in the trailer, sound made any other way is not.
|
|
123
|
+
- Rules for game sound (every action answers, what repeats, music under effects, intensity changes on
|
|
124
|
+
bar lines, the live-loop traps): `references/GAME-SOUND.md`.
|
|
125
|
+
- For sound made live in the browser (a synth voice per player, pitch that follows speed, a beat
|
|
126
|
+
clock), the open engine package `@homie-rocks/audio` has the building blocks; install it exactly
|
|
127
|
+
pinned (`npm install --save-exact @homie-rocks/audio@0.1.0`).
|
|
128
|
+
|
|
129
|
+
## 5. Prove it, and publish it
|
|
130
|
+
|
|
131
|
+
Build, run the game, and let the `playtest` skill's sound row listen: it copies what the game really
|
|
132
|
+
sends to the speaker while it is played, and says how loud it is, whether it clips, what a phone loses,
|
|
133
|
+
whether actions answer with a sound, and whether the music ever started. Never say a game "has sound"
|
|
134
|
+
from the code alone.
|
|
135
|
+
|
|
136
|
+
```sh
|
|
137
|
+
node <sound.mjs> add <slug> --title "<Title>" --blurb "<one line>" [--for-game <id>] --publish
|
|
138
|
+
node <sound.mjs> publish <slug> # rebuild and redeploy the site; the page answers 200
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
An effect set gets a page with its reel and every effect to download; a score gets its player, its
|
|
142
|
+
loops and its stems. The credits line says they were synthesized from code (no samples, no model).
|
|
143
|
+
Commit `music/<slug>/` specs and the manifest; the audio files stay out of git (the studio's
|
|
144
|
+
`.gitignore`) and regenerate from the spec, byte for byte.
|
|
145
|
+
|
|
146
|
+
## Tell the person
|
|
147
|
+
|
|
148
|
+
The page or the game, what was made (how many effects, the theme's length and bars), the loudness
|
|
149
|
+
numbers, the warnings you fixed and any you did not, and that it cost nothing.
|
|
150
|
+
|
|
151
|
+
## Never
|
|
152
|
+
|
|
153
|
+
- Never ship a sound you have not measured and looked at.
|
|
154
|
+
- Never use a mp3 for an effect or a loop: encoders pad the start with silence (a late hit) and the
|
|
155
|
+
ends (a gap at the loop). WAV or Ogg.
|
|
156
|
+
- Never imitate a real song, artist or brand jingle.
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Sound in a game
|
|
2
|
+
|
|
3
|
+
What makes a web game sound finished, and the ways a correct-looking sound system goes quiet. Each
|
|
4
|
+
rule below was learned by a game that broke it.
|
|
5
|
+
|
|
6
|
+
## What the game owes a player
|
|
7
|
+
|
|
8
|
+
- **Every action answers with a sound**, on the frame it happens: a press that does something makes
|
|
9
|
+
a sound; a press that is refused makes a different one (`deny`). Movement itself does not need a
|
|
10
|
+
sound per press, but landing, bumping and stopping do.
|
|
11
|
+
- **Sound starts on the first touch**, with no "tap for sound" screen: browsers keep audio suspended
|
|
12
|
+
until a gesture, so resume inside the first `pointerdown`, `touchend` or `keydown` (the player's
|
|
13
|
+
`sound.js`, or the port toolkit, does it). On iOS a held stick is only a gesture when the finger
|
|
14
|
+
lifts, so the first tap or release is when sound begins; the silent switch mutes Web Audio unless
|
|
15
|
+
`navigator.audioSession.type = 'playback'`.
|
|
16
|
+
- **Music carries the round**: calm at the start, a change when it gets tense, something for the last
|
|
17
|
+
ten seconds, a stinger for the result. A round that is all one intensity has no shape; silence
|
|
18
|
+
followed by a hit is worth more than a louder loop.
|
|
19
|
+
- **It sounds good on a phone speaker and on a TV.** A phone plays almost nothing under 300 Hz: every
|
|
20
|
+
important sound needs energy between 500 Hz and 4 kHz (a knock on a thud, an upper harmonic on a
|
|
21
|
+
bass). Measure it: `sound.mjs analyze` gives the loss through a phone-speaker curve.
|
|
22
|
+
- **Nothing clips** when many things happen at once: sounds share a bus that ends in a gentle limiter,
|
|
23
|
+
and each sound is capped at a few voices (`sound.js` keeps four of a name and 24 in all).
|
|
24
|
+
|
|
25
|
+
## Levels that work
|
|
26
|
+
|
|
27
|
+
| what | level |
|
|
28
|
+
| --- | --- |
|
|
29
|
+
| music bed under play | -18 to -20 LUFS (`score --lufs -18`) |
|
|
30
|
+
| title or menu music | -14 LUFS |
|
|
31
|
+
| effects | the four classes: ui -20, small -16, normal -12, big -9 dB (loudest 50 ms) |
|
|
32
|
+
| a whole game while played | -16 to -12 LUFS measured off its output (the playtest's sound row) |
|
|
33
|
+
|
|
34
|
+
Duck the music for the moments that matter (`sound.duck(0.4, 0.5)` for a knockout or a line of
|
|
35
|
+
speech) instead of raising the effect.
|
|
36
|
+
|
|
37
|
+
## Repetition
|
|
38
|
+
|
|
39
|
+
- A sound heard many times a round needs **variants** (three takes a touch apart in pitch) and a
|
|
40
|
+
little random pitch on each play; the same sample fired twenty times in a row reads as a machine gun.
|
|
41
|
+
- Signals the player must learn (start, win, lose, a warning) stay **the same every time**.
|
|
42
|
+
- A long session should not loop one 8-bar clip for ten minutes: make two or three sections and move
|
|
43
|
+
between them with the game's state.
|
|
44
|
+
|
|
45
|
+
## Changing music with the game
|
|
46
|
+
|
|
47
|
+
- Every loop the same tempo and key; switch on the **next bar line**
|
|
48
|
+
(`next = start + ceil((now - start) / bar) * bar` on the audio clock), which `sound.section()` does.
|
|
49
|
+
- Use the audio clock (`AudioContext.currentTime`), not timers, to schedule anything musical. A timer
|
|
50
|
+
or a `requestAnimationFrame` pump skips beats it sees late, and a loaded machine makes it late: a
|
|
51
|
+
bar counter that rides a pump lost three quarters of its sections under load while the audio
|
|
52
|
+
itself played on. Work the bar out from the audio clock each time you need it.
|
|
53
|
+
- A live arrangement's position is `bar % total bars`: changing the total while it plays makes the
|
|
54
|
+
playhead jump. Change the content of a section, not the arrangement's length.
|
|
55
|
+
- A section that plays once as an intro and a lap that repeats are different lengths: compute the lap
|
|
56
|
+
from the repeating part only.
|
|
57
|
+
- A cue that names a sound that does not exist fails silently in most engines: check every name a
|
|
58
|
+
cue or a section uses against what was loaded, once, at start.
|
|
59
|
+
- A gain, a bus or a send changed after a sound was wired may never reach the running graph (a loader
|
|
60
|
+
that skips "already wired" sounds). Set levels before the first play, or rebuild the node.
|
|
61
|
+
- An `AudioContext` clock stalls while the machine is busy (a render, a capture), so a long set drifts
|
|
62
|
+
from wall time: correct its start offset, never its cues.
|
|
63
|
+
- A module served from cache after a change runs the old code: version its URL (`?v=2`) when you
|
|
64
|
+
change it on a live page.
|
|
65
|
+
|
|
66
|
+
## Events, not counts
|
|
67
|
+
|
|
68
|
+
When sounds follow a game's event list (hits, pickups), give each event a rising sequence number and
|
|
69
|
+
play every event newer than the last one played, in order. A list's length is not a signal: an
|
|
70
|
+
expired event replaced at the same length, or two events in one frame, are lost otherwise.
|
|
71
|
+
|
|
72
|
+
## Mixing effects into a running bed
|
|
73
|
+
|
|
74
|
+
Send one-shots (hits, chimes, slams) straight to the final bus, not through the music's filters: a hit
|
|
75
|
+
routed through a lowpass meant for the bed disappears into it. Bright noise belongs on the bright
|
|
76
|
+
one-shots only. Never raise the master to make hits stand out; lower the bed or duck it.
|
|
77
|
+
|
|
78
|
+
## Proving it
|
|
79
|
+
|
|
80
|
+
Scheduling a node is not proof that anything is audible. Three instruments, cheapest first:
|
|
81
|
+
|
|
82
|
+
1. `sound.mjs analyze <file>` on each asset: level, start, clipping, phone loss, and its picture.
|
|
83
|
+
2. The `playtest` skill's sound row: the game's real output copied off its graph while it is played;
|
|
84
|
+
loudness, clipping, gaps, whether actions answer, whether music ever started.
|
|
85
|
+
3. An offline render of the game's own audio graph (`OfflineAudioContext` in headless Chrome) with a
|
|
86
|
+
clock you can set: it finds NaN voices (a NaN anywhere silences the whole graph) and feedback loops
|
|
87
|
+
that double their gain, without playing anything out loud. `currentTime` stays 0 offline, so give
|
|
88
|
+
the game's audio a clock the render can drive.
|
|
89
|
+
|
|
90
|
+
## A log for a recorder
|
|
91
|
+
|
|
92
|
+
`sound.js` keeps a log of everything it schedules when a recorder asks for one, and only then: the
|
|
93
|
+
recorder sets `window.__homieSoundCapture = { events: [] }` before the page's scripts run, and each
|
|
94
|
+
`play`, each music loop, each stop, duck and fader move is pushed onto `events` with `performance.now()`
|
|
95
|
+
and what is needed to mix it again (the file's address, gain, playback rate, pan, loop, delay). The
|
|
96
|
+
head of `sound.js` lists the events. The `video` skill's `trailer` films the game frame by frame and
|
|
97
|
+
rebuilds its sound from that log and the game's own files. A game with audio code of its own can push
|
|
98
|
+
the same events, and only what is in the log is in the trailer's sound.
|
|
99
|
+
|
|
100
|
+
A visual check can never hear a game. Report numbers, and look at the spectrogram.
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# The score language
|
|
2
|
+
|
|
3
|
+
A score is JSON: tempo and key, instruments, sections in whole bars, an arrangement. `sound.mjs
|
|
4
|
+
score` renders it the same way every time (every random choice is seeded), so the score IS the
|
|
5
|
+
music: commit it, and the audio regenerates byte for byte.
|
|
6
|
+
|
|
7
|
+
## A worked example (48 s, three sections, two loops)
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{
|
|
11
|
+
"title": "Night Market Theme",
|
|
12
|
+
"bpm": 128, "key": "A minor", "beatsPerBar": 4, "stepsPerBeat": 4, "swing": 0,
|
|
13
|
+
"instruments": {
|
|
14
|
+
"lead": { "preset": "pulse-lead", "db": -9, "pan": 0.1, "stepsPerBeat": 2, "echo": { "beats": 0.75, "feedback": 0.3, "mix": 0.2 }, "reverb": 0.15 },
|
|
15
|
+
"arp": { "preset": "pluck", "db": -15, "pan": -0.35, "reverb": 0.2 },
|
|
16
|
+
"bass": { "preset": "saw-bass", "db": -10, "duck": 0.35, "reverb": 0 },
|
|
17
|
+
"pad": { "preset": "pad", "db": -17, "duck": 0.5, "reverb": 0.3 },
|
|
18
|
+
"drums": { "preset": "kit", "db": -7, "reverb": 0.05 }
|
|
19
|
+
},
|
|
20
|
+
"sections": [
|
|
21
|
+
{ "name": "intro", "bars": 4, "chords": "Am F C G",
|
|
22
|
+
"play": { "pad": "chords", "arp": "arp:16", "drums": "hats" } },
|
|
23
|
+
{ "name": "main", "bars": 8, "chords": "Am F C G",
|
|
24
|
+
"play": {
|
|
25
|
+
"lead": "A4 . C5 . E5 - D5 C5 | B4 . G4 . B4*2 C5 D5 | E5 - . C5 A4 . C5 D5 | E5*3 . G5*2 E5*2 | F5 . E5 D5 C5 - A4 . | C5 . B4 . A4*2 G4*2 | A4 . C5 . E5 . A5 - | G5*4 E5*4",
|
|
26
|
+
"bass": "root8", "pad": "chords", "arp": "arp:16", "drums": "four" } },
|
|
27
|
+
{ "name": "chase", "bars": 4, "chords": "Dm E Am Am",
|
|
28
|
+
"play": {
|
|
29
|
+
"lead": { "notes": "D5 E5 F5 A5 | G#5*2 E5*2 | A5 E5 C5 A4 | E5*4", "stepsPerBeat": 1 },
|
|
30
|
+
"bass": "pulse", "arp": "arp-updown:16", "drums": "break+fill" } }
|
|
31
|
+
],
|
|
32
|
+
"arrangement": ["intro", "main", "chase", "main"],
|
|
33
|
+
"loops": ["main", "chase"],
|
|
34
|
+
"master": { "lufs": -14 }
|
|
35
|
+
}
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## The top
|
|
39
|
+
|
|
40
|
+
| field | meaning |
|
|
41
|
+
| --- | --- |
|
|
42
|
+
| `bpm` | 40 to 240 |
|
|
43
|
+
| `beatsPerBar` | 4 by default (3 for a waltz) |
|
|
44
|
+
| `stepsPerBeat` | the grid of a notes pattern: 4 = sixteenth notes (default), 2 = eighths, 1 = quarters. An instrument or a single pattern can set its own |
|
|
45
|
+
| `swing` | 0 to 0.5: every second step comes that fraction of a step late (0.2 to 0.33 is a shuffle) |
|
|
46
|
+
| `arrangement` | section names in playing order; a section can repeat. Default: the sections in order |
|
|
47
|
+
| `loops` | sections to render as seamless loops for the game (`--loops` overrides; `all` for every one) |
|
|
48
|
+
| `master.lufs` | -14 for a page or a trailer, -18 to -20 for a bed under game effects (`--lufs` overrides) |
|
|
49
|
+
| `room` | the shared reverb's size, 0 to 1 (0.55 by default) |
|
|
50
|
+
|
|
51
|
+
## Instruments
|
|
52
|
+
|
|
53
|
+
`{ "preset", "db", "pan", "octave", "legato", "echo", "reverb", "duck", "stepsPerBeat", "tone" }`
|
|
54
|
+
|
|
55
|
+
- `db` the level (-10 by default). Start the lead around -9, drums -7, bass -10, pads -17, and move
|
|
56
|
+
one at a time while listening to the numbers (`analyze` a stem).
|
|
57
|
+
- `pan` -1 (left) to 1 (right). Keep the bass and kick in the middle.
|
|
58
|
+
- `octave` shifts every note of the instrument by octaves.
|
|
59
|
+
- `legato` how much of each step a note holds (0.92 by default; 0.5 is staccato).
|
|
60
|
+
- `echo` `{ beats, feedback, mix, pingpong }`: a tempo-synced echo (0.75 beats is the dotted eighth).
|
|
61
|
+
- `reverb` the send to the shared room, 0 to 1.
|
|
62
|
+
- `duck` 0 to 0.95: the instrument dips on every kick and comes back within half a beat (the pump
|
|
63
|
+
that makes a pad breathe with the drums). Only when the section has a kick.
|
|
64
|
+
- `tone` (drums) shifts the kit's pitch in semitones.
|
|
65
|
+
|
|
66
|
+
Presets (`sound.mjs presets` lists them with what each is for): leads `pulse-lead`, `square-lead`,
|
|
67
|
+
`saw-lead`, `soft-lead`, `bell`; keys `epiano`, `pluck`, `organ`; pads `pad`, `warm-pad`; basses
|
|
68
|
+
`saw-bass`, `square-bass`, `triangle-bass`, `sub-bass`; drums `kit`, `chip-kit`.
|
|
69
|
+
|
|
70
|
+
## Chords
|
|
71
|
+
|
|
72
|
+
`"chords": "Am F C G"` is one chord a bar, repeated to fill the section. With bars marked,
|
|
73
|
+
`"Am F | C G | Dm | E7"` puts two chords in a bar when you write two. Qualities: major (`C`),
|
|
74
|
+
`m`, `7`, `maj7`, `m7`, `m9`, `9`, `add9`, `6`, `m6`, `5`, `dim`, `dim7`, `m7b5`, `aug`, `sus2`,
|
|
75
|
+
`sus4`, `7sus4`; a slash puts another note in the bass (`C/E`). Sharps and flats: `F#m`, `Bb`.
|
|
76
|
+
|
|
77
|
+
## Patterns
|
|
78
|
+
|
|
79
|
+
A pattern is either generated from the chords, or written note by note.
|
|
80
|
+
|
|
81
|
+
**From the chords**: `root` (the root held for the chord), `root8` (eighth-note roots; `root8:16`
|
|
82
|
+
for sixteenths), `pulse` (sixteenth-note roots, a driving bass), `octaves` (root and its octave in
|
|
83
|
+
turn), `chords` (the chord held: a pad), `stabs` (short chords on the off-beats), `arp`, `arp-down`,
|
|
84
|
+
`arp-updown` (the chord's notes one at a time; `:16` sixteenths by default, `:8` eighths). The bass
|
|
85
|
+
plays low, pads in the middle, arpeggios an octave up; `octave` moves any of them.
|
|
86
|
+
|
|
87
|
+
**Notes**: one token per step of the pattern's grid.
|
|
88
|
+
|
|
89
|
+
| token | meaning |
|
|
90
|
+
| --- | --- |
|
|
91
|
+
| `A4`, `C#5`, `Bb3` | a note (A4 = 440 Hz; octaves change at C) |
|
|
92
|
+
| `.` | a rest |
|
|
93
|
+
| `-` | hold the previous note one more step |
|
|
94
|
+
| `A4*3`, `.*4` | a note or rest lasting 3 (or 4) steps |
|
|
95
|
+
| `[A3 C4 E4]` | a chord |
|
|
96
|
+
| `A4!` | an accent (louder); with a length: `A4!*2` |
|
|
97
|
+
| a vertical bar | a bar line: optional, but when you write them every bar must be full, which catches a miscounted bar |
|
|
98
|
+
|
|
99
|
+
A pattern shorter than its section repeats (a one-bar riff over eight bars). With the default grid a
|
|
100
|
+
bar of 4/4 is 16 tokens; for a melody in eighth notes give the instrument `"stepsPerBeat": 2` (8
|
|
101
|
+
tokens a bar), and a single section can differ with `{ "notes": "…", "stepsPerBeat": 1 }`.
|
|
102
|
+
|
|
103
|
+
**Drums**: a groove by name, `four` (four on the floor), `backbeat`, `half` (half-time), `break`,
|
|
104
|
+
`shuffle`, `hats`, `pulse` (kick only), `none`; `+fill` puts a snare and tom fill in the section's last
|
|
105
|
+
bar (`"break+fill"`). Or write the bar yourself, one line per drum, `x` a hit, `X` an accent, `.`
|
|
106
|
+
nothing, 16 cells a bar in 4/4 (the line repeats every bar):
|
|
107
|
+
|
|
108
|
+
```json
|
|
109
|
+
"drums": { "kick": "x.....x...x.....", "snare": "....x.......x...", "hat": "x.x.x.x.x.x.x.x.", "openhat": "..............x.", "fill": true }
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Drums: `kick`, `snare`, `clap`, `hat`, `openhat`, `tom`, `rim`, `shaker`.
|
|
113
|
+
|
|
114
|
+
## What makes it good
|
|
115
|
+
|
|
116
|
+
- **A hook in the first bars.** The main melody should be singable: mostly steps, one leap, a note
|
|
117
|
+
held at the end of a phrase. Repeat it; change the last bar of the repeat.
|
|
118
|
+
- **Contrast between sections**: the calm section drops the drums or the bass; the chase doubles
|
|
119
|
+
the bass rhythm and adds the break; the final seconds go up a key or add the open hat.
|
|
120
|
+
- **Space for the game's effects.** A bed at -18 LUFS with the lead an octave away from the effects'
|
|
121
|
+
range; `duck` the pad and bass under the kick, and `sound.duck()` the music for big hits.
|
|
122
|
+
- **Loops in 4 or 8 bars**, every loop the same tempo and key, so the game can switch on a bar line.
|
|
123
|
+
- Listen with numbers: `analyze` each stem; a stem more than 12 dB under its neighbours is inaudible
|
|
124
|
+
in the mix, and a struck sound (a pluck, a bell) sits about 9 dB under a held one at the same peak,
|
|
125
|
+
so lengthen its decay instead of pushing its level.
|
|
126
|
+
|
|
127
|
+
## Errors that name the fix
|
|
128
|
+
|
|
129
|
+
A bar with the wrong number of steps, an unknown chord quality, a section that plays an instrument not
|
|
130
|
+
in `instruments`, an arrangement naming a missing section: each stops the render with a sentence
|
|
131
|
+
saying what and where. Fix that and render again; a render is free.
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Traps: ffmpeg, expressions and synthesis
|
|
2
|
+
|
|
3
|
+
Each of these produced silence, the wrong level or a click while every command exited 0.
|
|
4
|
+
|
|
5
|
+
## ffmpeg
|
|
6
|
+
|
|
7
|
+
- **Exit code 0 is not audio.** A graph that failed part-way still writes a file whose header promises
|
|
8
|
+
sound. Read it back: `ffprobe -v error -show_entries format=duration:stream=sample_rate,channels -of json <file>`,
|
|
9
|
+
and measure it (`sound.mjs analyze`). A duration of 0, or one far from what you asked for, is a
|
|
10
|
+
failed render with the right name.
|
|
11
|
+
- **`c=2` on a source makes one channel.** `aevalsrc` and friends want a layout name (`c=stereo`,
|
|
12
|
+
`c=mono`) or `2c`; a bare number is read as a channel mask. Ask for the count on the output
|
|
13
|
+
(`-ac 2 -ar 48000`) and the source cannot be wrong about it.
|
|
14
|
+
- **`amix` divides by the number of inputs** unless `normalize=0`: the levels you asked for never
|
|
15
|
+
arrive and nothing says so. (`sound.mjs mix` sums in JavaScript at the levels written.)
|
|
16
|
+
- **`adelay` takes one value per channel** (`adelay=500|500`); a single value delays the left channel only.
|
|
17
|
+
- **mp3 pads both ends with silence**: an effect starts late and a loop gaps at the wrap. Effects and
|
|
18
|
+
loops are WAV or Ogg; mp3 is for a page's player only.
|
|
19
|
+
- **`<audio loop>` gaps at the wrap** in every browser: loop a decoded buffer in Web Audio
|
|
20
|
+
(`source.loop = true`).
|
|
21
|
+
- There may be no `drawtext` in the user's ffmpeg build (it needs freetype): draw labels in a browser
|
|
22
|
+
or an image instead.
|
|
23
|
+
- `loudnorm` needs 400 ms blocks: integrated LUFS of a 0.2 s effect is meaningless. Level short
|
|
24
|
+
effects by their loudest 50 ms (what `sound.mjs` does).
|
|
25
|
+
- Encoders differ: many ffmpeg builds have no `libvorbis`; `libopus` in an `.ogg` is the fallback.
|
|
26
|
+
Older Safari cannot decode Ogg at all, so ship a WAV beside it.
|
|
27
|
+
|
|
28
|
+
## aevalsrc expressions
|
|
29
|
+
|
|
30
|
+
- **No comparison operators.** `t>=1` fails with a misleading "Missing ')'". Use `gte(t,1)`,
|
|
31
|
+
`lte()`, `gt()`, `lt()`, `between(t,a,b)`. There is no `sqr()`: `pow(x,2)`.
|
|
32
|
+
- **`0*inf` is NaN and a NaN silences the whole sum.** An envelope like `exp(-(t-a)*k)*gte(t,a)`
|
|
33
|
+
overflows before the gate: write `exp(-max(0,t-a)*k)`.
|
|
34
|
+
- **`random(x)` keeps its state in variable slot x** and overwrites it; put every `random` in one slot
|
|
35
|
+
(9) and keep `st()/ld()` state in 0..7. There is one random stream: to decorrelate a right channel,
|
|
36
|
+
advance it (`0*random(9)+…`). The first use of a slot must be its `st()`.
|
|
37
|
+
- Quotes, backslashes, semicolons and brackets belong to ffmpeg's graph parser, not the expression;
|
|
38
|
+
keep expressions to arithmetic and functions. Very long expressions (a few thousand characters)
|
|
39
|
+
are fragile: split the terms across parts, they sum to the same thing.
|
|
40
|
+
- `PI` may be missing: write `6.28318` for 2π.
|
|
41
|
+
- One expression renders mono; two (left, right) render true stereo.
|
|
42
|
+
|
|
43
|
+
## Pitch and phase
|
|
44
|
+
|
|
45
|
+
- **In `synth.mjs` a number is hertz, always.** `freq: 110` is 110 Hz and `slide.to: 45` is 45 Hz.
|
|
46
|
+
A note is a name (`freq: "A2"`) or a MIDI number said out loud (`midi: 45`, `slide.toMidi: 45`,
|
|
47
|
+
or `hzOfMidi(45)`). There is no guessing by size: a number of 127 or less used to be read as a
|
|
48
|
+
MIDI note, and every bass note is a frequency under 127 Hz, so a 110 Hz bass came out at 4.7 kHz.
|
|
49
|
+
- **A pitch bend is a change of frequency, integrated into phase.** `sin(2*PI*f(t)*t)` with a
|
|
50
|
+
changing `f` sweeps at twice the intended rate and runs backwards on the way down; the phase must be
|
|
51
|
+
the integral of the frequency (what `synth.mjs` does every sample). A scoop from below in closed
|
|
52
|
+
form is `+2π·df·τ·exp(-(t-a)/τ)` added to the phase.
|
|
53
|
+
- Vibrato depth scales with pitch: ±0.4% of the frequency is about ±7 cents at any note.
|
|
54
|
+
- A tremolo on one note is one voice whose envelope clock wraps (`mod(max(0,t-a)*rate,1)/rate`),
|
|
55
|
+
with the phase left running; restarting the phase each time clicks.
|
|
56
|
+
- An uneven pulse wave (duty other than 50%) has a DC offset: subtract `2·duty - 1`, or high-pass the
|
|
57
|
+
result, or it clicks at every start and stop.
|
|
58
|
+
|
|
59
|
+
## Loops and width
|
|
60
|
+
|
|
61
|
+
- **A held sound loops without a click only if every frequency in it, beating pairs included, is a
|
|
62
|
+
whole multiple of 1/loop-length** (0.125 Hz for an 8 s loop) and no channel starts at a phase
|
|
63
|
+
offset. Notes whose envelope closes before the loop point are exempt. Or render the section three
|
|
64
|
+
times and keep the middle (what `sound.mjs score` does).
|
|
65
|
+
- **Limit a loop before cutting it, not after.** A limiter run over the cut loop starts at full gain
|
|
66
|
+
and ends wherever the last bar left it: a step of gain at the wrap. `score` limits across the
|
|
67
|
+
three copies. A note, an echo or a reverb longer than the whole section still cannot repeat: the
|
|
68
|
+
seam goes over 1 and the report warns; make the section longer or the tail shorter.
|
|
69
|
+
- **A 1-3 ms delay between channels widens a transient but cancels a held tone** in mono (a phone
|
|
70
|
+
speaker folds to mono): correlation fell to -0.2 and the note lost 4 dB. Widen held sounds by
|
|
71
|
+
detuning or by panning different notes, never by a channel offset; offset only the attacks.
|
|
72
|
+
- Measure width with the stereo correlation and the mono loss (`analyze` gives both). Negative
|
|
73
|
+
correlation means parts cancel on a phone.
|
|
74
|
+
|
|
75
|
+
## Levels
|
|
76
|
+
|
|
77
|
+
- A struck sound (a pluck, a bell, a mallet) sits about 9 dB RMS under a held one at the same peak.
|
|
78
|
+
Lengthen its decay rather than raising its fader, then compare stems with their gains applied.
|
|
79
|
+
- A generated room tone or ambience can carry a steady hum (a narrow line in the spectrum): `analyze`
|
|
80
|
+
lists narrow tones; notch them or regenerate. Each one comes with `prominenceDb` (over its
|
|
81
|
+
neighbours, which a filter does not change: the neighbours fall with it), `levelDb` (how loud it
|
|
82
|
+
is, dBFS), `energyShare` and `presentShare` (the share of the file it is there). Only a tone that
|
|
83
|
+
is prominent, above -60 dBFS and there 60% of the time or more is called steady; a melody's brief
|
|
84
|
+
high partial is listed and not warned about. Read its level and share to judge it.
|
|
85
|
+
- Check every render at its head and tail as well as its peak: silence at the start of an effect is
|
|
86
|
+
latency; a tail cut mid-ring is a click.
|
|
87
|
+
|
|
88
|
+
## Provider renders (when a model made the audio)
|
|
89
|
+
|
|
90
|
+
- A paid render can land minutes after the command that asked for it timed out or failed. Before
|
|
91
|
+
asking again, list the output folder by time: the same request twice is twice the bill.
|
|
92
|
+
- Before cutting a straight loop out of a swung track, find the onsets in the rendered file: a hit a
|
|
93
|
+
few ms early is a musical flam, not an error, and triplet grids suit a shuffle.
|