rushes 0.10.1 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "rushes",
3
3
  "description": "A local review desk for video made with AI agents: timecoded notes, script edits and audio picks your agent reads and acts on.",
4
- "version": "0.10.1",
4
+ "version": "0.11.0",
5
5
  "author": { "name": "Red Morley Hewitt" },
6
6
  "homepage": "https://github.com/iamredmh/rushes",
7
7
  "repository": "https://github.com/iamredmh/rushes",
package/AGENTS.md CHANGED
@@ -122,7 +122,7 @@ The user can shelve a film they have set aside (**Shelve film** in the dashboard
122
122
  4. **Fix each note.** Before you start on a note, call `rushes_working_on` with its `note` id and what you are doing in a few plain words (`doing`, 200 characters at most, e.g. "Moving the title up"): the user sees it on the note while you work. It is only for a note that was sent to you; any other is refused (`note_not_sent`). It shows only while Rushes is running you on a batch or an answer; otherwise (the user pasted the prompt to you) it is accepted and changes nothing. A note has `stage`, `scope` (`point`, `range` or `whole` — on every tab, a voice read included: a voice range note is about that stretch of the read), `t`, `tOut`, `on` (the lane, variant, take, cue or section it's about), `text`, and optionally `box` (normalised 0 to 1), `grab` (a PNG path in `screenshots/`; older notes may point to `.rushes/grabs/`), `shot` (the shot it falls in, `{n, name}`, worked out by the server from the version's shots — not yours to set) and, on Picture, `format` (`null` when the note is for every format, or a format id such as `"9x16"` when it's for that shape only). When a note's `format` is not null, fix only that format and leave the others alone; an all-format note is for every shape, so fix each and say in `reply` which formats you fixed. `rushes_list_notes` takes `format` (an id such as `9x16`, or its label `9:16`: that format's Picture notes plus the all-format ones) and `onlyThisFormat: true` (only that format's). On an audio stage, `on` is `"<laneId>/<variantId>"` for a music bed, SFX pass or voice read (the lane and variant ids `rushes_add_variant` returns) — for a voice read the lane id is the round's slugged name, so `on` names the round and the read together — or `"<laneId>/<variantId>:<cueId>"` for an SFX cue. On a mix note (stage `mix`, made on the Timeline tab) it's `null` for the whole mix, `"vo"` for the voice (the newest round with a pick), or `"<laneId>/<variantId>"` for the music or the effects. Older notes may carry a bare variant id, `"vo"` on a voice note itself, a section id, `"<sectionId>:<takeId>"` for a take, `"<variantId>:<cueId>"` or a bare cue id; Rushes still lists these with a sensible label, but none of them name a read any more. A music, sfx or mix note can also carry `marks` — short feedback like `Fall` (bring the level down over the range) or `Quieter 3 dB` (3 dB lower over the range). `rushes_list_notes`, `rushes_get_batch` and exported notes all show them. Marks are the user's, like a note's text and times — you read and act on them, you don't set them. Voice notes carry no marks: a voice note is about the whole read (`scope: "whole"`), a point in it (`t`) or a range of it (`t` to `tOut`: replace or fix just that stretch), and its `text` says what to change. Older voice notes may begin with a word such as `Speaker`, `Pacing`, `Tone`, `Fix this` or `Keep this`, from quick-start buttons the dashboard no longer has.
123
123
  5. **Register the new cut** with `rushes_add_version`, with a short `label` and the detail in `note`. If the video is locked, the reply carries `warning: "Picture is locked at vN"`; the version is still added.
124
124
  6. **Reply to every note** in one `rushes_reply` call. For each, set `status: "done"`, a one-line `reply`, `fixT` (when the fix is visible in the new cut, in seconds) and `fixVersion`. If you didn't fix a note, leave it `todo` and say why in `reply`. Every note comes back with its `state`: `todo` (the user's, not sent yet), `sent` (in a batch the user sent: yours, and shown to them as "With your agent") or `done`. `rushes_list_notes` with `state: "sent"` lists everything you have been given and not finished, whichever batch it came in. A done note the user opens again is `todo` and goes out in their next batch.
125
- 7. **Audio batches.** For a note on the voice, the music, the effects or the mix (stage `mix` is the Timeline tab, where every take in use plays together), call `rushes_get_picks` first (pass the note's film as `video`: picks and levels are per film), so you know which read or variant is in use before you fix anything. It also returns `levels: { voice?, music?, sfx? }` in dB — where the user wants each sound to sit. Don't try to change these; they're set from the level on each sound's row of the Timeline tab and only tell you the balance the user has already chosen. Then fix each note:
125
+ 7. **Audio batches.** For a note on the voice, the music, the effects or the mix (stage `mix` is the Timeline tab, where every take in use plays together), call `rushes_get_picks` first (pass the note's film as `video`: picks and levels are per film), so you know which read or variant is in use before you fix anything. It also returns `levels: { voice?, music?, sfx? }` in dB — where the user wants each sound to sit. Don't try to change these; they're set from the level on each sound's row of the Timeline tab and only tell you the balance the user has already chosen. When `balanced` is there and its `levels` equal `levels`, they are Rushes' house balance (§29: music level with the voice and clear of it on every spoken line, each effect just under the music around it), set by Balance or by itself when a take was put in use; a level that differs was moved by hand. Rushes balances a new take in use against the others by itself while the levels are still the balance's own, so render a new bed, pass or read at a sensible level rather than leaning on the fader: `rushes balance --voice F --music F --sfx F` runs the same rule on files (no server) and says how many dB each would move; aim for moves near 0, and re-render a take it says hit the end of the fader. A finished mix you render yourself applies `levels` on top of the takes, as Mix plays them. Then fix each note:
126
126
  - **Voice:** register a new whole read with `rushes_add_variant` (`stage: "voice"`) — a new round for a new direction (another voice, a different tone), or the current round for a small fix to a marked word. Say in `description` what changed.
127
127
  - **Music or effects:** make the new bed or pass the text and marks ask for, and register it as in step 1.
128
128
 
package/dist/cli/main.js CHANGED
@@ -20,6 +20,9 @@ import { setup } from "../setup/setup.js";
20
20
  import { realSetupEnv } from "../setup/env.js";
21
21
  import { HARNESS_IDS } from "../setup/harnesses.js";
22
22
  import { runDoctor, realDoctorEnv } from "./doctor.js";
23
+ import { balanceFiles } from "../server/balance.js";
24
+ import { ffmpegAvailable } from "../server/loudness.js";
25
+ import { balanceSummary } from "../core/balance.js";
23
26
  export const HELP = `rushes ${VERSION}: a local review desk for video made with AI agents
24
27
 
25
28
  Usage
@@ -60,6 +63,9 @@ Usage
60
63
  rushes log add <text> [--area A] add a line to the Change Log
61
64
  flags first; text that starts with "-" goes after --:
62
65
  rushes log add --area mix -- "-3 dB on the bed"
66
+ rushes balance [--voice F] [--music F] [--sfx F] [--json]
67
+ the house balance for these files: how many dB to move the
68
+ music and effects so they sit against the voice (no server)
63
69
 
64
70
  Options
65
71
  --dir DIR project folder (default: current folder); where a command takes [dir], that wins
@@ -93,6 +99,10 @@ const OPTIONS = {
93
99
  only: { type: "string" },
94
100
  "dry-run": { type: "boolean" },
95
101
  "idle-minutes": { type: "string" },
102
+ // §29: `balance --voice F --music F --sfx F`.
103
+ voice: { type: "string" },
104
+ music: { type: "string" },
105
+ sfx: { type: "string" },
96
106
  // R17: `mcp --pinned`, the server Send to AI hands its agent: tools work on the given folder only.
97
107
  pinned: { type: "boolean" },
98
108
  help: { type: "boolean", short: "h" },
@@ -453,6 +463,26 @@ export async function main(argv, io) {
453
463
  }
454
464
  return usage(io, "rushes add version|format|variant|shots|file ...");
455
465
  }
466
+ case "balance": {
467
+ // §29: the same rule the desk's Balance runs, on files, with no server: for the agent that makes the sound.
468
+ const files = Object.fromEntries(["voice", "music", "sfx"].filter((k) => o[k]).map((k) => [k, resolve(io.cwd, o[k])]));
469
+ if (Object.keys(files).length === 0)
470
+ return usage(io, "rushes balance [--voice FILE] [--music FILE] [--sfx FILE] [--json]");
471
+ if (!(await ffmpegAvailable())) {
472
+ io.err("ffmpeg isn't on PATH: install it to measure loudness.");
473
+ return 1;
474
+ }
475
+ const { levels, report } = await balanceFiles(files);
476
+ if (o.json)
477
+ io.out(JSON.stringify({ levels, report, summary: balanceSummary(report) }, null, 2));
478
+ else {
479
+ for (const k of ["voice", "music", "sfx"])
480
+ if (levels[k] !== undefined)
481
+ io.out(`${k.padEnd(6)} ${levels[k] > 0 ? "+" : ""}${levels[k]} dB`);
482
+ io.out(balanceSummary(report));
483
+ }
484
+ return 0;
485
+ }
456
486
  case "lock": {
457
487
  const [video, lockVersion] = rest;
458
488
  if (!video || !lockVersion)
@@ -0,0 +1,238 @@
1
+ // §29: the house balance. Given each take's loudness over time, how far to move the music and effects faders so the
2
+ // three sounds sit against each other: the voice rings true over the music on every spoken line, the music holds a set
3
+ // level, and every effect is heard, not hidden under the bed. Pure: the readings come from ffmpeg (server/balance.ts),
4
+ // and `rushes balance` runs the same rule on files for the agent that makes the sound.
5
+ //
6
+ // The numbers are calibrated by ear on Red's Meet your agents mix (19 Nov 2026): the music he settled on sits level
7
+ // with the voice's integrated loudness, the voice still clears it in the speech band on every line, and the effects
8
+ // pass he wanted heard needed about +10 dB, which puts each hit about 1 dB under the music around it.
9
+ import { LEVEL_MAX, LEVEL_MIN, LEVEL_STEP } from "./schema.js";
10
+ export const BALANCE = {
11
+ /** LU the music's integrated loudness sits under the voice's (0: level with it). */
12
+ musicUnderVoice: 0,
13
+ /** LU the voice must clear the music by, in the speech band, on every spoken line. */
14
+ voiceClearance: 1,
15
+ /** dB each effect's momentary peak sits under the music's short-term level around it. */
16
+ sfxUnderGroove: 1,
17
+ /** With no music in use: dB each effect's peak sits under the voice's integrated loudness. */
18
+ sfxUnderVoice: 6,
19
+ /** With no voice in use: the music's integrated loudness, LUFS. */
20
+ musicAlone: -20,
21
+ /** With neither: each effect's momentary peak, LUFS. */
22
+ sfxAlone: -20,
23
+ /** dBTP the balanced mix may peak at: over it, every sound comes down together (`trimmed`). */
24
+ peakCeiling: -1,
25
+ };
26
+ /** The band a voice is understood in: ffmpeg filters applied before measuring voice and music for clearance. */
27
+ export const SPEECH_BAND = { low: 300, high: 4000 };
28
+ const SILENT = -70;
29
+ const FRAME_S = 0.1;
30
+ const finite = (n) => typeof n === "number" && Number.isFinite(n) && n > SILENT;
31
+ const pow = (db) => 10 ** (db / 10);
32
+ const toDb = (p) => (p > 0 ? 10 * Math.log10(p) : -120);
33
+ /** Loudness of several readings together: their power averaged, not their dB. */
34
+ const energyMean = (dbs) => (dbs.length === 0 ? -120 : toDb(dbs.reduce((a, d) => a + pow(d), 0) / dbs.length));
35
+ /** The frames of a reading by index (frame i covers t ≈ (i + 1) × 0.1 s), so two takes line up frame for frame. */
36
+ function byIndex(frames) {
37
+ const out = new Map();
38
+ for (const f of frames)
39
+ out.set(Math.round(f.t / FRAME_S) - 1, f);
40
+ return out;
41
+ }
42
+ /**
43
+ * Where the voice speaks: frames within 20 LU of its integrated loudness, joined across pauses of 0.4 s or less, and
44
+ * at least 0.3 s long. A breath between words stays inside one line; a gap between sentences ends it.
45
+ */
46
+ export function spokenLines(voice) {
47
+ if (!finite(voice.integrated))
48
+ return [];
49
+ const floor = Math.max(voice.integrated - 20, SILENT);
50
+ const lines = [];
51
+ let cur = null;
52
+ for (const f of voice.frames) {
53
+ if (f.m <= floor)
54
+ continue;
55
+ const i = Math.round(f.t / FRAME_S) - 1;
56
+ if (cur && f.t - cur.end <= 0.4 + 1e-9) {
57
+ cur.end = f.t;
58
+ cur.frames.push(i);
59
+ }
60
+ else {
61
+ cur = { start: f.t, end: f.t, frames: [i] };
62
+ lines.push(cur);
63
+ }
64
+ }
65
+ return lines.filter((l) => l.end - l.start >= 0.3 - 1e-9);
66
+ }
67
+ /** The effects in a pass: everything within 30 dB of the pass's loudest moment (and above -60 LUFS). */
68
+ export function effectHits(sfx) {
69
+ const loudest = Math.max(-120, ...sfx.frames.map((f) => f.m));
70
+ if (loudest <= SILENT)
71
+ return [];
72
+ const floor = Math.max(loudest - 30, -60);
73
+ const hits = [];
74
+ let cur = null;
75
+ for (const f of sfx.frames) {
76
+ if (f.m <= floor)
77
+ continue;
78
+ const i = Math.round(f.t / FRAME_S) - 1;
79
+ if (cur && f.t - cur.end <= 0.3 + 1e-9) {
80
+ cur.end = f.t;
81
+ if (f.m > cur.peak)
82
+ Object.assign(cur, { peak: f.m, t: f.t, index: i });
83
+ }
84
+ else {
85
+ cur = { t: f.t, index: i, peak: f.m, end: f.t };
86
+ hits.push(cur);
87
+ }
88
+ }
89
+ return hits.map(({ t, index, peak }) => ({ t, index, peak }));
90
+ }
91
+ const median = (xs) => {
92
+ const s = [...xs].sort((a, b) => a - b);
93
+ return s.length === 0 ? null : s.length % 2 ? s[(s.length - 1) / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2;
94
+ };
95
+ /** A fader setting: on the 0.5 dB step and inside the fader's range. */
96
+ export function toLevel(db) {
97
+ const stepped = Math.round(db / LEVEL_STEP) * LEVEL_STEP;
98
+ return Math.min(LEVEL_MAX, Math.max(LEVEL_MIN, stepped)) + 0; // + 0: never -0
99
+ }
100
+ /**
101
+ * The house balance for whichever of voice, music and effects are in use. The voice stays at 0 dB: it is the
102
+ * reference. Music is set level with the voice (BALANCE.musicUnderVoice), then pulled back if the voice would clear it
103
+ * by less than BALANCE.voiceClearance in the speech band on any spoken line. Effects are set so the median hit peaks
104
+ * BALANCE.sfxUnderGroove dB under the music's short-term level at that moment (after the music's own move). A sound
105
+ * that is silent, or wasn't read, gets no level.
106
+ */
107
+ export function balance(takes) {
108
+ const levels = {};
109
+ const clamped = [];
110
+ const set = (stage, db) => {
111
+ const level = toLevel(db);
112
+ if (db > LEVEL_MAX + LEVEL_STEP / 2 || db < LEVEL_MIN - LEVEL_STEP / 2)
113
+ clamped.push(stage);
114
+ levels[stage] = level;
115
+ return level;
116
+ };
117
+ const voice = takes.voice && finite(takes.voice.full.integrated) ? takes.voice : undefined;
118
+ const music = takes.music && finite(takes.music.full.integrated) ? takes.music : undefined;
119
+ const sfx = takes.sfx && takes.sfx.full.frames.some((f) => f.m > SILENT) ? takes.sfx : undefined;
120
+ const report = { voice: voice ? voice.full.integrated : null, music: null, sfx: null, clamped, trim: 0 };
121
+ if (voice)
122
+ levels.voice = 0;
123
+ // ---- music ----
124
+ let musicGain = 0;
125
+ if (music) {
126
+ const mI = music.full.integrated;
127
+ if (voice) {
128
+ const vI = voice.full.integrated;
129
+ let want = vI - BALANCE.musicUnderVoice - mI;
130
+ let by = "voice";
131
+ const margins = clearances(voice, music);
132
+ if (margins.length > 0) {
133
+ const closest = Math.min(...margins);
134
+ const limit = closest - BALANCE.voiceClearance;
135
+ if (limit < want) {
136
+ want = limit;
137
+ by = "clearance";
138
+ }
139
+ }
140
+ musicGain = set("music", want);
141
+ report.music = { level: musicGain, by, clearance: margins.length > 0 ? round1(Math.min(...margins) - musicGain) : null };
142
+ }
143
+ else {
144
+ musicGain = set("music", BALANCE.musicAlone - mI);
145
+ report.music = { level: musicGain, by: "alone", clearance: null };
146
+ }
147
+ }
148
+ // ---- effects ----
149
+ if (sfx) {
150
+ const hits = effectHits(sfx.full);
151
+ const groove = music ? byIndex(music.full.frames) : null;
152
+ const vI = voice ? voice.full.integrated : null;
153
+ let by = music ? "music" : voice ? "voice" : "alone";
154
+ const moves = hits.map((h) => {
155
+ const g = groove?.get(h.index)?.s;
156
+ // A hit where the music is silent (before it starts, say) is set against the voice, or the set level.
157
+ if (finite(g))
158
+ return g + musicGain - BALANCE.sfxUnderGroove - h.peak;
159
+ if (vI !== null)
160
+ return vI - BALANCE.sfxUnderVoice - h.peak;
161
+ return BALANCE.sfxAlone - h.peak;
162
+ });
163
+ const move = median(moves);
164
+ if (move !== null) {
165
+ if (music && hits.every((h) => !finite(groove?.get(h.index)?.s)))
166
+ by = voice ? "voice" : "alone";
167
+ report.sfx = { level: set("sfx", move), hits: hits.length, by };
168
+ }
169
+ }
170
+ return { levels, report };
171
+ }
172
+ /** On each spoken line: how far the voice is over the music in the speech band, LU, with the music at 0 dB. */
173
+ function clearances(voice, music) {
174
+ if (!voice.band || !music.band)
175
+ return [];
176
+ const vb = byIndex(voice.band.frames);
177
+ const mb = byIndex(music.band.frames);
178
+ return spokenLines(voice.full).map((line) => {
179
+ const v = energyMean(line.frames.map((i) => vb.get(i)?.m ?? -120));
180
+ const m = energyMean(line.frames.map((i) => mb.get(i)?.m ?? -120));
181
+ return v - m;
182
+ });
183
+ }
184
+ const round1 = (n) => Math.round(n * 10) / 10;
185
+ /**
186
+ * A balance brought down so the mix peaks at BALANCE.peakCeiling at most: every sound by the same amount (rounded up to
187
+ * the fader's step), voice included, so they still sit the same against each other. `truePeak` is the balanced mix's,
188
+ * dBTP. Unchanged when it is under the ceiling or unknown.
189
+ */
190
+ export function trimmed(b, truePeak) {
191
+ if (truePeak === null || !Number.isFinite(truePeak) || truePeak <= BALANCE.peakCeiling)
192
+ return b;
193
+ const by = Math.ceil((truePeak - BALANCE.peakCeiling) / LEVEL_STEP) * LEVEL_STEP;
194
+ const levels = {};
195
+ for (const stage of ["voice", "music", "sfx"])
196
+ if (b.levels[stage] !== undefined)
197
+ levels[stage] = toLevel(b.levels[stage] - by);
198
+ const { music, sfx } = b.report;
199
+ return {
200
+ levels,
201
+ report: { ...b.report, music: music && { ...music, level: levels.music }, sfx: sfx && { ...sfx, level: levels.sfx }, trim: -by },
202
+ };
203
+ }
204
+ const signed = (n) => `${n > 0 ? "+" : n < 0 ? "\u2212" : ""}${Math.abs(n)} dB`;
205
+ /** What a balance did, in a sentence or two, for the desk's toast, the agent and the CLI. */
206
+ export function balanceSummary(r) {
207
+ const parts = [];
208
+ if (r.music) {
209
+ const why = r.music.by === "alone" ? "at the set level, with no voice in use"
210
+ : r.music.by === "clearance" ? `held back so the voice clears it by ${BALANCE.voiceClearance} LU on every line`
211
+ : "level with the voice";
212
+ parts.push(`Music ${signed(r.music.level)}, ${why}.`);
213
+ }
214
+ if (r.sfx) {
215
+ const against = r.sfx.by === "music" ? "just under the music around them" : r.sfx.by === "voice" ? "under the voice" : "at the set level";
216
+ parts.push(`Effects ${signed(r.sfx.level)}: ${r.sfx.hits} ${r.sfx.hits === 1 ? "hit" : "hits"} ${against}.`);
217
+ }
218
+ if (parts.length === 0)
219
+ parts.push(r.voice !== null ? "Only the voice is in use: nothing to balance against it." : "Nothing in use to balance.");
220
+ if (r.trim < 0)
221
+ parts.push(`Everything ${signed(r.trim)} so the mix doesn't clip.`);
222
+ if (r.clamped.length > 0)
223
+ parts.push(`${r.clamped.map((s) => NAMES[s]).join(" and ")} hit the end of the fader: that take needs re-rendering at a different level.`);
224
+ return parts.join(" ");
225
+ }
226
+ const NAMES = { voice: "Voice", music: "Music", sfx: "Effects" };
227
+ /**
228
+ * The per-frame lines of an ebur128 run (`t: 1.2 TARGET:-23 LUFS M: -20.1 S: -22.3 ...`). `-inf` and `nan` read as
229
+ * silence.
230
+ */
231
+ export function parseFrames(stderr) {
232
+ const out = [];
233
+ const num = (s) => (s === "-inf" || s === "nan" || s === "inf" ? -120 : Number(s));
234
+ for (const m of stderr.matchAll(/\bt:\s*([\d.]+)\s+TARGET:[^\n]*?\bM:\s*(-?inf|nan|-?[\d.]+)\s+S:\s*(-?inf|nan|-?[\d.]+)/g)) {
235
+ out.push({ t: Number(m[1]), m: num(m[2]), s: num(m[3]) });
236
+ }
237
+ return out;
238
+ }
@@ -286,10 +286,10 @@ export const NotesFileSchema = fileObject({
286
286
  rev: z.number().int().nonnegative(),
287
287
  notes: z.array(NoteSchema).default([]),
288
288
  });
289
- // §19.6: a Mix lane's level, in dB. −24..6 in 0.5 dB steps; PUT /api/picks enforces the range and
289
+ // §19.6: a Mix lane's level, in dB. −24..12 in 0.5 dB steps (§29: +12, so a quiet take can be brought up to the house balance); PUT /api/picks enforces the range and
290
290
  // step (a value outside it, or off the step, is a 400), so the stored file is never checked again here.
291
291
  export const LEVEL_MIN = -24;
292
- export const LEVEL_MAX = 6;
292
+ export const LEVEL_MAX = 12;
293
293
  export const LEVEL_STEP = 0.5;
294
294
  export const LevelsSchema = fileObject({ voice: z.number(), music: z.number(), sfx: z.number() }).partial().default({});
295
295
  // What one film hears: its own pick per lane and its own Mix levels. Music, sound effects, the
@@ -301,10 +301,17 @@ export const LevelsSchema = fileObject({ voice: z.number(), music: z.number(), s
301
301
  */
302
302
  export const UseSchema = fileObject({ lane: id, variant: id, at: z.string(), seen: z.record(z.string(), z.string()).default({}) });
303
303
  export const InUseSchema = fileObject({ voice: UseSchema, music: UseSchema, sfx: UseSchema }).partial();
304
+ /**
305
+ * §29: the levels the last balance set, and when. While a film's levels still equal these (or none are set), its
306
+ * levels are the balance's own: a new take in use is balanced again. Once a fader is moved by hand they differ, and
307
+ * nothing moves it back until Balance is pressed.
308
+ */
309
+ export const BalancedSchema = fileObject({ levels: LevelsSchema, at: z.string() });
304
310
  export const VideoPicksSchema = fileObject({
305
311
  lanes: z.record(z.string(), z.string()).default({}),
306
312
  levels: LevelsSchema,
307
313
  use: InUseSchema.optional(),
314
+ balanced: BalancedSchema.optional(),
308
315
  });
309
316
  export const PicksSchema = fileObject({
310
317
  schema: z.literal(1),
@@ -318,6 +325,8 @@ export const PicksSchema = fileObject({
318
325
  // §27.6: the take in use per sound. In this top-level place only for the picks of one film as
319
326
  // picksForVideo hands them out (and a project from before picks were per film).
320
327
  use: InUseSchema.optional(),
328
+ // §29: the project-wide set's last balance, for a project with no cut yet.
329
+ balanced: BalancedSchema.optional(),
321
330
  // Each film's own picks and levels, keyed by video id. A film with no entry has picked nothing.
322
331
  videos: z.record(z.string(), VideoPicksSchema).default({}),
323
332
  });
@@ -326,7 +335,7 @@ export function picksForVideo(picks, videoId) {
326
335
  if (!videoId)
327
336
  return picks;
328
337
  const own = picks.videos[videoId];
329
- return { ...picks, lanes: { ...(own?.lanes ?? {}) }, levels: { ...(own?.levels ?? {}) }, use: { ...(own?.use ?? {}) }, sections: {}, videos: {} };
338
+ return { ...picks, lanes: { ...(own?.lanes ?? {}) }, levels: { ...(own?.levels ?? {}) }, use: { ...(own?.use ?? {}) }, balanced: own?.balanced, sections: {}, videos: {} };
330
339
  }
331
340
  /** The films a read, bed or pass is on: its `videos`, or an older project's one `video`, or none. */
332
341
  export function variantFilms(v) {
@@ -36,6 +36,8 @@ import { buildState, planWarning } from "../core/plan.js";
36
36
  import { broughtInEvent, cutEvent, fileEvent, filmEvent, formatEvent, lineEvent, lockEvent, notesSentEvent, picksEvent, repliesEvent, scriptEvent, takeEvent, variantEvent, variantFilmsEvent, visualEvent, } from "../core/logEvents.js";
37
37
  import { LOG_AREAS, LOG_MAX, changeLogFileName, logText, recentChanges } from "../core/logText.js";
38
38
  import { defaultRunner, measureMix } from "./loudness.js";
39
+ import { balanceFilm } from "./balance.js";
40
+ import { balanceSummary } from "../core/balance.js";
39
41
  import { clearDetectCache, detectAgents } from "./agent/detect.js";
40
42
  import { openInTerminal } from "./agent/terminal.js";
41
43
  import { agentRulesFor } from "./agent/rules.js";
@@ -208,6 +210,7 @@ const ShotsBody = z.object({
208
210
  const LockBody = z.object({ version: z.string().nullable() });
209
211
  const ShelveBody = z.object({ shelved: z.boolean() });
210
212
  const MixLoudnessBody = z.object({ lanes: z.array(LaneStageSchema).min(1), video: z.string().min(1).max(200).optional() });
213
+ const BalanceBody = z.object({ video: z.string().min(1).max(200).optional() });
211
214
  const SettingsBody = z.object({ autoProxy: z.boolean() }).strict();
212
215
  // A time in seconds, 0 to 24 h: an empty value, NaN, Infinity or anything huge is a 400.
213
216
  const FrameQuery = z.object({
@@ -1356,6 +1359,12 @@ export function createApp(store, opts = {}) {
1356
1359
  const usedNow = project ? filmUse(project, data, b.video) : {};
1357
1360
  const sameUse = ["voice", "music", "sfx"].every((s) => usedBefore[s]?.lane === usedNow[s]?.lane && usedBefore[s]?.variant === usedNow[s]?.variant);
1358
1361
  const changed = Object.keys({ ...before, ...after }).some((k) => before[k] !== after[k]) || !sameUse;
1362
+ // §29: a new take in use is balanced against the others by itself, unless a fader has been moved by hand.
1363
+ if (opts.autoBalance && project && !sameUse && b.levels === undefined) {
1364
+ const own = b.video ? data.videos[b.video] : data;
1365
+ if (own && balanceOwned(own))
1366
+ autoBalance(b.video, by(c));
1367
+ }
1359
1368
  if (changed) {
1360
1369
  const inUseLanes = Object.fromEntries(Object.values(usedNow).map((u) => [u.lane, u.variant]));
1361
1370
  if (project)
@@ -1365,6 +1374,79 @@ export function createApp(store, opts = {}) {
1365
1374
  }
1366
1375
  return c.json(project ? withUse(project, data, b.video) : b.video ? picksForVideo(data, b.video) : data);
1367
1376
  });
1377
+ // ---- §29: balance ----
1378
+ // Levels are the balance's own while none are set, or while they still equal what the last balance set
1379
+ // (an unset level is 0 dB). Moving a fader by hand ends that until Balance is pressed again.
1380
+ const sameLevels = (a, b) => ["voice", "music", "sfx"].every((k) => (a[k] ?? 0) === (b[k] ?? 0));
1381
+ const balanceOwned = (own) => own.balanced ? sameLevels(own.levels, own.balanced.levels) : ["voice", "music", "sfx"].every((k) => (own.levels[k] ?? 0) === 0);
1382
+ /**
1383
+ * Measures the film's takes in use and sets its levels to the house balance (core/balance.ts). `guard` (the
1384
+ * automatic run) is asked again inside the write, against the picks as they are then: if a fader was moved or
1385
+ * another take picked while ffmpeg ran, nothing is written.
1386
+ */
1387
+ const runBalance = async (video, who, guard) => {
1388
+ const [project, picks] = await Promise.all([store.read("project"), store.read("picks")]);
1389
+ const heard = { ...project, lanes: lanesForVideo(project.lanes, video, project.videos.length) };
1390
+ const result = await balanceFilm(heard, picksForVideo(picks, video), store.root, opts.loudnessRunner ?? defaultRunner, opts.loudnessTimeoutMs);
1391
+ if (!result.available)
1392
+ return { result, written: false, project, picks };
1393
+ let written = false;
1394
+ const { data } = await store.update("picks", (p) => {
1395
+ const own = video ? (p.videos[video] ??= { lanes: {}, levels: {} }) : p;
1396
+ if (guard && !guard(own, p, project))
1397
+ return;
1398
+ // Each sound in use takes its balanced level (0 dB is stored as unset, as a fader reset is); the others keep theirs.
1399
+ for (const stage of ["voice", "music", "sfx"]) {
1400
+ const v = result.levels[stage];
1401
+ if (v === undefined)
1402
+ continue;
1403
+ if (v === 0)
1404
+ delete own.levels[stage];
1405
+ else
1406
+ own.levels[stage] = v;
1407
+ }
1408
+ own.balanced = { levels: { ...own.levels }, at: new Date().toISOString() };
1409
+ written = true;
1410
+ });
1411
+ if (written) {
1412
+ const film = project.videos.find((v) => v.id === video);
1413
+ const event = { area: "mix", kind: "entry", merge: "replace", subject: `balance:${film?.id ?? ""}`, video: film?.id ?? null, version: null, text: `Mix balanced${film ? ` for ${film.name}` : ""}: ${balanceSummary(result.report)}` };
1414
+ await log.add(() => event, who);
1415
+ }
1416
+ return { result, written, project, picks: data };
1417
+ };
1418
+ // The automatic run, one at a time per film: a pick arriving while one is measuring runs again after it.
1419
+ const autoRuns = new Map();
1420
+ const autoBalance = (video, who) => {
1421
+ const key = video ?? "";
1422
+ const prev = autoRuns.get(key) ?? Promise.resolve();
1423
+ const run = prev.then(async () => {
1424
+ const before = await store.read("picks").catch(() => null);
1425
+ const project = await store.read("project").catch(() => null);
1426
+ if (!before || !project)
1427
+ return opts.onAutoBalance?.(video, null);
1428
+ const useAt = JSON.stringify(filmUse(project, before, video));
1429
+ const { result, written } = await runBalance(video, who, (own, p, proj) => balanceOwned(own) && JSON.stringify(filmUse(proj, p, video)) === useAt);
1430
+ opts.onAutoBalance?.(video, written && result.available ? balanceSummary(result.report) : null);
1431
+ }).catch(() => opts.onAutoBalance?.(video, null));
1432
+ autoRuns.set(key, run);
1433
+ void run.finally(() => {
1434
+ if (autoRuns.get(key) === run)
1435
+ autoRuns.delete(key);
1436
+ });
1437
+ };
1438
+ app.post("/api/mix/balance", async (c) => {
1439
+ const b = await body(c, BalanceBody);
1440
+ if (b.video) {
1441
+ const project = await store.read("project");
1442
+ if (!project.videos.some((v) => v.id === b.video))
1443
+ throw new RushesError(`No film "${b.video}"`, 404, "not_found");
1444
+ }
1445
+ const { result, project, picks } = await runBalance(b.video, by(c));
1446
+ if (!result.available)
1447
+ return c.json({ available: false, summary: "Install ffmpeg to balance the mix." });
1448
+ return c.json({ available: true, levels: result.levels, report: result.report, summary: balanceSummary(result.report), picks: withUse(project, picks, b.video) });
1449
+ });
1368
1450
  // ---- mix loudness (§17.6) ----
1369
1451
  app.post("/api/mix/loudness", async (c) => {
1370
1452
  const b = await body(c, MixLoudnessBody);
@@ -0,0 +1,103 @@
1
+ // §29: auto-balance on the server. Reads each take in use (voice, music, effects) with ffmpeg's ebur128 -- the full
2
+ // band, and for voice and music the speech band too -- and hands the readings to the house rule (core/balance.ts).
3
+ import { stat } from "node:fs/promises";
4
+ import { existsSync } from "node:fs";
5
+ import { balance, parseFrames, SPEECH_BAND, trimmed } from "../core/balance.js";
6
+ import { AUDIO_FORMATS } from "../core/media.js";
7
+ import { fromManifestPath } from "../core/paths.js";
8
+ import { inUse, inUseVariant } from "../core/picks.js";
9
+ import { defaultRunner, ffmpegAvailable, loudnessArgs, LoudnessTimeoutError, parseEbur128 } from "./loudness.js";
10
+ /** The ffmpeg arguments that read one file's loudness over time, optionally through the speech band. */
11
+ export function readingArgs(file, band) {
12
+ const filter = `${band ? `highpass=f=${SPEECH_BAND.low},lowpass=f=${SPEECH_BAND.high},` : ""}ebur128`;
13
+ return ["-nostats", "-protocol_whitelist", "file", "-format_whitelist", AUDIO_FORMATS, "-i", file, "-af", filter, "-f", "null", "-"];
14
+ }
15
+ /** One ffmpeg run, killed (and a 504) past `timeoutMs`. */
16
+ async function timed(run, args, timeoutMs) {
17
+ const controller = new AbortController();
18
+ let timer;
19
+ const timeout = new Promise((_r, reject) => {
20
+ timer = setTimeout(() => {
21
+ controller.abort();
22
+ reject(new LoudnessTimeoutError(timeoutMs));
23
+ }, timeoutMs);
24
+ });
25
+ try {
26
+ return await Promise.race([run(args, controller.signal), timeout]);
27
+ }
28
+ finally {
29
+ clearTimeout(timer);
30
+ }
31
+ }
32
+ async function read(run, file, band, timeoutMs) {
33
+ const r = await timed(run, readingArgs(file, band), timeoutMs);
34
+ const { integrated } = parseEbur128(r.stderr);
35
+ return { integrated: integrated !== null && Number.isFinite(integrated) ? integrated : null, frames: parseFrames(r.stderr) };
36
+ }
37
+ // One reading per (file, mtime, band), per runner: the same take is read once however often the balance runs.
38
+ const CACHE_LIMIT = 60;
39
+ const caches = new WeakMap();
40
+ async function cachedRead(run, file, band, timeoutMs) {
41
+ let key = null;
42
+ try {
43
+ key = JSON.stringify([file, (await stat(file)).mtimeMs, band]);
44
+ }
45
+ catch {
46
+ key = null;
47
+ }
48
+ let cache = caches.get(run);
49
+ if (!cache)
50
+ caches.set(run, (cache = new Map()));
51
+ if (key !== null) {
52
+ const hit = cache.get(key);
53
+ if (hit)
54
+ return hit;
55
+ }
56
+ const reading = await read(run, file, band, timeoutMs);
57
+ if (key !== null && reading.frames.length > 0) {
58
+ cache.set(key, reading);
59
+ if (cache.size > CACHE_LIMIT)
60
+ cache.delete(cache.keys().next().value);
61
+ }
62
+ return reading;
63
+ }
64
+ /**
65
+ * The house balance for these files. Voice and music are read in the speech band too. Then the balanced mix is measured
66
+ * once, and brought down together if it would peak over BALANCE.peakCeiling.
67
+ */
68
+ export async function balanceFiles(files, run = defaultRunner, timeoutMs = 60_000) {
69
+ const takes = {};
70
+ await Promise.all(Object.entries(files).map(async ([stage, file]) => {
71
+ const banded = stage !== "sfx" && files.voice !== undefined && files.music !== undefined;
72
+ const [full, band] = await Promise.all([cachedRead(run, file, false, timeoutMs), banded ? cachedRead(run, file, true, timeoutMs) : undefined]);
73
+ takes[stage] = band ? { full, band } : { full };
74
+ }));
75
+ const b = balance(takes);
76
+ const inputs = Object.entries(files)
77
+ .filter(([stage]) => b.levels[stage] !== undefined)
78
+ .map(([stage, file]) => ({ file, offset: 0, stage, gainDb: b.levels[stage] }));
79
+ if (inputs.length === 0)
80
+ return b;
81
+ const { truePeak } = parseEbur128((await timed(run, loudnessArgs(inputs), timeoutMs)).stderr);
82
+ return trimmed(b, truePeak);
83
+ }
84
+ /** The take in use for each sound (as Mix plays them), whose file is on disk. */
85
+ export function takesInUse(project, picks, root) {
86
+ const use = inUse(project, picks);
87
+ const files = {};
88
+ for (const stage of ["voice", "music", "sfx"]) {
89
+ const taken = inUseVariant(project, use, stage);
90
+ if (!taken)
91
+ continue;
92
+ const abs = fromManifestPath(root, taken.variant.file);
93
+ if (existsSync(abs))
94
+ files[stage] = abs;
95
+ }
96
+ return files;
97
+ }
98
+ /** The house balance for a film's takes in use; `available: false` without ffmpeg. */
99
+ export async function balanceFilm(project, picks, root, run = defaultRunner, timeoutMs) {
100
+ if (!(await ffmpegAvailable(run)))
101
+ return { available: false };
102
+ return { available: true, ...(await balanceFiles(takesInUse(project, picks, root), run, timeoutMs)) };
103
+ }
@@ -129,6 +129,8 @@ export async function startServer(rootDir, opts = {}) {
129
129
  const appOpts = {
130
130
  brief, storyboard, webDir: opts.webDir, onShutdown: () => void close(), reveal: opts.reveal, proxyJobs, peakJobs, found, formatProbe: opts.formats?.probe,
131
131
  agent: runner, detect: agentOpts?.detect, notify: opts.notify,
132
+ // §29: on for a real server; RUSHES_AUTO_BALANCE=0 turns it off (the browser tests, whose fixtures are test tones).
133
+ autoBalance: opts.autoBalance ?? process.env.RUSHES_AUTO_BALANCE !== "0",
132
134
  };
133
135
  const app = createApp(store, appOpts);
134
136
  const listener = getRequestListener(app.fetch);
package/dist/version.js CHANGED
@@ -1,2 +1,2 @@
1
1
  /** The Rushes version. A leaf module so any server file can import it without a cycle. */
2
- export const VERSION = "0.10.1";
2
+ export const VERSION = "0.11.0";
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "rushes",
3
- "version": "0.10.1",
3
+ "version": "0.11.0",
4
4
  "description": "A local review desk for video made with AI agents: timecoded notes, script edits and audio picks your agent can read and act on.",
5
5
  "keywords": [
6
6
  "video",
@@ -64,7 +64,7 @@ For stills, standalone images and carousels, register one visual per post with `
64
64
  - `rushes_add_variant` for each voice read, music bed or SFX pass, with `video` set to the film it is for, or a list of films when one bed is fitted to several (only those films' tabs list it; with more than one film, one without `video` is on no film until it is put on one, and `rushes_set_films` moves an existing one). Put a take only on the films it was fitted to (a read timed to one film's cut is that film's alone, even when another film shares its script), and when a new cut changes a film's length or timing, fit its bed and effects to that cut as new takes. Voice reads carry `stage: "voice"` and a `round` (its name — reads in one round are compared side by side; a new direction gets a new round); say what the read is, or what changed, in `description`. Music and SFX take a `round` too (each new batch is a round). A take you add is not in use until the user picks it. Music and SFX carry `meta.description` for their row in the list of takes, and SFX passes carry `cues` as `{name, t, file?}`: the same name each time a sound comes back, and `file` (an audio file) when you know which sample you placed;
65
65
  - `rushes_set_shots` for the storyboard's shots, once you have the first cut; send new timings when a later cut moves them.
66
66
  2. Call `rushes_open` and tell the user it's ready, with what it brought in and what it left (below). Each project's dashboard lives at its own address, `/p/<id>/`.
67
- 3. When the user sends a batch, call `rushes_get_batch` and fix every note in it. Before you start on each note, call `rushes_working_on` with its id and what you are doing, in a few plain words: the user sees it on the note while you work. A batch holds every open note from every tab and film, so each note's own `stage` and `video` say where it belongs. A note carries the `shot` it falls in; on an audio stage, call `rushes_get_picks` first — for a mix note (stage `mix`, made on the Timeline tab) it also returns `levels` (dB per sound), which you read, not set. Its `on` says what it's about: `"<lane>/<variant>"` for a bed, pass or voice read (for a voice read the lane names the round), `"<lane>/<variant>:<cue>"` for an SFX cue, `"vo"` for the voice in the mix, or `null` for the whole mix; older notes may carry a section id or `"<section>:<take>"`. On music, sfx and mix notes treat `marks` (e.g. "Fall", "Quieter 3 dB") as part of the note; voice notes carry none — they're on the whole read, a point in it, or a range of it (`t` to `tOut`: replace or fix just that stretch), and the `text` says what to change. A note on the cut (stage `picture`) has a `format` that is null for every format, or a format id such as `"9x16"`: when it isn't null, fix only that format. Picks are set from the dashboard only — no tool or CLI command sets one.
67
+ 3. When the user sends a batch, call `rushes_get_batch` and fix every note in it. Before you start on each note, call `rushes_working_on` with its id and what you are doing, in a few plain words: the user sees it on the note while you work. A batch holds every open note from every tab and film, so each note's own `stage` and `video` say where it belongs. A note carries the `shot` it falls in; on an audio stage, call `rushes_get_picks` first — for a mix note (stage `mix`, made on the Timeline tab) it also returns `levels` (dB per sound), which you read, not set; when they equal `balanced.levels` they are Rushes' house balance (Balance on the Timeline, or set by itself when a take is put in use), so render new takes near those levels: `rushes balance --voice F --music F --sfx F` measures files against the same rule and says how far each would move. Its `on` says what it's about: `"<lane>/<variant>"` for a bed, pass or voice read (for a voice read the lane names the round), `"<lane>/<variant>:<cue>"` for an SFX cue, `"vo"` for the voice in the mix, or `null` for the whole mix; older notes may carry a section id or `"<section>:<take>"`. On music, sfx and mix notes treat `marks` (e.g. "Fall", "Quieter 3 dB") as part of the note; voice notes carry none — they're on the whole read, a point in it, or a range of it (`t` to `tOut`: replace or fix just that stretch), and the `text` says what to change. A note on the cut (stage `picture`) has a `format` that is null for every format, or a format id such as `"9x16"`: when it isn't null, fix only that format. Picks are set from the dashboard only — no tool or CLI command sets one.
68
68
  4. Register the new cut with `rushes_add_version`. On a locked video the reply carries a `warning`.
69
69
  5. Reply to all the notes in one `rushes_reply` call. For each note set `status: "done"`, a one-line `reply`, `fixT` (when the fix is visible in the new cut, in seconds) and `fixVersion`. Each note has a `state`: `todo` (not sent yet), `sent` (yours to fix) or `done`; `rushes_list_notes` with `state: "sent"` finds every note you still owe. `rushes_get_picks` lists each take's `state` too (`in-use`, `new`, `earlier`, `put-away`): never offer a put-away take again.
70
70
  6. For script batches: when a section's `proposed` differs from `current`, adopt it with `rushes_set_script`, sending just that section with its `id`. If voiceover already exists, re-record the picked voice with the corrected script and register it with `rushes_add_variant` as a new read in a new round.