@aicopycoders/exodus 2026.9.2000-beta.14 → 2026.9.2000-beta.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -631,9 +631,12 @@ depends on the scene; see "Which scenes carry their voice where" below.
|
|
|
631
631
|
terms. `script[]` is the ad's spoken words in order — `{lineId, speakerId, text}`,
|
|
632
632
|
with `speakerId: null` meaning narrator VO. Each scene carries `sceneIndex`,
|
|
633
633
|
`durationSec`, `dialogue[]`, `voText`, `notes`, and `lineIds` (the script lines
|
|
634
|
-
that scene renders). A scene with `kind: "cutaway"` carries `cueLineId`
|
|
635
|
-
|
|
636
|
-
keeps running, so
|
|
634
|
+
that scene renders). A scene with `kind: "cutaway"` carries `cueLineId` and a
|
|
635
|
+
`cutawayType` instead. Most cutaways are a picture-only takeover of the frame
|
|
636
|
+
while the spoken track underneath keeps running, so they add no audio and take
|
|
637
|
+
no place in the running order. A `cutawayType: "reaction"` cutaway is the room
|
|
638
|
+
reacting out loud to its cue line: it is its own short beat after that line,
|
|
639
|
+
heard with its own sound while the speaker is silent. The
|
|
637
640
|
envelope also carries `overlays[]` (each with a `cueLineId` and a `placement`)
|
|
638
641
|
and `audio` (`roomTone`, `music`, `sfxCues[]`) — the planner's notes on what
|
|
639
642
|
should sit on top. The image and motion prompts (`framePrompt`, `videoPrompt`,
|
|
@@ -898,6 +901,13 @@ Read these out of `storyboard.json` before you build anything, in any tool:
|
|
|
898
901
|
running order. To find the moment, locate the A-roll scene whose `lineIds`
|
|
899
902
|
contain the cue line, then read the start time of the first word of that line
|
|
900
903
|
out of that scene's `words.json`.
|
|
904
|
+
- **Reactions.** A cutaway with `cutawayType: "reaction"` is the exception. Do
|
|
905
|
+
not lay it over the speech. Put it into the running order as its own beat
|
|
906
|
+
right after its cue line ends, play its own sound (the audience laughing),
|
|
907
|
+
keep it to about 1.5 s, then carry on with the speaker. Nobody talks during
|
|
908
|
+
it. When more lines follow the cue line in the same scene, cut that scene
|
|
909
|
+
where the cue line's last word ends in its `words.json` and put the reaction
|
|
910
|
+
between the two halves.
|
|
901
911
|
- **Overlays and SFX.** `overlays[]` and `audio.sfxCues[]` are cued the same
|
|
902
912
|
way, by `cueLineId`. Nothing renders them for you.
|
|
903
913
|
- **Captions.** Nothing is burned into any clip. Captions belong in post; the
|
|
@@ -914,12 +924,20 @@ the pieces:
|
|
|
914
924
|
holds its last frame for the difference. The narration is never sped up,
|
|
915
925
|
slowed down or cut short, so every scene plays at the speed it was recorded,
|
|
916
926
|
and the word times are handed on as they were measured.
|
|
927
|
+
- **It closes the dead air at the joins.** A clip that plays its own dialogue
|
|
928
|
+
keeps about 0.15 s after its last word and about 0.1 s before its first (0.25 s
|
|
929
|
+
when a different speaker answers), read from its `words.json`, so one clip
|
|
930
|
+
runs into the next like one conversation. It only ever shortens a clip's
|
|
931
|
+
silence and never cuts into a word. The ad's first lead-in and last tail, a
|
|
932
|
+
narrated scene, a still and a reaction beat are left as they are.
|
|
917
933
|
- **It ignores the continuous master.** It reads only `scene-NN.voice` files;
|
|
918
934
|
it never opens `narration.mp3` or `manifest.narration`. A tool that lays the
|
|
919
935
|
master down whole and cuts the pictures to it keeps the read intact.
|
|
920
|
-
- Otherwise: A-roll in manifest order as the spine, each
|
|
921
|
-
|
|
922
|
-
|
|
936
|
+
- Otherwise: A-roll in manifest order as the spine, each reaction put in as its
|
|
937
|
+
own beat of at most 1.5 s right after its cue line with its own sound, each
|
|
938
|
+
other cutaway laid over the spine at its cued word (falling back to a
|
|
939
|
+
proportional estimate when the scene has no word times, and saying which
|
|
940
|
+
method it used), a narrated scene's voice
|
|
923
941
|
replacing whatever the clip recorded, a dialogue scene keeping its performed
|
|
924
942
|
line, a clip-less scene becoming a still, everything normalized to 1080×1920
|
|
925
943
|
at 24 fps and −16 LUFS per segment, a generated room-tone bed under the whole
|
|
@@ -931,14 +949,15 @@ the pieces:
|
|
|
931
949
|
ad with its start and end) and `cut.srt` (the same words grouped into subtitle
|
|
932
950
|
lines), named from the `--out` basename. Those times are on the finished ad's
|
|
933
951
|
timeline, each scene's own word times offset by where that scene starts in the
|
|
934
|
-
cut and otherwise unaltered. Scenes with no word times leave
|
|
952
|
+
cut, less any lead-in trimmed at the join, and otherwise unaltered. Scenes with no word times leave
|
|
935
953
|
gaps and are named in the report; when no scene delivered any, neither file is
|
|
936
954
|
written. These sidecars are the script's own output — another tool will not
|
|
937
955
|
produce them.
|
|
938
956
|
|
|
939
957
|
Before uploading anything, from any tool, check what you made:
|
|
940
958
|
`ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 cut.mp4`
|
|
941
|
-
should be close to the sum of the scene durations you kept
|
|
959
|
+
should be close to the sum of the scene durations you kept (the bundled
|
|
960
|
+
script's cut runs somewhat shorter, by the silence it trimmed at the joins). A cut that is a few
|
|
942
961
|
seconds long when five clips went in means a filter dropped inputs; do not
|
|
943
962
|
upload it.
|
|
944
963
|
|
|
@@ -2,7 +2,10 @@
|
|
|
2
2
|
// A plain first cut from a folder written by `exodus video pull`: the A-roll
|
|
3
3
|
// scenes in the manifest's order make the spine, each cutaway lays over the
|
|
4
4
|
// spine at the moment its cued line is spoken, a room-tone bed and the music
|
|
5
|
-
// bed go underneath, and one MP4 comes out ready to upload.
|
|
5
|
+
// bed go underneath, and one MP4 comes out ready to upload. A reaction
|
|
6
|
+
// cutaway is the exception: it is its own short beat in the spine right after
|
|
7
|
+
// its cued line ends, heard with its own sound while the speaker is silent,
|
|
8
|
+
// and its sound fades out quietly under the next line rather than stopping.
|
|
6
9
|
//
|
|
7
10
|
// node first-cut.mjs <pulled-dir> [--out cut.mp4] [--skip 2,5] [--no-music]
|
|
8
11
|
//
|
|
@@ -11,9 +14,11 @@
|
|
|
11
14
|
// recorded: the voice replaces the clip's audio, and the picture is fitted to
|
|
12
15
|
// the voice, trimmed when the clip runs longer and held on its last frame when
|
|
13
16
|
// the voice runs longer. The voice itself is never altered. A dialogue scene
|
|
14
|
-
// keeps the audio it performed
|
|
15
|
-
//
|
|
16
|
-
//
|
|
17
|
+
// keeps the audio it performed; at each join its silence before the first word
|
|
18
|
+
// and after the last is cut back to a short beat, read from its word times. A
|
|
19
|
+
// scene with no clip but a keyframe becomes a still for the length of its voice
|
|
20
|
+
// track, else the storyboard's planned duration. Every segment is
|
|
21
|
+
// loudness-normalized to -16 LUFS before the join.
|
|
17
22
|
// --skip drops scenes.
|
|
18
23
|
//
|
|
19
24
|
// After the render it writes the whole ad's word times beside the MP4, as
|
|
@@ -30,15 +35,46 @@ const FPS = 24;
|
|
|
30
35
|
const norm = (label) =>
|
|
31
36
|
`[${label}]scale=${WIDTH}:${HEIGHT}:force_original_aspect_ratio=decrease,` +
|
|
32
37
|
`pad=${WIDTH}:${HEIGHT}:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=${FPS},format=yuv420p`;
|
|
38
|
+
/**
|
|
39
|
+
* Edges the cut chooses (join trims, reaction splits, narration lengths) land
|
|
40
|
+
* on whole frames, so the picture runs as long as its sound there and the
|
|
41
|
+
* timeline's times are the cut's times.
|
|
42
|
+
*/
|
|
43
|
+
const floorFrame = (t) => Math.floor(t * FPS + 1e-6) / FPS;
|
|
44
|
+
const ceilFrame = (t) => Math.ceil(t * FPS - 1e-6) / FPS;
|
|
33
45
|
const stereo = "aresample=48000,aformat=channel_layouts=stereo";
|
|
34
|
-
/**
|
|
35
|
-
|
|
46
|
+
/**
|
|
47
|
+
* Per-segment loudness, the level every clip was normalized to before the join.
|
|
48
|
+
* loudnorm stamps its output about 0.1 s late (its look-ahead), so a later
|
|
49
|
+
* atrim by time kept that much less sound than picture and every join pulled
|
|
50
|
+
* the sound further ahead (#2387). Re-stamping by sample count undoes it.
|
|
51
|
+
*/
|
|
52
|
+
const LOUDNORM = "loudnorm=I=-16:TP=-1.5:LRA=11,aresample=48000,asetpts=N/SR/TB";
|
|
53
|
+
/** A fade this short at each edge of a segment's sound stops a hard cut in the room tone clicking. */
|
|
54
|
+
const EDGE_FADE_SEC = 0.015;
|
|
36
55
|
/** Air after the last spoken word of a narrated scene, so the line lands. */
|
|
37
56
|
export const TAIL_AIR_SEC = 0.8;
|
|
38
57
|
/** A continuous low bed under the whole ad, so the joins between clips do not read as dead air. */
|
|
39
58
|
const ROOM_TONE =
|
|
40
59
|
"anoisesrc=color=brown:amplitude=0.0035:r=48000,highpass=f=60,lowpass=f=900,aformat=channel_layouts=stereo";
|
|
41
60
|
const MUSIC_VOLUME = 0.18;
|
|
61
|
+
/** The longest a reaction beat holds the spine (#2357); older pulls carry 4 s reaction clips. */
|
|
62
|
+
export const REACTION_MAX_SEC = 1.5;
|
|
63
|
+
/**
|
|
64
|
+
* A reaction's laugh carries into the next line instead of stopping dead at
|
|
65
|
+
* the join (#2406): it starts well under the speaker, at this share of its own
|
|
66
|
+
* level (about -9 dB), and fades to silence over this long.
|
|
67
|
+
*/
|
|
68
|
+
export const REACTION_TAIL_SEC = 0.75;
|
|
69
|
+
export const REACTION_TAIL_GAIN = 0.35;
|
|
70
|
+
/**
|
|
71
|
+
* The air a join keeps around its words (#2388): after one clip's last word, and
|
|
72
|
+
* before the next clip's first, less when the same speaker carries on than when
|
|
73
|
+
* someone new answers.
|
|
74
|
+
*/
|
|
75
|
+
export const JOIN_TAIL_SEC = 0.15;
|
|
76
|
+
export const JOIN_LEAD_SAME_SPEAKER_SEC = 0.1;
|
|
77
|
+
export const JOIN_LEAD_NEW_SPEAKER_SEC = 0.25;
|
|
42
78
|
|
|
43
79
|
/** The word rule the planner sized every line with (scout models.ts wordCount). */
|
|
44
80
|
export function wordCount(text) {
|
|
@@ -77,11 +113,17 @@ export function scenePlans(storyboard) {
|
|
|
77
113
|
const plans = new Map();
|
|
78
114
|
for (const scene of storyboard?.scenes ?? []) {
|
|
79
115
|
if (typeof scene?.sceneIndex !== "number") continue;
|
|
116
|
+
const speakers = (Array.isArray(scene.dialogue) ? scene.dialogue : [])
|
|
117
|
+
.filter((turn) => hasText(turn?.line) || hasText(turn?.text))
|
|
118
|
+
.map((turn) => (hasText(turn.speaker) ? turn.speaker.trim() : null));
|
|
80
119
|
plans.set(scene.sceneIndex, {
|
|
120
|
+
firstSpeaker: speakers[0] ?? null,
|
|
121
|
+
lastSpeaker: speakers.at(-1) ?? null,
|
|
81
122
|
cutaway: CUTAWAY_KINDS.has(scene.kind),
|
|
82
123
|
narrated: narratedScene(scene, storyboard),
|
|
83
124
|
lineIds: Array.isArray(scene.lineIds) ? scene.lineIds : [],
|
|
84
125
|
cueLineId: typeof scene.cueLineId === "string" ? scene.cueLineId : null,
|
|
126
|
+
cutawayType: typeof scene.cutawayType === "string" ? scene.cutawayType : null,
|
|
85
127
|
durationSec: typeof scene.durationSec === "number" ? scene.durationSec : null,
|
|
86
128
|
});
|
|
87
129
|
}
|
|
@@ -90,12 +132,12 @@ export function scenePlans(storyboard) {
|
|
|
90
132
|
|
|
91
133
|
/**
|
|
92
134
|
* Fit a picture to its voice track. The segment always runs as long as the
|
|
93
|
-
* voice plus a beat of air, because the voice is never
|
|
94
|
-
* runs longer is trimmed; a clip that runs shorter holds
|
|
95
|
-
* difference. A still (no clip) simply runs the voice's length.
|
|
135
|
+
* voice plus a beat of air, to the next whole frame, because the voice is never
|
|
136
|
+
* altered. A clip that runs longer is trimmed; a clip that runs shorter holds
|
|
137
|
+
* its last frame for the difference. A still (no clip) simply runs the voice's length.
|
|
96
138
|
*/
|
|
97
139
|
export function fitNarration({ clipSeconds, voiceSeconds }) {
|
|
98
|
-
const target = voiceSeconds + TAIL_AIR_SEC;
|
|
140
|
+
const target = ceilFrame(voiceSeconds + TAIL_AIR_SEC);
|
|
99
141
|
if (clipSeconds === null) return { seconds: target, holdSec: 0, trimmedBy: 0 };
|
|
100
142
|
if (target <= clipSeconds) {
|
|
101
143
|
return { seconds: target, holdSec: 0, trimmedBy: clipSeconds - target };
|
|
@@ -126,6 +168,121 @@ export function cueOffsetOnSpine(spine, counts, cueLineId) {
|
|
|
126
168
|
return null;
|
|
127
169
|
}
|
|
128
170
|
|
|
171
|
+
/**
|
|
172
|
+
* Where a segment's spoken cue line ends, in seconds into the segment, or null
|
|
173
|
+
* when the segment does not speak it. A cue line that closes the segment ends
|
|
174
|
+
* where the segment does.
|
|
175
|
+
*/
|
|
176
|
+
export function cueEndInSegment(seg, counts, cueLineId) {
|
|
177
|
+
const idx = seg.lineIds.indexOf(cueLineId);
|
|
178
|
+
if (idx === -1) return null;
|
|
179
|
+
if (idx === seg.lineIds.length - 1) return seg.seconds;
|
|
180
|
+
let through = 0;
|
|
181
|
+
let totalWords = 0;
|
|
182
|
+
seg.lineIds.forEach((id, i) => {
|
|
183
|
+
const n = counts.get(id) ?? 0;
|
|
184
|
+
totalWords += n;
|
|
185
|
+
if (i <= idx) through += n;
|
|
186
|
+
});
|
|
187
|
+
if (seg.words && seg.words.length > 0) {
|
|
188
|
+
const last = seg.words[Math.min(Math.max(through - 1, 0), seg.words.length - 1)];
|
|
189
|
+
return Math.min(Math.max(0, last?.e ?? 0), seg.seconds);
|
|
190
|
+
}
|
|
191
|
+
return totalWords > 0 ? (through / totalWords) * seg.seconds : seg.seconds;
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
/**
|
|
195
|
+
* Cut the dead air out of each join between clips that play their own sound: a
|
|
196
|
+
* clip's lead-in silence stacked on the previous clip's tail ran past a second
|
|
197
|
+
* (#2388). Each side comes back to a beat that reads as one conversation, a
|
|
198
|
+
* longer one when someone new answers. Only ever shortens, never into a word,
|
|
199
|
+
* and leaves the ad's first lead-in and last tail alone; a clip whose closing
|
|
200
|
+
* line cues a reaction is not the ad's end, since the beat follows it. Trims
|
|
201
|
+
* land on whole frames so the picture and its sound stay the same length.
|
|
202
|
+
*/
|
|
203
|
+
function trimJoins(spine, reactionCues) {
|
|
204
|
+
spine.forEach((seg, k) => {
|
|
205
|
+
if (seg.source !== "clip" || seg.audio !== "clip" || !seg.words?.length) return;
|
|
206
|
+
const prev = spine[k - 1];
|
|
207
|
+
const sameSpeaker = Boolean(prev?.lastSpeaker) && prev.lastSpeaker === seg.firstSpeaker;
|
|
208
|
+
const lead = sameSpeaker ? JOIN_LEAD_SAME_SPEAKER_SEC : JOIN_LEAD_NEW_SPEAKER_SEC;
|
|
209
|
+
const firstWord = Math.min(...seg.words.map((w) => w.s));
|
|
210
|
+
const lastWord = Math.max(...seg.words.map((w) => w.e));
|
|
211
|
+
const inSec = prev ? Math.max(0, floorFrame(firstWord - lead)) : 0;
|
|
212
|
+
const endsAd = k === spine.length - 1 && !reactionCues.has(seg.lineIds.at(-1));
|
|
213
|
+
const outSec = endsAd ? seg.seconds : Math.min(seg.seconds, ceilFrame(lastWord + JOIN_TAIL_SEC));
|
|
214
|
+
if (outSec <= inSec) return;
|
|
215
|
+
const how = [];
|
|
216
|
+
if (inSec > 0) how.push(`lead-in trimmed ${inSec.toFixed(2)}s`);
|
|
217
|
+
if (outSec < seg.seconds) how.push(`tail trimmed ${(seg.seconds - outSec).toFixed(2)}s`);
|
|
218
|
+
if (how.length === 0) return;
|
|
219
|
+
seg.inSec = inSec;
|
|
220
|
+
seg.seconds = outSec - inSec;
|
|
221
|
+
seg.words = seg.words.map((w) => ({ ...w, s: w.s - inSec, e: w.e - inSec }));
|
|
222
|
+
seg.note = `${seg.note} (${how.join(", ")})`;
|
|
223
|
+
});
|
|
224
|
+
}
|
|
225
|
+
|
|
226
|
+
const holdFor = (seg) =>
|
|
227
|
+
seg.source === "clip" ? Math.max(0, seg.seconds - Math.max(0, seg.clipSeconds - seg.inSec)) : 0;
|
|
228
|
+
|
|
229
|
+
/**
|
|
230
|
+
* Put a reaction beat into the spine right after its cue line ends, splitting
|
|
231
|
+
* the segment that speaks it when more lines follow in that segment. Returns
|
|
232
|
+
* the scene that speaks the cue, or null when no segment does.
|
|
233
|
+
*/
|
|
234
|
+
function placeReaction(spine, counts, reaction) {
|
|
235
|
+
for (let k = 0; k < spine.length; k++) {
|
|
236
|
+
const seg = spine[k];
|
|
237
|
+
if (seg.reaction) continue;
|
|
238
|
+
const cueEnd = cueEndInSegment(seg, counts, reaction.cueLineId);
|
|
239
|
+
if (cueEnd === null) continue;
|
|
240
|
+
const cut = Math.min(seg.seconds, ceilFrame(cueEnd));
|
|
241
|
+
const beat = {
|
|
242
|
+
sceneIndex: reaction.sceneIndex, source: reaction.source, file: reaction.file,
|
|
243
|
+
clipSeconds: reaction.source === "clip" ? reaction.seconds : null, inSec: 0,
|
|
244
|
+
seconds: Math.min(reaction.seconds, REACTION_MAX_SEC), holdSec: 0,
|
|
245
|
+
audio: reaction.hasAudio ? "clip" : "silence", voice: null, words: null, lineIds: [],
|
|
246
|
+
reaction: { cueLineId: reaction.cueLineId, cueSceneIndex: seg.sceneIndex },
|
|
247
|
+
};
|
|
248
|
+
if (cut >= seg.seconds - 0.05) {
|
|
249
|
+
spine.splice(k + 1, 0, beat);
|
|
250
|
+
return seg.sceneIndex;
|
|
251
|
+
}
|
|
252
|
+
const idx = seg.lineIds.indexOf(reaction.cueLineId);
|
|
253
|
+
const head = {
|
|
254
|
+
...seg, seconds: cut, lineIds: seg.lineIds.slice(0, idx + 1),
|
|
255
|
+
words: seg.words ? seg.words.filter((w) => w.s < cut) : null,
|
|
256
|
+
note: `${seg.note}, until ${reaction.cueLineId} ends at ${cut.toFixed(2)}s`,
|
|
257
|
+
};
|
|
258
|
+
const tail = {
|
|
259
|
+
...seg, inSec: seg.inSec + cut, seconds: seg.seconds - cut, lineIds: seg.lineIds.slice(idx + 1),
|
|
260
|
+
words: seg.words
|
|
261
|
+
? seg.words.filter((w) => w.s >= cut).map((w) => ({ ...w, s: w.s - cut, e: w.e - cut }))
|
|
262
|
+
: null,
|
|
263
|
+
note: `${seg.note}, resumed at ${cut.toFixed(2)}s after reaction ${reaction.sceneIndex}`,
|
|
264
|
+
};
|
|
265
|
+
head.holdSec = holdFor(head);
|
|
266
|
+
tail.holdSec = holdFor(tail);
|
|
267
|
+
spine.splice(k, 1, head, beat, tail);
|
|
268
|
+
return seg.sceneIndex;
|
|
269
|
+
}
|
|
270
|
+
return null;
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
/**
|
|
274
|
+
* The sound a reaction beat carries under the next segment (#2406): the clip's
|
|
275
|
+
* own sound past the beat when it has some (older 4 s pulls), else its last
|
|
276
|
+
* stretch replayed, so a clip exactly as long as its beat still fades out.
|
|
277
|
+
*/
|
|
278
|
+
function reactionTail(seg, next) {
|
|
279
|
+
const clipLen = seg.clipSeconds ?? seg.seconds;
|
|
280
|
+
const beatEnd = seg.inSec + seg.seconds;
|
|
281
|
+
const fromSec = Math.max(0, Math.min(beatEnd, clipLen - REACTION_TAIL_SEC));
|
|
282
|
+
const seconds = Math.min(REACTION_TAIL_SEC, next.seconds, clipLen - fromSec);
|
|
283
|
+
return { atSec: next.startSec, fromSec, seconds, gain: REACTION_TAIL_GAIN };
|
|
284
|
+
}
|
|
285
|
+
|
|
129
286
|
export function buildTimeline({ manifest, storyboard, skip, probe, exists, readWords }) {
|
|
130
287
|
const plans = scenePlans(storyboard);
|
|
131
288
|
const counts = lineWordCounts(storyboard);
|
|
@@ -133,11 +290,13 @@ export function buildTimeline({ manifest, storyboard, skip, probe, exists, readW
|
|
|
133
290
|
const cutaways = [];
|
|
134
291
|
const waiting = [];
|
|
135
292
|
const dropped = [];
|
|
136
|
-
let startSec = 0;
|
|
137
293
|
|
|
138
294
|
for (const scene of manifest?.scenes ?? []) {
|
|
139
295
|
const n = scene.sceneIndex;
|
|
140
|
-
const plan = plans.get(n) ?? {
|
|
296
|
+
const plan = plans.get(n) ?? {
|
|
297
|
+
cutaway: false, lineIds: [], cueLineId: null, cutawayType: null, durationSec: null,
|
|
298
|
+
firstSpeaker: null, lastSpeaker: null,
|
|
299
|
+
};
|
|
141
300
|
if (skip.has(n)) {
|
|
142
301
|
dropped.push({ n, reason: `scene ${n}: skipped by --skip` });
|
|
143
302
|
continue;
|
|
@@ -145,15 +304,17 @@ export function buildTimeline({ manifest, storyboard, skip, probe, exists, readW
|
|
|
145
304
|
const nothing = `nothing to cut with (clip ${scene.clipStatus}${scene.error ? `: ${scene.error}` : ""})`;
|
|
146
305
|
|
|
147
306
|
if (plan.cutaway) {
|
|
307
|
+
const cue = { cueLineId: plan.cueLineId, reaction: plan.cutawayType === "reaction" };
|
|
148
308
|
if (exists(scene.clip)) {
|
|
309
|
+
const clip = probe(scene.clip);
|
|
149
310
|
waiting.push({
|
|
150
311
|
sceneIndex: n, source: "clip", file: scene.clip,
|
|
151
|
-
seconds:
|
|
312
|
+
seconds: clip.seconds, hasAudio: clip.hasAudio, ...cue,
|
|
152
313
|
});
|
|
153
314
|
} else if (exists(scene.keyframe)) {
|
|
154
315
|
waiting.push({
|
|
155
316
|
sceneIndex: n, source: "still", file: scene.keyframe,
|
|
156
|
-
seconds: plan.durationSec ?? 4,
|
|
317
|
+
seconds: plan.durationSec ?? 4, hasAudio: false, ...cue,
|
|
157
318
|
});
|
|
158
319
|
} else {
|
|
159
320
|
dropped.push({ n, reason: `cutaway ${n}: ${nothing}` });
|
|
@@ -168,8 +329,8 @@ export function buildTimeline({ manifest, storyboard, skip, probe, exists, readW
|
|
|
168
329
|
if (exists(scene.clip)) {
|
|
169
330
|
const clip = probe(scene.clip);
|
|
170
331
|
const base = {
|
|
171
|
-
sceneIndex: n, source: "clip", file: scene.clip, clipSeconds: clip.seconds,
|
|
172
|
-
|
|
332
|
+
sceneIndex: n, source: "clip", file: scene.clip, clipSeconds: clip.seconds, inSec: 0,
|
|
333
|
+
lineIds: plan.lineIds, firstSpeaker: plan.firstSpeaker, lastSpeaker: plan.lastSpeaker,
|
|
173
334
|
};
|
|
174
335
|
if (voice && (narrated || !clip.hasAudio)) {
|
|
175
336
|
const fit = fitNarration({ clipSeconds: clip.seconds, voiceSeconds: probe(voice).seconds });
|
|
@@ -198,7 +359,6 @@ export function buildTimeline({ manifest, storyboard, skip, probe, exists, readW
|
|
|
198
359
|
words: null, note: `${flag}, silent clip`,
|
|
199
360
|
});
|
|
200
361
|
}
|
|
201
|
-
startSec += spine[spine.length - 1].seconds;
|
|
202
362
|
continue;
|
|
203
363
|
}
|
|
204
364
|
if (exists(scene.keyframe)) {
|
|
@@ -206,39 +366,64 @@ export function buildTimeline({ manifest, storyboard, skip, probe, exists, readW
|
|
|
206
366
|
? fitNarration({ clipSeconds: null, voiceSeconds: probe(voice).seconds })
|
|
207
367
|
: { seconds: plan.durationSec ?? 4, holdSec: 0 };
|
|
208
368
|
spine.push({
|
|
209
|
-
sceneIndex: n, source: "still", file: scene.keyframe, clipSeconds: null,
|
|
369
|
+
sceneIndex: n, source: "still", file: scene.keyframe, clipSeconds: null, inSec: 0,
|
|
210
370
|
seconds: fit.seconds, audio: voice ? "narration" : "silence", voice, holdSec: 0,
|
|
211
|
-
|
|
371
|
+
lineIds: plan.lineIds,
|
|
212
372
|
// #1689: on a narrated scene the words are the voice track's, and the
|
|
213
373
|
// voice starts where the segment does, so they need no offset here.
|
|
214
374
|
words: voice ? readWords(scene.words) : null,
|
|
215
375
|
note: `${flag}${voice ? ` + ${voice}` : ", silent"} (${fit.seconds.toFixed(1)}s, clip ${scene.clipStatus})`,
|
|
216
376
|
});
|
|
217
|
-
startSec += fit.seconds;
|
|
218
377
|
continue;
|
|
219
378
|
}
|
|
220
379
|
dropped.push({ n, reason: `scene ${n}: ${nothing}` });
|
|
221
380
|
}
|
|
222
381
|
|
|
223
|
-
|
|
382
|
+
const noCue = (c) =>
|
|
383
|
+
c.cueLineId ? `${c.cueLineId} is spoken by no scene in the cut` : "the storyboard names no cue line";
|
|
384
|
+
|
|
385
|
+
// Joins are closed first so a reaction beat lands right after its cue line
|
|
386
|
+
// ends in the trimmed clip. Reactions change the spine, so they go in before
|
|
387
|
+
// any overlay is placed on it.
|
|
388
|
+
trimJoins(spine, new Set(waiting.filter((w) => w.reaction).map((w) => w.cueLineId)));
|
|
389
|
+
for (const c of waiting.filter((w) => w.reaction)) {
|
|
390
|
+
if (!c.cueLineId || placeReaction(spine, counts, c) === null) {
|
|
391
|
+
dropped.push({ n: c.sceneIndex, reason: `cutaway ${c.sceneIndex}: ${noCue(c)}` });
|
|
392
|
+
}
|
|
393
|
+
}
|
|
394
|
+
let total = 0;
|
|
395
|
+
for (const seg of spine) {
|
|
396
|
+
seg.startSec = total;
|
|
397
|
+
total += seg.seconds;
|
|
398
|
+
}
|
|
399
|
+
spine.forEach((seg, k) => {
|
|
400
|
+
const next = spine[k + 1];
|
|
401
|
+
if (!seg.reaction || seg.audio !== "clip" || !next) return;
|
|
402
|
+
seg.reaction.tail = reactionTail(seg, next);
|
|
403
|
+
});
|
|
404
|
+
|
|
405
|
+
for (const c of waiting.filter((w) => !w.reaction)) {
|
|
224
406
|
const cue = c.cueLineId ? cueOffsetOnSpine(spine, counts, c.cueLineId) : null;
|
|
225
407
|
if (!cue) {
|
|
226
|
-
|
|
227
|
-
? `${c.cueLineId} is spoken by no scene in the cut`
|
|
228
|
-
: "the storyboard names no cue line";
|
|
229
|
-
dropped.push({ n: c.sceneIndex, reason: `cutaway ${c.sceneIndex}: ${why}` });
|
|
408
|
+
dropped.push({ n: c.sceneIndex, reason: `cutaway ${c.sceneIndex}: ${noCue(c)}` });
|
|
230
409
|
continue;
|
|
231
410
|
}
|
|
232
411
|
cutaways.push({ ...c, startSec: cue.startSec, method: cue.method, spineSceneIndex: cue.spineSceneIndex });
|
|
233
412
|
}
|
|
234
413
|
cutaways.sort((a, b) => a.startSec - b.startSec);
|
|
414
|
+
const beats = spine.filter((seg) => seg.reaction);
|
|
235
415
|
for (let i = 0; i < cutaways.length; i++) {
|
|
236
416
|
const c = cutaways[i];
|
|
417
|
+
const stops = [];
|
|
237
418
|
const next = cutaways[i + 1];
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
419
|
+
if (next) stops.push({ at: next.startSec, by: `cutaway ${next.sceneIndex} starts` });
|
|
420
|
+
const beat = beats.find((b) => b.startSec >= c.startSec);
|
|
421
|
+
if (beat) stops.push({ at: beat.startSec, by: `reaction ${beat.sceneIndex} starts` });
|
|
422
|
+
stops.push({ at: total, by: "the ad ends" });
|
|
423
|
+
const stop = stops.reduce((a, b) => (b.at < a.at ? b : a));
|
|
424
|
+
if (c.startSec + c.seconds > stop.at) {
|
|
425
|
+
c.seconds = Math.max(0, stop.at - c.startSec);
|
|
426
|
+
c.cutShortBy = stop.by;
|
|
242
427
|
}
|
|
243
428
|
}
|
|
244
429
|
|
|
@@ -262,6 +447,7 @@ export function buildFfmpegArgs(timeline, { resolve, music, out }) {
|
|
|
262
447
|
};
|
|
263
448
|
const filters = [];
|
|
264
449
|
const pairs = [];
|
|
450
|
+
const tails = [];
|
|
265
451
|
|
|
266
452
|
for (const seg of timeline.spine) {
|
|
267
453
|
const i =
|
|
@@ -269,16 +455,42 @@ export function buildFfmpegArgs(timeline, { resolve, music, out }) {
|
|
|
269
455
|
? addInput("-i", resolve(seg.file))
|
|
270
456
|
: addInput("-loop", "1", "-t", String(seg.seconds), "-i", resolve(seg.file));
|
|
271
457
|
const fit = [];
|
|
272
|
-
|
|
273
|
-
|
|
458
|
+
const inSec = seg.inSec ?? 0;
|
|
459
|
+
if (seg.source === "clip" && seg.clipSeconds !== null) {
|
|
460
|
+
const shown = seg.seconds - seg.holdSec;
|
|
461
|
+
if (inSec > 0.001 || shown < seg.clipSeconds - 0.01) {
|
|
462
|
+
// By frame number: a time printed to the millisecond can round past the
|
|
463
|
+
// frame it names and drop it.
|
|
464
|
+
const from = Math.round(inSec * FPS);
|
|
465
|
+
fit.push(`trim=start_frame=${from}:end_frame=${Math.round((inSec + shown) * FPS)},setpts=PTS-STARTPTS`);
|
|
466
|
+
}
|
|
274
467
|
}
|
|
275
468
|
if (seg.holdSec > 0.01) fit.push(`tpad=stop_mode=clone:stop_duration=${sec(seg.holdSec)}`);
|
|
276
469
|
filters.push(`${norm(`${i}:v`)}${fit.length ? `,${fit.join(",")}` : ""}[v${i}]`);
|
|
470
|
+
// A segment split around a reaction beat starts partway into its sound too.
|
|
471
|
+
const skipIn = inSec > 0.001 ? `atrim=start=${sec(inSec)},asetpts=PTS-STARTPTS,` : "";
|
|
472
|
+
const fitLength =
|
|
473
|
+
`apad,atrim=0:${sec(seg.seconds)},afade=t=in:d=${EDGE_FADE_SEC},` +
|
|
474
|
+
`afade=t=out:st=${sec(seg.seconds - EDGE_FADE_SEC)}:d=${EDGE_FADE_SEC}`;
|
|
475
|
+
const fitSound = `${LOUDNORM},${fitLength}`;
|
|
277
476
|
if (seg.audio === "narration") {
|
|
278
477
|
const v = addInput("-i", resolve(seg.voice));
|
|
279
|
-
filters.push(`[${v}:a]${stereo},${
|
|
478
|
+
filters.push(`[${v}:a]${stereo},${skipIn}${fitSound}[a${i}]`);
|
|
479
|
+
} else if (seg.audio === "clip" && seg.reaction?.tail) {
|
|
480
|
+
// The tail is cut from the same loudness-normalized sound as the beat,
|
|
481
|
+
// ducked, faded out, and laid in at the join; the next line's own sound
|
|
482
|
+
// plays at full level over it (#2406).
|
|
483
|
+
const t = seg.reaction.tail;
|
|
484
|
+
filters.push(`[${i}:a]${stereo},${skipIn}${LOUDNORM},asplit=2[as${i}][at${i}]`);
|
|
485
|
+
filters.push(`[as${i}]${fitLength}[a${i}]`);
|
|
486
|
+
filters.push(
|
|
487
|
+
`[at${i}]atrim=start=${sec(t.fromSec - inSec)}:duration=${sec(t.seconds)},asetpts=PTS-STARTPTS,` +
|
|
488
|
+
`volume=${t.gain},afade=t=in:d=${EDGE_FADE_SEC},afade=t=out:st=0:d=${sec(t.seconds)},` +
|
|
489
|
+
`adelay=delays=${Math.round(t.atSec * 1000)}:all=1[tail${i}]`,
|
|
490
|
+
);
|
|
491
|
+
tails.push(`[tail${i}]`);
|
|
280
492
|
} else if (seg.audio === "clip") {
|
|
281
|
-
filters.push(`[${i}:a]${stereo},${
|
|
493
|
+
filters.push(`[${i}:a]${stereo},${skipIn}${fitSound}[a${i}]`);
|
|
282
494
|
} else {
|
|
283
495
|
filters.push(`anullsrc=r=48000:cl=stereo,atrim=0:${sec(seg.seconds)}[a${i}]`);
|
|
284
496
|
}
|
|
@@ -308,7 +520,7 @@ export function buildFfmpegArgs(timeline, { resolve, music, out }) {
|
|
|
308
520
|
});
|
|
309
521
|
|
|
310
522
|
filters.push(`${ROOM_TONE}[room]`);
|
|
311
|
-
const beds = ["[room]"];
|
|
523
|
+
const beds = ["[room]", ...tails];
|
|
312
524
|
if (music) {
|
|
313
525
|
const m = addInput("-stream_loop", "-1", "-i", music);
|
|
314
526
|
filters.push(`[${m}:a]${stereo},volume=${MUSIC_VOLUME}[bed]`);
|
|
@@ -398,6 +610,20 @@ export function inTheCut(timeline) {
|
|
|
398
610
|
const rows = [];
|
|
399
611
|
for (const seg of timeline.spine) {
|
|
400
612
|
const label = seg.source === "clip" ? seg.file : `still ${seg.file}`;
|
|
613
|
+
if (seg.reaction) {
|
|
614
|
+
const sound = seg.audio === "clip" ? "its own sound" : "silent: the clip has no sound";
|
|
615
|
+
const tail = seg.reaction.tail;
|
|
616
|
+
const fades = tail
|
|
617
|
+
? `, fading out under the next line at ${tail.atSec.toFixed(2)}s over ${tail.seconds.toFixed(2)}s`
|
|
618
|
+
: "";
|
|
619
|
+
rows.push({
|
|
620
|
+
n: seg.sceneIndex,
|
|
621
|
+
line:
|
|
622
|
+
`reaction ${seg.sceneIndex}: ${label} for ${seg.seconds.toFixed(1)}s after ` +
|
|
623
|
+
`${seg.reaction.cueLineId} in scene ${seg.reaction.cueSceneIndex}, ${sound}${fades}`,
|
|
624
|
+
});
|
|
625
|
+
continue;
|
|
626
|
+
}
|
|
401
627
|
rows.push({ n: seg.sceneIndex, line: `scene ${seg.sceneIndex}: ${label}${seg.note}` });
|
|
402
628
|
}
|
|
403
629
|
for (const c of timeline.cutaways) {
|
|
@@ -420,7 +646,7 @@ export function inTheCut(timeline) {
|
|
|
420
646
|
export function noWordTimes(timeline) {
|
|
421
647
|
const rows = timeline.spine
|
|
422
648
|
.filter((seg) => !seg.words || seg.words.length === 0)
|
|
423
|
-
.map((seg) => `scene ${seg.sceneIndex} (no words.json)`);
|
|
649
|
+
.map((seg) => `scene ${seg.sceneIndex} (${seg.reaction ? "reaction" : "no words.json"})`);
|
|
424
650
|
for (const c of timeline.cutaways) rows.push(`scene ${c.sceneIndex} (cutaway)`);
|
|
425
651
|
return rows;
|
|
426
652
|
}
|
|
@@ -478,11 +704,14 @@ function main() {
|
|
|
478
704
|
};
|
|
479
705
|
const probe = (file) => {
|
|
480
706
|
const raw = execFileSync("ffprobe", [
|
|
481
|
-
"-v", "error", "-show_entries", "stream=codec_type:format=duration", "-of", "json", here(file),
|
|
707
|
+
"-v", "error", "-show_entries", "stream=codec_type,duration:stream_disposition=attached_pic:format=duration", "-of", "json", here(file),
|
|
482
708
|
]).toString();
|
|
483
709
|
const info = JSON.parse(raw);
|
|
710
|
+
// A clip's container runs a few ms past its picture on the audio's padding;
|
|
711
|
+
// the picture is what the cut shows, so its length is the clip's length.
|
|
712
|
+
const picture = Number((info.streams ?? []).find((s) => s.codec_type === "video" && !s.disposition?.attached_pic)?.duration);
|
|
484
713
|
return {
|
|
485
|
-
seconds: Number(info.format?.duration ?? 0),
|
|
714
|
+
seconds: picture > 0 ? picture : Number(info.format?.duration ?? 0),
|
|
486
715
|
hasAudio: (info.streams ?? []).some((s) => s.codec_type === "audio"),
|
|
487
716
|
};
|
|
488
717
|
};
|