musicfit 0.1.1 → 0.2.0

package/README.md

@@ -1,6 +1,6 @@
 # musicfit
 
-Fit a music track to a video cut. musicfit makes the track exactly as long as the video by cutting or looping whole bars, lands its drop on the second you pick, ends it on a beat, ducks it under a voice and sets its loudness.
+Fit a music track to a video cut. musicfit makes the track exactly as long as the video by cutting or looping whole bars, lands its drop on the second you pick, ends it on a beat, ducks it under a voice and sets its loudness. Give it the video too and it lays the track under it.
 
 ```bash
 npx musicfit@latest track.mp3 --length 32 --hit 27.5 --fps 30
@@ -10,7 +10,7 @@ npx musicfit@latest track.mp3 --length 32 --hit 27.5 --fps 30
 
 AI music models ignore the length you ask for. Ask for 30 seconds and you may get 47, or two minutes, and the drop the edit was built around comes whenever it comes. So the track gets cut at the video's last frame and stops in the middle of a bar, or it fades out on no beat in particular, and the big hit lands a second after the logo. Fixing that by hand means finding the bar lines in a waveform, cutting on them, nudging crossfades and doing the arithmetic for where the drop has to go. musicfit does it the way a music editor would: it finds the beats and the bar lines, cuts or loops whole bars where they sound alike, keeps the start and the ending of the track, and tells you which seconds to listen to.
 
-The track can come from any music model, and [Wiro](https://wiro.ai) has several.
+The track can come from any music model, and [Wiro](https://wiro.ai/explore/music-generation) has several.
 
 ## What you get
 
@@ -18,6 +18,7 @@ The track can come from any music model, and [Wiro](https://wiro.ai) has several
 musicfit-track/
   fitted.wav     the track, exactly --length long, 48 kHz stereo, at -14 LUFS
   mix.wav        with --voice: the voice over the ducked music
+  video.mp4      with --video: the video with the track under it, its picture as it was
   cues.json      every beat and bar, the hit, the seams and the fade, in seconds and frames
   seams.png      the track, the result and what went where, like the picture above
   FIT.md         what was done and where to listen, for people and agents
@@ -25,8 +26,9 @@ musicfit-track/
 
 - Cuts and loops are made only on bar lines, in whole bars, between bars that sound alike, and keep the track's four-bar phrases whole where they can. Each seam is a 30 ms crossfade that ends just before the downbeat, so the new bar's first hit plays whole, and the hits on either side are lined up to the millisecond.
 - The drop lands on `--hit` to the sample, on the start of its attack. Nothing that comes after the drop in the track plays before it, so it is heard first where you asked.
-- The track starts the way it starts, with its own pickup, and ends on its own last hit when it has one and there is room. Otherwise it ends with a fade that starts on a beat and ends on one.
+- The track starts the way it starts, with its own pickup, and ends on its own last hit when it has one and there is room. Otherwise it ends with a fade that starts on a beat and ends on one, and when the music ends before the fit does, FIT.md says how many seconds are silent. A fit too short for the track's opening and its ending both can start later in the track, on a bar, so without `--hit` it can be the track's last bars alone; FIT.md says where it starts.
 - `--voice` lowers the music 10 dB (`--duck`) wherever the voice speaks, starting 0.15 s before each phrase and coming back 0.5 s after.
+- `--video` lays the result under the video: see [Laying it under a video](#laying-it-under-a-video).
 - Loudness is measured with ffmpeg's EBU R128 meter and set with one gain; a true-peak limiter holds the peaks under -1 dBTP only when that gain would push them over.
 - `cues.json` lists `beats`, `bars` and `hits` (the drop and the track's own last hit) as frame numbers at `--fps`, ready for a Remotion `<Sequence from={cues.bars[8]}>`, and the same moments in seconds under `times`. The seams, sections, fade and loudness are in it too:
 
@@ -46,6 +48,46 @@ musicfit-track/
 
 The lists are cut short here; the file has every beat and bar.
 
+`--analyze` reads a track without fitting it and saves its `cues.json`, `seams.png` and `FIT.md` in `musicfit-track.analysis/` by default. No fit's folder can have that name, so reading a track never replaces a fit, unless both are given the same `--out`: an analysis that ends well in a fit's folder replaces that fit, its `fitted.wav`, `mix.wav` and `video.mp4` too, and says so.
+
+## Laying it under a video
+
+Give `--video` the video the track is for, and musicfit also writes `video.mp4`: that video's picture, copied as it is and never encoded again, with the fitted track under it at the loudness target. The video has to be `--length` long to a frame, counted from its first frame, and without `--fps` the frames count at its own frame rate.
+
+`--video-sound` says what becomes of the video's own sound:
+
+- `keep`, for sound effects. Each place where it plays is set 3 dB louder than the music there, and no quieter than the loudness target, so a place over quiet music or none plays at the target. It goes no louder than keeps its true peaks, with the music's, under -1 dBTP, so the limiter takes nothing off the music. Where that leaves a place short of 3 dB over the music, as a sharp sound or one on a loud drop can be, the music dips under that place alone by what is missing, `--duck` (10 dB) at most. A note names a place that stays short even so.
+- `voice`, for speech. It is set to the loudness target where it speaks, and the music dips `--duck` (10 dB) under it, in `fitted.wav` and `mix.wav` too, as under `--voice`.
+- `drop`, for music of its own. It is left out.
+
+With `keep` the music plays as in `fitted.wav` away from the places it dips under, and as it dips, `video.mp4` can come out under the target, a little at the default and more at a loud `--lufs`; a note says so when it is more than half a LU short. In a 29 s launch film whose own sound is five sound effects, measured in `video.mp4`, the logo sound on the drop came out 3.4 dB over the music with the music 7.7 dB down under it, another sound 2.9 dB over with the music 5.9 dB down, and the other three 2.5 to 2.9 dB over with the music 1.1 dB down under one of them; the film came out at -14.4 LUFS, and at -12.1 LUFS with `--lufs -11`.
+
+With `--voice` as well, the music dips under the voice as it always does and each place is set over that music, so a place the voice speaks over plays at the loudness target, as loud as the voice, and the music needs no further dip there. The voice's peaks hold no place back: where a place meets the voice a limiter holds the two under -1 dBTP, and takes a little off both for those moments, so `video.mp4` reaches the target. With a voice over that same film, the two sounds it speaks over all the way through came out 10.8 and 8.6 dB over the music as it dips, and the film at -14 LUFS.
+
+musicfit cannot tell speech from sound effects or from music, so a video with more than a faint room tone of its own is refused until you choose. Of several sound streams it reads only the first, and a note says how many the video has. When the first is silent or nearly so and a later one is not, the run is refused, with `keep` and `voice` too, as that sound would be left out: give a copy with that sound first, or `--video-sound drop` to go on without it. With `drop` no stream is read and no note comes. A voice the video already holds goes with `--video-sound voice`, not as a copy given to `--voice`, which would lay it in twice; when the video itself is the track or the `--voice` file, its sound is laid in once. `--help` says how loud each part is set, and FIT.md how `video.mp4` came out after the AAC encoding. `video.mp4` carries a comment that says musicfit made it, and names no file.
+
+A video is refused before any work when it:
+
+- cannot be read, or is cut short or damaged
+- has no picture, or only a still one
+- holds a picture an MP4 cannot take without encoding it again (ProRes, DNxHD, VP8): give an H.264 or HEVC copy
+- is not `--length` long, and the message gives the length to use
+- is a `video.mp4` musicfit made, unless `--video-sound drop` leaves out its sound and the fitted track in it
+
+## What a failed run leaves
+
+A run makes its files in a hidden `.musicfit-part-` folder, inside the out folder, or next to it when the out folder is not there yet, and puts them in place only once it has ended well. A run that fails, is refused or is stopped with Ctrl-C, Ctrl-\ or a signal (SIGTERM, SIGHUP, SIGUSR2, SIGALRM or SIGXCPU) leaves the out folder as it was: the earlier run's files stay and nothing is added. As the earlier files stay until the new ones are in place, a run needs room on the disk for both.
+
+A run that ends well replaces the earlier run's files and removes those it does not make again, such as the `mix.wav` of a fit with a voice, and a note says which it removed. Two runs that end at the same moment in one folder take turns, and a run that starts while another moves its files in waits for it.
+
+The earlier run's files are the ones its `cues.json` lists, while they are as it wrote them: `cues.json` records the size and the SHA-256 of each file it lists, and its own SHA-256, so a file changed since, `cues.json` included, is no longer the earlier run's. A folder musicfit 0.1 wrote has no such record; there a file its `cues.json` lists counts as musicfit's while it looks as 0.1 left it (`fitted.wav` and `mix.wav` as long as the fit, `seams.png` and `FIT.md` as musicfit makes them) and was not changed more than a minute before or after `cues.json`.
+
+Any other file under one of musicfit's names (`fitted.wav`, `mix.wav`, `video.mp4`, `cues.json`, `seams.png`, `FIT.md`) is taken for yours: a run that would write over it is refused before any work, and one that would not leaves it alone. A run whose track, voice or video is one of the files it would replace or remove is refused before any work too.
+
+A run killed outright (`kill -9`, or the system when memory runs out) cannot tidy up, and its hidden folder stays. A later run that ends well in the same place removes it once the killed run is gone and nothing in it has changed for 15 minutes; one it may not look into stays, and does not fail that run. A run killed in the moment it moved its files in also leaves some of its new files in the out folder, the earlier ones in its hidden folder and a `.musicfit-commit` file: the next run in that folder puts the earlier files back before anything else, and no other run removes that hidden folder. A failed or stopped run that cannot remove its own hidden folder names it in its message, and that folder goes the same way.
+
+When another program holds a file open so that it cannot be moved, the run says so, takes its new files out again and puts the earlier ones back. An earlier file it cannot put back waits in the folder the message names, with a note of where each file belongs; no later run removes that folder.
+
 ## Use it with your agent
 
 musicfit ships as an [Agent Skill](https://agentskills.io). Install it once and your agent runs it by itself when a video needs music that fits:
@@ -82,20 +124,42 @@ npx musicfit@latest <track> --length <time> [options]
   --length <time>   how long the result is, e.g. 32, 32s, 960f or 1:05
   --hit <time>      where the drop lands in the result
   --drop <time>     where the drop is in the track (default: found)
-  --fps <n>         frame rate for frame numbers, e.g. 30, 25, 29.97 (default: 30)
+  --fps <n>         frame rate for frame numbers, e.g. 30, 25, 29.97
+                    (default: the video's with --video, else 30)
   --voice <file>    a voice track on the video's timeline; the music dips under it
-  --duck <dB>       how far the music dips under the voice (default: 10)
+  --duck <dB>       how far the music dips under the voice (default: 10), and
+                    the most it dips under the video's own sound with keep
+  --video <file>    a video to lay the result under: also writes video.mp4,
+                    the video's picture as it was with this sound under it.
+                    The video has to be --length long, to a frame
+  --video-sound <how>
+                    what becomes of the video's own sound: keep (sound
+                    effects: each place it plays set 3 dB louder than the
+                    music there, between --lufs and 6 dB over it, and no
+                    louder than keeps its true peaks, with the music's,
+                    under -1 dBTP; where that leaves it less than 3 dB over
+                    the music, the music dips under that place alone by as
+                    much, --duck at most. With --voice the places are set
+                    over the music as it dips under the voice, and where
+                    one meets the voice a limiter holds the two under
+                    -1 dBTP), voice (speech: set to --lufs where it plays,
+                    and the music dips --duck under it) or drop (music of
+                    its own: left out). A faint room tone, or a voice with
+                    no speech found, is only turned down to --lufs if it is
+                    louder. Without it, a video with more than a faint room
+                    tone of its own is refused until you choose
   --lufs <n>        loudness target in LUFS (default: -14)
   --end <kind>      auto, natural (the track's own ending) or fade (default: auto)
   --fade <s>        length of the closing fade, in whole beats (default: a bar)
   --bpm <n>         the tempo, when the one found is half or double the real one
   --meter <n>       beats to a bar, 3 or 4 (default: found)
-  --out <dir>       where to save (default: ./musicfit-<track name>)
+  --out <dir>       where to save (default: ./musicfit-<track name>, or
+                    ./musicfit-<track name>.analysis with --analyze)
   --analyze         only read the track: its beats, bars, drop and ending
   --json            print cues.json instead of a summary
 ```
 
-Times are seconds unless they end in `f`, which counts frames at `--fps`. The track can be any file ffmpeg reads, the sound of a video included.
+Times are seconds unless they end in `f`, which counts frames at `--fps`, or with `--video` and no `--fps` at the video's frame rate. The track can be any file ffmpeg reads, the sound of a video included. musicfit reads files, not addresses: a track, a voice or a video given as `https://...` is refused, so download it first.
 
 Some runs:
 
@@ -103,12 +167,14 @@ Some runs:
 npx musicfit@latest track.mp3 --length 15
 npx musicfit@latest track.mp3 --length 900f --hit 450f --fps 30 --out public/music
 npx musicfit@latest track.mp3 --length 1:30 --hit 41 --voice vo.wav --lufs -16
+npx musicfit@latest track.mp3 --length 733f --hit 600f --video film.mp4 --video-sound keep
+npx musicfit@latest track.mp3 --length 32 --video talk.mp4 --video-sound voice
 npx musicfit@latest track.mp3 --analyze
 ```
 
 Installed globally (`npm install -g musicfit`), the command is `musicfit`. npx keeps the copy an earlier run fetched, so `npx musicfit@latest` is how you get a newer version.
 
-It needs Node 22 or newer and ffmpeg (`brew install ffmpeg`, `sudo apt install ffmpeg` or `winget install ffmpeg`). Set `FFMPEG_PATH` to pick the ffmpeg. There are no npm dependencies; the beat tracking is plain JavaScript and nothing leaves your machine.
+It needs Node 22 or newer and ffmpeg 4.4 or newer (`brew install ffmpeg`, `sudo apt install ffmpeg` or `winget install ffmpeg`); the tests run on Ubuntu 22.04's ffmpeg 4.4 and on the latest Ubuntu's, before every release too. Set `FFMPEG_PATH` to pick the ffmpeg. There are no npm dependencies; the beat tracking is plain JavaScript and nothing leaves your machine.
 
 ## How it works
 
@@ -118,7 +184,7 @@ It needs Node 22 or newer and ffmpeg (`brew install ffmpeg`, `sudo apt install f
 4. Finds the bar lines from where the kicks, the chord changes and the changes in loudness fall, and whether a bar has three beats or four.
 5. Describes every bar by its harmony, timbre, loudness and rhythm, finds where sections change, the drop (the first time the track reaches its fullest after a quieter stretch, on a phrase line) and how the track ends.
 6. Lays the result out as slots one bar long, lined up so a slot starts exactly on the hit, and fills them with bars of the track by dynamic programming: playing on costs nothing, a seam costs more the less its two sides sound like bars that follow each other in the track, and more again when it breaks a four-bar phrase or leaves a build before its drop.
-7. Joins the bars with short crossfades, lines up the hits at every seam, fades the head and the tail, ducks the voice, sets the loudness, and writes the files.
+7. Joins the bars with short crossfades, lines up the hits at every seam, fades the head and the tail, ducks the voice, sets the loudness, and writes the files. With `--video` it reads the video's packets first, to know its length and frame rate and to refuse what it cannot take, and at the end puts the new sound under its picture, which it copies as it is.
 
 ## Limits
 
@@ -130,6 +196,9 @@ It needs Node 22 or newer and ffmpeg (`brew install ffmpeg`, `sudo apt install f
 - A tempo that changes a lot within the track, or a bar of 2/4 in a 4/4 song, can put a seam a beat off.
 - Seams follow bars and sound, not words: in a song with singing a cut can fall in the middle of a line. An instrumental fits best.
 - Lengthening loops bars the track already has, so a 30 second track stretched to 3 minutes repeats, and so does the intro when the drop is asked for much later than the track has it (musicfit says how many times). Ask the music model for a little more than you need.
+- `--video` copies the picture as it is, so a picture an MP4 cannot hold that way (ProRes, DNxHD, VP8) is refused rather than encoded again. Only the picture and the new sound go into `video.mp4`: subtitles and other streams of the video are left out, a note says so of subtitles, and of several sound streams only the first is read.
+- A fit holds its sound in memory a few times over, so memory grows with `--length` and with the length of the track itself: a 10 minute fit of a 10 minute track took 0.9 to 1 GB, 1.2 GB with `--video` and its own sound kept, 1.4 to 1.5 GB with `--voice` and 1.7 GB with a voice and the video's own sound as well, on a desktop with ffmpeg 8 and in a 2 GB container with ffmpeg 4.4. A run takes a few seconds, and with `--video` of a long film a minute or more, most of it spent encoding the sound as AAC.
+- musicfit cannot tell speech from sound effects or from music, which is why it asks for `--video-sound`. With `voice` the music dips under anything in that sound that stands out, a cough or a door as much as a word, and a hiss or a hum under a quiet voice comes up with it. With `keep` each place is set on its own, so a soft effect and a loud one over the same music come out alike, and a hiss or a hum between them comes up with the places around it.
 - It does not change the tempo or the pitch, and it does not check whether you may use the track.
 
 ## See also

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "musicfit",
-  "version": "0.1.1",
-  "description": "Fit a music track to a video cut: exactly the length you need, cut and looped on bar lines, the drop on the second you pick, a fade that ends on a beat, ducked under a voice and at the right loudness.",
+  "version": "0.2.0",
+  "description": "Fit a music track to a video cut: exactly the length you need, cut and looped on bar lines, the drop on the second you pick, a fade that ends on a beat, ducked under a voice, at the right loudness, and laid under the video when you give it.",
   "license": "MIT",
   "author": "Emin Budak",
   "homepage": "https://github.com/EminBudak/musicfit#readme",

package/skills/musicfit/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: musicfit
-description: Fit a music track to a video before you put it under the video, such as a launch film, a product demo, an ad, a reel or a Remotion or HyperFrames composition. Makes the track exactly as long as the cut by cutting or looping whole bars, lands its drop on a chosen second, ends it with a fade that resolves on a beat, ducks it under a voice-over and sets its loudness, and writes every beat and bar as frame numbers. Use it whenever a track (from a music model, a library or the user) has to fit a video's length, or the cuts should land on the beat.
+description: Fit a music track to a video and lay it under the video, such as a launch film, a product demo, an ad, a reel or a Remotion or HyperFrames composition. Makes the track exactly as long as the cut by cutting or looping whole bars, lands its drop on a chosen second, ends it with a fade that resolves on a beat, ducks it under a voice-over, sets its loudness, writes every beat and bar as frame numbers, and with the video given writes the video with the track under it. Use it whenever a track (from a music model, a library or the user) has to fit a video's length, has to go under a finished video, or the cuts should land on the beat.
 ---
 
 # musicfit
@@ -13,19 +13,33 @@ Music models ignore the length you ask for, and a track cut at the video's last
 node <skill>/scripts/musicfit.js track.mp3 --length 32 --hit 27.5 --fps 30 --out audio/fit
 ```
 
-`<skill>` is this skill's folder. The script needs Node 22 or newer and ffmpeg on the machine; it installs nothing and needs no API key. A run takes a few seconds.
+`<skill>` is this skill's folder. The script needs Node 22 or newer and ffmpeg 4.4 or newer on the machine; it installs nothing and needs no API key. A run takes a few seconds, and with a long `--video` a minute or more. A long fit needs memory, more the longer the fit and the longer the track: a 10 minute fit of a 10 minute track took 0.9 to 1 GB, 1.2 GB with `--video`, 1.4 to 1.5 GB with `--voice` and 1.7 GB with a voice and the video's own sound.
 
 - `--length` is the video's length: seconds (`32`), frames (`960f`, at `--fps`) or minutes (`1:05`).
-- `--hit` is the second the drop should land on, for example the logo reveal. Leave it out and the track starts as it starts and ends on its own ending.
+- `--hit` is the second the drop should land on, for example the logo reveal. Leave it out and the track keeps its own ending; a short fit can then start later in the track, so its drop may not play (FIT.md says where it starts). Give `--hit` when the drop has to be heard.
 - `--drop 41.3` says where the drop is in the track when the one found is not the one you want. `--analyze` lists the candidates first.
 - `--voice vo.wav` ducks the music under a voice track laid out on the video's timeline and also writes `mix.wav`.
+- `--video film.mp4` also writes `video.mp4`, the video with the fitted track under it: see the next section.
 - `--fps` should match the composition, so the frame numbers in `cues.json` can be used as they are.
+- `--analyze` saves in `musicfit-<track name>.analysis` by default, never in a fit's folder unless you give both the same `--out`. An analysis given a fit's `--out` replaces that fit when it ends well: `fitted.wav`, `mix.wav` and `video.mp4` are removed. Give `--analyze` its own folder, or leave `--out` out.
+
+## Lay it under a video
+
+`--video film.mp4` also writes `video.mp4`: the video's picture as it was, copied and not encoded again, with the fitted track under it at the loudness target. The video has to be `--length` long to a frame; without `--fps` the frames count at its frame rate.
+
+A video with more than a faint room tone of its own, a single click included, is refused until `--video-sound` says what becomes of that sound. When you do not know what it is, ask the user:
+
+- `keep` for sound effects: each place where it plays is set 3 dB louder than the music there, and no quieter than the loudness target, so a place over quiet music or none plays at the target; it goes no louder than keeps its true peaks, with the music's, under -1 dBTP, so the limiter takes nothing off the music. Where that leaves a place less than 3 dB over the music, as a sharp sound or one on a loud drop can be, the music dips under that place alone by as much as is missing, `--duck` (10 dB) at most, and plays as in `fitted.wav` elsewhere; a note names a place that stays short even so. `video.mp4` can then come out under the target, more at a loud `--lufs`; a note says so when it is more than half a LU short. In a launch film whose own sound was five effects, the logo sound on the drop came out 3.4 dB over the music with the music 7.7 dB down under it, and the film at -14.4 LUFS. With `--voice` as well, the music dips under the voice as it always does and each place is set over that music, so one the voice speaks over plays at the target, as loud as the voice, with no further dip; where a place meets the voice a limiter holds the two under -1 dBTP and `video.mp4` reaches the target.
+- `voice` for speech: set to the loudness target where it speaks, and the music dips `--duck` (10 dB) under it, in `fitted.wav` and `mix.wav` too. With no speech found in it, it is only turned down if it is louder than the target, and a note says so.
+- `drop` for music of its own: left out.
+
+A voice the video already holds goes with `--video-sound voice`, not as a copy given to `--voice`, which would lay it in twice. For a launch film from launch-film that should carry your own track, make or plan the film with `--no-music`, or set `"music": false` in its film.json and render it again, so its sound is its effects only; then give its film.mp4 here as `--video` with `--video-sound keep`.
 
 ## Use the result
 
 1. Read `audio/fit/FIT.md` first. It says what was cut or looped and lists the moments a person should listen to, with their frames.
 2. Show `seams.png` to the user: the track on top, the fitted result below, every seam and the hit marked.
-3. Put `fitted.wav` (or `mix.wav` with a voice) under the whole video, starting at frame 0. Do not trim or stretch it: it is already exactly `--length`.
+3. With `--video`, hand over `video.mp4`: that is the video with the track under it. Without it, put `fitted.wav` (or `mix.wav` with a voice) under the whole video, starting at frame 0, and do not trim or stretch it: it is already exactly `--length`. Never say the track is in the video before a file with it exists.
 4. Time the edit to the music with `cues.json`: `beats` and `bars` are frame numbers at `--fps`, `hits` the frames of the drop and of the track's own last hit, `sections` where the music changes, and `times` has the same moments in seconds. In Remotion that is `<Sequence from={cues.bars[8]}>`; cut scenes on bar frames, small moves on beats.
 5. Tell the user to listen at the seams and the hit before the film goes out. `FIT.md` lists them with their frames.
 
@@ -37,4 +51,12 @@ node <skill>/scripts/musicfit.js track.mp3 --length 32 --hit 27.5 --fps 30 --out
 - A note that some bars play several times means `--hit` is much later than the track's own drop, so its intro repeats. Suggest an earlier `--hit`, or `--drop` with a later candidate.
 - With no steady beat, `beats` and `bars` in `cues.json` are empty lists. Time the edit by `hits` and `seams` instead.
 - The track can come from any music model. Ask for an instrumental a little longer than the video: musicfit shortens more gracefully than it lengthens, and its seams follow bars, not sung words.
+- `--video` refuses before any work a video it cannot read, one that is cut short or damaged, one with no picture, one whose picture an MP4 cannot hold without encoding it again (ProRes, DNxHD, VP8: ask for an H.264 or HEVC copy), one that is not `--length` long (the message gives the length to use), and a video musicfit made unless `--video-sound drop` leaves its sound out (else give the video it was made from, which the message names when the video's `cues.json` is beside it).
+- A video with sound of its own is refused until `--video-sound` is given: ask the user whether that sound is sound effects to keep (`keep`), speech (`voice`) or music of its own (`drop`). Read the notes in the summary too: one says when the video's own sound plays for most of its length, and with `keep` one names each place where that sound stays less than 3 dB louder than the music even with the music dipped `--duck` under it, as a sharp sound over the loudest part of the track can. Tell the user; a larger `--duck` lets the music dip further there, and deepens its dip under the `--voice` too when there is one, so no note suggests it then.
+- musicfit reads only a video's first sound stream, and a note says when it has more. When the first is silent or nearly so and a later one is not, the run is refused, with `keep` and `voice` too, as that sound would be left out: ask the user for a copy with that sound first, or give `--video-sound drop` to go on without it. With `drop` no stream is read and no note comes.
+- A note says when a video has subtitle streams: they are not in `video.mp4`, so tell the user.
+- A note that `video.mp4` is short of the loudness target, or over -1 dBTP after the AAC encoding, says how it came out; FIT.md says the same. Tell the user.
+- A run is refused before any work when its out folder holds, under a name the run writes, a file that is not as an earlier musicfit run wrote it there, such as the user's own `video.mp4` beside a copied `cues.json` for a run with `--video`, or a `FIT.md` or `cues.json` the user changed (move it, with the files it lists for a `cues.json`, or give another `--out`); a run that does not write that name leaves the file alone. It is refused too when the track, the voice or the video is one of the files the run would replace or remove.
+- A run that fails or is stopped (Ctrl-C, or a signal such as SIGTERM) leaves its folder as it was, so a fit that worked is never lost to one that did not. As the earlier files stay until the new ones are in place, a run needs room on the disk for both, and a full disk says so. A run that ends well removes the earlier files it does not make again, and a note names them: tell the user when one was a fit's `video.mp4` or `fitted.wav`. A run killed outright after it began writing can leave a hidden `.musicfit-part-` folder, inside the out folder or next to it when the out folder was not there yet, which a later run that ends well in the same place removes once the killed run's process is gone and 15 minutes have passed (one it may not look into stays, and does not fail that run); a run killed in the moment it moved its files in is undone by the next run there, which puts the earlier files back first, and no other run removes its hidden folder. When a failed or stopped run could not remove its hidden folder, its message names it, and a later run removes it the same way. When another program held a file open so that a run could not put every earlier file back, the message names the folder they are in, and its `musicfit-kept.txt` says where each one belongs: tell the user.
+- musicfit reads files: a track, a voice or a video given as an address is refused. Download it first.
 - Never say a track is cleared for use; musicfit does not check rights.

package/skills/musicfit/scripts/lib/analyze.js

@@ -358,8 +358,11 @@ function estimateTempo(feat, from, to, { bpm: forced } = {}) {
     const value = comb(acf, bpm);
     if (value > best.value) best = { bpm, value };
   }
-  // Refine to a hundredth of a beat per minute.
-  for (let bpm = best.bpm - 0.1; bpm <= best.bpm + 0.1; bpm += 0.01) {
+  // Refine to a hundredth of a beat per minute, around where the search
+  // found it: where the curve has no peak, as a steady tone has none, a
+  // bound that moved with the best would climb without end.
+  const around = best.bpm;
+  for (let bpm = around - 0.1; bpm <= around + 0.1; bpm += 0.01) {
     const value = comb(acf, bpm);
     if (value > best.value) best = { bpm, value };
   }
@@ -931,4 +934,4 @@ function finishEnergy(result, feat, chroma, audible, options) {
   return result;
 }
 
-module.exports = { analyze, RATE };
+module.exports = { analyze, RATE, barDistance };

package/skills/musicfit/scripts/lib/audio.js

@@ -2,72 +2,233 @@
 
 // ffmpeg does the decoding, the encoding and the loudness measurement.
 // Samples travel through pipes as raw 32-bit floats, so nothing is written
-// to disk but the final files.
+// to disk but the final files. Every call here works with ffmpeg 4.4 (the
+// one Ubuntu 22.04 has) and newer. A sound goes to ffmpeg as interleaved
+// floats, or as stereo { L, R }, which is interleaved a block at a time on
+// its way, so no interleaved copy of a long sound is ever held whole.
 
+const path = require("node:path");
 const { spawn } = require("node:child_process");
+const { CHUNK, breathe } = require("./breathe");
 
 const RATE = 48000;
 
+// A name made whole from the folder musicfit was started in. path.resolve
+// works out a .. by its letters, while the system goes up from wherever a
+// link on the way leads, so a name with a .. in it keeps it. Windows goes by
+// the letters too.
+function whole(name) {
+  if (process.platform === "win32" || !name.split("/").includes("..")) return path.resolve(name);
+  return path.isAbsolute(name) ? name : `${process.cwd()}/${name}`;
+}
+
+// A path with folders in it is read from where musicfit was started, also
+// for a call that runs ffmpeg in another folder.
 function ffmpegPath() {
-  return process.env.FFMPEG_PATH || "ffmpeg";
+  const named = process.env.FFMPEG_PATH;
+  if (!named) return "ffmpeg";
+  return path.basename(named) === named ? named : whole(named);
+}
+
+// The environment for a call run in another folder: a relative entry of PATH
+// reads from that folder when ffmpeg is looked up, so that entry is written
+// from where musicfit was started, and the others stay as they are. An empty
+// entry is the current folder too. Windows passes over an empty entry, takes
+// one in quotes, which may hold a ; of its own, and the variable's name in
+// any case.
+function fromHere() {
+  const windows = process.platform === "win32";
+  const { delimiter, isAbsolute } = windows ? path.win32 : path.posix;
+  const key = windows ? Object.keys(process.env).find((name) => name.toUpperCase() === "PATH") : "PATH";
+  if (!key || process.env[key] === undefined) return process.env;
+  const entries = windows ? process.env[key].match(/(?<=^|;)("[^"]*"?|'[^']*'?)?[^;]*/g) : process.env[key].split(delimiter);
+  const written = entries.map((entry) => {
+    const bare = windows ? entry.replace(/^["']|["']$/g, "") : entry;
+    return isAbsolute(bare) || (windows && !bare) ? entry : whole(bare);
+  });
+  return { ...process.env, [key]: written.join(delimiter) };
+}
+
+// A file as ffmpeg is given it: by its full path, since ffmpeg takes a name
+// whose part before a colon is all letters and digits (9:16.mp4) for a
+// protocol, and never a full path.
+const where = (file) => whole(file);
+
+// What ffmpeg prints once it has given up, after the line that says why:
+// how it stopped, which says nothing of the cause. ffmpeg 4.4 ends with the
+// first five, newer ones with the others.
+const CLOSING = [
+  /^Conversion failed!?$/,
+  /^Error while processing the decoded data/,
+  /^Failed to inject frame into filter network/,
+  /^Error reinitializing filters!?$/,
+  /^Exiting normally/,
+  /^Error initializing a simple filtergraph$/,
+  /^Error opening (input|output) files?\b/,
+  /^Error selecting an encoder$/,
+  /^Error closing file\b/,
+  /^Error muxing a packet$/,
+  /^Task finished with error code/,
+  /^Terminating thread with return code/,
+];
+// The words of a line that says what went wrong.
+const WRONG = /error|\b(invalid|unable|cannot|can't|could not|couldn't|failed|not found|no such|unknown|unrecognized|not supported|denied|no space|refused|not opened|is not|does not)\b/i;
+// What ffmpeg says of its input and output on the way, for a run that names no error.
+const INFO = /^(Input|Output) #\d|^Stream (mapping|#)|^Guessed Channel Layout|^Estimating duration|^Duration:|^Metadata:|^Press \[q\]|^(size|frame)=|^Summary:$/;
+
+// Whether ffmpeg was stopped by a signal: one it did not catch, or one it
+// caught, as Ctrl-C, after which it exits with 255. At -v info it then says
+// "Exiting normally, received signal N."; at -v error it says nothing at
+// all. ffmpeg 7 and newer also exit with 255 when the system refuses an
+// operation, but they say so first.
+function stopped(stderr, code, signal) {
+  return Boolean(signal) || /received signal \d+/.test(stderr || "") || (code === 255 && !String(stderr || "").trim());
+}
+
+// Why ffmpeg failed, from what it printed: the last line that names an
+// error once the closing lines and the indented ones (a summary, the
+// streams) are left out, without the "[name @ 0x...]" that starts it; else
+// the last line it printed that says something.
+function reason(stderr, code, signal) {
+  if (signal) return `ffmpeg was stopped (${signal})`;
+  // ffmpeg stopped by a signal it caught, as Ctrl-C.
+  const caught = /received signal (\d+)/.exec(stderr || "");
+  if (caught) return `ffmpeg was stopped (signal ${caught[1]})`;
+  if (stopped(stderr, code, signal)) return "ffmpeg was stopped";
+  const lines = String(stderr || "")
+    .split(/\r?\n|\r/)
+    .filter((line) => line.trim() && !/^\s/.test(line))
+    .map((line) => line.replace(/^(\[[^\]]*\]\s*)+/, "").trim())
+    .filter((line) => line && !CLOSING.some((closing) => closing.test(line)));
+  const line = lines.filter((l) => WRONG.test(l)).pop() || lines.filter((l) => !INFO.test(l)).pop();
+  if (!line) return `ffmpeg exited with ${code}`;
+  return /ffmpeg/i.test(line) ? line : `ffmpeg: ${line}`;
+}
+
+// ffmpeg 4.4 exits with 0 when it cannot finish a file: when the disk fills
+// as it writes an MP4's index or a WAV's sizes, it says "Error writing
+// trailer of <file>: No space left on device" and leaves a file no player
+// reads. A call that writes `file` and says so, in those words for that
+// file or in ffmpeg 7's for its output, has failed, whatever its exit
+// code. Nothing else it prints is read for them, so a file's name, as in
+// what ffmpeg says of a video it reads, is never taken for its words.
+function unfinished(stderr, file) {
+  const said = `\n${stderr}`;
+  return said.includes(`\nError writing trailer of ${file}: `) || /\n\[out#\d+\/\w+ @ 0x[0-9a-f]+\] Error writing trailer\b/.test(said);
+}
+
+// The ffmpeg processes running now, so a run that is stopped stops them
+// too: a signal sent to node alone does not reach them.
+const running = new Set();
+
+function stopChildren() {
+  for (const child of running) {
+    try {
+      child.kill("SIGKILL");
+    } catch {
+      // Gone already.
+    }
+  }
+  running.clear();
 }
 
-function run(args, input) {
+// Runs ffmpeg with `args`, and `input` on its stdin. `writes` is the file
+// the call writes, if any, and `cwd` the folder to run it in, if not this one.
+function run(args, input, { writes = null, cwd } = {}) {
   return new Promise((resolve, reject) => {
     let child;
     try {
-      child = spawn(ffmpegPath(), ["-hide_banner", ...(input ? [] : ["-nostdin"]), ...args], { stdio: [input ? "pipe" : "ignore", "pipe", "pipe"] });
+      child = spawn(ffmpegPath(), ["-hide_banner", ...(input ? [] : ["-nostdin"]), ...args], { cwd, env: cwd ? fromHere() : process.env, stdio: [input ? "pipe" : "ignore", "pipe", "pipe"] });
     } catch (err) {
       reject(err);
       return;
     }
+    running.add(child);
     const out = [];
     const err = [];
     child.stdout.on("data", (chunk) => out.push(chunk));
     child.stderr.on("data", (chunk) => err.push(chunk));
     child.on("error", (error) => {
-      if (error.code === "ENOENT") reject(new Error("musicfit needs ffmpeg. Install it (brew install ffmpeg, sudo apt install ffmpeg or winget install ffmpeg) or set FFMPEG_PATH to it."));
+      running.delete(child);
+      if (error.code === "ENOENT" && !cwd) reject(new Error("musicfit needs ffmpeg. Install it (brew install ffmpeg, sudo apt install ffmpeg or winget install ffmpeg) or set FFMPEG_PATH to it."));
       else reject(error);
     });
-    child.on("close", (code) => {
+    child.on("close", (code, signal) => {
+      running.delete(child);
       const stderr = Buffer.concat(err).toString("utf8");
-      if (code === 0) resolve({ chunks: out, stderr });
-      else {
-        const why = stderr.trim().split("\n").filter(Boolean).pop() || `ffmpeg exited with ${code}`;
-        reject(Object.assign(new Error(why), { stderr }));
-      }
+      if (code === 0 && !(writes && unfinished(stderr, writes))) resolve({ chunks: out, stderr });
+      else reject(Object.assign(new Error(reason(stderr, code, signal)), { stderr, signal, stopped: stopped(stderr, code, signal) }));
     });
     if (input) {
       // ffmpeg may stop reading early when it fails; the error that follows
       // tells why, the broken pipe does not.
       child.stdin.on("error", () => {});
-      child.stdin.end(Buffer.from(input.buffer, input.byteOffset, input.byteLength));
+      if (input.L) feed(child.stdin, input).catch(() => {});
+      else child.stdin.end(Buffer.from(input.buffer, input.byteOffset, input.byteLength));
     }
   });
 }
 
+// Stereo { L, R } written to ffmpeg as interleaved floats, a block at a
+// time, each written once the one before has gone down the pipe.
+async function feed(stdin, { L, R }) {
+  const block = 1 << 16;
+  const ready = () =>
+    new Promise((resolve) => {
+      const go = () => {
+        stdin.off("drain", go);
+        stdin.off("close", go);
+        resolve();
+      };
+      stdin.on("drain", go);
+      stdin.on("close", go);
+    });
+  for (let from = 0; from < L.length && !stdin.destroyed; from += block) {
+    const to = Math.min(L.length, from + block);
+    const out = new Float32Array(2 * (to - from));
+    for (let i = from; i < to; i++) {
+      out[2 * (i - from)] = L[i];
+      out[2 * (i - from) + 1] = R[i];
+    }
+    if (!stdin.write(Buffer.from(out.buffer))) await ready();
+  }
+  stdin.end();
+}
+
 // The pipe's chunks copied byte for byte into one array of floats, without
 // joining them first: a long track would be held twice.
-function floats(chunks) {
+async function floats(chunks) {
   const bytes = chunks.reduce((n, c) => n + c.length, 0);
   const out = new Float32Array(Math.floor(bytes / 4));
   const view = new Uint8Array(out.buffer);
   let at = 0;
+  let since = 0;
   for (const chunk of chunks) {
     const take = Math.min(chunk.length, view.length - at);
     if (take <= 0) break;
     view.set(chunk.subarray(0, take), at);
     at += take;
+    since += take;
+    if (since >= 4 * CHUNK) {
+      since = 0;
+      await breathe();
+    }
   }
   return out;
 }
 
-// The first audio stream of any file ffmpeg reads (a video's too), as
-// interleaved floats at the given rate and channel count.
-async function decode(file, { rate = RATE, channels = 2 } = {}) {
+// The first audio stream of any file ffmpeg reads (a video's too), or the
+// one numbered `stream` from 0, as interleaved floats at the given rate and
+// channel count. With `from` it is read on the file's own timeline from that
+// second on, for a video's own sound from its picture's first frame: cut
+// where the sound starts before it, silence where the sound starts after it
+// or has a gap, so a second into the sound is a second into the picture.
+async function decode(file, { rate = RATE, channels = 2, from = null, stream = 0 } = {}) {
+  const timeline = from === null ? [] : ["-copyts"];
+  const align = from === null ? [] : ["-af", `aresample=${rate}:async=1:first_pts=${Math.round(from * rate)}`];
   try {
-    const { chunks } = await run(["-v", "error", "-i", file, "-map", "0:a:0", "-vn", "-ac", String(channels), "-ar", String(rate), "-f", "f32le", "pipe:1"]);
-    const samples = floats(chunks);
+    const { chunks } = await run(["-v", "error", ...timeline, "-i", where(file), "-map", `0:a:${stream}`, "-vn", ...align, "-ac", String(channels), "-ar", String(rate), "-f", "f32le", "pipe:1"]);
+    const samples = await floats(chunks);
     if (!samples.length) throw new Error(`${file} has no audio`);
     return samples;
   } catch (err) {
@@ -80,20 +241,25 @@ async function decode(file, { rate = RATE, channels = 2 } = {}) {
   }
 }
 
-// Writes interleaved stereo floats as a 24-bit WAV.
+// Writes a stereo sound as a 24-bit WAV. bitexact leaves out the
+// header's note of the ffmpeg that wrote it, so any ffmpeg writes the same
+// bytes for the same sound.
 async function encodeWav(samples, file, { rate = RATE, channels = 2 } = {}) {
-  await run(["-v", "error", "-f", "f32le", "-ar", String(rate), "-ac", String(channels), "-i", "pipe:0", "-c:a", "pcm_s24le", "-y", file], samples);
+  await run(["-v", "error", "-f", "f32le", "-ar", String(rate), "-ac", String(channels), "-i", "pipe:0", "-c:a", "pcm_s24le", "-fflags", "+bitexact", "-y", where(file)], samples, { writes: where(file) });
 }
 
 // EBU R128 loudness as ffmpeg's ebur128 meter reads it: integrated
 // loudness (LUFS), true peak (dBTP, oversampled), loudness range (LU).
 // Silence reads -Infinity. loudnorm measures the same way but upsamples
-// everything to 192 kHz first, which is ten times slower.
-async function measure(samples, { rate = RATE, channels = 2 } = {}) {
-  const { stderr } = await run(["-v", "info", "-f", "f32le", "-ar", String(rate), "-ac", String(channels), "-i", "pipe:0", "-af", "ebur128=peak=true:framelog=quiet", "-f", "null", "-"], samples);
-  const summary = stderr.slice(stderr.lastIndexOf("Integrated loudness:"));
+// everything to 192 kHz first, which is ten times slower. framelog=verbose
+// keeps the meter's line for every 100 ms out of what -v info prints;
+// ffmpeg 4.4 has no quieter value, and the summary is the same.
+const METER = "ebur128=peak=true:framelog=verbose";
+
+function summary(stderr) {
+  const said = stderr.slice(stderr.lastIndexOf("Integrated loudness:"));
   const read = (label) => {
-    const m = new RegExp(`${label}:\\s*(-?inf|-?[\\d.]+)`).exec(summary);
+    const m = new RegExp(`${label}:\\s*(-?inf|-?[\\d.]+)`).exec(said);
     if (!m) throw new Error("ffmpeg's ebur128 printed no measurement");
     return /inf/.test(m[1]) ? -Infinity : Number(m[1]);
   };
@@ -101,4 +267,17 @@ async function measure(samples, { rate = RATE, channels = 2 } = {}) {
   return { integrated: integrated <= -70 ? -Infinity : integrated, truePeak: read("Peak"), range: read("LRA") };
 }
 
-module.exports = { RATE, decode, encodeWav, measure };
+async function measure(samples, { rate = RATE, channels = 2 } = {}) {
+  const { stderr } = await run(["-v", "info", "-nostats", "-f", "f32le", "-ar", String(rate), "-ac", String(channels), "-i", "pipe:0", "-af", METER, "-f", "null", "-"], samples);
+  return summary(stderr);
+}
+
+// The same for the first sound of a file the run made, as a player decodes
+// it. Named from its own folder: the hosted service's ffmpeg refuses an input
+// with a % in its path, and the name of the out folder is the user's.
+async function measureFile(file) {
+  const { stderr } = await run(["-v", "info", "-nostats", "-i", path.basename(file), "-map", "0:a:0", "-af", METER, "-f", "null", "-"], null, { cwd: path.dirname(file) });
+  return summary(stderr);
+}
+
+module.exports = { RATE, decode, encodeWav, fromHere, measure, measureFile, run, reason, stopChildren, where };

package/skills/musicfit/scripts/lib/breathe.js

@@ -0,0 +1,24 @@
+"use strict";
+
+// A stop (Ctrl-C, or the signal a supervisor sends) is heard only while the
+// main thread is free, and going through every sample of a long sound takes
+// seconds. So the loops over the samples go through them in blocks of half
+// a million and give way before each, and a stop is heard within a moment
+// at any point of a run. Each block is done in one plain call, which runs as
+// fast as the loop did in one go.
+
+const CHUNK = 1 << 19;
+const breathe = () => new Promise((resolve) => setImmediate(resolve));
+
+// body(from, to) for each block of [start, end), `size` long, first to
+// last. What a body carries from one block to the next it reads into its
+// own variables at the start of the block and writes back at the end,
+// never in its loop.
+async function blocks(start, end, body, size = CHUNK) {
+  for (let from = start; from < end; from += size) {
+    await breathe();
+    body(from, Math.min(end, from + size));
+  }
+}
+
+module.exports = { CHUNK, breathe, blocks };