@milkstraw/traceframe 0.4.0 → 0.5.1

package/README.md

@@ -2,7 +2,7 @@
 
 Traceframe records a scripted Chromium flow and renders an MP4 with a smooth cursor and click pulse. Everything runs locally: pages, flows, videos, and authentication state stay on your machine.
 
-Recordings are silent; audio is not captured.
+Browser capture is silent by default. Add an [external audio file](#add-audio) with explicit timing; tab audio and microphones are not captured automatically.
 
 https://github.com/user-attachments/assets/38e02cd5-a9ba-4f3c-938c-159955353141
 
@@ -170,7 +170,7 @@ The callback receives the same `page`, `signal` and five helpers as a flow file.
 
 Use Playwright 1.64.1 with Chromium, a fixed viewport, 100% browser zoom and an unscaled page. Traceframe checks the page's inner dimensions and visual viewport before starting capture, rejecting unsupported geometry with `CAPTURE_SIZE_MISMATCH`. `recordPage` keeps the page, context and browser open on success, failure and cancellation. It does not navigate, resize, run a `ready` hook, or change page timeouts. Your script owns setup and shutdown. The CLI and `record` continue to create and close their own browser.
 
-`page` and a function `flow` are required. `output`, `postRoll`, `flowTimeout`, `runDirectory` and `signal` have the same meaning as in `record`, and the return value is the same recording summary. Set `initialPointer: { x, y }` to choose the starting cursor position. The page's viewport determines the dimensions; `scale` defaults to 1 and accepts 1 or 2, and `fps` defaults to 30 and accepts 30 or 60. Use these explicit settings instead of a preset. Browser setup options such as `url`, `browser`, `storageState`, viewport dimensions and `readyTimeout` are rejected.
+`page` and a function `flow` are required. `output`, `audio`, `postRoll`, `flowTimeout`, `runDirectory` and `signal` have the same meaning as in `record`, and the return value is the same recording summary. Set `initialPointer: { x, y }` to choose the starting cursor position. The page's viewport determines the dimensions; `scale` defaults to 1 and accepts 1 or 2, and `fps` defaults to 30 and accepts 30 or 60. Use these explicit settings instead of a preset. Browser setup options such as `url`, `browser`, `storageState`, viewport dimensions and `readyTimeout` are rejected.
 
 Traceframe moves the caller's pointer only after validating the first frame. To show the starting hover state from frame 0, move the page's mouse to your chosen `initialPointer` before calling `recordPage`.
 
@@ -182,6 +182,39 @@ Await every helper. Helpers observe cancellation before further pointer or scrol
 
 Traceframe removes its own temporary outputs unless `runDirectory` is supplied. Playwright keeps original WebMs in its browser artifacts directory until the caller closes the browser, even after a take has been saved. For extension pages, use full Chromium; the headless shell alone does not load extensions. This path was checked on Linux with Playwright 1.64.1 and its bundled full Chromium using a synthetic extension.
 
+### Add audio
+
+Both `record` and `recordPage` accept `audio: { path: './sound.wav', offset: 0 }`. The CLI accepts the same settings:
+
+```sh
+npx @milkstraw/traceframe record --url http://localhost:3000 --flow ./flow.mjs \
+  --audio .traceframe/sound.wav --audio-offset 0.5 --out .traceframe/demo.mp4
+```
+
+`offset` is seconds from the first video frame to the first audio sample. Positive values add leading silence; negative values discard the beginning of the audio (in the CLI, use e.g. `--audio-offset=-0.25`). The default is 0. Short audio is padded with silence, and longer audio is trimmed to the video's end. Nothing is looped or stretched. The first audio stream is encoded as stereo AAC at 48 kHz, 192 kb/s; system FFmpeg must provide its standard `aac` encoder. Successful summaries include `audio: true`.
+
+The input must be a local file, such as WAV, MP3, M4A or audio WebM. Static inputs are checked before browser recording, so a missing file or missing audio stream cannot waste a take (`AUDIO_INVALID`). After capture, an audio start at or after the video's end rejects with `AUDIO_OUTSIDE_VIDEO`. A negative offset that discards the entire source produces a silent audio track.
+
+Traceframe reads the file without modifying or deleting it; use an input path different from the output and the reserved files in `runDirectory` (`capture.webm`, `render.filter`, `failure.png`). Keep recordings in an ignored directory.
+
+For audio recorded alongside the flow, supply an async function instead. It runs once after successful capture and post-roll, before rendering, so it can finalize the external recorder and return its file. `startedAt` is the first video frame's timestamp in epoch milliseconds:
+
+```js
+await recordPage({
+  page,
+  output: '.traceframe/demo.mp4',
+  flow,
+  audio: async ({ startedAt, signal }) => {
+    const { path, startedAt: audioStartedAt } = await finishAudioRecording({ signal })
+    return { path, offset: (audioStartedAt - startedAt) / 1000 }
+  },
+})
+```
+
+Here `finishAudioRecording` belongs to your script and reports the timestamp of its first recorded sample. Use timestamps on a shared clock, or determine the offset using an audiovisual marker. An offset corrects a constant alignment difference; it does not correct clock drift. Check markers near both ends of longer takes. See the [audio workflow](docs/recording-flows.md#align-external-audio).
+
+Your script owns the external recorder and its cleanup, including on flow failure or cancellation. The audio function is skipped if capture fails and must cooperate with `signal` when supplied. It may finish writing after capture; a static input must already exist before recording. Neither form captures browser or system sound by itself.
+
 ### Errors and cancellation
 
 Traceframe's own errors are `TraceframeError` instances (exported by the package) with a `code`, such as `READY_FAILED` or `FLOW_TARGET_OBSCURED`. Other exceptions thrown by the flow become `FLOW_FAILED`, with the original thrown value in `cause`; existing Traceframe error codes and causes are preserved.

package/docs/recording-flows.md

@@ -208,6 +208,14 @@ Inspect the reported screenshot, then the capture near the reported time if need
 
 Only paths saved for the current attempt are reported. A closed or unresponsive page may prevent a screenshot or capture from being saved. Errors before the first frame have no recording time, and a flow timeout may have no source location. Without `--run-dir`, flow locations and timing are still reported when available, but temporary captures are removed and no failure screenshot is saved. The API exposes the same context through [`error.diagnostics`](../README.md#errors-and-cancellation).
 
+## Align external audio
+
+Capture the application's actual processed output with your own audio recorder, then pass its local file through [`audio`](../README.md#add-audio). Use the same prepared page with `recordPage` when the audio setup needs a caller-owned context. Audio recording remains your script's responsibility; Traceframe aligns and encodes the supplied file in the final MP4.
+
+Use a distinct sound and visible change as a synchronization marker. If the visible change lands at 2.00 seconds in the video and its sound starts at 2.08 seconds in the audio file, pass `offset: -0.08`. Measure on the actual recorded streams, accounting for capture-device and encoder latency. For a live recorder, the audio supplier's `startedAt` gives the video clock; the recorder's timestamp must refer to its first sample on a shared clock.
+
+Inspect the rendered result with audio enabled, and compare another marker near the end. If the difference grows, a constant offset is insufficient: correct the external recording's rate separately before adding it. Traceframe never changes audio speed or pitch. Trimmed audio ends with the video, including post-roll; shorter audio leaves silence. For independent takes, calculate alignment for each take and close the external recorder in your script's `finally` block.
+
 ## Edit the finished video
 
 Edit the finished MP4 with FFmpeg rather than changing the flow. Editing finished frames keeps the page and cursor together. Keep the original, and say when a clip's pacing was edited.
@@ -225,6 +233,8 @@ export default async function ({ page, click, wait }) {
 
 Log to `console.error`, because Traceframe prints its JSON summary on standard output. This command plays seconds 2 to 9 at 4× speed. Set `fps` to the video's frame rate: 60 for the default preset, 30 for draft.
 
+This command drops the audio track. To keep audio synchronized, apply the same cuts and speed changes to its timeline, including when speeding up silent intervals.
+
 ```sh
 ffmpeg -i recording.mp4 -filter_complex "[0:v]trim=0:2,setpts=PTS-STARTPTS[a];[0:v]trim=2:9,setpts=(PTS-STARTPTS)/4[b];[0:v]trim=9,setpts=PTS-STARTPTS[c];[a][b][c]concat=n=3,fps=60[v]" -map "[v]" -c:v libx264 -crf 18 -movflags +faststart faster.mp4
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@milkstraw/traceframe",
-  "version": "0.4.0",
+  "version": "0.5.1",
   "description": "Record scripted browser flows and render smooth cursor overlays into video.",
   "keywords": [
     "playwright",

package/src/capture.mjs

@@ -65,7 +65,7 @@ export async function captureFlow({ config, flow, directory, page, signal: calle
     // A failed stop must not be retried: Playwright may still be waiting to finish its artifact.
     recording = false
     await page.screencast.stop()
-    return { rawPath, trace }
+    return { rawPath, trace, startedAt: clock }
   } catch (error) {
     failed = true
     const recordingTime = clock === undefined ? undefined : Math.max(0, (Date.now() - clock) / 1000)

package/src/cli.mjs

@@ -24,8 +24,10 @@ const OPTION_KEYS = {
   'ready-timeout': 'readyTimeout',
   'flow-timeout': 'flowTimeout',
   'run-dir': 'runDirectory',
+  audio: 'audio',
+  'audio-offset': 'audioOffset',
 }
-const NUMBER_KEYS = new Set(['viewportWidth', 'viewportHeight', 'scale', 'fps', 'postRoll', 'readyTimeout', 'flowTimeout'])
+const NUMBER_KEYS = new Set(['viewportWidth', 'viewportHeight', 'scale', 'fps', 'postRoll', 'readyTimeout', 'flowTimeout', 'audioOffset'])
 const HELP = `traceframe ${VERSION}
 
 Record a scripted Chromium flow and render a smooth cursor and click pulse.
@@ -46,6 +48,8 @@ Options:
   --ready-timeout <seconds>    Playwright default timeout during navigation and ready() (default: 30)
   --flow-timeout <seconds>     Time limit for the flow, including waits (default: 120)
   --run-dir <path>             Keep the raw capture, failure screenshot and FFmpeg filter
+  --audio <path>               Add an external audio file (AAC stereo in the MP4)
+  --audio-offset <seconds>     Audio start in the video; negative trims its start (default: 0)
   -h, --help                   Show this help
   -v, --version                Show the version
 
@@ -108,5 +112,9 @@ function parseCommand(args) {
   for (const [name, key] of Object.entries(OPTION_KEYS)) {
     if (values[name] !== undefined) options[key] = NUMBER_KEYS.has(key) ? Number(values[name]) : values[name]
   }
+  if (options.audio !== undefined || options.audioOffset !== undefined) {
+    options.audio = { path: options.audio, offset: options.audioOffset }
+    delete options.audioOffset
+  }
   return { options }
 }

package/src/config.mjs

@@ -62,18 +62,33 @@ function recordingConfig(options, viewport, defaults) {
   check(Number.isFinite(postRoll) && postRoll >= 0, 'Post-roll must be 0 or more seconds.')
   const flowTimeout = options.flowTimeout ?? 120
   check(Number.isFinite(flowTimeout) && flowTimeout > 0, 'Flow timeout must be more than 0 seconds.')
+  const runDirectory = options.runDirectory ? resolve(options.runDirectory) : undefined
 
   return {
     output,
+    audio: typeof options.audio === 'function' ? options.audio : createAudioConfig(options.audio, output, runDirectory),
     viewport: { ...viewport },
     video,
     scale,
     postRoll,
     flowTimeout,
-    runDirectory: options.runDirectory ? resolve(options.runDirectory) : undefined,
+    runDirectory,
   }
 }
 
+export function createAudioConfig(audio, output, directory) {
+  if (audio === undefined) return undefined
+  check(typeof audio?.path === 'string' && audio.path.length > 0, 'Audio must specify a local file path (--audio <path>, or audio: { path }).')
+  const path = resolve(audio.path)
+  check(path !== output, 'Audio input and video output must use different paths.')
+  if (directory) {
+    check(!['capture.webm', 'render.filter', 'failure.png'].some((name) => path === resolve(directory, name)), `Audio input must not use a reserved recording file: ${path}`)
+  }
+  const offset = audio.offset ?? 0
+  check(Number.isFinite(offset), 'Audio offset must be a finite number of seconds.')
+  return { path, offset }
+}
+
 function check(valid, message) {
   if (!valid) throw new TraceframeError(message, { code: 'INVALID_CONFIG' })
 }

package/src/ffmpeg.mjs

@@ -1,11 +1,12 @@
 import { execFile } from 'node:child_process'
+import { stat } from 'node:fs/promises'
 import { promisify } from 'node:util'
 
 import { TraceframeError } from './errors.mjs'
 
 const execFileAsync = promisify(execFile)
 
-export async function probeFfmpeg() {
+export async function probeFfmpeg({ audio = false } = {}) {
   const version = await runCommand('ffmpeg', ['-version'])
   const major = parseFfmpegMajor(version.stdout)
   if (!(major >= 5)) {
@@ -15,12 +16,27 @@ export async function probeFfmpeg() {
   if (!codecs.stdout.includes('libx264')) {
     throw new TraceframeError('System FFmpeg does not provide the libx264 encoder.', { code: 'FFMPEG_UNSUPPORTED' })
   }
+  if (audio && !/^\s*A\S*\s+aac\s/m.test(codecs.stdout)) {
+    throw new TraceframeError('System FFmpeg does not provide the AAC encoder required for audio.', { code: 'FFMPEG_UNSUPPORTED' })
+  }
   await runCommand('ffprobe', ['-version'])
   return { filterComplexFileOption: filterComplexFileOption(major) }
 }
 
-export async function probeDuration(path, signal) {
-  const { stdout } = await runCommand('ffprobe', ['-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', path], { signal })
+export async function probeAudio(path, signal) {
+  try {
+    if (!(await stat(path)).isFile()) throw new Error('The input is not a regular file.')
+    const { stdout } = await runCommand('ffprobe', ['-v', 'error', '-select_streams', 'a:0', '-show_entries', 'stream=index', '-of', 'json', path], { signal })
+    const { streams } = JSON.parse(stdout)
+    if (!streams.length) throw new Error('The file has no audio stream.')
+  } catch (cause) {
+    signal?.throwIfAborted()
+    throw new TraceframeError(`Could not read audio from ${path}: ${cause.message}`, { code: 'AUDIO_INVALID', cause })
+  }
+}
+
+export async function probeVideoDuration(path, signal) {
+  const { stdout } = await runCommand('ffprobe', ['-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=duration', '-of', 'csv=p=0', path], { signal })
   return Number(stdout)
 }
 

package/src/record.mjs

@@ -2,10 +2,10 @@ import { mkdir, mkdtemp, rm } from 'node:fs/promises'
 import { tmpdir } from 'node:os'
 import { dirname, join } from 'node:path'
 
-import { createConfig, createPageConfig } from './config.mjs'
+import { createAudioConfig, createConfig, createPageConfig } from './config.mjs'
 import { captureFlow } from './capture.mjs'
 import { addDiagnostics, retainedFile } from './diagnostics.mjs'
-import { probeFfmpeg } from './ffmpeg.mjs'
+import { probeAudio, probeFfmpeg } from './ffmpeg.mjs'
 import { createFlow, loadFlow } from './flow.mjs'
 import { renderVideo } from './render.mjs'
 import { TraceframeError } from './errors.mjs'
@@ -16,7 +16,7 @@ export async function record(options = {}) {
   const { signal } = options
   signal?.throwIfAborted()
   const config = createConfig(options)
-  const ffmpeg = await probeFfmpeg()
+  const ffmpeg = await probeFfmpeg({ audio: Boolean(config.audio) })
   const flow = await loadFlow(config.flow, config.viewport).catch(async (error) => {
     throw await addDiagnostics(error, { flow: config.flow })
   })
@@ -31,7 +31,7 @@ export async function recordPage(options = {}) {
   if (activePages.has(page)) throw new TraceframeError('This page is already being recorded by Traceframe.', { code: 'PAGE_RECORDING_ACTIVE' })
   activePages.add(page)
   try {
-    const ffmpeg = await probeFfmpeg()
+    const ffmpeg = await probeFfmpeg({ audio: Boolean(config.audio) })
     return await recordFlow({ config, flow, ffmpeg, signal, page })
   } finally {
     activePages.delete(page)
@@ -39,6 +39,8 @@ export async function recordPage(options = {}) {
 }
 
 async function recordFlow({ config, flow, ffmpeg, signal, page }) {
+  let audio = typeof config.audio === 'function' ? undefined : config.audio
+  if (audio) await probeAudio(audio.path, signal)
   await mkdir(dirname(config.output), { recursive: true })
   const directory = config.runDirectory ?? await mkdtemp(join(tmpdir(), 'traceframe-'))
   await mkdir(directory, { recursive: true })
@@ -48,7 +50,13 @@ async function recordFlow({ config, flow, ffmpeg, signal, page }) {
   try {
     capture = await captureFlow({ config, flow, directory, signal, page })
     const { rawPath, trace } = capture
-    const duration = await renderVideo({ rawPath, trace, directory, config, ffmpeg, signal })
+    if (typeof config.audio === 'function') {
+      signal?.throwIfAborted()
+      audio = createAudioConfig(await config.audio({ startedAt: capture.startedAt, signal }), config.output, directory)
+      if (!audio) throw new TraceframeError('The audio function must return { path, offset }.', { code: 'INVALID_CONFIG' })
+      await probeAudio(audio.path, signal)
+    }
+    const duration = await renderVideo({ rawPath, trace, directory, config, ffmpeg, signal, audio })
     return {
       output: config.output,
       width: config.video.width,
@@ -58,6 +66,7 @@ async function recordFlow({ config, flow, ffmpeg, signal, page }) {
       moveCount: trace.moves.length,
       clickCount: trace.clicks.length,
       runDirectory: config.runDirectory,
+      ...(audio ? { audio: true } : {}),
     }
   } catch (error) {
     failed = true

package/src/render.mjs

@@ -2,7 +2,7 @@ import { rename, rm, writeFile } from 'node:fs/promises'
 import { dirname, join } from 'node:path'
 import { randomUUID } from 'node:crypto'
 
-import { probeDuration, runCommand } from './ffmpeg.mjs'
+import { probeVideoDuration, runCommand } from './ffmpeg.mjs'
 import { TraceframeError } from './errors.mjs'
 
 const ASSETS = join(import.meta.dirname, 'assets')
@@ -13,11 +13,14 @@ const CLICK = { duration: 0.24, scaleStart: 0.7, scaleEnd: 1.25 }
 // format=yuv444 places overlays on whole pixels. The default yuv420 snaps them to even pixels, and RGB costs more for the same result.
 const OVERLAY = 'eof_action=pass:format=yuv444'
 
-export async function renderVideo({ rawPath, trace, directory, config, ffmpeg, signal }) {
+export async function renderVideo({ rawPath, trace, directory, config, ffmpeg, signal, audio }) {
   const { fps } = config.video
   const outputDuration = Math.max(trace.end + config.postRoll, ...trace.clicks.map((click) => click.at + CLICK.duration))
+  if (audio && audio.offset >= outputDuration) {
+    throw new TraceframeError(`Audio offset ${audio.offset} seconds starts at or after the video's end (${number(outputDuration)} seconds). Reduce its offset.`, { code: 'AUDIO_OUTSIDE_VIDEO' })
+  }
   const filterPath = join(directory, 'render.filter')
-  await writeFile(filterPath, buildFilterScript({ trace, scale: config.scale }))
+  await writeFile(filterPath, buildFilterScript({ trace, scale: config.scale, audio }))
   const outputTemp = join(dirname(config.output), `.traceframe-${randomUUID()}.tmp.mp4`)
   const args = [
     '-y', '-hide_banner', '-loglevel', 'error',
@@ -25,9 +28,13 @@ export async function renderVideo({ rawPath, trace, directory, config, ffmpeg, s
     '-loop', '1', '-framerate', String(fps), '-i', join(ASSETS, `cursor-${config.scale}x.png`),
   ]
   if (trace.clicks.length) args.push('-loop', '1', '-framerate', String(fps), '-t', String(CLICK.duration + 1 / fps), '-i', join(ASSETS, `click-pulse-${config.scale}x.png`))
+  if (audio) args.push('-i', audio.path)
   args.push(
     ffmpeg.filterComplexFileOption, filterPath,
     '-map', '[final]',
+  )
+  if (audio) args.push('-map', '[finalAudio]', '-c:a', 'aac', '-b:a', '192k', '-ar', '48000', '-ac', '2')
+  args.push(
     '-t', String(number(outputDuration)),
     '-r', String(fps),
     '-c:v', 'libx264', '-preset', 'fast', '-crf', '18',
@@ -37,7 +44,7 @@ export async function renderVideo({ rawPath, trace, directory, config, ffmpeg, s
   )
   try {
     await runCommand('ffmpeg', args, { signal })
-    const duration = await probeDuration(outputTemp, signal)
+    const duration = await probeVideoDuration(outputTemp, signal)
     if (!(Math.abs(duration - outputDuration) <= 1.5 / fps)) {
       throw new TraceframeError(`The rendered video is ${duration.toFixed(2)} seconds long, but the recording is ${outputDuration.toFixed(2)} seconds.`, { code: 'OUTPUT_MISMATCH' })
     }
@@ -51,7 +58,7 @@ export async function renderVideo({ rawPath, trace, directory, config, ffmpeg, s
 }
 
 // Event times are seconds from the first frame of the raw video.
-export function buildFilterScript({ trace, scale }) {
+export function buildFilterScript({ trace, scale, audio }) {
   const { clicks, moves } = trace
   // Playwright ends the raw video by the time it received the last frame, so a page that stays still can end it a little before the post-roll. The page did not change, so repeat its last frame.
   const filters = [`[0:v]setpts=PTS-STARTPTS,tpad=stop_mode=clone:stop_duration=1[video0]`]
@@ -75,6 +82,14 @@ export function buildFilterScript({ trace, scale }) {
     filters.push(`[cursorVideo${index}][cursor${index + 1}]overlay=x='${x}':y='${y}':${OVERLAY}:enable='${enable}'[cursorVideo${index + 1}]`)
   })
   filters.push(`[cursorVideo${moves.length}]colorspace=ispace=bt470bg:iprimaries=bt709:itrc=bt709:all=bt709:format=yuv420p[final]`)
+  if (audio) {
+    const input = clicks.length ? 3 : 2
+    const trim = number(Math.max(0, -audio.offset))
+    const delay = number(Math.max(0, audio.offset) * 1000)
+    // Reset the selected stream before trimming: its first sample may start after the container's video.
+    // Preserve real gaps before sample-count filters, tolerating normal recorder timestamp jitter.
+    filters.push(`[${input}:a:0]asetpts=PTS-STARTPTS,aresample=async=1:min_hard_comp=0.02,atrim=start=${trim},asetpts=PTS-STARTPTS,adelay=${delay}:all=1,apad[finalAudio]`)
+  }
   return filters.join(';')
 }