pitch-transcription 4.2.0 → 4.3.3

package/NOTICE

@@ -4,4 +4,5 @@ Copyright Jimmy Chen
 This package is derived from basic-pitch-ts (https://github.com/spotify/basic-pitch-ts),
 Copyright 2022 Spotify AB, licensed under the Apache License, Version 2.0. Its note
 creation, audio preparation and model call are rewritten from that source, and the
-trained model under `model/` is Spotify's Basic Pitch model, distributed unmodified.
+trained model under `model/` is Spotify's Basic Pitch model, its ONNX export `nmp.onnx` from
+`basic_pitch/saved_models/icassp_2022` in basic-pitch's repository, distributed unmodified.

package/README.md

@@ -5,7 +5,7 @@
 [![NPM downloads][badge-npm-downloads]][url-npm]
 [![NPM Unpacked Size (with version)][badge-npm-unpacked-size]][url-npm]
 
-The notes played in a recording: each note's pitch, start, length, loudness and pitch bend, heard by Spotify's Basic Pitch model on the current TensorFlow.js and written as MIDI if you want a file. It hears several notes at once, on any instrument. The trained model ships with the package, and the step that turns the model's readings into notes imports without TensorFlow.js.
+The notes played in a recording: each note's pitch, start, length, loudness and pitch bend, heard by Spotify's Basic Pitch model on ONNX Runtime and written as MIDI if you want a file. It hears several notes at once, on any instrument. The trained model ships with the package, and the step that turns the model's readings into notes imports without ONNX Runtime.
 
 ## Table of Contents
 
@@ -18,50 +18,49 @@ The notes played in a recording: each note's pitch, start, length, loudness and
 ## <a name="getting-started">🚀 Getting Started</a>
 
 ```bash
-pnpm i pitch-transcription @tensorflow/tfjs
+pnpm i pitch-transcription onnxruntime-common onnxruntime-node
 ```
 
 ```ts
-import { loadGraphModelSync } from "@tensorflow/tfjs";
-import { readFile } from "node:fs/promises";
+import { InferenceSession } from "onnxruntime-node";
+import { fileURLToPath } from "node:url";
 import {
   addPitchBends,
   convertNotesToSeconds,
   createNotes,
   MODEL_URL,
-  MODEL_WEIGHTS_URL,
   readModel,
   writeMidi,
 } from "pitch-transcription";
 
-// Under Node, the model is built from its two files; in a browser, `await loadGraphModel(servedModelUrl)` instead
-const [modelJson, weights] = await Promise.all([readFile(MODEL_URL, "utf8"), readFile(MODEL_WEIGHTS_URL)]);
-const model = loadGraphModelSync([JSON.parse(modelJson), new Uint8Array(weights).buffer]);
+// Under Node, the session reads the shipped model file; in a browser, `InferenceSession.create(servedModelUrl)` from `onnxruntime-web` instead
+const session = await InferenceSession.create(fileURLToPath(MODEL_URL));
 // One channel at 22050 Hz as a Float32Array, or an AudioBuffer at 22050 Hz with any number of channels
-const readings = await readModel(model, samples, (_windowReadings, progress) => console.log(progress));
+const readings = await readModel(session, samples, (_windowReadings, progress) => console.log(progress));
 const notes = createNotes(readings, { minNoteLength: 11 });
 const bentNotes = addPitchBends(readings.contours, notes);
 const timedNotes = convertNotesToSeconds(bentNotes);
 // Each note's startTimeSeconds, durationSeconds, pitchMidi, amplitude and pitchBends
 const midiBytes = writeMidi(timedNotes);
+await session.release();
 ```
 
-A worker or a server that already holds the readings imports note creation from `pitch-transcription/notes`, which never loads TensorFlow.js.
+A worker or a server that already holds the readings imports note creation from `pitch-transcription/notes`, which never loads ONNX Runtime. `pitch-transcription` takes `onnxruntime-common` as a peer, and the runtime the session runs on is yours to choose: `onnxruntime-node` under Node, `onnxruntime-web` in a browser.
 
 ## <a name="documentation">📖 Documentation</a>
 
-We highly recommend you take a look at the [documentation](https://esposter.com/docs/api/modules/pitch-transcription.html) to level up, and the [package's page](https://esposter.com/docs/pitch-transcription) for how it works and the absorption of `@spotify/basic-pitch` it replaced.
+We highly recommend you take a look at the [documentation](https://github.com/Esposter/Esposter/tree/main/openspec/specs/pitch-transcription) to level up. It explains how the package works and how it absorbed `@spotify/basic-pitch`.
 
 ### Key exports
 
-| Export                           | Role                                                                                                              |
-| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
-| `readModel`                      | The model's frame, onset and contour readings over a recording, a window at a time, each window's tensors freed   |
-| `createNotes`                    | The readings turned into notes, with Basic Pitch's thresholds, inferred onsets, frequency range and melodia trick |
-| `addPitchBends`                  | Each note's bend at every frame, in contour bins of a third of a semitone                                         |
-| `convertNotesToSeconds`          | Notes timed in seconds rather than the model's frames                                                             |
-| `writeMidi`                      | Notes as a MIDI file's bytes, each bend written as its share of the bend's range                                  |
-| `MODEL_URL`, `MODEL_WEIGHTS_URL` | The shipped model's two files                                                                                     |
+| Export                  | Role                                                                                                               |
+| ----------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `readModel`             | The model's frame, onset and contour readings over a recording, a window at a time, each output disposed once read |
+| `createNotes`           | The readings turned into notes, with Basic Pitch's thresholds, inferred onsets, frequency range and melodia trick  |
+| `addPitchBends`         | Each note's bend at every frame, in contour bins of a third of a semitone                                          |
+| `convertNotesToSeconds` | Notes timed in seconds rather than the model's frames                                                              |
+| `writeMidi`             | Notes as a MIDI file's bytes, each bend written as its share of the bend's range                                   |
+| `MODEL_URL`             | The shipped model file, Basic Pitch's ONNX export                                                                  |
 
 ### Commands
 

package/dist/index.d.ts

@@ -0,0 +1,21 @@
+import { ANNOTATIONS_FPS, ANNOTATIONS_FRAMES_PER_WINDOW, ANNOTATIONS_SEMITONES, AUDIO_SAMPLE_RATE, AUDIO_WINDOW_SAMPLES, CONTOURS_TENSOR, CONTOUR_BINS, CONTOUR_BINS_PER_SEMITONE, ENERGY_TOLERANCE, FFT_HOP, FRAMES_TENSOR, FRAME_THRESHOLD, INPUT_TENSOR, MIDI_MAX_VELOCITY, MIDI_MIN_VELOCITY, MIDI_OFFSET, MIDI_PITCH_BEND_SEMITONES, MIDI_PITCH_BEND_STEPS, MIN_NOTE_LENGTH, MODEL_URL, ModelReadings, NoteCreationOptions, NoteEvent, NoteEventTime, ONSETS_TENSOR, ONSET_THRESHOLD, OVERLAPPING_FRAMES, OVERLAP_SAMPLES, PITCH_BEND_STANDARD_DEVIATION, PITCH_BEND_TOLERANCE, WINDOW_HOP_SAMPLES, WINDOW_OFFSET_SECONDS, addPitchBends, convertNotesToSeconds, createNotes, writeMidi } from "./notes.js";
+import { InferenceSession, Tensor } from "onnxruntime-common";
+export interface AudioChannels {
+  getChannelData: (channel: number) => Float32Array;
+  numberOfChannels: number;
+  sampleRate: number;
+}
+export declare const clearPitch: (energy: number[][], frame: number, pitch: number) => void;
+export declare const constrainFrequency: (readings: number[][], maxFrequency?: number, minFrequency?: number) => number[][];
+export declare const findLargestReading: (readings: number[][]) => number;
+export declare const findOnsetPeaks: (onsets: number[][], threshold: number) => [number, number][];
+export declare const inferOnsets: (onsets: number[][], frames: number[][]) => number[][];
+export declare const mixDownChannels: (audio: AudioChannels | Float32Array) => Float32Array;
+export interface AudioWindows {
+  count: number;
+  getWindow: (index: number) => Tensor;
+}
+export declare const prepareAudioWindows: (samples: Float32Array) => AudioWindows;
+export declare const readModel: (session: Pick<InferenceSession, "run">, audio: AudioChannels | Float32Array, onWindow?: (readings: ModelReadings, progress: number) => void) => Promise<ModelReadings>;
+export declare const unwrapWindowReadings: (readings: Float32Array, binCount: number) => number[][];
+export { ANNOTATIONS_FPS, ANNOTATIONS_FRAMES_PER_WINDOW, ANNOTATIONS_SEMITONES, AUDIO_SAMPLE_RATE, AUDIO_WINDOW_SAMPLES, CONTOURS_TENSOR, CONTOUR_BINS, CONTOUR_BINS_PER_SEMITONE, ENERGY_TOLERANCE, FFT_HOP, FRAMES_TENSOR, FRAME_THRESHOLD, INPUT_TENSOR, MIDI_MAX_VELOCITY, MIDI_MIN_VELOCITY, MIDI_OFFSET, MIDI_PITCH_BEND_SEMITONES, MIDI_PITCH_BEND_STEPS, MIN_NOTE_LENGTH, MODEL_URL, ModelReadings, NoteCreationOptions, NoteEvent, NoteEventTime, ONSETS_TENSOR, ONSET_THRESHOLD, OVERLAPPING_FRAMES, OVERLAP_SAMPLES, PITCH_BEND_STANDARD_DEVIATION, PITCH_BEND_TOLERANCE, WINDOW_HOP_SAMPLES, WINDOW_OFFSET_SECONDS, addPitchBends, convertNotesToSeconds, createNotes, writeMidi };

package/dist/index.js

@@ -0,0 +1,85 @@
+import { A as ONSETS_TENSOR, C as MIDI_MAX_VELOCITY, D as MIDI_PITCH_BEND_STEPS, E as MIDI_PITCH_BEND_SEMITONES, F as PITCH_BEND_TOLERANCE, I as WINDOW_HOP_SAMPLES, L as WINDOW_OFFSET_SECONDS, M as OVERLAPPING_FRAMES, N as OVERLAP_SAMPLES, O as MIN_NOTE_LENGTH, P as PITCH_BEND_STANDARD_DEVIATION, S as INPUT_TENSOR, T as MIDI_OFFSET, _ as CONTOUR_BINS_PER_SEMITONE, a as findOnsetPeaks, b as FRAMES_TENSOR, c as convertNotesToSeconds, d as ANNOTATIONS_FRAMES_PER_WINDOW, f as ANNOTATIONS_SEMITONES, g as CONTOUR_BINS, h as CONTOURS_TENSOR, i as findLargestReading, j as ONSET_THRESHOLD, k as MODEL_URL, l as addPitchBends, m as AUDIO_WINDOW_SAMPLES, n as createNotes, o as constrainFrequency, p as AUDIO_SAMPLE_RATE, r as inferOnsets, s as clearPitch, t as writeMidi, u as ANNOTATIONS_FPS, v as ENERGY_TOLERANCE, w as MIDI_MIN_VELOCITY, x as FRAME_THRESHOLD, y as FFT_HOP } from "./notes-D6NakKKJ.js";
+import { InvalidOperationError, Operation, takeOne } from "@esposter/shared";
+import { Tensor } from "onnxruntime-common";
+// src/services/mixDownChannels.ts
+const mixDownChannels = (audio) => {
+	if (audio instanceof Float32Array) return audio;
+	if (audio.sampleRate !== 22050) throw new InvalidOperationError(Operation.Read, mixDownChannels.name, `audio at ${audio.sampleRate} Hz, the model reads ${AUDIO_SAMPLE_RATE} Hz`);
+	if (audio.numberOfChannels === 1) return audio.getChannelData(0);
+	const mixed = new Float32Array(audio.getChannelData(0).length);
+	for (let channel = 0; channel < audio.numberOfChannels; channel++) {
+		const channelData = audio.getChannelData(channel);
+		for (let index = 0; index < channelData.length; index++) mixed[index] = (mixed[index] ?? 0) + (channelData[index] ?? 0) / audio.numberOfChannels;
+	}
+	return mixed;
+};
+// src/services/prepareAudioWindows.ts
+const prepareAudioWindows = (samples) => {
+	const paddedLength = OVERLAP_SAMPLES / 2 + samples.length;
+	const count = Math.ceil(paddedLength / WINDOW_HOP_SAMPLES);
+	const padded = new Float32Array((count - 1) * WINDOW_HOP_SAMPLES + AUDIO_WINDOW_SAMPLES);
+	padded.set(samples, OVERLAP_SAMPLES / 2);
+	return {
+		count,
+		getWindow: (index) => {
+			const start = index * WINDOW_HOP_SAMPLES;
+			return new Tensor("float32", padded.subarray(start, start + AUDIO_WINDOW_SAMPLES), [
+				1,
+				AUDIO_WINDOW_SAMPLES,
+				1
+			]);
+		}
+	};
+};
+// src/services/unwrapWindowReadings.ts
+const unwrapWindowReadings = (readings, binCount) => {
+	const frameCount = readings.length / binCount;
+	const rows = [];
+	for (let frame = 15; frame < frameCount - 15; frame++) rows.push([...readings.subarray(frame * binCount, (frame + 1) * binCount)]);
+	return rows;
+};
+// src/services/readModel.ts
+const readOutput = async (outputs, name, binCount) => {
+	const output = takeOne(outputs, name);
+	const data = await output.getData();
+	output.dispose();
+	if (!(data instanceof Float32Array)) throw new InvalidOperationError(Operation.Read, readOutput.name, `${name} as ${output.type}, the model writes float32`);
+	return unwrapWindowReadings(data, binCount);
+};
+const readWindow = async (session, window, remainingFrames) => {
+	const outputs = await session.run({ [INPUT_TENSOR]: window }, [
+		FRAMES_TENSOR,
+		ONSETS_TENSOR,
+		CONTOURS_TENSOR
+	]);
+	const [frames, onsets, contours] = await Promise.all([
+		readOutput(outputs, FRAMES_TENSOR, 88),
+		readOutput(outputs, ONSETS_TENSOR, 88),
+		readOutput(outputs, CONTOURS_TENSOR, 264)
+	]);
+	return {
+		contours: contours.slice(0, remainingFrames),
+		frames: frames.slice(0, remainingFrames),
+		onsets: onsets.slice(0, remainingFrames)
+	};
+};
+const readModel = async (session, audio, onWindow) => {
+	const samples = mixDownChannels(audio);
+	const frameCount = Math.floor(samples.length * 86 / AUDIO_SAMPLE_RATE);
+	const windows = prepareAudioWindows(samples);
+	const readings = {
+		contours: [],
+		frames: [],
+		onsets: []
+	};
+	for (let index = 0; index < windows.count && readings.frames.length < frameCount; index++) {
+		const remainingFrames = frameCount - readings.frames.length;
+		const windowReadings = await readWindow(session, windows.getWindow(index), remainingFrames);
+		readings.frames.push(...windowReadings.frames);
+		readings.onsets.push(...windowReadings.onsets);
+		readings.contours.push(...windowReadings.contours);
+		onWindow?.(windowReadings, readings.frames.length / frameCount);
+	}
+	return readings;
+};
+export { ANNOTATIONS_FPS, ANNOTATIONS_FRAMES_PER_WINDOW, ANNOTATIONS_SEMITONES, AUDIO_SAMPLE_RATE, AUDIO_WINDOW_SAMPLES, CONTOURS_TENSOR, CONTOUR_BINS, CONTOUR_BINS_PER_SEMITONE, ENERGY_TOLERANCE, FFT_HOP, FRAMES_TENSOR, FRAME_THRESHOLD, INPUT_TENSOR, MIDI_MAX_VELOCITY, MIDI_MIN_VELOCITY, MIDI_OFFSET, MIDI_PITCH_BEND_SEMITONES, MIDI_PITCH_BEND_STEPS, MIN_NOTE_LENGTH, MODEL_URL, ONSETS_TENSOR, ONSET_THRESHOLD, OVERLAPPING_FRAMES, OVERLAP_SAMPLES, PITCH_BEND_STANDARD_DEVIATION, PITCH_BEND_TOLERANCE, WINDOW_HOP_SAMPLES, WINDOW_OFFSET_SECONDS, addPitchBends, clearPitch, constrainFrequency, convertNotesToSeconds, createNotes, findLargestReading, findOnsetPeaks, inferOnsets, mixDownChannels, prepareAudioWindows, readModel, unwrapWindowReadings, writeMidi };

package/dist/notes-D6NakKKJ.js

@@ -0,0 +1,307 @@
+import { takeOne } from "@esposter/shared";
+// src/constants.ts
+const AUDIO_SAMPLE_RATE = 22050;
+const FFT_HOP = 256;
+const AUDIO_WINDOW_SAMPLES = AUDIO_SAMPLE_RATE * 2 - 256;
+const ANNOTATIONS_FPS = Math.floor(AUDIO_SAMPLE_RATE / 256);
+const ANNOTATIONS_FRAMES_PER_WINDOW = 172;
+const OVERLAPPING_FRAMES = 30;
+const OVERLAP_SAMPLES = 7680;
+const WINDOW_HOP_SAMPLES = 36164;
+const WINDOW_OFFSET_SECONDS = .010326077097505668;
+const MIDI_OFFSET = 21;
+const ANNOTATIONS_SEMITONES = 88;
+const CONTOUR_BINS_PER_SEMITONE = 3;
+const CONTOUR_BINS = 264;
+const INPUT_TENSOR = "serving_default_input_2:0";
+const FRAMES_TENSOR = "StatefulPartitionedCall:1";
+const ONSETS_TENSOR = "StatefulPartitionedCall:2";
+const CONTOURS_TENSOR = "StatefulPartitionedCall:0";
+const MODEL_URL = new URL("../model/nmp.onnx", import.meta.url);
+const ONSET_THRESHOLD = .5;
+const FRAME_THRESHOLD = .3;
+const MIN_NOTE_LENGTH = 11;
+const ENERGY_TOLERANCE = 11;
+const PITCH_BEND_TOLERANCE = 25;
+const PITCH_BEND_STANDARD_DEVIATION = 5;
+const MIDI_PITCH_BEND_SEMITONES = 2;
+const MIDI_PITCH_BEND_STEPS = 8192;
+const MIDI_MIN_VELOCITY = 1;
+const MIDI_MAX_VELOCITY = 127;
+// src/services/addPitchBends.ts
+const PITCH_BEND_WEIGHTS = Array.from({ length: 51 }, (_value, offset) => Math.exp(-((offset - 25) ** 2) / 50));
+const addPitchBends = (contours, notes) => notes.map((note) => {
+	const bin = (note.pitchMidi - 21) * 3;
+	const start = Math.max(bin - 25, 0);
+	const end = Math.min(bin + 25 + 1, 264);
+	const pitchBends = contours.slice(note.startFrame, note.startFrame + note.durationFrames).map((row) => {
+		let bestBin = start;
+		let bestReading = -Infinity;
+		for (let contourBin = start; contourBin < end; contourBin++) {
+			const reading = takeOne(row, contourBin) * takeOne(PITCH_BEND_WEIGHTS, contourBin - bin + 25);
+			if (reading <= bestReading) continue;
+			bestBin = contourBin;
+			bestReading = reading;
+		}
+		return bestBin - bin;
+	});
+	return {
+		...note,
+		pitchBends
+	};
+});
+// src/services/convertNotesToSeconds.ts
+const toSeconds = (frame) => frame * 256 / AUDIO_SAMPLE_RATE - WINDOW_OFFSET_SECONDS * Math.floor(frame / 172);
+const convertNotesToSeconds = (notes) => notes.map(({ amplitude, durationFrames, pitchBends, pitchMidi, startFrame }) => {
+	const startTimeSeconds = toSeconds(startFrame);
+	return {
+		amplitude,
+		durationSeconds: toSeconds(startFrame + durationFrames) - startTimeSeconds,
+		pitchBends,
+		pitchMidi,
+		startTimeSeconds
+	};
+});
+// src/services/clearPitch.ts
+const clearPitch = (energy, frame, pitch) => {
+	const row = takeOne(energy, frame);
+	row[pitch] = 0;
+	if (pitch + 1 < row.length) row[pitch + 1] = 0;
+	if (pitch > 0) row[pitch - 1] = 0;
+};
+// src/services/constrainFrequency.ts
+const toPitchIndex = (frequency) => Math.round(12 * Math.log2(frequency / 440) + 69 - 21);
+const constrainFrequency = (readings, maxFrequency, minFrequency) => {
+	if (maxFrequency === void 0 && minFrequency === void 0) return readings;
+	const end = maxFrequency === void 0 ? Infinity : toPitchIndex(maxFrequency);
+	const start = minFrequency === void 0 ? 0 : toPitchIndex(minFrequency);
+	return readings.map((row) => row.map((reading, pitch) => pitch >= start && pitch < end ? reading : 0));
+};
+// src/services/findOnsetPeaks.ts
+const findOnsetPeaks = (onsets, threshold) => {
+	const peaks = [];
+	for (let frame = 1; frame < onsets.length - 1; frame++) {
+		const previous = takeOne(onsets, frame - 1);
+		const row = takeOne(onsets, frame);
+		const next = takeOne(onsets, frame + 1);
+		for (let pitch = 0; pitch < row.length; pitch++) {
+			const reading = takeOne(row, pitch);
+			if (reading > threshold && reading > takeOne(previous, pitch) && reading > takeOne(next, pitch)) peaks.push([frame, pitch]);
+		}
+	}
+	return peaks;
+};
+// src/services/findLargestReading.ts
+const findLargestReading = (readings) => {
+	let largest = 0;
+	for (const row of readings) for (const reading of row) largest = Math.max(largest, reading);
+	return largest;
+};
+// src/services/inferOnsets.ts
+const getJump = (row, previous, beforePrevious, pitch) => {
+	const reading = takeOne(row, pitch);
+	return Math.max(0, Math.min(reading - takeOne(previous, pitch), reading - takeOne(beforePrevious, pitch)));
+};
+const inferOnsets = (onsets, frames) => {
+	let largestJump = 0;
+	for (let frame = 2; frame < frames.length; frame++) {
+		const row = takeOne(frames, frame);
+		const previous = takeOne(frames, frame - 1);
+		const beforePrevious = takeOne(frames, frame - 2);
+		for (let pitch = 0; pitch < row.length; pitch++) largestJump = Math.max(largestJump, getJump(row, previous, beforePrevious, pitch));
+	}
+	if (largestJump === 0) return onsets;
+	const scale = findLargestReading(onsets) / largestJump;
+	return onsets.map((onsetRow, frame) => {
+		const row = takeOne(frames, frame);
+		const previous = frame < 2 ? row : takeOne(frames, frame - 1);
+		const beforePrevious = frame < 2 ? row : takeOne(frames, frame - 2);
+		return onsetRow.map((reading, pitch) => Math.max(reading, scale * getJump(row, previous, beforePrevious, pitch)));
+	});
+};
+// src/services/createNotes.ts
+const getMeanReading = (frames, start, end, pitch) => {
+	let sum = 0;
+	for (let frame = start; frame < end; frame++) sum += takeOne(takeOne(frames, frame), pitch);
+	return sum / (end - start);
+};
+const createNotes = ({ frames, onsets }, { energyTolerance = 11, frameThreshold = FRAME_THRESHOLD, isInferringOnsets = true, isMelodiaTrick = true, maxFrequency, minFrequency, minNoteLength = 11, onsetThreshold = ONSET_THRESHOLD } = {}) => {
+	const frameCount = frames.length;
+	const constrainedFrames = constrainFrequency(frames, maxFrequency, minFrequency);
+	const constrainedOnsets = constrainFrequency(onsets, maxFrequency, minFrequency);
+	const peakOnsets = isInferringOnsets ? inferOnsets(constrainedOnsets, constrainedFrames) : constrainedOnsets;
+	const energy = constrainedFrames.map((row) => [...row]);
+	const notes = [];
+	for (const [startFrame, pitch] of findOnsetPeaks(peakOnsets, onsetThreshold).toReversed()) {
+		if (startFrame >= frameCount - 1) continue;
+		let end = startFrame + 1;
+		let quietFrames = 0;
+		for (; end < frameCount - 1 && quietFrames < energyTolerance; end++) quietFrames = takeOne(takeOne(energy, end), pitch) < frameThreshold ? quietFrames + 1 : 0;
+		end -= quietFrames;
+		if (end - startFrame <= minNoteLength) continue;
+		for (let frame = startFrame; frame < end; frame++) clearPitch(energy, frame, pitch);
+		notes.push({
+			amplitude: getMeanReading(constrainedFrames, startFrame, end, pitch),
+			durationFrames: end - startFrame,
+			pitchMidi: pitch + 21,
+			startFrame
+		});
+	}
+	if (!isMelodiaTrick) return notes;
+	const candidates = [];
+	for (const [frame, row] of energy.entries()) for (let pitch = 0; pitch < row.length; pitch++) {
+		const reading = takeOne(row, pitch);
+		if (reading > frameThreshold) candidates.push([
+			reading,
+			frame,
+			pitch
+		]);
+	}
+	const sortedCandidates = candidates.toSorted(([firstReading, firstFrame, firstPitch], [secondReading, secondFrame, secondPitch]) => secondReading - firstReading || firstFrame - secondFrame || firstPitch - secondPitch);
+	for (const [reading, middle, pitch] of sortedCandidates) {
+		const middleRow = takeOne(energy, middle);
+		if (takeOne(middleRow, pitch) !== reading) continue;
+		middleRow[pitch] = 0;
+		let frame = middle + 1;
+		let quietFrames = 0;
+		for (; frame < frameCount - 1 && quietFrames < energyTolerance; frame++) {
+			quietFrames = takeOne(takeOne(energy, frame), pitch) < frameThreshold ? quietFrames + 1 : 0;
+			clearPitch(energy, frame, pitch);
+		}
+		const end = frame - 1 - quietFrames;
+		frame = middle - 1;
+		quietFrames = 0;
+		for (; frame > 0 && quietFrames < energyTolerance; frame--) {
+			quietFrames = takeOne(takeOne(energy, frame), pitch) < frameThreshold ? quietFrames + 1 : 0;
+			clearPitch(energy, frame, pitch);
+		}
+		const start = frame + 1 + quietFrames;
+		if (end - start <= minNoteLength) continue;
+		notes.push({
+			amplitude: getMeanReading(constrainedFrames, start, end, pitch),
+			durationFrames: end - start,
+			pitchMidi: pitch + 21,
+			startFrame: start
+		});
+	}
+	return notes;
+};
+// src/services/writeMidi.ts
+const MIDI_TICKS_PER_BEAT = 480;
+const MIDI_TICKS_PER_SECOND = MIDI_TICKS_PER_BEAT * 2;
+const writeVariableLength = (value) => {
+	const bytes = [value & 127];
+	for (let rest = value >> 7; rest > 0; rest >>= 7) bytes.unshift(rest & 127 | 128);
+	return bytes;
+};
+const toTicks = (seconds) => Math.round(seconds * MIDI_TICKS_PER_SECOND);
+const toVelocity = (amplitude) => {
+	const velocity = Math.floor(amplitude * 127);
+	return Math.min(127, Math.max(1, velocity));
+};
+const toUint16 = (value) => [value >> 8 & 255, value & 255];
+const toUint32 = (value) => [
+	value >>> 24 & 255,
+	value >> 16 & 255,
+	value >> 8 & 255,
+	value & 255
+];
+const writeChunk = (type, data) => [
+	...type,
+	...toUint32(data.length),
+	...data
+];
+const HEADER_CHUNK_TYPE = [
+	77,
+	84,
+	104,
+	100
+];
+const TRACK_CHUNK_TYPE = [
+	77,
+	84,
+	114,
+	107
+];
+const EMPTY_TRACK_NAME = [
+	255,
+	3,
+	0
+];
+const END_OF_TRACK = [
+	255,
+	47,
+	0
+];
+const PIANO_PROGRAM_CHANGE = [192, 0];
+const writeTrack = (events) => {
+	const data = [];
+	let lastTick = 0;
+	for (const { bytes, tick } of events.toSorted((firstEvent, secondEvent) => firstEvent.tick - secondEvent.tick)) {
+		data.push(...writeVariableLength(tick - lastTick), ...bytes);
+		lastTick = tick;
+	}
+	data.push(0, ...END_OF_TRACK);
+	return writeChunk(TRACK_CHUNK_TYPE, data);
+};
+const writeNoteEvents = (notes) => notes.map(({ amplitude, durationSeconds, pitchMidi, startTimeSeconds }) => ({
+	amplitude,
+	offTick: toTicks(startTimeSeconds) + toTicks(durationSeconds),
+	onTick: toTicks(startTimeSeconds),
+	pitchMidi
+})).toSorted((firstNote, secondNote) => firstNote.onTick - secondNote.onTick).flatMap(({ amplitude, offTick, onTick, pitchMidi }) => [{
+	bytes: [
+		144,
+		pitchMidi,
+		toVelocity(amplitude)
+	],
+	tick: onTick
+}, {
+	bytes: [
+		128,
+		pitchMidi,
+		0
+	],
+	tick: offTick
+}]);
+const writePitchBendEvents = (notes) => notes.flatMap(({ durationSeconds, pitchBends = [], startTimeSeconds }) => pitchBends.map((bend, index) => {
+	const share = bend / 3 / 2;
+	const value = MIDI_PITCH_BEND_STEPS + Math.max(-8192, Math.min(8191, Math.round(share * MIDI_PITCH_BEND_STEPS)));
+	return {
+		bytes: [
+			224,
+			value & 127,
+			value >> 7 & 127
+		],
+		tick: toTicks(startTimeSeconds + index * durationSeconds / pitchBends.length)
+	};
+}));
+const writeMidi = (notes) => {
+	const header = writeChunk(HEADER_CHUNK_TYPE, [
+		...toUint16(1),
+		...toUint16(2),
+		...toUint16(MIDI_TICKS_PER_BEAT)
+	]);
+	const tempoTrack = writeTrack([{
+		bytes: EMPTY_TRACK_NAME,
+		tick: 0
+	}]);
+	const noteTrack = writeTrack([
+		{
+			bytes: EMPTY_TRACK_NAME,
+			tick: 0
+		},
+		{
+			bytes: PIANO_PROGRAM_CHANGE,
+			tick: 0
+		},
+		...writeNoteEvents(notes),
+		...writePitchBendEvents(notes)
+	]);
+	return new Uint8Array([
+		...header,
+		...tempoTrack,
+		...noteTrack
+	]);
+};
+export { ONSETS_TENSOR as A, MIDI_MAX_VELOCITY as C, MIDI_PITCH_BEND_STEPS as D, MIDI_PITCH_BEND_SEMITONES as E, PITCH_BEND_TOLERANCE as F, WINDOW_HOP_SAMPLES as I, WINDOW_OFFSET_SECONDS as L, OVERLAPPING_FRAMES as M, OVERLAP_SAMPLES as N, MIN_NOTE_LENGTH as O, PITCH_BEND_STANDARD_DEVIATION as P, INPUT_TENSOR as S, MIDI_OFFSET as T, CONTOUR_BINS_PER_SEMITONE as _, findOnsetPeaks as a, FRAMES_TENSOR as b, convertNotesToSeconds as c, ANNOTATIONS_FRAMES_PER_WINDOW as d, ANNOTATIONS_SEMITONES as f, CONTOUR_BINS as g, CONTOURS_TENSOR as h, findLargestReading as i, ONSET_THRESHOLD as j, MODEL_URL as k, addPitchBends as l, AUDIO_WINDOW_SAMPLES as m, createNotes as n, constrainFrequency as o, AUDIO_SAMPLE_RATE as p, inferOnsets as r, clearPitch as s, writeMidi as t, ANNOTATIONS_FPS as u, ENERGY_TOLERANCE as v, MIDI_MIN_VELOCITY as w, FRAME_THRESHOLD as x, FFT_HOP as y };

package/dist/notes.d.ts

@@ -0,0 +1,62 @@
+export declare const AUDIO_SAMPLE_RATE = 22050;
+export declare const FFT_HOP = 256;
+export declare const AUDIO_WINDOW_SAMPLES: number;
+export declare const ANNOTATIONS_FPS: number;
+export declare const ANNOTATIONS_FRAMES_PER_WINDOW: number;
+export declare const OVERLAPPING_FRAMES = 30;
+export declare const OVERLAP_SAMPLES: number;
+export declare const WINDOW_HOP_SAMPLES: number;
+export declare const WINDOW_OFFSET_SECONDS: number;
+export declare const MIDI_OFFSET = 21;
+export declare const ANNOTATIONS_SEMITONES = 88;
+export declare const CONTOUR_BINS_PER_SEMITONE = 3;
+export declare const CONTOUR_BINS: number;
+export declare const INPUT_TENSOR = "serving_default_input_2:0";
+export declare const FRAMES_TENSOR = "StatefulPartitionedCall:1";
+export declare const ONSETS_TENSOR = "StatefulPartitionedCall:2";
+export declare const CONTOURS_TENSOR = "StatefulPartitionedCall:0";
+export declare const MODEL_URL: URL;
+export declare const ONSET_THRESHOLD = .5;
+export declare const FRAME_THRESHOLD = .3;
+export declare const MIN_NOTE_LENGTH = 11;
+export declare const ENERGY_TOLERANCE = 11;
+export declare const PITCH_BEND_TOLERANCE = 25;
+export declare const PITCH_BEND_STANDARD_DEVIATION = 5;
+export declare const MIDI_PITCH_BEND_SEMITONES = 2;
+export declare const MIDI_PITCH_BEND_STEPS: number;
+export declare const MIDI_MIN_VELOCITY = 1;
+export declare const MIDI_MAX_VELOCITY = 127;
+interface ModelReadings {
+  contours: number[][];
+  frames: number[][];
+  onsets: number[][];
+}
+interface NoteCreationOptions {
+  energyTolerance?: number;
+  frameThreshold?: number;
+  isInferringOnsets?: boolean;
+  isMelodiaTrick?: boolean;
+  maxFrequency?: number;
+  minFrequency?: number;
+  minNoteLength?: number;
+  onsetThreshold?: number;
+}
+interface NoteEvent {
+  amplitude: number;
+  durationFrames: number;
+  pitchBends?: number[];
+  pitchMidi: number;
+  startFrame: number;
+}
+interface NoteEventTime {
+  amplitude: number;
+  durationSeconds: number;
+  pitchBends?: number[];
+  pitchMidi: number;
+  startTimeSeconds: number;
+}
+export declare const addPitchBends: (contours: number[][], notes: NoteEvent[]) => NoteEvent[];
+export declare const convertNotesToSeconds: (notes: NoteEvent[]) => NoteEventTime[];
+export declare const createNotes: ({ frames, onsets }: Pick<ModelReadings, "frames" | "onsets">, { energyTolerance, frameThreshold, isInferringOnsets, isMelodiaTrick, maxFrequency, minFrequency, minNoteLength, onsetThreshold }?: NoteCreationOptions) => NoteEvent[];
+export declare const writeMidi: (notes: NoteEventTime[]) => Uint8Array;
+export type { ModelReadings, NoteCreationOptions, NoteEvent, NoteEventTime };

package/dist/notes.js

@@ -0,0 +1 @@
+export { u as ANNOTATIONS_FPS, d as ANNOTATIONS_FRAMES_PER_WINDOW, f as ANNOTATIONS_SEMITONES, p as AUDIO_SAMPLE_RATE, m as AUDIO_WINDOW_SAMPLES, h as CONTOURS_TENSOR, g as CONTOUR_BINS, _ as CONTOUR_BINS_PER_SEMITONE, v as ENERGY_TOLERANCE, y as FFT_HOP, b as FRAMES_TENSOR, x as FRAME_THRESHOLD, S as INPUT_TENSOR, C as MIDI_MAX_VELOCITY, w as MIDI_MIN_VELOCITY, T as MIDI_OFFSET, E as MIDI_PITCH_BEND_SEMITONES, D as MIDI_PITCH_BEND_STEPS, O as MIN_NOTE_LENGTH, k as MODEL_URL, A as ONSETS_TENSOR, j as ONSET_THRESHOLD, M as OVERLAPPING_FRAMES, N as OVERLAP_SAMPLES, P as PITCH_BEND_STANDARD_DEVIATION, F as PITCH_BEND_TOLERANCE, I as WINDOW_HOP_SAMPLES, L as WINDOW_OFFSET_SECONDS, l as addPitchBends, c as convertNotesToSeconds, n as createNotes, t as writeMidi } from "./notes-D6NakKKJ.js";

package/package.json

@@ -1,15 +1,16 @@
 {
   "name": "pitch-transcription",
-  "version": "4.2.0",
-  "description": "The pitched notes in a recording — Spotify's Basic Pitch model on current TensorFlow.js, its readings turned into notes with pitch bends and written as MIDI.",
+  "version": "4.3.3",
+  "description": "The pitched notes in a recording — Spotify's Basic Pitch model on ONNX Runtime, its readings turned into notes with pitch bends and written as MIDI.",
   "keywords": [
     "audio",
     "basic-pitch",
     "midi",
     "music",
+    "onnx",
+    "onnxruntime",
     "pitch",
     "polyphonic",
-    "tensorflow",
     "transcription",
     "typescript"
   ],
@@ -45,28 +46,23 @@
   "scripts": {
     "bench": "vitest bench --run",
     "build": "tsdown",
-    "format": "oxfmt",
-    "format:check": "oxfmt --check",
     "lint": "eslint .",
     "lint:fix": "eslint --fix .",
     "test": "vitest",
     "typecheck": "tsc"
   },
   "dependencies": {
-    "@esposter/shared": "4.2.0"
+    "@esposter/shared": "4.3.3"
   },
   "devDependencies": {
-    "@arethetypeswrong/core": "^0.18.5",
-    "@esposter/configuration": "4.2.0",
-    "@esposter/shared-node": "4.2.0",
-    "@tensorflow/tfjs": "^4.22.0",
+    "@esposter/configuration": "4.3.3",
+    "@esposter/shared-node": "4.3.3",
     "@types/node": "^26.6.5",
-    "publint": "^0.3.25",
-    "tsdown": "^0.23.1",
-    "vitest": "^5.0.4"
+    "onnxruntime-node": "^1.30.0",
+    "tsdown": "^0.23.1"
   },
   "peerDependencies": {
-    "@tensorflow/tfjs": "^4.22.0"
+    "onnxruntime-common": "^1.30.0"
   },
-  "gitHead": "fef4971cf6a621a2c1850f32fcec19ee4ec0de99"
+  "gitHead": "955bbc130ad9b28877fd4ae8efc73a045141ce01"
 }