@kairotic/analysis 0.3.0 → 0.7.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +24 -3
- package/dist/chroma.mjs +2 -2
- package/dist/features.mjs +2 -2
- package/dist/form.mjs +9 -0
- package/dist/index.mjs +21 -9
- package/dist/monitor.mjs +1 -1
- package/dist/onset.mjs +3 -3
- package/dist/pitch.mjs +2 -2
- package/dist/realize.mjs +9 -0
- package/dist/spectrum.mjs +2 -2
- package/dist/transcribe.mjs +3 -3
- package/package.json +5 -3
- package/src/chroma.cljs +5 -3
- package/src/features.cljs +3 -1
- package/src/form.cljs +138 -0
- package/src/onset.cljs +25 -13
- package/src/pitch.cljs +1 -0
- package/src/realize.cljs +218 -0
- package/src/spectrum.cljs +5 -3
- package/src/transcribe.cljs +110 -18
package/README.md
CHANGED
|
@@ -8,10 +8,12 @@ From any signal — an `adc~`, a `sample` played by a `reader~`, a `faust~` —
|
|
|
8
8
|
|---|---|
|
|
9
9
|
| `features~ [ms] [descriptor …]` | one outlet per descriptor written after it, every ms (20): `rms`, `energy`, `zcr`, `loudness`, `centroid` and `spread` (Hz), `flatness`, `rolloff` (Hz), `kurtosis`, `skewness`, `slope`, `crest`, `sharpness`, `perceptualSpread`, `mfcc` (13), `chroma` (12). Without words: `rms centroid flatness` |
|
|
10
10
|
| `pitch~ [lo hi] [threshold]` | the frequency in Hz (0 once when it stops hearing one) and how clearly periodic it is, by YIN, between lo and hi (60 – 1500 Hz) |
|
|
11
|
-
| `onset~ [threshold] [ms]` | a bang at the beginning of each sound (a note, a stroke, a syllable) and
|
|
11
|
+
| `onset~ [threshold] [ms]` | a bang at the beginning of each sound (a note, a stroke, a syllable), its strength, and the growth of the spectrum of every frame (the onset strength envelope). An onset is reported about 6 ms after it starts (one hop at most, measured); a sound cut at once is not one |
|
|
12
12
|
| `chroma~ [ms]` | 12 values from C to B and the strongest pitch class as a word (`:a`, `:c#`…) |
|
|
13
13
|
| `spectrum~ [lo hi]` | the spectrum drawn in the patch, on a log scale of frequency |
|
|
14
|
-
| `transcribe [bpm] [notes\|glide] [ms]` | what `pitch~` hears (and `onset~` splits), written into a `freqt` as strokes:
|
|
14
|
+
| `transcribe [bpm] [notes\|glide] [ms]` | what `pitch~` hears (and `onset~` splits), written into a `freqt` as strokes: notes where the contour settles on another pitch (a slide, a vibrato or an ornament that passes does not start notes), each at its median pitch, or contours that follow the voice; times in beats at bpm. `rec` starts from 0, `rec t` from beat t, `ovr t` over only what the take covers, `stop` ends |
|
|
15
|
+
| `form~ [seconds] [step] [kernel]` | how alike the moments of what arrives are (a chroma, mfcc, onset~'s envelope) over the last seconds (20): a self-similarity matrix of one frame every step (0.25 s), drawn; the novelty of each frame (how much the music changes there), a bang and `[:segment {:t :at :length :period}]` where a segment begins, and the period that repeats most, in seconds. `[:seconds x]` `[:step x]` `[:kernel x]` change it, `clear` forgets |
|
|
16
|
+
| `realize~ [settings] [midi]` | plays a `faust~` from descriptors ([Study 37](../../studies/37-notation/)): loudness (dB or a level), brightness (Hz), noisiness (0…1) and pitch (Hz, or MIDI with `midi`) in, `[:slider value]` for the `faust~` out — its `pitch`/`freq` and `gain` sliders follow pitch and loudness, the others take the settings of that `faust~` that sound nearest. It learns the `faust~` its outlet reaches first (steady settings played offline and heard, `@kairotic/faust-units`' `notation.mjs`), on the first values or `learn`. With `features~` of the `faust~`'s sound on its last three inlets (rms, centroid, flatness heard) it listens to what it plays and corrects what it asks, little by little. On the right, how far the nearest setting is (above 1: beyond that `faust~`) |
|
|
15
17
|
| `monitor [seconds] [name …]` | whatever arrives, drawn over the last seconds (6), one lane per inlet: a lane for each name written (`monitor 6 rms centroid chroma` has 3 inlets), or one. A number is a line in the range it has had in the window, a list of numbers a strip with a row per element (chroma, mfcc), a bang or a word a mark. It sends nothing |
|
|
16
18
|
|
|
17
19
|
Each sends only what changed. `pitch~` runs YIN in its own AudioWorklet; the others take frames of 2048
|
|
@@ -38,6 +40,25 @@ To watch what a sound does over time, several descriptors side by side:
|
|
|
38
40
|
```
|
|
39
41
|
|
|
40
42
|
An improvisation into notation: `adc~` → `pitch~` → `transcribe` → `freqt`, with `onset~` on the right
|
|
41
|
-
of `transcribe
|
|
43
|
+
of `transcribe` (its second outlet, the strength: each note is then written with the weight of its attack); `rec`,
|
|
44
|
+
play, `stop`; then a `quantize` box (the freqt's second outlet on its right inlet; `:time :track` to find its beats, a bang)
|
|
45
|
+
writes it in measures as a tree for a `tree` box — or `(track! :bN)` and `(view :bN :score)` on the freqt to preview it, or `(keep! …)` and
|
|
46
|
+
`mt`.
|
|
47
|
+
|
|
48
|
+
The form of what is played, and onsets that adapt to it: `form~` on a `chroma~`, its fourth outlet into
|
|
49
|
+
`onset~` (each `[:segment …]` makes `onset~` forget the usual change of the section before, so a quiet
|
|
50
|
+
section after a loud one is heard), or into a message box that sends `onset~` a `[:threshold x]` of its own.
|
|
51
|
+
The SSM, novelty and beat spectrum are those of the core (`kairotic.structure`, issue #120), in the API for any
|
|
52
|
+
component: `api/ssm`, `api/novelty`, `api/novelty-at`, `api/beat-spectrum`, `api/periods`, `api/boundaries`, and
|
|
53
|
+
`api/note-frames` for notes (a take, a tree) instead of sound.
|
|
54
|
+
|
|
55
|
+
```clojure
|
|
56
|
+
(box :b1 "adc~ 1" 40 30)
|
|
57
|
+
(box :b2 "chroma~" 40 90)
|
|
58
|
+
(box :b3 "onset~" 200 90)
|
|
59
|
+
(box :b4 "form~" 40 150)
|
|
60
|
+
(connect :b1 0 :b2 0) (connect :b1 0 :b3 0)
|
|
61
|
+
(connect :b2 0 :b4 0) (connect :b4 3 :b3 0)
|
|
62
|
+
```
|
|
42
63
|
|
|
43
64
|
Writing one of its classes in a work brings the package into it. Licence: [MPL-2.0](../../LICENSE).
|
package/dist/chroma.mjs
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
// Generated by packages/build.mjs from src/chroma.cljs
|
|
2
2
|
export default {
|
|
3
3
|
"class": "chroma~",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.7.5",
|
|
5
5
|
"author": "bonuoq",
|
|
6
6
|
"license": "MPL-2.0",
|
|
7
7
|
"doc": "How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.",
|
|
8
|
-
"source": ";; chroma~ · how much of each pitch class a sound has\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The chroma is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n gate 0.005 ; rms under it is silence: no strongest class\n classes [:c :c# :d :d# :e :f :f# :g :g# :a :a# :b]\n names [\"C\" \"C#\" \"D\" \"D#\" \"E\" \"F\" \"F#\" \"G\" \"G#\" \"A\" \"A#\" \"B\"]\n strongest (fn [xs] (when (seq xs) (nth classes (first (apply max-key (fn [[_ v]] v) (map-indexed vector xs))))))\n ;; a picture of 12 bars in 168 × 64\n bar-w 13 x0 6 base 46 tall 38]\n {;; ── about ──\n :class \"chroma~\"\n :license \"MPL-2.0\"\n :doc \"How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.\"\n :usage \"chroma~ [ms]\"\n :io [1 2]\n :size [168 64]\n :args [[:ms 40]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value) (> (or (:rms value) 0) gate) (= 12 (count (:chroma value))))\n (let [xs (:chroma value) top (strongest xs)\n outs (cond-> []\n (not= top (:top state)) (conj [1 top])\n (not= xs (:xs state)) (conj [0 xs]))]\n (if (seq outs) {:state {:xs xs :top top} :out outs :show xs} {}))\n :else {}))\n\n ;; ── view ──\n :draw (fn [{:keys [shown]}]\n (let [xs (if (and (sequential? shown) (= 12 (count shown))) shown (repeat 12 0))\n top (when (some pos? xs) (apply max xs))]\n (into [[:rect {:x 2 :y 2 :w 164 :h 60 :r 4 :stroke :grid :fill :bg}]]\n (mapcat (fn [i v n]\n (let [h (* tall (api/clamp v 0 1)) x (+ x0 (* i bar-w))]\n [[:rect {:x x :y (- base h) :w (- bar-w 3) :h h :r 1 :stroke nil\n :fill (if (and top (= v top)) :kairos :accent)}]\n [:text {:x (+ x (/ (- bar-w 3) 2)) :y 58 :anchor :middle :size 7 :fill :soft} n]]))\n (range 12) xs names))))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n ms (api/clamp (or (first (filter number? (:args b))) 40) 10 2000)\n node {:in in :M (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n (when-let [M @(:M node)]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"chroma\" \"rms\"] (first data))\n xs (mapv (fn [x] (/ (js/Math.round (* 100 (if (js/isFinite x) x 0))) 100))\n (array-seq (.-chroma r)))]\n (api/report! (:id b) {:chroma xs :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"chroma~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ ms 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (.connect in w) (
|
|
8
|
+
"source": ";; chroma~ · how much of each pitch class a sound has\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The chroma is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n gate 0.005 ; rms under it is silence: no strongest class\n classes [:c :c# :d :d# :e :f :f# :g :g# :a :a# :b]\n names [\"C\" \"C#\" \"D\" \"D#\" \"E\" \"F\" \"F#\" \"G\" \"G#\" \"A\" \"A#\" \"B\"]\n strongest (fn [xs] (when (seq xs) (nth classes (first (apply max-key (fn [[_ v]] v) (map-indexed vector xs))))))\n ;; a picture of 12 bars in 168 × 64\n bar-w 13 x0 6 base 46 tall 38]\n {;; ── about ──\n :class \"chroma~\"\n :license \"MPL-2.0\"\n :doc \"How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.\"\n :usage \"chroma~ [ms]\"\n :io [1 2]\n :size [168 64]\n :args [[:ms 40]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value) (> (or (:rms value) 0) gate) (= 12 (count (:chroma value))))\n (let [xs (:chroma value) top (strongest xs)\n outs (cond-> []\n (not= top (:top state)) (conj [1 top])\n (not= xs (:xs state)) (conj [0 xs]))]\n (if (seq outs) {:state {:xs xs :top top} :out outs :show xs} {}))\n :else {}))\n\n ;; ── view ──\n :draw (fn [{:keys [shown]}]\n (let [xs (if (and (sequential? shown) (= 12 (count shown))) shown (repeat 12 0))\n top (when (some pos? xs) (apply max xs))]\n (into [[:rect {:x 2 :y 2 :w 164 :h 60 :r 4 :stroke :grid :fill :bg}]]\n (mapcat (fn [i v n]\n (let [h (* tall (api/clamp v 0 1)) x (+ x0 (* i bar-w))]\n [[:rect {:x x :y (- base h) :w (- bar-w 3) :h h :r 1 :stroke nil\n :fill (if (and top (= v top)) :kairos :accent)}]\n [:text {:x (+ x (/ (- bar-w 3) 2)) :y 58 :anchor :middle :size 7 :fill :soft} n]]))\n (range 12) xs names))))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n ms (api/clamp (or (first (filter number? (:args b))) 40) 10 2000)\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"chroma\" \"rms\"] (first data))\n xs (mapv (fn [x] (/ (js/Math.round (* 100 (if (js/isFinite x) x 0))) 100))\n (array-seq (.-chroma r)))]\n (api/report! (:id b) {:chroma xs :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"chroma~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ ms 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w)) (api/untap w))))\n (.catch (fn [e] (js/console.error \"chroma~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
9
9
|
};
|
package/dist/features.mjs
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
// Generated by packages/build.mjs from src/features.cljs
|
|
2
2
|
export default {
|
|
3
3
|
"class": "features~",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.7.5",
|
|
5
5
|
"author": "bonuoq",
|
|
6
6
|
"license": "MPL-2.0",
|
|
7
7
|
"doc": "Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.",
|
|
8
|
-
"source": ";; features~ · descriptors of a sound, every few milliseconds\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The descriptors are Meyda's\n;; (MIT, meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n ;; every analysis box of the package uses frames of this size: Meyda is one object for the whole\n ;; page and keeps some filter banks for the size it saw first\n size 2048\n aliases {:centroid :spectralCentroid :flatness :spectralFlatness :rolloff :spectralRolloff\n :spread :spectralSpread :kurtosis :spectralKurtosis :skewness :spectralSkewness\n :slope :spectralSlope :crest :spectralCrest :sharpness :perceptualSharpness}\n known #{:rms :energy :zcr :loudness :mfcc :chroma :spectralCentroid :spectralFlatness :spectralRolloff\n :spectralSpread :spectralKurtosis :spectralSkewness :spectralSlope :spectralCrest\n :perceptualSharpness :perceptualSpread}\n feats (fn [args]\n (let [ws (->> args (filter keyword?) (map (fn [k] (get aliases k k))) (filter known) distinct vec)]\n (if (seq ws) ws [:rms :spectralCentroid :spectralFlatness])))\n hop-ms (fn [args] (api/clamp (or (first (filter number? args)) 20) 5 1000))\n ok? (fn [x] (and (number? x) (js/isFinite x)))\n ;; a value as it leaves the box: Hz where Meyda gives bins, the total of loudness, lists rounded\n value (fn [k v sr]\n (cond\n (#{:spectralCentroid :spectralSpread} k) (when (ok? v) (api/nice (* v (/ sr size))))\n (= k :loudness) (when v (let [t (.-total v)] (when (ok? t) (api/nice t))))\n (#{:mfcc :chroma} k) (when v (let [xs (vec (array-seq v))] (when (every? ok? xs) (mapv api/nice xs))))\n :else (when (ok? v) (api/nice v))))]\n {;; ── about ──\n :class \"features~\"\n :license \"MPL-2.0\"\n :doc \"Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.\"\n :usage \"features~ [ms] [descriptor …]\"\n :io (fn [b] [1 (count (feats (:args b)))])\n :dynamic-io true\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value))\n (let [fs (feats args)\n outs (vec (keep-indexed (fn [i k] (let [v (get value k)]\n (when (and (some? v) (not= v (get state k))) [i v])))\n fs))]\n (if (seq outs) {:state (merge state (select-keys value fs)) :out outs} {}))\n :else {}))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n fs (feats (:args b))\n names (clj->js (mapv name fs))\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n (when-let [M @(:M node)]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M names (first data))]\n (api/report! (:id b) (into {} (for [k fs] [k (value k (aget r (name k)) sr)])) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"features~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ (hop-ms (:args b)) 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"features~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
8
|
+
"source": ";; features~ · descriptors of a sound, every few milliseconds\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The descriptors are Meyda's\n;; (MIT, meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n ;; every analysis box of the package uses frames of this size: Meyda is one object for the whole\n ;; page and keeps some filter banks for the size it saw first\n size 2048\n aliases {:centroid :spectralCentroid :flatness :spectralFlatness :rolloff :spectralRolloff\n :spread :spectralSpread :kurtosis :spectralKurtosis :skewness :spectralSkewness\n :slope :spectralSlope :crest :spectralCrest :sharpness :perceptualSharpness}\n known #{:rms :energy :zcr :loudness :mfcc :chroma :spectralCentroid :spectralFlatness :spectralRolloff\n :spectralSpread :spectralKurtosis :spectralSkewness :spectralSlope :spectralCrest\n :perceptualSharpness :perceptualSpread}\n feats (fn [args]\n (let [ws (->> args (filter keyword?) (map (fn [k] (get aliases k k))) (filter known) distinct vec)]\n (if (seq ws) ws [:rms :spectralCentroid :spectralFlatness])))\n hop-ms (fn [args] (api/clamp (or (first (filter number? args)) 20) 5 1000))\n ok? (fn [x] (and (number? x) (js/isFinite x)))\n ;; a value as it leaves the box: Hz where Meyda gives bins, the total of loudness, lists rounded\n value (fn [k v sr]\n (cond\n (#{:spectralCentroid :spectralSpread} k) (when (ok? v) (api/nice (* v (/ sr size))))\n (= k :loudness) (when v (let [t (.-total v)] (when (ok? t) (api/nice t))))\n (#{:mfcc :chroma} k) (when v (let [xs (vec (array-seq v))] (when (every? ok? xs) (mapv api/nice xs))))\n :else (when (ok? v) (api/nice v))))]\n {;; ── about ──\n :class \"features~\"\n :license \"MPL-2.0\"\n :doc \"Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.\"\n :usage \"features~ [ms] [descriptor …]\"\n :io (fn [b] [1 (count (feats (:args b)))])\n :dynamic-io true\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value))\n (let [fs (feats args)\n outs (vec (keep-indexed (fn [i k] (let [v (get value k)]\n (when (and (some? v) (not= v (get state k))) [i v])))\n fs))]\n (if (seq outs) {:state (merge state (select-keys value fs)) :out outs} {}))\n :else {}))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n fs (feats (:args b))\n names (clj->js (mapv name fs))\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M names (first data))]\n (api/report! (:id b) (into {} (for [k fs] [k (value k (aget r (name k)) sr)])) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"features~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ (hop-ms (:args b)) 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"features~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
9
9
|
};
|
package/dist/form.mjs
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
// Generated by packages/build.mjs from src/form.cljs
|
|
2
|
+
export default {
|
|
3
|
+
"class": "form~",
|
|
4
|
+
"version": "0.7.5",
|
|
5
|
+
"author": "bonuoq",
|
|
6
|
+
"license": "MPL-2.0",
|
|
7
|
+
"doc": "How alike the moments of what arrives are, over the last seconds (20): a self-similarity matrix, drawn, of one frame every step seconds (0.25) — the mean of the lists that arrived in it (a chroma~, a features~ mfcc), or, for a stream of numbers (onset~'s third outlet, an rms), the shape of its values. On the left the novelty of each frame (0…1: how much the music changes there, by a checkerboard of kernel seconds (2) on each side, so it comes kernel seconds late); a bang on the second outlet where a segment begins (a novelty peak high for the window, two kernels after the last at least), and, on the fourth, [:segment {:t :at :length :period}] for the segment it ends; on the third, the period that repeats most in the window, in seconds (the bar, the phrase), when it changes. [:seconds x], [:step x] and [:kernel x] change them, :clear forgets. Into onset~ (a [:segment …] makes it forget the usual change of the section before: adaptive onsets), a message box with [:threshold x], or a print. Self-similarity after Foote (1999, 2000, 2001); the same functions for a whole take or a tree are api/ssm, api/novelty, api/boundaries and api/note-frames.",
|
|
8
|
+
"source": ";; form~ · how alike the moments are: a self-similarity matrix over the last seconds, its changes, its periods\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…): the matrix, novelty and beat spectrum\n;; are those of the core (kairotic.structure, issue #120), through api/ssm, api/novelty-at, api/beat-spectrum.\n;; What arrives (a chroma, descriptors, an onset envelope) is gathered into one frame every step seconds; each\n;; new frame is compared with all those of the window. A segment begins where the frames before are alike\n;; among themselves and unlike the ones after (Foote 2000), and the lag at which frames are most alike is the\n;; period that repeats (Foote & Uchihashi 2001).\n(let [G 20 ; the matrix drawn: G × G cells at most (the patch draws at most 600 shapes)\n bins 8 ; a stream of numbers: each frame is the shape of its step, in 8 slices\n settings (fn [args state]\n (let [o (:set state)\n seconds (api/clamp (or (:seconds o) (:seconds args) 20) 2 120)\n step (api/clamp (or (:step o) (:step args) 0.25) 0.02 4)\n kernel (api/clamp (or (:kernel o) (:kernel args) 2) (* 2 step) (/ seconds 3))]\n {:seconds seconds :step step :kernel kernel\n :L (max 1 (js/Math.round (/ kernel step)))\n :N (max 4 (js/Math.round (/ seconds step)))}))\n frame-of (fn [xs]\n ;; vectors: their mean; numbers: the values of the step, resampled into bins slices\n (if (sequential? (first xs))\n (let [d (reduce max 0 (map count xs))]\n (mapv (fn [j] (/ (reduce + (map (fn [v] (or (get v j) 0)) xs)) (count xs))) (range d)))\n (let [v (vec xs) n (count v)]\n (mapv (fn [j] (let [a (js/Math.floor (/ (* j n) bins)) b (max (inc a) (js/Math.floor (/ (* (inc j) n) bins)))\n s (subvec v (min a (dec n)) (min b n))]\n (/ (reduce + s) (max 1 (count s)))))\n (range bins)))))\n usable (fn [v] (cond (number? v) v\n (and (sequential? v) (seq v) (every? number? v)) (vec v)\n :else nil))\n ;; the matrix drawn: the mean of each block of frames\n cells (fn [S]\n (let [n (:n S) g (min G n) blk (fn [k] [(js/Math.floor (/ (* k n) g)) (js/Math.floor (/ (* (inc k) n) g))])]\n (vec (for [r (range g)]\n (let [[a b] (blk r)]\n (vec (for [c (range g)]\n (let [[x y] (blk c) cnt (* (- b a) (- y x))]\n (/ (reduce + (for [i (range a b) j (range x y)] (api/ssm-at S i j))) (max 1 cnt))))))))))\n mean-sd (fn [xs] (let [n (max 1 (count xs)) m (/ (reduce + xs) n)]\n [m (js/Math.sqrt (/ (reduce + (map (fn [x] (* (- x m) (- x m))) xs)) n))]))\n r3 (fn [x] (/ (js/Math.round (* 1000 x)) 1000))\n ;; a frame is complete: into the window, and what it tells\n close (fn [state {:keys [N L step kernel]}]\n (let [f (frame-of (:acc state))\n t (+ (:t0 state) (/ step 2))\n frames (vec (take-last N (conj (or (:frames state) []) [t f])))\n S (api/ssm (mapv second frames))\n n (:n S)\n i (- n 1 L)\n nov (when (>= i L) (api/novelty-at S L i))\n nt (when nov (first (frames i)))\n hist (if nov (vec (take-last N (conj (or (:nov state) []) [nt (max 0 nov)]))) (or (:nov state) []))\n ;; a boundary: the novelty of the frame before is a peak, high for this window, and the last\n ;; boundary is at least two kernels away\n vs (mapv second hist) k (count vs)\n [m sd] (mean-sd vs)\n peak (when (>= k 3)\n (let [[tp p] (hist (- k 2))]\n (when (and (> p (vs (- k 3))) (>= p (vs (dec k))) (> p (max 0.05 (+ m sd)))\n (or (nil? (:last state)) (>= (- tp (:last state)) (* 2 kernel))))\n tp)))\n bs (when (>= n 8) (api/beat-spectrum S (quot n 2)))\n ;; a period is a lag more alike than the lags before it (a real dip between): music that does\n ;; not change is alike at every lag, and has none\n lag (some (fn [[l x]] (when (and (>= l 2) (> (- x (reduce min (subvec bs 1 l))) 0.1)) l))\n (when bs (api/periods bs)))\n period (when lag (r3 (* lag step)))\n s (cond-> (assoc state :frames frames :nov hist\n :cells (cells S) :period (or period (:period state)))\n peak (assoc :last peak :marks (vec (take-last 8 (conj (or (:marks state) []) peak)))))]\n {:state s\n :out (cond-> []\n nov (conj [0 (r3 (max 0 nov))])\n (and period (not= period (:period state))) (conj [2 period])\n peak (conj [3 [:segment {:t (or (:last state) (first (first frames))) :at peak\n :length (r3 (- peak (or (:last state) (first (first frames)))))\n :period (or period (:period state))}]]\n [1 :bang]))}))]\n {;; ── about ──\n :class \"form~\"\n :license \"MPL-2.0\"\n :doc \"How alike the moments of what arrives are, over the last seconds (20): a self-similarity matrix, drawn, of one frame every step seconds (0.25) — the mean of the lists that arrived in it (a chroma~, a features~ mfcc), or, for a stream of numbers (onset~'s third outlet, an rms), the shape of its values. On the left the novelty of each frame (0…1: how much the music changes there, by a checkerboard of kernel seconds (2) on each side, so it comes kernel seconds late); a bang on the second outlet where a segment begins (a novelty peak high for the window, two kernels after the last at least), and, on the fourth, [:segment {:t :at :length :period}] for the segment it ends; on the third, the period that repeats most in the window, in seconds (the bar, the phrase), when it changes. [:seconds x], [:step x] and [:kernel x] change them, :clear forgets. Into onset~ (a [:segment …] makes it forget the usual change of the section before: adaptive onsets), a message box with [:threshold x], or a print. Self-similarity after Foote (1999, 2000, 2001); the same functions for a whole take or a tree are api/ssm, api/novelty, api/boundaries and api/note-frames.\"\n :usage \"form~ [seconds] [step] [kernel]\"\n :io [1 4]\n :size [220 128]\n :args [[:seconds 20] [:step 0.25] [:kernel 2]]\n\n ;; ── logic ──\n ;; the state: {:frames [[t vector] …] :acc [values of the frame under way] :t0 its start :nov [[t value] …]\n ;; :last the last boundary :marks boundaries :cells the matrix drawn :period :set what messages changed}\n :receive (fn [{:keys [value state args at]}]\n (cond\n (= value :clear) {:state (select-keys state [:set]) :show nil}\n (and (vector? value) (= 2 (count value)) (#{:seconds :step :kernel} (first value)) (number? (second value)))\n (let [s (-> (select-keys state [:set]) (assoc-in [:set (first value)] (second value)))] {:state s :show nil})\n (and (usable value) (number? at))\n (let [o (settings args state)\n ;; the engine started again (its clock went back), or the kind of what arrives changed: from nothing\n state (if (or (and (:t0 state) (< at (- (:t0 state) 1)))\n (and (seq (:acc state)) (not= (number? (first (:acc state))) (number? (usable value)))))\n (select-keys state [:set]) state)\n v (usable value)]\n ;; a value a step after the frame under way began closes it, and begins the next\n (if (and (:t0 state) (seq (:acc state)) (>= (- at (:t0 state)) (:step o)))\n (let [{s :state out :out} (close state o)\n s (assoc s :acc [v] :t0 at)]\n (cond-> {:state s :show (select-keys s [:cells :nov :marks :period :frames])}\n (seq out) (assoc :out out)))\n {:state (cond-> (update state :acc (fnil conj []) v) (nil? (:t0 state)) (assoc :t0 at))}))\n :else {}))\n\n ;; ── view: the matrix (time runs down and to the right), the novelty under it, the period ──\n :draw (fn [{:keys [shown args state w h]}]\n (let [{:keys [step kernel seconds]} (settings args state)\n cs (:cells shown) g (count cs)\n side (- h 12) x0 6 y0 6 cw (if (pos? g) (/ side g) side)\n frames (:frames shown)\n t0 (first (first frames)) t1 (first (peek frames))\n px0 (+ x0 side 10) px1 (- w 6) pw (- px1 px0)\n tx (fn [t] (if (and t0 t1 (> t1 t0)) (+ px0 (* pw (/ (- t t0) (- t1 t0)))) px0))\n ny0 (+ y0 34) ny1 (- h 8)\n nov (filter (fn [[t]] (and t0 (>= t t0))) (:nov shown))\n ty (fn [v] (- ny1 (* (- ny1 ny0) (api/clamp v 0 1))))]\n (into\n [[:rect {:x 2 :y 2 :w (- w 4) :h (- h 4) :r 4 :stroke :grid :fill :bg}]\n [:rect {:x x0 :y y0 :w side :h side :stroke :grid :fill :bg}]\n [:text {:x px0 :y (+ y0 10) :size 10 :fill :ink} \"form\"]\n [:text {:x px0 :y (+ y0 23) :size 8 :fill :soft}\n (if-let [p (:period shown)] (str \"period \" (api/nice p) \" s\") (str (api/nice seconds) \" s · \" (api/nice step) \" · \" (api/nice kernel)))]\n [:line {:x1 px0 :y1 ny1 :x2 px1 :y2 ny1 :stroke :grid :weight 1}]]\n (concat\n (for [r (range g) c (range g) :let [v ((cs r) c)] :when (> v 0.1)]\n [:rect {:x (+ x0 (* c cw)) :y (+ y0 (* r cw)) :w (+ cw 0.3) :h (+ cw 0.3)\n :fill :accent :stroke nil :opacity (api/clamp (* v v) 0 1)}])\n (let [xys (mapv (fn [[t v]] [(api/clamp (tx t) px0 px1) (ty v)]) nov)]\n (when (> (count xys) 1) [[:path {:points xys :stroke :accent :weight 1.2}]]))\n (for [t (:marks shown) :when (and t0 (>= t t0))]\n [:line {:x1 (tx t) :y1 ny0 :x2 (tx t) :y2 ny1 :stroke :kairos :weight 1.2}])))))})\n"
|
|
9
|
+
};
|
package/dist/index.mjs
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Generated by packages/build.mjs
|
|
2
2
|
export default {
|
|
3
3
|
"package": "@kairotic/analysis",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.7.5",
|
|
5
5
|
"api": 1,
|
|
6
6
|
"author": "bonuoq",
|
|
7
7
|
"license": "MPL-2.0",
|
|
@@ -10,13 +10,19 @@ export default {
|
|
|
10
10
|
"class": "chroma~",
|
|
11
11
|
"license": "MPL-2.0",
|
|
12
12
|
"doc": "How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.",
|
|
13
|
-
"source": ";; chroma~ · how much of each pitch class a sound has\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The chroma is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n gate 0.005 ; rms under it is silence: no strongest class\n classes [:c :c# :d :d# :e :f :f# :g :g# :a :a# :b]\n names [\"C\" \"C#\" \"D\" \"D#\" \"E\" \"F\" \"F#\" \"G\" \"G#\" \"A\" \"A#\" \"B\"]\n strongest (fn [xs] (when (seq xs) (nth classes (first (apply max-key (fn [[_ v]] v) (map-indexed vector xs))))))\n ;; a picture of 12 bars in 168 × 64\n bar-w 13 x0 6 base 46 tall 38]\n {;; ── about ──\n :class \"chroma~\"\n :license \"MPL-2.0\"\n :doc \"How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.\"\n :usage \"chroma~ [ms]\"\n :io [1 2]\n :size [168 64]\n :args [[:ms 40]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value) (> (or (:rms value) 0) gate) (= 12 (count (:chroma value))))\n (let [xs (:chroma value) top (strongest xs)\n outs (cond-> []\n (not= top (:top state)) (conj [1 top])\n (not= xs (:xs state)) (conj [0 xs]))]\n (if (seq outs) {:state {:xs xs :top top} :out outs :show xs} {}))\n :else {}))\n\n ;; ── view ──\n :draw (fn [{:keys [shown]}]\n (let [xs (if (and (sequential? shown) (= 12 (count shown))) shown (repeat 12 0))\n top (when (some pos? xs) (apply max xs))]\n (into [[:rect {:x 2 :y 2 :w 164 :h 60 :r 4 :stroke :grid :fill :bg}]]\n (mapcat (fn [i v n]\n (let [h (* tall (api/clamp v 0 1)) x (+ x0 (* i bar-w))]\n [[:rect {:x x :y (- base h) :w (- bar-w 3) :h h :r 1 :stroke nil\n :fill (if (and top (= v top)) :kairos :accent)}]\n [:text {:x (+ x (/ (- bar-w 3) 2)) :y 58 :anchor :middle :size 7 :fill :soft} n]]))\n (range 12) xs names))))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n ms (api/clamp (or (first (filter number? (:args b))) 40) 10 2000)\n node {:in in :M (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n (when-let [M @(:M node)]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"chroma\" \"rms\"] (first data))\n xs (mapv (fn [x] (/ (js/Math.round (* 100 (if (js/isFinite x) x 0))) 100))\n (array-seq (.-chroma r)))]\n (api/report! (:id b) {:chroma xs :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"chroma~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ ms 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (.connect in w) (
|
|
13
|
+
"source": ";; chroma~ · how much of each pitch class a sound has\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The chroma is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n gate 0.005 ; rms under it is silence: no strongest class\n classes [:c :c# :d :d# :e :f :f# :g :g# :a :a# :b]\n names [\"C\" \"C#\" \"D\" \"D#\" \"E\" \"F\" \"F#\" \"G\" \"G#\" \"A\" \"A#\" \"B\"]\n strongest (fn [xs] (when (seq xs) (nth classes (first (apply max-key (fn [[_ v]] v) (map-indexed vector xs))))))\n ;; a picture of 12 bars in 168 × 64\n bar-w 13 x0 6 base 46 tall 38]\n {;; ── about ──\n :class \"chroma~\"\n :license \"MPL-2.0\"\n :doc \"How much of each pitch class the signal on its left has, every ms (40 by default): a list of 12 values from C to B (0…1, the strongest 1) on the left, and the strongest class as a word (:c, :c#, … :b) on the right when it changes. Silence sends nothing. Chords, keys and drones become data for mt and the patch. Computed with Meyda.\"\n :usage \"chroma~ [ms]\"\n :io [1 2]\n :size [168 64]\n :args [[:ms 40]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value) (> (or (:rms value) 0) gate) (= 12 (count (:chroma value))))\n (let [xs (:chroma value) top (strongest xs)\n outs (cond-> []\n (not= top (:top state)) (conj [1 top])\n (not= xs (:xs state)) (conj [0 xs]))]\n (if (seq outs) {:state {:xs xs :top top} :out outs :show xs} {}))\n :else {}))\n\n ;; ── view ──\n :draw (fn [{:keys [shown]}]\n (let [xs (if (and (sequential? shown) (= 12 (count shown))) shown (repeat 12 0))\n top (when (some pos? xs) (apply max xs))]\n (into [[:rect {:x 2 :y 2 :w 164 :h 60 :r 4 :stroke :grid :fill :bg}]]\n (mapcat (fn [i v n]\n (let [h (* tall (api/clamp v 0 1)) x (+ x0 (* i bar-w))]\n [[:rect {:x x :y (- base h) :w (- bar-w 3) :h h :r 1 :stroke nil\n :fill (if (and top (= v top)) :kairos :accent)}]\n [:text {:x (+ x (/ (- bar-w 3) 2)) :y 58 :anchor :middle :size 7 :fill :soft} n]]))\n (range 12) xs names))))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n ms (api/clamp (or (first (filter number? (:args b))) 40) 10 2000)\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"chroma\" \"rms\"] (first data))\n xs (mapv (fn [x] (/ (js/Math.round (* 100 (if (js/isFinite x) x 0))) 100))\n (array-seq (.-chroma r)))]\n (api/report! (:id b) {:chroma xs :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"chroma~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ ms 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w)) (api/untap w))))\n (.catch (fn [e] (js/console.error \"chroma~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
14
14
|
},
|
|
15
15
|
{
|
|
16
16
|
"class": "features~",
|
|
17
17
|
"license": "MPL-2.0",
|
|
18
18
|
"doc": "Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.",
|
|
19
|
-
"source": ";; features~ · descriptors of a sound, every few milliseconds\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The descriptors are Meyda's\n;; (MIT, meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n ;; every analysis box of the package uses frames of this size: Meyda is one object for the whole\n ;; page and keeps some filter banks for the size it saw first\n size 2048\n aliases {:centroid :spectralCentroid :flatness :spectralFlatness :rolloff :spectralRolloff\n :spread :spectralSpread :kurtosis :spectralKurtosis :skewness :spectralSkewness\n :slope :spectralSlope :crest :spectralCrest :sharpness :perceptualSharpness}\n known #{:rms :energy :zcr :loudness :mfcc :chroma :spectralCentroid :spectralFlatness :spectralRolloff\n :spectralSpread :spectralKurtosis :spectralSkewness :spectralSlope :spectralCrest\n :perceptualSharpness :perceptualSpread}\n feats (fn [args]\n (let [ws (->> args (filter keyword?) (map (fn [k] (get aliases k k))) (filter known) distinct vec)]\n (if (seq ws) ws [:rms :spectralCentroid :spectralFlatness])))\n hop-ms (fn [args] (api/clamp (or (first (filter number? args)) 20) 5 1000))\n ok? (fn [x] (and (number? x) (js/isFinite x)))\n ;; a value as it leaves the box: Hz where Meyda gives bins, the total of loudness, lists rounded\n value (fn [k v sr]\n (cond\n (#{:spectralCentroid :spectralSpread} k) (when (ok? v) (api/nice (* v (/ sr size))))\n (= k :loudness) (when v (let [t (.-total v)] (when (ok? t) (api/nice t))))\n (#{:mfcc :chroma} k) (when v (let [xs (vec (array-seq v))] (when (every? ok? xs) (mapv api/nice xs))))\n :else (when (ok? v) (api/nice v))))]\n {;; ── about ──\n :class \"features~\"\n :license \"MPL-2.0\"\n :doc \"Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.\"\n :usage \"features~ [ms] [descriptor …]\"\n :io (fn [b] [1 (count (feats (:args b)))])\n :dynamic-io true\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value))\n (let [fs (feats args)\n outs (vec (keep-indexed (fn [i k] (let [v (get value k)]\n (when (and (some? v) (not= v (get state k))) [i v])))\n fs))]\n (if (seq outs) {:state (merge state (select-keys value fs)) :out outs} {}))\n :else {}))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n fs (feats (:args b))\n names (clj->js (mapv name fs))\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n (when-let [M @(:M node)]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M names (first data))]\n (api/report! (:id b) (into {} (for [k fs] [k (value k (aget r (name k)) sr)])) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"features~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ (hop-ms (:args b)) 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"features~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
19
|
+
"source": ";; features~ · descriptors of a sound, every few milliseconds\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The descriptors are Meyda's\n;; (MIT, meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n ;; every analysis box of the package uses frames of this size: Meyda is one object for the whole\n ;; page and keeps some filter banks for the size it saw first\n size 2048\n aliases {:centroid :spectralCentroid :flatness :spectralFlatness :rolloff :spectralRolloff\n :spread :spectralSpread :kurtosis :spectralKurtosis :skewness :spectralSkewness\n :slope :spectralSlope :crest :spectralCrest :sharpness :perceptualSharpness}\n known #{:rms :energy :zcr :loudness :mfcc :chroma :spectralCentroid :spectralFlatness :spectralRolloff\n :spectralSpread :spectralKurtosis :spectralSkewness :spectralSlope :spectralCrest\n :perceptualSharpness :perceptualSpread}\n feats (fn [args]\n (let [ws (->> args (filter keyword?) (map (fn [k] (get aliases k k))) (filter known) distinct vec)]\n (if (seq ws) ws [:rms :spectralCentroid :spectralFlatness])))\n hop-ms (fn [args] (api/clamp (or (first (filter number? args)) 20) 5 1000))\n ok? (fn [x] (and (number? x) (js/isFinite x)))\n ;; a value as it leaves the box: Hz where Meyda gives bins, the total of loudness, lists rounded\n value (fn [k v sr]\n (cond\n (#{:spectralCentroid :spectralSpread} k) (when (ok? v) (api/nice (* v (/ sr size))))\n (= k :loudness) (when v (let [t (.-total v)] (when (ok? t) (api/nice t))))\n (#{:mfcc :chroma} k) (when v (let [xs (vec (array-seq v))] (when (every? ok? xs) (mapv api/nice xs))))\n :else (when (ok? v) (api/nice v))))]\n {;; ── about ──\n :class \"features~\"\n :license \"MPL-2.0\"\n :doc \"Descriptors of the signal on its left, every ms (20 by default), one outlet for each word written after it, in order: rms, energy, zcr, loudness, centroid and spread (in Hz), flatness, rolloff (Hz), kurtosis, skewness, slope, crest, sharpness, perceptualSpread, mfcc (13 values) and chroma (12). Without words: rms centroid flatness. Each outlet sends only what changed. Computed with Meyda.\"\n :usage \"features~ [ms] [descriptor …]\"\n :io (fn [b] [1 (count (feats (:args b)))])\n :dynamic-io true\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (map? value))\n (let [fs (feats args)\n outs (vec (keep-indexed (fn [i k] (let [v (get value k)]\n (when (and (some? v) (not= v (get state k))) [i v])))\n fs))]\n (if (seq outs) {:state (merge state (select-keys value fs)) :out outs} {}))\n :else {}))\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n fs (feats (:args b))\n names (clj->js (mapv name fs))\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M names (first data))]\n (api/report! (:id b) (into {} (for [k fs] [k (value k (aget r (name k)) sr)])) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"features~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (* sr (/ (hop-ms (:args b)) 1000)))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"features~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
20
|
+
},
|
|
21
|
+
{
|
|
22
|
+
"class": "form~",
|
|
23
|
+
"license": "MPL-2.0",
|
|
24
|
+
"doc": "How alike the moments of what arrives are, over the last seconds (20): a self-similarity matrix, drawn, of one frame every step seconds (0.25) — the mean of the lists that arrived in it (a chroma~, a features~ mfcc), or, for a stream of numbers (onset~'s third outlet, an rms), the shape of its values. On the left the novelty of each frame (0…1: how much the music changes there, by a checkerboard of kernel seconds (2) on each side, so it comes kernel seconds late); a bang on the second outlet where a segment begins (a novelty peak high for the window, two kernels after the last at least), and, on the fourth, [:segment {:t :at :length :period}] for the segment it ends; on the third, the period that repeats most in the window, in seconds (the bar, the phrase), when it changes. [:seconds x], [:step x] and [:kernel x] change them, :clear forgets. Into onset~ (a [:segment …] makes it forget the usual change of the section before: adaptive onsets), a message box with [:threshold x], or a print. Self-similarity after Foote (1999, 2000, 2001); the same functions for a whole take or a tree are api/ssm, api/novelty, api/boundaries and api/note-frames.",
|
|
25
|
+
"source": ";; form~ · how alike the moments are: a self-similarity matrix over the last seconds, its changes, its periods\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…): the matrix, novelty and beat spectrum\n;; are those of the core (kairotic.structure, issue #120), through api/ssm, api/novelty-at, api/beat-spectrum.\n;; What arrives (a chroma, descriptors, an onset envelope) is gathered into one frame every step seconds; each\n;; new frame is compared with all those of the window. A segment begins where the frames before are alike\n;; among themselves and unlike the ones after (Foote 2000), and the lag at which frames are most alike is the\n;; period that repeats (Foote & Uchihashi 2001).\n(let [G 20 ; the matrix drawn: G × G cells at most (the patch draws at most 600 shapes)\n bins 8 ; a stream of numbers: each frame is the shape of its step, in 8 slices\n settings (fn [args state]\n (let [o (:set state)\n seconds (api/clamp (or (:seconds o) (:seconds args) 20) 2 120)\n step (api/clamp (or (:step o) (:step args) 0.25) 0.02 4)\n kernel (api/clamp (or (:kernel o) (:kernel args) 2) (* 2 step) (/ seconds 3))]\n {:seconds seconds :step step :kernel kernel\n :L (max 1 (js/Math.round (/ kernel step)))\n :N (max 4 (js/Math.round (/ seconds step)))}))\n frame-of (fn [xs]\n ;; vectors: their mean; numbers: the values of the step, resampled into bins slices\n (if (sequential? (first xs))\n (let [d (reduce max 0 (map count xs))]\n (mapv (fn [j] (/ (reduce + (map (fn [v] (or (get v j) 0)) xs)) (count xs))) (range d)))\n (let [v (vec xs) n (count v)]\n (mapv (fn [j] (let [a (js/Math.floor (/ (* j n) bins)) b (max (inc a) (js/Math.floor (/ (* (inc j) n) bins)))\n s (subvec v (min a (dec n)) (min b n))]\n (/ (reduce + s) (max 1 (count s)))))\n (range bins)))))\n usable (fn [v] (cond (number? v) v\n (and (sequential? v) (seq v) (every? number? v)) (vec v)\n :else nil))\n ;; the matrix drawn: the mean of each block of frames\n cells (fn [S]\n (let [n (:n S) g (min G n) blk (fn [k] [(js/Math.floor (/ (* k n) g)) (js/Math.floor (/ (* (inc k) n) g))])]\n (vec (for [r (range g)]\n (let [[a b] (blk r)]\n (vec (for [c (range g)]\n (let [[x y] (blk c) cnt (* (- b a) (- y x))]\n (/ (reduce + (for [i (range a b) j (range x y)] (api/ssm-at S i j))) (max 1 cnt))))))))))\n mean-sd (fn [xs] (let [n (max 1 (count xs)) m (/ (reduce + xs) n)]\n [m (js/Math.sqrt (/ (reduce + (map (fn [x] (* (- x m) (- x m))) xs)) n))]))\n r3 (fn [x] (/ (js/Math.round (* 1000 x)) 1000))\n ;; a frame is complete: into the window, and what it tells\n close (fn [state {:keys [N L step kernel]}]\n (let [f (frame-of (:acc state))\n t (+ (:t0 state) (/ step 2))\n frames (vec (take-last N (conj (or (:frames state) []) [t f])))\n S (api/ssm (mapv second frames))\n n (:n S)\n i (- n 1 L)\n nov (when (>= i L) (api/novelty-at S L i))\n nt (when nov (first (frames i)))\n hist (if nov (vec (take-last N (conj (or (:nov state) []) [nt (max 0 nov)]))) (or (:nov state) []))\n ;; a boundary: the novelty of the frame before is a peak, high for this window, and the last\n ;; boundary is at least two kernels away\n vs (mapv second hist) k (count vs)\n [m sd] (mean-sd vs)\n peak (when (>= k 3)\n (let [[tp p] (hist (- k 2))]\n (when (and (> p (vs (- k 3))) (>= p (vs (dec k))) (> p (max 0.05 (+ m sd)))\n (or (nil? (:last state)) (>= (- tp (:last state)) (* 2 kernel))))\n tp)))\n bs (when (>= n 8) (api/beat-spectrum S (quot n 2)))\n ;; a period is a lag more alike than the lags before it (a real dip between): music that does\n ;; not change is alike at every lag, and has none\n lag (some (fn [[l x]] (when (and (>= l 2) (> (- x (reduce min (subvec bs 1 l))) 0.1)) l))\n (when bs (api/periods bs)))\n period (when lag (r3 (* lag step)))\n s (cond-> (assoc state :frames frames :nov hist\n :cells (cells S) :period (or period (:period state)))\n peak (assoc :last peak :marks (vec (take-last 8 (conj (or (:marks state) []) peak)))))]\n {:state s\n :out (cond-> []\n nov (conj [0 (r3 (max 0 nov))])\n (and period (not= period (:period state))) (conj [2 period])\n peak (conj [3 [:segment {:t (or (:last state) (first (first frames))) :at peak\n :length (r3 (- peak (or (:last state) (first (first frames)))))\n :period (or period (:period state))}]]\n [1 :bang]))}))]\n {;; ── about ──\n :class \"form~\"\n :license \"MPL-2.0\"\n :doc \"How alike the moments of what arrives are, over the last seconds (20): a self-similarity matrix, drawn, of one frame every step seconds (0.25) — the mean of the lists that arrived in it (a chroma~, a features~ mfcc), or, for a stream of numbers (onset~'s third outlet, an rms), the shape of its values. On the left the novelty of each frame (0…1: how much the music changes there, by a checkerboard of kernel seconds (2) on each side, so it comes kernel seconds late); a bang on the second outlet where a segment begins (a novelty peak high for the window, two kernels after the last at least), and, on the fourth, [:segment {:t :at :length :period}] for the segment it ends; on the third, the period that repeats most in the window, in seconds (the bar, the phrase), when it changes. [:seconds x], [:step x] and [:kernel x] change them, :clear forgets. Into onset~ (a [:segment …] makes it forget the usual change of the section before: adaptive onsets), a message box with [:threshold x], or a print. Self-similarity after Foote (1999, 2000, 2001); the same functions for a whole take or a tree are api/ssm, api/novelty, api/boundaries and api/note-frames.\"\n :usage \"form~ [seconds] [step] [kernel]\"\n :io [1 4]\n :size [220 128]\n :args [[:seconds 20] [:step 0.25] [:kernel 2]]\n\n ;; ── logic ──\n ;; the state: {:frames [[t vector] …] :acc [values of the frame under way] :t0 its start :nov [[t value] …]\n ;; :last the last boundary :marks boundaries :cells the matrix drawn :period :set what messages changed}\n :receive (fn [{:keys [value state args at]}]\n (cond\n (= value :clear) {:state (select-keys state [:set]) :show nil}\n (and (vector? value) (= 2 (count value)) (#{:seconds :step :kernel} (first value)) (number? (second value)))\n (let [s (-> (select-keys state [:set]) (assoc-in [:set (first value)] (second value)))] {:state s :show nil})\n (and (usable value) (number? at))\n (let [o (settings args state)\n ;; the engine started again (its clock went back), or the kind of what arrives changed: from nothing\n state (if (or (and (:t0 state) (< at (- (:t0 state) 1)))\n (and (seq (:acc state)) (not= (number? (first (:acc state))) (number? (usable value)))))\n (select-keys state [:set]) state)\n v (usable value)]\n ;; a value a step after the frame under way began closes it, and begins the next\n (if (and (:t0 state) (seq (:acc state)) (>= (- at (:t0 state)) (:step o)))\n (let [{s :state out :out} (close state o)\n s (assoc s :acc [v] :t0 at)]\n (cond-> {:state s :show (select-keys s [:cells :nov :marks :period :frames])}\n (seq out) (assoc :out out)))\n {:state (cond-> (update state :acc (fnil conj []) v) (nil? (:t0 state)) (assoc :t0 at))}))\n :else {}))\n\n ;; ── view: the matrix (time runs down and to the right), the novelty under it, the period ──\n :draw (fn [{:keys [shown args state w h]}]\n (let [{:keys [step kernel seconds]} (settings args state)\n cs (:cells shown) g (count cs)\n side (- h 12) x0 6 y0 6 cw (if (pos? g) (/ side g) side)\n frames (:frames shown)\n t0 (first (first frames)) t1 (first (peek frames))\n px0 (+ x0 side 10) px1 (- w 6) pw (- px1 px0)\n tx (fn [t] (if (and t0 t1 (> t1 t0)) (+ px0 (* pw (/ (- t t0) (- t1 t0)))) px0))\n ny0 (+ y0 34) ny1 (- h 8)\n nov (filter (fn [[t]] (and t0 (>= t t0))) (:nov shown))\n ty (fn [v] (- ny1 (* (- ny1 ny0) (api/clamp v 0 1))))]\n (into\n [[:rect {:x 2 :y 2 :w (- w 4) :h (- h 4) :r 4 :stroke :grid :fill :bg}]\n [:rect {:x x0 :y y0 :w side :h side :stroke :grid :fill :bg}]\n [:text {:x px0 :y (+ y0 10) :size 10 :fill :ink} \"form\"]\n [:text {:x px0 :y (+ y0 23) :size 8 :fill :soft}\n (if-let [p (:period shown)] (str \"period \" (api/nice p) \" s\") (str (api/nice seconds) \" s · \" (api/nice step) \" · \" (api/nice kernel)))]\n [:line {:x1 px0 :y1 ny1 :x2 px1 :y2 ny1 :stroke :grid :weight 1}]]\n (concat\n (for [r (range g) c (range g) :let [v ((cs r) c)] :when (> v 0.1)]\n [:rect {:x (+ x0 (* c cw)) :y (+ y0 (* r cw)) :w (+ cw 0.3) :h (+ cw 0.3)\n :fill :accent :stroke nil :opacity (api/clamp (* v v) 0 1)}])\n (let [xys (mapv (fn [[t v]] [(api/clamp (tx t) px0 px1) (ty v)]) nov)]\n (when (> (count xys) 1) [[:path {:points xys :stroke :accent :weight 1.2}]]))\n (for [t (:marks shown) :when (and t0 (>= t t0))]\n [:line {:x1 (tx t) :y1 ny0 :x2 (tx t) :y2 ny1 :stroke :kairos :weight 1.2}])))))})\n"
|
|
20
26
|
},
|
|
21
27
|
{
|
|
22
28
|
"class": "monitor",
|
|
@@ -27,26 +33,32 @@ export default {
|
|
|
27
33
|
{
|
|
28
34
|
"class": "onset~",
|
|
29
35
|
"license": "MPL-2.0",
|
|
30
|
-
"doc": "The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the
|
|
31
|
-
"source": ";; onset~ · the beginnings of sounds: a note, a stroke, a syllable\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n bins 256 ; the spectrum up to about 6 kHz at 48 kHz: where onsets are heard\n ;; how much the spectrum grew since the frame before, relative to what sounds now (0 steady … 1\n ;; from silence): what grows counts, what fades does not\n flux (fn [cur prev]\n (if-not prev\n 0\n (loop [i 1 up 0 all 0]\n (if (< i bins)\n (let [a (aget cur i) d (- a (aget prev i))]\n (recur (inc i) (if (pos? d) (+ up d) up) (+ all a)))\n (if (> all 1e-6) (/ up all) 0)))))\n median (fn [xs] (if (empty? xs) 0 (nth (sort xs) (quot (count xs) 2))))\n gate 0.005 ; rms under it (about -46 dB) is silence\n ;; the decision, pure: an onset where the flux rises above threshold × the median of the last\n ;; frames (and above 0.1), not sooner than ms after the last one\n detect (fn [state {:keys [flux rms]} at threshold ms]\n (let [hist (vec (take-last 16 (:hist state)))\n m (median hist)\n over (and (> rms gate) (> flux 0.1) (> flux (* threshold m)))\n fire (and over (not (:over state)) (or (nil? (:last state)) (>= (- at (:last state)) (/ ms 1000))))]\n [(cond-> (assoc state :hist (conj hist flux) :over over) fire (assoc :last at))\n (when fire (/ (js/Math.round (* 100 flux)) 100))]))]\n {;; ── about ──\n :class \"onset~\"\n :license \"MPL-2.0\"\n :doc \"The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the
|
|
36
|
+
"doc": "The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the second outlet; on the third, that growth for every frame (the onset strength envelope, for beat tracking: issue #106). threshold (3) is how far above the usual change of the last frames an onset must rise, while the energy does not fall (a sound cut at once is not an onset); ms (80) the least time between two. Every 512 samples (about 11 ms), on frames of 2048. An onset is reported with the frame it is first seen in: measured on sines, plucked tones and noise bursts that start at a known sample, 6 ms after it as a median (0.7 to 10.7 ms: one hop), and 11 ms for a new note over a held one. [:threshold x] and [:ms y] change them while it listens, and a [:segment …] from form~ makes it forget the usual change of the section before (so a quiet section after a loud one is not deaf). Into a counter, a metro that follows, a sample's rec, or the right inlet of transcribe (its strength, as the weight of the note).",
|
|
37
|
+
"source": ";; onset~ · the beginnings of sounds: a note, a stroke, a syllable\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n bins 256 ; the spectrum up to about 6 kHz at 48 kHz: where onsets are heard\n ;; how much the spectrum grew since the frame before, relative to what sounds now (0 steady … 1\n ;; from silence): what grows counts, what fades does not\n flux (fn [cur prev]\n (if-not prev\n 0\n (loop [i 1 up 0 all 0]\n (if (< i bins)\n (let [a (aget cur i) d (- a (aget prev i))]\n (recur (inc i) (if (pos? d) (+ up d) up) (+ all a)))\n (if (> all 1e-6) (/ up all) 0)))))\n median (fn [xs] (if (empty? xs) 0 (nth (sort xs) (quot (count xs) 2))))\n gate 0.005 ; rms under it (about -46 dB) is silence\n ;; the decision, pure: an onset where the flux rises above threshold × the median of the last\n ;; frames (and above 0.1) while the energy does not fall (a sound cut at once spreads its spectrum like an\n ;; attack, but its energy goes down: not an onset), not sooner than ms after the last one\n detect (fn [state {:keys [flux rms]} at threshold ms]\n (let [hist (vec (take-last 16 (:hist state)))\n m (median hist)\n over (and (> rms gate) (> flux 0.1) (> flux (* threshold m)) (>= rms (or (:rms state) 0)))\n fire (and over (not (:over state)) (or (nil? (:last state)) (>= (- at (:last state)) (/ ms 1000))))]\n [(cond-> (assoc state :hist (conj hist flux) :over over :rms rms) fire (assoc :last at))\n (when fire (/ (js/Math.round (* 100 flux)) 100))]))]\n {;; ── about ──\n :class \"onset~\"\n :license \"MPL-2.0\"\n :doc \"The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the second outlet; on the third, that growth for every frame (the onset strength envelope, for beat tracking: issue #106). threshold (3) is how far above the usual change of the last frames an onset must rise, while the energy does not fall (a sound cut at once is not an onset); ms (80) the least time between two. Every 512 samples (about 11 ms), on frames of 2048. An onset is reported with the frame it is first seen in: measured on sines, plucked tones and noise bursts that start at a known sample, 6 ms after it as a median (0.7 to 10.7 ms: one hop), and 11 ms for a new note over a held one. [:threshold x] and [:ms y] change them while it listens, and a [:segment …] from form~ makes it forget the usual change of the section before (so a quiet section after a loud one is not deaf). Into a counter, a metro that follows, a sample's rec, or the right inlet of transcribe (its strength, as the weight of the note).\"\n :usage \"onset~ [threshold] [ms]\"\n :io [1 3]\n :size [64 30]\n :args [[:threshold 3] [:ms 80]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args at]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n ;; adaptive: [:threshold x] and [:ms y] change them while it listens (from form~, a message box…);\n ;; a [:segment …] from form~ forgets the usual change of the section before\n (and (vector? value) (= 2 (count value)) (#{:threshold :ms} (first value)) (number? (second value)))\n {:state (assoc-in state [:set (first value)] (second value)) :show {:label (str (name (first value)) \" \" (api/nice (second value)))}}\n (and (vector? value) (= :segment (first value)))\n {:state (assoc state :hist [])}\n (and (= inlet :audio) (map? value))\n (let [[st strength] (detect state value at (api/clamp (or (get-in state [:set :threshold]) (:threshold args)) 1 50)\n (api/clamp (or (get-in state [:set :ms]) (:ms args)) 10 2000))]\n ;; the flux of every frame on the third outlet: the onset strength envelope (issue #106)\n (let [env [2 (/ (js/Math.round (* 1000 (:flux value))) 1000)]]\n (if strength\n {:state st :out [[1 strength] [0 :bang] env] :show {:at at :strength strength}}\n {:state st :out [env]})))\n :else {}))\n\n ;; ── view: a dot that lights on each onset ──\n :derive (fn [{:keys [shown]}]\n {:label (cond (:strength shown) (str (:strength shown)) (:label shown) (:label shown) :else \"onset\")})\n :view [[:rect {:x 2 :y 2 :w 60 :h 26 :r 3 :stroke :grid :fill :bg}]\n [:circle {:cx 15 :cy 15 :r 6 :stroke :accent :fill :accent}]\n [:text {:x 26 :y 19 :size 10 :fill :soft} :$label]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n node {:in in :M (atom nil) :prev (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"amplitudeSpectrum\" \"rms\"] (first data))\n cur (.-amplitudeSpectrum r)\n f (flux cur @(:prev node))]\n (reset! (:prev node) (.slice cur 0 bins))\n (api/report! (:id b) {:flux f :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"onset~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop 512} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w)) (api/untap w))))\n (.catch (fn [e] (js/console.error \"onset~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
32
38
|
},
|
|
33
39
|
{
|
|
34
40
|
"class": "pitch~",
|
|
35
41
|
"license": "MPL-2.0",
|
|
36
42
|
"doc": "The pitch of the signal on its left, by YIN, every 1024 samples (about 21 ms): its frequency in Hz on the left, when it finds one between lo and hi (60 and 1500 by default), and 0 once when it no longer does (silence, noise, a chord); how clearly it is periodic (0…1) on the right. It sends only what changed. threshold (0.15) is how strict it is: lower finds fewer pitches, surer ones. Into ftom (Hz → MIDI pitch), a synth~, or a faust~.",
|
|
37
|
-
"source": ";; pitch~ · the pitch of a sound, by YIN\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…), with its AudioWorklet inside.\n(let [worklet \"class KairoticYin extends AudioWorkletProcessor {\n // YIN (de Cheveigné & Kawahara, 2002) on the last W samples, every `hop` samples: the period whose\n // cumulative mean normalized difference first falls under the threshold (refined by a parabola),\n // between lo and hi Hz. Hands {hz, clarity, rms}; hz 0 where it finds no pitch or hears silence.\n constructor(o) {\n super();\n const p = (o && o.processorOptions) || {};\n this.W = 1024; this.hop = p.hop || 1024; this.count = 0;\n this.set(p.lo || 60, p.hi || 1500, p.threshold || 0.15);\n this.port.onmessage = (e) => { const d = e.data || {}; this.set(d.lo || this.lo, d.hi || this.hi, d.threshold || this.th); };\n }\n set(lo, hi, th) {\n this.lo = Math.max(20, Math.min(lo, hi)); this.hi = Math.max(this.lo + 1, hi); this.th = Math.max(0.01, Math.min(0.9, th));\n this.T = Math.min(4096, Math.ceil(sampleRate / this.lo) + 1);\n this.tmin = Math.max(2, Math.floor(sampleRate / this.hi));\n this.N = this.W + this.T + 1;\n this.ring = new Float32Array(this.N); this.buf = new Float32Array(this.N); this.d = new Float32Array(this.T + 2);\n this.w = 0;\n }\n analyse() {\n const N = this.N, b = this.buf, r = this.ring, W = this.W, T = this.T, d = this.d;\n for (let i = 0; i < N; i++) b[i] = r[(this.w + i) % N];\n let e = 0; for (let i = N - W; i < N; i++) e += b[i] * b[i];\n const rms = Math.sqrt(e / W);\n if (rms < 0.002) return { hz: 0, clarity: 0, rms };\n // the difference function, stopped as soon as the first dip under the threshold has its minimum\n d[0] = 1; let sum = 0, best = -1;\n for (let tau = 1; tau <= T; tau++) {\n let s = 0;\n for (let j = 0; j < W; j++) { const x = b[j] - b[j + tau]; s += x * x; }\n sum += s; d[tau] = sum > 0 ? s * tau / sum : 1;\n if (tau - 1 >= this.tmin && d[tau - 1] < this.th && d[tau] >= d[tau - 1]) { best = tau - 1; break; }\n }\n if (best < 1) return { hz: 0, clarity: 0, rms };\n let t = best;\n const a = d[best - 1], c = d[best], z = d[best + 1], den = a + z - 2 * c;\n if (den > 0) t = best + (a - z) / (2 * den);\n return { hz: sampleRate / t, clarity: Math.max(0, Math.min(1, 1 - c)), rms };\n }\n process(inputs) {\n const ch = inputs[0] && inputs[0][0];\n const n = ch ? ch.length : 128;\n for (let i = 0; i < n; i++) {\n this.ring[this.w] = ch ? ch[i] : 0;\n this.w = (this.w + 1) % this.N;\n if (++this.count >= this.hop) {\n this.count = 0;\n const r = this.analyse();\n this.port.postMessage({ hz: r.hz, clarity: r.clarity, rms: r.rms, at: currentTime + (i + 1) / sampleRate });\n }\n }\n return true;\n }\n}\nregisterProcessor('kairotic-yin', KairoticYin);\n\"\n ;; what it says between a pitch it heard and the last one it sent: a change of more than a tenth of a cent\n changed? (fn [a b] (or (nil? a) (nil? b) (zero? a) (> (js/Math.abs (- (api/hz->midi b) (api/hz->midi a))) 0.001)))\n label (fn [hz]\n (if (and (number? hz) (pos? hz))\n (let [m (api/hz->midi hz) r (js/Math.round m) c (js/Math.round (* 100 (- m r)))]\n (str (api/midi-name r) \" \" (if (neg? c) \"\" \"+\") c \"¢\"))\n \"–\"))]\n {;; ── about ──\n :class \"pitch~\"\n :license \"MPL-2.0\"\n :doc \"The pitch of the signal on its left, by YIN, every 1024 samples (about 21 ms): its frequency in Hz on the left, when it finds one between lo and hi (60 and 1500 by default), and 0 once when it no longer does (silence, noise, a chord); how clearly it is periodic (0…1) on the right. It sends only what changed. threshold (0.15) is how strict it is: lower finds fewer pitches, surer ones. Into ftom (Hz → MIDI pitch), a synth~, or a faust~.\"\n :usage \"pitch~ [lo hi] [threshold]\"\n :io [1 2]\n :size [124 34]\n :args [[:lo 60] [:hi 1500] [:threshold 0.15]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (if (and (= inlet :audio) (map? value))\n (let [{:keys [hz clarity]} value\n hz (if (and (number? hz) (pos? hz)) (/ (js/Math.round (* hz 100)) 100) 0)\n cl (/ (js/Math.round (* 100 (or clarity 0))) 100)\n last-hz (:hz state)\n send-hz (cond (pos? hz) (when (changed? last-hz hz) hz)\n (and last-hz (pos? last-hz)) 0)\n outs (cond-> []\n (some? send-hz) (conj [0 send-hz])\n (not= cl (:clarity state)) (conj [1 cl]))]\n (if (seq outs)\n {:state {:hz (if (some? send-hz) send-hz last-hz) :clarity cl} :out outs :show {:hz hz :clarity cl}}\n {}))\n {}))\n\n ;; ── view ──\n :derive (fn [{:keys [shown]}]\n (let [hz (:hz shown)]\n {:note (label hz)\n :hz (if (and (number? hz) (pos? hz)) (str (api/nice hz) \" Hz\") \"\")\n :bar (* 116 (api/clamp (or (:clarity shown) 0) 0 1))}))\n :view [[:rect {:x 2 :y 2 :w 120 :h 30 :r 3 :stroke :grid :fill :bg}]\n [:rect {:x 4 :y 28 :w :$bar :h 2 :r 1 :stroke nil :fill :accent}]\n [:text {:x 6 :y 20 :size 13} :$note]\n [:text {:x 118 :y 20 :anchor :end :size 9 :fill :soft} :$hz]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n args (:args b)\n [lo hi th] (api/padded args [60 1500 0.15])\n node {:in in :w (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}]\n (-> (api/worklet-node ac \"kairotic-yin\" worklet\n {:numberOfInputs 1 :numberOfOutputs 0 :channelCount 1 :channelCountMode \"explicit\"\n :processorOptions {:lo lo :hi hi :threshold th}})\n (.then (fn [w]\n (if @(:alive node)\n (do (set! (.. w -port -onmessage)\n (fn [e] (let [d (.-data e)]\n (when @(:alive node)\n (api/report! (:id b) {:hz (.-hz d) :clarity (.-clarity d)} (.-at d))))))\n (.connect in w)\n (reset! (:w node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"pitch~\" e))))\n node))\n ;; a new text: new limits, without a new worklet\n :update (fn [node b _]\n (when-let [w @(:w node)]\n (let [[lo hi th] (api/padded (:args b) [60 1500 0.15])]\n (.postMessage (.-port w) #js {:lo lo :hi hi :threshold th}))))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
43
|
+
"source": ";; pitch~ · the pitch of a sound, by YIN\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…), with its AudioWorklet inside.\n(let [worklet \"class KairoticYin extends AudioWorkletProcessor {\n // YIN (de Cheveigné & Kawahara, 2002) on the last W samples, every `hop` samples: the period whose\n // cumulative mean normalized difference first falls under the threshold (refined by a parabola),\n // between lo and hi Hz. Hands {hz, clarity, rms}; hz 0 where it finds no pitch or hears silence.\n constructor(o) {\n super();\n const p = (o && o.processorOptions) || {};\n this.W = 1024; this.hop = p.hop || 1024; this.count = 0;\n this.set(p.lo || 60, p.hi || 1500, p.threshold || 0.15);\n this.port.onmessage = (e) => { const d = e.data || {}; this.set(d.lo || this.lo, d.hi || this.hi, d.threshold || this.th); };\n }\n set(lo, hi, th) {\n this.lo = Math.max(20, Math.min(lo, hi)); this.hi = Math.max(this.lo + 1, hi); this.th = Math.max(0.01, Math.min(0.9, th));\n this.T = Math.min(4096, Math.ceil(sampleRate / this.lo) + 1);\n this.tmin = Math.max(2, Math.floor(sampleRate / this.hi));\n this.N = this.W + this.T + 1;\n this.ring = new Float32Array(this.N); this.buf = new Float32Array(this.N); this.d = new Float32Array(this.T + 2);\n this.w = 0;\n }\n analyse() {\n const N = this.N, b = this.buf, r = this.ring, W = this.W, T = this.T, d = this.d;\n for (let i = 0; i < N; i++) b[i] = r[(this.w + i) % N];\n let e = 0; for (let i = N - W; i < N; i++) e += b[i] * b[i];\n const rms = Math.sqrt(e / W);\n if (rms < 0.002) return { hz: 0, clarity: 0, rms };\n // the difference function, stopped as soon as the first dip under the threshold has its minimum\n d[0] = 1; let sum = 0, best = -1;\n for (let tau = 1; tau <= T; tau++) {\n let s = 0;\n for (let j = 0; j < W; j++) { const x = b[j] - b[j + tau]; s += x * x; }\n sum += s; d[tau] = sum > 0 ? s * tau / sum : 1;\n if (tau - 1 >= this.tmin && d[tau - 1] < this.th && d[tau] >= d[tau - 1]) { best = tau - 1; break; }\n }\n if (best < 1) return { hz: 0, clarity: 0, rms };\n let t = best;\n const a = d[best - 1], c = d[best], z = d[best + 1], den = a + z - 2 * c;\n if (den > 0) t = best + (a - z) / (2 * den);\n return { hz: sampleRate / t, clarity: Math.max(0, Math.min(1, 1 - c)), rms };\n }\n process(inputs) {\n const ch = inputs[0] && inputs[0][0];\n const n = ch ? ch.length : 128;\n for (let i = 0; i < n; i++) {\n this.ring[this.w] = ch ? ch[i] : 0;\n this.w = (this.w + 1) % this.N;\n if (++this.count >= this.hop) {\n this.count = 0;\n const r = this.analyse();\n this.port.postMessage({ hz: r.hz, clarity: r.clarity, rms: r.rms, at: currentTime + (i + 1) / sampleRate });\n }\n }\n return true;\n }\n}\nregisterProcessor('kairotic-yin', KairoticYin);\n\"\n ;; what it says between a pitch it heard and the last one it sent: a change of more than a tenth of a cent\n changed? (fn [a b] (or (nil? a) (nil? b) (zero? a) (> (js/Math.abs (- (api/hz->midi b) (api/hz->midi a))) 0.001)))\n label (fn [hz]\n (if (and (number? hz) (pos? hz))\n (let [m (api/hz->midi hz) r (js/Math.round m) c (js/Math.round (* 100 (- m r)))]\n (str (api/midi-name r) \" \" (if (neg? c) \"\" \"+\") c \"¢\"))\n \"–\"))]\n {;; ── about ──\n :class \"pitch~\"\n :license \"MPL-2.0\"\n :doc \"The pitch of the signal on its left, by YIN, every 1024 samples (about 21 ms): its frequency in Hz on the left, when it finds one between lo and hi (60 and 1500 by default), and 0 once when it no longer does (silence, noise, a chord); how clearly it is periodic (0…1) on the right. It sends only what changed. threshold (0.15) is how strict it is: lower finds fewer pitches, surer ones. Into ftom (Hz → MIDI pitch), a synth~, or a faust~.\"\n :usage \"pitch~ [lo hi] [threshold]\"\n :io [1 2]\n :size [124 34]\n :args [[:lo 60] [:hi 1500] [:threshold 0.15]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state]}]\n (if (and (= inlet :audio) (map? value))\n (let [{:keys [hz clarity]} value\n hz (if (and (number? hz) (pos? hz)) (/ (js/Math.round (* hz 100)) 100) 0)\n cl (/ (js/Math.round (* 100 (or clarity 0))) 100)\n last-hz (:hz state)\n send-hz (cond (pos? hz) (when (changed? last-hz hz) hz)\n (and last-hz (pos? last-hz)) 0)\n outs (cond-> []\n (some? send-hz) (conj [0 send-hz])\n (not= cl (:clarity state)) (conj [1 cl]))]\n (if (seq outs)\n {:state {:hz (if (some? send-hz) send-hz last-hz) :clarity cl} :out outs :show {:hz hz :clarity cl}}\n {}))\n {}))\n\n ;; ── view ──\n :derive (fn [{:keys [shown]}]\n (let [hz (:hz shown)]\n {:note (label hz)\n :hz (if (and (number? hz) (pos? hz)) (str (api/nice hz) \" Hz\") \"\")\n :bar (* 116 (api/clamp (or (:clarity shown) 0) 0 1))}))\n :view [[:rect {:x 2 :y 2 :w 120 :h 30 :r 3 :stroke :grid :fill :bg}]\n [:rect {:x 4 :y 28 :w :$bar :h 2 :r 1 :stroke nil :fill :accent}]\n [:text {:x 6 :y 20 :size 13} :$note]\n [:text {:x 118 :y 20 :anchor :end :size 9 :fill :soft} :$hz]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n args (:args b)\n [lo hi th] (api/padded args [60 1500 0.15])\n node {:in in :w (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}]\n (-> (api/worklet-node ac \"kairotic-yin\" worklet\n {:numberOfInputs 1 :numberOfOutputs 0 :channelCount 1 :channelCountMode \"explicit\"\n :processorOptions {:lo lo :hi hi :threshold th}})\n (.then (fn [w]\n (if @(:alive node)\n (do (set! (.. w -port -onmessage)\n (fn [e] (let [d (.-data e)]\n (when @(:alive node)\n (api/report! (:id b) {:hz (.-hz d) :clarity (.-clarity d)} (.-at d))))))\n (.connect in w)\n (reset! (:w node) w))\n (try (.disconnect w) (catch :default _ nil)))))\n (.catch (fn [e] (js/console.error \"pitch~\" e))))\n node))\n ;; a new text: new limits, without a new worklet\n :update (fn [node b _]\n (when-let [w @(:w node)]\n (let [[lo hi th] (api/padded (:args b) [60 1500 0.15])]\n (.postMessage (.-port w) #js {:lo lo :hi hi :threshold th}))))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (when-let [w @(:w node)] (try (set! (.. w -port -onmessage) nil) (.disconnect w) (catch :default _ nil)))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
"class": "realize~",
|
|
47
|
+
"license": "MPL-2.0",
|
|
48
|
+
"doc": "Plays a faust~ from descriptors, as a score of them: loudness (dB, or a level 0…1 as features~ rms), brightness (Hz, the spectral centroid), noisiness (0…1, the spectral flatness) and pitch (Hz as pitch~ sends it, 0 for none; MIDI with `midi`). Its left outlet goes to the faust~: the faust~'s pitch or freq slider takes the pitch, its gain slider the loudness, and its other sliders are set to the settings of that faust~ that sound nearest to the brightness and noisiness written. It learns them first by playing steady settings of the faust~ offline and listening (300 by default), when the first values arrive or on learn; learn again after changing the faust~'s code. With features~ of the faust~'s sound on its last three inlets (rms, centroid, flatness) it listens to what it plays and asks for more where it falls short. On the right, how far the nearest setting is from what is written (above 1: the faust~ cannot sound like that).",
|
|
49
|
+
"source": ";; realize~ · descriptors written → the sliders of a faust~ that sound like them (discussion #114, Study 37)\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…).\n;; What arrives is a frame of a score: a loudness, a brightness (the spectral centroid, Hz), a noisiness (the\n;; spectral flatness) and a pitch — what features~ and pitch~ hear, so another sound can drive it, or curves\n;; drawn on canvases read back (reader~ → snapshot~). What leaves is [:slider value] for the faust~ on its\n;; left outlet: its `pitch` (MIDI) or `freq` (Hz) slider takes the pitch, its `gain` (dB or a level) the\n;; loudness, and its other sliders, the timbre, are set to the blend of the settings of that faust~ that come\n;; nearest to the brightness and noisiness written (where a pitch is written, among the settings in which YIN\n;; would hear it). To know them, it learns the faust~ first (the task :learn: steady settings played offline\n;; and heard, @kairotic/faust-units' notation.mjs), when the first frame arrives or on `learn`.\n;; It can also listen to what it plays: features~ of the faust~'s sound on its last three inlets (loudness,\n;; brightness, noisiness heard). Then it asks for more where what is heard falls short of what is written,\n;; little by little (a correction that grows while the difference lasts, within limits), as the study's\n;; passes did offline.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n origins [\"npm:@kairotic/faust-units@0.1.5/dist/notation.mjs\" \"kairotic/packages/faust-units/notation.mjs\"]\n silent -60 ; dB: below it nothing is written (a microphone in a room is often near -50)\n every 0.04 ; seconds: at most one new setting this often\n ;; listening: how much of each difference heard goes into the correction, and how far it may go\n rate 0.08\n leak 0.995 ; what it corrected fades unless it is still needed\n hold 0.25 ; seconds a sound has lasted before it is corrected (not its attack)\n limits {:loud 18 :lb 2 :nz 0.4}\n bank-size (fn [args] (js/Math.round (api/clamp (or (first (filter number? args)) 300) 40 2000)))\n midi-in? (fn [args] (some #{:midi} args))\n log2 (fn [x] (/ (js/Math.log x) js/Math.LN2))\n ;; a loudness as dB: a level (0 < v ≤ 1, as features~ rms) or dB already (≤ 0); 0 is silence\n db-of (fn [v] (cond (not (number? v)) nil (zero? v) -100 (and (pos? v) (<= v 1)) (api/db v) :else v))\n ;; the nearest settings of the bank to what is asked, blended: {:u [...] :loud0 dB :d distance}. The search\n ;; itself is the bank's query (notation.mjs: plain loops over typed arrays), asked for every frame it plays\n nearest (fn [bank {:keys [pitch lb nz voiced]}]\n (let [r ((:query bank) pitch lb nz (boolean voiced))]\n {:loud0 (aget r 0) :d (aget r 1) :u (vec (.slice r 2))}))\n ;; the values for the faust~, and only those that changed: [[0 [:cutoff 1800]] …]\n settle (fn [st at]\n (let [{:keys [bank]} st\n ;; a lane that never arrived keeps the middle of what the faust~ can do\n want (merge {:lb (first (:mid bank)) :nz (second (:mid bank))} (:want st))\n ;; what is asked of the bank: what is written, corrected by what was heard\n {cl :loud clb :lb cnz :nz} (:corr st)\n [lb-lo lb-hi] (or (:lbr bank) [0 20])\n ask (cond-> want\n (and cl (:loud want)) (update :loud + cl)\n clb (assoc :lb (api/clamp (+ (:lb want) clb) lb-lo lb-hi))\n cnz (assoc :nz (api/clamp (+ (:nz want) cnz) 0 1)))\n loud (:loud want)]\n (if (and (nil? loud) (nil? (:pitch want)) (nil? (get-in st [:want :lb])) (nil? (get-in st [:want :nz])))\n {:state (assoc st :t at)}\n (let [quiet? (and loud (< loud silent))\n {:keys [u loud0 d]} (if quiet? {:u (:u st) :loud0 0 :d (:reach st)} (nearest bank ask))\n ;; the timbre glides to where it goes (half the way every 40 ms), and keeps\n ;; gliding when nothing more arrives (a value written once, a canvas held)\n goal u\n u (if (and (:u st) (not quiet?)) (mapv (fn [a b] (+ a (* 0.5 (- b a)))) (:u st) u) u)\n moving? (and goal u (not quiet?) (some (fn [[a b]] (> (js/Math.abs (- a b)) 0.004)) (map vector u goal)))\n g (:gain bank) p (:pitch bank)\n gain-v (when (and g loud) (let [dB (if quiet? -100 (- (:loud ask) loud0))]\n (if (:db g) (api/clamp dB (:min g) (:max g))\n (api/clamp (js/Math.pow 10 (/ dB 20)) (:min g) (:max g)))))\n pitch-v (when (and p (:pitch want))\n (api/clamp (if (= \"hz\" (:kind p)) (api/midi->hz (:pitch want)) (:pitch want)) (:min p) (:max p)))\n vals (cond-> (if (and u (not quiet?))\n (into {} (map (fn [s x] [(:label s) (api/from-unit x (:min s) (:max s) (:log s))]) (:timbre bank) u))\n {})\n gain-v (assoc (:label g) gain-v)\n pitch-v (assoc (:label p) pitch-v))\n span (fn [label] (let [s (or (first (filter #(= label (:label %)) (:timbre bank))) (when (= label (:label g)) g) p)]\n (js/Math.abs (- (:max s) (:min s)))))\n changed (filter (fn [[k v]] (let [old (get (:sent st) k)]\n (or (nil? old) (> (js/Math.abs (- v old)) (* 0.002 (span k))))))\n vals)\n reach (when d (/ (js/Math.round (* 100 d)) 100))]\n {:state (-> st (assoc :t at :u (or u (:u st)) :reach reach :dirty (boolean moving?)\n :loud0 (if quiet? (:loud0 st) loud0)\n :tick (or (:tick st) (boolean moving?)))\n (update :sent merge (into {} changed)))\n :later (when (and moving? (not (:tick st))) [[(+ at every) 0 :realize-tick]])\n :out (cond-> (mapv (fn [[k v]] [0 [(keyword k) (api/nice v)]]) changed)\n (and reach (not= reach (:reach st))) (conj [1 reach]))\n :show {:status :ready :faust (:faust bank) :n (count (:entries bank)) :want want :reach reach\n :listening (some? (:corr st))}}))))\n ;; each request is numbered, so the answer to an older one is dropped\n learn (fn [st args]\n (let [k (inc (:k st 0))]\n {:state (assoc st :learning true :error nil :k k :corr nil)\n :show {:status :learning}\n :effects [[:learn {:n (bank-size args) :k k}]]}))]\n {;; ── about ──\n :class \"realize~\"\n :license \"MPL-2.0\"\n :doc \"Plays a faust~ from descriptors, as a score of them: loudness (dB, or a level 0…1 as features~ rms), brightness (Hz, the spectral centroid), noisiness (0…1, the spectral flatness) and pitch (Hz as pitch~ sends it, 0 for none; MIDI with `midi`). Its left outlet goes to the faust~: the faust~'s pitch or freq slider takes the pitch, its gain slider the loudness, and its other sliders are set to the settings of that faust~ that sound nearest to the brightness and noisiness written. It learns them first by playing steady settings of the faust~ offline and listening (300 by default), when the first values arrive or on learn; learn again after changing the faust~'s code. With features~ of the faust~'s sound on its last three inlets (rms, centroid, flatness) it listens to what it plays and asks for more where it falls short. On the right, how far the nearest setting is from what is written (above 1: the faust~ cannot sound like that).\"\n :usage \"realize~ [settings] [midi]\"\n :io [7 2]\n :size [236 46]\n\n ;; ── tasks: learning the faust~ it drives, outside its pure logic ──\n :tasks {:learn (fn [{:keys [n]} box]\n (let [f (first (filter (fn [w] (and (= \"faust~\" (:class w)) (zero? (:outlet w)))) (:wired box)))\n src (str (get-in f [:data :source]))\n ;; the compilation of its code, waited for (a faust~ just written is still compiling)\n compiled (fn compiled [tries]\n (let [r (api/compiled \"faust\" src)]\n (cond (= :ok (:status r)) (js/Promise.resolve r)\n (and (#{:compiling :empty} (:status r)) (pos? tries))\n (js/Promise. (fn [ok] (js/setTimeout (fn [] (ok nil)) 250)))\n :else (js/Promise.reject (js/Error. (str \"the faust~ does not compile: \" (or (:error r) \"no code\")))))))\n wait (fn wait [tries]\n (.then (compiled tries) (fn [r] (if r r (wait (dec tries))))))]\n (if-not f\n (js/Promise.reject (js/Error. \"connect its left outlet to a faust~: it learns the one it plays\"))\n (-> (wait 120)\n (.then (fn [r]\n (when (get-in r [:info :poly])\n (throw (js/Error. \"it plays a faust~ of one voice (this one declares nvoices)\")))\n (let [fac (.-factory (:value r))]\n (-> (js/Promise.all #js [(api/load-script meyda-url) (api/import-module origins)])\n (.then (fn [[_ m]]\n (.then (.learn m #js {:module (or (.-module fac) (.-code fac))\n :json (js/JSON.parse (.-json fac))\n :Meyda (.-Meyda js/window) :n n})\n (fn [b] [b (.index m b)]))))))))\n (.then (fn [[bjs query]]\n ;; the settings stay a JS array (the query reads them); the rest as data\n (let [es (.-entries bjs)\n col (fn [j] (map (fn [e] (aget e j)) (array-seq es)))\n avg (fn [j] (/ (reduce + (col j)) (.-length es)))]\n (-> (js->clj #js {:pitch (.-pitch bjs) :gain (.-gain bjs) :timbre (.-timbre bjs) :sd (.-sd bjs)}\n :keywordize-keys true)\n (assoc :query query :entries es :faust (name (:id f)) :mid [(avg 1) (avg 2)]\n :lbr [(reduce min (col 1)) (reduce max (col 1))])))))))))}\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args at]}]\n (let [st (or state {})]\n (cond\n ;; what learning found\n (= inlet :task)\n (cond\n (not= (:k (:arg value)) (:k st)) {}\n (:error value) {:state (assoc st :learning false :error (:error value))\n :show {:status :error :error (:error value)}}\n :else (let [st (assoc st :learning false :bank (:value value) :sent {} :u nil)]\n (settle st (or at (:t st) 0))))\n\n (#{:learn :bang} value) (learn st args)\n\n ;; a moment later: the last frame that arrived while it waited\n (= value :realize-tick) (if (and (:bank st) (:dirty st)) (settle (assoc st :tick false) at) {:state (assoc st :tick false)})\n\n ;; what is heard in what it plays (features~ of the faust~): the correction grows by a part of\n ;; the difference, while something is written and sounds\n (and (#{4 5 6} inlet) (number? value))\n (let [heard (case inlet 4 (db-of value) 5 (when (pos? value) (log2 value)) 6 value)\n k ({4 :loud 5 :lb 6 :nz} inlet)\n want (:want st)\n st (cond-> st (= k :loud) (assoc :heard-loud heard))\n sounding? (and (:bank st) (number? (:loud want)) (> (:loud want) silent)\n (> (or (:heard-loud st) -100) silent))]\n (if (and sounding? (number? heard) (number? (get want k)) (:since st) (>= (- at (:since st)) hold))\n ;; within what the faust~ can give (the bank's brightness, noisiness 0…1, the gain slider's\n ;; range): a correction that cannot be played does not grow, so it does not wind up\n (let [bank (:bank st) w (get want k) g (:gain bank) [lb-lo lb-hi] (or (:lbr bank) [0 20])\n [lo hi] (case k\n :lb [(- lb-lo w) (- lb-hi w)]\n :nz [(- w) (- 1 w)]\n :loud (if (and g (:db g) (:loud0 st))\n [(+ (:min g) (:loud0 st) (- w)) (+ (:max g) (:loud0 st) (- w))]\n [(- (limits k)) (limits k)]))\n c (-> (+ (* leak (get-in st [:corr k] 0)) (* rate (- w heard)))\n (api/clamp (max lo (- (limits k))) (min hi (limits k))))\n st (-> st (assoc-in [:corr k] c) (assoc :dirty true))]\n (if (or (nil? (:t st)) (>= (- at (:t st)) every))\n (settle st at)\n (if (:tick st) {:state st} {:state (assoc st :tick true) :later [[(+ (:t st) every) 0 :realize-tick]]})))\n {:state st}))\n\n (and (integer? inlet) (number? value))\n (let [midi (when (and (= inlet 3) (pos? value)) (if (midi-in? args) value (api/hz->midi value)))\n want (case inlet\n 0 (assoc (:want st) :loud (db-of value))\n 1 (if (pos? value) (assoc (:want st) :lb (log2 value)) (:want st))\n 2 (assoc (:want st) :nz (api/clamp value 0 1))\n 3 (if midi (assoc (:want st) :pitch midi :voiced true) (assoc (:want st) :voiced false))\n (:want st))\n ;; since when what is written sounds (its attack is not corrected)\n st (cond-> (assoc st :want want :dirty true)\n (= inlet 0) (assoc :since (if (> (or (:loud want) -100) silent) (or (:since st) at) nil)))]\n (cond\n (:bank st) (if (or (nil? (:t st)) (>= (- at (:t st)) every))\n (settle st at)\n (if (:tick st)\n {:state st}\n {:state (assoc st :tick true) :later [[(+ (:t st) every) 0 :realize-tick]]}))\n (or (:learning st) (:error st)) {:state st}\n :else (learn st args)))\n\n :else {})))\n\n ;; ── view: what it learnt, what is written now, and how far the faust~ is from it ──\n :draw (fn [{:keys [shown w h]}]\n (let [{:keys [status faust n want reach error]} shown\n line1 (case status\n :learning \"learning the faust~ it plays…\"\n :ready (str \"plays \" faust \" · \" n \" settings heard\" (when (:listening shown) \" · listening\"))\n :error (str \"✗ \" error)\n \"waits for a faust~ and a score\")\n fmt (fn [x d] (if (number? x) (.toFixed x d) \"–\"))\n line2 (when (= status :ready)\n (str (fmt (:loud want) 0) \" dB · \"\n (if (:lb want) (str (js/Math.round (js/Math.pow 2 (:lb want))) \" Hz\") \"–\") \" · \"\n (fmt (:nz want) 2)\n (when (and (:voiced want) (:pitch want)) (str \" · \" (api/midi-name (js/Math.round (:pitch want)))))))\n bw (- w 16) far (when (number? reach) (api/clamp (/ reach 2) 0 1))]\n (cond-> [[:rect {:x 2 :y 2 :w (- w 4) :h (- h 4) :r 4 :stroke :grid :fill :bg}]\n [:text {:x 8 :y 15 :size 9 :fill (if (= status :error) :kairos :ink)} line1]]\n line2 (conj [:text {:x 8 :y 28 :size 9 :fill :soft} line2])\n far (conj [:rect {:x 8 :y 35 :w bw :h 4 :r 2 :fill :grid :stroke nil}]\n [:rect {:x 8 :y 35 :w (max 2 (* bw far)) :h 4 :r 2 :fill (if (> reach 1) :kairos :accent) :stroke nil}]))))})\n"
|
|
38
50
|
},
|
|
39
51
|
{
|
|
40
52
|
"class": "spectrum~",
|
|
41
53
|
"license": "MPL-2.0",
|
|
42
54
|
"doc": "The spectrum of the signal on its left, drawn in the patch about 30 times a second: frequency on a log scale from lo to hi Hz (40 and 16000 by default), level in dB from -90 to 0. Computed with Meyda.",
|
|
43
|
-
"source": ";; spectrum~ · the spectrum of a sound, drawn in the patch\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n w 240 h 96 cols 116 ; the picture: 116 columns of 2 px, from x 4 to 236\n floor -90 ; dB at the bottom\n ;; for each bin of the spectrum, its column on a log scale of frequency from lo to hi (nil outside)\n column-map (fn [sr lo hi]\n (let [l0 (js/Math.log lo) span (- (js/Math.log hi) l0)]\n (vec (for [k (range (/ size 2))]\n (let [f (* k (/ sr size))]\n (when (and (>= f lo) (< f hi))\n (js/Math.floor (* cols (/ (- (js/Math.log f) l0) span)))))))))\n ;; the loudest bin of each column, in dB below a full-scale sine (whose bin is size/4 with Hann)\n columns (fn [spec cmap]\n (let [out (js/Array. cols)]\n (.fill out floor)\n (loop [k 1]\n (when (< k (count cmap))\n (when-let [c (nth cmap k)]\n (let [a (aget spec k)\n d (if (> a 1e-9) (* 20 (/ (js/Math.log (/ a (/ size 4))) js/Math.LN10)) floor)]\n (when (> d (aget out c)) (aset out c d))))\n (recur (inc k))))\n ;; columns with no bin of their own (low frequencies) take the one before\n (loop [c 1]\n (when (< c cols)\n (when (= floor (aget out c)) (aset out c (aget out (dec c))))\n (recur (inc c))))\n (vec out)))\n points (fn [ds]\n (vec (map-indexed (fn [i d] [(+ 4 (* 2 i)) (+ 4 (* (- h 18) (/ (- (api/clamp d floor 0)) (- floor))))]) ds)))]\n {;; ── about ──\n :class \"spectrum~\"\n :license \"MPL-2.0\"\n :doc \"The spectrum of the signal on its left, drawn in the patch about 30 times a second: frequency on a log scale from lo to hi Hz (40 and 16000 by default), level in dB from -90 to 0. Computed with Meyda.\"\n :usage \"spectrum~ [lo hi]\"\n :io [1 0]\n :size [240 96]\n :args [[:lo 40] [:hi 16000]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (vector? value)) {:show value}\n :else {}))\n\n ;; ── view ──\n :derive (fn [{:keys [shown args]}]\n {:pts (when (vector? shown) (points shown))\n :range (str (api/nice (:lo args)) \" – \" (api/nice (:hi args)) \" Hz\")})\n :view [[:rect {:x 2 :y 2 :w 236 :h 92 :r 4 :stroke :grid :fill :bg}]\n [:line {:x1 4 :y1 30 :x2 236 :y2 30 :stroke :grid}]\n [:line {:x1 4 :y1 54 :x2 236 :y2 54 :stroke :grid}]\n [:path {:points :$pts :stroke :accent :weight 1.2}]\n [:text {:x 234 :y 90 :anchor :end :size 8 :fill :soft} :$range]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n [lo hi] (api/padded (:args b) [40 16000])\n cmap (column-map sr (api/clamp (min lo hi) 10 20000) (api/clamp (max lo hi) 20 (/ sr 2)))\n node {:in in :M (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n (when-let [M @(:M node)]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [spec (.extract M \"amplitudeSpectrum\" (first data))]\n (api/report! (:id b) (columns spec cmap) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"spectrum~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (/ sr 30))} on-block)\n (.then (fn [w] (if @(:alive node) (.connect in w) (
|
|
55
|
+
"source": ";; spectrum~ · the spectrum of a sound, drawn in the patch\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n w 240 h 96 cols 116 ; the picture: 116 columns of 2 px, from x 4 to 236\n floor -90 ; dB at the bottom\n ;; for each bin of the spectrum, its column on a log scale of frequency from lo to hi (nil outside)\n column-map (fn [sr lo hi]\n (let [l0 (js/Math.log lo) span (- (js/Math.log hi) l0)]\n (vec (for [k (range (/ size 2))]\n (let [f (* k (/ sr size))]\n (when (and (>= f lo) (< f hi))\n (js/Math.floor (* cols (/ (- (js/Math.log f) l0) span)))))))))\n ;; the loudest bin of each column, in dB below a full-scale sine (whose bin is size/4 with Hann)\n columns (fn [spec cmap]\n (let [out (js/Array. cols)]\n (.fill out floor)\n (loop [k 1]\n (when (< k (count cmap))\n (when-let [c (nth cmap k)]\n (let [a (aget spec k)\n d (if (> a 1e-9) (* 20 (/ (js/Math.log (/ a (/ size 4))) js/Math.LN10)) floor)]\n (when (> d (aget out c)) (aset out c d))))\n (recur (inc k))))\n ;; columns with no bin of their own (low frequencies) take the one before\n (loop [c 1]\n (when (< c cols)\n (when (= floor (aget out c)) (aset out c (aget out (dec c))))\n (recur (inc c))))\n (vec out)))\n points (fn [ds]\n (vec (map-indexed (fn [i d] [(+ 4 (* 2 i)) (+ 4 (* (- h 18) (/ (- (api/clamp d floor 0)) (- floor))))]) ds)))]\n {;; ── about ──\n :class \"spectrum~\"\n :license \"MPL-2.0\"\n :doc \"The spectrum of the signal on its left, drawn in the patch about 30 times a second: frequency on a log scale from lo to hi Hz (40 and 16000 by default), level in dB from -90 to 0. Computed with Meyda.\"\n :usage \"spectrum~ [lo hi]\"\n :io [1 0]\n :size [240 96]\n :args [[:lo 40] [:hi 16000]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n (and (= inlet :audio) (vector? value)) {:show value}\n :else {}))\n\n ;; ── view ──\n :derive (fn [{:keys [shown args]}]\n {:pts (when (vector? shown) (points shown))\n :range (str (api/nice (:lo args)) \" – \" (api/nice (:hi args)) \" Hz\")})\n :view [[:rect {:x 2 :y 2 :w 236 :h 92 :r 4 :stroke :grid :fill :bg}]\n [:line {:x1 4 :y1 30 :x2 236 :y2 30 :stroke :grid}]\n [:line {:x1 4 :y1 54 :x2 236 :y2 54 :stroke :grid}]\n [:path {:points :$pts :stroke :accent :weight 1.2}]\n [:text {:x 234 :y 90 :anchor :end :size 8 :fill :soft} :$range]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n [lo hi] (api/padded (:args b) [40 16000])\n cmap (column-map sr (api/clamp (min lo hi) 10 20000) (api/clamp (max lo hi) 20 (/ sr 2)))\n node {:in in :M (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [spec (.extract M \"amplitudeSpectrum\" (first data))]\n (api/report! (:id b) (columns spec cmap) at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"spectrum~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop (js/Math.round (/ sr 30))} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w)) (api/untap w))))\n (.catch (fn [e] (js/console.error \"spectrum~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
44
56
|
},
|
|
45
57
|
{
|
|
46
58
|
"class": "transcribe",
|
|
47
59
|
"license": "MPL-2.0",
|
|
48
|
-
"doc": "Writes what pitch~ hears into a freqt, as strokes. Hz on the left (from pitch~: 0 when it hears none), rec to start from 0 (its first write clears the freqt) or rec t (msg rec 8) to start from beat t, keeping what is before it, or ovr t (msg ovr 8) to write over only what the take covers, keeping what comes after and stop to end; a bang on the right (from onset~) begins a new note even on the same pitch. In notes mode each
|
|
49
|
-
"source": ";; transcribe · what pitch~ hears, written as strokes of a freqt\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). Pure logic: no audio of its own.\n(let [;; ── what it understands ──\n flush-every 2 ; seconds between two writes into the freqt (each write is one event of the history)\n max-points 64 ; points of a stroke in glide mode\n cents (fn [a b] (* 1200 (/ (js/Math.log (/ b a)) js/Math.LN2)))\n ;; the pitch a note holds longest: pitch~ only sends what changed, so a steady note is one point and\n ;; its attack many; each pitch weighs the time until the next one\n held (fn [pts t1]\n (let [ws (map (fn [[t f] [t2 _]] [f (max 0 (- t2 t))]) pts (concat (rest pts) [[t1 nil]]))\n total (reduce + (map second ws))\n sorted (sort-by first ws)]\n (if (pos? total)\n (loop [[[f w] & more] sorted acc 0]\n (let [acc (+ acc w)] (if (or (>= acc (/ total 2)) (empty? more)) f (recur more acc))))\n (second (last pts)))))\n mode-of (fn [args] (if (= :glide (:mode args)) :glide :notes))\n beats (fn [state at] (* (- at (:t0 state)) (/ (:bpm state) 60)))\n r3 (fn [x] (/ (js/Math.round (* x 1000)) 1000))\n ;; ── a note that ends: a stroke, if it lasted long enough ──\n stroke-of (fn [{:keys [pts]} t1 mode]\n (let [t0 (ffirst pts) hz (held pts t1)]\n (if (= mode :glide)\n (let [kept (reduce (fn [acc [t f]]\n (if (> (js/Math.abs (cents (second (peek acc)) f)) 10) (conj acc [t f]) acc))\n [(first pts)] (rest pts))\n kept (if (> (count kept) max-points)\n (vec (take-nth (js/Math.ceil (/ (count kept) max-points)) kept))\n kept)]\n (mapv (fn [[t f]] [(r3 t) (r3 f)]) (conj kept [t1 (second (peek pts))])))\n [[(r3 t0) (r3 hz)] [(r3 t1) (r3 hz)]])))\n end-note (fn [state t]\n (if-let [cur (:cur state)]\n (let [t0 (ffirst (:pts cur))\n long-enough (>= (/ (- t t0) (/ (:bpm state) 60)) (/ (:min-ms state) 1000))]\n (cond-> (dissoc state :cur)\n long-enough (update :done conj (stroke-of cur t (:mode state)))\n long-enough (update :count inc)))\n state))\n start-note (fn [state t hz] (assoc state :cur {:pts [[t hz]] :midi (api/hz->midi hz)}))\n ;; ── writing: the strokes that ended, as one [:batch …], at most every flush-every seconds ──\n flush (fn [state at force]\n (let [now (r3 (beats state at))]\n (cond\n ;; ovr at stop: what the take covered since the last write is overwritten, even by silence\n (and (empty? (:done state)) (not (and force (:ovr state) (> now (:cut-from state))))) [state nil nil]\n (or force (nil? (:sent state)) (>= (- at (:sent state)) flush-every))\n ;; the first write of a take clears the freqt (as part of the same event); a take over (ovr)\n ;; cuts what it covered since the last write, and keeps what comes after\n [(cond-> (assoc state :done [] :sent at :pending false :fresh false)\n (:ovr state) (assoc :cut-from now))\n [:batch (into (cond (:ovr state) [[:cut (:cut-from state) now]]\n (:fresh state) [(if (:from state) [:cut (:from state)] [:clear])]\n :else [])\n (mapv (fn [pts] [:stroke pts]) (:done state)))] nil]\n (:pending state) [state nil nil]\n :else [(assoc state :pending true) nil [(+ (:sent state) flush-every) 0 [:flush]]])))\n result (fn [state [st edit later]]\n (cond-> {:state st :show (select-keys st [:on :count])}\n edit (assoc :out [[0 edit]])\n later (assoc :later [later])))\n ;; ── one value of pitch~ (Hz, 0 when it hears none) ──\n hear (fn [state hz at]\n (let [t (beats state at)\n cur (:cur state)\n state (cond\n (not (pos? hz)) (end-note state t)\n (nil? cur) (start-note state t hz)\n ;; notes: another note when the pitch moves more than 0.6 of a semitone away\n (and (= :notes (:mode state)) (> (js/Math.abs (- (api/hz->midi hz) (:midi cur))) 0.6))\n (-> state (end-note t) (start-note t hz))\n :else (update-in state [:cur :pts] conj [t hz]))]\n (flush state at false)))]\n {;; ── about ──\n :class \"transcribe\"\n :license \"MPL-2.0\"\n :doc \"Writes what pitch~ hears into a freqt, as strokes. Hz on the left (from pitch~: 0 when it hears none), rec to start from 0 (its first write clears the freqt) or rec t (msg rec 8) to start from beat t, keeping what is before it, or ovr t (msg ovr 8) to write over only what the take covers, keeping what comes after and stop to end; a bang on the right (from onset~) begins a new note even on the same pitch. In notes mode each note is a flat stroke at the pitch it holds longest (the freqt's tuning, or its score view, rounds it); in glide mode a stroke follows the pitch as it moves. Times are beats at bpm (60: one beat a second). Notes shorter than ms (60) are left out. It writes into the freqt at most every 2 seconds, each time one event of the history. Connect its outlet to the freqt's inlet.\"\n :usage \"transcribe [bpm] [notes|glide] [ms]\"\n :io [2 1]\n :size [124 30]\n :args [[:bpm 60] [:mode nil] [:min-ms 60]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args at]}]\n (let [state (or state {})]\n (cond\n ;; rec: a take from 0 (its first write clears the freqt); [:rec t] (msg rec 8): from beat t on,\n ;; keeping what is before t (punch-in)\n ;; ovr t (msg ovr 8): over what is there from beat t to where the take stops, keeping what comes after\n (or (#{:rec :ovr} value) (and (vector? value) (#{:rec :ovr} (first value))))\n (let [bpm (api/clamp (:bpm args) 10 400)\n ovr (or (= value :ovr) (and (vector? value) (= :ovr (first value))))\n from (or (when (vector? value) (let [x (second value)] (when (and (number? x) (>= x 0)) x)))\n (when ovr 0))]\n {:state (cond-> {:on true :t0 (- at (* (or from 0) (/ 60 bpm))) :bpm bpm :mode (mode-of args)\n :min-ms (api/clamp (:min-ms args) 0 2000) :done [] :count 0 :fresh true}\n from (assoc :from from)\n ovr (assoc :ovr true :cut-from from))\n :show {:on true :count 0}})\n (not (:on state)) {}\n (= value :stop)\n (let [st (end-note state (beats state at))\n [st edit _] (flush st at true)]\n (cond-> {:state (assoc st :on false) :show {:on false :count (:count st)}}\n edit (assoc :out [[0 edit]])))\n (= value [:flush]) (result state (flush (assoc state :pending false) at false))\n ;; an onset: the note that sounds ends here and a new one begins at its pitch\n (and (= inlet 1) (:cur state))\n (let [t (beats state at) hz (second (peek (get-in state [:cur :pts])))]\n (result state (flush (-> state (end-note t) (start-note t hz)) at false)))\n (and (= inlet 0) (number? value)) (result state (hear state value at))\n :else {})))\n\n ;; ── view ──\n :derive (fn [{:keys [shown]}]\n {:label (if (:on shown) (str \"● transcribing · \" (or (:count shown) 0)) (str \"transcribe · \" (or (:count shown) 0) \" notes\"))})\n :view [[:rect {:x 2 :y 2 :w 120 :h 26 :r 3 :stroke :grid :fill :bg}]\n [:text {:x 8 :y 19 :size 10 :fill :soft} :$label]]})\n"
|
|
60
|
+
"doc": "Writes what pitch~ hears into a freqt, as strokes. Hz on the left (from pitch~: 0 when it hears none), :rec to start from 0 (its first write clears the freqt) or :rec t (msg :rec 8) to start from beat t, keeping what is before it, or :ovr t (msg :ovr 8) to write over only what the take covers, keeping what comes after and :stop to end; a bang on the right (from onset~) begins a new note even on the same pitch, and a number there (onset~'s strength, 0…1) does too and gives the note that weight: its stroke is written heavier (as an event's :amp), and the quantizer reads it as the note's :vel, which helps find the beat and the bar (issue #106). In notes mode the contour of each voiced stretch is cut into notes where it settles on another pitch (a slide, a vibrato or an ornament that passes does not start notes; an onset on the right does), each a flat stroke at its median pitch (the freqt's tuning, or its score view, rounds it), written when the voice stops; in glide mode a stroke follows the pitch as it moves. Times are beats at bpm (60: one beat a second). Notes shorter than ms (80) are left out. It writes into the freqt at most every 2 seconds, each time one event of the history. Connect its outlet to the freqt's inlet.",
|
|
61
|
+
"source": ";; transcribe · what pitch~ hears, written as strokes of a freqt\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). Pure logic: no audio of its own.\n(let [;; ── what it understands ──\n flush-every 2 ; seconds between two writes into the freqt (each write is one event of the history)\n max-points 64 ; points of a stroke in glide mode\n cents (fn [a b] (* 1200 (/ (js/Math.log (/ b a)) js/Math.LN2)))\n ;; the pitch a note holds longest: pitch~ only sends what changed, so a steady note is one point and\n ;; its attack many; each pitch weighs the time until the next one\n held (fn [pts t1]\n (let [ws (map (fn [[t f] [t2 _]] [f (max 0 (- t2 t))]) pts (concat (rest pts) [[t1 nil]]))\n total (reduce + (map second ws))\n sorted (sort-by first ws)]\n (if (pos? total)\n (loop [[[f w] & more] sorted acc 0]\n (let [acc (+ acc w)] (if (or (>= acc (/ total 2)) (empty? more)) f (recur more acc))))\n (second (last pts)))))\n mode-of (fn [args] (if (= :glide (:mode args)) :glide :notes))\n beats (fn [state at] (* (- at (:t0 state)) (/ (:bpm state) 60)))\n r3 (fn [x] (/ (js/Math.round (* x 1000)) 1000))\n ;; ── a note that ends: a stroke, if it lasted long enough ──\n stroke-of (fn [{:keys [pts]} t1 mode]\n (let [t0 (ffirst pts) hz (held pts t1)]\n (if (= mode :glide)\n (let [kept (reduce (fn [acc [t f]]\n (if (> (js/Math.abs (cents (second (peek acc)) f)) 10) (conj acc [t f]) acc))\n [(first pts)] (rest pts))\n kept (if (> (count kept) max-points)\n (vec (take-nth (js/Math.ceil (/ (count kept) max-points)) kept))\n kept)]\n (mapv (fn [[t f]] [(r3 t) (r3 f)]) (conj kept [t1 (second (peek pts))])))\n [[(r3 t0) (r3 hz)] [(r3 t1) (r3 hz)]])))\n end-note (fn [state t]\n (if-let [cur (:cur state)]\n (let [t0 (ffirst (:pts cur))\n long-enough (>= (/ (- t t0) (/ (:bpm state) 60)) (/ (:min-ms state) 1000))]\n (cond-> (dissoc state :cur)\n long-enough (update :done conj {:pts (stroke-of cur t (:mode state)) :acc (:acc cur)})\n long-enough (update :count inc)))\n state))\n ;; a note takes the strength of the onset that began it (a number on the right inlet, from onset~), if it\n ;; came at most 0.15 s before: onset~ often hears the attack before pitch~ hears the pitch\n start-note (fn [state t hz]\n (let [[at s] (:acc state) acc (when (and at (<= (- t at) (* 0.15 (/ (:bpm state) 60)))) s)]\n (cond-> (assoc state :cur (cond-> {:pts [[t hz]] :midi (api/hz->midi hz)} acc (assoc :acc acc)))\n acc (dissoc :acc))))\n ;; how a note with the strength of its attack is written: the weight of its stroke, as an event's :amp is\n ;; (0 … 1 → 0.4 … 2.8 times the usual); the quantizer reads it back as the note's :vel (issue #106)\n edit-of (fn [{:keys [pts acc]}] (if (number? acc) [:stroke pts {:w (/ (js/Math.round (* 100 (+ 0.4 (* 2.4 acc)))) 100)}] [:stroke pts]))\n ;; ── writing: the strokes that ended, as one [:batch …], at most every flush-every seconds ──\n flush (fn [state at force]\n (let [now (r3 (beats state at))]\n (cond\n ;; ovr at stop: what the take covered since the last write is overwritten, even by silence\n (and (empty? (:done state)) (not (and force (:ovr state) (> now (:cut-from state))))) [state nil nil]\n (or force (nil? (:sent state)) (>= (- at (:sent state)) flush-every))\n ;; the first write of a take clears the freqt (as part of the same event); a take over (ovr)\n ;; cuts what it covered since the last write, and keeps what comes after\n [(cond-> (assoc state :done [] :sent at :pending false :fresh false)\n (:ovr state) (assoc :cut-from now))\n [:batch (into (cond (:ovr state) [[:cut (:cut-from state) now]]\n (:fresh state) [(if (:from state) [:cut (:from state)] [:clear])]\n :else [])\n (mapv edit-of (:done state)))] nil]\n (:pending state) [state nil nil]\n :else [(assoc state :pending true) nil [(+ (:sent state) flush-every) 0 [:flush]]])))\n result (fn [state [st edit later]]\n (cond-> {:state st :show (select-keys st [:on :count])}\n edit (assoc :out [[0 edit]])\n later (assoc :later [later])))\n ;; ── notes mode: the contour of each voiced stretch, segmented (issue #106, measured on cante2midi) ──\n ;; What pitch~ sends while a voice sounds is kept as a run of points; when the voice stops (or the run is long)\n ;; the run is read as a contour every 10 ms, smoothed by a median of 100 ms (vibrato and jitter), and cut into\n ;; notes with hysteresis: a note goes on while the contour stays within a semitone of its pitch; a new one\n ;; begins only where the contour, away from it, has held still (within 1.1 semitones) for 70 ms, so a slide\n ;; or an ornament that passes does not start notes. An onset (inlet 1) cuts there too. Each note is written\n ;; at the median pitch of its frames. Silences of 30 ms at most are bridged.\n median (fn [xs] (let [v (vec (sort xs))] (nth v (quot (count v) 2))))\n segment (fn [run marks t-end k minlen]\n (let [dt (* 0.01 k) w 5 band 1 stab 1.1 hold-n 7\n ts (vec (range (ffirst run) t-end dt))\n ;; the contour on the grid: each frame, the last point at or before it (as MIDI)\n raw (loop [i 0 [p & more :as ps] run cur nil out (transient [])]\n (if (>= i (count ts))\n (persistent! out)\n (let [t (ts i)]\n (if (and p (<= (first p) (+ t 1e-9)))\n (recur i more (api/hz->midi (second p)) out)\n (recur (inc i) ps cur (conj! out cur))))))\n n (count raw)\n sm (mapv (fn [i] (median (subvec raw (max 0 (- i w)) (min n (+ i w 1))))) (range n))\n cut? (fn [i] (let [a (if (pos? i) (ts (dec i)) (- (ts 0) dt)) b (ts i)] (some (fn [[mt]] (and (> mt a) (<= mt b))) marks)))\n note (fn [s e ms] {:s s :e e :ms ms})\n notes (loop [i 0 cur nil pend [] out []]\n (if (>= i n)\n (if cur (conj out (assoc cur :e t-end)) out)\n (let [t (ts i) m (sm i)]\n (cond\n (nil? cur) (recur (inc i) (note t (+ t dt) [m]) [] out)\n ;; (not while the contour is away from the note and not settled yet: there the contour\n ;; decides; measured on cante2midi with onsets from its Bark bands)\n (and (empty? pend) (cut? i) (> t (:s cur))) (recur (inc i) (note t (+ t dt) [m]) [] (conj out (assoc cur :e t)))\n (<= (js/Math.abs (- m (median (take-last 24 (:ms cur))))) band)\n (recur (inc i) (assoc cur :e (+ t dt) :ms (into (conj (:ms cur) m) (map second pend))) [] out)\n :else\n (let [pend (conj pend [t m]) tail (take-last hold-n pend) tm (map second tail)]\n (if (and (>= (count pend) hold-n) (< (- (apply max tm) (apply min tm)) stab))\n ;; the new note begins where the contour left the old one; its pitch is that of where it settled\n (recur (inc i) (note (ffirst pend) (+ t dt) (vec tm)) [] (conj out (assoc cur :e (ffirst pend))))\n (recur (inc i) cur pend out)))))))]\n (->> notes\n (filter (fn [{:keys [s e]}] (>= (- e s) minlen)))\n (mapv (fn [{:keys [s e ms]}]\n (let [hz (api/midi->hz (median ms))\n ;; the strength of an onset heard at most 0.15 s before the note (or just after its start)\n acc (some (fn [[mt x]] (when (and (number? x) (<= (- s (* 0.15 k)) mt (+ s (* 0.05 k)))) x)) marks)]\n (cond-> {:pts [[(r3 s) (r3 hz)] [(r3 e) (r3 hz)]]} acc (assoc :acc acc))))))))\n ;; the run segmented into :done: all of it (the voice stopped at t-end), or all but its last note (a long run\n ;; still sounding: that note goes on in the run)\n settle (fn [state t-end all?]\n (let [run (:run state) k (/ (:bpm state) 60) minlen (* k (/ (:min-ms state) 1000))]\n (if (or (empty? run) (<= t-end (ffirst run)))\n (dissoc state :run :silent)\n (let [ns (segment run (:marks state) t-end k minlen)\n [keep-ns rest-from] (if (or all? (< (count ns) 2)) [ns nil] [(pop ns) (ffirst (:pts (peek ns)))])\n state (-> state (update :done into keep-ns) (update :count + (count keep-ns)))]\n (if rest-from\n (let [before (last (filter (fn [[t]] (<= t rest-from)) run))]\n (assoc state :run (into [[rest-from (second before)]] (filter (fn [[t]] (> t rest-from)) run))\n :marks (filterv (fn [[t]] (>= t (- rest-from (* 0.15 k)))) (:marks state))))\n (-> state (dissoc :run :silent) (assoc :marks (filterv (fn [[t]] (> t t-end)) (:marks state)))))))))\n ;; a silence that lasted more than 30 ms: the run ends where it began\n settled (fn [state t]\n (if (and (:silent state) (> (- t (:silent state)) (* 0.03 (/ (:bpm state) 60))))\n (settle state (:silent state) true)\n state))\n hear-notes (fn [state hz t]\n (let [k (/ (:bpm state) 60)\n state (settled state t)]\n (cond\n (not (pos? hz)) (if (and (seq (:run state)) (nil? (:silent state))) (assoc state :silent t) state)\n :else (let [state (-> state (dissoc :silent) (update :run (fnil conj []) [t hz]))]\n ;; a run of 8 s and more: what is settled is written, the last note goes on\n (if (> (- t (ffirst (:run state))) (* 8 k)) (settle state t false) state)))))\n ;; ── one value of pitch~ (Hz, 0 when it hears none) ──\n hear (fn [state hz at]\n (let [t (beats state at)\n cur (:cur state)\n state (cond\n (= :notes (:mode state)) (hear-notes state hz t)\n (not (pos? hz)) (end-note state t)\n (nil? cur) (start-note state t hz)\n :else (update-in state [:cur :pts] conj [t hz]))\n [st edit later] (flush state at false)]\n ;; the voice stopped: a look again in a little while, to write its notes even if nothing else arrives\n [st edit (or later (when (and (:silent st) (not (:pending st))) [(+ at 0.05) 0 [:flush]]))]))]\n {;; ── about ──\n :class \"transcribe\"\n :license \"MPL-2.0\"\n :doc \"Writes what pitch~ hears into a freqt, as strokes. Hz on the left (from pitch~: 0 when it hears none), :rec to start from 0 (its first write clears the freqt) or :rec t (msg :rec 8) to start from beat t, keeping what is before it, or :ovr t (msg :ovr 8) to write over only what the take covers, keeping what comes after and :stop to end; a bang on the right (from onset~) begins a new note even on the same pitch, and a number there (onset~'s strength, 0…1) does too and gives the note that weight: its stroke is written heavier (as an event's :amp), and the quantizer reads it as the note's :vel, which helps find the beat and the bar (issue #106). In notes mode the contour of each voiced stretch is cut into notes where it settles on another pitch (a slide, a vibrato or an ornament that passes does not start notes; an onset on the right does), each a flat stroke at its median pitch (the freqt's tuning, or its score view, rounds it), written when the voice stops; in glide mode a stroke follows the pitch as it moves. Times are beats at bpm (60: one beat a second). Notes shorter than ms (80) are left out. It writes into the freqt at most every 2 seconds, each time one event of the history. Connect its outlet to the freqt's inlet.\"\n :usage \"transcribe [bpm] [notes|glide] [ms]\"\n :io [2 1]\n :size [124 30]\n :args [[:bpm 60] [:mode nil] [:min-ms 80]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args at]}]\n (let [state (or state {})]\n (cond\n ;; rec: a take from 0 (its first write clears the freqt); [:rec t] (msg :rec 8): from beat t on,\n ;; keeping what is before t (punch-in)\n ;; ovr t (msg :ovr 8): over what is there from beat t to where the take stops, keeping what comes after\n (or (#{:rec :ovr} value) (and (vector? value) (#{:rec :ovr} (first value))))\n (let [bpm (api/clamp (:bpm args) 10 400)\n ovr (or (= value :ovr) (and (vector? value) (= :ovr (first value))))\n from (or (when (vector? value) (let [x (second value)] (when (and (number? x) (>= x 0)) x)))\n (when ovr 0))]\n {:state (cond-> {:on true :t0 (- at (* (or from 0) (/ 60 bpm))) :bpm bpm :mode (mode-of args)\n :min-ms (api/clamp (:min-ms args) 0 2000) :done [] :count 0 :fresh true}\n from (assoc :from from)\n ovr (assoc :ovr true :cut-from from))\n :show {:on true :count 0}})\n (not (:on state)) {}\n (= value :stop)\n (let [t (beats state at)\n st (if (seq (:run state)) (settle state (or (:silent state) t) true) state)\n st (end-note st t)\n [st edit _] (flush st at true)]\n (cond-> {:state (assoc st :on false) :show {:on false :count (:count st)}}\n edit (assoc :out [[0 edit]])))\n (= value [:flush]) (result state (flush (settled (assoc state :pending false) (beats state at)) at false))\n ;; an onset (a bang, or its strength 0…1 from onset~'s second outlet): the note that sounds ends\n ;; here and a new one begins at its pitch, with that strength; in silence, the strength waits for\n ;; the next note to begin\n (and (= inlet 1) (= :notes (:mode state)))\n {:state (update state :marks (fnil conj []) [(beats state at) (when (number? value) (api/clamp value 0 1))])}\n (= inlet 1)\n (let [t (beats state at)\n state (if (number? value) (assoc state :acc [t (api/clamp value 0 1)]) state)]\n (if (:cur state)\n (let [hz (second (peek (get-in state [:cur :pts])))]\n (result state (flush (-> state (end-note t) (start-note t hz)) at false)))\n {:state state}))\n (and (= inlet 0) (number? value)) (result state (hear state value at))\n :else {})))\n\n ;; ── view ──\n :derive (fn [{:keys [shown]}]\n {:label (if (:on shown) (str \"● transcribing · \" (or (:count shown) 0)) (str \"transcribe · \" (or (:count shown) 0) \" notes\"))})\n :view [[:rect {:x 2 :y 2 :w 120 :h 26 :r 3 :stroke :grid :fill :bg}]\n [:text {:x 8 :y 19 :size 10 :fill :soft} :$label]]})\n"
|
|
50
62
|
}
|
|
51
63
|
]
|
|
52
64
|
};
|
package/dist/monitor.mjs
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
// Generated by packages/build.mjs from src/monitor.cljs
|
|
2
2
|
export default {
|
|
3
3
|
"class": "monitor",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.7.5",
|
|
5
5
|
"author": "bonuoq",
|
|
6
6
|
"license": "MPL-2.0",
|
|
7
7
|
"doc": "Draws the messages that arrive over the last seconds (6 by default), one lane per inlet: a lane for each word written after the seconds, which names it (monitor 6 rms centroid chroma has 3 inlets), or one lane without words. A number is drawn as a line in the range it has had in the window, a list of numbers as a strip with one row per element (chroma, mfcc), anything else (a bang, a word) as a mark. Time is that of the messages: the right edge is the last one. Sends nothing.",
|
package/dist/onset.mjs
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
// Generated by packages/build.mjs from src/onset.cljs
|
|
2
2
|
export default {
|
|
3
3
|
"class": "onset~",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.7.5",
|
|
5
5
|
"author": "bonuoq",
|
|
6
6
|
"license": "MPL-2.0",
|
|
7
|
-
"doc": "The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the
|
|
8
|
-
"source": ";; onset~ · the beginnings of sounds: a note, a stroke, a syllable\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n bins 256 ; the spectrum up to about 6 kHz at 48 kHz: where onsets are heard\n ;; how much the spectrum grew since the frame before, relative to what sounds now (0 steady … 1\n ;; from silence): what grows counts, what fades does not\n flux (fn [cur prev]\n (if-not prev\n 0\n (loop [i 1 up 0 all 0]\n (if (< i bins)\n (let [a (aget cur i) d (- a (aget prev i))]\n (recur (inc i) (if (pos? d) (+ up d) up) (+ all a)))\n (if (> all 1e-6) (/ up all) 0)))))\n median (fn [xs] (if (empty? xs) 0 (nth (sort xs) (quot (count xs) 2))))\n gate 0.005 ; rms under it (about -46 dB) is silence\n ;; the decision, pure: an onset where the flux rises above threshold × the median of the last\n ;; frames (and above 0.1), not sooner than ms after the last one\n detect (fn [state {:keys [flux rms]} at threshold ms]\n (let [hist (vec (take-last 16 (:hist state)))\n m (median hist)\n over (and (> rms gate) (> flux 0.1) (> flux (* threshold m)))\n fire (and over (not (:over state)) (or (nil? (:last state)) (>= (- at (:last state)) (/ ms 1000))))]\n [(cond-> (assoc state :hist (conj hist flux) :over over) fire (assoc :last at))\n (when fire (/ (js/Math.round (* 100 flux)) 100))]))]\n {;; ── about ──\n :class \"onset~\"\n :license \"MPL-2.0\"\n :doc \"The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the
|
|
7
|
+
"doc": "The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the second outlet; on the third, that growth for every frame (the onset strength envelope, for beat tracking: issue #106). threshold (3) is how far above the usual change of the last frames an onset must rise, while the energy does not fall (a sound cut at once is not an onset); ms (80) the least time between two. Every 512 samples (about 11 ms), on frames of 2048. An onset is reported with the frame it is first seen in: measured on sines, plucked tones and noise bursts that start at a known sample, 6 ms after it as a median (0.7 to 10.7 ms: one hop), and 11 ms for a new note over a held one. [:threshold x] and [:ms y] change them while it listens, and a [:segment …] from form~ makes it forget the usual change of the section before (so a quiet section after a loud one is not deaf). Into a counter, a metro that follows, a sample's rec, or the right inlet of transcribe (its strength, as the weight of the note).",
|
|
8
|
+
"source": ";; onset~ · the beginnings of sounds: a note, a stroke, a syllable\n;; Part of @kairotic/analysis. Written against the Kairotic API (api/…). The spectrum is Meyda's (MIT,\n;; meyda.js.org), loaded from jsDelivr the first time one is needed.\n(let [meyda-url \"npm:meyda@5.6.3/dist/web/meyda.min.js\"\n size 2048 ; the frames every analysis box of the package uses (see features~)\n bins 256 ; the spectrum up to about 6 kHz at 48 kHz: where onsets are heard\n ;; how much the spectrum grew since the frame before, relative to what sounds now (0 steady … 1\n ;; from silence): what grows counts, what fades does not\n flux (fn [cur prev]\n (if-not prev\n 0\n (loop [i 1 up 0 all 0]\n (if (< i bins)\n (let [a (aget cur i) d (- a (aget prev i))]\n (recur (inc i) (if (pos? d) (+ up d) up) (+ all a)))\n (if (> all 1e-6) (/ up all) 0)))))\n median (fn [xs] (if (empty? xs) 0 (nth (sort xs) (quot (count xs) 2))))\n gate 0.005 ; rms under it (about -46 dB) is silence\n ;; the decision, pure: an onset where the flux rises above threshold × the median of the last\n ;; frames (and above 0.1) while the energy does not fall (a sound cut at once spreads its spectrum like an\n ;; attack, but its energy goes down: not an onset), not sooner than ms after the last one\n detect (fn [state {:keys [flux rms]} at threshold ms]\n (let [hist (vec (take-last 16 (:hist state)))\n m (median hist)\n over (and (> rms gate) (> flux 0.1) (> flux (* threshold m)) (>= rms (or (:rms state) 0)))\n fire (and over (not (:over state)) (or (nil? (:last state)) (>= (- at (:last state)) (/ ms 1000))))]\n [(cond-> (assoc state :hist (conj hist flux) :over over :rms rms) fire (assoc :last at))\n (when fire (/ (js/Math.round (* 100 flux)) 100))]))]\n {;; ── about ──\n :class \"onset~\"\n :license \"MPL-2.0\"\n :doc \"The beginnings of sounds in the signal on its left — a note, a stroke, a syllable —: a bang on the left for each, and how strong it was (the growth of its spectrum, 0…1) on the second outlet; on the third, that growth for every frame (the onset strength envelope, for beat tracking: issue #106). threshold (3) is how far above the usual change of the last frames an onset must rise, while the energy does not fall (a sound cut at once is not an onset); ms (80) the least time between two. Every 512 samples (about 11 ms), on frames of 2048. An onset is reported with the frame it is first seen in: measured on sines, plucked tones and noise bursts that start at a known sample, 6 ms after it as a median (0.7 to 10.7 ms: one hop), and 11 ms for a new note over a held one. [:threshold x] and [:ms y] change them while it listens, and a [:segment …] from form~ makes it forget the usual change of the section before (so a quiet section after a loud one is not deaf). Into a counter, a metro that follows, a sample's rec, or the right inlet of transcribe (its strength, as the weight of the note).\"\n :usage \"onset~ [threshold] [ms]\"\n :io [1 3]\n :size [64 30]\n :args [[:threshold 3] [:ms 80]]\n\n ;; ── logic ──\n :receive (fn [{:keys [inlet value state args at]}]\n (cond\n (and (= inlet :audio) (:error value)) {:effects [[:print (:error value)]]}\n ;; adaptive: [:threshold x] and [:ms y] change them while it listens (from form~, a message box…);\n ;; a [:segment …] from form~ forgets the usual change of the section before\n (and (vector? value) (= 2 (count value)) (#{:threshold :ms} (first value)) (number? (second value)))\n {:state (assoc-in state [:set (first value)] (second value)) :show {:label (str (name (first value)) \" \" (api/nice (second value)))}}\n (and (vector? value) (= :segment (first value)))\n {:state (assoc state :hist [])}\n (and (= inlet :audio) (map? value))\n (let [[st strength] (detect state value at (api/clamp (or (get-in state [:set :threshold]) (:threshold args)) 1 50)\n (api/clamp (or (get-in state [:set :ms]) (:ms args)) 10 2000))]\n ;; the flux of every frame on the third outlet: the onset strength envelope (issue #106)\n (let [env [2 (/ (js/Math.round (* 1000 (:flux value))) 1000)]]\n (if strength\n {:state st :out [[1 strength] [0 :bang] env] :show {:at at :strength strength}}\n {:state st :out [env]})))\n :else {}))\n\n ;; ── view: a dot that lights on each onset ──\n :derive (fn [{:keys [shown]}]\n {:label (cond (:strength shown) (str (:strength shown)) (:label shown) (:label shown) :else \"onset\")})\n :view [[:rect {:x 2 :y 2 :w 60 :h 26 :r 3 :stroke :grid :fill :bg}]\n [:circle {:cx 15 :cy 15 :r 6 :stroke :accent :fill :accent}]\n [:text {:x 26 :y 19 :size 10 :fill :soft} :$label]]\n\n ;; ── audio ──\n :audio {:create (fn [ac b _]\n (let [in (.createGain ac)\n sr (.-sampleRate ac)\n node {:in in :M (atom nil) :prev (atom nil) :tap (atom nil) :alive (atom true) :inlets [[in 0]] :outlets []}\n on-block (fn [data at & _]\n ;; only while the box is here: a tap left behind would hand it silence\n (when-let [M (and @(:alive node) @(:M node))]\n (set! (.-bufferSize M) size)\n (set! (.-sampleRate M) sr)\n (let [r (.extract M #js [\"amplitudeSpectrum\" \"rms\"] (first data))\n cur (.-amplitudeSpectrum r)\n f (flux cur @(:prev node))]\n (reset! (:prev node) (.slice cur 0 bins))\n (api/report! (:id b) {:flux f :rms (.-rms r)} at))))]\n (-> (api/load-script meyda-url)\n (.then (fn [_] (reset! (:M node) (.-Meyda js/window))))\n (.catch (fn [e] (api/report! (:id b) {:error (str \"onset~: \" (.-message e))} nil))))\n (-> (api/tap ac {:mode :frames :size size :hop 512} on-block)\n (.then (fn [w] (if @(:alive node) (do (.connect in w) (reset! (:tap node) w)) (api/untap w))))\n (.catch (fn [e] (js/console.error \"onset~\" e))))\n node))\n :dispose (fn [node]\n (reset! (:alive node) false)\n (api/untap @(:tap node))\n (try (.disconnect (:in node)) (catch :default _ nil)))}})\n"
|
|
9
9
|
};
|