studio-engine 0.120.0 → 0.121.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +41 -0
- data/README.md +2 -0
- data/app/controllers/studio/voice_takes_controller.rb +93 -0
- data/app/javascript/studio/controllers/microphone_chip_controller.js +21 -0
- data/app/javascript/studio/controllers/voice_recorder_controller.js +29 -0
- data/app/javascript/studio/level_meter.js +167 -0
- data/app/javascript/studio/microphone.js +250 -0
- data/app/javascript/studio/microphone_chip.js +370 -0
- data/app/javascript/studio/stimulus.js +6 -1
- data/app/javascript/studio/voice_profile.js +29 -114
- data/app/javascript/studio/voice_recorder.js +352 -0
- data/app/views/studio/voice/_microphone_chip.html.erb +45 -0
- data/app/views/studio/voice/_microphone_help.html.erb +63 -0
- data/app/views/studio/voice/_recorder.html.erb +34 -0
- data/app/views/studio/voice/_styles.html.erb +90 -0
- data/lib/studio/engine.rb +3 -1
- data/lib/studio/version.rb +1 -1
- data/lib/studio/voice_capture.rb +95 -0
- data/lib/studio/voice_transcription.rb +404 -0
- data/lib/studio.rb +30 -0
- metadata +15 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 161d3e00f32dba8b6c36d769228b39815a79b85a702d76b80f89be4c4365b876
|
|
4
|
+
data.tar.gz: a56e3fe6902119b4e5ecba61c77347a95893c04c3d8a5543833246b0a674c38f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4c9506dfec92c4174d85f3d03002d63d222b42c1be0866184a6551faa075471aec3440265e7623378d804e0b7f4dd6eb9e7978ab50f8403ce4edabb8be186df2
|
|
7
|
+
data.tar.gz: 9ec9cfde7907b0c5dd09a0c19cda518da4fc6341c32003bdda71debdd0ff8de4a13c4091936223d646713c98d7e87c7201550ff725f8f8aecedaf6e53563e962
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,47 @@ The format is [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). This pro
|
|
|
4
4
|
|
|
5
5
|
## Unreleased
|
|
6
6
|
|
|
7
|
+
## 0.121.0 — 2026-11-01
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- **Voice capture layer, ported from Voicebox.** `studio/voice/_microphone_chip`
|
|
12
|
+
names the microphone a page records from (the track's own label, with a live
|
|
13
|
+
level dot) and lists the others; a pick is remembered in `localStorage` under
|
|
14
|
+
`studio.microphone` (`storage_key:` to change it) and opened by exact device
|
|
15
|
+
id, a pick that is gone falls back to the system default, and with nothing
|
|
16
|
+
picked the system default is asked for by id rather than Chrome's per-site
|
|
17
|
+
preference. Its native `<dialog>` help shows Chrome's site-settings steps or a
|
|
18
|
+
generic line, and opens on a switch, a refusal, a second silent take in a row,
|
|
19
|
+
or the help link. `studio/voice/_recorder` (controller `voice-recorder`, tap or
|
|
20
|
+
hold to talk) refuses a take under -40 dBFS and hands a heard take to the page
|
|
21
|
+
as `studio:voice-recorder:take` with the blob; it uploads nothing. A recorder
|
|
22
|
+
that cannot start lets go of the microphone and the meter. New modules:
|
|
23
|
+
`studio/microphone`, `studio/level_meter` (the one meter),
|
|
24
|
+
`studio/voice_recorder`, `studio/microphone_chip`. docs/VOICE_CAPTURE.md is
|
|
25
|
+
the contract.
|
|
26
|
+
- **Voice transcription: a take becomes `{text, speaker, score}`.**
|
|
27
|
+
`Studio::VoiceTranscription.hear(audio:, mime_type:, user:, embedding:, model:)`
|
|
28
|
+
transcribes a take with a mock-first adapter ported from Voicebox (`openai`:
|
|
29
|
+
`gpt-4o-transcribe`, one retry on `whisper-1`, told the language; `mock`
|
|
30
|
+
elsewhere) and names the speaker with `Studio::VoiceMatch` against the user's
|
|
31
|
+
own `users.voice_embedding`, same embedder revision only, else "Someone new".
|
|
32
|
+
Opt-in `POST /voice/takes` (`config.voice_transcription = true`) wraps it for a
|
|
33
|
+
signed-in, CSRF-protected page: 10 MB and 120 s caps, an allow-list of audio
|
|
34
|
+
types, a refused no-speech transcript, and a 503 with a showable line when the
|
|
35
|
+
app has no `OPENAI_API_KEY`. The real adapter raises `LiveCallAttempted`
|
|
36
|
+
before any socket while `VOICE_NO_LIVE_CALLS=1` and in every Rails test
|
|
37
|
+
environment. Nothing is stored; `voice[audio]` and `voice[embedding]` join the
|
|
38
|
+
parameter filter. docs/VOICE_TRANSCRIPTION.md is the contract.
|
|
39
|
+
|
|
40
|
+
### Changed
|
|
41
|
+
|
|
42
|
+
- The voice row on `/profile` (`studio/voice_profile`) runs on the shared meter
|
|
43
|
+
and recording format (`studio/level_meter`, `studio/voice_recorder`) instead
|
|
44
|
+
of its own copy; it still re-exports `SILENCE_RMS`, `meterLevel`,
|
|
45
|
+
`takeWasHeard`, `pickMimeType` and friends. Under reduced motion its ring now
|
|
46
|
+
steps in quarters, as the recorder's does.
|
|
47
|
+
|
|
7
48
|
## 0.120.0 — 2026-10-30
|
|
8
49
|
|
|
9
50
|
### Added
|
data/README.md
CHANGED
|
@@ -46,6 +46,8 @@ resolved.
|
|
|
46
46
|
- **Site footer**: `<%= studio_site_footer %>` in the layout renders the public site's footer. It is on by default: with nothing configured it prints the app's name, logo and the legal links the app has routes for, and no address. `config.site_footer` declares the app's own facts (brand, social profiles, link columns, an address with a live map, legal links), every part optional; `config.site_footer_address` adds a location in one setting; `config.site_footer = false` turns the footer off. Leaflet is vendored and served by the engine. See [`docs/SITE_FOOTER.md`](docs/SITE_FOOTER.md).
|
|
47
47
|
- **Booking**: a simple scheduler on Google Calendar appointment schedules, with or without the footer. With `config.booking_url`, `studio_booking_frame` embeds Google's booking page, `studio_booking_link` opens it in a popup, and `config.draw_booking_routes = true` draws `/schedule` (or name the app's own page with `config.booking_path`). `config.booking_crop` crops the frame to the slot picker at rest, from numbers `bin/booking-crop-measure` reads off the app's own schedule. See [`docs/BOOKING.md`](docs/BOOKING.md).
|
|
48
48
|
- **Voice profile**: an opt-in row on `/profile` ("Set up your voice"): the person reads three phrases with a live level meter, their device turns each take into a speaker embedding (WeSpeaker via transformers.js), and only the averaged vector is saved on their user, never audio. `Studio::VoiceMatch.identify(embedding, among: User.all, model:)` answers who is speaking. Turn it on with `config.voice_profile = true` after installing the migration. See [`docs/VOICE_PROFILE.md`](docs/VOICE_PROFILE.md).
|
|
49
|
+
- **Voice capture**: the microphone chip (names the microphone a page records from, picks another, Chrome help dialog), one level meter, and a record button that refuses a silent take and hands a heard one to the page as a `studio:voice-recorder:take` event with the blob; it uploads nothing. Render `studio/voice/microphone_chip` and `studio/voice/recorder`. See [`docs/VOICE_CAPTURE.md`](docs/VOICE_CAPTURE.md).
|
|
50
|
+
- **Voice transcription**: `POST /voice/takes` (opt-in, `config.voice_transcription = true`) turns a signed-in user's take into `{text, speaker, score}`: a mock-first transcriber (OpenAI `gpt-4o-transcribe` in production, a mock elsewhere, a no-live-calls trap in every test environment) and `Studio::VoiceMatch` against the user's own saved voice. Nothing is kept. See [`docs/VOICE_TRANSCRIPTION.md`](docs/VOICE_TRANSCRIPTION.md).
|
|
49
51
|
- **Surveys**: define a survey in app code (`config/surveys/*.rb`, `Studio.define_survey`) with six question types — emoji scale, 1–5 rating, single and multiple choice, short and long text — and the engine serves it at `/surveys/:slug`: one question per screen, autosave, resume, keyboard shortcuts, and a no-JavaScript fallback. `config.on_survey_completed` and `config.survey_ref_resolver` let the app fire its own goals and attribute email arrivals. An admin panel at `/admin/surveys` breaks results down per question and exports CSV. Opt in with `config.draw_survey_routes = true` and `config.draw_admin_survey_routes = true`. See [`docs/SURVEYS.md`](docs/SURVEYS.md).
|
|
50
52
|
|
|
51
53
|
## Tailwind
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Studio
|
|
4
|
+
# POST /voice/takes: one recorded take in, {text, speaker, score} out
|
|
5
|
+
# (docs/VOICE_TRANSCRIPTION.md). Drawn by Studio.routes only when the app
|
|
6
|
+
# sets Studio.voice_transcription.
|
|
7
|
+
#
|
|
8
|
+
# The page posts the recorder's take (studio:voice-recorder:take) as
|
|
9
|
+
# multipart/form-data:
|
|
10
|
+
#
|
|
11
|
+
# voice[audio] the blob (required, at most VoiceTranscription::MAX_BYTES)
|
|
12
|
+
# voice[mime_type] its mimeType, e.g. "audio/webm;codecs=opus" (else the
|
|
13
|
+
# part's own Content-Type)
|
|
14
|
+
# voice[duration_ms] durationMs (required, 1..MAX_DURATION_MS)
|
|
15
|
+
# voice[embedding] the take's speaker vector, JSON (optional)
|
|
16
|
+
# voice[model] the embedder id that made it (optional)
|
|
17
|
+
#
|
|
18
|
+
# SIGNED-IN ONLY (the host's require_authentication: a JSON 401 otherwise),
|
|
19
|
+
# CSRF-protected (send the page's token as X-CSRF-Token), JSON, no-store.
|
|
20
|
+
# The speaker is matched only against the signed-in user's own saved voice.
|
|
21
|
+
#
|
|
22
|
+
# NOTHING IS KEPT: the upload is read once, handed to the transcriber, and its
|
|
23
|
+
# tempfile is deleted before the answer; no audio or vector is stored or
|
|
24
|
+
# logged (voice[audio] and voice[embedding] are in the engine's parameter
|
|
25
|
+
# filter).
|
|
26
|
+
class VoiceTakesController < ::ApplicationController
|
|
27
|
+
before_action :require_authentication
|
|
28
|
+
protect_from_forgery with: :exception
|
|
29
|
+
before_action :no_store
|
|
30
|
+
|
|
31
|
+
CANT_HEAR = "I couldn't hear any words in that. Check your microphone and try again."
|
|
32
|
+
UNAVAILABLE = "I couldn't transcribe that just now. Try again in a moment."
|
|
33
|
+
|
|
34
|
+
def create
|
|
35
|
+
voice = params[:voice]
|
|
36
|
+
audio = voice.respond_to?(:[]) ? voice[:audio] : nil
|
|
37
|
+
return refuse("Record something first.") unless audio.respond_to?(:read) && audio.respond_to?(:size)
|
|
38
|
+
|
|
39
|
+
begin
|
|
40
|
+
return refuse("That recording is too long to send.", 413) if audio.size > VoiceTranscription::MAX_BYTES
|
|
41
|
+
return refuse("That recording was empty.") if audio.size.zero?
|
|
42
|
+
|
|
43
|
+
duration = voice[:duration_ms].to_s
|
|
44
|
+
unless duration.match?(/\A\d{1,9}\z/) && duration.to_i.between?(1, VoiceTranscription::MAX_DURATION_MS)
|
|
45
|
+
return refuse("That recording is too long, or has no length.")
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
mime = VoiceTranscription.mime_type(voice[:mime_type].presence ||
|
|
49
|
+
(audio.content_type if audio.respond_to?(:content_type)))
|
|
50
|
+
return refuse("That kind of recording isn't supported.", 415) if mime.nil?
|
|
51
|
+
|
|
52
|
+
heard = VoiceTranscription.hear(audio: audio, mime_type: mime, user: current_user,
|
|
53
|
+
embedding: voice[:embedding].presence, model: voice[:model].presence)
|
|
54
|
+
reason = VoiceTranscription.no_speech_reason(heard.text)
|
|
55
|
+
return refuse(CANT_HEAR, :unprocessable_entity, reason: "no_speech") if reason
|
|
56
|
+
|
|
57
|
+
render json: heard.to_h.merge("duration_ms" => duration.to_i), status: :ok
|
|
58
|
+
rescue VoiceTranscription::NotConfigured => e
|
|
59
|
+
capture(e)
|
|
60
|
+
refuse(VoiceTranscription::NOT_CONFIGURED_MESSAGE, :service_unavailable, reason: "not_configured")
|
|
61
|
+
rescue VoiceTranscription::HttpError => e
|
|
62
|
+
capture(e)
|
|
63
|
+
refuse(UNAVAILABLE, :service_unavailable, reason: "transcriber_failed", retry: true)
|
|
64
|
+
ensure
|
|
65
|
+
discard(audio)
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
private
|
|
70
|
+
|
|
71
|
+
def refuse(message, status = :unprocessable_entity, **extra)
|
|
72
|
+
render json: { error: message, **extra }, status: status
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def no_store
|
|
76
|
+
response.headers["Cache-Control"] = "no-store"
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# The upload's tempfile goes now, not at the next GC.
|
|
80
|
+
def discard(audio)
|
|
81
|
+
file = audio.respond_to?(:tempfile) ? audio.tempfile : nil
|
|
82
|
+
file.close! if file.respond_to?(:close!)
|
|
83
|
+
rescue StandardError
|
|
84
|
+
nil
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def capture(error)
|
|
88
|
+
::ErrorLog.capture!(error) if defined?(::ErrorLog) && ::ErrorLog.respond_to?(:capture!)
|
|
89
|
+
rescue StandardError
|
|
90
|
+
nil
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
// microphone-chip: the microphone chip (studio/voice/_microphone_chip), on the
|
|
2
|
+
// engine's Stimulus application (studio/stimulus registers it lazily, the
|
|
3
|
+
// first time a page renders a chip):
|
|
4
|
+
//
|
|
5
|
+
// <div data-studio-controller="microphone-chip" data-microphone-copy='{...}'>
|
|
6
|
+
//
|
|
7
|
+
// The listing, the pick, the live dot and the help dialog are
|
|
8
|
+
// studio/microphone_chip; which device opens is studio/microphone.
|
|
9
|
+
import { Controller } from "@hotwired/stimulus"
|
|
10
|
+
import { mountMicrophoneChip } from "studio/microphone_chip"
|
|
11
|
+
|
|
12
|
+
export default class extends Controller {
|
|
13
|
+
connect() {
|
|
14
|
+
this.chip = mountMicrophoneChip(this.element)
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
disconnect() {
|
|
18
|
+
if (this.chip) this.chip.destroy()
|
|
19
|
+
this.chip = null
|
|
20
|
+
}
|
|
21
|
+
}
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
// voice-recorder: one record button (studio/voice/_recorder, or a page's own
|
|
2
|
+
// markup), on the engine's Stimulus application (studio/stimulus registers it
|
|
3
|
+
// lazily, the first time a page renders one):
|
|
4
|
+
//
|
|
5
|
+
// <div data-studio-controller="voice-recorder" data-voice-recorder-mode="toggle">
|
|
6
|
+
//
|
|
7
|
+
// The take, the meter and the events are studio/voice_recorder. A page reads
|
|
8
|
+
// the finished audio from the studio:voice-recorder:take event; this
|
|
9
|
+
// controller uploads nothing. Pages that drive it from their own controls call
|
|
10
|
+
// the actions: data-studio-action="voice-recorder#start" (or #stop, #toggle).
|
|
11
|
+
import { Controller } from "@hotwired/stimulus"
|
|
12
|
+
import { mountVoiceRecorder } from "studio/voice_recorder"
|
|
13
|
+
|
|
14
|
+
export default class extends Controller {
|
|
15
|
+
connect() {
|
|
16
|
+
this.recorder = mountVoiceRecorder(this.element)
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
disconnect() {
|
|
20
|
+
if (this.recorder) this.recorder.destroy()
|
|
21
|
+
this.recorder = null
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
start() { if (this.recorder) this.recorder.start() }
|
|
25
|
+
|
|
26
|
+
stop() { if (this.recorder) this.recorder.stop() }
|
|
27
|
+
|
|
28
|
+
toggle() { if (this.recorder) this.recorder.toggle() }
|
|
29
|
+
}
|
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
// studio/level_meter: the ONE level meter every engine recorder uses (the voice
|
|
2
|
+
// recorder, studio/voice_recorder, and the voice row, studio/voice_profile). It
|
|
3
|
+
// reads a live microphone stream through a Web Audio AnalyserNode, keeps the
|
|
4
|
+
// take's loudest frame, and paints a 0..1 level the page draws as a ring or a
|
|
5
|
+
// dot. Voicebox carried two copies of this (its recorder and its voice setup);
|
|
6
|
+
// the engine carries one (docs/VOICE_CAPTURE.md).
|
|
7
|
+
//
|
|
8
|
+
// const meter = createLevelMeter(stream, { onPaint: (level) => ..., onSample: (rms) => ... })
|
|
9
|
+
// ... meter.sample() ... // the recorder also samples on its own timer
|
|
10
|
+
// const heard = meter.heard() // before close: did the take clear SILENCE_RMS?
|
|
11
|
+
// meter.close() // stops frames and the timer, closes the context, paints 0
|
|
12
|
+
//
|
|
13
|
+
// WHY A TAKE IS JUDGED. A muted or wrong microphone records perfect silence,
|
|
14
|
+
// and a speech model handed silence makes a word up. A take whose loudest frame
|
|
15
|
+
// stayed under SILENCE_RMS is refused by the recorder before it goes anywhere.
|
|
16
|
+
//
|
|
17
|
+
// It imports nothing, so test/javascript/level_meter.test.mjs loads it as a
|
|
18
|
+
// data: module.
|
|
19
|
+
|
|
20
|
+
// The loudest frame (RMS of the samples, 0..1) a take must reach to count as
|
|
21
|
+
// heard: 0.01 is -40 dBFS. A muted or absent microphone sits at 0, a quiet room
|
|
22
|
+
// on a laptop microphone near 0.002-0.006, and speech at arm's length peaks
|
|
23
|
+
// well above 0.05, so a real voice clears it many times over. Measured in
|
|
24
|
+
// Voicebox.
|
|
25
|
+
export const SILENCE_RMS = 0.01
|
|
26
|
+
|
|
27
|
+
// How often the meter samples on a timer as well as on animation frames: a
|
|
28
|
+
// background tab stops frames, and a take recorded there is still measured.
|
|
29
|
+
export const SAMPLE_INTERVAL_MS = 100
|
|
30
|
+
|
|
31
|
+
// Reduced motion: the level steps four times a second, in quarters, instead
|
|
32
|
+
// of gliding.
|
|
33
|
+
export const REDUCED_PAINT_MS = 250
|
|
34
|
+
|
|
35
|
+
export function rootMeanSquare(samples) {
|
|
36
|
+
if (!samples || samples.length === 0) return 0
|
|
37
|
+
let sum = 0
|
|
38
|
+
for (let i = 0; i < samples.length; i++) sum += samples[i] * samples[i]
|
|
39
|
+
return Math.sqrt(sum / samples.length)
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
// The loudest window's RMS, 0..1 (800 samples is 50 ms at 16 kHz): the same
|
|
43
|
+
// silence rule applied to decoded samples rather than a live stream.
|
|
44
|
+
export function peakWindowRms(samples, windowSize = 800) {
|
|
45
|
+
let peak = 0
|
|
46
|
+
if (!samples) return peak
|
|
47
|
+
for (let start = 0; start < samples.length; start += windowSize) {
|
|
48
|
+
const end = Math.min(samples.length, start + windowSize)
|
|
49
|
+
let sum = 0
|
|
50
|
+
for (let i = start; i < end; i++) sum += samples[i] * samples[i]
|
|
51
|
+
const rms = Math.sqrt(sum / Math.max(1, end - start))
|
|
52
|
+
if (rms > peak) peak = rms
|
|
53
|
+
}
|
|
54
|
+
return peak
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
// The meter's reading, 0..1, for an RMS level: speech lives between about
|
|
58
|
+
// 0.01 and 0.3, so the scale is logarithmic from -40 dBFS to -10 dBFS.
|
|
59
|
+
export function meterLevel(rms) {
|
|
60
|
+
if (!(rms > 0)) return 0
|
|
61
|
+
const db = 20 * Math.log10(rms)
|
|
62
|
+
return Math.min(1, Math.max(0, (db + 40) / 30))
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
// Heard, unless the meter measured the take and it stayed under SILENCE_RMS. A
|
|
66
|
+
// take the meter never measured (no Web Audio, a context that never ran) is let
|
|
67
|
+
// through, and whatever receives it (a server, an embedder) decides.
|
|
68
|
+
export function takeWasHeard(peakRms, framesMeasured) {
|
|
69
|
+
if (!framesMeasured) return true
|
|
70
|
+
return peakRms >= SILENCE_RMS
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
// The level as painted: in quarters under reduced motion.
|
|
74
|
+
export function steppedLevel(level, reduced) {
|
|
75
|
+
return reduced ? Math.round(level * 4) / 4 : level
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
export function prefersReducedMotion(win = globalThis) {
|
|
79
|
+
try { return !!(win.matchMedia && win.matchMedia("(prefers-reduced-motion: reduce)").matches) } catch (_) { return false }
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
// Starts measuring a live stream. Opened after the microphone prompt resolves,
|
|
83
|
+
// so a strict autoplay rule (iOS) can leave the context suspended: those frames
|
|
84
|
+
// read zeros and prove nothing, so they are not counted, and an unmeasured take
|
|
85
|
+
// is let through (takeWasHeard). Without Web Audio the meter measures nothing
|
|
86
|
+
// and paints nothing beyond 0; the take still records.
|
|
87
|
+
//
|
|
88
|
+
// onSample(rms) runs on every counted frame (the recorder tells the chip);
|
|
89
|
+
// onPaint(level) runs with the 0..1 level to draw.
|
|
90
|
+
export function createLevelMeter(stream, { win = globalThis, onSample = () => {}, onPaint = () => {}, intervalMs = SAMPLE_INTERVAL_MS } = {}) {
|
|
91
|
+
let audioContext = null
|
|
92
|
+
let analyser = null
|
|
93
|
+
let buffer = null
|
|
94
|
+
let frame = null
|
|
95
|
+
let timer = null
|
|
96
|
+
let peak = 0
|
|
97
|
+
let frames = 0
|
|
98
|
+
let lastPaint = 0
|
|
99
|
+
let closed = false
|
|
100
|
+
const reduced = prefersReducedMotion(win)
|
|
101
|
+
const now = () => (win.performance && win.performance.now ? win.performance.now() : Date.now())
|
|
102
|
+
|
|
103
|
+
function paint(level) {
|
|
104
|
+
try { onPaint(steppedLevel(level, reduced)) } catch (_) { /* a painter's error never stops the take */ }
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
function sample() {
|
|
108
|
+
if (closed || !analyser || !audioContext || audioContext.state !== "running") return null
|
|
109
|
+
analyser.getFloatTimeDomainData(buffer)
|
|
110
|
+
const rms = rootMeanSquare(buffer)
|
|
111
|
+
frames += 1
|
|
112
|
+
if (rms > peak) peak = rms
|
|
113
|
+
try { onSample(rms) } catch (_) { /* as above */ }
|
|
114
|
+
const t = now()
|
|
115
|
+
if (!reduced || t - lastPaint >= REDUCED_PAINT_MS) {
|
|
116
|
+
lastPaint = t
|
|
117
|
+
paint(meterLevel(rms))
|
|
118
|
+
}
|
|
119
|
+
return rms
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
function close() {
|
|
123
|
+
if (frame != null && win.cancelAnimationFrame) win.cancelAnimationFrame(frame)
|
|
124
|
+
if (timer != null && win.clearInterval) win.clearInterval(timer)
|
|
125
|
+
frame = null
|
|
126
|
+
timer = null
|
|
127
|
+
analyser = null
|
|
128
|
+
if (audioContext && audioContext.state !== "closed" && audioContext.close) {
|
|
129
|
+
try { audioContext.close().catch(() => {}) } catch (_) { /* already gone */ }
|
|
130
|
+
}
|
|
131
|
+
audioContext = null
|
|
132
|
+
closed = true
|
|
133
|
+
paint(0)
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
paint(0)
|
|
137
|
+
const AudioContextClass = win.AudioContext || win.webkitAudioContext
|
|
138
|
+
if (AudioContextClass && stream) {
|
|
139
|
+
try {
|
|
140
|
+
audioContext = new AudioContextClass()
|
|
141
|
+
const source = audioContext.createMediaStreamSource(stream)
|
|
142
|
+
analyser = audioContext.createAnalyser()
|
|
143
|
+
analyser.fftSize = 1024
|
|
144
|
+
buffer = new Float32Array(analyser.fftSize)
|
|
145
|
+
source.connect(analyser)
|
|
146
|
+
if (audioContext.state === "suspended" && audioContext.resume) audioContext.resume().catch(() => {})
|
|
147
|
+
if (win.requestAnimationFrame) {
|
|
148
|
+
const loop = () => {
|
|
149
|
+
sample()
|
|
150
|
+
if (!closed) frame = win.requestAnimationFrame(loop)
|
|
151
|
+
}
|
|
152
|
+
frame = win.requestAnimationFrame(loop)
|
|
153
|
+
}
|
|
154
|
+
if (win.setInterval) timer = win.setInterval(sample, intervalMs)
|
|
155
|
+
} catch (_) {
|
|
156
|
+
close() // measured nothing: heard() lets the take through
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
return {
|
|
161
|
+
sample,
|
|
162
|
+
close,
|
|
163
|
+
heard: () => takeWasHeard(peak, frames),
|
|
164
|
+
get peak() { return peak },
|
|
165
|
+
get frames() { return frames }
|
|
166
|
+
}
|
|
167
|
+
}
|
|
@@ -0,0 +1,250 @@
|
|
|
1
|
+
// studio/microphone: which microphone a page records from. The one the person
|
|
2
|
+
// picked on the microphone chip (studio/microphone_chip), remembered on this
|
|
3
|
+
// device, or the system default when they never picked one or the picked one
|
|
4
|
+
// is gone. Ported from Voicebox's microphone.js (docs/VOICE_CAPTURE.md).
|
|
5
|
+
//
|
|
6
|
+
// Every recorder opens the microphone through openMicrophone, so the recorder
|
|
7
|
+
// and the chip never disagree about the device. The recorder then tells the
|
|
8
|
+
// chip what it opened, how loud the take is, and when it let go, with EVENTS on
|
|
9
|
+
// window; the chip only listens.
|
|
10
|
+
//
|
|
11
|
+
// Why it exists: Chrome's site setting can pick a microphone that is switched
|
|
12
|
+
// off (a wireless headset) instead of the system's own, and every take then
|
|
13
|
+
// records silence with nothing on the page saying which microphone was used.
|
|
14
|
+
//
|
|
15
|
+
// It imports nothing, so test/javascript/microphone.test.mjs loads it as a
|
|
16
|
+
// data: module.
|
|
17
|
+
|
|
18
|
+
// Where the pick is remembered (localStorage). One key for every engine page
|
|
19
|
+
// on an origin, so the chip on the hub's chat and on /profile agree. A host
|
|
20
|
+
// that wants its own key calls configureMicrophone({ storageKey }) or renders
|
|
21
|
+
// the chip with storage_key: (studio/voice/_microphone_chip).
|
|
22
|
+
export const STORAGE_KEY = "studio.microphone"
|
|
23
|
+
|
|
24
|
+
let currentKey = STORAGE_KEY
|
|
25
|
+
|
|
26
|
+
export function configureMicrophone({ storageKey } = {}) {
|
|
27
|
+
currentKey = storageKey ? String(storageKey) : STORAGE_KEY
|
|
28
|
+
return currentKey
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
export function storageKey() { return currentKey }
|
|
32
|
+
|
|
33
|
+
export const EVENTS = {
|
|
34
|
+
opened: "studio:microphone:opened", // detail: { label, deviceId, fellBack }
|
|
35
|
+
level: "studio:microphone:level", // detail: { rms }
|
|
36
|
+
closed: "studio:microphone:closed",
|
|
37
|
+
denied: "studio:microphone:denied",
|
|
38
|
+
chosen: "studio:microphone:chosen", // detail: { deviceId }
|
|
39
|
+
silent: "studio:microphone:silent" // detail: { label }: a take came back silent
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
// Which help to show: Chrome's (and Chromium Edge's, Opera's) site-settings
|
|
43
|
+
// steps, or one generic line for Safari and Firefox. Only desktop and Android
|
|
44
|
+
// Chromium carry "Chrome/" (or Edge's "Edg/"); Chrome and Edge on iOS say
|
|
45
|
+
// "CriOS/" and "EdgiOS/", WebKit underneath with no site-settings icon by the
|
|
46
|
+
// address, so they get the generic line too.
|
|
47
|
+
export function browserFamily(userAgent) {
|
|
48
|
+
return /(Chrome|Chromium|Edg)\//.test(String(userAgent || "")) ? "chrome" : "other"
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
// Chrome's two stand-ins for "whatever the system picks". They repeat a real
|
|
52
|
+
// device under another id, so the picker leaves them out when real ones exist.
|
|
53
|
+
const PSEUDO_IDS = ["default", "communications"]
|
|
54
|
+
|
|
55
|
+
// "Arctis Nova Pro Wireless (1038:12e0)" -> "Arctis Nova Pro Wireless", and
|
|
56
|
+
// Chrome's "Default - MacBook Pro Microphone (Built-in)" ->
|
|
57
|
+
// "MacBook Pro Microphone". The USB vendor:product pair and the "(Built-in)"
|
|
58
|
+
// tag mean nothing to a person.
|
|
59
|
+
export function trimLabel(label) {
|
|
60
|
+
return String(label || "")
|
|
61
|
+
.replace(/^(Default|Communications)\s+-\s+/i, "")
|
|
62
|
+
.replace(/\s*\([0-9a-f]{4}:[0-9a-f]{4}\)/gi, "")
|
|
63
|
+
.replace(/\s*\(Built-in\)/gi, "")
|
|
64
|
+
.replace(/\s+/g, " ")
|
|
65
|
+
.trim()
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
export function audioInputs(devices) {
|
|
69
|
+
return Array.from(devices || []).filter((device) => device && device.kind === "audioinput")
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// Before the person allows the microphone the browser lists inputs with empty
|
|
73
|
+
// labels (and often empty ids): there is nothing to show yet.
|
|
74
|
+
export function permissionGranted(inputs) {
|
|
75
|
+
return (inputs || []).some((device) => !!device.label)
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
// The devices worth offering in the picker: real ones, named, once each.
|
|
79
|
+
export function pickerChoices(inputs) {
|
|
80
|
+
const named = (inputs || []).filter((device) => device.label && device.deviceId)
|
|
81
|
+
const real = named.filter((device) => !PSEUDO_IDS.includes(device.deviceId))
|
|
82
|
+
const seen = new Set()
|
|
83
|
+
return (real.length ? real : named).flatMap((device) => {
|
|
84
|
+
if (seen.has(device.deviceId)) return []
|
|
85
|
+
seen.add(device.deviceId)
|
|
86
|
+
return [{ deviceId: device.deviceId, label: trimLabel(device.label) }]
|
|
87
|
+
})
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
// The remembered device, if it is still plugged in; null means "let the
|
|
91
|
+
// browser pick". Before permission the ids are blank and nothing can be
|
|
92
|
+
// checked, so the remembered one is tried and openMicrophone falls back if the
|
|
93
|
+
// browser says it is gone.
|
|
94
|
+
export function chooseDeviceId(inputs, rememberedId) {
|
|
95
|
+
if (!rememberedId) return null
|
|
96
|
+
if (inputs.some((device) => device.deviceId === rememberedId)) return rememberedId
|
|
97
|
+
if (inputs.every((device) => !device.deviceId)) return rememberedId
|
|
98
|
+
return null
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
// The system default, as a device to ask for by id. Chrome lists it as the
|
|
102
|
+
// "default" entry, and a bare { audio: true } does NOT open it: Chrome opens
|
|
103
|
+
// its per-site preference instead, which can be a headset switched off in a
|
|
104
|
+
// drawer. So with nothing picked the page asks for the system default by id,
|
|
105
|
+
// and the chip, which names the system default, names what is opened. The
|
|
106
|
+
// real twin's id when there is one (the track then reports that id), else
|
|
107
|
+
// "default" itself; null where the browser lists no "default" (Safari,
|
|
108
|
+
// Firefox), and there the browser's own choice stands and the opened track
|
|
109
|
+
// corrects the chip.
|
|
110
|
+
export function systemDefaultId(inputs) {
|
|
111
|
+
const entry = inputs.find((device) => device.deviceId === "default" && device.label)
|
|
112
|
+
if (!entry) return null
|
|
113
|
+
const name = trimLabel(entry.label)
|
|
114
|
+
const twin = inputs.find((device) => device.deviceId && !PSEUDO_IDS.includes(device.deviceId) &&
|
|
115
|
+
trimLabel(device.label) === name)
|
|
116
|
+
return twin ? twin.deviceId : "default"
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
export function audioConstraints(deviceId) {
|
|
120
|
+
return deviceId ? { audio: { deviceId: { exact: deviceId } } } : { audio: true }
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
// The errors getUserMedia gives when an exact device is not there.
|
|
124
|
+
export function isMissingDevice(error) {
|
|
125
|
+
return !!error && ["OverconstrainedError", "NotFoundError"].includes(error.name)
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
// What the chip names before a take has opened anything: the picked device,
|
|
129
|
+
// else the system default's name, else the first input. After a take, the
|
|
130
|
+
// track's own label wins (activeLabel), because that is the device that was
|
|
131
|
+
// actually recorded from.
|
|
132
|
+
export function displayedLabel(inputs, rememberedId, activeLabel) {
|
|
133
|
+
if (activeLabel) return trimLabel(activeLabel)
|
|
134
|
+
if (!permissionGranted(inputs)) return null
|
|
135
|
+
const remembered = rememberedId && inputs.find((device) => device.deviceId === rememberedId && device.label)
|
|
136
|
+
if (remembered) return trimLabel(remembered.label)
|
|
137
|
+
const fallback = inputs.find((device) => device.deviceId === "default" && device.label) ||
|
|
138
|
+
inputs.find((device) => device.label)
|
|
139
|
+
return fallback ? trimLabel(fallback.label) : null
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
// ---- Remembering the choice (localStorage can throw: private mode, blocked
|
|
143
|
+
// site data), so every touch is guarded and a failure means "no choice". ----
|
|
144
|
+
|
|
145
|
+
function storageOf(storage) {
|
|
146
|
+
if (storage !== undefined) return storage
|
|
147
|
+
try { return globalThis.localStorage || null } catch (_) { return null }
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
export function rememberedDeviceId(storage, key = currentKey) {
|
|
151
|
+
try { return storageOf(storage)?.getItem(key) || null } catch (_) { return null }
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
export function rememberDevice(deviceId, storage, key = currentKey) {
|
|
155
|
+
try {
|
|
156
|
+
const store = storageOf(storage)
|
|
157
|
+
if (!store) return
|
|
158
|
+
if (deviceId) store.setItem(key, deviceId)
|
|
159
|
+
else store.removeItem(key)
|
|
160
|
+
} catch (_) { /* nothing remembered: the browser picks next time */ }
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
export function forgetDevice(storage, key = currentKey) { rememberDevice(null, storage, key) }
|
|
164
|
+
|
|
165
|
+
export function trackOf(stream) {
|
|
166
|
+
return (stream && stream.getAudioTracks && stream.getAudioTracks()[0]) || null
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
// Opens the microphone the person picked, or the system default. A picked
|
|
170
|
+
// device that has gone away is forgotten and the default opened instead
|
|
171
|
+
// (fellBack), so a headset left in a drawer never blocks recording. Other
|
|
172
|
+
// errors (NotAllowedError above all) are the caller's to word.
|
|
173
|
+
//
|
|
174
|
+
// With nothing picked it asks for the system default by id (systemDefaultId),
|
|
175
|
+
// so Chrome's per-site preference never silently wins. Before permission the
|
|
176
|
+
// ids are blank and only a bare request can ask; once that grants permission,
|
|
177
|
+
// a stream that landed on another device is closed and the system default
|
|
178
|
+
// opened in its place, so even the first take records from what the chip will
|
|
179
|
+
// name.
|
|
180
|
+
export async function openMicrophone(mediaDevices = globalThis.navigator && globalThis.navigator.mediaDevices, storage) {
|
|
181
|
+
const remembered = rememberedDeviceId(storage)
|
|
182
|
+
const inputs = await listInputs(mediaDevices)
|
|
183
|
+
const picked = chooseDeviceId(inputs, remembered)
|
|
184
|
+
let fellBack = !!remembered && !picked
|
|
185
|
+
if (fellBack) forgetDevice(storage)
|
|
186
|
+
let deviceId = picked || systemDefaultId(inputs)
|
|
187
|
+
|
|
188
|
+
let stream
|
|
189
|
+
try {
|
|
190
|
+
stream = await mediaDevices.getUserMedia(audioConstraints(deviceId))
|
|
191
|
+
} catch (error) {
|
|
192
|
+
if (!deviceId || !isMissingDevice(error)) throw error
|
|
193
|
+
if (picked) {
|
|
194
|
+
forgetDevice(storage)
|
|
195
|
+
fellBack = true
|
|
196
|
+
}
|
|
197
|
+
deviceId = null
|
|
198
|
+
stream = await mediaDevices.getUserMedia(audioConstraints(null))
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
if (!deviceId && !permissionGranted(inputs)) stream = await reopenSystemDefault(mediaDevices, stream)
|
|
202
|
+
|
|
203
|
+
const track = trackOf(stream)
|
|
204
|
+
const settings = settingsOf(track)
|
|
205
|
+
const opened = { label: trimLabel(track && track.label), deviceId: settings.deviceId || deviceId || null, fellBack }
|
|
206
|
+
announce(EVENTS.opened, opened)
|
|
207
|
+
return { stream, ...opened }
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
async function listInputs(mediaDevices) {
|
|
211
|
+
if (!mediaDevices || !mediaDevices.enumerateDevices) return []
|
|
212
|
+
try { return audioInputs(await mediaDevices.enumerateDevices()) } catch (_) { return [] }
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
function settingsOf(track) {
|
|
216
|
+
try { return (track && track.getSettings && track.getSettings()) || {} } catch (_) { return {} }
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
// After the permission prompt: is the bare request's stream the system
|
|
220
|
+
// default? If it landed elsewhere (Chrome's site preference), let it go and
|
|
221
|
+
// open the system default by id. Any doubt keeps the stream it has.
|
|
222
|
+
export async function reopenSystemDefault(mediaDevices, stream) {
|
|
223
|
+
const inputs = await listInputs(mediaDevices)
|
|
224
|
+
const wanted = systemDefaultId(inputs)
|
|
225
|
+
const track = trackOf(stream)
|
|
226
|
+
if (!wanted || !track) return stream
|
|
227
|
+
const got = settingsOf(track).deviceId
|
|
228
|
+
const wantedLabel = trimLabel((inputs.find((device) => device.deviceId === wanted) || {}).label)
|
|
229
|
+
if (got === wanted || got === "default" || (wantedLabel && trimLabel(track.label) === wantedLabel)) return stream
|
|
230
|
+
try {
|
|
231
|
+
const replacement = await mediaDevices.getUserMedia(audioConstraints(wanted))
|
|
232
|
+
stream.getTracks().forEach((t) => t.stop())
|
|
233
|
+
return replacement
|
|
234
|
+
} catch (_) {
|
|
235
|
+
return stream
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
// Tells the chip (and anything else listening on window).
|
|
240
|
+
export function announce(name, detail = {}, win = globalThis) {
|
|
241
|
+
try { win.dispatchEvent(new CustomEvent(name, { detail })) } catch (_) { /* no listeners to tell */ }
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
// The silent-take line that names the microphone, or null when the track had
|
|
245
|
+
// no name to give (then the page's general line stands).
|
|
246
|
+
export function silentMessage(template, label) {
|
|
247
|
+
const name = trimLabel(label)
|
|
248
|
+
if (!template || !name) return null
|
|
249
|
+
return template.replace("%{microphone}", name)
|
|
250
|
+
}
|