@micdrop/client 2.2.10 → 3.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -23
- package/dist/index.d.mts +581 -120
- package/dist/index.d.ts +581 -120
- package/dist/index.js +1100 -696
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +1079 -694
- package/dist/index.mjs.map +1 -1
- package/package.json +10 -6
package/README.md
CHANGED
|
@@ -1,47 +1,69 @@
|
|
|
1
1
|
# 🖐️🎤 Micdrop: Real-Time Voice Conversations with AI
|
|
2
2
|
|
|
3
|
-
[Micdrop website](https://micdrop.dev) | [Documentation](https://micdrop.dev/docs/client)
|
|
3
|
+
[Micdrop website](https://micdrop.dev) | [Documentation](https://micdrop.dev/docs/client)
|
|
4
4
|
|
|
5
|
-
Micdrop is a set of open source Typescript packages to build real-time voice conversations with AI agents. It handles all the complexities on the
|
|
5
|
+
Micdrop is a set of open source Typescript packages to build real-time voice conversations with AI agents. It handles all the complexities on the client and server side (microphone, speaker, VAD, network communication, etc) and provides ready-to-use implementations for various AI providers.
|
|
6
6
|
|
|
7
7
|
# @micdrop/client
|
|
8
8
|
|
|
9
|
-
The
|
|
9
|
+
The heart of a Micdrop call, with nothing platform specific in it: the protocol, the call state, the voice activity detection, the recording and the playback scheduling.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
**You install it through a platform package rather than on its own:**
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
- [`@micdrop/web`](../web) for a browser
|
|
14
|
+
- [`@micdrop/react-native`](../react-native) for iOS and Android
|
|
14
15
|
|
|
15
|
-
|
|
16
|
+
Both re-export everything here, so `import { Micdrop } from '@micdrop/web'` gives you the same objects this package defines.
|
|
16
17
|
|
|
17
|
-
|
|
18
|
-
- 🗣️ Voice activity detection (VAD)
|
|
19
|
-
- 🔊 Devices selection and testing
|
|
20
|
-
- 🔌 Full state and events for UI integration
|
|
21
|
-
- 🌐 WebSocket-based audio streaming
|
|
18
|
+
## What lives here
|
|
22
19
|
|
|
23
|
-
|
|
20
|
+
- `MicdropClient` and the `Micdrop` singleton: the WebSocket protocol, the call state, reconnection, interruption
|
|
21
|
+
- `Mic` and `Speaker`: one microphone and one speaker for the app, in front of a pluggable driver
|
|
22
|
+
- `MicRecorder`: the reserve of audio, the chunking at 16 kHz, the wiring to the VAD
|
|
23
|
+
- `VolumeVAD`, `SileroVAD`, `MultipleVAD`: voice activity detection, the same on every platform
|
|
24
|
+
- `Pcm16AudioStream`: gapless playback of the chunks as they arrive
|
|
25
|
+
- `VolumeMeter`: the level the VAD and the level meters read
|
|
24
26
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
+
## Writing a platform package
|
|
28
|
+
|
|
29
|
+
A platform provides two things, and gets everything above for free.
|
|
30
|
+
|
|
31
|
+
```ts
|
|
32
|
+
import { Mic, MicDriver, Speaker, SpeakerDriver } from '@micdrop/client'
|
|
33
|
+
|
|
34
|
+
class MyMic extends MicDriver {
|
|
35
|
+
// start() captures, then emits Frames with mono float samples
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
class MySpeaker extends SpeakerDriver {
|
|
39
|
+
// play() queues 16 kHz PCM16
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
Mic.setDriver(new MyMic())
|
|
43
|
+
Speaker.setDriver(new MySpeaker())
|
|
27
44
|
```
|
|
28
45
|
|
|
29
|
-
|
|
46
|
+
`Pcm16AudioStream` does the playback scheduling for you if the platform offers something like Web Audio: give it an `AudioSink`, which is the small slice of it that Micdrop needs.
|
|
30
47
|
|
|
31
|
-
|
|
48
|
+
For Silero, provide the inference and the state machine comes from here:
|
|
32
49
|
|
|
33
|
-
```
|
|
34
|
-
import {
|
|
50
|
+
```ts
|
|
51
|
+
import { setSileroModelLoader } from '@micdrop/client'
|
|
35
52
|
|
|
36
|
-
//
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
53
|
+
setSileroModelLoader(async () => myModel) // process(frame) => probability
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Tests
|
|
57
|
+
|
|
58
|
+
The whole package runs in Node, from the audio helpers up to a full call against a real `MicdropServer`.
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
pnpm --filter @micdrop/client test
|
|
40
62
|
```
|
|
41
63
|
|
|
42
64
|
## Documentation
|
|
43
65
|
|
|
44
|
-
Read full [documentation
|
|
66
|
+
Read the full [client documentation](https://micdrop.dev/docs/client) on the [website](https://micdrop.dev).
|
|
45
67
|
|
|
46
68
|
## License
|
|
47
69
|
|