claude-threads 1.33.1 → 1.34.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +9 -0
- package/dist/index.js +752 -467
- package/dist/mcp/mcp-server.js +110 -88
- package/docs/CONFIGURATION.md +33 -0
- package/docs/audio-transcription-spec.md +151 -0
- package/package.json +1 -1
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
# Audio transcription: voice notes → text before Claude sees them
|
|
2
|
+
|
|
3
|
+
Upstream discussion: anneschuth/claude-threads#519.
|
|
4
|
+
|
|
5
|
+
## What it does
|
|
6
|
+
|
|
7
|
+
When an inbound attachment has an `audio/*` MIME type and `transcription` is
|
|
8
|
+
configured, the bot transcribes it and puts the text into the message Claude
|
|
9
|
+
receives. Today the attachment is only saved to disk and listed by path — a
|
|
10
|
+
`.webm` Claude cannot hear. With this feature a Slack audio clip (or any
|
|
11
|
+
uploaded audio file) becomes a spoken message.
|
|
12
|
+
|
|
13
|
+
Behaviour, in order:
|
|
14
|
+
|
|
15
|
+
1. `saveFilesToUploadDir` runs unchanged — the file lands on disk and its
|
|
16
|
+
path is still listed in the `[Attached files from chat …]` header.
|
|
17
|
+
2. Every saved file whose `mimeType` starts with `audio/` — or whose
|
|
18
|
+
extension is a known audio one (`m4a mp3 ogg opus wav aac flac webm`) when
|
|
19
|
+
the platform reported only a generic type — is sent to the configured
|
|
20
|
+
transcriber. `webm` is on the list because Slack's own clips are
|
|
21
|
+
`voice.webm`; a WebM *video* that reaches this fallback gets its soundtrack
|
|
22
|
+
transcribed, which costs a fraction of a cent and is harmless. Properly
|
|
23
|
+
typed `video/*` is still excluded. Files are transcribed sequentially.
|
|
24
|
+
3. The prompt gains one block per transcript, after the file list and before
|
|
25
|
+
the user's own text:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
[Transcript of voice.webm (elevenlabs):]
|
|
29
|
+
<text>
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
4. The bot posts each transcript back into the thread/channel as a quote
|
|
33
|
+
(`🎙️ Transcript of voice.webm:` + every line blockquoted — a bare `>` on
|
|
34
|
+
the first line only breaks at the first pause in speech). Teammates and
|
|
35
|
+
the audit log see what Claude heard; a bad transcript is visible before
|
|
36
|
+
Claude acts on it.
|
|
37
|
+
5. A transcription failure is reported through the existing skipped-files
|
|
38
|
+
feedback (`⚠️ Some files could not be processed`) with reason
|
|
39
|
+
`Transcription failed: <message>` and the note that the raw file was still
|
|
40
|
+
handed to Claude. Never silent, never fatal to the message.
|
|
41
|
+
6. No `transcription` block in config → nothing changes. Non-audio files are
|
|
42
|
+
untouched.
|
|
43
|
+
|
|
44
|
+
Out of scope for round 1: starting a session from a voice note (the first
|
|
45
|
+
message still needs the typed @-mention), `video/*`, Slack's own async
|
|
46
|
+
transcript, streaming/realtime.
|
|
47
|
+
|
|
48
|
+
## Config (top-level)
|
|
49
|
+
|
|
50
|
+
```yaml
|
|
51
|
+
transcription:
|
|
52
|
+
provider: elevenlabs # the only provider in round 1
|
|
53
|
+
apiKey: ... # ElevenLabs key; 0600 file, never in a repo
|
|
54
|
+
model: scribe_v2 # optional, default scribe_v2
|
|
55
|
+
languageCode: hrv # optional; passed through verbatim (ElevenLabs
|
|
56
|
+
# accepts ISO-639-1 and -3); omitted = auto-detect
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Validation happens at boot: an unknown `provider` or a missing `apiKey` is a
|
|
60
|
+
config error and the daemon does not start. One key per daemon — the provider
|
|
61
|
+
is a property of the deployment, not of a chat platform, and it applies to
|
|
62
|
+
Slack and Mattermost alike.
|
|
63
|
+
|
|
64
|
+
## Provider interface
|
|
65
|
+
|
|
66
|
+
```ts
|
|
67
|
+
// src/transcription/types.ts
|
|
68
|
+
interface Transcriber {
|
|
69
|
+
readonly provider: string;
|
|
70
|
+
transcribe(input: { path: string; mimeType: string; name: string }): Promise<string>;
|
|
71
|
+
}
|
|
72
|
+
createTranscriber(config: TranscriptionConfig): Transcriber // src/transcription/index.ts
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
`ElevenLabsTranscriber` (`src/transcription/elevenlabs.ts`) POSTs multipart
|
|
76
|
+
`file` + `model_id` (+ `language_code` when set) to
|
|
77
|
+
`https://api.elevenlabs.io/v1/speech-to-text` with header `xi-api-key`, 120 s
|
|
78
|
+
timeout, and returns `text` from the JSON body. A non-2xx response or an
|
|
79
|
+
empty `text` is an error with the HTTP status and the first 200 chars of the
|
|
80
|
+
body. No SDK: `fetch` + `FormData` on Bun/Node 20. `fetch` is injectable for
|
|
81
|
+
tests.
|
|
82
|
+
|
|
83
|
+
## Wiring
|
|
84
|
+
|
|
85
|
+
- `Config.transcription?: TranscriptionConfig` (`src/config/types.ts`).
|
|
86
|
+
- `index.ts`: `createTranscriber(config.transcription)` once at boot, then
|
|
87
|
+
`sessionManager.setTranscriber(t)`.
|
|
88
|
+
- `SessionManager` passes it as the new last parameter of
|
|
89
|
+
`streaming.buildMessageContent(text, platform, uploadDir, files, debug, transcriber?)`.
|
|
90
|
+
- `BuiltMessageContent.transcripts?: Transcript[]` (`{ name, text }`).
|
|
91
|
+
- `postTranscriptFeedback(platform, threadId, transcripts)` next to
|
|
92
|
+
`postSkippedFilesFeedback`, called at the same three places that post
|
|
93
|
+
skipped-file feedback for a user message with files: follow-up
|
|
94
|
+
(`message-manager.ts`), session start (`lifecycle.ts`), context-prompt
|
|
95
|
+
resolution (`context-prompt/handler.ts`).
|
|
96
|
+
- **Build once.** The session-start path used to build the message content
|
|
97
|
+
(downloading every attachment) and then hand the *built* text plus the
|
|
98
|
+
same files to `offerContextPrompt`, whose send paths build again — double
|
|
99
|
+
download, duplicated file header, and now a double transcription and echo.
|
|
100
|
+
`startSession` now passes the raw prompt and files into
|
|
101
|
+
`offerContextPrompt` and builds only on the fallback path. Behaviour for
|
|
102
|
+
direct-channel-mode sessions (the fallback path) is unchanged.
|
|
103
|
+
|
|
104
|
+
- **The deferred context prompt keeps its files.** When the user answers
|
|
105
|
+
the "include thread context?" prompt with a reaction, the completion event
|
|
106
|
+
carries only simplified file refs; the original `PlatformFile[]` were
|
|
107
|
+
parked in the context-prompt module. The lifecycle listener now takes them
|
|
108
|
+
from there (`takeContextPromptFiles`) and builds with them, posting the
|
|
109
|
+
usual skipped-file and transcript feedback. Before, attachments on that
|
|
110
|
+
path survived only because the pre-built file header rode along in the
|
|
111
|
+
queued prompt text, which the once-only build removed. Regression test in
|
|
112
|
+
`lifecycle.test.ts`.
|
|
113
|
+
|
|
114
|
+
Known, pre-existing, out of scope: the worktree-prompt skip adapters
|
|
115
|
+
(`reaction-router.ts`, `manager.ts` worktree skip) still drop queued files.
|
|
116
|
+
Direct-channel-mode task channels never take that path.
|
|
117
|
+
|
|
118
|
+
## Tests
|
|
119
|
+
|
|
120
|
+
- `elevenlabs.test.ts`: sends the file and model as multipart with the key
|
|
121
|
+
header; includes `language_code` only when configured; returns `text`;
|
|
122
|
+
throws on non-2xx with status and body excerpt; throws on empty text.
|
|
123
|
+
- `index.test.ts`: factory returns an ElevenLabs transcriber; rejects unknown
|
|
124
|
+
provider; rejects missing key.
|
|
125
|
+
- `handler.test.ts` (streaming): audio attachment produces a transcript block
|
|
126
|
+
and a `transcripts` entry; non-audio attachment is not transcribed; no
|
|
127
|
+
transcriber means no transcript; transcriber failure surfaces in `skipped`
|
|
128
|
+
and the file path stays in the header; `postTranscriptFeedback` posts one
|
|
129
|
+
quote per transcript and nothing when empty.
|
|
130
|
+
|
|
131
|
+
## Decisions
|
|
132
|
+
|
|
133
|
+
| Decision | Why |
|
|
134
|
+
|---|---|
|
|
135
|
+
| Top-level config, not per-platform | one vendor key per daemon; the question is open upstream (#519) and the shape can move if Anne prefers per-platform |
|
|
136
|
+
| Transcript echoed to the thread by default | the whole point of the Slack surface is that teammates see the task; a silent transcript hides the one step most likely to be wrong |
|
|
137
|
+
| Keep the raw file in the prompt | Claude may want the audio itself later (e.g. to re-transcribe with a hint), and it costs nothing |
|
|
138
|
+
| Failure = warning, message still sent | fail loud, but a bad vendor day must not eat a teammate's message |
|
|
139
|
+
| `audio/*` only, plus an audio-extension fallback for generic MIME types | Slack audio clips are `audio/webm`; a client that reports `application/octet-stream` for a `.m4a` still gets transcribed; video is a separate question asked upstream |
|
|
140
|
+
| Flat config keys (`apiKey`, `model` beside `provider`) rather than nested per provider | one provider today; #519 asks the maintainer which shape they want and the move is mechanical |
|
|
141
|
+
| Transcript is framed with a bracket header like the file list, not XML tags | it is the sender's own message, exactly as trusted as typed text from the same user; a reviewer suggested XML "boundaries", but the file-list header sets the house style and Claude already treats the block as user content |
|
|
142
|
+
| ElevenLabs error bodies are parsed for `detail.message` | the raw JSON in a thread warning is noise; the status code and a body excerpt remain the fallback |
|
|
143
|
+
|
|
144
|
+
## Review notes (2026-09-02)
|
|
145
|
+
|
|
146
|
+
Gemini flagged `scribe_v2` as non-existent and `language_code` as ISO-639-1
|
|
147
|
+
only; a grounded web search the same day listed `scribe_v2` as current, and
|
|
148
|
+
the Telegram bot on the same box has run `hrv` (ISO-639-3) in production for
|
|
149
|
+
weeks. Both settled against the live API at deploy time; both are
|
|
150
|
+
configurable if the default is ever wrong. Codex found no design flaw beyond
|
|
151
|
+
the once-only build and the pre-existing file-loss paths recorded above.
|
package/package.json
CHANGED