talkthrough-mcp 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- talkthrough_mcp-0.1.0/LICENSE +21 -0
- talkthrough_mcp-0.1.0/PKG-INFO +584 -0
- talkthrough_mcp-0.1.0/README.md +552 -0
- talkthrough_mcp-0.1.0/pyproject.toml +103 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/__init__.py +9 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/__main__.py +5 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/cli.py +139 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/__init__.py +1 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/audio.py +30 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/dedup.py +63 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/errors.py +39 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/ffmpeg.py +74 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/frames.py +158 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/jobs.py +140 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/manifest.py +268 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/ocr.py +149 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/pipeline.py +365 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/probe.py +68 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/stt.py +99 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/core/wallclock.py +167 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/guidance.py +389 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/py.typed +0 -0
- talkthrough_mcp-0.1.0/src/talkthrough_mcp/server.py +415 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 korovin-aa97
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,584 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: talkthrough-mcp
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Local-first MCP server that turns narrated screen recordings into agent-ready structured data: timestamped transcript, scene keyframes, OCR text, and wall-clock anchoring.
|
|
5
|
+
Keywords: mcp,mcp server,model-context-protocol,screen-recording,video,transcription,whisper,ocr,feedback
|
|
6
|
+
Author: korovin-aa97
|
|
7
|
+
Author-email: korovin-aa97 <korovin.aa97@gmail.com>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Operating System :: MacOS
|
|
13
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Topic :: Multimedia :: Video
|
|
19
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
20
|
+
Classifier: Typing :: Typed
|
|
21
|
+
Requires-Dist: faster-whisper>=1.2.1
|
|
22
|
+
Requires-Dist: mcp>=1.28.1
|
|
23
|
+
Requires-Dist: pillow>=12.3.0
|
|
24
|
+
Requires-Dist: rapidocr>=3.9.1
|
|
25
|
+
Requires-Dist: static-ffmpeg>=3.0
|
|
26
|
+
Requires-Python: >=3.11, <3.14
|
|
27
|
+
Project-URL: Homepage, https://github.com/korovin-aa97/talkthrough-mcp
|
|
28
|
+
Project-URL: Repository, https://github.com/korovin-aa97/talkthrough-mcp
|
|
29
|
+
Project-URL: Issues, https://github.com/korovin-aa97/talkthrough-mcp/issues
|
|
30
|
+
Project-URL: Changelog, https://github.com/korovin-aa97/talkthrough-mcp/blob/main/CHANGELOG.md
|
|
31
|
+
Description-Content-Type: text/markdown
|
|
32
|
+
|
|
33
|
+
# talkthrough-mcp
|
|
34
|
+
|
|
35
|
+
<!-- mcp-name: io.github.korovin-aa97/talkthrough-mcp -->
|
|
36
|
+
|
|
37
|
+
[](https://github.com/korovin-aa97/talkthrough-mcp/actions/workflows/ci.yml)
|
|
38
|
+
[](LICENSE)
|
|
39
|
+
[](pyproject.toml)
|
|
40
|
+
<!-- uncomment at PyPI publish: [](https://pypi.org/project/talkthrough-mcp/) -->
|
|
41
|
+
|
|
42
|
+
[Quickstart](#quickstart) · [Tools](#tools) · [FAQ](#faq) ·
|
|
43
|
+
[Troubleshooting](docs/TROUBLESHOOTING.md) · [Changelog](CHANGELOG.md) ·
|
|
44
|
+
[Contributing](CONTRIBUTING.md)
|
|
45
|
+
|
|
46
|
+
**Feedback ingestion for AI agents.** Record your screen and talk; your agent
|
|
47
|
+
does the rest — files the bugs, writes the spec, builds the backlog.
|
|
48
|
+
|
|
49
|
+

|
|
50
|
+
|
|
51
|
+
`talkthrough-mcp` is a local-first MCP server that turns a narrated screen
|
|
52
|
+
recording (or any video/audio file) into agent-ready structured data:
|
|
53
|
+
timestamped transcript segments, scene-change keyframes, OCR'd on-screen text,
|
|
54
|
+
and wall-clock anchoring. Everything is served through lazy retrieval tools, so
|
|
55
|
+
a 30-minute recording never floods the model context — the agent pulls exactly
|
|
56
|
+
the transcript slice, moment bundle, or frame it needs.
|
|
57
|
+
|
|
58
|
+
There is no LLM inside the server and no cloud anywhere in the path: ffmpeg,
|
|
59
|
+
faster-whisper, and RapidOCR run on your machine, and the calling agent brings
|
|
60
|
+
the intelligence. What makes it different from screen-recorder SaaS and
|
|
61
|
+
video-analyzer MCPs: it works on arbitrary local files, it ships the agent
|
|
62
|
+
workflows (server prompts + example agents), and it anchors every timestamp to
|
|
63
|
+
**wall-clock time** — so "the moment I said the checkout hung" maps straight to
|
|
64
|
+
the right window of your server logs.
|
|
65
|
+
|
|
66
|
+
## Quickstart
|
|
67
|
+
|
|
68
|
+
One command, no system dependencies: ffmpeg falls back to a bundled build,
|
|
69
|
+
OCR is pip-only, and whisper models download themselves on first use.
|
|
70
|
+
|
|
71
|
+
<!-- gen:install — generated by scripts/gen_integrations.py; do not hand-edit -->
|
|
72
|
+
|
|
73
|
+
[](https://cursor.com/en/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
|
|
74
|
+
[](https://vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D)
|
|
75
|
+
[](https://insiders.vscode.dev/redirect/mcp/install?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%2C%22type%22%3A%22stdio%22%7D&quality=insiders)
|
|
76
|
+
[](https://lmstudio.ai/install-mcp?name=talkthrough&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ0YWxrdGhyb3VnaC1tY3AiXX0%3D)
|
|
77
|
+
[](https://kiro.dev/launch/mcp/add?name=talkthrough&config=%7B%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22talkthrough-mcp%22%5D%7D)
|
|
78
|
+
[](https://block.github.io/goose/extension?cmd=uvx&arg=talkthrough-mcp&id=talkthrough&name=Talkthrough&description=Turn%20narrated%20screen%20recordings%20into%20agent-ready%20data)
|
|
79
|
+
|
|
80
|
+
### Claude Code
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
claude mcp add -s user talkthrough -- uvx talkthrough-mcp
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Or install the full plugin (server + the five workflow commands + the triage
|
|
87
|
+
agent + an agent skill):
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
/plugin marketplace add korovin-aa97/talkthrough-mcp
|
|
91
|
+
/plugin install talkthrough@talkthrough
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### Every other MCP client
|
|
95
|
+
|
|
96
|
+
<details>
|
|
97
|
+
<summary><b>Claude Desktop</b></summary>
|
|
98
|
+
|
|
99
|
+
`claude_desktop_config.json`:
|
|
100
|
+
|
|
101
|
+
```json
|
|
102
|
+
{
|
|
103
|
+
"mcpServers": {
|
|
104
|
+
"talkthrough": {
|
|
105
|
+
"command": "uvx",
|
|
106
|
+
"args": [
|
|
107
|
+
"talkthrough-mcp"
|
|
108
|
+
]
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
More: [`integrations/claude-desktop/`](integrations/claude-desktop/)
|
|
115
|
+
|
|
116
|
+
</details>
|
|
117
|
+
|
|
118
|
+
<details>
|
|
119
|
+
<summary><b>Cursor</b></summary>
|
|
120
|
+
|
|
121
|
+
`~/.cursor/mcp.json (or project .cursor/mcp.json)`:
|
|
122
|
+
|
|
123
|
+
```json
|
|
124
|
+
{
|
|
125
|
+
"mcpServers": {
|
|
126
|
+
"talkthrough": {
|
|
127
|
+
"command": "uvx",
|
|
128
|
+
"args": [
|
|
129
|
+
"talkthrough-mcp"
|
|
130
|
+
]
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
More: [`integrations/cursor/`](integrations/cursor/)
|
|
137
|
+
|
|
138
|
+
</details>
|
|
139
|
+
|
|
140
|
+
<details>
|
|
141
|
+
<summary><b>OpenAI Codex CLI</b></summary>
|
|
142
|
+
|
|
143
|
+
`~/.codex/config.toml (or project-scoped .codex/config.toml in trusted projects)`:
|
|
144
|
+
|
|
145
|
+
```toml
|
|
146
|
+
[mcp_servers.talkthrough]
|
|
147
|
+
command = "uvx"
|
|
148
|
+
args = ["talkthrough-mcp"]
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
More: [`integrations/codex/`](integrations/codex/)
|
|
152
|
+
|
|
153
|
+
</details>
|
|
154
|
+
|
|
155
|
+
<details>
|
|
156
|
+
<summary><b>Gemini CLI</b></summary>
|
|
157
|
+
|
|
158
|
+
`~/.gemini/settings.json`:
|
|
159
|
+
|
|
160
|
+
```json
|
|
161
|
+
{
|
|
162
|
+
"mcpServers": {
|
|
163
|
+
"talkthrough": {
|
|
164
|
+
"command": "uvx",
|
|
165
|
+
"args": [
|
|
166
|
+
"talkthrough-mcp"
|
|
167
|
+
]
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
More: [`integrations/gemini-cli/`](integrations/gemini-cli/)
|
|
174
|
+
|
|
175
|
+
</details>
|
|
176
|
+
|
|
177
|
+
<details>
|
|
178
|
+
<summary><b>Cline / Roo Code</b></summary>
|
|
179
|
+
|
|
180
|
+
`cline_mcp_settings.json (via MCP Servers UI)`:
|
|
181
|
+
|
|
182
|
+
```json
|
|
183
|
+
{
|
|
184
|
+
"mcpServers": {
|
|
185
|
+
"talkthrough": {
|
|
186
|
+
"command": "uvx",
|
|
187
|
+
"args": [
|
|
188
|
+
"talkthrough-mcp"
|
|
189
|
+
]
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
}
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
More: [`integrations/cline/`](integrations/cline/)
|
|
196
|
+
|
|
197
|
+
</details>
|
|
198
|
+
|
|
199
|
+
<details>
|
|
200
|
+
<summary><b>OpenClaw</b></summary>
|
|
201
|
+
|
|
202
|
+
`~/.openclaw/openclaw.json`:
|
|
203
|
+
|
|
204
|
+
```json
|
|
205
|
+
{
|
|
206
|
+
"mcp": {
|
|
207
|
+
"servers": {
|
|
208
|
+
"talkthrough": {
|
|
209
|
+
"command": "uvx",
|
|
210
|
+
"args": [
|
|
211
|
+
"talkthrough-mcp"
|
|
212
|
+
]
|
|
213
|
+
}
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
More: [`integrations/openclaw/`](integrations/openclaw/)
|
|
220
|
+
|
|
221
|
+
</details>
|
|
222
|
+
|
|
223
|
+
<details>
|
|
224
|
+
<summary><b>OpenCode</b></summary>
|
|
225
|
+
|
|
226
|
+
`opencode.json (project) or ~/.config/opencode/opencode.json`:
|
|
227
|
+
|
|
228
|
+
```json
|
|
229
|
+
{
|
|
230
|
+
"mcp": {
|
|
231
|
+
"talkthrough": {
|
|
232
|
+
"type": "local",
|
|
233
|
+
"command": [
|
|
234
|
+
"uvx",
|
|
235
|
+
"talkthrough-mcp"
|
|
236
|
+
],
|
|
237
|
+
"enabled": true
|
|
238
|
+
}
|
|
239
|
+
}
|
|
240
|
+
}
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
More: [`integrations/opencode/`](integrations/opencode/)
|
|
244
|
+
|
|
245
|
+
</details>
|
|
246
|
+
|
|
247
|
+
<details>
|
|
248
|
+
<summary><b>Goose</b></summary>
|
|
249
|
+
|
|
250
|
+
`~/.config/goose/config.yaml`:
|
|
251
|
+
|
|
252
|
+
```yaml
|
|
253
|
+
extensions:
|
|
254
|
+
talkthrough:
|
|
255
|
+
enabled: true
|
|
256
|
+
type: stdio
|
|
257
|
+
cmd: uvx
|
|
258
|
+
args: ["talkthrough-mcp"]
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
More: [`integrations/goose/`](integrations/goose/)
|
|
262
|
+
|
|
263
|
+
</details>
|
|
264
|
+
|
|
265
|
+
<details>
|
|
266
|
+
<summary><b>GitHub Copilot CLI</b></summary>
|
|
267
|
+
|
|
268
|
+
`~/.copilot/mcp-config.json`:
|
|
269
|
+
|
|
270
|
+
```json
|
|
271
|
+
{
|
|
272
|
+
"mcpServers": {
|
|
273
|
+
"talkthrough": {
|
|
274
|
+
"command": "uvx",
|
|
275
|
+
"args": [
|
|
276
|
+
"talkthrough-mcp"
|
|
277
|
+
]
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
}
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
More: [`integrations/copilot-cli/`](integrations/copilot-cli/)
|
|
284
|
+
|
|
285
|
+
</details>
|
|
286
|
+
|
|
287
|
+
<details>
|
|
288
|
+
<summary><b>Windsurf</b></summary>
|
|
289
|
+
|
|
290
|
+
`~/.codeium/windsurf/mcp_config.json`:
|
|
291
|
+
|
|
292
|
+
```json
|
|
293
|
+
{
|
|
294
|
+
"mcpServers": {
|
|
295
|
+
"talkthrough": {
|
|
296
|
+
"command": "uvx",
|
|
297
|
+
"args": [
|
|
298
|
+
"talkthrough-mcp"
|
|
299
|
+
]
|
|
300
|
+
}
|
|
301
|
+
}
|
|
302
|
+
}
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
More: [`integrations/windsurf/`](integrations/windsurf/)
|
|
306
|
+
|
|
307
|
+
</details>
|
|
308
|
+
|
|
309
|
+
<details>
|
|
310
|
+
<summary><b>Zed</b></summary>
|
|
311
|
+
|
|
312
|
+
`settings.json (Zed)`:
|
|
313
|
+
|
|
314
|
+
```json
|
|
315
|
+
{
|
|
316
|
+
"context_servers": {
|
|
317
|
+
"talkthrough": {
|
|
318
|
+
"source": "custom",
|
|
319
|
+
"command": {
|
|
320
|
+
"path": "uvx",
|
|
321
|
+
"args": [
|
|
322
|
+
"talkthrough-mcp"
|
|
323
|
+
]
|
|
324
|
+
}
|
|
325
|
+
}
|
|
326
|
+
}
|
|
327
|
+
}
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
More: [`integrations/zed/`](integrations/zed/)
|
|
331
|
+
|
|
332
|
+
</details>
|
|
333
|
+
|
|
334
|
+
Any other MCP stdio client uses the same server command: `uvx talkthrough-mcp`.
|
|
335
|
+
Per-engine folders with exactly these snippets plus verification steps live
|
|
336
|
+
in [`integrations/`](integrations/); agents can self-install via
|
|
337
|
+
[`llms-install.md`](llms-install.md).
|
|
338
|
+
|
|
339
|
+
<!-- /gen:install -->
|
|
340
|
+
|
|
341
|
+
### Local checkout (development)
|
|
342
|
+
|
|
343
|
+
```bash
|
|
344
|
+
git clone https://github.com/korovin-aa97/talkthrough-mcp
|
|
345
|
+
claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
Then, in your agent:
|
|
349
|
+
|
|
350
|
+
> Process `~/Desktop/recording.mov` and triage it — or just invoke the
|
|
351
|
+
> `triage-recording` server prompt.
|
|
352
|
+
|
|
353
|
+
## Tools
|
|
354
|
+
|
|
355
|
+
| Tool | What it does |
|
|
356
|
+
|---|---|
|
|
357
|
+
| `process_media(path, recorded_at?, vocabulary?, language?, model?, force?)` | Ingest a video/audio file: local STT, keyframes, OCR, wall-clock. Returns a compact summary. Idempotent by content hash — re-calls are instant. |
|
|
358
|
+
| `get_transcript(job_id, start_ms?, end_ms?, format?)` | Paginated transcript as `segments`, `text`, or `srt`; truncation returns `next_start_ms`. |
|
|
359
|
+
| `get_frames(job_id, at_ms? \| start_ms?+end_ms?, max_frames?, include_duplicates?)` | Keyframe images nearest a timestamp or evenly thinned across a range (unique frames by default, max 6/call). |
|
|
360
|
+
| `get_moment(job_id, start_ms, end_ms)` | The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range. |
|
|
361
|
+
| `search(job_id, query)` | Substring search over the transcript AND on-screen OCR text; hits carry `t_ms`/`t_wall` and frame refs. |
|
|
362
|
+
| `extract_frame(job_id, at_ms, crop?)` | Exact-timestamp full-resolution re-extract from the source video (optional crop) when keyframes miss the instant. |
|
|
363
|
+
| `list_jobs()` | Recent processed recordings with durations, wall-clock starts, and counts. |
|
|
364
|
+
|
|
365
|
+
Every tool description ships 10+ usage examples, so agents pick the right tool
|
|
366
|
+
without extra prompting.
|
|
367
|
+
|
|
368
|
+
## Server prompts (slash commands in MCP clients)
|
|
369
|
+
|
|
370
|
+
| Prompt | Workflow |
|
|
371
|
+
|---|---|
|
|
372
|
+
| `triage-recording` | Narrated screencast → precise findings JSON (bug/feature/question routing, frame evidence) |
|
|
373
|
+
| `spec-from-workshop` | Recorded workshop → structured spec with quoted decisions and open questions |
|
|
374
|
+
| `backlog-from-demo` | Product demo → prioritized backlog with timestamped evidence |
|
|
375
|
+
| `meeting-actions` | Meeting audio → action items, decisions, open questions |
|
|
376
|
+
| `correlate-with-logs` | Recording remarks ↔ system logs via wall-clock windows |
|
|
377
|
+
|
|
378
|
+
The same prompts live as plain files in [`examples/prompts/`](examples/prompts/)
|
|
379
|
+
if your client doesn't surface MCP prompts. The findings contract used by
|
|
380
|
+
`triage-recording` is [`examples/output-contract.schema.json`](examples/output-contract.schema.json).
|
|
381
|
+
|
|
382
|
+
## Works as a skill too (no MCP required)
|
|
383
|
+
|
|
384
|
+
The same workflow ships as a cross-engine [Agent Skill](https://agentskills.io)
|
|
385
|
+
at [`.agents/skills/talkthrough/`](.agents/skills/talkthrough/) — Claude Code,
|
|
386
|
+
Codex CLI (`$talkthrough`), Cursor, Copilot, Gemini CLI, Goose and other
|
|
387
|
+
SKILL.md-compatible tools read it. Agents without MCP wiring can drive the
|
|
388
|
+
[CLI](#cli) directly: `talkthrough-mcp process recording.mov --json` prints the
|
|
389
|
+
same summary the MCP tool returns, and the job store is shared either way.
|
|
390
|
+
|
|
391
|
+
## Wall-clock anchoring
|
|
392
|
+
|
|
393
|
+
Every timestamped result carries both `t_ms` (video-relative) and `t_wall`
|
|
394
|
+
(ISO 8601 real time) once the recording start is known. Resolution ladder:
|
|
395
|
+
|
|
396
|
+
1. `recorded_at` parameter (agent/user override) → confidence `exact`
|
|
397
|
+
2. QuickTime `com.apple.quicktime.creationdate` tag, carries the local
|
|
398
|
+
timezone (QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26) → `high`
|
|
399
|
+
3. Container `creation_time` tag (UTC) → `medium` — macOS 26+ ⌘⇧5/ReplayKit
|
|
400
|
+
screen recordings land here (no `creationdate` tag anymore); pass
|
|
401
|
+
`recorded_at=` when local-tz `t_wall` matters
|
|
402
|
+
4. File mtime minus duration (recorders finalize files at recording END) → `low`
|
|
403
|
+
5. Nothing → tools still work with relative `t_ms` only
|
|
404
|
+
|
|
405
|
+
Why it matters: "the upload spinner froze *here*" becomes a ±30 s grep window
|
|
406
|
+
in your server logs.
|
|
407
|
+
|
|
408
|
+
## Privacy
|
|
409
|
+
|
|
410
|
+
Everything runs locally: your recordings never leave your machine, speech is
|
|
411
|
+
transcribed by a local whisper model, OCR is local ONNX inference, and there is
|
|
412
|
+
no telemetry. The only network access is one-time tool/model downloads (ffmpeg
|
|
413
|
+
build, whisper model, OCR models).
|
|
414
|
+
|
|
415
|
+
## Languages
|
|
416
|
+
|
|
417
|
+
Narration in any of Whisper's ~99 languages works: the language is
|
|
418
|
+
auto-detected per recording, and the summary reports both `language` and
|
|
419
|
+
`language_probability` so agents can tell a confident detection from a shaky
|
|
420
|
+
one (silence or music at the start can fool the detector — pin it with
|
|
421
|
+
`language="ru"` and `force=true` when that happens).
|
|
422
|
+
|
|
423
|
+
Pick the model for your languages — per call (`model=` parameter, agents do
|
|
424
|
+
this themselves when a transcript comes back garbled) or as the server
|
|
425
|
+
default (`TALKTHROUGH_WHISPER_MODEL`):
|
|
426
|
+
|
|
427
|
+
| Model | Size | Best for |
|
|
428
|
+
|---|---|---|
|
|
429
|
+
| `small` (default) | 464 MB | English and major-language narration on CPU |
|
|
430
|
+
| `large-v3-turbo` | ~1.5 GB | **recommended for non-English** — near-large quality at near-small speed |
|
|
431
|
+
| `medium` | ~1.5 GB | conservative alternative to turbo |
|
|
432
|
+
| `tiny` / `base` | 75–145 MB | quick drafts, CI |
|
|
433
|
+
| `*.en` variants | — | English-only, slightly faster/better for EN |
|
|
434
|
+
|
|
435
|
+
Tips that work in every language: pass product names via
|
|
436
|
+
`vocabulary="Term1, Term2"` (biases the decoder so jargon survives), and note
|
|
437
|
+
that the workflow prompts instruct agents to write digests in the
|
|
438
|
+
**narrator's language** while keeping quotes verbatim — the server never
|
|
439
|
+
translates (exact quotes are evidence; translation is the agent's job).
|
|
440
|
+
|
|
441
|
+
On-screen text (OCR) defaults to RapidOCR's Latin + Chinese models. For other
|
|
442
|
+
scripts set `TALKTHROUGH_OCR_LANG` to your language — `ru`/`uk` (→ the
|
|
443
|
+
`eslav` pack), `ja`, `ko`, `ar`, `hi`, `el`, `th`, or any RapidOCR pack name
|
|
444
|
+
like `cyrillic` — and reprocess with `force=true`; the matching recognition
|
|
445
|
+
model downloads once. Spoken-language support is unaffected either way.
|
|
446
|
+
|
|
447
|
+
## Configuration
|
|
448
|
+
|
|
449
|
+
| Env var | Default | Meaning |
|
|
450
|
+
|---|---|---|
|
|
451
|
+
| `TALKTHROUGH_WHISPER_MODEL` | `small` | default whisper model (`tiny`/`base`/`small`/`medium`/`large-v3`/`large-v3-turbo`); the `model` tool param overrides per call |
|
|
452
|
+
| `TALKTHROUGH_OCR` | `on` | set `off` to skip OCR |
|
|
453
|
+
| `TALKTHROUGH_OCR_LANG` | Latin+Chinese | recognition script for on-screen text: a language code (`ru`, `ja`, `ko`, `ar`, `hi`, …) or a RapidOCR pack name (`eslav`, `cyrillic`, `latin`, …); the model downloads once |
|
|
454
|
+
| `TALKTHROUGH_OCR_PARAMS` | — | advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. `{"Rec.lang_type": "cyrillic"}` |
|
|
455
|
+
| `TALKTHROUGH_MAX_SECONDS` | `7200` | max media duration |
|
|
456
|
+
| `TALKTHROUGH_MAX_FRAMES` | `600` | keyframe cap per job |
|
|
457
|
+
| `TALKTHROUGH_HOME` | `~/.talkthrough` | job store root |
|
|
458
|
+
|
|
459
|
+
## CLI
|
|
460
|
+
|
|
461
|
+
The pipeline is also a CLI — useful for pre-processing long recordings outside
|
|
462
|
+
an agent session (the store is content-addressed, so the agent then queries the
|
|
463
|
+
same job instantly):
|
|
464
|
+
|
|
465
|
+
```bash
|
|
466
|
+
talkthrough-mcp process ~/Videos/long-session.mov # prints the summary
|
|
467
|
+
talkthrough-mcp process demo.mov --json # machine-readable
|
|
468
|
+
talkthrough-mcp gc --keep-days 30 # clean the job store
|
|
469
|
+
talkthrough-mcp serve # stdio MCP server (default)
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
First run notes: missing system ffmpeg triggers a one-time `static-ffmpeg`
|
|
473
|
+
download; the first transcription downloads the whisper model (~460 MB for
|
|
474
|
+
`small`); both are cached. After that, expect roughly 3× faster than real time
|
|
475
|
+
on an Apple-Silicon CPU with the default model, OCR included (a 2-minute clip
|
|
476
|
+
processes in ~40 s) — and instant re-runs on the same file. Progress streams
|
|
477
|
+
as MCP progress notifications, and the CLI prints stage lines. More:
|
|
478
|
+
[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
|
|
479
|
+
|
|
480
|
+
## Windows (best-effort)
|
|
481
|
+
|
|
482
|
+
CI runs lint, the unit suite, and a full CLI smoke on `windows-latest`
|
|
483
|
+
(static-ffmpeg Windows build, whisper `tiny` transcription, OCR, and the
|
|
484
|
+
instant idempotent re-run). Notes: the per-job lock is POSIX `fcntl` and
|
|
485
|
+
degrades to a no-op on Windows — fine for a single-user machine; quote paths
|
|
486
|
+
with spaces (`uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4"`).
|
|
487
|
+
Windows is not a release gate — if something breaks, please open an issue.
|
|
488
|
+
|
|
489
|
+
## Supported inputs
|
|
490
|
+
|
|
491
|
+
Video: `.mov` `.mp4` `.webm` `.mkv` — audio-only: `.m4a` `.mp3` `.wav` `.ogg`
|
|
492
|
+
`.flac` (transcript tools only; frame tools explain why they're unavailable).
|
|
493
|
+
Local files only.
|
|
494
|
+
|
|
495
|
+
## Limitations
|
|
496
|
+
|
|
497
|
+
Honest edges, so you can decide fast:
|
|
498
|
+
|
|
499
|
+
- **One speaker stream.** No diarization yet — "who said it" isn't tracked
|
|
500
|
+
([#4](https://github.com/korovin-aa97/talkthrough-mcp/issues/4)).
|
|
501
|
+
- **Local files only.** No URL/YouTube ingestion
|
|
502
|
+
([#5](https://github.com/korovin-aa97/talkthrough-mcp/issues/5)) — download
|
|
503
|
+
first.
|
|
504
|
+
- **Keyframes + transcript, not motion analysis.** A glitch *between* scene
|
|
505
|
+
changes can be invisible in the frame set; `extract_frame` re-checks any
|
|
506
|
+
instant, but frame-by-frame motion reasoning is your multimodal model's job.
|
|
507
|
+
- **STT quality tracks the model you pick.** The default `small` favors speed;
|
|
508
|
+
non-English narration wants `model="large-v3-turbo"` (see
|
|
509
|
+
[Languages](#languages)).
|
|
510
|
+
- **OCR reads crisp UI text well;** tiny or low-contrast print is best-effort.
|
|
511
|
+
- **Wall-clock confidence depends on recorder metadata** — worst case pass
|
|
512
|
+
`recorded_at=` (see the ladder above).
|
|
513
|
+
- **Windows is best-effort** (see above).
|
|
514
|
+
|
|
515
|
+
## How it compares
|
|
516
|
+
|
|
517
|
+
| | talkthrough | cloud recorder SaaS | meeting notetakers | typical video-analyzer MCPs |
|
|
518
|
+
|---|---|---|---|---|
|
|
519
|
+
| Runs fully locally | ✅ | ❌ | ❌ | varies |
|
|
520
|
+
| Any local video/audio file | ✅ | browser/app captures | meetings only | ✅ |
|
|
521
|
+
| Wall-clock anchoring (log correlation) | ✅ | ❌ | ❌ | ❌ |
|
|
522
|
+
| Ships agent workflows (prompts, skill, findings contract) | ✅ | ❌ | ❌ | ❌ |
|
|
523
|
+
| OCR of on-screen text, searchable | ✅ | some | ❌ | rare |
|
|
524
|
+
|
|
525
|
+
## FAQ
|
|
526
|
+
|
|
527
|
+
**Why not just upload the video to a multimodal model (e.g. Gemini)?**
|
|
528
|
+
For a short, non-sensitive clip — do that. The trade-offs appear with length
|
|
529
|
+
and sensitivity: an hour of screen recording costs on the order of a million
|
|
530
|
+
tokens *per question*, the file leaves your machine, and you still can't map a
|
|
531
|
+
remark to `14:32:07 UTC` to grep your server logs. talkthrough indexes once,
|
|
532
|
+
locally, then answers any number of follow-ups from the index.
|
|
533
|
+
|
|
534
|
+
**Why not screenpipe?**
|
|
535
|
+
Different job. screenpipe is an always-on recorder of *your* machine going
|
|
536
|
+
forward (commercial license). It can't open the `.mov` a teammate or customer
|
|
537
|
+
just sent you. talkthrough analyzes any file it's handed — the two compose
|
|
538
|
+
fine.
|
|
539
|
+
|
|
540
|
+
**There are agent skills that "watch" videos. Why a server with an index?**
|
|
541
|
+
Watch-style skills push a budgeted frame dump into the context window (and go
|
|
542
|
+
sparse on long videos), often call cloud STT for the audio, and keep nothing.
|
|
543
|
+
talkthrough builds a persistent local index — transcript + OCR, full-text
|
|
544
|
+
searchable — retrieves exact frames lazily, anchors everything to wall-clock
|
|
545
|
+
time, and answers the next question without reprocessing.
|
|
546
|
+
|
|
547
|
+
**I use Jam for bug reports — do I need this?**
|
|
548
|
+
Keep Jam for browser bugs: console+network captured at record time is great
|
|
549
|
+
evidence. talkthrough covers what a browser extension can't — desktop apps,
|
|
550
|
+
mobile screencasts, ops incidents, meetings, any file — with no account, and
|
|
551
|
+
correlates with *server-side* logs via wall-clock time.
|
|
552
|
+
|
|
553
|
+
**Can't I just script ffmpeg + whisper myself?**
|
|
554
|
+
Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change
|
|
555
|
+
detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock
|
|
556
|
+
ladder, MCP tools with embedded usage examples, five workflow prompts, and a
|
|
557
|
+
findings contract. One `uvx` command instead of an afternoon of glue.
|
|
558
|
+
|
|
559
|
+
**Is it really local? What leaves my machine?**
|
|
560
|
+
Nothing at runtime. The network is used only for one-time downloads (ffmpeg
|
|
561
|
+
build, whisper/OCR models). No telemetry. See [Privacy](#privacy) — and
|
|
562
|
+
[SECURITY.md](SECURITY.md) treats a violation of this promise as a
|
|
563
|
+
vulnerability.
|
|
564
|
+
|
|
565
|
+
## For agents & tooling
|
|
566
|
+
|
|
567
|
+
Machine-readable entry points, so AI agents can install and use this server
|
|
568
|
+
without a human reading docs:
|
|
569
|
+
|
|
570
|
+
- [`llms-install.md`](llms-install.md) — step-by-step install instructions for agents
|
|
571
|
+
- [`llms.txt`](llms.txt) — index of the documentation
|
|
572
|
+
- [`.agents/skills/talkthrough/SKILL.md`](.agents/skills/talkthrough/SKILL.md) — an [Agent Skill](https://agentskills.io) teaching the tool workflow; discovered automatically inside a checkout by Codex CLI (`$talkthrough`) and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools
|
|
573
|
+
- [`AGENTS.md`](AGENTS.md) — instructions for coding agents contributing to this repo
|
|
574
|
+
- [`server.json`](server.json) — MCP registry manifest
|
|
575
|
+
- [`integrations/`](integrations/) — per-engine adapters, all generated from one source of truth and drift-tested (incl. the Claude Code plugin under [`integrations/claude-code/`](integrations/claude-code/))
|
|
576
|
+
|
|
577
|
+
## Roadmap (not in v1)
|
|
578
|
+
|
|
579
|
+
URL/YouTube ingestion · speaker diarization · cloud STT · embeddings/semantic
|
|
580
|
+
search · hosted/remote mode · `.mcpb` bundle · whisper.cpp backend
|
|
581
|
+
|
|
582
|
+
## License
|
|
583
|
+
|
|
584
|
+
MIT
|