osborn 0.9.88 → 0.9.91

package/.claude/skills/meetings/SKILL.md

@@ -22,18 +22,64 @@ Two trigger patterns:
 
 **Do NOT use this skill** for normal user voice-native messages that don't fit those patterns — those get spoken responses as usual.
 
+## CRITICAL: delegate the file + transcript work to the `writer` sub-agent
+
+The main orchestrator agent has a **hard limit of 3 direct tool calls per turn** (enforced in PreToolUse — Read/Write/Bash/Glob are DENIED after the 3rd call). Writing `meeting-todos.md` and pulling transcripts (curl + jq + Write) is far more than 3 calls, so **doing it directly gets you blocked** ("all tools blocked").
+
+**Sub-agents are exempt from this budget and have full permissions.** So for ALL meeting file/transcript work, **delegate to the `writer` sub-agent in ONE `Task` call** and let it do the whole job (fetch transcript, parse, write `meeting-todos.md`). That's a single tool call for you, and the writer has no budget cap.
+
+```
+Task(
+  subagent_type: 'writer',
+  run_in_background: true,   // silent — don't block voice
+  description: 'update meeting-todos.md',
+  prompt: '<the full instructions below: workspace path, bot ID, what to fetch/parse/write>'
+)
+```
+
+Give the writer everything it needs in the prompt: the session-workspace path, the bot ID, the `us-west-2.recall.ai` endpoint rule, and the `meeting-todos.md` structure. The writer runs the curl/jq/Write steps itself. For research, delegate to the `researcher` sub-agent the same way.
+
+## How to SPEAK INTO the meeting (out of silent mode)
+
+You can talk directly into the Google Meet / Zoom — your words play as the bot's
+voice. The bot casts a "meeting canvas" webpage as its camera+mic; POSTing to
+your own HTTP API pushes speech (and visuals) to it, and Recall pipes the
+canvas audio into the call. This is ONE Bash call, within budget.
+
+**Speak into the meeting:**
+```bash
+curl -sS -X POST http://localhost:8741/canvas -H 'Content-Type: application/json' \
+  -d '{"kind":"say","text":"YOUR WORDS HERE"}'
+```
+(Port is `OSBORN_API_PORT`, default 8741.)
+
+**Show a visual on the bot's camera** (notes, a link, a title):
+```bash
+curl -sS -X POST http://localhost:8741/canvas -H 'Content-Type: application/json' \
+  -d '{"kind":"show","mode":"notes","title":"...","items":["...","..."]}'
+```
+`mode` = `idle` | `notes` (title+items) | `link` (url) | `web` (iframe url) | `text` (title+text).
+
+**When to speak into the meeting:** By DEFAULT stay silent (observer) for
+`[MEETING — *]:` chunks — take notes, don't interrupt. Speak into the meeting
+ONLY when: (a) the voice-native user explicitly tells you to say something to the
+meeting / "tell them X" / "answer that", or (b) you're directly addressed by name
+in the meeting and the user has enabled active mode. When you do speak, keep it
+short and let the room continue. This is the toggle between silent-observer and
+active-participant.
+
 ## How to behave (auto-tagged chunks)
 
 For every `[MEETING — *]:` message:
 
 1. **Do NOT speak.** No TTS output. No conversational reply.
-2. **Update `meeting-todos.md`** in the session workspace. Append new action items, decisions, open questions. One file, evolving.
-3. **Optionally trigger background research silently** via Task tool.
+2. **Delegate to the `writer` sub-agent** (see above) to append new action items, decisions, and open questions to `meeting-todos.md`. Do NOT write the file yourself — you'll hit the 3-call budget and get blocked. Batch chunks if they arrive faster than the writer finishes; one evolving file.
+3. **Delegate research to the `researcher` sub-agent** via `Task` (background, silent) when a chunk warrants it.
 4. **Don't consume voice-native attention.** The user can interrupt with a voice-native message at any time — that's the only kind that gets spoken responses.
 
 ## How to pull transcripts on demand (Bash + curl)
 
-When the user explicitly asks (see triggers above), run these commands. Speak briefly first ("On it"), do the work, then speak the result.
+When the user explicitly asks (see triggers above): speak briefly first ("On it"), then **delegate the fetch+parse+write to the `writer` sub-agent** in one `Task` call (the steps below are what you put in the writer's prompt — they're 4+ Bash/Write calls, over your 3-call budget). When the writer finishes, speak the result. The commands below are the recipe the writer runs, not calls you make directly.
 
 ### Step 1: Get the bot ID
 

package/Dockerfile.sandbox

@@ -164,7 +164,14 @@ PKG_SKILLS_DIR="/usr/local/lib/node_modules/osborn/.claude/skills"
 SEED_VERSION_FILE="${HOME_SKILLS_DIR}/.seed-version"
 mkdir -p "$HOME_SKILLS_DIR"
 CURRENT_SEED_VERSION=$(cat "$SEED_VERSION_FILE" 2>/dev/null | tr -d '[:space:]' || echo "")
-IMAGE_SEED_VERSION="${OSBORN_IMAGE_VERSION:-latest}"
+# Seed version = the INSTALLED package's own version (reliable on every boot),
+# NOT the OSBORN_IMAGE_VERSION env — that env isn't threaded through
+# `fly machine update`, so it went stale and left the marker permanently pinned
+# at 0.9.48, meaning the refresh below never fired and new-version skills never
+# reached the load path. Reading package.json makes CURRENT != IMAGE trip on a
+# real version bump. Falls back to the env / "latest" if the package is unreadable.
+IMAGE_SEED_VERSION=$(grep -m1 '"version"' /usr/local/lib/node_modules/osborn/package.json 2>/dev/null | sed 's/.*"version"[^"]*"\([^"]*\)".*/\1/')
+[ -z "$IMAGE_SEED_VERSION" ] && IMAGE_SEED_VERSION="${OSBORN_IMAGE_VERSION:-latest}"
 if [ -d "$PKG_SKILLS_DIR" ]; then
   REFRESHED=0
   SEEDED=0

package/dist/index.js

@@ -515,6 +515,41 @@ function startApiServer(workingDir, port) {
             console.log(`[canvas] client connected (${canvasClients.size} total)`);
             return;
         }
+        // ── Meeting canvas: TTS audio for speaking INTO the meeting ──────────────
+        // GET /tts?text=... → mp3 (OpenAI TTS). The canvas plays this as a real
+        // <audio> element so Recall's webpage output pipes it into the meeting —
+        // speechSynthesis is NOT captured by Recall, a media element IS.
+        if (req.method === 'GET' && url.pathname === '/tts') {
+            const text = (url.searchParams.get('text') || '').slice(0, 4000);
+            const voice = url.searchParams.get('voice') || 'alloy';
+            const key = process.env.OPENAI_API_KEY;
+            if (!text || !key) {
+                res.writeHead(400, { 'Content-Type': 'application/json' });
+                res.end(JSON.stringify({ error: !key ? 'no OPENAI_API_KEY' : 'no text' }));
+                return;
+            }
+            try {
+                const tts = await fetch('https://api.openai.com/v1/audio/speech', {
+                    method: 'POST',
+                    headers: { 'Authorization': `Bearer ${key}`, 'Content-Type': 'application/json' },
+                    body: JSON.stringify({ model: 'gpt-4o-mini-tts', voice, input: text, response_format: 'mp3' }),
+                });
+                if (!tts.ok) {
+                    const e = await tts.text().catch(() => '');
+                    res.writeHead(502, { 'Content-Type': 'application/json' });
+                    res.end(JSON.stringify({ error: `tts ${tts.status}`, detail: e.slice(0, 200) }));
+                    return;
+                }
+                const buf = Buffer.from(await tts.arrayBuffer());
+                res.writeHead(200, { 'Content-Type': 'audio/mpeg', 'Cache-Control': 'no-store', 'Content-Length': buf.length });
+                res.end(buf);
+            }
+            catch (e) {
+                res.writeHead(500, { 'Content-Type': 'application/json' });
+                res.end(JSON.stringify({ error: e.message }));
+            }
+            return;
+        }
         // ── Meeting canvas: control endpoint (director / agent tool) ─────────────
         // POST /canvas  { kind:'say', text } | { kind:'show', mode, title?, items?, url?, text? }
         if (req.method === 'POST' && url.pathname === '/canvas') {
@@ -1521,6 +1556,36 @@ async function main() {
     let currentUserId = '';
     let activeMeetingBotId = null; // Recall.ai bot ID if in a meeting
     let activeMeetingPoller = null; // Transcript poller bound to that bot
+    // LIVE meeting transcript → LLM (buffered webhook finals). See recall.on('transcript').
+    const meetingTranscriptBuffer = [];
+    let meetingFlushTimer = null;
+    const startMeetingFlush = (botId) => {
+        stopMeetingFlush();
+        console.log(`📓 Meeting transcript flush timer started (bot ${botId}, 20s)`);
+        meetingFlushTimer = setInterval(() => {
+            if (!meetingTranscriptBuffer.length || !currentLLM)
+                return;
+            const turns = meetingTranscriptBuffer.splice(0); // drain
+            const tagged = `[MEETING — ${botId}]:\n${turns.join('\n')}`;
+            try {
+                const ctx = new llm.ChatContext();
+                ctx.addMessage({ role: 'user', content: tagged });
+                currentLLM.chat({ chatCtx: ctx });
+                console.log(`📓 Flushed ${turns.length} meeting turn(s) to LLM`);
+            }
+            catch (err) {
+                console.warn(`⚠️ Meeting flush failed: ${err.message}`);
+            }
+        }, 20_000);
+    };
+    const stopMeetingFlush = () => {
+        if (meetingFlushTimer) {
+            clearInterval(meetingFlushTimer);
+            meetingFlushTimer = null;
+            console.log('📓 Meeting flush timer stopped');
+        }
+        meetingTranscriptBuffer.length = 0;
+    };
     // Track the active resume session ID across scopes (ParticipantConnected + DataReceived)
     // Updated by resume_session, session_selected, continue_session, switch_session handlers
     let currentResumeSessionId;
@@ -1558,26 +1623,18 @@ async function main() {
     // "what was said in the meeting" display, separate from the LLM input path).
     const recall = getRecallClient();
     if (recall) {
-        console.log('🎥 Recall.ai client initialized (webhook STT receiver — LLM forwarding disabled, see meeting-bot Phase 2)');
-        recall.on('transcript', ({ botId, speaker, text }) => {
-            console.log(`📝 Meeting transcript [${speaker}]: ${text}`);
-            // INTENTIONALLY DISABLED — see comment above. Audio path is now LiveKit
-            // → meeting-bot page publishes meeting audio → agent STT processes it.
-            // The line below is preserved as a reference for future re-enablement
-            // (e.g. as a display-only feature, NOT as LLM input).
-            //
-            // if (currentLLM && currentSession) {
-            //   const meetingText = `[Meeting — ${speaker}]: ${text}`
-            //   try {
-            //     if (currentVoiceMode === 'pipeline' || currentVoiceMode === 'direct') {
-            //       const chatCtx = new llm.ChatContext()
-            //       chatCtx.addMessage({ role: 'user', content: meetingText })
-            //       ;(currentLLM as any).chat({ chatCtx })
-            //     }
-            //   } catch (err) {
-            //     console.error('❌ Failed to route meeting transcript:', err)
-            //   }
-            // }
+        console.log('🎥 Recall.ai client initialized — LIVE webhook finals buffered → LLM (batch poller download_url is empty mid-call, so the webhook is the only live transcript source)');
+        // LIVE meeting transcript → LLM. The realtime webhook (handleWebhook) emits
+        // every partial + final. We buffer FINALS (partials are too noisy) and a
+        // flush timer (started on join_meeting) batches them to currentLLM.chat()
+        // as [MEETING — botId]: every ~20s — so the agent actually sees the meeting
+        // and can take notes / delegate to the writer. Previously this was DISABLED
+        // and the poller (batch endpoint) was the only LLM path — but that endpoint
+        // is empty until the meeting ENDS, so mid-call the LLM saw nothing.
+        recall.on('transcript', ({ botId, speaker, text, partial }) => {
+            console.log(`📝 Meeting transcript [${speaker}]${partial ? ' (partial)' : ''}: ${text}`);
+            if (!partial && text.trim())
+                meetingTranscriptBuffer.push(`${speaker}: ${text.trim()}`);
         });
     }
     // ============================================================
@@ -4066,6 +4123,7 @@ async function main() {
         clearFastBrainSession();
         clearPipelineFastBrainSession();
         // Auto-leave any active meeting bot when user disconnects from the room
+        stopMeetingFlush();
         if (activeMeetingPoller) {
             activeMeetingPoller.stop();
             activeMeetingPoller = null;
@@ -4751,6 +4809,9 @@ async function main() {
                                 },
                             });
                             activeMeetingPoller.start();
+                            // LIVE path: buffer webhook finals + flush to the LLM every 20s.
+                            // (The poller above only lands data after the meeting ENDS.)
+                            startMeetingFlush(botId);
                         }
                         catch (err) {
                             console.error('❌ Recall.ai join error:', err);
@@ -4764,8 +4825,9 @@ async function main() {
                 const recallLeave = getRecallClient();
                 if (recallLeave && botId) {
                     try {
-                        // Stop the transcript poller FIRST so no more transcript chunks get
-                        // forwarded to the LLM during the leave.
+                        // Stop the transcript poller + live flush FIRST so no more chunks
+                        // get forwarded to the LLM during the leave.
+                        stopMeetingFlush();
                         if (activeMeetingPoller) {
                             activeMeetingPoller.stop();
                             activeMeetingPoller = null;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "osborn",
-  "version": "0.9.88",
+  "version": "0.9.91",
   "description": "Voice AI coding assistant - local agent that connects to Osborn frontend",
   "type": "module",
   "bin": {