osborn 0.8.28 → 0.8.30

package/.claude/skills/learned-behaviors/SKILL.md

@@ -0,0 +1,41 @@
+# Learned Behaviors
+
+Auto-extracted from voice sessions via PreCompact.
+Last updated: 2026-05-01 | Session: unknown...
+
+USER CORRECTIONS:
+- First write attempt for interruption timer was DENIED because it replaced existing LiveKit code instead of adding on top — always ADD fallbacks, never replace existing mechanisms; user said "you're basically deleting what we have"
+- User denied permission dialog for file edit because they wanted to verify the approach first before committing — ask "should I build this?" before delegating write tasks on significant changes
+- When proposing solution approach, verify against LiveKit SDK source AND docs before implementing — user asked "can you verify that we're doing it the right way in terms of live kits"
+
+USER PREFERENCES:
+- Voice transcription: "MPM"=npm, "mobile publish"=npm publish, "live gate"=LiveKit, "left kit"=LiveKit, "basketball man"=bash command, "Osborne"=osborn — interpret flexibly
+- Sprite is the publish path for npm going forward (not Mac), set up with automation token
+- Prefers additive code changes: keep existing mechanisms as primary, add new approaches as fallback tiers
+- Wants to understand the "why" before approving implementation — explain the mechanism, not just the solution
+- Wants confirmation before committing to solutions when uncertain: "before we commit to doing this"
+
+DOMAIN KNOWLEDGE:
+- `osborn --version` has no version flag — it starts a new LiveKit-connected instance; use `npm list -g osborn` or `npm list -g osborn --json` for version
+- npm automation token (not web login token) required for headless publish — web login token still requires 2FA for publish
+- `playbackPosition` is 0 because interrupt often fires before TTS audio frames start flowing (`pushedDuration = 0`); also race condition between `addDoneCallback` and `waitForPlayoutTask`
+- `skipTTSQueue: true` does NOT bypass audio frame tracking — `ttsTask` code path is identical
+- `EVENT_PLAYBACK_STARTED` ('playbackStarted') is public, fires on first audio frame, accessible via `audioOutput.on()`
+- OpenAI TTS-1 rate: ~14 chars/second confirmed from community benchmarks
+- `check-version` in sandbox/route.ts was incorrectly using `osborn --version` (starts instance); fixed to `npm list -g osborn --json`
+- execInSprite works any time the Sprite container is running, independent of osborn agent health
+- `checkVersion()` already existed in dashboard/page.tsx (lines 325-338); `installedVersion` and `latestVersion` state already existed
+- Dashboard handleRestart/handleUpdate were not reading API response bodies — server returns rich data but client discarded it
+- Railway auto-deploys from GitHub main branch push
+
+EFFECTIVE PATTERNS:
+- Layered fallback approach: preserve existing SDK mechanisms, add custom approach only as fallback — explicit user preference confirmed
+- Taking checkpoint before significant changes gives confidence to experiment
+- Reading API response bodies and surfacing errors/success messages is a simple fix for "silent" operations
+
+ANTI-PATTERNS:
+- Running `osborn --version` on the Sprite created stray processes connected to LiveKit room — never run the osborn binary to check version
+- Replacing existing LiveKit code with custom implementation instead of adding on top — user explicitly denied and corrected this
+- Delegating write task without confirming approach with user first on novel/uncertain changes
+- Not reading the API response body after `await fetch()` — silent failure mode
+</summary>

package/dist/index.js

@@ -1109,13 +1109,61 @@ async function main() {
                 if (handle && typeof handle.addDoneCallback === 'function') {
                     // SpeechHandle — track it and register interruption callback
                     currentSpeechHandle = handle;
+                    // Wall-clock timer: capture when audio actually starts playing (first frame)
+                    // Used as fallback if LiveKit's playbackPosition is 0 (race condition)
+                    let playbackStartedAt = null;
+                    const audioOutputRef = currentSession?.output?.audio;
+                    if (audioOutputRef && typeof audioOutputRef.on === 'function') {
+                        const onPlaybackStarted = () => {
+                            playbackStartedAt = Date.now();
+                            console.log(`🔊 [${sayId}] audio first frame out (playbackStarted)`);
+                            audioOutputRef.off('playbackStarted', onPlaybackStarted);
+                        };
+                        audioOutputRef.on('playbackStarted', onPlaybackStarted);
+                    }
                     handle.addDoneCallback((sh) => {
                         if (sh.interrupted) {
                             console.log(`🔇 [${sayId}] session.say INTERRUPTED`);
-                            const audioOutput = currentSession?._activity?.agentSession?.output?.audio;
-                            const spokenText = audioOutput?.lastPlaybackEvent?.synchronizedTranscript || data.text;
-                            const playbackPositionSec = audioOutput?.lastPlaybackEvent?.playbackPosition ?? 0;
-                            console.log('🔇 Synchronized transcript:', JSON.stringify({ chars: spokenText.length, fullChars: data.text.length, playbackSec: playbackPositionSec, isSynced: !!audioOutput?.lastPlaybackEvent?.synchronizedTranscript }));
+                            const audioOutput = currentSession?.output?.audio;
+                            const sdkTranscript = audioOutput?.lastPlaybackEvent?.synchronizedTranscript;
+                            const sdkPlaybackSec = audioOutput?.lastPlaybackEvent?.playbackPosition ?? 0;
+                            let spokenText;
+                            let method;
+                            if (sdkTranscript) {
+                                // Best case: LiveKit gave us word-accurate transcript (requires alignedTranscript TTS)
+                                spokenText = sdkTranscript;
+                                method = 'sdk-transcript';
+                            }
+                            else if (sdkPlaybackSec > 0) {
+                                // Second: LiveKit gave us playback duration — estimate chars from it
+                                const CHARS_PER_SEC = 14;
+                                const charCount = Math.min(Math.round(sdkPlaybackSec * CHARS_PER_SEC), data.text.length);
+                                const slicePoint = data.text.lastIndexOf(' ', charCount) || charCount;
+                                spokenText = slicePoint > 0 ? data.text.slice(0, slicePoint) : data.text;
+                                method = 'sdk-position';
+                            }
+                            else if (playbackStartedAt !== null) {
+                                // Third: use our wall-clock timer from first audio frame
+                                const elapsedSec = (Date.now() - playbackStartedAt) / 1000;
+                                const CHARS_PER_SEC = 14;
+                                const charCount = Math.min(Math.round(elapsedSec * CHARS_PER_SEC), data.text.length);
+                                const slicePoint = data.text.lastIndexOf(' ', charCount) || charCount;
+                                spokenText = slicePoint > 0 ? data.text.slice(0, slicePoint) : data.text;
+                                method = 'wall-clock';
+                            }
+                            else {
+                                // Fallback: interrupt fired before first frame — pass full block
+                                spokenText = data.text;
+                                method = 'full-block-fallback';
+                            }
+                            console.log('🔇 Interruption estimate:', JSON.stringify({
+                                method,
+                                sdkPlaybackSec,
+                                isSynced: !!sdkTranscript,
+                                spokenChars: spokenText.length,
+                                fullChars: data.text.length,
+                                heard: spokenText.slice(0, 80) + (spokenText.length > 80 ? '...' : '')
+                            }));
                             handleSpeechDone(sh, spokenText, data.text);
                         }
                         else {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "osborn",
-  "version": "0.8.28",
+  "version": "0.8.30",
   "description": "Voice AI coding assistant - local agent that connects to Osborn frontend",
   "type": "module",
   "bin": {