escribano 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Eduardo Sanchez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,297 @@
+# Escribano
+
+Record your screen. Get a structured summary of what you did.
+
+> **Platform:** macOS (Apple Silicon) required. Linux/Windows on the roadmap.
+> **Minimum:** 16GB unified memory (32GB recommended for best quality)
+
+---
+
+## What you put in
+
+A screen recording. Could be 20 minutes, could be 3 hours. You didn't take notes.
+
+## What you get back (~9 minutes later)
+
+```markdown
+# Session Card - Feb 25, 2026
+
+## Escribano Pipeline Optimization
+**1h 53m** | coding 22m, debugging 30m, terminal 24m, review 58m, planning 6m
+
+- Optimized the video processing pipeline by evaluating skip-frame strategies 
+  and removing scene detection for 180-minute videos.
+- Resolved persistent VLM parsing failures and truncation errors by implementing 
+  raw response logging and fallback mechanisms.
+- Executed database migrations to add the new observations table schema.
+- Benchmarked the performance of the GLM-5 and Qwen-VL models.
+
+## Frame Extraction & Scene Detection
+**19m** | coding 11m, debugging 4m, terminal 4m
+
+- Developed TypeScript scripts for video frame extraction using FFmpeg.
+- Debugged a critical parsing failure at Frame 3.
+- Monitored terminal logs to track progress of a 792-second video file.
+
+## Research & System Analysis
+**22m** | review 3m, research 2m, coding 7m, terminal 6m
+
+- Reviewed GitHub Copilot pricing and Screenpipe repository architecture.
+- Investigated the database schema in TablePlus.
+
+---
+*Personal time: 2h 38m (WhatsApp, Instagram, Email)*
+```
+
+That's the **card** format. Two others:
+
+### Standup format
+
+```markdown
+## Standup - Feb 25, 2026
+
+**What I did:**
+- Debugged VLM parsing failures by implementing raw response logging
+- Optimized video frame extraction pipeline using FFmpeg
+- Analyzed GLM-5 and Qwen-VL model performance
+- Implemented database schema migrations
+
+**Key outcomes:**
+- Resolved truncated response issues with fallback parsing
+- Identified scene detection as a latency bottleneck
+- Validated new batch extraction strategy
+
+**Next:**
+- Merge scene detection optimization branch
+- Benchmark qwen3_next model
+- Add unit tests for fallback parsing
+```
+
+Paste straight into Slack.
+
+### Narrative format
+
+```markdown
+# Session Summary: Sunday, February 22, 2026
+
+## Overview
+I spent nearly three hours optimizing the VLM inference pipeline. The main focus 
+was resolving JSON parsing errors during batch processing and benchmarking the 
+qwen3-vl:4b model against InternVL-14B. By the end, I'd identified the truncation 
+root cause, adjusted MAX_TOKENS, and validated the fix against 342 frames — 
+resulting in a 4x speedup with continuous batching.
+
+## Timeline
+* **0:00** (45m): Terminal work, running benchmark scripts
+* **45:00** (60m): Debugging JSON parsing in VS Code
+* **1:45:00** (40m): Researching model quantization
+* **2:25:00** (34m): Documenting performance metrics
+...
+```
+
+Good for retrospectives or blog drafts.
+
+---
+
+## Benchmarks
+
+Ran the full pipeline on 11 real screen recordings:
+
+| Metric | Result |
+|--------|--------|
+| Videos processed | 11 |
+| Artifacts generated | 33 (3 formats × 11 videos) |
+| Success rate | 100% |
+| Total time | 1h 41m |
+| Avg per video | **~9 min** (pipeline + all 3 formats) |
+| Hardware | MacBook Pro M4 Max, 128GB |
+
+Everything runs locally. No API keys. Nothing leaves your machine.
+
+---
+
+## Why this exists
+
+Most screen recording tools just give you a video file. If you want to remember what you did, you have to watch it back.
+
+Escribano watches it for you. It extracts frames, runs them through a vision-language model, transcribes any audio, and writes up what happened — broken into topics, with timestamps and time per activity.
+
+Built for developers: understands the difference between debugging, coding, reading docs, and scrolling Slack. Doesn't just OCR text (which produces garbage when every screen has "function" and "const" on it).
+
+---
+
+## How it works
+
+```
+Screen recording
+     │
+     ├──► Audio: Silero VAD → Whisper → transcripts
+     │
+     └──► Video: FFmpeg frames → scene detection → adaptive sampling
+                                              │
+                                              ▼
+                                    VLM inference (MLX-VLM, Qwen3-VL-2B)
+                                              │
+                                              ▼
+                                    "Debugging in terminal"
+                                    "Reading docs in Chrome"
+                                    "Coding in VS Code"
+     │
+     ▼
+Activity segmentation → temporal audio alignment → TopicBlocks
+     │
+     ▼
+LLM summary (Ollama, auto-detected) → Markdown artifact
+```
+
+Uses VLM-first visual understanding, not OCR + text clustering. OCR fails for developer work because all code screens produce similar tokens. VLMs understand the *activity*, not just the text.
+
+---
+
+## Quick Start
+
+### Prerequisites
+
+```bash
+# macOS (Homebrew)
+brew install ollama whisper-cpp ffmpeg
+
+# MLX-VLM for frame analysis (Apple Silicon)
+pip install mlx-vlm
+```
+
+### LLM Model Setup
+
+Escribano auto-detects the best model for your hardware:
+
+| Your RAM | Auto-selected | Install command |
+|----------|---------------|-----------------|
+| 16GB | `qwen3:8b` | `ollama pull qwen3:8b` |
+| 32GB | `qwen3:14b` | `ollama pull qwen3:14b` |
+| 64GB+ | `qwen3.5:27b` | `ollama pull qwen3.5:27b` |
+
+```bash
+# Minimum (16GB)
+ollama pull qwen3:8b
+
+# Or best quality (64GB+)
+ollama pull qwen3.5:27b
+```
+
+### Run
+
+```bash
+# Check prerequisites
+npx escribano doctor
+
+# Process a recording
+npx escribano --file "~/Desktop/Screen Recording.mov"
+```
+
+### Local Development
+
+```bash
+git clone https://github.com/eduardosanzb/escribano.git
+cd escribano
+pnpm install
+pnpm escribano --file "~/Desktop/Screen Recording.mov"
+```
+
+Output: `~/.escribano/artifacts/`
+
+---
+
+## CLI
+
+### Flags
+
+| Flag | What it does |
+|------|--------------|
+| `--file <path>` | Process a video file |
+| `--mic-audio <path>` | External mic audio |
+| `--system-audio <path>` | External system audio |
+| `--format <format>` | `card`, `standup`, or `narrative` (default: card) |
+| `--force` | Reprocess from scratch |
+| `--skip-summary` | Process frames only, skip artifact |
+| `--include-personal` | Include personal time (filtered by default) |
+| `--copy` | Copy to clipboard |
+| `--stdout` | Print to stdout |
+| `--help` | Show all options |
+
+### Formats
+
+| Format | Use for | Style |
+|--------|---------|-------|
+| `card` | Personal review, daily notes | Time breakdowns per subject, bullets |
+| `standup` | Daily standup, async updates | What I did / Outcomes / Next |
+| `narrative` | Retrospectives, blog drafts | Prose with timeline |
+
+### Examples
+
+```bash
+# Process and copy
+npx escribano --file "~/Desktop/Screen Recording.mov" --format standup --copy
+
+# Narrative format
+npx escribano --file session.mp4 --format narrative --force
+
+# With external audio
+npx escribano --file recording.mov --mic-audio mic.wav
+```
+
+---
+
+## Supported inputs
+
+| Source | Command |
+|--------|---------|
+| QuickTime recording | `--file video.mov` |
+| Cap recording | Auto-detected in `~/Movies/Cap/` |
+| Any MP4/MOV | `--file /path/to/video.mp4` |
+| External audio | `--mic-audio mic.wav --system-audio system.wav` |
+
+---
+
+## Architecture
+
+Clean architecture: domain entities, pure services, adapter interfaces for external systems (MLX-VLM, Ollama, Whisper, FFmpeg, SQLite).
+
+Deep dives:
+- [Why OCR fails for developers](docs/adr/005-vlm-first-visual-pipeline.md)
+- [MLX-VLM migration for 4x speedup](docs/adr/006-mlx-vlm-adapter.md)
+- [Benchmarks and learnings](docs/learnings.md)
+
+Full architecture: [docs/architecture.md](docs/architecture.md)
+
+---
+
+## Requirements
+
+- **macOS** (Apple Silicon for MLX-VLM)
+- **Node.js 20+**
+- **16GB+ RAM** (see model tiers above)
+- **~10GB disk** for models
+
+---
+
+## Roadmap
+
+- [x] VLM-first visual pipeline
+- [x] MLX-VLM migration
+- [x] Activity segmentation
+- [x] Multiple artifact formats
+- [x] Auto-detect best LLM model
+- [ ] Auto-detect ffmpeg hardware acceleration
+- [ ] OCR on keyframes for code/URLs
+- [ ] MCP server for AI assistants
+- [ ] Cross-recording queries
+
+---
+
+## License
+
+MIT
+
+---
+
+*Escribano = "The Scribe"*

package/dist/0_types.js

@@ -0,0 +1,279 @@
+/**
+ * Escribano - Core Types
+ *
+ * All types and interfaces in one place.
+ */
+import { z } from 'zod';
+// =============================================================================
+// RECORDING
+// =============================================================================
+export const recordingSchema = z.object({
+    id: z.string(),
+    source: z.object({
+        type: z.enum(['cap', 'meetily', 'raw']),
+        originalPath: z.string(),
+        metadata: z.record(z.string(), z.any()).optional(),
+    }),
+    videoPath: z.string().nullable(),
+    audioMicPath: z.string().nullable(),
+    audioSystemPath: z.string().nullable(),
+    duration: z.number(),
+    capturedAt: z.date(),
+});
+// =============================================================================
+// TRANSCRIPT
+// =============================================================================
+export const transcriptSegmentSchema = z.object({
+    id: z.string(),
+    start: z.number(),
+    end: z.number(),
+    text: z.string(),
+    speaker: z.string().nullable().optional(),
+});
+export const transcriptSchema = z.object({
+    fullText: z.string(),
+    segments: z.array(transcriptSegmentSchema),
+    language: z.string().default('en'),
+    duration: z.number(),
+});
+// =============================================================================
+// SESSION
+// =============================================================================
+// Tagged transcript to identify audio source
+export const taggedTranscriptSchema = z.object({
+    source: z.enum(['mic', 'system']),
+    transcript: transcriptSchema,
+});
+// =============================================================================
+// CLASSIFICATION
+// =============================================================================
+export const sessionTypeSchema = z.enum([
+    'meeting',
+    'debugging',
+    'tutorial',
+    'learning',
+    'working',
+]);
+export const classificationSchema = z.object({
+    meeting: z.number().min(0).max(100),
+    debugging: z.number().min(0).max(100),
+    tutorial: z.number().min(0).max(100),
+    learning: z.number().min(0).max(100),
+    working: z.number().min(0).max(100),
+});
+// =============================================================================
+// TRANSCRIPT METADATA
+// =============================================================================
+const speakerSchema = z.object({
+    name: z.string(),
+    role: z.string().optional(),
+});
+const keyMomentSchema = z.object({
+    timestamp: z.number(),
+    description: z.string(),
+    importance: z.enum(['high', 'medium', 'low']),
+});
+const actionItemSchema = z.object({
+    description: z.string(),
+    owner: z.string().nullable(),
+    priority: z.enum(['high', 'medium', 'low']).optional(),
+});
+const technicalTermSchema = z.object({
+    term: z.string(),
+    context: z.string(),
+    type: z.enum(['error', 'file', 'function', 'variable', 'other']),
+});
+const codeSnippetSchema = z.object({
+    language: z.string().optional(),
+    code: z.string(),
+    description: z.string().optional(),
+    timestamp: z.number().optional(),
+});
+export const transcriptMetadataSchema = z.object({
+    speakers: z.array(speakerSchema).optional(),
+    keyMoments: z.array(keyMomentSchema).optional(),
+    actionItems: z.array(actionItemSchema).optional(),
+    technicalTerms: z.array(technicalTermSchema).optional(),
+    codeSnippets: z.array(codeSnippetSchema).optional(),
+});
+// =============================================================================
+// ARTIFACTS
+// =============================================================================
+export const artifactTypeSchema = z.enum([
+    'summary',
+    'action-items',
+    'runbook',
+    'step-by-step',
+    'notes',
+    'code-snippets',
+    'blog-research',
+    'blog-draft',
+]);
+export const artifactSchema = z.object({
+    id: z.string(),
+    type: artifactTypeSchema,
+    content: z.string(),
+    format: z.enum(['markdown']).default('markdown'),
+    createdAt: z.date(),
+});
+// =============================================================================
+// VISUAL LOG
+// =============================================================================
+export const visualLogEntrySchema = z.object({
+    timestamp: z.number(),
+    imagePath: z.string(),
+    description: z.string().optional(),
+    ocrSummary: z.string().optional(),
+    heuristicLabel: z.string().optional(),
+    sceneScore: z.number().optional(),
+});
+export const visualLogSchema = z.object({
+    entries: z.array(visualLogEntrySchema),
+    source: z.enum(['screen', 'camera']).default('screen'),
+});
+// =============================================================================
+// VISUAL ANALYSIS (External Tool Output)
+// =============================================================================
+export const visualIndexFrameSchema = z.object({
+    index: z.number(),
+    timestamp: z.number(),
+    imagePath: z.string(),
+    ocrText: z.string(),
+    clusterId: z.number(),
+    changeScore: z.number(),
+});
+export const visualIndexClusterSchema = z.object({
+    id: z.number(),
+    heuristicLabel: z.string(),
+    timeRange: z.tuple([z.number(), z.number()]),
+    frameCount: z.number(),
+    representativeIdx: z.number(),
+    avgOcrCharacters: z.number(),
+    mediaIndicators: z.array(z.string()),
+});
+export const visualIndexSchema = z.object({
+    frames: z.array(visualIndexFrameSchema),
+    clusters: z.array(visualIndexClusterSchema),
+    processingTime: z.object({
+        ocrMs: z.number(),
+        clipMs: z.number(),
+        totalMs: z.number(),
+    }),
+});
+export const visualDescriptionSchema = z.object({
+    clusterId: z.number(),
+    timestamp: z.number(),
+    description: z.string(),
+});
+export const visualDescriptionsSchema = z.object({
+    descriptions: z.array(visualDescriptionSchema),
+    processingTime: z.object({
+        vlmMs: z.number(),
+        framesProcessed: z.number(),
+    }),
+});
+// =============================================================================
+// INTELLIGENCE & EMBEDDINGS
+// =============================================================================
+export const activityContextSchema = z.object({
+    type: z.enum(['url', 'file', 'app', 'topic', 'unknown']),
+    value: z.string(),
+    confidence: z.number().min(0).max(1),
+});
+export const sessionSegmentSchema = z.object({
+    id: z.string(),
+    timeRange: z.tuple([z.number(), z.number()]),
+    visualClusterIds: z.array(z.number()),
+    contexts: z.array(activityContextSchema),
+    transcriptSlice: taggedTranscriptSchema.nullable(),
+    classification: classificationSchema.nullable(),
+    isNoise: z.boolean(),
+});
+export const embeddingConfigSchema = z.object({
+    model: z.string().default('qwen3-embedding:8b'),
+    similarityThreshold: z.number().min(0).max(1).default(0.4),
+});
+// =============================================================================
+// SESSION
+// =============================================================================
+export const outlineSyncStateSchema = z.object({
+    collectionId: z.string(),
+    sessionDocumentId: z.string(),
+    sessionDocumentUrl: z.string(),
+    artifacts: z.array(z.object({
+        type: artifactTypeSchema,
+        documentId: z.string(),
+        documentUrl: z.string(),
+        syncedAt: z.date(),
+        contentHash: z.string(),
+    })),
+    lastSyncedAt: z.date(),
+});
+export const sessionSchema = z.object({
+    id: z.string(),
+    recording: recordingSchema,
+    transcripts: z.array(taggedTranscriptSchema),
+    visualLogs: z.array(visualLogSchema).default([]),
+    segments: z.array(sessionSegmentSchema).default([]),
+    status: z.enum([
+        'raw',
+        'transcribed',
+        'visual-logged',
+        'classified',
+        'metadata-extracted',
+        'complete',
+        'error',
+    ]),
+    classification: classificationSchema.nullable(),
+    metadata: transcriptMetadataSchema.nullable(),
+    artifacts: z.array(artifactSchema).default([]),
+    outlineSyncState: outlineSyncStateSchema.nullable().optional(),
+    createdAt: z.date(),
+    updatedAt: z.date(),
+    errorMessage: z.string().nullable().optional(),
+});
+// =============================================================================
+// CONFIG
+// =============================================================================
+export const capConfigSchema = z.object({
+    recordingsPath: z
+        .string()
+        .default('~/Library/Application Support/so.cap.desktop/recordings'),
+});
+export const whisperConfigSchema = z.object({
+    binaryPath: z.string().default('whisper-cli'),
+    model: z.string().default('large-v3'),
+    cwd: z.string().optional(),
+    outputFormat: z.enum(['json', 'txt', 'srt', 'vtt']).default('json'),
+    language: z.string().optional(),
+});
+export const intelligenceConfigSchema = z.object({
+    provider: z.enum(['ollama', 'mlx']).default('ollama'),
+    endpoint: z.string().default('http://localhost:11434/api/chat'),
+    model: z.string().default('qwen3.5:27b'),
+    generationModel: z.string().default('qwen3.5:27b'),
+    visionModel: z.string().default('minicpm-v:8b'),
+    maxRetries: z.number().default(3),
+    timeout: z.number().default(600000), // 10 minutes
+    keepAlive: z.string().default('10m'),
+    maxContextSize: z.number().default(131072), // qwen3:8b supports up to 128K
+    embedding: embeddingConfigSchema.default({
+        model: 'nomic-embed-text',
+        similarityThreshold: 0.75,
+    }),
+    // MLX-VLM specific config
+    vlmBatchSize: z.number().default(4),
+    vlmMaxTokens: z.number().default(2000),
+    mlxSocketPath: z.string().default('/tmp/escribano-mlx.sock'),
+});
+export const DEFAULT_INTELLIGENCE_CONFIG = intelligenceConfigSchema.parse({});
+const artifactConfigSchema = z.object({
+    parallelGeneration: z.boolean().default(false),
+    maxParallel: z.number().default(3),
+    maxScreenshots: z.number().default(10),
+});
+export const outlineConfigSchema = z.object({
+    url: z.string().url(),
+    token: z.string(),
+    collectionName: z.string().default('Escribano Sessions'),
+});

package/dist/actions/classify-session.js

@@ -0,0 +1,77 @@
+/**
+ * Escribano - Classify Session Action
+ *
+ * Classifies a session using IntelligenceService.
+ * If segments exist, it classifies each segment individually and aggregates the results.
+ */
+import { Session } from '../domain/session.js';
+import { Transcript } from '../domain/transcript.js';
+export async function classifySession(session, intelligence) {
+    if (session.transcripts.length === 0) {
+        throw new Error('Cannot classify session without transcripts');
+    }
+    // 1. If segments exist, classify each segment sequentially
+    if (session.segments.length > 0) {
+        const nonNoiseSegments = session.segments.filter((s) => !s.isNoise);
+        const noiseCount = session.segments.length - nonNoiseSegments.length;
+        console.log(`Classifying ${nonNoiseSegments.length} segments (${noiseCount} noise skipped)...`);
+        // Sequential classification to avoid parallel warmup race + Ollama overload
+        const classifiedSegments = [];
+        let classifiedCount = 0;
+        for (const segment of session.segments) {
+            if (segment.isNoise) {
+                classifiedSegments.push({
+                    ...segment,
+                    classification: {
+                        meeting: 0,
+                        debugging: 0,
+                        tutorial: 0,
+                        learning: 0,
+                        working: 0,
+                    },
+                });
+                continue;
+            }
+            classifiedCount++;
+            console.log(`  Segment ${classifiedCount}/${nonNoiseSegments.length}: ${segment.id}`);
+            try {
+                const classification = await intelligence.classifySegment(segment);
+                classifiedSegments.push({ ...segment, classification });
+            }
+            catch (error) {
+                console.error(`  Failed to classify segment ${segment.id}:`, error);
+                classifiedSegments.push(segment);
+            }
+        }
+        // Update session with classified segments
+        const updatedSession = {
+            ...session,
+            segments: classifiedSegments,
+            status: 'classified',
+            updatedAt: new Date(),
+        };
+        // 2. Derive session-level classification from aggregated segments
+        const aggregatedClassification = Session.getActivityBreakdown(updatedSession);
+        // Convert Record to Classification type
+        updatedSession.classification = {
+            meeting: aggregatedClassification.meeting || 0,
+            debugging: aggregatedClassification.debugging || 0,
+            tutorial: aggregatedClassification.tutorial || 0,
+            learning: aggregatedClassification.learning || 0,
+            working: aggregatedClassification.working || 0,
+        };
+        return updatedSession;
+    }
+    // Fallback to legacy whole-session classification if no segments exist
+    console.log('No segments found, falling back to session-level classification...');
+    const transcriptForClassification = session.transcripts.length === 1
+        ? session.transcripts[0].transcript
+        : Transcript.interleave(session.transcripts);
+    const classification = await intelligence.classify(transcriptForClassification, session.visualLogs);
+    return {
+        ...session,
+        status: 'classified',
+        classification,
+        updatedAt: new Date(),
+    };
+}