@tyvm/knowhow 0.0.158 → 0.0.160

package/PROCESSING.md

@@ -1,450 +0,0 @@
-# Media Processing with Knowhow
-
-This guide covers how to process audio, video, and PDF files using Knowhow's configuration system. Knowhow automatically converts media files to text for AI processing, supports pipeline chaining where outputs become inputs, and allows assignment of processing tasks to specific agents.
-
----
-
-## Supported File Types
-
-Knowhow's `convertToText()` function automatically handles these file types:
-
-**Audio Files:**
-- `.mp3` - MP3 audio files
-- `.wav` - WAV audio files  
-- `.m4a` - M4A audio files
-- `.mpga` - MPEG audio files
-
-**Video Files:**
-- `.mp4` - MP4 video files
-- `.webm` - WebM video files
-- `.mov` - QuickTime video files
-- `.mpeg` - MPEG video files
-
-**Document Files:**
-- `.pdf` - PDF documents
-
-All other file types are processed as plain text files.
-
----
-
-## Audio Processing
-
-### Basic Audio Transcription
-
-```json
-{
-  "sources": [
-    {
-      "input": "./recordings/**/*.mp3",
-      "output": "./transcripts/",
-      "prompt": "BasicTranscript"
-    }
-  ]
-}
-```
-
-Audio files are automatically:
-1. Split into 30-second chunks by default (configurable via `chunkTime` parameter)
-2. Transcribed using speech-to-text via `Downloader.transcribeChunks()`
-3. Saved as `transcript.json` in a folder named after the audio file
-4. Formatted with timestamps: `[0:30s] transcript text [30:60s] more text`
-
-### Audio with Agent Assignment
-
-```json
-{
-  "sources": [
-    {
-      "input": "./meetings/**/*.wav",
-      "output": "./meeting-notes/",
-      "prompt": "MeetingNotes",
-      "agent": "patcher"
-    }
-  ]
-}
-```
-
-### Meeting Recording Pipeline
-
-```json
-{
-  "sources": [
-    {
-      "input": "./meetings/**/*.m4a",
-      "output": "./meetings/transcripts/",
-      "prompt": "MeetingTranscriber"
-    },
-    {
-      "input": "./meetings/transcripts/**/*.mdx",
-      "output": "./meetings/summaries/",
-      "prompt": "MeetingSummarizer"
-    },
-    {
-      "input": "./meetings/summaries/**/*.mdx", 
-      "output": "./meetings/action-items.txt",
-      "prompt": "ActionItemExtractor"
-    }
-  ]
-}
-```
-
-This pipeline demonstrates chaining:
-1. Audio files → transcripts (multi-output to directory)
-2. Transcripts → summaries (multi-output to directory)  
-3. Summaries → single action items file (single output)
-
----
-
-## Video Processing
-
-### Basic Video Processing
-
-```json
-{
-  "sources": [
-    {
-      "input": "./videos/**/*.mp4",
-      "output": "./video-analysis/",
-      "prompt": "VideoAnalyzer"
-    }
-  ]
-}
-```
-
-Video files are processed by:
-1. Extracting and transcribing audio (same process as audio files)
-2. Extracting keyframes at regular intervals using `Downloader.extractKeyframes()`
-3. Analyzing visual content of each keyframe
-4. Combining transcript and visual analysis with timestamps
-
-### Keyframe Extraction Interval
-
-The `interval` parameter controls how often (in seconds) a frame is sampled from the video during keyframe extraction. It is accepted by both `streamKeyFrameExtraction()` and `extractKeyframes()` with a default of **10 seconds**.
-
-| Value | Effect |
-|-------|--------|
-| `5`   | Sample a frame every 5 seconds — more frames, finer detail |
-| `10`  | Default — one frame every 10 seconds, balanced coverage |
-| `30`  | One frame every 30 seconds — fewer frames, lower cost |
-
-**Example usage (programmatic):**
-```typescript
-// Extract a keyframe every 5 seconds
-await mediaProcessorService.extractKeyframes(filePath, outputPath, true, 5);
-
-// Stream keyframes every 30 seconds
-for await (const frame of mediaProcessorService.streamKeyFrameExtraction(filePath, videoJsonPath, true, 30)) {
-  console.log(frame);
-}
-```
-
-The actual output format from `convertVideoToText()` includes:
-```
-Chunk: (1/10):
-Start Timestamp: [0s]
-Visual: description of keyframe
-Audio: transcribed audio
-End Timestamp: [30s]
-```
-
-### Video Content Organization
-
-```json
-{
-  "sources": [
-    {
-      "input": "./raw-videos/**/*.webm",
-      "output": "./organized-videos/",
-      "prompt": "VideoOrganizer",
-      "agent": "patcher"
-    }
-  ]
-}
-```
-
-This example shows assigning video processing to the "patcher" agent.
-
----
-
-## PDF Processing
-
-### Document Analysis
-
-```json
-{
-  "sources": [
-    {
-      "input": "./documents/**/*.pdf",
-      "output": "./document-summaries/",
-      "prompt": "DocumentSummarizer"
-    }
-  ]
-}
-```
-
-PDF files are processed by:
-1. Reading the file using `fs.readFileSync()`
-2. Extracting text content using the `pdf-parse` library
-3. Returning the extracted text via `data.text`
-4. Applying the specified prompt for analysis
-
-### Multi-Document Research Pipeline
-
-```json
-{
-  "sources": [
-    {
-      "input": "./research-papers/**/*.pdf",
-      "output": "./paper-summaries/",
-      "prompt": "AcademicSummarizer"
-    },
-    {
-      "input": "./paper-summaries/**/*.mdx",
-      "output": "./research-synthesis.md",
-      "prompt": "ResearchSynthesizer"
-    }
-  ]
-}
-```
-
----
-
-## Agent Assignment
-
-### Assigning Processing to Specific Agents
-
-```json
-{
-  "sources": [
-    {
-      "input": "./meetings/**/*.mov",
-      "output": "./meeting-notes/",
-      "prompt": "MeetingNotes",
-      "agent": "patcher"
-    }
-  ]
-}
-```
-
-The `agent` parameter assigns processing to a specific agent defined in your configuration. The agent receives the file content and prompt, then processes it according to its instructions and capabilities.
-
-### Multi-Agent Pipeline
-
-```json
-{
-  "sources": [
-    {
-      "input": "./interviews/**/*.wav",
-      "output": "./interview-transcripts/",
-      "prompt": "InterviewTranscriber",
-      "agent": "transcriber"
-    },
-    {
-      "input": "./interview-transcripts/**/*.mdx",
-      "output": "./interview-insights/",
-      "prompt": "InsightExtractor", 
-      "agent": "analyst"
-    }
-  ]
-}
-```
-
-This assigns transcription to a "transcriber" agent and analysis to an "analyst" agent, each specialized for their respective tasks.
-
----
-
-## Embedding Generation from Processed Media
-
-### Audio Embeddings
-
-```json
-{
-  "embedSources": [
-    {
-      "input": "./podcasts/**/*.mp3",
-      "output": ".knowhow/embeddings/podcasts.json",
-      "chunkSize": 2000,
-      "prompt": "PodcastEmbeddingExplainer"
-    }
-  ]
-}
-```
-
-Audio files are automatically transcribed using `convertAudioToText()`, then chunked and embedded for semantic search.
-
-### Video Embeddings
-
-```json
-{
-  "embedSources": [
-    {
-      "input": "./tutorials/**/*.mp4", 
-      "output": ".knowhow/embeddings/tutorials.json",
-      "chunkSize": 1500
-    }
-  ]
-}
-```
-
-Video files are processed using `convertVideoToText()` to extract both visual and audio content, then embedded for search.
-
-### PDF Document Embeddings
-
-```json
-{
-  "embedSources": [
-    {
-      "input": "./documentation/**/*.pdf",
-      "output": ".knowhow/embeddings/docs.json", 
-      "chunkSize": 2000,
-      "prompt": "DocumentChunker"
-    }
-  ]
-}
-```
-
----
-
-## Pipeline Chaining Examples
-
-### Complete Media Processing Workflow
-
-```json
-{
-  "sources": [
-    {
-      "input": "./raw-content/**/*.{mp4,mp3,pdf}",
-      "output": "./processed-content/",
-      "prompt": "ContentProcessor"
-    },
-    {
-      "input": "./processed-content/**/*.mdx",
-      "output": "./content-categories/",
-      "prompt": "ContentCategorizer"
-    },
-    {
-      "input": "./content-categories/**/*.mdx",
-      "output": "./final-report.md",
-      "prompt": "ReportGenerator"
-    }
-  ]
-}
-```
-
-This three-stage pipeline:
-1. Processes mixed media files (video, audio, PDF) into text summaries
-2. Categorizes the processed content  
-3. Generates a final consolidated report
-
-### Meeting-to-Tasks Pipeline
-
-```json
-{
-  "sources": [
-    {
-      "input": "./weekly-meetings/**/*.mov",
-      "output": "./meeting-transcripts/",
-      "prompt": "MeetingTranscriber"
-    },
-    {
-      "input": "./meeting-transcripts/**/*.mdx", 
-      "output": "./extracted-tasks/",
-      "prompt": "TaskExtractor"
-    },
-    {
-      "input": "./extracted-tasks/**/*.mdx",
-      "output": "./project-tasks.json",
-      "prompt": "TaskConsolidator",
-      "agent": "patcher"
-    }
-  ]
-}
-```
-
----
-
-## Processing Workflow Details
-
-### Transcript Caching and Reuse
-
-The system implements intelligent caching for audio/video processing:
-
-**Audio Processing (`processAudio()`):**
-- Transcripts saved as `transcript.json` in `{dir}/{filename}/transcript.json`
-- Uses `reusePreviousTranscript` parameter (default: true)
-- Checks if transcript exists with `fileExists()` before reprocessing
-- Enables fast re-processing with different prompts
-
-**Video Processing (`processVideo()`):**
-- Audio transcripts cached the same way as audio files
-- Video analysis cached as `video.json` in `{dir}/{filename}/video.json`
-- Keyframes extracted using `Downloader.extractKeyframes()`
-
-### Chunking Behavior
-
-**Audio Files:**
-- Default chunk time: 30 seconds (configurable via `chunkTime` parameter)
-- Uses `Downloader.chunk()` to split audio files
-- Each chunk transcribed separately then combined with timestamps
-
-**Video Files:**
-- Same 30-second default chunking for audio track
-- Keyframes extracted at chunk intervals
-- Visual and audio analysis combined per chunk
-
-### Output Structure
-
-**Multi-Output (directory ending with `/`):**
-- Creates one output file per input file
-- Preserves directory structure relative to input pattern
-- Uses `outputExt` (default: "mdx") for file extensions
-- Uses `outputName` or original filename
-
-**Single Output (specific filename):**
-- Combines all input files into one output
-- Useful for reports, summaries, and consolidated documents
-
----
-
-## Troubleshooting
-
-### Common Issues
-
-**Audio/Video Processing Fails:**
-- Ensure ffmpeg is installed and accessible (required by Downloader)
-- Check file permissions and disk space
-- Verify audio/video files aren't corrupted
-- Check that `Downloader.chunk()` and `Downloader.transcribeChunks()` are working
-
-**PDF Processing Fails:**
-- Some PDFs may have restrictions or encryption
-- Scanned PDFs without OCR won't extract text properly with `pdf-parse`
-- Large PDFs may cause memory issues when loading with `fs.readFileSync()`
-- Ensure PDF file isn't corrupted
-
-**Pipeline Chaining Issues:**
-- Ensure output directory of one stage matches input pattern of next stage
-- Check that intermediate files are being created successfully
-- Verify file extensions match between stages (default: .mdx)
-
-**Agent Assignment Problems:**
-- Ensure the specified agent exists in your configuration
-- Check that the agent has appropriate permissions and tools
-- Verify the agent can handle the specific prompt and content type
-
-### Performance Optimization
-
-**Large File Processing:**
-- Audio/video files are automatically chunked (30s default) for efficient processing
-- Consider adjusting `chunkTime` parameter for very long recordings
-- Transcript caching avoids reprocessing unchanged files
-
-**Batch Processing:**
-- Process files in smaller batches if memory becomes an issue
-- Use specific file patterns rather than overly broad glob patterns
-- Consider processing different file types in separate pipeline stages
-
-**File System Considerations:**
-- Transcripts create subdirectories: `{filename}/transcript.json`
-- Video processing creates: `{filename}/video.json`
-- Ensure sufficient disk space for intermediate files

package/WORKER.md

@@ -1,260 +0,0 @@
-# `knowhow worker` — what it is
-
-The `knowhow worker` command starts a **Knowhow Worker process** that exposes an **MCP (Model Context Protocol) server** backed by a curated set of **tools**.
-
-At a high level (from `src/worker.ts` + CLI wiring):
-
-1. Loads `knowhow.json` via `getConfig()`.
-2. Optionally runs **passkey setup / reset** flows (browser-based), then exits.
-3. Chooses an execution mode:
-   - **Host mode** (default), or
-   - **Docker sandbox mode** (`--sandbox`)
-4. In host mode, it connects to Knowhow’s API over **WebSockets** and serves MCP tools.
-5. In sandbox mode, it verifies Docker, rebuilds the worker image, then runs the worker inside Docker and forwards logs.
-
-> The CLI entry point (e.g. in `src/cli.ts`) mainly defines the subcommand surface and ensures shared services are initialized; the runtime behavior and config handling are implemented in `src/worker.ts`.
-
----
-
-## What the worker exposes
-
-In **host mode**, the worker:
-- Starts an **MCP server** (`clientName = "knowhow-worker"`, `clientVersion = "1.1.1"`).
-- Registers the enabled tool set (see `worker.allowedTools` below).
-- Connects to the Knowhow backend WebSocket endpoint(s) so the backend can drive tool usage.
-- Reconnects continuously if the WebSocket connection drops.
-
-Key connection endpoints (host mode):
-- Worker WS: `${API_URL}/ws/worker`
-- Optional tunnel WS: `${API_URL}/ws/tunnel`
-
-WebSocket headers include:
-- `Authorization: Bearer <jwt>` (from `loadJwt()`)
-- `User-Agent: knowhow-worker/1.1.1/<hostname>`
-- `Root: <computed from WORKER_ROOT or cwd>`
-- Optional share visibility:
-  - `Shared: "true"` with `--share`
-  - `Shared: "false"` with `--unshare`
-  - no `Shared` header if neither flag is provided
-
----
-
-# CLI arguments supported by `knowhow worker`
-
-These flags are handled by the exported worker launcher in `src/worker.ts` (and exposed by the `worker` subcommand wiring).
-
-| Flag | Type | Effect |
-|------|------|--------|
-| `--register` | boolean | Registers the current directory as a worker path (`registerWorkerPath(process.cwd())`) and then exits. |
-| `--share` | boolean | Sets `Shared: "true"` header when connecting to the API (marks worker as shared). |
-| `--unshare` | boolean | Sets `Shared: "false"` header when connecting to the API (marks worker as private/unshared). |
-| `--sandbox` | boolean | Runs worker in Docker sandbox mode; persists `worker.sandbox = true` into config. |
-| `--no-sandbox` | boolean | Forces host mode; persists `worker.sandbox = false` into config. |
-| `--passkey` | boolean | Runs browser-based passkey setup (`PasskeySetupService().setup(jwt)`) and exits. |
-| `--passkey-reset` | boolean | Clears stored passkey credentials (`PasskeySetupService().reset()`) and exits. |
-
-## Sandbox selection priority (important)
-
-When not already inside Docker, sandbox mode is selected by this priority:
-
-1. `--sandbox` / `--no-sandbox` CLI flags (highest priority)
-2. `knowhow.json` config: `config.worker.sandbox`
-3. Default: `false` (host mode)
-
-### Special case: already inside Docker
-If `process.env.KNOWHOW_DOCKER === "true"`, the worker forces:
-- `options.sandbox = false`
-- `options.noSandbox = true`
-
-This prevents nested Docker execution.
-
----
-
-# `knowhow.json` config options supported (as used by `src/worker.ts`)
-
-The worker primarily reads/writes under `config.worker.*`.
-
-## 1) `worker.sandbox` (boolean)
-Controls which mode runs **when no CLI sandbox flags are provided**.
-
-- Updated by:
-  - `--sandbox` → `worker.sandbox = true`
-  - `--no-sandbox` → `worker.sandbox = false`
-
----
-
-## 2) `worker.allowedTools` (string[])
-Controls which tools are exposed to the MCP server.
-
-Behavior:
-- If missing, the worker:
-  1. populates it with `Tools.getToolNames()`
-  2. saves it back to `knowhow.json`
-  3. then exits early (so you can edit/curate the allowlist)
-
-So: first run creates an allowlist; subsequent runs enforce it.
-
----
-
-## 3) Passkey auth: `worker.auth.passkey`
-If configured, the worker starts in a **locked** state and wraps tool execution so tools fail unless unlocked.
-
-The code implies an expected shape like:
-
-```ts
-config.worker.auth.passkey.publicKey
-config.worker.auth.passkey.credentialId
-config.worker.auth.sessionDurationHours // optional; defaults to ~3
-```
-
-Effects when enabled:
-- `WorkerPasskeyAuthService` is created.
-- Tool functions are wrapped:
-  - When locked: return an error object like `WORKER_LOCKED` / “unlock first”.
-  - When unlocked: execute the original tool function.
-- The worker registers additional auth tools:
-  - `unlock` (`makeUnlockTool`)
-  - `lock` (`makeLockTool`)
-
-> Exact error message/response format depends on the auth service implementation, but the lock/unlock gating is explicit.
-
----
-
-## 4) Tunnel configuration: `worker.tunnel.*`
-If enabled, the worker also opens a second WebSocket (`/ws/tunnel`) so the backend can reach allowed local services.
-
-### `worker.tunnel.enabled` (boolean)
-- `true` → tunnel WS created
-- `false` / missing → tunneling disabled
-
-### `worker.tunnel.allowedPorts` (number[])
-- Restricts which ports can be used through the tunnel.
-- If tunnel is enabled but this list is empty, the worker logs a warning.
-
-### `worker.tunnel.localHost` (string, optional)
-Hostname for reaching local services.
-
-- If not set:
-  - inside Docker (`KNOWHOW_DOCKER === "true"`): `host.docker.internal`
-  - otherwise: `127.0.0.1`
-
-### `worker.tunnel.portMapping` (object/map)
-The code treats it as a mapping:
-
-- `config.worker.tunnel.portMapping[containerPort] = hostPort`
-
-It logs these mappings and passes them to the tunnel handler.
-
-### `worker.tunnel.maxConcurrentStreams` (number, optional)
-- Passed to tunnel config
-- Default used in `src/worker.ts`: `50`
-
-### `worker.tunnel.enableUrlRewriting` (boolean, optional)
-- Passed to tunnel config as:
-  - `config.worker.tunnel.enableUrlRewriting !== false`
-- So URL rewriting defaults to **enabled** unless explicitly set to `false`.
-
----
-
-# Execution modes (host vs Docker sandbox)
-
-## Host mode (default)
-Key responsibilities:
-- Connect to `${API_URL}/ws/worker` using the JWT and headers (including optional `Shared`).
-- Optionally connect to `${API_URL}/ws/tunnel`.
-- Serve MCP tools via `mcpServer.createServer(...).withTools(toolsToUse)`.
-- If the worker WS reconnects:
-  - it resets MCP server state and re-registers the tool set.
-
-## Docker sandbox mode (`--sandbox`)
-When sandbox mode is selected:
-1. Check Docker availability.
-2. Build the worker Docker image.
-3. Load JWT.
-4. Run the worker in Docker (`Docker.runWorkerContainer(...)`) with:
-   - `workspaceDir: process.cwd()`
-   - `jwt`
-   - `apiUrl: API_URL`
-   - `config` passed into container
-   - `share` / `unshare` options
-5. Forward container logs and stop the container on exit.
-
----
-
-# Summary checklist
-
-- **Use `knowhow worker` to**: run a Knowhow Worker that exposes an MCP server and a filtered tool allowlist.
-- **Common CLI flags**:
-  - `--register`, `--share`, `--unshare`
-  - `--sandbox` / `--no-sandbox`
-  - `--passkey` / `--passkey-reset`
-- **Key config fields**:
-  - `worker.sandbox`
-  - `worker.allowedTools` (auto-populated on first run if missing)
-  - `worker.auth.passkey.*` (enables lock/unlock tool gating)
-  - `worker.tunnel.enabled`, `worker.tunnel.allowedPorts`, `worker.tunnel.localHost`, `worker.tunnel.portMapping`, `worker.tunnel.maxConcurrentStreams`, `worker.tunnel.enableUrlRewriting`
-```
-
-If you paste `src/worker.ts` (or the exact relevant portions) and `src/cli.ts`, I can tighten any remaining ambiguities (e.g., exact auth config field names and the precise tool/error response shapes).
-# Knowhow Workers Guide
-
-Knowhow workers allow you to expose your local tools and development environment to the Knowhow platform, enabling remote agents to execute commands, access files, and leverage your local setup through a secure WebSocket connection.
-
----
-
-## Quick Start
-
-### 1. Initial Setup
-
-First, authenticate with the Knowhow platform:
-
-```bash
-knowhow login
-```
-
-This will open your browser and guide you through the authentication process.
-
-### 2. Generate Worker Configuration
-
-Run the worker command to generate the initial configuration:
-
-```bash
-knowhow worker
-```
-
-This will:
-- Generate a `worker` configuration block in your `knowhow.json`
-- Display available tools that can be exposed
-- Create a secure connection to the Knowhow platform
-
-### 3. Configure Allowed Tools
-
-Edit your `knowhow.json` to specify which tools you want to expose:
-
-```json
-{
-  "worker": {
-    "allowedTools": [
-      "readFile",
-      "writeFileChunk",
-      "patchFile",
-      "execCommand",
-      "textSearch",
-      "fileSearch",
-      "embeddingSearch"
-    ]
-  }
-}
-```
-
-### 4. Start the Worker
-
-Run the worker again to start the connection:
-
-```bash
-knowhow worker
-```
-
-Your local tools are now available to Knowhow behaviors and agents!
-
-