@framers/agentos-skills-registry 0.13.0 → 0.15.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@framers/agentos-skills-registry",
-  "version": "0.13.0",
+  "version": "0.15.0",
   "files": [
     "dist",
     "registry",

package/registry/curated/audio-generation/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: audio-generation
+version: '1.0.0'
+description: Music and sound effects generation — 8 providers with fallback chains, user-configurable preferences, local and cloud options.
+author: Wunderland
+namespace: wunderland
+category: media
+tags: [audio, music, sound-effects, sfx, generation, suno, elevenlabs, stable-audio, musicgen]
+requires_secrets: []
+requires_tools: []
+metadata:
+  agentos:
+    emoji: "\U0001F3B5"
+---
+
+# Audio Generation (Music & Sound Effects)
+
+Use this skill when the user wants to generate music compositions or sound effects from text descriptions. The system supports 8 provider backends with automatic fallback chains and user-configurable provider preferences.
+
+This skill covers two complementary APIs:
+
+1. **generateMusic()** — Full-length musical compositions from text prompts
+2. **generateSFX()** — Short sound effects from text descriptions
+
+## Music Generation
+
+### Basic Usage
+
+Generate music from a text prompt. The system auto-detects the best available provider from environment variables in priority order: `SUNO_API_KEY` (highest quality) -> `UDIO_API_KEY` -> `STABILITY_API_KEY` -> `REPLICATE_API_TOKEN` -> `FAL_API_KEY` -> local MusicGen (no key required).
+
+```typescript
+import { generateMusic } from 'agentos';
+
+const result = await generateMusic({
+  prompt: 'Upbeat lo-fi hip hop beat with vinyl crackle and mellow piano',
+  durationSec: 60,
+});
+console.log(result.audio[0].url);
+```
+
+### Prompt Tips for Music
+
+- **Specify genre and mood first**: "melancholic jazz ballad", "aggressive drum and bass", "peaceful ambient soundscape"
+- **Include instrumentation**: "with acoustic guitar, soft brushed drums, and upright bass"
+- **Mention tempo and energy**: "slow tempo, 70 BPM", "high energy, driving rhythm"
+- **Add texture and production**: "lo-fi with vinyl crackle", "clean studio recording", "reverb-heavy shoegaze"
+- **Reference eras or styles**: "1970s progressive rock", "modern trap production", "classical baroque harpsichord"
+- **Use negative prompts** where supported: `negativePrompt: 'vocals, singing, lyrics'`
+
+### Music Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `prompt` | (required) | Text description of the desired music |
+| `provider` | auto-detect | Provider ID (`"suno"`, `"udio"`, `"stable-audio"`, etc.) |
+| `model` | provider default | Model identifier within the provider |
+| `durationSec` | provider default | Output duration in seconds (Suno: up to ~240s, Stable Audio: ~47s) |
+| `negativePrompt` | — | Musical elements to avoid (not all providers support this) |
+| `outputFormat` | `"mp3"` | Output format: `"mp3"`, `"wav"`, `"flac"`, `"ogg"`, `"aac"` |
+| `seed` | random | Seed for reproducible output (provider-dependent) |
+| `n` | `1` | Number of clips to generate |
+
+## Sound Effect Generation
+
+### Basic Usage
+
+Generate a sound effect from a text description. The SFX detection order is: `ELEVENLABS_API_KEY` (highest quality) -> `STABILITY_API_KEY` -> `REPLICATE_API_TOKEN` -> `FAL_API_KEY` -> local AudioGen (no key required).
+
+```typescript
+import { generateSFX } from 'agentos';
+
+const result = await generateSFX({
+  prompt: 'Thunder crack followed by heavy rain on a tin roof',
+  durationSec: 5,
+});
+console.log(result.audio[0].url);
+```
+
+### Prompt Tips for Sound Effects
+
+- **Be specific about the sound**: "glass bottle shattering on concrete floor" rather than just "glass breaking"
+- **Describe the environment**: "footsteps on gravel in an empty parking garage with echo"
+- **Layer multiple sounds**: "busy city intersection with car horns, distant sirens, and pedestrian chatter"
+- **Specify duration context**: short stingers (1-3s) vs ambient loops (10-15s)
+- **Include physical properties**: "heavy wooden door creaking open slowly", "small metallic click of a light switch"
+
+### SFX Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `prompt` | (required) | Text description of the desired sound effect |
+| `provider` | auto-detect | Provider ID (`"elevenlabs-sfx"`, `"stable-audio"`, etc.) |
+| `model` | provider default | Model identifier within the provider |
+| `durationSec` | provider default | Output duration in seconds (SFX is typically 1-15s) |
+| `outputFormat` | `"mp3"` | Output format: `"mp3"`, `"wav"`, `"flac"`, `"ogg"`, `"aac"` |
+| `seed` | random | Seed for reproducible output (provider-dependent) |
+| `n` | `1` | Number of clips to generate |
+
+## Provider Selection Guide
+
+### Music Providers
+
+| Provider | ID | Best For | Env Var | Key Required |
+|----------|-----|----------|---------|-------------|
+| **Suno** | `suno` | Highest quality vocals + instrumentals, full songs | `SUNO_API_KEY` | Yes |
+| **Udio** | `udio` | High quality music, alternative to Suno | `UDIO_API_KEY` | Yes |
+| **Stable Audio** | `stable-audio` | Instrumentals, loops, ambient, fast generation | `STABILITY_API_KEY` | Yes |
+| **Replicate** | `replicate-audio` | Open-source models (MusicGen), pay-per-use | `REPLICATE_API_TOKEN` | Yes |
+| **Fal** | `fal-audio` | Fast serverless GPU, cost-effective | `FAL_API_KEY` | Yes |
+| **MusicGen Local** | `musicgen-local` | Offline generation, no API key needed, privacy | — | No |
+
+### SFX Providers
+
+| Provider | ID | Best For | Env Var | Key Required |
+|----------|-----|----------|---------|-------------|
+| **ElevenLabs** | `elevenlabs-sfx` | Highest quality SFX, fast turnaround | `ELEVENLABS_API_KEY` | Yes |
+| **Stable Audio** | `stable-audio` | Good SFX + music in one provider | `STABILITY_API_KEY` | Yes |
+| **Replicate** | `replicate-audio` | Open-source AudioGen model, pay-per-use | `REPLICATE_API_TOKEN` | Yes |
+| **Fal** | `fal-audio` | Fast serverless GPU | `FAL_API_KEY` | Yes |
+| **AudioGen Local** | `audiogen-local` | Offline SFX generation, no API key needed | — | No |
+
+### Forcing a Specific Provider
+
+```typescript
+const result = await generateMusic({
+  prompt: 'Chill synthwave with arpeggiated synths',
+  provider: 'stable-audio',
+  apiKey: 'your-stability-key',
+  durationSec: 30,
+});
+```
+
+## Provider Preferences
+
+Use `providerPreferences` to control the fallback chain ordering and filtering without hardcoding a single provider. This is useful for load balancing, cost optimization, or respecting user preferences.
+
+```typescript
+import { generateMusic } from 'agentos';
+
+// Prefer Suno, fall back to Stable Audio, never use Udio
+const result = await generateMusic({
+  prompt: 'Orchestral film score with dramatic strings',
+  providerPreferences: {
+    preferred: ['suno', 'stable-audio'],
+    blocked: ['udio'],
+  },
+});
+```
+
+### Preference Fields
+
+| Field | Description |
+|-------|-------------|
+| `preferred` | Ordered list of provider IDs to try first. Providers not in this list are excluded. |
+| `blocked` | Provider IDs to unconditionally exclude from the chain. |
+| `weights` | Weight map for weighted random selection (useful for A/B testing or load balancing). |
+
+Provider preferences work identically across `generateMusic()`, `generateSFX()`, `generateImage()`, and `generateVideo()`.
+
+## When to Use Music vs SFX vs TTS
+
+| Need | API | Why |
+|------|-----|-----|
+| Background music, songs, jingles | `generateMusic()` | Optimized for musical compositions with melody, harmony, rhythm |
+| Sound effects, foley, ambient sounds | `generateSFX()` | Optimized for short, non-musical audio (impacts, nature, UI sounds) |
+| Speech, narration, voice cloning | TTS (speech subsystem) | Use the speech/TTS APIs instead — audio generation is for non-speech |
+| Podcast intros with music + voice | Combine both | Generate music with `generateMusic()`, speech with TTS, mix externally |
+
+## Combining Audio
+
+The audio generation APIs return URLs or base64 data that can be combined in downstream workflows:
+
+1. **Generate background music**: `generateMusic({ prompt: 'Gentle ambient pad' })`
+2. **Generate SFX stingers**: `generateSFX({ prompt: 'Notification chime' })`
+3. **Generate speech**: Use the TTS subsystem for narration
+4. **Mix**: Use ffmpeg or a Web Audio API pipeline to layer the tracks
+
+```typescript
+import { generateMusic, generateSFX } from 'agentos';
+
+// Generate assets in parallel
+const [music, sfx] = await Promise.all([
+  generateMusic({ prompt: 'Calm podcast background music', durationSec: 120 }),
+  generateSFX({ prompt: 'Soft transition whoosh', durationSec: 2 }),
+]);
+
+// Use the URLs/base64 data in your mixing pipeline
+console.log('Music:', music.audio[0].url);
+console.log('SFX:', sfx.audio[0].url);
+```
+
+## Local Providers (No API Key)
+
+Both MusicGen and AudioGen can run locally without any API keys using HuggingFace Transformers.js. The models are downloaded on first use and cached locally.
+
+**Requirements:**
+- `@huggingface/transformers` must be installed as a peer dependency
+- Sufficient RAM for model inference (MusicGen Small ~1GB, AudioGen Medium ~2GB)
+
+```typescript
+// Explicitly use local generation
+const result = await generateMusic({
+  prompt: 'Simple piano melody',
+  provider: 'musicgen-local',
+});
+```
+
+Local providers are automatically used as the last fallback when no cloud API keys are configured.
+
+## Prerequisites
+
+- At least one audio provider API key for cloud generation, OR `@huggingface/transformers` for local generation
+- For music: `SUNO_API_KEY`, `UDIO_API_KEY`, `STABILITY_API_KEY`, `REPLICATE_API_TOKEN`, or `FAL_API_KEY`
+- For SFX: `ELEVENLABS_API_KEY`, `STABILITY_API_KEY`, `REPLICATE_API_TOKEN`, or `FAL_API_KEY`
+
+## Examples
+
+- "Generate a 60-second lo-fi hip hop beat for a study playlist"
+- "Create a thunder and rain sound effect for my podcast intro"
+- "Make upbeat electronic music for a product demo video"
+- "Generate a notification chime sound effect"
+- "Create ambient forest sounds with birds and a gentle stream"
+- "Generate a dramatic orchestral score for a trailer"
+- "Make a retro 8-bit video game soundtrack"
+- "Create footstep sounds on different surfaces — wood, gravel, snow"

package/registry/curated/video-generation/SKILL.md

@@ -0,0 +1,210 @@
+---
+name: video-generation
+version: '1.0.0'
+description: Video generation, analysis, and scene detection — text-to-video, image-to-video, structured scene descriptions with RAG indexing, and general-purpose visual change detection.
+author: Wunderland
+namespace: wunderland
+category: media
+tags: [video, generation, analysis, scene-detection, RAG, multimodal, runway, replicate, fal]
+requires_secrets: []
+requires_tools: []
+metadata:
+  agentos:
+    emoji: "\U0001F3AC"
+---
+
+# Video Generation, Analysis & Scene Detection
+
+Use this skill when the user wants to create AI-generated videos, analyse existing video content for structured scene descriptions, or detect visual changes in live/recorded frame streams.
+
+This skill covers three complementary APIs:
+
+1. **generateVideo()** — Text-to-video and image-to-video generation
+2. **analyzeVideo()** — Structured video analysis with scene descriptions, transcription, and optional RAG indexing
+3. **detectScenes()** — Real-time or batch scene boundary detection from frame streams
+
+## Video Generation
+
+### Text-to-Video
+
+Generate a video from a text prompt. The system auto-detects the best available provider from environment variables in priority order: `RUNWAY_API_KEY` (highest quality), `REPLICATE_API_TOKEN` (widest model variety), `FAL_API_KEY` (fast serverless GPU).
+
+```typescript
+import { generateVideo } from 'agentos';
+
+const result = await generateVideo({
+  prompt: 'A drone flying over a misty forest at sunrise, cinematic 4K',
+  durationSec: 5,
+  aspectRatio: '16:9',
+});
+console.log(result.videos[0].url);
+```
+
+### Image-to-Video
+
+Animate a still image by providing it as a Buffer via `opts.image`. The prompt describes the desired motion rather than the scene itself.
+
+```typescript
+import { generateVideo } from 'agentos';
+import { readFileSync } from 'fs';
+
+const result = await generateVideo({
+  prompt: 'Camera slowly zooms out, gentle wind moves the leaves',
+  image: readFileSync('landscape.png'),
+  provider: 'runway',
+});
+```
+
+### Provider Selection
+
+| Provider | Best For | Env Var |
+|----------|----------|---------|
+| **Runway** | Highest quality, cinematic output, image-to-video | `RUNWAY_API_KEY` |
+| **Replicate** | Widest model variety (Kling, HunyuanVideo, MiniMax), open-source models | `REPLICATE_API_TOKEN` |
+| **Fal** | Fast serverless GPU, cost-effective, Kling/CogVideo | `FAL_API_KEY` |
+
+When multiple provider API keys are set, the system wraps the primary in a `FallbackVideoProxy` so a transient failure on one provider automatically retries on the next.
+
+To force a specific provider:
+
+```typescript
+const result = await generateVideo({
+  prompt: 'A cat playing piano',
+  provider: 'replicate',
+  model: 'klingai/kling-v1',
+  apiKey: 'your-replicate-token',
+});
+```
+
+### Prompt Tips for Video
+
+- **Be specific about motion**: "camera pans left to right", "person walks toward camera", "time-lapse of clouds moving"
+- **Specify style early**: "cinematic 4K", "hand-drawn animation", "vintage film grain"
+- **Keep prompts concise**: Video models respond best to clear, focused descriptions (1-3 sentences)
+- **Use negative prompts** to avoid unwanted artifacts: `negativePrompt: 'blurry, distorted faces, watermark'`
+
+### Image-to-Video Motion Strength
+
+When doing image-to-video, the prompt controls how much the image changes:
+
+- **Gentle motion**: "subtle camera drift", "soft wind blowing through hair" — minimal departure from source
+- **Moderate motion**: "person turns head and smiles", "camera orbits subject" — clear movement while preserving subject
+- **Strong motion**: "explosion of confetti", "character runs toward camera" — significant scene change
+
+The provider's motion strength interpretation varies. Runway tends to be conservative (good for preserving the source image), while Replicate/Fal models may be more aggressive. Start with gentle prompts and increase intensity.
+
+## Video Analysis
+
+### Structured Scene Analysis
+
+Analyse a video to extract structured scene descriptions, detected objects, on-screen text, and optional audio transcription.
+
+```typescript
+import { analyzeVideo } from 'agentos';
+
+const analysis = await analyzeVideo({
+  videoUrl: 'https://example.com/product-demo.mp4',
+  prompt: 'Identify all products shown and their key features',
+  transcribeAudio: true,
+  descriptionDetail: 'detailed',
+});
+
+console.log(analysis.description);
+for (const scene of analysis.scenes ?? []) {
+  console.log(`[${scene.startSec}s - ${scene.endSec}s] ${scene.description}`);
+}
+```
+
+### RAG Integration
+
+Enable `indexForRAG: true` to automatically index scene descriptions and transcripts into the vector store for later retrieval. This is especially useful for building searchable video libraries.
+
+```typescript
+const analysis = await analyzeVideo({
+  videoBuffer: videoData,
+  indexForRAG: true,
+  descriptionDetail: 'detailed',
+  transcribeAudio: true,
+});
+
+// Scene descriptions and transcripts are now searchable via RAG
+console.log(`Indexed ${analysis.ragChunkIds?.length ?? 0} chunks`);
+```
+
+Each scene description becomes a separate vector chunk with metadata including timestamps, scene index, and cut type. This enables queries like "find the part where the presenter shows the pricing slide" to return precise timestamp ranges.
+
+### Analysis Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `sceneThreshold` | `0.3` | Scene change sensitivity (0-1, lower = more scenes) |
+| `transcribeAudio` | `true` | Transcribe audio via configured STT provider |
+| `descriptionDetail` | `'detailed'` | `'brief'`, `'detailed'`, or `'exhaustive'` |
+| `maxScenes` | `100` | Cap on detected scenes (prevents runaway on long videos) |
+| `indexForRAG` | `false` | Index results into RAG vector store |
+
+## Scene Detection
+
+### Live Stream / Batch Detection
+
+Use `detectScenes()` for real-time visual change detection on frame streams. Returns an AsyncGenerator that yields `SceneBoundary` objects as visual discontinuities are detected.
+
+```typescript
+import { detectScenes } from 'agentos';
+
+// From a pre-recorded video (frames extracted via ffmpeg)
+for await (const boundary of detectScenes({ frames: extractedFrameStream })) {
+  console.log(`Scene ${boundary.index} at ${boundary.startTimeSec}s`);
+  console.log(`  Type: ${boundary.cutType}, Confidence: ${boundary.confidence}`);
+}
+```
+
+### Use Cases
+
+- **Webcam / security camera**: Detect motion or scene changes in real-time surveillance feeds
+- **Screen recording**: Identify slide transitions in presentations, page changes in demos
+- **Video editing**: Automatically segment raw footage at cut points
+- **Content moderation**: Flag rapid scene changes that may indicate problematic content
+
+### Configuration
+
+```typescript
+for await (const boundary of detectScenes({
+  frames: webcamStream,
+  hardCutThreshold: 0.4,     // Less sensitive to hard cuts
+  gradualThreshold: 0.15,    // Standard sensitivity for dissolves/fades
+  minSceneDurationSec: 2.0,  // Suppress very short scenes
+  methods: ['histogram'],     // Fast histogram-only detection
+})) {
+  handleSceneChange(boundary);
+}
+```
+
+### Cut Type Classification
+
+The detector classifies each scene boundary:
+
+| Cut Type | Description |
+|----------|-------------|
+| `hard-cut` | Abrupt frame-to-frame change (most common) |
+| `dissolve` | Cross-dissolve / superimposition transition |
+| `fade` | Fade from/to black or white |
+| `gradual` | Other gradual visual change |
+
+## Prerequisites
+
+- At least one video provider API key for generation (`RUNWAY_API_KEY`, `REPLICATE_API_TOKEN`, or `FAL_API_KEY`)
+- **ffmpeg** on PATH for video analysis (frame extraction and audio demuxing)
+- A vision-capable LLM (`OPENAI_API_KEY` or equivalent) for scene description
+- An STT provider for audio transcription (when `transcribeAudio` is enabled)
+
+Scene detection (`detectScenes()`) has zero external dependencies — it works purely on RGB pixel buffers.
+
+## Examples
+
+- "Generate a 5-second cinematic video of a sunset over the ocean"
+- "Turn this product photo into a video with a slow camera orbit"
+- "Analyse this tutorial video and index it for search"
+- "Detect scene changes in this security camera feed"
+- "Extract structured scenes from this presentation recording"
+- "Create a video from this image with gentle parallax motion"

package/registry/curated/vision-ocr/SKILL.md

@@ -1,22 +1,82 @@
 ---
 name: vision-ocr
-description: Extract text from images using OCR and vision AI
-version: 1.0.0
+version: '1.1.0'
+description: Extract text from images using OCR and vision AI with the performOCR() high-level API or the full VisionPipeline.
+author: Wunderland
+namespace: wunderland
+category: vision
 tags: [vision, ocr, text-extraction, document, handwriting]
-tools_required: [vision-pipeline]
+requires_secrets: []
+requires_tools: [vision-pipeline]
 ---
 
 # Vision & OCR
 
-Extract text from images, documents, and handwritten notes using a progressive 3-tier pipeline: local OCR (PaddleOCR) -> local vision models (TrOCR, Florence-2) -> cloud vision (GPT-4o, Claude).
+Extract text from images, documents, and handwritten notes using a progressive 3-tier pipeline: local OCR (PaddleOCR / Tesseract) -> local vision models (TrOCR, Florence-2) -> cloud vision LLM (GPT-4o, Claude, Gemini).
+
+## High-Level API: `performOCR()`
+
+For one-shot text extraction, use the top-level `performOCR()` function. It handles input resolution, pipeline lifecycle, and cleanup automatically.
+
+```typescript
+import { performOCR } from '@framers/agentos';
+
+const result = await performOCR({
+  image: '/path/to/receipt.png',   // file path, URL, base64, or Buffer
+  strategy: 'progressive',         // 'progressive' | 'local-only' | 'cloud-only'
+  confidenceThreshold: 0.7,        // min confidence before escalating tier
+});
+
+console.log(result.text);       // extracted text
+console.log(result.confidence); // 0–1 score
+console.log(result.tier);       // 'ocr' | 'handwriting' | 'document-ai' | 'cloud-vision'
+console.log(result.provider);   // 'paddle' | 'tesseract' | 'openai' | etc.
+console.log(result.regions);    // bounding boxes (when available)
+```
+
+## When to use `performOCR()` vs `VisionPipeline`
+
+| Use case | Recommendation |
+|----------|---------------|
+| One-shot text extraction from a single image | `performOCR()` — simplest API |
+| Batch processing many images | `VisionPipeline` — create once, reuse, dispose when done |
+| Need CLIP embeddings or document layout | `VisionPipeline` — richer result shape |
+| Quick scripts and integrations | `performOCR()` — zero boilerplate |
+
+## Progressive Tier System
+
+The pipeline tries the cheapest/fastest tier first and only escalates when confidence is below threshold:
+
+1. **Tier 1 — Local OCR** (PaddleOCR or Tesseract.js): Fast, free, offline. Handles printed text in documents, receipts, screenshots.
+2. **Tier 2 — Local Vision Models** (TrOCR / Florence-2): Still offline. Handles handwritten notes, complex document layouts with tables and figures.
+3. **Tier 3 — Cloud Vision LLM** (GPT-4o / Claude / Gemini): Best quality. Handles photographs, diagrams, mixed content, anything the local tiers can't confidently read.
+
+## Strategy Selection
+
+- **`'progressive'`** (default): Start local, escalate only if needed. Best cost/quality balance for most use cases.
+- **`'local-only'`**: Never call cloud APIs. Use for air-gapped environments, privacy-sensitive data (medical records, financial docs), or when no API keys are available.
+- **`'cloud-only'`**: Skip local tiers entirely, send straight to a cloud vision LLM. Use when you need the highest quality output and cost is not a concern.
+
+## Input Formats
+
+`performOCR()` accepts four input types:
+
+- **File path**: `'/tmp/scan.png'` — reads from disk
+- **URL**: `'https://example.com/receipt.jpg'` — fetches via HTTP
+- **Base64 string**: Raw base64 or `data:image/png;base64,...` data URIs — decoded in-memory
+- **Buffer**: Raw image bytes — passed directly to the pipeline
 
 ## Capabilities
-- **Printed text OCR**: Extract text from documents, receipts, screenshots
+
+- **Printed text OCR**: Extract text from documents, receipts, screenshots, PDFs
 - **Handwriting recognition**: Read handwritten notes and forms via TrOCR
-- **Document layout**: Understand tables, figures, headings via Florence-2
-- **Image embeddings**: Generate CLIP vectors for semantic image search
+- **Document layout understanding**: Parse tables, figures, headings via Florence-2
+- **Bounding box regions**: Spatial text locations for overlay rendering
+- **Image embeddings**: Generate CLIP vectors for semantic image search (via `VisionPipeline` only)
+
+## Examples
 
-## Example
-"Read the text from this receipt"
-"What does this handwritten note say?"
-"Extract the table data from this PDF page"
+- "Read the text from this receipt"
+- "What does this handwritten note say?"
+- "Extract the table data from this PDF page"
+- "OCR this screenshot and return the error message"