@framers/agentos-skills-registry 0.10.0 → 0.12.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@framers/agentos-skills-registry",
-  "version": "0.10.0",
+  "version": "0.12.0",
   "files": [
     "dist",
     "registry",

package/registry/curated/image-gen/SKILL.md

@@ -1,12 +1,12 @@
 ---
 name: image-gen
-version: '1.0.0'
-description: Generate images from text prompts using AI image generation APIs like DALL-E, Stable Diffusion, or Midjourney.
+version: '2.0.0'
+description: Generate, edit, upscale, and variate images using the AgentOS multi-provider image pipeline with automatic fallback.
 author: Wunderland
 namespace: wunderland
 category: creative
-tags: [image-generation, ai-art, dall-e, stable-diffusion, creative, visual]
-requires_secrets: [openai.api_key]
+tags: [image-generation, ai-art, dall-e, stable-diffusion, flux, replicate, stability, fal, creative, visual]
+requires_secrets: []
 requires_tools: [generate_image]
 metadata:
   agentos:
@@ -15,36 +15,89 @@ metadata:
     homepage: https://platform.openai.com/docs/guides/images
 ---
 
-# AI Image Generation
+# AI Image Generation Workflow
 
-Use the `generate_image` tool to create images from text descriptions. Two providers are supported:
-- **DALL-E 3** (OpenAI) — requires `OPENAI_API_KEY`
-- **Stability AI** (SDXL) — requires `STABILITY_API_KEY`
+Use this skill when the user wants to create, edit, upscale, or create variations of images. AgentOS provides four high-level APIs that route to any configured provider with automatic fallback when multiple providers have credentials set.
+
+## The Four High-Level APIs
+
+1. **`generateImage()`** — Create new images from text prompts.
+2. **`editImage()`** — Transform existing images via img2img, inpainting, or outpainting.
+3. **`upscaleImage()`** — Increase resolution (2x or 4x super-resolution).
+4. **`variateImage()`** — Generate visual variations of an existing image.
 
 If the `generate_image` tool is not loaded, enable it with `extensions_enable image-generation`.
 
-Craft detailed, effective prompts that translate the user's creative vision into high-quality generated images.
+## Provider Selection Guide
+
+Choose the provider based on the user's priority:
+
+| Priority | Provider | Env Var | Best For |
+|----------|----------|---------|----------|
+| Quality | **OpenAI** (GPT-Image-1, DALL-E 3) | `OPENAI_API_KEY` | Highest fidelity, prompt adherence, text-in-image |
+| Control | **Stability AI** (SDXL, SD3, Ultra) | `STABILITY_API_KEY` | Negative prompts, style presets, cfg/steps tuning |
+| Speed | **BFL / Flux** (Flux Pro 1.1) | `BFL_API_KEY` | Fast generation with strong quality |
+| Speed | **Fal** (Flux Dev) | `FAL_API_KEY` | Serverless Flux inference, low latency |
+| Variety | **Replicate** (Flux, SDXL, community models) | `REPLICATE_API_TOKEN` | Access to thousands of community models |
+| Cost | **OpenRouter** (routes to cheapest) | `OPENROUTER_API_KEY` | Provider-agnostic routing, best price |
+| Privacy | **Local SD** (A1111 / ComfyUI) | `STABLE_DIFFUSION_LOCAL_BASE_URL` | Fully offline, no data leaves the machine |
+
+When multiple providers are configured, AgentOS wraps them in a **FallbackImageProxy** — if the primary provider fails (rate limit, outage, etc.), the request automatically retries on the next available provider in priority order.
+
+## Operation Decision Tree
+
+Use this to pick the right API for the user's request:
+
+- **"Generate / create / draw / imagine"** -> `generateImage()`
+- **"Edit / change / modify / transform"** -> `editImage()` with `mode: 'img2img'`
+- **"Remove / fill in / fix this area"** -> `editImage()` with `mode: 'inpaint'` + mask
+- **"Extend / expand the borders"** -> `editImage()` with `mode: 'outpaint'`
+- **"Make it higher resolution / sharper"** -> `upscaleImage()` with `scale: 2` or `4`
+- **"Show me variations / alternatives"** -> `variateImage()` with `n: 3-4`
+
+## Prompt Engineering Tips
+
+A strong image prompt has five components:
+
+1. **Subject** — What is in the image. Be specific: "a red panda sitting on a mossy branch" not "an animal."
+2. **Style** — Artistic approach: photorealistic, watercolor, pixel art, oil painting, vector illustration, cinematic, anime.
+3. **Composition** — Camera angle and framing: close-up portrait, wide establishing shot, overhead flat lay, isometric.
+4. **Lighting and Color** — Mood through light: golden hour, dramatic side-lighting, neon glow, muted earth tones, high contrast.
+5. **Atmosphere** — Emotional tone: serene, ominous, whimsical, nostalgic, futuristic.
 
-When generating images, help the user refine their prompt for best results. A good image prompt includes: subject description, style (photorealistic, illustration, watercolor, etc.), composition (close-up, wide shot, overhead), lighting (natural, dramatic, soft), color palette, and mood/atmosphere. Offer prompt suggestions when the user's description is vague or underspecified.
+Additional tips:
+- Front-load the most important elements. Models weight earlier tokens more heavily.
+- Use negative prompts (Stability, Local SD) to exclude unwanted elements: "no text, no watermark, no blurry."
+- For text-in-image, OpenAI GPT-Image-1 is the most reliable. Other models struggle with legible text.
+- Request `quality: 'hd'` for DALL-E 3 when detail matters (doubles cost).
+- For consistent characters across multiple images, describe the character in detail each time or use img2img with a reference.
 
-Support different image sizes and aspect ratios based on the API capabilities (1024x1024, 1792x1024, 1024x1792 for DALL-E 3). For iterative refinement, maintain context from previous generations so the user can say "make it more vibrant" or "change the background to a beach." Save generated images to the filesystem when the user requests it, with descriptive filenames.
+## Sizes and Aspect Ratios
 
-When the user requests variations or edits of existing images, use the appropriate API endpoints (variations, inpainting) when available. For batch generation, create multiple variations with slightly different prompts to give the user options. Always inform the user of the model and settings used for each generation.
+| Provider | Supported Sizes | Aspect Ratio Support |
+|----------|----------------|---------------------|
+| OpenAI | 1024x1024, 1792x1024, 1024x1792 | Via size selection |
+| Stability | Flexible | `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, etc. |
+| Replicate/Flux | Flexible | `aspectRatio` parameter |
+| Local SD | Any (multiples of 64) | Via `width`/`height` |
 
 ## Examples
 
-- "Generate an image of a cozy cabin in the mountains at sunset, watercolor style"
-- "Create a professional logo for a coffee shop called 'Bean There'"
-- "Make the previous image more dramatic with storm clouds"
-- "Generate 3 variations of a cyberpunk cityscape at night"
-- "Create a 16:9 landscape of a serene Japanese garden in spring"
+- "Generate a photorealistic image of a cozy cabin in the mountains at sunset."
+- "Create a professional logo for a coffee shop called 'Bean There' — vector illustration style, clean lines."
+- "Edit this photo: make the sky more dramatic with storm clouds." (img2img)
+- "Remove the person from the background of this product photo." (inpaint + mask)
+- "Upscale this thumbnail to 4x resolution for print."
+- "Show me 3 variations of this hero image with different color palettes."
+- "Generate a 16:9 cinematic landscape of a neon-lit Tokyo street at night in the rain."
 
 ## Constraints
 
 - Image generation costs API credits per request; inform the user of approximate costs when possible.
-- Content policy restrictions apply: no realistic faces of real people, no violent/explicit content.
-- DALL-E 3 does not support exact image editing or inpainting; describe the full desired output.
+- Content policy restrictions apply per provider: no realistic faces of real people, no violent/explicit content.
+- DALL-E 3 does not support native inpainting — use GPT-Image-1 or Stability for mask-based editing.
+- Upscaling is not supported by OpenAI or OpenRouter — use Stability, Replicate, or Local SD.
 - Generated images may not perfectly match the prompt; iterative refinement is expected.
-- Maximum prompt length varies by model (DALL-E 3: 4,000 characters).
-- Image quality and style depend on the model version and generation parameters.
-- Generated images should not be represented as photographs or real events.
+- Maximum prompt length varies by model (DALL-E 3: 4,000 chars; Stability: 2,000 chars).
+- Local SD requires a running A1111 or ComfyUI instance with the API enabled.
+- The fallback chain only activates when the primary provider fails; it does not merge results from multiple providers.

package/registry/curated/video-ingestion/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: video-ingestion
+version: '1.0.0'
+description: Video processing for RAG — extract frames via vision pipeline + audio via STT, index into knowledge base.
+author: Wunderland
+namespace: wunderland
+category: productivity
+tags: [video, ffmpeg, frames, transcription, multimodal, RAG]
+requires_secrets: []
+requires_tools: []
+metadata:
+  agentos:
+    emoji: "\U0001F3AC"
+---
+
+# Video Ingestion for Multimodal RAG
+
+Use this skill when the user wants to index video content into the agent's knowledge base so it can be searched and recalled during conversation.
+
+Video ingestion works through the `MultimodalMemoryBridge`, which orchestrates two parallel extraction pipelines and feeds the results into both the RAG vector store and (optionally) cognitive memory.
+
+## How It Works
+
+1. **Frame extraction** — ffmpeg samples frames at a configurable interval (default: 1 frame every 5 seconds). Each frame is passed to a vision-capable LLM (e.g. GPT-4o) which generates a text description. That description is embedded and indexed into the vector store with `modality: 'image'` metadata.
+
+2. **Audio extraction** — ffmpeg demuxes the audio track and pipes it to the configured STT provider (e.g. Whisper). The resulting transcript is chunked, embedded, and indexed with `modality: 'audio'` metadata.
+
+3. **Memory traces** — When cognitive memory is enabled, the bridge encodes both visual descriptions and audio transcript chunks as memory traces so the agent can recall video content during future conversations.
+
+## When to Ingest Video vs. Just Extract Audio
+
+- **Ingest full video** when visual content matters: tutorials, screen recordings, product demos, surveillance, presentations with slides, anything where "what is shown" conveys information the transcript alone misses.
+- **Extract audio only** when the video is essentially a podcast, voice memo, meeting recording, or phone call where the visual track adds no information. Audio-only ingestion is faster, cheaper (no vision LLM calls), and produces smaller index footprints.
+
+If you are unsure, prefer full video ingestion. The frame extraction is lightweight and the vision descriptions are short — the marginal cost is small compared to the value of not losing visual context.
+
+## Prerequisites
+
+- **ffmpeg** must be installed and on the system PATH. The bridge shells out to `ffmpeg` for frame and audio extraction. Without it, video ingestion will fail with a clear error.
+- A **vision-capable LLM** must be configured (OPENAI_API_KEY or equivalent) for frame description.
+- An **STT provider** must be configured for audio transcription.
+
+## Usage
+
+Video ingestion is triggered through the `MultimodalMemoryBridge.ingestVideo()` method. When using the HTTP API, POST the video file to:
+
+```
+POST /api/agentos/rag/multimodal/documents/ingest
+Content-Type: multipart/form-data
+```
+
+with the video file in the `document` field. The system auto-detects video MIME types and routes to the video pipeline.
+
+Programmatic usage:
+
+```typescript
+import { MultimodalMemoryBridge } from 'agentos/rag/multimodal';
+
+await bridge.ingestVideo(videoBuffer, {
+  source: 'user-upload',
+  fileName: 'meeting-2024-03-15.mp4',
+  extractFrames: true,       // default true
+  frameIntervalSeconds: 10,  // sample 1 frame every 10s (default 5)
+  language: 'en',            // STT language hint
+});
+```
+
+## Configuration Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `extractFrames` | `true` | Set `false` for audio-only ingestion |
+| `frameIntervalSeconds` | `5` | Seconds between sampled frames |
+| `language` | auto-detect | BCP-47 language code for STT |
+| `collection` | `'multimodal'` | Target vector store collection |
+
+## Examples
+
+- "Ingest this tutorial video so I can search it later."
+- "Extract the audio from this meeting recording and add it to my knowledge base."
+- "Index this product demo video — I need to reference the UI screenshots shown at 2:30."
+- "Process all MP4 files in this folder and make them searchable."
+
+## Constraints
+
+- ffmpeg must be installed. The system does not bundle or auto-install it.
+- Long videos (>1 hour) produce many frames; consider increasing `frameIntervalSeconds` to 15-30 for very long content.
+- Vision LLM calls are billed per frame. A 1-hour video at the default 5-second interval generates ~720 frames.
+- Supported container formats: MP4, MKV, WebM, AVI, MOV (anything ffmpeg can demux).
+- Video ingestion is not real-time; expect processing time proportional to video length.