noosphere 0.5.0 → 0.8.0

package/README.md

@@ -6,11 +6,14 @@ One import. Every model. Every modality.
 
 ## Features
 
-- **4 modalities** — LLM chat, image generation, video generation, and text-to-speech
+- **7 modalities** — LLM, image, video, TTS, STT, music, and embeddings
 - **Always up-to-date models** — Dynamic auto-fetch from ALL provider APIs at runtime (OpenAI, Anthropic, Google, Groq, Mistral, xAI, Cerebras, OpenRouter)
+- **Dynamic descriptions** — Model descriptions fetched from source (Ollama library, HuggingFace READMEs, CivitAI API) — no hardcoded strings
+- **Modality-filtered sync** — `syncModels('llm')` only fetches LLM providers, avoiding unnecessary requests
 - **867+ media endpoints** — via FAL (Flux, SDXL, Kling, Sora 2, VEO 3, Kokoro, ElevenLabs, and hundreds more)
 - **30+ HuggingFace tasks** — LLM, image, TTS, translation, summarization, classification, and more
-- **Local-first architecture** — Auto-detects ComfyUI, Ollama, Piper, and Kokoro on your machine
+- **Local-first architecture** — Auto-detects Ollama, ComfyUI, Whisper, AudioCraft, Piper, and Kokoro on your machine
+- **Org-aware logos** — HuggingFace models show the real org logo (Meta, Google, NVIDIA) instead of generic HF logo
 - **Agentic capabilities** — Tool use, function calling, reasoning/thinking, vision, and agent loops via Pi-AI
 - **Failover & retry** — Automatic retries with exponential backoff and cross-provider failover
 - **Usage tracking** — Real-time cost, latency, and token tracking across all providers
@@ -324,6 +327,288 @@ const imageModels = await ai.getModels('image');
 
 ---
 
+## Local Models — Run Everything on Your Machine
+
+Noosphere has **comprehensive local model support** across all modalities — LLM, image, video, TTS, STT, and music. Auto-discovers what's installed, catalogs what's available to download, and provides a unified API for everything.
+
+### Quick Start
+
+```typescript
+const ai = new Noosphere();
+await ai.syncModels();
+
+// 774 models discovered — cloud + local, all modalities
+const all = await ai.getModels();
+
+// Filter by what you can run locally
+const localModels = all.filter(m => m.local || m.status === 'installed');
+
+// What's installed vs what's available to download
+const installed = all.filter(m => m.status === 'installed');   // 39 models ready to use
+const available = all.filter(m => m.status === 'available');   // 251 models you can download
+
+// Chat with a local Ollama model — same API as cloud
+const result = await ai.chat({
+  model: 'qwen3:8b',
+  provider: 'ollama',
+  messages: [{ role: 'user', content: 'Hello!' }],
+});
+console.log(result.content);   // "Hello! How can I help?"
+console.log(result.usage);     // { cost: 0, input: 24, output: 198, unit: 'tokens' }
+
+// Install a new model from Ollama library
+await ai.installModel('deepseek-r1:14b');
+
+// Uninstall
+await ai.uninstallModel('deepseek-r1:14b');
+```
+
+### 8 Providers, 5 Modalities, 774+ Models
+
+| Provider | Modality | Models | Source | Auto-Detect |
+|---|---|---|---|---|
+| **pi-ai** | LLM | 482 | OpenAI, Anthropic, Google, Groq, Mistral, xAI, OpenRouter, Cerebras | API keys |
+| **ollama** | LLM, embedding | 70 | 38 installed + 32 from Ollama web catalog | `localhost:11434` |
+| **hf-local** | image, video, tts, stt, music | 220 | HuggingFace catalog (FLUX, SDXL, Wan2.2, Whisper, MusicGen) | Always (no API key) |
+| **huggingface** | LLM, image, tts | dynamic | HuggingFace Inference API | `HUGGINGFACE_TOKEN` |
+| **comfyui** | image, video | dynamic | Installed checkpoints + CivitAI catalog | `localhost:8188` |
+| **openai-compat** | LLM | dynamic | llama.cpp, LM Studio, vLLM, LocalAI, KoboldCpp, Jan, TabbyAPI | Scans ports |
+| **fal** | image, video, tts | 867+ | FAL.ai (Flux, SDXL, Kling, Sora 2, Kokoro, ElevenLabs) | `FAL_KEY` |
+| **piper** | TTS | 2+ | Piper voices installed locally | Binary detection |
+| **whisper-local** | STT | 8 | Whisper/Faster-Whisper (tiny → large-v3) | Python detection |
+| **audiocraft** | music | 5 | MusicGen (small/medium/large/melody) + AudioGen | Python detection |
+
+### Modality-Filtered Sync — Only Fetch What You Need
+
+Sync **only the providers relevant to a specific modality** instead of fetching everything. This avoids unnecessary network requests (e.g., fetching 270+ HuggingFace READMEs when you only need LLMs).
+
+```typescript
+// Sync only LLM providers (Ollama, pi-ai, openai-compat, huggingface)
+await ai.syncModels('llm');
+
+// Sync only image providers (hf-local, comfyui, fal, huggingface)
+await ai.syncModels('image');
+
+// Sync only STT providers (whisper-local, hf-local)
+await ai.syncModels('stt');
+
+// Sync everything (backward compatible)
+await ai.syncModels();
+```
+
+**Which providers sync for each modality:**
+
+| Modality | Providers Synced |
+|---|---|
+| `llm` | pi-ai, ollama, openai-compat, huggingface (cloud, needs API key) |
+| `image` | hf-local, comfyui, fal, huggingface (cloud) |
+| `video` | hf-local, comfyui, fal |
+| `tts` | hf-local (speech models), fal, piper, kokoro, huggingface (cloud) |
+| `stt` | hf-local, whisper-local |
+| `music` | hf-local (MusicGen, AudioLDM, etc.), audiocraft |
+| `embedding` | ollama |
+
+### Models by Modality
+
+```typescript
+const models = await ai.getModels();
+
+// Filter by modality
+const llm    = models.filter(m => m.modality === 'llm');    // 552 (cloud + Ollama local)
+const image  = models.filter(m => m.modality === 'image');  // 101 (FLUX, SDXL, SD3, PixArt...)
+const tts    = models.filter(m => m.modality === 'tts');    //  61 (MusicGen, Bark, Piper, Kokoro...)
+const video  = models.filter(m => m.modality === 'video');  //  30 (Wan2.2, CogVideoX, AnimateDiff...)
+const stt    = models.filter(m => m.modality === 'stt');    //  30 (Whisper, wav2vec2...)
+```
+
+### Ollama Provider — Local LLM
+
+Full integration with Ollama's API:
+
+```typescript
+// Auto-detected on startup — no config needed
+// Models include full metadata from Ollama
+
+const ollamaModels = models.filter(m => m.provider === 'ollama');
+for (const m of ollamaModels) {
+  console.log(m.id);                      // "llama3.3:70b"
+  console.log(m.status);                  // "installed" | "available" | "running"
+  console.log(m.localInfo.parameterSize); // "70.6B"
+  console.log(m.localInfo.quantization);  // "Q4_K_M"
+  console.log(m.localInfo.sizeBytes);     // 42520413916
+  console.log(m.localInfo.family);        // "llama"
+  console.log(m.logo);                    // { svg: "...meta.svg", png: "...meta.png" }
+}
+
+// Chat with streaming
+const stream = ai.stream({
+  model: 'qwen3:8b',
+  provider: 'ollama',
+  messages: [{ role: 'user', content: 'Explain quantum computing' }],
+});
+
+for await (const event of stream) {
+  if (event.type === 'text_delta') process.stdout.write(event.delta);
+}
+
+const finalResult = await stream.result();
+
+// Model management
+await ai.installModel('deepseek-r1:14b');     // Downloads from Ollama library
+await ai.uninstallModel('old-model:7b');       // Removes from disk
+
+// Hardware info
+const hw = await ai.getHardware();
+// { ollama: true, runningModels: [{ name: 'qwen3:8b', size: 5200000000, ... }] }
+```
+
+### OpenAI-Compatible Provider — Any Local Server
+
+Connects to ANY server that implements the OpenAI API:
+
+```typescript
+// Auto-detects servers on common ports:
+// llama.cpp (:8080), LM Studio (:1234), vLLM (:8000)
+// LocalAI (:8080), TabbyAPI (:5000), KoboldCpp (:5001), Jan (:1337)
+
+// Or configure manually:
+const ai = new Noosphere({
+  openaiCompat: [
+    { baseUrl: 'http://localhost:1234/v1', name: 'LM Studio' },
+    { baseUrl: 'http://192.168.1.100:8080/v1', name: 'Remote llama.cpp' },
+  ],
+});
+```
+
+### HuggingFace Local Catalog
+
+Auto-fetches the top models by downloads for each modality:
+
+```typescript
+const imageModels = models.filter(m => m.provider === 'hf-local' && m.modality === 'image');
+// → FLUX.1-dev, FLUX.1-schnell, SDXL, SD 3.5, PixArt-Σ, Playground v2.5, Kolors...
+
+const videoModels = models.filter(m => m.provider === 'hf-local' && m.modality === 'video');
+// → Wan2.2-T2V, CogVideoX-5b, AnimateDiff, Stable Video Diffusion...
+
+const ttsModels = models.filter(m => m.provider === 'hf-local' && m.modality === 'tts');
+// → MusicGen, Stable Audio Open, Bark, ACE-Step...
+
+const sttModels = models.filter(m => m.provider === 'hf-local' && m.modality === 'stt');
+// → Whisper large-v3, Whisper large-v3-turbo, wav2vec2...
+```
+
+Models already downloaded to `~/.cache/huggingface/hub/` are automatically detected as `status: 'installed'`.
+
+### ComfyUI — Dynamic Workflow Engine
+
+When ComfyUI is running, noosphere discovers all installed checkpoints, LoRAs, and models:
+
+```typescript
+// Auto-detected on localhost:8188
+const comfyModels = models.filter(m => m.provider === 'comfyui');
+// → All checkpoints (SD 1.5, SDXL, FLUX, Pony, etc.)
+
+// Also fetches top models from CivitAI as "available"
+const civitai = comfyModels.filter(m => m.status === 'available');
+```
+
+### Model Descriptions — Dynamic from Source
+
+Every model includes a `description` field fetched dynamically from its source — no hardcoded strings:
+
+```typescript
+const models = await ai.getModels('llm');
+
+for (const m of models) {
+  console.log(m.name, m.description);
+  // "llama3.1"  "Llama 3.1 is a new state-of-the-art model from Meta available in 8B, 70B and 405B"
+  // "qwen3"     "Qwen3 is the latest generation of large language models in Qwen series"
+  // "gemma3"    "The current, most capable model that runs on a single GPU"
+}
+
+const imageModels = await ai.getModels('image');
+for (const m of imageModels) {
+  console.log(m.name, m.description);
+  // "stable-diffusion-xl-base-1.0"  "Stable Diffusion XL (SDXL) is a latent text-to-image..."
+  // "FLUX.1-dev"                     "FLUX.1 [dev] is a 12 billion parameter rectified flow..."
+}
+```
+
+| Provider | Description Source |
+|---|---|
+| **Ollama** | Scraped from `ollama.com/library` page |
+| **HuggingFace Local** | Parsed from each model's `README.md` on HuggingFace Hub |
+| **CivitAI/ComfyUI** | Extracted from CivitAI API response |
+| **Whisper** | Parsed from OpenAI's Whisper README on HuggingFace |
+| **AudioCraft** | Parsed from Meta's AudioCraft README on HuggingFace |
+
+All description fetches are **parallel and fail-safe** — if a source is unreachable, models are returned without descriptions. No API keys required.
+
+### Model Status & Local Info
+
+Every local model includes rich metadata:
+
+```typescript
+interface ModelInfo {
+  id: string;
+  provider: string;
+  name: string;
+  description?: string;          // Dynamic from source (Ollama library, HF README, CivitAI)
+  modality: 'llm' | 'image' | 'video' | 'tts' | 'stt' | 'music' | 'embedding';
+  status?: 'installed' | 'available' | 'downloading' | 'running' | 'error';
+  local: boolean;
+  logo?: { svg?: string; png?: string };
+  localInfo?: {
+    sizeBytes: number;
+    family?: string;              // "llama", "gemma3", "qwen2"
+    parameterSize?: string;       // "70.6B", "7B", "3.2B"
+    quantization?: string;        // "Q4_K_M", "Q8_0", "F16"
+    format?: string;              // "gguf", "safetensors", "onnx"
+    digest?: string;
+    modifiedAt?: string;
+    running?: boolean;
+    runtime: string;              // "ollama", "diffusers", "comfyui", "piper", "whisper"
+  };
+  capabilities: {
+    contextWindow?: number;
+    maxTokens?: number;
+    supportsVision?: boolean;
+    supportsStreaming?: boolean;
+  };
+}
+```
+
+### Web Catalogs (Auto-Fetched)
+
+| Source | API | What it provides |
+|---|---|---|
+| **Ollama Library** | `ollama.com/api/tags` | 215+ LLM families with sizes and quantizations |
+| **HuggingFace** | `huggingface.co/api/models?pipeline_tag=...` | Top models per modality (image, video, TTS, STT) |
+| **CivitAI** | `civitai.com/api/v1/models` | SD/SDXL/FLUX checkpoints with previews |
+
+### Auto-Detection — Zero Config
+
+Noosphere auto-detects all local runtimes on startup:
+
+| Runtime | Detection Method | Default Port |
+|---|---|---|
+| Ollama | `GET localhost:11434/api/version` | 11434 |
+| ComfyUI | `GET localhost:8188/system_stats` | 8188 |
+| llama.cpp | `GET localhost:8080/health` | 8080 |
+| LM Studio | `GET localhost:1234/v1/models` | 1234 |
+| vLLM | `GET localhost:8000/v1/models` | 8000 |
+| KoboldCpp | `GET localhost:5001/v1/models` | 5001 |
+| TabbyAPI | `GET localhost:5000/v1/models` | 5000 |
+| Jan | `GET localhost:1337/v1/models` | 1337 |
+| Piper | Binary in PATH | — |
+| Whisper | Python package detection | — |
+| AudioCraft | Python package detection | — |
+
+> 📄 **Full research:** [`docs/LOCAL_AI_RESEARCH.md`](./docs/LOCAL_AI_RESEARCH.md) — 44KB covering 12+ runtimes across all modalities
+
+---
+
 ## Configuration
 
 API keys are resolved from the constructor config or environment variables (config takes priority):