noosphere 0.7.0 → 0.9.0

package/README.md

@@ -6,11 +6,16 @@ One import. Every model. Every modality.
 
 ## Features
 
-- **4 modalities** — LLM chat, image generation, video generation, and text-to-speech
+- **7 modalities** — LLM, image, video, TTS, STT, music, and embeddings
+- **OpenAI media** — GPT-Image-1/1.5, DALL-E 2/3, Sora 2/Pro (video), TTS-1/HD, Whisper — all auto-fetched from `OPENAI_API_KEY`
+- **Google media** — Imagen 4.0 (image), Veo 2/3/3.1 (video), Gemini TTS — all auto-fetched from `GEMINI_API_KEY`
 - **Always up-to-date models** — Dynamic auto-fetch from ALL provider APIs at runtime (OpenAI, Anthropic, Google, Groq, Mistral, xAI, Cerebras, OpenRouter)
+- **Dynamic descriptions** — Model descriptions fetched from source (Ollama library, HuggingFace READMEs, CivitAI API) — no hardcoded strings
+- **Modality-filtered sync** — `syncModels('llm')` only fetches LLM providers, avoiding unnecessary requests
 - **867+ media endpoints** — via FAL (Flux, SDXL, Kling, Sora 2, VEO 3, Kokoro, ElevenLabs, and hundreds more)
 - **30+ HuggingFace tasks** — LLM, image, TTS, translation, summarization, classification, and more
-- **Local-first architecture** — Auto-detects ComfyUI, Ollama, Piper, and Kokoro on your machine
+- **Local-first architecture** — Auto-detects Ollama, ComfyUI, Whisper, AudioCraft, Piper, and Kokoro on your machine
+- **Org-aware logos** — HuggingFace models show the real org logo (Meta, Google, NVIDIA) instead of generic HF logo
 - **Agentic capabilities** — Tool use, function calling, reasoning/thinking, vision, and agent loops via Pi-AI
 - **Failover & retry** — Automatic retries with exponential backoff and cross-provider failover
 - **Usage tracking** — Real-time cost, latency, and token tracking across all providers
@@ -35,13 +40,29 @@ const response = await ai.chat({
 });
 console.log(response.content);
 
-// Generate an image
+// Generate an image with GPT-Image-1 (OpenAI) — just needs OPENAI_API_KEY
 const image = await ai.image({
   prompt: 'A sunset over mountains',
+  provider: 'openai-media',
+});
+// image.buffer contains the PNG data
+
+// Generate an image with Imagen 4.0 (Google) — just needs GEMINI_API_KEY
+const googleImage = await ai.image({
+  prompt: 'A sunset over mountains',
+  provider: 'google-media',
+});
+// googleImage.buffer contains the PNG data
+
+// Generate an image with DALL-E 3
+const dalle = await ai.image({
+  prompt: 'A sunset over mountains',
+  provider: 'openai-media',
+  model: 'dall-e-3',
   width: 1024,
   height: 1024,
 });
-console.log(image.url);
+console.log(dalle.url);
 
 // Generate a video
 const video = await ai.video({
@@ -50,7 +71,7 @@ const video = await ai.video({
 });
 console.log(video.url);
 
-// Text-to-speech
+// Text-to-speech with OpenAI TTS — just needs OPENAI_API_KEY
 const audio = await ai.speak({
   text: 'Welcome to Noosphere',
   voice: 'alloy',
@@ -365,14 +386,48 @@ await ai.uninstallModel('deepseek-r1:14b');
 | Provider | Modality | Models | Source | Auto-Detect |
 |---|---|---|---|---|
 | **pi-ai** | LLM | 482 | OpenAI, Anthropic, Google, Groq, Mistral, xAI, OpenRouter, Cerebras | API keys |
+| **openai-media** | image, video, tts, stt | 12 | GPT-Image-1/1.5, DALL-E 2/3, Sora 2/Pro, TTS-1/HD, Whisper | `OPENAI_API_KEY` |
+| **google-media** | image, video, tts | 10 | Imagen 4.0, Veo 2/3/3.1, Gemini TTS (Flash/Pro) | `GEMINI_API_KEY` |
 | **ollama** | LLM, embedding | 70 | 38 installed + 32 from Ollama web catalog | `localhost:11434` |
-| **hf-local** | image, video, tts, stt | 220 | HuggingFace catalog (FLUX, SDXL, Wan2.2, Whisper, MusicGen) | Always |
+| **hf-local** | image, video, tts, stt, music | 220 | HuggingFace catalog (FLUX, SDXL, Wan2.2, Whisper, MusicGen) | Always (no API key) |
+| **huggingface** | LLM, image, tts | dynamic | HuggingFace Inference API | `HUGGINGFACE_TOKEN` |
 | **comfyui** | image, video | dynamic | Installed checkpoints + CivitAI catalog | `localhost:8188` |
 | **openai-compat** | LLM | dynamic | llama.cpp, LM Studio, vLLM, LocalAI, KoboldCpp, Jan, TabbyAPI | Scans ports |
+| **fal** | image, video, tts | 867+ | FAL.ai (Flux, SDXL, Kling, Sora 2, Kokoro, ElevenLabs) | `FAL_KEY` |
 | **piper** | TTS | 2+ | Piper voices installed locally | Binary detection |
 | **whisper-local** | STT | 8 | Whisper/Faster-Whisper (tiny → large-v3) | Python detection |
 | **audiocraft** | music | 5 | MusicGen (small/medium/large/melody) + AudioGen | Python detection |
 
+### Modality-Filtered Sync — Only Fetch What You Need
+
+Sync **only the providers relevant to a specific modality** instead of fetching everything. This avoids unnecessary network requests (e.g., fetching 270+ HuggingFace READMEs when you only need LLMs).
+
+```typescript
+// Sync only LLM providers (Ollama, pi-ai, openai-compat, huggingface)
+await ai.syncModels('llm');
+
+// Sync only image providers (hf-local, comfyui, fal, huggingface)
+await ai.syncModels('image');
+
+// Sync only STT providers (whisper-local, hf-local)
+await ai.syncModels('stt');
+
+// Sync everything (backward compatible)
+await ai.syncModels();
+```
+
+**Which providers sync for each modality:**
+
+| Modality | Providers Synced |
+|---|---|
+| `llm` | pi-ai, ollama, openai-compat, huggingface (cloud) |
+| `image` | **openai-media** (GPT-Image-1, DALL-E), **google-media** (Imagen 4.0), hf-local, comfyui, fal, huggingface (cloud) |
+| `video` | **openai-media** (Sora 2/Pro), **google-media** (Veo 2/3/3.1), hf-local, comfyui, fal |
+| `tts` | **openai-media** (TTS-1, TTS-1-HD), **google-media** (Gemini TTS), hf-local, fal, piper, kokoro, huggingface (cloud) |
+| `stt` | **openai-media** (Whisper), hf-local, whisper-local |
+| `music` | hf-local (MusicGen, AudioLDM, etc.), audiocraft |
+| `embedding` | ollama |
+
 ### Models by Modality
 
 ```typescript
@@ -478,6 +533,38 @@ const comfyModels = models.filter(m => m.provider === 'comfyui');
 const civitai = comfyModels.filter(m => m.status === 'available');
 ```
 
+### Model Descriptions — Dynamic from Source
+
+Every model includes a `description` field fetched dynamically from its source — no hardcoded strings:
+
+```typescript
+const models = await ai.getModels('llm');
+
+for (const m of models) {
+  console.log(m.name, m.description);
+  // "llama3.1"  "Llama 3.1 is a new state-of-the-art model from Meta available in 8B, 70B and 405B"
+  // "qwen3"     "Qwen3 is the latest generation of large language models in Qwen series"
+  // "gemma3"    "The current, most capable model that runs on a single GPU"
+}
+
+const imageModels = await ai.getModels('image');
+for (const m of imageModels) {
+  console.log(m.name, m.description);
+  // "stable-diffusion-xl-base-1.0"  "Stable Diffusion XL (SDXL) is a latent text-to-image..."
+  // "FLUX.1-dev"                     "FLUX.1 [dev] is a 12 billion parameter rectified flow..."
+}
+```
+
+| Provider | Description Source |
+|---|---|
+| **Ollama** | Scraped from `ollama.com/library` page |
+| **HuggingFace Local** | Parsed from each model's `README.md` on HuggingFace Hub |
+| **CivitAI/ComfyUI** | Extracted from CivitAI API response |
+| **Whisper** | Parsed from OpenAI's Whisper README on HuggingFace |
+| **AudioCraft** | Parsed from Meta's AudioCraft README on HuggingFace |
+
+All description fetches are **parallel and fail-safe** — if a source is unreachable, models are returned without descriptions. No API keys required.
+
 ### Model Status & Local Info
 
 Every local model includes rich metadata:
@@ -486,6 +573,8 @@ Every local model includes rich metadata:
 interface ModelInfo {
   id: string;
   provider: string;
+  name: string;
+  description?: string;          // Dynamic from source (Ollama library, HF README, CivitAI)
   modality: 'llm' | 'image' | 'video' | 'tts' | 'stt' | 'music' | 'embedding';
   status?: 'installed' | 'available' | 'downloading' | 'running' | 'error';
   local: boolean;