@tryhamster/gerbil 1.1.2 → 1.2.0

package/docs/tools.md

@@ -1,304 +0,0 @@
-# Gerbil Tools & Agents
-
-Gerbil supports tool calling with Qwen3 models, enabling agentic workflows where the LLM can call functions to accomplish tasks.
-
-## Quick Start
-
-### Simple Tool (Recommended)
-
-Create a file in `.gerbil/tools/` with the `.tool.ts` extension:
-
-```typescript
-// .gerbil/tools/get_weather.tool.ts
-
-export default {
-  name: "get_weather",
-  description: "Get current weather for a city",
-  parameters: {
-    city: { type: "string", description: "City name" },
-  },
-  execute: async (params: { city: string }) => {
-    return `Weather in ${params.city}: 72°F, sunny`;
-  },
-};
-```
-
-That's it! Gerbil automatically loads tools from `.gerbil/tools/` when the REPL starts.
-
-### Advanced Tool (with Zod)
-
-For more complex validation, use `defineTool` with Zod schemas:
-
-```typescript
-import { defineTool } from "@tryhamster/gerbil";
-import { z } from "zod";
-
-export const weatherTool = defineTool({
-  name: "get_weather",
-  description: "Get current weather for a city",
-  parameters: z.object({
-    city: z.string().describe("City name"),
-    units: z.enum(["celsius", "fahrenheit"]).optional().default("fahrenheit"),
-  }),
-  execute: async ({ city, units }) => {
-    return `Weather in ${city}: ${units === "celsius" ? "22°C" : "72°F"}, sunny`;
-  },
-});
-```
-
-## Project Tools
-
-Store your custom tools in `.gerbil/tools/`:
-
-```
-.gerbil/
-└── tools/
-    ├── get_weather.tool.ts
-    ├── search_docs.tool.ts
-    └── calculator.tool.ts
-```
-
-### Tool File Format
-
-Each `.tool.ts` file exports a config object:
-
-```typescript
-// .gerbil/tools/my_tool.tool.ts
-
-export default {
-  name: "my_tool",                    // Tool name (snake_case)
-  description: "What the tool does",  // LLM uses this to decide when to call
-  parameters: {                       // Optional input parameters
-    param1: { type: "string", description: "First param" },
-    param2: { type: "number", optional: true },
-  },
-  execute: async (params) => {
-    // Your implementation
-    return "result string";
-  },
-};
-```
-
-### Managing Tools in REPL
-
-Press `3` or select **Tools** from the menu to:
-- View all registered tools (builtin + project)
-- Create new tools with the guided wizard
-- Execute tools directly with `x`
-- Open tool files in VS Code with `o`
-
-## Built-in Tools
-
-### `gerbil_docs`
-
-Search Gerbil documentation for any topic.
-
-```typescript
-import { docsTool } from "@tryhamster/gerbil";
-
-const result = await docsTool({ query: "streaming" });
-// Returns documentation about streaming responses
-```
-
-**Available topics:**
-- `quickstart` — Getting started
-- `generate` — Text generation options
-- `stream` — Streaming responses
-- `json` — Structured JSON output
-- `thinking` — Chain-of-thought reasoning
-- `embed` — Embeddings
-- `models` — Available models
-- `ai-sdk` — Vercel AI SDK integration
-- `next`, `express`, `react`, `hono` — Framework integrations
-- `skills` — Skills system
-- `cli` — CLI commands
-- `tools` — Tool calling
-
-## Tool Examples
-
-### Calculator
-
-```typescript
-// .gerbil/tools/calculator.tool.ts
-
-export default {
-  name: "calculator",
-  description: "Perform basic math calculations",
-  parameters: {
-    expression: { type: "string", description: "Math expression like '2 + 2'" },
-  },
-  execute: async (params: { expression: string }) => {
-    try {
-      // Simple evaluator (use a proper math lib in production)
-      const result = Function(`return ${params.expression}`)();
-      return `${params.expression} = ${result}`;
-    } catch {
-      return `Error: Invalid expression "${params.expression}"`;
-    }
-  },
-};
-```
-
-### File Reader
-
-```typescript
-// .gerbil/tools/read_file.tool.ts
-
-import fs from "fs/promises";
-
-export default {
-  name: "read_file",
-  description: "Read contents of a file",
-  parameters: {
-    path: { type: "string", description: "File path to read" },
-  },
-  execute: async (params: { path: string }) => {
-    try {
-      const content = await fs.readFile(params.path, "utf-8");
-      return content.slice(0, 2000); // Limit output size
-    } catch (e) {
-      return `Error reading file: ${e}`;
-    }
-  },
-};
-```
-
-### API Caller
-
-```typescript
-// .gerbil/tools/fetch_url.tool.ts
-
-export default {
-  name: "fetch_url",
-  description: "Fetch content from a URL",
-  parameters: {
-    url: { type: "string", description: "URL to fetch" },
-  },
-  execute: async (params: { url: string }) => {
-    try {
-      const res = await fetch(params.url);
-      const text = await res.text();
-      return text.slice(0, 2000);
-    } catch (e) {
-      return `Error fetching URL: ${e}`;
-    }
-  },
-};
-```
-
-### Joke Generator
-
-```typescript
-// .gerbil/tools/generate_joke.tool.ts
-
-export default {
-  name: "generate_joke",
-  description: "Generates a joke based on a theme",
-  parameters: {
-    theme: { type: "string", description: "The theme (optional)", optional: true },
-  },
-  execute: async (params: { theme?: string }) => {
-    const theme = params?.theme || "programming";
-    const jokes: Record<string, string> = {
-      programming: "Why do programmers prefer dark mode? Because light attracts bugs!",
-      coffee: "Why do Java developers wear glasses? Because they can't C#!",
-      ai: "Why did the neural network break up with the algorithm? Too many trust issues!",
-    };
-    return jokes[theme.toLowerCase()] || `Why did the ${theme} cross the road? To optimize the other side!`;
-  },
-};
-```
-
-## Tool Registry API
-
-```typescript
-import { 
-  defineTool, 
-  getTool, 
-  listTools, 
-  getToolDefinitions,
-  loadProjectTools
-} from "@tryhamster/gerbil";
-
-// Load tools from .gerbil/tools/
-await loadProjectTools();
-
-// Get tool by name
-const tool = getTool("get_weather");
-await tool({ city: "NYC" });
-
-// List all tool names
-console.log(listTools());
-// ["gerbil_docs", "get_weather", "calculator"]
-
-// Get all definitions (for prompt injection)
-const definitions = getToolDefinitions();
-```
-
-## Using Tools in Chat
-
-### REPL Agent Mode
-
-The easiest way to use tools is in the REPL's Agent mode:
-
-```bash
-gerbil repl
-# Press ⌘A (Cmd+A) to toggle agent mode
-```
-
-**Toggle shortcuts:**
-- **⌘A (Cmd+A)**: Toggle agent mode on/off (works anywhere)
-- **⌘T (Cmd+T)**: Toggle thinking mode on/off
-
-When agent mode is enabled:
-- Shows `+tools` badge in the header
-- Tool calls display in cyan boxes with results
-- Works with any chat persona
-
-In Agent mode, the model can:
-1. See available tools in its system prompt
-2. Call tools using the `<tool_call>` format
-3. Receive tool results and synthesize a response
-
-### Programmatic Usage
-
-```typescript
-import { 
-  formatToolsForPrompt, 
-  parseToolCall, 
-  executeToolCall,
-  getToolDefinitions 
-} from "@tryhamster/gerbil";
-
-// Add tools to system prompt
-const tools = getToolDefinitions();
-const systemPrompt = `You are a helpful assistant.\n\n${formatToolsForPrompt(tools)}`;
-
-// Generate response
-const response = await gerbil.generate(userQuery, { system: systemPrompt });
-
-// Check for tool calls
-const toolCall = parseToolCall(response.text);
-if (toolCall) {
-  const result = await executeToolCall(toolCall.tool, toolCall.params);
-  // Continue conversation with tool result
-}
-```
-
-## Best Practices
-
-1. **Clear descriptions** — The model uses descriptions to decide when to call tools
-2. **Handle missing params** — Use defaults for optional parameters
-3. **Limit output size** — Long tool outputs consume context (aim for <2000 chars)
-4. **Return strings** — Execute function must return a string
-5. **Graceful errors** — Catch exceptions and return helpful error messages
-6. **Snake_case names** — Use `get_weather` not `getWeather`
-
-## Model Compatibility
-
-Tool calling works best with:
-- **Qwen3-0.6B** — Excellent tool calling support, fast
-- **Qwen3-1.7B** — Better reasoning, still fast
-- **Qwen2.5 models** — Good support
-
-Smaller models (SmolLM2) may struggle with consistent tool call formatting.

package/docs/tts.md

@@ -1,147 +0,0 @@
-# Text-to-Speech
-
-Gerbil's native WebGPU engine synthesizes speech with **Kani-TTS-2** — an LFM2-350M
-codec-LM backbone driving NVIDIA's NeMo **NanoCodec** decoder (FSQ + causal HiFi-GAN).
-`engine.speak()` returns **22.05 kHz mono PCM**, fully on-device, no ONNX.
-
-> **Pre-1.0.** Kani-TTS-2 is the only TTS path. The old Kokoro / Supertonic ONNX /
-> transformers.js lane has been removed. The `Gerbil`-class `speak()` method still works for
-> backward compatibility but is now a thin wrapper over the native engine (see
-> [below](#gerbil-class-speak-native-wrapper)).
-
-## Quick Start
-
-### Node
-
-```typescript
-import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
-
-const engine = await WebGPUEngine.create({ repo: "nineninesix/kani-tts-2-en" });
-
-const { pcm, sampleRate, audioSeconds } = await engine.speak("Hello, I'm Gerbil!");
-// pcm: Float32Array in [-1, 1], sampleRate === 22050
-console.log(`${audioSeconds.toFixed(2)}s of audio`);
-
-engine.destroy();
-```
-
-### Options
-
-```typescript
-const result = await engine.speak("Speak with feeling.", {
-  languageTag: "en_us",     // prepended as "{tag}: {text}" (default "en_us")
-  temperature: 1.0,         // sampling temperature (default 1.0)
-  topP: 0.95,               // nucleus threshold (default 0.95)
-  repetitionPenalty: 1.1,   // (default 1.1)
-  maxFrames: 2000,          // cap audio length (default: unbounded up to maxSeqLen)
-});
-```
-
-`speak()` requires a Kani-TTS-2 checkpoint (architecture `KaniTTS2ForCausalLM`). The
-NanoCodec decoder checkpoint is downloaded lazily on first `speak()` call.
-
-## Result
-
-```typescript
-interface SpeakResult {
-  /** Mono PCM in [-1, 1]. */
-  pcm: Float32Array;
-  /** Sample rate — always 22050. */
-  sampleRate: number;
-  /** Number of audio frames decoded. */
-  frames: number;
-  /** Audio duration in seconds. */
-  audioSeconds: number;
-}
-```
-
-## Standalone KaniTTS
-
-`engine.speak()` lazily builds a `KaniTTS` engine under the hood. You can also use it
-directly (e.g. without a separate text model):
-
-```typescript
-import { KaniTTS } from "@tryhamster/gerbil/gpu";
-
-const tts = await KaniTTS.create({
-  repo: "nineninesix/kani-tts-2-en",
-  // codecRepo defaults to the NeMo 22 kHz NanoCodec MLX checkpoint
-});
-
-const { pcm, sampleRate } = await tts.speak("Hello world!");
-tts.destroy();
-```
-
-## Playing / Saving Audio
-
-### Browser (Web Audio API)
-
-```typescript
-const ctx = new AudioContext({ sampleRate });
-const buffer = ctx.createBuffer(1, pcm.length, sampleRate);
-buffer.copyToChannel(pcm, 0);
-const source = ctx.createBufferSource();
-source.buffer = buffer;
-source.connect(ctx.destination);
-source.start();
-```
-
-### Node (save to WAV)
-
-```typescript
-import { writeFileSync } from "node:fs";
-
-function saveWav(filename: string, audio: Float32Array, sampleRate: number) {
-  const buffer = Buffer.alloc(44 + audio.length * 2);
-  buffer.write("RIFF", 0);
-  buffer.writeUInt32LE(36 + audio.length * 2, 4);
-  buffer.write("WAVE", 8);
-  buffer.write("fmt ", 12);
-  buffer.writeUInt32LE(16, 16);
-  buffer.writeUInt16LE(1, 20); // PCM
-  buffer.writeUInt16LE(1, 22); // mono
-  buffer.writeUInt32LE(sampleRate, 24);
-  buffer.writeUInt32LE(sampleRate * 2, 28);
-  buffer.writeUInt16LE(2, 32);
-  buffer.writeUInt16LE(16, 34);
-  buffer.write("data", 36);
-  buffer.writeUInt32LE(audio.length * 2, 40);
-  for (let i = 0; i < audio.length; i++) {
-    const s = Math.max(-1, Math.min(1, audio[i]));
-    buffer.writeInt16LE(Math.round(s * 32767), 44 + i * 2);
-  }
-  writeFileSync(filename, buffer);
-}
-
-const { pcm, sampleRate } = await engine.speak("Hello!");
-saveWav("output.wav", pcm, sampleRate); // 22.05 kHz mono
-// macOS: execSync("afplay output.wav");
-```
-
-## Model
-
-### Kani-TTS-2
-
-- **Backbone:** LFM2-350M codec-LM (architecture `KaniTTS2ForCausalLM`)
-- **Codec:** NVIDIA NeMo NanoCodec (FSQ + causal HiFi-GAN), validated bit-exact
-- **Sample rate:** 22.05 kHz
-- **Repos:** `nineninesix/kani-tts-2-en`. License varies by variant (the `kani-tts-2-en`
-  checkpoint is LFM1.0/other; a 450M variant is Apache 2.0).
-
----
-
-## `Gerbil`-class `speak()` (native wrapper)
-
-> The Kokoro-82M / Supertonic-66M ONNX/transformers.js TTS lane has been removed. The
-> `Gerbil`-class `speak()` method below now runs the native Kani-TTS-2 engine under the hood
-> (it requires WebGPU and returns 22.05 kHz PCM). The browser `useSpeech` hook is gone — use
-> `useTTS` from `@tryhamster/gerbil/gpu/hooks`. The AI SDK `gerbil.speech()` provider also
-> routes through this native path.
-
-```typescript
-import { Gerbil } from "@tryhamster/gerbil";
-
-const g = new Gerbil();
-const result = await g.speak("Hello!");
-// result.audio = Float32Array, result.sampleRate = 22050 (native Kani-TTS-2)
-```

package/docs/vision.md

@@ -1,158 +0,0 @@
-# Vision (Image → Text)
-
-Gerbil's native WebGPU engine runs **vision language models** image-in → text-out. Two
-vision towers are supported natively, both with the same `describeImage` API:
-
-- **Qwen3.5 ViT** — bundled with `Qwen/Qwen3.5-0.8B` (the 0.8B text model's own built-in
-  vision tower). Bit-exact vs HuggingFace transformers.
-- **Gemma 4 ViT** — bundled with `mlx-community/gemma-4-e2b-it-4bit`.
-
-There is no separate "vision model" — vision is the text model's own ViT, loaded on
-demand with `enableVision: true`.
-
-> **Pre-1.0.** This is the native engine surface. The legacy `Gerbil` class (ONNX /
-> transformers.js) once exposed an `images:` array on `generate()`; that path is retired and
-> not documented here.
-
-## Quick Start
-
-### Node
-
-In Node you decode the image to RGB pixels yourself (HWC layout, 0..255), then pass
-`{ pixels, width, height }`. The engine handles smart-resize, normalization, and patchify
-internally to match the HF image processor.
-
-```typescript
-import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
-
-const engine = await WebGPUEngine.create({
-  repo: "Qwen/Qwen3.5-0.8B", // BF16 checkpoint ships the ViT weights
-  enableVision: true,
-});
-
-// pixels: Uint8ClampedArray | Uint8Array | Float32Array, RGB, length = width*height*3
-const { text, tokensPerSecond } = await engine.describeImage(
-  { pixels, width, height },
-  "What's in this image?",
-  { maxTokens: 150 },
-);
-
-console.log(text, `(${tokensPerSecond.toFixed(1)} tok/s)`);
-engine.destroy();
-```
-
-### React (Browser)
-
-The React hook decodes URLs and data-URLs for you, so you can pass an image source string
-directly. Load with `enableVision: true`.
-
-```tsx
-import { useEngine } from "@tryhamster/gerbil/gpu/hooks";
-
-function ImageDescriber() {
-  const { describeImage, completion, isLoading, isGenerating } = useEngine({
-    model: "Qwen/Qwen3.5-0.8B",
-    enableVision: true,
-    autoLoad: true,
-  });
-
-  if (isLoading) return <div>Loading vision model…</div>;
-
-  return (
-    <div>
-      <button
-        disabled={isGenerating}
-        onClick={() =>
-          describeImage("https://example.com/photo.jpg", "Describe this image in detail")
-        }
-      >
-        Describe
-      </button>
-      <p>{completion}</p>
-    </div>
-  );
-}
-```
-
-`describeImage` accepts either an image source string (data URL or http(s) URL) or
-pre-decoded `{ pixels, width, height }`. The hook caps the longest side at ~448px before
-decoding — ViT attention memory scales with patch-count², so this keeps mobile coherent.
-
-## Supported Towers
-
-| Tower | Load from | dtype | Notes |
-|-------|-----------|-------|-------|
-| Qwen3.5 ViT | `Qwen/Qwen3.5-0.8B` (`enableVision: true`) | BF16 (ships ViT) | Bit-exact vs HF (per-token cosine 1.000000) |
-| Gemma 4 ViT | `mlx-community/gemma-4-e2b-it-4bit` (`enableVision: true`) | MLX 4-bit | Native projector + multimodal-embedder norms |
-
-> The MLX 4-bit Qwen3.5 repo (`mlx-community/Qwen3.5-0.8B-4bit`) is text-only — load the
-> BF16 `Qwen/Qwen3.5-0.8B` repo to get the ViT weights for vision.
-
-## How it works
-
-`describeImage` runs the full image-in → text-out pipeline natively:
-
-1. **Preprocess** — pixels are smart-resized, normalized, and patchified to match the HF
-   image processor (`preprocessImage` / `preprocessImageGemma4`).
-2. **Encode** — the ViT turns patches into merged image-embedding tokens (`encodeImage`).
-3. **Splice** — image tokens are scattered into the `image_token_id` rows of the text
-   sequence (`EmbedSplice` in the multimodal graph).
-4. **Decode** — Qwen3.5 uses multimodal M-RoPE positions; Gemma 4 uses standard sequential
-   1D RoPE. The LM then generates the description.
-
-The low-level pieces are exposed if you need them: `engine.encodeImage(patches, gridTHW)`
-returns the merged image tokens, and `engine.hasVision` reports whether the engine was
-built with a ViT.
-
-## Image input
-
-```typescript
-// Pre-decoded RGB pixels (Node, or browser if you've already decoded)
-await engine.describeImage({ pixels, width, height }, prompt);
-
-// Already-built patch tensor + grid (skips host preprocessing — for reference parity)
-await engine.describeImage({ patches, gridTHW: [1, gridH, gridW] }, prompt);
-```
-
-In React, the hook also accepts a source string:
-
-```tsx
-describeImage("data:image/png;base64,iVBOR…", "What is this?");
-describeImage("https://example.com/photo.jpg", "Describe it"); // must be CORS-accessible
-```
-
-## Options
-
-`describeImage(image, prompt?, options?)` takes the standard generation options:
-
-```typescript
-await engine.describeImage(image, "Describe this image.", {
-  maxTokens: 150,
-  sampling: { temperature: 0.7 },
-  stopSequences: ["\n\n"],
-  onToken: (t) => process.stdout.write(t),
-});
-```
-
-Returns a `GenerateResult`: `{ text, tokensGenerated, tokensPerSecond, totalTime, finishReason }`.
-
-## Performance Tips
-
-- **Resize before sending.** ViT attention cost grows with the square of the patch count.
-  The React hook already caps the longest side at 448px; in Node, downscale large photos
-  (≈448–512px longest side) before decoding to pixels.
-- **Mobile.** On iOS/iPadOS the engine reserves fewer vision patches by default to stay
-  under the WebKit GPU watchdog and memory budget; very large images may still need a
-  smaller `maxVisionPatches` (a `WebGPUEngine.create` option).
-
-## Troubleshooting
-
-### "describeImage() requires a vision encoder"
-
-Load with `{ enableVision: true }` on a vision-capable checkpoint (Qwen3.5 BF16 or Gemma 4).
-The text-only MLX-4bit Qwen3.5 repo does not ship the ViT weights.
-
-### Out of memory on mobile
-
-Use a smaller image and/or lower `maxVisionPatches` at create time. WebKit kills the page
-content process around 1.5–2 GB.