@warlock.js/ai 4.5.0 → 4.6.1

package/skills/rag-loaders-and-stores/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: rag-loaders-and-stores
+description: 'Turn any source into a RagDocument and index it in a production vector store — the document loaders ai.rag.loadText / loadHtml / loadWeb (SSRF-safe via guardedFetch) / loadPdf (lazy pdf-parse peer), plus the swappable stores ai.rag.pgVectorStore({client}) (pgvector + ensureSchema DDL + hnsw/ivfflat index) and ai.rag.cacheVectorStore(driver), both satisfying VectorStoreContract (upsert / query / removeNamespace). Loaders return the exact RagDocument[] that kb.index() consumes — no adapter. Triggers: `ai.rag.loadText`, `ai.rag.loadHtml`, `ai.rag.loadWeb`, `ai.rag.loadPdf`, `loadText`, `loadHtml`, `loadWeb`, `loadPdf`, `ai.rag.pgVectorStore`, `ai.rag.cacheVectorStore`, `pgVectorStore`, `cacheVectorStore`, `VectorStore`, `PgVectorStoreOptions`, `PgVectorStoreInstance`, `ensureSchema`, `schema()`, `RagLoaderResult`, `LoadWebOptions`, `LoadPdfOptions`, `perPage`, `OutboundPolicy`, `guardedFetch`, `hnsw`, `ivfflat`, `pgvector`, `dimensions`, `PgClientLike`, `PDF_PARSE_INSTALL_INSTRUCTIONS`; ''load a website into a knowledge base'', ''index a PDF for RAG'', ''strip HTML to text for embedding'', ''pgvector store for RAG'', ''SSRF-safe document fetch'', ''one document per PDF page'', ''swap the vector store''; typical import `import { ai } from "@warlock.js/ai"`. Skip: the chunk → embed → retrieve → rerank → cite pipeline that consumes these — `@warlock.js/ai/run-ai-rag/SKILL.md`; the raw embedder primitive — `@warlock.js/ai/embed-text/SKILL.md`; cache similarity internals — `@warlock.js/cache/use-cache-similarity/SKILL.md`; competing libs `langchain` loaders, `llamaindex` readers.'
+---
+
+# RAG loaders + vector stores — source → `RagDocument` → durable store
+
+Two feature groups that bracket `ai.rag()`: **loaders** turn a source (string, raw HTML, a URL, or PDF bytes) into the exact `RagDocument[]` shape `kb.index()` consumes, and **stores** are the swappable backends that hold the embeddings. Both live on the `ai.rag.*` namespace — present the moment `@warlock.js/ai` is imported, no side-effect import, no module augmentation.
+
+## Contract — what each side produces / satisfies
+
+Every loader returns `RagLoaderResult` — a plain `RagDocument[]` — so a load feeds `index()` with no adapter and callers never branch on arity (one page ⇒ 1 doc, a per-page PDF ⇒ N docs):
+
+```ts
+type RagLoaderResult = RagDocument[];
+type RagDocument = { id: string; text: string; metadata?: Record<string, unknown>; tags?: string[] };
+```
+
+Every store satisfies the three-method `VectorStore` contract (a thin narrowing of the cache `similar()` surface — NOT a new engine):
+
+```ts
+interface VectorStore {
+  upsert(key: string, value: unknown, vector: number[], tags?: string[]): Promise<void>;
+  query<T>(vector: number[], options: { topK: number; threshold?: number; tags?: string[] }): Promise<{ key: string; value: T; score: number }[]>;
+  removeNamespace(namespace: string): Promise<void>;
+}
+```
+
+## Loaders
+
+| Loader | Input | Deps | Emits |
+|---|---|---|---|
+| `ai.rag.loadText(input, opts?)` | `string` \| `{ id, text }` \| array of either | none | one doc per non-empty item |
+| `ai.rag.loadHtml(html, opts?)` | raw HTML string | none (regex strip) | one doc, `metadata.title` from `<title>` |
+| `ai.rag.loadWeb(url, opts?)` | absolute URL | none (uses core `guardedFetch`) | one doc, SSRF-safe fetch |
+| `ai.rag.loadPdf(bytes, opts?)` | `Buffer` \| `ArrayBuffer` \| `Uint8Array` | lazy `pdf-parse` peer | one doc, or one per page with `perPage: true` |
+
+Shared options (`RagLoaderOptions`): `id` (source id — falls back to the URL for web, `"document"` otherwise), `metadata` (merged **over** the loader-derived keys, so an explicit `metadata.title` always wins), and `tags` (applied to every chunk for `retrieve({ tags })` filtering). Loader-derived keys: `source`, `loader` (`"text" | "html" | "web" | "pdf"`), plus `title` / `page` / `pageCount` / `contentType` where determinable. Empty / whitespace-only / all-markup inputs emit **no** document — never a no-op record for `index()` to skip.
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+// Bare string, or many records → many distinctly-identified docs:
+await kb.index(ai.rag.loadText([
+  { id: "faq-billing", text: "…", metadata: { section: "billing" } },
+  { id: "faq-shipping", text: "…" },
+]));
+
+// Raw HTML → readable text (scripts/styles dropped, entities decoded):
+await kb.index(ai.rag.loadHtml(rawHtml, { id: "landing", tags: ["marketing"] }));
+```
+
+### `loadWeb` is SSRF-safe — never a raw `fetch`
+
+Every request goes through core's `guardedFetch` under an `OutboundPolicy`. The strict defaults (https-only, private-IP-deny on, 10s timeout, 5 MiB cap) apply even when you pass no `policy`, so an untuned call is already hardened. Tighten it per call:
+
+```ts
+await kb.index(await ai.rag.loadWeb("https://docs.example.com/guide", {
+  policy: { hostAllowlist: ["docs.example.com"], maxBytes: 2_000_000, timeoutMs: 5_000 },
+  tags: ["docs"],
+}));
+```
+
+HTML responses run through the same tag-strip pass as `loadHtml`; non-HTML text (`text/plain`, markdown) is used verbatim. `metadata.source` is the resolved URL, `metadata.contentType` the server-reported type. A non-OK response, a policy block, a timeout, or an over-cap body throws `OutboundPolicyError`.
+
+### `loadPdf` — lazy optional peer, page-precise citations
+
+`pdf-parse` is an **optional** peer, dynamic-imported on the FIRST `loadPdf` call — importing `@warlock.js/ai` never forces it. When it is absent, the curated `PDF_PARSE_INSTALL_INSTRUCTIONS` string is thrown as a plain `Error` (a missing infra peer, not a content problem), never a raw module-resolution stack trace.
+
+```ts
+import { readFile } from "node:fs/promises";
+
+// Whole PDF → one doc carrying metadata.pageCount:
+await kb.index(await ai.rag.loadPdf(await readFile("manual.pdf"), { id: "manual" }));
+
+// One doc per page → citations stay page-precise (id suffixed `#p<n>`, metadata.page set):
+await kb.index(await ai.rag.loadPdf(bytes, { id: "manual", perPage: true }));
+```
+
+An image-only / scanned page has no text layer and is dropped, so a fully-scanned PDF yields zero docs (nothing to embed).
+
+## Stores
+
+### `ai.rag.cacheVectorStore(driver)` — adapt any `@warlock.js/cache` driver
+
+The cache driver **is** the vector store — `upsert → set({ vector, tags })`, `query → similar()`, `removeNamespace → removeNamespace()`. A driver without similarity support throws `CacheUnsupportedError` unchanged (pointing you at the `pg` / `redis` cache drivers).
+
+```ts
+import { MemoryCacheDriver } from "@warlock.js/cache";
+
+const store = ai.rag.cacheVectorStore(new MemoryCacheDriver()); // dev / tests
+```
+
+### `ai.rag.pgVectorStore(options)` — production pgvector
+
+One durable row per chunk keyed by the pipeline's dotted key, the chunk payload in a `JSONB` `value` column, the embedding in a pgvector `vector` column. Pass a live pool (`{ client }` — `@warlock.js/ai` imports **nothing**) or a `{ connectionString }` and let the store lazily `import("pg")` (the optional peer; curated install string on first use if absent). Exactly one of the two is required.
+
+```ts
+type PgVectorStoreOptions = {
+  client?: PgClientLike;               // a pg.Pool / pg.Client — only `query` is ever called
+  connectionString?: string;           // else the store builds its own Pool lazily
+  table?: string;                      // default "warlock_ai_rag_vectors"; must be a safe identifier
+  dimensions?: number;                 // vector(N) width in the DDL, default 1536
+  index?: "hnsw" | "ivfflat" | "none"; // ANN strategy, default "hnsw"
+  ivfflatLists?: number;               // ivfflat only, default 100
+};
+```
+
+`schema()` (alias `ensureSchema()`) returns the reference migration DDL — `CREATE EXTENSION vector`, the table, a GIN index on `tags`, and the chosen ANN index (`USING hnsw (embedding vector_cosine_ops)`). It **only returns the string**; the framework never auto-migrates — you run it once through your own tool. Index and query MUST use the same embedding model: the `vector(N)` width is fixed at table-creation time from `dimensions`.
+
+```ts
+import { Pool } from "pg";
+
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+const store = ai.rag.pgVectorStore({ client: pool, dimensions: 1536, index: "hnsw" });
+
+// Once, via your migration tooling — never auto-run:
+await pool.query(store.ensureSchema());
+```
+
+`query()` runs the cosine floor (`threshold`) and `tags` overlap filter **in SQL** (a below-floor row never crosses the wire), orders by `embedding <=> $vec`, caps at `topK`, and maps the pgvector distance back to a `[0,1]` cosine-similarity `score` — the same scale the cache store emits. `removeNamespace()` is a prefix DELETE that escapes `_` / `%` so dropping `ai.rag.docs` never also catches `ai.rag.docs2`.
+
+## Pattern — a knowledge base from a website, backed by pgvector
+
+```ts
+import { Pool } from "pg";
+import { ai } from "@warlock.js/ai";
+import { OpenAISDK } from "@warlock.js/ai-openai";
+
+const openai = new OpenAISDK({ apiKey: process.env.OPENAI_API_KEY! });
+const pool = new Pool({ connectionString: process.env.DATABASE_URL });
+
+const store = ai.rag.pgVectorStore({ client: pool, dimensions: 1536 });
+await pool.query(store.ensureSchema()); // once at boot / migration
+
+const kb = ai.rag({
+  name: "docs",
+  embedder: openai.embedder({ name: "text-embedding-3-small" }), // 1536 dims — matches the DDL
+  store,
+});
+
+// Crawl a few pages (SSRF-safe) + a spec PDF into the same namespace:
+for (const url of ["https://docs.example.com/intro", "https://docs.example.com/config"]) {
+  await kb.index(await ai.rag.loadWeb(url, { policy: { hostAllowlist: ["docs.example.com"] }, tags: ["docs"] }));
+}
+await kb.index(await ai.rag.loadPdf(await readFile("spec.pdf"), { id: "spec", perPage: true, tags: ["spec"] }));
+
+// Now retrieve — every hit's citation traces back to its source URL / page:
+const { chunks } = await kb.retrieve("how do I configure caching?", { topK: 4, tags: ["docs"] });
+```
+
+The `embedder`'s `dimensions` MUST equal the store's `dimensions` — a mismatch is a runtime insert failure at the pgvector column, not a type error.
+
+## Cost + testing
+
+- **Loaders are cheap.** `loadText` / `loadHtml` are zero-dependency string passes; `loadWeb` costs one guarded HTTP round-trip; `loadPdf` costs the `pdf-parse` parse. **None embed** — embedding cost lands entirely in `kb.index()` (batched, 96 texts per `embedMany` call). The token spend is per chunk, so `perPage` PDFs and finer chunking mean more, smaller vectors.
+- **`pgVectorStore` construction is synchronous and does no I/O** — the `pg` import + pool build are deferred to first `query`/`upsert`. Table-name validation (`/^[A-Za-z_][A-Za-z0-9_]*$/`) throws at construction, so a `table: "bad; DROP TABLE x"` fails fast.
+- **Unit-test loaders with fixtures** (a stubbed `policy.fetch` for `loadWeb`, `vi.mock("pdf-parse")` for `loadPdf` — the literal specifier is mockable). Test stores against a `FakePgClient` implementing `{ query }`, or `cacheVectorStore(new MemoryCacheDriver())` for a real end-to-end index/retrieve with no external service.
+
+## See also
+
+- [[run-ai-rag]] — the chunk → embed → retrieve → rerank → cite pipeline that **consumes** these loaders and stores (`ai.rag({ embedder, store })`, `index()` / `retrieve()`).
+- [[embed-text]] — the `sdk.embedder` primitive whose `dimensions` must match the store's `vector(N)` width.
+- [`@warlock.js/cache/use-cache-similarity/SKILL.md`](@warlock.js/cache/use-cache-similarity/SKILL.md) — the cache driver `cacheVectorStore` adapts.

package/skills/transcribe-audio/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: transcribe-audio
+description: 'Speech-to-text via ai.transcribe({ model: sdk.transcribe({ name }), audio }) — the audio-INPUT verb (Theme I), returning the uniform never-throws { data, error, usage, report } envelope with cost-truth + panoptic observation. Feed it an AudioInput = { base64; mediaType; filename? } — build one with ai.audioFromFile(path) (reads disk, infers media type incl. WhatsApp .ogg/.opus) or ai.audioFromBuffer(bytes, mediaType). Models: OpenAI whisper-1 (verbose_json, per-minute, segments + durationSeconds) or gpt-4o-transcribe (json, per-token). Triggers: `ai.transcribe`, `ai.audioFromFile`, `ai.audioFromBuffer`, `sdk.transcribe`, `openai.transcribe`, `TranscriptionModelContract`, `AudioInput`, `TranscriptionSegment`, `MockTranscriptionModel`; ''speech to text'', ''transcribe audio'', ''voice note to text'', ''WhatsApp voice message'', ''whisper'', ''gpt-4o-transcribe'', ''subtitle segments'', ''audio input''; typical import `import { ai } from "@warlock.js/ai"` + `import { OpenAISDK } from "@warlock.js/ai-openai"`. Skip: text-to-speech / synthesizing a voice — [[generate-speech]]; competing libs raw `openai.audio.transcriptions.create`, `whisper.cpp`.'
+---
+
+# Transcribe audio — the speech-to-text verb (`ai.transcribe`)
+
+`ai.transcribe()` is the inverse of `ai.speech()` on the modality track (Theme I). Audio-in / text-out, wrapped in the same uniform result contract every executable returns — so transcribing a support voicemail slots into cost dashboards and panoptic traces exactly like an agent run.
+
+**Extracting text from an audio file NEEDS AI** — that is the `ai.transcribe` step. The file handling (`ai.audioFromFile` / `ai.audioFromBuffer`) is pure, non-AI **utility** that just packages bytes into an `AudioInput`; it does no I/O to a provider on its own.
+
+This is audio **input** (STT). For audio **output** (synthesizing a voice line), see [[generate-speech]].
+
+## Shape — WhatsApp voice note → text, end to end
+
+```ts
+import { ai } from "@warlock.js/ai";
+import { OpenAISDK } from "@warlock.js/ai-openai";
+
+const openai = new OpenAISDK({ apiKey: process.env.OPENAI_API_KEY! });
+
+// audioFromFile reads the file + infers the media type from the extension.
+// .ogg / .opus (Android WhatsApp) and .m4a (iOS) are recognized out of the box.
+const audio = await ai.audioFromFile("./voice-note.ogg");
+
+const { data, error } = await ai.transcribe({
+  model: openai.transcribe({ name: "whisper-1" }),
+  audio,
+  language: "en", // BCP-47 hint — improves accuracy + latency
+});
+
+if (error) console.warn(error.code); // typed AIError
+else console.log(data.text);         // the transcript
+```
+
+`TranscriptionModelContract` mirrors `SpeechModelContract` — a peer primitive produced by the adapter's optional `transcribe?()` factory. A non-STT model id (`openai.transcribe({ name: "gpt-4o" })`) throws `InvalidRequestError` **at construction** — fail fast, like the speech/embedder guards.
+
+## The `AudioInput` shape + the two builders
+
+```ts
+type AudioInput = {
+  base64: string;    // base64-encoded audio bytes
+  mediaType: string; // IANA type, e.g. "audio/ogg", "audio/mpeg"
+  filename?: string; // helps providers infer the codec from the extension
+};
+
+// From a file on disk — reads + infers media type (override for extensionless files).
+const fromDisk = await ai.audioFromFile("./meeting.m4a");
+const forced   = await ai.audioFromFile("./blob", { mediaType: "audio/ogg" });
+
+// From bytes you already hold (an upload buffer, a downloaded blob) — no I/O, no AI.
+const fromBytes = ai.audioFromBuffer(uploadBuffer, "audio/ogg", "note.ogg");
+```
+
+Keeping `AudioInput` as inlined base64 + explicit media type makes the verb provider-neutral and serializable — there is no `fs` coupling in core, so the same request can cross a queue or an RPC boundary.
+
+## The result envelope
+
+```ts
+type TranscriptionResult = {
+  type: "transcription";
+  data?: {
+    text: string;                    // full transcript
+    segments?: TranscriptionSegment[]; // timestamped, in verbose mode
+  };                                 // undefined on failure
+  error?: AIError;                   // undefined on success — NEVER thrown
+  usage: Usage;                      // tokens (gpt-4o-transcribe) + cost when priced
+  report: TranscriptionReport;       // type:"transcription", model, durationSeconds, lineage
+};
+
+type TranscriptionSegment = { text: string; start?: number; end?: number };
+```
+
+`segments` and `report.durationSeconds` appear only when the provider returns them (whisper's `verbose_json` mode). Use segments to build subtitles or to jump-to-timestamp in a player.
+
+## Transcribe options (provider-neutral)
+
+```ts
+await ai.transcribe({
+  model,
+  audio,
+  language: "en",              // BCP-47 hint
+  prompt: "Names: Acme, Zoë",  // priming — spelling / style hints
+  format: "verbose_json",      // response-format override (segments + duration)
+  signal,                      // AbortSignal
+  observe: collector,          // route the report to an Observer (panoptic)
+  sessionId: "ticket-88",      // group into a session for flat cost/trace queries
+  options: { /* provider passthrough */ },
+});
+```
+
+## OpenAI — whisper-1 (per-minute) + gpt-4o-transcribe (per-token)
+
+```ts
+// whisper-1 — defaults to verbose_json → segments + duration; billed PER MINUTE.
+const whisper = openai.transcribe({ name: "whisper-1", pricing: { perMinute: 0.006 } });
+
+// gpt-4o-transcribe — defaults to json; billed PER TOKEN like a chat model.
+const gpt = openai.transcribe({ name: "gpt-4o-transcribe", pricing: { input: 2.5, output: 10 } });
+
+const { data, usage } = await ai.transcribe({ model: whisper, audio });
+// data.segments → [{ text, start, end }, …]; usage.cost from report.durationSeconds
+```
+
+The adapter wraps the base64 bytes in an uploadable via the SDK's `toFile`, using `audio.filename` (or `"audio"`) and `audio.mediaType` so the codec is declared correctly.
+
+## Cost-truth — one rollup, two metering models
+
+`ai.transcribe` fills `usage.cost` so STT spend folds into the **same** `Usage.cost` rollup as text:
+
+- **Per-minute** (`whisper-1`): `{ perMinute }` × `(durationSeconds / 60)` → `cost.input`. If the provider didn't report a duration, cost stays **`undefined`** (no guessing).
+- **Token-metered** (`gpt-4o-transcribe`): `{ input, output }` USD-per-1M-tokens → standard `computeCost` against the returned token usage.
+
+Per-minute wins when both are set; an unpriced model leaves `usage.cost` **`undefined`** (honest "cost unknown", never a false zero).
+
+## Pattern — inbound voice-message webhook
+
+```ts
+const stt = openai.transcribe({ name: "whisper-1" });
+
+async function onVoiceMessage(buffer: Buffer, mediaType: string) {
+  const audio = ai.audioFromBuffer(buffer, mediaType, "inbound.ogg");
+  const { data, error } = await ai.transcribe({ model: stt, audio, sessionId: "inbox" });
+
+  if (error) return replyWith("Sorry, I couldn't understand that audio.");
+  return routeToAgent(data.text); // hand the transcript to an ai.agent for a reply
+}
+```
+
+## Observability
+
+The completed `TranscriptionReport` (with `report.durationSeconds` and cost/latency attributed to `report.model`) routes to any registered `Observer` (panoptic, OTel, …) through the shared `observe` seam — `observe: true` (global), an `Observer` (flow-local), or observe-all. See [[observe-ai-flows]]. Provider faults surface as typed `AIError`s on `result.error`; see [[handle-ai-errors]].
+
+## Testing
+
+`MockTranscriptionModel(name, responses, pricing?)` is a deterministic `TranscriptionModelContract` double — no HTTP. Script text/segments/duration/usage/errors and inspect `model.calls`. `MockSDK({ transcriptionResponses, transcriptionPricing }).transcribe({ name })` wires the same double behind a full adapter.
+
+```ts
+import { MockTranscriptionModel, transcribe } from "@warlock.js/ai";
+
+const AUDIO = { base64: "QUJD", mediaType: "audio/mpeg", filename: "clip.mp3" };
+
+const model = new MockTranscriptionModel("whisper-1", [{ durationSeconds: 120 }], { perMinute: 0.006 });
+const { data, usage, report } = await transcribe({ model, audio: AUDIO });
+// data.text → "mock transcript"
+// usage.cost.input → (120 / 60) * 0.006   report.durationSeconds → 120
+// model.calls[0] records { audio, options } for assertions
+```
+
+Scripting `[{ error: new ProviderRateLimitError("slow down") }]` drives the never-throws path — `result.error` is the typed error and `result.data` is `undefined`.
+
+## See also
+
+- [[generate-speech]] — the inverse verb (`ai.speech`), text → audio
+- [[generate-images]] — the sibling image-output verb (`ai.image`)
+- [[observe-ai-flows]] — routing the `TranscriptionReport` to panoptic / OTel
+- [[handle-ai-errors]] — the typed `AIError` taxonomy on `result.error`