markdown-lsp 0.2.2 → 1.1.0

package/README.md

@@ -6,87 +6,238 @@
 [![license](https://img.shields.io/npm/l/markdown-lsp.svg?style=flat-square)](https://github.com/Docsbook-io/markdown-lsp/blob/main/LICENSE)
 [![node](https://img.shields.io/node/v/markdown-lsp.svg?style=flat-square)](https://www.npmjs.com/package/markdown-lsp)
 
-Language Server Protocol implementation for Markdown documentation. Optional AI-powered semantic layer on top.
+CLI and library for querying Markdown documentation graphs. Point it at a folder of `.md` files and get instant full-text search, outline, link analysis, and symbol lookup — all as JSON.
 
-**Status: M1 complete, M2 (AI layer) opt-in.**
+**Status: v1.1.0. CLI is the default interface. LSP stdio mode available as a subcommand.**
 
-## Two layers
+---
 
-### Structural (default, no AI)
+## Quick Start
 
-Out of the box — like Marksman, but persisted in Postgres and addressable from a service.
+```bash
+# List all pages
+npx markdown-lsp workspace-outline ./docs
+
+# Full-text search (natural-language, ranked)
+npx markdown-lsp search-text ./docs "getting started"
+
+# Fuzzy heading search
+npx markdown-lsp search-symbols ./docs "auth" --limit 10
+```
+
+---
+
+## Installation
+
+```bash
+npm install -g markdown-lsp
+# or per-project
+npm install markdown-lsp
+```
+
+Node.js >= 20 required.
 
-- `textDocument/documentSymbol` — heading outline
-- `workspace/symbol` — fuzzy subsequence search across all headings (e.g. `oaf` matches `OAuth flow`)
-- `textDocument/definition` — jump from a link to its target document
-- `textDocument/references` — find every page linking to the current document
-- `textDocument/completion` — wiki-link completion `[[...]]`
-- `textDocument/publishDiagnostics` — warnings for unresolved link targets
-- `workspace.executeCommand("markdownLsp/reindex")` — force re-index of the workspace
-- Incremental indexing via content-hash diff; watched-files cleanup
+---
 
-This layer is fully deterministic, free, and runs offline against your Postgres.
+## Subcommands
 
-### Semantic (optional, AI-powered)
+All subcommands accept a **`--pretty`** flag for indented JSON output (compact by default).
 
-Off by default. When enabled, an extract pass identifies canonical concepts per section so that
-references survive synonym variation (`auth` ≡ `authentication` ≡ `OAuth` ≡ `login`).
+| Subcommand | Arguments | Description |
+|---|---|---|
+| `workspace-outline` | `<docs-dir> [--prefix p] [--limit n]` | List all pages with metadata |
+| `outline` | `<docs-dir> <page>` | Heading outline of a single page |
+| `search-text` | `<docs-dir> <query> [--mode ranked\|verbatim] [--regex] [--case-sensitive] [--prefix p] [--limit n] [--context n]` | Full-text search |
+| `search-symbols` | `<docs-dir> <query> [--limit n]` | Fuzzy subsequence search across headings |
+| `search-paths` | `<docs-dir> <glob>` | List pages matching a glob pattern |
+| `links-to` | `<docs-dir> <page>` | All pages that link to `<page>` |
+| `links-from` | `<docs-dir> <page>` | All links originating from `<page>` |
+| `resolve-link` | `<docs-dir> <from-page> <link-text>` | Resolve a specific link text from a page |
+| `get-section` | `<docs-dir> <page> <anchor>` | Get a section by anchor slug |
+| `lsp` / `serve` | `[--stdio]` | Start the LSP stdio server |
+| `graph` | `<docs-dir> [--format json\|dot\|mermaid\|html] [--out file]` | Export the doc link graph |
+| `semantic-search` | `<docs-dir> <query> [--limit n] [--model model]` | AI semantic search via embeddings |
 
-Enable with:
+### search-text modes
+
+- **ranked** (default) — tokenizes query, drops stop words, ranks by coverage/heading/proximity. Best for natural-language questions.
+- **verbatim** (`--mode verbatim`) — literal substring match. Use `--regex` for regex.
+
+### Output format
+
+All subcommands print JSON to stdout. Use `--pretty` for human-readable output:
 
 ```bash
-export MARKDOWN_LSP_AI_ENABLED=1
-export AI_GATEWAY_API_KEY=...   # Vercel AI Gateway
+markdown-lsp search-text ./docs "authentication" --limit 5 --pretty
 ```
 
-If the flag is not set, no AI calls are ever made. The server starts and behaves as a
-pure-structural LSP — no key required.
+---
 
-## Architecture
+## Examples
+
+```bash
+# Workspace overview
+markdown-lsp workspace-outline ./docs --limit 20 --pretty
+
+# Find pages about authentication
+markdown-lsp search-text ./docs "authentication flow" --pretty
 
-- LSP over stdio (`vscode-languageserver/node`) — works in any editor
-- pgvector (Neon serverless) for cosine search on canonical-term embeddings (only when AI layer is enabled)
-- Drizzle ORM; all tables prefixed `mdlsp_`
-- Vercel AI Gateway (`text-embedding-3-small` for embeddings, `gpt-4o-mini` for extraction) — when AI on
-- An optional MCP HTTP facade (M3) over the same handlers — for AI agents like Claude Code
+# What links to README.md?
+markdown-lsp links-to ./docs README.md
 
-## Setup
+# Glob search: all files under api/
+markdown-lsp search-paths ./docs "api/**"
+
+# Outline of a specific page
+markdown-lsp outline ./docs quick-start.md --pretty
+
+# Find headings containing "auth"
+markdown-lsp search-symbols ./docs "auth" --limit 10
+
+# Resolve a link from a page
+markdown-lsp resolve-link ./docs README.md "Getting Started"
+
+# Get a specific section by anchor
+markdown-lsp get-section ./docs overview.md quick-links --pretty
+```
+
+---
+
+## Graph export
+
+Export the full page link graph — nodes are pages, edges are markdown links.
 
 ```bash
-pnpm install
-cp .env.example .env.local        # fill DATABASE_URL; AI_GATEWAY_API_KEY only if you want the AI layer
-pnpm migrate                       # runs scripts/apply-migration.ts against DATABASE_URL
-pnpm build
+# JSON (nodes + edges — machine-readable)
+markdown-lsp graph ./docs --format json --pretty
+
+# Graphviz DOT
+markdown-lsp graph ./docs --format dot > graph.dot
+
+# Mermaid flowchart (embed in markdown)
+markdown-lsp graph ./docs --format mermaid
+
+# Self-contained interactive HTML with D3 force-directed graph
+# (drag, zoom, hover highlights neighbours, click to inspect)
+markdown-lsp graph ./docs --format html --out graph.html
 ```
 
-## Run
+JSON output shape:
+```json
+{
+  "nodes": [{"id": "README.md", "title": "Docsbook", "charCount": 2634, "sectionsCount": 10}],
+  "edges": [{"source": "README.md", "target": "quick-start.md", "kind": "inline", "label": "Get started"}],
+  "unresolvedCount": 3
+}
+```
+
+---
 
-LSP via stdio (for editor integration):
+## Semantic search
+
+AI-powered semantic search using text embeddings — finds conceptually related pages even if they
+don't contain the exact query words.
 
 ```bash
-node dist/server.js --stdio
+# Requires OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway)
+OPENROUTER_API_KEY=sk-or-... markdown-lsp semantic-search ./docs "how to configure webhooks" --limit 5
+
+# Override embedding model
+markdown-lsp semantic-search ./docs "authentication" --model openai/text-embedding-3-small --limit 3
 ```
 
-`bin/markdown-lsp` wraps the same entry point as a CLI.
+- Default embedding model: `openai/text-embedding-3-small` (via OpenRouter — model prefix required)
+- Results cached in `.markdown-lsp-cache/embeddings/` — second run is instant, no API call
+- Returns `[{ pagePath, pageTitle, score, snippet }]` sorted by cosine similarity
 
-## Use from Docsbook
+**Environment variables:**
 
-The structural layer is what Docsbook's "Source of Truth" feature wants. Wire it in like this:
+| Variable | Purpose |
+|---|---|
+| `OPENROUTER_API_KEY` | OpenRouter API key (takes priority if set) |
+| `AI_GATEWAY_API_KEY` | Vercel AI Gateway key (fallback) |
+| `EMBEDDING_MODEL` | Override default embedding model |
+
+---
+
+## LSP mode (editor integration)
+
+`markdown-lsp` also works as a Language Server Protocol server for editors (VS Code, Zed, Neovim, etc.).
+
+```bash
+# Recommended (v1.0.0+)
+npx markdown-lsp lsp --stdio
+
+# Back-compat — old LSP flag style still works so existing editor configs don't break
+npx markdown-lsp --stdio
+```
+
+The LSP server speaks the standard protocol over stdio. It requires a Postgres database for the structural index (see Setup below).
+
+### Editor configuration example (VS Code)
+
+In your `settings.json`:
+
+```json
+{
+  "markdown-lsp.serverPath": "markdown-lsp",
+  "markdown-lsp.args": ["lsp", "--stdio"]
+}
+```
+
+---
+
+## Use as a library
 
 ```ts
-import { ensureWorkspace, indexWorkspace } from "@docsbook/markdown-lsp/indexer"
-import { getDocumentSymbols, getWorkspaceSymbols } from "@docsbook/markdown-lsp/core"
+import { buildGraph, loadDocsAsFiles } from "markdown-lsp/graph"
+import { searchTextRanked, searchSymbols, listPages } from "markdown-lsp/bridge"
+
+const graph = buildGraph("./docs")
+const hits = searchTextRanked(graph, "authentication flow")
+const pages = listPages(graph, { limit: 50 })
+```
+
+Available entry points:
+- `markdown-lsp/bridge` — search functions + `RichDocGraph`, `buildInMemoryGraph`, types
+- `markdown-lsp/graph` — `buildGraph(docsRoot)`, `loadDocsAsFiles(docsRoot)`
+- `markdown-lsp/indexer` — SQLite/Postgres workspace indexer (for LSP use)
+- `markdown-lsp/core` — document symbols and references (for LSP use)
+- `markdown-lsp/parser` — raw Markdown parser
+
+---
+
+## LSP Setup (for editor / structural indexer use)
 
-// after cloning a workspace repo into ./tmp/<workspace-id>/
-const ws = await ensureWorkspace("./tmp/42")
-await indexWorkspace(ws)
+The CLI subcommands work **without any database** — they build an in-memory graph on the fly.
 
-// MCP tools then call:
-await getWorkspaceSymbols(ws, "auth")
-await findReferencesToDocument(ws, authDocId)
+The LSP server requires Postgres (for the incremental index):
+
+```bash
+pnpm install
+cp .env.example .env.local        # fill DATABASE_URL; AI_GATEWAY_API_KEY only if you want the AI layer
+pnpm migrate                       # runs scripts/apply-migration.ts against DATABASE_URL
+pnpm build
+```
+
+Optional AI layer (semantic synonym resolution):
+
+```bash
+export MARKDOWN_LSP_AI_ENABLED=1
+export AI_GATEWAY_API_KEY=...   # Vercel AI Gateway
 ```
 
-No AI required.
+---
+
+## Architecture
+
+- **CLI** — `node:util parseArgs`, zero extra deps, reads `.md` files into an in-memory graph
+- **Graph** — pure TypeScript, no DB needed; `buildGraph(docsRoot)` walks the directory tree
+- **LSP** — `vscode-languageserver/node` over stdio; requires Postgres (Drizzle ORM, `mdlsp_` prefix)
+- **AI layer** (opt-in) — pgvector cosine search on canonical-term embeddings; `text-embedding-3-small` via Vercel AI Gateway
+- **Bridge** — pure in-memory search (searchText, searchTextRanked, searchSymbols, searchPaths, listPages)
+
+---
 
 ## Tests
 
@@ -94,17 +245,21 @@ No AI required.
 pnpm test
 ```
 
-27 tests cover the parser, indexer, and core handlers (plus a small suite for the AI feature flag).
+27 tests cover the parser, indexer, core handlers, and bridge search functions.
+
+---
 
 ## Milestones
 
 - **M0 — Scaffold** ✅
 - **M1 — Structural layer** ✅
-- M2 — Semantic extract (opt-in, code present, awaiting live AI Gateway credit)
-- M3 — MCP HTTP facade
+- **M2 — Semantic extract** (opt-in, code present, awaiting live AI Gateway credit)
+- **M3 — CLI-first interface** ✅ (v1.0.0)
 - M4 — User overrides for the glossary (merge / split / rename / add_synonym)
 - M5 — Docsbook integration
 
+---
+
 ## License
 
 MIT

package/bin/markdown-lsp

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import("../dist/server.js")
+import("../dist/cli.js")

package/dist/ai/cache.d.ts

@@ -0,0 +1,3 @@
+export declare function cachedEmbedding(text: string, model: string): number[] | null;
+export declare function saveEmbedding(text: string, model: string, vec: number[]): void;
+//# sourceMappingURL=cache.d.ts.map

package/dist/ai/cache.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.d.ts","sourceRoot":"","sources":["../../src/ai/cache.ts"],"names":[],"mappings":"AAUA,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,IAAI,CAW5E;AAED,wBAAgB,aAAa,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,IAAI,CAK9E"}

package/dist/ai/cache.js

@@ -0,0 +1,27 @@
+import { createHash } from "node:crypto";
+import { readFileSync, writeFileSync, mkdirSync, existsSync } from "node:fs";
+import { join } from "node:path";
+const CACHE_DIR = ".markdown-lsp-cache/embeddings";
+function cacheKey(text, model) {
+    return createHash("sha256").update(model + "\x00" + text).digest("hex");
+}
+export function cachedEmbedding(text, model) {
+    const key = cacheKey(text, model);
+    const filePath = join(CACHE_DIR, `${key}.json`);
+    if (existsSync(filePath)) {
+        try {
+            return JSON.parse(readFileSync(filePath, "utf8"));
+        }
+        catch {
+            return null;
+        }
+    }
+    return null;
+}
+export function saveEmbedding(text, model, vec) {
+    mkdirSync(CACHE_DIR, { recursive: true });
+    const key = cacheKey(text, model);
+    const filePath = join(CACHE_DIR, `${key}.json`);
+    writeFileSync(filePath, JSON.stringify(vec), "utf8");
+}
+//# sourceMappingURL=cache.js.map

package/dist/ai/cache.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.js","sourceRoot":"","sources":["../../src/ai/cache.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAA;AACxC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,SAAS,CAAA;AAC5E,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAEhC,MAAM,SAAS,GAAG,gCAAgC,CAAA;AAElD,SAAS,QAAQ,CAAC,IAAY,EAAE,KAAa;IAC3C,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAA;AACzE,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,IAAY,EAAE,KAAa;IACzD,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACjC,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,OAAO,CAAC,CAAA;IAC/C,IAAI,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;QACzB,IAAI,CAAC;YACH,OAAO,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAa,CAAA;QAC/D,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,IAAI,CAAA;QACb,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAA;AACb,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,IAAY,EAAE,KAAa,EAAE,GAAa;IACtE,SAAS,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;IACzC,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACjC,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,OAAO,CAAC,CAAA;IAC/C,aAAa,CAAC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC,CAAA;AACtD,CAAC"}

package/dist/ai/config.d.ts

@@ -3,7 +3,13 @@ export interface AiConfig {
     embeddingModel: string;
     extractModel: string;
     hasGatewayKey: boolean;
+    hasOpenRouterKey: boolean;
 }
 export declare function getAiConfig(): AiConfig;
 export declare function assertAiEnabled(): void;
+/**
+ * Lighter check for CLI commands (semantic-search) that don't need MARKDOWN_LSP_AI_ENABLED.
+ * Only checks that at least one API key is present. Exits with a clear message on failure.
+ */
+export declare function assertApiKey(): void;
 //# sourceMappingURL=config.d.ts.map

package/dist/ai/config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,QAAQ;IACvB,OAAO,EAAE,OAAO,CAAA;IAChB,cAAc,EAAE,MAAM,CAAA;IACtB,YAAY,EAAE,MAAM,CAAA;IACpB,aAAa,EAAE,OAAO,CAAA;CACvB;AAED,wBAAgB,WAAW,IAAI,QAAQ,CAQtC;AAED,wBAAgB,eAAe,IAAI,IAAI,CAYtC"}
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,QAAQ;IACvB,OAAO,EAAE,OAAO,CAAA;IAChB,cAAc,EAAE,MAAM,CAAA;IACtB,YAAY,EAAE,MAAM,CAAA;IACpB,aAAa,EAAE,OAAO,CAAA;IACtB,gBAAgB,EAAE,OAAO,CAAA;CAC1B;AAED,wBAAgB,WAAW,IAAI,QAAQ,CAStC;AAED,wBAAgB,eAAe,IAAI,IAAI,CAYtC;AAED;;;GAGG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAQnC"}

package/dist/ai/config.js

@@ -1,10 +1,12 @@
+import { DEFAULT_EMBEDDING_MODEL } from "./gateway.js";
 export function getAiConfig() {
     const enabled = process.env.MARKDOWN_LSP_AI_ENABLED === "1" || process.env.MARKDOWN_LSP_AI_ENABLED === "true";
     return {
         enabled,
-        embeddingModel: process.env.EMBEDDING_MODEL ?? "text-embedding-3-small",
+        embeddingModel: process.env.EMBEDDING_MODEL ?? DEFAULT_EMBEDDING_MODEL,
         extractModel: process.env.EXTRACT_MODEL ?? "gpt-4o-mini",
         hasGatewayKey: Boolean(process.env.AI_GATEWAY_API_KEY),
+        hasOpenRouterKey: Boolean(process.env.OPENROUTER_API_KEY),
     };
 }
 export function assertAiEnabled() {
@@ -12,8 +14,19 @@ export function assertAiEnabled() {
     if (!cfg.enabled) {
         throw new Error("AI features are disabled. Set MARKDOWN_LSP_AI_ENABLED=1 to enable embeddings + semantic extraction.");
     }
-    if (!cfg.hasGatewayKey) {
-        throw new Error("AI features require AI_GATEWAY_API_KEY (Vercel AI Gateway) — or fork the gateway module to use a different provider.");
+    if (!cfg.hasGatewayKey && !cfg.hasOpenRouterKey) {
+        throw new Error("AI features require OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway).");
+    }
+}
+/**
+ * Lighter check for CLI commands (semantic-search) that don't need MARKDOWN_LSP_AI_ENABLED.
+ * Only checks that at least one API key is present. Exits with a clear message on failure.
+ */
+export function assertApiKey() {
+    const cfg = getAiConfig();
+    if (!cfg.hasGatewayKey && !cfg.hasOpenRouterKey) {
+        process.stderr.write("Error: set OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway) to use semantic-search.\n");
+        process.exit(1);
     }
 }
 //# sourceMappingURL=config.js.map

package/dist/ai/config.js.map

@@ -1 +1 @@
-{"version":3,"file":"config.js","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAOA,MAAM,UAAU,WAAW;IACzB,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,MAAM,CAAA;IAC7G,OAAO;QACL,OAAO;QACP,cAAc,EAAE,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,wBAAwB;QACvE,YAAY,EAAE,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa;QACxD,aAAa,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;KACvD,CAAA;AACH,CAAC;AAED,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC;QACjB,MAAM,IAAI,KAAK,CACb,qGAAqG,CACtG,CAAA;IACH,CAAC;IACD,IAAI,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC;QACvB,MAAM,IAAI,KAAK,CACb,sHAAsH,CACvH,CAAA;IACH,CAAC;AACH,CAAC"}
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,uBAAuB,EAAE,MAAM,cAAc,CAAA;AAUtD,MAAM,UAAU,WAAW;IACzB,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,MAAM,CAAA;IAC7G,OAAO;QACL,OAAO;QACP,cAAc,EAAE,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,uBAAuB;QACtE,YAAY,EAAE,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa;QACxD,aAAa,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;QACtD,gBAAgB,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;KAC1D,CAAA;AACH,CAAC;AAED,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC;QACjB,MAAM,IAAI,KAAK,CACb,qGAAqG,CACtG,CAAA;IACH,CAAC;IACD,IAAI,CAAC,GAAG,CAAC,aAAa,IAAI,CAAC,GAAG,CAAC,gBAAgB,EAAE,CAAC;QAChD,MAAM,IAAI,KAAK,CACb,gGAAgG,CACjG,CAAA;IACH,CAAC;AACH,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,YAAY;IAC1B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,aAAa,IAAI,CAAC,GAAG,CAAC,gBAAgB,EAAE,CAAC;QAChD,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,gHAAgH,CACjH,CAAA;QACD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;IACjB,CAAC;AACH,CAAC"}

package/dist/ai/embeddings.d.ts

@@ -2,6 +2,13 @@ export interface EmbeddingResult {
     vectors: number[][];
     tokensUsed: number;
 }
-export declare function embedTexts(texts: string[]): Promise<EmbeddingResult>;
-export declare function embedOne(text: string): Promise<number[]>;
+/**
+ * Embed multiple texts with disk-cache support.
+ * Checks cache per-text before hitting the API; saves results after.
+ * @param texts - texts to embed
+ * @param modelOverride - override the embedding model (default: EMBEDDING_MODEL env or openai/text-embedding-3-small)
+ * @param useCache - whether to use disk cache (default: true)
+ */
+export declare function embedTexts(texts: string[], modelOverride?: string, useCache?: boolean): Promise<EmbeddingResult>;
+export declare function embedOne(text: string, modelOverride?: string): Promise<number[]>;
 //# sourceMappingURL=embeddings.d.ts.map

package/dist/ai/embeddings.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"embeddings.d.ts","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAOA,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,EAAE,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;CACnB;AAED,wBAAsB,UAAU,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,eAAe,CAAC,CAiB1E;AAED,wBAAsB,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAG9D"}
+{"version":3,"file":"embeddings.d.ts","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAQA,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,EAAE,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;CACnB;AAED;;;;;;GAMG;AACH,wBAAsB,UAAU,CAC9B,KAAK,EAAE,MAAM,EAAE,EACf,aAAa,CAAC,EAAE,MAAM,EACtB,QAAQ,UAAO,GACd,OAAO,CAAC,eAAe,CAAC,CAgD1B;AAED,wBAAsB,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,aAAa,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAGtF"}

package/dist/ai/embeddings.js

@@ -1,27 +1,65 @@
 import { embedMany } from "ai";
 import { getGateway, EMBEDDING_MODEL } from "./gateway.js";
 import { assertAiEnabled } from "./config.js";
+import { cachedEmbedding, saveEmbedding } from "./cache.js";
 const BATCH_SIZE = 96;
 const MAX_INPUT_CHARS = 6000;
-export async function embedTexts(texts) {
+/**
+ * Embed multiple texts with disk-cache support.
+ * Checks cache per-text before hitting the API; saves results after.
+ * @param texts - texts to embed
+ * @param modelOverride - override the embedding model (default: EMBEDDING_MODEL env or openai/text-embedding-3-small)
+ * @param useCache - whether to use disk cache (default: true)
+ */
+export async function embedTexts(texts, modelOverride, useCache = true) {
     if (texts.length === 0)
         return { vectors: [], tokensUsed: 0 };
     assertAiEnabled();
     const gw = getGateway();
-    const model = gw.embedding(EMBEDDING_MODEL);
-    const out = [];
+    const modelName = modelOverride ?? EMBEDDING_MODEL;
+    const model = gw.embedding(modelName);
+    const out = new Array(texts.length).fill(null);
     let tokensUsed = 0;
-    for (let i = 0; i < texts.length; i += BATCH_SIZE) {
-        const batch = texts.slice(i, i + BATCH_SIZE).map((t) => t.slice(0, MAX_INPUT_CHARS));
-        const res = await embedMany({ model, values: batch });
-        for (const v of res.embeddings)
-            out.push(v);
+    // Check cache first
+    const uncachedIndices = [];
+    const uncachedTexts = [];
+    if (useCache) {
+        for (let i = 0; i < texts.length; i++) {
+            const cached = cachedEmbedding(texts[i], modelName);
+            if (cached !== null) {
+                out[i] = cached;
+            }
+            else {
+                uncachedIndices.push(i);
+                uncachedTexts.push(texts[i]);
+            }
+        }
+    }
+    else {
+        for (let i = 0; i < texts.length; i++) {
+            uncachedIndices.push(i);
+            uncachedTexts.push(texts[i]);
+        }
+    }
+    // Embed uncached texts in batches
+    for (let b = 0; b < uncachedTexts.length; b += BATCH_SIZE) {
+        const batchTexts = uncachedTexts.slice(b, b + BATCH_SIZE).map((t) => t.slice(0, MAX_INPUT_CHARS));
+        const batchIndices = uncachedIndices.slice(b, b + BATCH_SIZE);
+        const res = await embedMany({ model, values: batchTexts });
+        for (let j = 0; j < res.embeddings.length; j++) {
+            const vec = res.embeddings[j];
+            const originalIndex = batchIndices[j];
+            out[originalIndex] = vec;
+            if (useCache) {
+                saveEmbedding(uncachedTexts[b + j], modelName, vec);
+            }
+        }
         tokensUsed += res.usage?.tokens ?? 0;
     }
     return { vectors: out, tokensUsed };
 }
-export async function embedOne(text) {
-    const { vectors } = await embedTexts([text]);
+export async function embedOne(text, modelOverride) {
+    const { vectors } = await embedTexts([text], modelOverride);
     return vectors[0];
 }
 //# sourceMappingURL=embeddings.js.map

package/dist/ai/embeddings.js.map

@@ -1 +1 @@
-{"version":3,"file":"embeddings.js","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,IAAI,CAAA;AAC9B,OAAO,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAA;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AAE7C,MAAM,UAAU,GAAG,EAAE,CAAA;AACrB,MAAM,eAAe,GAAG,IAAI,CAAA;AAO5B,MAAM,CAAC,KAAK,UAAU,UAAU,CAAC,KAAe;IAC9C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAA;IAC7D,eAAe,EAAE,CAAA;IACjB,MAAM,EAAE,GAAG,UAAU,EAAE,CAAA;IACvB,MAAM,KAAK,GAAG,EAAE,CAAC,SAAS,CAAC,eAAe,CAAC,CAAA;IAE3C,MAAM,GAAG,GAAe,EAAE,CAAA;IAC1B,IAAI,UAAU,GAAG,CAAC,CAAA;IAElB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,UAAU,EAAE,CAAC;QAClD,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,eAAe,CAAC,CAAC,CAAA;QACpF,MAAM,GAAG,GAAG,MAAM,SAAS,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,CAAA;QACrD,KAAK,MAAM,CAAC,IAAI,GAAG,CAAC,UAAU;YAAE,GAAG,CAAC,IAAI,CAAC,CAAa,CAAC,CAAA;QACvD,UAAU,IAAI,GAAG,CAAC,KAAK,EAAE,MAAM,IAAI,CAAC,CAAA;IACtC,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,GAAG,EAAE,UAAU,EAAE,CAAA;AACrC,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,IAAY;IACzC,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,UAAU,CAAC,CAAC,IAAI,CAAC,CAAC,CAAA;IAC5C,OAAO,OAAO,CAAC,CAAC,CAAE,CAAA;AACpB,CAAC"}
+{"version":3,"file":"embeddings.js","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,IAAI,CAAA;AAC9B,OAAO,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAA;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AAC7C,OAAO,EAAE,eAAe,EAAE,aAAa,EAAE,MAAM,YAAY,CAAA;AAE3D,MAAM,UAAU,GAAG,EAAE,CAAA;AACrB,MAAM,eAAe,GAAG,IAAI,CAAA;AAO5B;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,UAAU,CAC9B,KAAe,EACf,aAAsB,EACtB,QAAQ,GAAG,IAAI;IAEf,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAA;IAC7D,eAAe,EAAE,CAAA;IACjB,MAAM,EAAE,GAAG,UAAU,EAAE,CAAA;IACvB,MAAM,SAAS,GAAG,aAAa,IAAI,eAAe,CAAA;IAClD,MAAM,KAAK,GAAG,EAAE,CAAC,SAAS,CAAC,SAAS,CAAC,CAAA;IAErC,MAAM,GAAG,GAAwB,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;IACnE,IAAI,UAAU,GAAG,CAAC,CAAA;IAElB,oBAAoB;IACpB,MAAM,eAAe,GAAa,EAAE,CAAA;IACpC,MAAM,aAAa,GAAa,EAAE,CAAA;IAElC,IAAI,QAAQ,EAAE,CAAC;QACb,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,MAAM,MAAM,GAAG,eAAe,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,SAAS,CAAC,CAAA;YACpD,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;gBACpB,GAAG,CAAC,CAAC,CAAC,GAAG,MAAM,CAAA;YACjB,CAAC;iBAAM,CAAC;gBACN,eAAe,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;gBACvB,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAE,CAAC,CAAA;YAC/B,CAAC;QACH,CAAC;IACH,CAAC;SAAM,CAAC;QACN,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,eAAe,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;YACvB,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAE,CAAC,CAAA;QAC/B,CAAC;IACH,CAAC;IAED,kCAAkC;IAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,aAAa,CAAC,MAAM,EAAE,CAAC,IAAI,UAAU,EAAE,CAAC;QAC1D,MAAM,UAAU,GAAG,aAAa,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,eAAe,CAAC,CAAC,CAAA;QACjG,MAAM,YAAY,GAAG,eAAe,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAA;QAC7D,MAAM,GAAG,GAAG,MAAM,SAAS,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,UAAU,EAAE,CAAC,CAAA;QAC1D,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,GAAG,CAAC,UAAU,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC/C,MAAM,GAAG,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAa,CAAA;YACzC,MAAM,aAAa,GAAG,YAAY,CAAC,CAAC,CAAE,CAAA;YACtC,GAAG,CAAC,aAAa,CAAC,GAAG,GAAG,CAAA;YACxB,IAAI,QAAQ,EAAE,CAAC;gBACb,aAAa,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAE,EAAE,SAAS,EAAE,GAAG,CAAC,CAAA;YACtD,CAAC;QACH,CAAC;QACD,UAAU,IAAI,GAAG,CAAC,KAAK,EAAE,MAAM,IAAI,CAAC,CAAA;IACtC,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,GAAiB,EAAE,UAAU,EAAE,CAAA;AACnD,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,IAAY,EAAE,aAAsB;IACjE,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,UAAU,CAAC,CAAC,IAAI,CAAC,EAAE,aAAa,CAAC,CAAA;IAC3D,OAAO,OAAO,CAAC,CAAC,CAAE,CAAA;AACpB,CAAC"}

package/dist/ai/gateway.d.ts

@@ -1,5 +1,8 @@
+export declare const DEFAULT_EMBEDDING_MODEL = "openai/text-embedding-3-small";
 export declare const EMBEDDING_MODEL: string;
 export declare const EMBEDDING_DIM = 1536;
 export declare const EXTRACT_MODEL: string;
 export declare function getGateway(): import("@ai-sdk/openai").OpenAIProvider;
+/** Reset the cached gateway (useful for tests or when env changes mid-process) */
+export declare function resetGateway(): void;
 //# sourceMappingURL=gateway.d.ts.map

package/dist/ai/gateway.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"gateway.d.ts","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAEA,eAAO,MAAM,eAAe,QAA0D,CAAA;AACtF,eAAO,MAAM,aAAa,OAAO,CAAA;AACjC,eAAO,MAAM,aAAa,QAA6C,CAAA;AAIvE,wBAAgB,UAAU,4CASzB"}
+{"version":3,"file":"gateway.d.ts","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAEA,eAAO,MAAM,uBAAuB,kCAAkC,CAAA;AACtE,eAAO,MAAM,eAAe,QAAyD,CAAA;AACrF,eAAO,MAAM,aAAa,OAAO,CAAA;AACjC,eAAO,MAAM,aAAa,QAA6C,CAAA;AAIvE,wBAAgB,UAAU,4CAoBzB;AAED,kFAAkF;AAClF,wBAAgB,YAAY,IAAI,IAAI,CAEnC"}

package/dist/ai/gateway.js

@@ -1,18 +1,27 @@
 import { createOpenAI } from "@ai-sdk/openai";
-export const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL ?? "text-embedding-3-small";
+export const DEFAULT_EMBEDDING_MODEL = "openai/text-embedding-3-small";
+export const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL ?? DEFAULT_EMBEDDING_MODEL;
 export const EMBEDDING_DIM = 1536;
 export const EXTRACT_MODEL = process.env.EXTRACT_MODEL ?? "gpt-4o-mini";
 let _gw = null;
 export function getGateway() {
     if (_gw)
         return _gw;
-    const apiKey = process.env.AI_GATEWAY_API_KEY;
-    if (!apiKey)
-        throw new Error("AI_GATEWAY_API_KEY is not set");
-    _gw = createOpenAI({
-        apiKey,
-        baseURL: process.env.AI_GATEWAY_BASE_URL ?? "https://ai-gateway.vercel.sh/v1",
-    });
+    // OpenRouter takes priority if OPENROUTER_API_KEY is set
+    const openrouterKey = process.env.OPENROUTER_API_KEY;
+    const gatewayKey = process.env.AI_GATEWAY_API_KEY;
+    const apiKey = openrouterKey ?? gatewayKey;
+    if (!apiKey) {
+        throw new Error("Set OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway) to use AI features.");
+    }
+    const baseURL = openrouterKey
+        ? (process.env.AI_GATEWAY_BASE_URL ?? "https://openrouter.ai/api/v1")
+        : (process.env.AI_GATEWAY_BASE_URL ?? "https://ai-gateway.vercel.sh/v1");
+    _gw = createOpenAI({ apiKey, baseURL });
     return _gw;
 }
+/** Reset the cached gateway (useful for tests or when env changes mid-process) */
+export function resetGateway() {
+    _gw = null;
+}
 //# sourceMappingURL=gateway.js.map

package/dist/ai/gateway.js.map

@@ -1 +1 @@
-{"version":3,"file":"gateway.js","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAA;AAE7C,MAAM,CAAC,MAAM,eAAe,GAAG,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,wBAAwB,CAAA;AACtF,MAAM,CAAC,MAAM,aAAa,GAAG,IAAI,CAAA;AACjC,MAAM,CAAC,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa,CAAA;AAEvE,IAAI,GAAG,GAA2C,IAAI,CAAA;AAEtD,MAAM,UAAU,UAAU;IACxB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAA;IACnB,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IAC7C,IAAI,CAAC,MAAM;QAAE,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAA;IAC7D,GAAG,GAAG,YAAY,CAAC;QACjB,MAAM;QACN,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,iCAAiC;KAC9E,CAAC,CAAA;IACF,OAAO,GAAG,CAAA;AACZ,CAAC"}
+{"version":3,"file":"gateway.js","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAA;AAE7C,MAAM,CAAC,MAAM,uBAAuB,GAAG,+BAA+B,CAAA;AACtE,MAAM,CAAC,MAAM,eAAe,GAAG,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,uBAAuB,CAAA;AACrF,MAAM,CAAC,MAAM,aAAa,GAAG,IAAI,CAAA;AACjC,MAAM,CAAC,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa,CAAA;AAEvE,IAAI,GAAG,GAA2C,IAAI,CAAA;AAEtD,MAAM,UAAU,UAAU;IACxB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAA;IAEnB,yDAAyD;IACzD,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IACpD,MAAM,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IACjD,MAAM,MAAM,GAAG,aAAa,IAAI,UAAU,CAAA;IAE1C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,mGAAmG,CACpG,CAAA;IACH,CAAC;IAED,MAAM,OAAO,GAAG,aAAa;QAC3B,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,8BAA8B,CAAC;QACrE,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,iCAAiC,CAAC,CAAA;IAE1E,GAAG,GAAG,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAA;IACvC,OAAO,GAAG,CAAA;AACZ,CAAC;AAED,kFAAkF;AAClF,MAAM,UAAU,YAAY;IAC1B,GAAG,GAAG,IAAI,CAAA;AACZ,CAAC"}

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":""}