markdown-lsp 1.0.0 → 1.2.0

package/README.md

@@ -6,9 +6,9 @@
 [![license](https://img.shields.io/npm/l/markdown-lsp.svg?style=flat-square)](https://github.com/Docsbook-io/markdown-lsp/blob/main/LICENSE)
 [![node](https://img.shields.io/node/v/markdown-lsp.svg?style=flat-square)](https://www.npmjs.com/package/markdown-lsp)
 
-CLI and library for querying Markdown documentation graphs. Point it at a folder of `.md` files and get instant full-text search, outline, link analysis, and symbol lookup — all as JSON.
+CLI and library for querying Markdown documentation graphs. Point it at a folder of `.md` files and get instant full-text search, outline, link analysis, symbol lookup, interactive HTML graphs, and AI-powered semantic search — all as JSON.
 
-**Status: v1.0.0. CLI is the default interface. LSP stdio mode available as a subcommand.**
+**Status: v1.2.0. CLI is the default interface. LSP stdio mode available as a subcommand.**
 
 ---
 
@@ -18,11 +18,40 @@ CLI and library for querying Markdown documentation graphs. Point it at a folder
 # List all pages
 npx markdown-lsp workspace-outline ./docs
 
+# Heading outline of a page
+npx markdown-lsp outline ./docs introduction.md
+
 # Full-text search (natural-language, ranked)
 npx markdown-lsp search-text ./docs "getting started"
 
 # Fuzzy heading search
 npx markdown-lsp search-symbols ./docs "auth" --limit 10
+
+# Find pages by glob
+npx markdown-lsp search-paths ./docs "ai/*.md"
+
+# Backlinks and outgoing links
+npx markdown-lsp links-to ./docs quick-start.md
+npx markdown-lsp links-from ./docs README.md
+
+# Resolve a link text / read a section
+npx markdown-lsp resolve-link ./docs README.md "Getting Started"
+npx markdown-lsp get-section ./docs overview.md "quick-links"
+
+# Interactive link graph (HTML, JSON, DOT, Mermaid)
+npx markdown-lsp graph ./docs --format html --out graph.html
+npx markdown-lsp graph ./docs --format json --pretty
+npx markdown-lsp graph ./docs --format dot | dot -Tsvg > graph.svg
+npx markdown-lsp graph ./docs --format mermaid
+
+# Build embeddings + interactive semantic graph in one command
+OPENROUTER_API_KEY=sk-or-... npx markdown-lsp graph ./docs --format html --semantic --out graph.html
+
+# AI semantic search
+OPENROUTER_API_KEY=sk-or-... npx markdown-lsp semantic-search ./docs "how to configure webhooks"
+
+# LSP server (for editors)
+npx markdown-lsp lsp --stdio
 ```
 
 ---
@@ -54,6 +83,8 @@ All subcommands accept a **`--pretty`** flag for indented JSON output (compact b
 | `links-from` | `<docs-dir> <page>` | All links originating from `<page>` |
 | `resolve-link` | `<docs-dir> <from-page> <link-text>` | Resolve a specific link text from a page |
 | `get-section` | `<docs-dir> <page> <anchor>` | Get a section by anchor slug |
+| `graph` | `<docs-dir> [--format json\|dot\|mermaid\|html] [--out file] [--semantic] [--sim-threshold n] [--sim-top-k n] [--model m]` | Export doc link graph; `--semantic` adds AI similarity edges |
+| `semantic-search` | `<docs-dir> <query> [--limit n] [--model m]` | AI semantic search via embeddings |
 | `lsp` / `serve` | `[--stdio]` | Start the LSP stdio server |
 
 ### search-text modes
@@ -101,6 +132,102 @@ markdown-lsp get-section ./docs overview.md quick-links --pretty
 
 ---
 
+## Graph export
+
+Export the full page link graph — nodes are pages, edges are markdown links.
+
+```bash
+# JSON (nodes + edges — machine-readable)
+markdown-lsp graph ./docs --format json --pretty
+
+# Graphviz DOT
+markdown-lsp graph ./docs --format dot > graph.dot
+
+# Mermaid flowchart (embed in markdown)
+markdown-lsp graph ./docs --format mermaid
+
+# Self-contained interactive HTML with D3 force-directed graph
+# (drag, zoom, hover highlights neighbours, click to inspect side-panel)
+markdown-lsp graph ./docs --format html --out graph.html
+```
+
+JSON output shape:
+```json
+{
+  "nodes": [{"id": "README.md", "title": "Docsbook", "charCount": 2634, "sectionsCount": 10,
+              "sections": [...], "outgoing": [...], "incoming": [...], "topSimilar": []}],
+  "edges": [{"source": "README.md", "target": "quick-start.md", "kind": "inline", "label": "Get started"}],
+  "semanticEdges": [],
+  "unresolvedCount": 3
+}
+```
+
+---
+
+## Turnkey semantic graph (v1.2)
+
+Overlay AI-powered semantic similarity edges on top of the link graph — one command, no pipeline.
+
+```bash
+# Build embeddings + interactive semantic graph in one command
+OPENROUTER_API_KEY=sk-or-... markdown-lsp graph ./docs --format html --semantic --out graph.html
+
+# With explicit thresholds
+OPENROUTER_API_KEY=sk-or-... markdown-lsp graph ./docs --format html --semantic \
+  --sim-threshold 0.75 --sim-top-k 5 --out graph.html
+```
+
+**What you get in the HTML:**
+
+- Two types of edges — **solid lines** (explicit markdown links) and **dashed amber lines** (semantic similarity)
+- **Checkboxes** in the toolbar to toggle each edge type independently
+- **Click any node** to open a side-panel showing: title, path, sections, outgoing links, incoming links, and top semantically similar pages with scores
+- Clicking a linked page in the side-panel focuses the graph on that node
+- Background click closes the panel and clears selection
+- Drag and zoom preserved from v1.1
+
+**Semantic flags:**
+
+| Flag | Default | Description |
+|---|---|---|
+| `--semantic` | off | Enable AI similarity edges |
+| `--sim-threshold` | `0.75` | Minimum cosine similarity score to draw an edge |
+| `--sim-top-k` | `5` | Max semantic neighbours per node |
+| `--model` | `openai/text-embedding-3-small` | Embedding model override |
+
+**Caching:** embeddings are cached in `.markdown-lsp-cache/embeddings/` — the second run is instant with 0 API calls.
+
+**OpenRouter model naming:** when using `OPENROUTER_API_KEY`, the model name requires the `openai/` prefix (e.g. `openai/text-embedding-3-small`). When using `AI_GATEWAY_API_KEY` (Vercel AI Gateway), use the bare name (`text-embedding-3-small`). If the model is rejected, the CLI outputs a clear hint to try the other form.
+
+---
+
+## Semantic search
+
+AI-powered semantic search using text embeddings — finds conceptually related pages even if they
+don't contain the exact query words.
+
+```bash
+# Requires OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway)
+OPENROUTER_API_KEY=sk-or-... markdown-lsp semantic-search ./docs "how to configure webhooks" --limit 5
+
+# Override embedding model
+markdown-lsp semantic-search ./docs "authentication" --model openai/text-embedding-3-small --limit 3
+```
+
+- Default embedding model: `openai/text-embedding-3-small` (via OpenRouter — model prefix required)
+- Results cached in `.markdown-lsp-cache/embeddings/` — second run is instant, no API call
+- Returns `[{ pagePath, pageTitle, score, snippet }]` sorted by cosine similarity
+
+**Environment variables:**
+
+| Variable | Purpose |
+|---|---|
+| `OPENROUTER_API_KEY` | OpenRouter API key (takes priority if set) |
+| `AI_GATEWAY_API_KEY` | Vercel AI Gateway key (fallback) |
+| `EMBEDDING_MODEL` | Override default embedding model |
+
+---
+
 ## LSP mode (editor integration)
 
 `markdown-lsp` also works as a Language Server Protocol server for editors (VS Code, Zed, Neovim, etc.).
@@ -174,6 +301,7 @@ export AI_GATEWAY_API_KEY=...   # Vercel AI Gateway
 
 - **CLI** — `node:util parseArgs`, zero extra deps, reads `.md` files into an in-memory graph
 - **Graph** — pure TypeScript, no DB needed; `buildGraph(docsRoot)` walks the directory tree
+- **Semantic graph** — in-memory cosine similarity (N×N in ~5ms); embeddings via OpenRouter or Vercel AI Gateway; disk-cached per sha256(model+text)
 - **LSP** — `vscode-languageserver/node` over stdio; requires Postgres (Drizzle ORM, `mdlsp_` prefix)
 - **AI layer** (opt-in) — pgvector cosine search on canonical-term embeddings; `text-embedding-3-small` via Vercel AI Gateway
 - **Bridge** — pure in-memory search (searchText, searchTextRanked, searchSymbols, searchPaths, listPages)
@@ -196,8 +324,10 @@ pnpm test
 - **M1 — Structural layer** ✅
 - **M2 — Semantic extract** (opt-in, code present, awaiting live AI Gateway credit)
 - **M3 — CLI-first interface** ✅ (v1.0.0)
-- M4 — User overrides for the glossary (merge / split / rename / add_synonym)
-- M5 — Docsbook integration
+- **M4 — Graph export + HTML D3 visualisation** ✅ (v1.1.0)
+- **M5 — Turnkey semantic graph (graph --semantic)** ✅ (v1.2.0)
+- M6 — User overrides for the glossary (merge / split / rename / add_synonym)
+- M7 — Docsbook integration
 
 ---
 

package/dist/ai/cache.d.ts

@@ -0,0 +1,3 @@
+export declare function cachedEmbedding(text: string, model: string): number[] | null;
+export declare function saveEmbedding(text: string, model: string, vec: number[]): void;
+//# sourceMappingURL=cache.d.ts.map

package/dist/ai/cache.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.d.ts","sourceRoot":"","sources":["../../src/ai/cache.ts"],"names":[],"mappings":"AAUA,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,IAAI,CAW5E;AAED,wBAAgB,aAAa,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,IAAI,CAK9E"}

package/dist/ai/cache.js

@@ -0,0 +1,27 @@
+import { createHash } from "node:crypto";
+import { readFileSync, writeFileSync, mkdirSync, existsSync } from "node:fs";
+import { join } from "node:path";
+const CACHE_DIR = ".markdown-lsp-cache/embeddings";
+function cacheKey(text, model) {
+    return createHash("sha256").update(model + "\x00" + text).digest("hex");
+}
+export function cachedEmbedding(text, model) {
+    const key = cacheKey(text, model);
+    const filePath = join(CACHE_DIR, `${key}.json`);
+    if (existsSync(filePath)) {
+        try {
+            return JSON.parse(readFileSync(filePath, "utf8"));
+        }
+        catch {
+            return null;
+        }
+    }
+    return null;
+}
+export function saveEmbedding(text, model, vec) {
+    mkdirSync(CACHE_DIR, { recursive: true });
+    const key = cacheKey(text, model);
+    const filePath = join(CACHE_DIR, `${key}.json`);
+    writeFileSync(filePath, JSON.stringify(vec), "utf8");
+}
+//# sourceMappingURL=cache.js.map

package/dist/ai/cache.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.js","sourceRoot":"","sources":["../../src/ai/cache.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAA;AACxC,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,SAAS,EAAE,UAAU,EAAE,MAAM,SAAS,CAAA;AAC5E,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAEhC,MAAM,SAAS,GAAG,gCAAgC,CAAA;AAElD,SAAS,QAAQ,CAAC,IAAY,EAAE,KAAa;IAC3C,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAA;AACzE,CAAC;AAED,MAAM,UAAU,eAAe,CAAC,IAAY,EAAE,KAAa;IACzD,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACjC,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,OAAO,CAAC,CAAA;IAC/C,IAAI,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;QACzB,IAAI,CAAC;YACH,OAAO,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAa,CAAA;QAC/D,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,IAAI,CAAA;QACb,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAA;AACb,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,IAAY,EAAE,KAAa,EAAE,GAAa;IACtE,SAAS,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAA;IACzC,MAAM,GAAG,GAAG,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACjC,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,OAAO,CAAC,CAAA;IAC/C,aAAa,CAAC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC,CAAA;AACtD,CAAC"}

package/dist/ai/config.d.ts

@@ -3,7 +3,13 @@ export interface AiConfig {
     embeddingModel: string;
     extractModel: string;
     hasGatewayKey: boolean;
+    hasOpenRouterKey: boolean;
 }
 export declare function getAiConfig(): AiConfig;
 export declare function assertAiEnabled(): void;
+/**
+ * Lighter check for CLI commands (semantic-search) that don't need MARKDOWN_LSP_AI_ENABLED.
+ * Only checks that at least one API key is present. Exits with a clear message on failure.
+ */
+export declare function assertApiKey(): void;
 //# sourceMappingURL=config.d.ts.map

package/dist/ai/config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,QAAQ;IACvB,OAAO,EAAE,OAAO,CAAA;IAChB,cAAc,EAAE,MAAM,CAAA;IACtB,YAAY,EAAE,MAAM,CAAA;IACpB,aAAa,EAAE,OAAO,CAAA;CACvB;AAED,wBAAgB,WAAW,IAAI,QAAQ,CAQtC;AAED,wBAAgB,eAAe,IAAI,IAAI,CAYtC"}
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,QAAQ;IACvB,OAAO,EAAE,OAAO,CAAA;IAChB,cAAc,EAAE,MAAM,CAAA;IACtB,YAAY,EAAE,MAAM,CAAA;IACpB,aAAa,EAAE,OAAO,CAAA;IACtB,gBAAgB,EAAE,OAAO,CAAA;CAC1B;AAED,wBAAgB,WAAW,IAAI,QAAQ,CAStC;AAED,wBAAgB,eAAe,IAAI,IAAI,CAYtC;AAED;;;GAGG;AACH,wBAAgB,YAAY,IAAI,IAAI,CAQnC"}

package/dist/ai/config.js

@@ -1,10 +1,12 @@
+import { DEFAULT_EMBEDDING_MODEL } from "./gateway.js";
 export function getAiConfig() {
     const enabled = process.env.MARKDOWN_LSP_AI_ENABLED === "1" || process.env.MARKDOWN_LSP_AI_ENABLED === "true";
     return {
         enabled,
-        embeddingModel: process.env.EMBEDDING_MODEL ?? "text-embedding-3-small",
+        embeddingModel: process.env.EMBEDDING_MODEL ?? DEFAULT_EMBEDDING_MODEL,
         extractModel: process.env.EXTRACT_MODEL ?? "gpt-4o-mini",
         hasGatewayKey: Boolean(process.env.AI_GATEWAY_API_KEY),
+        hasOpenRouterKey: Boolean(process.env.OPENROUTER_API_KEY),
     };
 }
 export function assertAiEnabled() {
@@ -12,8 +14,19 @@ export function assertAiEnabled() {
     if (!cfg.enabled) {
         throw new Error("AI features are disabled. Set MARKDOWN_LSP_AI_ENABLED=1 to enable embeddings + semantic extraction.");
     }
-    if (!cfg.hasGatewayKey) {
-        throw new Error("AI features require AI_GATEWAY_API_KEY (Vercel AI Gateway) — or fork the gateway module to use a different provider.");
+    if (!cfg.hasGatewayKey && !cfg.hasOpenRouterKey) {
+        throw new Error("AI features require OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway).");
+    }
+}
+/**
+ * Lighter check for CLI commands (semantic-search) that don't need MARKDOWN_LSP_AI_ENABLED.
+ * Only checks that at least one API key is present. Exits with a clear message on failure.
+ */
+export function assertApiKey() {
+    const cfg = getAiConfig();
+    if (!cfg.hasGatewayKey && !cfg.hasOpenRouterKey) {
+        process.stderr.write("Error: set OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway) to use semantic-search.\n");
+        process.exit(1);
     }
 }
 //# sourceMappingURL=config.js.map

package/dist/ai/config.js.map

@@ -1 +1 @@
-{"version":3,"file":"config.js","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAOA,MAAM,UAAU,WAAW;IACzB,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,MAAM,CAAA;IAC7G,OAAO;QACL,OAAO;QACP,cAAc,EAAE,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,wBAAwB;QACvE,YAAY,EAAE,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa;QACxD,aAAa,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;KACvD,CAAA;AACH,CAAC;AAED,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC;QACjB,MAAM,IAAI,KAAK,CACb,qGAAqG,CACtG,CAAA;IACH,CAAC;IACD,IAAI,CAAC,GAAG,CAAC,aAAa,EAAE,CAAC;QACvB,MAAM,IAAI,KAAK,CACb,sHAAsH,CACvH,CAAA;IACH,CAAC;AACH,CAAC"}
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../../src/ai/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,uBAAuB,EAAE,MAAM,cAAc,CAAA;AAUtD,MAAM,UAAU,WAAW;IACzB,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,MAAM,CAAA;IAC7G,OAAO;QACL,OAAO;QACP,cAAc,EAAE,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,uBAAuB;QACtE,YAAY,EAAE,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa;QACxD,aAAa,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;QACtD,gBAAgB,EAAE,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAC;KAC1D,CAAA;AACH,CAAC;AAED,MAAM,UAAU,eAAe;IAC7B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC;QACjB,MAAM,IAAI,KAAK,CACb,qGAAqG,CACtG,CAAA;IACH,CAAC;IACD,IAAI,CAAC,GAAG,CAAC,aAAa,IAAI,CAAC,GAAG,CAAC,gBAAgB,EAAE,CAAC;QAChD,MAAM,IAAI,KAAK,CACb,gGAAgG,CACjG,CAAA;IACH,CAAC;AACH,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,YAAY;IAC1B,MAAM,GAAG,GAAG,WAAW,EAAE,CAAA;IACzB,IAAI,CAAC,GAAG,CAAC,aAAa,IAAI,CAAC,GAAG,CAAC,gBAAgB,EAAE,CAAC;QAChD,OAAO,CAAC,MAAM,CAAC,KAAK,CAClB,gHAAgH,CACjH,CAAA;QACD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;IACjB,CAAC;AACH,CAAC"}

package/dist/ai/embeddings.d.ts

@@ -2,6 +2,13 @@ export interface EmbeddingResult {
     vectors: number[][];
     tokensUsed: number;
 }
-export declare function embedTexts(texts: string[]): Promise<EmbeddingResult>;
-export declare function embedOne(text: string): Promise<number[]>;
+/**
+ * Embed multiple texts with disk-cache support.
+ * Checks cache per-text before hitting the API; saves results after.
+ * @param texts - texts to embed
+ * @param modelOverride - override the embedding model (default: EMBEDDING_MODEL env or openai/text-embedding-3-small)
+ * @param useCache - whether to use disk cache (default: true)
+ */
+export declare function embedTexts(texts: string[], modelOverride?: string, useCache?: boolean): Promise<EmbeddingResult>;
+export declare function embedOne(text: string, modelOverride?: string): Promise<number[]>;
 //# sourceMappingURL=embeddings.d.ts.map

package/dist/ai/embeddings.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"embeddings.d.ts","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAOA,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,EAAE,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;CACnB;AAED,wBAAsB,UAAU,CAAC,KAAK,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,eAAe,CAAC,CAiB1E;AAED,wBAAsB,QAAQ,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAG9D"}
+{"version":3,"file":"embeddings.d.ts","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAQA,MAAM,WAAW,eAAe;IAC9B,OAAO,EAAE,MAAM,EAAE,EAAE,CAAA;IACnB,UAAU,EAAE,MAAM,CAAA;CACnB;AAED;;;;;;GAMG;AACH,wBAAsB,UAAU,CAC9B,KAAK,EAAE,MAAM,EAAE,EACf,aAAa,CAAC,EAAE,MAAM,EACtB,QAAQ,UAAO,GACd,OAAO,CAAC,eAAe,CAAC,CAgD1B;AAED,wBAAsB,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,aAAa,CAAC,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAGtF"}

package/dist/ai/embeddings.js

@@ -1,27 +1,65 @@
 import { embedMany } from "ai";
 import { getGateway, EMBEDDING_MODEL } from "./gateway.js";
 import { assertAiEnabled } from "./config.js";
+import { cachedEmbedding, saveEmbedding } from "./cache.js";
 const BATCH_SIZE = 96;
 const MAX_INPUT_CHARS = 6000;
-export async function embedTexts(texts) {
+/**
+ * Embed multiple texts with disk-cache support.
+ * Checks cache per-text before hitting the API; saves results after.
+ * @param texts - texts to embed
+ * @param modelOverride - override the embedding model (default: EMBEDDING_MODEL env or openai/text-embedding-3-small)
+ * @param useCache - whether to use disk cache (default: true)
+ */
+export async function embedTexts(texts, modelOverride, useCache = true) {
     if (texts.length === 0)
         return { vectors: [], tokensUsed: 0 };
     assertAiEnabled();
     const gw = getGateway();
-    const model = gw.embedding(EMBEDDING_MODEL);
-    const out = [];
+    const modelName = modelOverride ?? EMBEDDING_MODEL;
+    const model = gw.embedding(modelName);
+    const out = new Array(texts.length).fill(null);
     let tokensUsed = 0;
-    for (let i = 0; i < texts.length; i += BATCH_SIZE) {
-        const batch = texts.slice(i, i + BATCH_SIZE).map((t) => t.slice(0, MAX_INPUT_CHARS));
-        const res = await embedMany({ model, values: batch });
-        for (const v of res.embeddings)
-            out.push(v);
+    // Check cache first
+    const uncachedIndices = [];
+    const uncachedTexts = [];
+    if (useCache) {
+        for (let i = 0; i < texts.length; i++) {
+            const cached = cachedEmbedding(texts[i], modelName);
+            if (cached !== null) {
+                out[i] = cached;
+            }
+            else {
+                uncachedIndices.push(i);
+                uncachedTexts.push(texts[i]);
+            }
+        }
+    }
+    else {
+        for (let i = 0; i < texts.length; i++) {
+            uncachedIndices.push(i);
+            uncachedTexts.push(texts[i]);
+        }
+    }
+    // Embed uncached texts in batches
+    for (let b = 0; b < uncachedTexts.length; b += BATCH_SIZE) {
+        const batchTexts = uncachedTexts.slice(b, b + BATCH_SIZE).map((t) => t.slice(0, MAX_INPUT_CHARS));
+        const batchIndices = uncachedIndices.slice(b, b + BATCH_SIZE);
+        const res = await embedMany({ model, values: batchTexts });
+        for (let j = 0; j < res.embeddings.length; j++) {
+            const vec = res.embeddings[j];
+            const originalIndex = batchIndices[j];
+            out[originalIndex] = vec;
+            if (useCache) {
+                saveEmbedding(uncachedTexts[b + j], modelName, vec);
+            }
+        }
         tokensUsed += res.usage?.tokens ?? 0;
     }
     return { vectors: out, tokensUsed };
 }
-export async function embedOne(text) {
-    const { vectors } = await embedTexts([text]);
+export async function embedOne(text, modelOverride) {
+    const { vectors } = await embedTexts([text], modelOverride);
     return vectors[0];
 }
 //# sourceMappingURL=embeddings.js.map

package/dist/ai/embeddings.js.map

@@ -1 +1 @@
-{"version":3,"file":"embeddings.js","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,IAAI,CAAA;AAC9B,OAAO,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAA;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AAE7C,MAAM,UAAU,GAAG,EAAE,CAAA;AACrB,MAAM,eAAe,GAAG,IAAI,CAAA;AAO5B,MAAM,CAAC,KAAK,UAAU,UAAU,CAAC,KAAe;IAC9C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAA;IAC7D,eAAe,EAAE,CAAA;IACjB,MAAM,EAAE,GAAG,UAAU,EAAE,CAAA;IACvB,MAAM,KAAK,GAAG,EAAE,CAAC,SAAS,CAAC,eAAe,CAAC,CAAA;IAE3C,MAAM,GAAG,GAAe,EAAE,CAAA;IAC1B,IAAI,UAAU,GAAG,CAAC,CAAA;IAElB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,UAAU,EAAE,CAAC;QAClD,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,eAAe,CAAC,CAAC,CAAA;QACpF,MAAM,GAAG,GAAG,MAAM,SAAS,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,CAAA;QACrD,KAAK,MAAM,CAAC,IAAI,GAAG,CAAC,UAAU;YAAE,GAAG,CAAC,IAAI,CAAC,CAAa,CAAC,CAAA;QACvD,UAAU,IAAI,GAAG,CAAC,KAAK,EAAE,MAAM,IAAI,CAAC,CAAA;IACtC,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,GAAG,EAAE,UAAU,EAAE,CAAA;AACrC,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,IAAY;IACzC,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,UAAU,CAAC,CAAC,IAAI,CAAC,CAAC,CAAA;IAC5C,OAAO,OAAO,CAAC,CAAC,CAAE,CAAA;AACpB,CAAC"}
+{"version":3,"file":"embeddings.js","sourceRoot":"","sources":["../../src/ai/embeddings.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,MAAM,IAAI,CAAA;AAC9B,OAAO,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAA;AAC1D,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAA;AAC7C,OAAO,EAAE,eAAe,EAAE,aAAa,EAAE,MAAM,YAAY,CAAA;AAE3D,MAAM,UAAU,GAAG,EAAE,CAAA;AACrB,MAAM,eAAe,GAAG,IAAI,CAAA;AAO5B;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,UAAU,CAC9B,KAAe,EACf,aAAsB,EACtB,QAAQ,GAAG,IAAI;IAEf,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,OAAO,EAAE,EAAE,EAAE,UAAU,EAAE,CAAC,EAAE,CAAA;IAC7D,eAAe,EAAE,CAAA;IACjB,MAAM,EAAE,GAAG,UAAU,EAAE,CAAA;IACvB,MAAM,SAAS,GAAG,aAAa,IAAI,eAAe,CAAA;IAClD,MAAM,KAAK,GAAG,EAAE,CAAC,SAAS,CAAC,SAAS,CAAC,CAAA;IAErC,MAAM,GAAG,GAAwB,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAA;IACnE,IAAI,UAAU,GAAG,CAAC,CAAA;IAElB,oBAAoB;IACpB,MAAM,eAAe,GAAa,EAAE,CAAA;IACpC,MAAM,aAAa,GAAa,EAAE,CAAA;IAElC,IAAI,QAAQ,EAAE,CAAC;QACb,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,MAAM,MAAM,GAAG,eAAe,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,SAAS,CAAC,CAAA;YACpD,IAAI,MAAM,KAAK,IAAI,EAAE,CAAC;gBACpB,GAAG,CAAC,CAAC,CAAC,GAAG,MAAM,CAAA;YACjB,CAAC;iBAAM,CAAC;gBACN,eAAe,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;gBACvB,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAE,CAAC,CAAA;YAC/B,CAAC;QACH,CAAC;IACH,CAAC;SAAM,CAAC;QACN,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,eAAe,CAAC,IAAI,CAAC,CAAC,CAAC,CAAA;YACvB,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAE,CAAC,CAAA;QAC/B,CAAC;IACH,CAAC;IAED,kCAAkC;IAClC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,aAAa,CAAC,MAAM,EAAE,CAAC,IAAI,UAAU,EAAE,CAAC;QAC1D,MAAM,UAAU,GAAG,aAAa,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,eAAe,CAAC,CAAC,CAAA;QACjG,MAAM,YAAY,GAAG,eAAe,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,UAAU,CAAC,CAAA;QAC7D,MAAM,GAAG,GAAG,MAAM,SAAS,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,UAAU,EAAE,CAAC,CAAA;QAC1D,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,GAAG,CAAC,UAAU,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YAC/C,MAAM,GAAG,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAa,CAAA;YACzC,MAAM,aAAa,GAAG,YAAY,CAAC,CAAC,CAAE,CAAA;YACtC,GAAG,CAAC,aAAa,CAAC,GAAG,GAAG,CAAA;YACxB,IAAI,QAAQ,EAAE,CAAC;gBACb,aAAa,CAAC,aAAa,CAAC,CAAC,GAAG,CAAC,CAAE,EAAE,SAAS,EAAE,GAAG,CAAC,CAAA;YACtD,CAAC;QACH,CAAC;QACD,UAAU,IAAI,GAAG,CAAC,KAAK,EAAE,MAAM,IAAI,CAAC,CAAA;IACtC,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,GAAiB,EAAE,UAAU,EAAE,CAAA;AACnD,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,IAAY,EAAE,aAAsB;IACjE,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,UAAU,CAAC,CAAC,IAAI,CAAC,EAAE,aAAa,CAAC,CAAA;IAC3D,OAAO,OAAO,CAAC,CAAC,CAAE,CAAA;AACpB,CAAC"}

package/dist/ai/gateway.d.ts

@@ -1,5 +1,8 @@
+export declare const DEFAULT_EMBEDDING_MODEL = "openai/text-embedding-3-small";
 export declare const EMBEDDING_MODEL: string;
 export declare const EMBEDDING_DIM = 1536;
 export declare const EXTRACT_MODEL: string;
 export declare function getGateway(): import("@ai-sdk/openai").OpenAIProvider;
+/** Reset the cached gateway (useful for tests or when env changes mid-process) */
+export declare function resetGateway(): void;
 //# sourceMappingURL=gateway.d.ts.map

package/dist/ai/gateway.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"gateway.d.ts","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAEA,eAAO,MAAM,eAAe,QAA0D,CAAA;AACtF,eAAO,MAAM,aAAa,OAAO,CAAA;AACjC,eAAO,MAAM,aAAa,QAA6C,CAAA;AAIvE,wBAAgB,UAAU,4CASzB"}
+{"version":3,"file":"gateway.d.ts","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAEA,eAAO,MAAM,uBAAuB,kCAAkC,CAAA;AACtE,eAAO,MAAM,eAAe,QAAyD,CAAA;AACrF,eAAO,MAAM,aAAa,OAAO,CAAA;AACjC,eAAO,MAAM,aAAa,QAA6C,CAAA;AAIvE,wBAAgB,UAAU,4CAoBzB;AAED,kFAAkF;AAClF,wBAAgB,YAAY,IAAI,IAAI,CAEnC"}

package/dist/ai/gateway.js

@@ -1,18 +1,27 @@
 import { createOpenAI } from "@ai-sdk/openai";
-export const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL ?? "text-embedding-3-small";
+export const DEFAULT_EMBEDDING_MODEL = "openai/text-embedding-3-small";
+export const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL ?? DEFAULT_EMBEDDING_MODEL;
 export const EMBEDDING_DIM = 1536;
 export const EXTRACT_MODEL = process.env.EXTRACT_MODEL ?? "gpt-4o-mini";
 let _gw = null;
 export function getGateway() {
     if (_gw)
         return _gw;
-    const apiKey = process.env.AI_GATEWAY_API_KEY;
-    if (!apiKey)
-        throw new Error("AI_GATEWAY_API_KEY is not set");
-    _gw = createOpenAI({
-        apiKey,
-        baseURL: process.env.AI_GATEWAY_BASE_URL ?? "https://ai-gateway.vercel.sh/v1",
-    });
+    // OpenRouter takes priority if OPENROUTER_API_KEY is set
+    const openrouterKey = process.env.OPENROUTER_API_KEY;
+    const gatewayKey = process.env.AI_GATEWAY_API_KEY;
+    const apiKey = openrouterKey ?? gatewayKey;
+    if (!apiKey) {
+        throw new Error("Set OPENROUTER_API_KEY (OpenRouter) or AI_GATEWAY_API_KEY (Vercel AI Gateway) to use AI features.");
+    }
+    const baseURL = openrouterKey
+        ? (process.env.AI_GATEWAY_BASE_URL ?? "https://openrouter.ai/api/v1")
+        : (process.env.AI_GATEWAY_BASE_URL ?? "https://ai-gateway.vercel.sh/v1");
+    _gw = createOpenAI({ apiKey, baseURL });
     return _gw;
 }
+/** Reset the cached gateway (useful for tests or when env changes mid-process) */
+export function resetGateway() {
+    _gw = null;
+}
 //# sourceMappingURL=gateway.js.map

package/dist/ai/gateway.js.map

@@ -1 +1 @@
-{"version":3,"file":"gateway.js","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAA;AAE7C,MAAM,CAAC,MAAM,eAAe,GAAG,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,wBAAwB,CAAA;AACtF,MAAM,CAAC,MAAM,aAAa,GAAG,IAAI,CAAA;AACjC,MAAM,CAAC,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa,CAAA;AAEvE,IAAI,GAAG,GAA2C,IAAI,CAAA;AAEtD,MAAM,UAAU,UAAU;IACxB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAA;IACnB,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IAC7C,IAAI,CAAC,MAAM;QAAE,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAA;IAC7D,GAAG,GAAG,YAAY,CAAC;QACjB,MAAM;QACN,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,iCAAiC;KAC9E,CAAC,CAAA;IACF,OAAO,GAAG,CAAA;AACZ,CAAC"}
+{"version":3,"file":"gateway.js","sourceRoot":"","sources":["../../src/ai/gateway.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAA;AAE7C,MAAM,CAAC,MAAM,uBAAuB,GAAG,+BAA+B,CAAA;AACtE,MAAM,CAAC,MAAM,eAAe,GAAG,OAAO,CAAC,GAAG,CAAC,eAAe,IAAI,uBAAuB,CAAA;AACrF,MAAM,CAAC,MAAM,aAAa,GAAG,IAAI,CAAA;AACjC,MAAM,CAAC,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,aAAa,IAAI,aAAa,CAAA;AAEvE,IAAI,GAAG,GAA2C,IAAI,CAAA;AAEtD,MAAM,UAAU,UAAU;IACxB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAA;IAEnB,yDAAyD;IACzD,MAAM,aAAa,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IACpD,MAAM,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,kBAAkB,CAAA;IACjD,MAAM,MAAM,GAAG,aAAa,IAAI,UAAU,CAAA;IAE1C,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,mGAAmG,CACpG,CAAA;IACH,CAAC;IAED,MAAM,OAAO,GAAG,aAAa;QAC3B,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,8BAA8B,CAAC;QACrE,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,iCAAiC,CAAC,CAAA;IAE1E,GAAG,GAAG,YAAY,CAAC,EAAE,MAAM,EAAE,OAAO,EAAE,CAAC,CAAA;IACvC,OAAO,GAAG,CAAA;AACZ,CAAC;AAED,kFAAkF;AAClF,MAAM,UAAU,YAAY;IAC1B,GAAG,GAAG,IAAI,CAAA;AACZ,CAAC"}