@virstack/doc-ingest 1.0.0

package/README.md

@@ -0,0 +1,203 @@
+# 🚀 Virstack Doc Ingest
+
+**Virstack Doc Ingest** is a high-performance, parallelized document ingestion and vectorization pipeline designed for scalable Retrieval-Augmented Generation (RAG) applications.
+
+Powered by **LangGraph** for resilient orchestration, **OpenRouter / Gemini** for advanced vision/text extraction, and natively supporting **Upstash Vector** (with easily injectable custom adapters), this library acts as a universal bridge between your raw documents and your AI applications.
+
+---
+
+## ✨ Key Features
+
+- **Universal Multi-Format Support:** Natively processes PDF, DOCX, XLSX, PPTX, CSV, TXT, HTML, and EPUB files.
+- **Dual-Tier Parallelism:** Concurrently processes multiple files while simultaneously splitting and routing large PDFs into parallel Vision-API execution nodes.
+- **Smart Type Routing:** Automatically identifies MIME types and dynamically routes files to the most optimal, parser-specific extraction graph.
+- **Provider Agnostic Architecture:** Built entirely on Dependency Injection. Easily swap out LLMs, Embeddings, and Vector Databases (Pinecone, Qdrant, etc.) to fit your specific stack.
+- **Gorgeous TUI (Text User Interface):** Features a beautiful, interactive command-line interface with interactive menus and live, non-tearing spinners.
+
+---
+
+## 🛠️ System Prerequisites
+
+**IMPORTANT:** For parsing complex Office documents (e.g., `.docx`, `.pptx`, `.xlsx`, `.epub`), the pipeline relies on **LibreOffice** for high-fidelity conversion.
+
+If you are only parsing PDFs, TXT, or CSV files, LibreOffice is **not** required.
+
+### Installing LibreOffice:
+
+- **macOS:** `brew install --cask libreoffice`
+- **Ubuntu/Debian:** `sudo apt-get install libreoffice`
+- **Windows:** Download the installer from [libreoffice.org](https://www.libreoffice.org/)
+
+---
+
+## 📦 Installation
+
+You can install Virstack Doc Ingest globally to use as a standalone CLI tool, or locally to utilize its powerful API in your custom Node.js applications.
+
+### Global Install (CLI Usage)
+
+```bash
+npm install -g virstack-doc-ingest
+```
+
+### Local Install (Library Usage)
+
+```bash
+npm install virstack-doc-ingest
+```
+
+---
+
+## 💻 Usage Mode 1: Interactive CLI
+
+The CLI offers a completely interactive, wizard-based experience.
+
+### 1. Environment Configuration
+
+Create a `.env` file in the directory where you plan to run the command:
+
+```env
+OPENROUTER_API_KEY=sk-or-v1-...
+UPSTASH_VECTOR_URL=https://...
+UPSTASH_VECTOR_TOKEN=...
+LLM_MODEL=google/gemini-2.0-flash-001
+EMBEDDING_MODEL=text-embedding-3-large
+MAX_CONCURRENT_FILES=3
+MAX_CONCURRENT_API_CALLS=15
+```
+
+### 2. Running the Tool
+
+To launch the interactive wizard (which allows you to select files, folders, or paste raw text):
+
+```bash
+virstack-doc-ingest
+```
+
+To bypass the wizard and directly ingest a specific file or directory:
+
+```bash
+# Process a single contract
+virstack-doc-ingest ./documents/contract.pdf
+
+# Process all documents in a directory
+virstack-doc-ingest ./company-knowledge-base/
+
+# Run with verbose, node-level diagnostics
+virstack-doc-ingest ./documents/ --verbose
+```
+
+### Example Output
+
+```text
+ __     __  _                _                    _        ____                     ___                                _
+ \ \   / / (_)  _ __   ___  | |_    __ _    ___  | | __   |  _ \    ___     ___    |_ _|  _ __     __ _    ___   ___  | |_
+  \ \ / /  | | | '__| / __| | __|  / _` |  / __| | |/ /   | | | |  / _ \   / __|    | |  | '_ \   / _` |  / _ \ / __| | __|
+   \ V /   | | | |    \__ \ | |_  | (_| | | (__  |   <    | |_| | | (_) | | (__     | |  | | | | | (_| | |  __/ \__ \ | |_
+    \_/    |_| |_|    |___/  \__|  \__,_|  \___| |_|\_\   |____/   \___/   \___|   |___| |_| |_|  \__, |  \___| |___/  \__|
+                                                                                                  |___/
+┌   Welcome to Virstack Doc Ingest
+│
+◇  What file or directory would you like to process?
+│  ./docs
+│
+◇  Found 2 file(s). Ready to process?
+│  Yes, start ingestion
+│
+◇  ✔ Processing complete in 41.8s!
+│
+◇  Final Results: 2 succeeded, 0 failed
+│
+│    ✔ PRES1 CIS 6006-Updated Assessment.p │   28 chunks │   28 vectors │ 41.7s
+│
+│    ✔ VAI-020-021-Webhook-Implementation. │   12 chunks │   12 vectors │ 27.9s
+│
+└   Pipeline Finished Successfully!
+```
+
+---
+
+## 🛠️ Usage Mode 2: Node.js Library (100% Provider Agnostic)
+
+Virstack Doc Ingest is designed to be fully embedded into your own SaaS backends or ETL pipelines. It is rigidly decoupled from concrete implementations.
+
+### Default Built-In Adapters
+
+The package exports fully functional adapters for typical stacks:
+
+- `OpenRouterLlmAdapter`
+- `OpenRouterEmbeddingAdapter`
+- `UpstashAdapter`
+
+### Custom Adapter Example (Pinecone & Local LLM)
+
+Here is how you inject your own custom logic into the LangGraph pipeline:
+
+```typescript
+import {
+  initializeConfig,
+  batchGraph,
+  type VectorStoreAdapter,
+  type LlmAdapter,
+  type EmbeddingAdapter,
+  OpenRouterEmbeddingAdapter,
+} from "virstack-doc-ingest";
+import { Pinecone } from "@pinecone-database/pinecone";
+
+// 1. Define your own Vector Store connection
+class CustomPineconeAdapter implements VectorStoreAdapter {
+  async upsert(records: any[]) {
+    /* ... */
+  }
+}
+
+// 2. Define a custom Local AI processor (e.g. Ollama)
+class LocalLLMAdapter implements LlmAdapter {
+  async extractText(image: Buffer, mime: string) {
+    return "extracted text";
+  }
+}
+
+// 3. Mount the adapters to the global configuration
+initializeConfig({
+  llm: new LocalLLMAdapter(),
+  embedder: new OpenRouterEmbeddingAdapter(
+    process.env.OPENROUTER_API_KEY!,
+    "text-embedding-3-large",
+  ),
+  vectorStore: new CustomPineconeAdapter(),
+  maxConcurrentFiles: 5,
+});
+
+// 4. Invoke the ingestion orchestrator
+async function processData() {
+  const files = ["./uploads/report_2024.pdf", "./uploads/financials.xlsx"];
+
+  console.log("Orchestrating batch ingestion...");
+  const result = await batchGraph.invoke({ files });
+  console.log("Success! Extracted documents:", result.results.length);
+}
+
+processData();
+```
+
+---
+
+## ⚙️ Configuration Reference
+
+When invoking `initializeConfig(options)`, the `VirstackDocIngestConfig` interface accepts the following properties:
+
+| Property              | Type                 | Default      | Description                                                                       |
+| :-------------------- | :------------------- | :----------- | :-------------------------------------------------------------------------------- |
+| `llm`                 | `LlmAdapter`         | **Required** | Provider for extracting text (especially from PDF images via Vision APIs).        |
+| `embedder`            | `EmbeddingAdapter`   | **Required** | Provider for transforming text chunks into vector arrays.                         |
+| `vectorStore`         | `VectorStoreAdapter` | **Required** | Provider targeting your target vector database for final persistence.             |
+| `openRouterApiKey`    | `string`             | `undefined`  | Required if utilizing the built-in OpenRouter adapters.                           |
+| `maxConcurrentFiles`  | `number`             | `3`          | Maximum files mapped into the parallel processing queue simultaneously.           |
+| `maxConcurrentApi`    | `number`             | `15`         | Global connection limit to prevent 429 Rate Limit errors across all active nodes. |
+| `maxTokens`           | `number`             | `16384`      | Maximum allowable context window for the Vision LLM extraction.                   |
+| `embeddingDimensions` | `number`             | `1536`       | Target dimensions for the output vectors.                                         |
+| `chunkSize`           | `number`             | `1000`       | Target character length for Markdown recursive section chunking.                  |
+| `chunkOverlap`        | `number`             | `100`        | Overlapping character padding between contiguous chunk segments.                  |
+| `pdfPagesPerChunk`    | `number`             | `10`         | Number of PDF pages grouped together before a parallel Vision evaluation.         |
+| `systemPrompt`        | `string`             | _(default)_  | Injection of custom instructions overriding the default extraction constraints.   |

package/dist/adapters/aiAdapters.d.ts

@@ -0,0 +1,25 @@
+export interface LlmInput {
+    systemPrompt: string;
+    userText: string;
+    base64PdfChunk?: string;
+}
+export interface LlmAdapter {
+    generateMarkdown(input: LlmInput): Promise<string>;
+}
+export interface EmbeddingAdapter {
+    embed(chunks: string[]): Promise<number[][]>;
+}
+export declare class OpenRouterLlmAdapter implements LlmAdapter {
+    private client;
+    private model;
+    constructor(apiKey: string, model: string);
+    generateMarkdown(input: LlmInput): Promise<string>;
+}
+export declare class OpenRouterEmbeddingAdapter implements EmbeddingAdapter {
+    private client;
+    private model;
+    private dimensions;
+    constructor(apiKey: string, model: string, dimensions?: number);
+    embed(chunks: string[]): Promise<number[][]>;
+}
+//# sourceMappingURL=aiAdapters.d.ts.map

package/dist/adapters/aiAdapters.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"aiAdapters.d.ts","sourceRoot":"","sources":["../../src/adapters/aiAdapters.ts"],"names":[],"mappings":"AAIA,MAAM,WAAW,QAAQ;IACvB,YAAY,EAAE,MAAM,CAAC;IACrB,QAAQ,EAAE,MAAM,CAAC;IACjB,cAAc,CAAC,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,UAAU;IACzB,gBAAgB,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;CACpD;AAED,MAAM,WAAW,gBAAgB;IAC/B,KAAK,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;CAC9C;AAID,qBAAa,oBAAqB,YAAW,UAAU;IACrD,OAAO,CAAC,MAAM,CAAa;IAC3B,OAAO,CAAC,KAAK,CAAS;gBAEV,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM;IAKnC,gBAAgB,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,MAAM,CAAC;CAgCzD;AAED,qBAAa,0BAA2B,YAAW,gBAAgB;IACjE,OAAO,CAAC,MAAM,CAAa;IAC3B,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,UAAU,CAAS;gBAEf,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,UAAU,GAAE,MAAa;IAM9D,KAAK,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;CA4BnD"}

package/dist/adapters/aiAdapters.js

@@ -0,0 +1,73 @@
+import { OpenRouter } from "@openrouter/sdk";
+// --- BUILT-IN ADAPTERS (For CLI to use by default) ---
+export class OpenRouterLlmAdapter {
+    client;
+    model;
+    constructor(apiKey, model) {
+        this.client = new OpenRouter({ apiKey });
+        this.model = model;
+    }
+    async generateMarkdown(input) {
+        const userContent = [];
+        if (input.base64PdfChunk) {
+            userContent.push({
+                type: "image_url",
+                imageUrl: { url: `data:application/pdf;base64,${input.base64PdfChunk}` },
+            });
+        }
+        userContent.push({ type: "text", text: input.userText });
+        const response = await this.client.chat.send({
+            chatGenerationParams: {
+                model: this.model,
+                messages: [
+                    { role: "system", content: input.systemPrompt },
+                    { role: "user", content: userContent },
+                ],
+                temperature: 0,
+            }
+        });
+        // The SDK returns ChatResponse when not streaming
+        const chatResponse = response;
+        const content = chatResponse.choices?.[0]?.message?.content;
+        if (Array.isArray(content)) {
+            return content.map(item => (item.type === 'text' ? item.text : '')).join('').trim();
+        }
+        return (typeof content === "string" ? content.trim() : "");
+    }
+}
+export class OpenRouterEmbeddingAdapter {
+    client;
+    model;
+    dimensions;
+    constructor(apiKey, model, dimensions = 1536) {
+        this.client = new OpenRouter({ apiKey });
+        this.model = model;
+        this.dimensions = dimensions;
+    }
+    async embed(chunks) {
+        const response = await this.client.embeddings.generate({
+            requestBody: {
+                model: this.model,
+                input: chunks,
+                dimensions: this.dimensions,
+            }
+        });
+        if (typeof response === "string") {
+            throw new Error(`OpenRouter Embeddings API returned unexpected string response: ${response}`);
+        }
+        // Maintain chunk order based on OpenRouter response structure
+        let embeddingsList = response.data;
+        if (embeddingsList.length > 0 && typeof embeddingsList[0].index === "number") {
+            embeddingsList = embeddingsList.sort((a, b) => a.index - b.index);
+        }
+        return embeddingsList.map((item) => {
+            const emb = item.embedding;
+            if (typeof emb === "string") {
+                // Some models might return base64 if requested, but we expect float arrays
+                throw new Error("Received unexpected string embedding from OpenRouter");
+            }
+            return emb;
+        });
+    }
+}
+//# sourceMappingURL=aiAdapters.js.map

package/dist/adapters/aiAdapters.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"aiAdapters.js","sourceRoot":"","sources":["../../src/adapters/aiAdapters.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,iBAAiB,CAAC;AAkB7C,wDAAwD;AAExD,MAAM,OAAO,oBAAoB;IACvB,MAAM,CAAa;IACnB,KAAK,CAAS;IAEtB,YAAY,MAAc,EAAE,KAAa;QACvC,IAAI,CAAC,MAAM,GAAG,IAAI,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC;QACzC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;IAED,KAAK,CAAC,gBAAgB,CAAC,KAAe;QACpC,MAAM,WAAW,GAAU,EAAE,CAAC;QAE9B,IAAI,KAAK,CAAC,cAAc,EAAE,CAAC;YACzB,WAAW,CAAC,IAAI,CAAC;gBACf,IAAI,EAAE,WAAW;gBACjB,QAAQ,EAAE,EAAE,GAAG,EAAE,+BAA+B,KAAK,CAAC,cAAc,EAAE,EAAE;aACzE,CAAC,CAAC;QACL,CAAC;QACD,WAAW,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC;QAEzD,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC;YAC3C,oBAAoB,EAAE;gBACpB,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,QAAQ,EAAE;oBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,KAAK,CAAC,YAAY,EAAE;oBAC/C,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,WAAkB,EAAE;iBAC9C;gBACD,WAAW,EAAE,CAAC;aACf;SACF,CAAC,CAAC;QAEH,kDAAkD;QAClD,MAAM,YAAY,GAAG,QAAe,CAAC;QACrC,MAAM,OAAO,GAAG,YAAY,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,CAAC;QAE5D,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YAC3B,OAAO,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,IAAI,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;QACtF,CAAC;QAED,OAAO,CAAC,OAAO,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IAC7D,CAAC;CACF;AAED,MAAM,OAAO,0BAA0B;IAC7B,MAAM,CAAa;IACnB,KAAK,CAAS;IACd,UAAU,CAAS;IAE3B,YAAY,MAAc,EAAE,KAAa,EAAE,aAAqB,IAAI;QAClE,IAAI,CAAC,MAAM,GAAG,IAAI,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC,CAAC;QACzC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;QACnB,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;IAC/B,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,MAAgB;QAC1B,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,UAAU,CAAC,QAAQ,CAAC;YACrD,WAAW,EAAE;gBACX,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,KAAK,EAAE,MAAM;gBACb,UAAU,EAAE,IAAI,CAAC,UAAU;aAC5B;SACF,CAAC,CAAC;QAEH,IAAI,OAAO,QAAQ,KAAK,QAAQ,EAAE,CAAC;YACjC,MAAM,IAAI,KAAK,CAAC,kEAAkE,QAAQ,EAAE,CAAC,CAAC;QAChG,CAAC;QAED,8DAA8D;QAC9D,IAAI,cAAc,GAAG,QAAQ,CAAC,IAAI,CAAC;QACnC,IAAI,cAAc,CAAC,MAAM,GAAG,CAAC,IAAI,OAAO,cAAc,CAAC,CAAC,CAAC,CAAC,KAAK,KAAK,QAAQ,EAAE,CAAC;YAC7E,cAAc,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC,CAAM,EAAE,CAAM,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QAC9E,CAAC;QAED,OAAO,cAAc,CAAC,GAAG,CAAC,CAAC,IAAS,EAAE,EAAE;YACtC,MAAM,GAAG,GAAG,IAAI,CAAC,SAAS,CAAC;YAC3B,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;gBAC3B,2EAA2E;gBAC3E,MAAM,IAAI,KAAK,CAAC,sDAAsD,CAAC,CAAC;YAC3E,CAAC;YACD,OAAO,GAAG,CAAC;QACb,CAAC,CAAC,CAAC;IACL,CAAC;CACF"}

package/dist/adapters/vectorStore.d.ts

@@ -0,0 +1,24 @@
+/**
+ * The standard shape of a record that the pipeline will produce.
+ */
+export interface VectorRecord {
+    id: string;
+    vector: number[];
+    metadata: Record<string, any>;
+}
+/**
+ * The contract that any vector database adapter must follow.
+ */
+export interface VectorStoreAdapter {
+    upsert(records: VectorRecord[]): Promise<void>;
+}
+/**
+ * Built-in adapter for Upstash Vector.
+ * Used by default when running via the CLI.
+ */
+export declare class UpstashAdapter implements VectorStoreAdapter {
+    private index;
+    constructor(url: string, token: string);
+    upsert(records: VectorRecord[]): Promise<void>;
+}
+//# sourceMappingURL=vectorStore.d.ts.map

package/dist/adapters/vectorStore.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"vectorStore.d.ts","sourceRoot":"","sources":["../../src/adapters/vectorStore.ts"],"names":[],"mappings":"AAEA;;GAEG;AACH,MAAM,WAAW,YAAY;IAC3B,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;CAC/B;AAED;;GAEG;AACH,MAAM,WAAW,kBAAkB;IACjC,MAAM,CAAC,OAAO,EAAE,YAAY,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;CAChD;AAED;;;GAGG;AACH,qBAAa,cAAe,YAAW,kBAAkB;IACvD,OAAO,CAAC,KAAK,CAAQ;gBAET,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM;IAIhC,MAAM,CAAC,OAAO,EAAE,YAAY,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;CAWrD"}

package/dist/adapters/vectorStore.js

@@ -0,0 +1,22 @@
+import { Index } from "@upstash/vector";
+/**
+ * Built-in adapter for Upstash Vector.
+ * Used by default when running via the CLI.
+ */
+export class UpstashAdapter {
+    index;
+    constructor(url, token) {
+        this.index = new Index({ url, token });
+    }
+    async upsert(records) {
+        const upstashRecords = records.map((r) => ({
+            id: r.id,
+            vector: r.vector,
+            metadata: r.metadata,
+            // For Upstash, the string payload goes in 'data' usually, but metadata is fine.
+            data: r.metadata.text || "",
+        }));
+        await this.index.upsert(upstashRecords);
+    }
+}
+//# sourceMappingURL=vectorStore.js.map

package/dist/adapters/vectorStore.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"vectorStore.js","sourceRoot":"","sources":["../../src/adapters/vectorStore.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,iBAAiB,CAAC;AAkBxC;;;GAGG;AACH,MAAM,OAAO,cAAc;IACjB,KAAK,CAAQ;IAErB,YAAY,GAAW,EAAE,KAAa;QACpC,IAAI,CAAC,KAAK,GAAG,IAAI,KAAK,CAAC,EAAE,GAAG,EAAE,KAAK,EAAE,CAAC,CAAC;IACzC,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,OAAuB;QAClC,MAAM,cAAc,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;YACzC,EAAE,EAAE,CAAC,CAAC,EAAE;YACR,MAAM,EAAE,CAAC,CAAC,MAAM;YAChB,QAAQ,EAAE,CAAC,CAAC,QAAQ;YACpB,gFAAgF;YAChF,IAAI,EAAE,CAAC,CAAC,QAAQ,CAAC,IAAI,IAAI,EAAE;SAC5B,CAAC,CAAC,CAAC;QAEJ,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,cAAc,CAAC,CAAC;IAC1C,CAAC;CACF"}

package/dist/aiAdapters.d.ts

@@ -0,0 +1,25 @@
+export interface LlmInput {
+    systemPrompt: string;
+    userText: string;
+    base64PdfChunk?: string;
+}
+export interface LlmAdapter {
+    generateMarkdown(input: LlmInput): Promise<string>;
+}
+export interface EmbeddingAdapter {
+    embed(chunks: string[]): Promise<number[][]>;
+}
+export declare class OpenRouterLlmAdapter implements LlmAdapter {
+    private client;
+    private model;
+    constructor(apiKey: string, model: string);
+    generateMarkdown(input: LlmInput): Promise<string>;
+}
+export declare class OpenRouterEmbeddingAdapter implements EmbeddingAdapter {
+    private client;
+    private model;
+    private dimensions;
+    constructor(apiKey: string, model: string, dimensions?: number);
+    embed(chunks: string[]): Promise<number[][]>;
+}
+//# sourceMappingURL=aiAdapters.d.ts.map

package/dist/aiAdapters.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"aiAdapters.d.ts","sourceRoot":"","sources":["../src/aiAdapters.ts"],"names":[],"mappings":"AAIA,MAAM,WAAW,QAAQ;IACvB,YAAY,EAAE,MAAM,CAAC;IACrB,QAAQ,EAAE,MAAM,CAAC;IACjB,cAAc,CAAC,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,UAAU;IACzB,gBAAgB,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;CACpD;AAED,MAAM,WAAW,gBAAgB;IAC/B,KAAK,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;CAC9C;AAID,qBAAa,oBAAqB,YAAW,UAAU;IACrD,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,KAAK,CAAS;gBAEV,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM;IAKnC,gBAAgB,CAAC,KAAK,EAAE,QAAQ,GAAG,OAAO,CAAC,MAAM,CAAC;CAsBzD;AAED,qBAAa,0BAA2B,YAAW,gBAAgB;IACjE,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,UAAU,CAAS;gBAEf,MAAM,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,UAAU,GAAE,MAAa;IAM9D,KAAK,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;CAWnD"}

package/dist/aiAdapters.js

@@ -0,0 +1,50 @@
+import { OpenAI } from "openai";
+// --- BUILT-IN ADAPTERS (For CLI to use by default) ---
+export class OpenRouterLlmAdapter {
+    client;
+    model;
+    constructor(apiKey, model) {
+        this.client = new OpenAI({ baseURL: "https://openrouter.ai/api/v1", apiKey });
+        this.model = model;
+    }
+    async generateMarkdown(input) {
+        const userContent = [];
+        if (input.base64PdfChunk) {
+            userContent.push({
+                type: "file",
+                file: { filename: "chunk.pdf", file_data: `data:application/pdf;base64,${input.base64PdfChunk}` },
+            });
+        }
+        userContent.push({ type: "text", text: input.userText });
+        const response = await this.client.chat.completions.create({
+            model: this.model,
+            messages: [
+                { role: "system", content: input.systemPrompt },
+                { role: "user", content: userContent },
+            ],
+            temperature: 0,
+        });
+        return response.choices[0]?.message?.content?.trim() ?? "";
+    }
+}
+export class OpenRouterEmbeddingAdapter {
+    client;
+    model;
+    dimensions;
+    constructor(apiKey, model, dimensions = 1536) {
+        this.client = new OpenAI({ baseURL: "https://openrouter.ai/api/v1", apiKey });
+        this.model = model;
+        this.dimensions = dimensions;
+    }
+    async embed(chunks) {
+        const response = await this.client.embeddings.create({
+            model: this.model,
+            input: chunks,
+            dimensions: this.dimensions,
+        });
+        // Sort to maintain chunk order
+        const sorted = response.data.sort((a, b) => a.index - b.index);
+        return sorted.map((item) => item.embedding);
+    }
+}
+//# sourceMappingURL=aiAdapters.js.map

package/dist/aiAdapters.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"aiAdapters.js","sourceRoot":"","sources":["../src/aiAdapters.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAE,MAAM,QAAQ,CAAC;AAkBhC,wDAAwD;AAExD,MAAM,OAAO,oBAAoB;IACvB,MAAM,CAAS;IACf,KAAK,CAAS;IAEtB,YAAY,MAAc,EAAE,KAAa;QACvC,IAAI,CAAC,MAAM,GAAG,IAAI,MAAM,CAAC,EAAE,OAAO,EAAE,8BAA8B,EAAE,MAAM,EAAE,CAAC,CAAC;QAC9E,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;IAED,KAAK,CAAC,gBAAgB,CAAC,KAAe;QACpC,MAAM,WAAW,GAAU,EAAE,CAAC;QAE9B,IAAI,KAAK,CAAC,cAAc,EAAE,CAAC;YACzB,WAAW,CAAC,IAAI,CAAC;gBACf,IAAI,EAAE,MAAM;gBACZ,IAAI,EAAE,EAAE,QAAQ,EAAE,WAAW,EAAE,SAAS,EAAE,+BAA+B,KAAK,CAAC,cAAc,EAAE,EAAE;aAClG,CAAC,CAAC;QACL,CAAC;QACD,WAAW,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC;QAEzD,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC;YACzD,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,QAAQ,EAAE;gBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,KAAK,CAAC,YAAY,EAAE;gBAC/C,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,WAAkB,EAAE;aAC9C;YACD,WAAW,EAAE,CAAC;SACf,CAAC,CAAC;QAEH,OAAO,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;IAC7D,CAAC;CACF;AAED,MAAM,OAAO,0BAA0B;IAC7B,MAAM,CAAS;IACf,KAAK,CAAS;IACd,UAAU,CAAS;IAE3B,YAAY,MAAc,EAAE,KAAa,EAAE,aAAqB,IAAI;QAClE,IAAI,CAAC,MAAM,GAAG,IAAI,MAAM,CAAC,EAAE,OAAO,EAAE,8BAA8B,EAAE,MAAM,EAAE,CAAC,CAAC;QAC9E,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;QACnB,IAAI,CAAC,UAAU,GAAG,UAAU,CAAC;IAC/B,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,MAAgB;QAC1B,MAAM,QAAQ,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,UAAU,CAAC,MAAM,CAAC;YACnD,KAAK,EAAE,IAAI,CAAC,KAAK;YACjB,KAAK,EAAE,MAAM;YACb,UAAU,EAAE,IAAI,CAAC,UAAU;SACrB,CAAC,CAAC;QAEV,+BAA+B;QAC/B,MAAM,MAAM,GAAG,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAM,EAAE,CAAM,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;QACzE,OAAO,MAAM,CAAC,GAAG,CAAC,CAAC,IAAS,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACnD,CAAC;CACF"}

package/dist/batchPipeline.d.ts

@@ -0,0 +1,52 @@
+/**
+ * State for the batch document processing graph.
+ */
+export declare const BatchStateAnnotation: import("@langchain/langgraph").AnnotationRoot<{
+    /** Input: List of absolute file paths to process */
+    files: {
+        (): import("@langchain/langgraph").LastValue<string[]>;
+        (annotation: import("@langchain/langgraph").SingleReducer<string[], string[]>): import("@langchain/langgraph").BinaryOperatorAggregate<string[], string[]>;
+        Root: <S extends import("@langchain/langgraph").StateDefinition>(sd: S) => import("@langchain/langgraph").AnnotationRoot<S>;
+    };
+    /** Output: Collection of results from each individual document run */
+    results: import("@langchain/langgraph").BinaryOperatorAggregate<any[], any[]>;
+}>;
+export type BatchState = typeof BatchStateAnnotation.State;
+export declare const graph: import("@langchain/langgraph").CompiledStateGraph<import("@langchain/langgraph").StateType<{
+    /** Input: List of absolute file paths to process */
+    files: {
+        (): import("@langchain/langgraph").LastValue<string[]>;
+        (annotation: import("@langchain/langgraph").SingleReducer<string[], string[]>): import("@langchain/langgraph").BinaryOperatorAggregate<string[], string[]>;
+        Root: <S extends import("@langchain/langgraph").StateDefinition>(sd: S) => import("@langchain/langgraph").AnnotationRoot<S>;
+    };
+    /** Output: Collection of results from each individual document run */
+    results: import("@langchain/langgraph").BinaryOperatorAggregate<any[], any[]>;
+}>, import("@langchain/langgraph").UpdateType<{
+    /** Input: List of absolute file paths to process */
+    files: {
+        (): import("@langchain/langgraph").LastValue<string[]>;
+        (annotation: import("@langchain/langgraph").SingleReducer<string[], string[]>): import("@langchain/langgraph").BinaryOperatorAggregate<string[], string[]>;
+        Root: <S extends import("@langchain/langgraph").StateDefinition>(sd: S) => import("@langchain/langgraph").AnnotationRoot<S>;
+    };
+    /** Output: Collection of results from each individual document run */
+    results: import("@langchain/langgraph").BinaryOperatorAggregate<any[], any[]>;
+}>, "__start__" | "workerNode" | "orchestrator" | "summaryNode", {
+    /** Input: List of absolute file paths to process */
+    files: {
+        (): import("@langchain/langgraph").LastValue<string[]>;
+        (annotation: import("@langchain/langgraph").SingleReducer<string[], string[]>): import("@langchain/langgraph").BinaryOperatorAggregate<string[], string[]>;
+        Root: <S extends import("@langchain/langgraph").StateDefinition>(sd: S) => import("@langchain/langgraph").AnnotationRoot<S>;
+    };
+    /** Output: Collection of results from each individual document run */
+    results: import("@langchain/langgraph").BinaryOperatorAggregate<any[], any[]>;
+}, {
+    /** Input: List of absolute file paths to process */
+    files: {
+        (): import("@langchain/langgraph").LastValue<string[]>;
+        (annotation: import("@langchain/langgraph").SingleReducer<string[], string[]>): import("@langchain/langgraph").BinaryOperatorAggregate<string[], string[]>;
+        Root: <S extends import("@langchain/langgraph").StateDefinition>(sd: S) => import("@langchain/langgraph").AnnotationRoot<S>;
+    };
+    /** Output: Collection of results from each individual document run */
+    results: import("@langchain/langgraph").BinaryOperatorAggregate<any[], any[]>;
+}, import("@langchain/langgraph").StateDefinition>;
+//# sourceMappingURL=batchPipeline.d.ts.map

package/dist/batchPipeline.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"batchPipeline.d.ts","sourceRoot":"","sources":["../src/batchPipeline.ts"],"names":[],"mappings":"AAKA;;GAEG;AACH,eAAO,MAAM,oBAAoB;IAC/B,oDAAoD;;;;;;IAGpD,sEAAsE;;EAKtE,CAAC;AAEH,MAAM,MAAM,UAAU,GAAG,OAAO,oBAAoB,CAAC,KAAK,CAAC;AAyE3D,eAAO,MAAM,KAAK;IAnFhB,oDAAoD;;;;;;IAGpD,sEAAsE;;;IAHtE,oDAAoD;;;;;;IAGpD,sEAAsE;;;IAHtE,oDAAoD;;;;;;IAGpD,sEAAsE;;;IAHtE,oDAAoD;;;;;;IAGpD,sEAAsE;;kDAgF/B,CAAC"}

package/dist/batchPipeline.js

@@ -0,0 +1,81 @@
+import { Annotation, StateGraph, Send, END } from "@langchain/langgraph";
+import { graph as singleDocGraph } from "./pipeline.js";
+import path from "node:path";
+import { logger, LogSource } from "./logger.js";
+/**
+ * State for the batch document processing graph.
+ */
+export const BatchStateAnnotation = Annotation.Root({
+    /** Input: List of absolute file paths to process */
+    files: (Annotation),
+    /** Output: Collection of results from each individual document run */
+    results: Annotation({
+        reducer: (x, y) => x.concat(y),
+        default: () => [],
+    }),
+});
+/**
+ * Orchestrator node: Prepares the batch and sends it to workers.
+ */
+function orchestrator(state) {
+    logger.info(LogSource.BATCH, `Starting processing of ${state.files.length} documents.`);
+    return {};
+}
+/**
+ * Conditional edge: Uses the Send API to spawn parallel worker nodes for each file.
+ */
+function distributeFiles(state) {
+    return state.files.map((file) => new Send("workerNode", { filePath: file }));
+}
+/**
+ * Worker node: Invokes the original single-document pipeline.
+ */
+async function workerNode(state) {
+    const fileName = path.basename(state.filePath);
+    const startTime = Date.now();
+    try {
+        // Invoke the existing compiled single-document graph
+        const result = await singleDocGraph.invoke({ filePath: state.filePath });
+        const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
+        return {
+            results: [{
+                    file: fileName,
+                    status: "success",
+                    chunks: result.textChunks?.length ?? 0,
+                    vectors: result.vectors?.length ?? 0,
+                    durationSec: elapsed,
+                }]
+        };
+    }
+    catch (error) {
+        const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
+        return {
+            results: [{
+                    file: fileName,
+                    status: "error",
+                    chunks: 0,
+                    vectors: 0,
+                    durationSec: elapsed,
+                    error: error.message,
+                }]
+        };
+    }
+}
+/**
+ * Final node: Prints a summary of the entire batch.
+ */
+function summaryNode(state) {
+    logger.success(LogSource.BATCH, "All documents processed.");
+    return {};
+}
+// Build the batch graph
+const batchGraph = new StateGraph(BatchStateAnnotation)
+    .addNode("orchestrator", orchestrator)
+    .addNode("workerNode", workerNode)
+    .addNode("summaryNode", summaryNode)
+    .addEdge("__start__", "orchestrator")
+    .addConditionalEdges("orchestrator", distributeFiles, ["workerNode"])
+    .addEdge("workerNode", "summaryNode")
+    .addEdge("summaryNode", END);
+export const graph = batchGraph.compile();
+//# sourceMappingURL=batchPipeline.js.map

package/dist/batchPipeline.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"batchPipeline.js","sourceRoot":"","sources":["../src/batchPipeline.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,GAAG,EAAE,MAAM,sBAAsB,CAAC;AACzE,OAAO,EAAE,KAAK,IAAI,cAAc,EAAE,MAAM,eAAe,CAAC;AACxD,OAAO,IAAI,MAAM,WAAW,CAAC;AAC7B,OAAO,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAEhD;;GAEG;AACH,MAAM,CAAC,MAAM,oBAAoB,GAAG,UAAU,CAAC,IAAI,CAAC;IAClD,oDAAoD;IACpD,KAAK,EAAE,CAAA,UAAoB,CAAA;IAE3B,sEAAsE;IACtE,OAAO,EAAE,UAAU,CAAQ;QACzB,OAAO,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC;QAC9B,OAAO,EAAE,GAAG,EAAE,CAAC,EAAE;KAClB,CAAC;CACH,CAAC,CAAC;AAIH;;GAEG;AACH,SAAS,YAAY,CAAC,KAAiB;IACrC,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,0BAA0B,KAAK,CAAC,KAAK,CAAC,MAAM,aAAa,CAAC,CAAC;IACxF,OAAO,EAAE,CAAC;AACZ,CAAC;AAED;;GAEG;AACH,SAAS,eAAe,CAAC,KAAiB;IACxC,OAAO,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAC9B,IAAI,IAAI,CAAC,YAAY,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAC3C,CAAC;AACJ,CAAC;AAED;;GAEG;AACH,KAAK,UAAU,UAAU,CAAC,KAA2B;IACnD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;IAC/C,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;IAE7B,IAAI,CAAC;QACH,qDAAqD;QACrD,MAAM,MAAM,GAAG,MAAM,cAAc,CAAC,MAAM,CAAC,EAAE,QAAQ,EAAE,KAAK,CAAC,QAAQ,EAAE,CAAC,CAAC;QACzE,MAAM,OAAO,GAAG,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC,GAAG,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAE7D,OAAO;YACL,OAAO,EAAE,CAAC;oBACR,IAAI,EAAE,QAAQ;oBACd,MAAM,EAAE,SAAS;oBACjB,MAAM,EAAE,MAAM,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC;oBACtC,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,MAAM,IAAI,CAAC;oBACpC,WAAW,EAAE,OAAO;iBACrB,CAAC;SACH,CAAC;IACJ,CAAC;IAAC,OAAO,KAAU,EAAE,CAAC;QACpB,MAAM,OAAO,GAAG,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,SAAS,CAAC,GAAG,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;QAC7D,OAAO;YACL,OAAO,EAAE,CAAC;oBACR,IAAI,EAAE,QAAQ;oBACd,MAAM,EAAE,OAAO;oBACf,MAAM,EAAE,CAAC;oBACT,OAAO,EAAE,CAAC;oBACV,WAAW,EAAE,OAAO;oBACpB,KAAK,EAAE,KAAK,CAAC,OAAO;iBACrB,CAAC;SACH,CAAC;IACJ,CAAC;AACH,CAAC;AAED;;GAEG;AACH,SAAS,WAAW,CAAC,KAAiB;IACpC,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,KAAK,EAAE,0BAA0B,CAAC,CAAC;IAC5D,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,wBAAwB;AACxB,MAAM,UAAU,GAAG,IAAI,UAAU,CAAC,oBAAoB,CAAC;KACpD,OAAO,CAAC,cAAc,EAAE,YAAY,CAAC;KACrC,OAAO,CAAC,YAAY,EAAE,UAAU,CAAC;KACjC,OAAO,CAAC,aAAa,EAAE,WAAW,CAAC;KACnC,OAAO,CAAC,WAAW,EAAE,cAAc,CAAC;KACpC,mBAAmB,CAAC,cAAc,EAAE,eAAe,EAAE,CAAC,YAAY,CAAC,CAAC;KACpE,OAAO,CAAC,YAAY,EAAE,aAAa,CAAC;KACpC,OAAO,CAAC,aAAa,EAAE,GAAG,CAAC,CAAC;AAE/B,MAAM,CAAC,MAAM,KAAK,GAAG,UAAU,CAAC,OAAO,EAAE,CAAC"}

package/dist/cli.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+import "dotenv/config";
+//# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AACA,OAAO,eAAe,CAAC"}