@devilsdev/rag-pipeline-utils 1.0.0 → 2.0.0

package/docs-site/docs/Architecture.md

@@ -1,76 +1,76 @@
-# Architecture
-
-This page outlines the internal architecture of the RAG pipeline utilities, emphasizing modularity, plugin design, and SOLID-compliant structure.
-
----
-
-## 🧩 Core Design Philosophy
-
-The architecture adheres to enterprise-grade practices:
-
-- **Single Responsibility**: Each component handles one domain concern
-- **Pluggable Interfaces**: Any layer (loader, retriever, LLM) can be swapped
-- **Streaming-Ready**: Async flows support token-based output
-- **Environment-Safe**: Config via `.ragrc.json`, not hardcoded
-- **Testable**: All modules can be mocked and unit tested
-
----
-
-## 🏗️ Layered Components
-
-```
-+-------------------------+
-| createRagPipeline()     |
-+-------------------------+
-        |     |     |
-        ▼     ▼     ▼
-  Loader   Embedder  Retriever
-     ▼        |         ▼
-  Chunks   Embeddings  Context
-        \     |     /
-         ▼    ▼    ▼
-           LLM Runner
-             |
-           Output
-```
-
----
-
-## 📦 Core Interfaces
-
-Each plugin is registered via the `PluginRegistry` and looked up by key.
-
-```ts
-registry.register('loader', 'pdf', new PDFLoader())
-const loader = registry.get('loader', 'pdf');
-```
-
-All plugins implement interface contracts like:
-
-```ts
-interface Loader {
-  load(path: string): Promise<{ chunk(): string[] }[]>;
-}
-
-interface Embedder {
-  embed(chunks: string[]): Vector[];
-  embedQuery(prompt: string): Vector;
-}
-
-interface Retriever {
-  store(vectors: Vector[]): Promise<void>;
-  retrieve(query: Vector): Promise<Context[]>;
-}
-```
-
----
-
-## 🔁 DAG Support
-
-The `dag-engine.js` module supports chaining multiple components:
-- Example: Summarize → Retrieve → Rerank → LLM
-- Enables more complex workflows than linear ingestion/query
-
----
-
-Next → [Evaluation](./Evaluation.md)
+# Architecture
+
+This page outlines the internal architecture of the RAG pipeline utilities, emphasizing modularity, plugin design, and SOLID-compliant structure.
+
+---
+
+## 🧩 Core Design Philosophy
+
+The architecture adheres to enterprise-grade practices:
+
+- **Single Responsibility**: Each component handles one domain concern
+- **Pluggable Interfaces**: Any layer (loader, retriever, LLM) can be swapped
+- **Streaming-Ready**: Async flows support token-based output
+- **Environment-Safe**: Config via `.ragrc.json`, not hardcoded
+- **Testable**: All modules can be mocked and unit tested
+
+---
+
+## 🏗️ Layered Components
+
+```
++-------------------------+
+| createRagPipeline()     |
++-------------------------+
+        |     |     |
+        ▼     ▼     ▼
+  Loader   Embedder  Retriever
+     ▼        |         ▼
+  Chunks   Embeddings  Context
+        \     |     /
+         ▼    ▼    ▼
+           LLM Runner
+             |
+           Output
+```
+
+---
+
+## 📦 Core Interfaces
+
+Each plugin is registered via the `PluginRegistry` and looked up by key.
+
+```ts
+registry.register('loader', 'pdf', new PDFLoader())
+const loader = registry.get('loader', 'pdf');
+```
+
+All plugins implement interface contracts like:
+
+```ts
+interface Loader {
+  load(path: string): Promise<{ chunk(): string[] }[]>;
+}
+
+interface Embedder {
+  embed(chunks: string[]): Vector[];
+  embedQuery(prompt: string): Vector;
+}
+
+interface Retriever {
+  store(vectors: Vector[]): Promise<void>;
+  retrieve(query: Vector): Promise<Context[]>;
+}
+```
+
+---
+
+## 🔁 DAG Support
+
+The `dag-engine.js` module supports chaining multiple components:
+- Example: Summarize → Retrieve → Rerank → LLM
+- Enables more complex workflows than linear ingestion/query
+
+---
+
+Next → [Evaluation](./Evaluation.md)

package/docs-site/docs/CLI.md

@@ -1,69 +1,69 @@
-# CLI Reference
-
-The CLI gives you interactive access to the RAG pipeline tools via terminal.
-
-Install globally (optional):
-```bash
-npm install -g @yourorg/rag-pipeline-utils
-```
-
-Or run directly from your project:
-```bash
-npx rag-pipeline <command>
-```
-
----
-
-## 🔧 Commands
-
-### `ingest`
-```bash
-rag-pipeline ingest ./docs/sample.pdf --loader pdf
-```
-- Loads and indexes documents into vector store
-
-### `query`
-```bash
-rag-pipeline query "Explain vector databases" --llm openai-gpt-4
-```
-- Retrieves context and generates response
-
-### `evaluate`
-```bash
-rag-pipeline evaluate ./fixtures/eval.json
-```
-- Runs batch evaluation with BLEU/ROUGE scoring
-
-### `rerank`
-```bash
-rag-pipeline rerank "What is RAG?" --retriever pinecone
-```
-- Reorders retrieved context using an LLM reranker
-
----
-
-## ⚙️ `.ragrc.json` Config
-
-Defaults can be set in a file:
-```json
-{
-  "loader": "directory",
-  "embedder": "openai",
-  "retriever": "pinecone",
-  "llm": "openai-gpt-4",
-  "useReranker": true
-}
-```
-
----
-
-## 🐳 Docker Usage
-
-```bash
-docker build -t rag-pipeline .
-docker run --rm -v $PWD:/data rag-pipeline query "What is retrieval?"
-```
-
----
-
-Next → [Architecture](./Architecture.md)
+# CLI Reference
+
+The CLI gives you interactive access to the RAG pipeline tools via terminal.
+
+Install globally (optional):
+```bash
+npm install -g @yourorg/rag-pipeline-utils
+```
+
+Or run directly from your project:
+```bash
+npx rag-pipeline <command>
+```
+
+---
+
+## 🔧 Commands
+
+### `ingest`
+```bash
+rag-pipeline ingest ./docs/sample.pdf --loader pdf
+```
+- Loads and indexes documents into vector store
+
+### `query`
+```bash
+rag-pipeline query "Explain vector databases" --llm openai-gpt-4
+```
+- Retrieves context and generates response
+
+### `evaluate`
+```bash
+rag-pipeline evaluate ./fixtures/eval.json
+```
+- Runs batch evaluation with BLEU/ROUGE scoring
+
+### `rerank`
+```bash
+rag-pipeline rerank "What is RAG?" --retriever pinecone
+```
+- Reorders retrieved context using an LLM reranker
+
+---
+
+## ⚙️ `.ragrc.json` Config
+
+Defaults can be set in a file:
+```json
+{
+  "loader": "directory",
+  "embedder": "openai",
+  "retriever": "pinecone",
+  "llm": "openai-gpt-4",
+  "useReranker": true
+}
+```
+
+---
+
+## 🐳 Docker Usage
+
+```bash
+docker build -t rag-pipeline .
+docker run --rm -v $PWD:/data rag-pipeline query "What is retrieval?"
+```
+
+---
+
+Next → [Architecture](./Architecture.md)

package/docs-site/docs/Evaluation.md

@@ -1,61 +1,61 @@
-# Evaluation
-
-This module allows you to evaluate the quality of generated LLM responses in a RAG pipeline.
-
----
-
-## 🎯 Objective
-
-Quantify how well the pipeline performs on question-answer pairs using:
-- **BLEU**: Precision on matching tokens
-- **ROUGE**: Recall + overlap on longest subsequences
-- **LLM Feedback** *(planned)*: Score generation based on LLM ratings
-
----
-
-## 📄 Dataset Format
-
-Each entry in your JSON dataset should include:
-
-```json
-{
-  "prompt": "What is a vector index?",
-  "groundTruth": "A data structure for fast similarity search in vector spaces."
-}
-```
-
-Saved as: `sample-eval-dataset.json`
-
----
-
-## 🚀 CLI Evaluation
-
-```bash
-rag-pipeline evaluate ./fixtures/sample-eval-dataset.json
-```
-
-Outputs:
-- ✅ Pass/fail per item
-- 📊 BLEU/ROUGE scores
-- 📈 Aggregated metrics
-
----
-
-## 📊 Dashboard Visualization
-
-Use the React dashboard under `/public`:
-
-```bash
-node server.js
-```
-
-Visit: [http://localhost:3000](http://localhost:3000)
-
-Displays:
-- Prompt vs Answer
-- Success criteria
-- Metric charts (BLEU, ROUGE)
-
----
-
-Next → [Plugins](./Plugins.md)
+# Evaluation
+
+This module allows you to evaluate the quality of generated LLM responses in a RAG pipeline.
+
+---
+
+## 🎯 Objective
+
+Quantify how well the pipeline performs on question-answer pairs using:
+- **BLEU**: Precision on matching tokens
+- **ROUGE**: Recall + overlap on longest subsequences
+- **LLM Feedback** *(planned)*: Score generation based on LLM ratings
+
+---
+
+## 📄 Dataset Format
+
+Each entry in your JSON dataset should include:
+
+```json
+{
+  "prompt": "What is a vector index?",
+  "groundTruth": "A data structure for fast similarity search in vector spaces."
+}
+```
+
+Saved as: `sample-eval-dataset.json`
+
+---
+
+## 🚀 CLI Evaluation
+
+```bash
+rag-pipeline evaluate ./fixtures/sample-eval-dataset.json
+```
+
+Outputs:
+- ✅ Pass/fail per item
+- 📊 BLEU/ROUGE scores
+- 📈 Aggregated metrics
+
+---
+
+## 📊 Dashboard Visualization
+
+Use the React dashboard under `/public`:
+
+```bash
+node server.js
+```
+
+Visit: [http://localhost:3000](http://localhost:3000)
+
+Displays:
+- Prompt vs Answer
+- Success criteria
+- Metric charts (BLEU, ROUGE)
+
+---
+
+Next → [Plugins](./Plugins.md)

package/docs-site/docs/FAQ.md

@@ -1,50 +1,50 @@
-# FAQ
-
-### ❓ Can I use this with my own vector database?
-Yes! You can implement the `Retriever` interface and register it:
-```ts
-registry.register('retriever', 'mydb', new MyRetriever());
-```
-Then use it via CLI or `.ragrc.json`.
-
----
-
-### ❓ Is streaming LLM response supported?
-Yes. LLM runners can yield async output tokens. This is handled internally via `AsyncIterable` in compatible models.
-
----
-
-### ❓ How do I evaluate performance?
-Use `rag-pipeline evaluate dataset.json` or open the dashboard at `http://localhost:3000` to visualize BLEU/ROUGE and pass rates.
-
----
-
-### ❓ What file types can I ingest?
-Currently supported:
-- `.pdf` (mocked)
-- `.md`
-- `.html`
-- `.csv`
-- Full directory ingestion
-
----
-
-### ❓ Can I rerank results using a local model?
-Yes, as long as your local LLM implements:
-```ts
-llm.generate(prompt, context): Promise<string>
-```
-Use it to power `LLMReranker`.
-
----
-
-### ❓ How do I contribute?
-Fork the repo → add your plugin or CLI feature → test → submit PR.
-
----
-
-Still stuck? Open an issue at [GitHub](https://github.com/DevilsDev/rag-pipeline-utils/issues).
-
----
-
-Next → [Back to Introduction](./Introduction.md)
+# FAQ
+
+### ❓ Can I use this with my own vector database?
+Yes! You can implement the `Retriever` interface and register it:
+```ts
+registry.register('retriever', 'mydb', new MyRetriever());
+```
+Then use it via CLI or `.ragrc.json`.
+
+---
+
+### ❓ Is streaming LLM response supported?
+Yes. LLM runners can yield async output tokens. This is handled internally via `AsyncIterable` in compatible models.
+
+---
+
+### ❓ How do I evaluate performance?
+Use `rag-pipeline evaluate dataset.json` or open the dashboard at `http://localhost:3000` to visualize BLEU/ROUGE and pass rates.
+
+---
+
+### ❓ What file types can I ingest?
+Currently supported:
+- `.pdf` (mocked)
+- `.md`
+- `.html`
+- `.csv`
+- Full directory ingestion
+
+---
+
+### ❓ Can I rerank results using a local model?
+Yes, as long as your local LLM implements:
+```ts
+llm.generate(prompt, context): Promise<string>
+```
+Use it to power `LLMReranker`.
+
+---
+
+### ❓ How do I contribute?
+Fork the repo → add your plugin or CLI feature → test → submit PR.
+
+---
+
+Still stuck? Open an issue at [GitHub](https://github.com/DevilsDev/rag-pipeline-utils/issues).
+
+---
+
+Next → [Back to Introduction](./Introduction.md)

package/docs-site/docs/Plugins.md

@@ -1,61 +1,61 @@
-# Plugins
-
-The toolkit is fully modular. You can inject your own loaders, retrievers, embedders, or LLMs.
-
----
-
-## 🔌 Plugin Architecture
-
-All plugins follow a strict interface pattern. Each is registered under a namespace using the plugin registry:
-
-```ts
-registry.register('loader', 'my-custom', new MyLoader());
-const loader = registry.get('loader', 'my-custom');
-```
-
-You can register plugins for:
-- `loader`
-- `embedder`
-- `retriever`
-- `llm`
-
----
-
-## 🧱 Loader Example
-
-```ts
-class MyMarkdownLoader {
-  async load(filePath) {
-    const raw = fs.readFileSync(filePath, 'utf-8');
-    return [{ chunk: () => raw.split('\n\n') }];
-  }
-}
-```
-
-Register it:
-```ts
-registry.register('loader', 'md', new MyMarkdownLoader());
-```
-
----
-
-## ⚡ Runtime Override
-
-CLI and API will resolve plugin names from `pluginRegistry`. Use:
-
-```json
-{
-  "loader": "md",
-  "embedder": "cohere",
-  "retriever": "chroma",
-  "llm": "openai-gpt-4"
-}
-```
-
-in `.ragrc.json`
-
----
-
-This design keeps the core minimal and lets you evolve.
-
-Next → [FAQ](./FAQ.md)
+# Plugins
+
+The toolkit is fully modular. You can inject your own loaders, retrievers, embedders, or LLMs.
+
+---
+
+## 🔌 Plugin Architecture
+
+All plugins follow a strict interface pattern. Each is registered under a namespace using the plugin registry:
+
+```ts
+registry.register('loader', 'my-custom', new MyLoader());
+const loader = registry.get('loader', 'my-custom');
+```
+
+You can register plugins for:
+- `loader`
+- `embedder`
+- `retriever`
+- `llm`
+
+---
+
+## 🧱 Loader Example
+
+```ts
+class MyMarkdownLoader {
+  async load(filePath) {
+    const raw = fs.readFileSync(filePath, 'utf-8');
+    return [{ chunk: () => raw.split('\n\n') }];
+  }
+}
+```
+
+Register it:
+```ts
+registry.register('loader', 'md', new MyMarkdownLoader());
+```
+
+---
+
+## ⚡ Runtime Override
+
+CLI and API will resolve plugin names from `pluginRegistry`. Use:
+
+```json
+{
+  "loader": "md",
+  "embedder": "cohere",
+  "retriever": "chroma",
+  "llm": "openai-gpt-4"
+}
+```
+
+in `.ragrc.json`
+
+---
+
+This design keeps the core minimal and lets you evolve.
+
+Next → [FAQ](./FAQ.md)