@devilsdev/rag-pipeline-utils 1.0.0

package/docs-site/blog/authors.yml

@@ -0,0 +1,23 @@
+yangshun:
+  name: Yangshun Tay
+  title: Front End Engineer @ Facebook
+  url: https://github.com/yangshun
+  image_url: https://github.com/yangshun.png
+  page: true
+  socials:
+    x: yangshunz
+    github: yangshun
+
+slorber:
+  name: Sébastien Lorber
+  title: Docusaurus maintainer
+  url: https://sebastienlorber.com
+  image_url: https://github.com/slorber.png
+  page:
+    # customize the url of the author page at /blog/authors/<permalink>
+    permalink: '/all-sebastien-lorber-articles'
+  socials:
+    x: sebastienlorber
+    linkedin: sebastienlorber
+    github: slorber
+    newsletter: https://thisweekinreact.com

package/docs-site/blog/tags.yml

@@ -0,0 +1,19 @@
+facebook:
+  label: Facebook
+  permalink: /facebook
+  description: Facebook tag description
+
+hello:
+  label: Hello
+  permalink: /hello
+  description: Hello tag description
+
+docusaurus:
+  label: Docusaurus
+  permalink: /docusaurus
+  description: Docusaurus tag description
+
+hola:
+  label: Hola
+  permalink: /hola
+  description: Hola tag description

package/docs-site/docs/Architecture.md

@@ -0,0 +1,76 @@
+# Architecture
+
+This page outlines the internal architecture of the RAG pipeline utilities, emphasizing modularity, plugin design, and SOLID-compliant structure.
+
+---
+
+## 🧩 Core Design Philosophy
+
+The architecture adheres to enterprise-grade practices:
+
+- **Single Responsibility**: Each component handles one domain concern
+- **Pluggable Interfaces**: Any layer (loader, retriever, LLM) can be swapped
+- **Streaming-Ready**: Async flows support token-based output
+- **Environment-Safe**: Config via `.ragrc.json`, not hardcoded
+- **Testable**: All modules can be mocked and unit tested
+
+---
+
+## 🏗️ Layered Components
+
+```
++-------------------------+
+| createRagPipeline()     |
++-------------------------+
+        |     |     |
+        ▼     ▼     ▼
+  Loader   Embedder  Retriever
+     ▼        |         ▼
+  Chunks   Embeddings  Context
+        \     |     /
+         ▼    ▼    ▼
+           LLM Runner
+             |
+           Output
+```
+
+---
+
+## 📦 Core Interfaces
+
+Each plugin is registered via the `PluginRegistry` and looked up by key.
+
+```ts
+registry.register('loader', 'pdf', new PDFLoader())
+const loader = registry.get('loader', 'pdf');
+```
+
+All plugins implement interface contracts like:
+
+```ts
+interface Loader {
+  load(path: string): Promise<{ chunk(): string[] }[]>;
+}
+
+interface Embedder {
+  embed(chunks: string[]): Vector[];
+  embedQuery(prompt: string): Vector;
+}
+
+interface Retriever {
+  store(vectors: Vector[]): Promise<void>;
+  retrieve(query: Vector): Promise<Context[]>;
+}
+```
+
+---
+
+## 🔁 DAG Support
+
+The `dag-engine.js` module supports chaining multiple components:
+- Example: Summarize → Retrieve → Rerank → LLM
+- Enables more complex workflows than linear ingestion/query
+
+---
+
+Next → [Evaluation](./Evaluation.md)

package/docs-site/docs/CLI.md

@@ -0,0 +1,69 @@
+# CLI Reference
+
+The CLI gives you interactive access to the RAG pipeline tools via terminal.
+
+Install globally (optional):
+```bash
+npm install -g @yourorg/rag-pipeline-utils
+```
+
+Or run directly from your project:
+```bash
+npx rag-pipeline <command>
+```
+
+---
+
+## 🔧 Commands
+
+### `ingest`
+```bash
+rag-pipeline ingest ./docs/sample.pdf --loader pdf
+```
+- Loads and indexes documents into vector store
+
+### `query`
+```bash
+rag-pipeline query "Explain vector databases" --llm openai-gpt-4
+```
+- Retrieves context and generates response
+
+### `evaluate`
+```bash
+rag-pipeline evaluate ./fixtures/eval.json
+```
+- Runs batch evaluation with BLEU/ROUGE scoring
+
+### `rerank`
+```bash
+rag-pipeline rerank "What is RAG?" --retriever pinecone
+```
+- Reorders retrieved context using an LLM reranker
+
+---
+
+## ⚙️ `.ragrc.json` Config
+
+Defaults can be set in a file:
+```json
+{
+  "loader": "directory",
+  "embedder": "openai",
+  "retriever": "pinecone",
+  "llm": "openai-gpt-4",
+  "useReranker": true
+}
+```
+
+---
+
+## 🐳 Docker Usage
+
+```bash
+docker build -t rag-pipeline .
+docker run --rm -v $PWD:/data rag-pipeline query "What is retrieval?"
+```
+
+---
+
+Next → [Architecture](./Architecture.md)

package/docs-site/docs/Evaluation.md

@@ -0,0 +1,61 @@
+# Evaluation
+
+This module allows you to evaluate the quality of generated LLM responses in a RAG pipeline.
+
+---
+
+## 🎯 Objective
+
+Quantify how well the pipeline performs on question-answer pairs using:
+- **BLEU**: Precision on matching tokens
+- **ROUGE**: Recall + overlap on longest subsequences
+- **LLM Feedback** *(planned)*: Score generation based on LLM ratings
+
+---
+
+## 📄 Dataset Format
+
+Each entry in your JSON dataset should include:
+
+```json
+{
+  "prompt": "What is a vector index?",
+  "groundTruth": "A data structure for fast similarity search in vector spaces."
+}
+```
+
+Saved as: `sample-eval-dataset.json`
+
+---
+
+## 🚀 CLI Evaluation
+
+```bash
+rag-pipeline evaluate ./fixtures/sample-eval-dataset.json
+```
+
+Outputs:
+- ✅ Pass/fail per item
+- 📊 BLEU/ROUGE scores
+- 📈 Aggregated metrics
+
+---
+
+## 📊 Dashboard Visualization
+
+Use the React dashboard under `/public`:
+
+```bash
+node server.js
+```
+
+Visit: [http://localhost:3000](http://localhost:3000)
+
+Displays:
+- Prompt vs Answer
+- Success criteria
+- Metric charts (BLEU, ROUGE)
+
+---
+
+Next → [Plugins](./Plugins.md)

package/docs-site/docs/FAQ.md

@@ -0,0 +1,50 @@
+# FAQ
+
+### ❓ Can I use this with my own vector database?
+Yes! You can implement the `Retriever` interface and register it:
+```ts
+registry.register('retriever', 'mydb', new MyRetriever());
+```
+Then use it via CLI or `.ragrc.json`.
+
+---
+
+### ❓ Is streaming LLM response supported?
+Yes. LLM runners can yield async output tokens. This is handled internally via `AsyncIterable` in compatible models.
+
+---
+
+### ❓ How do I evaluate performance?
+Use `rag-pipeline evaluate dataset.json` or open the dashboard at `http://localhost:3000` to visualize BLEU/ROUGE and pass rates.
+
+---
+
+### ❓ What file types can I ingest?
+Currently supported:
+- `.pdf` (mocked)
+- `.md`
+- `.html`
+- `.csv`
+- Full directory ingestion
+
+---
+
+### ❓ Can I rerank results using a local model?
+Yes, as long as your local LLM implements:
+```ts
+llm.generate(prompt, context): Promise<string>
+```
+Use it to power `LLMReranker`.
+
+---
+
+### ❓ How do I contribute?
+Fork the repo → add your plugin or CLI feature → test → submit PR.
+
+---
+
+Still stuck? Open an issue at [GitHub](https://github.com/DevilsDev/rag-pipeline-utils/issues).
+
+---
+
+Next → [Back to Introduction](./Introduction.md)

package/docs-site/docs/Introduction.md

@@ -0,0 +1,58 @@
+---
+sidebar_position: 1
+---
+
+# Introduction
+
+Welcome to **@yourorg/rag-pipeline-utils**, a modular Node.js toolkit for building scalable, pluggable Retrieval-Augmented Generation (RAG) systems.
+
+---
+
+## 🚀 What is RAG?
+
+RAG combines information retrieval with large language models to enhance generation with factual grounding. It typically involves:
+
+1. **Retrieving** relevant context (documents, chunks) from a vector store
+2. **Generating** answers using an LLM, augmented by that context
+
+This project provides all the composable parts to build your own RAG pipeline.
+
+---
+
+## 🔧 Why Use This Toolkit?
+
+- **Plugin-based**: Swap in your own loaders, retrievers, LLMs
+- **Streaming-ready**: Async-friendly, token-by-token output
+- **CLI & API**: Use from terminal or integrate programmatically
+- **Evaluation built-in**: BLEU, ROUGE, dashboard UI
+- **Enterprise-quality**: SOLID, tested, typed, and CI-ready
+
+---
+
+## 🧱 Architecture
+
+Each RAG pipeline includes:
+
+- `loader`: Document loader (e.g., PDF, Markdown, HTML)
+- `embedder`: Embedding model interface (OpenAI, Cohere, local)
+- `retriever`: Vector store adapter (e.g., Pinecone, Chroma)
+- `llm`: Language model runner (OpenAI, Ollama, GPT-4-V)
+- `reranker`: (Optional) Context re-ranking module via LLM
+
+The pipeline can be used via code or via CLI commands.
+
+---
+
+## ✅ Status
+
+This project is:
+- ✅ Production-ready
+- ✅ Covered by unit/integration tests
+- ✅ Compatible with Node.js `>=18`
+- ✅ Published to npm under `@yourorg`
+
+---
+
+Let’s get started!
+
+Next → [Usage](./Usage.md)

package/docs-site/docs/Plugins.md

@@ -0,0 +1,61 @@
+# Plugins
+
+The toolkit is fully modular. You can inject your own loaders, retrievers, embedders, or LLMs.
+
+---
+
+## 🔌 Plugin Architecture
+
+All plugins follow a strict interface pattern. Each is registered under a namespace using the plugin registry:
+
+```ts
+registry.register('loader', 'my-custom', new MyLoader());
+const loader = registry.get('loader', 'my-custom');
+```
+
+You can register plugins for:
+- `loader`
+- `embedder`
+- `retriever`
+- `llm`
+
+---
+
+## 🧱 Loader Example
+
+```ts
+class MyMarkdownLoader {
+  async load(filePath) {
+    const raw = fs.readFileSync(filePath, 'utf-8');
+    return [{ chunk: () => raw.split('\n\n') }];
+  }
+}
+```
+
+Register it:
+```ts
+registry.register('loader', 'md', new MyMarkdownLoader());
+```
+
+---
+
+## ⚡ Runtime Override
+
+CLI and API will resolve plugin names from `pluginRegistry`. Use:
+
+```json
+{
+  "loader": "md",
+  "embedder": "cohere",
+  "retriever": "chroma",
+  "llm": "openai-gpt-4"
+}
+```
+
+in `.ragrc.json`
+
+---
+
+This design keeps the core minimal and lets you evolve.
+
+Next → [FAQ](./FAQ.md)

package/docs-site/docs/Usage.md

@@ -0,0 +1,72 @@
+# Usage
+
+This guide covers how to use `@yourorg/rag-pipeline-utils` programmatically via API and through the CLI.
+
+---
+
+## 📦 Installation
+
+```bash
+npm install @yourorg/rag-pipeline-utils
+```
+
+---
+
+## 🔧 Programmatic API
+
+Import and instantiate a RAG pipeline:
+
+```js
+import { createRagPipeline } from '@yourorg/rag-pipeline-utils';
+
+const pipeline = createRagPipeline({
+  loader: 'markdown',
+  embedder: 'openai',
+  retriever: 'pinecone',
+  llm: 'openai-gpt-4',
+  useReranker: true
+});
+
+const answer = await pipeline.query("What is retrieval-augmented generation?");
+console.log(answer);
+```
+
+---
+
+## 🔁 Pipeline Methods
+
+- `pipeline.ingest(path: string)` → Load, chunk, embed, and store vectors
+- `pipeline.query(prompt: string)` → Retrieve context, rerank (optional), and call LLM
+
+---
+
+## 🧩 Plugin Configuration
+
+You can customize the pipeline by registering plugins:
+
+```js
+registry.register('loader', 'custom', new MyCustomLoader());
+registry.register('retriever', 'opensearch', new MyOpenSearchRetriever());
+```
+
+---
+
+## ⚙️ Configuration via `.ragrc.json`
+
+Create a JSON file at your root:
+
+```json
+{
+  "loader": "directory",
+  "embedder": "openai",
+  "retriever": "pinecone",
+  "llm": "openai-gpt-4",
+  "useReranker": true
+}
+```
+
+Used automatically when no CLI args are passed.
+
+---
+
+Next → [CLI](./CLI.md)

package/docs-site/docusaurus.config.js

@@ -0,0 +1,57 @@
+/**
+ * Version: 0.3.0
+ * Path: /docs-site/docusaurus.config.js
+ * Description: Final production config with cleaned branding and image setup
+ * Author: Ali Kahwaji
+ */
+
+// @ts-check
+
+export default {
+  title: 'RAG Pipeline Utils',
+  tagline: 'Composable RAG pipelines with Node.js',
+  url: 'https://devilsdev.github.io',
+  baseUrl: '/rag-pipeline-utils/',
+  onBrokenLinks: 'throw',
+  onBrokenMarkdownLinks: 'warn',
+
+  // ✅ Final favicon and social card for branding
+  favicon: 'img/favicon.ico',
+  themeConfig: {
+    image: 'img/social-card.jpg',
+    navbar: {
+      title: 'RAG Utils',
+      logo: {
+        alt: '', // ✅ Clean alt to avoid raw rendering
+        src: '/img/logo.svg'
+      },
+      items: [
+        { to: '/docs/Introduction', label: 'Docs', position: 'left' },
+        { href: 'https://github.com/DevilsDev/rag-pipeline-utils', label: 'GitHub', position: 'right' }
+      ]
+    },
+    footer: {
+      style: 'dark',
+      copyright: `© ${new Date().getFullYear()} Ali Kahwaji`
+    }
+  },
+
+  organizationName: 'DevilsDev',
+  projectName: 'rag-pipeline-utils',
+  deploymentBranch: 'gh-pages',
+
+  presets: [
+    [
+      'classic',
+      {
+        docs: {
+          sidebarPath: require.resolve('./sidebars.js'),
+          editUrl: 'https://github.com/DevilsDev/rag-pipeline-utils/edit/main/docs-site/'
+        },
+        theme: {
+          customCss: require.resolve('./src/css/custom.css')
+        }
+      }
+    ]
+  ]
+};