@devilsdev/rag-pipeline-utils 2.1.7 → 2.1.8

package/docs-site/blog/2025-04-21-release-v2.1.3.md

@@ -1,13 +0,0 @@
----
-slug: release-v2.1.3
-title: ✨ Version v2.1.3 Released
-authors: [ali]
-tags: [release, changelog]
----
-
-RAG Pipeline Utils vv2.1.3 is now available on NPM!
-
-#### Highlights
-- 🚀 Auto-published blog from GitHub Release
-- 📦 Version bump: v2.1.3
-- 📘 See CHANGELOG.md for full details

package/docs-site/blog/2025-04-21-release-v2.1.4.md

@@ -1,13 +0,0 @@
----
-slug: release-v2.1.4
-title: ✨ Version v2.1.4 Released
-authors: [ali]
-tags: [release, changelog]
----
-
-RAG Pipeline Utils vv2.1.4 is now available on NPM!
-
-#### Highlights
-- 🚀 Auto-published blog from GitHub Release
-- 📦 Version bump: v2.1.4
-- 📘 See CHANGELOG.md for full details

package/docs-site/blog/2025-04-22-release-v2.1.5.md

@@ -1,13 +0,0 @@
----
-slug: release-v2.1.5
-title: ✨ Version v2.1.5 Released
-authors: [ali]
-tags: [release, changelog]
----
-
-RAG Pipeline Utils vv2.1.5 is now available on NPM!
-
-#### Highlights
-- 🚀 Auto-published blog from GitHub Release
-- 📦 Version bump: v2.1.5
-- 📘 See CHANGELOG.md for full details

package/docs-site/blog/2025-04-24-v2.1.6.md

@@ -1,21 +0,0 @@
----
-slug: release-v2.1.6
-title: "🚀 Version v2.1.6 Released"
-authors: [ali]
-tags: [release, changelog]
----
-
-RAG Pipeline Utils **v2.1.6** is now available on NPM!
-
-## 🔧 Changes
-
-- chore: remove legacy changelog workflow and finalize blog badge
-- ci(workflows): add blog badge verification workflow
-- ci(workflows): add blog badge verification workflow
-- ci(workflows): add blog badge verification workflow
-- docs(blog): add release post for v2.1.5
-
-## 🔗 Resources
-
-- GitHub Compare: [v2.1.5...v2.1.6](https://github.com/DevilsDev/rag-pipeline-utils/compare/v2.1.5...v2.1.6)
-- View full [CHANGELOG.md](../../CHANGELOG.md)

package/docs-site/blog/authors.yml

@@ -1,5 +0,0 @@
-ali:
-  name: Ali Kahwaji
-  title: Creator & Maintainer of RAG Pipeline Utils
-  url: https://github.com/alikahwaji
-  image_url: /img/authors/ali.jpg

package/docs-site/blog/tags.yml

@@ -1,60 +0,0 @@
-# docs-site/blog/tags.yml
-# Description: Central tag definitions for Docusaurus blog filtering and grouping
-
-release:
-  label: Release
-  permalink: /blog/tags/release
-
-architecture:
-  label: Architecture
-  permalink: /blog/tags/architecture
-
-prompt-engineering:
-  label: Prompt Engineering
-  permalink: /blog/tags/prompt-engineering
-
-evaluation:
-  label: Evaluation
-  permalink: /blog/tags/evaluation
-
-tutorials:
-  label: Tutorials
-  permalink: /blog/tags/tutorials
-
-changelog:
-  label: Changelog
-  permalink: /blog/tags/changelog
-
-design:
-  label: Design
-  permalink: /blog/tags/design
-
-rag:
-  label: RAG
-  permalink: /blog/tags/rag
-
-developer-tools:
-  label: Developer Tools
-  permalink: /blog/tags/developer-tools
-
-modular:
-  label: Modular
-  permalink: /blog/tags/modular
-
-image:
-  label: Image
-  permalink: /blog/tags/image
-
-facebook:
-  label: Facebook
-  permalink: /blog/tags/facebook
-
-hello:
-  label: Hello
-  permalink: /blog/tags/hello
-
-docusaurus:
-  label: Docusaurus
-  permalink: /blog/tags/docusaurus
-
-

package/docs-site/docs/Architecture.md

@@ -1,76 +0,0 @@
-# Architecture
-
-This page outlines the internal architecture of the RAG pipeline utilities, emphasizing modularity, plugin design, and SOLID-compliant structure.
-
----
-
-## Core Design Philosophy
-
-The architecture adheres to enterprise-grade practices:
-
-- **Single Responsibility**: Each component handles one domain concern
-- **Pluggable Interfaces**: Any layer (loader, retriever, LLM) can be swapped
-- **Streaming-Ready**: Async flows support token-based output
-- **Environment-Safe**: Config via `.ragrc.json`, not hardcoded
-- **Testable**: All modules can be mocked and unit tested
-
----
-
-## Layered Components
-
-```
-+-------------------------+
-| createRagPipeline()     |
-+-------------------------+
-        |     |     |
-        ▼     ▼     ▼
-  Loader   Embedder  Retriever
-     ▼        |         ▼
-  Chunks   Embeddings  Context
-        \     |     /
-         ▼    ▼    ▼
-           LLM Runner
-             |
-           Output
-```
-
----
-
-## Core Interfaces
-
-Each plugin is registered via the `PluginRegistry` and looked up by key.
-
-```ts
-registry.register('loader', 'pdf', new PDFLoader())
-const loader = registry.get('loader', 'pdf');
-```
-
-All plugins implement interface contracts like:
-
-```ts
-interface Loader {
-  load(path: string): Promise<{ chunk(): string[] }[]>;
-}
-
-interface Embedder {
-  embed(chunks: string[]): Vector[];
-  embedQuery(prompt: string): Vector;
-}
-
-interface Retriever {
-  store(vectors: Vector[]): Promise<void>;
-  retrieve(query: Vector): Promise<Context[]>;
-}
-```
-
----
-
-## DAG Support
-
-The `dag-engine.js` module supports chaining multiple components:
-- Example: Summarize → Retrieve → Rerank → LLM
-- Enables more complex workflows than linear ingestion/query
-
----
-
-Next → [Evaluation](./Evaluation.md)

package/docs-site/docs/CLI.md

@@ -1,69 +0,0 @@
-# CLI Reference
-
-The CLI gives you interactive access to the RAG pipeline tools via terminal.
-
-Install globally (optional):
-```bash
-npm install -g @yourorg/rag-pipeline-utils
-```
-
-Or run directly from your project:
-```bash
-npx rag-pipeline <command>
-```
-
----
-
-## Commands
-
-### `ingest`
-```bash
-rag-pipeline ingest ./docs/sample.pdf --loader pdf
-```
-- Loads and indexes documents into vector store
-
-### `query`
-```bash
-rag-pipeline query "Explain vector databases" --llm openai-gpt-4
-```
-- Retrieves context and generates response
-
-### `evaluate`
-```bash
-rag-pipeline evaluate ./fixtures/eval.json
-```
-- Runs batch evaluation with BLEU/ROUGE scoring
-
-### `rerank`
-```bash
-rag-pipeline rerank "What is RAG?" --retriever pinecone
-```
-- Reorders retrieved context using an LLM reranker
-
----
-
-## `.ragrc.json` Config
-
-Defaults can be set in a file:
-```json
-{
-  "loader": "directory",
-  "embedder": "openai",
-  "retriever": "pinecone",
-  "llm": "openai-gpt-4",
-  "useReranker": true
-}
-```
-
----
-
-##  Docker Usage
-
-```bash
-docker build -t rag-pipeline .
-docker run --rm -v $PWD:/data rag-pipeline query "What is retrieval?"
-```
-
----
-
-Next → [Architecture](./Architecture.md)

package/docs-site/docs/Evaluation.md

@@ -1,61 +0,0 @@
-# Evaluation
-
-This module allows you to evaluate the quality of generated LLM responses in a RAG pipeline.
-
----
-
-## Objective
-
-Quantify how well the pipeline performs on question-answer pairs using:
-- **BLEU**: Precision on matching tokens
-- **ROUGE**: Recall + overlap on longest subsequences
-- **LLM Feedback** *(planned)*: Score generation based on LLM ratings
-
----
-
-## Dataset Format
-
-Each entry in your JSON dataset should include:
-
-```json
-{
-  "prompt": "What is a vector index?",
-  "groundTruth": "A data structure for fast similarity search in vector spaces."
-}
-```
-
-Saved as: `sample-eval-dataset.json`
-
----
-
-## CLI Evaluation
-
-```bash
-rag-pipeline evaluate ./fixtures/sample-eval-dataset.json
-```
-
-Outputs:
-- Pass/fail per item
-- BLEU/ROUGE scores
-- Aggregated metrics
-
----
-
-## Dashboard Visualization
-
-Use the React dashboard under `/public`:
-
-```bash
-node server.js
-```
-
-Visit: [http://localhost:3000](http://localhost:3000)
-
-Displays:
-- Prompt vs Answer
-- Success criteria
-- Metric charts (BLEU, ROUGE)
-
----
-
-Next → [Plugins](./Plugins.md)

package/docs-site/docs/FAQ.md

@@ -1,50 +0,0 @@
-# FAQ
-
-### Can I use this with my own vector database?
-Yes! You can implement the `Retriever` interface and register it:
-```ts
-registry.register('retriever', 'mydb', new MyRetriever());
-```
-Then use it via CLI or `.ragrc.json`.
-
----
-
-### Is streaming LLM response supported?
-Yes. LLM runners can yield async output tokens. This is handled internally via `AsyncIterable` in compatible models.
-
----
-
-### How do I evaluate performance?
-Use `rag-pipeline evaluate dataset.json` or open the dashboard at `http://localhost:3000` to visualize BLEU/ROUGE and pass rates.
-
----
-
-### What file types can I ingest?
-Currently supported:
-- `.pdf` (mocked)
-- `.md`
-- `.html`
-- `.csv`
-- Full directory ingestion
-
----
-
-### Can I rerank results using a local model?
-Yes, as long as your local LLM implements:
-```ts
-llm.generate(prompt, context): Promise<string>
-```
-Use it to power `LLMReranker`.
-
----
-
-### How do I contribute?
-Fork the repo → add your plugin or CLI feature → test → submit PR.
-
----
-
-Still stuck? Open an issue at [GitHub](https://github.com/DevilsDev/rag-pipeline-utils/issues).
-
----
-
-Next → [Back to Introduction](./Introduction.md)

package/docs-site/docs/Introduction.md

@@ -1,58 +0,0 @@
----
-sidebar_position: 1
----
-
-# Introduction
-
-Welcome to **@DevilsDev/rag-pipeline-utils**, a modular Node.js toolkit for building scalable, pluggable Retrieval-Augmented Generation (RAG) systems.
-
----
-
-##  What is RAG?
-
-RAG combines information retrieval with large language models to enhance generation with factual grounding. It typically involves:
-
-1. **Retrieving** relevant context (documents, chunks) from a vector store
-2. **Generating** answers using an LLM, augmented by that context
-
-This project provides all the composable parts to build your own RAG pipeline.
-
----
-
-##  Why Use This Toolkit?
-
-- **Plugin-based**: Swap in your own loaders, retrievers, LLMs
-- **Streaming-ready**: Async-friendly, token-by-token output
-- **CLI & API**: Use from terminal or integrate programmatically
-- **Evaluation built-in**: BLEU, ROUGE, dashboard UI
-- **Enterprise-quality**: SOLID, tested, typed, and CI-ready
-
----
-
-## Architecture
-
-Each RAG pipeline includes:
-
-- `loader`: Document loader (e.g., PDF, Markdown, HTML)
-- `embedder`: Embedding model interface (OpenAI, Cohere, local)
-- `retriever`: Vector store adapter (e.g., Pinecone, Chroma)
-- `llm`: Language model runner (OpenAI, Ollama, GPT-4-V)
-- `reranker`: (Optional) Context re-ranking module via LLM
-
-The pipeline can be used via code or via CLI commands.
-
----
-
-## Status
-
-This project is:
-- Production-ready
-- Covered by unit/integration tests
-- Compatible with Node.js `>=18`
-- Published to npm under `DevilsDev`
-
----
-
-Let’s get started!
-
-Next → [Usage](./Usage.md)

package/docs-site/docs/Plugins.md

@@ -1,61 +0,0 @@
-# Plugins
-
-The toolkit is fully modular. You can inject your own loaders, retrievers, embedders, or LLMs.
-
----
-
-## Plugin Architecture
-
-All plugins follow a strict interface pattern. Each is registered under a namespace using the plugin registry:
-
-```ts
-registry.register('loader', 'my-custom', new MyLoader());
-const loader = registry.get('loader', 'my-custom');
-```
-
-You can register plugins for:
-- `loader`
-- `embedder`
-- `retriever`
-- `llm`
-
----
-
-## Loader Example
-
-```ts
-class MyMarkdownLoader {
-  async load(filePath) {
-    const raw = fs.readFileSync(filePath, 'utf-8');
-    return [{ chunk: () => raw.split('\n\n') }];
-  }
-}
-```
-
-Register it:
-```ts
-registry.register('loader', 'md', new MyMarkdownLoader());
-```
-
----
-
-## Runtime Override
-
-CLI and API will resolve plugin names from `pluginRegistry`. Use:
-
-```json
-{
-  "loader": "md",
-  "embedder": "cohere",
-  "retriever": "chroma",
-  "llm": "openai-gpt-4"
-}
-```
-
-in `.ragrc.json`
-
----
-
-This design keeps the core minimal and lets you evolve.
-
-Next → [FAQ](./FAQ.md)

package/docs-site/docs/Tutorials.md

@@ -1,106 +0,0 @@
----
-id: Tutorials
-title: Tutorials & Onboarding
-sidebar_label: Tutorials
----
-
-Welcome to the **RAG Pipeline Utils** tutorial section. These guides will help developers quickly onboard, integrate, and extend the modular RAG system.
-
----
-
-##  1. Install the Package
-
-```bash
-npm install @devilsdev/rag-pipeline-utils
-```
-
----
-
-## 2. Ingest Documents Using CLI
-
-This CLI command loads PDFs, embeds them, and stores the vectors.
-
-```bash
-npx rag-utils ingest ./sample.pdf --config .ragrc.json
-```
-
-Ensure you have a valid `.ragrc.json` config:
-
-```json
-{
-  "loader": { "pdf": "./src/mocks/pdf-loader.js" },
-  "embedder": { "openai": "./src/mocks/openai-embedder.js" },
-  "retriever": { "pinecone": "./src/mocks/pinecone-retriever.js" },
-  "llm": { "openai": "./src/mocks/openai-llm.js" },
-  "namespace": "docs-example",
-  "pipeline": ["loader", "embedder", "retriever"]
-}
-```
-
----
-
-## 3. Run a Query Using CLI
-
-```bash
-npx rag-utils query "What is this document about?" --config .ragrc.json
-```
-
----
-
-## 4. Register a Custom Plugin
-
-You can create your own loader/retriever/LLM:
-
-```js
-// src/plugins/my-custom-llm.js
-export default class MyCustomLLM {
-  ask(prompt) {
-    return `Custom response to: ${prompt}`;
-  }
-}
-```
-
-Then reference it in `.ragrc.json`:
-
-```json
-"llm": {
-  "custom": "./src/plugins/my-custom-llm.js"
-}
-```
-
----
-
-## 5. Validate Plugin Mocks
-
-Ensure all mocks meet contract interfaces before commit:
-
-```bash
-npm run validate-fixtures
-```
-
----
-
-## 6. Run All Tests + Coverage
-
-```bash
-npm run ci
-```
-
-This runs:
-
--  Linting
--  Fixture validation
--  Unit/integration tests
--  Coverage reporting
-
----
-
-## Next Steps
-
-- Explore `/src/core/plugin-registry.js` for plugin orchestration
-- Browse `/scripts/` for CI and release utilities
-- View `/__tests__/` for end-to-end RAG validation flows
-
----
-
-Looking to build on top of this? Check out [Use Cases](./Use-Cases.md) for advanced scenarios.

package/docs-site/docs/Usage.md

@@ -1,72 +0,0 @@
-# Usage
-
-This guide covers how to use `@yourorg/rag-pipeline-utils` programmatically via API and through the CLI.
-
----
-
-## Installation
-
-```bash
-npm install @yourorg/rag-pipeline-utils
-```
-
----
-
-## Programmatic API
-
-Import and instantiate a RAG pipeline:
-
-```js
-import { createRagPipeline } from '@yourorg/rag-pipeline-utils';
-
-const pipeline = createRagPipeline({
-  loader: 'markdown',
-  embedder: 'openai',
-  retriever: 'pinecone',
-  llm: 'openai-gpt-4',
-  useReranker: true
-});
-
-const answer = await pipeline.query("What is retrieval-augmented generation?");
-console.log(answer);
-```
-
----
-
-##  Pipeline Methods
-
-- `pipeline.ingest(path: string)` → Load, chunk, embed, and store vectors
-- `pipeline.query(prompt: string)` → Retrieve context, rerank (optional), and call LLM
-
----
-
-##  Plugin Configuration
-
-You can customize the pipeline by registering plugins:
-
-```js
-registry.register('loader', 'custom', new MyCustomLoader());
-registry.register('retriever', 'opensearch', new MyOpenSearchRetriever());
-```
-
----
-
-## Configuration via `.ragrc.json`
-
-Create a JSON file at your root:
-
-```json
-{
-  "loader": "directory",
-  "embedder": "openai",
-  "retriever": "pinecone",
-  "llm": "openai-gpt-4",
-  "useReranker": true
-}
-```
-
-Used automatically when no CLI args are passed.
-
----
-
-Next → [CLI](./CLI.md)