@devilsdev/rag-pipeline-utils 2.2.0 → 2.3.0

package/README.md

@@ -1,918 +1,519 @@
-# @devilsdev/rag-pipeline-utils
+# rag-pipeline-utils
+
+**Modular toolkit for building production-ready RAG pipelines in Node.js**
 
 [![CI](https://github.com/DevilsDev/rag-pipeline-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/DevilsDev/rag-pipeline-utils/actions)
-[![npm version](https://badge.fury.io/js/%40devilsdev%2Frag-pipeline-utils.svg)](https://badge.fury.io/js/%40devilsdev%2Frag-pipeline-utils)
+[![npm version](https://badge.fury.io/js/%40devilsdev%2Frag-pipeline-utils.svg)](https://www.npmjs.com/package/@devilsdev/rag-pipeline-utils)
+[![codecov](https://codecov.io/gh/DevilsDev/rag-pipeline-utils/branch/main/graph/badge.svg)](https://codecov.io/gh/DevilsDev/rag-pipeline-utils)
+[![License: GPL-3.0](https://img.shields.io/badge/License-GPL%203.0-blue.svg)](https://opensource.org/licenses/GPL-3.0)
 [![Node.js Version](https://img.shields.io/node/v/@devilsdev/rag-pipeline-utils.svg)](https://nodejs.org/)
-[![License](https://img.shields.io/npm/l/@devilsdev/rag-pipeline-utils.svg)](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/LICENSE)
 [![Downloads](https://img.shields.io/npm/dm/@devilsdev/rag-pipeline-utils.svg)](https://www.npmjs.com/package/@devilsdev/rag-pipeline-utils)
-
-Enterprise-grade toolkit for building Retrieval-Augmented Generation (RAG) pipelines in Node.js with modular plugin architecture, streaming support, and observability.
-
-## 🎉 What's New in v2.2.0
-
-### Major Improvements
-
-**✅ Fixed ESM Build (Critical)**
-
-- Implemented CJS→ESM interop pattern for proper module compatibility
-- All 15 exports now working correctly in both CommonJS and ES Modules
-- Verified compatibility with Node.js 18.x, 20.x, and 22.x
-
-**📦 50% Smaller Install Size**
-
-- Reduced from ~30MB to ~15MB by optimizing dependencies
-- Moved CLI-only tools to devDependencies
-- Zero production vulnerabilities (npm audit clean)
-
-**📝 Enhanced TypeScript Support**
-
-- Added 5 complete plugin contract interfaces:
-  - `LoaderPlugin` - Document loading interface
-  - `EmbedderPlugin` - Text embedding interface
-  - `RetrieverPlugin` - Vector search interface
-  - `LLMPlugin` - Language model interface
-  - `RerankerPlugin` - Result reranking interface
-- Added 4 helper types: `Document`, `SearchResult`, `RetrieveOptions`, `LLMResponse`
-- 236 lines of comprehensive type definitions
-- Full IDE autocomplete and type checking support
-
-**🔧 Real-World Example**
-
-- Complete OpenAI + Pinecone integration example in `examples/openai-pinecone/`
-- Mock mode for testing without API keys
-- Production-ready error handling and cost estimation guidance
-- See [OpenAI + Pinecone Example](#openai--pinecone-example) below
-
-**📚 Documentation Enhancements**
-
-- Platform compatibility matrix (Linux, macOS, Windows, Docker)
-- Module system examples (CommonJS + ES Modules)
-- Docker usage examples
-- Troubleshooting guides
-
-### Migration from v2.1.x
-
-**No breaking changes!** v2.2.0 is fully backward compatible. Simply update:
-
-```bash
-npm install @devilsdev/rag-pipeline-utils@latest
-```
-
-All existing code will continue to work without modifications.
+[![GitHub Discussions](https://img.shields.io/github/discussions/DevilsDev/rag-pipeline-utils)](https://github.com/DevilsDev/rag-pipeline-utils/discussions)
 
 ---
 
-## Installation
+## Why This Exists
 
-```bash
-npm install @devilsdev/rag-pipeline-utils
-```
+Building Retrieval-Augmented Generation (RAG) systems is harder than it should be. Most solutions lock you into specific vendors, force opinionated architectures, or sacrifice observability for simplicity.
 
-**Requirements:** Node.js >= 18.0.0
+**rag-pipeline-utils** takes a different approach: it provides the building blocks—loaders, embedders, retrievers, LLM connectors, and rerankers—that you compose into pipelines that fit your needs. Every component follows clear contracts. Every integration is optional. Every decision is yours.
 
-## Compatibility
+We built this toolkit because we believe infrastructure should be:
 
-### Platform Support
+- **Modular** – swap any component without rewriting your pipeline
+- **Observable** – metrics, tracing, and audit logs built in from day one
+- **Secure** – hardened JWT validation with replay protection, multi-layer path traversal defense, and enterprise-grade defaults
+- **Open** – no vendor lock-in, no proprietary APIs, just JavaScript
 
-| Platform    | Status             | Notes                                                         |
-| ----------- | ------------------ | ------------------------------------------------------------- |
-| **Linux**   | ✅ Fully Supported | Tested on Ubuntu 20.04+, Debian 11+, RHEL 8+                  |
-| **macOS**   | ✅ Fully Supported | Tested on macOS 12+ (Intel & Apple Silicon)                   |
-| **Windows** | ✅ Fully Supported | Tested on Windows 10/11, WSL2 recommended for best experience |
-| **Docker**  | ✅ Fully Supported | Alpine & Debian-based images available                        |
+Whether you're prototyping a document Q&A system or scaling a production knowledge base, this toolkit grows with you.
 
-### Module Systems
+---
 
-This package supports both CommonJS and ES Modules:
+## Quick Start
 
-**CommonJS (require)**
+Here's a complete RAG pipeline in 15 lines:
 
 ```javascript
-const {
-  createRagPipeline,
-  DAGEngine,
-} = require("@devilsdev/rag-pipeline-utils");
+const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils");
 
+// Implement your plugins (or use existing ones)
 const pipeline = createRagPipeline({
-  /* config */
+  loader: new MyPDFLoader(),
+  embedder: new MyOpenAIEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
+  retriever: new MyVectorDBRetriever({ url: "http://localhost:8000" }),
+  llm: new MyLLMConnector({ model: "gpt-4" }),
 });
-const dag = new DAGEngine();
+
+// Ingest documents
+await pipeline.ingest("./company-docs.pdf");
+
+// Query with natural language
+const answer = await pipeline.query("What is the vacation policy?");
+console.log(answer.text);
+// Output: "Based on the company handbook, employees receive 20 days of vacation per year..."
 ```
 
-**ES Modules (import)**
+**Requirements:** Node.js >= 18.0.0
 
-```javascript
-import { createRagPipeline, DAGEngine } from "@devilsdev/rag-pipeline-utils";
+Install with:
 
-const pipeline = createRagPipeline({
-  /* config */
-});
-const dag = new DAGEngine();
+```bash
+npm install @devilsdev/rag-pipeline-utils
 ```
 
-### TypeScript Support
-
-**New in v2.2.0:** Complete plugin contract interfaces with 236 lines of type definitions:
-
-```typescript
-import {
-  createRagPipeline,
-  DAGEngine,
-  RagPipelineConfig,
-  // Plugin contract interfaces (NEW in v2.2.0)
-  LoaderPlugin,
-  EmbedderPlugin,
-  RetrieverPlugin,
-  LLMPlugin,
-  RerankerPlugin,
-  // Helper types (NEW in v2.2.0)
-  Document,
-  SearchResult,
-  RetrieveOptions,
-  LLMResponse,
-} from "@devilsdev/rag-pipeline-utils";
-
-// Type-safe plugin implementation
-class MyLoader implements LoaderPlugin {
-  async load(source: string, options?: any): Promise<Document[]> {
-    // Your implementation with full type checking
-    return [
-      {
-        id: "doc-1",
-        content: "Document content",
-        metadata: { source },
-      },
-    ];
-  }
-}
+### What's New in v2.3.0 (Released November 7, 2025)
 
-const config: RagPipelineConfig = {
-  loader: new MyLoader(),
-  embedder: "my-embedder",
-};
+**Major Security Enhancements:**
 
-const pipeline = createRagPipeline(config);
-```
+- **Advanced JWT replay protection** with self-signed token reusability
+- **Hardened path traversal defense** with iterative URL decoding
+- **Consistent validation behavior** eliminating duplicate checks
+- **Race condition mitigation** in concurrent token verification
 
-### Package Size & Dependencies
+**All security tests passing:** 113 security-focused tests with 100% coverage on critical paths.
 
-- **Package size**: 244KB (gzipped), 1.1MB (unpacked)
-- **Runtime dependencies**: Only 11 production dependencies
-- **Total install size**: ~15MB (including dependencies) - **50% smaller than v2.1.x**
-- **Tree-shakeable**: ES module exports support tree-shaking
-- **Zero vulnerabilities**: npm audit clean (production dependencies)
+[See full security details below](#security-and-quality) | [Migration guide](#upgrading-from-v22x)
 
-### Docker Usage
+---
 
-**Basic Docker setup:**
+## How It Works
 
-```dockerfile
-FROM node:18-alpine
+**rag-pipeline-utils** orchestrates data through a clear, predictable flow:
 
-WORKDIR /app
-COPY package*.json ./
-RUN npm ci --production
-COPY . .
+```mermaid
+graph LR
+    A[Documents] --> B[Loader]
+    B --> C[Embedder]
+    C --> D[Vector Store]
+    D --> E[Retriever]
+    F[User Query] --> C
+    F --> E
+    E --> G[Reranker]
+    G --> H[LLM]
+    H --> I[Response]
 
-CMD ["node", "index.js"]
+    style B fill:#e1f5e1
+    style C fill:#e1f5e1
+    style E fill:#e1f5e1
+    style G fill:#fff4e1
+    style H fill:#e1f5e1
+    style I fill:#e3f2fd
 ```
 
-**Docker Compose example:**
-
-```yaml
-version: "3.8"
-services:
-  rag-pipeline:
-    image: node:18-alpine
-    working_dir: /app
-    volumes:
-      - ./:/app
-    environment:
-      - NODE_ENV=production
-    command: npm start
-```
+1. **Loader** – Parse documents (PDF, HTML, Markdown, etc.)
+2. **Embedder** – Convert text to vector embeddings
+3. **Vector Store** – Index and persist embeddings
+4. **Retriever** – Find semantically similar content
+5. **Reranker** (optional) – Refine retrieval results
+6. **LLM** – Generate contextual responses
 
-## Quick Start
+Every component is replaceable. Every step is observable.
 
-### Basic Pipeline Setup
+---
 
-```javascript
-const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils");
+## Features
 
-// Create pipeline with your plugin implementations
-const pipeline = createRagPipeline({
-  loader: yourLoaderPlugin,
-  embedder: yourEmbedderPlugin,
-  retriever: yourRetrieverPlugin,
-  llm: yourLLMPlugin,
-});
+### Modular Plugin Architecture
 
-// Use the pipeline
-await pipeline.ingest("./document.pdf");
-const result = await pipeline.query("What is this document about?");
-console.log(result);
-```
+Swap any component without rewriting your pipeline. Use OpenAI embeddings with Pinecone storage, or mix Hugging Face models with PostgreSQL—your choice.
 
-### Using Configuration File
-
-Create `.ragrc.json`:
-
-```json
-{
-  "loader": {
-    "pdf": "your-pdf-loader-plugin"
-  },
-  "embedder": {
-    "default": "your-embedder-plugin"
-  },
-  "retriever": {
-    "default": "your-vector-db-plugin"
-  },
-  "llm": {
-    "default": "your-llm-plugin"
-  }
-}
-```
+### Enterprise Observability
 
-Load and use configuration:
+Built-in structured logging, Prometheus metrics, OpenTelemetry tracing, and audit logs for compliance-grade transparency.
 
-```javascript
-const {
-  loadConfig,
-  createRagPipeline,
-} = require("@devilsdev/rag-pipeline-utils");
+### DAG Workflow Engine
 
-const config = await loadConfig(".ragrc.json");
-const pipeline = createRagPipeline(config);
-```
+Chain complex multi-step operations with retry logic, parallel execution, and graceful error handling. Perfect for batch ingestion or multi-stage transformations.
 
-## Core API
+### Type-Safe Contracts
 
-### Exported Modules
+Complete TypeScript interfaces for all plugin types. Full IDE autocomplete and compile-time validation.
 
-```javascript
-const {
-  // Core pipeline
-  createRagPipeline,
+### Production-Ready Security
 
-  // Configuration
-  loadConfig,
-  validateRagrc,
-  normalizeConfig,
+Hardened security with defense-in-depth: JWT replay protection with race condition mitigation, multi-layer path traversal defense with iterative URL decoding, validated JSON parsing, algorithm whitelisting, automatic secret redaction, and comprehensive audit logging.
 
-  // Plugin system
-  pluginRegistry,
+### Streaming & Batching
 
-  // DAG workflow engine
-  DAGEngine,
+Stream LLM responses in real-time. Batch embed thousands of documents efficiently with parallel processing and backpressure control.
 
-  // Observability
-  eventLogger,
-  metrics,
+### Framework Agnostic
 
-  // Performance
-  ParallelProcessor,
+Works with Express, Fastify, Next.js, or any Node.js runtime. Deploy to AWS Lambda, Vercel, Docker, or bare metal.
 
-  // AI/ML
-  MultiModalProcessor,
-  AdaptiveRetrievalEngine,
+---
 
-  // Enterprise
-  AuditLogger,
-  DataGovernance,
+## Join the Community
 
-  // Utilities
-  logger,
-} = require("@devilsdev/rag-pipeline-utils");
-```
+We're building this together.
 
-### createRagPipeline(config)
+Whether you're implementing a new retriever, writing documentation, reporting bugs, or sharing ideas—every contribution makes this toolkit better for everyone.
 
-Creates a RAG pipeline instance.
+**Getting Started:**
 
-**Parameters:**
+- [GitHub Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions) – Ask questions, share use cases, propose features
+- [Issue Tracker](https://github.com/DevilsDev/rag-pipeline-utils/issues) – Report bugs or request features
+- [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md) – Learn how to submit pull requests
+- [Code of Conduct](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/CODE_OF_CONDUCT.md) – Our community standards
 
-- `config.loader` - Document loader plugin
-- `config.embedder` - Text embedding plugin
-- `config.retriever` - Vector storage and retrieval plugin
-- `config.llm` - Language model plugin
-- `config.reranker` (optional) - Result reranking plugin
+**Good First Issues:** Look for issues tagged `good first issue` to find beginner-friendly contributions.
 
-**Returns:** Pipeline instance with `ingest()` and `query()` methods
+**Office Hours:** Join our monthly community calls to discuss roadmap priorities and architectural decisions.
 
-### Configuration Management
+---
 
-```javascript
-// Load configuration from file
-const config = await loadConfig(".ragrc.json");
+## Security and Quality
 
-// Validate configuration
-const isValid = validateRagrc(config);
+Security isn't optional. This toolkit is built with defense in depth:
 
-// Normalize configuration
-const normalized = normalizeConfig(config);
-```
+### Core Security Features
 
-### Plugin Registry
+- **Validated Inputs** – All JSON parsed through schema validation (AJV)
+- **Secure Logging** – Automatic redaction of 36 sensitive field patterns (API keys, JWTs, tokens)
+- **JWT Best Practices** – Algorithm whitelisting, issuer/audience validation, expiration enforcement
+- **Dependency Scanning** – Weekly Dependabot updates, CI blocks on critical vulnerabilities
+- **Audit Trail** – Immutable compliance-grade logs for all security-sensitive operations
+- **Plugin Sandboxing** – Third-party plugins run with declared permissions and resource limits
+- **Supply Chain Security** – All GitHub Actions pinned to commit SHAs, SBOM generation available
 
-```javascript
-const { pluginRegistry } = require("@devilsdev/rag-pipeline-utils");
+### Recent Security Enhancements (v2.3.0)
 
-// Register a plugin
-pluginRegistry.register("my-plugin", MyPluginClass);
+#### Hardened JWT Validation
 
-// Get a plugin
-const Plugin = pluginRegistry.get("my-plugin");
+**Advanced Replay Protection:**
 
-// List all plugins
-const plugins = pluginRegistry.list();
+- Self-signed tokens can be verified multiple times (essential for refresh flows and load balancer retries)
+- External tokens are tracked and blocked on replay attempts
+- Race condition mitigation with optimized check-then-set pattern
+- Separate tracking for reusable vs. single-use tokens
 
-// Check if plugin exists
-const exists = pluginRegistry.has("my-plugin");
-```
+**Consistent Validation Behavior:**
 
-### DAG Workflow Engine
+- `strictValidation` flag now consistently controls issuer/audience validation
+- Eliminated duplicate validation logic to prevent configuration confusion
+- Clear separation of concerns: jsonwebtoken handles cryptographic validation, custom logic handles business rules
 
-Build complex multi-step workflows:
+**Example:**
 
 ```javascript
-const { DAGEngine } = require("@devilsdev/rag-pipeline-utils");
-
-const dag = new DAGEngine({
-  nodes: [
-    {
-      id: "load",
-      type: "loader",
-      config: {
-        /* loader config */
-      },
-    },
-    {
-      id: "embed",
-      type: "embedder",
-      dependencies: ["load"],
-      config: {
-        /* embedder config */
-      },
-    },
-    {
-      id: "store",
-      type: "retriever",
-      dependencies: ["embed"],
-      config: {
-        /* retriever config */
-      },
-    },
-  ],
+const { JWTValidator } = require("@devilsdev/rag-pipeline-utils");
+
+const validator = new JWTValidator({
+  secret: process.env.JWT_SECRET,
+  algorithm: "HS256",
+  issuer: "my-app",
+  audience: "api-users",
+  strictValidation: true, // Enforces iss/aud validation
+  enableJtiTracking: true, // Prevents replay attacks
 });
 
-await dag.execute();
+// Self-signed tokens can be verified multiple times
+const token = validator.sign({ sub: "user-123" });
+validator.verify(token); // First verification
+validator.verify(token); // Still works (refresh flow)
+
+// External tokens are blocked on replay
+const externalToken = getTokenFromThirdParty();
+validator.verify(externalToken); // First use
+validator.verify(externalToken); // Throws "Token replay detected"
 ```
 
-### Observability
+#### Path Traversal Defense
 
-```javascript
-const { eventLogger, metrics } = require("@devilsdev/rag-pipeline-utils");
+**Multi-Layer Protection:**
 
-// Configure event logging
-eventLogger.configure({
-  level: "info",
-  format: "json",
-});
+- Iterative URL decoding (up to 5 passes) to catch sophisticated encoding attacks
+- Detects double-encoded paths: `%252e%252e%252f` → `%2e%2e%2f` → `../`
+- Malformed encoding treated as attack indicator (throws error)
+- Critical security violations always throw, regardless of configuration
 
-// Log events
-eventLogger.log("info", "Pipeline started", { pipelineId: "abc123" });
+**Attack Vectors Blocked:**
 
-// Access metrics
-const pipelineMetrics = metrics.getMetrics();
-console.log(pipelineMetrics);
-```
+- Standard traversal: `../../../etc/passwd`
+- Windows paths: `..\..\windows\system32`
+- URL encoded: `%2e%2e%2f`, `%2e%2e%5c`
+- Double encoded: `%252e%252e%252f`
+- Mixed encoding: combinations of above
 
-### Parallel Processing
+**Example:**
 
 ```javascript
-const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils");
+const { sanitizePath } = require("@devilsdev/rag-pipeline-utils");
 
-const processor = new ParallelProcessor({
-  maxConcurrency: 5,
-});
+// Safe paths are normalized
+sanitizePath("docs/README.md"); // Returns: "docs/README.md"
 
-await processor.process(items, async (item) => {
-  // Process each item
-  return processItem(item);
-});
+// Dangerous paths throw errors
+sanitizePath("../../../etc/passwd"); // Throws
+sanitizePath("%2e%2e%2f%2e%2e%2fpasswd"); // Throws
+sanitizePath("%252e%252e%252fconfig"); // Throws (double-encoded)
 ```
 
-## Plugin Development
+#### Defense-in-Depth Architecture
 
-This package provides a plugin-based architecture. You need to implement plugins for your specific use case.
+**Critical Security Errors:**
 
-### Plugin Contracts
+- Path traversal violations **always throw**, even if `throwOnInvalid=false`
+- Object depth limit violations **always throw** to prevent DoS
+- Security-critical operations are prioritized over configuration flexibility
 
-**New in v2.2.0:** TypeScript interfaces for all plugin contracts. See [TypeScript Support](#typescript-support) for full type definitions.
+**Security Monitoring:**
 
-#### Loader Plugin
+- All blocked attempts tracked in `stats.blocked` counter
+- Audit events emitted for replay detection, algorithm mismatches, and validation failures
+- Structured logging with security event correlation
 
-```javascript
-/**
- * Loader plugin interface (NEW TypeScript interface in v2.2.0)
- * Implements: LoaderPlugin
- */
-class MyLoader {
-  async load(source, options) {
-    // Load and parse document
-    // Return array of Document objects
-    return [
-      {
-        id: "doc-1",
-        content: "Document text...",
-        metadata: { page: 1, source },
-        // embedding: optional pre-computed embedding
-      },
-    ];
-  }
-}
-```
+### Quality Metrics
 
-#### Embedder Plugin
+**Zero Production Vulnerabilities:** `npm audit --production` returns clean on every release.
 
-```javascript
-/**
- * Embedder plugin interface (NEW TypeScript interface in v2.2.0)
- * Implements: EmbedderPlugin
- */
-class MyEmbedder {
-  async embed(text, options) {
-    // Generate embedding for a single text string
-    // Return: Promise<number[]> - embedding vector
-    return [0.1, 0.2, 0.3 /* ... 1536 dimensions for OpenAI */];
-  }
+**Test Coverage:** 113 security-focused tests across 2 dedicated security suites:
 
-  // Optional batch embedding method
-  async embedBatch(texts) {
-    // Generate embeddings for multiple texts
-    return texts.map((text) => this.embed(text));
-  }
-}
-```
+- JWT Validator: 44 tests covering algorithm confusion, replay attacks, and validation edge cases
+- Input Sanitizer: 69 tests covering XSS, SQL injection, command injection, and path traversal
 
-#### Retriever Plugin
+**Supported Node Versions:** 18.x, 20.x, 22.x (tested in CI)
 
-```javascript
-/**
- * Retriever plugin interface (NEW TypeScript interface in v2.2.0)
- * Implements: RetrieverPlugin
- */
-class MyRetriever {
-  async retrieve(query, options) {
-    // query can be:
-    // - string: search by text
-    // - { embedding: number[] }: search by vector
-    const { topK = 5, minScore = 0.7, filter } = options || {};
-
-    // Search for similar vectors
-    // Return: Promise<SearchResult[]>
-    return [
-      {
-        id: "result-1",
-        score: 0.95,
-        document: {
-          id: "doc-1",
-          content: "Document text...",
-          metadata: { source: "file.pdf" },
-        },
-        metadata: {
-          /* additional search metadata */
-        },
-      },
-    ];
-  }
-}
-```
-
-#### LLM Plugin
-
-```javascript
-/**
- * LLM plugin interface (NEW TypeScript interface in v2.2.0)
- * Implements: LLMPlugin
- */
-class MyLLM {
-  async generate(prompt, options) {
-    // Generate text from prompt
-    // Return: Promise<LLMResponse>
-    return {
-      text: "Generated response...",
-      usage: {
-        promptTokens: 100,
-        completionTokens: 50,
-        totalTokens: 150,
-      },
-      metadata: {
-        model: "gpt-4",
-        finishReason: "stop",
-      },
-    };
-  }
+---
 
-  // Optional streaming support
-  async *stream(prompt, options) {
-    // Stream response chunks
-    yield "Generated ";
-    yield "response ";
-    yield "chunks...";
-  }
-}
-```
+## Installation & Development
 
-#### Reranker Plugin (Optional)
+### Install
 
-```javascript
-/**
- * Reranker plugin interface (NEW TypeScript interface in v2.2.0)
- * Implements: RerankerPlugin
- */
-class MyReranker {
-  async rerank(results, query, options) {
-    // Rerank search results based on relevance to query
-    // Return: Promise<SearchResult[]> - reranked results
-    return results
-      .map((result) => ({
-        ...result,
-        score: calculateRelevanceScore(query, result),
-      }))
-      .sort((a, b) => b.score - a.score);
-  }
-}
+```bash
+npm install @devilsdev/rag-pipeline-utils
 ```
 
-## Environment Configuration
+### Development Setup
 
 ```bash
-# Example environment variables for your plugins
-OPENAI_API_KEY=your_api_key
-PINECONE_API_KEY=your_api_key
-PINECONE_ENVIRONMENT=us-east-1
-CHROMA_URL=http://localhost:8000
-
-# Performance settings
-RAG_MAX_CONCURRENCY=5
-RAG_BATCH_SIZE=10
-```
+# Clone the repository
+git clone https://github.com/DevilsDev/rag-pipeline-utils.git
+cd rag-pipeline-utils
 
-## Advanced Features
+# Install dependencies
+npm install
 
-### Multi-Modal Processing
+# Run tests
+npm test
 
-```javascript
-const { MultiModalProcessor } = require("@devilsdev/rag-pipeline-utils");
+# Run linter
+npm run lint
 
-const processor = new MultiModalProcessor({
-  // Configuration for processing text, images, audio, etc.
-});
+# Build distribution
+npm run build
 
-await processor.process(multiModalData);
+# Run development tools
+npm run dev
 ```
 
-### Adaptive Retrieval
-
-```javascript
-const { AdaptiveRetrievalEngine } = require("@devilsdev/rag-pipeline-utils");
-
-const retriever = new AdaptiveRetrievalEngine({
-  // Configuration for adaptive retrieval strategies
-});
+### Common Commands
 
-const results = await retriever.retrieve(query, context);
+```bash
+npm test                # Run test suite
+npm run test:watch      # Watch mode for TDD
+npm run lint            # Check code style
+npm run lint:fix        # Auto-fix linting issues
+npm run security:audit  # Scan dependencies
+npm run docs:api        # Generate API documentation
+npm run benchmark       # Performance benchmarks
 ```
 
-### Enterprise Audit Logging
+---
 
-```javascript
-const { AuditLogger } = require("@devilsdev/rag-pipeline-utils");
+## Roadmap
 
-const auditLogger = new AuditLogger({
-  destination: "./audit-logs",
-});
+### v2.3.0 (Q2 2025)
 
-auditLogger.log("query", {
-  user: "user@example.com",
-  query: "sensitive query",
-  timestamp: new Date(),
-});
-```
+- **Enhanced CLI** – Interactive wizard for pipeline setup, validation commands, and migration tools
+- **Plugin Marketplace** – Curated registry of community-contributed plugins with ratings and verification
+- **Streaming Embeddings** – Real-time embedding generation with backpressure control
+- **Multi-Tenant Support** – Workspace isolation, resource quotas, and tenant-specific configurations
 
-### Data Governance
+### v2.4.0 (Q3 2025)
 
-```javascript
-const { DataGovernance } = require("@devilsdev/rag-pipeline-utils");
-
-const governance = new DataGovernance({
-  policies: {
-    retention: "90d",
-    encryption: true,
-  },
-});
-
-await governance.enforce(data);
-```
+- **GraphRAG Support** – Native knowledge graph integration for entity-relationship retrieval
+- **Federated Learning** – Privacy-preserving model training across distributed datasets
+- **Advanced Reranking** – Cross-encoder models and learned ranking functions
+- **Performance Dashboard** – Web UI for monitoring pipeline health and cost metrics
 
-## CLI Usage
+### Beyond
 
-The package includes a CLI tool for pipeline operations:
+- Native Rust bindings for performance-critical paths
+- Kubernetes operator for production deployments
+- Real-time collaboration features for team-based pipelines
 
-```bash
-# Initialize configuration
-npx rag-pipeline init
+**Vote on features:** Share your priorities in [GitHub Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions/categories/roadmap).
 
-# Run pipeline operations (requires plugin implementation)
-npx rag-pipeline ingest ./documents
-npx rag-pipeline query "Your question"
+---
 
-# Run system diagnostics
-npx rag-pipeline doctor
-```
+## Upgrading from v2.2.x
 
-## Error Handling
+The v2.3.0 release includes important security enhancements that are **100% backward compatible** for most use cases. However, there are a few behavioral changes to be aware of:
 
-```javascript
-try {
-  const result = await pipeline.query("What is AI?");
-} catch (error) {
-  if (error.code === "PLUGIN_NOT_FOUND") {
-    console.error("Required plugin not found:", error.message);
-  } else if (error.code === "INVALID_CONFIG") {
-    console.error("Invalid configuration:", error.message);
-  } else {
-    console.error("Pipeline error:", error.message);
-  }
-}
-```
+### JWT Validator Changes
 
-## Use Cases
+#### No Action Required (Recommended Use Case)
 
-### Document Q&A System
+If you're using the default configuration with `strictValidation: true` (the default), everything works the same:
 
 ```javascript
-// Implement your plugins
-const loader = new PDFLoaderPlugin();
-const embedder = new OpenAIEmbedderPlugin({
-  apiKey: process.env.OPENAI_API_KEY,
-});
-const retriever = new ChromaRetrieverPlugin({ url: "http://localhost:8000" });
-const llm = new OpenAILLMPlugin({ model: "gpt-4" });
-
-// Create pipeline
-const pipeline = createRagPipeline({
-  loader,
-  embedder,
-  retriever,
-  llm,
+const validator = new JWTValidator({
+  secret: process.env.JWT_SECRET,
+  algorithm: "HS256",
+  issuer: "my-app",
+  audience: "api-users",
+  // strictValidation defaults to true
 });
 
-// Ingest documents
-await pipeline.ingest("./company-handbook.pdf");
-
-// Query
-const answer = await pipeline.query("What is the vacation policy?");
-console.log(answer);
+// Behavior unchanged - tokens validated correctly
 ```
 
-### Knowledge Base Search
-
-```javascript
-// Use your custom database retriever
-class PostgresRetriever {
-  constructor(connectionString) {
-    this.db = new PostgresClient(connectionString);
-  }
-
-  async retrieve(queryVector, options) {
-    const results = await this.db.query(
-      "SELECT content, metadata, vector <-> $1 as score FROM embeddings ORDER BY score LIMIT $2",
-      [queryVector, options.topK],
-    );
-    return results.rows;
-  }
-}
-
-const pipeline = createRagPipeline({
-  loader: yourLoader,
-  embedder: yourEmbedder,
-  retriever: new PostgresRetriever("postgresql://localhost/kb"),
-  llm: yourLLM,
-});
-```
+#### Behavior Change (Advanced Use Case)
 
-## Performance Tips
+If you explicitly set `strictValidation: false`, the behavior has changed for consistency:
 
-### Batch Processing
+**Before v2.3.0:**
 
 ```javascript
-const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils");
-
-const processor = new ParallelProcessor({
-  maxConcurrency: 5,
-  batchSize: 10,
-});
-
-// Process documents in parallel
-await processor.process(documents, async (doc) => {
-  return await pipeline.ingest(doc);
+const validator = new JWTValidator({
+  secret: process.env.JWT_SECRET,
+  strictValidation: false,
+  issuer: "my-app", // These were still validated
+  audience: "api-users", // even with strictValidation=false
 });
 ```
 
-### Logging and Monitoring
+**After v2.3.0:**
 
 ```javascript
-const { logger, metrics } = require("@devilsdev/rag-pipeline-utils");
-
-// Set log level
-logger.setLevel("debug");
-
-// Monitor performance
-const stats = metrics.getMetrics();
-console.log("Total queries:", stats.queries?.total || 0);
-console.log("Average latency:", stats.queries?.avgLatency || 0);
+const validator = new JWTValidator({
+  secret: process.env.JWT_SECRET,
+  strictValidation: false,
+  issuer: "my-app", // Now properly ignored
+  audience: "api-users", // when strictValidation=false
+});
+// issuer/audience validation is now truly disabled
 ```
 
-## Common Integration Examples
-
-### OpenAI + Pinecone Example
+**Migration:** If you were relying on the old inconsistent behavior, set `strictValidation: true` explicitly.
 
-**New in v2.2.0:** Complete working example with OpenAI embeddings and Pinecone vector database.
+#### Enhanced Behavior (Self-Signed Tokens)
 
-See the full example in [`examples/openai-pinecone/`](examples/openai-pinecone/) with:
+Self-signed tokens can now be verified multiple times, which is essential for:
 
-- ✅ Complete plugin implementations
-- ✅ Mock mode (no API keys required for testing)
-- ✅ Real OpenAI integration
-- ✅ Real Pinecone integration
-- ✅ Error handling and cost estimation
-- ✅ Comprehensive documentation
+- Refresh token flows
+- Load balancer retries
+- Distributed systems with eventual consistency
 
-**Quick start:**
-
-```bash
-cd examples/openai-pinecone
-npm install
-
-# Test with mock mode (no API keys needed)
-USE_MOCK_MODE=true npm start
+```javascript
+const token = validator.sign({ sub: "user-123" });
 
-# Run with real APIs (requires API keys in .env)
-npm start
+// Before v2.3.0: Second verification would throw "replay detected"
+// After v2.3.0: Works correctly for self-signed tokens
+validator.verify(token); // Works
+validator.verify(token); // Now works!
 ```
 
-### With OpenAI
+External tokens (not signed by your validator) are still correctly blocked on replay.
 
-You need to implement OpenAI plugin wrappers:
+### Input Sanitizer Changes
 
-```javascript
-class OpenAIEmbedder {
-  constructor({ apiKey, model = "text-embedding-ada-002" }) {
-    this.client = new OpenAI({ apiKey });
-    this.model = model;
-  }
+#### Path Traversal Now Always Throws
 
-  async embed(texts) {
-    const response = await this.client.embeddings.create({
-      model: this.model,
-      input: texts,
-    });
-    return response.data.map((d) => d.embedding);
-  }
-}
+Path traversal attempts now **always throw errors**, even with `throwOnInvalid: false`:
 
-class OpenAILLM {
-  constructor({ apiKey, model = "gpt-4" }) {
-    this.client = new OpenAI({ apiKey });
-    this.model = model;
-  }
+**Before v2.3.0:**
 
-  async generate(prompt) {
-    const response = await this.client.chat.completions.create({
-      model: this.model,
-      messages: [{ role: "user", content: prompt }],
-    });
-    return { text: response.choices[0].message.content };
-  }
-}
+```javascript
+const sanitizer = new InputSanitizer({ throwOnInvalid: false });
+const result = sanitizePath("../../../etc/passwd");
+// Returned: null (silent failure)
 ```
 
-### With ChromaDB
+**After v2.3.0:**
 
 ```javascript
-class ChromaRetriever {
-  constructor({ url, collection }) {
-    this.client = new ChromaClient({ path: url });
-    this.collectionName = collection;
-  }
+const sanitizer = new InputSanitizer({ throwOnInvalid: false });
+const result = sanitizePath("../../../etc/passwd");
+// Throws: "Potential path traversal detected"
+```
 
-  async store(vectors, metadata) {
-    const collection = await this.client.getOrCreateCollection({
-      name: this.collectionName,
-    });
+**Rationale:** Path traversal is a critical security violation that should never be silently ignored.
 
-    await collection.add({
-      embeddings: vectors,
-      metadatas: metadata,
-      ids: metadata.map((_, i) => `doc_${i}`),
-    });
-  }
+**Migration:** Wrap path sanitization in try-catch if you need to handle errors gracefully:
 
-  async retrieve(queryVector, options) {
-    const collection = await this.client.getCollection({
-      name: this.collectionName,
-    });
-
-    const results = await collection.query({
-      queryEmbeddings: [queryVector],
-      nResults: options.topK || 5,
-    });
-
-    return results.metadatas[0].map((meta, i) => ({
-      content: results.documents[0][i],
-      metadata: meta,
-      score: 1 - results.distances[0][i],
-    }));
+```javascript
+try {
+  const safePath = sanitizePath(userInput);
+  // Use safePath
+} catch (error) {
+  if (error.message.includes("path traversal")) {
+    // Handle attack attempt
+    logger.warn("Path traversal blocked", { input: userInput });
+    return res.status(400).json({ error: "Invalid path" });
   }
+  throw error;
 }
 ```
 
-## Troubleshooting
-
-### Common Issues
+### Breaking Changes
 
-**Error: Cannot find module**
+**None.** All changes are security enhancements that improve correctness without breaking existing valid use cases.
 
-```bash
-npm install @devilsdev/rag-pipeline-utils
-```
+### Testing Your Upgrade
 
-**Error: Plugin not registered**
+Run your test suite after upgrading:
 
-```javascript
-// Make sure to register your plugins
-pluginRegistry.register("my-embedder", MyEmbedderClass);
+```bash
+npm test
 ```
 
-**Error: Invalid configuration**
+If you encounter issues, please [open an issue](https://github.com/DevilsDev/rag-pipeline-utils/issues) with details about your configuration.
 
-```javascript
-// Validate your configuration
-const { validateRagrc } = require("@devilsdev/rag-pipeline-utils");
-const isValid = validateRagrc(config);
-if (!isValid) {
-  console.error("Configuration is invalid");
-}
-```
+---
 
-### Debug Mode
+## Contributing
 
-```javascript
-const { logger } = require("@devilsdev/rag-pipeline-utils");
+We welcome contributions of all kinds:
 
-// Enable debug logging
-logger.setLevel("debug");
-```
+- **Code** – Fix bugs, add features, improve performance
+- **Documentation** – Write guides, improve examples, clarify concepts
+- **Testing** – Expand test coverage, add integration tests, report edge cases
+- **Design** – Propose API improvements, suggest architectural patterns
+- **Community** – Answer questions, review pull requests, mentor new contributors
 
-## Examples
+**Before You Start:**
 
-### Real-World Integration Examples
+1. Read the [Contributing Guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md)
+2. Check existing [Issues](https://github.com/DevilsDev/rag-pipeline-utils/issues) and [Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions)
+3. Open an issue to discuss major changes before coding
+4. Follow our [Code of Conduct](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/CODE_OF_CONDUCT.md)
 
-**[OpenAI + Pinecone Example](examples/openai-pinecone/)** (NEW in v2.2.0)
+**Pull Request Checklist:**
 
-- Complete RAG pipeline implementation
-- OpenAI embeddings (text-embedding-3-small)
-- Pinecone vector database integration
-- Mock mode for testing without API keys
-- Production-ready with error handling
+- [ ] Tests pass (`npm test`)
+- [ ] Linter passes (`npm run lint`)
+- [ ] Documentation updated
+- [ ] Changelog entry added
+- [ ] Type definitions updated (if applicable)
 
-**Basic Pipeline Example**
+---
 
-- See `examples/basic-pipeline/` for minimal setup
-- Simple in-memory implementation
-- Perfect for learning and prototyping
+## License
 
-## Documentation
+**GPL-3.0** – See [LICENSE](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/LICENSE) for full terms.
 
-- [GitHub Repository](https://github.com/DevilsDev/rag-pipeline-utils)
-- [Issue Tracker](https://github.com/DevilsDev/rag-pipeline-utils/issues)
-- [Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions)
-- [OpenAI + Pinecone Example](examples/openai-pinecone/) (NEW)
-- [Release Notes](https://github.com/DevilsDev/rag-pipeline-utils/releases)
-- [CHANGELOG](CHANGELOG.md)
+This is free software: you are free to change and redistribute it under the terms of the GNU General Public License version 3.
 
-## Contributing
+---
 
-Contributions are welcome. Please see the [contribution guidelines](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md).
+## What's Next?
 
-## License
+**Star this repository** if you find it useful—it helps others discover the project.
 
-GPL-3.0 - See [LICENSE](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/LICENSE) for details.
+**Share your use case** in [Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions/categories/show-and-tell) to inspire the community.
 
-## Support
+**Contribute** your first pull request—check out issues tagged [good first issue](https://github.com/DevilsDev/rag-pipeline-utils/labels/good%20first%20issue).
 
-- Issues: https://github.com/DevilsDev/rag-pipeline-utils/issues
-- Discussions: https://github.com/DevilsDev/rag-pipeline-utils/discussions
+Happy building.