@devilsdev/rag-pipeline-utils 2.1.11 → 2.2.0

package/README.md

@@ -8,315 +8,332 @@
 
 Enterprise-grade toolkit for building Retrieval-Augmented Generation (RAG) pipelines in Node.js with modular plugin architecture, streaming support, and observability.
 
-## Installation
+## 🎉 What's New in v2.2.0
 
-```bash
-npm install @devilsdev/rag-pipeline-utils
-```
+### Major Improvements
 
-**Requirements:** Node.js >= 18.0.0
+**✅ Fixed ESM Build (Critical)**
 
-## Quick Start
+- Implemented CJS→ESM interop pattern for proper module compatibility
+- All 15 exports now working correctly in both CommonJS and ES Modules
+- Verified compatibility with Node.js 18.x, 20.x, and 22.x
 
-### Basic Pipeline
+**📦 50% Smaller Install Size**
 
-```javascript
-const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils");
+- Reduced from ~30MB to ~15MB by optimizing dependencies
+- Moved CLI-only tools to devDependencies
+- Zero production vulnerabilities (npm audit clean)
 
-// Initialize pipeline with plugins
-const pipeline = createRagPipeline({
-  loader: new PDFLoader(),
-  embedder: new OpenAIEmbedder({
-    apiKey: process.env.OPENAI_API_KEY,
-    model: "text-embedding-ada-002",
-  }),
-  retriever: new ChromaRetriever({
-    url: "http://localhost:8000",
-    collection: "documents",
-  }),
-  llm: new OpenAILLM({
-    apiKey: process.env.OPENAI_API_KEY,
-    model: "gpt-4",
-  }),
-});
+**📝 Enhanced TypeScript Support**
 
-// Ingest documents
-await pipeline.ingest("./documents/whitepaper.pdf");
+- Added 5 complete plugin contract interfaces:
+  - `LoaderPlugin` - Document loading interface
+  - `EmbedderPlugin` - Text embedding interface
+  - `RetrieverPlugin` - Vector search interface
+  - `LLMPlugin` - Language model interface
+  - `RerankerPlugin` - Result reranking interface
+- Added 4 helper types: `Document`, `SearchResult`, `RetrieveOptions`, `LLMResponse`
+- 236 lines of comprehensive type definitions
+- Full IDE autocomplete and type checking support
 
-// Query
-const result = await pipeline.query("What is the main conclusion?");
-console.log(result.answer);
-```
+**🔧 Real-World Example**
 
-### Streaming Responses
+- Complete OpenAI + Pinecone integration example in `examples/openai-pinecone/`
+- Mock mode for testing without API keys
+- Production-ready error handling and cost estimation guidance
+- See [OpenAI + Pinecone Example](#openai--pinecone-example) below
 
-```javascript
-// Stream tokens in real-time
-const stream = await pipeline.query("Explain the methodology", {
-  stream: true,
-});
+**📚 Documentation Enhancements**
 
-for await (const chunk of stream) {
-  process.stdout.write(chunk);
-}
-```
+- Platform compatibility matrix (Linux, macOS, Windows, Docker)
+- Module system examples (CommonJS + ES Modules)
+- Docker usage examples
+- Troubleshooting guides
 
-### Batch Processing
+### Migration from v2.1.x
 
-```javascript
-// Process multiple documents concurrently
-const results = await pipeline.ingestBatch(
-  ["./docs/paper1.pdf", "./docs/paper2.pdf", "./docs/paper3.pdf"],
-  {
-    parallel: true,
-    maxConcurrency: 5,
-  },
-);
-```
+**No breaking changes!** v2.2.0 is fully backward compatible. Simply update:
 
-## Core API
+```bash
+npm install @devilsdev/rag-pipeline-utils@latest
+```
 
-### createRagPipeline(config)
+All existing code will continue to work without modifications.
 
-Creates a pipeline instance with the specified plugins.
+---
 
-**Parameters:**
+## Installation
 
-- `config.loader` - Document loader plugin
-- `config.embedder` - Text embedding plugin
-- `config.retriever` - Vector storage and retrieval plugin
-- `config.llm` - Language model plugin
-- `config.reranker` (optional) - Result reranking plugin
+```bash
+npm install @devilsdev/rag-pipeline-utils
+```
 
-**Returns:** Pipeline instance
+**Requirements:** Node.js >= 18.0.0
 
-### Pipeline Methods
+## Compatibility
 
-#### `pipeline.ingest(filePath, options)`
+### Platform Support
 
-Ingest a document into the pipeline.
+| Platform    | Status             | Notes                                                         |
+| ----------- | ------------------ | ------------------------------------------------------------- |
+| **Linux**   | ✅ Fully Supported | Tested on Ubuntu 20.04+, Debian 11+, RHEL 8+                  |
+| **macOS**   | ✅ Fully Supported | Tested on macOS 12+ (Intel & Apple Silicon)                   |
+| **Windows** | ✅ Fully Supported | Tested on Windows 10/11, WSL2 recommended for best experience |
+| **Docker**  | ✅ Fully Supported | Alpine & Debian-based images available                        |
 
-```javascript
-await pipeline.ingest("./document.pdf", {
-  chunkSize: 1000,
-  chunkOverlap: 200,
-  metadata: { source: "research", year: 2024 },
-});
-```
+### Module Systems
 
-#### `pipeline.query(question, options)`
+This package supports both CommonJS and ES Modules:
 
-Query the pipeline.
+**CommonJS (require)**
 
 ```javascript
-const result = await pipeline.query("What is RAG?", {
-  topK: 5,
-  similarityThreshold: 0.7,
-  stream: false,
+const {
+  createRagPipeline,
+  DAGEngine,
+} = require("@devilsdev/rag-pipeline-utils");
+
+const pipeline = createRagPipeline({
+  /* config */
 });
+const dag = new DAGEngine();
 ```
 
-**Result structure:**
+**ES Modules (import)**
 
 ```javascript
-{
-  answer: string,
-  sources: Array<{
-    content: string,
-    metadata: object,
-    score: number
-  }>,
-  metrics: {
-    retrievalTime: number,
-    generationTime: number,
-    totalTime: number
-  }
-}
-```
-
-#### `pipeline.ingestBatch(files, options)`
-
-Batch ingest multiple documents.
+import { createRagPipeline, DAGEngine } from "@devilsdev/rag-pipeline-utils";
 
-```javascript
-await pipeline.ingestBatch(["file1.pdf", "file2.pdf"], {
-  parallel: true,
-  maxConcurrency: 3,
+const pipeline = createRagPipeline({
+  /* config */
 });
+const dag = new DAGEngine();
 ```
 
-## Plugin System
-
-### Plugin Contracts
+### TypeScript Support
 
-All plugins implement standardized interfaces for interoperability.
+**New in v2.2.0:** Complete plugin contract interfaces with 236 lines of type definitions:
 
-#### Loader Plugin
+```typescript
+import {
+  createRagPipeline,
+  DAGEngine,
+  RagPipelineConfig,
+  // Plugin contract interfaces (NEW in v2.2.0)
+  LoaderPlugin,
+  EmbedderPlugin,
+  RetrieverPlugin,
+  LLMPlugin,
+  RerankerPlugin,
+  // Helper types (NEW in v2.2.0)
+  Document,
+  SearchResult,
+  RetrieveOptions,
+  LLMResponse,
+} from "@devilsdev/rag-pipeline-utils";
 
-```javascript
-class CustomLoader {
-  async load(filePath) {
-    // Return array of document chunks
-    return [{ content: string, metadata: object }];
+// Type-safe plugin implementation
+class MyLoader implements LoaderPlugin {
+  async load(source: string, options?: any): Promise<Document[]> {
+    // Your implementation with full type checking
+    return [
+      {
+        id: "doc-1",
+        content: "Document content",
+        metadata: { source },
+      },
+    ];
   }
 }
-```
-
-#### Embedder Plugin
 
-```javascript
-class CustomEmbedder {
-  async embed(texts) {
-    // Return array of vectors
-    return [[0.1, 0.2, ...], [0.3, 0.4, ...]];
-  }
-
-  async embedQuery(query) {
-    // Return single vector
-    return [0.5, 0.6, ...];
-  }
+const config: RagPipelineConfig = {
+  loader: new MyLoader(),
+  embedder: "my-embedder",
+};
 
-  getDimensions() {
-    return 1536; // Vector dimensions
-  }
-}
+const pipeline = createRagPipeline(config);
 ```
 
-#### Retriever Plugin
+### Package Size & Dependencies
 
-```javascript
-class CustomRetriever {
-  async store(vectors, metadata) {
-    // Store vectors with metadata
-  }
+- **Package size**: 244KB (gzipped), 1.1MB (unpacked)
+- **Runtime dependencies**: Only 11 production dependencies
+- **Total install size**: ~15MB (including dependencies) - **50% smaller than v2.1.x**
+- **Tree-shakeable**: ES module exports support tree-shaking
+- **Zero vulnerabilities**: npm audit clean (production dependencies)
 
-  async retrieve(queryVector, options = {}) {
-    const { topK = 5, threshold = 0.7 } = options;
-    // Return similar documents
-    return [{ content: string, metadata: object, score: number }];
-  }
-}
-```
+### Docker Usage
 
-#### LLM Plugin
+**Basic Docker setup:**
 
-```javascript
-class CustomLLM {
-  async generate(prompt, options = {}) {
-    // Return generated text
-    return { text: string, usage: object };
-  }
+```dockerfile
+FROM node:18-alpine
 
-  async stream(prompt, options = {}) {
-    // Return async iterable
-    async function* generate() {
-      yield chunk1;
-      yield chunk2;
-    }
-    return generate();
-  }
-}
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --production
+COPY . .
+
+CMD ["node", "index.js"]
 ```
 
-#### Reranker Plugin (Optional)
+**Docker Compose example:**
 
-```javascript
-class CustomReranker {
-  async rerank(query, documents, options = {}) {
-    // Return reranked documents
-    return documents.map((doc) => ({
-      ...doc,
-      score: newScore,
-    }));
-  }
-}
+```yaml
+version: "3.8"
+services:
+  rag-pipeline:
+    image: node:18-alpine
+    working_dir: /app
+    volumes:
+      - ./:/app
+    environment:
+      - NODE_ENV=production
+    command: npm start
 ```
 
-## Configuration
-
-### Environment Variables
+## Quick Start
 
-```bash
-# OpenAI
-OPENAI_API_KEY=sk-...
-OPENAI_MODEL=gpt-4
+### Basic Pipeline Setup
 
-# Pinecone
-PINECONE_API_KEY=...
-PINECONE_ENVIRONMENT=us-east-1
+```javascript
+const { createRagPipeline } = require("@devilsdev/rag-pipeline-utils");
 
-# ChromaDB
-CHROMA_URL=http://localhost:8000
+// Create pipeline with your plugin implementations
+const pipeline = createRagPipeline({
+  loader: yourLoaderPlugin,
+  embedder: yourEmbedderPlugin,
+  retriever: yourRetrieverPlugin,
+  llm: yourLLMPlugin,
+});
 
-# Performance
-RAG_MAX_CONCURRENCY=5
-RAG_BATCH_SIZE=10
+// Use the pipeline
+await pipeline.ingest("./document.pdf");
+const result = await pipeline.query("What is this document about?");
+console.log(result);
 ```
 
-### Configuration File
+### Using Configuration File
 
 Create `.ragrc.json`:
 
 ```json
 {
   "loader": {
-    "pdf": "@devilsdev/pdf-loader",
-    "markdown": "@devilsdev/markdown-loader"
+    "pdf": "your-pdf-loader-plugin"
   },
   "embedder": {
-    "openai": "@devilsdev/openai-embedder"
+    "default": "your-embedder-plugin"
   },
   "retriever": {
-    "chroma": "@devilsdev/chroma-retriever"
+    "default": "your-vector-db-plugin"
   },
   "llm": {
-    "openai": "@devilsdev/openai-llm"
-  },
-  "pipeline": {
-    "loader": "pdf",
-    "embedder": "openai",
-    "retriever": "chroma",
-    "llm": "openai"
-  },
-  "performance": {
-    "maxConcurrency": 5,
-    "enableStreaming": true
+    "default": "your-llm-plugin"
   }
 }
 ```
 
-Load configuration:
+Load and use configuration:
 
 ```javascript
-const { loadConfig } = require("@devilsdev/rag-pipeline-utils");
+const {
+  loadConfig,
+  createRagPipeline,
+} = require("@devilsdev/rag-pipeline-utils");
+
 const config = await loadConfig(".ragrc.json");
+const pipeline = createRagPipeline(config);
 ```
 
-## Advanced Usage
+## Core API
 
-### Custom Pipeline Components
+### Exported Modules
 
 ```javascript
 const {
+  // Core pipeline
   createRagPipeline,
+
+  // Configuration
+  loadConfig,
+  validateRagrc,
+  normalizeConfig,
+
+  // Plugin system
   pluginRegistry,
+
+  // DAG workflow engine
+  DAGEngine,
+
+  // Observability
+  eventLogger,
+  metrics,
+
+  // Performance
+  ParallelProcessor,
+
+  // AI/ML
+  MultiModalProcessor,
+  AdaptiveRetrievalEngine,
+
+  // Enterprise
+  AuditLogger,
+  DataGovernance,
+
+  // Utilities
+  logger,
 } = require("@devilsdev/rag-pipeline-utils");
+```
 
-// Register custom plugin
-pluginRegistry.register("my-embedder", MyCustomEmbedder);
+### createRagPipeline(config)
 
-// Use in pipeline
-const pipeline = createRagPipeline({
-  loader: new PDFLoader(),
-  embedder: pluginRegistry.get("my-embedder"),
-  retriever: new ChromaRetriever({ url: "http://localhost:8000" }),
-  llm: new OpenAILLM({ model: "gpt-4" }),
-});
+Creates a RAG pipeline instance.
+
+**Parameters:**
+
+- `config.loader` - Document loader plugin
+- `config.embedder` - Text embedding plugin
+- `config.retriever` - Vector storage and retrieval plugin
+- `config.llm` - Language model plugin
+- `config.reranker` (optional) - Result reranking plugin
+
+**Returns:** Pipeline instance with `ingest()` and `query()` methods
+
+### Configuration Management
+
+```javascript
+// Load configuration from file
+const config = await loadConfig(".ragrc.json");
+
+// Validate configuration
+const isValid = validateRagrc(config);
+
+// Normalize configuration
+const normalized = normalizeConfig(config);
 ```
 
-### DAG Workflows
+### Plugin Registry
 
-Build complex multi-step pipelines using the DAG engine:
+```javascript
+const { pluginRegistry } = require("@devilsdev/rag-pipeline-utils");
+
+// Register a plugin
+pluginRegistry.register("my-plugin", MyPluginClass);
+
+// Get a plugin
+const Plugin = pluginRegistry.get("my-plugin");
+
+// List all plugins
+const plugins = pluginRegistry.list();
+
+// Check if plugin exists
+const exists = pluginRegistry.has("my-plugin");
+```
+
+### DAG Workflow Engine
+
+Build complex multi-step workflows:
 
 ```javascript
 const { DAGEngine } = require("@devilsdev/rag-pipeline-utils");
@@ -326,19 +343,25 @@ const dag = new DAGEngine({
     {
       id: "load",
       type: "loader",
-      config: { source: "./docs" },
+      config: {
+        /* loader config */
+      },
     },
     {
       id: "embed",
       type: "embedder",
       dependencies: ["load"],
-      config: { model: "text-embedding-ada-002" },
+      config: {
+        /* embedder config */
+      },
     },
     {
       id: "store",
       type: "retriever",
       dependencies: ["embed"],
-      config: { collection: "documents" },
+      config: {
+        /* retriever config */
+      },
     },
   ],
 });
@@ -348,114 +371,325 @@ await dag.execute();
 
 ### Observability
 
-Enable monitoring and tracing:
-
 ```javascript
 const { eventLogger, metrics } = require("@devilsdev/rag-pipeline-utils");
 
-// Configure logging
+// Configure event logging
 eventLogger.configure({
   level: "info",
   format: "json",
 });
 
+// Log events
+eventLogger.log("info", "Pipeline started", { pipelineId: "abc123" });
+
 // Access metrics
 const pipelineMetrics = metrics.getMetrics();
-console.log(pipelineMetrics.queries.total);
-console.log(pipelineMetrics.queries.avgLatency);
+console.log(pipelineMetrics);
 ```
 
-### Error Handling
+### Parallel Processing
 
 ```javascript
-try {
-  const result = await pipeline.query("What is AI?");
-} catch (error) {
-  if (error.code === "RETRIEVAL_ERROR") {
-    console.error("Vector search failed:", error.message);
-  } else if (error.code === "LLM_ERROR") {
-    console.error("Generation failed:", error.message);
+const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils");
+
+const processor = new ParallelProcessor({
+  maxConcurrency: 5,
+});
+
+await processor.process(items, async (item) => {
+  // Process each item
+  return processItem(item);
+});
+```
+
+## Plugin Development
+
+This package provides a plugin-based architecture. You need to implement plugins for your specific use case.
+
+### Plugin Contracts
+
+**New in v2.2.0:** TypeScript interfaces for all plugin contracts. See [TypeScript Support](#typescript-support) for full type definitions.
+
+#### Loader Plugin
+
+```javascript
+/**
+ * Loader plugin interface (NEW TypeScript interface in v2.2.0)
+ * Implements: LoaderPlugin
+ */
+class MyLoader {
+  async load(source, options) {
+    // Load and parse document
+    // Return array of Document objects
+    return [
+      {
+        id: "doc-1",
+        content: "Document text...",
+        metadata: { page: 1, source },
+        // embedding: optional pre-computed embedding
+      },
+    ];
   }
 }
 ```
 
-## CLI Usage
+#### Embedder Plugin
+
+```javascript
+/**
+ * Embedder plugin interface (NEW TypeScript interface in v2.2.0)
+ * Implements: EmbedderPlugin
+ */
+class MyEmbedder {
+  async embed(text, options) {
+    // Generate embedding for a single text string
+    // Return: Promise<number[]> - embedding vector
+    return [0.1, 0.2, 0.3 /* ... 1536 dimensions for OpenAI */];
+  }
+
+  // Optional batch embedding method
+  async embedBatch(texts) {
+    // Generate embeddings for multiple texts
+    return texts.map((text) => this.embed(text));
+  }
+}
+```
 
-The package includes a CLI for common operations:
+#### Retriever Plugin
+
+```javascript
+/**
+ * Retriever plugin interface (NEW TypeScript interface in v2.2.0)
+ * Implements: RetrieverPlugin
+ */
+class MyRetriever {
+  async retrieve(query, options) {
+    // query can be:
+    // - string: search by text
+    // - { embedding: number[] }: search by vector
+    const { topK = 5, minScore = 0.7, filter } = options || {};
+
+    // Search for similar vectors
+    // Return: Promise<SearchResult[]>
+    return [
+      {
+        id: "result-1",
+        score: 0.95,
+        document: {
+          id: "doc-1",
+          content: "Document text...",
+          metadata: { source: "file.pdf" },
+        },
+        metadata: {
+          /* additional search metadata */
+        },
+      },
+    ];
+  }
+}
+```
+
+#### LLM Plugin
+
+```javascript
+/**
+ * LLM plugin interface (NEW TypeScript interface in v2.2.0)
+ * Implements: LLMPlugin
+ */
+class MyLLM {
+  async generate(prompt, options) {
+    // Generate text from prompt
+    // Return: Promise<LLMResponse>
+    return {
+      text: "Generated response...",
+      usage: {
+        promptTokens: 100,
+        completionTokens: 50,
+        totalTokens: 150,
+      },
+      metadata: {
+        model: "gpt-4",
+        finishReason: "stop",
+      },
+    };
+  }
+
+  // Optional streaming support
+  async *stream(prompt, options) {
+    // Stream response chunks
+    yield "Generated ";
+    yield "response ";
+    yield "chunks...";
+  }
+}
+```
+
+#### Reranker Plugin (Optional)
+
+```javascript
+/**
+ * Reranker plugin interface (NEW TypeScript interface in v2.2.0)
+ * Implements: RerankerPlugin
+ */
+class MyReranker {
+  async rerank(results, query, options) {
+    // Rerank search results based on relevance to query
+    // Return: Promise<SearchResult[]> - reranked results
+    return results
+      .map((result) => ({
+        ...result,
+        score: calculateRelevanceScore(query, result),
+      }))
+      .sort((a, b) => b.score - a.score);
+  }
+}
+```
+
+## Environment Configuration
 
 ```bash
-# Initialize configuration
-npx rag-pipeline init
+# Example environment variables for your plugins
+OPENAI_API_KEY=your_api_key
+PINECONE_API_KEY=your_api_key
+PINECONE_ENVIRONMENT=us-east-1
+CHROMA_URL=http://localhost:8000
 
-# Ingest documents
-npx rag-pipeline ingest ./docs --loader pdf
+# Performance settings
+RAG_MAX_CONCURRENCY=5
+RAG_BATCH_SIZE=10
+```
 
-# Query interactively
-npx rag-pipeline query "Your question here"
+## Advanced Features
 
-# Run health check
-npx rag-pipeline doctor
+### Multi-Modal Processing
+
+```javascript
+const { MultiModalProcessor } = require("@devilsdev/rag-pipeline-utils");
+
+const processor = new MultiModalProcessor({
+  // Configuration for processing text, images, audio, etc.
+});
+
+await processor.process(multiModalData);
 ```
 
-## Use Cases
+### Adaptive Retrieval
+
+```javascript
+const { AdaptiveRetrievalEngine } = require("@devilsdev/rag-pipeline-utils");
+
+const retriever = new AdaptiveRetrievalEngine({
+  // Configuration for adaptive retrieval strategies
+});
 
-### Document Q&A Systems
+const results = await retriever.retrieve(query, context);
+```
 
-Build question-answering systems over proprietary documents:
+### Enterprise Audit Logging
 
 ```javascript
-const pipeline = createRagPipeline({
-  /* config */
+const { AuditLogger } = require("@devilsdev/rag-pipeline-utils");
+
+const auditLogger = new AuditLogger({
+  destination: "./audit-logs",
 });
 
-// Ingest company documentation
-await pipeline.ingestBatch([
-  "./docs/handbook.pdf",
-  "./docs/policies.pdf",
-  "./docs/procedures.pdf",
-]);
+auditLogger.log("query", {
+  user: "user@example.com",
+  query: "sensitive query",
+  timestamp: new Date(),
+});
+```
 
-// Answer questions
-const answer = await pipeline.query("What is the vacation policy?");
+### Data Governance
+
+```javascript
+const { DataGovernance } = require("@devilsdev/rag-pipeline-utils");
+
+const governance = new DataGovernance({
+  policies: {
+    retention: "90d",
+    encryption: true,
+  },
+});
+
+await governance.enforce(data);
 ```
 
-### Semantic Search
+## CLI Usage
+
+The package includes a CLI tool for pipeline operations:
+
+```bash
+# Initialize configuration
+npx rag-pipeline init
 
-Implement semantic search over large document collections:
+# Run pipeline operations (requires plugin implementation)
+npx rag-pipeline ingest ./documents
+npx rag-pipeline query "Your question"
+
+# Run system diagnostics
+npx rag-pipeline doctor
+```
+
+## Error Handling
 
 ```javascript
-const results = await pipeline.query("machine learning papers", {
-  topK: 10,
-  similarityThreshold: 0.75,
-  returnSources: true,
+try {
+  const result = await pipeline.query("What is AI?");
+} catch (error) {
+  if (error.code === "PLUGIN_NOT_FOUND") {
+    console.error("Required plugin not found:", error.message);
+  } else if (error.code === "INVALID_CONFIG") {
+    console.error("Invalid configuration:", error.message);
+  } else {
+    console.error("Pipeline error:", error.message);
+  }
+}
+```
+
+## Use Cases
+
+### Document Q&A System
+
+```javascript
+// Implement your plugins
+const loader = new PDFLoaderPlugin();
+const embedder = new OpenAIEmbedderPlugin({
+  apiKey: process.env.OPENAI_API_KEY,
 });
+const retriever = new ChromaRetrieverPlugin({ url: "http://localhost:8000" });
+const llm = new OpenAILLMPlugin({ model: "gpt-4" });
 
-results.sources.forEach((source) => {
-  console.log(`${source.metadata.title} (score: ${source.score})`);
+// Create pipeline
+const pipeline = createRagPipeline({
+  loader,
+  embedder,
+  retriever,
+  llm,
 });
-```
 
-### Knowledge Base Integration
+// Ingest documents
+await pipeline.ingest("./company-handbook.pdf");
+
+// Query
+const answer = await pipeline.query("What is the vacation policy?");
+console.log(answer);
+```
 
-Integrate with existing knowledge bases:
+### Knowledge Base Search
 
 ```javascript
-// Custom retriever for your database
+// Use your custom database retriever
 class PostgresRetriever {
   constructor(connectionString) {
     this.db = new PostgresClient(connectionString);
   }
 
-  async store(vectors, metadata) {
-    await this.db.query(
-      "INSERT INTO embeddings (vector, metadata) VALUES ($1, $2)",
-      [vectors, metadata],
-    );
-  }
-
   async retrieve(queryVector, options) {
     const results = await this.db.query(
-      "SELECT * FROM embeddings ORDER BY vector <-> $1 LIMIT $2",
+      "SELECT content, metadata, vector <-> $1 as score FROM embeddings ORDER BY score LIMIT $2",
       [queryVector, options.topK],
     );
     return results.rows;
@@ -463,191 +697,216 @@ class PostgresRetriever {
 }
 
 const pipeline = createRagPipeline({
-  loader: new PDFLoader(),
-  embedder: new OpenAIEmbedder({ apiKey: process.env.OPENAI_API_KEY }),
+  loader: yourLoader,
+  embedder: yourEmbedder,
   retriever: new PostgresRetriever("postgresql://localhost/kb"),
-  llm: new OpenAILLM({ model: "gpt-4" }),
+  llm: yourLLM,
 });
 ```
 
-### Multi-Source Aggregation
+## Performance Tips
 
-Combine data from multiple sources:
+### Batch Processing
 
 ```javascript
-const sources = [
-  { type: "pdf", path: "./research/*.pdf" },
-  { type: "markdown", path: "./docs/*.md" },
-  { type: "api", url: "https://api.example.com/docs" },
-];
-
-for (const source of sources) {
-  const loader = getLoaderForType(source.type);
-  await pipeline.ingest(source.path || source.url, {
-    loader,
-    metadata: { source: source.type },
-  });
-}
-```
+const { ParallelProcessor } = require("@devilsdev/rag-pipeline-utils");
 
-## Performance Optimization
-
-### Chunking Strategies
+const processor = new ParallelProcessor({
+  maxConcurrency: 5,
+  batchSize: 10,
+});
 
-```javascript
-await pipeline.ingest("./large-document.pdf", {
-  chunkSize: 1000, // Characters per chunk
-  chunkOverlap: 200, // Overlap between chunks
-  chunkStrategy: "sentence", // 'sentence' | 'paragraph' | 'fixed'
+// Process documents in parallel
+await processor.process(documents, async (doc) => {
+  return await pipeline.ingest(doc);
 });
 ```
 
-### Concurrent Processing
+### Logging and Monitoring
 
 ```javascript
-await pipeline.ingestBatch(files, {
-  parallel: true,
-  maxConcurrency: 5, // Parallel operations
-  batchSize: 10, // Embeddings per batch
-});
-```
+const { logger, metrics } = require("@devilsdev/rag-pipeline-utils");
 
-### Caching
+// Set log level
+logger.setLevel("debug");
 
-```javascript
-const pipeline = createRagPipeline({
-  loader: new PDFLoader(),
-  embedder: new OpenAIEmbedder({
-    apiKey: process.env.OPENAI_API_KEY,
-    cache: true, // Cache embeddings
-    cacheTTL: 3600, // Cache lifetime in seconds
-  }),
-  retriever: new ChromaRetriever({ url: "http://localhost:8000" }),
-  llm: new OpenAILLM({ model: "gpt-4" }),
-});
+// Monitor performance
+const stats = metrics.getMetrics();
+console.log("Total queries:", stats.queries?.total || 0);
+console.log("Average latency:", stats.queries?.avgLatency || 0);
 ```
 
-## TypeScript Support
+## Common Integration Examples
 
-Full TypeScript definitions included:
+### OpenAI + Pinecone Example
 
-```typescript
-import {
-  createRagPipeline,
-  Pipeline,
-  LoaderPlugin,
-  EmbedderPlugin,
-  RetrieverPlugin,
-  LLMPlugin,
-} from "@devilsdev/rag-pipeline-utils";
+**New in v2.2.0:** Complete working example with OpenAI embeddings and Pinecone vector database.
 
-const pipeline: Pipeline = createRagPipeline({
-  loader: new PDFLoader(),
-  embedder: new OpenAIEmbedder({ apiKey: string }),
-  retriever: new ChromaRetriever({ url: string }),
-  llm: new OpenAILLM({ model: string }),
-});
+See the full example in [`examples/openai-pinecone/`](examples/openai-pinecone/) with:
 
-interface QueryResult {
-  answer: string;
-  sources: Source[];
-  metrics: Metrics;
-}
+- ✅ Complete plugin implementations
+- ✅ Mock mode (no API keys required for testing)
+- ✅ Real OpenAI integration
+- ✅ Real Pinecone integration
+- ✅ Error handling and cost estimation
+- ✅ Comprehensive documentation
+
+**Quick start:**
+
+```bash
+cd examples/openai-pinecone
+npm install
+
+# Test with mock mode (no API keys needed)
+USE_MOCK_MODE=true npm start
 
-const result: QueryResult = await pipeline.query("question");
+# Run with real APIs (requires API keys in .env)
+npm start
 ```
 
-## API Reference
+### With OpenAI
 
-### Exported Modules
+You need to implement OpenAI plugin wrappers:
 
 ```javascript
-const {
-  // Core
-  createRagPipeline,
-  loadConfig,
-
-  // Plugin System
-  pluginRegistry,
+class OpenAIEmbedder {
+  constructor({ apiKey, model = "text-embedding-ada-002" }) {
+    this.client = new OpenAI({ apiKey });
+    this.model = model;
+  }
 
-  // Workflow Engine
-  DAGEngine,
+  async embed(texts) {
+    const response = await this.client.embeddings.create({
+      model: this.model,
+      input: texts,
+    });
+    return response.data.map((d) => d.embedding);
+  }
+}
 
-  // Observability
-  eventLogger,
-  metrics,
+class OpenAILLM {
+  constructor({ apiKey, model = "gpt-4" }) {
+    this.client = new OpenAI({ apiKey });
+    this.model = model;
+  }
 
-  // Utilities
-  logger,
-} = require("@devilsdev/rag-pipeline-utils");
+  async generate(prompt) {
+    const response = await this.client.chat.completions.create({
+      model: this.model,
+      messages: [{ role: "user", content: prompt }],
+    });
+    return { text: response.choices[0].message.content };
+  }
+}
 ```
 
-### Plugin Registry API
+### With ChromaDB
 
 ```javascript
-// Register plugin
-pluginRegistry.register("name", PluginClass);
-
-// Get plugin
-const Plugin = pluginRegistry.get("name");
+class ChromaRetriever {
+  constructor({ url, collection }) {
+    this.client = new ChromaClient({ path: url });
+    this.collectionName = collection;
+  }
 
-// List plugins
-const plugins = pluginRegistry.list();
+  async store(vectors, metadata) {
+    const collection = await this.client.getOrCreateCollection({
+      name: this.collectionName,
+    });
+
+    await collection.add({
+      embeddings: vectors,
+      metadatas: metadata,
+      ids: metadata.map((_, i) => `doc_${i}`),
+    });
+  }
 
-// Check if plugin exists
-const exists = pluginRegistry.has("name");
+  async retrieve(queryVector, options) {
+    const collection = await this.client.getCollection({
+      name: this.collectionName,
+    });
+
+    const results = await collection.query({
+      queryEmbeddings: [queryVector],
+      nResults: options.topK || 5,
+    });
+
+    return results.metadatas[0].map((meta, i) => ({
+      content: results.documents[0][i],
+      metadata: meta,
+      score: 1 - results.distances[0][i],
+    }));
+  }
+}
 ```
 
 ## Troubleshooting
 
 ### Common Issues
 
-**Error: Module not found**
+**Error: Cannot find module**
 
 ```bash
-npm install --save @devilsdev/rag-pipeline-utils
+npm install @devilsdev/rag-pipeline-utils
 ```
 
-**Error: Invalid API key**
+**Error: Plugin not registered**
 
 ```javascript
-// Ensure environment variables are set
-require("dotenv").config();
-const embedder = new OpenAIEmbedder({
-  apiKey: process.env.OPENAI_API_KEY,
-});
+// Make sure to register your plugins
+pluginRegistry.register("my-embedder", MyEmbedderClass);
 ```
 
-**Error: Connection refused (ChromaDB)**
+**Error: Invalid configuration**
 
-```bash
-# Start ChromaDB server
-docker run -p 8000:8000 chromadb/chroma
+```javascript
+// Validate your configuration
+const { validateRagrc } = require("@devilsdev/rag-pipeline-utils");
+const isValid = validateRagrc(config);
+if (!isValid) {
+  console.error("Configuration is invalid");
+}
 ```
 
 ### Debug Mode
 
-Enable verbose logging:
-
 ```javascript
 const { logger } = require("@devilsdev/rag-pipeline-utils");
 
+// Enable debug logging
 logger.setLevel("debug");
 ```
 
 ## Examples
 
-See the [examples directory](https://github.com/DevilsDev/rag-pipeline-utils/tree/main/examples) for complete working examples:
+### Real-World Integration Examples
+
+**[OpenAI + Pinecone Example](examples/openai-pinecone/)** (NEW in v2.2.0)
+
+- Complete RAG pipeline implementation
+- OpenAI embeddings (text-embedding-3-small)
+- Pinecone vector database integration
+- Mock mode for testing without API keys
+- Production-ready with error handling
+
+**Basic Pipeline Example**
+
+- See `examples/basic-pipeline/` for minimal setup
+- Simple in-memory implementation
+- Perfect for learning and prototyping
+
+## Documentation
 
-- Basic RAG pipeline
-- Streaming responses
-- Custom plugins
-- DAG workflows
-- Multi-source ingestion
+- [GitHub Repository](https://github.com/DevilsDev/rag-pipeline-utils)
+- [Issue Tracker](https://github.com/DevilsDev/rag-pipeline-utils/issues)
+- [Discussions](https://github.com/DevilsDev/rag-pipeline-utils/discussions)
+- [OpenAI + Pinecone Example](examples/openai-pinecone/) (NEW)
+- [Release Notes](https://github.com/DevilsDev/rag-pipeline-utils/releases)
+- [CHANGELOG](CHANGELOG.md)
 
 ## Contributing
 
-Contributions welcome. Please read the [contribution guidelines](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md) and [plugin developer guide](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/PLUGIN_DEVELOPER_GUIDE.md).
+Contributions are welcome. Please see the [contribution guidelines](https://github.com/DevilsDev/rag-pipeline-utils/blob/main/docs/CONTRIBUTING.md).
 
 ## License
 
@@ -655,6 +914,5 @@ GPL-3.0 - See [LICENSE](https://github.com/DevilsDev/rag-pipeline-utils/blob/mai
 
 ## Support
 
-- Documentation: https://devilsdev.github.io/rag-pipeline-utils/
 - Issues: https://github.com/DevilsDev/rag-pipeline-utils/issues
-- Discussions: https://github.com/DevilsDev/rag-pipeline-utils/discussionsDo
+- Discussions: https://github.com/DevilsDev/rag-pipeline-utils/discussions