@devilsdev/rag-pipeline-utils 2.5.0 → 2.6.1

package/CHANGELOG.md

@@ -9,6 +9,105 @@ Older history (pre-2.4.0) is preserved in the
 [GitHub releases](https://github.com/DevilsDev/rag-pipeline-utils/releases)
 page.
 
+## [Unreleased]
+
+## [2.6.1] - 2026-10-29
+
+Documentation only; no code changes.
+
+### Changed
+
+- **README covers the 2.6.0 features**, so the npm package page shows
+  them: `checkAnswer` for answers from other frameworks, the `/testing`
+  entry, caching, the `query` `--explain`, `--trace`, `--stats` and
+  `--streaming` options, `OpenAIConnector` `baseURL` and the AI SDK
+  adapter.
+- **StackBlitz examples checked**: the basic pipeline, OpenAI with
+  Pinecone (offline mock) and the Fastify service each install and run.
+  The docs now say to reload if StackBlitz stalls while cloning.
+
+## [2.6.0] - 2026-10-29
+
+### Added
+
+- **`OpenAIConnector` `baseURL`** for OpenAI-compatible APIs (Groq,
+  Together, the Hugging Face router, vLLM, LM Studio, Ollama's `/v1`
+  endpoint). With a custom `baseURL` the key is never read from
+  `OPENAI_API_KEY`, so an OpenAI key is not sent to another provider; pass
+  `apiKey`, or omit it for local servers that need none.
+- **`validatePlugin(stage, plugin, { registry })`** returns
+  `{ valid, errors, warnings }` for a plugin without throwing, for use in
+  plugin tests. `registry: true` also applies the rules
+  `pluginRegistry.register()` enforces.
+- **`ParallelEmbedder` `mode`** (`'auto'`, `'batch'`, `'single'`).
+- **`checkAnswer({ question, answer, sources })`** checks an answer from any
+  framework (LangChain, LlamaIndex, Mastra, the AI SDK, your own code)
+  against its sources: `grounded`, `groundedness`, the `unsupported`
+  sentences, per-sentence `citations` and `evaluation` scores. Sources may be
+  strings, `{ id, content }`, LangChain Documents or LlamaIndex nodes;
+  `normalizeSources` is exported for the conversion alone.
+- **Caching**: `createRagPipeline({ cache })` caches query embeddings,
+  retrieval results and answers (`true`, a store, or per layer with time to
+  live). `MemoryCache` (LRU and TTL) and `RedisCache` (node-redis or
+  ioredis) are included; any `get`/`set` store works. A layer caches only
+  when its component identifies itself (`getCacheKey(role)` or `cacheKey`;
+  all built-ins do), retrieval is keyed by the store's state and the query
+  vector, answers by the exact documents and generation options, and
+  answers are not cached when the temperature is above 0. Retrieval is
+  cached before `filterResults` and the reranker, so guardrail ACLs run on
+  every request. Nothing from a failed, aborted or timed-out run, or from an
+  unfinished stream, is stored. Results carry `cache` per layer and the
+  pipeline has `cacheStats()`.
+- **`@devilsdev/rag-pipeline-utils/testing`**: deterministic, offline test
+  helpers. `HashEmbedder`, `EchoLLM` (answers with the top document or a set
+  answer, records every call, streams), `fixtures.documents` and
+  `createTestPipeline()`. Shares classes with the main entry and is not
+  loaded by it.
+- **`computeSemanticSimilarity(answer, reference, { embedder })`**: cosine
+  similarity of the two embeddings, 0 to 1. `checkAnswer` adds it as
+  `evaluation.scores.semanticSimilarity` when given a `referenceAnswer` and
+  an `embedder`.
+- **`rag-pipeline query`** options: `--trace` (time per stage), `--explain`
+  (passage scores, and the passages supporting each answer sentence, with
+  unsupported sentences flagged), `--stats` (latency, estimated tokens and
+  cost), `--streaming` and `--timeout <ms>`; global `--quiet`. `--json`
+  includes trace, stats and explanation when asked.
+- **`createAiSdkAdapter({ ai, embeddingModel, languageModel })`** wraps
+  Vercel AI SDK models as an `embedder` and an `llm` for
+  `createRagPipeline()`, with streaming, `embedMany` batching and the run's
+  `AbortSignal`. You pass the AI SDK functions, so it is not a dependency and
+  works with AI SDK 5, 6 and 7 from ESM or CommonJS. The RAG system prompt
+  is sent as both `instructions` (AI SDK 7) and `system` (5 and 6). Tested
+  against AI SDK 6 and 7 with their mock models.
+
+### Changed
+
+- **Faster CLI.** Commands load only their own dependencies: `--help`
+  takes about 0.5 s and `query` about 0.5 s instead of about 3.6 s. Loading
+  the CLI drops from 1281 modules and 93 packages to 55 and 13. The HTML,
+  Markdown and CSV loaders load `jsdom`, `marked` and `csv-parse` when they
+  read a file.
+- **Faster import.** `require('@devilsdev/rag-pipeline-utils')` takes about
+  125 ms instead of about 515 ms and loads 31 modules from 9 packages
+  instead of 245 from 54 (published bundle, Node 22). `axios`,
+  `jsonwebtoken`, `ajv` and `chokidar` now load when the feature that needs
+  them is first used (the Anthropic, Cohere and Ollama connectors,
+  `JWTValidator`, plugin registration and config validation, hot reload).
+  The shared `pluginRegistry` loads its contracts on first use; registries
+  created with `new PluginRegistry()` still load them in the constructor.
+
+### Fixed
+
+- **`ExecutionTracer.exportTraces('json')`** threw "circular structure"
+  with the default `captureOutputs: true`: traces stored the live result,
+  which carries the trace. Traces now store JSON snapshots, which also keeps
+  later changes to a result out of its recorded trace.
+- **`ParallelEmbedder`** failed with the built-in connectors, which embed
+  one string at a time, because it always called `embed(batch)` with an
+  array. In the default `'auto'` mode it now embeds texts one by one for
+  those connectors and uses `embedBatch(texts)` when an embedder has it;
+  other embedders are still called with the array.
+
 ## [2.5.0] - 2026-10-28
 
 ### Added
@@ -468,6 +567,9 @@ First public release on the 2.4 line, introducing the v2.4 feature set.
 For release notes prior to v2.4.0, see
 [github.com/DevilsDev/rag-pipeline-utils/releases](https://github.com/DevilsDev/rag-pipeline-utils/releases).
 
+[2.6.1]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.6.1
+[2.6.0]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.6.0
+[2.5.0]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.5.0
 [2.4.5]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.5
 [2.4.4]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.4
 [2.4.3]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.3

package/README.md

@@ -143,7 +143,15 @@ npx rag-pipeline serve --port 3000         # local HTTP API: GET /health, POST /
 npx rag-pipeline doctor                    # checks config, API keys, plugins and the index
 ```
 
-The default `local` template needs no API keys. Use `init --template openai`, `anthropic` or `ollama` to generate answers with a model. Other commands include `validate`, `hub install` for plugins and `ai benchmark` for measured latency. See the [CLI reference](https://devilsdev.github.io/rag-pipeline-utils/docs/CLI).
+The default `local` template needs no API keys. Use `init --template openai`, `anthropic` or `ollama` to generate answers with a model. Other commands include `validate`, `hub install` for plugins and `ai benchmark` for measured latency.
+
+`query` can also show its work: `--explain` lists each passage's score and which passages support each sentence, `--trace` times each stage, `--stats` estimates tokens and cost, and `--streaming` prints the answer as it is generated:
+
+```bash
+npx rag-pipeline query "How many vacation days do I get?" --explain --trace
+```
+
+See the [CLI reference](https://devilsdev.github.io/rag-pipeline-utils/docs/CLI).
 
 ## Use cases
 
@@ -198,6 +206,45 @@ console.log(report.scores); // includes contextRecall because a reference was gi
 if (report.scores.faithfulness < 0.8) process.exitCode = 1;
 ```
 
+### Check answers from another framework
+
+Already on LangChain, LlamaIndex, Mastra or the AI SDK? `checkAnswer` checks the answer your chain produced against the sources it was given, in the shape your framework returns them:
+
+```js
+import { checkAnswer } from "@devilsdev/rag-pipeline-utils";
+
+const check = await checkAnswer({
+  question: "How many vacation days do I get?",
+  answer: "Full-time employees receive thirty days of paid vacation per year.",
+  sources: [
+    {
+      pageContent:
+        "Full-time employees receive twenty days of paid vacation per year.",
+      metadata: { source: "handbook.md" },
+    },
+  ],
+});
+
+console.log(check.grounded); // false
+console.log(check.unsupported.length); // 1
+```
+
+### Test your RAG code offline
+
+`@devilsdev/rag-pipeline-utils/testing` has a deterministic embedder, an LLM that echoes the top passage and records its calls, and fixture documents, so tests need no network or API keys:
+
+```js
+import { createTestPipeline } from "@devilsdev/rag-pipeline-utils/testing";
+
+const { pipeline, llm } = await createTestPipeline();
+const result = await pipeline.run({
+  query: "How many vacation days do I get?",
+});
+
+console.log(result.results[0].id); // "vacation"
+console.log(llm.calls.length); // 1
+```
+
 ### Protect an internal assistant
 
 `GuardrailsPipeline` wraps any pipeline. Access control runs before the LLM sees a document, prompt-injection detection runs on the query, and PII is redacted from the answer, its citations and its evaluation details.
@@ -354,6 +401,30 @@ const result = await pipeline.run({
 if (!result.success) console.error(result.error); // "Execution timeout" or "Execution aborted"
 ```
 
+### Cache repeated questions
+
+`cache: true` caches query embeddings, retrieval results and answers in memory, so a repeated question makes no model calls. Use `RedisCache` to share the cache between processes. Cached results are not reused when the index, embedder, model or its settings change, and access-control filtering still runs on every request:
+
+```js
+import { createRagPipeline } from "@devilsdev/rag-pipeline-utils";
+import {
+  createTestPipeline,
+  EchoLLM,
+} from "@devilsdev/rag-pipeline-utils/testing";
+
+const { embedder, retriever } = await createTestPipeline();
+const llm = new EchoLLM();
+const pipeline = createRagPipeline({ embedder, retriever, llm, cache: true });
+
+await pipeline.run({ query: "How many vacation days do I get?" });
+const again = await pipeline.run({ query: "How many vacation days do I get?" });
+
+console.log(again.cache.generation); // "hit"
+console.log(llm.calls.length); // 1
+```
+
+Answers are cached only at temperature 0. See [Caching](https://devilsdev.github.io/rag-pipeline-utils/docs/Caching).
+
 ### Run fully local
 
 Pair the `LocalEmbedder` or Ollama embeddings with an Ollama model and `FileVectorStore` for a private, offline setup. The CLI's `local` and `ollama` templates set this up for you:
@@ -391,19 +462,22 @@ The result of `run()` is:
 
 ## Capabilities
 
-| Area           | What you get                                                                                               |
-| -------------- | ---------------------------------------------------------------------------------------------------------- |
-| **Chunking**   | `ChunkingEngine` with sentence, fixed-size, recursive, semantic and structure-aware strategies             |
-| **Retrieval**  | `MemoryRetriever`, persistent `FileVectorStore`, `BM25Search`, `HybridRetriever` (Reciprocal Rank Fusion)  |
-| **Reranking**  | `ScoringReranker`, `EmbeddingReranker` (cached), `CascadeReranker`                                         |
-| **Citations**  | Per-sentence source mapping, hallucination report, numeric-consistency check                               |
-| **Evaluation** | Faithfulness, relevance, context precision, context recall (with a reference), groundedness, BLEU/ROUGE    |
-| **Guardrails** | Injection detection with input normalisation, ACL and relevance filtering before generation, PII redaction |
-| **Agentic**    | `QueryPlanner`, `IterativeRetriever`, `SelfCritiqueChecker`, `AgenticPipeline`                             |
-| **GraphRAG**   | `KnowledgeGraph`, `EntityExtractor`, `GraphRetriever`                                                      |
-| **Streaming**  | Token streaming with a final chunk carrying citations and evaluation; `SSEAdapter`, `WebSocketAdapter`     |
-| **Cost**       | `CostTracker`, `TokenBudget`, `CostCalculator` with sourced, overridable pricing                           |
-| **Operations** | `DevServer` HTTP API, `ExecutionTracer`, `AuditLogger`, MCP tool server (`MCPServer`)                      |
+| Area             | What you get                                                                                                                                   |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Chunking**     | `ChunkingEngine` with sentence, fixed-size, recursive, semantic and structure-aware strategies                                                 |
+| **Retrieval**    | `MemoryRetriever`, persistent `FileVectorStore`, `BM25Search`, `HybridRetriever` (Reciprocal Rank Fusion)                                      |
+| **Reranking**    | `ScoringReranker`, `EmbeddingReranker` (cached), `CascadeReranker`                                                                             |
+| **Citations**    | Per-sentence source mapping, hallucination report, numeric-consistency check                                                                   |
+| **Evaluation**   | Faithfulness, relevance, context precision, context recall (with a reference), groundedness, BLEU/ROUGE, semantic similarity with any embedder |
+| **Integrations** | `checkAnswer` for answers from LangChain, LlamaIndex, Mastra or the AI SDK; `createAiSdkAdapter`                                               |
+| **Caching**      | Opt-in caches for query embeddings, retrieval and answers, in memory (`MemoryCache`) or Redis (`RedisCache`)                                   |
+| **Testing**      | `/testing` entry: offline `HashEmbedder`, `EchoLLM`, fixtures, `createTestPipeline()`; `validatePlugin` for plugin tests                       |
+| **Guardrails**   | Injection detection with input normalisation, ACL and relevance filtering before generation, PII redaction                                     |
+| **Agentic**      | `QueryPlanner`, `IterativeRetriever`, `SelfCritiqueChecker`, `AgenticPipeline`                                                                 |
+| **GraphRAG**     | `KnowledgeGraph`, `EntityExtractor`, `GraphRetriever`                                                                                          |
+| **Streaming**    | Token streaming with a final chunk carrying citations and evaluation; `SSEAdapter`, `WebSocketAdapter`                                         |
+| **Cost**         | `CostTracker`, `TokenBudget`, `CostCalculator` with sourced, overridable pricing                                                               |
+| **Operations**   | `DevServer` HTTP API, `ExecutionTracer`, `AuditLogger`, MCP tool server (`MCPServer`)                                                          |
 
 ## How it compares
 
@@ -422,17 +496,18 @@ A short summary of the [comparison page](https://devilsdev.github.io/rag-pipelin
 | Stability        | Documented [SEMVER policy](SEMVER.md) with deprecations | Versioned per package                 | Versioned                            |
 
 Choose LangChain.js for the widest integration catalogue or work that goes well beyond RAG, and LlamaIndex.TS for rich index types and structured data sources. Choose this library when evaluation, citations and guardrails are requirements and you want a small surface to review.
-3
 
 ## Connectors and plugins
 
-| Connector            | Embeddings | Generation | Notes                                      |
-| -------------------- | :--------: | :--------: | ------------------------------------------ |
-| `OpenAIConnector`    |     ✓      |     ✓      | `OPENAI_API_KEY`                           |
-| `AnthropicConnector` |            |     ✓      | `ANTHROPIC_API_KEY`                        |
-| `CohereConnector`    |     ✓      |            | Embeddings and reranking, `COHERE_API_KEY` |
-| `OllamaConnector`    |     ✓      |     ✓      | Local models, no key                       |
-| `LocalEmbedder`      |     ✓      |            | Offline TF-IDF, fitted to your corpus      |
+| Connector            | Embeddings | Generation | Notes                                                                                    |
+| -------------------- | :--------: | :--------: | ---------------------------------------------------------------------------------------- |
+| `OpenAIConnector`    |     ✓      |     ✓      | `OPENAI_API_KEY`; `baseURL` for OpenAI-compatible APIs (Groq, Together, vLLM, LM Studio) |
+| `AnthropicConnector` |            |     ✓      | `ANTHROPIC_API_KEY`                                                                      |
+| `CohereConnector`    |     ✓      |            | Embeddings and reranking, `COHERE_API_KEY`                                               |
+| `OllamaConnector`    |     ✓      |     ✓      | Local models, no key                                                                     |
+| `LocalEmbedder`      |     ✓      |            | Offline TF-IDF, fitted to your corpus                                                    |
+
+Vercel AI SDK models (any provider it supports) work as the embedder and LLM through `createAiSdkAdapter`; see [Use with other frameworks](https://devilsdev.github.io/rag-pipeline-utils/docs/Integrations).
 
 Connectors read API keys when they are created, retry only transient failures (network errors, 429, 5xx) with backoff, and send retrieved documents to the model as clearly delimited, untrusted reference data rather than as instructions.
 
@@ -451,7 +526,7 @@ Simulated or incomplete modules live under a separate entry point and are not co
 import { TenantManager } from "@devilsdev/rag-pipeline-utils/experimental";
 ```
 
-They are still exported from the main entry in 2.5 with a one-time deprecation warning, and will be removed from it in 3.0. Do not use them for authentication, tenant isolation or compliance. The [Experimental modules](https://devilsdev.github.io/rag-pipeline-utils/docs/Experimental) page describes what each one really does.
+They are still exported from the main entry in 2.x with a one-time deprecation warning, and will be removed from it in 3.0. Do not use them for authentication, tenant isolation or compliance. The [Experimental modules](https://devilsdev.github.io/rag-pipeline-utils/docs/Experimental) page describes what each one really does.
 
 ## Compatibility
 
@@ -469,7 +544,7 @@ They are still exported from the main entry in 2.5 with a one-time deprecation w
 - [CLI reference](https://devilsdev.github.io/rag-pipeline-utils/docs/CLI)
 - [API reference](https://devilsdev.github.io/rag-pipeline-utils/docs/API-Reference)
 - [Evaluation guide](https://devilsdev.github.io/rag-pipeline-utils/docs/Evaluation)
-- [Examples](examples/): basic pipeline, OpenAI with Pinecone, and a deployable Fastify service
+- [Examples](examples/): basic pipeline, OpenAI with Pinecone, and a deployable Fastify service ([run the basic pipeline in your browser](https://stackblitz.com/github/DevilsDev/rag-pipeline-utils/tree/main/examples/basic-pipeline))
 - [Changelog](CHANGELOG.md) and [benchmarks methodology](BENCHMARKS.md)
 
 ## Security