@devilsdev/rag-pipeline-utils 2.6.2 → 2.7.0

package/BENCHMARKS.md

@@ -2,7 +2,13 @@
 
 This document describes **how we measure** performance in
 `@devilsdev/rag-pipeline-utils` and **how you reproduce the numbers**
-yourself. It deliberately does not publish a headline "N ops/sec" figure.
+yourself.
+
+**Published results:** pipeline overhead and vector store latency and
+recall (`benchmarks/speed.js`, run in CI against pgvector and Qdrant) are
+on the [Benchmarks page](https://devilsdev.github.io/rag-pipeline-utils/docs/Benchmarks),
+generated from `benchmarks/results/`. The rest of this document covers the
+engine micro-benchmark suite and benchmarking your own stack.
 
 ## Philosophy
 

package/CHANGELOG.md

@@ -11,6 +11,126 @@ page.
 
 ## [Unreleased]
 
+## [2.7.0] - 2026-10-30
+
+### Added
+
+- **`validateDataset(dataset, { requireExpected, requireSourceIds })`**
+  checks an evaluation test set before it runs and reports every problem
+  with its case index, id and field, without throwing. A dataset is an
+  array of `{ prompt, expected, sourceIds, id, tags }` cases or
+  `{ cases: [...] }`; `question` and `referenceAnswer` are accepted as
+  aliases, and the `{ prompt, expected }` shape `evaluateRagDataset` reads
+  is valid unchanged. Valid datasets come back normalised in `cases`.
+- **`runEvaluation(dataset, pipeline)`** runs every case with citations and
+  evaluation on and returns a JSON report: per-case answers, retrieved ids,
+  scores (faithfulness, relevance, context precision and recall,
+  groundedness, citation coverage, and `sourceRecall` against the case's
+  `sourceIds`) and per-metric means. Failed cases are recorded, not thrown;
+  options for `concurrency`, a per-case `timeout`, `signal` and `onCase`.
+  Reports record the `libraryVersion`, and `compareConfigurations` warns
+  when two runs used different versions, since scoring can change between
+  releases.
+- **`compareConfigurations(dataset, { baseline, candidate })`** compares two
+  runs, where each side is a pipeline or a saved report, and reports
+  `regressed`, per-metric deltas, and the cases that now fail or dropped by
+  more than `caseTolerance`. `tolerance` (default 0.02) sets the allowed
+  drop in a metric's mean.
+- **`generateSyntheticDataset(documents, { llm, count })`** drafts a test set:
+  an LLM writes question and answer pairs about one document at a time, and
+  each case records the document in `sourceIds`. Answers their document
+  doesn't support (by `computeFaithfulness`), repeated questions, malformed
+  replies and short documents are skipped and listed with a reason. Cases
+  are tagged `synthetic` and marked `reviewed: false`.
+- **`rag-pipeline generate-dataset --output <file>`** does the same from the
+  index with the configured `llm`, and won't overwrite a file without
+  `--force`.
+- **`PgVectorRetriever`** stores and searches documents in PostgreSQL with
+  the pgvector extension: cosine similarity, upserts in one transaction, an
+  HNSW index with an `efSearch` option for recall, and `removeBySource`, `delete`, `clear`, `size` and `list` for
+  the CLI. Pass a `pg` Pool, or a connection string with `pg` installed (an
+  optional peer dependency, loaded only when used). In `.ragrc.json` it is
+  the `pgvector` retriever and reads its connection string from
+  `DATABASE_URL`. It isn't cached unless you give it a `cacheKey`.
+- **`QdrantRetriever`** stores and searches documents in a Qdrant
+  collection through its REST API, with no client package. Document ids map
+  to stable UUIDs, with the original id kept in the payload; writes wait
+  until Qdrant applies them. In `.ragrc.json` it is the `qdrant` retriever,
+  reading `QDRANT_URL` and `QDRANT_API_KEY`. Tested against Qdrant 1.20.
+- **`rag-pipeline eval <dataset>`** runs a `.json` or `.jsonl` test set
+  through the configured pipeline and exits 1 when a case fails, a metric
+  is below its `--min metric=value`, or the run regressed from a
+  `--baseline` report. `--output` saves the run as a baseline, `--report`
+  writes a standalone HTML page (`.html`) or a Markdown summary for a pull
+  request comment, and `--json` prints the verdict. Also `--tolerance`,
+  `--case-tolerance`, `--concurrency`, `--timeout` and `--top-k`.
+- **[`DevilsDev/rag-eval-action`](https://github.com/DevilsDev/rag-eval-action)**,
+  a GitHub Action that runs `rag-pipeline eval` on pull requests, compares
+  them with the default branch's report from the Actions cache, posts a
+  summary comment that later runs update, and fails the check when scores
+  drop. Its `v1` is published with this release.
+- **`renderEvaluationReport`** (Markdown or HTML) and
+  **`evaluationVerdict`** for the same reports in code.
+- `sourceIds` match a retrieved result's `id` or its `metadata.source`, so
+  a case can name a file when the index stores chunks of it.
+- **A generated API reference** on the docs site: a page for every export,
+  with its methods, options and types, built from the published type
+  declarations by `npm run docs:api`. CI fails when it is out of date.
+- **Published speed benchmarks** on the docs site: pipeline overhead, and
+  ingest throughput, query latency and recall@10 for `MemoryRetriever`,
+  `FileVectorStore`, `PgVectorRetriever` and `QdrantRetriever`, measured in
+  CI against real pgvector and Qdrant (`npm run bench:speed`), and
+  retrieval quality (nDCG@10, recall, MRR) of the offline retrievers on
+  BEIR SciFact (`npm run bench:quality`), where `BM25Search` matches the
+  published BM25 baseline. The tables are generated from the committed
+  reports in `benchmarks/results/`.
+- `logger.info()`, `.warn()`, `.error()` and `.debug()` log through the
+  shared logger. The `logger` export is the logger module, so these were
+  undefined; `logger.logger.info()` keeps working.
+
+### Fixed
+
+- **`LocalEmbedder` ignored most words on a large corpus.** When a corpus
+  had more distinct tokens than `maxVocabularySize` (10,000), `train()`
+  kept the rarest ones, mostly words that occur in a single document, so
+  most queries embedded to all zeros. It now keeps the most common tokens.
+  On BEIR SciFact nDCG@10 rose from 0.12 to 0.47. Indexes built before
+  keep their saved vocabulary until you ingest again; re-run `ingest` to
+  benefit.
+- **The TypeScript declarations for the core API didn't match the code.**
+  Plugins written to them failed at runtime. `RetrieverPlugin.retrieve`
+  takes one `{ query, queryVector, topK, signal }` object,
+  `RerankerPlugin.rerank` takes `(query, documents)`, and `LLMPlugin.generate`
+  takes `(query, documents, options)` and must return the answer text: an
+  `LLMResponse` object became the answer `"[object Object]"`.
+  `PipelineRunResult` now has `answer`, `citations` and `evaluation`, and
+  `run()` with `options: { stream: true }` is typed to resolve a stream.
+  The
+  declarations of `loadConfig` (synchronous, takes a path), `validateRagrc`,
+  `normalizeConfig`, `pluginRegistry`, `createError` and `wrapError`,
+  `computeBLEU`, `computeROUGE` and `scoreAnswer` (token arrays,
+  `(actual, expected)`), `retry`, `withRetry`, `RateLimiter`,
+  `BatchProcessor`, `ConnectionPoolManager`, `JWTValidator`,
+  `PluginSignatureVerifier`, `validateInput`, `DAG`, `evaluateRagDataset`,
+  `HotReloadManager`, `DevServer`, `AuditLogger`, `PluginHub`, `metrics`,
+  `eventLogger` and `ParallelProcessor` are now generated from the source,
+  so they describe what each one does. A call that matched an old
+  declaration but not the code, such as `rerank(results, query)` or
+  `await loadConfig({ configPath })`, may no longer compile; it didn't
+  work when run.
+  `LLMResponse`, `LoadConfigOptions`, `DAGNode`, `DAGOptions`,
+  `ErrorFormatter` and `EvalResult` are deprecated.
+
+- **The CLI didn't wait for retrievers whose methods are async**, as a
+  remote vector store's are: `ingest` didn't wait for `removeBySource`, so
+  re-ingesting a file could keep its old chunks or delete the new ones, and `generate-dataset`,
+  `ai feedback` and `ai benchmark` failed on `list()`. Every retriever call
+  is now awaited. The fitted `local` embedder with a store that can't save
+  its state now stops with a message instead of embedding queries with an
+  empty vocabulary.
+- A bad option value (`--top-k 0`, `--min speed=1`) printed a stack trace;
+  the CLI now prints a one-line error naming the option and exits 1.
+
 ## [2.6.2] - 2026-10-29
 
 ### Fixed
@@ -580,6 +700,7 @@ First public release on the 2.4 line, introducing the v2.4 feature set.
 For release notes prior to v2.4.0, see
 [github.com/DevilsDev/rag-pipeline-utils/releases](https://github.com/DevilsDev/rag-pipeline-utils/releases).
 
+[2.7.0]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.7.0
 [2.6.2]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.6.2
 [2.6.1]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.6.1
 [2.6.0]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.6.0

package/README.md

@@ -143,6 +143,8 @@ npx rag-pipeline ingest ./docs             # indexes .md, .html, .csv and .txt f
 npx rag-pipeline query "What is the remote work policy?"
 npx rag-pipeline serve --port 3000         # local HTTP API: GET /health, POST /query
 npx rag-pipeline doctor                    # checks config, API keys, plugins and the index
+npx rag-pipeline generate-dataset --output tests.json     # drafts a test set from the index with your llm
+npx rag-pipeline eval tests.json --min faithfulness=0.8   # scores a test set; exits 1 on failure
 ```
 
 The default `local` template needs no API keys. Use `init --template openai`, `anthropic` or `ollama` to generate answers with a model. Other commands include `validate`, `hub install` for plugins and `ai benchmark` for measured latency.
@@ -208,6 +210,10 @@ console.log(report.scores); // includes contextRecall because a reference was gi
 if (report.scores.faithfulness < 0.8) process.exitCode = 1;
 ```
 
+Keep the question set in a JSON file of `{ prompt, expected, sourceIds }` cases; `generateSyntheticDataset` (or `rag-pipeline generate-dataset`) drafts one from your documents for you to review. `validateDataset` checks it before a run and reports every malformed case, with its index and field; `runEvaluation` runs it and saves a JSON report, which `renderEvaluationReport` turns into Markdown or HTML; and `compareConfigurations` compares two pipelines, or a pipeline against the report saved from your main branch, and lists the cases that got worse. See the [dataset format](https://devilsdev.github.io/rag-pipeline-utils/docs/Evaluation#dataset-format).
+
+On GitHub, [`DevilsDev/rag-eval-action`](https://github.com/DevilsDev/rag-eval-action) runs this on every pull request, compares it with your main branch and posts the summary as a comment.
+
 ### Check answers from another framework
 
 Already on LangChain, LlamaIndex, Mastra or the AI SDK? `checkAnswer` checks the answer your chain produced against the sources it was given, in the shape your framework returns them:
@@ -339,6 +345,8 @@ const results = await hybrid.retrieve({
 console.log(results[0].id); // "err-4012"
 ```
 
+On the BEIR SciFact benchmark, `BM25Search` alone matches the published BM25 baseline (nDCG@10 0.665); fusion helps when the vector side is a strong embedder, so weight the retrievers to suit. See [retrieval quality](https://devilsdev.github.io/rag-pipeline-utils/docs/Benchmarks#retrieval-quality).
+
 ### Answer multi-part questions
 
 `AgenticPipeline` splits compound questions into sub-queries, retrieves for each, and self-checks the answer against the sources before returning it.
@@ -464,22 +472,22 @@ The result of `run()` is:
 
 ## Capabilities
 
-| Area             | What you get                                                                                                                                   |
-| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Chunking**     | `ChunkingEngine` with sentence, fixed-size, recursive, semantic and structure-aware strategies                                                 |
-| **Retrieval**    | `MemoryRetriever`, persistent `FileVectorStore`, `BM25Search`, `HybridRetriever` (Reciprocal Rank Fusion)                                      |
-| **Reranking**    | `ScoringReranker`, `EmbeddingReranker` (cached), `CascadeReranker`                                                                             |
-| **Citations**    | Per-sentence source mapping, hallucination report, numeric-consistency check                                                                   |
-| **Evaluation**   | Faithfulness, relevance, context precision, context recall (with a reference), groundedness, BLEU/ROUGE, semantic similarity with any embedder |
-| **Integrations** | `checkAnswer` for answers from LangChain, LlamaIndex, Mastra or the AI SDK; `createAiSdkAdapter`                                               |
-| **Caching**      | Opt-in caches for query embeddings, retrieval and answers, in memory (`MemoryCache`) or Redis (`RedisCache`)                                   |
-| **Testing**      | `/testing` entry: offline `HashEmbedder`, `EchoLLM`, fixtures, `createTestPipeline()`; `validatePlugin` for plugin tests                       |
-| **Guardrails**   | Injection detection with input normalisation, ACL and relevance filtering before generation, PII redaction                                     |
-| **Agentic**      | `QueryPlanner`, `IterativeRetriever`, `SelfCritiqueChecker`, `AgenticPipeline`                                                                 |
-| **GraphRAG**     | `KnowledgeGraph`, `EntityExtractor`, `GraphRetriever`                                                                                          |
-| **Streaming**    | Token streaming with a final chunk carrying citations and evaluation; `SSEAdapter`, `WebSocketAdapter`                                         |
-| **Cost**         | `CostTracker`, `TokenBudget`, `CostCalculator` with sourced, overridable pricing                                                               |
-| **Operations**   | `DevServer` HTTP API, `ExecutionTracer`, `AuditLogger`, MCP tool server (`MCPServer`)                                                          |
+| Area             | What you get                                                                                                                                                                                                                                                                   |
+| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| **Chunking**     | `ChunkingEngine` with sentence, fixed-size, recursive, semantic and structure-aware strategies                                                                                                                                                                                 |
+| **Retrieval**    | `MemoryRetriever`, persistent `FileVectorStore`, `PgVectorRetriever` (PostgreSQL), `QdrantRetriever`, `BM25Search`, `HybridRetriever` (Reciprocal Rank Fusion)                                                                                                                 |
+| **Reranking**    | `ScoringReranker`, `EmbeddingReranker` (cached), `CascadeReranker`                                                                                                                                                                                                             |
+| **Citations**    | Per-sentence source mapping, hallucination report, numeric-consistency check                                                                                                                                                                                                   |
+| **Evaluation**   | Faithfulness, relevance, context precision, context recall (with a reference), groundedness, BLEU/ROUGE, semantic similarity with any embedder; test sets with `validateDataset`, `runEvaluation`, `compareConfigurations`, `generateSyntheticDataset` and `rag-pipeline eval` |
+| **Integrations** | `checkAnswer` for answers from LangChain, LlamaIndex, Mastra or the AI SDK; `createAiSdkAdapter`                                                                                                                                                                               |
+| **Caching**      | Opt-in caches for query embeddings, retrieval and answers, in memory (`MemoryCache`) or Redis (`RedisCache`)                                                                                                                                                                   |
+| **Testing**      | `/testing` entry: offline `HashEmbedder`, `EchoLLM`, fixtures, `createTestPipeline()`; `validatePlugin` for plugin tests                                                                                                                                                       |
+| **Guardrails**   | Injection detection with input normalisation, ACL and relevance filtering before generation, PII redaction                                                                                                                                                                     |
+| **Agentic**      | `QueryPlanner`, `IterativeRetriever`, `SelfCritiqueChecker`, `AgenticPipeline`                                                                                                                                                                                                 |
+| **GraphRAG**     | `KnowledgeGraph`, `EntityExtractor`, `GraphRetriever`                                                                                                                                                                                                                          |
+| **Streaming**    | Token streaming with a final chunk carrying citations and evaluation; `SSEAdapter`, `WebSocketAdapter`                                                                                                                                                                         |
+| **Cost**         | `CostTracker`, `TokenBudget`, `CostCalculator` with sourced, overridable pricing                                                                                                                                                                                               |
+| **Operations**   | `DevServer` HTTP API, `ExecutionTracer`, `AuditLogger`, MCP tool server (`MCPServer`)                                                                                                                                                                                          |
 
 ## How it compares
 
@@ -509,6 +517,8 @@ Choose LangChain.js for the widest integration catalogue or work that goes well
 | `OllamaConnector`    |     ✓      |     ✓      | Local models, no key                                                                     |
 | `LocalEmbedder`      |     ✓      |            | Offline TF-IDF, fitted to your corpus                                                    |
 
+`PgVectorRetriever` and `QdrantRetriever` store and search documents in PostgreSQL with pgvector and in Qdrant ([vector stores](https://devilsdev.github.io/rag-pipeline-utils/docs/Concepts-Connectors#postgresql-with-pgvector)); `efSearch` trades pgvector query speed for recall. Their ingest speed, query latency and recall, measured in CI, are on the [Benchmarks page](https://devilsdev.github.io/rag-pipeline-utils/docs/Benchmarks).
+
 Vercel AI SDK models (any provider it supports) work as the embedder and LLM through `createAiSdkAdapter`; see [Use with other frameworks](https://devilsdev.github.io/rag-pipeline-utils/docs/Integrations).
 
 Connectors read API keys when they are created, retry only transient failures (network errors, 429, 5xx) with backoff, and send retrieved documents to the model as clearly delimited, untrusted reference data rather than as instructions.
@@ -532,22 +542,23 @@ They are still exported from the main entry in 2.x with a one-time deprecation w
 
 ## Compatibility
 
-|                |                                                                           |
-| -------------- | ------------------------------------------------------------------------- |
-| Node.js        | 18, 20, 22 (Node 18 support ends with 3.0)                                |
-| Module formats | CommonJS (`require`) and ESM (`import`)                                   |
-| TypeScript     | Declarations for every export, including `/experimental`                  |
-| Versioning     | Semantic versioning with a deprecation window, see [SEMVER.md](SEMVER.md) |
-| Supply chain   | Published from GitHub Actions with npm provenance                         |
+|                |                                                                                                      |
+| -------------- | ---------------------------------------------------------------------------------------------------- |
+| Node.js        | 18, 20, 22 (Node 18 support ends with 3.0)                                                           |
+| Module formats | CommonJS (`require`) and ESM (`import`)                                                              |
+| TypeScript     | Declarations for every export, including `/experimental`, compiled and run against the package in CI |
+| Versioning     | Semantic versioning with a deprecation window, see [SEMVER.md](SEMVER.md)                            |
+| Supply chain   | Published from GitHub Actions with npm provenance                                                    |
 
 ## Documentation
 
 - [Documentation site](https://devilsdev.github.io/rag-pipeline-utils/): guides, API reference and deployment
 - [CLI reference](https://devilsdev.github.io/rag-pipeline-utils/docs/CLI)
 - [API reference](https://devilsdev.github.io/rag-pipeline-utils/docs/API-Reference)
+- [Generated API reference](https://devilsdev.github.io/rag-pipeline-utils/docs/api): every export, with its methods, options and types
 - [Evaluation guide](https://devilsdev.github.io/rag-pipeline-utils/docs/Evaluation)
 - [Examples](examples/): basic pipeline, OpenAI with Pinecone, and a deployable Fastify service ([run the basic pipeline in your browser](https://stackblitz.com/github/DevilsDev/rag-pipeline-utils/tree/main/examples/basic-pipeline))
-- [Changelog](CHANGELOG.md) and [benchmarks methodology](BENCHMARKS.md)
+- [Changelog](CHANGELOG.md), [benchmark results](https://devilsdev.github.io/rag-pipeline-utils/docs/Benchmarks) and [methodology](BENCHMARKS.md)
 
 ## Security