@devilsdev/rag-pipeline-utils 2.4.3 → 2.4.5

package/BENCHMARKS.md

@@ -0,0 +1,338 @@
+# Benchmarks
+
+This document describes **how we measure** performance in
+`@devilsdev/rag-pipeline-utils` and **how you reproduce the numbers**
+yourself. It deliberately does not publish a headline "N ops/sec" figure.
+
+## Philosophy
+
+Benchmark numbers that are not reproducible are marketing, not engineering.
+RAG pipelines are particularly sensitive to factors outside the framework:
+embedding model choice, vector store, LLM provider and region, corpus
+size, query distribution, and hardware. A single number hides all of
+that.
+
+Instead, we ship:
+
+- A **deterministic, in-process benchmark suite** that measures framework
+  overhead — DAG execution, retry policies, logging, memory patterns.
+  These numbers are comparable across runs on the same hardware and
+  make regressions easy to catch.
+- A **methodology and reference harness** that lets you benchmark your
+  own end-to-end pipeline with your actual provider, data, and hardware.
+  These are the only numbers that matter for your workload.
+
+When you need a headline number for a decision, generate it on the
+system you'll deploy to, with the providers you'll actually use.
+
+## What the suite measures
+
+The built-in suite (`scripts/benchmark.js`) exercises **framework
+overhead only**. Every benchmark is in-process, network-free, and
+deterministic up to GC and OS jitter.
+
+| Category                                              | What it measures                                          | Why it matters                                         |
+| ----------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------ |
+| **DAG execution** (5 / 10 / 25 / 50 nodes)            | Linear chain of async nodes, cold and warm                | Baseline overhead of the pipeline orchestrator         |
+| **Concurrent DAG** (1 / 2 / 5 / 10 parallelism)       | Tree of 20 nodes with configurable concurrency            | How well the engine scales intra-query parallelism     |
+| **Retry policies** (none / conservative / aggressive) | Simulated 30% failure rate, retries with jittered backoff | Cost of reliability guarantees on the happy path       |
+| **Structured logging** (JSON / console / correlated)  | Single log call with contextual payload                   | Per-log-line overhead when observability is on         |
+| **Memory allocation**                                 | Array + Map + Set construction and processing             | Establishes a memory-pressure floor to compare against |
+| **CPU intensive**                                     | Fibonacci + JSON round-trip + string ops                  | Sanity check against the hosting CPU                   |
+
+Each benchmark reports:
+
+- `mean`, `median`, `min`, `max` latency (ms)
+- `p95`, `p99` latency (ms)
+- `standardDeviation` (ms)
+- `throughput` (ops/sec)
+- `memory.avgDeltaHeapUsed`, `memory.peakHeapUsed`
+- `successRate` (%) and `errors` count
+
+All measurements include a warmup phase (default 10 iterations) and use
+`perf_hooks.performance.now()` for timing. If the Node process is
+started with `--expose-gc`, the suite forces a GC before each run.
+
+## What the suite does **not** measure
+
+The built-in suite does not exercise:
+
+- **Embedder throughput** — provider- and model-dependent
+- **Retrieval latency** — vector store, index size, and network bound
+- **LLM inference latency** — model, region, and token-count bound
+- **End-to-end query latency** — dominated by the three above
+- **Cost per query** — provider pricing + token usage
+
+These are intentional omissions. Claiming a number here without
+disclosing the full stack would be misleading. Use the
+**end-to-end harness template** below to measure them against your
+actual stack.
+
+## Running the suite
+
+Three presets ship in `package.json`:
+
+```bash
+# 20 iterations per benchmark — fast sanity check (~30s)
+npm run benchmark:quick
+
+# 100 iterations, verbose — default (~2 min)
+npm run benchmark
+
+# 500 iterations, verbose — release-gate quality (~8 min)
+npm run benchmark:full
+```
+
+Direct invocation:
+
+```bash
+node scripts/benchmark.js \
+  --iterations 200 \
+  --output ./my-results \
+  --verbose
+
+# With deterministic GC
+node --expose-gc scripts/benchmark.js --iterations 200
+```
+
+Results are written to `./benchmark-results/`:
+
+- `latest.json` — most recent run
+- `benchmark-<timestamp>.json` — archived run
+
+A human-readable summary is printed to stdout on completion.
+
+## Environment disclosure template
+
+When sharing benchmark numbers (issue reports, blog posts, PR descriptions),
+include the following so they're interpretable:
+
+```text
+Package:      @devilsdev/rag-pipeline-utils@<version>
+Node:         <version> (v8 <version>)
+OS:           <distro/version>, kernel <version>
+CPU:          <model>, <cores> cores, <base/boost GHz>
+Memory:       <total GB>
+Storage:      <NVMe/SSD/HDD>, <filesystem>
+Container:    <bare-metal / Docker / K8s / Lambda / etc.>
+Providers:    embedder=<…>, retriever=<…>, llm=<…>
+Corpus:       <N documents, total tokens, chunk size>
+Query set:    <N queries, source>
+Iterations:   <N>, warmup: <N>
+GC control:   --expose-gc=<yes/no>
+```
+
+The JSON report already captures `nodeVersion`, `platform`, `arch`,
+`cpus`, `totalMemory`, and `loadAverage` automatically in its
+`environment` block.
+
+## Report schema
+
+Each run produces a JSON document of the shape:
+
+```json
+{
+  "timestamp": "2026-04-16T22:00:00.000Z",
+  "environment": {
+    "nodeVersion": "v20.x.x",
+    "platform": "linux",
+    "arch": "x64",
+    "cpus": 8,
+    "totalMemory": "16384MB",
+    "loadAverage": [0.12, 0.08, 0.05]
+  },
+  "benchmarks": {
+    "DAG execution (10 nodes)": {
+      "iterations": 100,
+      "errors": 0,
+      "successRate": 100,
+      "timing": {
+        "mean": 0.0,
+        "median": 0.0,
+        "min": 0.0,
+        "max": 0.0,
+        "p95": 0.0,
+        "p99": 0.0,
+        "standardDeviation": 0.0
+      },
+      "memory": {
+        "avgDeltaHeapUsed": "0KB",
+        "totalHeapDelta": "0KB",
+        "peakHeapUsed": "0KB"
+      },
+      "throughput": 0.0
+    }
+  },
+  "meta": {
+    "totalDuration": "0ms",
+    "totalBenchmarks": 0,
+    "completedAt": "2026-04-16T22:00:00.000Z"
+  }
+}
+```
+
+The numeric fields are intentionally left at `0.0` in this schema —
+fill them by running the suite.
+
+## End-to-end benchmarking your pipeline
+
+The harness below is the minimal template for honest end-to-end numbers.
+Adapt it to your pipeline, provider mix, and query distribution.
+
+```js
+// scripts/e2e-benchmark.mjs
+import { performance } from "node:perf_hooks";
+import { writeFileSync } from "node:fs";
+import { createRagPipeline } from "@devilsdev/rag-pipeline-utils";
+
+const pipeline = createRagPipeline({
+  /* your real config — same as production */
+});
+
+const queries = [
+  /* your real query set — at least 50 queries drawn from production logs */
+];
+
+const WARMUP = 10;
+const ITERATIONS = 100;
+
+async function run(query) {
+  const t0 = performance.now();
+  const result = await pipeline.run({ query });
+  const t1 = performance.now();
+  return {
+    ms: t1 - t0,
+    retrievedCount: result.results?.length ?? 0,
+    answerChars: result.answer?.length ?? 0,
+    tokens: result.usage?.totalTokens ?? null,
+  };
+}
+
+// Warmup
+for (let i = 0; i < WARMUP; i++) {
+  await run(queries[i % queries.length]);
+}
+
+// Measured run
+const samples = [];
+for (let i = 0; i < ITERATIONS; i++) {
+  samples.push(await run(queries[i % queries.length]));
+}
+
+const ms = samples.map((s) => s.ms).sort((a, b) => a - b);
+const percentile = (p) => ms[Math.floor(ms.length * p)];
+
+const report = {
+  timestamp: new Date().toISOString(),
+  iterations: ITERATIONS,
+  warmup: WARMUP,
+  latencyMs: {
+    mean: ms.reduce((a, b) => a + b, 0) / ms.length,
+    p50: percentile(0.5),
+    p95: percentile(0.95),
+    p99: percentile(0.99),
+    min: ms[0],
+    max: ms[ms.length - 1],
+  },
+  avgTokens: samples.reduce((a, b) => a + (b.tokens ?? 0), 0) / samples.length,
+  avgRetrieved:
+    samples.reduce((a, b) => a + b.retrievedCount, 0) / samples.length,
+};
+
+writeFileSync("e2e-benchmark.json", JSON.stringify(report, null, 2));
+console.table(report.latencyMs);
+```
+
+Run it the same way on every comparison — same corpus, same query set,
+same providers, same hardware. Don't compare a warm run to a cold one.
+
+## Interpreting results
+
+### Timing percentiles
+
+- **mean** is easy to move around by outliers; don't report it alone
+- **p50** (median) is the "typical" experience
+- **p95** is the tail a user will notice
+- **p99** is where GC, cold caches, and network hiccups live — useful
+  for SLO design
+- **standardDeviation** much larger than the mean signals high variance;
+  your benchmark is probably too short, too concurrent, or sharing a
+  noisy host
+
+### Throughput
+
+- Throughput is computed as `iterations / (mean_seconds)`. It's
+  single-process throughput with the harness's concurrency settings —
+  not a statement about your production service's throughput under a
+  real load mix.
+- If you want realistic multi-tenant throughput numbers, build a
+  load-test harness (`autocannon`, `k6`, `artillery`) that hits the
+  HTTP surface you actually deploy, not the library directly.
+
+### Memory
+
+- `avgDeltaHeapUsed` is the _growth per iteration_. Trending up across
+  a long run means you have a leak.
+- `peakHeapUsed` tells you the upper bound for right-sizing containers.
+- On Windows or containerized Linux, RSS is not the same as heap used;
+  prefer platform metrics for capacity planning.
+
+### Common pitfalls
+
+1. **No warmup** — first-iteration numbers are dominated by JIT and
+   cold caches. Always discard a warmup window.
+2. **Too few iterations** — 10 iterations on a ~1ms operation is mostly
+   noise. 100+ is a reasonable floor for framework benchmarks;
+   end-to-end benchmarks need enough iterations to cover your query
+   distribution.
+3. **Correlated runs** — running the suite twice back-to-back on a
+   laptop will show the second run as faster because caches are warm.
+   For comparisons, use fresh processes and cold caches, or explicitly
+   measure warm paths.
+4. **Shared hosts** — CI runners, personal machines with open browsers,
+   and containers on bin-packed hosts all have neighbors that will add
+   noise. Dedicated bare metal or bare VMs give the cleanest numbers.
+5. **Stream-based workloads** — retrieval latency is not end-to-end
+   latency when the LLM streams tokens. Measure time-to-first-token
+   (TTFT) separately from total completion time.
+
+## Regression testing in CI
+
+The benchmark suite is deterministic enough to catch gross regressions
+in CI. Recommended pattern:
+
+1. Run `npm run benchmark:quick` on every PR against the default branch
+2. Compare `latest.json` to the baseline in `main` using the keys in
+   `benchmarks.*.timing.mean`
+3. Fail the gate if any benchmark's mean regresses by more than a
+   disclosed threshold (e.g. 25%)
+
+This project does not yet gate on benchmark regressions in CI by default
+— numbers on shared GitHub runners are too noisy to trust automatically.
+If you run the suite on dedicated hardware as part of a release gate,
+we'd welcome the workflow contribution.
+
+## Contributing benchmark data
+
+If you run the suite on a representative production-class host and want
+to share the results:
+
+1. Open a PR adding your report to `benchmark-results/community/` as
+   `<hostname-slug>-<date>.json`
+2. Include the full environment disclosure block above in the PR
+   description
+3. We will not average, normalize, or rank contributed numbers — they
+   remain raw data points for other users comparing against similar
+   hardware profiles
+
+Community benchmark data is licensed under CC0 1.0 (public domain)
+when contributed.
+
+## See also
+
+- [Performance guide](docs-site/docs/Performance.md) — optimization
+  strategies by component (embedding, retrieval, LLM, memory)
+- [`scripts/benchmark.js`](scripts/benchmark.js) — the benchmark
+  harness source
+- [SECURITY.md](SECURITY.md) — security hardening, not performance,
+  but often on the same reviewer checklist

package/CHANGELOG.md

@@ -9,6 +9,52 @@ Older history (pre-2.4.0) is preserved in the
 [GitHub releases](https://github.com/DevilsDev/rag-pipeline-utils/releases)
 page.
 
+## [2.4.4] - 2026-04-16
+
+### Added
+
+- **`BENCHMARKS.md`** (root, ships in tarball) — methodology-first
+  benchmarks doc covering what we measure, what we don't, how to
+  run the suite, environment disclosure template, JSON report schema,
+  end-to-end harness template for benchmarking your own stack,
+  interpretation guide (percentiles, throughput, memory), common
+  pitfalls, and a CI regression pattern.
+- **`SEMVER.md`** (root, ships in tarball) — API stability policy.
+  Defines public surface, what counts as breaking, deprecation
+  process (one-minor warning window minimum), supported runtimes,
+  TypeScript and ESM/CJS commitments.
+- **`docs-site/docs/Security-Capabilities.md`** — capability matrix
+  distinguishing what's battle-tested (🟢) vs. recommended practice
+  (🟡) vs. example code requiring hardening (🟠) vs. out of scope
+  (🔴), across pre-retrieval, retrieval-time, and post-generation
+  guardrails plus supply-chain, auth, logging, input validation, and
+  data governance.
+- **`docs-site/docs/Concepts-Core.md`**,
+  `Concepts-Connectors.md`, `Concepts-Evaluation.md`,
+  `Concepts-Guardrails.md` — four conceptual-module overviews mapped
+  to a future `rag-core` / `rag-connectors` / `rag-eval` /
+  `rag-guardrails` package boundary, so reviewers can reason about
+  blast radius without us splitting the codebase.
+- **`docs-site/docs/Comparison.md`** — honest comparison vs.
+  LangChain.js and LlamaIndex.TS, including where we lose and a
+  decision framework. Currency dated to April 2026.
+- **`examples/fastify-rag-demo/`** — runnable end-to-end RAG service:
+  Fastify 5 + OpenAI + in-memory retriever + 3-layer guardrails
+  - citations + evaluation. Ships with Dockerfile, Fly.io `fly.toml`,
+    Railway `railway.json`, ingest + smoke-test scripts, and a complete
+    README. ~150 lines of `server.js`.
+- **`examples/README.md`** — index of runnable examples with
+  contribution guidance.
+
+### Changed
+
+- README "Documentation" section now links to BENCHMARKS, SECURITY,
+  CHANGELOG (added in 2.4.3) plus SEMVER, the Security Capabilities
+  matrix, the Comparison page, and the runnable demo.
+- Sidebar adds two new categories: "Concepts" (4 pages) and a
+  Comparison page under Overview. Security-Capabilities slotted under
+  Enterprise.
+
 ## [2.4.3] - 2026-04-16
 
 ### Security
@@ -78,6 +124,7 @@ First public release on the 2.4 line, introducing the v2.4 feature set.
 For release notes prior to v2.4.0, see
 [github.com/DevilsDev/rag-pipeline-utils/releases](https://github.com/DevilsDev/rag-pipeline-utils/releases).
 
+[2.4.4]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.4
 [2.4.3]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.3
 [2.4.2]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.2
 [2.4.1]: https://github.com/DevilsDev/rag-pipeline-utils/releases/tag/v2.4.1

package/README.md

@@ -157,6 +157,13 @@ import {
 - **[Architecture](https://devilsdev.github.io/rag-pipeline-utils/docs/Architecture)** — plugin contracts, pipeline internals
 - **[API reference](https://devilsdev.github.io/rag-pipeline-utils/docs/API-Reference)** — complete export index
 - **[Examples](https://devilsdev.github.io/rag-pipeline-utils/docs/Examples)** — chatbot, Q&A, code search, support
+- **[Comparison vs LangChain.js / LlamaIndex.TS](https://devilsdev.github.io/rag-pipeline-utils/docs/Comparison)** — honest decision framework
+- **[Runnable demo](examples/fastify-rag-demo)** — Fastify + OpenAI + guardrails, deployable in one command
+- **[Benchmarks & methodology](BENCHMARKS.md)** — how we measure, how you reproduce
+- **[Security policy](SECURITY.md)** — responsible disclosure, supported versions
+- **[Security capability matrix](https://devilsdev.github.io/rag-pipeline-utils/docs/Security-Capabilities)** — what's battle-tested vs. recommended vs. example
+- **[API stability policy (SEMVER.md)](SEMVER.md)** — what counts as breaking, deprecation windows
+- **[Changelog](CHANGELOG.md)** — release history
 
 ## Roadmap
 

package/SEMVER.md

@@ -0,0 +1,184 @@
+# API Stability Policy
+
+This document defines what counts as a breaking change for
+`@devilsdev/rag-pipeline-utils`, how long each release line is
+supported, and the deprecation process for public API surface.
+
+It is the contract you can plan against.
+
+## Versioning
+
+We follow [Semantic Versioning 2.0](https://semver.org/spec/v2.0.0.html):
+
+| Release           | When it lands                                                                 | What changes                                                                |
+| ----------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
+| **Patch** (2.4.x) | Bug fixes, security patches, docs, internal refactors                         | Nothing public-facing breaks. Safe to adopt without reading release notes.  |
+| **Minor** (2.x.0) | New features, additive API surface, new plugins, non-breaking behavior tweaks | Review the [CHANGELOG](CHANGELOG.md) before upgrading. Backward compatible. |
+| **Major** (x.0.0) | Breaking changes to public API or required Node versions                      | Requires active migration. Each major ships with a migration guide.         |
+
+Pre-release versions (`x.y.z-rc.N`, `-beta.N`, `-alpha.N`) are for
+testing. They have no stability guarantees and may change or disappear
+before general availability.
+
+## Public API surface
+
+The **public API surface** — everything covered by SemVer — is
+everything reachable from the package entry point:
+
+```js
+import { ... } from "@devilsdev/rag-pipeline-utils";
+```
+
+Plus the CLI (`npx rag-pipeline …`) and the plugin contract schemas
+in `contracts/`.
+
+The following are **not** part of the public API surface and may
+change in any release, including patches:
+
+- Any import path with `/internal/` in it
+- Default export behavior of submodules not documented in the
+  [API reference](https://devilsdev.github.io/rag-pipeline-utils/docs/API-Reference)
+- Imports reaching into `dist/` or `src/` directly (e.g.
+  `@devilsdev/rag-pipeline-utils/src/core/dag/executor.js`)
+- Behavior of items marked `@internal` in JSDoc
+- Behavior marked **experimental** in release notes or JSDoc
+- The on-disk format of internal caches, logs, or temporary files
+- Exact wording of log messages and error `message` strings
+- Node version support _below_ the documented minimum (see "Supported
+  runtime" below)
+
+If you depend on any of the above, you accept that patch-level upgrades
+may break your integration.
+
+## Breaking changes
+
+A **breaking change** is any change that would force a user of the
+public API surface to modify their code, configuration, or deployment.
+Concretely:
+
+- Removing or renaming a public export
+- Removing a method or property on a public class
+- Changing the signature of a public method in a way existing callers
+  can't satisfy (added required parameter, removed optional parameter
+  type, etc.)
+- Changing the shape of a public configuration object
+- Tightening a type that was previously accepting
+- Changing the default value of a public option when the new default
+  produces different behavior
+- Throwing a new error class from a public method where callers
+  previously got a different class or a string
+- Raising the minimum Node.js version
+- Raising the minimum TypeScript version for typed consumers
+- Removing support for a bundler or module format (ESM/CJS)
+
+The following are **not** breaking changes (they ship in minors and
+patches freely):
+
+- Adding new exports, methods, or options
+- Relaxing a type that was previously restrictive
+- Widening an accepted input (e.g. accepting `Buffer` where we
+  previously only accepted `string`)
+- Adding new optional parameters at the end of a signature
+- Improving error messages or adding context to them
+- Changing the shape of _logs_ (not errors)
+- Performance improvements that don't change observable behavior
+- Internal refactors invisible to the public API
+
+## Deprecation process
+
+When we need to remove public API surface, the removal is preceded by
+at least **one minor release** with a deprecation warning.
+
+Timeline:
+
+1. **Minor X.Y.0** — API marked deprecated
+   - JSDoc `@deprecated` tag with migration guidance
+   - TypeScript `@deprecated` in type declarations (strikethrough in
+     IDEs)
+   - Runtime warning printed once per process when the API is called
+     (`util.deprecate` or equivalent)
+   - Documentation updated with the recommended replacement
+   - Mentioned in the [CHANGELOG](CHANGELOG.md)
+2. **Minor X.Y.Z** and beyond — API still present, still warning
+3. **Major (X+1).0.0** — API removed
+
+The minimum one-minor window means a user running `npm update`
+regularly will see deprecation warnings before the next major ships.
+We will extend the window longer than one minor for widely-used API
+surface.
+
+## Supported runtime
+
+The minimum supported Node.js version is declared in
+[`package.json` `engines.node`](./package.json). We support Node LTS
+versions that are in active maintenance at the time of each major
+release.
+
+Dropping support for a Node version is a **breaking change** and
+lands only in major releases.
+
+Current: **Node.js ≥ 18**. The next major release is planned to
+require **Node.js ≥ 20** to track LTS support timelines.
+
+## Supported release lines
+
+See [SECURITY.md](SECURITY.md#supported-versions). In summary:
+
+- **Latest minor** receives bug fixes and security patches
+- **Previous minor** receives a 30-day security-only patch window
+  after a new minor ships
+- **Older minors** are unsupported
+
+The same matrix applies to non-security bug fixes.
+
+## TypeScript
+
+Type declarations ship in `dist/index.d.ts` and are considered part of
+the public API surface. Breaking changes to _types_ that require
+consumer code changes follow the same SemVer rules as runtime API.
+
+Type-widening changes (making a type more permissive) are not breaking.
+Type-narrowing changes (making a type more restrictive, causing
+previously-compiling code to fail) are breaking.
+
+We target the TypeScript version currently listed in
+`package.json` `devDependencies`. Consumers using older TypeScript
+versions may see type errors that are not regressions.
+
+## ESM and CommonJS
+
+The package ships **dual-module builds** — both ESM (`dist/index.mjs`)
+and CommonJS (`dist/index.cjs`). `package.json` `exports` resolves the
+correct entry point based on how the consumer imports it.
+
+The ESM build is the primary development target. The CommonJS build
+is a compatibility layer. When a feature can only be expressed in ESM
+(e.g. top-level await, real dynamic imports), we ship it in ESM and
+document the CJS limitation.
+
+Dropping CommonJS support is a breaking change and will land only in
+a major release, with a full minor of deprecation warnings first.
+
+## What this means for you
+
+If you are building on top of this package, you can:
+
+- **Auto-merge patch updates** with confidence. Nothing public-facing
+  breaks.
+- **Review minor updates** via the CHANGELOG before merging. No
+  required code changes, but new features may be worth adopting.
+- **Plan for major updates.** Each major ships with a
+  `docs-site/docs/Migration.md` covering every breaking change and the
+  new call site.
+
+If you find a case where a patch or minor change broke your
+integration without a deprecation warning, open an issue — we treat
+that as a bug against this policy.
+
+## See also
+
+- [SECURITY.md](SECURITY.md) — supported version windows and CVE
+  handling
+- [CHANGELOG.md](CHANGELOG.md) — what actually changed per release
+- [Migration guide](https://devilsdev.github.io/rag-pipeline-utils/docs/Migration)
+  — upgrade instructions between majors

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@devilsdev/rag-pipeline-utils",
-  "version": "2.4.3",
+  "version": "2.4.5",
   "description": "Composable RAG for Node.js — with built-in evaluation, citations, guardrails, and observability.",
   "main": "dist/index.cjs",
   "type": "commonjs",
@@ -153,7 +153,9 @@
     "README.md",
     "LICENSE",
     "SECURITY.md",
-    "CHANGELOG.md"
+    "CHANGELOG.md",
+    "BENCHMARKS.md",
+    "SEMVER.md"
   ],
   "publishConfig": {
     "access": "public"