@reportwright/engine 0.11.0 → 0.12.1

package/CHANGELOG.md

@@ -2,6 +2,87 @@
 
 All `@reportwright/*` packages (formerly `@pagewrightjs/*`) share one version. The versioning policy is in [CONTRIBUTING.md](CONTRIBUTING.md#versioning-semver).
 
+## 0.12.1 — 2026-10-08
+
+### Fixed
+- **Word export: long reports.** The document XML goes to the zip in pieces instead of one string: a 500k-row report
+  hit V8's string limit (RangeError: Invalid string length). A report with no tables exports without `exportData`.
+- **`exportHtml(model, { fonts, ...options })`** takes the options object like the other exporters; the 0.7 form
+  `exportHtml(model, fontStore, options)` still works with a deprecation warning. It threw before ("reading 'get'").
+- **Charts: scatter and bubble axes** no longer pad past zero the data does not stay above (0-100 data gives 0 to 120,
+  not -50 to 150). Warnings: a row one series leaves blank is drawn by the others, so it is not reported as not drawn;
+  a gantt chart has no series by design and gets no "has no series" warning.
+- **Variable CJK fonts** are cut at Regular (wght 400), not their default Thin instance, in the PDF and HTML subsets.
+- **Data fetch:** gzip, deflate and br bodies are decoded (the byte cap counts decoded bytes); `file:` and `data:` URLs
+  are refused with the scheme named, not "its host is made by a parameter".
+- **Format characters** (RLM, LRE/LRI, ZWJ/ZWNJ, soft hyphen) need no glyph: no "No font has these characters" warning,
+  and no notdef box in the PDF.
+- **Warnings:** an unnamed item is named by its type (no "undefined:"), and a sub-report warning that validate and the
+  render both say is said once.
+- **`parameterOptions()`** lists a parameter's fixed options first, then the fetched ones, each value once (as the viewer does).
+- **HTML Tables: date sort keys** use the wall clock, so the same report gives the same keys in every host time zone.
+
+- **Streaming PDF (`exportPdfStream`), from an independent 1M-row verification:**
+  - Rows given as an array are read where they are (never copied whole or sorted, tested); JSON given as text is read
+    row by row instead of parsed whole (a 300,000-row text source: 910 → 500 MB peak). The README says why a large array
+    still shows a large RSS (it is the caller's, and V8 lets garbage grow beside it) and to pass a cursor or text.
+  - Grouped and sorted tables: in Node the engine package sorts through a disk spill by default (`spill: { dir, maxMB }`,
+    `spill: false`), so a grouped export's memory is flat (300,000 rows: 300 MB, was 670 MB and growing); `presorted:
+    true` reads rows that come in the report's order without sorting, checking each against the one before (a row out of
+    order stops the export with an error).
+  - A data set without declared `fields` streams: its fields are detected from the first 50 rows, as the whole render
+    detects them. A report written whole now says why in its first warning, not only in `why`.
+  - The window that reaches the last rows is the last one: a table whose footer keeps with its rows could stop with
+    "The data changed while the PDF was written".
+  - Tagged PDFs are about 27% smaller (a page's borders are one artifact and its cells' text one text object; plain cells
+    carry no attributes; the xref stream is predicted). Past a million structure elements the result warns that tools
+    which load every object (qpdf, pikepdf) may run out of memory; `tagged.maxElements` caps it.
+  - `parallel: N` is documented with its memory (150–300 MB a thread) and when it helps; the example runs serial.
+  - The comments of `exportPdfStream` said tagged and PDF/A files do not stream; they do.
+  - Note: the non-streaming `exportPdf` still uses pdf-lib, which stays a dependency.
+- **BIRT import: SUM and other aggregates total their column.** A bound column with `aggregateFunction` (SUM, AVE,
+  MIN, MAX, FIRST, LAST…) had no listed argument, so it fell back to a row count: `Sum(1, …)` showed 3 for a total of
+  250.75. The column's own expression is now the argument. An aggregate with no expression is counted as approximated.
+  The Jasper and SSRS importers were checked: they already take the expression.
+- **Docs: no npm provenance badge.** The repository is private, so npm provenance cannot be generated. The badges and
+  the claims are removed from the package READMEs and `docs/RELEASING.md`; the publish workflow stays, and its notes say
+  provenance needs a public repository.
+- **Docs: the viewer's `data` example** says which shape matches the data set's path: `{ orders: { rows: [...] } }` for
+  `"path": "$.rows"`, `{ orders: [...] }` for a data set with no path. A mismatch gives empty tables, silently.
+- **Docs: a missing font** stops the viewer's run with a message naming the font and a Retry button; the README said the
+  report draws in a fallback font, which it does not.
+- **Docs: the engine README** says `pw` is in `@reportwright/cli`, and its quick start has a `statement.pw.json` to save.
+- **CLI help** marks `pw audit` and `pw migrate-storage` as server only; the npm CLI does not include them.
+- **Engine text:** the CJK font warning names `npx reportwright-fonts-cjk` first (the app's `npm run fonts:cjk` second),
+  and the deprecated positional export forms say they are removed in 1.0.
+
+## 0.12.0 — 2026-10-08
+
+Streaming PDF: very large reports are written page by page, with flat memory.
+
+### Added
+- **`exportPdfStream(definition, options, sink)`** in `@reportwright/engine`: a report whose body is a flowing table
+  (plain or grouped: repeating group headers, subtotals, `RowNumber()`, widow control, keep-together, page breaks) is laid
+  out a window of rows at a time and each finished page goes straight to the sink. Peak memory stays flat: about 260 MB
+  for the whole render process from 20,000 to 100,000 rows (projected about 250–275 MB at 1,000,000).
+  "Page N of M" and footer totals work (a counting pass first). PDF/A-2b, PDF/UA-1 and PDF/A-2a + PDF/UA-1 stream too
+  (veraPDF passes). Anything else (images, charts, matrices, rich text, subreports, signing, encryption) falls back to
+  `exportPdf`, with the same output.
+- **The server streams large PDFs** (`/api/reports/<id>/pdf`, `/api/v1/render`) with the same row caps, deadline, stall
+  timeout and abort handling as the CSV/Excel streaming; the viewer's large-report button gets the streamed file.
+- **`parallel: N`** (Node, opt-in): page ranges painted on worker threads, byte-identical to the serial file. Measure on
+  your hardware before relying on it; each worker costs about 150 MB.
+- `examples/stream-1m.mjs` in the engine package: streams a generated million-row ledger to a file and prints peak memory.
+- `planPaged` exported for custom pipelines.
+
+### Changed
+- The PDF painter is shared between `exportPdf` and the streaming writer; `exportPdf` output is byte-for-byte unchanged.
+- The engine package ships its readable bundle without a source map (1.3 MB unpacked).
+
+### Known limits
+- In the npm engine, grouped reports sort rows in memory (the server spills them to disk), so a 1M-row grouped report is
+  not flat outside the server yet.
+
 ## 0.11.0 — 2026-10-08
 
 Everything since 0.10.0. Items marked **Behaviour change** alter output that a pixel-locked layout or a locale-sensitive

package/README.md

@@ -1,13 +1,34 @@
 # @reportwright/engine
 
-[![npm](https://img.shields.io/npm/v/%40reportwright%2Fengine)](https://www.npmjs.com/package/@reportwright/engine) [![license](https://img.shields.io/badge/license-MIT-blue)](https://github.com/MrArun005/reportwright/blob/main/LICENSE) [![Socket](https://socket.dev/api/badge/npm/package/@reportwright/engine)](https://socket.dev/npm/package/@reportwright/engine) [![provenance](https://img.shields.io/badge/npm-provenance-green)](https://docs.npmjs.com/generating-provenance-statements)
+[![npm](https://img.shields.io/npm/v/%40reportwright%2Fengine)](https://www.npmjs.com/package/@reportwright/engine) [![license](https://img.shields.io/badge/license-MIT-blue)](https://github.com/MrArun005/reportwright/blob/main/LICENSE) [![Socket](https://socket.dev/api/badge/npm/package/@reportwright/engine)](https://socket.dev/npm/package/@reportwright/engine)
 
-The ReportWright report engine for Node and browsers: a JSON report definition and data in, pages out, then PDF (also PDF/A-2b), Word, Excel, PowerPoint (exportPptx), HTML or CSV. Includes the `pw` command line.
+The ReportWright report engine for Node and browsers: a JSON report definition and data in, pages out, then PDF (also PDF/A-2b), Word, Excel, PowerPoint (exportPptx), HTML or CSV. The command line is `@reportwright/cli` (`pw`).
 
 ```bash
 npm i @reportwright/engine @reportwright/fonts
 ```
 
+Save this as `statement.pw.json` (a minimal report; the repository's `examples/` has more):
+
+```json
+{
+  "$schema": "pagewright/report@1",
+  "name": "Statement",
+  "page": { "size": "A4", "margins": [36, 36, 36, 36] },
+  "parameters": [{ "name": "accountId", "type": "string" }],
+  "dataSources": [{ "name": "ledger", "type": "json", "data": { "rows": [{ "item": "Design", "amount": 1200 }, { "item": "Hosting", "amount": 99.5 }] } }],
+  "dataSets": [{ "name": "Lines", "source": "ledger", "path": "$.rows" }],
+  "sections": { "body": { "items": [
+    { "type": "textbox", "name": "Title", "x": 0, "y": 0, "w": 400, "h": 24, "value": "Statement", "style": { "fontSize": 18, "fontWeight": "bold" } },
+    { "type": "table", "name": "Table", "x": 0, "y": 40, "w": 400, "h": 60, "dataSet": "Lines",
+      "columns": [{ "width": 250 }, { "width": 150 }],
+      "header": [{ "height": 18, "cells": [{ "value": "Item" }, { "value": "Amount", "style": { "textAlign": "right" } }] }],
+      "detail": [{ "height": 18, "cells": [{ "value": "=Fields.item" }, { "value": "=Fields.amount", "style": { "format": "C2", "textAlign": "right" } }] }],
+      "footer": [{ "height": 18, "cells": [{ "value": "Total" }, { "value": "=Sum(Fields.amount)", "style": { "format": "C2", "textAlign": "right", "fontWeight": "bold" } }] }] }
+  ] } }
+}
+```
+
 ```js
 import { render, defaultFontStore, exportPdf } from '@reportwright/engine';
 import fs from 'node:fs';
@@ -46,6 +67,65 @@ const bundledFonts = defaultFontStore({ fontsUrl: '/pw/fonts/' }); // a missing
   `timeoutMs` as well; the PDF exporter checks every PNG before decoding it and leaves a damaged one out (`onWarning`).
 - **Currency.** `C2` with no `currency` on the report uses the locale's own (`en-IN` ₹, `de-DE` €, `en-US` $).
 
+## Streaming PDF for large reports
+
+`exportPdfStream(definition, options, sink)` writes the PDF as it is laid out: a flowing table is laid out a window of
+rows at a time and each page leaves memory once it is written, so a million-row report's PDF needs about the memory of a
+few pages (measured: peak RSS flat from 20,000 to 300,000 rows, about 250 MB for the whole Node process; most of it is
+garbage V8 has not collected yet, not rows: it does not shrink with a smaller `window`, and
+`node --max-old-space-size=128` holds a 300,000-row export to about 180 MB). The file reads as
+`exportPdf`'s: the same pages, text and positions.
+
+```js
+import fs from 'node:fs';
+import { exportPdfStream } from '@reportwright/engine';
+
+const out = fs.createWriteStream('ledger.pdf');
+const r = await exportPdfStream(definition, { parameters: { year: 2026 }, title: 'Ledger' }, out);
+out.end();
+console.log(r.streamed ? `${r.pages} pages, streamed` : `written whole: ${r.why}`);
+```
+
+The sink is a Node `Writable` or anything with `write(bytes)` that may return a promise (back-pressure); it is not
+closed for you. Options are `render()`'s and `exportPdf`'s (`fonts`, `parameters`, `title`, `pdfa`, `tagged`, …), plus
+`window` (rows laid out at once, 1,000), `maxRows` (rows read, 5,000,000), `presorted`, `spill` and `parallel`. Rows come
+from the report's source a batch at a time: CSV and REST (JSON) sources are read as streams, JSON given as text is read
+row by row, an array given in the definition (or in `sources`) is read where it is, never copied or sorted, and
+`sqlRows(source, params)`, an async iterable of row batches, stands in for a database cursor. An array you pass is yours
+and stays in memory for the export: the export's own memory stays flat beside it (measured: the array plus about
+150 MB), but V8 lets garbage grow with a large live heap, so pass a cursor (`sqlRows`) or JSON text when the rows are
+many.
+
+- **Sorts and groups.** A table sort or groups put the rows in order first. In Node the sort spills to the system temp
+  folder (`spill: { dir, maxMB }`; 2 GB at most), so memory stays flat; `spill: false`, and in a
+  browser, it is in memory (all the rows: measured about 1.3 KB a row, so keep browser exports under about 100,000 sorted or
+  grouped rows). When the source already gives the rows in the report's order (`order by region, type` for groups by
+  region then type), pass `presorted: true`: nothing is sorted or held, each row is checked against the one before as it
+  passes, and a row out of order stops the export with an error.
+- **parallel: N** (Node, off by default) lays out and paints with N worker threads; the file is byte for byte the same.
+  Each thread loads the engine and fonts and lays out its own windows: about 150–300 MB a thread (measured: 260 MB
+  serial, 590 MB with 2, 890 MB with 4, for 100,000 rows). It helps only with idle cores and a report whose layout, not
+  its source, is the slow part: 15% faster with 2 threads on a 2-core machine. Leave it off unless you have measured.
+- **Tagged files.** A tagged (PDF/UA) table has a structure element a cell: a million-row ledger has some 8 million,
+  about 200 MB of file. Readers open it; tools that load every object (qpdf, pikepdf) may need gigabytes. Past a
+  million elements the result warns; `tagged: { lang, maxElements }` caps it (the export stops with an error).
+
+**What streams:** a body that is one table (with items above or below it), its rows' expressions row by row (fields,
+parameters, functions of the row, `RowNumber()`), groups with headers that repeat, subtotals (`Sum`, `Count`, `Avg`,
+`Min`, `Max` of a row expression, `CountRows()`), keep-together and page breaks, filters and sorts, page headers and
+footers without aggregates ("Page N of M" included: the pages are counted first, so the source is read twice), PDF/A-2b,
+and accessible PDFs (PDF/UA-1, PDF/A-2a; validated with veraPDF). A data set without declared `fields` streams too: its fields are
+detected from its first 50 rows, as the whole render detects them (declare them when a column first appears later).
+
+**What is written whole** (`exportPdf`; the result says why, in `why` and in its first warning, since its memory grows
+with its pages): matrices, charts, lists, subreports, images, rich text,
+lookups and aggregates outside footers, more than one data set, group sections (`newSection`), signed or encrypted PDFs.
+Grouped and sorted tables sort through a disk spill in Node (above), in memory in a browser.
+
+`examples/stream-1m.mjs` streams a generated million-row ledger to a file and prints its peak memory:
+`node node_modules/@reportwright/engine/examples/stream-1m.mjs 1000000 ledger.pdf` (`--grouped`, `--presorted`,
+`--tagged`, `--plain`, `--array`, `--array=text`; serial unless `--parallel=N`).
+
 ## Many reports at once (Node): a worker pool
 
 ```js