@dipanshuhandoo/excelio 1.0.0 → 1.1.0

package/CHANGELOG.md

@@ -1,6 +1,37 @@
 # Changelog
 
-## 0.1.0 (Unreleased)
+## 1.1.0 (2026-11-06)
+
+### Added
+
+- Sheet `data` accepts any iterable or async iterable (generators, database cursors,
+  object-mode streams) as well as arrays; iterable sheets are written inline with
+  output backlog bounded at about 8 MiB, so memory stays flat for large exports.
+- `maxRowsPerSheet` write option: sheets continue on `"<name> (2)"`, `"<name> (3)"`, ….
+- `sharedStrings: 'auto' | true | false` write option (`'auto'`: off when a workbook
+  has an iterable sheet).
+- `totalRows` sheet hint for progress, and `sheetPart` in `beforeWrite` context and
+  write progress events (which now also carry `sheetName`).
+- `SOURCE_ERROR` error code for failing row sources.
+- `npm run test:memory` release gate.
+
+### Fixed
+
+- Non-array `data` (for example a generator) silently wrote an empty sheet.
+- Sheets over 1,048,575 rows produced files Excel cannot open; they now roll over.
+- Failed writes to a file path left a partial file and an open handle; the file is
+  now removed. Failed writes to a caller's stream no longer end that stream.
+- Early file errors (such as a missing directory) could hang the write.
+- `batchSize: 0` silently disabled progress events.
+
+### Changed
+
+- Invalid `batchSize` (not a positive integer) now throws `INVALID_SPEC`.
+- Specs and write options are validated before worker routing.
+- `SheetSpec.data` is optional in the type declarations. TypeScript code with an
+  exhaustive switch over `ExcelIoErrorCode` must handle `SOURCE_ERROR`.
+
+## 1.0.0 (2026-10-08)
 
 - Package as `@dipanshuhandoo/excelio`, supporting Node.js 22 and 24 LTS.
 - Add readable/minified ESM and CommonJS builds, worker bundles, and declarations.

package/README.md

@@ -1,9 +1,80 @@
 # ExcelIO
 
 `@dipanshuhandoo/excelio` converts JavaScript row objects to XLSX workbooks and
-reads XLSX files into structured data. It supports streaming writes, multiple
-sheets/workbooks, row transforms and validation, and configurable worker-thread
-offloading. It is a Node.js library, not a browser library.
+reads XLSX files into structured data. Rows can come from arrays or be streamed
+from generators, database cursors, and paged APIs with bounded memory. It is a
+Node.js library, not a browser library. Its only runtime dependency is ExcelJS.
+
+## Features
+
+**Writing**
+
+- One call writes one or many workbooks, each with multiple sheets, to a Buffer,
+  a file path, a writable stream, or a readable stream (`writeStream`).
+- Sheet `data` can be an array, a sync iterable, or an async iterable (async
+  generator, MongoDB/Mongoose cursor, object-mode stream, paged API).
+- Large exports: streamed rows with a file or stream target keep memory flat
+  (about 50 MB growth for 1,000,000 rows), with output backpressure built in.
+- Automatic sheet rollover at Excel's row limit, or earlier with
+  `maxRowsPerSheet`: `Links`, `Links (2)`, `Links (3)` … keep columns, header,
+  and styling.
+- Columns: explicit keys, headers, widths, types (`string`, `number`, `boolean`,
+  `date`), and Excel number formats, or keys inferred from the first row.
+- Write-side coercion of numeric/boolean strings and dates; multiline and
+  multilingual text.
+- Styling: bold header, frozen header row, auto-filter; workbook properties
+  (author, company, title, created, modified).
+- Safe sheet names: forbidden characters replaced, 31-character limit,
+  duplicate suffixes.
+- Shared-strings control (`sharedStrings`) to trade file size for memory.
+
+**Reading**
+
+- Reads a Buffer, file path, or readable stream into
+  `{ ok, data, errors, stats }`, covering all sheets with headers as keys.
+- Options for header row, blank handling, trimming, and date output (ISO string,
+  Date, or epoch).
+- Formula cells return cached values; rich text and hyperlinks return displayed
+  text.
+
+**Transforms and validation**
+
+- `beforeWrite` and `afterRead` row hooks with row context (workbook, sheet,
+  part, Excel row number).
+- `validateRow` collects per-row errors (partial data accepted) or fails fast
+  with `onRowError: 'fail'`.
+
+**Operations**
+
+- Worker-thread offload for large plain jobs (`useWorker: 'auto' | true | false`,
+  configurable thresholds); hooks, streams, and iterables run inline
+  automatically.
+- Progress callbacks (`onProgress`, `batchSize`) with sheet, part, and total row
+  information.
+- Cancellation through `AbortSignal`; streamed sheets respond during the write.
+- Failure cleanup: a failed file write removes the partial file, and a caller's
+  stream is not ended or destroyed; row-source errors are reported as
+  `SOURCE_ERROR` with row details.
+- Typed `ExcelIoError` codes: `INVALID_SPEC`, `IO_ERROR`, `ABORTED`,
+  `WORKER_ERROR`, `ROW_VALIDATION`, `SOURCE_ERROR`.
+
+**Packaging**
+
+- ESM and CommonJS, readable and minified (`/min`) builds, worker bundles,
+  source maps, and TypeScript declarations.
+
+| Use case | Example |
+| --- | --- |
+| In-memory round-trip | [01-buffer-roundtrip](https://github.com/DipanshuHandoo/excelio/blob/main/examples/01-buffer-roundtrip.mjs) |
+| Styled multi-sheet report to a file | [02-sales-report](https://github.com/DipanshuHandoo/excelio/blob/main/examples/02-sales-report.mjs) |
+| Validated import with row errors | [03-import-validation](https://github.com/DipanshuHandoo/excelio/blob/main/examples/03-import-validation.mjs) |
+| Stream a workbook to a destination | [04-streaming](https://github.com/DipanshuHandoo/excelio/blob/main/examples/04-streaming.mjs) |
+| Workers and concurrent exports | [05-workers-and-concurrency](https://github.com/DipanshuHandoo/excelio/blob/main/examples/05-workers-and-concurrency.mjs) |
+| Progress and cancellation | [06-progress-and-cancellation](https://github.com/DipanshuHandoo/excelio/blob/main/examples/06-progress-and-cancellation.mjs) |
+| CommonJS and minified builds | [07-commonjs-and-minified](https://github.com/DipanshuHandoo/excelio/blob/main/examples/07-commonjs-and-minified.cjs) |
+| XLSX to JSON | [08-excel-to-json](https://github.com/DipanshuHandoo/excelio/blob/main/examples/08-excel-to-json.mjs) |
+| Service export migration | [09-service-exports](https://github.com/DipanshuHandoo/excelio/blob/main/examples/09-service-exports.mjs) |
+| Stream rows from a paged source with rollover | [10-streaming-rows](https://github.com/DipanshuHandoo/excelio/blob/main/examples/10-streaming-rows.mjs) |
 
 ## Installation
 
@@ -41,6 +112,21 @@ const result = await excelio.read(buffer);
 console.log(result.data[0].sheets[0].data);
 ```
 
+Stream a large export straight from a source, without building an array:
+
+```js
+async function* orders() {
+  for await (const doc of Order.find({}).lean().cursor({ batchSize: 1000 })) {
+    yield { customer: doc.customer, amount: doc.amount, date: doc.createdAt };
+  }
+}
+
+await excelio.write([{
+  workbook: 'orders',
+  sheets: [{ sheet: 'Orders', columns: spec[0].sheets[0].columns, data: orders() }]
+}], { to: 'orders.xlsx', signal, onProgress: console.log });
+```
+
 CommonJS exposes the same named exports:
 
 ```js
@@ -61,8 +147,9 @@ workbook-processing time. Both builds include source maps and declarations.
 
 See the [example guide](https://github.com/DipanshuHandoo/excelio/blob/main/examples/README.md)
 for buffer round-trips, styled multi-sheet reports, validated imports, streaming,
-workers/concurrent exports, progress/cancellation, CommonJS/minified usage, and
-XLSX-to-JSON conversion.
+workers/concurrent exports, progress/cancellation, CommonJS/minified usage,
+XLSX-to-JSON conversion, service export migration, and streamed rows with sheet
+rollover.
 Examples live in the repository and use public package exports.
 
 From a checkout, run all cases or select one:
@@ -79,10 +166,49 @@ Run the file examples directly after building to keep their XLSX/JSON output:
 node examples/02-sales-report.mjs
 node examples/04-streaming.mjs
 node examples/08-excel-to-json.mjs
+node examples/10-streaming-rows.mjs
 ```
 
 These save to the ignored example-output/ directory by default.
 
+## Options Reference
+
+Common to `write`, `writeStream`, and `read`:
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `onProgress` | none | `(event) => void`, called every `batchSize` rows; writes also emit a final event |
+| `batchSize` | 5000 | Progress interval; on write it must be a positive integer and also sets the backpressure/yield interval for streamed sheets |
+| `signal` | none | `AbortSignal`; rejects with `ABORTED` |
+| `logger` | none | `{ info?, warn?, error? }` |
+| `useWorker` | `'auto'` | `'auto'`, `true`, or `false` |
+| `workerThreshold` | `{ rows: 50_000, bytes: 10 MiB }` | Auto-mode offload thresholds |
+
+Write only:
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `to` | none (returns Buffer/Buffer[]) | File path or writable stream |
+| `beforeWrite` | none | `(row, ctx) => row`, synchronous transform |
+| `maxRowsPerSheet` | 1,048,575 | Rows per worksheet before continuing on `"<name> (2)"` |
+| `sharedStrings` | `'auto'` | `'auto'` (off when a workbook has a streamed sheet), `true`, or `false` |
+
+Read only:
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `headerRow` | 1 | Row containing headers |
+| `nullForBlank` | `true` | Empty cells become `null` |
+| `trimStrings` | `true` | Trim string cells |
+| `dateFormat` | `'iso'` | `'iso'`, `'date'`, or `'epoch'` |
+| `afterRead` | none | `(row, ctx) => row`, synchronous transform |
+| `validateRow` | none | `(row, ctx) => null \| { message, code?, column? }` |
+| `onRowError` | `'collect'` | `'collect'` or `'fail'` |
+
+Sheet spec: `sheet`, `columns`, `styling` (`headerBold`, `freezeHeader`,
+`autoFilter`), `data` (array, iterable, or async iterable; optional), and
+`totalRows` (progress hint for streamed sheets).
+
 ## API
 
 ### Writing
@@ -98,15 +224,64 @@ These save to the ignored example-output/ directory by default.
   separate buffers, or write each workbook individually. No ZIP wrapper is added.
 
 `spec` is a non-empty array of workbooks, each containing a non-empty `sheets`
-array. Sheets contain `sheet`, optional `columns` and `styling`, and a `data`
-array. Workbook properties support author, company, title, created, and modified.
+array. Sheets contain `sheet`, optional `columns` and `styling`, and optional
+`data`. Workbook properties support author, company, title, created, and modified.
 The `workbook` field is logical metadata, not an automatic filename.
 
+`data` accepts an array, any iterable (for example a `function*` generator), or
+any async iterable (an `async function*`, an object-mode stream, or any async
+iterable such as a database cursor; excelio has no database dependency). Omitted
+or null `data` writes an empty sheet. Iterables are consumed once, so pass a
+separate iterable per sheet; reusing one iterable object across sheets throws
+`INVALID_SPEC` (the same array may be reused).
+
+```js
+async function* rows() {
+  for await (const doc of collection.find({}).batchSize(1000)) {
+    yield { url: doc.url, status: doc.status };
+  }
+}
+
+await excelio.write([{
+  workbook: 'audit',
+  sheets: [{ sheet: 'Pages', columns: [{ key: 'url' }, { key: 'status' }], data: rows() }]
+}], { to: 'audit.xlsx' });
+```
+
+When excelio stops early (error or abort) it calls the iterator's `return()`,
+which ends a `for await` loop in the generator and closes the underlying cursor.
+
 Columns support key, header, width, type (`string`, `number`, `boolean`, `date`),
 and Excel number-format string. Without explicit columns, keys are inferred from
-the first row only. Width defaults to 15. Sheet names replace forbidden Excel
-characters, truncate to 31 characters, and receive suffixes when duplicated.
-`logger.warn` reports truncation.
+the first row only (for iterables, the first row is peeked and still written).
+Explicit columns are recommended for iterables: stable order and types. Width
+defaults to 15. Sheet names replace forbidden Excel characters, truncate to 31
+characters, and receive suffixes when duplicated. `logger.warn` reports truncation.
+
+### Large Exports
+
+- Combine iterable `data` with `to` (a file path or writable stream). Buffer
+  output still collects the whole file in memory.
+- Iterable sheets wait whenever about 8 MiB of produced XML is queued for
+  compression or the target, checking every `batchSize` rows. Compression is
+  slower than row production, so this matters for local files too.
+- `sharedStrings: 'auto' | true | false` (default `'auto'`): the Excel
+  shared-strings table is on unless a workbook has an iterable sheet. On gives a
+  smaller file, but memory grows with the number of distinct strings (1,000,000
+  rows of distinct URLs added 1.4–2.1 GB). Off keeps memory flat and writes
+  plain strings as `t="str"` cells.
+- `maxRowsPerSheet` (default and maximum 1,048,575, Excel's limit minus the
+  header) continues a sheet on `"<name> (2)"`, `"<name> (3)"`, … with the same
+  columns, header, and styling. It applies to arrays and iterables. Names are
+  allocated after every logical sheet's own name, so a continuation never takes
+  a name another sheet uses.
+- `totalRows` on an iterable sheet is an optional hint used for progress
+  `totalRows`; without it, progress `totalRows` is undefined.
+- Measured with exceljs 4.4.0 on Node 22 (Windows): 1,000,000 rows with six
+  distinct-string columns to a file grew RSS by about 50 MB and took 24–58 s
+  across runs; a slow 64 KB sink and two 200,000-row sheets stayed under 25 MB. The
+  bound relies on exceljs/archiver internals, so it is re-measured on every
+  exceljs upgrade (`npm run test:memory`).
 
 ### Reading
 
@@ -155,7 +330,10 @@ await excelio.write(spec, {
 ```
 
 Hooks are synchronous. They receive `(row, ctx)` where ctx contains workbookIndex,
-workbookName, sheetIndex, sheetName, and the 1-based Excel rowNumber. Read applies
+workbookName, sheetIndex, sheetName, and the 1-based Excel rowNumber. On write,
+ctx also has `sheetPart` (1, 2, … after `maxRowsPerSheet` rollover); sheetIndex
+stays the spec index, sheetName is the actual worksheet name, and rowNumber
+restarts at 2 on each continuation sheet. Read applies
 afterRead before validateRow. Invalid rows are excluded from data and collected
 in errors by default; `onRowError: 'fail'` throws on the first issue. A hook
 returning null/undefined keeps the original row. Write transforms do not add
@@ -176,19 +354,25 @@ reusable worker pool, concurrency cap, or parallel splitting of a workbook.
 Submit independent operations concurrently only when memory and CPU permit.
 
 Functions (`beforeWrite`, `afterRead`, `validateRow`, `onProgress`), `logger`,
-`signal`, stream inputs, and writable stream destinations force inline execution,
-even when useWorker is true. Worker payloads must be structured-cloneable;
+`signal`, stream inputs, writable stream destinations, and iterable sheet data
+force inline execution, even when useWorker is true (with iterable data,
+`logger.warn` reports the ignored useWorker). Worker payloads must be structured-cloneable;
 functions or unsupported values embedded in row data cannot be sent to workers.
 Bundlers/serverless deployments must preserve the emitted entrypoint and its
 sibling workers directory. Import the package normally rather than copying a
 single bundled file.
 
 `onProgress` receives phase and rowsProcessed plus available workbook/sheet
-metadata. `batchSize` defaults to 5,000 and controls notifications, not memory
-or yielding. Writes emit a final notification; reads notify at batch boundaries.
-`signal` is checked between rows and sheets, but cannot interrupt initial XLSX
-loading, commit, or synchronous row loops. Do not rely on timely cancellation of
-an in-progress operation from an event-loop callback.
+metadata; write events include sheetIndex, sheetName, and sheetPart. `batchSize`
+defaults to 5,000 and must be a positive integer. Writes emit a final
+notification; reads notify at batch boundaries.
+
+`signal` is checked between rows and sheets. Iterable sheets check it on every
+row and yield to the event loop every `batchSize` rows, so an abort fired from a
+timer or request handler is observed during long iterable writes. A source stuck
+inside `next()` cannot be interrupted by excelio; pass the same signal to the
+source. Array sheets still run as a synchronous loop, and initial XLSX loading
+and commit cannot be interrupted.
 
 ### Values and Errors
 
@@ -198,16 +382,28 @@ the strings 'true'/'false', and 1/0. Unsupported coercions are left unchanged;
 these hints are not schema validation. Multiline and multilingual text is supported.
 
 `ExcelIoError` exposes code, optional cause, and optional details. Codes include
-INVALID_SPEC, IO_ERROR, ABORTED, WORKER_ERROR, and ROW_VALIDATION. Collected read
+INVALID_SPEC, IO_ERROR, ABORTED, WORKER_ERROR, ROW_VALIDATION, and SOURCE_ERROR.
+SOURCE_ERROR means an iterable sheet's source threw (or returned a non-object
+iterator result); `cause` is the original error and `details` holds
+workbookIndex, sheetIndex, sheetName, and rowsRead. If the signal is already
+aborted when the source fails, the error is ABORTED with the source error as
+`cause`. Collected read
 errors contain workbook index, sheet, row, column, code, and message. `ok` is
 false when collected errors exist; callers decide whether to accept partial data.
 Some lower-level ExcelJS/write errors can propagate as ordinary Error instances.
 
+On a failed write, a file-path target is closed and removed; a caller's writable
+stream is unpiped but neither ended nor destroyed; the original error is thrown,
+with `err.cleanupError` set if cleanup itself failed. On success a caller's
+stream is ended by the write, as before.
+
 ## Memory and Scope
 
-Writes use ExcelJS's streaming XLSX writer and commit rows incrementally, but
-input arrays and shared strings remain in memory; buffer output also collects
-the complete file. Direct file/stream output avoids that final buffer.
+Writes use ExcelJS's streaming XLSX writer and commit rows incrementally. Array
+`data` holds all rows in memory, and so does the shared-strings table when it is
+on. Iterable `data` with the default `sharedStrings` and a file or stream target
+keeps write memory bounded (see Large Exports). Buffer output collects the
+complete file either way; direct file/stream output avoids that final buffer.
 Reading currently loads the entire workbook and collects accepted rows.
 Workers keep processing off the main thread but do not guarantee lower total
 memory or faster completion; cloning large inputs adds overhead.