sheet-to-schema 1.0.0 → 1.1.0-rc.2

package/README.md

@@ -11,8 +11,10 @@ one record per row, or per group of rows, reporting every cell it could not use.
   with memory bounded by a batch, and serves the import over HTTP.
 - **The same engine on both sides.** A file imports to the same records in the browser and
   on the server, because both run the same function over the same plan.
+- **The same reader on both sides.** `sheet-to-schema/browser` reads a CSV, TSV or `.xlsx`
+  file in a browser into exactly the rows the server reads from the same bytes.
 
-> **Status: stable (1.0.0).** It is the import engine of
+> **Status: stable (1.0.0); 1.1.0-rc.2 on `next`.** It is the import engine of
 > [Dynamic Entity](https://github.com/berserker5619/Dynamic-Entity#readme) 3.0, extracted so it can be used without it.
 > Members marked `@internal` are not covered by semver and may change in any release.
 
@@ -20,8 +22,8 @@ one record per row, or per group of rows, reporting every cell it could not use.
 
 ```bash
 npm install sheet-to-schema
-# Only for sheet-to-schema/node, and only the ones you use:
-npm install exceljs   # to read or write .xlsx
+# Only for sheet-to-schema/node and /browser, and only the ones you use:
+npm install exceljs   # to read or write .xlsx (on a server or in a browser)
 npm install busboy    # to receive uploads
 npm install express   # for the ready-made router
 ```
@@ -135,8 +137,20 @@ result.warnings;
 ```
 
 Rows of a group that disagree on a parent field keep the first row's value and raise a
-**warning**: the record is still imported. Grouping works in `applyMapping`. The streaming
-runner in `/node` does not group yet, and refuses a grouped plan instead of ignoring it.
+**warning**: the record is still imported. Grouping works in `applyMapping`. On the server it
+is opt-in in 1.1: `runImport` and `createImportRouter` group when given `grouping: true`, and
+give the same records as `applyMapping` (a group that straddles a batch is carried into the next
+one and written whole). Without it a grouped plan is refused, as in 1.0. The runner groups
+contiguous rows only (sort the sheet by the key columns); `contiguous: false` works in
+`applyMapping` and is refused by the runner, since it would hold every group to the end of the
+file.
+
+Grouping is bounded: a group of more than 10,000 rows, more than 200,000 groups, or (for
+contiguous groups) more than 8 MiB of group keys stops the import with an `ImportFailure`
+(`GROUP_TOO_LARGE` or `TOO_MANY_GROUPS`), as does a key that comes back after its rows ended
+(`GROUP_NOT_CONTIGUOUS`). Raise a limit with
+`applyMapping(rows, plan, adapter, ctx, { groupLimits: { maxGroupRows: 50_000 } })`, or with the
+server's `limits`. Over HTTP these answer 413 and 422.
 
 ## Dates are never guessed
 
@@ -160,6 +174,42 @@ For anything else, implement `SchemaAdapter`: `id`, `adapterVersion`, and
 its own cells (`coerce`), finish a record (`finalize`) and check one (`validate`). That is how
 Dynamic Entity's form configs run on this engine.
 
+## In a browser: `sheet-to-schema/browser`
+
+```ts
+import { applyMapping, decimalMarkFor } from 'sheet-to-schema';
+import { readSheetFile } from 'sheet-to-schema/browser';
+
+// `file` from an <input type="file">: a CSV, a TSV or an .xlsx, decided by its bytes.
+const sheet = await readSheetFile(file, { filename: file.name });
+const result = applyMapping(sheet.rows, plan, adapter, undefined, { decimal: decimalMarkFor(sheet.delimiter) });
+```
+
+`readSheetFile` returns the grid the server's `readSheet` reads from the same file: the same
+headers, the same rows with the same typed cells (a date cell is a `Date`, a formula is its
+cached result), the same separator, and the same refusals, with the server's error codes and
+messages. Pass `sheet.delimiter` to `decimalMarkFor`, as the server does, so a semicolon file
+reads `1,5` as 1.5 on both sides. `readXlsx` is the same for a caller that already knows the
+file is a workbook.
+
+- **A CSV needs nothing installed.** `exceljs` is loaded with `import()` the first time a
+  workbook is read, so a CSV-only app never downloads it. Without it, a workbook fails with
+  what to install.
+- **The limits are the server's**, with its defaults: `maxBytes`, `maxRows`, `maxColumns`,
+  `maxCellLength` and `maxZipEntries`, through `limits`.
+- **A workbook's zip is checked by what it declares** (entry count, sizes, compression ratio)
+  before ExcelJS sees it. The server also checks what each entry actually inflates to; the
+  browser does not, because there the file is the user's own and only their tab is at risk.
+- **The first sheet, header row 1**, as on the server.
+
+Known limitations of the browser path:
+
+- **It needs `DecompressionStream`** (every current browser, and Node 20) to read a workbook;
+  without it, a workbook fails with a message saying so. A CSV needs nothing.
+- **A broken archive may be described differently.** Both readers refuse it, but the server
+  reads entries front to back and the browser reads the zip's directory, so the message for a
+  damaged `.xlsx` can differ. Every well-formed workbook in the parity fixtures reads the same.
+
 ## On a server: `sheet-to-schema/node`
 
 ```ts
@@ -172,6 +222,8 @@ app.use(
   createImportRouter(express.Router(), {
     bindings: { order: { adapter, ctx: undefined, kind: 'json-schema' } },
     dateOrder: 'DMY',
+    // Opt-in in 1.1: import plans that group rows into one record (previews then say "group").
+    grouping: true,
     // Called once per batch, and awaited. It must be idempotent: a client will retry.
     onImport: async (records, { name }) => saveAll(name, records),
   }),
@@ -203,13 +255,18 @@ createImportRouter(express.Router(), {
 Four things to know before deploying it:
 
 1. **An import is not transactional.** A file that fails at row 30,000 has written 29,999
-   records. Run `/validate` first, and make your writer idempotent.
+   records. Run `/validate` first, and make your writer idempotent. A grouped import can stop
+   on a group limit or a key that comes back, after earlier groups were written; `/validate`
+   runs the same pipeline and finds that first.
 2. **Authentication, authorization and concurrency are yours.** The router checks none.
-3. **Every limit is on by default**: bytes, rows, columns, cell size, zip expansion, idle and
-   total time. See `DEFAULT_LIMITS`; pass `limits` to change one.
-4. **A client must check `planVersions`.** `planForServer(plan, preview.planVersions)` returns
-   the plan to send to that server, or the reason not to: a server from before plan v2 would
-   ignore `group` and import one record per row.
+3. **Every limit is on by default**: bytes, rows, columns, cell size, zip expansion, group
+   size and count, idle and total time. See `DEFAULT_LIMITS`; pass `limits` to change one.
+4. **A client must check what the server reads.** `planForServer(plan, preview)` returns the
+   plan to send to that server, or the reason not to: a server from before plan v2 would ignore
+   `group` and import one record per row, and a 1.0 server reads plan v2 but refuses `group`.
+   A preview lists `planVersions` and, from 1.1, `features` (`["group"]` on a router created
+   with `grouping: true`); a missing `features` means the server will not group. The 1.0 form, `planForServer(plan, preview.planVersions)`,
+   still works and checks plan versions only.
 
 ## Problem codes
 
@@ -221,6 +278,15 @@ Four things to know before deploying it:
 
 `PLAN_PROBLEM_CODES`, `CELL_PROBLEM_CODES` and `RECORD_PROBLEM_CODES` list them all.
 
+A grouped import can also stop as a whole, with an `ImportFailure` (`IMPORT_FAILURE_CODES`).
+`applyMapping` and `runImport` throw it; over HTTP it is the error body's `code`:
+
+| Code | HTTP | Raised when |
+| --- | --- | --- |
+| `GROUP_NOT_CONTIGUOUS` | 422 | A group's key comes back after its rows ended. `details` has the key and both rows. |
+| `GROUP_TOO_LARGE` | 413 | A group has more than `maxGroupRows` rows. |
+| `TOO_MANY_GROUPS` | 413 | More than `maxGroups` groups, or more than `maxGroupKeyBytes` of group keys. |
+
 ## Licence
 
 MIT

package/dist/browser/index.d.mts

@@ -0,0 +1,41 @@
+import { SheetGrid, CsvDelimiter } from 'sheet-to-schema';
+import { I as ImportLimits } from '../import-error-Co6lRAqv.mjs';
+export { a as ImportError, b as ImportErrorCode } from '../import-error-Co6lRAqv.mjs';
+
+/** The limits a browser read enforces. Each one left out is the server's default. */
+type BrowserLimits = Pick<ImportLimits, 'maxBytes' | 'maxRows' | 'maxColumns' | 'maxCellLength' | 'maxZipEntries'>;
+interface ReadSheetFileOptions {
+    /**
+     * The file's name. A delimiter hint only, as on the server: once the bytes have said the file
+     * is text, a name ending in `.tsv` means tab. It never decides the format.
+     */
+    filename?: string;
+    limits?: Partial<BrowserLimits>;
+}
+/** What `readSheetFile` reads: the grid, what the bytes turned out to be, and a CSV's separator. */
+interface SheetFile extends SheetGrid {
+    format: 'csv' | 'xlsx';
+    /** The separator a CSV turned out to use; pass it to `decimalMarkFor`. Absent for a workbook. */
+    delimiter?: CsvDelimiter;
+}
+/** What a file can arrive as: a `File` or other `Blob`, or its bytes. */
+type SheetFileInput = Blob | ArrayBuffer | Uint8Array;
+/**
+ * Read a CSV, TSV or `.xlsx` file into exactly the grid the server's `readSheet` reads from the
+ * same bytes: the same headers, the same rows with the same typed cells, the same separator,
+ * and the same refusals with the same codes and messages.
+ *
+ * @throws ImportError `TOO_LARGE`, `SHEET_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `MALFORMED_FILE`,
+ *   `ARCHIVE_REFUSED`, as the server would. A workbook read without `exceljs` installed throws a
+ *   plain `Error` saying what to install.
+ */
+declare function readSheetFile(input: SheetFileInput, options?: ReadSheetFileOptions): Promise<SheetFile>;
+/**
+ * Read an `.xlsx` file by the server's rules, for a caller that already knows the format.
+ *
+ * @throws ImportError `UNSUPPORTED_FORMAT` for anything that is not a zip, and whatever
+ *   `readSheetFile` throws for a workbook.
+ */
+declare function readXlsx(input: SheetFileInput, options?: Omit<ReadSheetFileOptions, 'filename'>): Promise<SheetGrid>;
+
+export { type BrowserLimits, type ReadSheetFileOptions, type SheetFile, type SheetFileInput, readSheetFile, readXlsx };

package/dist/browser/index.d.ts

@@ -0,0 +1,41 @@
+import { SheetGrid, CsvDelimiter } from 'sheet-to-schema';
+import { I as ImportLimits } from '../import-error-Co6lRAqv.js';
+export { a as ImportError, b as ImportErrorCode } from '../import-error-Co6lRAqv.js';
+
+/** The limits a browser read enforces. Each one left out is the server's default. */
+type BrowserLimits = Pick<ImportLimits, 'maxBytes' | 'maxRows' | 'maxColumns' | 'maxCellLength' | 'maxZipEntries'>;
+interface ReadSheetFileOptions {
+    /**
+     * The file's name. A delimiter hint only, as on the server: once the bytes have said the file
+     * is text, a name ending in `.tsv` means tab. It never decides the format.
+     */
+    filename?: string;
+    limits?: Partial<BrowserLimits>;
+}
+/** What `readSheetFile` reads: the grid, what the bytes turned out to be, and a CSV's separator. */
+interface SheetFile extends SheetGrid {
+    format: 'csv' | 'xlsx';
+    /** The separator a CSV turned out to use; pass it to `decimalMarkFor`. Absent for a workbook. */
+    delimiter?: CsvDelimiter;
+}
+/** What a file can arrive as: a `File` or other `Blob`, or its bytes. */
+type SheetFileInput = Blob | ArrayBuffer | Uint8Array;
+/**
+ * Read a CSV, TSV or `.xlsx` file into exactly the grid the server's `readSheet` reads from the
+ * same bytes: the same headers, the same rows with the same typed cells, the same separator,
+ * and the same refusals with the same codes and messages.
+ *
+ * @throws ImportError `TOO_LARGE`, `SHEET_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `MALFORMED_FILE`,
+ *   `ARCHIVE_REFUSED`, as the server would. A workbook read without `exceljs` installed throws a
+ *   plain `Error` saying what to install.
+ */
+declare function readSheetFile(input: SheetFileInput, options?: ReadSheetFileOptions): Promise<SheetFile>;
+/**
+ * Read an `.xlsx` file by the server's rules, for a caller that already knows the format.
+ *
+ * @throws ImportError `UNSUPPORTED_FORMAT` for anything that is not a zip, and whatever
+ *   `readSheetFile` throws for a workbook.
+ */
+declare function readXlsx(input: SheetFileInput, options?: Omit<ReadSheetFileOptions, 'filename'>): Promise<SheetGrid>;
+
+export { type BrowserLimits, type ReadSheetFileOptions, type SheetFile, type SheetFileInput, readSheetFile, readXlsx };