@xeplr/actions 1.1.20 → 1.1.22

package/README.md

@@ -283,7 +283,7 @@ await actions.uploader.upload({
   signal,                 // optional AbortSignal: stop on request
   staged                  // optional: see Staged loads
 })
-// → { movementId, tables: { main, errors }, columns, totalRows, totalBatches, completed, dropped, aborted, merged, durationMs }
+// → { movementId, tables: { main, errors }, columns, totalRows, totalBatches, completed, dropped, errorRows, aborted, merged, durationMs }
 ```
 
 `onProgress({ rowsRead, batches, rowsWritten, phase? })` fires after each batch is read, with `rowsWritten` from the write queue's own count, so a caller can show both and see whether writing keeps up. After the read, while the queue drains, it fires again each time the written count moves (`phase: 'writing'`), never on a bare timer, so a stuck write goes quiet.
@@ -292,6 +292,8 @@ await actions.uploader.upload({
 
 Rows are spooled to NDJSON files on disk between source and database. From the first batch it creates, if missing: the target table, `<targetTable>_import_errors` (`movement_id`, `row_num`, `error_description`, `underlying_sql`, `raw_row`, `recorded_at`), a `__xeplr_movement_id__` column, and a unique index on `primaryKeys` (not on DuckDB: its upserts are staged and keep none). Server dialects also add `__xeplr_id__`; DuckDB does not.
 
+**Rows the database refuses are kept, in `<targetTable>_import_errors`.** When a batch fails, the queue splits it until it finds the rows the database will not take; each one is written to the error table with `row_num` (its 1-based place in the source, so the first data row is 1), `error_description` (the database's own message and code, e.g. `… violates check constraint "t_qty_check" [23514]`), the statement, and `raw_row` (the row as JSON, every column). They load nowhere else, so this table is the only way to see, fix and re-load them. `errorRows` in the result says how many were recorded: `{ recorded, notRecorded, reason? }`. Recording is best-effort and never fails the movement; `notRecorded` counts rows that could not be written (with the first `reason`). Rows dropped because the movement gave up (stopped, aborted) are counted in `dropped` but were never refused, so they are not recorded. Rolling a movement back deletes its error rows too. A queue's own `onErrorTable` hook still runs first. The statement is built once for every dialect (`lib/drivers/db/errorRows.js`), exposed per driver as `buildInsertErrorRowsSql(table, movementId, entries)`.
+
 Logical types and what each driver creates:
 
 | type | postgres | mysql | mssql | duckdb |
@@ -348,10 +350,16 @@ Upserts need no index: they are staged and match keys in the move. A read-only o
 | format | reads | notes |
 |---|---|---|
 | `csv` | stream | `formatConfig`: `columns` (default first row), `delimiter` (`,`), `cast` (true), `skipEmptyLines`, `trim`. |
-| `excel` | file path | `.xlsx` via `unzipper` + `saxes`; `sheet`, `startRow`. Date-styled cells arrive as `Date`. |
+| `excel` | file path | `.xlsx` via `unzipper` + `saxes`; `sheet`, `startRow`. Date-styled cells arrive as `Date`. Excel's own escapes in cell text are decoded (`_x000D_` → a real line break; `_x005F_` → `_`), and Japanese phonetic runs (`<rPh>`, furigana) are not joined into the value. |
 | `json` | stream | A JSON array or NDJSON; buffers the whole file. |
 | `txt` | — | Placeholder, not implemented. |
 
+**Rows pasted into one cell** (`lib/formats/pastedRows.js`). Data copied out of an ERP and pasted into Excel: a value starting with `"` (an inch mark) makes Excel put everything up to the next `"` — tabs, line breaks, whole rows — into that one cell. The rows it swallowed exist nowhere else. Seen for real: 465 rows folded into 11 cells.
+- `formats.scanPastedRows(format, filePath, { sheet })` → `{ cells, rows, examples: [{ row, column, rows, first }] }` — found, nothing changed. For an import's inspect step.
+- `formatConfig.repairPastedRows: true` on `file-upload` puts them back: the cell splits on its line breaks into the rest of its own row, whole rows, and the start of the last row, whose remaining values are the merged row's later cells. The eaten quotes go back where they opened and closed. The action's output says what it did: `repairedPastedRows: { cells, rows, at }`.
+- **Only an exact shape is repaired:** the pieces must have the sheet's column counts (and the cells the last row's values would leave empty must be empty). Ordinary multi-line text never matches, so a genuine note with line breaks is never touched. On the client's real file: 11 of 11 cells, 465 of 465 rows, no recovered entry number duplicating an existing one.
+- `pastedRows.decodeOoxml`, `splitPastedRows(rec, headers)`, `expandPastedRows(rows)`, `scanForPastedRows(rows)` for other readers.
+
 File sources: only `local` is implemented (`path`). `sftp`, `sharepoint` and `google` are placeholders.
 
 ## Movement metadata (`xeplr_configs`)
@@ -448,7 +456,7 @@ node --test test/uploader/reconcile.test.js
 
 | needs nothing | needs a database |
 |---|---|
-| `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
+| `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, which needs `@xeplr/db`: a peer dependency, installed for development) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js` (also honours `PG_HOST`/`PG_PORT`/`PG_USER`), `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
 | `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`; each skipped when unset), `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/duckdb-index-guard.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
 | | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
 | | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |

package/lib/builtins/file/upload.js

@@ -11,6 +11,7 @@
 var { upload } = require('../../uploader');
 var fileSources = require('../../drivers/file');
 var formats     = require('../../formats');
+var { expandPastedRows } = require('../../formats/pastedRows');
 var dbDrivers   = require('../../drivers/db');
 var { SqlQueue } = require('@xeplr/utils/lib/queue');
 var { generateId } = require('@xeplr/utils/lib/helpers');
@@ -82,6 +83,13 @@ module.exports = {
       var stream = await source.openStream(sourceCfg);
       rowsIterable = format.parseStream(stream, input.formatConfig || {});
     }
+    // Rows that a paste folded into one cell, put back — only when asked
+    // (formatConfig.repairPastedRows), after the person saw what was found.
+    // See lib/formats/pastedRows.js.
+    var repaired = [];
+    if (input.formatConfig && input.formatConfig.repairPastedRows) {
+      rowsIterable = expandPastedRows(rowsIterable, function(r) { repaired.push(r); });
+    }
 
     // Set up target
     var movementId = input.movementId || system.occurrenceId || ('mv_' + generateId());
@@ -97,7 +105,7 @@ module.exports = {
     });
 
     try {
-      return await upload({
+      var result = await upload({
         source:         rowsIterable,
         driver:         driver,
         connection:     pool,
@@ -120,6 +128,15 @@ module.exports = {
         onProgress:     system.onProgress || null,
         signal:         system.signal || null
       });
+      // What the repair put back, for the import's report.
+      if (repaired.length) {
+        result = Object.assign({}, result, { repairedPastedRows: {
+          cells: repaired.length,
+          rows: repaired.reduce(function(n, r) { return n + r.rows; }, 0),
+          at: repaired
+        } });
+      }
+      return result;
     } finally {
       queue.stop();
       await driver.close(pool);

package/lib/drivers/db/duckdb.js

@@ -966,6 +966,10 @@ function buildRetypeToLocalDateTimeSql(tableName, column) {
   return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' SET DATA TYPE TIMESTAMP';
 }
 
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
   buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
   requires: ['@duckdb/node-api'],
@@ -979,6 +983,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('postgres'), tableName, movementId, entries);
+  },
   buildUpsertIndexSql:      buildUpsertIndexSql,
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,

package/lib/drivers/db/errorRows.js

@@ -0,0 +1,37 @@
+// ROWS THE DATABASE REFUSED, written to `<table>_import_errors`.
+//
+// Every dialect creates that table (buildCreateErrorTableSql) with the same
+// columns; only the quoting and the bind marks differ. So the statement is
+// built once, here, and each driver passes its own quoteIdent and mark.
+
+var COLUMNS = ['movement_id', 'row_num', 'error_description', 'underlying_sql', 'raw_row'];
+
+/**
+ * One INSERT for several refused rows.
+ *
+ * @param {function} quoteIdent  the driver's identifier quoting
+ * @param {function} mark        bind mark for param n (0-based): '?', '$1', '@p0'
+ * @param {string}   tableName   the TARGET table — the error table is named from it
+ * @param {string}   movementId
+ * @param {Array<{rowNum, error, sql, row}>} entries
+ * @returns {{ sql: string, params: any[] }}
+ */
+function buildInsertErrorRowsSql(quoteIdent, mark, tableName, movementId, entries) {
+  var params = [];
+  var groups = entries.map(function(e) {
+    var marks = [];
+    [movementId, e.rowNum == null ? null : e.rowNum, e.error || null, e.sql || null,
+      e.row == null ? null : JSON.stringify(e.row)].forEach(function(v) {
+      marks.push(mark(params.length));
+      params.push(v);
+    });
+    return '(' + marks.join(', ') + ')';
+  });
+  return {
+    sql: 'INSERT INTO ' + quoteIdent(tableName + '_import_errors') +
+      ' (' + COLUMNS.map(quoteIdent).join(', ') + ') VALUES ' + groups.join(', '),
+    params: params
+  };
+}
+
+module.exports = { buildInsertErrorRowsSql: buildInsertErrorRowsSql };

package/lib/drivers/db/mssql.js

@@ -450,6 +450,10 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
   requires: ['mssql'],
   maxParams: MAX_PARAMS,
@@ -465,6 +469,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('mssql'), tableName, movementId, entries);
+  },
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,
 

package/lib/drivers/db/mysql.js

@@ -451,6 +451,10 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
   requires: ['mysql2'],
   maxParams: MAX_PARAMS,
@@ -466,6 +470,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('mysql'), tableName, movementId, entries);
+  },
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,
 

package/lib/drivers/db/postgres.js

@@ -403,6 +403,10 @@ function buildRetypeToLocalDateTimeSql(tableName, column) {
   return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' TYPE timestamp USING ' + quoteIdent(column) + " AT TIME ZONE 'UTC'";
 }
 
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
   buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
   requires: ['pg'],
@@ -418,6 +422,9 @@ module.exports = {
   quoteIdent:              quoteIdent,
   buildCreateTableSql:     buildCreateTableSql,
   buildCreateErrorTableSql:buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('postgres'), tableName, movementId, entries);
+  },
   buildUpsertIndexSql:     buildUpsertIndexSql,
   buildAlterTableAddSql:   buildAlterTableAddSql,
   buildInsertSql:          buildInsertSql,

package/lib/formats/_xlsx-stream.js

@@ -28,6 +28,7 @@
 var unzipper = require('unzipper');
 var { SaxesParser } = require('saxes');
 var { StringDecoder } = require('string_decoder');
+var { decodeOoxml } = require('./pastedRows');
 
 // Strip an XML namespace prefix: "x:row" -> "row", "row" -> "row".
 function localName(n) {
@@ -194,18 +195,25 @@ async function parseSharedStrings(directory) {
   var f = findFile(directory, 'xl/sharedStrings.xml');
   if (!f) return null;
   var strings = [];
-  var cur = null, inT = 0, buf = '';
+  // inRph: inside a phonetic run (<rPh>, the furigana Japanese Excel stores
+  // beside a cell's text). Its <t> is a reading aid, not the cell's value —
+  // joining it in gave "東京とうきょう" for 東京.
+  var cur = null, inT = 0, inRph = 0, buf = '';
   await saxParseStream(f.stream(), {
     opentag: function(tag) {
       var ln = localName(tag.name);
-      if      (ln === 'si') { cur = ''; }
-      else if (ln === 't')  { inT++; buf = ''; }
+      if      (ln === 'si')  { cur = ''; }
+      else if (ln === 'rPh') { inRph++; }
+      else if (ln === 't')   { inT++; buf = ''; }
     },
     text: function(t) { if (inT > 0) buf += t; },
     closetag: function(tag) {
       var ln = localName(tag.name);
-      if      (ln === 't')  { inT--; cur = (cur || '') + buf; buf = ''; }
-      else if (ln === 'si') { strings.push(cur || ''); cur = null; }
+      if      (ln === 't')   { inT--; if (!inRph) cur = (cur || '') + buf; buf = ''; }
+      else if (ln === 'rPh') { inRph--; }
+      // Excel's own escapes (_x000D_ for a line break inside the cell) decoded
+      // here, once per distinct string.
+      else if (ln === 'si')  { strings.push(decodeOoxml(cur || '')); cur = null; }
     }
   });
   return strings;
@@ -288,7 +296,7 @@ async function stream_xlsx_sheet_rows(filePath, sheet_name, handlers) {
     } else if (curType === 'b') {
       val = textBuf === '1';
     } else if (curType === 'str' || curType === 'inlineStr' || closingLocal === 't') {
-      val = textBuf;
+      val = decodeOoxml(textBuf);
     } else {
       var n = Number(textBuf);
       val = (textBuf !== '' && !Number.isNaN(n)) ? n : (textBuf === '' ? null : textBuf);

package/lib/formats/index.js

@@ -33,8 +33,27 @@ function checkFormatRequires(actionName, format, mod) {
   }
 }
 
+/**
+ * Which cells of a file hold rows a paste folded into them (pastedRows.js),
+ * read without changing anything — for an import's inspect step.
+ * opts: the format's own options (`sheet` for excel), plus `examples`.
+ */
+async function scanPastedRows(type, filePath, opts) {
+  var format = getFormat(type);
+  var config = opts || {};
+  var rows = format.inputMode === 'path'
+    ? format.parsePath(filePath, config)
+    : format.parseStream(require('fs').createReadStream(filePath), config);
+  return pastedRows.scanForPastedRows(rows, { examples: config.examples });
+}
+
+var pastedRows = require('./pastedRows');
+
 module.exports = {
   getFormat: getFormat,
   checkFormatRequires: checkFormatRequires,
-  SUPPORTED: Object.keys(FORMATS)
+  SUPPORTED: Object.keys(FORMATS),
+  scanPastedRows: scanPastedRows,
+  // decodeOoxml, splitPastedRows, expandPastedRows, scanForPastedRows
+  pastedRows: pastedRows
 };

package/lib/formats/pastedRows.js

@@ -0,0 +1,146 @@
+// ROWS PASTED INTO ONE CELL — found, and put back.
+//
+// Data copied out of an ERP is tab-separated text, a row per line. Pasted into
+// Excel, a value that STARTS with a double quote (an inch mark: `"4 INCH`)
+// switches Excel into quoted mode, and everything up to the next `"` — tabs,
+// line breaks, whole rows — lands in that one cell. The sheet then holds one
+// row whose cell carries the next N rows, and those rows exist nowhere else.
+// Seen for real: 465 rows folded into 11 Item Description cells.
+//
+// The damage has an exact shape, which is what makes it safe to undo:
+//
+//   cell = <its own row's value> \t <its own row's remaining columns>
+//          \n <a whole row> \n <a whole row> …
+//          \n <the last row's columns up to and including this one>
+//
+// and the merged row's OTHER columns after the cell belong to that last row.
+// So the cell splits on its line breaks into a first piece of (n - c) values,
+// whole rows of n values, and a last piece of L values — n columns, the cell
+// at column c. Usually L is c + 1 (the closing quote was in the same column);
+// when it closed in a later column k, L is k + 1, and the last row's
+// remaining (n - L) values sit in the merged row's cells right after c, the
+// cells beyond them empty. A cell that does not split exactly like that is
+// left alone: ordinary multi-line text never has those counts, so a genuine
+// note with line breaks is never "repaired".
+//
+// The quote that opened the cell and the one that closed it were eaten by the
+// paste; they are put back — at the start of the first row's value, where
+// quoted mode began, and at the end of the last row's value where it ended
+// (an inch mark: `Painting brush 4"`).
+
+// Excel's own escape inside cell text: `_xHHHH_` is the character with that
+// hex code. A line break inside a cell is saved as `_x000D_` + a real newline;
+// `_x005F_` is a literal underscore, which is how text that really says
+// "_x000D_" is stored ("_x005F_x000D_") — left to right, that decodes right.
+var ESCAPE = /_x([0-9A-Fa-f]{4})_/g;
+
+function decodeOoxml(s) {
+  if (typeof s !== 'string' || s.indexOf('_x') === -1) return s;
+  return s.replace(ESCAPE, function(_, hex) { return String.fromCharCode(parseInt(hex, 16)); });
+}
+
+var LINE = /\r\n|\r|\n/;
+var NUMBER = /^-?\d+(\.\d+)?$/;
+
+// A value from inside the cell: text, typed as the sheet's own cells would be.
+function cast(v) {
+  if (v === '') return null;
+  return NUMBER.test(v) ? Number(v) : v;
+}
+
+/**
+ * The rows folded into one of `rec`'s cells, put back — or null when no cell
+ * has that exact shape. `headers` in sheet order.
+ *
+ * @returns {{ column: string, rows: object[] } | null}  rows[0] is `rec` itself, rebuilt
+ */
+function splitPastedRows(rec, headers) {
+  var n = headers.length;
+  for (var c = 0; c < n; c++) {
+    var cell = decodeOoxml(rec[headers[c]]);
+    if (typeof cell !== 'string' || cell.indexOf('\t') === -1 || !LINE.test(cell)) continue;
+    var lines = cell.split(LINE);
+    var first = lines[0].split('\t');
+    var last = lines[lines.length - 1].split('\t');
+    var middle = lines.slice(1, -1).map(function(l) { return l.split('\t'); });
+    var L = last.length;
+    if (first.length !== n - c || L < c + 1 || L > n) continue;
+    if (middle.some(function(m) { return m.length !== n; })) continue;
+    // The last row's values after the closing quote: the merged row's cells
+    // right after c — and every cell beyond those must be empty, or this is
+    // not the shape a paste leaves.
+    var tail = headers.slice(c + 1, c + 1 + (n - L));
+    var beyond = headers.slice(c + 1 + (n - L));
+    if (beyond.some(function(h) { return rec[h] !== null && rec[h] !== undefined && rec[h] !== ''; })) continue;
+
+    var rows = [];
+    var top = {};
+    headers.forEach(function(h, i) { top[h] = i < c ? rec[h] : cast(first[i - c]); });
+    top[headers[c]] = '"' + first[0];
+    rows.push(top);
+    middle.forEach(function(m) {
+      var r = {};
+      headers.forEach(function(h, i) { r[h] = cast(m[i]); });
+      rows.push(r);
+    });
+    var end = {};
+    headers.forEach(function(h, i) { end[h] = i < L ? cast(last[i]) : rec[tail[i - L]]; });
+    // The closing quote, at the end of the value it closed.
+    end[headers[L - 1]] = String(last[L - 1]) + '"';
+    rows.push(end);
+    return { column: headers[c], rows: rows };
+  }
+  return null;
+}
+
+/**
+ * Rows as they were before the paste folded them: each record whose cell
+ * holds pasted rows becomes those rows. Everything else passes through.
+ * `onRepair({ row, column, rows })` hears each one (row: 1-based data row).
+ */
+async function* expandPastedRows(records, onRepair) {
+  var headers = null;
+  var index = 0;
+  for await (var rec of records) {
+    index++;
+    if (!headers) headers = Object.keys(rec);
+    var split = splitPastedRows(rec, headers);
+    if (!split) { yield rec; continue; }
+    if (onRepair) onRepair({ row: index, column: split.column, rows: split.rows.length - 1 });
+    for (var r of split.rows) yield r;
+  }
+}
+
+/**
+ * Read a whole file's rows and say which cells hold pasted rows, without
+ * changing anything — for the inspect step, before anything is loaded.
+ *
+ * @returns {{ cells: number, rows: number, examples: [{ row, column, rows, first }] }}
+ *          rows: how many rows the repair would recover
+ */
+async function scanForPastedRows(records, opts) {
+  var limit = (opts && opts.examples) || 5;
+  var headers = null;
+  var index = 0;
+  var found = { cells: 0, rows: 0, examples: [] };
+  for await (var rec of records) {
+    index++;
+    if (!headers) headers = Object.keys(rec);
+    var split = splitPastedRows(rec, headers);
+    if (!split) continue;
+    found.cells++;
+    found.rows += split.rows.length - 1;
+    if (found.examples.length < limit) {
+      found.examples.push({ row: index, column: split.column, rows: split.rows.length - 1,
+        first: String(split.rows[0][split.column]).slice(0, 80) });
+    }
+  }
+  return found;
+}
+
+module.exports = {
+  decodeOoxml: decodeOoxml,
+  splitPastedRows: splitPastedRows,
+  expandPastedRows: expandPastedRows,
+  scanForPastedRows: scanForPastedRows
+};

package/lib/uploader/index.js

@@ -96,6 +96,17 @@ async function upload(opts) {
   var onStop = opts.signal ? function() { opts.queue.abort(opts.movementId, 'stopped'); } : null;
   if (onStop) opts.signal.addEventListener('abort', onStop, { once: true });
 
+  // WHERE A REFUSED ROW GOES: `<targetTable>_import_errors`, with the
+  // database's reason. The queue splits a failing batch down to the rows the
+  // database refuses and hands each one to its onErrorTable hook — whose
+  // default does nothing, so every one of them used to vanish, leaving only a
+  // count ("63 dropped") and nothing anyone could look at, fix or re-load.
+  var refused = recordRefusedRows(opts, driver, pool);
+  // Each row's place in the source (1-based), to say WHICH row was refused.
+  // Keyed by the row object itself: a split batch slices the same objects, so
+  // the number survives however far the queue bisects.
+  var rowNumbers = refused.rowNumbers;
+
   try {
     var summary = await spool(opts.source, {
       runId:        opts.movementId,
@@ -156,6 +167,9 @@ async function upload(opts) {
         onBatch: async function(rows, meta) {
           if (opts.signal && opts.signal.aborted) throw stoppedError();
           if (!columns) throw new Error('uploader: onBatch before beforeAll — spool contract violated');
+          for (var n = 0; n < rows.length; n++) {
+            if (rows[n] && typeof rows[n] === 'object') rowNumbers.set(rows[n], rowsSeen + n + 1);
+          }
           for (var off = 0; off < rows.length; off += effectiveBatchSize) {
             var chunk = rows.slice(off, off + effectiveBatchSize);
             // A NAME FOR THIS BATCH, carried all the way to the failure.
@@ -231,6 +245,7 @@ async function upload(opts) {
     } finally {
       if (drainWatch) drainWatch.stop();
     }
+    var errorRows = await refused.flush();
     // Stopped while the queue drained: not a finished movement.
     if (opts.signal && opts.signal.aborted) throw stoppedError();
 
@@ -269,12 +284,17 @@ async function upload(opts) {
       totalBatches: totalBatches,
       completed:    stats.completed || 0,
       dropped:      stats.dropped || 0,
+      // Of the dropped rows, how many are in the error table to look at —
+      // { recorded, notRecorded, reason? }. Rows dropped without the database
+      // refusing them (a movement that gave up) are counted, never recorded.
+      errorRows:    errorRows,
       aborted:      !!stats.aborted,
       // Staged only: what the move put in the target — { inserted, replaced }.
       merged:       merged,
       durationMs:   Date.now() - startedAt
     };
   } finally {
+    refused.detach();
     // A staged load that did not move its rows: stopped, failed, aborted.
     // Its staging table is all it wrote, and dropping it is the whole undo.
     if (staged && !merged && pool) {
@@ -290,6 +310,66 @@ async function upload(opts) {
   }
 }
 
+// How many refused rows to write in one INSERT into the error table.
+var ERROR_ROWS_PER_INSERT = 200;
+
+/**
+ * Hook this movement into the queue's onErrorTable, writing every row the
+ * database refuses to `<targetTable>_import_errors` — row number, reason,
+ * the statement, and the row itself as JSON. Whatever hook the caller gave
+ * the queue still runs first. Writing is best-effort: a failure to record is
+ * counted and reported, never a failure of the movement.
+ *
+ * @returns {{ rowNumbers: WeakMap, flush: () => Promise<{recorded, notRecorded, reason?}>, detach: () => void }}
+ */
+function recordRefusedRows(opts, driver, pool) {
+  var queue = opts.queue;
+  var previous = queue.onErrorTable;
+  var rowNumbers = new WeakMap();
+  var pending = [];
+  var writes = [];
+  var tally = { recorded: 0, notRecorded: 0, reason: null };
+  var canWrite = typeof driver.buildInsertErrorRowsSql === 'function';
+
+  function write(entries) {
+    if (!canWrite) {
+      tally.notRecorded += entries.length;
+      tally.reason = tally.reason || 'this database driver cannot write the error table';
+      return;
+    }
+    var built = driver.buildInsertErrorRowsSql(opts.targetTable, opts.movementId, entries);
+    writes.push(driver.query(pool, built.sql, built.params).then(
+      function() { tally.recorded += entries.length; },
+      function(err) { tally.notRecorded += entries.length; tally.reason = tally.reason || err.message; }
+    ));
+  }
+
+  queue.onErrorTable = function(info) {
+    try { if (previous) previous.apply(queue, arguments); } catch (_) { /* the caller's hook cannot stop recording */ }
+    var item = info && info.item;
+    if (!item || item.movementId !== opts.movementId) return;
+    var rows = (item.meta && item.meta.rows) || [];
+    var reason = (info.error && info.error.message) || info.reason || 'unknown error';
+    if (info.errorCode != null) reason = reason + ' [' + info.errorCode + ']';
+    rows.forEach(function(row) {
+      pending.push({ rowNum: rowNumbers.get(row), error: reason, sql: rows.length === 1 ? item.sql : null, row: row });
+    });
+    while (pending.length >= ERROR_ROWS_PER_INSERT) write(pending.splice(0, ERROR_ROWS_PER_INSERT));
+  };
+
+  return {
+    rowNumbers: rowNumbers,
+    flush: async function() {
+      if (pending.length) write(pending.splice(0));
+      await Promise.all(writes);
+      var out = { recorded: tally.recorded, notRecorded: tally.notRecorded };
+      if (tally.reason) out.reason = tally.reason;
+      return out;
+    },
+    detach: function() { queue.onErrorTable = previous; }
+  };
+}
+
 /** The error a stopped movement ends with: code 'STOPPED', so a caller can tell it from a failure. */
 function stoppedError() {
   var err = new Error('Stopped by request');

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "1.1.20",
+  "version": "1.1.22",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "files": [
@@ -37,6 +37,7 @@
   },
   "peerDependencies": {
     "@duckdb/node-api": "1.5.5-r.3",
+    "@xeplr/db": "^1.0.8",
     "@xeplr/email": "^1.0.0",
     "@xeplr/utils": "^1.0.0",
     "csv-parse": "^7.0.2",
@@ -117,6 +118,7 @@
   },
   "devDependencies": {
     "@duckdb/node-api": "^1.5.5-r.3",
+    "@xeplr/db": "^1.0.8",
     "@xeplr/utils": "^1.0.0",
     "csv-parse": "^7.0.2",
     "dejavu-fonts-ttf": "^2.37.3",