@xeplr/actions 1.1.21 → 1.1.23

package/README.md

@@ -117,7 +117,7 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 | `dbListViews` | `db-list-views` | `{ views }`. |
 | `dbListProcedures` | `db-list-procedures` | `{ procedures }` (functions excluded). |
 | `dbListColumns` | `db-list-columns` | `{ columns }` — `name`, `dataType`, `udtName` and `maxLength` (characters; `-1` is SQL Server `(max)`, `null` when it has none) for one table. |
-| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
+| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`) as a New table, Append or Replace (`mode`, see [Import modes](#import-modes)). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
 | `sendEmail` | `email-send` | Send one message. See [Email](#email). |
 | `emailRead` | `email-read` | List messages, newest first, with `unread_only`, `limit` (1–1000, default 50), `since`, `from_equals`, `from_contains`, `subject_contains`, `body: 'full' \| 'text' \| 'none'`, optional NDJSON spooling. |
 | `emailMove` | `email-move` | Move a message (`messageId`, `folder`) to `toFolder`. |
@@ -283,7 +283,7 @@ await actions.uploader.upload({
   signal,                 // optional AbortSignal: stop on request
   staged                  // optional: see Staged loads
 })
-// → { movementId, tables: { main, errors }, columns, totalRows, totalBatches, completed, dropped, aborted, merged, durationMs }
+// → { movementId, tables: { main, errors }, columns, totalRows, totalBatches, completed, dropped, errorRows, aborted, merged, durationMs }
 ```
 
 `onProgress({ rowsRead, batches, rowsWritten, phase? })` fires after each batch is read, with `rowsWritten` from the write queue's own count, so a caller can show both and see whether writing keeps up. After the read, while the queue drains, it fires again each time the written count moves (`phase: 'writing'`), never on a bare timer, so a stuck write goes quiet.
@@ -292,6 +292,8 @@ await actions.uploader.upload({
 
 Rows are spooled to NDJSON files on disk between source and database. From the first batch it creates, if missing: the target table, `<targetTable>_import_errors` (`movement_id`, `row_num`, `error_description`, `underlying_sql`, `raw_row`, `recorded_at`), a `__xeplr_movement_id__` column, and a unique index on `primaryKeys` (not on DuckDB: its upserts are staged and keep none). Server dialects also add `__xeplr_id__`; DuckDB does not.
 
+**Rows the database refuses are kept, in `<targetTable>_import_errors`.** When a batch fails, the queue splits it until it finds the rows the database will not take; each one is written to the error table with `row_num` (its 1-based place in the source, so the first data row is 1), `error_description` (the database's own message and code, e.g. `… violates check constraint "t_qty_check" [23514]`), the statement, and `raw_row` (the row as JSON, every column). They load nowhere else, so this table is the only way to see, fix and re-load them. `errorRows` in the result says how many were recorded: `{ recorded, notRecorded, reason? }`. Recording is best-effort and never fails the movement; `notRecorded` counts rows that could not be written (with the first `reason`). Rows dropped because the movement gave up (stopped, aborted) are counted in `dropped` but were never refused, so they are not recorded. Rolling a movement back deletes its error rows too. A queue's own `onErrorTable` hook still runs first. The statement is built once for every dialect (`lib/drivers/db/errorRows.js`), exposed per driver as `buildInsertErrorRowsSql(table, movementId, entries)`.
+
 Logical types and what each driver creates:
 
 | type | postgres | mysql | mssql | duckdb |
@@ -329,6 +331,23 @@ With `staged: true` a load **never writes into its target while it runs**. Batch
 
 A driver opts in by implementing `prepareStaging`, `mergeStaged`, `dropStaging` and `committedMovement`; `upload` refuses `staged` for one that does not. A driver that sets `stagesUpserts: true` (DuckDB) has every upsert staged, asked or not.
 
+### Import modes
+
+`mode` says what the target must be, checked before a row is read:
+
+| mode | the table | then |
+|---|---|---|
+| `'new'` | must not exist | created from the file. One that exists is refused ("already exists. Choose Append … or Replace …"), nothing written |
+| `'append'` | must exist | the rows are added. A name that matches no table is refused: a typo is not a new table |
+| `'replace'` | must exist | **staged**: the rows go to `<table>__xeplr_staging` while the import runs; when all are in, one transaction empties the table and moves them in. Stopped, failed or killed before that, the table is as it was. `merged` is `{ inserted, replaced }`, `replaced` being the rows it held before. No `primaryKeys` (that is an upsert), no `replaceFrom` |
+| omitted | | created if missing, else appended (as before modes existed) |
+
+A refused target throws with `code: 'TARGET_REFUSED'`. The mode is recorded on the import (`import_meta.meta.mode`), and **a Replace cannot be rolled back**: deleting its rows would leave the table empty, not as it was, so `uploader.rollback` refuses it.
+
+On Postgres, MySQL and SQL Server the staging table is shaped like the target (`lib/drivers/db/staging.js`; each driver gives only how to create it and how to run a transaction), so a row the target refuses is refused while staged and recorded in the error table like any other. Postgres copies NOT NULL, defaults and CHECKs (`LIKE … INCLUDING DEFAULTS INCLUDING CONSTRAINTS`); MySQL copies everything (`CREATE TABLE … LIKE`); SQL Server copies columns, NULL-ness and identity but not CHECKs (`SELECT TOP 0 * INTO`), so there a row only a CHECK refuses fails the final move, which leaves the table as it was. These three stage only a whole-table Replace (`stagedReplaceOnly`); DuckDB stages upserts and range replaces too.
+
+**Before loading: the file against the table.** `uploader.compareToTable({ driver, connection, targetTable, columns })` → `{ exists, matched, extra, missing: [{ name, required }] }`, matched by exact name, the framework's columns left out. `extra` (in the file, not the table) means the import will be refused; `missing` columns are left empty, and `required` ones (no empty value, no default) mean every row will be refused. `formats.columnsOf(format, filePath, { sheet? })` gives a file's columns from its first row. Every driver's `getTableSchema` now says `nullable` and `hasDefault` (an identity or auto-increment counts as a default).
+
 ### A written DuckDB file keeps no index
 
 DuckDB 1.5.4–1.5.6 has a bug ([duckdb/duckdb#26106](https://github.com/duckdb/duckdb/issues/26106), open upstream): when a process dies with rows still only in the WAL, the next open replays them into the table and the next checkpoint drops them from every index on it. The first `DELETE` that reaches one fails with *"Failed to delete all rows from index"*, which is **FATAL**: the whole file is unusable until the process restarts. Any index, unique or not. A table with no index is unaffected.
@@ -454,7 +473,7 @@ node --test test/uploader/reconcile.test.js
 
 | needs nothing | needs a database |
 |---|---|
-| `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
+| `test/import-modes.test.js` (DuckDB in memory; its Postgres cases run when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, its MySQL cases when `MYSQL_HOST`/`MYSQL_PORT`/`MYSQL_USER`/`MYSQL_PASSWORD` are), `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, which needs `@xeplr/db`: a peer dependency, installed for development) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js` (also honours `PG_HOST`/`PG_PORT`/`PG_USER`), `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
 | `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`; each skipped when unset), `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/duckdb-index-guard.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
 | | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
 | | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |

package/lib/builtins/file/upload.js

@@ -43,6 +43,9 @@ module.exports = {
     { name: 'dbConnection',  type: 'object',  required: true, order: 11 },
     { name: 'targetTable',   type: 'string',  required: true, order: 12 },
     { name: 'primaryKeys',   type: 'array',   order: 13, group: 'Advanced' },
+    { name: 'mode',          type: 'string',  order: 14, group: 'Advanced',
+      description: "What the target must be: 'new' (refused if it exists), 'append' (it must exist) or 'replace' " +
+        "(it must exist; its rows are replaced in one transaction once every row is in). Omitted: created if missing, else appended." },
 
     // CONTROL
     { name: 'movementId',    type: 'string',  order: 20, group: 'Advanced' },
@@ -111,6 +114,7 @@ module.exports = {
         connection:     pool,
         targetTable:    input.targetTable,
         primaryKeys:    (input.primaryKeys && input.primaryKeys.length) ? input.primaryKeys : null,
+        mode:           input.mode || null,
         movementId:     movementId,
         queue:          queue,
         connectionName: input.targetTable,

package/lib/drivers/db/duckdb.js

@@ -611,11 +611,12 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
 // 'public' here returns an empty column list for a table that plainly exists.
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT column_name, data_type FROM information_schema.columns ' +
+    'SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns ' +
     'WHERE table_schema = $1 AND table_name = $2 ORDER BY ordinal_position',
     [schema || 'main', tableName]);
   return r.rows.map(function(row) {
-    return { name: row.column_name, dataType: row.data_type, udtName: row.data_type };
+    return { name: row.column_name, dataType: row.data_type, udtName: row.data_type,
+      nullable: row.is_nullable === 'YES', hasDefault: row.column_default != null };
   });
 }
 
@@ -913,6 +914,12 @@ async function mergeStaged(pool, opts) {
       // away from it (a DATE read as UTC midnight here, compared at the
       // source's local midnight there) — and a day deleted but never read
       // back is a day lost. Nothing staged: MIN is NULL, nothing is deleted.
+      // THE WHOLE TABLE REPLACED: an import in Replace mode. Its rows go and
+      // the staged ones take their place, in this same transaction.
+      if (opts.replaceAll) {
+        var all = await conn.run('DELETE FROM ' + target);
+        replaced = Number(all.rowsChanged) || 0;
+      }
       if (opts.replaceFrom) {
         var col = quoteIdent(opts.replaceFrom.column);
         var cut = await conn.run('DELETE FROM ' + target + ' WHERE ' + col + ' >= (SELECT MIN(' + col + ') FROM ' + staging + ')');
@@ -966,6 +973,10 @@ function buildRetypeToLocalDateTimeSql(tableName, column) {
   return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' SET DATA TYPE TIMESTAMP';
 }
 
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
   buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
   requires: ['@duckdb/node-api'],
@@ -979,6 +990,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('postgres'), tableName, movementId, entries);
+  },
   buildUpsertIndexSql:      buildUpsertIndexSql,
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,

package/lib/drivers/db/errorRows.js

@@ -0,0 +1,37 @@
+// ROWS THE DATABASE REFUSED, written to `<table>_import_errors`.
+//
+// Every dialect creates that table (buildCreateErrorTableSql) with the same
+// columns; only the quoting and the bind marks differ. So the statement is
+// built once, here, and each driver passes its own quoteIdent and mark.
+
+var COLUMNS = ['movement_id', 'row_num', 'error_description', 'underlying_sql', 'raw_row'];
+
+/**
+ * One INSERT for several refused rows.
+ *
+ * @param {function} quoteIdent  the driver's identifier quoting
+ * @param {function} mark        bind mark for param n (0-based): '?', '$1', '@p0'
+ * @param {string}   tableName   the TARGET table — the error table is named from it
+ * @param {string}   movementId
+ * @param {Array<{rowNum, error, sql, row}>} entries
+ * @returns {{ sql: string, params: any[] }}
+ */
+function buildInsertErrorRowsSql(quoteIdent, mark, tableName, movementId, entries) {
+  var params = [];
+  var groups = entries.map(function(e) {
+    var marks = [];
+    [movementId, e.rowNum == null ? null : e.rowNum, e.error || null, e.sql || null,
+      e.row == null ? null : JSON.stringify(e.row)].forEach(function(v) {
+      marks.push(mark(params.length));
+      params.push(v);
+    });
+    return '(' + marks.join(', ') + ')';
+  });
+  return {
+    sql: 'INSERT INTO ' + quoteIdent(tableName + '_import_errors') +
+      ' (' + COLUMNS.map(quoteIdent).join(', ') + ') VALUES ' + groups.join(', '),
+    params: params
+  };
+}
+
+module.exports = { buildInsertErrorRowsSql: buildInsertErrorRowsSql };

package/lib/drivers/db/mssql.js

@@ -303,7 +303,8 @@ function coerceValue(v, type) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, CHARACTER_MAXIMUM_LENGTH AS maxLength, IS_NULLABLE AS isNullable, ' +
+    'COLUMN_DEFAULT AS columnDefault, COLUMNPROPERTY(OBJECT_ID(QUOTENAME(TABLE_SCHEMA) + \'.\' + QUOTENAME(TABLE_NAME)), COLUMN_NAME, \'IsIdentity\') AS isIdentity ' +
     'FROM INFORMATION_SCHEMA.COLUMNS ' +
     'WHERE TABLE_NAME = @p0 AND (@p1 IS NULL OR TABLE_SCHEMA = @p1) ' +
     'ORDER BY ORDINAL_POSITION',
@@ -311,7 +312,8 @@ async function getTableSchema(pool, tableName, schema) {
   );
   return r.rows.map(function(row) {
     // maxLength -1 is (max): nvarchar(max), varchar(max), varbinary(max).
-    return { name: row.name, dataType: row.dataType, udtName: row.dataType, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
+    return { name: row.name, dataType: row.dataType, udtName: row.dataType, maxLength: row.maxLength == null ? null : Number(row.maxLength),
+      nullable: row.isNullable === 'YES', hasDefault: row.columnDefault != null || Number(row.isIdentity) === 1 };
   });
 }
 
@@ -450,7 +452,41 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// STAGED REPLACE — see staging.js. SELECT … INTO copies the columns, their
+// NULL-ness and the identity, but not CHECK or DEFAULT constraints: a row only
+// a CHECK refuses fails the final move, which then leaves the target as it was.
+var staged = require('./staging').stagingFor({
+  name: 'mssql', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['SELECT TOP 0 * INTO ' + s + ' FROM ' + t]; },
+  transaction: async function(pool, fn) {
+    var sql = require('mssql');
+    var tx = new sql.Transaction(pool);
+    await tx.begin();
+    try {
+      var out = await fn(async function(text) {
+        var r = await new sql.Request(tx).query(text);
+        return (r.rowsAffected || []).reduce(function(a, b) { return a + b; }, 0);
+      });
+      await tx.commit();
+      return out;
+    } catch (err) {
+      await tx.rollback().catch(function() {});
+      throw err;
+    }
+  }
+});
+
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   requires: ['mssql'],
   maxParams: MAX_PARAMS,
 
@@ -465,6 +501,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('mssql'), tableName, movementId, entries);
+  },
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,
 

package/lib/drivers/db/mysql.js

@@ -302,14 +302,16 @@ function coerceValue(v, type) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName, CHARACTER_MAXIMUM_LENGTH AS maxLength, ' +
+    'IS_NULLABLE AS isNullable, COLUMN_DEFAULT AS columnDefault, EXTRA AS extra ' +
     'FROM information_schema.COLUMNS ' +
     'WHERE TABLE_SCHEMA = COALESCE(?, DATABASE()) AND TABLE_NAME = ? ' +
     'ORDER BY ORDINAL_POSITION',
     [schema || null, tableName]
   );
   return r.rows.map(function(row) {
-    return { name: row.name, dataType: row.dataType, udtName: row.udtName, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
+    return { name: row.name, dataType: row.dataType, udtName: row.udtName, maxLength: row.maxLength == null ? null : Number(row.maxLength),
+      nullable: row.isNullable === 'YES', hasDefault: row.columnDefault != null || /auto_increment|DEFAULT_GENERATED/i.test(row.extra || '') };
   });
 }
 
@@ -451,7 +453,38 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// STAGED REPLACE — see staging.js. CREATE TABLE … LIKE copies the columns,
+// NOT NULL, defaults, indexes and (8.0.16+) CHECK constraints.
+var staged = require('./staging').stagingFor({
+  name: 'mysql', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['CREATE TABLE ' + s + ' LIKE ' + t]; },
+  transaction: async function(pool, fn) {
+    var conn = await pool.getConnection();
+    try {
+      await conn.beginTransaction();
+      var out = await fn(async function(sql) { var r = await conn.query(sql); return (r[0] && r[0].affectedRows) || 0; });
+      await conn.commit();
+      return out;
+    } catch (err) {
+      await conn.rollback().catch(function() {});
+      throw err;
+    } finally {
+      conn.release();
+    }
+  }
+});
+
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   requires: ['mysql2'],
   maxParams: MAX_PARAMS,
 
@@ -466,6 +499,9 @@ module.exports = {
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('mysql'), tableName, movementId, entries);
+  },
   buildAlterTableAddSql:    buildAlterTableAddSql,
   buildInsertSql:           buildInsertSql,
 

package/lib/drivers/db/postgres.js

@@ -289,7 +289,7 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
 // in ordinal-position order. Empty array if the table doesn't exist.
 async function getTableSchema(pool, tableName, schema) {
   var r = await pool.query(
-    "SELECT column_name, data_type, udt_name, character_maximum_length " +
+    "SELECT column_name, data_type, udt_name, character_maximum_length, is_nullable, column_default, is_identity " +
     "FROM information_schema.columns " +
     "WHERE table_schema = $2 AND table_name = $1 " +
     "ORDER BY ordinal_position",
@@ -297,7 +297,8 @@ async function getTableSchema(pool, tableName, schema) {
   );
   return r.rows.map(function(row) {
     return { name: row.column_name, dataType: row.data_type, udtName: row.udt_name,
-      maxLength: row.character_maximum_length == null ? null : Number(row.character_maximum_length) };
+      maxLength: row.character_maximum_length == null ? null : Number(row.character_maximum_length),
+      nullable: row.is_nullable === 'YES', hasDefault: row.column_default != null || row.is_identity === 'YES' };
   });
 }
 
@@ -403,7 +404,39 @@ function buildRetypeToLocalDateTimeSql(tableName, column) {
   return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' TYPE timestamp USING ' + quoteIdent(column) + " AT TIME ZONE 'UTC'";
 }
 
+// STAGED REPLACE — see staging.js. LIKE brings NOT NULL, defaults and CHECKs,
+// so a row the target would refuse is refused while it is staged; indexes stay
+// behind (a whole-table replace matches no keys).
+var staged = require('./staging').stagingFor({
+  name: 'postgres', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['CREATE TABLE ' + s + ' (LIKE ' + t + ' INCLUDING DEFAULTS INCLUDING CONSTRAINTS)']; },
+  transaction: async function(pool, fn) {
+    var client = await pool.connect();
+    try {
+      await client.query('BEGIN');
+      var out = await fn(async function(sql) { return (await client.query(sql)).rowCount || 0; });
+      await client.query('COMMIT');
+      return out;
+    } catch (err) {
+      await client.query('ROLLBACK').catch(function() {});
+      throw err;
+    } finally {
+      client.release();
+    }
+  }
+});
+
+// Rows the database refused, into <table>_import_errors — one builder for every dialect.
+var errorRows = require('./errorRows');
+var placeholders = require('./procedure').placeholders;
+
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
   requires: ['pg'],
 
@@ -418,6 +451,9 @@ module.exports = {
   quoteIdent:              quoteIdent,
   buildCreateTableSql:     buildCreateTableSql,
   buildCreateErrorTableSql:buildCreateErrorTableSql,
+  buildInsertErrorRowsSql: function(tableName, movementId, entries) {
+    return errorRows.buildInsertErrorRowsSql(quoteIdent, placeholders('postgres'), tableName, movementId, entries);
+  },
   buildUpsertIndexSql:     buildUpsertIndexSql,
   buildAlterTableAddSql:   buildAlterTableAddSql,
   buildInsertSql:          buildInsertSql,

package/lib/drivers/db/staging.js

@@ -0,0 +1,70 @@
+// A STAGED REPLACE on the server databases (Postgres, MySQL, SQL Server).
+//
+// An import that replaces a table must never leave it empty because the
+// import failed. So the rows are written to `<target>__xeplr_staging` while
+// the import runs, and only when every one is in does ONE transaction empty
+// the target and move them across. Stopped, failed or killed before that:
+// the target is exactly as it was, and the staging table is all there is to
+// drop (the next load drops a leftover one anyway).
+//
+// The staging table is shaped like the target, with its NOT NULL and (where
+// the dialect copies them) CHECK constraints — so a row the target would
+// refuse is refused while it is staged, one row at a time, and recorded in
+// `<target>_import_errors` like any other, rather than failing the move.
+//
+// Each driver supplies only what differs: how to make the staging table and
+// how to run statements in one transaction. DuckDB keeps its own (it also
+// upserts and replaces ranges there — see its "staged loads" section).
+
+var SUFFIX = '__xeplr_staging';
+var MOVEMENT_COL = '__xeplr_movement_id__';
+
+function stagingTableFor(tableName) { return tableName + SUFFIX; }
+
+/**
+ * @param {object} dialect - { name, quoteIdent, query(pool, sql), createStagingSql(target, staging) → string[],
+ *                             transaction(pool, fn(run)) where run(sql) → rows affected }
+ */
+function stagingFor(dialect) {
+  var q = dialect.quoteIdent;
+
+  async function prepareStaging(pool, tableName) {
+    var staging = stagingTableFor(tableName);
+    await dropStaging(pool, tableName);
+    var sqls = dialect.createStagingSql(q(tableName), q(staging));
+    for (var i = 0; i < sqls.length; i++) await dialect.query(pool, sqls[i]);
+    return staging;
+  }
+
+  async function dropStaging(pool, tableName) {
+    await dialect.query(pool, dialect.dropTableSql(q(stagingTableFor(tableName))));
+  }
+
+  /**
+   * Replace the target's rows with the staged ones, in one transaction, then
+   * drop the staging table. Only a whole-table replace: upserts and range
+   * replaces are staged on DuckDB alone.
+   *
+   * @returns {{ inserted, replaced }} replaced: the rows the target held before
+   */
+  async function mergeStaged(pool, opts) {
+    if (!opts.replaceAll || (opts.primaryKeys && opts.primaryKeys.length) || opts.replaceFrom) {
+      throw new Error(dialect.name + ': a staged load replaces the whole table here — upserts and range replaces are not staged on this database');
+    }
+    var target = q(opts.targetTable);
+    var staging = q(stagingTableFor(opts.targetTable));
+    var cols = [MOVEMENT_COL].concat(opts.columns.map(function(c) { return c.name; })).map(q).join(', ');
+    var counts = await dialect.transaction(pool, async function(run) {
+      var replaced = await run('DELETE FROM ' + target);
+      var inserted = await run('INSERT INTO ' + target + ' (' + cols + ') SELECT ' + cols + ' FROM ' + staging);
+      return { inserted: inserted, replaced: replaced };
+    });
+    // After the commit, and allowed to fail: the rows have landed.
+    await dropStaging(pool, opts.targetTable).catch(function() {});
+    return counts;
+  }
+
+  return { prepareStaging: prepareStaging, dropStaging: dropStaging, mergeStaged: mergeStaged };
+}
+
+module.exports = { stagingFor: stagingFor, stagingTableFor: stagingTableFor, SUFFIX: SUFFIX };

package/lib/formats/index.js

@@ -47,6 +47,27 @@ async function scanPastedRows(type, filePath, opts) {
   return pastedRows.scanForPastedRows(rows, { examples: config.examples });
 }
 
+/**
+ * A file's column names — its header (CSV), one sheet's (Excel), or the first
+ * record's keys (JSON) — read from the first row, the rest left unread. What
+ * an import compares with an existing table before loading anything.
+ * opts: the format's own options (`sheet` for excel).
+ */
+async function columnsOf(type, filePath, opts) {
+  var format = getFormat(type);
+  var config = opts || {};
+  var rows = format.inputMode === 'path'
+    ? format.parsePath(filePath, config)
+    : format.parseStream(require('fs').createReadStream(filePath), config);
+  var iterator = rows[Symbol.asyncIterator]();
+  try {
+    var first = await iterator.next();
+    return first.done || !first.value ? [] : Object.keys(first.value);
+  } finally {
+    if (typeof iterator.return === 'function') await iterator.return();
+  }
+}
+
 var pastedRows = require('./pastedRows');
 
 module.exports = {
@@ -54,6 +75,7 @@ module.exports = {
   checkFormatRequires: checkFormatRequires,
   SUPPORTED: Object.keys(FORMATS),
   scanPastedRows: scanPastedRows,
+  columnsOf: columnsOf,
   // decodeOoxml, splitPastedRows, expandPastedRows, scanForPastedRows
   pastedRows: pastedRows
 };

package/lib/uploader/index.js

@@ -30,6 +30,9 @@
 //     keepBatchFiles,  // default false — clean up NDJSON files after success
 //     replaceFrom,     // optional, staged only: { column } — the move deletes target
 //                       // rows from the earliest staged value on first (see mergeStaged)
+//     mode,            // optional: 'new' | 'append' | 'replace' — what the target
+//                       // must be (see MODES). Omitted: create it if missing,
+//                       // else append, as before modes existed.
 //     staged,          // default false. true: batches land in <targetTable>__xeplr_staging
 //                       // and move into the target in ONE transaction at the end, so a
 //                       // load that stops or dies midway never touched the target.
@@ -96,7 +99,23 @@ async function upload(opts) {
   var onStop = opts.signal ? function() { opts.queue.abort(opts.movementId, 'stopped'); } : null;
   if (onStop) opts.signal.addEventListener('abort', onStop, { once: true });
 
+  // WHERE A REFUSED ROW GOES: `<targetTable>_import_errors`, with the
+  // database's reason. The queue splits a failing batch down to the rows the
+  // database refuses and hands each one to its onErrorTable hook — whose
+  // default does nothing, so every one of them used to vanish, leaving only a
+  // count ("63 dropped") and nothing anyone could look at, fix or re-load.
+  var refused = recordRefusedRows(opts, driver, pool);
+  // Each row's place in the source (1-based), to say WHICH row was refused.
+  // Keyed by the row object itself: a split batch slices the same objects, so
+  // the number survives however far the queue bisects.
+  var rowNumbers = refused.rowNumbers;
+
   try {
+    // WHAT THE TARGET MUST BE, checked before a row is read: a New table must
+    // not exist, and Append and Replace need one that does — a typo in the
+    // name would otherwise make a fresh table and "append" to nothing.
+    if (opts.mode) await checkMode(opts, driver, pool);
+
     var summary = await spool(opts.source, {
       runId:        opts.movementId,
       batchDir:     opts.batchDir,
@@ -143,6 +162,7 @@ async function upload(opts) {
             dbType:        opts.dbType,            // recorded when the caller supplies it
             columns:       columns,
             primaryKeys:   opts.primaryKeys,
+            mode:          opts.mode || null,
             mtId1: opts.mtId1, mtId2: opts.mtId2, mtId3: opts.mtId3, mtId4: opts.mtId4,
             details:       opts.details
           });
@@ -156,6 +176,9 @@ async function upload(opts) {
         onBatch: async function(rows, meta) {
           if (opts.signal && opts.signal.aborted) throw stoppedError();
           if (!columns) throw new Error('uploader: onBatch before beforeAll — spool contract violated');
+          for (var n = 0; n < rows.length; n++) {
+            if (rows[n] && typeof rows[n] === 'object') rowNumbers.set(rows[n], rowsSeen + n + 1);
+          }
           for (var off = 0; off < rows.length; off += effectiveBatchSize) {
             var chunk = rows.slice(off, off + effectiveBatchSize);
             // A NAME FOR THIS BATCH, carried all the way to the failure.
@@ -231,6 +254,7 @@ async function upload(opts) {
     } finally {
       if (drainWatch) drainWatch.stop();
     }
+    var errorRows = await refused.flush();
     // Stopped while the queue drained: not a finished movement.
     if (opts.signal && opts.signal.aborted) throw stoppedError();
 
@@ -250,7 +274,8 @@ async function upload(opts) {
         columns:     columns,
         primaryKeys: opts.primaryKeys,
         movementId:  opts.movementId,
-        replaceFrom: opts.replaceFrom
+        replaceFrom: opts.replaceFrom,
+        replaceAll:  opts.replaceAll
       });
     }
     await opts.metaStore.recordEnd(opts.movementId, {
@@ -269,12 +294,19 @@ async function upload(opts) {
       totalBatches: totalBatches,
       completed:    stats.completed || 0,
       dropped:      stats.dropped || 0,
+      // Of the dropped rows, how many are in the error table to look at —
+      // { recorded, notRecorded, reason? }. Rows dropped without the database
+      // refusing them (a movement that gave up) are counted, never recorded.
+      errorRows:    errorRows,
       aborted:      !!stats.aborted,
       // Staged only: what the move put in the target — { inserted, replaced }.
+      // For a Replace, `replaced` is how many rows the table held before.
       merged:       merged,
+      mode:         opts.mode || null,
       durationMs:   Date.now() - startedAt
     };
   } finally {
+    refused.detach();
     // A staged load that did not move its rows: stopped, failed, aborted.
     // Its staging table is all it wrote, and dropping it is the whole undo.
     if (staged && !merged && pool) {
@@ -290,6 +322,66 @@ async function upload(opts) {
   }
 }
 
+// How many refused rows to write in one INSERT into the error table.
+var ERROR_ROWS_PER_INSERT = 200;
+
+/**
+ * Hook this movement into the queue's onErrorTable, writing every row the
+ * database refuses to `<targetTable>_import_errors` — row number, reason,
+ * the statement, and the row itself as JSON. Whatever hook the caller gave
+ * the queue still runs first. Writing is best-effort: a failure to record is
+ * counted and reported, never a failure of the movement.
+ *
+ * @returns {{ rowNumbers: WeakMap, flush: () => Promise<{recorded, notRecorded, reason?}>, detach: () => void }}
+ */
+function recordRefusedRows(opts, driver, pool) {
+  var queue = opts.queue;
+  var previous = queue.onErrorTable;
+  var rowNumbers = new WeakMap();
+  var pending = [];
+  var writes = [];
+  var tally = { recorded: 0, notRecorded: 0, reason: null };
+  var canWrite = typeof driver.buildInsertErrorRowsSql === 'function';
+
+  function write(entries) {
+    if (!canWrite) {
+      tally.notRecorded += entries.length;
+      tally.reason = tally.reason || 'this database driver cannot write the error table';
+      return;
+    }
+    var built = driver.buildInsertErrorRowsSql(opts.targetTable, opts.movementId, entries);
+    writes.push(driver.query(pool, built.sql, built.params).then(
+      function() { tally.recorded += entries.length; },
+      function(err) { tally.notRecorded += entries.length; tally.reason = tally.reason || err.message; }
+    ));
+  }
+
+  queue.onErrorTable = function(info) {
+    try { if (previous) previous.apply(queue, arguments); } catch (_) { /* the caller's hook cannot stop recording */ }
+    var item = info && info.item;
+    if (!item || item.movementId !== opts.movementId) return;
+    var rows = (item.meta && item.meta.rows) || [];
+    var reason = (info.error && info.error.message) || info.reason || 'unknown error';
+    if (info.errorCode != null) reason = reason + ' [' + info.errorCode + ']';
+    rows.forEach(function(row) {
+      pending.push({ rowNum: rowNumbers.get(row), error: reason, sql: rows.length === 1 ? item.sql : null, row: row });
+    });
+    while (pending.length >= ERROR_ROWS_PER_INSERT) write(pending.splice(0, ERROR_ROWS_PER_INSERT));
+  };
+
+  return {
+    rowNumbers: rowNumbers,
+    flush: async function() {
+      if (pending.length) write(pending.splice(0));
+      await Promise.all(writes);
+      var out = { recorded: tally.recorded, notRecorded: tally.notRecorded };
+      if (tally.reason) out.reason = tally.reason;
+      return out;
+    },
+    detach: function() { queue.onErrorTable = previous; }
+  };
+}
+
 /** The error a stopped movement ends with: code 'STOPPED', so a caller can tell it from a failure. */
 function stoppedError() {
   var err = new Error('Stopped by request');
@@ -316,6 +408,13 @@ async function rollback(opts) {
 
   if (opts.metaStore && opts.metaStore.getById) {
     var meta = await opts.metaStore.getById(opts.movementId);
+    // A REPLACE emptied the table before its rows went in. Deleting its rows
+    // now would leave the table empty, not as it was — the old rows are gone.
+    var metaOf = meta && meta.meta && typeof meta.meta === 'string' ? JSON.parse(meta.meta) : (meta && meta.meta);
+    if (metaOf && metaOf.mode === 'replace') {
+      throw new Error('rollback: this import replaced the table\'s rows — they cannot be brought back by deleting the new ones, ' +
+        'so it cannot be rolled back. Import the earlier data again to restore it.');
+    }
     if (meta && meta.primary_keys && meta.primary_keys.length) {
       throw new Error(
         'rollback: this movement used upsert (primary keys: ' + meta.primary_keys.join(', ') + ') — ' +
@@ -378,6 +477,18 @@ function normalizeOptions(opts) {
   if (opts.driver && opts.driver.stagesUpserts && (opts.primaryKeys || []).length) opts.staged = true;
   opts.writeTable         = null;
   opts.replaceFrom        = opts.replaceFrom || null;
+  if (opts.mode != null && MODES.indexOf(opts.mode) === -1) {
+    throw new Error('upload: mode must be one of ' + MODES.join(', ') + ' — got "' + opts.mode + '"');
+  }
+  // REPLACE IS STAGED: the rows go to a staging table while the import runs,
+  // and one transaction empties the target and moves them in at the end — so
+  // an import that fails or is stopped never leaves the table empty.
+  opts.replaceAll = opts.mode === 'replace';
+  if (opts.replaceAll) {
+    if ((opts.primaryKeys || []).length) throw new Error('upload: Replace empties the table first, so it takes no primaryKeys (that is an upsert)');
+    if (opts.replaceFrom) throw new Error('upload: Replace replaces the whole table; replaceFrom replaces a range — choose one');
+    opts.staged = true;
+  }
   if (opts.replaceFrom) {
     if (!opts.replaceFrom.column) throw new Error('upload: replaceFrom needs a column');
     if ((opts.primaryKeys || []).length) throw new Error('upload: replaceFrom replaces a range, so it takes no primaryKeys');
@@ -386,9 +497,71 @@ function normalizeOptions(opts) {
   if (opts.staged && typeof opts.driver.mergeStaged !== 'function') {
     throw new Error('upload: staged loads are not supported for this database — its driver has no mergeStaged');
   }
+  if (opts.staged && opts.driver.stagedReplaceOnly && !opts.replaceAll) {
+    throw new Error('upload: on this database a load is staged only to replace a whole table (mode: replace) — not for upserts or replaceFrom');
+  }
   return opts;
 }
 
+// What an import may do to its target. 'new': make it — refused if it exists.
+// 'append': add to it — it must exist. 'replace': empty it and load — it must
+// exist; staged, so a failed import leaves it as it was.
+var MODES = ['new', 'append', 'replace'];
+
+async function checkMode(opts, driver, pool) {
+  var schema = await driver.getTableSchema(pool, opts.targetTable);
+  var exists = !!(schema && schema.length);
+  if (opts.mode === 'new' && exists) {
+    throw refusal('Table "' + opts.targetTable + '" already exists. Choose Append to add to it or Replace to replace its rows, or give a new name.');
+  }
+  if (opts.mode !== 'new' && !exists) {
+    throw refusal('Table "' + opts.targetTable + '" does not exist, so there is nothing to ' + (opts.mode === 'append' ? 'append to' : 'replace') +
+      '. Check the name, or choose New table.');
+  }
+}
+
+function refusal(message) {
+  var err = new Error(message);
+  err.code = 'TARGET_REFUSED';
+  return err;
+}
+
+/**
+ * The file's columns against an existing table — what an import would do,
+ * said before it does it. Matched by exact name; the framework's own columns
+ * are left out.
+ *
+ * @param {object} opts - { driver, connection (config or pool), targetTable, columns: string[] }
+ * @returns {Promise<{ exists, matched: string[], extra: string[], missing: [{ name, required }] }>}
+ *   extra: in the file, not the table — the import would be refused.
+ *   missing: in the table, not the file — left empty; `required` when the
+ *   column takes no empty value and has no default, so every row would be refused.
+ */
+async function compareToTable(opts) {
+  var driver = opts.driver;
+  var poolIsExternal = looksLikePool(opts.connection);
+  var pool = poolIsExternal ? opts.connection : await driver.connect(opts.connection);
+  try {
+    var schema = (await driver.getTableSchema(pool, opts.targetTable)) || [];
+    var table = schema.filter(function(c) { return c.name.indexOf('__xeplr_') !== 0; });
+    var fileCols = (opts.columns || []).filter(function(n) { return n != null && n !== ''; }).map(String);
+    var inTable = {};
+    table.forEach(function(c) { inTable[c.name] = c; });
+    var inFile = {};
+    fileCols.forEach(function(n) { inFile[n] = true; });
+    return {
+      exists: schema.length > 0,
+      matched: fileCols.filter(function(n) { return inTable[n]; }),
+      extra: schema.length ? fileCols.filter(function(n) { return !inTable[n]; }) : [],
+      missing: table.filter(function(c) { return !inFile[c.name]; }).map(function(c) {
+        return { name: c.name, required: c.nullable === false && !c.hasDefault };
+      })
+    };
+  } finally {
+    if (!poolIsExternal) await driver.close(pool);
+  }
+}
+
 function safeBatchSize(columnCount, maxParams) {
   var paramsPerRow = columnCount + 1;   // +1 for __xeplr_movement_id__
   return Math.max(1, Math.floor(maxParams / paramsPerRow));
@@ -521,5 +694,7 @@ module.exports = {
   wasCommitted:    wasCommitted,
   inferColumns:    inferColumns,
   inferColumnType: inferColumnType,
+  compareToTable:  compareToTable,
+  MODES:           MODES,
   makeMetaStore:   require('./meta-store-knex')
 };

package/lib/uploader/meta-store-knex.js

@@ -88,6 +88,9 @@ module.exports = function makeKnexMetaStore(knex, options) {
           status:         'running',
           primary_keys:   plan.primaryKeys ? JSON.stringify(plan.primaryKeys) : null,
           columns:        plan.columns     ? JSON.stringify(plan.columns)     : null,
+          // What the import did to its target ('new' | 'append' | 'replace'):
+          // a Replace cannot be rolled back, and rollback reads it here.
+          meta:           plan.mode ? JSON.stringify({ mode: plan.mode }) : null,
           error:          null,
           started_at:         knex.fn.now(),
           ended_at:           null,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "1.1.21",
+  "version": "1.1.23",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "files": [
@@ -37,6 +37,7 @@
   },
   "peerDependencies": {
     "@duckdb/node-api": "1.5.5-r.3",
+    "@xeplr/db": "^1.0.8",
     "@xeplr/email": "^1.0.0",
     "@xeplr/utils": "^1.0.0",
     "csv-parse": "^7.0.2",
@@ -117,6 +118,7 @@
   },
   "devDependencies": {
     "@duckdb/node-api": "^1.5.5-r.3",
+    "@xeplr/db": "^1.0.8",
     "@xeplr/utils": "^1.0.0",
     "csv-parse": "^7.0.2",
     "dejavu-fonts-ttf": "^2.37.3",