@xeplr/actions 1.1.22 → 1.1.24

package/README.md

@@ -117,7 +117,7 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 | `dbListViews` | `db-list-views` | `{ views }`. |
 | `dbListProcedures` | `db-list-procedures` | `{ procedures }` (functions excluded). |
 | `dbListColumns` | `db-list-columns` | `{ columns }` — `name`, `dataType`, `udtName` and `maxLength` (characters; `-1` is SQL Server `(max)`, `null` when it has none) for one table. |
-| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
+| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`) as a New table, Append or Replace (`mode`, see [Import modes](#import-modes)). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
 | `sendEmail` | `email-send` | Send one message. See [Email](#email). |
 | `emailRead` | `email-read` | List messages, newest first, with `unread_only`, `limit` (1–1000, default 50), `since`, `from_equals`, `from_contains`, `subject_contains`, `body: 'full' \| 'text' \| 'none'`, optional NDJSON spooling. |
 | `emailMove` | `email-move` | Move a message (`messageId`, `folder`) to `toFolder`. |
@@ -331,6 +331,27 @@ With `staged: true` a load **never writes into its target while it runs**. Batch
 
 A driver opts in by implementing `prepareStaging`, `mergeStaged`, `dropStaging` and `committedMovement`; `upload` refuses `staged` for one that does not. A driver that sets `stagesUpserts: true` (DuckDB) has every upsert staged, asked or not.
 
+### Import modes
+
+`mode` says what the target must be, checked before a row is read:
+
+| mode | the table | then |
+|---|---|---|
+| `'new'` | must not exist | created from the file. One that exists is refused ("already exists. Choose Append … or Replace …"), nothing written |
+| `'append'` | must exist | the rows are added. A name that matches no table is refused: a typo is not a new table |
+| `'replace'` | must exist | **staged**: the rows go to `<table>__xeplr_staging` while the import runs; when all are in, one transaction empties the table and moves them in. Stopped, failed or killed before that, the table is as it was. `merged` is `{ inserted, replaced }`, `replaced` being the rows it held before. No `primaryKeys` (that is an upsert), no `replaceFrom` |
+| omitted | | created if missing, else appended (as before modes existed) |
+
+A refused target throws with `code: 'TARGET_REFUSED'`. The mode is recorded on the import (`import_meta.meta.mode`), and **a Replace cannot be rolled back**: deleting its rows would leave the table empty, not as it was, so `uploader.rollback` refuses it.
+
+On Postgres, MySQL and SQL Server the staging table is shaped like the target (`lib/drivers/db/staging.js`; each driver gives only how to create it and how to run a transaction), so a row the target refuses is refused while staged and recorded in the error table like any other. Postgres copies NOT NULL, defaults and CHECKs (`LIKE … INCLUDING DEFAULTS INCLUDING CONSTRAINTS`); MySQL copies everything (`CREATE TABLE … LIKE`); SQL Server copies columns, NULL-ness and identity but not CHECKs (`SELECT TOP 0 * INTO`), so there a row only a CHECK refuses fails the final move, which leaves the table as it was. These three stage only a whole-table Replace (`stagedReplaceOnly`); DuckDB stages upserts and range replaces too.
+
+**Before loading: the file against the table.** `uploader.compareToTable({ driver, connection, targetTable, columns })` → `{ exists, matched, extra, missing: [{ name, required }] }`, matched by exact name, the framework's columns left out. `extra` (in the file, not the table) means the import will be refused; `missing` columns are left empty, and `required` ones (no empty value, no default) mean every row will be refused. `formats.columnsOf(format, filePath, { sheet? })` gives a file's columns from its first row. Every driver's `getTableSchema` now says `nullable` and `hasDefault` (an identity or auto-increment counts as a default).
+
+### The uploader's own tables
+
+Beside the tables it loads, the uploader makes three of its own: `<table>_import_errors` (refused rows), `<table>__xeplr_staging` (a staged load in progress) and, in a DuckDB file, `__xeplr_loads` (its ledger). `isInternalTable(name)` (also accepts a listing row `{ name }`) says whether a table is one of these, so an app listing a database's tables for people to pick from can leave them out. They still exist and are read where they belong.
+
 ### A written DuckDB file keeps no index
 
 DuckDB 1.5.4–1.5.6 has a bug ([duckdb/duckdb#26106](https://github.com/duckdb/duckdb/issues/26106), open upstream): when a process dies with rows still only in the WAL, the next open replays them into the table and the next checkpoint drops them from every index on it. The first `DELETE` that reaches one fails with *"Failed to delete all rows from index"*, which is **FATAL**: the whole file is unusable until the process restarts. Any index, unique or not. A table with no index is unaffected.
@@ -456,7 +477,7 @@ node --test test/uploader/reconcile.test.js
 
 | needs nothing | needs a database |
 |---|---|
-| `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, which needs `@xeplr/db`: a peer dependency, installed for development) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js` (also honours `PG_HOST`/`PG_PORT`/`PG_USER`), `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
+| `test/import-modes.test.js` (DuckDB in memory; its Postgres cases run when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, its MySQL cases when `MYSQL_HOST`/`MYSQL_PORT`/`MYSQL_USER`/`MYSQL_PASSWORD` are), `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, which needs `@xeplr/db`: a peer dependency, installed for development) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js` (also honours `PG_HOST`/`PG_PORT`/`PG_USER`), `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
 | `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`; each skipped when unset), `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/duckdb-index-guard.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
 | | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
 | | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |

package/index.js

@@ -37,6 +37,8 @@ module.exports = {
   // their own — e.g. a UI wizard showing inferred columns from a CSV sample
   // before actually committing to an import).
   uploader:      uploader,
+  // The uploader's own tables (_import_errors, __xeplr_staging, __xeplr_loads), to leave out of table pickers.
+  isInternalTable: require('./lib/drivers/db/internalTables').isInternalTable,
   inferColumns:  uploader.inferColumns,
   streaming:     streaming,
   drivers:       { db: dbDrivers, file: fileDrivers },

package/lib/builtins/file/upload.js

@@ -43,6 +43,9 @@ module.exports = {
     { name: 'dbConnection',  type: 'object',  required: true, order: 11 },
     { name: 'targetTable',   type: 'string',  required: true, order: 12 },
     { name: 'primaryKeys',   type: 'array',   order: 13, group: 'Advanced' },
+    { name: 'mode',          type: 'string',  order: 14, group: 'Advanced',
+      description: "What the target must be: 'new' (refused if it exists), 'append' (it must exist) or 'replace' " +
+        "(it must exist; its rows are replaced in one transaction once every row is in). Omitted: created if missing, else appended." },
 
     // CONTROL
     { name: 'movementId',    type: 'string',  order: 20, group: 'Advanced' },
@@ -111,6 +114,7 @@ module.exports = {
         connection:     pool,
         targetTable:    input.targetTable,
         primaryKeys:    (input.primaryKeys && input.primaryKeys.length) ? input.primaryKeys : null,
+        mode:           input.mode || null,
         movementId:     movementId,
         queue:          queue,
         connectionName: input.targetTable,

package/lib/drivers/db/duckdb.js

@@ -611,11 +611,12 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
 // 'public' here returns an empty column list for a table that plainly exists.
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT column_name, data_type FROM information_schema.columns ' +
+    'SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns ' +
     'WHERE table_schema = $1 AND table_name = $2 ORDER BY ordinal_position',
     [schema || 'main', tableName]);
   return r.rows.map(function(row) {
-    return { name: row.column_name, dataType: row.data_type, udtName: row.data_type };
+    return { name: row.column_name, dataType: row.data_type, udtName: row.data_type,
+      nullable: row.is_nullable === 'YES', hasDefault: row.column_default != null };
   });
 }
 
@@ -913,6 +914,12 @@ async function mergeStaged(pool, opts) {
       // away from it (a DATE read as UTC midnight here, compared at the
       // source's local midnight there) — and a day deleted but never read
       // back is a day lost. Nothing staged: MIN is NULL, nothing is deleted.
+      // THE WHOLE TABLE REPLACED: an import in Replace mode. Its rows go and
+      // the staged ones take their place, in this same transaction.
+      if (opts.replaceAll) {
+        var all = await conn.run('DELETE FROM ' + target);
+        replaced = Number(all.rowsChanged) || 0;
+      }
       if (opts.replaceFrom) {
         var col = quoteIdent(opts.replaceFrom.column);
         var cut = await conn.run('DELETE FROM ' + target + ' WHERE ' + col + ' >= (SELECT MIN(' + col + ') FROM ' + staging + ')');

package/lib/drivers/db/internalTables.js

@@ -0,0 +1,31 @@
+// THE TABLES THE UPLOADER MAKES FOR ITSELF, beside the tables it loads:
+//
+//   <table>_import_errors    rows the database refused (errorRows.js)
+//   <table>__xeplr_staging   a staged load's rows until they move in (staging.js, duckdb.js)
+//   __xeplr_loads            a DuckDB file's ledger of staged loads (duckdb.js)
+//
+// They are the uploader's bookkeeping, not data anybody models: an app that
+// lists a database's tables for people to pick from (a dataset builder, a
+// cube's source picker) leaves them out with isInternalTable(). They still
+// exist and are read where they belong (an import's dropped rows).
+
+var ERRORS_SUFFIX = '_import_errors';
+var STAGING_SUFFIX = require('./staging').SUFFIX;
+var LOADS_LEDGER = '__xeplr_loads';
+
+function endsWith(s, suffix) {
+  return s.length > suffix.length && s.slice(-suffix.length) === suffix;
+}
+
+/** Whether a table is one the uploader made for itself. Case as the database reports it. */
+function isInternalTable(name) {
+  var s = String(name == null ? '' : (typeof name === 'object' ? name.name || name.table || '' : name));
+  return endsWith(s, ERRORS_SUFFIX) || endsWith(s, STAGING_SUFFIX) || s === LOADS_LEDGER;
+}
+
+module.exports = {
+  isInternalTable: isInternalTable,
+  ERRORS_SUFFIX: ERRORS_SUFFIX,
+  STAGING_SUFFIX: STAGING_SUFFIX,
+  LOADS_LEDGER: LOADS_LEDGER
+};

package/lib/drivers/db/mssql.js

@@ -303,7 +303,8 @@ function coerceValue(v, type) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, CHARACTER_MAXIMUM_LENGTH AS maxLength, IS_NULLABLE AS isNullable, ' +
+    'COLUMN_DEFAULT AS columnDefault, COLUMNPROPERTY(OBJECT_ID(QUOTENAME(TABLE_SCHEMA) + \'.\' + QUOTENAME(TABLE_NAME)), COLUMN_NAME, \'IsIdentity\') AS isIdentity ' +
     'FROM INFORMATION_SCHEMA.COLUMNS ' +
     'WHERE TABLE_NAME = @p0 AND (@p1 IS NULL OR TABLE_SCHEMA = @p1) ' +
     'ORDER BY ORDINAL_POSITION',
@@ -311,7 +312,8 @@ async function getTableSchema(pool, tableName, schema) {
   );
   return r.rows.map(function(row) {
     // maxLength -1 is (max): nvarchar(max), varchar(max), varbinary(max).
-    return { name: row.name, dataType: row.dataType, udtName: row.dataType, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
+    return { name: row.name, dataType: row.dataType, udtName: row.dataType, maxLength: row.maxLength == null ? null : Number(row.maxLength),
+      nullable: row.isNullable === 'YES', hasDefault: row.columnDefault != null || Number(row.isIdentity) === 1 };
   });
 }
 
@@ -450,11 +452,41 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// STAGED REPLACE — see staging.js. SELECT … INTO copies the columns, their
+// NULL-ness and the identity, but not CHECK or DEFAULT constraints: a row only
+// a CHECK refuses fails the final move, which then leaves the target as it was.
+var staged = require('./staging').stagingFor({
+  name: 'mssql', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['SELECT TOP 0 * INTO ' + s + ' FROM ' + t]; },
+  transaction: async function(pool, fn) {
+    var sql = require('mssql');
+    var tx = new sql.Transaction(pool);
+    await tx.begin();
+    try {
+      var out = await fn(async function(text) {
+        var r = await new sql.Request(tx).query(text);
+        return (r.rowsAffected || []).reduce(function(a, b) { return a + b; }, 0);
+      });
+      await tx.commit();
+      return out;
+    } catch (err) {
+      await tx.rollback().catch(function() {});
+      throw err;
+    }
+  }
+});
+
 // Rows the database refused, into <table>_import_errors — one builder for every dialect.
 var errorRows = require('./errorRows');
 var placeholders = require('./procedure').placeholders;
 
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   requires: ['mssql'],
   maxParams: MAX_PARAMS,
 

package/lib/drivers/db/mysql.js

@@ -302,14 +302,16 @@ function coerceValue(v, type) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName, CHARACTER_MAXIMUM_LENGTH AS maxLength, ' +
+    'IS_NULLABLE AS isNullable, COLUMN_DEFAULT AS columnDefault, EXTRA AS extra ' +
     'FROM information_schema.COLUMNS ' +
     'WHERE TABLE_SCHEMA = COALESCE(?, DATABASE()) AND TABLE_NAME = ? ' +
     'ORDER BY ORDINAL_POSITION',
     [schema || null, tableName]
   );
   return r.rows.map(function(row) {
-    return { name: row.name, dataType: row.dataType, udtName: row.udtName, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
+    return { name: row.name, dataType: row.dataType, udtName: row.udtName, maxLength: row.maxLength == null ? null : Number(row.maxLength),
+      nullable: row.isNullable === 'YES', hasDefault: row.columnDefault != null || /auto_increment|DEFAULT_GENERATED/i.test(row.extra || '') };
   });
 }
 
@@ -451,11 +453,38 @@ async function rollbackMovement(pool, tableName, movementId) {
   return result;
 }
 
+// STAGED REPLACE — see staging.js. CREATE TABLE … LIKE copies the columns,
+// NOT NULL, defaults, indexes and (8.0.16+) CHECK constraints.
+var staged = require('./staging').stagingFor({
+  name: 'mysql', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['CREATE TABLE ' + s + ' LIKE ' + t]; },
+  transaction: async function(pool, fn) {
+    var conn = await pool.getConnection();
+    try {
+      await conn.beginTransaction();
+      var out = await fn(async function(sql) { var r = await conn.query(sql); return (r[0] && r[0].affectedRows) || 0; });
+      await conn.commit();
+      return out;
+    } catch (err) {
+      await conn.rollback().catch(function() {});
+      throw err;
+    } finally {
+      conn.release();
+    }
+  }
+});
+
 // Rows the database refused, into <table>_import_errors — one builder for every dialect.
 var errorRows = require('./errorRows');
 var placeholders = require('./procedure').placeholders;
 
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   requires: ['mysql2'],
   maxParams: MAX_PARAMS,
 

package/lib/drivers/db/postgres.js

@@ -289,7 +289,7 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
 // in ordinal-position order. Empty array if the table doesn't exist.
 async function getTableSchema(pool, tableName, schema) {
   var r = await pool.query(
-    "SELECT column_name, data_type, udt_name, character_maximum_length " +
+    "SELECT column_name, data_type, udt_name, character_maximum_length, is_nullable, column_default, is_identity " +
     "FROM information_schema.columns " +
     "WHERE table_schema = $2 AND table_name = $1 " +
     "ORDER BY ordinal_position",
@@ -297,7 +297,8 @@ async function getTableSchema(pool, tableName, schema) {
   );
   return r.rows.map(function(row) {
     return { name: row.column_name, dataType: row.data_type, udtName: row.udt_name,
-      maxLength: row.character_maximum_length == null ? null : Number(row.character_maximum_length) };
+      maxLength: row.character_maximum_length == null ? null : Number(row.character_maximum_length),
+      nullable: row.is_nullable === 'YES', hasDefault: row.column_default != null || row.is_identity === 'YES' };
   });
 }
 
@@ -403,11 +404,39 @@ function buildRetypeToLocalDateTimeSql(tableName, column) {
   return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' TYPE timestamp USING ' + quoteIdent(column) + " AT TIME ZONE 'UTC'";
 }
 
+// STAGED REPLACE — see staging.js. LIKE brings NOT NULL, defaults and CHECKs,
+// so a row the target would refuse is refused while it is staged; indexes stay
+// behind (a whole-table replace matches no keys).
+var staged = require('./staging').stagingFor({
+  name: 'postgres', quoteIdent: quoteIdent, query: query,
+  dropTableSql: function(t) { return 'DROP TABLE IF EXISTS ' + t; },
+  createStagingSql: function(t, s) { return ['CREATE TABLE ' + s + ' (LIKE ' + t + ' INCLUDING DEFAULTS INCLUDING CONSTRAINTS)']; },
+  transaction: async function(pool, fn) {
+    var client = await pool.connect();
+    try {
+      await client.query('BEGIN');
+      var out = await fn(async function(sql) { return (await client.query(sql)).rowCount || 0; });
+      await client.query('COMMIT');
+      return out;
+    } catch (err) {
+      await client.query('ROLLBACK').catch(function() {});
+      throw err;
+    } finally {
+      client.release();
+    }
+  }
+});
+
 // Rows the database refused, into <table>_import_errors — one builder for every dialect.
 var errorRows = require('./errorRows');
 var placeholders = require('./procedure').placeholders;
 
 module.exports = {
+  prepareStaging: staged.prepareStaging,
+  dropStaging:    staged.dropStaging,
+  mergeStaged:    staged.mergeStaged,
+  // Staged only to replace a whole table (an import's Replace mode).
+  stagedReplaceOnly: true,
   buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
   requires: ['pg'],
 

package/lib/drivers/db/staging.js

@@ -0,0 +1,70 @@
+// A STAGED REPLACE on the server databases (Postgres, MySQL, SQL Server).
+//
+// An import that replaces a table must never leave it empty because the
+// import failed. So the rows are written to `<target>__xeplr_staging` while
+// the import runs, and only when every one is in does ONE transaction empty
+// the target and move them across. Stopped, failed or killed before that:
+// the target is exactly as it was, and the staging table is all there is to
+// drop (the next load drops a leftover one anyway).
+//
+// The staging table is shaped like the target, with its NOT NULL and (where
+// the dialect copies them) CHECK constraints — so a row the target would
+// refuse is refused while it is staged, one row at a time, and recorded in
+// `<target>_import_errors` like any other, rather than failing the move.
+//
+// Each driver supplies only what differs: how to make the staging table and
+// how to run statements in one transaction. DuckDB keeps its own (it also
+// upserts and replaces ranges there — see its "staged loads" section).
+
+var SUFFIX = '__xeplr_staging';
+var MOVEMENT_COL = '__xeplr_movement_id__';
+
+function stagingTableFor(tableName) { return tableName + SUFFIX; }
+
+/**
+ * @param {object} dialect - { name, quoteIdent, query(pool, sql), createStagingSql(target, staging) → string[],
+ *                             transaction(pool, fn(run)) where run(sql) → rows affected }
+ */
+function stagingFor(dialect) {
+  var q = dialect.quoteIdent;
+
+  async function prepareStaging(pool, tableName) {
+    var staging = stagingTableFor(tableName);
+    await dropStaging(pool, tableName);
+    var sqls = dialect.createStagingSql(q(tableName), q(staging));
+    for (var i = 0; i < sqls.length; i++) await dialect.query(pool, sqls[i]);
+    return staging;
+  }
+
+  async function dropStaging(pool, tableName) {
+    await dialect.query(pool, dialect.dropTableSql(q(stagingTableFor(tableName))));
+  }
+
+  /**
+   * Replace the target's rows with the staged ones, in one transaction, then
+   * drop the staging table. Only a whole-table replace: upserts and range
+   * replaces are staged on DuckDB alone.
+   *
+   * @returns {{ inserted, replaced }} replaced: the rows the target held before
+   */
+  async function mergeStaged(pool, opts) {
+    if (!opts.replaceAll || (opts.primaryKeys && opts.primaryKeys.length) || opts.replaceFrom) {
+      throw new Error(dialect.name + ': a staged load replaces the whole table here — upserts and range replaces are not staged on this database');
+    }
+    var target = q(opts.targetTable);
+    var staging = q(stagingTableFor(opts.targetTable));
+    var cols = [MOVEMENT_COL].concat(opts.columns.map(function(c) { return c.name; })).map(q).join(', ');
+    var counts = await dialect.transaction(pool, async function(run) {
+      var replaced = await run('DELETE FROM ' + target);
+      var inserted = await run('INSERT INTO ' + target + ' (' + cols + ') SELECT ' + cols + ' FROM ' + staging);
+      return { inserted: inserted, replaced: replaced };
+    });
+    // After the commit, and allowed to fail: the rows have landed.
+    await dropStaging(pool, opts.targetTable).catch(function() {});
+    return counts;
+  }
+
+  return { prepareStaging: prepareStaging, dropStaging: dropStaging, mergeStaged: mergeStaged };
+}
+
+module.exports = { stagingFor: stagingFor, stagingTableFor: stagingTableFor, SUFFIX: SUFFIX };

package/lib/formats/index.js

@@ -47,6 +47,27 @@ async function scanPastedRows(type, filePath, opts) {
   return pastedRows.scanForPastedRows(rows, { examples: config.examples });
 }
 
+/**
+ * A file's column names — its header (CSV), one sheet's (Excel), or the first
+ * record's keys (JSON) — read from the first row, the rest left unread. What
+ * an import compares with an existing table before loading anything.
+ * opts: the format's own options (`sheet` for excel).
+ */
+async function columnsOf(type, filePath, opts) {
+  var format = getFormat(type);
+  var config = opts || {};
+  var rows = format.inputMode === 'path'
+    ? format.parsePath(filePath, config)
+    : format.parseStream(require('fs').createReadStream(filePath), config);
+  var iterator = rows[Symbol.asyncIterator]();
+  try {
+    var first = await iterator.next();
+    return first.done || !first.value ? [] : Object.keys(first.value);
+  } finally {
+    if (typeof iterator.return === 'function') await iterator.return();
+  }
+}
+
 var pastedRows = require('./pastedRows');
 
 module.exports = {
@@ -54,6 +75,7 @@ module.exports = {
   checkFormatRequires: checkFormatRequires,
   SUPPORTED: Object.keys(FORMATS),
   scanPastedRows: scanPastedRows,
+  columnsOf: columnsOf,
   // decodeOoxml, splitPastedRows, expandPastedRows, scanForPastedRows
   pastedRows: pastedRows
 };

package/lib/uploader/index.js

@@ -30,6 +30,9 @@
 //     keepBatchFiles,  // default false — clean up NDJSON files after success
 //     replaceFrom,     // optional, staged only: { column } — the move deletes target
 //                       // rows from the earliest staged value on first (see mergeStaged)
+//     mode,            // optional: 'new' | 'append' | 'replace' — what the target
+//                       // must be (see MODES). Omitted: create it if missing,
+//                       // else append, as before modes existed.
 //     staged,          // default false. true: batches land in <targetTable>__xeplr_staging
 //                       // and move into the target in ONE transaction at the end, so a
 //                       // load that stops or dies midway never touched the target.
@@ -108,6 +111,11 @@ async function upload(opts) {
   var rowNumbers = refused.rowNumbers;
 
   try {
+    // WHAT THE TARGET MUST BE, checked before a row is read: a New table must
+    // not exist, and Append and Replace need one that does — a typo in the
+    // name would otherwise make a fresh table and "append" to nothing.
+    if (opts.mode) await checkMode(opts, driver, pool);
+
     var summary = await spool(opts.source, {
       runId:        opts.movementId,
       batchDir:     opts.batchDir,
@@ -154,6 +162,7 @@ async function upload(opts) {
             dbType:        opts.dbType,            // recorded when the caller supplies it
             columns:       columns,
             primaryKeys:   opts.primaryKeys,
+            mode:          opts.mode || null,
             mtId1: opts.mtId1, mtId2: opts.mtId2, mtId3: opts.mtId3, mtId4: opts.mtId4,
             details:       opts.details
           });
@@ -265,7 +274,8 @@ async function upload(opts) {
         columns:     columns,
         primaryKeys: opts.primaryKeys,
         movementId:  opts.movementId,
-        replaceFrom: opts.replaceFrom
+        replaceFrom: opts.replaceFrom,
+        replaceAll:  opts.replaceAll
       });
     }
     await opts.metaStore.recordEnd(opts.movementId, {
@@ -290,7 +300,9 @@ async function upload(opts) {
       errorRows:    errorRows,
       aborted:      !!stats.aborted,
       // Staged only: what the move put in the target — { inserted, replaced }.
+      // For a Replace, `replaced` is how many rows the table held before.
       merged:       merged,
+      mode:         opts.mode || null,
       durationMs:   Date.now() - startedAt
     };
   } finally {
@@ -396,6 +408,13 @@ async function rollback(opts) {
 
   if (opts.metaStore && opts.metaStore.getById) {
     var meta = await opts.metaStore.getById(opts.movementId);
+    // A REPLACE emptied the table before its rows went in. Deleting its rows
+    // now would leave the table empty, not as it was — the old rows are gone.
+    var metaOf = meta && meta.meta && typeof meta.meta === 'string' ? JSON.parse(meta.meta) : (meta && meta.meta);
+    if (metaOf && metaOf.mode === 'replace') {
+      throw new Error('rollback: this import replaced the table\'s rows — they cannot be brought back by deleting the new ones, ' +
+        'so it cannot be rolled back. Import the earlier data again to restore it.');
+    }
     if (meta && meta.primary_keys && meta.primary_keys.length) {
       throw new Error(
         'rollback: this movement used upsert (primary keys: ' + meta.primary_keys.join(', ') + ') — ' +
@@ -458,6 +477,18 @@ function normalizeOptions(opts) {
   if (opts.driver && opts.driver.stagesUpserts && (opts.primaryKeys || []).length) opts.staged = true;
   opts.writeTable         = null;
   opts.replaceFrom        = opts.replaceFrom || null;
+  if (opts.mode != null && MODES.indexOf(opts.mode) === -1) {
+    throw new Error('upload: mode must be one of ' + MODES.join(', ') + ' — got "' + opts.mode + '"');
+  }
+  // REPLACE IS STAGED: the rows go to a staging table while the import runs,
+  // and one transaction empties the target and moves them in at the end — so
+  // an import that fails or is stopped never leaves the table empty.
+  opts.replaceAll = opts.mode === 'replace';
+  if (opts.replaceAll) {
+    if ((opts.primaryKeys || []).length) throw new Error('upload: Replace empties the table first, so it takes no primaryKeys (that is an upsert)');
+    if (opts.replaceFrom) throw new Error('upload: Replace replaces the whole table; replaceFrom replaces a range — choose one');
+    opts.staged = true;
+  }
   if (opts.replaceFrom) {
     if (!opts.replaceFrom.column) throw new Error('upload: replaceFrom needs a column');
     if ((opts.primaryKeys || []).length) throw new Error('upload: replaceFrom replaces a range, so it takes no primaryKeys');
@@ -466,9 +497,71 @@ function normalizeOptions(opts) {
   if (opts.staged && typeof opts.driver.mergeStaged !== 'function') {
     throw new Error('upload: staged loads are not supported for this database — its driver has no mergeStaged');
   }
+  if (opts.staged && opts.driver.stagedReplaceOnly && !opts.replaceAll) {
+    throw new Error('upload: on this database a load is staged only to replace a whole table (mode: replace) — not for upserts or replaceFrom');
+  }
   return opts;
 }
 
+// What an import may do to its target. 'new': make it — refused if it exists.
+// 'append': add to it — it must exist. 'replace': empty it and load — it must
+// exist; staged, so a failed import leaves it as it was.
+var MODES = ['new', 'append', 'replace'];
+
+async function checkMode(opts, driver, pool) {
+  var schema = await driver.getTableSchema(pool, opts.targetTable);
+  var exists = !!(schema && schema.length);
+  if (opts.mode === 'new' && exists) {
+    throw refusal('Table "' + opts.targetTable + '" already exists. Choose Append to add to it or Replace to replace its rows, or give a new name.');
+  }
+  if (opts.mode !== 'new' && !exists) {
+    throw refusal('Table "' + opts.targetTable + '" does not exist, so there is nothing to ' + (opts.mode === 'append' ? 'append to' : 'replace') +
+      '. Check the name, or choose New table.');
+  }
+}
+
+function refusal(message) {
+  var err = new Error(message);
+  err.code = 'TARGET_REFUSED';
+  return err;
+}
+
+/**
+ * The file's columns against an existing table — what an import would do,
+ * said before it does it. Matched by exact name; the framework's own columns
+ * are left out.
+ *
+ * @param {object} opts - { driver, connection (config or pool), targetTable, columns: string[] }
+ * @returns {Promise<{ exists, matched: string[], extra: string[], missing: [{ name, required }] }>}
+ *   extra: in the file, not the table — the import would be refused.
+ *   missing: in the table, not the file — left empty; `required` when the
+ *   column takes no empty value and has no default, so every row would be refused.
+ */
+async function compareToTable(opts) {
+  var driver = opts.driver;
+  var poolIsExternal = looksLikePool(opts.connection);
+  var pool = poolIsExternal ? opts.connection : await driver.connect(opts.connection);
+  try {
+    var schema = (await driver.getTableSchema(pool, opts.targetTable)) || [];
+    var table = schema.filter(function(c) { return c.name.indexOf('__xeplr_') !== 0; });
+    var fileCols = (opts.columns || []).filter(function(n) { return n != null && n !== ''; }).map(String);
+    var inTable = {};
+    table.forEach(function(c) { inTable[c.name] = c; });
+    var inFile = {};
+    fileCols.forEach(function(n) { inFile[n] = true; });
+    return {
+      exists: schema.length > 0,
+      matched: fileCols.filter(function(n) { return inTable[n]; }),
+      extra: schema.length ? fileCols.filter(function(n) { return !inTable[n]; }) : [],
+      missing: table.filter(function(c) { return !inFile[c.name]; }).map(function(c) {
+        return { name: c.name, required: c.nullable === false && !c.hasDefault };
+      })
+    };
+  } finally {
+    if (!poolIsExternal) await driver.close(pool);
+  }
+}
+
 function safeBatchSize(columnCount, maxParams) {
   var paramsPerRow = columnCount + 1;   // +1 for __xeplr_movement_id__
   return Math.max(1, Math.floor(maxParams / paramsPerRow));
@@ -601,5 +694,7 @@ module.exports = {
   wasCommitted:    wasCommitted,
   inferColumns:    inferColumns,
   inferColumnType: inferColumnType,
+  compareToTable:  compareToTable,
+  MODES:           MODES,
   makeMetaStore:   require('./meta-store-knex')
 };

package/lib/uploader/meta-store-knex.js

@@ -88,6 +88,9 @@ module.exports = function makeKnexMetaStore(knex, options) {
           status:         'running',
           primary_keys:   plan.primaryKeys ? JSON.stringify(plan.primaryKeys) : null,
           columns:        plan.columns     ? JSON.stringify(plan.columns)     : null,
+          // What the import did to its target ('new' | 'append' | 'replace'):
+          // a Replace cannot be rolled back, and rollback reads it here.
+          meta:           plan.mode ? JSON.stringify({ mode: plan.mode }) : null,
           error:          null,
           started_at:         knex.fn.now(),
           ended_at:           null,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "1.1.22",
+  "version": "1.1.24",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "files": [