@xeplr/actions 1.1.16 → 1.1.18

package/README.md

@@ -24,7 +24,7 @@ Everything else is an **optional peer**, required only when an action actually u
 | `unzipper`, `saxes` | `format: 'excel'` (.xlsx) |
 | `imapflow` | every inbound email action |
 | `mailparser` | `email-read` (unless `body: 'none'`), `email-download-attachments` |
-| `nodemailer` | `email-send` with its own connection |
+| `nodemailer` (^10.0.13 — up to 10.0.5 has high-severity advisories) | `email-send` with its own connection |
 | `@xeplr/email` | `email-send` with `templateName` |
 | `knex` | the knex meta store |
 | `mongodb`, `ssh2-sftp-client`, `exceljs` | declared, but their drivers/sources are not implemented |
@@ -106,6 +106,7 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 
 | key | action name | does |
 |---|---|---|
+| `dataOperation` | `data-operation` | From a list of rows, make the list the next step needs — keep some rows, pick and rename columns, compute new ones, sort, optionally render. See [Data operation](#data-operation). |
 | `dbFetch` | `db-fetch` | Stream rows from a table (`mode: 'table'`) or SQL (`mode: 'query'`, `params`). Default `streaming_mode: true` → NDJSON file `{ filePath, format: 'jsonl', bytes, rows }`; `false` → `{ rows, rowCount }`. |
 | `dbPush` | `db-push` | Write `rows` or an NDJSON `filePath` into `targetTable` through the uploader. `primaryKeys` → upsert. |
 | `dbMove` | `db-move` | One window of rows from a source table, query or procedure into a target table in one streamed pass. See [db-move](#db-move). |
@@ -113,8 +114,8 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 | `dbListTables` | `db-list-tables` | `{ tables }` — base tables in a schema/database. |
 | `dbListViews` | `db-list-views` | `{ views }`. |
 | `dbListProcedures` | `db-list-procedures` | `{ procedures }` (functions excluded). |
-| `dbListColumns` | `db-list-columns` | `{ columns }` — name and data type for one table. |
-| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`). |
+| `dbListColumns` | `db-list-columns` | `{ columns }` — `name`, `dataType`, `udtName` and `maxLength` (characters; `-1` is SQL Server `(max)`, `null` when it has none) for one table. |
+| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
 | `sendEmail` | `email-send` | Send one message. See [Email](#email). |
 | `emailRead` | `email-read` | List messages, newest first, with `unread_only`, `limit` (1–1000, default 50), `since`, `from_equals`, `from_contains`, `subject_contains`, `body: 'full' \| 'text' \| 'none'`, optional NDJSON spooling. |
 | `emailMove` | `email-move` | Move a message (`messageId`, `folder`) to `toFolder`. |
@@ -130,7 +131,7 @@ The db actions `db-fetch`, `db-push` and `db-procedure` accept a connection two
 - `connectionInfoId` (+ `dbInfoId`) — ids of a saved connection. The action does not resolve these; a host does, before `runAction` (e.g. `@xeplr/jobs` `resolveConnection`, which turns `connectionInfoId` into `connection` and `dbType`).
 - `connection: { host, port, user, password, database }` and `dbType` — literal, for scripts, tests and callers with no host.
 
-`db-move` takes literal `sourceConnection` / `targetConnection`; `file-upload` takes `dbConnection`. A DuckDB connection is `{ file, access?: 'ro' | 'rw', openTimeoutMs?: 60000, maxConnections?: 4 }`.
+`db-move` takes literal `sourceConnection` / `targetConnection`; `file-upload` takes `dbConnection`. A DuckDB connection is `{ file, access?: 'ro' | 'rw', openTimeoutMs?: 60000, maxConnections?: 4, keepIndexes?: false }` (see [A written DuckDB file keeps no index](#a-written-duckdb-file-keeps-no-index)).
 
 ### db-move
 
@@ -141,7 +142,8 @@ The db actions `db-fetch`, `db-push` and `db-procedure` accept a connection two
 | `writeMode` | `append` (default), `replace` (TRUNCATE the target first if it exists, DELETE if TRUNCATE fails), `upsert` (**requires** `primaryKeys`). |
 | `window` | `{ column, from, to }` → `column >= from AND column < to` (a null edge is left out). `{ columns: [{ column, from }] }` → OR of `column >= from` (null `from`s left out). For a procedure: `{ from, to, fromParam, toParam }`, passed as parameters. A window with none of `column`, `columns`, `fromParam`, `toParam` is treated as no window. |
 | `where` | Raw SQL condition, ANDed with the window in parentheses, run on the source. Not parameterised. Ignored for `mode: 'procedure'`. |
-| `staged` | `true`: write into `<targetTable>__xeplr_staging` and move it into the target in one transaction at the end (see [Staged loads](#staged-loads)). DuckDB targets only. Default `false`. |
+| `replaceFrom` | `{ column }` with `staged`: the final move deletes target rows from the earliest staged value of `column` on, before inserting the staged ones — see [Staged loads](#staged-loads). |
+| `staged` | `true`: write into `<targetTable>__xeplr_staging` and move it into the target in one transaction at the end (see [Staged loads](#staged-loads)). DuckDB targets only. Default `false`, except that **every DuckDB upsert is staged** whatever this says. |
 | `movementId` | Correlation key. Default `system.occurrenceId`, else `mv_<id>`. |
 | `batchSize`, `concurrency` | Default 5000 and 4. |
 
@@ -170,6 +172,54 @@ For `mode: 'table'` the source's declared column types (`getTableSchema`) are pa
 
 Message ids are IMAP UIDs **scoped to `folder`**, and a move changes the id — download and delete before moving.
 
+## Data operation
+
+The step between steps. A fetch returns everything a table holds; an email
+wants four columns, the active rows, sorted, and sometimes rendered. Without
+this, each of those is a module somebody writes and maintains — and the third
+is written differently from the first two.
+
+```js
+await runAction({ name: 'data-operation', input: {
+  from: rows,                                   // a list; one object is one row
+  where: 'status = "active" and [due] <= today()',
+  columns: ['name', { name: 'late', label: 'Days late', formula: 'datediff([due], today())' },
+            { name: 'id', hidden: true }],
+  sort: '-late',
+  format: true, title: 'Overdue'
+} })
+// → { rows, count, html, text }
+```
+
+**Rows and a rendering, both.** `rows` and `count` are always there; `html`
+and `text` only when `format` is on, because building a table for ten
+thousand rows nobody renders is work done and stored on every run. Neither is
+privileged — the next step binds whichever it wants.
+
+**`count` is the point of the step as often as `rows` is**: it is what lets
+the step after it branch on *nothing to report* instead of sending an empty
+email.
+
+**Formulas are `@xeplr/expression-handler`** — the engine running BI's report
+formulas and workflow's conditions, with the same functions. `where` and a
+computed column are the same language.
+
+**What `today()` means comes from the run's context**, not from the action:
+`context.formula` carries `today` and `fiscalYearStart`, so a workspace that
+reckons days elsewhere and a run replayed from history agree on what
+`yesterday()` filtered. The action never calls `new Date()`.
+
+**Only visible columns are rendered.** A column kept for the next step but
+marked `hidden` stays in `rows` and never reaches the table — an id the flow
+needs is not a column a person wants printed.
+
+**A ragged row does not fail the step.** A column missing from one row reads
+as nothing (`missing: 'null'`), because one bad row out of nine hundred
+should not take the other eight hundred and ninety-nine with it.
+
+**No aggregation.** `group by status, count` changes what a row *is*; that is
+a different shape and, when it is wanted, a different step.
+
 ## Uploader
 
 `uploader.upload(opts)` is what `db-push`, `db-move` and `file-upload` use:
@@ -196,7 +246,7 @@ await actions.uploader.upload({
 
 `signal` (an `AbortSignal`, passed to db-move as `system.signal`) stops a running movement: the writes still queued are dropped at once (in-flight ones settle), the read stops at the next batch, and it ends with an error whose `code` is `'STOPPED'` (the runner keeps `error.code`), neither a failure nor a success. What it wrote stays, under its `movementId`, for the caller to roll back or keep.
 
-Rows are spooled to NDJSON files on disk between source and database. From the first batch it creates, if missing: the target table, `<targetTable>_import_errors` (`movement_id`, `row_num`, `error_description`, `underlying_sql`, `raw_row`, `recorded_at`), a `__xeplr_movement_id__` column, and a unique index on `primaryKeys`. Server dialects also add `__xeplr_id__`; DuckDB does not.
+Rows are spooled to NDJSON files on disk between source and database. From the first batch it creates, if missing: the target table, `<targetTable>_import_errors` (`movement_id`, `row_num`, `error_description`, `underlying_sql`, `raw_row`, `recorded_at`), a `__xeplr_movement_id__` column, and a unique index on `primaryKeys` (not on DuckDB: its upserts are staged and keep none). Server dialects also add `__xeplr_id__`; DuckDB does not.
 
 Logical types and what each driver creates:
 
@@ -219,9 +269,22 @@ With `staged: true` a load **never writes into its target while it runs**. Batch
 - **Crash during the move**: the transaction is discarded on reopen and the target is as it was.
 - **Did it land?** `uploader.wasCommitted({ driver, connection, movementId })` reads the ledger row, written in the same transaction as the move: a row means it landed, `null` means nothing did. No scan of the target.
 - **Upsert** replaces target rows whose key is staged (`DELETE … USING` staging, then `INSERT`). Within one load the last row staged for a key wins, as under `ON CONFLICT DO UPDATE`; rows with a NULL key match nothing and are kept. **No unique index**: a staged load drops `<targetTable>_upsert_uniq` and never creates it. DuckDB's DELETE through an index that a killed load left out of step with its rows fails with "Failed to delete all rows from index", and that failure invalidates the whole database.
+- **A range** (`replaceFrom: { column }`, staged append only, no `primaryKeys`) replaces the target's rows from **the earliest staged value** of `column` on by the staged ones, in the same one transaction: *re-load the last N days*, for a table with no key or a date with no time, where reading "past the latest" would duplicate today on every run. The caller reads the source from its cutoff; the boundary is taken from what that read brought, never from a date passed in, because a cutoff computed elsewhere sits a time zone away (a `DATE` read as UTC midnight here, compared at local midnight at the source) and a day deleted but not read back is lost. Nothing staged deletes nothing. `merged.replaced` is the rows deleted.
 - **Cost** is one statement at the end. Measured with 50 lakh staged rows into 50 crore: append 0.3 s (independent of the target's size), upsert 0.4–7 s depending on how widely the replaced keys are spread.
 
-A driver opts in by implementing `prepareStaging`, `mergeStaged`, `dropStaging` and `committedMovement`; `upload` refuses `staged` for one that does not.
+A driver opts in by implementing `prepareStaging`, `mergeStaged`, `dropStaging` and `committedMovement`; `upload` refuses `staged` for one that does not. A driver that sets `stagesUpserts: true` (DuckDB) has every upsert staged, asked or not.
+
+### A written DuckDB file keeps no index
+
+DuckDB 1.5.4–1.5.6 has a bug ([duckdb/duckdb#26106](https://github.com/duckdb/duckdb/issues/26106), open upstream): when a process dies with rows still only in the WAL, the next open replays them into the table and the next checkpoint drops them from every index on it. The first `DELETE` that reaches one fails with *"Failed to delete all rows from index"*, which is **FATAL**: the whole file is unusable until the process restarts. Any index, unique or not. A table with no index is unaffected.
+
+So the DuckDB driver, on every **read-write** open of a file:
+
+1. runs `CHECKPOINT` first, which folds a replayed WAL in safely before anything touches it (the upstream workaround);
+2. **drops every index** in the file, logging which, so one left from before this rule cannot be what a later crash breaks;
+3. **refuses `CREATE INDEX`** on that pool from then on (`ensureUpsertIndex` included).
+
+Upserts need no index: they are staged and match keys in the move. A read-only open changes nothing. `keepIndexes: true` turns 2 and 3 off, only for a file that is never written after its index exists; `:memory:` is exempt, having no WAL to replay. `test/duckdb-index-guard.test.js` kills a real writer process to bring the failure about, proves it still happens on the pinned DuckDB, and proves each step above prevents it.
 
 `uploader.rollback({ movementId, driver, connection, targetTable, metaStore?, queue? })` aborts the movement's queued work, deletes its rows from the target and error tables by `__xeplr_movement_id__` / `movement_id`, and records `rolled-back`.
 
@@ -331,7 +394,7 @@ node --test test/uploader/reconcile.test.js
 | needs nothing | needs a database |
 |---|---|
 | `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
-| `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
+| `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`; each skipped when unset), `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/duckdb-index-guard.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
 | | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
 | | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |
 

package/lib/builtins/data-operation.js

@@ -0,0 +1,241 @@
+// data-operation action — from what a step got, make what the next one needs.
+//
+// The step between steps. A fetch returns everything a table holds; an email
+// wants four columns, only the active rows, sorted, and sometimes rendered.
+// Without this every one of those is a module somebody writes and maintains,
+// and the third one is written slightly differently from the first two.
+//
+// ROWS AND A RENDERING, BOTH. `rows` is always there and `html` only when
+// asked for, and neither is privileged: the next step binds
+// {steps.x.output.rows} or {steps.x.output.html} and the consumer decides,
+// the same rule that settled the list/item-wise question.
+//
+// NO AGGREGATION. `group by status, count` changes what a row IS, which is a
+// different shape and a different step. Filtering, picking, computing and
+// sorting all leave a list of rows a list of rows.
+//
+// FORMULAS ARE @xeplr/expression-handler — the engine already running BI's
+// report formulas and workflow's conditions. `where` and a computed column
+// are the same language, with the same functions, evaluated with the run's
+// own context: what `today()` means, and therefore what `yesterday()` means,
+// is that engine's business and not this action's.
+
+var xf = require('@xeplr/expression-handler');
+
+/** A cell as it goes into HTML: never as markup. */
+function escapeHtml(value) {
+  if (value === null || value === undefined) return '';
+  return String(value)
+    .replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;')
+    .replace(/"/g, '&quot;').replace(/'/g, '&#39;');
+}
+
+/** What a column is called in the output, and what its header reads. */
+function columnSpec(col) {
+  if (typeof col === 'string') return { name: col.trim(), label: col.trim(), formula: null };
+  if (!col || !col.name) return null;
+  return {
+    name: String(col.name).trim(),
+    label: String(col.label || col.name).trim(),
+    formula: col.formula ? String(col.formula) : null,
+    // Hidden columns are computed and kept in `rows` — a later step may want
+    // an id it does not want printed — but are not in the rendering. See the
+    // note on `format` below.
+    hidden: Boolean(col.hidden)
+  };
+}
+
+/**
+ * A formula, ready to run against a row.
+ *
+ * The RUN'S CONTEXT is compiled in — what today is, and when the financial
+ * year starts. A workspace whose day starts elsewhere, and a run replayed
+ * from history, have to agree on what `yesterday()` meant, and that answer
+ * belongs to the engine and the context rather than to this action.
+ *
+ * `missing: 'null'` — a column that is not on a row reads as nothing, not as
+ * an error. Rows out of a real system are ragged, and one row missing a field
+ * should not fail the step that was filtering the other nine hundred.
+ */
+function compile(formula, where, ctx) {
+  try {
+    return xf.compile(formula, Object.assign({ missing: 'null' }, ctx));
+  } catch (err) {
+    var e = new Error('Cannot read ' + where + ': ' + err.message);
+    e.status = 400;
+    throw e;
+  }
+}
+
+module.exports = {
+  name: 'data-operation',
+  description: 'From a list of rows, make the list the next step needs: keep some rows, pick and ' +
+               'rename columns, compute new ones, sort — and optionally render an HTML table.',
+  requires: [],
+
+  inputSchema: [
+    { name: 'from', type: 'array', required: true, order: 1,
+      description: 'The rows to work on — usually an earlier step\'s output. One object is treated as one row.' },
+
+    { name: 'where', type: 'string', order: 2,
+      description: 'Keep only the rows this is true for, e.g. status = "active" and createdDate >= yesterday(). ' +
+                   'Blank keeps every row.' },
+
+    // A list rather than a comma-separated string: a computed column needs a
+    // formula and a label beside its name, and three parallel strings that
+    // have to line up is a shape nobody can edit by hand.
+    { name: 'columns', type: 'array', order: 3,
+      description: 'Which columns to keep, in order: "name", or { name, label, formula, hidden }. ' +
+                   'Blank keeps every column of the first row, as it is.' },
+
+    { name: 'sort', type: 'string', order: 4,
+      description: 'A column to sort by. Put - in front for descending, e.g. -dueDate.' },
+
+    // OFF BY DEFAULT. Building a table for ten thousand rows nobody renders
+    // is work done and stored on every run — so it is asked for rather than
+    // assumed, and the rows are there either way.
+    { name: 'format', type: 'boolean', default: false, order: 5,
+      description: 'Also give back an HTML table (and a plain-text one) of the visible columns.' },
+    { name: 'title', type: 'string', order: 6, group: 'Formatting',
+      description: 'A heading above the table.' },
+    { name: 'emptyText', type: 'string', order: 7, group: 'Formatting',
+      description: 'What the table says when no rows are left. Default: "Nothing to show".' }
+  ],
+
+  outputSchema: [
+    { name: 'rows', type: 'array', description: 'What is left, with the columns asked for.' },
+    { name: 'count', type: 'number', description: 'How many rows — so the next step can branch on none.' },
+    { name: 'html', type: 'string', description: 'A table of the visible columns. Only when `format` is on.' },
+    { name: 'text', type: 'string', description: 'The same, as plain text, for a mail client that refuses HTML.' }
+  ],
+
+  execute: async function(input, context) {
+    var given = input.from;
+    // A list, always — one object is a list of one, and nothing is an empty
+    // list rather than an error. The step after this one should not have to
+    // ask which it got.
+    var source = given === null || given === undefined ? []
+      : Array.isArray(given) ? given
+      : [given];
+
+    // The run's own context decides what today is — and therefore what
+    // yesterday is. See the note at the top.
+    var ctx = (context && context.formula) || {};
+
+    var keep = null;
+    if (input.where && String(input.where).trim()) {
+      var test = compile(String(input.where), 'the condition', ctx);
+      // Anything the formula calls true. `status = "active"` gives a boolean,
+      // but `daysLate` gives a number, and a step that filtered on it would
+      // otherwise keep every row including the zeroes.
+      keep = function(row) {
+        var v = test(row);
+        return v !== null && v !== undefined && v !== false && v !== 0 && v !== '';
+      };
+    }
+
+    var specs = (Array.isArray(input.columns) ? input.columns : [])
+      .map(columnSpec)
+      .filter(Boolean);
+
+    // No columns asked for: every column of the data, as it is. A step that
+    // only filters should not have to list what it is not changing.
+    if (!specs.length) {
+      var first = source.find(function(r) { return r && typeof r === 'object'; });
+      specs = Object.keys(first || {}).map(function(k) { return { name: k, label: k, formula: null }; });
+    }
+
+    var computed = specs.map(function(spec) {
+      return spec.formula ? Object.assign({}, spec, { fn: compile(spec.formula, '"' + spec.label + '"', ctx) }) : spec;
+    });
+
+    var rows = [];
+    for (var i = 0; i < source.length; i++) {
+      var row = source[i] && typeof source[i] === 'object' ? source[i] : {};
+      if (keep && !keep(row)) continue;
+
+      var out = {};
+      for (var c = 0; c < computed.length; c++) {
+        var spec = computed[c];
+        out[spec.name] = spec.fn ? spec.fn(row) : row[spec.name];
+      }
+      rows.push(out);
+    }
+
+    if (input.sort && String(input.sort).trim()) {
+      var desc = String(input.sort).trim().charAt(0) === '-';
+      var by = desc ? String(input.sort).trim().slice(1) : String(input.sort).trim();
+      rows.sort(function(a, b) {
+        var x = a[by], y = b[by];
+        if (x === y) return 0;
+        // Nothing sorts last either way: an empty cell is not the smallest
+        // value, it is the absence of one, and burying it under the top of a
+        // descending list is how it gets missed.
+        if (x === null || x === undefined || x === '') return 1;
+        if (y === null || y === undefined || y === '') return -1;
+        var out = x < y ? -1 : 1;
+        return desc ? -out : out;
+      });
+    }
+
+    var result = { rows: rows, count: rows.length };
+
+    if (input.format) {
+      // THE VISIBLE COLUMNS ONLY. A column kept for the next step but not
+      // meant to be printed is not printed — and its label, which exists for
+      // a header, goes with it.
+      var shown = computed.filter(function(s) { return !s.hidden; });
+      var empty = input.emptyText || 'Nothing to show';
+      result.html = toHtml(rows, shown, input.title, empty);
+      result.text = toText(rows, shown, input.title, empty);
+    }
+
+    return result;
+  }
+};
+
+function toHtml(rows, columns, title, empty) {
+  var head = '<tr>' + columns.map(function(c) {
+    return '<th align="left" style="padding:6px 10px;border-bottom:1px solid #ddd">' + escapeHtml(c.label) + '</th>';
+  }).join('') + '</tr>';
+
+  var body = rows.map(function(r) {
+    return '<tr>' + columns.map(function(c) {
+      return '<td style="padding:6px 10px;border-bottom:1px solid #f0f0f0">' + escapeHtml(r[c.name]) + '</td>';
+    }).join('') + '</tr>';
+  }).join('');
+
+  // Inline styles and no stylesheet: an email client strips <style> and most
+  // of what it does not strip, it ignores.
+  var table = rows.length
+    ? '<table cellspacing="0" cellpadding="0" style="border-collapse:collapse;font-family:sans-serif;font-size:14px">' +
+      '<thead>' + head + '</thead><tbody>' + body + '</tbody></table>'
+    : '<p style="font-family:sans-serif;font-size:14px">' + escapeHtml(empty) + '</p>';
+
+  return title ? '<h3 style="font-family:sans-serif;font-size:15px;margin:0 0 8px">' + escapeHtml(title) + '</h3>' + table : table;
+}
+
+function toText(rows, columns, title, empty) {
+  if (!rows.length) return (title ? title + '\n\n' : '') + empty;
+
+  var cells = [columns.map(function(c) { return String(c.label); })].concat(
+    rows.map(function(r) {
+      return columns.map(function(c) { return r[c.name] === null || r[c.name] === undefined ? '' : String(r[c.name]); });
+    })
+  );
+
+  // Padded to the widest cell in each column, so it still reads as a table in
+  // a client that shows plain text in a proportional font badly.
+  var widths = columns.map(function(_, i) {
+    return cells.reduce(function(max, line) { return Math.max(max, line[i].length); }, 0);
+  });
+  var line = function(parts) {
+    return parts.map(function(p, i) { return p.padEnd(widths[i]); }).join('  ').trimEnd();
+  };
+
+  var out = [line(cells[0]), widths.map(function(w) { return '-'.repeat(w); }).join('  ')]
+    .concat(cells.slice(1).map(line))
+    .join('\n');
+
+  return title ? title + '\n\n' + out : out;
+}

package/lib/builtins/db/move.js

@@ -99,6 +99,12 @@ module.exports = {
                    'the end, so a movement that stops or fails midway never touches the target. ' +
                    'DuckDB targets only.' },
 
+    { name: 'replaceFrom',     type: 'object', order: 12.6, group: 'Write options',
+      description: 'With staged and append: { column }. The final move first deletes the target ' +
+                   'rows where column >= the earliest staged value, then inserts the staged rows, in the ' +
+                   'same one transaction — "re-load the last N days" for a table with no key, or a date ' +
+                   'with no time. Read the source from the cutoff; nothing staged deletes nothing.' },
+
     { name: 'window',          type: 'object', order: 13, group: 'Advanced',
       description: 'One window: { column, from, to }. Half-open — from <= x < to. ' +
                    'Omitted = no filter. For mode="procedure", also needs { fromParam, toParam }.' },
@@ -315,7 +321,8 @@ module.exports = {
         onProgress:     system.onProgress || null,
         // An AbortSignal from the caller: stop reading, drop the queued writes.
         signal:         system.signal || null,
-        staged:         Boolean(input.staged)
+        staged:         Boolean(input.staged),
+        replaceFrom:    input.replaceFrom || null
       });
 
       // Every count upload() has, not just the headline one — "read" and

package/lib/builtins/file/upload.js

@@ -113,7 +113,12 @@ module.exports = {
         // opaque tenant/app context — see runner.js's ctx.system passthrough.
         metaStore:      ctx.metaStore,
         mtId1: system.mtId1, mtId2: system.mtId2, mtId3: system.mtId3, mtId4: system.mtId4,
-        details:        system.details
+        details:        system.details,
+        // Progress and the stop switch, as db-move passes them: the uploader
+        // supports both, and without them a long import could neither say how
+        // far it had got nor be stopped.
+        onProgress:     system.onProgress || null,
+        signal:         system.signal || null
       });
     } finally {
       queue.stop();

package/lib/builtins/index.js

@@ -18,6 +18,8 @@
 module.exports = {
   // Generic actions (no runtime-config dependency)
   httpRequest:  require('./http-request'),
+  // The step between steps: from what one step got, make what the next needs.
+  dataOperation: require('./data-operation'),
   spawnProgram: require('./spawn-program'),
   sendEmail:    require('./send-email'),
 

package/lib/drivers/db/duckdb.js

@@ -196,6 +196,9 @@ async function openWithWait(api, key, options, access, timeoutMs) {
  *   config.openTimeoutMs            how long to wait out another holder
  *                                   (default 60s — a load can be slow)
  *   config.maxConnections           default 4
+ *   config.keepIndexes              true to allow indexes in this file. Off by
+ *                                   default — see "a written file keeps no
+ *                                   index" below.
  *
  * READ-ONLY UNLESS ASKED. A caller that forgets `access: 'rw'` gets an error
  * on its first write, in the code doing the writing — not a silent success on
@@ -248,10 +251,78 @@ async function connect(config) {
     max: Math.max(1, parseInt(config.maxConnections || 4, 10))
   };
   entry.pool = { __duck: entry, access: access, file: key };
+  // ':memory:' never outlives its process, so it has no WAL to replay and
+  // nothing for the guard to protect.
+  entry.keepIndexes = Boolean(config.keepIndexes) || key === ':memory:';
+  if (access === 'rw' && key !== ':memory:') {
+    try {
+      await guardOnOpen(entry);
+    } catch (err) {
+      try { instance.closeSync(); } catch (_) {}
+      throw err;
+    }
+  }
   INSTANCE_CACHE[key] = entry;
   return entry.pool;
 }
 
+// ── a written file keeps no index ────────────────────────────────────────
+//
+// DuckDB 1.5.4–1.5.6 (duckdb/duckdb#26106, open upstream): after a process
+// dies with rows still only in the WAL, the next open replays them into the
+// table, and the next checkpoint drops them from every ART index on it. The
+// rows are still there; the index no longer knows them. The first DELETE that
+// reaches one fails with "Failed to delete all rows from index" — FATAL, and
+// the whole file is unusable until the process restarts. Unique or not, any
+// index. With no index there is nothing to fall out of step.
+//
+// So on every read-write open of a file:
+//   1. CHECKPOINT first — folds a replayed WAL into the file before anything
+//      else touches it, the upstream workaround (proven here: kill, reopen,
+//      CHECKPOINT, DELETE succeeds even with an index).
+//   2. DROP every index in it — one left from before this rule, or one that
+//      slipped in, cannot be the thing a later crash breaks.
+// and CREATE INDEX is refused on the pool from then on (see query). Upserts do
+// not need one: they are staged and matched in the move (mergeStaged).
+//
+// A reader cannot trip this: a read-only open replays nothing and deletes
+// nothing, and a file a crash left damaged gets its index dropped the first
+// time anything opens it to write.
+async function guardOnOpen(entry) {
+  var conn = await entry.instance.connect();
+  try {
+    await conn.run('CHECKPOINT');
+    if (entry.keepIndexes) return;
+    var reader = await conn.runAndReadAll(
+      "SELECT schema_name, index_name, table_name FROM duckdb_indexes()"
+    );
+    var found = reader.getRowObjectsJS();
+    for (var i = 0; i < found.length; i++) {
+      await conn.run('DROP INDEX IF EXISTS ' + quoteIdent(found[i].schema_name) + '.' + quoteIdent(found[i].index_name));
+    }
+    if (found.length) {
+      await conn.run('CHECKPOINT');
+      console.warn('[duckdb] ' + entry.key + ': dropped ' + found.length + ' index(es) (' +
+        found.map(function(f) { return f.table_name + '.' + f.index_name; }).join(', ') +
+        ') — a written file keeps none; see duckdb/duckdb#26106');
+    }
+  } finally {
+    try { conn.closeSync(); } catch (_) {}
+  }
+}
+
+var CREATE_INDEX = /^\s*CREATE\s+(UNIQUE\s+)?INDEX\b/i;
+
+function refuseIndex(entry, sql) {
+  if (entry.keepIndexes || !CREATE_INDEX.test(String(sql || ''))) return;
+  throw new Error(
+    'duckdb: CREATE INDEX refused on "' + entry.key + '". A file that is written keeps no index: ' +
+    'DuckDB loses rows from an index after a crash (duckdb/duckdb#26106), and the next DELETE ' +
+    'then fails fatally. Upserts are staged and need none. Open with keepIndexes: true only for ' +
+    'a file that is never written after the index exists.'
+  );
+}
+
 /** Check out a connection, creating one lazily up to `max`, else queueing. */
 async function acquire(entry) {
   if (entry.free.length) return entry.free.pop();
@@ -341,6 +412,7 @@ function plainDeep(v) {
  */
 async function query(pool, sql, params) {
   var entry = entryOf(pool);
+  refuseIndex(entry, sql);
   var conn = await acquire(entry);
   try {
     var api = require('@duckdb/node-api');
@@ -528,7 +600,7 @@ async function ensureMovementColumn(pool, tableName) {
 
 async function ensureUpsertIndex(pool, tableName, primaryKeys) {
   var sql = buildUpsertIndexSql(tableName, primaryKeys);
-  if (sql) await query(pool, sql);
+  if (sql) await query(pool, sql);   // refused on a guarded file — the uploader stages DuckDB upserts instead
 }
 
 // ── introspection ────────────────────────────────────────────────────────
@@ -800,6 +872,11 @@ async function dropStaging(pool, tableName) {
  * under ON CONFLICT DO UPDATE; a row with a NULL key matches nothing, and is
  * kept, as ON CONFLICT kept it.
  *
+ * With replaceFrom { column } it replaces a RANGE: target rows with column >=
+ * the earliest staged value are deleted first ("re-load the last N days" —
+ * the caller reads the source from its cutoff; the boundary here is what that
+ * read actually brought). `replaced` is then the rows deleted.
+ *
  * Returns { inserted, replaced }.
  */
 async function mergeStaged(pool, opts) {
@@ -824,6 +901,21 @@ async function mergeStaged(pool, opts) {
     await conn.run('BEGIN TRANSACTION');
     try {
       var replaced = 0;
+      // A RANGE REPLACED: the target's rows from the EARLIEST STAGED value
+      // on go, and the staged rows take their place — inside this one
+      // transaction, so a crash leaves the old rows, never neither.
+      //
+      // From what was staged, not from a date passed in. The staged values
+      // came through the same reads and conversions as the target's own, so
+      // this boundary is exact; a cutoff computed elsewhere is a time zone
+      // away from it (a DATE read as UTC midnight here, compared at the
+      // source's local midnight there) — and a day deleted but never read
+      // back is a day lost. Nothing staged: MIN is NULL, nothing is deleted.
+      if (opts.replaceFrom) {
+        var col = quoteIdent(opts.replaceFrom.column);
+        var cut = await conn.run('DELETE FROM ' + target + ' WHERE ' + col + ' >= (SELECT MIN(' + col + ') FROM ' + staging + ')');
+        replaced = Number(cut.rowsChanged) || 0;
+      }
       if (pks) {
         var del = await conn.run('DELETE FROM ' + target + ' AS t USING ' + staging + ' AS s WHERE ' +
           pks.map(function(k) { return 't.' + quoteIdent(k) + ' = s.' + quoteIdent(k); }).join(' AND '));
@@ -881,6 +973,9 @@ module.exports = {
 
   ensureMovementColumn: ensureMovementColumn,
   ensureUpsertIndex:    ensureUpsertIndex,
+  // An upsert into DuckDB is always staged: it matches keys in the move, so it
+  // needs no index, which a written DuckDB file must not keep.
+  stagesUpserts:        true,
 
   getTableSchema:  getTableSchema,
   listTables:      listTables,

package/lib/drivers/db/mssql.js

@@ -314,14 +314,15 @@ function toMssqlDatetime(v) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
     'FROM INFORMATION_SCHEMA.COLUMNS ' +
     'WHERE TABLE_NAME = @p0 AND (@p1 IS NULL OR TABLE_SCHEMA = @p1) ' +
     'ORDER BY ORDINAL_POSITION',
     [tableName, schema || null]
   );
   return r.rows.map(function(row) {
-    return { name: row.name, dataType: row.dataType, udtName: row.dataType };
+    // maxLength -1 is (max): nvarchar(max), varchar(max), varbinary(max).
+    return { name: row.name, dataType: row.dataType, udtName: row.dataType, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
   });
 }
 

package/lib/drivers/db/mysql.js

@@ -311,14 +311,14 @@ function toMysqlDatetime(v) {
 
 async function getTableSchema(pool, tableName, schema) {
   var r = await query(pool,
-    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName ' +
+    'SELECT COLUMN_NAME AS name, DATA_TYPE AS dataType, COLUMN_TYPE AS udtName, CHARACTER_MAXIMUM_LENGTH AS maxLength ' +
     'FROM information_schema.COLUMNS ' +
     'WHERE TABLE_SCHEMA = COALESCE(?, DATABASE()) AND TABLE_NAME = ? ' +
     'ORDER BY ORDINAL_POSITION',
     [schema || null, tableName]
   );
   return r.rows.map(function(row) {
-    return { name: row.name, dataType: row.dataType, udtName: row.udtName };
+    return { name: row.name, dataType: row.dataType, udtName: row.udtName, maxLength: row.maxLength == null ? null : Number(row.maxLength) };
   });
 }
 

package/lib/drivers/db/postgres.js

@@ -290,14 +290,15 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
 // in ordinal-position order. Empty array if the table doesn't exist.
 async function getTableSchema(pool, tableName, schema) {
   var r = await pool.query(
-    "SELECT column_name, data_type, udt_name " +
+    "SELECT column_name, data_type, udt_name, character_maximum_length " +
     "FROM information_schema.columns " +
     "WHERE table_schema = $2 AND table_name = $1 " +
     "ORDER BY ordinal_position",
     [tableName, schema || 'public']
   );
   return r.rows.map(function(row) {
-    return { name: row.column_name, dataType: row.data_type, udtName: row.udt_name };
+    return { name: row.column_name, dataType: row.data_type, udtName: row.udt_name,
+      maxLength: row.character_maximum_length == null ? null : Number(row.character_maximum_length) };
   });
 }
 

package/lib/uploader/index.js

@@ -28,6 +28,8 @@
 //     errorTable,      // defaults to `<targetTable>_import_errors`
 //     batchDir,        // where NDJSON files spool to. Defaults to os.tmpdir()/xeplr-actions
 //     keepBatchFiles,  // default false — clean up NDJSON files after success
+//     replaceFrom,     // optional, staged only: { column } — the move deletes target
+//                       // rows from the earliest staged value on first (see mergeStaged)
 //     staged,          // default false. true: batches land in <targetTable>__xeplr_staging
 //                       // and move into the target in ONE transaction at the end, so a
 //                       // load that stops or dies midway never touched the target.
@@ -247,7 +249,8 @@ async function upload(opts) {
         targetTable: opts.targetTable,
         columns:     columns,
         primaryKeys: opts.primaryKeys,
-        movementId:  opts.movementId
+        movementId:  opts.movementId,
+        replaceFrom: opts.replaceFrom
       });
     }
     await opts.metaStore.recordEnd(opts.movementId, {
@@ -370,7 +373,16 @@ function normalizeOptions(opts) {
     (process.env.XEPLR_ACTIONS_TMP_DIR || path.join(os.tmpdir(), 'xeplr-actions'));
   opts.keepBatchFiles     = !!opts.keepBatchFiles;
   opts.staged             = !!opts.staged;
+  // A driver that matches upsert keys only in a staged move (DuckDB: its
+  // written files keep no index) stages every upsert, asked or not.
+  if (opts.driver && opts.driver.stagesUpserts && (opts.primaryKeys || []).length) opts.staged = true;
   opts.writeTable         = null;
+  opts.replaceFrom        = opts.replaceFrom || null;
+  if (opts.replaceFrom) {
+    if (!opts.replaceFrom.column) throw new Error('upload: replaceFrom needs a column');
+    if ((opts.primaryKeys || []).length) throw new Error('upload: replaceFrom replaces a range, so it takes no primaryKeys');
+    opts.staged = true;
+  }
   if (opts.staged && typeof opts.driver.mergeStaged !== 'function') {
     throw new Error('upload: staged loads are not supported for this database — its driver has no mergeStaged');
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "1.1.16",
+  "version": "1.1.18",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "files": [
@@ -28,7 +28,8 @@
     "access": "public"
   },
   "dependencies": {
-    "@xeplr/schema-handler": "^1.0.0"
+    "@xeplr/schema-handler": "^1.0.0",
+    "@xeplr/expression-handler": "^2.0.0"
   },
   "peerDependencies": {
     "@duckdb/node-api": "1.5.5-r.3",
@@ -42,7 +43,7 @@
     "mongodb": "^6.0.0",
     "mssql": "^11.0.0",
     "mysql2": "^3.0.0",
-    "nodemailer": "^8.0.2",
+    "nodemailer": "^10.0.13",
     "pg": "^8.0.0",
     "saxes": "^6.0.0",
     "ssh2-sftp-client": "^11.0.0",