@xeplr/actions 2.2.5 → 2.2.7

package/README.md

@@ -80,7 +80,8 @@ var moved = await actions.runAction(actions.builtins.dbMove, {
 | `streaming` | `{ spool, resume, readBatchFile, checkpoint }` — rows to rotating NDJSON batch files with ordered hooks and resumable checkpoints. |
 | `drivers.db` | `{ getDriver(type), checkDriverRequires, SUPPORTED }` — `postgres`, `mysql`, `mssql`, `mongo`, `duckdb`. |
 | `drivers.file` | `{ getSource(type), checkSourceRequires, SUPPORTED }` — `local`, `sharepoint`, `google`, `sftp`. |
-| `formats` | `{ getFormat(type), checkFormatRequires, SUPPORTED }` — `excel`, `csv`, `txt`, `json`. |
+| `formats` | `{ getFormat(type), checkFormatRequires, SUPPORTED }` — `excel`, `csv`, `txt`, `json`, `parquet`. |
+| `parquet` | `{ open, writeAll, readRows, columnsOf, isParquet }` — rows to a folder of Parquet parts and back, through DuckDB. See [Parquet](#parquet-the-hand-off-to-python-and-warehouses). |
 | `attachConfig({ service, applicationId? })` | Connect to the shared `xeplr_configs` database; see [Movement metadata](#movement-metadata-xeplr_configs). |
 | `configRequiredEnv` | `['XCFG_DB_NAME']` — spread into your app's required-env list. |
 
@@ -133,10 +134,10 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 | key | action name | does |
 |---|---|---|
 | `dataOperation` | `data-operation` | From a list of rows, make the list the next step needs — keep some rows, pick and rename columns, compute new ones, sort, optionally render. See [Data operation](#data-operation). |
-| `exportTable` | `export-table` | Rows to a **CSV, Excel (.xlsx) or PDF** file. See [Export](#export). |
+| `exportTable` | `export-table` | Rows to a **CSV, Excel (.xlsx), PDF or Parquet** file. See [Export](#export). |
 | `exportPages` | `export-pages` | Pages of placed blocks (text, tables, images, shapes) to a PDF — a dashboard as it looked. See [Export](#export). |
-| `dbFetch` | `db-fetch` | Stream rows from a table (`mode: 'table'`) or SQL (`mode: 'query'`, `params`). Default `streaming_mode: true` → NDJSON file `{ filePath, format: 'jsonl', bytes, rows }`; `false` → `{ rows, rowCount }`. |
-| `dbPush` | `db-push` | Write `rows` or an NDJSON `filePath` into `targetTable` through the uploader. `primaryKeys` → upsert. |
+| `dbFetch` | `db-fetch` | Stream rows from a table (`mode: 'table'`) or SQL (`mode: 'query'`, `params`). Default `streaming_mode: true` → NDJSON file `{ filePath, format: 'jsonl', bytes, rows }`; with `format: 'parquet'` a folder of typed Parquet parts `{ filePath, format: 'parquet', parts, bytes, rows }` (see [Parquet](#parquet-the-hand-off-to-python-and-warehouses)); `false` → `{ rows, rowCount }`. |
+| `dbPush` | `db-push` | Write `rows`, or a `filePath` — NDJSON, a `.parquet` file or a folder of Parquet parts — into `targetTable` through the uploader. A Parquet source declares its column types, so the target gets them instead of types guessed from values. `primaryKeys` → upsert. |
 | `dbMove` | `db-move` | One window of rows from a source table, query or procedure into a target table in one streamed pass. See [db-move](#db-move). |
 | `dbDump` | `db-dump` | A whole database to one backup file with its own dump program: `pg_dump -Fc` (Postgres, `.dump`) or `mysqldump` (MySQL, `.sql.gz`). See [Backups](#backups-db-dump-folder-archive-and-s3-upload). |
 | `dbProcedure` | `db-procedure` | Call a stored procedure; returns `{ rows, rowCount, procedure, sql }`. See [db-procedure](#db-procedure). |
@@ -446,6 +447,7 @@ Decided for the whole step before anything is written, so `fail` never leaves ha
 | `csv` | RFC 4180; a UTF-8 byte-order mark so Excel reads `₹` and accents; a text cell starting with `= + - @` gets a leading `'` (no formula injection; numbers untouched) | `values` |
 | `xlsx` | numbers and dates as **real typed cells carrying the column's Excel number format** (`"₹"#,##0.00`, `0.0%`, `mmm yyyy`, `#,##0.0,,"M"`), so the sheet looks like the screen and still sums; an optional title and subtitle; a bold, frozen header with an autofilter; bold subtotals and a tinted total row; column widths from the first 200 rows. Excel's 1,048,576-row limit is respected (`truncated`) | `values` |
 | `pdf` | each cell exactly as displayed, numbers right-aligned by `align`, header repeated on every page, landscape past 6 columns, "Page x of y", bold or tinted total rows. Capped at 5,000 rows (`maxRows`) with a closing note | `display` |
+| `parquet` | one `.parquet` file of the raw values, typed (numbers, dates, booleans as themselves), the headers as column names (a repeated header gets ` (2)`). For pandas, DuckDB or a warehouse — see [Parquet](#parquet-the-hand-off-to-python-and-warehouses) | `values` |
 
 ```js
 await runAction(builtins.exportTable, {
@@ -479,7 +481,50 @@ await runAction(builtins.exportTable, {
 
 What it is given is assumed to come from a browser, so it is bounded: 50 pages, 200 blocks a page, 8 MB per image, 500 table rows, and 5,000 characters of text. An image that will not decode draws a note in its box instead of failing the whole file.
 
-Peers (optional, needed only by the format used): `fflate` for xlsx; `pdfkit` and `dejavu-fonts-ttf` for pdf. `writers` (`getWriter`, `INFO`, `SUPPORTED`) is exported for code that wants a writer without the action.
+Peers (optional, needed only by the format used): `fflate` for xlsx; `pdfkit` and `dejavu-fonts-ttf` for pdf; `@duckdb/node-api` for parquet. `writers` (`getWriter`, `INFO`, `SUPPORTED`) is exported for code that wants a writer without the action.
+
+## Parquet: the hand-off to Python and warehouses
+
+For moving rows between programs — Node to a Python model and back, or into a warehouse — NDJSON is the wrong file: every row repeats every column name, numbers are text, and a date comes back a string the reader has to guess. **Parquet** is stored by column and compressed (typically several times smaller), keeps each column's type, and pandas, pyarrow and DuckDB read it many times faster. It is written **by DuckDB** (already a peer here), not by a JavaScript Parquet library: native, multi-threaded, and correct on dates, timestamps and decimals.
+
+**A stream becomes a folder of parts.** A Parquet file cannot be appended to and is unreadable until its last byte (its index is written at the end), so rows stream into a folder:
+
+```
+<folder>/part-00001.parquet    partRows rows each (default 100,000)
+<folder>/part-00002.parquet
+<folder>/_done                 written last: { rows, parts }
+```
+
+Rows go into a table in a DuckDB of their own, in memory; at `partRows` it is copied out as the next part and emptied. **Memory stays about one part, whatever the total, and each row is written to disk once.** A part is written under a dot-name and renamed when whole, so a reader never sees half of one. Every reader takes the folder as one table and ignores `_done`:
+
+```python
+df = pandas.read_parquet('<folder>')                     # pandas / pyarrow
+duckdb.sql("SELECT * FROM '<folder>/*.parquet'")         # DuckDB
+```
+
+Keep parts large (the default): thousands of tiny parts cost a file open each, which on a network disk (EFS) is a round trip.
+
+```js
+var p = actions.parquet;
+var done = await p.writeAll(rows, { dir, columns?, partRows?, signal? });
+// → { path, parts: ['part-00001.parquet', …], rows, bytes, columns }
+for await (var row of p.readRows(done.path)) { … }        // a file or a folder
+await p.columnsOf(done.path);                              // [{ name, type }]
+var w = p.open({ dir }); await w.write(row); …; await w.close();   // or abort()
+```
+
+`{ file }` instead of `{ dir }` writes one file at close, its rows held until then — for a bounded export (`export-table`), not a stream.
+
+**Types.** `columns: [{ name, type }]` (the logical types every driver speaks) are used as given; otherwise they are inferred from the first `scanRows` (1,000) rows by the uploader's own inference. A value lands as it would in a DuckDB table — the DuckDB driver's type map and coercion are used as they are: `number` → DOUBLE, `datetime` → TIMESTAMPTZ, `date` → DATE, `boolean`, objects and arrays → JSON (read back as their text). `db-fetch` with `mode: 'table'` declares the table's own types, so a Postgres `numeric` (which reaches node as a string) is a number in the file, not text.
+
+**Refused, never bent:**
+- A value that does not fit its column fails the write — `Row 6: column "amount" (number) cannot hold "twelve"` (`code: 'PARQUET_ROW'`).
+- A field no earlier row had fails it too: a folder has one set of columns, and dropping a field quietly is worse than stopping. Declare the columns when the first rows do not show them all.
+- A failed or stopped write (`signal` → `code: 'STOPPED'`) removes the parts it wrote and writes no `_done`.
+
+Our bookkeeping columns (`__xeplr_id__`, `__xeplr_movement_id__`) never come along, as in every load. No rows with declared columns gives one empty part (a reader still finds the columns); no rows and none declared gives only `_done`, which reads as empty.
+
+In a workflow: `db-fetch` (`format: 'parquet'`) → the step that reads it → `db-push` (`filePath`: the folder). `export-table` with `format: 'parquet'` writes one typed file under the headers as column names (made unique).
 
 ## Uploader
 
@@ -569,6 +614,8 @@ On Postgres, MySQL and SQL Server the staging table is shaped like the target (`
 
 Beside the tables it loads, the uploader makes three of its own: `<table>_import_errors` (refused rows), `<table>__xeplr_staging` (a staged load in progress) and, in a DuckDB file, `__xeplr_loads` (its ledger). `isInternalTable(name)` (also accepts a listing row `{ name }`) says whether a table is one of these, so an app listing a database's tables for people to pick from can leave them out. They still exist and are read where they belong.
 
+It also adds its own columns to every table it loads: `__xeplr_id__` (Postgres, MySQL, SQL Server) and `__xeplr_movement_id__` (every target), the row's id and the load that wrote it. `isInternalColumn(name)` (or `{ name }`) says whether a column is one of these. **They are never carried into another table:** when a loaded table is loaded somewhere else (an import copied into a cube's warehouse, a move between databases), the uploader drops them from the target's columns, whether inferred from the rows or declared by the caller, and the new load stamps its own. Carried along, they clashed with the target's own column (`Column with name __xeplr_movement_id__ already exists`) and would have marked rows with a load that never wrote them.
+
 ### A written DuckDB file keeps no index
 
 DuckDB 1.5.4–1.5.6 has a bug ([duckdb/duckdb#26106](https://github.com/duckdb/duckdb/issues/26106), open upstream): when a process dies with rows still only in the WAL, the next open replays them into the table and the next checkpoint drops them from every index on it. The first `DELETE` that reaches one fails with *"Failed to delete all rows from index"*, which is **FATAL**: the whole file is unusable until the process restarts. Any index, unique or not. A table with no index is unaffected.
@@ -590,6 +637,7 @@ Upserts need no index: they are staged and match keys in the move. A read-only o
 | `csv` | stream | `formatConfig`: `columns` (default first row), `delimiter` (`,`), `cast` (true), `skipEmptyLines`, `trim`. |
 | `excel` | file path | `.xlsx` via `unzipper` + `saxes`; `sheet`, `startRow`. Date-styled cells arrive as `Date`. Excel's own escapes in cell text are decoded (`_x000D_` → a real line break; `_x005F_` → `_`), and Japanese phonetic runs (`<rPh>`, furigana) are not joined into the value. |
 | `json` | stream | A JSON array or NDJSON; buffers the whole file. |
+| `parquet` | file path | A `.parquet` file or a folder of parts, streamed a chunk at a time through DuckDB (`@duckdb/node-api`). Typed: no `formatConfig`. |
 | `txt` | — | Placeholder, not implemented. |
 
 **Rows pasted into one cell** (`lib/formats/pastedRows.js`). Data copied out of an ERP and pasted into Excel: a value starting with `"` (an inch mark) makes Excel put everything up to the next `"` — tabs, line breaks, whole rows — into that one cell. The rows it swallowed exist nowhere else. Seen for real: 465 rows folded into 11 cells.
@@ -733,6 +781,7 @@ node --test test/uploader/reconcile.test.js
 - **No `*_HOST`:** that database's tests are **skipped**, and each says which settings run it.
 - **`*_HOST` set:** they run, and a login the database refuses **fails** them. A configured run never passes by not connecting.
 - **A data root and an application per test process** (`--require ./test/setup-data-dir.cjs`, in `npm test`, CI, and the gate through `@flags` on `test/data-dir.test.js`), as an app has at boot. `test/data-dir.test.js` registers tenants: two companies with the same names, each refused the other's files by full path, `../`, a link and a move; `*` and no tenant; email attachments, db-push and a local upload.
+- **Parquet** (`test/formats/parquet.test.js`): parts written and read back by DuckDB's own Parquet reader — 300,000 rows in parts, declared and inferred types, a bad value and a stray column refused, a stop leaving nothing, `export-table` and `file-upload`. `test/builtins/parquet-pg.test.js` takes a real Postgres table through `db-fetch` → Parquet → `db-push` and checks the new table's types and every row.
 - **One file at a time** (`--test-concurrency=1`): the database suites share the `xeplr_actions_test` database and race to create it when run side by side.
 - **Why:** these files used to carry a fixed host, port and password each (Postgres on `:5435`). On any other machine they all failed, so the pre-commit gate and CI ran only `test/*.test.js` and never this package's sub-folders. That is how `data-operation` shipped returning no rows (fixed in 1.4.2).
 

package/index.js

@@ -48,12 +48,16 @@ module.exports = {
   uploader:      uploader,
   // The uploader's own tables (_import_errors, __xeplr_staging, __xeplr_loads), to leave out of table pickers.
   isInternalTable: require('./lib/drivers/db/internalTables').isInternalTable,
+  isInternalColumn: require('./lib/drivers/db/internalTables').isInternalColumn,
   inferColumns:  uploader.inferColumns,
   streaming:     streaming,
   drivers:       { db: dbDrivers, file: fileDrivers },
   formats:       formats,
-  // { getWriter(type), INFO, SUPPORTED } — csv / xlsx / pdf writers behind export-table.
+  // { getWriter(type), INFO, SUPPORTED } — csv / xlsx / pdf / parquet writers behind export-table.
   writers:       require('./lib/formats/writers'),
+  // { open, writeAll, readRows, columnsOf, isParquet } — rows to a folder of
+  // Parquet parts and back, through DuckDB (lib/formats/parquet.js).
+  parquet:       require('./lib/formats/parquet'),
 
   // xeplr_configs — shared control-plane DB (import/movement metadata).
   // One call per app: attachConfig({ service }); await xcfg.ready(). See

package/lib/builtins/db/columns.js

@@ -0,0 +1,63 @@
+// The columns a database source DECLARES — shared by db-move and db-fetch, so
+// both carry a table's own types instead of guessing them from values.
+
+function normalizeMapping(columns) {
+  return (columns || [])
+    .filter(function(c) { return c && c.from; })
+    .map(function(c) { return { from: c.from, to: c.to || c.from }; });
+}
+
+/**
+ * The source's DECLARED column types, in the target's names — or null when the
+ * source has none to declare.
+ *
+ * Inference reads values, and for a database source that is throwing away an
+ * answer the server already holds. Three ways it goes wrong, all of them quiet
+ * and all of them baked into a CREATE TABLE that outlives the sample:
+ *
+ *   Postgres returns `numeric` to node as a STRING. A money column arrives as
+ *   strings, infers as text, and the target column is created TEXT. You find
+ *   out when somebody tries to SUM it.
+ *
+ *   A column that is all-null in the first batch has no type, so it becomes
+ *   text — and stays text once the table exists.
+ *
+ *   A zero-padded code ('007') looks like a number and loses its zeros.
+ *
+ * Only for mode 'table': a query or a procedure has no single table to
+ * describe. Both dialects CAN describe a result set — Postgres by preparing,
+ * SQL Server via sp_describe_first_result_set — but that is a per-driver
+ * capability that does not exist yet, so those fall back to inference and the
+ * caveat above still applies to them.
+ */
+async function declaredColumns(driver, pool, input) {
+  var mode = input.mode || (input.table ? 'table' : 'query');
+  if (mode !== 'table' || !input.table) return null;
+  if (typeof driver.getTableSchema !== 'function' || typeof driver.dataTypeToLogical !== 'function') return null;
+
+  var schema = await driver.getTableSchema(pool, input.table);
+  if (!schema || !schema.length) return null;
+
+  var byName = {};
+  schema.forEach(function(c) { byName[c.name] = c; });
+
+  var mapping = normalizeMapping(input.columns);
+  // No mapping means every source column, unrenamed — so the declaration is
+  // the source's own schema in its own order.
+  var wanted = mapping.length
+    ? mapping
+    : schema.map(function(c) { return { from: c.name, to: c.name }; });
+
+  var out = [];
+  for (var i = 0; i < wanted.length; i++) {
+    var src = byName[wanted[i].from];
+    // A mapped column the source does not have is left out rather than
+    // guessed at — the SELECT will fail on it anyway, with a better message
+    // than anything invented here.
+    if (!src) return null;
+    out.push({ name: wanted[i].to, type: driver.dataTypeToLogical(src.dataType) });
+  }
+  return out.length ? out : null;
+}
+
+module.exports = { declaredColumns: declaredColumns, normalizeMapping: normalizeMapping };

package/lib/builtins/db/fetch.js

@@ -17,8 +17,17 @@
 //   Temp file: the tenant's tmp/ under XEPLR_DATA_DIR (lib/dataDir.js),
 //   named <occurrenceId>_db-fetch_<seq>.jsonl. Cleanup: none in-action — rely
 //   on the sweep-temp-files action (or db-push after it consumes the file).
+//
+// format: 'parquet' writes a FOLDER of Parquet parts instead
+// (<occurrenceId>_db-fetch_<seq>/part-00001.parquet …, then _done), typed —
+// see formats/parquet.js. For a hand-off to Python or a warehouse: smaller,
+// typed, and read far faster than NDJSON. mode 'table' declares the table's
+// own column types; a query's are inferred from its first rows. db-push reads
+// the folder back as it is.
 
 var dbDrivers  = require('../../drivers/db');
+var parquet    = require('../../formats/parquet');
+var { declaredColumns } = require('./columns');
 var { generateId } = require('@xeplr/utils/lib/helpers');
 var path = require('path');
 var fs = require('fs');
@@ -74,7 +83,11 @@ module.exports = {
     { name: 'params',         type: 'array',   order: 6,
       description: 'Bind params for mode="query".' },
     { name: 'streaming_mode', type: 'boolean', default: true, order: 7, group: 'Performance',
-      description: 'Default TRUE → spool to NDJSON file. false → inline output.rows (small reads only).' },
+      description: 'Default TRUE → spool to a file (see format). false → inline output.rows (small reads only).' },
+    { name: 'format',         type: 'string',  default: 'jsonl', order: 7, options: ['jsonl', 'parquet'], group: 'Performance',
+      description: 'The spooled file: jsonl (one JSON row per line) or parquet (a folder of typed Parquet parts, for Python or a warehouse).' },
+    { name: 'partRows',       type: 'number',  default: parquet.PART_ROWS, order: 8, group: 'Performance',
+      description: 'parquet only: rows per part file.' },
     { name: 'batchSize',      type: 'number',  default: 1000, order: 8, group: 'Performance',
       description: 'Rows pulled per cursor round-trip / stream highWaterMark.' }
   ],
@@ -100,7 +113,7 @@ module.exports = {
     }
     var output = input.streaming_mode === false
       ? { rows: rows, rowCount: rows.length, streaming: false }
-      : { filePath: '(a file in this company\'s tmp/, written when it runs)', format: 'jsonl', bytes: 0, rows: rows.length, streaming: true };
+      : { filePath: '(a file in this company\'s tmp/, written when it runs)', format: formatOf(input), bytes: 0, rows: rows.length, streaming: true };
     return { output: output, rows: rows };
   },
 
@@ -129,7 +142,29 @@ module.exports = {
         return { rows: rows, rowCount: rows.length, streaming: false };
       }
 
-      var filePath = await resolveOutputPath(system);
+      if (formatOf(input) === 'parquet') {
+        var columns = await declaredColumns(driver, pool, input);
+        var folder = await resolveOutputPath(system, '');
+        var written;
+        try {
+          written = await parquet.writeAll(rowStream, {
+            dir: folder, columns: columns, partRows: input.partRows, signal: system.signal
+          });
+        } catch (err) {
+          fs.rmSync(folder, { recursive: true, force: true });
+          throw err;
+        }
+        return {
+          filePath:  written.path,
+          format:    'parquet',
+          parts:     written.parts.length,
+          bytes:     written.bytes,
+          rows:      written.rows,
+          streaming: true
+        };
+      }
+
+      var filePath = await resolveOutputPath(system, '.jsonl');
       var stats = await writeNDJSON(filePath, rowStream);
       return {
         filePath:  filePath,
@@ -160,10 +195,16 @@ function buildFetchSql(driver, input) {
 }
 
 
-async function resolveOutputPath(system) {
-  // The tenant's own temporary folder (lib/dataDir.js).
+function formatOf(input) {
+  var f = String(input.format || 'jsonl').toLowerCase();
+  if (f !== 'jsonl' && f !== 'parquet') throw new Error('db-fetch: format must be "jsonl" or "parquet" — got "' + input.format + '"');
+  return f;
+}
+
+// The tenant's own temporary folder (lib/dataDir.js). No extension: a folder.
+async function resolveOutputPath(system, extension) {
   var dir = require('../../dataDir').scratch();
-  var base = (system.occurrenceId || generateId()) + '_db-fetch_' + generateId() + '.jsonl';
+  var base = (system.occurrenceId || generateId()) + '_db-fetch_' + generateId() + extension;
   return path.join(dir, base);
 }
 // Write an async iterable of rows to an NDJSON file, honoring write

package/lib/builtins/db/move.js

@@ -55,6 +55,7 @@ var { SqlQueue } = require('@xeplr/utils/lib/queue');
 // action — this file used to own that knowledge, which meant a procedure could
 // only be called by moving its output somewhere. See drivers/db/procedure.js.
 var procedureCall = require('../../drivers/db/procedure');
+var { declaredColumns, normalizeMapping } = require('./columns');
 var { generateId } = require('@xeplr/utils/lib/helpers');
 
 var WRITE_MODES = ['replace', 'upsert', 'append'];
@@ -275,7 +276,7 @@ module.exports = {
       //
       // Without this upload() infers from the first batch's VALUES, which is
       // correct for a CSV and quietly lossy here — see declaredColumns.
-      var declared = await declaredColumns(sourceDriver, sourcePool, input, plan);
+      var declared = await declaredColumns(sourceDriver, sourcePool, input);
 
       // The actual SQL, so a discrepancy is checkable against the source
       // directly rather than trusted on faith — this is what actually ran,
@@ -457,60 +458,7 @@ function describeWriteMode(writeMode, primaryKeys) {
   return writeMode;
 }
 
-// ── what the columns actually are ────────────────────────────────────────
-
-/**
- * The source's DECLARED column types, in the target's names — or null when the
- * source has none to declare.
- *
- * Inference reads values, and for a database source that is throwing away an
- * answer the server already holds. Three ways it goes wrong, all of them quiet
- * and all of them baked into a CREATE TABLE that outlives the sample:
- *
- *   Postgres returns `numeric` to node as a STRING. A money column arrives as
- *   strings, infers as text, and the target column is created TEXT. You find
- *   out when somebody tries to SUM it.
- *
- *   A column that is all-null in the first batch has no type, so it becomes
- *   text — and stays text once the table exists.
- *
- *   A zero-padded code ('007') looks like a number and loses its zeros.
- *
- * Only for mode 'table': a query or a procedure has no single table to
- * describe. Both dialects CAN describe a result set — Postgres by preparing,
- * SQL Server via sp_describe_first_result_set — but that is a per-driver
- * capability that does not exist yet, so those fall back to inference and the
- * caveat above still applies to them.
- */
-async function declaredColumns(driver, pool, input, plan) {
-  var mode = input.mode || (input.table ? 'table' : 'query');
-  if (mode !== 'table' || !input.table) return null;
-  if (typeof driver.getTableSchema !== 'function' || typeof driver.dataTypeToLogical !== 'function') return null;
-
-  var schema = await driver.getTableSchema(pool, input.table);
-  if (!schema || !schema.length) return null;
-
-  var byName = {};
-  schema.forEach(function(c) { byName[c.name] = c; });
-
-  var mapping = normalizeMapping(input.columns);
-  // No mapping means every source column, unrenamed — so the declaration is
-  // the source's own schema in its own order.
-  var wanted = mapping.length
-    ? mapping
-    : schema.map(function(c) { return { from: c.name, to: c.name }; });
-
-  var out = [];
-  for (var i = 0; i < wanted.length; i++) {
-    var src = byName[wanted[i].from];
-    // A mapped column the source does not have is left out rather than
-    // guessed at — the SELECT will fail on it anyway, with a better message
-    // than anything invented here.
-    if (!src) return null;
-    out.push({ name: wanted[i].to, type: driver.dataTypeToLogical(src.dataType) });
-  }
-  return out.length ? out : null;
-}
+// What the columns actually are: see ./columns.js (shared with db-fetch).
 
 // ── the source query ─────────────────────────────────────────────────────
 
@@ -664,11 +612,6 @@ function procedurePlan(driver, input, mapping, win) {
 // from there rather than kept as a second copy.
 var quoteQualifiedIdent = procedureCall.quoteQualifiedIdent;
 
-function normalizeMapping(columns) {
-  return (columns || [])
-    .filter(function(c) { return c && c.from; })
-    .map(function(c) { return { from: c.from, to: c.to || c.from }; });
-}
 
 /** Dialect placeholder for bind param n (0-based). */
 function placeholders(dbType) {

package/lib/builtins/db/push.js

@@ -2,12 +2,16 @@
 //
 // Composes: drivers.db.<type> + uploader.upload + SqlQueue
 //
-// Inputs accept EITHER `rows` (inline) OR `filePath` (NDJSON, one row per line
-// — matches spool()'s output shape). The action creates its own SqlQueue per
+// Inputs accept EITHER `rows` (inline) OR `filePath`: an NDJSON file (one row
+// per line — matches spool()'s output shape), or Parquet — a .parquet file or
+// a folder of parts, as db-fetch format 'parquet' writes. A Parquet source
+// declares its columns' types, and the target is created with them rather
+// than with types guessed from values. The action creates its own SqlQueue per
 // call and tears it down when done; movement lifecycle is fully self-contained.
 
 var { upload } = require('../../uploader');
 var dbDrivers  = require('../../drivers/db');
+var parquet    = require('../../formats/parquet');
 var { SqlQueue } = require('@xeplr/utils/lib/queue');
 var { generateId } = require('@xeplr/utils/lib/helpers');
 var fs = require('fs');
@@ -16,7 +20,7 @@ var readline = require('readline');
 module.exports = {
   name: 'db-push',
   description: 'Push rows into a target DB table. Supports UPSERT via primaryKeys, ' +
-               'NDJSON streaming via filePath, movement-scoped rollback, and error-table ' +
+               'NDJSON or Parquet streaming via filePath, movement-scoped rollback, and error-table ' +
                'routing on failure. Routes to the driver named by input.dbType.',
   requires: [],   // per-driver requires checked at runtime once dbType is known
 
@@ -41,7 +45,7 @@ module.exports = {
     { name: 'rows',        type: 'array',   order: 5,
       description: 'Inline rows (mutually exclusive with filePath).' },
     { name: 'filePath',    type: 'string',  order: 6,
-      description: 'Path to an NDJSON file (one row per line). Use for large uploads.' },
+      description: 'An NDJSON file (one row per line), a .parquet file, or a folder of Parquet parts. Use for large uploads.' },
     { name: 'movementId',  type: 'string',  order: 7, group: 'Advanced',
       description: 'Correlation key for rollback. Defaults to system.occurrenceId or a fresh id.' },
     { name: 'batchSize',   type: 'number',  default: 5000, order: 8, group: 'Performance' },
@@ -57,6 +61,8 @@ module.exports = {
 
     var movementId = input.movementId || system.occurrenceId || ('mv_' + generateId());
     var source = resolveSource(input);
+    // A Parquet file says what its columns are; nothing is guessed.
+    var columns = source.parquet ? await parquet.columnsOf(source.path) : null;
 
     var pool = await driver.connect(input.connection);
     var connectionName = input.targetTable;
@@ -75,7 +81,7 @@ module.exports = {
 
     try {
       return await upload({
-        source:              source,
+        source:              source.rows,
         driver:              driver,
         connection:          pool,
         targetTable:         input.targetTable,
@@ -84,6 +90,7 @@ module.exports = {
         queue:               queue,
         connectionName:      connectionName,
         dbType:              input.dbType,       // recorded in import_meta
+        columns:             columns && columns.length ? columns : undefined,
         // Movement metadata persists to xeplr_config when the framework supplies
         // a meta store (meta-store-knex bound to xeplr_config); else no-op.
         metaStore:           ctx.metaStore,
@@ -96,10 +103,14 @@ module.exports = {
   }
 };
 
-// Return an async iterable for whichever input shape the caller supplied.
+// { rows: async iterable, parquet?, path? } for whichever input the caller supplied.
 function resolveSource(input) {
-  if (Array.isArray(input.rows) && input.rows.length > 0) return asyncFromArray(input.rows);
-  if (input.filePath)                                     return asyncFromNDJSONFile(require('../../dataDir').resolve(input.filePath, 'The rows file'));
+  if (Array.isArray(input.rows) && input.rows.length > 0) return { rows: asyncFromArray(input.rows) };
+  if (input.filePath) {
+    var file = require('../../dataDir').resolve(input.filePath, 'The rows file');
+    if (parquet.isParquet(file)) return { rows: parquet.readRows(file), parquet: true, path: file };
+    return { rows: asyncFromNDJSONFile(file) };
+  }
   throw new Error('db-push: either input.rows (non-empty) or input.filePath is required');
 }
 

package/lib/builtins/export/table.js

@@ -1,4 +1,4 @@
-// export-table — rows to a CSV, Excel or PDF file.
+// export-table — rows to a CSV, Excel, PDF or Parquet file.
 //
 // The rows arrive already DECIDED: which columns, in what order, under what
 // heading, and for each cell both its raw value and the text a reader saw.
@@ -10,6 +10,8 @@
 //   xlsx  the raw values as TYPED cells carrying each column's number format,
 //         so the sheet looks like the screen and still sums
 //   pdf   the display text, exactly as shown
+//   parquet  the raw values, typed, under the headers as column names — a
+//         file for pandas, DuckDB or a warehouse (formats/parquet.js)
 //
 // Rows come inline (`rows`) or from an NDJSON file (`rowsFile`, one row per
 // line — what streaming/spool writes), so a large result never has to be one
@@ -59,13 +61,13 @@ module.exports = {
   name: 'export-table',
   // What it hands on, with example values: it writes, so it is never previewed.
   example: { filePath: 'exports/sales.xlsx', fileName: 'sales.xlsx', contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', bytes: 18432, rows: 1420, truncated: false, skipped: false },
-  description: 'Write rows to a CSV, Excel (.xlsx) or PDF file. Excel cells stay numbers with their number format; ' +
-               'a PDF shows each cell as it was displayed.',
+  description: 'Write rows to a CSV, Excel (.xlsx), PDF or Parquet file. Excel cells stay numbers with their number format; ' +
+               'a PDF shows each cell as it was displayed; Parquet keeps each column typed.',
   requires: [],
 
   inputSchema: [
     { name: 'format', type: 'string', required: true, order: 1, enum: writers.SUPPORTED,
-      description: 'csv, xlsx or pdf.' },
+      description: 'csv, xlsx, pdf or parquet.' },
     { name: 'document', type: 'object', order: 2,
       description: '{ title?, subtitle?, footer?, columns: [{ header, key?, kind?: number|date|text, numFmt?, align?, width? }] }. ' +
                    'numFmt is an Excel number format ("#,##0.00", "0.0%", "mmm yyyy"). key: which field of a plain record ' +

package/lib/drivers/db/duckdb.js

@@ -1044,6 +1044,9 @@ module.exports = {
 
   toPgType:                 toPgType,
   dataTypeToLogical:        dataTypeToLogical,
+  // The Parquet writer (formats/parquet.js) appends through it, so a value
+  // reaches a .parquet file exactly as it would reach a DuckDB table.
+  coerceValue:              coerceValue,
   quoteIdent:               quoteIdent,
   buildCreateTableSql:      buildCreateTableSql,
   buildCreateErrorTableSql: buildCreateErrorTableSql,

package/lib/drivers/db/internalTables.js

@@ -13,6 +13,13 @@ var ERRORS_SUFFIX = '_import_errors';
 var STAGING_SUFFIX = require('./staging').SUFFIX;
 var LOADS_LEDGER = '__xeplr_loads';
 
+// THE COLUMNS IT ADDS TO EVERY TABLE IT LOADS: a row's own id and the
+// movement that wrote it. They describe a load into THAT table, so they are
+// never carried into another one — loading a loaded table elsewhere (an
+// import copied into a cube's warehouse, a move between databases) leaves
+// them behind and the new load stamps its own (uploader/index.js).
+var INTERNAL_COLUMNS = ['__xeplr_id__', '__xeplr_movement_id__'];
+
 function endsWith(s, suffix) {
   return s.length > suffix.length && s.slice(-suffix.length) === suffix;
 }
@@ -23,8 +30,16 @@ function isInternalTable(name) {
   return endsWith(s, ERRORS_SUFFIX) || endsWith(s, STAGING_SUFFIX) || s === LOADS_LEDGER;
 }
 
+/** Whether a column is one the uploader adds itself. A name, or { name }. */
+function isInternalColumn(column) {
+  var s = String(column == null ? '' : (typeof column === 'object' ? column.name || '' : column));
+  return INTERNAL_COLUMNS.indexOf(s) !== -1;
+}
+
 module.exports = {
   isInternalTable: isInternalTable,
+  isInternalColumn: isInternalColumn,
+  INTERNAL_COLUMNS: INTERNAL_COLUMNS,
   ERRORS_SUFFIX: ERRORS_SUFFIX,
   STAGING_SUFFIX: STAGING_SUFFIX,
   LOADS_LEDGER: LOADS_LEDGER

package/lib/formats/index.js

@@ -7,7 +7,8 @@ var FORMATS = {
   excel: function() { return require('./excel'); },
   csv:   function() { return require('./csv'); },
   txt:   function() { return require('./txt'); },
-  json:  function() { return require('./json'); }
+  json:  function() { return require('./json'); },
+  parquet: function() { return require('./parquet'); }
 };
 
 function getFormat(type) {

package/lib/formats/parquet.js

@@ -0,0 +1,282 @@
+// Parquet — read and write, through DuckDB. No JavaScript Parquet library:
+// DuckDB is already a peer here, writes Parquet in native code across
+// threads, and keeps decimals, dates and timestamps the way pandas and
+// pyarrow read them back.
+//
+// ── WRITING: a folder of parts ──────────────────────────────────────────
+//
+// A Parquet file cannot be appended to, and its index is written last, so a
+// stream of rows becomes a FOLDER of part files:
+//
+//   <dir>/part-00001.parquet   each partRows rows (default 100,000)
+//   <dir>/part-00002.parquet
+//   <dir>/_done                written last: every part is complete
+//
+// Rows are appended into a table in a DuckDB of their own, in memory; at
+// partRows the table is copied out as the next part and emptied. Memory stays
+// about one part whatever the total, and every row is written to disk once.
+// A part is written under a dot-name and renamed when complete, so nothing
+// ever reads half a part. pandas (`read_parquet(dir)`), pyarrow and DuckDB
+// (`'<dir>/*.parquet'`) all read the folder as one table and skip `_done`.
+//
+// `file` instead of `dir` writes ONE file at close — for an export somebody
+// downloads. Its rows are held in memory until then, so it is for bounded row
+// counts, not for a stream of unknown size.
+//
+// ── TYPES ───────────────────────────────────────────────────────────────
+//
+// Declared (`columns: [{ name, type }]`, the logical types every driver
+// speaks) or, when not, inferred from the first scanRows rows with the
+// uploader's own inference (uploader/infer.js). The DuckDB driver's type map
+// and value coercion are used as they are, so a value lands in a .parquet
+// file exactly as it would in a DuckDB table: 'number' is DOUBLE, 'datetime'
+// is TIMESTAMPTZ, 'date' DATE, objects and arrays JSON.
+//
+// A value that does not fit its column fails the write naming the row and
+// the column — never stored as something else. A key no earlier row had
+// fails it too: a Parquet folder has one set of columns, and dropping a
+// field quietly is worse than stopping. Declare the columns when the first
+// rows do not show them all. Our own bookkeeping columns (__xeplr_…__) never
+// come along, as in every load.
+//
+// ── READING ─────────────────────────────────────────────────────────────
+//
+// readRows(path) streams a file or a folder of parts, a chunk at a time,
+// through the DuckDB driver's own fetchStream — the same values a DuckDB
+// table gives (dates as Date, BIGINT narrowed, a JSON column as its text).
+
+var fs = require('fs');
+var path = require('path');
+var duck = require('../drivers/db/duckdb');
+var { inferColumns } = require('../uploader/infer');
+var { isInternalColumn } = require('../drivers/db/internalTables');
+
+var PART_ROWS = 100000;
+var SCAN_ROWS = 1000;
+var DONE = '_done';
+var TABLE = 'rows';
+
+function literal(p) { return "'" + String(p).replace(/'/g, "''") + "'"; }
+function partName(n) { return 'part-' + String(n).padStart(5, '0') + '.parquet'; }
+
+/** The path DuckDB reads: a file as it is, a folder as its parts. */
+function sourceOf(target) {
+  return fs.statSync(target).isDirectory() ? path.join(target, '*.parquet') : target;
+}
+
+/**
+ * True when `target` is a .parquet file, or a folder of parts — including one
+ * this writer finished with no rows (only `_done`), which reads as empty.
+ */
+function isParquet(target) {
+  try {
+    var st = fs.statSync(target);
+    if (st.isFile()) return /\.parquet$/i.test(target);
+    return st.isDirectory() && fs.readdirSync(target).some(function(f) { return /\.parquet$/i.test(f) || f === DONE; });
+  } catch (_) { return false; }
+}
+
+/** Whether there is anything to read: a file, or a folder with at least one part. */
+function hasRows(target) {
+  var st = fs.statSync(target);
+  return st.isFile() || fs.readdirSync(target).some(function(f) { return /\.parquet$/i.test(f); });
+}
+
+/**
+ * Open a writer.
+ *
+ * @param {object} opts
+ *   dir       folder to write parts into (created if missing), or
+ *   file      one .parquet file, written at close
+ *   columns   [{ name, type }] — declared; otherwise inferred
+ *   partRows  rows per part (dir only), default 100,000
+ *   scanRows  rows read before inferring types, default 1,000
+ * @returns {{ write(row): Promise<void>, close(): Promise<object>, abort(): Promise<void> }}
+ *   close() → { path, parts: [names], rows, bytes, columns }
+ */
+function open(opts) {
+  opts = opts || {};
+  if (!opts.dir === !opts.file) throw new Error('parquet: give a dir (parts) or a file — one of them.');
+  var single = Boolean(opts.file);
+  var target = single ? opts.file : opts.dir;
+  fs.mkdirSync(single ? path.dirname(target) : target, { recursive: true });
+  var partRows = single ? Infinity : Math.max(1, opts.partRows || PART_ROWS);
+  var scanRows = Math.max(1, opts.scanRows || SCAN_ROWS);
+
+  var instance = null, conn = null, appender = null;
+  var columns = null, known = null;
+  var buffer = [];
+  var rows = 0, inPart = 0;
+  var parts = [], written = [];
+  var finished = false;
+
+  async function start() {
+    var declared = (opts.columns && opts.columns.length) ? opts.columns : inferColumns(buffer);
+    columns = declared.filter(function(c) { return !isInternalColumn(c); })
+      .map(function(c) { return { name: c.name, type: c.type || 'string' }; });
+    known = new Set(columns.map(function(c) { return c.name; }));
+    var held = buffer;
+    buffer = null;
+    if (!columns.length) return;
+    var api = require('@duckdb/node-api');
+    instance = await api.DuckDBInstance.create(':memory:', { TimeZone: 'UTC' });
+    conn = await instance.connect();
+    await conn.run('CREATE TABLE ' + TABLE + ' (' + columns.map(function(c) {
+      return duck.quoteIdent(c.name) + ' ' + duck.toPgType(c.type);
+    }).join(', ') + ')');
+    appender = await conn.createAppender(TABLE);
+    for (var i = 0; i < held.length; i++) await append(held[i], rows - held.length + i + 1);
+  }
+
+  function refuse(n, message) {
+    return Object.assign(new Error('Row ' + n.toLocaleString('en-IN') + ': ' + message), { code: 'PARQUET_ROW' });
+  }
+
+  async function append(row, n) {
+    for (var k in row) {
+      if (!known.has(k) && !isInternalColumn(k)) {
+        throw refuse(n, 'it has a column "' + k + '" that the rows before it did not. ' +
+          'Declare the columns when the first rows do not show them all.');
+      }
+    }
+    for (var i = 0; i < columns.length; i++) {
+      var c = columns[i];
+      var v = duck.coerceValue(row[c.name], c.type);
+      try {
+        if (v === null) appender.appendNull();
+        else if (typeof v === 'number' && c.type === 'number') appender.appendDouble(v);
+        else if (typeof v === 'boolean' && c.type === 'boolean') appender.appendBoolean(v);
+        else appender.appendVarchar(String(v));
+      } catch (err) {
+        throw refuse(n, 'column "' + c.name + '" (' + c.type + ') cannot hold ' + JSON.stringify(row[c.name]) + ' — ' + err.message);
+      }
+    }
+    appender.endRow();
+    inPart++;
+    if (inPart >= partRows) await flush();
+  }
+
+  // Copy the table out as the next part, under a dot-name until it is whole.
+  async function flush() {
+    appender.closeSync();
+    var name = single ? path.basename(target) : partName(parts.length + 1);
+    var dest = single ? target : path.join(target, name);
+    var tmp = path.join(path.dirname(dest), '.' + path.basename(dest) + '.tmp');
+    written.push(tmp);
+    await conn.run('COPY ' + TABLE + ' TO ' + literal(tmp) + ' (FORMAT parquet, COMPRESSION zstd)');
+    fs.renameSync(tmp, dest);
+    written[written.length - 1] = dest;
+    parts.push(name);
+    inPart = 0;
+    await conn.run('DELETE FROM ' + TABLE);
+    appender = await conn.createAppender(TABLE);
+  }
+
+  function shut() {
+    try { if (appender) appender.closeSync(); } catch (_) {}
+    try { if (conn) conn.closeSync(); } catch (_) {}
+    try { if (instance) instance.closeSync(); } catch (_) {}
+    appender = conn = instance = null;
+  }
+
+  return {
+    write: async function(row) {
+      if (finished) throw new Error('parquet: write after close');
+      rows++;
+      if (buffer) {
+        buffer.push(row || {});
+        if (buffer.length >= scanRows) await start();
+        return;
+      }
+      if (columns.length) return append(row || {}, rows);
+      // No columns: the first rows had none. A later one with fields is refused, never dropped.
+      var extra = Object.keys(row || {}).filter(function(k) { return !isInternalColumn(k); });
+      if (extra.length) throw refuse(rows, 'it has a column "' + extra[0] + '" that the rows before it did not.');
+    },
+
+    close: async function() {
+      if (finished) throw new Error('parquet: closed twice');
+      finished = true;
+      try {
+        if (buffer) await start();
+        // One part even when empty, so a reader still finds the columns.
+        // With no rows and none declared there are no columns to write.
+        if (columns.length && (inPart > 0 || parts.length === 0)) await flush();
+        if (!single) fs.writeFileSync(path.join(target, DONE), JSON.stringify({ rows: rows, parts: parts.length }) + '\n');
+      } finally {
+        shut();
+      }
+      var bytes = parts.reduce(function(sum, name) {
+        return sum + fs.statSync(single ? target : path.join(target, name)).size;
+      }, 0);
+      return { path: target, parts: parts, rows: rows, bytes: bytes, columns: columns };
+    },
+
+    /** Stop and remove what this writer wrote. The folder itself is the caller's. */
+    abort: async function() {
+      finished = true;
+      shut();
+      written.forEach(function(f) { try { fs.rmSync(f, { force: true }); } catch (_) {} });
+      if (!single) { try { fs.rmSync(path.join(target, DONE), { force: true }); } catch (_) {} }
+    }
+  };
+}
+
+/** Write every row of an (async) iterable; aborts and rethrows on failure. */
+async function writeAll(rowIterable, opts) {
+  var w = open(opts);
+  try {
+    for await (var row of rowIterable) {
+      if (opts.signal && opts.signal.aborted) throw Object.assign(new Error('Stopped by request'), { code: 'STOPPED' });
+      await w.write(row);
+    }
+    return await w.close();
+  } catch (err) {
+    await w.abort();
+    throw err;
+  }
+}
+
+// Reading goes through a shared in-memory DuckDB: read_parquet keeps no state
+// in it, so callers never meet each other there.
+async function withReader(fn) {
+  var pool = await duck.connect({ file: ':memory:', access: 'rw' });
+  try { return await fn(pool); } finally { await duck.close(pool); }
+}
+
+/** A file's or a folder's columns, as logical types: [{ name, type }]. */
+async function columnsOf(target) {
+  if (!isParquet(target) || !hasRows(target)) return [];
+  return withReader(async function(pool) {
+    var res = await duck.query(pool, 'DESCRIBE SELECT * FROM read_parquet(' + literal(sourceOf(target)) + ')', []);
+    return res.rows.map(function(r) { return { name: r.column_name, type: duck.dataTypeToLogical(r.column_type) }; });
+  });
+}
+
+/** Stream the rows of a .parquet file or a folder of parts. */
+async function* readRows(target, opts) {
+  opts = opts || {};
+  if (!isParquet(target)) throw new Error('parquet: "' + target + '" is not a .parquet file or a folder of parts.');
+  if (!hasRows(target)) return;
+  var pool = await duck.connect({ file: ':memory:', access: 'rw' });
+  try {
+    yield* duck.fetchStream(pool, { sql: 'SELECT * FROM read_parquet(' + literal(sourceOf(target)) + ')', signal: opts.signal });
+  } finally {
+    await duck.close(pool);
+  }
+}
+
+module.exports = {
+  // As a source format (formats/index.js): file-upload and columnsOf take a path.
+  requires: ['@duckdb/node-api'],
+  inputMode: 'path',
+  parsePath: function(filePath, opts) { return readRows(filePath, opts); },
+
+  open: open,
+  writeAll: writeAll,
+  readRows: readRows,
+  columnsOf: columnsOf,
+  isParquet: isParquet,
+  PART_ROWS: PART_ROWS,
+  DONE: DONE
+};

package/lib/formats/writers/index.js

@@ -5,13 +5,15 @@
 var WRITERS = {
   csv:  function() { return require('./csv'); },
   xlsx: function() { return require('./xlsx'); },
-  pdf:  function() { return require('./pdf'); }
+  pdf:  function() { return require('./pdf'); },
+  parquet: function() { return require('./parquet'); }
 };
 
 var INFO = {
   csv:  { extension: 'csv',  contentType: 'text/csv; charset=utf-8' },
   xlsx: { extension: 'xlsx', contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' },
-  pdf:  { extension: 'pdf',  contentType: 'application/pdf' }
+  pdf:  { extension: 'pdf',  contentType: 'application/pdf' },
+  parquet: { extension: 'parquet', contentType: 'application/vnd.apache.parquet' }
 };
 
 function getWriter(type) {

package/lib/formats/writers/parquet.js

@@ -0,0 +1,48 @@
+// Parquet writer for export-table — one .parquet file, the raw values, typed.
+//
+// Column names are the headers (made unique: Parquet columns need distinct
+// names). Types come from the values themselves (formats/parquet.js), so a
+// number stays a number and a date a date for whatever program reads it.
+// The rows are held until close, in DuckDB's memory: an export is bounded.
+
+var parquet = require('../parquet');
+
+module.exports = {
+  requires: ['@duckdb/node-api'],
+
+  /**
+   * @param {string} filePath
+   * @param {object} doc - { columns: [{ header }] }
+   * @returns {{ writeRow(row): Promise<boolean>, close(): Promise<{ rows, truncated }> }}
+   */
+  open: function(filePath, doc) {
+    var seen = {};
+    var names = (doc.columns || []).map(function(c, i) {
+      var base = c.header == null || c.header === '' ? 'column_' + (i + 1) : String(c.header);
+      var name = base;
+      for (var n = 2; seen[name]; n++) name = base + ' (' + n + ')';
+      seen[name] = true;
+      return name;
+    });
+    var w = parquet.open({ file: filePath });
+
+    return {
+      writeRow: async function(row) {
+        var values = row.values || [];
+        var record = {};
+        for (var i = 0; i < names.length; i++) record[names[i]] = values[i] === undefined ? null : values[i];
+        try { await w.write(record); } catch (err) { await w.abort(); throw err; }
+        return true;
+      },
+      close: async function() {
+        try {
+          var done = await w.close();
+          return { rows: done.rows, truncated: false };
+        } catch (err) {
+          await w.abort();
+          throw err;
+        }
+      }
+    };
+  }
+};

package/lib/uploader/index.js

@@ -59,6 +59,7 @@
 //
 // See project_xeplr_actions_streaming.md for the full contract.
 
+var { isInternalColumn } = require('../drivers/db/internalTables');
 var path = require('path');
 var fsp = require('fs/promises');
 
@@ -149,6 +150,12 @@ async function upload(opts) {
             var scanLimit = Math.min(sample.length, opts.firstBatchScanRows);
             declared = inferColumns(sample.slice(0, scanLimit));
           }
+          // OUR OWN COLUMNS NEVER COME ALONG. A source that was itself loaded
+          // by the uploader (an import's table, a replica) has its own
+          // __xeplr_id__ and __xeplr_movement_id__; carried here they clashed
+          // with this table's ("already exists") and would have claimed rows
+          // for a load that never wrote them. This load stamps its own.
+          declared = declared.filter(function(c) { return !isInternalColumn(c); });
           columns = await runBootstrap(driver, pool, opts.targetTable, declared, opts.primaryKeys, opts.staged);
           if (opts.staged) {
             opts.writeTable = await driver.prepareStaging(pool, opts.targetTable);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "2.2.5",
+  "version": "2.2.7",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "scripts": {
@@ -23,7 +23,8 @@
     "export",
     "csv",
     "xlsx",
-    "pdf"
+    "pdf",
+    "parquet"
   ],
   "author": "xeplr",
   "license": "MIT",