@xeplr/actions 2.2.4 → 2.2.6

package/README.md

@@ -122,7 +122,7 @@ The input is checked exactly as a run checks it.
 
 | previewed (reads, up to 5) | example only (writes or sends; never run) |
 |---|---|
-| `db-fetch` (through the run's own cursor, then stops; nothing spooled), `db-list-tables` / `-views` / `-procedures` / `-columns`, `file-list`, `file-read`, `data-operation` (the run itself: it only computes), `email-read` (the mailbox opened **read-only**, IMAP `EXAMINE`, so nothing is marked read) | `file-move`, `file-delete`, `email-send`, `push-send`, `email-move`, `email-delete`, `email-download-email`, `email-download-attachments`, `export-table`, `export-pages`, `db-procedure` (a procedure may write) |
+| `db-fetch` (through the run's own cursor, then stops; nothing spooled), `db-list-tables` / `-views` / `-procedures` / `-columns` / `-foreign-keys`, `file-list`, `file-read`, `data-operation` (the run itself: it only computes), `email-read` (the mailbox opened **read-only**, IMAP `EXAMINE`, so nothing is marked read) | `file-move`, `file-delete`, `email-send`, `push-send`, `email-move`, `email-delete`, `email-download-email`, `email-download-attachments`, `export-table`, `export-pages`, `db-procedure` (a procedure may write) |
 
 Neither: `db-push`, `db-move`, `http-request` (an outside system, which may change something), `spawn-program`, `file-upload`; a designer asks for an example by hand. `test/previews.test.js` checks it on a real folder, file and Postgres: five rows read and no file spooled, the real count kept, a deleting step's file still there.
 
@@ -144,6 +144,7 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
 | `dbListViews` | `db-list-views` | `{ views }`. |
 | `dbListProcedures` | `db-list-procedures` | `{ procedures }` (functions excluded). |
 | `dbListColumns` | `db-list-columns` | `{ columns }` — `name`, `dataType`, `udtName` and `maxLength` (characters; `-1` is SQL Server `(max)`, `null` when it has none) for one table. |
+| `dbListForeignKeys` | `db-list-foreign-keys` | `{ foreignKeys }` — each foreign key between the tables of a schema/database: `{ name, table, columns, refTable, refColumns }`, a composite key's columns in key order. Only keys whose **both** tables are in the schema `db-list-tables` lists, so every name is one it returns. Postgres (`public`), MySQL (the connection's database), SQL Server (every schema unless `schema`), DuckDB (`main`). What the BI dataset canvas joins two tables on without asking. |
 | `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`) as a New table, Append or Replace (`mode`, see [Import modes](#import-modes)). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
 | `sendEmail` | `email-send` | Send one message. See [Email](#email). |
 | `pushSend` | `push-send` | A push notification on people's phones and computers, even with the app closed, by `emails` and/or `userIds`: `title` (required), `body`, `url` (a page of the app a tap opens), `tag`. Returns `{ devices, sent, removed, failed }`. Through [`@xeplr/push`](https://www.npmjs.com/package/@xeplr/push), which the host sets up. |
@@ -568,6 +569,8 @@ On Postgres, MySQL and SQL Server the staging table is shaped like the target (`
 
 Beside the tables it loads, the uploader makes three of its own: `<table>_import_errors` (refused rows), `<table>__xeplr_staging` (a staged load in progress) and, in a DuckDB file, `__xeplr_loads` (its ledger). `isInternalTable(name)` (also accepts a listing row `{ name }`) says whether a table is one of these, so an app listing a database's tables for people to pick from can leave them out. They still exist and are read where they belong.
 
+It also adds its own columns to every table it loads: `__xeplr_id__` (Postgres, MySQL, SQL Server) and `__xeplr_movement_id__` (every target), the row's id and the load that wrote it. `isInternalColumn(name)` (or `{ name }`) says whether a column is one of these. **They are never carried into another table:** when a loaded table is loaded somewhere else (an import copied into a cube's warehouse, a move between databases), the uploader drops them from the target's columns, whether inferred from the rows or declared by the caller, and the new load stamps its own. Carried along, they clashed with the target's own column (`Column with name __xeplr_movement_id__ already exists`) and would have marked rows with a load that never wrote them.
+
 ### A written DuckDB file keeps no index
 
 DuckDB 1.5.4–1.5.6 has a bug ([duckdb/duckdb#26106](https://github.com/duckdb/duckdb/issues/26106), open upstream): when a process dies with rows still only in the WAL, the next open replays them into the table and the next checkpoint drops them from every index on it. The first `DELETE` that reaches one fails with *"Failed to delete all rows from index"*, which is **FATAL**: the whole file is unusable until the process restarts. Any index, unique or not. A table with no index is unaffected.

package/index.js

@@ -48,6 +48,7 @@ module.exports = {
   uploader:      uploader,
   // The uploader's own tables (_import_errors, __xeplr_staging, __xeplr_loads), to leave out of table pickers.
   isInternalTable: require('./lib/drivers/db/internalTables').isInternalTable,
+  isInternalColumn: require('./lib/drivers/db/internalTables').isInternalColumn,
   inferColumns:  uploader.inferColumns,
   streaming:     streaming,
   drivers:       { db: dbDrivers, file: fileDrivers },

package/lib/builtins/db/list-foreign-keys.js

@@ -0,0 +1,41 @@
+// db-list-foreign-keys action — the foreign keys between the tables of one
+// schema/database, each with its columns in key order: { name, table,
+// columns, refTable, refColumns }. What an app joins two tables on without
+// asking (the BI dataset canvas does). Only keys whose both tables are in the
+// schema db-list-tables lists. A small catalogue read, like db-list-tables.
+
+var dbDrivers = require('../../drivers/db');
+
+module.exports = {
+  name: 'db-list-foreign-keys',
+  description: 'List foreign keys between tables of a schema/database. Routes to the driver named by input.dbType.',
+  requires: [],   // per-driver requires checked at runtime once dbType is known
+
+  inputSchema: [
+    { name: 'dbType',     type: 'string', required: true, default: 'postgres', order: 1,
+      description: 'postgres | mysql | mssql | duckdb' },
+    { name: 'connection', type: 'object', required: true, order: 2,
+      description: 'Connection config: { host, port, user, password, database } (duckdb: { file })' },
+    { name: 'schema',     type: 'string', order: 3,
+      description: 'Schema to read (postgres public, duckdb main, mssql every schema, mysql the connection database by default)' }
+  ],
+
+  // Only reads the database's own catalogue, so the preview is the run.
+  preview: async function(ctx) { return { output: await module.exports.execute(ctx) }; },
+
+  execute: async function(ctx) {
+    var input = ctx.input || {};
+    var driver = dbDrivers.getDriver(input.dbType);
+    dbDrivers.checkDriverRequires('db-list-foreign-keys', input.dbType, driver);
+    if (typeof driver.listForeignKeys !== 'function') {
+      throw new Error('db-list-foreign-keys: driver "' + input.dbType + '" does not implement listForeignKeys');
+    }
+
+    var pool = await driver.connect(input.connection);
+    try {
+      return { foreignKeys: await driver.listForeignKeys(pool, input.schema) };
+    } finally {
+      await driver.close(pool);
+    }
+  }
+};

package/lib/builtins/index.js

@@ -49,6 +49,8 @@ module.exports = {
   dbListViews:      require('./db/list-views'),
   dbListProcedures: require('./db/list-procedures'),
   dbListColumns:    require('./db/list-columns'),
+  // The foreign keys between a schema's tables, composite ones in key order.
+  dbListForeignKeys: require('./db/list-foreign-keys'),
 
   // File actions — factory-shaped for the same reason (meta store binding).
   fileUpload:   require('./file/upload'),

package/lib/drivers/db/duckdb.js

@@ -354,6 +354,7 @@ async function acquire(entry) {
 // own connection, and the time limit is a timer that does the same. A query
 // still waiting for a free connection is let go of before it ever runs.
 var stop = require('./stop');
+var foreignKeys = require('./foreignKeys');
 
 /** acquire(), unless the stop comes first; a connection that arrives too late goes straight back. */
 function acquireUnlessStopped(entry, signal) {
@@ -658,6 +659,28 @@ async function listTables(pool, schema) {
   return r.rows.map(function(row) { return row.table_name; });
 }
 
+// Foreign keys between this schema's tables — see foreignKeys.js.
+async function listForeignKeys(pool, schema) {
+  var r = await query(pool,
+    "SELECT constraint_index, table_name, constraint_column_names, referenced_table, referenced_column_names " +
+    "FROM duckdb_constraints() WHERE constraint_type = 'FOREIGN KEY' AND schema_name = $1 " +
+    'ORDER BY table_name, constraint_index',
+    [schema || 'main']);
+  var rows = [];
+  r.rows.forEach(function(row) {
+    // LIST columns: this driver hands them back as arrays.
+    var cols = row.constraint_column_names || [];
+    var refs = row.referenced_column_names || [];
+    cols.forEach(function(col, i) {
+      rows.push({ name: row.table_name + '_fk_' + row.constraint_index, table: row.table_name,
+        refTable: row.referenced_table, column: col, refColumn: refs[i] });
+    });
+  });
+  return foreignKeys.group(rows);
+}
+
+
+
 async function listViews(pool, schema) {
   var r = await query(pool,
     'SELECT table_name FROM information_schema.views WHERE table_schema = $1 ORDER BY table_name',
@@ -1039,6 +1062,7 @@ module.exports = {
 
   getTableSchema:  getTableSchema,
   listTables:      listTables,
+  listForeignKeys: listForeignKeys,
   listViews:       listViews,
   listProcedures:  listProcedures,
 

package/lib/drivers/db/foreignKeys.js

@@ -0,0 +1,26 @@
+// Foreign keys between the tables of one schema, as every SQL driver's
+// listForeignKeys returns them — what the BI dataset canvas joins two tables
+// on without asking. One entry per constraint, its columns in key order (a
+// composite key keeps every pair): { name, table, columns, refTable,
+// refColumns }. Each driver reads its own catalogue one row per column pair
+// and hands the rows here; only keys whose BOTH tables are in the schema
+// listTables lists, so every name is one listTables also returns.
+
+/** Rows of { name, table, refTable, column, refColumn }, in key order → one entry per key. */
+function group(rows) {
+  var byKey = {};
+  var out = [];
+  (rows || []).forEach(function(r) {
+    var key = r.table + '\u0000' + r.name;
+    var fk = byKey[key];
+    if (!fk) {
+      fk = byKey[key] = { name: r.name, table: r.table, columns: [], refTable: r.refTable, refColumns: [] };
+      out.push(fk);
+    }
+    fk.columns.push(r.column);
+    fk.refColumns.push(r.refColumn);
+  });
+  return out;
+}
+
+module.exports = { group: group };

package/lib/drivers/db/internalTables.js

@@ -13,6 +13,13 @@ var ERRORS_SUFFIX = '_import_errors';
 var STAGING_SUFFIX = require('./staging').SUFFIX;
 var LOADS_LEDGER = '__xeplr_loads';
 
+// THE COLUMNS IT ADDS TO EVERY TABLE IT LOADS: a row's own id and the
+// movement that wrote it. They describe a load into THAT table, so they are
+// never carried into another one — loading a loaded table elsewhere (an
+// import copied into a cube's warehouse, a move between databases) leaves
+// them behind and the new load stamps its own (uploader/index.js).
+var INTERNAL_COLUMNS = ['__xeplr_id__', '__xeplr_movement_id__'];
+
 function endsWith(s, suffix) {
   return s.length > suffix.length && s.slice(-suffix.length) === suffix;
 }
@@ -23,8 +30,16 @@ function isInternalTable(name) {
   return endsWith(s, ERRORS_SUFFIX) || endsWith(s, STAGING_SUFFIX) || s === LOADS_LEDGER;
 }
 
+/** Whether a column is one the uploader adds itself. A name, or { name }. */
+function isInternalColumn(column) {
+  var s = String(column == null ? '' : (typeof column === 'object' ? column.name || '' : column));
+  return INTERNAL_COLUMNS.indexOf(s) !== -1;
+}
+
 module.exports = {
   isInternalTable: isInternalTable,
+  isInternalColumn: isInternalColumn,
+  INTERNAL_COLUMNS: INTERNAL_COLUMNS,
   ERRORS_SUFFIX: ERRORS_SUFFIX,
   STAGING_SUFFIX: STAGING_SUFFIX,
   LOADS_LEDGER: LOADS_LEDGER

package/lib/drivers/db/mssql.js

@@ -57,6 +57,7 @@ var NATIVE_TO_LOGICAL = {
 
 // Dates and times: one shared rule for every driver (dateValues.js).
 var dateValues = require('./dateValues');
+var foreignKeys = require('./foreignKeys');
 
 var INTERNAL_ID_COL       = '__xeplr_id__';
 var INTERNAL_MOVEMENT_COL = '__xeplr_movement_id__';
@@ -348,6 +349,25 @@ async function listTables(pool, schema) {
   return r.rows.map(function(row) { return row.name; });
 }
 
+// Foreign keys between this schema's tables — see foreignKeys.js.
+async function listForeignKeys(pool, schema) {
+  var r = await query(pool,
+    'SELECT fk.name AS name, t.name AS [table], rt.name AS refTable, ' +
+    'c.name AS [column], rc.name AS refColumn ' +
+    'FROM sys.foreign_keys fk ' +
+    'JOIN sys.foreign_key_columns fkc ON fkc.constraint_object_id = fk.object_id ' +
+    'JOIN sys.tables t ON t.object_id = fk.parent_object_id ' +
+    'JOIN sys.tables rt ON rt.object_id = fk.referenced_object_id ' +
+    'JOIN sys.columns c ON c.object_id = fkc.parent_object_id AND c.column_id = fkc.parent_column_id ' +
+    'JOIN sys.columns rc ON rc.object_id = fkc.referenced_object_id AND rc.column_id = fkc.referenced_column_id ' +
+    'WHERE (@p0 IS NULL OR (SCHEMA_NAME(t.schema_id) = @p0 AND SCHEMA_NAME(rt.schema_id) = @p0)) ' +
+    'ORDER BY t.name, fk.name, fkc.constraint_column_id',
+    [schema || null]
+  );
+  return foreignKeys.group(r.rows);
+}
+
+
 async function listViews(pool, schema) {
   var r = await query(pool,
     'SELECT TABLE_NAME AS name FROM INFORMATION_SCHEMA.TABLES ' +
@@ -545,6 +565,7 @@ module.exports = {
   // Introspection (used by uploader.reconcile + the db-list-* actions) + rollback
   getTableSchema:   getTableSchema,
   listTables:       listTables,
+  listForeignKeys: listForeignKeys,
   listViews:        listViews,
   listProcedures:   listProcedures,
   rollbackMovement: rollbackMovement,

package/lib/drivers/db/mysql.js

@@ -55,6 +55,7 @@ var NATIVE_TO_LOGICAL = {
 
 // Dates and times: one shared rule for every driver (dateValues.js).
 var dateValues = require('./dateValues');
+var foreignKeys = require('./foreignKeys');
 
 var INTERNAL_ID_COL       = '__xeplr_id__';
 var INTERNAL_MOVEMENT_COL = '__xeplr_movement_id__';
@@ -383,6 +384,21 @@ async function listTables(pool, schema) {
   return r.rows.map(function(row) { return row.name; });
 }
 
+// Foreign keys between this schema's tables — see foreignKeys.js.
+async function listForeignKeys(pool, schema) {
+  var r = await query(pool,
+    'SELECT CONSTRAINT_NAME AS name, TABLE_NAME AS `table`, REFERENCED_TABLE_NAME AS refTable, ' +
+    'COLUMN_NAME AS `column`, REFERENCED_COLUMN_NAME AS refColumn ' +
+    'FROM information_schema.KEY_COLUMN_USAGE ' +
+    'WHERE TABLE_SCHEMA = COALESCE(?, DATABASE()) AND REFERENCED_TABLE_NAME IS NOT NULL ' +
+    'AND REFERENCED_TABLE_SCHEMA = TABLE_SCHEMA ' +
+    'ORDER BY TABLE_NAME, CONSTRAINT_NAME, ORDINAL_POSITION',
+    [schema || null]
+  );
+  return foreignKeys.group(r.rows);
+}
+
+
 async function listViews(pool, schema) {
   var r = await query(pool,
     "SELECT TABLE_NAME AS name FROM information_schema.TABLES " +
@@ -577,6 +593,7 @@ module.exports = {
   // Introspection (used by uploader.reconcile + the db-list-* actions) + rollback
   getTableSchema:   getTableSchema,
   listTables:       listTables,
+  listForeignKeys: listForeignKeys,
   listViews:        listViews,
   listProcedures:   listProcedures,
   rollbackMovement: rollbackMovement,

package/lib/drivers/db/postgres.js

@@ -89,6 +89,7 @@ async function close(pool) {
 // time limit is statement_timeout, so Postgres enforces it even if this
 // process dies; our own timer, a little later, is only the backstop.
 var stop = require('./stop');
+var foreignKeys = require('./foreignKeys');
 var QUERY_CANCELED = '57014';
 var BACKSTOP_MS = 2000;
 
@@ -381,6 +382,25 @@ async function listTables(pool, schema) {
   return r.rows.map(function(row) { return row.table_name; });
 }
 
+// Foreign keys between this schema's tables — see foreignKeys.js.
+async function listForeignKeys(pool, schema) {
+  var r = await pool.query(
+    'SELECT c.conname AS name, t.relname AS "table", rt.relname AS "refTable", ' +
+    'a.attname AS "column", ra.attname AS "refColumn" ' +
+    'FROM pg_constraint c ' +
+    'JOIN pg_class t ON t.oid = c.conrelid JOIN pg_namespace n ON n.oid = t.relnamespace ' +
+    'JOIN pg_class rt ON rt.oid = c.confrelid JOIN pg_namespace rn ON rn.oid = rt.relnamespace ' +
+    'CROSS JOIN LATERAL unnest(c.conkey, c.confkey) WITH ORDINALITY AS k(col, refcol, ord) ' +
+    'JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = k.col ' +
+    'JOIN pg_attribute ra ON ra.attrelid = c.confrelid AND ra.attnum = k.refcol ' +
+    "WHERE c.contype = 'f' AND n.nspname = $1 AND rn.nspname = $1 " +
+    'ORDER BY t.relname, c.conname, k.ord',
+    [schema || 'public']
+  );
+  return foreignKeys.group(r.rows);
+}
+
+
 async function listViews(pool, schema) {
   var r = await pool.query(
     "SELECT table_name FROM information_schema.views " +
@@ -546,6 +566,7 @@ module.exports = {
   // Introspection (used by uploader.reconcile + the db-list-* actions)
   getTableSchema:  getTableSchema,
   listTables:      listTables,
+  listForeignKeys: listForeignKeys,
   listViews:       listViews,
   listProcedures:  listProcedures,
 

package/lib/uploader/index.js

@@ -59,6 +59,7 @@
 //
 // See project_xeplr_actions_streaming.md for the full contract.
 
+var { isInternalColumn } = require('../drivers/db/internalTables');
 var path = require('path');
 var fsp = require('fs/promises');
 
@@ -149,6 +150,12 @@ async function upload(opts) {
             var scanLimit = Math.min(sample.length, opts.firstBatchScanRows);
             declared = inferColumns(sample.slice(0, scanLimit));
           }
+          // OUR OWN COLUMNS NEVER COME ALONG. A source that was itself loaded
+          // by the uploader (an import's table, a replica) has its own
+          // __xeplr_id__ and __xeplr_movement_id__; carried here they clashed
+          // with this table's ("already exists") and would have claimed rows
+          // for a load that never wrote them. This load stamps its own.
+          declared = declared.filter(function(c) { return !isInternalColumn(c); });
           columns = await runBootstrap(driver, pool, opts.targetTable, declared, opts.primaryKeys, opts.staged);
           if (opts.staged) {
             opts.writeTable = await driver.prepareStaging(pool, opts.targetTable);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "2.2.4",
+  "version": "2.2.6",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "scripts": {