@xeplr/actions 1.1.11 → 1.1.13

package/README.md

@@ -255,6 +255,24 @@ None published. The package has no HTTP routes (a host serves `list()`, e.g. `@x
 
 The repository has `scripts/upload-file.js` (not in the npm package) — a CLI that loads a CSV or Excel file into Postgres with `file-upload`; its header documents the flags (`--file`, `--table`, `--conn-name`, `--conn`, `--host`/`--port`/`--db`/`--user`/`--password`, `--encrypt-conn`).
 
+## Read-only connections — for SQL you did not write
+
+A report preview, a user's query, anything whose SQL text came from outside your own code: connect with `readOnly: true`.
+
+```js
+const pool = await driver.connect({ ...connection, readOnly: true })
+await driver.query(pool, userSql, params)          // reads only
+```
+
+| Driver | What `readOnly` does |
+|---|---|
+| postgres | every `query` and `fetchStream` runs inside `BEGIN TRANSACTION READ ONLY`, always rolled back, over the **extended protocol** — one statement only, so `…; COMMIT; DROP TABLE t` is refused, and nothing inside can switch the transaction back to read-write |
+| mysql | every connection the pool opens is set to `TRANSACTION READ ONLY`; `multipleStatements` stays off, so one statement per query |
+| mssql | `readOnlyIntent` (routes to a readable secondary where one exists) — **not enforcement**: SQL Server runs whole batches and has no session read-only mode. Give such SQL a login with only `db_datareader` |
+| duckdb | read-only unless `access: 'rw'` |
+
+**Values are bound, never written into SQL.** On MySQL, `stringifyObjects` is always on, so an object or array passed as a value is compared as text instead of being expanded into `key = value` pairs. **Procedure parameter names** are written into the call (`@region = @p0`, `region => $1`) and must be plain names — letters, digits and `_`.
+
 ## Rules the code enforces, and why
 
 | rule | why |
@@ -275,6 +293,8 @@ The repository has `scripts/upload-file.js` (not in the npm package) — a CLI t
 | **Rollback is refused for a movement that used `primaryKeys`.** | Upsert stamps the movement id on rows it only updated, so deleting by it would remove rows that already existed. |
 | Movement logs carry identifiers, counts, SQL and the error text — never parameter values or row contents; a failed batch shows one row's primary key, not its data. | Logs are copied into tickets and chats. |
 | DuckDB opens **read-only unless the call says `access: 'rw'`**; no environment variable can grant write. A file already open read-only in the process refuses a write. Opening waits on the lock (`openTimeoutMs`, default 60 s). | One read-write process locks every other process out, even readers. Write access belongs at the call site that writes. |
+| **Every DuckDB value comes out plain**, from `query` and `fetchStream` alike: a number, text, boolean, `Date`, array or object, never a BigInt or a DuckDB wrapper object. DECIMAL is a number; DATE and TIMESTAMP are `Date`s; a whole number too big to hold exactly (beyond ±9,007,199,254,740,991) is **text**, never rounded; lists, structs and maps are arrays and objects; TIME, INTERVAL, UUID and any other type are text. `fetchStream` has DuckDB do the conversion inside its engine by wrapping the query (about +3% on a million rows); a query with nothing to convert runs as given. | A streamed read used to hand DECIMAL and TIMESTAMP back as wrapper objects with a BigInt inside, and the first `JSON.stringify` downstream failed with "Do not know how to serialize a BigInt". Text for huge numbers is what the Postgres, MySQL and SQL Server drivers already return. |
+| **A DuckDB timestamp without a zone is read as UTC**; one with a zone keeps its exact instant. | Data is kept in UTC; the viewer's time zone is applied when rendering, never in the driver. |
 | `meta-store-knex` requires `service` and `applicationId`, and scopes every statement by `applicationId`. | `import_meta` is shared by every app; unattributed rows, or a rollback decided from another app's row, are worse than refusing. |
 | Mail filenames are sanitised (`[^\w.-]` → `_`, all-dot names → `attachment`, max 180 chars) and prefixed with a random id. | Attachment names come from the sender and could escape the output directory; common names collide. |
 | The system mail sender refuses `text`, `bcc`, `from`, `replyTo` rather than dropping them. | A silently lost bcc is a compliance problem found a year later. |
@@ -291,7 +311,7 @@ node --test test/uploader/reconcile.test.js
 | needs nothing | needs a database |
 |---|---|
 | `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
-| `test/drivers/duckdb.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
+| `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
 | | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
 | | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |
 

package/lib/drivers/db/duckdb.js

@@ -299,11 +299,27 @@ async function close(pool) {
 
 // BigInt is what DuckDB hands back for BIGINT/HUGEINT, and it poisons anything
 // downstream that does arithmetic or JSON.stringify on a result. Narrowed to a
-// number where that is lossless, left alone where it is not — silently losing
-// precision on a genuinely huge value would be worse than an awkward type.
+// number where that is lossless, and to TEXT where it is not: silently losing
+// digits on a genuinely huge value would be wrong, and a BigInt left in a row
+// crashes the first JSON.stringify that meets it. Text is also what Postgres,
+// MySQL and SQL Server already return for such a value, so every source
+// behaves alike.
 function unBig(v) {
   if (typeof v !== 'bigint') return v;
-  return (v <= 9007199254740991n && v >= -9007199254740991n) ? Number(v) : v;
+  return (v <= 9007199254740991n && v >= -9007199254740991n) ? Number(v) : v.toString();
+}
+
+// The same, all the way down: a BigInt can sit INSIDE a value the normal read
+// returns (an INTERVAL's micros, a list of BIGINTs). Arrays and plain objects
+// keep their shape; a Date, a string or a number is left as it is.
+function plainDeep(v) {
+  if (typeof v === 'bigint') return unBig(v);
+  if (v === null || typeof v !== 'object' || v instanceof Date) return v;
+  if (Array.isArray(v)) return v.map(plainDeep);
+  if (Object.getPrototypeOf(v) !== Object.prototype) return v;
+  var out = {};
+  for (var k in v) out[k] = plainDeep(v[k]);
+  return out;
 }
 
 /**
@@ -337,7 +353,7 @@ async function query(pool, sql, params) {
     var columns = reader.columnNames();
     var rows = reader.getRowObjectsJS().map(function(row) {
       var out = {};
-      for (var k in row) out[k] = unBig(row[k]);
+      for (var k in row) out[k] = plainDeep(row[k]);
       return out;
     });
     return { rows: rows, rowCount: rows.length, columns: columns };
@@ -551,6 +567,108 @@ async function listProcedures(_pool, _schema) {
   return [];
 }
 
+// ── plain values out of a stream ─────────────────────────────────────────
+//
+// A streamed chunk (getRowObjects) hands DECIMAL, TIMESTAMP, INTERVAL, UUID
+// and the nested types back as DuckDB WRAPPER objects, several with a BigInt
+// inside. They are not the values the rest of the code expects, and the first
+// JSON.stringify that meets one fails ("Do not know how to serialize a
+// BigInt") — which is how a report grouped by an amount or a timestamp from a
+// cube used to die.
+//
+// Converting each wrapper in JavaScript doubles the cost of a streamed read.
+// So DuckDB converts them inside its own engine instead, by wrapping the
+// query: measured on 1M rows, +2% (+6% with the timestamp turned into a Date
+// here), against +100% for the JS converter.
+//
+//   passes through   booleans, integers (BigInt narrowed by unBig), floats,
+//                    text, enums
+//   DECIMAL          DOUBLE: what the non-streamed read (getRowObjectsJS)
+//                    returns too
+//   DATE, TIMESTAMP* epoch milliseconds, turned into a Date as the row is
+//                    built — the same type Postgres and the normal read give.
+//                    A timestamp WITHOUT a zone is read as UTC (data is kept
+//                    in UTC; the viewer's zone is applied when rendering). One
+//                    WITH a zone keeps its exact instant: epoch_ms is taken on
+//                    it directly, never after a cast that would apply the
+//                    session's zone.
+//   LIST, STRUCT,    JSON, parsed back here: arrays and objects, the shape the
+//   MAP, ARRAY       normal read gives
+//   anything else    VARCHAR (TIME, INTERVAL, UUID, …). A type nobody has met
+//                    yet becomes text rather than a crash.
+//
+// When no column needs converting, the query runs exactly as given.
+var PASS_THROUGH_TYPES = null;
+var DATE_TYPES = null;
+var NESTED_TYPES = null;
+function typeSets() {
+  if (PASS_THROUGH_TYPES) return;
+  var T = require('@duckdb/node-api').DuckDBTypeId;
+  PASS_THROUGH_TYPES = new Set([T.BOOLEAN, T.TINYINT, T.SMALLINT, T.INTEGER, T.BIGINT, T.HUGEINT,
+    T.UTINYINT, T.USMALLINT, T.UINTEGER, T.UBIGINT, T.UHUGEINT, T.FLOAT, T.DOUBLE,
+    T.VARCHAR, T.ENUM, T.SQLNULL].filter(function(x) { return x !== undefined; }));
+  DATE_TYPES = {};
+  DATE_TYPES[T.DATE] = 'plain';
+  DATE_TYPES[T.TIMESTAMP] = 'plain';
+  DATE_TYPES[T.TIMESTAMP_S] = 'plain';
+  DATE_TYPES[T.TIMESTAMP_MS] = 'plain';
+  DATE_TYPES[T.TIMESTAMP_NS] = 'plain';
+  DATE_TYPES[T.TIMESTAMP_TZ] = 'zoned';
+  NESTED_TYPES = new Set([T.LIST, T.STRUCT, T.MAP, T.ARRAY, T.UNION].filter(function(x) { return x !== undefined; }));
+}
+
+function stripTrailingSemicolon(sql) {
+  return String(sql).replace(/[\s;]+$/, '');
+}
+
+async function plainValuesPlan(conn, sql) {
+  typeSets();
+  var prepared = await conn.prepare(sql);
+  var count = prepared.columnCount;
+  var select = [];
+  var inner = [];
+  var dateColumns = [];
+  var jsonColumns = [];
+  var changed = false;
+  for (var i = 0; i < count; i++) {
+    var name = prepared.columnName(i);
+    var typeId = prepared.columnTypeId(i);
+    var col = '"__c' + i + '"';
+    inner.push(col);
+    var expr;
+    if (PASS_THROUGH_TYPES.has(typeId)) {
+      expr = col;
+    } else if (DATE_TYPES[typeId] === 'zoned') {
+      expr = 'epoch_ms(' + col + ')';
+      dateColumns.push(name);
+      changed = true;
+    } else if (DATE_TYPES[typeId] === 'plain') {
+      expr = 'epoch_ms(CAST(' + col + ' AS TIMESTAMP))';
+      dateColumns.push(name);
+      changed = true;
+    } else if (typeId === require('@duckdb/node-api').DuckDBTypeId.DECIMAL) {
+      expr = 'CAST(' + col + ' AS DOUBLE)';
+      changed = true;
+    } else if (NESTED_TYPES.has(typeId)) {
+      expr = 'CAST(to_json(' + col + ') AS VARCHAR)';
+      jsonColumns.push(name);
+      changed = true;
+    } else {
+      expr = 'CAST(' + col + ' AS VARCHAR)';
+      changed = true;
+    }
+    select.push(expr + ' AS ' + quoteIdent(name));
+  }
+  if (!changed) return { sql: sql, dateColumns: [], jsonColumns: [] };
+  // Positional renames (__c0, __c1, …) so two source columns with one name
+  // stay two columns, whatever they are called.
+  return {
+    sql: 'SELECT ' + select.join(', ') + ' FROM (' + stripTrailingSemicolon(sql) + '\n) AS "__plain"(' + inner.join(', ') + ')',
+    dateColumns: dateColumns,
+    jsonColumns: jsonColumns
+  };
+}
+
 // ── streaming read ───────────────────────────────────────────────────────
 
 /**
@@ -565,8 +683,11 @@ async function* fetchStream(pool, opts) {
   var entry = entryOf(pool);
   var conn = await acquire(entry);
   try {
-    var result = await conn.stream(opts.sql, (opts.params && opts.params.length) ? opts.params : undefined);
+    var plan = await plainValuesPlan(conn, opts.sql);
+    var result = await conn.stream(plan.sql, (opts.params && opts.params.length) ? opts.params : undefined);
     var names = result.columnNames();
+    var dates = plan.dateColumns;
+    var jsons = plan.jsonColumns;
     while (true) {
       var chunk = await result.fetchChunk();
       if (!chunk || chunk.rowCount === 0) break;
@@ -575,6 +696,14 @@ async function* fetchStream(pool, opts) {
         var row = rows[i];
         var out = {};
         for (var k in row) out[k] = unBig(row[k]);
+        for (var d = 0; d < dates.length; d++) {
+          var v = out[dates[d]];
+          if (v !== null && v !== undefined) out[dates[d]] = new Date(v);
+        }
+        for (var j = 0; j < jsons.length; j++) {
+          var jv = out[jsons[j]];
+          if (typeof jv === 'string') out[jsons[j]] = JSON.parse(jv);
+        }
         yield out;
       }
     }

package/lib/drivers/db/mssql.js

@@ -87,6 +87,11 @@ async function connect(config) {
     pool:     { max: config.maxConnections || 10 },
     options:  Object.assign(
       { encrypt: true, trustServerCertificate: true },
+      // readOnly: ApplicationIntent=ReadOnly — routed to a readable secondary
+      // where there is one. It is NOT enforcement: SQL Server runs a whole
+      // batch and has no session-level read-only mode, so SQL the caller did
+      // not write needs a login that can only read (db_datareader). See README.
+      config.readOnly ? { readOnlyIntent: true } : {},
       config.options || {}
     )
   });

package/lib/drivers/db/mysql.js

@@ -89,8 +89,21 @@ async function connect(config) {
     timezone:          'Z',
     dateStrings:       true,
     supportBigNumbers: true,
-    bigNumberStrings:  true
+    bigNumberStrings:  true,
+    // A VALUE IS A VALUE. Without this mysql2 expands an object bound to `?`
+    // into `key` = value pairs and nested arrays into lists — so a filter value
+    // of {"a":1} rewrites the predicate instead of being compared to it.
+    stringifyObjects:  true
+    // multipleStatements stays at its default (off): one statement per query.
   });
+  // READ-ONLY CONNECTIONS — `connect({ ..., readOnly: true })` for SQL the
+  // caller did not write itself. Every connection the pool opens is put into
+  // read-only transaction mode before it is used, so a write fails.
+  if (config.readOnly) {
+    pool.pool.on('connection', function(conn) {
+      conn.query('SET SESSION TRANSACTION READ ONLY');
+    });
+  }
   return pool;
 }
 

package/lib/drivers/db/postgres.js

@@ -41,6 +41,17 @@ var TYPE_MAP = {
 var INTERNAL_ID_COL       = '__xeplr_id__';
 var INTERNAL_MOVEMENT_COL = '__xeplr_movement_id__';
 
+// READ-ONLY CONNECTIONS — for SQL the caller did not write itself (a report
+// preview, a user's query). `connect({ ..., readOnly: true })` makes every
+// query and fetchStream on that pool:
+//
+//   • run inside BEGIN TRANSACTION READ ONLY, always rolled back — a write
+//     fails with "cannot execute … in a read-only transaction", and nothing
+//     inside the transaction can switch it back to read-write
+//   • go over the EXTENDED protocol, which carries exactly one statement — so
+//     `…; COMMIT; DROP TABLE t` is refused rather than run after the SELECT
+//     (the simple protocol, which pg uses when there are no parameters, runs
+//     every statement in the string)
 async function connect(config) {
   var pg = require('pg');
   var pool = new pg.Pool({
@@ -51,6 +62,7 @@ async function connect(config) {
     database: config.database,
     max:      config.maxConnections || 10
   });
+  if (config.readOnly) pool.__xeplrReadOnly = true;
   return pool;
 }
 
@@ -67,7 +79,19 @@ async function close(pool) {
 // this as the canonical shape the other drivers normalize TO, so replacing
 // the object risks dropping a property something else reads.
 async function query(pool, sql, params) {
-  var result = await pool.query(sql, params || []);
+  var result;
+  if (pool.__xeplrReadOnly) {
+    var client = await pool.connect();
+    try {
+      await client.query('BEGIN TRANSACTION READ ONLY');
+      result = await client.query({ text: sql, values: params || [], queryMode: 'extended' });
+    } finally {
+      try { await client.query('ROLLBACK'); } catch (_) { /* the connection is released either way */ }
+      client.release();
+    }
+  } else {
+    result = await pool.query(sql, params || []);
+  }
   result.columns = (result.fields || []).map(function(f) { return f.name; });
   return result;
 }
@@ -323,8 +347,13 @@ async function* fetchStream(pool, opts) {
   var client = await pool.connect();
   var declared = false;
   try {
-    await client.query('BEGIN');
-    await client.query('DECLARE __xeplr_fetch_cur NO SCROLL CURSOR FOR ' + opts.sql, opts.params || []);
+    var readOnly = Boolean(pool.__xeplrReadOnly);
+    await client.query(readOnly ? 'BEGIN TRANSACTION READ ONLY' : 'BEGIN');
+    // Extended protocol when read-only: one statement, so the caller's SQL
+    // cannot close the cursor declaration and carry on with its own.
+    await client.query(readOnly
+      ? { text: 'DECLARE __xeplr_fetch_cur NO SCROLL CURSOR FOR ' + opts.sql, values: opts.params || [], queryMode: 'extended' }
+      : 'DECLARE __xeplr_fetch_cur NO SCROLL CURSOR FOR ' + opts.sql, readOnly ? undefined : (opts.params || []));
     declared = true;
     while (true) {
       var res = await client.query('FETCH FORWARD ' + batchSize + ' FROM __xeplr_fetch_cur');
@@ -333,7 +362,7 @@ async function* fetchStream(pool, opts) {
     }
   } finally {
     try { if (declared) await client.query('CLOSE __xeplr_fetch_cur'); } catch (_) {}
-    try { await client.query('COMMIT'); } catch (_) {}
+    try { await client.query(pool.__xeplrReadOnly ? 'ROLLBACK' : 'COMMIT'); } catch (_) {}
     client.release();
   }
 }

package/lib/drivers/db/procedure.js

@@ -183,6 +183,12 @@ function buildProcedureCall(opts) {
   var marks = entries.map(function(p, i) {
     var mark = ph(i);
     if (!p.name || !supportsNames) return mark;
+    // Written into the call as text (`@region = @p0`, `region => $1`), so a
+    // plain name only — a saved or requested parameter name must not be able
+    // to close the call and start another statement.
+    if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(String(p.name))) {
+      throw new Error('procedure parameter name "' + String(p.name).slice(0, 40) + '" — use letters, digits and _ only');
+    }
     if (dbType === 'mssql') return '@' + p.name + ' = ' + mark;
     return p.name + ' => ' + mark;              // postgres named-argument call
   }).join(', ');

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xeplr/actions",
-  "version": "1.1.11",
+  "version": "1.1.13",
   "description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
   "main": "index.js",
   "files": [