@vida-global/core 2.3.2 → 2.3.4

package/AGENTS.md

@@ -2,12 +2,53 @@
 
 Use this guide to understand and write code for the vida-core repo.
 
+`@vida-global/core` is the shared library behind all Vida apps. It also ships to
+customers inside `vida-apps-tools`, so the exported surface is a public API.
+Treat a changed or removed export as a breaking change, not a refactor.
+
+CommonJS throughout (`require` / `module.exports`). There is no linter in this
+repo — the style guide below is the only enforcement, so follow it closely.
+
+
 # Style and conventions
+
+Always in force:
+
 @agents/style.md
-@agents/server.md
-@agents/db.md
-@agents/apis.md
 @agents/testing.md
 
+Read only when the work touches that area:
+
+- `agents/server.md` — Express endpoints, controllers, validation, status codes
+- `agents/db.md` — ActiveRecord, Sequelize, migrations, caching
+- `agents/apis.md` — outbound HTTP clients to third parties
+- `agents/git.md` — branches, commits, GitHub
+- `agents/frontend.md` — Next.js/React only. Not used by this package.
+
+
+# Commands
+
+    npm test                     # jest, coverage on by default
+    npm test -- test/server      # narrow to a directory
+    npm test -- -t 'name'        # narrow by test name
+    npm run develop              # prerelease via @vida-global/release
+    npm run release              # release via @vida-global/release
+
+`npm test` carries required flags (`LOG_LEVEL=test`, `NODE_NO_WARNINGS=1`,
+`node --experimental-vm-modules`). Always go through the script. Calling `jest`
+directly drops them and the run misbehaves.
+
+Coverage is collected on every run. `/helpers/` is excluded by design.
+
+
+# Repository map
+
+`index.js` is the whole public surface. `lib/` holds the modules; `test/` mirrors
+`lib/` one directory per module. Per-module guides live next to the code and are
+listed in `README.md`: `activeRecord`, `http`, `logger`, `server`, `jobQueue`,
+plus `cache`, `redis`, `apm`, `observability`, `utils`.
 
-# Helpers for common use cases
+One non-obvious mechanic in `index.js`: server error classes are exported by
+filtering `lib/server` exports on `prototype instanceof AbstractServerError`. A
+new error class that extends it is exported automatically — do not add it to
+`module.exports` by hand. One that does not extend it is silently absent.

package/lib/activeRecord/README.md

@@ -249,7 +249,7 @@ user.fullName; // "Ada Lovelace"
 Opt a model in to Redis-backed caching by overriding `isCacheable`. With caching on:
 
 - `Model.find(id)` and `Model.find([id1, id2])` consult Redis first and fall back to the database for misses, repopulating the cache for next time.
-- `afterSave` and `afterDestroy` hooks automatically invalidate the cached record.
+- An `afterCommit` hook automatically invalidates the cached record, so a write that is rolled back never disturbs the cache.
 - Cache keys are namespaced as `AR:<ClassName>` (`<ClassName>#find:<id>`).
 
 ```js
@@ -294,7 +294,29 @@ User.addHook('afterSave', async (user, options) => {
 });
 ```
 
-See [Sequelize hooks](https://sequelize.org/docs/v6/other-topics/hooks/) for the full list of hook points. The caching layer above uses `afterSave` and `afterDestroy` under the hood when `isCacheable` is true.
+See [Sequelize hooks](https://sequelize.org/docs/v6/other-topics/hooks/) for the full list of hook points.
+
+
+### afterCommit
+
+Every hook above runs *inside* the transaction the write belongs to, so its work is undone if that
+transaction rolls back. Use `afterCommit` for anything that must not happen until the data is
+durable — cache invalidation, broadcasts, enqueuing jobs:
+
+```js
+User.addHook('afterCommit', async (user, options) => {
+    await broadcast('users.changed', { id: user.id });   // never fires if the transaction rolls back
+});
+```
+
+- Fires once per record written, after the transaction commits. A write outside a transaction fires
+  it immediately.
+- Fires on create, update, destroy, restore and upsert. Bulk statements such as
+  `Model.destroy({ where })` only fire it when `individualHooks: true` is set, the same as
+  `afterDestroy`.
+- With nested transactions, only the outermost commit fires it. A savepoint that commits inside a
+  transaction that later rolls back fires nothing.
+- A record written more than once in the same transaction still fires each callback once.
 
 
 ## Associations

package/lib/activeRecord/afterCommitHooks.js

@@ -0,0 +1,167 @@
+// Sequelize has no model level afterCommit hook, only Transaction#afterCommit. This module holds
+// the callbacks registered for one and defers them onto the transaction a write belongs to, so
+// BaseRecord can offer afterCommit alongside the hooks sequelize does provide.
+
+// Keyed by model class so callbacks stay scoped to the model they were registered on. Module scope
+// is what keeps them unreachable from outside: a static # field is unreadable through a subclass,
+// and a plain static field on BaseRecord would be one collection shared by every model.
+const modelCallbacks       = new WeakMap();
+const claimedRegistrations = new WeakMap();
+
+
+/***************************************************************************************************
+* REGISTRATION
+***************************************************************************************************/
+function addCallback(model, name, fn) {
+    const callback = buildCallback(name, fn);
+    ownCallbacks(model).push(callback);
+}
+
+
+// Sequelize's addHook takes either (name, fn) or (fn), and stores the named form as an object so it
+// can be removed by name later. Match both.
+function buildCallback(name, fn) {
+    if (typeof name === 'function') return name;
+
+    return { name, fn };
+}
+
+
+function removeCallback(model, name) {
+    const remaining = ownCallbacks(model).filter(callback => !callbackMatches(callback, name));
+    modelCallbacks.set(model, remaining);
+}
+
+
+// Only named callbacks are removable by name. An unnamed one carries its function name, which would
+// otherwise match a string by accident.
+function callbackMatches(callback, name) {
+    if (typeof name === 'function')  return callback === name;
+    if (typeof callback === 'object') return callback.name === name;
+
+    return false;
+}
+
+
+function ownCallbacks(model) {
+    if (!modelCallbacks.has(model)) modelCallbacks.set(model, []);
+
+    return modelCallbacks.get(model);
+}
+
+
+/***************************************************************************************************
+* LOOKUP
+***************************************************************************************************/
+// Ancestors first, so callbacks run in the order they were registered as you read down a hierarchy.
+function callbacksFor(model) {
+    const callbacks = [];
+    for (const ancestor of modelChain(model)) {
+        callbacks.unshift(...inheritableCallbacks(ancestor));
+    }
+    return callbacks;
+}
+
+
+function inheritableCallbacks(model) {
+    const callbacks = modelCallbacks.get(model) || [];
+    return callbacks.map(callback => callbackFunction(callback));
+}
+
+
+function callbackFunction(callback) {
+    if (typeof callback === 'function') return callback;
+
+    return callback.fn;
+}
+
+
+function* modelChain(model) {
+    let ancestor = model;
+    while (typeof ancestor === 'function') {
+        yield ancestor;
+        ancestor = Object.getPrototypeOf(ancestor);
+    }
+}
+
+
+/***************************************************************************************************
+* DISPATCH
+***************************************************************************************************/
+async function runCallbacks(model, record, options) {
+    const transaction = rootTransaction(options.transaction);
+
+    if (!transaction) return await callImmediately(model, record, options);
+
+    deferToTransaction(transaction, model, record, options);
+}
+
+
+// A savepoint commits while its parent can still roll back, so the callbacks belong to the
+// outermost transaction.
+function rootTransaction(transaction) {
+    let root = transaction;
+    while (root?.parent) {
+        root = root.parent;
+    }
+    return root;
+}
+
+
+async function callImmediately(model, record, options) {
+    for (const callback of callbacksFor(model)) {
+        await callback.call(model, record, options);
+    }
+}
+
+
+function deferToTransaction(transaction, model, record, options) {
+    for (const callback of callbacksFor(model)) {
+        deferCallback(transaction, callback, model, record, options);
+    }
+}
+
+
+function deferCallback(transaction, callback, model, record, options) {
+    if (!claimRegistration(transaction, callback, record)) return;
+
+    transaction.afterCommit(async () => await callback.call(model, record, options));
+}
+
+
+/***************************************************************************************************
+* DEDUPLICATION
+***************************************************************************************************/
+// Overlapping hooks can report the same write twice. Each callback runs once per record per
+// transaction; returns false when this one is already spoken for.
+function claimRegistration(transaction, callback, record) {
+    const records = claimedRecords(transaction, callback);
+    if (records.has(record)) return false;
+
+    records.add(record);
+    return true;
+}
+
+
+// A WeakSet so claiming a record doesn't hold it in memory for the life of the transaction.
+function claimedRecords(transaction, callback) {
+    const callbacks = transactionClaims(transaction);
+    if (!callbacks.has(callback)) callbacks.set(callback, new WeakSet());
+
+    return callbacks.get(callback);
+}
+
+
+function transactionClaims(transaction) {
+    if (!claimedRegistrations.has(transaction)) claimedRegistrations.set(transaction, new Map());
+
+    return claimedRegistrations.get(transaction);
+}
+
+
+module.exports = {
+    addCallback,
+    callbacksFor,
+    removeCallback,
+    runCallbacks,
+}

package/lib/activeRecord/baseRecord.js

@@ -6,10 +6,14 @@ const { Model, Op, Sequelize }            = require('sequelize');
 const   nodeUtil                          = require('util');
 const { redisClientFactory }              = require('../redis');
 const { camelize, tableize }              = require('inflection');
+const   afterCommitHooks                  = require('./afterCommitHooks');
 const   utils                             = require('./utils');
 const   validations                       = require('./validations');
 
 
+const AFTER_COMMIT = 'afterCommit';
+
+
 class BaseRecord extends Model {
     static _connection;
     static _initialized = false;
@@ -197,14 +201,76 @@ class BaseRecord extends Model {
     }
 
 
+    /***********************************************************************************************
+    * HOOKS
+    ***********************************************************************************************/
     static initializeHooks() {
+        this._initializeAfterCommitHooks();
+
         if (this.isCacheable) {
-            this.addHook('afterSave',    this._afterSaveCacheHook);
-            this.addHook('afterDestroy', this._afterDestroyCacheHook);
+            this.addHook(AFTER_COMMIT, this._afterCommitCacheHook);
+        }
+    }
+
+
+    static _initializeAfterCommitHooks() {
+        // Every hook that reports a write to a single record. afterSave is deliberately the only save hook
+        // listed: sequelize proxies it into afterCreate and afterUpdate, so adding those would double fire.
+        const AFTER_COMMIT_SOURCE_HOOKS = ['afterSave', 'afterDestroy', 'afterRestore', 'afterUpsert'];
+        for (const hookType of AFTER_COMMIT_SOURCE_HOOKS) {
+            this.addHook(hookType, this._afterCommitHandler);
         }
     }
 
 
+    static addHook(hookType, name, fn) {
+        if (hookType != AFTER_COMMIT) return super.addHook(hookType, name, fn);
+
+        afterCommitHooks.addCallback(this, name, fn);
+        return this;
+    }
+
+
+    static removeHook(hookType, name) {
+        if (hookType != AFTER_COMMIT) return super.removeHook(hookType, name);
+
+        afterCommitHooks.removeCallback(this, name);
+        return this;
+    }
+
+
+    static hasHook(hookType) {
+        if (hookType != AFTER_COMMIT) return super.hasHook(hookType);
+
+        return Boolean(afterCommitHooks.callbacksFor(this).length);
+    }
+
+
+    // Sequelize aliases hasHooks onto its own hasHook, so overriding hasHook alone leaves it behind.
+    static hasHooks(hookType) {
+        return this.hasHook(hookType);
+    }
+
+
+    // Sequelize generates a shorthand registration method for each of its hook types. Match it.
+    static afterCommit(name, fn) {
+        return this.addHook(AFTER_COMMIT, name, fn);
+    }
+
+
+    static async _afterCommitHandler(recordOrResult, options) {
+        const record = this._afterCommitRecord(recordOrResult);
+        await afterCommitHooks.runCallbacks(this, record, options);
+    }
+
+
+    static _afterCommitRecord(recordOrResult) {
+        if (Array.isArray(recordOrResult)) return recordOrResult[0]; // afterUpsert: [record, created]
+
+        return recordOrResult;
+    }
+
+
     /***********************************************************************************************
     * VALIDATIONS
     ***********************************************************************************************/
@@ -350,7 +416,7 @@ class BaseRecord extends Model {
         const client       = await this._getRedisClient();
         const cachedValues = clear ? [] : await client.mGet(keys);
         const kvPairs      = keys.map((key,idx) => {
-            const val = this._unmarshallCachedData(cachedValues[idx]);
+            const val = this._decodeCachedValue(cachedValues[idx]);
             return [key, val]
         });
         const cachedData   = Object.fromEntries(kvPairs);
@@ -374,7 +440,7 @@ class BaseRecord extends Model {
         const toCache = {};
 
         for (const [key, value] of Object.entries(fetchedData)) {
-            toCache[key] = this._marshallDataForCaching(value);
+            toCache[key] = await this._encodeForCache(value);
         }
 
         const client = await this._getRedisClient();
@@ -448,42 +514,41 @@ class BaseRecord extends Model {
     static get isCacheable() { return false; }
 
 
-    static async _afterSaveCacheHook(record, options) {
+    static async _afterCommitCacheHook(record, options) {
         await record.clearSelfCache();
         await record.updateCache(options);
     }
 
 
-    static async _afterDestroyCacheHook(record, options) {
-        await record.clearSelfCache();
-        await record.updateCache(options);
+    // The data a record contributes to its cache entry. Override to cache associated data
+    // alongside the record so a single cache read returns both.
+    async toCachePayload() {
+        return this.toJSON();
     }
 
 
-    static _marshallDataForCaching(value) {
-      let marshalledValue;
-      if (value instanceof this) {
-          marshalledValue = JSON.stringify(value.toJSON());
-          marshalledValue = `${this.recordCachingPrefix}:${marshalledValue}`;
-      } else {
-          marshalledValue = JSON.stringify(value);
-      }
-      return marshalledValue;
+    // The inverse of toCachePayload: rebuilds a record from its cached data.
+    static fromCachePayload(data) {
+        return new this(data, {isNewRecord: false});
     }
 
 
-    static _unmarshallCachedData(value) {
+    static async _encodeForCache(value) {
+        if (!(value instanceof this)) return JSON.stringify(value);
+
+        const payload = await value.toCachePayload();
+        return `${this.recordCachingPrefix}:${JSON.stringify(payload)}`;
+    }
+
+
+    static _decodeCachedValue(value) {
         if (value == null)            return null;
         if (typeof value != 'string') return value;
+        if (!value.startsWith(this.recordCachingPrefix)) return JSON.parse(value);
 
-        if (value.startsWith(this.recordCachingPrefix)) {
-            const regExp = new RegExp(`^${this.recordCachingPrefix}:`);
-            value        = value.replace(regExp, '');
-            const data   = JSON.parse(value);
-            return new this(data, {isNewRecord: false});
-        } else {
-            return JSON.parse(value);
-        }
+        const regExp = new RegExp(`^${this.recordCachingPrefix}:`);
+        const data   = JSON.parse(value.replace(regExp, ''));
+        return this.fromCachePayload(data);
     }
 
 

package/lib/activeRecord/db/migrator.js

@@ -119,8 +119,11 @@ class Migrator {
 
 
     idColumn(idType=DEFAULT_ID_TYPE) {
+        if (idType === false) return {};
+
         if (!ID_TYPES.includes(idType)) {
-            throw new Error(`Unknown id type "${idType}". Use one of: ${ID_TYPES.join(', ')}`);
+            throw new Error(`Unknown id type "${idType}". Use one of: ${ID_TYPES.join(', ')}, or `
+                          + `false for a table that declares its own primary key.`);
         }
 
         return this[`${idType}IdColumn`];

package/lib/server/controllerMixins/renderer.js

@@ -9,6 +9,13 @@ class StreamInProgressError extends Error {}
 class NoActiveStreamError   extends Error {}
 
 
+// xmlbuilder2's default object-notation control prefixes: @attribute, #text, $cdata,
+// !comment, ?processing-instruction. A key using one of these must survive sanitization
+// unmangled, or an intended attribute (e.g. {'@track': 'both'}) silently turns into a
+// child element (`<_track>both</_track>`) instead of `track="both"`.
+const XML_CONVERT_PREFIXES = ['@', '#', '$', '!', '?'];
+
+
 const CONTENT_TYPE_BY_EXTENSION = {
     css:   'text/css; charset=utf-8',
     gif:   'image/gif',
@@ -88,12 +95,17 @@ const InstanceMethods = {
         if (!this.responseContentType) this.responseContentType = CONTENT_TYPE_BY_EXTENSION.xml;
         const doc = create({ [this.xmlRootElement]: body });
         const xml = doc.end();
-        this._response.send(xml);
+        this._send(xml);
     },
 
 
     renderTextResponse(body) {
         if (!this.responseContentType) this.responseContentType = CONTENT_TYPE_BY_EXTENSION.txt;
+        this._send(body);
+    },
+
+
+    _send(body) {
         this._response.send(body);
     },
 
@@ -182,8 +194,23 @@ const InstanceMethods = {
 
     // XML element names can't start with a digit or contain spaces/other invalid characters,
     // so sanitize keys that would otherwise make xmlbuilder2 throw on dynamic response data.
+    // A leading xmlbuilder2 control prefix (see XML_CONVERT_PREFIXES) is preserved as-is;
+    // only the remainder (the attribute/PI name, if any) is sanitized.
     _toXMLElementName(key) {
-        let name = `${key}`.replace(/[^a-zA-Z0-9_.-]/g, '_');
+        key = `${key}`;
+
+        const prefix = XML_CONVERT_PREFIXES.find(p => key.startsWith(p));
+        if (prefix) {
+            const rest = key.slice(prefix.length);
+            return rest ? prefix + this._sanitizeXMLNamePart(rest) : prefix;
+        }
+
+        return this._sanitizeXMLNamePart(key);
+    },
+
+
+    _sanitizeXMLNamePart(name) {
+        name = name.replace(/[^a-zA-Z0-9_.-]/g, '_');
         if (!/^[a-zA-Z_]/.test(name)) name = `_${name}`;
         return name;
     },

package/lib/server/controllerMixins/requestDetails.js

@@ -47,8 +47,10 @@ const Accessors = {
 
     contentType:     { get() { return this.requestHeaders['content-type']          }},
     cookies:         { get() { return this._request.cookies || {}                  }},
+    isProxiedRequest:{ get() { return Boolean(this.requestHeaders['x-forwarded-for']) }},
     requestBody:     { get() { return this._request.body                           }},
     requestHeaders:  { get() { return structuredClone(this._request.headers || {}) }},
+    requestHost:     { get() { return this.requestHeaders.host                     }},
     requestId:       { get() { return this._request.id                             }},
     requestIp:       { get() { return this._request.ip                             }},
     requestMethod:   { get() { return this._request.method                         }},
@@ -73,6 +75,36 @@ const Accessors = {
             return match[1].trim();
         }
     },
+
+
+    // The absolute URL the caller addressed, rebuilt from the request. Webhook signature schemes
+    // sign this string, so it has to match what the sender used character for character.
+    requestUrl: {
+        get() { return `${this.requestProtocol}://${this.requestHost}${this.url}` }
+    },
+
+
+    // TLS usually terminates at a load balancer, and not every balancer forwards
+    // X-Forwarded-Proto, so `_request.protocol` can report http on a request the caller made over
+    // https. Prefer the forwarded protocol, assume https for anything else that arrived through a
+    // proxy, and fall back to the connection's own protocol for direct requests.
+    requestProtocol: {
+        get() {
+            if (this.forwardedProtocol) return this.forwardedProtocol;
+            if (this.isProxiedRequest)  return 'https';
+            return this._request.protocol;
+        }
+    },
+
+
+    // A comma-joined chain lists the original client's protocol first.
+    forwardedProtocol: {
+        get() {
+            const header = this.requestHeaders['x-forwarded-proto'];
+            if (!header) return null;
+            return header.split(',')[0].trim();
+        }
+    },
 }
 
 

package/lib/server/controllerMixins/validations.js

@@ -70,7 +70,7 @@ const InstanceMethods = {
 
     validateIsString(value, options) {
         if (typeof value !== 'string') return 'must be a string';
-        if (options?.allowEmpty === false && value.length === 0) return 'must not be empty';
+        if (options?.allowEmpty === false && value.trim().length === 0) return 'must not be empty';
 
         if (options?.length?.gte !== undefined) {
             if (value.length < options.length.gte) return `must be greater than or equal to ${options.length.gte} characters`;

package/lib/server/doc/requestDetails.md

@@ -19,6 +19,11 @@ Within an action, the controller exposes:
 | `this.requestMethod` | HTTP method. |
 | `this.requestIp` | Client IP address. |
 | `this.url` | Original URL. |
+| `this.requestHost` | Shortcut for `requestHeaders['host']`. |
+| `this.requestUrl` | Absolute URL the caller addressed: `requestProtocol://requestHost + url`. Use this when verifying a webhook signature that covers the URL. |
+| `this.requestProtocol` | `forwardedProtocol`, else `https` when `isProxiedRequest`, else the connection's own protocol. |
+| `this.forwardedProtocol` | First entry of `X-Forwarded-Proto`, or `null`. |
+| `this.isProxiedRequest` | `true` when `X-Forwarded-For` is present. |
 | `this.bearerToken` | Parsed `Authorization: Bearer <token>` value, or `null`. |
 | `this.statusCode` | Current outgoing status (settable, but prefer the render helpers below). |
 | `this.logger` | Per-request child logger. |

package/lib/server/server.js

@@ -192,7 +192,7 @@ class VidaServer {
         const method         = action.method.toLowerCase();
         const requestHandler = this.requestHandler(action.action, controllerCls)
         if (process.env.NODE_ENV != 'test') {
-            this.logger.verbose(`ROUTE: ${method.toUpperCase().padEnd(6)} ${action.path}`);
+            this.logger.debug(`ROUTE: ${method.toUpperCase().padEnd(6)} ${action.path}`);
         }
         this['_'+method](action.path, requestHandler);
     }

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@vida-global/core",
-    "version": "2.3.2",
+    "version": "2.3.4",
     "description": "Core libraries for supporting Vida development",
     "author": "",
     "license": "ISC",