@ti-engine/core 1.10.0 → 1.11.0

package/CHANGELOG.md

@@ -2,6 +2,68 @@
 
 This document contains the list of changes made to the framework. The format is based on the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification.
 
+## Version 1.11.0
+
+`decycle` silently lost a key named `__proto__` and corrupted the surrounding document. It built its replica as `{}`
+and copied keys in by bracket assignment, so that one name hit the inherited prototype setter instead of becoming an
+own property: the key vanished and its value became the replica's prototype, which `stringifyJSON` then flattened
+back in as unrelated top-level keys. An object with an own `__proto__` key holding `{ hello: 1 }` alongside an `ada`
+key serialized to `{"ada":{…},"hello":1}` — the key gone and `hello` promoted to a sibling. (Note the input cannot be
+written as an object literal: in `{ __proto__: x }` that name is a prototype setter, not a key, even quoted. Only a
+computed key or an assignment onto `Object.create( null )` produces the property this bug needed.)
+
+This reached Redis: `cache.setJSON` serializes with `stringifyJSON`, so any caller storing such a key lost it against
+a real cache. `decycle` also runs on the message-exchange integrity-hash path, and over an exception's `data` whenever
+`raise` is given one. No test caught it because web-framework's in-memory cache double clones with
+`JSON.parse( JSON.stringify( … ) )` and never reaches `stringifyJSON`.
+
+**This is a minor rather than a patch release because it requires a consumer code change** — see the note below. The
+fix is a bug fix, but the replica's prototype is part of the observable contract.
+
+* fix(tools): build `decycle`'s object replica with `Object.create( null )`, so a key named `__proto__` is an ordinary
+  own key with no setter to hit. Verified against both consumers of the output — `_.isPlainObject` accepts a
+  null-prototype object, so `decomposeJSON` (message hashing) and `_.toPlainObject` (serialization) are unaffected
+* fix(tools): the same guard in `errorToJSON`, which copied an error's own property names with the identical pattern
+* fix(cache)!: `Cache.getValues( keys, prefix )` resolved **every key to `null`**, whatever Redis returned. Its ternary
+  inspected `results`, the accumulator being built, instead of `result`, the per-key `[ error, value ]` entry — and
+  `{}.length` is `undefined`, so the comparison was always false. The single-key `getValue()` had the correct form; the
+  two expressions were near-identical and one drifted. No first-party caller existed, which is why nothing caught it.
+  The per-entry decode and the key mapping are now two shared module-level functions (`decodeCommandValue`,
+  `mapCommandValues`, exported for testing) rather than duplicated inline expressions, so the drift cannot recur. The
+  mapping iterates the requested keys rather than the raw results, so a short response yields an entry per requested
+  key instead of silently omitting some, and its accumulator is a null-prototype object for the same reason as above —
+  cache keys come from the caller, and one named `__proto__` would otherwise repoint the map's prototype. Marked
+  breaking only because the returned map now has a null prototype (see the note below); the value behaviour is a
+  straight bug fix
+* test(tools): pin the regression, the cycle handling it must not break, the `decomposeJSON` and `retrocycle` paths,
+  and the fact that a key named `constructor` was **never** affected — it is an ordinary writable data property, so
+  bracket assignment always shadowed it correctly, and the fix's shape should not imply otherwise
+* build(release): bump package version from `1.10.0` to `1.11.0`
+
+`retrocycle` was checked and is not affected: it mutates objects that already came through `JSON.parse`, which creates
+`__proto__` as an own data property, and assignment to an existing own property shadows the inherited setter.
+
+**Breaking for consumers who touch the returned object directly.** The objects returned by `decycle`, by `errorToJSON`
+for an `Error`, and by `Cache.getValues` now have a **null prototype**, so they inherit nothing from `Object.prototype`. Passing
+them to `JSON.stringify`, lodash, or `decomposeJSON` is unaffected; the following are not, and the first two throw
+rather than merely behaving differently:
+
+| Expression on the returned object | Before | Now |
+|---|---|---|
+| `` `${ result }` ``, `String( result )`, `result + ""` | `"[object Object]"` | **`TypeError: Cannot convert object to primitive value`** |
+| `result.toString()`, `result.valueOf()` | works | **`TypeError`** |
+| `result.hasOwnProperty( key )` | works | **`TypeError`** — use `Object.prototype.hasOwnProperty.call( result, key )` |
+| `result instanceof Object` | `true` | `false` |
+| `result.constructor` | `Object` | `undefined` |
+
+The likeliest way to hit this is interpolating an `errorToJSON` result straight into a log line. Wrap such a value in
+`_.cloneDeep( … )` — which normalizes the prototype back — or serialize it explicitly. No call site in this monorepo
+does any of the above; every first-party `hasOwnProperty` use is already the `Object.prototype.…call` form.
+
+**Mixed-version note:** for the rare message carrying a literal `__proto__` key, an old sender and an upgraded receiver
+compute different `createMessageHash` values, so the receiver raises `E_SEC_MESSAGE_TAMPERING_DETECTED`. Such messages
+were already being corrupted in transit, so this surfaces a pre-existing problem rather than creating one.
+
 ## Version 1.10.0
 
 * feat(localization): `getLabel( label, language, fallback )` takes an optional third argument returned when the key

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ti-engine/core",
-  "version": "1.10.0",
+  "version": "1.11.0",
   "description": "Microservice framework for Node.js: a Redis-backed message exchange with end-to-end call tracing, retries and tamper-evident message envelopes.",
   "keywords": [
     "microservices",

package/types/utils/cache.d.ts

@@ -1,6 +1,42 @@
 declare const _exported: Readonly<CommonMemoryCache>;
 export { _exported as instance };
+export { decodeCommandValue };
+export { mapCommandValues };
 import ConnectionObserver = require("#connection-observer");
+/**
+ * Decodes one entry of a `multi(...).exec()` result into the value it carries.
+ * <br/>
+ * Each entry is an ioredis `[ error, value ]` pair, so the value sits at index 1 and is a string when the key existed.
+ * Returns `undefined` for a miss, an error entry, or a malformed entry.
+ * <br/>
+ * This lives outside the class, and is shared by {@link CommonMemoryCache#getValue} and
+ * {@link CommonMemoryCache#getValues}, because it previously existed as two near-identical inline expressions and one
+ * of them drifted: `getValues` inspected its own accumulator instead of the per-key entry, so `.length` was
+ * `undefined`, the comparison was always false, and **every key resolved to `null`** whatever Redis returned.
+ *
+ * @method
+ * @param {Array} [result] One `[ error, value ]` entry.
+ * @returns {*} The parsed value, or `undefined` when there is none.
+ * @private
+ */
+declare function decodeCommandValue(result?: any[]): any;
+/**
+ * Maps a set of requested keys onto the values a `multi(...).exec()` returned for them, using `null` for a miss.
+ * <br/>
+ * Iterates the requested `keys` rather than the raw results, so a short or absent response still yields one entry per
+ * requested key instead of silently omitting some — the caller's map always has the shape it asked for.
+ * <br/>
+ * The accumulator has no prototype on purpose: the key names come from the caller, and a cache key named `__proto__`
+ * written by bracket assignment onto an ordinary `{}` would repoint the accumulator's prototype instead of creating
+ * the entry. Same class as the `decycle` defect fixed in `tools.js`; see the 1.11.0 changelog entry.
+ *
+ * @method
+ * @param {string[]} keys The keys that were requested, in command order.
+ * @param {Array} [rawResults] The `multi(...).exec()` result.
+ * @returns {Object} A null-prototype map of key to value, `null` where the key was absent.
+ * @private
+ */
+declare function mapCommandValues(keys: string[], rawResults?: any[]): Object;
 /**
  * Used to create and/or return a Common Memory Cache singleton instance.
  *

package/utils/cache.js

@@ -13,6 +13,55 @@ const tools = require( "#tools" );
 const redis = require( "#redis-integration" );
 const exceptions = require( "#exceptions" );
 
+/**
+ * Decodes one entry of a `multi(...).exec()` result into the value it carries.
+ * <br/>
+ * Each entry is an ioredis `[ error, value ]` pair, so the value sits at index 1 and is a string when the key existed.
+ * Returns `undefined` for a miss, an error entry, or a malformed entry.
+ * <br/>
+ * This lives outside the class, and is shared by {@link CommonMemoryCache#getValue} and
+ * {@link CommonMemoryCache#getValues}, because it previously existed as two near-identical inline expressions and one
+ * of them drifted: `getValues` inspected its own accumulator instead of the per-key entry, so `.length` was
+ * `undefined`, the comparison was always false, and **every key resolved to `null`** whatever Redis returned.
+ *
+ * @method
+ * @param {Array} [result] One `[ error, value ]` entry.
+ * @returns {*} The parsed value, or `undefined` when there is none.
+ * @private
+ */
+function decodeCommandValue( result ) {
+    // `Array.isArray` rather than a bare truthy-and-length test: a string also has a `length` and an indexable
+    // character at 1, so a malformed non-array entry would otherwise be parsed as if it were a value.
+    return ( Array.isArray( result ) && result.length > 1 && _.isString( result[ 1 ] ) ) ? tools.parseJSON( result[ 1 ] ) : undefined;
+}
+
+/**
+ * Maps a set of requested keys onto the values a `multi(...).exec()` returned for them, using `null` for a miss.
+ * <br/>
+ * Iterates the requested `keys` rather than the raw results, so a short or absent response still yields one entry per
+ * requested key instead of silently omitting some — the caller's map always has the shape it asked for.
+ * <br/>
+ * The accumulator has no prototype on purpose: the key names come from the caller, and a cache key named `__proto__`
+ * written by bracket assignment onto an ordinary `{}` would repoint the accumulator's prototype instead of creating
+ * the entry. Same class as the `decycle` defect fixed in `tools.js`; see the 1.11.0 changelog entry.
+ *
+ * @method
+ * @param {string[]} keys The keys that were requested, in command order.
+ * @param {Array} [rawResults] The `multi(...).exec()` result.
+ * @returns {Object} A null-prototype map of key to value, `null` where the key was absent.
+ * @private
+ */
+function mapCommandValues( keys, rawResults ) {
+    let values = Object.create( null );
+
+    _.forEach( keys, ( key, idx ) => {
+        let decoded = decodeCommandValue( rawResults ? rawResults[ idx ] : undefined );
+        values[ key ] = ( decoded === undefined ) ? null : decoded;
+    } );
+
+    return values;
+}
+
 /**
  * Used to create and/or return a Common Memory Cache singleton instance.
  *
@@ -260,8 +309,7 @@ class CommonMemoryCache extends ConnectionObserver {
             if ( this.#isOperational === true ) {
                 let commandGetValue = [ redis.cacheCommands.GET_VALUE, key ];
                 this.#redisClient.executeCommands( [ commandGetValue ] ).then( ( results ) => {
-                    results = results[ 0 ];
-                    resolve( ( results && results.length > 1 && _.isString( results[ 1 ] ) ) ? tools.parseJSON( results[ 1 ] ) : undefined );
+                    resolve( decodeCommandValue( results ? results[ 0 ] : undefined ) );
                 } ).catch( ( error ) => {
                     reject( error );
                 } );
@@ -288,11 +336,7 @@ class CommonMemoryCache extends ConnectionObserver {
                     commands.push( [ redis.cacheCommands.GET_VALUE, ( ( prefix ) ? prefix : "" ) + key ] );
                 } );
                 this.#redisClient.executeCommands( commands ).then( ( rawResults ) => {
-                    let results = {};
-                    _.forEach( rawResults, ( result, idx ) => {
-                        results[ keys[ idx ] ] = ( results && results.length > 1 && _.isString( results[ 1 ] ) ) ? tools.parseJSON( results[ 1 ] ) : null;
-                    } );
-                    resolve( results );
+                    resolve( mapCommandValues( keys, rawResults ) );
                 } ).catch( ( error ) => {
                     reject( error );
                 } );
@@ -769,4 +813,9 @@ class CommonMemoryCache extends ConnectionObserver {
 }
 
 const instance = new CommonMemoryCache();
-module.exports.instance = Object.freeze( instance );
+module.exports.instance = Object.freeze( instance );
+
+// Exported for testing. The cache singleton builds its own Redis client in its constructor, so `getValues` cannot be
+// driven without a live server — these are the pure halves of it, and they are where the defect was.
+module.exports.decodeCommandValue = decodeCommandValue;
+module.exports.mapCommandValues = mapCommandValues;

package/utils/tools.js

@@ -182,7 +182,11 @@ module.exports.getEnumName = ( enumList, enumValue, placeholder = undefined ) =>
  * @public
  */
 module.exports.errorToJSON = ( value ) => {
-    let error = {};
+    // Null prototype for the same reason as in `decycle`: the keys come from the error's own property names, and a
+    // key named `__proto__` written by bracket assignment onto an ordinary `{}` would repoint the prototype rather
+    // than be copied. An error carrying such a property is exotic — it takes a deliberate `defineProperty` — but the
+    // pattern is identical and the guard costs nothing.
+    let error = Object.create( null );
 
     if ( value instanceof Error ) {
         Object.getOwnPropertyNames( value ).forEach( ( key ) => {
@@ -337,7 +341,19 @@ module.exports.decycle = ( object, replacer ) => {
                 } );
             } else {
                 // If it is an object, replicate the object.
-                newItem = {};
+                // The replica has NO prototype on purpose. A key is copied in by bracket assignment, and on an
+                // ordinary `{}` a key named `__proto__` would hit the inherited setter instead of creating an own
+                // property: the key vanished from the replica and its value silently became the replica's prototype.
+                // `stringifyJSON` then ran `_.toPlainObject` over that, flattening the lost value's own fields back
+                // in as unrelated top-level keys, so the key vanished and its contents surfaced as siblings. Silent
+                // corruption, not a crash, and it reached Redis through `cache.setJSON`. With a null prototype there
+                // is no setter to hit, so `__proto__` is an ordinary key.
+                // Both consumers of this output are safe: `_.isPlainObject` accepts a null-prototype object, so
+                // `decomposeJSON` behaves identically, and `_.toPlainObject` preserves the key — but only because
+                // lodash's own `baseAssignValue` special-cases `__proto__` with `defineProperty`. That is a dependency
+                // this fix leans on, so `test/tools-proto-keys.test.js` asserts the whole chain end to end rather than
+                // just this function.
+                newItem = Object.create( null );
                 Object.keys( value ).forEach( ( name ) => {
                     newItem[ name ] = derez(
                         value[ name ],