soajs.controller 4.3.19 → 4.3.20

package/docs/maintenance-routes.md

@@ -18,6 +18,7 @@ The maintenance server runs on a separate port from the main gateway to allow in
 | `/loadProvision` | GET | Reload provision/tenant data |
 | `/reloadRegistry` | GET | Reload environment registry |
 | `/awarenessStat` | GET | Get service awareness statistics |
+| `/cache/invalidate` | POST | Remove cached GET responses a service knows are out of date |
 
 ---
 
@@ -226,6 +227,36 @@ curl http://localhost:5000/awarenessStat
 
 ---
 
+## 5. Cache Invalidation (`/cache/invalidate`)
+
+**Endpoint:** `POST /cache/invalidate`
+
+**Purpose:** Lets a service remove cached GET responses as soon as it knows they changed, instead of
+waiting for their TTL. Needs the mongo cache model.
+
+### Request
+
+```bash
+curl -X POST http://localhost:5000/cache/invalidate \
+  -H "Content-Type: application/json" \
+  -d '{"tenant": "5c0e74ba9acc3c5a84a51259", "service": "spaces", "api": "GET /spaces", "user": "5c8d0c505653de3985aa2001"}'
+```
+
+### Response
+
+```json
+{
+  "result": true,
+  "ts": 1792430000000,
+  "service": { "service": "soajs.controller", "version": "4.3.20", "type": "rest", "route": "/cache/invalidate" },
+  "data": { "removed": 3 }
+}
+```
+
+What each field matches, and the error cases, are in [Cache, Invalidation](middleware/cache.md#invalidation).
+
+---
+
 ## Security Considerations
 
 ### Network Access

package/docs/middleware/cache.md

@@ -157,7 +157,7 @@ APIs are matched using `GET /path` format with support for path parameters:
 - **tenant_id**: Isolates cache per tenant
 - **serviceName**: The target service name
 - **path**: The API path
-- **queryHash**: MD5 hash of sorted query parameters
+- **queryHash**: MD5 hash of sorted query parameters, without `access_token`
 - **user_id**: The logged-in user id (only for `tenant_user` scope), isolating cache per user within the tenant
 
 ### Query Parameter Handling
@@ -168,8 +168,14 @@ Query parameters are included in the cache key via MD5 hash:
 GET /users?page=1&limit=10  → key includes hash of {"limit":"10","page":"1"}
 GET /users?limit=10&page=1  → same hash (sorted)
 GET /users                  → different key (no query hash)
+GET /users?access_token=abc → same key as GET /users
 ```
 
+`access_token` is left out: it identifies the caller, not what is asked for. With it in the key, every
+token got its own entry, so a `tenant` scoped API was never shared between users, and a user's entries
+stopped matching after each login or token refresh. The user of a `tenant_user` entry is already in the
+key.
+
 ## Storage Backends
 
 ### Memory (Default)
@@ -229,6 +235,73 @@ Content-Type: application/json
 - Error responses (4xx, 5xx)
 - APIs not configured for caching
 
+## Invalidation
+
+A cached response normally lives until its TTL. A service that knows something changed can remove the
+affected entries right away by calling the gateway's maintenance port:
+
+```
+POST /cache/invalidate
+Content-Type: application/json
+
+{
+  "tenant": "5c0e74ba9acc3c5a84a51259",   // required, the tenant the GET was cached under
+  "service": "spaces",                    // required
+  "api": "GET /spaces",                   // optional, as written in the cache configuration
+  "user": "5c8d0c505653de3985aa2001"      // optional, the user id of a tenant_user entry
+}
+```
+
+```json
+{ "result": true, "data": { "removed": 3 } }
+```
+
+| Given | Removed |
+|-------|---------|
+| `tenant`, `service`, `api`, `user` | That user's entries of that api |
+| `tenant`, `service`, `api` | That api's entries for every user, and its tenant scoped entry |
+| `tenant`, `service`, `user` | That user's entries of every api of the service |
+| `tenant`, `service` | Every entry of the service in the tenant |
+
+Every cached variant of a matching entry is removed: every query string, and every value of a path
+parameter. `"api": "GET /space/:id"` removes `/space/1`, `/space/2` and so on. A `user` only matches
+`tenant_user` entries, a `tenant` scoped entry has no user.
+
+**Only the mongo model is supported.** It is the one shared by every gateway, so one call clears the
+entry everywhere. With the memory model the route answers `400`.
+
+**Example.** A service caches `GET /spaces`, the spaces a user is invited to, per user. When user A
+invites user B, the invite API removes B's list before it answers, so B's next call loads the new one:
+
+```javascript
+// at the end of the invite API, before replying
+await post("http://" + gatewayHost + ":" + maintenancePort + "/cache/invalidate", {
+	"tenant": req.soajs.tenant.id,
+	"service": "spaces",
+	"api": "GET /spaces",
+	"user": invitedUserId
+});
+```
+
+Wait for it with a short timeout. If the gateway cannot be reached, the change still stands and the
+entry expires on its TTL.
+
+**Requests already in flight.** A GET that started before the invalidation may have read the data
+before the change. Its response is still sent to the client but not stored, so it cannot put the old
+list back after the removal. Each invalidation is remembered for 15 minutes in the `cache_invalidation`
+collection for this check. It compares the time the gateway handling the GET received it with the time
+the invalidation ran, keep the gateways' clocks in sync.
+
+Entries cached before this version carry no service, api or user, an invalidation does not match them,
+they expire on their TTL.
+
+| Status | Reason |
+|--------|--------|
+| `200` | Removed, `data.removed` is the number of entries |
+| `400` | Missing `tenant` or `service`, an `api` not written as `GET /path`, a `user` that is not a string, caching not configured, or not the mongo model |
+| `405` | Not a `POST` |
+| `500` | The cache store failed or is not ready |
+
 ## Usage Notes
 
 - Only GET requests are cached

package/mw/cache/index.js

@@ -16,17 +16,28 @@ const { logOnce } = require('../../lib/logOnce.js');
 
 let model = { memory, mongo };
 
+//NOTE: the access token is how the caller is identified, not what is asked for. Kept in, every token
+//		gets its own entry and a tenant scoped api is never shared, the user is already in the key of a
+//		tenant_user entry.
+const QUERY_PARAMS_NOT_IN_KEY = ['access_token'];
+
 function hashQuery(query) {
-	if (!query || Object.keys(query).length === 0) {
+	if (!query) {
+		return '';
+	}
+	const params = Object.keys(query).filter((key) => !QUERY_PARAMS_NOT_IN_KEY.includes(key));
+	if (params.length === 0) {
 		return '';
 	}
-	const sortedQuery = Object.keys(query).sort().reduce((obj, key) => {
+	const sortedQuery = params.sort().reduce((obj, key) => {
 		obj[key] = query[key];
 		return obj;
 	}, {});
 	return crypto.createHash('md5').update(JSON.stringify(sortedQuery)).digest('hex');
 }
 
+//NOTE: returns the entry name as written in the configuration with its config, an invalidation names
+//		the api the same way
 function matchesApi(method, path, apis) {
 	if (!apis || typeof apis !== 'object') {
 		return null;
@@ -34,7 +45,7 @@ function matchesApi(method, path, apis) {
 	const requestSignature = `${method} ${path}`;
 
 	if (apis[requestSignature]) {
-		return apis[requestSignature];
+		return { 'name': requestSignature, 'config': apis[requestSignature] };
 	}
 
 	for (const api in apis) {
@@ -59,7 +70,7 @@ function matchesApi(method, path, apis) {
 				}
 			}
 			if (matches) {
-				return apis[api];
+				return { 'name': api, 'config': apis[api] };
 			}
 		}
 	}
@@ -96,7 +107,8 @@ module.exports = function (configuration) {
 			return next();
 		}
 
-		const apiConfig = matchesApi(method, apiPath, serviceConfig.apis);
+		const matchedApi = matchesApi(method, apiPath, serviceConfig.apis);
+		const apiConfig = matchedApi ? matchedApi.config : null;
 		if (!apiConfig || !apiConfig.enabled) {
 			return next();
 		}
@@ -141,6 +153,7 @@ module.exports = function (configuration) {
 			'l2': `${serviceName}:GET:${apiPath}:${queryHash}${userPart}`
 		};
 
+		const startedAt = Date.now();
 		const processCache = async () => {
 			try {
 				const cached = await model[modelType].get(key);
@@ -157,6 +170,12 @@ module.exports = function (configuration) {
 				req.soajs.cacheContext = {
 					key: key,
 					ttl: ttl,
+					meta: {
+						service: serviceName,
+						api: matchedApi.name,
+						user: (scope === 'tenant_user') ? String(userId) : null,
+						startedAt: startedAt
+					},
 					modelType: modelType,
 					model: model[modelType],
 					log: configuration.log

package/mw/cache/invalidate.js

@@ -0,0 +1,71 @@
+'use strict';
+
+/**
+ * @license
+ * Copyright SOAJS All Rights Reserved.
+ *
+ * Use of this source code is governed by an Apache license that can be
+ * found in the LICENSE file at the root of this repository
+ */
+
+const get = (p, o) => p.reduce((xs, x) => (xs && xs[x]) ? xs[x] : null, o);
+const mongo = require('./model/mongo.js');
+
+const API_REGEX = /^GET \/\S*$/;
+
+function isNonEmptyString(value) {
+	return typeof value === 'string' && value.length > 0;
+}
+
+/**
+ * Remove cached GET responses a service knows are out of date, called on the maintenance port.
+ *
+ * NOTE: only the mongo model is supported, it is the one shared by every gateway, the memory model
+ *       is per gateway and meant for testing
+ *
+ * @param registry the gateway registry
+ * @param input {tenant, service, api, user} tenant and service are required
+ * @returns {Promise<{status: number, removed: number}|{status: number, error: string}>}
+ */
+async function invalidate(registry, input) {
+	if (!input || typeof input !== 'object') {
+		return { 'status': 400, 'error': 'A JSON body with tenant and service is required.' };
+	}
+	if (!isNonEmptyString(input.tenant) || !isNonEmptyString(input.service)) {
+		return { 'status': 400, 'error': 'tenant and service are required.' };
+	}
+	if (input.api !== undefined && (!isNonEmptyString(input.api) || !API_REGEX.test(input.api))) {
+		return { 'status': 400, 'error': 'api must be written as in the cache configuration, for example "GET /spaces".' };
+	}
+	if (input.user !== undefined && !isNonEmptyString(input.user)) {
+		return { 'status': 400, 'error': 'user must be a user id.' };
+	}
+
+	const cacheConfig = get(["custom", "gateway", "value", "cache"], registry);
+	if (!cacheConfig) {
+		return { 'status': 400, 'error': 'Caching is not configured.' };
+	}
+	if (cacheConfig.model !== 'mongo') {
+		return { 'status': 400, 'error': 'Cache invalidation needs the mongo model, the cache model is [' + (cacheConfig.model || 'memory') + '].' };
+	}
+	if (!mongo.mongo) {
+		return { 'status': 500, 'error': 'The cache store is not ready.' };
+	}
+
+	try {
+		const removed = await mongo.invalidateMatching({
+			'tenant': input.tenant,
+			'service': input.service,
+			'api': input.api,
+			'user': input.user
+		});
+		return { 'status': 200, 'removed': removed };
+	} catch (e) {
+		if (mongo.log) {
+			mongo.log.error('Cache invalidate:', e.message);
+		}
+		return { 'status': 500, 'error': 'Cache invalidation failed.' };
+	}
+}
+
+module.exports = invalidate;

package/mw/cache/model/mongo.js

@@ -12,6 +12,11 @@ const core = require('soajs.core.modules');
 const Mongo = core.mongo;
 
 const cache_store = 'cache_store';
+const cache_invalidation = 'cache_invalidation';
+
+//NOTE: an invalidation is remembered long enough to outlive any request that started before it,
+//		so that request cannot store what it read before the change
+const INVALIDATION_RETENTION = 15 * 60 * 1000;
 
 let model = {
 	'mongo': null,
@@ -35,6 +40,27 @@ let model = {
 					log.debug("Cache TTL index created: " + index);
 				}
 			});
+			model.mongo.createIndex(cache_store, { 'l1': 1, 'service': 1, 'api': 1, 'user': 1 }, {}, (err, index) => {
+				if (err) {
+					log.error("Cache invalidation lookup index creation failed: " + err.message);
+				} else {
+					log.debug("Cache invalidation lookup index created: " + index);
+				}
+			});
+			model.mongo.createIndex(cache_invalidation, { 'l1': 1, 'service': 1, 'api': 1, 'user': 1 }, { unique: true }, (err, index) => {
+				if (err) {
+					log.error("Cache invalidation index creation failed: " + err.message);
+				} else {
+					log.debug("Cache invalidation index created: " + index);
+				}
+			});
+			model.mongo.createIndex(cache_invalidation, { 'expiresAt': 1 }, { expireAfterSeconds: 0 }, (err, index) => {
+				if (err) {
+					log.error("Cache invalidation TTL index creation failed: " + err.message);
+				} else {
+					log.debug("Cache invalidation TTL index created: " + index);
+				}
+			});
 		}
 	},
 
@@ -58,7 +84,15 @@ let model = {
 		}
 	},
 
-	'set': async (key, response, ttl) => {
+	/**
+	 * @param key {l1, l2}
+	 * @param response {statusCode, headers, body}
+	 * @param ttl
+	 * @param meta {service, api, user, startedAt} what an invalidation matches on, user is null for a
+	 *        tenant scoped entry
+	 * @returns {Promise<boolean>} false when not stored
+	 */
+	'set': async (key, response, ttl, meta) => {
 		let condition = {
 			'l1': key.l1,
 			'l2': key.l2
@@ -70,13 +104,73 @@ let model = {
 			'cachedAt': new Date(),
 			'expiresAt': new Date(Date.now() + ttl)
 		};
+		if (meta) {
+			doc.service = meta.service;
+			doc.api = meta.api;
+			doc.user = meta.user || null;
+		}
 		let options = { 'upsert': true };
 		let s = { '$set': doc };
 		try {
+			//NOTE: a request that started before an invalidation covering it may have read the data
+			//		before the change, its response is passed on but not stored
+			if (meta && meta.startedAt) {
+				let invalidated = await model.mongo.findOne(cache_invalidation, {
+					'l1': key.l1,
+					'service': meta.service,
+					'api': { '$in': [meta.api, null] },
+					'user': { '$in': [doc.user, null] },
+					'clearedAt': { '$gt': new Date(meta.startedAt) }
+				});
+				if (invalidated) {
+					return false;
+				}
+			}
 			await model.mongo.updateOne(cache_store, condition, s, options);
+			return true;
 		} catch (e) {
 			model.log.error('Cache set:', e.message);
+			return false;
+		}
+	},
+
+	/**
+	 * Remove every cached variant of the matching entries, all query strings and path params.
+	 *
+	 * NOTE: the invalidation is recorded before the entries are removed, a request that read before the
+	 *       change and stores after the removal is caught by set
+	 *
+	 * @param criteria {tenant, service, api, user} api and user are optional, without them every api of
+	 *        the service, for every user and the tenant scoped entries
+	 * @returns {Promise<number>} the number of entries removed
+	 */
+	'invalidateMatching': async (criteria) => {
+		let marker = {
+			'l1': criteria.tenant,
+			'service': criteria.service,
+			'api': criteria.api || null,
+			'user': criteria.user || null
+		};
+		let now = new Date();
+		await model.mongo.updateOne(cache_invalidation, marker, {
+			'$set': Object.assign({}, marker, {
+				'clearedAt': now,
+				'expiresAt': new Date(now.getTime() + INVALIDATION_RETENTION)
+			})
+		}, { 'upsert': true });
+
+		let condition = {
+			'l1': criteria.tenant,
+			'service': criteria.service
+		};
+		if (criteria.api) {
+			condition.api = criteria.api;
+		}
+		if (criteria.user) {
+			condition.user = criteria.user;
 		}
+		let response = await model.mongo.deleteMany(cache_store, condition);
+		return (response && response.deletedCount) || 0;
 	},
 
 	'invalidate': async (key) => {

package/mw/gotoService/redirectToService.js

@@ -411,7 +411,7 @@ module.exports = (configuration) => {
 							headers: cacheResponseHeaders,
 							body: cacheBody
 						};
-						cacheContext.model.set(cacheContext.key, cacheData, cacheContext.ttl);
+						cacheContext.model.set(cacheContext.key, cacheData, cacheContext.ttl, cacheContext.meta);
 					}
 					// Complete idempotency if applicable
 					//NOTE: only an answer of the service is replayed, a 5xx means it could not answer, release

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "soajs.controller",
   "description": "soajs multi tenant API gateway",
-  "version": "4.3.19",
+  "version": "4.3.20",
   "author": {
     "name": "soajs team",
     "email": "team@soajs.org"

package/server/maintenance.js

@@ -11,6 +11,7 @@
 const get = (p, o) => p.reduce((xs, x) => (xs && xs[x]) ? xs[x] : null, o);
 
 const registryModule = require("./../modules/registry");
+const cacheInvalidate = require("./../mw/cache/invalidate.js");
 
 const httpProxy = require('http-proxy');
 const http = require('http');
@@ -311,6 +312,46 @@ let Maintenance = (core, log, param, serviceIp, regEnvironment, awareness_mw, so
 			};
 			proxy.web(req, res, {target: haTarget});
 		}
+		else if (parsedUrl.pathname === '/cache/invalidate') {
+			let response = maintenanceResponse(parsedUrl, param);
+			let reply = (status, extra) => {
+				res.writeHead(status, {'Content-Type': 'application/json'});
+				return res.end(JSON.stringify(Object.assign(response, extra)));
+			};
+			if (req.method !== 'POST') {
+				return reply(405, {"error": "Use POST."});
+			}
+			let body = "";
+			let tooLarge = false;
+			req.on('data', function (chunk) {
+				body += chunk;
+				//NOTE: the body only names what to remove, anything larger is not a valid request
+				if (body.length > 16384) {
+					tooLarge = true;
+					req.destroy();
+				}
+			});
+			req.on('end', function () {
+				if (tooLarge) {
+					return;
+				}
+				let input = null;
+				try {
+					input = body ? JSON.parse(body) : null;
+				} catch (e) {
+					return reply(400, {"error": "The body is not valid JSON."});
+				}
+				cacheInvalidate(registryModule.get(), input).then((outcome) => {
+					if (outcome.status !== 200) {
+						return reply(outcome.status, {"error": outcome.error});
+					}
+					log.info("Cache invalidated for tenant [" + input.tenant + "] service [" + input.service + "]" +
+						(input.api ? " api [" + input.api + "]" : "") + (input.user ? " user [" + input.user + "]" : "") +
+						", removed [" + outcome.removed + "]");
+					return reply(200, {"result": true, "data": {"removed": outcome.removed}});
+				});
+			});
+		}
 		else if (parsedUrl.pathname === '/loadProvision') {
 			provision.loadProvision(function (loaded) {
 				let response = maintenanceResponse(parsedUrl, param);

package/test/data/provision/customRegistry/gateway.js

@@ -16,6 +16,21 @@ let doc = {
 				}
 			}
 		},
+		"cache": {
+			"model": "mongo",
+			"defaultTTL": 60000,
+			"services": {
+				"spaces": {
+					"enabled": true,
+					"apis": {
+						"GET /spaces": { "enabled": true, "scope": "tenant_user" },
+						"GET /space/:id": { "enabled": true, "scope": "tenant_user" },
+						"GET /slow": { "enabled": true, "scope": "tenant_user" },
+						"GET /catalog": { "enabled": true, "scope": "tenant" }
+					}
+				}
+			}
+		},
 		"traffic": {
 			"throttling": {
 				"idempotent": {

package/test/data/provision/hosts/spaces.js

@@ -0,0 +1,12 @@
+'use strict';
+
+let host = {
+    _id: "5da078e22357684b57effcf5",
+    env: "dev",
+    ip: "127.0.0.1",
+    name: "spaces",
+    version: "1",
+    port: 4015
+};
+
+module.exports = host;

package/test/data/provision/marketplace/spaces.js

@@ -0,0 +1,36 @@
+'use strict';
+
+let service = {
+	_id: "5da078c92357684b57effcf4",
+	"type": "service",
+	name: "spaces",
+	configuration: {
+		group: "SOAJS test",
+		port: 4015,
+		requestTimeout: 30,
+		requestTimeoutRenewal: 5
+	},
+	versions: [{
+		"version": "1",
+		maintenance: {
+			"port": {
+				"type": "maintenance"
+			},
+			"readiness": "/heartbeat"
+		},
+		oauth: true,
+		extKeyRequired: true,
+		urac: true,
+		urac_Profile: false,
+		urac_ACL: false,
+		provision_ACL: false,
+		apis: [
+			{ l: "Spaces of the user", v: "/spaces", m: "get", group: "Integration test" },
+			{ l: "One space", v: "/space/:id", m: "get", group: "Integration test" },
+			{ l: "Spaces after a delay", v: "/slow", m: "get", group: "Integration test" },
+			{ l: "Same for the whole tenant", v: "/catalog", m: "get", group: "Integration test" }
+		]
+	}
+	]
+};
+module.exports = service;

package/test/data/provision/oauth/token/second_owner_token.js

@@ -0,0 +1,72 @@
+'use strict';
+
+let token = {
+    _id: "5cdc190fd52c82e0ddb1dcd8",
+    type: "accessToken",
+    token: "c4c4e5e5f6f6a7a7b8b8c9c9d0d0e1e1f2f2a301",
+    clientId: "5c0e74ba9acc3c5a84a51259",
+    user: {
+        _id: "5c8d0c505653de3985aa2001",
+        locked: true,
+        username: "second_owner",
+        firstName: "owner3'sèį",
+        lastName: "owner",
+        email: "me@localhost.com",
+        "ts": new Date().getTime(),
+        status: "active",
+        profile: {},
+        groups: [
+            "owner"
+        ],
+        config: {
+            packages: {},
+            keys: {},
+            allowedTenants: [
+                {
+                    tenant: {
+                        id: "THYME_tID",
+                        code: "THYME_CODE",
+                        pin: {
+                            code: "5678",
+                            allowed: true
+                        }
+                    },
+                    groups: [
+                        "waiter"
+                    ]
+                },
+                {
+                    tenant: {
+                        id: "ELVIRA_tID",
+                        code: "ELVIRA_CODE"
+                    },
+                    groups: [
+                        "manager"
+                    ]
+                }
+            ]
+        },
+        tenant: {
+            id: "5c0e74ba9acc3c5a84a51259",
+            code: "DBTN",
+            pin: {
+                code: "1235",
+                allowed: true
+            }
+        },
+        lastLogin: new Date().getTime(),
+        groupsConfig: {
+            allowedPackages: {
+                DSBRD: [
+                    "DSBRD_OWNER"
+                ]
+            },
+            allowedEnvironments: {}
+        },
+        loginMode: "urac",
+        id: "5c8d0c505653de3985aa2001"
+    },
+    env: "dev",
+    expires: new Date((new Date().getFullYear()) + 2, 0, 1)
+};
+module.exports = token;

package/test/data/provision/products/dsbrd.js

@@ -9,6 +9,22 @@ let prod = {
 	"scope": {
 		"acl": {
 			"dev": {
+				"spaces": {
+					"1": {
+						"access": true,
+						"get": [
+							{
+								"group": "Spaces group",
+								"apis": {
+									"/spaces": {},
+									"/space/:id": {},
+									"/slow": {},
+									"/catalog": {}
+								}
+							}
+						]
+					}
+				},
 				"idempotent": {
 					"1": {
 						"access": true,
@@ -217,6 +233,14 @@ let prod = {
 			"locked": true,
 			"acl": {
 				"dev": {
+					"spaces": [
+						{
+							"version": "1",
+							"get": [
+								"Spaces group"
+							]
+						}
+					],
 					"idempotent": [
 						{
 							"version": "1",

package/test/data/provision/urac/users/second_owner.js

@@ -0,0 +1,61 @@
+'use strict';
+let user = {
+    _id: "5c8d0c505653de3985aa2001",
+    locked: true,
+    username: "second_owner",
+    password: "$2a$12$geJJfv33wkYIXEAlDkeeuOgiQ6y6MjP/YxbqLdHdDSK7LDG.7n7Pq",
+    firstName: "owner3'sèį",
+    lastName: "owner",
+    email: "second_owner@localhost.com",
+    ts: 1552747600152,
+    status: "active",
+    profile: {
+
+    },
+    groups: [
+        "owner"
+    ],
+    config: {
+        packages: {
+
+        },
+        keys: {
+
+        },
+        allowedTenants: [
+            {
+                tenant: {
+                    id: "THYME_tID",
+                    code: "THYME_CODE",
+                    pin: {
+                        code: "5678",
+                        allowed: true
+                    }
+                },
+                groups: [
+                    "waiter"
+                ]
+            },
+            {
+                tenant: {
+                    id: "ELVIRA_tID",
+                    code: "ELVIRA_CODE"
+                },
+                groups: [
+                    "manager"
+                ]
+            }
+        ]
+    },
+    tenant: {
+        id: "5c0e74ba9acc3c5a84a51259",
+        code: "DBTN",
+        pin: {
+            code: "1235",
+            allowed: true
+        }
+    },
+    lastLogin: 1559836949372
+};
+
+module.exports = user;

package/test/integration/UC7-cache-invalidate/index.js

@@ -0,0 +1,209 @@
+"use strict";
+
+const assert = require('assert');
+const express = require('express');
+const request = require('request');
+const helper = require("../../helper.js");
+let Controller = helper.requireModule('./server/controller');
+let localConfig = helper.requireModule('./config.js');
+
+//NOTE: the gateway custom registry caches the spaces service with the mongo model, per user except
+//		/catalog which is per tenant. Both tokens are users of the console tenant.
+const tenantId = "5c0e74ba9acc3c5a84a51259";
+const ownerToken = "cfb209a91b23896820f510aadbf1f4284b512123";
+const ownerId = "5c8d0c505653de3985aa0ffd";
+const secondToken = "c4c4e5e5f6f6a7a7b8b8c9c9d0d0e1e1f2f2a301";
+const secondId = "5c8d0c505653de3985aa2001";
+const maintenanceUri = 'http://127.0.0.1:5001/cache/invalidate';
+
+function startService(calls, callback) {
+	const sApp = express();
+	const mApp = express();
+	let answer = (name) => (req, res) => {
+		calls[name] = (calls[name] || 0) + 1;
+		res.json({ "result": true, "data": { "call": calls[name], "url": req.url } });
+	};
+	sApp.get('/spaces', answer("spaces"));
+	sApp.get('/space/:id', answer("space"));
+	sApp.get('/catalog', answer("catalog"));
+	sApp.get('/slow', (req, res) => {
+		calls.slow = (calls.slow || 0) + 1;
+		let call = calls.slow;
+		setTimeout(() => {
+			res.json({ "result": true, "data": { "call": call } });
+		}, 1000);
+	});
+	mApp.get('/heartbeat', (req, res) => {
+		res.json({ "result": true, "ts": Date.now(), "service": { "service": "spaces", "type": "rest", "route": "/heartbeat" } });
+	});
+	let sAppServer = sApp.listen(4015, () => {
+		let mAppServer = mApp.listen(5016, () => {
+			return callback({ "sAppServer": sAppServer, "mAppServer": mAppServer });
+		});
+	});
+}
+
+describe("Integration for Usecase 7 - cache invalidation", function () {
+	let extKey = "3d90163cf9d6b3076ad26aa5ed58556348069258e5c6c941ee0f18448b570ad1c5c790e2d2a1989680c55f4904e2005ff5f8e71606e4aa641e67882f4210ebbc5460ff305dcb36e6ec2a2299cf0448ef60b9e38f41950ec251c1cf41f05f3ce9";
+
+	let c = new Controller({
+		"serviceName": localConfig.serviceName,
+		"serviceVersion": localConfig.serviceVersion,
+		"serviceGroup": localConfig.serviceGroup,
+		"servicePort": localConfig.servicePort,
+		"maintenance": localConfig.maintenance
+	});
+	let registry = null;
+	let log = null;
+	let service = null;
+	let server = null;
+	let serverMaintenance = null;
+	let mockServers = null;
+	let calls = {};
+
+	// GET through the gateway, resolves with the X-Cache header and the body
+	function get(path, token) {
+		return new Promise((resolve) => {
+			request.get({
+				"uri": 'http://127.0.0.1:' + localConfig.servicePort + '/spaces' + path,
+				"json": true,
+				"headers": {
+					"key": extKey,
+					"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36"
+				},
+				"qs": { "access_token": token }
+			}, (error, response, body) => {
+				assert.ifError(error);
+				resolve({ "cache": response.headers['x-cache'], "status": response.statusCode, body });
+			});
+		});
+	}
+
+	function invalidate(body, method) {
+		return new Promise((resolve) => {
+			request({
+				"method": method || 'POST',
+				"uri": maintenanceUri,
+				"json": body === undefined ? true : body
+			}, (error, response, responseBody) => {
+				assert.ifError(error);
+				resolve({ "status": response.statusCode, "body": responseBody });
+			});
+		});
+	}
+
+	// the response is stored once the proxy finishes, after the client got it
+	const settle = () => new Promise((r) => setTimeout(r, 150));
+
+	before((done) => {
+		//NOTE: the unit tests run in the same process and connect the cache mongo model to a dummy
+		//		database, the model connects only once, drop it so the gateway connects it to the real one
+		helper.requireModule('./mw/cache/model/mongo').mongo = null;
+		startService(calls, (servers) => {
+			mockServers = servers;
+			c.init((_registry, _log, _service, _server, _serverMaintenance) => {
+				registry = _registry;
+				log = _log;
+				service = _service;
+				server = _server;
+				serverMaintenance = _serverMaintenance;
+				c.start(registry, log, service, server, serverMaintenance, () => {
+					setTimeout(function () {
+						done();
+					}, 5000);
+				});
+			});
+		});
+	});
+
+	it("a cached response is served from the cache", async () => {
+		let first = await get('/spaces', ownerToken);
+		assert.strictEqual(first.status, 200, JSON.stringify(first.body));
+		assert.strictEqual(first.cache, 'MISS');
+		await settle();
+		let second = await get('/spaces', ownerToken);
+		assert.strictEqual(second.cache, 'HIT');
+		assert.strictEqual(second.body.data.call, first.body.data.call);
+	});
+
+	it("one user: every query variant of the api is removed, the other user keeps theirs", async () => {
+		await get('/spaces?page=1', ownerToken);
+		await get('/spaces?page=2', ownerToken);
+		await get('/spaces', secondToken);
+		await settle();
+		assert.strictEqual((await get('/spaces?page=1', ownerToken)).cache, 'HIT');
+		assert.strictEqual((await get('/spaces', secondToken)).cache, 'HIT');
+
+		let outcome = await invalidate({ "tenant": tenantId, "service": "spaces", "api": "GET /spaces", "user": ownerId });
+		assert.strictEqual(outcome.status, 200, JSON.stringify(outcome.body));
+		assert.strictEqual(outcome.body.result, true);
+		assert.strictEqual(outcome.body.data.removed, 3);
+
+		assert.strictEqual((await get('/spaces', ownerToken)).cache, 'MISS');
+		assert.strictEqual((await get('/spaces?page=1', ownerToken)).cache, 'MISS');
+		assert.strictEqual((await get('/spaces?page=2', ownerToken)).cache, 'MISS');
+		assert.strictEqual((await get('/spaces', secondToken)).cache, 'HIT');
+	});
+
+	it("without a user every user's entries are removed", async () => {
+		await settle();
+		assert.strictEqual((await get('/spaces', ownerToken)).cache, 'HIT');
+		assert.strictEqual((await get('/spaces', secondToken)).cache, 'HIT');
+		let outcome = await invalidate({ "tenant": tenantId, "service": "spaces", "api": "GET /spaces" });
+		assert.strictEqual(outcome.status, 200);
+		assert.ok(outcome.body.data.removed >= 2);
+		assert.strictEqual((await get('/spaces', ownerToken)).cache, 'MISS');
+		assert.strictEqual((await get('/spaces', secondToken)).cache, 'MISS');
+	});
+
+	it("an api with a path param is removed for every value", async () => {
+		await get('/space/1', ownerToken);
+		await get('/space/2', ownerToken);
+		await settle();
+		assert.strictEqual((await get('/space/1', ownerToken)).cache, 'HIT');
+		let outcome = await invalidate({ "tenant": tenantId, "service": "spaces", "api": "GET /space/:id", "user": ownerId });
+		assert.strictEqual(outcome.body.data.removed, 2);
+		assert.strictEqual((await get('/space/1', ownerToken)).cache, 'MISS');
+		assert.strictEqual((await get('/space/2', ownerToken)).cache, 'MISS');
+	});
+
+	it("a tenant scoped entry is shared by users, and removed without a user", async () => {
+		await get('/catalog', ownerToken);
+		await settle();
+		assert.strictEqual((await get('/catalog', secondToken)).cache, 'HIT');
+		let outcome = await invalidate({ "tenant": tenantId, "service": "spaces", "api": "GET /catalog" });
+		assert.strictEqual(outcome.body.data.removed, 1);
+		assert.strictEqual((await get('/catalog', secondToken)).cache, 'MISS');
+	});
+
+	it("a request that started before the invalidation is not stored", async () => {
+		let inFlight = get('/slow', ownerToken);
+		await new Promise((r) => setTimeout(r, 300));
+		let outcome = await invalidate({ "tenant": tenantId, "service": "spaces", "api": "GET /slow", "user": ownerId });
+		assert.strictEqual(outcome.status, 200);
+		let first = await inFlight;
+		assert.strictEqual(first.status, 200);
+		await settle();
+		let next = await get('/slow', ownerToken);
+		assert.strictEqual(next.cache, 'MISS');
+		assert.strictEqual(next.body.data.call, first.body.data.call + 1);
+		await settle();
+		assert.strictEqual((await get('/slow', ownerToken)).cache, 'HIT');
+	});
+
+	it("refuses an invalid request", async () => {
+		assert.strictEqual((await invalidate({ "service": "spaces" })).status, 400);
+		assert.strictEqual((await invalidate({ "tenant": tenantId, "service": "spaces", "api": "/spaces" })).status, 400);
+		let wrongMethod = await invalidate(undefined, 'GET');
+		assert.strictEqual(wrongMethod.status, 405);
+		assert.strictEqual(wrongMethod.body.result, false);
+	});
+
+	after((done) => {
+		mockServers.sAppServer.close();
+		mockServers.mAppServer.close();
+		c.stop(registry, log, service, server, serverMaintenance, () => {
+			done();
+		});
+	});
+});

package/test/integration/index.js

@@ -44,6 +44,10 @@ describe("starting integration tests", () => {
 		// to test idempotency with the mongo model
 		require("./UC6-idempotency/index.js");
 		
+		
+		// to test cache invalidation with the mongo model
+		require("./UC7-cache-invalidate/index.js");
+		
 		done();
 	});
 });

package/test/unit/index.js

@@ -42,6 +42,8 @@ describe("Starting Gateway Unit test", () => {
         require("./mw/idempotency/mongo.js");
         require("./mw/cache/index.js");
         require("./mw/cache/memory.js");
+        require("./mw/cache/mongo.js");
+        require("./mw/cache/invalidate.js");
         require("./mw/lastSeen/index.js");
 
         require("./utilities/utils.js");

package/test/unit/mw/cache/index.js

@@ -575,4 +575,126 @@ describe("Unit test for: mw - cache", () => {
 		});
 	});
 
+	describe("what an invalidation matches on", () => {
+		let noopRes = {
+			setHeader: function() {},
+			writeHead: function() { return this; },
+			write: function() { return true; },
+			end: function() {}
+		};
+
+		function buildReq(path, isAPIPublic, apis) {
+			return {
+				method: 'GET',
+				query: { page: '2' },
+				soajs: {
+					tenant: { id: 'tenant-meta' },
+					uracDriver: { id: 'user-7' },
+					controller: {
+						serviceParams: { name: 'spaces', serviceInfo: ['', 'spaces'].concat(path.split('/').slice(1)), isAPIPublic: isAPIPublic }
+					},
+					registry: {
+						custom: {
+							gateway: {
+								value: {
+									cache: {
+										model: 'memory',
+										services: {
+											spaces: { enabled: true, apis: apis }
+										}
+									}
+								}
+							}
+						}
+					}
+				}
+			};
+		}
+
+		it("records the service, the api as configured and the user", (done) => {
+			let before = Date.now();
+			let req = buildReq('/space/42', false, { 'GET /space/:id': { enabled: true } });
+			mw(mockConfig)(req, noopRes, (error) => {
+				assert.ifError(error);
+				let meta = req.soajs.cacheContext.meta;
+				assert.strictEqual(meta.service, 'spaces');
+				assert.strictEqual(meta.api, 'GET /space/:id');
+				assert.strictEqual(meta.user, 'user-7');
+				assert.ok(meta.startedAt >= before && meta.startedAt <= Date.now());
+				done();
+			});
+		});
+
+		it("records no user for a tenant scoped api", (done) => {
+			let req = buildReq('/catalog', false, { 'GET /catalog': { enabled: true, scope: 'tenant' } });
+			mw(mockConfig)(req, noopRes, (error) => {
+				assert.ifError(error);
+				assert.strictEqual(req.soajs.cacheContext.meta.api, 'GET /catalog');
+				assert.strictEqual(req.soajs.cacheContext.meta.user, null);
+				done();
+			});
+		});
+	});
+
+	describe("the access token is not part of the key", () => {
+		let noopRes = {
+			setHeader: function() {},
+			writeHead: function() { return this; },
+			write: function() { return true; },
+			end: function() {}
+		};
+
+		function buildReq(query, userId) {
+			return {
+				method: 'GET',
+				query: query,
+				soajs: {
+					tenant: { id: 'tenant-token' },
+					uracDriver: { id: userId },
+					controller: { serviceParams: { name: 'test', serviceInfo: ['', 'test', 'catalog'], isAPIPublic: true } },
+					registry: {
+						custom: {
+							gateway: {
+								value: {
+									cache: {
+										model: 'memory',
+										services: {
+											test: { enabled: true, apis: { 'GET /catalog': { enabled: true, scope: 'tenant' } } }
+										}
+									}
+								}
+							}
+						}
+					}
+				}
+			};
+		}
+
+		function keyOf(query, userId) {
+			return new Promise((resolve) => {
+				let req = buildReq(query, userId);
+				mw(mockConfig)(req, noopRes, () => {
+					resolve(req.soajs.cacheContext.key.l2);
+				});
+			});
+		}
+
+		it("two users with their own tokens share a tenant scoped entry", async () => {
+			let ann = await keyOf({ access_token: 'AAA', page: '1' }, 'ann');
+			let bob = await keyOf({ access_token: 'BBB', page: '1' }, 'bob');
+			assert.strictEqual(ann, bob);
+		});
+
+		it("a token alone gives the same key as no query", async () => {
+			let withToken = await keyOf({ access_token: 'AAA' }, 'ann');
+			let without = await keyOf({}, 'ann');
+			assert.strictEqual(withToken, without);
+		});
+
+		it("the other query params still make the key", async () => {
+			let one = await keyOf({ access_token: 'AAA', page: '1' }, 'ann');
+			let two = await keyOf({ access_token: 'AAA', page: '2' }, 'ann');
+			assert.notStrictEqual(one, two);
+		});
+	});
 });

package/test/unit/mw/cache/invalidate.js

@@ -0,0 +1,99 @@
+"use strict";
+
+const helper = require("../../../helper.js");
+const invalidate = helper.requireModule('./mw/cache/invalidate');
+const model = helper.requireModule('./mw/cache/model/mongo');
+const assert = require('assert');
+
+describe("Unit test for: mw - cache invalidate", () => {
+
+	let received = null;
+	let failure = null;
+	let originalInvalidateMatching = null;
+	let originalMongo = null;
+
+	function registry(model) {
+		return { custom: { gateway: { value: { cache: { model: model, services: {} } } } } };
+	}
+
+	before(() => {
+		originalInvalidateMatching = model.invalidateMatching;
+		originalMongo = model.mongo;
+		model.mongo = {};
+		model.invalidateMatching = async (criteria) => {
+			received = criteria;
+			if (failure) {
+				throw failure;
+			}
+			return 2;
+		};
+	});
+	after(() => {
+		model.invalidateMatching = originalInvalidateMatching;
+		model.mongo = originalMongo;
+	});
+	beforeEach(() => {
+		received = null;
+		failure = null;
+	});
+
+	it("removes the matching entries", async () => {
+		let outcome = await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces', api: 'GET /spaces', user: 'u1' });
+		assert.deepStrictEqual(outcome, { status: 200, removed: 2 });
+		assert.deepStrictEqual(received, { tenant: 't1', service: 'spaces', api: 'GET /spaces', user: 'u1' });
+	});
+
+	it("api and user are optional", async () => {
+		let outcome = await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces' });
+		assert.strictEqual(outcome.status, 200);
+		assert.deepStrictEqual(received, { tenant: 't1', service: 'spaces', api: undefined, user: undefined });
+	});
+
+	it("refuses the memory model", async () => {
+		let outcome = await invalidate(registry('memory'), { tenant: 't1', service: 'spaces' });
+		assert.strictEqual(outcome.status, 400);
+		assert.ok(outcome.error.indexOf('mongo') !== -1);
+		assert.strictEqual(received, null);
+	});
+
+	it("refuses when no model is set, memory is the default", async () => {
+		let outcome = await invalidate(registry(undefined), { tenant: 't1', service: 'spaces' });
+		assert.strictEqual(outcome.status, 400);
+		assert.strictEqual(received, null);
+	});
+
+	it("refuses when caching is not configured", async () => {
+		let outcome = await invalidate({ custom: {} }, { tenant: 't1', service: 'spaces' });
+		assert.strictEqual(outcome.status, 400);
+	});
+
+	it("requires tenant and service", async () => {
+		assert.strictEqual((await invalidate(registry('mongo'), null)).status, 400);
+		assert.strictEqual((await invalidate(registry('mongo'), { service: 'spaces' })).status, 400);
+		assert.strictEqual((await invalidate(registry('mongo'), { tenant: 't1' })).status, 400);
+		assert.strictEqual(received, null);
+	});
+
+	it("requires the api as written in the configuration", async () => {
+		assert.strictEqual((await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces', api: '/spaces' })).status, 400);
+		assert.strictEqual((await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces', api: 'POST /spaces' })).status, 400);
+		assert.strictEqual((await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces', user: 42 })).status, 400);
+		assert.strictEqual(received, null);
+	});
+
+	it("answers 500 when the store fails", async () => {
+		failure = new Error("down");
+		let outcome = await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces' });
+		assert.strictEqual(outcome.status, 500);
+	});
+
+	it("answers 500 when the store is not ready", async () => {
+		model.mongo = null;
+		try {
+			let outcome = await invalidate(registry('mongo'), { tenant: 't1', service: 'spaces' });
+			assert.strictEqual(outcome.status, 500);
+		} finally {
+			model.mongo = {};
+		}
+	});
+});

package/test/unit/mw/cache/mongo.js

@@ -0,0 +1,104 @@
+"use strict";
+
+const helper = require("../../../helper.js");
+const model = helper.requireModule('./mw/cache/model/mongo');
+const assert = require('assert');
+
+describe("Unit test for: mw - cache/model/mongo invalidation", () => {
+
+	let calls = null;
+	let invalidation = null;
+	let deleted = 0;
+	let originalMongo = null;
+	let originalLog = null;
+
+	before(() => {
+		originalMongo = model.mongo;
+		originalLog = model.log;
+		model.log = {
+			debug: () => {},
+			info: () => {},
+			error: () => {}
+		};
+		model.mongo = {
+			findOne: async (col, condition) => {
+				calls.push({ "op": "findOne", col, condition });
+				return invalidation;
+			},
+			updateOne: async (col, condition, s, options) => {
+				calls.push({ "op": "updateOne", col, condition, s, options });
+			},
+			deleteMany: async (col, condition) => {
+				calls.push({ "op": "deleteMany", col, condition });
+				return { deletedCount: deleted };
+			}
+		};
+	});
+	after(() => {
+		model.mongo = originalMongo;
+		model.log = originalLog;
+	});
+	beforeEach(() => {
+		calls = [];
+		invalidation = null;
+		deleted = 0;
+	});
+
+	let key = { l1: 'tenant1', l2: 'spaces:GET:/spaces::u:user-1' };
+	let response = { statusCode: 200, headers: {}, body: '[]' };
+
+	it("set stores the fields an invalidation matches on", async () => {
+		let stored = await model.set(key, response, 60000, { service: 'spaces', api: 'GET /spaces', user: 'user-1', startedAt: Date.now() });
+		assert.strictEqual(stored, true);
+		let update = calls.find((c) => c.op === "updateOne");
+		assert.strictEqual(update.col, 'cache_store');
+		assert.strictEqual(update.s.$set.service, 'spaces');
+		assert.strictEqual(update.s.$set.api, 'GET /spaces');
+		assert.strictEqual(update.s.$set.user, 'user-1');
+	});
+
+	it("set looks for an invalidation after the request started, for that api or all, that user or all", async () => {
+		let startedAt = Date.now();
+		await model.set(key, response, 60000, { service: 'spaces', api: 'GET /spaces', user: 'user-1', startedAt: startedAt });
+		let lookup = calls.find((c) => c.op === "findOne");
+		assert.strictEqual(lookup.col, 'cache_invalidation');
+		assert.deepStrictEqual(lookup.condition, {
+			l1: 'tenant1',
+			service: 'spaces',
+			api: { $in: ['GET /spaces', null] },
+			user: { $in: ['user-1', null] },
+			clearedAt: { $gt: new Date(startedAt) }
+		});
+	});
+
+	it("set does not store a response read before an invalidation", async () => {
+		invalidation = { clearedAt: new Date() };
+		let stored = await model.set(key, response, 60000, { service: 'spaces', api: 'GET /spaces', user: 'user-1', startedAt: Date.now() - 1000 });
+		assert.strictEqual(stored, false);
+		assert.strictEqual(calls.filter((c) => c.op === "updateOne").length, 0);
+	});
+
+	it("set without meta stores as before", async () => {
+		let stored = await model.set(key, response, 60000);
+		assert.strictEqual(stored, true);
+		assert.strictEqual(calls.filter((c) => c.op === "findOne").length, 0);
+	});
+
+	it("invalidateMatching records the invalidation before removing", async () => {
+		deleted = 3;
+		let removed = await model.invalidateMatching({ tenant: 'tenant1', service: 'spaces', api: 'GET /spaces', user: 'user-1' });
+		assert.strictEqual(removed, 3);
+		assert.deepStrictEqual(calls.map((c) => c.op), ['updateOne', 'deleteMany']);
+		assert.strictEqual(calls[0].col, 'cache_invalidation');
+		assert.deepStrictEqual(calls[0].condition, { l1: 'tenant1', service: 'spaces', api: 'GET /spaces', user: 'user-1' });
+		assert.ok(calls[0].s.$set.clearedAt instanceof Date);
+		assert.ok(calls[0].s.$set.expiresAt > calls[0].s.$set.clearedAt);
+		assert.deepStrictEqual(calls[1].condition, { l1: 'tenant1', service: 'spaces', api: 'GET /spaces', user: 'user-1' });
+	});
+
+	it("invalidateMatching without api and user covers the whole service", async () => {
+		await model.invalidateMatching({ tenant: 'tenant1', service: 'spaces' });
+		assert.deepStrictEqual(calls[0].condition, { l1: 'tenant1', service: 'spaces', api: null, user: null });
+		assert.deepStrictEqual(calls[1].condition, { l1: 'tenant1', service: 'spaces' });
+	});
+});