@boxyhq/saml-jackson 0.1.3 → 0.1.5-beta.101

package/README.md

@@ -7,12 +7,30 @@ You need someone like Jules Winnfield to save you from the vagaries of SAML logi
 ## Source code visualizer
 [CodeSee codebase visualizer](https://app.codesee.io/maps/public/53e91640-23b5-11ec-a724-79d7dd589517)
 
-# npm library
-Jackson is available as an npm library (https://www.npmjs.com/package/@boxyhq/saml-jackson) that can be integrated into express.js routes. In theory, the library should be usable with other node.js web application frameworks but is currently untested. Please file an issue or submit a PR if you encounter any issues.
+## Getting Started
 
-Refer to https://github.com/boxyhq/jackson#configuration for the configuration options to pass to the library.
+There are two ways to use this repo.
+- As an npm library
+- As a separate service
+
+### Install as an npm library
+Jackson is available as an [npm package](https://www.npmjs.com/package/@boxyhq/saml-jackson) that can be integrated into Express.js routes. library should be usable with other node.js web application frameworks but is currently untested. Please file an issue or submit a PR if you encounter any issues.
+
+```
+npm i @boxyhq/saml-jackson
+```
+
+### Docker
+The docker container can be found at [boxyhq/jackson](https://hub.docker.com/r/boxyhq/jackson/tags). It is preferable to use a specific version instead of the `latest` tag. Jackson uses two ports (configurable if needed, see below) 5000 and 6000. 6000 is the internal port and ideally should not be exposed to a public network.
+
+```
+docker run -p 5000:5000 -p 6000:6000 boxyhq/jackson:78e9099d
+```
+
+### Usage
+
+#### 1. Add Express Routes
 
-Here's how to use the npm library:
 ```
 // express
 const express = require('express');
@@ -94,66 +112,17 @@ router.get('/oauth/userinfo', cors(), async (req, res) => {
 app.user('/sso', router);
 
 ```
-You can also refer to our usage of the library internally in the Jackson service here - https://github.com/boxyhq/jackson/blob/main/src/jackson.js
-
-# Deployment
-The docker container can be found at https://hub.docker.com/r/boxyhq/jackson/tags. It is preferable to use a specific version instead of the `latest` tag. Jackson uses two ports (configurable if needed, see below) 5000 and 6000. 6000 is the internal port and ideally should not be exposed to a public network.
-
-Example of a docker run:
-```
-docker run -p 5000:5000 -p 6000:6000 boxyhq/jackson:78e9099d
-```
-
-# Database Support
-Jackson currently supports SQL databases (Postgres, CockroachDB, MySQL and MariaDB), MongoDB and Redis.
-
-# Configuration
-Configuration is done via env vars (and in the case of the npm library via an options object). The following options are supported and will have to be configured during deployment:
-- HOST_URL: The URL to bind to, defaults to `localhost`
-- HOST_PORT: The port to bind to, defaults to `5000`
-- EXTERNAL_URL (npm: externalUrl): The public URL to reach this service, used internally for documenting the SAML configuration instructions. Defaults to `http://{HOST_URL}:{HOST_PORT}` for Jackson service, required for npm library
-- INTERNAL_HOST_URL: The URL to bind to expose the internal APIs, defaults to `localhost`. Do not configure this to a public network
-- INTERNAL_HOST_PORT: The port to bind to for the internal APIs, defaults to `6000`
-- SAML_AUDIENCE (npm: samlAudience): This is just an identitifer to validate the SAML audience, this value will also get configured in the SAML apps created by your customers. Once set do not change this value unless you get your customers to reconfigure their SAML again. Defaults to `https://saml.boxyhq.com` and is case sensitive. This does not have be a real URL
-- IDP_ENABLED (npm: idpEnabled): Set to `true` to enable IdP initiated login for SAML. SP initiated login is the only recommended flow but you might have to support IdP login at times. Defaults to `false`
-- DB_ENGINE (npm: db.engine): Supported values are `redis`, `sql`, `mongo`, `mem`. Defaults to `sql`
-- DB_URL (npm: db.url): The database URL to connect to, for example `postgres://postgres:postgres@localhost:5450/jackson`
-- DB_TYPE (npm: db.type): Only needed when DB_ENGINE is `sql`. Supported values are `postgres`, `cockroachdb`, `mysql`, `mariadb`. Defaults to `postgres`
-- PRE_LOADED_CONFIG: If you only need a single tenant or a handful of pre-configured tenants then this config will help you read and load SAML configs. It works well with the mem DB engine so you don't have to configure any external databases for this to work (though it works with those as well). This is a path (absolute or relative) to a directory that contains files organized in the format described in the next section.
-
-# Pre-loaded SAML Configuration
-If PRE_LOADED_CONFIG is set then it should point to a directory with the following structure (example below):-
-```
-boxyhq.js
-boxyhq.xml
-anothertenant.js
-anothertenant.xml
-```
-The JS file has the following structure:-
-```
-module.exports = {
-  defaultRedirectUrl: 'http://localhost:3000/login/saml',
-  redirectUrl: '["http://localhost:3000/*", "http://localhost:5000/*"]',
-  tenant: 'boxyhq.com',
-  product: 'demo',
-};
-```
-The XML file (should share the name with the .js file) is the raw XML metadata file you receive from your Identity Provider. Please ensure it is saved in the `utf-8` encoding.
 
-The config and XML above correspond to the `SAML API config` (see below).
-
-# SAML Login flows
-There are two kinds of SAML login flows - SP-initiated and IdP-initiated. We highly recommend sticking to the SP-initiated flow since it is more secure but Jackson also supports the IdP-initiated flow if you enable it. For an in-depth understanding of SAML and the two flows please refer to Okta's comprehensive guide - https://developer.okta.com/docs/concepts/saml/.
-
-# Setting up SAML with your customer's Identity Provider
+#### 2. Setting up SAML with your customer's Identity Provider
 Please follow the instructions here to guide your customer's in setting up SAML correctly for your product(s). You should create a copy of the doc and modify it with your custom settings, we have used the values that work for our demo apps - https://docs.google.com/document/d/1fk---Z9Ln59u-2toGKUkyO3BF6Dh3dscT2u4J2xHANE.
 
-# SAML config API
+#### 3. SAML config API
 Once your customer has set up the SAML app on their Identity Provider, the Identity Provider will generate an IdP or SP metadata file. Some Identity Providers only generate an IdP metadata file but it usually works for the SP login flow as well. It is an XML file that contains various attributes Jackson needs in order to validate incoming SAML login requests. This step is the equivalent of setting an OAuth 2.0 app and generating a client ID and client secret that will be used in the login flow.
 
 You will need to provide a place in the UI for your customers (The account settings page is usually a good place for this) to configure this and then call the API below.
 
 The following API call sets up the configuration in Jackson:
+
 ```
 curl --location --request POST 'http://localhost:6000/api/v1/saml/config' \
 --header 'Content-Type: application/x-www-form-urlencoded' \
@@ -167,18 +136,19 @@ curl --location --request POST 'http://localhost:6000/api/v1/saml/config' \
 - rawMetadata: The XML metadata file your customer gets from their Identity Provider
 - defaultRedirectUrl: The redirect URL to use in the IdP login flow. Jackson will call this URL after completing an IdP login flow
 - redirectUrl: JSON encoded array containing a list of allowed redirect URLs. Jackson will disallow any redirects not on this list (or not the default URL above)
-- tenant: Jackson supports a multi-tenant architecture, this is a unique identifier you set from your side that relates back to your customer's tenant. This is normally an email, domain, an account id, or user id
+- tenant: Jackson supports a multi-tenant architecture, this is a unique identifier you set from your side that relates back to your customer's tenant. This is normally an email, domain, an account id, or user-id
 - product: Jackson support multiple products, this is a unique identifier you set from your side that relates back to the product your customer is using
 
 The response returns a JSON with `client_id` and `client_secret` that can be stored against your tenant and product for a more secure OAuth 2.0 flow. If you do not want to store the `client_id` and `client_secret` you can alternatively use `client_id=tentant=<tenantID>&product=<productID>` and any arbitrary value for `client_secret` when setting up the OAuth 2.0 flow.
 
-# OAuth 2.0 Flow
+#### 4. OAuth 2.0 Flow
 Jackson has been designed to abstract the SAML login flow as a pure OAuth 2.0 flow. This means it's compatible with any standard OAuth 2.0 library out there, both client-side and server-side. It is important to remember that SAML is configured per customer unlike OAuth 2.0 where you can have a single OAuth app supporting logins for all customers.
 
 Jackson also supports the PKCE authorization flow (https://oauth.net/2/pkce/), so you can protect your SPAs.
 
 If for any reason you need to implement the flow on your own, the steps are outlined below:
-## Authorize
+
+#### 5. Authorize
 The OAuth flow begins with redirecting your user to the `authorize` URL:
 ```
 https://localhost:5000/oauth/authorize
@@ -193,7 +163,7 @@ https://localhost:5000/oauth/authorize
 - redirect_uri: This is where the user will be taken back once the authorization flow is complete
 - state: Use a randomly generated string as the state, this will be echoed back as a query parameter when taking the user back to the `redirect_uri` above. You should validate the state to prevent XSRF attacks
 
-## Code exchange
+#### 6. Code Exchange
 After successful authorization, the user is redirected back to the `redirect_uri`. The query parameters will include the `code` and `state` parameters. You should validate that the state matches the one you sent in the authorize request.
 
 The code can then be exchanged for a token by making the following request:
@@ -221,7 +191,7 @@ If everything goes well you should receive a JSON response that includes the acc
 }
 ```
 
-## Profile request
+#### 7. Profile Request
 The short-lived access token can now be used to request the user's profile. You'll need to make the following request:
 ```
 curl --request GET \
@@ -244,3 +214,78 @@ If everything goes well you should receive a JSON response with the user's profi
 - id: The id of the user as provided by the Identity Provider
 - firstName: The first name of the user as provided by the Identity Provider
 - lastName: The last name of the user as provided by the Identity Provider
+
+## Examples
+To Do
+
+## Database Support
+Jackson currently supports the following databases.
+
+- Postgres
+- CockroachDB
+- MySQL
+- MariaDB
+- MongoDB
+- Redis
+
+## Configuration
+Configuration is done via env vars (and in the case of the npm library via an options object). 
+
+The following options are supported and will have to be configured during deployment.
+
+| Key                               | Description                                                                                                                                                                                                                                                                                                                                                                                                          | Default                         |
+|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|
+| HOST_URL                          | The URL to bind to                                                                                                                                                                                                                                                                                                                                                                                                   | `localhost`                     |
+| HOST_PORT                         | The port to bind to                                                                                                                                                                                                                                                                                                                                                                                                  | `5000`                          |
+| EXTERNAL_URL (npm: externalUrl)   | The public URL to reach this service, used internally for documenting the SAML configuration instructions.                                                                                                                                                                                                                                                                                                           | `http://{HOST_URL}:{HOST_PORT}` |
+| INTERNAL_HOST_URL                 | The URL to bind to expose the internal APIs. Do not configure this to a public network.                                                                                                                                                                                                                                                                                                                              | `localhost`                     |
+| INTERNAL_HOST_PORT                | The port to bind to for the internal APIs.                                                                                                                                                                                                                                                                                                                                                                           | `6000`                          |
+| SAML_AUDIENCE (npm: samlAudience) | This is just an identifier to validate the SAML audience, this value will also get configured in the SAML apps created by your customers.  Once set do not change this value unless you get your customers to reconfigure their SAML again. It is case-sensitive. This does not have to be a real URL.                                                                                                               | `https://saml.boxyhq.com`       |
+| IDP_ENABLED (npm: idpEnabled)     | Set to `true` to enable IdP initiated login for SAML. SP initiated login is the only recommended flow but you might have to support IdP login at times.                                                                                                                                                                                                                                                              | `false`                         |
+| DB_ENGINE (npm: db.engine)        | Supported values are `redis`, `sql`, `mongo`, `mem`.                                                                                                                                                                                                                                                                                                                                                                 | `sql`                           |
+| DB_URL (npm: db.url)              | The database URL to connect to. For example `postgres://postgres:postgres@localhost:5450/jackson`                                                                                                                                                                                                                                                                                                                    |                                 |
+| DB_TYPE (npm: db.type)            | Only needed when DB_ENGINE is `sql`. Supported values are `postgres`, `cockroachdb`, `mysql`, `mariadb`.                                                                                                                                                                                                                                                                                                             | `postgres`                      |
+| PRE_LOADED_CONFIG                 | If you only need a single tenant or a handful of pre-configured tenants then this config will help you read and load SAML configs. It works well with the mem DB engine so you don't have to configure any external databases for this to work (though it works with those as well). This is a path (absolute or relative) to a directory that contains files organized in the format described in the next section. |                                 |
+
+## Pre-loaded SAML Configuration
+If PRE_LOADED_CONFIG is set then it should point to a directory with the following structure (example below):-
+```
+boxyhq.js
+boxyhq.xml
+anothertenant.js
+anothertenant.xml
+```
+The JS file has the following structure:-
+```
+module.exports = {
+  defaultRedirectUrl: 'http://localhost:3000/login/saml',
+  redirectUrl: '["http://localhost:3000/*", "http://localhost:5000/*"]',
+  tenant: 'boxyhq.com',
+  product: 'demo',
+};
+```
+The XML file (should share the name with the .js file) is the raw XML metadata file you receive from your Identity Provider. Please ensure it is saved in the `utf-8` encoding.
+
+The config and XML above correspond to the `SAML API config` (see below).
+
+## SAML Login flows
+There are two kinds of SAML login flows - SP-initiated and IdP-initiated. We highly recommend sticking to the SP-initiated flow since it is more secure but Jackson also supports the IdP-initiated flow if you enable it. For an in-depth understanding of SAML and the two flows please refer to Okta's comprehensive guide - https://developer.okta.com/docs/concepts/saml/.
+
+## Contributing
+Thanks for taking the time to contribute! Contributions are what makes the open-source community such an amazing place to learn, inspire, and create. Any contributions you make will benefit everybody else and are appreciated.
+
+Please try to create bug reports that are:
+
+- _Reproducible._ Include steps to reproduce the problem.
+- _Specific._ Include as much detail as possible: which version, what environment, etc.
+- _Unique._ Do not duplicate existing opened issues.
+- _Scoped to a Single Bug._ One bug per report.
+
+## Support
+Reach out to the maintainer at one of the following places:
+
+- [GitHub Issues](https://github.com/boxyhq/jackson/issues)
+- The email which is located [in GitHub profile](https://github.com/deepakprabhakara)
+
+## License
+[Apache 2.0 License](https://github.com/boxyhq/jackson/blob/main/LICENSE)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@boxyhq/saml-jackson",
-  "version": "0.1.3",
+  "version": "0.1.5-beta.101",
   "license": "Apache 2.0",
   "description": "SAML 2.0 service",
   "main": "src/index.js",
@@ -20,7 +20,7 @@
     "calendso": "cross-env DB_URL=postgresql://postgres:postgres@localhost:5450/calendso nodemon src/jackson.js",
     "mongo": "cross-env DB_ENGINE=mongo DB_URL=mongodb://localhost:27017/jackson nodemon src/jackson.js",
     "pre-loaded": "cross-env DB_ENGINE=mem PRE_LOADED_CONFIG='./_config' nodemon src/jackson.js",
-    "test": "tap src/**/*.test.js",
+    "test": "tap --timeout=100 src/**/*.test.js",
     "dev-dbs": "docker-compose -f ./_dev/docker-compose.yml up -d"
   },
   "tap": {
@@ -32,27 +32,27 @@
   },
   "dependencies": {
     "@boxyhq/saml20": "0.2.0",
-    "@peculiar/webcrypto": "1.1.7",
-    "@peculiar/x509": "1.4.1",
+    "@peculiar/webcrypto": "1.2.2",
+    "@peculiar/x509": "1.6.0",
     "cors": "2.8.5",
     "express": "4.17.1",
-    "mongodb": "4.1.3",
-    "mysql2": "^2.3.3-rc.0",
+    "mongodb": "4.2.0",
+    "mysql2": "2.3.3",
     "pg": "8.7.1",
     "rambda": "6.9.0",
-    "redis": "4.0.0-rc.3",
+    "redis": "4.0.0-rc.4",
     "reflect-metadata": "0.1.13",
     "ripemd160": "2.0.2",
     "thumbprint": "0.0.1",
-    "typeorm": "0.2.38",
+    "typeorm": "0.2.41",
     "xml-crypto": "2.1.3",
     "xml2js": "0.4.23",
     "xmlbuilder": "15.1.1"
   },
   "devDependencies": {
     "cross-env": "7.0.3",
-    "eslint": "^8.2.0",
-    "nodemon": "2.0.13",
-    "tap": "15.0.10"
+    "eslint": "8.3.0",
+    "nodemon": "2.0.15",
+    "tap": "15.1.2"
   }
 }

package/src/controller/oauth.js

@@ -28,10 +28,16 @@ const extractBearerToken = (req) => {
 function getEncodedClientId(client_id) {
   try {
     const sp = new URLSearchParams(client_id);
-    return {
-      tenant: sp.get('tenant'),
-      product: sp.get('product'),
-    };
+    const tenant = sp.get('tenant');
+    const product = sp.get('product');
+    if (tenant && product) {
+      return {
+        tenant: sp.get('tenant'),
+        product: sp.get('product'),
+      };
+    }
+
+    return null;
   } catch (err) {
     return null;
   }
@@ -287,7 +293,7 @@ const token = async (req, res) => {
   res.json({
     access_token: token,
     token_type: 'bearer',
-    expires_in: 300,
+    expires_in: options.db.ttl,
   });
 };
 

package/src/db/db.js

@@ -19,7 +19,7 @@ class DB {
 
   // ttl is in seconds
   async put(namespace, key, val, ttl = 0, ...indexes) {
-    if (this.ttl > 0 && indexes && indexes.length > 0) {
+    if (ttl > 0 && indexes && indexes.length > 0) {
       throw new Error('secondary indexes not allow on a store with ttl');
     }
 

package/src/db/db.test.js

@@ -3,6 +3,8 @@ const t = require('tap');
 const DB = require('./db.js');
 
 let configStores = [];
+let ttlStores = [];
+const ttl = 3;
 
 const record1 = {
   id: '1',
@@ -18,48 +20,46 @@ const record2 = {
 const dbs = [
   {
     engine: 'mem',
-    options: {},
+    ttl: 1,
   },
   {
     engine: 'redis',
-    options: { url: 'redis://localhost:6379' },
+    url: 'redis://localhost:6379',
   },
   {
     engine: 'sql',
-    options: {
-      url: 'postgresql://postgres:postgres@localhost:5432/postgres',
-      type: 'postgres',
-    },
+    url: 'postgresql://postgres:postgres@localhost:5432/postgres',
+    type: 'postgres',
+    ttl: 1,
+    limit: 1,
   },
   {
     engine: 'mongo',
-    options: { url: 'mongodb://localhost:27017/jackson' },
+    url: 'mongodb://localhost:27017/jackson',
   },
   {
     engine: 'sql',
-    options: {
-      url: 'mysql://root:mysql@localhost:3307/mysql',
-      type: 'mysql',
-    },
+    url: 'mysql://root:mysql@localhost:3307/mysql',
+    type: 'mysql',
+    ttl: 1,
+    limit: 1,
   },
   {
     engine: 'sql',
-    options: {
-      url: 'mariadb://root@localhost:3306/mysql',
-      type: 'mariadb',
-    },
+    url: 'mariadb://root@localhost:3306/mysql',
+    type: 'mariadb',
+    ttl: 1,
+    limit: 1,
   },
 ];
 
 t.before(async () => {
   for (const idx in dbs) {
-    const config = dbs[idx];
-    const engine = config.engine;
-    const opts = config.options;
-    opts.engine = engine;
+    const opts = dbs[idx];
     const db = await DB.new(opts);
 
     configStores.push(db.store('saml:config'));
+    ttlStores.push(db.store('oauth:session', ttl));
   }
 });
 
@@ -70,9 +70,10 @@ t.teardown(async () => {
 t.test('dbs', ({ end }) => {
   for (const idx in configStores) {
     const configStore = configStores[idx];
+    const ttlStore = ttlStores[idx];
     let dbEngine = dbs[idx].engine;
-    if (dbs[idx].options.type) {
-      dbEngine += ': ' + dbs[idx].options.type;
+    if (dbs[idx].type) {
+      dbEngine += ': ' + dbs[idx].type;
     }
     t.test('put(): ' + dbEngine, async (t) => {
       await configStore.put(
@@ -109,8 +110,8 @@ t.test('dbs', ({ end }) => {
     });
 
     t.test('get(): ' + dbEngine, async (t) => {
-      const ret1 = await configStore.get('1');
-      const ret2 = await configStore.get('2');
+      const ret1 = await configStore.get(record1.id);
+      const ret2 = await configStore.get(record2.id);
 
       t.same(ret1, record1, 'unable to get record1');
       t.same(ret2, record2, 'unable to get record2');
@@ -151,8 +152,8 @@ t.test('dbs', ({ end }) => {
 
       await configStore.delete(record2.id);
 
-      const ret1 = await configStore.get('1');
-      const ret2 = await configStore.get('2');
+      const ret1 = await configStore.get(record1.id);
+      const ret2 = await configStore.get(record2.id);
 
       const ret3 = await configStore.getByIndex({
         name: 'name',
@@ -171,6 +172,69 @@ t.test('dbs', ({ end }) => {
 
       t.end();
     });
+
+    t.test('ttl indexes: ' + dbEngine, async (t) => {
+      try {
+        await ttlStore.put(
+          record1.id,
+          record1,
+          {
+            // secondary index on city
+            name: 'city',
+            value: record1.city,
+          },
+          {
+            // secondary index on name
+            name: 'name',
+            value: record1.name,
+          }
+        );
+
+        t.fail('expecting a secondary indexes not allow on a store with ttl');
+      } catch (err) {
+        t.ok(err, 'got expected error');
+      }
+
+      t.end();
+    });
+
+    t.test('ttl put(): ' + dbEngine, async (t) => {
+      await ttlStore.put(record1.id, record1);
+
+      await ttlStore.put(record2.id, record2);
+
+      t.end();
+    });
+
+    t.test('ttl get(): ' + dbEngine, async (t) => {
+      const ret1 = await ttlStore.get(record1.id);
+      const ret2 = await ttlStore.get(record2.id);
+
+      t.same(ret1, record1, 'unable to get record1');
+      t.same(ret2, record2, 'unable to get record2');
+
+      t.end();
+    });
+
+    t.test('ttl expiry: ' + dbEngine, async (t) => {
+      // mongo runs ttl task every 60 seconds
+      if (dbEngine.startsWith('mongo')) {
+        t.end();
+        return;
+      }
+
+      await new Promise((resolve) =>
+        setTimeout(resolve, ((dbEngine === 'mem' ? 5 : 0) + ttl + 0.5) * 1000)
+      );
+
+      const ret1 = await ttlStore.get(record1.id);
+      const ret2 = await ttlStore.get(record2.id);
+
+      t.same(ret1, null, 'ttl for record1 failed');
+      t.same(ret2, null, 'ttl for record2 failed');
+
+      t.end();
+    });
   }
 
   t.test('db.new() error', async (t) => {

package/src/db/mem.js

@@ -2,14 +2,32 @@
 const dbutils = require('./utils.js');
 
 class Mem {
-  constructor(/*options*/) {
+  constructor(options) {
     return (async () => {
       this.store = {}; // map of key, value
       this.indexes = {}; // map of key, Set
       this.cleanup = {}; // map of indexes for cleanup when store key is deleted
       this.ttlStore = {}; // map of key to ttl
 
-      return this; // Return the newly-created instance
+      if (options.ttl) {
+        this.ttlCleanup = async () => {
+          const now = Date.now();
+          for (const k in this.ttlStore) {
+            if (this.ttlStore[k].expiresAt < now) {
+              await this.delete(
+                this.ttlStore[k].namespace,
+                this.ttlStore[k].key
+              );
+            }
+          }
+
+          this.timerId = setTimeout(this.ttlCleanup, options.ttl * 1000);
+        };
+
+        this.timerId = setTimeout(this.ttlCleanup, options.ttl * 1000);
+      }
+
+      return this;
     })();
   }
 
@@ -39,8 +57,11 @@ class Mem {
     this.store[k] = JSON.stringify(val);
 
     if (ttl) {
-      // TODO: implement cleanup of TTL
-      this.ttlStore[k] = Date.now() + ttl * 1000;
+      this.ttlStore[k] = {
+        namespace,
+        key,
+        expiresAt: Date.now() + ttl * 1000,
+      };
     }
 
     // no ttl support for secondary indexes
@@ -79,6 +100,7 @@ class Mem {
     }
 
     delete this.cleanup[idxKey];
+    delete this.ttlStore[k];
   }
 }
 

package/src/db/sql/entity/JacksonIndex.js

@@ -13,9 +13,11 @@ module.exports = new EntitySchema({
     },
     key: {
       type: 'varchar',
+      length: 1500,
     },
     storeKey: {
       type: 'varchar',
+      length: 1500,
     }
   },
   relations: {

package/src/db/sql/entity/JacksonStore.js

@@ -1,20 +1,36 @@
 const EntitySchema = require('typeorm').EntitySchema;
 const JacksonStore = require('../model/JacksonStore.js');
 
-module.exports = new EntitySchema({
-  name: 'JacksonStore',
-  target: JacksonStore,
-  columns: {
-    key: {
-      primary: true,
-      type: 'varchar',
-    },
-    value: {
-      type: 'varchar',
+const valueType = (type) => {
+  switch (type) {
+    case 'postgres':
+    case 'cockroachdb':
+      return 'text';
+    case 'mysql':
+    case 'mariadb':
+      return 'mediumtext';
+    default:
+      return 'varchar';
+  }
+};
+
+module.exports = (type) => {
+  return new EntitySchema({
+    name: 'JacksonStore',
+    target: JacksonStore,
+    columns: {
+      key: {
+        primary: true,
+        type: 'varchar',
+        length: 1500,
+      },
+      value: {
+        type: valueType(type),
+      },
+      expiresAt: {
+        type: 'bigint',
+        nullable: true,
+      },
     },
-    expiresAt: {
-      type: 'bigint',
-      nullable: true,
-    }
-  },
-});
+  });
+};

package/src/db/sql/sql.js

@@ -8,22 +8,60 @@ const dbutils = require('../utils.js');
 class Sql {
   constructor(options) {
     return (async () => {
-      this.connection = await typeorm.createConnection({
-        name: options.type,
-        type: options.type,
-        url: options.url,
-        synchronize: true,
-        logging: false,
-        entities: [
-          require('./entity/JacksonStore.js'),
-          require('./entity/JacksonIndex.js'),
-        ],
-      });
+      while (true) {
+        try {
+          this.connection = await typeorm.createConnection({
+            name: options.type,
+            type: options.type,
+            url: options.url,
+            synchronize: true,
+            migrationsTableName: '_jackson_migrations',
+            logging: false,
+            entities: [
+              require('./entity/JacksonStore.js')(options.type),
+              require('./entity/JacksonIndex.js'),
+            ],
+          });
+
+          break;
+        } catch (err) {
+          console.error(`error connecting to ${options.type} db: ${err}`);
+          await dbutils.sleep(1000);
+          continue;
+        }
+      }
 
       this.storeRepository = this.connection.getRepository(JacksonStore);
       this.indexRepository = this.connection.getRepository(JacksonIndex);
 
-      return this; // Return the newly-created instance
+      if (options.ttl && options.limit) {
+        this.ttlCleanup = async () => {
+          const now = Date.now();
+
+          while (true) {
+            const ids = await this.storeRepository.find({
+              expiresAt: typeorm.MoreThan(now),
+              take: options.limit,
+            });
+
+            if (ids.length <= 0) {
+              break;
+            }
+
+            await this.storeRepository.remove(ids);
+          }
+
+          this.timerId = setTimeout(this.ttlCleanup, options.ttl * 1000);
+        };
+
+        this.timerId = setTimeout(this.ttlCleanup, options.ttl * 1000);
+      } else {
+        console.log(
+          'Warning: ttl cleanup not enabled, set both "ttl" and "limit" options to enable it!'
+        );
+      }
+
+      return this;
     })();
   }
 

package/src/db/utils.js

@@ -16,14 +16,15 @@ const keyFromParts = (...parts) => {
   return parts.join(':'); // TODO: pick a better strategy, keys can collide now
 };
 
+const sleep = (ms) => {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+};
+
 module.exports = {
   key,
-
   keyForIndex,
-
   keyDigest,
-
   keyFromParts,
-
+  sleep,
   indexPrefix: '_index',
 };

package/src/env.js

@@ -15,6 +15,7 @@ const db = {
   engine: process.env.DB_ENGINE,
   url: process.env.DB_URL,
   type: process.env.DB_TYPE,
+  ttl: process.env.DB_TTL,
 };
 
 module.exports = {

package/src/index.js

@@ -21,6 +21,8 @@ const defaultOpts = (opts) => {
   newOpts.db.url =
     newOpts.db.url || 'postgres://postgres:postgres@localhost:5432/jackson';
   newOpts.db.type = newOpts.db.type || 'postgres'; // Only needed if DB_ENGINE is sql. Supported values: postgres, cockroachdb, mysql, mariadb
+  newOpts.db.ttl = (newOpts.db.ttl || 300) * 1; // TTL for the code, session and token stores (in seconds)
+  newOpts.db.limit = (newOpts.db.limit || 1000) * 1; // Limit ttl cleanup to this many items at a time
 
   return newOpts;
 };
@@ -30,9 +32,9 @@ module.exports = async function (opts) {
 
   const db = await DB.new(opts.db);
   const configStore = db.store('saml:config');
-  const sessionStore = db.store('oauth:session', 300);
-  const codeStore = db.store('oauth:code', 300);
-  const tokenStore = db.store('oauth:token', 300);
+  const sessionStore = db.store('oauth:session', opts.db.ttl);
+  const codeStore = db.store('oauth:code', opts.db.ttl);
+  const tokenStore = db.store('oauth:token', opts.db.ttl);
 
   const apiController = require('./controller/api.js')({ configStore });
   const oauthController = require('./controller/oauth.js')({
@@ -54,7 +56,8 @@ module.exports = async function (opts) {
     }
   }
 
-  console.log(`Using engine: ${opts.db.engine}`);
+  const type = opts.db.type ? ' Type: ' + opts.db.type : '';
+  console.log(`Using engine: ${opts.db.engine}.${type}`);
 
   return {
     apiController,

package/src/jackson.js

@@ -78,7 +78,7 @@ internalApp.post(apiPath + '/config', async (req, res) => {
 
 let internalServer = server;
 if (env.useInternalServer) {
-  internalServer = internalApp.listen(env.internalPort, async () => {
+  internalServer = internalApp.listen(env.internalHostPort, async () => {
     console.log(
       `🚀 The path of the righteous internal server: http://${env.internalHostUrl}:${env.internalHostPort}`
     );