imapkit 5.1.0 → 5.3.0

package/README.md

@@ -27,7 +27,7 @@ imapkit -p 1143
 
 Point your IMAP client to `localhost:1143` and log in with user name `testuser` and password `testpass`. Without `-p` the server listens on port 143 (993 with `--secure`), which usually needs root privileges.
 
-`imapkit --smtpPort=1025` also starts an SMTP server that appends every message it receives to INBOX. SMTP needs the optional [smtp-server](https://www.npmjs.com/package/smtp-server) package, which is not installed with ImapKit: `npm install -g smtp-server` (or `npm install smtp-server` next to a local install).
+`imapkit -p 1143 --tls-port=1993` also listens for implicit TLS (IMAPS) connections on port 1993, both ports serve the same mailboxes. `imapkit --smtpPort=1025` also starts an SMTP server that appends every message it receives to INBOX. SMTP needs the optional [smtp-server](https://www.npmjs.com/package/smtp-server) package, which is not installed with ImapKit: `npm install -g smtp-server` (or `npm install smtp-server` next to a local install).
 
 Run `imapkit --help` to see all command line options, the environment variables (`IMAPKIT_PORT`, `IMAPKIT_PLUGINS`, ...) and sample configuration data. For example, `imapkit -p 1143 --plugin=IDLE,MOVE,CONDSTORE --storage=storage.json` loads three plugins and the mailboxes of `storage.json`.
 
@@ -142,6 +142,21 @@ ImapKit follows these of the strategies that RFC 2180 (IMAP4 Multi-Accessed Mail
 
 By default the only account is user name `"testuser"` with password `"testpass"` (and access token `"testtoken"` for XOAUTH2 and OAUTHBEARER). The `users` option replaces the default account list, for example `{ "testuser": { "password": "testpass" }, "otheruser": { "password": "secret" } }`, and `server.control.addUser()` adds users at runtime (see [Control API](#control-api)). All users share the same mailbox tree.
 
+## OAuth 2.0 and OpenID Connect
+
+The `oauth` option (`--oauth-port`) starts a test identity provider, so a mail client can run its whole OAuth sign-in against ImapKit: the authorization code flow with PKCE, a consent page that needs no password, and tokens that log in to IMAP (XOAUTH2, OAUTHBEARER) and SMTP (AUTH XOAUTH2), expire on the server clock, refresh and can be revoked.
+
+```javascript
+const server = imapkit({
+    plugins: ['SASL-IR', 'XOAUTH2', 'OAUTHBEARER'],
+    users: { 'alice@example.com': { password: 'pw', displayName: 'Alice' }, 'denied@example.com': { password: 'pw', oauth: { consent: 'deny' } } },
+    oauth: { port: 8085 }
+});
+await server.start(1143); // discovery: http://127.0.0.1:8085/.well-known/openid-configuration
+```
+
+A consent policy decides every sign-in, from the first that has one: a queued decision (`control.queueOAuthDecision()`), the user that `login_hint` names, the client, the server default. `prompt` shows the consent page (pick an account, Allow or Deny), `approve` and `deny` redirect right away with a plain HTTP 302, so a test can follow the redirect without a browser, and an error object answers with that error. The clients `imapkit`, `imapkit-approve` and `imapkit-deny` are registered by default, with loopback redirect URIs on any port. The ID token's `email` claim is the user name, so use address-style user names for OAuth tests. The provider is strict like the IMAP side: PKCE S256 for public clients, single-use codes, exact redirect URIs, no unknown scopes. Token faults, forced expiry and revocation are in the control API and the REST API. See [OAuth 2.0 and OpenID Connect](https://imapkit.com/docs/guides/oauth).
+
 ## Status
 
 ### IMAP4rev1
@@ -444,7 +459,7 @@ Plugins add operations for their own data, they exist only when the plugin is lo
 
 A plugin of your own adds operations with `server.control.register(name, fn, routes)`, the routes go to the REST API.
 
-`server.start(port, host)` resolves with the port, `server.stop()` closes the server and every session. `listen()` and `close()` still work. With the `smtp` option (`imapkit({ smtp: { port, host } })`) `start()` also starts an SMTP server that appends every message it receives to INBOX, `server.smtpServer` is that server. It needs the optional `smtp-server` package (`npm install smtp-server`), `start()` rejects with an error that says so when it is missing.
+`server.start(port, host)` resolves with the port, `server.stop()` closes the server and every session. `listen()` and `close()` still work. With the `smtp` option (`imapkit({ smtp: { port, host } })`) `start()` also starts an SMTP server that appends every message it receives to INBOX, `server.smtpServer` is that server. It needs the optional `smtp-server` package (`npm install smtp-server`), `start()` rejects with an error that says so when it is missing. With the `tls` option (`imapkit({ tls: { port, host } })`) `start()` also listens for implicit TLS connections, next to the plaintext port, so one server answers on 143 and 993 alike; `server.tlsServer` is that listener.
 
 ### Events
 
@@ -453,8 +468,10 @@ Tests can wait for events instead of polling:
 - `session`: `{ type, session }`, `type` is `open`, `login`, `select` (also when the same mailbox is selected again), `unselect` (CLOSE, UNSELECT, a failed SELECT, BYE), `logout` (UNAUTHENTICATE), `waiting` (a command waits for client input, with `command`, e.g. `IDLE` after its continuation) or `close`, `session` is the session as `sessions()` describes it.
 - `command`: `{ session, tag, command, status, user }` when the tagged response of a command goes out, `status` is `OK`, `NO` or `BAD`. A line that is refused before it runs (it does not parse, or it is pipelined ambiguously) and a command that a script rule answers have no `command` event.
 - `mailbox`: `{ type, path, oldPath, mailbox, origin }` for CREATE, DELETE, RENAME, SUBSCRIBE and UNSUBSCRIBE, `origin` is null for the control API.
+- `append`: `(mailbox, messages, origin)` when a message is added, by APPEND, COPY, MOVE, REPLACE, SMTP or the control API, before the sessions are told.
 - `expunge`: `(mailbox, messages, origin)` before the sessions are told, and `flags`: `(mailbox, messages, origin)` for flag changes of the control API.
 - `acl`: `(mailbox, previousAcl)` when the ACL of a mailbox changes (ACL plugin), `script`: `{ rule, event, session, tag, command }` when a script rule fires, `reset`: after `control.reset()`.
+- `trace`: `{ session, direction, data }` for the protocol traffic of a session, `direction` is `in` or `out` and `data` a Buffer. It is the IMAP layer above TLS and COMPRESS, so a log shows plaintext commands and responses. Nothing is traced while no one listens.
 
 ```javascript
 const idling = new Promise(resolve => server.on('session', event => event.type === 'waiting' && event.command === 'IDLE' && resolve(event)));
@@ -478,7 +495,7 @@ The REST API controls the whole server, including the mail of every user, so it
 
 Requests and responses are JSON, and every POST needs `Content-Type: application/json`, also without a body (HTTP 415 otherwise). Errors are `{ "error": { "code", "message" } }` with HTTP status 404 for `NONEXISTENT`, 409 for `ALREADYEXISTS` and the response codes of a failed mailbox operation (`CANNOT`, `HASCHILDREN` ...), 400 for `INVALID`, 401 for a missing or wrong token, 404 `NOTFOUND` and 405 `METHOD` for an unknown route or method, 413 `TOOBIG` for a body over 64 MiB (and `addMessage` checks refused by APPENDLIMIT), 500 for a server error. A mailbox is its storage name (modified UTF-7) in the URL, URL encoded with a `/` in the name as `%2F` (`/v1/mailboxes/Work%2FProjects`). Message sources are base64 in responses (`"encoding": "base64"`), a request sends `raw` as text or as base64 with `"encoding": "base64"`. `GET /v1/openapi.json` describes every endpoint.
 
-`GET /v1/events` streams the [events](#events) as [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) (`event: <type>` with JSON `data`), so a test in another language can wait for "the client selected INBOX" or "a script rule fired" without polling. `?types=session,command` chooses the types (`session`, `command`, `mailbox`, `expunge`, `flags`, `acl`, `script`, `reset`), mailboxes are paths, messages UIDs and sessions numbers in the data:
+`GET /v1/events` streams the [events](#events) as [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) (`event: <type>` with JSON `data`), so a test in another language can wait for "the client selected INBOX" or "a script rule fired" without polling. `?types=session,command` chooses the types (`session`, `command`, `mailbox`, `append`, `expunge`, `flags`, `acl`, `script`, `reset`, `oauth`, and `trace`, which is only sent when asked for, with base64 `data`), mailboxes are paths, messages UIDs and sessions numbers in the data:
 
 ```bash
 curl -N 'http://127.0.0.1:8143/v1/events?types=session'
@@ -486,26 +503,27 @@ curl -N 'http://127.0.0.1:8143/v1/events?types=session'
 # data: {"type":"select","session":{"session":1,"user":"testuser","state":"Selected","mailbox":"INBOX",...}}
 ```
 
-| Endpoint                                                                                             | Control API                                                                            |
-| ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
-| `GET /v1/snapshot`                                                                                   | `snapshot()`                                                                           |
-| `POST /v1/reset`                                                                                     | `reset()`                                                                              |
-| `POST /v1/shutdown` (`{ graceful }`)                                                                 | `shutdown()`                                                                           |
-| `GET /v1/sessions`, `DELETE /v1/sessions/{n}` (`{ text, reset }`)                                    | `sessions()`, `disconnect()`                                                           |
-| `POST /v1/sessions/{n}/inject` (`{ data, encoding }`)                                                | `inject()`                                                                             |
-| `GET`, `POST /v1/users`, `PUT`, `DELETE /v1/users/{name}`                                            | `listUsers()`, `addUser()`, `updateUser()`, `deleteUser()`                             |
-| `GET`, `POST /v1/mailboxes`, `GET`, `DELETE /v1/mailboxes/{path}`                                    | `listMailboxes()`, `createMailbox()`, `getMailbox()`, `deleteMailbox()`                |
-| `POST /v1/mailboxes/{path}/rename` (`{ newPath }`)                                                   | `renameMailbox()`                                                                      |
-| `PUT`, `DELETE /v1/mailboxes/{path}/subscription`                                                    | `subscribe()`, `unsubscribe()`                                                         |
-| `POST /v1/mailboxes/{path}/uidvalidity`                                                              | `resetUidValidity()`                                                                   |
-| `GET /v1/mailboxes/{path}/messages` (`?uids=1,2&raw=true`), `POST` (`{ raw, flags }`)                | `listMessages()`, `addMessage()`                                                       |
-| `GET`, `DELETE /v1/mailboxes/{path}/messages/{uid}`                                                  | `getMessage()`, `expungeMessages()`                                                    |
-| `POST /v1/mailboxes/{path}/messages/flags`, `.../expunge`, `.../copy`, `.../move`                    | `setFlags()`, `expungeMessages()`, `copyMessages()`, `moveMessages()`                  |
-| `GET /v1/mailboxes/{path}/acl`, `PUT`, `DELETE /v1/mailboxes/{path}/acl/{identifier}` (`{ rights }`) | `getAcl()`, `setAcl()`, `deleteAcl()` (ACL)                                            |
-| `GET`, `PUT /v1/quota`                                                                               | `getQuota()`, `setQuota()` (QUOTA)                                                     |
-| `GET`, `PUT /v1/metadata`, `GET`, `PUT /v1/mailboxes/{path}/metadata`                                | `getMetadata()`, `setMetadata()` (METADATA)                                            |
-| `PUT /v1/mailboxes/{path}/special-use` (`{ specialUse }`)                                            | `setSpecialUse()` (SPECIAL-USE)                                                        |
-| `GET`, `POST`, `DELETE /v1/script/rules`, `DELETE /v1/script/rules/{id}`                             | [script rules](#scripted-faults) in their JSON form, so runtime faults need no restart |
+| Endpoint                                                                                                                       | Control API                                                                            |
+| ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
+| `GET /v1/snapshot`                                                                                                             | `snapshot()`                                                                           |
+| `POST /v1/reset`                                                                                                               | `reset()`                                                                              |
+| `POST /v1/shutdown` (`{ graceful }`)                                                                                           | `shutdown()`                                                                           |
+| `GET /v1/sessions`, `DELETE /v1/sessions/{n}` (`{ text, reset }`)                                                              | `sessions()`, `disconnect()`                                                           |
+| `POST /v1/sessions/{n}/inject` (`{ data, encoding }`)                                                                          | `inject()`                                                                             |
+| `GET`, `POST /v1/users`, `PUT`, `DELETE /v1/users/{name}`                                                                      | `listUsers()`, `addUser()`, `updateUser()`, `deleteUser()`                             |
+| `GET`, `POST /v1/mailboxes`, `GET`, `DELETE /v1/mailboxes/{path}`                                                              | `listMailboxes()`, `createMailbox()`, `getMailbox()`, `deleteMailbox()`                |
+| `POST /v1/mailboxes/{path}/rename` (`{ newPath }`)                                                                             | `renameMailbox()`                                                                      |
+| `PUT`, `DELETE /v1/mailboxes/{path}/subscription`                                                                              | `subscribe()`, `unsubscribe()`                                                         |
+| `POST /v1/mailboxes/{path}/uidvalidity`                                                                                        | `resetUidValidity()`                                                                   |
+| `GET /v1/mailboxes/{path}/messages` (`?uids=1,2&raw=true`), `POST` (`{ raw, flags }`)                                          | `listMessages()`, `addMessage()`                                                       |
+| `GET`, `DELETE /v1/mailboxes/{path}/messages/{uid}`                                                                            | `getMessage()`, `expungeMessages()`                                                    |
+| `POST /v1/mailboxes/{path}/messages/flags`, `.../expunge`, `.../copy`, `.../move`                                              | `setFlags()`, `expungeMessages()`, `copyMessages()`, `moveMessages()`                  |
+| `GET /v1/mailboxes/{path}/acl`, `PUT`, `DELETE /v1/mailboxes/{path}/acl/{identifier}` (`{ rights }`)                           | `getAcl()`, `setAcl()`, `deleteAcl()` (ACL)                                            |
+| `GET`, `PUT /v1/quota`                                                                                                         | `getQuota()`, `setQuota()` (QUOTA)                                                     |
+| `GET`, `PUT /v1/metadata`, `GET`, `PUT /v1/mailboxes/{path}/metadata`                                                          | `getMetadata()`, `setMetadata()` (METADATA)                                            |
+| `PUT /v1/mailboxes/{path}/special-use` (`{ specialUse }`)                                                                      | `setSpecialUse()` (SPECIAL-USE)                                                        |
+| `GET`, `POST`, `DELETE /v1/script/rules`, `DELETE /v1/script/rules/{id}`                                                       | [script rules](#scripted-faults) in their JSON form, so runtime faults need no restart |
+| `GET /v1/oauth`, `/v1/oauth/clients`, `/v1/oauth/consent`, `/v1/oauth/decisions`, `/v1/oauth/token-faults`, `/v1/oauth/tokens` | the [OAuth provider](#oauth-20-and-openid-connect), with the `oauth` option            |
 
 ## Scripted faults
 

package/bin/help.txt

@@ -22,6 +22,8 @@ Usage: imapkit [OPTS]
                            "Scripted faults" below)
   --plugin=<plugin>        Enables a plugin for the server. See below for
                            available plugins.
+  --tls-port=<port>        Also listens for implicit TLS (IMAPS) connections
+                           on this port, next to the plaintext port
   --smtpPort=<port>        Port number for incoming SMTP server. If not set
                            SMTP server is not started. --smtp-port works too
   --rest-port=<port>       Starts the REST API (the control API over HTTP,
@@ -29,6 +31,15 @@ Usage: imapkit [OPTS]
   --rest-host=<address>    Address of the REST API, defaults to 127.0.0.1.
                            Any other than a loopback address needs a token
   --rest-token=<token>     Bearer token that REST requests must send
+  --oauth-port=<port>      Starts the OAuth 2.0 / OpenID Connect provider:
+                           tokens for IMAP (XOAUTH2, OAUTHBEARER) and SMTP
+                           (AUTH XOAUTH2), no passwords, see "OAuth" below
+  --oauth-host=<address>   Address of the OAuth provider, 127.0.0.1 default
+  --oauth-consent=<policy> What a sign-in does: prompt (consent page, the
+                           default), approve or deny (redirect right away)
+  --oauth-secure=true      Serves the OAuth provider over HTTPS with the
+                           bundled localhost certificate
+  --oauth-issuer=<url>     Public base URL of the OAuth provider (a tunnel)
   --quirk=<name>           Makes the server behave like a known real server
                            (james-fetchgroup, james-late-fetch,
                            yahoo-quoted-sections, m365-throttle, no-uidplus,
@@ -38,9 +49,11 @@ Usage: imapkit [OPTS]
 
 The options can also be set with environment variables: IMAPKIT_PORT,
 IMAPKIT_SECURE, IMAPKIT_DEBUG, IMAPKIT_STORAGE, IMAPKIT_CONFIG,
-IMAPKIT_SCRIPT, IMAPKIT_PLUGINS (comma separated), IMAPKIT_SMTPPORT,
-IMAPKIT_REST_PORT, IMAPKIT_REST_HOST, IMAPKIT_REST_TOKEN, IMAPKIT_QUIRKS
-and IMAPKIT_SCRIPT_SEED.
+IMAPKIT_SCRIPT, IMAPKIT_PLUGINS (comma separated), IMAPKIT_TLS_PORT,
+IMAPKIT_SMTPPORT, IMAPKIT_REST_PORT, IMAPKIT_REST_HOST, IMAPKIT_REST_TOKEN,
+IMAPKIT_OAUTH_PORT, IMAPKIT_OAUTH_HOST, IMAPKIT_OAUTH_CONSENT,
+IMAPKIT_OAUTH_SECURE, IMAPKIT_OAUTH_ISSUER, IMAPKIT_QUIRKS and
+IMAPKIT_SCRIPT_SEED.
 
 NB! If port or smtpPort values are below 1024 you most probably need to
 use sudo or run the command with administrator rights.
@@ -78,6 +91,19 @@ Configuration file takes the following structure
     }
 }
 
+OAuth
+-----
+
+--oauth-port starts a test identity provider. Its discovery document is
+at <issuer>/.well-known/openid-configuration. Three public clients are
+registered (loopback redirect URIs on any port, PKCE S256 required):
+"imapkit" follows the default consent policy, "imapkit-approve" always
+approves and "imapkit-deny" always denies. The consent page needs no
+password: pick an account and click Allow or Deny. The email claim is
+the user name, so use address-style user names for OAuth tests. Load
+SASL-IR with XOAUTH2 or OAUTHBEARER to log in to IMAP with the tokens.
+More clients, scopes and policies go into "oauth" of the --config file.
+
 Scripted faults
 ---------------
 

package/bin/imapkit.js

@@ -23,9 +23,15 @@ const { values: argv } = parseArgs({
         plugin: { type: 'string', multiple: true },
         smtpPort: { type: 'string' },
         'smtp-port': { type: 'string' },
+        'tls-port': { type: 'string' },
         'rest-port': { type: 'string' },
         'rest-host': { type: 'string' },
         'rest-token': { type: 'string' },
+        'oauth-port': { type: 'string' },
+        'oauth-host': { type: 'string' },
+        'oauth-consent': { type: 'string' },
+        'oauth-secure': { type: 'boolean' },
+        'oauth-issuer': { type: 'string' },
         quirk: { type: 'string', multiple: true },
         'script-seed': { type: 'string' }
     }
@@ -48,9 +54,15 @@ const configLocation = argv.config || process.env.IMAPKIT_CONFIG;
 const storageLocation = argv.storage || process.env.IMAPKIT_STORAGE;
 const scriptLocation = argv.script || process.env.IMAPKIT_SCRIPT;
 const smtpPort = argv.smtpPort || argv['smtp-port'] || process.env.IMAPKIT_SMTPPORT;
+const tlsPort = argv['tls-port'] || process.env.IMAPKIT_TLS_PORT;
 const restPort = argv['rest-port'] || process.env.IMAPKIT_REST_PORT;
 const restHost = argv['rest-host'] || process.env.IMAPKIT_REST_HOST;
 const restToken = argv['rest-token'] || process.env.IMAPKIT_REST_TOKEN;
+const oauthPort = argv['oauth-port'] || process.env.IMAPKIT_OAUTH_PORT;
+const oauthHost = argv['oauth-host'] || process.env.IMAPKIT_OAUTH_HOST;
+const oauthConsent = argv['oauth-consent'] || process.env.IMAPKIT_OAUTH_CONSENT;
+const oauthSecure = isTrue(argv['oauth-secure'] || process.env.IMAPKIT_OAUTH_SECURE);
+const oauthIssuer = argv['oauth-issuer'] || process.env.IMAPKIT_OAUTH_ISSUER;
 const quirksList = listOption(argv.quirk || process.env.IMAPKIT_QUIRKS);
 const scriptSeed = argv['script-seed'] || process.env.IMAPKIT_SCRIPT_SEED;
 const pluginsList = listOption(argv.plugin || process.env.IMAPKIT_PLUGINS).map(plugin => plugin.toUpperCase());
@@ -115,9 +127,24 @@ if (argv.help) {
     if (scriptSeed !== undefined) {
         config.scriptSeed = Number(scriptSeed);
     }
+    if (tlsPort) {
+        config.tls = { port: Number(tlsPort) };
+    }
     if (restPort) {
         config.rest = { port: Number(restPort), host: restHost, token: restToken };
     }
+    if (oauthPort) {
+        // the other oauth options (clients, scopes ...) come from the --config file
+        config.oauth = Object.assign({}, config.oauth, { port: Number(oauthPort) });
+        for (const [key, value] of Object.entries({ host: oauthHost, consent: oauthConsent, issuer: oauthIssuer })) {
+            if (value) {
+                config.oauth[key] = value;
+            }
+        }
+        if (oauthSecure) {
+            config.oauth.secure = true;
+        }
+    }
     if (smtpPort) {
         // smtp-server is an optional dependency, start() loads it only when SMTP is enabled
         config.smtp = { port: Number(smtpPort) };
@@ -135,9 +162,25 @@ if (argv.help) {
         listening => {
             // the actual port, also for -p 0
             console.log('ImapKit successfully%s listening on port %s', isSecure ? ' and securely' : '', listening);
+            if (server.tlsServer) {
+                console.log('ImapKit listening for TLS connections on port %s', server.tlsServer.address().port);
+            }
             if (server.restServer) {
                 console.log('REST API listening on %s:%s', (config.rest && config.rest.host) || '127.0.0.1', server.restServer.address().port);
             }
+            if (server.oauthServer) {
+                const clients = server.control.listOAuthClients().map(client => client.id);
+                const consent = server.oauth.consent;
+                console.log(
+                    'OAuth provider listening on %s (clients: %s; default consent: %s)',
+                    server.oauth.issuer,
+                    clients.join(', '),
+                    typeof consent === 'string' ? consent : 'error ' + consent.error
+                );
+                if (!server.capabilities['AUTH=XOAUTH2'] && !server.capabilities['AUTH=OAUTHBEARER']) {
+                    console.log('Load the XOAUTH2 or OAUTHBEARER plugin (with SASL-IR) to log in to IMAP with its tokens');
+                }
+            }
             if (server.smtpServer) {
                 console.log('Incoming SMTP server up and running on port %s', server.smtpServer.server.address().port);
             }