@seriousme/opifex 1.11.3 → 2.0.3

package/README.md

@@ -6,21 +6,58 @@
 
 Opifex aims to provide a MQTT server and MQTT client in Typescript to be used
 with [NodeJS](https://nodejs.org), [Deno](https://deno.land) or
-[Bun](https://bun.sh) It has _no_ third party dependencies, it only relies on
-built in modules.
+[Bun](https://bun.sh). It has _no_ third-party dependencies, it only relies on
+built-in modules.
 
 # Compatibility
 
 The following MQTT versions are supported:
 
-|                | MQTT 3.1 (v3) | MQTT 3.1.1 (v4) | MQTT 5.0 (v5) |
-| -------------- | ------------- | --------------- | ------------- |
-| Packet/encoder | ✅            | ✅              | ✅            |
-| Client         | ✅            | ✅              | ✅ partially  |
-| Server         | ❌            | ✅              | ❌            |
+|                | MQTT 3.1 (v3) | MQTT 3.1.1 (v4) | MQTT 5.0 (v5)   |
+| -------------- | ------------- | --------------- | --------------- |
+| Packet/encoder | ✅            | ✅              | ✅              |
+| Client         | ✅            | ✅              | ✅ <sup>1</sup> |
+| Server         | ❌            | ✅              | ✅              |
 
-Client and server currently only have memory based persistence, but one can
-provide its own persistence. (see [Architecture](#architecture))
+<sup>1</sup> The client API fully supports MQTTv5, the CLI only supports a
+subset
+
+Version 5 support includes:
+
+- Client Topic Alias ✅
+- Server Topic Alias ✅
+- Subscription Identifiers ✅
+- Subscription options (NoLocal, RetainAsPublished, RetainHandling) ✅
+- Message expiry interval ✅
+- Will delay ✅
+- Assigned Client Identifier ✅
+- Client receive maximum ✅
+- Server receive maximum ✅
+- Shared Subscriptions ✅
+
+# Transports
+
+The following transports are currently supported:
+
+- plain TCP ✅
+- TLS ✅
+- WebSockets ✅ <sup>2</sup>
+
+<sup>2</sup> Deno + Browser only, NodeJS has no native websocket support.
+
+You can add your own transports.
+
+# Persistence
+
+Opifex offers pluggable persistence, but one can provide its own persistence.
+(see [Architecture](#architecture))
+
+The following types are currently provided:
+
+|        | Plugable | Memory | Sqlite |
+| ------ | -------- | ------ | ------ |
+| Client | ✅       | ✅     | ❌     |
+| Server | ✅       | ✅     | ✅     |
 
 ## Usage
 
@@ -35,8 +72,8 @@ can be done with CLI options then the next step is to clone the demo server
 and/or the client scripts and modify them to your liking.
 
 If you want to port the platform independent client and server libs to other
-types of transports (e.g. Unix sockets or websocketstream) then it's recommended
-to clone and modify the platform specific code in `/node` or `/deno` as well.
+types of transports (e.g. Unix sockets) then it's recommended to clone and
+modify the platform specific code in `/node` , `/deno` or `/web` as well.
 
 If you want to port the platform independent client and server libs to another
 platform then the platform specific code in `/node` or `/deno` might serve as
@@ -47,20 +84,25 @@ but for historic reasons `/deno` still exists.
 
 ## Example
 
-A simple server example using Deno.
+A simple server example:
 
 ```typescript
-import { MqttServer } from "./server/mod.ts";
-
-const listener = Deno.listen();
-const mqttServer = new MqttServer(mqttOptions);
-for await (const conn of this.listener) {
-  mqttServer.serve(conn);
-}
+import { TcpServer } from "@seriousme/opifex/tcpServer";
+
+const server = new TcpServer({ port: 1883 }, {
+  // just an example of how to configure the server
+  configuration: {
+    context: { maximumConnectPacketSize: 3000 },
+  },
+});
+server.start();
+console.log(
+  `MQTT server running on port: ${server.port}, address: ${server.address}`,
+);
 ```
 
-A more elaborate example including client and server can be found in the
-[examples](/examples/) folder.
+More elaborate examples including client and server, TLS, Websockets and sqlite
+can be found in the [examples](/examples/) folder.
 
 ## Architecture
 
@@ -69,10 +111,10 @@ A more elaborate example including client and server can be found in the
    encode and decode packets.
 
 2. On top of mqttPacket sits the MQTT connection module
-   ([mqttConn/mod.ts](mqttConn/mod.ts)) module that reads packets from a
-   Readable stream and writes them to Writable stream. It will take care of
-   incomplete and/or malformed packets. mqttConn provides an async iterable that
-   can be awaited for new packets.
+   ([mqttConn/mod.ts](mqttConn/mod.ts)) that reads packets from a Readable
+   stream and writes them to Writable stream. It will take care of incomplete
+   and/or malformed packets. mqttConn provides an async iterable that can be
+   awaited for new packets.
 
 3. On top of mqttConn live the MQTT server ([server/mod.ts](server/mod.ts)) and
    MQTT client ([client/mod.ts](client/mod.ts)) that take care of the MQTT
@@ -88,29 +130,46 @@ A more elaborate example including client and server can be found in the
    keep track of its state and associated timers.
 
 5. Persistence of data is handled by a pluggable persistence module
-   ([persistence](persistence)) which currently only offers memory persistence
-   ([persistence/memory](persistence/memory)) but can be extended with database
-   backed persistence supported by third party modules.
+   ([persistence](persistence)) which currently offers single process
+   persistence which you can replace by something more scalable (e.g. redis
+   based clusters etc) This is backed by two storage providers:
+   - [memory](persistence/memory)
+   - [sqlite](persistence/sqlite), based on node:sqlite
+
+   but can be extended with other database backend persistence supported by
+   third-party modules.
+
+6. Server behavior can be customized by passing a
+   [ConfigurationInput object](server/config.ts) via
+   `MqttServerOptions.configuration`, see the example above.
 
-6. The demo server listens to a platform specific socket and runs the `serve()`
+7. The demo server listens to a platform specific socket and runs the `serve()`
    method from the server module on the platform independent streams of every
    connection.
 
-7. The demo client opens a platform specific socket and passes the resulting
+8. The demo client opens a platform specific socket and passes the resulting
    platform independent streams to the client module.
 
 ## Exports
 
-| Export                        | Description                                             |
-| ----------------------------- | ------------------------------------------------------- |
-| @seriousme/opifex/tcpClient   | Exports a MQTT over TCP client                          |
-| @seriousme/opifex/tcpServer   | Exports a MQTT over TCP server                          |
-| @seriousme/opifex/server      | Exports a transport agnostic MQTT server                |
-| @seriousme/opifex/client      | Exports a transport agnostic MQTT client                |
-| @seriousme/opifex/persistence | Exports an Typescript interface for storage persistence |
-| @seriousme/opifex/mqttConn    | Exports MQTT connection handling                        |
-| @seriousme/opifex/mqttPacket  | Exports MQTT packet handling                            |
-| @seriousme/opifex/utils       | Exports various utilities                               |
+| Export                        | Description                                                  |
+| ----------------------------- | ------------------------------------------------------------ |
+| @seriousme/opifex/tcpClient   | Exports a MQTT over TCP/TLS client                           |
+| @seriousme/opifex/tcpServer   | Exports a MQTT over TCP server                               |
+| @seriousme/opifex/tlsServer   | Exports a MQTT over TLS server                               |
+| @seriousme/opifex/wsClient    | Exports a MQTT over Websocket client (deno and browser only) |
+| @seriousme/opifex/wsServer    | Exports a MQTT over Websocket server (deno only)             |
+| @seriousme/opifex/server      | Exports a transport agnostic MQTT server                     |
+| @seriousme/opifex/client      | Exports a transport agnostic MQTT client                     |
+| @seriousme/opifex/persistence | Exports a Typescript interface for storage persistence       |
+| @seriousme/opifex/mqttConn    | Exports MQTT connection handling                             |
+| @seriousme/opifex/mqttPacket  | Exports MQTT packet handling                                 |
+| @seriousme/opifex/utils       | Exports various utilities                                    |
+
+## API docs
+
+The latest API documentation can be found at:
+https://jsr.io/@seriousme/opifex@latest/doc
 
 ## Naming
 

package/bin/conformanceServer.ts

@@ -7,8 +7,14 @@
  */
 import { parseArgs } from "node:util";
 import { TcpServer } from "../node/tcpServer.ts";
-import type { Context, TAuthenticationResult, Topic } from "../server/mod.ts";
-import { AuthenticationResult } from "../server/mod.ts";
+import type {
+  AuthenticatedResult,
+  ConnectPacket,
+  Context,
+  ShareName,
+  Topic,
+} from "../server/mod.ts";
+import { ReasonCode } from "../server/mod.ts";
 import { logger, LogLevel } from "../utils/mod.ts";
 
 /**
@@ -42,35 +48,49 @@ const checkUsername = true;
  * @param {string} clientId - Client identifier
  * @param {string} username - Username to authenticate
  * @param {Uint8Array} password - Password as byte array
- * @returns {TAuthenticationResult} Authentication result
+ * @param {ConnectPacket} connectPacket - The raw connect packet
+ * @returns {IsAuthenticationResult} Authentication result
  */
 function isAuthenticated(
   _ctx: Context,
   clientId: string,
   username: string,
   password: Uint8Array,
-): TAuthenticationResult {
+  connectPacket: ConnectPacket,
+): AuthenticatedResult {
   const pwd = utf8Decoder.decode(password);
   logger.info(
-    `Verifying authentication of client '${clientId}' with username '${username}' and password '${pwd}'`,
+    "Verifying authentication of client",
+    clientId,
+    "with username",
+    username,
+  );
+  logger.debug(
+    "Client",
+    clientId,
+    "connecting with protocol level",
+    connectPacket.protocolLevel,
   );
 
   if (!checkUsername) {
     // allow all users access
-    return AuthenticationResult.ok;
+    return { reasonCode: ReasonCode.success };
   }
   // is the username valid according to MQTT specification
   if (!strictUsername.test(username)) {
-    return AuthenticationResult.badUsernameOrPassword;
+    return { reasonCode: ReasonCode.badUserNameOrPassword };
   }
   // Does user and password match an entry in userTable
   if (userTable.has(username)) {
     const pass = userTable.get(username);
     if (pwd === pass) {
-      return AuthenticationResult.ok;
+      return { reasonCode: ReasonCode.success };
     }
   }
-  return AuthenticationResult.notAuthorized;
+  return {
+    reasonCode: ReasonCode.notAuthorized,
+    reasonString: "Authentication failed",
+  };
 }
 
 /**
@@ -81,7 +101,10 @@ function isAuthenticated(
  */
 function isAuthorizedToPublish(ctx: Context, topic: Topic): boolean {
   logger.debug(
-    `Checking authorization of client '${ctx.store?.clientId}' to publish on topic '${topic}'`,
+    "Checking authorization of client",
+    ctx.clientId,
+    "to publish on topic",
+    topic,
   );
   return true;
 }
@@ -92,9 +115,18 @@ function isAuthorizedToPublish(ctx: Context, topic: Topic): boolean {
  * @param {Topic} topic - Topic to subscribe to
  * @returns {boolean} True if authorized
  */
-function isAuthorizedToSubscribe(ctx: Context, topic: Topic): boolean {
+function isAuthorizedToSubscribe(
+  ctx: Context,
+  topic: Topic,
+  shareName: ShareName,
+): boolean {
   logger.debug(
-    `Checking authorization of client '${ctx.store?.clientId}' to subscribe to topic '${topic}'`,
+    "Checking authorization of client",
+    ctx.clientId,
+    "to subscribe to topic",
+    topic,
+    "using shareName",
+    shareName,
   );
   return true;
 }
@@ -118,4 +150,4 @@ const tcpServer = new TcpServer({ port, hostname }, {
   },
 });
 tcpServer.start();
-logger.info(`Server started on port ${tcpServer.port}`);
+logger.info("Server started on port", tcpServer.port);

package/bin/demoServer.ts

@@ -6,8 +6,14 @@
  */
 import { parseArgs } from "node:util";
 import { TcpServer } from "../node/tcpServer.ts";
-import type { Context, TAuthenticationResult, Topic } from "../server/mod.ts";
-import { AuthenticationResult } from "../server/mod.ts";
+import type {
+  AuthenticatedResult,
+  ConnectPacket,
+  Context,
+  ShareName,
+  Topic,
+} from "../server/mod.ts";
+import { ReasonCode } from "../server/mod.ts";
 import { logger, LogLevel } from "../utils/mod.ts";
 
 /**
@@ -41,35 +47,42 @@ const checkUsername = false;
  * @param {string} clientId - Client identifier
  * @param {string} username - Username to authenticate
  * @param {Uint8Array} password - Password as byte array
- * @returns {TAuthenticationResult} Authentication result
+ * @returns {IsAuthenticationResult} Authentication result
  */
 function isAuthenticated(
   _ctx: Context,
   clientId: string,
   username: string,
   password: Uint8Array,
-): TAuthenticationResult {
+  connectPacket: ConnectPacket,
+): AuthenticatedResult {
   const pwd = utf8Decoder.decode(password);
   logger.info(
-    `Verifying authentication of client '${clientId}' with username '${username}' and password '${pwd}'`,
+    `Verifying authentication of client '${clientId}' with username '${username}'`,
+  );
+  logger.debug(
+    `Client '${clientId}' connecting with protocol level ${connectPacket.protocolLevel}`,
   );
 
   if (!checkUsername) {
     // allow all users access
-    return AuthenticationResult.ok;
+    return { reasonCode: ReasonCode.success };
   }
   // is the username valid according to MQTT specification
   if (!strictUsername.test(username)) {
-    return AuthenticationResult.badUsernameOrPassword;
+    return { reasonCode: ReasonCode.badUserNameOrPassword };
   }
   // Does user and password match an entry in userTable
   if (userTable.has(username)) {
     const pass = userTable.get(username);
     if (pwd === pass) {
-      return AuthenticationResult.ok;
+      return { reasonCode: ReasonCode.success };
     }
   }
-  return AuthenticationResult.notAuthorized;
+  return {
+    reasonCode: ReasonCode.notAuthorized,
+    reasonString: "Authentication failed",
+  };
 }
 
 /**
@@ -80,7 +93,10 @@ function isAuthenticated(
  */
 function isAuthorizedToPublish(ctx: Context, topic: Topic): boolean {
   logger.debug(
-    `Checking authorization of client '${ctx.store?.clientId}' to publish on topic '${topic}'`,
+    "Checking authorization of client",
+    ctx.clientId,
+    "to publish on topic",
+    topic,
   );
   return true;
 }
@@ -91,9 +107,18 @@ function isAuthorizedToPublish(ctx: Context, topic: Topic): boolean {
  * @param {Topic} topic - Topic to subscribe to
  * @returns {boolean} True if authorized
  */
-function isAuthorizedToSubscribe(ctx: Context, topic: Topic): boolean {
+function isAuthorizedToSubscribe(
+  ctx: Context,
+  topic: Topic,
+  shareName: ShareName,
+): boolean {
   logger.debug(
-    `Checking authorization of client '${ctx.store?.clientId}' to subscribe to topic '${topic}'`,
+    "Checking authorization of client",
+    ctx.clientId,
+    "to subscribe to topic",
+    topic,
+    "using shareName",
+    shareName,
   );
   return true;
 }
@@ -117,4 +142,4 @@ const tcpServer = new TcpServer({ port, hostname }, {
   },
 });
 tcpServer.start();
-logger.info(`Server started on port ${tcpServer.port}`);
+logger.info("Server started on port", tcpServer.port);

package/bin/mqtt.ts

@@ -229,7 +229,7 @@ async function subscribe() {
   } catch (err) {
     if (err instanceof Error) {
       const errorWithCode = err as Error & { code?: string };
-      logger.info(`Error: ${errorWithCode.message || errorWithCode.code}`);
+      logger.info("Error:", errorWithCode.message || errorWithCode.code);
     }
   }
 }
@@ -283,7 +283,7 @@ async function publish() {
   } catch (err) {
     if (err instanceof Error) {
       const errorWithCode = err as Error & { code?: string };
-      logger.info(`Error: ${errorWithCode.message || errorWithCode.code}`);
+      logger.info("Error:", errorWithCode.message || errorWithCode.code);
     }
   }
 }

package/client/ConnectionState.ts

@@ -13,6 +13,8 @@ export const ConnectionState = {
   disconnecting: "disconnecting",
   /** System has been disconnected */
   disconnected: "disconnected",
+  /** System is authenticating mid session */
+  authenticating: "authenticating",
 } as const;
 
 /**

package/client/client.ts

@@ -4,21 +4,23 @@ import {
   MemoryStore,
   MQTTLevel,
   PacketType,
+  ReasonCode,
 } from "./deps.ts";
 
 import type {
   ConnectPacket,
+  IStore,
   PublishPacket,
   SockConn,
   SubscribePacket,
   TAuthenticationResult,
 } from "./deps.ts";
 
-import { noop } from "../utils/utils.ts";
+import { noop } from "../utils/mod.ts";
 
 import { Context } from "./context.ts";
+import type { AuthenticatedResult } from "./context.ts";
 import type { TConnectionState } from "./ConnectionState.ts";
-import { assert } from "../utils/assert.ts";
 import { BufferedAsyncIterable } from "./deps.ts";
 
 /**
@@ -84,6 +86,20 @@ export const DEFAULT_KEEPALIVE = 60; // 60 seconds
 const DEFAULT_RETRIES = 3; // on first connect
 const CLIENTID_PREFIX = "opifex"; // on first connect
 
+// convert an ErrorEvent into an Error
+// NodeJS does not know ErrorEvent
+function normalizeError(err: unknown): Error {
+  if (typeof ErrorEvent !== "undefined" && err instanceof ErrorEvent) {
+    return new Error(err.message);
+  }
+
+  if (err instanceof Error) {
+    return err;
+  }
+
+  throw err;
+}
+
 /**
  * The Client class provides an MQTT Client that can be used to connect to
  * a MQTT broker and publish/subscribe messages.
@@ -94,6 +110,15 @@ const CLIENTID_PREFIX = "opifex"; // on first connect
  * connection type that is supported by the subclass.
  */
 export class Client {
+  public onAuth: (
+    authMethod: string,
+    authData: Uint8Array,
+  ) => AuthenticatedResult = () => {
+    return {
+      reasonCode: ReasonCode.badAuthenticationMethod,
+      reasonString: "No authentication method configured",
+    };
+  };
   public onError: (err: Error) => void = noop;
   public onPacket: (pkt: PublishPacket) => void | Promise<void> = noop;
   public onConnected: () => void = noop;
@@ -102,51 +127,43 @@ export class Client {
 
   protected clientIdPrefix = CLIENTID_PREFIX;
   protected numberOfRetries = DEFAULT_RETRIES;
-  protected url: URL = new URL(DEFAULT_URL);
+  protected connectUrl: URL = new URL(DEFAULT_URL);
   protected keepAlive = DEFAULT_KEEPALIVE;
   protected protocolLevel = DEFAULT_PROTOCOLLEVEL;
   protected autoReconnect = true;
-  private caCerts?: string[] | undefined;
-  private cert?: string | undefined;
-  private key?: string | undefined;
-  private clientId: string;
+  protected caCerts?: string[] | undefined;
+  protected cert?: string | undefined;
+  protected key?: string | undefined;
+  protected clientId: string;
   private ctx: Context;
   private connectPacket?: ConnectPacket;
 
   /**
    * Creates a new MQTT client instance
    */
-  constructor() {
-    this.ctx = new Context(new MemoryStore(), this);
+  constructor(store?: IStore) {
+    const cStore = store ? store : new MemoryStore();
+    this.ctx = new Context(cStore, this);
     this.clientId = generateClientId(this.clientIdPrefix);
     this.numberOfRetries = DEFAULT_RETRIES;
   }
 
-  get connectionState(): TConnectionState {
+  public get connectionState(): TConnectionState {
     return this.ctx.connectionState;
   }
 
+  public get url(): URL {
+    return this.connectUrl;
+  }
+
   /**
    * Creates a new connection to the MQTT broker
-   * @param protocol - The protocol to use (mqtt, mqtts, etc)
-   * @param _hostname - The hostname to connect to
-   * @param _port - The port to connect to
-   * @param _caCerts - Optional CA certificates
-   * @param _cert - Optional client certificate
-   * @param _key - Optional client key
    * @returns Promise resolving to a SockConn connection
    */
-  protected createConn(
-    protocol: string,
-    _hostname: string,
-    _port?: number,
-    _caCerts?: string[],
-    _cert?: string,
-    _key?: string,
-  ): Promise<SockConn> {
+  protected createConn(): Promise<SockConn> {
     // if you need to support alternative connection types just
     // overload this method in your subclass
-    throw new Error(`Unsupported protocol: ${protocol}`);
+    throw new Error(`Unsupported protocol: ${this.url.protocol}`);
   }
 
   /**
@@ -163,16 +180,12 @@ export class Client {
     let lastMessage = new Error();
     let tryConnect = true;
     while (tryConnect) {
-      logger.debug(`${isReconnect ? "re" : ""}connecting, attempt ${attempt}`);
+      logger.debug(
+        isReconnect ? "reconnecting, attempt" : "connecting, attempt",
+        attempt,
+      );
       try {
-        const conn = await this.createConn(
-          this.url.protocol,
-          this.url.hostname,
-          Number(this.url.port) ?? undefined,
-          this.caCerts,
-          this.cert,
-          this.key,
-        );
+        const conn = await this.createConn();
         // if we get this far we have a connection
         tryConnect =
           (await this.ctx.handleConnection(conn, this.connectPacket)) &&
@@ -182,12 +195,9 @@ export class Client {
         this.connectPacket.clean = false;
         this.ctx.close();
       } catch (err) {
-        assert(
-          err instanceof Error,
-          `Caught something that is not an instance of Error: ${err}`,
-        );
-        queueMicrotask(() => this.onError(err));
-        lastMessage = err;
+        const normalizedErr = normalizeError(err);
+        queueMicrotask(() => this.onError(normalizedErr));
+        lastMessage = normalizedErr;
         logger.debug({ lastMessage });
       }
 
@@ -213,7 +223,7 @@ export class Client {
    * @returns Promise resolving to authentication result
    */
   connect(params: ConnectParameters = {}): Promise<TAuthenticationResult> {
-    this.url = params?.url || this.url;
+    this.connectUrl = params?.url || this.connectUrl;
     this.numberOfRetries = params.numberOfRetries || this.numberOfRetries;
     this.caCerts = params?.caCerts;
     this.cert = params?.cert;