mongo-realtime 2.0.4 → 3.0.0

package/README.md

@@ -1,438 +1,362 @@
-# Mongo Realtime
-
-A Node.js package that combines Socket.IO and MongoDB Change Streams to deliver real-time database updates to your WebSocket clients.
-
-![Banner](logo.png)
-
-## 🚀 Features
-
-- **Real-time updates**: Automatically detects changes in MongoDB and broadcasts them via Socket.IO
-- **Granular events**: Emits specific events by operation type, collection, and document
-- **Connection management**: Customizable callbacks for socket connections/disconnections
-- **TypeScript compatible**: JSDoc annotations for better development experience
-
-## 📦 Installation
-
-```bash
-npm install mongo-realtime
-```
-
-## Setup
-
-### Prerequisites
-
-- MongoDB running as a replica set (required for Change Streams)
-- Node.js HTTP server (See below how to configure an HTTP server with Express)
-
-### Example setup
-
-```javascript
-const express = require("express");
-const http = require("http");
-const MongoRealtime = require("mongo-realtime");
-
-const app = express();
-const server = http.createServer(app);
-
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  onDbConnect: (conn) => {
-    console.log("Connected to db", conn.name);
-  },
-  onDbError: (err) => {
-    console.log(err.message);
-  },
-  server: server,
-  ignore: ["posts"], // ignore 'posts' collection
-  onSocket: (socket) => {
-    console.log(`Client connected: ${socket.id}`);
-    socket.emit("welcome", { message: "Connection successful!" });
-  },
-  offSocket: (socket, reason) => {
-    console.log(`Client disconnected: ${socket.id}, reason: ${reason}`);
-  },
-}).then(() => {
-  // It's better to start the server after the db connection is established
-  server.listen(3000, () => {
-    console.log("Server listening on port 3000");
-  });
-});
-
-```
-
-## Breaking Changes since v2.x.x
-
-In older versions, streams receive all documents and send them to sockets. This can lead to performance issues with large collections.\
-From version 2.x.x, streams now require a limit (default 100 documents) from the client side to avoid overloading the server and network.\
-Receiving streamed docs now works like this:
-
-```javascript
-socket.emit("realtime", {
-  limit: 100,
-  streamId: "users",
-  reverse: true,
-  registerId: "random-client-generated-id",
-});
-/*
- * Request 100 documents from 'users' stream (streamId defined server-side or collection name if autoStream)
- * 'reverse' to get the latest documents first
- * 'registerId' is a unique id generated by the client to identify this request. It will be used to listen to the response event.
- * The response return an object {results: [...], count: number, total:number, remaining:number, coll:string}
- */
-
-socket.on("realtime:users:random-client-generated-id", (data) => {
-  console.log("Received streamed documents:", data.results); // 100 max per
-});
-
-socket.emit("realtime", {
-  limit: 50,
-  streamId: "usersWithEmail", // make sure this stream is defined server-side
-  registerId: "specific-request-id",
-});
-
-socket.on("realtime:usersWithEmail:specific-request-id", (data) => {
-  console.log("Received streamed documents with email:", data.results); // 50 max per
-});
-
-// Make sure to listen before emitting the request to avoid missing the response
-// To stop receiving streamed documents for a specific request just use socket.off
-```
-
-## 📋 API
-
-### `MongoRealtime.init(options)`
-
-Initializes the socket system and MongoDB Change Streams.
-
-#### Parameters
-
-\* means required
-
-| Parameter             | Type              | Description                                                                      |
-| --------------------- | ----------------- | -------------------------------------------------------------------------------- |
-| `options.dbUri`       | `String`\*        | Database URI                                                                     |
-| `options.dbOptions`   | `Object`          | Mongoose connection options                                                      |
-| `options.onDbConnect` | `Function`        | Callback on successful database connection                                       |
-| `options.onDbError`   | `Function`        | Callback on database connection error                                            |
-| `options.server`      | `http.Server`\*   | HTTP server to attach Socket.IO                                                  |
-| `options.authentify`  | `Function`        | Function to authenticate socket connections. Should return true if authenticated |
-| `options.middlewares` | `Array[Function]` | Array of Socket.IO middlewares                                                   |
-| `options.onSocket`    | `Function`        | Callback on socket connection                                                    |
-| `options.offSocket`   | `Function`        | Callback on socket disconnection                                                 |
-| `options.watch`       | `Array[String]`   | Collections to only watch. Listen to all when is empty                           |
-| `options.ignore`      | `Array[String]`   | Collections to only ignore. Overrides watch array                                |
-| `options.autoStream`  | `Array[String]`   | Collections to automatically stream to clients. Default is all                   |
-| `options.debug`       | `Boolean`         | Enable debug logs                                                                |
-
-#### Static Properties and Methods
-
-- `MongoRealtime.addStream(streamId, collection, filter)`: Manually add a list stream for a specific collection and filter
-- `MongoRealtime.connection`: MongoDB connection
-- `MongoRealtime.collections`: Array of database collections
-- `MongoRealtime.io`: Socket.IO server instance
-- `MongoRealtime.init(options)`: Initialize the package with options
-- `MongoRealtime.listen(event, callback)`: Add an event listener for database changes
-- `MongoRealtime.notifyListeners(event, data)`: Manually emit an event to all listeners
-- `MongoRealtime.removeStream(streamId)`: Remove a previously added stream by id
-- `MongoRealtime.removeListener(event, callback)`: Remove a specific event listener or all listeners for an event
-- `MongoRealtime.removeAllListeners()`: Remove all event listeners
-- `MongoRealtime.sockets()`: Returns an array of connected sockets
-
-## 🎯 Emitted Events
-
-The package automatically emits six types of events for each database change:
-
-### Event Types
-
-| Event                              | Description       | Example                                    |
-| ---------------------------------- | ----------------- | ------------------------------------------ |
-| `db:change`                        | All changes       | Any collection change                      |
-| `db:{type}`                        | By operation type | `db:insert`, `db:update`, `db:delete`      |
-| `db:change:{collection}`           | By collection     | `db:change:users`, `db:change:posts`       |
-| `db:{type}:{collection}`           | Type + collection | `db:insert:users`, `db:update:posts`       |
-| `db:change:{collection}:{id}`      | Specific document | `db:change:users:507f1f77bcf86cd799439011` |
-| `db:{type}:{collection}:{id}`      | Type + document   | `db:insert:users:507f1f77bcf86cd799439011` |
-| `realtime:{streamId}:{registerId}` | By stream         | `realtime:myStreamId:registerId`           |
-
-### Event listeners
-
-You can add serverside listeners to those db events to trigger specific actions on the server:
-
-```js
-function sendNotification(change) {
-  const userId = change.docId; // or change.documentKey._id
-  NotificationService.send(userId, "Welcome to DB");
-}
-
-MongoRealtime.listen("db:insert:users", sendNotification);
-```
-
-#### Adding many callback to one event
-
-```js
-MongoRealtime.listen("db:insert:users", anotherAction);
-MongoRealtime.listen("db:insert:users", anotherAction2);
-```
-
-#### Removing event listeners
-
-```js
-MongoRealtime.removeListener("db:insert:users", sendNotification); // remove this specific action from this event
-MongoRealtime.removeListener("db:insert:users"); // remove all actions from this event
-MongoRealtime.removeAllListeners(); // remove all listeners
-```
-
-### Event Payload Structure
-
-Each event contains the full MongoDB change object:
-
-```javascript
-{
-  "_id": {...},
-  "col":"users", // same as ns.coll
-  "docId":"...", // same as documentKey._id
-  "operationType": "insert|update|delete|replace",
-  "documentKey": { "_id": "..." },
-  "ns": { "db": "mydb", "coll": "users" },
-  "fullDocument": {...},
-  "fullDocumentBeforeChange": {...}
-}
-```
-
-## 🔨 Usage Examples
-
-### Server-side - Listening to specific events
-
-```javascript
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  onSocket: (socket) => {
-    socket.on("subscribe:users", () => {
-      socket.join("users-room");
-    });
-  },
-});
-
-MongoRealtime.io.to("users-room").emit("custom-event", data);
-```
-
-### Client-side - Receiving updates
-
-```html
-<!DOCTYPE html>
-<html>
-  <head>
-    <script src="/socket.io/socket.io.js"></script>
-  </head>
-  <body>
-    <script>
-      const socket = io();
-
-      socket.on("db:change", (change) => {
-        console.log("Detected change:", change);
-      });
-
-      socket.on("db:insert:users", (change) => {
-        console.log("New user:", change.fullDocument);
-      });
-
-      const userId = "507f1f77bcf86cd799439011";
-      socket.on(`db:update:users:${userId}`, (change) => {
-        console.log("Updated user:", change.fullDocument);
-      });
-
-      socket.on("db:delete", (change) => {
-        console.log("Deleted document:", change.documentKey);
-      });
-    </script>
-  </body>
-</html>
-```
-
-## Error Handling
-
-```javascript
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  onSocket: (socket) => {
-    socket.on("error", (error) => {
-      console.error("Socket error:", error);
-    });
-  },
-  offSocket: (socket, reason) => {
-    if (reason === "transport error") {
-      console.log("Transport error detected");
-    }
-  },
-});
-```
-
-## 🔒 Security
-
-### Socket Authentication
-
-You can provide an `authentify` function in the init options to authenticate socket connections.\
-The function receives the token (from `socket.handshake.auth.token` or `socket.handshake.headers.authorization`) and the socket object.\
-When setted, it rejects connections based on this logic:
-
-- Token not provided -> error `NO_TOKEN_PROVIDED`
-- Token invalid or returns `false` -> error `UNAUTHORIZED`
-- Any other error -> error `AUTH_ERROR`
-- Return `true` to accept the connection
-
-```javascript
-function authenticateSocket(token, socket) {
-  const verify = AuthService.verifyToken(token);
-  if (verify) {
-    socket.user = verify.user; // attach user info to socket
-    return true; // should return true to accept the connection
-  }
-  return false;
-}
-
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  authentify: authenticateSocket,
-  middlewares: [
-    (socket, next) => {
-      console.log(`User is authenticated: ${socket.user.email}`);
-      next();
-    },
-  ],
-  offSocket: (socket, reason) => {
-    console.log(`Socket ${socket.id} disconnected: ${reason}`);
-  },
-});
-```
-
-### Setup streams
-
-The server will automatically emit a list of filtered documents from the specified collections after each change.\
-Each list stream requires an unique `streamId`, the `collection` name, and an optional `filter` function that returns a boolean or a promise resolving to a boolean.
-Clients receive the list on the event `db:stream:{streamId}`.\
-
-```javascript
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  autoStream: ["users"], // automatically stream users collection only
-});
-
-MongoRealtime.addStream("users", "users", (doc) => !!doc.email); // will throw an error as streamId 'users' already exists
-
-MongoRealtime.removeStream("users"); // remove the previous stream
-MongoRealtime.addStream("users", "users", (doc) => !!doc.email); // client can listen to db:stream:users
-
-MongoRealtime.addStream("usersWithEmail", "users", (doc) => !!doc.email); // client can listen to db:stream:usersWithEmail
-```
-
-#### ⚠️ NOTICE
-
-When `autoStream` is not set, all collections are automatically streamed and WITHOUT any filter.\
-That means that if you have a `posts` collection, all documents from this collection will be sent to the clients on each change.\
-
-```javascript
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  autoStream: [], // stream no collection automatically (you can add your own filtered streams later)
-});
-// or
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  // Stream all collections automatically but you can override them
-}):
-
-MongoRealtime.addStream("postsWithTitle", "posts", (doc) => !!doc.title); // client can listen to db:stream:posts
-MongoRealtime.addStream("users", "users", (doc) => !!doc.email); // will not throw an error cause 'users' is already streamed but with no filter
-```
-
-#### Usecase for id based streams
-
-```javascript
-MongoRealtime.init({
-  dbUri: "mongodb://localhost:27017/mydb",
-  server: server,
-  authentify: (token, socket) => {
-    try {
-      socket.uid = decodeToken(token).uid; // setup user id from token
-      return true;
-    } catch (error) {
-      return false;
-    }
-  },
-  onSocket: (socket) => {
-    // setup a personal stream for each connected user
-    MongoRealtime.addStream(
-      `userPost:${socket.uid}`,
-      "posts",
-      (doc) => doc._id == socket.uid
-    );
-  },
-  offSocket: (socket) => {
-    // clean up when user disconnects
-    MongoRealtime.removeStream(`userPost:${socket.uid}`);
-  },
-});
-
-// ...
-// or activate stream from a controller or middleware
-app.get("/my-posts", (req, res) => {
-  const { user } = req;
-  MongoRealtime.addStream(
-    `userPosts:${user._id}`,
-    "posts",
-    (doc) => doc.authorId === user._id
-  );
-
-  res.send("Stream activated");
-});
-```
-
-### Usecase with async filter
-
-```javascript
-// MongoRealtime.init({...});
-
-MongoRealtime.addStream("authorizedUsers", "users", async (doc) => {
-  const isAdmin = await UserService.isAdmin(doc._id);
-  return isAdmin && doc.email.endsWith("@mydomain.com");
-});
-
-MongoRealtime.addStream(
-  "bestPosts",
-  "posts",
-  async (doc) => doc.likes > (await PostService.getLikesThreshold())
-);
-```
-
-## 📚 Dependencies
-
-- `socket.io`: WebSocket management
-- `mongoose`: MongoDB ODM with Change Streams support
-
-## 🐛 Troubleshooting
-
-### MongoDB must be in Replica Set mode
-
-To use Change Streams, MongoDB must be running as a replica set. For local development, you can initiate a single-node replica set:
-
-```bash
-mongod --replSet rs0
-
-rs.initiate()
-```
-
-For any other issues, open an issue on the GitHub repository.
-
-## 📄 License
-
-MIT
-
-## 🤝 Contributing
-
-Contributions are welcome! Feel free to open an issue or submit a pull request.
+# Mongo Realtime
+
+MongoRealTime exposes MongoDB Change Streams over native WebSockets.\
+Clients can subscribe to live updates on query results, perform CRUD operations, and send custom commands to the server.
+
+## 📦 Installation
+
+```bash
+npm install mongo-realtime
+```
+
+## Quick start
+
+```js
+const { MongoRealTimeServer } = require("mongo-realtime");
+
+const server = new MongoRealTimeServer({
+  mongoUri: "mongodb://localhost:27017/mydb",
+  dbName: "mydb",
+});
+
+// Register a handler for custom commands sent with `realtime:emit`.
+server.on("calculate", (payload) => {
+  return payload.reduce((sum, value) => sum + value, 0);
+});
+
+await server.start();
+```
+
+## Server configuration
+
+The constructor accepts these options:
+
+- `host` - Host for the built-in HTTP server (`0.0.0.0` by default).
+- `port` - Port for the built-in HTTP server (`3000` by default).
+- `path` - WebSocket upgrade path (`/` by default).
+- `mongoUri` - MongoDB connection URI.
+- `dbName` - MongoDB database name.
+- `cacheTtlMs` - Query result cache TTL in milliseconds (`300000` by default).
+- `authenticate` - Optional async function to validate incoming socket connections.
+- `server` - Optional existing HTTP server to attach the WebSocket endpoint.
+- `mongoClient` - Optional existing `MongoClient` instance.
+- `db` - Optional existing MongoDB `Db` instance.
+- `logger` - Optional `{ info?, warn? }` logger object.
+
+When `server` is omitted, the package creates and owns an HTTP server.
+
+## Environment variables
+
+The package can also read configuration from `.env`:
+
+- `HOST`
+- `PORT`
+- `WS_PATH`
+- `MONGODB_URI` or `MONGO_URI`
+- `MONGODB_DB_NAME` or `MONGO_DB`
+- `CACHE_TTL_MS`
+- `CACHE_TTL_SECONDS`
+
+## WebSocket protocol
+
+### Supported message types
+
+From client to server:
+
+- `realtime:subscribe`
+- `realtime:unsubscribe`
+- `realtime:fetch`
+- `realtime:insert`
+- `realtime:update`
+- `realtime:delete`
+- `realtime:emit`
+
+### `realtime:subscribe`
+
+Subscribe to live matching documents and receive an initial result set.
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:subscribe",
+    collection: "users",
+    filter: { active: true },
+    sort: { createdAt: -1 },
+    limit: 50,
+    queryId: "my-query-id",
+  }),
+);
+```
+
+The server replies with:
+
+```js
+{
+  type: 'realtime:initial',
+  collection: 'users',
+  queryId: 'my-query-id',
+  documents: [ ... ],
+}
+```
+
+Live changes are delivered as separate events:
+
+- `realtime:insert`
+- `realtime:update`
+- `realtime:delete`
+
+### `realtime:fetch`
+
+Fetch the current document set without keeping a live subscription.
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:fetch",
+    collection: "users",
+    filter: { active: true },
+    sort: { createdAt: -1 },
+    limit: 50,
+    queryId: "fetch-1",
+  }),
+);
+```
+
+### `realtime:unsubscribe`
+
+Stop a live subscription by its `queryId`:
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:unsubscribe",
+    queryId: "my-query-id",
+  }),
+);
+```
+
+### `realtime:insert`
+
+Insert a new document into a collection:
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:insert",
+    collection: "users",
+    document: { name: "Alice", active: true },
+  }),
+);
+```
+
+### `realtime:update`
+
+Update matching documents in a collection:
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:update",
+    collection: "users",
+    filter: { _id: "507f1f77bcf86cd799439011" },
+    update: { $set: { active: false } },
+  }),
+);
+```
+
+The server accepts both operator-style updates (`$set`, `$inc`, etc.) and replacement-style updates.
+
+### `realtime:delete`
+
+Delete matching documents from a collection:
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:delete",
+    collection: "users",
+    filter: { active: false },
+  }),
+);
+```
+
+### `realtime:emit`
+
+Send a custom command to the server and receive a response.
+
+```js
+socket.send(
+  JSON.stringify({
+    type: "realtime:emit",
+    event: "calculate",
+    payload: [1, 2, 3],
+    requestId: "request-1",
+  }),
+);
+```
+
+Server responses:
+
+```js
+{
+  type: 'realtime:emit:result',
+  event: 'calculate',
+  requestId: 'request-1',
+  data: 6,
+}
+```
+
+or:
+
+```js
+{
+  type: 'realtime:emit:error',
+  event: 'calculate',
+  requestId: 'request-1',
+  error: 'No handler registered for event "calculate".',
+}
+```
+
+### Server internal events
+
+The server emit internal events that can be listened inside the backend code using `server.on('MY_INTERNAL_EVENT', handler)`.\
+`MY_INTERNAL_EVENT` follows these patterns:
+
+- `db:OPERATION_TYPE` : For any operation type on any collection
+- `db:OPERATION_TYPE:COLLECTION_NAME` : For a specific operation type on a specific collection
+- `db:OPERATION_TYPE:COLLECTION_NAME:DOCUMENT_ID` : For a specific operation type on a specific document
+
+`OPERATION_TYPE` can be `insert`, `update`, `delete` or `change` (matches all).\
+`COLLECTION_NAME` is the name of the collection, e.g. `users`.\
+`DOCUMENT_ID` is the string representation of the document's `_id`, e.g. `507f1f77bcf86cd799439011`.
+
+Example:
+
+```js
+server.on("db:insert:users", (change) => {
+  console.log("A new user was inserted:", change.document);
+});
+server.on("db:update:orders:507f1f77bcf86cd799439011", (change) => {
+  console.log(
+    "Order 507f1f77bcf86cd799439011 was updated:",
+    change.updateDescription,
+  );
+});
+```
+
+### Error responses
+
+If a request fails, the server sends:
+
+```js
+{
+  type: 'realtime:error',
+  error: 'Error message',
+  queryId?: 'my-query-id',
+}
+```
+
+## Message payloads from server
+
+Live change messages follow this shape:
+
+```js
+{
+  type: 'realtime:insert' | 'realtime:update' | 'realtime:delete',
+  collection: 'users',
+  document: { ... } | null,
+  before?: { ... },
+  documentId?: '507f1f77bcf86cd799439011',
+}
+```
+
+## Query filters
+
+Supported filter operators include:
+
+- `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`
+- `$in`, `$nin`
+- `$exists`
+- `$regex`
+- `$and`, `$or`, `$nor`
+
+Nested document paths are supported. `_id` strings are automatically converted to `ObjectId` when possible.
+
+## API
+
+### `new MongoRealTimeServer(options)`
+
+Creates a new server instance.
+
+### `server.start()`
+
+Connects to MongoDB, attaches WebSocket handlers, and starts listening if the package owns the HTTP server.
+
+### `server.stop()`
+
+Closes active subscriptions, connected sockets, and owned resources.
+
+### `server.on(eventName, handler)`
+
+Registers a handler for `realtime:emit` messages.
+
+### `server.collection(name)`
+
+Returns a MongoDB collection handle for direct access.
+
+## Authentication
+
+Provide an `authenticate` function to validate WebSocket connections. The incoming payload is read from the `auth` request header and parsed as JSON when possible.
+
+Example:
+
+```js
+const server = new MongoRealTimeServer({
+  authenticate: async (authData) => {
+    // Perform your authentication logic here, e.g. check a token or session.
+    // In this example, we simply check if the authData matches a hardcoded token.
+    return authData === "my-auth-token";
+  },
+});
+```
+
+## Example: attach to Express
+
+```js
+const http = require("node:http");
+const express = require("express");
+const { MongoRealTimeServer } = require("mongo-realtime");
+
+const app = express();
+const httpServer = http.createServer(app);
+
+const realtimeServer = new MongoRealTimeServer({
+  server: httpServer, // needs to be the raw HTTP server, not the Express app
+  path: "/", // WebSocket path
+  mongoUri: "mongodb://localhost:27017/mydb",
+  dbName: "mydb",
+});
+
+await new Promise((resolve, reject) => {
+  httpServer.once("error", reject);
+  httpServer.listen(3000, "0.0.0.0", resolve);
+});
+
+await realtimeServer.start(); // start the MongoRealTimeServer after the HTTP server is listening
+```
+
+## Notes
+
+- MongoDB must run as a replica set for Change Streams.
+- The package now uses native WebSockets (`ws`), no longer Socket.IO.
+- Query results are cached for `cacheTtlMs` milliseconds when using `subscribe` or `fetch`.
+
+## Dependencies
+
+- `mongodb`
+- `ws`
+- `dotenv`
+
+## License
+
+MIT