allserver 1.1.0 → 1.3.0

package/README.md

@@ -2,7 +2,7 @@
 
 Multi-transport and multi-protocol simple RPC server and (optional) client. Boilerplate-less. Opinionated. Minimalistic. DX-first.
 
-Think HTTP, gRPC, GraphQL, WebSockets, Lambda, inter-process, unix sockets, etc Remote Procedure Calls using exactly the same client and server code.
+Think of Remote Procedure Calls using exactly the same client and server code but via multiple protocols/mechanisms such as HTTP, gRPC, GraphQL, WebSockets, job queues, Lambda, inter-process, (Web)Workers, unix sockets, etc etc etc.
 
 Should be used in (micro)services where JavaScript is able to run - your computer, Docker, k8s, virtual machines, serverless functions (Lambdas, Google Cloud Functions, Azure Functions, etc), RaspberryPI, SharedWorker, thread, you name it.
 
@@ -12,6 +12,8 @@ Superpowers the `Allserver` gives you:
 - Run your HTTP server as gRPC with a single line change (almost).
 - Serve same logic via HTTP and gRPC (or more) simultaneously in the same node.js process.
 - Deploy and run your HTTP server on AWS Lambda with no code changes.
+- Embed same exact code into your existing Express.js, no need to implement any handlers/middleware.
+- Use Redis-backed job queue [BullMQ](https://docs.bullmq.io) to call remote procedures reliably.
 - And moar!
 
 Superpowers the `AllserverClient` gives you:
@@ -88,6 +90,7 @@ When calling a remote procedure I want something which:
 - Can be easily mapped to any language, any protocol. Especially to upstream GraphQL mutations.
 - Is simple to read in the source code, just like a method/function call. Without thinking of protocol-level details for every damn call.
 - Allows me to test gRPC server from my browser/Postman/curl (via HTTP!) by a simple one line config change.
+- Replace flaky HTTP with Kafka
 - Does not bring tons of npm dependencies with it.
 
 Also, the main driving force was my vast experience splitting monolith servers onto (micro)services. Here is how I do it with much success.
@@ -141,15 +144,21 @@ Please note, that Allserver depends only on a single tiny npm module [`stampit`]
 
 The default `HttpTransport` is using the [`micro`](http://npmjs.com/package/micro) npm module as an optional dependency.
 
-```shell script
+```shell
 npm i allserver micro
 ```
 
+Alternatively, you can embed Allserver into your existing Express.js application:
+
+```shell
+npm i allserver express
+```
+
 #### Client
 
 Optionally, you can use Allserver's built-in client:
 
-```shell script
+```shell
 npm i allserver node-fetch
 ```
 
@@ -161,7 +170,7 @@ Or do HTTP requests using any module you like.
 
 No dependencies other than `allserver` itself.
 
-```shell script
+```shell
 npm i allserver
 ```
 
@@ -175,7 +184,7 @@ Same as the HTTP protocol client above.
 
 The default `GrpcTransport` is using the standard the [`@grpc/grpc-js`](https://www.npmjs.com/package/@grpc/grpc-js) npm module as an optional dependency.
 
-```shell script
+```shell
 npm i allserver @grpc/grpc-js@1 @grpc/proto-loader@0.5
 ```
 
@@ -185,12 +194,32 @@ Note, with gRPC server and client you'd need to have your own `.proto` file. See
 
 Optionally, you can use Allserver's built-in client:
 
-```shell script
+```shell
 npm i allserver @grpc/grpc-js@1 @grpc/proto-loader@0.5
 ```
 
 Or do gRPC requests using any module you like.
 
+### [BullMQ](https://docs.bullmq.io) job queue
+
+#### Server
+
+The default `BullmqTransport` is using the [`bullmq`](https://www.npmjs.com/package/bullmq) module as a dependency, connects to Redis using `Worker` class.
+
+```shell
+npm i allserver bullmq
+```
+
+#### Client
+
+Optionally, you can use Allserver's built-in client:
+
+```shell
+npm i allserver bullmq
+```
+
+Or use the `bullmq` module directly. You don't need to use Allserver to call remote procedures. See code example below.
+
 ## Code examples
 
 ### Procedures
@@ -297,19 +326,36 @@ const procedures = {
   async processEntity(_, ctx) {
     const micro = ctx.allserver.transport.micro; // same as require("micro")
     const req = ctx.http.req; // node.js Request
-    
+
     // as a string
     const text = await micro.text(req);
     // as a node.js buffer
     const buffer = await micro.buffer(req);
-    
-    // ... process the request here ... 
+
+    // ... process the request here ...
   },
 };
 ```
 
 More info can be found in the [`micro`](http://npmjs.com/package/micro) NPM module docs.
 
+### HTTP server in Express.js
+
+Doesn't require a dedicated client transport. Use the HTTP client below.
+
+NB: not yet tested in production.
+
+```js
+const { Allserver, ExpressTransport } = require("allserver");
+
+const middleware = Allserver({
+  procedures,
+  transport: ExpressTransport(),
+}).start();
+
+app.use("/route-with-allsever", middleware);
+```
+
 ### HTTP server in AWS Lambda
 
 Doesn't require a dedicated client transport. Use the HTTP client below.
@@ -343,7 +389,7 @@ exports = Allserver({
 
 You'd need to install `node-fetch` optional dependency.
 
-```shell script
+```shell
 npm i allserver node-fetch
 ```
 
@@ -354,7 +400,7 @@ const { AllserverClient } = require("allserver");
 // or
 const AllserverClient = require("allserver/Client");
 
-const client = AllserverClient({ uri: "http://localhost:4000" });
+const client = AllserverClient({ uri: "http://localhost:40000" });
 
 const { success, code, message, user } = await client.updateUser({
   id: "123412341234123412341234",
@@ -363,7 +409,7 @@ const { success, code, message, user } = await client.updateUser({
 });
 ```
 
-The `AllserverClient` will issue `HTTP POST` request to this URL: `http://localhost:4000/updateUser`.
+The `AllserverClient` will issue `HTTP POST` request to this URL: `http://localhost:40000/updateUser`.
 The path of the URL is dynamically taken straight from the `client.updateUser` calling code using the ES6 [`Proxy`](https://stackoverflow.com/a/20147219/188475) class. In other words, `AllserverClient` intercepts non-existent property access.
 
 #### Using any HTTP client (axios in this example)
@@ -373,7 +419,7 @@ It's a regular HTTP `POST` call with JSON request and response. URI is `/updateU
 ```js
 import axios from "axios";
 
-const response = await axios.post("http://localhost:4000/updateUser", {
+const response = await axios.post("http://localhost:40000/updateUser", {
   id: "123412341234123412341234",
   firstName: "Fred",
   lastName: "Flinstone",
@@ -479,15 +525,80 @@ const { success, code, message, user } = data;
 1. You can't have `import` statements in your `.proto` file. (Yet.)
 1. Your server-side `.proto` file must include Allserver's [mandatory declarations](./mandatory.proto). (Yet.)
 
+### BullMQ server side
+
+Note that we are reusing the `procedures` from the example above.
+
+Here is how your BullMQ server can look like:
+
+```js
+const { Allserver, BullmqTransport } = require("allserver");
+
+Allserver({
+  procedures,
+  transport: BullmqTransport({
+    connectionOptions: { host: "localhost", port: 6379 },
+  }),
+}).start();
+```
+
+### BullMQ client side
+
+#### Using built-in client
+
+Note, that this code is **same** as the HTTP client code example above! The only difference is the URI.
+
+```js
+const { AllserverClient, BullmqClientTransport } = require("allserver");
+// or
+const AllserverClient = require("allserver/Client");
+
+const client = AllserverClient({ uri: "bullmq://localhost:6379" });
+// or
+const client = AllserverClient({
+  transport: BullmqClientTransport({ uri: "redis://localhost:6379" }),
+});
+
+const { success, code, message, user } = await client.updateUser({
+  id: "123412341234123412341234",
+  firstName: "Fred",
+  lastName: "Flinstone",
+});
+```
+
+The `bullmq://` schema uses same connection string as Redis: `bullmq://[[username:]password@]host[:port][/database]`
+
+#### Using any BullMQ `Queue` class without Allserver
+
+```js
+const { Queue, QueueEvents } = require("bullmq");
+
+const queue = new Queue("Allserver", {
+  connection: { host: "localhost", port },
+});
+const queueEvents = new QueueEvents("Allserver", {
+  connection: { host: "localhost", port },
+});
+
+const job = await queue.add("updateUser", {
+  id: "123412341234123412341234",
+  firstName,
+  lastName,
+});
+const data = await job.waitUntilFinished(queueEvents, 30_000);
+
+const { success, code, message, user } = data;
+```
+
 ## `AllserverClient` options
 
 **All the arguments are optional.** But either `uri` or `transport` must be provided. We are trying to keep the highest possible DX here.
 
 - `uri`<br>
-  The remote server address string. Out of box supported schemas are: `http`, `https`, `grpc`. (More to come.)
+  The remote server address string. Out of box supported schemas are: `http`, `https`, `grpc`, `bullmq`. (More to come.)
 
 - `transport`<br>
-  The transport implementation object. The `uri` is ignored if this option provided. If not given then it will be automatically created based on the `uri` schema. E.g. if it starts with `http://` or `https:/` then `HttpClientTransport` will be used. If starts with `grpc://` then `GrpcClientTransport` will be used.
+  The transport implementation object. The `uri` is ignored if this option provided. If not given then it will be automatically created based on the `uri` schema. E.g. if it starts with `http://` or `https://` then `HttpClientTransport` will be used. If starts with `grpc://` then `GrpcClientTransport` will be used. If starts with `bullmq://` then `BullmqClientTransport` is used.
 
 - `neverThrow=true`<br>
   Set it to `false` if you want to get exceptions when there are a network, or a server errors during a procedure call. Otherwise, the standard `{success,code,message}` object is returned from method calls. The Allserver error `code`s are always start with `"ALLSERVER_"`. E.g. `"ALLSERVER_CLIENT_MALFORMED_INTROSPECTION"`.
@@ -653,7 +764,7 @@ Yep.
 const { AllserverClient } = require("allserver");
 
 const client = AllserverClient({
-  uri: "http://example.com:4000",
+  uri: "http://example.com:40000",
 
   async before(ctx) {
     console.log(ctx.procedureName, ctx.arg);
@@ -685,7 +796,7 @@ const { AllserverClient, HttpClientTransport } = require("allserver");
 
 const client = AllserverClient({
   transport: HttpClientTransport({
-    uri: "http://my-server:4000",
+    uri: "http://my-server:40000",
     headers: { authorization: "Basic my-token" },
   }),
 });
@@ -712,7 +823,7 @@ If something more sophisticated is needed - you would need to mangle the `ctx` i
 const { AllserverClient } = require("allserver");
 
 const client = AllserverClient({
-  uri: "http://my-server:4000",
+  uri: "http://my-server:40000",
   async before(ctx) {
     console.log(ctx.procedureName, ctx.arg);
     ctx.http.mode = "cors";
@@ -739,7 +850,7 @@ const MyAllserverClientWithAuth = AllserverClient.defaults({
 });
 
 const client = MyAllserverClientWithAuth({
-  uri: "http://my-server:4000",
+  uri: "http://my-server:40000",
 });
 ```
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "allserver",
-  "version": "1.1.0",
+  "version": "1.3.0",
   "description": "Multi-protocol simple RPC server and [optional] client. Boilerplate-less. Opinionated. Minimalistic. DX-first.",
   "main": "src/index.js",
   "scripts": {
@@ -34,10 +34,12 @@
   "devDependencies": {
     "@grpc/grpc-js": "^1.1.7",
     "@grpc/proto-loader": "^0.5.5",
+    "bullmq": "^3.5.7",
     "eslint": "^7.9.0",
+    "express": "^4.18.2",
     "lambda-local": "^1.7.3",
     "micro": "^9.3.4",
-    "mocha": "^8.1.3",
+    "mocha": "^10.2.0",
     "node-fetch": "^2.6.1",
     "nyc": "^15.1.0",
     "prettier": "^2.1.1"

package/src/client/AllserverClient.js

@@ -164,6 +164,7 @@ module.exports = require("stampit")({
             http() { return require("./HttpClientTransport"); },
             https() { return require("./HttpClientTransport"); },
             grpc() { return require("./GrpcClientTransport"); },
+            bullmq() { return require("./BullmqClientTransport"); },
         },
     },
 

package/src/client/BullmqClientTransport.js

@@ -0,0 +1,71 @@
+module.exports = require("./ClientTransport").compose({
+    name: "BullmqClientTransport",
+
+    props: {
+        Queue: require("bullmq").Queue,
+        QueueEvents: require("bullmq").QueueEvents,
+        _timeout: 60000,
+        _queue: null,
+        _queueEvents: null,
+        _jobsOptions: null,
+    },
+
+    init({ queueName = "Allserver", connectionOptions, timeout, jobsOptions }) {
+        if (!connectionOptions) {
+            const bullmqUrl = new URL(this.uri);
+            connectionOptions = {
+                host: bullmqUrl.hostname,
+                port: bullmqUrl.port,
+                username: bullmqUrl.username,
+                password: bullmqUrl.password,
+                db: Number(bullmqUrl.pathname.substr(1)) || 0,
+                retryStrategy: null, // only one attempt to connect
+            };
+        }
+        this._timeout = timeout || this._timeout;
+
+        this._queue = new this.Queue(queueName, { connection: connectionOptions });
+        this._queue.on("error", () => {}); // The only reason we subscribe is to avoid bullmq to print errors to console
+        this._queueEvents = new this.QueueEvents(queueName, { connection: connectionOptions });
+        this._queueEvents.on("error", () => {}); // The only reason we subscribe is to avoid bullmq to print errors to console
+
+        this._jobsOptions = jobsOptions || {};
+        if (this._jobsOptions.removeOnComplete === undefined) this._jobsOptions.removeOnComplete = 100;
+        if (this._jobsOptions.removeOnFail === undefined) this._jobsOptions.removeOnFail = 100;
+        if (this._jobsOptions.sizeLimit === undefined) this._jobsOptions.sizeLimit = 524288; // max data JSON size is 512KB
+    },
+
+    methods: {
+        async introspect(ctx) {
+            ctx.procedureName = "introspect";
+            const jobReturnResult = await this.call(ctx);
+            // The server-side Transport will not have job result if introspection is not available on the server side,
+            // but the server itself is up and running processing calls.
+            if (jobReturnResult == null) throw new Error("The bullmq introspection job returned nothing");
+            return jobReturnResult;
+        },
+
+        async call({ procedureName, bullmq }) {
+            try {
+                await this._queue.waitUntilReady();
+                const job = await bullmq.queue.add(procedureName, bullmq.data, bullmq.jobsOptions);
+                return await job.waitUntilFinished(bullmq.queueEvents, this._timeout);
+            } catch (err) {
+                if (err.code === "ECONNREFUSED") err.noNetToServer = true;
+                throw err;
+            }
+        },
+
+        createCallContext(defaultCtx) {
+            return {
+                ...defaultCtx,
+                bullmq: {
+                    data: defaultCtx.arg,
+                    queue: this._queue,
+                    queueEvents: this._queueEvents,
+                    jobsOptions: this._jobsOptions,
+                },
+            };
+        },
+    },
+});

package/src/client/HttpClientTransport.js

@@ -6,7 +6,9 @@ module.exports = require("./ClientTransport").compose({
     props: {
         // eslint-disable-next-line no-undef
         fetch: (typeof self !== "undefined" && self.fetch) || require("node-fetch"),
-        headers: {},
+        headers: {
+            "Content-Type": "application/json; charset=utf-8",
+        },
     },
 
     init({ headers, fetch }) {

package/src/index.js

@@ -11,12 +11,18 @@ module.exports = {
     get HttpTransport() {
         return require("./server/HttpTransport");
     },
+    get ExpressTransport() {
+        return require("./server/ExpressTransport");
+    },
     get LambdaTransport() {
         return require("./server/LambdaTransport");
     },
     get GrpcTransport() {
         return require("./server/GrpcTransport");
     },
+    get BullmqTransport() {
+        return require("./server/BullmqTransport");
+    },
 
     // client
 
@@ -32,4 +38,7 @@ module.exports = {
     get GrpcClientTransport() {
         return require("./client/GrpcClientTransport");
     },
+    get BullmqClientTransport() {
+        return require("./client/BullmqClientTransport");
+    },
 };

package/src/server/Allserver.js

@@ -65,7 +65,7 @@ module.exports = require("stampit")({
             try {
                 result = await ctx.procedure(ctx.arg, ctx);
             } catch (err) {
-                const code = err.code || "ALLSERVER_PROCEDURE_ERROR"
+                const code = err.code || "ALLSERVER_PROCEDURE_ERROR";
                 this.logger.error(code, err);
                 ctx.error = err;
                 ctx.result = {
@@ -103,7 +103,7 @@ module.exports = require("stampit")({
                         break;
                     }
                 } catch (err) {
-                    const code = err.code || "ALLSERVER_MIDDLEWARE_ERROR";  
+                    const code = err.code || "ALLSERVER_MIDDLEWARE_ERROR";
                     this.logger.error(code, err);
                     ctx.error = err;
                     ctx.result = {
@@ -136,7 +136,7 @@ module.exports = require("stampit")({
             // Warning! This call might overwrite an existing result.
             await this._callMiddlewares(ctx, "after");
 
-            this.transport.reply(ctx);
+            return this.transport.reply(ctx);
         },
 
         start() {

package/src/server/BullmqTransport.js

@@ -0,0 +1,54 @@
+module.exports = require("./Transport").compose({
+    name: "BullmqTransport",
+
+    props: {
+        Worker: require("bullmq").Worker,
+        _worker: null,
+        _queueName: null,
+        _connectionOptions: null,
+        _workerOptions: null,
+    },
+
+    init({ queueName = "Allserver", connectionOptions, workerOptions }) {
+        if (!connectionOptions) {
+            connectionOptions = { host: "localhost", port: 6379 };
+        }
+        this._queueName = queueName || this._queueName;
+        this._connectionOptions = connectionOptions || this._connectionOptions;
+        this._workerOptions = workerOptions || this._workerOptions || {};
+        if (this._workerOptions.autorun === undefined) this._workerOptions.autorun = true;
+    },
+
+    methods: {
+        async startServer(defaultCtx) {
+            this._worker = new this.Worker(
+                this._queueName,
+                async (job) => {
+                    const ctx = { ...defaultCtx, bullmq: { job }, arg: job.data };
+                    return await ctx.allserver.handleCall(ctx);
+                },
+                {
+                    connection: this._connectionOptions,
+                    ...this._workerOptions,
+                }
+            );
+            return await this._worker.waitUntilReady();
+        },
+
+        reply(ctx) {
+            return ctx.result;
+        },
+
+        async stopServer() {
+            return this._worker.close();
+        },
+
+        getProcedureName(ctx) {
+            return ctx.bullmq.job.name;
+        },
+
+        isIntrospection(ctx) {
+            return this.getProcedureName(ctx) === "introspect"; // could be conflicting with procedure name(s)
+        },
+    },
+});

package/src/server/ExpressTransport.js

@@ -0,0 +1,74 @@
+const { parse: parseUrl } = require("url");
+
+module.exports = require("./Transport").compose({
+    name: "ExpressTransport",
+
+    props: {
+        _express: require("express"),
+    },
+
+    methods: {
+        async deserializeRequest(ctx) {
+            if (ctx.http.req.deserializationFailed) return false;
+            // If there is no body we will use request query (aka search params)
+            let arg = ctx.http.query;
+            if (ctx.http.req.body && Object.keys(ctx.http.req.body).length > 0) arg = ctx.http.req.body;
+            ctx.arg = arg;
+            return true;
+        },
+
+        async _handleRequest(ctx) {
+            if (await this.deserializeRequest(ctx)) {
+                await ctx.allserver.handleCall(ctx);
+            } else {
+                // HTTP protocol request was malformed (not expected structure).
+                // We are not going to process it.
+                ctx.result = { success: false, code: "ALLSERVER_BAD_REQUEST", message: "Can't parse JSON" };
+                ctx.http.statusCode = 400;
+                this.reply(ctx);
+            }
+        },
+
+        startServer(defaultCtx) {
+            return [
+                this._express.json(),
+                (err, req, res, next) => {
+                    if (err.statusCode) req.deserializationFailed = err; // overriding the error, processing the request as normal
+                    next();
+                },
+                async (req, res) => {
+                    const ctx = {
+                        ...defaultCtx,
+                        http: { req, res, query: req.query || {} },
+                    };
+
+                    await this._handleRequest(ctx);
+                },
+            ];
+        },
+
+        getProcedureName(ctx) {
+            return parseUrl(ctx.http.req.url).pathname.substr(1);
+        },
+
+        isIntrospection(ctx) {
+            return this.getProcedureName(ctx) === "";
+        },
+
+        prepareNotFoundReply(ctx) {
+            ctx.http.statusCode = 404;
+        },
+        prepareProcedureErrorReply(ctx) {
+            // Generic exception.
+            ctx.http.statusCode = 500;
+
+            // nodejs assert() exception. In HTTP world this likely means 400 "Bad Request".
+            if (ctx.error.code === "ERR_ASSERTION") ctx.http.statusCode = 400;
+        },
+
+        reply(ctx) {
+            if (!ctx.http.statusCode) ctx.http.statusCode = 200;
+            ctx.http.res.status(ctx.http.statusCode).json(ctx.result);
+        },
+    },
+});