fulmine.js 5.13.2 → 5.13.3

package/NOTICE

@@ -79,6 +79,37 @@ significant changes made to the original work:
   - The app is callable as a request listener, so http.createServer(app),
     supertest and anything else that invokes an app directly keeps working
     through a node:http shim.
+  - Routing rebuilt on Express 5 semantics: its path matcher, case-insensitive
+    matching by default, req.route, and assigning req.url inside a middleware
+    re-routes the rest of the stack.
+  - The routes listen() can read are compiled into one handler, with an overlap
+    analysis deciding which of them may skip the middleware chain, and a
+    self-check mode that serves the same application twice to show the compiled
+    path answers as the chain it stands in for.
+  - ETags are generated in this package rather than by the etag module, and an
+    "etag methods" setting limits them to the methods it names.
+  - express.compression() added, the compression module's middleware built in: a
+    body that arrives whole is compressed in one call, and partial content is
+    left alone.
+  - express.serverTiming() added, which reports on the response how the request
+    was routed.
+  - express.static serves the .br and .gz twins of a file with preCompressed.
+  - app.ws() serves websockets on the same port, with an upgrade hook and the
+    request on the socket.
+  - express({ cluster: "auto" }) forks one worker per core on the same port.
+  - express.Route and express.testing added, the second one asserting what
+    listen() decided about a route.
+  - X-Powered-By is not sent unless the application asks for it, and the header
+    methods, res.flushHeaders and the node members that were missing are filled
+    in.
+  - An adapter for NestJS ships as fulmine.js/nest.
+  - A fulmine command added: it profiles and explains what listen() worked out
+    about each route, verifies the machine, migrates a project, and prints the
+    differences from Express.
+  - Random applications are generated and compared against Express round by
+    round, with the request bytes, the methods that compute a header value and
+    keep-alive sequences each having their own fuzzer, and applications built on
+    four frameworks run as an integration suite.
   - Releases are built and published to npm from CI.
   - Benchmark harness reworked: wrk replaced by autocannon so the suite runs
     anywhere Node does, NODE_ENV is set, load errors and response validation are

package/README.md

@@ -44,7 +44,6 @@ See [Migrating](#migrating) for what it handles and what it deliberately does no
 - [Why this exists](#why-this-exists)
 - [Performance](#performance)
 - [Public benchmarks](#public-benchmarks)
-- [Attribution](#attribution)
 - [Difference from similar projects](#difference-from-similar-projects)
 - [Migrating](#migrating)
     - [Angular SSR](#angular-ssr)
@@ -69,6 +68,7 @@ See [Migrating](#migrating) for what it handles and what it deliberately does no
 - [Tested frameworks](#tested-frameworks)
 - [Tested view engines](#tested-view-engines)
 - [Examples](./examples/README.md)
+- [Attribution](#attribution)
 - [Working on Fulmine](./CONTRIBUTING.md)
 
 ## Why this exists
@@ -77,6 +77,8 @@ There are several fast HTTP servers for Node built on [µWebSockets.js](https://
 
 Compatibility here is not a claim, it is a test suite. Every test runs against real Express first and then against Fulmine, and the outputs have to match byte for byte. That is what makes `helmet`, `cors`, `passport`, `morgan`, `multer`, `express-session` and the rest of the ecosystem work rather than "mostly work". Express 5's own test suite runs against Fulmine too, and passes whole: 1130 passing, 0 failing at the pinned Express version.
 
+It started as a fork of [Ultimate Express](https://github.com/dimdenGD/ultimate-express), which is where the hard part was already done. See [Attribution](#attribution).
+
 ## Performance
 
 Fulmine is faster than Express where the framework itself is doing the work, and the same speed where it is not. Both halves of that sentence matter, so here is the honest version.
@@ -104,16 +106,6 @@ Numbers produced by a project about itself deserve suspicion, so Fulmine also st
 
 More to come as their maintainers take the entries in.
 
-## Attribution
-
-Fulmine is a derivative work of [Ultimate Express](https://github.com/dimdenGD/ultimate-express) by [@dimdenGD](https://github.com/dimdenGD), used under the Apache License 2.0. The full commit history is preserved, so the original authorship is visible in the repository itself.
-
-**Special thanks to [@dimdenGD](https://github.com/dimdenGD).** Ultimate Express is the hard part of this project, and it was already done before Fulmine existed. Everything here stands on that work.
-
-Fulmine is not affiliated with, endorsed by, or maintained by the authors of Ultimate Express. See [`NOTICE`](./NOTICE) for the list of significant changes.
-
-It is likewise not affiliated with the OpenJS Foundation or the Express.js project. Express is a trademark of the OpenJS Foundation.
-
 ## Difference from similar projects
 
 - **`ultimate-express`** is what Fulmine is derived from, and is the closest relative by far. It targets Express 4, keeps the v4 API surface and its deprecations. Fulmine targets Express 5 only, which removes the compatibility layer for everything v5 dropped, and is typed. If you are on Express 4, use `ultimate-express`.
@@ -927,6 +919,16 @@ npm install
 node websocket.js
 ```
 
+## Attribution
+
+Fulmine is a derivative work of [Ultimate Express](https://github.com/dimdenGD/ultimate-express) by [@dimdenGD](https://github.com/dimdenGD), used under the Apache License 2.0. The full commit history is preserved, so the original authorship is visible in the repository itself.
+
+**Special thanks to [@dimdenGD](https://github.com/dimdenGD).** Ultimate Express is the hard part of this project, and it was already done before Fulmine existed. Everything here stands on that work.
+
+Fulmine is not affiliated with, endorsed by, or maintained by the authors of Ultimate Express. See [`NOTICE`](./NOTICE) for the list of significant changes.
+
+It is likewise not affiliated with the OpenJS Foundation or the Express.js project. Express is a trademark of the OpenJS Foundation.
+
 ## Working on Fulmine
 
 How to run the suites, what each of them is for, and how to write a comparison test:

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "fulmine.js",
-    "version": "5.13.2",
+    "version": "5.13.3",
     "description": "Drop-in Express 5 replacement on uWebSockets.js. Your existing middleware keeps working.",
     "main": "src/index.js",
     "exports": {

package/src/request.js

@@ -17,7 +17,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-const { deprecated } = require("./utils.js");
+const { deprecated, fastQueryParse } = require("./utils.js");
 const accepts = require("accepts");
 const typeis = require("type-is");
 const parseRange = require("range-parser");
@@ -157,6 +157,29 @@ const READABLE_OPTIONS = { highWaterMark: 128 * 1024 };
 // `{"a":1}GET /path HTTP/1.1` is a request to it, with `{"A":1}GET` as the method. See _mustRefuse.
 const KNOWN_METHODS = new Set(require("http").METHODS);
 
+/**
+ * Whether a request target is bytes node's parser would have accepted, which is printable ASCII
+ * and nothing else.
+ *
+ * µWS takes the target as it finds it and decodes it as UTF-8, so `GET /café` arrives here
+ * as a path with an é in it and the overlong encoding of a slash arrives as replacement
+ * characters. Node refuses both with a 400 before any application sees them, and it has to: what
+ * reaches req.url otherwise is not what is on the wire, and a proxy in front reading the same
+ * bytes can disagree with this server about which path was asked for. Control characters are µWS's
+ * own to refuse and it does, so the test is one comparison per character rather than two.
+ *
+ * @param {string} target the path or the query string, as µWS decoded it
+ * @returns {boolean}
+ */
+function isAsciiTarget(target) {
+    for (let i = 0; i < target.length; i++) {
+        if (target.charCodeAt(i) > 0x7e) {
+            return false;
+        }
+    }
+    return true;
+}
+
 /**
  * Whether a transfer-encoding leaves the body's length knowable, which is RFC 9112's rule that
  * `chunked` comes last. `gzip, chunked` is fine and `chunked, gzip` is not: with a coding applied
@@ -174,7 +197,65 @@ function endsWithChunked(value) {
     const last = value.slice(value.lastIndexOf(",") + 1).trim();
     // a coding may carry parameters, which are not part of its name
     const semicolon = last.indexOf(";");
-    return (semicolon === -1 ? last : last.slice(0, semicolon)).trim().toLowerCase() === "chunked";
+    if ((semicolon === -1 ? last : last.slice(0, semicolon)).trim().toLowerCase() !== "chunked") {
+        return false;
+    }
+    // and only once. "chunked, chunked" ends with it and is still nonsense: a sender may not frame
+    // a body twice, and where node refuses the request outright µWS frames it as one chunked body
+    // and reads whatever follows as the next request on the connection
+    const codings = value.split(",");
+    let chunkedCount = 0;
+    for (const coding of codings) {
+        const parameter = coding.indexOf(";");
+        if ((parameter === -1 ? coding : coding.slice(0, parameter)).trim().toLowerCase() === "chunked") {
+            chunkedCount++;
+        }
+    }
+    return chunkedCount === 1;
+}
+
+/**
+ * Whether a Connection header says the connection ends with this response.
+ *
+ * It is a list, and "keep-alive, close" closes as much as "close" alone does. Compared against an
+ * exact "close", this server kept a connection the client had said it was done with, and then read
+ * the bytes after it as another request: node closes there, so the two disagreed on how many
+ * requests the same bytes carried, which is what a desync is.
+ *
+ * Written as a scan rather than a split and a lowercase, because almost every request that carries
+ * this header carries "keep-alive", and both of those allocate per request.
+ *
+ * @param {string} value as µWS hands it over
+ * @returns {boolean}
+ */
+function saysClose(value) {
+    const length = value.length;
+    let at = 0;
+    while (at < length) {
+        while (at < length && (value.charCodeAt(at) === 0x20 || value.charCodeAt(at) === 0x09)) {
+            at++;
+        }
+        const start = at;
+        while (at < length && value.charCodeAt(at) !== 0x2c) {
+            at++;
+        }
+        let end = at;
+        while (end > start && (value.charCodeAt(end - 1) === 0x20 || value.charCodeAt(end - 1) === 0x09)) {
+            end--;
+        }
+        if (
+            end - start === 5 &&
+            (value.charCodeAt(start) | 0x20) === 0x63 &&
+            (value.charCodeAt(start + 1) | 0x20) === 0x6c &&
+            (value.charCodeAt(start + 2) | 0x20) === 0x6f &&
+            (value.charCodeAt(start + 3) | 0x20) === 0x73 &&
+            (value.charCodeAt(start + 4) | 0x20) === 0x65
+        ) {
+            return true;
+        }
+        at++;
+    }
+    return false;
 }
 
 /**
@@ -401,12 +482,7 @@ module.exports = class Request extends LazyReadable {
         // spotted in the loop that is running anyway: a client asking for the connection to be
         // closed must not be answered that it is being kept alive. The response is built right
         // after this and reads the flag.
-        if (
-            headerKey.length === 10 &&
-            headerKey === "connection" &&
-            value.length === 5 &&
-            value.toLowerCase() === "close"
-        ) {
+        if (headerKey.length === 10 && headerKey === "connection" && saysClose(value)) {
             r._connectionClose = true;
         } else if (
             (headerKey.length === 14 && headerKey === "content-length") ||
@@ -644,7 +720,7 @@ module.exports = class Request extends LazyReadable {
                 const connection = req.getHeader("connection");
                 if (connection !== "") {
                     entries.push("connection", connection);
-                    if (connection.length === 5 && connection.toLowerCase() === "close") {
+                    if (saysClose(connection)) {
                         this._connectionClose = true;
                     }
                 }
@@ -689,6 +765,9 @@ module.exports = class Request extends LazyReadable {
             const rawQuery = req.getQuery();
             this._rawQuery = rawQuery ?? "";
             this.urlQuery = rawQuery === undefined ? "" : "?" + rawQuery;
+            if (rawQuery !== undefined && rawQuery.length !== 0 && !isAsciiTarget(rawQuery)) {
+                this._mustRefuse = true;
+            }
         }
         if (preset) {
             // the registration's constants: two native crossings and their strings not asked for
@@ -707,6 +786,12 @@ module.exports = class Request extends LazyReadable {
             // again. Building originalUrl and picking the path back out of it with indexOf and
             // substring was a search and a second string for something uWS had just handed over.
             this._path = req.getUrl();
+            // the target as it arrived, which node would have refused before this ran. A preset
+            // needs no check: it is a literal registration, and µWS only matched it because the
+            // bytes were that literal
+            if (!isAsciiTarget(this._path)) {
+                this._mustRefuse = true;
+            }
             this.originalUrl = this._path + this.urlQuery;
             this.url = this.originalUrl;
             // what the router last wrote to req.url. A middleware assigning something else is a
@@ -1183,11 +1268,21 @@ module.exports = class Request extends LazyReadable {
         // the vendored default already answers on a bare null prototype, so it goes out as is;
         // any other parser is copied onto one, which is what kept fast-querystring's result from
         // inspecting as "Empty <[Object: null prototype] {}>" where Express shows the bare form
-        return qp
-            ? qp === parseQuery
-                ? parseQuery(this._rawQuery)
-                : Object.assign(Object.create(null), qp(this._rawQuery))
-            : Object.create(null);
+        // A parser of the application's own is handed what express hands it, which is
+        // parseurl's `query`: null when the url carries no "?" at all, and the text after it
+        // otherwise, the empty string included. Passing "" for both meant a parser written for
+        // express, which may check for null before it reads the string, saw a request that had no
+        // query as one with an empty query. The two built in parsers take the raw string.
+        if (!qp) {
+            return Object.create(null);
+        }
+        if (qp === parseQuery) {
+            return parseQuery(this._rawQuery);
+        }
+        if (qp === fastQueryParse) {
+            return Object.assign(Object.create(null), fastQueryParse(this._rawQuery));
+        }
+        return Object.assign(Object.create(null), qp(this.urlQuery === "" ? null : this._rawQuery));
     }
 
     /**

package/src/response.js

@@ -790,13 +790,24 @@ module.exports = class Response extends LazyWritable {
             this.writeHeaders(true);
         }
         const contentLength = this.headers["content-length"];
+        // The client said this connection ends here, and it is this end() that has to make it so.
+        // µWS closes by itself for a bare "close" and not for a list, so "keep-alive, close" left
+        // the socket open and the bytes after that request were read as another one: node closes
+        // there, and a server that does not is a server the client and it disagree with about how
+        // many requests were sent. See saysClose.
+        //
+        // Only where a length goes out with it. endWithoutBody takes the flag as its second
+        // argument and reads the first as the length whatever it holds, so asking it to close
+        // without one writes "Content-Length: 9223372036854775808" onto a 204. Those two paths keep
+        // µWS's own rule, which closes for a bare "close" and not for a list.
+        const closeConnection = this.req._connectionClose === true;
         if (STATUSES_WITHOUT_BODY.has(this.statusCode) || this.statusCode < 200) {
             // no body and no length describing one, whatever the caller passed. node decides
             // this the same way, from the status alone, so res.status(304).end("x") sends the
             // status and nothing else on either.
             this._res.endWithoutBody();
         } else if (!data && contentLength) {
-            this._res.endWithoutBody(contentLength.toString());
+            this._res.endWithoutBody(contentLength.toString(), closeConnection);
         } else if (headWasAlreadyOut && this.chunkedTransfer) {
             // whatever is still queued goes first: end() must not overtake the body written before it
             this.#flushQueued(null);
@@ -817,7 +828,7 @@ module.exports = class Response extends LazyWritable {
             if (this.req.method === "HEAD") {
                 const length = Buffer.byteLength(data ?? "");
                 this.headers["content-length"] = String(length);
-                this._res.endWithoutBody(length.toString());
+                this._res.endWithoutBody(length.toString(), closeConnection);
             } else {
                 // remembered rather than measured: only a caller that asks for content-length pays
                 // for it, and uWS is measuring the same bytes for the wire anyway
@@ -825,7 +836,7 @@ module.exports = class Response extends LazyWritable {
                 // and null is sent as the empty body it means. uWS answers end(null) with a
                 // response the client never sees the end of, where node and express send an empty
                 // 200: res.end(null) is what the compression module's own test suite does
-                this._res.end(data ?? "");
+                this._res.end(data ?? "", closeConnection);
             }
         }
 

package/src/router.js

@@ -1790,6 +1790,55 @@ module.exports = class Router extends EventEmitter {
         method = method.toUpperCase();
         callbacks = callbacks.flat(Infinity);
         checkHandlers(callbacks);
+        // What express hangs off req.route as its methods, and the three registrations do not
+        // agree on it: app.all() registers every verb one at a time, so the map names all of
+        // them; router.all() and app.route().all() mark the route _all instead; and everything
+        // hung off one app.route() shares one map, since express builds one Route for the lot.
+        // Built in node's own order, which is the order the methods package hands express, so
+        // the map reads back key for key as express's does.
+        let methodMap;
+        let stack;
+        if (method !== "USE") {
+            methodMap = this._pendingGroupMethods ?? new NullObject();
+            // and the layers behind them, which is express's Route#stack: one per handler per verb
+            // the route was registered for, in the order express pushes them. app.all() therefore
+            // has one for every verb, since that is how many times express registers the handler
+            stack = this._pendingGroupStack ?? [];
+            let verbs;
+            if (method === "ALL") {
+                if (this._isApplication && this._pendingGroup === undefined) {
+                    verbs = [];
+                    for (const known of METHODS) {
+                        const lowered = known.toLowerCase();
+                        methodMap[lowered] = true;
+                        verbs.push(lowered);
+                    }
+                } else {
+                    methodMap._all = true;
+                    // Route#all leaves the layer without one, and express reads that as any verb
+                    verbs = [undefined];
+                }
+            } else {
+                methodMap[method.toLowerCase()] = true;
+                verbs = [method.toLowerCase()];
+            }
+            for (const verb of verbs) {
+                for (const handle of callbacks) {
+                    stack.push({
+                        handle,
+                        name: handle.name || "<anonymous>",
+                        params: undefined,
+                        path: undefined,
+                        keys: [],
+                        method: verb
+                    });
+                }
+            }
+        }
+        // Several paths at once are one route to express, whose path is the array it was given,
+        // and several here, one per path, so they share the map and the stack and read back with
+        // the array as their path.
+        const writtenPath = path;
         const paths = Array.isArray(path) ? path : [path];
         const routes = [];
         for (let path of paths) {
@@ -1843,6 +1892,12 @@ module.exports = class Router extends EventEmitter {
                 regexMount: method === "USE" && path instanceof RegExp,
                 // written by the application, so express matches it as it stands
                 userRegexp: path instanceof RegExp,
+                // express reads these off req.route, and a middleware has none: see _preprocessRequest
+                methods: methodMap,
+                stack,
+                // the route as a request sees it, which is the route itself unless the path was
+                // normalised. Written into the literal so every route keeps one shape
+                exposed: /** @type {any} */ (undefined),
                 routeKey: routeKey++,
                 // which app.route() this came from, when it came from one, so the routes it built
                 // count as one route where an error is concerned. undefined for every other route
@@ -1859,6 +1914,17 @@ module.exports = class Router extends EventEmitter {
                 all: method === "ALL" || method === "USE",
                 gettable: method === "GET" || method === "HEAD"
             };
+            // Everything here matches on the normalised path, and express hands out the written
+            // one: a route registered as "/users/" is matched as "/users" with strict routing off
+            // and still reads back with its slash. Rather than carry two paths through the
+            // optimizer, a route whose path was normalised gets a view of itself with the written
+            // path on top, and that is the one the request is given.
+            route.exposed = route;
+            if (writtenPath !== path) {
+                const view = Object.create(route);
+                view.path = writtenPath;
+                route.exposed = view;
+            }
             if (
                 route.pattern instanceof RegExp &&
                 // a RegExp the application wrote: its capture groups are params too
@@ -2643,7 +2709,13 @@ module.exports = class Router extends EventEmitter {
      * @returns {any} a promise only when a param callback is involved
      */
     _preprocessRequest(req, res, route) {
-        req.route = route;
+        // express sets this inside Route#dispatch, so only a route ever writes one: a middleware
+        // reads undefined there, and so does a request nothing routed. Code that tells a route
+        // from a middleware by asking for req.route, which is how a metric gets its name, read
+        // the mount here and named itself after it
+        if (route.use !== true) {
+            req.route = route.exposed;
+        }
         // both, not the route flag alone: the flag says the route was registered natively, the
         // values say this request came in that way
         if (route.optimizedParams && req.optimizedParams) {
@@ -2966,12 +3038,24 @@ module.exports = class Router extends EventEmitter {
         // far as an error is concerned, see errorHop
         const group = ++routeGroups;
         const fns = new NullObject();
+        // one map for the whole chain, because express builds one Route for it: a request answered
+        // by the get() of an app.route() reads post() in its req.route.methods too
+        const groupMethods = new NullObject();
+        const groupStack = [];
+        // express hands back a Route, which carries these three beside the verb methods
+        fns.path = path;
+        fns.methods = groupMethods;
+        fns.stack = groupStack;
         const inGroup = (method, callbacks) => {
             this._pendingGroup = group;
+            this._pendingGroupMethods = groupMethods;
+            this._pendingGroupStack = groupStack;
             try {
                 return this.createRoute(method, path, /** @type {any} */ (fns), ...callbacks);
             } finally {
                 this._pendingGroup = undefined;
+                this._pendingGroupMethods = undefined;
+                this._pendingGroupStack = undefined;
             }
         };
         for (const method of methods) {