@zone-eu/mailsplit 5.4.26 → 5.4.28

package/README.md

@@ -29,9 +29,9 @@ let splitter = new Splitter(options);
 
 Where
 
--   **options** is an optional options object
-    -   **options.ignoreEmbedded** (boolean, defaults to false) if true then treat message/rfc822 node as normal leaf node and do not try to parse it
-    -   **options.maxHeadSize** (number, defaults to 1 MiB) limits the header size of a MIME node in bytes, every header line counting for at least 64 bytes
+- **options** is an optional options object
+    - **options.ignoreEmbedded** (boolean, defaults to false) if true then treat message/rfc822 node as normal leaf node and do not try to parse it
+    - **options.maxHeadSize** (number, defaults to 1 MiB) limits the header size of a MIME node in bytes, every header line counting for at least 64 bytes
 
 #### Events
 
@@ -39,13 +39,13 @@ Where
 
 #### Data objects
 
--   **type**
-    -   `'node'` means that we reached the next mime node and the previous one is completely processed
-    -   `'data'` provides us multipart body parts, including boundaries. This data is not directly related to any specific multipart node, basically it includes everything between the end of one normal node and the header of next node
-    -   `'body'` provides us next chunk for the last seen `'node'` element
--   **value** is a buffer value for `'body'` and `'data'` parts
--   **getDecoder()** is a function that returns a stream object you can use to decode node contents. Write data from 'body' to decoder and read decoded Buffer value out from it
--   **getEncoder()** is a function that returns a stream object you can use to encode node contents. Write buffer data to encoder and read encoded object value out that you can pass to a Joiner
+- **type**
+    - `'node'` means that we reached the next mime node and the previous one is completely processed
+    - `'data'` provides us multipart body parts, including boundaries. This data is not directly related to any specific multipart node, basically it includes everything between the end of one normal node and the header of next node
+    - `'body'` provides us next chunk for the last seen `'node'` element
+- **value** is a buffer value for `'body'` and `'data'` parts
+- **getDecoder()** is a function that returns a stream object you can use to decode node contents. Write data from 'body' to decoder and read decoded Buffer value out from it
+- **getEncoder()** is a function that returns a stream object you can use to encode node contents. Write buffer data to encoder and read encoded object value out that you can pass to a Joiner
 
 Element with type `'node'` has a bunch of header related methods and properties, see [below](#manipulating-headers).
 
@@ -82,31 +82,31 @@ someMessagStream.pipe(splitter);
 
 If the data object has `type='node'` then you can modify headers for that node (headers can be modified until the data object is passed over to a `Joiner`)
 
--   **node.getHeaders()** returns a Buffer value with generated headers. If you have not modified the headers object in any way then you should get the exact copy of the original. In case you have done something (for example removed a key, or added a new header key), then all linebreaks are forced to <CR><LF> even if the original headers used just <LF>
--   **node.setContentType(contentType)** sets or updates mime type for the node
--   **node.setCharset(charset)** sets or updates character set in the Content-Type header
--   **node.setFilename(filename)** sets or updates filename in the Content-Disposition header (unicode allowed)
+- **node.getHeaders()** returns a Buffer value with generated headers. If you have not modified the headers object in any way then you should get the exact copy of the original. In case you have done something (for example removed a key, or added a new header key), then all linebreaks are forced to <CR><LF> even if the original headers used just <LF>
+- **node.setContentType(contentType)** sets or updates mime type for the node
+- **node.setCharset(charset)** sets or updates character set in the Content-Type header
+- **node.setFilename(filename)** sets or updates filename in the Content-Disposition header (unicode allowed)
 
 You can manipulate specific header keys as well using the `headers` object
 
--   **node.headers.get(key)** returns an array of strings with all header rows for the selected key (these are full header lines, so key name is part of the row string, eg `["Subject: This is subject line"]`)
--   **node.headers.getFirst(key)** returns string value of the specified header key (eg `"This is subject line"`)
--   **node.headers.hasHeader(key)** returns boolean value if the specified header key exists
--   **node.headers.add(key, value [,index])** adds a new header value to the specified index or to the top of the header block if index is not specified
--   **node.headers.update(key, value, [,relativeKeyIndex])** replaces a header value to the specified relative key index (note that relative key index means relative to the same header key, eg if multiple exist and you specify `1` as the value, then it will update the second) or if no relative key index is specified, then it will remove all header value matches found for the key and append one at the last matching key index found with the specified value. If a relative key index is specified and it does not exist then it will be replaced (eg if there are two headers of `X-Foo-Bar` and you pass `2`, meaning it will update the third, no updates will be made since the third did not exist)
--   **node.headers.remove(key)** remove header value
--   **node.headers.mbox** If this is a MBOX formatted message then this value holds the prefix line (eg. "From MAILER-DAEMON Fri Jul 8 12:08:34 2011")
--   **node.headers.mbox** If this is a POST form-data then this value holds the HTTP prefix line (eg. "POST /upload.php HTTP/1.1")
+- **node.headers.get(key)** returns an array of strings with all header rows for the selected key (these are full header lines, so key name is part of the row string, eg `["Subject: This is subject line"]`)
+- **node.headers.getFirst(key)** returns string value of the specified header key (eg `"This is subject line"`)
+- **node.headers.hasHeader(key)** returns boolean value if the specified header key exists
+- **node.headers.add(key, value [,index])** adds a new header value to the specified index or to the top of the header block if index is not specified
+- **node.headers.update(key, value, [,relativeKeyIndex])** replaces a header value to the specified relative key index (note that relative key index means relative to the same header key, eg if multiple exist and you specify `1` as the value, then it will update the second) or if no relative key index is specified, then it will remove all header value matches found for the key and append one at the last matching key index found with the specified value. If a relative key index is specified and it does not exist then it will be replaced (eg if there are two headers of `X-Foo-Bar` and you pass `2`, meaning it will update the third, no updates will be made since the third did not exist)
+- **node.headers.remove(key)** remove header value
+- **node.headers.mbox** If this is a MBOX formatted message then this value holds the prefix line (eg. "From MAILER-DAEMON Fri Jul 8 12:08:34 2011")
+- **node.headers.mbox** If this is a POST form-data then this value holds the HTTP prefix line (eg. "POST /upload.php HTTP/1.1")
 
 Additionally you can check the details of the node with the following properties automatically parsed from the headers:
 
--   **node.root** if true then it means this is the message root, so this node should contain Subject, From, To etc. headers
--   **node.contentType** returns the mime type of the node (eg. 'text/html')
--   **node.disposition** either `'attachment'`, `'inline'` or `false` if not set
--   **node.charset** returns the charset of the node as defined in 'Content-Type' header (eg. 'UTF-8') or false if not defined
--   **node.encoding** returns the Transfer-Encoding value (eg. 'base64' or 'quoted-printable') or false if not defined
--   **node.multipart** if has value, then this is a multipart node (does not have 'body' parts)
--   **node.filename** is set if the headers contain a filename value. This is decoded to unicode, so it is a normal string or false if not found
+- **node.root** if true then it means this is the message root, so this node should contain Subject, From, To etc. headers
+- **node.contentType** returns the mime type of the node (eg. 'text/html')
+- **node.disposition** either `'attachment'`, `'inline'` or `false` if not set
+- **node.charset** returns the charset of the node as defined in 'Content-Type' header (eg. 'UTF-8') or false if not defined
+- **node.encoding** returns the Transfer-Encoding value (eg. 'base64' or 'quoted-printable') or false if not defined
+- **node.multipart** if has value, then this is a multipart node (does not have 'body' parts)
+- **node.filename** is set if the headers contain a filename value. This is decoded to unicode, so it is a normal string or false if not found
 
 ### Join parsed message stream
 
@@ -118,10 +118,7 @@ let Joiner = require('mailsplit').Joiner;
 let splitter = new Splitter();
 let joiner = new Joiner();
 // pipe a message source to splitter, then joiner and finally to stdout
-someMessagStream
-    .pipe(splitter)
-    .pipe(joiner)
-    .pipe(process.stdout);
+someMessagStream.pipe(splitter).pipe(joiner).pipe(process.stdout);
 ```
 
 ### Rewrite specific nodes
@@ -130,14 +127,14 @@ someMessagStream
 
 Rewriter takes the following argument:
 
--   **filterFunc** gets the current node as argument and starts processing it if `filterFunc` returns true
+- **filterFunc** gets the current node as argument and starts processing it if `filterFunc` returns true
 
 Once Rewriter finds a matching node, it emits the following event:
 
--   _'node'_ with an object argument `data`
-    -   `data.node` includes the current node with headers
-    -   `data.decoder` is the decoder stream that you can read data from
-    -   `data.encoder` is the encoder stream that you can write data to. Whatever you write into that stream will be encoded properly and inserted as the content of the current node
+- _'node'_ with an object argument `data`
+    - `data.node` includes the current node with headers
+    - `data.decoder` is the decoder stream that you can read data from
+    - `data.encoder` is the encoder stream that you can write data to. Whatever you write into that stream will be encoded properly and inserted as the content of the current node
 
 ```javascript
 let Splitter = require('mailsplit').Splitter;
@@ -153,11 +150,7 @@ rewriter.on('node', data => {
     data.decoder.pipe(data.encoder);
 });
 // pipe a message source to splitter, then rewriter, then joiner and finally to stdout
-someMessagStream
-    .pipe(splitter)
-    .pipe(rewriter)
-    .pipe(joiner)
-    .pipe(process.stdout);
+someMessagStream.pipe(splitter).pipe(rewriter).pipe(joiner).pipe(process.stdout);
 ```
 
 ### Stream specific nodes
@@ -166,14 +159,14 @@ someMessagStream
 
 Streamer takes the following argument:
 
--   **filterFunc** gets the current node as argument and starts processing it if `filterFunc` returns true
+- **filterFunc** gets the current node as argument and starts processing it if `filterFunc` returns true
 
 Once Streamer finds a matching node, it emits the following event:
 
--   _'node'_ with an object argument `data`
-    -   `data.node` includes the current node with headers (informational only, you can't modify it)
-    -   `data.decoder` is the decoder stream that you can read data from
-    -   `data.done` is a function you must call once you have processed the stream
+- _'node'_ with an object argument `data`
+    - `data.node` includes the current node with headers (informational only, you can't modify it)
+    - `data.decoder` is the decoder stream that you can read data from
+    - `data.done` is a function you must call once you have processed the stream
 
 ```javascript
 let Splitter = require('mailsplit').Splitter;
@@ -189,11 +182,7 @@ streamer.on('node', data => {
     data.done();
 });
 // pipe a message source to splitter, then streamer, then joiner and finally to stdout
-someMessagStream
-    .pipe(splitter)
-    .pipe(streamer)
-    .pipe(joiner)
-    .pipe(process.stdout);
+someMessagStream.pipe(splitter).pipe(streamer).pipe(joiner).pipe(process.stdout);
 ```
 
 ### Benchmark
@@ -211,4 +200,4 @@ Done. 20000 messages [1244 MB] processed in 55.882 s. with average of 358 messag
 
 Dual licensed under **MIT** or **EUPLv1.1+**
 
-> `mailsplit` is part of the Zone Mail Suite (ZMS). Suite of programs and modules for an efficient, fast and modern email server.
+> `mailsplit` is part of the Zone Mail Suite (ZMS). Suite of programs and modules for an efficient, fast and modern email server.

package/lib/message-splitter.js

@@ -20,9 +20,17 @@ const MAX_PENDING_LINE_SIZE = 64 * 1024;
 // how many separate writes the pending line may be kept in before it is compacted
 const MAX_PENDING_LINE_CHUNKS = 1024;
 
+// The most SP and HTAB a delimiter line may carry between the boundary value, or its closing
+// "--", and the line ending. RFC 2046 5.1.1 allows the whitespace, and Python and postal-mime
+// accept any amount of it, so a line that ends in a few spaces delimits the same parts here
+// as there. The line still has to fit a header line (RFC 5322 2.1.1), so the pending line
+// kept as a possible delimiter stays bounded
+const BOUNDARY_LINE_TRAILING_WSP = 998;
+
 // what a delimiter line may carry after the boundary value: the "--" prefix, an optional
-// "--" suffix and the line terminator. This is the bound compareBoundary() accepts.
-const BOUNDARY_LINE_SUFFIX = 2 /* "--" prefix */ + 2 /* "--" suffix */ + 2; /* trailing <CR><LF> */
+// "--" suffix, the trailing whitespace and the line terminator. This is the bound
+// compareBoundary() accepts.
+const BOUNDARY_LINE_SUFFIX = 2 /* "--" prefix */ + 2 /* "--" suffix */ + BOUNDARY_LINE_TRAILING_WSP + 2; /* trailing <CR><LF> */
 
 // checkBoundary() additionally allows a line ending in front of the delimiter, so this is
 // the longest a delimiter line can ever be once the boundary value is subtracted
@@ -389,7 +397,8 @@ class MessageSplitter extends Transform {
      * @returns {1 | 2 | false}
      */
     compareBoundary(line, startpos, boundary) {
-        // --{boundary}\r\n or --{boundary}--\r\n
+        // --{boundary}\r\n or --{boundary}--\r\n, with SP and HTAB allowed in front of the
+        // line ending (RFC 2046 5.1.1), where Python and postal-mime accept them as well
         if (line.length < boundary.length + 3 + startpos || line.length > boundary.length + BOUNDARY_LINE_SUFFIX + startpos) {
             return false;
         }
@@ -399,34 +408,37 @@ class MessageSplitter extends Transform {
             }
         }
 
-        let pos = 0;
-        for (let i = boundary.length + 2 + startpos; i < line.length; i++) {
-            let c = line[i];
-            if (pos === 0 && (c === 0x0d || c === 0x0a)) {
-                // 1: next node
-                return 1;
-            }
-            if (pos === 0 && c !== 0x2d) {
-                // expecting "-"
-                return false;
-            }
-            if (pos === 1 && c !== 0x2d) {
-                // expecting "-"
-                return false;
-            }
-            if (pos === 2 && c !== 0x0d && c !== 0x0a) {
-                // expecting line terminator, either <CR> or <LF>
+        let i = boundary.length + 2 + startpos;
+        // 1: next node, 2: multipart end
+        let result = /** @type {1 | 2} */ (1);
+        if (line[i] === 0x2d) {
+            if (line[i + 1] !== 0x2d) {
+                // expecting "--"
                 return false;
             }
-            if (pos === 3 && c !== 0x0a) {
-                // expecting line terminator <LF>
-                return false;
-            }
-            pos++;
+            result = 2;
+            i += 2;
+        }
+
+        while (i < line.length && (line[i] === 0x20 || line[i] === 0x09)) {
+            i++;
         }
 
-        // 2: multipart end
-        return 2;
+        if (i === line.length) {
+            // only the closing delimiter may end the input without a line ending
+            return result === 2 ? 2 : false;
+        }
+
+        // expecting the line terminator and nothing after it: <CR><LF>, <CR> or <LF>, where
+        // the <CR> may be a run of them, the artifact of a second CR to CRLF conversion,
+        // which postal-mime reads as the line ending as well
+        while (i < line.length && line[i] === 0x0d) {
+            i++;
+        }
+        if (i < line.length && line[i] === 0x0a) {
+            i++;
+        }
+        return i === line.length ? result : false;
     }
 
     /**

package/lib/mime-node.d.ts

@@ -46,7 +46,7 @@ declare class MimeNode implements MimeNodeShape {
     /** Decoded filename from Content-Disposition or Content-Type parameters, or `false` when absent. */
     filename: string | false;
 
-    /** Whether this node is `text/*` with `format=flowed`. */
+    /** Whether this node is `text/plain` with `format=flowed`. */
     flowed: boolean;
 
     /** Whether flowed text uses `delsp=yes`. */

package/lib/mime-node.js

@@ -27,6 +27,9 @@ const HEADER_LINE_MIN_COST = 64;
  * `value.replace(/\(.*\)/g, '')`, but in linear time. The regex backtracks quadratically
  * on a value with many `(` characters and no closing `)`.
  *
+ * Used for the Content-Transfer-Encoding only. The MIME headers with parameters go through
+ * libmime's parseHeaderValue with stripComments, which knows where a comment may stand.
+ *
  * @param {string} value
  * @returns {string}
  */
@@ -73,7 +76,11 @@ function rfc2231ParamNames(value) {
         if (eq < 0) {
             return;
         }
-        let match = segment.substring(0, eq).trim().toLowerCase().match(/^(.+?)\*(\d+\*?)?$/);
+        let match = segment
+            .substring(0, eq)
+            .trim()
+            .toLowerCase()
+            .match(/^(.+?)\*(\d+\*?)?$/);
         if (match) {
             names.add(match[1]);
         }
@@ -212,8 +219,14 @@ class MimeNode {
         this._headersLines = [];
         let headers = this.headers;
 
+        // The comments of the MIME headers are dropped, so that `boundary=b (comment)` names
+        // the boundary `b` that postal-mime and Python's email.policy.default split by. Read as
+        // content, it matched no delimiter line and the parts both of them find stayed hidden
+        // here. Python's legacy compat32 policy, the default of message_from_bytes() without a
+        // policy, keeps the comment and splits by `--b (comment)` lines instead: a split
+        // against it is the price of agreeing with the two readers a renderer is built on
         let dispositionHeader = headers.getFirst('Content-Disposition');
-        this._parsedContentDisposition = this.libmime.parseHeaderValue(dispositionHeader);
+        this._parsedContentDisposition = this.libmime.parseHeaderValue(dispositionHeader, { stripComments: true });
         let parsedContentDisposition = /** @type {StructuredHeader} */ (this._parsedContentDisposition);
 
         // if content-type is missing default to plaintext
@@ -236,9 +249,12 @@ class MimeNode {
             }
         }
 
-        this._parsedContentType = this.libmime.parseHeaderValue(contentHeader);
+        this._parsedContentType = this.libmime.parseHeaderValue(contentHeader, { stripComments: true });
         let parsedContentType = /** @type {StructuredHeader} */ (this._parsedContentType);
 
+        // The encoding is a bare token, and one written in quotes stays quoted, so that
+        // "base64" is not decoded, as Python and postal-mime do not decode it. parseHeaderValue
+        // would unquote it, so the transfer encoding keeps its own comment stripper
         this.encoding = stripComments(headers.getFirst('Content-Transfer-Encoding')).toLowerCase().trim();
         this.contentType = (parsedContentType.value || '').toLowerCase().trim() || false;
         this.charset = parsedContentType.params.charset || false;
@@ -248,14 +264,18 @@ class MimeNode {
         if (this.disposition) {
             try {
                 this.disposition = this.libmime.decodeWords(this.disposition);
-            } catch (E) {
+            } catch (_E) {
                 // failed to parse disposition, keep as is (most probably an unknown charset is used)
             }
         }
 
         this.filename = parsedContentDisposition.params.filename || parsedContentType.params.name || false;
 
-        if (parsedContentType.params.format && parsedContentType.params.format.toLowerCase().trim() === 'flowed') {
+        // format=flowed is defined for text/plain (RFC 3676), and a browser never unfolds an
+        // html part. Flagged on any text part, a text/html part with delsp=yes had its
+        // `<scri \r\npt>` joined into a script tag by the consumer that unfolded it, a tag
+        // neither Python nor a browser given the raw bytes sees
+        if (this.contentType === 'text/plain' && parsedContentType.params.format && parsedContentType.params.format.toLowerCase().trim() === 'flowed') {
             this.flowed = true;
             if (parsedContentType.params.delsp && parsedContentType.params.delsp.toLowerCase().trim() === 'yes') {
                 this.delSp = true;
@@ -270,7 +290,7 @@ class MimeNode {
             if (!decoded) {
                 try {
                     this.filename = this.libmime.decodeWords(this.filename);
-                } catch (E) {
+                } catch (_E) {
                     // failed to parse filename, keep as is (most probably an unknown charset is used)
                 }
             }
@@ -281,7 +301,16 @@ class MimeNode {
                 this.contentType.substr(0, this.contentType.indexOf('/')) === 'multipart' &&
                 this.contentType.substr(this.contentType.indexOf('/') + 1)) ||
             false;
-        this._boundary = (parsedContentType.params.boundary && Buffer.from(parsedContentType.params.boundary)) || false;
+        // A boundary is ASCII and does not end in whitespace (RFC 2046 5.1.1, bchars), and
+        // Python matches no line against one that is not. One holding a non-ASCII character
+        // came out of a header read as latin1 or as UTF-8, and matched under `--ab\xC3\xA9cd`
+        // lines parts that a scanner built on Python never saw. Such a boundary counts as
+        // none, as a boundary parameter that is missing does. Trailing whitespace is dropped
+        // as Python drops it, a delimiter line may end in whitespace anyway
+        let boundary = (parsedContentType.params.boundary || '').trimEnd();
+        // a non-ASCII character takes more than one byte in UTF-8
+        let ascii = Buffer.byteLength(boundary) === boundary.length;
+        this._boundary = boundary && ascii ? Buffer.from(boundary) : false;
 
         this.rfc822 = this.contentType === 'message/rfc822';
 

package/lib/types.d.ts

@@ -107,7 +107,7 @@ export interface MimeNode {
     /** Decoded filename from Content-Disposition or Content-Type parameters, or `false` when absent. */
     filename: string | false;
 
-    /** Whether this node is `text/*` with `format=flowed`. */
+    /** Whether this node is `text/plain` with `format=flowed`. */
     flowed: boolean;
 
     /** Whether flowed text uses `delsp=yes`. */

package/package.json

@@ -1,38 +1,43 @@
 {
     "name": "@zone-eu/mailsplit",
-    "version": "5.4.26",
+    "version": "5.4.28",
     "description": "Split email messages into an object stream",
     "main": "index.js",
     "types": "index.d.ts",
+    "engines": {
+        "node": ">=20.0.0"
+    },
     "directories": {
         "test": "test"
     },
     "scripts": {
-        "test": "grunt",
-        "update": "rm -rf node_modules package-lock.json && ncu -u && npm install",
-        "typecheck": "tsc -p tsconfig.json"
+        "test": "node --test test/*-test.js",
+        "test:cjs": "NODE_OPTIONS=--no-experimental-require-module npm test",
+        "test:coverage": "c8 npm test",
+        "lint": "npm run typecheck && eslint .",
+        "lint:fix": "eslint . --fix",
+        "format": "prettier --write .",
+        "format:check": "prettier --check .",
+        "typecheck": "tsc -p tsconfig.json",
+        "update": "rm -rf node_modules package-lock.json && ncu -u && npm install"
     },
     "author": "Andris Reinman",
     "license": "(MIT OR EUPL-1.1+)",
     "dependencies": {
         "libbase64": "1.4.1",
-        "libmime": "5.5.1",
+        "libmime": "5.6.3",
         "libqp": "3.0.1"
     },
     "devDependencies": {
-        "@types/eslint": "9.6.1",
-        "@types/eslint-config-prettier": "6.11.3",
-        "@types/grunt": "0.4.32",
+        "@eslint/js": "10.0.1",
         "@types/libmime": "5.3.0",
         "@types/libqp": "1.1.3",
         "@types/node": "26.6.5",
-        "eslint": "8.29.0",
-        "eslint-config-nodemailer": "1.2.0",
-        "eslint-config-prettier": "9.1.0",
-        "grunt": "1.6.3",
-        "grunt-cli": "1.5.0",
-        "grunt-contrib-nodeunit": "5.0.0",
-        "grunt-eslint": "24.0.1",
+        "c8": "12.0.0",
+        "eslint": "10.13.0",
+        "eslint-config-prettier": "10.1.8",
+        "globals": "17.13.0",
+        "prettier": "3.9.10",
         "random-message": "1.1.0",
         "typescript": "7.0.2"
     },