Haraka 3.1.5 → 3.1.6

package/docs/Transaction.md

@@ -1,140 +1,135 @@
 # Transaction Object
 
-An SMTP transaction is valid from MAIL FROM time until RSET or "final-dot".
+An SMTP transaction begins at `MAIL FROM` and ends at `RSET` or end-of-data (the "final dot"). A connection can carry multiple transactions; the current one is available as `connection.transaction`.
 
-## API
+## Properties
 
-- transaction.uuid
+### transaction.uuid
 
-A unique UUID for this transaction. Is equal to the connection.uuid + '.N' where N increments for each transaction on this connection.
+A unique UUID for this transaction, of the form `<connection.uuid>.N`, where `N` increments per transaction on the connection.
 
-- transaction.mail_from
+### transaction.mail_from
 
-The value of the MAIL FROM command is an `Address`[1] object.
+The `MAIL FROM` argument as an [`Address`][address] object.
 
-- transaction.rcpt_to
+### transaction.rcpt_to
 
-An Array of `Address`[1] objects of recipients from the RCPT TO command.
+An array of [`Address`][address] objects, one per accepted `RCPT TO`.
 
-- transaction.message_stream
+### transaction.header
 
-A node.js Readable Stream object for the message.
+The parsed message header. See [haraka-email-message → Header](https://github.com/haraka/email-message#header).
 
-You use it like this:
+### transaction.body
 
-```js
-transaction.message_stream.pipe(WritableStream, options)
-```
+The parsed message body, available only when `parse_body` is `true`. See [haraka-email-message → Body](https://github.com/haraka/email-message#body).
 
-Where WritableStream is a node.js Writable Stream object such as a net.socket, fs.writableStream, process.stdout/stderr or custom stream.
+### transaction.message_stream
 
-The options argument should be an object that overrides the following properties:
+A Node.js `Readable` stream for the message (headers + body). Pipe it into any `Writable` — a socket, file, stdout, or your own stream:
+
+```js
+transaction.message_stream.pipe(writable, options)
+```
 
-    * line_endings (default: "\r\n")
-    * dot_stuffed  (default: true)
-    * ending_dot   (default: false)
-    * end          (default: true)
-    * buffer_size  (default: 65535)
-    * clamd_style  (default: false)
+`options` may override:
 
-e.g.
+| Option         | Default  | Description |
+| -------------- | -------- | --- |
+| `line_endings` | `"\r\n"` | newline sequence |
+| `dot_stuffed`  | `true`   | emit SMTP dot-stuffed output |
+| `ending_dot`   | `false`  | terminate with `.\r\n` (SMTP end-of-data) |
+| `end`          | `true`   | call `.end()` on the writable when finished |
+| `buffer_size`  | `65535`  | internal buffer size |
+| `clamd_style`  | `false`  | ClamAV CLAMSCAN-INSTREAM framing |
 
 ```js
 transaction.message_stream.pipe(socket, { ending_dot: true })
 ```
 
-- transaction.data_bytes
+### transaction.data_bytes
+
+Number of bytes received during `DATA`.
+
+### transaction.parse_body
+
+`false` by default. Set to `true` (in `hook_data` or earlier) to enable MIME body parsing, after which `transaction.body` becomes available. `attachment_hooks()`, `set_banner()`, and `add_body_filter()` set this automatically.
 
-The number of bytes in the email after DATA.
+### transaction.discard_data
 
-- transaction.add_data(line)
+Set to `true` to drop the raw message as it arrives instead of buffering it in `message_stream`. The parsed body and attachments are still available when `parse_body` is `true`. Useful for plugins that only need attachments or text without retaining the whole message.
 
-Adds a line of data to the email. Note this is RAW email - it isn't useful for adding banners to the email.
+### transaction.notes
 
-- transaction.notes
+A `haraka-notes` instance scoped to this transaction. Use it to pass state between hooks; for structured per-test output prefer `transaction.results`. See [haraka-notes](https://github.com/haraka/haraka-notes).
 
-A safe place to store transaction specific values. See also [haraka-results](https://github.com/haraka/haraka-results) and [haraka-notes](https://github.com/haraka/haraka-notes).
+`transaction.notes.skip_plugins` is honoured by the plugin runner — push plugin names into it to bypass them for the remainder of the transaction.
 
-- transaction.add_leading_header(key, value)
+### transaction.results
 
-Adds a header to the top of the header list. This should only be used in very specific cases. Most cases will use `add_header()` instead.
+Structured store for plugin results. See [haraka-results](https://github.com/haraka/haraka-results).
 
-- transaction.add_header(key, value)
+### transaction.rcpt_count
 
-Adds a header to the email.
+Per-disposition counters (`accept`, `tempfail`, `reject`) tracking recipients in this transaction.
 
-- transaction.remove_header(key)
+### transaction.mime_part_count
 
-Deletes a header from the email.
+Number of MIME parts seen so far (when `parse_body` is enabled).
 
-- transaction.header
+### transaction.encoding
 
-The header of the email. See `Header Object`.
+Character encoding used to convert incoming bytes to strings. Defaults to `'utf8'`.
 
-- transaction.parse_body = true|false [default: false]
+## Methods
 
-Set to `true` to enable parsing of the mail body. Make sure you set this in hook_data or before. Storing a transaction hook (with transaction.attachment_hooks) will set this to true.
+### transaction.add_header(key, value)
 
-- transaction.body
+Append a header to the message.
 
-The body of the email if you set `parse_body` above. See `Body Object`.
+### transaction.add_leading_header(key, value)
 
-- transaction.attachment_hooks(start)
+Prepend a header to the message. Most plugins want `add_header()`; use this only when ordering matters (e.g. `Received:` chains).
 
-Sets a callback for when we see an attachment.
+### transaction.remove_header(key)
 
-The `start` event will receive `(content_type, filename, body, stream)` as parameters.
+Remove all headers with `key`.
 
-The stream is a [ReadableStream](http://nodejs.org/api/stream.html)
+### transaction.add_data(line)
 
-If you set stream.connection then the stream will apply backpressure to the connection, allowing you to process attachments before the connection has ended. Here is an example which stores attachments in temporary files using the `tmp` library from npm and tells us the size of the file:
+Append a raw line to the message. The input must already be in SMTP wire format (CRLF newlines, dot-stuffed). Not the right tool for adding banners or transforming body parts — see `set_banner()` and `add_body_filter()`.
+
+### transaction.attachment_hooks(start)
+
+Register a callback fired for each attachment. `start` is called with `(content_type, filename, body, stream)`; `stream` is a Node.js `Readable`. Setting `stream.connection = connection` applies backpressure to the SMTP connection so attachments can be processed before the message ends.
 
 ```js
-exports.hook_data = function (next, connection) {
-  // enable mail body parsing
-  connection.transaction.attachment_hooks(function (ct, fn, body, stream) {
-    start_att(connection, ct, fn, body, stream)
-  })
-  next()
+exports.hook_data = (next, connection) => {
+    connection.transaction.attachment_hooks((ct, fn, body, stream) => {
+        start_att(connection, ct, fn, body, stream)
+    })
+    next()
 }
 
 function start_att(connection, ct, fn, body, stream) {
-  connection.loginfo(`Got attachment: ${ct}, ${fn} for user id: ${connection.transaction.notes.hubdoc_user.email}`)
-  connection.transaction.notes.attachment_count++
-
-  stream.connection = connection // Allow backpressure
-  stream.pause()
-
-  require('tmp').file((err, path, fd) => {
-    connection.loginfo(`Got tempfile: ${path} (${fd})`)
-    const ws = fs.createWriteStream(path)
-    stream.pipe(ws)
-    stream.resume()
-    ws.on('close', () => {
-      connection.loginfo('End of stream reached')
-      fs.fstat(fd, (err, stats) => {
-        connection.loginfo(`Got data of length: ${stats.size}`)
-        fs.close(fd, () => {}) // Close the tmp file descriptor
-      })
+    connection.loginfo(`attachment: ${ct} ${fn}`)
+    stream.connection = connection // enable backpressure
+    stream.pause()
+
+    require('node:tmp').file((err, path, fd) => {
+        const ws = require('node:fs').createWriteStream(path)
+        stream.pipe(ws)
+        stream.resume()
     })
-  })
 }
 ```
 
-- transaction.discard_data = true|false [default: false]
-
-Set this flag to true to discard all data as it arrives and not store in memory or on disk (in the message_stream property). You can still access the attachments and body if you set parse_body to true. This is useful for systems which do not need the full email, just the attachments or mail text.
-
-- transaction.set_banner(text, html)
-
-Sets a banner to be added to the end of the email. If the html part is not given (optional) then the text part will have each line ending replaced with `<br/>` when being inserted into HTML parts.
-
-- transaction.add_body_filter(ct_match, filter)
+### transaction.set_banner(text, html)
 
-Adds a filter to be applied to body parts in the email. ct_match should be a regular expression to match against the full content-type line, or a string to match at the start, e.g. `/^text\/html/` or `'text/plain'`. filter will be called when each body part matching ct_match is complete. It receives three parameters: the content-type line, the encoding name, and a buffer with the full body part. It should return a buffer with the desired contents of the body in the same encoding.
+Append a banner to the end of the message. If `html` is omitted, each newline in `text` is replaced with `<br/>\n` when inserted into HTML parts.
 
-- transaction.results
+### transaction.add_body_filter(ct_match, filter)
 
-Store [results](https://github.com/haraka/haraka-results) of processing in a structured format.
+Register a filter applied to body parts. `ct_match` is either a regex matched against the content-type line, or a string matched as a prefix (e.g. `/^text\/html/` or `'text/plain'`). `filter` receives `(content_type, encoding, buffer)` and must return a `Buffer` with the replacement body (in the same encoding).
 
-[1]: `Address` objects are [address-rfc2821](https://github.com/haraka/node-address-rfc2821) objects.
+[address]: https://github.com/haraka/node-address-rfc2821

package/docs/Tutorial.md

@@ -1,270 +1,183 @@
 # Writing Haraka Plugins
 
-Part of the joy of using Haraka as your main mail server is having a strong
-plugin based system which means you control all aspects of how your mail is
-processed, accepted, and delivered.
+Part of the joy of using Haraka as your main mail server is having a strong plugin system: you control every aspect of how mail is processed, accepted, and delivered.
 
-Of course in order to control this you may at some point need to edit some
-sort of plugin file of your own to customise how things work. The good news
-is that writing plugins in Haraka is simple, even for novice coders. You
-just need a little knowledge of Javascript (and maybe some understanding of
-Node.js) and the world is your oyster.
+This tutorial walks through a small plugin that supports *disposable addresses*: an email like `user-20271231@example.com` is accepted up until 31 December 2027, after which delivery is rejected. Mail that is still within the validity window is rewritten back to `user@example.com` before it is forwarded on.
 
-This tutorial will run through a simple plugin which allows you to have
-email addresses that expire in a short period of time. This is handy if you
-want a _disposable email address_ to use to sign up for a web site that you
-don't wish to continually receive communication from.
+## What You'll Need
 
-## The Design
-
-In order to make this simple, we are going to simply let you have tagged
-email addresses such as `user-20120515@domain.com` which will expire on the
-15th May, 2012. Haraka will then check the email has yet to expire, and
-reject mails to that address after the expiry date. If the address hasn't
-expired yet it will re-write the address to `user@domain.com` before onward
-delivery.
-
-## What You Will Need
-
-- Node.js and npm
+- Node.js (an active LTS release) and npm
 - Haraka
 - A text editor
-- [swaks][1]
-- A screwdriver
-
-[1]: http://jetmore.org/john/code/swaks/
+- [swaks][swaks] for sending test mail
 
 ## Getting Started
 
-First install Haraka via npm if you haven't already:
+Install Haraka and create a project:
 
-    $ sudo npm -g install Haraka
+```sh
+sudo npm install -g Haraka
+haraka -i /path/to/new_project
+```
 
-Now we can create our project directory to get started with:
+Use a directory that does not yet exist. Now scaffold a plugin:
 
-    $ haraka -i /path/to/new_project
+```sh
+haraka -c /path/to/new_project -p rcpt_to.disposable
+```
 
-Make sure you use a directory that doesn't exist for your project.
+`haraka -p` reports the files it created:
 
-Next, let's create a new plugin:
+```
+Plugin rcpt_to.disposable created
+Now edit javascript in:    /path/to/new_project/plugins/rcpt_to.disposable.js
+Add the plugin to config:  /path/to/new_project/config/plugins
+And edit documentation in: /path/to/new_project/docs/plugins/rcpt_to.disposable.md
+```
 
-    $ haraka -c /path/to/new_project -p rcpt_to.disposable
+Edit `config/plugins` so the only enabled lines are:
 
-This should output a bunch of information about files it has created:
+```
+rcpt_to.disposable
+rcpt_to.in_host_list
+queue/test
+```
 
-    Plugin rcpt_to.disposable created
-    Now edit javascript in:    /path/to/new_project/plugins/rcpt_to.disposable.js
-    Add the plugin to config:  /path/to/new_project/config/plugins
-    And edit documentation in: /path/to/new_project/docs/plugins/rcpt_to.disposable.md
+The ordering matters — the disposable plugin must run *before* `rcpt_to.in_host_list`, which accepts mail for domains listed in `config/host_list`. `queue/test` writes accepted mail to a `.eml` file in `os.tmpdir()` so you can confirm delivery.
 
-So let's do the second part now - load up the `config/plugins` file and
-set it up to test. Comment out most of the plugins, except for
-`rcpt_to.in_host_list` and add in our new plugin, and change the queue
-plugin to `test_queue`. The final file should look like this:
+Open `plugins/rcpt_to.disposable.js` and start with:
 
-    # default list of plugins
+```js
+exports.hook_rcpt = (next, connection, params) => {
+    const rcpt = params[0]
+    connection.loginfo(`got recipient: ${rcpt}`)
+    next()
+}
+```
 
-    #dns-lists
-    #data.signatures
+Verify it works. In one terminal:
 
-    # block mail from some known bad HELOs - see config/helo.checks.ini for configuration
-    #helo.checks
+```sh
+echo LOGDEBUG > config/loglevel
+echo myserver.com >> config/host_list
+sudo haraka -c /path/to/new_project
+```
 
-    # Only accept mail where the MAIL FROM domain is resolvable to an MX record
-    #mail_from.is_resolvable
+In another:
 
-    # Allow dated tagged addresses
-    rcpt_to.disposable
+```sh
+swaks -h example.com -t booya@myserver.com -f sender@example.com -s localhost -p 25
+```
 
-    # Only accept mail for your personal list of hosts
-    rcpt_to.in_host_list
+You should see something like this in the Haraka log:
 
-    # Queue mail via qmail-queue
-    #queue/qmail-queue
+```
+[INFO] [<uuid>] [rcpt_to.disposable] got recipient: <booya@myserver.com>
+```
 
-    test_queue
-
-The ordering here is important - our new plugin has to come before `rcpt_to.in_host_list`.
-
-Fire up your favourite editor and put the following into the `plugins/rcpt_to.disposable.js` file:
-
-    exports.hook_rcpt = function (next, connection, params) {
-    	const rcpt = params[0];
-    	this.loginfo("Got recipient: " + rcpt);
-    	next();
-    }
+…and a `.eml` file in your system temp directory containing the message.
 
-Here we log that we got the recipient.
+## Parsing Out the Date
 
-Check this works. You'll need two terminal windows. In window 1:
+Detect addresses of the form `user-YYYYMMDD` and parse the date:
 
-    $ echo LOGDEBUG > config/loglevel
-    $ echo myserver.com >> config/host_list
-    $ sudo haraka -c /path/to/new_project
+```js
+exports.hook_rcpt = (next, connection, params) => {
+    const rcpt = params[0]
+    connection.loginfo(`got recipient: ${rcpt}`)
 
-And in window 2:
+    const match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user)
+    if (!match) return next()
 
-    $ swaks -h domain.com -t booya@myserver.com -f somewhere@example.com \
-      -s localhost -p 25
+    // Date constructor uses zero-indexed months (Dec === 11)
+    const expiry = new Date(match[2], match[3] - 1, match[4])
+    connection.loginfo(`expires on: ${expiry.toISOString()}`)
 
-In the logs you should see:
+    next()
+}
+```
 
-    [INFO] [rcpt_to.disposable] Got recipient: <booya@myserver.com>
+Restart Haraka and send:
 
-Which indicates everything is working. You should also have a file
-`/tmp/mail.eml` containing the email that swaks sent.
+```sh
+swaks -h example.com -t booya-20271231@myserver.com \
+    -f sender@example.com -s localhost -p 25
+```
 
-## Parsing Out The Date
+Logs:
 
-Now check for emails with an expire date in them and turn them into
-`Date` objects. Edit your plugin file as follows:
+```
+[INFO] [rcpt_to.disposable] got recipient: <booya-20271231@myserver.com>
+[INFO] [rcpt_to.disposable] expires on: 2027-12-31T00:00:00.000Z
+```
 
-    exports.hook_rcpt = function (next, connection, params) {
-    	var rcpt = params[0];
-    	this.loginfo("Got recipient: " + rcpt);
+## Rejecting Expired Addresses
 
-    	// Check user matches regex 'user-YYYYMMDD':
-    	var match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user);
-    	if (!match) {
-    		return next();
-    	}
+Compare the parsed date to today and reject if it has already passed:
 
-    	// get date - note Date constructor takes month-1 (i.e. Dec == 11).
-    	var expiry_date = new Date(match[2], match[3]-1, match[4]);
+```js
+exports.hook_rcpt = (next, connection, params) => {
+    const rcpt = params[0]
 
-    	this.loginfo("Email expires on: " + expiry_date);
+    const match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user)
+    if (!match) return next()
 
-    	next();
+    const expiry = new Date(match[2], match[3] - 1, match[4])
+    if (expiry < new Date()) {
+        return next(DENY, 'Expired email address')
     }
 
-Start haraka again and pass it the following email via swaks:
+    next()
+}
+```
 
-    $ swaks -h domain.com -t booya-20120101@myserver.com \
-      -f somewhere@example.com -s localhost -p 25
+Send mail to an expired address:
 
-And you should see now in the logs:
+```sh
+swaks -h example.com -t booya-20200101@myserver.com \
+    -f sender@example.com -s localhost -p 25
+```
 
-    [INFO] [rcpt_to.disposable] Got recipient: <booya-20120101@myserver.com>
-    [INFO] [rcpt_to.disposable] Email expires on: Sun, 01 Jan 2012 05:00:00 GMT
+The remote end sees:
 
-The exact time may vary depending on your timezone, but it should be obvious
-we now have a date object, which we can now compare to the current date.
+```
+<** 550 Expired email address
+```
 
-## Rejecting Expired Emails
+## Rewriting Live Addresses
 
-The next edit we have to do is to add in code to compare to the current date
-and reject expired emails. Again, this is very simple:
+When the address is still valid, strip the date tag so the downstream mail store receives plain `user@domain`:
 
-    exports.hook_rcpt = function (next, connection, params) {
-    	var rcpt = params[0];
-    	this.loginfo("Got recipient: " + rcpt);
+```js
+exports.hook_rcpt = (next, connection, params) => {
+    const rcpt = params[0]
 
-    	// Check user matches regex 'user-YYYYMMDD':
-    	var match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user);
-    	if (!match) {
-    		return next();
-    	}
+    const match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user)
+    if (!match) return next()
 
-    	// get date - note Date constructor takes month-1 (i.e. Dec == 11).
-    	var expiry_date = new Date(match[2], match[3]-1, match[4]);
-
-    	this.loginfo("Email expires on: " + expiry_date);
-
-    	var today = new Date();
-
-    	if (expiry_date < today) {
-    		// If we get here, the email address has expired
-    		return next(DENY, "Expired email address");
-    	}
-
-    	next();
+    const expiry = new Date(match[2], match[3] - 1, match[4])
+    if (expiry < new Date()) {
+        return next(DENY, 'Expired email address')
     }
 
-And we can easily check that with swaks (remember to restart Haraka):
-
-    $ swaks -h foo.com -t booya-20110101@haraka.local -f somewhere@example.com \
-      -s localhost -p 25
-    === Trying localhost:25...
-    === Connected to localhost.
-    <-  220 sergeant.org ESMTP Haraka 0.3 ready
-     -> EHLO foo.com
-    <-  250-Haraka says hi Unknown [127.0.0.1]
-    <-  250-PIPELINING
-    <-  250-8BITMIME
-    <-  250 SIZE 500000
-     -> MAIL FROM:<somewhere@example.com>
-    <-  250 From address is OK
-     -> RCPT TO:<booya-20110101@haraka.local>
-    <** 550 Expired email address
-     -> QUIT
-    <-  221 closing connection. Have a jolly good day.
-    === Connection closed with remote host.
-
-Now we need to do one more thing...
-
-## Fixing Up Unexpired Emails
-
-The last thing we need to do, is if we have an email that isn't expired, we
-need to normalise it back to the real email address, because wherever we
-deliver this to is unlikely to recognise these new email addresses.
-
-Here's how our final plugin will look:
-
-    exports.hook_rcpt = function (next, connection, params) {
-    	var rcpt = params[0];
-    	this.loginfo("Got recipient: " + rcpt);
-
-    	// Check user matches regex 'user-YYYYMMDD':
-    	var match = /^(.*)-(\d{4})(\d{2})(\d{2})$/.exec(rcpt.user);
-    	if (!match) {
-    		return next();
-    	}
-
-    	// get date - note Date constructor takes month-1 (i.e. Dec == 11).
-    	var expiry_date = new Date(match[2], match[3]-1, match[4]);
-
-    	this.loginfo("Email expires on: " + expiry_date);
-
-    	var today = new Date();
-
-    	if (expiry_date < today) {
-    		// If we get here, the email address has expired
-    		return next(DENY, "Expired email address");
-    	}
-
-    	// now get rid of the extension:
-    	rcpt.user = match[1];
-    	this.loginfo("Email address now: " + rcpt);
-
-    	next();
-    }
-
-And when we test this with an unexpired address via swaks:
-
-    $ swaks -h foo.com -t booya-20120101@haraka.local \
-      -f somewhere@example.com -s localhost -p 25
-
-We get in the logs:
+    rcpt.user = match[1]
+    connection.loginfo(`rewrote recipient to: ${rcpt}`)
+    next()
+}
+```
 
-    [INFO] [rcpt_to.disposable] Got recipient: <booya-20120101@haraka.local>
-    [INFO] [rcpt_to.disposable] Email expires on: Sun Jan 01 2012 00:00:00 GMT-0500 (EST)
-    [INFO] [rcpt_to.disposable] Email address now: <booya@haraka.local>
+Send to a live tagged address and watch the log:
 
-Which indicates that we have successfully modified the email address.
+```
+[INFO] [rcpt_to.disposable] rewrote recipient to: <booya@myserver.com>
+```
 
-# Further Reading
+## Further Reading
 
-There are many more features of the Haraka API to explore, including access
-to the body of the email and the headers, access to the HELO string, and
-implementing ESMTP extensions, among many others.
+The Haraka API offers much more — body and header access, ESMTP extension hooks, the outbound delivery hooks, structured results, attachments, and so on. Two good starting points:
 
-There are two good places to read up on these. Firstly is the documentation
-in the Haraka "docs" directory. Start with the `Plugins.md` file, and work
-your way through the API from there.
+- The [Plugins guide](Plugins.md) and the rest of the `docs/` directory.
+- The [Plugin registry](https://github.com/haraka/Haraka/blob/master/PLUGINS.md) for an inventory of real-world plugins.
+- The plugins shipped in the [`plugins/`](../plugins/) directory. Even the most elaborate are under 200 lines; many are under 20.
 
-The second place is simply reading the source code for the plugins themselves.
-The plugins that Haraka ships with use almost all parts of the API and so
-should give you a good starting point if you want to implement a particular
-piece of functionality. Even the most complicated plugins are under 200 lines
-of code, so don't be intimidated by them! The simplest one is a mere 5 lines
-of code.
+[swaks]: https://www.jetmore.org/john/code/swaks/