Haraka 3.1.5 → 3.1.7

package/docs/Plugins.md

@@ -1,7 +1,6 @@
 # Plugins
 
-Most aspects of receiving an email in Haraka are controlled by plugins. Mail cannot even be received unless at least a 'rcpt' and 'queue' plugin are
-enabled.
+Most aspects of receiving an email in Haraka are controlled by plugins. Mail cannot even be received unless at least a 'rcpt' and 'queue' plugin are enabled.
 
 Recipient (_rcpt_) plugins determine if a particular recipient is allowed to be relayed or received for. A _queue_ plugin queues the message somewhere - normally to disk or to an another SMTP server.
 
@@ -9,7 +8,7 @@ Recipient (_rcpt_) plugins determine if a particular recipient is allowed to be
 
 Get a list of installed plugins by running `haraka -l`. To include locally installed plugins, add the `-c /path/to/config` option.
 
-We also have a [registry of known plugins](https://github.com/haraka/Haraka/blob/master/Plugins.md).
+The [top-level Plugins.md](../Plugins.md) is the registry of known plugins — both core and community.
 
 Display the help text for a plugin by running:
 
@@ -21,12 +20,9 @@ Display the help text for a plugin by running:
 
 ## Anatomy of a Plugin
 
-Plugins in Haraka are JS files in the `plugins` directory (legacy) and npm
-modules in the node_modules directory. See "Plugins as Modules" below.
+Plugins in Haraka are JS files in the `plugins` directory (legacy) and npm modules in the node_modules directory. See "Plugins as Modules" below.
 
-Plugins can be installed in the Haraka global directory (default:
-/$os/$specific/lib/node_modules/Haraka) or in the Haraka install directory
-(whatever you chose when you typed `haraka -i`. Example: `haraka -i /etc/haraka`
+Plugins can be installed in the Haraka global directory (default: /$os/$specific/lib/node_modules/Haraka) or in the Haraka install directory (whatever you chose when you typed `haraka -i`. Example: `haraka -i /etc/haraka`
 
 To enable a plugin, add its name to `config/plugins`. For npm packaged plugins, the name does not include the `haraka-plugin` prefix.
 
@@ -123,6 +119,8 @@ need to define them:
 
 - DENYDISCONNECT - Reject with a 5xx error and immediately disconnect.
 
+- DENYSOFTDISCONNECT - Reject with a 4xx error and immediately disconnect.
+
 - DISCONNECT - Immediately disconnect
 
 - OK
@@ -178,6 +176,7 @@ These are the hook and their parameters (next excluded):
 - delivered (hmail, [host, ip, response, delay, port, mode, ok_recips, secured, authenticated]) - called when outbound mail is delivered
 - send_email (hmail) - called when outbound is about to be sent
 - pre_send_trans_email (fake_connection) - called just before an email is queued to disk with a faked connection object
+- log (logger, log_item) - called for every log message; log plugins (e.g. haraka-plugin-syslog) use this hook to ship logs elsewhere
 
 ### rcpt
 
@@ -287,47 +286,28 @@ Plugins inherit all the logging methods of `logger.js`, which are:
 - logalert
 - logemerg
 
-If plugins throw an exception when in a hook, the exception will be caught
-and generate a logcrit level error. However, exceptions will not be caught
-as gracefully when plugins are running async code. Use error codes for that,
-log the error, and run your next() function appropriately.
+If plugins throw an exception when in a hook, the exception will be caught and generate a logcrit level error. However, exceptions will not be caught as gracefully when plugins are running async code. Use error codes for that, log the error, and run your next() function appropriately.
 
 ## Sharing State
 
-There are several cases where you might need to share information between
-plugins. This is done using `notes` - there are three types available:
+There are several cases where you might need to share information between plugins. This is done using `notes` - there are three types available:
 
 - server.notes
 
-  Available in all plugins. This is created at PID start-up and is shared
-  amongst all plugins on the same PID and listener.
-  Typical uses for notes at this level would be to share database
-  connections between multiple plugins or connection pools etc.
+  Available in all plugins. This is created at PID start-up and is shared amongst all plugins on the same PID and listener. Typical uses for notes at this level would be to share database connections between multiple plugins or connection pools etc.
 
 - connection.notes
 
-  Available on any hook that passes 'connection' as a function parameter.
-  This is shared amongst all plugins for a single connection and is
-  destroyed after the client disconnects.
-  Typical uses for notes at this level would be to store information
-  about the connected client e.g. rDNS names, HELO/EHLO, white/black
-  list status etc.
+  Available on any hook that passes 'connection' as a function parameter. This is shared amongst all plugins for a single connection and is destroyed after the client disconnects. Typical uses for notes at this level would be to store information about the connected client e.g. rDNS names, HELO/EHLO, white/black list status etc.
 
 - connection.transaction.notes
 
-  Available on any hook that passes 'connection' as a function parameter
-  between hook_mail and hook_data_post.
-  This is shared amongst all plugins for this transaction (e.g. MAIL FROM
-  through until a message is received or the connection is reset).
-  Typical uses for notes at this level would be to store information
-  on things like greylisting which uses client, sender and recipient
-  information etc.
+  Available on any hook that passes 'connection' as a function parameter between hook_mail and hook_data_post.
+  This is shared amongst all plugins for this transaction (e.g. MAIL FROM through until a message is received or the connection is reset). Typical uses for notes at this level would be to store information on things like greylisting which uses client, sender and recipient information etc.
 
 - hmail.todo.notes
 
-  Available on any outbound hook that passes `hmail` as a function parameter.
-  This is the same object as 'connection.transaction.notes', so anything
-  you store in the transaction notes is automatically available in the
+  Available on any outbound hook that passes `hmail` as a function parameter. This is the same object as 'connection.transaction.notes', so anything you store in the transaction notes is automatically available in the
   outbound functions here.
 
 All of these notes are JS objects - use them as simple key/value store e.g.
@@ -336,39 +316,40 @@ All of these notes are JS objects - use them as simple key/value store e.g.
 
 ## Plugins as Modules
 
-Plugins as NPM modules are named with the `haraka-plugin` prefix. Therefore, a
-plugin that frobnobricates might be called `haraka-plugin-frobnobricate` and
-published to NPM with that name. The prefix is not required in the
+Plugins as NPM modules are named with the `haraka-plugin` prefix. Therefore, a plugin that frobnobricates might be called `haraka-plugin-frobnobricate` and published to NPM with that name. The prefix is not required in the
 `config/plugins` file.
 
-Plugins loaded as NPM modules behave slightly different than plugins loaded
-as plain JS files.
+Plugins loaded as NPM modules behave slightly different than plugins loaded as plain JS files.
+
+Plain JS plugins have a custom `require()` which allows loading core Haraka modules via specifying `require('./name')` (note the `./` prefix). Although the core modules aren't in the same folder, the custom `require` intercepts
+this and look for core modules. Note that if there is a module in your plugins folder of the same name that will not take preference, so avoid using names similar to core modules.
+
+Plugins loaded as modules do not have the special `require()`. To load a core Haraka module you must use `this.haraka_require('name')`. This should also be preferred for plain JS plugins, as the `./` hack is likely to be removed in the future.
+
+Plugins loaded as modules are not compiled in the Haraka plugin sandbox, which blocks access to certain globals and provides a global `server` object. To access the `server` object, use `connection.server` instead.
+
+Module plugins support default config in their local `config` directory. See the "Default Config and Overrides" section in [haraka-config](https://github.com/haraka/haraka-config#default-config-and-overrides).
 
-Plain JS plugins have a custom `require()` which allows loading core Haraka
-modules via specifying `require('./name')` (note the `./` prefix). Although
-the core modules aren't in the same folder, the custom `require` intercepts
-this and look for core modules. Note that if there is a module in your plugins
-folder of the same name that will not take preference, so avoid using names
-similar to core modules.
+### Inheriting from another plugin
 
-Plugins loaded as modules do not have the special `require()`. To load
-a core Haraka module you must use `this.haraka_require('name')`.
-This should also be preferred for plain JS plugins, as the
-`./` hack is likely to be removed in the future.
+A plugin can inherit methods from another plugin by calling `plugin.inherits(name)` from its `register()`. The parent's exported methods become available on `this` (without overwriting any methods the child has already defined), and the parent's `register()` runs in the child's context. `rcpt_to.host_list_base` is a typical parent used by multiple `rcpt_to.*` plugins.
+
+```js
+exports.register = function () {
+    this.inherits('rcpt_to.host_list_base')
+    this.register_hook('rcpt', 'my_rcpt')
+}
+```
 
-Plugins loaded as modules are not compiled in the Haraka plugin sandbox,
-which blocks access to certain globals and provides a global `server` object.
-To access the `server` object, use `connection.server` instead.
+### Deprecated plugin names
 
-Module plugins support default config in their local `config` directory. See the
-"Default Config and Overrides" section in [Config](Config.md).
+Some plugin names have been folded into newer packages — for example `connect.fcrdns` is now `fcrdns`, `dnsbl` is now `dns-list`, and `rate_limit` / `max_unrecognized_commands` are now `limit`. Haraka logs a notice and loads the replacement automatically; the full mapping lives in `plugins.js → plugins.deprecated`.
 
 ## Shutdown
 
 On graceful reload, Haraka will call a plugin's `shutdown` method.
 
-This is so you can clear any timers or intervals, or shut down any connections
-to remote servers. See [Issue 2024](https://github.com/haraka/Haraka/issues/2024).
+This is so you can clear any timers or intervals, or shut down any connections to remote servers. See [Issue 2024](https://github.com/haraka/Haraka/issues/2024).
 
 e.g.
 
@@ -378,9 +359,7 @@ exports.shutdown = function () {
 }
 ```
 
-If you don't implement this in your plugin and have a connection open or a
-timer running then Haraka will take 30 seconds to shut down and have to
-forcibly kill your process.
+If you don't implement this in your plugin and have a connection open or a timer running then Haraka will take 30 seconds to shut down and have to forcibly kill your process.
 
 Note: This only applies when running with a `nodes=...` value in smtp.ini.
 

package/docs/Transaction.md

@@ -1,140 +1,135 @@
 # Transaction Object
 
-An SMTP transaction is valid from MAIL FROM time until RSET or "final-dot".
+An SMTP transaction begins at `MAIL FROM` and ends at `RSET` or end-of-data (the "final dot"). A connection can carry multiple transactions; the current one is available as `connection.transaction`.
 
-## API
+## Properties
 
-- transaction.uuid
+### transaction.uuid
 
-A unique UUID for this transaction. Is equal to the connection.uuid + '.N' where N increments for each transaction on this connection.
+A unique UUID for this transaction, of the form `<connection.uuid>.N`, where `N` increments per transaction on the connection.
 
-- transaction.mail_from
+### transaction.mail_from
 
-The value of the MAIL FROM command is an `Address`[1] object.
+The `MAIL FROM` argument as an [`Address`][address] object.
 
-- transaction.rcpt_to
+### transaction.rcpt_to
 
-An Array of `Address`[1] objects of recipients from the RCPT TO command.
+An array of [`Address`][address] objects, one per accepted `RCPT TO`.
 
-- transaction.message_stream
+### transaction.header
 
-A node.js Readable Stream object for the message.
+The parsed message header. See [haraka-email-message → Header](https://github.com/haraka/email-message#header).
 
-You use it like this:
+### transaction.body
 
-```js
-transaction.message_stream.pipe(WritableStream, options)
-```
+The parsed message body, available only when `parse_body` is `true`. See [haraka-email-message → Body](https://github.com/haraka/email-message#body).
 
-Where WritableStream is a node.js Writable Stream object such as a net.socket, fs.writableStream, process.stdout/stderr or custom stream.
+### transaction.message_stream
 
-The options argument should be an object that overrides the following properties:
+A Node.js `Readable` stream for the message (headers + body). Pipe it into any `Writable` — a socket, file, stdout, or your own stream:
+
+```js
+transaction.message_stream.pipe(writable, options)
+```
 
-    * line_endings (default: "\r\n")
-    * dot_stuffed  (default: true)
-    * ending_dot   (default: false)
-    * end          (default: true)
-    * buffer_size  (default: 65535)
-    * clamd_style  (default: false)
+`options` may override:
 
-e.g.
+| Option         | Default  | Description |
+| -------------- | -------- | --- |
+| `line_endings` | `"\r\n"` | newline sequence |
+| `dot_stuffed`  | `true`   | emit SMTP dot-stuffed output |
+| `ending_dot`   | `false`  | terminate with `.\r\n` (SMTP end-of-data) |
+| `end`          | `true`   | call `.end()` on the writable when finished |
+| `buffer_size`  | `65535`  | internal buffer size |
+| `clamd_style`  | `false`  | ClamAV CLAMSCAN-INSTREAM framing |
 
 ```js
 transaction.message_stream.pipe(socket, { ending_dot: true })
 ```
 
-- transaction.data_bytes
+### transaction.data_bytes
+
+Number of bytes received during `DATA`.
+
+### transaction.parse_body
+
+`false` by default. Set to `true` (in `hook_data` or earlier) to enable MIME body parsing, after which `transaction.body` becomes available. `attachment_hooks()`, `set_banner()`, and `add_body_filter()` set this automatically.
 
-The number of bytes in the email after DATA.
+### transaction.discard_data
 
-- transaction.add_data(line)
+Set to `true` to drop the raw message as it arrives instead of buffering it in `message_stream`. The parsed body and attachments are still available when `parse_body` is `true`. Useful for plugins that only need attachments or text without retaining the whole message.
 
-Adds a line of data to the email. Note this is RAW email - it isn't useful for adding banners to the email.
+### transaction.notes
 
-- transaction.notes
+A `haraka-notes` instance scoped to this transaction. Use it to pass state between hooks; for structured per-test output prefer `transaction.results`. See [haraka-notes](https://github.com/haraka/haraka-notes).
 
-A safe place to store transaction specific values. See also [haraka-results](https://github.com/haraka/haraka-results) and [haraka-notes](https://github.com/haraka/haraka-notes).
+`transaction.notes.skip_plugins` is honoured by the plugin runner — push plugin names into it to bypass them for the remainder of the transaction.
 
-- transaction.add_leading_header(key, value)
+### transaction.results
 
-Adds a header to the top of the header list. This should only be used in very specific cases. Most cases will use `add_header()` instead.
+Structured store for plugin results. See [haraka-results](https://github.com/haraka/haraka-results).
 
-- transaction.add_header(key, value)
+### transaction.rcpt_count
 
-Adds a header to the email.
+Per-disposition counters (`accept`, `tempfail`, `reject`) tracking recipients in this transaction.
 
-- transaction.remove_header(key)
+### transaction.mime_part_count
 
-Deletes a header from the email.
+Number of MIME parts seen so far (when `parse_body` is enabled).
 
-- transaction.header
+### transaction.encoding
 
-The header of the email. See `Header Object`.
+Character encoding used to convert incoming bytes to strings. Defaults to `'utf8'`.
 
-- transaction.parse_body = true|false [default: false]
+## Methods
 
-Set to `true` to enable parsing of the mail body. Make sure you set this in hook_data or before. Storing a transaction hook (with transaction.attachment_hooks) will set this to true.
+### transaction.add_header(key, value)
 
-- transaction.body
+Append a header to the message.
 
-The body of the email if you set `parse_body` above. See `Body Object`.
+### transaction.add_leading_header(key, value)
 
-- transaction.attachment_hooks(start)
+Prepend a header to the message. Most plugins want `add_header()`; use this only when ordering matters (e.g. `Received:` chains).
 
-Sets a callback for when we see an attachment.
+### transaction.remove_header(key)
 
-The `start` event will receive `(content_type, filename, body, stream)` as parameters.
+Remove all headers with `key`.
 
-The stream is a [ReadableStream](http://nodejs.org/api/stream.html)
+### transaction.add_data(line)
 
-If you set stream.connection then the stream will apply backpressure to the connection, allowing you to process attachments before the connection has ended. Here is an example which stores attachments in temporary files using the `tmp` library from npm and tells us the size of the file:
+Append a raw line to the message. The input must already be in SMTP wire format (CRLF newlines, dot-stuffed). Not the right tool for adding banners or transforming body parts — see `set_banner()` and `add_body_filter()`.
+
+### transaction.attachment_hooks(start)
+
+Register a callback fired for each attachment. `start` is called with `(content_type, filename, body, stream)`; `stream` is a Node.js `Readable`. Setting `stream.connection = connection` applies backpressure to the SMTP connection so attachments can be processed before the message ends.
 
 ```js
-exports.hook_data = function (next, connection) {
-  // enable mail body parsing
-  connection.transaction.attachment_hooks(function (ct, fn, body, stream) {
-    start_att(connection, ct, fn, body, stream)
-  })
-  next()
+exports.hook_data = (next, connection) => {
+    connection.transaction.attachment_hooks((ct, fn, body, stream) => {
+        start_att(connection, ct, fn, body, stream)
+    })
+    next()
 }
 
 function start_att(connection, ct, fn, body, stream) {
-  connection.loginfo(`Got attachment: ${ct}, ${fn} for user id: ${connection.transaction.notes.hubdoc_user.email}`)
-  connection.transaction.notes.attachment_count++
-
-  stream.connection = connection // Allow backpressure
-  stream.pause()
-
-  require('tmp').file((err, path, fd) => {
-    connection.loginfo(`Got tempfile: ${path} (${fd})`)
-    const ws = fs.createWriteStream(path)
-    stream.pipe(ws)
-    stream.resume()
-    ws.on('close', () => {
-      connection.loginfo('End of stream reached')
-      fs.fstat(fd, (err, stats) => {
-        connection.loginfo(`Got data of length: ${stats.size}`)
-        fs.close(fd, () => {}) // Close the tmp file descriptor
-      })
+    connection.loginfo(`attachment: ${ct} ${fn}`)
+    stream.connection = connection // enable backpressure
+    stream.pause()
+
+    require('node:tmp').file((err, path, fd) => {
+        const ws = require('node:fs').createWriteStream(path)
+        stream.pipe(ws)
+        stream.resume()
     })
-  })
 }
 ```
 
-- transaction.discard_data = true|false [default: false]
-
-Set this flag to true to discard all data as it arrives and not store in memory or on disk (in the message_stream property). You can still access the attachments and body if you set parse_body to true. This is useful for systems which do not need the full email, just the attachments or mail text.
-
-- transaction.set_banner(text, html)
-
-Sets a banner to be added to the end of the email. If the html part is not given (optional) then the text part will have each line ending replaced with `<br/>` when being inserted into HTML parts.
-
-- transaction.add_body_filter(ct_match, filter)
+### transaction.set_banner(text, html)
 
-Adds a filter to be applied to body parts in the email. ct_match should be a regular expression to match against the full content-type line, or a string to match at the start, e.g. `/^text\/html/` or `'text/plain'`. filter will be called when each body part matching ct_match is complete. It receives three parameters: the content-type line, the encoding name, and a buffer with the full body part. It should return a buffer with the desired contents of the body in the same encoding.
+Append a banner to the end of the message. If `html` is omitted, each newline in `text` is replaced with `<br/>\n` when inserted into HTML parts.
 
-- transaction.results
+### transaction.add_body_filter(ct_match, filter)
 
-Store [results](https://github.com/haraka/haraka-results) of processing in a structured format.
+Register a filter applied to body parts. `ct_match` is either a regex matched against the content-type line, or a string matched as a prefix (e.g. `/^text\/html/` or `'text/plain'`). `filter` receives `(content_type, encoding, buffer)` and must return a `Buffer` with the replacement body (in the same encoding).
 
-[1]: `Address` objects are [address-rfc2821](https://github.com/haraka/node-address-rfc2821) objects.
+[address]: https://github.com/haraka/node-address-rfc2821