haraka 0.0.33 → 3.3.1

package/docs/CoreConfig.md

@@ -0,0 +1,96 @@
+# Core Configuration Files
+
+Haraka reads the configuration files in this document directly. Plugins typically own their own files (named after the plugin); see the individual plugin docs.
+
+See also [Logging](Logging.md), [HAProxy](HAProxy.md), and [Outbound](Outbound.md).
+
+## smtp.yaml / smtp.json
+
+If either of these files exists it is loaded first. It uses the YAML/JSON override mechanism from [haraka-config](https://github.com/haraka/haraka-config) and can provide the entire configuration as a single file.
+
+## plugins
+
+A newline-delimited list of plugins to load. Lines starting with `#` are ignored.
+
+## smtp.ini
+
+Controls the SMTP listener and the master process.
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `listen` | `[::0]:25` | Comma-separated `IP:port` pairs to listen on (e.g. `127.0.0.1:25,127.0.0.1:587`). |
+| `listen_host` / `port` | — | Legacy. If set, a `<listen_host>:<port>` entry is prepended to `listen`. Prefer `listen`. |
+| `smtps_port` | `465` | Port used by the optional implicit-TLS listener. |
+| `public_ip` | none | The server's public IP. Helps NAT-aware plugins (SPF, GeoIP) when Haraka is behind NAT. If `stun` is on `$PATH` Haraka will try to discover it automatically. |
+| `inactivity_timeout` | `300` | Idle seconds before a client socket is dropped. |
+| `nodes` | `1` | Number of worker processes to fork. The string `cpus` forks one per CPU. |
+| `user` / `group` | — | User and group to drop privileges to (name or numeric ID). |
+| `ignore_bad_plugins` | `0` | If `1`, Haraka starts even if some plugins fail to compile. |
+| `daemonize` | `false` | If `true`, fork into the background at start-up. |
+| `daemon_log_file` | `/var/log/haraka.log` | Where to redirect stdout/stderr when daemonized. |
+| `daemon_pid_file` | `/var/run/haraka.pid` | Where to write the PID file. |
+| `graceful_shutdown` | `false` | If `true`, wait for in-flight sockets on shutdown. |
+| `force_shutdown_timeout` | `30` | Seconds to wait before forcing shutdown. |
+
+## me
+
+A single-line file containing the server name used in `Received:` headers
+and elsewhere. Defaults to `hostname(1)`.
+
+## connection.ini
+
+Per-connection limits and behaviours. See inline comments in the shipped
+`config/connection.ini` for full details.
+
+| Section / Key | Default | Description |
+| --- | --- | --- |
+| `main.spool_dir` | `/tmp` | Directory for temporary spool files. |
+| `main.spool_after` | `-1` | Size (bytes) at which to spool the message to disk. `-1` never spools; `0` always spools. |
+| `main.strict_rfc1869` | `false` | Reject `MAIL FROM` / `RCPT TO` that violates RFC 1869/821 (spurious spaces, missing brackets). |
+| `main.smtputf8` | `true` | Advertise `SMTPUTF8` (RFC 6531). |
+| `main.postel` | `false` | Liberal envelope parsing. |
+| `haproxy.enabled` | `true` | Enable HAProxy PROXY protocol handling. Set `false` to bypass PROXY support. |
+| `haproxy.hosts` | empty | Array of IPs/CIDRs allowed to send the PROXY protocol. See [HAProxy.md](HAProxy.md). |
+| `headers.add_received` | `true` | Add a `Received:` header to incoming mail. |
+| `headers.clean_auth_results` | `true` | Strip inbound `Authentication-Results:` headers before plugins run. |
+| `headers.show_version` | `true` | Include the Haraka version in the SMTP banner and the `Received:` header. |
+| `headers.max_lines` | `1000` | Maximum number of header lines accepted. |
+| `headers.max_received` | `100` | Maximum number of `Received:` headers before mail is rejected as looping. |
+| `max.bytes` | `26214400` | Maximum message size (advertised as the `SIZE` ESMTP extension). |
+| `max.line_length` | `512` | SMTP command line length cap. Clients exceeding this are dropped with `521`. |
+| `max.data_line_length` | `992` | Maximum line length in `DATA`. Longer lines are wrapped with `CRLF SPACE` (Sendmail behaviour) and `transaction.notes.data_line_length_exceeded` is set. |
+| `max.mime_parts` | `1000` | Maximum MIME parts per message. |
+| `message.greeting` | — | Array. Lines used as the SMTP greeting banner. |
+| `message.helo` | `Haraka is at your service.` | Reply text for `HELO` / `EHLO`. |
+| `message.close` | `closing connection. …` | Reply text on `QUIT`. |
+| `uuid.banner_chars` | `6` | Number of UUID chars included in the SMTP banner (`0` to disable, `40` for the full UUID). |
+| `uuid.deny_chars` | `0` | Number of UUID chars prepended (in brackets) to deny messages. |
+
+## plugin_timeout
+
+Single-integer file. Seconds to allow a plugin hook to run before Haraka
+automatically advances to the next hook. Default: `30`.
+
+A per-plugin override may be placed at `config/<plugin>.timeout`. A value
+of `0` disables the timeout for that plugin — use with care. Plugins in
+subdirectories need a matching path: `queue/smtp_forward` looks for
+`config/queue/smtp_forward.timeout`.
+
+## outbound.ini
+
+Configures the outbound delivery engine. The most common keys are listed
+below; see [Outbound.md](Outbound.md) for the full set.
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `disabled` | `false` | Disable outbound delivery entirely. |
+| `concurrency_max` | `100` | Maximum simultaneous outbound deliveries. |
+| `enable_tls` | `true` | Use STARTTLS opportunistically on outbound deliveries. |
+| `maxTempFailures` | `13` | Number of temporary failures before a message bounces. |
+| `always_split` | `false` | Create one queue file per recipient instead of one per destination domain. |
+| `received_header` | `Haraka outbound` | Value used in the outbound `Received:` header. |
+| `inet_prefer` | `default` | IP family preference for MX lookups: `v4`, `v6`, or `default` (OS preference). |
+
+## outbound.bounce_message
+
+Template used when an outbound message bounces. The default is usually fine. Available template variables are documented in the source of `outbound/hmail.js`.

package/docs/CustomReturnCodes.md

@@ -0,0 +1,3 @@
+# Custom Return Codes
+
+This content has moved to [haraka-dsn](https://github.com/haraka/haraka-dsn).

package/docs/HAProxy.md

@@ -0,0 +1,62 @@
+# HAProxy PROXY Protocol Support
+
+Haraka supports version 1 (text format) of the HAProxy [PROXY protocol][proxy-spec], which allows an upstream proxy to tell Haraka the real client IP and port. DNSBLs, allow-lists, and other IP-based plugins then see the original client rather than the proxy.
+
+The PROXY v2 binary header is not currently supported.
+
+## Configuration
+
+PROXY support is enabled by default, but inactive until trusted proxy IPs or CIDRs are listed in `connection.ini`:
+
+```ini
+[haproxy]
+enabled=true
+hosts[] = 192.0.2.4
+hosts[] = 192.0.2.5/30
+hosts[] = 2001:db8::1
+```
+
+Set `enabled=false` to disable PROXY protocol handling entirely. On SMTPS listeners this bypasses pre-TLS PROXY parsing and uses the standard implicit TLS server.
+
+Connections from any other IP get a `DENYSOFTDISCONNECT` if they send a `PROXY` command. `DENYSOFT` is deliberate — it avoids permanently rejecting valid mail when a misconfiguration causes a legitimate proxy to fall outside the allow-list.
+
+When a listed proxy connects, Haraka **does not** send the SMTP banner; it waits for the `PROXY` command. If none arrives within 30 seconds the connection is closed with `421 PROXY timed out`.
+
+## What plugins see after PROXY is parsed
+
+| Property | Value |
+| --- | --- |
+| `connection.remote.ip` / `.port` | the **real** client address from the PROXY header |
+| `connection.local.ip` / `.port` | the destination IP/port from the PROXY header |
+| `connection.proxy.allowed` | `true` |
+| `connection.proxy.ip` | the proxy's address (i.e. the original socket peer) |
+| `connection.proxy.type` | `'haproxy'` |
+| `connection.notes.proxy` | full parsed record: `{ type, proto, src_ip, src_port, dst_ip, dst_port, proxy_ip }` |
+
+## HAProxy configuration
+
+You need HAProxy ≥ 1.5. The `send-proxy` option on each backend server tells HAProxy to emit the v1 header on every connection.
+
+```
+listen smtp :25
+    mode tcp
+    option tcplog
+    balance roundrobin
+    server smtp1 ip.of.haraka1:25 check-send-proxy check inter 10s send-proxy
+    server smtp2 ip.of.haraka2:25 check-send-proxy check inter 10s send-proxy
+```
+
+The `check-send-proxy` flag is required for HAProxy's health checks because Haraka does not respond with a banner before the PROXY header arrives.
+
+### Health checks
+
+`option smtpchk` drops the connection mid-handshake and shows up as `CONNRESET` in Haraka's logs. A cleaner check is to use `tcp-check` and politely close with `QUIT`:
+
+```
+    option tcp-check
+    tcp-check expect rstring ^220
+    tcp-check send QUIT\r\n
+    tcp-check expect rstring ^221
+```
+
+[proxy-spec]: https://www.haproxy.org/download/2.8/doc/proxy-protocol.txt

package/docs/Header.md

@@ -0,0 +1 @@
+moved to [Header](https://github.com/haraka/email-message#header)

package/docs/Logging.md

@@ -0,0 +1,129 @@
+# Haraka Logging
+
+Haraka has a built-in logger (described below) and a plugin hook (`log`) that lets log plugins ship messages elsewhere — for example to syslog via [haraka-plugin-syslog](https://github.com/haraka/haraka-plugin-syslog).
+
+## Configuration
+
+### log.ini
+
+```ini
+[main]
+; data, protocol, debug, info, notice, warn, error, crit, alert, emerg
+level=info
+
+; prepend ISO-8601 timestamps to log entries (built-in logger only)
+timestamps=false
+
+; default, logfmt, json
+format=default
+```
+
+### loglevel
+
+A single-line file for quick CLI tweaks:
+
+```sh
+echo DEBUG > config/loglevel
+```
+
+When both `log.ini` and the `loglevel` file are present, whichever was edited most recently wins at runtime — `loglevel` is convenient for an interactive bump without touching `log.ini`.
+
+### log_timestamps
+
+A single-value file that toggles timestamp prepending. Equivalent to `main.timestamps` in `log.ini`. If either source enables timestamps, they are enabled.
+
+## Log Levels
+
+In ascending severity (and decreasing verbosity):
+
+| Level    | Numeric | Use |
+| -------- | ------- | --- |
+| DATA     | 9       | message body bytes — extremely verbose |
+| PROTOCOL | 8       | SMTP wire protocol |
+| DEBUG    | 7       | developer diagnostics |
+| INFO     | 6       | general informational |
+| NOTICE   | 5       | normal but significant events (connect/disconnect, summary lines) |
+| WARN     | 4       | recoverable problems |
+| ERROR    | 3       | non-fatal errors |
+| CRIT     | 2       | critical errors |
+| ALERT    | 1       | needs immediate attention |
+| EMERG    | 0       | unusable |
+
+A message is emitted when its level ≤ the configured level.
+
+## Logging API
+
+Every log call ultimately produces:
+
+    [level] [uuid] [origin] message
+
+`origin` is `core` or the plugin name; `uuid` is the connection UUID (with `.N` appended for the Nth transaction).
+
+The simplest call is on the connection or plugin object — origin and uuid are filled in automatically:
+
+```js
+connection.logdebug('turtles all the way down')
+plugin.loginfo('checking sender', connection)
+```
+
+Each of the level names has a matching method:
+`logdata`, `logprotocol`, `logdebug`, `loginfo`, `lognotice`, `logwarn`,
+`logerror`, `logcrit`, `logalert`, `logemerg`.
+
+Calling the logger directly works too — pass the plugin and/or connection anywhere in the arguments and the logger sniffs them:
+
+```js
+logger.logdebug('i like turtles', plugin, connection)
+// → [DEBUG] [7F1C820F-…] [dnsbl] i like turtles
+```
+
+Plain objects mixed into the arguments are merged into the log record (in `logfmt` / `json` formats) or stringified (`key=value` pairs in the default format).
+
+## Log Formats
+
+Set `main.format` in `log.ini` to one of `default`, `logfmt`, or `json`.
+
+`logfmt`:
+
+    level=PROTOCOL uuid=9FF7F70E-…1 source=core message="S: 354 go ahead, make my day"
+
+`json`:
+
+```json
+{
+  "level": "PROTOCOL",
+  "uuid": "9FF7F70E-…1",
+  "source": "core",
+  "message": "S: 354 go ahead, make my day"
+}
+```
+
+A typical structured disconnect line looks like:
+
+```json
+{
+  "level": "NOTICE",
+  "uuid": "9FF7F70E-…1",
+  "source": "core",
+  "message": "disconnect",
+  "ip": "127.0.0.1",
+  "rdns": "Unknown",
+  "helo": "3h2dnz8a0if",
+  "relay": "N",
+  "early": "N",
+  "esmtp": "N",
+  "tls": "N",
+  "pipe": "N",
+  "errors": 0,
+  "txns": 1,
+  "rcpts": "1/0/0",
+  "msgs": "1/0/0",
+  "bytes": 222,
+  "lr": "",
+  "time": 0.052
+}
+```
+
+## The `log` hook
+
+Each log message becomes a `log` hook invocation. The built-in handler writes to stdout (with ANSI colour when stdout is a TTY); log plugins can return `OK` or `STOP` to suppress the built-in output and ship the message elsewhere. Messages emitted before plugins finish loading are buffered and replayed once the plugin chain is ready.

package/docs/Outbound.md

@@ -0,0 +1,210 @@
+# Outbound Mail with Haraka
+
+A default Haraka installation queues outbound mail to disk and delivers it to the appropriate MX for each recipient domain. Temporary failures are retried automatically using the configured backoff schedule.
+
+A mail is treated as outbound when a plugin sets `connection.relaying` to `true`. The simplest way is SMTP AUTH using `auth/flat_file` or one of the [auth plugins](plugins/auth/); the `relay` plugin offers allow-list-based variants, and a custom plugin can apply any policy.
+
+For live stats on the outbound queue see the [`process_title`](plugins/process_title.md) plugin.
+
+To flush the temp-fail queue (e.g. after fixing network or DNS), send `SIGHUP` to the Haraka master process.
+
+## Outbound Configuration
+
+### outbound.ini
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `disabled` | `false` | Pause outbound delivery while still queuing inbound mail. Reloadable at runtime. |
+| `concurrency_max` | `10000` | Maximum concurrent outbound deliveries **per worker**. Effective total is `concurrency_max × nodes`. |
+| `enable_tls` | `true` | Use opportunistic STARTTLS on outbound. |
+| `maxTempFailures` | `13` | Maximum temp-fail retries before the message bounces. Ignored if `temp_fail_intervals` is set. |
+| `temp_fail_intervals` | derived | Comma-separated `<n><unit>[*<count>]` pattern. `1m, 5m*2, 1h*3` → `[60,300,300,3600,3600,3600]` seconds. `none` bounces on first temp-fail. |
+| `always_split` | `false` | Create one queue file per recipient (instead of one per destination domain). Hurts throughput but simplifies bounce handling. |
+| `received_header` | `Haraka outbound` | Text used in the outbound `Received:` header. Set to the literal `disabled` to omit it. |
+| `connect_timeout` | `30` | Seconds to wait for TCP connect to the remote MX. |
+| `local_mx_ok` | `false` | Allow outbound delivery to local/private IPs (otherwise blocked to prevent loops). |
+| `inet_prefer` | `default` | `default` (prefer IPv6 at equal MX priority), `v4`, or `v6`. Delivery still follows MX priority. |
+
+TLS configuration is shared with the `tls` plugin (`tls_key.pem`, `tls_cert.pem`, and `tls.ini`). Outbound-specific overrides go under `[outbound]` in `tls.ini`:
+
+```ini
+[outbound]
+ciphers=!DES
+minVersion=TLSv1.2
+```
+
+### outbound.bounce_message
+
+Template for the bounce message body. See "Bounce Messages" below.
+
+## The HMail Object
+
+Most outbound hooks pass an `hmail` (HMailItem). You rarely need its methods, but these properties are useful:
+
+| Property | Description |
+| --- | --- |
+| `path` | Full filesystem path to the queue file. |
+| `filename` | Queue file's base name. |
+| `num_failures` | Number of temp-fail attempts so far. |
+| `notes` | Plain object for plugin state, scoped to this queue item. |
+| `todo` | The `TODOItem` describing what to deliver (see below). |
+
+## The TODO Object
+
+`hmail.todo` describes the delivery:
+
+| Property | Description |
+| --- | --- |
+| `mail_from` | `Address`<sup>[1](#fn1)</sup> — the envelope sender. |
+| `rcpt_to` | `Address`<sup>[1](#fn1)</sup> array — envelope recipients. |
+| `domain` | Recipient domain (a single domain unless `always_split` is set). |
+| `notes` | The original `transaction.notes`. Keys you may set: |
+| `notes.outbound_ip` | IP to bind the outbound socket to. **Set via the `get_mx` hook**, not directly. |
+| `notes.outbound_helo` | EHLO domain. **Set via the `get_mx` hook**, not directly. |
+| `queue_time` | When the mail was queued (epoch ms). |
+| `uuid` | Inherited from the source `transaction.uuid`. |
+| `force_tls` | If `true`, defer instead of delivering in plaintext. |
+
+## Outbound Hooks
+
+### queue_outbound
+
+Runs before queuing. Returning `CONT` (or having no hook) queues the mail. `OK` indicates the plugin queued it itself; the `DENY*` codes reject the message.
+
+### pre_send_trans_email
+
+Parameters: `next, connection`
+
+Fired by `outbound.send_trans_email()` before the transaction is serialized to disk. Useful for plugins that synthesize mail programmatically — they can attach final headers or notes here.
+
+### send_email
+
+Parameters: `next, hmail`
+
+Called just before delivery starts. `next(DELAY, seconds)` defers the attempt.
+
+### get_mx
+
+Parameters: `next, hmail, domain`
+
+Called when delivery begins, with the destination domain. Plugins can override MX lookup; most installs leave Haraka to do DNS. Respond with `next(OK, mx)` where `mx` is a [HarakaMx][url-harakamx] object, an array of them, or any HarakaMx-compatible input. Set `mx.auth_user` / `mx.auth_pass` to AUTH against the remote, or `mx.bind` / `mx.bind_helo`
+to control source address and EHLO.
+
+### deferred
+
+Parameters: `next, hmail, { delay, err }`
+
+Fired on temporary failure. Return `OK` to drop the mail silently; return `DENYSOFT, seconds` to override the retry delay (useful for custom backoff indexed on `hmail.num_failures`).
+
+### bounce
+
+Parameters: `next, hmail, error`
+
+Fired on permanent failure (5xx). Not called for temp-fails. `error` may carry:
+
+- `mx` — the MX that caused the bounce
+- `deferred_rcpt` — recipients that eventually bounced after deferrals
+- `bounced_rcpt` — recipients that bounced outright
+
+Return `OK` to suppress the DSN to the original sender.
+
+### delivered
+
+Parameters: `next, hmail, [host, ip, response, delay, port, mode, ok_recips, secured, authenticated]`
+
+Fired after a successful delivery. Return codes are ignored; the hook is for logging / accounting.
+
+| Element | Description |
+| --- | --- |
+| `host` | Hostname of the receiving MX. |
+| `ip` | IP we delivered to. |
+| `response` | Remote SMTP response text (typically includes the remote queue ID). |
+| `delay` | Seconds between queue write and delivery. |
+| `port` | Destination port. |
+| `mode` | `'smtp'` or `'lmtp'`. |
+| `ok_recips` | `Address`<sup>[1](#fn1)</sup> array of successfully delivered recipients. |
+| `secured` | `true` if STARTTLS succeeded. |
+| `authenticated` | `true` if outbound AUTH succeeded. |
+
+## Outbound IP Address
+
+By default the OS routing table chooses the source IP. To pin outbound to a specific IP (per-sender, per-domain, etc.), bind that address to a local interface or alias, then set `mx.bind` (source IP) and `mx.bind_helo` (EHLO domain) in your `get_mx` hook.
+
+## Outbound AUTH
+
+Force AUTH for a domain or smart host by returning an MX with `auth_user` and `auth_pass` set from the `get_mx` hook. If the remote end does not advertise AUTH (or no compatible mechanism is found), delivery proceeds without AUTH and a warning is logged.
+
+## Bounce Messages
+
+The bounce body comes from `config/outbound.bounce_message`. Curly-brace template variables are filled in at bounce time:
+
+- `pid` — current process id
+- `date` — bounce timestamp
+- `me` — contents of `config/me`
+- `from` — original sender
+- `msgid` — original message UUID
+- `to` — original recipient (or first, for multi-recipient mail)
+- `reason` — remote server's rejection text
+
+The original message is appended to the bounce.
+
+For HTML bounces, add `config/outbound.bounce_message_html` (and optionally an inline image in `config/outbound.bounce_message_image`).
+
+## Generating Mail from a Plugin
+
+To create and queue a new message from inside a plugin, use the `outbound` module:
+
+```js
+const outbound = require('./outbound')
+
+const from = 'sender@example.com'
+const to = 'user@example.com'
+
+const contents = [
+    `From: ${from}`,
+    `To: ${to}`,
+    'MIME-Version: 1.0',
+    'Content-Type: text/plain; charset=us-ascii',
+    'Subject: Hello',
+    '',
+    'Body here.',
+    '',
+].join('\n')
+
+outbound.send_email(from, to, contents, (code, msg) => {
+    switch (code) {
+        case OK:
+            plugin.loginfo('queued')
+            break
+        case DENY:
+            plugin.logerror(`queue failed: ${msg}`)
+            break
+    }
+})
+```
+
+The callback fires when the mail is **queued**, not delivered — hook `delivered` and `bounce` to observe delivery outcomes.
+
+The callback may be omitted if you don't need to handle queue failure:
+
+```js
+outbound.send_email(from, to, contents)
+```
+
+Options accepted by `send_email(from, to, contents, next, options)`:
+
+| Option | Description |
+| --- | --- |
+| `dot_stuffed: true` | Content is already SMTP dot-stuffed. |
+| `notes: { … }` | Seed the new transaction's `notes`. |
+| `remove_msgid: true` | Drop any existing `Message-Id:` so Haraka generates one. Useful when releasing from quarantine. |
+| `remove_date: true` | Drop any existing `Date:` so Haraka generates one. |
+| `origin: <object>` | Object passed to the logger to identify the source plugin / connection / HMailItem. |
+
+To send an already-built `Transaction` directly, use `outbound.send_trans_email(transaction, next)`. This is what `send_email()` calls internally and fires the `pre_send_trans_email` hook.
+
+<a name="fn1">1</a>: `Address` objects are [@haraka/email-address](https://github.com/haraka/email-address) objects.
+
+[url-tls]: plugins/tls.md
+[url-harakamx]: https://github.com/haraka/haraka-net-utils?tab=readme-ov-file#harakamx
+[url-rfc2821]: https://tools.ietf.org/html/rfc2821#section-4.5.2