Haraka 3.1.4 → 3.1.6

package/docs/CoreConfig.md

@@ -1,76 +1,94 @@
 # Core Configuration Files
 
-See [Logging](Logging.md).
-
-The Haraka core reads some configuration files to determine a few actions:
-
-- smtp.yaml or smtp.json
-
-If either of these files exist then they are loaded first.
-This file is designed to use the JSON/YAML file overrides documented in
-[haraka-config](https://github.com/haraka/haraka-config) to optionally provide the entire configuration in a single file.
-
-- plugins
-
-The list of plugins to load
-
-- smtp.ini
-
-  Keys:
-
-  - listen_host, port - the host and port to listen on (default: ::0 and 25)
-  - listen - (default: [::0]:25) Comma separated IP:Port addresses to listen on
-  - inactivity_time - how long to let clients idle in seconds (default: 300)
-  - nodes - specifies how many processes to fork. The string "cpus" will fork as many children as there are CPUs (default: 1, which enables cluster mode with a single process)
-  - user - optionally a user to drop privileges to. Can be a string or UID.
-  - group - optionally a group to drop privileges to. Can be a string or GID.
-  - ignore_bad_plugins - If a plugin fails to compile by default Haraka will stop at load time.
-    If, however, you wish to continue on without that plugin's facilities, then
-    set this config option
-  - daemonize - enable this to cause Haraka to fork into the background on start-up (default: 0)
-  - daemon_log_file - (default: /var/log/haraka.log) where to redirect stdout/stderr when daemonized
-  - daemon_pid_file - (default: /var/run/haraka.pid) where to write a PID file to
-  - graceful_shutdown - (default: false) enable this to wait for sockets on shutdown instead of closing them quickly
-  - force_shutdown_timeout - (default: 30) number of seconds to wait for a graceful shutdown
-
-- me
-
-  A name to use for this server. Used in received lines and elsewhere. Setup
-  by default to be your hostname.
-
-- connection.ini
-
-  See inline comments in connection.ini for the following settings:
-
-  - main.spool_dir
-  - main.spool_after
-  - haproxy.hosts_ipv4
-  - haproxy.hosts_ipv6
-  - headers.\*
-  - max.bytes
-  - max.line_length
-  - max.data_line_length
-  - max.mime_parts
-  - message.greeting
-  - message.close
-  - smtputf8
-  - strict_rfc1869
-  - uuid.deny_chars
-  - uuid.banner_bytes
-
-- plugin_timeout
-
-  Seconds to allow a plugin to run before the next hook is called automatically
-  default: 30
-
-  Note also that each plugin can have a `config/<plugin_name>.timeout`
-  file specifying a per-plugin timeout. In this file you can set a timeout of 0 to mean that this plugin's hooks never time out. Use this with care.
-
-  If the plugin is in a sub-directory of plugins, then you must create this file
-  in the equivalent path e.g. the queue/smtp_forward would need a timeout file in `config/queue/smtp_forward.timeout`
-
-- outbound.ini
-
-- outbound.bounce_message
-
-  The bounce message if delivery of the message fails. The default is normally fine. Bounce messages contain a number of template replacement values which are best discovered by looking at the source code.
+Haraka reads the configuration files in this document directly. Plugins typically own their own files (named after the plugin); see the individual plugin docs.
+
+See also [Logging](Logging.md), [HAProxy](HAProxy.md), and [Outbound](Outbound.md).
+
+## smtp.yaml / smtp.json
+
+If either of these files exists it is loaded first. It uses the YAML/JSON override mechanism from [haraka-config](https://github.com/haraka/haraka-config) and can provide the entire configuration as a single file.
+
+## plugins
+
+A newline-delimited list of plugins to load. Lines starting with `#` are ignored.
+
+## smtp.ini
+
+Controls the SMTP listener and the master process.
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `listen` | `[::0]:25` | Comma-separated `IP:port` pairs to listen on (e.g. `127.0.0.1:25,127.0.0.1:587`). |
+| `listen_host` / `port` | — | Legacy. If set, a `<listen_host>:<port>` entry is prepended to `listen`. Prefer `listen`. |
+| `smtps_port` | `465` | Port used by the optional implicit-TLS listener. |
+| `public_ip` | none | The server's public IP. Helps NAT-aware plugins (SPF, GeoIP) when Haraka is behind NAT. If `stun` is on `$PATH` Haraka will try to discover it automatically. |
+| `inactivity_timeout` | `300` | Idle seconds before a client socket is dropped. |
+| `nodes` | `1` | Number of worker processes to fork. The string `cpus` forks one per CPU. |
+| `user` / `group` | — | User and group to drop privileges to (name or numeric ID). |
+| `ignore_bad_plugins` | `0` | If `1`, Haraka starts even if some plugins fail to compile. |
+| `daemonize` | `false` | If `true`, fork into the background at start-up. |
+| `daemon_log_file` | `/var/log/haraka.log` | Where to redirect stdout/stderr when daemonized. |
+| `daemon_pid_file` | `/var/run/haraka.pid` | Where to write the PID file. |
+| `graceful_shutdown` | `false` | If `true`, wait for in-flight sockets on shutdown. |
+| `force_shutdown_timeout` | `30` | Seconds to wait before forcing shutdown. |
+
+## me
+
+A single-line file containing the server name used in `Received:` headers
+and elsewhere. Defaults to `hostname(1)`.
+
+## connection.ini
+
+Per-connection limits and behaviours. See inline comments in the shipped
+`config/connection.ini` for full details.
+
+| Section / Key | Default | Description |
+| --- | --- | --- |
+| `main.spool_dir` | `/tmp` | Directory for temporary spool files. |
+| `main.spool_after` | `-1` | Size (bytes) at which to spool the message to disk. `-1` never spools; `0` always spools. |
+| `main.strict_rfc1869` | `false` | Reject `MAIL FROM` / `RCPT TO` that violates RFC 1869/821 (spurious spaces, missing brackets). |
+| `main.smtputf8` | `true` | Advertise `SMTPUTF8` (RFC 6531). |
+| `haproxy.hosts` | empty | Array of IPs/CIDRs allowed to send the PROXY protocol. See [HAProxy.md](HAProxy.md). |
+| `headers.add_received` | `true` | Add a `Received:` header to incoming mail. |
+| `headers.clean_auth_results` | `true` | Strip inbound `Authentication-Results:` headers before plugins run. |
+| `headers.show_version` | `true` | Include the Haraka version in the SMTP banner and the `Received:` header. |
+| `headers.max_lines` | `1000` | Maximum number of header lines accepted. |
+| `headers.max_received` | `100` | Maximum number of `Received:` headers before mail is rejected as looping. |
+| `max.bytes` | `26214400` | Maximum message size (advertised as the `SIZE` ESMTP extension). |
+| `max.line_length` | `512` | SMTP command line length cap. Clients exceeding this are dropped with `521`. |
+| `max.data_line_length` | `992` | Maximum line length in `DATA`. Longer lines are wrapped with `CRLF SPACE` (Sendmail behaviour) and `transaction.notes.data_line_length_exceeded` is set. |
+| `max.mime_parts` | `1000` | Maximum MIME parts per message. |
+| `message.greeting` | — | Array. Lines used as the SMTP greeting banner. |
+| `message.helo` | `Haraka is at your service.` | Reply text for `HELO` / `EHLO`. |
+| `message.close` | `closing connection. …` | Reply text on `QUIT`. |
+| `uuid.banner_chars` | `6` | Number of UUID chars included in the SMTP banner (`0` to disable, `40` for the full UUID). |
+| `uuid.deny_chars` | `0` | Number of UUID chars prepended (in brackets) to deny messages. |
+
+## plugin_timeout
+
+Single-integer file. Seconds to allow a plugin hook to run before Haraka
+automatically advances to the next hook. Default: `30`.
+
+A per-plugin override may be placed at `config/<plugin>.timeout`. A value
+of `0` disables the timeout for that plugin — use with care. Plugins in
+subdirectories need a matching path: `queue/smtp_forward` looks for
+`config/queue/smtp_forward.timeout`.
+
+## outbound.ini
+
+Configures the outbound delivery engine. The most common keys are listed
+below; see [Outbound.md](Outbound.md) for the full set.
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `disabled` | `false` | Disable outbound delivery entirely. |
+| `concurrency_max` | `100` | Maximum simultaneous outbound deliveries. |
+| `enable_tls` | `true` | Use STARTTLS opportunistically on outbound deliveries. |
+| `maxTempFailures` | `13` | Number of temporary failures before a message bounces. |
+| `always_split` | `false` | Create one queue file per recipient instead of one per destination domain. |
+| `received_header` | `Haraka outbound` | Value used in the outbound `Received:` header. |
+| `inet_prefer` | `default` | IP family preference for MX lookups: `v4`, `v6`, or `default` (OS preference). |
+
+## outbound.bounce_message
+
+Template used when an outbound message bounces. The default is usually fine. Available template variables are documented in the source of `outbound/hmail.js`.

package/docs/HAProxy.md

@@ -1,43 +1,59 @@
-# HAProxy PROXY protocol extension support
+# HAProxy PROXY Protocol Support
 
-Haraka supports PROXY protocol [1].
+Haraka supports version 1 (text format) of the HAProxy [PROXY protocol][proxy-spec], which allows an upstream proxy to tell Haraka the real client IP and port. DNSBLs, allow-lists, and other IP-based plugins then see the original client rather than the proxy.
 
-This allows an upstream proxy to pass the IP address and port of the remote client. Haraka will use the remote IP instead of the socket IP address (which is the proxy). This allows DNSBLs and access control lists to use the correct source address.
+The PROXY v2 binary header is not currently supported.
 
-Support is disabled by default. Attempts to send a PROXY command will return a DENYSOFTDISCONNECT error. DENYSOFT is used to prevent configuration errors from rejecting valid mail.
+## Configuration
 
-To enable support for PROXY you must populate connection.ini[haproxy]hosts[] with the IP addresses of the HAProxy hosts that MUST send the PROXY command. Ranges can be specified with CIDR notation.
+PROXY support is disabled by default. To enable it, list the IPs (or CIDRs) of trusted proxies in `connection.ini`:
 
-When a proxy host connects to Haraka, a banner is not sent. Instead Haraka awaits the PROXY command. The connection will timeout with `421 PROXY timed out` if the command is not sent within 30 seconds.
+```ini
+[haproxy]
+hosts[] = 192.0.2.4
+hosts[] = 192.0.2.5/30
+hosts[] = 2001:db8::1
+```
+
+Connections from any other IP get a `DENYSOFTDISCONNECT` if they send a `PROXY` command. `DENYSOFT` is deliberate — it avoids permanently rejecting valid mail when a misconfiguration causes a legitimate proxy to fall outside the allow-list.
+
+When a listed proxy connects, Haraka **does not** send the SMTP banner; it waits for the `PROXY` command. If none arrives within 30 seconds the connection is closed with `421 PROXY timed out`.
 
-NOTE: because Haraka does not send a banner when a listed HAProxy host connects you must set check-send-proxy to ensure that the service checks send a PROXY command before they run.
+## What plugins see after PROXY is parsed
 
-[1] http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
+| Property | Value |
+| --- | --- |
+| `connection.remote.ip` / `.port` | the **real** client address from the PROXY header |
+| `connection.local.ip` / `.port` | the destination IP/port from the PROXY header |
+| `connection.proxy.allowed` | `true` |
+| `connection.proxy.ip` | the proxy's address (i.e. the original socket peer) |
+| `connection.proxy.type` | `'haproxy'` |
+| `connection.notes.proxy` | full parsed record: `{ type, proto, src_ip, src_port, dst_ip, dst_port, proxy_ip }` |
 
-HAProxy supports the PROXY protocol in version 1.5 or later.
+## HAProxy configuration
 
-Here is an example listener section for haproxy.cfg:
+You need HAProxy ≥ 1.5. The `send-proxy` option on each backend server tells HAProxy to emit the v1 header on every connection.
 
 ```
 listen smtp :25
-        mode tcp
-        option tcplog
-        option smtpchk
-        balance roundrobin
-        server smtp1 ip.of.haraka.server1:25 check-send-proxy check inter 10s send-proxy
-        server smtp2 ip.of.haraka.server2:25 check-send-proxy check inter 10s send-proxy
-        server smtp3 ip.of.haraka.server3:25 check-send-proxy check inter 10s send-proxy
-        server smtp4 ip.of.haraka.server4:25 check-send-proxy check inter 10s send-proxy
-        server smtp5 ip.of.haraka.server5:25 check-send-proxy check inter 10s send-proxy
+    mode tcp
+    option tcplog
+    balance roundrobin
+    server smtp1 ip.of.haraka1:25 check-send-proxy check inter 10s send-proxy
+    server smtp2 ip.of.haraka2:25 check-send-proxy check inter 10s send-proxy
 ```
 
-The important part is `send-proxy` which causes HAProxy to send the PROXY extension on connection.
+The `check-send-proxy` flag is required for HAProxy's health checks because Haraka does not respond with a banner before the PROXY header arrives.
 
-When using `option smtpchk` you will see CONNRESET errors reported in the Haraka logs as smtpchk drops the connection before the HELO response is still being written. You can use the `option tcp-check` instead to provide a better service check by having the check wait for the banner, send QUIT and then check the response:
+### Health checks
+
+`option smtpchk` drops the connection mid-handshake and shows up as `CONNRESET` in Haraka's logs. A cleaner check is to use `tcp-check` and politely close with `QUIT`:
 
 ```
-        option tcp-check
-        tcp-check expect rstring ^220\
-        tcp-check send QUIT\r\n
-        tcp-check expect rstring ^221\
+    option tcp-check
+    tcp-check expect rstring ^220
+    tcp-check send QUIT\r\n
+    tcp-check expect rstring ^221
 ```
+
+[proxy-spec]: https://www.haproxy.org/download/2.8/doc/proxy-protocol.txt

package/docs/Logging.md

@@ -1,83 +1,109 @@
 # Haraka Logging
 
-Haraka has built-in logging (see API docs below) and support for log plugins.
+Haraka has a built-in logger (described below) and a plugin hook (`log`) that lets log plugins ship messages elsewhere — for example to syslog via [haraka-plugin-syslog](https://github.com/haraka/haraka-plugin-syslog).
 
-- log.ini
+## Configuration
 
-Contains settings for log level, timestamps, and format. See the example log.ini file for examples.
+### log.ini
 
-- loglevel
+```ini
+[main]
+; data, protocol, debug, info, notice, warn, error, crit, alert, emerg
+level=info
 
-The loglevel file provides a finger-friendly way to change the loglevel on the CLI. Use it like so: `echo DEBUG > config/loglevel`. When the level in log.ini is set and the loglevel file is present, the loglevel file wins. During runtime, whichever was edited most recently wins.
+; prepend ISO-8601 timestamps to log entries (built-in logger only)
+timestamps=false
 
-## Logging API
+; default, logfmt, json
+format=default
+```
 
-Logging conventions within Haraka
+### loglevel
 
-This section pertains to the built in logging. For log plugins like ([haraka-plugin-syslog](https://github.com/haraka/haraka-plugin-syslog)), refer to the plugin's docs.
+A single-line file for quick CLI tweaks:
 
-The logline by default will be in the form of:
+```sh
+echo DEBUG > config/loglevel
+```
 
-    [level] [uuid] [origin] message
+When both `log.ini` and the `loglevel` file are present, whichever was edited most recently wins at runtime — `loglevel` is convenient for an interactive bump without touching `log.ini`.
 
-Where origin is "core" or the name of the plugin which triggered the message, and "uuid" is the ID of the connection associated with the message.
+### log_timestamps
 
-When calling a log method on logger, you should provide the plugin object and the connection object anywhere in the arguments
-to the log method.
+A single-value file that toggles timestamp prepending. Equivalent to `main.timestamps` in `log.ini`. If either source enables timestamps, they are enabled.
 
-    logger.logdebug("i like turtles", plugin, connection);
+## Log Levels
 
-Will yield, for example,
+In ascending severity (and decreasing verbosity):
 
-    [DEBUG] [7F1C820F-DC79-4192-9AA6-5307354B20A6] [dnsbl] i like turtles
+| Level    | Numeric | Use |
+| -------- | ------- | --- |
+| DATA     | 9       | message body bytes — extremely verbose |
+| PROTOCOL | 8       | SMTP wire protocol |
+| DEBUG    | 7       | developer diagnostics |
+| INFO     | 6       | general informational |
+| NOTICE   | 5       | normal but significant events (connect/disconnect, summary lines) |
+| WARN     | 4       | recoverable problems |
+| ERROR    | 3       | non-fatal errors |
+| CRIT     | 2       | critical errors |
+| ALERT    | 1       | needs immediate attention |
+| EMERG    | 0       | unusable |
 
-If you call the log method on the connection object, you can forego the connection as argument:
+A message is emitted when its level ≤ the configured level.
 
-    connection.logdebug("turtles all the way down", plugin);
+## Logging API
 
-and similarly for the log methods on the plugin object:
+Every log call ultimately produces:
 
-    plugin.logdebug("he just really likes turtles", connection);
+    [level] [uuid] [origin] message
 
-failing to provide a connection and/or plugin object will leavethe default values in the log (currently "core").
+`origin` is `core` or the plugin name; `uuid` is the connection UUID (with `.N` appended for the Nth transaction).
 
-This is implemented by testing for argument type in the logger.js log\* method. objects-as-arguments are then sniffed to try to determine if they're a connection or plugin instance.
+The simplest call is on the connection or plugin object — origin and uuid are filled in automatically:
 
-### Log formats
+```js
+connection.logdebug('turtles all the way down')
+plugin.loginfo('checking sender', connection)
+```
 
-Apart from the default log format described above, Haraka also supports logging as [`logfmt`](https://brandur.org/logfmt) and JSON. These can be used by changing the `format` attribute in `log.ini` to the desired format, e.g.:
+Each of the level names has a matching method:
+`logdata`, `logprotocol`, `logdebug`, `loginfo`, `lognotice`, `logwarn`,
+`logerror`, `logcrit`, `logalert`, `logemerg`.
 
-```ini
-; format=default
-; format=json
-format=logfmt
+Calling the logger directly works too — pass the plugin and/or connection anywhere in the arguments and the logger sniffs them:
+
+```js
+logger.logdebug('i like turtles', plugin, connection)
+// → [DEBUG] [7F1C820F-…] [dnsbl] i like turtles
 ```
 
-Here's an example of a log line generated with `logfmt`:
+Plain objects mixed into the arguments are merged into the log record (in `logfmt` / `json` formats) or stringified (`key=value` pairs in the default format).
+
+## Log Formats
+
+Set `main.format` in `log.ini` to one of `default`, `logfmt`, or `json`.
+
+`logfmt`:
 
-    level=PROTOCOL uuid=9FF7F70E-5D57-435A-AAD9-EA069B6159D9.1 source=core message="S: 354 go ahead, make my day"
+    level=PROTOCOL uuid=9FF7F70E-…1 source=core message="S: 354 go ahead, make my day"
 
-And the same line formatted as JSON:
+`json`:
 
 ```json
 {
   "level": "PROTOCOL",
-  "uuid": "9FF7F70E-5D57-435A-AAD9-EA069B6159D9.1",
+  "uuid": "9FF7F70E-…1",
   "source": "core",
   "message": "S: 354 go ahead, make my day"
 }
 ```
 
-Any objects passed to the log methods will also have their properties included in the log line. For example, using `logfmt`:
-
-    level=NOTICE uuid=9FF7F70E-5D57-435A-AAD9-EA069B6159D9.1 source=core message=disconnect ip=127.0.0.1 rdns=Unknown helo=3h2dnz8a0if relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=1/0/0 msgs=1/0/0 bytes=222 lr="" time=0.052
-
-And using JSON:
+A typical structured disconnect line looks like:
 
 ```json
 {
   "level": "NOTICE",
-  "uuid": "9FF7F70E-5D57-435A-AAD9-EA069B6159D9.1",
+  "uuid": "9FF7F70E-…1",
   "source": "core",
   "message": "disconnect",
   "ip": "127.0.0.1",
@@ -97,3 +123,7 @@ And using JSON:
   "time": 0.052
 }
 ```
+
+## The `log` hook
+
+Each log message becomes a `log` hook invocation. The built-in handler writes to stdout (with ANSI colour when stdout is a TTY); log plugins can return `OK` or `STOP` to suppress the built-in output and ship the message elsewhere. Messages emitted before plugins finish loading are buffered and replayed once the plugin chain is ready.