messagefoundry 0.2.8__tar.gz → 0.2.9__tar.gz

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/.github/workflows/ci.yml

@@ -23,9 +23,11 @@ jobs:
   test:
     name: test (${{ matrix.os }}, py${{ matrix.python-version }})
     runs-on: ${{ matrix.os }}
-    # Wall-clock backstop: bound each leg so a hung test fails in minutes, not the 6h default. The
-    # pytest-timeout watchdog (60s/test) should catch a culprit first; this is the belt-and-suspenders cap
-    # if a whole process ever deadlocks below pytest.
+    # Wall-clock backstop: bound each leg so a hung test fails in minutes, not the 6h default. Three
+    # nested watchdogs catch a hang at increasing scope (#55): the pytest-timeout per-test cap (60s,
+    # pyproject addopts) + the faulthandler belt fire FIRST and name the stuck frame; the pytest STEP
+    # has its own `timeout-minutes: 13` so a process-level deadlock below pytest fails the step fast;
+    # this job cap is the outermost belt if even that is somehow out-raced.
     timeout-minutes: 15
     strategy:
       fail-fast: false
@@ -75,10 +77,27 @@ jobs:
       - name: Type-check (mypy, strict)
         run: mypy messagefoundry
 
+      # Step-level watchdog UNDER the 15-min job cap (#55): the windows-2022 leg intermittently
+      # wedges ~25% in (a Windows ProactorEventLoop listener-teardown / socket wait that the shared
+      # session event loop can't get past), emits no output for ~12 min, then the JOB cap CANCELS it
+      # — a red ✗ with no stack and no named test. Two belts make that fail FAST and NAMED instead:
+      #   * `--timeout-method=thread` (the ONLY method on Windows — SIGALRM is POSIX-only) dumps ALL
+      #     thread stacks at the per-test cap (60s, set in pyproject addopts), naming the stuck frame.
+      #   * PYTHONFAULTHANDLER=1 + `-o faulthandler_timeout=90` is the belt that fires even when the
+      #     thread-timer CANNOT interrupt a main-thread C-level wait (a Proactor overlapped op /
+      #     blocking accept / subprocess.wait): pytest's faulthandler plugin arms a separate watchdog
+      #     thread that dumps the NATIVE stack of every thread at 90s (> the 60s pytest-timeout, so the
+      #     per-test dump is attributed first; this is the last-resort stack when the thread method is
+      #     itself out-raced). NB the plugin exposes this as the `faulthandler_timeout` ini key, not a
+      #     `--faulthandler-*` CLI flag, so it is passed with `-o`; it only DUMPS (never kills), safe.
+      # The step `timeout-minutes: 13` is the outer backstop, well under the job's 15-min cap, so a
+      # hang both watchdogs somehow miss still fails the STEP (not the whole job at the silent cap).
       - name: Tests (pytest)
+        timeout-minutes: 13
         env:
           QT_QPA_PLATFORM: offscreen
-        run: pytest -q
+          PYTHONFAULTHANDLER: "1"
+        run: pytest -q -o faulthandler_timeout=90
 
   # Build + type-check the VS Code extension, and run its integration tests. The Python jobs never
   # touch ide/, so a dep bump or IDE code change that breaks the bundle or the types would otherwise
@@ -592,7 +611,14 @@ jobs:
           # install-service.ps1 -Environment threads it into the service command + resolves
           # environments/prod.toml). INFO not DEBUG: `serve` refuses DEBUG in prod (Gate #1 — DEBUG can
           # surface PHI); INFO is enough to smoke service start + MLLP.
-          .\scripts\service\install-service.ps1 -AppExe $exe -LogLevel INFO -Environment prod
+          # -LockConfigDir: the runner checks out samples/config with the default ACL, which grants
+          # BUILTIN\Users (S-1-5-32-545) write. The in-process config-source trust guard (SEC-003,
+          # wiring._assert_safe_config_source_windows) then REFUSES to load it ("writable-by-others
+          # path") and `serve` crash-loops to SERVICE_PAUSED, so /health never comes up. Locking the
+          # config dir (strip inheritance -> SYSTEM/Administrators only; the LocalSystem service still
+          # reads it) is the production-correct posture docs/SERVICE.md prescribes, and exercises the
+          # real -LockConfigDir install path rather than the MEFOR_ALLOW_INSECURE_CONFIG_SOURCE dev escape.
+          .\scripts\service\install-service.ps1 -AppExe $exe -LogLevel INFO -Environment prod -LockConfigDir
           # The shipped samples graph includes a mutual-TLS WS-* SOAP feed (IB_IMMUNIZATION_VXU) whose
           # connector LOADS its client cert at wiring time, so the engine refuses to start the graph
           # without a readable cert+key. Mint a throwaway pair for the smoke — the smoke sends only an

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/CHANGELOG.md

@@ -6,6 +6,63 @@ All notable changes to MessageFoundry are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.9] — 2026-06-27 — Early Access
+
+A retention + security-hardening + observability release: per-connection retention and
+embedded-document pruning windows, dual-control config reloads with startup code-attestation,
+operational-health metrics, and a fix for the intermittent Windows listener-teardown / CI hang.
+
+### Added
+- **Per-connection retention windows (ADR 0027).** Optional `messages_days` (inbound) and
+  `dead_letter_days` (outbound) on a connection, layered over the global `[retention]` window and
+  authored on the connection spec or `connections.toml` (the same override idiom as the delivery
+  knobs): `None` inherits the global window, `0` keeps forever. The `RetentionRunner` threads a
+  per-connection cutoff through the body and dead-letter purge on **all three** store backends; the
+  never-purge-an-in-flight-body guard and the single per-pass audit row (now recording the overrides)
+  are unchanged. (#34)
+- **Embedded-document pruning (ADR 0042).** Optional `prune_documents_after` (+ a size threshold)
+  per inbound connection: after the window, bulky **base64 embedded documents** — HL7 **OBX-5 ED**
+  and the generic `mfb64:v1:` carriage — are stripped **in place** to a small size/content-type
+  tombstone (via the parsed model / codec, **never** string-slicing HL7), keeping the rest of the
+  message parseable; the row is never deleted and a `documents_pruned` flag is set. All three
+  backends. (The ingest-time offload variant remains deferred.) (#47)
+- **Dual-control `config:reload` (ADR 0041 D2).** `config_reload` is now a gateable
+  `[approvals].operations` op — a **distinct** second approver must release a live config reload
+  (the requester can never self-approve; both identities land in the hash-chained audit). Opt-in /
+  deny-by-default, so single-operator deployments are unchanged. (#53)
+- **Startup code self-attestation (ADR 0041 D3).** At startup the engine hashes its loaded modules
+  against the wheel's `dist-info/RECORD`; on drift it records a hash-chained, off-box-teed
+  `startup_integrity` audit row and raises an alert (**alert-only by default**; opt-in
+  `[integrity].fail_closed_on_drift` refuses to start). A no-op on an editable (`pip install -e .`)
+  install, so development is never bricked. (#54)
+- **Operational-health metrics.** `GET /status` now meters the app-log directory's disk usage
+  alongside the database, and a per-connection **message-stall** alert rule fires when a connection's
+  oldest-undelivered age crosses a configurable threshold. (#50)
+
+### Changed
+- **Non-editable, hash-locked wheel is the enforced production default (ADR 0017 amendment).** The
+  prior recommendation is now the default for production deployments; editable installs remain a
+  no-op for development. (#54)
+
+### Fixed
+- **Intermittent Windows listener-teardown hang.** `MLLPSource` / `TcpSource` / `X12Source` no longer
+  `await server.wait_closed()` / `writer.wait_closed()` **unbounded** on the Windows Proactor loop
+  during teardown — a wait that never completes can no longer stall a shared event loop (the same
+  class as the resolved py3.11 hang). Added CI guards (a per-test `faulthandler` stack dump and a
+  step-level watchdog) so a future hang fails fast and names itself instead of silently timing out. (#55)
+
+### Docs
+- Refreshed `benchmarks/TUNING-BASELINE.md` with measured multi-process sharding throughput from the
+  Windows Server 2025 box (η ≈ 0.85 speedup shape; per-shard E_core ≈ 42 msg/s — a test-box SQLite
+  floor), plus the still-unmeasured hardware-gated follow-ups (enterprise E_core, the shared-DB
+  commit-wall sweep). (#28, #29)
+- Authored **ADR 0027** (per-connection retention) and **ADR 0042** (embedded-document pruning); added
+  EARS acceptance criteria to **ADR 0041** D2/D3; amended **ADR 0017** for the enforced wheel.
+
+### CI
+- Locked the smoke job's config directory (#603) and skipped a mirror-only Dependabot guardrail test
+  on the OSS mirror (#606), greening `main` CI post-0.2.8.
+
 ## [0.2.8] — 2026-06-27 — Early Access
 
 A tooling/ops release: the load harness gains a **multi-shard driver** so one harness can drive a

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messagefoundry
-Version: 0.2.8
+Version: 0.2.9
 Summary: Open-source healthcare integration engine — route, transform, and validate messages across many formats and connection types
 Project-URL: Homepage, https://messagefoundry.org/
 Project-URL: Documentation, https://messagefoundry.org/

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/docs/CONFIGURATION.md

@@ -390,6 +390,7 @@ the rest are forward-compat placeholders for the future engine broker (accepted-
 |---|---|---|---|
 | `level` | enum | `info` | log level; never run prod at `debug` (PHI) — `serve` refuses it (Gate #1) |
 | `format` | enum | `text` | stdout rendering: `text` (default) or structured `json` (one object per line). Stdlib only — no structlog |
+| `log_dir` | str | _unset_ | the directory NSSM (or another supervisor) **rotates the engine's captured stdout/stderr into**. The engine never writes log **files** itself (it logs to stdout); set this only to tell it where the supervisor parks them, and `GET /status` then **meters that directory's total bytes + filesystem free space** alongside the DB metrics (#50). Unset = stdout-only, no metering. **Metadata only** — the file contents are never read. |
 | `forward_enabled` | bool | `false` | ship a copy of every record off-box to a syslog/SIEM collector (sec-offbox-log) so evidence survives a host compromise; requires `forward_host` |
 | `forward_host` | str | — | syslog/SIEM collector host (**required** when `forward_enabled`) |
 | `forward_port` | int | `514` | collector port (1–65535) |
@@ -412,12 +413,21 @@ so retention is opt-in.
 | `messages_days` | int | `0` | past N days, null inbound bodies (`raw`/`summary`/`error`) of **fully-resolved** messages (no `pending`/`inflight` delivery), keeping metadata. `0` = keep |
 | `dead_letter_days` | int | `0` | past N days, null the bodies of **dead-lettered** outbound rows (their own window — a dead row stays replayable until purged). `0` = keep |
 | `state_max_age_days` | int | `0` | past N days, **delete** transform-state entries (ADR 0005) last written before the cutoff — keeps the in-memory state cache + table bounded. A simple global age purge (by `set_at`); per-namespace policy is a follow-up. `0` = keep |
+| `connection_event_retention_hours` | int | `0` | past N **hours**, **delete** `connection_event` rows (the `[diagnostics]` #46 transport/lifecycle log — high-volume under a connect-per-message sender or a probe storm, so its own short window in **hours**, not days). `0` = inherit the `messages_days` body window (the ADR 0021 §7.5 default). |
 | `audit_days` | int | `0` | **reserved / not enforced.** The `audit_log` is a tamper-evident hash chain and HIPAA expects ~6-year retention, so audit is **keep-forever by design**; archive-first pruning is a tracked follow-up. Accepted so a forward-looking file still loads |
 | `max_db_mb` | int | `0` | advisory only: warn (WARNING log + an `AlertSink` `storage_threshold` event) when the DB (+ `-wal`/`-shm`) exceeds this. Never auto-deletes. `0` = off |
 | `purge_interval_seconds` | float | `3600` | how often the purge/maintenance loop runs a pass |
 | `wal_checkpoint_seconds` | float | `0` | `PRAGMA wal_checkpoint(TRUNCATE)` cadence (SQLite). `0` = off (rely on auto-checkpoint). Evaluated once per pass, so a value below `purge_interval_seconds` is effectively rounded up to it |
 | `vacuum_at` | str | `""` | daily local `"HH:MM"` to run `VACUUM` (SQLite; reclaims space freed by purges). `""` = off. A daily off-peak time, **not** a cron expression (no new dependency); VACUUM holds a write lock on the whole DB while it runs |
 
+> **Per-connection overrides ([ADR 0027](adr/0027-per-connection-retention.md)).** `messages_days` and
+> `dead_letter_days` are **global defaults** an individual connection may override: an **inbound** sets its
+> own `messages_days`, an **outbound** its own `dead_letter_days` (both `None` = inherit this global window,
+> `0` = keep that connection's bodies forever, `>0` = days). An inbound may also opt into **embedded-document
+> pruning** (`prune_documents_after` + `prune_documents_min_bytes`, [ADR 0042](adr/0042-embedded-document-pruning.md))
+> to strip bulky base64 attachments while keeping the readable message. These live on the connection (code-first
+> or in `connections.toml`) — see [CONNECTIONS.md](CONNECTIONS.md).
+
 > **SQLite-only.** Retention/maintenance runs on the SQLite backend. On the SQL Server
 > backend it is a DBA concern (TDE + a SQL Agent purge/shrink job). Each pass that does real work
 > writes one `retention_purge` `audit_log` entry (cutoffs + counts, **no** message content). Design:
@@ -432,6 +442,7 @@ so retention is opt-in.
 | `internal_error` | enum | `continue` | what a delivery worker does on an **internal/code error** (a non-`DeliveryError` exception from `send` — our bug, not the partner's): `continue` (dead-letter the row + advance) or `stop` (halt the connection's worker, preserve the message for replay, raise a `connection_stopped` alert). Per-outbound `internal_error=` overrides. Partner NAKs / transport failures are unaffected. |
 | `buildup_max_depth` | int | _unset_ | raise a `queue_buildup` alert when an outbound lane's pending depth reaches this. Unset = depth dimension off (a healthy ceiling is throughput-specific, so there's no safe default). Per-outbound `buildup=BuildupThreshold(...)` overrides. |
 | `buildup_max_oldest_seconds` | num | 300 | raise `queue_buildup` when the lane's **oldest** pending message has waited this long (a stuck/retry-forever head is the classic cause). On by default — a head stuck >5 min is a problem in any environment. Set to unset/`0`-disable via a per-outbound override. |
+| `stall_max_oldest_seconds` | num | _unset_ | raise a `message_stall` alert (Corepoint "Max Message Stall", [ADR 0014](adr/0014-alert-routing-rules.md)) when an outbound lane's **oldest undelivered message** has waited this long. **Unset (the default) = the stall alert is OFF** — deny-by-default/opt-in, because it overlaps `buildup_max_oldest_seconds`'s age dimension and would double-page if both fired. Set a threshold to turn it on; a per-outbound `stall=StallThreshold(...)` overrides it. The stall event routes through `[[alerts.rules]]` like any other ([ADR 0014](adr/0014-alerting-rules-engine.md)). |
 | `outbox_workers` | int | per-outbound | delivery concurrency (planned) |
 | `dead_letter` | enum | `keep` | `keep`/`drop`-after-N (planned) |
 
@@ -440,6 +451,22 @@ so retention is opt-in.
 |---|---|---|---|
 | `max_correlation_depth` | int (≥1) | 8 | **Re-ingress loop cap** (ADR 0013 Increment 2). When a captured reply is re-ingressed (`reingress_to=`/`Loopback()`), the re-ingressed message carries a `correlation_depth`; a message at this depth still routes, but the next hop (depth+1) **dead-letters** its re-ingress work-row and marks the origin `ERROR`. Coarse by design — it bounds *total work*, not topology, so a chain that legitimately bounces A→B→A a few times needs headroom. 8 is safe for typical request→response→route feeds; raise it for deep correlation chains, lower it to fence a misbehaving loop. (A value of 0 would dead-letter every re-ingress, so the floor is 1.) |
 
+### `[diagnostics]`
+The Corepoint-style **event log** (#46) — a metadata-only record of connection lifecycle / pre-ingress
+failures and the ACK/NAK the engine returns. **Both master switches are on by default and safe to be:**
+they store only non-PHI metadata (connection name, peer IP, a scrubbed reason, the ACK disposition), and
+the AA-ACK *body* is stored only when the store is encrypted (else NULL); a NAK body is never persisted.
+A per-connection `capture_connection_errors` / `capture_ack` flag overrides the matching master switch for
+one connection (see [CONNECTIONS.md](CONNECTIONS.md)).
+
+| Key | Type | Default | Notes |
+|---|---|---|---|
+| `connection_events` | bool | `true` | master switch for the **connection/transport event log**: inbound lifecycle (established/closed) + pre-ingress failures (allowlist/capacity/oversize/peer-reset/framing) + outbound lane transitions (connection_lost/restored). Metadata-only, written off the hot path by a drain task. Per-connection `capture_connection_errors` overrides it. |
+| `response_sent` | bool | `true` | master switch for **"Response Sent"** — the ACK/NAK returned to an inbound sender. Always captures the disposition metadata (`ack_code`/`phase`/`outcome`); the AA body is stored only on an encrypted store, and every NAK body is NULL. Per-connection `capture_ack` overrides it. |
+
+> Retention for the event log has its own short window — `[retention].connection_event_retention_hours`
+> (in **hours**; `0` = inherit `messages_days`).
+
 ### `[egress]`
 Fail-closed **outbound destination allowlist** (WP-11c; ASVS 13.2.4/13.2.5/14.2.3) — bounds where the
 engine may **send** PHI, so a fat-fingered or hostile destination can't exfiltrate it. Each list is
@@ -484,7 +511,8 @@ bytes/SQL leave the box) and retains the would-send payload for parity compariso
 > the message just finalizes `PROCESSED`.
 
 ### `[alerts]`
-Where the delivery pipeline's operational alerts (`connection_stopped`, `queue_buildup`) are
+Where the delivery pipeline's operational alerts (e.g. `connection_stopped`, `queue_buildup`,
+`connection_error`, `message_stall`, `integrity_drift`) are
 delivered. **Both transports are off by default** — with neither configured, events are logged at
 `WARNING` (the `LoggingAlertSink`). A transport turns on when its essentials are present. Payloads
 carry the connection name + queue shape only — **never a message body** (no PHI). Delivery is
@@ -515,7 +543,7 @@ silences an event you didn't name. Matching is pure config (no code/`eval`).
 
 | Key | Type | Default | Notes |
 |---|---|---|---|
-| `event_type` | str | `any` | match this event: `any`, `connection_stopped`, `queue_buildup`, `storage_threshold`, or `cert_expiry` |
+| `event_type` | str | `any` | match this event: `any`, `connection_stopped`, `queue_buildup`, `storage_threshold`, `cert_expiry`, `connection_error`, `message_stall`, or `integrity_drift` |
 | `connection` | str (glob) | `*` | glob over the connection name (e.g. `OB_*`, `IB_ACME_*`) |
 | `min_depth` | int | _unset_ | `queue_buildup` only — match only when pending depth is at/over this |
 | `min_oldest_seconds` | num | _unset_ | `queue_buildup` only — …or the oldest pending message has waited at least this long |
@@ -708,6 +736,21 @@ Optional **dual-control (maker-checker)** approval for high-value actions (ASVS
 | `operations` | list[str] | `["connection_purge", "dead_letter_replay"]` | which operations require approval; each must be a known op key (a typo is refused at startup) |
 | `expiry_hours` | num | 72 | a pending request can no longer be approved after this many hours (`0` = never expires) |
 
+### `[integrity]`
+Startup **self-attestation of the installed engine wheel** ([ADR 0041](adr/0041-load-path-attestation-and-change-attribution.md)
+D3) — a runtime in-place-tamper tripwire (`messagefoundry/integrity.py`). At startup (and on demand) the
+engine hashes every **loaded** first-party `messagefoundry` module file against the installed wheel's
+`*.dist-info/RECORD` baseline; on **drift** (a loaded module no longer matching its RECORD hash) it records
+a hash-chained `startup_integrity` audit row and fires the `AlertSink`. It complements ADR 0036 (which guards
+the *config dir*) by covering the installed *site-packages* an admin with venv-write + restart rights could
+edit in place. Both keys default **safe**: attestation is on but **alert-only** (it never blocks startup), and
+an **editable** install (`pip install -e .` — no RECORD baseline) is a **no-op**, so dev is never bricked.
+
+| Key | Type | Default | Notes |
+|---|---|---|---|
+| `enabled` | bool | `true` | run startup attestation at all. On by default (alert-only is harmless); a **no-op** off an editable install. Set `false` only to suppress the check entirely (e.g. an unusual packaging where RECORD is known-stale) — you then lose the in-place-tamper tripwire. |
+| `fail_closed_on_drift` | bool | `false` | when `true`, drift makes `serve` **refuse to start** (after recording the audit row + alerting). Default `false` = **alert-only**: a legitimate reviewed in-place security hotfix (the documented vendored-parser patch contingency) would itself trip a RECORD mismatch, so fail-closed-by-default would brick a legitimate patch. Opt in for hard enforcement on a locked-down instance. |
+
 ### `[engine]`
 | Key | Type | Default | Notes |
 |---|---|---|---|

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/docs/CONNECTIONS.md

@@ -173,7 +173,8 @@ collision.
 | `tls_check_hostname` | out | `true` | require the server cert to match `host` (SNI + hostname check). |
 
 Plus on `inbound(...)`: `ack_mode` (`original`/`enhanced`/`none`), `strict`, `hl7_version`. On
-`outbound(...)`: `retry` (`RetryPolicy`), `ordering`, `internal_error`, `buildup`, and `simulate`
+`outbound(...)`: `retry` (`RetryPolicy`), `ordering`, `internal_error`, `buildup`, `stall`
+(`StallThreshold` — Corepoint "Max Message Stall", #50; off unless set, see below), and `simulate`
 (`bool`, default `false`). `simulate=True` puts the outbound in **shadow / parallel-run mode** (#15): it
 runs the full transform + count-and-log and finalizes the message `PROCESSED`, but **suppresses the real
 egress** (no bytes/SQL leave the box) and retains the would-send payload for parity comparison — so a
@@ -1011,6 +1012,90 @@ outbound("MLLP-OUT_PAYER_ELIG", MLLP(host=env("payer_host"), port=2575, reingres
 # a Handler Sends the eligibility query to MLLP-OUT_PAYER_ELIG; its reply re-ingresses into IB-LOOP_PAYER_ELIG.
 ```
 
+## Per-connection retention, document pruning & diagnostics overrides
+
+A connection may **override** several service-wide `[…]` defaults for just itself. Each is set the same
+two ways as `retry`/`buildup` — **code-first** on `inbound(...)`/`outbound(...)`, **or** as a key in
+`connections.toml` (ADR 0007) — and each defaults to **inherit the global setting** when omitted.
+
+### Retention overrides ([ADR 0027](adr/0027-per-connection-retention.md))
+
+Override the global `[retention]` body-null windows per connection. `None` (omitted) = inherit the global
+window; `0` = keep this connection's bodies **forever**; `>0` = days.
+
+| Key | Dir | Type | Default | Meaning |
+|-----|-----|------|---------|---------|
+| `messages_days` | in | int | inherit `[retention].messages_days` | past N days, null this **inbound's** received message bodies (keyed on the receiving inbound), keeping the metadata row. `0` = keep forever |
+| `dead_letter_days` | out | int | inherit `[retention].dead_letter_days` | past N days, null the bodies of **this outbound's** dead-lettered rows (keyed on the outbound that dead-lettered them). A dead row stays replayable until its body is purged. `0` = keep forever |
+
+### Embedded-document pruning ([ADR 0042](adr/0042-embedded-document-pruning.md), #47)
+
+A separate **inbound** lever that evicts only the bulky base64 **embedded document** (a `mfb64:v1:`
+carriage value / an HL7 `OBX-5` ED embed) **in place** to a small tombstone — keeping the surrounding,
+readable message — distinct from `messages_days`, which nulls the **whole** body.
+
+| Key | Dir | Type | Default | Meaning |
+|-----|-----|------|---------|---------|
+| `prune_documents_after` | in | int | `None` = **never prune** (back-compat) | after N **days**, strip each embedded document for this inbound. Must be `> 0` |
+| `prune_documents_min_bytes` | in | int | `None` = strip **any** size | skip an embed whose decoded size is **below** this byte threshold (keep small embeds, evict only the bulky ones). Setting it **requires** `prune_documents_after` (else a wiring error) |
+
+### Diagnostics / event-log overrides ([ADR 0021](adr/0021-inbound-ack-nak-capture-response-sent.md), #46)
+
+Override the `[diagnostics]` master switches for one connection. **Tri-state:** omitted = inherit the
+matching master switch; `true`/`false` = explicit per-connection override.
+
+| Key | Dir | Type | Default | Meaning |
+|-----|-----|------|---------|---------|
+| `capture_ack` | in | bool | inherit `[diagnostics].response_sent` | record the **"Response Sent"** ACK/NAK metadata for this inbound (the AA body only on an encrypted store; a NAK body is never stored) |
+| `capture_connection_errors` | in | bool | inherit `[diagnostics].connection_events` | record this connection's **lifecycle + pre-ingress failure** events (established/closed, allowlist/capacity/oversize/peer-reset/framing) |
+
+### `stall` — Max Message Stall ([ADR 0014](adr/0014-alerting-rules-engine.md), #50)
+
+An **outbound** override of the `[delivery].stall_max_oldest_seconds` global: raise a `message_stall`
+alert when this lane's **oldest undelivered message** has waited too long.
+
+| Key | Dir | Type | Default | Meaning |
+|-----|-----|------|---------|---------|
+| `stall` | out | `StallThreshold` | inherit `[delivery]` (off unless set) | `StallThreshold(max_oldest_seconds=…)` — `None` keeps the stall alert **off** (it overlaps `buildup`'s age dimension, so it's opt-in to avoid double-paging). In `connections.toml` it is an `[outbound.stall]` table with `max_oldest_seconds` (see the example below) |
+
+```python
+from messagefoundry import MLLP, inbound, outbound
+from messagefoundry.config.models import StallThreshold
+
+# Inbound: keep this feed's bodies only 7 days, and prune embedded documents >256 KiB after 1 day.
+inbound("IB_ACME_RAD", MLLP(port=2576), router="rad_router",
+        messages_days=7, prune_documents_after=1, prune_documents_min_bytes=256 * 1024,
+        capture_ack=True)                         # force-capture the ACK even if the master switch is off
+
+# Outbound: keep this destination's dead-letter bodies 90 days; alert if a message stalls >10 min.
+outbound("OB_PACS_RAD", MLLP(host="pacs", port=11112),
+         dead_letter_days=90, stall=StallThreshold(max_oldest_seconds=600))
+```
+
+```toml
+# connections.toml — the same overrides as data.
+[[inbound]]
+name = "IB_ACME_RAD"
+transport = "mllp"
+router = "rad_router"
+messages_days = 7
+prune_documents_after = 1
+prune_documents_min_bytes = 262144
+capture_ack = true
+  [inbound.settings]
+  port = 2576
+
+[[outbound]]
+name = "OB_PACS_RAD"
+transport = "mllp"
+dead_letter_days = 90
+  [outbound.settings]
+  host = { env = "pacs_host" }
+  port = 11112
+  [outbound.stall]
+  max_oldest_seconds = 600
+```
+
 ## Resource management & limits (ASVS 13.1.2 / 13.1.3 / 13.2.6)
 
 How the engine bounds connections, threads, and retries per external system, and what happens **when a

{messagefoundry-0.2.8 → messagefoundry-0.2.9}/docs/adr/0017-consumer-deployment-model.md

@@ -215,3 +215,42 @@ installs use the pinned wheel. That migration already had to repurpose the built
   the security-posture tier — both Blocker rows must land together.
 - Most of the model already works; the remaining effort is **packaging, documentation, scaffolding, one
   env-name lift, and a few isolation/drift guards** — not a redesign of the config or value layers.
+
+## Amendment (2026-06-27) — non-editable wheel from recommendation to enforced production default
+
+> Append-only amendment from the **insider-code-tampering review** (see
+> [ADR 0041](0041-load-path-attestation-and-change-attribution.md) D3, BACKLOG #54). It **does not rewrite**
+> the accepted three-tier model or the eight ratified decisions above — it **tightens** Ratified Decision 5
+> ("Boundary enforcement → packaging-primary") from a documented *recommendation* into an **enforced
+> production default**.
+
+**What changes.** The original decision said the non-editable wheel is the *recommended* boundary and an
+optional Windows venv ACL-lock is *defense-in-depth*. ADR 0041 D3 adds a **runtime tripwire** that converts
+this from advice into an enforced posture:
+
+- The **non-editable, hash-locked wheel is the production default.** Editable `pip install -e .` is
+  **retired from production deployment docs** ([EARLY-ADOPTER-GUIDE.md](../EARLY-ADOPTER-GUIDE.md),
+  [DEPLOYMENT.md](../DEPLOYMENT.md), SERVICE.md): editable installs are documented **only** for engine
+  co-development, never for a deployed instance. The two-repo consumer model (pinned wheel + org config repo)
+  is now the single documented production install path.
+- Enforcement is **startup self-attestation** (ADR 0041 D3): the engine hashes its loaded `messagefoundry`
+  module files against the wheel's `*.dist-info/RECORD` baseline at boot. On drift the **default is
+  alert-only** (a `startup_integrity` audit row + `AlertSink`); an **opt-in** `[integrity].fail_closed_on_drift`
+  hard-refuses to start. This makes the read-only-engine boundary **detectable at runtime**, not merely
+  enforced by file packaging.
+
+**Implemented as a no-op on editable dev installs.** An editable install has **no `dist-info/RECORD`**
+baseline to attest against, so the tripwire is a **no-op/advisory** there — `pip install -e ../MessageFoundry`
+during co-development (the workflow named in *First instance of the pattern* above) is **byte-for-byte
+unchanged** and never bricked. The enforcement applies precisely where it should: a **deployed, wheel-installed
+production instance**.
+
+**Why the default is alert-only, not fail-closed.** The documented vendored-patch contingency for the dormant
+`python-hl7`/`hl7apy` parsers (a legitimate, reviewed security hotfix applied in place) would itself trip a
+`RECORD` mismatch. A fail-closed-by-default posture would therefore **brick a legitimate security patch** at
+the worst moment. Alert-only-by-default detects-and-attributes the drift while keeping the service up;
+operators who want hard enforcement opt in via `[integrity].fail_closed_on_drift`.
+
+**What this does not change.** The accepted three-tier ownership model, the same-commit-everywhere promotion
+model (Decision 4), the env-name decoupling Blockers, and the AGPL/licensing posture (Decision 6) are all
+unchanged. This amendment narrows **only** the install/boundary-enforcement surface of Decision 5.

messagefoundry-0.2.9/docs/adr/0027-per-connection-retention.md

@@ -0,0 +1,208 @@
+# ADR 0027 — Per-connection retention / pruning windows
+
+- **Status:** Proposed  <!-- Proposed (no code yet) → Accepted (build may start) → Superseded by NNNN / Rejected -->
+- **Date:** 2026-06-27
+- **Related:** BACKLOG #34 · [ADR 0001](0001-staged-pipeline-architecture.md) (staged queue + the
+  count-and-log / at-least-once invariants this purge must preserve) · [ADR 0007](0007-gui-manageable-connections-toml.md)
+  (transport-config-as-data — the per-connection keys are also hand-/GUI-editable) ·
+  [ADR 0021 §7](0021-inbound-ack-nak-capture-response-sent.md) (the `connection_event` log + the **#46
+  per-connection diagnostics-override plumbing this reuses**) · [ADR 0005](0005-transform-accessible-state.md)
+  (transform-state purge — stays global) · [ADR 0017](0017-consumer-deployment-model.md) (org-owned config
+  repo across instances) · [CLAUDE.md](../../CLAUDE.md) §1 (no grouping unit; connections may be data),
+  §2 (count-and-log + reliability), §9 ([PHI.md](../PHI.md) §8 retention) ·
+  [`config/settings.py`](../../messagefoundry/config/settings.py) `RetentionSettings`/`DiagnosticsSettings` ·
+  [`config/wiring.py`](../../messagefoundry/config/wiring.py) `InboundConnection`/`OutboundConnection` ·
+  [`pipeline/retention.py`](../../messagefoundry/pipeline/retention.py) `RetentionRunner.run_once` ·
+  [`store/store.py`](../../messagefoundry/store/store.py) `purge_message_bodies`/`purge_dead_letters`
+
+---
+
+## Context
+
+Data retention today is a **single, store-wide policy**. The `[retention]` service-settings section
+([`config/settings.py`](../../messagefoundry/config/settings.py) `RetentionSettings`) is enforced by
+**one** global [`RetentionRunner`](../../messagefoundry/pipeline/retention.py) per process. Its windows
+(`messages_days`, `dead_letter_days`) drive the store purge methods — `purge_message_bodies` nulls inbound
+message bodies, `purge_dead_letters` nulls dead-lettered outbound bodies — and each takes a **single
+`older_than` cutoff and purges store-wide by message age only**. The inbound `InboundConnection` and
+outbound `OutboundConnection` ([`config/wiring.py`](../../messagefoundry/config/wiring.py)) carry no
+retention field. So every feed shares one window: an operator cannot keep ADT for 90 days while pruning a
+high-volume / low-value lab feed at 7, or null bodies sooner for one chatty connection to bound its PHI
+footprint feed-by-feed. Mirth, by contrast, sets **message storage + pruning per channel** — the standard
+operator lever for bounding PHI footprint feed-by-feed.
+
+The mechanism to layer a per-connection override over a global default **already exists and was just
+extended** for the #46 Corepoint-style event log ([ADR 0021 §7](0021-inbound-ack-nak-capture-response-sent.md),
+PR #541): `InboundConnection.capture_ack` / `capture_connection_errors` are `bool | None` where
+**`None` = inherit the matching `[diagnostics]` master switch** and an explicit value overrides it for one
+connection — the same shape as `OutboundConnection.retry`/`ordering`/`buildup`
+(`RetryPolicy`/`OrderingMode`/`BuildupThreshold`) and the inbound FIFO binding. The `inbound()`/`outbound()`
+factories thread the field through `build_inbound_connection`/`build_outbound_connection`, and the ADR 0007
+`connections.toml` loader desugars the same key through those same factories. Critically,
+`retention.py run_once` **already interleaves** body-purge **and** the #46 `connection_event` purge in **one
+pass** (the RETENTION hook). This ADR extends that **same** pass and **reuses that same override plumbing**
+rather than inventing a parallel one.
+
+Two [CLAUDE.md](../../CLAUDE.md) invariants bound the design and **must not** be relaxed:
+
+- **Count-and-log** (§2): "**every received message is persisted before the ACK** … nothing is
+  accepted-and-dropped." A purge therefore **NULLs the PHI *body* while keeping the metadata row** (counts,
+  disposition, audit intact — the Mirth Data-Pruner pattern, `purge_message_bodies`' existing contract); it
+  never deletes a `messages` row.
+- **Reliability / at-least-once** (§2, ADR 0001): a body still in flight is never purged
+  (`purge_message_bodies` already guards `NOT EXISTS (… queue.status IN pending/inflight)`); a per-connection
+  cutoff only changes *which age* is eligible, never that guard.
+- **No grouping unit / connections may be data** (§1): "*Routers/Handlers (logic) stay code-first*, but a
+  Connection's *transport config* … may live in an optional **`connections.toml`**." Retention is transport
+  *config* for a connection, so the override rides the connection spec + `connections.toml`, **not** a
+  Router/Handler API and **not** a built "channel" object.
+
+## Decision
+
+**Add a per-connection retention override layered over the global `[retention]` default, authored on the
+inbound/outbound connection (code-first **and** `connections.toml`), and thread a per-connection cutoff into
+the existing purge SQL** — reusing the #46 override pattern, not a new one.
+
+### D1 — Override field on the connection spec (= #46 / FIFO / RetryPolicy / BuildupThreshold)
+
+Add `messages_days: int | None` to `InboundConnection` and `dead_letter_days: int | None` to
+`OutboundConnection` (and their `inbound()` / `outbound()` factories + `build_inbound_connection` /
+`build_outbound_connection` cores), where **`None` = inherit the global `[retention]` window** and an
+explicit value overrides it for that one connection (`0` = keep-forever, `>0` = days). Authored code-first
+(`inbound(..., messages_days=90)`) **and** as a `connections.toml` key (ADR 0007 desugars through the same
+factory), so it stays hand- and GUI-editable. This is the **identical** override idiom #46 just shipped for
+`capture_ack`/`capture_connection_errors` — no new settings section, no new resolver.
+
+- `messages_days` rides the **inbound** because `purge_message_bodies` is keyed by the receiving connection
+  (`messages.channel_id` = inbound name). `dead_letter_days` rides the **outbound** because
+  `purge_dead_letters` is keyed by the connection that dead-lettered the row (`queue.destination_name` =
+  outbound name).
+
+### D2 — Thread a per-connection cutoff through the purge SQL on all three backends
+
+The `RetentionRunner` resolves, per pass, a **`{connection_name → older_than}` map** (per-connection override
+→ global `[retention]` default → keep-forever) and passes it to the purge methods alongside the global
+cutoff. `purge_message_bodies` (keyed on `messages.channel_id`) / `purge_dead_letters` (keyed on
+`queue.destination_name`) gain an **optional per-connection-cutoff argument** (default empty ⇒ byte-identical
+to today's single global cutoff): a connection in the map is purged at its own cutoff; any connection absent
+from the map falls back to the global window. The per-connection cutoff predicate is **AND-ed** with the
+existing *never-purge-an-in-flight-body* predicate (`NOT EXISTS (pending/inflight)`), so a purge can never
+evict a body still being routed/transformed/delivered. This must land on **all three** store backends
+(**SQLite / Postgres / SQL Server**) with a purge-SQL parity test. Lane A is the **sole store-writer** —
+coordinate land-order with the pool-prewarm store refactor so the purge-SQL change rebases cleanly onto the
+shared read pool.
+
+### D3 — One audit row per pass, recording per-connection cutoffs + counts
+
+`RetentionRunner.run_once` still emits **exactly one** `retention_purge` `audit_log` entry per pass that did
+work, now recording the **per-connection cutoffs + per-connection purged counts** (connection name + days +
+count — **never** message content, no PHI) alongside the existing global fields. This is the same
+count-and-log discipline the global runner already follows, and the same single pass that already interleaves
+the #46 `connection_event` purge.
+
+### What this must not break
+
+- **Count-and-log / Mirth Data-Pruner.** Still null-body-keep-metadata — never delete a `messages` row;
+  counts, disposition, and the audit trail stay intact. A per-connection cutoff changes *when* a body is
+  nulled, never *that the row survives*.
+- **At-least-once / reliability (ADR 0001).** The `NOT EXISTS (pending/inflight)` in-flight guard is
+  AND-ed in unchanged — a message mid-pipeline is never purged regardless of its connection's window.
+- **No grouping unit / code-first logic.** The override is connection transport-config (spec field +
+  `connections.toml`); Router/Handler *logic* is untouched. No declarative "channel" element.
+- **Global-only deployments.** Every new field defaults to `None` (inherit); a config with no per-connection
+  override resolves to exactly one global cutoff and the purge SQL is byte-identical to today.
+- **Stays global:** `audit_days` (keep-forever — tamper-evident hash chain, ~6-yr HIPAA expectation),
+  `state_max_age_days` (per-*namespace*, not per-connection — a separate follow-up),
+  `connection_event_retention_hours` (#46; already its own window), and `max_db_mb` / WAL / VACUUM (they
+  govern the one store *file*, not a feed).
+
+## Acceptance Criteria
+
+> EARS form; each linked (`→`) to its test/fixture. `messagefoundry adr-analyze` checks each `→` resolves.
+
+- **AC-1** — WHERE an inbound connection sets `messages_days`, WHEN `run_once` purges message bodies, THE
+  SYSTEM SHALL apply that connection's cutoff to rows it received (`messages.channel_id`) and the global
+  `[retention].messages_days` cutoff to every connection without an override.
+  → `tests/test_per_connection_retention.py::test_override_applies_per_inbound`
+- **AC-2** — WHERE an outbound connection sets `dead_letter_days`, WHEN `run_once` purges dead-letters, THE
+  SYSTEM SHALL key the cutoff off the **outbound** that dead-lettered each row (`queue.destination_name`).
+  → `tests/test_per_connection_retention.py::test_dead_letter_override_keys_off_outbound`
+- **AC-3** — WHEN no per-connection override is configured, THE SYSTEM SHALL purge exactly as the global
+  store-wide policy does today (a single global cutoff; byte-identical behaviour).
+  → `tests/test_per_connection_retention.py::test_global_only_unchanged`
+- **AC-4** — IF a message body is still pending/inflight, THEN THE SYSTEM SHALL NOT purge it even when its
+  per-connection cutoff has elapsed (the cutoff AND-s with the in-flight guard).
+  → `tests/test_per_connection_retention.py::test_in_flight_body_never_purged`
+- **AC-5** — WHEN a per-connection override is set to `0`, THE SYSTEM SHALL keep that connection's bodies
+  forever even while the global window prunes others (per-feed opt-out).
+  → `tests/test_per_connection_retention.py::test_per_connection_zero_keeps_forever`
+- **AC-6** — WHEN a pass purges across connections, THE SYSTEM SHALL write **exactly one** `retention_purge`
+  audit row recording the per-connection cutoffs + counts and **no message content**.
+  → `tests/test_per_connection_retention.py::test_audit_records_per_connection_cutoffs`
+- **AC-7** — THE SYSTEM SHALL accept `messages_days` / `dead_letter_days` from both a code-first
+  `inbound()`/`outbound()` and a `connections.toml` entry, resolving to identical `InboundConnection` /
+  `OutboundConnection` overrides.
+  → `tests/test_connections_file.py::test_retention_override_roundtrips_toml`
+- **AC-8** — THE SYSTEM SHALL apply the per-connection cutoff identically on SQLite, Postgres, and SQL
+  Server (purge-SQL parity).
+  → `tests/test_per_connection_retention.py::test_three_backend_parity`
+
+## Options considered
+
+1. **Per-connection override field on the inbound/outbound spec, reusing the #46 override plumbing —
+   CHOSEN.** Adds `messages_days`/`dead_letter_days` (`None` = inherit) to the connection spec exactly as
+   `capture_ack`/`retry`/`buildup` already do, threaded through the same `build_*` factories + the same ADR
+   0007 `connections.toml` desugaring, and threads a `{connection → cutoff}` map into the existing purge
+   methods + the existing single-pass / single-audit `run_once`. Minimal new surface; one override idiom; the
+   keys are already GUI/TOML-editable. Matches BACKLOG #34's stated scope.
+2. **A new `[retention.connections.<name>]` settings overlay (a parallel resolver).** Rejected: invents a
+   *second* override idiom parallel to FIFO/RetryPolicy/BuildupThreshold/#46-diagnostics, splits a
+   connection's config across two surfaces, and re-derives a connection→cutoff resolver the #46 work already
+   provides. The override belongs *on the connection*, where every other per-connection knob lives — the
+   brief's explicit "reuse #46 plumbing, don't invent a parallel one."
+3. **A `retention` field on the lower-level `Source`/`Destination` model** ([config/models.py](../../messagefoundry/config/models.py)).
+   Rejected: those carry transport *type + settings*; the per-connection lifecycle knobs (router binding,
+   FIFO, retry, #46 diagnostics) all live on the higher-level `InboundConnection`/`OutboundConnection` spec —
+   retention rides there for consistency.
+4. **A second `RetentionRunner` per connection.** Rejected: retention is a leader-only WRITE singleton
+   (PHI-body nulling + audit) that must run on exactly one node and write **one** audit row per pass; N
+   runners multiply the audit rows, the leader-election surface, and WAL/VACUUM contention. One runner with a
+   per-connection cutoff map is strictly simpler.
+5. **Status quo (store-wide only).** Rejected: forces a clinically-important feed onto a noisy feed's window
+   and blocks feed-by-feed PHI minimization — the explicit owner ask and the Mirth per-channel gap.
+
+## Consequences
+
+**Positive** — Operators get the Mirth per-channel lever: keep ADT 90 days, prune a high-volume lab feed at
+7, opt one feed out entirely — bounding PHI footprint feed-by-feed. It reuses one override idiom (no new
+mental model, no new settings section, no new resolver), extends the existing single pass + single audit row,
+and is purely additive (every default `None` ⇒ today's behaviour).
+
+**Negative / risks** — The purge SQL gains a per-connection branch on three backends — a small parity surface
+that must stay in lock-step (covered by AC-8 + the parity test). The per-connection cutoff map is resolved
+each pass from the live registry, so a reload that changes an override takes effect on the next pass (not
+retroactively un-purging already-nulled bodies — the intended one-way trade-off). Per-connection counts in
+the audit detail grow the row with the number of connections that purged in a pass (still metadata-only).
+Mis-keying inbound (`channel_id`) vs outbound (`destination_name`) would purge the wrong rows — the AC-1/AC-2
+split pins it.
+
+**Out of scope / stays global** — `audit_days` (keep-forever; tamper-evident chain, HIPAA ~6 yr),
+`state_max_age_days` (per-*namespace* follow-up, not per-connection), `connection_event_retention_hours`
+(#46; already its own window), `max_db_mb` / `wal_checkpoint_seconds` / `vacuum_at` (govern the one store
+file, not a feed). Deleting metadata rows (vs nulling bodies) stays **declined** — count-and-log keeps the
+row forever.
+
+## To resolve on acceptance
+
+- [ ] **Map vs join.** Resolve the per-connection cutoff as an in-Python `{connection → older_than}` map
+  passed to the purge call, **or** push a `(connection, cutoff)` temp/VALUES join into the purge SQL? (Map is
+  simpler and backend-uniform for a handful of overrides; a join scales to many connections. Decide before
+  the Postgres / SQL Server parity build.)
+- [ ] **Land-order with the store refactor.** Confirm the purge-SQL change rebases onto the pool-prewarm /
+  shared-read-pool store refactor (Lane A is sole store-writer) — coordinate the merge order so neither
+  clobbers the other's `store.py` purge methods.
+- [ ] **Leader/audit semantics under sharding.** Confirm the leader-only `run_once` (one owner per process
+  today; NullCoordinator single-node) stays correct for per-connection cutoffs under ADR 0037 multi-process
+  sharding, where inbounds are partitioned across shards but outbound/logic are shared.
+- [ ] **GUI surfacing.** Confirm the `connections.toml` GUI editor (ADR 0007) renders the two new keys
+  (inbound `messages_days`, outbound `dead_letter_days`) — or defer to a follow-up GUI task.