messagefoundry 0.2.12__tar.gz → 0.2.13__tar.gz

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/.github/workflows/benchmark.yml

@@ -84,7 +84,7 @@ jobs:
       MEFOR_STORE_DATABASE: messagefoundry
       MEFOR_STORE_USERNAME: postgres
       MEFOR_STORE_PASSWORD: mefor
-      MEFOR_STORE_POOL_SIZE: "5"
+      MEFOR_STORE_POOL_SIZE: "40"  # track the shipped default (ADR 0062: 5 -> 40)
       MEFOR_STORE_ENCRYPT: "false"  # plaintext to the container; the TLS-hardening guard needs the escape
       MEFOR_ALLOW_INSECURE_TLS: "1"
     steps:
@@ -151,7 +151,7 @@ jobs:
       MEFOR_STORE_AUTH: sql
       MEFOR_STORE_USERNAME: sa
       MEFOR_STORE_PASSWORD: "Str0ng_P@ssw0rd!"
-      MEFOR_STORE_POOL_SIZE: "5"
+      MEFOR_STORE_POOL_SIZE: "40"  # track the shipped default (ADR 0062: 5 -> 40)
       MEFOR_STORE_TRUST_SERVER_CERTIFICATE: "true"
       MEFOR_ALLOW_INSECURE_TLS: "1"
     steps:

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/.github/workflows/ci.yml

@@ -569,7 +569,7 @@ jobs:
           # FORCE a below-default pool (4 connections) so the perf_counter-measured acquire-WAIT
           # histogram (the PRIMARY pool-wait signal) is non-trivial even at small N: ~3N inbound workers
           # contend for 4 connections, so acquires actually queue and the percentiles populate. The
-          # default pool (5+) would mask the wall. 4 (not 1-2) is deliberate: GET /status runs
+          # default pool (40) would mask the wall. 4 (not 1-2) is deliberate: GET /status runs
           # db_status()'s four sequential COUNT(*) queries on every poll, and an extremely tiny pool
           # starves those acquires past the poller's HTTP timeout under the empty-claim herd on a slow
           # runner — every poll then fails and the drain loops burn the test budget (the PR #675 hang).

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/CHANGELOG.md

@@ -6,6 +6,46 @@ All notable changes to MessageFoundry are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.13] — 2026-07-01 — Early Access
+
+The **store connection-scale sizing** wave — right-size the server-DB connection pool to the measured
+inverted-U optimum, guard against over-provisioning, and guarantee the message store stays unified. All
+changes are **server-DB-only**; the single-node SQLite default is unaffected.
+
+### Added
+- **Soft store-pool over-provisioning warning** ([ADR 0062](docs/adr/0062-default-store-pool-size.md)) — a
+  server-DB engine now logs an advisory `WARNING` at graph start if `[store].pool_size` is sized past the
+  connection-pool inverted-U optimum: at/beyond the ~80 catastrophic cliff, or oversized for the engine's
+  inbound-interface count (`~2.5 ×` interfaces). Advisory only — it never blocks startup; SQLite has no pool
+  so it is skipped, and the default (40) never trips it. Guards the "set a huge pool for 1500 connections"
+  footgun (which is a *sharding* problem, not a pool one).
+
+### Changed
+- **Default server-DB store connection pool size raised 5 → 40** ([`[store].pool_size`](docs/CONFIGURATION.md),
+  env `MEFOR_STORE_POOL_SIZE`; [ADR 0062](docs/adr/0062-default-store-pool-size.md)). A three-sweep
+  connection-scale study found the pool is an **inverted-U**: it helps up to ~40 per engine, and
+  **over-provisioning is catastrophic** — past ~40 the extra connections thrash one shared SQL instance
+  (WRITELOG serialization + per-message finalizer applocks), and ACK latency explodes 30–90×. 40 is the
+  measured optimum — **do not set it higher to chase connection count.** **Server-DB backends only** (Postgres
+  / SQL Server) — the default **single-node SQLite** backend is unaffected (fixed read pool + single writer;
+  never reads `pool_size`). **Existing explicit `[store].pool_size` / `MEFOR_STORE_POOL_SIZE` values are
+  unchanged** — only the unset default moves. Behavioral deltas on server-DB engines: ~**8×** the steady-state
+  DB sessions per engine, and the startup pool pre-warm rises from ~2 to **~20 connections per engine**
+  (bounded by `warm_pool_timeout`, off the intake path, self-releasing, never raises). **Connection-budget
+  caution:** `pool_size` is **per engine**, so on a shared server DB `engines × pool_size` all count against
+  one `max_connections` (Postgres default ~100 → ~2 engines at 40) — raise `max_connections`, front the DB
+  with a pooler (PgBouncer), or use SQL Server; or size `pool_size` down. **Never split the store** to fit the
+  budget ([ADR 0063](docs/adr/0063-no-split-store-unified-store-for-sharding.md)). See
+  [`docs/DEPLOY-SERVER-DB.md`](docs/DEPLOY-SERVER-DB.md) §3.
+- **No split data store: multi-shard engine sharding now requires a server DB** ([ADR 0063](docs/adr/0063-no-split-store-unified-store-for-sharding.md),
+  amends [ADR 0037](docs/adr/0037-multi-process-sharding-l3.md)). `messagefoundry supervise` with **more than
+  one shard** on a **SQLite** store is now **refused at startup** — the old SQLite-file-per-shard behavior
+  split the message store into one database per shard, fragmenting search/reporting/audit/replay. A sharded
+  deployment must share **one unified store**, so `>1` shard requires `[store].backend = 'postgres'` or
+  `'sqlserver'` (every shard connects to the same database). **A single un-sharded engine on SQLite is
+  unaffected** (byte-identical to `serve`). Migrating an existing SQLite-sharded deployment: drain each shard
+  store to empty, then re-point `supervise` at one server DB (not an offline store merge).
+
 ## [0.2.12] — 2026-07-01 — Early Access
 
 The **throughput & connection-scale wave.** The staged-queue per-message commit chain is shortened

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messagefoundry
-Version: 0.2.12
+Version: 0.2.13
 Summary: Open-source healthcare integration engine — route, transform, and validate messages across many formats and connection types
 Project-URL: Homepage, https://messagefoundry.org/
 Project-URL: Documentation, https://messagefoundry.org/

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docker/compose.yaml

@@ -49,7 +49,7 @@ x-ha-engine-env: &ha-engine-env
   MEFOR_STORE_PORT: "5432"
   MEFOR_STORE_DATABASE: "messagefoundry"
   MEFOR_STORE_USERNAME: "mefor"
-  MEFOR_STORE_POOL_SIZE: "5"          # >= 2 required under [cluster]; >= 3 recommended
+  MEFOR_STORE_POOL_SIZE: "40"         # default (ADR 0062); >= 2 required under [cluster]
   MEFOR_CLUSTER_ENABLED: "true"
   # DEV-ONLY: the local Postgres has no CA-trusted cert, so verification can't succeed here. This
   # disables TLS verification across ALL transports — it is a CI/lab switch and must NEVER be set in

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/BACKLOG.md

@@ -426,6 +426,14 @@ client. "Moderate risk"; not verifiable in offscreen Qt tests, so it deserves a
 **FIFO per outbound connection** (simple, safe). Per-key ordering is the leading-edge refinement to
 revisit once FIFO is solid and a real workload needs the parallelism.
 
+**Naming (locked 2026-06-30):** the canonical vocabulary for this concept is **sequence key** (the
+per-message partition value — ADT = `facility + MRN`; same key → same lane → FIFO preserved,
+different keys → parallel), **sequence group** (the set of messages sharing a sequence key), and
+**sequence-keyed lanes** / **sequence sharding** (the mechanism). The older terms used elsewhere in
+this item — `partition_key` (config setting) and "order-group sharding" — refer to the same thing.
+"Order-group" was retired because "order" collides with clinical orders (CPOE/ORM); "partition key"
+was retired because "partition" is reserved for the planned store-sharding-by-channel axis.
+
 **What:** preserve order only *within* a partition key (e.g. MRN / encounter / sending facility)
 while processing *across* keys in parallel — the sweet spot between strict per-connection FIFO (safe,
 serial) and unordered parallelism (fast, unsafe). Same idiom as a broker partition key. Recent
@@ -3063,11 +3071,11 @@ parity analysis.
 
 **Cluster:** Minor gaps. **Priority:** P3. **Verdict:** demand-gate.
 
-**Scope:** Per-connection send pacing, verify the reply's MSA-2 == the sent MSH-10, and keep-alive / persistent-reconnect on MLLP/TCP outbounds.
+**Scope:** Per-connection send pacing and verify the reply's MSA-2 == the sent MSH-10. *(Keep-alive / persistent-reconnect on MLLP/TCP outbounds was promoted out to its own tracked item — see [#97](#97-keep-alive--persistent-outbound-connections--per-connector-setting-p3-on-trigger).)*
 
-**Trigger:** build when a partner needs paced sending, strict response-correlation, or a persistent connection.
+**Trigger:** build when a partner needs paced sending or strict response-correlation.
 
-**Why:** All confirmed (no pacing/keepalive; `MLLPDestination._check_ack` reads MSA-1/MSA-3 only, never matches MSA-2↔MSH-10; `mllp.py` notes persistent/pooled "can come later"). Cheap per-connection-override additions; bundle on real partner need.
+**Why:** Both confirmed (no pacing; `MLLPDestination._check_ack` reads MSA-1/MSA-3 only, never matches MSA-2↔MSH-10). Cheap per-connection-override additions; bundle on real partner need. Keep-alive is tracked separately at #97.
 
 **Source:** promoted from [#52](#52-corepoint-capability-parity-gaps--prioritized-roadmap-input-2026-06-27) gap synthesis (adversarially reviewed, 2026-06-28).
 
@@ -3559,6 +3567,81 @@ reconciled the same day.
 
 ---
 
+## 96. Built-in "setup tester" — self-service capacity estimator that benchmarks the deployed setup and reports how much traffic it can handle (P2, adopter-facing)
+
+**Type:** feature — an operator/adopter-facing **capacity self-test** shipped *with the engine*. It runs the
+same style of measurement we do for throughput testing, but as a first-class, on-demand command an adopter
+points at **their own** setup (this box, this store backend, this config) to get back an **estimate of how
+much traffic that setup can sustain**.
+
+**What:** a `messagefoundry` subcommand (e.g. `messagefoundry capacity` / `setup-test`) that drives a
+controlled synthetic load through the real engine and reports an **estimated sustainable throughput** — a
+headline **msg/s** and **msg/day** figure, ideally **per-inbound-interface** *and* engine-wide, plus the
+**limiting factor** (commit-bound / pool-saturated / CPU / disk) and a confidence caveat. It ramps to the
+saturation knee (where `in_pipeline`/`backlog_seconds` start rising faster than drain — the #93 signal) and
+reports the last rate that drained cleanly with no loss, rather than a raw peak. Reuses the **BUILT load
+harness** measurement machinery ([`harness/load/`](../harness/load/), [`docs/LOAD-TESTING.md`](LOAD-TESTING.md))
+— the rate governor, the fast correlation sink (true end-to-end latency), the drain gauge (`in_pipeline`), and
+the no-loss reconciliation — packaged as a supported engine capability rather than a dev-only tool.
+
+**Distinct from what already exists (don't duplicate):**
+- **#28 / #29 (DONE)** are the *developer/benchmark* runs of the harness against a synthetic high-fan-out
+  system-under-test, producing the **project** [`benchmarks/TUNING-BASELINE.md`](benchmarks/TUNING-BASELINE.md)
+  baseline. This item is the *adopter-run* inverse: point it at **the real deployed config on the real box** and
+  get a sizing number for *that* deployment — not a project baseline, and not something that needs the harness's
+  synthetic SUT config or the denylist-guarded estate profiles.
+- **#93 (P2)** is the *passive, runtime* counterpart — it watches real traffic and **warns** when live load is
+  approaching capacity. This item is the *active, pre-cutover* counterpart — it **measures** where that capacity
+  is in the first place, so #93's overload threshold can be calibrated against it. They pair.
+- **#40** is the enterprise-hardware CI leg; this tester is what an adopter would run **on their own hardware**
+  to reproduce a sizing number without CI access.
+
+**Design constraints (for the eventual ADR):**
+- **Must not pollute production.** A capacity run generates real store writes and would otherwise inflate the
+  true inbound counts (the count-and-log invariant persists *every* received message). It must run against an
+  **isolated/ephemeral store** (temp DB) or a clearly-marked test namespace, and never leave synthetic rows in,
+  or skew the metrics of, the live message store.
+- **Synthetic payloads only — never real PHI.** Drive it from the conformant generators
+  ([`generators/`](../messagefoundry/generators/)) / the anon framework (ADR 0030), consistent with the
+  dryrun/generate PHI rule (never against real PHI, never redirected to a committed file/CI log).
+- **Respect the per-interface bound.** Report capacity **per inbound interface** and note that a single strictly-
+  ordered feed is core-bound (owner principle: fan out feeds at source, not infinite single-feed speed) — an
+  engine-wide total is the sum across interfaces, not a single-feed number. Sequence-keyed lanes (#3) are the
+  sanctioned single-feed escape hatch when one feed outgrows a core.
+- **Name the limiting factor**, reusing the #93/#64 signals (commit/write latency, `[store].pool_size`
+  busy/wait, CPU/mem via #74, `in_pipeline` growth) so the output is *"~N msg/s, commit-bound"* rather than a
+  bare number — the sizing memos already conclude the pipeline is serial-per-inbound commit-bound, not
+  disk/CPU/pool-bound on the tested boxes.
+
+**Why P2 / on-trigger.** Turns capacity sizing — today a manual "run the dev harness + read TUNING-BASELINE by
+hand" exercise — into a **supported operation** an adopter can self-serve before a cutover (*"will this box carry
+our ~1.6M ADT/day?"*). The measurement machinery already exists; the net-new is the operator-facing command, the
+isolated-store harness, the ramp-to-knee estimator, and the capacity report. **Trigger:** a pilot/adopter needing
+a self-service pre-cutover capacity check on their own hardware (the ADR 0017 consumer-deployment pattern), or the
+#93 overload-alert threshold needing a per-deployment capacity baseline to calibrate against. Relates to
+**#28**/**#29** (the harness it wraps), **#40** (enterprise-box runs), **#64** (the throughput-performance
+roadmap), **#93** (the runtime overload-alert counterpart), and the recorded sizing work (throughput matrix /
+per-interface bound / commit-bottleneck analyses).
+
+**Source:** owner request (2026-06-30) — "add a setup tester to the engine … do tests like we're doing for
+throughput testing and report back an estimate of how much traffic the setup can handle."
+
+---
+
+## 97. Keep-alive / persistent outbound connections — per-connector setting (P3, on-trigger)
+
+> **On-trigger / demand-gate.** Numbered for tracking only — build when the trigger below fires (“demand-gate, don’t schedule”). Promotes the keep-alive sub-scope of [#82](#82-sender-transport-polish-bundle--pacing--msa-2msh-10-matching--tcp-keep-alive) into its own tracked item at the owner's request.
+
+**Type:** feature — a per-outbound-connection option to **hold the TCP link open across deliveries** (keep-alive / persistent) instead of the current connect-per-message behavior.
+
+**What:** an opt-in **per-connector setting** (e.g. `keepalive = true` / a `connection_mode = "persistent" | "on_demand"` knob in the outbound's `settings`, default `on_demand` so existing configs stay byte-identical) on the MLLP / raw-TCP / X12 outbound connectors. When enabled, the delivery worker reuses one open connection (reconnecting on drop/idle), rather than opening + closing a fresh socket every message as it does today. Wants: a bounded idle-close / max-lifetime, reconnect-with-backoff on a dropped link, and clean teardown on `stop()`/reload — all per-connection, with the setting validated at build (dry-run / `check`), consistent with the other outbound knobs.
+
+**Why:** confirmed gap — every TCP-family outbound opens a **fresh connection per delivery** today and there is no toggle: `MLLPDestination` ([`transports/mllp.py`](../messagefoundry/transports/mllp.py), *"Phase 1 opens a fresh connection per delivery … a persistent/pooled connection can come later"*) and `TcpDestination` ([`transports/tcp.py`](../messagefoundry/transports/tcp.py), *"Opens a fresh connection per delivery … pooling can come later"*); listed as an unbuilt MLLP feature gap in [`CONNECTIONS.md`](CONNECTIONS.md) ("keep-connection-open/pooling"). Inbound listeners are already persistent (peer-driven, idle-bounded by `receive_timeout`) — this closes the outbound half. The connect-per-message default is simple and robust to flaky peers, so this is genuinely additive and stays **off by default**; the at-least-once / idempotent-receiver contract is unchanged (a reused link that drops mid-ACK still retries, same as today). **Trigger:** a partner that needs a held-open link (a persistent-session receiver, or a high-rate feed where per-message connect setup is measurable overhead). Relates to **#82** (the sender-polish bundle this splits from — pacing + MSA-2↔MSH-10 matching stay there), **#46** (connection lifecycle events would gain reconnect/retry signals), and **#65** (outbound-connector option surface).
+
+**Source:** owner request (2026-06-30) — "add keepalive feature for outbound connections, controlled by a setting per outbound connector."
+
+---
+
 *Everything else from the 2026-06-10 full-codebase review (1 critical, 13 high, 33 medium, 31 low —
 78 findings) has been remediated; see the review report's §6 action order. The two items it still
 sourced — **#1 (SQL Server concurrency)** and **#2 (console off-thread)** — are now both **DONE**

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/CLOUD-DEPLOYMENT.md

@@ -77,7 +77,7 @@ is verified against [`config/settings.py`](../messagefoundry/config/settings.py)
 | `MEFOR_STORE_DATABASE` | `[store].database` | `messagefoundry` | the database name |
 | `MEFOR_STORE_USERNAME` | `[store].username` | `mefor` | the login |
 | `MEFOR_STORE_PASSWORD` | `[store].password` | *(secret)* | **env/secret only** — never the config file |
-| `MEFOR_STORE_POOL_SIZE` | `[store].pool_size` | `5` | `>= 2` **required** under `[cluster]`, `>= 3` recommended |
+| `MEFOR_STORE_POOL_SIZE` | `[store].pool_size` | `40` | default 40, the inverted-U optimum — don't set higher ([ADR 0062](adr/0062-default-store-pool-size.md)); `>= 2` under `[cluster]`. **Budget:** `engines × pool_size` hit one managed-PG `max_connections` (~2 engines at 40/100) — raise it, add a pooler, or size down; never split the store ([DEPLOY-SERVER-DB.md](DEPLOY-SERVER-DB.md) §3) |
 | `MEFOR_CLUSTER_ENABLED` | `[cluster].enabled` | `true` | turns on the leader lease + graph gating |
 | `MEFOR_STORE_REQUIRE_ENCRYPTION` | `[store].require_encryption` | `true` | fail-closed: refuse to start keyless (PHI at rest) |
 | `MEFOR_STORE_ENCRYPTION_KEY` | `[store].encryption_key` | *(secret)* | base64 32 bytes — `messagefoundry gen-key` |

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/CLUSTERING.md

@@ -42,7 +42,7 @@ backend   = "postgres"   # or "sqlserver"
 server    = "db.internal"
 database  = "messagefoundry"
 username  = "mefor"
-pool_size = 5          # >= 2
+pool_size = 40         # default (ADR 0062); >= 2 under [cluster]
 
 [cluster]
 enabled = true

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/CONFIGURATION.md

@@ -66,11 +66,11 @@ backend that consumes the server-DB keys lands incrementally (the settings + val
 | `username` | str | — | server DBs (required when `auth = sql`) |
 | `password` | secret | — | **env only** (`MEFOR_STORE_PASSWORD`) |
 | `encrypt`, `trust_server_certificate` | bool | `true`/`false` | TLS to the DB |
-| `pool_size` | int | 5 | server DBs |
+| `pool_size` | int | 40 | server DBs — **server-DB only** (no-op on SQLite). The inverted-U optimum (raised from 5; do **not** set higher — over-provisioning is catastrophic, [ADR 0062](adr/0062-default-store-pool-size.md)). **Per engine:** `engines × pool_size` share one `max_connections` — see [`DEPLOY-SERVER-DB.md`](DEPLOY-SERVER-DB.md) §3 |
 | `connect_timeout`, `command_timeout` | int (s) | 15 / 30 | server DBs |
 | `warm_pool` | bool | `true` | server DBs — pre-open pooled connections in the background on graph start/promotion so a connection burst (the post-promotion delivery workers, or a cold start) finds them warm instead of paying cold connects (TCP+TLS+login). Best-effort, self-releasing, **no-op on SQLite**. On by default (it touches no commit/correctness seam); set `false` to opt out on a connection-constrained/licensed site. |
 | `warm_pool_timeout` | num (s) | 15 | server DBs — upper bound on the background warm-up; on expiry it logs and continues with a partially warm pool. Must be `> 0`. A **clustered** server-DB node also rejects an **explicit** value `>= [cluster].leader_fence_timeout_seconds` (a warm should finish within the leadership term that started it); the default (15 < the 20 fence) never trips this. |
-| `warm_pool_target` | int | — | server DBs — how many connections to pre-open. Unset (default) = a safe fraction of the pool (`min(pool_size-1, pool_size//2)`), so the warm never pins more than half the pool; an explicit value is clamped to `pool_size-1`. A pool of 1 is never warmed. |
+| `warm_pool_target` | int | — | server DBs — how many connections to pre-open. Unset (default) = a safe fraction of the pool (`min(pool_size-1, pool_size//2)`), so the warm never pins more than half the pool; an explicit value is clamped to `pool_size-1`. A pool of 1 is never warmed. At the default `pool_size = 40` this is `min(39, 20) = 20` pre-opened per server-DB engine at startup. |
 | `db_schema`, `application_name` | str | — / `messagefoundry` | optional (`db_schema` ⇒ env `MEFOR_STORE_DB_SCHEMA`) |
 
 > Selecting `backend = "sqlserver"` validates that `server`/`database` (and `username` when

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/DEPLOY-SERVER-DB.md

@@ -60,12 +60,44 @@ Configure `[store]` in the service settings (full reference: [`CONFIGURATION.md`
 
 ## 3. Pool sizing
 
+The default `[store].pool_size` is **40** (server-DB only; a no-op on the SQLite default, which uses a fixed
+read pool + single writer). Raised from 5 in [ADR 0062](adr/0062-default-store-pool-size.md): the
+connection-scale study found the pool is an **inverted-U** — it helps up to ~40 per engine, and
+**over-provisioning is catastrophic** (past ~40 the extra connections thrash one shared instance — WRITELOG +
+finalizer applocks — and ACK latency explodes 30–90×). 40 is the measured **optimum**, not a floor: **do not
+set it higher to chase connection count** (that path is refuted — scale is *sharding*, not a bigger pool).
+Tunable via `[store].pool_size` / `MEFOR_STORE_POOL_SIZE`; re-measure for a materially different deployment
+shape (transform cost, message size, disk, SQL-box sizing).
+
 - **Single node:** `[store].pool_size ≥ 3` recommended. Each stage handoff is a committed round-trip and
   the per-stage workers (router, transform, per-outbound delivery) run concurrently against the pool — a
   pool of 1 serializes them against intake.
 - **Clustered (active-passive):** `pool_size ≥ 2` is **required** (a cross-section validator refuses a
   smaller pool when `[cluster].enabled`), `≥ 3` recommended — a clustered node also drives the
   membership / lease-renewal maintenance loop against the pool.
+- **Connection-budget ceiling (important on a shared server DB):** `pool_size` is **per engine**. Every
+  Postgres/SQL Server engine that shares the store — multi-process shards (all connect to the *same*
+  database) and the active-passive standby's warm pool — counts against one server limit. So budget
+  **peak connections ≈ engines × `pool_size`** (+ the standby's `warm_pool_target`) and keep it well under
+  the DB server's `max_connections` (**Postgres default ~100**; SQL Server is bounded by sessions/memory).
+  At the default `pool_size = 40`, ~2 co-located engines already reach ~100 — **a config that connected at
+  5 can fail to connect at 40.** Co-locating several server-DB engines (a sharded fan-out)? **Raise
+  `max_connections`, front the DB with a connection pooler (PgBouncer), or use SQL Server** (more sessions);
+  or size `pool_size` **down**. **Do NOT give each shard its own database/server** — that is a split store,
+  which is disallowed ([ADR 0063](adr/0063-no-split-store-unified-store-for-sharding.md)); one unified store
+  scales *vertically* (faster box/disk) + *cheaper-per-message*, never by fragmenting. Also budget the startup
+  warm burst (~20 pre-opened/engine at the default; set `warm_pool_target` or `warm_pool = false` on a
+  connection-/license-constrained site).
+- **Commit-depth at high interface counts (`fifo_claim_batch`):** the pool is not the only per-engine wall —
+  above ~48 inbound interfaces an engine becomes bound by *commits per message* on the shared store (WRITELOG
+  + the per-message finalizer applock), which a bigger pool **cannot** fix. For a **high-interface-count,
+  commit-bound** server-DB engine, set **`[store].fifo_claim_batch = 8`** (range 8–16, [ADR 0058](adr/0058-batch-claim-fifo-prefix.md)):
+  the INGRESS/ROUTED FIFO claim then takes the contiguous due head-prefix in one commit instead of one-per-row,
+  amortizing the claim commit toward `1/N` on backlogged lanes. It **preserves strict per-lane FIFO (#285) +
+  at-least-once**, is **byte-identical at `1` (the default)** and on caught-up (low-backlog) lanes, and never
+  batches the outbound/delivery claim. Size it against worst-case message size (N decrypted bodies are resident
+  per lane between the claim and its N handoffs). This — not the pool — is the *raise-interfaces-per-engine*
+  lever.
 
 ---
 
@@ -173,7 +205,8 @@ update the pin in lockstep — pin the **CA**, not the leaf, to keep rotations m
 - [ ] `[store].encrypt = true` (and **not** `MEFOR_ALLOW_INSECURE_TLS`) for any PHI deployment.
 - [ ] DB CA trusted so `trust_server_certificate = false` validates — Postgres `ssl_root_cert` **or**
       machine-store import; SQL Server **machine store only** (§5). Never `TrustServerCertificate=true`.
-- [ ] `[store].pool_size ≥ 3` (≥ 2 hard-required in cluster mode).
+- [ ] `[store].pool_size` sized (default **40**, server-DB only; ≥ 2 hard-required in cluster mode) **and**
+      the connection budget checked: engines × `pool_size` (+ standby warm) well under the DB `max_connections` (§3).
 - [ ] Bootstrap login can create the schema on first open, **or** the schema is pre-created.
 - [ ] SQL Server: RCSI enabled (auto, or pre-enabled by a DBA).
 - [ ] Source store drained (`in_pipeline → 0`) before cutover — greenfield, no in-place migration.

{messagefoundry-0.2.12 → messagefoundry-0.2.13}/docs/MENTAL-MODEL.md

@@ -422,7 +422,7 @@ The built-in HA model is **active-passive failover** (the Corepoint/Rhapsody mod
 backend = "postgres"   # or "sqlserver"; SQLite cannot cluster
 server = "db.internal"
 database = "messagefoundry"
-pool_size = 5          # >= 2 (>= 3 preferred)
+pool_size = 40         # default (ADR 0062); >= 2 (>= 3 preferred)
 
 [cluster]
 enabled = true

messagefoundry-0.2.13/docs/THROUGHPUT.md

@@ -0,0 +1,304 @@
+# Understanding MessageFoundry Throughput
+
+*A plain-language guide to what determines how many messages MessageFoundry can move — and why a
+single "messages per day" number rarely tells you what you need to know.*
+
+> **Read this first.** Every throughput figure in this document is a **theoretical maximum** measured
+> in a controlled lab under ideal conditions. Your real-world throughput will almost always be **lower**,
+> and it is usually set by **how fast your partner systems accept and acknowledge each message** — not by
+> the engine. MessageFoundry adds only a few milliseconds of its own overhead per message; the rest of the
+> time is spent waiting on round-trips to your database and your downstream systems. Size against *your*
+> systems, not against a headline number.
+
+---
+
+## Executive summary
+
+- **The engine is nearly transparent.** In our reference lab, MessageFoundry adds on the order of
+  **~16 milliseconds** of its own processing per message. For an ordered (strictly in-sequence) feed that
+  works out to roughly **60 messages/second end-to-end** and **~450 messages/second at the point of
+  receipt** — but those are *ceilings under ideal conditions*, not promises for your environment.
+
+- **Your partner systems are the biggest factor.** For ordered delivery, each message is sent, the engine
+  **waits for the partner to accept and acknowledge it**, and only then does the next one go. The partner's
+  round-trip time lands *directly* in the critical path. A partner that takes 50 ms to acknowledge caps a
+  single ordered stream at roughly **15 messages/second** — no matter how fast the engine or database is.
+
+- **"Messages per day" is a weak unit.** Healthcare traffic is **bursty**. In the production ADT feeds
+  we've profiled, the **busiest hour runs about 2.7× the all-day average rate**. You have to size for the
+  *peak hour*, not the daily average — so a raw "X million/day" figure, on its own, tells you very little.
+
+- **Scale by fanning out, not by one giant pipe.** A strictly-ordered interface is a single serial lane
+  with a bounded rate — that's the physics of guaranteed ordering, not a product limitation. Real capacity
+  comes from running **many interfaces** (split feeds at the source: by hospital, facility, service line).
+  One node's message store easily absorbs the combined load of dozens of interfaces.
+
+- **Four questions to ask of *any* throughput claim** — ours or a competitor's:
+  **(1)** How big are the messages? **(2)** Is delivery strictly ordered or parallel?
+  **(3)** How fast does the receiving system acknowledge? **(4)** What's the *peak-hour* rate behind that
+  daily average? Without those four, a msg/s or msg/day number is marketing, not engineering.
+
+---
+
+## 1. How a message flows (and why there are two different "speeds")
+
+A message moves through MessageFoundry in stages, each of which is written durably to the message store
+before the next begins (that's what guarantees nothing is lost):
+
+```
+   sender ──▶ [ receive + persist ] ──▶ [ route ] ──▶ [ transform ] ──▶ [ deliver ] ──▶ partner
+                     │                                                        │
+                 ACK to sender                                         partner ACK
+              (fast — "intake")                                    (paced by the partner)
+```
+
+This gives you **two distinct throughput numbers**, and they are often very far apart:
+
+| Rate | What it measures | Typical relationship |
+|---|---|---|
+| **Intake rate** (ACK-on-receipt) | How fast the engine can accept a message, store it durably, and acknowledge the *sender*. The sender never waits for downstream delivery. | **Higher** — ~450/s in our lab. Protects upstream senders from downstream slowness. |
+| **Delivery rate** (end-to-end) | How fast the engine can get a message all the way to the *partner* and confirm it landed. | **Lower** — ~60/s in our lab. Paced by everything downstream, especially the partner. |
+
+The gap between them is not waste — it's the buffer. MessageFoundry accepts and safely stores messages
+faster than it delivers them, so a momentary slowdown at a partner doesn't push back on the sender. The
+backlog drains as the partner catches up.
+
+**Takeaway:** when someone quotes a throughput number, ask *which* rate they mean. Intake numbers look
+impressive but say nothing about how fast messages actually reach their destination.
+
+---
+
+## 2. The single biggest factor: your partner systems and round-trip times
+
+This is the point most capacity conversations miss.
+
+**Ordered delivery is inherently serial.** When you require messages to arrive in the exact order they were
+received (the default, and a hard requirement for most clinical feeds), the engine can only have **one
+message in flight per destination at a time**:
+
+1. Send message *N* to the partner.
+2. **Wait** for the partner to receive it, process it (often persist it to *its own* database), and send
+   back an acknowledgement.
+3. Only then send message *N+1*.
+
+That waiting time is not the engine's — it belongs to the **network round-trip and the partner's own
+processing**. And because the stream is serial, it lands squarely in the critical path.
+
+### The round-trips that make up a message's journey
+
+Every message pays for several round-trips, and each one adds latency:
+
+- **Sender ↔ engine** — the intake acknowledgement. Fast, and it doesn't block delivery.
+- **Engine ↔ message store** — each durable stage handoff is a commit. If the store is on another server,
+  every commit is a network round-trip. (See §4.)
+- **Engine ↔ partner** — the delivery send *and the partner's acknowledgement*. **This is usually the
+  dominant term**, and MessageFoundry does not control it.
+- **Engine ↔ enrichment source** *(optional)* — if a handler does a live lookup (e.g. a provider or
+  eligibility check), that adds another round-trip during transformation. (See §5.)
+
+### What this does to throughput
+
+For an ordered stream, throughput is approximately:
+
+> **messages/second ≈ 1000 ÷ ( engine overhead + partner round-trip )**, in milliseconds
+
+The engine overhead is small and roughly fixed (~16 ms in our lab). The **partner round-trip is yours**,
+and it dominates as soon as it's more than a few milliseconds:
+
+| Partner acknowledges in… | Approx. ordered throughput | "Uniform" msgs/day* | Realistic msgs/day (peak-aware)† |
+|---|---|---|---|
+| ~0 ms (lab ideal) | ~60 /s | ~5.2 M | ~1.9 M |
+| 10 ms | ~38 /s | ~3.3 M | ~1.2 M |
+| 25 ms | ~24 /s | ~2.1 M | ~0.8 M |
+| 50 ms | ~15 /s | ~1.3 M | ~0.5 M |
+| 100 ms | ~9 /s | ~0.8 M | ~0.3 M |
+| 250 ms | ~4 /s | ~0.35 M | ~0.13 M |
+
+<sub>\* "Uniform" assumes traffic is perfectly flat around the clock — it never is (see §6).
+† "Peak-aware" divides by a 2.7× burst factor, which is the realistic planning number. Illustrative
+first-order model anchored to the measured ~16 ms engine floor; your numbers depend on your systems.</sub>
+
+The lesson is stark: a partner that takes a quarter-second to acknowledge each message will hold a single
+ordered stream to a few messages per second, and **no amount of engine or database speed changes that.**
+The bottleneck has moved outside MessageFoundry entirely.
+
+### What you can do about a slow partner
+
+- **Relax ordering where it's safe.** If a feed tolerates out-of-order delivery, unordered mode lets the
+  engine keep many messages in flight at once, hiding the partner's round-trip behind concurrency (see §3).
+- **Open multiple connections to the partner.** Each destination connection is its own independent stream,
+  so *N* connections give you *N* parallel ordered lanes (if the partner accepts concurrent connections).
+- **Fan out at the source.** Split one hot feed into several interfaces (see §7) so the aggregate isn't
+  gated by a single serial lane.
+- **Ask your partner about their acknowledgement latency.** It is often the cheapest thing to improve, and
+  it's the term that matters most.
+
+---
+
+## 3. Ordering: the throughput-vs-order trade-off
+
+Ordering guarantees and raw throughput pull in opposite directions. MessageFoundry lets you choose per
+feed:
+
+| Mode | Guarantee | Throughput | Use when |
+|---|---|---|---|
+| **Strict FIFO** *(default)* | Messages delivered in exactly the order received. | Bounded — one message in flight per destination (serial). | Order matters: ADT streams, anything where a later message corrects an earlier one. |
+| **Unordered** | No ordering guarantee. | Higher — many messages in flight at once; the partner round-trip is hidden by concurrency. | Order doesn't matter: independent results, logging, feeds keyed only by their own content. |
+
+Strict ordering being serial is not unique to MessageFoundry — it is a property of *any* system that
+guarantees order over a single stream. This is exactly why integration engines let you opt out of ordering
+when a feed can tolerate it: it's the most direct throughput lever available for a single interface.
+
+---
+
+## 4. The message store (durability has a cost, and location matters)
+
+Every stage handoff is committed durably to the store before the next stage runs — that's what makes
+delivery at-least-once and crash-safe. So the store's commit latency is part of every message's journey.
+
+| Choice | Effect on throughput |
+|---|---|
+| **SQLite (embedded, default)** | Runs *in-process* with no network hop — the fastest per-commit path. Its ceiling is single-writer serialization, which is the floor case in our testing, not a problem for most single-node deployments. |
+| **SQL Server / PostgreSQL (server database)** | Chosen for concurrency, high availability, and enterprise operations — **not** for raw single-stream speed. Every commit is now a **network round-trip** to the database server, which *adds* latency to each handoff. |
+| **Store location — local vs. across the network** | This matters more than the database brand. A commit to a co-located, fast local disk is dramatically quicker than a commit that crosses the network to a remote database server. In our testing a local NVMe store committed on the order of **~65× faster** than a round-trip to a database on another box. |
+| **Disk speed (NVMe vs. shared SAN)** | The store forces data to disk on commit for durability; faster storage lowers that per-commit cost. |
+
+**One store serves many interfaces.** A single node's store has enormous headroom — in our lab a local
+store sustained on the order of **tens of thousands of commits per second**, far more than any one ordered
+interface can generate. This is *why* the scaling story is "add interfaces," not "make one interface
+faster": the shared store is nowhere near the wall.
+
+---
+
+## 5. Message and transformation factors
+
+- **Message size.** A small ADT message and a large message carrying an embedded document are not the same
+  unit of work. Bigger messages cost more to parse, store, encrypt, and transmit.
+- **Parsing and validation.** Fast, tolerant field inspection on the routing path is cheap. Full strict
+  validation is deliberately opt-in per feed because it is the slow path — turn it on where you need
+  conformance, not everywhere.
+- **Transformation complexity.** How much work your handler does per message directly affects the rate.
+  Heavy transformation is one of the largest hardware-independent reducers of throughput.
+- **Live enrichment lookups.** A handler may do a sanctioned read-only lookup (e.g. a provider or
+  eligibility check against a database or FHIR endpoint). Each lookup is another round-trip that adds to a
+  message's time in the pipeline — powerful, but use it deliberately.
+- **Fan-out.** If one inbound message is delivered to *N* destinations, that's *N* deliveries to commit and
+  *N* partners to wait on. Each destination is its own independent lane.
+
+---
+
+## 6. Why "messages per day" is a misleading unit
+
+**Healthcare message traffic is bursty — it is nothing like a flat, around-the-clock stream.** It follows
+the operational rhythm of the facilities behind it, and that shape matters far more than the daily total.
+
+### The shape of real traffic
+
+- **Within a day:** volume is light overnight, ramps up sharply in the morning as registration, admissions,
+  rounds, lab draws, and shift changes begin, peaks during the busy clinical hours, and tapers into the
+  evening. Emergency-department and registration surges create short, sharp spikes on top of that curve.
+- **Across a week:** weekdays run **several times heavier** than weekends, driven by elective admissions,
+  scheduled procedures, and clinic activity.
+- **Across a year:** seasonal load (e.g. respiratory-illness season) shifts the baseline further.
+
+### The number that actually matters: the peak hour
+
+A system doesn't experience the daily *average* — it experiences the **busiest hour**. If you size for the
+average, you will be underwater exactly when volume is highest: queues back up, and end-to-end latency
+climbs right when timeliness matters most.
+
+**In the production ADT feeds we've profiled, the busiest hour runs about 2.7× the all-day average rate.**
+So a feed that averages out to some tidy daily figure is actually asking for **nearly three times that
+average rate** during its peak — and *that* is the rate your interface has to sustain.
+
+### Same interface, very different "daily capacity"
+
+Because you must size to the peak, the realistic daily capacity of an interface is well below the naive
+"sustained rate × 86,400 seconds" figure:
+
+| | Naive (assumes flat traffic) | Realistic (peak = 2.7× average) |
+|---|---|---|
+| One ordered interface at 60 msg/s | ~5.2 M messages/day | **~1.9 M messages/day** |
+
+Nothing about the engine changed between those two columns — only the assumption about traffic shape. The
+naive number is off by nearly 3×.
+
+**This is why a bare "handles X million messages/day" claim is close to meaningless.** Two feeds with the
+*same* daily total but different burst profiles need very different capacity. Always ask for the
+**peak-hour rate**, the **message size**, and the **ordering guarantee** behind any daily figure — those,
+not the daily total, tell you whether it fits your environment.
+
+---
+
+## 7. Putting it together: how to size a deployment
+
+There is no single throughput number — there's a short calculation using *your* inputs:
+
+1. **Start from the per-interface ceiling** for your ordering mode (our lab reference: ~60 msg/s ordered,
+   with an instant partner).
+2. **Apply your partner's real acknowledgement time** (§2). This is usually the biggest reduction.
+3. **Apply your traffic's burst factor** (§6) — divide by ~2.7 for ADT-shaped traffic (or measure your own
+   feed's peak-hour ÷ daily-average ratio).
+4. **Fan out to enough interfaces** to cover your aggregate peak, splitting feeds at the source.
+
+### A worked (illustrative) example
+
+Suppose a partner acknowledges in ~25 ms and you have ADT-shaped traffic:
+
+- Ordered throughput per interface ≈ 1000 ÷ (16 + 25) ≈ **~24 msg/s**.
+- Peak-aware daily capacity ≈ 24 × 86,400 ÷ 2.7 ≈ **~0.8 M messages/day per interface**.
+- To carry, say, a 5-million-message/day aggregate, you'd fan out to roughly **6–7 interfaces** — each
+  comfortably within one node's shared-store headroom.
+
+Your inputs will differ; the point is the *method*. Plug in your partner latency and your peak factor, then
+count interfaces.
+
+### Fan-out is the scaling story
+
+A mega health system cannot push all of its ADT through one interface, and it doesn't need to. It splits
+the feed at the source — by hospital, facility, region, or service line — across **multiple interfaces**.
+Aggregate capacity is the sum of the interfaces; the shared message store on a single node has ample
+headroom for the combined load, so you scale out by adding interfaces rather than trying to make one pipe
+infinitely fast.
+
+---
+
+## 8. Reference lab measurements (and their caveats)
+
+These are the figures referenced above, stated with full conditions so you can judge how they map to your
+environment. **They are theoretical maxima under controlled conditions using synthetic data — not
+guarantees, and not measured against any live clinical system.**
+
+| Measurement | Value | Conditions |
+|---|---|---|
+| End-to-end delivery, one ordered interface | **~60 msg/s** | Single strictly-ordered MLLP interface, synthetic ADT, pass-through (no heavy transform), server database over a LAN, default settings, **instant-acknowledging partner**. |
+| Intake (ACK-on-receipt) | **~450 msg/s** | Same setup; measures accept-and-persist, not end-to-end delivery. |
+| Single-node store commit ceiling | **~tens of thousands of commits/s** | Local NVMe store; the shared store is far from the bottleneck for realistic interface counts. |
+| Engine's own per-message overhead | **~16 ms** | Derived from the ordered end-to-end rate with an instant partner; this is the portion MessageFoundry contributes. |
+
+**Why these are conservative and why your mileage will vary:**
+
+- **Ordered, single interface.** The default strict-FIFO mode is serial by design. Unordered feeds and
+  multiple interfaces both go faster.
+- **Instant partner.** The lab partner acknowledges immediately. **Real partners don't** — and as §2 shows,
+  their round-trip time is usually the dominant factor. This is the single largest reason your end-to-end
+  number will differ from the lab number.
+- **No independent industry benchmark exists.** There is no trustworthy public per-node throughput number
+  for the major integration engines either — every published msg/s figure is hardware- and
+  workload-dependent. Treat *all* such numbers (including these) as starting points for your own
+  measurement, not as guarantees.
+
+---
+
+## Takeaways
+
+1. MessageFoundry adds only a few milliseconds per message; **your partner systems and round-trip times set
+   the real throughput.**
+2. There are **two** rates — fast intake and slower end-to-end delivery. Know which one a number refers to.
+3. **Strict ordering is serial** and therefore bounded; relax ordering, add connections, or fan out when you
+   need more.
+4. Traffic is **bursty** — size to the **peak hour (≈ 2.7× the average)**, not the daily total. "Messages
+   per day" alone tells you little.
+5. **Scale by fanning out** to multiple interfaces; one node's store has ample headroom.
+6. Interrogate every throughput claim with four questions: **message size, ordering, partner
+   acknowledgement time, and peak-hour rate.**