messagefoundry 0.2.7__tar.gz → 0.2.8__tar.gz

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/CHANGELOG.md

@@ -6,6 +6,46 @@ All notable changes to MessageFoundry are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.8] — 2026-06-27 — Early Access
+
+A tooling/ops release: the load harness gains a **multi-shard driver** so one harness can drive a
+`supervise` cluster (unblocking the multi-core throughput measurement), `supervise` resolves
+`--env` files for its shards, and a prominent upgrade note for the config-directory permission
+guard introduced in 0.2.6.
+
+> ### ⚠ Upgrading from ≤ 0.2.5 — tighten config-dir ACLs first
+> The config-directory permission guard (SEC-003 / ADR 0036), added in **0.2.6**, refuses to load a
+> `--config` directory that is **writable by a broad principal** (e.g. `Authenticated Users` /
+> `S-1-5-11`). A deployment whose config dir inherits that write — common under `C:\srv\…` — will
+> **fail to start on first upgrade to ≥ 0.2.6** with *"refusing to load config from writable-by-others
+> path …"*. **Before upgrading**, tighten the directory (elevated):
+> ```powershell
+> icacls "<config-dir>" /inheritance:d /T
+> icacls "<config-dir>" /remove:g *S-1-5-11 /T          # drop Authenticated Users
+> icacls "<config-dir>" /grant *S-1-5-18:(OI)(CI)F /grant *S-1-5-32-544:(OI)(CI)F /T  # SYSTEM + Admins
+> ```
+> See [`docs/SERVICE.md`](docs/SERVICE.md) → *Update to a new build* and *Lock down the config
+> directory (CONFIG-2)*.
+
+### Added
+- **Multi-shard load driving (`messagefoundry-harness`).** `python -m harness` gains
+  **`--skip-preflight`** (drive shard MLLP ports that no single `--engine` owns) and a repeatable
+  **`--shard-engine <url>`**: the engine poller now takes a list of shard APIs and **sums** each
+  shard's `/stats` (read/written/backlog/in_pipeline/queue_depth/dead) into one cluster sample, so
+  the no-loss reconcile and drain are **cluster-aggregate** — a healthy K-shard run reports pass,
+  not a false "lost on intake". With no `--shard-engine` the behavior is byte-identical to before.
+  Two sample graphs ship for the throughput suite: `harness/config/store_once` (the
+  dedup-triggering one-handler-`list[Send]`-of-identical-body shape for store-once) and
+  `harness/config/passthrough` (an internal `PassThrough()` re-ingress hop); the load graph
+  (`harness/config/load`) is now shard-taggable via `MEFOR_LOAD_SHARD_ADT`/`_RESULTS`/`_OTHER`. (#604)
+
+### Fixed
+- **`supervise --project-root`.** `supervise` now accepts `--project-root` and forwards it to each
+  spawned `serve --shard`, so `supervise --config <dir> --env <env>` resolves each shard's
+  `environments/<env>.toml` (previously the shards resolved nothing from their spawned cwd and
+  required an explicit `--service-config` posture). Backward compatible — no `--project-root` is
+  unchanged. (#602)
+
 ## [0.2.7] — 2026-06-27 — Early Access
 
 A docs/packaging release that fixes the broken badge images on the PyPI project page
@@ -299,7 +339,8 @@ tests, but the external code review + penetration test (the bar for a security-c
 - Releases are built, SBOM'd (CycloneDX), and signed with [Sigstore](https://www.sigstore.dev/) — see the
   `release` workflow.
 
-[Unreleased]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.7...HEAD
+[Unreleased]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.8...HEAD
+[0.2.8]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.7...v0.2.8
 [0.2.7]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.6...v0.2.7
 [0.2.6]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.5...v0.2.6
 [0.2.5]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.4...v0.2.5

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messagefoundry
-Version: 0.2.7
+Version: 0.2.8
 Summary: Open-source healthcare integration engine — route, transform, and validate messages across many formats and connection types
 Project-URL: Homepage, https://messagefoundry.org/
 Project-URL: Documentation, https://messagefoundry.org/

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/docs/SERVICE.md

@@ -91,6 +91,16 @@ definition drifted — the install script is idempotent (it stops and reconfigur
 If the package was installed **non-editable** (a plain `pip install .`), the venv holds a
 snapshot of the old code — run `.venv\Scripts\python.exe -m pip install -e .` first, then restart.
 
+> **⚠ Upgrading from ≤ 0.2.5 → ≥ 0.2.6: tighten the config-dir ACLs first.** The config-directory
+> permission guard (SEC-003 / ADR 0036) did **not** exist in 0.2.5. From 0.2.6 on, `serve` refuses to
+> start against a `--config` directory **writable by a broad principal** (e.g. `Authenticated Users` /
+> `S-1-5-11`) — *"refusing to load config from writable-by-others path …"*. A config dir that inherits
+> that write (common under `C:\srv\…`) therefore **fails its first start after the upgrade**. Lock it
+> down **before** restarting onto the new build — use the surgical recipe under
+> [Lock down the config directory (CONFIG-2)](#lock-down-the-config-directory-config-2)
+> (`icacls /inheritance:d /T` + `/remove:g *S-1-5-11 /T` + grant SYSTEM/Admins), the lighter-touch
+> alternative to a full `/inheritance:r` reset for a shared tree.
+
 ## Start / stop / status
 
 ```powershell
@@ -190,6 +200,15 @@ trust boundary: anyone who can write a `.py` file there can run code as the serv
   (full) and the service account (read+execute). It is **opt-in** because the config dir often lives
   inside a developer's repo where stripping inheritance is surprising — for production, point
   `-Config` at a dedicated admin-owned directory and pass `-LockConfigDir`.
+- **Fix an existing tree that inherits a broad write grant** without a full ACL reset. A config dir
+  placed under a shared root (e.g. `C:\srv\…`) often inherits `Authenticated Users` (`S-1-5-11`)
+  write, which trips the guard below. The lighter-touch alternative to `/inheritance:r` is to break
+  inheritance, surgically drop just the broad principal, and grant the run-as user read+execute:
+  ```powershell
+  icacls "C:\srv\mefor\config" /inheritance:d /T
+  icacls "C:\srv\mefor\config" /remove:g *S-1-5-11 /T
+  icacls "C:\srv\mefor\config" /grant "<run-as-user>:(OI)(CI)RX" /T
+  ```
 - The loader **actively enforces** this at load time (and on `/config/reload`), not just as a
   documented recommendation (ADR 0036, SEC-003):
   - On **Windows** the loader now parses the directory's and each `*.py`'s NTFS owner + DACL and

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/harness/__main__.py

@@ -69,6 +69,24 @@ def main(argv: list[str] | None = None) -> int:
         "--sink-port", type=int, default=2700, help="load: base correlation-sink port"
     )
     parser.add_argument("--sink-ports", type=int, default=1, help="load: contiguous sink ports")
+    parser.add_argument(
+        "--skip-preflight",
+        action="store_true",
+        help="load: skip the 'engine serves all target ports' check — for driving a `supervise` "
+        "multi-shard cluster from one harness (the MLLP ports are spread across the shard engines, "
+        "so no single --engine serves them all). Point every shard's MEFOR_LOAD_SINK_PORT at this "
+        "run's --sink-port so the one correlation sink aggregates all shards' end-to-end throughput.",
+    )
+    parser.add_argument(
+        "--shard-engine",
+        action="append",
+        metavar="URL",
+        help="load: an EXTRA engine API base URL to poll + aggregate alongside --engine (repeatable). "
+        "Drives a `supervise` multi-shard cluster: pass --engine for the primary shard and one "
+        "--shard-engine per other shard, and the harness sums every shard's /stats so the no-loss "
+        "reconcile + drain see true cluster totals. Pair with --skip-preflight (no one engine serves "
+        "all MLLP ports). With none given the single --engine is polled exactly as before.",
+    )
     parser.add_argument("--report-json", help="load: write the JSON report to this path")
     parser.add_argument("--report-csv", help="load: write the per-phase CSV to this path")
     parser.add_argument("--baseline", help="load: compare against this saved JSON report")
@@ -166,6 +184,8 @@ def _run_load(args: argparse.Namespace) -> int:
                 sink_port=args.sink_port,
                 sink_ports=args.sink_ports,
                 db_backend=args.db_backend,
+                skip_preflight=args.skip_preflight,
+                shard_engines=tuple(args.shard_engine or ()),
             )
         )
     except PreflightError as exc:

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/harness/config/load/_shape.py

@@ -18,6 +18,14 @@ Knobs (all optional, with safe defaults so ``serve`` works with none set):
 * ``MEFOR_LOAD_ADT_PORT``       — ADT hub inbound MLLP port (default 2600).
 * ``MEFOR_LOAD_RESULTS_PORT``   — results hub inbound MLLP port (default 2601).
 * ``MEFOR_LOAD_OTHER_PORT``     — other hub inbound MLLP port (default 2602).
+* ``MEFOR_LOAD_SHARD_ADT``      — ``supervise`` shard id for the ADT hub (default unset → no tag).
+* ``MEFOR_LOAD_SHARD_RESULTS``  — ``supervise`` shard id for the results hub (default unset → no tag).
+* ``MEFOR_LOAD_SHARD_OTHER``    — ``supervise`` shard id for the other hub (default unset → no tag).
+
+Shards (all optional): tagging a hub with ``shard=`` routes it to a named ``messagefoundry supervise``
+subprocess. **Unset (the default) = no tag = a single implicit shard**, so the SAME load graph serves
+both unsharded (default) and sharded (e.g. ``MEFOR_LOAD_SHARD_ADT=a`` / ``_RESULTS=b`` / ``_OTHER=b``
+→ a 2-shard layout) with no other change. The tags never affect routing.
 
 The graph is **synthetic and generic** — it models the *shape* of a high-fan-out estate (one big ADT
 hub plus results/orders hubs), never a real site. See ``docs/LOAD-TESTING.md``.
@@ -64,6 +72,13 @@ def _env_float(name: str, default: float, *, minimum: float = 0.0) -> float:
     return value
 
 
+def _env_shard(name: str) -> str | None:
+    """A shard id from the environment, or ``None`` when unset/blank (= no shard tag)."""
+    raw = os.environ.get(name)
+    raw = raw.strip() if raw else ""
+    return raw or None
+
+
 @dataclass(frozen=True)
 class Shape:
     """Resolved load-graph parameters (read once from the environment at load time)."""
@@ -78,6 +93,9 @@ class Shape:
     adt_port: int
     results_port: int
     other_port: int
+    shard_adt: str | None
+    shard_results: str | None
+    shard_other: str | None
 
     def sink_endpoint(self, index: int) -> tuple[str, int]:
         """Round-robin a destination index across the contiguous sink port range."""
@@ -101,6 +119,9 @@ def load_shape() -> Shape:
         adt_port=_env_int("MEFOR_LOAD_ADT_PORT", 2600, minimum=1),
         results_port=_env_int("MEFOR_LOAD_RESULTS_PORT", 2601, minimum=1),
         other_port=_env_int("MEFOR_LOAD_OTHER_PORT", 2602, minimum=1),
+        shard_adt=_env_shard("MEFOR_LOAD_SHARD_ADT"),
+        shard_results=_env_shard("MEFOR_LOAD_SHARD_RESULTS"),
+        shard_other=_env_shard("MEFOR_LOAD_SHARD_OTHER"),
     )
 
 

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/harness/config/load/graph.py

@@ -69,10 +69,19 @@ _OTH_HANDLERS = _register_lane("OTH", _SHAPE.results_fanout, _SHAPE)
 
 
 # Three inbound MLLP hubs matching the load profiles' target ports. Each fans every received message
-# to its lane's full handler set (the sender sends only the matching types to each port).
-inbound("IB_Load_ADT", MLLP(port=_SHAPE.adt_port), router="adt_router")
-inbound("IB_Load_Results", MLLP(port=_SHAPE.results_port), router="results_router")
-inbound("IB_Load_Other", MLLP(port=_SHAPE.other_port), router="other_router")
+# to its lane's full handler set (the sender sends only the matching types to each port). Each hub may
+# be tagged to a `supervise` shard via MEFOR_LOAD_SHARD_* (default unset → shard=None → no tag → a
+# single implicit shard = byte-identical to the unsharded graph); see _shape.
+inbound("IB_Load_ADT", MLLP(port=_SHAPE.adt_port), router="adt_router", shard=_SHAPE.shard_adt)
+inbound(
+    "IB_Load_Results",
+    MLLP(port=_SHAPE.results_port),
+    router="results_router",
+    shard=_SHAPE.shard_results,
+)
+inbound(
+    "IB_Load_Other", MLLP(port=_SHAPE.other_port), router="other_router", shard=_SHAPE.shard_other
+)
 
 
 @router("adt_router")

messagefoundry-0.2.8/harness/config/passthrough/graph.py

@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""Pass-through (PT) re-ingress system-under-test (§7 S7.4/S7.5).
+
+A minimal graph exercising the ADR 0013 (generalized) **pass-through** primitive: an MLLP entry hub
+routes every message to a handler that ``Send``\\ s it *into* an internal :func:`PassThrough` inbound
+(``PT_Relay``) — naming it like an outbound. The engine re-ingresses that body as a new, independent
+inbound message on ``PT_Relay``, whose **own** router/handler then forward it to the real outbound
+(the harness correlation sink). So one logical feed crosses an internal hop before delivery, with no
+external round-trip — the shape the PT re-ingress throughput rows measure.
+
+Synthetic + generic; reuses the load graph's ``_shape`` for the entry port and sink endpoint, so
+``MEFOR_LOAD_ADT_PORT`` / ``MEFOR_LOAD_SINK_PORT`` behave as in ``harness/config/load``. Serve it::
+
+    MEFOR_LOAD_ADT_PORT=2600 MEFOR_LOAD_SINK_PORT=2700 \
+      python -m messagefoundry serve --config harness/config/passthrough --db ./pt.db
+"""
+
+from __future__ import annotations
+
+from messagefoundry import MLLP, PassThrough, Send, handler, inbound, outbound, router
+from messagefoundry.config.models import RetryPolicy
+from messagefoundry.parsing.message import Message, RawMessage
+
+from harness.config.load._shape import load_shape
+
+_SHAPE = load_shape()
+
+_RETRY = RetryPolicy(
+    max_attempts=5, backoff_seconds=0.5, backoff_multiplier=2.0, max_backoff_seconds=5.0
+)
+
+# The real egress: one outbound delivering to the harness correlation sink.
+_SINK_HOST, _SINK_PORT = _SHAPE.sink_endpoint(0)
+outbound(
+    "OB_PT_Sink",
+    MLLP(host=_SINK_HOST, port=_SINK_PORT, connect_timeout=2.0, timeout_seconds=5.0),
+    retry=_RETRY,
+)
+
+# The entry hub: an external MLLP inbound whose handler hands every message off to the PT inbound.
+inbound("IB_PT_Entry", MLLP(port=_SHAPE.adt_port), router="pt_entry_router")
+
+# The internal pass-through inbound: no socket; fed only by the Send-into-PT handoff below. Its own
+# router/handler re-route the re-ingressed body to the real sink outbound.
+inbound("PT_Relay", PassThrough(), router="pt_relay_router")
+
+
+@router("pt_entry_router")
+def route_entry(msg: Message | RawMessage) -> list[str]:
+    return ["pt_entry_handler"]
+
+
+@handler("pt_entry_handler")
+def to_passthrough(msg: Message | RawMessage) -> Send:
+    # Send INTO the PT inbound (named like an outbound) → the engine re-ingresses it on PT_Relay.
+    return Send("PT_Relay", msg)
+
+
+@router("pt_relay_router")
+def route_relay(msg: Message | RawMessage) -> list[str]:
+    return ["pt_relay_handler"]
+
+
+@handler("pt_relay_handler")
+def to_sink(msg: Message | RawMessage) -> Send:
+    # The PT inbound's own handler forwards the re-ingressed body to the real outbound.
+    return Send("OB_PT_Sink", msg)

messagefoundry-0.2.8/harness/config/store_once/graph.py

@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""Store-once-deliver-many (L2b) system-under-test — the shape that actually triggers dedup.
+
+The load graph (``harness/config/load``) fans out via N **separate** handlers (one ``Send`` each → N
+``transform_handoff``s, each a single delivery → the body is stored **inline**), and its ``edit``
+transform rewrites MSH-6 *per destination* → N **distinct** bodies. So the load graph never exercises
+store-once-deliver-many.
+
+This graph fans out via **one** handler returning ``list[Send]`` of the **identical** body to N
+destinations — a single ``transform_handoff`` with N deliveries — which is exactly the shape that
+triggers L2b: the body is content-addressed and written **once** to ``shared_body``, referenced by N
+``queue.body_ref`` rows (instead of N inline copies). Serve it for §7 **J1/S7.1** (store-once
+functional) and **J9/S7.6** (write-amplification)::
+
+    MEFOR_LOAD_FANOUT=20 MEFOR_LOAD_SINK_PORT=2700 \
+      python -m messagefoundry serve --config harness/config/store_once --db ./store_once.db
+
+then inject a few ADT messages on the inbound port and inspect the store (``shared_body`` row count,
+``queue.body_ref``). Synthetic + generic; reuses the load graph's ``_shape`` for ports/sink/fan-out so
+``MEFOR_LOAD_FANOUT`` / ``MEFOR_LOAD_SINK_PORT`` / ``MEFOR_LOAD_ADT_PORT`` behave the same as there.
+"""
+
+from __future__ import annotations
+
+from messagefoundry import MLLP, Send, handler, inbound, outbound, router
+from messagefoundry.config.models import RetryPolicy
+from messagefoundry.parsing.message import Message, RawMessage
+
+from harness.config.load._shape import load_shape
+
+_SHAPE = load_shape()
+
+# A few attempts with brief backoff (the sink always AA's, so retries shouldn't fire) — a transient
+# hiccup rides out rather than dead-lettering and dirtying the no-loss / store-once inspection.
+_RETRY = RetryPolicy(
+    max_attempts=5, backoff_seconds=0.5, backoff_multiplier=2.0, max_backoff_seconds=5.0
+)
+
+
+def _register_dests(fanout: int) -> list[str]:
+    """Declare ``fanout`` outbound destinations, all delivering to the harness correlation sink."""
+    names: list[str] = []
+    for i in range(fanout):
+        dest = f"OB_StoreOnce_{i:02d}"
+        host, port = _SHAPE.sink_endpoint(i)
+        outbound(
+            dest,
+            MLLP(host=host, port=port, connect_timeout=2.0, timeout_seconds=5.0),
+            retry=_RETRY,
+        )
+        names.append(dest)
+    return names
+
+
+_DESTS = _register_dests(_SHAPE.fanout)
+
+inbound("IB_StoreOnce", MLLP(port=_SHAPE.adt_port), router="store_once_router")
+
+
+@router("store_once_router")
+def route(msg: Message | RawMessage) -> list[str]:
+    return ["fanout_identical_body"]
+
+
+@handler("fanout_identical_body")
+def fanout_identical_body(msg: Message | RawMessage) -> list[Send]:
+    # ONE handler → list[Send] of the SAME body to every destination. All N deliveries share this one
+    # transform_handoff AND carry the identical body, so the store dedups: one content-addressed
+    # shared_body row + N body_ref rows (not N inline copies). Deliberately NO per-destination
+    # transform — that would make the bodies distinct and defeat dedup (see harness/config/load for the
+    # realistic N-distinct-bodies shape).
+    return [Send(dest, msg) for dest in _DESTS]

messagefoundry-0.2.8/harness/load/enginepoll.py

@@ -0,0 +1,244 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""Engine-side sampling — the aggregate view the per-message metrics can't give.
+
+Polls the engine's HTTP API on an interval (``/stats``, ``/connections``, ``/status``) to track
+engine-side throughput (Δdone/Δt), backlog, dead-letter accumulation, and DB/WAL growth over the run,
+then measures **drain time** after offered load stops. The :class:`~messagefoundry.console.client`
+``EngineClient`` is synchronous (httpx), so every call runs in a thread via ``run_in_executor`` — the
+load engine's event loop is never blocked. The harness reaches the engine only through this API; it
+never touches the store.
+
+**Cluster-wide aggregation.** A ``messagefoundry supervise`` cluster spreads inbounds across several
+shard subprocesses, each with its own API. The poller takes a **list** of engine base-URLs (the
+primary ``--engine`` plus any ``--shard-engine``, de-duped), polls each in turn off the event loop,
+and **sums** each shard's read/written/backlog/in_pipeline/queue_depth/dead into one cluster sample
+(the sum is order-independent, so the sequential per-shard reads need no ordering) — so the no-loss
+reconcile compares cluster-aggregate ``read``/``written``/``backlog`` against the (already cluster-
+aggregate) client ``sent``/``sink_received``, and drain requires **every** shard to empty. With a
+single URL (the default) a sample is byte-identical to the one-shard behavior.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from dataclasses import dataclass
+from typing import Sequence
+
+from messagefoundry.console.client import ApiError, EngineClient
+
+
+@dataclass(frozen=True)
+class EngineSample:
+    """One engine-side observation. ``read``/``written``/``done``/``dead`` are cumulative since engine
+    start, so run totals are last − first. Under a multi-shard cluster every field is the **sum**
+    across all polled shards."""
+
+    elapsed_s: float
+    pending: int  # outbound stage, status=pending
+    inflight: int  # outbound stage, status=inflight
+    done: int  # outbound stage, status=done (delivered)
+    dead: int  # outbound stage, status=dead (dead-lettered)
+    read: int  # Σ inbound `read` (messages received)
+    written: int  # Σ outbound `written` (deliveries made)
+    out_dead: int  # Σ outbound `errored` (deliveries dead-lettered)
+    queue_depth: int  # Σ outbound queue_depth (pending + inflight)
+    in_pipeline: (
+        int  # NOT-DONE rows across ALL stages (ingress+routed+outbound) — whole-pipeline gauge
+    )
+    db_size_bytes: int
+    journal_mode: str
+    uptime_s: float
+
+    @property
+    def backlog(self) -> int:
+        return self.pending + self.inflight
+
+
+@dataclass(frozen=True)
+class _ShardSample:
+    """A single shard's contribution to one cluster sample (the per-URL summable parts)."""
+
+    pending: int
+    inflight: int
+    done: int
+    dead: int
+    read: int
+    written: int
+    out_dead: int
+    queue_depth: int
+    in_pipeline: int
+    db_size_bytes: int
+    uptime_s: float
+    journal_mode: str
+
+
+class EnginePoller:
+    """Samples one or more engine APIs off the event loop, aggregates them, and detects post-load
+    drain across the whole cluster."""
+
+    def __init__(
+        self, engine_urls: str | Sequence[str], token: str | None, *, origin: float
+    ) -> None:
+        # Accept a single URL (back-compat) or a list. The first URL is the "primary" whose `client`
+        # is exposed for one-off preflight reads (served-ports check).
+        urls = [engine_urls] if isinstance(engine_urls, str) else list(engine_urls)
+        if not urls:
+            raise ValueError("EnginePoller needs at least one engine URL")
+        # De-dup, order-preserving (primary first): passing the primary --engine ALSO as a
+        # --shard-engine would otherwise double-count that shard's read/written/backlog and mask real
+        # loss. Distinct shard APIs are unaffected; the single-URL default stays exactly one client.
+        seen: set[str] = set()
+        deduped: list[str] = []
+        for url in urls:
+            if url not in seen:
+                seen.add(url)
+                deduped.append(url)
+        self._urls = deduped
+        self._token = token
+        self._origin = origin
+        self._clients: list[EngineClient] = []
+        self._samples: list[EngineSample] = []
+
+    @property
+    def client(self) -> EngineClient | None:
+        """The PRIMARY shard's client (set after :meth:`open`) — for one-off preflight reads."""
+        return self._clients[0] if self._clients else None
+
+    @property
+    def samples(self) -> list[EngineSample]:
+        return self._samples
+
+    @property
+    def baseline(self) -> EngineSample | None:
+        return self._samples[0] if self._samples else None
+
+    @property
+    def final(self) -> EngineSample | None:
+        return self._samples[-1] if self._samples else None
+
+    async def open(self) -> None:
+        await asyncio.get_running_loop().run_in_executor(None, self._open_sync)
+
+    async def close(self) -> None:
+        loop = asyncio.get_running_loop()
+        clients, self._clients = self._clients, []
+        for client in clients:
+            await loop.run_in_executor(None, client.close)
+
+    async def sample_once(self) -> EngineSample | None:
+        sample = await asyncio.get_running_loop().run_in_executor(None, self._sample_sync)
+        if sample is not None:
+            self._samples.append(sample)
+        return sample
+
+    async def run(self, interval: float, stop: asyncio.Event) -> None:
+        """Sample every ``interval`` seconds until ``stop`` is set."""
+        await self.sample_once()
+        while not stop.is_set():
+            try:
+                await asyncio.wait_for(stop.wait(), timeout=interval)
+            except TimeoutError:
+                await self.sample_once()
+
+    async def await_drain(self, *, timeout: float, interval: float) -> float | None:
+        """Poll until the **whole cluster's** pipeline is empty and inbound/delivery counters stop
+        moving. Returns seconds-to-drain, or ``None`` on timeout.
+
+        Drain requires the *aggregate* ``in_pipeline == 0`` (no NOT-DONE rows in ANY stage of ANY
+        shard — ingress, routed, or outbound), the summed outbound backlog + per-edge ``queue_depth``
+        at zero, and ``read``/``written`` unchanged across a poll. Because the cluster sample sums
+        every shard, this only completes once **every** shard reports ``in_pipeline == 0`` and an
+        empty backlog. The ``in_pipeline`` gauge (from ``/stats``) closes the prior blind spot: a
+        fully **stalled** router/transform leaves the outbound backlog at 0 but ``in_pipeline > 0``,
+        so it no longer reads as drained."""
+        loop = asyncio.get_running_loop()
+        start = loop.time()
+        prev = self.final or await self.sample_once()
+        while loop.time() - start < timeout:
+            try:
+                await asyncio.wait_for(asyncio.sleep(interval), timeout=interval + 1.0)
+            except TimeoutError:  # pragma: no cover - defensive
+                pass
+            cur = await self.sample_once()
+            if cur is None:
+                continue
+            stable = prev is not None and cur.read == prev.read and cur.written == prev.written
+            if cur.backlog == 0 and cur.queue_depth == 0 and cur.in_pipeline == 0 and stable:
+                return loop.time() - start
+            prev = cur
+        return None
+
+    # --- sync helpers (run in the executor) ----------------------------------
+
+    def _open_sync(self) -> None:
+        clients: list[EngineClient] = []
+        for url in self._urls:
+            client = EngineClient(url)
+            if self._token:
+                client.set_token(self._token)  # does a /me request to validate
+            clients.append(client)
+        self._clients = clients
+
+    def _sample_sync(self) -> EngineSample | None:
+        """Sample every shard and SUM into one cluster observation.
+
+        Reachability mirrors the single-shard semantics: a shard that is transiently unreachable makes
+        the whole sample unavailable (return ``None`` → skip this tick, keep polling), rather than
+        silently reporting a too-low aggregate that would poison the baseline/final no-loss math."""
+        if not self._clients:
+            return None
+        shard_samples: list[_ShardSample] = []
+        for client in self._clients:
+            shard = self._sample_shard(client)
+            if shard is None:
+                return None  # one shard unreachable → skip the aggregate (keep polling)
+            shard_samples.append(shard)
+        # Journal mode is reported per shard; they share a backend in practice, so take the first
+        # (informational only — it doesn't feed the no-loss check).
+        return EngineSample(
+            elapsed_s=time.perf_counter() - self._origin,
+            pending=sum(s.pending for s in shard_samples),
+            inflight=sum(s.inflight for s in shard_samples),
+            done=sum(s.done for s in shard_samples),
+            dead=sum(s.dead for s in shard_samples),
+            read=sum(s.read for s in shard_samples),
+            written=sum(s.written for s in shard_samples),
+            out_dead=sum(s.out_dead for s in shard_samples),
+            queue_depth=sum(s.queue_depth for s in shard_samples),
+            in_pipeline=sum(s.in_pipeline for s in shard_samples),
+            db_size_bytes=sum(s.db_size_bytes for s in shard_samples),
+            journal_mode=shard_samples[0].journal_mode,
+            uptime_s=max(s.uptime_s for s in shard_samples),
+        )
+
+    @staticmethod
+    def _sample_shard(client: EngineClient) -> _ShardSample | None:
+        try:
+            stats = client.stats()
+            conns = client.connections()
+            status = client.status()
+        except ApiError:
+            return None  # transient unreachability — caller skips the whole sample
+        ob = stats.outbox_by_status
+        # `read` is populated only on inbound rows, `written` only on outbound rows — so summing the
+        # non-None values partitions inbound vs outbound without guessing role/direction strings.
+        read = sum(r.read for r in conns if r.read is not None)
+        written = sum(r.written for r in conns if r.written is not None)
+        out_dead = sum(r.errored or 0 for r in conns if r.written is not None)
+        queue_depth = sum(r.queue_depth or 0 for r in conns if r.queue_depth is not None)
+        return _ShardSample(
+            pending=ob.get("pending", 0),
+            inflight=ob.get("inflight", 0),
+            done=ob.get("done", 0),
+            dead=ob.get("dead", 0),
+            read=read,
+            written=written,
+            out_dead=out_dead,
+            queue_depth=queue_depth,
+            in_pipeline=stats.in_pipeline,
+            db_size_bytes=status.db.size_bytes,
+            uptime_s=status.engine.uptime_seconds,
+            journal_mode=status.db.journal_mode,
+        )

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/harness/load/runner.py

@@ -13,6 +13,7 @@ from __future__ import annotations
 import asyncio
 import contextlib
 import time
+from typing import Sequence
 
 from messagefoundry.console.client import ApiError
 
@@ -45,6 +46,8 @@ async def run_load(
     sink_port: int = 2700,
     sink_ports: int = 1,
     db_backend: str | None = None,
+    skip_preflight: bool = False,
+    shard_engines: Sequence[str] = (),
 ) -> RunReport:
     ids = ControlIds(prefix=id_prefix)
     # Generate + parse the corpus off the event loop (hl7apy validation is slow) before anything runs.
@@ -60,7 +63,10 @@ async def run_load(
         host=sink_host,
         ports=tuple(sink_port + i for i in range(sink_ports)),
     )
-    poller = EnginePoller(engine_url, token, origin=time.perf_counter())
+    # Poll the primary --engine plus every --shard-engine and AGGREGATE (sum) their /stats, so the
+    # no-loss reconcile and drain see CLUSTER totals — not just the one shard the --engine names. With
+    # no shard_engines this is exactly [engine_url] = byte-identical to the single-shard behavior.
+    poller = EnginePoller([engine_url, *shard_engines], token, origin=time.perf_counter())
     pools = [
         (t, ConnectionPool(t, profile.pool_size, correlator, metrics)) for t in profile.targets
     ]
@@ -71,7 +77,14 @@ async def run_load(
     try:
         await sink.start()
         await poller.open()
-        await _preflight(poller, profile)  # raises PreflightError if unreachable / ports missing
+        if not skip_preflight:
+            await _preflight(
+                poller, profile
+            )  # raises PreflightError if unreachable / ports missing
+        # When skipped (multi-shard driving): one harness drives MLLP ports spread across several
+        # `supervise` shard engines, so no single --engine serves them all. The served-ports check is
+        # bypassed; the correlation sink still measures aggregate E2E + no-loss across every shard
+        # delivering to this run's --sink-port (set each shard's MEFOR_LOAD_SINK_PORT to match).
 
         dispatcher.start()
         poll_task = asyncio.create_task(poller.run(profile.poll_interval_s, poll_stop))

{messagefoundry-0.2.7 → messagefoundry-0.2.8}/messagefoundry/__init__.py

@@ -61,7 +61,7 @@ from messagefoundry.parsing.message import Message, RawMessage
 from messagefoundry.parsing.split import split_by_obr
 from messagefoundry.timezone import convert_hl7_timestamp, to_zone
 
-__version__ = "0.2.7"
+__version__ = "0.2.8"
 
 __all__ = [
     "Message",