messagefoundry 0.2.15__tar.gz → 0.3.0__tar.gz

{messagefoundry-0.2.15 → messagefoundry-0.3.0}/CHANGELOG.md

@@ -6,6 +6,46 @@ All notable changes to MessageFoundry are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.3.0] — 2026-07-13 — Early Access
+
+Highlights since 0.2.15 — streaming attachments end-to-end, a copy-on-Send message model, richer
+connectivity, and a run of security hardening. (Concise highlights; the git history is the full change set.)
+
+### Added
+- **Streaming large attachments end-to-end** (#149, ADR 0105) — very-large OBX-5 documents are detached from
+  the message skeleton at ingress and streamed through routing → transform → delivery on all three stores
+  (SQLite / SQL Server / Postgres), with an operator read/download surface.
+- **Copy-on-Send message model** (ADR 0104) — `Send` snapshots the message at construction (opt-in
+  `[pipeline].snapshot_on_send`), plus `Message.copy()` and a recognition-first `message_type_of(accepts=)`
+  predicate; and in the IDE, a point-and-click HL7 field picker and the Steps-view authoring palette
+  (ADR 0103 / 0106).
+- **Connectivity** — generic HTTP auth (OAuth 2.0 client-credentials + Digest) and HTTP response-header
+  capture (#154, #65); a connector `SecretProvider` seam with a HashiCorp Vault backend (#196).
+- **Monitoring** — an engine-wide KPI roll-up with a saturation-derivative alert and DB-pool metrics (#93).
+
+### Changed
+- **pipeline: `batch_handoff_statements` now defaults ON on SQL Server** ([ADR 0075](docs/adr/0075-per-hop-sql-statement-batching.md))
+  — per-hop SQL statement batching (a **distance-insurance** lever: folds each message's per-hop store round-trips
+  into the fewest `pyodbc.execute()` T-SQL batches, cutting **network round-trips, NOT transactions** — the
+  single per-hop `COMMIT` is untouched, `commits/msg` stays 2.000) now activates **by default** on the SQL Server
+  store. Promoted 2026-07-08 after Bench B showed **harmless-near** (batch ON vs OFF within ±0.4% at ~0.28 ms RTT,
+  zero-loss) and **helps-far** (constant ~−18% ACK-p99, absolute saving widening with RTT: −84 ms @ +20 ms,
+  −212 ms @ +50 ms) over a green SS correctness precondition (`tests/test_adr0075_batch_sqlserver.py`, 9 passed).
+  **Fail-closed + SQL-Server-only** — Postgres (asyncpg) and SQLite are byte-identical no-ops; the flag is retained
+  **only as an emergency off-switch**: set `[pipeline].batch_handoff_statements = false` to disable.
+
+### Security
+- **The published PyPI source distribution no longer ships the whole repo** (#1020) — the sdist is pinned to
+  the `messagefoundry` package + its metadata, with a fail-closed "sdist is package-only" gate in the release
+  workflow. No release ever exposed PHI or customer data.
+- **Transport hardening** — certificate-revocation refusal extended to outbound-connector TLS (#201); in-use
+  memory protection for the store (#198); the publish deny-list is read from the ref being published (#983).
+
+### Fixed
+- **CI reliability** — bound PHI-retention on the Windows service smoke (#1011); wrap pyodbc-heavy SQL Server
+  steps with a native-crash retry (#1010); multi-message finalizers take their per-message locks in canonical
+  order (#980).
+
 ## [0.2.15] — 2026-07-06 — Early Access
 
 **Thread-hop fusion (ADR 0071 B5, flagged default-OFF) + the browser ops dashboard, pooled-claimer
@@ -28,11 +68,25 @@ is additive / opt-in.
   crossing-count micro-bench + a Windows CI mechanism gate, and a connscale B0/B1 fusion A/B harness axis
   (`fuse_ab` profile, `trials`-banked); the throughput GO/NO-GO (ship-by-default vs escalate to
   free-threading, ADR 0053) is a separate SQL-Server bench, not a merge gate.
+  - **Promote-gate resolved 2026-07-06 — NO-GO** ([ADR 0071](docs/adr/0071-cut-executor-round-trips-b5.md)
+    §8/§10, #787): the SQL-Server `fuse_ab` bench measured a real but sub-threshold **+6.5 / +9.3 / +10.0 %**
+    lift (below the ≥10% bar; zero-loss held), so `fuse_thread_hops` **stays default-OFF** and the lever
+    escalates to free-threading (ADR 0053). Bench-SHA provenance: the run was at commit `8bab40e2`, which is
+    **not** an ancestor of `main` (PR5 was squash-merged as `90f80a3`, #780) — but `git diff 8bab40e2 90f80a3`
+    is **empty**, so the NO-GO bench ran a code tree byte-identical to merged PR5.
 - **store: pooled-claimer primitives** `claim_fifo_heads` / `list_fifo_lanes` / `release_claimed` on all
-  three backends ([ADR 0066](docs/adr/0066-pooled-stage-claimers.md) PR 2; **unwired — no engine caller
-  yet**): a FIFO-safe multi-lane head-claim (probe-then-claim, EMPTY-on-locked-head — never a #285 skip
+  three backends ([ADR 0066](docs/adr/0066-pooled-stage-claimers.md) PR 2 — **now wired by the
+  `StageDispatcher` and the default claim path since #755/#744, see _Changed_ below**): a FIFO-safe
+  multi-lane head-claim (probe-then-claim, EMPTY-on-locked-head — never a #285 skip
   to seq N+1), a read-only head-due-aware lane discovery for the pooled sweep, and an attempts-neutral
   claim release. `claim_next_fifo` / `claim_next_fifo_batch` / `claim_ready` are untouched.
+- **pipeline: bounded pooled-mode infra-fault handling (T17)** (ADR 0070; #766) — in `pooled` mode a lane
+  whose head keeps failing on an infrastructure fault now **re-pends its head at an exponential-capped
+  backoff** (cap `[pipeline].infra_fault_backoff_cap`, default **60 s**) instead of spinning the ~4×/s
+  discovery sweep, and after `[pipeline].infra_fault_stop_after` (default **10**) consecutive zero-progress
+  faults (~4 min of wall clock) applies `[pipeline].infra_fault_policy` (default **`stop`**): STOP-the-lane
+  with a throttled `lane_stuck` alert — **never** auto-dead-letter. `retry_forever` instead keeps re-pending
+  at the cap and alerts once the horizon is crossed. `per_lane` mode is unaffected.
 - **Browser ops dashboard (read-only, M1)** (#75, [ADR 0065](docs/adr/0065-web-ops-dashboard.md)) — an
   **opt-in** (`[api].serve_ui`, default **off**), same-origin, zero-install browser ops view served under
   `/ui` by the engine's FastAPI app. Read-only: a live-polling connections dashboard (In/Out/Queued/
@@ -130,6 +184,17 @@ is additive / opt-in.
   `ix_queue_ready(stage, status, …)` index instead of one unindexed status-only full scan of the
   queue (Postgres additionally drops an unsargable `OR $n IS NULL` form). Same rows recovered, same
   single transaction — all three backends.
+- **Default staged-pipeline claim path flipped to pooled per-stage claimers**
+  ([ADR 0066](docs/adr/0066-pooled-stage-claimers.md); `[pipeline].claim_mode` default `per_lane` →
+  **`pooled`** — issue #744, shipped via PR #765). The `StageDispatcher` was first wired in behind
+  `[pipeline].claim_mode` **default-OFF** (#755, ADR 0066 PR4) and is now the **default** claim topology:
+  one dispatcher per stage running a handful of pooled claimer tasks over the `claim_fifo_heads` /
+  `list_fifo_lanes` / `release_claimed` primitives (batch-claiming head-prefixes across lanes), collapsing
+  the ~4,500 per-(lane×stage) claim loops that saturated a shared server DB at high fan-out and holding
+  zero-loss where `per_lane` dropped messages. **`per_lane` stays fully selectable as the byte-identical
+  opt-out** (`[pipeline].claim_mode = "per_lane"`), enforced by the zero-pooled-construction test sentinel.
+  Reliability-core — read once at engine start (a `/config/reload` does not toggle it; restart to change).
+  Single-node scope; the at-least-once / per-lane-FIFO / poison-guard invariants are unchanged in both modes.
 
 ## [0.2.14] — 2026-07-01 — Early Access
 

{messagefoundry-0.2.15 → messagefoundry-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messagefoundry
-Version: 0.2.15
+Version: 0.3.0
 Summary: Open-source healthcare integration engine — route, transform, and validate messages across many formats and connection types
 Project-URL: Homepage, https://messagefoundry.org/
 Project-URL: Documentation, https://messagefoundry.org/
@@ -32,17 +32,13 @@ Requires-Dist: hl7>=0.4.5
 Requires-Dist: hl7apy>=1.3
 Requires-Dist: ldap3>=2.9
 Requires-Dist: prometheus-client>=0.20
+Requires-Dist: psutil>=6.0
 Requires-Dist: pydantic>=2.6
 Requires-Dist: pyspnego>=0.10
 Requires-Dist: starlette>=1.3.1
 Requires-Dist: tomlkit>=0.12
 Requires-Dist: tzdata>=2024.1
 Requires-Dist: uvicorn[standard]>=0.29
-Provides-Extra: console
-Requires-Dist: httpx>=0.27; extra == 'console'
-Requires-Dist: keyring>=24.0; extra == 'console'
-Requires-Dist: pyside6>=6.6; extra == 'console'
-Requires-Dist: truststore>=0.10; extra == 'console'
 Provides-Extra: dev
 Requires-Dist: httpx>=0.27; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
@@ -57,6 +53,10 @@ Requires-Dist: pynetdicom<4,>=3.0.4; extra == 'dicom'
 Provides-Extra: fhir
 Requires-Dist: fhir-resources>=7.1.0; extra == 'fhir'
 Requires-Dist: fhirpathpy>=2.2.0; extra == 'fhir'
+Provides-Extra: harness
+Requires-Dist: httpx>=0.27; extra == 'harness'
+Requires-Dist: pyside6>=6.6; extra == 'harness'
+Requires-Dist: truststore>=0.10; extra == 'harness'
 Provides-Extra: otel
 Requires-Dist: opentelemetry-exporter-otlp>=1.20; extra == 'otel'
 Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
@@ -66,6 +66,8 @@ Provides-Extra: sftp
 Requires-Dist: paramiko>=5.0; extra == 'sftp'
 Provides-Extra: sqlserver
 Requires-Dist: aioodbc>=0.5; extra == 'sqlserver'
+Provides-Extra: vault
+Requires-Dist: hvac>=2.3.0; extra == 'vault'
 Provides-Extra: webauthn
 Requires-Dist: webauthn<4,>=3.0.0; extra == 'webauthn'
 Provides-Extra: x12
@@ -94,7 +96,8 @@ guided tooling or extend it in Python; it runs on SQLite, PostgreSQL, or SQL Ser
 authentication, RBAC, audit logging, and encryption-at-rest built in.
 
 > Python import package: `messagefoundry`. Built with **hl7apy** + **python-hl7** (HL7 parsing/
-> validation), **FastAPI** (engine API), and **PySide6** (admin console).
+> validation), **FastAPI** (engine API + the same-origin `/ui` web console), and **PySide6** (the
+> standalone test harness).
 
 ## What it is
 
@@ -107,8 +110,8 @@ control; either way the configuration is version-controlled and yours.
 ## Architecture
 
 **Engine-as-library + localhost API.** The engine is an importable Python package.
-The PySide6 console talks to it over a localhost HTTP + WebSocket API — the same way
-whether the engine runs in-process, as a local daemon, or (later) on a remote host.
+A browser **web console** (served same-origin at `/ui`) talks to it over a localhost HTTP + WebSocket
+API — the same way whether the engine runs in-process, as a local daemon, or (later) on a remote host.
 No hand-rolled IPC; the deployment split is a config choice, not an architectural fork.
 
 See **[docs/architecture-diagram.md](https://github.com/MEFORORG/MessageFoundry/blob/main/docs/architecture-diagram.md)** for the rendered diagrams —
@@ -149,7 +152,7 @@ Full documentation lives on **[messagefoundry.org](https://messagefoundry.org/)*
 
 - **[Mental map](https://messagefoundry.org/assets/docs/MessageFoundry-Mental-Model.pdf)** — an
   orientation guide to how the pieces fit: Connections → Router → Handlers → Connections, with the
-  headless engine and the console/IDE that drive it.
+  headless engine and the web console/IDE that drive it.
 - **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)** — install,
   configure, secure, and roll out to production.
 - **[User Guide](https://messagefoundry.org/assets/docs/MessageFoundry-User-Guide.pdf)** — author and
@@ -179,19 +182,21 @@ deliberate — replace `<version>` with the current release shown at the top of
 needs (each is opt-in and lazy-imported):
 
 ```bash
-pip install "messagefoundry[console]==<version>"     # admin console (PySide6 GUI) — the operator UI; most operators want this
+pip install "messagefoundry-webconsole==<version>"   # the browser web console (/ui) — the operator UI; most operators want this
 pip install "messagefoundry[postgres]==<version>"    # PostgreSQL store backend (production server DB)
 pip install "messagefoundry[sqlserver]==<version>"   # SQL Server store backend (+ OS-level ODBC Driver 18)
 pip install "messagefoundry[sftp]==<version>"        # SFTP transport for the REMOTEFILE connector
 pip install "messagefoundry[dicom]==<version>"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
+pip install "messagefoundry[harness]==<version>"     # the standalone PySide6 test harness GUI
 ```
 
-**What's in the `messagefoundry` package — and what isn't.** It is the **engine plus the admin
-console**: the console ships in the same wheel, with its PySide6/GUI dependencies as the opt-in
-`[console]` extra, so a headless server, container, or adopter install stays lean — the console is one
-flag away (`messagefoundry[console]`) when you want the operator UI. The **VS Code extension is a
-separate product, not on PyPI** (a VS Code extension is a different ecosystem); see *VS Code extension &
-test harness* below for where to get it.
+**What's in the `messagefoundry` package — and what isn't.** It is the **engine**; the operator UI is
+the browser **web console** served same-origin at `/ui`, which ships as a separate, version-matched
+wheel (`messagefoundry-webconsole`) the engine mounts in-process (turn it on with `[api].serve_ui`), so
+a headless server, container, or adopter install stays lean. (The former PySide6 desktop console was
+retired in favour of the web console — BACKLOG #103; PySide6 now backs only the opt-in `[harness]` test
+tooling.) The **VS Code extension is a separate product, not on PyPI** (a VS Code extension is a
+different ecosystem); see *VS Code extension & test harness* below for where to get it.
 
 > **Verify before you install (supply chain).** Every release is built by a GitHub Actions workflow,
 > Sigstore-signed, and carries SLSA build-provenance + PEP 740 attestations. Verify a downloaded
@@ -226,10 +231,11 @@ python -m messagefoundry serve --config samples/config --db messagefoundry.db --
 # API on http://127.0.0.1:8765 — GET /connections, /messages, /stats, WS /ws/stats
 ```
 
-Then open the admin console (needs the `console` extra: `pip install -e ".[console]"`):
+Then open the admin console in a browser (install the web console alongside the engine —
+`pip install -e packaging/messagefoundry-webconsole` — and set `[api].serve_ui = true`):
 
 ```bash
-python -m messagefoundry.console --url http://127.0.0.1:8765
+# browse to http://127.0.0.1:8765/ui and sign in
 ```
 
 ### VS Code extension & test harness

{messagefoundry-0.2.15 → messagefoundry-0.3.0}/README.md

@@ -16,7 +16,8 @@ guided tooling or extend it in Python; it runs on SQLite, PostgreSQL, or SQL Ser
 authentication, RBAC, audit logging, and encryption-at-rest built in.
 
 > Python import package: `messagefoundry`. Built with **hl7apy** + **python-hl7** (HL7 parsing/
-> validation), **FastAPI** (engine API), and **PySide6** (admin console).
+> validation), **FastAPI** (engine API + the same-origin `/ui` web console), and **PySide6** (the
+> standalone test harness).
 
 ## What it is
 
@@ -29,8 +30,8 @@ control; either way the configuration is version-controlled and yours.
 ## Architecture
 
 **Engine-as-library + localhost API.** The engine is an importable Python package.
-The PySide6 console talks to it over a localhost HTTP + WebSocket API — the same way
-whether the engine runs in-process, as a local daemon, or (later) on a remote host.
+A browser **web console** (served same-origin at `/ui`) talks to it over a localhost HTTP + WebSocket
+API — the same way whether the engine runs in-process, as a local daemon, or (later) on a remote host.
 No hand-rolled IPC; the deployment split is a config choice, not an architectural fork.
 
 See **[docs/architecture-diagram.md](https://github.com/MEFORORG/MessageFoundry/blob/main/docs/architecture-diagram.md)** for the rendered diagrams —
@@ -71,7 +72,7 @@ Full documentation lives on **[messagefoundry.org](https://messagefoundry.org/)*
 
 - **[Mental map](https://messagefoundry.org/assets/docs/MessageFoundry-Mental-Model.pdf)** — an
   orientation guide to how the pieces fit: Connections → Router → Handlers → Connections, with the
-  headless engine and the console/IDE that drive it.
+  headless engine and the web console/IDE that drive it.
 - **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)** — install,
   configure, secure, and roll out to production.
 - **[User Guide](https://messagefoundry.org/assets/docs/MessageFoundry-User-Guide.pdf)** — author and
@@ -101,19 +102,21 @@ deliberate — replace `<version>` with the current release shown at the top of
 needs (each is opt-in and lazy-imported):
 
 ```bash
-pip install "messagefoundry[console]==<version>"     # admin console (PySide6 GUI) — the operator UI; most operators want this
+pip install "messagefoundry-webconsole==<version>"   # the browser web console (/ui) — the operator UI; most operators want this
 pip install "messagefoundry[postgres]==<version>"    # PostgreSQL store backend (production server DB)
 pip install "messagefoundry[sqlserver]==<version>"   # SQL Server store backend (+ OS-level ODBC Driver 18)
 pip install "messagefoundry[sftp]==<version>"        # SFTP transport for the REMOTEFILE connector
 pip install "messagefoundry[dicom]==<version>"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
+pip install "messagefoundry[harness]==<version>"     # the standalone PySide6 test harness GUI
 ```
 
-**What's in the `messagefoundry` package — and what isn't.** It is the **engine plus the admin
-console**: the console ships in the same wheel, with its PySide6/GUI dependencies as the opt-in
-`[console]` extra, so a headless server, container, or adopter install stays lean — the console is one
-flag away (`messagefoundry[console]`) when you want the operator UI. The **VS Code extension is a
-separate product, not on PyPI** (a VS Code extension is a different ecosystem); see *VS Code extension &
-test harness* below for where to get it.
+**What's in the `messagefoundry` package — and what isn't.** It is the **engine**; the operator UI is
+the browser **web console** served same-origin at `/ui`, which ships as a separate, version-matched
+wheel (`messagefoundry-webconsole`) the engine mounts in-process (turn it on with `[api].serve_ui`), so
+a headless server, container, or adopter install stays lean. (The former PySide6 desktop console was
+retired in favour of the web console — BACKLOG #103; PySide6 now backs only the opt-in `[harness]` test
+tooling.) The **VS Code extension is a separate product, not on PyPI** (a VS Code extension is a
+different ecosystem); see *VS Code extension & test harness* below for where to get it.
 
 > **Verify before you install (supply chain).** Every release is built by a GitHub Actions workflow,
 > Sigstore-signed, and carries SLSA build-provenance + PEP 740 attestations. Verify a downloaded
@@ -148,10 +151,11 @@ python -m messagefoundry serve --config samples/config --db messagefoundry.db --
 # API on http://127.0.0.1:8765 — GET /connections, /messages, /stats, WS /ws/stats
 ```
 
-Then open the admin console (needs the `console` extra: `pip install -e ".[console]"`):
+Then open the admin console in a browser (install the web console alongside the engine —
+`pip install -e packaging/messagefoundry-webconsole` — and set `[api].serve_ui = true`):
 
 ```bash
-python -m messagefoundry.console --url http://127.0.0.1:8765
+# browse to http://127.0.0.1:8765/ui and sign in
 ```
 
 ### VS Code extension & test harness

{messagefoundry-0.2.15 → messagefoundry-0.3.0}/messagefoundry/__init__.py

@@ -13,11 +13,13 @@ Config modules define the message graph against this surface::
 
 from messagefoundry.config.models import (
     AckMode,
+    BatchConfig,
     BuildupThreshold,
     ContentType,
     InternalErrorPolicy,
     OrderingMode,
     RetryPolicy,
+    SaturationThreshold,
     StallThreshold,
 )
 from messagefoundry.config.active_environment import current_environment
@@ -33,6 +35,7 @@ from messagefoundry.config.wiring import (
     DatabaseLookup,
     DatabasePoll,
     DatabaseRef,
+    Direct,
     Email,
     FhirLookup,
     File,
@@ -54,14 +57,35 @@ from messagefoundry.config.wiring import (
     Timer,
     X12,
     Send,
+    SetMeta,
     SetState,
+    MessageTypeError,
     code_set,
     env,
     handler,
     inbound,
+    message_type_of,
     outbound,
     router,
 )
+from messagefoundry.actions import (
+    append_to_field,
+    arith_field,
+    code_lookup,
+    convert_case,
+    copy_field,
+    copy_segment,
+    date_diff_field,
+    delete_segment,
+    format_date,
+    pad_field,
+    replace_literal,
+    set_field,
+    split_field,
+    substring_field,
+    trim_field,
+)
+from messagefoundry.diagnostics import checkpoint, log_note
 from messagefoundry.parsing.groups import SegmentGroup
 from messagefoundry.parsing.message import Message, RawMessage
 from messagefoundry.parsing.split import split_by_obr
@@ -74,7 +98,7 @@ from messagefoundry.timezone import (
     to_zone,
 )
 
-__version__ = "0.2.15"
+__version__ = "0.3.0"
 
 __all__ = [
     "Message",
@@ -83,6 +107,7 @@ __all__ = [
     "split_by_obr",
     "Send",
     "SetState",
+    "SetMeta",
     "state_get",
     "response_get",
     "MLLP",
@@ -94,6 +119,7 @@ __all__ = [
     "Loopback",
     "PassThrough",
     "Rest",
+    "Direct",
     "Email",
     "SMTP",
     "FHIR",
@@ -125,11 +151,32 @@ __all__ = [
     "InternalErrorPolicy",
     "BuildupThreshold",
     "StallThreshold",
+    "SaturationThreshold",
+    "BatchConfig",
     "ContentType",
     "inbound",
     "outbound",
     "router",
     "handler",
+    "message_type_of",
+    "MessageTypeError",
+    "copy_field",
+    "set_field",
+    "append_to_field",
+    "trim_field",
+    "substring_field",
+    "pad_field",
+    "replace_literal",
+    "convert_case",
+    "arith_field",
+    "format_date",
+    "date_diff_field",
+    "split_field",
+    "code_lookup",
+    "copy_segment",
+    "delete_segment",
+    "log_note",
+    "checkpoint",
     "convert_hl7_timestamp",
     "to_zone",
     "parse_hl7_timestamp",