messagefoundry 0.2.2__tar.gz → 0.2.3__tar.gz

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/.gitignore

@@ -74,3 +74,9 @@ TRANSCRIPTS.md
 /docs/security/DEPENDENCY-RESPONSE-A4-HANDOFF.md
 # vuln_metrics.py local CSV output (the CI run uploads it as an artifact; not committed)
 /docs/security/metrics/
+
+# Operator-local load/throughput profiles — tuned to a specific deployment's volume; kept OUT of the
+# OSS repo + public mirror (the plan itself lives on the operator share, WIN2025-LOAD-THROUGHPUT-MATRIX.md).
+# Still usable by name (they sit in PROFILES_DIR) and copy-on to the test box.
+/harness/load/profiles/hospital-baseline.toml
+/harness/load/profiles/soak-12h.toml

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/CHANGELOG.md

@@ -6,6 +6,36 @@ All notable changes to MessageFoundry are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.3] — 2026-06-26 — Early Access
+
+A bug-fix + feature release: the SQL Server store no longer raises "connection busy" errors under
+concurrent load, plus connection/transport event logging, GUI-managed translation tables, and inbound
+listener port-conflict detection.
+
+### Fixed
+- **SQL Server: "Connection is busy with results for another command" under concurrent load (EF-6).**
+  `claim_next_fifo` — and three sibling sites (`_maybe_finalize`, `consume_recovery_code_hash`,
+  `consume_totp_step`) — read a result-set-returning statement with a lone `fetchone()` and could return
+  the pooled connection to the pool with the result set still pending, so the next borrower's first
+  command raced an `HY000` busy error (ODBC Driver 18, no MARS). All affected sites now fully drain the
+  result set (`fetchall`) before commit/release. SQLite and PostgreSQL were unaffected (asyncpg
+  materializes rows; SQLite has no shared pooled-connection single-result-set constraint). (#543)
+
+### Added
+- **Connection/transport event log + "Response Sent" ACK capture** (ADR 0020 / ADR 0021). A new id-keyed,
+  metadata-only `connection_event` table records inbound connection lifecycle, pre-ingress failures, and
+  outbound lane transitions, with a `[diagnostics]` config block (per-connection overrides + retention),
+  a `GET /events` read API, and a console **Event Log** page. Event reasons are scrubbed and encrypted at
+  rest. (#541)
+- **GUI-managed translation tables (code sets)** (ADR 0033). A code-set CLI + writer and a VS Code
+  extension grid editor / **Translation Tables** view for maintaining code-set mappings. (#540)
+- **Inbound listener port-conflict detection** — static + runtime checks that flag two inbound
+  connections bound to the same host:port before they collide at startup. (#538)
+
+### Changed
+- Docs: README install instructions are now version-agnostic and link the website docs; the roadmap
+  section is replaced with a features summary. (#542, #544)
+
 ## [0.2.2] — 2026-06-24 — Early Access
 
 A security-hardening release: PHI-at-rest encryption is closed across every backend, the active-passive
@@ -141,7 +171,9 @@ tests, but the external code review + penetration test (the bar for a security-c
 - Releases are built, SBOM'd (CycloneDX), and signed with [Sigstore](https://www.sigstore.dev/) — see the
   `release` workflow.
 
-[Unreleased]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.1...HEAD
+[Unreleased]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.3...HEAD
+[0.2.3]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.2...v0.2.3
+[0.2.2]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.1...v0.2.2
 [0.2.1]: https://github.com/wshallwshall/MessageFoundry/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/wshallwshall/MessageFoundry/compare/v0.1.0...v0.2.0
 [0.1.0]: https://github.com/wshallwshall/MessageFoundry/releases/tag/v0.1.0

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: messagefoundry
-Version: 0.2.2
+Version: 0.2.3
 Summary: Open-source healthcare integration engine — route, transform, and validate messages across many formats and connection types
 License-Expression: AGPL-3.0-or-later
 License-File: LICENSE
@@ -102,62 +102,34 @@ renders on GitHub and in the VS Code preview). The prose source of truth is
   and **transport TLS** (HTTPS/WSS for the API plus MLLP-over-TLS) are **built**; MFA and off-box log
   shipping remain on the roadmap. See [docs/PHI.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/PHI.md) for the full data-protection map.
 
-## Roadmap
-
-**Phase 1 — minimum reliable engine**
-- [x] Connection/Router/Handler model + config-module loader
-- [x] Durable message store / queue (SQLite WAL, outbox pattern)
-- [x] Parse / validate (tolerant peek + opt-in strict validation)
-- [x] MLLP source + destination (correct `0x0B … 0x1C 0x0D` framing, ACK/NACK)
-- [x] File source + destination
-- [x] Pipeline: source → parse/validate/filter/transform → outbox → per-dest workers,
-      with retry/backoff, dead-letter, and replay
-- [x] localhost API (connections start/stop, message track/search/detail, replay, stats,
-      live WebSocket feed) + `python -m messagefoundry serve`
-- [x] PySide6 console: connection dashboard, message browser, HL7 parse-tree viewer,
-      delivery/audit trail, replay (`python -m messagefoundry.console`)
-
-**Phase 1 complete.**
-
-**Since Phase 1 — now built**
-- [x] Staged pipeline (ingress → routed → outbound): at-least-once handoff, dead-letter, replay
-- [x] Authentication, RBAC, user-attributed audit log, at-rest body encryption (AES-256-GCM)
-- [x] **PostgreSQL** store backend (production, single-node)
-- [x] **Microsoft SQL Server** store backend (production, single-node)
-- [x] REST, SOAP, and Database destinations
-- [x] Database poll source
-- [x] Reference / lookup tables (`code_set`) for enrichment
-- [x] Alerting — logging sink + webhook/email notifier
-- [x] Connections-as-data (`connections.toml`) editable by hand or a VS Code GUI
-- [x] **Active-passive high availability** — self-fencing leadership lease, leader-gated graph, and a
-      failover-load test harness (kill-the-primary-under-load), on **both** PostgreSQL and SQL Server
-- [x] **Native transport TLS** — in-process API TLS (HTTPS/WSS) and MLLP-over-TLS, with an off-loopback
-      bind guard and a certificate-expiry monitor
-- [x] Published throughput + active-passive failover **baseline** ([docs/benchmarks/TUNING-BASELINE.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/benchmarks/TUNING-BASELINE.md))
-- [x] **SMART Backend Services token provider** — OAuth2 `client_credentials` + signed-JWT
-      `client_assertion` (RS384/ES384) authenticating the FHIR/REST outbound against real SMART-secured
-      servers (Epic, Oracle Health); a `with_smart_backend()` composer over `FHIR()`/`Rest()` that mints,
-      caches, and re-mints (on 401) a short-lived bearer, token endpoint gated by `[egress].allowed_http`
-      ([ADR 0024](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0024-smart-backend-services-token-provider.md))
-- [x] **base64 binary-carriage codec** — an `mfb64:v1:` marker carries arbitrary NUL-safe **bytes** over
-      the str/TEXT ingress + store (`RawMessage.from_bytes()`/`.raw_bytes`/`.binary()`/`.is_binary`), plus
-      HL7 OBX-5 ED (Encapsulated Data) embedding helpers ([ADR 0028](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0028-base64-binary-carriage-codec.md))
-- [x] **DICOM codec + C-STORE SCP** (Phase 1) — a pure codec (routing peek, headers/Structured Report,
-      code-first SR→HL7 mapping helpers; **headers + SR only, no pixel data**) on `content_type=dicom`
-      payload-agnostic ingress, plus an inbound C-STORE SCP listener (`DICOM()`) ([ADR 0025](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0025-dicom-codec-store-connectors.md))
-- [x] **Anonymizer / de-identification** — builds PHI-free test datasets from real traffic with
-      deterministic secret-per-dataset pseudonymization, field-anchored site-code scrub, and fail-closed
-      emission (never an un-scrubbed body), via a `tee anonymize-captures` subcommand and test-harness
-      hooks ([ADR 0030](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0030-anonymization-test-harness-tee.md))
-
-**Later** — higher-throughput delivery (a pooled/persistent MLLP connector); a read-only **component
-SDK** (fork-to-customize); DICOM Phase 2 (C-STORE SCU + C-ECHO + DICOMweb STOW-RS, designed); MFA and
-off-box log shipping. See [docs/EARLY-ADOPTER-GUIDE.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/EARLY-ADOPTER-GUIDE.md) §2 for the current
-built-vs-experimental map.
-
-Horizontal **active-active** scale-out (the multi-node cluster path) was **dropped on 2026-06-18 and
-its code removed** — it is not a planned milestone; single-leader **active-passive** HA (above) is the
-supported HA model.
+## Features
+
+MessageFoundry ships a reliable, PHI-aware engine today: the Connection/Router/Handler graph
+over a durable staged pipeline (at-least-once delivery, retries, dead-letter, replay), backed by
+SQLite, PostgreSQL, or SQL Server. It speaks MLLP (plain and over TLS), file/SFTP, REST, SOAP,
+FHIR, database, and DICOM (C-STORE SCP), parses HL7 v2 tolerantly (with opt-in strict validation)
+and carries other formats payload-agnostically (JSON, XML/SOAP, X12, binary). Security is
+first-class — authentication, RBAC, a user-attributed audit log, at-rest body encryption, and
+transport TLS — and it runs single-node or in active-passive high availability.
+
+**See the full, up-to-date feature breakdown — built vs. planned — at
+[messagefoundry.org/features-table.html](https://messagefoundry.org/features-table.html).**
+
+## Documentation
+
+Full documentation lives on **[messagefoundry.org](https://messagefoundry.org/)**:
+
+- **[Mental map](https://messagefoundry.org/assets/docs/MessageFoundry-Mental-Model.pdf)** — a one-page
+  picture of how the pieces fit: Connections → Router → Handlers → Connections, with the headless
+  engine and the console/IDE that drive it.
+- **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)** — install,
+  configure, secure, and roll out to production.
+- **[User Guide](https://messagefoundry.org/assets/docs/MessageFoundry-User-Guide.pdf)** — author and
+  operate interfaces day to day.
+
+**Get started:** [Quickstart](https://messagefoundry.org/getting-started.html) ·
+[Guides](https://messagefoundry.org/guides/) ·
+[Documents](https://messagefoundry.org/documents.html)
 
 ## Installing & rolling out
 
@@ -167,21 +139,23 @@ supported production artifact, with no source checkout required. Install it as a
 dependency**, then scaffold your own config repo ([ADR 0017](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0017-consumer-deployment-model.md)):
 
 ```bash
-pip install "messagefoundry==0.2.1"   # pin the exact engine version (core runtime, SQLite store)
-messagefoundry init ./my-config-repo     # scaffold a standalone config repo
+pip install "messagefoundry==<version>"   # pin the exact engine version (core runtime, SQLite store)
+messagefoundry init ./my-config-repo      # scaffold a standalone config repo
 cd ./my-config-repo
 messagefoundry serve --config config --env dev
 ```
 
-`0.2.1` is the current **Early Access** release on PyPI. Always **pin the exact version** so
-upgrades stay deliberate. Add the extras your deployment needs (each is opt-in and lazy-imported):
+MessageFoundry is in **Early Access**. Always **pin the exact version** so upgrades stay
+deliberate — replace `<version>` with the current release shown at the top of the
+[PyPI project page](https://pypi.org/project/messagefoundry/). Add the extras your deployment
+needs (each is opt-in and lazy-imported):
 
 ```bash
-pip install "messagefoundry[postgres]==0.2.1"    # PostgreSQL store backend (production server DB)
-pip install "messagefoundry[sqlserver]==0.2.1"   # SQL Server store backend (+ OS-level ODBC Driver 18)
-pip install "messagefoundry[console]==0.2.1"     # PySide6 admin console
-pip install "messagefoundry[sftp]==0.2.1"        # SFTP transport for the REMOTEFILE connector
-pip install "messagefoundry[dicom]==0.2.1"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
+pip install "messagefoundry[postgres]==<version>"    # PostgreSQL store backend (production server DB)
+pip install "messagefoundry[sqlserver]==<version>"   # SQL Server store backend (+ OS-level ODBC Driver 18)
+pip install "messagefoundry[console]==<version>"     # PySide6 admin console
+pip install "messagefoundry[sftp]==<version>"        # SFTP transport for the REMOTEFILE connector
+pip install "messagefoundry[dicom]==<version>"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
 ```
 
 > **Verify before you install (supply chain).** Every release is built by a GitHub Actions workflow,
@@ -193,7 +167,7 @@ pip install "messagefoundry[dicom]==0.2.1"       # DICOM codec + C-STORE SCP (py
 >
 > *(Engine developers install from a checkout instead — see [Development](#development).)*
 
-Piloting MessageFoundry? The **[Early-Adopter Installation & Rollout Guide](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/EARLY-ADOPTER-GUIDE.md)**
+Piloting MessageFoundry? The **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)**
 takes you from first install through a staged, go/no-go-gated path to full production
 (Lab → Shadow/Parallel → Limited → Full). It leads with an honest built-vs-experimental
 maturity map and covers prerequisites, install, security/PHI hardening, reliability

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/README.md

@@ -49,62 +49,34 @@ renders on GitHub and in the VS Code preview). The prose source of truth is
   and **transport TLS** (HTTPS/WSS for the API plus MLLP-over-TLS) are **built**; MFA and off-box log
   shipping remain on the roadmap. See [docs/PHI.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/PHI.md) for the full data-protection map.
 
-## Roadmap
-
-**Phase 1 — minimum reliable engine**
-- [x] Connection/Router/Handler model + config-module loader
-- [x] Durable message store / queue (SQLite WAL, outbox pattern)
-- [x] Parse / validate (tolerant peek + opt-in strict validation)
-- [x] MLLP source + destination (correct `0x0B … 0x1C 0x0D` framing, ACK/NACK)
-- [x] File source + destination
-- [x] Pipeline: source → parse/validate/filter/transform → outbox → per-dest workers,
-      with retry/backoff, dead-letter, and replay
-- [x] localhost API (connections start/stop, message track/search/detail, replay, stats,
-      live WebSocket feed) + `python -m messagefoundry serve`
-- [x] PySide6 console: connection dashboard, message browser, HL7 parse-tree viewer,
-      delivery/audit trail, replay (`python -m messagefoundry.console`)
-
-**Phase 1 complete.**
-
-**Since Phase 1 — now built**
-- [x] Staged pipeline (ingress → routed → outbound): at-least-once handoff, dead-letter, replay
-- [x] Authentication, RBAC, user-attributed audit log, at-rest body encryption (AES-256-GCM)
-- [x] **PostgreSQL** store backend (production, single-node)
-- [x] **Microsoft SQL Server** store backend (production, single-node)
-- [x] REST, SOAP, and Database destinations
-- [x] Database poll source
-- [x] Reference / lookup tables (`code_set`) for enrichment
-- [x] Alerting — logging sink + webhook/email notifier
-- [x] Connections-as-data (`connections.toml`) editable by hand or a VS Code GUI
-- [x] **Active-passive high availability** — self-fencing leadership lease, leader-gated graph, and a
-      failover-load test harness (kill-the-primary-under-load), on **both** PostgreSQL and SQL Server
-- [x] **Native transport TLS** — in-process API TLS (HTTPS/WSS) and MLLP-over-TLS, with an off-loopback
-      bind guard and a certificate-expiry monitor
-- [x] Published throughput + active-passive failover **baseline** ([docs/benchmarks/TUNING-BASELINE.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/benchmarks/TUNING-BASELINE.md))
-- [x] **SMART Backend Services token provider** — OAuth2 `client_credentials` + signed-JWT
-      `client_assertion` (RS384/ES384) authenticating the FHIR/REST outbound against real SMART-secured
-      servers (Epic, Oracle Health); a `with_smart_backend()` composer over `FHIR()`/`Rest()` that mints,
-      caches, and re-mints (on 401) a short-lived bearer, token endpoint gated by `[egress].allowed_http`
-      ([ADR 0024](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0024-smart-backend-services-token-provider.md))
-- [x] **base64 binary-carriage codec** — an `mfb64:v1:` marker carries arbitrary NUL-safe **bytes** over
-      the str/TEXT ingress + store (`RawMessage.from_bytes()`/`.raw_bytes`/`.binary()`/`.is_binary`), plus
-      HL7 OBX-5 ED (Encapsulated Data) embedding helpers ([ADR 0028](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0028-base64-binary-carriage-codec.md))
-- [x] **DICOM codec + C-STORE SCP** (Phase 1) — a pure codec (routing peek, headers/Structured Report,
-      code-first SR→HL7 mapping helpers; **headers + SR only, no pixel data**) on `content_type=dicom`
-      payload-agnostic ingress, plus an inbound C-STORE SCP listener (`DICOM()`) ([ADR 0025](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0025-dicom-codec-store-connectors.md))
-- [x] **Anonymizer / de-identification** — builds PHI-free test datasets from real traffic with
-      deterministic secret-per-dataset pseudonymization, field-anchored site-code scrub, and fail-closed
-      emission (never an un-scrubbed body), via a `tee anonymize-captures` subcommand and test-harness
-      hooks ([ADR 0030](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0030-anonymization-test-harness-tee.md))
-
-**Later** — higher-throughput delivery (a pooled/persistent MLLP connector); a read-only **component
-SDK** (fork-to-customize); DICOM Phase 2 (C-STORE SCU + C-ECHO + DICOMweb STOW-RS, designed); MFA and
-off-box log shipping. See [docs/EARLY-ADOPTER-GUIDE.md](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/EARLY-ADOPTER-GUIDE.md) §2 for the current
-built-vs-experimental map.
-
-Horizontal **active-active** scale-out (the multi-node cluster path) was **dropped on 2026-06-18 and
-its code removed** — it is not a planned milestone; single-leader **active-passive** HA (above) is the
-supported HA model.
+## Features
+
+MessageFoundry ships a reliable, PHI-aware engine today: the Connection/Router/Handler graph
+over a durable staged pipeline (at-least-once delivery, retries, dead-letter, replay), backed by
+SQLite, PostgreSQL, or SQL Server. It speaks MLLP (plain and over TLS), file/SFTP, REST, SOAP,
+FHIR, database, and DICOM (C-STORE SCP), parses HL7 v2 tolerantly (with opt-in strict validation)
+and carries other formats payload-agnostically (JSON, XML/SOAP, X12, binary). Security is
+first-class — authentication, RBAC, a user-attributed audit log, at-rest body encryption, and
+transport TLS — and it runs single-node or in active-passive high availability.
+
+**See the full, up-to-date feature breakdown — built vs. planned — at
+[messagefoundry.org/features-table.html](https://messagefoundry.org/features-table.html).**
+
+## Documentation
+
+Full documentation lives on **[messagefoundry.org](https://messagefoundry.org/)**:
+
+- **[Mental map](https://messagefoundry.org/assets/docs/MessageFoundry-Mental-Model.pdf)** — a one-page
+  picture of how the pieces fit: Connections → Router → Handlers → Connections, with the headless
+  engine and the console/IDE that drive it.
+- **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)** — install,
+  configure, secure, and roll out to production.
+- **[User Guide](https://messagefoundry.org/assets/docs/MessageFoundry-User-Guide.pdf)** — author and
+  operate interfaces day to day.
+
+**Get started:** [Quickstart](https://messagefoundry.org/getting-started.html) ·
+[Guides](https://messagefoundry.org/guides/) ·
+[Documents](https://messagefoundry.org/documents.html)
 
 ## Installing & rolling out
 
@@ -114,21 +86,23 @@ supported production artifact, with no source checkout required. Install it as a
 dependency**, then scaffold your own config repo ([ADR 0017](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/adr/0017-consumer-deployment-model.md)):
 
 ```bash
-pip install "messagefoundry==0.2.1"   # pin the exact engine version (core runtime, SQLite store)
-messagefoundry init ./my-config-repo     # scaffold a standalone config repo
+pip install "messagefoundry==<version>"   # pin the exact engine version (core runtime, SQLite store)
+messagefoundry init ./my-config-repo      # scaffold a standalone config repo
 cd ./my-config-repo
 messagefoundry serve --config config --env dev
 ```
 
-`0.2.1` is the current **Early Access** release on PyPI. Always **pin the exact version** so
-upgrades stay deliberate. Add the extras your deployment needs (each is opt-in and lazy-imported):
+MessageFoundry is in **Early Access**. Always **pin the exact version** so upgrades stay
+deliberate — replace `<version>` with the current release shown at the top of the
+[PyPI project page](https://pypi.org/project/messagefoundry/). Add the extras your deployment
+needs (each is opt-in and lazy-imported):
 
 ```bash
-pip install "messagefoundry[postgres]==0.2.1"    # PostgreSQL store backend (production server DB)
-pip install "messagefoundry[sqlserver]==0.2.1"   # SQL Server store backend (+ OS-level ODBC Driver 18)
-pip install "messagefoundry[console]==0.2.1"     # PySide6 admin console
-pip install "messagefoundry[sftp]==0.2.1"        # SFTP transport for the REMOTEFILE connector
-pip install "messagefoundry[dicom]==0.2.1"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
+pip install "messagefoundry[postgres]==<version>"    # PostgreSQL store backend (production server DB)
+pip install "messagefoundry[sqlserver]==<version>"   # SQL Server store backend (+ OS-level ODBC Driver 18)
+pip install "messagefoundry[console]==<version>"     # PySide6 admin console
+pip install "messagefoundry[sftp]==<version>"        # SFTP transport for the REMOTEFILE connector
+pip install "messagefoundry[dicom]==<version>"       # DICOM codec + C-STORE SCP (pydicom + pynetdicom)
 ```
 
 > **Verify before you install (supply chain).** Every release is built by a GitHub Actions workflow,
@@ -140,7 +114,7 @@ pip install "messagefoundry[dicom]==0.2.1"       # DICOM codec + C-STORE SCP (py
 >
 > *(Engine developers install from a checkout instead — see [Development](#development).)*
 
-Piloting MessageFoundry? The **[Early-Adopter Installation & Rollout Guide](https://github.com/wshallwshall/MessageFoundry/blob/main/docs/EARLY-ADOPTER-GUIDE.md)**
+Piloting MessageFoundry? The **[Install Guide](https://messagefoundry.org/assets/docs/MessageFoundry-Install-Guide.pdf)**
 takes you from first install through a staged, go/no-go-gated path to full production
 (Lab → Shadow/Parallel → Limited → Full). It leads with an honest built-vs-experimental
 maturity map and covers prerequisites, install, security/PHI hardening, reliability

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/docs/BACKLOG.md

@@ -931,6 +931,10 @@ console-view scope.
 
 **Source:** Corepoint event-log gap analysis (2026-06-17); ADRs 0020 + 0021.
 
+**See also #46** — the complementary *happy-path* connection-state lifecycle log (established / connecting /
+retrying / lost). #16's retained scope is pre-message *failure* events + "Response Sent" ACK; #46 is the
+routine Transport-event transitions. Build them together (one event log) if either is un-deferred.
+
 ---
 
 ## 17. CI: the `py3.11` test leg hangs (pytest deadlock) — OBSOLETE (py3.11/3.13 legs removed)
@@ -1255,6 +1259,14 @@ too. Fold the alert-rule view/test (ADR 0014 rules) into the Alerts page.
 **Why P2:** no new engine work — purely the console surface. Matters for operator parity, but the
 API/CLI already cover the underlying capability.
 
+> **#22c Event Log page (committed, from #46 — do NOT drop).** When the **#46** Corepoint-style event logging
+> lands engine-side ("logging like Corepoint", owner go 2026-06-25), it ships **engine capture first** with the
+> operator-facing **console Event Log page as a committed fast-follow, not optional.** A filterable PySide6 page
+> (by connection / direction / kind / time, Corepoint Transport/Diagnostic/Alert/Misc filter) over
+> `GET /events` + `GET /connections/{name}/events`, plus the Response-Sent ("ACK returned") view off
+> `GET /messages/{id}/responses?kind=ack_sent`. Replaces a `PlaceholderPage`, same pattern as #22a/#22b.
+> Tracked here so the GUI surface is built once the #46 engine API exists. See **#46**.
+
 **Source:** Mirth brochure gap analysis (2026-06-18); FEATURE-MAP §10.
 
 ---
@@ -2127,6 +2139,76 @@ avoid. Build when a private-CA adopter is blocked.
 
 ---
 
+## 46. Connection lifecycle event log — "established / lost / connecting / retrying" (Corepoint Transport-event parity)
+
+> **✅ SCOPED FOR BUILD (owner go, 2026-06-25): "logging like Corepoint."** Build a unified, metadata-only
+> `connection_event` log capturing **inbound lifecycle** (established/closed) + the **ADR 0021 §7 failures**
+> (allowlist/capacity/oversize/peer-reset/framing) + **outbound lifecycle** (connection_lost/restored,
+> edge-triggered — no per-delivery spam), plus **Response Sent ACK/NAK** (ADR 0021 §§1-6, PHI, encrypted).
+> **ON by default** for the no-PHI connection events (master `[diagnostics].connection_events`). Engine
+> capture-first; the **console "Event Log" viewer is a committed fast-follow, NOT optional** — see
+> *Console deliverable* below. Raw protocol trace (ADR 0020) stays declined. Build increments + the two
+> confirm-items are in the 2026-06-25 plan.
+>
+> **Console deliverable (do NOT drop).** The point of "like Corepoint" is the *operator-facing* event log, so
+> the engine increments are not "done" until the **PySide6 console Event Log page** ships: a filterable view
+> (by connection / direction / kind / time) over `GET /events` + `GET /connections/{name}/events`, with the
+> Corepoint-style Transport / Diagnostic / Alert / Misc filter, plus the Response-Sent ("ACK returned") view
+> off `GET /messages/{id}/responses?kind=ack_sent`. This rides the **#22** console-page workstream (its natural
+> home) — tracked there too so it can't fall through the gap between the engine work and the GUI work.
+
+**Type:** feature — operational/diagnostic observability. Closely related to **#16** (Corepoint event-log
+parity) — see *Relationship* below; this is the broader, happy-path slice #16's narrowed scope does **not**
+cover.
+
+**What:** MessageFoundry's technical log does **not** emit the routine per-connection lifecycle play-by-play
+that Corepoint/Mirth surface under their **Transport** event filter — "connection established", "connection
+lost", "trying to connect", "reconnecting". Today the connection layer is silent on the happy path and only
+records the *exceptional* edges:
+- **Inbound (MLLP/TCP listeners):** a successful client accept is **not logged** — only refusals
+  (`source_ip_allowlist`), at-capacity (silent), frame-over-cap, and unexpected per-connection errors are
+  ([`transports/mllp.py`](../messagefoundry/transports/mllp.py) `_serve_client`,
+  [`transports/tcp.py`](../messagefoundry/transports/tcp.py)). When a peer connects and sends normally, the
+  **message** is what's counted/dispositioned in the store — there is no "accepted connection from <peer>" event.
+- **Outbound (delivery):** [`MLLPDestination`](../messagefoundry/transports/mllp.py) opens a **fresh connection
+  per delivery** (connect → send → ACK → close), so there is no persistent connection to "lose" or
+  "reconnect". A connect/IO failure becomes a `DeliveryError` → retry-with-backoff, and **each failed attempt
+  is not written to the technical log** — the detail goes to the store row's `last_error`, surfaced to
+  operators only via the `AlertSink` `queue_buildup` when a lane backs up
+  ([`pipeline/wiring_runner.py`](../messagefoundry/pipeline/wiring_runner.py) delivery loop, the
+  `except DeliveryError` arm). So there is no "trying to connect… refused… retrying" stream.
+
+What the technical log *does* carry at connection level: engine/wiring lifecycle (`wiring started: N inbound,
+M outbound`, `wiring stopped/reloaded`), connection-failed-to-bind (isolated, ADR 0031), worker
+crashed/respawned, STOP-policy halts, and egress/connect allowlist denials.
+
+**Proposed shape (when triggered):** a lightweight **structured connection *event* log — metadata only, no raw
+bytes / no PHI** (peer, direction, connection name, transition, timestamp, reason) recording the lifecycle
+transitions: inbound accept/close, outbound connect-attempt/connected/failed/retry-scheduled, and lane
+stop/resume. Reuse the existing `AlertSink` seam + the planned lightweight connection-error event log from
+#16's narrowed ADR 0020 scope rather than a second mechanism; emit-points are connector lifecycle hooks in
+`transports/` (accept/close on the listeners; connect/send/close on the outbound) plus the delivery-worker
+retry transitions in `wiring_runner.py`. Keep it **off-by-default / metadata-only** so it never reintroduces a
+raw-PHI-at-rest tier (the exact reason #16 dropped ADR 0020's raw-frame capture).
+
+**Relationship to #16:** #16's *retained* slice is **pre-message *failure* events that have no `message_id`**
+(bad framing, TLS-accept failure, peer reset, allowlist refuse) + ADR 0021's "Response Sent" ACK/NAK capture.
+This item is the complementary **happy-path connection-state lifecycle** (established / connecting / retrying /
+lost) — the routine Transport-event transitions a successful connection goes through, which today are silent.
+Build the two together (one event log, two event classes) if either is un-deferred, to avoid a split design.
+
+**Why deferred (on-trigger):** no customer pull yet — internal Corepoint-checklist origin, same posture as
+ADR 0020. Operator visibility for *failures* is already met via per-message disposition + `last_error` + the
+`queue_buildup`/`connection_stopped` alerts; this adds **diagnostic** visibility of normal connection churn,
+valuable mainly to operators migrating from an engine that shows it. Build when an adopter needs Corepoint-style
+connection-event visibility. **Trigger:** a pilot/adopter asks for a connection-state/transport event log.
+
+**Source:** session question 2026-06-25 ("do our connection logs show connection established / lost / trying to
+connect?") — confirmed against `transports/mllp.py`, `transports/tcp.py`, and the `wiring_runner.py` delivery
+loop; relationship to the #16 Corepoint event-log gap analysis (2026-06-17) + ADRs 0020/0021.
+
+---
+
 *Everything else from the 2026-06-10 full-codebase review (1 critical, 13 high, 33 medium, 31 low —
 78 findings) has been remediated; see the review report's §6 action order. The two items it still
 sourced — **#1 (SQL Server concurrency)** and **#2 (console off-thread)** — are now both **DONE**

messagefoundry-0.2.3/docs/CODESETS.md

@@ -0,0 +1,109 @@
+# Code sets — translation tables (operator reference)
+
+A **code set** is read-only reference data a code-first Router/Handler looks up by name — an Epic
+diet code → a food-service value, a facility code → a downstream mnemonic. This is the operator
+reference for **editing** code sets: the VS Code grid editor and the `messagefoundry codeset` CLI it
+shells. For how a Handler *consumes* a code set (`code_set("name")`), the on-disk format, and the
+purity/reload semantics, see [CONFIGURATION.md → "Code sets"](CONFIGURATION.md#code-sets--reference-lookup-tables-codesets).
+The design record is [ADR 0033](adr/0033-gui-manageable-code-sets.md) (a sibling of the connection
+editor, [ADR 0007](adr/0007-gui-manageable-connections-toml.md)).
+
+## Where the files live
+
+Code sets live in `codesets/` **relative to the `--config` dir** — a config bundle carries its own
+reference tables and they **reload with the graph** (`POST /config/reload`). The code-set **name** is
+the file's stem (`codesets/epic_diets.csv` → `"epic_diets"`). A missing `codesets/` dir is fine (no
+code sets); the directory is created on the first `upsert`.
+
+- **CSV** (`<name>.csv`) — the editable, GUI-canonical format. A header row; the **first column is the
+  lookup key**. One other column → the value is that scalar (`str`); several other columns → the value
+  is a `dict` `{header: cell}`. A duplicate key is a **load error** (fail loud).
+- **TOML** (`<name>.toml`) — hand-authored / legacy. Summarized by `list` and shown **read-only** in
+  the grid; the CLI never *writes* TOML. (A TOML-in-grid editor is a fast-follow.)
+
+Both editors — a hand edit and a GUI save — write the same `codesets/<name>.csv`, so they are
+interchangeable.
+
+## The grid editor (VS Code)
+
+The extension opens a **grid** (rows × columns of strings; the first column is the lookup key) to
+**create / edit / rename / delete** a translation table. It never writes a file itself: it shells the
+`messagefoundry codeset` CLI, which owns all validation and the atomic write. A CLI error comes back
+inline and the grid stays open (file unchanged) so you can fix it. When the opened set is a `.toml`
+file the grid is **read-only** (Save disabled).
+
+## The `messagefoundry codeset` CLI
+
+The CLI is **offline** (no engine start, no egress check — a code set is standalone data) and
+validates against the **same loader** that runs at startup. Under `--json` it prints a single-line
+JSON result on success, or `{"error": "<message>"}` (exit 1) on failure, so the IDE can surface the
+loader's own wording inline.
+
+| Command | What it does |
+|---|---|
+| `messagefoundry codeset list --config DIR` | Summarize every set under `codesets/` (`.csv` **and** `.toml`), sorted by name. |
+| `messagefoundry codeset show --config DIR --name N` | The grid for set `N` (headers + rows); `format:"toml"` ⇒ read-only. |
+| `messagefoundry codeset upsert --config DIR [--data JSON]` | Validate → write `codesets/N.csv` atomically (temp + replace, owner-only perms) → **re-load the written file as the final check**; a bad save rolls back. DETAIL JSON comes from `--data` or stdin. |
+| `messagefoundry codeset rename --config DIR --name N --to M` | Atomic `os.replace` of `codesets/N.<ext>` → `codesets/M.<ext>`; rejects a stem collision. |
+| `messagefoundry codeset remove --config DIR --name N` | Delete `codesets/N.csv` (else `.toml`). |
+
+Add `--json` to any command for machine-readable output.
+
+### DETAIL shape (for `upsert` / returned by `show`)
+
+```json
+{
+  "name": "epic_diets",
+  "format": "csv",
+  "columns": ["code", "value"],
+  "rows": [["A", "Apple"], ["B", "Banana"]]
+}
+```
+
+`rows` is an **array of arrays** (`string[][]`), each row positionally aligned to `columns`; the
+first column is the key. One value column ⇒ a scalar value, two or more ⇒ a `{header: cell}` dict —
+exactly the loader's rule.
+
+### The operator-supplied name is untrusted
+
+`upsert`'s `name`, `rename`'s `--to`, **and** the `--name` on `show`/`remove` are all treated as
+**untrusted data** (CLAUDE.md §5/§8). The CLI rejects a name that contains a path separator, `..`, an
+absolute / drive-prefixed path, or an embedded `.csv`/`.toml` extension, and applies a final
+`resolve()` check that the target stays inside `codesets/` — so a name can never read or write a file
+outside the code-sets directory.
+
+### Validation rules (mirror the loader exactly)
+
+A bad `upsert` is rejected **before** any file is touched: a non-empty key column plus at least one
+value column, unique non-empty headers, all-string cells, no row longer than `columns`, a fully-blank
+row dropped while a blank-key row that carries data is rejected (fail loud — never silently dropped),
+and no duplicate key. A stem that collides with an existing
+`.toml` is rejected (the same ambiguity the loader fails loud on). After the write, the file is
+**re-loaded** as the final authority; any failure rolls the prior content back (or unlinks a
+brand-new file), so a bad edit never lands.
+
+## Promote to apply — and the rename/remove caveat
+
+Editing a `codesets/` file changes nothing live; the running graph adopts the change only through the
+**existing audited `POST /config/reload`** (the IDE promote), exactly like a connection or handler
+change.
+
+**Renaming or removing a code set can break a handler reference.** A code set is referenced by **name**
+from a Handler (`code_set("epic_diets")`), and that reference is resolved at the loader/runner — not by
+`messagefoundry codeset`. So after a rename/remove, a `code_set("old_name")` call raises at run time
+(that message's `ERROR` disposition). A plain `validate` only confirms each *file* parses, so it
+**won't** catch a now-dangling reference.
+
+> **After a rename or remove, run `messagefoundry check` before promoting.** Its dry-run actually
+> executes the transforms, so it triggers the lookup and surfaces a broken `code_set(...)` reference at
+> the gate rather than at the first message in production. The editor deliberately does **not** scan
+> handler source for references (that would couple a data tool to code parsing and still miss a
+> call-time-computed name) — the `messagefoundry check` dry-run is the safety net.
+
+## Out of scope
+
+**Reference sets** ([ADR 0006](adr/0006-external-data-lookups.md)) and **`db_lookup`**
+([ADR 0010](adr/0010-handler-callable-db-lookup.md)) are **not** editable here: their data is
+externally owned (synced on a cadence, or a live database read), so editing it in this grid would be
+overwritten by the next sync/query and is therefore meaningless. This editor governs only the
+bundle-shipped, operator-owned code sets.

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/docs/CONFIGURATION.md

@@ -188,6 +188,34 @@ system value, a facility code → a downstream mnemonic. Rather than a hand-main
   crash-re-run can make the re-run derive a different output. That's acceptable for reference data (a
   code set is deliberately operator-editable, and a reload is an explicit, audited act), but it is the
   one way a transform's re-run can legitimately differ — note it where you document the transform.
+- **Editing — by hand or from the IDE.** A code set is a plain `codesets/<name>.csv` you can edit in any
+  editor, **and** a GUI-manageable artifact ([ADR 0033](adr/0033-gui-manageable-code-sets.md)). The VS
+  Code extension opens a **grid editor** (rows × columns of strings — the first column is the lookup
+  key) to **create / edit / rename / delete** a translation table; it shells a new
+  **`messagefoundry codeset`** CLI that owns validation and the atomic write. Both editors write the
+  same file (CSV-first), so a hand edit and a GUI save are interchangeable — mirroring the connections
+  editor ([ADR 0007](adr/0007-gui-manageable-connections-toml.md)).
+  - `messagefoundry codeset list  --config DIR` — summarize every set under `codesets/` (`.csv` **and**
+    `.toml`; TOML sets are summarized and shown **read-only** in the grid — TOML-in-grid editing is a
+    fast-follow).
+  - `messagefoundry codeset show   --config DIR --name N` — the grid (headers + rows).
+  - `messagefoundry codeset upsert --config DIR --data '{…}'` — validate → write `codesets/N.csv`
+    atomically (temp + replace, owner-only perms) → **re-load the written file as the final check**;
+    a bad save rolls back, so the CLI never leaves an unloadable table.
+  - `messagefoundry codeset rename --config DIR --name N --to M` / `… remove --config DIR --name N`.
+
+  The CLI is **offline** (no engine start, no egress check — a code set is standalone data); it validates
+  against the **same loader** that runs at startup, and the operator-supplied **name is treated as
+  untrusted data** (rejecting path separators, `..`, absolute/drive paths, and an embedded extension, so
+  a name can't escape `codesets/`). Apply a change with the existing audited promote/reload below.
+- **Promote to apply (rename/remove caveat).** Editing a `codesets/` file changes nothing live; the
+  running graph adopts the change only through **`POST /config/reload`** (the IDE promote), exactly like
+  a connection or handler change. **Renaming or removing a code set can break a handler reference** — a
+  `code_set("old_name")` call then raises at run time (that message's `ERROR` disposition). A plain
+  `validate` only confirms each file parses, so it **won't** catch a now-dangling reference; **run
+  `messagefoundry check` after a rename/remove**, whose dry-run executes the transforms and surfaces the
+  broken `code_set(...)` lookup before you promote. See [docs/CODESETS.md](CODESETS.md) for the full grid
+  editor + CLI reference.
 
 ### Transform state — cross-message correlation ([ADR 0005](adr/0005-transform-accessible-state.md))
 

{messagefoundry-0.2.2 → messagefoundry-0.2.3}/docs/CONNECTIONS.md

@@ -200,6 +200,16 @@ or leave empty for no restriction.
 > scoped to one listener; the same off-loopback risk applies), and **`source_ip_allowlist`** restricts
 > which peers that listener accepts. See [docs/CONFIGURATION.md](CONFIGURATION.md).
 
+> **Port-conflict detection.** Two inbound listeners that bind the **same port on overlapping
+> interfaces** are caught **statically** — at `messagefoundry check` / dry-run and at engine
+> start/reload — naming **both** connections, instead of aborting at the bare OS bind. The check is
+> **interface-aware**: two listeners on the same port but **different** explicit `bind_address`es (a
+> multi-NIC host) don't conflict, while a `0.0.0.0` (all-interfaces) bind conflicts with any specific
+> interface on that port. `env()`-resolved ports and the engine's own **API listener port** (`[api].port`)
+> are included in the start/reload pass. At runtime, a port already held by **another process** (a
+> second instance, an OS service) is reported as a clear, named conflict and the affected inbound is
+> **isolated** (the engine still comes up; see [ADR 0031](adr/0031-startup-connection-fault-isolation.md)).
+
 #### Inspecting & testing a connection (API)
 
 Two read/diagnostic endpoints back the console's connection view (auth + per-channel RBAC apply — see