messagefoundry-harness 0.2.4__py3-none-any.whl

harness/README.md

@@ -0,0 +1,101 @@
+# Test harness
+
+A standalone PySide6 tool to exercise **everything the engine can do** with synthetic, PHI-free
+traffic — send and receive HL7 v2 over MLLP and file, send malformed messages, inject delivery
+faults, and watch what a running engine actually did with each message.
+
+```powershell
+python -m harness                      # launch the GUI
+python -m harness --list-scenarios     # list headless scenarios
+python -m harness --scenario processed # run one scenario in CI (exit 0 pass / 1 fail)
+python -m harness --list-profiles      # list headless load profiles
+python -m harness --load smoke         # run a load profile (exit 0 SLOs met / 1 violation / 2 setup)
+```
+
+It reuses the engine's own MLLP framing + ACK builder (`messagefoundry.transports.mllp`), message
+generators (`messagefoundry/generators`), and API client (`messagefoundry.console.client`), so it
+frames, acknowledges, and reads engine state exactly as the real components do. New message types
+light up automatically as they're added to `messagefoundry/generators/all_types.py`.
+
+## GUI tabs
+
+- **Send** — pick a message type/trigger (or "random across all"), a count, target `host:port`,
+  and an optional rate; fire them and watch per-message ACK code / latency / errors. Sending runs
+  on a worker thread so the UI stays responsive.
+- **Receive** — start a localhost MLLP listener and reply with a configurable mode. Beyond
+  **AA / AE / AR / none**, the reply can inject faults to drive the engine's *outbound* retry /
+  dead-letter / independent-draining behavior: **delay then AA** (set the delay past the engine's
+  timeout to force a retry), **close (no reply)**, and **fail N then AA** (reject the first N
+  deliveries of a control id, then accept). Repeated control ids — the engine's at-least-once
+  retries — are counted and highlighted.
+- **File** — *Drop* generated messages into the engine's File-inbound directory (atomic writes,
+  so the engine never polls a half-written file), and *Watch* its File-outbound directory, parsing
+  and displaying each file that appears. Defaults match `harness/config`.
+- **Compose** — send an arbitrary, hand-edited message (preset seeds: valid, no-MSH,
+  wrong-version) over MLLP with an explicit **ACK expectation** (Accept / Reject / No ACK), flagged
+  against the actual reply — or drop it as a file. This reaches the ERROR / AR / AE / strict-
+  validation paths the generators can't.
+- **Monitor** — connect to a running engine's API (reusing the console's sign-in) and observe what
+  it did: live outbox stats + a connections table (polled off the UI thread), the message store
+  with per-message disposition and full delivery/audit trail, the dead-letter queue with scoped +
+  bulk replay, and a config-reload button. Connection start/stop/restart/purge controls included.
+
+## Driving a complete engine
+
+`harness/config` is a self-contained config graph wired to produce **every** disposition
+and delivery path (see its docstring). Serve it, then drive it from the tabs above:
+
+```powershell
+python -m messagefoundry serve --config harness/config --db ./messagefoundry.db --env dev
+```
+
+| Send (Send/Compose tab → 127.0.0.1:2575) | Disposition (Monitor tab) |
+|------------------------------------------|---------------------------|
+| ADT^A01 / A04 / A08                      | PROCESSED (fan-out: MLLP echo + file) |
+| any other ADT trigger                    | PROCESSED (file archive) |
+| ADT^A02                                  | FILTERED |
+| ADT^A03                                  | ERROR (AE NAK) |
+| any non-ADT type                         | UNROUTED |
+| malformed / wrong version (→ 2577)       | ERROR (AE NAK) |
+
+To see retries → dead-letter → replay: leave the Receive tab **not** listening on 2576 (or set it
+to *fail N then AA* / *close*), send an ADT^A01, and watch the echo delivery dead-letter in the
+Monitor's Dead Letters tab — the file archive for the same message still succeeds (independent
+draining). Then replay it from there.
+
+## Headless scenarios (CI)
+
+The scenario runner generates traffic, sends it, and asserts the engine's resulting disposition
+(or dead-lettering) over the API — Qt-free, so it runs on a display-less runner. Built-in
+scenarios target `harness/config`; serve it, then:
+
+```powershell
+python -m harness --scenario processed   # ADT^A05 → file → PROCESSED
+python -m harness --scenario filtered     # ADT^A02 → FILTERED
+python -m harness --scenario unrouted      # ORU → UNROUTED
+python -m harness --scenario error         # ADT^A03 → ERROR
+python -m harness --scenario dead_letter   # ADT^A01 echo with nothing on 2576
+```
+
+Pass `--engine <url>` for a non-default API address and `--token <t>` for an auth-enabled engine.
+
+## Load testing (headless)
+
+A separate, **Qt-free** asyncio load engine (`harness/load/`) drives the engine under heavy MLLP
+traffic and measures it — the GUI's single-thread sender can't saturate it. A pool of **persistent,
+pipelined** connections offers a data-driven [load profile](load/profiles/) (warmup → ramp →
+sustained → spike → soak); a fast **correlation sink** absorbs the engine's outbound fan-out and
+times every message end-to-end; an engine poller samples the API for throughput, backlog, DB growth,
+and post-load drain. The run ends in an SLO verdict + a no-loss reconciliation and a JSON/CSV report.
+
+Serve the synthetic high-fan-out [system-under-test](config/load/) (separate from `harness/config`),
+then run a profile:
+
+```powershell
+$env:MEFOR_LOAD_FANOUT=20; $env:MEFOR_LOAD_TRANSFORM="edit"; $env:MEFOR_LOAD_SINK_PORT=2700
+python -m messagefoundry serve --config harness/config/load --db ./load.db   # swap --db for backends
+python -m harness --load fanout-baseline --engine URL --token T --report-json out/load/run.json
+```
+
+Full guide — profile schema, the env knobs, reading the report/SLOs, exit codes, baseline
+comparison, and the backend-comparison recipe — is in [docs/LOAD-TESTING.md](../docs/LOAD-TESTING.md).

harness/__main__.py

@@ -0,0 +1,293 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""Test harness entrypoint.
+
+``python -m harness``                 → launch the GUI (Send/Receive/File/Compose/Monitor).
+``python -m harness --list-scenarios``→ list the built-in scenarios.
+``python -m harness --scenario NAME`` → run one scenario headless against a running engine
+                                                and exit 0 (pass) / 1 (fail) — for CI.
+``python -m harness --list-profiles`` → list the built-in load profiles.
+``python -m harness --load NAME``     → run a load profile headless against a running engine and exit
+                                                0 (SLOs met) / 1 (SLO violation, incl. zero_loss when
+                                                the profile sets it, or a baseline regression) / 2
+                                                (setup error) / 3 (interrupted).
+``python -m harness --failover NAME`` → run the two-node primary-kill scenario (the profile must carry a
+                                                [load.failover] table) against a shared server DB
+                                                (MEFOR_STORE_BACKEND=postgres|sqlserver + MEFOR_STORE_*);
+                                                the harness OWNS the two engines, SIGKILLs the primary
+                                                mid-load, and exits 0 (recovery + no-loss + ordering met)
+                                                / 1 (an SLO violated) / 2 (setup) / 3 (interrupted).
+
+The headless paths import no PySide6, so they run on a display-less runner; the GUI import is
+deferred into :func:`_launch_gui`.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+
+def main(argv: list[str] | None = None) -> int:
+    # Scenario text uses arrows (U+2192); a legacy Windows console (cp1252) would otherwise raise
+    # UnicodeEncodeError when --list-scenarios / --scenario prints them, breaking the documented
+    # CI use. Force UTF-8 on the CLI streams — best-effort, since a pytest/redirect wrapper may
+    # not support reconfigure.
+    for _stream in (sys.stdout, sys.stderr):
+        try:
+            _stream.reconfigure(encoding="utf-8", errors="replace")  # type: ignore[attr-defined]
+        except (AttributeError, ValueError, OSError):
+            pass
+
+    parser = argparse.ArgumentParser(prog="harness", description="MessageFoundry test harness")
+    parser.add_argument(
+        "--scenario", help="run this scenario headless and exit (see --list-scenarios)"
+    )
+    parser.add_argument("--list-scenarios", action="store_true", help="list built-in scenarios")
+    parser.add_argument(
+        "--load", help="run this load profile (built-in name or path to a .toml) and exit"
+    )
+    parser.add_argument(
+        "--failover",
+        help="run this profile's two-node primary-kill scenario (needs a [load.failover] table + a "
+        "shared server DB via MEFOR_STORE_*) and exit",
+    )
+    parser.add_argument(
+        "--inbound-base-port",
+        type=int,
+        default=2600,
+        help="failover: base inbound MLLP port (ADT hub; results/other = base+1/+2). Both nodes bind it",
+    )
+    parser.add_argument("--list-profiles", action="store_true", help="list built-in load profiles")
+    parser.add_argument("--engine", default="http://127.0.0.1:8765", help="engine API base URL")
+    parser.add_argument("--token", help="bearer token for an auth-enabled engine")
+    parser.add_argument(
+        "--timeout", type=float, default=30.0, help="scenario: seconds to wait for the outcome"
+    )
+    # Load-run options.
+    parser.add_argument(
+        "--sink-port", type=int, default=2700, help="load: base correlation-sink port"
+    )
+    parser.add_argument("--sink-ports", type=int, default=1, help="load: contiguous sink ports")
+    parser.add_argument("--report-json", help="load: write the JSON report to this path")
+    parser.add_argument("--report-csv", help="load: write the per-phase CSV to this path")
+    parser.add_argument("--baseline", help="load: compare against this saved JSON report")
+    parser.add_argument(
+        "--tolerance",
+        type=float,
+        default=0.1,
+        help="load: baseline regression tolerance (fraction)",
+    )
+    parser.add_argument("--db-backend", help="load: label the store backend in the report")
+    args = parser.parse_args(argv)
+
+    if args.list_scenarios:
+        return _list_scenarios()
+    if args.list_profiles:
+        return _list_profiles()
+    if sum(bool(x) for x in (args.load, args.scenario, args.failover)) > 1:
+        print("--load, --failover, and --scenario are mutually exclusive", file=sys.stderr)
+        return 2
+    if args.failover:
+        return _run_failover(args)
+    if args.load:
+        return _run_load(args)
+    if args.scenario:
+        return _run_scenario(args.scenario, args.engine, args.token, args.timeout)
+    return _launch_gui()
+
+
+def _list_scenarios() -> int:
+    from harness.scenarios import SCENARIOS
+
+    for name, scenario in SCENARIOS.items():
+        print(f"  {name:<12} {scenario.description}")
+    return 0
+
+
+def _run_scenario(name: str, engine_url: str, token: str | None, timeout: float) -> int:
+    from messagefoundry.console.client import ApiError, EngineClient
+    from harness.scenarios import SCENARIOS, run_scenario
+
+    scenario = SCENARIOS.get(name)
+    if scenario is None:
+        print(f"unknown scenario {name!r}; choices: {', '.join(SCENARIOS)}", file=sys.stderr)
+        return 2
+    try:
+        with EngineClient(engine_url) as client:
+            if token:
+                client.set_token(token)
+            result = run_scenario(scenario, client, timeout=timeout)
+    except ApiError as exc:
+        print(f"FAIL  {name}: {exc}", file=sys.stderr)
+        return 1
+    print(f"{'PASS' if result.ok else 'FAIL'}  {name}: {result.detail}")
+    return 0 if result.ok else 1
+
+
+def _list_profiles() -> int:
+    from harness.load.profile import list_profiles
+
+    for name, description in list_profiles().items():
+        print(f"  {name:<18} {description}")
+    return 0
+
+
+def _run_load(args: argparse.Namespace) -> int:
+    import asyncio
+    import json
+    import os
+    import time
+    from pathlib import Path
+
+    from messagefoundry.console.client import ApiError
+
+    from harness.load.profile import LoadProfileError, get_profile
+    from harness.load.report import compare_to_baseline
+    from harness.load.runner import PreflightError, run_load
+
+    try:
+        profile = get_profile(args.load)
+    except LoadProfileError as exc:
+        print(f"bad profile: {exc}", file=sys.stderr)
+        return 2
+
+    # A run-scoped, ASCII-alnum control-id prefix so a re-run can't collide with a prior run's ids in
+    # a long-lived DB. (pid + monotonic ns; no wall clock needed.)
+    prefix = f"L{os.getpid():x}{time.perf_counter_ns():x}"[:16]
+
+    try:
+        report = asyncio.run(
+            run_load(
+                profile,
+                engine_url=args.engine,
+                id_prefix=prefix,
+                token=args.token,
+                sink_port=args.sink_port,
+                sink_ports=args.sink_ports,
+                db_backend=args.db_backend,
+            )
+        )
+    except PreflightError as exc:
+        print(f"preflight failed: {exc}", file=sys.stderr)
+        return 2
+    except ApiError as exc:
+        # A bad/expired --token or an engine that's down surfaces here (the client validates the token
+        # via /auth/me before preflight). That's a setup failure, not an SLO violation → exit 2.
+        print(f"engine setup failed: {exc}", file=sys.stderr)
+        return 2
+    except KeyboardInterrupt:
+        print("interrupted", file=sys.stderr)
+        return 3
+
+    print(report.render_console())
+
+    if args.report_json:
+        Path(args.report_json).write_text(report.to_json(), encoding="utf-8")
+    if args.report_csv:
+        Path(args.report_csv).write_text(report.to_csv(), encoding="utf-8")
+
+    exit_code = report.exit_code
+    if args.baseline:
+        try:
+            baseline = json.loads(Path(args.baseline).read_text(encoding="utf-8"))
+        except (OSError, ValueError) as exc:
+            print(f"could not read baseline {args.baseline!r}: {exc}", file=sys.stderr)
+            return 2
+        regressions = compare_to_baseline(report.to_json_dict(), baseline, tolerance=args.tolerance)
+        if regressions:
+            print("\nBASELINE REGRESSIONS:", file=sys.stderr)
+            for r in regressions:
+                print(f"  - {r}", file=sys.stderr)
+            exit_code = exit_code or 1
+    return exit_code
+
+
+def _run_failover(args: argparse.Namespace) -> int:
+    import asyncio
+    import json
+    import os
+    import socket
+    from pathlib import Path
+
+    from harness.load.failover import FailoverError, FailoverPorts, run_failover_load
+    from harness.load.profile import LoadProfileError, get_profile
+
+    try:
+        profile = get_profile(args.failover)
+    except LoadProfileError as exc:
+        print(f"bad profile: {exc}", file=sys.stderr)
+        return 2
+    if profile.failover is None:
+        print(
+            f"profile {profile.name!r} has no [load.failover] table — not a failover profile",
+            file=sys.stderr,
+        )
+        return 2
+    # A failover needs a SHARED server DB (SQLite is single-file/single-node — it can't cluster).
+    backend = os.environ.get("MEFOR_STORE_BACKEND", "").strip().lower()
+    if backend not in ("postgres", "sqlserver"):
+        print(
+            "failover needs a shared server DB: set MEFOR_STORE_BACKEND=postgres|sqlserver (+ the "
+            f"MEFOR_STORE_* connection env); got {backend or '(unset → sqlite)'!r}",
+            file=sys.stderr,
+        )
+        return 2
+
+    def _two_free_ports() -> tuple[int, int]:
+        # Hold BOTH sockets open while reading their ports so the kernel can't hand the same ephemeral
+        # port back to the second bind (the close->rebind race that would launch both nodes on one --port).
+        s1, s2 = socket.socket(), socket.socket()
+        for s in (s1, s2):
+            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+            s.bind(("127.0.0.1", 0))
+        try:
+            return int(s1.getsockname()[1]), int(s2.getsockname()[1])
+        finally:
+            s1.close()
+            s2.close()
+
+    base = args.inbound_base_port
+    api_a, api_b = _two_free_ports()
+    ports = FailoverPorts(
+        inbound_adt=base,
+        inbound_results=base + 1,
+        inbound_other=base + 2,
+        sink=args.sink_port,
+        sink_count=args.sink_ports,
+        api_a=api_a,
+        api_b=api_b,
+    )
+    try:
+        report = asyncio.run(
+            run_failover_load(profile, ports=ports, db_backend=args.db_backend or backend)
+        )
+    except FailoverError as exc:
+        print(f"failover setup failed: {exc}", file=sys.stderr)
+        return 2
+    except KeyboardInterrupt:
+        print("interrupted", file=sys.stderr)
+        return 3
+
+    print(report.render_console())
+    if args.report_json:
+        Path(args.report_json).write_text(
+            json.dumps(report.to_json_dict(), indent=2), encoding="utf-8"
+        )
+    return report.exit_code
+
+
+def _launch_gui() -> int:
+    from PySide6.QtWidgets import QApplication
+
+    from harness.window import HarnessWindow
+
+    app = QApplication(sys.argv)
+    window = HarnessWindow()
+    window.resize(1100, 750)
+    window.show()
+    return app.exec()
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

harness/acceptance/__init__.py

@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""Windows Server 2025 on-server acceptance harness.
+
+Turns ``docs/testing/WIN2025-TEST-MATRIX.md`` into an executable runner: each matrix row is bound to
+a *probe* (a live environment/host check), a set of *pytest node ids* (the existing suites that
+already assert that row, run against whatever backends ``MEFOR_*`` env makes reachable), a *harness*
+command, or a *manual* step. The runner executes what it can, leaves the rest clearly marked MANUAL
+(never faked green), and emits a PASS/FAIL/SKIP/MANUAL report — optionally writing the verdict back
+into the matrix spreadsheet's Status column.
+
+Run it with ``python -m harness.acceptance`` (see ``--help``). It imports no PySide6 on the headless
+path, so it runs on the server over a remote session as well as the console host.
+"""
+
+from __future__ import annotations
+
+from harness.acceptance.matrix import MATRIX, Coverage, MatrixRow, Status
+from harness.acceptance.runner import RowResult, run_matrix
+
+__all__ = ["MATRIX", "Coverage", "MatrixRow", "Status", "RowResult", "run_matrix"]

harness/acceptance/__main__.py

@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# Copyright (C) 2026 MessageFoundry Organization and contributors
+"""CLI for the WIN2025 acceptance harness.
+
+``python -m harness.acceptance``                       → run probes + the backing pytest suites, print
+                                                          a per-row summary, exit 0 (no FAIL/ERROR) / 1.
+``python -m harness.acceptance --no-pytest``           → probes only (fast host check; no suite run).
+``python -m harness.acceptance --section A,B``         → run only those matrix sections.
+``python -m harness.acceptance --report-md r.md --report-csv r.csv``
+                                                       → also write Markdown / CSV reports.
+``python -m harness.acceptance --xlsx WIN2025-TEST-MATRIX.xlsx``
+                                                       → write each verdict back into the workbook's
+                                                          Status column (needs openpyxl).
+
+Exit codes: 0 = no FAIL/ERROR (MANUAL/SKIP are not failures), 1 = at least one FAIL/ERROR, 2 = bad
+usage / write error.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from harness.acceptance.matrix import SECTIONS
+from harness.acceptance.report import (
+    exit_code,
+    render_console,
+    render_csv,
+    render_markdown,
+    write_xlsx_status,
+)
+from harness.acceptance.runner import run_matrix
+
+
+def main(argv: list[str] | None = None) -> int:
+    for _stream in (sys.stdout, sys.stderr):
+        try:
+            _stream.reconfigure(encoding="utf-8", errors="replace")  # type: ignore[attr-defined]
+        except (AttributeError, ValueError, OSError):
+            pass
+
+    parser = argparse.ArgumentParser(
+        prog="harness.acceptance",
+        description="Run the Windows Server 2025 on-server acceptance matrix.",
+    )
+    parser.add_argument(
+        "--no-pytest",
+        action="store_true",
+        help="run probes only; skip the backing pytest suites (PYTEST rows report SKIP)",
+    )
+    parser.add_argument(
+        "--section", help="comma-separated section letters to run (e.g. A,B,C); default = all"
+    )
+    parser.add_argument("--report-md", help="write the Markdown report to this path")
+    parser.add_argument("--report-csv", help="write the CSV report to this path")
+    parser.add_argument(
+        "--xlsx", help="write each verdict back into this matrix workbook's Status column"
+    )
+    args = parser.parse_args(argv)
+
+    include_sections = None
+    if args.section:
+        include_sections = [s.strip().upper() for s in args.section.split(",") if s.strip()]
+        unknown = [s for s in include_sections if s not in SECTIONS]
+        if unknown:
+            print(
+                f"unknown section(s): {', '.join(unknown)}; choices: {', '.join(SECTIONS)}",
+                file=sys.stderr,
+            )
+            return 2
+
+    results = run_matrix(run_pytest=not args.no_pytest, include_sections=include_sections)
+
+    print(render_console(results))
+
+    if args.report_md:
+        Path(args.report_md).write_text(render_markdown(results), encoding="utf-8")
+    if args.report_csv:
+        Path(args.report_csv).write_text(render_csv(results), encoding="utf-8")
+    if args.xlsx:
+        try:
+            n = write_xlsx_status(results, Path(args.xlsx))
+        except (RuntimeError, OSError) as exc:
+            print(f"xlsx write-back failed: {exc}", file=sys.stderr)
+            return 2
+        print(f"  wrote {n} verdict(s) into {args.xlsx}", file=sys.stderr)
+
+    return exit_code(results)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())