gea-program 0.15.0__tar.gz → 0.16.0__tar.gz

{gea_program-0.15.0 → gea_program-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gea-program
-Version: 0.15.0
+Version: 0.16.0
 Summary: GEA-Program: downhole gauge monitoring - canonical measurement records, quality rules, drift evaluation with a re-fit change log, well-test validation, alarm management, model cards, store-and-forward, monthly SLA measurement, FAT/SAT protocols and a dashboard, generated from live objects in the client's reporting outline.
 Author: Daniel T. Murphy
 License: MPL-2.0
@@ -72,7 +72,7 @@ reaches a client document. Nothing unmeasured is ever reported as met.
   rate-controlled replay, duplicate suppression, per-record latency.
 - **Configuration versioning, SBOM, monthly SLA, FAT/SAT**
   (`config_versioning.py`, `sbom.py`, `sla_report.py`, `fat_sat.py`).
-- **Live protocol ports** (`live_ports.py`, `wits0.py`, `witsml.py`, `opcua_port.py`,
+- **Live protocol ports** (`live_ports.py`, `wits0.py`, `witsml.py`, `etp.py`, `opcua_port.py`,
   `mqtt_port.py`, `modbus.py`): WITS Level 0 from the drill floor (TCP connect,
   TCP listen, serial), WITSML 1.4.1 stores (read-only, polled), OPC UA (read and
   subscribe), MQTT (number, JSON, Sparkplug B), Modbus TCP; every message becomes
@@ -125,7 +125,7 @@ gea dashboard --catalog-well volve_f12_f14_production_excerpt:15/9-F-12:10000 --
 gea client-report --report accuracy --out client_report
 gea model-cards --out model_cards
 gea sbom --out sbom
-gea accept                     # the product gate (333 checks)
+gea accept                     # the product gate (337 checks)
 gea help drift                 # the help library, by the job (16 pages; the same text is on every dashboard page)
 gea guide                      # the click-by-click tester guide (docs/TESTER_GUIDE.md)
 gea gui                        # the desktop window (pip install "gea-program[desktop]")
@@ -143,7 +143,7 @@ gea swaps / gea certificates / gea notify / gea housekeeping / gea loadtest   #
 
 ## Live data
 
-The engine reads live data through five protocol ports. All are **read-only**,
+The engine reads live data through six protocol ports. All are **read-only**,
 all map only what the site declares (a tag map the client owns; an empty map
 is declined), and all write every received message to a JSON-lines recording
 that `--replay` turns back into records without a connection - which is how a
@@ -156,6 +156,9 @@ gea wits0-sim --port 5001                             # a WITS0 sender to rehear
 gea wits0  --config wits0.json --seconds 60 --record floor/session.jsonl --out floor --stream-csv floor/stream.csv   # in the first window, while the sender runs
 gea witsml --write-example-config witsml.json         # a WITSML 1.4.1 store: url, uids, mnemonics -> tags; credentials by env name
 gea witsml --config witsml.json --seconds 60 --out store                 # needs a real store: the example url is a placeholder until you edit it
+gea etp    --write-example-config etp.json            # a WITSML 2.x store over ETP v1.2: ws:// or wss:// url, bearer token by env name, channel URIs -> tags, units
+gea etp-sim --port 9800                               # an ETP store with two channels to rehearse with - run it in a SECOND window and leave it running
+gea etp    --config etp.json --seconds 60 --record rig/session.jsonl --out rig   # subscribes; the store pushes every value as it is written
 gea opcua  --write-example-config opcua.json          # edit: endpoint, security, credential env names, nodes
 gea opcua  --config opcua.json --seconds 60 --record opc/session.jsonl --out opc --stream-csv opc/stream.csv   # needs a real OPC UA server, likewise
 gea mqtt   --write-example-config mqtt.json           # edit: broker, TLS, credential env names, topics
@@ -167,6 +170,7 @@ gea ingest --file floor/stream.csv                    # a stream CSV feeds the h
 |---|---|---|---|
 | `wits0` | WITS Level 0 frames (`&&` ... `!!`, four-digit item codes) over TCP connect, TCP listen or serial; record-01 dictionary built in | numeric GOOD; sentinels (-9999, -999.25) and non-numeric -> GAP with the reason | items 0105/0106 (date, time) when present, else arrival; ingest at arrival |
 | `witsml` | WITSML 1.4.1 SOAP store: GetVersion, GetCap, GetFromStore on one log object, rows newer than the last seen at each poll | numeric GOOD; the store's nullValue and empty fields -> GAP | the index curve of a time log; arrival for a depth log |
+| `etp` | Energistics ETP v1.2 over WebSocket (`etp12.energistics.org`, Avro binary to the published schemas): RequestSession/OpenSession, GetChannelMetadata for the declared URIs, SubscribeChannels, ChannelData pushed by the store (Protocol 21) | numeric GOOD; a non-numeric or null value -> BAD with the reason; a channel the map does not name is counted unmapped | the store's own index: an ETP DateTime index is microseconds since the epoch; an ElapsedTime index is seconds from its declared start; a depth index leaves the arrival time standing, named |
 | `opcua` | opc.tcp, read + subscribe; Basic256Sha256 SignAndEncrypt when a certificate and key are given | StatusCode severity bits: Good -> GOOD, Uncertain -> STALE, Bad -> GAP (value withheld) | source timestamp, else server timestamp, else arrival |
 | `mqtt` | MQTT v5 / v3.1.1, TLS optional, QoS per topic; `+`/`#` wildcards | number: GOOD or GAP; JSON: value/time/quality paths; Sparkplug B: `is_null` -> GAP, `Quality` != 192 -> STALE | JSON time path, Sparkplug metric timestamp, else arrival |
 | `modbus_g6` | Modbus TCP, user-supplied register map with a citation | read failures -> GAP | arrival |
@@ -619,6 +623,29 @@ promoted; the group and the support are declarations; a matched frequency
 is not a fault size. CSV or miniSEED/SAC; `gea vibration` runs the labelled
 pump; `gea help vibration`.
 
+### WITSML 2.x over ETP
+
+The newer rigs and stores do not poll: Energistics ETP v1.2 is a WebSocket
+session, Avro-encoded to the published schemas, on which the customer names
+the channels it wants and the store pushes every value as it is written.
+`gea/etp.py` is that customer: the handshake with the subprotocol, the
+binary encoding and the payload limits; RequestSession for Discovery, Store
+and ChannelSubscribe and what the store answered; the channel metadata for
+the URIs the mapping declares (a URI the store does not have is named as
+missing); the subscription; and each ChannelData DataItem as a record under
+the mapping's tag and unit, with the store's own index as the source time -
+microseconds since the epoch for a DateTime index, seconds from the declared
+start for ElapsedTime, the arrival time standing (and named) for a depth
+index - and the arrival as the ingest time, so the latency is per record.
+Client message ids even, the store's odd; a ProtocolException carried with
+its code; a client or a store without the subprotocol refused at the
+handshake. The patch supervisor runs it like every other port; the page
+offers it; `gea etp-sim` is a store with two channels of a synthetic
+disposal well; `gea etp --selftest` runs the codec and a loopback session;
+`gea help etp`. The doctor now also names a panel started through the
+`gea.exe` launcher, which pip cannot replace while it runs - stop, update,
+start with `python -m gea serve`.
+
 ### Conformity, in the standards' words
 
 The drift report's "bias inside the instrument's class" is a statement of

{gea_program-0.15.0 → gea_program-0.16.0}/README.md

@@ -33,7 +33,7 @@ reaches a client document. Nothing unmeasured is ever reported as met.
   rate-controlled replay, duplicate suppression, per-record latency.
 - **Configuration versioning, SBOM, monthly SLA, FAT/SAT**
   (`config_versioning.py`, `sbom.py`, `sla_report.py`, `fat_sat.py`).
-- **Live protocol ports** (`live_ports.py`, `wits0.py`, `witsml.py`, `opcua_port.py`,
+- **Live protocol ports** (`live_ports.py`, `wits0.py`, `witsml.py`, `etp.py`, `opcua_port.py`,
   `mqtt_port.py`, `modbus.py`): WITS Level 0 from the drill floor (TCP connect,
   TCP listen, serial), WITSML 1.4.1 stores (read-only, polled), OPC UA (read and
   subscribe), MQTT (number, JSON, Sparkplug B), Modbus TCP; every message becomes
@@ -86,7 +86,7 @@ gea dashboard --catalog-well volve_f12_f14_production_excerpt:15/9-F-12:10000 --
 gea client-report --report accuracy --out client_report
 gea model-cards --out model_cards
 gea sbom --out sbom
-gea accept                     # the product gate (333 checks)
+gea accept                     # the product gate (337 checks)
 gea help drift                 # the help library, by the job (16 pages; the same text is on every dashboard page)
 gea guide                      # the click-by-click tester guide (docs/TESTER_GUIDE.md)
 gea gui                        # the desktop window (pip install "gea-program[desktop]")
@@ -104,7 +104,7 @@ gea swaps / gea certificates / gea notify / gea housekeeping / gea loadtest   #
 
 ## Live data
 
-The engine reads live data through five protocol ports. All are **read-only**,
+The engine reads live data through six protocol ports. All are **read-only**,
 all map only what the site declares (a tag map the client owns; an empty map
 is declined), and all write every received message to a JSON-lines recording
 that `--replay` turns back into records without a connection - which is how a
@@ -117,6 +117,9 @@ gea wits0-sim --port 5001                             # a WITS0 sender to rehear
 gea wits0  --config wits0.json --seconds 60 --record floor/session.jsonl --out floor --stream-csv floor/stream.csv   # in the first window, while the sender runs
 gea witsml --write-example-config witsml.json         # a WITSML 1.4.1 store: url, uids, mnemonics -> tags; credentials by env name
 gea witsml --config witsml.json --seconds 60 --out store                 # needs a real store: the example url is a placeholder until you edit it
+gea etp    --write-example-config etp.json            # a WITSML 2.x store over ETP v1.2: ws:// or wss:// url, bearer token by env name, channel URIs -> tags, units
+gea etp-sim --port 9800                               # an ETP store with two channels to rehearse with - run it in a SECOND window and leave it running
+gea etp    --config etp.json --seconds 60 --record rig/session.jsonl --out rig   # subscribes; the store pushes every value as it is written
 gea opcua  --write-example-config opcua.json          # edit: endpoint, security, credential env names, nodes
 gea opcua  --config opcua.json --seconds 60 --record opc/session.jsonl --out opc --stream-csv opc/stream.csv   # needs a real OPC UA server, likewise
 gea mqtt   --write-example-config mqtt.json           # edit: broker, TLS, credential env names, topics
@@ -128,6 +131,7 @@ gea ingest --file floor/stream.csv                    # a stream CSV feeds the h
 |---|---|---|---|
 | `wits0` | WITS Level 0 frames (`&&` ... `!!`, four-digit item codes) over TCP connect, TCP listen or serial; record-01 dictionary built in | numeric GOOD; sentinels (-9999, -999.25) and non-numeric -> GAP with the reason | items 0105/0106 (date, time) when present, else arrival; ingest at arrival |
 | `witsml` | WITSML 1.4.1 SOAP store: GetVersion, GetCap, GetFromStore on one log object, rows newer than the last seen at each poll | numeric GOOD; the store's nullValue and empty fields -> GAP | the index curve of a time log; arrival for a depth log |
+| `etp` | Energistics ETP v1.2 over WebSocket (`etp12.energistics.org`, Avro binary to the published schemas): RequestSession/OpenSession, GetChannelMetadata for the declared URIs, SubscribeChannels, ChannelData pushed by the store (Protocol 21) | numeric GOOD; a non-numeric or null value -> BAD with the reason; a channel the map does not name is counted unmapped | the store's own index: an ETP DateTime index is microseconds since the epoch; an ElapsedTime index is seconds from its declared start; a depth index leaves the arrival time standing, named |
 | `opcua` | opc.tcp, read + subscribe; Basic256Sha256 SignAndEncrypt when a certificate and key are given | StatusCode severity bits: Good -> GOOD, Uncertain -> STALE, Bad -> GAP (value withheld) | source timestamp, else server timestamp, else arrival |
 | `mqtt` | MQTT v5 / v3.1.1, TLS optional, QoS per topic; `+`/`#` wildcards | number: GOOD or GAP; JSON: value/time/quality paths; Sparkplug B: `is_null` -> GAP, `Quality` != 192 -> STALE | JSON time path, Sparkplug metric timestamp, else arrival |
 | `modbus_g6` | Modbus TCP, user-supplied register map with a citation | read failures -> GAP | arrival |
@@ -580,6 +584,29 @@ promoted; the group and the support are declarations; a matched frequency
 is not a fault size. CSV or miniSEED/SAC; `gea vibration` runs the labelled
 pump; `gea help vibration`.
 
+### WITSML 2.x over ETP
+
+The newer rigs and stores do not poll: Energistics ETP v1.2 is a WebSocket
+session, Avro-encoded to the published schemas, on which the customer names
+the channels it wants and the store pushes every value as it is written.
+`gea/etp.py` is that customer: the handshake with the subprotocol, the
+binary encoding and the payload limits; RequestSession for Discovery, Store
+and ChannelSubscribe and what the store answered; the channel metadata for
+the URIs the mapping declares (a URI the store does not have is named as
+missing); the subscription; and each ChannelData DataItem as a record under
+the mapping's tag and unit, with the store's own index as the source time -
+microseconds since the epoch for a DateTime index, seconds from the declared
+start for ElapsedTime, the arrival time standing (and named) for a depth
+index - and the arrival as the ingest time, so the latency is per record.
+Client message ids even, the store's odd; a ProtocolException carried with
+its code; a client or a store without the subprotocol refused at the
+handshake. The patch supervisor runs it like every other port; the page
+offers it; `gea etp-sim` is a store with two channels of a synthetic
+disposal well; `gea etp --selftest` runs the codec and a loopback session;
+`gea help etp`. The doctor now also names a panel started through the
+`gea.exe` launcher, which pip cannot replace while it runs - stop, update,
+start with `python -m gea serve`.
+
 ### Conformity, in the standards' words
 
 The drift report's "bias inside the instrument's class" is a statement of

{gea_program-0.15.0 → gea_program-0.16.0}/gea/__init__.py

@@ -87,6 +87,7 @@ from .opcua_port import ASYNCUA_AVAILABLE, OpcUaTap          # registers the 'op
 from .mqtt_port import PAHO_AVAILABLE, MqttTap                # registers the 'mqtt' port at import
 from .wits0 import Wits0Tap, SERIAL_AVAILABLE                  # registers the 'wits0' port at import
 from .witsml import WitsmlTap                                  # registers the 'witsml' port at import
+from .etp import EtpTap                                        # registers the 'etp' port at import (WITSML 2.x over ETP v1.2)
 from .well_assembler import (
     WellAssembly, WellComponent, assemble, assemble_ktb_hb, assemble_odp_504b,
     assemble_site_1027, assemble_u1324, BUILTIN_ASSEMBLIES,
@@ -122,7 +123,7 @@ from .profile_catalog import (
     read_pangaea_txt,
 )
 
-__version__ = "0.15.0"
+__version__ = "0.16.0"
 __all__ = [
     "aging_rate", "rate_psi_yr", "accuracy_psi",
     "Sensor", "SimulatorConfig", "DownholeEngine",

{gea_program-0.15.0 → gea_program-0.16.0}/gea/__main__.py

@@ -265,7 +265,8 @@ def main(argv=None) -> int:
     for name, helptxt in (("opcua", "OPC UA client port: read once or subscribe; record; replay a recording offline"),
                           ("mqtt", "MQTT subscriber port (3.1.1/5.0; number | json | Sparkplug B): run; record; replay offline"),
                           ("wits0", "WITS Level 0 port (drill floor: mud logger / EDR) over TCP connect, TCP listen or serial: run; record; replay offline"),
-                          ("witsml", "WITSML 1.4.1 read-only client: poll a log object from the store; record; replay offline")):
+                          ("witsml", "WITSML 1.4.1 read-only client: poll a log object from the store; record; replay offline"),
+                          ("etp", "WITSML 2.x over Energistics ETP v1.2 (WebSocket, Avro): subscribe to channels and the store pushes; record; replay the session's shape")):
         p_lp = sub.add_parser(name, help=helptxt)
         p_lp.add_argument("--config", type=str, default=None, help="site map JSON (client-owned)")
         p_lp.add_argument("--write-example-config", type=str, default=None, help="write an example map here and exit")
@@ -275,6 +276,7 @@ def main(argv=None) -> int:
         p_lp.add_argument("--read-once", action="store_true", help="opcua: one Read of all nodes instead of a subscription")
         p_lp.add_argument("--out", type=str, default=None, help="write the records CSV here")
         p_lp.add_argument("--stream-csv", type=str, default=None, help="also write a historian-style CSV the other commands ingest")
+        p_lp.add_argument("--selftest", action="store_true", help="etp: the codec round-trip and a session against the simulated store on the loopback")
 
     p_ws = sub.add_parser("workspace", help="the client site folder: init, add wells (file | catalogue | live), list, migrate a --out folder, refresh the dashboard, audit log")
     p_ws.add_argument("--path", type=str, required=True, help="the workspace folder")
@@ -400,6 +402,11 @@ def main(argv=None) -> int:
     p_us.add_argument("--password-env", type=str, default="GEA_PASSWORD", help="environment variable holding the password (never a flag, never on the command line)")
     p_us.add_argument("--actor", type=str, default="cli")
 
+    p_es = sub.add_parser("etp-sim", help="an ETP v1.2 store for testing a patch without a rig: two channels of a synthetic disposal well, pushed as a customer subscribes")
+    p_es.add_argument("--port", type=int, default=9800)
+    p_es.add_argument("--frames", type=int, default=600)
+    p_es.add_argument("--interval", type=float, default=1.0, help="seconds between samples")
+    p_es.add_argument("--seed", type=int, default=1)
     p_w0 = sub.add_parser("wits0-sim", help="a WITS0 sender for testing a patch without a rig: listens once, then streams a deterministic drilling sequence")
     p_w0.add_argument("--port", type=int, default=5001)
     p_w0.add_argument("--frames", type=int, default=600)
@@ -1966,10 +1973,30 @@ def main(argv=None) -> int:
             stop.set()
             print("wits0-sim: stopped")
         return 0
-    elif a.cmd in ("opcua", "mqtt", "wits0", "witsml"):
+    elif a.cmd == "etp-sim":
+        from . import etp as _E
+        th, port, stop = _E.simulate_store(port=a.port, frames=a.frames, interval_s=a.interval, seed=a.seed, verbose=True)
+        print(f"etp-sim: an ETP v1.2 store listening on ws://127.0.0.1:{port}/ - this window is the STORE and stays here; nothing happens until a customer connects.\n"
+              f"  Open a SECOND window and run:  gea etp --config etp.json --seconds 60 --out floor   (with etp.json: url ws://127.0.0.1:{port}/ and the two example channel URIs)\n"
+              f"  or on the dashboard: Patch panel -> Add a patch -> etp, url ws://127.0.0.1:{port}/. Then it pushes {a.frames} samples per channel {a.interval:g} s apart. Ctrl+C stops it.", flush=True)
+        try:
+            while th.is_alive():
+                th.join(0.5)
+        except KeyboardInterrupt:
+            stop.set()
+            print("etp-sim: stopped")
+        return 0
+    elif a.cmd in ("opcua", "mqtt", "wits0", "witsml", "etp"):
         import json as _json
         from .live_ports import records_to_stream, summarize, write_records_csv
-        mod = __import__({"opcua": "gea.opcua_port", "mqtt": "gea.mqtt_port", "wits0": "gea.wits0", "witsml": "gea.witsml"}[a.cmd], fromlist=["x"])
+        mod = __import__({"opcua": "gea.opcua_port", "mqtt": "gea.mqtt_port", "wits0": "gea.wits0", "witsml": "gea.witsml", "etp": "gea.etp"}[a.cmd], fromlist=["x"])
+        if a.cmd == "etp" and getattr(a, "selftest", False):
+            r = mod.selftest()
+            print(f"etp selftest [{r['label']}]: {r['status']}")
+            for k, v in r["checks"].items():
+                print(f"  {'ok ' if v else 'BAD'} {k}")
+            print(f"  {r['n_records']} record(s) on {r['tags']}; store {r['store'].get('application')} protocols {r['store'].get('protocols')}; counts {r['counts']}")
+            return 0 if r["status"] == "OK" else 1
         if a.write_example_config:
             print("config:", mod.write_example_config(a.write_example_config))
             return 0
@@ -1994,6 +2021,11 @@ def main(argv=None) -> int:
                 tap = mod.Wits0Tap(a.config, recording_path=a.record)
                 recs = tap.run(a.seconds)
                 src = f"{tap.cfg['transport']} {tap.cfg.get('host') or tap.cfg.get('device') or ('listen:' + str(tap.cfg.get('listen_port')))}"
+            elif a.cmd == "etp":
+                tap = mod.EtpTap(a.config, recording_path=a.record)
+                recs = tap.run(a.seconds)
+                src = tap.cfg["url"] + (f"  (store: {tap.state.get('store', {}).get('application')}, {tap.state.get('channels')} channel(s) subscribed"
+                                        + (f", missing {tap.state.get('missing_uris')}" if tap.state.get('missing_uris') else '') + ')')
             else:
                 tap = mod.WitsmlTap(a.config, recording_path=a.record)
                 recs = tap.run(a.seconds)
@@ -2001,11 +2033,17 @@ def main(argv=None) -> int:
         except (NotImplementedError, ConnectionError, ValueError) as e:
             hint = {"wits0": "nothing is sending on that host:port - start `gea wits0-sim --port 5001` in a SECOND window first (it must be running while this reads), or point the config at the rig's real WITS0 feed",
                     "witsml": "no WITSML store answers at that url - the example config points at 127.0.0.1:8000, a placeholder; put the real store's url, uids and credential env names in it",
+                    "etp": "no ETP store answers at that url - start `gea etp-sim --port 9800` in a SECOND window first, or point the config at the rig's ETP endpoint (wss://...) with the channel URIs it publishes",
                     "opcua": "no OPC UA server answers at that endpoint - the example config is a placeholder; put the real server's endpoint, security and node ids in it",
                     "mqtt": "no MQTT broker answers at that host:port - the example config is a placeholder; put the real broker in it"}.get(a.cmd, "")
             raise SystemExit(f"{a.cmd}: {e}" + (f"\n  {hint}" if hint else ""))          # the reason and what to do, without a traceback
         print(f"{a.cmd}: {src}")
-        print(_json.dumps(summarize(recs), indent=1))
+        if a.cmd == "etp" and a.replay:
+            shape = getattr(mod.replay, "summary", {})
+            print(_json.dumps({"session_shape": shape, "records": 0,
+                               "note": "an ETP recording holds the messages as received with ChannelData as counts (the samples went to --out); the replay reproduces the session's shape, not the samples"}, indent=1))
+        else:
+            print(_json.dumps(summarize(recs), indent=1))
         if a.out:
             print("  records:", write_records_csv(recs, a.out))
         if a.stream_csv and recs:

{gea_program-0.15.0 → gea_program-0.16.0}/gea/acceptance_tests.py

@@ -5453,6 +5453,134 @@ def section_bi_conformance(tmp: str) -> None:
     finally:
         svc.stop()
 
+
+def section_bj_etp(tmp: str) -> None:
+    """Section BJ - the live-port leg on ETP v1.2: the Avro codec to the specification's schemas, the WebSocket
+    with the ETP handshake, the session, the ChannelSubscribe stream into records, the patch supervisor
+    running it, the API and the page knowing the protocol, and a store that is not ETP refused at the door."""
+    import http.cookiejar
+    import socket as _socket
+    import subprocess as _sp
+    import urllib.request
+    import urllib.error
+    from . import etp as E
+    from . import helplib as H
+    from .patches import PatchSupervisor, PROTOCOLS
+    from .ports import PORT_REGISTRY
+    from .telemetry import TelemetryRecorder, TelemetryConfig
+    from .downhole_engine import DownholeEngine, SimulatorConfig
+    from .workspace import Workspace, PORTS
+    from .service import Service, Users
+    pkg = Path(__file__).parent
+    d = Path(tmp, "bj"); d.mkdir()
+
+    # -- the codec, to the schemas as published ------------------------------------------------------------
+    a = E.avro()
+    buf = E.encode_message(0, 1, {'applicationName': 'g', 'applicationVersion': '1', 'clientInstanceId': b'\x07' * 16, 'requestedProtocols': [], 'supportedDataObjects': [],
+                                  'currentDateTime': 5, 'earliestRetainedChangeTime': 0}, message_id=2)
+    hdr, body, name = E.decode_message(buf)
+    zz = [E.Avro._zigzag(x) for x in (0, -1, 1, -2, 2, 63, 64, -64, 2 ** 40)]
+    back = [E.Avro._unzigzag(b, 0)[0] for b in zz]
+    bad_mt = E.decode_message(a.encode(E.HEADER, {'protocol': 99, 'messageType': 99, 'correlationId': 0, 'messageId': 1, 'messageFlags': 0}) + b'\x00')
+    ok(len(a.messages) == 20 and (21, 4) in a.messages and (0, 1000) in a.messages and E.SCHEMA_FILE.endswith('etp12_schemas.json') and os.path.isfile(E.SCHEMA_FILE)
+       and name == 'RequestSession' and body['applicationName'] == 'g' and body['supportedFormats'] == ['xml'] and body['serverAuthorizationRequired'] is False
+       and hdr == {'protocol': 0, 'messageType': 1, 'correlationId': 0, 'messageId': 2, 'messageFlags': 0}
+       and zz[0] == b'\x00' and zz[1] == b'\x01' and zz[2] == b'\x02' and zz[5] == b'\x7e' and zz[6] == b'\x80\x01' and back == [0, -1, 1, -2, 2, 63, 64, -64, 2 ** 40]
+       and bad_mt[1] is None and 'no schema' in bad_mt[2]
+       and E.FLAG_MULTIPART == 0x01 and E.FLAG_FINAL == 0x02 and E.FLAG_ACK == 0x10 and E.FLAG_EXTENSION == 0x20 and E.ETP_SUBPROTOCOL == 'etp12.energistics.org',
+       "BJ1 the Avro binary codec is driven by the ETP v1.2 message schemas as published (twenty messages, the header and the error record): zig-zag "
+       "varints at their boundaries, defaults filled for fields not given, a record decoded back to what was encoded, a message of a kind this port "
+       "does not know left undecoded and named; the message flags and the subprotocol are the specification's")
+
+    # -- the session on the loopback: the self-test, and what the records carry ----------------------------------
+    r = E.selftest(frames=4, interval_s=0.15)
+    ok(r["status"] == "OK" and all(r["checks"].values()) and r["label"] == "SIMULATION_SELF_TEST" and r["n_records"] == 8
+       and r["store_messages"][:3] == ['RequestSession', 'GetChannelMetadata', 'SubscribeChannels'] and r["counts"]["exceptions"] == 0,
+       "BJ2 a customer session against the simulated store over a real WebSocket on the loopback: the ETP handshake (subprotocol, binary encoding, "
+       "payload limits), RequestSession answered by OpenSession, the channel metadata fetched for the declared URIs, the subscription, ChannelData "
+       "pushed at the store's own index (microseconds since the epoch) and carried as each record's source time with the latency measured, the "
+       "client's ids even and the store's odd, a URI the store does not have named as missing, a client without the subprotocol refused at the door")
+
+    # -- the patch supervisor, the API, the page ---------------------------------------------------------------------
+    hist = str(d / "hist.csv")
+    TelemetryRecorder(engine=DownholeEngine(SimulatorConfig()), config=TelemetryConfig(duration_hours=0.5, seed=3)).run().export_csv(hist)
+    wsp = str(d / "ws"); ws = Workspace.create(wsp, "Pad 3", actor="tester")
+    w = ws.add_well_file(hist, display="SWD 1", actor="tester")
+    th, port, stop = E.simulate_store(frames=400, interval_s=0.1, seed=5)
+    cfg = json.loads(json.dumps(E.EXAMPLE_CONFIG)); cfg["url"] = f"ws://127.0.0.1:{port}/"
+    sup = PatchSupervisor(ws)
+    sup.store.add("rig", "etp", w["id"], cfg, "tester", priority=2, stale_after_s=5)
+    sup.start("rig", "tester")
+    time.sleep(3.0)
+    s1 = {x["name"]: x for x in sup.states()}["rig"]
+    sup.stop("rig", "tester")
+    s2 = {x["name"]: x for x in sup.states()}["rig"]
+    stop.set(); th.join(timeout=3)
+    users = Users(str(Path(wsp, "users.json"))); users.add("op", "operator-pass-1", "operator")
+    svc = Service(wsp, port=0, scheduler=False).start()
+    base = svc.url.rstrip("/")
+    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()))
+
+    def call(method, path, body=None):
+        req = urllib.request.Request(base + path, method=method, data=(json.dumps(body).encode() if body is not None else None))
+        req.add_header("Content-Type", "application/json"); req.add_header("X-GEA-Action", "1")
+        try:
+            with opener.open(req, timeout=120) as rr:
+                return rr.status, json.loads(rr.read())
+        except urllib.error.HTTPError as e:
+            return e.code, json.loads(e.read())
+    try:
+        call("POST", "/api/login", {"name": "op", "password": "operator-pass-1"})
+        st_def, dflt = call("GET", "/api/config/defaults/etp")
+        st_pl, pl = call("GET", "/api/patches")
+        st_bad = call("POST", "/api/patches/add", {"name": "bad", "protocol": "etp", "well_id": w["id"], "config": {"url": "http://x/"}})[0]
+        st_add = call("POST", "/api/patches/add", {"name": "rig2", "protocol": "etp", "well_id": w["id"], "config": cfg, "priority": 1, "stale_after_s": 5})[0]
+    finally:
+        svc.stop()
+    page_src = (pkg / "web" / "app.html").read_text(encoding="utf-8")
+    r1 = _sp.run([sys.executable, "-m", "gea", "etp", "--write-example-config", str(d / "etp.json")], capture_output=True, text=True)
+    r2 = _sp.run([sys.executable, "-m", "gea", "etp", "--config", str(d / "etp.json"), "--seconds", "1"], capture_output=True, text=True)
+    th5, port5, stop5 = E.simulate_store(frames=8, interval_s=0.1)
+    cfg5 = json.loads(json.dumps(E.EXAMPLE_CONFIG)); cfg5["url"] = f"ws://127.0.0.1:{port5}/"
+    (d / "etp5.json").write_text(json.dumps(cfg5), encoding="utf-8")
+    r3 = _sp.run([sys.executable, "-m", "gea", "etp", "--config", str(d / "etp5.json"), "--seconds", "3", "--out", str(d / "recs.csv"), "--record", str(d / "rec.jsonl")], capture_output=True, text=True)
+    stop5.set(); th5.join(timeout=3)
+    r4 = _sp.run([sys.executable, "-m", "gea", "etp", "--config", str(d / "etp5.json"), "--replay", str(d / "rec.jsonl")], capture_output=True, text=True)
+    help_md = (pkg / "help" / "etp.md").read_text(encoding="utf-8")
+    ok(s1["status"] == "CONNECTED" and s1["samples_total"] >= 20 and set(s1["tags"]) == {"P_surf_psi", "Q_bpm"} and s1["tags"]["P_surf_psi"]["unit"] == "psi"
+       and 1400 < s1["tags"]["P_surf_psi"]["value"] < 1600 and s1["latency_p95_s"] is not None and s1["latency_p95_s"] < 3.0 and s2["status"] == "STOPPED"
+       and any(f.startswith("patch_rig_") and f.endswith(".records.csv") for f in os.listdir(Path(wsp, "wells", w["id"], "records", "live")))
+       and "etp" in PROTOCOLS and "etp" in PORTS and PORT_REGISTRY["etp"].status == "IMPLEMENTED_REQUIRES_SITE_CONFIG"
+       and st_def == 200 and dflt["content"]["url"].startswith("ws://") and "etp" in pl["protocols"] and st_bad == 400 and st_add == 200
+       and '<option value="etp">' in page_src
+       and r1.returncode == 0 and r2.returncode != 0 and "etp-sim" in (r2.stderr + r2.stdout)
+       and r3.returncode == 0 and (d / "recs.csv").is_file() and (d / "rec.jsonl").is_file() and '"records": 16' in r3.stdout and '"GOOD": 16' in r3.stdout
+       and r4.returncode == 0 and '"channel_data": 8' in r4.stdout and '"exceptions": 0' in r4.stdout and "session's shape" in r4.stdout and any(t["topic"] == "etp" for t in H.topics()) and not H.check("etp")
+       and "microseconds" in help_md and "arrival time" in help_md.split("will not call a measurement")[1],
+       "BJ3 the patch supervisor runs an ETP patch like any other port - CONNECTED with the two channels under their declared tags and units, "
+       "latency measured, records on disk, STOPPED on request; the service knows the protocol (its example config, the protocol list, a bad "
+       "config refused, a good one added); the page offers it; `gea etp` writes the example, names the missing store with the simulator to "
+       "start, records a session and replays its shape; the help page names the depth-indexed sample it will not give a time to")
+
+    # -- the doctor names a panel that cannot be updated in place ---------------------------------------------------
+    from . import doctor as D
+    argv0 = sys.argv[0]
+    try:
+        sys.argv[0] = r"C:\Users\op\AppData\Roaming\Python\Python312\Scripts\gea.exe"
+        env_l = D.check_environment()
+        sys.argv[0] = os.path.join(os.path.dirname(argv0) or ".", "__main__.py")
+        env_m = D.check_environment()
+    finally:
+        sys.argv[0] = argv0
+    fl = [x for x in env_l if "console launcher" in x["what"]]
+    ok(D.started_through_launcher(r"C:\py\Scripts\gea.exe").endswith("gea.exe") and D.started_through_launcher("/home/op/.local/bin/gea") == "/home/op/.local/bin/gea"
+       and D.started_through_launcher("/x/gea/__main__.py") is None and D.started_through_launcher("-m") is None
+       and len(fl) == 1 and fl[0]["level"] == "warn" and "WinError 32" in fl[0]["what"] and "python -m gea serve" in fl[0]["fix"]
+       and not any("console launcher" in x["what"] for x in env_m),
+       "BJ4 the doctor (and so `gea serve` at start) names a panel started through the gea / gea.exe console launcher: pip cannot replace that "
+       "launcher while it runs, so the program cannot be updated in place - stop, update, start with `python -m gea serve`; a panel started "
+       "with `python -m gea` draws no such finding, and `gea update` from a launcher-started panel on Windows refuses to run pip and says why")
+
 def main() -> int:
     print("GEA-Program - ACCEPTANCE SUITE (the product gate)")
     # a handle a job subprocess still holds at the end must not turn a finished gate into a traceback on Windows:
@@ -5517,6 +5645,7 @@ def main() -> int:
         section_bg_ppdm(tmp)
         section_bh_vibration(tmp)
         section_bi_conformance(tmp)
+        section_bj_etp(tmp)
     if _FAILS:
         print(f"[ACCEPTANCE] {len(_FAILS)} FAILURES ({_PASS} passed):")
         for f in _FAILS:

{gea_program-0.15.0 → gea_program-0.16.0}/gea/doctor.py

@@ -22,6 +22,7 @@ from __future__ import annotations
 import importlib.util
 import json
 import os
+import re
 import socket
 import sys
 import sysconfig
@@ -116,6 +117,11 @@ def check_environment() -> List[dict]:
         f.append(_finding('info', f'no `gea` launcher in {" or ".join(candidates)}; use `python -m gea ...`'))
     elif found and not any(c in on_path for c in found):
         f.append(_finding('warn', f'the `gea` launcher is in {found[0]}, which is not on PATH', 'use `python -m gea ...`, or add that folder to PATH'))
+    launcher = started_through_launcher()
+    if launcher:
+        f.append(_finding('warn', f'this process was started through the console launcher {launcher}: while it runs, pip cannot replace '
+                          f'that launcher on Windows (WinError 32), so the program cannot be updated in place',
+                          'stop the panel, then update, then start it with `python -m gea serve` (the kit launcher and start-gea.cmd do)'))
     newest = pypi_newest()
 
     def _vt(v):
@@ -131,6 +137,18 @@ def check_environment() -> List[dict]:
     return f
 
 
+def started_through_launcher(argv0: Optional[str] = None) -> Optional[str]:
+    """The path of the `gea` / `gea.exe` console launcher this process was started through, or None when it was
+    started as `python -m gea` (or from a checkout). On Windows the launcher .exe is held open by the process it
+    started, so pip cannot replace it while the panel runs: `gea update` from such a panel, and `pip install
+    --upgrade` beside it, fail with WinError 32. The kit launchers and start-gea.cmd use `python -m gea` for this."""
+    a0 = sys.argv[0] if argv0 is None else argv0
+    base = re.split(r'[\\/]', a0 or '')[-1].lower()                      # either separator: the launcher path may be Windows-shaped
+    if base in ('gea', 'gea.exe', 'gea-script.py', 'gea-script.pyw'):
+        return a0
+    return None
+
+
 def pypi_newest(timeout_s: float = 3.0) -> Optional[str]:
     try:
         import urllib.request
@@ -150,9 +168,15 @@ def update(check_only: bool = False, extras: str = 'live,plotting,xls,desktop',
     vt = lambda v: tuple(int(x) if x.isdigit() else 0 for x in str(v).split('.'))
     res = {'running': __version__, 'newest_pypi': newest, 'python': sys.executable, 'extras': extras,
            'state': 'unknown' if newest is None else ('current' if vt(newest) <= vt(__version__) else 'behind'), 'ran_pip': False, 'returncode': None}
+    launcher = started_through_launcher()
+    res['started_through_launcher'] = launcher
     if newest is None:
         res['note'] = 'PyPI is not reachable from here; an offline installation is updated by installing the newer kit from the release page'
-    if not check_only and newest is not None and res['state'] == 'behind':
+    if not check_only and newest is not None and res['state'] == 'behind' and launcher and os.name == 'nt':
+        res['note'] = (f'not run: this program was started through {launcher}, which pip cannot replace while it runs (WinError 32); '
+                       'stop the panel, update from a prompt (`python -m pip install --upgrade "gea-program[live]"`), then start it with `python -m gea serve`')
+        res['returncode'] = 2
+    elif not check_only and newest is not None and res['state'] == 'behind':
         spec = f'gea-program[{extras}]=={newest}' if extras else f'gea-program=={newest}'
         r = subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade', spec], capture_output=True, text=True)
         res.update({'ran_pip': True, 'returncode': r.returncode, 'pip_tail': (r.stdout + r.stderr)[-1500:]})