@senpi/runtime 3.0.99 → 3.0.100

package/dist/scanners/external/scaffold/senpi_mcp.py

@@ -12,13 +12,27 @@ the OFFICIAL MCP Python SDK (`mcp`), which is async-only:
             await session.initialize()
             result = await session.call_tool(name, arguments=args)
 
-DECISION (locked, execution-plan B5): use the SDK the STANDARD way — a SINGLE
-PERSISTENT `ClientSession` created ONCE (and `initialize()`d once), reused across
-every call. NO extra guards: no custom timeout, no reconnect/retry wrapper, no
-pooling toggle, no fresh-connection-per-call. The async context managers must
-STAY OPEN for the client's lifetime, so we drive them on a dedicated event loop
-running in a background thread; `call_tool` submits work onto that loop and blocks
-for the result (the sync↔async bridge).
+DECISION (execution-plan B5, AMENDED 2026-09-04): use the SDK the STANDARD way —
+ONE persistent `ClientSession` created once (and `initialize()`d once), reused
+across every call, with no pooling toggle and no fresh-connection-per-call. The
+async context managers must STAY OPEN across calls, so we drive them on a
+dedicated event loop running in a background thread; `call_tool` submits work
+onto that loop and blocks for the result (the sync↔async bridge).
+
+AMENDMENT (RCA 2026-09-04, "External scanners go permanently blind after one
+MCP transport failure"): B5 originally also locked "no reconnect". In
+production the MCP server's `POST /mcp` stream is severed at the 60 s Kong
+upstream read timeout; the SDK then closes its streams but the client kept the
+dead session for the life of the process, so every later `call_tool` raised
+`ClosedResourceError` in ~1 ms until a runtime restart — 16–25 scanner MIDs
+blind per day, and authored `scan()` code swallowing the error made every tick
+report `ok` with zero candidates. The session is therefore now REBUILT: a
+transport-level failure (closed/broken stream, connection reset, broken pipe,
+httpx transport error) discards the dead session and the NEXT call opens and
+initialises a fresh one. The failing call still fails — there is no retry
+inside a call (bounding a call is a separate ticket) — the point is that the
+failure is no longer permanent. The failure is logged once as
+`senpi_mcp_error` with `session_rebuilt: true`.
 
 Response normalization MATCHES producer.mjs `createSenpiToolClient`:
     1. isError truthy            -> RAISE with the error text (never a silent None).
@@ -42,18 +56,27 @@ The `mcp` package is LAZY-IMPORTED on first connect (NOT at module top), so
 `import scaffold.senpi_mcp` works even where `mcp` is not installed. Missing creds
 are LOUD AT FIRST USE: `call_tool` raises, construction never does.
 
+SESSION LIFETIME: each session lives inside ONE long-lived task on the loop
+(`_session_lifetime`), which enters the transport + ClientSession context
+managers, publishes the initialised session, then parks until a close is
+requested and exits the same context managers from the same task. That
+same-task shape is required: the SDK's transport is an anyio task group, and
+anyio refuses to exit a cancel scope from a task other than the one that
+entered it — so a dead session cannot be closed by calling `aclose()` from the
+`call_tool` task. A rebuild only flips the session's close event; the old
+lifetime task unwinds by itself, and `call_tool` never blocks on that teardown,
+so a wedged transport cannot hang a tick.
+
 TEARDOWN (intentional, one-process-per-scanner model): there is NO explicit
-close()/shutdown for the persistent session. The background loop runs on a DAEMON
-thread, so it dies with the process. The scaffold runs exactly ONE client per
-process for that process's whole life — the supervisor restarts by spawning a
-FRESH child process (not by re-running run_scaffold in-place), so a single client
-is never replaced in-process and there is no loop/session leak across restart.
-On graceful stop (SIGTERM/SIGINT) run_scaffold exits its loop and the process
-returns; on a hard kill the OS reclaims the thread. We deliberately do NOT drive
-an AsyncExitStack.aclose() from a shutdown hook: it buys nothing over the dropped
-TCP connection (streamable-http is request/response, no protocol goodbye) and a
-cross-thread aclose() on a wedged transport could itself hang a path that today
-cannot hang.
+close()/shutdown for the client. The background loop runs on a DAEMON thread,
+so it dies with the process. The scaffold runs exactly ONE client per process
+for that process's whole life — the supervisor restarts by spawning a FRESH
+child process (not by re-running run_scaffold in-place), so a single client is
+never replaced in-process and there is no loop leak across restart. On
+graceful stop (SIGTERM/SIGINT) run_scaffold exits its loop and the process
+returns; on a hard kill the OS reclaims the thread. We deliberately do NOT
+drive a close from a shutdown hook: it buys nothing over the dropped TCP
+connection (streamable-http is request/response, no protocol goodbye).
 """
 
 from __future__ import annotations
@@ -356,13 +379,69 @@ def _cast_market_numerics(name: str, payload: Any, args: Optional[dict] = None)
     return payload
 
 
+# --------------------------------------------------------------------------
+# Transport-failure classification — which errors mean "the session is dead".
+# --------------------------------------------------------------------------
+#
+# Matched by class NAME along the MRO so the module keeps its lazy-import
+# contract (no `anyio`/`httpx` import at module top). anyio raises the first
+# three when the SDK's streams are closed or broken under a session; httpx's
+# `TransportError` is the base of every connect/read/write/protocol/timeout
+# failure on the POST itself. OSError covers ConnectionError and its subclasses
+# (reset, refused, aborted), BrokenPipeError and — deliberately — TimeoutError:
+# a call that timed out at the socket, or under a future per-call bound, has
+# left the session in an unknown state and the next call must not inherit it.
+# Our own pre-connect raises (PermissionError for a blocked mutation tool,
+# RuntimeError for missing creds) happen before any session is held, so they
+# never reach this classification.
+_TRANSPORT_ERROR_NAMES: frozenset[str] = frozenset(
+    {
+        "ClosedResourceError",
+        "BrokenResourceError",
+        "EndOfStream",
+        "TransportError",
+    }
+)
+
+
+def _is_transport_error(exc: BaseException) -> bool:
+    """True when `exc` means the MCP session's transport is unusable."""
+    if isinstance(exc, (OSError, EOFError)):
+        return True
+    return any(klass.__name__ in _TRANSPORT_ERROR_NAMES for klass in type(exc).__mro__)
+
+
+class _McpSession:
+    """One live MCP session: the SDK session plus the handles that end it.
+
+    `lifetime` is the loop task that entered the SDK's context managers and
+    will exit them once `close_requested` is set. `lifetime.done()` while the
+    session is still held means the transport died on its own.
+    """
+
+    __slots__ = ("session", "lifetime", "close_requested")
+
+    def __init__(
+        self, session: Any, lifetime: "asyncio.Task[None]", close_requested: asyncio.Event
+    ) -> None:
+        self.session = session
+        self.lifetime = lifetime
+        self.close_requested = close_requested
+
+    @property
+    def alive(self) -> bool:
+        return not self.lifetime.done()
+
+
 class SenpiMcpClient:
     """Sync MCP client over the official async SDK with one persistent session.
 
     Construction is cheap and never touches the network or imports `mcp`. The
     first `call_tool` lazily imports the SDK, spins up a private event loop on a
-    background thread, opens the transport + ClientSession ONCE, and initializes
-    it ONCE. Every subsequent call reuses that same session.
+    background thread (once, for the client's life), opens the transport +
+    ClientSession, and initializes it once. Every subsequent call reuses that
+    same session — until a transport failure, after which the dead session is
+    discarded and the next call opens a fresh one (B5 amendment, RCA 2026-09-04).
     """
 
     def __init__(
@@ -377,12 +456,12 @@ class SenpiMcpClient:
         # → every MUTATION_TOOLS entry is blocked (read-only producer boundary).
         self._write_allowlist: frozenset[str] = frozenset(write_allowlist)
 
-        # Persistent bridge state — created lazily on first call_tool.
+        # Persistent bridge state. The loop + thread are created lazily on the
+        # first call_tool and live for the client's life; the session is
+        # (re)opened lazily whenever there is no live one.
         self._loop: Optional[asyncio.AbstractEventLoop] = None
         self._thread: Optional[threading.Thread] = None
-        self._session: Any = None
-        self._stack: Any = None
-        self._connected = False
+        self._current: Optional[_McpSession] = None
         self._connect_lock = threading.Lock()
 
     # -- public sync surface ----------------------------------------------
@@ -396,7 +475,12 @@ class SenpiMcpClient:
         scaffold event) before it is re-raised, so even when the author's scan()
         swallows the exception in its own try/except, the failure still leaves a
         visible record at this boundary instead of vanishing into an empty result.
+
+        A transport-level failure (see `_is_transport_error`) additionally
+        discards the session it happened on, so the NEXT call opens a fresh one
+        instead of failing forever; its log event carries `session_rebuilt: true`.
         """
+        used: Optional[_McpSession] = None
         try:
             # Read-only producer boundary, enforced PRE-CONNECT (no transport
             # opened for a blocked write). A mutation tool passes only if its exact
@@ -411,30 +495,39 @@ class SenpiMcpClient:
                     "not reachable by authored producer code)."
                 )
 
-            self._ensure_connected()
-            result = self._run(self._session.call_tool(name, arguments=args))
+            used = self._ensure_connected()
+            result = self._run(used.session.call_tool(name, arguments=args))
             return _cast_market_numerics(name, self._unwrap(name, result), args)
         except Exception as exc:
             # Catch every failure — mutation block, missing creds, the lazy
             # `import mcp`, transport errors, tool `isError` — log it, then re-raise
             # to the author's scan(). Scoped to Exception so a tick timeout
             # (_TickTimeout, a BaseException) still unwinds the tick untouched.
+            rebuilt = used is not None and _is_transport_error(exc) and self._discard(used)
             log_event(
                 "senpi_mcp_error",
                 tool=name,
                 error_type=type(exc).__name__,
                 error=str(exc)[:500],
+                **({"session_rebuilt": True} if rebuilt else {}),
             )
             raise
 
     # -- persistent session lifecycle -------------------------------------
 
-    def _ensure_connected(self) -> None:
-        if self._connected:
-            return
+    def _ensure_connected(self) -> _McpSession:
+        """Return the live session, opening one when there is none.
+
+        A session whose lifetime task has ended on its own (the transport died
+        between calls) counts as absent and is replaced here, before the call.
+        """
+        current = self._current
+        if current is not None and current.alive:
+            return current
         with self._connect_lock:
-            if self._connected:
-                return
+            current = self._current
+            if current is not None and current.alive:
+                return current
 
             # Loud at first use: creds are required to reach the MCP server.
             if not self._api_key:
@@ -448,43 +541,83 @@ class SenpiMcpClient:
                     "(creds are resolved at launch from SENPI_API_KEY/SENPI_MCP_URL)."
                 )
 
-            # Dedicated event loop on a background thread. The async context
-            # managers (transport + session) must stay OPEN for the client's
-            # lifetime, so we cannot use asyncio.run() per call — that would
-            # tear the session down between calls.
-            self._loop = asyncio.new_event_loop()
-            self._thread = threading.Thread(
-                target=self._loop.run_forever,
-                name="senpi-mcp-loop",
-                daemon=True,
-            )
-            self._thread.start()
+            if self._loop is None:
+                # Dedicated event loop on a background thread, created ONCE per
+                # client. The async context managers (transport + session) must
+                # stay OPEN across calls, so we cannot use asyncio.run() per
+                # call — that would tear the session down between calls. A
+                # rebuild reuses this loop; it never spawns another thread.
+                self._loop = asyncio.new_event_loop()
+                self._thread = threading.Thread(
+                    target=self._loop.run_forever,
+                    name="senpi-mcp-loop",
+                    daemon=True,
+                )
+                self._thread.start()
 
-            self._run(self._connect())
-            self._connected = True
+            self._current = self._run(self._open_session())
+            return self._current
 
-    async def _connect(self) -> None:
-        """Open the transport + ClientSession ONCE and initialize ONCE.
+    def _discard(self, dead: _McpSession) -> bool:
+        """Drop `dead` so the next call opens a fresh session; ask it to close.
+
+        Returns False (and touches nothing) when `dead` is no longer the held
+        session — a concurrent caller already replaced it, and its successor
+        must not be thrown away for the predecessor's failure. The close is
+        fire-and-forget: the lifetime task exits its own context managers, and
+        the caller never waits on a possibly-wedged transport.
+        """
+        with self._connect_lock:
+            if self._current is not dead:
+                return False
+            self._current = None
+        self._loop.call_soon_threadsafe(dead.close_requested.set)
+        return True
+
+    async def _open_session(self) -> _McpSession:
+        """Start a session lifetime task and wait until its session is initialised.
+
+        A failure while opening propagates to the caller; the lifetime task has
+        already unwound whatever it had entered, so nothing is left half-built.
+        """
+        loop = asyncio.get_running_loop()
+        ready: "asyncio.Future[Any]" = loop.create_future()
+        close_requested = asyncio.Event()
+        lifetime = loop.create_task(self._session_lifetime(ready, close_requested))
+        session = await ready
+        return _McpSession(session, lifetime, close_requested)
+
+    async def _session_lifetime(self, ready: "asyncio.Future[Any]", close_requested: asyncio.Event) -> None:
+        """Own one session from open to close, all within this single task.
 
         Lazy-import the SDK here so `import scaffold.senpi_mcp` succeeds in envs
         without `mcp` installed (tests inject a fake `mcp` into sys.modules).
         """
-        from mcp import ClientSession
-        from mcp.client.streamable_http import streamablehttp_client
-
-        # Keep the async context managers alive by holding their __aenter__
-        # values on self and only __aexit__-ing them at close. We use
-        # AsyncExitStack so both nest correctly and unwind in order.
-        from contextlib import AsyncExitStack
-
-        self._stack = AsyncExitStack()
-        headers = {"Authorization": f"Bearer {self._api_key}"}
-        read, write, _ = await self._stack.enter_async_context(
-            streamablehttp_client(self._mcp_url, headers=headers)
-        )
-        session = await self._stack.enter_async_context(ClientSession(read, write))
-        await session.initialize()
-        self._session = session
+        try:
+            from mcp import ClientSession
+            from mcp.client.streamable_http import streamablehttp_client
+
+            headers = {"Authorization": f"Bearer {self._api_key}"}
+            async with streamablehttp_client(self._mcp_url, headers=headers) as (read, write, _):
+                async with ClientSession(read, write) as session:
+                    await session.initialize()
+                    ready.set_result(session)
+                    await close_requested.wait()
+        except BaseException as exc:
+            if not ready.done():
+                # Opening failed: surface it to the waiting call_tool. A
+                # non-Exception (task cancellation) is wrapped so the waiter
+                # gets a plain error instead of blocking forever.
+                ready.set_exception(
+                    exc
+                    if isinstance(exc, Exception)
+                    else RuntimeError(f"senpi_mcp: session open aborted by {type(exc).__name__}")
+                )
+            # Otherwise the transport died under a published session (the
+            # next call_tool sees `alive` False and rebuilds) or its teardown
+            # complained after a rebuild — neither has a caller left to tell.
+            if not isinstance(exc, Exception):
+                raise
 
     def _run(self, coro: Any) -> Any:
         """Drive a coroutine to completion on the persistent loop, blocking."""

package/openclaw.plugin.json

@@ -2,7 +2,7 @@
   "id": "runtime",
   "name": "Senpi Runtime",
   "description": "Senpi runtime plugin for OpenClaw. Orchestrates scanners, LLM-based trade decisions, and a DSL trailing stop-loss exit engine.",
-  "version": "3.0.99",
+  "version": "3.0.100",
   "commandAliases": [
     {
       "name": "senpi"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@senpi/runtime",
-  "version": "3.0.99",
+  "version": "3.0.100",
   "description": "Senpi runtime plugin for OpenClaw",
   "type": "module",
   "main": "dist/index.js",