flowjet 2.1.4__tar.gz → 2.1.6__tar.gz

{flowjet-2.1.4 → flowjet-2.1.6}/CHANGELOG.md

@@ -10,6 +10,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.6] — 2026-10-24
+
+### Fixed
+
+- **ArcBox tool sandboxes stay usable across recycles, crashes and restarts.**
+  The manager trusted its in-memory lease map, and the agent graph pinned its
+  sandbox id at build time, so a guest that died behind a still-`ready` daemon
+  record (host sleep, idle recycle, daemon restart) left the worker talking to a
+  corpse: every file/shell tool failed and the agent reported an empty
+  workspace even though the host tree was fully staged. `open_workspace` now
+  probes readiness once per turn — a control-plane `info()` tier that catches a
+  record outliving its VM plus terminal states, then a guest round-trip that
+  catches a wedged guest — and kills, drops and replaces a lease that fails.
+  Every new guest, whether spawned or adopted, is seeded from the host
+  workspace, and in-flight run counts carry over so peer open/close pairs stay
+  balanced. The agent cache key is now workspace **+ sandbox id**, so a respawn
+  rebuilds the graph against the live guest.
+- **Crash and restart recovery.** VMs are created with a labelled orphan
+  deadline (`idle_timeout` + `on_idle=kill`) that the reaper re-arms every 60 s,
+  so a process that dies leaves nothing behind — the daemon reaps it, with no
+  cross-process coordination and no startup kill-list. On first use the manager
+  scans owner-scoped sandboxes and adopts a surviving VM, pulling guest state to
+  the host before pushing (recovering a crashed turn's artifacts) instead of
+  spawning a duplicate; a failed create reconnects the client and retries once.
+- **Every control-plane RPC is bounded** (`kill`/`set_lifecycle` 15 s, `connect`
+  by `connect_timeout`). The SDK's per-RPC deadline is opt-in and was
+  unconfigured: one wedged call stalled the reaper loop, and a stalled heartbeat
+  reads to the daemon exactly like a dead owner — which gets healthy, in-use VMs
+  reaped at their deadlines.
+
+### Changed
+
+- `sandbox.orphan_timeout_seconds` defaults to `max(900, 3 × idle_timeout_seconds)`
+  instead of a fixed 900, and an explicit value is validated only when the
+  sandbox is enabled — existing configs keep parsing unchanged. New keys:
+  `recover_orphans` (adopt surviving VMs after a restart) and `owner` (label
+  namespace, env `FLOWJET_SANDBOX_OWNER`) so planes sharing one ArcBox daemon
+  never adopt or reap each other's sandboxes. See RFC-005 §5.1/§5.2.
+
+## [2.1.5] — 2026-10-24
+
+### Fixed
+
+- **`ask_user` pauses render as Markdown on ACP/UI hosts.** A clarification
+  turn's reply reused the CLI terminal block (space-indented `⏸ Paused for
+  input:` followed by `• option` lines). Hosts that render replies as Markdown
+  collapse the indentation, so the pause and its options displayed as a single
+  run-on paragraph. The ACP path now emits `format_questions_markdown()`: a
+  bold pause heading, a bold question (Q1:/Q2: numbering for multiple
+  questions), and options as `- **label** — description` bullets, keeping the
+  `(Recommended)` → ★ convention. The CLI keeps the plain-text block.
+
 ## [2.1.4] — 2026-10-23
 
 ### Added

{flowjet-2.1.4 → flowjet-2.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: flowjet
-Version: 2.1.4
+Version: 2.1.6
 Summary: FlowJet — agent CLI (fj) and server (ACP + OpenAI-compatible HTTP) powered by Soothe
 Project-URL: Homepage, https://github.com/caesar0301/flowjet
 Project-URL: Repository, https://github.com/caesar0301/flowjet

{flowjet-2.1.4 → flowjet-2.1.6}/docs/specs/RFC-005-arcbox-tool-sandbox.md

@@ -68,6 +68,9 @@ sandbox:
   connect_timeout: 60
   idle_timeout_seconds: 300  # idle sandbox recycle
   max_sandboxes: 8          # 0 (unlimited), None (70% usable CPU / vcpus, min 8), or int
+  orphan_timeout_seconds: null  # daemon-side crash deadline; null → max(900, 3× idle)
+  recover_orphans: true        # adopt a surviving VM after a restart
+  owner: flowjet               # sandbox label namespace; env FLOWJET_SANDBOX_OWNER
   # FLOWJET_SANDBOX_MAX (int; 0=unlimited) and FLOWJET_SANDBOX_MAX_PERCENT (1-100, default 70) env overrides
   # api_url / api_key: optional; env fallback ARCBOX_API_URL / ARCBOX_API_KEY
   # soothe-nano placement policy (pass-through):
@@ -92,7 +95,7 @@ flowchart LR
     E[WorkspaceResolver<br/>/ SyncAware] -->|open: push host→guest<br/>close: pull guest→host| D
 ```
 
-- `flowjet.core.backends.tool_sandbox.ArcBoxToolSandboxManager` — one guest VM
+- `flowjet.core.sandbox.arbox.ArcBoxToolSandboxManager` — one guest VM
   per workspace: lazy spawn (`AsyncArcBox.create(..., wait_until_ready=True)`),
   idle reaper, LRU eviction at `max_sandboxes` (explicit int, `None`, or `0`
   for unlimited — `None` sizes from host CPU: 70% usable ÷ per-sandbox vCPUs,
@@ -102,11 +105,73 @@ flowchart LR
   runs inside the session lifecycle: open = inner materialize → host push;
   close = guest pull → inner checkpoint (the sync checkpoint must see guest
   outputs).
-- Adapter binding — the sandbox backend is bound at agent build time; the
-  adapter rebuilds its agent when the pool worker switches workspaces. In
+- Adapter binding — the sandbox backend is pinned at agent build time, so the
+  adapter caches the built agent on **workspace + sandbox id**, not the path
+  alone: a respawn, idle recycle or eviction yields a new id and the agent is
+  rebuilt (a stale binding fails every guest tool call with `VM not found`). In
   sandbox mode the request's `configurable["workspace"]` is the **guest root**
   (`/workspace`), not the host path — matching soothe-nano sandbox semantics
   (system prompt, workspace policy, and recon all target the guest root).
+
+### 5.1 Readiness invariant
+
+`open_workspace` runs once per turn on the main loop, *before* the agent
+streams, and is the only place a guest is created or validated; the sync
+`backend_for` (worker threads) never spawns. It guarantees a *usable* guest:
+
+1. **Probe** — two tiers, cheap first. Tier 1 is `sandbox.info()`: one unary
+   RPC, "always fresh, never cached", no guest execution, no lifecycle side
+   effects — it catches the observed failure class (the daemon's record
+   *outliving the VM*: `list()` still says `ready` while every call fails
+   with `VM not found`) and terminal states (`stopping`/`stopped`/`failed`).
+   Tier 2 is a guest round-trip (`true`), catching a wedged guest (VM
+   present, nothing executing); commands run as independent process groups,
+   so it never queues behind a peer's long-running command. A record is
+   never trusted on its own.
+2. **Replace** — a lease that fails the probe is killed and dropped; the
+   replacement is seeded from the host tree. In-flight run counts carry over so
+   peer open/close pairs stay balanced, and the reseed is forced even with a
+   peer mid-run (a new VM is empty — skipping the push is how materials go
+   missing).
+3. **Reseed** — every path that yields a new guest (spawn, respawn, adoption)
+   pushes the host workspace before the turn runs.
+
+Every control-plane RPC in the manager is bounded (`kill` / `set_lifecycle`
+by a 15 s deadline, `connect` by `connect_timeout`): the SDK's per-RPC
+deadline is opt-in and unconfigured here, and one wedged call would otherwise
+stall the reaper loop — a stalled heartbeat reads to the daemon exactly like
+a dead owner, which gets *healthy* VMs reaped at their deadlines.
+
+### 5.2 Crash and restart
+
+The manager holds its leases in memory, so a `flowjet-server` crash loses the
+map while the daemon keeps the VMs. Three mechanisms keep that safe:
+
+- **Daemon-side orphan deadline** — each VM is created with
+  `idle_timeout=<orphan deadline>` + `on_idle="kill"`; the manager
+  *re-arms* that timer every `_HEARTBEAT_INTERVAL_SECONDS` (60 s, one
+  bounded `set_lifecycle` per lease). Expiry therefore means "the owning
+  process is gone", and the daemon reaps the VM with no cross-process
+  coordination and no startup kill-list. A lease that cannot be re-armed is
+  unreachable and is dropped immediately. The deadline defaults to
+  `max(900, 3 × idle_timeout_seconds)` when unset, so an existing config with
+  a long idle reaper parses unchanged and is never preempted by a daemon
+  kill; an explicit value must exceed `idle_timeout_seconds` (validated only
+  when the sandbox is enabled).
+- **Startup scan + adoption** — on first use the manager lists owner-scoped
+  sandboxes carrying its labels (`flowjet.sandbox{,-owner,-instance,-workspace}`).
+  Entries from another instance are recovery *candidates*, not victims: the
+  first turn for that workspace connects, probes, then **pulls guest → host**
+  (recovering files the interrupted turn wrote but never pulled) before pushing
+  the host tree back. No candidate (or a dead one) falls through to a fresh,
+  seeded VM. `recover_orphans: false` disables adoption.
+- **Owner scoping** — `owner` namespaces the labels so several flowjet planes
+  sharing one daemon never adopt or reap each other's sandboxes. Adoption
+  additionally assumes one process per workspace at a time (the shipped
+  agentspace plane runs a single replica).
+
+A failing `create` reconnects the client and retries once, covering a daemon
+that is itself restarting.
 - Credential boundary — LLM keys and the nano config never enter the guest;
   only the staged workspace bytes do.
 

{flowjet-2.1.4 → flowjet-2.1.6}/docs/specs/rfc-history.md

@@ -21,7 +21,7 @@ This document tracks all RFC lifecycle events in chronological order (newest fir
 
 | Date | Event | RFC / Document | Summary |
 |------|-------|----------------|---------|
-| 2026-10-18 | Created | [RFC-005](RFC-005-arcbox-tool-sandbox.md) | ArcBox tool sandbox via soothe-nano 2.0; whole-worker arcbox isolation removed |
+| 2026-10-24 | Updated | [RFC-005](RFC-005-arcbox-tool-sandbox.md) | §5.1 two-tier readiness probe + sandbox-id agent cache key; §5.2 crash/restart recovery (daemon orphan deadline, heartbeat, orphan adoption); bounded control-plane RPCs; orphan deadline derives when unset || 2026-10-18 | Created | [RFC-005](RFC-005-arcbox-tool-sandbox.md) | ArcBox tool sandbox via soothe-nano 2.0; whole-worker arcbox isolation removed |
 | 2026-07-31 | Reference Updated | [TEST-COVERAGE-MATRIX](../impl/TEST-COVERAGE-MATRIX.md) | Feature → test attachment for RFC-001–003 |
 | 2026-07-31 | Created | [IG-003](../impl/IG-003-production-isolation-hardening.md) | Production isolation hardening guide for RFC-003 |
 | 2026-07-31 | Created | [RFC-003](RFC-003-production-isolation-hardening.md) | Production-ready ThreadPool lifecycle, bridge, and policy |

{flowjet-2.1.4 → flowjet-2.1.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "flowjet"
-version = "2.1.4"
+version = "2.1.6"
 description = "FlowJet — agent CLI (fj) and server (ACP + OpenAI-compatible HTTP) powered by Soothe"
 readme = "README.md"
 license = { text = "MIT" }

{flowjet-2.1.4 → flowjet-2.1.6}/src/flowjet/cli/stream.py

@@ -420,6 +420,39 @@ def format_questions_block(
     return "\n".join(lines) + "\n"
 
 
+def format_questions_markdown(questions: tuple[dict[str, Any], ...]) -> str:
+    """Render ``ask_user`` questions as Markdown for hosts that render replies.
+
+    Same content as :func:`format_questions_block` (the CLI renders that block
+    to a terminal), but in Markdown so ACP/UI hosts show a bold pause heading
+    with bulleted, bolded options. The plain-text block cannot be reused here:
+    its space indentation collapses under Markdown rendering and the whole
+    block displays as one run-on paragraph.
+    """
+    if not questions:
+        return "**⏸ Paused for input**\n"
+    lines: list[str] = ["**⏸ Paused for input**"]
+    for qi, q in enumerate(questions, 1):
+        header = str(q.get("header") or "").strip()
+        question = str(q.get("question") or "").strip()
+        label = f"[{header}] {question}" if header else question
+        prefix = f"Q{qi}: " if len(questions) > 1 else ""
+        lines.append(f"\n**{prefix}{label}**")
+        options = q.get("options")
+        if isinstance(options, list):
+            for opt in options:
+                if not isinstance(opt, dict):
+                    continue
+                opt_label = str(opt.get("label") or "").strip()
+                desc = str(opt.get("description") or "").strip()
+                marker = " ★" if "(Recommended)" in opt_label or "Recommended" in opt_label else ""
+                if desc:
+                    lines.append(f"- **{opt_label}{marker}** — {desc}")
+                else:
+                    lines.append(f"- **{opt_label}{marker}**")
+    return "\n".join(lines) + "\n"
+
+
 async def stream_query(
     agent: SootheNanoAgent,
     query: str,

{flowjet-2.1.4 → flowjet-2.1.6}/src/flowjet/core/bridges/base.py

@@ -66,10 +66,12 @@ class AgentAdapterBase:
 
     Manages one agent instance per pool worker: lazily built on first use,
     bound to a workspace's sandbox when a ``sandbox_manager`` is present
-    (rebuilt when the worker switches workspaces), and recycled after a failed
-    or cancelled run — cancellation can interrupt middleware cleanup and leave
-    mutable graph state (for example edit-coalescing buffers) incomplete, so
-    that graph is never reused for another request.
+    (rebuilt when the worker switches workspaces *or* that workspace's sandbox
+    is replaced — the graph's backend is pinned to a sandbox id, not a path),
+    and recycled after a failed or cancelled run — cancellation can interrupt
+    middleware cleanup and leave mutable graph state (for example
+    edit-coalescing buffers) incomplete, so that graph is never reused for
+    another request.
 
     Thread state, by contrast, is *not* worker-local: when
     ``checkpointer_policy`` is enabled the adapter opens one checkpointer on
@@ -100,6 +102,8 @@ class AgentAdapterBase:
         self._checkpointer_cm: Any | None = None
         self._checkpointer_failed = False
         self._agent_workspace: str | None = None
+        #: Sandbox id the cached agent's backend is bound to (``None`` off-sandbox).
+        self._agent_sandbox_id: str | None = None
         self._active = False
         self._tainted = False
         self._generation = 1 if agent is not None else 0
@@ -148,20 +152,53 @@ class AgentAdapterBase:
 
     async def _ensure_agent(self, *, workspace: str | None = None) -> Any:
         """Async ensure: open the checkpointer before a config-driven build."""
-        needs_build = self._agent is None or (
-            self._sandbox_manager is not None and self._agent_workspace != workspace
-        )
-        if needs_build and self._agent_factory is None:
+        if self._agent_needs_rebuild(workspace) and self._agent_factory is None:
             await self._ensure_checkpointer()
         return self._ensure(workspace=workspace)
 
+    def _live_sandbox_id(self, workspace: str | None) -> str | None:
+        """Id of the guest the manager currently binds to ``workspace``."""
+        if self._sandbox_manager is None or workspace is None:
+            return None
+        return self._sandbox_manager.sandbox_id_for(workspace)
+
+    def _agent_needs_rebuild(self, workspace: str | None) -> bool:
+        """True when this worker must build a new agent for ``workspace``.
+
+        Without a sandbox manager the built agent is workspace-agnostic, so an
+        existing instance is reused. With one, the graph holds an
+        ``ArcBoxBackend`` bound at build time: either a workspace switch or a
+        replacement of *that workspace's* sandbox (idle recycle, dead VM,
+        eviction) leaves it pointing at a guest that no longer exists, where
+        every file and shell tool fails with ``VM not found``. The path alone
+        cannot detect the second case — it stays the same across generations —
+        so the live sandbox id is part of the cache key.
+        """
+        if self._agent is None:
+            return True
+        if self._sandbox_manager is None:
+            return False
+        return (
+            self._agent_workspace != workspace
+            or self._agent_sandbox_id != self._live_sandbox_id(workspace)
+        )
+
     def _ensure(self, *, workspace: str | None = None) -> Any:
+        if not self._agent_needs_rebuild(workspace):
+            assert self._agent is not None
+            return self._agent
         if self._agent is not None:
-            if self._sandbox_manager is None or self._agent_workspace == workspace:
-                return self._agent
-            # The sandbox backend is bound at build time — a workspace
-            # switch needs a fresh agent bound to the new sandbox.
-            self._agent = None
+            # Bound to a workspace/sandbox generation that is no longer current.
+            logger.info(
+                "%s: rebuilding agent for %s (sandbox %s -> %s)",
+                self.label,
+                workspace,
+                self._agent_sandbox_id,
+                self._live_sandbox_id(workspace),
+            )
+        self._agent = None
+        self._agent_sandbox_id = None
+        live_sandbox_id = self._live_sandbox_id(workspace)
         if self._agent_factory is not None:
             self._agent = self._agent_factory()
         else:
@@ -170,6 +207,7 @@ class AgentAdapterBase:
                 sandbox_backend = self._sandbox_manager.backend_for(workspace)
             self._agent = self._build(sandbox_backend=sandbox_backend)
         self._agent_workspace = workspace
+        self._agent_sandbox_id = live_sandbox_id
         self._generation += 1
         return self._agent
 
@@ -216,11 +254,13 @@ class AgentAdapterBase:
         if self._tainted:
             self._agent = None
             self._agent_workspace = None
+            self._agent_sandbox_id = None
             self._tainted = False
 
     async def cleanup(self) -> None:
         self._agent = None
         self._agent_workspace = None
+        self._agent_sandbox_id = None
         self._tainted = False
         self._active = False
         cm, self._checkpointer_cm = self._checkpointer_cm, None