switchroom 0.19.27 → 0.19.29

package/vendor/hindsight-memory/scripts/drain_pending.py

@@ -23,6 +23,49 @@ memory" from degrading into "never drain anything" — the drain is
 sequential and oldest-first, so without the demotion three chronically
 failing entries sit at the head and end every run at zero progress.
 
+Serialisation
+-------------
+``drain()`` takes an EXCLUSIVE, non-blocking lock (``_exclusive_drain``,
+``_drain_lock_path``) for the whole run and returns a zero summary with
+``skipped_locked=True`` if another drain already holds it. That lives HERE,
+not in a caller, because the queue has several independent drain paths and a
+guarantee that depends on every caller remembering to wrap itself in
+``flock`` is not a guarantee:
+
+* the SessionStart hook (``session_start.py`` imports ``drain`` directly),
+* the ``hindsight-drain`` sidecar in ``profiles/_base/start.sh.hbs``,
+* the operator's out-of-band replay, which ``switchroom doctor`` documents
+  as a bare ``docker exec … drain_pending.py --backlog``.
+
+Two of those overlapping means two processes iterate the same queue and
+re-POST the same entry — ~168s of a 4-slot fleet-wide LLM lane, twice, for
+one memory. The lock is ``fcntl.flock``, so the kernel releases it if the
+process dies; there is no stale-lock recovery to get wrong.
+
+The lock file is ``drain-pending.lock``, deliberately NOT the ``drain.lock``
+that the interim host cron wraps its ``docker exec`` in. Same-path would
+mean that wrapper's outer ``flock`` starves the ``drain_pending.py`` it just
+launched — a silent no-op, i.e. the drain quietly stops. With a distinct
+name an external ``flock`` wrapper on the historical path is redundant but
+harmless, and this module's guarantee holds no matter who calls it.
+
+Circuit breaker
+---------------
+Demotion (above) bounds head-of-line blocking, but not TOTAL work: a
+chronically failing entry is retried on every run forever. That was
+tolerable when the only drains were a 4-second hook and a supervised manual
+sweep. It is not tolerable for the unattended sidecar, whose backlog budget
+is 3600s against a 900s cooldown — a wedged agent would spend ~80% of
+wall-clock re-POSTing the same unsaveable entries into a shared lane, for
+good. So an entry past ``_attempt_ceiling()`` (default ``MAX_ATTEMPTS`` x 4
+= 20 attempts, ``HINDSIGHT_DRAIN_ATTEMPT_CEILING``) is PARKED: skipped by
+the retain pass, counted in ``summary["parked"]``, still on disk, still
+swept for free by the reconcile pass, and still retried in full under
+``--force``. Parking destroys nothing; it stops paying for the same failure
+without end. HOW MUCH WALL-CLOCK 20 ATTEMPTS BUYS IS QUEUE-SIZE DEPENDENT,
+and is shorter than it looks — the measured table is at
+``ATTEMPT_CEILING_MULTIPLE``, and the trade it records is deliberate.
+
 Boundaries
 ----------
 * Per-entry HTTP timeout: ``HINDSIGHT_DRAIN_TIMEOUT`` (default 5s), but
@@ -118,6 +161,8 @@ Standalone usage::
 from __future__ import annotations
 
 import argparse
+import contextlib
+import fcntl
 import os
 import subprocess
 import sys
@@ -136,6 +181,7 @@ from lib.pending import (
     is_permanent_failure,
     iter_entries,
     mark_dead,
+    pending_dir,
     resplit_over_bound_entries,
     sweep_legacy_dead_markers,
     update_attempt,
@@ -145,6 +191,179 @@ from lib.retain_split import retain_client_deadline, retain_content_limit
 
 STALL_THRESHOLD = 3
 
+#: Attempt ceiling, as a multiple of ``MAX_ATTEMPTS`` — see ``_attempt_ceiling``
+#: and the "Circuit breaker" section of the module docstring. 4 x 5 = 20
+#: attempts.
+#:
+#: HOW LONG THAT BUYS IS QUEUE-SIZE DEPENDENT, which is the non-obvious part
+#: and the reason an earlier revision of this comment was wrong (it claimed 20
+#: attempts was "most of a day … far past any transient upstream outage"). A
+#: drain run attempts each live entry at most once, and ``pending.enqueue``
+#: already records attempt 1, so the FLOOR is 19 further runs — on the
+#: sidecar's 900s cooldown, **under 5 hours**, not most of a day. Above that
+#: floor the only thing that slows parking down is ``STALL_THRESHOLD`` (3),
+#: and it stops applying exactly when it would start to help: past
+#: ``MAX_ATTEMPTS`` an entry abstains from the stall guard (``_over_budget``),
+#: so once a queue is uniformly over budget EVERY entry is attempted on EVERY
+#: run.
+#:
+#: Measured against this module by ``CeilingArithmeticTest`` — total upstream
+#: outage, every attempt failing, counting drain runs until the whole queue is
+#: parked:
+#:
+#: ====  ====  ==============
+#: queue runs  at 900s
+#: ====  ====  ==============
+#: 1     20    5.0 h
+#: 4     24    6.0 h
+#: 13    36    9.0 h
+#: 30    56    14.0 h
+#: ====  ====  ==============
+#:
+#: So the guarantee runs BACKWARDS from the intuition: the SMALLER and
+#: healthier the queue, the SOONER a single overnight outage parks it whole.
+#: A 4-entry queue is gone in 6 hours. That is a real cost and it is accepted
+#: rather than papered over by a bigger number, because:
+#:
+#: * the ceiling is the ONLY bound on unattended lane waste, and it is a
+#:   per-run bound of ``len(queue)`` x ~168s once the stall guard abstains
+#:   (the n=30 run above retries all 30 entries every run for 14 hours).
+#:   Sizing the ceiling to survive a 24h outage means ~100 attempts, i.e.
+#:   5x that window — weakening the very mechanism this change adds;
+#: * parking destroys nothing. The entry stays on disk, the free reconcile
+#:   pass still sweeps it, ``--force`` replays it in full, and since the
+#:   summary-line fix below it is REPORTED on every drain path rather than
+#:   only the backlog one;
+#: * no attempt count can make a wall-clock promise anyway. Attempts convert
+#:   to hours only via the queue size, so a larger constant would restate the
+#:   same category error with a different number. Stating the conversion is
+#:   the durable fix.
+#:
+#: An operator who knows the upstream will be down longer than the table
+#: allows raises ``HINDSIGHT_DRAIN_ATTEMPT_CEILING`` for the duration, and
+#: replays with ``--force`` afterwards; ``switchroom doctor``'s backlog
+#: remediation names both.
+ATTEMPT_CEILING_MULTIPLE = 4
+
+#: Basename of the drain lock, inside the same directory that holds
+#: ``pending-retains/``. NOT ``drain.lock`` — see the module docstring:
+#: the interim host cron wraps its ``docker exec`` in ``flock -n
+#: $HOME/.hindsight/drain.lock``, and sharing that path would make that
+#: wrapper starve the very ``drain_pending.py`` it launches.
+DRAIN_LOCK_BASENAME = "drain-pending.lock"
+
+
+def _drain_lock_path() -> str:
+    """Absolute path of the per-agent drain lock.
+
+    Derived from ``pending_dir()`` rather than ``$HOME`` so it follows the
+    queue it protects: one lock per queue, including under
+    ``HINDSIGHT_PENDING_DIR``. In production that resolves to
+    ``$HOME/.hindsight/drain-pending.lock``.
+    """
+    override = os.environ.get("HINDSIGHT_DRAIN_LOCK")
+    if override:
+        return override
+    return os.path.join(
+        os.path.dirname(os.path.abspath(pending_dir())), DRAIN_LOCK_BASENAME
+    )
+
+
+@contextlib.contextmanager
+def _exclusive_drain():
+    """Hold the drain lock for the duration of a run.
+
+    Yields ``True`` if this process owns the drain, ``False`` if another
+    drain already holds the lock (caller must do nothing).
+
+    ``fcntl.flock`` and not a pidfile: the kernel releases it when the fd
+    closes OR the process dies, so a SIGKILLed drain cannot wedge the queue
+    and there is no stale-lock reaper to get wrong. ``LOCK_NB`` and not a
+    blocking wait, because every caller has somewhere better to be — the
+    SessionStart hook has a ~9s budget, and the sidecar has another tick in
+    900s.
+
+    FAILING TO OPEN the lock file is NOT contention and must not stop the
+    drain: an unwritable ``.hindsight/`` (read-only mount, ENOSPC) would
+    otherwise silently disable memory replay altogether, which is strictly
+    worse than an unserialised drain. That path warns and proceeds.
+    """
+    path = _drain_lock_path()
+    fd = None
+    try:
+        os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
+        fd = os.open(path, os.O_CREAT | os.O_RDWR, 0o600)
+    except OSError as e:
+        if fd is not None:
+            os.close(fd)
+        print(
+            f"[Hindsight] drain_pending: cannot open drain lock {path} ({e}); "
+            f"draining WITHOUT serialisation",
+            file=sys.stderr,
+        )
+        yield True
+        return
+    try:
+        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except OSError:
+        os.close(fd)
+        yield False
+        return
+    try:
+        yield True
+    finally:
+        try:
+            fcntl.flock(fd, fcntl.LOCK_UN)
+        finally:
+            os.close(fd)
+
+
+def _attempt_ceiling() -> int:
+    """Attempts after which an entry is PARKED rather than retried again.
+
+    Floored at ``MAX_ATTEMPTS``: a ceiling below the attempt budget would
+    park entries the demotion logic is still trying to drain normally.
+    """
+    return _env_num(
+        "HINDSIGHT_DRAIN_ATTEMPT_CEILING",
+        MAX_ATTEMPTS * ATTEMPT_CEILING_MULTIPLE,
+        int,
+        lo=MAX_ATTEMPTS,
+    )
+
+
+def _circuit_broken(entry: dict) -> bool:
+    """Has this entry failed so many times that retrying it is just waste?"""
+    try:
+        return int(entry.get("attempt_count", 0)) >= _attempt_ceiling()
+    except (TypeError, ValueError):
+        # Same rule as ``_over_budget``: a corrupt counter must never decide
+        # policy and must never raise on the drain path. Treat it as live.
+        return False
+
+
+def _park_broken(
+    entries: list[tuple[str, dict]], summary: dict, force: bool
+) -> list[tuple[str, dict]]:
+    """Drop circuit-broken entries from a RETAIN pass, recording the count.
+
+    Applied to the POST paths only. The free reconcile pass
+    (``_reconcile_phase``) still sweeps every entry on disk, so a parked
+    entry whose document did land is still retired without a POST — parking
+    withholds the expensive retry, not the cheap proof.
+    """
+    if force:
+        return entries
+    live: list[tuple[str, dict]] = []
+    parked = 0
+    for path, entry in entries:
+        if _circuit_broken(entry):
+            parked += 1
+        else:
+            live.append((path, entry))
+    summary["parked"] = parked
+    return live
+
 
 def _clamp(timeout: int, budget: float, started: float) -> int:
     """Per-REQUEST HTTP timeout: ``timeout`` capped by the budget LEFT NOW.
@@ -381,6 +600,10 @@ def _retry_one(entry: dict, timeout: int) -> None:
         tags=entry.get("tags"),
         timeout=timeout,
         async_processing=False,
+        # `.get` (not `[...]`): entries queued by a pre-#observation_scopes build
+        # are on disk RIGHT NOW and carry no such key. They must drain, not
+        # KeyError — a crash here loses the queue entry's only copy of a turn.
+        observation_scopes=entry.get("observation_scopes"),
     )
 
 
@@ -552,6 +775,14 @@ def _new_summary() -> dict:
         "drained": 0,
         "retried": 0,
         "dead": 0,
+        # Entries past `_attempt_ceiling()` — skipped by the retain pass so an
+        # unattended sidecar cannot spend a shared LLM lane re-POSTing the same
+        # unsaveable entry forever. Still queued, still reconciled for free,
+        # still drained in full under `--force`. See the module docstring.
+        "parked": 0,
+        # True when another drain already held the lock, so THIS run did
+        # nothing at all. Distinguishes "no work" from "did not run".
+        "skipped_locked": False,
         "reconciled": 0,
         "unknown": 0,
         # Presence WAS established, but the entry could not be moved into
@@ -566,6 +797,14 @@ def _new_summary() -> dict:
         "dead_relocated": 0,
         "resplit": 0,
         "resplit_parts": 0,
+        # Labels of the pre-drain phases ("0"/"0b"/"0c") that raised and were
+        # stepped over this run — see `_phase_failed`. A LIST, not a count,
+        # because which phase broke is what an operator needs. IN THE GATE,
+        # not only in the summary line, for exactly the reason `parked` is
+        # (see `main()`): every pre-drain counter reads 0 both on a run whose
+        # phases all blew up and on a clean, empty, healthy queue, so without
+        # this key the two runs are byte-identical from outside.
+        "phase_failures": [],
         "stalled": False,
         "budget_exceeded": False,
     }
@@ -587,17 +826,28 @@ def drain(
     backlog: bool = False,
     phase: str = "both",
     dry_run: bool = False,
+    force: bool = False,
 ) -> dict:
     """Walk the pending-retains directory and retry each entry.
 
     ``backlog=False`` (default) is the bounded in-hook drain.
     ``backlog=True`` is the operator backlog replay — see ``drain_backlog()``.
+    ``force=True`` ignores the per-entry circuit breaker (module docstring).
+
+    THE SERIALISATION POINT for every drain path — the SessionStart hook, the
+    `hindsight-drain` sidecar and the operator's out-of-band replay all reach
+    the queue through this function, so the lock is taken here rather than in
+    any one caller. A run that finds the lock held does NOTHING and returns a
+    zero summary with ``skipped_locked=True``; it never queues behind the
+    holder, because every caller has a next tick and none has time to wait.
 
     Returns a summary dict::
 
         {"drained": int,   # successful retries (entries archived)
          "retried": int,   # failures kept for next session
          "dead":    int,   # entries promoted to .dead this run
+         "parked":  int,   # past the attempt ceiling: not retried this run
+         "skipped_locked": bool,  # another drain held the lock; did nothing
          "reconciled": int,# already durable, archived without a POST
          "unknown": int,   # presence unknown, left queued
          "archive_failed": int,  # durable, but the archive was unwritable
@@ -606,12 +856,32 @@ def drain(
          "dead_relocated": int,  # legacy .dead markers moved out of the queue dir
          "resplit":   int, # over-bound entries split into drainable parts
          "resplit_parts": int,  # parts those entries became
+         "phase_failures": list[str],  # pre-drain phases that raised and
+                                       # were stepped over (backlog only)
          "stalled": bool,  # stall guard tripped
          "budget_exceeded": bool}
     """
     config = config or load_config()
-    if backlog:
-        return _drain_backlog_impl(config, phase=phase, dry_run=dry_run)
+    with _exclusive_drain() as acquired:
+        if not acquired:
+            summary = _new_summary()
+            summary["skipped_locked"] = True
+            print(
+                f"[Hindsight] drain_pending: another drain holds "
+                f"{_drain_lock_path()} — skipping this run so the same entry "
+                f"is not retained twice.",
+                file=sys.stderr,
+            )
+            return summary
+        if backlog:
+            return _drain_backlog_impl(
+                config, phase=phase, dry_run=dry_run, force=force
+            )
+        return _drain_inhook_impl(config, force=force)
+
+
+def _drain_inhook_impl(config: dict, force: bool = False) -> dict:
+    """The bounded SessionStart drain. The caller holds the drain lock."""
     timeout = _per_entry_timeout()
     budget = _budget_seconds()
     started = time.monotonic()
@@ -622,9 +892,12 @@ def drain(
     # behind them — see ``_drain_order``. Matters even more here than in the
     # backlog drain: the in-hook budget is ~4s, so a single entry at the head
     # that always burns its clamped timeout consumes the entire run.
-    entries = _drain_order(iter_entries())
+    entries = _park_broken(_drain_order(iter_entries()), summary, force)
     if not entries:
-        debug_log(config, "drain_pending: queue empty")
+        debug_log(
+            config,
+            f"drain_pending: nothing to retry (parked={summary['parked']})",
+        )
         return summary
 
     debug_log(config, f"drain_pending: {len(entries)} entries to retry")
@@ -742,6 +1015,47 @@ def _blog(msg: str) -> None:
     print(f"[Hindsight] drain_pending(backlog): {msg}", file=sys.stderr)
 
 
+def _phase_failed(summary: dict, phase: str, e: BaseException, cost: str) -> None:
+    """Record a pre-drain phase that raised, and let the drain continue.
+
+    ONE mechanism for phases 0, 0b and 0c (#3894 / #3895). They sit in the same
+    ``if not dry_run:`` block and run BEFORE the phases that actually
+    drain, so a raise out of any of them takes phases 1 and 2 with it and
+    the backlog becomes immortal — for every OTHER entry too, none of
+    which had anything to do with the failure. Phases 0 and 0c PARSE
+    queued entries, so a semantically-malformed-but-valid-JSON entry
+    reaches both — precisely the class ``iter_entries``' quarantine cannot
+    catch. Phase 0b parses nothing and its guard is defence-in-depth on
+    the caller contract; the comment at its call site says so plainly.
+    None of the three is loss-bearing to skip — a collapse, a sweep and a
+    re-split only MOVE or REWRITE bytes already on disk, and none of them
+    deletes — so all three are logged and stepped over, not fatal.
+
+    Reported in TWO places on purpose. The stderr line is for whoever is
+    watching the run; ``summary["phase_failures"]`` is what makes the
+    failure survive into the return value, into ``main()``'s summary line,
+    and into its "anything to report?" gate. Without the second, a run
+    where every pre-drain phase blew up prints exactly what an empty,
+    healthy queue prints — every pre-drain counter is 0 either way — and a
+    permanently broken phase is invisible to anything not tailing stderr.
+    Same argument, same shape, as ``parked`` (#3893).
+
+    ``cost`` states what skipping this phase actually costs, in the same
+    voice as the phase's own log lines: the point of the message is that
+    the operator can tell at a glance that no memory was lost.
+
+    The message format is deliberately the one #3894 wrote inline for its
+    phase-0 guard, so folding that guard onto this helper changed no
+    output at all. Three hand-rolled ``except`` blocks in one block is
+    exactly the shape that drifts.
+    """
+    summary["phase_failures"].append(phase)
+    _blog(
+        f"phase {phase} FAILED ({type(e).__name__}: {e}) — continuing to the "
+        f"phases that actually drain. Nothing was lost: {cost}"
+    )
+
+
 def _reconcile_phase(config: dict, summary: dict, dry_run: bool) -> None:
     """PHASE 1 — free pass: drop entries whose document already exists.
 
@@ -822,7 +1136,7 @@ def _wait_for_upstream(backoff_ms: int, started: float, budget: float) -> bool:
 
 
 def _drain_backlog_impl(
-    config: dict, phase: str = "both", dry_run: bool = False
+    config: dict, phase: str = "both", dry_run: bool = False, force: bool = False
 ) -> dict:
     """Concurrent-capable, long-budget, three-phase backlog replay."""
     summary = _new_summary()
@@ -840,8 +1154,30 @@ def _drain_backlog_impl(
     # the SessionStart drain's whole contract is a hard wall-clock ceiling
     # on hook latency. New duplicates cannot accumulate there anyway:
     # `pending.enqueue`'s filename-keyed guard stops those at the producer.
+    #
+    # GUARDED, because phase 0 runs BEFORE the phases that actually drain
+    # (#3688 review R1-M1). `iter_entries()` quarantines an unparsable entry
+    # precisely so no single corrupt file can make the whole queue immortal;
+    # an unguarded collapse re-creates that failure one level up, where a
+    # semantically-malformed-but-valid-JSON entry raises and takes phases 1
+    # and 2 with it — a strict regression against the pre-#3688 drain, where
+    # the same entry drains. A collapse failure costs duplicated work, never
+    # a memory, so it is logged and stepped over, exactly as every other
+    # phase treats a per-entry failure. `_attempt_count` closes the one
+    # measured raise; this guard is what keeps the NEXT one from being a
+    # fleet-wide drain stall.
     if not dry_run:
-        summary["collapsed"] = collapse_duplicates()
+        try:
+            summary["collapsed"] = collapse_duplicates()
+        except Exception as e:  # noqa: BLE001 — see comment above
+            _phase_failed(
+                summary,
+                "0",
+                e,
+                "a collapse only ever MOVES a byte-identical copy into "
+                "pending-duplicate/, so the cost of skipping it is duplicated "
+                "work, not a memory.",
+            )
         if summary["collapsed"]:
             _blog(
                 f"phase 0: collapsed {summary['collapsed']} duplicate entries "
@@ -853,7 +1189,29 @@ def _drain_backlog_impl(
         # queue directory. Free, local, and an UPGRADE step: markers written
         # by an older build sit in the directory external janitors sweep, and
         # a marker is the only remaining copy of its memory.
-        moved = sweep_legacy_dead_markers()
+        #
+        # GUARDED (#3895) because it runs BEFORE the phases that drain, but
+        # be honest about WHY: unlike phase 0c this one parses no entry — it
+        # walks filenames — and it already catches `OSError` per marker,
+        # which covers `shutil.Error` too (an `OSError` subclass). No queue
+        # state reachable today makes it raise, and none was found looking.
+        # The guard is on the CALLER contract rather than a live defect:
+        # nothing running before the drain may be able to stop the drain,
+        # and any future line inside that loop re-opens the hole. Skipping
+        # the sweep is free — the markers stay exactly where every earlier
+        # build left them.
+        try:
+            moved = sweep_legacy_dead_markers()
+        except Exception as e:  # noqa: BLE001 — see `_phase_failed`
+            moved = 0
+            _phase_failed(
+                summary,
+                "0b",
+                e,
+                "a sweep only ever MOVES a `.dead` marker out of the live "
+                "queue directory, so the cost of skipping it is that the "
+                "markers stay where they already were, not a memory.",
+            )
         if moved:
             summary["dead_relocated"] = moved
             _blog(
@@ -871,7 +1229,30 @@ def _drain_backlog_impl(
         # part drainable, which is the difference between a lost memory and a
         # slow one. Runs after the duplicate collapse so a duplicated
         # over-bound entry is split ONCE, not once per copy.
-        entries_split, parts_written = resplit_over_bound_entries()
+        #
+        # GUARDED (#3895), and this is the most exposed of the pre-drain
+        # phases: it PARSES every queued entry to measure it, so a
+        # semantically-malformed-but-valid-JSON entry — the exact class
+        # `iter_entries`' quarantine hands over rather than quarantines —
+        # reaches it. Measured: one entry whose `metadata` is a string
+        # instead of a mapping raises `ValueError` out of the payload
+        # rebuild. Unguarded, that ONE entry stops phases 1 and 2 for every
+        # other entry in the queue, on every run. Skipping the split costs
+        # the over-bound entries only: they stay exactly as undrainable as
+        # they were before phase 0c existed, and nothing is deleted.
+        try:
+            entries_split, parts_written = resplit_over_bound_entries()
+        except Exception as e:  # noqa: BLE001 — see `_phase_failed`
+            entries_split, parts_written = 0, 0
+            _phase_failed(
+                summary,
+                "0c",
+                e,
+                "a re-split only ever REWRITES an over-bound entry into "
+                "drainable parts and archives the original, so the cost of "
+                "skipping it is that those entries stay as undrainable as "
+                "they were before phase 0c existed. Nothing is deleted.",
+            )
         if entries_split:
             summary["resplit"] = entries_split
             summary["resplit_parts"] = parts_written
@@ -897,8 +1278,18 @@ def _drain_backlog_impl(
 
     # See ``_drain_order``: without this, three budget-exhausted entries at the
     # head trip the stall guard on every run forever and the drain never makes
-    # progress again.
-    entries = _drain_order(iter_entries())
+    # progress again. Then the CIRCUIT BREAKER: demotion bounds head-of-line
+    # blocking but not total work, and this pass is the unattended one — a
+    # 3600s budget every 900s means a wedged agent would otherwise re-POST the
+    # same unsaveable entries into a 4-slot shared lane indefinitely.
+    entries = _park_broken(_drain_order(iter_entries()), summary, force)
+    if summary["parked"]:
+        _blog(
+            f"parked {summary['parked']} entries past the attempt ceiling "
+            f"({_attempt_ceiling()} attempts) — still queued and still "
+            f"reconciled for free, but not retained again. `--force` to retry "
+            f"them anyway."
+        )
     if not entries:
         _blog("phase 2: nothing left to retain")
         return summary
@@ -1045,6 +1436,13 @@ def _parse_args(argv: list[str] | None):
         action="store_true",
         help="with --backlog: report what would happen, issue no writes",
     )
+    ap.add_argument(
+        "--force",
+        action="store_true",
+        help="retry entries past the attempt ceiling too (they are skipped by "
+        "default so an unattended drain cannot spend the shared retain lane "
+        "on the same permanently-failing entry forever)",
+    )
     return ap.parse_args(argv)
 
 
@@ -1058,7 +1456,11 @@ def main(argv: list[str] | None = None) -> int:
         return 2
     config = load_config()
     summary = drain(
-        config, backlog=args.backlog, phase=args.phase, dry_run=args.dry_run
+        config,
+        backlog=args.backlog,
+        phase=args.phase,
+        dry_run=args.dry_run,
+        force=args.force,
     )
     if any(
         summary[k]
@@ -1072,6 +1474,28 @@ def main(argv: list[str] | None = None) -> int:
             "reconciled",
             "unknown",
             "archive_failed",
+            # PARKED IS IN THE GATE, not only in the line below. A run whose
+            # entire queue is past the attempt ceiling parks everything and
+            # does nothing else, so every other counter is 0 — without this
+            # key the gate stays shut and the run is completely silent: rc=0,
+            # empty stdout, empty stderr, byte-identical to "queue empty".
+            #
+            # The backlog path narrates its own parking (`_drain_backlog_impl`
+            # logs "parked N entries past the attempt ceiling"). The IN-HOOK
+            # path only `debug_log`s it, so it says nothing unless debug is on
+            # — and that is the path that runs on every session boot. Nothing
+            # else can supply the signal either: `switchroom doctor` reads the
+            # queue DIRECTORY, where a parked entry and a backlogged entry are
+            # the same file. An operator watching a queue that will not shrink
+            # could not tell "parked by the circuit breaker, needs --force"
+            # from "ordinary backlog, will drain on its own".
+            "parked",
+            # Same argument as `parked` directly above, one stage earlier in
+            # the run (#3895). A run whose pre-drain phases all raised has
+            # every pre-drain counter at 0 — identical to a clean, empty
+            # queue — so without this key a permanently broken phase
+            # 0/0b/0c prints nothing at all from the CLI.
+            "phase_failures",
         )
     ):
         print(
@@ -1083,6 +1507,8 @@ def main(argv: list[str] | None = None) -> int:
             f"retried={summary['retried']} dead={summary['dead']} "
             f"unknown={summary['unknown']} "
             f"archive_failed={summary['archive_failed']} "
+            f"parked={summary['parked']} "
+            f"phase_failures={','.join(summary['phase_failures']) or 'none'} "
             f"stalled={summary['stalled']} budget_exceeded={summary['budget_exceeded']}",
             file=sys.stderr,
         )

package/vendor/hindsight-memory/scripts/lib/client.py

@@ -177,9 +177,17 @@ class HindsightClient:
         tags: Optional[list] = None,
         timeout: int = 15,
         async_processing: bool = True,
+        observation_scopes: Optional[str] = None,
     ) -> dict:
         """Retain content into a bank's memory.
 
+        ``observation_scopes`` is a per-row Hindsight field controlling which
+        observation scope consolidation writes this item's observations into.
+        ``"shared"`` puts them in ONE global untagged scope instead of a scope
+        per tag. ``None`` (the default) omits the field entirely, so the
+        engine's own default stands and the wire body is byte-identical to a
+        pre-``observation_scopes`` client.
+
         By default posts with ``async=true`` so the server processes extraction
         in the background (a 200 is an ack-of-receipt, not proof of durable
         persistence). The context field helps Hindsight cluster memories by
@@ -253,6 +261,7 @@ class HindsightClient:
                 tags=tags,
                 timeout=part_timeout,
                 async_processing=async_processing,
+                observation_scopes=observation_scopes,
             )
         if total > 1 and isinstance(response, dict):
             response = dict(response)
@@ -269,6 +278,7 @@ class HindsightClient:
         tags: Optional[list],
         timeout: int,
         async_processing: bool,
+        observation_scopes: Optional[str] = None,
     ) -> dict:
         """POST exactly one retain item. Raises on any HTTP/transport error."""
         path = f"/v1/default/banks/{urllib.parse.quote(bank_id, safe='')}/memories"
@@ -281,6 +291,10 @@ class HindsightClient:
             item["context"] = context
         if tags:
             item["tags"] = tags
+        # Omitted entirely when unset — the engine default must stay in force
+        # and the body must match a pre-observation_scopes client byte for byte.
+        if observation_scopes:
+            item["observation_scopes"] = observation_scopes
         body = {
             "items": [item],
             "async": bool(async_processing),