ctrlrelay 0.20.0__tar.gz → 0.20.2__tar.gz

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/CHANGELOG.md

@@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.20.2] - 2026-11-04
+
+### Fixed
+
+- A node no longer stops taking work when the agent CLI reports its usage
+  status as `allowed_warning`. That status means a usage window has passed
+  a warning line, and calls are still served; it was treated as a refusal,
+  so nodes at 26% of the 7-day window paused until that window reset, a
+  week away. How close a window is to its limit is still decided by the
+  pause threshold, and any other status still pauses (#338).
+
+## [0.20.1] - 2026-11-04
+
+### Fixed
+
+- The status page's server panel reads memory and uptime on macOS too. It
+  read those two from `/proc`, which macOS does not have, so on a Mac those
+  rows were empty and the page printed the read error. On macOS it now
+  uses `sysctl` and `vm_stat`. macOS has no `MemAvailable`, so "available"
+  there is free plus inactive pages: an estimate, and not the same
+  measurement as Linux's (#321).
+
+- A run waiting for the operator's answer is no longer lost when the poller
+  stops during the wait. The question is now saved, and the session marked
+  `blocked`, before the wait starts rather than after it ends (if that
+  write fails it is logged and the wait goes ahead as before). If the
+  poller stops, a later answer in chat reaches the saved question and
+  resumes the same agent session; when the poller starts again it drops the
+  repo lock such a run held, if the process that ran it is gone, so the
+  repo is not wedged until then. An answer that arrives during the wait
+  still resumes the run at once and closes the saved question; if a reply
+  already answered the saved question first, that first answer stands and
+  the run is resumed once, with it. A planning session's question resumes
+  the plan, not the code. The claim on the issue is not marked held while the
+  question waits, so after a stop it goes stale as before (#324).
+
+- An issue is read again from GitHub just before its run starts. A poll
+  returns a batch that is handled one issue at a time, so with one slot the
+  last issue could start an hour after the poll saw it, after its trigger
+  label was removed or the issue was closed. Now an issue that is closed,
+  has gained an `exclude_labels` label, or no longer meets any trigger is
+  skipped and left for the next poll to judge; so is one that cannot be read.
+  The run is routed on the labels read then, not the poll's. Who added a
+  trigger is not asked again: a trigger the operator removed and someone
+  else put back in that window still counts (#326).
+
 ## [0.20.0] - 2026-11-04
 
 ### Added
@@ -2313,7 +2359,9 @@ pipeline).
   per-phase implementation plans (Phase 0 through Phase 4).
 - `docs/Claude_Code_Project_Guide.md` — project development guide.
 
-[Unreleased]: https://github.com/AInvirion/ctrlrelay/compare/v0.20.0...HEAD
+[Unreleased]: https://github.com/AInvirion/ctrlrelay/compare/v0.20.2...HEAD
+[0.20.2]: https://github.com/AInvirion/ctrlrelay/compare/v0.20.1...v0.20.2
+[0.20.1]: https://github.com/AInvirion/ctrlrelay/compare/v0.20.0...v0.20.1
 [0.20.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.19.1...v0.20.0
 [0.19.1]: https://github.com/AInvirion/ctrlrelay/compare/v0.19.0...v0.19.1
 [0.19.0]: https://github.com/AInvirion/ctrlrelay/compare/v0.18.0...v0.19.0

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ctrlrelay
-Version: 0.20.0
+Version: 0.20.2
 Summary: Local-first orchestrator for headless coding agents across multiple GitHub repos
 Project-URL: Homepage, https://github.com/AInvirion/ctrlrelay
 Project-URL: Documentation, https://ainvirion.github.io/ctrlrelay/

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ctrlrelay"
-version = "0.20.0"
+version = "0.20.2"
 description = "Local-first orchestrator for headless coding agents across multiple GitHub repos"
 readme = "README.md"
 requires-python = ">=3.12"

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/cli.py

@@ -3801,7 +3801,8 @@ def poller_start(
                                 )
                                 try:
                                     state_db.mark_pending_resume_resumed(
-                                        session_id
+                                        session_id, answered_at=row["answered_at"],
+                                        answer=row["answer"],
                                     )
                                 except Exception:
                                     pass
@@ -3844,7 +3845,8 @@ def poller_start(
                             )
                             try:
                                 state_db.mark_pending_resume_resumed(
-                                    session_id
+                                    session_id, answered_at=row["answered_at"],
+                                    answer=row["answer"],
                                 )
                             except Exception:
                                 pass
@@ -3862,7 +3864,10 @@ def poller_start(
                         # same broken row. Operator can inspect via the
                         # sessions table.
                         try:
-                            state_db.mark_pending_resume_resumed(session_id)
+                            state_db.mark_pending_resume_resumed(
+                                session_id, answered_at=row["answered_at"],
+                                answer=row["answer"],
+                            )
                         except Exception:
                             pass
                         continue
@@ -3926,7 +3931,10 @@ def poller_start(
                         continue
 
                     try:
-                        state_db.mark_pending_resume_resumed(session_id)
+                        state_db.mark_pending_resume_resumed(
+                            session_id, answered_at=row["answered_at"],
+                            answer=row["answer"],
+                        )
                     except Exception:
                         pass
 
@@ -4224,6 +4232,12 @@ def poller_start(
                     f"[yellow]Run {r['session_id']} for {where} stopped "
                     f"when the relay did; marked failed.{kept}[/yellow]"
                 )
+            # A run that stopped while waiting for an answer (#324) is
+            # `blocked` with its question saved, and is resumed when the
+            # answer comes; the repo lock it held would only wedge the
+            # repo until then.
+            from ctrlrelay.core.orphans import release_dead_blocked_locks
+            release_dead_blocked_locks(state_db)
 
             try:
                 await github_budget.refresh()

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/core/github.py

@@ -561,6 +561,19 @@ class GitHubCLI:
         )
         return json.loads(output)
 
+    async def get_issue_gate(self, repo: str, issue_number: int) -> dict[str, Any]:
+        """Return ``{"state", "labels", "assignees"}`` for one issue, read
+        live over REST: the fields that decide whether the relay may still
+        start it (#326). ``labels`` and ``assignees`` are lists of names.
+        """
+        output = await self._run_gh(
+            "api", f"repos/{repo}/issues/{issue_number}",
+            "--jq",
+            "{state: .state, labels: [.labels[].name], "
+            "assignees: [.assignees[].login]}",
+        )
+        return json.loads(output)
+
     async def create_pr(
         self,
         repo: str,

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/core/orphans.py

@@ -386,3 +386,48 @@ async def reap_orphaned_sessions(
             "handed_back": handed_back,
         })
     return settled
+
+
+def release_dead_blocked_locks(
+    state_db: Any, *, is_alive: Callable[[int], bool] = owner_is_alive,
+) -> list[str]:
+    """Drop repo locks held by ``blocked`` runs whose process is gone (#324).
+
+    A run that asks a question is ``blocked`` with its question saved
+    while its process waits in memory for the answer, and it keeps its
+    repo lock for that wait. If the process stops, the lock only wedges
+    the repo until the answer comes, and the resume takes the lock again
+    itself. Runs this process started, runs whose process is alive (a
+    ``ctrlrelay run dev`` waiting in another terminal), and rows that name
+    no process are left alone, on the same terms as
+    :func:`reap_orphaned_sessions`. Returns the session ids released.
+    """
+    try:
+        rows = state_db.list_blocked_sessions_holding_locks()
+    except Exception as e:
+        log_event(
+            _logger, "orphans.blocked_locks_list_failed",
+            reason=type(e).__name__, error=str(e)[:200],
+        )
+        return []
+    released: list[str] = []
+    for row in rows:
+        pid = row.get("owner_pid")
+        if row.get("owner_run") == PROCESS_RUN or pid is None:
+            continue
+        if is_alive(int(pid)):
+            continue
+        try:
+            if state_db.release_blocked_session_lock(row["id"]):
+                released.append(row["id"])
+                log_event(
+                    _logger, "orphans.blocked_lock_released",
+                    session_id=row["id"], repo=row.get("repo"),
+                )
+        except Exception as e:
+            log_event(
+                _logger, "orphans.blocked_lock_release_failed",
+                session_id=row["id"], reason=type(e).__name__,
+                error=str(e)[:200],
+            )
+    return released

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/core/poller.py

@@ -954,6 +954,84 @@ class IssuePoller:
                 return True
         return False
 
+    async def recheck_before_dispatch(
+        self, repo: str, issue: dict[str, Any],
+    ) -> dict[str, Any] | None:
+        """Read the issue again just before it is handled (#326).
+
+        ``poll()`` returns a batch and the loop handles it one issue at a
+        time, so with one slot and long runs the last issue in a batch can
+        start an hour after the poll saw it. In that time the operator may
+        have removed the trigger label or closed the issue, and the run
+        would start anyway.
+
+        Returns the issue with today's ``state``, ``labels`` and
+        ``assignees`` when it may still start. Returns ``None`` when it is
+        closed, now carries an ``exclude_labels`` label, no longer meets
+        any trigger, or could not be read; then it is also taken out of
+        ``seen_issues``, so the next poll judges it afresh rather than
+        never again. Whether the operator was the one who added a label or
+        assignment is not asked again: that was settled by ``poll()``.
+        """
+        number = int(issue["number"])
+        try:
+            live = await self.github.get_issue_gate(repo, number)
+            state = str(live["state"]).lower()
+            labels = [str(n) for n in live["labels"]]
+            assignees = [str(n) for n in live["assignees"]]
+        except asyncio.CancelledError:
+            raise
+        except Exception as e:
+            self.unmark_seen(repo, number)
+            log_event(
+                _logger, "poll.issue.recheck_failed", repo=repo,
+                issue_number=number, reason=type(e).__name__,
+                error=str(e)[:200],
+            )
+            return None
+
+        fresh = {
+            **issue,
+            "state": state,
+            "labels": [{"name": n} for n in labels],
+            "assignees": [{"login": n} for n in assignees],
+        }
+        reason: str | None = None
+        if state != "open":
+            reason = "closed"
+        else:
+            excluded = self._matched_exclude_label(fresh, {
+                x.lower() for x in self.exclude_labels_by_repo.get(repo, [])
+            })
+            include_lowered = {
+                x.lower() for x in self.include_labels_by_repo.get(repo, [])
+            }
+            require_lowered = {
+                x.lower() for x in self.require_labels_by_repo.get(repo, [])
+            }
+            # Logins compared without case, as the assignee search that
+            # admitted the issue does: a case-only mismatch would skip the
+            # issue on every poll and never start it.
+            me = (self.username or "").lower()
+            assigned = bool(me) and me in {n.lower() for n in assignees} and (
+                not require_lowered
+                or self._matched_require_label(fresh, require_lowered)
+                is not None
+            )
+            labelled = self._matched_include_label(fresh, include_lowered)
+            if excluded is not None:
+                reason = f"excluded_by_label:{excluded}"
+            elif not assigned and labelled is None:
+                reason = "trigger_gone"
+        if reason is not None:
+            self.unmark_seen(repo, number)
+            log_event(
+                _logger, "poll.issue.stale_at_dispatch", repo=repo,
+                issue_number=number, reason=reason,
+            )
+            return None
+        return fresh
+
     def mark_seen(self, repo: str, issue_number: int) -> None:
         """Mark an issue as seen without triggering a poll.
 
@@ -1310,7 +1388,15 @@ async def run_poll_loop(
         # would otherwise be silently dropped until daemon restart.
         for item in new_issues:
             try:
-                await handler(item["repo"], item["issue"])
+                # The batch can be an hour old by the time an item's turn
+                # comes (#326): read it again, and skip it if it may no
+                # longer start.
+                fresh = await poller.recheck_before_dispatch(
+                    item["repo"], item["issue"],
+                )
+                if fresh is None:
+                    continue
+                await handler(item["repo"], fresh)
             except asyncio.CancelledError:
                 raise
             except Exception as e:

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/core/state.py

@@ -1021,6 +1021,117 @@ class StateDB:
         )
         self._conn.commit()
 
+    def park_blocked_question(
+        self, session_id: str, pipeline: str, repo: str, question: str,
+    ) -> None:
+        """Record a question BEFORE the run waits on the chat answer (#324).
+
+        The session goes ``running`` -> ``blocked`` and its question gets a
+        ``pending_resumes`` row, in one commit. While the poller waits in
+        process, this is only a record; if the poller stops during the
+        wait, it is what the bridge routes a late answer to and what the
+        sweeper resumes. Without it, the only record of the question was a
+        coroutine, and a restart lost it with the session row left
+        ``running``.
+        """
+        try:
+            self._conn.execute(
+                "UPDATE sessions SET status = 'blocked' "
+                "WHERE id = ? AND status = 'running'",
+                (session_id,),
+            )
+            # Commits both statements.
+            self.add_pending_resume(
+                session_id=session_id, pipeline=pipeline, repo=repo,
+                question=question,
+            )
+        except BaseException:
+            # Never leave the half-done write open through the wait: it
+            # would hold the database's write lock against the bridge, and
+            # a later commit would keep only half of it.
+            if self._conn.in_transaction:
+                self._conn.rollback()
+            raise
+
+    def take_parked_answer(self, session_id: str, answer: str) -> bool:
+        """The in-process wait got its answer (#324): mark the parked row
+        answered AND resumed, so neither the bridge nor the sweeper acts on
+        it again, and put the session back to ``running``. One commit.
+
+        Only an unanswered, open row is taken. Returns ``False`` when no
+        such row is there, and changes nothing: the row may have been
+        answered some other way first (a reply the bridge routed to the
+        saved question), expired, or be missing. The caller tells these
+        apart; only the first means another answer stands."""
+        now = int(time.time())
+        ours = not self._conn.in_transaction
+        try:
+            cursor = self._conn.execute(
+                "UPDATE pending_resumes SET answer = ?, answered_at = ?, "
+                "resumed_at = ? WHERE session_id = ? AND answered_at IS NULL "
+                "AND resumed_at IS NULL AND expired_at IS NULL",
+                (answer, now, now, session_id),
+            )
+            if cursor.rowcount > 0:
+                self._conn.execute(
+                    "UPDATE sessions SET status = 'running' "
+                    "WHERE id = ? AND status = 'blocked'",
+                    (session_id,),
+                )
+            if ours:
+                self._conn.commit()
+        except BaseException:
+            if ours and self._conn.in_transaction:
+                self._conn.rollback()
+            raise
+        return cursor.rowcount > 0
+
+    def mark_session_running(self, session_id: str) -> None:
+        """Put a ``blocked`` session back to ``running`` (#324): its run
+        resumed in process with an answer the saved question did not
+        record (that row expired or is missing)."""
+        self._conn.execute(
+            "UPDATE sessions SET status = 'running' "
+            "WHERE id = ? AND status = 'blocked'",
+            (session_id,),
+        )
+        self._conn.commit()
+
+    def has_open_pending_resume(self, session_id: str, question: str) -> bool:
+        """Whether this session already has a row for this same question
+        that is neither resumed nor expired. Re-writing such a row would
+        clear an answer already attached to it (#324)."""
+        row = self._conn.execute(
+            "SELECT 1 FROM pending_resumes WHERE session_id = ? "
+            "AND question = ? AND resumed_at IS NULL AND expired_at IS NULL",
+            (session_id, question),
+        ).fetchone()
+        return row is not None
+
+    def list_blocked_sessions_holding_locks(self) -> list[dict[str, Any]]:
+        """``blocked`` sessions that still hold a repo lock, with the
+        process that ran them (#324). A blocked run holds its lock only
+        while that process waits in memory for the answer; see
+        :func:`ctrlrelay.core.orphans.release_dead_blocked_locks`."""
+        rows = self._conn.execute(
+            "SELECT s.id, s.repo, s.owner_pid, s.owner_run "
+            "FROM repo_locks l JOIN sessions s ON s.id = l.session_id "
+            "WHERE s.status = 'blocked'"
+        ).fetchall()
+        return [dict(r) for r in rows]
+
+    def release_blocked_session_lock(self, session_id: str) -> bool:
+        """Drop the lock of a session that is still ``blocked`` (#324).
+        Guarded on the status, so a session resumed in the meantime keeps
+        its lock."""
+        cursor = self._conn.execute(
+            "DELETE FROM repo_locks WHERE session_id = ? AND EXISTS ("
+            "SELECT 1 FROM sessions WHERE id = ? AND status = 'blocked')",
+            (session_id, session_id),
+        )
+        self._conn.commit()
+        return cursor.rowcount > 0
+
     def get_oldest_unanswered_pending_resume(
         self,
         pipeline: str | None = None,
@@ -1186,7 +1297,12 @@ class StateDB:
         ).fetchall()
         return [dict(row) for row in rows]
 
-    def mark_pending_resume_resumed(self, session_id: str) -> bool:
+    def mark_pending_resume_resumed(
+        self,
+        session_id: str,
+        answered_at: int | None = None,
+        answer: str | None = None,
+    ) -> bool:
         """Mark a pending resume as executed so the sweeper doesn't pick
         it up again.
 
@@ -1201,12 +1317,27 @@ class StateDB:
         wedging the session. The guard makes this a no-op when the
         row has been refreshed.
 
+        ``answered_at`` and ``answer``, when given, are those of the row
+        the caller took (#324). Since the question is saved before the
+        wait, a resume that re-blocks can have its NEW question answered
+        (through the bridge) before this call; matching the answer and its
+        time keeps this call from marking that new answer resumed, which
+        would hide it for ever.
+
         Returns True iff a row was updated."""
-        cursor = self._conn.execute(
-            "UPDATE pending_resumes SET resumed_at = ? "
-            "WHERE session_id = ? AND answered_at IS NOT NULL",
-            (int(time.time()), session_id),
-        )
+        if answered_at is None:
+            cursor = self._conn.execute(
+                "UPDATE pending_resumes SET resumed_at = ? "
+                "WHERE session_id = ? AND answered_at IS NOT NULL",
+                (int(time.time()), session_id),
+            )
+        else:
+            cursor = self._conn.execute(
+                "UPDATE pending_resumes SET resumed_at = ? "
+                "WHERE session_id = ? AND answered_at = ? "
+                "AND answer IS ?",
+                (int(time.time()), session_id, answered_at, answer),
+            )
         self._conn.commit()
         return cursor.rowcount > 0
 

{ctrlrelay-0.20.0 → ctrlrelay-0.20.2}/src/ctrlrelay/core/usage.py

@@ -43,6 +43,14 @@ UNKNOWN_RESET_PAUSE_SECONDS = 3600
 # recursion limit (RecursionError). Any of them is unreadable, not fatal.
 _UNPARSEABLE = (ValueError, RecursionError)
 ALLOWED = "allowed"
+# Statuses that still let a run through. The agent CLI reports
+# ``allowed_warning`` once a window passes a warning line well short of
+# its limit (seen at 26% of the 7-day window), and the call is served.
+# Pausing on it idled machines for the rest of the 7-day window, so it
+# counts as allowed; how close a window is to its limit is the
+# threshold's question, not the status's. Any other status, including
+# one not seen before, still pauses.
+RUNNABLE_STATUSES = frozenset({ALLOWED, "allowed_warning"})
 
 
 @dataclass(frozen=True)
@@ -225,7 +233,7 @@ def pause_for(
 ) -> Pause | None:
     """Whether to stop taking new work.
 
-    Paused when the status is not ``allowed`` or a window is at or above
+    Paused when the status is not one of ``RUNNABLE_STATUSES`` or a window is at or above
     ``threshold``, until that window's reset (for the status, the event's
     own reset). A reading whose reset time has passed says nothing about
     now, so it does not pause. When the reset time is missing or cannot
@@ -243,7 +251,7 @@ def pause_for(
     # From the reading, or from now if the reading claims to be from the
     # future (a clock set back): an hour, never longer.
     fallback = int(min(reading.read_at, now)) + UNKNOWN_RESET_PAUSE_SECONDS
-    if reading.status != ALLOWED:
+    if reading.status not in RUNNABLE_STATUSES:
         until = reading.resets_at if reading.resets_at is not None else fallback
         hits.append((f"usage status is {reading.status}", until))
     for name, w in reading.windows.items():