ctrlrelay 0.1.7__tar.gz → 0.1.8__tar.gz

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/CHANGELOG.md

@@ -7,6 +7,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.8] - 2026-04-21
+
+The "reply to BLOCKED in Telegram and it actually resumes" release.
+Two operator-visibility fixes surfaced from running a 79-repo secops
+sweep: a noisy log-spam issue and a silently-dropped-reply issue. The
+latter turned into a proper resume flow.
+
+### Added
+
+- **Resume BLOCKED secops via Telegram reply.** When a scheduled
+  secops sweep escalates BLOCKED and exits, the question is now
+  persisted in a new `pending_resumes` table. Replying in Telegram
+  matches against that table and queues the answer; a new per-minute
+  `pending_resume_sweeper` scheduler job inside the poller drains
+  answered rows — re-acquires the repo lock, re-creates the worktree,
+  calls `SecopsPipeline.resume(ctx, answer)`, and Telegrams the
+  result (success / re-blocked / failed). First reply-to-resume
+  round-trip is ≤60s.
+- **Disambiguation when multiple BLOCKED sessions exist.** Replying
+  "merge it" when both `repoA` and `repoB` are blocked used to route
+  to FIFO (wrong repo, possibly destructive). The bridge now refuses
+  to guess: with >1 unanswered BLOCKED sessions it returns a Telegram
+  list of pending session_ids so the operator can reply with one
+  included. Single-BLOCKED case stays unambiguous.
+
+### Fixed
+
+- **Poller log spam on issues-disabled repos.** Repos with GitHub's
+  Issues feature disabled (template repos, signature repos, GitHub
+  Pages sites) returned a permanent `GitHubError(... has disabled
+  issues)` that the poller classified as transient, retrying every
+  120s and escalating to WARNING after 3 cycles. `poll()` and
+  `seed_current()` now detect the specific error, mark the repo in
+  an in-memory permanent-skip set, log once at INFO as
+  `poll.repo.issues_disabled`, and skip the `gh` call on subsequent
+  cycles. Resets on daemon restart.
+- **Orphan Telegram replies silently dropped.** When a BLOCKED
+  session had already torn down (scheduled secops), the bridge's
+  in-memory `_pending_questions` entry died with the ASK socket and
+  the operator's reply disappeared with just an `info` log line. The
+  bridge now replies via Telegram so the failure is visible (and,
+  with the resume flow above, actually lands as an answer).
+- **Pending_resumes rows no longer dropped on sweeper lock
+  contention.** When the per-minute sweeper raced the 6am scheduled
+  secops on the same repo, it used to `mark_pending_resume_resumed`
+  unconditionally and lose the queued answer. The sweeper now detects
+  the specific `"Repository locked by another session"` error and
+  leaves the row pending for the next tick.
+
+### Schema migration
+
+State DB gains a `pending_resumes` table (session_id PK, pipeline,
+repo, question, created_at, answer, answered_at, resumed_at). Two
+partial indexes: `idx_pending_resumes_unanswered` (for orphan-reply
+lookup) and `idx_pending_resumes_answered_unresumed` (for sweeper
+load). Created automatically on daemon start; no backfill needed.
+
+### Operator notes
+
+- Upgrade via `uv tool upgrade ctrlrelay` (or
+  `uv tool install ctrlrelay@latest --force` if pinned), restart
+  poller and bridge so the new sweeper schedules and the bridge
+  sees the new schema.
+- To exercise the resume-via-Telegram path: let a scheduled secops
+  escalate BLOCKED, reply to the Telegram notification with your
+  decision (or a fresh message that mentions the session_id if
+  multiple repos are BLOCKED). Expect a `✅ Answer queued` ack within
+  seconds and a result message within ~1 minute.
+- Dev pipeline resume-via-Telegram is not yet wired; the sweeper
+  skips non-secops rows.
+
 ## [0.1.7] - 2026-04-20
 
 Patch release. Fixes one drift bug in how the package reports its own

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ctrlrelay
-Version: 0.1.7
+Version: 0.1.8
 Summary: Local-first orchestrator for headless coding agents across multiple GitHub repos
 Project-URL: Homepage, https://github.com/AInvirion/ctrlrelay
 Project-URL: Documentation, https://ainvirion.github.io/ctrlrelay/

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ctrlrelay"
-version = "0.1.7"
+version = "0.1.8"
 description = "Local-first orchestrator for headless coding agents across multiple GitHub repos"
 readme = "README.md"
 requires-python = ">=3.12"

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/src/ctrlrelay/bridge/__main__.py

@@ -21,6 +21,15 @@ def main() -> None:
         help="Environment variable holding the Telegram bot token",
     )
     parser.add_argument("--chat-id", type=int, required=True, help="Telegram chat ID")
+    parser.add_argument(
+        "--state-db",
+        default=None,
+        help=(
+            "Path to the orchestrator state.db. When provided, orphan "
+            "Telegram replies route to persisted BLOCKED sessions in "
+            "pending_resumes. Required for the resume-via-Telegram flow."
+        ),
+    )
     args = parser.parse_args()
 
     bot_token = os.environ.get(args.bot_token_env)
@@ -31,11 +40,17 @@ def main() -> None:
         )
         sys.exit(2)
 
+    state_db = None
+    if args.state_db:
+        from ctrlrelay.core.state import StateDB
+        state_db = StateDB(Path(args.state_db))
+
     socket_path = Path(args.socket_path)
     server = BridgeServer(
         socket_path=socket_path,
         bot_token=bot_token,
         chat_id=args.chat_id,
+        state_db=state_db,
     )
 
     loop = asyncio.new_event_loop()
@@ -63,6 +78,11 @@ def main() -> None:
         pass
     finally:
         loop.close()
+        if state_db is not None:
+            try:
+                state_db.close()
+            except Exception:
+                pass
 
 
 if __name__ == "__main__":

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/src/ctrlrelay/bridge/server.py

@@ -9,6 +9,7 @@ import stat
 from collections import OrderedDict
 from datetime import datetime, timezone
 from pathlib import Path
+from typing import TYPE_CHECKING
 
 from ctrlrelay.bridge.protocol import (
     BridgeMessage,
@@ -20,6 +21,9 @@ from ctrlrelay.bridge.protocol import (
 from ctrlrelay.bridge.telegram_handler import TelegramHandler
 from ctrlrelay.core.obs import get_logger, hash_text, log_event
 
+if TYPE_CHECKING:
+    from ctrlrelay.core.state import StateDB
+
 _logger = get_logger("bridge.server")
 _log = logging.getLogger(__name__)
 
@@ -53,10 +57,18 @@ class BridgeServer:
         socket_path: Path,
         bot_token: str,
         chat_id: int,
+        state_db: "StateDB | None" = None,
     ) -> None:
         self.socket_path = socket_path
         self.bot_token = bot_token
         self.chat_id = chat_id
+        # Optional: when provided, orphan Telegram replies (no live
+        # _pending_question to match) are routed to the oldest unanswered
+        # BLOCKED session in state_db's pending_resumes table. The poller's
+        # pending-resume sweeper then picks up the answer and drives the
+        # actual pipeline resume. Without state_db, orphan replies still
+        # get a "didn't land" Telegram notice but nothing gets queued.
+        self.state_db = state_db
         self._server: asyncio.Server | None = None
         self._running = False
         self._telegram: TelegramHandler | None = None
@@ -250,6 +262,55 @@ class BridgeServer:
                     "bridge: incoming telegram msg with no pending question; "
                     "text=%r", text[:80],
                 )
+                # Try to route to a persisted BLOCKED session in state_db
+                # so the operator's reply actually drives a resume. Without
+                # this, the reply disappears the instant the session's ASK
+                # socket closes — which is exactly what happens when a
+                # scheduled secops sweep escalates BLOCKED and exits.
+                outcome = await self._queue_orphan_reply_as_resume_answer(text)
+                if self._telegram is not None:
+                    try:
+                        if outcome["status"] == "queued":
+                            row = outcome["row"]
+                            await self._telegram.send(
+                                "✅ Answer queued for BLOCKED session "
+                                f"`{row['session_id']}` "
+                                f"(pipeline={row['pipeline']}, "
+                                f"repo={row['repo']}).\n"
+                                "The pending-resume sweeper will drive it "
+                                "on the next tick — you'll get another "
+                                "message with the result."
+                            )
+                        elif outcome["status"] == "ambiguous":
+                            pending_list = "\n".join(
+                                f"  • `{r['session_id']}` ({r['repo']}): "
+                                f"{(r['question'] or '')[:80]}"
+                                for r in outcome["rows"]
+                            )
+                            await self._telegram.send(
+                                "⚠️ Your reply wasn't routed — multiple "
+                                "BLOCKED sessions are unanswered and your "
+                                "message didn't include a session_id to "
+                                "disambiguate.\n\n"
+                                "Pending:\n"
+                                f"{pending_list}\n\n"
+                                "Reply again with the session_id included "
+                                "(just paste it anywhere in your message)."
+                            )
+                        else:
+                            await self._telegram.send(
+                                "⚠️ Your reply wasn't routed — no active "
+                                "session is waiting on input and no "
+                                "persisted BLOCKED session is unanswered. "
+                                "To act manually, re-run the pipeline, "
+                                "e.g. `ctrlrelay run secops --repo "
+                                "<owner>/<repo>`."
+                            )
+                    except Exception as e:
+                        _log.warning(
+                            "bridge: failed to notify orphan-reply sender: %s",
+                            e,
+                        )
                 return
             self._pending_questions.pop(match.request_id, None)
 
@@ -283,3 +344,78 @@ class BridgeServer:
                 "bridge: failed to deliver ANSWER request_id=%s err=%s",
                 match.request_id, e,
             )
+
+    async def _queue_orphan_reply_as_resume_answer(
+        self, text: str
+    ) -> dict:
+        """Try to route an orphan Telegram reply to a persisted BLOCKED
+        session so the pending-resume sweeper can pick it up and drive a
+        pipeline resume.
+
+        Returns a dict with ``status`` set to one of:
+        - ``"queued"`` with ``row`` (dict) — answer was attached.
+        - ``"ambiguous"`` with ``rows`` (list[dict]) — multiple BLOCKED
+          sessions exist and the reply didn't name one, so we refuse to
+          guess. The sender is told which session_ids exist so they can
+          retry with one included.
+        - ``"none"`` — no state_db, no unanswered rows, or DB error.
+
+        Disambiguation rule: if the reply text contains exactly one of
+        the unanswered session_ids as a substring, route to that row.
+        Otherwise, with >1 unanswered rows and no substring match,
+        return ambiguous. With exactly one unanswered row and no
+        substring match, route anyway (single-repo case is unambiguous).
+        """
+        if self.state_db is None:
+            return {"status": "none"}
+        try:
+            rows = self.state_db.list_unanswered_pending_resumes()
+        except Exception as e:
+            log_event(
+                _logger,
+                "bridge.pending_resume.list_failed",
+                reason=type(e).__name__,
+                error=str(e)[:200],
+            )
+            return {"status": "none"}
+
+        if not rows:
+            return {"status": "none"}
+
+        matched_by_id = [r for r in rows if r["session_id"] in text]
+        if len(matched_by_id) == 1:
+            target = matched_by_id[0]
+        elif len(matched_by_id) > 1:
+            # Multiple session_ids named in the same reply — refuse to
+            # pick one. Let the operator send a single-session reply.
+            return {"status": "ambiguous", "rows": matched_by_id}
+        elif len(rows) == 1:
+            target = rows[0]
+        else:
+            # Multiple unanswered, no session_id hint — can't route safely.
+            return {"status": "ambiguous", "rows": rows}
+
+        try:
+            if not self.state_db.answer_pending_resume(
+                target["session_id"], text
+            ):
+                return {"status": "none"}
+        except Exception as e:
+            log_event(
+                _logger,
+                "bridge.pending_resume.update_failed",
+                reason=type(e).__name__,
+                error=str(e)[:200],
+            )
+            return {"status": "none"}
+
+        log_event(
+            _logger,
+            "bridge.pending_resume.queued",
+            session_id=target["session_id"],
+            pipeline=target["pipeline"],
+            repo=target["repo"],
+            answer_length=len(text),
+            answer_hash=hash_text(text),
+        )
+        return {"status": "queued", "row": target}

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/src/ctrlrelay/cli.py

@@ -306,10 +306,18 @@ def bridge_start(
         console.print(f"Starting bridge on {socket_path}")
         console.print("Press Ctrl+C to stop")
 
+        # Open the state DB so the bridge can route orphan Telegram replies
+        # to persisted BLOCKED sessions in pending_resumes. Both daemons
+        # share ~/.ctrlrelay/state.db — SQLite's WAL mode handles concurrent
+        # readers/writers for the low contention we see here.
+        from ctrlrelay.core.state import StateDB
+        state_db = StateDB(config.paths.state_db)
+
         server = BridgeServer(
             socket_path=socket_path,
             bot_token=bot_token,
             chat_id=telegram_config.chat_id,
+            state_db=state_db,
         )
 
         loop = asyncio.new_event_loop()
@@ -340,6 +348,10 @@ def bridge_start(
                 pass
         finally:
             loop.close()
+            try:
+                state_db.close()
+            except Exception:
+                pass
             pid_file.unlink(missing_ok=True)
     else:
         # Pass the token via environment, never argv. Putting it on the command
@@ -354,6 +366,8 @@ def bridge_start(
             telegram_config.bot_token_env,
             "--chat-id",
             str(telegram_config.chat_id),
+            "--state-db",
+            str(config.paths.state_db),
         ]
         proc = subprocess.Popen(
             cmd,
@@ -1321,6 +1335,147 @@ def poller_start(
                     except Exception:
                         pass
 
+        async def _run_pending_resume_sweeper() -> None:
+            """Drain pending_resumes rows that an operator answered via
+            Telegram while the original BLOCKED session had already torn
+            down.
+
+            Runs every minute inside the poller. For each answered row
+            it acquires the repo lock, re-creates a worktree, calls
+            ``SecopsPipeline.resume(ctx, answer)`` via the shared
+            ``resume_secops_from_pending`` helper, then marks the row
+            resumed and fans out a Telegram notification with the result
+            so the operator knows whether the resume landed.
+            """
+            try:
+                pending = state_db.list_pending_resumes_to_execute()
+            except Exception as e:
+                console.print(
+                    f"[yellow]pending_resume_sweeper: list failed ({e})"
+                    f"[/yellow]"
+                )
+                return
+            if not pending:
+                return
+
+            from ctrlrelay.pipelines.secops import resume_secops_from_pending
+
+            sweeper_transport = None
+            if config.transport.type.value == "telegram" and config.transport.telegram:
+                from ctrlrelay.transports import SocketTransport
+                sock = config.transport.telegram.socket_path.expanduser().resolve()
+                if sock.exists():
+                    try:
+                        candidate = SocketTransport(sock)
+                        await candidate.connect()
+                        sweeper_transport = candidate
+                    except Exception:
+                        sweeper_transport = None
+
+            try:
+                for row in pending:
+                    session_id = row["session_id"]
+                    repo = row["repo"]
+                    pipeline_name = row["pipeline"]
+                    answer = row["answer"] or ""
+                    if pipeline_name != "secops":
+                        # dev pipeline resume-from-pending not wired yet —
+                        # leave the row marked-answered so a later sweep
+                        # picks it up once that path lands.
+                        continue
+
+                    if sweeper_transport:
+                        try:
+                            await sweeper_transport.send(
+                                f"🔁 Resuming BLOCKED session "
+                                f"`{session_id}` on {repo} with your "
+                                f"answer..."
+                            )
+                        except Exception:
+                            pass
+
+                    try:
+                        result = await resume_secops_from_pending(
+                            session_id=session_id,
+                            repo=repo,
+                            answer=answer,
+                            dispatcher=dispatcher,
+                            github=github,
+                            worktree=worktree,
+                            dashboard=scheduled_dashboard,
+                            state_db=state_db,
+                            transport=sweeper_transport,
+                            contexts_dir=config.paths.contexts,
+                        )
+                    except Exception as e:
+                        if sweeper_transport:
+                            try:
+                                await sweeper_transport.send(
+                                    f"❌ Resume of `{session_id}` on "
+                                    f"{repo} crashed: {e}"
+                                )
+                            except Exception:
+                                pass
+                        # Mark resumed so the sweeper doesn't hot-loop the
+                        # same broken row. Operator can inspect via the
+                        # sessions table.
+                        try:
+                            state_db.mark_pending_resume_resumed(session_id)
+                        except Exception:
+                            pass
+                        continue
+
+                    # Lock-contention is retryable: the 6am secops cron or
+                    # an in-flight dev session holds the repo lock. Leave
+                    # the pending_resumes row as-is so the next sweeper
+                    # tick tries again. Without this guard the operator's
+                    # queued answer is silently dropped.
+                    if not result.success and result.error == (
+                        "Repository locked by another session"
+                    ):
+                        console.print(
+                            "[dim]pending_resume_sweeper: "
+                            f"lock contention on {repo}, will retry "
+                            f"next tick (session={session_id})[/dim]"
+                        )
+                        continue
+
+                    try:
+                        state_db.mark_pending_resume_resumed(session_id)
+                    except Exception:
+                        pass
+
+                    if sweeper_transport:
+                        try:
+                            if result.success:
+                                await sweeper_transport.send(
+                                    f"✅ Resume succeeded on {repo}\n"
+                                    f"Session: `{session_id}`\n"
+                                    f"\n{result.summary}"
+                                )
+                            elif result.blocked:
+                                q = result.question or "(no question text)"
+                                await sweeper_transport.send(
+                                    f"⏸️ Resume re-blocked on {repo}\n"
+                                    f"Session: `{session_id}`\n"
+                                    f"\n{q}"
+                                )
+                            else:
+                                err = result.error or result.summary
+                                await sweeper_transport.send(
+                                    f"❌ Resume failed on {repo}\n"
+                                    f"Session: `{session_id}`\n"
+                                    f"\n{err}"
+                                )
+                        except Exception:
+                            pass
+            finally:
+                if sweeper_transport:
+                    try:
+                        await sweeper_transport.close()
+                    except Exception:
+                        pass
+
         async def _main() -> None:
             # Register + start the scheduler FIRST, before any potentially
             # slow startup work. Otherwise a 6am fire that lands during
@@ -1333,10 +1488,20 @@ def poller_start(
                 cron_expr=config.schedules.secops_cron,
                 func=_run_scheduled_secops,
             )
+            # Drain answered pending_resumes every minute so a Telegram
+            # reply to a BLOCKED session turns into an actual pipeline
+            # resume within ~60s, not 24h (the next scheduled secops
+            # cron). Cheap: no-ops when the pending_resumes table is empty.
+            scheduler.add_cron_job(
+                name="pending_resume_sweeper",
+                cron_expr="* * * * *",
+                func=_run_pending_resume_sweeper,
+            )
             scheduler.start()
             console.print(
                 f"[dim]Scheduler: secops cron={config.schedules.secops_cron} "
-                f"tz={config.timezone}[/dim]"
+                f"tz={config.timezone} | "
+                f"pending_resume_sweeper=every 1m[/dim]"
             )
 
             # Now the slow startup: first-run seeding (one gh call per

{ctrlrelay-0.1.7 → ctrlrelay-0.1.8}/src/ctrlrelay/core/poller.py

@@ -31,6 +31,16 @@ _TRANSIENT_POLL_ERRORS = (TimeoutError, GitHubError, OSError)
 _REPO_FAILURE_WARN_THRESHOLD = 3
 
 
+def _is_issues_disabled_error(exc: Exception) -> bool:
+    """Detect the specific GitHubError raised when a repo has its Issues
+    feature disabled. This is a permanent state (not a transient API
+    failure), so callers should skip the repo rather than retry it on every
+    poll cycle."""
+    if not isinstance(exc, GitHubError):
+        return False
+    return "has disabled issues" in str(exc).lower()
+
+
 @dataclass
 class IssuePoller:
     """Polls GitHub repos for newly assigned issues.
@@ -63,6 +73,12 @@ class IssuePoller:
     # seed_current(). Not persisted — intentionally resets on daemon
     # restart so an operator fix is exercised before we re-escalate.
     _repo_failure_counts: dict[str, int] = field(default_factory=dict, repr=False)
+    # Repos with GitHub Issues feature disabled — a permanent state, not a
+    # transient fetch error. Populated on first encounter and kept for the
+    # daemon lifetime so we don't spam WARNING logs every 120s cycle.
+    # Resets on daemon restart so a fresh detection still runs if the repo
+    # re-enables issues in the meantime.
+    _issues_disabled_repos: set[str] = field(default_factory=set, repr=False)
 
     def __post_init__(self) -> None:
         self._load_state()
@@ -142,6 +158,24 @@ class IssuePoller:
         """Reset the failure counter after a successful repo lookup."""
         self._repo_failure_counts.pop(repo, None)
 
+    def _mark_issues_disabled(self, repo: str) -> None:
+        """Mark a repo as having GitHub Issues disabled. Logged once at INFO
+        level so the operator can see which repos won't be polled; future
+        cycles skip the `gh` call entirely until daemon restart."""
+        if repo in self._issues_disabled_repos:
+            return
+        self._issues_disabled_repos.add(repo)
+        # Any accumulated transient-failure count is meaningless once we've
+        # identified the error as permanent — clear it so the restart counter
+        # starts fresh if the repo ever re-enables issues.
+        self._repo_failure_counts.pop(repo, None)
+        log_event(
+            _logger,
+            "poll.repo.issues_disabled",
+            repo=repo,
+            action="skipping permanently until daemon restart",
+        )
+
     # ------------------------------------------------------------------
     # Public API
     # ------------------------------------------------------------------
@@ -167,6 +201,10 @@ class IssuePoller:
         new_issues: list[dict[str, Any]] = []
 
         for repo in self.repos:
+            # Repos with GitHub Issues disabled will never return issues; skip
+            # before the `gh` call so we don't log the same error every cycle.
+            if repo in self._issues_disabled_repos:
+                continue
             try:
                 issues = await self.github.list_assigned_issues(
                     repo, assignee=self.username
@@ -174,6 +212,9 @@ class IssuePoller:
             except asyncio.CancelledError:
                 raise
             except Exception as e:
+                if _is_issues_disabled_error(e):
+                    self._mark_issues_disabled(repo)
+                    continue
                 # Transient-ish (TimeoutError/GitHubError/OSError) goes through
                 # the failure counter so persistent misconfig escalates; any
                 # other unexpected exception is logged as a skip too so the
@@ -371,6 +412,8 @@ class IssuePoller:
         treated as new and picked up — that's safer than crashing first-run.
         """
         for repo in self.repos:
+            if repo in self._issues_disabled_repos:
+                continue
             try:
                 issues = await self.github.list_assigned_issues(
                     repo, assignee=self.username
@@ -378,6 +421,9 @@ class IssuePoller:
             except asyncio.CancelledError:
                 raise
             except _TRANSIENT_POLL_ERRORS as e:
+                if _is_issues_disabled_error(e):
+                    self._mark_issues_disabled(repo)
+                    continue
                 self._record_repo_failure(repo, e, phase="seed")
                 continue
             self._clear_repo_failure(repo)