@capagents/forgeqe 2.3.0 → 2.3.2

package/docs/user-guide.html

@@ -598,7 +598,7 @@ bun src/index.ts init --global --profile azure-gpt4o
 <p>Copilot runs one SDK session per model request, with none of Copilot's own tools and your request's system prompt in place of Copilot's. The agent's tools are offered to the model as the session's only tools; when it calls one, the gateway ends the turn and hands the call back to ForgeQE's agent runtime, which runs it (with the same permissions as on any other model) and sends the result with the next request. The gateway itself runs nothing.</p>
 <p><strong>With Copilot and a key-based model both in <code>llms.json</code></strong>, Copilot comes first: a sign-in makes it the default, and without a <code>&quot;default&quot;</code> it is the active profile. Before a model is used, ForgeQE checks that it works (its sign-in or key, and one live call, kept for a day once it passes). If Copilot does not work (not signed in, the Copilot SDK missing, the model not on your seat), the other model is used and ForgeQE says why and how to fix Copilot; if a key-based model you made the default does not work, Copilot is tried first. A model that fails its check is never used, and when none works the error lists each one with what to do. <code>--profile</code> turns this off: the model you name is used as it is. Each request is billed to the seat (GitHub AI credits or premium requests, by your plan), and GitHub may rate-limit frequent automated requests. If a model is refused, check that your organization's Copilot policies allow it. ForgeQE shows no cost for a <code>github_copilot</code> profile unless you give it a <code>price</code>.</p>
 <p><strong>In pipelines.</strong> <code>forgeqe suite ... --no-code</code> uses only the browser engine. Set <code>COPILOT_GITHUB_TOKEN</code> to a fine-grained token with the Copilot Requests permission, stored as a pipeline secret. GitHub also documents the workflow's own <code>GITHUB_TOKEN</code> with <code>permissions: copilot-requests: write</code> for organization repositories whose Copilot policy allows it (<a href="https://docs.github.com/en/copilot/how-tos/copilot-sdk/auth/server-to-server-tokens" rel="noopener">server tokens</a>). A Copilot seat belongs to one person: prefer an organization-billed token over signing a shared pipeline in to someone's seat.</p>
-<p>GitHub Models, which ForgeQE used for the engine before, was <a href="https://github.blog/changelog/2026-07-30-github-models-is-now-retired/" rel="noopener">retired by GitHub</a> on July 30, 2026. A profile that still has <code>engine_model: &quot;openai/...&quot;</code> or <code>github_org</code> needs updating: drop <code>github_org</code>, and use a Copilot model id.</p>
+<p>GitHub Models, which ForgeQE used for the engine before, was <a href="https://github.blog/changelog/2026-07-30-github-models-is-now-retired/" rel="noopener">retired by GitHub</a> on July 30, 2026. A profile that still has <code>engine_model: &quot;openai/...&quot;</code> or <code>github_org</code> needs updating: drop <code>github_org</code>, and use a Copilot model id. A profile whose endpoint is GitHub Models (<code>models.github.ai</code>) is never called: remove it from <code>llms.json</code>. Copilot needs no <code>GITHUB_TOKEN</code> or <code>GH_TOKEN</code>; ForgeQE does not pass them to Copilot.</p>
 <p><strong>When something goes wrong.</strong></p>
 <div class="tbl"><table>
 <thead>
@@ -609,6 +609,7 @@ bun src/index.ts init --global --profile azure-gpt4o
 <tr><td><code>The GitHub sign-in did not finish</code></td><td>The browser sign-in was cancelled or denied. Run <code>forgeqe auth login copilot</code> again; with SAML single sign-on, authorize your organization when GitHub asks.</td></tr>
 <tr><td><code>GitHub Copilot does not take classic tokens (ghp_)</code></td><td>Sign in in the browser instead, or make a fine-grained token with the Copilot Requests permission, owned by your personal account.</td></tr>
 <tr><td><code>GitHub Copilot did not accept the token</code></td><td>The token has no Copilot Requests permission, has expired, or the account has no Copilot seat.</td></tr>
+<tr><td><code>GitHub Copilot refused the sign-in</code> or <code>refused the token</code> (<code>Authorization error. Your credentials may be expired or invalid</code>)</td><td>Copilot received the credential and would not take it. A token wins over the browser sign-in: if <code>COPILOT_GITHUB_TOKEN</code> is set, or you once pasted a token, that is what was refused; remove it and run <code>forgeqe auth login copilot</code> in the browser. Otherwise sign in again. If it still fails and your seat comes from your company, a GitHub admin must allow Copilot CLI in the Copilot policies: the Copilot SDK runs on it.</td></tr>
 <tr><td><code>… must be a Copilot model id</code>, <code>engine_model must be a Copilot model id</code></td><td>Fix that field in <code>llms.json</code>: the sign-in lists the seat's models. A GitHub Models id such as <code>openai/gpt-4.1</code> is not a Copilot id; use <code>gpt-4.1</code>.</td></tr>
 <tr><td><code>No model in llms.json can … right now</code></td><td>Each model is listed with why it cannot be used and the fix. <code>forgeqe doctor</code> checks everything at once.</td></tr>
 <tr><td><code>GitHub's Copilot SDK is not installed in the ForgeQE engine</code></td><td>Run <code>forgeqe setup</code>. It installs the SDK from pypi.org; if that is blocked, <code>forgeqe doctor</code> says what to ask IT for.</td></tr>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@capagents/forgeqe",
-  "version": "2.3.0",
+  "version": "2.3.2",
   "description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

package/py/forgeqe_engine/copilot_gateway.py

@@ -30,7 +30,7 @@ import sys
 import tempfile
 import time
 import uuid
-from typing import Any, Dict, List, Optional, TextIO, Tuple
+from typing import Any, Callable, Dict, List, Optional, TextIO, Tuple
 
 from .systemcerts import use_system_certificates
 
@@ -290,6 +290,9 @@ class Copilot:
         self.workdir = tempfile.mkdtemp(prefix="forgeqe-copilot-")
         self.signed_in_user = not token
         self.client = CopilotClient(**client_options(token, home, self.workdir))
+        # Caller tools named like one of Copilot's own: they must say they replace it. Learned from Copilot's answers
+        # (BUILTIN_TOOLS to start with), kept for the gateway's life, so only a first request pays a retry.
+        self.replaces: set = set(BUILTIN_TOOLS)
 
     async def start(self) -> None:
         await self.client.start()
@@ -300,6 +303,14 @@ class Copilot:
                 raise GatewayError(401, "GitHub Copilot is not signed in: run forgeqe auth login copilot, which opens GitHub in your browser.")
             raise GatewayError(401, "GitHub Copilot did not accept the token: use a fine-grained token with the Copilot Requests permission, for an account with a Copilot seat.")
 
+    def refusal(self, message: str) -> "GatewayError":
+        """Copilot's error as the gateway reports it. A refused sign-in says whose, and whether it was a token."""
+        if not signin_refused(message):
+            return GatewayError(502, f"GitHub Copilot: {message}"[:2000])
+        what = "the sign-in" if self.signed_in_user else "the token"
+        who = f" of {self.login}" if getattr(self, "login", "") else ""
+        return GatewayError(401, f"GitHub Copilot refused {what}{who}: {message}"[:2000])
+
     async def stop(self) -> None:
         try:
             await self.client.stop()
@@ -326,9 +337,14 @@ class Copilot:
             })
         return out
 
-    async def complete(self, req: Dict[str, Any]) -> Dict[str, Any]:
+    async def complete(self, req: Dict[str, Any], on_text: Optional[Callable[[str], None]] = None) -> Dict[str, Any]:
+        """The answer to one request. With on_text, the reply's text goes to it as Copilot writes it (not for a JSON
+        reply, whose code fences are removed once it is whole)."""
         from copilot.generated.rpc import PermissionDecisionReject
-        from copilot.session_events import AssistantMessageData, AssistantUsageData, SessionErrorData
+        from copilot.session_events import AssistantMessageData, AssistantMessageDeltaData, AssistantUsageData, SessionErrorData
+
+        # The text callback while this request streams; None once it is asked for again without streaming.
+        stream: Dict[str, Any] = {"to": None if req["schema"] is not None or req["json_only"] else on_text, "sent": False}
 
         usage = {"input": 0, "output": 0, "cached": 0}
         errors: List[str] = []
@@ -342,42 +358,68 @@ class Copilot:
                 usage["cached"] += data.cache_read_tokens or 0
             elif isinstance(data, SessionErrorData):
                 errors.append(data.message or data.error_type or "error")
-
-        tools = relay_tools(req["tools"], calls)
-        options: Dict[str, Any] = dict(
-            model=req["model"],
-            # The caller's prompt is the whole prompt; Copilot's own tools are never offered, only the caller's.
-            system_message={"mode": "replace", "content": req["system"] or "You are a helpful assistant."},
-            tools=tools or None,
-            available_tools=[t.name for t in tools],
-            on_permission_request=lambda *_: PermissionDecisionReject(),
-            working_directory=self.workdir,
-            skip_custom_instructions=True,
-            enable_config_discovery=False,
-            enable_skills=False,
-            enable_session_store=False,
-            on_event=on_event,
-        )
-        try:
-            session = await self.client.create_session(**options)
-        except Exception as e:
-            # A caller's tool named like one of Copilot's (bash, grep, glob) must say it replaces it.
-            if not tools or "built" not in str(e).lower():
-                raise
-            tools = relay_tools(req["tools"], calls, override=True)
-            session = await self.client.create_session(**{**options, "tools": tools})
-        try:
-            reply = await session.send_and_wait(req["prompt"], attachments=req["attachments"] or None, response_schema=req["schema"], timeout=REQUEST_TIMEOUT_S)
-        except TimeoutError:
-            raise GatewayError(504, f"GitHub Copilot did not answer within {int(REQUEST_TIMEOUT_S)} seconds.")
-        finally:
+            # A sub-agent's text is not the reply.
+            elif stream["to"] and isinstance(data, AssistantMessageDeltaData) and data.delta_content and not data.parent_tool_call_id:
+                stream["sent"] = True
+                stream["to"](data.delta_content)
+
+        tried: set = set()
+        reply: Any = None
+        for _ in range(MAX_TOOL_RETRIES + 2):
+            errors.clear()
+            calls.clear()
+            names = {t["name"] for t in req["tools"]}
+            tools = relay_tools(req["tools"], calls, replaces=self.replaces & names)
+            options: Dict[str, Any] = dict(
+                model=req["model"],
+                # The caller's prompt is the whole prompt; Copilot's own tools are never offered, only the caller's.
+                system_message={"mode": "replace", "content": req["system"] or "You are a helpful assistant."},
+                tools=tools or None,
+                available_tools=[t.name for t in tools],
+                on_permission_request=lambda *_: PermissionDecisionReject(),
+                working_directory=self.workdir,
+                skip_custom_instructions=True,
+                enable_config_discovery=False,
+                enable_skills=False,
+                enable_session_store=False,
+                on_event=on_event,
+                **({"streaming": True} if stream["to"] else {}),
+            )
+            failure: Optional[str] = None
+            session = None
             try:
-                await session.disconnect()
-            except Exception:
-                pass
+                session = await self.client.create_session(**options)
+                reply = await session.send_and_wait(req["prompt"], attachments=req["attachments"] or None, response_schema=req["schema"], timeout=REQUEST_TIMEOUT_S)
+            except TimeoutError:
+                raise GatewayError(504, f"GitHub Copilot did not answer within {int(REQUEST_TIMEOUT_S)} seconds.")
+            except Exception as e:  # Copilot names a tool it will not take; anything else is reported below.
+                failure = str(e)
+            finally:
+                if session is not None:
+                    try:
+                        await session.disconnect()
+                    except Exception:
+                        pass
+            # Copilot reports a clash with its own tools when the session starts or when the turn starts.
+            fix = tool_conflict(failure or (errors[-1] if errors and not calls else ""), names)
+            if fix and fix not in tried:
+                tried.add(fix)
+                name, replace = fix
+                (self.replaces.add if replace else self.replaces.discard)(name)
+                continue
+            # A streamed request that failed before any text: asked once more the way that worked before streaming, so
+            # streaming is never what stops Copilot. A refused sign-in would fail the same way, so it is not asked again.
+            problem = failure or (errors[-1] if errors and not calls else "")
+            if problem and stream["to"] and not stream["sent"] and not signin_refused(problem):
+                stream["to"] = None
+                continue
+            if failure:
+                raise self.refusal(failure)
+            break
+        else:
+            raise GatewayError(502, "GitHub Copilot kept refusing the names of the caller's tools.")
         if errors and not calls:
-            msg = errors[-1]
-            raise GatewayError(401 if "auth" in msg.lower() or "401" in msg else 502, f"GitHub Copilot: {msg}")
+            raise self.refusal(errors[-1])
         data = getattr(reply, "data", None)
         content = (data.content or "") if isinstance(data, AssistantMessageData) else ""
         if calls:
@@ -389,10 +431,44 @@ class Copilot:
         return completion(req["model"], content, usage)
 
 
-def relay_tools(specs: List[Dict[str, Any]], calls: List[Dict[str, Any]], override: bool = False) -> List[Any]:
+# Copilot's own tools that the agent runtime's tools are named like (bash, glob, grep, edit, …). A caller tool with one
+# of these names says it replaces Copilot's: the caller's tool is the one offered, and it runs nothing here.
+BUILTIN_TOOLS = frozenset({"bash", "powershell", "glob", "grep", "view", "edit", "create", "task", "skill", "web_fetch", "think", "update_todo", "report_intent"})
+MAX_TOOL_RETRIES = 8
+
+_CONFLICT = re.compile(r'(?:external )?tool "([A-Za-z0-9_-]{1,64})" conflicts with a built-in tool', re.I)
+_NO_BUILTIN = re.compile(r'tool "([A-Za-z0-9_-]{1,64})"[^"\n]{0,80}(?:does not|doesn\'t|no) (?:override|built-in|builtin)', re.I)
+
+
+_SIGNIN_REFUSED = re.compile(r"authori[sz]ation error|credentials may be expired|bad credentials|unauthori[sz]ed|not authenticated|no github oauth token|\b401\b", re.I)
+
+
+def signin_refused(message: str) -> bool:
+    """Whether Copilot's error says it would not take the sign-in or token (as opposed to the model or the request)."""
+    return bool(_SIGNIN_REFUSED.search(message or ""))
+
+
+def tool_conflict(message: str, names: set) -> Optional[Tuple[str, bool]]:
+    """
+    From Copilot's error, the caller tool to retry with: (name, True) when it clashes with a built-in and must replace
+    it, (name, False) when Copilot has no built-in of that name to replace. None for any other error.
+    """
+    if not message:
+        return None
+    m = _CONFLICT.search(message)
+    if m and m.group(1) in names:
+        return m.group(1), True
+    m = _NO_BUILTIN.search(message)
+    if m and m.group(1) in names:
+        return m.group(1), False
+    return None
+
+
+def relay_tools(specs: List[Dict[str, Any]], calls: List[Dict[str, Any]], override: bool = False, replaces: Optional[set] = None) -> List[Any]:
     """
     The caller's tools as Copilot SDK tools that run nothing: each records the call in `calls` and, being terminal, ends
-    the turn, so the call goes back to the caller as OpenAI tool_calls.
+    the turn, so the call goes back to the caller as OpenAI tool_calls. A tool in `replaces` (or every tool, with
+    `override`) replaces Copilot's built-in tool of the same name.
     """
     if not specs:
         return []
@@ -408,7 +484,7 @@ def relay_tools(specs: List[Dict[str, Any]], calls: List[Dict[str, Any]], overri
         return ToolResult(text_result_for_llm="The caller runs this tool and sends its result in the next message.")
 
     return [
-        Tool(name=t["name"], description=t["description"], parameters=t["parameters"], handler=handler, skip_permission=True, is_terminal=True, defer="never", overrides_built_in_tool=override)
+        Tool(name=t["name"], description=t["description"], parameters=t["parameters"], handler=handler, skip_permission=True, is_terminal=True, defer="never", overrides_built_in_tool=override or t["name"] in (replaces or set()))
         for t in specs
     ]
 
@@ -435,6 +511,61 @@ def _error(status: int, message: str) -> bytes:
     return _response(status, {"error": {"message": message, "type": "forgeqe_copilot_gateway"}})
 
 
+class Stream:
+    """
+    A streaming answer sent as Copilot writes it. The head goes out with the first piece of text, so a request that
+    fails before any text still gets a plain HTTP error; one that fails later ends with an error event.
+    """
+
+    def __init__(self, writer: Any, model: str) -> None:
+        self.writer = writer
+        self.base = {"id": f"chatcmpl-{uuid.uuid4().hex}", "object": "chat.completion.chunk", "created": int(time.time()), "model": model}
+        self.sent = ""
+        self.loop = asyncio.get_running_loop()
+
+    def text(self, delta: str) -> None:
+        """Called by the SDK, maybe from a thread of its own: the write happens on the server's loop."""
+        try:
+            on_loop = asyncio.get_running_loop() is self.loop
+        except RuntimeError:
+            on_loop = False
+        if on_loop:
+            self._write(delta)
+        else:
+            self.loop.call_soon_threadsafe(self._write, delta)
+
+    def _write(self, delta: str) -> None:
+        if not self.sent:
+            self.writer.write(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nCache-Control: no-cache\r\nConnection: close\r\n\r\n")
+        role = {} if self.sent else {"role": "assistant"}
+        self.sent += delta
+        self._event({**self.base, "choices": [{"index": 0, "delta": {**role, "content": delta}, "finish_reason": None}]})
+
+    def _event(self, payload: Any) -> None:
+        self.writer.write(f"data: {json.dumps(payload)}\n\n".encode())
+
+    async def finish(self, done: Dict[str, Any]) -> None:
+        await asyncio.sleep(0)  # text the SDK handed over from its own thread is written first
+        if not self.sent:
+            self.writer.write(_stream(done))
+            return
+        message = done["choices"][0]["message"]
+        content = message.get("content") or ""
+        # Only what was not sent yet: the final text normally is exactly what streamed.
+        rest = content[len(self.sent):] if content.startswith(self.sent) else ""
+        for c in chunks({**done, **self.base, "choices": [{**done["choices"][0], "message": {**message, "content": rest}}]}):
+            c["choices"] = [{**ch, "delta": {k: v for k, v in ch["delta"].items() if k != "role" and (k != "content" or v)}} for ch in c["choices"]]
+            self._event(c)
+        self.writer.write(b"data: [DONE]\n\n")
+
+    async def fail(self, status: int, message: str) -> None:
+        await asyncio.sleep(0)
+        if not self.sent:
+            self.writer.write(_error(status, message))
+            return
+        self._event({"error": {"message": message, "type": "forgeqe_copilot_gateway"}})
+
+
 async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, copilot: Any, key: str) -> None:
     try:
         try:
@@ -477,8 +608,16 @@ async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, cop
             writer.write(_error(400, "The request body is not JSON."))
             return
         req = translate(body)
-        done = await copilot.complete(req)
-        writer.write(_stream(done) if req["stream"] else _response(200, done))
+        if req["stream"]:
+            stream = Stream(writer, req["model"])
+            try:
+                await stream.finish(await copilot.complete(req, stream.text))
+            except GatewayError as e:
+                await stream.fail(e.status, e.message)
+            except Exception as e:
+                await stream.fail(502, f"GitHub Copilot: {type(e).__name__}: {e}"[:2000])
+            return
+        writer.write(_response(200, await copilot.complete(req)))
     except GatewayError as e:
         writer.write(_error(e.status, e.message))
     except Exception as e:  # The SDK's own errors: report them rather than drop the connection.