@capagents/forgeqe 1.2.0 → 1.2.1

package/HOW_TO_USE.md

@@ -301,6 +301,8 @@ A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of
 
 **The same flow asked again.** When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (*✓ Existing test passed · 19s · no browser run or new test needed*). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.
 
+**When the flow cannot be reproduced.** If ForgeQE's own replay of the run could not reach the run's final page and Scribe's first test fails too, the job stops there and says which replayed step failed and why, instead of spending its remaining attempts on a flow no script can repeat as recorded.
+
 **Why a test failed.** When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's `E` lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.
 
 **Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
@@ -1210,7 +1212,19 @@ rag:
 
 The answer can also be an object with `results`, `passages`, `documents`, or `hits`, and a passage's text can be `text`, `content`, or `page_content` (LangChain's shape). A RAG server with an MCP interface works through `mcp:` instead: ForgeQE's agent calls its tools.
 
-`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check.
+`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check. Ask to run a case and it runs, whatever the check found; ForgeQE's agent reports the findings with the result instead of refusing. A case whose steps have no expected results runs too: it passes when every step can be done, and the check lists it **to review**, since it checks no result the page should show.
+
+### API steps in a test case
+
+A test case can mix API and UI steps ("add three products with POST /add_to_cart/{id}, then remove the first one in the cart, then pay"). The browser agent makes each API step with its `api_request` action, from the page, in the browser's own session: the logged-in user's cookies go with the call, and the CSRF token comes from the cookie or the page's form when the site needs one. So what an API step changes (a cart, an order) is what the next UI step sees.
+
+- **The verdict** of an API step rests on the status the server answered, not on the agent's account of it.
+- **The run records it** as an `api` step in `workflow.yaml`: method, URL, body, CSRF source, and the status and final URL the server gave.
+- **ForgeQE's replay repeats it** the same way, and fails that step when the server answers differently (403 where the run got 200), so the replay and the test get past it instead of stalling on an empty cart.
+- **Scribe's handoff** lists each API step with what was sent and answered, and tells Scribe to make it through the framework's API request context that shares the browser's cookies (Playwright's `page.request`), asserting the status.
+- **Kept for reuse**: the calls are saved as an API scenario, one per test case (`out/api/case-ado-2287/`, updated by each run), which `forgeqe api` and the API tools run again without a browser. The run's card says where (*API  2 calls saved as a reusable API scenario*).
+
+URLs the case gives are opened exactly as given: when the model adds a trailing slash (`/login/`, a 404 on many sites), ForgeQE opens and records the URL from the case.
 
 When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. `update_test_case` shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.
 

package/docs/user-guide.html

@@ -590,6 +590,7 @@ forgeqe auth logout jira
 </table></div>
 <p>A card says which it did and why (<em>Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing</em>). Say how when you know (&quot;extend the login test&quot;, &quot;write a separate test&quot;, &quot;update it&quot;) and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.</p>
 <p><strong>The same flow asked again.</strong> When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (<em>✓ Existing test passed · 19s · no browser run or new test needed</em>). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for &quot;a fresh run&quot; or &quot;a new test&quot; to skip the reuse.</p>
+<p><strong>When the flow cannot be reproduced.</strong> If ForgeQE's own replay of the run could not reach the run's final page and Scribe's first test fails too, the job stops there and says which replayed step failed and why, instead of spending its remaining attempts on a flow no script can repeat as recorded.</p>
 <p><strong>Why a test failed.</strong> When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's <code>E</code> lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.</p>
 <p><strong>Keeping Scribe's cost down.</strong> ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. <code>code.max_attempts</code> (default 3) and <code>code.max_tokens</code> (default 1,500,000; <code>0</code> turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in <code>max_attempts</code>) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (<code>code.tidy: false</code> skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. <code>code.max_steps</code> (default 50) caps the tool calls in one of Scribe's turns. <code>code.review_profile</code> runs the reviewer on a cheaper llms.json profile.</p>
 <p><code>forgeqe.yaml</code> selects the active profile:</p>
@@ -917,6 +918,17 @@ forgeqe memory clear booking.com --yes
 <h4>Test cases and suites</h4>
 <div class="prose"><p>Test cases you already have can be run two ways. In the shell or with <code>forgeqe run</code>, ask in plain words (&quot;run SHOP-12 and SHOP-14&quot;, &quot;run test plan 12 suite 34&quot;, &quot;run the cases in smoke.yaml&quot;) or paste the cases, and the agent works out the source. <code>forgeqe suite</code> takes explicit references and runs the same way every time, with no agent involved, which is what pipelines call. Each case is one goal: ForgeQE tells the agent to do the steps in order and to check every expected result, and the agent's verdict decides whether the case passed. A case with no steps (plain text, a Jira story) is run from its description.</p>
 </div>
+<section class="doc sub" id="doc-how_to_use-api-steps-in-a-test-case"><h5>API steps in a test case</h5><div class="prose"><p>A test case can mix API and UI steps (&quot;add three products with POST /add_to_cart/{id}, then remove the first one in the cart, then pay&quot;). The browser agent makes each API step with its <code>api_request</code> action, from the page, in the browser's own session: the logged-in user's cookies go with the call, and the CSRF token comes from the cookie or the page's form when the site needs one. So what an API step changes (a cart, an order) is what the next UI step sees.</p>
+<ul>
+<li><strong>The verdict</strong> of an API step rests on the status the server answered, not on the agent's account of it.</li>
+<li><strong>The run records it</strong> as an <code>api</code> step in <code>workflow.yaml</code>: method, URL, body, CSRF source, and the status and final URL the server gave.</li>
+<li><strong>ForgeQE's replay repeats it</strong> the same way, and fails that step when the server answers differently (403 where the run got 200), so the replay and the test get past it instead of stalling on an empty cart.</li>
+<li><strong>Scribe's handoff</strong> lists each API step with what was sent and answered, and tells Scribe to make it through the framework's API request context that shares the browser's cookies (Playwright's <code>page.request</code>), asserting the status.</li>
+<li><strong>Kept for reuse</strong>: the calls are saved as an API scenario, one per test case (<code>out/api/case-ado-2287/</code>, updated by each run), which <code>forgeqe api</code> and the API tools run again without a browser. The run's card says where (<em>API  2 calls saved as a reusable API scenario</em>).</li>
+</ul>
+<p>URLs the case gives are opened exactly as given: when the model adds a trailing slash (<code>/login/</code>, a 404 on many sites), ForgeQE opens and records the URL from the case.</p>
+<p>When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. <code>update_test_case</code> shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.</p>
+</div></section>
 <section class="doc sub" id="doc-how_to_use-when-a-case-fails"><h5>When a case fails</h5><div class="prose"><p>The browser run of a test case is the test, so when a case does not pass, ForgeQE works out what kind of failure it was and acts on it. The browser agent reports a cause for each failed step (it saw the page); clear evidence in the run, such as an unreachable site or a 503 page, overrides &quot;I could not operate it&quot;.</p>
 <div class="tbl"><table>
 <thead>
@@ -1148,8 +1160,7 @@ forgeqe suite cases.yaml --no-code --no-write-back --out .forgeqe/reports/smoke
 #   command: [&quot;python&quot;, &quot;rag/search.py&quot;]
 </code></pre>
 <p>The answer can also be an object with <code>results</code>, <code>passages</code>, <code>documents</code>, or <code>hits</code>, and a passage's text can be <code>text</code>, <code>content</code>, or <code>page_content</code> (LangChain's shape). A RAG server with an MCP interface works through <code>mcp:</code> instead: ForgeQE's agent calls its tools.</p>
-<p><code>suite.case_check</code> decides what happens before a run: <code>warn</code> (the default) shows the findings and runs every case, <code>block</code> leaves out the cases that cannot run as written, <code>off</code> skips the check.</p>
-<p>When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. <code>update_test_case</code> shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.</p>
+<p><code>suite.case_check</code> decides what happens before a run: <code>warn</code> (the default) shows the findings and runs every case, <code>block</code> leaves out the cases that cannot run as written, <code>off</code> skips the check. Ask to run a case and it runs, whatever the check found; ForgeQE's agent reports the findings with the result instead of refusing. A case whose steps have no expected results runs too: it passes when every step can be done, and the check lists it <strong>to review</strong>, since it checks no result the page should show.</p>
 </div>
 </section>
 </div>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@capagents/forgeqe",
-  "version": "1.2.0",
+  "version": "1.2.1",
   "description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
   "license": "MIT",
   "type": "module",

package/prompts/brain.md

@@ -160,6 +160,7 @@ Read data from Azure DevOps, Jira, Xray, Confluence, and other systems in this s
 - run_test_cases files bugs for failed cases itself when "This session" says so, only for functional defects. You MUST NOT file them again. Its write_back list names each bug.
 - run_test_cases checks the cases first (case_check in its result). When a case needs a review or cannot run as written, say so briefly, with the correction.
 - When the user asks if cases are good, ready, clear, or complete, you MUST call check_test_cases.
+- When the user asks to run cases, you MUST call run_test_cases, also when check_test_cases or its card found problems. run_test_cases skips a case only when the setup says so (suite.case_check: block). You MUST NOT refuse a run because of a check verdict. Report what the check found with the result, and offer to fix the case with update_test_case.
 - check_test_cases also reads what each case tests: the story, the feature, the epic, linked Confluence pages, the RAG service, and page memory.
 - You MUST judge each case against that. Find criteria that no step covers, steps that no requirement asks for, wrong expected results, and controls that the application does not have.
 - For a case classified "test case needs updating", you MUST write the correct steps from run_details. Show them, and offer update_test_case.

package/prompts/browser-api.md

@@ -0,0 +1,13 @@
+---
+name: browser-api
+description: "How the browser agent makes the API steps of a test case, in the browser's session, so they are recorded and repeated."
+used_by: "the browser agent (every browser run)"
+kind: fragment
+---
+## API steps
+- For a step that calls an API (an HTTP method, an endpoint, or a payload), you MUST use api_request. You MUST NOT call fetch or XMLHttpRequest from evaluate.
+- Before the first api_request, you MUST open a page of the site. api_request runs in the browser's session and sends its cookies.
+- When the site needs a CSRF token, you MUST set csrf to "cookie" or "form". Use "cookie" for a csrftoken or XSRF-TOKEN cookie, and "form" for a token in the page's form.
+- For a form payload, you MUST give body as name=value pairs joined with &, and content_type application/x-www-form-urlencoded. For JSON, you MUST give the JSON text and content_type application/json.
+- You MUST judge an API step on the server's answer: its status, or a value in the response. You MUST NOT judge it on what the page shows. You MUST NOT record an API answer with record_check. The api_request result is the evidence of an API step. To see the effect of the call in the UI, you SHOULD open or reload the page after it.
+- You MUST use every URL exactly as the task gives it. You MUST NOT add or remove a trailing slash, path segments, or parameters. When a URL that you changed fails, you MUST use the URL as given.

package/prompts/browser-capture.md

@@ -6,10 +6,10 @@ kind: fragment
 ---
 ## Acting so the run can become a test
 ForgeQE turns a passed run into an automated test. The test repeats your actions with real clicks and typing.
-- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector.
+- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector, and through api_request for a step that calls an API.
 - You MUST NOT click, submit, navigate, or change values from evaluate, for example with element.click(), dispatchEvent, `location.href = …`, or `.value`. A test cannot repeat these actions.
 - You MAY use evaluate only to read the page.
-- You MUST open a URL only for the start page, or when the task gives the URL.
+- You MUST open a URL only for the start page, or when the task gives the URL. Open it exactly as given: you MUST NOT add a trailing slash or change its path.
 - You MUST NOT add #section or ?parameters to a URL to go where a step takes you. That skips the step.
 - When you open a wrong URL, you MUST go back to the start page. Then do the steps.
 - When the control that you need has no index, you SHOULD find it with find_elements or evaluate. Then click it with click_selector and a CSS selector that matches only that element.

package/py/forgeqe_engine/api_step.py

@@ -0,0 +1,130 @@
+"""
+API steps in a browser run: api_request calls an HTTP endpoint from the page, in the browser's own session (its
+cookies go with the call), so a test case can mix API and UI steps on one logged-in user. Each call is recorded as an
+"api" step with its method, URL, body, and the response's status, so the verdict rests on the status the server
+gave, the replay and the written test repeat the call, and ForgeQE keeps it as a reusable API scenario.
+"""
+# No `from __future__ import annotations`: the library matches the injected browser_session parameter by its real type.
+import asyncio
+import json
+import logging
+from typing import Any, Dict
+
+from browser_use import ActionResult, BrowserSession
+
+logger = logging.getLogger("forgeqe_engine.api_step")
+
+# An endpoint that does not answer in this time fails the step instead of holding the run.
+CALL_TIMEOUT = 30.0
+
+API_SYSTEM_MESSAGE = """
+## API steps
+- For a step that calls an API (an HTTP method, an endpoint, or a payload), you MUST use api_request. You MUST NOT call fetch or XMLHttpRequest from evaluate.
+- Before the first api_request, you MUST open a page of the site. api_request runs in the browser's session and sends its cookies.
+- When the site needs a CSRF token, you MUST set csrf to "cookie" or "form". Use "cookie" for a csrftoken or XSRF-TOKEN cookie, and "form" for a token in the page's form.
+- For a form payload, you MUST give body as name=value pairs joined with &, and content_type application/x-www-form-urlencoded. For JSON, you MUST give the JSON text and content_type application/json.
+- You MUST judge an API step on the server's answer: its status, or a value in the response. You MUST NOT judge it on what the page shows. You MUST NOT record an API answer with record_check. The api_request result is the evidence of an API step. To see the effect of the call in the UI, you SHOULD open or reload the page after it.
+- You MUST use every URL exactly as the task gives it. You MUST NOT add or remove a trailing slash, path segments, or parameters. When a URL that you changed fails, you MUST use the URL as given.
+""".strip()
+
+# Runs in the page: the call carries the page's cookies; a CSRF token is read where the site keeps it.
+_FETCH_JS = r"""async (a) => {
+  const cookie = (n) => { const c = document.cookie.split('; ').find((x) => x.startsWith(n + '=')); return c ? decodeURIComponent(c.slice(n.length + 1)) : '' };
+  let token = '';
+  if (a.csrf === 'cookie') token = cookie('csrftoken') || cookie('XSRF-TOKEN') || cookie('csrf_token') || cookie('_csrf');
+  if (a.csrf === 'form') {
+    const f = document.querySelector('input[name="csrfmiddlewaretoken"], input[name="_csrf"], input[name="csrf_token"], input[name="authenticity_token"], input[name="__RequestVerificationToken"]');
+    const m = document.querySelector('meta[name="csrf-token"], meta[name="_csrf"]');
+    token = (f && f.value) || (m && m.content) || '';
+  }
+  const headers = {};
+  if (a.contentType) headers['Content-Type'] = a.contentType;
+  let body = a.body || undefined;
+  if (token) {
+    headers['X-CSRFToken'] = token;
+    headers['X-XSRF-TOKEN'] = token;
+    headers['X-CSRF-Token'] = token;
+    // A form post carries the token in the form too, as the site's own form would.
+    if (/x-www-form-urlencoded/i.test(a.contentType || '') && !/(^|&)(csrfmiddlewaretoken|_csrf|authenticity_token)=/.test(body || '')) body = (body ? body + '&' : '') + 'csrfmiddlewaretoken=' + encodeURIComponent(token);
+  }
+  const method = (a.method || 'GET').toUpperCase();
+  try {
+    const r = await fetch(a.url, { method, headers, body: method === 'GET' || method === 'HEAD' ? undefined : body, credentials: 'include', redirect: 'follow' });
+    const text = await r.text();
+    return { status: r.status, ok: r.ok, final_url: r.url, redirected: r.redirected, content_type: r.headers.get('content-type') || '', body: text.slice(0, 4000), csrf_sent: Boolean(token) };
+  } catch (e) {
+    return { error: String(e && e.message || e), page: location.href };
+  }
+}"""
+
+
+def _snippet(text: str, n: int = 300) -> str:
+    """The start of a response body on one line: JSON as it is, HTML without its tags."""
+    s = text.strip()
+    if s.startswith("<"):
+        import re
+
+        title = re.search(r"<title[^>]*>(.*?)</title>", s, re.I | re.S)
+        s = (f"[HTML: {title.group(1).strip()}] " if title else "[HTML] ") + re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", s))
+    return " ".join(s.split())[:n]
+
+
+def describe(call: Dict[str, Any]) -> str:
+    """"POST https://shop.test/payment → 200, ended at https://shop.test/payment_done/1 · response: …" for the agent and the trace."""
+    if call.get("error"):
+        return f"{call['method']} {call['url']} failed: {call['error']}"
+    end = f", ended at {call['final_url']}" if call.get("redirected") and call.get("final_url") else ""
+    csrf = " (CSRF token sent)" if call.get("csrf_sent") else ""
+    return f"{call['method']} {call['url']}{csrf} → {call['status']}{end} · response: {call.get('snippet') or '(empty)'}"
+
+
+def register_api_request(tools: Any, captured: Dict[str, Any]) -> None:
+    """api_request: an HTTP call from the page's session; the call and its answer are captured for the step."""
+
+    @tools.action(
+        "Call an HTTP API endpoint from this browser's session (the session's cookies are sent) and get the status and "
+        "the start of the response. Use it for every step that calls an API; never fetch() from evaluate. "
+        "method: GET, POST, PUT, PATCH, or DELETE. url: exactly as the task gives it. body: name=value pairs joined with & "
+        "for a form, or JSON text. content_type: application/x-www-form-urlencoded or application/json. "
+        "csrf: empty, cookie, or form."
+    )
+    async def api_request(
+        method: str,
+        url: str,
+        browser_session: BrowserSession,
+        body: str = "",
+        content_type: str = "",
+        csrf: str = "",
+    ) -> ActionResult:
+        args = {"method": (method or "GET").upper(), "url": url, "body": body, "contentType": content_type, "csrf": (csrf or "").lower()}
+        try:
+            cdp = await browser_session.get_or_create_cdp_session()
+            res = await asyncio.wait_for(
+                cdp.cdp_client.send.Runtime.evaluate(
+                    params={"expression": f"({_FETCH_JS})({json.dumps(args)})", "awaitPromise": True, "returnByValue": True},
+                    session_id=cdp.session_id,
+                ),
+                timeout=CALL_TIMEOUT,
+            )
+            value = ((res or {}).get("result") or {}).get("value") or {}
+        except Exception as ex:  # The page went away, or the call could not be made.
+            value = {"error": str(ex)}
+        call: Dict[str, Any] = {"method": args["method"], "url": url, "body": body, "content_type": content_type, "csrf": args["csrf"]}
+        if value.get("error"):
+            call["error"] = value["error"]
+            captured["api"] = call
+            hint = " Open a page of this site first: the call is made from the page." if str(value.get("page", "")).startswith("about:") else ""
+            return ActionResult(error=f"api_request: {describe(call)}.{hint}")
+        call.update(
+            status=int(value.get("status") or 0),
+            ok=bool(value.get("ok")),
+            final_url=str(value.get("final_url") or url),
+            redirected=bool(value.get("redirected")),
+            content_type_received=str(value.get("content_type") or ""),
+            csrf_sent=bool(value.get("csrf_sent")),
+            snippet=_snippet(str(value.get("body") or "")),
+        )
+        captured["api"] = call
+        line = describe(call)
+        # A 4xx or 5xx is the server's answer, not a failed action: the agent judges it against what the step expects.
+        return ActionResult(extracted_content=line, long_term_memory=line)

package/py/forgeqe_engine/capture.py

@@ -12,6 +12,7 @@ Live capture for the test a passed run becomes, done while the page is still the
 import asyncio
 import json
 import logging
+import re
 import time
 from typing import Any, Dict, List, Optional
 
@@ -36,10 +37,10 @@ MAX_CLIMB = 5
 CAPTURE_SYSTEM_MESSAGE = """
 ## Acting so the run can become a test
 ForgeQE turns a passed run into an automated test. The test repeats your actions with real clicks and typing.
-- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector.
+- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector, and through api_request for a step that calls an API.
 - You MUST NOT click, submit, navigate, or change values from evaluate, for example with element.click(), dispatchEvent, `location.href = …`, or `.value`. A test cannot repeat these actions.
 - You MAY use evaluate only to read the page.
-- You MUST open a URL only for the start page, or when the task gives the URL.
+- You MUST open a URL only for the start page, or when the task gives the URL. Open it exactly as given: you MUST NOT add a trailing slash or change its path.
 - You MUST NOT add #section or ?parameters to a URL to go where a step takes you. That skips the step.
 - When you open a wrong URL, you MUST go back to the start page. Then do the steps.
 - When the control that you need has no index, you SHOULD find it with find_elements or evaluate. Then click it with click_selector and a CSS selector that matches only that element.
@@ -114,6 +115,45 @@ def action_name(action: Any) -> str:
     return next((k for k, v in data.items() if v is not None), "")
 
 
+_URL = re.compile(r"https?://[^\s\"'<>)\]]+")
+
+
+def task_urls(task: str) -> List[str]:
+    """The URLs the task gives, without the punctuation that ends a sentence around them."""
+    return [u.rstrip(".,;:!?") for u in _URL.findall(task or "")]
+
+
+def exact_task_url(url: str, given: Optional[List[str]]) -> Optional[str]:
+    """
+    The task's own URL when `url` differs from it only by a trailing slash. Models add one (/login/), and on many
+    sites that is a 404: the URL the task gave is the one to open. None when `url` is fine as it is.
+    """
+    if not url or not given or url in given:
+        return None
+    return next((t for t in given if t != url and t.rstrip("/") == url.rstrip("/")), None)
+
+
+def correct_task_urls(payload: Dict[str, Any], given: Optional[List[str]]) -> None:
+    """
+    A step's actions, as shown and recorded, with the task's own URL where the model added a slash: what install_effects
+    opens. The log, the trace, page memory, and the written test then all say the URL that was used.
+    """
+    actions = payload.get("actions") or []
+    for i, action in enumerate(actions):
+        for name in ("navigate", "api_request"):
+            params = action.get(name) if isinstance(action, dict) else None
+            if not isinstance(params, dict) or not params.get("url"):
+                continue
+            fixed = exact_task_url(str(params["url"]), given)
+            if not fixed:
+                continue
+            for key in ("action_lines", "plain_actions"):
+                lines = payload.get(key) or []
+                if i < len(lines) and isinstance(lines[i], str):
+                    lines[i] = lines[i].replace(str(params["url"]), fixed)
+            params["url"] = fixed
+
+
 def install_effects(tools: Any, effects: List[Dict[str, Any]], captured: Dict[str, Any]) -> None:
     """
     Wrap tools.act so each action call leaves {action, effect} in `effects`, in call order; click_selector also leaves
@@ -124,14 +164,25 @@ def install_effects(tools: Any, effects: List[Dict[str, Any]], captured: Dict[st
     async def act(action: Any, browser_session: Any = None, *args: Any, **kwargs: Any):
         session = browser_session if browser_session is not None else kwargs.get("browser_session")
         name = action_name(action)
+        # A URL the task gave, opened with a slash it does not have: open it exactly as given.
+        if name in ("navigate", "api_request"):
+            # The library's ActionModel is a root model: the action's parameters sit under .root.
+            params = getattr(getattr(action, "root", action), name, None)
+            fixed = exact_task_url(str(getattr(params, "url", "") or ""), captured.get("task_urls"))
+            if fixed and params is not None:
+                logger.info("%s: using the task's URL %s instead of %s", name, fixed, params.url)
+                params.url = fixed
         before = await page_state(session) if name in EFFECT_ACTIONS and session is not None else None
         captured.pop("target", None)
+        captured.pop("api", None)
         result = await original(action, browser_session, *args, **kwargs)
         entry: Dict[str, Any] = {"action": name, "effect": None}
         if name in EFFECT_ACTIONS and session is not None and not getattr(result, "error", None):
             entry["effect"] = await effect_after(session, before)
         if captured.get("target") is not None:
             entry["target"] = captured.pop("target")
+        if captured.get("api") is not None:
+            entry["api"] = captured.pop("api")
         effects.append(entry)
         return result
 

package/py/forgeqe_engine/memory.py

@@ -1001,4 +1001,15 @@ def _compact_step(step: Dict[str, Any]) -> Dict[str, Any]:
         return {"op": "back"}
     if action == "evaluate":
         return {"op": "evaluate", "code": str(p.get("code") or "")[:2000]}
+    if action == "api_request":
+        # An HTTP call made in the page's session: what was sent, and what the server answered when the run made it.
+        out = {"op": "api", "method": str(p.get("method") or "GET").upper(), "url": p.get("url") or ""}
+        for k in ("body", "content_type", "csrf"):
+            if p.get(k):
+                out[k] = p[k]
+        answer = step.get("api") or {}
+        for k in ("status", "final_url", "redirected", "snippet"):
+            if answer.get(k) not in (None, "", False):
+                out[k] = answer[k]
+        return out
     return {"op": "unknown", "action_keys": [action]}

package/py/forgeqe_engine/service.py

@@ -432,6 +432,12 @@ class MemoryRun:
         return data
 
     def add(self, step: Dict[str, Any], agent_step: int = 0) -> None:
+        # A navigate redone at once (a wrong URL, then the right one) leaves only the one that stood: the flow, its
+        # replay, and the test then never open the page that was a mistake.
+        if step.get("action") == "navigate" and self.steps and self.steps[-1].get("action") == "navigate":
+            self.steps[-1] = step
+            self.step_of[-1] = agent_step
+            return
         self.steps.append(step)
         self.step_of.append(agent_step)
 
@@ -521,6 +527,14 @@ def _plan_memory_actions(
         if not name:
             continue
         params = action[name] if isinstance(action[name], dict) else {}
+        # The URL the task gave, where the model added a slash: what was opened (see capture.install_effects), recorded.
+        if name in ("navigate", "api_request") and params.get("url"):
+            from .capture import exact_task_url, task_urls
+
+            fixed = exact_task_url(str(params["url"]), task_urls(getattr(mem, "goal", "") or ""))
+            if fixed:
+                line = line.replace(str(params["url"]), fixed)
+                params = {**params, "url": fixed}
         target = None
         if params.get("index") is not None:
             try:
@@ -606,6 +620,8 @@ def _record_memory_results(
             step["overlay"] = target["overlay"]
         if name == "evaluate" and eff.get("effect") is True:
             step["scripted"] = True
+        if name == "api_request" and eff.get("api"):
+            step["api"] = {k: v for k, v in eff["api"].items() if k in ("status", "final_url", "redirected", "snippet", "csrf_sent")}
         if note:
             step["note"] = note
         mem.add(step, agent_step)
@@ -756,7 +772,8 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
 
     from browser_use import Agent, Browser, Tools
 
-    from .capture import CAPTURE_SYSTEM_MESSAGE, install_effects, register_click_selector
+    from .api_step import API_SYSTEM_MESSAGE, register_api_request
+    from .capture import CAPTURE_SYSTEM_MESSAGE, correct_task_urls, install_effects, register_click_selector, task_urls
     from .checks import CHECKS_SYSTEM_MESSAGE, register_checks
     from .skills import choose_skills, skills_block
 
@@ -875,6 +892,7 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
 
         async def _on_new_step(browser_state: Any, agent_output: Any, n_steps: int) -> None:
             payload = serialize_browser_step_payload(browser_state, agent_output, n_steps + replay_offset)
+            correct_task_urls(payload, captured.get("task_urls"))
             collected_steps.append(payload)
             _evaluate_control_loop()
             emit({"type": "step", **{k: v for k, v in payload.items() if k not in ("actions", "boxes")}})
@@ -961,7 +979,7 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
                 handoff = handoff_task(goal, flow, replay, kind=kind)
                 emit({"type": "memory", "text": "The agent checks the result" if replay.complete and kind == "exact" else "The agent takes over from here"})
 
-        extra_parts: List[str] = [prompt_text("browser-checks", CHECKS_SYSTEM_MESSAGE), prompt_text("browser-capture", CAPTURE_SYSTEM_MESSAGE)]
+        extra_parts: List[str] = [prompt_text("browser-checks", CHECKS_SYSTEM_MESSAGE), prompt_text("browser-capture", CAPTURE_SYSTEM_MESSAGE), prompt_text("browser-api", API_SYSTEM_MESSAGE)]
         login_block = build_default_browser_login_system_extension()
         if login_block:
             extra_parts.append(login_block)
@@ -1000,7 +1018,9 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
         )
         agent_tools = Tools()
         register_checks(agent_tools, recorded_checks)
+        captured["task_urls"] = task_urls(req.goal)
         register_click_selector(agent_tools, captured)
+        register_api_request(agent_tools, captured)
         install_effects(agent_tools, action_effects, captured)
         agent_kw["tools"] = agent_tools
         agent_kw.update(_browser_use_agent_tuning_kwargs())

package/src/api/fromSteps.ts

@@ -0,0 +1,111 @@
+import { existsSync, readFileSync } from "node:fs"
+import path from "node:path"
+
+import YAML from "yaml"
+
+import type { AppConfig } from "../config.ts"
+import { scenarioFromRun } from "./scenario.ts"
+import { saveSet } from "./store.ts"
+import { contractsFromTraffic, readNetwork, type NetworkCall } from "./traffic.ts"
+import type { ContractSet } from "./types.ts"
+
+/**
+ * The API steps a browser run made (api_request, in the browser's session) kept in ForgeQE's own API format: a
+ * contract set with one scenario that makes the calls in the run's order, carrying values from one answer to the next.
+ * It runs again without a browser (forgeqe api run, or ForgeQE's agent), and the API tools plan from it.
+ */
+
+type ApiStep = { op: "api"; method?: string; url?: string; body?: string; content_type?: string; csrf?: string; status?: number; final_url?: string; snippet?: string }
+
+export type SavedApiSteps = { id: string; dir: string; file: string; calls: number }
+
+function apiSteps(runDir: string): ApiStep[] {
+  const file = path.join(runDir, "workflow.yaml")
+  if (!existsSync(file)) return []
+  try {
+    const wf = YAML.parse(readFileSync(file, "utf8")) as { steps?: { op?: string }[] }
+    return (wf?.steps || []).filter((s): s is ApiStep => s?.op === "api" && typeof (s as ApiStep).url === "string")
+  } catch {
+    return []
+  }
+}
+
+const samePath = (a: string, b: string) => {
+  try {
+    const x = new URL(a)
+    const y = new URL(b)
+    return x.host === y.host && x.pathname.replace(/\/$/, "") === y.pathname.replace(/\/$/, "")
+  } catch {
+    return a === b
+  }
+}
+
+/**
+ * Each API step with the call network.json recorded for it (its headers and the whole answer), in order; a step the
+ * capture did not see (capture off, or over its limit) is kept from what the step itself recorded.
+ */
+export function callsForSteps(steps: ApiStep[], network: NetworkCall[]): NetworkCall[] {
+  const used = new Set<NetworkCall>()
+  const ordered = [...network].sort((a, b) => a.t - b.t)
+  return steps.map((s, i) => {
+    const method = (s.method || "GET").toUpperCase()
+    const hit = ordered.find((c) => !used.has(c) && c.method.toUpperCase() === method && samePath(c.url, s.url!))
+    if (hit) {
+      used.add(hit)
+      return hit
+    }
+    return {
+      t: i,
+      method,
+      url: s.url!,
+      type: "Fetch",
+      request: { ...(s.content_type ? { headers: { "content-type": s.content_type } } : {}), ...(s.body ? { body: s.body } : {}) },
+      response: { status: s.status || 0, ...(s.snippet ? { body: s.snippet } : {}) },
+    }
+  })
+}
+
+/** Saves the run's API steps as a contract set with their scenario; null when the run made no API step. */
+/** "case-ado-2287", or "flow-" and a short hash of the goal: stable across runs of the same case or flow. */
+export function setIdFor(caseId: string | undefined, goal: string): string {
+  if (caseId) return `case-${caseId.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}`
+  let h = 0
+  for (const ch of goal) h = (h * 31 + ch.charCodeAt(0)) | 0
+  return `flow-${(h >>> 0).toString(36)}`
+}
+
+export function saveApiSteps(config: AppConfig, runDir: string, opts: { caseId?: string; caseTitle?: string } = {}): SavedApiSteps | null {
+  const steps = apiSteps(runDir)
+  if (!steps.length) return null
+  const calls = callsForSteps(steps, readNetwork(runDir)).filter((c) => c.response?.status)
+  if (!calls.length) return null
+  const { contracts, skipped, baseUrl } = contractsFromTraffic(calls, { thirdParty: true })
+  let goal = ""
+  try {
+    goal = (JSON.parse(readFileSync(path.join(runDir, "run.json"), "utf8")) as { goal?: string }).goal || ""
+  } catch {}
+  // One set per test case (or per flow), updated by each run: the latest calls, not a new copy every time.
+  const id = setIdFor(opts.caseId, goal || path.basename(runDir))
+  for (const c of contracts) c.source.ref = path.basename(runDir)
+  const set: ContractSet = {
+    version: 1,
+    id,
+    name: opts.caseId ? `API steps of ${opts.caseId}${opts.caseTitle ? `: ${opts.caseTitle}` : ""}` : `API steps of: ${goal.slice(0, 80) || path.basename(runDir)}`,
+    baseUrl,
+    createdAt: new Date().toISOString(),
+    source: { kind: "run", ref: runDir },
+    contracts,
+    ...(skipped.length ? { skipped } : {}),
+  }
+  const scenario = scenarioFromRun(calls, set, { goal, runId: path.basename(runDir), thirdParty: true })
+  if (scenario) {
+    scenario.notes = [
+      ...(scenario.notes || []),
+      "The run made these calls in the browser's logged-in session. Run without a browser, they need that session: a login step first, or its cookie and CSRF token in api headers.",
+    ]
+    if (opts.caseId) Object.assign(scenario, { caseId: opts.caseId, ...(opts.caseTitle ? { caseTitle: opts.caseTitle } : {}) })
+    set.scenarios = [scenario]
+  }
+  const saved = saveSet(config, set)
+  return { id, dir: saved.dir, file: saved.file, calls: calls.length }
+}

package/src/cases/quality.ts

@@ -54,7 +54,9 @@ export function checkCase(tc: TestCase): CaseFinding[] {
   }
   const withExpected = steps.filter((s) => s.expected?.trim())
   if (!withExpected.length) {
-    out.push({ code: "no_expected_results", severity: "blocker", message: "No step has an expected result, so the run cannot pass or fail on anything.", suggestion: "Give at least the steps that matter an expected result the page shows (a message, a value, a page)." })
+    // It still runs: each step is done and verified, so the case passes when every step can be done. What it does not
+    // have is a result to check, so it is worth fixing, not a reason to refuse it.
+    out.push({ code: "no_expected_results", severity: "warning", message: "No step has an expected result: the run passes when every step can be done, and checks no result the page should show.", suggestion: "Give at least the steps that matter an expected result the page shows (a message, a value, a page)." })
   }
   steps.forEach((s, i) => {
     const n = i + 1

package/src/code/handoff.ts

@@ -43,6 +43,16 @@ export type Step = {
   scripted?: boolean
   /** click: the control sat in an overlay (cookie banner, sign-in prompt, dialog) that this click dismissed. */
   overlay?: { kind?: string; name?: string; id?: string }
+  /** api: an HTTP call made in the browser's session, and what the server answered when the run made it. */
+  method?: string
+  body?: string
+  content_type?: string
+  /** api: where the CSRF token came from: the "cookie", or the page's "form". */
+  csrf?: string
+  status?: number
+  final_url?: string
+  redirected?: boolean
+  snippet?: string
 }
 
 /** An element the engine found on the live page, with its locators (verified ones first). */
@@ -174,6 +184,10 @@ function stepLines(step: Step, n: number): string[] {
       break
     case "press":
       return [`${n}. press ${step.keys}`]
+    case "api":
+      return [
+        `${n}. API ${step.method || "GET"} ${step.url}${step.body ? ` with ${step.content_type?.includes("json") ? "JSON" : "form"} body ${quote(step.body.slice(0, 300))}` : ""}${step.csrf ? `, CSRF token from the ${step.csrf === "form" ? "page's form" : "csrftoken cookie"}` : ""}: the server answered ${step.status ?? "?"}${step.redirected && step.final_url ? `, ending at ${step.final_url}` : ""}${step.snippet ? `; response starts: ${quote(step.snippet.slice(0, 160))}` : ""}`,
+      ]
     case "evaluate":
       return step.scripted
         ? [
@@ -558,6 +572,12 @@ export function buildHandoff(runDir: string, memoryDir?: string, opts: { replay?
       "",
     )
   }
+  if (steps.some((s) => s.op === "api")) {
+    md.push(
+      "Steps marked API are HTTP calls the run made in the browser's own session (its cookies went with each call, so a login and a cart carry over between the API and the UI steps). Make each one in the test the same way: through the framework's API request context that shares the browser's cookies (Playwright: page.request in TypeScript, Page.APIRequest in .NET, page.request in Python and Java), with the same method, URL, and body, reading the CSRF token where the step says (the page's form, or the csrftoken cookie) and sending it as the site expects. Assert the status the run got. Do not replace an API step with clicks, or a UI step with an API call.",
+      "",
+    )
+  }
   const slices = stepSlices(steps, opts.replay)
   if (slices.size) {
     md.push(

package/src/code/loop.ts

@@ -605,6 +605,16 @@ export async function runCodeAgent(opts: {
         break
       }
       const why = run.exitCode === null ? `timed out after ${config.code.testTimeoutSeconds}s` : `exit ${run.exitCode}`
+      // ForgeQE's own replay of the run could not get through the flow either: no script reproduces it as recorded,
+      // so more attempts only spend tokens. Stop with what the replay met, for the user to act on.
+      const unreproducible = replay && !replay.error && replay.passes.length > 0 && replay.passes.every((p) => !p.reached)
+      if (unreproducible) {
+        const stuck = replay!.passes[0]!.steps.find((st) => !st.ok && !st.skipped)
+        const reason = failureReason(run.output)[0] || why
+        result.notes = `Stopped after attempt ${attempt}: the test failed (${reason}), and ForgeQE's replay of the run could not reach its final page either${stuck ? ` (step ${stuck.n}, ${stuck.action}: ${String(stuck.error || "").split("\n")[0]!.slice(0, 200)})` : ""}. The flow as recorded cannot be repeated by a script: usually a step the run did in a way a test cannot (a script in the page, state left from an earlier run, a pop-up only the agent dismissed). Run the case again after fixing that step, or check the handoff's replay section.`
+        emit({ type: "code", kind: "test", ok: false, attempt, text: `stopped: the flow cannot be reproduced; ForgeQE's replay stopped at ${stuck ? `step ${stuck.n}` : "a step"} as well (attempt ${attempt}/${maxAttempts})` })
+        break
+      }
       const mismatch = browserMismatch(runBrowser, run.browsers, report.command)
       if (mismatch) emit({ type: "code", kind: "info", text: `The test's browser differs from the run's: ${run.browsers.join(", ")}` })
       const key = failureKey(run.output)

package/src/code/replay.ts

@@ -135,11 +135,31 @@ function describe(step: Step): string {
       return "go back"
     case "evaluate":
       return "run the agent's JavaScript"
+    case "api":
+      return `API ${step.method || "GET"} ${step.url}`
     default:
       return `${step.op || "unknown"}${step.action_keys?.length ? ` (${step.action_keys.join(", ")})` : ""}`
   }
 }
 
+/**
+ * An API step as the engine's api_request makes it (py/forgeqe_engine/api_step.py): fetch in the page with its cookies,
+ * the CSRF token from the cookie or the page's form, sent in headers and, for a form post, in the form.
+ */
+export const API_CALL_JS = `async (a) => {
+  const cookie = (n) => { const c = document.cookie.split('; ').find((x) => x.startsWith(n + '=')); return c ? decodeURIComponent(c.slice(n.length + 1)) : '' };
+  let token = '';
+  if (a.csrf === 'cookie') token = cookie('csrftoken') || cookie('XSRF-TOKEN') || cookie('csrf_token') || cookie('_csrf');
+  if (a.csrf === 'form') { const f = document.querySelector('input[name="csrfmiddlewaretoken"], input[name="_csrf"], input[name="csrf_token"], input[name="authenticity_token"], input[name="__RequestVerificationToken"]'); const m = document.querySelector('meta[name="csrf-token"], meta[name="_csrf"]'); token = (f && f.value) || (m && m.content) || ''; }
+  const headers = {};
+  if (a.contentType) headers['Content-Type'] = a.contentType;
+  let body = a.body || undefined;
+  if (token) { headers['X-CSRFToken'] = token; headers['X-XSRF-TOKEN'] = token; headers['X-CSRF-Token'] = token; if (/x-www-form-urlencoded/i.test(a.contentType || '') && !/(^|&)(csrfmiddlewaretoken|_csrf|authenticity_token)=/.test(body || '')) body = (body ? body + '&' : '') + 'csrfmiddlewaretoken=' + encodeURIComponent(token); }
+  const method = (a.method || 'GET').toUpperCase();
+  try { const r = await fetch(a.url, { method, headers, body: method === 'GET' || method === 'HEAD' ? undefined : body, credentials: 'include', redirect: 'follow' }); return { status: r.status, final_url: r.url }; }
+  catch (e) { return { error: String(e && e.message || e) }; }
+}`
+
 /** The body of one step in the script, or null when it has no Playwright equivalent. */
 function stepCode(step: Step): string | null {
   const loc = step.locators?.map(locatorExpr).find(Boolean) || null
@@ -171,6 +191,19 @@ function stepCode(step: Step): string | null {
       return "(page) => page.goBack()"
     case "evaluate":
       return step.code ? `(page) => page.evaluate(${q(step.code)})` : null
+    case "api": {
+      if (!step.url) return null
+      // The call is made in the page, as the run made it: the session's cookies and its CSRF token go with it. The step
+      // fails when the server answers differently than it did in the run (403 where the run got 200).
+      const args = JSON.stringify({ method: step.method || "GET", url: step.url, body: step.body || "", contentType: step.content_type || "", csrf: step.csrf || "" })
+      return [
+        "async (page) => {",
+        `    const r = await page.evaluate(${API_CALL_JS}, ${args})`,
+        "    if (r.error) throw new Error(`API ${r.error}`)",
+        ...(step.status ? [`    if (r.status !== ${Number(step.status)}) throw new Error(\`API answered \${r.status}, the run got ${Number(step.status)}\`)`] : []),
+        "  }",
+      ].join("\n")
+    }
     default:
       return null
   }

package/src/run.ts

@@ -1,3 +1,4 @@
+import { saveApiSteps } from "./api/fromSteps.ts"
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"
 import { createRequire } from "node:module"
 import path from "node:path"
@@ -23,6 +24,8 @@ import { knowledgeBlock, readKnowledge, relevantKnowledge } from "./knowledge.ts
 export type ExecuteOptions = {
   config: AppConfig
   goal: string
+  /** The test case the run is for: its API steps are saved as that case's scenario. */
+  caseRef?: { caseId?: string; caseTitle?: string }
   browserProvider?: BrowserProvider
   /** A named device from the provider files (browser_devices); it also picks the provider. */
   browserDevice?: string
@@ -156,6 +159,13 @@ export async function executeRun(opts: ExecuteOptions): Promise<{ result: RunRes
   const withReport = (out: { result: RunResult; files: string[] }) => {
     out.result.cost = costOf(out.result.usage, opts.config.llm.price)
     applyQuality(opts.config, dir, out.result, opts.onEvent)
+    // API steps the run made in the browser's session, kept as a scenario the API tools run again without a browser.
+    try {
+      const api = saveApiSteps(opts.config, dir, opts.caseRef || {})
+      if (api) out.result.api = { id: api.id, dir: api.dir, calls: api.calls }
+    } catch (err) {
+      opts.onEvent?.({ type: "log", level: "warn", text: `API steps not saved as a scenario: ${(err as Error).message}` })
+    }
     logEvent("run.done", { out_dir: dir, status: out.result.status, summary: out.result.summary, usage: out.result.usage, memory: out.result.memory, ...(out.result.skills ? { skills: out.result.skills } : {}) })
     try {
       out.files.push(writeRunReport(dir, out.result, reportEnv(opts.config, { url: startUrl, headed: opts.headed, browser }), t0))

package/src/suite/report.ts

@@ -77,11 +77,28 @@ export function suiteUsage(results: CaseResult[], analysis?: { usage: Usage; cos
   return { usage, cost: priced ? cost : undefined }
 }
 
+/**
+ * The suite's headline, saying what each part did: the cases, and the tests written from them. A suite whose cases
+ * all passed but whose test did not pass is not called failed with "1/1 passed" beside it.
+ */
+export function suiteHeadline(s: Pick<SuiteResult, "results" | "stopped">): { mark: "✓" | "✗" | "■"; title: string; counts: string; ok: boolean } {
+  const c = counts(s.results)
+  const coded = s.results.filter((r) => r.code)
+  const testsOk = coded.filter((r) => r.code!.status === "passed").length
+  const casesOk = c.passed === c.total
+  const testsLine = coded.length ? ` · ${testsOk}/${coded.length} ${coded.length === 1 ? "test" : "tests"} passing` : ""
+  const countsLine = `${c.passed}/${c.total} ${c.total === 1 ? "case" : "cases"} passed${testsLine}`
+  if (s.stopped) return { mark: "■", title: "Suite stopped", counts: countsLine, ok: false }
+  if (!casesOk) return { mark: "✗", title: "Suite failed", counts: countsLine, ok: false }
+  if (testsOk < coded.length) return { mark: "✗", title: `${c.total === 1 ? "Case" : "Cases"} passed, ${coded.length - testsOk === 1 ? "test" : "tests"} not passing`, counts: countsLine, ok: false }
+  return { mark: "✓", title: "Suite passed", counts: countsLine, ok: true }
+}
+
 /** The end-of-suite summary for the terminal. */
 export function formatSuite(s: SuiteResult, cwd = process.cwd()): string {
-  const c = counts(s.results)
   const tokens = s.results.reduce((n, r) => n + r.tokens, 0)
-  const head = `${s.stopped ? "■ Suite stopped" : c.passed === c.total ? "✓ Suite passed" : "✗ Suite failed"} · ${c.passed}/${c.total} passed · ${duration(s.ms)} · ${(tokens / 1000).toFixed(1)}K tokens`
+  const h = suiteHeadline(s)
+  const head = `${h.mark} ${h.title} · ${h.counts} · ${duration(s.ms)} · ${(tokens / 1000).toFixed(1)}K tokens`
   const show = (f: string) => path.relative(cwd, f) || f
   const total = suiteUsage(s.results, s.analysis)
   const lines = [head, ""]
@@ -116,7 +133,8 @@ export function formatSuite(s: SuiteResult, cwd = process.cwd()): string {
 export function markdownSummary(s: SuiteResult): string {
   const c = counts(s.results)
   const cell = (t: string) => t.replace(/\|/g, "\\|").replace(/\s+/g, " ").trim()
-  const head = s.stopped ? "■ Suite stopped" : c.passed === c.total ? "✓ Suite passed" : "✗ Suite failed"
+  const h = suiteHeadline(s)
+  const head = `${h.mark} ${h.title}`
   const rows = s.results.map((r) => {
     const test = r.code ? `${r.code.status}${r.code.testFile ? ` \`${cell(r.code.testFile)}\`` : ""}` : ""
     const steps = r.stepResults?.length && r.case.steps.length ? `${r.stepResults.filter((v) => v.status === "passed").length}/${r.case.steps.length}` : ""
@@ -140,7 +158,7 @@ export function markdownSummary(s: SuiteResult): string {
       ]
     : []
   return [
-    `### ${head} · ${c.passed}/${c.total} passed · ${duration(s.ms)}`,
+    `### ${head} · ${h.counts} · ${duration(s.ms)}`,
     "",
     "| | Case | Title | Steps | Time | Result | Test |",
     "|---|---|---|---|---|---|---|",

package/src/suite/run.ts

@@ -151,6 +151,7 @@ export async function runSuite(opts: SuiteOptions): Promise<SuiteResult> {
         const { result } = await execute({
           config,
           goal: goalFor(original),
+          caseRef: { caseId: original.id, caseTitle: original.title },
           browserProvider: opts.browserProvider,
           browserDevice: device,
           url: tc.startUrl || opts.url,

package/src/summary.ts

@@ -25,6 +25,8 @@ export type RunSummary = {
   usage: string
   /** WCAG rules passed and failed, Core Web Vitals, and whether the budgets held. */
   quality?: string
+  /** The API steps saved as a reusable scenario: "2 calls saved as API scenario 20261013-…" and where. */
+  api?: string
 }
 
 export const STATUS_TITLE: Record<RunStatus, string> = {
@@ -80,6 +82,7 @@ export function buildSummary(opts: {
   failedChecks?: { expected: string; actual: string }[]
   quality?: QualityReport
   budgetExceeded?: string[]
+  api?: { id: string; dir: string; calls: number }
 }): RunSummary {
   const dir = opts.files.length ? path.relative(process.cwd(), path.dirname(opts.files[0])) || "." : ""
   const assertion = opts.outcome === "assertion_failed"
@@ -97,6 +100,7 @@ export function buildSummary(opts: {
     savedFiles: opts.files.map((f) => path.basename(f)),
     usage: opts.usage ? usageLine(opts.usage, opts.cost) : "",
     ...(opts.quality && qualityLine(opts.quality) ? { quality: qualityLine(opts.quality) } : {}),
+    ...(opts.api ? { api: `${opts.api.calls} ${opts.api.calls === 1 ? "call" : "calls"} saved as a reusable API scenario · ${path.relative(process.cwd(), opts.api.dir) || opts.api.dir}/` } : {}),
   }
 }
 
@@ -106,6 +110,7 @@ export function formatSummary(s: RunSummary): string {
   if (s.finalUrl) lines.push(`Ended on  ${s.finalUrl}`)
   if (s.memory) lines.push(`Memory    ${s.memory}`)
   if (s.quality) lines.push(`Quality   ${s.quality}`)
+  if (s.api) lines.push(`API       ${s.api}`)
   if (s.usage) lines.push(`Usage     ${s.usage}`)
   if (s.savedDir) lines.push(`Saved     ${s.savedDir}/  ${s.savedFiles.join(" · ")}`)
   return lines.join("\n")

package/src/tui/App.tsx

@@ -893,6 +893,7 @@ export function App({ options }: { options: TuiOptions }) {
           failedChecks: r?.failedChecks,
           quality: r?.quality,
           budgetExceeded: r?.budgetExceeded,
+          api: r?.api,
           ms: e.ms,
           tokens: r ? r.usage.input + r.usage.output : 0,
           steps: stepRef.current,

package/src/tui/transcript.tsx

@@ -6,7 +6,7 @@ import { createContext, useContext, type ReactNode } from "react"
 
 import type { CodeResult, RunStatus } from "../types.ts"
 import { CARD_MARK, type Card } from "../card.ts"
-import { counts } from "../suite/report.ts"
+import { counts, suiteHeadline } from "../suite/report.ts"
 import { suiteExitCode, type SuiteResult } from "../suite/run.ts"
 import { STATUS_MARK, shortPath, type RunSummary } from "../summary.ts"
 import { BAR } from "./components.tsx"
@@ -192,6 +192,7 @@ export function SummaryCard({ summary }: { summary: RunSummary }) {
         {summary.finalUrl ? detail("Ended on", <span fg={theme.muted}>{summary.finalUrl}</span>) : null}
         {summary.memory ? detail("Memory", <span fg={theme.muted}>{summary.memory}</span>) : null}
         {summary.quality ? detail("Quality", <span fg={theme.muted}>{summary.quality}</span>) : null}
+        {summary.api ? detail("API", <Linked text={summary.api} color={theme.muted} lead={UNDER + 10} />) : null}
         {summary.savedDir ? (
           <text>
             <span fg={theme.faint}>{"Saved     "}</span>
@@ -272,7 +273,7 @@ export function Row({ label, value, color = theme.muted }: { label: string; valu
 export const CASE_MARK = { passed: "✓", failed: "✗", error: "!", stopped: "■" } as const
 
 export function SuiteCard({ suite }: { suite: SuiteResult }) {
-  const c = counts(suite.results)
+  const head = suiteHeadline(suite)
   const ok = !suite.stopped && suiteExitCode(suite) === 0
   const color = suite.stopped ? theme.warn : ok ? theme.ok : theme.err
   const tokens = suite.results.reduce((n, r) => n + r.tokens, 0)
@@ -283,9 +284,9 @@ export function SuiteCard({ suite }: { suite: SuiteResult }) {
       <Marked mark={MARK.item} color={color} gap>
         <text>
           <span fg={theme.text}>
-            <strong>{suite.stopped ? "Suite stopped" : ok ? "Suite passed" : "Suite failed"}</strong>
+            <strong>{head.title}</strong>
           </span>
-          <span fg={theme.faint}>{`  ${c.passed}/${c.total} passed · ${formatDuration(suite.ms)} · ${formatTokens(tokens)} tokens`}</span>
+          <span fg={theme.faint}>{`  ${head.counts} · ${formatDuration(suite.ms)} · ${formatTokens(tokens)} tokens`}</span>
         </text>
       </Marked>
       <Under first>

package/src/types.ts

@@ -59,6 +59,8 @@ export type RunResult = {
   quality?: QualityReport
   /** Budgets this run exceeded: the run counts as failed, though its steps passed. */
   budgetExceeded?: string[]
+  /** The run's API steps, saved as a reusable API scenario: where, and how many calls. */
+  api?: { id: string; dir: string; calls: number }
 }
 
 /** Why a failed step failed, in the browser agent's judgement: see src/cases/failure.ts. */