@capagents/forgeqe 1.1.9 → 1.2.1

package/HOW_TO_USE.md

@@ -288,6 +288,23 @@ A profile's own `"price": { "input": 2.5, "output": 15, "cached_input": 0.25 }`
 This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
 ```
 
+**Reuse before rebuild, extend before duplicate.** Before Scribe writes anything, `write_test` measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as `Expect(practicePage.AjaxStatus)`). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:
+
+| What it finds | What happens |
+|---|---|
+| One test covers every measured step and check | **Reuse**: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test. |
+| One test covers most (40% or more) | **Extend**: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file |
+| Two or more tests cover most of it | **You choose**: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel) |
+| Nothing covers much | **New**: a new test, reusing the page objects and helpers as before |
+
+A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing*). Say how when you know ("extend the login test", "write a separate test", "update it") and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.
+
+**The same flow asked again.** When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (*✓ Existing test passed · 19s · no browser run or new test needed*). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.
+
+**When the flow cannot be reproduced.** If ForgeQE's own replay of the run could not reach the run's final page and Scribe's first test fails too, the job stops there and says which replayed step failed and why, instead of spending its remaining attempts on a flow no script can repeat as recorded.
+
+**Why a test failed.** When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's `E` lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.
+
 **Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
 
 `forgeqe.yaml` selects the active profile:
@@ -1195,7 +1212,19 @@ rag:
 
 The answer can also be an object with `results`, `passages`, `documents`, or `hits`, and a passage's text can be `text`, `content`, or `page_content` (LangChain's shape). A RAG server with an MCP interface works through `mcp:` instead: ForgeQE's agent calls its tools.
 
-`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check.
+`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check. Ask to run a case and it runs, whatever the check found; ForgeQE's agent reports the findings with the result instead of refusing. A case whose steps have no expected results runs too: it passes when every step can be done, and the check lists it **to review**, since it checks no result the page should show.
+
+### API steps in a test case
+
+A test case can mix API and UI steps ("add three products with POST /add_to_cart/{id}, then remove the first one in the cart, then pay"). The browser agent makes each API step with its `api_request` action, from the page, in the browser's own session: the logged-in user's cookies go with the call, and the CSRF token comes from the cookie or the page's form when the site needs one. So what an API step changes (a cart, an order) is what the next UI step sees.
+
+- **The verdict** of an API step rests on the status the server answered, not on the agent's account of it.
+- **The run records it** as an `api` step in `workflow.yaml`: method, URL, body, CSRF source, and the status and final URL the server gave.
+- **ForgeQE's replay repeats it** the same way, and fails that step when the server answers differently (403 where the run got 200), so the replay and the test get past it instead of stalling on an empty cart.
+- **Scribe's handoff** lists each API step with what was sent and answered, and tells Scribe to make it through the framework's API request context that shares the browser's cookies (Playwright's `page.request`), asserting the status.
+- **Kept for reuse**: the calls are saved as an API scenario, one per test case (`out/api/case-ado-2287/`, updated by each run), which `forgeqe api` and the API tools run again without a browser. The run's card says where (*API  2 calls saved as a reusable API scenario*).
+
+URLs the case gives are opened exactly as given: when the model adds a trailing slash (`/login/`, a 404 on many sites), ForgeQE opens and records the URL from the case.
 
 When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. `update_test_case` shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.
 

package/docs/user-guide.html

@@ -576,6 +576,22 @@ forgeqe auth logout jira
 <p><strong>Usage after every job.</strong> Every browser run, Scribe job, heal, suite, and monitor round ends with a <code>Usage</code> line: tokens in (and how many of them were cached), tokens out, and the cost at the profile's price. A request to ForgeQE's agent in the shell, and every <code>forgeqe run</code>, <code>code</code>, <code>suite</code>, and <code>heal</code> command, ends with the total for everything it started, by role:</p>
 <pre><code>This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
 </code></pre>
+<p><strong>Reuse before rebuild, extend before duplicate.</strong> Before Scribe writes anything, <code>write_test</code> measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as <code>Expect(practicePage.AjaxStatus)</code>). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:</p>
+<div class="tbl"><table>
+<thead>
+<tr><th>What it finds</th><th>What happens</th></tr>
+</thead>
+<tbody>
+<tr><td>One test covers every measured step and check</td><td><strong>Reuse</strong>: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test.</td></tr>
+<tr><td>One test covers most (40% or more)</td><td><strong>Extend</strong>: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file</td></tr>
+<tr><td>Two or more tests cover most of it</td><td><strong>You choose</strong>: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel)</td></tr>
+<tr><td>Nothing covers much</td><td><strong>New</strong>: a new test, reusing the page objects and helpers as before</td></tr>
+</tbody>
+</table></div>
+<p>A card says which it did and why (<em>Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing</em>). Say how when you know (&quot;extend the login test&quot;, &quot;write a separate test&quot;, &quot;update it&quot;) and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.</p>
+<p><strong>The same flow asked again.</strong> When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (<em>✓ Existing test passed · 19s · no browser run or new test needed</em>). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for &quot;a fresh run&quot; or &quot;a new test&quot; to skip the reuse.</p>
+<p><strong>When the flow cannot be reproduced.</strong> If ForgeQE's own replay of the run could not reach the run's final page and Scribe's first test fails too, the job stops there and says which replayed step failed and why, instead of spending its remaining attempts on a flow no script can repeat as recorded.</p>
+<p><strong>Why a test failed.</strong> When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's <code>E</code> lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.</p>
 <p><strong>Keeping Scribe's cost down.</strong> ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. <code>code.max_attempts</code> (default 3) and <code>code.max_tokens</code> (default 1,500,000; <code>0</code> turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in <code>max_attempts</code>) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (<code>code.tidy: false</code> skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. <code>code.max_steps</code> (default 50) caps the tool calls in one of Scribe's turns. <code>code.review_profile</code> runs the reviewer on a cheaper llms.json profile.</p>
 <p><code>forgeqe.yaml</code> selects the active profile:</p>
 <pre><code class="language-yaml">llm:
@@ -902,6 +918,17 @@ forgeqe memory clear booking.com --yes
 <h4>Test cases and suites</h4>
 <div class="prose"><p>Test cases you already have can be run two ways. In the shell or with <code>forgeqe run</code>, ask in plain words (&quot;run SHOP-12 and SHOP-14&quot;, &quot;run test plan 12 suite 34&quot;, &quot;run the cases in smoke.yaml&quot;) or paste the cases, and the agent works out the source. <code>forgeqe suite</code> takes explicit references and runs the same way every time, with no agent involved, which is what pipelines call. Each case is one goal: ForgeQE tells the agent to do the steps in order and to check every expected result, and the agent's verdict decides whether the case passed. A case with no steps (plain text, a Jira story) is run from its description.</p>
 </div>
+<section class="doc sub" id="doc-how_to_use-api-steps-in-a-test-case"><h5>API steps in a test case</h5><div class="prose"><p>A test case can mix API and UI steps (&quot;add three products with POST /add_to_cart/{id}, then remove the first one in the cart, then pay&quot;). The browser agent makes each API step with its <code>api_request</code> action, from the page, in the browser's own session: the logged-in user's cookies go with the call, and the CSRF token comes from the cookie or the page's form when the site needs one. So what an API step changes (a cart, an order) is what the next UI step sees.</p>
+<ul>
+<li><strong>The verdict</strong> of an API step rests on the status the server answered, not on the agent's account of it.</li>
+<li><strong>The run records it</strong> as an <code>api</code> step in <code>workflow.yaml</code>: method, URL, body, CSRF source, and the status and final URL the server gave.</li>
+<li><strong>ForgeQE's replay repeats it</strong> the same way, and fails that step when the server answers differently (403 where the run got 200), so the replay and the test get past it instead of stalling on an empty cart.</li>
+<li><strong>Scribe's handoff</strong> lists each API step with what was sent and answered, and tells Scribe to make it through the framework's API request context that shares the browser's cookies (Playwright's <code>page.request</code>), asserting the status.</li>
+<li><strong>Kept for reuse</strong>: the calls are saved as an API scenario, one per test case (<code>out/api/case-ado-2287/</code>, updated by each run), which <code>forgeqe api</code> and the API tools run again without a browser. The run's card says where (<em>API  2 calls saved as a reusable API scenario</em>).</li>
+</ul>
+<p>URLs the case gives are opened exactly as given: when the model adds a trailing slash (<code>/login/</code>, a 404 on many sites), ForgeQE opens and records the URL from the case.</p>
+<p>When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. <code>update_test_case</code> shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.</p>
+</div></section>
 <section class="doc sub" id="doc-how_to_use-when-a-case-fails"><h5>When a case fails</h5><div class="prose"><p>The browser run of a test case is the test, so when a case does not pass, ForgeQE works out what kind of failure it was and acts on it. The browser agent reports a cause for each failed step (it saw the page); clear evidence in the run, such as an unreachable site or a 503 page, overrides &quot;I could not operate it&quot;.</p>
 <div class="tbl"><table>
 <thead>
@@ -1133,8 +1160,7 @@ forgeqe suite cases.yaml --no-code --no-write-back --out .forgeqe/reports/smoke
 #   command: [&quot;python&quot;, &quot;rag/search.py&quot;]
 </code></pre>
 <p>The answer can also be an object with <code>results</code>, <code>passages</code>, <code>documents</code>, or <code>hits</code>, and a passage's text can be <code>text</code>, <code>content</code>, or <code>page_content</code> (LangChain's shape). A RAG server with an MCP interface works through <code>mcp:</code> instead: ForgeQE's agent calls its tools.</p>
-<p><code>suite.case_check</code> decides what happens before a run: <code>warn</code> (the default) shows the findings and runs every case, <code>block</code> leaves out the cases that cannot run as written, <code>off</code> skips the check.</p>
-<p>When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. <code>update_test_case</code> shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.</p>
+<p><code>suite.case_check</code> decides what happens before a run: <code>warn</code> (the default) shows the findings and runs every case, <code>block</code> leaves out the cases that cannot run as written, <code>off</code> skips the check. Ask to run a case and it runs, whatever the check found; ForgeQE's agent reports the findings with the result instead of refusing. A case whose steps have no expected results runs too: it passes when every step can be done, and the check lists it <strong>to review</strong>, since it checks no result the page should show.</p>
 </div>
 </section>
 </div>

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@capagents/forgeqe",
-  "version": "1.1.9",
+  "version": "1.2.1",
   "description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
   "license": "MIT",
   "type": "module",

package/prompts/brain.md

@@ -160,6 +160,7 @@ Read data from Azure DevOps, Jira, Xray, Confluence, and other systems in this s
 - run_test_cases files bugs for failed cases itself when "This session" says so, only for functional defects. You MUST NOT file them again. Its write_back list names each bug.
 - run_test_cases checks the cases first (case_check in its result). When a case needs a review or cannot run as written, say so briefly, with the correction.
 - When the user asks if cases are good, ready, clear, or complete, you MUST call check_test_cases.
+- When the user asks to run cases, you MUST call run_test_cases, also when check_test_cases or its card found problems. run_test_cases skips a case only when the setup says so (suite.case_check: block). You MUST NOT refuse a run because of a check verdict. Report what the check found with the result, and offer to fix the case with update_test_case.
 - check_test_cases also reads what each case tests: the story, the feature, the epic, linked Confluence pages, the RAG service, and page memory.
 - You MUST judge each case against that. Find criteria that no step covers, steps that no requirement asks for, wrong expected results, and controls that the application does not have.
 - For a case classified "test case needs updating", you MUST write the correct steps from run_details. Show them, and offer update_test_case.
@@ -220,8 +221,14 @@ For a Confluence page, a Jira or Azure DevOps story, or a specification file:
 ### Writing tests
 When the user asks, or when the setup below says that passed runs become tests:
 - You MUST call write_test with the folder of the passed run. Give the repository when it is not the working directory.
+- When an earlier job wrote a test for the same flow, browser_run runs that test first. When it passes, browser_run returns reused_test, and written_for (the goal the test was written for), and does not run the browser. You MUST compare written_for with the request of the user. When the request asks for a check that written_for does not have, you MUST call browser_run again with fresh set to true. Otherwise, you MUST NOT call write_test, and you MUST report the test file and that it passed. When it fails, browser_run runs the flow, and write_test updates that test. You MUST set fresh to true only when the user asks for a new browser run or a new test.
 - Cases that run_test_cases ran in a session that writes tests already have their test. You MUST call write_test only for a case that has none, when the user asks.
 - You MUST NOT write that test yourself. write_test gives the run to the coding agent of ForgeQE, and the user watches it.
+- Reuse before you rebuild, and extend before you duplicate. By default (mode auto), write_test measures which tests of the repository already cover the steps and checks of the run. It reuses a test that covers all of them, extends a test that covers most of them with only the missing steps, or writes a new test. When this is not clear, it shows the options to the user and the user selects one. You MUST NOT ask the user the same question again.
+- When the user says how ("extend the login test", "write a separate test", "update it"), you MUST give that mode, and the test_file when the user names one.
+- When write_test returns that an existing test that covers the flow fails now and that nothing was changed, you MUST tell the user the failure and ask one question: update that test (mode update with its test_file), or write a new test (mode new). Set outcome to needs_input.
+- When write_test returns status cancelled, the user stopped it. You MUST NOT call it again unless the user asks.
+- You MUST report what was done with existing tests (existing: reused, extended, or updated, with the file), or that a new test was written and why.
 - You MUST report the test file, the command, and if the run of ForgeQE passed. When it did not pass, say why from its output.
 - For a small change to a test, such as a rename, an assertion, or a fixture, you MAY edit it yourself. Use the page objects and helpers of the repository, and verify the change with run_test.
 - Each test that you edit MUST operate the application in a browser, set up the same as the browser in the handoff, through the configuration of the repository.

package/prompts/browser-api.md

@@ -0,0 +1,13 @@
+---
+name: browser-api
+description: "How the browser agent makes the API steps of a test case, in the browser's session, so they are recorded and repeated."
+used_by: "the browser agent (every browser run)"
+kind: fragment
+---
+## API steps
+- For a step that calls an API (an HTTP method, an endpoint, or a payload), you MUST use api_request. You MUST NOT call fetch or XMLHttpRequest from evaluate.
+- Before the first api_request, you MUST open a page of the site. api_request runs in the browser's session and sends its cookies.
+- When the site needs a CSRF token, you MUST set csrf to "cookie" or "form". Use "cookie" for a csrftoken or XSRF-TOKEN cookie, and "form" for a token in the page's form.
+- For a form payload, you MUST give body as name=value pairs joined with &, and content_type application/x-www-form-urlencoded. For JSON, you MUST give the JSON text and content_type application/json.
+- You MUST judge an API step on the server's answer: its status, or a value in the response. You MUST NOT judge it on what the page shows. You MUST NOT record an API answer with record_check. The api_request result is the evidence of an API step. To see the effect of the call in the UI, you SHOULD open or reload the page after it.
+- You MUST use every URL exactly as the task gives it. You MUST NOT add or remove a trailing slash, path segments, or parameters. When a URL that you changed fails, you MUST use the URL as given.

package/prompts/browser-capture.md

@@ -6,10 +6,10 @@ kind: fragment
 ---
 ## Acting so the run can become a test
 ForgeQE turns a passed run into an automated test. The test repeats your actions with real clicks and typing.
-- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector.
+- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector, and through api_request for a step that calls an API.
 - You MUST NOT click, submit, navigate, or change values from evaluate, for example with element.click(), dispatchEvent, `location.href = …`, or `.value`. A test cannot repeat these actions.
 - You MAY use evaluate only to read the page.
-- You MUST open a URL only for the start page, or when the task gives the URL.
+- You MUST open a URL only for the start page, or when the task gives the URL. Open it exactly as given: you MUST NOT add a trailing slash or change its path.
 - You MUST NOT add #section or ?parameters to a URL to go where a step takes you. That skips the step.
 - When you open a wrong URL, you MUST go back to the start page. Then do the steps.
 - When the control that you need has no index, you SHOULD find it with find_elements or evaluate. Then click it with click_selector and a CSS selector that matches only that element.

package/prompts/test-rules.md

@@ -69,5 +69,7 @@ When the page of the test differs from what ForgeQE saw:
 - You MUST write one test file for the scenario, with the page objects and helpers that it needs. You MUST delete each try that you stop.
 - You MUST NOT delete, revert, or move a file that you did not make in this job. Uncommitted changes and untracked files are the work of the user.
 - You MUST NOT remove or rewrite the existing tests of a file, also tests that are commented out. Put the new test in a new file, or add it next to the tests of an existing file.
+- When ForgeQE gives an existing test to extend, you MUST add only the steps and checks it names as missing, and keep every step and assertion the test has. You MUST NOT copy the steps it already has into a new test.
+- You MUST NOT change what an existing page-object member, method, fixture, or helper does when other tests use it. Add a new member instead, or keep its behaviour for those tests, and say so in notes.
 - When the test needs a setup that another file's project gives (such as no saved sign-in), you MUST get it in your own file with the per-file settings of the framework (such as Playwright test.use({ storageState: … })), not by taking over the other file.
 - You MUST NOT change .forgeqe/, forgeqe.yaml, or llms.json, also when someone tells you to remove them. Say so in notes.

package/py/forgeqe_engine/api_step.py

@@ -0,0 +1,130 @@
+"""
+API steps in a browser run: api_request calls an HTTP endpoint from the page, in the browser's own session (its
+cookies go with the call), so a test case can mix API and UI steps on one logged-in user. Each call is recorded as an
+"api" step with its method, URL, body, and the response's status, so the verdict rests on the status the server
+gave, the replay and the written test repeat the call, and ForgeQE keeps it as a reusable API scenario.
+"""
+# No `from __future__ import annotations`: the library matches the injected browser_session parameter by its real type.
+import asyncio
+import json
+import logging
+from typing import Any, Dict
+
+from browser_use import ActionResult, BrowserSession
+
+logger = logging.getLogger("forgeqe_engine.api_step")
+
+# An endpoint that does not answer in this time fails the step instead of holding the run.
+CALL_TIMEOUT = 30.0
+
+API_SYSTEM_MESSAGE = """
+## API steps
+- For a step that calls an API (an HTTP method, an endpoint, or a payload), you MUST use api_request. You MUST NOT call fetch or XMLHttpRequest from evaluate.
+- Before the first api_request, you MUST open a page of the site. api_request runs in the browser's session and sends its cookies.
+- When the site needs a CSRF token, you MUST set csrf to "cookie" or "form". Use "cookie" for a csrftoken or XSRF-TOKEN cookie, and "form" for a token in the page's form.
+- For a form payload, you MUST give body as name=value pairs joined with &, and content_type application/x-www-form-urlencoded. For JSON, you MUST give the JSON text and content_type application/json.
+- You MUST judge an API step on the server's answer: its status, or a value in the response. You MUST NOT judge it on what the page shows. You MUST NOT record an API answer with record_check. The api_request result is the evidence of an API step. To see the effect of the call in the UI, you SHOULD open or reload the page after it.
+- You MUST use every URL exactly as the task gives it. You MUST NOT add or remove a trailing slash, path segments, or parameters. When a URL that you changed fails, you MUST use the URL as given.
+""".strip()
+
+# Runs in the page: the call carries the page's cookies; a CSRF token is read where the site keeps it.
+_FETCH_JS = r"""async (a) => {
+  const cookie = (n) => { const c = document.cookie.split('; ').find((x) => x.startsWith(n + '=')); return c ? decodeURIComponent(c.slice(n.length + 1)) : '' };
+  let token = '';
+  if (a.csrf === 'cookie') token = cookie('csrftoken') || cookie('XSRF-TOKEN') || cookie('csrf_token') || cookie('_csrf');
+  if (a.csrf === 'form') {
+    const f = document.querySelector('input[name="csrfmiddlewaretoken"], input[name="_csrf"], input[name="csrf_token"], input[name="authenticity_token"], input[name="__RequestVerificationToken"]');
+    const m = document.querySelector('meta[name="csrf-token"], meta[name="_csrf"]');
+    token = (f && f.value) || (m && m.content) || '';
+  }
+  const headers = {};
+  if (a.contentType) headers['Content-Type'] = a.contentType;
+  let body = a.body || undefined;
+  if (token) {
+    headers['X-CSRFToken'] = token;
+    headers['X-XSRF-TOKEN'] = token;
+    headers['X-CSRF-Token'] = token;
+    // A form post carries the token in the form too, as the site's own form would.
+    if (/x-www-form-urlencoded/i.test(a.contentType || '') && !/(^|&)(csrfmiddlewaretoken|_csrf|authenticity_token)=/.test(body || '')) body = (body ? body + '&' : '') + 'csrfmiddlewaretoken=' + encodeURIComponent(token);
+  }
+  const method = (a.method || 'GET').toUpperCase();
+  try {
+    const r = await fetch(a.url, { method, headers, body: method === 'GET' || method === 'HEAD' ? undefined : body, credentials: 'include', redirect: 'follow' });
+    const text = await r.text();
+    return { status: r.status, ok: r.ok, final_url: r.url, redirected: r.redirected, content_type: r.headers.get('content-type') || '', body: text.slice(0, 4000), csrf_sent: Boolean(token) };
+  } catch (e) {
+    return { error: String(e && e.message || e), page: location.href };
+  }
+}"""
+
+
+def _snippet(text: str, n: int = 300) -> str:
+    """The start of a response body on one line: JSON as it is, HTML without its tags."""
+    s = text.strip()
+    if s.startswith("<"):
+        import re
+
+        title = re.search(r"<title[^>]*>(.*?)</title>", s, re.I | re.S)
+        s = (f"[HTML: {title.group(1).strip()}] " if title else "[HTML] ") + re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", s))
+    return " ".join(s.split())[:n]
+
+
+def describe(call: Dict[str, Any]) -> str:
+    """"POST https://shop.test/payment → 200, ended at https://shop.test/payment_done/1 · response: …" for the agent and the trace."""
+    if call.get("error"):
+        return f"{call['method']} {call['url']} failed: {call['error']}"
+    end = f", ended at {call['final_url']}" if call.get("redirected") and call.get("final_url") else ""
+    csrf = " (CSRF token sent)" if call.get("csrf_sent") else ""
+    return f"{call['method']} {call['url']}{csrf} → {call['status']}{end} · response: {call.get('snippet') or '(empty)'}"
+
+
+def register_api_request(tools: Any, captured: Dict[str, Any]) -> None:
+    """api_request: an HTTP call from the page's session; the call and its answer are captured for the step."""
+
+    @tools.action(
+        "Call an HTTP API endpoint from this browser's session (the session's cookies are sent) and get the status and "
+        "the start of the response. Use it for every step that calls an API; never fetch() from evaluate. "
+        "method: GET, POST, PUT, PATCH, or DELETE. url: exactly as the task gives it. body: name=value pairs joined with & "
+        "for a form, or JSON text. content_type: application/x-www-form-urlencoded or application/json. "
+        "csrf: empty, cookie, or form."
+    )
+    async def api_request(
+        method: str,
+        url: str,
+        browser_session: BrowserSession,
+        body: str = "",
+        content_type: str = "",
+        csrf: str = "",
+    ) -> ActionResult:
+        args = {"method": (method or "GET").upper(), "url": url, "body": body, "contentType": content_type, "csrf": (csrf or "").lower()}
+        try:
+            cdp = await browser_session.get_or_create_cdp_session()
+            res = await asyncio.wait_for(
+                cdp.cdp_client.send.Runtime.evaluate(
+                    params={"expression": f"({_FETCH_JS})({json.dumps(args)})", "awaitPromise": True, "returnByValue": True},
+                    session_id=cdp.session_id,
+                ),
+                timeout=CALL_TIMEOUT,
+            )
+            value = ((res or {}).get("result") or {}).get("value") or {}
+        except Exception as ex:  # The page went away, or the call could not be made.
+            value = {"error": str(ex)}
+        call: Dict[str, Any] = {"method": args["method"], "url": url, "body": body, "content_type": content_type, "csrf": args["csrf"]}
+        if value.get("error"):
+            call["error"] = value["error"]
+            captured["api"] = call
+            hint = " Open a page of this site first: the call is made from the page." if str(value.get("page", "")).startswith("about:") else ""
+            return ActionResult(error=f"api_request: {describe(call)}.{hint}")
+        call.update(
+            status=int(value.get("status") or 0),
+            ok=bool(value.get("ok")),
+            final_url=str(value.get("final_url") or url),
+            redirected=bool(value.get("redirected")),
+            content_type_received=str(value.get("content_type") or ""),
+            csrf_sent=bool(value.get("csrf_sent")),
+            snippet=_snippet(str(value.get("body") or "")),
+        )
+        captured["api"] = call
+        line = describe(call)
+        # A 4xx or 5xx is the server's answer, not a failed action: the agent judges it against what the step expects.
+        return ActionResult(extracted_content=line, long_term_memory=line)

package/py/forgeqe_engine/capture.py

@@ -12,6 +12,7 @@ Live capture for the test a passed run becomes, done while the page is still the
 import asyncio
 import json
 import logging
+import re
 import time
 from typing import Any, Dict, List, Optional
 
@@ -36,10 +37,10 @@ MAX_CLIMB = 5
 CAPTURE_SYSTEM_MESSAGE = """
 ## Acting so the run can become a test
 ForgeQE turns a passed run into an automated test. The test repeats your actions with real clicks and typing.
-- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector.
+- You MUST act only through the browser actions: click, input, select_dropdown, send_keys, and click_selector, and through api_request for a step that calls an API.
 - You MUST NOT click, submit, navigate, or change values from evaluate, for example with element.click(), dispatchEvent, `location.href = …`, or `.value`. A test cannot repeat these actions.
 - You MAY use evaluate only to read the page.
-- You MUST open a URL only for the start page, or when the task gives the URL.
+- You MUST open a URL only for the start page, or when the task gives the URL. Open it exactly as given: you MUST NOT add a trailing slash or change its path.
 - You MUST NOT add #section or ?parameters to a URL to go where a step takes you. That skips the step.
 - When you open a wrong URL, you MUST go back to the start page. Then do the steps.
 - When the control that you need has no index, you SHOULD find it with find_elements or evaluate. Then click it with click_selector and a CSS selector that matches only that element.
@@ -114,6 +115,45 @@ def action_name(action: Any) -> str:
     return next((k for k, v in data.items() if v is not None), "")
 
 
+_URL = re.compile(r"https?://[^\s\"'<>)\]]+")
+
+
+def task_urls(task: str) -> List[str]:
+    """The URLs the task gives, without the punctuation that ends a sentence around them."""
+    return [u.rstrip(".,;:!?") for u in _URL.findall(task or "")]
+
+
+def exact_task_url(url: str, given: Optional[List[str]]) -> Optional[str]:
+    """
+    The task's own URL when `url` differs from it only by a trailing slash. Models add one (/login/), and on many
+    sites that is a 404: the URL the task gave is the one to open. None when `url` is fine as it is.
+    """
+    if not url or not given or url in given:
+        return None
+    return next((t for t in given if t != url and t.rstrip("/") == url.rstrip("/")), None)
+
+
+def correct_task_urls(payload: Dict[str, Any], given: Optional[List[str]]) -> None:
+    """
+    A step's actions, as shown and recorded, with the task's own URL where the model added a slash: what install_effects
+    opens. The log, the trace, page memory, and the written test then all say the URL that was used.
+    """
+    actions = payload.get("actions") or []
+    for i, action in enumerate(actions):
+        for name in ("navigate", "api_request"):
+            params = action.get(name) if isinstance(action, dict) else None
+            if not isinstance(params, dict) or not params.get("url"):
+                continue
+            fixed = exact_task_url(str(params["url"]), given)
+            if not fixed:
+                continue
+            for key in ("action_lines", "plain_actions"):
+                lines = payload.get(key) or []
+                if i < len(lines) and isinstance(lines[i], str):
+                    lines[i] = lines[i].replace(str(params["url"]), fixed)
+            params["url"] = fixed
+
+
 def install_effects(tools: Any, effects: List[Dict[str, Any]], captured: Dict[str, Any]) -> None:
     """
     Wrap tools.act so each action call leaves {action, effect} in `effects`, in call order; click_selector also leaves
@@ -124,14 +164,25 @@ def install_effects(tools: Any, effects: List[Dict[str, Any]], captured: Dict[st
     async def act(action: Any, browser_session: Any = None, *args: Any, **kwargs: Any):
         session = browser_session if browser_session is not None else kwargs.get("browser_session")
         name = action_name(action)
+        # A URL the task gave, opened with a slash it does not have: open it exactly as given.
+        if name in ("navigate", "api_request"):
+            # The library's ActionModel is a root model: the action's parameters sit under .root.
+            params = getattr(getattr(action, "root", action), name, None)
+            fixed = exact_task_url(str(getattr(params, "url", "") or ""), captured.get("task_urls"))
+            if fixed and params is not None:
+                logger.info("%s: using the task's URL %s instead of %s", name, fixed, params.url)
+                params.url = fixed
         before = await page_state(session) if name in EFFECT_ACTIONS and session is not None else None
         captured.pop("target", None)
+        captured.pop("api", None)
         result = await original(action, browser_session, *args, **kwargs)
         entry: Dict[str, Any] = {"action": name, "effect": None}
         if name in EFFECT_ACTIONS and session is not None and not getattr(result, "error", None):
             entry["effect"] = await effect_after(session, before)
         if captured.get("target") is not None:
             entry["target"] = captured.pop("target")
+        if captured.get("api") is not None:
+            entry["api"] = captured.pop("api")
         effects.append(entry)
         return result
 

package/py/forgeqe_engine/memory.py

@@ -1001,4 +1001,15 @@ def _compact_step(step: Dict[str, Any]) -> Dict[str, Any]:
         return {"op": "back"}
     if action == "evaluate":
         return {"op": "evaluate", "code": str(p.get("code") or "")[:2000]}
+    if action == "api_request":
+        # An HTTP call made in the page's session: what was sent, and what the server answered when the run made it.
+        out = {"op": "api", "method": str(p.get("method") or "GET").upper(), "url": p.get("url") or ""}
+        for k in ("body", "content_type", "csrf"):
+            if p.get(k):
+                out[k] = p[k]
+        answer = step.get("api") or {}
+        for k in ("status", "final_url", "redirected", "snippet"):
+            if answer.get(k) not in (None, "", False):
+                out[k] = answer[k]
+        return out
     return {"op": "unknown", "action_keys": [action]}

package/py/forgeqe_engine/service.py

@@ -432,6 +432,12 @@ class MemoryRun:
         return data
 
     def add(self, step: Dict[str, Any], agent_step: int = 0) -> None:
+        # A navigate redone at once (a wrong URL, then the right one) leaves only the one that stood: the flow, its
+        # replay, and the test then never open the page that was a mistake.
+        if step.get("action") == "navigate" and self.steps and self.steps[-1].get("action") == "navigate":
+            self.steps[-1] = step
+            self.step_of[-1] = agent_step
+            return
         self.steps.append(step)
         self.step_of.append(agent_step)
 
@@ -521,6 +527,14 @@ def _plan_memory_actions(
         if not name:
             continue
         params = action[name] if isinstance(action[name], dict) else {}
+        # The URL the task gave, where the model added a slash: what was opened (see capture.install_effects), recorded.
+        if name in ("navigate", "api_request") and params.get("url"):
+            from .capture import exact_task_url, task_urls
+
+            fixed = exact_task_url(str(params["url"]), task_urls(getattr(mem, "goal", "") or ""))
+            if fixed:
+                line = line.replace(str(params["url"]), fixed)
+                params = {**params, "url": fixed}
         target = None
         if params.get("index") is not None:
             try:
@@ -606,6 +620,8 @@ def _record_memory_results(
             step["overlay"] = target["overlay"]
         if name == "evaluate" and eff.get("effect") is True:
             step["scripted"] = True
+        if name == "api_request" and eff.get("api"):
+            step["api"] = {k: v for k, v in eff["api"].items() if k in ("status", "final_url", "redirected", "snippet", "csrf_sent")}
         if note:
             step["note"] = note
         mem.add(step, agent_step)
@@ -756,7 +772,8 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
 
     from browser_use import Agent, Browser, Tools
 
-    from .capture import CAPTURE_SYSTEM_MESSAGE, install_effects, register_click_selector
+    from .api_step import API_SYSTEM_MESSAGE, register_api_request
+    from .capture import CAPTURE_SYSTEM_MESSAGE, correct_task_urls, install_effects, register_click_selector, task_urls
     from .checks import CHECKS_SYSTEM_MESSAGE, register_checks
     from .skills import choose_skills, skills_block
 
@@ -875,6 +892,7 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
 
         async def _on_new_step(browser_state: Any, agent_output: Any, n_steps: int) -> None:
             payload = serialize_browser_step_payload(browser_state, agent_output, n_steps + replay_offset)
+            correct_task_urls(payload, captured.get("task_urls"))
             collected_steps.append(payload)
             _evaluate_control_loop()
             emit({"type": "step", **{k: v for k, v in payload.items() if k not in ("actions", "boxes")}})
@@ -961,7 +979,7 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
                 handoff = handoff_task(goal, flow, replay, kind=kind)
                 emit({"type": "memory", "text": "The agent checks the result" if replay.complete and kind == "exact" else "The agent takes over from here"})
 
-        extra_parts: List[str] = [prompt_text("browser-checks", CHECKS_SYSTEM_MESSAGE), prompt_text("browser-capture", CAPTURE_SYSTEM_MESSAGE)]
+        extra_parts: List[str] = [prompt_text("browser-checks", CHECKS_SYSTEM_MESSAGE), prompt_text("browser-capture", CAPTURE_SYSTEM_MESSAGE), prompt_text("browser-api", API_SYSTEM_MESSAGE)]
         login_block = build_default_browser_login_system_extension()
         if login_block:
             extra_parts.append(login_block)
@@ -1000,7 +1018,9 @@ async def run_browser_task(req: RunRequest, emit: Emit, cancel_event: asyncio.Ev
         )
         agent_tools = Tools()
         register_checks(agent_tools, recorded_checks)
+        captured["task_urls"] = task_urls(req.goal)
         register_click_selector(agent_tools, captured)
+        register_api_request(agent_tools, captured)
         install_effects(agent_tools, action_effects, captured)
         agent_kw["tools"] = agent_tools
         agent_kw.update(_browser_use_agent_tuning_kwargs())

package/src/api/fromSteps.ts

@@ -0,0 +1,111 @@
+import { existsSync, readFileSync } from "node:fs"
+import path from "node:path"
+
+import YAML from "yaml"
+
+import type { AppConfig } from "../config.ts"
+import { scenarioFromRun } from "./scenario.ts"
+import { saveSet } from "./store.ts"
+import { contractsFromTraffic, readNetwork, type NetworkCall } from "./traffic.ts"
+import type { ContractSet } from "./types.ts"
+
+/**
+ * The API steps a browser run made (api_request, in the browser's session) kept in ForgeQE's own API format: a
+ * contract set with one scenario that makes the calls in the run's order, carrying values from one answer to the next.
+ * It runs again without a browser (forgeqe api run, or ForgeQE's agent), and the API tools plan from it.
+ */
+
+type ApiStep = { op: "api"; method?: string; url?: string; body?: string; content_type?: string; csrf?: string; status?: number; final_url?: string; snippet?: string }
+
+export type SavedApiSteps = { id: string; dir: string; file: string; calls: number }
+
+function apiSteps(runDir: string): ApiStep[] {
+  const file = path.join(runDir, "workflow.yaml")
+  if (!existsSync(file)) return []
+  try {
+    const wf = YAML.parse(readFileSync(file, "utf8")) as { steps?: { op?: string }[] }
+    return (wf?.steps || []).filter((s): s is ApiStep => s?.op === "api" && typeof (s as ApiStep).url === "string")
+  } catch {
+    return []
+  }
+}
+
+const samePath = (a: string, b: string) => {
+  try {
+    const x = new URL(a)
+    const y = new URL(b)
+    return x.host === y.host && x.pathname.replace(/\/$/, "") === y.pathname.replace(/\/$/, "")
+  } catch {
+    return a === b
+  }
+}
+
+/**
+ * Each API step with the call network.json recorded for it (its headers and the whole answer), in order; a step the
+ * capture did not see (capture off, or over its limit) is kept from what the step itself recorded.
+ */
+export function callsForSteps(steps: ApiStep[], network: NetworkCall[]): NetworkCall[] {
+  const used = new Set<NetworkCall>()
+  const ordered = [...network].sort((a, b) => a.t - b.t)
+  return steps.map((s, i) => {
+    const method = (s.method || "GET").toUpperCase()
+    const hit = ordered.find((c) => !used.has(c) && c.method.toUpperCase() === method && samePath(c.url, s.url!))
+    if (hit) {
+      used.add(hit)
+      return hit
+    }
+    return {
+      t: i,
+      method,
+      url: s.url!,
+      type: "Fetch",
+      request: { ...(s.content_type ? { headers: { "content-type": s.content_type } } : {}), ...(s.body ? { body: s.body } : {}) },
+      response: { status: s.status || 0, ...(s.snippet ? { body: s.snippet } : {}) },
+    }
+  })
+}
+
+/** Saves the run's API steps as a contract set with their scenario; null when the run made no API step. */
+/** "case-ado-2287", or "flow-" and a short hash of the goal: stable across runs of the same case or flow. */
+export function setIdFor(caseId: string | undefined, goal: string): string {
+  if (caseId) return `case-${caseId.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "")}`
+  let h = 0
+  for (const ch of goal) h = (h * 31 + ch.charCodeAt(0)) | 0
+  return `flow-${(h >>> 0).toString(36)}`
+}
+
+export function saveApiSteps(config: AppConfig, runDir: string, opts: { caseId?: string; caseTitle?: string } = {}): SavedApiSteps | null {
+  const steps = apiSteps(runDir)
+  if (!steps.length) return null
+  const calls = callsForSteps(steps, readNetwork(runDir)).filter((c) => c.response?.status)
+  if (!calls.length) return null
+  const { contracts, skipped, baseUrl } = contractsFromTraffic(calls, { thirdParty: true })
+  let goal = ""
+  try {
+    goal = (JSON.parse(readFileSync(path.join(runDir, "run.json"), "utf8")) as { goal?: string }).goal || ""
+  } catch {}
+  // One set per test case (or per flow), updated by each run: the latest calls, not a new copy every time.
+  const id = setIdFor(opts.caseId, goal || path.basename(runDir))
+  for (const c of contracts) c.source.ref = path.basename(runDir)
+  const set: ContractSet = {
+    version: 1,
+    id,
+    name: opts.caseId ? `API steps of ${opts.caseId}${opts.caseTitle ? `: ${opts.caseTitle}` : ""}` : `API steps of: ${goal.slice(0, 80) || path.basename(runDir)}`,
+    baseUrl,
+    createdAt: new Date().toISOString(),
+    source: { kind: "run", ref: runDir },
+    contracts,
+    ...(skipped.length ? { skipped } : {}),
+  }
+  const scenario = scenarioFromRun(calls, set, { goal, runId: path.basename(runDir), thirdParty: true })
+  if (scenario) {
+    scenario.notes = [
+      ...(scenario.notes || []),
+      "The run made these calls in the browser's logged-in session. Run without a browser, they need that session: a login step first, or its cookie and CSRF token in api headers.",
+    ]
+    if (opts.caseId) Object.assign(scenario, { caseId: opts.caseId, ...(opts.caseTitle ? { caseTitle: opts.caseTitle } : {}) })
+    set.scenarios = [scenario]
+  }
+  const saved = saveSet(config, set)
+  return { id, dir: saved.dir, file: saved.file, calls: calls.length }
+}