@capagents/forgeqe 1.2.1 → 1.3.0

package/prompts/brain.md

@@ -26,9 +26,11 @@ A message can also be a task specification with the parts Role, Objective, Conte
 **Tools** (ForgeQE tools; the client adds the forgeqe_ prefix):
 - browser_run: does one goal in a real browser, and verifies it. Use it for all work on a website. Set purpose to test for a flow of the application under test. Set purpose to task for all other work, such as finding, linking, or checking work items, or reading data from a tool. A task never becomes an automated test.
 - find_test_cases: reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, GitHub issues, files, or text. It does not run them.
-- save_test_cases: saves test cases that you wrote from a story into Azure DevOps (Test Case work items) or Jira, linked to the story. It gives a link for each case.
+- story_test_cases: gives the test cases already linked to an Azure DevOps or Jira story, with their steps. It changes nothing.
+- save_test_cases: saves test cases that you wrote from a story into Azure DevOps (Test Case work items) or Jira, linked to the story. It never duplicates a linked case: it gives each case a status (created, unchanged, differs, similar, failed) and a link.
 - check_test_cases: checks test cases before they run. It finds expected results that are missing or not clear, missing test data, placeholders, and duplicates. It runs nothing.
-- update_test_case: writes the steps of an Azure DevOps Test Case again, after the user agrees. It asks the user.
+- update_test_case: writes the steps of an Azure DevOps Test Case or a Jira test issue again, in place, after the user agrees. It asks the user.
+- e2e_journey: plans end-to-end journeys from test cases that exist (the cases of a story, or any sources). It keeps the steps they share once, and gives each case as a section. It changes nothing.
 - release_readiness: gives requirement coverage and a GO / GO WITH RISKS / NO-GO verdict for a sprint, area, epic, or stories. It uses test cases, their results, bugs, and flaky tests. It can publish to the wiki. Examples: "are we ready to release?", "coverage of sprint 12".
 - run_test_cases: runs test cases in real browsers, with a verdict for each step. It writes report.html (steps, screenshots, timings) and junit.xml. It sends results back to the source system. It files or updates bugs for failures when bugs are set up. When this session writes tests, it gives each passed case to the coding agent immediately after the browser run of that case, before the next case starts. It gives the test of each case.
 - read_confluence: reads Confluence pages (requirements, specifications) as text, by id, link, or CQL.
@@ -43,6 +45,9 @@ A message can also be a task specification with the parts Role, Objective, Conte
 - flaky_tests: gives the flaky tests from that history, with their probable cause.
 - defect_trends: gives how the defects of the saved suites went over time: still there, gone, or back after a fix, the time to fix, the most seen, and defects by area. It runs nothing.
 - predict_defects: gives the areas where a change in the product repository can cause defects, with the reasons and the cases to run first. It is a heuristic from the history. It runs nothing.
+- build_regression_pack: builds a regression pack for a release from an Azure DevOps sprint or work items, a commit range, pull requests, or changes you read from other connected tools: the tests the changes can break, each with why, and the changes no test covers. It runs nothing.
+- run_regression_pack: runs a saved pack after the user confirms, on this machine, on TestMu AI, or in a pipeline, and compares the run with the last one.
+- list_regression_packs: gives the saved packs and their last runs.
 - quarantine_tests: puts flaky tests in quarantine, or takes them out, after the user agrees. The failures of a test in quarantine do not fail runs or CI.
 - review_pull_request: Scribe reviews an Azure DevOps or GitHub pull request from the local clone (bugs, security, requirements, tests). The user sees the findings. It posts nothing.
 - post_pr_review: posts a review_pull_request review to the pull request after the user confirms.
@@ -165,6 +170,7 @@ Read data from Azure DevOps, Jira, Xray, Confluence, and other systems in this s
 - You MUST judge each case against that. Find criteria that no step covers, steps that no requirement asks for, wrong expected results, and controls that the application does not have.
 - For a case classified "test case needs updating", you MUST write the correct steps from run_details. Show them, and offer update_test_case.
 - You MUST NOT change a test case without update_test_case, which asks the user first.
+- When the user reviews a saved case and asks for a change, you MUST call update_test_case with the id of that case. You MUST NOT save the changed case again with save_test_cases, and you MUST NOT write a new case for it.
 - To link cases to a story, or to check a link, you MUST use the MCP tools or query_api, not the browser. When the browser is the only way, call browser_run with purpose task.
 - A browser run of a test case is the test. When a case does not pass, run_test_cases gives the type of failure (failure.type). You MUST report each failed case by that type, in plain words:
   - A functional defect: the application is wrong. Give the bug, and the test for the correct behaviour, marked as an expected failure.
@@ -183,17 +189,27 @@ Domain applies when you write and check test cases, test from requirements, do e
 ### Testing from requirements
 For a Confluence page, a Jira or Azure DevOps story, or a specification file:
 - You MUST read the requirements with read_confluence or the Confluence MCP tools, the MCP tools of the story or query_api, or the file.
+- For an Azure DevOps or Jira story, you MUST call story_test_cases before you write a case. Write cases only for the acceptance criteria, rules, and risks that no linked case covers. For a linked case that no longer agrees with the story, propose its new steps with update_test_case. You MUST NOT write a linked case again. When the linked cases cover everything, say so and write nothing.
 - You MUST write one case for each acceptance criterion or rule, and the main negative paths.
 - You MUST then write one case for each domain risk and obligation on your list that the story does not cover.
 - Each case MUST have a title, the start page when you know it, and numbered steps with an expected result, specific enough for a browser.
 - When the story is in Azure DevOps or Jira, and "This session" says that ForgeQE saves cases there, you MUST save them with save_test_cases before they run.
 - Give save_test_cases the system, the story, and each case with its steps and expected results. You MUST NOT create the cases with the MCP tools or the browser.
-- In the reply, you MUST list each saved case as a markdown link to its work item, with its title, and without its full steps.
+- save_test_cases writes nothing for a case with status differs or similar. You MUST NOT say that such a case was saved or updated. Show how it differs from linked_case, and offer update_test_case. Use force_new only when the user says that it is a different case.
+- In the reply, you MUST list each saved case as a markdown link to its work item, with its title and its status, and without its full steps.
 - When ForgeQE does not save cases there, or the user says not to, you MUST show them as a short markdown list with numbered titles, steps, and expected results.
 - You MUST then run them with run_test_cases, unless the user asked only for the cases.
 - You MUST run saved Azure DevOps cases by their ids (save_test_cases gives run_with), so that the results go back. Run other cases as text sources, one entry for each case.
 - You MUST save cases from other requirements into a test management system only when the user asks. Use save_test_cases when there is a story to link them to. Give the new ids as links.
 
+### End-to-end journeys from test cases
+For "an end-to-end test for story 1234", "combine these cases into one flow", or "E2E from the test plan":
+- You MUST build on the cases that exist. Call e2e_journey with the story, or with the sources. You MUST NOT write the cases again first.
+- When the story has no linked cases yet, write and save them by "Testing from requirements" first, then call e2e_journey.
+- Order the sections so that each one leaves the application in the state that the next one needs. A negative section goes last, or stays its own case.
+- Show each journey with the cases that it covers. When the user agrees, save it with save_test_cases as one case titled "E2E: …", with a description that names the cases it covers. Keep the original cases.
+- For end-to-end API scenarios, use api_scenarios.
+
 ### Signing in
 - The browser agent signs in to the sites under "Saved logins" with stored credentials and a saved session. You MUST NOT ask the user for those passwords or codes.
 - You MUST NOT put credentials in goals.
@@ -218,10 +234,25 @@ For a Confluence page, a Jira or Azure DevOps story, or a specification file:
 - You MUST name the files that it made: report.html in the run folder, and the test that write_test wrote. Say if the run of that test passed. When ForgeQE wrote no test, say so and why.
 - When the user wants logs or details to share for debugging, tell them to type /logs, and add shots for screenshots. Outside the shell, forgeqe logs bundle writes one archive with secrets replaced.
 
+### Regression packs
+When the user asks for a regression pack, or which tests to run for a release, a sprint, or recent pull requests:
+- You MUST give build_regression_pack every source the user names. Give an Azure DevOps sprint, area, epic, or work items directly. Give a commit range as from and to, with range_repo when it is the product repository. Give pull requests as prs or merged_days.
+- When the changes are in another connected tool (a Jira release or fix version, GitHub issues, a release note in Confluence), you MUST read them with that tool first. Then you MUST pass each one in items with its id, title, text, url, and the test cases and work items it links to. You MUST NOT shorten their text: the words find the tests.
+- You MUST report the tests by why they are in, the changes no test covers with the closest test for each, and the risk areas.
+- When the user did not say where to run the pack, you MUST ask one question: on this machine, on TestMu AI, or in a pipeline. You MUST NOT run a pack until the user agrees.
+- After a run, you MUST name the failing tests with their messages, and the tests that fail now but passed in the last run.
+
 ### Writing tests
 When the user asks, or when the setup below says that passed runs become tests:
 - You MUST call write_test with the folder of the passed run. Give the repository when it is not the working directory.
-- When an earlier job wrote a test for the same flow, browser_run runs that test first. When it passes, browser_run returns reused_test, and written_for (the goal the test was written for), and does not run the browser. You MUST compare written_for with the request of the user. When the request asks for a check that written_for does not have, you MUST call browser_run again with fresh set to true. Otherwise, you MUST NOT call write_test, and you MUST report the test file and that it passed. When it fails, browser_run runs the flow, and write_test updates that test. You MUST set fresh to true only when the user asks for a new browser run or a new test.
+- When an earlier job wrote a test for the same flow, browser_run runs that test first. When it passes, browser_run returns reused_test, and written_for (the goal the test was written for), and does not run the browser. You MUST compare written_for with the request of the user. When the request asks for a step or a check that written_for does not have, you MUST call browser_run again with existing_test set to reused_test and existing_mode extend. Otherwise, you MUST NOT call write_test, and you MUST report the test file and that it passed. When it fails, browser_run runs the flow, and write_test updates that test. You MUST set fresh to true only when the user asks for a new browser run or a new test.
+- Before a browser run for a test, browser_run searches the tests of the repository. When some are close to the request but not the same flow, it returns status existing_tests_found and does not start the browser. For each test it gives its steps, the elements of the request that it covers and lacks, and written_for when an earlier job wrote it. You MUST compare them with the request of the user before any browser run. Then you MUST call browser_run again with the same goal and one of these:
+  - existing_test (the test, Class.Method) and existing_mode reuse, when that test already does and checks everything the request asks.
+  - existing_mode extend, when the test covers the start of the journey and the request adds a few steps or checks to it.
+  - existing_mode sibling, when the request is a variant of the test: another path, other data, more than a few added steps, or a test that is long already. A new test goes beside it in its class and reuses its setup and steps.
+  - fresh set to true, when no test is this journey. The new test still reuses the page objects and helpers of the repository.
+  When existing_tests_found has sent_as task, you sent a check of the application as a task, and tests of the repository are close to it. When one of them does what the check asks, you MUST answer the check with it: call browser_run again with the same goal and purpose, existing_test set to that test, and existing_mode reuse. When the user asked for a test, you MUST use purpose test and one of the choices above. You MUST set fresh to true only when no test does what the check asks, or when the check needs data that a passing test does not give, such as a value to read.
+  You MUST NOT write a new test file for a journey that an existing test covers in part. After extend or sibling, write_test does what you chose. You MUST NOT give it a mode.
 - Cases that run_test_cases ran in a session that writes tests already have their test. You MUST call write_test only for a case that has none, when the user asks.
 - You MUST NOT write that test yourself. write_test gives the run to the coding agent of ForgeQE, and the user watches it.
 - Reuse before you rebuild, and extend before you duplicate. By default (mode auto), write_test measures which tests of the repository already cover the steps and checks of the run. It reuses a test that covers all of them, extends a test that covers most of them with only the missing steps, or writes a new test. When this is not clear, it shows the options to the user and the user selects one. You MUST NOT ask the user the same question again.

package/prompts/browser-checks.md

@@ -12,6 +12,8 @@ ForgeQE turns a passed run into an automated test. The test asserts the results
 - You MUST set `dynamic` to true for live data that changes on a different run, such as the date, a price, or a count. The test then asserts the rule, not this value.
 - You MUST record only the results that the task tells you to verify.
 - When the task lists expected results, you MUST record one check for each, and copy `expected` from that line.
+- When the task is a test case with numbered steps, you MUST set `step` to the number of the step that the result belongs to.
+- You MUST record the check of a step as soon as its expected result shows, before you start the next step.
 - ForgeQE finds the element that shows `text`. You MUST select text that points to the subject of the line:
   - For the items in a list, copy text from one item, such as the name of one result. Do not copy a summary line such as "2,463 properties found".
   - For a page or a section that opens, copy its heading or a different label that names it.

package/py/forgeqe_engine/checks.py

@@ -25,6 +25,8 @@ ForgeQE turns a passed run into an automated test. The test asserts the results
 - You MUST set `dynamic` to true for live data that changes on a different run, such as the date, a price, or a count. The test then asserts the rule, not this value.
 - You MUST record only the results that the task tells you to verify.
 - When the task lists expected results, you MUST record one check for each, and copy `expected` from that line.
+- When the task is a test case with numbered steps, you MUST set `step` to the number of the step that the result belongs to.
+- You MUST record the check of a step as soon as its expected result shows, before you start the next step.
 - ForgeQE finds the element that shows `text`. You MUST select text that points to the subject of the line:
   - For the items in a list, copy text from one item, such as the name of one result. Do not copy a summary line such as "2,463 properties found".
   - For a page or a section that opens, copy its heading or a different label that names it.
@@ -72,9 +74,10 @@ def register_checks(tools: Any, checks: List[Dict[str, Any]]) -> None:
         "Record one result the task asks you to verify, before done: expected (the result in a few words), "
         "text (the shortest exact text on the page that shows it, copied as shown), and dynamic=true when that text "
         "is live data that will differ on another run (today's date, a price, a count, today's featured item). "
-        "holds=false when the page does not show the expected result: then text is what the page shows instead."
+        "holds=false when the page does not show the expected result: then text is what the page shows instead. "
+        "step: when the task is a test case with numbered steps, the number of the step whose expected result this is."
     )
-    async def record_check(expected: str, text: str, browser_session: BrowserSession, dynamic: bool = False, holds: bool = True) -> ActionResult:
+    async def record_check(expected: str, text: str, browser_session: BrowserSession, dynamic: bool = False, holds: bool = True, step: int = 0) -> ActionResult:
         expected, text = " ".join(str(expected or "").split())[:200], " ".join(str(text or "").split())[:200]
         if not text:
             return ActionResult(error="Give the text on the page that shows the result.")
@@ -91,6 +94,9 @@ def register_checks(tools: Any, checks: List[Dict[str, Any]]) -> None:
         if not holds:
             # A failed expectation is a test result: `text` is what the page shows instead.
             check["holds"] = False
+        if isinstance(step, int) and step > 0:
+            # Which test step this checks: memory records where that step ended, to replay it after the case is edited.
+            check["step"] = step
         # The element showing the text and its section, with locators checked on this page, for the test's assertion.
         found = await locate_check(browser_session, text, dynamic=dynamic)
         if found:

package/py/forgeqe_engine/llm.py

@@ -6,7 +6,7 @@ import logging
 import os
 import re
 import time
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Callable, Dict, List, Optional, Tuple
 
 logger = logging.getLogger("forgeqe_engine.service")
 
@@ -157,18 +157,46 @@ def _build_llm(llm_config: Optional[Dict[str, Any]]):
     raise ValueError(f"Unsupported LLM provider for the engine: {provider}")
 
 
-def _time_llm_calls(llm: Any, calls: List[Tuple[float, float]]) -> None:
-    """Record when each model call started and ended, so the report can split step time into model and browser."""
+def call_usage(result: Any) -> Dict[str, int]:
+    """One model call's tokens from its completion (browser-use ChatInvokeUsage); prompt tokens include cached ones."""
+    u = getattr(result, "usage", None)
+    if u is None:
+        return {}
+    out = {"input": int(getattr(u, "prompt_tokens", 0) or 0), "output": int(getattr(u, "completion_tokens", 0) or 0)}
+    cached = int(getattr(u, "prompt_cached_tokens", 0) or 0)
+    written = int(getattr(u, "prompt_cache_creation_tokens", 0) or 0)
+    if cached:
+        out["cached"] = cached
+    if written:
+        out["cache_write"] = written
+    return out
+
+
+def _time_llm_calls(llm: Any, calls: List[Tuple[float, float]], on_call: Optional[Callable[[Dict[str, Any]], None]] = None) -> None:
+    """
+    Record when each model call started and ended, so the report can split step time into model and browser;
+    on_call also gets each call's time and tokens (counts only, never the prompt), for ForgeQE's usage ledger.
+    """
     inner = getattr(llm, "ainvoke", None)
     if inner is None:
         return
 
     async def timed(*args: Any, **kwargs: Any) -> Any:
         started = time.time()
+        result: Any = None
+        ok = False
         try:
-            return await inner(*args, **kwargs)
+            result = await inner(*args, **kwargs)
+            ok = True
+            return result
         finally:
-            calls.append((started, time.time()))
+            ended = time.time()
+            calls.append((started, ended))
+            if on_call is not None:
+                try:
+                    on_call({"ms": int((ended - started) * 1000), "ok": ok, **call_usage(result)})
+                except Exception:
+                    logger.debug("model call usage not reported", exc_info=True)
 
     try:
         object.__setattr__(llm, "ainvoke", timed)

package/py/forgeqe_engine/memory.py

@@ -5,6 +5,7 @@ Layout under the memory directory:
 
   pages/<host>/<page>.json   every interactive element seen on that page, each with ranked locators
   flows/<goal>-<hash>.json   the actions of the last successful run of a goal, replayable without the LLM
+  cases/<hash>.json          for a test case: which flow it last passed with, so an edited case replays its unchanged steps
 
 A locator's confidence (1-99 %) multiplies four factors: how stable its kind is (test id > id >
 role+name > ... > xpath), how unique it was on the page, how consistently it reappeared on later
@@ -650,22 +651,44 @@ class MemoryStore:
         replayed: int = 0,
         llm_steps: int = 0,
         resume: Optional[Dict[str, Any]] = None,
+        case: Optional[Dict[str, Any]] = None,
+        checks: Optional[List[Dict[str, Any]]] = None,
     ) -> Optional[str]:
         """
         Update the goal's flow after a run. Only successful runs replace its steps. Returns "saved" or "updated".
 
         A failed run with `resume` (the steps that worked before it went wrong) keeps them under "resume", so the
         next run of this task replays them and the agent picks up where the failed run went wrong.
+
+        `case` ({id, keys, ends}) is the test case the goal came from: a key per case step, and for each the number
+        of flow steps done by the end of it (None where unknown). A successful run records it and points the case's
+        index at this flow, so a run of the case after an edit replays the steps before the first one that changed.
         """
         if status == "stopped":
             return None
         with _locked(self.root):
             return self._save_flow_result(goal, start_url, status=status, steps=steps, summary=summary,
-                                          final_url=final_url, replayed=replayed, llm_steps=llm_steps, resume=resume)
+                                          final_url=final_url, replayed=replayed, llm_steps=llm_steps, resume=resume, case=case,
+                                          checks=checks)
+
+    def case_file(self, case_id: str) -> Path:
+        return self.root / "cases" / f"{_sha(case_id, 16)}.json"
+
+    def load_case_flow(self, case_id: str) -> Optional[Dict[str, Any]]:
+        """The flow a test case last passed with, when it recorded where each case step ended."""
+        index = _read_json(self.case_file(case_id)) if case_id else None
+        name = str((index or {}).get("flow") or "")
+        if not name or "/" in name or "\\" in name:
+            return None
+        flow = _read_json(self.root / "flows" / name)
+        if not flow or flow.get("case_id") != case_id or not flow.get("successes") or not isinstance(flow.get("steps"), list):
+            return None
+        return flow
 
     def _save_flow_result(self, goal: str, start_url: Optional[str], *, status: str, steps: List[Dict[str, Any]],
                           summary: str, final_url: str, replayed: int, llm_steps: int,
-                          resume: Optional[Dict[str, Any]] = None) -> Optional[str]:
+                          resume: Optional[Dict[str, Any]] = None, case: Optional[Dict[str, Any]] = None,
+                          checks: Optional[List[Dict[str, Any]]] = None) -> Optional[str]:
         file = self.flow_file(goal, start_url)
         flow = self.load_flow(goal, start_url)
         created = flow is None
@@ -689,6 +712,18 @@ class MemoryStore:
             flow["llm_steps"] = llm_steps
             flow["replayed_steps"] = replayed
             flow.pop("resume", None)
+            # checks: what the run checked, with the flow step each came after (observe.py); None keeps the flow's.
+            # New steps from the agent with no checks given leave none, since old positions would not fit them.
+            if checks is not None:
+                flow["checks"] = checks
+            elif llm_steps and flow.get("checks"):
+                flow.pop("checks", None)
+            if case and case.get("id") and case.get("keys"):
+                kept = len(flow["steps"])
+                flow["case_id"] = str(case["id"])
+                flow["case_step_keys"] = [str(k) for k in case["keys"]]
+                flow["case_step_ends"] = [e if isinstance(e, int) and 0 < e <= kept else None for e in (case.get("ends") or [])][: len(case["keys"])]
+                _write_json(self.case_file(str(case["id"])), {"case_id": str(case["id"]), "flow": file.name, "updated": now})
         elif status != "complete":
             flow["failures"] = int(flow.get("failures") or 0) + 1
         if keep_resume and resume:
@@ -702,6 +737,28 @@ class MemoryStore:
             return None
         return "saved" if created else "updated"
 
+    @staticmethod
+    def case_prefix(flow: Dict[str, Any], keys: List[str]) -> Tuple[int, int]:
+        """
+        How much of a case's earlier flow still fits its steps now: (case steps, flow steps). The case steps are the
+        leading ones whose keys did not change, up to the last of them where the flow recorded its end. (0, 0) when none.
+
+        An end is trusted only when the next step that recorded one ended later: when the agent did the actions of two
+        steps before recording either check, both ends are the same, and replaying up to the first would also redo
+        the second step's actions, which may be the step that changed.
+        """
+        old = [str(k) for k in (flow.get("case_step_keys") or [])]
+        ends = [e if isinstance(e, int) and 0 < e <= len(flow.get("steps") or []) else None for e in (flow.get("case_step_ends") or [])]
+        same = 0
+        while same < min(len(old), len(keys)) and old[same] == keys[same]:
+            same += 1
+        for n in range(same, 0, -1):
+            end = ends[n - 1] if n - 1 < len(ends) else None
+            later = next((e for e in ends[n:] if e is not None), None)
+            if end is not None and (later is None or later > end):
+                return n, end
+        return 0, 0
+
     def recent_flows(self, host: str, *, exclude: Optional[Path] = None, limit: int = 2) -> List[Dict[str, Any]]:
         folder = self.root / "flows"
         if not folder.is_dir():

package/py/forgeqe_engine/observe.py

@@ -0,0 +1,123 @@
+"""
+Replaying a flow without the model, and knowing it still passes (docs/COST_ARCHITECTURE.md, phase 2).
+
+Two parts:
+- read_only_script: whether an `evaluate` step in a flow only read the page. The agent often runs a script to look at
+  the page (list the fields, read their values) before it decides; replay can leave such a step out. The test is on
+  the script's source and conservative: anything that could write (assign, call a method that changes the page or the
+  browser's state, send a request, schedule code) counts as writing, and the step then needs the agent as before.
+- checks_for_flow / flow_checks_hold: the results a passing run recorded (record_check), kept in its flow with the
+  step each was recorded after, and checked again on the replayed page with no model.
+"""
+from __future__ import annotations
+
+import asyncio
+import re
+from typing import Any, Awaitable, Callable, Dict, List, Optional
+
+# Anything that writes to the page, the browser, or the network, or runs code later. A script with any of these
+# is not read-only. False positives only cost a model call (today's path); a false negative could skip a real step,
+# so the list leans wide.
+_WRITES = re.compile(
+    r"""
+    (?<![=!<>])=(?![=>])                                   # any assignment (=, +=, -=, ...), not ==, ===, =>, <=, >=, !=
+    | \+\+ | --                                            # increments
+    | \b(?:delete)\b
+    | \.(?:click|submit|requestSubmit|reset|focus|blur|select|setSelectionRange|setRangeText|showModal|show|close|open|
+         dispatchEvent|remove|removeChild|append|appendChild|prepend|insertBefore|insertAdjacent\w*|replaceChild|
+         replaceWith|replaceChildren|before|after|setAttribute\w*|removeAttribute\w*|toggleAttribute|
+         scroll\w*|play|pause|load|reload|assign|pushState|replaceState|back|forward|go|
+         setItem|removeItem|clear|add|toggle|set|setProperty|removeProperty|attachShadow|
+         sendBeacon|postMessage|writeText|write|writeln|execCommand|requestFullscreen|exitFullscreen)\s*\(
+    | \b(?:fetch|XMLHttpRequest|WebSocket|EventSource|eval|Function|setTimeout|setInterval|requestAnimationFrame|
+         queueMicrotask|importScripts|import)\s*\(
+    | \bnew\s+(?:\w*Event|XMLHttpRequest|WebSocket|Worker|SharedWorker|Function|MutationObserver|BroadcastChannel)\b
+    | \b(?:document\.cookie|location|indexedDB|caches|navigator\.serviceWorker)\b
+    """,
+    re.VERBOSE,
+)
+
+# Declarations (const x = ..., let x = ..., var x = ..., function parameters with defaults) write only local names.
+_DECLARATION = re.compile(r"\b(?:const|let|var)\s+(?:[A-Za-z_$][\w$]*|\{[^}]*\}|\[[^\]]*\])\s*=(?![=>])")
+# Text inside string and template literals: an "=" or ".click(" in a selector or a message is not code.
+_STRINGS = re.compile(r"""'(?:\\.|[^'\\])*'|"(?:\\.|[^"\\])*"|`(?:\\.|[^`\\])*`""", re.DOTALL)
+# Comments.
+_COMMENTS = re.compile(r"//[^\n]*|/\*.*?\*/", re.DOTALL)
+
+
+def read_only_script(code: str) -> bool:
+    """Whether a script only reads the page: no assignment to anything but its own new local names, no call that changes
+    the page, the browser, or storage, no request, and no code run later."""
+    src = str(code or "")
+    if not src.strip() or len(src) > 20_000:
+        return False
+    src = _COMMENTS.sub(" ", src)
+    src = _STRINGS.sub('""', src)
+    # `const x = ...`: the assignment is the declaration of a local name. Remove the "=" of each declaration only;
+    # the expression after it is still checked.
+    src = _DECLARATION.sub(lambda m: m.group(0)[:-1] + " ", src)
+    # Parameters with defaults, `(a = 1) =>`: rare in these scripts; treated as an assignment, so not read-only.
+    return _WRITES.search(src) is None
+
+
+def read_only_step(step: Dict[str, Any]) -> bool:
+    """A flow step replay can leave out: an `evaluate` whose script only reads the page."""
+    return step.get("action") == "evaluate" and read_only_script(str((step.get("params") or {}).get("code") or ""))
+
+
+# ---------------------------------------------------------------------------------------------
+# Recorded checks
+# ---------------------------------------------------------------------------------------------
+
+CHECK_ATTEMPTS = 6
+CHECK_DELAY_SECONDS = 0.5
+
+
+def checks_for_flow(checks: List[Dict[str, Any]], at: Dict[str, int]) -> List[Dict[str, Any]]:
+    """The checks a passing run recorded, as its flow keeps them: what to find on the page, and after which flow step."""
+    out: List[Dict[str, Any]] = []
+    for c in checks or []:
+        text = " ".join(str(c.get("text") or "").split())
+        if not text:
+            continue
+        entry: Dict[str, Any] = {"expected": str(c.get("expected") or text)[:200], "text": text[:200], "at": int(at.get(check_key(c), 0))}
+        if c.get("url"):
+            entry["url"] = str(c["url"])
+        if c.get("dynamic"):
+            entry["dynamic"] = True
+        if c.get("holds") is False:
+            entry["holds"] = False
+        if isinstance(c.get("step"), int) and c["step"] > 0:
+            entry["step"] = c["step"]
+        out.append(entry)
+    return out
+
+
+def check_key(c: Dict[str, Any]) -> str:
+    return str(c.get("expected") or c.get("text") or "").strip().lower()
+
+
+def checkable(flow: Dict[str, Any]) -> Optional[str]:
+    """None when the flow's checks can decide a replay on their own; otherwise why not, in words."""
+    checks = flow.get("checks")
+    if not isinstance(checks, list) or not checks:
+        return "the flow has no recorded checks"
+    if any(c.get("dynamic") for c in checks):
+        return "a recorded check is live data (dynamic), which only the agent can judge"
+    if any(c.get("holds") is False for c in checks):
+        return "a recorded check did not hold when it was recorded"
+    return None
+
+
+async def check_holds(shows: Callable[[str], Awaitable[Optional[bool]]], text: str, attempts: Optional[int] = None, delay: Optional[float] = None) -> Optional[bool]:
+    """Whether the page shows `text`, asking again for a few seconds while it settles; None when it could not be asked."""
+    attempts = CHECK_ATTEMPTS if attempts is None else attempts
+    delay = CHECK_DELAY_SECONDS if delay is None else delay
+    seen: Optional[bool] = None
+    for n in range(max(1, attempts)):
+        seen = await shows(text)
+        if seen:
+            return True
+        if n + 1 < attempts:
+            await asyncio.sleep(delay)
+    return seen

package/py/forgeqe_engine/replay.py

@@ -16,6 +16,7 @@ from typing import Any, Awaitable, Callable, Dict, List, Optional, Tuple
 
 from . import brand
 from .logins import totp_code
+from .observe import check_holds, read_only_step
 from .memory import (
     ELEMENT_ACTIONS,
     READ_ONLY_ACTIONS,
@@ -45,6 +46,10 @@ class ReplayOutcome:
     times: List[Tuple[float, float]] = field(default_factory=list)
     # Steps whose recorded locator no longer matched and another one found the element.
     heals: List[Dict[str, Any]] = field(default_factory=list)
+    # The flow's recorded checks found on the page again, each after the step it was recorded after (observe.py).
+    checked: List[Dict[str, Any]] = field(default_factory=list)
+    # Steps left out because they only read the page (an `evaluate` with a read-only script).
+    skipped_reads: int = 0
 
     @property
     def complete(self) -> bool:
@@ -86,7 +91,13 @@ async def replay_flow(
     cancel_event: asyncio.Event,
     on_step: Optional[Callable[[], Awaitable[None]]] = None,
     overlay: Any = None,
+    checks_first: bool = False,
 ) -> ReplayOutcome:
+    """
+    `checks_first` (registry.mode on, docs/COST_ARCHITECTURE.md): steps that only read the page are left out, and the
+    flow's recorded checks are found on the page again after the step each was recorded after. A check that does not
+    hold stops the replay there, so the agent takes over on that page as after any other stop.
+    """
     from browser_use import Tools
 
     tools = Tools()
@@ -99,6 +110,23 @@ async def replay_flow(
         out.reason = reason
         return out
 
+    checks = [c for c in (flow.get("checks") or []) if isinstance(c, dict) and c.get("text")] if checks_first else []
+
+    async def verify_checks(after: int) -> Optional[str]:
+        """Finds again the checks recorded after flow step `after`; the first that does not hold, in words."""
+        due = [c for c in checks if (int(c.get("at") or 0) == after) or (after == len(steps) and not 0 < int(c.get("at") or 0) <= len(steps))]
+        if not due:
+            return None
+        from .checks import page_shows
+
+        for c in due:
+            holds = await check_holds(lambda t: page_shows(browser, t), str(c["text"]))
+            out.checked.append({**c, "holds_now": bool(holds)})
+            emit({"type": "result", "step": after, "items": [{"ok": bool(holds), "done": False, "text": f'check "{c.get("expected") or c["text"]}": {"shows" if holds else "does not show"} "{c["text"]}"'[:240]}]})
+            if not holds:
+                return f'the recorded check "{c.get("expected") or c["text"]}" does not hold: the page does not show "{c["text"]}"'
+        return None
+
     for n, step in enumerate(steps, 1):
         if cancel_event.is_set():
             stop(n, "stopped by user")
@@ -106,6 +134,17 @@ async def replay_flow(
         action = str(step.get("action") or "")
         params = dict(step.get("params") or {})
         line = str(step.get("line") or action)
+        if checks_first and read_only_step(step):
+            # The agent looked at the page here; replay has nothing to do. Kept in the flow so check positions hold.
+            out.done.append(step)
+            out.times.append((time.time(), time.time()))
+            out.emitted = n
+            out.skipped_reads += 1
+            failed = await verify_checks(n)
+            if failed:
+                stop(n + 1 if n < len(steps) else n, failed)
+                break
+            continue
         if action not in REPLAYABLE_ACTIONS:
             stop(n, f"`{line}` needs the agent")
             break
@@ -188,6 +227,10 @@ async def replay_flow(
                 out.heals.append({"step": n, "line": line, "label": step.get("label") or "", "stale": [_locator_text(k) for k in gone], "used": _locator_text(used)})
         if on_step is not None:
             await on_step()
+        failed = await verify_checks(n)
+        if failed:
+            stop(n + 1 if n < len(steps) else n, failed)
+            break
 
     out.url = await _current_url(browser)
     return out
@@ -197,11 +240,14 @@ def handoff_task(goal: str, flow: Dict[str, Any], outcome: ReplayOutcome, *, kin
     """
     The agent's task after a replay: the goal, what already happened, and where to pick up.
 
-    `kind` is "exact" (an earlier successful run of this task), "resume" (the steps that worked in a run of this
+    `kind` is "exact" (an earlier successful run of this task), "prefix" (the recorded steps of a test case's unchanged
+    first steps, after the case was edited), "resume" (the steps that worked in a run of this
     task that later went wrong), or "interrupted" (the steps a run of this task got through before it was cut off: closed, crashed, or the machine restarted).
     """
     when = str(flow.get("last_success") or flow.get("at") or "")[:10]
-    if kind == "resume":
+    if kind == "prefix":
+        source = "an earlier successful run of this test case, before its later steps were changed"
+    elif kind == "resume":
         source = "an earlier run of this task that went wrong later"
     elif kind == "interrupted":
         source = "an earlier run of this task that was cut off before it finished"
@@ -221,6 +267,13 @@ def handoff_task(goal: str, flow: Dict[str, Any], outcome: ReplayOutcome, *, kin
             f"All recorded steps ran and the browser is on {outcome.url or 'the final page'}. Check this page: "
             "if the task is complete, call done right away and report the result. Otherwise continue from here."
         )
+    elif kind == "prefix":
+        n = int(flow.get("case_steps_done") or 0)
+        lines.append(
+            f"They did test steps 1 to {n}, which have not changed. The browser is on {outcome.url or 'the current page'}. "
+            f"Check that the page agrees, then continue with test step {n + 1}. In your step report, report steps 1 to {n} "
+            "from what the page shows now."
+        )
     elif kind == "interrupted":
         lines.append(
             f"That is as far as the earlier run got before it was cut off. The browser is on {outcome.url or 'the current page'}. "