@capagents/forgeqe 1.2.0 → 1.3.0

package/HOW_TO_USE.md

@@ -50,13 +50,14 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
 | Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory`, `browser_devices` | [docs](#4-without-the-shell) |
 | Browser | **Saved logins for the application under test**: Signs the browser agent in with saved credentials (including TOTP) and reuses the session; a user can also sign in by hand once in a browser. | `logins` | [docs](#signing-in) |
 | Test cases | **Run manual test cases in browsers**: Reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, a file, or pasted text, runs each in a real browser with a verdict per step, writes report.html and junit.xml, posts results back, and files or updates bugs for failures. | `find_test_cases`, `run_test_cases` | [docs](#7-test-cases-and-suites) |
-| Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps cases once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
-| Test cases | **Write test cases from a story**: Writes test cases from a story's acceptance criteria and saves them in Azure DevOps or Jira, linked to the story. | `save_test_cases` | [docs](#where-cases-come-from) |
+| Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps or Jira cases in place once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
+| Test cases | **Write test cases from a story**: Writes test cases from a story's acceptance criteria and saves them in Azure DevOps or Jira, linked to the story. Reads the cases already linked first, so asking again adds only what is missing and updates cases in place instead of duplicating them; plans end-to-end journeys from the existing cases. | `story_test_cases`, `save_test_cases`, `e2e_journey` | [docs](#where-cases-come-from) |
 | Test cases | **Domain and regulatory test design**: Every agent works as an expert in the product's industry would, automatically: the risks the industry is known for and the obligations of its regulations and standards (for example DORA, PSD2, GDPR, ISO 27001), declared in forgeqe.yaml or worked out from the story and the application. It applies to analysing requirements, writing and checking test cases, exploration, writing, reviewing, and healing test code, API tests, finding tests, and pull request reviews, labelled so coverage can be traced and citing only the control ids in the team's compliance file. forgeqe eval domain measures how well the agent does it. |  | [docs](#16-domain-and-regulations) |
 | Test cases | **Exploratory testing of a story**: Explores a story in real browsers beyond its scripted cases (edge cases, invalid input, boundaries, roles, state), confirms each defect by reproducing it, proposes test cases, and files bugs once the user agrees. | `explore_story` | [docs](#exploratory-testing-from-a-story) |
 | Automated tests | **Turn a passed run into an automated test**: Scribe, the coding agent, writes a test for a passed browser run in the repository's own framework, page objects, and helpers; ForgeQE runs it until it passes and a reviewer approves it. | `write_test`, `run_test` | [docs](#6-tests) |
 | Automated tests | **Heal failing tests**: Runs a repository's tests and updates the ones the application has outgrown against the live app, or reports a real application bug instead. | `heal_tests` | [docs](#fixing-broken-tests) |
 | Automated tests | **Find and run existing tests**: Finds the automated tests that match a feature, tags, ids, a folder, or a sprint, confirms the selection, and runs it once the user agrees, with a verdict per test. | `find_tests`, `run_tests`, `list_sprints` | [docs](#finding-and-running-existing-tests) |
+| Automated tests | **Regression packs for a release**: Builds a regression pack from what goes into a release (an Azure DevOps sprint or work items, a commit range, pull requests, or changes read from any connected tool): the tests those changes can break, each with why, plus risk-area and smoke tests, and the changes no test covers. Runs it locally, on TestMu AI, or in a pipeline, and compares each run with the last. | `build_regression_pack`, `run_regression_pack`, `list_regression_packs` | [docs](#regression-packs-for-a-release) |
 | Automated tests | **Flaky test detection and quarantine**: Finds tests that flip between passing and failing in the repository's run history, with their likely cause, and quarantines them once the user agrees. | `flaky_tests`, `quarantine_tests` | [docs](#flaky-tests) |
 | Pull requests | **Review pull requests**: Reviews an Azure DevOps or GitHub pull request from the local clone for bugs, security, requirements, and tests, and posts the review once the user agrees. | `review_pull_request`, `post_pr_review` | [docs](#reviewing-pull-requests) |
 | Pull requests | **Pull request impact and test selection**: Works out which product areas, pages, requirements, and tests a pull request touches, picks the tests it can break, runs them, and posts the impact or results on the pull request once the user agrees. | `analyze_pr_impact`, `select_pr_tests`, `post_pr_test_results`, `comment_on_pr` | [docs](#what-a-pull-request-impacts-in-the-product) |
@@ -301,6 +302,8 @@ A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of
 
 **The same flow asked again.** When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (*✓ Existing test passed · 19s · no browser run or new test needed*). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.
 
+**When the flow cannot be reproduced.** If ForgeQE's own replay of the run could not reach the run's final page and Scribe's first test fails too, the job stops there and says which replayed step failed and why, instead of spending its remaining attempts on a flow no script can repeat as recorded.
+
 **Why a test failed.** When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's `E` lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.
 
 **Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
@@ -857,6 +860,42 @@ Review my open pull requests
 
 Through `forgeqe mcp`, `post_pr_review` posts only when called again with `confirmed: true`; the MCP client should ask you first.
 
+### Regression packs for a release
+
+A regression pack is the set of your repository's tests that a release's changes can break, saved under a name so you can run it again (here, on TestMu AI, or in a pipeline) and compare runs. Ask in your own words:
+
+```text
+Build a regression pack for sprint 12 and run it on TestMu
+Which tests should we run for the PRs merged this week?
+Build a pack for Jira fix version 2.4 and the commits since release/2.3 in ../shop-app
+```
+
+**Where the changes come from.** Any mix of:
+
+- an Azure DevOps sprint, area, epic or feature, or work item ids: their stories and bugs, the test cases each is tested by, and the files their linked pull requests changed;
+- the commits between two refs of a repository (`from` and `to`): the product's repository, or the test repository's own; work items the commit messages name (`AB#123`, `#123`, `PAY-12`) are read too;
+- pull requests by number or URL, or the ones merged in the last N days (Azure DevOps or GitHub);
+- anything ForgeQE's agent can read through its connected tools (a Jira release, GitHub issues, a Confluence release note): it reads the items and passes them in with their text and links.
+
+**Why each test is in.** A test can be in for more than one reason, and the card and `<name>.md` say each one:
+
+| Reason | Meaning |
+|---|---|
+| linked | it automates a test case that tests a changed work item (`.forgeqe/test-cases.json`, or a case id in the test's attributes) |
+| test code changed | the change is in the test repository, in code the test uses |
+| matches a change | the work item's or pull request's words name what the test does: the elements it uses, its page, its name |
+| risk area | the release touches an area with defects or failures before, and the test works there |
+| smoke | the test is tagged smoke, so it is in every pack |
+
+**Gaps.** Changes no test is in for are listed as *not covered*, each with the closest existing test and what it lacks, so you can extend that test rather than write a new one.
+
+**Running it.** ForgeQE asks before it runs anything.
+
+- *This machine*: ForgeQE builds one command for exactly the pack's tests, the way the repository already runs a test (for .NET, a `--filter` with each test, the settings an earlier job used, TRX results), runs it, and reads each result. A failed test runs once more; one that then passes is flaky.
+- *TestMu AI*: the same command, with the variable `FORGEQE_PW_WS_ENDPOINT` set to TestMu AI's Playwright endpoint for the browser and platform in `testmu.yaml` (the access key goes in the variable, never in the command, and is removed from the output). The tests must connect to it when it is set: for NUnit, override `ConnectOptionsAsync` in the test base class; for Playwright Test, set `connectOptions` from it in `playwright.config`. ForgeQE refuses a TestMu run when the tests do not read the variable, so a run never looks remote while it ran locally.
+
+Each run is kept with the pack: the card says which tests fail now that passed in the last run, and which were fixed. Packs are saved in `.forgeqe/packs/<name>.json` and `<name>.md`; run output and results in `.forgeqe/reports/packs/<name>/<time>/`.
+
 ### Flaky tests
 
 Every `run_tests` and `pr-tests` run keeps each test's result in `.forgeqe/test-history.json` in the repository (commit it to share the history, or keep it between CI runs as a cache). A test that fails runs once more in the same run, with the framework's own option (Playwright `--last-failed`, pytest `--lf`, Jest `--onlyFailures`; otherwise the whole command again when the selection is small): when it then passes it is **flaky**, not failed, and the run does not fail because of it.
@@ -1210,7 +1249,19 @@ rag:
 
 The answer can also be an object with `results`, `passages`, `documents`, or `hits`, and a passage's text can be `text`, `content`, or `page_content` (LangChain's shape). A RAG server with an MCP interface works through `mcp:` instead: ForgeQE's agent calls its tools.
 
-`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check.
+`suite.case_check` decides what happens before a run: `warn` (the default) shows the findings and runs every case, `block` leaves out the cases that cannot run as written, `off` skips the check. Ask to run a case and it runs, whatever the check found; ForgeQE's agent reports the findings with the result instead of refusing. A case whose steps have no expected results runs too: it passes when every step can be done, and the check lists it **to review**, since it checks no result the page should show.
+
+### API steps in a test case
+
+A test case can mix API and UI steps ("add three products with POST /add_to_cart/{id}, then remove the first one in the cart, then pay"). The browser agent makes each API step with its `api_request` action, from the page, in the browser's own session: the logged-in user's cookies go with the call, and the CSRF token comes from the cookie or the page's form when the site needs one. So what an API step changes (a cart, an order) is what the next UI step sees.
+
+- **The verdict** of an API step rests on the status the server answered, not on the agent's account of it.
+- **The run records it** as an `api` step in `workflow.yaml`: method, URL, body, CSRF source, and the status and final URL the server gave.
+- **ForgeQE's replay repeats it** the same way, and fails that step when the server answers differently (403 where the run got 200), so the replay and the test get past it instead of stalling on an empty cart.
+- **Scribe's handoff** lists each API step with what was sent and answered, and tells Scribe to make it through the framework's API request context that shares the browser's cookies (Playwright's `page.request`), asserting the status.
+- **Kept for reuse**: the calls are saved as an API scenario, one per test case (`out/api/case-ado-2287/`, updated by each run), which `forgeqe api` and the API tools run again without a browser. The run's card says where (*API  2 calls saved as a reusable API scenario*).
+
+URLs the case gives are opened exactly as given: when the model adds a trailing slash (`/login/`, a 404 on many sites), ForgeQE opens and records the URL from the case.
 
 When a run finds a test case does not fit the application, the agent drafts the steps as they should be, from what the run found, and offers to save them. `update_test_case` shows the current and the new steps and asks before it changes the Azure DevOps Test Case; the reason goes into the work item's history, where the old steps stay.
 

package/README.md

@@ -236,7 +236,7 @@ sources:
 
 ### Reports
 
-Every suite writes `report.html`, `junit.xml`, and `suite.json` to `.forgeqe/reports/suites/<time>/`, with each case's run folder beside them; every single run writes its own `report.html` too. The report is one HTML file with the screenshots embedded, so you can open it straight from disk, attach it to a build, or mail it; no server is needed. It has an overview (verdict, totals, step checks, tokens, cost, page memory savings, browser errors, accessibility issues, bugs filed, what changed since the last run, results over earlier runs, the cases that need attention, results by source and tag, the slowest cases, and what was posted where), a test results page (each case's step table with expected and actual results, why it failed, its last passing screenshot beside this run's, browser errors per step, page memory with healed locators, the browser agent's steps with a screenshot each and the clicked element outlined, a timing waterfall, accessibility issues, and the trace and workflow to read in place), a failures page, since last run, trends and flaky tests from the earlier suite runs in the same `.forgeqe/reports/suites/` folder, time and cost (model vs browser time, a parallel worker timeline, cost per case from the model's `price` in `llms.json`), coverage by requirement and group from the test system, accessibility checks for every page visited, and the environment (model, browser, ForgeQE version, CI build, branch, commit). It exports Markdown and JSON and prints cleanly. See `samples/reports/nightly/20260929-020005/report.html` for an example. `report.screenshots: failures` keeps screenshots for failed runs only, `off` drops them; `report.accessibility: false` skips the page checks. On GitHub Actions the suite also writes a job summary.
+Every suite writes `report.html`, `junit.xml`, and `suite.json` to `.forgeqe/reports/suites/<time>/`, with each case's run folder beside them; every single run writes its own `report.html` too. The report is one HTML file with the screenshots embedded, so you can open it straight from disk, attach it to a build, or mail it; no server is needed. It has an overview (verdict, totals, step checks, tokens, cost, page memory savings, browser errors, accessibility issues, bugs filed, what changed since the last run, results over earlier runs, the cases that need attention, results by source and tag, the slowest cases, and what was posted where), a test results page (each case's step table with expected and actual results, why it failed, its last passing screenshot beside this run's, browser errors per step, page memory with healed locators, the browser agent's steps with a screenshot each and the clicked element outlined, a timing waterfall, accessibility issues, and the trace and workflow to read in place), a failures page, since last run, trends and flaky tests from the earlier suite runs in the same `.forgeqe/reports/suites/` folder, time and cost (model vs browser time, a parallel worker timeline, cost per case from the model's `price` in `llms.json`), coverage by requirement and group from the test system, accessibility checks for every page visited, the environment (model, browser, ForgeQE version, CI build, branch, commit), and Tokenomics: every model call by every agent from the usage ledger in `~/.forgeqe/usage` (tokens, cache, cost, time, by agent, job, task, purpose, model, user and machine, never prompt text), the models compared, spend per day with a month forecast, the `budget:` limits from `forgeqe.yaml` (warnings at 80%, 90% and 100%, nothing stopped), and a calculator for what runs would cost at another size, cache rate or model. It exports Markdown and JSON and prints cleanly. See `samples/reports/nightly/20260929-020005/report.html` for an example. `report.screenshots: failures` keeps screenshots for failed runs only, `off` drops them; `report.accessibility: false` skips the page checks. On GitHub Actions the suite also writes a job summary.
 
 ### Static BI
 

package/docs/COST_ARCHITECTURE.md

@@ -0,0 +1,280 @@
+# Cost-aware execution: registry, tiers, and escalation
+
+Status: proposal · 2026-10-15
+
+ForgeQE pays the model for work it has already done. This document sets out how ForgeQE decides when the
+model is needed at all, and how much of it. The approach is built from published, measured techniques.
+The goal for known work: **a rerun of an existing test costs 0–5% of its first run**, and the model is
+paid only for what is new or broken.
+
+## 1. What a request costs today (measured)
+
+From the session logs in a live project (`.forgeqe/logs`, Azure OpenAI gpt-5.4):
+
+| Request | Agent calls | Input tokens | Output tokens | Cached |
+|---|---|---|---|---|
+| "Are we ready to release?" (one release-readiness tool call) | 12 | 577K | 2.6K | 518K |
+| Test-authoring session | many | 1.9M | 8.9K | 1.65M |
+
+These numbers show three things:
+
+1. **Fixed overhead per call.** Every chat-agent call resends about 45K tokens of system prompt, tool
+   definitions and history. Output is under 0.5% of tokens, so the cost is the context, not the work.
+2. **Calls multiply.** The deterministic release check took 12 calls, because the agent asked
+   questions, searched for files, read config and summarised.
+3. **Known work is not recognised:**
+   - Every request goes through the chat agent, including the **rerun** link.
+   - Page memory finds a saved flow only by the exact goal text plus the URL (`memory.py: goal_key`), so a
+     reworded goal misses it.
+   - `replay: verify` replays every step, then starts the browser agent to check the result.
+
+Prompt caching already works (about 90% of input is cached). It lowers the price per token, but not the
+number of tokens or calls, so on its own it cannot reach the target.
+
+## 2. The approaches this design uses
+
+| Technique | Evidence | Where it applies here |
+|---|---|---|
+| **Workflows before agents.** Use fixed code paths for predictable tasks; keep agents for open-ended ones. | Anthropic, [*Building effective agents*](https://www.anthropic.com/research/building-effective-agents) (2024) | Tier 0: rerun, run case, run suite, release check |
+| **Plan caching.** Extract a plan template from a finished run, match new requests by key, and adapt the template with a small model. | Zhang et al., [*Agentic Plan Caching*](https://arxiv.org/abs/2506.14852), NeurIPS 2025: 50.3% lower cost and 27.3% lower latency on average, performance maintained | Tier 1: known flow, new wording or new data |
+| **Workflow memory.** Turn successful trajectories into reusable routines. | Wang et al., [*Agent Workflow Memory*](https://arxiv.org/abs/2409.07429), ICML 2025: +51.1% relative success on WebArena, fewer steps | Flows saved from passing runs, shared across cases |
+| **Model cascade and routing.** Try the cheapest model or step that can answer, and escalate only when a check fails. | Chen et al., [*FrugalGPT*](https://arxiv.org/abs/2305.05176) (up to 98% savings, task-dependent); Ong et al., [*RouteLLM*](https://arxiv.org/abs/2406.18665), ICLR 2025 (over 2x savings at about 95% quality) | The router; result checks on the reviewer model |
+| **Content-addressed memoization.** Key cached work by a fingerprint of its inputs; any change to the inputs invalidates it. | Standard in build systems (Bazel, Gradle remote cache) and in test impact analysis | Registry fingerprints, the tool-result cache |
+| **Record–replay with self-healing locators.** | Already in ForgeQE (`replay.py`, ranked locators) | Tier 0 and Tier 2 execution |
+
+The published figures come from the authors' own benchmarks. ForgeQE has to measure its own (section 7).
+
+## 3. The execution ladder
+
+Every request is resolved to the **lowest tier that can do it**. A tier escalates only on a concrete
+failure signal, never on a guess.
+
+| Tier | When | Model use | Target cost vs. first run |
+|---|---|---|---|
+| **0: Deterministic** | The request resolves to a known entity, and replay and end-state checks pass | None. The reply is filled in from the result. | 0% |
+| **1: Adapt** | A known flow, but with new wording, new test data, or an end-state check that is unsure | One call to the small (reviewer) model with a short prompt (about 2K tokens) | ≤5% |
+| **2: Heal** | Replay stopped at step *k*, or a check failed | The agent starts at step *k* with the steps so far as context, not from the beginning | Proportional to what broke |
+| **3: Explore** | No match in the registry: a new goal or new kind of task | Full agent, as today | 100% |
+
+```
+request ──► Resolver (no model) ──hit──► Tier 0 run ──checks pass──► reply from template
+               │                            │ checks fail / replay stops
+               │ ambiguous                  ▼
+               ▼                         Tier 2 heal (agent from step k)
+         Router (small model, ~2K) ──known──► Tier 1 adapt
+               │ novel
+               ▼
+         Tier 3 full agent ──passes──► registry learns the flow, plan and checks
+```
+
+**Resolver (no model).** Matches IDs (`TC-123`, `SHOP-12`, plan/suite numbers, run folders,
+`last`), file and test names, and aliases from the registry. The **rerun** link and `/rerun` call it
+directly. Plain requests try it before any model is called.
+
+**Router (small model).** Runs only when the resolver is unsure. It sees the request and the top
+registry candidates (ranked by alias and keyword match), not the full tool catalogue. It returns
+`{entity, tier}` or `novel`.
+
+**Safety rules (non-negotiable):**
+- A Tier 0 pass needs evidence: replay complete **and** every recorded end-state check passing. Anything
+  else escalates.
+- **Freshness.** A flow is marked *stale* when the app changes: a new deploy or build id (when
+  configured), a changed page fingerprint seen during replay, or the age limit
+  (`registry.maxAgeDays`). A stale flow runs at Tier 1 once to re-confirm it.
+- **Canary.** Every *N*th Tier 0 run of an entity (default 10) runs at Tier 1, so a check that silently
+  passes is caught.
+- The user can always force a tier: `/rerun --verify`, `/rerun --fresh`.
+
+## 4. The registry: one indexed catalogue
+
+The registry is one SQLite file per project: `.forgeqe/registry/registry.db`.
+- Bun ships `bun:sqlite` and Python ships `sqlite3`, so there are no new dependencies.
+- It is local, and git-ignored by default (its folder carries its own `.gitignore`).
+- Existing JSON stores (page memory, code graph, quality history) stay where they are. The registry
+  **indexes and links** them, it does not replace them.
+
+### 4.1 Taxonomy
+
+Every row has a `kind`, and every kind has a single source of truth:
+
+| Kind | Source of truth | Example |
+|---|---|---|
+| `test_case` | ADO / Jira / Xray / TestRail / cases file | `ado:QI-Claims:TC-4812` |
+| `code_test` | Repo test code (code graph) | `PlaywrightPom.Web.Tests.LoginTests.ValidLogin` |
+| `flow` | Page memory `flows/` | recorded steps for a goal |
+| `page`, `element` | Page memory `pages/` | `/login`, `#username` |
+| `suite`, `plan` | ADO / cases file | `plan 12 / suite 34` |
+| `work_item` | ADO / Jira | story, bug, feature |
+| `pipeline`, `environment`, `login` | Config, ADO, saved sessions | `QI-Claims nightly`, `staging`, `saucedemo-standard` |
+| `run`, `report` | `.forgeqe/reports` | run folders |
+
+### 4.2 Tables
+
+```sql
+-- Everything ForgeQE knows about, categorised. fingerprint = hash of the inputs that define it.
+entities(id TEXT PK, kind TEXT, source TEXT, external_id TEXT, title TEXT,
+         path TEXT, fingerprint TEXT, version TEXT, updated_at TEXT)
+-- Every name a person might use for an entity: ids, titles, file and method names, normalised.
+aliases(alias TEXT, entity_id TEXT, weight REAL, PRIMARY KEY(alias, entity_id))
+-- Links between entities: case↔code_test, case↔flow, flow↔page, test↔page_object, story↔case.
+links(from_id TEXT, rel TEXT, to_id TEXT, PRIMARY KEY(from_id, rel, to_id))
+-- Reusable plans (the plan cache): what to call, with which arguments, how to judge the result.
+plans(id TEXT PK, entity_id TEXT, intent TEXT, tool TEXT, args_template TEXT,
+      checks TEXT, input_fingerprint TEXT, successes INT, failures INT,
+      last_success TEXT, stale INT DEFAULT 0)
+-- End-state checks recorded from a passing run: url, visible text, element, API status.
+checks(plan_id TEXT, kind TEXT, target TEXT, expected TEXT)
+-- The cost ledger: every request, its tier, and what it cost.
+runs(id TEXT PK, entity_id TEXT, plan_id TEXT, tier INT, outcome TEXT,
+     escalated_from INT, tokens_in INT, tokens_out INT, tokens_cached INT,
+     cost REAL, currency TEXT, ms INT, started_at TEXT)
+-- Memoised reads from external systems (ADO, Jira, GitHub), keyed by arguments and source version.
+tool_cache(tool TEXT, args_hash TEXT, source_version TEXT, result_path TEXT,
+           fetched_at TEXT, ttl_s INT, PRIMARY KEY(tool, args_hash, source_version))
+-- Pre-built context briefs per entity, rebuilt only when the fingerprint changes.
+context_packs(entity_id TEXT PK, fingerprint TEXT, brief TEXT, tokens INT, built_at TEXT)
+```
+
+### 4.3 Fingerprints and invalidation
+
+A cached item is valid only while its fingerprint matches its source:
+
+- **test_case:** hash of the normalised steps plus expected results, together with the ADO `rev`.
+  This replaces matching on goal text, so rewording a request no longer misses the flow.
+- **code_test:** git blob sha of the test file and of the page objects it reaches (from the code graph).
+- **flow:** hash of its steps, together with the page fingerprints seen on its last pass.
+- **tool_cache:** the source's own version (work item `rev`, sprint id, ETag) where one exists;
+  otherwise a short TTL. Writes always bypass it.
+
+## 5. Building context (for Tiers 1–3)
+
+The agent's fixed 45K-token context is the main cost driver. Context is built from the registry instead
+of being discovered by the agent:
+
+- **Context packs.** For each entity, a brief of a few hundred tokens: what it is, its linked case, code
+  test, flow, pages and last results. The agent gets the brief rather than spending calls on
+  `read`, `grep` and `code_outline`.
+- **Tool listing by tier:**
+  - Tier 1 gets one tool.
+  - Tier 2 gets the tools of the capability that broke.
+  - Tier 3 gets today's progressive listing (`routing.ts`).
+- **Stable prefix first, variable last**, so prompt caching keeps hitting.
+- **History compaction.** After a request finishes, only its outcome and ids are carried forward, not
+  its tool results.
+
+## 6. Learning: how the registry fills itself
+
+- **Indexing.**
+  - On `forgeqe init`, and incrementally on file change, the code graph and cases sources are indexed
+    into `entities`, `aliases` and `links`.
+  - ADO and Jira items are indexed when they are first fetched.
+- **Promotion.** A Tier 2 or Tier 3 run that passes, and passes the judge when it is on, writes or updates
+  its `plan` and `checks`. A plan needs two passes, or one pass plus user confirmation, before it is
+  allowed to run at Tier 0.
+- **Demotion.** Failures increment `failures`. Two failures in a row mark the plan *stale*, and the next
+  run goes to Tier 1 or 2.
+
+## 7. Measurement and acceptance
+
+The ledger (`runs`) makes cost measurable per tier and per entity. `forgeqe cost` reports the following
+metrics. Each must meet its target before a phase counts as done.
+
+| Metric | Target |
+|---|---|
+| Cost of a rerun of a known passing test, vs. its first run | ≤5% (Tier 0: 0 tokens) |
+| Share of reruns served at Tier 0 or Tier 1 | ≥90% |
+| Tier escalation rate on unchanged apps | ≤5% |
+| **False pass rate** (Tier 0 pass that a Tier 1 canary then fails) | 0 known; every one investigated |
+| Release check ("are we ready", scope known) | 0–1 agent calls |
+
+The evaluation is an offline replay suite under `src/evals`. It holds recorded requests (rerun, reworded
+rerun, changed app, new goal), each with an expected tier and outcome. It runs on every change to the
+resolver, router or registry, and live against the real model before release.
+
+## 8. Compatibility and rollout safety
+
+Nothing here may change what ForgeQE does today until it has been shown to give the same results.
+
+**Built alongside, never in place of**
+- `registry.mode: off | shadow | on` in `forgeqe.yaml`. It defaulted to `off` through phases 1 and 2,
+  and to `on` from 2026-10-15, after the live results in sections 9–10. With `off`, every request takes the
+  agent's path, unchanged.
+- The new code lives in `src/registry/`. Existing modules gain only small hooks at their entry points:
+  the chat request handler, the command parser and config. The chat agent, page memory, replay modes and
+  tools keep their behaviour.
+- Any registry error (missing, locked or corrupt database, an exception in the resolver) falls back to
+  today's path. The registry is a cache: deleting it loses savings, never behaviour.
+- Existing stores (page memory, code graph, quality history, run folders) keep their formats. The registry
+  reads and indexes them and never writes to them.
+- Tier 0 runs through the same `browser_run` tool the agent calls. Existing-test reuse, page memory
+  replay, the configured replay mode, reports and quality checks all apply as they do today.
+
+**Proven before it is switched on**
+- **Tests that lock in today's behaviour.** Written before the hooks, they pin down today's results:
+  a request with the registry `off` goes to the agent, and the **rerun** link goes to the agent.
+  They have to pass unchanged afterwards.
+- **Shadow mode.** The resolver decides and records what it would do (`runs` with `mode = shadow`), while
+  the request still takes today's path. The decisions are compared with real outcomes before `on`.
+- **False pass check.** During rollout, every Tier 0 pass is re-checked at Tier 1. Any disagreement blocks
+  the next phase.
+- **The full suite.** Every existing test passes unchanged in every phase, with no existing test edited
+  to make it pass.
+
+**Switched on gradually**
+- One phase per release. Each one can be turned off on its own.
+- A live check against the real model, Azure DevOps and Teams before each phase is called done.
+- Every reply says which tier answered it. `/rerun` is an explicit command, and the agent remains one
+  sentence away: anything the resolver is unsure of goes to it.
+
+## 9. Phase 1 live result (2026-10-15)
+
+Live run against Azure OpenAI gpt-5.4 and a real headless Chrome: a `/rerun` of a 27-step practice-form run
+whose first run used 242K tokens.
+
+| | Result |
+|---|---|
+| Resolver | Matched the run; no chat-agent calls |
+| Page memory | Replayed 26 of 27 steps without the model |
+| Step 27 | A read-only `evaluate` (reading the fields to check them) is not replayable, so the browser agent took over for 14 model steps |
+| Cost | **178K tokens: 74% of the first run.** The target is not met |
+
+The remaining cost is the browser agent's check after replay. Phase 2 is the step that reaches the target.
+Read-only steps (`evaluate`, extract) and the final check must be answered from recorded end-state checks
+(`run.json` already records `checks` with expected values). The agent comes in only when a check fails.
+
+## 10. Phase 2 live result (2026-10-15)
+
+What phase 2 adds, with `registry.mode: on` only:
+- `evaluate` steps whose script only reads the page are left out of a replay. The test is on the script's
+  source and conservative: any assignment, any page-changing call, any request or deferred code counts
+  as writing. Checked against every `evaluate` in a live project's flows: no write was classed as read-only.
+- A passing run's `record_check` results are kept in its flow, each with the step it was recorded after.
+- A replay finds each check on the page again after that step, with no model. If every check holds, the
+  run passes with no model. The first check that does not hold stops the replay there, and the agent
+  takes over on that page (today's path).
+- The agent still judges the run when: a check is dynamic, the flow has not passed for 30 days, or it is
+  every 10th replay of the flow (the canary).
+
+Live, Azure OpenAI gpt-5.4 and headless Chrome:
+
+| Flow | Run | Tokens | vs. first run |
+|---|---|---|---|
+| Wikipedia search (3 steps, 1 check) | first run | 50.3K | 100% |
+| | rerun: the search suggestion did not show in time, so the agent took 2 steps | 25.1K | 50% |
+| | rerun: replay complete, the check held | **0** | **0%** (6 s) |
+| | rerun with the saved check changed to text the page does not show | 25.1K | the replay refused to pass it, and the agent judged the real page |
+| Practice-form playground (27+ steps) | reruns | 122K, 135K, 82K | 50%, 56%, 34% |
+
+The practice page is built to resist replay: double-click targets, clipboard paste and per-key typing that
+replay's direct input does not reproduce. Its checks correctly never passed a replay that ended in a
+different state. Flows like that stay with the agent until replay can repeat those inputs (a later phase).
+
+## 11. Rollout
+
+| Phase | Scope | Done when |
+|---|---|---|
+| **1** | Registry (entities, aliases, plans, runs). Resolver plus Tier 0 for `/rerun`, the rerun link, and case and run ids. Fingerprint matching for test cases. Reply filled in from the result. | A rerun of a known passing case logs 0 tokens; a reworded request still finds its flow |
+| **2** | Recorded end-state checks replace the post-replay agent check. Canary and freshness rules. | `verify` default costs one small-model call or none; the false pass suite stays green |
+| **3** | Router (small model) plus Tier 1 adapt. Repo code tests run directly (`dotnet test` / `npx playwright test` with filters, results files parsed). | Reworded and new-data reruns at ≤5% |
+| **4** | Tool-result cache, context packs, history compaction. Release check as a fixed workflow. | The release check takes 0–1 agent calls; the agent's average context per call is down by half |
+| **5** | `forgeqe cost` report and the evaluation suite in CI | Metrics in section 7 reported per release |

package/docs/feature-test-prompts.md

@@ -0,0 +1,95 @@
+# ForgeQE feature test prompts
+
+One prompt for each ForgeQE feature, to check the agent end to end. Type each prompt in the ForgeQE terminal (`forgeqe start`), started in the test project's folder (for example `playwright-csharp-pom-framework`).
+
+## Before you start
+
+- Replace the `<…>` placeholders with your own ids (a story, a pull request).
+- ⚠ marks prompts that change something outside ForgeQE: Azure DevOps, Teams, git, or the repository. Run those on test data only.
+- Some prompts need an earlier run. #2, #17, #27, and #28 depend on #1 or #9, so run them after it.
+- Don't use test case 2287 for the API + UI check. As written, its step 2 gets a 403 (`POST /add_to_cart` is not how that site adds to the cart), and it signs in to a personal account. Use #26 instead.
+
+
+
+## Browser and test cases
+
+
+| #   | Feature                      | Prompt                                                                                                                                      | What to check                                                                                     |
+| --- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
+| 1   | Browser goals                | `Go to https://www.playwrightautomation.com, open the Practice tab, fill the Text Inputs with realistic values and check they are accepted` | Steps appear live; the run ends with a pass verdict and a report link                             |
+| 2   | Page memory                  | Run #1 again, then: `What does ForgeQE know about the Practice page?`                                                                       | The rerun replays from memory, faster and with fewer tokens; the answer lists locators            |
+| 3   | Saved logins                 | `Sign in to https://www.automationexercise.com/login with the test account and keep the session`                                            | The login is saved; the password never appears in the chat or the logs                            |
+| 4   | Run test cases               | `Run test case 2287 from QI-Claims and don't post results`                                                                                  | Each step has a verdict; report.html and junit.xml are written; nothing is posted to Azure DevOps |
+| 5   | Test case quality            | `Is test case 2287 in QI-Claims good enough to run? Check it against PBI 2286`                                                              | The card says "to review", not "not runnable"; the missing expected results are listed            |
+| 6   | ⚠ Write test cases           | `Write test cases for PBI 2286 in QI-Claims and show them before saving`                                                                    | Each case has expected results and links to PBI 2286; ForgeQE asks before saving                  |
+| 7   | Domain and regulatory design | `Write test cases for PBI 2286 covering PCI DSS obligations for card payments`                                                              | Cases cover card data and masking, and name the regulation                                        |
+| 8   | Exploratory testing          | `Explore PBI 2286 on automationexercise.com and find bugs beyond the test cases`                                                            | A charter, findings with evidence, and suggested bugs                                             |
+
+
+
+
+## Automated tests (Scribe)
+
+
+| #   | Feature                  | Prompt                                                    | What to check                                                                                                     |
+| --- | ------------------------ | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
+| 9   | ⚠ Turn a run into a test | `Write a Playwright test for the last passed run`         | A handoff; a C# test that uses the repository's page objects; ForgeQE runs the test; the reviewer gives a verdict |
+| 10  | ⚠ Heal failing tests     | `Heal the failing tests in tests/PlaywrightPom.Web.Tests` | Each failure is healed or reported as an application bug; no test is deleted or skipped                           |
+| 11  | Find and run tests       | `Run the Wikipedia tests in this repo`                    | The right `dotnet test --filter` command; a pass or fail for each test                                            |
+| 12  | Flaky tests              | `Which tests in this repo are flaky?`                     | Tests ranked, each with evidence (retries, mixed results)                                                         |
+
+
+
+
+## Pull requests
+
+
+| #   | Feature                   | Prompt                                                         | What to check                                                 |
+| --- | ------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------- |
+| 13  | Pull request review       | `Review PR <n> in QI-Claims and focus on test coverage`        | Findings with file and line, and a verdict                    |
+| 14  | Impact and test selection | `What does PR <n> impact, and run only the tests it can break` | The test list comes from the code graph; only those tests run |
+
+
+
+
+## API and quality
+
+
+| #   | Feature                       | Prompt                                                                        | What to check                                                          |
+| --- | ----------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
+| 15  | API and contract tests        | `Turn the last run's API calls into contract tests and run them`              | Contracts are saved and run without a browser; a JUnit file is written |
+| 16  | Accessibility and performance | `Is the automationexercise.com checkout page accessible, and how fast is it?` | WCAG findings by severity; LCP, CLS, and INP figures                   |
+| 17  | Visual comparison             | Run #1 twice, then: `What changed visually since the last run?`               | Each page is marked same or changed, with diff images                  |
+
+
+
+
+## Release, operations, and systems
+
+
+| #   | Feature                | Prompt                                                                                      | What to check                                                              |
+| --- | ---------------------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
+| 18  | Defect intelligence    | `Which defects in QI-Claims keep coming back?`                                              | Recurring defects, with counts and areas                                   |
+| 19  | Release readiness      | `Are we ready to release QI-Claims? Give requirement coverage`                              | A coverage table and a GO, GO WITH RISKS, or NO-GO verdict                 |
+| 20  | ⚠ Scheduled monitoring | `Check the automationexercise.com login every 15 minutes and alert Teams if it fails`       | The monitor is scheduled and `/monitor` lists it; the alert text is clear  |
+| 21  | ⚠ Notifications        | `Post the last suite's results to the Teams QA channel`                                     | A message arrives in Teams, with a link to the report                      |
+| 22  | Reports                | `Write up the open defects in QI-Claims as a report I can share`                            | report.html and report.md, linked from the card                            |
+| 23  | ⚠ CI pipelines         | `Add an Azure Pipelines workflow that runs tests/cases.yaml on every PR, and make it green` | A pipeline file, the secrets it needs stored, a pushed branch (never main) |
+| 24  | Azure DevOps           | `List the last 3 pipeline runs in QI-Claims and QI-Payments and which failed`               | The agent's plan shows; calls appear in Activity; failure causes are given |
+| 25  | Setup help             | `What can you do here, and what is still not set up?`                                       | The features, each with what it still needs                                |
+
+
+
+
+## Recent additions
+
+
+| #   | Feature                        | Prompt                                                                                                                                                                                                                                           | What to check                                                                                                                                                |
+| --- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| 26  | API and UI steps in one case   | Run a case with these steps: `1. Open https://automationexercise.com/products 2. Call POST https://automationexercise.com/api/searchProduct with form body search_product=top, expect 200 3. Search "top" in the UI, expect "Searched Products"` | `api_request` is used and the server answers 200; workflow.yaml has an `api` step; the card says "API … saved as a reusable API scenario"; the replay passes |
+| 27  | Reuse or extend existing tests | Run #9's flow again with one extra check added                                                                                                                                                                                                   | ForgeQE offers reuse, extend, or new, with coverage evidence; no duplicate test file appears                                                                 |
+| 28  | Reviewer exchange              | Any Scribe job that needs two or more attempts (for example #9 on a new flow)                                                                                                                                                                    | "Reviewer asked for changes" lists each issue with its fix; "Scribe tries again" shows the real error; then the verdict                                      |
+| 29  | Activities panel               | During #24, press **ctrl+o**                                                                                                                                                                                                                     | Activity takes half the width: the plan with spinners, Now and Next, each call shown as it starts; each request has a header row and a result row            |
+| 30  | Refine                         | Type `run 2287 and write automation`, then press **ctrl+r**                                                                                                                                                                                      | The panel scrolls with ↑↓ and PgUp/PgDn; open questions come first; Use, Edit, and Keep stay visible                                                         |
+
+