@capagents/forgeqe 2.2.0 → 2.2.2

package/HOW_TO_USE.md

@@ -38,6 +38,7 @@ Every agent has its own name, icon, and colour in the shell: the icon moves besi
 | a pencil, green | Scribe | `forgeqe-test` | Scribe: writes a test from a verified run | Yes, the repository |
 | an open book, leaf | Scribe, reading | `forgeqe-read` | Scribe, reading the repository before the run arrives | No |
 | a check mark, pink | Reviewer | `forgeqe-review` | Reviewer of Scribe's tests | No |
+| a route between two points, violet | Test planner | `forgeqe-test-plan` | Plans how a test recorded by hand fits the repository | No |
 | a cross, lime | Healer | `forgeqe-heal` | Scribe healing tests the application outgrew | Yes, the repository |
 | a magnifier, cyan | Test finder | `forgeqe-find` | Finds the repository tests a request means | No |
 | a comment bubble, rose | PR reviewer | `forgeqe-pr-review` | Reviews pull requests | No |
@@ -59,23 +60,23 @@ The icons as the shell draws them, each in its agent's colour:
 █▖▗█  Scribe, reading    ▄ ▗▛  Reviewer
 ▀▜▛▀                     ▝█▛
 
-▗▟▙▖  Healer             ▟█▖   Test finder
-▝▜▛▘                     ▝▀▐▖
+▚▖    Test planner       ▗▟▙▖  Healer
+ ▝▀▄                     ▝▜▛▘
 
-▛▛▛█  PR reviewer        ▗▙▟▖  API test writer
-▜▛▀▀                     ▝▜▛▘
+▟█▖   Test finder        ▛▛▛█  PR reviewer
+▝▀▐▖                     ▜▛▀▀
 
-▘▀▀▀  API planner        ███▘  Explorer
-▘▀▀                      █
+▗▙▟▖  API test writer    ▘▀▀▀  API planner
+▝▜▛▘                     ▘▀▀
 
-▚▟▙▞  Root cause         ▜██▛  Refine
-▜██▛                      ▜▛
+███▘  Explorer           ▚▟▙▞  Root cause
+█                        ▜██▛
 
-▐█▙   Judge              ▄▟▙▄  Model profiler
-▄▖▝▙                     ▙▖ ▙
+▜██▛  Refine             ▐█▙   Judge
+ ▜▛                      ▄▖▝▙
 
-▛▀▀▜  Browser agent
-▙▄▄▟
+▄▟▙▄  Model profiler     ▛▀▀▜  Browser agent
+▙▖ ▙                     ▙▄▄▟
 ```
 <!-- subagents:end -->
 
@@ -88,6 +89,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
 |------|------------|-------|-|
 | Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory`, `browser_devices` | [docs](#4-without-the-shell) |
 | Browser | **Saved logins for the application under test**: Signs the browser agent in with saved credentials (including TOTP) and reuses the session; a user can also sign in by hand once in a browser. | `logins` | [docs](#signing-in) |
+| Browser | **Record a web test by hand**: Opens a browser with the Recorder panel docked in the page while the user does the steps of a web test themself, and saves their clicks, typing, choices and keys in order as a run, with the checks, kept values, test data and notes they add; the coding agent then turns it into a test in the repository's framework. | `web_record` | [docs](#recording-a-web-test-by-hand) |
 | Test cases | **Run manual test cases in browsers**: Reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, a file, or pasted text, runs each in a real browser with a verdict per step, writes report.html and junit.xml, posts results back, and files or updates bugs for failures. | `find_test_cases`, `run_test_cases` | [docs](#7-test-cases-and-suites) |
 | Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps or Jira cases in place once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
 | Test cases | **Write test cases from a story**: Writes test cases from a story's acceptance criteria and saves them in Azure DevOps or Jira, linked to the story. Reads the cases already linked first, so asking again adds only what is missing and updates cases in place instead of duplicating them; plans end-to-end journeys from the existing cases. | `story_test_cases`, `save_test_cases`, `e2e_journey` | [docs](#where-cases-come-from) |
@@ -103,7 +105,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
 | Automated tests | **Turn a passed run into an automated test**: Scribe, the coding agent, writes a test for a passed browser run in the repository's own framework, page objects, and helpers; ForgeQE runs it until it passes and a reviewer approves it. | `write_test`, `run_test` | [docs](#6-tests) |
 | Automated tests | **Heal failing tests**: Runs a repository's tests and updates the ones the application has outgrown against the live app, or reports a real application bug instead. | `heal_tests` | [docs](#fixing-broken-tests) |
 | Automated tests | **Find and run existing tests**: Finds the automated tests that match a feature, tags, ids, a folder, or a sprint, confirms the selection, and runs it once the user agrees, with a verdict per test. | `find_tests`, `run_tests`, `list_sprints` | [docs](#finding-and-running-existing-tests) |
-| Automated tests | **Regression packs for a release**: Builds a regression pack from what goes into a release (an Azure DevOps sprint or work items, a commit range, pull requests, or changes read from any connected tool): the tests those changes can break, each with why, plus risk-area and smoke tests, and the changes no test covers. Runs it locally, on TestMu AI, or in a pipeline, and compares each run with the last. | `build_regression_pack`, `run_regression_pack`, `list_regression_packs` | [docs](#regression-packs-for-a-release) |
+| Automated tests | **Regression packs for a release**: Builds a regression pack from what goes into a release (an Azure DevOps sprint or work items, a commit range, pull requests, or changes read from any connected tool): the tests those changes can break, each with why, plus risk-area and smoke tests, and the changes no test covers. Runs it locally, on TestMu AI, or in a pipeline, compares each run with the last, and triages each failure: flaky, environment, broken test code, heal, or an AI browser re-run to tell a test the app outgrew from an app bug. | `build_regression_pack`, `run_regression_pack`, `list_regression_packs` | [docs](#regression-packs-for-a-release) |
 | Automated tests | **Flaky test detection and quarantine**: Finds tests that flip between passing and failing in the repository's run history, with their likely cause, and quarantines them once the user agrees. | `flaky_tests`, `quarantine_tests` | [docs](#flaky-tests) |
 | Pull requests | **Review pull requests**: Reviews an Azure DevOps or GitHub pull request from the local clone for bugs, security, requirements, and tests, and posts the review once the user agrees. | `review_pull_request`, `post_pr_review` | [docs](#reviewing-pull-requests) |
 | Pull requests | **Pull request impact and test selection**: Works out which product areas, pages, requirements, and tests a pull request touches, picks the tests it can break, runs them, and posts the impact or results on the pull request once the user agrees. | `analyze_pr_impact`, `select_pr_tests`, `post_pr_test_results`, `comment_on_pr` | [docs](#what-a-pull-request-impacts-in-the-product) |
@@ -1049,7 +1051,21 @@ Build a pack for Jira fix version 2.4 and the commits since release/2.3 in ../sh
 - *This machine*: ForgeQE builds one command for exactly the pack's tests, the way the repository already runs a test (for .NET, a `--filter` with each test, the settings an earlier job used, TRX results), runs it, and reads each result. A failed test runs once more; one that then passes is flaky.
 - *TestMu AI*: the same command, with the variable `FORGEQE_PW_WS_ENDPOINT` set to TestMu AI's Playwright endpoint for the browser and platform in `testmu.yaml` (the access key goes in the variable, never in the command, and is removed from the output). The tests must connect to it when it is set: for NUnit, override `ConnectOptionsAsync` in the test base class; for Playwright Test, set `connectOptions` from it in `playwright.config`. ForgeQE refuses a TestMu run when the tests do not read the variable, so a run never looks remote while it ran locally.
 
-Each run is kept with the pack: the card says which tests fail now that passed in the last run, and which were fixed. Packs are saved in `.forgeqe/packs/<name>.json` and `<name>.md`; run output and results in `.forgeqe/reports/packs/<name>/<time>/`.
+Each run is kept with the pack: the card says which tests fail now that passed in the last run, and which were fixed.
+
+**Triage: heal, re-run, or file a bug?** After each run ForgeQE sorts every failed or flaky test into one route, from evidence that costs nothing first (the retry in the same run, the failure message, the test's history), so the AI is spent only where it decides something:
+
+| Route | When | What happens |
+|---|---|---|
+| flaky | it passed when run again in the same run, or its history flip-flops | not healed: quarantine it, or fix the cause it flakes on |
+| environment | network errors, 5xx, a crashed browser, sign-in failing | run again once the environment works; no heal and no bug |
+| test code broken | a missing module, a compile error, a missing fixture | fix the test's code; healing against the page cannot fix it |
+| heal | an element the test uses is no longer found (high confidence when it passed its last runs) | the healing agent re-runs the flow in the browser and updates the steps and locators |
+| AI re-run decides | an expected value does not hold, a timeout, or something covers the element | the healing agent re-runs the flow in the browser: heals the test when the flow works, reports an app bug when it fails at the same step |
+
+Failures that name the same element or the same message are shown as **one cause** (a page object or a shared step changed), so one fix can heal them together. When most failures are the environment, ForgeQE says to run the pack again before healing anything, and `heal_tests` refuses until you name the tests.
+
+The triage is on the run's card and in `triage.md` and `triage.json` in the run's folder. To heal only the tests it sent to healing, ask "heal the failures of pack release-1.5", or run `forgeqe pack heal release-1.5` (`--tests` for some of them): ForgeQE builds a command for just those tests and the healing agent re-runs each flow in the browser before it changes anything. `forgeqe pack triage <name>` shows the last run's triage again. Packs are saved in `.forgeqe/packs/<name>.json` and `<name>.md`; run output and results in `.forgeqe/reports/packs/<name>/<time>/`.
 
 ### Flaky tests
 
@@ -1172,6 +1188,24 @@ forgeqe pr-tests 42 --run --post                     # and post the results on p
 
 Without `--agent`, the impacted files run as they are; this builds the command for Playwright and pytest projects (for other frameworks add `--agent`, or run the listed files).
 
+### Reports for tests run outside ForgeQE
+
+ForgeQE's HTML report is not only for runs it makes. `run_tests` and `pr-tests --run` write one, and so can any run in a terminal, an IDE's test runner, or CI:
+
+```bash
+forgeqe report test-results/junit.xml               # from JUnit XML or TRX: any framework, after the run
+forgeqe report setup                                # a report after every Playwright or pytest run, automatically
+```
+
+`forgeqe report` reads JUnit XML or TRX files, or the folder they are in (Playwright `--reporter=junit`, pytest `--junitxml=results.xml`, `dotnet test --logger trx`), and exits 1 when a test failed. Each report is saved in `<out_dir>/results/<time>/`, so its Trends and Flaky tests pages cover the earlier runs, whoever ran them. JUnit carries one line per failure; a skipped test is counted, not shown.
+
+`forgeqe report setup` sets up the framework the repository uses:
+
+- **Playwright**: it copies a reporter to `.forgeqe/playwright-reporter.mjs` (commit it) and prints the line to add to `playwright.config`: `reporter: [["list"], ["./.forgeqe/playwright-reporter.mjs"]]`. Every run then ends with a report, from `npx playwright test`, the VS Code Playwright extension, or CI. With `screenshot: "only-on-failure"` and `trace: "retain-on-failure"`, each failure shows its screenshot, the full error and stack, and the trace.
+- **pytest**: install `forgeqe-cli` in the environment that runs the tests (`pip install forgeqe-cli`). Its plugin loads by itself, from the terminal, VS Code's test explorer, or CI, in a folder with `forgeqe.yaml` or `.forgeqe/` (elsewhere with `pytest --forgeqe-report`). A test that passes when pytest-rerunfailures runs it again is reported as flaky.
+
+`FORGEQE_REPORT=0` turns the reporters off for a run (ForgeQE's own `run_tests` does this, as it writes its own report). The reporters call `forgeqe` on PATH, or `FORGEQE_BIN`; without it they keep the results file and print the command that builds the report. Screenshots and traces are copied into the report only from the working folder, never from a hidden folder or a file named like a key.
+
 ## 7. Test cases and suites
 
 Test cases you already have can be run two ways. In the shell or with `forgeqe run`, ask in plain words ("run SHOP-12 and SHOP-14", "run test plan 12 suite 34", "run the cases in smoke.yaml") or paste the cases, and the agent works out the source. `forgeqe suite` takes explicit references and runs the same way every time, with no agent involved, which is what pipelines call. Each case is one goal: ForgeQE tells the agent to do the steps in order and to check every expected result, and the agent's verdict decides whether the case passed. A case with no steps (plain text, a Jira story) is run from its description.
@@ -1543,6 +1577,91 @@ There is no scenario file to write: ask in plain language ("place an order in th
 Billing, in UAT") and ForgeQE's agent picks the apps from their descriptions, shows the plan, runs each step with the
 right engine, and carries values from one app to the next.
 
+### Recording a web test by hand
+
+When you would rather show the steps than describe them, record them. ForgeQE opens Chrome on the app's page with the
+**Recorder** panel docked in a corner of the page, in the same window: you click, type and choose as usual, and each
+action is listed in the panel as a numbered step, in order. Drag the panel by its title to another corner, or make it
+small (a pill that shows the step count); it comes back where you left it on every page.
+
+```bash
+forgeqe record shop --name "Apply a coupon at checkout"     # a web app from env.yaml
+forgeqe record --url https://staging.shop.test/cart         # or any page
+```
+
+In the shell, say "I'll show you the checkout, record it" (or `/record shop`). What you can add while recording:
+
+| In the panel | In the page: Ctrl+Shift+Space, then | What it adds |
+|---|---|---|
+| Check, then click an element | **A** | A check: its text, or its value for a field, or checked for a checkbox. Change the kind (text, visible, hidden, value) and mark **live data** (check the pattern, not this value) in the panel |
+| Variable, then click an element | **V** | Keeps the element's text for later steps: typing the same text later is written as `{{name}}` |
+| Wait for, then click an element | **W** | The test waits for the element to show |
+| Test data | **D** | The last value you typed becomes test data the test reads from its data |
+| `note: …` in the box | **N** | A note on the last step: "this is the cart page object" |
+| Pause / Undo / Stop, ✕ on a step | **P** / **Z** / **S** | Pause records nothing until you resume; ✕ removes that step |
+
+Type a check in words in the panel's box, or in the terminal: `the total shows "$40.00"`. Quoted text the page shows
+becomes a check on the element showing it; without quotes, one model call finds the text that shows it on the page. A
+check the page cannot show yet is kept in your words, and the coding agent writes it. In the terminal, `undo`, `pause`,
+`delete 4` and `stop` work too.
+
+What is recorded: clicks (a double click as one step), the final text of each field you typed in (not each key), choices
+in drop-downs, Enter and Escape, key chords, pages you open from the address bar, and Back. A click on a menu item that
+opened when the pointer went over its trigger gets a hover on the trigger first. Password fields are written as an
+environment variable (the saved login's `password_env`, or `<SITE>_PASSWORD`), and text equal to a saved login's value
+as its variable: what you type in them is never stored. A file you choose for an upload is recorded by its path in the
+working folder; one outside it, a hidden file (`.env`, anything under `.git`), or one named like credentials or keys is
+listed as not recorded. Drag and drop is listed as not recorded. `--signed-in` starts with the saved sessions of
+`logins:`; without it, record the sign-in too.
+
+**Data that changes from run to run.** A test that spells out this recording's price, date, or order number fails on
+the next run, so ForgeQE keeps what holds and makes the rest a pattern, a variable, or a value made fresh:
+
+| In the recording | In the test |
+|---|---|
+| A check of a price, date, time, order number, or count ("$14.15", "2,463 found") | A check of its pattern: `/^\$\d[\d,]*\.\d{2}$/` |
+| The same value checked again later (the product's price, then the cart total) | The first check keeps it as `{{price}}`; the later one checks `{{price}}` |
+| A link or button named with such data ("Combination Pliers $14.15") | Found by the part that holds: `getByRole("link", { name: /Combination Pliers/ })` |
+| A date you typed | The date the same number of days from today: `{{$date:+3:%Y-%m-%d}}` |
+| A value the app refuses twice (a new account's email) | A part new on each run: `alice+{{$unique}}@example.com` |
+
+Prices, dates, times, ids, and long numbers are found by their form; a small number on its own ("1" in the cart) stays
+exact, as it is usually what the check is about. When you stop, one model call sorts the values the forms cannot
+tell (a new account's email, "today's deal"); without a model, those stay as recorded. Each change shows in the panel
+with a tick box: untick it to keep the value as recorded.
+
+**Recording a test case.** `forgeqe record --case SHOP-12` (any source ForgeQE reads cases from: Jira, Azure DevOps,
+Xray, TestRail, a file) starts on the case's page and shows its steps as a checklist in the panel. Click the step you
+are on (or **Next case step**): what you record belongs to it, and a check you add there takes its expected result.
+Each expected result shows **✓ checked** or **no check yet**; Stop warns once about the ones with no check, and a second
+Stop stops. The test is named and tagged for the case, and mapped to it once it passes.
+
+When you stop, the run folder holds `workflow.yaml` and `trace.md` as for any run. With a repository (`--repo`, or
+`code.repo`), ForgeQE asks whether to turn it into a test. First a planner reads the repository and shows the plan:
+
+```
+Test      tests/add-to-cart.spec.ts › adds pliers to the cart
+Steps
+   2. search: HomePage.searchFor()
+   4. open the product: new HomePage.product(name)
+   6. add to the cart: new ProductPage.addToCart()
+New code
+  ProductPage in pages/ProductPage.ts
+Checks
+  after step 5: expect(product.price).toHaveText(/^\$\d+\.\d{2}$/)
+  after step 6: expect(product.cartQuantity).toHaveText("1")
+```
+
+It asks at most three questions where the repository allows two good answers (a new file, or the existing spec?), then
+**Write it**, **Change the plan** (say what to change, in your words), or **Cancel**. The coding agent replays the
+recording in Playwright, writes the test to the plan you agreed in the repository's framework, and asserts each check at
+the step where you added it. In the shell, "change the plan" comes back to ForgeQE's agent, which asks you what to
+change. `--no-plan` (or `--yes`) skips the plan. Later: `forgeqe code <run folder>`.
+
+The recorder runs in an isolated world in each page, so the page's own scripts can neither see it nor add steps, and it
+records only real input: clicks and keys a script fakes are ignored. The panel's own clicks and keys never reach the page
+and are never recorded, and it stays above the page's dialogs. Recording uses local Chrome or Chromium.
+
 ### Desktop apps (Windows)
 
 ForgeQE also automates Windows thick-client apps (Win32, WinForms, WPF). In the shell, ask for a goal in the app