@capagents/forgeqe 2.2.1 → 2.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/HOW_TO_USE.md +235 -51
- package/README.md +2 -2
- package/THIRD_PARTY_NOTICES.md +2 -1
- package/dist/forgeqe.js +1145 -951
- package/dist/template.html +36 -36
- package/docs/ARCHITECTURE.md +6 -2
- package/docs/CONTAINERS.md +80 -4
- package/docs/user-guide.html +1509 -143
- package/forgeqe.example.yaml +3 -1
- package/llms.example.json +2 -4
- package/package.json +5 -2
- package/prompts/README.md +1 -0
- package/prompts/brain.md +3 -1
- package/prompts/test-plan.md +61 -0
- package/py/forgeqe_engine/browser.py +106 -1
- package/py/forgeqe_engine/cli.py +57 -0
- package/py/forgeqe_engine/copilot_gateway.py +369 -0
- package/py/forgeqe_engine/dynamic.py +149 -0
- package/py/forgeqe_engine/recordsession.py +908 -0
- package/py/forgeqe_engine/service.py +29 -0
- package/py/forgeqe_engine/webrecord.py +589 -0
- package/py/forgeqe_engine/webrecord_page.py +530 -0
- package/reporters/playwright.mjs +83 -0
package/HOW_TO_USE.md
CHANGED
|
@@ -38,6 +38,7 @@ Every agent has its own name, icon, and colour in the shell: the icon moves besi
|
|
|
38
38
|
| a pencil, green | Scribe | `forgeqe-test` | Scribe: writes a test from a verified run | Yes, the repository |
|
|
39
39
|
| an open book, leaf | Scribe, reading | `forgeqe-read` | Scribe, reading the repository before the run arrives | No |
|
|
40
40
|
| a check mark, pink | Reviewer | `forgeqe-review` | Reviewer of Scribe's tests | No |
|
|
41
|
+
| a route between two points, violet | Test planner | `forgeqe-test-plan` | Plans how a test recorded by hand fits the repository | No |
|
|
41
42
|
| a cross, lime | Healer | `forgeqe-heal` | Scribe healing tests the application outgrew | Yes, the repository |
|
|
42
43
|
| a magnifier, cyan | Test finder | `forgeqe-find` | Finds the repository tests a request means | No |
|
|
43
44
|
| a comment bubble, rose | PR reviewer | `forgeqe-pr-review` | Reviews pull requests | No |
|
|
@@ -59,23 +60,23 @@ The icons as the shell draws them, each in its agent's colour:
|
|
|
59
60
|
█▖▗█ Scribe, reading ▄ ▗▛ Reviewer
|
|
60
61
|
▀▜▛▀ ▝█▛
|
|
61
62
|
|
|
62
|
-
▗▟▙▖ Healer
|
|
63
|
-
▝▜▛▘
|
|
63
|
+
▚▖ Test planner ▗▟▙▖ Healer
|
|
64
|
+
▝▀▄ ▝▜▛▘
|
|
64
65
|
|
|
65
|
-
▛▛▛█ PR reviewer
|
|
66
|
-
▜▛▀▀
|
|
66
|
+
▟█▖ Test finder ▛▛▛█ PR reviewer
|
|
67
|
+
▝▀▐▖ ▜▛▀▀
|
|
67
68
|
|
|
68
|
-
▘▀▀▀ API planner
|
|
69
|
-
▘▀▀
|
|
69
|
+
▗▙▟▖ API test writer ▘▀▀▀ API planner
|
|
70
|
+
▝▜▛▘ ▘▀▀
|
|
70
71
|
|
|
71
|
-
▚▟▙▞ Root cause
|
|
72
|
-
▜██▛
|
|
72
|
+
███▘ Explorer ▚▟▙▞ Root cause
|
|
73
|
+
█ ▜██▛
|
|
73
74
|
|
|
74
|
-
▐█▙ Judge
|
|
75
|
-
▄▖▝▙
|
|
75
|
+
▜██▛ Refine ▐█▙ Judge
|
|
76
|
+
▜▛ ▄▖▝▙
|
|
76
77
|
|
|
77
|
-
▛▀▀▜ Browser agent
|
|
78
|
-
▙▄▄▟
|
|
78
|
+
▄▟▙▄ Model profiler ▛▀▀▜ Browser agent
|
|
79
|
+
▙▖ ▙ ▙▄▄▟
|
|
79
80
|
```
|
|
80
81
|
<!-- subagents:end -->
|
|
81
82
|
|
|
@@ -88,6 +89,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
|
|
|
88
89
|
|------|------------|-------|-|
|
|
89
90
|
| Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory`, `browser_devices` | [docs](#4-without-the-shell) |
|
|
90
91
|
| Browser | **Saved logins for the application under test**: Signs the browser agent in with saved credentials (including TOTP) and reuses the session; a user can also sign in by hand once in a browser. | `logins` | [docs](#signing-in) |
|
|
92
|
+
| Browser | **Record a web test by hand**: Opens a browser with the Recorder panel docked in the page while the user does the steps of a web test themself, and saves their clicks, typing, choices and keys in order as a run, with the checks, kept values, test data and notes they add; the coding agent then turns it into a test in the repository's framework. | `web_record` | [docs](#recording-a-web-test-by-hand) |
|
|
91
93
|
| Test cases | **Run manual test cases in browsers**: Reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, a file, or pasted text, runs each in a real browser with a verdict per step, writes report.html and junit.xml, posts results back, and files or updates bugs for failures. | `find_test_cases`, `run_test_cases` | [docs](#7-test-cases-and-suites) |
|
|
92
94
|
| Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps or Jira cases in place once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
|
|
93
95
|
| Test cases | **Write test cases from a story**: Writes test cases from a story's acceptance criteria and saves them in Azure DevOps or Jira, linked to the story. Reads the cases already linked first, so asking again adds only what is missing and updates cases in place instead of duplicating them; plans end-to-end journeys from the existing cases. | `story_test_cases`, `save_test_cases`, `e2e_journey` | [docs](#where-cases-come-from) |
|
|
@@ -96,14 +98,14 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
|
|
|
96
98
|
| Setup | **Test environments and their apps**: Knows the test environments in env.yaml (QA, UAT, …) and each one's apps: web apps and APIs by URL, Windows desktop apps, and mobile apps, with what each is for. Every agent works from the active one; switches environment on request, picks the apps a scenario needs, and carries values such as an order number from one app to the next. | `test_apps`, `use_environment` | [docs](#environments-and-apps) |
|
|
97
99
|
| Desktop | **Automate Windows desktop apps**: Carries out a plain-language goal in a Windows thick-client app with UFO² and saves it as a workflow with checks of the outcome, or records the steps a tester does by hand, with no model; replays saved workflows with no model, healing a step whose control moved, and fixes a workflow from chat (a timeout, a wait, a dialog to dismiss) after showing the change. Apps and the secrets they may type come from env.yaml or forgeqe.yaml desktop:. | `desktop_explore`, `desktop_record`, `desktop_replay`, `desktop_edit_workflow` | [docs](#desktop-apps-windows) |
|
|
98
100
|
| Mobile | **Automate Android and iOS apps**: Carries out a plain-language goal in an Android or iOS app through Appium and saves it as a mobile workflow with checks of the outcome; replays workflows with no model on Appium (healing a moved control) or Maestro; runs the team's own Maestro flows after checking them against an allowlist; uploads an app build to TestMu AI and saves its lt:// id as the app's build. Apps come from env.yaml (kind: mobile) or forgeqe.yaml mobile:. | `mobile_explore`, `mobile_replay`, `mobile_maestro`, `mobile_upload` | [docs](#mobile-apps-android-ios) |
|
|
99
|
-
| Operations | **Services and
|
|
101
|
+
| Operations | **Services, mocks, workspace and browser in Docker or Podman**: Starts the services a test run needs (a database, a queue, the API under test) in Docker or Podman, as forgeqe.yaml declares them, and waits until each is ready; stops them, and shows their status and logs. Starts mocks of an API on WireMock, from the calls a browser run recorded or from an OpenAPI description, and can point an API app of the environment at the mock for the session. With containers.workspace, the coding agents' shell commands (build, restore, run) and every test ForgeQE runs go to a container of the project's image; with browser.provider docker, the browser agent, the replay and the coding agent's live browser use Chromium in a container. | `services`, `mock_service` | [docs](#containers-and-mock-services) |
|
|
100
102
|
| Data | **Ask a database in plain language**: Connects to the project's databases (PostgreSQL, MySQL and MariaDB, SQL Server and Azure SQL, Oracle, Snowflake, Redshift, CockroachDB, SQLite, DuckDB, and MongoDB) from a connection string or host and user, signing in with a password, Microsoft Entra ID or AWS IAM. Reads the schema through DB-GPT's connectors and turns questions into SQL that is checked before it runs: reads in a read-only session that is rolled back, writes only on a database marked writable, after you confirm them. | `databases`, `database_schema`, `run_sql` | [docs](#databases) |
|
|
101
103
|
| Data | **Test data on demand**: Makes realistic test data from a spec that fits the database's schema (types, sizes, NOT NULL, unique keys, CHECK lists, enums), parents before children, the same rows for the same seed. Or takes a masked subset of a real database: the rows you pick and every row they need, with names, emails, phones, addresses and other personal columns replaced the same way every time. Shows a preview, writes CSV, JSON or SQL files for automation, or inserts the rows into a database marked writable after you confirm, as a batch that can be removed again. Specs live in the project, so tests and pipelines make the same data with forgeqe data gen. | `generate_test_data`, `test_data_batches` | [docs](#test-data) |
|
|
102
104
|
| Data | **Data flows between systems**: Checks the data that passes between your systems' databases (the digital channel, CRM, billing, a warehouse), by batch ETL or in real time: each hop's rows arrived, complete, with every mapped column right after its rule, on time for its SLA, without duplicates. Follows one record end to end, says what a change to a table or column reaches downstream, and writes the flows from your mapping documents wherever they are, or suggests them from the schemas. | `data_flows`, `check_flows`, `trace_record`, `read_mapping_file` | [docs](#data-flows) |
|
|
103
105
|
| Automated tests | **Turn a passed run into an automated test**: Scribe, the coding agent, writes a test for a passed browser run in the repository's own framework, page objects, and helpers; ForgeQE runs it until it passes and a reviewer approves it. | `write_test`, `run_test` | [docs](#6-tests) |
|
|
104
106
|
| Automated tests | **Heal failing tests**: Runs a repository's tests and updates the ones the application has outgrown against the live app, or reports a real application bug instead. | `heal_tests` | [docs](#fixing-broken-tests) |
|
|
105
107
|
| Automated tests | **Find and run existing tests**: Finds the automated tests that match a feature, tags, ids, a folder, or a sprint, confirms the selection, and runs it once the user agrees, with a verdict per test. | `find_tests`, `run_tests`, `list_sprints` | [docs](#finding-and-running-existing-tests) |
|
|
106
|
-
| Automated tests | **Regression packs for a release**: Builds a regression pack from what goes into a release (an Azure DevOps sprint or work items, a commit range, pull requests, or changes read from any connected tool): the tests those changes can break, each with why, plus risk-area and smoke tests, and the changes no test covers. Runs it locally, on TestMu AI, or in a pipeline,
|
|
108
|
+
| Automated tests | **Regression packs for a release**: Builds a regression pack from what goes into a release (an Azure DevOps sprint or work items, a commit range, pull requests, or changes read from any connected tool): the tests those changes can break, each with why, plus risk-area and smoke tests, and the changes no test covers. Runs it locally, on TestMu AI, or in a pipeline, compares each run with the last, and triages each failure: flaky, environment, broken test code, heal, or an AI browser re-run to tell a test the app outgrew from an app bug. | `build_regression_pack`, `run_regression_pack`, `list_regression_packs` | [docs](#regression-packs-for-a-release) |
|
|
107
109
|
| Automated tests | **Flaky test detection and quarantine**: Finds tests that flip between passing and failing in the repository's run history, with their likely cause, and quarantines them once the user agrees. | `flaky_tests`, `quarantine_tests` | [docs](#flaky-tests) |
|
|
108
110
|
| Pull requests | **Review pull requests**: Reviews an Azure DevOps or GitHub pull request from the local clone for bugs, security, requirements, and tests, and posts the review once the user agrees. | `review_pull_request`, `post_pr_review` | [docs](#reviewing-pull-requests) |
|
|
109
111
|
| Pull requests | **Pull request impact and test selection**: Works out which product areas, pages, requirements, and tests a pull request touches, picks the tests it can break, runs them, and posts the impact or results on the pull request once the user agrees. | `analyze_pr_impact`, `select_pr_tests`, `post_pr_test_results`, `comment_on_pr` | [docs](#what-a-pull-request-impacts-in-the-product) |
|
|
@@ -118,7 +120,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
|
|
|
118
120
|
| Operations | **CI pipelines**: Writes a GitHub Actions or Azure Pipelines workflow that runs ForgeQE, stores the secrets it needs, pushes a branch, and makes the pipeline green. | `push_branch`, `set_ci_secret` | [docs](#8-pipelines) |
|
|
119
121
|
| Systems | **Work with Azure DevOps, Jira, GitHub, and Confluence**: Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, puts run evidence (screenshots, reports) on work items and issues as attachments or comments, and builds Azure DevOps dashboards and wiki pages. | `read_confluence`, `create_ado_dashboard`, `write_ado_wiki_page`, `attach_to_work_item` | [docs](README.md#azure-devops-jira-and-other-systems) |
|
|
120
122
|
| Systems | **Read any data in Azure DevOps, Jira, Xray, and Confluence**: Reads through a system's REST API what its MCP tools do not cover: pull requests and repositories, test plans and suites, classic releases, and anything else in Azure DevOps, Jira, Xray, or Confluence. Given on every turn once a system is set up, so the agent never turns to the browser or the shell to read it. | `query_api` | [docs](README.md#azure-devops-jira-and-other-systems) |
|
|
121
|
-
| Setup | **What ForgeQE wrote**: Says how much of the code ForgeQE wrote is still in the repository, and gives the trailer and footer that mark a commit message or pull request description covering code it wrote. Works with any repository, whatever system the team uses. | `generated_attribution` | [docs](#
|
|
123
|
+
| Setup | **What ForgeQE wrote**: Says how much of the code ForgeQE wrote is still in the repository, and gives the trailer and footer that mark a commit message or pull request description covering code it wrote. Works with any repository, whatever system the team uses. | `generated_attribution` | [docs](#generated-code) |
|
|
122
124
|
| Setup | **Every forgeqe command from the shell**: Runs the forgeqe commands no other tool covers when the user asks in words: the context budget, guardrails, prompts (show, diff, check, reset), the domain eval, skills on or off, session logs, saved API sets and packs, pack pipelines, the BI site, a test notification, desktop and mobile doctor and validate, and updating ForgeQE. Reading commands run at once; the others are shown and asked first. | `forgeqe_command` | [docs](#15-commands-cheat-sheet) |
|
|
123
125
|
| Setup | **Setup, skills, and documentation**: Shows the model profiles and MCP servers, installs or repairs the browser engine, creates a forgeqe.yaml, manages skills, and answers any question about ForgeQE from its documentation, this list, and its own code (the prompts, skills, and source it runs). | `forgeqe_setup`, `capabilities`, `playbook`, `use_capabilities` | [docs](#2-scaffold-config) |
|
|
124
126
|
|
|
@@ -165,11 +167,10 @@ forgeqe models # check the profiles
|
|
|
165
167
|
forgeqe start # the shell
|
|
166
168
|
```
|
|
167
169
|
|
|
168
|
-
With GitHub Copilot licences instead of a model key, sign in rather than exporting one, and set `llm.active: copilot` in `forgeqe.yaml` (or `"default": "copilot"` in `llms.json`):
|
|
170
|
+
With GitHub Copilot licences instead of a model key, sign in rather than exporting one, and set `llm.active: copilot` in `forgeqe.yaml` (or `"default": "copilot"` in `llms.json`). The browser engine then runs on your Copilot seat; ForgeQE's agent and Scribe need another profile for now:
|
|
169
171
|
|
|
170
172
|
```bash
|
|
171
|
-
forgeqe auth login copilot #
|
|
172
|
-
forgeqe auth login github # the browser engine, through GitHub Models
|
|
173
|
+
forgeqe auth login copilot # a fine-grained token with Copilot Requests, or the GitHub CLI's sign-in
|
|
173
174
|
forgeqe models # the copilot profile is active
|
|
174
175
|
```
|
|
175
176
|
|
|
@@ -274,60 +275,63 @@ Give that identity the **Cognitive Services OpenAI User** role on the resource.
|
|
|
274
275
|
|
|
275
276
|
### GitHub Copilot seats as the model
|
|
276
277
|
|
|
277
|
-
With GitHub Copilot licences (Pro, Pro+, Business or Enterprise), a `github_copilot` profile runs
|
|
278
|
+
With GitHub Copilot licences (Pro, Pro+, Business or Enterprise), a `github_copilot` profile runs the browser engine and the desktop and mobile agents on your Copilot seat, through [GitHub's Copilot SDK](https://docs.github.com/en/copilot/how-tos/copilot-sdk). Copilot gives other tools no model API of its own, so ForgeQE starts a small gateway on `127.0.0.1` that runs the SDK and speaks the OpenAI chat API to the engine. It signs in with a GitHub token, so no other app is authorized on your GitHub account.
|
|
278
279
|
|
|
279
280
|
```json
|
|
280
281
|
"copilot": {
|
|
281
282
|
"provider": "github_copilot",
|
|
282
|
-
"model": "
|
|
283
|
-
"engine_model": "openai/gpt-4.1",
|
|
284
|
-
"github_org": "acme"
|
|
283
|
+
"model": "gpt-4.1"
|
|
285
284
|
}
|
|
286
285
|
```
|
|
287
286
|
|
|
288
287
|
| Field | What it is |
|
|
289
288
|
|---|---|
|
|
290
|
-
| `model` | The Copilot model
|
|
291
|
-
| `engine_model` |
|
|
292
|
-
| `
|
|
293
|
-
| `api_key_env` | Optional. The variable holding the GitHub Models token; default `GITHUB_TOKEN`, else the GitHub CLI's sign-in. |
|
|
289
|
+
| `model` | The Copilot model, as the Copilot SDK names it (`gpt-4.1`, `claude-sonnet-4.6`). It must take screenshots. The sign-in lists your seat's models and which of them take images. |
|
|
290
|
+
| `engine_model` | Optional. Another Copilot model for the browser engine and the desktop agent. |
|
|
291
|
+
| `api_key_env` | Optional. The variable holding the GitHub token; without it ForgeQE reads `COPILOT_GITHUB_TOKEN`, then the token `forgeqe auth login copilot` saved, then the GitHub CLI's sign-in. |
|
|
294
292
|
|
|
295
293
|
Sign in once per machine:
|
|
296
294
|
|
|
297
295
|
```bash
|
|
298
|
-
forgeqe auth login copilot
|
|
299
|
-
forgeqe auth login github # the GitHub CLI, whose sign-in the browser engine uses for GitHub Models
|
|
296
|
+
forgeqe auth login copilot
|
|
300
297
|
```
|
|
301
298
|
|
|
302
|
-
|
|
299
|
+
It offers two ways:
|
|
303
300
|
|
|
304
|
-
|
|
301
|
+
- **A fine-grained token** (recommended). ForgeQE opens GitHub's token page: set **Resource owner** to your personal account and add the **Copilot Requests** permission, then paste the token (`github_pat_...`). The token goes to your system's credential store like ForgeQE's other secrets. Classic tokens (`ghp_`) do not work with Copilot.
|
|
302
|
+
- **The GitHub CLI's sign-in**, when `gh` is signed in.
|
|
303
|
+
|
|
304
|
+
Either way ForgeQE checks the token by asking Copilot for your seat's models and lists them; the first time, the Copilot SDK downloads Copilot's runtime from GitHub (checked against GitHub's checksums). `forgeqe auth` shows the sign-in on its `Copilot` line; `forgeqe auth logout copilot` forgets a saved token.
|
|
305
|
+
|
|
306
|
+
The token reaches only the gateway's process. The engine gets the gateway's address and a one-time key that works nowhere else; the gateway refuses requests without that key and any request a web page sends. Model-run shell commands never see the token or the key.
|
|
305
307
|
|
|
306
308
|
Which part uses what:
|
|
307
309
|
|
|
308
310
|
| Part | Model | Signs in with |
|
|
309
311
|
|---|---|---|
|
|
310
|
-
|
|
|
311
|
-
|
|
|
312
|
-
|
|
|
313
|
-
| The desktop agent | `engine_model`, on GitHub Models | `GITHUB_TOKEN`, or `forgeqe auth login github` |
|
|
312
|
+
| The browser engine (every browser run: `run --direct`, suites, monitoring) | `engine_model` or `model`, on the Copilot seat | `forgeqe auth login copilot` |
|
|
313
|
+
| The desktop and mobile agents | `engine_model` or `model`, on the Copilot seat | `forgeqe auth login copilot` |
|
|
314
|
+
| ForgeQE's agent and Scribe | Not on Copilot yet: give them another profile with `--profile`, or `code.model` | |
|
|
314
315
|
|
|
315
|
-
|
|
316
|
+
Copilot runs one SDK session per model request, with no tools of its own and your request's system prompt in place of Copilot's. Each request is billed to the seat (GitHub AI credits or premium requests, by your plan), and GitHub may rate-limit frequent automated requests. If a model is refused, check that your organization's Copilot policies allow it. ForgeQE shows no cost for a `github_copilot` profile unless you give it a `price`.
|
|
316
317
|
|
|
317
|
-
|
|
318
|
+
**In pipelines.** `forgeqe suite ... --no-code` uses only the browser engine. Set `COPILOT_GITHUB_TOKEN` to a fine-grained token with the Copilot Requests permission, stored as a pipeline secret. GitHub also documents the workflow's own `GITHUB_TOKEN` with `permissions: copilot-requests: write` for organization repositories whose Copilot policy allows it ([server tokens](https://docs.github.com/en/copilot/how-tos/copilot-sdk/auth/server-to-server-tokens)). A Copilot seat belongs to one person: prefer an organization-billed token over signing a shared pipeline in to someone's seat.
|
|
318
319
|
|
|
319
|
-
|
|
320
|
+
GitHub Models, which ForgeQE used for the engine before, was [retired by GitHub](https://github.blog/changelog/2026-07-30-github-models-is-now-retired/) on July 30, 2026. A profile that still has `engine_model: "openai/..."` or `github_org` needs updating: drop `github_org`, and use a Copilot model id.
|
|
320
321
|
|
|
321
322
|
**When something goes wrong.**
|
|
322
323
|
|
|
323
324
|
| ForgeQE says | What to do |
|
|
324
325
|
|---|---|
|
|
325
|
-
| `
|
|
326
|
-
| `
|
|
327
|
-
|
|
|
328
|
-
|
|
|
329
|
-
|
|
|
330
|
-
| The
|
|
326
|
+
| `GitHub Copilot needs a GitHub token, and there is none` | Run `forgeqe auth login copilot` (or `/auth copilot` in the shell), or set `COPILOT_GITHUB_TOKEN`. |
|
|
327
|
+
| `GitHub Copilot does not take classic tokens (ghp_)` | Make a fine-grained token with the Copilot Requests permission, owned by your personal account. |
|
|
328
|
+
| `GitHub Copilot did not accept the token` | The token has no Copilot Requests permission, has expired, or the account has no Copilot seat. |
|
|
329
|
+
| `… must be a Copilot model id`, `engine_model must be a Copilot model id` | Fix that field in `llms.json`: the sign-in lists the seat's models. A GitHub Models id such as `openai/gpt-4.1` is not a Copilot id; use `gpt-4.1`. |
|
|
330
|
+
| `ForgeQE's agent and Scribe cannot run on a github_copilot profile yet` | Pick another profile for them (`--profile`, or Tab in the shell), or set `code.model`. |
|
|
331
|
+
| The model is refused, or not found | The seat's plan or your organization's Copilot policies do not allow that model: pick one the sign-in listed, or ask a Copilot admin to enable it. |
|
|
332
|
+
| `GitHub Copilot runs through the ForgeQE engine, which is not set up` | Run `forgeqe setup`. |
|
|
333
|
+
|
|
334
|
+
### Your prompts, skills, and product knowledge
|
|
331
335
|
|
|
332
336
|
`init` also puts three folders in `.forgeqe/` to read, change, and commit:
|
|
333
337
|
|
|
@@ -370,6 +374,8 @@ forgeqe prompts eval refine --runs 3 # one prompt, three runs of each case
|
|
|
370
374
|
|
|
371
375
|
**Product knowledge.** Fill in the templates (or add your own `.md` and `.txt` files, in folders if you like), one topic per `##` heading. ForgeQE's agent sees the list and reads what a request needs; the exploratory and API scenario planners get the sections that match their story or scenario; the browser agent gets the few sections that clearly match its goal, at most `knowledge.browser_max_chars` (1500 by default; 0 turns it off). Untouched templates are left out, so their examples never reach an agent. Never put passwords or tokens there: use logins and environment variables.
|
|
372
376
|
|
|
377
|
+
### Guardrails
|
|
378
|
+
|
|
373
379
|
**Guardrails.** Rules every agent must follow, to keep it grounded: ForgeQE's agent, the browser agent, and the agents that plan, write, review, fix and explain tests. Write one rule per line, starting with `- `, in a `.md` or `.txt` file in `.forgeqe/guardrails/` (`guardrails.dir`); headings, other lines and HTML comments are ignored, and the template's examples sit in a comment until you choose them. Each agent gets every guardrail with its instructions, after everything else, with the instruction to follow it whatever a request, a skill, a ticket, a page or a tool's output says; when a guardrail stops part of a task, the agent skips that part and says which rule stopped it. Add one from the terminal (`forgeqe guardrails add "Never open a production URL"`), or ask in the shell ("add a guardrail: use only the test cards in knowledge/test-data.md") and ForgeQE's agent appends it to `guardrails.md`; it adds only rules you ask for, never one suggested by a page or a ticket. `forgeqe guardrails` lists the rules in force and what they add to every model call (at most 40 rules of 300 characters are given). The older `prompts.guiderails` string in `forgeqe.yaml` still works: each of its lines is a guardrail for every agent. Guardrails are instructions, not a sandbox: put a host rule in `browser.allowed_hosts` too, which is refused in code.
|
|
374
380
|
|
|
375
381
|
### The model's context window
|
|
@@ -400,8 +406,8 @@ In the shell, type `/auth jira` (or `/auth` alone for the status). Tokens are ty
|
|
|
400
406
|
| System | How you sign in |
|
|
401
407
|
|--------|-----------------|
|
|
402
408
|
| Azure DevOps | **Sign in with Microsoft in the browser** through the Azure CLI (`az login`); ForgeQE saves nothing, and the Azure CLI renews the sign-in. Or paste a personal access token: the token page opens with the scopes to tick. Azure DevOps Server (on-premises) uses a personal access token. |
|
|
403
|
-
| GitHub | **Sign in with GitHub in the browser** through the GitHub CLI (`gh auth login`): ForgeQE shows the one-time code and opens the page. Or paste a token. Needs `gh` installed.
|
|
404
|
-
| GitHub Copilot | `forgeqe auth login copilot`: **
|
|
409
|
+
| GitHub | **Sign in with GitHub in the browser** through the GitHub CLI (`gh auth login`): ForgeQE shows the one-time code and opens the page. Or paste a token. Needs `gh` installed. `forgeqe auth login copilot` can use this sign-in for a `github_copilot` profile. |
|
|
410
|
+
| GitHub Copilot | `forgeqe auth login copilot`: **Paste a fine-grained token** with the Copilot Requests permission (ForgeQE opens GitHub's token page), or use the GitHub CLI's sign-in. ForgeQE checks it with Copilot and lists the models your seat has. No other app is authorized on your GitHub account. For a `github_copilot` profile; see [GitHub Copilot seats as the model](#github-copilot-seats-as-the-model). |
|
|
405
411
|
| Jira, Confluence | Your Atlassian API tokens page opens (Cloud) or your personal access tokens page (Server/Data Center); create one and paste it. Confluence on the same Cloud site as Jira reuses the Jira sign-in. |
|
|
406
412
|
| TestRail, Zephyr Scale, qTest, Xray Cloud | The page with your API key opens, or ForgeQE says where to find it; paste it. |
|
|
407
413
|
| Anything else | `forgeqe auth login OPENAI_API_KEY` (any variable name) saves that one secret: a model key, a Teams webhook URL, an SMTP or saved-login password. |
|
|
@@ -410,6 +416,8 @@ Atlassian's OAuth sign-in needs a client secret, which a command-line tool canno
|
|
|
410
416
|
|
|
411
417
|
Each token is checked against the system before it is saved, and the system is switched on: a saved Jira sign-in sets the site URL too, so `sources.jira` is not needed in `forgeqe.yaml`. Saved values go in under the variable names the config already reads (`JIRA_API_TOKEN`, `ADO_PAT`, ...). A variable exported in the terminal is used before a saved sign-in, and a saved sign-in before a `.env` file, so pipelines keep using their secrets. The shell commands the agents run never see the saved secrets.
|
|
412
418
|
|
|
419
|
+
### Model prices and usage
|
|
420
|
+
|
|
413
421
|
**Prices.** ForgeQE prices every model call at the model's **list price**, from two public price lists: [models.dev](https://models.dev) (the list the agent runtime uses) and LiteLLM's model price list, which also has Azure's Data Zone prices. It keeps the few fields it needs in `~/.forgeqe/prices.json`, waits a few seconds for it the first time, refreshes it once a day in the background, and uses the last copy offline; only the public lists are downloaded, nothing about your runs is sent (`FORGEQE_PRICES=off` stops the downloads). Costs at list price say so: `$1.73 (list price of gpt-5.4 (Azure Global) from models.dev, 2026-10-06)`. For an Azure Data Zone deployment, which costs about 10% more, set `"deployment_type": "data_zone"` on the profile.
|
|
414
422
|
|
|
415
423
|
**Model parameters.** Models differ in what a request may carry: reasoning models (o1, o3, o4, gpt-5) take `max_completion_tokens` instead of `max_tokens` and only the default temperature, and some OpenAI-compatible servers refuse `max_completion_tokens`, `reasoning_effort` or `response_format`. ForgeQE works this out by itself. A profile is a reasoning model when its `model` or `deployment` name says so (an Azure deployment called `chat-prod` with `"model": "gpt-5.4"` counts), or when it sets `"reasoning": true`; `"reasoning": false` says it is not. When the API still refuses a parameter, the browser, desktop and mobile agents send the call again without it and leave it out of later calls to that model. Nothing needs setting for a new model.
|
|
@@ -422,6 +430,32 @@ A profile's own `"price": { "input": 2.5, "output": 15, "cached_input": 0.25 }`
|
|
|
422
430
|
This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
|
|
423
431
|
```
|
|
424
432
|
|
|
433
|
+
### Spending budgets
|
|
434
|
+
|
|
435
|
+
A budget warns you before model spend gets out of hand. Set it under `budget:` in `forgeqe.yaml`, in the currency of your prices (see [Model prices and usage](#model-prices-and-usage)):
|
|
436
|
+
|
|
437
|
+
```yaml
|
|
438
|
+
budget:
|
|
439
|
+
run: 2 # one browser run
|
|
440
|
+
day: 25
|
|
441
|
+
month: 400
|
|
442
|
+
# currency: USD # default: the currency of the model's price
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
Set any of the three. A budget **only warns; it never stops a run**. Each budget warns once at 80%, once at 90%, and once at 100%. When spend jumps past more than one of these at once, you get one warning, for the highest:
|
|
446
|
+
|
|
447
|
+
```
|
|
448
|
+
Budget: this month's model spend is at $245.10, 82% of the $300.00 month budget (budget.month in forgeqe.yaml). Nothing is stopped.
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
- **Where you see it.** In the shell, the warning is a line in the conversation. Every other `forgeqe` command writes it to the terminal's error output.
|
|
452
|
+
- **What counts.** `run` is the spend of one browser run. `day` and `month` add up every ForgeQE process on this machine (days and months are in UTC). Runs that the agent starts through the MCP server count too, because every process writes its model calls to the usage ledger in `~/.forgeqe/usage/` (`FORGEQE_USAGE_DIR` moves it). The ledger holds counts, times, and names, never prompt or response text. With `FORGEQE_USAGE=off`, nothing is kept on disk, and a budget counts only the calls of the process that warns.
|
|
453
|
+
- **What does not count.** A call with no price, such as a `github_copilot` profile without a `price`, and a call priced in a different currency than the budget.
|
|
454
|
+
- **Spend from before.** If you start ForgeQE when the day or month is already past 80%, it does not repeat that warning. It warns only for a threshold that spend passes from then on.
|
|
455
|
+
- **In reports.** The **Trends and forecast** page of a report's Tokenomics shows spend on this machine and the next days at the recent pace, against the budget.
|
|
456
|
+
|
|
457
|
+
### How Scribe writes a test
|
|
458
|
+
|
|
425
459
|
**Reuse before rebuild, extend before duplicate.** Before Scribe writes anything, `write_test` measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as `Expect(practicePage.AjaxStatus)`). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:
|
|
426
460
|
|
|
427
461
|
| What it finds | What happens |
|
|
@@ -441,8 +475,12 @@ A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of
|
|
|
441
475
|
|
|
442
476
|
**Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
|
|
443
477
|
|
|
478
|
+
### Generated code
|
|
479
|
+
|
|
444
480
|
**Generated code.** What ForgeQE writes is marked, so the team can see it and measure it. A file ForgeQE created starts with one line in that language's comment, such as `// Generated with ForgeQE · fq:k3x7m2qa4bdi · run 20261019-162847`; the `fq:` id is a hash of the file's contents without that line, so it shows whether the file was changed since (`code.header: false` leaves the line out; a JSON file never gets one). Files it only edited get no header. Every file it created or edited is recorded per run in `.forgeqe/generated.jsonl` (commit it with the code): the lines it added, as short hashes, never their text. `forgeqe generated` (or `/generated` in the shell, `--json` for a pipeline) says how many lines ForgeQE wrote, how many are still in the code, its share of the files it touched, and which created files are as it wrote them. Commit messages ForgeQE's agent writes end with a `Generated-by: ForgeQE (fq:…)` trailer, and its pull request descriptions with a line naming the runs' `fq:` ids and how much they wrote. The ids are plain hashes anyone can recompute: they show what ForgeQE wrote, not who owns the code.
|
|
445
481
|
|
|
482
|
+
### Choosing the model and the browser
|
|
483
|
+
|
|
446
484
|
`forgeqe.yaml` selects the active profile:
|
|
447
485
|
|
|
448
486
|
```yaml
|
|
@@ -460,6 +498,8 @@ browser:
|
|
|
460
498
|
allowed_hosts: []
|
|
461
499
|
judge: false # true: a second model call checks each finished run and can fail it
|
|
462
500
|
overlay: true
|
|
501
|
+
window: maximized # visible browser: maximized | viewport (a window of viewport's size)
|
|
502
|
+
video: true # video.mp4 of each run, shown in its report
|
|
463
503
|
# start_url: https://staging.example.com
|
|
464
504
|
```
|
|
465
505
|
|
|
@@ -469,8 +509,12 @@ Each step waits for the tab the agent is on to finish loading (up to 8 seconds,
|
|
|
469
509
|
|
|
470
510
|
With a visible browser (`headless: false` or `--headed`), ForgeQE opens on its own start page, and every page it drives gets a glowing edge and a bar along the bottom showing, in plain words, what the step is for and what the agent is doing ("Step 19 · Check whether the results can be sorted by Fastest · Looking through the page's buttons"). Your mouse and keyboard are locked while the agent thinks and unlocked while it clicks and types, so a stray click can't change the test. None of this appears in the screenshots the model sees or in the report. `overlay: false` turns it off. `forgeqe login <name>` shows the same bar without the lock, since you are the one signing in.
|
|
471
511
|
|
|
512
|
+
A visible browser opens maximized. `window: viewport` opens it at `viewport`'s width and height instead; a headless page is always that size (1280×800 by default). Each run is recorded to `video.mp4` in its run folder, and the report plays it on the run's page; `video: false` turns recording off. The video needs packages that `forgeqe setup` adds to the engine (an engine set up before this version gets them on its next run).
|
|
513
|
+
|
|
472
514
|
`start_url` is optional. Leave it out and the agent picks the site from the goal; ForgeQE never guesses a URL from the goal text. `allowed_hosts` empty allows every host. Set `["example.com"]` to keep the browser on that domain and its subdomains. `prompts.step_instruction` is appended to the engine's system prompt; the lines of `prompts.guiderails` are guardrails for every agent (see Guardrails above).
|
|
473
515
|
|
|
516
|
+
### Model routing
|
|
517
|
+
|
|
474
518
|
`llm.routing` is `off` by default for existing projects (`forgeqe init` writes `auto`); `shadow` logs each pick in the session log but changes nothing. `--profile` and `code.model` always win, and `llm.routes` limits or pins the profiles an area may use. ForgeQE checks and rates the models once and keeps the result in `.forgeqe/model-profile.json` (you can edit the ratings). Details: [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md).
|
|
475
519
|
|
|
476
520
|
List profiles, and the model each area gets:
|
|
@@ -542,7 +586,7 @@ The slash commands control the session and show a few things directly (`/help` l
|
|
|
542
586
|
|
|
543
587
|
`start_url` in `forgeqe.yaml` or `--url` pins a start page for browser runs, such as a staging app. Passed runs become tests in the current folder; `forgeqe start --repo ../app` puts them in that repo instead, and `--no-code` turns tests off.
|
|
544
588
|
|
|
545
|
-
ForgeQE's agent runs on the agent runtime with the active profile's model (or `code.model`), or, with `llm.routing: auto`, the model the router picks for the `agent` area when the session starts (`--profile` and `code.model` still win); a `github_copilot` profile
|
|
589
|
+
ForgeQE's agent runs on the agent runtime with the active profile's model (or `code.model`), or, with `llm.routing: auto`, the model the router picks for the `agent` area when the session starts (`--profile` and `code.model` still win); it cannot run on a `github_copilot` profile yet. ForgeQE downloads the agent runtime into `~/.forgeqe/tools` when it is not installed. When the agent runtime cannot be downloaded, ForgeQE says the agent is off and runs each message as one browser goal.
|
|
546
590
|
|
|
547
591
|
While a run is going, the bar under the prompt shows a spinner, the tokens used, and the elapsed time, and the transcript ends with what the agent is doing now (`Thinking about step 3…`, `Acting in the browser (step 3)…`). Each step starts with `+ Thought: 4.2s`, the time the model took to decide. When the run ends, a summary card closes the transcript:
|
|
548
592
|
|
|
@@ -888,7 +932,7 @@ After ForgeQE's reply, the request ends with the files it produced:
|
|
|
888
932
|
|
|
889
933
|
When no test was written, the Test row says why: the run did not pass, the session has `--no-code`, or the run was not handed to the coding agent.
|
|
890
934
|
|
|
891
|
-
Scribe uses the active `llms.json` profile, or, with `llm.routing: auto`, the profile the router picks for the `coding` area (`--profile` and `code.model` still win; see [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md)). The key stays in your environment: the agent runtime's config refers to it as `{env:WP_AZURE_API_KEY}` and never contains the value. An `auth: entra` profile reaches the model through ForgeQE's local token gateway (see [Signing in to the model without a key](#signing-in-to-the-model-without-a-key-microsoft-entra-id)). An Azure profile goes through the agent runtime's `azure` provider on `<endpoint>/openai/v1` (the Responses API, with the deployment as the model). `openai_compatible` and `ollama` profiles use their `base_url`. A `github_copilot` profile
|
|
935
|
+
Scribe uses the active `llms.json` profile, or, with `llm.routing: auto`, the profile the router picks for the `coding` area (`--profile` and `code.model` still win; see [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md)). The key stays in your environment: the agent runtime's config refers to it as `{env:WP_AZURE_API_KEY}` and never contains the value. An `auth: entra` profile reaches the model through ForgeQE's local token gateway (see [Signing in to the model without a key](#signing-in-to-the-model-without-a-key-microsoft-entra-id)). An Azure profile goes through the agent runtime's `azure` provider on `<endpoint>/openai/v1` (the Responses API, with the deployment as the model). `openai_compatible` and `ollama` profiles use their `base_url`. A `github_copilot` profile cannot run Scribe yet, and `code.model: github-copilot/...` is refused: the agent runtime's own Copilot provider signs in through a third-party app on your GitHub account. `code.model: provider/model` (or `--code-model`) uses one of the agent runtime's built-in providers instead, with that provider's usual key variable (for example `ANTHROPIC_API_KEY` for `anthropic/...`).
|
|
892
936
|
|
|
893
937
|
Scribe may edit files and run commands anywhere in the repo without asking. `git push` in any form is denied (pushing goes through ForgeQE's guarded `push_branch` tool, section 8), and it cannot touch files outside the repo except the run and memory directories, which it may read.
|
|
894
938
|
|
|
@@ -1049,7 +1093,21 @@ Build a pack for Jira fix version 2.4 and the commits since release/2.3 in ../sh
|
|
|
1049
1093
|
- *This machine*: ForgeQE builds one command for exactly the pack's tests, the way the repository already runs a test (for .NET, a `--filter` with each test, the settings an earlier job used, TRX results), runs it, and reads each result. A failed test runs once more; one that then passes is flaky.
|
|
1050
1094
|
- *TestMu AI*: the same command, with the variable `FORGEQE_PW_WS_ENDPOINT` set to TestMu AI's Playwright endpoint for the browser and platform in `testmu.yaml` (the access key goes in the variable, never in the command, and is removed from the output). The tests must connect to it when it is set: for NUnit, override `ConnectOptionsAsync` in the test base class; for Playwright Test, set `connectOptions` from it in `playwright.config`. ForgeQE refuses a TestMu run when the tests do not read the variable, so a run never looks remote while it ran locally.
|
|
1051
1095
|
|
|
1052
|
-
Each run is kept with the pack: the card says which tests fail now that passed in the last run, and which were fixed.
|
|
1096
|
+
Each run is kept with the pack: the card says which tests fail now that passed in the last run, and which were fixed.
|
|
1097
|
+
|
|
1098
|
+
**Triage: heal, re-run, or file a bug?** After each run ForgeQE sorts every failed or flaky test into one route, from evidence that costs nothing first (the retry in the same run, the failure message, the test's history), so the AI is spent only where it decides something:
|
|
1099
|
+
|
|
1100
|
+
| Route | When | What happens |
|
|
1101
|
+
|---|---|---|
|
|
1102
|
+
| flaky | it passed when run again in the same run, or its history flip-flops | not healed: quarantine it, or fix the cause it flakes on |
|
|
1103
|
+
| environment | network errors, 5xx, a crashed browser, sign-in failing | run again once the environment works; no heal and no bug |
|
|
1104
|
+
| test code broken | a missing module, a compile error, a missing fixture | fix the test's code; healing against the page cannot fix it |
|
|
1105
|
+
| heal | an element the test uses is no longer found (high confidence when it passed its last runs) | the healing agent re-runs the flow in the browser and updates the steps and locators |
|
|
1106
|
+
| AI re-run decides | an expected value does not hold, a timeout, or something covers the element | the healing agent re-runs the flow in the browser: heals the test when the flow works, reports an app bug when it fails at the same step |
|
|
1107
|
+
|
|
1108
|
+
Failures that name the same element or the same message are shown as **one cause** (a page object or a shared step changed), so one fix can heal them together. When most failures are the environment, ForgeQE says to run the pack again before healing anything, and `heal_tests` refuses until you name the tests.
|
|
1109
|
+
|
|
1110
|
+
The triage is on the run's card and in `triage.md` and `triage.json` in the run's folder. To heal only the tests it sent to healing, ask "heal the failures of pack release-1.5", or run `forgeqe pack heal release-1.5` (`--tests` for some of them): ForgeQE builds a command for just those tests and the healing agent re-runs each flow in the browser before it changes anything. `forgeqe pack triage <name>` shows the last run's triage again. Packs are saved in `.forgeqe/packs/<name>.json` and `<name>.md`; run output and results in `.forgeqe/reports/packs/<name>/<time>/`.
|
|
1053
1111
|
|
|
1054
1112
|
### Flaky tests
|
|
1055
1113
|
|
|
@@ -1172,6 +1230,24 @@ forgeqe pr-tests 42 --run --post # and post the results on p
|
|
|
1172
1230
|
|
|
1173
1231
|
Without `--agent`, the impacted files run as they are; this builds the command for Playwright and pytest projects (for other frameworks add `--agent`, or run the listed files).
|
|
1174
1232
|
|
|
1233
|
+
### Reports for tests run outside ForgeQE
|
|
1234
|
+
|
|
1235
|
+
ForgeQE's HTML report is not only for runs it makes. `run_tests` and `pr-tests --run` write one, and so can any run in a terminal, an IDE's test runner, or CI:
|
|
1236
|
+
|
|
1237
|
+
```bash
|
|
1238
|
+
forgeqe report test-results/junit.xml # from JUnit XML or TRX: any framework, after the run
|
|
1239
|
+
forgeqe report setup # a report after every Playwright or pytest run, automatically
|
|
1240
|
+
```
|
|
1241
|
+
|
|
1242
|
+
`forgeqe report` reads JUnit XML or TRX files, or the folder they are in (Playwright `--reporter=junit`, pytest `--junitxml=results.xml`, `dotnet test --logger trx`), and exits 1 when a test failed. Each report is saved in `<out_dir>/results/<time>/`, so its Trends and Flaky tests pages cover the earlier runs, whoever ran them. JUnit carries one line per failure; a skipped test is counted, not shown.
|
|
1243
|
+
|
|
1244
|
+
`forgeqe report setup` sets up the framework the repository uses:
|
|
1245
|
+
|
|
1246
|
+
- **Playwright**: it copies a reporter to `.forgeqe/playwright-reporter.mjs` (commit it) and prints the line to add to `playwright.config`: `reporter: [["list"], ["./.forgeqe/playwright-reporter.mjs"]]`. Every run then ends with a report, from `npx playwright test`, the VS Code Playwright extension, or CI. With `screenshot: "only-on-failure"` and `trace: "retain-on-failure"`, each failure shows its screenshot, the full error and stack, and the trace.
|
|
1247
|
+
- **pytest**: install `forgeqe-cli` in the environment that runs the tests (`pip install forgeqe-cli`). Its plugin loads by itself, from the terminal, VS Code's test explorer, or CI, in a folder with `forgeqe.yaml` or `.forgeqe/` (elsewhere with `pytest --forgeqe-report`). A test that passes when pytest-rerunfailures runs it again is reported as flaky.
|
|
1248
|
+
|
|
1249
|
+
`FORGEQE_REPORT=0` turns the reporters off for a run (ForgeQE's own `run_tests` does this, as it writes its own report). The reporters call `forgeqe` on PATH, or `FORGEQE_BIN`; without it they keep the results file and print the command that builds the report. Screenshots and traces are copied into the report only from the working folder, never from a hidden folder or a file named like a key.
|
|
1250
|
+
|
|
1175
1251
|
## 7. Test cases and suites
|
|
1176
1252
|
|
|
1177
1253
|
Test cases you already have can be run two ways. In the shell or with `forgeqe run`, ask in plain words ("run SHOP-12 and SHOP-14", "run test plan 12 suite 34", "run the cases in smoke.yaml") or paste the cases, and the agent works out the source. `forgeqe suite` takes explicit references and runs the same way every time, with no agent involved, which is what pipelines call. Each case is one goal: ForgeQE tells the agent to do the steps in order and to check every expected result, and the agent's verdict decides whether the case passed. A case with no steps (plain text, a Jira story) is run from its description.
|
|
@@ -1543,6 +1619,91 @@ There is no scenario file to write: ask in plain language ("place an order in th
|
|
|
1543
1619
|
Billing, in UAT") and ForgeQE's agent picks the apps from their descriptions, shows the plan, runs each step with the
|
|
1544
1620
|
right engine, and carries values from one app to the next.
|
|
1545
1621
|
|
|
1622
|
+
### Recording a web test by hand
|
|
1623
|
+
|
|
1624
|
+
When you would rather show the steps than describe them, record them. ForgeQE opens Chrome on the app's page with the
|
|
1625
|
+
**Recorder** panel docked in a corner of the page, in the same window: you click, type and choose as usual, and each
|
|
1626
|
+
action is listed in the panel as a numbered step, in order. Drag the panel by its title to another corner, or make it
|
|
1627
|
+
small (a pill that shows the step count); it comes back where you left it on every page.
|
|
1628
|
+
|
|
1629
|
+
```bash
|
|
1630
|
+
forgeqe record shop --name "Apply a coupon at checkout" # a web app from env.yaml
|
|
1631
|
+
forgeqe record --url https://staging.shop.test/cart # or any page
|
|
1632
|
+
```
|
|
1633
|
+
|
|
1634
|
+
In the shell, say "I'll show you the checkout, record it" (or `/record shop`). What you can add while recording:
|
|
1635
|
+
|
|
1636
|
+
| In the panel | In the page: Ctrl+Shift+Space, then | What it adds |
|
|
1637
|
+
|---|---|---|
|
|
1638
|
+
| Check, then click an element | **A** | A check: its text, or its value for a field, or checked for a checkbox. Change the kind (text, visible, hidden, value) and mark **live data** (check the pattern, not this value) in the panel |
|
|
1639
|
+
| Variable, then click an element | **V** | Keeps the element's text for later steps: typing the same text later is written as `{{name}}` |
|
|
1640
|
+
| Wait for, then click an element | **W** | The test waits for the element to show |
|
|
1641
|
+
| Test data | **D** | The last value you typed becomes test data the test reads from its data |
|
|
1642
|
+
| `note: …` in the box | **N** | A note on the last step: "this is the cart page object" |
|
|
1643
|
+
| Pause / Undo / Stop, ✕ on a step | **P** / **Z** / **S** | Pause records nothing until you resume; ✕ removes that step |
|
|
1644
|
+
|
|
1645
|
+
Type a check in words in the panel's box, or in the terminal: `the total shows "$40.00"`. Quoted text the page shows
|
|
1646
|
+
becomes a check on the element showing it; without quotes, one model call finds the text that shows it on the page. A
|
|
1647
|
+
check the page cannot show yet is kept in your words, and the coding agent writes it. In the terminal, `undo`, `pause`,
|
|
1648
|
+
`delete 4` and `stop` work too.
|
|
1649
|
+
|
|
1650
|
+
What is recorded: clicks (a double click as one step), the final text of each field you typed in (not each key), choices
|
|
1651
|
+
in drop-downs, Enter and Escape, key chords, pages you open from the address bar, and Back. A click on a menu item that
|
|
1652
|
+
opened when the pointer went over its trigger gets a hover on the trigger first. Password fields are written as an
|
|
1653
|
+
environment variable (the saved login's `password_env`, or `<SITE>_PASSWORD`), and text equal to a saved login's value
|
|
1654
|
+
as its variable: what you type in them is never stored. A file you choose for an upload is recorded by its path in the
|
|
1655
|
+
working folder; one outside it, a hidden file (`.env`, anything under `.git`), or one named like credentials or keys is
|
|
1656
|
+
listed as not recorded. Drag and drop is listed as not recorded. `--signed-in` starts with the saved sessions of
|
|
1657
|
+
`logins:`; without it, record the sign-in too.
|
|
1658
|
+
|
|
1659
|
+
**Data that changes from run to run.** A test that spells out this recording's price, date, or order number fails on
|
|
1660
|
+
the next run, so ForgeQE keeps what holds and makes the rest a pattern, a variable, or a value made fresh:
|
|
1661
|
+
|
|
1662
|
+
| In the recording | In the test |
|
|
1663
|
+
|---|---|
|
|
1664
|
+
| A check of a price, date, time, order number, or count ("$14.15", "2,463 found") | A check of its pattern: `/^\$\d[\d,]*\.\d{2}$/` |
|
|
1665
|
+
| The same value checked again later (the product's price, then the cart total) | The first check keeps it as `{{price}}`; the later one checks `{{price}}` |
|
|
1666
|
+
| A link or button named with such data ("Combination Pliers $14.15") | Found by the part that holds: `getByRole("link", { name: /Combination Pliers/ })` |
|
|
1667
|
+
| A date you typed | The date the same number of days from today: `{{$date:+3:%Y-%m-%d}}` |
|
|
1668
|
+
| A value the app refuses twice (a new account's email) | A part new on each run: `alice+{{$unique}}@example.com` |
|
|
1669
|
+
|
|
1670
|
+
Prices, dates, times, ids, and long numbers are found by their form; a small number on its own ("1" in the cart) stays
|
|
1671
|
+
exact, as it is usually what the check is about. When you stop, one model call sorts the values the forms cannot
|
|
1672
|
+
tell (a new account's email, "today's deal"); without a model, those stay as recorded. Each change shows in the panel
|
|
1673
|
+
with a tick box: untick it to keep the value as recorded.
|
|
1674
|
+
|
|
1675
|
+
**Recording a test case.** `forgeqe record --case SHOP-12` (any source ForgeQE reads cases from: Jira, Azure DevOps,
|
|
1676
|
+
Xray, TestRail, a file) starts on the case's page and shows its steps as a checklist in the panel. Click the step you
|
|
1677
|
+
are on (or **Next case step**): what you record belongs to it, and a check you add there takes its expected result.
|
|
1678
|
+
Each expected result shows **✓ checked** or **no check yet**; Stop warns once about the ones with no check, and a second
|
|
1679
|
+
Stop stops. The test is named and tagged for the case, and mapped to it once it passes.
|
|
1680
|
+
|
|
1681
|
+
When you stop, the run folder holds `workflow.yaml` and `trace.md` as for any run. With a repository (`--repo`, or
|
|
1682
|
+
`code.repo`), ForgeQE asks whether to turn it into a test. First a planner reads the repository and shows the plan:
|
|
1683
|
+
|
|
1684
|
+
```
|
|
1685
|
+
Test tests/add-to-cart.spec.ts › adds pliers to the cart
|
|
1686
|
+
Steps
|
|
1687
|
+
2. search: HomePage.searchFor()
|
|
1688
|
+
4. open the product: new HomePage.product(name)
|
|
1689
|
+
6. add to the cart: new ProductPage.addToCart()
|
|
1690
|
+
New code
|
|
1691
|
+
ProductPage in pages/ProductPage.ts
|
|
1692
|
+
Checks
|
|
1693
|
+
after step 5: expect(product.price).toHaveText(/^\$\d+\.\d{2}$/)
|
|
1694
|
+
after step 6: expect(product.cartQuantity).toHaveText("1")
|
|
1695
|
+
```
|
|
1696
|
+
|
|
1697
|
+
It asks at most three questions where the repository allows two good answers (a new file, or the existing spec?), then
|
|
1698
|
+
**Write it**, **Change the plan** (say what to change, in your words), or **Cancel**. The coding agent replays the
|
|
1699
|
+
recording in Playwright, writes the test to the plan you agreed in the repository's framework, and asserts each check at
|
|
1700
|
+
the step where you added it. In the shell, "change the plan" comes back to ForgeQE's agent, which asks you what to
|
|
1701
|
+
change. `--no-plan` (or `--yes`) skips the plan. Later: `forgeqe code <run folder>`.
|
|
1702
|
+
|
|
1703
|
+
The recorder runs in an isolated world in each page, so the page's own scripts can neither see it nor add steps, and it
|
|
1704
|
+
records only real input: clicks and keys a script fakes are ignored. The panel's own clicks and keys never reach the page
|
|
1705
|
+
and are never recorded, and it stays above the page's dialogs. Recording uses local Chrome or Chromium.
|
|
1706
|
+
|
|
1546
1707
|
### Desktop apps (Windows)
|
|
1547
1708
|
|
|
1548
1709
|
ForgeQE also automates Windows thick-client apps (Win32, WinForms, WPF). In the shell, ask for a goal in the app
|
|
@@ -1710,6 +1871,29 @@ same rules (forgeqe.yaml's own declaration of a service wins):
|
|
|
1710
1871
|
`x-forgeqe: { ready: … }` sets the ready check. Without it, a service is ready when its first port takes connections.
|
|
1711
1872
|
The project's containers share their own network, so services reach each other by name (`db:5432`).
|
|
1712
1873
|
|
|
1874
|
+
**Docker as the workspace and the browser.** Two settings move the rest of the work into containers:
|
|
1875
|
+
|
|
1876
|
+
```yaml
|
|
1877
|
+
browser:
|
|
1878
|
+
provider: docker # the browser agent, the replay and Scribe's live browser use Chromium in a container
|
|
1879
|
+
containers:
|
|
1880
|
+
workspace: # Scribe's shell commands and every test ForgeQE runs use this image
|
|
1881
|
+
image: mcr.microsoft.com/playwright/dotnet:v1.63.0-noble
|
|
1882
|
+
env: { NEXUS_ENVIRONMENT: CI }
|
|
1883
|
+
browser: # optional
|
|
1884
|
+
headless: false # visible on a virtual display inside the container (some sites refuse headless)
|
|
1885
|
+
```
|
|
1886
|
+
|
|
1887
|
+
- **Workspace:** Scribe builds, restores, runs and fixes tests in a Linux container of the project's image, with the
|
|
1888
|
+
repository at the same path. Each test run gets its own container from that image, and `HOME` is a volume of the
|
|
1889
|
+
project's, so package caches outlive the session. The agents and the model key stay on this machine, and ForgeQE's
|
|
1890
|
+
own agent keeps this machine's shell. `.env`, keys and other credential files in the repository are covered inside
|
|
1891
|
+
the containers.
|
|
1892
|
+
- **Browser:** each browser run gets a fresh container, reached through a gateway that wants a random token. A
|
|
1893
|
+
declared service is reached by its name; an app on this machine is `host.docker.internal`, not `localhost`.
|
|
1894
|
+
`--browser-provider docker` picks it for one run.
|
|
1895
|
+
- `FORGEQE_HOST_RUN=1` runs Scribe's commands and the tests on this machine for a session.
|
|
1896
|
+
|
|
1713
1897
|
**Tests in a container.** With `containers.tests`, `run_tests` runs the repository's tests in that image instead of
|
|
1714
1898
|
on this machine, after starting the services it lists:
|
|
1715
1899
|
|
|
@@ -2281,7 +2465,7 @@ What it does:
|
|
|
2281
2465
|
5. Stores the keys the pipeline needs (the model's key from `llms.json`, and `ADO_PAT`, `JIRA_EMAIL` and `JIRA_API_TOKEN`, or `XRAY_CLIENT_ID` and `XRAY_CLIENT_SECRET`) with ForgeQE's `set_ci_secret` tool. The agent only names the variable; ForgeQE passes the value from its own environment to `gh secret set` (on stdin) or `az pipelines variable` (through the environment), and never shows it. A variable that is not set in ForgeQE's environment is reported, not guessed.
|
|
2282
2466
|
6. Triggers the run with `gh workflow run` or `az pipelines run`, watches it, reads the logs of failed steps, fixes the pipeline (or the code), pushes again, and repeats until the run passes or `code.max_attempts` runs have failed. It reports the run link and the result.
|
|
2283
2467
|
|
|
2284
|
-
It needs `gh` logged in for GitHub, or `az` with the `azure-devops` extension logged in for Azure DevOps. Without them it still writes, validates, and commits the pipeline, then tells you the commands to run. Set `llm.active` to a live profile in the committed `forgeqe.yaml`. With a `github_copilot` profile the pipeline needs
|
|
2468
|
+
It needs `gh` logged in for GitHub, or `az` with the `azure-devops` extension logged in for Azure DevOps. Without them it still writes, validates, and commits the pipeline, then tells you the commands to run. Set `llm.active` to a live profile in the committed `forgeqe.yaml`. With a `github_copilot` profile the pipeline needs a GitHub token for Copilot in `COPILOT_GITHUB_TOKEN` (see [In pipelines](#github-copilot-seats-as-the-model)).
|
|
2285
2469
|
|
|
2286
2470
|
## 9. Monitoring
|
|
2287
2471
|
|
|
@@ -2583,8 +2767,8 @@ the list again. For a client that ignores it, `forgeqe mcp --all-tools` lists ev
|
|
|
2583
2767
|
| `WP_LLMS_JSON` | Path to `llms.json` |
|
|
2584
2768
|
| `WP_LLM_PROFILE` | Active profile, unless `--profile` is set |
|
|
2585
2769
|
| `WP_AZURE_API_KEY` / `OPENAI_API_KEY` / `WP_API_KEY` | Keys referenced by profiles |
|
|
2586
|
-
| `
|
|
2587
|
-
| `
|
|
2770
|
+
| `COPILOT_GITHUB_TOKEN` | A `github_copilot` profile's GitHub token (fine-grained, with the Copilot Requests permission), unless the profile's `api_key_env` names another; else the saved token, then the GitHub CLI's sign-in |
|
|
2771
|
+
| `FORGEQE_COPILOT_TOKEN` | The token `forgeqe auth login copilot` saves; set by ForgeQE from the credential store, not by hand |
|
|
2588
2772
|
| `FORGEQE_CONFIG_DIR` | Global config folder (default `~/.config/forgeqe`): `forgeqe.yaml`, `llms.json` and `.env` used when a project has none |
|
|
2589
2773
|
| `FORGEQE_PYTHON` | A Python that already has the engine's libraries, used instead of the managed venv (skipped, with a note, when it cannot run the engine) |
|
|
2590
2774
|
| `FORGEQE_DEBUG` | `1` logs at debug level, like `--debug` |
|
package/README.md
CHANGED
|
@@ -51,7 +51,7 @@ ForgeQE sends no telemetry and turns it off in everything it starts:
|
|
|
51
51
|
- The agent runtime: auto-update, session sharing, OpenTelemetry spans, and the model catalogue download are off; it uses its bundled model list.
|
|
52
52
|
- The repo's test command: `DO_NOT_TRACK=1`, plus the opt-outs for Next.js, Nuxt, Astro, Gatsby, Storybook, Turborepo, Angular, .NET, and Cypress.
|
|
53
53
|
|
|
54
|
-
`OTEL_EXPORTER_OTLP_*` variables are removed from those processes, and your environment cannot turn any of this back on. The
|
|
54
|
+
`OTEL_EXPORTER_OTLP_*` variables are removed from those processes, and your environment cannot turn any of this back on. The traffic is to your model endpoint and the sites the agent visits. Besides that, ForgeQE downloads only public files: the model price lists from models.dev and LiteLLM, once a day (`FORGEQE_PRICES=off` stops them); the agent runtime from its GitHub releases, when it is not installed; and the engine's ad-block and cookie-banner extensions from the Chrome Web Store, once. Nothing about your runs is sent.
|
|
55
55
|
|
|
56
56
|
## Configure
|
|
57
57
|
|
|
@@ -88,7 +88,7 @@ The active profile is `llm.active` in `forgeqe.yaml`, else `default` in `llms.js
|
|
|
88
88
|
|
|
89
89
|
**Model router.** With more than one model in `llms.json`, `llm.routing: auto` gives each area of work (ForgeQE's agent, browser runs, coding, review, analysis, planning) the cheapest configured model that is good enough for it. ForgeQE checks each model once and rates it, and keeps the result in `.forgeqe/model-profile.json`. `forgeqe models --routes` shows the model each area gets, and why. `--profile` and `code.model` still win. See [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md).
|
|
90
90
|
|
|
91
|
-
**GitHub Copilot.** With Copilot licences, a `github_copilot` profile runs
|
|
91
|
+
**GitHub Copilot.** With Copilot licences, a `github_copilot` profile runs the browser engine and the desktop and mobile agents on the Copilot seat through GitHub's Copilot SDK, behind a gateway on `127.0.0.1`. It signs in with a GitHub token (`forgeqe auth login copilot`: a fine-grained token with the Copilot Requests permission, or the GitHub CLI's sign-in), so no third-party app is authorized on the account. ForgeQE's agent and Scribe do not run on Copilot yet. See HOW_TO_USE.md, *GitHub Copilot seats as the model*.
|
|
92
92
|
|
|
93
93
|
## CLI
|
|
94
94
|
|
package/THIRD_PARTY_NOTICES.md
CHANGED
|
@@ -60,7 +60,8 @@ ForgeQE downloads these onto the machine it runs on, from their own sources, whe
|
|
|
60
60
|
| Bun | the JavaScript runtime, when the machine has none (bin/forgeqe.js) | MIT | https://bun.sh |
|
|
61
61
|
| OpenCode | the agent runtime for ForgeQE's agent and Scribe (src/code/runtime.ts) | MIT | https://opencode.ai |
|
|
62
62
|
| uv and Python | the browser engine's interpreter (src/engine/setup.ts) | MIT or Apache-2.0; PSF | https://github.com/astral-sh/uv |
|
|
63
|
-
| browser-use, Playwright, PyYAML, azure-identity | the browser engine's libraries and Chromium (src/engine/setup.ts) | MIT; Apache-2.0; MIT; MIT | https://pypi.org |
|
|
63
|
+
| browser-use, Playwright, PyYAML, azure-identity, github-copilot-sdk | the browser engine's libraries and Chromium (src/engine/setup.ts) | MIT; Apache-2.0; MIT; MIT; MIT | https://pypi.org |
|
|
64
|
+
| GitHub Copilot CLI runtime | downloaded by GitHub's Copilot SDK for github_copilot profiles (src/copilot.ts) | GitHub Copilot CLI license (GitHub) | https://github.com/github/copilot-cli |
|
|
64
65
|
|
|
65
66
|
## Licence texts
|
|
66
67
|
|