@capagents/forgeqe 1.3.1 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/HOW_TO_USE.md +373 -9
- package/README.md +55 -4
- package/docs/ARCHITECTURE.md +62 -1
- package/docs/CONTAINERS.md +227 -0
- package/docs/COST_ARCHITECTURE.md +32 -0
- package/docs/DESKTOP-CI.md +87 -0
- package/docs/DESKTOP.md +387 -0
- package/docs/MOBILE.md +228 -0
- package/docs/feature-test-prompts.md +25 -0
- package/docs/security-data-flow.html +440 -3
- package/docs/user-guide.html +797 -415
- package/env.example.yaml +67 -0
- package/forgeqe.example.yaml +56 -1
- package/llms.example.json +7 -0
- package/package.json +4 -1
- package/prompts/brain.md +31 -0
- package/prompts/find-tests.md +1 -0
- package/py/forgeqe_desktop/__init__.py +1 -0
- package/py/forgeqe_desktop/__main__.py +3 -0
- package/py/forgeqe_desktop/apps.py +80 -0
- package/py/forgeqe_desktop/checks.py +168 -0
- package/py/forgeqe_desktop/cli.py +90 -0
- package/py/forgeqe_desktop/handhooks.py +293 -0
- package/py/forgeqe_desktop/handrecord.py +239 -0
- package/py/forgeqe_desktop/heal.py +107 -0
- package/py/forgeqe_desktop/models.py +208 -0
- package/py/forgeqe_desktop/pywinauto_driver.py +217 -0
- package/py/forgeqe_desktop/record.py +150 -0
- package/py/forgeqe_desktop/replay.py +345 -0
- package/py/forgeqe_desktop/service.py +356 -0
- package/py/forgeqe_desktop/session.py +60 -0
- package/py/forgeqe_desktop/ufo_bridge.py +86 -0
- package/py/forgeqe_desktop/ufo_runner.py +305 -0
- package/py/forgeqe_desktop/visual.py +112 -0
- package/py/forgeqe_desktop/workflow.py +338 -0
- package/py/forgeqe_engine/cli.py +10 -0
- package/py/forgeqe_engine/llm.py +54 -4
- package/py/forgeqe_engine/model_params.py +169 -0
- package/py/forgeqe_engine/summary.py +6 -0
- package/py/forgeqe_mobile/__init__.py +1 -0
- package/py/forgeqe_mobile/__main__.py +3 -0
- package/py/forgeqe_mobile/appium_driver.py +342 -0
- package/py/forgeqe_mobile/apps.py +84 -0
- package/py/forgeqe_mobile/cli.py +70 -0
- package/py/forgeqe_mobile/explore.py +203 -0
- package/py/forgeqe_mobile/hyperexecute.py +261 -0
- package/py/forgeqe_mobile/maestro.py +321 -0
- package/py/forgeqe_mobile/service.py +332 -0
- package/skills/testcontainers/SKILL.md +42 -0
- package/src/agents.ts +11 -2
- package/src/api/plan.ts +2 -0
- package/src/api/write.ts +2 -0
- package/src/auth/copilot.ts +130 -0
- package/src/auth/signin.ts +54 -1
- package/src/brain/index.ts +26 -4
- package/src/capabilities/guide.ts +2 -0
- package/src/capabilities/registry.ts +110 -2
- package/src/capabilities/status.ts +15 -0
- package/src/cases/defects.ts +2 -1
- package/src/cases/rca.ts +2 -0
- package/src/childEnv.ts +4 -1
- package/src/cli/args.ts +3 -2
- package/src/cli/help.ts +24 -3
- package/src/cli/index.ts +47 -3
- package/src/code/find.ts +26 -0
- package/src/code/loop.ts +4 -1
- package/src/code/prompt.ts +2 -1
- package/src/code/prreview.ts +2 -0
- package/src/code/review.ts +6 -2
- package/src/code/runtime.ts +34 -3
- package/src/commands.ts +6 -1
- package/src/config.ts +159 -3
- package/src/containers/command.ts +68 -0
- package/src/containers/compose.ts +173 -0
- package/src/containers/config.ts +283 -0
- package/src/containers/mock.ts +156 -0
- package/src/containers/run.ts +362 -0
- package/src/containers/tests.ts +80 -0
- package/src/context.ts +33 -6
- package/src/copilot.ts +155 -0
- package/src/desktop/bridge.ts +229 -0
- package/src/desktop/command.ts +350 -0
- package/src/desktop/edit.ts +217 -0
- package/src/desktop/junit.ts +35 -0
- package/src/desktop/report.ts +155 -0
- package/src/desktop/run.ts +252 -0
- package/src/desktop/setup.ts +134 -0
- package/src/engine/bridge.ts +12 -5
- package/src/engine/setup.ts +6 -3
- package/src/environments.ts +566 -0
- package/src/evals/domain.ts +1 -0
- package/src/explore/charters.ts +2 -0
- package/src/guardrails.ts +145 -0
- package/src/heal/loop.ts +4 -1
- package/src/heal/prompt.ts +2 -1
- package/src/init.ts +5 -2
- package/src/llm.ts +23 -2
- package/src/mcp/attachtools.ts +16 -4
- package/src/mcp/captools.ts +27 -0
- package/src/mcp/containertools.ts +212 -0
- package/src/mcp/desktoptools.ts +334 -0
- package/src/mcp/mobiletools.ts +132 -0
- package/src/mcp/reporttools.ts +17 -2
- package/src/mcp/repotests.ts +22 -3
- package/src/mcp/routing.ts +11 -5
- package/src/mcp/scope.ts +210 -0
- package/src/mcp/server.ts +80 -5
- package/src/mcp/tools.ts +38 -1
- package/src/mobile/command.ts +245 -0
- package/src/mobile/run.ts +226 -0
- package/src/pack/pipeline.ts +44 -2
- package/src/refine.ts +3 -0
- package/src/report/html.ts +3 -0
- package/src/report/template.html +48 -48
- package/src/run.ts +7 -3
- package/src/tokenomics/ledger.ts +3 -1
- package/src/tui/App.tsx +23 -0
- package/src/update.ts +118 -8
- package/src/usage.ts +1 -1
package/HOW_TO_USE.md
CHANGED
|
@@ -54,6 +54,10 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
|
|
|
54
54
|
| Test cases | **Write test cases from a story**: Writes test cases from a story's acceptance criteria and saves them in Azure DevOps or Jira, linked to the story. Reads the cases already linked first, so asking again adds only what is missing and updates cases in place instead of duplicating them; plans end-to-end journeys from the existing cases. | `story_test_cases`, `save_test_cases`, `e2e_journey` | [docs](#where-cases-come-from) |
|
|
55
55
|
| Test cases | **Domain and regulatory test design**: Every agent works as an expert in the product's industry would, automatically: the risks the industry is known for and the obligations of its regulations and standards (for example DORA, PSD2, GDPR, ISO 27001), declared in forgeqe.yaml or worked out from the story and the application. It applies to analysing requirements, writing and checking test cases, exploration, writing, reviewing, and healing test code, API tests, finding tests, and pull request reviews, labelled so coverage can be traced and citing only the control ids in the team's compliance file. forgeqe eval domain measures how well the agent does it. | | [docs](#16-domain-and-regulations) |
|
|
56
56
|
| Test cases | **Exploratory testing of a story**: Explores a story in real browsers beyond its scripted cases (edge cases, invalid input, boundaries, roles, state), confirms each defect by reproducing it, proposes test cases, and files bugs once the user agrees. | `explore_story` | [docs](#exploratory-testing-from-a-story) |
|
|
57
|
+
| Setup | **Test environments and their apps**: Knows the test environments in env.yaml (QA, UAT, …) and each one's apps: web apps and APIs by URL, Windows desktop apps, and mobile apps, with what each is for. Every agent works from the active one; switches environment on request, picks the apps a scenario needs, and carries values such as an order number from one app to the next. | `test_apps`, `use_environment` | [docs](#environments-and-apps) |
|
|
58
|
+
| Desktop | **Automate Windows desktop apps**: Carries out a plain-language goal in a Windows thick-client app with UFO² and saves it as a workflow with checks of the outcome, or records the steps a tester does by hand, with no model; replays saved workflows with no model, healing a step whose control moved, and fixes a workflow from chat (a timeout, a wait, a dialog to dismiss) after showing the change. Apps and the secrets they may type come from env.yaml or forgeqe.yaml desktop:. | `desktop_explore`, `desktop_record`, `desktop_replay`, `desktop_edit_workflow` | [docs](#desktop-apps-windows) |
|
|
59
|
+
| Mobile | **Automate Android and iOS apps**: Carries out a plain-language goal in an Android or iOS app through Appium and saves it as a mobile workflow with checks of the outcome; replays workflows with no model on Appium (healing a moved control) or Maestro; runs the team's own Maestro flows after checking them against an allowlist. Apps come from env.yaml (kind: mobile) or forgeqe.yaml mobile:. | `mobile_explore`, `mobile_replay`, `mobile_maestro` | [docs](#mobile-apps-android-ios) |
|
|
60
|
+
| Operations | **Services and mocks in Docker or Podman**: Starts the services a test run needs (a database, a queue, the API under test) in Docker or Podman, as forgeqe.yaml declares them, and waits until each is ready; stops them, and shows their status and logs. Starts mocks of an API on WireMock, from the calls a browser run recorded or from an OpenAPI description, and can point an API app of the environment at the mock for the session. | `services`, `mock_service` | [docs](#containers-and-mock-services) |
|
|
57
61
|
| Automated tests | **Turn a passed run into an automated test**: Scribe, the coding agent, writes a test for a passed browser run in the repository's own framework, page objects, and helpers; ForgeQE runs it until it passes and a reviewer approves it. | `write_test`, `run_test` | [docs](#6-tests) |
|
|
58
62
|
| Automated tests | **Heal failing tests**: Runs a repository's tests and updates the ones the application has outgrown against the live app, or reports a real application bug instead. | `heal_tests` | [docs](#fixing-broken-tests) |
|
|
59
63
|
| Automated tests | **Find and run existing tests**: Finds the automated tests that match a feature, tags, ids, a folder, or a sprint, confirms the selection, and runs it once the user agrees, with a verdict per test. | `find_tests`, `run_tests`, `list_sprints` | [docs](#finding-and-running-existing-tests) |
|
|
@@ -71,7 +75,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
|
|
|
71
75
|
| Release | **Reports you can share**: Writes up what ForgeQE's agent found, such as open defects across projects, release status, or pipeline health, as a page with filters (project, state, priority, assignee, …), charts, and a sortable table, plus Markdown and CSV. Every table is built from evidence: the results of the tool calls the agent made, or a read-only query ForgeQE runs itself, listed on the page's Sources with each raw result kept beside the report. | `write_report` | [docs](#reports-from-the-agent) |
|
|
72
76
|
| Operations | **CI pipelines**: Writes a GitHub Actions or Azure Pipelines workflow that runs ForgeQE, stores the secrets it needs, pushes a branch, and makes the pipeline green. | `push_branch`, `set_ci_secret` | [docs](#8-pipelines) |
|
|
73
77
|
| Systems | **Work with Azure DevOps, Jira, GitHub, and Confluence**: Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, puts run evidence (screenshots, reports) on work items and issues as attachments or comments, and builds Azure DevOps dashboards and wiki pages. | `query_api`, `read_confluence`, `create_ado_dashboard`, `write_ado_wiki_page`, `attach_to_work_item` | [docs](README.md#azure-devops-jira-and-other-systems) |
|
|
74
|
-
| Setup | **Setup, skills, and documentation**: Shows the model profiles and MCP servers, installs or repairs the browser engine, creates a forgeqe.yaml, manages skills, and answers questions about ForgeQE from its documentation and this list. | `forgeqe_setup`, `capabilities`, `playbook` | [docs](#2-scaffold-config) |
|
|
78
|
+
| Setup | **Setup, skills, and documentation**: Shows the model profiles and MCP servers, installs or repairs the browser engine, creates a forgeqe.yaml, manages skills, and answers questions about ForgeQE from its documentation and this list. | `forgeqe_setup`, `capabilities`, `playbook`, `use_capabilities` | [docs](#2-scaffold-config) |
|
|
75
79
|
|
|
76
80
|
Not supported: Native mobile apps (iOS, Android) and desktop applications (instead: web applications in local Chrome or Chromium, or configured remote web grids such as BrowserStack and TestMu AI/LambdaTest); Load, stress, or soak testing with many simultaneous users (instead: per-page Core Web Vitals with budgets); Penetration testing or dynamic security scanning (instead: security findings in pull request reviews).
|
|
77
81
|
<!-- capabilities:end -->
|
|
@@ -86,7 +90,7 @@ Not supported: Native mobile apps (iOS, Android) and desktop applications (inste
|
|
|
86
90
|
|-------------|-------|
|
|
87
91
|
| **npm (Node.js 18+) or pip (Python 3.9+)** | To install `@capagents/forgeqe` or `forgeqe-cli`; either one gives the same `forgeqe` command |
|
|
88
92
|
| **A terminal** | macOS Terminal, iTerm2, VS Code or Cursor's terminal, Windows Terminal |
|
|
89
|
-
| **A model** | A profile in `llms.json` with a working key, or Microsoft Entra ID sign-in for Azure (Azure OpenAI, OpenAI, an OpenAI-compatible gateway, or Ollama) |
|
|
93
|
+
| **A model** | A profile in `llms.json` with a working key, or Microsoft Entra ID sign-in for Azure (Azure OpenAI, OpenAI, an OpenAI-compatible gateway, or Ollama), or a GitHub Copilot seat (see [GitHub Copilot seats as the model](#github-copilot-seats-as-the-model)) |
|
|
90
94
|
| **Internet access on first use** | ForgeQE downloads Bun, the agent runtime, uv, Python, and the engine into `~/.forgeqe` once |
|
|
91
95
|
|
|
92
96
|
### Optional
|
|
@@ -116,6 +120,14 @@ forgeqe models # check the profiles
|
|
|
116
120
|
forgeqe start # the shell
|
|
117
121
|
```
|
|
118
122
|
|
|
123
|
+
With GitHub Copilot licences instead of a model key, sign in rather than exporting one, and set `llm.active: copilot` in `forgeqe.yaml` (or `"default": "copilot"` in `llms.json`):
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
forgeqe auth login copilot # ForgeQE's agent and Scribe on your Copilot seat
|
|
127
|
+
forgeqe auth login github # the browser engine, through GitHub Models
|
|
128
|
+
forgeqe models # the copilot profile is active
|
|
129
|
+
```
|
|
130
|
+
|
|
119
131
|
In the shell, type a request and press Enter:
|
|
120
132
|
|
|
121
133
|
```
|
|
@@ -213,6 +225,63 @@ Give that identity the **Cognitive Services OpenAI User** role on the resource.
|
|
|
213
225
|
|
|
214
226
|
`auth: entra` also works on an `openai_compatible` profile, for a gateway such as API Management that checks Entra ID tokens: set `token_scope` to the scope it expects (for example `api://<app id>/.default`; the default is `https://cognitiveservices.azure.com/.default`). The engine needs the `azure-identity` package, which `forgeqe setup` installs (and adds to environments set up before it was needed); with `FORGEQE_PYTHON`, install it there yourself.
|
|
215
227
|
|
|
228
|
+
### GitHub Copilot seats as the model
|
|
229
|
+
|
|
230
|
+
With GitHub Copilot licences (Pro, Pro+, Business or Enterprise), a `github_copilot` profile runs ForgeQE's agent and Scribe on the Copilot seat. They work through the agent runtime's own GitHub Copilot sign-in, [which GitHub supports](https://github.blog/changelog/2026-01-16-github-copilot-now-supports-opencode/). Copilot gives other tools no model API of their own, so the browser engine and the desktop agent use [GitHub Models](https://docs.github.com/en/github-models) instead, with your GitHub sign-in:
|
|
231
|
+
|
|
232
|
+
```json
|
|
233
|
+
"copilot": {
|
|
234
|
+
"provider": "github_copilot",
|
|
235
|
+
"model": "claude-sonnet-4.6",
|
|
236
|
+
"engine_model": "openai/gpt-4.1",
|
|
237
|
+
"github_org": "acme"
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
| Field | What it is |
|
|
242
|
+
|---|---|
|
|
243
|
+
| `model` | The Copilot model for ForgeQE's agent and Scribe, as Copilot names it (`claude-sonnet-4.6`, `gpt-5.4`). The sign-in lists the ones your seat has. |
|
|
244
|
+
| `engine_model` | The GitHub Models model for the browser engine and the desktop agent, as `publisher/model`. It must take images. Default `openai/gpt-4.1`. |
|
|
245
|
+
| `github_org` | Optional. Bills the engine's GitHub Models calls to this organization and applies its policies, rather than your personal free allowance. |
|
|
246
|
+
| `api_key_env` | Optional. The variable holding the GitHub Models token; default `GITHUB_TOKEN`, else the GitHub CLI's sign-in. |
|
|
247
|
+
|
|
248
|
+
Sign in once per machine:
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
forgeqe auth login copilot # Copilot, for ForgeQE's agent and Scribe (GitHub.com or GitHub Enterprise, e.g. acme.ghe.com)
|
|
252
|
+
forgeqe auth login github # the GitHub CLI, whose sign-in the browser engine uses for GitHub Models
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
`forgeqe auth login copilot` shows a one-time code and opens GitHub's device page; authorize with the account that has the seat. The sign-in goes to your system's credential store like ForgeQE's other secrets. The agent runtime gets it in its environment for each session, never in a file, and the agent's shell commands never see it. A Copilot sign-in made with `opencode auth login` also works. In CI, set `GITHUB_TOKEN` to a token with the `models:read` permission for the engine.
|
|
256
|
+
|
|
257
|
+
`forgeqe auth` shows the sign-in on its `Copilot` line (`saved sign-in as dana`, or `agent runtime's own sign-in` for one made with `opencode auth login`); `forgeqe auth logout copilot` forgets it.
|
|
258
|
+
|
|
259
|
+
Which part uses what:
|
|
260
|
+
|
|
261
|
+
| Part | Model | Signs in with |
|
|
262
|
+
|---|---|---|
|
|
263
|
+
| ForgeQE's agent (the shell, `forgeqe run`) | `model`, on the Copilot seat | `forgeqe auth login copilot` |
|
|
264
|
+
| Scribe (writing, fixing and reviewing tests) | `model`, on the Copilot seat | `forgeqe auth login copilot` |
|
|
265
|
+
| The browser engine (every browser run: `run --direct`, suites, monitoring) | `engine_model`, on GitHub Models | `GITHUB_TOKEN`, or `forgeqe auth login github` |
|
|
266
|
+
| The desktop agent | `engine_model`, on GitHub Models | `GITHUB_TOKEN`, or `forgeqe auth login github` |
|
|
267
|
+
|
|
268
|
+
`code.model` still overrides `model` for ForgeQE's agent and Scribe; `code.model: github-copilot/<model>` uses the Copilot sign-in too, with any other profile active.
|
|
269
|
+
|
|
270
|
+
What each part costs: the agent's and Scribe's requests count against the seat's premium requests. The engine's GitHub Models calls are billed by GitHub Models: the free allowance is rate-limited, so for suites set `github_org` to an organization with paid GitHub Models usage turned on. If a Copilot model is refused, check that your organization's Copilot policies allow that model. ForgeQE shows no cost for a `github_copilot` profile unless you give it a `price`.
|
|
271
|
+
|
|
272
|
+
**In pipelines.** `forgeqe suite ... --no-code`, which pipelines run, uses only the browser engine, so it needs GitHub Models and no Copilot sign-in. In GitHub Actions, give the job `permissions: models: read` and pass the workflow's own token as `GITHUB_TOKEN`; on Azure Pipelines or elsewhere, store a fine-grained token with the `models:read` permission as the `GITHUB_TOKEN` secret. A Copilot seat belongs to one person, so keep ForgeQE's agent and Scribe on developers' machines rather than signing a shared pipeline in to someone's seat.
|
|
273
|
+
|
|
274
|
+
**When something goes wrong.**
|
|
275
|
+
|
|
276
|
+
| ForgeQE says | What to do |
|
|
277
|
+
|---|---|
|
|
278
|
+
| `The github_copilot profile for … needs a GitHub Copilot sign-in` | Run `forgeqe auth login copilot` (or `/auth copilot` in the shell). |
|
|
279
|
+
| `The browser engine reaches GitHub Models with a GitHub sign-in, and there is none` | Run `forgeqe auth login github`, or set `GITHUB_TOKEN` to a token with `models:read`. |
|
|
280
|
+
| `… must be a Copilot model id`, `engine_model must be a GitHub Models id`, `github_org must be a GitHub organization name` | Fix that field in `llms.json`: the sign-in lists the seat's Copilot models, and [the GitHub Models catalog](https://github.com/marketplace?type=models) lists `publisher/model` ids. |
|
|
281
|
+
| The agent's model is refused, or not found | The seat's plan or your organization's Copilot policies do not allow that model: pick one the sign-in listed, or ask a Copilot admin to enable it. |
|
|
282
|
+
| The browser engine gets `429` / rate limit errors | The GitHub Models free allowance is used up: set `github_org` to an organization with paid GitHub Models usage, or wait for the limit to reset. |
|
|
283
|
+
| The code page does not open, or the sign-in says `not a GitHub Enterprise domain` | Give only the host of your GitHub Enterprise (`acme.ghe.com`). ForgeQE opens only GitHub's own `https` device page. |
|
|
284
|
+
|
|
216
285
|
`init` also puts three folders in `.forgeqe/` to read, change, and commit:
|
|
217
286
|
|
|
218
287
|
| Folder | What it holds |
|
|
@@ -254,13 +323,15 @@ forgeqe prompts eval refine --runs 3 # one prompt, three runs of each case
|
|
|
254
323
|
|
|
255
324
|
**Product knowledge.** Fill in the templates (or add your own `.md` and `.txt` files, in folders if you like), one topic per `##` heading. ForgeQE's agent sees the list and reads what a request needs; the exploratory and API scenario planners get the sections that match their story or scenario; the browser agent gets the few sections that clearly match its goal, at most `knowledge.browser_max_chars` (1500 by default; 0 turns it off). Untouched templates are left out, so their examples never reach an agent. Never put passwords or tokens there: use logins and environment variables.
|
|
256
325
|
|
|
326
|
+
**Guardrails.** Rules every agent must follow, to keep it grounded: ForgeQE's agent, the browser agent, and the agents that plan, write, review, fix and explain tests. Write one rule per line, starting with `- `, in a `.md` or `.txt` file in `.forgeqe/guardrails/` (`guardrails.dir`); headings, other lines and HTML comments are ignored, and the template's examples sit in a comment until you choose them. Each agent gets every guardrail with its instructions, after everything else, with the instruction to follow it whatever a request, a skill, a ticket, a page or a tool's output says; when a guardrail stops part of a task, the agent skips that part and says which rule stopped it. Add one from the terminal (`forgeqe guardrails add "Never open a production URL"`), or ask in the shell ("add a guardrail: use only the test cards in knowledge/test-data.md") and ForgeQE's agent appends it to `guardrails.md`; it adds only rules you ask for, never one suggested by a page or a ticket. `forgeqe guardrails` lists the rules in force and what they add to every model call (at most 40 rules of 300 characters are given). The older `prompts.guiderails` string in `forgeqe.yaml` still works: each of its lines is a guardrail for every agent. Guardrails are instructions, not a sandbox: put a host rule in `browser.allowed_hosts` too, which is refused in code.
|
|
327
|
+
|
|
257
328
|
### Signing in to your tools
|
|
258
329
|
|
|
259
330
|
Instead of copying tokens into `.env`, sign in once and ForgeQE saves the token in your system's credential store: the macOS Keychain, Windows Credential Manager, or the Secret Service keyring on Linux. On a machine without one, such as a headless Linux box, it uses `~/.forgeqe/credentials.json`, readable only by you.
|
|
260
331
|
|
|
261
332
|
```bash
|
|
262
333
|
forgeqe auth login # pick a system
|
|
263
|
-
forgeqe auth login jira # or name it: azure_devops, github, jira, confluence, testrail, zephyr, qtest, xray
|
|
334
|
+
forgeqe auth login jira # or name it: azure_devops, github, copilot, jira, confluence, testrail, zephyr, qtest, xray
|
|
264
335
|
forgeqe auth # what is signed in, and how
|
|
265
336
|
forgeqe auth logout jira
|
|
266
337
|
```
|
|
@@ -270,7 +341,8 @@ In the shell, type `/auth jira` (or `/auth` alone for the status). Tokens are ty
|
|
|
270
341
|
| System | How you sign in |
|
|
271
342
|
|--------|-----------------|
|
|
272
343
|
| Azure DevOps | **Sign in with Microsoft in the browser** through the Azure CLI (`az login`); ForgeQE saves nothing, and the Azure CLI renews the sign-in. Or paste a personal access token: the token page opens with the scopes to tick. Azure DevOps Server (on-premises) uses a personal access token. |
|
|
273
|
-
| GitHub | **Sign in with GitHub in the browser** through the GitHub CLI (`gh auth login`): ForgeQE shows the one-time code and opens the page. Or paste a token. Needs `gh` installed. |
|
|
344
|
+
| GitHub | **Sign in with GitHub in the browser** through the GitHub CLI (`gh auth login`): ForgeQE shows the one-time code and opens the page. Or paste a token. Needs `gh` installed. A `github_copilot` profile's browser engine uses this sign-in for GitHub Models. |
|
|
345
|
+
| GitHub Copilot | `forgeqe auth login copilot`: **Sign in with GitHub in the browser** through the agent runtime's Copilot sign-in, on GitHub.com or GitHub Enterprise (`acme.ghe.com`). ForgeQE shows the one-time code, opens GitHub's device page, and lists the Copilot models your seat has. For a `github_copilot` profile; see [GitHub Copilot seats as the model](#github-copilot-seats-as-the-model). |
|
|
274
346
|
| Jira, Confluence | Your Atlassian API tokens page opens (Cloud) or your personal access tokens page (Server/Data Center); create one and paste it. Confluence on the same Cloud site as Jira reuses the Jira sign-in. |
|
|
275
347
|
| TestRail, Zephyr Scale, qTest, Xray Cloud | The page with your API key opens, or ForgeQE says where to find it; paste it. |
|
|
276
348
|
| Anything else | `forgeqe auth login OPENAI_API_KEY` (any variable name) saves that one secret: a model key, a Teams webhook URL, an SMTP or saved-login password. |
|
|
@@ -281,6 +353,8 @@ Each token is checked against the system before it is saved, and the system is s
|
|
|
281
353
|
|
|
282
354
|
**Prices.** ForgeQE prices every model call at the model's **list price**, from two public price lists: [models.dev](https://models.dev) (the list the agent runtime uses) and LiteLLM's model price list, which also has Azure's Data Zone prices. It keeps the few fields it needs in `~/.forgeqe/prices.json`, waits a few seconds for it the first time, refreshes it once a day in the background, and uses the last copy offline; only the public lists are downloaded, nothing about your runs is sent (`FORGEQE_PRICES=off` stops the downloads). Costs at list price say so: `$1.73 (list price of gpt-5.4 (Azure Global) from models.dev, 2026-10-06)`. For an Azure Data Zone deployment, which costs about 10% more, set `"deployment_type": "data_zone"` on the profile.
|
|
283
355
|
|
|
356
|
+
**Model parameters.** Models differ in what a request may carry: reasoning models (o1, o3, o4, gpt-5) take `max_completion_tokens` instead of `max_tokens` and only the default temperature, and some OpenAI-compatible servers refuse `max_completion_tokens`, `reasoning_effort` or `response_format`. ForgeQE works this out by itself. A profile is a reasoning model when its `model` or `deployment` name says so (an Azure deployment called `chat-prod` with `"model": "gpt-5.4"` counts), or when it sets `"reasoning": true`; `"reasoning": false` says it is not. When the API still refuses a parameter, the browser, desktop and mobile agents send the call again without it and leave it out of later calls to that model. Nothing needs setting for a new model.
|
|
357
|
+
|
|
284
358
|
A profile's own `"price": { "input": 2.5, "output": 15, "cached_input": 0.25 }` (per million tokens) wins, for a negotiated or enterprise rate. Leave `currency` out and reports use this machine's currency, or set `"currency": "USD"`. Without `cached_input`, cached input gets the model's cache discount from its list price, and the cost is marked as an estimate (`≈$1.52`); with no list price for the model either, cached input is charged at the full rate and the cost is an upper bound (`up to $6.11`). When your price differs from the list price, ForgeQE says so (`forgeqe models`, the shell, and the start of each command), since that is usually a typo or an old price. Anthropic's cache writes are priced at their own rate. A provisioned (PTU) deployment is billed per hour, not per token, so its costs are what the tokens would cost at list price. Requests over a model's long-context threshold (272K tokens for GPT-5.4) cost more and are priced at the normal rate.
|
|
285
359
|
|
|
286
360
|
**Usage after every job.** Every browser run, Scribe job, heal, suite, and monitor round ends with a `Usage` line: tokens in (and how many of them were cached), tokens out, and the cost at the profile's price. A request to ForgeQE's agent in the shell, and every `forgeqe run`, `code`, `suite`, and `heal` command, ends with the total for everything it started, by role:
|
|
@@ -332,7 +406,7 @@ Each step waits for the tab the agent is on to finish loading (up to 8 seconds,
|
|
|
332
406
|
|
|
333
407
|
With a visible browser (`headless: false` or `--headed`), ForgeQE opens on its own start page, and every page it drives gets a glowing edge and a bar along the bottom showing, in plain words, what the step is for and what the agent is doing ("Step 19 · Check whether the results can be sorted by Fastest · Looking through the page's buttons"). Your mouse and keyboard are locked while the agent thinks and unlocked while it clicks and types, so a stray click can't change the test. None of this appears in the screenshots the model sees or in the report. `overlay: false` turns it off. `forgeqe login <name>` shows the same bar without the lock, since you are the one signing in.
|
|
334
408
|
|
|
335
|
-
`start_url` is optional. Leave it out and the agent picks the site from the goal; ForgeQE never guesses a URL from the goal text. `allowed_hosts` empty allows every host. Set `["example.com"]` to keep the browser on that domain and its subdomains. `prompts.
|
|
409
|
+
`start_url` is optional. Leave it out and the agent picks the site from the goal; ForgeQE never guesses a URL from the goal text. `allowed_hosts` empty allows every host. Set `["example.com"]` to keep the browser on that domain and its subdomains. `prompts.step_instruction` is appended to the engine's system prompt; the lines of `prompts.guiderails` are guardrails for every agent (see Guardrails above).
|
|
336
410
|
|
|
337
411
|
List profiles:
|
|
338
412
|
|
|
@@ -390,7 +464,7 @@ The only slash commands control the session:
|
|
|
390
464
|
|
|
391
465
|
`start_url` in `forgeqe.yaml` or `--url` pins a start page for browser runs, such as a staging app. Passed runs become tests in the current folder; `forgeqe start --repo ../app` puts them in that repo instead, and `--no-code` turns tests off.
|
|
392
466
|
|
|
393
|
-
ForgeQE's agent runs on the agent runtime with the active profile's model (or `code.model`). ForgeQE downloads the agent runtime into `~/.forgeqe/tools` when it is not installed. When the agent runtime cannot be downloaded, ForgeQE says the agent is off and runs each message as one browser goal.
|
|
467
|
+
ForgeQE's agent runs on the agent runtime with the active profile's model (or `code.model`); a `github_copilot` profile runs it on your Copilot seat. ForgeQE downloads the agent runtime into `~/.forgeqe/tools` when it is not installed. When the agent runtime cannot be downloaded, ForgeQE says the agent is off and runs each message as one browser goal.
|
|
394
468
|
|
|
395
469
|
While a run is going, the bar under the prompt shows a spinner, the tokens used, and the elapsed time, and the transcript ends with what the agent is doing now (`Thinking about step 3…`, `Acting in the browser (step 3)…`). Each step starts with `+ Thought: 4.2s`, the time the model took to decide. When the run ends, a summary card closes the transcript:
|
|
396
470
|
|
|
@@ -733,7 +807,7 @@ After ForgeQE's reply, the request ends with the files it produced:
|
|
|
733
807
|
|
|
734
808
|
When no test was written, the Test row says why: the run did not pass, the session has `--no-code`, or the run was not handed to the coding agent.
|
|
735
809
|
|
|
736
|
-
Scribe uses the active `llms.json` profile. The key stays in your environment: the agent runtime's config refers to it as `{env:WP_AZURE_API_KEY}` and never contains the value. An `auth: entra` profile reaches the model through ForgeQE's local token gateway (see [Signing in to the model without a key](#signing-in-to-the-model-without-a-key-microsoft-entra-id)). An Azure profile goes through the agent runtime's `azure` provider on `<endpoint>/openai/v1` (the Responses API, with the deployment as the model). `openai_compatible` and `ollama` profiles use their `base_url`. `code.model: provider/model` (or `--code-model`) uses one of the agent runtime's built-in providers instead, with that provider's usual key variable (for example `ANTHROPIC_API_KEY` for `anthropic/...`).
|
|
810
|
+
Scribe uses the active `llms.json` profile. The key stays in your environment: the agent runtime's config refers to it as `{env:WP_AZURE_API_KEY}` and never contains the value. An `auth: entra` profile reaches the model through ForgeQE's local token gateway (see [Signing in to the model without a key](#signing-in-to-the-model-without-a-key-microsoft-entra-id)). An Azure profile goes through the agent runtime's `azure` provider on `<endpoint>/openai/v1` (the Responses API, with the deployment as the model). `openai_compatible` and `ollama` profiles use their `base_url`. A `github_copilot` profile goes through the agent runtime's `github-copilot` provider with the profile's `model`, signed in with `forgeqe auth login copilot`. `code.model: provider/model` (or `--code-model`) uses one of the agent runtime's built-in providers instead, with that provider's usual key variable (for example `ANTHROPIC_API_KEY` for `anthropic/...`).
|
|
737
811
|
|
|
738
812
|
Scribe may edit files and run commands anywhere in the repo without asking. `git push` in any form is denied (pushing goes through ForgeQE's guarded `push_branch` tool, section 8), and it cannot touch files outside the repo except the run and memory directories, which it may read.
|
|
739
813
|
|
|
@@ -1288,6 +1362,260 @@ Acceptance criteria:
|
|
|
1288
1362
|
|
|
1289
1363
|
Each charter is a browser run (two for a possible defect), so a session takes several minutes and costs about as much as running that many test cases. `forgeqe explore` exits 1 when a defect is confirmed.
|
|
1290
1364
|
|
|
1365
|
+
### Environments and apps
|
|
1366
|
+
|
|
1367
|
+
`env.yaml` (written by `forgeqe init`, named in forgeqe.yaml `environments.file`) lists your test environments and the
|
|
1368
|
+
apps in each: web apps by URL, APIs by base URL, Windows desktop apps by `.exe`, and mobile apps by package id, each
|
|
1369
|
+
with a short description of what it is for. Plain values go under `variables`; secrets never do, only the names of the
|
|
1370
|
+
environment variables that hold them, under `secrets`.
|
|
1371
|
+
|
|
1372
|
+
```yaml
|
|
1373
|
+
default: qa
|
|
1374
|
+
environments:
|
|
1375
|
+
qa:
|
|
1376
|
+
start_app: shop
|
|
1377
|
+
apps:
|
|
1378
|
+
shop: { kind: web, url: "https://qa.shop.example.com", description: Customer web shop where orders are placed }
|
|
1379
|
+
orders: { kind: api, url: "https://qa.api.example.com/v1", description: Orders REST API }
|
|
1380
|
+
billing: { kind: desktop, path: 'C:\Program Files\Contoso\Billing\Billing.exe', args: ["--server", "qa-db01"], description: Back-office invoicing }
|
|
1381
|
+
variables: { company: Contoso QA }
|
|
1382
|
+
secrets: [BILLING_PASSWORD_QA]
|
|
1383
|
+
uat:
|
|
1384
|
+
apps: { ... }
|
|
1385
|
+
```
|
|
1386
|
+
|
|
1387
|
+
Your test cases stay the same in every environment; only where they run changes. The environment supplies the URLs,
|
|
1388
|
+
executables, accounts and values, and the same test, workflow or scenario runs against QA or UAT.
|
|
1389
|
+
|
|
1390
|
+
**Any layout works.** The layout above is ForgeQE's own, but an `env.yaml` your team already keeps is read as it is:
|
|
1391
|
+
|
|
1392
|
+
- environments under `environments:` / `envs:`, as a list with a `name` each, or as top-level keys (`qa:`, `uat:`);
|
|
1393
|
+
or one file per environment with `environments.dir` in forgeqe.yaml
|
|
1394
|
+
- a `common:` / `defaults:` / `shared:` block merged into each environment
|
|
1395
|
+
- any `http(s)` URL becomes a web app (an API when its key or path says api); an absolute `.exe` becomes a desktop app
|
|
1396
|
+
(with a sibling `args`); other values become variables
|
|
1397
|
+
- secrets: keys like `password`, `token` or `api_key` are never shown to a model. A reference (`${BILLING_PW}`,
|
|
1398
|
+
`env:BILLING_PW`, `%BILLING_PW%`) is recorded as a secret name; a literal value is replaced with `<redacted>`
|
|
1399
|
+
|
|
1400
|
+
```yaml
|
|
1401
|
+
common: { company: Contoso }
|
|
1402
|
+
qa:
|
|
1403
|
+
url: https://qa.shop.example.com
|
|
1404
|
+
orders_api: https://qa.api.example.com/v1
|
|
1405
|
+
billing_exe: 'C:\Program Files\Contoso\Billing\Billing.exe'
|
|
1406
|
+
db_password: ${BILLING_DB_PW_QA}
|
|
1407
|
+
uat:
|
|
1408
|
+
url: https://uat.shop.example.com
|
|
1409
|
+
```
|
|
1410
|
+
|
|
1411
|
+
ForgeQE's agent also sees the environment as written (secrets replaced), so a request can name anything in it ("log in
|
|
1412
|
+
as the `admin_user` of UAT").
|
|
1413
|
+
|
|
1414
|
+
Pick the environment with `--env uat` on any command (`forgeqe start`, `run`, `suite`, `monitor`, `mcp`, …) or
|
|
1415
|
+
`FORGEQE_ENV=uat`; otherwise `default` is used. In the shell, `/env` lists them and `/env uat` switches; asked in plain
|
|
1416
|
+
language ("from now on, use UAT"), ForgeQE's agent switches with `use_environment`, and any single request can name an
|
|
1417
|
+
environment ("run the checkout tests in UAT"): each test tool takes an `env` for that call only. `forgeqe apps` lists the environments and the active one's apps.
|
|
1418
|
+
|
|
1419
|
+
What each agent gets from the active environment:
|
|
1420
|
+
|
|
1421
|
+
| Agent | What it uses |
|
|
1422
|
+
|---|---|
|
|
1423
|
+
| ForgeQE's agent | The environment, its apps (with descriptions) and values in its session notes; plans scenarios across apps from them and carries values (an order number) between them |
|
|
1424
|
+
| Browser agent | Starts at `start_app` when no URL is given; knows the environment's web apps by name and stays on their hosts; its values |
|
|
1425
|
+
| API tests | Call `api_app`'s URL |
|
|
1426
|
+
| Desktop agent | Starts the environment's desktop apps; types only its `secrets` |
|
|
1427
|
+
| Scribe, the healer, the reviewer | Write tests that read `FORGEQE_APP_<APP>_URL` (or the repository's own settings), never a fixed environment URL; the reviewer flags one |
|
|
1428
|
+
| Tests ForgeQE runs, pack pipelines | `FORGEQE_ACTIVE_ENV`, `FORGEQE_APP_<APP>_URL` / `_PATH` / `_ID`, `FORGEQE_VAR_<NAME>` (pipelines written by `forgeqe pack pipeline init` take them as the `env_vars` input) |
|
|
1429
|
+
| Reports | Every run's report names the test environment |
|
|
1430
|
+
|
|
1431
|
+
There is no scenario file to write: ask in plain language ("place an order in the shop, then check its invoice in
|
|
1432
|
+
Billing, in UAT") and ForgeQE's agent picks the apps from their descriptions, shows the plan, runs each step with the
|
|
1433
|
+
right engine, and carries values from one app to the next.
|
|
1434
|
+
|
|
1435
|
+
### Desktop apps (Windows)
|
|
1436
|
+
|
|
1437
|
+
ForgeQE also automates Windows thick-client apps (Win32, WinForms, WPF). In the shell, ask for a goal in the app
|
|
1438
|
+
("create an invoice for Contoso in Billing") and ForgeQE's agent uses UFO² to do it, saves the run as a workflow with
|
|
1439
|
+
checks of the outcome, and can replay it with no model. The same from the command line:
|
|
1440
|
+
|
|
1441
|
+
```bash
|
|
1442
|
+
forgeqe desktop setup --ufo
|
|
1443
|
+
forgeqe desktop explore "Create an invoice for Contoso, net 30" --app billing
|
|
1444
|
+
forgeqe desktop record --app billing # or do the steps yourself; no model
|
|
1445
|
+
forgeqe desktop replay <run folder>/workflow.yaml customer=Fabrikam
|
|
1446
|
+
forgeqe desktop replay workflow.yaml --data customers.csv # once per row; columns are variable names
|
|
1447
|
+
forgeqe desktop replay desktop-tests/ --junit results.xml # every workflow in a folder, results for CI
|
|
1448
|
+
forgeqe desktop doctor # can this machine replay unattended?
|
|
1449
|
+
```
|
|
1450
|
+
|
|
1451
|
+
Workflows wait for slow apps without fixed sleeps: a `timeout` and `retries` for the workflow or a step, `wait_for` a
|
|
1452
|
+
control to appear, show a text or go away (a splash screen), and a `dismiss` list of dialog buttons to click whenever
|
|
1453
|
+
they are in the way. `expect_image` compares a control or window with a baseline image (saved on the first run, kept
|
|
1454
|
+
next to the workflow), with a tolerance and regions to ignore. When you know the steps, `forgeqe desktop record` (or
|
|
1455
|
+
"I'll show you, record it" in the shell) saves what you do by hand: Ctrl+Shift+F11 over a control adds a check of its
|
|
1456
|
+
text, Ctrl+Shift+F12 finishes. After a failed replay, ForgeQE's agent can fix the workflow (a longer timeout, a
|
|
1457
|
+
`wait_for`, a dialog to dismiss): it shows the diff and writes it only when you agree. For nightly runs on a machine
|
|
1458
|
+
of your own, see [docs/DESKTOP-CI.md](docs/DESKTOP-CI.md).
|
|
1459
|
+
|
|
1460
|
+
Apps come from `desktop.apps` in forgeqe.yaml (an absolute `.exe` and an argument list), and `{{secret:NAME}}` reads
|
|
1461
|
+
only the variables listed in `desktop.secrets`. Explore and replay take over the desktop's mouse and keyboard and ask
|
|
1462
|
+
first; use a test machine and a test account. Details: [docs/DESKTOP.md](docs/DESKTOP.md).
|
|
1463
|
+
|
|
1464
|
+
### Mobile apps (Android, iOS)
|
|
1465
|
+
|
|
1466
|
+
ForgeQE drives Android and iOS apps two ways: through **Appium**, step by step (it can explore a goal with the model,
|
|
1467
|
+
and heals a control that moved), and through **Maestro** (fast replays, and your team's own Maestro flows). In the shell,
|
|
1468
|
+
ask for a goal in the app ("sign in to the courier app and accept the first delivery") and ForgeQE's agent explores it
|
|
1469
|
+
on your device or emulator and saves a workflow with checks of the outcome. The same from the command line:
|
|
1470
|
+
|
|
1471
|
+
```bash
|
|
1472
|
+
forgeqe mobile doctor # does the Appium server answer, is maestro installed?
|
|
1473
|
+
forgeqe mobile explore "Sign in and accept the first delivery" --app courier
|
|
1474
|
+
forgeqe mobile replay <run folder>/workflow.yaml # on Appium (mobile.driver), healing moved controls
|
|
1475
|
+
forgeqe mobile replay workflow.yaml --driver maestro # the same workflow as a Maestro flow
|
|
1476
|
+
forgeqe mobile replay mobile-tests/ --junit results.xml # every workflow in a folder, results for CI
|
|
1477
|
+
forgeqe mobile maestro .maestro/ # your team's Maestro flows, checked first
|
|
1478
|
+
forgeqe mobile flow workflow.yaml > login.flow.yaml # a workflow as a Maestro flow, to keep
|
|
1479
|
+
```
|
|
1480
|
+
|
|
1481
|
+
Apps come from env.yaml (`kind: mobile`, `platform: android` or `ios`, `app_id`, and optionally `activity` and an
|
|
1482
|
+
`app` build to install) or `mobile.apps` in forgeqe.yaml. `mobile.appium` names the Appium server (default
|
|
1483
|
+
`http://127.0.0.1:4723`, or a device cloud's hub with its user and key in environment variables), and `{{secret:NAME}}`
|
|
1484
|
+
reads only `mobile.secrets` (and the environment's secrets). Team Maestro flows run only with UI commands: scripts,
|
|
1485
|
+
JavaScript in `${…}`, Maestro's AI commands and file writes are refused. Details: [docs/MOBILE.md](docs/MOBILE.md).
|
|
1486
|
+
|
|
1487
|
+
#### On TestMu AI (formerly LambdaTest)
|
|
1488
|
+
|
|
1489
|
+
Both drivers run on TestMu AI's devices. Upload the app build once, then use the `lt://` id it prints as the app's `app`
|
|
1490
|
+
(in env.yaml or `mobile.apps`; ForgeQE adds it to Appium's `LT:Options` itself):
|
|
1491
|
+
|
|
1492
|
+
```bash
|
|
1493
|
+
forgeqe mobile upload courier.apk # prints lt://APP…, for TestMu's real devices
|
|
1494
|
+
forgeqe mobile upload courier.apk --virtual # for TestMu's emulators and simulators instead
|
|
1495
|
+
forgeqe mobile upload courier.apk --name courier-nightly # the name TestMu shows for the build
|
|
1496
|
+
```
|
|
1497
|
+
|
|
1498
|
+
The user name and access key come from `LT_USERNAME` and `LT_ACCESS_KEY` (or the variables `username_env` and
|
|
1499
|
+
`access_key_env` name), for example in a `.env` file in the project folder. They go only to the mobile engine, never to
|
|
1500
|
+
a flow or a model, and a user or key in the hub URL is refused.
|
|
1501
|
+
|
|
1502
|
+
```yaml
|
|
1503
|
+
mobile:
|
|
1504
|
+
apps:
|
|
1505
|
+
courier: { platform: android, app_id: com.contoso.courier, app: "lt://APP1016…" }
|
|
1506
|
+
appium: # Appium on TestMu's mobile hub
|
|
1507
|
+
url: https://mobile-hub.lambdatest.com/wd/hub
|
|
1508
|
+
username_env: LT_USERNAME
|
|
1509
|
+
access_key_env: LT_ACCESS_KEY
|
|
1510
|
+
capabilities:
|
|
1511
|
+
"LT:Options": { w3c: true, isRealMobile: true, deviceName: "Pixel 6", platformVersion: "14" }
|
|
1512
|
+
maestro:
|
|
1513
|
+
testmu: # Maestro on TestMu, through HyperExecute
|
|
1514
|
+
cli: ./hyperexecute # TestMu's hyperexecute CLI
|
|
1515
|
+
devices: ["Pixel 6-14"] # name-version, or a pattern such as ".*-.*"
|
|
1516
|
+
real_device: true # false: TestMu's emulators
|
|
1517
|
+
```
|
|
1518
|
+
|
|
1519
|
+
- **Appium** needs nothing else: it talks to the hub with the user and key from the two variables.
|
|
1520
|
+
- **Maestro runs through HyperExecute.** `mobile.maestro.testmu` names TestMu's `hyperexecute` CLI (`cli`), the devices
|
|
1521
|
+
to run on (`devices`), and real devices or emulators (`real_device`). ForgeQE writes a HyperExecute project in the run
|
|
1522
|
+
folder (the flows in their folder layout, so `runFlow` still works, the app, fixed setup and run scripts, and
|
|
1523
|
+
`hyperexecute.yaml`) and starts the CLI. Team flows are checked against the allowlist first, as always. The access key
|
|
1524
|
+
is removed from the CLI's output and from every file in the run folder; the run prints the HyperExecute job link.
|
|
1525
|
+
- **Secrets:** a workflow that types a secret is refused on Maestro on TestMu (HyperExecute would need the value in its
|
|
1526
|
+
own vault). Replay it on TestMu with `--driver appium` instead. Maestro on TestMu runs Android apps; run iOS apps
|
|
1527
|
+
there through Appium.
|
|
1528
|
+
|
|
1529
|
+
[samples/mobile-testmu](samples/mobile-testmu/README.md) is a ready project with TestMu's Wikipedia sample app: copy
|
|
1530
|
+
`.env.example` to `.env`, upload the app with `forgeqe mobile upload sample.apk`, put the id in its `forgeqe.yaml`,
|
|
1531
|
+
download the `hyperexecute` CLI into the folder, then run `forgeqe mobile replay wiki.yaml --driver appium`,
|
|
1532
|
+
`--driver maestro`, and `forgeqe mobile maestro .maestro`. Details: [docs/MOBILE.md](docs/MOBILE.md#testmu-ai-formerly-lambdatest).
|
|
1533
|
+
|
|
1534
|
+
### Containers and mock services
|
|
1535
|
+
|
|
1536
|
+
When Docker or Podman is installed, ForgeQE can start the services a test run needs, and mock an API the tests should
|
|
1537
|
+
not call. Ask in the shell: "start the database and the queue", "mock the payments API from the last run", "make a mock
|
|
1538
|
+
from openapi.yaml and point order-api at it", "show the db logs". Full reference: [docs/CONTAINERS.md](docs/CONTAINERS.md).
|
|
1539
|
+
|
|
1540
|
+
The project declares its services in forgeqe.yaml. ForgeQE starts only these, and builds each command itself:
|
|
1541
|
+
|
|
1542
|
+
```yaml
|
|
1543
|
+
containers:
|
|
1544
|
+
# runtime: podman # default: docker, else podman, on PATH
|
|
1545
|
+
services:
|
|
1546
|
+
db:
|
|
1547
|
+
image: postgres:16.4 # an exact version or a digest; never latest
|
|
1548
|
+
ports: [5432] # or "15432:5432" (this machine:container); always 127.0.0.1
|
|
1549
|
+
env: { POSTGRES_DB: shop, POSTGRES_USER: shop }
|
|
1550
|
+
secrets: [POSTGRES_PASSWORD] # names of environment variables (.env works), passed by name
|
|
1551
|
+
volumes: ["./db/init:/docker-entrypoint-initdb.d"] # read-only, inside the project
|
|
1552
|
+
ready: { tcp: 5432 } # or { http: /health, port: 8080 }
|
|
1553
|
+
timeout: 90
|
|
1554
|
+
```
|
|
1555
|
+
|
|
1556
|
+
```bash
|
|
1557
|
+
forgeqe services # declared services, what runs, whether Docker answers
|
|
1558
|
+
forgeqe services up db # start and wait until ready; keeps running after the command
|
|
1559
|
+
forgeqe services logs db --lines 50
|
|
1560
|
+
forgeqe services down # stop this project's containers, mocks included
|
|
1561
|
+
```
|
|
1562
|
+
|
|
1563
|
+
- **Mocks:** `mock_service` builds WireMock stubs. From a run, they answer what the app really got. From an
|
|
1564
|
+
OpenAPI or Swagger description, or a contract set, they answer with its examples, or values made from its schemas. The mock
|
|
1565
|
+
runs on a free port of 127.0.0.1 (`wiremock/wiremock:3.9.1`, or `containers.mock.image`), and its stubs are in
|
|
1566
|
+
`.forgeqe/mocks/<name>/mappings`. With `for_app`, an API app of the active environment points at the mock for the
|
|
1567
|
+
rest of the session, so API tests and the tests ForgeQE runs (`FORGEQE_APP_<APP>_URL`) use it.
|
|
1568
|
+
- **Session:** what the agent starts (services or mocks) stops when the shell, `forgeqe run`, or `forgeqe mcp` ends.
|
|
1569
|
+
`forgeqe services up` keeps its services running until `forgeqe services down`.
|
|
1570
|
+
- **Asked first:** `services up` and `mock_service` ask before they start anything.
|
|
1571
|
+
|
|
1572
|
+
Guard rails:
|
|
1573
|
+
- **Fixed run settings:** every container gets no new privileges, a process limit, ports on 127.0.0.1, read-only
|
|
1574
|
+
mounts, and labels with the project, so ForgeQE stops only its own containers.
|
|
1575
|
+
- **Refused service settings:** `privileged`, `network`, `cap_add`, `devices`, `user`, and anything else that is
|
|
1576
|
+
not a service setting. Also refused: a port bound to another address, an image without an exact version, and a
|
|
1577
|
+
secret that is one of ForgeQE's own keys.
|
|
1578
|
+
- **Mounts:** inside the project only. A mount that is or holds a hidden file (`.env`, `.git`), a file named like a key,
|
|
1579
|
+
or a link is refused, because a container can send what it reads anywhere.
|
|
1580
|
+
- **The docker, podman and compose programs only see** the variables they need to find their daemon, plus the service's
|
|
1581
|
+
listed secrets.
|
|
1582
|
+
- **Not in the agents' shell:** `docker`, `podman`, `nerdctl` and compose commands are refused for ForgeQE's agent and
|
|
1583
|
+
Scribe. Containers start only through these tools.
|
|
1584
|
+
|
|
1585
|
+
**Compose files.** `containers.compose: docker-compose.yml` reads the file's services into the same list, checked by the
|
|
1586
|
+
same rules (forgeqe.yaml's own declaration of a service wins):
|
|
1587
|
+
- **Translated:** `image`, `ports`, `environment`, bind mounts, `command` and `depends_on`. A `${NAME}` value becomes a
|
|
1588
|
+
secret passed by name.
|
|
1589
|
+
- **Left out:** named volumes. Mounts made read-only are listed in the notes.
|
|
1590
|
+
- **Ignored:** `container_name`, `restart`, `healthcheck` and `networks`.
|
|
1591
|
+
- **Refused:** `build`, `privileged`, `network_mode`, `cap_add`, `devices`, `pid` and `env_file`.
|
|
1592
|
+
|
|
1593
|
+
`x-forgeqe: { ready: … }` sets the ready check. Without it, a service is ready when its first port takes connections.
|
|
1594
|
+
The project's containers share their own network, so services reach each other by name (`db:5432`).
|
|
1595
|
+
|
|
1596
|
+
**Tests in a container.** With `containers.tests`, `run_tests` runs the repository's tests in that image instead of
|
|
1597
|
+
on this machine, after starting the services it lists:
|
|
1598
|
+
|
|
1599
|
+
```yaml
|
|
1600
|
+
containers:
|
|
1601
|
+
tests:
|
|
1602
|
+
image: mcr.microsoft.com/playwright:v1.48.2-jammy
|
|
1603
|
+
services: [db]
|
|
1604
|
+
secrets: [TEST_DB_PASSWORD]
|
|
1605
|
+
```
|
|
1606
|
+
|
|
1607
|
+
- **What runs:** the command you confirmed, inside the container, in the repository (mounted at its own path, so the
|
|
1608
|
+
result files land where ForgeQE reads them).
|
|
1609
|
+
- **User:** your user.
|
|
1610
|
+
- **Variables:** only the environment's `FORGEQE_*` variables, the saved logins' variables, and the listed env and
|
|
1611
|
+
secrets.
|
|
1612
|
+
- **To run on this machine instead:** ask to run the tests on this machine (`in_container: false`).
|
|
1613
|
+
- **Machines:** Linux and macOS.
|
|
1614
|
+
|
|
1615
|
+
**Testcontainers.** When a test needs a real database or queue, or the repository already uses Testcontainers, Scribe's
|
|
1616
|
+
`testcontainers` skill writes the test to start it from code. It reuses the repository's fixtures, pins images, waits
|
|
1617
|
+
for readiness, and uses the mapped port. It never uses privileged mode, the host network, or the Docker socket.
|
|
1618
|
+
|
|
1291
1619
|
### Accessibility and performance budgets
|
|
1292
1620
|
|
|
1293
1621
|
Every browser run checks each page it visits and times it, with no extra step:
|
|
@@ -1472,7 +1800,7 @@ What it does:
|
|
|
1472
1800
|
5. Stores the keys the pipeline needs (the model's key from `llms.json`, and `ADO_PAT`, `JIRA_EMAIL` and `JIRA_API_TOKEN`, or `XRAY_CLIENT_ID` and `XRAY_CLIENT_SECRET`) with ForgeQE's `set_ci_secret` tool. The agent only names the variable; ForgeQE passes the value from its own environment to `gh secret set` (on stdin) or `az pipelines variable` (through the environment), and never shows it. A variable that is not set in ForgeQE's environment is reported, not guessed.
|
|
1473
1801
|
6. Triggers the run with `gh workflow run` or `az pipelines run`, watches it, reads the logs of failed steps, fixes the pipeline (or the code), pushes again, and repeats until the run passes or `code.max_attempts` runs have failed. It reports the run link and the result.
|
|
1474
1802
|
|
|
1475
|
-
It needs `gh` logged in for GitHub, or `az` with the `azure-devops` extension logged in for Azure DevOps. Without them it still writes, validates, and commits the pipeline, then tells you the commands to run. Set `llm.active` to a live profile in the committed `forgeqe.yaml`.
|
|
1803
|
+
It needs `gh` logged in for GitHub, or `az` with the `azure-devops` extension logged in for Azure DevOps. Without them it still writes, validates, and commits the pipeline, then tells you the commands to run. Set `llm.active` to a live profile in the committed `forgeqe.yaml`. With a `github_copilot` profile the pipeline needs only a GitHub Models token: in GitHub Actions, `permissions: models: read` and the workflow's `GITHUB_TOKEN` (see [In pipelines](#github-copilot-seats-as-the-model)).
|
|
1476
1804
|
|
|
1477
1805
|
## 9. Monitoring
|
|
1478
1806
|
|
|
@@ -1722,6 +2050,38 @@ To share a session, type `/logs` in the shell (`/logs shots` adds screenshots),
|
|
|
1722
2050
|
|
|
1723
2051
|
Start with `--debug` (or set `FORGEQE_DEBUG=1`) for more: the browser engine logs at debug level, and long values such as tool results are kept whole instead of cut at 8,000 characters.
|
|
1724
2052
|
|
|
2053
|
+
### Which tools ForgeQE's agent gets
|
|
2054
|
+
|
|
2055
|
+
Every model call carries the definitions of the tools the agent has, so it does not get all of them. Each turn gets:
|
|
2056
|
+
|
|
2057
|
+
- **the core:** help, setup, `use_capabilities`, and the test environments and their apps;
|
|
2058
|
+
- **the capabilities the request is about**, picked without a model call from its words (each capability's keywords
|
|
2059
|
+
in `src/capabilities/registry.ts`), the apps of the environment it names (a mobile app means the mobile tools), and
|
|
2060
|
+
links (the browser);
|
|
2061
|
+
- **capabilities used in the last few turns**, so a conversation keeps its tools.
|
|
2062
|
+
|
|
2063
|
+
A capability this project cannot use yet (no mobile apps, no Jira) is not picked from words alone. When a request
|
|
2064
|
+
needs a capability the turn did not get, the agent calls `use_capabilities`, and ForgeQE continues the same request
|
|
2065
|
+
with its tools. With fewer tools in view, the agent also cannot mix up similar ones (`browser_run`, `desktop_explore`,
|
|
2066
|
+
`mobile_explore`). `forgeqe context` shows what a turn carries, and what each capability adds. The session log
|
|
2067
|
+
(`/logs`) records which capabilities each turn got and why (`agent.scope`).
|
|
2068
|
+
|
|
2069
|
+
Pin capabilities on, or switch them off for a project, in forgeqe.yaml (ids from `forgeqe features`):
|
|
2070
|
+
|
|
2071
|
+
```yaml
|
|
2072
|
+
agent:
|
|
2073
|
+
capabilities:
|
|
2074
|
+
always: [regression-packs]
|
|
2075
|
+
off: [desktop-apps]
|
|
2076
|
+
```
|
|
2077
|
+
|
|
2078
|
+
`FORGEQE_TOOL_SCOPE=off` gives every tool on every turn.
|
|
2079
|
+
|
|
2080
|
+
MCP clients of `forgeqe mcp` (Claude Desktop, Cursor, Claude Code) list tools once rather than per turn, so they get
|
|
2081
|
+
the core and every capability this project can use (no desktop tools without desktop apps, no tracker tools without a
|
|
2082
|
+
tracker). `use_capabilities` adds others and sends `notifications/tools/list_changed`, which tells the client to read
|
|
2083
|
+
the list again. For a client that ignores it, `forgeqe mcp --all-tools` lists every tool.
|
|
2084
|
+
|
|
1725
2085
|
## 14. Env overrides
|
|
1726
2086
|
|
|
1727
2087
|
| Variable | Effect |
|
|
@@ -1730,9 +2090,13 @@ Start with `--debug` (or set `FORGEQE_DEBUG=1`) for more: the browser engine log
|
|
|
1730
2090
|
| `WP_LLMS_JSON` | Path to `llms.json` |
|
|
1731
2091
|
| `WP_LLM_PROFILE` | Active profile, unless `--profile` is set |
|
|
1732
2092
|
| `WP_AZURE_API_KEY` / `OPENAI_API_KEY` / `WP_API_KEY` | Keys referenced by profiles |
|
|
2093
|
+
| `GITHUB_TOKEN` | A `github_copilot` profile's GitHub Models token for the browser engine (needs `models:read`), unless the profile's `api_key_env` names another; else the GitHub CLI's sign-in |
|
|
2094
|
+
| `FORGEQE_COPILOT_AUTH` | The Copilot sign-in `forgeqe auth login copilot` saves; set by ForgeQE from the credential store, not by hand |
|
|
1733
2095
|
| `FORGEQE_CONFIG_DIR` | Global config folder (default `~/.config/forgeqe`): `forgeqe.yaml`, `llms.json` and `.env` used when a project has none |
|
|
1734
2096
|
| `FORGEQE_PYTHON` | A Python that already has the engine's libraries, used instead of the managed venv (skipped, with a note, when it cannot run the engine) |
|
|
1735
2097
|
| `FORGEQE_DEBUG` | `1` logs at debug level, like `--debug` |
|
|
2098
|
+
| `FORGEQE_TOOL_SCOPE` | `off` gives ForgeQE's agent every tool on every turn, instead of the tools of the capabilities each request is about |
|
|
2099
|
+
| `FORGEQE_ROUTING` | `off` also puts every playbook in the agent's prompt (for comparing, or ruling out a routing fault) |
|
|
1736
2100
|
| `FORGEQE_HOME` | Where the managed engine lives, user skills (`skills/`), and page memory when there is no config file (default `~/.forgeqe`) |
|
|
1737
2101
|
| `ADO_ORG_URL` / `ADO_PROJECT` / `ADO_PAT` | Azure DevOps organization, project, and token when `sources.azure_devops` doesn't set them |
|
|
1738
2102
|
| `JIRA_URL` / `JIRA_EMAIL` / `JIRA_API_TOKEN` | Jira site and credentials when `sources.jira` doesn't set them |
|
|
@@ -1756,7 +2120,7 @@ Start with `--debug` (or set `FORGEQE_DEBUG=1`) for more: the browser engine log
|
|
|
1756
2120
|
| `forgeqe features` | What ForgeQE can do, and what each capability still needs in this setup (works without a model) |
|
|
1757
2121
|
| `forgeqe docs [--guide\|--open] [--copy DIR]` | The bundled docs; `--guide` opens the user guide, `--open` the architecture page |
|
|
1758
2122
|
| `forgeqe mcp list` | The MCP servers ForgeQE's agent gets from your settings |
|
|
1759
|
-
| `forgeqe auth login [system\|NAME]` | Sign in to Azure DevOps, GitHub, Jira, Confluence, TestRail, Zephyr, qTest, or Xray in the browser, or save one secret, in the system's credential store |
|
|
2123
|
+
| `forgeqe auth login [system\|NAME]` | Sign in to Azure DevOps, GitHub, GitHub Copilot (`copilot`), Jira, Confluence, TestRail, Zephyr, qTest, or Xray in the browser, or save one secret, in the system's credential store |
|
|
1760
2124
|
| `forgeqe auth [status]`, `forgeqe auth logout <system>` | What is signed in; forget a sign-in |
|
|
1761
2125
|
|
|
1762
2126
|
### Run
|