@capagents/forgeqe 1.3.2 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (115) hide show
  1. package/HOW_TO_USE.md +373 -9
  2. package/README.md +55 -4
  3. package/docs/ARCHITECTURE.md +62 -1
  4. package/docs/CONTAINERS.md +227 -0
  5. package/docs/COST_ARCHITECTURE.md +32 -0
  6. package/docs/DESKTOP-CI.md +87 -0
  7. package/docs/DESKTOP.md +387 -0
  8. package/docs/MOBILE.md +228 -0
  9. package/docs/feature-test-prompts.md +25 -0
  10. package/docs/security-data-flow.html +440 -3
  11. package/docs/user-guide.html +797 -415
  12. package/env.example.yaml +67 -0
  13. package/forgeqe.example.yaml +56 -1
  14. package/llms.example.json +7 -0
  15. package/package.json +4 -1
  16. package/prompts/brain.md +31 -0
  17. package/prompts/find-tests.md +1 -0
  18. package/py/forgeqe_desktop/__init__.py +1 -0
  19. package/py/forgeqe_desktop/__main__.py +3 -0
  20. package/py/forgeqe_desktop/apps.py +80 -0
  21. package/py/forgeqe_desktop/checks.py +168 -0
  22. package/py/forgeqe_desktop/cli.py +90 -0
  23. package/py/forgeqe_desktop/handhooks.py +293 -0
  24. package/py/forgeqe_desktop/handrecord.py +239 -0
  25. package/py/forgeqe_desktop/heal.py +107 -0
  26. package/py/forgeqe_desktop/models.py +208 -0
  27. package/py/forgeqe_desktop/pywinauto_driver.py +217 -0
  28. package/py/forgeqe_desktop/record.py +150 -0
  29. package/py/forgeqe_desktop/replay.py +345 -0
  30. package/py/forgeqe_desktop/service.py +356 -0
  31. package/py/forgeqe_desktop/session.py +60 -0
  32. package/py/forgeqe_desktop/ufo_bridge.py +86 -0
  33. package/py/forgeqe_desktop/ufo_runner.py +305 -0
  34. package/py/forgeqe_desktop/visual.py +112 -0
  35. package/py/forgeqe_desktop/workflow.py +338 -0
  36. package/py/forgeqe_engine/llm.py +54 -4
  37. package/py/forgeqe_engine/model_params.py +169 -0
  38. package/py/forgeqe_mobile/__init__.py +1 -0
  39. package/py/forgeqe_mobile/__main__.py +3 -0
  40. package/py/forgeqe_mobile/appium_driver.py +342 -0
  41. package/py/forgeqe_mobile/apps.py +84 -0
  42. package/py/forgeqe_mobile/cli.py +70 -0
  43. package/py/forgeqe_mobile/explore.py +203 -0
  44. package/py/forgeqe_mobile/hyperexecute.py +261 -0
  45. package/py/forgeqe_mobile/maestro.py +321 -0
  46. package/py/forgeqe_mobile/service.py +332 -0
  47. package/skills/testcontainers/SKILL.md +42 -0
  48. package/src/agents.ts +11 -2
  49. package/src/api/plan.ts +2 -0
  50. package/src/api/write.ts +2 -0
  51. package/src/auth/copilot.ts +130 -0
  52. package/src/auth/signin.ts +54 -1
  53. package/src/brain/index.ts +26 -4
  54. package/src/capabilities/guide.ts +2 -0
  55. package/src/capabilities/registry.ts +110 -2
  56. package/src/capabilities/status.ts +15 -0
  57. package/src/cases/rca.ts +2 -0
  58. package/src/childEnv.ts +4 -1
  59. package/src/cli/args.ts +3 -2
  60. package/src/cli/help.ts +24 -3
  61. package/src/cli/index.ts +47 -3
  62. package/src/code/find.ts +26 -0
  63. package/src/code/loop.ts +4 -1
  64. package/src/code/prompt.ts +2 -1
  65. package/src/code/prreview.ts +2 -0
  66. package/src/code/review.ts +6 -2
  67. package/src/code/runtime.ts +34 -3
  68. package/src/commands.ts +6 -1
  69. package/src/config.ts +159 -3
  70. package/src/containers/command.ts +68 -0
  71. package/src/containers/compose.ts +173 -0
  72. package/src/containers/config.ts +283 -0
  73. package/src/containers/mock.ts +156 -0
  74. package/src/containers/run.ts +362 -0
  75. package/src/containers/tests.ts +80 -0
  76. package/src/context.ts +33 -6
  77. package/src/copilot.ts +155 -0
  78. package/src/desktop/bridge.ts +229 -0
  79. package/src/desktop/command.ts +350 -0
  80. package/src/desktop/edit.ts +217 -0
  81. package/src/desktop/junit.ts +35 -0
  82. package/src/desktop/report.ts +155 -0
  83. package/src/desktop/run.ts +252 -0
  84. package/src/desktop/setup.ts +134 -0
  85. package/src/engine/bridge.ts +12 -5
  86. package/src/engine/setup.ts +3 -3
  87. package/src/environments.ts +566 -0
  88. package/src/evals/domain.ts +1 -0
  89. package/src/explore/charters.ts +2 -0
  90. package/src/guardrails.ts +145 -0
  91. package/src/heal/loop.ts +4 -1
  92. package/src/heal/prompt.ts +2 -1
  93. package/src/init.ts +5 -2
  94. package/src/llm.ts +23 -2
  95. package/src/mcp/attachtools.ts +16 -4
  96. package/src/mcp/captools.ts +27 -0
  97. package/src/mcp/containertools.ts +212 -0
  98. package/src/mcp/desktoptools.ts +334 -0
  99. package/src/mcp/mobiletools.ts +132 -0
  100. package/src/mcp/reporttools.ts +17 -2
  101. package/src/mcp/repotests.ts +22 -3
  102. package/src/mcp/routing.ts +11 -5
  103. package/src/mcp/scope.ts +210 -0
  104. package/src/mcp/server.ts +80 -5
  105. package/src/mcp/tools.ts +38 -1
  106. package/src/mobile/command.ts +245 -0
  107. package/src/mobile/run.ts +226 -0
  108. package/src/pack/pipeline.ts +44 -2
  109. package/src/refine.ts +3 -0
  110. package/src/report/html.ts +3 -0
  111. package/src/report/template.html +48 -48
  112. package/src/run.ts +7 -3
  113. package/src/tokenomics/ledger.ts +3 -1
  114. package/src/tui/App.tsx +23 -0
  115. package/src/usage.ts +1 -1
package/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # ForgeQE
2
2
 
3
- Browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome locally, or BrowserStack and TestMu AI/LambdaTest through their Chrome DevTools Protocol (CDP) endpoints, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
3
+ Test agent CLI for web, Windows desktop and mobile apps. Say what you want in plain language: a goal to carry out in a browser, a Windows app or an Android or iOS app, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome locally, or BrowserStack and TestMu AI/LambdaTest through their Chrome DevTools Protocol (CDP) endpoints, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
4
4
 
5
5
  The engine is the browser service from test-agent-nexus, copied into this package (`py/forgeqe_engine`). It keeps the nexus behaviour: error-recovery system prompt, fast-mode agent tuning, Chrome launch flags, the search/select loop breaker, the 600 s run timeout, compact workflow YAML with semantic locators. There is no dependency on test-agent-nexus, FastAPI, or a database.
6
6
 
@@ -60,10 +60,11 @@ ForgeQE sends no telemetry and turns it off in everything it starts:
60
60
  | File | Purpose |
61
61
  |------|---------|
62
62
  | `forgeqe.yaml` | Browser, prompts, export, active profile |
63
- | `llms.json` | Named LLM profiles (azure, openai, ollama, openai_compatible) |
63
+ | `llms.json` | Named LLM profiles (azure, openai, ollama, openai_compatible, github_copilot) |
64
64
  | `.forgeqe/prompts/` | Every agent prompt as a markdown file, to change how an agent works (`forgeqe prompts` lists them) |
65
65
  | `.forgeqe/skills/` | The built-in skills, to adapt; add your own beside them |
66
66
  | `.forgeqe/knowledge/` | Product knowledge templates: product, roles, glossary, business rules, test data, environments |
67
+ | `.forgeqe/guardrails/` | Your guardrails: rules every agent must follow, one `- rule` line each (`forgeqe guardrails` lists them, `forgeqe guardrails add "…"` adds one) |
67
68
  | `.forgeqe/browserstack.yaml` / `.forgeqe/testmu.yaml` | Sample remote-browser provider configs, ready to point `browser.providers` at |
68
69
 
69
70
  An unchanged copy keeps following ForgeQE's updates; once you edit it, yours wins.
@@ -77,6 +78,8 @@ forgeqe run "Read the homepage of example.com" --plain
77
78
 
78
79
  The active profile is `llm.active` in `forgeqe.yaml`, else `default` in `llms.json`, else the first profile; Tab switches in the shell. With none set up, ForgeQE says so and how to add one.
79
80
 
81
+ **GitHub Copilot.** With Copilot licences, a `github_copilot` profile runs ForgeQE's agent and Scribe on the Copilot seat through the agent runtime's GitHub Copilot sign-in (`forgeqe auth login copilot`, GitHub.com or GitHub Enterprise). The browser engine uses GitHub Models with your GitHub sign-in (`forgeqe auth login github`, or `GITHUB_TOKEN`), optionally billed to an organization with `github_org`. See HOW_TO_USE.md, *GitHub Copilot seats as the model*.
82
+
80
83
  ## CLI
81
84
 
82
85
  ```bash
@@ -93,6 +96,8 @@ forgeqe run --direct "Read the homepage" --url https://example.com --out ./.forg
93
96
 
94
97
  There are no commands to pick a source or a mode. ForgeQE's agent reads the request and decides: a browser goal, test cases (Jira keys, Azure DevOps ids or test plans, Xray, TestRail, Zephyr Scale or qTest cases, files in the working directory, or cases pasted into the message), requirements to turn into cases (a Confluence page, a story), a test, or a pipeline. When a reference could mean more than one thing, it asks. The agent picks the site to open; `--url` only pins a start page.
95
98
 
99
+ **Tools per turn.** ForgeQE's agent does not carry every tool on every turn. Each turn gets the core (help, setup, the environments) and the capabilities the request is about, picked from its words, the apps it names and its links, plus those used in the last few turns. When it needs another, it calls `use_capabilities` and carries on with the same request. Fewer tools mean a smaller prompt and no mix-ups between similar ones (`browser_run`, `desktop_explore`, `mobile_explore`). `agent.capabilities.always` and `off` in `forgeqe.yaml` pin or remove capabilities, and `FORGEQE_TOOL_SCOPE=off` gives every tool. See [HOW_TO_USE.md](HOW_TO_USE.md#which-tools-forgeqes-agent-gets).
100
+
96
101
  Inside the shell, type the request and press Enter (Ctrl+J starts a new line). The only commands are for the session: `/stop`, `/headed`, `/details`, `/profile`, `/logs`, `/skills`, `/clear`, `/help`, `/exit`. Everything the other commands do can be asked for in the shell too, since each one is a tool the agent calls: "heal the checkout tests", "check the smoke cases once", "monitor them every 10 minutes", "sign in to shop", "forget the shop session", "which profiles do I have?". While the agents work, the shell shows each step's result, the files the coding agent writes, test results, and problems, with what is happening now on the activity line; `/details on` (or `--verbose`) adds the browser agent's reasoning, page memory, engine logs, and every file the coding agent reads. Results appear as cards in the transcript, and when a step needs you (signing in by hand), a **Your turn** prompt waits until you pick **Done, continue** or **Cancel**.
97
102
 
98
103
  ForgeQE's agent needs the agent runtime (downloaded automatically, see Install) and a live profile. It uses the active `llms.json` profile, or `code.model: provider/model`. Without the agent runtime, the request runs as one browser goal.
@@ -116,7 +121,7 @@ forgeqe run "..." --replay trust # or --replay off, --no-memory
116
121
 
117
122
  ## Skills
118
123
 
119
- Skills teach the agents what a prompt can't know: your repo's test conventions, a framework, or how to drive an app. A skill is a folder with a `SKILL.md` (the Agent Skills format: `name`, `description`, then instructions), plus ForgeQE's `agents: [forgeqe, scribe, browser]` to say who uses it. ForgeQE ships `playwright-typescript` and `dotnet-nunit-playwright` for Scribe, `dynamics-365` for the browser agent, and `writing-skills` for ForgeQE's agent. Your own go in `.forgeqe/skills` next to `forgeqe.yaml` (commit them) or `~/.forgeqe/skills`; a project skill replaces a built-in one with the same name. Skills in `~/.claude/skills` and `~/.agents/skills` are used only with `skills.external: true`.
124
+ Skills teach the agents what a prompt can't know: your repo's test conventions, a framework, or how to drive an app. A skill is a folder with a `SKILL.md` (the Agent Skills format: `name`, `description`, then instructions), plus ForgeQE's `agents: [forgeqe, scribe, browser]` to say who uses it. ForgeQE ships `playwright-typescript`, `dotnet-nunit-playwright` and `testcontainers` for Scribe, `dynamics-365` for the browser agent, and `writing-skills` for ForgeQE's agent. Your own go in `.forgeqe/skills` next to `forgeqe.yaml` (commit them) or `~/.forgeqe/skills`; a project skill replaces a built-in one with the same name. Skills in `~/.claude/skills` and `~/.agents/skills` are used only with `skills.external: true`.
120
125
 
121
126
  Each agent sees the skills meant for it and loads the ones whose description fits the job. For browser runs, ForgeQE's agent passes the skills a goal needs; in `forgeqe suite`, `monitor` and `run --direct` the engine picks them from the goal once, and `--skill NAME` (or `--skill none`) fixes the list. Reports and `run.json` show which skills a run used.
122
127
 
@@ -182,6 +187,52 @@ forgeqe api scenarios --text "create a user, add two items, check out" --spec op
182
187
  forgeqe api run-scenarios --allow-writes
183
188
  ```
184
189
 
190
+ ### Environments
191
+
192
+ `env.yaml` lists your test environments (dev, qa, uat, …) and the apps in each: web apps by URL, APIs by base URL, Windows apps by `.exe`, and mobile apps by package or bundle id. Its layout is up to you; ForgeQE reads any shape. The same test cases run in every environment: "run the smoke cases on uat" picks it for one request, and `--env NAME` (or `FORGEQE_ENV`) for a command. Secrets are listed by the name of the environment variable that holds them, never by value. `forgeqe apps` shows the environments and their apps. See [HOW_TO_USE.md](HOW_TO_USE.md#environments-and-apps).
193
+
194
+ ### Windows desktop apps
195
+
196
+ ForgeQE drives Windows apps with Microsoft's UFO² (explore a goal with the model) and pywinauto (replay with no model). A run that passes is saved as a `forgeqe-desktop-v1` workflow you can read and edit.
197
+
198
+ - **Replay** with timeouts, retries, `wait_for` (a control gone, or showing some text), and `dismiss` for dialogs that get in the way. A control that moved is healed with one model call and reported for review.
199
+ - **Visual checks:** `expect_image` compares a control or the window with a baseline next to the workflow.
200
+ - **Record by hand:** `forgeqe desktop record` saves the steps you do yourself as a workflow.
201
+ - **Edit from chat:** "make step 4 wait for the Save button" changes the workflow with `desktop_edit_workflow`, after showing you the diff.
202
+ - **CI:** `forgeqe desktop doctor` checks the machine (a signed-in, unlocked desktop session; Windows services run in session 0, which has none), and `--junit` writes results for the pipeline.
203
+
204
+ ```bash
205
+ forgeqe desktop explore "Create an invoice for Contoso" --app erp
206
+ forgeqe desktop record --app erp --name "New invoice"
207
+ forgeqe desktop replay desktop-tests/ --junit results.xml
208
+ ```
209
+
210
+ See [docs/DESKTOP.md](docs/DESKTOP.md) and [docs/DESKTOP-CI.md](docs/DESKTOP-CI.md).
211
+
212
+ ### Mobile apps (Android, iOS)
213
+
214
+ One workflow format (`forgeqe-mobile-v1`) runs on **Appium** (explore with the model, replay step by step, healing, visual checks) and on **Maestro** (the workflow becomes a Maestro flow). Your team's own Maestro flows run too, after an allowlist check that refuses scripts, AI commands and file access. Both run on a local emulator or device, or on TestMu AI's devices: `forgeqe mobile upload app.apk` uploads a build and returns its `lt://` id, and Maestro runs there through HyperExecute. `samples/mobile-testmu/` is a ready project that reads `LT_USERNAME` and `LT_ACCESS_KEY` from a `.env` file.
215
+
216
+ ```bash
217
+ forgeqe mobile explore "Sign in as the courier and accept the first delivery" --app courier
218
+ forgeqe mobile replay workflow.yaml --driver maestro
219
+ forgeqe mobile maestro .maestro/
220
+ ```
221
+
222
+ See [docs/MOBILE.md](docs/MOBILE.md).
223
+
224
+ ### Services and mocks in Docker
225
+
226
+ With Docker or Podman installed, ForgeQE starts the services a test run needs (a database, a queue, the API under test) as `containers.services` in `forgeqe.yaml` declares them, waits until each is ready, and stops them when the session ends. `mock_service` runs a WireMock mock of an API, built from the calls a browser run recorded or from an OpenAPI description, and can point an API app of the environment at it. ForgeQE builds every container command itself: ports bind to 127.0.0.1, mounts are read-only and inside the project, and privileged containers, the host network and the runtime's socket are refused. `docker` and `podman` are blocked in the agents' own shell.
227
+
228
+ ```bash
229
+ forgeqe services up db # start and wait until ready
230
+ forgeqe services # what runs
231
+ forgeqe services down
232
+ ```
233
+
234
+ Services can also come from a docker-compose file (`containers.compose`), checked by the same rules. With `containers.tests`, `run_tests` runs the repository's tests in a container, after starting the services they need, which the tests reach by name. Scribe's `testcontainers` skill writes tests that start their own services from code. Or ask: "mock the payments API from the last run and point payments at it". See [docs/CONTAINERS.md](docs/CONTAINERS.md).
235
+
185
236
  ## Test cases and suites
186
237
 
187
238
  Run existing test cases instead of typing goals: ask for them by name ("run SHOP-12", "run test plan 12 suite 34", "run the cases in smoke.yaml") or paste them into the message. Each case becomes one goal: its steps are what the browser agent does and its expected results are what it checks, and the agent's own verdict decides pass or fail. With a repo set, every passed case also gets a test.
@@ -367,4 +418,4 @@ Rounds go to `.forgeqe/reports/monitor/<time>/`, the same as suites, so the repo
367
418
  { "mcpServers": { "forgeqe": { "command": "forgeqe", "args": ["mcp", "--repo", "/path/to/app"] } } }
368
419
  ```
369
420
 
370
- Tools: `browser_run`, `find_test_cases`, `save_test_cases`, `run_test_cases`, `run_details`, `run_test`, `page_memory`, `read_confluence`, `query_api`, `push_branch`, `set_ci_secret`, `find_tests`, `run_tests` (runs only with `confirmed: true`, once the user agreed), `review_pull_request`, `post_pr_review` (posts only with `confirmed: true`), `list_sprints`, and one tool per command: `write_test`, `heal_tests`, `monitor`, `logins`, `forgeqe_setup`. Browser runs and suites log their progress to stderr, and long calls send progress notifications.
421
+ Tools: `browser_run`, `find_test_cases`, `save_test_cases`, `run_test_cases`, `run_details`, `run_test`, `page_memory`, `read_confluence`, `query_api`, `push_branch`, `set_ci_secret`, `find_tests`, `run_tests` (runs only with `confirmed: true`, once the user agreed), `review_pull_request`, `post_pr_review` (posts only with `confirmed: true`), `list_sprints`, and one tool per command: `write_test`, `heal_tests`, `monitor`, `logins`, `forgeqe_setup`. Browser runs and suites log their progress to stderr, and long calls send progress notifications. Desktop (`desktop_explore`, `desktop_replay`, `desktop_record`, `desktop_edit_workflow`) and mobile (`mobile_explore`, `mobile_replay`, `mobile_maestro`) tools are listed when the environment has such apps. The client gets the tools this project can use; `use_capabilities` adds others and sends `notifications/tools/list_changed`, and `forgeqe mcp --all-tools` lists every tool.
@@ -1,6 +1,6 @@
1
1
  # ForgeQE CLI architecture
2
2
 
3
- Three processes. The Bun CLI owns the shell, config, profiles, and ForgeQE's tools. ForgeQE's agent runs on the agent runtime, reads each request, and calls those tools over MCP. Scribe, the coding agent, runs on the same runtime and writes, heals, and reviews tests. A Python engine drives Chrome. There is no FastAPI server and no database.
3
+ Three processes. The Bun CLI owns the shell, config, profiles, and ForgeQE's tools. ForgeQE's agent runs on the agent runtime, reads each request, and calls those tools over MCP. Scribe, the coding agent, runs on the same runtime and writes, heals, and reviews tests. A Python engine drives Chrome; two more drive Windows apps (`py/forgeqe_desktop`, on UFO² and pywinauto) and Android and iOS apps (`py/forgeqe_mobile`, on Appium and Maestro). There is no FastAPI server and no database.
4
4
 
5
5
  The single-view map of subagents, tools, and capabilities is [`forgeqe-architecture.html`](../forgeqe-architecture.html) (`forgeqe docs --open`). Suite `report.html` includes the browser agent's token counts, and a cost in the machine's currency when the profile has a price.
6
6
 
@@ -182,6 +182,21 @@ Tools take structured arguments. `run_test_cases` and `find_test_cases` get a li
182
182
 
183
183
  **Callers.** The TUI opens the brain on the first message and keeps it for the session, so follow-up answers and "write a test for that run" have context; `/profile` (or Tab) closes it and keeps the turns so far: the next agent session's first prompt is `carryOver(turns, text)`, the last 20 turns with remembered secrets redacted, so switching models does not lose the conversation. It renders `card` events as result panels and `agent` events as `⚙` and test lines with their own token counter, and its `waitForUser` pushes the prompt into the transcript and opens a **Your turn** dialog (Done, continue / Cancel); opening another dialog or Esc settles it as cancelled. `forgeqe run` opens the brain for one request, prints tool events (cards as `cardText`) and the reply as plain lines, waits for Enter in `waitForUser` when stdin is a terminal, and exits 0 done, 1 failed, 2 stopped, 3 needs input. `forgeqe run --direct` and a missing agent runtime run `browser_run` directly. `forgeqe mcp` serves the same `TOOLS` on stdio and logs `ToolEvent`s to stderr; it has no `waitForUser`. `forgeqe start` and `forgeqe mcp` stop background monitors when they return, because `src/index.ts` calls `process.exit`.
184
184
 
185
+ ## Tool scope (`src/mcp/scope.ts`, `src/capabilities`)
186
+
187
+ Every model call carries the definitions of the tools the agent has, so the agent does not get every tool on every turn. The registry (`capabilities/registry.ts`) groups the tools into capabilities. Each capability has its tools, the requirements it needs (a model, a runtime, an app kind, a tracker), and keywords.
188
+
189
+ | Module | Contents |
190
+ |--------|----------|
191
+ | `mcp/scope.ts` | `CapabilityScope`, one per agent session. `beginTurn(text)` picks the turn's capabilities and returns the per-prompt `tools` switches (`forgeqe_<tool>: true\|false`). The picks are: the core (`setup-help`, `test-apps`), `agent.capabilities.always`, the ones `use_capabilities` asked for, those used in the last `STICKY_TURNS` (3) turns, and up to `MAX_PICKED` (4) from `scoreRequest`. `scoreRequest` makes no model call: it matches keywords and tool names, URLs (the browser), issue keys (test cases), and the names of the active environment's apps (by kind). A capability whose required apps or system are missing (`usableCapabilities`) is not picked from words alone, and nothing is picked from `agent.capabilities.off`. When nothing matches, `browser-goals` is the fallback. `request` (from `use_capabilities`) refuses unknown and switched-off ids and queues the added ones in `takePending`. `listedTools` is the list for MCP clients. `capabilityIndex` is the short list of capabilities in the agent's prompt |
192
+ | `mcp/captools.ts` | `use_capabilities`: adds capabilities for the rest of the session |
193
+ | `mcp/routing.ts` | `createRouter` passes every ForgeQE tool that runs to `scope.used` |
194
+ | `brain/index.ts` | `ask` starts each turn with `beginTurn` and prompts with its switches. When `use_capabilities` added capabilities during the turn, it continues the same request with them, at most `MAX_SCOPE_CONTINUATIONS` (2) times. `FORGEQE_TOOL_SCOPE=off` turns the scope off |
195
+ | `mcp/server.ts` | With a scope, `tools/list` returns `listedTools`, the server advertises `listChanged`, and after a call that added capabilities it sends `notifications/tools/list_changed`. `forgeqe mcp --all-tools` serves without a scope |
196
+ | `context.ts`, `agents.ts` | `forgeqe context`: the fixed context and what each capability adds. Budgets: 10000 tokens for the forgeqe agent's fixed context, 2500 per capability (`CAPABILITY_TOOL_BUDGET`) |
197
+
198
+ The tools stay registered with the agent runtime for the whole session; only the switches change, so the conversation is never restarted. `tests/scope.test.ts` covers routing, gating, sticky turns, config, hostile ids, and the MCP list change. `tests/brain.e2e.test.ts` checks against the real agent runtime that a turn sees only its tools and that `use_capabilities` continues the request.
199
+
185
200
  ## Scribe, the coding agent (`src/code`)
186
201
 
187
202
  The test-writing loop used by `forgeqe code`, by `forgeqe suite` with a repo, and by the `write_test` tool, which is how ForgeQE's agent writes tests in the shell and `forgeqe run`. It uses the agent runtime SDK and the runtime binary that `code/runtime.ts` finds.
@@ -330,6 +345,51 @@ Every write-back includes the step verdicts, and retries once without the step d
330
345
 
331
346
  `runSuite` gives each case a run folder `NN-<id>` and runs `executeRun` with a pool of `parallel` workers. With a repo, a passed case is queued for the coding agent as soon as its browser run ends and its worker awaits that job before taking the next case, so each case is explored, handed off, and coded before the next one starts; the queue is one promise, so coding jobs never overlap. A case that is not coded records `codeSkipped` with the reason. Write-back and `fileBugs` run after every case finishes and skip stopped cases. `report.ts` writes `junit.xml` (failed → `<failure>`, error → `<error>`, stopped → `<skipped>`, step verdicts and posted links in `system-out`), `report.html` (`src/report/data.ts` turns the results into `ReportData`: step verdicts, the `steps.json` timeline with screenshots as data URIs up to a 40 MB budget, failed cases first, small text files such as the trace and workflow, posted records, each run folder's `run.json` (page memory with heals and repairs, browser diagnostics) and `accessibility.json`, the case's group and requirements, usage and cost, and history from the sibling `.forgeqe/reports/suites/*/suite.json`. From that history it also finds each failed case's last passing run folder (next to its `suite.json`, else where it was written) for the "last good vs now" screenshot of the same URL path, and each memory-replayed case's last run with no replay, as the baseline for page memory savings; `src/report/html.ts` injects it as `window.__FORGEQE_REPORT__` into `src/report/template.html`, a React app from `report-ui/` built into one file by `bun run build:report`; every single run writes one too), `suite.json`, the terminal summary, and the GitHub job summary (appended to `GITHUB_STEP_SUMMARY` when set). The `run_test_cases` tool calls the same `runSuite`, passing the working directory and `runCodeAgent` as the coder when the session writes tests (so the agent never calls `write_test` for those cases afterwards), and the shell renders a suite card. There is no pipeline template: the agent writes pipelines to fit each repository, and they call `forgeqe suite`.
332
347
 
348
+ ## Environments (`src/environments.ts`)
349
+
350
+ `env.yaml` (named by `environments.file`) lists the environments and their apps, in any layout: `environments:` maps, top-level maps, or lists. They are all read into the same `Environment` (`apps` by kind: web, api, desktop, mobile; `variables`; `secrets` as names of environment variables). `activeEnvironment` picks one: `--env`, `FORGEQE_ENV`, `use_environment`, or the default. `withEnvironment` gives a config for one tool call's `env`. `exportEnvironment` puts the environment's `FORGEQE_APP_*` and variables into the environment of the tests ForgeQE runs. `environmentBrief` is what each agent is told: ForgeQE's agent gets every app, the browser agent its hosts, and Scribe the names to read so a test runs in every environment.
351
+
352
+ ## Desktop apps (`src/desktop`, `py/forgeqe_desktop`)
353
+
354
+ `python -m forgeqe_desktop serve` takes JSON-lines requests from `src/desktop/bridge.ts`, as the browser engine does.
355
+
356
+ - **Explore** runs UFO² in-process (`ufo_runner.py`, at a pinned commit) with the profile's model. Only UI actions are allowed, Win-key chords are refused, and the action trace becomes a `forgeqe-desktop-v1` workflow (`record.py`).
357
+ - **Replay** (`replay.py` on `pywinauto_driver.py`) needs no model. It handles timeouts, retries, `wait_for` and `dismiss`, `expect_image` (`visual.py`), heal by numbered choice (`heal.py`), and checks of the outcome (`checks.py`).
358
+ - **Recording by hand** (`handrecord.py`, `handhooks.py`) uses Windows hooks and UI Automation.
359
+ - **Changes from chat:** `src/desktop/edit.ts` makes fixed-shape changes, validates them, shows the diff, and keeps a backup.
360
+ - **Before a run:** `session.py` refuses a session that cannot be driven (session 0, locked, no screen), and `forgeqe desktop doctor` reports it.
361
+
362
+ The full file list is in [DESKTOP.md](DESKTOP.md#code), and unattended runs are in [DESKTOP-CI.md](DESKTOP-CI.md).
363
+
364
+ ## Mobile apps (`src/mobile`, `py/forgeqe_mobile`)
365
+
366
+ `forgeqe-mobile-v1` shares the desktop engine's workflow validation, replay, heal, checks and visual code.
367
+
368
+ - **Appium:** `appium_driver.py` is a W3C WebDriver client over HTTP, with locators quoted as XPath literals. It drives replay and the model's explore loop (`explore.py`).
369
+ - **Maestro:** `maestro.py` turns a workflow into a Maestro flow, with secrets as `MAESTRO_<NAME>` environment variables. It checks a team's own flows against an allowlist and runs `maestro test` with an argument list.
370
+ - **TestMu AI:** `hyperexecute.py` writes a HyperExecute project and runs its CLI, scrubbing the access key from the output and the run folder. `src/mobile/run.ts` uploads builds (`forgeqe mobile upload`).
371
+
372
+ The full file list is in [MOBILE.md](MOBILE.md#code).
373
+
374
+ ## Containers (`src/containers`)
375
+
376
+ `containers:` in forgeqe.yaml declares services (`config.ts`). It refuses any setting outside the fixed list, an image without an exact version, a port bound to an address other than 127.0.0.1, and a mount option.
377
+
378
+ - **`run.ts`** finds docker or podman and builds `run` arguments:
379
+ - ForgeQE's labels;
380
+ - `--security-opt no-new-privileges` and `--pids-limit`;
381
+ - `--publish 127.0.0.1:…`;
382
+ - `--env NAME` for secrets;
383
+ - `:ro` mounts.
384
+
385
+ It checks mounts as uploads are checked (inside the project; nothing hidden, key-named or linked). It waits for a TCP or HTTP ready check. Docker's port proxy accepts connections before the service listens, so a TCP connection counts only when it stays open. It scrubs secrets from logs and errors, and tracks the containers it started, which the CLI removes on exit.
386
+ - **`mock.ts`** makes WireMock stubs from `network.json` or a contract set.
387
+ - **`compose.ts`** translates docker-compose services into forgeqe.yaml services, refusing settings that widen what a container reaches. `startOrder` (`config.ts`) orders services by `depends_on`. Every container joins the project network `forgeqe-<key>` under its service name.
388
+ - **`tests.ts`** is a `runTestCommand` for `run_tests` with `containers.tests`. It runs the confirmed command with `sh -c` in the image, with the repository mounted at its own path, `--user` this machine's user, and only the environment's FORGEQE_* variables, the saved-login variables, and the listed env and secrets, by name.
389
+ - **The agent's tools** are `services` and `mock_service` in `src/mcp/containertools.ts`, and the CLI is `forgeqe services` (`command.ts`).
390
+ - **The agents' shell:** `BASH_PERMISSION` denies docker, podman, nerdctl and compose commands for the agent and Scribe.
391
+ - **Tests:** `tests/containers.test.ts` uses a fake runtime. `tests/containers.live.test.ts` uses real Docker and is skipped without it.
392
+
333
393
  ## Monitoring (`src/monitor`)
334
394
 
335
395
  `forgeqe monitor` is `runSuite` on a timer, plus memory between rounds of which checks are failing, plus alerts.
@@ -405,4 +465,5 @@ Rounds are ordinary suite folders, so `report.html` gets trends and flaky checks
405
465
  - `tests/heal.test.ts`: the healing loop with a scripted session that calls ForgeQE's tools over the real MCP server (only the four heal tools are listed; the browser run lands in the heal folder): nothing to heal, healed and confirmed, an app bug left failing, a healed test that still fails or has no command, finding the command, `code.test_command`, a missing report, a fatal error, and stop. `tests/monitor.test.ts`: `decide` sequences (fail_after, flaky, reminders, recovery, stopped, a round that could not run), duration and `monitor:` parsing, rounds through the real `runSuite` with state carried over, report links, attachments, `alerts.json`, pruning, the loop and `--once` exit codes, the lock against a live pid, the Teams card against a fake webhook, email through nodemailer to a local fake SMTP server (login, recipients, subject, screenshot), and secrets scrubbed from errors. `tests/cli.test.ts` runs `forgeqe monitor --test-alerts` and `--once` with the mock profile against a fake Teams webhook.
406
466
  - `tests/commands.test.ts`: the command tools called as the agent calls them: `write_test` with a scripted session (agent events, the card, the mock profile refused), `heal_tests` clean and healed, `monitor` round and status, `start` refused in a `once` session, start/status/stop in a `shell` session (a round card, a second start and a one-off round refused, the lock released), `test_alerts` against a fake webhook, `logins` list/clear and `sign_in` without a person, `page_memory` flows and clear, and `forgeqe_setup` profiles, MCP servers, docs by topic, and init. `tests/repotests.test.ts`: `parseResults` on JUnit and TRX; `find_tests` with a scripted Scribe (the read-only agent, the listing fed back, a corrected list command listed again, nothing found, mock profile and bad dates refused); `run_tests` declined (nothing runs), confirmed (per-test card from JUnit, the heal suggestion), and without `ask` (`needs_confirmation`, then `confirmed: true`, exit code only); sprints from fake Azure DevOps and Jira APIs (paging, the pick through `ask`, cancel, `needs_choice`, a name matched). `tests/engine/test_step_notes.py`: the agent's note built from its own fields and stored on the flow steps a step recorded, `evaluate` steps with their code. `tests/code.test.ts` (handoff): notes before each group of steps, once per step, and the JavaScript shown. `tests/update.test.ts`: bun and pip installs found in PATH order; waiting while the registry 404s, then the cache-skipping commands per install; `--check` running nothing; a release never served failing with a retry hint. `tests/sessionLog.test.ts`: the session header and events with remembered passwords, secret environment variables, and inline API keys replaced; tool calls and results; the bundle's contents without screenshots or keys; `/logs`. `tests/engine/test_page_settle.py`: the judge off by default and on by setting, `page_settled`, `settle_page` waiting for a new tab and giving up on one that stays blank, and note actions as read-only. `tests/resume.test.ts`: resume steps in the run result, `loadResume` by folder or run name, and `browser_run` returning the retry and passing `resume_from`. `tests/engine/test_resume.py`: `resume_point`, `resume_steps`, the loop start and nudge text, `_install_nudges` draining after the library's hook, resume steps in the flow file, the resume hand-off, checkpoints (kept apart from flows, cleared only by their own run, kept by a stopped run), and the hand-off after a cut-off run. `tests/code.test.ts` (resuming): a stopped job keeps `scribe-session.json` and a finished one removes it, a rerun reopens the session with the saved replay and a carry-on prompt, and a gone session starts fresh; `tests/code.e2e.test.ts` reopens a session by id after the runtime restarted and finds its earlier messages. `tests/engine.e2e.test.ts` runs a page where clicking "One way" resets the date: a model that takes the nudge finishes; one that ignores it is stopped with its resume steps saved, and a retry with `resume_from` replays the cookie click and needs two model calls. `tests/sessionLogins.test.ts`: remembered logins (domains from a URL, hosts, and wildcards; the engine payload with values; no process env), redaction next to punctuation and in carried-over turns, a remembered login replacing a configured one in the notes, ForgeQE's test runs getting the env names with the output redacted, and the `logins` tool's remember/list/clear without the password in any result. `tests/engine/test_session_logins.py`: remembered values as placeholders on every sign-in domain, env values still working, `step_outcomes`, and `making_progress`. `tests/tui.test.tsx` renders a monitor round's card in the transcript and checks the **Your turn** dialog holds a tool until **Done, continue**.
407
467
  - `tests/skills.test.ts`: `skills:` parsing; the prebuilt skills loading cleanly; precedence and `overrides` across user, project, the repo's own folder, and `dirs`; the repo's `.claude` skills only with `external`; the default and opt-in `agents`; `disable` and unknown names; every validation problem (including the unquoted `: ` hint); the per-agent `skill` permission and `skills.paths` in the runtime config; links readable through and removed without touching the skill; `browserSkills` given and choose, unknown names, and the remembered choice; the engine request through a stub engine and `skills` back on the result; the skills catalogue in the environment notes; `newSkill`, `setSkillDisabled` keeping comments, and `ignoredNote`; the overview and `/skills`; and `forgeqe skills new/show/disable/enable/check` end to end. `tests/code.e2e.test.ts` checks, against the real agent runtime, that Scribe's skill listing has only the registry's skills for it, an allowed skill's text arrives, and a disabled skill, the repo's `.opencode` and `.claude` skills, the runtime's built-in skill, and `~/.claude/skills` stay out unless `external` adds them through the registry. `tests/engine/test_skills.py`: `choose_skills` order, the cap of 3, errors and an empty catalogue without a model call; `skills_block` within `skills_max_chars`; the request fields.
468
+ - `tests/desktop.test.ts`, `tests/mobile.test.ts`, `tests/scope.test.ts`, `tests/capabilities.test.ts`: the desktop and mobile commands and tools, workflow edits, uploads, and the tool scope. `<engine-python> -m pytest tests/desktop`: the desktop and mobile engines against fake drivers, a fake Appium server and a stand-in Maestro (any OS). The `*_live` tests need Windows or an emulator, and run in `.github/workflows/forgeqe-desktop-windows.yml` and `forgeqe-mobile-android.yml`.
408
469
  - `<engine-python> -m pytest tests/engine`: the engine directly, with a scripted model, including `test_memory.py` (locators, confidence, identity, flows, secrets) and replay, trust mode, and self-healing against real Chrome. `test_brand.py` checks the overlay in real Chrome: the start page, trusted CDP clicks and keys dropped while locked and let through while acting, the unlocked state and step text on a page loaded mid-step, and a clean screenshot and DOM for the model. `WP_E2E_HEADED=1` runs the bun e2e test in a visible browser, with the overlay on.
@@ -0,0 +1,227 @@
1
+ # ForgeQE containers: services and mocks in Docker or Podman
2
+
3
+ Status: the tests run against a fake runtime on any OS. They check the commands ForgeQE builds, the checks on config,
4
+ compose files, mounts and secrets, the mock stubs, and tests in a container. `tests/containers.live.test.ts` runs on
5
+ real Docker (and is skipped without it):
6
+ - Postgres from forgeqe.yaml and from a compose file.
7
+ - A WireMock mock that answers a recorded run's calls.
8
+ - A tests container that reaches the database by name and writes its results into the repository as this machine's
9
+ user.
10
+
11
+ `.github/workflows/forgeqe-containers.yml` runs both on GitHub's Docker. There, a skipped live test fails the job.
12
+
13
+ ForgeQE uses Docker or Podman, when one is installed, for three jobs:
14
+
15
+ - **Services a test run needs:** a database, a queue, a cache, or the API under test. The project declares them in
16
+ forgeqe.yaml, or in a docker-compose file that forgeqe.yaml names.
17
+ - **Mocks of an API:** WireMock stubs made from the calls a browser run recorded, or from an OpenAPI description, so
18
+ the tests can run without the real service.
19
+ - **The repository's tests in a container:** `run_tests` runs the command the user confirmed in an image, instead of
20
+ on this machine, after starting the services the tests need.
21
+
22
+ Scribe also has a `testcontainers` skill, for tests that start their own services from code (below).
23
+
24
+ ForgeQE's agent uses them through two tools, `services` and `mock_service`. The CLI has `forgeqe services`. When
25
+ neither Docker nor Podman is on PATH, the agent isn't offered these tools.
26
+
27
+ ## Use
28
+
29
+ ```bash
30
+ forgeqe services # the declared services, what runs, whether the runtime answers
31
+ forgeqe services up # start every declared service and wait until each is ready
32
+ forgeqe services up db queue # just these
33
+ forgeqe services logs db --lines 50 # last lines, secrets removed
34
+ forgeqe services down # stop and remove this project's containers (mocks included)
35
+ ```
36
+
37
+ In the shell or `forgeqe run`, you can ask, for example:
38
+ - "start the database"
39
+ - "is the queue up?"
40
+ - "mock the payments API from the last run"
41
+ - "make a mock service from openapi.yaml and point order-api at it"
42
+ - "stop the services"
43
+
44
+ The agent asks before it starts anything.
45
+
46
+ ## Declaring services
47
+
48
+ ```yaml
49
+ # forgeqe.yaml
50
+ containers:
51
+ runtime: docker # docker | podman; default: docker, else podman, on PATH
52
+ services:
53
+ db:
54
+ description: Orders database for the API tests
55
+ image: postgres:16.4 # an exact version, or name@sha256:<digest>
56
+ ports: [5432] # 127.0.0.1:5432 -> 5432; "15432:5432" maps another port of this machine
57
+ env: # plain values only
58
+ POSTGRES_DB: shop
59
+ POSTGRES_USER: shop
60
+ secrets: [POSTGRES_PASSWORD] # environment variable names; their values come from the environment or .env
61
+ volumes:
62
+ - ./db/init:/docker-entrypoint-initdb.d # read-only; a folder or file inside the project
63
+ command: ["-c", "log_statement=all"] # arguments for the image's entrypoint
64
+ ready: { tcp: 5432 } # or { http: /health, port: 8080 }; without it: the container runs
65
+ timeout: 90 # seconds to wait until ready (default 60)
66
+ queue:
67
+ image: rabbitmq:3.13.7-management
68
+ ports: [5672, 15672]
69
+ ready: { http: /, port: 15672 }
70
+ mock:
71
+ image: wiremock/wiremock:3.9.1 # the default
72
+ ```
73
+
74
+ | Setting | Rule |
75
+ |---|---|
76
+ | Service name | Lower-case letters, digits and `-`, starting with a letter. Names starting with `mock-` are kept for mocks. |
77
+ | `image` | Must have an exact tag or a digest. An untagged image or `latest` is refused, because it changes under the same name. |
78
+ | `ports` | Always bound to 127.0.0.1. An address such as `0.0.0.0:5432:5432` is refused. |
79
+ | `env` | Plain values on one line. Secrets go under `secrets`. |
80
+ | `secrets` | Variable names. They are passed to the runtime by name (`--env NAME`), so the value is never on a command line. One of ForgeQE's own keys (a model key, a tracker token) is refused. |
81
+ | `volumes` | Always read-only. Must be inside the folder of forgeqe.yaml, and can't be the whole folder. |
82
+ | `command` | A list of arguments, never one command string. |
83
+ | `depends_on` | Services to start first. |
84
+ | Anything else | Refused with an error, not ignored. That includes `privileged`, `network`, `cap_add`, `devices`, `user`, `pid` and the socket. |
85
+
86
+ Every service, whether from forgeqe.yaml or a compose file, can list `depends_on`. `up` starts a service after the
87
+ ones it depends on; a circle is refused.
88
+
89
+ ## Compose files
90
+
91
+ ```yaml
92
+ containers:
93
+ compose: docker-compose.yml # or a list; inside the project
94
+ ```
95
+
96
+ ForgeQE reads each service of the file and translates it into a forgeqe.yaml service. The same rules then check it,
97
+ and a service declared in forgeqe.yaml wins over the compose file's service of the same name. Compose itself never
98
+ runs.
99
+
100
+ | Compose | ForgeQE |
101
+ |---|---|
102
+ | `image` | Required; an exact version or a digest. `build` is refused: build and tag the image first. |
103
+ | `ports` | `"5432"`, `"15432:5432"`, `"127.0.0.1:15432:5432"`, or the long form; always bound to 127.0.0.1. Any other address, and UDP, are refused. |
104
+ | `environment` | Plain values are passed as they are. `KEY: ${KEY}`, `KEY` alone in the list form, and `KEY: ${KEY:-default}` pass the variable by name as a secret. `${OTHER}` under another name, and values that mix text and `${…}`, are refused. |
105
+ | `volumes` | Bind mounts (`./db/init:/docker-entrypoint-initdb.d`) are mounted read-only, and `:rw` is noted. Named and anonymous volumes are left out, so the container keeps no data between runs. A path outside the project is refused when the service starts. |
106
+ | `command` | A list, or a string split into arguments (compose uses no shell, and neither does ForgeQE). |
107
+ | `depends_on` | List or map form: the start order. `condition` is not used; ForgeQE waits until each service is ready. |
108
+ | `x-forgeqe` | `ready`, `timeout`, `description`, as in forgeqe.yaml. Without `ready`, a service with ports is ready when its first port takes connections. |
109
+ | `container_name`, `restart`, `labels`, `healthcheck`, `networks`, `expose`, `deploy`, `profiles`, … | Ignored: ForgeQE names, labels and networks the containers itself. |
110
+ | `privileged`, `network_mode`, `cap_add`, `devices`, `pid`, `ipc`, `user`, `security_opt`, `volumes_from`, `env_file`, `entrypoint`, `extends`, … | Refused, and the error names the setting. |
111
+
112
+ What was left out (a named volume, a mount made read-only) is listed in `notes`.
113
+
114
+ ## The project network
115
+
116
+ Each project's containers join their own network, `forgeqe-<project>`, under their service name. So the API container
117
+ reaches the database as `db:5432`, and the tests container reaches both. No other container joins the network, and no
118
+ container is on the host's network. `forgeqe services down` with no names also removes the network.
119
+
120
+ ## Tests in a container
121
+
122
+ ```yaml
123
+ containers:
124
+ tests:
125
+ image: mcr.microsoft.com/playwright:v1.48.2-jammy # an exact version; the tests' toolchain
126
+ services: [db, order-api] # started first (when not running yet); the tests reach them by name
127
+ env: { DB_HOST: db }
128
+ secrets: [TEST_DB_PASSWORD]
129
+ ```
130
+
131
+ With `containers.tests` set, `run_tests` runs the tests in that image unless it is called with `in_container: false`.
132
+ Its confirmation says where the tests will run. The container:
133
+
134
+ - **Runs the command the user confirmed** with the image's own shell, `sh -c "<command>"`, inside the container,
135
+ never on this machine.
136
+ - **Mounts the repository read-write at its own path** and works in it, so results land where ForgeQE reads them.
137
+ On this machine, the tests already have the whole disk. A folder outside the repository that the results go to is
138
+ mounted the same way.
139
+ - **Runs as this machine's user** (`--user uid:gid`), so files it writes are yours. It also gets
140
+ `--security-opt no-new-privileges`, `--pids-limit 4096` and `--rm`, and it is removed by name when the run ends or is
141
+ stopped.
142
+ - **Gets only these variables:** `CI=1`, `NO_COLOR`, `HOME=/tmp`, the environment's `FORGEQE_ACTIVE_ENV`,
143
+ `FORGEQE_APP_*` and `FORGEQE_VAR_*`, the saved logins' variables, and `containers.tests` `env` and `secrets`.
144
+ Secrets are passed by name. ForgeQE's own keys are never passed.
145
+ - **Has secret values removed from its output**, as on this machine.
146
+
147
+ This needs a Linux or macOS machine, because the repository is mounted at its own path.
148
+
149
+ ## Testcontainers (Scribe)
150
+
151
+ When a test needs a real service, or the repository already uses Testcontainers (Java, .NET, Node, Python, Go), Scribe
152
+ loads the `testcontainers` skill. It reuses the repository's containers and fixtures, pins images, waits for
153
+ readiness, uses the mapped port, and sets the app's own configuration from it. It never uses privileged mode, the
154
+ host network, the Docker socket, or a `docker` command from a test. These containers belong to the tests in the
155
+ repository: the tests start and stop them, wherever they run.
156
+
157
+ ## Mocks
158
+
159
+ `mock_service` builds WireMock stubs and runs WireMock on a free port of 127.0.0.1:
160
+
161
+ - **From a run** (the default: the latest run, or `run_dir`):
162
+ - One stub per method, path and query of the app's own API calls, answering with the last status, content type
163
+ and body the app got.
164
+ - Other sites' calls are left out unless `third_party` is set.
165
+ - Failed calls are left out.
166
+ - Answers that were cut when recorded are listed under `skipped`.
167
+ - **From a description** (`spec`: OpenAPI 3 or Swagger 2, a link or a file):
168
+ - One stub per REST operation.
169
+ - Each stub answers with the operation's first success status and a body from the schema's examples, or values
170
+ made from its types.
171
+ - A path parameter (`/orders/{id}`) matches any one path segment.
172
+ - Instead of a description, you can give a contract set from `api_contracts` (`set_id`).
173
+ - **GraphQL, SOAP and gRPC** are listed under `skipped`, not mocked.
174
+
175
+ A spec given as a link is fetched by ForgeQE itself, with no credentials, as `api_contracts` does. The stubs are written to `.forgeqe/mocks/<name>/mappings/`. You can read and edit them there. The folder is mounted
176
+ read-only at `/home/wiremock/mappings`. WireMock runs with `--enable-stub-cors`, so a web app on another local port
177
+ can call it.
178
+
179
+ With `for_app: <api app>`, the active environment's API app points at the mock for the rest of the session. That
180
+ covers the API tools, and the tests ForgeQE runs, which read `FORGEQE_APP_<APP>_URL`. env.yaml itself is not
181
+ changed.
182
+
183
+ ## Lifetime
184
+
185
+ - **Services and mocks the agent starts** are removed when the shell, `forgeqe run`, or `forgeqe mcp` ends.
186
+ - **`forgeqe services up`** keeps its services running after the command ends, until `forgeqe services down`.
187
+ - **Container names** are `forgeqe-<project>-<service>`. `<project>` is a hash of the project folder, so two projects
188
+ never share a container. Starting a service replaces its earlier container. Removing a container also removes the anonymous volumes its image declares (a database's data folder), so nothing is left on the machine. A container is never reused with old
189
+ settings.
190
+ - **Labels:** each container carries `forgeqe=1`, `forgeqe.project`, `forgeqe.service` and `forgeqe.kind`.
191
+ `down` and `status` look only at this project's containers.
192
+
193
+ ## Security
194
+
195
+ ForgeQE's agent reads text other people control: tickets, web pages, repositories. A container command built from that
196
+ text could mount the whole disk or run privileged, so:
197
+
198
+ - **No commands from the model.** The agent names services that forgeqe.yaml declares, and ForgeQE builds the
199
+ argument array (`docker run …`). It never uses a shell. Images, flags and mounts never come from a request.
200
+ - **The same restrictions for every container:** `--security-opt no-new-privileges`, `--pids-limit 1024`, ports on
201
+ 127.0.0.1, and read-only mounts. There is no setting to turn these off.
202
+ - **Mounts are checked like uploads** (except a mock's stubs folder, which ForgeQE writes itself and mounts read-only). The path is resolved with `realpath` and must be inside the project. A mount
203
+ that is or holds a hidden file or folder (`.env`, `.git`, `.npmrc`), a file named like a key (`*.pem`, `id_rsa`,
204
+ `llms.json`) or a link is refused. A container can send what it reads anywhere.
205
+ - **Environment:**
206
+ - The `docker` and `podman` programs see only the system settings and the variables they need to find their daemon
207
+ (`DOCKER_HOST`, `DOCKER_CONTEXT`, `CONTAINER_HOST`, …), plus the service's listed secrets.
208
+ - A container gets only its `env` and `secrets`.
209
+ - Logs and errors have secret values replaced with `***`.
210
+ - **Not in the agents' shell:** `docker`, `podman`, `nerdctl`, `docker-compose` and `podman-compose` are denied in the
211
+ shell of ForgeQE's agent and of Scribe. So the only way to start a container is these tools.
212
+ - **Asked first:** `services up` and `mock_service` ask the user before starting, as `run_tests` does.
213
+
214
+ ## Code
215
+
216
+ | File | Contents |
217
+ |---|---|
218
+ | `src/containers/config.ts` | `containers:` in forgeqe.yaml: services, images, ports, env, secrets, mounts, ready checks |
219
+ | `src/containers/run.ts` | The runtime (docker or podman), `run` arguments, mount and secret checks, start and wait, status, logs, stop, the session's containers |
220
+ | `src/containers/mock.ts` | WireMock stubs from run traffic and from contract sets; values from JSON schemas |
221
+ | `src/containers/compose.ts` | docker-compose files as services: what is translated, ignored, left out or refused |
222
+ | `src/containers/tests.ts` | `run_tests` in a container: arguments, environment, mounts, output |
223
+ | `src/containers/command.ts` | `forgeqe services` |
224
+ | `skills/testcontainers/SKILL.md` | Scribe's Testcontainers skill |
225
+ | `src/mcp/containertools.ts` | The agent's `services` and `mock_service` |
226
+
227
+ Tests: `bun test tests/containers.test.ts`, which uses a fake runtime that records each command. `tests/containers.live.test.ts` uses real Docker.
@@ -278,3 +278,35 @@ different state. Flows like that stay with the agent until replay can repeat tho
278
278
  | **3** | Router (small model) plus Tier 1 adapt. Repo code tests run directly (`dotnet test` / `npx playwright test` with filters, results files parsed). | Reworded and new-data reruns at ≤5% |
279
279
  | **4** | Tool-result cache, context packs, history compaction. Release check as a fixed workflow. | The release check takes 0–1 agent calls; the agent's average context per call is down by half |
280
280
  | **5** | `forgeqe cost` report and the evaluation suite in CI | Metrics in section 7 reported per release |
281
+
282
+ ## 12. Per-turn tool scope (2026-10-17)
283
+
284
+ Tool definitions are part of the fixed overhead in section 1, and they grew with the desktop, mobile and
285
+ TestMu tools. The forgeqe agent's fixed context went past its budget. ForgeQE's agent now gets the tools of
286
+ the capabilities a turn needs, not all of them (`src/mcp/scope.ts`; see
287
+ [ARCHITECTURE.md](ARCHITECTURE.md#tool-scope-srcmcpscopets-srccapabilities)):
288
+
289
+ - **Every turn:** the core (help and setup, `use_capabilities`, the test environments and their apps).
290
+ - **Picked without a model call:** up to four capabilities matched from the request's words, the apps it
291
+ names, its links and issue keys. A capability the project has no apps or tracker for is not picked from
292
+ words alone.
293
+ - **Sticky:** capabilities whose tools ran in the last three turns.
294
+ - **On request:** `use_capabilities` adds one when the request turns out to need it, and ForgeQE continues the
295
+ same request.
296
+
297
+ `forgeqe context`, measured on this repo:
298
+
299
+ | | Tokens |
300
+ |---|---|
301
+ | The forgeqe agent's fixed context (prompt 6.2K + core tools 1.6K) | 7.8K of a 10K budget |
302
+ | A turn with the four largest capabilities | 14.7K |
303
+ | Every tool on every turn (`FORGEQE_TOOL_SCOPE=off`) | about 26.7K |
304
+ | Largest single capability (`api-testing`, `systems`) | 1.8K of a 2.5K budget per capability |
305
+
306
+ A typical turn ("replay the courier login on Maestro") adds one or two capabilities, around 1–3K. Beyond
307
+ the cost, fewer tools in view keep the agent from mixing up similar ones (`browser_run`,
308
+ `desktop_explore`, `mobile_explore`). The test `tests/agents.test.ts` fails when an agent or a capability passes its
309
+ budget. That keeps new tools from bringing the fixed overhead back.
310
+
311
+ **Still to measure:** live-model evaluations of the tool picks against `FORGEQE_TOOL_SCOPE=off`: the success
312
+ rate, the `use_capabilities` calls per request, and tokens per request.
@@ -0,0 +1,87 @@
1
+ # Running desktop workflows unattended (nightly CI)
2
+
3
+ Desktop replay clicks and types into real windows. It needs what a person at the machine would have: a signed-in
4
+ Windows session, unlocked, with a screen. GitHub's hosted `windows-2022` runners have that, and ForgeQE's own CI runs
5
+ Notepad there. Your own apps usually need a machine of your own, with the app installed, its test database, and a
6
+ network route to its servers. This page sets one up as a GitHub Actions self-hosted runner. The same rules apply to
7
+ Azure DevOps agents and Jenkins nodes.
8
+
9
+ ## What breaks a desktop run, and how ForgeQE tells you
10
+
11
+ | Situation | Effect | Fix |
12
+ |---|---|---|
13
+ | The runner is a Windows service | Session 0 has no desktop; every control lookup times out | Run the runner in the user's session (below) |
14
+ | The screen is locked | Windows sends no input to apps | No inactivity lock, no screen-saver lock |
15
+ | A Remote Desktop session was disconnected | The session is locked | Leave Remote Desktop with `tscon` (below), or don't use it on the test machine |
16
+ | The machine slept or the screen turned off | Input and screenshots stop | Power plan: never sleep, never turn off the screen |
17
+ | After a restart nobody signs in | No session at all | Sysinternals Autologon |
18
+
19
+ Before each replay or explore, the engine checks the session (`py/forgeqe_desktop/session.py`). On a session it cannot
20
+ drive, the run fails at once with the reason, instead of timing out step by step. `forgeqe desktop doctor` runs the
21
+ same check from a shell or a job:
22
+
23
+ ```text
24
+ engine ready (C:\Users\qa\.forgeqe\desktop\.venv\Scripts\python.exe)
25
+ explore not set up (forgeqe desktop setup --ufo)
26
+ session 1, screen 1920x1080
27
+ desktop logged in and unlocked
28
+ ```
29
+
30
+ It exits with 1 when the desktop cannot be driven, so a job stops before the suite runs.
31
+
32
+ ## Set up the test machine
33
+
34
+ Use a test machine or VM, and a test account: the runner's jobs control its mouse and keyboard.
35
+
36
+ 1. **Sign in** as the account the tests will run as.
37
+ 2. **Install the runner** (GitHub: Settings → Actions → Runners → New self-hosted runner). Give it the label
38
+ `forgeqe-desktop`. When `config.cmd` asks whether to run as a service, answer **no**.
39
+ 3. **Run `ci/desktop-runner/prepare-runner.ps1`** in an elevated PowerShell (`-WhatIf` first to see the changes):
40
+
41
+ ```powershell
42
+ .\prepare-runner.ps1 -RunnerDir C:\actions-runner -WhatIf
43
+ .\prepare-runner.ps1 -RunnerDir C:\actions-runner
44
+ ```
45
+
46
+ It stops sleep, screen-off and the screen-saver lock. It turns the inactivity lock off and registers a logon task
47
+ that starts `run.cmd` in this account's session. It warns you if the runner is also installed as a service. It
48
+ never handles a password.
49
+ 4. **Sign in automatically after a restart** with
50
+ [Sysinternals Autologon](https://learn.microsoft.com/sysinternals/downloads/autologon), run as the test account.
51
+ It stores the password encrypted by Windows. Never put the password in a script, a registry value you write
52
+ yourself, or the repository.
53
+ 5. **Fix the screen size.** A visual check (`expect_image`) compares pixels, so the resolution and scale (100%, 125%)
54
+ must match the machine where its baselines were saved. Set them once and leave them.
55
+ 6. **Restart** and check that the runner shows as idle on GitHub. Then run `forgeqe desktop doctor` from a job.
56
+
57
+ ### Leaving Remote Desktop without locking the session
58
+
59
+ Closing a Remote Desktop window disconnects the session, and Windows locks it. To leave it running instead, hand the
60
+ session back to the machine's console. In an elevated prompt inside the Remote Desktop session:
61
+
62
+ ```bat
63
+ for /f "skip=1 tokens=3" %s in ('query user %USERNAME%') do tscon %s /dest:console
64
+ ```
65
+
66
+ On a VM, the hypervisor's console (Hyper-V, VMware, VirtualBox) needs no such step.
67
+
68
+ ## The nightly job
69
+
70
+ `ci/desktop-runner/desktop-nightly.example.yml` is a ready workflow. Copy it into `.github/workflows/` in the
71
+ repository that holds your desktop workflows. It:
72
+
73
+ - runs at night on `[self-hosted, windows, forgeqe-desktop]`, one run at a time;
74
+ - installs a pinned ForgeQE and runs `forgeqe desktop setup` and `forgeqe desktop doctor`;
75
+ - replays every workflow in a folder: `forgeqe desktop replay desktop-tests/ --junit results/desktop.xml`;
76
+ - keeps the reports, screenshots and image diffs as an artifact.
77
+
78
+ `--junit` writes one test case per workflow (and per CSV row with `--data`), which CI servers show as test results.
79
+ `runs.json` in the output folder has the same, with the path of each report.
80
+
81
+ Secrets the workflows type (`{{secret:NAME}}`) come from the CI's secret store as environment variables, and must be
82
+ listed in `desktop.secrets` or the environment's `secrets` in `env.yaml`. Pick the environment with `FORGEQE_ENV` or
83
+ `--env`.
84
+
85
+ Healing calls the model. In a nightly run, either give the job the model's key so moved controls are healed and
86
+ reported in `heals.json` (never written into the workflow without `--accept-heals`), or pass `--no-heal` so a moved
87
+ control fails the run.