@capagents/forgeqe 1.3.2 → 2.0.0

package/docs/DESKTOP.md

@@ -0,0 +1,387 @@
+# ForgeQE desktop: thick-client automation
+
+Status: runs on Windows. The CI job `.github/workflows/forgeqe-desktop-windows.yml` (GitHub's `windows-2022`) installs the
+engine and the pinned UFO², replays Notepad live with the pywinauto driver and through `forgeqe desktop replay`, heals a
+moved control from the live window, and runs a whole explore: UFO² drives Notepad with a scripted model behind an
+OpenAI-compatible server, the action is recorded with the control's AutomationId, and the recording replays. Not yet
+run: explore with a real model (the job has no key), and your own apps. See
+[Before first use on Windows](#before-first-use-on-windows).
+
+ForgeQE drives web apps through Chrome. The desktop engine does the same for Windows thick clients (Win32, WinForms,
+WPF, and the native parts of Electron and WebView2 apps): say what you want in plain language, UFO² does it on the live
+app, and the run is saved as a YAML workflow that replays later without a model.
+
+## Use
+
+```bash
+forgeqe desktop setup --ufo                       # once, on Windows: the engine, pywinauto, and the pinned UFO²
+forgeqe desktop apps                              # what forgeqe.yaml allows
+forgeqe desktop explore "Create an invoice for Contoso, net 30" --app billing
+forgeqe desktop record --app billing              # or do the steps yourself: no model, no UFO
+forgeqe desktop replay .forgeqe/reports/desktop-explore-…/workflow.yaml customer=Fabrikam
+forgeqe desktop replay workflow.yaml --data customers.csv   # once per row; the header names the variables
+forgeqe desktop replay desktop-tests/ --junit results.xml   # every workflow in a folder, results for CI
+forgeqe desktop replay workflow.yaml --update-baselines     # save new baselines for its expect_image steps
+forgeqe desktop doctor                            # can this machine run replay unattended?
+forgeqe desktop validate workflow.yaml            # any OS
+```
+
+Or ask ForgeQE's agent in the shell or `forgeqe run`: "create an invoice for Contoso in Billing and save it as a test",
+or "I'll show you the steps in Billing, record them". Its tools are `test_apps`, `desktop_explore`, `desktop_record`,
+`desktop_replay` and `desktop_edit_workflow`. It asks before it starts the app or takes over the desktop, and before it
+changes a workflow. Where it cannot ask (an MCP client), the call answers `needs_confirmation` until the user agrees.
+
+Desktop apps come from the active environment in `env.yaml` (see HOW_TO_USE.md, "Environments and apps"), or from
+`desktop.apps` in `forgeqe.yaml`, which works without environments:
+
+```yaml
+desktop:
+  apps:
+    billing:
+      path: C:\Program Files\Contoso\Billing\Billing.exe
+      args: ["--profile", "test"]
+      # window: "Billing.*"        # for a launcher that starts the real app and exits: connect by title
+  secrets: [BILLING_PASSWORD]      # the only environment variables {{secret:…}} may read
+  heal: true                       # replay asks the model to re-find a control that moved (default on)
+  screenshots: true                # one screenshot per replayed step (default on)
+```
+
+Each run writes a folder under the reports folder (`desktop-explore-…` or `desktop-replay-…`): `run.json`,
+`report.html`, `steps/NNN.png`, and for explore `workflow.yaml`, `trace.json` and UFO's own logs in `ufo/`. A replay
+whose steps were healed also writes `heals.json` and asks whether to write the new locators into the workflow
+(`--accept-heals` writes them without asking; with no terminal they are never written).
+
+## Decisions
+
+| Question | Decision | Why |
+|---|---|---|
+| Agent that explores an app | Microsoft UFO² (`microsoft/UFO`, MIT), pinned to commit `eae3c0a8d168` and never edited, behind one adapter (`ufo_runner.py`) | It already solves UI Automation (UIA) control detection, merging it with vision, and the per-app agent loop. Behind the adapter it can be swapped for our own engine without touching the CLI, the YAML or replay. |
+| Saved scripts | ForgeQE's own YAML (`forgeqe-desktop-v1`), replayed on pywinauto (UIA backend) | UFO calls a model at every step. Replay must be fast, free, repeatable and fit for CI, so no model runs unless a step needs healing. |
+| Models | `llms.json`, the same profiles as the browser engine. UFO never gets its own model config | One place to configure models, keys and Microsoft Entra ID. |
+| Python environment | Its own (`~/.forgeqe/desktop/.venv`, Python 3.10), not the browser engine's | UFO pins `openai==1.66.2` and `pydantic`, which the browser engine's library does not run with, and `pandas==1.4.3`, which has Windows wheels up to Python 3.10. Replay's packages use UFO's pins, so one environment runs both. |
+| Not chosen: `windowsagent` | Reference only | Very early (v0.1.0), replay not built, and it exposes a model-driven `/shell` endpoint. |
+
+## How a run flows
+
+```
+forgeqe desktop explore ──► src/desktop/bridge.ts ──JSON lines──► python -m forgeqe_desktop serve
+                                                                    │ start the app from desktop.apps (no shell)
+                                                                    │ UFO² in-process: model = llms.json profile,
+                                                                    │ UI tools only, every action traced with its
+                                                                    ▼ control's UIA properties
+                                                       record.py ──► workflow.yaml
+forgeqe desktop replay ───► bridge ──► serve (replay.py on pywinauto, no model)
+                                          │ a control is not found
+                                          ▼
+                                       heal.py: the model picks one of the window's real controls, by number
+                                          ▼
+                                       proposed locator → heals.json, written into the workflow only when accepted
+```
+
+## One model config: `llms.json` → UFO
+
+UFO picks its model client by `API_TYPE` through `ufo.llm.base.BaseService.get_service`, which every model call goes
+through (`ufo/llm/llm_call.py`). The adapter adds the type `forgeqe`:
+
+- `write_ufo_config` writes UFO's `config/ufo/agents.yaml` from its template, with `API_TYPE: forgeqe`, the model's name,
+  and no endpoint and no key, for `HOST_AGENT`, `APP_AGENT`, `BACKUP_AGENT` and `EVALUATION_AGENT`.
+- `ufo_bridge.install` makes `get_service("forgeqe", …)` return `ForgeQEService`, which calls the profile through the
+  `openai` SDK (`models.py`): `azure` (key or `auth: entra`), `openai`, `openai_compatible` (key or Entra), and `ollama`
+  through its OpenAI-compatible `/v1`. Keys, Azure endpoints and Entra use the browser engine's own helpers
+  (`forgeqe_engine.llm`), so a profile means the same thing in both. Keys stay in environment variables.
+- Each call's time and tokens go to ForgeQE's usage ledger as the "desktop agent" (counts only, never the prompt).
+
+Explore sends screenshots, so its profile must take images: `"vision": true` or `false` on the profile says so; without
+it, `ollama` profiles are refused and others accepted. Healing sends a screenshot only to a profile that takes images.
+
+The request parameters suit the model by themselves: a reasoning model (o1, o3, o4, gpt-5, by the profile's `model` or
+`deployment` name, or `"reasoning": true` on the profile) gets `max_completion_tokens` and no temperature from the first
+call, and when the API refuses a parameter (`max_tokens`, `temperature`, `frequency_penalty`, `response_format`, …) the
+call is sent again without it, and later calls to that model leave it out (`py/forgeqe_engine/model_params.py`). The
+browser agent does the same.
+
+## Workflow YAML (`forgeqe-desktop-v1`)
+
+```yaml
+format: forgeqe-desktop-v1
+name: Create an invoice
+app: billing                 # a name from forgeqe.yaml desktop.apps, never a path or a command
+variables:
+  customer: Contoso
+steps:
+  - op: launch
+  - op: click
+    target: {automation_id: btnNewInvoice, control_type: Button, name: New invoice}
+  - op: type
+    target: {automation_id: txtCustomer, control_type: Edit}
+    text: "{{customer}}"
+  - op: type
+    target: {automation_id: txtPassword, control_type: Edit}
+    text: "{{secret:BILLING_PASSWORD}}"
+  - op: select
+    target: {automation_id: cmbTerms, control_type: ComboBox}
+    value: Net 30
+  - op: keys
+    keys: ctrl+s
+  - op: expect
+    target: {automation_id: lblStatus}
+    text: Saved
+  - op: close
+```
+
+- **Ops:** `launch`, `click`, `double_click`, `type`, `select`, `keys`, `expect`, `expect_image`, `wait` (at most
+  60 s), `wait_for`, `close`.
+- **Locators** use UIA properties, never screen coordinates: `automation_id`, `name`, `control_type`, `class_name`,
+  `index` (when several match), and `path` (the ancestors' `control_type` and `name`). A target needs an
+  `automation_id` or a `name`. Replay looks in the launched app's own windows only.
+- **Variables:** `{{name}}` from the workflow's `variables` or `name=value` on the command line; `{{secret:NAME}}` from
+  the environment, only for names in `desktop.secrets`. A secret ForgeQE itself uses (a model key, a tracker token) is
+  refused there. Values are substituted once and never expanded again. Targets take `{{name}}` too (a CSV row's
+  `{{customer}}` as the control to click), never a secret.
+- **Keys:** `ctrl`, `shift` and `alt` with one named key or character (`ctrl+s`, `alt+f4`, `enter`, `f5`). The Windows
+  and context-menu keys are refused: Win+R followed by typing runs any command.
+- **Typing** is literal: pywinauto's key syntax (`^`, `%`, `+`, `~`, `{ENTER}`) in the text is escaped.
+
+## Recording by hand
+
+When you know the steps, do them yourself instead of describing a goal: no model and no UFO are involved, so it costs
+nothing and does exactly what you did.
+
+```bash
+forgeqe desktop record --app billing --name "Create an invoice"
+```
+
+ForgeQE starts the app and watches the mouse and keyboard (low-level Windows hooks), asking UI Automation which control
+each click and key went to. Do the steps, then:
+
+- **Ctrl+Shift+F11** with the mouse over a control adds a check: an `expect` step with the text it shows now (a
+  status message, a saved total). Add one at the end, so the replay tests the outcome, not only the steps.
+- **Ctrl+Shift+F12**, or closing the app, finishes. The console shows each step as it is recorded.
+
+What becomes a step:
+
+| You do | Step |
+|---|---|
+| Click a control (two quick clicks: `double_click`) | `click` with its AutomationId, name and control type |
+| Pick an item in a drop-down | `select` on the drop-down, with the item's name |
+| Type into a field (corrections and pastes included) | one `type` with the field's text when you move on, read from the field |
+| A key chord outside a text field (`ctrl+s`, `enter`, `f5`) | `keys` |
+| Ctrl+Shift+F11 over a control | `expect` with its text |
+
+- **Passwords:** a password field is never read; its step types `{{secret:NAME}}` (the one name in `desktop.secrets`,
+  or `<APP>_PASSWORD`, which the result tells you to set). Text equal to a `desktop.secrets` value is written as its
+  placeholder. The typed characters themselves are never kept, only the field's text when you move on.
+- **Only the app:** clicks and keys outside the app's own process are not recorded; nor are the Windows and menu keys,
+  or controls with neither an AutomationId nor a name. The result lists what was left out.
+- The run folder has `workflow.yaml`, `run.json` and `report.html`, like explore. Replay it with
+  `forgeqe desktop replay`.
+
+## Fixing a workflow from chat
+
+When a replayed step fails because the app was slow or a dialog was in the way, ForgeQE's agent fixes the workflow
+with `desktop_edit_workflow`: it proposes the change (a longer `timeout` or `retries` on the step, a `wait_for` before
+it, a dialog button in `dismiss`, a healed locator, a corrected text), shows you the diff, and writes the file only
+when you agree. For example, after "step 4 timed out while the invoice list loaded":
+
+```text
+Change invoice.yaml?
+• new wait_for step before step 4
+• step 4: retries 2
+    - op: click
+  +   target: {automation_id: btnPost}
+  +   retries: 2
+```
+
+- Each change has a fixed shape (`set`, `unset`, `set_step`, `add_dismiss`, `insert_step`, `remove_step`,
+  `set_variable`); step numbers are the replay report's. Anything else is refused.
+- The changed workflow is checked by the engine's own validator before it is shown; one that would not run is not
+  offered. Comments and layout are kept.
+- The old text is kept in the reports folder (`desktop-edits/`), and a file that changed since the edit was proposed is
+  not overwritten.
+- The workflow must be a `.yaml` inside the working folder or the reports folder.
+
+## Waits, retries and dialogs
+
+Real apps are slow in places: a splash screen, a form that loads its data, a "Tip of the day" or "Save changes?"
+dialog. A workflow says how to wait for them, without fixed `wait` steps:
+
+```yaml
+timeout: 20              # seconds a step waits for its control (default 10, at most 300)
+retries: 1               # tries a failed action again, after a short pause (default 0, at most 5)
+dismiss:                 # clicked whenever one of them is in the way
+  - {name: OK, control_type: Button, automation_id: btnTipClose}
+  - {name: "Don't Save", control_type: Button}
+steps:
+  - op: launch
+    timeout: 90          # a slow start: how long to wait for the app's window
+  - op: wait_for         # the splash screen goes away
+    target: {automation_id: splashForm}
+    gone: true
+    timeout: 120
+  - op: wait_for         # the form has loaded its data
+    target: {automation_id: lblStatus}
+    text: Ready
+  - op: click
+    target: {automation_id: btnPost}
+    timeout: 60          # per step, over the workflow's timeout
+    retries: 3
+```
+
+- **Each step waits** for its control up to its `timeout`. `click`, `type` and `select` also wait until the control is
+  enabled and visible (a form that is still loading greys its buttons out).
+- **`wait_for`** waits until a control is there, shows a `text`, or with `gone: true` has gone.
+- **`retries`** applies to `click`, `double_click`, `type`, `select`, `keys`, `expect` and `expect_image`. Between
+  tries, replay pauses (1 s, 2 s, … up to 5 s) and closes any `dismiss` dialog. An error in the workflow itself (a
+  missing variable) is not retried.
+- **`dismiss`** lists at most 10 targets, checked like any target. When a control is not found, or something waits to
+  go away, replay clicks the `dismiss` controls that are on screen and looks again, up to 3 times per lookup. The
+  report and the console name every dialog closed, and how many tries a step took.
+
+## Visual checks (`expect_image`)
+
+`expect` compares text. When what matters is drawn (a chart, a signature, a colour, a layout), `expect_image`
+compares pixels with a baseline image:
+
+```yaml
+  - op: expect_image
+    target: {automation_id: chartRevenue}   # without a target: the app's top window
+    image: revenue-chart.png                # a file name in <workflow name>.baselines/ next to the workflow
+    tolerance: 0.01                         # share of pixels that may differ (default 0.01, i.e. 1%)
+    ignore: [[0, 0, 120, 24]]               # regions [x, y, width, height] that change every run (a date, a clock)
+```
+
+- **The first run** with no baseline saves what it captured and passes; the report says so. Commit the baselines
+  folder with the workflow. `--update-baselines` replaces them after an intended change.
+- **A pixel counts as different** when a colour channel differs by more than 32 of 255, so anti-aliasing and ClearType
+  do not fail a check. A different size (another screen scale or window size) fails with both sizes named.
+- **The report** shows the baseline, what was captured, and the differences in red, with the share of pixels that
+  differ (`images/NNN-baseline|actual|diff.png` in the run folder).
+- **Image names** are file names only (letters, digits, `_`, `-`, ending `.png`). A path, `..`, or a link that leads
+  out of the baselines folder is refused.
+- Run visual checks on one machine with a fixed resolution and scale (see [DESKTOP-CI.md](DESKTOP-CI.md)).
+
+## Unattended runs
+
+Replay needs a signed-in, unlocked Windows session. Before it starts the app, it checks the session and fails at once
+with the reason if it is a Windows service (session 0), a locked screen, or a disconnected Remote Desktop session.
+`forgeqe desktop doctor` runs the same check. To run the suite every night on a machine of your own, see
+[DESKTOP-CI.md](DESKTOP-CI.md): a runner set up in the user's session (`ci/desktop-runner/prepare-runner.ps1`), a
+nightly workflow (`ci/desktop-runner/desktop-nightly.example.yml`), and `--junit` results.
+
+## Updating UFO²
+
+UFO² is pinned to one commit. Moving it is checked before it is pinned:
+
+1. `.github/workflows/forgeqe-desktop-ufo-update.yml` runs weekly, or by hand with a commit. It sets up that UFO on
+   Windows (`FORGEQE_UFO_COMMIT`, which takes a full 40-character commit id only), runs the adapter tests against its
+   real code and a scripted explore of Notepad, and says in the job summary whether it passed.
+2. When it passed: `bun scripts/ufo-bump.ts <commit>` writes the new pin into `src/desktop/setup.ts`,
+   `py/forgeqe_desktop/ufo_runner.py` and this page. Run the tests and push; the Windows CI job tests the pin again.
+
+When it fails, the failing step names what changed in UFO (usually the action executor or the config layout that the
+adapter wraps). Keep the pin until the adapter is updated for it.
+
+## Checks of the outcome
+
+A recording of actions only proves the steps run. When UFO has finished, and before the app closes, explore shows the
+model the goal and the app's controls with the text each shows now, and the model picks up to three whose text proves
+the goal (a status message, a saved value), by number, with the part of the text to expect. A check is kept only when
+that text really is in that control, so the model can choose among what is on screen but never invent it; text holding
+a `desktop.secrets` value is never offered. The checks are saved as `expect` steps at the end of the workflow, and the
+report says how many there are. With none, the report and the agent say so: add `expect` steps by hand.
+
+## Recording
+
+UFO's own step log names the control and a per-step id, not its AutomationId, so the adapter wraps UFO's
+`ActionExecutor.execute` and records each action with the UIA properties of the control it acted on. `record.py` turns
+that trace into steps: `click_input` → `click`/`double_click`, `set_edit_text` → `type`, `keyboard_input` → `keys` (one
+allowed chord) or `type`, `select` → `select`. Coordinate clicks, scrolls and anything else are listed as not recorded
+in the report. A typed value equal to a `desktop.secrets` value is written as its placeholder; typed text that only
+looks like `{{…}}` is kept literal, so a model told to type `{{secret:X}}` cannot make replay type the secret.
+
+## Security
+
+The YAML, the trace, the app's screen text and the model's replies can all be written by someone else. The rules in the
+repository's `CLAUDE.md` apply; for this engine:
+
+- **No command in the workflow.** `app:` names a `desktop.apps` entry: an absolute `.exe` (a `.bat` or `.cmd` would run
+  through `cmd.exe`, which reparses its arguments) and an argument list, started with `subprocess.Popen([path, *args])`.
+- **UFO gets UI tools only.** Each explore writes UFO's `mcp.yaml` with `UICollector`, `HostUIExecutor` and
+  `AppUIExecutor`. UFO's default `CommandLineExecutor`, which runs shell commands, and its Word, Excel, PowerPoint and
+  PDF executors are left out. The adapter starts the app, so UFO never launches a program.
+- **Keys:** during explore the adapter refuses UFO's Windows and menu keys too, and the report lists what it refused.
+- **No one at the console:** UFO's safe guard asks before sensitive actions; explore answers no, since nobody is there.
+- **Environment:** the engine process gets system settings, the profile's key variable (or Entra's) and
+  `desktop.secrets`, nothing else (`desktopEnvNames`).
+- **Explore controls the real mouse and keyboard.** It asks before starting (`--yes` to skip; required with no
+  terminal). Use a test machine or VM and a test account.
+- **Agent tools:** a workflow path the agent gives must be a `.yaml` inside the working folder or the reports folder (real
+  path, so a link out is refused); variables are names and text only; heals are reported, never written.
+- **Healing:** the model picks one of the window's real controls by number; control text is passed as data, and any
+  other answer heals nothing. Heals are written into the workflow only when accepted.
+- **Report:** every value is escaped; images are only the engine's own `steps/NNN.png` and
+  `images/NNN-baseline|actual|diff.png`.
+- **Baselines:** `expect_image` names are file names, and their real path must be inside the workflow's baselines
+  folder, so a workflow cannot read or overwrite another file.
+- **Unattended machines:** `prepare-runner.ps1` never handles a password; automatic sign-in uses Sysinternals
+  Autologon, which stores it encrypted.
+- **Install:** UFO is cloned and checked out at the pinned commit (or `FORGEQE_UFO_COMMIT`, a full commit id only),
+  and the checkout's `HEAD` is verified; replay's packages are pinned to exact versions.
+
+## Code
+
+| File | Contents |
+|---|---|
+| `src/desktop/command.ts` | `forgeqe desktop`: setup, apps, doctor, validate, explore, record, replay (folders, `--data`, `--junit`, `--update-baselines`); heal acceptance |
+| `src/desktop/junit.ts` | JUnit XML for a suite |
+| `src/desktop/edit.ts` | Changes to a workflow from chat: shapes, applied with comments kept, validated, diff, backup |
+| `src/desktop/run.ts` | Explore and replay as the CLI and the agent's tools run them; variables, CSV rows, workflow paths |
+| `src/mcp/desktoptools.ts` | ForgeQE's agent's `test_apps`, `desktop_explore`, `desktop_record`, `desktop_replay`, `desktop_edit_workflow` |
+| `src/desktop/bridge.ts` | Starts `python -m forgeqe_desktop serve`, passes events on, records model calls in the usage ledger |
+| `src/desktop/setup.ts` | The desktop environment, the pinned UFO checkout (`ufoCommit`), `desktopDoctor` |
+| `scripts/ufo-bump.ts` | Pins a new UFO commit in every place that names it |
+| `ci/desktop-runner/` | `prepare-runner.ps1` and the nightly workflow example for unattended runs |
+| `src/desktop/report.ts` | `report.html` for a run |
+| `py/forgeqe_desktop/service.py` | `serve`: the JSON-lines protocol, replay and explore runs, `run.json` |
+| `py/forgeqe_desktop/workflow.py` | Load, validate and save workflows; variables and secrets; the key-chord grammar |
+| `py/forgeqe_desktop/apps.py` | `desktop.apps`: the app allowlist and the argument array a launch uses |
+| `py/forgeqe_desktop/models.py` | The profile through the `openai` SDK; image checks |
+| `py/forgeqe_desktop/ufo_bridge.py` | UFO's model blocks, and the `forgeqe` model service |
+| `py/forgeqe_desktop/ufo_runner.py` | UFO's config, the action trace and key guard, running UFO in-process |
+| `py/forgeqe_desktop/record.py` | Trace to workflow steps |
+| `py/forgeqe_desktop/replay.py` | Runs a workflow on a `Driver`: timeouts, retries, `wait_for`, dismissing dialogs, heal and stop |
+| `py/forgeqe_desktop/handrecord.py` | Recording by hand: clicks, typing, chords and checks as steps; secrets |
+| `py/forgeqe_desktop/handhooks.py` | The Windows hooks and UI Automation lookups that feed it |
+| `py/forgeqe_desktop/visual.py` | `expect_image`: baselines, pixel comparison, diff images |
+| `py/forgeqe_desktop/session.py` | Whether this Windows session can be driven (service, locked, no screen) |
+| `py/forgeqe_desktop/heal.py` | One model call to re-find a moved control |
+| `py/forgeqe_desktop/checks.py` | One model call for checks of the outcome, kept only when on screen |
+| `py/forgeqe_desktop/pywinauto_driver.py` | The Windows `Driver` |
+
+Tests: `bun test tests/desktop.test.ts`, and `<engine-python> -m pytest tests/desktop -q` (needs `openai==1.66.2`
+and PyYAML; they use a fake driver and a fake UFO, so they run on any OS).
+
+## Before first use on Windows
+
+Verified in CI: setup with and without `--ufo`, replay, healing, the CLI, and UFO's own code with the adapter (its
+config, its model calls routed to the profile, the action trace, UI tools only). Still to check, on a Windows test
+machine with your apps:
+
+1. **UIA coverage:** `Inspect.exe` shows the controls that matter with stable `AutomationId`s. Custom grids, Java
+   (needs Java Access Bridge), Delphi, SAP GUI and Citrix (vision only) are where UIA falls short.
+2. **Explore with your model:** one goal with an Azure profile and one with `auth: entra`; `trace.json` has the
+   controls' AutomationIds, and `ufo/` has UFO's logs.
+3. **Replay:** the recorded workflow passes 10 runs out of 10; a renamed control is healed.
+4. **Cost and time** of one explore, from `forgeqe` usage.
+
+Found by the Windows runs and fixed: UFO's pins need Python 3.10 (pandas 1.4.3); UFO prints emoji panels to stdout, which
+is the engine's event stream and crashed on the cp1252 console (now sent to stderr, characters replaced); an error in
+UFO's own evaluation lost the recorded actions (the trace is now kept and saved).
+
+## Next steps
+
+1. The checks above.
+2. `forgeqe suite` and test cases from Azure DevOps, Jira and Xray for desktop workflows, with results written back.
+3. Scribe: export a workflow as pywinauto code (or C# with FlaUI) in the user's repo.
+4. App adapters where UIA falls short: Java Access Bridge, SAP GUI scripting, vision-only for Citrix.
+5. A pipeline template for a self-hosted Windows runner signed in to a desktop session.

package/docs/MOBILE.md

@@ -0,0 +1,228 @@
+# ForgeQE mobile: Android and iOS apps
+
+Status: the engine and its tests run on any OS, against a fake Appium server and a stand-in for Maestro. The CI job
+`.github/workflows/forgeqe-mobile-android.yml` runs it on an Android emulator: Appium explores and replays the
+emulator's Settings app, and Maestro replays the same workflow. iOS runs the same code with the XCUITest driver, but
+no CI job tests it yet (it needs a macOS runner with a simulator).
+
+ForgeQE drives mobile apps two ways, and one workflow format runs on both:
+
+| | Appium | Maestro |
+|---|---|---|
+| Runs | step by step, through an Appium server (yours or a device cloud's) | the whole workflow as one Maestro flow |
+| Explore a goal with the model | yes | no |
+| Heal a control that moved | yes, one model call, reported for review | no |
+| Visual checks (`expect_image`) | yes | no |
+| Your team's own flows | n/a | `forgeqe mobile maestro`, checked against an allowlist |
+| Needs | `appium` with UiAutomator2 (Android) or XCUITest (iOS), a device or emulator | `maestro` and a device or emulator |
+
+## Use
+
+```bash
+forgeqe mobile doctor                                   # does the Appium server answer, is maestro installed?
+forgeqe mobile apps                                     # the mobile apps of the active environment
+forgeqe mobile explore "Sign in as the courier and accept the first delivery" --app courier
+forgeqe mobile replay .forgeqe/reports/mobile-explore-…/workflow.yaml
+forgeqe mobile replay workflow.yaml --driver maestro
+forgeqe mobile replay mobile-tests/ --data couriers.csv --junit results.xml
+forgeqe mobile maestro .maestro/                        # the team's flows
+forgeqe mobile flow workflow.yaml                       # print a workflow as a Maestro flow
+forgeqe mobile validate workflow.yaml
+```
+
+Or ask ForgeQE's agent in the shell or `forgeqe run`: "sign in to the courier app and save it as a test", "replay the
+courier login on Maestro", "run our Maestro flows". Its tools are `mobile_explore`, `mobile_replay` and
+`mobile_maestro`. It asks before each one starts, and `desktop_edit_workflow` fixes a mobile workflow after a failed
+replay, the same way as a desktop one.
+
+## Apps and settings
+
+Mobile apps come from the active environment in env.yaml, or from `mobile.apps` in forgeqe.yaml:
+
+```yaml
+# env.yaml
+environments:
+  qa:
+    apps:
+      courier:
+        kind: mobile
+        platform: android               # or ios
+        app_id: com.contoso.courier     # Android package or iOS bundle id
+        activity: .MainActivity         # Android only, optional
+        app: /builds/courier-qa.apk     # optional: install this build (.apk/.aab/.ipa/.app/.zip, https URL, bs://…)
+        description: Driver app that confirms delivery
+    secrets: [COURIER_PASSWORD_QA]
+```
+
+```yaml
+# forgeqe.yaml
+mobile:
+  driver: appium                        # appium | maestro: the default for replays
+  appium:
+    url: http://127.0.0.1:4723
+    capabilities:                       # added to every session
+      appium:udid: emulator-5554
+    # username_env: BROWSERSTACK_USERNAME
+    # access_key_env: BROWSERSTACK_ACCESS_KEY
+  maestro:
+    command: [maestro]
+    # device: emulator-5554
+  secrets: [COURIER_PASSWORD]
+  heal: true
+  screenshots: true
+```
+
+Each run writes a folder under the reports folder (`mobile-explore-…`, `mobile-replay-…`, `mobile-maestro-…`) with
+`run.json`, `report.html`, step screenshots, and for Maestro runs the flow, `maestro.log` and Maestro's JUnit report.
+
+## Workflow YAML (`forgeqe-mobile-v1`)
+
+```yaml
+format: forgeqe-mobile-v1
+name: Courier signs in
+app: courier
+timeout: 20
+dismiss:
+  - {name: Allow, control_type: android.widget.Button}   # permission dialogs, when they are in the way
+steps:
+  - op: launch
+  - op: type
+    target: {automation_id: username}
+    text: "{{courier}}"
+  - op: type
+    target: {automation_id: password}
+    text: "{{secret:COURIER_PASSWORD}}"
+  - op: tap
+    target: {name: Sign in}
+  - op: wait_for
+    target: {automation_id: progress}
+    gone: true
+  - op: scroll_to
+    target: {name: Deliveries}
+  - op: expect
+    target: {automation_id: status}
+    text: Welcome
+```
+
+- **Ops:** `launch`, `tap`, `double_tap`, `long_press`, `type`, `expect`, `expect_image`, `swipe` (`direction` up,
+  down, left or right, optionally on a `target`), `scroll_to` (scrolls until the target is on screen), `back`,
+  `hide_keyboard`, `keys` (`enter`, `back`, `tab`, `delete`, `search`), `wait`, `wait_for`, `close`. Timeouts,
+  retries and `dismiss` work as in desktop workflows. Explore saves a tap on a system dialog (another package's
+  control, such as Android's notification prompt) under `dismiss`, not as a step, so the workflow still runs where the
+  dialog never shows; on Maestro, each `dismiss` target is tapped after a launch if it is there.
+- **Targets** use the desktop keys, mapped per platform: `automation_id` is the Android resource-id (with or without the
+  `package:id/` prefix) or the iOS accessibility identifier; `name` is the Android text or content-desc, or the iOS
+  label; `control_type` is the Android class (`android.widget.Button`) or the iOS type (`XCUIElementTypeButton`);
+  `index` picks one of several matches.
+
+## Explore
+
+The model sees the goal, its steps so far, the numbered controls on screen with the text each shows, and a screenshot
+when the profile takes images. It answers with one action on a control by its number (tap, long press, type), a swipe
+or scroll, back, or done. So it can only act on a control that is on screen. When it says done, it picks checks of the
+outcome from the controls' current text, kept only when that text is really there, as on the desktop. The app's text
+is passed as data.
+
+## Maestro
+
+**ForgeQE workflows on Maestro.** `--driver maestro` writes the workflow as a Maestro flow (`flow.yaml` in the run
+folder) and runs `maestro test`. Maestro's debug output gives each workflow step its result. Variables are filled in
+first. Values are matched literally, so Maestro's regular expressions never see them as patterns. A secret becomes
+`${MAESTRO_NAME}`, which Maestro reads from the environment ForgeQE gives it, so a secret's value never lands in the flow
+file or on a command line. A value holding `${…}` is refused, because Maestro would run it as JavaScript.
+
+**Your team's flows.** `forgeqe mobile maestro <files or folders>` checks each flow, and every flow it runs, before
+Maestro gets it:
+
+- Only UI commands pass: `launchApp`, `tapOn`, `inputText`, `assertVisible`, `scrollUntilVisible`, `swipe`,
+  `extendedWaitUntil`, `runFlow`, `repeat`, `retry` and the like.
+- These are refused:
+  - `runScript`, `evalScript` and `assertTrue`, which run JavaScript.
+  - Maestro's AI commands, which send screenshots to a service.
+  - `takeScreenshot`, recordings and `addMedia`, which read or write files.
+- `${…}` may only name a variable (`${USERNAME}`), never an expression.
+- `when: {true: …}` is refused, because it is JavaScript.
+- `openLink` takes http(s) or the app's own deep links, never `javascript:`, `file:` or `intent:`.
+- `runFlow` files must be `.yaml` inside the working folder; links out of it are refused too. Nesting stops at 5 levels.
+
+A refused flow does not run, and the error names the command.
+
+## TestMu AI (formerly LambdaTest)
+
+Both drivers run on TestMu's devices. Upload the app once, then use the `lt://` id it returns as the app's `app`:
+
+```bash
+forgeqe mobile upload courier.apk          # → lt://APP…  (--virtual for TestMu's emulators)
+```
+
+The user and key come from `LT_USERNAME` and `LT_ACCESS_KEY` (or the variables the config names), for example in a
+`.env` file in the project folder. `samples/mobile-testmu/` is a ready project with TestMu's Wikipedia sample app.
+
+```yaml
+mobile:
+  apps:
+    courier: { platform: android, app_id: com.contoso.courier, app: "lt://APP1016…" }
+  appium:                                   # Appium on TestMu's mobile hub
+    url: https://mobile-hub.lambdatest.com/wd/hub
+    username_env: LT_USERNAME
+    access_key_env: LT_ACCESS_KEY
+    capabilities:
+      "LT:Options": { w3c: true, isRealMobile: true, deviceName: "Pixel 6", platformVersion: "14", build: nightly }   # app: added from the app's own
+  maestro:
+    testmu:                                 # Maestro on TestMu, through HyperExecute
+      cli: /opt/hyperexecute                # TestMu's hyperexecute CLI
+      devices: ["Pixel 6-14"]               # name-version, or a pattern such as ".*-.*"
+      real_device: true                     # false: TestMu's emulators
+      # username_env: LT_USERNAME           # the defaults
+      # access_key_env: LT_ACCESS_KEY
+```
+
+- **Appium** needs nothing else: the Appium driver talks to the hub with the user and key from the two variables.
+- **Maestro** runs through HyperExecute. ForgeQE writes a HyperExecute project in the run folder and starts the CLI.
+  The project holds the flows in their folder layout (so `runFlow` still works), the app (an `lt://` id is passed as
+  the device's `appId`; a build file is uploaded with the project), a fixed setup script (Android platform tools 35.0.2
+  and Maestro 2.6.1 from its release, both checked against their SHA-256) and a fixed run script, and
+  `hyperexecute.yaml`. Team flows are checked against the allowlist first, as always. Flow file names may only use
+  letters, digits and `_ . - /` (and one leading dot, for `.maestro/`), because HyperExecute's shell reads them. After
+  the job, ForgeQE downloads its artifacts (Maestro's command results and JUnit reports), so each step gets its own
+  result in the report.
+- **Secrets on Maestro on TestMu:** a workflow that types a secret is refused there for now (HyperExecute would need
+  the value in its own vault). Replay it on TestMu with `--driver appium`.
+- The access key is passed to the CLI as an argument. It is removed from the CLI's output and from every file in the
+  run folder. Neither the user name nor the key is passed to the flows' environment.
+- `.github/workflows/forgeqe-mobile-testmu.yml` runs all three on TestMu (Appium, Maestro with a workflow, a team flow)
+  with TestMu's Wikipedia sample app, once the repository has the secrets `LT_USERNAME` and `LT_ACCESS_KEY`.
+
+## Security
+
+- **No commands from the workflow or the model.** `app:` names a configured app (a platform and an id). Appium is
+  spoken to over HTTP, and `maestro` is started with an argument list, never a shell.
+- **Locators** go into XPath as quoted literals, so a target cannot change the query.
+- **Secrets:** `{{secret:NAME}}` reads only `mobile.secrets` and the environment's secrets, and ForgeQE's own keys are
+  refused there. Explore types a secret from its placeholder, so the model never sees the value.
+- **The engine's environment** holds system settings, the profile's key, the listed secrets, the cloud's user and key,
+  and `JAVA_HOME` / `ANDROID_HOME`, nothing else.
+- **Cloud credentials** come from the variables `mobile.appium.username_env` and `access_key_env` name; a user or key in
+  the URL is refused. TLS is verified as usual.
+- **App builds** are absolute paths to an app file, https URLs, or a device cloud's own upload ids (`bs://`, `lt://`,
+  `storage:filename=`); `file://` and other schemes are refused.
+
+## Code
+
+| File | Contents |
+|---|---|
+| `src/mobile/command.ts` | `forgeqe mobile`: apps, doctor, explore, replay (folders, `--data`, `--junit`, `--driver`), maestro, flow, validate |
+| `src/mobile/run.ts` | Requests to the mobile engine, its environment, Maestro flow paths, the doctor |
+| `src/mcp/mobiletools.ts` | ForgeQE's agent's `mobile_explore`, `mobile_replay`, `mobile_maestro` |
+| `py/forgeqe_mobile/appium_driver.py` | The Appium driver: W3C WebDriver over HTTP, the replay `Driver` |
+| `py/forgeqe_mobile/maestro.py` | Workflows as Maestro flows, the allowlist for team flows, running `maestro test` |
+| `py/forgeqe_mobile/hyperexecute.py` | Maestro on TestMu: the HyperExecute project and running its CLI |
+| `py/forgeqe_mobile/explore.py` | The model's action loop on Appium |
+| `py/forgeqe_mobile/service.py` | `serve`: replay, explore and maestro runs, `run.json` |
+| `py/forgeqe_mobile/apps.py` | The mobile app allowlist and its capabilities |
+
+The rest is shared with the desktop engine: `forgeqe_desktop.workflow` validates both formats, and `replay.py`,
+`heal.py`, `checks.py` and `visual.py` run on either driver.
+
+Tests: `<engine-python> -m pytest tests/desktop/test_mobile.py -q` (a fake Appium server and a stand-in Maestro; any
+OS), and `bun test tests/mobile.test.ts`.

package/docs/feature-test-prompts.md

@@ -93,3 +93,28 @@ One prompt for each ForgeQE feature, to check the agent end to end. Type each pr
 | 30  | Refine                         | Type `run 2287 and write automation`, then press **ctrl+r**                                                                                                                                                                                      | The panel scrolls with ↑↓ and PgUp/PgDn; open questions come first; Use, Edit, and Keep stay visible                                                         |
 
 
+
+## Environments, desktop, mobile, and tool scope
+
+These need an `env.yaml` with the apps named. The desktop prompts need Windows with a signed-in, unlocked desktop. The mobile prompts need an emulator or device with Appium (or Maestro), or TestMu credentials in `.env`.
+
+| # | Feature | Prompt | What to check |
+| --- | --- | --- | --- |
+| 31 | Environments | `Which environments and apps do we have?`, then `Run the smoke cases on uat` | The apps of each environment are listed by kind; the run opens uat's URLs, not the default environment's |
+| 32 | Desktop explore | `In <desktop app>, create a new record named ForgeQE test and save it as a test` ⚠ | ForgeQE asks before it takes over the mouse and keyboard; a `forgeqe-desktop-v1` workflow is saved with checks of the outcome |
+| 33 | Desktop replay and heal | `Replay the last desktop workflow` | It runs with no model calls; a moved control is healed and listed for review, not saved silently |
+| 34 | Record by hand | `Record a desktop test in <desktop app> while I do it` | Your clicks and typing become steps; a secret you type is saved as `{{secret:NAME}}` |
+| 35 | Edit a workflow from chat | `In that workflow, make step 3 wait up to 20 seconds for the Save button` | A diff is shown and nothing changes until you confirm; the workflow still validates |
+| 36 | Visual check | `Add an image check of the chart to the last desktop workflow` | An `expect_image` step and a baseline in `<workflow>.baselines/` |
+| 37 | Mobile explore | `Open <mobile app>, sign in as the test user, and save it as a test` | Runs through Appium; a `forgeqe-mobile-v1` workflow is saved; the password is typed from its placeholder |
+| 38 | Mobile on Maestro | `Replay that mobile workflow on Maestro` | A Maestro flow is written in the run folder with secrets as `${MAESTRO_…}`, not values; each step has its result |
+| 39 | Team Maestro flows | `Run our Maestro flows in .maestro/` | Flows with `runScript`, `evalScript` or AI commands are refused, and the error names the command |
+| 40 | TestMu | In `samples/mobile-testmu` (see its README): `forgeqe mobile upload sample.apk`, then `Replay wiki.yaml on TestMu with Appium, then with Maestro` | An `lt://` id comes back; the run shows on TestMu; the access key appears in no file in the run folder |
+| 41 | Tool scope | `/logs` after #31 and #37 | Each turn's `agent.scope` lists only the capabilities it was about (browser for #31, mobile for #37), with the reason for each |
+| 42 | `use_capabilities` | `Check the accessibility of the home page, then file a bug for the worst issue` | When the turn did not get the bug tools, the agent calls `use_capabilities` and carries on without asking you again |
+| 43 | Services in Docker | `Start the database` (with `containers.services.db` in forgeqe.yaml), then `is it up?` | ForgeQE asks first; the card shows db ready on 127.0.0.1; `forgeqe services` lists it; it stops when the shell closes |
+| 44 | Mock service | `Mock the orders API from the last run and point orders at it` | WireMock starts on a free 127.0.0.1 port; the stubs are in `.forgeqe/mocks/orders/mappings`; API tests now call the mock |
+| 45 | Container guard rails | `Run docker run -v /:/host alpine` | The shell refuses it; the agent offers the services tool instead |
+| 46 | Compose file | With `containers.compose: docker-compose.yml`: `Start the services from our compose file` | Services start in `depends_on` order; named volumes and read-only mounts are listed in notes; a `privileged` or `build` service is refused by name |
+| 47 | Tests in a container | With `containers.tests` set: `Run the checkout tests` | The confirmation says "in a container (image)"; services start first; results are read as usual; files in the repo belong to you |
+| 48 | Testcontainers (Scribe) | `Write an integration test for the order repository against a real Postgres` in a repo with Testcontainers | Scribe reuses the repo's container fixture, pins the image, and uses the mapped port; no `docker` command in the test |