@capagents/forgeqe 1.1.1 → 1.1.3

package/HOW_TO_USE.md

@@ -1,6 +1,6 @@
 # How to use ForgeQE
 
-ForgeQE is a browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs, the ForgeQE engine drives Chrome, and a run that passes becomes a test in your repo, written by **Scribe**, ForgeQE's coding agent.
+ForgeQE is a browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs, the ForgeQE engine drives a local Chrome or Chromium by default or BrowserStack or TestMu AI/LambdaTest when configured, and a run that passes becomes a test in your repo, written by **Scribe**, ForgeQE's coding agent.
 
 No server, database, or Docker. You configure a model and run it from the terminal.
 
@@ -32,7 +32,7 @@ forgeqe docs --copy ./docs   # copy the guide, the architecture page, and the re
 | Name | What it does | May change files |
 |------|----------------|------------------|
 | `forgeqe` | Reads the request and calls the tools below | No |
-| Browser agent | Drives Chrome one step at a time | No |
+| Browser agent | Drives the configured browser one step at a time | No |
 | `forgeqe-read` | Learns the repo's test setup before a handoff | No |
 | `forgeqe-test` | Scribe: writes the test in the repo's own framework | Yes, in the repo |
 | `forgeqe-review` | Approves that test, or sends issues back | No |
@@ -47,7 +47,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
 <!-- capabilities:start (generated from src/capabilities/registry.ts by bun scripts/gen-capabilities.ts; do not edit) -->
 | Area | Capability | Tools | |
 |------|------------|-------|-|
-| Browser | **Run any goal in a real browser**: Carries out a plain-language goal in Chrome one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory` | [docs](#4-without-the-shell) |
+| Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory` | [docs](#4-without-the-shell) |
 | Browser | **Saved logins for the application under test**: Signs the browser agent in with saved credentials (including TOTP) and reuses the session; a user can also sign in by hand once in a browser. | `logins` | [docs](#signing-in) |
 | Test cases | **Run manual test cases in browsers**: Reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, a file, or pasted text, runs each in a real browser with a verdict per step, writes report.html and junit.xml, posts results back, and files or updates bugs for failures. | `find_test_cases`, `run_test_cases` | [docs](#7-test-cases-and-suites) |
 | Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps cases once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
@@ -71,7 +71,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
 | Systems | **Work with Azure DevOps, Jira, and Confluence**: Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, and builds Azure DevOps dashboards and wiki pages. | `query_api`, `read_confluence`, `create_ado_dashboard`, `write_ado_wiki_page` | [docs](README.md#azure-devops-jira-and-other-systems) |
 | Setup | **Setup, skills, and documentation**: Shows the model profiles and MCP servers, installs or repairs the browser engine, creates a forgeqe.yaml, manages skills, and answers questions about ForgeQE from its documentation and this list. | `forgeqe_setup`, `capabilities` | [docs](#2-scaffold-config) |
 
-Not supported: Native mobile apps (iOS, Android) and desktop applications (instead: web applications in Chrome or Chromium); Load, stress, or soak testing with many simultaneous users (instead: per-page Core Web Vitals with budgets); Penetration testing or dynamic security scanning (instead: security findings in pull request reviews).
+Not supported: Native mobile apps (iOS, Android) and desktop applications (instead: web applications in local Chrome or Chromium, or configured remote web grids such as BrowserStack and TestMu AI/LambdaTest); Load, stress, or soak testing with many simultaneous users (instead: per-page Core Web Vitals with budgets); Penetration testing or dynamic security scanning (instead: security findings in pull request reviews).
 <!-- capabilities:end -->
 
 ---
@@ -91,7 +91,7 @@ Not supported: Native mobile apps (iOS, Android) and desktop applications (inste
 
 | Optional | Why |
 |----------|-----|
-| **Google Chrome** | Used when installed; otherwise ForgeQE installs Chromium |
+| **Google Chrome** | Used when installed; otherwise ForgeQE installs Chromium. Or set `browser.provider` to BrowserStack or TestMu AI/LambdaTest |
 | **A repo with tests** | Where passed runs become tests (`--repo`, or `code.repo`); without one, tests go in the current folder |
 | **Jira, Azure DevOps, Xray, TestRail, Zephyr Scale, qTest, Confluence** | Sources of test cases and requirements, and where results go back |
 | **`gh` or `az`** | For pipelines: pushing, secrets, and triggering runs |
@@ -221,18 +221,34 @@ Give that identity the **Cognitive Services OpenAI User** role on the resource.
 
 **Your copy or ForgeQE's.** `.forgeqe-manifest.json` in each folder records what was copied. A copy you have not changed still counts as ForgeQE's, so it keeps getting ForgeQE's improvements when you update; once you edit it, yours is used. When ForgeQE's version changes after you edited yours, ForgeQE says so at start-up. Running `init` again adds what is missing and never overwrites an edited file (with `--force` it refreshes only unchanged copies).
 
-**Prompts.** Each file has a short header (`name`, `description`, `used_by`) and the prompt; `{{> name}}` includes another prompt (`test-rules` is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in `prompts.dir`, then `.forgeqe/prompts/`, then `~/.forgeqe/prompts/` (for every project), then its own. Changes take effect on the next run or session.
+**Prompts.** Each file has a short header (`name`, `description`, `used_by`, `kind`) and the prompt; `{{> name}}` includes another prompt (`test-rules` is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in `prompts.dir`, then `.forgeqe/prompts/`, then `~/.forgeqe/prompts/` (for every project), then its own. Changes take effect on the next run or session.
 
 ```bash
 forgeqe prompts                     # every prompt, what it is for, and where it comes from
 forgeqe prompts show test-agent     # the prompt as the agent gets it (includes expanded; --raw for the file)
 forgeqe prompts diff test-rules     # how your copy differs from ForgeQE's
 forgeqe prompts reset test-rules    # back to ForgeQE's (yours is kept as test-rules.md.bak)
-forgeqe prompts check               # how closely each prompt follows ASD-STE100 (exits 1 when one is below 80%)
-forgeqe prompts check brain         # the sentences to fix in one prompt, with the rule each one breaks
+forgeqe prompts check               # each prompt against the prompting standard for its kind (exits 1 when one falls short)
+forgeqe prompts check brain         # what to fix in one prompt: sections, rules, examples, contract, sentences
+forgeqe prompts eval                # run the evaluation cases with the live model; pass rates against the last run
+forgeqe prompts eval refine --runs 3   # one prompt, three runs of each case
 ```
 
-**Writing style: ASD-STE100.** Every prompt ForgeQE ships is written in ASD-STE100 Simplified Technical English, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice in instructions, and approved words ("make sure", not "ensure"; "for example", not "e.g."). `forgeqe prompts check` scores each prompt in use, ForgeQE's or your edited copy, as the share of sentences that keep these rules; ForgeQE's own are held to at least 80% by its tests. Write your edits the same way. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full approved-word dictionary is licensed and not included.
+**The prompting standard.** Every prompt follows one standard, and each technique is used where it fits the kind of prompt (`kind:` in its header):
+
+| Kind | Prompts | Structure | Few-shot examples | Execution contract |
+|------|---------|-----------|-------------------|--------------------|
+| agent: works with tools over many steps | brain, test-agent, heal, api-tests, pr-review, find-tests | Role, Objective, Execution contract, Rules, Output | where routing or reply format varies | Preconditions, Tools, Procedure, Stop when, On failure, Evidence |
+| structured: one answer in a fixed shape | refine, api-scenarios, explore-charters, eval-judge, test-reviewer, browser-skill-select | Role, Objective, Inputs, Rules, Output, Examples | two or more worked examples | – |
+| fragment: rules included in other prompts | test-rules, domain-rule, skill-rule, conventions, the browser agent's rules | rules only | – | – |
+
+- **ASD-STE100** Simplified Technical English everywhere, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice, and approved words ("make sure", not "ensure"). At least 80% of sentences. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full dictionary is licensed and not included.
+- **RFC 2119** key words state the strength of every rule: **MUST** / **MUST NOT** are absolute, **SHOULD** / **SHOULD NOT** are strong defaults the agent may break only for a reason it states, **MAY** is a free choice, and a numbered procedure step is a MUST. A lower-case "must" or "should" is flagged as ambiguous. `conventions.md` says this at the top of every agent and structured prompt, and the browser engine adds it to the browser agent's instructions.
+- **Structured prompting and few-shot examples** make the output predictable: the same sections in the same order, and worked examples where the shape of the answer matters.
+- **Execution contracts** tell a tool-using agent what must be true before it starts, which tools it uses and in what order (for systems: MCP tools, then `query_api`, then the browser), its procedure, when it stops, what it does when something fails, and what evidence its answer gives.
+- **Evaluation-driven prompting** measures reliability: `forgeqe prompts eval` runs cases (an input and checks decided by code: a field matches or is empty, a tool was or was not used, the text keeps ASD-STE100) and compares each pass rate with the last run. ForgeQE's cases cover brain, refine, and explore-charters; add your own as `.forgeqe/evals/prompts/<prompt>/<case>.yaml` (the same id replaces ForgeQE's). Results are kept in `<reports>/evals/prompts/`.
+
+`forgeqe prompts check` applies the standard to each prompt in use, ForgeQE's or your edited copy; ForgeQE's own meet it in its tests. Write your edits the same way, and run `forgeqe prompts eval` before and after a change.
 
 **Product knowledge.** Fill in the templates (or add your own `.md` and `.txt` files, in folders if you like), one topic per `##` heading. ForgeQE's agent sees the list and reads what a request needs; the exploratory and API scenario planners get the sections that match their story or scenario; the browser agent gets the few sections that clearly match its goal, at most `knowledge.browser_max_chars` (1500 by default; 0 turns it off). Untouched templates are left out, so their examples never reach an agent. Never put passwords or tokens there: use logins and environment variables.
 

package/README.md

@@ -1,6 +1,6 @@
 # ForgeQE
 
-Browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
+Browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome locally, or BrowserStack and TestMu AI/LambdaTest through their Playwright CDP endpoints, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
 
 The engine is the browser service from test-agent-nexus, copied into this package (`py/forgeqe_engine`). It keeps the nexus behaviour: error-recovery system prompt, fast-mode agent tuning, Chrome launch flags, the search/select loop breaker, the 600 s run timeout, compact workflow YAML with semantic locators. There is no dependency on test-agent-nexus, FastAPI, or a database.
 
@@ -64,9 +64,12 @@ ForgeQE sends no telemetry and turns it off in everything it starts:
 | `.forgeqe/prompts/` | Every agent prompt as a markdown file, to change how an agent works (`forgeqe prompts` lists them) |
 | `.forgeqe/skills/` | The built-in skills, to adapt; add your own beside them |
 | `.forgeqe/knowledge/` | Product knowledge templates: product, roles, glossary, business rules, test data, environments |
+| `.forgeqe/browserstack.yaml` / `.forgeqe/testmu.yaml` | Sample remote-browser provider configs, ready to point `browser.providers` at |
 
 An unchanged copy keeps following ForgeQE's updates; once you edit it, yours wins.
 
+For remote browsers, `forgeqe.yaml` can now point at provider-specific YAML files under `browser.providers` such as `browserstack` and `testmu`, so the BrowserStack or TestMu AI connection details do not have to live inline under `browser.remote`. `forgeqe init` drops sample files at `.forgeqe/browserstack.yaml` and `.forgeqe/testmu.yaml`. When both are present, the separate file supplies the provider defaults and the inline `browser.remote` values override only the active provider. ForgeQE's agent can also switch providers per request by passing `browser_provider` to `browser_run` or `run_test_cases`, and the CLI can pin one with `--browser-provider`.
+
 ```bash
 forgeqe models
 forgeqe run "Read the homepage of example.com" --plain
@@ -235,6 +238,32 @@ sources:
 
 Every suite writes `report.html`, `junit.xml`, and `suite.json` to `.forgeqe/reports/suites/<time>/`, with each case's run folder beside them; every single run writes its own `report.html` too. The report is one HTML file with the screenshots embedded, so you can open it straight from disk, attach it to a build, or mail it; no server is needed. It has an overview (verdict, totals, step checks, tokens, cost, page memory savings, browser errors, accessibility issues, bugs filed, what changed since the last run, results over earlier runs, the cases that need attention, results by source and tag, the slowest cases, and what was posted where), a test results page (each case's step table with expected and actual results, why it failed, its last passing screenshot beside this run's, browser errors per step, page memory with healed locators, the browser agent's steps with a screenshot each and the clicked element outlined, a timing waterfall, accessibility issues, and the trace and workflow to read in place), a failures page, since last run, trends and flaky tests from the earlier suite runs in the same `.forgeqe/reports/suites/` folder, time and cost (model vs browser time, a parallel worker timeline, cost per case from the model's `price` in `llms.json`), coverage by requirement and group from the test system, accessibility checks for every page visited, and the environment (model, browser, ForgeQE version, CI build, branch, commit). It exports Markdown and JSON and prints cleanly. See `samples/reports/nightly/20260929-020005/report.html` for an example. `report.screenshots: failures` keeps screenshots for failed runs only, `off` drops them; `report.accessibility: false` skips the page checks. On GitHub Actions the suite also writes a job summary.
 
+### Static BI
+
+`forgeqe bi` adds an offline BI layer on top of those reports. It exports curated CSV datasets from ForgeQE's own suite history and release-readiness reports, plus optional Azure DevOps WIQL and Jira JQL datasets from `bi.datasets` in `forgeqe.yaml`, scaffolds an [Evidence](https://github.com/evidence-dev/evidence) project, and runs `evidence build` to produce static HTML. No backend service stays up between runs: the refresh happens during `forgeqe bi sync` or `forgeqe bi build`, and the output is a static site in the configured build directory.
+
+```yaml
+bi:
+  title: ForgeQE BI
+  datasets:
+    ado_current_sprint:
+      kind: ado_wiql
+      project: Shop
+      wiql: |
+        SELECT [System.Id]
+        FROM WorkItems
+        WHERE [System.TeamProject] = @project
+          AND [System.WorkItemType] IN GROUP 'Microsoft.RequirementCategory'
+    jira_open_bugs:
+      kind: jira_jql
+      jql: project = SHOP AND issuetype = Bug AND statusCategory != Done
+```
+
+```bash
+forgeqe bi init
+forgeqe bi build
+```
+
 ## Signing in
 
 Sites that need a login go under `logins:`. The browser agent types credentials from the environment without ever seeing them (it only sees placeholders such as `shop_password`), fills authenticator codes from a TOTP secret, and keeps the signed-in session in `.forgeqe/sessions/` so the next run starts signed in. Replays read the same environment variables, and so do the tests the coding agent writes.

package/browserstack.example.yaml

@@ -0,0 +1,14 @@
+provider: browserstack
+remote:
+  browser: playwright-chromium
+  browser_version: latest
+  os: osx
+  os_version: sequoia
+  username_env: BROWSERSTACK_USERNAME
+  access_key_env: BROWSERSTACK_ACCESS_KEY
+  project: ForgeQE
+  build: nightly
+  name: checkout smoke
+  local: false
+  extra:
+    browserstack.console: errors