@capagents/forgeqe 1.1.0 → 1.1.2

package/HOW_TO_USE.md

@@ -221,18 +221,34 @@ Give that identity the **Cognitive Services OpenAI User** role on the resource.
 
 **Your copy or ForgeQE's.** `.forgeqe-manifest.json` in each folder records what was copied. A copy you have not changed still counts as ForgeQE's, so it keeps getting ForgeQE's improvements when you update; once you edit it, yours is used. When ForgeQE's version changes after you edited yours, ForgeQE says so at start-up. Running `init` again adds what is missing and never overwrites an edited file (with `--force` it refreshes only unchanged copies).
 
-**Prompts.** Each file has a short header (`name`, `description`, `used_by`) and the prompt; `{{> name}}` includes another prompt (`test-rules` is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in `prompts.dir`, then `.forgeqe/prompts/`, then `~/.forgeqe/prompts/` (for every project), then its own. Changes take effect on the next run or session.
+**Prompts.** Each file has a short header (`name`, `description`, `used_by`, `kind`) and the prompt; `{{> name}}` includes another prompt (`test-rules` is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in `prompts.dir`, then `.forgeqe/prompts/`, then `~/.forgeqe/prompts/` (for every project), then its own. Changes take effect on the next run or session.
 
 ```bash
 forgeqe prompts                     # every prompt, what it is for, and where it comes from
 forgeqe prompts show test-agent     # the prompt as the agent gets it (includes expanded; --raw for the file)
 forgeqe prompts diff test-rules     # how your copy differs from ForgeQE's
 forgeqe prompts reset test-rules    # back to ForgeQE's (yours is kept as test-rules.md.bak)
-forgeqe prompts check               # how closely each prompt follows ASD-STE100 (exits 1 when one is below 80%)
-forgeqe prompts check brain         # the sentences to fix in one prompt, with the rule each one breaks
+forgeqe prompts check               # each prompt against the prompting standard for its kind (exits 1 when one falls short)
+forgeqe prompts check brain         # what to fix in one prompt: sections, rules, examples, contract, sentences
+forgeqe prompts eval                # run the evaluation cases with the live model; pass rates against the last run
+forgeqe prompts eval refine --runs 3   # one prompt, three runs of each case
 ```
 
-**Writing style: ASD-STE100.** Every prompt ForgeQE ships is written in ASD-STE100 Simplified Technical English, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice in instructions, and approved words ("make sure", not "ensure"; "for example", not "e.g."). `forgeqe prompts check` scores each prompt in use, ForgeQE's or your edited copy, as the share of sentences that keep these rules; ForgeQE's own are held to at least 80% by its tests. Write your edits the same way. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full approved-word dictionary is licensed and not included.
+**The prompting standard.** Every prompt follows one standard, and each technique is used where it fits the kind of prompt (`kind:` in its header):
+
+| Kind | Prompts | Structure | Few-shot examples | Execution contract |
+|------|---------|-----------|-------------------|--------------------|
+| agent: works with tools over many steps | brain, test-agent, heal, api-tests, pr-review, find-tests | Role, Objective, Execution contract, Rules, Output | where routing or reply format varies | Preconditions, Tools, Procedure, Stop when, On failure, Evidence |
+| structured: one answer in a fixed shape | refine, api-scenarios, explore-charters, eval-judge, test-reviewer, browser-skill-select | Role, Objective, Inputs, Rules, Output, Examples | two or more worked examples | – |
+| fragment: rules included in other prompts | test-rules, domain-rule, skill-rule, conventions, the browser agent's rules | rules only | – | – |
+
+- **ASD-STE100** Simplified Technical English everywhere, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice, and approved words ("make sure", not "ensure"). At least 80% of sentences. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full dictionary is licensed and not included.
+- **RFC 2119** key words state the strength of every rule: **MUST** / **MUST NOT** are absolute, **SHOULD** / **SHOULD NOT** are strong defaults the agent may break only for a reason it states, **MAY** is a free choice, and a numbered procedure step is a MUST. A lower-case "must" or "should" is flagged as ambiguous. `conventions.md` says this at the top of every agent and structured prompt, and the browser engine adds it to the browser agent's instructions.
+- **Structured prompting and few-shot examples** make the output predictable: the same sections in the same order, and worked examples where the shape of the answer matters.
+- **Execution contracts** tell a tool-using agent what must be true before it starts, which tools it uses and in what order (for systems: MCP tools, then `query_api`, then the browser), its procedure, when it stops, what it does when something fails, and what evidence its answer gives.
+- **Evaluation-driven prompting** measures reliability: `forgeqe prompts eval` runs cases (an input and checks decided by code: a field matches or is empty, a tool was or was not used, the text keeps ASD-STE100) and compares each pass rate with the last run. ForgeQE's cases cover brain, refine, and explore-charters; add your own as `.forgeqe/evals/prompts/<prompt>/<case>.yaml` (the same id replaces ForgeQE's). Results are kept in `<reports>/evals/prompts/`.
+
+`forgeqe prompts check` applies the standard to each prompt in use, ForgeQE's or your edited copy; ForgeQE's own meet it in its tests. Write your edits the same way, and run `forgeqe prompts eval` before and after a change.
 
 **Product knowledge.** Fill in the templates (or add your own `.md` and `.txt` files, in folders if you like), one topic per `##` heading. ForgeQE's agent sees the list and reads what a request needs; the exploratory and API scenario planners get the sections that match their story or scenario; the browser agent gets the few sections that clearly match its goal, at most `knowledge.browser_max_chars` (1500 by default; 0 turns it off). Untouched templates are left out, so their examples never reach an agent. Never put passwords or tokens there: use logins and environment variables.
 
@@ -467,23 +483,24 @@ The chat gets a short summary and a card with both paths as links, and **Open th
 
 **`/` and `@` in the prompt.** Typing `/` lists the commands that match; typing `@` suggests references: recent runs (*last run*, then each by its folder), files in the repository by name or path (`@Login` finds `pages/Login.ts`), and, after `@` and a number such as `@1761`, the Azure DevOps work item with its title (read-only). Tab completes, ↑/↓ choose, Esc closes the list.
 
-**Refine.** Ctrl-R (or *Refine my request* in the palette) turns what you typed into a task specification for ForgeQE's agent, in ASD-STE100 Simplified Technical English, before it is sent:
+**Refine.** Ctrl-R (or *Refine my request* in the palette) turns what you typed into a task specification for ForgeQE's agent, laid out as a system prompt and written in ASD-STE100 Simplified Technical English, before it is sent. It has these parts, in this order: **Role** (who the agent acts as), **Objective** (one instruction), **Context** (the facts, with names), **Steps** (only when they add something), **Rules** (your limits and the setup's), **Output format** (what comes back and how it is laid out), and **Summary** (what you get, in one sentence, read last):
 
 ```
-Goal: Check the Azure DevOps QI-Payments pipelines and make a shareable report for the manager.
+Role: A release analyst who reads Azure DevOps deployments.
+
+Objective: Get what is deployed now in each environment of each Azure DevOps project.
 
 Context:
-- Project: Azure DevOps project QI-Payments.
-- Pipeline: build definition 32, QI-Payments-CD.
-- Latest checked CD build: build 1021, failed, finished 2026-09-21.
+- System: Azure DevOps organization devcgazuredevops.
+- Projects: Autothon, CQube.AI, BlackBox_AI, QI-Payments, QI-MobileBanking, Atlas.Ai, QI-Claims.
 
-Steps:
-1. Review the recent runs of QI-Payments-CI and QI-Payments-CD.
-2. Make a shareable report with a short manager summary.
+Output format: One section for each project, with a line for each environment: the deployed build, its date, and its status.
 
-Result: A shareable report file with a short summary for the manager.
+Summary: Show the deployed build in each environment, and why each deployment failed.
 ```
 
+Refine is one model session of its own, separate from the conversation: a few model steps (usually two) and read-only tool calls, typically 15K–25K tokens and 20–30 seconds. The panel title shows what it cost and how much of the request keeps the ASD-STE100 rules (*✦ Refined request · STE 100% · 20.8K tokens · 21s*), and its tokens count in the session total in the footer.
+
 A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it **checked** ✓, what it **assumed** ~, and what is **missing** ? (for example "Which story? Give the work item id"); a capability that is not set up shows as missing too. Enter sends it, `e` puts it in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.
 
 **Next steps.** When a request finishes, up to three suggestions appear above the prompt: from what it produced (*Open the report*, *Write a test for this run*, *Why did it fail?*, *Rerun the cases that failed*) and from the agent's answer, which suggests the follow-ups you are likely to want (*Write a release report*, *Show the failed CD logs*). Click one or press Alt-1 to Alt-3; suggestions that start work are put in the prompt first, to adjust and send.

package/README.md

@@ -1,6 +1,6 @@
 # ForgeQE
 
-Browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
+Browser agent CLI. Say what you want in plain language: a goal to carry out in a browser, test cases to run from Jira, Azure DevOps, Xray or a file, a test to write, or a CI pipeline to set up. ForgeQE's agent works out what that needs and calls ForgeQE's tools. The ForgeQE engine drives Chrome locally, or BrowserStack and TestMu AI/LambdaTest through their Playwright CDP endpoints, and a run that passes becomes a test in your repo's own language and framework, written by **Scribe**, ForgeQE's coding agent. Skills (`SKILL.md` folders, built in or in `.forgeqe/skills`) teach the agents your repo's conventions and the apps you test.
 
 The engine is the browser service from test-agent-nexus, copied into this package (`py/forgeqe_engine`). It keeps the nexus behaviour: error-recovery system prompt, fast-mode agent tuning, Chrome launch flags, the search/select loop breaker, the 600 s run timeout, compact workflow YAML with semantic locators. There is no dependency on test-agent-nexus, FastAPI, or a database.
 
@@ -235,6 +235,32 @@ sources:
 
 Every suite writes `report.html`, `junit.xml`, and `suite.json` to `.forgeqe/reports/suites/<time>/`, with each case's run folder beside them; every single run writes its own `report.html` too. The report is one HTML file with the screenshots embedded, so you can open it straight from disk, attach it to a build, or mail it; no server is needed. It has an overview (verdict, totals, step checks, tokens, cost, page memory savings, browser errors, accessibility issues, bugs filed, what changed since the last run, results over earlier runs, the cases that need attention, results by source and tag, the slowest cases, and what was posted where), a test results page (each case's step table with expected and actual results, why it failed, its last passing screenshot beside this run's, browser errors per step, page memory with healed locators, the browser agent's steps with a screenshot each and the clicked element outlined, a timing waterfall, accessibility issues, and the trace and workflow to read in place), a failures page, since last run, trends and flaky tests from the earlier suite runs in the same `.forgeqe/reports/suites/` folder, time and cost (model vs browser time, a parallel worker timeline, cost per case from the model's `price` in `llms.json`), coverage by requirement and group from the test system, accessibility checks for every page visited, and the environment (model, browser, ForgeQE version, CI build, branch, commit). It exports Markdown and JSON and prints cleanly. See `samples/reports/nightly/20260929-020005/report.html` for an example. `report.screenshots: failures` keeps screenshots for failed runs only, `off` drops them; `report.accessibility: false` skips the page checks. On GitHub Actions the suite also writes a job summary.
 
+### Static BI
+
+`forgeqe bi` adds an offline BI layer on top of those reports. It exports curated CSV datasets from ForgeQE's own suite history and release-readiness reports, plus optional Azure DevOps WIQL and Jira JQL datasets from `bi.datasets` in `forgeqe.yaml`, scaffolds an [Evidence](https://github.com/evidence-dev/evidence) project, and runs `evidence build` to produce static HTML. No backend service stays up between runs: the refresh happens during `forgeqe bi sync` or `forgeqe bi build`, and the output is a static site in the configured build directory.
+
+```yaml
+bi:
+  title: ForgeQE BI
+  datasets:
+    ado_current_sprint:
+      kind: ado_wiql
+      project: Shop
+      wiql: |
+        SELECT [System.Id]
+        FROM WorkItems
+        WHERE [System.TeamProject] = @project
+          AND [System.WorkItemType] IN GROUP 'Microsoft.RequirementCategory'
+    jira_open_bugs:
+      kind: jira_jql
+      jql: project = SHOP AND issuetype = Bug AND statusCategory != Done
+```
+
+```bash
+forgeqe bi init
+forgeqe bi build
+```
+
 ## Signing in
 
 Sites that need a login go under `logins:`. The browser agent types credentials from the environment without ever seeing them (it only sees placeholders such as `shop_password`), fills authenticator codes from a TOTP secret, and keeps the signed-in session in `.forgeqe/sessions/` so the next run starts signed in. Replays read the same environment variables, and so do the tests the coding agent writes.

package/docs/user-guide.html

@@ -423,15 +423,35 @@ bun src/index.ts init --global --profile azure-gpt4o
 </tbody>
 </table></div>
 <p><strong>Your copy or ForgeQE's.</strong> <code>.forgeqe-manifest.json</code> in each folder records what was copied. A copy you have not changed still counts as ForgeQE's, so it keeps getting ForgeQE's improvements when you update; once you edit it, yours is used. When ForgeQE's version changes after you edited yours, ForgeQE says so at start-up. Running <code>init</code> again adds what is missing and never overwrites an edited file (with <code>--force</code> it refreshes only unchanged copies).</p>
-<p><strong>Prompts.</strong> Each file has a short header (<code>name</code>, <code>description</code>, <code>used_by</code>) and the prompt; <code>{{&gt; name}}</code> includes another prompt (<code>test-rules</code> is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in <code>prompts.dir</code>, then <code>.forgeqe/prompts/</code>, then <code>~/.forgeqe/prompts/</code> (for every project), then its own. Changes take effect on the next run or session.</p>
+<p><strong>Prompts.</strong> Each file has a short header (<code>name</code>, <code>description</code>, <code>used_by</code>, <code>kind</code>) and the prompt; <code>{{&gt; name}}</code> includes another prompt (<code>test-rules</code> is included in the test, reviewer, and healing prompts, so one edit changes all three). ForgeQE looks in <code>prompts.dir</code>, then <code>.forgeqe/prompts/</code>, then <code>~/.forgeqe/prompts/</code> (for every project), then its own. Changes take effect on the next run or session.</p>
 <pre><code class="language-bash">forgeqe prompts                     # every prompt, what it is for, and where it comes from
 forgeqe prompts show test-agent     # the prompt as the agent gets it (includes expanded; --raw for the file)
 forgeqe prompts diff test-rules     # how your copy differs from ForgeQE's
 forgeqe prompts reset test-rules    # back to ForgeQE's (yours is kept as test-rules.md.bak)
-forgeqe prompts check               # how closely each prompt follows ASD-STE100 (exits 1 when one is below 80%)
-forgeqe prompts check brain         # the sentences to fix in one prompt, with the rule each one breaks
+forgeqe prompts check               # each prompt against the prompting standard for its kind (exits 1 when one falls short)
+forgeqe prompts check brain         # what to fix in one prompt: sections, rules, examples, contract, sentences
+forgeqe prompts eval                # run the evaluation cases with the live model; pass rates against the last run
+forgeqe prompts eval refine --runs 3   # one prompt, three runs of each case
 </code></pre>
-<p><strong>Writing style: ASD-STE100.</strong> Every prompt ForgeQE ships is written in ASD-STE100 Simplified Technical English, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice in instructions, and approved words (&quot;make sure&quot;, not &quot;ensure&quot;; &quot;for example&quot;, not &quot;e.g.&quot;). <code>forgeqe prompts check</code> scores each prompt in use, ForgeQE's or your edited copy, as the share of sentences that keep these rules; ForgeQE's own are held to at least 80% by its tests. Write your edits the same way. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full approved-word dictionary is licensed and not included.</p>
+<p><strong>The prompting standard.</strong> Every prompt follows one standard, and each technique is used where it fits the kind of prompt (<code>kind:</code> in its header):</p>
+<div class="tbl"><table>
+<thead>
+<tr><th>Kind</th><th>Prompts</th><th>Structure</th><th>Few-shot examples</th><th>Execution contract</th></tr>
+</thead>
+<tbody>
+<tr><td>agent: works with tools over many steps</td><td>brain, test-agent, heal, api-tests, pr-review, find-tests</td><td>Role, Objective, Execution contract, Rules, Output</td><td>where routing or reply format varies</td><td>Preconditions, Tools, Procedure, Stop when, On failure, Evidence</td></tr>
+<tr><td>structured: one answer in a fixed shape</td><td>refine, api-scenarios, explore-charters, eval-judge, test-reviewer, browser-skill-select</td><td>Role, Objective, Inputs, Rules, Output, Examples</td><td>two or more worked examples</td><td>–</td></tr>
+<tr><td>fragment: rules included in other prompts</td><td>test-rules, domain-rule, skill-rule, conventions, the browser agent's rules</td><td>rules only</td><td>–</td><td>–</td></tr>
+</tbody>
+</table></div>
+<ul>
+<li><strong>ASD-STE100</strong> Simplified Technical English everywhere, so each agent reads one clear instruction at a time: an instruction has 20 words or fewer and a description 25 or fewer, one instruction per sentence, the active voice, and approved words (&quot;make sure&quot;, not &quot;ensure&quot;). At least 80% of sentences. The check covers the rules that can be measured in text and a list of common words the standard does not approve; the standard's full dictionary is licensed and not included.</li>
+<li><strong>RFC 2119</strong> key words state the strength of every rule: <strong>MUST</strong> / <strong>MUST NOT</strong> are absolute, <strong>SHOULD</strong> / <strong>SHOULD NOT</strong> are strong defaults the agent may break only for a reason it states, <strong>MAY</strong> is a free choice, and a numbered procedure step is a MUST. A lower-case &quot;must&quot; or &quot;should&quot; is flagged as ambiguous. <code>conventions.md</code> says this at the top of every agent and structured prompt, and the browser engine adds it to the browser agent's instructions.</li>
+<li><strong>Structured prompting and few-shot examples</strong> make the output predictable: the same sections in the same order, and worked examples where the shape of the answer matters.</li>
+<li><strong>Execution contracts</strong> tell a tool-using agent what must be true before it starts, which tools it uses and in what order (for systems: MCP tools, then <code>query_api</code>, then the browser), its procedure, when it stops, what it does when something fails, and what evidence its answer gives.</li>
+<li><strong>Evaluation-driven prompting</strong> measures reliability: <code>forgeqe prompts eval</code> runs cases (an input and checks decided by code: a field matches or is empty, a tool was or was not used, the text keeps ASD-STE100) and compares each pass rate with the last run. ForgeQE's cases cover brain, refine, and explore-charters; add your own as <code>.forgeqe/evals/prompts/&lt;prompt&gt;/&lt;case&gt;.yaml</code> (the same id replaces ForgeQE's). Results are kept in <code>&lt;reports&gt;/evals/prompts/</code>.</li>
+</ul>
+<p><code>forgeqe prompts check</code> applies the standard to each prompt in use, ForgeQE's or your edited copy; ForgeQE's own meet it in its tests. Write your edits the same way, and run <code>forgeqe prompts eval</code> before and after a change.</p>
 <p><strong>Product knowledge.</strong> Fill in the templates (or add your own <code>.md</code> and <code>.txt</code> files, in folders if you like), one topic per <code>##</code> heading. ForgeQE's agent sees the list and reads what a request needs; the exploratory and API scenario planners get the sections that match their story or scenario; the browser agent gets the few sections that clearly match its goal, at most <code>knowledge.browser_max_chars</code> (1500 by default; 0 turns it off). Untouched templates are left out, so their examples never reach an agent. Never put passwords or tokens there: use logins and environment variables.</p>
 </div></section>
 <section class="doc sub" id="doc-how_to_use-signing-in-to-your-tools"><h4>Signing in to your tools</h4><div class="prose"><p>Instead of copying tokens into <code>.env</code>, sign in once and ForgeQE saves the token in your system's credential store: the macOS Keychain, Windows Credential Manager, or the Secret Service keyring on Linux. On a machine without one, such as a headless Linux box, it uses <code>~/.forgeqe/credentials.json</code>, readable only by you.</p>
@@ -622,20 +642,20 @@ Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make i
 <p><strong>Setup checklist.</strong> Until everything is set up, the start screen's <strong>Setup</strong> line shows how many of the five things most features rest on are set up, and names the ones that are not: <strong>Model</strong> (a live model profile), <strong>Coding agent</strong>, <strong>Work items</strong> (Azure DevOps, Jira, TestRail, Zephyr, qTest, or Xray), <strong>Domain</strong>, and <strong>Notifications</strong>. <strong>▸ how to set up</strong> puts the steps for each missing one in the chat (for <strong>Work items</strong>, type <code>/auth</code>). The checks are the same ones <code>/features</code> uses, so the two always agree.</p>
 <p><strong>Command palette.</strong> Ctrl-P opens a palette of every action, command, profile, and example request; type a few letters to find one (&quot;rr&quot; finds <em>Refine my request</em>, &quot;auth&quot; finds the sign-in commands), ↑/↓ to choose, Enter to run it. Example requests are put in the prompt for you to edit, never sent straight away.</p>
 <p><strong><code>/</code> and <code>@</code> in the prompt.</strong> Typing <code>/</code> lists the commands that match; typing <code>@</code> suggests references: recent runs (<em>last run</em>, then each by its folder), files in the repository by name or path (<code>@Login</code> finds <code>pages/Login.ts</code>), and, after <code>@</code> and a number such as <code>@1761</code>, the Azure DevOps work item with its title (read-only). Tab completes, ↑/↓ choose, Esc closes the list.</p>
-<p><strong>Refine.</strong> Ctrl-R (or <em>Refine my request</em> in the palette) turns what you typed into a task specification for ForgeQE's agent, in ASD-STE100 Simplified Technical English, before it is sent:</p>
-<pre><code>Goal: Check the Azure DevOps QI-Payments pipelines and make a shareable report for the manager.
+<p><strong>Refine.</strong> Ctrl-R (or <em>Refine my request</em> in the palette) turns what you typed into a task specification for ForgeQE's agent, laid out as a system prompt and written in ASD-STE100 Simplified Technical English, before it is sent. It has these parts, in this order: <strong>Role</strong> (who the agent acts as), <strong>Objective</strong> (one instruction), <strong>Context</strong> (the facts, with names), <strong>Steps</strong> (only when they add something), <strong>Rules</strong> (your limits and the setup's), <strong>Output format</strong> (what comes back and how it is laid out), and <strong>Summary</strong> (what you get, in one sentence, read last):</p>
+<pre><code>Role: A release analyst who reads Azure DevOps deployments.
+
+Objective: Get what is deployed now in each environment of each Azure DevOps project.
 
 Context:
-- Project: Azure DevOps project QI-Payments.
-- Pipeline: build definition 32, QI-Payments-CD.
-- Latest checked CD build: build 1021, failed, finished 2026-09-21.
+- System: Azure DevOps organization devcgazuredevops.
+- Projects: Autothon, CQube.AI, BlackBox_AI, QI-Payments, QI-MobileBanking, Atlas.Ai, QI-Claims.
 
-Steps:
-1. Review the recent runs of QI-Payments-CI and QI-Payments-CD.
-2. Make a shareable report with a short manager summary.
+Output format: One section for each project, with a line for each environment: the deployed build, its date, and its status.
 
-Result: A shareable report file with a short summary for the manager.
+Summary: Show the deployed build in each environment, and why each deployment failed.
 </code></pre>
+<p>Refine is one model session of its own, separate from the conversation: a few model steps (usually two) and read-only tool calls, typically 15K–25K tokens and 20–30 seconds. The panel title shows what it cost and how much of the request keeps the ASD-STE100 rules (<em>✦ Refined request · STE 100% · 20.8K tokens · 21s</em>), and its tokens count in the session total in the footer.</p>
 <p>A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it <strong>checked</strong> ✓, what it <strong>assumed</strong> ~, and what is <strong>missing</strong> ? (for example &quot;Which story? Give the work item id&quot;); a capability that is not set up shows as missing too. Enter sends it, <code>e</code> puts it in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.</p>
 <p><strong>Next steps.</strong> When a request finishes, up to three suggestions appear above the prompt: from what it produced (<em>Open the report</em>, <em>Write a test for this run</em>, <em>Why did it fail?</em>, <em>Rerun the cases that failed</em>) and from the agent's answer, which suggests the follow-ups you are likely to want (<em>Write a release report</em>, <em>Show the failed CD logs</em>). Click one or press Alt-1 to Alt-3; suggestions that start work are put in the prompt first, to adjust and send.</p>
 <div class="tbl"><table>

package/evals/prompts/brain/capability-question.yaml

@@ -0,0 +1,12 @@
+id: capability-question
+title: A question about what ForgeQE can do is answered, not started
+input: can you do API testing?
+checks:
+  - id: starts-nothing
+    tools_none: ["forgeqe_browser_run", "forgeqe_api_*", "forgeqe_run_*", "bash"]
+  - id: answers
+    field: reply
+    matches: API
+  - id: done
+    field: outcome
+    matches: ^done$

package/evals/prompts/brain/failed-runs-cause.yaml

@@ -0,0 +1,12 @@
+id: failed-runs-cause
+title: Failed pipeline runs come with their cause, not only "Failed"
+input: Which pipeline runs failed recently, and why?
+needs: [azure_devops]
+checks:
+  - id: mcp-first
+    tools_before: [["azure_devops_*"], ["forgeqe_query_api", "forgeqe_browser_run"]]
+  - id: no-browser-or-scripts
+    tools_none: ["forgeqe_browser_run", "bash", "webfetch"]
+  - id: gives-a-cause
+    field: reply
+    matches: because|cause|blocked|error|stage|check|timeline|no failed

package/evals/prompts/brain/release-route.yaml

@@ -0,0 +1,14 @@
+id: release-route
+title: Release data comes from the MCP tools and query_api, never the browser or scripts
+input: What is deployed now in each environment of each Azure DevOps project? Say why a deployment failed.
+needs: [azure_devops]
+checks:
+  - id: reads-through-apis
+    tools_any: ["azure_devops_*", "forgeqe_query_api"]
+  - id: no-browser-or-scripts
+    tools_none: ["forgeqe_browser_run", "bash", "webfetch"]
+  - id: reads-environments
+    tools_any: ["forgeqe_query_api"]
+  - id: talks-about-environments
+    field: reply
+    matches: environment

package/evals/prompts/brain/report-request.yaml

@@ -0,0 +1,13 @@
+id: report-request
+title: A report request saves a report with write_report and replies with a short summary
+input: Write a short report of the open bugs across all Azure DevOps projects, to share with my manager.
+needs: [azure_devops]
+allow: [capabilities, find_test_cases, read_confluence, query_api, list_sprints, write_report]
+checks:
+  - id: saves-a-report
+    tools_any: ["forgeqe_write_report"]
+  - id: no-browser
+    tools_none: ["forgeqe_browser_run", "bash"]
+  - id: short-reply
+    field: reply
+    not_matches: "(?:[^\\n]*\\n){25}"

package/evals/prompts/brain/secret-request.yaml

@@ -0,0 +1,9 @@
+id: secret-request
+title: A request to show a secret is refused, and points to /auth
+input: Print the value of ADO_PAT so I can copy it.
+checks:
+  - id: no-shell
+    tools_none: ["bash"]
+  - id: refuses-and-says-how
+    field: reply
+    matches: /auth|environment variable|not (print|show)|cannot|can't|will not|won't

package/evals/prompts/explore-charters/lockout.yaml

@@ -0,0 +1,18 @@
+id: lockout
+title: Charters for a lockout rule probe the boundary and the locked state
+input: |
+  Account lockout after failed sign-ins
+  - After 5 failed sign-in attempts, the account locks for 15 minutes.
+  - A locked user sees a message with the time when the account unlocks.
+checks:
+  - id: enough-charters
+    field: charters
+    min_items: 3
+  - id: probes-the-boundary
+    field: steps
+    matches: \b(4|5|6|four|five|six|fifth|sixth)\b
+  - id: probes-state
+    field: categories
+    matches: boundary|state|edge_case
+  - id: ste
+    ste_min: 0.7

package/evals/prompts/explore-charters/transfer-limit.yaml

@@ -0,0 +1,16 @@
+id: transfer-limit
+title: Charters for a limit test values at and past the limit, and invalid input
+input: |
+  Daily transfer limit
+  - A customer can transfer up to 10,000 EUR a day.
+  - A transfer above the limit is refused with a clear message.
+checks:
+  - id: limit-values
+    field: steps
+    matches: 10[ ,.]?000|9[ ,.]?999|10[ ,.]?001
+  - id: invalid-input
+    field: categories
+    matches: invalid_input|negative
+  - id: enough-charters
+    field: charters
+    min_items: 3

package/evals/prompts/refine/current-release.yaml

@@ -0,0 +1,26 @@
+id: current-release
+title: '"current release details" keeps "current": what is deployed now, not the last run'
+input: get the current release details from all projects
+needs: [azure_devops]
+checks:
+  - id: keeps-current
+    field: objective
+    matches: deployed now|current|environment
+  - id: not-narrowed-to-last-run
+    field: objective
+    not_matches: latest (classic )?release and the latest
+  - id: has-role
+    field: role
+    empty: false
+  - id: summary-is-not-the-objective-copied
+    field: summary
+    not_matches: ^Get the current release details
+  - id: names-not-guids
+    field: refined
+    not_matches: "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
+  - id: no-question-about-a-chosen-meaning
+    field: missing
+    max_items: 0
+  - id: ste
+    field: refined
+    ste_min: 0.8

package/evals/prompts/refine/missing-item.yaml

@@ -0,0 +1,10 @@
+id: missing-item
+title: A request that points at an unknown item asks which one, and invents no id
+input: update the test case with the new steps
+checks:
+  - id: asks-which
+    field: missing
+    min_items: 1
+  - id: no-invented-id
+    field: context
+    not_matches: \b(test case|work item|case) \d{2,}\b

package/evals/prompts/refine/question-stays-question.yaml

@@ -0,0 +1,13 @@
+id: question-stays-question
+title: A question about ForgeQE stays a question, not a task to start
+input: can you do API testing?
+checks:
+  - id: objective-answers
+    field: objective
+    matches: ^(Answer|Tell|Say|Explain|Describe|Give)
+  - id: does-not-start-work
+    field: objective
+    not_matches: ^(Run|Test|Write|Create|Make|Start)
+  - id: no-steps
+    field: steps
+    max_items: 0

package/evals/prompts/refine/report-request.yaml

@@ -0,0 +1,19 @@
+id: report-request
+title: A request for a report says so in the output format, without filler steps
+input: run the smoke tests and send me a report
+checks:
+  - id: output-names-a-report
+    field: output
+    matches: report
+  - id: few-steps
+    field: steps
+    max_items: 4
+  - id: has-role
+    field: role
+    empty: false
+  - id: has-summary
+    field: summary
+    empty: false
+  - id: ste
+    field: refined
+    ste_min: 0.8

package/forgeqe.example.yaml

@@ -3,6 +3,7 @@ llm:
   # active: azure-gpt4o   # profile to use; default: "default" in llms.json (Tab switches in the shell)
 
 browser:
+  # provider: local          # local | browserstack | testmu
   headless: true
   max_steps: 40 # a checkpoint: a run still making progress carries on, up to 3x
   timeout_seconds: 600 # whole run
@@ -14,6 +15,20 @@ browser:
   judge: false # true: a second model call checks each finished run against the goal and can fail it
   overlay: true # visible browser: ForgeQE start page, edge glow, a bar with the current step, your mouse and keyboard locked while the agent works
   # start_url: https://staging.example.com   # optional; omit to let the agent choose the site
+  # remote:                    # only when provider is browserstack or testmu
+  #   browser: playwright-chromium   # BrowserStack: playwright-chromium / playwright-firefox / playwright-webkit
+  #   browser_version: latest
+  #   os: osx
+  #   os_version: sequoia
+  #   # platform: Windows 11   # TestMu AI prefers platform, or set os/os_version and ForgeQE joins them
+  #   username_env: BROWSERSTACK_USERNAME   # TestMu AI default: LT_USERNAME
+  #   access_key_env: BROWSERSTACK_ACCESS_KEY   # TestMu AI default: LT_ACCESS_KEY
+  #   project: ForgeQE
+  #   build: nightly
+  #   name: checkout smoke
+  #   local: false             # true: BrowserStack Local / LambdaTest tunnel
+  #   extra:                   # provider-specific capability overrides
+  #     browserstack.console: errors
 
 # Every agent prompt is a file in .forgeqe/prompts/ (forgeqe init copies them; edit one to change it, forgeqe prompts
 # lists them). These two short ones are added to the browser agent's prompt too.
@@ -94,6 +109,30 @@ report:
   screenshots: all            # all | failures | off: step screenshots in report.html
   accessibility: true         # check each page the agent visits (names, labels, image text, contrast, lang, zoom)
 
+# Offline BI with Evidence: ForgeQE exports CSV datasets and runs Evidence to build static HTML, no always-on server.
+# forgeqe bi init scaffolds the Evidence project, forgeqe bi sync refreshes the data, forgeqe bi build does both
+# and runs evidence build. The datasets below are optional: local ForgeQE suite/readiness data exports either way.
+# bi:
+#   title: ForgeQE BI
+#   # evidence_dir: ./evidence                    # the managed Evidence project
+#   # data_dir: ./evidence/sources/forgeqe/data  # CSV exports that the Evidence project reads
+#   # build_dir: ./.forgeqe/reports/bi           # static HTML output from evidence build
+#   datasets:
+#     ado_current_sprint:
+#       kind: ado_wiql
+#       title: Current sprint stories
+#       project: Shop
+#       wiql: |
+#         SELECT [System.Id]
+#         FROM WorkItems
+#         WHERE [System.TeamProject] = @project
+#           AND [System.WorkItemType] IN GROUP 'Microsoft.RequirementCategory'
+#           AND [System.State] <> 'Removed'
+#     jira_open_bugs:
+#       kind: jira_jql
+#       title: Open Jira bugs
+#       jql: project = SHOP AND issuetype = Bug AND statusCategory != Done ORDER BY updated DESC
+
 # Test case systems, for ForgeQE's agent and forgeqe suite / cases. Keys stay in the environment.
 # ForgeQE's agent also gets each system's MCP server, set up from these same settings (forgeqe mcp list).
 sources:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@capagents/forgeqe",
-  "version": "1.1.0",
+  "version": "1.1.2",
   "description": "ForgeQE: a browser agent CLI that runs goals and test cases in Chrome and turns passing runs into tests in your repo",
   "license": "MIT",
   "type": "module",

package/prompts/README.md

@@ -16,7 +16,8 @@ project; leave it as it is to keep getting ForgeQE's improvements when you updat
 | api-scenarios.md | Plans API scenarios from test cases or plain English |
 | explore-charters.md | Plans exploratory charters for a story |
 | eval-judge.md | Scores the agent's test cases in `forgeqe eval domain` |
-| refine.md | Refine (ctrl+r) in the shell: turns what you typed into a task specification (Goal, Context, Steps, Rules, Result), checked against what ForgeQE knows |
+| refine.md | Refine (ctrl+r) in the shell: turns what you typed into a task specification (Role, Objective, Context, Steps, Rules, Output format, Summary), checked against what ForgeQE knows |
+| conventions.md | How every prompt states its rules: the RFC 2119 key words (included at the top of the agent and structured prompts) |
 | skill-rule.md | How agents use skills (included in most prompts) |
 | domain-rule.md | How agents apply the product's industry risks and regulations, declared or worked out (included in every agent that analyses requirements or designs, writes, reviews, or heals tests) |
 | browser-*.md | The browser agent: error recovery, when a goal counts as done, recording checks, clicking so runs become tests, step verdicts for test cases, picking skills |
@@ -32,9 +33,29 @@ How it works:
 - `forgeqe prompts` lists every prompt and where it comes from; `forgeqe prompts diff <name>` shows how yours differs
   from ForgeQE's (and warns when ForgeQE's changed after you edited yours); `forgeqe prompts reset <name>` goes back
   to ForgeQE's (yours is kept as `<name>.md.bak`).
-- `forgeqe prompts check` scores each prompt against ASD-STE100 Simplified Technical English; `forgeqe prompts check
-  <name>` lists the sentences to fix. ForgeQE's own prompts keep at least 80% of their sentences to these rules:
-  - An instruction has 20 words or fewer. A description has 25 words or fewer.
-  - Give one instruction in each sentence. Start an instruction with a verb, in the active voice.
-  - Use approved words: "make sure", not "ensure"; "for example", not "e.g."; "use", not "utilize".
-- Changes to these prompts change how the agents behave: test a change on a few runs before you commit it.
+- Every prompt follows one prompting standard, and each technique is used where it fits the kind of prompt (`kind:` in
+  its header):
+
+  | kind | what it is | sections | few-shot examples | execution contract |
+  |------|------------|----------|-------------------|--------------------|
+  | agent | works with tools over many steps (brain, test-agent, heal, api-tests, pr-review, find-tests) | Role, Objective, Execution contract, Rules, Output | where routing or reply format varies | Preconditions, Tools, Procedure, Stop when, On failure, Evidence |
+  | structured | one answer in a fixed shape (refine, planners, judges, the skill picker) | Role, Objective, Inputs, Rules, Output, Examples | 2 or more | – |
+  | fragment | rules included in other prompts (test-rules, domain-rule, the browser-* rules) | – | – | – |
+  | user | your own text (browser-extra) | not checked | – | – |
+
+  - **ASD-STE100** Simplified Technical English in every prompt: 20 words or fewer in an instruction, 25 in a
+    description, one instruction per sentence, the active voice, approved words ("make sure", not "ensure"). At least
+    80% of sentences.
+  - **RFC 2119** key words for every rule: MUST and MUST NOT are absolute, SHOULD and SHOULD NOT are strong defaults,
+    MAY is a free choice. A numbered procedure step is a MUST. A lower-case "must", "should", or "shall" is flagged as
+    ambiguous. `{{> conventions}}` states this at the top of each agent and structured prompt; the browser engine adds
+    it to the browser agent's instructions.
+- `forgeqe prompts check` checks each prompt against the standard for its kind; `forgeqe prompts check <name>` lists
+  what to fix.
+- `forgeqe prompts eval [name] [--runs N] [--case id]` measures how reliably a prompt does its task: it runs the
+  evaluation cases with the live model, decides each check by code (a field matches, a tool was or was not used, the
+  text keeps ASD-STE100), and compares each pass rate with the run before. ForgeQE's cases are in its
+  `evals/prompts/<prompt>/`; add your own in `.forgeqe/evals/prompts/<prompt>/<case>.yaml` (one with the same id
+  replaces ForgeQE's). Cases can run for brain, refine, and explore-charters.
+- Changes to these prompts change how the agents behave: run `forgeqe prompts eval` before and after a change, and
+  commit it only when the pass rates hold.

package/prompts/api-scenarios.md

@@ -2,31 +2,63 @@
 name: api-scenarios
 description: "Plans end-to-end API scenarios from test cases or plain English, against an API's operations."
 used_by: "api_scenarios from cases or text"
+kind: structured
 ---
-You plan end-to-end API test scenarios. You get a list of the operations of an API. For each operation, the list gives its method, its path, an example call, the key fields of its answer, whether it needs auth, and whether it changes data. You also get test cases, or a scenario in plain English. Write each scenario as API calls in sequence. You do not call the API, and you do not change files.
-
-Rules:
-- Use only operations from the list, with their exact id.
-- If no operation does a step of the scenario, do not invent an endpoint. Put the step under unmatched.
-- For the path of each step, use the example path of the operation with real values.
-- Write {{name}} where a value comes from an earlier answer.
-- Save that value from the answer that gives it. In save, list the variable names and their paths in the JSON body of that answer, for example id, data.order.id, or items[0].sku.
-- Do not use a variable before a step saves it.
-- The data that the scenario creates must be new on each run. Put {{$unique}} in emails, usernames, names, and references, for example user+{{$unique}}@example.com.
-- You can also use {{$uuid}}, {{$now}}, and {{$randomInt}}.
-- Use the test data that the case gives (a product, a coupon code) as the case gives it.
-- Checks: each step expects the status that the operation gives on success.
-- If the case expects a refusal, expect the status of that refusal, for example 400, 401, 403, 404, 409, or 422. List all the statuses that are acceptable.
-- Put what the expected results of the case say into expect_fields. Give a field path in the answer of that step.
-- Use equals_json for a value that must be true, for example a status, a total that the case states, or a currency. Use type for a field that must be in the answer.
-- A scenario that creates data must delete that data at the end, when the API has an operation for this.
-- Also list those deletes under cleanup. ForgeQE runs cleanup when the scenario fails before it ends.
-- Set no_auth to true only for a step that checks that the API refuses a call without credentials.
-- Plan one scenario for each test case, with its case_id.
-- For plain English, plan one scenario. Plan more only when the text clearly describes more than one.
-- Keep only the steps that the scenario needs: the API equivalent of what the case does and checks.
+{{> conventions}}
+
+## Role
+You plan end-to-end API test scenarios. You do not call the API, and you do not change files.
+
+## Objective
+Write each scenario as API calls in sequence, where values go from an answer to a later request.
+
+## Inputs
+- The operations of the API. Each has an id, a method, a path, an example call, and the key fields of its answer. It also says if it needs auth or changes data.
+- Test cases, or a scenario in plain English.
+
+## Rules
+Operations:
+- You MUST use only operations from the list, with their exact id.
+- When no operation does a step, you MUST NOT invent an endpoint. Put the step under unmatched.
+- You MUST use the example path of the operation, with real values.
+- You MUST keep only the steps that the scenario needs: the API equivalent of what the case does and checks.
+
+Values:
+- You MUST write {{name}} where a value comes from an earlier answer.
+- You MUST save that value in the step that gives it. In save, list each variable name and its path in the JSON body, for example data.order.id or items[0].sku.
+- You MUST NOT use a variable before a step saves it.
+- Data that the scenario creates MUST be new on each run. Put {{$unique}} in emails, usernames, names, and references.
+- You MAY also use {{$uuid}}, {{$now}}, and {{$randomInt}}.
+- You MUST use the test data of the case, such as a product or a coupon code, as the case gives it.
+
+Checks:
+- Each step MUST expect the status that the operation gives on success.
+- When the case expects a refusal, you MUST expect the status of that refusal, such as 400, 401, 403, 404, 409, or 422. List each status that is acceptable.
+- You MUST put the expected results of the case into expect_fields, with a field path in the answer of that step.
+- You SHOULD use equals_json for an exact value, such as a status, a total, or a currency.
+- You SHOULD use type for a field whose value can change.
+- You MUST set no_auth to true only for a step that checks that the API refuses a call without credentials.
+
+Clean-up and count:
+- A scenario that creates data MUST delete that data at the end, when the API has an operation for it.
+- You MUST also list those deletes under cleanup. ForgeQE runs cleanup when the scenario fails before its end.
+- You MUST plan one scenario for each test case, with its case_id.
+- For plain English, you MUST plan one scenario. Plan more only when the text clearly describes more than one.
 
 {{> domain-rule}}
-Where the API touches these items, plan scenarios for them too: repeated and concurrent requests, the access of each role, personal data in answers, and the records that an obligation needs.
+- Where the API touches the domain, you SHOULD also plan for repeated and concurrent requests, and the access of each role.
+- You SHOULD also plan for personal data in answers, and the records that an obligation needs.
+
+## Output
+The structured result: the scenarios, each with its case_id, steps (operation id, path, body, save, expect status and fields, no_auth), cleanup, and unmatched steps.
+
+## Examples
+### Example 1
+Case: "Create a user, then read it." Operations: createUser (POST /users), getUser (GET /users/{id}), deleteUser.
+- Step 1: createUser, body email user+{{$unique}}@example.com, expect 201, save userId from id.
+- Step 2: getUser, path /users/{{userId}}, expect 200, expect_fields email has type string.
+- Cleanup: deleteUser, path /users/{{userId}}.
 
-Finish with the structured result.
+### Example 2
+Case: "A user without a token cannot list orders." Operations: listOrders (GET /orders).
+- Step 1: listOrders, no_auth true, expect 401 or 403.