@capagents/forgeqe 1.1.4 → 1.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (65) hide show
  1. package/HOW_TO_USE.md +64 -17
  2. package/README.md +2 -1
  3. package/browserstack.example.yaml +14 -2
  4. package/docs/user-guide.html +78 -23
  5. package/forgeqe.example.yaml +18 -5
  6. package/package.json +2 -1
  7. package/prompts/README.md +1 -0
  8. package/prompts/brain.md +8 -1
  9. package/prompts/defect-rca.md +43 -0
  10. package/prompts/test-agent.md +13 -2
  11. package/prompts/test-reviewer.md +8 -3
  12. package/prompts/test-rules.md +10 -4
  13. package/py/forgeqe_engine/browser.py +1 -1
  14. package/skills/dotnet-nunit-playwright/SKILL.md +2 -1
  15. package/skills/playwright-typescript/SKILL.md +2 -1
  16. package/src/brain/index.ts +1 -1
  17. package/src/capabilities/registry.ts +34 -6
  18. package/src/cases/bugs.ts +209 -22
  19. package/src/cases/defectHistory.ts +193 -0
  20. package/src/cases/defects.ts +255 -0
  21. package/src/cases/github.ts +162 -0
  22. package/src/cases/goal.ts +1 -1
  23. package/src/cases/index.ts +23 -4
  24. package/src/cases/predict.ts +372 -0
  25. package/src/cases/rca.ts +258 -0
  26. package/src/cases/readiness.ts +38 -1
  27. package/src/cases/repro.ts +202 -0
  28. package/src/cases/types.ts +8 -1
  29. package/src/cli/help.ts +8 -2
  30. package/src/cli/index.ts +36 -7
  31. package/src/code/graph/graph.ts +518 -0
  32. package/src/code/graph/locators.ts +398 -0
  33. package/src/code/graph/parse.ts +439 -0
  34. package/src/code/graph/tools.ts +113 -0
  35. package/src/code/handoff.ts +93 -18
  36. package/src/code/loop.ts +30 -5
  37. package/src/code/profile.ts +14 -3
  38. package/src/code/replay.ts +1 -1
  39. package/src/code/review.ts +22 -0
  40. package/src/code/runtime.ts +8 -1
  41. package/src/config.ts +142 -18
  42. package/src/mcp/attachtools.ts +223 -0
  43. package/src/mcp/commands.ts +1 -1
  44. package/src/mcp/defecttools.ts +106 -0
  45. package/src/mcp/devicetools.ts +125 -0
  46. package/src/mcp/external.ts +42 -1
  47. package/src/mcp/server.ts +3 -1
  48. package/src/mcp/tools.ts +95 -10
  49. package/src/monitor/run.ts +1 -1
  50. package/src/report/data.ts +14 -0
  51. package/src/report/html.ts +15 -3
  52. package/src/report/template.html +44 -43
  53. package/src/run.ts +4 -2
  54. package/src/suite/report.ts +44 -11
  55. package/src/suite/run.ts +163 -16
  56. package/src/tui/App.tsx +153 -94
  57. package/src/tui/components.tsx +27 -23
  58. package/src/tui/inspector.tsx +140 -83
  59. package/src/tui/landing.ts +3 -4
  60. package/src/tui/refine-panel.tsx +102 -52
  61. package/src/tui/shortcuts.tsx +2 -0
  62. package/src/tui/transcript.tsx +32 -2
  63. package/src/types.ts +3 -1
  64. package/src/usage.ts +1 -1
  65. package/testmu.example.yaml +11 -0
package/HOW_TO_USE.md CHANGED
@@ -47,7 +47,7 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
47
47
  <!-- capabilities:start (generated from src/capabilities/registry.ts by bun scripts/gen-capabilities.ts; do not edit) -->
48
48
  | Area | Capability | Tools | |
49
49
  |------|------------|-------|-|
50
- | Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory` | [docs](#4-without-the-shell) |
50
+ | Browser | **Run any goal in a real browser**: Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper. | `browser_run`, `run_details`, `page_memory`, `browser_devices` | [docs](#4-without-the-shell) |
51
51
  | Browser | **Saved logins for the application under test**: Signs the browser agent in with saved credentials (including TOTP) and reuses the session; a user can also sign in by hand once in a browser. | `logins` | [docs](#signing-in) |
52
52
  | Test cases | **Run manual test cases in browsers**: Reads test cases from Azure DevOps, Jira, Xray, TestRail, Zephyr Scale, qTest, a file, or pasted text, runs each in a real browser with a verdict per step, writes report.html and junit.xml, posts results back, and files or updates bugs for failures. | `find_test_cases`, `run_test_cases` | [docs](#7-test-cases-and-suites) |
53
53
  | Test cases | **Check and improve test cases**: Checks test cases before they run (vague or missing expected results, missing test data, placeholders, duplicates) against the story, feature, and epic they test, then rewrites Azure DevOps cases once the user agrees. | `check_test_cases`, `update_test_case` | [docs](#checking-test-cases-before-they-run) |
@@ -63,12 +63,13 @@ Everything ForgeQE can do, by area. ForgeQE's agent answers "can you…?" questi
63
63
  | API | **API and contract testing**: Builds API contracts (status, schema, key fields) from the calls a browser run saw or from an API description (OpenAPI, Swagger, WSDL, GraphQL, .proto, gRPC reflection), and end-to-end API scenarios from runs, descriptions, test cases, or plain English. Runs them straight against the API without a browser, or writes them as tests in the repository. REST, GraphQL, SOAP, and gRPC. | `api_contracts`, `run_api_contracts`, `api_scenarios`, `run_api_scenarios`, `write_api_tests` | [docs](#api-and-contract-tests) |
64
64
  | Quality | **Accessibility and page performance**: Checks every page a run visits against WCAG rules (axe-core) and measures its Core Web Vitals, against budgets, with trends over time per page. | `quality_report`, `quality_trends` | [docs](#accessibility-and-performance-budgets) |
65
65
  | Quality | **Visual comparison with baselines**: Compares each page a run captures with its approved baseline and shows what changed; intended changes become the new baseline once the user agrees. | `visual_compare`, `visual_approve` | [docs](#visual-comparison) |
66
+ | Quality | **Defect intelligence: trends and prediction**: Groups failures into defects, keeps their history across suites (new, known, gone, back again), and gives defect trends and where a change is likely to cause defects, with the cases to run first. | `defect_trends`, `predict_defects` | [docs](#bugs-for-failures) |
66
67
  | Release | **Requirement coverage and release readiness**: Gives requirement coverage and a GO / GO WITH RISKS / NO-GO verdict for a sprint, area, epic, or stories, from test cases, results, bugs, and flaky tests; can publish it to the wiki. | `release_readiness` | [docs](#requirement-coverage-and-release-readiness) |
67
68
  | Operations | **Scheduled monitoring with alerts**: Runs checks on a schedule and alerts Teams or email when one keeps failing, and again when it recovers. | `monitor` | [docs](#9-monitoring) |
68
69
  | Operations | **Teams and email notifications**: Tells the team in Microsoft Teams channels or by email when a suite fails or passes, when a release readiness verdict is in, and when the domain evaluation falls below its gate, with the failing cases, their bugs, and links to the report and the CI build. Each channel picks its events; the agent also posts a summary to a channel when asked. | `notify` | [docs](#17-notifications) |
69
70
  | Release | **Reports you can share**: Writes up what ForgeQE's agent found, such as release status across projects, open defects, or pipeline health, as a Markdown file and an HTML page in the reports folder, with a summary in the chat and links to open them. | `write_report` | [docs](#reports-from-the-agent) |
70
71
  | Operations | **CI pipelines**: Writes a GitHub Actions or Azure Pipelines workflow that runs ForgeQE, stores the secrets it needs, pushes a branch, and makes the pipeline green. | `push_branch`, `set_ci_secret` | [docs](#8-pipelines) |
71
- | Systems | **Work with Azure DevOps, Jira, and Confluence**: Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, and builds Azure DevOps dashboards and wiki pages. | `query_api`, `read_confluence`, `create_ado_dashboard`, `write_ado_wiki_page` | [docs](README.md#azure-devops-jira-and-other-systems) |
72
+ | Systems | **Work with Azure DevOps, Jira, GitHub, and Confluence**: Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, puts run evidence (screenshots, reports) on work items and issues as attachments or comments, and builds Azure DevOps dashboards and wiki pages. | `query_api`, `read_confluence`, `create_ado_dashboard`, `write_ado_wiki_page`, `attach_to_work_item` | [docs](README.md#azure-devops-jira-and-other-systems) |
72
73
  | Setup | **Setup, skills, and documentation**: Shows the model profiles and MCP servers, installs or repairs the browser engine, creates a forgeqe.yaml, manages skills, and answers questions about ForgeQE from its documentation and this list. | `forgeqe_setup`, `capabilities` | [docs](#2-scaffold-config) |
73
74
 
74
75
  Not supported: Native mobile apps (iOS, Android) and desktop applications (instead: web applications in local Chrome or Chromium, or configured remote web grids such as BrowserStack and TestMu AI/LambdaTest); Load, stress, or soak testing with many simultaneous users (instead: per-page Core Web Vitals with budgets); Penetration testing or dynamic security scanning (instead: security findings in pull request reviews).
@@ -404,7 +405,7 @@ The chat gets a short summary and a card with both paths as links, and **Open th
404
405
  **The start screen.** Before your first request, the screen is one column under the logo:
405
406
 
406
407
  ```
407
- playwright-csharp-pom-framework · .NET · Azure DevOps · azure/gpt-5.4
408
+ playwright-csharp-pom-framework · .NET · Azure DevOps
408
409
 
409
410
  ┃ Describe what you want to test…
410
411
  ✦ Refine ctrl+r ⌕ Search ctrl+p ? shortcuts ctrl+c close
@@ -416,9 +417,10 @@ The chat gets a short summary and a card with both paths as links, and **Open th
416
417
  ▤ Report QI-Payments pipeline status report 18 min ago open
417
418
  ✓ Run Search hotels in Mumbai on booking.com 2 days ago open rerun
418
419
  Setup ▰▰▰▱▱ 3 of 5 · Domain and Notifications are not set up ▸ how to set up
420
+ ● Tip Press Tab to switch LLM profile azure/gpt-5.4 (default) · ForgeQE
419
421
  ```
420
422
 
421
- - The line under the logo is what ForgeQE works with here: the folder, what it is built with (from its files, such as a `.sln` or `playwright.config.ts`), the systems that are set up, and the model.
423
+ - The line under the logo is what ForgeQE works with here: the folder, what it is built with (from its files, such as a `.sln` or `playwright.config.ts`), the systems that are set up. The model and its profile are in the footer, the status line on every screen; Tab switches the profile.
422
424
  - The prompt starts five rows high and grows with a long request, up to nearly half the screen, before it scrolls; the sections below give way, last first, to make room.
423
425
  - **Start with** offers up to four things to ask, from the capabilities that work in this setup, each with an example in the words of your setup. Click one, or press Alt-1 to Alt-4, to put it in the prompt to adjust and send.
424
426
  - In a folder without `forgeqe.yaml`, or without a live model, the line under the logo says so and **Get started** takes its place: **Set up ForgeQE in this folder** (runs `/init`), **Add a model**, and **Sign in to Azure DevOps or Jira** (runs `/auth`). Start ForgeQE in your test project's folder to work with its settings.
@@ -442,26 +444,29 @@ The chat gets a short summary and a card with both paths as links, and **Open th
442
444
  ⏺ No classic releases exist in any of the seven projects.
443
445
  …
444
446
 
445
- ⠹ Thinking… (24s · 57.4K tokens · esc to interrupt)
447
+ ⠹ Acting in the browser (step 3)… (24s · step 3/40 · 57.4K tokens · esc to interrupt)
446
448
  ╭──────────────────────────────────────────────────────────╮
447
449
  │ > Ask ForgeQE… (/ for commands, @ to reference a run…) │
448
450
  ╰──────────────────────────────────────────────────────────╯
449
451
  ✦ Refine ctrl+r ⌕ Search ctrl+p ▦ Panels ctrl+o ctrl+c close
450
- ~/playwright-csharp-pom-framework azure/gpt-5.4 · 57.4K tokens · ForgeQE
452
+ ~/playwright-csharp-pom-framework azure/gpt-5.4 (default) · 57.4K tokens · ForgeQE
451
453
  ```
452
454
 
453
455
  - `>` is what you asked. `⏺` starts each reply, tool call, browser run, test case, and result; calls in a row sit together, with the arguments dimmed. What came of a call is indented under it after `⎿`: the agent's notes, each browser step and its actions (✓ or ✗), and a result's details.
454
- - While a request runs, one line above the input says what is happening, for how long, and with how many tokens. The footer is the status line: the model and what the session used. Prices are not shown in the conversation; reports and `forgeqe models` have them.
456
+ - While a request runs, one line above the input says what is happening, for how long, how far a browser run or suite has got, and with how many tokens. The footer is the status line, and the one place for the folder, the model and its profile, and what the session used.
457
+ - PgUp and PgDn scroll the conversation from the prompt (the mouse wheel works too); PgDn back at the end follows new lines again.
458
+ - A long request you sent (a refined one, a pasted spec) shows its first lines and *… 27 more lines, sent in full*. Prices are not shown in the conversation; reports and `forgeqe models` have them.
455
459
  - A result that made nothing (a run that only looked something up) adds no card.
456
460
 
457
- **The panels.** **▦ Panels** (Ctrl-O) opens the session, plan, activity, files, context, and tokens panels beside the conversation on a terminal 120 columns wide or more, or in its place on a narrower one; Ctrl-O again, or **✕ Close**, closes them:
461
+ **The panels.** **▦ Panels** (Ctrl-O) splits the screen in half on a terminal 120 columns wide or more: the **Activities** column on the right, and the conversation with the files, context, and tokens panels on the left. On a narrower terminal the panels take the conversation's place. Ctrl-O again, or **✕ Close**, closes them.
458
462
 
459
- - the status line, with who is working, for how long, a progress bar for the plan or a suite's cases, and **■ Stop**;
460
- - **Plan**: the agent's to-do list, then each browser step or suite case with ✓, ✗ and why, ◐ running;
461
- - **Activity**: every tool call, browser step, and the agents' reasoning;
462
- - **Files**: each file read ◉, written ✚, or edited ✎, newest first;
463
- - **Context**: how full the model's context window is, and the skills, product knowledge, domain, and MCP servers in it;
464
- - **Tokens**: total, input, output, cached.
463
+ The Activities column shows what the agents do as they do it:
464
+
465
+ - the status line: who is working, a progress bar for the plan or a suite's cases, for how long, and **■ Stop**; under it, the whole width, what is happening now (*Agent: Azure DevOps API · QI-Payments/_apis/build/builds…*);
466
+ - **Plan**: the agent's own task list as soon as it writes one, then each browser step or suite case: ○ pending, ◐ in progress, ✓ done, ✗ failed and why. ForgeQE's agent writes a plan for any request of three steps or more and updates it as it goes;
467
+ - **Activity**: **Now** (the task in progress) and **Next** (the next task), then every tool call, browser step, and note. A call shows ◐ the moment it starts and becomes ✓ or ✗ in the same row when it ends, so a long API query or test run is never invisible. Reads and searches that the conversation leaves out at the brief detail level are listed here. A call takes one row and a note up to three; focus the panel to scroll through everything or filter it.
468
+
469
+ Over the conversation: **Files** (each file read ◉, written ✚, or edited ✎, newest first), **Context** (how full the model's context window is, and the skills, product knowledge, domain, and MCP servers in it), and **Tokens** (total, input, output, cached). When that half is under 80 columns they fold into one line: *Files 4 (1 changed) · Context 26% · Skills 2 · 52.9K tokens*.
465
470
 
466
471
  **Buttons.** Under the input: **✦ Refine** (Ctrl-R; it fills in once there is text to refine), **⌕ Search** (Ctrl-P, the command palette), and **▦ Panels** (Ctrl-O); while a request runs, **■ Stop** (Esc) takes Refine's place. With the panels open, **▦ Focus panels** (Shift-Tab) and **✕ Close** (Ctrl-O) are in their status line. On a narrow terminal, **◧ Details** (Ctrl-O) and **⇄ Profile** (Tab) are under the input. Each is clickable.
467
472
 
@@ -501,7 +506,7 @@ Summary: Show the deployed build in each environment, and why each deployment fa
501
506
 
502
507
  Refine is one model session of its own, separate from the conversation: a few model steps (usually two) and read-only tool calls, typically 15K–25K tokens and 20–30 seconds. The panel title shows what it cost and how much of the request keeps the ASD-STE100 rules (*✦ Refined request · STE 100% · 20.8K tokens · 21s*), and its tokens count in the session total in the footer.
503
508
 
504
- A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it **checked** ✓, what it **assumed** ~, and what is **missing** ? (for example "Which story? Give the work item id"); a capability that is not set up shows as missing too. Enter sends it, `e` puts it in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.
509
+ A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it **checked** ✓, what it **assumed** ~, and what is **missing** ? (for example "Which story? Give the work item id"); a capability that is not set up shows as missing too. What is missing comes first, and the title counts it (*✦ Refined request · 2 to answer*). A refined request longer than the screen scrolls inside the panel with ↑/↓ and PgUp/PgDn (the border says so); the title and the **Use refined**, **Edit**, and **Keep mine** buttons stay on screen. ←/→ or Tab picks one, Enter does it; `e` puts the request in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.
505
510
 
506
511
  **Next steps.** When a request finishes, up to three suggestions appear above the prompt: from what it produced (*Open the report*, *Write a test for this run*, *Why did it fail?*, *Rerun the cases that failed*) and from the agent's answer, which suggests the follow-ups you are likely to want (*Write a release report*, *Show the failed CD logs*). Click one or press Alt-1 to Alt-3; suggestions that start work are put in the prompt first, to adjust and send.
507
512
 
@@ -510,6 +515,7 @@ A read-only agent writes it. It starts knowing every capability and whether it w
510
515
  | Enter | Send the message or command |
511
516
  | Tab | Next LLM profile from `llms.json`; the conversation carries over to the new model. While `/` or `@` suggestions show, completes the selected one |
512
517
  | Shift-Tab | Move between the chat and the Plan, Activity, and Files panels |
518
+ | PgUp / PgDn | Scroll the conversation |
513
519
  | Ctrl-P | Command palette |
514
520
  | Ctrl-R | Refine the request in the prompt |
515
521
  | Ctrl-O | Open or close the panels (beside the conversation, or in its place on a narrow terminal) |
@@ -612,6 +618,7 @@ code:
612
618
  stall_after: 5 # stop the agent's turn when one command it runs has failed this many times
613
619
  replay: true # before the agent starts, run the steps as a Playwright script and hand over what worked
614
620
  live_browser: true # once a step fails, the agent looks at that page in a Playwright browser
621
+ graph: true # the code graph: find page objects without reading files, and reuse what already locates an element
615
622
  # model: anthropic/claude-sonnet-4-5
616
623
  # runtime: /path/to/agent-runtime # default: the runtime ForgeQE downloads
617
624
  ```
@@ -625,10 +632,12 @@ What happens after the run passes:
625
632
 
626
633
  Scribe first reads the repo (as a read-only agent: it cannot change files or use the live browser yet), while the replay runs when there is one, so the two overlap. What it learns (the framework, the command that runs one test, where page objects and config live, the conventions) is saved with page memory in `scribe-notes.json`, with the tests each job writes there. The next job in the same repo starts from those notes and that list instead of reading the repo again, and extends those tests' page objects when its flow overlaps; notes older than 30 days are read again.
627
634
  3. Scribe starts in the repo. It reads `AGENTS.md`, the test framework config, run scripts, page objects, fixtures, and helpers, and follows the imports between them. It writes the test in the repo's language and framework (a C# NUnit repo gets a C# NUnit test) and to fit its code: it extends an existing page object instead of duplicating it and reuses the login flow, test data, and config. A repo with no tests gets Playwright Test in TypeScript. It may change anything the test needs, including dependencies and config.
635
+
636
+ Before Scribe starts, ForgeQE reads the repo into a **code graph**, without a model: every class, page object, method, property, and test (TypeScript, JavaScript, C#, Java, Python), the calls between them, and the locators each one builds, in any framework's spelling (Playwright, Selenium, Cypress, WebdriverIO, Testing Library). It is kept in `~/.forgeqe/cache/code-graph`, nothing leaves the machine, and the next job reads again only the files that changed. The graph is tied to page memory, which knows each element by all its locators (its id, role and name, text, test id): so when the run clicked `getByRole('button', { name: 'Login' })` and the repo's `LoginPage.LoginButton` uses `#login-button`, the handoff says under that step "in this repository: LoginPage.LoginButton … used by LoginPage.SignInAsync", and Scribe calls the existing method instead of writing a second locator. Scribe also gets tools that answer from the graph instead of grep and whole-file reads: `code_find` (classes, methods, and tests by name, or by the text a locator uses), `code_outline` (a file's or class's signatures and locators, or the source of one member), `code_element` (which member already locates an element), `code_references` (what calls a member, before it is changed), and `code_impact` (the tests a change reaches, through flows and base classes, whatever the test files are called). `graph: false` turns it off. The handoff also outlines the page objects its steps use and includes the repo's test that uses the most of them, so Scribe takes the test conventions from that test instead of reading other tests, and reads a settings file only to change it.
628
637
  4. It writes the test from the handoff and the replay script: steps that worked are ported with their locators, steps that failed are fixed first. Every locator, including those for assertions, comes from the replay script, the page snapshots, page memory, or a check on the page; steps where the browser agent ran JavaScript to look around are left out. It does not walk the flow again or build tests to explore. Once a step fails (in the replay, or in its test), ForgeQE switches on its live browser: a Chrome that Playwright drives, set up like ForgeQE's, where it looks at the failing step's page one tool call per action (open, snapshot, click, type, wait, screenshot, run JavaScript, console errors, failed requests). Until then the live browser tools are off, so no tokens go into looking at pages that work. The live browser needs Node.js (`npx`), and Microsoft's Playwright MCP server is downloaded on first use; with `live_browser: false`, or without `npx`, it looks with a small throwaway script instead. When its browser gets a different page than ForgeQE's did (an error, a challenge, a control that will not open), it compares the two browsers (user agent, `navigator.webdriver`, headless, failed requests and CORS errors, headers the framework adds, console errors) and fixes the difference in the repo's browser configuration. It never judges the site with `curl` or `webfetch`.
629
638
  5. Scribe does not run the test itself: it may build or compile, then reports the test file and the command that runs only that test, and ForgeQE runs it. If one command it runs fails `stall_after` times in a row within its work (the same build again and again, say), ForgeQE stops that turn and sends the command and its last output back, asking it to change approach. If the same command stalls again, the job stops as a failure with that output in the notes.
630
639
  6. ForgeQE runs that command (with `CI=1`) and checks the exit code, and it watches the processes the command starts for a browser launched by Playwright, Puppeteer, Selenium, or Cypress. On a failure it sends the command, exit code, and output back to Scribe. The output is kept short so the error is never cut off: the page dump Playwright puts into a failed assertion ("Aria snapshot:", often thousands of lines) is taken out and saved as `<run>/test-runs/attempt-N-page.aria.yml`, then the start (where the first error and its stack trace are) and the end (the summary) are sent; the full output is in `<run>/test-runs/attempt-N.log`, and the prompt names both files. A command that exits 0 without starting a browser also goes back: the test must drive the app in a browser. If the repo's browser runs on another machine (Selenium Grid, a cloud service, a container), the agent says where and the check is skipped.
631
- 7. A passing test then goes to a separate reviewer agent, which can read files and `git diff` but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (`.forgeqe/`, `forgeqe.yaml`, `llms.json`) are never part of the review. If it finds problems, they go back to Scribe as the next attempt.
640
+ 7. A passing test then goes to a separate reviewer agent, which can read files and `git diff` but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (`.forgeqe/`, `forgeqe.yaml`, `llms.json`) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same `code_*` tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt.
632
641
 
633
642
  Scribe cannot edit ForgeQE's files or run folders, and cannot run `git clean`, `git reset --hard`, `git stash`, `git restore`, or `git checkout -- <path>`. If it still removes a file that existed before the job, or reverts your uncommitted edits, ForgeQE puts the file back after that attempt and tells the agent.
634
643
  8. The job is done when ForgeQE's run passes with a browser and the reviewer approves. It stops as a failure after `max_attempts` attempts, when the job passes `code.max_tokens`, when the reviewer rejects a passing test twice, if the review cannot finish, when the test fails the same way three times in a row (the same output apart from numbers such as timings and line numbers; the second time, Scribe is told its change did not change the failure and to look at the failing step before changing code), or straight away on an auth, API, or context-limit error from the model.
@@ -1109,9 +1118,28 @@ sources:
1109
1118
  labels: [forgeqe] # Jira labels or Azure DevOps tags
1110
1119
  # area_path: Shop\Web # Azure DevOps
1111
1120
  # on_error: true # also for cases that could not run
1121
+ # group_by: defect # defect (default): cases failing the same way share a bug; case: a bug per case
1112
1122
  ```
1113
1123
 
1114
- When results are posted, each case that failed with a functional defect (below) gets a bug: the title is `[ForgeQE] <case id>: <title>`, the description has the result, the steps with verdicts, the preconditions, where the run ended, the build link, and the run folder, and the last screenshot is attached. In Jira the bug is linked to the Jira or Xray test ("relates to"); in Azure DevOps it is "tested by" the test case work item. Each bug carries a label or tag that fingerprints the case (`forgeqe-` and ten hex digits), so the next failure comments "Still failing" on the open bug instead of filing a new one, and a pass adds a note that it passes again. Closed bugs are left alone; a new failure after that files a new bug.
1124
+ When results are posted, each case that failed with a functional defect (below) gets a bug: the title is `[ForgeQE] <case id>: <title>`, the description has the result, the steps with verdicts, the preconditions, where the run ended, the build link, and the run folder, and the last screenshot is attached. In Jira the bug is linked to the Jira or Xray test ("relates to"); in Azure DevOps it is "tested by" the test case work item. Each bug carries a label or tag that fingerprints the case (`forgeqe-` and ten hex digits), so the next failure comments "Still failing" on the open bug instead of filing a new one, and a pass adds a note that it passes again. A closed bug is not reopened: a new failure after that files a new bug, titled `[ForgeQE] Regression: …`, that says when the old one was closed and links to it (Jira "relates to", Azure DevOps "Related", a mention on GitHub; a GitHub issue closed as not planned does not count).
1125
+
1126
+ **One bug per defect.** Cases that fail the same way share one bug (`group_by: defect`, the default). ForgeQE gives each failing case a signature from the strongest evidence its run recorded: a failed request (a 5xx before a 4xx) by its route and status, then an uncaught page error, then a console error, with ids, numbers and times taken out, so `POST /api/cart/123 → 500` and `/api/cart/456 → 500` match. Errors that the suite's passing runs show too (an analytics call, a missing favicon) are noise and do not count. The first case of a defect files the bug, titled after the defect (`[ForgeQE] shop.example.com/api/cart/:id returns 500`) and labelled with the defect's fingerprint (`forgeqe-d-` and ten hex digits) as well as the case's; every other case failing the same way is added as a comment and gets its own case label on that bug. The next run finds the open bug by either label. A failure with no browser evidence is matched by its page and what the agent saw, within one run only, and its bug is titled after the case as before. `group_by: case` files one bug per failing case. The report's **Defects** page, the terminal summary and the CI job summary list every defect with the cases it breaks.
1127
+
1128
+ **Defect history.** Each suite adds its defects to `defect-history.json` next to the suite folders (keep it between CI runs as a cache, like the suite folders, to track a pipeline's defects). A defect is **new** the first time, then **known** ("seen in 3 earlier runs since 2 Oct"). It **went away** when a run did not see it while a case it broke passed; seen again after that, or matching a closed bug in the tracker, it is a **regression**. A defect whose cases keep changing between passing and failing across the last 20 suites (pass, fail, pass) is marked **intermittent**: it still gets a bug, which says so. When a new bug is filed, ForgeQE also searches the tracker once for open bugs people filed about the same route or error (`/api/cart`, `price is undefined`) and lists them on the bug as possibly related; it never merges them. Only defects found by browser evidence are kept in the history: a defect matched by the agent's words gets new words every run. The Defects page shows the history and filters new defects and regressions; the summaries say it next to each defect.
1129
+
1130
+ **Reproducing and root cause.** Each defect gets a repro file, `defects/<defect id>.md` in the suite folder, and the same steps go in its bug: how reproducible it is, the case's steps with the agent's verdicts, what the browser did, where it ended, the environment (browser, device, build, commit), the failing request as a `curl` command with its response (the engine takes credentials out of the requests it records), and the run's files (screenshot, replay flow, every step, every API call). The first case of a new product defect runs once more into its `reproduce/` folder: failing the same way makes it **always** reproducible, not failing again makes it **intermittent**; a defect seen in earlier runs is not run again. Then the root cause, always as a hypothesis:
1131
+
1132
+ - From the evidence, at no cost: the layer (the server, the front end, access, a route that is not there, the environment) and the component (the route or the script).
1133
+ - With `defects.product_repo` set: **suspect commits**, made since the defect's cases last passed, whose changes mention its route or error (`git log -G`), or whose files are named after it.
1134
+ - For new defects and regressions, a read-only agent (prompt `defect-rca`) reads the evidence, the suspect commits and the product's code, and writes a hypothesis with its confidence (high only when it read the code that fails and the change that broke it), what it rests on, and checks to confirm it. The hypothesis is added to the bug as a comment and shown on the Defects page. It runs for at most `max_analyses` defects per suite; each is one agent session (about 90K–120K tokens with gpt-5.4 in our runs), shown on its own line under Usage.
1135
+
1136
+ ```yaml
1137
+ defects:
1138
+ product_repo: ../shop # the application's repository: suspect commits, and the code the analysis reads
1139
+ # reproduce: true # run the first case of each new defect once more
1140
+ # analyze: true # a model writes a root-cause hypothesis for new defects and regressions
1141
+ # max_analyses: 3 # most analyses per suite
1142
+ ```
1115
1143
 
1116
1144
 
1117
1145
  ### Checking test cases before they run
@@ -1234,6 +1262,25 @@ forgeqe visual # the latest run's pages against their base
1234
1262
  forgeqe visual approve --page /home # make this run's /home the baseline (asks first; --yes to skip)
1235
1263
  ```
1236
1264
 
1265
+ ### Defect trends and prediction
1266
+
1267
+ `forgeqe defects` (or "which defects keep coming back?") reads the suites saved in `suite.out_dir` and their `defect-history.json`, and writes `defect-trends.md` next to them: how many defects are still there, gone, or came back after a fix; the time they took to go away; the most seen ones with their bugs; defects by area (a page or an API route, by its path without ids); defects per suite; and how often predictions were right. The report's Defects page shows the same under **Across runs**.
1268
+
1269
+ `forgeqe defects predict [--repo DIR] [--base REF] [--head REF]` (or "what can the changes since main break?") says where a change is likely to cause defects. It reads the change between the merge base of `base` (default: the remote's default branch, else `main` or `master`) and `head` in `defects.product_repo` (else the working directory). A change touches an area when it changes a file named after it (`server/cart.ts` for `/api/cart/:id`) or its diff mentions the route. Each touched area gets a score from what the saved suites know:
1270
+
1271
+ | Signal | Score |
1272
+ |--------|-------|
1273
+ | The diff mentions the area's route | +3 |
1274
+ | A changed file is named after it | +2 |
1275
+ | Each earlier defect there (three at most) | +2 |
1276
+ | Defects there came back after a fix | +2 |
1277
+ | A defect there is still open | +1 |
1278
+ | Its cases failed in 20% (50%) of recent runs | +1 (+2) |
1279
+
1280
+ 6 or more is high, 3 or more medium. It lists the cases that ended on the high and medium areas as the ones to run first, and the changed product code that no case reaches. It is a heuristic, not a trained model: each prediction is saved in `defect-predictions.json`, and the first suite after it scores it (did its defects fall in the areas rated medium or high; did the high areas get one), so `forgeqe defects` gives its accuracy.
1281
+
1282
+ Release readiness uses the defects too: a regression on a case in scope blocks the release (NO-GO), and a new defect with no bug is a risk.
1283
+
1237
1284
  ### Requirement coverage and release readiness
1238
1285
 
1239
1286
  "Are we ready to release sprint 12?", "coverage of epic 300" (or `release_readiness`) builds the coverage matrix for an Azure DevOps scope: a sprint (`current` or a name), an area path, an epic or feature and every story under it, or story ids.
package/README.md CHANGED
@@ -68,7 +68,7 @@ ForgeQE sends no telemetry and turns it off in everything it starts:
68
68
 
69
69
  An unchanged copy keeps following ForgeQE's updates; once you edit it, yours wins.
70
70
 
71
- For remote browsers, `forgeqe.yaml` can now point at provider-specific YAML files under `browser.providers` such as `browserstack` and `testmu`, so the BrowserStack or TestMu AI connection details do not have to live inline under `browser.remote`. `forgeqe init` drops sample files at `.forgeqe/browserstack.yaml` and `.forgeqe/testmu.yaml`. When both are present, the separate file supplies the provider defaults and the inline `browser.remote` values override only the active provider. ForgeQE's agent can also switch providers per request by passing `browser_provider` to `browser_run` or `run_test_cases`, and the CLI can pin one with `--browser-provider`. ForgeQE drives remote browsers over CDP, so `remote.browser` must be Chrome or Edge, and the default endpoints are the providers' CDP (`/puppeteer`) ones. On a remote grid, the browser stops taking real clicks and keystrokes on a tab once a password form has been submitted (most likely the browser's password manager), so after a sign-in leaves the login page ForgeQE reopens the page it landed on in a fresh tab, with the session intact, and the run log says so.
71
+ For remote browsers, `forgeqe.yaml` can now point at provider-specific YAML files under `browser.providers` such as `browserstack` and `testmu`, so the BrowserStack or TestMu AI connection details do not have to live inline under `browser.remote`. `forgeqe init` drops sample files at `.forgeqe/browserstack.yaml` and `.forgeqe/testmu.yaml`. When both are present, the separate file supplies the provider defaults and the inline `browser.remote` values override only the active provider. ForgeQE's agent can also switch providers per request by passing `browser_provider` to `browser_run` or `run_test_cases`, and the CLI can pin one with `--browser-provider`. **Devices.** A provider file can list named devices under `devices:`, each the file's shared `remote:` settings with its own browser, version, and platform on top. In the shell, describe the devices you want ("add Chrome on Windows 11 and Chrome 128 on Windows 10 on TestMu") and ForgeQE's agent saves them there with the `browser_devices` tool, keeping the file's comments; then "run SHOP-12 on both" runs the case on every device at the same time, each with its own run folder and row in the report (`SHOP-12@win11-chrome`). From the command line, `forgeqe suite <source> --devices win11-chrome,win10-edge` does the same, and `forgeqe start --devices ...` pins devices for a shell session. Devices run side by side, up to eight at once unless `--parallel` or `suite.parallel` says more; the grid queues sessions past your plan's limit. Each case posts one combined result back to its test system, failed when any device failed, and gets one test, written from its run on the first device. ForgeQE drives remote browsers over CDP, so `remote.browser` must be Chrome or Edge (TestMu AI's CDP endpoint serves Chrome only: an Edge session there never opens), and the default endpoints are the providers' CDP (`/puppeteer`) ones. On a remote grid, the browser stops taking real clicks and keystrokes on a tab once a password form has been submitted (most likely the browser's password manager), so after a sign-in leaves the login page ForgeQE reopens the page it landed on in a fresh tab, with the session intact, and the run log says so.
72
72
 
73
73
  ```bash
74
74
  forgeqe models
@@ -291,6 +291,7 @@ ForgeQE's agent connects to each system's MCP server, configured from the same `
291
291
  | Azure DevOps Services | Microsoft's [Azure DevOps MCP server](https://github.com/microsoft/azure-devops-mcp) (`npx @azure-devops/mcp`), with all its tools: work items, boards, test plans, pipelines, repos, wiki pages, search, and security alerts | Node.js, and `ADO_PAT` or `az login` |
292
292
  | Jira Cloud, Server/Data Center | [mcp-atlassian](https://github.com/sooperset/mcp-atlassian) (`uvx`), with issues, JQL search, comments, and transitions. `mcp: rovo` uses [Atlassian's hosted server](https://github.com/atlassian/atlassian-mcp-server) instead (Cloud, and an admin must allow API tokens) | uv; `JIRA_EMAIL` and `JIRA_API_TOKEN` (Cloud) or a personal access token (Server/DC) |
293
293
  | Confluence | mcp-atlassian, shared with Jira (or its own when Jira isn't set up); Atlassian's hosted server covers it on the same site | `confluence: true` on a Jira Cloud site, or `confluence.url` + tokens |
294
+ | GitHub | GitHub's [hosted MCP server](https://github.com/github/github-mcp-server) with the repos, issues, pull requests, and Actions toolsets (`sources.github.toolsets` changes them), on in a repository with a GitHub remote or when `sources.github` is set. GitHub issues are also a test case source (`github:#12`, `github:acme/shop#12,#13`, `github:label=login`, `github:q=login in:title`): results go back as issue comments, and `sources.bugs.system: github` files bugs as issues. GitHub has no API to upload a screenshot to an issue, so a bug names the run folder instead | `GITHUB_TOKEN`, or the GitHub CLI's sign-in (`forgeqe auth login github`) |
294
295
  | Xray, TestRail, Zephyr Scale, qTest | none; ForgeQE's API | `XRAY_CLIENT_ID` / `XRAY_CLIENT_SECRET`, `TESTRAIL_EMAIL` / `TESTRAIL_API_KEY`, `ZEPHYR_API_TOKEN`, `QTEST_TOKEN` |
295
296
  | Anything else | servers you add under `mcp:` | |
296
297
 
@@ -2,8 +2,8 @@ provider: browserstack
2
2
  remote:
3
3
  browser: chrome
4
4
  browser_version: latest
5
- os: osx
6
- os_version: sequoia
5
+ os: OS X
6
+ os_version: Sequoia
7
7
  username_env: BROWSERSTACK_USERNAME
8
8
  access_key_env: BROWSERSTACK_ACCESS_KEY
9
9
  project: ForgeQE
@@ -12,3 +12,15 @@ remote:
12
12
  local: false
13
13
  extra:
14
14
  browserstack.console: errors
15
+
16
+ # Named devices: each is the remote settings above with its own on top. Run on them with --devices win11-chrome,mac-edge
17
+ # (or ask ForgeQE's agent: it adds, changes, and removes devices here). Chrome or Edge only: ForgeQE drives the browser over CDP.
18
+ # devices:
19
+ # win11-chrome:
20
+ # browser: chrome
21
+ # os: Windows
22
+ # os_version: "11"
23
+ # mac-edge:
24
+ # browser: edge
25
+ # os: OS X
26
+ # os_version: Sequoia
@@ -381,10 +381,10 @@ forgeqe docs --copy ./docs # copy the guide, the architecture page, and the re
381
381
  <div class="area"><a class="area-name" href="#area-automated-tests"><span class="mono">Automated tests</span></a><ul class="tiles"><li><a class="tile" href="#cap-write-tests"><b>Turn a passed run into an automated test</b><span>Write a Playwright test for that run in ../app</span></a></li><li><a class="tile" href="#cap-heal-tests"><b>Heal failing tests</b><span>Heal the failing tests in ../app</span></a></li><li><a class="tile" href="#cap-find-run-tests"><b>Find and run existing tests</b><span>Run the checkout tests</span></a></li><li><a class="tile" href="#cap-flaky-tests"><b>Flaky test detection and quarantine</b><span>Which tests are flaky?</span></a></li></ul></div>
382
382
  <div class="area"><a class="area-name" href="#area-pull-requests"><span class="mono">Pull requests</span></a><ul class="tiles"><li><a class="tile" href="#cap-pr-review"><b>Review pull requests</b><span>Review PR 42 and focus on error handling</span></a></li><li><a class="tile" href="#cap-pr-impact"><b>Pull request impact and test selection</b><span>What does PR 42 impact?</span></a></li></ul></div>
383
383
  <div class="area"><a class="area-name" href="#area-api"><span class="mono">API</span></a><ul class="tiles"><li><a class="tile" href="#cap-api-testing"><b>API and contract testing</b><span>Turn the last run's API calls into tests</span></a></li></ul></div>
384
- <div class="area"><a class="area-name" href="#area-quality"><span class="mono">Quality</span></a><ul class="tiles"><li><a class="tile" href="#cap-accessibility-performance"><b>Accessibility and page performance</b><span>Is the checkout page accessible?</span></a></li><li><a class="tile" href="#cap-visual-comparison"><b>Visual comparison with baselines</b><span>What changed visually since the last run?</span></a></li></ul></div>
384
+ <div class="area"><a class="area-name" href="#area-quality"><span class="mono">Quality</span></a><ul class="tiles"><li><a class="tile" href="#cap-accessibility-performance"><b>Accessibility and page performance</b><span>Is the checkout page accessible?</span></a></li><li><a class="tile" href="#cap-visual-comparison"><b>Visual comparison with baselines</b><span>What changed visually since the last run?</span></a></li><li><a class="tile" href="#cap-defect-intelligence"><b>Defect intelligence: trends and prediction</b><span>Which defects keep coming back?</span></a></li></ul></div>
385
385
  <div class="area"><a class="area-name" href="#area-release"><span class="mono">Release</span></a><ul class="tiles"><li><a class="tile" href="#cap-release-readiness"><b>Requirement coverage and release readiness</b><span>Are we ready to release?</span></a></li><li><a class="tile" href="#cap-reports"><b>Reports you can share</b><span>Create a release report for all projects</span></a></li></ul></div>
386
386
  <div class="area"><a class="area-name" href="#area-operations"><span class="mono">Operations</span></a><ul class="tiles"><li><a class="tile" href="#cap-monitoring"><b>Scheduled monitoring with alerts</b><span>Check the login flow every 15 minutes and alert Teams</span></a></li><li><a class="tile" href="#cap-notifications"><b>Teams and email notifications</b><span>Post the results to the QA channel</span></a></li><li><a class="tile" href="#cap-pipelines"><b>CI pipelines</b><span>Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make it green</span></a></li></ul></div>
387
- <div class="area"><a class="area-name" href="#area-systems"><span class="mono">Systems</span></a><ul class="tiles"><li><a class="tile" href="#cap-systems"><b>Work with Azure DevOps, Jira, and Confluence</b><span>Show the open bugs in this sprint</span></a></li></ul></div>
387
+ <div class="area"><a class="area-name" href="#area-systems"><span class="mono">Systems</span></a><ul class="tiles"><li><a class="tile" href="#cap-systems"><b>Work with Azure DevOps, Jira, GitHub, and Confluence</b><span>Show the open bugs in this sprint</span></a></li></ul></div>
388
388
  <div class="area"><a class="area-name" href="#area-setup"><span class="mono">Setup</span></a><ul class="tiles"><li><a class="tile" href="#cap-setup-help"><b>Setup, skills, and documentation</b><span>What can you do?</span></a></li></ul></div>
389
389
  <h3 class="band-title">Not supported</h3>
390
390
  <ul class="plainlist"><li><b>Native mobile apps (iOS, Android) and desktop applications</b>. Instead: web applications in local Chrome or Chromium, or configured remote web grids such as BrowserStack and TestMu AI/LambdaTest.</li><li><b>Load, stress, or soak testing with many simultaneous users</b>. Instead: per-page Core Web Vitals with budgets.</li><li><b>Penetration testing or dynamic security scanning</b>. Instead: security findings in pull request reviews.</li></ul>
@@ -664,7 +664,7 @@ Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make i
664
664
  <p>The text is the agent's own result. Failed runs show <code>✗ Task failed</code> with the reason, and stopped runs show <code>■ Run stopped</code>. <code>forgeqe run</code> prints the same summary at the end.</p>
665
665
  </div>
666
666
  <section class="doc sub" id="doc-how_to_use-the-screen"><h4>The screen</h4><div class="prose"><p><strong>The start screen.</strong> Before your first request, the screen is one column under the logo:</p>
667
- <pre><code> playwright-csharp-pom-framework · .NET · Azure DevOps · azure/gpt-5.4
667
+ <pre><code> playwright-csharp-pom-framework · .NET · Azure DevOps
668
668
 
669
669
  ┃ Describe what you want to test…
670
670
  ✦ Refine ctrl+r ⌕ Search ctrl+p ? shortcuts ctrl+c close
@@ -676,9 +676,10 @@ Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make i
676
676
  ▤ Report QI-Payments pipeline status report 18 min ago open
677
677
  ✓ Run Search hotels in Mumbai on booking.com 2 days ago open rerun
678
678
  Setup ▰▰▰▱▱ 3 of 5 · Domain and Notifications are not set up ▸ how to set up
679
+ ● Tip Press Tab to switch LLM profile azure/gpt-5.4 (default) · ForgeQE
679
680
  </code></pre>
680
681
  <ul>
681
- <li>The line under the logo is what ForgeQE works with here: the folder, what it is built with (from its files, such as a <code>.sln</code> or <code>playwright.config.ts</code>), the systems that are set up, and the model.</li>
682
+ <li>The line under the logo is what ForgeQE works with here: the folder, what it is built with (from its files, such as a <code>.sln</code> or <code>playwright.config.ts</code>), the systems that are set up. The model and its profile are in the footer, the status line on every screen; Tab switches the profile.</li>
682
683
  <li>The prompt starts five rows high and grows with a long request, up to nearly half the screen, before it scrolls; the sections below give way, last first, to make room.</li>
683
684
  <li><strong>Start with</strong> offers up to four things to ask, from the capabilities that work in this setup, each with an example in the words of your setup. Click one, or press Alt-1 to Alt-4, to put it in the prompt to adjust and send.</li>
684
685
  <li>In a folder without <code>forgeqe.yaml</code>, or without a live model, the line under the logo says so and <strong>Get started</strong> takes its place: <strong>Set up ForgeQE in this folder</strong> (runs <code>/init</code>), <strong>Add a model</strong>, and <strong>Sign in to Azure DevOps or Jira</strong> (runs <code>/auth</code>). Start ForgeQE in your test project's folder to work with its settings.</li>
@@ -700,27 +701,28 @@ Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make i
700
701
  ⏺ No classic releases exist in any of the seven projects.
701
702
  …
702
703
 
703
- ⠹ Thinking… (24s · 57.4K tokens · esc to interrupt)
704
+ ⠹ Acting in the browser (step 3)… (24s · step 3/40 · 57.4K tokens · esc to interrupt)
704
705
  ╭──────────────────────────────────────────────────────────╮
705
706
  │ &gt; Ask ForgeQE… (/ for commands, @ to reference a run…) │
706
707
  ╰──────────────────────────────────────────────────────────╯
707
708
  ✦ Refine ctrl+r ⌕ Search ctrl+p ▦ Panels ctrl+o ctrl+c close
708
- ~/playwright-csharp-pom-framework azure/gpt-5.4 · 57.4K tokens · ForgeQE
709
+ ~/playwright-csharp-pom-framework azure/gpt-5.4 (default) · 57.4K tokens · ForgeQE
709
710
  </code></pre>
710
711
  <ul>
711
712
  <li><code>&gt;</code> is what you asked. <code>⏺</code> starts each reply, tool call, browser run, test case, and result; calls in a row sit together, with the arguments dimmed. What came of a call is indented under it after <code>⎿</code>: the agent's notes, each browser step and its actions (✓ or ✗), and a result's details.</li>
712
- <li>While a request runs, one line above the input says what is happening, for how long, and with how many tokens. The footer is the status line: the model and what the session used. Prices are not shown in the conversation; reports and <code>forgeqe models</code> have them.</li>
713
+ <li>While a request runs, one line above the input says what is happening, for how long, how far a browser run or suite has got, and with how many tokens. The footer is the status line, and the one place for the folder, the model and its profile, and what the session used.</li>
714
+ <li>PgUp and PgDn scroll the conversation from the prompt (the mouse wheel works too); PgDn back at the end follows new lines again.</li>
715
+ <li>A long request you sent (a refined one, a pasted spec) shows its first lines and <em>… 27 more lines, sent in full</em>. Prices are not shown in the conversation; reports and <code>forgeqe models</code> have them.</li>
713
716
  <li>A result that made nothing (a run that only looked something up) adds no card.</li>
714
717
  </ul>
715
- <p><strong>The panels.</strong> <strong>▦ Panels</strong> (Ctrl-O) opens the session, plan, activity, files, context, and tokens panels beside the conversation on a terminal 120 columns wide or more, or in its place on a narrower one; Ctrl-O again, or <strong>✕ Close</strong>, closes them:</p>
718
+ <p><strong>The panels.</strong> <strong>▦ Panels</strong> (Ctrl-O) splits the screen in half on a terminal 120 columns wide or more: the <strong>Activities</strong> column on the right, and the conversation with the files, context, and tokens panels on the left. On a narrower terminal the panels take the conversation's place. Ctrl-O again, or <strong>✕ Close</strong>, closes them.</p>
719
+ <p>The Activities column shows what the agents do as they do it:</p>
716
720
  <ul>
717
- <li>the status line, with who is working, for how long, a progress bar for the plan or a suite's cases, and <strong>■ Stop</strong>;</li>
718
- <li><strong>Plan</strong>: the agent's to-do list, then each browser step or suite case with ✓, ✗ and why, ◐ running;</li>
719
- <li><strong>Activity</strong>: every tool call, browser step, and the agents' reasoning;</li>
720
- <li><strong>Files</strong>: each file read ◉, written ✚, or edited ✎, newest first;</li>
721
- <li><strong>Context</strong>: how full the model's context window is, and the skills, product knowledge, domain, and MCP servers in it;</li>
722
- <li><strong>Tokens</strong>: total, input, output, cached.</li>
721
+ <li>the status line: who is working, a progress bar for the plan or a suite's cases, for how long, and <strong>■ Stop</strong>; under it, the whole width, what is happening now (<em>Agent: Azure DevOps API · QI-Payments/_apis/build/builds…</em>);</li>
722
+ <li><strong>Plan</strong>: the agent's own task list as soon as it writes one, then each browser step or suite case: ○ pending, ◐ in progress, ✓ done, ✗ failed and why. ForgeQE's agent writes a plan for any request of three steps or more and updates it as it goes;</li>
723
+ <li><strong>Activity</strong>: <strong>Now</strong> (the task in progress) and <strong>Next</strong> (the next task), then every tool call, browser step, and note. A call shows ◐ the moment it starts and becomes ✓ or ✗ in the same row when it ends, so a long API query or test run is never invisible. Reads and searches that the conversation leaves out at the brief detail level are listed here. A call takes one row and a note up to three; focus the panel to scroll through everything or filter it.</li>
723
724
  </ul>
725
+ <p>Over the conversation: <strong>Files</strong> (each file read ◉, written ✚, or edited ✎, newest first), <strong>Context</strong> (how full the model's context window is, and the skills, product knowledge, domain, and MCP servers in it), and <strong>Tokens</strong> (total, input, output, cached). When that half is under 80 columns they fold into one line: <em>Files 4 (1 changed) · Context 26% · Skills 2 · 52.9K tokens</em>.</p>
724
726
  <p><strong>Buttons.</strong> Under the input: <strong>✦ Refine</strong> (Ctrl-R; it fills in once there is text to refine), <strong>⌕ Search</strong> (Ctrl-P, the command palette), and <strong>▦ Panels</strong> (Ctrl-O); while a request runs, <strong>■ Stop</strong> (Esc) takes Refine's place. With the panels open, <strong>▦ Focus panels</strong> (Shift-Tab) and <strong>✕ Close</strong> (Ctrl-O) are in their status line. On a narrow terminal, <strong>◧ Details</strong> (Ctrl-O) and <strong>⇄ Profile</strong> (Tab) are under the input. Each is clickable.</p>
725
727
  <p><strong>Replies.</strong> The agent's answers lead with the answer, then short headed sections; lists keep their bullets in a column, so a wrapped line stays under its text instead of running back to the margin.</p>
726
728
  <p><strong>Moving between panels.</strong> With the panels open, Shift-Tab (or a click) moves the keys from the prompt to Plan, Activity, then Files; the focused panel's border lights up and shows its keys. Esc, or typing, goes back to the prompt.</p>
@@ -752,7 +754,7 @@ Output format: One section for each project, with a line for each environment: t
752
754
  Summary: Show the deployed build in each environment, and why each deployment failed.
753
755
  </code></pre>
754
756
  <p>Refine is one model session of its own, separate from the conversation: a few model steps (usually two) and read-only tool calls, typically 15K–25K tokens and 20–30 seconds. The panel title shows what it cost and how much of the request keeps the ASD-STE100 rules (<em>✦ Refined request · STE 100% · 20.8K tokens · 21s</em>), and its tokens count in the session total in the footer.</p>
755
- <p>A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it <strong>checked</strong> ✓, what it <strong>assumed</strong> ~, and what is <strong>missing</strong> ? (for example &quot;Which story? Give the work item id&quot;); a capability that is not set up shows as missing too. Enter sends it, <code>e</code> puts it in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.</p>
757
+ <p>A read-only agent writes it. It starts knowing every capability and whether it works in this setup, and the domain; it checks names and ids in the systems that are set up (work items, test plans, pipelines, sprints, recent runs). The panel shows the request with what it <strong>checked</strong> ✓, what it <strong>assumed</strong> ~, and what is <strong>missing</strong> ? (for example &quot;Which story? Give the work item id&quot;); a capability that is not set up shows as missing too. What is missing comes first, and the title counts it (<em>✦ Refined request · 2 to answer</em>). A refined request longer than the screen scrolls inside the panel with ↑/↓ and PgUp/PgDn (the border says so); the title and the <strong>Use refined</strong>, <strong>Edit</strong>, and <strong>Keep mine</strong> buttons stay on screen. ←/→ or Tab picks one, Enter does it; <code>e</code> puts the request in the prompt to edit, Esc keeps yours. Refine never runs anything, writes files, or changes a work item; it never invents an id, URL, project, or file.</p>
756
758
  <p><strong>Next steps.</strong> When a request finishes, up to three suggestions appear above the prompt: from what it produced (<em>Open the report</em>, <em>Write a test for this run</em>, <em>Why did it fail?</em>, <em>Rerun the cases that failed</em>) and from the agent's answer, which suggests the follow-ups you are likely to want (<em>Write a release report</em>, <em>Show the failed CD logs</em>). Click one or press Alt-1 to Alt-3; suggestions that start work are put in the prompt first, to adjust and send.</p>
757
759
  <div class="tbl"><table>
758
760
  <thead>
@@ -762,6 +764,7 @@ Summary: Show the deployed build in each environment, and why each deployment fa
762
764
  <tr><td>Enter</td><td>Send the message or command</td></tr>
763
765
  <tr><td>Tab</td><td>Next LLM profile from <code>llms.json</code>; the conversation carries over to the new model. While <code>/</code> or <code>@</code> suggestions show, completes the selected one</td></tr>
764
766
  <tr><td>Shift-Tab</td><td>Move between the chat and the Plan, Activity, and Files panels</td></tr>
767
+ <tr><td>PgUp / PgDn</td><td>Scroll the conversation</td></tr>
765
768
  <tr><td>Ctrl-P</td><td>Command palette</td></tr>
766
769
  <tr><td>Ctrl-R</td><td>Refine the request in the prompt</td></tr>
767
770
  <tr><td>Ctrl-O</td><td>Open or close the panels (beside the conversation, or in its place on a narrow terminal)</td></tr>
@@ -803,10 +806,10 @@ forgeqe run &quot;Run the cases in smoke.yaml and don't post results&quot; --pla
803
806
  <div class="hd"><span class="mono">Browser</span><h3>Run any goal in a real browser</h3></div>
804
807
  <p class="ds">Carries out a plain-language goal in a real browser one step at a time and verifies the outcome, with screenshots, a report, and the steps it took. The browser comes from forgeqe.yaml: local Chrome or Chromium by default, or BrowserStack or TestMu AI/LambdaTest when configured. Page memory replays steps that already worked on a site, so repeat runs are faster and cheaper.</p>
805
808
  <div class="facts">
806
- <div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Go to booking.com and search hotels in Mumbai for next weekend</li><li>What does ForgeQE know about the checkout page?</li></ul></div>
809
+ <div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Go to booking.com and search hotels in Mumbai for next weekend</li><li>What does ForgeQE know about the checkout page?</li><li>Add Chrome on Windows 11 and Chrome 128 on Windows 10 as TestMu devices, then run SHOP-12 on both</li></ul></div>
807
810
  <div class="fact"><span class="mono">What it needs</span><ul class="needs"><li><span class="need req">Required</span><span>A live model profile in llms.json (an API key or Microsoft Entra ID sign-in)</span></li></ul></div>
808
811
  <div class="fact"><span class="mono">Limits</span><ul class="plainlist"><li>Web applications only: local Chrome or Chromium, or configured remote web grids such as BrowserStack and TestMu AI/LambdaTest; not native mobile or desktop applications.</li></ul></div>
809
- <div class="fact"><span class="mono">Tools</span><p class="tools"><code>browser_run</code> <code>run_details</code> <code>page_memory</code></p></div>
812
+ <div class="fact"><span class="mono">Tools</span><p class="tools"><code>browser_run</code> <code>run_details</code> <code>page_memory</code> <code>browser_devices</code></p></div>
810
813
  </div>
811
814
  <div class="detail"><span class="mono">In detail</span>
812
815
  <p class="see">See <a href="#doc-how_to_use-4-without-the-shell">4. Without the shell</a>.</p>
@@ -1074,8 +1077,23 @@ forgeqe suite cases.yaml --no-code --no-write-back --out .forgeqe/reports/smoke
1074
1077
  labels: [forgeqe] # Jira labels or Azure DevOps tags
1075
1078
  # area_path: Shop\Web # Azure DevOps
1076
1079
  # on_error: true # also for cases that could not run
1080
+ # group_by: defect # defect (default): cases failing the same way share a bug; case: a bug per case
1081
+ </code></pre>
1082
+ <p>When results are posted, each case that failed with a functional defect (below) gets a bug: the title is <code>[ForgeQE] &lt;case id&gt;: &lt;title&gt;</code>, the description has the result, the steps with verdicts, the preconditions, where the run ended, the build link, and the run folder, and the last screenshot is attached. In Jira the bug is linked to the Jira or Xray test (&quot;relates to&quot;); in Azure DevOps it is &quot;tested by&quot; the test case work item. Each bug carries a label or tag that fingerprints the case (<code>forgeqe-</code> and ten hex digits), so the next failure comments &quot;Still failing&quot; on the open bug instead of filing a new one, and a pass adds a note that it passes again. A closed bug is not reopened: a new failure after that files a new bug, titled <code>[ForgeQE] Regression: …</code>, that says when the old one was closed and links to it (Jira &quot;relates to&quot;, Azure DevOps &quot;Related&quot;, a mention on GitHub; a GitHub issue closed as not planned does not count).</p>
1083
+ <p><strong>One bug per defect.</strong> Cases that fail the same way share one bug (<code>group_by: defect</code>, the default). ForgeQE gives each failing case a signature from the strongest evidence its run recorded: a failed request (a 5xx before a 4xx) by its route and status, then an uncaught page error, then a console error, with ids, numbers and times taken out, so <code>POST /api/cart/123 → 500</code> and <code>/api/cart/456 → 500</code> match. Errors that the suite's passing runs show too (an analytics call, a missing favicon) are noise and do not count. The first case of a defect files the bug, titled after the defect (<code>[ForgeQE] shop.example.com/api/cart/:id returns 500</code>) and labelled with the defect's fingerprint (<code>forgeqe-d-</code> and ten hex digits) as well as the case's; every other case failing the same way is added as a comment and gets its own case label on that bug. The next run finds the open bug by either label. A failure with no browser evidence is matched by its page and what the agent saw, within one run only, and its bug is titled after the case as before. <code>group_by: case</code> files one bug per failing case. The report's <strong>Defects</strong> page, the terminal summary and the CI job summary list every defect with the cases it breaks.</p>
1084
+ <p><strong>Defect history.</strong> Each suite adds its defects to <code>defect-history.json</code> next to the suite folders (keep it between CI runs as a cache, like the suite folders, to track a pipeline's defects). A defect is <strong>new</strong> the first time, then <strong>known</strong> (&quot;seen in 3 earlier runs since 2 Oct&quot;). It <strong>went away</strong> when a run did not see it while a case it broke passed; seen again after that, or matching a closed bug in the tracker, it is a <strong>regression</strong>. A defect whose cases keep changing between passing and failing across the last 20 suites (pass, fail, pass) is marked <strong>intermittent</strong>: it still gets a bug, which says so. When a new bug is filed, ForgeQE also searches the tracker once for open bugs people filed about the same route or error (<code>/api/cart</code>, <code>price is undefined</code>) and lists them on the bug as possibly related; it never merges them. Only defects found by browser evidence are kept in the history: a defect matched by the agent's words gets new words every run. The Defects page shows the history and filters new defects and regressions; the summaries say it next to each defect.</p>
1085
+ <p><strong>Reproducing and root cause.</strong> Each defect gets a repro file, <code>defects/&lt;defect id&gt;.md</code> in the suite folder, and the same steps go in its bug: how reproducible it is, the case's steps with the agent's verdicts, what the browser did, where it ended, the environment (browser, device, build, commit), the failing request as a <code>curl</code> command with its response (the engine takes credentials out of the requests it records), and the run's files (screenshot, replay flow, every step, every API call). The first case of a new product defect runs once more into its <code>reproduce/</code> folder: failing the same way makes it <strong>always</strong> reproducible, not failing again makes it <strong>intermittent</strong>; a defect seen in earlier runs is not run again. Then the root cause, always as a hypothesis:</p>
1086
+ <ul>
1087
+ <li>From the evidence, at no cost: the layer (the server, the front end, access, a route that is not there, the environment) and the component (the route or the script).</li>
1088
+ <li>With <code>defects.product_repo</code> set: <strong>suspect commits</strong>, made since the defect's cases last passed, whose changes mention its route or error (<code>git log -G</code>), or whose files are named after it.</li>
1089
+ <li>For new defects and regressions, a read-only agent (prompt <code>defect-rca</code>) reads the evidence, the suspect commits and the product's code, and writes a hypothesis with its confidence (high only when it read the code that fails and the change that broke it), what it rests on, and checks to confirm it. The hypothesis is added to the bug as a comment and shown on the Defects page. It runs for at most <code>max_analyses</code> defects per suite; each is one agent session (about 90K–120K tokens with gpt-5.4 in our runs), shown on its own line under Usage.</li>
1090
+ </ul>
1091
+ <pre><code class="language-yaml">defects:
1092
+ product_repo: ../shop # the application's repository: suspect commits, and the code the analysis reads
1093
+ # reproduce: true # run the first case of each new defect once more
1094
+ # analyze: true # a model writes a root-cause hypothesis for new defects and regressions
1095
+ # max_analyses: 3 # most analyses per suite
1077
1096
  </code></pre>
1078
- <p>When results are posted, each case that failed with a functional defect (below) gets a bug: the title is <code>[ForgeQE] &lt;case id&gt;: &lt;title&gt;</code>, the description has the result, the steps with verdicts, the preconditions, where the run ended, the build link, and the run folder, and the last screenshot is attached. In Jira the bug is linked to the Jira or Xray test (&quot;relates to&quot;); in Azure DevOps it is &quot;tested by&quot; the test case work item. Each bug carries a label or tag that fingerprints the case (<code>forgeqe-</code> and ten hex digits), so the next failure comments &quot;Still failing&quot; on the open bug instead of filing a new one, and a pass adds a note that it passes again. Closed bugs are left alone; a new failure after that files a new bug.</p>
1079
1097
  </div>
1080
1098
  </section>
1081
1099
  </div>
@@ -1256,6 +1274,7 @@ Acceptance criteria:
1256
1274
  stall_after: 5 # stop the agent's turn when one command it runs has failed this many times
1257
1275
  replay: true # before the agent starts, run the steps as a Playwright script and hand over what worked
1258
1276
  live_browser: true # once a step fails, the agent looks at that page in a Playwright browser
1277
+ graph: true # the code graph: find page objects without reading files, and reuse what already locates an element
1259
1278
  # model: anthropic/claude-sonnet-4-5
1260
1279
  # runtime: /path/to/agent-runtime # default: the runtime ForgeQE downloads
1261
1280
  </code></pre>
@@ -1271,6 +1290,7 @@ Acceptance criteria:
1271
1290
  </li>
1272
1291
  <li>
1273
1292
  <p>Scribe starts in the repo. It reads <code>AGENTS.md</code>, the test framework config, run scripts, page objects, fixtures, and helpers, and follows the imports between them. It writes the test in the repo's language and framework (a C# NUnit repo gets a C# NUnit test) and to fit its code: it extends an existing page object instead of duplicating it and reuses the login flow, test data, and config. A repo with no tests gets Playwright Test in TypeScript. It may change anything the test needs, including dependencies and config.</p>
1293
+ <p>Before Scribe starts, ForgeQE reads the repo into a <strong>code graph</strong>, without a model: every class, page object, method, property, and test (TypeScript, JavaScript, C#, Java, Python), the calls between them, and the locators each one builds, in any framework's spelling (Playwright, Selenium, Cypress, WebdriverIO, Testing Library). It is kept in <code>~/.forgeqe/cache/code-graph</code>, nothing leaves the machine, and the next job reads again only the files that changed. The graph is tied to page memory, which knows each element by all its locators (its id, role and name, text, test id): so when the run clicked <code>getByRole('button', { name: 'Login' })</code> and the repo's <code>LoginPage.LoginButton</code> uses <code>#login-button</code>, the handoff says under that step &quot;in this repository: LoginPage.LoginButton … used by LoginPage.SignInAsync&quot;, and Scribe calls the existing method instead of writing a second locator. Scribe also gets tools that answer from the graph instead of grep and whole-file reads: <code>code_find</code> (classes, methods, and tests by name, or by the text a locator uses), <code>code_outline</code> (a file's or class's signatures and locators, or the source of one member), <code>code_element</code> (which member already locates an element), <code>code_references</code> (what calls a member, before it is changed), and <code>code_impact</code> (the tests a change reaches, through flows and base classes, whatever the test files are called). <code>graph: false</code> turns it off. The handoff also outlines the page objects its steps use and includes the repo's test that uses the most of them, so Scribe takes the test conventions from that test instead of reading other tests, and reads a settings file only to change it.</p>
1274
1294
  </li>
1275
1295
  <li>
1276
1296
  <p>It writes the test from the handoff and the replay script: steps that worked are ported with their locators, steps that failed are fixed first. Every locator, including those for assertions, comes from the replay script, the page snapshots, page memory, or a check on the page; steps where the browser agent ran JavaScript to look around are left out. It does not walk the flow again or build tests to explore. Once a step fails (in the replay, or in its test), ForgeQE switches on its live browser: a Chrome that Playwright drives, set up like ForgeQE's, where it looks at the failing step's page one tool call per action (open, snapshot, click, type, wait, screenshot, run JavaScript, console errors, failed requests). Until then the live browser tools are off, so no tokens go into looking at pages that work. The live browser needs Node.js (<code>npx</code>), and Microsoft's Playwright MCP server is downloaded on first use; with <code>live_browser: false</code>, or without <code>npx</code>, it looks with a small throwaway script instead. When its browser gets a different page than ForgeQE's did (an error, a challenge, a control that will not open), it compares the two browsers (user agent, <code>navigator.webdriver</code>, headless, failed requests and CORS errors, headers the framework adds, console errors) and fixes the difference in the repo's browser configuration. It never judges the site with <code>curl</code> or <code>webfetch</code>.</p>
@@ -1282,7 +1302,7 @@ Acceptance criteria:
1282
1302
  <p>ForgeQE runs that command (with <code>CI=1</code>) and checks the exit code, and it watches the processes the command starts for a browser launched by Playwright, Puppeteer, Selenium, or Cypress. On a failure it sends the command, exit code, and output back to Scribe. The output is kept short so the error is never cut off: the page dump Playwright puts into a failed assertion (&quot;Aria snapshot:&quot;, often thousands of lines) is taken out and saved as <code>&lt;run&gt;/test-runs/attempt-N-page.aria.yml</code>, then the start (where the first error and its stack trace are) and the end (the summary) are sent; the full output is in <code>&lt;run&gt;/test-runs/attempt-N.log</code>, and the prompt names both files. A command that exits 0 without starting a browser also goes back: the test must drive the app in a browser. If the repo's browser runs on another machine (Selenium Grid, a cloud service, a container), the agent says where and the check is skipped.</p>
1283
1303
  </li>
1284
1304
  <li>
1285
- <p>A passing test then goes to a separate reviewer agent, which can read files and <code>git diff</code> but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (<code>.forgeqe/</code>, <code>forgeqe.yaml</code>, <code>llms.json</code>) are never part of the review. If it finds problems, they go back to Scribe as the next attempt.</p>
1305
+ <p>A passing test then goes to a separate reviewer agent, which can read files and <code>git diff</code> but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (<code>.forgeqe/</code>, <code>forgeqe.yaml</code>, <code>llms.json</code>) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same <code>code_*</code> tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt.</p>
1286
1306
  <p>Scribe cannot edit ForgeQE's files or run folders, and cannot run <code>git clean</code>, <code>git reset --hard</code>, <code>git stash</code>, <code>git restore</code>, or <code>git checkout -- &lt;path&gt;</code>. If it still removes a file that existed before the job, or reverts your uncommitted edits, ForgeQE puts the file back after that attempt and tells the agent.</p>
1287
1307
  </li>
1288
1308
  <li>
@@ -1715,6 +1735,40 @@ forgeqe visual approve --page /home # make this run's /home the baseline (asks
1715
1735
  </section>
1716
1736
  </div>
1717
1737
  </article>
1738
+ <article class="spec feat" id="cap-defect-intelligence">
1739
+ <div class="hd"><span class="mono">Quality</span><h3>Defect intelligence: trends and prediction</h3></div>
1740
+ <p class="ds">Groups failures into defects, keeps their history across suites (new, known, gone, back again), and gives defect trends and where a change is likely to cause defects, with the cases to run first.</p>
1741
+ <div class="facts">
1742
+ <div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Which defects keep coming back?</li><li>What can the changes since main break?</li></ul></div>
1743
+ <div class="fact"><span class="mono">What it needs</span><ul class="needs"><li><span class="need input">You give</span><span>suites run into one folder (suite.out_dir)</span></li><li><span class="need opt">Optional</span><span>git on PATH, for predictions from the product's repository</span></li></ul></div>
1744
+ <div class="fact"><span class="mono">Limits</span><ul class="plainlist"><li>Prediction is a heuristic from the saved suites and the change, not a trained model; its accuracy shows once suites have scored it.</li></ul></div>
1745
+ <div class="fact"><span class="mono">Tools</span><p class="tools"><code>defect_trends</code> <code>predict_defects</code></p></div>
1746
+ </div>
1747
+ <div class="detail"><span class="mono">In detail</span>
1748
+ <p class="see">See <a href="#doc-how_to_use-bugs-for-failures">Bugs for failures</a>.</p>
1749
+ <section class="doc" id="doc-how_to_use-defect-trends-and-prediction">
1750
+ <h4>Defect trends and prediction</h4>
1751
+ <div class="prose"><p><code>forgeqe defects</code> (or &quot;which defects keep coming back?&quot;) reads the suites saved in <code>suite.out_dir</code> and their <code>defect-history.json</code>, and writes <code>defect-trends.md</code> next to them: how many defects are still there, gone, or came back after a fix; the time they took to go away; the most seen ones with their bugs; defects by area (a page or an API route, by its path without ids); defects per suite; and how often predictions were right. The report's Defects page shows the same under <strong>Across runs</strong>.</p>
1752
+ <p><code>forgeqe defects predict [--repo DIR] [--base REF] [--head REF]</code> (or &quot;what can the changes since main break?&quot;) says where a change is likely to cause defects. It reads the change between the merge base of <code>base</code> (default: the remote's default branch, else <code>main</code> or <code>master</code>) and <code>head</code> in <code>defects.product_repo</code> (else the working directory). A change touches an area when it changes a file named after it (<code>server/cart.ts</code> for <code>/api/cart/:id</code>) or its diff mentions the route. Each touched area gets a score from what the saved suites know:</p>
1753
+ <div class="tbl"><table>
1754
+ <thead>
1755
+ <tr><th>Signal</th><th>Score</th></tr>
1756
+ </thead>
1757
+ <tbody>
1758
+ <tr><td>The diff mentions the area's route</td><td>+3</td></tr>
1759
+ <tr><td>A changed file is named after it</td><td>+2</td></tr>
1760
+ <tr><td>Each earlier defect there (three at most)</td><td>+2</td></tr>
1761
+ <tr><td>Defects there came back after a fix</td><td>+2</td></tr>
1762
+ <tr><td>A defect there is still open</td><td>+1</td></tr>
1763
+ <tr><td>Its cases failed in 20% (50%) of recent runs</td><td>+1 (+2)</td></tr>
1764
+ </tbody>
1765
+ </table></div>
1766
+ <p>6 or more is high, 3 or more medium. It lists the cases that ended on the high and medium areas as the ones to run first, and the changed product code that no case reaches. It is a heuristic, not a trained model: each prediction is saved in <code>defect-predictions.json</code>, and the first suite after it scores it (did its defects fall in the areas rated medium or high; did the high areas get one), so <code>forgeqe defects</code> gives its accuracy.</p>
1767
+ <p>Release readiness uses the defects too: a regression on a case in scope blocks the release (NO-GO), and a new defect with no bug is a risk.</p>
1768
+ </div>
1769
+ </section>
1770
+ </div>
1771
+ </article>
1718
1772
  </section>
1719
1773
  <section class="chapter" id="area-release" aria-labelledby="area-release-h">
1720
1774
  <header class="chead"><span class="mono">10</span><h2 id="area-release-h">Release</h2><p>Requirement coverage and a release verdict</p></header>
@@ -1927,12 +1981,12 @@ forgeqe run &quot;The ForgeQE workflow is failing, fix it&quot;
1927
1981
  <section class="chapter" id="area-systems" aria-labelledby="area-systems-h">
1928
1982
  <header class="chead"><span class="mono">12</span><h2 id="area-systems-h">Systems</h2><p>Azure DevOps, Jira, and Confluence</p></header>
1929
1983
  <article class="spec feat" id="cap-systems">
1930
- <div class="hd"><span class="mono">Systems</span><h3>Work with Azure DevOps, Jira, and Confluence</h3></div>
1931
- <p class="ds">Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, and builds Azure DevOps dashboards and wiki pages.</p>
1984
+ <div class="hd"><span class="mono">Systems</span><h3>Work with Azure DevOps, Jira, GitHub, and Confluence</h3></div>
1985
+ <p class="ds">Reads anything in the configured systems (work items, issues, queries, boards, pages), makes changes the user asks for, puts run evidence (screenshots, reports) on work items and issues as attachments or comments, and builds Azure DevOps dashboards and wiki pages.</p>
1932
1986
  <div class="facts">
1933
- <div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Show the open bugs in this sprint</li><li>Read the checkout requirements from Confluence</li><li>Make a dashboard of this sprint's test results</li></ul></div>
1987
+ <div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Show the open bugs in this sprint</li><li>Read the checkout requirements from Confluence</li><li>Make a dashboard of this sprint's test results</li><li>Add the last screenshot of that run to ADO 256 as a comment</li><li>Run the GitHub issues labelled login as test cases</li></ul></div>
1934
1988
  <div class="fact"><span class="mono">What it needs</span><ul class="needs"><li><span class="need req">Required</span><span>Azure DevOps or Jira or Xray or Confluence: set up under sources: in forgeqe.yaml, or sign in with /auth</span></li></ul></div>
1935
- <div class="fact"><span class="mono">Tools</span><p class="tools"><code>query_api</code> <code>read_confluence</code> <code>create_ado_dashboard</code> <code>write_ado_wiki_page</code></p></div>
1989
+ <div class="fact"><span class="mono">Tools</span><p class="tools"><code>query_api</code> <code>read_confluence</code> <code>create_ado_dashboard</code> <code>write_ado_wiki_page</code> <code>attach_to_work_item</code></p></div>
1936
1990
  </div>
1937
1991
  <div class="detail"><span class="mono">In detail</span>
1938
1992
  <section class="doc" id="doc-readme-azure-devops-jira-and-other-systems">
@@ -1946,6 +2000,7 @@ forgeqe run &quot;The ForgeQE workflow is failing, fix it&quot;
1946
2000
  <tr><td>Azure DevOps Services</td><td>Microsoft's <a href="https://github.com/microsoft/azure-devops-mcp" rel="noopener">Azure DevOps MCP server</a> (<code>npx @azure-devops/mcp</code>), with all its tools: work items, boards, test plans, pipelines, repos, wiki pages, search, and security alerts</td><td>Node.js, and <code>ADO_PAT</code> or <code>az login</code></td></tr>
1947
2001
  <tr><td>Jira Cloud, Server/Data Center</td><td><a href="https://github.com/sooperset/mcp-atlassian" rel="noopener">mcp-atlassian</a> (<code>uvx</code>), with issues, JQL search, comments, and transitions. <code>mcp: rovo</code> uses <a href="https://github.com/atlassian/atlassian-mcp-server" rel="noopener">Atlassian's hosted server</a> instead (Cloud, and an admin must allow API tokens)</td><td>uv; <code>JIRA_EMAIL</code> and <code>JIRA_API_TOKEN</code> (Cloud) or a personal access token (Server/DC)</td></tr>
1948
2002
  <tr><td>Confluence</td><td>mcp-atlassian, shared with Jira (or its own when Jira isn't set up); Atlassian's hosted server covers it on the same site</td><td><code>confluence: true</code> on a Jira Cloud site, or <code>confluence.url</code> + tokens</td></tr>
2003
+ <tr><td>GitHub</td><td>GitHub's <a href="https://github.com/github/github-mcp-server" rel="noopener">hosted MCP server</a> with the repos, issues, pull requests, and Actions toolsets (<code>sources.github.toolsets</code> changes them), on in a repository with a GitHub remote or when <code>sources.github</code> is set. GitHub issues are also a test case source (<code>github:#12</code>, <code>github:acme/shop#12,#13</code>, <code>github:label=login</code>, <code>github:q=login in:title</code>): results go back as issue comments, and <code>sources.bugs.system: github</code> files bugs as issues. GitHub has no API to upload a screenshot to an issue, so a bug names the run folder instead</td><td><code>GITHUB_TOKEN</code>, or the GitHub CLI's sign-in (<code>forgeqe auth login github</code>)</td></tr>
1949
2004
  <tr><td>Xray, TestRail, Zephyr Scale, qTest</td><td>none; ForgeQE's API</td><td><code>XRAY_CLIENT_ID</code> / <code>XRAY_CLIENT_SECRET</code>, <code>TESTRAIL_EMAIL</code> / <code>TESTRAIL_API_KEY</code>, <code>ZEPHYR_API_TOKEN</code>, <code>QTEST_TOKEN</code></td></tr>
1950
2005
  <tr><td>Anything else</td><td>servers you add under <code>mcp:</code></td><td></td></tr>
1951
2006
  </tbody>