@capagents/forgeqe 1.1.8 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/HOW_TO_USE.md +31 -1
- package/docs/user-guide.html +30 -5
- package/package.json +1 -1
- package/prompts/brain.md +6 -0
- package/prompts/test-rules.md +2 -0
- package/src/code/coverage.ts +117 -0
- package/src/code/exchange.ts +74 -0
- package/src/code/graph/graph.ts +30 -3
- package/src/code/loop.ts +70 -7
- package/src/code/notes.ts +76 -0
- package/src/code/prompt.ts +52 -3
- package/src/code/review.ts +13 -0
- package/src/detail.ts +2 -1
- package/src/mcp/commands.ts +76 -0
- package/src/mcp/tools.ts +69 -2
- package/src/print.ts +9 -0
- package/src/tui/App.tsx +23 -6
- package/src/tui/components.tsx +4 -3
- package/src/tui/text.ts +16 -0
- package/src/tui/theme.ts +25 -26
- package/src/tui/transcript.tsx +22 -9
- package/src/types.ts +8 -0
package/HOW_TO_USE.md
CHANGED
|
@@ -288,6 +288,21 @@ A profile's own `"price": { "input": 2.5, "output": 15, "cached_input": 0.25 }`
|
|
|
288
288
|
This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
|
|
289
289
|
```
|
|
290
290
|
|
|
291
|
+
**Reuse before rebuild, extend before duplicate.** Before Scribe writes anything, `write_test` measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as `Expect(practicePage.AjaxStatus)`). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:
|
|
292
|
+
|
|
293
|
+
| What it finds | What happens |
|
|
294
|
+
|---|---|
|
|
295
|
+
| One test covers every measured step and check | **Reuse**: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test. |
|
|
296
|
+
| One test covers most (40% or more) | **Extend**: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file |
|
|
297
|
+
| Two or more tests cover most of it | **You choose**: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel) |
|
|
298
|
+
| Nothing covers much | **New**: a new test, reusing the page objects and helpers as before |
|
|
299
|
+
|
|
300
|
+
A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing*). Say how when you know ("extend the login test", "write a separate test", "update it") and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.
|
|
301
|
+
|
|
302
|
+
**The same flow asked again.** When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (*✓ Existing test passed · 19s · no browser run or new test needed*). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.
|
|
303
|
+
|
|
304
|
+
**Why a test failed.** When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's `E` lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.
|
|
305
|
+
|
|
291
306
|
**Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
|
|
292
307
|
|
|
293
308
|
`forgeqe.yaml` selects the active profile:
|
|
@@ -647,7 +662,22 @@ What happens after the run passes:
|
|
|
647
662
|
4. It writes the test from the handoff and the replay script: steps that worked are ported with their locators, steps that failed are fixed first. Every locator, including those for assertions, comes from the replay script, the page snapshots, page memory, or a check on the page; steps where the browser agent ran JavaScript to look around are left out. It does not walk the flow again or build tests to explore. Once a step fails (in the replay, or in its test), ForgeQE switches on its live browser: a Chrome that Playwright drives, set up like ForgeQE's, where it looks at the failing step's page one tool call per action (open, snapshot, click, type, wait, screenshot, run JavaScript, console errors, failed requests). Until then the live browser tools are off, so no tokens go into looking at pages that work. The live browser needs Node.js (`npx`), and Microsoft's Playwright MCP server is downloaded on first use; with `live_browser: false`, or without `npx`, it looks with a small throwaway script instead. When its browser gets a different page than ForgeQE's did (an error, a challenge, a control that will not open), it compares the two browsers (user agent, `navigator.webdriver`, headless, failed requests and CORS errors, headers the framework adds, console errors) and fixes the difference in the repo's browser configuration. It never judges the site with `curl` or `webfetch`.
|
|
648
663
|
5. Scribe does not run the test itself: it may build or compile, then reports the test file and the command that runs only that test, and ForgeQE runs it. If one command it runs fails `stall_after` times in a row within its work (the same build again and again, say), ForgeQE stops that turn and sends the command and its last output back, asking it to change approach. If the same command stalls again, the job stops as a failure with that output in the notes.
|
|
649
664
|
6. ForgeQE runs that command (with `CI=1`) and checks the exit code, and it watches the processes the command starts for a browser launched by Playwright, Puppeteer, Selenium, or Cypress. On a failure it sends the command, exit code, and output back to Scribe. The output is kept short so the error is never cut off: the page dump Playwright puts into a failed assertion ("Aria snapshot:", often thousands of lines) is taken out and saved as `<run>/test-runs/attempt-N-page.aria.yml`, then the start (where the first error and its stack trace are) and the end (the summary) are sent; the full output is in `<run>/test-runs/attempt-N.log`, and the prompt names both files. A command that exits 0 without starting a browser also goes back: the test must drive the app in a browser. If the repo's browser runs on another machine (Selenium Grid, a cloud service, a container), the agent says where and the check is skipped.
|
|
650
|
-
7. A passing test then goes to a separate reviewer agent, which can read files and `git diff` but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (`.forgeqe/`, `forgeqe.yaml`, `llms.json`) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same `code_*` tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt.
|
|
665
|
+
7. A passing test then goes to a separate reviewer agent, which can read files and `git diff` but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (`.forgeqe/`, `forgeqe.yaml`, `llms.json`) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same `code_*` tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt. The terminal shows the exchange as it happens, in order: the reviewer's verdict with its reason and each issue (✗ correctness, ~ coding standard) with the file and the fix it asks for; then why Scribe tries again and what it is fixing (the reviewer's issues, or the test's first error); then what Scribe says it did and which files it changed; then the next verdict:
|
|
666
|
+
|
|
667
|
+
```
|
|
668
|
+
⏺ Reviewer asked for changes attempt 1 of 5 · 1 to fix, 1 coding-standard
|
|
669
|
+
⎿ The test never checks that the pie chart is shown.
|
|
670
|
+
✗ Correctness · tests/PracticeTests.cs: No assertion on the SVG pie chart.
|
|
671
|
+
Fix: Assert the chart's heading and SVG are visible.
|
|
672
|
+
~ Coding standard · tests/PracticeTests.cs: Test data is inline.
|
|
673
|
+
Fix: Move it to the page object.
|
|
674
|
+
⏺ Scribe tries again · attempt 2 of 5 the reviewer asked for changes
|
|
675
|
+
⎿ → tests/PracticeTests.cs: No assertion on the SVG pie chart. — fix: Assert the chart's heading and SVG are visible.
|
|
676
|
+
⎿ Scribe (attempt 2): Added the pie chart assertion · changed tests/PracticeTests.cs, Pages/PracticePage.cs
|
|
677
|
+
⏺ Reviewer approved the test attempt 2 of 5
|
|
678
|
+
```
|
|
679
|
+
|
|
680
|
+
The Activities panel has one line for each verdict and attempt, and `forgeqe` commands outside the terminal UI print the same blocks.
|
|
651
681
|
|
|
652
682
|
Scribe cannot edit ForgeQE's files or run folders, and cannot run `git clean`, `git reset --hard`, `git stash`, `git restore`, or `git checkout -- <path>`. If it still removes a file that existed before the job, or reverts your uncommitted edits, ForgeQE puts the file back after that attempt and tells the agent.
|
|
653
683
|
8. The job is done when ForgeQE's run passes with a browser and the reviewer approves. It stops as a failure after `max_attempts` attempts, when the job passes `code.max_tokens`, when the reviewer rejects a passing test twice, if the review cannot finish, when the test fails the same way three times in a row (the same output apart from numbers such as timings and line numbers; the second time, Scribe is told its change did not change the failure and to look at the failing step before changing code), or straight away on an auth, API, or context-limit error from the model.
|
package/docs/user-guide.html
CHANGED
|
@@ -576,6 +576,21 @@ forgeqe auth logout jira
|
|
|
576
576
|
<p><strong>Usage after every job.</strong> Every browser run, Scribe job, heal, suite, and monitor round ends with a <code>Usage</code> line: tokens in (and how many of them were cached), tokens out, and the cost at the profile's price. A request to ForgeQE's agent in the shell, and every <code>forgeqe run</code>, <code>code</code>, <code>suite</code>, and <code>heal</code> command, ends with the total for everything it started, by role:</p>
|
|
577
577
|
<pre><code>This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
|
|
578
578
|
</code></pre>
|
|
579
|
+
<p><strong>Reuse before rebuild, extend before duplicate.</strong> Before Scribe writes anything, <code>write_test</code> measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as <code>Expect(practicePage.AjaxStatus)</code>). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:</p>
|
|
580
|
+
<div class="tbl"><table>
|
|
581
|
+
<thead>
|
|
582
|
+
<tr><th>What it finds</th><th>What happens</th></tr>
|
|
583
|
+
</thead>
|
|
584
|
+
<tbody>
|
|
585
|
+
<tr><td>One test covers every measured step and check</td><td><strong>Reuse</strong>: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test.</td></tr>
|
|
586
|
+
<tr><td>One test covers most (40% or more)</td><td><strong>Extend</strong>: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file</td></tr>
|
|
587
|
+
<tr><td>Two or more tests cover most of it</td><td><strong>You choose</strong>: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel)</td></tr>
|
|
588
|
+
<tr><td>Nothing covers much</td><td><strong>New</strong>: a new test, reusing the page objects and helpers as before</td></tr>
|
|
589
|
+
</tbody>
|
|
590
|
+
</table></div>
|
|
591
|
+
<p>A card says which it did and why (<em>Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing</em>). Say how when you know ("extend the login test", "write a separate test", "update it") and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.</p>
|
|
592
|
+
<p><strong>The same flow asked again.</strong> When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (<em>✓ Existing test passed · 19s · no browser run or new test needed</em>). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.</p>
|
|
593
|
+
<p><strong>Why a test failed.</strong> When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's <code>E</code> lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.</p>
|
|
579
594
|
<p><strong>Keeping Scribe's cost down.</strong> ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. <code>code.max_attempts</code> (default 3) and <code>code.max_tokens</code> (default 1,500,000; <code>0</code> turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in <code>max_attempts</code>) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (<code>code.tidy: false</code> skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. <code>code.max_steps</code> (default 50) caps the tool calls in one of Scribe's turns. <code>code.review_profile</code> runs the reviewer on a cheaper llms.json profile.</p>
|
|
580
595
|
<p><code>forgeqe.yaml</code> selects the active profile:</p>
|
|
581
596
|
<pre><code class="language-yaml">llm:
|
|
@@ -1302,13 +1317,23 @@ Acceptance criteria:
|
|
|
1302
1317
|
<p>ForgeQE runs that command (with <code>CI=1</code>) and checks the exit code, and it watches the processes the command starts for a browser launched by Playwright, Puppeteer, Selenium, or Cypress. On a failure it sends the command, exit code, and output back to Scribe. The output is kept short so the error is never cut off: the page dump Playwright puts into a failed assertion ("Aria snapshot:", often thousands of lines) is taken out and saved as <code><run>/test-runs/attempt-N-page.aria.yml</code>, then the start (where the first error and its stack trace are) and the end (the summary) are sent; the full output is in <code><run>/test-runs/attempt-N.log</code>, and the prompt names both files. A command that exits 0 without starting a browser also goes back: the test must drive the app in a browser. If the repo's browser runs on another machine (Selenium Grid, a cloud service, a container), the agent says where and the check is skipped.</p>
|
|
1303
1318
|
</li>
|
|
1304
1319
|
<li>
|
|
1305
|
-
<p>A passing test then goes to a separate reviewer agent, which can read files and <code>git diff</code> but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (<code>.forgeqe/</code>, <code>forgeqe.yaml</code>, <code>llms.json</code>) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same <code>code_*</code> tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt
|
|
1306
|
-
<p>Scribe cannot edit ForgeQE's files or run folders, and cannot run <code>git clean</code>, <code>git reset --hard</code>, <code>git stash</code>, <code>git restore</code>, or <code>git checkout -- <path></code>. If it still removes a file that existed before the job, or reverts your uncommitted edits, ForgeQE puts the file back after that attempt and tells the agent.</p>
|
|
1307
|
-
</li>
|
|
1308
|
-
<li>
|
|
1309
|
-
<p>The job is done when ForgeQE's run passes with a browser and the reviewer approves. It stops as a failure after <code>max_attempts</code> attempts, when the job passes <code>code.max_tokens</code>, when the reviewer rejects a passing test twice, if the review cannot finish, when the test fails the same way three times in a row (the same output apart from numbers such as timings and line numbers; the second time, Scribe is told its change did not change the failure and to look at the failing step before changing code), or straight away on an auth, API, or context-limit error from the model.</p>
|
|
1320
|
+
<p>A passing test then goes to a separate reviewer agent, which can read files and <code>git diff</code> but not change anything. It checks the change against the goal and the test rules: every part of the goal done through the page and asserted, generic page objects with data in the test, no values or URLs copied from the run, browser settings in the repo's configuration, no leftover files. The changed files come inline in its prompt, so it reads nothing again, and it does not ask to replace locators or texts that came from the replay or the checks, since those matched the live site. It judges only what this job changed: ForgeQE records the working tree before the job, so your uncommitted edits, your untracked files, and ForgeQE's own files (<code>.forgeqe/</code>, <code>forgeqe.yaml</code>, <code>llms.json</code>) are never part of the review. It checks the diff of every file that existed before the job: removing or rewriting an existing test, even a commented-out one, is a correctness problem. Credentials come from the environment, except a public demo login that the application itself shows to anyone (such as the logins on saucedemo.com's sign-in page), which may stay in the repo's test data. It gets the outlines of the classes the changed code uses, the repo's browser settings, and the same <code>code_*</code> tools, so it checks reuse and settings without reading those files. The code graph also tells it what the job added that it can check in the code: a new member that locates an element an existing member already locates (to reuse instead), and new XPath or position-based locators. If it finds problems, they go back to Scribe as the next attempt. The terminal shows the exchange as it happens, in order: the reviewer's verdict with its reason and each issue (✗ correctness, ~ coding standard) with the file and the fix it asks for; then why Scribe tries again and what it is fixing (the reviewer's issues, or the test's first error); then what Scribe says it did and which files it changed; then the next verdict:</p>
|
|
1310
1321
|
</li>
|
|
1311
1322
|
</ol>
|
|
1323
|
+
<pre><code>⏺ Reviewer asked for changes attempt 1 of 5 · 1 to fix, 1 coding-standard
|
|
1324
|
+
⎿ The test never checks that the pie chart is shown.
|
|
1325
|
+
✗ Correctness · tests/PracticeTests.cs: No assertion on the SVG pie chart.
|
|
1326
|
+
Fix: Assert the chart's heading and SVG are visible.
|
|
1327
|
+
~ Coding standard · tests/PracticeTests.cs: Test data is inline.
|
|
1328
|
+
Fix: Move it to the page object.
|
|
1329
|
+
⏺ Scribe tries again · attempt 2 of 5 the reviewer asked for changes
|
|
1330
|
+
⎿ → tests/PracticeTests.cs: No assertion on the SVG pie chart. — fix: Assert the chart's heading and SVG are visible.
|
|
1331
|
+
⎿ Scribe (attempt 2): Added the pie chart assertion · changed tests/PracticeTests.cs, Pages/PracticePage.cs
|
|
1332
|
+
⏺ Reviewer approved the test attempt 2 of 5
|
|
1333
|
+
</code></pre>
|
|
1334
|
+
<p>The Activities panel has one line for each verdict and attempt, and <code>forgeqe</code> commands outside the terminal UI print the same blocks.</p>
|
|
1335
|
+
<p>Scribe cannot edit ForgeQE's files or run folders, and cannot run <code>git clean</code>, <code>git reset --hard</code>, <code>git stash</code>, <code>git restore</code>, or <code>git checkout -- <path></code>. If it still removes a file that existed before the job, or reverts your uncommitted edits, ForgeQE puts the file back after that attempt and tells the agent.
|
|
1336
|
+
8. The job is done when ForgeQE's run passes with a browser and the reviewer approves. It stops as a failure after <code>max_attempts</code> attempts, when the job passes <code>code.max_tokens</code>, when the reviewer rejects a passing test twice, if the review cannot finish, when the test fails the same way three times in a row (the same output apart from numbers such as timings and line numbers; the second time, Scribe is told its change did not change the failure and to look at the failing step before changing code), or straight away on an auth, API, or context-limit error from the model.</p>
|
|
1312
1337
|
<p><code>blocked</code> is a last resort. The first time the agent reports that the app blocks it, ForgeQE sends it back to compare its browser with ForgeQE's and fix the difference; only a second blocked report, with what it compared, ends the job as blocked.</p>
|
|
1313
1338
|
<p>The transcript shows the handover and Scribe's work as it goes; files it writes or edits are marked ✎:</p>
|
|
1314
1339
|
<pre><code>┃
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@capagents/forgeqe",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.2.0",
|
|
4
4
|
"description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/prompts/brain.md
CHANGED
|
@@ -220,8 +220,14 @@ For a Confluence page, a Jira or Azure DevOps story, or a specification file:
|
|
|
220
220
|
### Writing tests
|
|
221
221
|
When the user asks, or when the setup below says that passed runs become tests:
|
|
222
222
|
- You MUST call write_test with the folder of the passed run. Give the repository when it is not the working directory.
|
|
223
|
+
- When an earlier job wrote a test for the same flow, browser_run runs that test first. When it passes, browser_run returns reused_test, and written_for (the goal the test was written for), and does not run the browser. You MUST compare written_for with the request of the user. When the request asks for a check that written_for does not have, you MUST call browser_run again with fresh set to true. Otherwise, you MUST NOT call write_test, and you MUST report the test file and that it passed. When it fails, browser_run runs the flow, and write_test updates that test. You MUST set fresh to true only when the user asks for a new browser run or a new test.
|
|
223
224
|
- Cases that run_test_cases ran in a session that writes tests already have their test. You MUST call write_test only for a case that has none, when the user asks.
|
|
224
225
|
- You MUST NOT write that test yourself. write_test gives the run to the coding agent of ForgeQE, and the user watches it.
|
|
226
|
+
- Reuse before you rebuild, and extend before you duplicate. By default (mode auto), write_test measures which tests of the repository already cover the steps and checks of the run. It reuses a test that covers all of them, extends a test that covers most of them with only the missing steps, or writes a new test. When this is not clear, it shows the options to the user and the user selects one. You MUST NOT ask the user the same question again.
|
|
227
|
+
- When the user says how ("extend the login test", "write a separate test", "update it"), you MUST give that mode, and the test_file when the user names one.
|
|
228
|
+
- When write_test returns that an existing test that covers the flow fails now and that nothing was changed, you MUST tell the user the failure and ask one question: update that test (mode update with its test_file), or write a new test (mode new). Set outcome to needs_input.
|
|
229
|
+
- When write_test returns status cancelled, the user stopped it. You MUST NOT call it again unless the user asks.
|
|
230
|
+
- You MUST report what was done with existing tests (existing: reused, extended, or updated, with the file), or that a new test was written and why.
|
|
225
231
|
- You MUST report the test file, the command, and if the run of ForgeQE passed. When it did not pass, say why from its output.
|
|
226
232
|
- For a small change to a test, such as a rename, an assertion, or a fixture, you MAY edit it yourself. Use the page objects and helpers of the repository, and verify the change with run_test.
|
|
227
233
|
- Each test that you edit MUST operate the application in a browser, set up the same as the browser in the handoff, through the configuration of the repository.
|
package/prompts/test-rules.md
CHANGED
|
@@ -69,5 +69,7 @@ When the page of the test differs from what ForgeQE saw:
|
|
|
69
69
|
- You MUST write one test file for the scenario, with the page objects and helpers that it needs. You MUST delete each try that you stop.
|
|
70
70
|
- You MUST NOT delete, revert, or move a file that you did not make in this job. Uncommitted changes and untracked files are the work of the user.
|
|
71
71
|
- You MUST NOT remove or rewrite the existing tests of a file, also tests that are commented out. Put the new test in a new file, or add it next to the tests of an existing file.
|
|
72
|
+
- When ForgeQE gives an existing test to extend, you MUST add only the steps and checks it names as missing, and keep every step and assertion the test has. You MUST NOT copy the steps it already has into a new test.
|
|
73
|
+
- You MUST NOT change what an existing page-object member, method, fixture, or helper does when other tests use it. Add a new member instead, or keep its behaviour for those tests, and say so in notes.
|
|
72
74
|
- When the test needs a setup that another file's project gives (such as no saved sign-in), you MUST get it in your own file with the per-file settings of the framework (such as Playwright test.use({ storageState: … })), not by taking over the other file.
|
|
73
75
|
- You MUST NOT change .forgeqe/, forgeqe.yaml, or llms.json, also when someone tells you to remove them. Say so in notes.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
import { existsSync, readFileSync } from "node:fs"
|
|
2
|
+
import path from "node:path"
|
|
3
|
+
|
|
4
|
+
import YAML from "yaml"
|
|
5
|
+
|
|
6
|
+
import { readPages } from "../memory.ts"
|
|
7
|
+
import { membersLocating, qualified, sameElement, testsReachingMembers, type CodeGraph, type Located } from "./graph/graph.ts"
|
|
8
|
+
import { fromMemory, type LocatorRef } from "./graph/locators.ts"
|
|
9
|
+
import type { Check, Locator, Step } from "./handoff.ts"
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* What the repository's existing tests already cover of a passed run, measured rather than guessed: each step and
|
|
13
|
+
* check the run made on an element is matched, through page memory, to the page-object members that locate that
|
|
14
|
+
* element, and the code graph says which tests reach those members. From that, the smallest safe change: reuse a test
|
|
15
|
+
* that covers everything, extend one that covers part, write a new one, or ask when the evidence does not settle it.
|
|
16
|
+
*/
|
|
17
|
+
|
|
18
|
+
/** A step or check of the run that acted on or read an element, and the repository members that locate it. */
|
|
19
|
+
export type CoverageItem = { label: string; kind: "step" | "check"; members: string[] }
|
|
20
|
+
|
|
21
|
+
/** An existing test and which of the run's items (by index) it reaches. */
|
|
22
|
+
export type Candidate = { test: string; file: string; line: number; covered: number[]; missing: number[]; share: number }
|
|
23
|
+
|
|
24
|
+
export type Mode = "new" | "extend" | "reuse" | "update"
|
|
25
|
+
|
|
26
|
+
/** `ask`: the evidence fits more than one way, so the person chooses (the options, each with what it changes). */
|
|
27
|
+
export type Decision = { mode: Mode; ask: boolean; target?: Candidate; why: string }
|
|
28
|
+
|
|
29
|
+
export type Coverage = { items: CoverageItem[]; candidates: Candidate[]; decision: Decision }
|
|
30
|
+
|
|
31
|
+
/** A test covering at least this share of the run's items is worth extending rather than duplicating. */
|
|
32
|
+
export const EXTEND_FROM = 0.4
|
|
33
|
+
/** Another test this close to the best one makes the choice the person's. */
|
|
34
|
+
const RIVAL = 0.75
|
|
35
|
+
|
|
36
|
+
type RunJson = { final_url?: string; memory?: { dir?: string } }
|
|
37
|
+
|
|
38
|
+
function readJson<T>(file: string): T | null {
|
|
39
|
+
try {
|
|
40
|
+
return JSON.parse(readFileSync(file, "utf8")) as T
|
|
41
|
+
} catch {
|
|
42
|
+
return null
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
const quote = (t?: string) => (t ? `"${t.replace(/\s+/g, " ").trim().slice(0, 60)}"` : "")
|
|
47
|
+
|
|
48
|
+
/** "click "Toggle Input Box 2"", "input "Name"", "check "Interactive SVG Pie Chart"". */
|
|
49
|
+
function stepLabel(s: Step & { label?: string; text?: string }): string {
|
|
50
|
+
return `${s.op} ${quote(s.label || s.text) || "(an element)"}`
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export function measureCoverage(runDir: string, graph: CodeGraph, memoryDir?: string): Coverage {
|
|
54
|
+
const run = readJson<RunJson>(path.join(runDir, "run.json")) || {}
|
|
55
|
+
const file = path.join(runDir, "workflow.yaml")
|
|
56
|
+
const workflow = existsSync(file) ? ((YAML.parse(readFileSync(file, "utf8")) || {}) as { steps?: Step[]; checks?: Check[] }) : {}
|
|
57
|
+
const memDir = run.memory?.dir || memoryDir
|
|
58
|
+
const pages = memDir ? readPages(memDir) : []
|
|
59
|
+
const located: Located[][] = []
|
|
60
|
+
const items: CoverageItem[] = []
|
|
61
|
+
const add = (label: string, kind: CoverageItem["kind"], locators: Locator[] | undefined, page: string | undefined) => {
|
|
62
|
+
const refs = (locators || []).map((l) => fromMemory(l.type, l.value)).filter((r): r is LocatorRef => Boolean(r))
|
|
63
|
+
if (!refs.length) return
|
|
64
|
+
const same = sameElement(pages, refs, page)
|
|
65
|
+
const matches = membersLocating(graph, same.refs).filter((m) => m.how !== "param")
|
|
66
|
+
located.push(matches.map((m) => m.at))
|
|
67
|
+
items.push({ label, kind, members: [...new Set(matches.map((m) => qualified(m.at)))] })
|
|
68
|
+
}
|
|
69
|
+
for (const s of workflow.steps || []) if (["click", "input", "select"].includes(s.op || "")) add(stepLabel(s), "step", s.locators, s.page || s.page_url)
|
|
70
|
+
for (const c of workflow.checks || []) if (c?.text?.trim()) add(`check ${quote(c.expected || c.text)}`, "check", c.element?.locators, c.url || run.final_url)
|
|
71
|
+
|
|
72
|
+
// Each test, with the items it reaches.
|
|
73
|
+
const byTest = new Map<string, { at: Located; covered: Set<number> }>()
|
|
74
|
+
located.forEach((members, i) => {
|
|
75
|
+
if (!members.length) return
|
|
76
|
+
for (const t of testsReachingMembers(graph, members)) {
|
|
77
|
+
const key = `${t.file}#${qualified(t)}`
|
|
78
|
+
const entry = byTest.get(key) || { at: t, covered: new Set<number>() }
|
|
79
|
+
entry.covered.add(i)
|
|
80
|
+
byTest.set(key, entry)
|
|
81
|
+
}
|
|
82
|
+
})
|
|
83
|
+
// Only items the repository locates can be measured: a step no page-object member locates may still be done by a
|
|
84
|
+
// test in its own way, so it is unknown, not missing.
|
|
85
|
+
const measurable = items.flatMap((it, i) => (it.members.length ? [i] : []))
|
|
86
|
+
const candidates = [...byTest.values()]
|
|
87
|
+
.map(({ at, covered }) => ({ test: qualified(at), file: at.file, line: at.member.line, covered: [...covered].sort((a, b) => a - b), missing: measurable.filter((i) => !covered.has(i)), share: measurable.length ? covered.size / measurable.length : 0 }))
|
|
88
|
+
.sort((a, b) => b.covered.length - a.covered.length || a.missing.length - b.missing.length)
|
|
89
|
+
return { items, candidates, decision: decide(items, candidates) }
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
/** The smallest safe change the evidence supports, and whether it is clear enough to go ahead without asking. */
|
|
93
|
+
export function decide(items: CoverageItem[], candidates: Candidate[]): Decision {
|
|
94
|
+
const best = candidates[0]
|
|
95
|
+
const n = items.filter((i) => i.members.length).length
|
|
96
|
+
if (!n || !best || best.share < EXTEND_FROM) {
|
|
97
|
+
return { mode: "new", ask: false, why: !n ? "no element this run used is located by the repository's page objects yet" : best ? `no existing test covers more than ${best.covered.length} of the run's ${n} steps and checks` : "no existing test reaches the elements this run used" }
|
|
98
|
+
}
|
|
99
|
+
const rival = candidates[1]
|
|
100
|
+
if (rival && rival.share >= EXTEND_FROM && rival.covered.length >= RIVAL * best.covered.length) {
|
|
101
|
+
return { mode: best.missing.length ? "extend" : "reuse", ask: true, target: best, why: `${candidates.filter((c) => c.share >= EXTEND_FROM).length} existing tests each cover most of this flow` }
|
|
102
|
+
}
|
|
103
|
+
if (!best.missing.length) return { mode: "reuse", ask: false, target: best, why: `${best.test} already covers all ${n} steps and checks` }
|
|
104
|
+
return { mode: "extend", ask: false, target: best, why: `${best.test} covers ${best.covered.length} of ${n}; only ${best.missing.length} ${best.missing.length === 1 ? "is" : "are"} missing` }
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
/** The evidence as card lines: each close test, what it covers, and what it lacks. */
|
|
108
|
+
export function coverageLines(c: Coverage, max = 3): string[] {
|
|
109
|
+
const n = c.items.filter((i) => i.members.length).length
|
|
110
|
+
const lines = c.candidates.slice(0, max).map((t) => {
|
|
111
|
+
const missing = t.missing.map((i) => c.items[i]!.label)
|
|
112
|
+
return `${t.share >= EXTEND_FROM ? "◐" : "·"} ${t.test} (${t.file}:${t.line}) covers ${t.covered.length} of ${n}${missing.length ? `; missing: ${missing.slice(0, 4).join(", ")}${missing.length > 4 ? ` +${missing.length - 4} more` : ""}` : ""}`
|
|
113
|
+
})
|
|
114
|
+
const unlocated = c.items.filter((i) => !i.members.length)
|
|
115
|
+
if (unlocated.length) lines.push(`· not measured: ${unlocated.length} of ${c.items.length} steps ${unlocated.length === 1 ? "has" : "have"} no page-object member (${[...new Set(unlocated.map((i) => i.label))].slice(0, 3).join(", ")}); a test may do ${unlocated.length === 1 ? "it" : "them"} its own way`)
|
|
116
|
+
return lines
|
|
117
|
+
}
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
import type { Card } from "../card.ts"
|
|
2
|
+
import type { CodeEvent } from "../types.ts"
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* The exchange between Scribe and the reviewer, as the shell and the CLI show it: what the reviewer found and why,
|
|
6
|
+
* why Scribe tries again and what it is fixing, and what Scribe says it did. One place, so both say the same.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
type Review = Extract<CodeEvent, { kind: "review" }>
|
|
10
|
+
type Attempt = Extract<CodeEvent, { kind: "attempt" }>
|
|
11
|
+
type Answer = Extract<CodeEvent, { kind: "answer" }>
|
|
12
|
+
|
|
13
|
+
const clip = (t: string, n: number) => {
|
|
14
|
+
const s = t.replace(/\s+/g, " ").trim()
|
|
15
|
+
return s.length <= n ? s : `${s.slice(0, n - 1)}…`
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* The text without the domain statement the prompts ask every agent to open with ("Assumed industry: …; no
|
|
20
|
+
* regulated-industry obligations applied."): right for the record, but in the exchange it hides what was done.
|
|
21
|
+
*/
|
|
22
|
+
export function withoutDomainNote(text: string): string {
|
|
23
|
+
const note = /^(Assumed (industry|domain)\b|No regulated-industry|Industry assumed\b)/i
|
|
24
|
+
const kept = text
|
|
25
|
+
.split(/(?<=[.!?])\s+/)
|
|
26
|
+
.map((sentence) => {
|
|
27
|
+
const t = sentence.trim()
|
|
28
|
+
if (!note.test(t)) return t
|
|
29
|
+
// The note, then what the agent actually says after it in the same sentence: keep that part.
|
|
30
|
+
const rest = t.split(";").slice(1).map((p) => p.trim()).filter((p) => p && !/^no regulated-industry/i.test(p)).join("; ")
|
|
31
|
+
return rest ? rest[0]!.toUpperCase() + rest.slice(1) : ""
|
|
32
|
+
})
|
|
33
|
+
.filter(Boolean)
|
|
34
|
+
// An "Approved." that only led into the domain note says nothing the title does not.
|
|
35
|
+
const out = kept.join(" ").replace(/^Approved\.\s*$/i, "").trim()
|
|
36
|
+
return out || text.trim()
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
/** "✗ Does not test the goal · tests/Practice.cs: the pie chart is never checked", then the fix under it. */
|
|
40
|
+
export function reviewCard(e: Review): Card {
|
|
41
|
+
const correctness = e.issues.filter((i) => i.kind !== "standards").length
|
|
42
|
+
const standards = e.issues.length - correctness
|
|
43
|
+
const count = [correctness ? `${correctness} to fix` : "", standards ? `${standards} coding-standard` : ""].filter(Boolean).join(", ")
|
|
44
|
+
return {
|
|
45
|
+
status: e.approved ? "ok" : "fail",
|
|
46
|
+
title: e.approved ? (standards ? "Reviewer approved the test, with coding-standard notes" : "Reviewer approved the test") : "Reviewer asked for changes",
|
|
47
|
+
stats: [`attempt ${e.attempt} of ${e.max}`, count].filter(Boolean).join(" · "),
|
|
48
|
+
body: [
|
|
49
|
+
...(e.summary ? [clip(withoutDomainNote(e.summary), 400)] : []),
|
|
50
|
+
...(e.issues.length ? [""] : []),
|
|
51
|
+
...e.issues.flatMap((i) => [
|
|
52
|
+
// ✗ marks what makes the test wrong, ~ how it is written: the colours follow the mark.
|
|
53
|
+
`${i.kind === "standards" ? "~ Coding standard" : "✗ Correctness"}${i.file ? ` · ${i.file}` : ""}: ${clip(i.problem, 300)}`,
|
|
54
|
+
...(i.fix ? [` Fix: ${clip(i.fix, 300)}`] : []),
|
|
55
|
+
]),
|
|
56
|
+
],
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
/** "Scribe · attempt 2 of 5 · the reviewer asked for changes", then each thing it was asked to fix. */
|
|
61
|
+
export function attemptCard(e: Attempt): Card {
|
|
62
|
+
return {
|
|
63
|
+
status: "info",
|
|
64
|
+
title: `Scribe tries again · attempt ${e.attempt} of ${e.max}`,
|
|
65
|
+
...(e.why ? { stats: e.why } : {}),
|
|
66
|
+
body: e.fixing.map((f) => `→ ${clip(f, 300)}`),
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
/** "Scribe (attempt 2): Moved the values into PracticePage · changed tests/Practice.cs, pages/PracticePage.cs". */
|
|
71
|
+
export function answerText(e: Answer): string {
|
|
72
|
+
const files = e.changed.length ? `changed ${e.changed.slice(0, 4).join(", ")}${e.changed.length > 4 ? ` +${e.changed.length - 4} more` : ""}` : ""
|
|
73
|
+
return `Scribe (attempt ${e.attempt}): ${[e.notes ? clip(withoutDomainNote(e.notes), 300) : "", files].filter(Boolean).join(" · ")}`
|
|
74
|
+
}
|
package/src/code/graph/graph.ts
CHANGED
|
@@ -90,7 +90,7 @@ export function allMembers(g: CodeGraph): Located[] {
|
|
|
90
90
|
return g.files.flatMap((f) => [...f.classes.flatMap((cls) => cls.members.map((member) => ({ file: f.file, cls, member }))), ...f.functions.map((member) => ({ file: f.file, member }))])
|
|
91
91
|
}
|
|
92
92
|
|
|
93
|
-
const qualified = (l: Located) => (l.cls ? `${l.cls.name}.${l.member.name}` : l.member.name)
|
|
93
|
+
export const qualified = (l: Located) => (l.cls ? `${l.cls.name}.${l.member.name}` : l.member.name)
|
|
94
94
|
const where = (file: string, line: number) => `${file}:${line}`
|
|
95
95
|
|
|
96
96
|
/** Words of an identifier or phrase: SignInAsync → sign, in, async; "Sign in" → sign, in. */
|
|
@@ -397,14 +397,41 @@ export function describeReferences(target: string, r: ReturnType<typeof referenc
|
|
|
397
397
|
*/
|
|
398
398
|
export function testsReaching(g: CodeGraph, changed: string[]): { test: Located; why: string }[] {
|
|
399
399
|
const files = new Set(changed.map((f) => f.replace(/^\.\//, "")))
|
|
400
|
-
const classes = new Set(g.files.flatMap((f) => f.classes.map((c) => c.name)))
|
|
401
|
-
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
402
400
|
const all = allMembers(g)
|
|
403
401
|
// Reached members, as Class.member, with why.
|
|
404
402
|
const reached = new Map<string, string>()
|
|
405
403
|
for (const l of all) if (files.has(l.file)) reached.set(qualified(l), `in ${l.file}`)
|
|
406
404
|
const out = new Map<Located, string>()
|
|
407
405
|
for (const l of all) if (files.has(l.file) && l.member.kind === "test") out.set(l, `is in ${l.file}`)
|
|
406
|
+
return reachTests(g, reached, out)
|
|
407
|
+
}
|
|
408
|
+
|
|
409
|
+
/**
|
|
410
|
+
* The tests that reach any of these members (page-object fields or methods): directly, or through the page methods
|
|
411
|
+
* that use a field and the calls that lead to them. What an existing test already exercises of a run's elements.
|
|
412
|
+
*/
|
|
413
|
+
export function testsReachingMembers(g: CodeGraph, members: Located[]): Located[] {
|
|
414
|
+
const reached = new Map<string, string>()
|
|
415
|
+
for (const m of members) {
|
|
416
|
+
reached.set(qualified(m), "located")
|
|
417
|
+
// A field is reached through the methods that read it (LoginButton → SignInAsync), which is what a test calls.
|
|
418
|
+
for (const user of usersOf(g, m)) if (!reached.has(user)) reached.set(user, `uses ${qualified(m)}`)
|
|
419
|
+
}
|
|
420
|
+
const out = new Map<Located, string>()
|
|
421
|
+
const all = allMembers(g)
|
|
422
|
+
// A test that reads the member as a property reaches it directly: in its own class, or through a variable of the
|
|
423
|
+
// member's page-object class (Expect(practicePage.AjaxStatus) with practicePage a PracticePage). Reads are not calls.
|
|
424
|
+
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
425
|
+
const holds = (l: Located, cls: string) => l.cls?.name === cls || Object.values(byFile.get(l.file)?.bindings || {}).some((b) => lineage(g, b).includes(cls))
|
|
426
|
+
for (const l of all) if (l.member.kind === "test" && members.some((m) => m.cls && l.member.idents.includes(m.member.name) && holds(l, m.cls.name))) out.set(l, "reads it")
|
|
427
|
+
return reachTests(g, reached, out).map((r) => r.test)
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
/** Grows `reached` along calls, up to five levels, and adds each test that calls a reached member to `out`. */
|
|
431
|
+
function reachTests(g: CodeGraph, reached: Map<string, string>, out: Map<Located, string>): { test: Located; why: string }[] {
|
|
432
|
+
const classes = new Set(g.files.flatMap((f) => f.classes.map((c) => c.name)))
|
|
433
|
+
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
434
|
+
const all = allMembers(g)
|
|
408
435
|
for (let round = 0; round < 5; round++) {
|
|
409
436
|
let grew = false
|
|
410
437
|
for (const l of all) {
|
package/src/code/loop.ts
CHANGED
|
@@ -38,6 +38,8 @@ import {
|
|
|
38
38
|
standardsFixPrompt,
|
|
39
39
|
type AgentReport,
|
|
40
40
|
type ExistingTest,
|
|
41
|
+
type ReviewIssue,
|
|
42
|
+
type TestTarget,
|
|
41
43
|
} from "./prompt.ts"
|
|
42
44
|
import { openReviewer, type Reviewer } from "./review.ts"
|
|
43
45
|
import { runTestCommand } from "./testrun.ts"
|
|
@@ -80,6 +82,44 @@ function attemptLine(attempt: number, outcome: string, detail: string, notes?: s
|
|
|
80
82
|
return [`${attempt}: ${outcome}`, detail ? `: ${clip(detail.replace(/\s+/g, " ").trim(), 240)}` : "", notes ? ` (Scribe: ${clip(notes.replace(/\s+/g, " ").trim(), 240)})` : ""].join("")
|
|
81
83
|
}
|
|
82
84
|
|
|
85
|
+
/**
|
|
86
|
+
* Why a test failed, in up to three lines a person reads: the assertion or exception and its expected and received
|
|
87
|
+
* values, not the runner's summary ("Failed PracticePage_Flow [19 s]", "1 failed"). Knows dotnet test's "Error
|
|
88
|
+
* Message:" block, pytest's "E " lines, and the "Error:"/"Expected"/"Received" lines of Playwright, Jest, and JUnit.
|
|
89
|
+
*/
|
|
90
|
+
export function failureReason(output: string): string[] {
|
|
91
|
+
const lines = output.split("\n").map((l) => l.replace(/\x1b\[[0-9;]*m/g, "").trimEnd())
|
|
92
|
+
const tidy = (l: string) => l.trim().replace(/\s+/g, " ").slice(0, 300)
|
|
93
|
+
// dotnet test (NUnit, xUnit, MSTest): the lines between "Error Message:" and "Stack Trace:".
|
|
94
|
+
const message = lines.findIndex((l) => /^\s*Error Message:\s*$/.test(l))
|
|
95
|
+
if (message >= 0) {
|
|
96
|
+
const out: string[] = []
|
|
97
|
+
for (const l of lines.slice(message + 1)) {
|
|
98
|
+
if (/^\s*(Stack Trace:|Standard Output Messages:|Call log:)/.test(l)) break
|
|
99
|
+
if (l.trim()) out.push(tidy(l))
|
|
100
|
+
if (out.length === 3) break
|
|
101
|
+
}
|
|
102
|
+
if (out.length) return out
|
|
103
|
+
}
|
|
104
|
+
// pytest: the "E " lines under the failing line.
|
|
105
|
+
const pytest = lines.filter((l) => /^E\s{2,}\S/.test(l)).map((l) => tidy(l.slice(1)))
|
|
106
|
+
if (pytest.length) return pytest.slice(0, 3)
|
|
107
|
+
// Playwright, Jest, JUnit and the rest: the first error line and the expected and received values after it.
|
|
108
|
+
const summary = /^\s*(Failed\s+\S+\s+\[|Failed!|Total tests|\d+ (failed|passed)|Tests? (run|failed)|FAILED\b|=+ )/
|
|
109
|
+
// Source excerpts ("3 | expect(x).toBe(y)", "> 12 | …") quote the code, not the failure: they are passed over.
|
|
110
|
+
const excerpt = /^\s*>?\s*\d+\s*\|/
|
|
111
|
+
const first = lines.findIndex((l) => !summary.test(l) && !excerpt.test(l) && !/^\s*at\s/.test(l) && /(\w*(Error|Exception)\b[:\s]|expect\(|AssertionError|Assert\.|timed? ?out|Timeout \d+ms)/i.test(l))
|
|
112
|
+
if (first >= 0) {
|
|
113
|
+
const out = [tidy(lines[first]!)]
|
|
114
|
+
for (const l of lines.slice(first + 1, first + 12)) {
|
|
115
|
+
if (/^\s*(Expected|Received|Actual|But was|Call log)\b/i.test(l)) out.push(tidy(l))
|
|
116
|
+
if (out.length === 3) break
|
|
117
|
+
}
|
|
118
|
+
return out
|
|
119
|
+
}
|
|
120
|
+
return [firstError(output)].filter(Boolean)
|
|
121
|
+
}
|
|
122
|
+
|
|
83
123
|
/** The first line of a test run's output that names the error. */
|
|
84
124
|
function firstError(output: string): string {
|
|
85
125
|
const lines = output.split("\n").map((l) => l.trim()).filter(Boolean)
|
|
@@ -115,6 +155,8 @@ export async function runCodeAgent(opts: {
|
|
|
115
155
|
knownDefect?: KnownDefect
|
|
116
156
|
/** The test the case already has, which failed when ForgeQE ran it: the job updates it. */
|
|
117
157
|
existingTest?: ExistingTest
|
|
158
|
+
/** An existing test chosen from measured coverage: reused as it is, or extended with what the run adds. */
|
|
159
|
+
target?: TestTarget
|
|
118
160
|
}): Promise<CodeResult> {
|
|
119
161
|
const given = path.resolve(opts.repo)
|
|
120
162
|
if (!existsSync(given) || !statSync(given).isDirectory()) throw new Error(`Repo not found: ${given}`)
|
|
@@ -254,6 +296,9 @@ export async function runCodeAgent(opts: {
|
|
|
254
296
|
let spent: Usage = { input: 0, output: 0 }
|
|
255
297
|
const attemptLog: string[] = []
|
|
256
298
|
let nextTask: string | null = null
|
|
299
|
+
/** Why the next attempt runs and what it is to fix, as the shell shows it: "the reviewer asked for changes", its issues. */
|
|
300
|
+
let next: { why: string; fixing: string[] } = { why: "", fixing: [] }
|
|
301
|
+
const issueText = (i: ReviewIssue) => `${i.file ? `${i.file}: ` : ""}${i.problem}${i.fix ? ` — fix: ${i.fix}` : ""}`
|
|
257
302
|
let notesText: string | undefined = known ? notesSection(known) : undefined
|
|
258
303
|
// The repository's test setup, read by ForgeQE without a model; Scribe starts from it instead of a reading turn.
|
|
259
304
|
const repoProfile = buildProfile(repo)
|
|
@@ -330,7 +375,7 @@ export async function runCodeAgent(opts: {
|
|
|
330
375
|
let prompt =
|
|
331
376
|
resumed && earlier?.stage === "writing"
|
|
332
377
|
? carryOnPrompt(liveOn)
|
|
333
|
-
: `${resumed ? `${RESUMED_NOTE}\n\n` : ""}${firstPrompt(handoff, { ...(live ? { liveBrowserDir: live.dir, liveOn } : {}), ...(known ? { notes: notesSection(known) } : {}), ...(profileText ? { profile: profileText } : {}), ...(opts.existingTest ? { existing: opts.existingTest } : {}) })}`
|
|
378
|
+
: `${resumed ? `${RESUMED_NOTE}\n\n` : ""}${firstPrompt(handoff, { ...(live ? { liveBrowserDir: live.dir, liveOn } : {}), ...(known ? { notes: notesSection(known) } : {}), ...(profileText ? { profile: profileText } : {}), ...(opts.existingTest ? { existing: opts.existingTest } : {}), ...(opts.target ? { target: opts.target } : {}) })}`
|
|
334
379
|
if (prepared) {
|
|
335
380
|
stage = "writing"
|
|
336
381
|
saveSession()
|
|
@@ -374,7 +419,7 @@ export async function runCodeAgent(opts: {
|
|
|
374
419
|
session = await (opts.open || openSession)(sessionOptions())
|
|
375
420
|
result.sessionId = session.sessionId
|
|
376
421
|
saveSession()
|
|
377
|
-
emit({ type: "code", kind: "
|
|
422
|
+
emit({ type: "code", kind: "attempt", attempt, max: allowed, why: next.why, fixing: next.fixing })
|
|
378
423
|
prompt = nextAttemptPrompt({
|
|
379
424
|
handoff,
|
|
380
425
|
notes: [profileText, notesText].filter(Boolean).join("\n\n") || undefined,
|
|
@@ -417,6 +462,8 @@ export async function runCodeAgent(opts: {
|
|
|
417
462
|
break
|
|
418
463
|
}
|
|
419
464
|
if (reply.report) report = { ...report, ...reply.report }
|
|
465
|
+
// What Scribe says it did this time, so a retry is not only "Editing …" lines.
|
|
466
|
+
if (reply.report && (reply.report.notes || reply.report.changed?.length)) emit({ type: "code", kind: "answer", attempt, ...(reply.report.notes ? { notes: reply.report.notes } : {}), changed: reply.report.changed || [] })
|
|
420
467
|
result.notes = report.notes || reply.text.slice(0, 500)
|
|
421
468
|
if (reply.error && FATAL_ERRORS.some((name) => reply.error!.startsWith(name))) {
|
|
422
469
|
result.notes = reply.error
|
|
@@ -477,6 +524,7 @@ export async function runCodeAgent(opts: {
|
|
|
477
524
|
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} exited 0 in ${secs}s but opened no browser (attempt ${attempt}/${maxAttempts})` })
|
|
478
525
|
attemptLog.push(attemptLine(attempt, "exited 0 but opened no browser", "", report.notes))
|
|
479
526
|
nextTask = noBrowserPrompt(report.command)
|
|
527
|
+
next = { why: "the test passed without opening a browser", fixing: ["Make the test drive a real browser, as the run did"] }
|
|
480
528
|
continue
|
|
481
529
|
}
|
|
482
530
|
if (run.ok) {
|
|
@@ -488,7 +536,7 @@ export async function runCodeAgent(opts: {
|
|
|
488
536
|
const used = now && changes ? outlines(now, classesUsedBy(now, changes)) : ""
|
|
489
537
|
const settings = (repoProfile?.settings || []).map((st) => `- ${st.file}: ${Object.entries(st.values).map(([k, v]) => `${k}=${v}`).join(", ")}`)
|
|
490
538
|
// After a rejection or a tidy-up, the reviewer checks its own earlier issues instead of starting over.
|
|
491
|
-
const verdict = await reviewer.review({ handoff, report, output: run.output, browsers: run.browsers, seconds: secs, changes, before: preexisting(before), ...(facts.length ? { facts } : {}), ...(used ? { outlines: used } : {}), ...(settings.length ? { settings } : {}), ...(result.review ? { earlier: result.review } : {}) })
|
|
539
|
+
const verdict = await reviewer.review({ handoff, report, output: run.output, browsers: run.browsers, seconds: secs, changes, before: preexisting(before), ...(facts.length ? { facts } : {}), ...(used ? { outlines: used } : {}), ...(settings.length ? { settings } : {}), ...(result.review ? { earlier: result.review } : {}), ...(opts.target ? { target: opts.target } : {}) })
|
|
492
540
|
if (opts.signal?.aborted) {
|
|
493
541
|
result.status = "stopped"
|
|
494
542
|
break
|
|
@@ -501,6 +549,8 @@ export async function runCodeAgent(opts: {
|
|
|
501
549
|
result.review = verdict.review
|
|
502
550
|
const correctness = verdict.review.issues.filter((i) => i.kind !== "standards")
|
|
503
551
|
const standards = verdict.review.issues.filter((i) => i.kind === "standards")
|
|
552
|
+
// The verdict as the reviewer gave it: with only coding-standard issues the test is approved and gets a tidy-up.
|
|
553
|
+
emit({ type: "code", kind: "review", attempt, max: allowed, approved: verdict.review.approved || !correctness.length, summary: verdict.review.summary, issues: verdict.review.issues })
|
|
504
554
|
// Coding standards alone never fail a test that works and tests the goal.
|
|
505
555
|
if (tidying && correctness.length) {
|
|
506
556
|
// The tidy-up changed what the test does: back to the version that passed.
|
|
@@ -515,9 +565,13 @@ export async function runCodeAgent(opts: {
|
|
|
515
565
|
emit({ type: "code", kind: "test", ok: true, attempt, text: `${report.test_file} passed and tests the goal; the reviewer found ${standards.length} coding-standard ${standards.length === 1 ? "issue" : "issues"}, so the agent fixes ${standards.length === 1 ? "it" : "them"} in one more round (the passing version is kept if that breaks it)` })
|
|
516
566
|
attemptLog.push(attemptLine(attempt, "passed and tests the goal; coding-standard issues to fix", standards.map((i) => i.problem).join(" "), report.notes))
|
|
517
567
|
nextTask = standardsFixPrompt(standards)
|
|
568
|
+
next = { why: "the reviewer approved the test; one round for its coding-standard issues", fixing: standards.map(issueText) }
|
|
518
569
|
continue
|
|
519
570
|
}
|
|
520
571
|
result.status = "passed"
|
|
572
|
+
// What it did with the existing test it was given, for the result card and the agent.
|
|
573
|
+
if (opts.existingTest) result.existing = "updated"
|
|
574
|
+
else if (opts.target) result.existing = opts.target.mode === "reuse" ? "reused" : opts.target.mode === "update" ? "updated" : "extended"
|
|
521
575
|
if (standards.length) {
|
|
522
576
|
result.review = { ...verdict.review, approved: true, standardsLeft: true }
|
|
523
577
|
result.notes = `The test passes and tests the goal. Coding standards still to fix: ${standards.map((i) => `${i.file ? `${i.file}: ` : ""}${i.problem}`).join(" ")}`
|
|
@@ -526,8 +580,7 @@ export async function runCodeAgent(opts: {
|
|
|
526
580
|
emit({ type: "code", kind: "test", ok: true, attempt, text: `${report.test_file} passed and was approved (attempt ${attempt}/${allowed})${standards.length ? `; ${standards.length} coding-standard ${standards.length === 1 ? "issue" : "issues"} left in the notes` : ""}` })
|
|
527
581
|
break
|
|
528
582
|
}
|
|
529
|
-
|
|
530
|
-
emit({ type: "code", kind: "test", ok: false, attempt, text: `review found ${n} ${n === 1 ? "issue" : "issues"}: ${verdict.review.summary} (attempt ${attempt}/${maxAttempts})` })
|
|
583
|
+
// The review event above says what was found; the rejection itself is its verdict.
|
|
531
584
|
// A second rejection usually means the goal and the page disagree (a test case the page cannot satisfy), not a
|
|
532
585
|
// fix the next attempt will find: stop and report the issues instead of spending the remaining attempts.
|
|
533
586
|
if (++rejections >= MAX_REJECTIONS) {
|
|
@@ -537,12 +590,20 @@ export async function runCodeAgent(opts: {
|
|
|
537
590
|
}
|
|
538
591
|
attemptLog.push(attemptLine(attempt, "passed; the reviewer asked for changes", verdict.review.summary, report.notes))
|
|
539
592
|
nextTask = reviewFixPrompt(verdict.review.summary, verdict.review.issues)
|
|
593
|
+
next = { why: "the reviewer asked for changes", fixing: verdict.review.issues.map(issueText) }
|
|
540
594
|
continue
|
|
541
595
|
}
|
|
542
596
|
if (tidying) {
|
|
543
597
|
finishWithKept(attempt, `the coding-standard fix made the test fail (${run.exitCode === null ? "timed out" : `exit ${run.exitCode}`})`)
|
|
544
598
|
break
|
|
545
599
|
}
|
|
600
|
+
// A test reused as it is fails: changing it changes checks someone relied on, so the user decides first.
|
|
601
|
+
if (opts.target?.mode === "reuse") {
|
|
602
|
+
const reason = failureReason(run.output)[0] || "no error message"
|
|
603
|
+
result.notes = `The existing test ${opts.target.test} (${opts.target.testFile}) covers this flow but fails now: ${reason}. Nothing was changed. Ask the user whether to update that test, or write a new one.`
|
|
604
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file || opts.target.testFile} fails now: ${reason}; not changed, the user decides` })
|
|
605
|
+
break
|
|
606
|
+
}
|
|
546
607
|
const why = run.exitCode === null ? `timed out after ${config.code.testTimeoutSeconds}s` : `exit ${run.exitCode}`
|
|
547
608
|
const mismatch = browserMismatch(runBrowser, run.browsers, report.command)
|
|
548
609
|
if (mismatch) emit({ type: "code", kind: "info", text: `The test's browser differs from the run's: ${run.browsers.join(", ")}` })
|
|
@@ -555,14 +616,16 @@ export async function runCodeAgent(opts: {
|
|
|
555
616
|
break
|
|
556
617
|
}
|
|
557
618
|
if (sameFailures === 1) {
|
|
558
|
-
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed the same way as the last attempt, ${why} (attempt ${attempt}/${maxAttempts})` })
|
|
619
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed the same way as the last attempt, ${why} (attempt ${attempt}/${maxAttempts}): ${failureReason(run.output)[0] || "no error message"}` })
|
|
559
620
|
attemptLog.push(attemptLine(attempt, `failed, ${why}, the same way as before`, firstError(run.output), report.notes))
|
|
560
621
|
nextTask = repeatedFailurePrompt(sameFailures + 1, report.command, run.exitCode, run.output, files) + mismatch + switchLiveOn()
|
|
622
|
+
next = { why: `the test failed the same way again (${why}); Scribe looks at the failing step first`, fixing: failureReason(run.output) }
|
|
561
623
|
continue
|
|
562
624
|
}
|
|
563
|
-
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed, ${why} (attempt ${attempt}/${maxAttempts})` })
|
|
625
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed, ${why} (attempt ${attempt}/${maxAttempts}): ${failureReason(run.output)[0] || "no error message"}` })
|
|
564
626
|
attemptLog.push(attemptLine(attempt, `failed, ${why}`, firstError(run.output), report.notes))
|
|
565
627
|
nextTask = fixPrompt(attempt, report.command, run.exitCode, run.output, files) + mismatch
|
|
628
|
+
next = { why: `the test failed (${why})`, fixing: failureReason(run.output) }
|
|
566
629
|
}
|
|
567
630
|
keepSession = result.status === "stopped" || runtimeFailed
|
|
568
631
|
if (report.test_name) result.testName = report.test_name
|