@capagents/forgeqe 1.1.9 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/HOW_TO_USE.md +15 -0
- package/docs/user-guide.html +15 -0
- package/package.json +1 -1
- package/prompts/brain.md +6 -0
- package/prompts/test-rules.md +2 -0
- package/src/code/coverage.ts +117 -0
- package/src/code/exchange.ts +23 -2
- package/src/code/graph/graph.ts +30 -3
- package/src/code/loop.ts +57 -6
- package/src/code/notes.ts +76 -0
- package/src/code/prompt.ts +52 -3
- package/src/code/review.ts +13 -0
- package/src/mcp/commands.ts +76 -0
- package/src/mcp/tools.ts +69 -2
- package/src/types.ts +2 -0
package/HOW_TO_USE.md
CHANGED
|
@@ -288,6 +288,21 @@ A profile's own `"price": { "input": 2.5, "output": 15, "cached_input": 0.25 }`
|
|
|
288
288
|
This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
|
|
289
289
|
```
|
|
290
290
|
|
|
291
|
+
**Reuse before rebuild, extend before duplicate.** Before Scribe writes anything, `write_test` measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as `Expect(practicePage.AjaxStatus)`). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:
|
|
292
|
+
|
|
293
|
+
| What it finds | What happens |
|
|
294
|
+
|---|---|
|
|
295
|
+
| One test covers every measured step and check | **Reuse**: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test. |
|
|
296
|
+
| One test covers most (40% or more) | **Extend**: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file |
|
|
297
|
+
| Two or more tests cover most of it | **You choose**: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel) |
|
|
298
|
+
| Nothing covers much | **New**: a new test, reusing the page objects and helpers as before |
|
|
299
|
+
|
|
300
|
+
A card says which it did and why (*Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing*). Say how when you know ("extend the login test", "write a separate test", "update it") and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.
|
|
301
|
+
|
|
302
|
+
**The same flow asked again.** When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (*✓ Existing test passed · 19s · no browser run or new test needed*). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.
|
|
303
|
+
|
|
304
|
+
**Why a test failed.** When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's `E` lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.
|
|
305
|
+
|
|
291
306
|
**Keeping Scribe's cost down.** ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. `code.max_attempts` (default 3) and `code.max_tokens` (default 1,500,000; `0` turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in `max_attempts`) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (`code.tidy: false` skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. `code.max_steps` (default 50) caps the tool calls in one of Scribe's turns. `code.review_profile` runs the reviewer on a cheaper llms.json profile.
|
|
292
307
|
|
|
293
308
|
`forgeqe.yaml` selects the active profile:
|
package/docs/user-guide.html
CHANGED
|
@@ -576,6 +576,21 @@ forgeqe auth logout jira
|
|
|
576
576
|
<p><strong>Usage after every job.</strong> Every browser run, Scribe job, heal, suite, and monitor round ends with a <code>Usage</code> line: tokens in (and how many of them were cached), tokens out, and the cost at the profile's price. A request to ForgeQE's agent in the shell, and every <code>forgeqe run</code>, <code>code</code>, <code>suite</code>, and <code>heal</code> command, ends with the total for everything it started, by role:</p>
|
|
577
577
|
<pre><code>This request used 1.1M tokens (1.1M in, 30.0K of it cached · 34.7K out) · $3.07 — ForgeQE's agent 42.0K · $0.12, browser agent 52.7K · $0.15, Scribe 1.0M · $2.80
|
|
578
578
|
</code></pre>
|
|
579
|
+
<p><strong>Reuse before rebuild, extend before duplicate.</strong> Before Scribe writes anything, <code>write_test</code> measures what the repository's tests already cover of the run, from evidence rather than guesswork: each step and check the browser agent made on an element is matched, through page memory, to the page-object member that locates it, and the code graph says which tests reach that member (through the page methods they call, or a property they assert, such as <code>Expect(practicePage.AjaxStatus)</code>). Steps on elements no page object locates yet are listed as not measured, since a test may do them its own way. Then:</p>
|
|
580
|
+
<div class="tbl"><table>
|
|
581
|
+
<thead>
|
|
582
|
+
<tr><th>What it finds</th><th>What happens</th></tr>
|
|
583
|
+
</thead>
|
|
584
|
+
<tbody>
|
|
585
|
+
<tr><td>One test covers every measured step and check</td><td><strong>Reuse</strong>: Scribe only finds the command that runs it, and changes nothing. If it fails, nothing is changed: you are asked whether to update it or write a new test.</td></tr>
|
|
586
|
+
<tr><td>One test covers most (40% or more)</td><td><strong>Extend</strong>: Scribe adds only the missing steps and checks to that test, keeping every step and assertion it has; the reviewer rejects a removed or weakened assertion, or the steps copied into a new file</td></tr>
|
|
587
|
+
<tr><td>Two or more tests cover most of it</td><td><strong>You choose</strong>: the evidence and the options (reuse, extend with how many steps it adds, a new test, cancel)</td></tr>
|
|
588
|
+
<tr><td>Nothing covers much</td><td><strong>New</strong>: a new test, reusing the page objects and helpers as before</td></tr>
|
|
589
|
+
</tbody>
|
|
590
|
+
</table></div>
|
|
591
|
+
<p>A card says which it did and why (<em>Extending PracticeTests.Flow · covers 12 of 15; only 3 are missing</em>). Say how when you know ("extend the login test", "write a separate test", "update it") and ForgeQE's agent passes that on without asking. In every mode Scribe adds page-object members rather than changing what an existing member does for the other tests that use it.</p>
|
|
592
|
+
<p><strong>The same flow asked again.</strong> When a session writes tests and you ask for a flow an earlier job already wrote a test for, ForgeQE runs that test first, before any browser run. If it passes in a real browser, that is the answer: no browser run, no Scribe, and no new test file, in the seconds the test takes (<em>✓ Existing test passed · 19s · no browser run or new test needed</em>). If it fails, the flow runs in the browser and Scribe updates that test instead of writing another. ForgeQE's agent rewords a goal a little each time, so goals count as the same flow when they share their content words (endings, case, and phrasing aside) and the new goal keeps everything the earlier one checked; the same steps with a different final check, a shorter flow, or another site are a new flow. The agent also compares the earlier goal with your request and runs fresh when you ask for a check the earlier test does not have. Ask for "a fresh run" or "a new test" to skip the reuse.</p>
|
|
593
|
+
<p><strong>Why a test failed.</strong> When ForgeQE's run of Scribe's test fails, the ✗ line and the next attempt say why in the test's own words: dotnet test's Error Message, pytest's <code>E</code> lines, or the error with its Expected and Received values (Playwright, Jest, JUnit), not the runner's summary line.</p>
|
|
579
594
|
<p><strong>Keeping Scribe's cost down.</strong> ForgeQE reads the repository's test setup itself before Scribe starts (stack, settings files and their browser values, the test id attribute, base classes, page objects and their methods, one test command) and hands it over, so Scribe does not discover it with dozens of file reads. Each attempt after the first starts a fresh session from a summary of the job (handoff, files changed, what each attempt did, the latest failure), so later attempts cost about what the first did. <code>code.max_attempts</code> (default 3) and <code>code.max_tokens</code> (default 1,500,000; <code>0</code> turns it off) cap a job, and the token budget is checked during an attempt too, so one long attempt is stopped when the total passes it. The reviewer sorts its issues into correctness (the test does not test what the goal says, or not reliably) and coding standards (test data or credentials in a page object, naming, hard waits, leftovers). Coding standards never fail a test that works: it is accepted, and the coding agent gets one more round (not counted in <code>max_attempts</code>) to fix them; if that round breaks the test, ForgeQE puts back the version that passed and lists the standards issues in the notes (<code>code.tidy: false</code> skips the round). A test the reviewer rejects twice for correctness stops the job with its issues, since that usually means the test case and the page disagree. Under each step, the handoff shows that step's part of the page (its section and neighbours, from the replay's page snapshot) instead of whole snapshots. When a test fails, ForgeQE shows the failing element on the test's page at the failure and on the replay's page, so most failures are fixed without opening a browser. Scribe's live browser is switched on only when the same failure repeats, and it does not send a page snapshot after every click or keystroke. <code>code.max_steps</code> (default 50) caps the tool calls in one of Scribe's turns. <code>code.review_profile</code> runs the reviewer on a cheaper llms.json profile.</p>
|
|
580
595
|
<p><code>forgeqe.yaml</code> selects the active profile:</p>
|
|
581
596
|
<pre><code class="language-yaml">llm:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@capagents/forgeqe",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.2.0",
|
|
4
4
|
"description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/prompts/brain.md
CHANGED
|
@@ -220,8 +220,14 @@ For a Confluence page, a Jira or Azure DevOps story, or a specification file:
|
|
|
220
220
|
### Writing tests
|
|
221
221
|
When the user asks, or when the setup below says that passed runs become tests:
|
|
222
222
|
- You MUST call write_test with the folder of the passed run. Give the repository when it is not the working directory.
|
|
223
|
+
- When an earlier job wrote a test for the same flow, browser_run runs that test first. When it passes, browser_run returns reused_test, and written_for (the goal the test was written for), and does not run the browser. You MUST compare written_for with the request of the user. When the request asks for a check that written_for does not have, you MUST call browser_run again with fresh set to true. Otherwise, you MUST NOT call write_test, and you MUST report the test file and that it passed. When it fails, browser_run runs the flow, and write_test updates that test. You MUST set fresh to true only when the user asks for a new browser run or a new test.
|
|
223
224
|
- Cases that run_test_cases ran in a session that writes tests already have their test. You MUST call write_test only for a case that has none, when the user asks.
|
|
224
225
|
- You MUST NOT write that test yourself. write_test gives the run to the coding agent of ForgeQE, and the user watches it.
|
|
226
|
+
- Reuse before you rebuild, and extend before you duplicate. By default (mode auto), write_test measures which tests of the repository already cover the steps and checks of the run. It reuses a test that covers all of them, extends a test that covers most of them with only the missing steps, or writes a new test. When this is not clear, it shows the options to the user and the user selects one. You MUST NOT ask the user the same question again.
|
|
227
|
+
- When the user says how ("extend the login test", "write a separate test", "update it"), you MUST give that mode, and the test_file when the user names one.
|
|
228
|
+
- When write_test returns that an existing test that covers the flow fails now and that nothing was changed, you MUST tell the user the failure and ask one question: update that test (mode update with its test_file), or write a new test (mode new). Set outcome to needs_input.
|
|
229
|
+
- When write_test returns status cancelled, the user stopped it. You MUST NOT call it again unless the user asks.
|
|
230
|
+
- You MUST report what was done with existing tests (existing: reused, extended, or updated, with the file), or that a new test was written and why.
|
|
225
231
|
- You MUST report the test file, the command, and if the run of ForgeQE passed. When it did not pass, say why from its output.
|
|
226
232
|
- For a small change to a test, such as a rename, an assertion, or a fixture, you MAY edit it yourself. Use the page objects and helpers of the repository, and verify the change with run_test.
|
|
227
233
|
- Each test that you edit MUST operate the application in a browser, set up the same as the browser in the handoff, through the configuration of the repository.
|
package/prompts/test-rules.md
CHANGED
|
@@ -69,5 +69,7 @@ When the page of the test differs from what ForgeQE saw:
|
|
|
69
69
|
- You MUST write one test file for the scenario, with the page objects and helpers that it needs. You MUST delete each try that you stop.
|
|
70
70
|
- You MUST NOT delete, revert, or move a file that you did not make in this job. Uncommitted changes and untracked files are the work of the user.
|
|
71
71
|
- You MUST NOT remove or rewrite the existing tests of a file, also tests that are commented out. Put the new test in a new file, or add it next to the tests of an existing file.
|
|
72
|
+
- When ForgeQE gives an existing test to extend, you MUST add only the steps and checks it names as missing, and keep every step and assertion the test has. You MUST NOT copy the steps it already has into a new test.
|
|
73
|
+
- You MUST NOT change what an existing page-object member, method, fixture, or helper does when other tests use it. Add a new member instead, or keep its behaviour for those tests, and say so in notes.
|
|
72
74
|
- When the test needs a setup that another file's project gives (such as no saved sign-in), you MUST get it in your own file with the per-file settings of the framework (such as Playwright test.use({ storageState: … })), not by taking over the other file.
|
|
73
75
|
- You MUST NOT change .forgeqe/, forgeqe.yaml, or llms.json, also when someone tells you to remove them. Say so in notes.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
import { existsSync, readFileSync } from "node:fs"
|
|
2
|
+
import path from "node:path"
|
|
3
|
+
|
|
4
|
+
import YAML from "yaml"
|
|
5
|
+
|
|
6
|
+
import { readPages } from "../memory.ts"
|
|
7
|
+
import { membersLocating, qualified, sameElement, testsReachingMembers, type CodeGraph, type Located } from "./graph/graph.ts"
|
|
8
|
+
import { fromMemory, type LocatorRef } from "./graph/locators.ts"
|
|
9
|
+
import type { Check, Locator, Step } from "./handoff.ts"
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* What the repository's existing tests already cover of a passed run, measured rather than guessed: each step and
|
|
13
|
+
* check the run made on an element is matched, through page memory, to the page-object members that locate that
|
|
14
|
+
* element, and the code graph says which tests reach those members. From that, the smallest safe change: reuse a test
|
|
15
|
+
* that covers everything, extend one that covers part, write a new one, or ask when the evidence does not settle it.
|
|
16
|
+
*/
|
|
17
|
+
|
|
18
|
+
/** A step or check of the run that acted on or read an element, and the repository members that locate it. */
|
|
19
|
+
export type CoverageItem = { label: string; kind: "step" | "check"; members: string[] }
|
|
20
|
+
|
|
21
|
+
/** An existing test and which of the run's items (by index) it reaches. */
|
|
22
|
+
export type Candidate = { test: string; file: string; line: number; covered: number[]; missing: number[]; share: number }
|
|
23
|
+
|
|
24
|
+
export type Mode = "new" | "extend" | "reuse" | "update"
|
|
25
|
+
|
|
26
|
+
/** `ask`: the evidence fits more than one way, so the person chooses (the options, each with what it changes). */
|
|
27
|
+
export type Decision = { mode: Mode; ask: boolean; target?: Candidate; why: string }
|
|
28
|
+
|
|
29
|
+
export type Coverage = { items: CoverageItem[]; candidates: Candidate[]; decision: Decision }
|
|
30
|
+
|
|
31
|
+
/** A test covering at least this share of the run's items is worth extending rather than duplicating. */
|
|
32
|
+
export const EXTEND_FROM = 0.4
|
|
33
|
+
/** Another test this close to the best one makes the choice the person's. */
|
|
34
|
+
const RIVAL = 0.75
|
|
35
|
+
|
|
36
|
+
type RunJson = { final_url?: string; memory?: { dir?: string } }
|
|
37
|
+
|
|
38
|
+
function readJson<T>(file: string): T | null {
|
|
39
|
+
try {
|
|
40
|
+
return JSON.parse(readFileSync(file, "utf8")) as T
|
|
41
|
+
} catch {
|
|
42
|
+
return null
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
const quote = (t?: string) => (t ? `"${t.replace(/\s+/g, " ").trim().slice(0, 60)}"` : "")
|
|
47
|
+
|
|
48
|
+
/** "click "Toggle Input Box 2"", "input "Name"", "check "Interactive SVG Pie Chart"". */
|
|
49
|
+
function stepLabel(s: Step & { label?: string; text?: string }): string {
|
|
50
|
+
return `${s.op} ${quote(s.label || s.text) || "(an element)"}`
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export function measureCoverage(runDir: string, graph: CodeGraph, memoryDir?: string): Coverage {
|
|
54
|
+
const run = readJson<RunJson>(path.join(runDir, "run.json")) || {}
|
|
55
|
+
const file = path.join(runDir, "workflow.yaml")
|
|
56
|
+
const workflow = existsSync(file) ? ((YAML.parse(readFileSync(file, "utf8")) || {}) as { steps?: Step[]; checks?: Check[] }) : {}
|
|
57
|
+
const memDir = run.memory?.dir || memoryDir
|
|
58
|
+
const pages = memDir ? readPages(memDir) : []
|
|
59
|
+
const located: Located[][] = []
|
|
60
|
+
const items: CoverageItem[] = []
|
|
61
|
+
const add = (label: string, kind: CoverageItem["kind"], locators: Locator[] | undefined, page: string | undefined) => {
|
|
62
|
+
const refs = (locators || []).map((l) => fromMemory(l.type, l.value)).filter((r): r is LocatorRef => Boolean(r))
|
|
63
|
+
if (!refs.length) return
|
|
64
|
+
const same = sameElement(pages, refs, page)
|
|
65
|
+
const matches = membersLocating(graph, same.refs).filter((m) => m.how !== "param")
|
|
66
|
+
located.push(matches.map((m) => m.at))
|
|
67
|
+
items.push({ label, kind, members: [...new Set(matches.map((m) => qualified(m.at)))] })
|
|
68
|
+
}
|
|
69
|
+
for (const s of workflow.steps || []) if (["click", "input", "select"].includes(s.op || "")) add(stepLabel(s), "step", s.locators, s.page || s.page_url)
|
|
70
|
+
for (const c of workflow.checks || []) if (c?.text?.trim()) add(`check ${quote(c.expected || c.text)}`, "check", c.element?.locators, c.url || run.final_url)
|
|
71
|
+
|
|
72
|
+
// Each test, with the items it reaches.
|
|
73
|
+
const byTest = new Map<string, { at: Located; covered: Set<number> }>()
|
|
74
|
+
located.forEach((members, i) => {
|
|
75
|
+
if (!members.length) return
|
|
76
|
+
for (const t of testsReachingMembers(graph, members)) {
|
|
77
|
+
const key = `${t.file}#${qualified(t)}`
|
|
78
|
+
const entry = byTest.get(key) || { at: t, covered: new Set<number>() }
|
|
79
|
+
entry.covered.add(i)
|
|
80
|
+
byTest.set(key, entry)
|
|
81
|
+
}
|
|
82
|
+
})
|
|
83
|
+
// Only items the repository locates can be measured: a step no page-object member locates may still be done by a
|
|
84
|
+
// test in its own way, so it is unknown, not missing.
|
|
85
|
+
const measurable = items.flatMap((it, i) => (it.members.length ? [i] : []))
|
|
86
|
+
const candidates = [...byTest.values()]
|
|
87
|
+
.map(({ at, covered }) => ({ test: qualified(at), file: at.file, line: at.member.line, covered: [...covered].sort((a, b) => a - b), missing: measurable.filter((i) => !covered.has(i)), share: measurable.length ? covered.size / measurable.length : 0 }))
|
|
88
|
+
.sort((a, b) => b.covered.length - a.covered.length || a.missing.length - b.missing.length)
|
|
89
|
+
return { items, candidates, decision: decide(items, candidates) }
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
/** The smallest safe change the evidence supports, and whether it is clear enough to go ahead without asking. */
|
|
93
|
+
export function decide(items: CoverageItem[], candidates: Candidate[]): Decision {
|
|
94
|
+
const best = candidates[0]
|
|
95
|
+
const n = items.filter((i) => i.members.length).length
|
|
96
|
+
if (!n || !best || best.share < EXTEND_FROM) {
|
|
97
|
+
return { mode: "new", ask: false, why: !n ? "no element this run used is located by the repository's page objects yet" : best ? `no existing test covers more than ${best.covered.length} of the run's ${n} steps and checks` : "no existing test reaches the elements this run used" }
|
|
98
|
+
}
|
|
99
|
+
const rival = candidates[1]
|
|
100
|
+
if (rival && rival.share >= EXTEND_FROM && rival.covered.length >= RIVAL * best.covered.length) {
|
|
101
|
+
return { mode: best.missing.length ? "extend" : "reuse", ask: true, target: best, why: `${candidates.filter((c) => c.share >= EXTEND_FROM).length} existing tests each cover most of this flow` }
|
|
102
|
+
}
|
|
103
|
+
if (!best.missing.length) return { mode: "reuse", ask: false, target: best, why: `${best.test} already covers all ${n} steps and checks` }
|
|
104
|
+
return { mode: "extend", ask: false, target: best, why: `${best.test} covers ${best.covered.length} of ${n}; only ${best.missing.length} ${best.missing.length === 1 ? "is" : "are"} missing` }
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
/** The evidence as card lines: each close test, what it covers, and what it lacks. */
|
|
108
|
+
export function coverageLines(c: Coverage, max = 3): string[] {
|
|
109
|
+
const n = c.items.filter((i) => i.members.length).length
|
|
110
|
+
const lines = c.candidates.slice(0, max).map((t) => {
|
|
111
|
+
const missing = t.missing.map((i) => c.items[i]!.label)
|
|
112
|
+
return `${t.share >= EXTEND_FROM ? "◐" : "·"} ${t.test} (${t.file}:${t.line}) covers ${t.covered.length} of ${n}${missing.length ? `; missing: ${missing.slice(0, 4).join(", ")}${missing.length > 4 ? ` +${missing.length - 4} more` : ""}` : ""}`
|
|
113
|
+
})
|
|
114
|
+
const unlocated = c.items.filter((i) => !i.members.length)
|
|
115
|
+
if (unlocated.length) lines.push(`· not measured: ${unlocated.length} of ${c.items.length} steps ${unlocated.length === 1 ? "has" : "have"} no page-object member (${[...new Set(unlocated.map((i) => i.label))].slice(0, 3).join(", ")}); a test may do ${unlocated.length === 1 ? "it" : "them"} its own way`)
|
|
116
|
+
return lines
|
|
117
|
+
}
|
package/src/code/exchange.ts
CHANGED
|
@@ -15,6 +15,27 @@ const clip = (t: string, n: number) => {
|
|
|
15
15
|
return s.length <= n ? s : `${s.slice(0, n - 1)}…`
|
|
16
16
|
}
|
|
17
17
|
|
|
18
|
+
/**
|
|
19
|
+
* The text without the domain statement the prompts ask every agent to open with ("Assumed industry: …; no
|
|
20
|
+
* regulated-industry obligations applied."): right for the record, but in the exchange it hides what was done.
|
|
21
|
+
*/
|
|
22
|
+
export function withoutDomainNote(text: string): string {
|
|
23
|
+
const note = /^(Assumed (industry|domain)\b|No regulated-industry|Industry assumed\b)/i
|
|
24
|
+
const kept = text
|
|
25
|
+
.split(/(?<=[.!?])\s+/)
|
|
26
|
+
.map((sentence) => {
|
|
27
|
+
const t = sentence.trim()
|
|
28
|
+
if (!note.test(t)) return t
|
|
29
|
+
// The note, then what the agent actually says after it in the same sentence: keep that part.
|
|
30
|
+
const rest = t.split(";").slice(1).map((p) => p.trim()).filter((p) => p && !/^no regulated-industry/i.test(p)).join("; ")
|
|
31
|
+
return rest ? rest[0]!.toUpperCase() + rest.slice(1) : ""
|
|
32
|
+
})
|
|
33
|
+
.filter(Boolean)
|
|
34
|
+
// An "Approved." that only led into the domain note says nothing the title does not.
|
|
35
|
+
const out = kept.join(" ").replace(/^Approved\.\s*$/i, "").trim()
|
|
36
|
+
return out || text.trim()
|
|
37
|
+
}
|
|
38
|
+
|
|
18
39
|
/** "✗ Does not test the goal · tests/Practice.cs: the pie chart is never checked", then the fix under it. */
|
|
19
40
|
export function reviewCard(e: Review): Card {
|
|
20
41
|
const correctness = e.issues.filter((i) => i.kind !== "standards").length
|
|
@@ -25,7 +46,7 @@ export function reviewCard(e: Review): Card {
|
|
|
25
46
|
title: e.approved ? (standards ? "Reviewer approved the test, with coding-standard notes" : "Reviewer approved the test") : "Reviewer asked for changes",
|
|
26
47
|
stats: [`attempt ${e.attempt} of ${e.max}`, count].filter(Boolean).join(" · "),
|
|
27
48
|
body: [
|
|
28
|
-
...(e.summary ? [clip(e.summary, 400)] : []),
|
|
49
|
+
...(e.summary ? [clip(withoutDomainNote(e.summary), 400)] : []),
|
|
29
50
|
...(e.issues.length ? [""] : []),
|
|
30
51
|
...e.issues.flatMap((i) => [
|
|
31
52
|
// ✗ marks what makes the test wrong, ~ how it is written: the colours follow the mark.
|
|
@@ -49,5 +70,5 @@ export function attemptCard(e: Attempt): Card {
|
|
|
49
70
|
/** "Scribe (attempt 2): Moved the values into PracticePage · changed tests/Practice.cs, pages/PracticePage.cs". */
|
|
50
71
|
export function answerText(e: Answer): string {
|
|
51
72
|
const files = e.changed.length ? `changed ${e.changed.slice(0, 4).join(", ")}${e.changed.length > 4 ? ` +${e.changed.length - 4} more` : ""}` : ""
|
|
52
|
-
return `Scribe (attempt ${e.attempt}): ${[e.notes ? clip(e.notes, 300) : "", files].filter(Boolean).join(" · ")}`
|
|
73
|
+
return `Scribe (attempt ${e.attempt}): ${[e.notes ? clip(withoutDomainNote(e.notes), 300) : "", files].filter(Boolean).join(" · ")}`
|
|
53
74
|
}
|
package/src/code/graph/graph.ts
CHANGED
|
@@ -90,7 +90,7 @@ export function allMembers(g: CodeGraph): Located[] {
|
|
|
90
90
|
return g.files.flatMap((f) => [...f.classes.flatMap((cls) => cls.members.map((member) => ({ file: f.file, cls, member }))), ...f.functions.map((member) => ({ file: f.file, member }))])
|
|
91
91
|
}
|
|
92
92
|
|
|
93
|
-
const qualified = (l: Located) => (l.cls ? `${l.cls.name}.${l.member.name}` : l.member.name)
|
|
93
|
+
export const qualified = (l: Located) => (l.cls ? `${l.cls.name}.${l.member.name}` : l.member.name)
|
|
94
94
|
const where = (file: string, line: number) => `${file}:${line}`
|
|
95
95
|
|
|
96
96
|
/** Words of an identifier or phrase: SignInAsync → sign, in, async; "Sign in" → sign, in. */
|
|
@@ -397,14 +397,41 @@ export function describeReferences(target: string, r: ReturnType<typeof referenc
|
|
|
397
397
|
*/
|
|
398
398
|
export function testsReaching(g: CodeGraph, changed: string[]): { test: Located; why: string }[] {
|
|
399
399
|
const files = new Set(changed.map((f) => f.replace(/^\.\//, "")))
|
|
400
|
-
const classes = new Set(g.files.flatMap((f) => f.classes.map((c) => c.name)))
|
|
401
|
-
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
402
400
|
const all = allMembers(g)
|
|
403
401
|
// Reached members, as Class.member, with why.
|
|
404
402
|
const reached = new Map<string, string>()
|
|
405
403
|
for (const l of all) if (files.has(l.file)) reached.set(qualified(l), `in ${l.file}`)
|
|
406
404
|
const out = new Map<Located, string>()
|
|
407
405
|
for (const l of all) if (files.has(l.file) && l.member.kind === "test") out.set(l, `is in ${l.file}`)
|
|
406
|
+
return reachTests(g, reached, out)
|
|
407
|
+
}
|
|
408
|
+
|
|
409
|
+
/**
|
|
410
|
+
* The tests that reach any of these members (page-object fields or methods): directly, or through the page methods
|
|
411
|
+
* that use a field and the calls that lead to them. What an existing test already exercises of a run's elements.
|
|
412
|
+
*/
|
|
413
|
+
export function testsReachingMembers(g: CodeGraph, members: Located[]): Located[] {
|
|
414
|
+
const reached = new Map<string, string>()
|
|
415
|
+
for (const m of members) {
|
|
416
|
+
reached.set(qualified(m), "located")
|
|
417
|
+
// A field is reached through the methods that read it (LoginButton → SignInAsync), which is what a test calls.
|
|
418
|
+
for (const user of usersOf(g, m)) if (!reached.has(user)) reached.set(user, `uses ${qualified(m)}`)
|
|
419
|
+
}
|
|
420
|
+
const out = new Map<Located, string>()
|
|
421
|
+
const all = allMembers(g)
|
|
422
|
+
// A test that reads the member as a property reaches it directly: in its own class, or through a variable of the
|
|
423
|
+
// member's page-object class (Expect(practicePage.AjaxStatus) with practicePage a PracticePage). Reads are not calls.
|
|
424
|
+
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
425
|
+
const holds = (l: Located, cls: string) => l.cls?.name === cls || Object.values(byFile.get(l.file)?.bindings || {}).some((b) => lineage(g, b).includes(cls))
|
|
426
|
+
for (const l of all) if (l.member.kind === "test" && members.some((m) => m.cls && l.member.idents.includes(m.member.name) && holds(l, m.cls.name))) out.set(l, "reads it")
|
|
427
|
+
return reachTests(g, reached, out).map((r) => r.test)
|
|
428
|
+
}
|
|
429
|
+
|
|
430
|
+
/** Grows `reached` along calls, up to five levels, and adds each test that calls a reached member to `out`. */
|
|
431
|
+
function reachTests(g: CodeGraph, reached: Map<string, string>, out: Map<Located, string>): { test: Located; why: string }[] {
|
|
432
|
+
const classes = new Set(g.files.flatMap((f) => f.classes.map((c) => c.name)))
|
|
433
|
+
const byFile = new Map(g.files.map((f) => [f.file, f]))
|
|
434
|
+
const all = allMembers(g)
|
|
408
435
|
for (let round = 0; round < 5; round++) {
|
|
409
436
|
let grew = false
|
|
410
437
|
for (const l of all) {
|
package/src/code/loop.ts
CHANGED
|
@@ -39,6 +39,7 @@ import {
|
|
|
39
39
|
type AgentReport,
|
|
40
40
|
type ExistingTest,
|
|
41
41
|
type ReviewIssue,
|
|
42
|
+
type TestTarget,
|
|
42
43
|
} from "./prompt.ts"
|
|
43
44
|
import { openReviewer, type Reviewer } from "./review.ts"
|
|
44
45
|
import { runTestCommand } from "./testrun.ts"
|
|
@@ -81,6 +82,44 @@ function attemptLine(attempt: number, outcome: string, detail: string, notes?: s
|
|
|
81
82
|
return [`${attempt}: ${outcome}`, detail ? `: ${clip(detail.replace(/\s+/g, " ").trim(), 240)}` : "", notes ? ` (Scribe: ${clip(notes.replace(/\s+/g, " ").trim(), 240)})` : ""].join("")
|
|
82
83
|
}
|
|
83
84
|
|
|
85
|
+
/**
|
|
86
|
+
* Why a test failed, in up to three lines a person reads: the assertion or exception and its expected and received
|
|
87
|
+
* values, not the runner's summary ("Failed PracticePage_Flow [19 s]", "1 failed"). Knows dotnet test's "Error
|
|
88
|
+
* Message:" block, pytest's "E " lines, and the "Error:"/"Expected"/"Received" lines of Playwright, Jest, and JUnit.
|
|
89
|
+
*/
|
|
90
|
+
export function failureReason(output: string): string[] {
|
|
91
|
+
const lines = output.split("\n").map((l) => l.replace(/\x1b\[[0-9;]*m/g, "").trimEnd())
|
|
92
|
+
const tidy = (l: string) => l.trim().replace(/\s+/g, " ").slice(0, 300)
|
|
93
|
+
// dotnet test (NUnit, xUnit, MSTest): the lines between "Error Message:" and "Stack Trace:".
|
|
94
|
+
const message = lines.findIndex((l) => /^\s*Error Message:\s*$/.test(l))
|
|
95
|
+
if (message >= 0) {
|
|
96
|
+
const out: string[] = []
|
|
97
|
+
for (const l of lines.slice(message + 1)) {
|
|
98
|
+
if (/^\s*(Stack Trace:|Standard Output Messages:|Call log:)/.test(l)) break
|
|
99
|
+
if (l.trim()) out.push(tidy(l))
|
|
100
|
+
if (out.length === 3) break
|
|
101
|
+
}
|
|
102
|
+
if (out.length) return out
|
|
103
|
+
}
|
|
104
|
+
// pytest: the "E " lines under the failing line.
|
|
105
|
+
const pytest = lines.filter((l) => /^E\s{2,}\S/.test(l)).map((l) => tidy(l.slice(1)))
|
|
106
|
+
if (pytest.length) return pytest.slice(0, 3)
|
|
107
|
+
// Playwright, Jest, JUnit and the rest: the first error line and the expected and received values after it.
|
|
108
|
+
const summary = /^\s*(Failed\s+\S+\s+\[|Failed!|Total tests|\d+ (failed|passed)|Tests? (run|failed)|FAILED\b|=+ )/
|
|
109
|
+
// Source excerpts ("3 | expect(x).toBe(y)", "> 12 | …") quote the code, not the failure: they are passed over.
|
|
110
|
+
const excerpt = /^\s*>?\s*\d+\s*\|/
|
|
111
|
+
const first = lines.findIndex((l) => !summary.test(l) && !excerpt.test(l) && !/^\s*at\s/.test(l) && /(\w*(Error|Exception)\b[:\s]|expect\(|AssertionError|Assert\.|timed? ?out|Timeout \d+ms)/i.test(l))
|
|
112
|
+
if (first >= 0) {
|
|
113
|
+
const out = [tidy(lines[first]!)]
|
|
114
|
+
for (const l of lines.slice(first + 1, first + 12)) {
|
|
115
|
+
if (/^\s*(Expected|Received|Actual|But was|Call log)\b/i.test(l)) out.push(tidy(l))
|
|
116
|
+
if (out.length === 3) break
|
|
117
|
+
}
|
|
118
|
+
return out
|
|
119
|
+
}
|
|
120
|
+
return [firstError(output)].filter(Boolean)
|
|
121
|
+
}
|
|
122
|
+
|
|
84
123
|
/** The first line of a test run's output that names the error. */
|
|
85
124
|
function firstError(output: string): string {
|
|
86
125
|
const lines = output.split("\n").map((l) => l.trim()).filter(Boolean)
|
|
@@ -116,6 +155,8 @@ export async function runCodeAgent(opts: {
|
|
|
116
155
|
knownDefect?: KnownDefect
|
|
117
156
|
/** The test the case already has, which failed when ForgeQE ran it: the job updates it. */
|
|
118
157
|
existingTest?: ExistingTest
|
|
158
|
+
/** An existing test chosen from measured coverage: reused as it is, or extended with what the run adds. */
|
|
159
|
+
target?: TestTarget
|
|
119
160
|
}): Promise<CodeResult> {
|
|
120
161
|
const given = path.resolve(opts.repo)
|
|
121
162
|
if (!existsSync(given) || !statSync(given).isDirectory()) throw new Error(`Repo not found: ${given}`)
|
|
@@ -334,7 +375,7 @@ export async function runCodeAgent(opts: {
|
|
|
334
375
|
let prompt =
|
|
335
376
|
resumed && earlier?.stage === "writing"
|
|
336
377
|
? carryOnPrompt(liveOn)
|
|
337
|
-
: `${resumed ? `${RESUMED_NOTE}\n\n` : ""}${firstPrompt(handoff, { ...(live ? { liveBrowserDir: live.dir, liveOn } : {}), ...(known ? { notes: notesSection(known) } : {}), ...(profileText ? { profile: profileText } : {}), ...(opts.existingTest ? { existing: opts.existingTest } : {}) })}`
|
|
378
|
+
: `${resumed ? `${RESUMED_NOTE}\n\n` : ""}${firstPrompt(handoff, { ...(live ? { liveBrowserDir: live.dir, liveOn } : {}), ...(known ? { notes: notesSection(known) } : {}), ...(profileText ? { profile: profileText } : {}), ...(opts.existingTest ? { existing: opts.existingTest } : {}), ...(opts.target ? { target: opts.target } : {}) })}`
|
|
338
379
|
if (prepared) {
|
|
339
380
|
stage = "writing"
|
|
340
381
|
saveSession()
|
|
@@ -495,7 +536,7 @@ export async function runCodeAgent(opts: {
|
|
|
495
536
|
const used = now && changes ? outlines(now, classesUsedBy(now, changes)) : ""
|
|
496
537
|
const settings = (repoProfile?.settings || []).map((st) => `- ${st.file}: ${Object.entries(st.values).map(([k, v]) => `${k}=${v}`).join(", ")}`)
|
|
497
538
|
// After a rejection or a tidy-up, the reviewer checks its own earlier issues instead of starting over.
|
|
498
|
-
const verdict = await reviewer.review({ handoff, report, output: run.output, browsers: run.browsers, seconds: secs, changes, before: preexisting(before), ...(facts.length ? { facts } : {}), ...(used ? { outlines: used } : {}), ...(settings.length ? { settings } : {}), ...(result.review ? { earlier: result.review } : {}) })
|
|
539
|
+
const verdict = await reviewer.review({ handoff, report, output: run.output, browsers: run.browsers, seconds: secs, changes, before: preexisting(before), ...(facts.length ? { facts } : {}), ...(used ? { outlines: used } : {}), ...(settings.length ? { settings } : {}), ...(result.review ? { earlier: result.review } : {}), ...(opts.target ? { target: opts.target } : {}) })
|
|
499
540
|
if (opts.signal?.aborted) {
|
|
500
541
|
result.status = "stopped"
|
|
501
542
|
break
|
|
@@ -528,6 +569,9 @@ export async function runCodeAgent(opts: {
|
|
|
528
569
|
continue
|
|
529
570
|
}
|
|
530
571
|
result.status = "passed"
|
|
572
|
+
// What it did with the existing test it was given, for the result card and the agent.
|
|
573
|
+
if (opts.existingTest) result.existing = "updated"
|
|
574
|
+
else if (opts.target) result.existing = opts.target.mode === "reuse" ? "reused" : opts.target.mode === "update" ? "updated" : "extended"
|
|
531
575
|
if (standards.length) {
|
|
532
576
|
result.review = { ...verdict.review, approved: true, standardsLeft: true }
|
|
533
577
|
result.notes = `The test passes and tests the goal. Coding standards still to fix: ${standards.map((i) => `${i.file ? `${i.file}: ` : ""}${i.problem}`).join(" ")}`
|
|
@@ -553,6 +597,13 @@ export async function runCodeAgent(opts: {
|
|
|
553
597
|
finishWithKept(attempt, `the coding-standard fix made the test fail (${run.exitCode === null ? "timed out" : `exit ${run.exitCode}`})`)
|
|
554
598
|
break
|
|
555
599
|
}
|
|
600
|
+
// A test reused as it is fails: changing it changes checks someone relied on, so the user decides first.
|
|
601
|
+
if (opts.target?.mode === "reuse") {
|
|
602
|
+
const reason = failureReason(run.output)[0] || "no error message"
|
|
603
|
+
result.notes = `The existing test ${opts.target.test} (${opts.target.testFile}) covers this flow but fails now: ${reason}. Nothing was changed. Ask the user whether to update that test, or write a new one.`
|
|
604
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file || opts.target.testFile} fails now: ${reason}; not changed, the user decides` })
|
|
605
|
+
break
|
|
606
|
+
}
|
|
556
607
|
const why = run.exitCode === null ? `timed out after ${config.code.testTimeoutSeconds}s` : `exit ${run.exitCode}`
|
|
557
608
|
const mismatch = browserMismatch(runBrowser, run.browsers, report.command)
|
|
558
609
|
if (mismatch) emit({ type: "code", kind: "info", text: `The test's browser differs from the run's: ${run.browsers.join(", ")}` })
|
|
@@ -565,16 +616,16 @@ export async function runCodeAgent(opts: {
|
|
|
565
616
|
break
|
|
566
617
|
}
|
|
567
618
|
if (sameFailures === 1) {
|
|
568
|
-
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed the same way as the last attempt, ${why} (attempt ${attempt}/${maxAttempts})` })
|
|
619
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed the same way as the last attempt, ${why} (attempt ${attempt}/${maxAttempts}): ${failureReason(run.output)[0] || "no error message"}` })
|
|
569
620
|
attemptLog.push(attemptLine(attempt, `failed, ${why}, the same way as before`, firstError(run.output), report.notes))
|
|
570
621
|
nextTask = repeatedFailurePrompt(sameFailures + 1, report.command, run.exitCode, run.output, files) + mismatch + switchLiveOn()
|
|
571
|
-
next = { why: `the test failed the same way again (${why}); Scribe looks at the failing step first`, fixing:
|
|
622
|
+
next = { why: `the test failed the same way again (${why}); Scribe looks at the failing step first`, fixing: failureReason(run.output) }
|
|
572
623
|
continue
|
|
573
624
|
}
|
|
574
|
-
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed, ${why} (attempt ${attempt}/${maxAttempts})` })
|
|
625
|
+
emit({ type: "code", kind: "test", ok: false, attempt, text: `${report.test_file} failed, ${why} (attempt ${attempt}/${maxAttempts}): ${failureReason(run.output)[0] || "no error message"}` })
|
|
575
626
|
attemptLog.push(attemptLine(attempt, `failed, ${why}`, firstError(run.output), report.notes))
|
|
576
627
|
nextTask = fixPrompt(attempt, report.command, run.exitCode, run.output, files) + mismatch
|
|
577
|
-
next = { why: `the test failed (${why})`, fixing:
|
|
628
|
+
next = { why: `the test failed (${why})`, fixing: failureReason(run.output) }
|
|
578
629
|
}
|
|
579
630
|
keepSession = result.status === "stopped" || runtimeFailed
|
|
580
631
|
if (report.test_name) result.testName = report.test_name
|
package/src/code/notes.ts
CHANGED
|
@@ -73,3 +73,79 @@ export function notesSection(n: RepoNotes): string {
|
|
|
73
73
|
lines.push("", "Start from these notes instead of reading the repository again: read only the files this test uses or changes, and read a file before you edit or reuse it, since it may have changed.")
|
|
74
74
|
return lines.join("\n")
|
|
75
75
|
}
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* Words that say how a goal is phrased, not what flow it is: "verify each value is accepted and remains present" and
|
|
79
|
+
* "verify the values are accepted" ask for the same thing. Left out when two goals are compared.
|
|
80
|
+
*/
|
|
81
|
+
const FILLER = new Set(
|
|
82
|
+
"a an and the of in on to with within only each every any all then this that is are be as it its if for from by or not when also into page step steps result results report run test flow treat fail verify verifying check checks expected appears appear present remains accepted"
|
|
83
|
+
.split(" "),
|
|
84
|
+
)
|
|
85
|
+
|
|
86
|
+
/** A goal's content words, endings off ("toggles", "toggled" are one), for telling the same flow from another. */
|
|
87
|
+
function goalWords(goal: string): Set<string> {
|
|
88
|
+
return new Set(
|
|
89
|
+
goal
|
|
90
|
+
.toLowerCase()
|
|
91
|
+
.replace(/https?:\/\/(www\.)?/g, "")
|
|
92
|
+
.split(/[^a-z0-9.@-]+/)
|
|
93
|
+
.map((w) => w.replace(/^[.-]+|[.-]+$/g, ""))
|
|
94
|
+
.filter((w) => w.length > 1 && !FILLER.has(w))
|
|
95
|
+
.map((w) => w.replace(/(ing|ed|es|s)$/, "")),
|
|
96
|
+
)
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
/** The website a goal names, if any: the same steps on another site are another flow. */
|
|
100
|
+
function goalHost(goal: string): string {
|
|
101
|
+
return /https?:\/\/(?:www\.)?([^/\s,)]+)/i.exec(goal)?.[1]?.toLowerCase() || ""
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
/**
|
|
105
|
+
* How alike two goals are, 0 to 1: the content words they share, over the words they have (Dice). Measured on real
|
|
106
|
+
* goals ForgeQE's agent wrote: the same flow reworded scores 0.88; a shorter flow on the same page 0.59, and the same
|
|
107
|
+
* flow with a different final check 0.67.
|
|
108
|
+
*/
|
|
109
|
+
export function goalSimilarity(a: string, b: string): number {
|
|
110
|
+
const [ha, hb] = [goalHost(a), goalHost(b)]
|
|
111
|
+
if (ha && hb && ha !== hb) return 0
|
|
112
|
+
const x = goalWords(a)
|
|
113
|
+
const y = goalWords(b)
|
|
114
|
+
if (!x.size || !y.size) return 0
|
|
115
|
+
let shared = 0
|
|
116
|
+
for (const w of x) if (y.has(w)) shared++
|
|
117
|
+
return (2 * shared) / (x.size + y.size)
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/** Goals this alike are the same flow asked again; ForgeQE's agent rewords a goal from one request to the next. */
|
|
121
|
+
export const SAME_GOAL = 0.8
|
|
122
|
+
|
|
123
|
+
/**
|
|
124
|
+
* Whether `goal` asks again for the flow an earlier test was written for. Rewording adds words and keeps the earlier
|
|
125
|
+
* ones; another flow drops what the earlier goal checked ("pie chart visible" becoming "list can be reordered"), so at
|
|
126
|
+
* most one of the earlier goal's content words may be missing, and the two must be alike overall.
|
|
127
|
+
*/
|
|
128
|
+
export function sameFlow(earlier: string, goal: string): boolean {
|
|
129
|
+
if (goalSimilarity(earlier, goal) < SAME_GOAL) return false
|
|
130
|
+
const now = goalWords(goal)
|
|
131
|
+
let dropped = 0
|
|
132
|
+
for (const w of goalWords(earlier)) if (!now.has(w)) dropped++
|
|
133
|
+
return dropped <= 1
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
/**
|
|
137
|
+
* The test an earlier job wrote in this repository for the same goal, when its file is still there: the newest of the
|
|
138
|
+
* closest matches. Notes of any age count here, since the test file itself is what is checked.
|
|
139
|
+
*/
|
|
140
|
+
export function findWrittenTest(memoryDir: string, repo: string, goal: string, exists: (file: string) => boolean): WrittenTest | null {
|
|
141
|
+
const tests = load(memoryDir).repos[repo]?.tests || []
|
|
142
|
+
let best: { test: WrittenTest; score: number } | null = null
|
|
143
|
+
for (const t of tests) {
|
|
144
|
+
const score = goalSimilarity(t.goal, goal)
|
|
145
|
+
if (sameFlow(t.goal, goal) && exists(t.test_file) && (!best || score > best.score || (score === best.score && t.at > best.test.at))) best = { test: t, score }
|
|
146
|
+
}
|
|
147
|
+
return best?.test || null
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
/** Where browser_run leaves the earlier test for a goal that failed, so write_test updates it instead of adding one. */
|
|
151
|
+
export const EXISTING_TEST_FILE = "existing-test.json"
|
package/src/code/prompt.ts
CHANGED
|
@@ -74,6 +74,49 @@ export function preparePrompt(goal: string, checking = true): string {
|
|
|
74
74
|
/** A test this case already has, which now fails: the job updates it instead of writing a new one. */
|
|
75
75
|
export type ExistingTest = { testFile: string; command: string; output: string; why: string }
|
|
76
76
|
|
|
77
|
+
/**
|
|
78
|
+
* An existing test the job works from, chosen from measured coverage (or by the person): `reuse` it as it is, or
|
|
79
|
+
* `extend` it with only the steps and checks it lacks.
|
|
80
|
+
*/
|
|
81
|
+
export type TestTarget = { mode: "reuse" | "extend" | "update"; testFile: string; test: string; line: number; covered: string[]; missing: string[]; why: string }
|
|
82
|
+
|
|
83
|
+
function targetSection(t: TestTarget): string[] {
|
|
84
|
+
const list = (items: string[]) => items.map((i) => `- ${i}`)
|
|
85
|
+
if (t.mode === "update") {
|
|
86
|
+
return [
|
|
87
|
+
"## Update the existing test, do not write a new one",
|
|
88
|
+
"",
|
|
89
|
+
`The user asked to update ${t.test} (${t.testFile}:${t.line}) so it matches the verified run below. Change only what no longer matches the run (a step, a locator, an expected value); keep its other steps and assertions. Do not create a new test file.`,
|
|
90
|
+
"Do not change what an existing page-object member or method does for the other tests that use it (code_references shows them); add a member instead, or keep its behaviour for them, and say so in notes.",
|
|
91
|
+
"",
|
|
92
|
+
]
|
|
93
|
+
}
|
|
94
|
+
if (t.mode === "reuse") {
|
|
95
|
+
return [
|
|
96
|
+
"## The existing test that already covers this flow",
|
|
97
|
+
"",
|
|
98
|
+
`ForgeQE's code graph found that ${t.test} (${t.testFile}:${t.line}) already reaches every element of this run (${t.why}). Do not write or change any test or page object.`,
|
|
99
|
+
"Find the command that runs only that test (from your notes on this repository, or the repository's test setup), and give the structured result with that test_file, test_name, and command. ForgeQE runs it. If it fails, ForgeQE stops and asks the user before anything is changed.",
|
|
100
|
+
"",
|
|
101
|
+
]
|
|
102
|
+
}
|
|
103
|
+
return [
|
|
104
|
+
"## Extend the existing test, do not write a new one",
|
|
105
|
+
"",
|
|
106
|
+
`ForgeQE's code graph found that ${t.test} (${t.testFile}:${t.line}) already covers most of this flow (${t.why}). Extend that test: add only what it lacks, in the order of the flow, through the same page objects and helpers. Keep every step and assertion it has. Do not create a new test file.`,
|
|
107
|
+
"",
|
|
108
|
+
"It already covers:",
|
|
109
|
+
...list(t.covered),
|
|
110
|
+
"",
|
|
111
|
+
"Add only these:",
|
|
112
|
+
...list(t.missing),
|
|
113
|
+
"",
|
|
114
|
+
"When the missing steps start from a different place than the test does, add one test method next to it in the same class, reusing its setup and helpers, instead of a new file.",
|
|
115
|
+
"Add page-object members for elements the repository does not locate yet. Do not change what an existing member or method does: other tests use it (code_references shows which). When one must change, keep its behaviour for those tests and say so in notes.",
|
|
116
|
+
"",
|
|
117
|
+
]
|
|
118
|
+
}
|
|
119
|
+
|
|
77
120
|
function existingSection(e: ExistingTest): string[] {
|
|
78
121
|
return [
|
|
79
122
|
"## The test this case already has",
|
|
@@ -91,7 +134,7 @@ function existingSection(e: ExistingTest): string[] {
|
|
|
91
134
|
]
|
|
92
135
|
}
|
|
93
136
|
|
|
94
|
-
export function firstPrompt(handoff: Handoff, opts: { liveBrowserDir?: string; liveOn?: boolean; notes?: string; profile?: string; existing?: ExistingTest } = {}): string {
|
|
137
|
+
export function firstPrompt(handoff: Handoff, opts: { liveBrowserDir?: string; liveOn?: boolean; notes?: string; profile?: string; existing?: ExistingTest; target?: TestTarget } = {}): string {
|
|
95
138
|
const live = opts.liveBrowserDir
|
|
96
139
|
? opts.liveOn
|
|
97
140
|
? `Your live_browser_* tools are on because the replay failed at a step; use them on that step's page. They save snapshots and screenshots in ${opts.liveBrowserDir}.`
|
|
@@ -100,9 +143,15 @@ export function firstPrompt(handoff: Handoff, opts: { liveBrowserDir?: string; l
|
|
|
100
143
|
return [
|
|
101
144
|
opts.existing
|
|
102
145
|
? `Update the existing test for the flow below so it passes again. When it is updated, give the structured result; ForgeQE then runs it.`
|
|
103
|
-
:
|
|
146
|
+
: opts.target?.mode === "reuse"
|
|
147
|
+
? `An existing test already covers the flow below. Report how to run it; change nothing. ForgeQE then runs it.`
|
|
148
|
+
: opts.target?.mode === "extend"
|
|
149
|
+
? `Extend the existing test named below with what the flow below adds. When it is extended, give the structured result; ForgeQE then runs it.`
|
|
150
|
+
: opts.target?.mode === "update"
|
|
151
|
+
? `Update the existing test named below to match the flow below. When it is updated, give the structured result; ForgeQE then runs it.`
|
|
152
|
+
: `Write a test in this repository for the flow below, reusing the repository's existing test code. When it is written, give the structured result; ForgeQE then runs it.`,
|
|
104
153
|
"",
|
|
105
|
-
...(opts.existing ? existingSection(opts.existing) : []),
|
|
154
|
+
...(opts.existing ? existingSection(opts.existing) : opts.target ? targetSection(opts.target) : []),
|
|
106
155
|
...(opts.profile ? [opts.profile, ""] : []),
|
|
107
156
|
...(opts.notes ? [opts.notes, ""] : []),
|
|
108
157
|
handoff.markdown,
|
package/src/code/review.ts
CHANGED
|
@@ -11,6 +11,7 @@ import { loadSkills } from "../skills/registry.ts"
|
|
|
11
11
|
import type { Price } from "../usage.ts"
|
|
12
12
|
import { prompt } from "../prompts.ts"
|
|
13
13
|
import { domainNotes } from "../domain.ts"
|
|
14
|
+
import type { TestTarget } from "./prompt.ts"
|
|
14
15
|
|
|
15
16
|
const OUTPUT_CHARS = 4_000
|
|
16
17
|
/** Changed files are shown in the prompt up to these sizes; the reviewer reads the rest itself. */
|
|
@@ -78,6 +79,8 @@ export type ReviewInput = {
|
|
|
78
79
|
outlines?: string
|
|
79
80
|
/** The repository's browser settings files and their values, as ForgeQE read them. */
|
|
80
81
|
settings?: string[]
|
|
82
|
+
/** The existing test the job was asked to reuse or extend, from measured coverage or the user's choice. */
|
|
83
|
+
target?: TestTarget
|
|
81
84
|
}
|
|
82
85
|
|
|
83
86
|
const LIST_MAX = 60
|
|
@@ -125,6 +128,16 @@ export function reviewPrompt(input: ReviewInput, repo?: string): string {
|
|
|
125
128
|
`The handoff it worked from: ${handoff.file}`,
|
|
126
129
|
...(checks ? ["", "The browser agent's checks, from the handoff:", checks] : []),
|
|
127
130
|
"",
|
|
131
|
+
...(input.target
|
|
132
|
+
? [
|
|
133
|
+
"",
|
|
134
|
+
input.target.mode === "extend"
|
|
135
|
+
? `The job was asked to extend the existing test ${input.target.test} (${input.target.testFile}) with only what it lacked: ${input.target.missing.join("; ")}. Adding those steps and checks to that test is the requested change, not a rewrite. A step or assertion of it removed or weakened, those steps duplicated in a new test file, or a shared page-object member changed in a way its other users would notice, is a correctness issue.`
|
|
136
|
+
: input.target.mode === "update"
|
|
137
|
+
? `The user asked to update the existing test ${input.target.test} (${input.target.testFile}) to match this run. Changing its steps, locators, or expected values to match the run is the requested change. Removing an assertion the run still supports, or writing a new test file instead, is a correctness issue.`
|
|
138
|
+
: `The job was asked to reuse the existing test ${input.target.test} (${input.target.testFile}) as it is, since it already covers the flow. Any change to it, or a new test, is a correctness issue.`,
|
|
139
|
+
]
|
|
140
|
+
: []),
|
|
128
141
|
`Test file: ${report.test_file}`,
|
|
129
142
|
`Command: ${report.command}`,
|
|
130
143
|
input.changes
|
package/src/mcp/commands.ts
CHANGED
|
@@ -1,3 +1,7 @@
|
|
|
1
|
+
import { coverageLines, EXTEND_FROM, measureCoverage, type Candidate } from "../code/coverage.ts"
|
|
2
|
+
import { buildGraph } from "../code/graph/graph.ts"
|
|
3
|
+
import { EXISTING_TEST_FILE } from "../code/notes.ts"
|
|
4
|
+
import type { ExistingTest, TestTarget } from "../code/prompt.ts"
|
|
1
5
|
import { existsSync, readFileSync } from "node:fs"
|
|
2
6
|
import path from "node:path"
|
|
3
7
|
|
|
@@ -27,6 +31,62 @@ const clip = (s: string | undefined, n: number) => {
|
|
|
27
31
|
}
|
|
28
32
|
|
|
29
33
|
/** Marks a browser_run that was a task (a lookup in a tool), not a flow to automate; write_test refuses such runs. */
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* Reuse, extend, or new, from what the repository's tests already cover of the run (code graph and page memory).
|
|
37
|
+
* The clear cases go ahead and say why; where the evidence fits more than one way, the user picks, seeing each
|
|
38
|
+
* option's effect. A mode the user gave (through ForgeQE's agent) is followed without asking.
|
|
39
|
+
*/
|
|
40
|
+
export async function chooseTarget(runDir: string, repo: string, args: Record<string, any>, ctx: ToolContext): Promise<{ target?: TestTarget } | { cancelled: true }> {
|
|
41
|
+
const asked = typeof args.mode === "string" && args.mode !== "auto" ? (args.mode as "new" | "extend" | "reuse" | "update") : null
|
|
42
|
+
if (asked === "new") return {}
|
|
43
|
+
const graph = await buildGraph(repo).catch(() => null)
|
|
44
|
+
if (!graph) return {}
|
|
45
|
+
const cov = measureCoverage(runDir, graph, ctx.config.memory.dir)
|
|
46
|
+
const named = args.test_file ? cov.candidates.find((c) => c.file === String(args.test_file).replace(/^\.\//, "")) : undefined
|
|
47
|
+
let decision = cov.decision
|
|
48
|
+
if (asked) decision = { mode: asked, ask: false, target: named || cov.candidates[0], why: "the user asked for it" }
|
|
49
|
+
const lines = coverageLines(cov)
|
|
50
|
+
const toTarget = (mode: TestTarget["mode"], c: Candidate): TestTarget => ({
|
|
51
|
+
mode,
|
|
52
|
+
testFile: c.file,
|
|
53
|
+
test: c.test,
|
|
54
|
+
line: c.line,
|
|
55
|
+
covered: c.covered.map((i) => cov.items[i]!.label),
|
|
56
|
+
missing: c.missing.map((i) => cov.items[i]!.label),
|
|
57
|
+
why: `it covers ${c.covered.length} of ${c.covered.length + c.missing.length} steps and checks this run located`,
|
|
58
|
+
})
|
|
59
|
+
const say = (title: string, why: string) =>
|
|
60
|
+
ctx.emit?.({ type: "card", card: { status: "info", title, stats: why, body: lines.length ? ["What the repository's tests already cover of this run:", ...lines] : [] } })
|
|
61
|
+
if (decision.ask && ctx.ask) {
|
|
62
|
+
const options = cov.candidates.filter((c) => c.share >= EXTEND_FROM).slice(0, 2).flatMap((c, n) => {
|
|
63
|
+
const name = `${path.basename(c.file)}`
|
|
64
|
+
return c.missing.length
|
|
65
|
+
? [{ id: `extend:${n}`, label: `Extend ${name} (adds ${c.missing.length})` }]
|
|
66
|
+
: [{ id: `reuse:${n}`, label: `Reuse ${name} as it is` }]
|
|
67
|
+
})
|
|
68
|
+
const answer = await ctx.ask({
|
|
69
|
+
title: "Existing tests already cover this flow. Which way should Scribe go?",
|
|
70
|
+
detail: `${decision.why}. Extending keeps the test's steps and adds only what is missing; a new test duplicates what they cover.`,
|
|
71
|
+
lines,
|
|
72
|
+
options: [...options, { id: "new", label: "Write a new test" }, { id: "cancel", label: "Cancel" }],
|
|
73
|
+
})
|
|
74
|
+
if (!answer || answer === "cancel") return { cancelled: true }
|
|
75
|
+
if (answer === "new") return {}
|
|
76
|
+
const [mode, n] = answer.split(":") as ["reuse" | "extend", string]
|
|
77
|
+
const c = cov.candidates.filter((x) => x.share >= EXTEND_FROM)[Number(n)]!
|
|
78
|
+
say(mode === "reuse" ? `Reusing ${c.test}` : `Extending ${c.test}`, "the user chose it")
|
|
79
|
+
return { target: toTarget(mode, c) }
|
|
80
|
+
}
|
|
81
|
+
if (decision.mode === "new" || !decision.target) {
|
|
82
|
+
if (cov.candidates.length) say("Writing a new test", decision.why)
|
|
83
|
+
return {}
|
|
84
|
+
}
|
|
85
|
+
const mode: TestTarget["mode"] = decision.mode === "reuse" ? "reuse" : decision.mode === "update" ? "update" : "extend"
|
|
86
|
+
say(mode === "reuse" ? `Reusing ${decision.target.test}` : mode === "update" ? `Updating ${decision.target.test}` : `Extending ${decision.target.test}`, decision.why)
|
|
87
|
+
return { target: toTarget(mode, decision.target) }
|
|
88
|
+
}
|
|
89
|
+
|
|
30
90
|
export const TASK_FILE = "task.json"
|
|
31
91
|
|
|
32
92
|
const inCwd = (ctx: ToolContext, p: unknown) => (p ? path.resolve(ctx.cwd, String(p)) : ctx.cwd)
|
|
@@ -96,6 +156,13 @@ export const COMMAND_TOOLS: McpTool[] = [
|
|
|
96
156
|
properties: {
|
|
97
157
|
run_dir: { type: "string", description: "Run folder from browser_run or run_test_cases (default: the latest run)" },
|
|
98
158
|
repo: { type: "string", description: "Repository to write the test in, relative to the working directory (default: the working directory)" },
|
|
159
|
+
mode: {
|
|
160
|
+
type: "string",
|
|
161
|
+
enum: ["auto", "new", "extend", "reuse", "update"],
|
|
162
|
+
description:
|
|
163
|
+
"auto (default): ForgeQE measures which existing tests already cover the run, then reuses one that covers it all, extends one that covers most, or writes a new test, and asks the user when that is not clear. new, extend, reuse, update: only when the user said which (\"extend the login test\", \"write a separate test\", \"update it\" after a reused test failed).",
|
|
164
|
+
},
|
|
165
|
+
test_file: { type: "string", description: "With mode extend, reuse, or update: the existing test file (from the user, or from an earlier write_test result)" },
|
|
99
166
|
},
|
|
100
167
|
},
|
|
101
168
|
async run(args, ctx) {
|
|
@@ -108,6 +175,12 @@ export const COMMAND_TOOLS: McpTool[] = [
|
|
|
108
175
|
const t0 = Date.now()
|
|
109
176
|
const repo = inCwd(ctx, args.repo)
|
|
110
177
|
const goal = (readJson(path.join(runDir, "run.json")) as { goal?: string } | null)?.goal || ""
|
|
178
|
+
// browser_run found an earlier test for this flow and it failed: Scribe updates it rather than adding another.
|
|
179
|
+
const existingTest = readJson(path.join(runDir, EXISTING_TEST_FILE)) as ExistingTest | null
|
|
180
|
+
// Otherwise: what the repository's tests already cover of this run decides reuse, extend, or new.
|
|
181
|
+
const chosen = existingTest?.testFile ? { target: undefined } : await chooseTarget(runDir, repo, args, ctx)
|
|
182
|
+
if ("cancelled" in chosen) return { status: "cancelled", notes: "The user cancelled before Scribe started; nothing was changed." }
|
|
183
|
+
const target = chosen.target
|
|
111
184
|
ctx.emit?.({ type: "code-start", runDir, repo, goal })
|
|
112
185
|
let r
|
|
113
186
|
try {
|
|
@@ -119,6 +192,8 @@ export const COMMAND_TOOLS: McpTool[] = [
|
|
|
119
192
|
onEvent: (event) => ctx.emit?.({ type: "agent", event }),
|
|
120
193
|
open: ctx.openSession as never,
|
|
121
194
|
runTest: ctx.runTest,
|
|
195
|
+
...(existingTest?.testFile ? { existingTest } : {}),
|
|
196
|
+
...(target ? { target } : {}),
|
|
122
197
|
})
|
|
123
198
|
} catch (err) {
|
|
124
199
|
ctx.emit?.({ type: "code-done", error: (err as Error).message, ms: Date.now() - t0 })
|
|
@@ -136,6 +211,7 @@ export const COMMAND_TOOLS: McpTool[] = [
|
|
|
136
211
|
notes: r.notes || undefined,
|
|
137
212
|
browsers: r.browsers,
|
|
138
213
|
review: r.review,
|
|
214
|
+
...(r.existing ? { existing: r.existing } : {}),
|
|
139
215
|
output: r.status === "passed" ? undefined : clip(r.output, 4000),
|
|
140
216
|
}
|
|
141
217
|
},
|
package/src/mcp/tools.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { existsSync, readFileSync, writeFileSync } from "node:fs"
|
|
1
|
+
import { existsSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs"
|
|
2
2
|
import path from "node:path"
|
|
3
3
|
|
|
4
4
|
import type { AgentEvent, CodeEvent, CodeResult, ReplayMode, RunResult } from "../types.ts"
|
|
@@ -16,6 +16,7 @@ import { buildHandoff } from "../code/handoff.ts"
|
|
|
16
16
|
import { latestRunDir, runCodeAgent } from "../code/index.ts"
|
|
17
17
|
import type { OpencodeSession } from "../code/runtime.ts"
|
|
18
18
|
import { runTestCommand } from "../code/testrun.ts"
|
|
19
|
+
import { EXISTING_TEST_FILE, findWrittenTest } from "../code/notes.ts"
|
|
19
20
|
import { browserProvider, type AppConfig } from "../config.ts"
|
|
20
21
|
import { clearMemory, memoryFilesFor, memoryFlows, memoryOverview, memoryShow } from "../memory.ts"
|
|
21
22
|
import { executeRun, runDirectory } from "../run.ts"
|
|
@@ -329,6 +330,59 @@ function devicesFor(args: Record<string, unknown>, ctx: ToolContext): string[] {
|
|
|
329
330
|
return [...new Set(devices)]
|
|
330
331
|
}
|
|
331
332
|
|
|
333
|
+
/**
|
|
334
|
+
* Runs the test an earlier job wrote for this goal, if there is one. `passed`: the tool's answer, the test having
|
|
335
|
+
* passed in a browser. `failed`: the test to update, for write_test. Null when there is no such test.
|
|
336
|
+
*/
|
|
337
|
+
async function existingTestFirst(goal: string, runDir: string, ctx: ToolContext): Promise<{ passed?: Record<string, unknown>; failed?: { testFile: string; command: string; output: string; why: string } } | null> {
|
|
338
|
+
const cfg = ctx.config
|
|
339
|
+
if (!cfg.memory.enabled) return null
|
|
340
|
+
let repo = ctx.cwd
|
|
341
|
+
try {
|
|
342
|
+
repo = realpathSync(ctx.cwd)
|
|
343
|
+
} catch {}
|
|
344
|
+
const known = findWrittenTest(cfg.memory.dir, repo, goal, (f) => existsSync(path.resolve(repo, f)))
|
|
345
|
+
if (!known) return null
|
|
346
|
+
const emit = ctx.emit || (() => {})
|
|
347
|
+
emit({ type: "info", text: `This flow already has a test, ${known.test_file} (written ${known.at.slice(0, 10)}): ForgeQE runs it first` })
|
|
348
|
+
mkdirSync(runDir, { recursive: true })
|
|
349
|
+
const run = await (ctx.runTest || runTestCommand)({
|
|
350
|
+
command: known.command,
|
|
351
|
+
cwd: repo,
|
|
352
|
+
timeoutSeconds: cfg.code.testTimeoutSeconds,
|
|
353
|
+
signal: ctx.signal,
|
|
354
|
+
hide: privateEnvNames(cfg),
|
|
355
|
+
save: path.join(runDir, "existing-test"),
|
|
356
|
+
})
|
|
357
|
+
if (ctx.signal?.aborted) return null
|
|
358
|
+
const secs = Math.round(run.ms / 1000)
|
|
359
|
+
if (run.ok && run.browsers.length) {
|
|
360
|
+
emit({
|
|
361
|
+
type: "card",
|
|
362
|
+
card: {
|
|
363
|
+
status: "ok",
|
|
364
|
+
title: "Existing test passed",
|
|
365
|
+
stats: `${secs}s · no browser run or new test needed`,
|
|
366
|
+
body: [`Test ${known.test_file}`, `Command ${known.command}`, `Browser ${run.browsers.join(", ")}`, `Written ${known.at.slice(0, 10)} for the same flow`],
|
|
367
|
+
},
|
|
368
|
+
})
|
|
369
|
+
return {
|
|
370
|
+
passed: {
|
|
371
|
+
status: "passed",
|
|
372
|
+
reused_test: known.test_file,
|
|
373
|
+
written_for: known.goal,
|
|
374
|
+
command: known.command,
|
|
375
|
+
browsers: run.browsers,
|
|
376
|
+
seconds: secs,
|
|
377
|
+
note: "An earlier job wrote a test for this flow (written_for) and it passed as it is, in a real browser: no browser run and no new test were needed. If the user's request asks for something written_for does not check, call browser_run again with fresh: true; otherwise say in one or two lines that the existing test passed.",
|
|
378
|
+
},
|
|
379
|
+
}
|
|
380
|
+
}
|
|
381
|
+
const why = run.ok ? "It exits 0 but opened no browser." : `It fails now (${run.exitCode === null ? "timed out" : `exit ${run.exitCode}`}): the application or the page changed since it was written.`
|
|
382
|
+
emit({ type: "info", text: `${known.test_file} ${run.ok ? "opened no browser" : "fails now"}: ForgeQE runs the flow in the browser, and Scribe updates that test instead of writing a new one` })
|
|
383
|
+
return { failed: { testFile: known.test_file, command: known.command, output: run.output, why } }
|
|
384
|
+
}
|
|
385
|
+
|
|
332
386
|
export const TOOLS: McpTool[] = [
|
|
333
387
|
{
|
|
334
388
|
name: "browser_run",
|
|
@@ -362,6 +416,11 @@ export const TOOLS: McpTool[] = [
|
|
|
362
416
|
description:
|
|
363
417
|
"test: a user flow of the app under test, to verify and turn into an automated test. task: anything else done in a browser, such as looking up or linking work items, checking a relationship, or reading data from a tool; no test is written for it.",
|
|
364
418
|
},
|
|
419
|
+
fresh: {
|
|
420
|
+
type: "boolean",
|
|
421
|
+
description:
|
|
422
|
+
"true: run in the browser even when an earlier job wrote a test for this goal. Only when the user asks for a new browser run or a new test; by default an existing test is run first and reused when it passes.",
|
|
423
|
+
},
|
|
365
424
|
},
|
|
366
425
|
required: ["goal", "purpose"],
|
|
367
426
|
},
|
|
@@ -378,6 +437,10 @@ export const TOOLS: McpTool[] = [
|
|
|
378
437
|
if (devices.length > 1) return runGoalOnDevices(goal, url, devices, args, ctx)
|
|
379
438
|
const runDir = runDirectory(ctx.config.export.outDir)
|
|
380
439
|
const emit = ctx.emit || (() => {})
|
|
440
|
+
// The same flow asked again: an earlier job's test for it runs first. Passing in a browser, it answers the goal
|
|
441
|
+
// in seconds, with no browser run and no new test; failing, the run goes ahead and Scribe updates that test.
|
|
442
|
+
const reuse = ctx.writeTests && args.purpose !== "task" && !args.fresh && !resumeFrom && devices.length <= 1 ? await existingTestFirst(goal, runDir, ctx) : null
|
|
443
|
+
if (reuse?.passed) return reuse.passed
|
|
381
444
|
const t0 = Date.now()
|
|
382
445
|
emit({ type: "run-start", goal, url })
|
|
383
446
|
try {
|
|
@@ -406,7 +469,11 @@ export const TOOLS: McpTool[] = [
|
|
|
406
469
|
return summary
|
|
407
470
|
}
|
|
408
471
|
if (ctx.writeTests && result.status === "complete" && result.driver !== "synthetic") {
|
|
409
|
-
|
|
472
|
+
if (reuse?.failed) writeFileSync(path.join(runDir, EXISTING_TEST_FILE), `${JSON.stringify(reuse.failed, null, 2)}\n`)
|
|
473
|
+
return {
|
|
474
|
+
...summary,
|
|
475
|
+
next_step: `This session turns passed runs into tests: call write_test with run_dir ${runDir} now, before you reply.${reuse?.failed ? ` It updates the existing test ${reuse.failed.testFile} instead of writing a new one.` : ""}`,
|
|
476
|
+
}
|
|
410
477
|
}
|
|
411
478
|
return summary
|
|
412
479
|
} catch (err) {
|
package/src/types.ts
CHANGED
|
@@ -116,6 +116,8 @@ export type CodeResult = {
|
|
|
116
116
|
knownDefect?: boolean
|
|
117
117
|
/** The job did not finish: what it wrote was moved to the needs-attention folder (see src/code/attention.ts). */
|
|
118
118
|
attention?: { dir: string; files: string[]; patch?: string; readme: string }
|
|
119
|
+
/** The job worked from an existing test: ran it as it was, extended it, or updated it. */
|
|
120
|
+
existing?: "reused" | "extended" | "updated"
|
|
119
121
|
}
|
|
120
122
|
|
|
121
123
|
/** What an agent did to a file, for the shell's file list. */
|