@capagents/forgeqe 0.0.0-stage → 1.0.0

package/skills/dotnet-nunit-playwright/SKILL.md

@@ -0,0 +1,34 @@
+---
+name: dotnet-nunit-playwright
+description: "Use when the repository's browser tests are C# with Microsoft.Playwright.NUnit (PageTest or ContextTest base classes, .runsettings files): browser settings, base classes and page objects, async style and ConfigureAwait, test data and categories, and running one test with dotnet test."
+agents: [scribe]
+---
+
+# Playwright for .NET with NUnit
+
+## Read the setup first
+- The test project: the `.csproj` that references `Microsoft.Playwright.NUnit`. Note its target framework, `Nullable`, `TreatWarningsAsErrors`, analyzers, and anything set in `Directory.Build.props` and `.editorconfig`.
+- Base classes: tests inherit `PageTest` (gives `Page`, `Context`, `Browser`, and `Expect`), or `ContextTest` / `BrowserTest`. Repositories usually add their own base (for example `BaseTest : PageTest`) that signs in, sets context options, or starts tracing. Inherit the repository's base, not `PageTest` directly.
+- Page objects: classes that take `IPage`, expose `ILocator` properties, and have `async Task` methods. Extend the existing one for a page.
+
+## Browser settings
+- Launch settings live in the `.runsettings` file's `<Playwright>` section: `<BrowserName>chromium</BrowserName>` and `<LaunchOptions><Channel>chrome</Channel><Headless>true</Headless></LaunchOptions>`. It is used through `dotnet test --settings <file>` or `<RunSettingsFilePath>` in the `.csproj`. The `BROWSER` and `HEADED=1` environment variables override it.
+- Context settings (viewport, locale, time zone, base URL, storage state) go in an override of `public override BrowserNewContextOptions ContextOptions()` in the base class.
+
+## Run one test
+- Build: `dotnet build <Tests.csproj>`.
+- Install browsers once: `pwsh <Tests>/bin/Debug/<tfm>/playwright.ps1 install chromium` (use `install chrome` for the Chrome channel). Without PowerShell, install it first (`dotnet tool install --global PowerShell`).
+- Run: `dotnet test <Tests.csproj> --no-build --settings <file>.runsettings --filter "FullyQualifiedName~MyTests.CheckoutTests.PlacesOrder" --logger "console;verbosity=normal"`. To run a category, filter with `Category=Smoke`.
+
+## Code style
+- Tests are `public async Task` methods that await every call. Never use `.Result`, `.Wait()`, `Thread.Sleep`, or `Task.Delay`.
+- Use `ConfigureAwait(false)` exactly where the repository does (an analyzer such as CA2007 asks for it). Follow the analyzers so the build gains no warnings.
+- Locators: `Page.GetByRole(AriaRole.Button, new() { Name = "Save" })`, `GetByLabel`, `GetByPlaceholder`, `GetByText`, and `GetByTestId`, then CSS. Avoid XPath and position-based selectors.
+- Assert page state with retrying assertions: `await Expect(locator).ToBeVisibleAsync()`, `ToHaveTextAsync`, `ToHaveValueAsync`, and `await Expect(Page).ToHaveURLAsync(...)`. Use NUnit's `Assert.That` for plain values only, never on `await locator.IsVisibleAsync()`.
+- Wait for outcomes with `Page.WaitForURLAsync`, `Page.RunAndWaitForResponseAsync`, or an assertion.
+- Overlays that appear at random (cookie banners, surveys): register `Page.AddLocatorHandlerAsync(...)` in `[SetUp]` of the base class rather than clicking them inside a test.
+
+## Data, secrets, attributes
+- Data: `[TestCase(...)]` or `[TestCaseSource(nameof(Cases))]`, or the repository's data files and builders. Compute dates when the test runs.
+- Secrets: `Environment.GetEnvironmentVariable("SHOP_USER") ?? throw new InvalidOperationException("Set SHOP_USER")`. If the repository reads `TestContext.Parameters` from `.runsettings`, follow that, without committing secret values.
+- Match the repository's `[Category]`, `[Parallelizable]`, `[Description]`, and naming. `PageTest` gives each test its own browser context, so tests must not share state through static fields.

package/skills/dynamics-365/SKILL.md

@@ -0,0 +1,21 @@
+---
+name: dynamics-365
+description: "Use when the site is Microsoft Dynamics 365 or a Power Apps model-driven app (hosts like *.crm.dynamics.com or *.crm4.dynamics.com, URLs with main.aspx?appid=): lookup fields, choices, date fields, grids and views, the command bar, saving, business process flows, and loading indicators."
+agents: [browser]
+---
+
+# Dynamics 365 / model-driven apps
+
+- Wait for loading to end before acting: spinners, moving progress dots, and grey placeholder bars mean the form or grid is still loading. A field showing "---" is loaded and empty.
+- Fields: click the field, type the value, then press Tab so the form takes it. Required fields have a red asterisk.
+- Forms are split into tabs (for example Summary, Details, Related). A field you cannot find may be on another tab.
+- Lookup fields: type part of the name, wait for the results list under the field, and click the row whose name matches exactly (never "+ New"). Use "Search for more records" when it is not listed. Check that the field now shows the record as a link. To clear one, hover over it and click its X.
+- Choice (option set) fields open a list; pick the option by its text. Yes/No fields toggle when clicked.
+- Date fields: type the date in the format the field already shows or its placeholder (for example 6/15/2026), then press Tab. Use the calendar button only when typing does not stick. The time is a separate box next to the date.
+- Save with the command bar's Save or Ctrl+S. Wait until "Saving..." disappears and the title no longer says "Unsaved"; the form reloads after a save, so look again before the next step. "Save & Close" returns to the list.
+- A red banner at the top of the form lists what blocked the save (usually required fields left empty). Fill those and save again.
+- Command bar buttons that do not fit are under "More commands" (⋯). Open it when a button is missing.
+- Lists and subgrids: open a record with the link in its primary name column; select rows with the round check at the left of the row. "Search this view" filters the current view; press Enter after typing. Change views from the view name's dropdown above the grid. Sort and filter from a column header's menu.
+- Business process flow (the stage bar under the header): click a stage to open its fields; "Next Stage" moves on once its required fields are filled.
+- Leaving a form with changes opens an "Unsaved changes" dialog: choose "Save and continue" or "Discard changes" as the goal needs.
+- Go back with the form's back arrow, not the browser's Back button.

package/skills/playwright-typescript/SKILL.md

@@ -0,0 +1,35 @@
+---
+name: playwright-typescript
+description: "Use when the repository's browser tests are Playwright Test in TypeScript or JavaScript (@playwright/test, playwright.config.ts): where browser settings go, fixtures and page objects, sign-in state, locators and waiting, pop-up overlays, and the command that runs one test."
+agents: [scribe]
+---
+
+# Playwright Test (TypeScript / JavaScript)
+
+## Read the setup first
+- Config: `playwright.config.ts` (or `.js`, `.mjs`; a repo can have several, run with `--config`). Note `testDir`, `projects` and their `dependencies`, `use` (baseURL, channel, headless, viewport, locale, timezoneId, storageState, testIdAttribute), `webServer`, `globalSetup`, `timeout`, `expect.timeout`, `retries`.
+- Fixtures: files that call `test.extend<...>()` (often `fixtures.ts` or `tests/fixtures/`). When tests import `test` and `expect` from such a file, the new test imports from it too, never straight from `@playwright/test`.
+- Page objects: classes that take `page: Page` in the constructor and expose locators and actions. Extend the existing one for a page; when the repo hands page objects out as fixtures, add yours the same way.
+- Sign-in: a setup project (`*.setup.ts`, `dependencies: ['setup']`) that saves `storageState`. Reuse it; sign in inside the test only when sign-in is what the test checks.
+
+## Run one test
+- `npx playwright test tests/checkout.spec.ts --project=chromium --reporter=line`, plus `-g "title"` for one test in a file. Use the repo's package manager when its lockfile says so (`pnpm exec playwright`, `yarn playwright`). Add `--workers=1` while debugging.
+- Missing browsers: `npx playwright install chromium`; `channel: 'chrome'` needs `npx playwright install chrome`; on Linux CI add `--with-deps`.
+- Never commit `test.only`, `--headed`, `--debug`, or `page.pause()`.
+
+## Browser settings
+- Put them in `use` of the config, at the top level or in the project the test runs in: `channel`, `headless`, `viewport`, `locale`, `timezoneId`, and `userAgent` only when the site needs it. Use `test.use()` in a spec file only if the repo already overrides settings per file.
+
+## Locators and assertions
+- Prefer `getByRole(role, { name })`, `getByLabel`, `getByPlaceholder`, `getByText`, and `getByTestId` (check `testIdAttribute`), then CSS. Avoid XPath and position-based selectors (`nth`, `:nth-child`); narrow with `.filter({ hasText })` or a parent locator instead.
+- A locator that matches several elements throws on click or fill (strict mode). Make it unique; use `.first()` only when any match is fine.
+- Assert with retrying web-first assertions: `await expect(locator).toBeVisible()`, `toHaveText`, `toHaveValue`, `toHaveURL`, `toHaveCount`. Never `expect(await locator.isVisible()).toBe(true)`.
+- Wait for outcomes, never time: no `page.waitForTimeout`. Use an assertion, `page.waitForURL`, or a `page.waitForResponse(...)` promise created before the click that triggers the request.
+
+## Overlays that come and go
+- Cookie banners, chat prompts, and surveys that appear at random: register `page.addLocatorHandler(page.getByRole('button', { name: 'Accept all' }), async (button) => { await button.click() })` in a fixture or `beforeEach`, instead of clicking them inside the test.
+
+## Data and secrets
+- Read secrets with `process.env.NAME`, and fail with a clear message when one is missing. If the config loads `dotenv`, add new names to `.env.example` without values.
+- Compute dates such as "tomorrow" when the test runs. Run one test over several inputs with a `for (const c of cases) test(\`... ${c.name}\`, ...)` loop or the repo's data files.
+- Tag tests the repo's way: `test('title', { tag: '@smoke' }, ...)` or tags in the title.

package/skills/writing-skills/SKILL.md

@@ -0,0 +1,30 @@
+---
+name: writing-skills
+description: "Use when the user asks to create, change, or review a ForgeQE skill (a SKILL.md under .forgeqe/skills or ~/.forgeqe/skills), for example \"make a skill for our date picker\" or \"teach Scribe our test conventions\": choosing who uses it, a description agents choose by, and short concrete instructions."
+agents: [forgeqe]
+---
+
+# Writing a ForgeQE skill
+
+A skill is a folder `<name>/SKILL.md`: YAML frontmatter with `name` (lowercase words joined by hyphens, the same as the folder), `description`, and `agents`, then Markdown instructions.
+
+## Who uses it (agents)
+- `scribe`: how tests are written in a repository (framework, folders, base classes, page objects, the command that runs one test). Scribe can also read files placed next to SKILL.md, such as `references/locators.md`, when SKILL.md names them by relative path.
+- `browser`: how to drive one site or widget during browser runs (lookups, date pickers, grids, save buttons). The browser agent sees only the SKILL.md text, so put everything it needs there and keep it under about 1,500 characters; all chosen skills together are capped by skills.browser_max_chars.
+- `forgeqe`: how to handle a kind of request (which sources, tools, and order).
+- With no `agents`, a skill is for forgeqe and scribe. Browser skills must say `agents: [browser]`.
+
+## The description
+- Agents choose skills by the description alone, so name the concrete signs it applies: package and framework names, config files, URL hosts, product names, the kind of task.
+- Good: "Use when the site is the Contoso ERP (erp.contoso.com): the order grid, the customer lookup, and saving orders." Weak: "Tips for testing."
+
+## The instructions
+- Short imperative rules, one per line, with real names: labels on screen, file paths, class names, commands.
+- Say what to do, not background. Don't repeat ForgeQE's own rules; a skill adds to them and never overrides them.
+- No secrets or passwords: name the environment variables instead.
+- Gather the facts first (read the repository's config and tests, or ask the user what the widget does). Don't invent them.
+
+## Creating it
+- Use forgeqe_setup with action skills, op new: name, description, agents, and body (the instructions). It goes in this project's .forgeqe/skills; set user to true only when the user wants it for every project.
+- Then tell the user where it is, that it applies from the next session or run, and that the project skill should be committed. `forgeqe skills check` validates every skill.
+- To change a built-in skill, create a project skill with the same name; it replaces the built-in one. `forgeqe skills disable <name>` turns a skill off for this project.

package/src/api/plan.ts

@@ -0,0 +1,246 @@
+import type { AppConfig } from "../config.ts"
+import { privateEnvNames } from "../childEnv.ts"
+import type { TestCase } from "../cases/types.ts"
+import { checkCase } from "../cases/quality.ts"
+import type { CodeEvent, Usage } from "../types.ts"
+import { openSession, type AgentSpec, type OpencodeSession } from "../code/runtime.ts"
+import { scenarioProblems, slug } from "./scenario.ts"
+import { knowledgeBlock, readKnowledge, relevantKnowledge } from "../knowledge.ts"
+import type { Contract, ContractSet, KeyField, Scenario, ScenarioStep } from "./types.ts"
+import { prompt } from "../prompts.ts"
+import { domainNotes } from "../domain.ts"
+
+/**
+ * Scenarios planned by a model from test cases (Azure DevOps, Jira, Xray, TestRail, files) or plain English, against
+ * the operations a contract set has (from an API description or a browser run). The planner only picks operations
+ * from that list, in order, with the data, the values to carry from one answer to the next, and the checks the
+ * expected results ask for; it changes nothing and calls nothing. ForgeQE checks the plan (every operation exists,
+ * every variable is saved before it is used) and shows it before anything runs.
+ */
+
+export const planAgentPrompt = () => prompt("api-scenarios")
+
+export const PLAN_AGENT: AgentSpec = {
+  name: "forgeqe-api-plan",
+  description: "Plans end-to-end API scenarios from test cases or plain English, against an API's operations",
+  get prompt() {
+    return planAgentPrompt()
+  },
+  permission: { edit: "deny", bash: "deny", webfetch: "deny", task: "deny", "live_*": "deny" },
+}
+
+const pair = { type: "object", properties: { name: { type: "string" }, value: { type: "string" } }, required: ["name", "value"] }
+const PLAN_STEP = {
+  type: "object",
+  properties: {
+    name: { type: "string", description: "What the step does, in a few words" },
+    operation: { type: "string", description: "The operation's id from the list, exactly" },
+    path: { type: "string", description: "The path to call, with real values or {{variables}}" },
+    query: { type: "array", items: pair },
+    headers: { type: "array", items: pair, description: "Only headers the operation needs besides credentials (ForgeQE adds those)" },
+    body_json: { type: "string", description: "The request body as JSON text, with {{variables}} inside strings; empty for none" },
+    save: { type: "array", items: { type: "object", properties: { variable: { type: "string" }, path: { type: "string" } }, required: ["variable", "path"] } },
+    expect_status: { type: "integer" },
+    expect_status_any: { type: "array", items: { type: "integer" }, description: "For an expected refusal: every acceptable status" },
+    expect_fields: { type: "array", items: { type: "object", properties: { path: { type: "string" }, equals_json: { type: "string" }, type: { type: "string" } }, required: ["path"] } },
+    no_auth: { type: "boolean" },
+  },
+  required: ["name", "operation", "path"],
+}
+
+export const PLAN_SCHEMA = {
+  type: "object",
+  properties: {
+    scenarios: {
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          name: { type: "string" },
+          case_id: { type: "string", description: "The test case it automates, when it comes from one" },
+          steps: { type: "array", items: PLAN_STEP },
+          cleanup: { type: "array", items: PLAN_STEP },
+          notes: { type: "string" },
+        },
+        required: ["name", "steps"],
+      },
+    },
+    unmatched: { type: "array", items: { type: "string" }, description: "Steps no operation does, with the scenario they belong to" },
+  },
+  required: ["scenarios"],
+} as const
+
+type PlanStep = {
+  name: string
+  operation: string
+  path: string
+  query?: { name: string; value: string }[]
+  headers?: { name: string; value: string }[]
+  body_json?: string
+  save?: { variable: string; path: string }[]
+  expect_status?: number
+  expect_status_any?: number[]
+  expect_fields?: { path: string; equals_json?: string; type?: string }[]
+  no_auth?: boolean
+}
+export type PlanReport = { scenarios: { name: string; case_id?: string; steps: PlanStep[]; cleanup?: PlanStep[]; notes?: string }[]; unmatched?: string[] }
+
+const clip = (t: string, n: number) => (t.length > n ? `${t.slice(0, n - 1)}…` : t)
+
+/** The operations as short lines: what the planner needs, without full schemas. */
+export function catalog(set: ContractSet): string {
+  return set.contracts
+    .map((c) => {
+      const call = `${c.method} ${c.example.path}${c.example.query ? `?${new URLSearchParams(c.example.query)}` : ""}`
+      const body = c.example.body === undefined ? "" : `\n    body: ${clip(typeof c.example.body === "string" ? c.example.body : JSON.stringify(c.example.body), 400)}`
+      const fields = c.expect.fields?.length ? `\n    answer: ${c.expect.fields.map((f) => f.path).join(", ")}` : ""
+      const top = c.expect.schema?.properties ? `\n    answer fields: ${Object.keys(c.expect.schema.properties).slice(0, 15).join(", ")}` : ""
+      const flags = [c.writes ? "changes data" : "", c.auth ? `auth ${c.auth.type}` : "", `→ ${c.expect.status}`].filter(Boolean).join(", ")
+      return `- ${c.id}${c.name !== c.id ? ` (${clip(c.name, 80)})` : ""}: ${call} [${flags}]${body}${fields || top}`
+    })
+    .join("\n")
+}
+
+function caseText(tc: TestCase): string {
+  return [
+    `### ${tc.id}: ${tc.title}`,
+    tc.description ? clip(tc.description, 1500) : "",
+    tc.preconditions?.length ? `Preconditions: ${tc.preconditions.join("; ")}` : "",
+    ...tc.steps.map((s, i) => `${i + 1}. ${s.action}${s.data ? ` (data: ${s.data})` : ""}${s.expected ? `\n   Expected: ${s.expected}` : ""}`),
+  ]
+    .filter(Boolean)
+    .join("\n")
+}
+
+export function planPrompt(set: ContractSet, input: { cases?: TestCase[]; text?: string }, knowledge = ""): string {
+  return [
+    ...(knowledge ? [knowledge, ""] : []),
+    `The API: ${set.name}${set.baseUrl ? ` (${set.baseUrl})` : ""}. Its operations:`,
+    "",
+    catalog(set),
+    "",
+    input.cases?.length ? `Plan one API scenario for each of these test cases:\n\n${input.cases.map(caseText).join("\n\n")}` : `Plan the API scenario for: ${input.text}`,
+  ].join("\n")
+}
+
+function json(text: string | undefined): unknown {
+  if (!text?.trim()) return undefined
+  try {
+    return JSON.parse(text)
+  } catch {
+    return text
+  }
+}
+
+function toStep(p: PlanStep, ops: Map<string, Contract>, notes: string[]): ScenarioStep | null {
+  const op = ops.get(p.operation) || [...ops.values()].find((c) => `${c.method} ${c.path}` === p.operation)
+  if (!op) {
+    notes.push(`${p.name}: no operation "${p.operation}" in the set, left out`)
+    return null
+  }
+  const status = p.expect_status ?? p.expect_status_any?.[0] ?? op.expect.status
+  const fields: KeyField[] = (p.expect_fields || []).map((f) => ({ path: f.path, ...(f.type ? { type: f.type } : {}), ...(f.equals_json !== undefined && f.equals_json !== "" ? { equals: json(f.equals_json) } : {}) }))
+  const sameAsOp = status === op.expect.status && !p.expect_status_any?.length
+  const expect: Contract["expect"] = {
+    status,
+    ...(p.expect_status_any?.length ? { statusIn: p.expect_status_any } : {}),
+    ...(sameAsOp && op.expect.contentType ? { contentType: op.expect.contentType } : {}),
+    ...(sameAsOp && op.expect.schema ? { schema: op.expect.schema } : {}),
+    ...(fields.length ? { fields } : sameAsOp && op.expect.fields?.length ? { fields: op.expect.fields } : {}),
+    ...(op.expect.elements ? { elements: op.expect.elements } : {}),
+  }
+  const pairs = (xs?: { name: string; value: string }[]) => (xs?.length ? Object.fromEntries(xs.map((x) => [x.name, x.value])) : undefined)
+  const body = json(p.body_json)
+  const query = pairs(p.query)
+  const headers = { ...(op.example.headers || {}), ...(pairs(p.headers) || {}) }
+  return {
+    name: p.name,
+    operation: op.id,
+    protocol: op.protocol,
+    method: op.method,
+    path: p.path || op.example.path,
+    ...(query ? { query } : {}),
+    ...(Object.keys(headers).length ? { headers } : {}),
+    ...(body !== undefined ? { body } : op.protocol === "graphql" || op.protocol === "soap" ? { body: op.example.body } : {}),
+    ...(p.save?.length ? { save: Object.fromEntries(p.save.map((s) => [s.variable, s.path])) } : {}),
+    expect,
+    ...(p.no_auth ? { noAuth: true } : {}),
+  }
+}
+
+/** The planner's answer as scenarios ForgeQE can run, with what it had to leave out or could not check. */
+export function toScenarios(report: PlanReport, set: ContractSet, input: { cases?: TestCase[]; text?: string }): Scenario[] {
+  const ops = new Map(set.contracts.map((c) => [c.id, c]))
+  return report.scenarios.map((p) => {
+    const notes: string[] = []
+    const steps = p.steps.map((s) => toStep(s, ops, notes)).filter((s): s is ScenarioStep => Boolean(s))
+    const cleanup = (p.cleanup || []).map((s) => toStep(s, ops, notes)).filter((s): s is ScenarioStep => Boolean(s))
+    const tc = input.cases?.find((c) => c.id === p.case_id) || (input.cases?.length === 1 ? input.cases[0] : undefined)
+    const s: Scenario = {
+      id: slug(tc ? `${tc.id} ${p.name}` : p.name),
+      name: p.name,
+      kind: "planned",
+      source: tc ? { kind: "case", ref: tc.id } : { kind: "text", ref: clip(input.text || "", 200) },
+      ...(tc ? { caseId: tc.id, caseTitle: tc.title, caseSource: tc.source, ...(tc.url ? { caseUrl: tc.url } : {}), ...(tc.ref ? { caseRef: tc.ref } : {}) } : {}),
+      steps,
+      ...(cleanup.length ? { cleanup } : {}),
+      writes: steps.some((st) => ops.get(st.operation || "")?.writes || !["GET", "HEAD", "OPTIONS"].includes(st.method)),
+    }
+    notes.push(...scenarioProblems(s))
+    if (p.notes) notes.push(p.notes)
+    if (notes.length) s.notes = notes
+    return s
+  })
+}
+
+export type PlanResult = { scenarios: Scenario[]; unmatched: string[]; notRunnable: { caseId: string; reason: string }[]; usage: Usage; model: string }
+
+/** Plan scenarios with the model. Test cases a rule check finds nothing to test in are left out, with the reason. */
+export async function planScenarios(opts: {
+  config: AppConfig
+  set: ContractSet
+  cases?: TestCase[]
+  text?: string
+  cwd: string
+  signal?: AbortSignal
+  onEvent?: (e: CodeEvent) => void
+  open?: (o: Parameters<typeof openSession>[0]) => Promise<OpencodeSession<PlanReport>>
+}): Promise<PlanResult> {
+  if (!opts.set.contracts.length) throw new Error("The contract set has no operations to plan with: make contracts from an API description or a browser run first.")
+  const notRunnable: PlanResult["notRunnable"] = []
+  const cases = (opts.cases || []).filter((tc) => {
+    const blocker = checkCase(tc).find((f) => f.severity === "blocker")
+    if (blocker) notRunnable.push({ caseId: tc.id, reason: blocker.message })
+    return !blocker
+  })
+  if (opts.cases?.length && !cases.length) return { scenarios: [], unmatched: [], notRunnable, usage: { input: 0, output: 0 }, model: "" }
+  if (!cases.length && !opts.text?.trim()) throw new Error("Give test cases or a scenario in plain English.")
+  const input = cases.length ? { cases } : { text: opts.text }
+  const open = (opts.open || openSession) as (o: Parameters<typeof openSession>[0]) => Promise<OpencodeSession<PlanReport>>
+  const session = await open({
+    repo: opts.cwd,
+    profile: opts.config.llm,
+    domain: domainNotes(opts.config),
+    model: opts.config.code.model,
+    bin: opts.config.code.opencode,
+    readDirs: [opts.cwd],
+    hide: privateEnvNames(opts.config),
+    signal: opts.signal,
+    onEvent: opts.onEvent,
+    agent: PLAN_AGENT,
+    schema: PLAN_SCHEMA as unknown as Record<string, unknown>,
+    title: "ForgeQE API scenarios",
+  })
+  try {
+    opts.onEvent?.({ type: "code", kind: "info", text: `Planning API scenarios (${session.model}) against ${opts.set.contracts.length} operations` })
+    const k = opts.config.knowledge
+    const task = [opts.text || "", ...(input.cases || []).map((c) => `${c.title} ${c.description || ""} ${c.steps.map((s) => `${s.action} ${s.expected || ""}`).join(" ")}`)].join("\n")
+    const knowledge = k ? knowledgeBlock(relevantKnowledge(readKnowledge(k.dir), task, 5000)) : ""
+    let reply = await session.prompt(planPrompt(opts.set, input, knowledge))
+    if (!reply.report && !reply.error && !opts.signal?.aborted) reply = await session.prompt("Give the structured result: scenarios and unmatched.")
+    if (!reply.report) throw new Error(`No scenarios were planned${reply.error ? `: ${reply.error}` : "."}`)
+    return { scenarios: toScenarios(reply.report, opts.set, input), unmatched: reply.report.unmatched || [], notRunnable, usage: { ...session.usage }, model: session.model }
+  } finally {
+    await session.close()
+  }
+}

package/src/api/run.ts

@@ -0,0 +1,204 @@
+import { mkdirSync, writeFileSync } from "node:fs"
+import path from "node:path"
+
+import { checkFields, validate } from "./schema.ts"
+import { grpcTarget, runGrpcurl, type Fetcher } from "./spec.ts"
+import { soapOperation, xmlElements } from "./traffic.ts"
+import type { CheckResult, Contract, ContractResult, ContractRun, ContractSet } from "./types.ts"
+
+/**
+ * Run contracts straight against the API, without a browser or the coding agent: each example call is made and its
+ * answer checked for the status, content type, schema, and key fields (GraphQL: no errors; SOAP: no fault and the
+ * expected elements). Calls that change data are left out unless allowed, and so are calls the description gave no
+ * real values for. Credentials come from api.headers in forgeqe.yaml, with ${NAME} read from the environment; they are
+ * sent, never written to the results.
+ */
+
+export type RunOptions = {
+  baseUrl?: string
+  headers?: Record<string, string>
+  allowWrites?: boolean
+  /** Run contracts whose example has made-up values too. */
+  includeNeedsData?: boolean
+  only?: string[]
+  timeoutSeconds?: number
+  signal?: AbortSignal
+  fetch?: Fetcher
+  grpcurl?: typeof runGrpcurl
+  dir: string
+  onResult?: (r: ContractResult, c: Contract) => void
+}
+
+/** Header values with ${NAME} read from the environment; the names of variables not set, so the caller can say. */
+export function expandHeaders(headers: Record<string, string> = {}, env: Record<string, string | undefined> = process.env): { headers: Record<string, string>; missing: string[] } {
+  const missing: string[] = []
+  const out: Record<string, string> = {}
+  for (const [k, v] of Object.entries(headers)) {
+    let gap = false
+    const value = v.replace(/\$\{(\w+)\}|\$(\w+)/g, (_, a, b) => {
+      const name = a || b
+      const got = env[name]
+      if (got === undefined || got === "") {
+        missing.push(name)
+        gap = true
+        return ""
+      }
+      return got
+    })
+    if (!gap) out[k] = value
+  }
+  return { headers: out, missing }
+}
+
+const hasHeader = (h: Record<string, string>, name: string) => Object.keys(h).some((k) => k.toLowerCase() === name.toLowerCase())
+
+function urlFor(c: Contract, base: string): string {
+  const p = c.example.path
+  const u = new URL(/^https?:\/\//.test(p) ? p : `${base.replace(/\/+$/, "")}${p.startsWith("/") ? "" : "/"}${p}`)
+  for (const [k, v] of Object.entries(c.example.query || {})) u.searchParams.set(k, v)
+  return u.href
+}
+
+const sameType = (got: string, want: string) => {
+  const g = got.split(";")[0].trim().toLowerCase()
+  const w = want.split(";")[0].trim().toLowerCase()
+  return g === w || (g.includes("json") && w.includes("json")) || (g.includes("xml") && w.includes("xml"))
+}
+
+type Checked = { result: ContractResult; body?: unknown }
+
+const wanted = (e: Contract["expect"]) => (e.statusIn?.length ? e.statusIn.join(" or ") : String(e.status))
+const statusMatches = (e: Contract["expect"], got: number) => (e.statusIn?.length ? e.statusIn.includes(got) : got === e.status)
+
+async function httpCheck(c: Contract, base: string, headers: Record<string, string>, opts: Pick<RunOptions, "fetch" | "timeoutSeconds" | "signal">): Promise<Checked> {
+  const started = Date.now()
+  const body = c.example.body === undefined ? undefined : typeof c.example.body === "string" ? c.example.body : JSON.stringify(c.example.body)
+  const sendHeaders: Record<string, string> = { Accept: c.protocol === "soap" ? "text/xml, application/soap+xml" : "application/json, */*", ...(c.example.headers || {}), ...headers }
+  if (body !== undefined && !hasHeader(sendHeaders, "content-type")) sendHeaders["Content-Type"] = "application/json"
+  const timeout = AbortSignal.timeout((opts.timeoutSeconds || 30) * 1000)
+  const res = await (opts.fetch || fetch)(urlFor(c, base), { method: c.method, headers: sendHeaders, body, signal: opts.signal ? AbortSignal.any([opts.signal, timeout]) : timeout, redirect: "manual" })
+  const text = await res.text()
+  const ms = Date.now() - started
+  const checks: CheckResult[] = []
+  const statusOk = statusMatches(c.expect, res.status)
+  checks.push({ name: "status", ok: statusOk, detail: statusOk ? String(res.status) : `got ${res.status}, expected ${wanted(c.expect)}${res.status === 401 || res.status === 403 ? authHint(c, headers) : ""}` })
+  const type = res.headers.get("content-type") || ""
+  let parsed: unknown
+  try {
+    parsed = text ? JSON.parse(text) : undefined
+  } catch {}
+  // A negative case checks the status only: an error answer has its own shape.
+  const negative = Boolean(c.expect.statusIn?.length)
+  if (negative) return { result: { id: c.id, status: statusOk ? "passed" : "failed", ms, httpStatus: res.status, checks }, body: parsed }
+  if (c.expect.contentType && text) checks.push({ name: "content type", ok: sameType(type, c.expect.contentType), detail: type || "none" })
+  if (c.protocol === "soap") {
+    const fault = /<(?:\w+:)?Fault[\s>]/.test(text)
+    checks.push({ name: "no SOAP fault", ok: !fault, ...(fault ? { detail: (text.match(/<(?:\w+:)?(faultstring|Text)[^>]*>([^<]*)</) || [])[2] || "Fault" } : {}) })
+    if (c.expect.elements?.length) {
+      const got = new Set(xmlElements(text, 200))
+      const missing = c.expect.elements.filter((e) => !got.has(e) && soapOperation(text) !== e)
+      checks.push({ name: "elements", ok: !missing.length, detail: missing.length ? `missing ${missing.join(", ")}` : c.expect.elements.join(", ") })
+    }
+  } else if (c.expect.schema || c.expect.fields?.length || c.protocol === "graphql") {
+    const json = parsed
+    if (json === undefined) checks.push({ name: "JSON body", ok: false, detail: text ? `not JSON: ${text.slice(0, 80)}` : "empty body" })
+    if (json !== undefined) {
+      if (c.protocol === "graphql") {
+        const errors = (json as { errors?: { message?: string }[] }).errors
+        checks.push({ name: "no GraphQL errors", ok: !errors?.length, ...(errors?.length ? { detail: errors.map((e) => e.message).slice(0, 3).join("; ") } : {}) })
+      }
+      if (c.expect.schema) {
+        const problems = validate(json, c.expect.schema)
+        checks.push({ name: "schema", ok: !problems.length, ...(problems.length ? { detail: problems.slice(0, 5).join("; ") } : {}) })
+      }
+      if (c.expect.fields?.length) {
+        const problems = checkFields(json, c.expect.fields)
+        checks.push({ name: "key fields", ok: !problems.length, detail: problems.length ? problems.slice(0, 5).join("; ") : c.expect.fields.map((f) => f.path).join(", ") })
+      }
+    }
+  }
+  return { result: { id: c.id, status: checks.every((x) => x.ok) ? "passed" : "failed", ms, httpStatus: res.status, checks }, body: parsed ?? text }
+}
+
+function authHint(c: Contract, headers: Record<string, string>): string {
+  if (!c.auth) return ""
+  const name = c.auth.type === "header" ? c.auth.name : c.auth.type === "cookie" ? "Cookie" : "Authorization"
+  return name && !hasHeader(headers, name) ? ` (the call needs ${c.auth.type === "header" ? `the ${name} header` : c.auth.type === "cookie" ? "a session cookie" : `${c.auth.type} auth`}: set it under api.headers in forgeqe.yaml, e.g. ${name}: ${c.auth.type === "basic" ? "Basic ${API_BASIC}" : c.auth.type === "header" ? "${API_KEY}" : c.auth.type === "cookie" ? "${API_COOKIE}" : "Bearer ${API_TOKEN}"})` : ""
+}
+
+const GRPC_CODES = ["OK", "Canceled", "Unknown", "InvalidArgument", "DeadlineExceeded", "NotFound", "AlreadyExists", "PermissionDenied", "ResourceExhausted", "FailedPrecondition", "Aborted", "OutOfRange", "Unimplemented", "Internal", "Unavailable", "DataLoss", "Unauthenticated"]
+
+async function grpcCheck(c: Contract, base: string, headers: Record<string, string>, opts: Pick<RunOptions, "grpcurl" | "timeoutSeconds">): Promise<Checked> {
+  const { address, plaintext } = grpcTarget(base)
+  const started = Date.now()
+  const args = [...(plaintext ? ["-plaintext"] : []), "-max-time", String(opts.timeoutSeconds || 30), "-format", "json", ...Object.entries(headers).flatMap(([k, v]) => ["-H", `${k}: ${v}`]), "-d", "@", address, c.path.replace(/^\//, "")]
+  const r = await (opts.grpcurl || runGrpcurl)(args, JSON.stringify(c.example.body ?? {}))
+  const ms = Date.now() - started
+  const code = r.ok ? 0 : Math.max(0, GRPC_CODES.indexOf((r.out.match(/Code:\s*(\w+)/) || [])[1] || "Unknown"))
+  const ok = statusMatches(c.expect, code)
+  const checks: CheckResult[] = [{ name: "status", ok, detail: ok ? GRPC_CODES[code] : `got ${GRPC_CODES[code]}, expected ${(c.expect.statusIn?.length ? c.expect.statusIn : [c.expect.status]).map((n) => GRPC_CODES[n] || n).join(" or ")}${r.ok ? "" : `: ${(r.out.match(/Message:\s*(.*)/) || [])[1] || r.out.trim().split("\n")[0]}`}` }]
+  let body: unknown
+  if (r.ok) {
+    try {
+      body = JSON.parse(r.out || "{}")
+    } catch {}
+  }
+  if (r.ok && c.expect.schema) {
+    const problems = body === undefined ? ["the answer was not JSON"] : validate(body, c.expect.schema)
+    checks.push({ name: "schema", ok: !problems.length, ...(problems.length ? { detail: problems.join("; ") } : {}) })
+  }
+  return { result: { id: c.id, status: checks.every((x) => x.ok) ? "passed" : "failed", ms, checks }, body }
+}
+
+/** Make one call and check its answer; the parsed answer comes back for scenarios to take values from. */
+export async function callContract(c: Contract, base: string, headers: Record<string, string>, opts: Pick<RunOptions, "fetch" | "grpcurl" | "timeoutSeconds" | "signal">): Promise<Checked> {
+  return c.protocol === "grpc" ? grpcCheck(c, base, headers, opts) : httpCheck(c, base, headers, opts)
+}
+
+export async function runContracts(set: ContractSet, opts: RunOptions): Promise<ContractRun> {
+  const base = (opts.baseUrl || set.baseUrl || "").replace(/\/+$/, "")
+  const headers = opts.headers || {}
+  const only = (opts.only || []).map((s) => s.toLowerCase()).filter(Boolean)
+  const results: ContractResult[] = []
+  for (const c of set.contracts) {
+    if (only.length && !only.some((o) => c.id.toLowerCase().includes(o) || c.name.toLowerCase().includes(o))) continue
+    if (opts.signal?.aborted) break
+    let r: ContractResult
+    if (c.writes && !opts.allowWrites) r = { id: c.id, status: "skipped", checks: [], reason: "changes data: run with allow_writes on a test environment" }
+    else if (c.needsData?.length && !opts.includeNeedsData) r = { id: c.id, status: "skipped", checks: [], reason: `needs real values for ${c.needsData.join(", ")}: set them in the example in contracts.json` }
+    else if (!base && !/^https?:\/\//.test(c.example.path)) r = { id: c.id, status: "skipped", checks: [], reason: "no base URL: pass base_url or set api.base_url" }
+    else {
+      try {
+        r = (await callContract(c, base, headers, opts)).result
+      } catch (err) {
+        r = { id: c.id, status: "error", checks: [], reason: err instanceof Error ? (err.name === "TimeoutError" ? `no answer in ${opts.timeoutSeconds || 30}s` : err.message) : String(err) }
+      }
+    }
+    results.push(r)
+    opts.onResult?.(r, c)
+  }
+  const counts = { passed: 0, failed: 0, skipped: 0, error: 0 }
+  for (const r of results) counts[r.status]++
+  const run: ContractRun = { set: set.id, baseUrl: base, at: new Date().toISOString(), results, counts, dir: opts.dir }
+  mkdirSync(opts.dir, { recursive: true })
+  writeFileSync(path.join(opts.dir, "results.json"), JSON.stringify(run, null, 2))
+  writeFileSync(path.join(opts.dir, "junit.xml"), junit(set, run))
+  return run
+}
+
+const esc = (s: string) => s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;").replace(/"/g, "&quot;")
+
+export const xmlEscape = esc
+
+export function junit(set: ContractSet, run: ContractRun): string {
+  const cases = run.results.map((r) => {
+    const head = `  <testcase classname="${esc(set.name)}" name="${esc(r.id)}" time="${((r.ms || 0) / 1000).toFixed(3)}"`
+    if (r.status === "passed") return `${head}/>`
+    if (r.status === "skipped") return `${head}><skipped message="${esc(r.reason || "")}"/></testcase>`
+    const failed = r.checks.filter((c) => !c.ok)
+    const msg = r.reason || failed.map((c) => `${c.name}: ${c.detail || "failed"}`).join("; ")
+    return `${head}><${r.status === "error" ? "error" : "failure"} message="${esc(msg)}"/></testcase>`
+  })
+  const c = run.counts
+  return `<?xml version="1.0" encoding="UTF-8"?>\n<testsuite name="${esc(set.name)}" tests="${run.results.length}" failures="${c.failed}" errors="${c.error}" skipped="${c.skipped}">\n${cases.join("\n")}\n</testsuite>\n`
+}