@capagents/forgeqe 2.3.4 → 2.3.5

package/docs/user-guide.html

@@ -389,7 +389,7 @@ forgeqe docs --copy ./docs   # copy the guide, the architecture page, and the re
 <div class="area"><a class="area-name" href="#area-setup"><span class="mono">Setup</span></a><ul class="tiles"><li><a class="tile" href="#cap-test-apps"><b>Test environments and their apps</b><span>Place an order in the shop, then check its invoice in Billing, in UAT</span></a></li><li><a class="tile" href="#cap-generated-code"><b>What ForgeQE wrote</b><span>How much of the code ForgeQE wrote is still there?</span></a></li><li><a class="tile" href="#cap-forgeqe-commands"><b>Every forgeqe command from the shell</b><span>Show me the context budget</span></a></li><li><a class="tile" href="#cap-setup-help"><b>Setup, skills, and documentation</b><span>What can you do?</span></a></li></ul></div>
 <div class="area"><a class="area-name" href="#area-desktop"><span class="mono">Desktop</span></a><ul class="tiles"><li><a class="tile" href="#cap-desktop-apps"><b>Automate Windows desktop apps</b><span>Create an invoice for Contoso in the Billing app and save it as a test</span></a></li></ul></div>
 <div class="area"><a class="area-name" href="#area-mobile"><span class="mono">Mobile</span></a><ul class="tiles"><li><a class="tile" href="#cap-mobile-apps"><b>Automate Android and iOS apps</b><span>Sign in to the courier app and accept the first delivery, and save it as a test</span></a></li></ul></div>
-<div class="area"><a class="area-name" href="#area-operations"><span class="mono">Operations</span></a><ul class="tiles"><li><a class="tile" href="#cap-containers"><b>Services, mocks, workspace and browser in Docker or Podman</b><span>Start the database and the queue in Docker</span></a></li><li><a class="tile" href="#cap-monitoring"><b>Scheduled monitoring with alerts</b><span>Check the login flow every 15 minutes and alert Teams</span></a></li><li><a class="tile" href="#cap-notifications"><b>Teams and email notifications</b><span>Post the results to the QA channel</span></a></li><li><a class="tile" href="#cap-pipelines"><b>CI pipelines</b><span>Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make it green</span></a></li></ul></div>
+<div class="area"><a class="area-name" href="#area-operations"><span class="mono">Operations</span></a><ul class="tiles"><li><a class="tile" href="#cap-containers"><b>Services, mocks, workspace and browser in Docker or Podman</b><span>Start the database and the queue in Docker</span></a></li><li><a class="tile" href="#cap-monitoring"><b>Scheduled monitoring with alerts</b><span>Check the login flow every 15 minutes and alert Teams</span></a></li><li><a class="tile" href="#cap-jobs"><b>Background jobs on a schedule</b><span>Run the release-2.3 pack every weekday at 2am in the background</span></a></li><li><a class="tile" href="#cap-notifications"><b>Teams and email notifications</b><span>Post the results to the QA channel</span></a></li><li><a class="tile" href="#cap-pipelines"><b>CI pipelines</b><span>Add a GitHub Actions workflow that runs tests/cases.yaml on every PR, and make it green</span></a></li></ul></div>
 <div class="area"><a class="area-name" href="#area-data"><span class="mono">Data</span></a><ul class="tiles"><li><a class="tile" href="#cap-databases"><b>Ask a database in plain language</b><span>Which customers in India have paid orders? Ask the orders database</span></a></li><li><a class="tile" href="#cap-test-data"><b>Test data on demand</b><span>Generate test data: 20 customers with 1 to 3 orders each in the shop database</span></a></li><li><a class="tile" href="#cap-data-flows"><b>Data flows between systems</b><span>Check that customers from the digital database reach the CRM correctly</span></a></li></ul></div>
 <div class="area"><a class="area-name" href="#area-automated-tests"><span class="mono">Automated tests</span></a><ul class="tiles"><li><a class="tile" href="#cap-write-tests"><b>Turn a passed run into an automated test</b><span>Write a Playwright test for that run in ../app</span></a></li><li><a class="tile" href="#cap-heal-tests"><b>Heal failing tests</b><span>Heal the failing tests in ../app</span></a></li><li><a class="tile" href="#cap-find-run-tests"><b>Find and run existing tests</b><span>Run the checkout tests</span></a></li><li><a class="tile" href="#cap-regression-packs"><b>Regression packs for a release</b><span>Build a regression pack for sprint 12 and run it on TestMu</span></a></li><li><a class="tile" href="#cap-flaky-tests"><b>Flaky test detection and quarantine</b><span>Which tests are flaky?</span></a></li></ul></div>
 <div class="area"><a class="area-name" href="#area-pull-requests"><span class="mono">Pull requests</span></a><ul class="tiles"><li><a class="tile" href="#cap-pr-review"><b>Review pull requests</b><span>Review PR 42 and focus on error handling</span></a></li><li><a class="tile" href="#cap-pr-impact"><b>Pull request impact and test selection</b><span>What does PR 42 impact?</span></a></li></ul></div>
@@ -3238,6 +3238,62 @@ forgeqe monitor --once                         # one round, then exit (cron, sch
 </section>
 </div>
 </article>
+<article class="spec feat" id="cap-jobs">
+<div class="hd"><span class="mono">Operations</span><h3>Background jobs on a schedule</h3></div>
+<p class="ds">Runs ForgeQE commands (suites, monitor rounds, regression packs, healing, pull-request tests, browser goals) on a schedule or once in the jobs service, apart from the conversation and after it closes, and watches an existing GitHub Actions or Azure DevOps pipeline (read-only). A failed pipeline run or pack run is investigated: the healing agent works on each failure on a local branch of its own (a fix for review, or an app bug), never pushing. Says how each job goes when asked, shows runs as they end, and sends failures, recoveries and findings to Teams or email.</p>
+<div class="facts">
+<div class="fact"><span class="mono">Ask it</span><ul class="asks"><li>Run the release-2.3 pack every weekday at 2am in the background</li><li>Watch the pipeline in the background and investigate failures</li><li>How are my jobs doing?</li></ul></div>
+<div class="fact"><span class="mono">What it needs</span><ul class="needs"><li><span class="need opt">Optional</span><span>A notification channel: notify.channels in forgeqe.yaml, each a Teams Workflows webhook (in an environment variable) or email</span></li></ul></div>
+<div class="fact"><span class="mono">Tools</span><p class="tools"><code>jobs</code></p></div>
+</div>
+<div class="detail"><span class="mono">In detail</span>
+<section class="doc" id="doc-how_to_use-the-jobs-service-scheduled-and-long-jobs-apart-from-the-shell">
+<h4>The jobs service: scheduled and long jobs apart from the shell</h4>
+<div class="prose"><p><code>forgeqe jobs</code> runs ForgeQE commands on a schedule, or once, in a service of their own, so they keep going while you work on something else in the shell, or with the shell closed. Each run is its own <code>forgeqe</code> process with its own output and session log; you ask how a job is going at any time, and its failures and recoveries go to Teams.</p>
+<pre><code class="language-bash">forgeqe jobs add nightly-pack --cron &quot;0 2 * * 1-5&quot; -- pack run release-2.3   # weekdays at 02:00, this machine's time
+forgeqe jobs add smoke --every 30m -- suite checks/smoke.yaml
+forgeqe jobs add heal-login --now -- heal --focus login                     # once, now
+forgeqe jobs                       # every job: running or how its last run ended, and when it runs next
+forgeqe jobs status nightly-pack   # its last runs, and the last lines of the latest
+forgeqe jobs logs nightly-pack --tail 100
+forgeqe jobs run nightly-pack      # run it now as well
+forgeqe jobs stop nightly-pack     # stop the run that is going
+forgeqe jobs disable|enable|remove nightly-pack
+forgeqe jobs shutdown              # stop the service; scheduled jobs wait until it starts again
+</code></pre>
+<ul>
+<li><strong>What a job runs.</strong> One ForgeQE command: <code>suite</code>, <code>monitor</code>, <code>run</code>, <code>heal</code>, <code>pr-tests</code>, <code>pack run</code>, <code>pack triage</code> or <code>pipeline watch</code>, with its arguments after <code>--</code>. Nobody is there to answer a question while a job runs, so a <code>run</code> job is a browser run of the goal (<code>--direct</code>), and a <code>pack run</code> job carries <code>--yes</code>: adding the job is your approval, and the command you agree to shows it. It runs with the project's own <code>forgeqe.yaml</code>, model profile and approvals: a job cannot set <code>--config</code>, <code>--profile</code> or <code>--yes</code>, and nothing else is started (no shell, no other program). A <code>monitor</code> job runs one round each time it is due (<code>--once</code>), on the job's schedule; it sends its own <code>monitor.alerts</code>, so its job is quiet unless you add it with notices on.</li>
+<li><strong>When.</strong> <code>--every 15m</code> (at least 1m), <code>--cron &quot;0 2 * * 1-5&quot;</code> (minute hour day month weekday, local time), or neither: it runs when asked, and <code>add</code> starts it at once. <code>--now</code> also runs a scheduled job straight away. A run missed while the service was down starts once when it is back. When several are due, the one that has waited longest goes first.</li>
+<li><strong>How many.</strong> <code>jobs.parallel</code> in <code>forgeqe.yaml</code> (default 2) runs at once over all jobs, and never two runs of one job. A run past <code>--timeout</code> (default 6h, at most 24h) is stopped and marked timed out.</li>
+<li><strong>The service.</strong> <code>forgeqe jobs add</code> and <code>forgeqe jobs run</code> start it when it is not running, as a process of its own that outlives the command and the shell; <code>forgeqe jobs serve</code> runs it in the foreground (for a container or a system service). It is one per project, on this machine, until you stop it or the machine restarts. It listens on 127.0.0.1 only, with a random token kept in your own ForgeQE folder (<code>~/.forgeqe/jobs/</code>, readable by your user only, never in the project, which others may read), and refuses requests from web pages. A second service for the same project is refused. <code>.forgeqe/jobs/</code> and <code>.forgeqe/pipelines/</code> get their own <code>.gitignore</code>, so runs and logs are never committed.</li>
+<li><strong>Files.</strong> <code>.forgeqe/jobs/&lt;job&gt;/job.json</code> and, for each of its last 50 runs, <code>runs/&lt;run&gt;/run.json</code> and <code>output.log</code>; <code>.forgeqe/jobs/events.jsonl</code> lists what happened; <code>.forgeqe/jobs/service.log</code> is the service's own output. A run that was going when the service stopped is marked interrupted.</li>
+<li><strong>Watching a CI pipeline.</strong> <code>forgeqe jobs add ci --every 10m -- pipeline watch --github org/app --workflow ci.yml --branch main</code> (or <code>--ado-project Shop --pipeline 12</code>) looks at the pipeline's latest completed run each time, read-only: it starts, re-runs and changes nothing. A new failed run is triaged like a pack's (the JUnit or TRX files in its artifacts on GitHub, the Test Runs API on Azure DevOps; without results, the job and step that failed), in <code>.forgeqe/reports/pipelines/</code>. The job fails while the latest run is failed, so the team hears once when the pipeline breaks and once when it is green again; a cancelled run changes nothing. <code>forgeqe pipeline watch …</code> on its own does one look. GitHub needs the <code>gh</code> CLI signed in; Azure DevOps the same sign-in as the rest of ForgeQE.</li>
+<li><strong>Investigating failures.</strong> When a <code>pipeline watch</code> or <code>pack run</code> job fails, the failures that healing can decide (an element not found, an expected value that does not hold; not the environment, flaky tests or broken test code, and nothing while the environment is down) are investigated, at most 3 a run, <code>jobs.parallel</code> at once. Each goes to the healing agent in a git worktree of its own, on a new local branch <code>forgeqe/investigate/&lt;test&gt;-&lt;time&gt;-&lt;n&gt;</code>, from the commit the failing run tested when this clone has it (fetched when it can be, else <code>HEAD</code>, which the finding says). The agent re-runs the flow in the browser: when the flow works it heals the test, and the fix is committed on that branch for you to review; when the browser fails at the same step it is an app bug, and no branch is kept. Your checkout, its branch and its files are never touched; nothing is pushed or merged; the worktree is removed at the end. <code>node_modules</code>, <code>.venv</code> and <code>venv</code> from your checkout are linked into the worktree so the tests can run, and never committed. The failure is told at once (<code>job_failed</code>), the findings when they are in (<code>job_investigated</code>); <code>forgeqe jobs status</code> lists them, <code>.forgeqe/jobs/&lt;job&gt;/runs/&lt;run&gt;/investigations.json</code> and <code>investigate.log</code> have the detail, and <code>forgeqe investigate &lt;run folder&gt;</code> runs an investigation by hand. <code>--no-investigate</code> turns it off for a job.</li>
+<li><strong>From the shell.</strong> Ask ForgeQE's agent: &quot;run the release-2.3 pack every weekday at 2am in the background&quot; makes it propose the job, and ForgeQE asks you to agree to its exact command and schedule before adding it (<strong>Add the job</strong> / <strong>Don't add it</strong>). You keep working; when a job's run ends, a line says so in the conversation (&quot;Job nightly-pack failed: …&quot;), and &quot;how are my jobs?&quot; or &quot;show the end of the nightly-pack log&quot; get answers from the service. The agent adds a job only when you ask for one, and never one that runs anything but the commands above.</li>
+<li><strong>Notices.</strong> With <code>notify:</code> channels set up (section 17), a job that fails sends <code>job_failed</code> once (not every run while it keeps failing), <code>job_recovered</code> when it passes again, and a job that runs only when asked sends <code>job_done</code> when it ends. <code>--no-notify</code> keeps a job quiet.</li>
+</ul>
+<pre><code class="language-yaml">jobs:
+  parallel: 2
+</code></pre>
+<p><strong>On company machines.</strong> Nothing here needs administrator rights:</p>
+<div class="tbl"><table>
+<thead>
+<tr><th>What</th><th>Needs</th></tr>
+</thead>
+<tbody>
+<tr><td>The jobs service</td><td>Runs as you, in your project and your ForgeQE folder. It listens on 127.0.0.1 only, so no firewall rule is needed. Endpoint security may see a background <code>bun</code> process: it is ForgeQE's own (in its install folder), which IT can allow. Jobs pause while the machine sleeps or is off and a missed run starts once when it is back; for always-on jobs, run <code>forgeqe jobs serve</code> on a VM or build agent under its service manager (systemd, a Windows Task Scheduler task &quot;at log on&quot;, which needs no administrator, or a container).</td></tr>
+<tr><td><code>pipeline watch</code> on GitHub</td><td>The <code>gh</code> CLI signed in (<code>gh auth login</code>) with read access to the repository's Actions. A fine-grained token needs Actions: read and Contents: read; with SAML single sign-on, authorize the token for the organization. GitHub Enterprise Server: <code>--github host/owner/repo</code> and <code>gh auth login --hostname host</code>.</td></tr>
+<tr><td><code>pipeline watch</code> on Azure DevOps</td><td>The Azure DevOps sign-in ForgeQE already uses (<code>forgeqe auth login azure_devops</code>, or a token in <code>ADO_PAT</code>) with Build: read and Test Management: read. Azure DevOps Server works through its collection URL.</td></tr>
+<tr><td>Investigating failures</td><td>git, read access to fetch the failing commit, the tests' dependencies installed in your checkout, and a model: a key-based profile, or GitHub Copilot where the organization's policy allows Copilot CLI and the SDK. Branches are local; nothing is pushed.</td></tr>
+<tr><td>Teams notices</td><td>A Workflows webhook (&quot;post to a channel when a webhook request is received&quot;), or an incoming webhook where the tenant still allows them; where neither is allowed, email.</td></tr>
+<tr><td>Proxy and inspected HTTPS</td><td>The same as the rest of ForgeQE (section &quot;Company networks&quot;); <code>gh</code> uses <code>HTTPS_PROXY</code>. <code>forgeqe doctor</code> checks the machine and the network.</td></tr>
+</tbody>
+</table></div>
+<p>On Windows, the dependency folders are linked into worktrees as junctions (no administrator or developer mode needed), a stopped run ends at once rather than being asked to finish, and the token file is protected by your user profile's permissions.</p>
+</div>
+</section>
+</div>
+</article>
 <article class="spec feat" id="cap-notifications">
 <div class="hd"><span class="mono">Operations</span><h3>Teams and email notifications</h3></div>
 <p class="ds">Tells the team in Microsoft Teams channels or by email when a suite fails or passes, when a release readiness verdict is in, and when the domain evaluation falls below its gate, with the failing cases, their bugs, and links to the report and the CI build. Each channel picks its events; the agent also posts a summary to a channel when asked.</p>
@@ -3273,9 +3329,13 @@ forgeqe monitor --once                         # one round, then exit (cron, sch
 <tr><td><code>suite_passed</code></td><td>Every case passed</td><td>The count, how long it took, where it ran</td></tr>
 <tr><td><code>release</code></td><td>A release readiness verdict</td><td>GO, GO WITH RISKS, or NO-GO, the counts, every blocker and risk, and the wiki page when published</td></tr>
 <tr><td><code>eval_failed</code></td><td><code>forgeqe eval domain --min</code> falls below its gate</td><td>Coverage for each variant and invented ids</td></tr>
+<tr><td><code>job_failed</code></td><td>A job of the jobs service fails or runs out of time, after passing or for the first time (not again while it keeps failing)</td><td>The job, its command and schedule, how long it ran, its last lines of output, and where its output is</td></tr>
+<tr><td><code>job_recovered</code></td><td>A job passes again after failing</td><td>The same, for the run that passed</td></tr>
+<tr><td><code>job_done</code></td><td>A job that runs only when asked ends</td><td>The same, whether it passed or failed</td></tr>
+<tr><td><code>job_investigated</code></td><td>The investigation of a job's failed run is done</td><td>Each failure: a test out of date with its fix on a branch, an app bug, or not decided, with the agent's reasons</td></tr>
 </tbody>
 </table></div>
-<p>A channel without <code>on</code> gets <code>suite_failed</code>, <code>release</code>, and <code>eval_failed</code>. Email adds the failure screenshots. A stopped suite is not announced. Monitoring keeps its own <code>monitor.alerts</code>, with the same cards.</p>
+<p>A channel without <code>on</code> gets <code>suite_failed</code>, <code>release</code>, <code>eval_failed</code>, <code>job_failed</code>, <code>job_recovered</code>, and <code>job_investigated</code>. Email adds the failure screenshots. A stopped suite is not announced. Monitoring keeps its own <code>monitor.alerts</code>, with the same cards.</p>
 <p>Events are sent from runs nobody is watching: <code>forgeqe suite</code> in a pipeline or on a schedule, <code>forgeqe run</code>, and the MCP server. In the shell, where people try things out, nothing is sent on its own; ask instead (&quot;post the results to the QA channel&quot;, &quot;tell the release channel the verdict&quot;), and ForgeQE's agent posts a card with <code>notify</code>.</p>
 <p>Links: with <code>report_url</code>, cards open the report where the reports folder is published. In Azure Pipelines and GitHub Actions they also open the build, where the report is an artifact. Without either, the card names the report's path.</p>
 <p><strong>Setting up a Teams channel:</strong></p>

package/forgeqe.example.yaml

@@ -133,7 +133,7 @@ prompts:
 # industry would, and add each regulation's obligations a feature touches. Your applicable controls, with their ids,
 # go in .forgeqe/knowledge/compliance.md; agents cite only ids listed there.
 # Teams and email notifications for the team: each channel picks its events (suite_failed, suite_passed, release,
-# eval_failed). The Teams webhook URL is a secret: keep it in the environment, never here. See HOW_TO_USE.md section 17.
+# eval_failed, job_failed, job_recovered, job_done). The Teams webhook URL is a secret: keep it in the environment, never here. See HOW_TO_USE.md section 17.
 # notify:
 #   report_url: https://reports.example.com/forgeqe   # where .forgeqe/reports is published; cards link each report
 #   channels:
@@ -360,3 +360,8 @@ suite:
 #       password_env: SMTP_PASSWORD
 #       # from: ForgeQE <qa-bot@example.com>   # default: the username
 #       to: [qa-team@example.com]
+
+# forgeqe jobs: ForgeQE commands on a schedule or once, in a service apart from the shell (HOW_TO_USE.md, "The jobs
+# service"). Jobs are added with forgeqe jobs add; this sets how many runs go at once over all jobs.
+# jobs:
+#   parallel: 2

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@capagents/forgeqe",
-  "version": "2.3.4",
+  "version": "2.3.5",
   "description": "ForgeQE: a browser agent CLI that runs goals and test cases in local Chrome or Chromium, or BrowserStack and TestMu AI/LambdaTest when configured, and turns passing runs into tests in your repo",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {

package/prompts/brain.md

@@ -356,6 +356,15 @@ For REST, GraphQL, SOAP, and gRPC:
 - For monitoring that continues after this session, you SHOULD set up forgeqe monitor on a computer that is always on, or forgeqe monitor --once from cron or a scheduled pipeline.
 - That pipeline MUST keep .forgeqe/reports/monitor/state.json between runs (a cache), so that alerts start only on changes.
 
+### Background jobs
+- Work that repeats on a schedule, or that takes long while the user works on other things, is a job: jobs add, with a name, the ForgeQE command (suite, monitor, run, heal, pr-tests, pack run, pack triage) and every or cron. A job with neither runs one time, now.
+- ForgeQE asks the user to agree to the job before it is added. You MUST NOT add a job the user did not ask for.
+- After you add a job, you SHOULD continue with the other work. Runs that end come in as notices. jobs status and jobs log tell how a job goes.
+- When the user asks how jobs are going, you MUST call jobs list, or jobs status for one job, and MUST NOT guess.
+- A monitor job runs one round each time. It sends its own monitor alerts.
+- To watch a CI pipeline that the team has, add a job on pipeline watch with --github owner/repo --workflow FILE, or --ado-project NAME --pipeline ID, and --branch. It only reads the pipeline.
+- When a pipeline watch or pack run job fails, ForgeQE investigates the failures on local branches. You MUST report each finding as the job reports it: a fix on a branch for review, an app bug, or not decided. You MUST NOT push, merge or delete these branches unless the user asks.
+
 ### ForgeQE itself
 - Questions about what ForgeQE can do, such as "can you do API testing?", are questions, not tasks. You MUST answer them, and MUST NOT start the work.
 - The capability list in "This session" is complete, and says what works in this setup. For details, you MUST call capabilities with the ids that fit.