@engineeros/connector 1.56.11 → 1.58.0

package/README.md

@@ -455,6 +455,20 @@ Include exactly one designed case reference in each implemented test name, such
 
 Use connector 1.40.0 with backend migration `20261015_000132`. If verification says the per-case report or inventory is missing, correct the reporting command or connector and continue through the existing Goal. Work and failed measurements are preserved. Delete the implementation Goal only when intentionally starting over; repeated selections reuse its ownership.
 
+## Checked writing (Automate tests keep checks)
+
+**Automate tests** writes tests for designed cases in a worker of its own and leaves them uncommitted. With connector 1.58.0 and a backend that sends `checks`, a written test is kept only for a reason. The connector advertises `test_checks` when it can do this.
+
+- **What runs.** In a clean copy of the repository (the checkout's commit plus the task's test files, nothing else) the selected tests run three times; the third run puts every test file on a command of its own, which finds tests that depend on each other's order. A test that passes fewer than three times is flaky and goes back to the writing session. Then small, repeatable changes (a comparison flipped, a sign swapped, a condition negated, a returned value dropped) are made, one at a time, to the lines the test covers, and the test is kept only when at least one of them makes it fail. A change that breaks the build, or a run that times out, is discarded and counts for nothing.
+- **Where.** Two Git worktrees under `.engineeros/test-workers/`: the task's worker (`1`, `2` ...) and the clean copy (`v1`, `v2` ...). Mutated files are restored in a `finally`, whatever happens. The covered lines come from the runner's own coverage where it is already installed (coverage.py, Jest, a Vitest coverage provider, `node --test` through V8, Go's cover profile); otherwise from the symbols the test names in the files it imports. Nothing is installed.
+- **What it writes.** Nothing in your checkout from the checks. A test that passed but was rejected is removed and a copy is kept under `.engineeros/test-automation/<task>/rejected/`. A reproducer for a bug the product really has is saved under `.engineeros/proven-bugs/<task>/` and never left in the suite. Both folders are git-ignored.
+- **Where no mutant can be made** (a UI test, an end-to-end test, a language this does not mutate) an ordinary case is kept on three clean passes and an assertion, and is labelled `repeat_only`. Behaviour-as-found and catching tests have no fallback: no killed mutant, no keep.
+- **Repairs.** A case that fails gets two plain repairs and then one more as the `replan` task class, a step higher, before its tests are removed.
+- **Cost in time.** The mutant work has a wall-clock budget of 20 minutes per task (`ENGINEEROS_MUTANT_BUDGET_MS`, in milliseconds). Cases not reached are reported as skipped for the budget; it never limits tokens.
+- **Turn it off.** Set `ENGINEEROS_TEST_CHECKS=off` in the connector's environment. The connector then stops advertising `test_checks`, and tests are kept as they were before 1.58.0.
+
+Jest, Go (`go test -json`), Maven, Gradle and Playwright are written and checked as well as pytest, Vitest and `node --test`. Playwright has nothing to mutate and is kept `repeat_only`. A gap design is sent the diff of the commit it is scoped to (`git diff --unified=2`, without generated files, lockfiles and tests), bounded per file with the cut stated.
+
 ## Record CI test results
 
 Your CI can record each test suite's results in EngineerOS, where they sit beside manual and scheduled results on the project's Quality page, link to the test inventory, and open or resolve defects. No pairing is needed. Have your test runner write JUnit XML (pytest `--junitxml`, Vitest `--reporter=junit`, `jest-junit`, Playwright's `junit` reporter, `node --test --test-reporter=junit`, Maven Surefire, `go-junit-report`, `dotnet test --logger junit`), then run this after the tests, also when they failed:
@@ -494,6 +508,26 @@ Inputs: `label` and `junit` (required; one path or glob per line), `coverage`, `
 
 A project can schedule a test command on the Quality page (**CI and schedules**): a plain command line, the folder it runs in, a cron expression and its time zone, and optionally a test environment. When a run is due, EngineerOS sends it to an online connector of the person who last saved the schedule. The connector runs it exactly like a Goal's command Proof - without a shell, stopped after 15 minutes unless `ENGINEEROS_TEST_COMMAND_TIMEOUT_MS` says otherwise, with the output redacted - collects the JUnit and coverage reports the command wrote, and reports what it measured, together with the checked-out branch and commit. Scheduled runs on one connector run one at a time. When no connector is online the run is recorded as skipped, with the reason, and is not queued.
 
+## Onboarding baseline (existing tests, run once)
+
+After a repository is onboarded, an operator can ask EngineerOS to run the project's existing tests once (**Run your existing tests once**). It costs no model tokens and is never started without that click, unless the project switched on automatic baselines.
+
+- **What runs.** Every test runner the repository declares and that needs no running app: pytest, Vitest, Jest, `node --test`, `go test` and Maven or Gradle. Playwright and Cypress are left for a confirmed environment, and other runners (cargo, rspec, xunit, mocha ...) are listed as unsupported with the reason. Failures that look like missing services (connection refused, a missing environment variable) are reported as needing setup, not as product failures.
+- **Where.** In a Git worktree of its own at the checkout's current commit, `.engineeros/test-workers/baseline`, with the checkout's dependency folders (`node_modules`, virtual environments) linked in. It measures committed code: your uncommitted edits and untracked files are not run.
+- **What it writes.** Nothing in your checkout. Reports and coverage go to a temporary folder that is deleted when each shard settles. The worktree is reset at the next baseline. Nothing is installed: where a runner's coverage tooling is missing (no `pytest-cov`, no Vitest coverage provider) the runner still runs for the pass rate and flaky tests, and reports no coverage.
+- **How.** Each runner's test files are cut into shards of about five minutes (about 40 files for pytest and `node --test`, 60 for Vitest and Jest, one package for Go, one module for Maven or Gradle) and run one after another; the shard's results are posted as soon as it settles, so a machine that sleeps keeps what already ran. A shard has no wall-clock limit; it is stopped only after ten minutes without output (`ENGINEEROS_BASELINE_IDLE_MS` changes that), which costs that shard and nothing else. Each failing test file is run twice more on its own, to tell a flaky test from a broken one. Runners run side by side only when you set several workers (`/workers`), at most three.
+- **Turn it off.** Set `ENGINEEROS_QUALITY_BASELINE=off` in the connector's environment, or `"quality_baseline": false` in the connector's settings file. The connector then stops advertising the `quality_baseline` capability and refuses a baseline message.
+
+## Quality map and commit reports
+
+With each workspace snapshot the connector sends a **quality map**, so EngineerOS can tell which tests a change reaches instead of flagging whole capabilities. It is built from the local context index and `git log`, runs no code from the repository and costs no model tokens:
+
+- **Test footprints.** For each test file, the files its imports reach (three levels deep, with `tsconfig`/`jsconfig` `paths`, `baseUrl` and workspace package names resolved), and for browser tests the route files they visit. A file that many others import is recorded but not followed.
+- **File facts.** Per code file over the last 180 days: commits, fix commits, authors, last change, lines, symbols and decision points. Author names are counted, never sent.
+- **Inputs and launch recipes.** The files whose change should rerun everything (manifests, lockfiles, CI, migration folders, `conftest.py`), the app's routes and how it is started (`docker compose`, `npm run dev`, `uvicorn`, `manage.py`, Rails, Spring, Go). Nothing is started.
+
+The connector also looks at `HEAD` on every maintenance tick (ten seconds). A new commit or a branch switch that stays put for three seconds is reported as a small delta: the changed files with their added and deleted line counts, the cases of the changed test files, and the map entries that moved. No snapshot is uploaded unless EngineerOS asks for one. Commits that only save EngineerOS's own artifacts are not reported. A workspace that is not a Git repository, and a hunt host, report nothing.
+
 ## Bug Hunter
 
 Bug Hunter tests a running test environment of the project (`bug_hunt.session`). The coding agent writes test cases; the connector runs them in a browser, with no model involved, so a recorded case is re-run on every new version for free. The connector does not need the project's code: a connector paired in an empty folder hunts from the project's EngineerOS artifacts.
@@ -512,6 +546,8 @@ On Linux, missing Chromium system libraries require administrator access. At a t
 - **Catalogue runs** execute the recorded cases. A failure is run twice more from a fresh start: failing all three times it is reproduced, otherwise flaky.
 - **Hunts** run one agent turn in `.engineeros/bug-hunt/<session>/`. The agent writes each scenario as a case file in `cases/` and probes a draft with `node .engineeros/bug-hunt/<session>/probe.mjs <case file>`, which runs it through the same runner. Case files that cannot be read get one correction turn. The connector registers the cases with EngineerOS, which refuses duplicates, then runs each one.
 - **Review**: newly reproduced failures go to one reviewer turn in a fresh session. Its verdicts decide what becomes a bug.
+- **What pages do on their own**: every run listens, without sending anything, for console and page errors, failed and 5xx or 404 requests, loading that never finishes, broken images and links and mixed content, scans each page once for serious accessibility violations (axe-core) and samples Web Vitals. A signal never changes a case's result. A cause that shows again on a clean re-run is sent to EngineerOS, which files it as a bug; nothing but text (messages, URLs without query values, selectors, numbers) leaves the machine. A production check only listens.
+- **Routes and API calls**: a map also reads `robots.txt`, `sitemap.xml`, the application's own bundles and public source maps (and, in a checkout, its source) for routes, then presses navigation buttons and tabs with every write aborted. It records the application's own XHR and fetch calls as a map of endpoints, reduced to their shape; no response value is kept.
 - **Rules of engagement** are enforced by the runner. Only the environment's hosts and the allowed hosts may be opened or written to. A step that a forbidden-action rule names (a quoted element name or a path) is refused.
 - **Stopping**: EngineerOS can stop a session at any time. Stop ends the agent's turn and still runs and reviews what it wrote; stop without review skips the review; abort ends everything at once.
 - **One at a time**: a hunt starts only while the coding agent is idle, and one session runs on a connector at a time. The session folder is removed when it ends.
@@ -583,3 +619,7 @@ The API body has `report` (the SARIF object) and `complete` (a boolean). Set `co
 full scan: only complete reports resolve missing findings. Uploads require an authorized non-production hunter,
 and every finding URL must be in its allowed hosts. Findings deduplicate by environment, rule and URL and can be
 triaged and filed through Findings. The connector does not run ZAP; CI owns the scanner and policy.
+
+## Third-party code
+
+Accessibility scans use [axe-core](https://github.com/dequelabs/axe-core) (MPL-2.0), installed as an unmodified npm dependency and never edited or bundled into other files.