@activate-core/test-forge 0.14.0 → 0.15.0

package/README.md

@@ -35,13 +35,44 @@ When your code changes later:
 testforge update     # re-check and refresh only what changed
 ```
 
+### Working on one journey
+
+`update` is the whole-project command: if the code has moved it re-maps,
+re-drafts **every** doc and regenerates. Most of the time you want one
+journey, and both steps take `--journey` (or `--area`):
+
+```bash
+# the spec for one journey changed — re-draft just that doc, then its tests
+testforge discover --journey payment
+testforge generate --journey payment
+
+# adding a journey that the reference now describes
+testforge discover --journey refund-flow
+testforge generate --journey refund-flow
+
+# everything in one area
+testforge generate --area ops
+```
+
+A journey **is** its doc: `generate` reads `docs/` and treats every markdown
+file there as one journey. So a doc you write by hand — for a feature the
+reference does not cover yet, or one taken from a ticket — works exactly like a
+drafted one. Write the doc, then `generate --journey <name>`.
+
+If you edited a doc yourself, skip `discover` entirely and go straight to
+`generate --journey <name>`.
+
+Regeneration replaces: when a journey is regenerated, test files for checks that
+no longer exist are deleted — but only files TestForge wrote. Your own tests are
+never touched.
+
 ## Commands
 
 | Command              | What it does                                                                                                                       |
 | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
 | `testforge setup`    | One-time setup: runs init, map, and discover in order, with progress.                                                              |
 | `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. |
-| `testforge update`   | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |
+| `testforge update`   | Refreshes the map, docs and tests when the **source code** or the **reference** has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |
 
 ### Advanced (run a single step)
 
@@ -51,6 +82,41 @@ testforge update     # re-check and refresh only what changed
 | `testforge map`      | Rebuild the project map (`testforge.md`) only.              |
 | `testforge discover` | Redraft the per-journey docs in `docs/` only.               |
 | `testforge env`      | Rebuild `.env.testing.example` from the docs only.          |
+| `testforge run`      | Run the generated tests (`--journey` / `--area` / `--env-file`). |
+| `testforge report`   | Publish results to the dashboard — `--results` after a run, `--inventory` to register every generated check. |
+
+### Flags worth knowing
+
+| Flag                       | On                     | What it does                                                                 |
+| -------------------------- | ---------------------- | ---------------------------------------------------------------------------- |
+| `--journey <name>`         | discover, generate, run | Work on one journey instead of all of them.                                  |
+| `--area <name>`            | discover, generate, run | Work on one area (api, web, ops…).                                           |
+| `--force`                  | generate               | Regenerate even when the doc has not changed.                                |
+| `--refresh-docs`           | update                 | Re-draft the docs even when nothing detectably changed — see below.          |
+| `--reference <file\|url>`  | setup                  | Base the docs on a spec instead of the code map.                             |
+| `--source map`             | setup                  | Base the docs on `testforge.md`, no prompt.                                  |
+
+#### When the reference cannot be checked
+
+`update` compares two things: a hash of your source code, and a hash of the
+reference the docs came from. A reference it can read (a file, or a plain URL)
+is compared directly, so a spec that moved while the code stood still is
+detected — that case used to pass silently as "everything is up to date".
+
+Some references cannot be read at all. A Notion or Confluence link answers a
+plain fetch with the application, not the document: no page content, and a build
+stamp that changes on every vendor deploy. Hashing that would be worse than not
+checking — it would report "unchanged" while the spec moved, and "changed" every
+time the vendor shipped. TestForge refuses it and tells you what it could not
+check:
+
+```
+No source changes detected.
+
+  Note: the reference could not be checked — app.notion.com serves an app
+  shell, not the document — its content needs the API.
+  If the specification has changed, run `testforge update --refresh-docs`.
+```
 
 ## How it works
 
@@ -70,8 +136,18 @@ becomes a Jest test (HTTP + database checks); `Type: web` becomes a
 Playwright test. Other areas use the framework set in config. It also
 generates a seed file that provisions the test data each test needs.
 
-**update** hashes your source code and compares it to the last run. No
-change means no work and no AI calls; a change triggers a refresh.
+**generate** hands each file it writes to the project's own ESLint with
+`--fix` before moving on. Formatting differences are corrected in place — CI
+runs lint before it runs a test, and a file that is correct but indented
+differently still fails the build. Anything the linter cannot fix (an unused
+declaration, say) is reported as a warning against that journey, because a
+declaration nothing checks is worth a person's attention. If the project has no
+ESLint, the step is skipped.
+
+**update** hashes your source code AND the reference the docs came from,
+and compares both to the last run. Neither changed means no work and no AI
+calls; either one changed triggers a refresh, and it says which. A reference it
+cannot read is reported as exactly that, never as "up to date".
 
 ## Source of truth
 
@@ -108,7 +184,7 @@ website, or a Notion page. testforge then:
 
 The reference becomes the source of truth — expected behavior from a human
 or client, not just what the code happens to do. The choice is remembered
-in `.testforge/source.json`, so a later `testforge update` refreshes the
+in `test-artifacts/source.json`, so a later `testforge update` refreshes the
 docs from the same source without asking again. Your review of
 `testforge.diff.md` stays as an audit of what differed and why.
 
@@ -204,7 +280,8 @@ The test runner produces its own pass/fail report.
     "ops": "playwright"
   },
   "seedOutput": "packages/api/test/helpers/seed",
-  "testExtension": ".test.ts"
+  "testExtension": ".test.ts",
+  "externalModules": ["shipengine", "@customerio/cdp-analytics"]
 }
 ```
 
@@ -221,6 +298,15 @@ The test runner produces its own pass/fail report.
 - **framework** — which test framework each area uses.
 - **seedOutput** — where seed files go.
 - **testExtension** — `.test.ts` or `.test.js`.
+- **externalModules** — the third-party network services this project
+  talks to, named as a test would import them. A generated backend test
+  that imports one without mocking it is rejected before it is written:
+  left alone it would reach the live service, so it breaks offline and
+  can fire real charges, messages or shipments. testforge already knows
+  the common SDKs (payment, SMS, email, cloud, identity) and this list is
+  added to them, never instead of them — so name only what is particular
+  to your product. Every domain has something: a shipping carrier, a CMS,
+  a ticketing or streaming backend, an ad network.
 
 ## Notes
 

package/dist/cli/index.js

@@ -13,6 +13,10 @@
  *   testforge generate                 Generate tests from the journey docs.
  *   testforge generate --basic         Use the built-in basic generator only.
  *   testforge generate --force         Regenerate everything, ignoring the cache.
+ *   testforge update                   Refresh only what changed (code or reference).
+ *   testforge update --refresh-docs    Re-derive the docs even with no detected
+ *                                      change — for a reference we cannot read
+ *                                      (an auth-gated page can move unseen).
  */
 import { VERSION } from '../core/index.js';
 import { runGenerate } from '../core/run-generate.js';
@@ -65,7 +69,7 @@ function printUsage() {
     console.log('Usage:');
     console.log('  testforge setup             Run init + map + discover in order');
     console.log('  testforge setup --reference <file|url>  Base the docs on a reference');
-    console.log('  testforge setup --source map            Base the docs on testforge.md');
+    console.log('  testforge setup --source map            Base the docs on testforge.md', '  testforge update --refresh-docs         Re-derive docs from the reference');
     console.log('  testforge generate           Generate tests from the journey docs');
     console.log('  testforge generate --basic   Use the basic generator only');
     console.log('  testforge generate --force   Regenerate everything');
@@ -83,7 +87,7 @@ function printUsage() {
 }
 async function main() {
     if (command === 'update') {
-        await runUpdate('.');
+        await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
     }
     else if (command === 'setup') {
         await runSetup('.', sourceOptions());

package/dist/core/ai-generator.js

@@ -129,20 +129,88 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
         `    test with a one-line reason naming exactly what was missing. Never`,
         `    let a missing precondition reach an assertion and report as a bug.`,
         `    Guard such tests at the top: check the precondition, skip if unmet.`,
+        `    A CREDENTIAL THAT EXISTS IS NOT A CREDENTIAL THAT WORKS. A supplied`,
+        `    token expires; treat "present" and "usable" as different checks.`,
+        `    Where the harness accepts a pre-minted token, verify it is still in`,
+        `    date (a JWT carries its own exp) before adopting it, and mint a`,
+        `    fresh one when it is not. If no working credential can be obtained,`,
+        `    skip and say so — a dead credential answers every authenticated`,
+        `    request with 401/403, and a suite that asserts on those reports an`,
+        `    entire product as broken when nothing is wrong with it.`,
         `(3) NO FLAKY TESTS. Every test must pass or fail for the same reason on`,
         `    every run, in any order. Do not depend on execution order, on data`,
         `    another test creates or mutates, on wall-clock timing, or on`,
         `    ambient totals that other tests change (assert on the rows YOU`,
         `    seeded, by id — never on a global count). Set up what you assert`,
         `    on, and assert only on that. Wait for a specific condition, never a`,
-        `    fixed sleep.`,
+        `    fixed sleep. Prefer a retrying assertion to a one-shot read: a value`,
+        `    the page writes after hydration, or re-renders on its way somewhere`,
+        `    else, is not there at the instant you first look. And a read that`,
+        `    runs in the page (evaluate) dies when a redirect lands mid-poll —`,
+        `    tolerate that error and let the poll retry, rather than reporting`,
+        `    a destroyed context as a product failure.`,
         `(4) NEVER ASSERT AN UNFIXED VALUE. Do not assert exact user-facing copy,`,
         `    incidental ordering, generated ids, timestamps, or counts the`,
         `    document does not fix. Assert the behaviour the document specifies,`,
         `    at the loosest form that still proves it.`,
-        `(5) A TIMEOUT IS NOT A BUG REPORT. If an action legitimately waits on`,
+        `(5) PROVE YOUR NEGATIVE INPUT IS ACTUALLY NEGATIVE. A security test that`,
+        `    feeds a "bad" credential, signature, id or payload is only as good as`,
+        `    the badness of that input. Construct it so it MUST differ from the`,
+        `    valid one at the level the product checks — mutate decoded bytes, not`,
+        `    an encoded character, since encodings carry spare bits that make`,
+        `    edits invisible. Where you can, assert the control too: the valid`,
+        `    input is accepted and the invalid one is not. An input that only`,
+        `    looks tampered turns correct behaviour into a reported breach, which`,
+        `    is the most damaging false alarm there is.`,
+        `(6) A STUB MUST NEVER INTERCEPT THE APPLICATION ITSELF. When you fake a`,
+        `    third party, match it by HOST (or an exact URL), never by a substring`,
+        `    of the URL: a pattern like "**evil.example**" also matches your own`,
+        `    page's "?next=https%3A%2F%2Fevil.example", so the app is served the`,
+        `    stub, nothing renders, and the test fails on a screen that never`,
+        `    loaded. Compare hostnames case-insensitively. After installing`,
+        `    stubs, satisfy yourself the page under test still loads.`,
+        `(7) BIND A FIXTURE TO THE IDENTITY THE SCREEN IS SCOPED BY. Before`,
+        `    seeding for a role-scoped page, find the exact key the server scopes`,
+        `    on — a user _id, a phone number, an account id — and which sign-in`,
+        `    the page uses. Seed against THAT identity. Do not reuse whatever the`,
+        `    harness minted first: the primary token is usually a different role,`,
+        `    and an auth/IdP subject is not the same value as the database id`,
+        `    (stripping a "sms|" prefix yields something id-SHAPED that belongs`,
+        `    to nobody). Fixtures hung off the wrong identity make every figure`,
+        `    read zero and every list read empty, which looks exactly like the`,
+        `    product losing data.`,
+        `(8) SEED ONLY STATES THE PRODUCT CAN REACH. Build every fixture the way`,
+        `    the code would: follow the transition that produces it and copy the`,
+        `    fields that transition writes. A hand-assembled record combining`,
+        `    fields no code path puts together is not a test case — the product`,
+        `    owes it no behaviour, so whatever it does with it proves nothing.`,
+        `    Before seeding a state, find the write that produces it; if there`,
+        `    is none, the scenario is imaginary — drop it.`,
+        `(9) A TIMEOUT IS NOT A BUG REPORT. If an action legitimately waits on`,
         `    something external, verify availability first and skip when absent.`,
         `    A test that times out tells the team nothing about the product.`,
+        `(10) THE HARNESS IS SOURCE TOO — READ IT BEFORE YOU CALL IT. Open the`,
+        `    helper you are about to use and check what it actually RETURNS: a`,
+        `    setup routine that publishes env and returns nothing hands you`,
+        `    undefined, and every later reference through it dies in beforeAll,`,
+        `    taking the whole file down as "failures" the product never caused.`,
+        `    The same applies to locators: a query must resolve to exactly ONE`,
+        `    node. Component libraries label a control twice (the <label> and the`,
+        `    widget both answer getByLabel), so prefer the role the widget`,
+        `    actually exposes. Where a sibling test in this journey already`,
+        `    drives the same control, reuse ITS helper verbatim rather than`,
+        `    writing a second one that has to rediscover the same trap.`,
+        `(11) LEAVE SHARED STATE AS YOU FOUND IT, EVEN WHEN YOU FAIL. A fixture`,
+        `    every other suite reads — a shared account, the one seeded record`,
+        `    the whole journey keys on — must be put back the moment you are`,
+        `    done with it. Read the value first, restore it in a finally or an`,
+        `    after-EACH hook, never an after-ALL: the specs that mutate shared`,
+        `    state are usually the ones asserting the write is REFUSED, so they`,
+        `    throw by design the day the product is wrong, and a worker that dies`,
+        `    on that throw never reaches after-all. The forged value then sits in`,
+        `    the fixture and every later suite reads it as real — one defect`,
+        `    becomes a hundred failures in journeys that were never broken, and`,
+        `    the run cannot be trusted until someone repairs the data by hand.`,
         `When rules conflict, prefer reporting LESS: a pending test costs a line`,
         `in the report; a false failure costs the team a release.`,
         ``,
@@ -174,8 +242,8 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
         `call in the file.`,
         `AUTH & SCOPING: a route may be declared auth:false yet STILL populate`,
         `request.user when a Bearer token is sent (a pre-auth extension). So`,
-        `owner-scoped data ("my bookings/payouts") is derived from the TOKEN`,
-        `owner — NOT from a query param. Do NOT assume a "?ownerId=/?makerId="`,
+        `owner-scoped data ("my orders", "my messages") is derived from the`,
+        `TOKEN owner — NOT from a query param. Do NOT assume an "?ownerId="`,
         `fallback overrides the token for an AUTHENTICATED call: the testkit`,
         `seeds the owner fixtures under the authenticated identity, so expect`,
         `the token owner's rows. A second, non-owner identity (for IDOR/scope`,
@@ -204,13 +272,13 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
                 `process.env.DB_URL) — never hardcode a connection string.`,
                 `MOCKING (MANDATORY — the test must be HERMETIC): the test must`,
                 `NEVER depend on a live external service. For EVERY external`,
-                `service the exercised code reaches (identity/IM, payment/Stripe,`,
-                `SMS, email, etc.), mock it AT THE SEAM THE CODE ACTUALLY USES —`,
-                `read the source to see which client it imports (e.g. axios, the`,
-                `stripe SDK) and how it is called, then jest.mock that EXACT module`,
+                `service the exercised code reaches — whatever this product happens`,
+                `to call out to — mock it AT THE SEAM THE CODE ACTUALLY USES —`,
+                `read the source to see which client it imports (an HTTP client, a`,
+                `vendor SDK) and how it is called, then jest.mock that EXACT module`,
                 `so the mock truly intercepts. A test that leaves any such call`,
-                `unmocked (so it would hit a real host like localhost/an IM/Stripe`,
-                `URL) is WRONG — it will fail whenever that service is down and is`,
+                `unmocked (so it would reach a real host rather than your mock)`,
+                `is WRONG — it will fail whenever that service is down and is`,
                 `not deterministic. Do NOT mock a client the code doesn't use.`,
                 `HERMETIC BASE URLS: if the client's target is built from an env`,
                 `var (e.g. process.env.IM_API_URL), do NOT rely on the ambient`,

package/dist/core/cache.js

@@ -105,3 +105,89 @@ export function hashProjectSource(rootDir) {
     walk(rootDir);
     return hashContent(parts.join('\n---\n'));
 }
+/**
+ * Is this response the document, or the application that would render it?
+ *
+ * Notion (and Confluence, and Google Docs) answer a plain fetch with 200 and a
+ * single-page-app shell: megabytes of script, none of the document's text, and
+ * a build stamp that changes every time they deploy. Hashing that is worse than
+ * not checking at all — it reports "unchanged" while the specification moves,
+ * and "changed" every time the vendor ships. Refuse it and say why.
+ */
+function looksLikeAppShell(url, body) {
+    let host = '';
+    try {
+        host = new URL(url).hostname.toLowerCase();
+    }
+    catch { /* keep '' */ }
+    const HOSTED_EDITORS = ['notion.so', 'notion.com', 'notion.site', 'atlassian.net', 'docs.google.com', 'sharepoint.com'];
+    if (HOSTED_EDITORS.some((h) => host === h || host.endsWith('.' + h))) {
+        return `${host} serves an app shell, not the document — its content needs the API`;
+    }
+    // Generic net for anything else that ships a shell: almost all markup and
+    // script, almost no readable text.
+    const text = body.replace(/<script[\s\S]*?<\/script>/gi, ' ')
+        .replace(/<style[\s\S]*?<\/style>/gi, ' ')
+        .replace(/<[^>]+>/g, ' ')
+        .replace(/\s+/g, ' ')
+        .trim();
+    if (body.length > 50_000 && text.length / body.length < 0.02) {
+        return 'the page returned an app shell rather than readable content';
+    }
+    if (/<title>[^<]*(log ?in|sign ?in|authenticate)/i.test(body)) {
+        return 'the page requires sign-in';
+    }
+    return null;
+}
+/**
+ * Hash the reference the journey docs are derived from.
+ *
+ * The source hash covers the CODE, which answers "has the implementation
+ * moved?". It says nothing about the specification. A reference that changes
+ * while the code stands still is the case that matters most — the docs, and so
+ * every assertion, are now out of date — and it used to pass silently as
+ * "everything is up to date".
+ *
+ * A local path is read directly. An http(s) reference is fetched. Anything we
+ * cannot read (a Notion page behind auth, no network) returns `unreadable` with
+ * the reason, so the caller can say so plainly instead of implying it checked.
+ */
+export async function hashReference(reference) {
+    const ref = (reference || '').trim();
+    if (!ref)
+        return { unreadable: 'no reference recorded' };
+    if (/^https?:\/\//i.test(ref)) {
+        try {
+            const res = await fetch(ref, { redirect: 'follow' });
+            if (!res.ok)
+                return { unreadable: `fetch returned ${res.status}` };
+            const body = await res.text();
+            const shell = looksLikeAppShell(ref, body);
+            if (shell)
+                return { unreadable: shell };
+            return { hash: hashContent(body) };
+        }
+        catch (e) {
+            return { unreadable: e.message.slice(0, 80) };
+        }
+    }
+    try {
+        const st = statSync(ref);
+        if (st.isDirectory()) {
+            const parts = [];
+            for (const entry of readdirSync(ref).sort()) {
+                const full = join(ref, entry);
+                try {
+                    if (statSync(full).isFile())
+                        parts.push(entry + ':' + hashContent(readFileSync(full, 'utf-8')));
+                }
+                catch { /* unreadable entry — skip */ }
+            }
+            return { hash: hashContent(parts.join('\n')) };
+        }
+        return { hash: hashContent(readFileSync(ref, 'utf-8')) };
+    }
+    catch (e) {
+        return { unreadable: e.message.slice(0, 80) };
+    }
+}

package/dist/core/config.js

@@ -38,6 +38,7 @@ const DEFAULT_CONFIG = {
     testExtension: '.test.ts',
     testCommand: {},
     envFile: '.env.testing',
+    externalModules: [],
 };
 /** Load config from the project, falling back to defaults. */
 export function loadConfig() {

package/dist/core/harness.js

@@ -58,7 +58,7 @@ export const HARNESS_CONTRACT = [
     `                                  EVERY id MUST be env-overridable: resolve`,
     `                                  each as process.env.TEST_<NAME>_ID ||`,
     `                                  '<stable default>' (e.g. TEST_PROJECT_ID,`,
-    `                                  TEST_WEARER_ID). This lets a caller point`,
+    `                                  TEST_OWNER_ID). This lets a caller point`,
     `                                  the suite at data created BY HAND on`,
     `                                  staging/prod (with SEED_MODE=existing)`,
     `                                  without editing code. Resolve non-id`,
@@ -116,8 +116,8 @@ export const HARNESS_CONTRACT = [
     `                                  the UI shows the seeded data.`,
     `Use only env var NAMES for secrets/connection — never literal values.`,
     `The ONLY credentials a developer should have to provide are the DB`,
-    `connection, a Stripe test key, and ONE seed/refresh token (or seeded`,
-    `login) — everything else setup() derives at runtime.`,
+    `connection, a test key for any paid third party, and ONE seed/refresh`,
+    `token (or seeded login) — everything else setup() derives at runtime.`,
     ``,
     `SAFETY — this testkit must NEVER be able to damage a real environment:`,
     `  - reset() and ANY delete/drop MUST refuse unless BOTH hold: the opt-in`,
@@ -140,9 +140,9 @@ export const HARNESS_CONTRACT = [
     `extension when a Bearer is sent. Therefore:`,
     `  - setup() MUST resolve the authenticated identity's REAL id (decode`,
     `    the minted token's subject/_id claim, or call the profile/whoami`,
-    `    endpoint) and seed the OWNER fixtures (the maker/wearer/customer`,
-    `    whose data the tests read) under THAT id — then set the TEST_IDS`,
-    `    owner ids to it.`,
+    `    endpoint) and seed the OWNER fixtures (whichever role in this`,
+    `    product owns the data the tests read) under THAT id — then set`,
+    `    the TEST_IDS owner ids to it.`,
     `  - Do NOT hardcode an owner id the token does not own: the seeded rows`,
     `    would never be returned to the authenticated caller, and every`,
     `    owner-scoped test would fail for a fake reason.`,
@@ -392,15 +392,15 @@ export function fixturesDoc() {
         ``,
         `## Running against data you created by hand (staging / prod)`,
         `1. Create the fixture entities in the target database yourself (a`,
-        `   project, its price line(s), the wearer/maker users, etc.) — mirror`,
-        `   what the seed builders in \`helpers/testkit\` produce.`,
+        `   parent record, its line items, the users on each side, etc.) —`,
+        `   mirror what the seed builders in \`helpers/testkit\` produce.`,
         `2. Point the suite at them — every well-known id is env-overridable:`,
         ``,
         `      TEST_PROJECT_ID=<your project id>`,
-        `      TEST_WEARER_ID=<...>        # and any other TEST_*_ID the suite reads`,
+        `      TEST_OWNER_ID=<...>         # and any other TEST_*_ID the suite reads`,
         ``,
         `3. Provide auth + connection env (DB url/name(s), a seed/refresh token`,
-        `   or seeded login, Stripe test key).`,
+        `   or seeded login, any third-party test key).`,
         `4. Run with \`SEED_MODE=existing\` so nothing is seeded or deleted.`,
         ``,
         `## Safety`,

package/dist/core/lint.js

@@ -0,0 +1,105 @@
+/**
+ * testforge — run the project's own linter over what we generated.
+ *
+ * A generated file is written to a repository that already has a house style,
+ * and CI usually runs `npm run lint` before it runs a single test. Files that
+ * are correct but indented differently fail that build, and somebody then fixes
+ * two hundred formatting errors by hand — work the generator caused and the
+ * generator can undo.
+ *
+ * So: after writing, hand the files to the project's ESLint with --fix. Style
+ * problems disappear silently, because they were never interesting. Anything
+ * left is reported, because an unused declaration in a test is a claim nothing
+ * checks, and that is worth a human's attention.
+ *
+ * The project's config is used, never one of ours — the point is to match the
+ * repository, not to impose a preference.
+ */
+import { spawnSync } from 'node:child_process';
+import { existsSync } from 'node:fs';
+import { join, dirname } from 'node:path';
+/** Walk up from a file to the nearest package that has ESLint installed. */
+function eslintFor(file) {
+    let dir = dirname(file);
+    for (let i = 0; i < 8; i++) {
+        const bin = join(dir, 'node_modules', '.bin', 'eslint');
+        if (existsSync(bin))
+            return { cwd: dir, bin };
+        const up = dirname(dir);
+        if (up === dir)
+            break;
+        dir = up;
+    }
+    return null;
+}
+/**
+ * Lint (and fix) the given files. Returns what was fixed and what was not.
+ * Never throws: a missing or misconfigured linter must not fail a generation
+ * that otherwise succeeded.
+ */
+export function lintGenerated(files) {
+    const real = files.filter((f) => f && existsSync(f));
+    if (!real.length)
+        return { checked: 0, fixed: 0, remaining: [] };
+    // Group by the package whose ESLint owns them — a monorepo has one per app.
+    const groups = new Map();
+    for (const f of real) {
+        const found = eslintFor(f);
+        if (!found)
+            continue;
+        const g = groups.get(found.cwd) || { bin: found.bin, files: [] };
+        g.files.push(f);
+        groups.set(found.cwd, g);
+    }
+    if (!groups.size) {
+        return { checked: 0, fixed: 0, remaining: [], skipped: 'no ESLint found in the project' };
+    }
+    let checked = 0;
+    let fixed = 0;
+    const remaining = [];
+    for (const [cwd, g] of groups) {
+        const before = countProblems(g.bin, cwd, g.files);
+        spawnSync(g.bin, ['--fix', '--no-error-on-unmatched-pattern', ...g.files], {
+            cwd,
+            encoding: 'utf-8',
+            timeout: 120_000,
+        });
+        const after = readProblems(g.bin, cwd, g.files);
+        checked += g.files.length;
+        fixed += Math.max(0, before - after.count);
+        remaining.push(...after.lines);
+    }
+    return { checked, fixed, remaining };
+}
+function countProblems(bin, cwd, files) {
+    return readProblems(bin, cwd, files).count;
+}
+function readProblems(bin, cwd, files) {
+    const res = spawnSync(bin, ['-f', 'json', '--no-error-on-unmatched-pattern', ...files], {
+        cwd,
+        encoding: 'utf-8',
+        timeout: 120_000,
+        maxBuffer: 32 * 1024 * 1024,
+    });
+    let parsed;
+    try {
+        parsed = JSON.parse(res.stdout || '[]');
+    }
+    catch {
+        return { count: 0, lines: [] }; // linter unavailable or output unusable
+    }
+    const lines = [];
+    let count = 0;
+    for (const file of parsed) {
+        for (const m of file.messages) {
+            if (m.severity !== 2)
+                continue; // warnings do not fail a build
+            count += 1;
+            if (lines.length < 20) {
+                const rel = file.filePath.replace(cwd + '/', '');
+                lines.push(`${rel}:${m.line} — ${m.message}${m.ruleId ? ` (${m.ruleId})` : ''}`);
+            }
+        }
+    }
+    return { count, lines };
+}

package/dist/core/report.js

@@ -99,6 +99,70 @@ export async function publishReport(opts) {
     });
     return { status: res.status, text: await res.text() };
 }
+const PENDING_CALL = /^[ \t]*(?:it|test)\.(?:todo|fixme)\(\s*(['"`])([\s\S]*?)\1/;
+/** Read the pending checks out of a test file, each with the reason recorded
+ *  beside it: the contiguous `//` comment block immediately above the call. */
+export function scanPending(file) {
+    let lines;
+    try {
+        lines = fs.readFileSync(file, 'utf8').split('\n');
+    }
+    catch {
+        return [];
+    }
+    const out = [];
+    for (let i = 0; i < lines.length; i++) {
+        const m = PENDING_CALL.exec(lines[i]);
+        if (!m)
+            continue;
+        const parts = [];
+        for (let j = i - 1; j >= 0 && j > i - 8; j--) {
+            const t = lines[j].trim();
+            if (!t.startsWith('//')) {
+                if (t)
+                    break;
+                else
+                    continue;
+            }
+            // Section banners ("// ----- Pending — ... -----") head a whole block and
+            // say nothing about THIS check; strip the rule and drop what is left if
+            // it is only decoration.
+            const body = t.replace(/^\/+\s?/, '').replace(/-{3,}/g, ' ').replace(/\s+/g, ' ').trim();
+            if (!body)
+                continue;
+            parts.unshift(body);
+        }
+        let reason = parts.join(' ').replace(/\s+/g, ' ').trim();
+        // A generic block header is not a reason for one check.
+        if (/^pending\s*[—-]/i.test(reason) && reason.length < 90)
+            reason = '';
+        if (reason.length < 15)
+            reason = '';
+        out.push({ title: m[2].replace(/\s+/g, ' ').trim().slice(0, 300), ...(reason ? { reason: reason.slice(0, 400) } : {}) });
+    }
+    return out;
+}
+/** Attach pending-case reasons to every suite whose file we can locate. */
+function attachPending(dir, ext, suites) {
+    const files = [];
+    try {
+        walk(dir, ext, files);
+    }
+    catch {
+        return;
+    }
+    const byName = new Map();
+    for (const f of files)
+        byName.set(suiteName(f), f);
+    for (const s of suites) {
+        const f = byName.get(s.name);
+        if (!f || s.pending)
+            continue;
+        const pending = scanPending(f);
+        if (pending.length)
+            s.pending = pending;
+    }
+}
 function walk(dir, ext, out) {
     for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
         const full = path.join(dir, e.name);
@@ -143,8 +207,15 @@ for (const f of ['jest.config.js','jest.config.cjs','jest.config.json']) {
   if (fs.existsSync(path.join(__dirname, f))) { base = require('./' + f); break; }
 }
 if (!Object.keys(base).length && fs.existsSync(path.join(__dirname,'package.json'))) base = require('./package.json').jest || {};
+// Strip the project's reporters. Spreading them in meant OUR OWN publish
+// reporter ran during a count — and since every test body is a no-op here, it
+// published a full, entirely green run over the real results. A counting pass
+// must be invisible: it reports nothing, because it tested nothing.
+const { reporters, ...rest } = base;
+void reporters;
 module.exports = {
-  ...base,
+  ...rest,
+  reporters: ['default'],
   testEnvironment: 'node', // plain node env for a count-only run
   setupFilesAfterEnv: [...(base.setupFilesAfterEnv || []), path.resolve(__dirname, '.tf-collect-patch.cjs')],
 };
@@ -212,7 +283,10 @@ function jestCount(dir, ext, category, out) {
  */
 function playwrightList(pkgDir, category, out) {
     try {
-        const res = execSync('npx playwright test --list', {
+        // `--reporter=list` so the project's own reporters — ours included — do
+        // not run during a count. Listing publishes nothing on its own, but a
+        // reporter that fires on end would report a run that never happened.
+        const res = execSync('npx playwright test --list --reporter=list', {
             cwd: pkgDir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], timeout: 180000,
         });
         const counts = {};
@@ -247,8 +321,10 @@ export function scanInventory(cwd, testOutput, ext, framework = {}) {
             continue;
         if (framework[category] === 'playwright') {
             // package root (holds playwright.config) is the parent of the tests dir
-            if (playwrightList(path.dirname(dir), category, suites))
+            if (playwrightList(path.dirname(dir), category, suites)) {
+                attachPending(dir, ext, suites);
                 continue;
+            }
             console.warn(`  ! playwright --list failed for "${category}" — using static estimate`);
             jestStatic(dir, ext, category, suites);
         }
@@ -256,6 +332,7 @@ export function scanInventory(cwd, testOutput, ext, framework = {}) {
             // jest (and ops/lambda, also jest-based) — accurate collect-only counter
             jestCount(dir, ext, category, suites);
         }
+        attachPending(dir, ext, suites);
     }
     return suites;
 }

package/dist/core/run-generate.js

@@ -19,6 +19,7 @@
 import { readFileSync, writeFileSync, mkdirSync, existsSync, readdirSync, rmSync, } from 'node:fs';
 import { join, dirname } from 'node:path';
 import { CACHE_PATH, COVERAGE_PATH, PLANS_DIR, STATE_DIR } from './paths.js';
+import { lintGenerated } from './lint.js';
 import { parseJourney } from './parser.js';
 import { generateTestCases } from './generator.js';
 import { writeDeferredStub } from './playwright-writer.js';
@@ -402,7 +403,12 @@ export async function runGenerate(docsDir, useAI = true, force = false, only, ar
                 selectorsByArea.set(areaDir, knownSelectors);
             }
         }
-        const testGate = (content) => checkTestGates(content, ext, { framework, allowedModules, knownSelectors });
+        const testGate = (content) => checkTestGates(content, ext, {
+            framework,
+            allowedModules,
+            knownSelectors,
+            externalModules: config.externalModules,
+        });
         // Skip if unchanged — but NOT if the journey still has deferred stubs
         // to retry (a resume pass fills those in without touching real files).
         if (!force && cache[file] === hash && !hasDeferredStubs(testDir, ext)) {
@@ -561,6 +567,23 @@ export async function runGenerate(docsDir, useAI = true, force = false, only, ar
         // it — so a later `generate` (once the limit resets) RE-DOES it with
         // AI instead of skipping it as "done". Persist per-journey so an
         // interrupted run still resumes from where it stopped.
+        // Hand what we just wrote to the project's own linter. CI runs lint before
+        // it runs a test, so a file that is correct but formatted differently fails
+        // the build — and somebody fixes it by hand afterwards. Style is fixed in
+        // place; whatever survives is reported, because an unused declaration in a
+        // test is a claim nothing checks.
+        const lintTargets = [
+            ...testFiles.map((t) => (typeof t === 'string' ? t : t.path)),
+            ...(seedPath ? [seedPath] : []),
+        ];
+        const lint = lintGenerated(lintTargets);
+        if (lint.fixed > 0) {
+            console.log(`  tidied ${lint.fixed} lint problem(s) in ${lint.checked} generated file(s)`);
+        }
+        if (lint.remaining.length) {
+            warnings.push(`lint: ${lint.remaining.length} problem(s) the linter could not fix — ` +
+                lint.remaining.slice(0, 5).join('; '));
+        }
         const degraded = useAI && (status === 'basic' || seedStatus === 'failed');
         if (!degraded) {
             cache[file] = hash;

package/dist/core/update.js

@@ -12,11 +12,13 @@ import { existsSync } from 'node:fs';
 import { runMap } from './map.js';
 import { runDiscover } from './discover.js';
 import { runGenerate } from './run-generate.js';
-import { hashProjectSource, loadCache, saveCache } from './cache.js';
+import { hashProjectSource, hashReference, loadCache, saveCache } from './cache.js';
+import { loadSourceChoice } from './source-of-truth.js';
 import { runEnvTemplate } from './env-template.js';
 import { CACHE_PATH } from './paths.js';
 const SOURCE_KEY = '__source__';
-export async function runUpdate(srcDir) {
+const REFERENCE_KEY = '__reference__';
+export async function runUpdate(srcDir, opts = {}) {
     if (!existsSync('testforge.md') || !existsSync('testforge.config.json')) {
         console.log("This project isn't set up yet.");
         console.log('Run `testforge setup` first.');
@@ -26,12 +28,41 @@ export async function runUpdate(srcDir) {
     const cache = loadCache(CACHE_PATH);
     const oldHash = cache[SOURCE_KEY];
     const newHash = hashProjectSource(srcDir);
-    if (oldHash === newHash) {
-        console.log('No source changes detected. Everything is up to date.');
-        console.log('(Nothing regenerated — no AI calls made.)');
+    const codeChanged = oldHash !== newHash;
+    // The docs are the source of truth for every assertion, so a reference that
+    // moved matters at least as much as code that moved. Checking only the code
+    // meant a changed spec was reported as "everything is up to date" — the tests
+    // then kept asserting a specification nobody uses any more.
+    const src = loadSourceChoice() || { mode: 'map' };
+    const refState = src.mode === 'reference' && src.reference
+        ? await hashReference(src.reference)
+        : { unreadable: 'docs are derived from the code map, not a reference' };
+    const refChanged = Boolean(refState.hash) && cache[REFERENCE_KEY] !== undefined && cache[REFERENCE_KEY] !== refState.hash;
+    const refFirstSeen = Boolean(refState.hash) && cache[REFERENCE_KEY] === undefined;
+    if (!codeChanged && !refChanged && !opts.refreshDocs) {
+        console.log(`No source changes detected.${refState.hash ? ' The reference is unchanged too.' : ''}`);
+        if (refState.unreadable && src.mode === 'reference') {
+            // Say what was NOT checked. Claiming "up to date" while the specification
+            // could have moved unseen is the one thing this command must not do.
+            console.log(`\n  Note: the reference could not be checked — ${refState.unreadable}.`);
+            console.log(`  ${src.reference}`);
+            console.log('  If the specification has changed, run `testforge update --refresh-docs`.');
+        }
+        else {
+            console.log('(Nothing regenerated — no AI calls made.)');
+        }
+        if (refFirstSeen) {
+            cache[REFERENCE_KEY] = refState.hash;
+            saveCache(CACHE_PATH, cache);
+        }
         return;
     }
-    console.log('Source code changed — updating the project.\n');
+    const why = [
+        codeChanged ? 'source code changed' : null,
+        refChanged ? 'the reference changed' : null,
+        opts.refreshDocs && !codeChanged && !refChanged ? 'refresh requested' : null,
+    ].filter(Boolean).join(' and ');
+    console.log(`Updating the project — ${why}.\n`);
     console.log('Re-mapping the project...');
     runMap(srcDir);
     console.log('  done — testforge.md\n');
@@ -46,6 +77,8 @@ export async function runUpdate(srcDir) {
     const skipped = results.filter((r) => r.status === 'skipped').length;
     console.log(`  done — ${made} updated, ${skipped} unchanged.\n`);
     cache[SOURCE_KEY] = newHash;
+    if (refState.hash)
+        cache[REFERENCE_KEY] = refState.hash;
     saveCache(CACHE_PATH, cache);
     console.log('Refreshing env template...');
     runEnvTemplate('docs', '.env.testing.example', srcDir);

package/dist/core/validate.js

@@ -20,6 +20,28 @@ import { execFileSync } from 'node:child_process';
 import { writeFileSync, mkdtempSync, rmSync } from 'node:fs';
 import { join } from 'node:path';
 import { tmpdir } from 'node:os';
+/**
+ * Third-party network SDKs common enough to be worth gating everywhere:
+ * a backend test that imports one without mocking it is talking to a real
+ * service, which flakes offline and can fire real charges, messages or
+ * emails. Deliberately NOT exhaustive and deliberately NOT tied to any one
+ * industry — every project names its own in `externalModules`, and a
+ * shipping API, a CMS, a games backend or an ad network is as much a live
+ * dependency as a payment provider. Database clients are excluded on
+ * purpose: a backend integration test may legitimately use a real local DB.
+ */
+export const DEFAULT_EXTERNAL_MODULES = [
+    // payment
+    'stripe', 'braintree', '@paypal/checkout-server-sdk', 'razorpay', 'square',
+    // messaging / email
+    'twilio', '@sendgrid/mail', '@sendgrid/client', 'nodemailer', 'postmark',
+    'mailgun.js', '@slack/web-api', 'resend',
+    // cloud
+    'aws-sdk', '@aws-sdk/client-ses', '@aws-sdk/client-sns', '@aws-sdk/client-s3',
+    '@google-cloud/storage', '@azure/storage-blob',
+    // identity / support / analytics
+    'auth0', 'firebase-admin', 'node-zendesk', 'intercom-client', 'analytics-node',
+];
 /**
  * Semantic gate for a generated test file. Returns a human-readable
  * reason string if the content violates a guardrail, or null if it is
@@ -70,14 +92,14 @@ export function checkTestGates(content, _ext, ctx) {
         }
     }
     // F11 — hermetic seam (jest/backend): a test that imports a known
-    // EXTERNAL network service (payment/SMS/email/cloud) must mock it, or the
+    // EXTERNAL network service must mock it, or the
     // suite silently depends on a live third party — it flakes offline and can
     // fire real charges/messages. DB clients are intentionally NOT listed:
     // backend integration tests legitimately use a real (local) database.
     if (ctx.framework === 'jest') {
         const EXTERNAL = new Set([
-            'stripe', 'twilio', '@sendgrid/mail', '@sendgrid/client',
-            'nodemailer', 'aws-sdk', '@aws-sdk/client-ses', '@aws-sdk/client-sns',
+            ...DEFAULT_EXTERNAL_MODULES,
+            ...(ctx.externalModules || []),
         ]);
         for (const mod of importedModules(content)) {
             const top = topLevelModule(mod);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activate-core/test-forge",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "AI-powered QA automation. Scans a project, drafts journey docs, and generates seeded Jest/Playwright tests. Works in any project, with any AI CLI.",
   "type": "module",
   "bin": {

package/reporters/jest-reporter.cjs

@@ -27,6 +27,13 @@ module.exports = class TestForgeJestReporter {
       }
     }
     const durationSec = Math.round((results.startTime ? Date.now() - results.startTime : 0) / 1000);
-    await publish(this.category, Object.values(suites), durationSec);
+    const list = Object.values(suites);
+    // Same reasoning as the Playwright reporter: no suites means nothing ran,
+    // and a published 0/0/0 run reads as a clean result.
+    if (!list.length) {
+      console.log('[testforge] nothing ran — no run published');
+      return;
+    }
+    await publish(this.category, list, durationSec);
   }
 };

package/reporters/playwright-reporter.cjs

@@ -26,6 +26,15 @@ module.exports = class TestForgePlaywrightReporter {
   }
   async onEnd(result) {
     const durationSec = Math.round(((result && result.duration) || (Date.now() - this.start)) / 1000);
-    await publish(this.category, Object.values(this.suites), durationSec);
+    const suites = Object.values(this.suites);
+    // A run that collected no suites did not test anything: global setup failed,
+    // or the filter matched nothing. Publishing it records a 0/0/0 result that
+    // reads as "the suite ran and found nothing wrong" — the most misleading
+    // thing this reporter could say. Say nothing instead.
+    if (!suites.length) {
+      console.log('[testforge] nothing ran — no run published');
+      return;
+    }
+    await publish(this.category, suites, durationSec);
   }
 };