@activate-core/test-forge 0.14.1 → 0.15.0

package/README.md

@@ -136,6 +136,14 @@ becomes a Jest test (HTTP + database checks); `Type: web` becomes a
 Playwright test. Other areas use the framework set in config. It also
 generates a seed file that provisions the test data each test needs.
 
+**generate** hands each file it writes to the project's own ESLint with
+`--fix` before moving on. Formatting differences are corrected in place — CI
+runs lint before it runs a test, and a file that is correct but indented
+differently still fails the build. Anything the linter cannot fix (an unused
+declaration, say) is reported as a warning against that journey, because a
+declaration nothing checks is worth a person's attention. If the project has no
+ESLint, the step is skipped.
+
 **update** hashes your source code AND the reference the docs came from,
 and compares both to the last run. Neither changed means no work and no AI
 calls; either one changed triggers a refresh, and it says which. A reference it
@@ -272,7 +280,8 @@ The test runner produces its own pass/fail report.
     "ops": "playwright"
   },
   "seedOutput": "packages/api/test/helpers/seed",
-  "testExtension": ".test.ts"
+  "testExtension": ".test.ts",
+  "externalModules": ["shipengine", "@customerio/cdp-analytics"]
 }
 ```
 
@@ -289,6 +298,15 @@ The test runner produces its own pass/fail report.
 - **framework** — which test framework each area uses.
 - **seedOutput** — where seed files go.
 - **testExtension** — `.test.ts` or `.test.js`.
+- **externalModules** — the third-party network services this project
+  talks to, named as a test would import them. A generated backend test
+  that imports one without mocking it is rejected before it is written:
+  left alone it would reach the live service, so it breaks offline and
+  can fire real charges, messages or shipments. testforge already knows
+  the common SDKs (payment, SMS, email, cloud, identity) and this list is
+  added to them, never instead of them — so name only what is particular
+  to your product. Every domain has something: a shipping carrier, a CMS,
+  a ticketing or streaming backend, an ad network.
 
 ## Notes
 

package/dist/core/ai-generator.js

@@ -189,6 +189,28 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
         `(9) A TIMEOUT IS NOT A BUG REPORT. If an action legitimately waits on`,
         `    something external, verify availability first and skip when absent.`,
         `    A test that times out tells the team nothing about the product.`,
+        `(10) THE HARNESS IS SOURCE TOO — READ IT BEFORE YOU CALL IT. Open the`,
+        `    helper you are about to use and check what it actually RETURNS: a`,
+        `    setup routine that publishes env and returns nothing hands you`,
+        `    undefined, and every later reference through it dies in beforeAll,`,
+        `    taking the whole file down as "failures" the product never caused.`,
+        `    The same applies to locators: a query must resolve to exactly ONE`,
+        `    node. Component libraries label a control twice (the <label> and the`,
+        `    widget both answer getByLabel), so prefer the role the widget`,
+        `    actually exposes. Where a sibling test in this journey already`,
+        `    drives the same control, reuse ITS helper verbatim rather than`,
+        `    writing a second one that has to rediscover the same trap.`,
+        `(11) LEAVE SHARED STATE AS YOU FOUND IT, EVEN WHEN YOU FAIL. A fixture`,
+        `    every other suite reads — a shared account, the one seeded record`,
+        `    the whole journey keys on — must be put back the moment you are`,
+        `    done with it. Read the value first, restore it in a finally or an`,
+        `    after-EACH hook, never an after-ALL: the specs that mutate shared`,
+        `    state are usually the ones asserting the write is REFUSED, so they`,
+        `    throw by design the day the product is wrong, and a worker that dies`,
+        `    on that throw never reaches after-all. The forged value then sits in`,
+        `    the fixture and every later suite reads it as real — one defect`,
+        `    becomes a hundred failures in journeys that were never broken, and`,
+        `    the run cannot be trusted until someone repairs the data by hand.`,
         `When rules conflict, prefer reporting LESS: a pending test costs a line`,
         `in the report; a false failure costs the team a release.`,
         ``,
@@ -220,8 +242,8 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
         `call in the file.`,
         `AUTH & SCOPING: a route may be declared auth:false yet STILL populate`,
         `request.user when a Bearer token is sent (a pre-auth extension). So`,
-        `owner-scoped data ("my bookings/payouts") is derived from the TOKEN`,
-        `owner — NOT from a query param. Do NOT assume a "?ownerId=/?makerId="`,
+        `owner-scoped data ("my orders", "my messages") is derived from the`,
+        `TOKEN owner — NOT from a query param. Do NOT assume an "?ownerId="`,
         `fallback overrides the token for an AUTHENTICATED call: the testkit`,
         `seeds the owner fixtures under the authenticated identity, so expect`,
         `the token owner's rows. A second, non-owner identity (for IDOR/scope`,
@@ -250,13 +272,13 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
                 `process.env.DB_URL) — never hardcode a connection string.`,
                 `MOCKING (MANDATORY — the test must be HERMETIC): the test must`,
                 `NEVER depend on a live external service. For EVERY external`,
-                `service the exercised code reaches (identity/IM, payment/Stripe,`,
-                `SMS, email, etc.), mock it AT THE SEAM THE CODE ACTUALLY USES —`,
-                `read the source to see which client it imports (e.g. axios, the`,
-                `stripe SDK) and how it is called, then jest.mock that EXACT module`,
+                `service the exercised code reaches — whatever this product happens`,
+                `to call out to — mock it AT THE SEAM THE CODE ACTUALLY USES —`,
+                `read the source to see which client it imports (an HTTP client, a`,
+                `vendor SDK) and how it is called, then jest.mock that EXACT module`,
                 `so the mock truly intercepts. A test that leaves any such call`,
-                `unmocked (so it would hit a real host like localhost/an IM/Stripe`,
-                `URL) is WRONG — it will fail whenever that service is down and is`,
+                `unmocked (so it would reach a real host rather than your mock)`,
+                `is WRONG — it will fail whenever that service is down and is`,
                 `not deterministic. Do NOT mock a client the code doesn't use.`,
                 `HERMETIC BASE URLS: if the client's target is built from an env`,
                 `var (e.g. process.env.IM_API_URL), do NOT rely on the ambient`,

package/dist/core/config.js

@@ -38,6 +38,7 @@ const DEFAULT_CONFIG = {
     testExtension: '.test.ts',
     testCommand: {},
     envFile: '.env.testing',
+    externalModules: [],
 };
 /** Load config from the project, falling back to defaults. */
 export function loadConfig() {

package/dist/core/harness.js

@@ -58,7 +58,7 @@ export const HARNESS_CONTRACT = [
     `                                  EVERY id MUST be env-overridable: resolve`,
     `                                  each as process.env.TEST_<NAME>_ID ||`,
     `                                  '<stable default>' (e.g. TEST_PROJECT_ID,`,
-    `                                  TEST_WEARER_ID). This lets a caller point`,
+    `                                  TEST_OWNER_ID). This lets a caller point`,
     `                                  the suite at data created BY HAND on`,
     `                                  staging/prod (with SEED_MODE=existing)`,
     `                                  without editing code. Resolve non-id`,
@@ -116,8 +116,8 @@ export const HARNESS_CONTRACT = [
     `                                  the UI shows the seeded data.`,
     `Use only env var NAMES for secrets/connection — never literal values.`,
     `The ONLY credentials a developer should have to provide are the DB`,
-    `connection, a Stripe test key, and ONE seed/refresh token (or seeded`,
-    `login) — everything else setup() derives at runtime.`,
+    `connection, a test key for any paid third party, and ONE seed/refresh`,
+    `token (or seeded login) — everything else setup() derives at runtime.`,
     ``,
     `SAFETY — this testkit must NEVER be able to damage a real environment:`,
     `  - reset() and ANY delete/drop MUST refuse unless BOTH hold: the opt-in`,
@@ -140,9 +140,9 @@ export const HARNESS_CONTRACT = [
     `extension when a Bearer is sent. Therefore:`,
     `  - setup() MUST resolve the authenticated identity's REAL id (decode`,
     `    the minted token's subject/_id claim, or call the profile/whoami`,
-    `    endpoint) and seed the OWNER fixtures (the maker/wearer/customer`,
-    `    whose data the tests read) under THAT id — then set the TEST_IDS`,
-    `    owner ids to it.`,
+    `    endpoint) and seed the OWNER fixtures (whichever role in this`,
+    `    product owns the data the tests read) under THAT id — then set`,
+    `    the TEST_IDS owner ids to it.`,
     `  - Do NOT hardcode an owner id the token does not own: the seeded rows`,
     `    would never be returned to the authenticated caller, and every`,
     `    owner-scoped test would fail for a fake reason.`,
@@ -392,15 +392,15 @@ export function fixturesDoc() {
         ``,
         `## Running against data you created by hand (staging / prod)`,
         `1. Create the fixture entities in the target database yourself (a`,
-        `   project, its price line(s), the wearer/maker users, etc.) — mirror`,
-        `   what the seed builders in \`helpers/testkit\` produce.`,
+        `   parent record, its line items, the users on each side, etc.) —`,
+        `   mirror what the seed builders in \`helpers/testkit\` produce.`,
         `2. Point the suite at them — every well-known id is env-overridable:`,
         ``,
         `      TEST_PROJECT_ID=<your project id>`,
-        `      TEST_WEARER_ID=<...>        # and any other TEST_*_ID the suite reads`,
+        `      TEST_OWNER_ID=<...>         # and any other TEST_*_ID the suite reads`,
         ``,
         `3. Provide auth + connection env (DB url/name(s), a seed/refresh token`,
-        `   or seeded login, Stripe test key).`,
+        `   or seeded login, any third-party test key).`,
         `4. Run with \`SEED_MODE=existing\` so nothing is seeded or deleted.`,
         ``,
         `## Safety`,

package/dist/core/lint.js

@@ -0,0 +1,105 @@
+/**
+ * testforge — run the project's own linter over what we generated.
+ *
+ * A generated file is written to a repository that already has a house style,
+ * and CI usually runs `npm run lint` before it runs a single test. Files that
+ * are correct but indented differently fail that build, and somebody then fixes
+ * two hundred formatting errors by hand — work the generator caused and the
+ * generator can undo.
+ *
+ * So: after writing, hand the files to the project's ESLint with --fix. Style
+ * problems disappear silently, because they were never interesting. Anything
+ * left is reported, because an unused declaration in a test is a claim nothing
+ * checks, and that is worth a human's attention.
+ *
+ * The project's config is used, never one of ours — the point is to match the
+ * repository, not to impose a preference.
+ */
+import { spawnSync } from 'node:child_process';
+import { existsSync } from 'node:fs';
+import { join, dirname } from 'node:path';
+/** Walk up from a file to the nearest package that has ESLint installed. */
+function eslintFor(file) {
+    let dir = dirname(file);
+    for (let i = 0; i < 8; i++) {
+        const bin = join(dir, 'node_modules', '.bin', 'eslint');
+        if (existsSync(bin))
+            return { cwd: dir, bin };
+        const up = dirname(dir);
+        if (up === dir)
+            break;
+        dir = up;
+    }
+    return null;
+}
+/**
+ * Lint (and fix) the given files. Returns what was fixed and what was not.
+ * Never throws: a missing or misconfigured linter must not fail a generation
+ * that otherwise succeeded.
+ */
+export function lintGenerated(files) {
+    const real = files.filter((f) => f && existsSync(f));
+    if (!real.length)
+        return { checked: 0, fixed: 0, remaining: [] };
+    // Group by the package whose ESLint owns them — a monorepo has one per app.
+    const groups = new Map();
+    for (const f of real) {
+        const found = eslintFor(f);
+        if (!found)
+            continue;
+        const g = groups.get(found.cwd) || { bin: found.bin, files: [] };
+        g.files.push(f);
+        groups.set(found.cwd, g);
+    }
+    if (!groups.size) {
+        return { checked: 0, fixed: 0, remaining: [], skipped: 'no ESLint found in the project' };
+    }
+    let checked = 0;
+    let fixed = 0;
+    const remaining = [];
+    for (const [cwd, g] of groups) {
+        const before = countProblems(g.bin, cwd, g.files);
+        spawnSync(g.bin, ['--fix', '--no-error-on-unmatched-pattern', ...g.files], {
+            cwd,
+            encoding: 'utf-8',
+            timeout: 120_000,
+        });
+        const after = readProblems(g.bin, cwd, g.files);
+        checked += g.files.length;
+        fixed += Math.max(0, before - after.count);
+        remaining.push(...after.lines);
+    }
+    return { checked, fixed, remaining };
+}
+function countProblems(bin, cwd, files) {
+    return readProblems(bin, cwd, files).count;
+}
+function readProblems(bin, cwd, files) {
+    const res = spawnSync(bin, ['-f', 'json', '--no-error-on-unmatched-pattern', ...files], {
+        cwd,
+        encoding: 'utf-8',
+        timeout: 120_000,
+        maxBuffer: 32 * 1024 * 1024,
+    });
+    let parsed;
+    try {
+        parsed = JSON.parse(res.stdout || '[]');
+    }
+    catch {
+        return { count: 0, lines: [] }; // linter unavailable or output unusable
+    }
+    const lines = [];
+    let count = 0;
+    for (const file of parsed) {
+        for (const m of file.messages) {
+            if (m.severity !== 2)
+                continue; // warnings do not fail a build
+            count += 1;
+            if (lines.length < 20) {
+                const rel = file.filePath.replace(cwd + '/', '');
+                lines.push(`${rel}:${m.line} — ${m.message}${m.ruleId ? ` (${m.ruleId})` : ''}`);
+            }
+        }
+    }
+    return { count, lines };
+}

package/dist/core/report.js

@@ -207,8 +207,15 @@ for (const f of ['jest.config.js','jest.config.cjs','jest.config.json']) {
   if (fs.existsSync(path.join(__dirname, f))) { base = require('./' + f); break; }
 }
 if (!Object.keys(base).length && fs.existsSync(path.join(__dirname,'package.json'))) base = require('./package.json').jest || {};
+// Strip the project's reporters. Spreading them in meant OUR OWN publish
+// reporter ran during a count — and since every test body is a no-op here, it
+// published a full, entirely green run over the real results. A counting pass
+// must be invisible: it reports nothing, because it tested nothing.
+const { reporters, ...rest } = base;
+void reporters;
 module.exports = {
-  ...base,
+  ...rest,
+  reporters: ['default'],
   testEnvironment: 'node', // plain node env for a count-only run
   setupFilesAfterEnv: [...(base.setupFilesAfterEnv || []), path.resolve(__dirname, '.tf-collect-patch.cjs')],
 };
@@ -276,7 +283,10 @@ function jestCount(dir, ext, category, out) {
  */
 function playwrightList(pkgDir, category, out) {
     try {
-        const res = execSync('npx playwright test --list', {
+        // `--reporter=list` so the project's own reporters — ours included — do
+        // not run during a count. Listing publishes nothing on its own, but a
+        // reporter that fires on end would report a run that never happened.
+        const res = execSync('npx playwright test --list --reporter=list', {
             cwd: pkgDir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], timeout: 180000,
         });
         const counts = {};

package/dist/core/run-generate.js

@@ -19,6 +19,7 @@
 import { readFileSync, writeFileSync, mkdirSync, existsSync, readdirSync, rmSync, } from 'node:fs';
 import { join, dirname } from 'node:path';
 import { CACHE_PATH, COVERAGE_PATH, PLANS_DIR, STATE_DIR } from './paths.js';
+import { lintGenerated } from './lint.js';
 import { parseJourney } from './parser.js';
 import { generateTestCases } from './generator.js';
 import { writeDeferredStub } from './playwright-writer.js';
@@ -402,7 +403,12 @@ export async function runGenerate(docsDir, useAI = true, force = false, only, ar
                 selectorsByArea.set(areaDir, knownSelectors);
             }
         }
-        const testGate = (content) => checkTestGates(content, ext, { framework, allowedModules, knownSelectors });
+        const testGate = (content) => checkTestGates(content, ext, {
+            framework,
+            allowedModules,
+            knownSelectors,
+            externalModules: config.externalModules,
+        });
         // Skip if unchanged — but NOT if the journey still has deferred stubs
         // to retry (a resume pass fills those in without touching real files).
         if (!force && cache[file] === hash && !hasDeferredStubs(testDir, ext)) {
@@ -561,6 +567,23 @@ export async function runGenerate(docsDir, useAI = true, force = false, only, ar
         // it — so a later `generate` (once the limit resets) RE-DOES it with
         // AI instead of skipping it as "done". Persist per-journey so an
         // interrupted run still resumes from where it stopped.
+        // Hand what we just wrote to the project's own linter. CI runs lint before
+        // it runs a test, so a file that is correct but formatted differently fails
+        // the build — and somebody fixes it by hand afterwards. Style is fixed in
+        // place; whatever survives is reported, because an unused declaration in a
+        // test is a claim nothing checks.
+        const lintTargets = [
+            ...testFiles.map((t) => (typeof t === 'string' ? t : t.path)),
+            ...(seedPath ? [seedPath] : []),
+        ];
+        const lint = lintGenerated(lintTargets);
+        if (lint.fixed > 0) {
+            console.log(`  tidied ${lint.fixed} lint problem(s) in ${lint.checked} generated file(s)`);
+        }
+        if (lint.remaining.length) {
+            warnings.push(`lint: ${lint.remaining.length} problem(s) the linter could not fix — ` +
+                lint.remaining.slice(0, 5).join('; '));
+        }
         const degraded = useAI && (status === 'basic' || seedStatus === 'failed');
         if (!degraded) {
             cache[file] = hash;

package/dist/core/validate.js

@@ -20,6 +20,28 @@ import { execFileSync } from 'node:child_process';
 import { writeFileSync, mkdtempSync, rmSync } from 'node:fs';
 import { join } from 'node:path';
 import { tmpdir } from 'node:os';
+/**
+ * Third-party network SDKs common enough to be worth gating everywhere:
+ * a backend test that imports one without mocking it is talking to a real
+ * service, which flakes offline and can fire real charges, messages or
+ * emails. Deliberately NOT exhaustive and deliberately NOT tied to any one
+ * industry — every project names its own in `externalModules`, and a
+ * shipping API, a CMS, a games backend or an ad network is as much a live
+ * dependency as a payment provider. Database clients are excluded on
+ * purpose: a backend integration test may legitimately use a real local DB.
+ */
+export const DEFAULT_EXTERNAL_MODULES = [
+    // payment
+    'stripe', 'braintree', '@paypal/checkout-server-sdk', 'razorpay', 'square',
+    // messaging / email
+    'twilio', '@sendgrid/mail', '@sendgrid/client', 'nodemailer', 'postmark',
+    'mailgun.js', '@slack/web-api', 'resend',
+    // cloud
+    'aws-sdk', '@aws-sdk/client-ses', '@aws-sdk/client-sns', '@aws-sdk/client-s3',
+    '@google-cloud/storage', '@azure/storage-blob',
+    // identity / support / analytics
+    'auth0', 'firebase-admin', 'node-zendesk', 'intercom-client', 'analytics-node',
+];
 /**
  * Semantic gate for a generated test file. Returns a human-readable
  * reason string if the content violates a guardrail, or null if it is
@@ -70,14 +92,14 @@ export function checkTestGates(content, _ext, ctx) {
         }
     }
     // F11 — hermetic seam (jest/backend): a test that imports a known
-    // EXTERNAL network service (payment/SMS/email/cloud) must mock it, or the
+    // EXTERNAL network service must mock it, or the
     // suite silently depends on a live third party — it flakes offline and can
     // fire real charges/messages. DB clients are intentionally NOT listed:
     // backend integration tests legitimately use a real (local) database.
     if (ctx.framework === 'jest') {
         const EXTERNAL = new Set([
-            'stripe', 'twilio', '@sendgrid/mail', '@sendgrid/client',
-            'nodemailer', 'aws-sdk', '@aws-sdk/client-ses', '@aws-sdk/client-sns',
+            ...DEFAULT_EXTERNAL_MODULES,
+            ...(ctx.externalModules || []),
         ]);
         for (const mod of importedModules(content)) {
             const top = topLevelModule(mod);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activate-core/test-forge",
-  "version": "0.14.1",
+  "version": "0.15.0",
   "description": "AI-powered QA automation. Scans a project, drafts journey docs, and generates seeded Jest/Playwright tests. Works in any project, with any AI CLI.",
   "type": "module",
   "bin": {