@activate-core/test-forge 0.14.0 → 0.14.1

package/README.md

@@ -35,13 +35,44 @@ When your code changes later:
 testforge update     # re-check and refresh only what changed
 ```
 
+### Working on one journey
+
+`update` is the whole-project command: if the code has moved it re-maps,
+re-drafts **every** doc and regenerates. Most of the time you want one
+journey, and both steps take `--journey` (or `--area`):
+
+```bash
+# the spec for one journey changed — re-draft just that doc, then its tests
+testforge discover --journey payment
+testforge generate --journey payment
+
+# adding a journey that the reference now describes
+testforge discover --journey refund-flow
+testforge generate --journey refund-flow
+
+# everything in one area
+testforge generate --area ops
+```
+
+A journey **is** its doc: `generate` reads `docs/` and treats every markdown
+file there as one journey. So a doc you write by hand — for a feature the
+reference does not cover yet, or one taken from a ticket — works exactly like a
+drafted one. Write the doc, then `generate --journey <name>`.
+
+If you edited a doc yourself, skip `discover` entirely and go straight to
+`generate --journey <name>`.
+
+Regeneration replaces: when a journey is regenerated, test files for checks that
+no longer exist are deleted — but only files TestForge wrote. Your own tests are
+never touched.
+
 ## Commands
 
 | Command              | What it does                                                                                                                       |
 | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
 | `testforge setup`    | One-time setup: runs init, map, and discover in order, with progress.                                                              |
 | `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. |
-| `testforge update`   | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |
+| `testforge update`   | Refreshes the map, docs and tests when the **source code** or the **reference** has changed. Skips everything if neither has (no AI calls). Broad by design — for one journey, see below. |
 
 ### Advanced (run a single step)
 
@@ -51,6 +82,41 @@ testforge update     # re-check and refresh only what changed
 | `testforge map`      | Rebuild the project map (`testforge.md`) only.              |
 | `testforge discover` | Redraft the per-journey docs in `docs/` only.               |
 | `testforge env`      | Rebuild `.env.testing.example` from the docs only.          |
+| `testforge run`      | Run the generated tests (`--journey` / `--area` / `--env-file`). |
+| `testforge report`   | Publish results to the dashboard — `--results` after a run, `--inventory` to register every generated check. |
+
+### Flags worth knowing
+
+| Flag                       | On                     | What it does                                                                 |
+| -------------------------- | ---------------------- | ---------------------------------------------------------------------------- |
+| `--journey <name>`         | discover, generate, run | Work on one journey instead of all of them.                                  |
+| `--area <name>`            | discover, generate, run | Work on one area (api, web, ops…).                                           |
+| `--force`                  | generate               | Regenerate even when the doc has not changed.                                |
+| `--refresh-docs`           | update                 | Re-draft the docs even when nothing detectably changed — see below.          |
+| `--reference <file\|url>`  | setup                  | Base the docs on a spec instead of the code map.                             |
+| `--source map`             | setup                  | Base the docs on `testforge.md`, no prompt.                                  |
+
+#### When the reference cannot be checked
+
+`update` compares two things: a hash of your source code, and a hash of the
+reference the docs came from. A reference it can read (a file, or a plain URL)
+is compared directly, so a spec that moved while the code stood still is
+detected — that case used to pass silently as "everything is up to date".
+
+Some references cannot be read at all. A Notion or Confluence link answers a
+plain fetch with the application, not the document: no page content, and a build
+stamp that changes on every vendor deploy. Hashing that would be worse than not
+checking — it would report "unchanged" while the spec moved, and "changed" every
+time the vendor shipped. TestForge refuses it and tells you what it could not
+check:
+
+```
+No source changes detected.
+
+  Note: the reference could not be checked — app.notion.com serves an app
+  shell, not the document — its content needs the API.
+  If the specification has changed, run `testforge update --refresh-docs`.
+```
 
 ## How it works
 
@@ -70,8 +136,10 @@ becomes a Jest test (HTTP + database checks); `Type: web` becomes a
 Playwright test. Other areas use the framework set in config. It also
 generates a seed file that provisions the test data each test needs.
 
-**update** hashes your source code and compares it to the last run. No
-change means no work and no AI calls; a change triggers a refresh.
+**update** hashes your source code AND the reference the docs came from,
+and compares both to the last run. Neither changed means no work and no AI
+calls; either one changed triggers a refresh, and it says which. A reference it
+cannot read is reported as exactly that, never as "up to date".
 
 ## Source of truth
 
@@ -108,7 +176,7 @@ website, or a Notion page. testforge then:
 
 The reference becomes the source of truth — expected behavior from a human
 or client, not just what the code happens to do. The choice is remembered
-in `.testforge/source.json`, so a later `testforge update` refreshes the
+in `test-artifacts/source.json`, so a later `testforge update` refreshes the
 docs from the same source without asking again. Your review of
 `testforge.diff.md` stays as an audit of what differed and why.
 

package/dist/cli/index.js

@@ -13,6 +13,10 @@
  *   testforge generate                 Generate tests from the journey docs.
  *   testforge generate --basic         Use the built-in basic generator only.
  *   testforge generate --force         Regenerate everything, ignoring the cache.
+ *   testforge update                   Refresh only what changed (code or reference).
+ *   testforge update --refresh-docs    Re-derive the docs even with no detected
+ *                                      change — for a reference we cannot read
+ *                                      (an auth-gated page can move unseen).
  */
 import { VERSION } from '../core/index.js';
 import { runGenerate } from '../core/run-generate.js';
@@ -65,7 +69,7 @@ function printUsage() {
     console.log('Usage:');
     console.log('  testforge setup             Run init + map + discover in order');
     console.log('  testforge setup --reference <file|url>  Base the docs on a reference');
-    console.log('  testforge setup --source map            Base the docs on testforge.md');
+    console.log('  testforge setup --source map            Base the docs on testforge.md', '  testforge update --refresh-docs         Re-derive docs from the reference');
     console.log('  testforge generate           Generate tests from the journey docs');
     console.log('  testforge generate --basic   Use the basic generator only');
     console.log('  testforge generate --force   Regenerate everything');
@@ -83,7 +87,7 @@ function printUsage() {
 }
 async function main() {
     if (command === 'update') {
-        await runUpdate('.');
+        await runUpdate('.', { refreshDocs: process.argv.includes('--refresh-docs') });
     }
     else if (command === 'setup') {
         await runSetup('.', sourceOptions());

package/dist/core/ai-generator.js

@@ -129,18 +129,64 @@ export function buildPrompt(journey, map, testPath, focus, harnessPath, transpor
         `    test with a one-line reason naming exactly what was missing. Never`,
         `    let a missing precondition reach an assertion and report as a bug.`,
         `    Guard such tests at the top: check the precondition, skip if unmet.`,
+        `    A CREDENTIAL THAT EXISTS IS NOT A CREDENTIAL THAT WORKS. A supplied`,
+        `    token expires; treat "present" and "usable" as different checks.`,
+        `    Where the harness accepts a pre-minted token, verify it is still in`,
+        `    date (a JWT carries its own exp) before adopting it, and mint a`,
+        `    fresh one when it is not. If no working credential can be obtained,`,
+        `    skip and say so — a dead credential answers every authenticated`,
+        `    request with 401/403, and a suite that asserts on those reports an`,
+        `    entire product as broken when nothing is wrong with it.`,
         `(3) NO FLAKY TESTS. Every test must pass or fail for the same reason on`,
         `    every run, in any order. Do not depend on execution order, on data`,
         `    another test creates or mutates, on wall-clock timing, or on`,
         `    ambient totals that other tests change (assert on the rows YOU`,
         `    seeded, by id — never on a global count). Set up what you assert`,
         `    on, and assert only on that. Wait for a specific condition, never a`,
-        `    fixed sleep.`,
+        `    fixed sleep. Prefer a retrying assertion to a one-shot read: a value`,
+        `    the page writes after hydration, or re-renders on its way somewhere`,
+        `    else, is not there at the instant you first look. And a read that`,
+        `    runs in the page (evaluate) dies when a redirect lands mid-poll —`,
+        `    tolerate that error and let the poll retry, rather than reporting`,
+        `    a destroyed context as a product failure.`,
         `(4) NEVER ASSERT AN UNFIXED VALUE. Do not assert exact user-facing copy,`,
         `    incidental ordering, generated ids, timestamps, or counts the`,
         `    document does not fix. Assert the behaviour the document specifies,`,
         `    at the loosest form that still proves it.`,
-        `(5) A TIMEOUT IS NOT A BUG REPORT. If an action legitimately waits on`,
+        `(5) PROVE YOUR NEGATIVE INPUT IS ACTUALLY NEGATIVE. A security test that`,
+        `    feeds a "bad" credential, signature, id or payload is only as good as`,
+        `    the badness of that input. Construct it so it MUST differ from the`,
+        `    valid one at the level the product checks — mutate decoded bytes, not`,
+        `    an encoded character, since encodings carry spare bits that make`,
+        `    edits invisible. Where you can, assert the control too: the valid`,
+        `    input is accepted and the invalid one is not. An input that only`,
+        `    looks tampered turns correct behaviour into a reported breach, which`,
+        `    is the most damaging false alarm there is.`,
+        `(6) A STUB MUST NEVER INTERCEPT THE APPLICATION ITSELF. When you fake a`,
+        `    third party, match it by HOST (or an exact URL), never by a substring`,
+        `    of the URL: a pattern like "**evil.example**" also matches your own`,
+        `    page's "?next=https%3A%2F%2Fevil.example", so the app is served the`,
+        `    stub, nothing renders, and the test fails on a screen that never`,
+        `    loaded. Compare hostnames case-insensitively. After installing`,
+        `    stubs, satisfy yourself the page under test still loads.`,
+        `(7) BIND A FIXTURE TO THE IDENTITY THE SCREEN IS SCOPED BY. Before`,
+        `    seeding for a role-scoped page, find the exact key the server scopes`,
+        `    on — a user _id, a phone number, an account id — and which sign-in`,
+        `    the page uses. Seed against THAT identity. Do not reuse whatever the`,
+        `    harness minted first: the primary token is usually a different role,`,
+        `    and an auth/IdP subject is not the same value as the database id`,
+        `    (stripping a "sms|" prefix yields something id-SHAPED that belongs`,
+        `    to nobody). Fixtures hung off the wrong identity make every figure`,
+        `    read zero and every list read empty, which looks exactly like the`,
+        `    product losing data.`,
+        `(8) SEED ONLY STATES THE PRODUCT CAN REACH. Build every fixture the way`,
+        `    the code would: follow the transition that produces it and copy the`,
+        `    fields that transition writes. A hand-assembled record combining`,
+        `    fields no code path puts together is not a test case — the product`,
+        `    owes it no behaviour, so whatever it does with it proves nothing.`,
+        `    Before seeding a state, find the write that produces it; if there`,
+        `    is none, the scenario is imaginary — drop it.`,
+        `(9) A TIMEOUT IS NOT A BUG REPORT. If an action legitimately waits on`,
         `    something external, verify availability first and skip when absent.`,
         `    A test that times out tells the team nothing about the product.`,
         `When rules conflict, prefer reporting LESS: a pending test costs a line`,

package/dist/core/cache.js

@@ -105,3 +105,89 @@ export function hashProjectSource(rootDir) {
     walk(rootDir);
     return hashContent(parts.join('\n---\n'));
 }
+/**
+ * Is this response the document, or the application that would render it?
+ *
+ * Notion (and Confluence, and Google Docs) answer a plain fetch with 200 and a
+ * single-page-app shell: megabytes of script, none of the document's text, and
+ * a build stamp that changes every time they deploy. Hashing that is worse than
+ * not checking at all — it reports "unchanged" while the specification moves,
+ * and "changed" every time the vendor ships. Refuse it and say why.
+ */
+function looksLikeAppShell(url, body) {
+    let host = '';
+    try {
+        host = new URL(url).hostname.toLowerCase();
+    }
+    catch { /* keep '' */ }
+    const HOSTED_EDITORS = ['notion.so', 'notion.com', 'notion.site', 'atlassian.net', 'docs.google.com', 'sharepoint.com'];
+    if (HOSTED_EDITORS.some((h) => host === h || host.endsWith('.' + h))) {
+        return `${host} serves an app shell, not the document — its content needs the API`;
+    }
+    // Generic net for anything else that ships a shell: almost all markup and
+    // script, almost no readable text.
+    const text = body.replace(/<script[\s\S]*?<\/script>/gi, ' ')
+        .replace(/<style[\s\S]*?<\/style>/gi, ' ')
+        .replace(/<[^>]+>/g, ' ')
+        .replace(/\s+/g, ' ')
+        .trim();
+    if (body.length > 50_000 && text.length / body.length < 0.02) {
+        return 'the page returned an app shell rather than readable content';
+    }
+    if (/<title>[^<]*(log ?in|sign ?in|authenticate)/i.test(body)) {
+        return 'the page requires sign-in';
+    }
+    return null;
+}
+/**
+ * Hash the reference the journey docs are derived from.
+ *
+ * The source hash covers the CODE, which answers "has the implementation
+ * moved?". It says nothing about the specification. A reference that changes
+ * while the code stands still is the case that matters most — the docs, and so
+ * every assertion, are now out of date — and it used to pass silently as
+ * "everything is up to date".
+ *
+ * A local path is read directly. An http(s) reference is fetched. Anything we
+ * cannot read (a Notion page behind auth, no network) returns `unreadable` with
+ * the reason, so the caller can say so plainly instead of implying it checked.
+ */
+export async function hashReference(reference) {
+    const ref = (reference || '').trim();
+    if (!ref)
+        return { unreadable: 'no reference recorded' };
+    if (/^https?:\/\//i.test(ref)) {
+        try {
+            const res = await fetch(ref, { redirect: 'follow' });
+            if (!res.ok)
+                return { unreadable: `fetch returned ${res.status}` };
+            const body = await res.text();
+            const shell = looksLikeAppShell(ref, body);
+            if (shell)
+                return { unreadable: shell };
+            return { hash: hashContent(body) };
+        }
+        catch (e) {
+            return { unreadable: e.message.slice(0, 80) };
+        }
+    }
+    try {
+        const st = statSync(ref);
+        if (st.isDirectory()) {
+            const parts = [];
+            for (const entry of readdirSync(ref).sort()) {
+                const full = join(ref, entry);
+                try {
+                    if (statSync(full).isFile())
+                        parts.push(entry + ':' + hashContent(readFileSync(full, 'utf-8')));
+                }
+                catch { /* unreadable entry — skip */ }
+            }
+            return { hash: hashContent(parts.join('\n')) };
+        }
+        return { hash: hashContent(readFileSync(ref, 'utf-8')) };
+    }
+    catch (e) {
+        return { unreadable: e.message.slice(0, 80) };
+    }
+}

package/dist/core/report.js

@@ -99,6 +99,70 @@ export async function publishReport(opts) {
     });
     return { status: res.status, text: await res.text() };
 }
+const PENDING_CALL = /^[ \t]*(?:it|test)\.(?:todo|fixme)\(\s*(['"`])([\s\S]*?)\1/;
+/** Read the pending checks out of a test file, each with the reason recorded
+ *  beside it: the contiguous `//` comment block immediately above the call. */
+export function scanPending(file) {
+    let lines;
+    try {
+        lines = fs.readFileSync(file, 'utf8').split('\n');
+    }
+    catch {
+        return [];
+    }
+    const out = [];
+    for (let i = 0; i < lines.length; i++) {
+        const m = PENDING_CALL.exec(lines[i]);
+        if (!m)
+            continue;
+        const parts = [];
+        for (let j = i - 1; j >= 0 && j > i - 8; j--) {
+            const t = lines[j].trim();
+            if (!t.startsWith('//')) {
+                if (t)
+                    break;
+                else
+                    continue;
+            }
+            // Section banners ("// ----- Pending — ... -----") head a whole block and
+            // say nothing about THIS check; strip the rule and drop what is left if
+            // it is only decoration.
+            const body = t.replace(/^\/+\s?/, '').replace(/-{3,}/g, ' ').replace(/\s+/g, ' ').trim();
+            if (!body)
+                continue;
+            parts.unshift(body);
+        }
+        let reason = parts.join(' ').replace(/\s+/g, ' ').trim();
+        // A generic block header is not a reason for one check.
+        if (/^pending\s*[—-]/i.test(reason) && reason.length < 90)
+            reason = '';
+        if (reason.length < 15)
+            reason = '';
+        out.push({ title: m[2].replace(/\s+/g, ' ').trim().slice(0, 300), ...(reason ? { reason: reason.slice(0, 400) } : {}) });
+    }
+    return out;
+}
+/** Attach pending-case reasons to every suite whose file we can locate. */
+function attachPending(dir, ext, suites) {
+    const files = [];
+    try {
+        walk(dir, ext, files);
+    }
+    catch {
+        return;
+    }
+    const byName = new Map();
+    for (const f of files)
+        byName.set(suiteName(f), f);
+    for (const s of suites) {
+        const f = byName.get(s.name);
+        if (!f || s.pending)
+            continue;
+        const pending = scanPending(f);
+        if (pending.length)
+            s.pending = pending;
+    }
+}
 function walk(dir, ext, out) {
     for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
         const full = path.join(dir, e.name);
@@ -247,8 +311,10 @@ export function scanInventory(cwd, testOutput, ext, framework = {}) {
             continue;
         if (framework[category] === 'playwright') {
             // package root (holds playwright.config) is the parent of the tests dir
-            if (playwrightList(path.dirname(dir), category, suites))
+            if (playwrightList(path.dirname(dir), category, suites)) {
+                attachPending(dir, ext, suites);
                 continue;
+            }
             console.warn(`  ! playwright --list failed for "${category}" — using static estimate`);
             jestStatic(dir, ext, category, suites);
         }
@@ -256,6 +322,7 @@ export function scanInventory(cwd, testOutput, ext, framework = {}) {
             // jest (and ops/lambda, also jest-based) — accurate collect-only counter
             jestCount(dir, ext, category, suites);
         }
+        attachPending(dir, ext, suites);
     }
     return suites;
 }

package/dist/core/update.js

@@ -12,11 +12,13 @@ import { existsSync } from 'node:fs';
 import { runMap } from './map.js';
 import { runDiscover } from './discover.js';
 import { runGenerate } from './run-generate.js';
-import { hashProjectSource, loadCache, saveCache } from './cache.js';
+import { hashProjectSource, hashReference, loadCache, saveCache } from './cache.js';
+import { loadSourceChoice } from './source-of-truth.js';
 import { runEnvTemplate } from './env-template.js';
 import { CACHE_PATH } from './paths.js';
 const SOURCE_KEY = '__source__';
-export async function runUpdate(srcDir) {
+const REFERENCE_KEY = '__reference__';
+export async function runUpdate(srcDir, opts = {}) {
     if (!existsSync('testforge.md') || !existsSync('testforge.config.json')) {
         console.log("This project isn't set up yet.");
         console.log('Run `testforge setup` first.');
@@ -26,12 +28,41 @@ export async function runUpdate(srcDir) {
     const cache = loadCache(CACHE_PATH);
     const oldHash = cache[SOURCE_KEY];
     const newHash = hashProjectSource(srcDir);
-    if (oldHash === newHash) {
-        console.log('No source changes detected. Everything is up to date.');
-        console.log('(Nothing regenerated — no AI calls made.)');
+    const codeChanged = oldHash !== newHash;
+    // The docs are the source of truth for every assertion, so a reference that
+    // moved matters at least as much as code that moved. Checking only the code
+    // meant a changed spec was reported as "everything is up to date" — the tests
+    // then kept asserting a specification nobody uses any more.
+    const src = loadSourceChoice() || { mode: 'map' };
+    const refState = src.mode === 'reference' && src.reference
+        ? await hashReference(src.reference)
+        : { unreadable: 'docs are derived from the code map, not a reference' };
+    const refChanged = Boolean(refState.hash) && cache[REFERENCE_KEY] !== undefined && cache[REFERENCE_KEY] !== refState.hash;
+    const refFirstSeen = Boolean(refState.hash) && cache[REFERENCE_KEY] === undefined;
+    if (!codeChanged && !refChanged && !opts.refreshDocs) {
+        console.log(`No source changes detected.${refState.hash ? ' The reference is unchanged too.' : ''}`);
+        if (refState.unreadable && src.mode === 'reference') {
+            // Say what was NOT checked. Claiming "up to date" while the specification
+            // could have moved unseen is the one thing this command must not do.
+            console.log(`\n  Note: the reference could not be checked — ${refState.unreadable}.`);
+            console.log(`  ${src.reference}`);
+            console.log('  If the specification has changed, run `testforge update --refresh-docs`.');
+        }
+        else {
+            console.log('(Nothing regenerated — no AI calls made.)');
+        }
+        if (refFirstSeen) {
+            cache[REFERENCE_KEY] = refState.hash;
+            saveCache(CACHE_PATH, cache);
+        }
         return;
     }
-    console.log('Source code changed — updating the project.\n');
+    const why = [
+        codeChanged ? 'source code changed' : null,
+        refChanged ? 'the reference changed' : null,
+        opts.refreshDocs && !codeChanged && !refChanged ? 'refresh requested' : null,
+    ].filter(Boolean).join(' and ');
+    console.log(`Updating the project — ${why}.\n`);
     console.log('Re-mapping the project...');
     runMap(srcDir);
     console.log('  done — testforge.md\n');
@@ -46,6 +77,8 @@ export async function runUpdate(srcDir) {
     const skipped = results.filter((r) => r.status === 'skipped').length;
     console.log(`  done — ${made} updated, ${skipped} unchanged.\n`);
     cache[SOURCE_KEY] = newHash;
+    if (refState.hash)
+        cache[REFERENCE_KEY] = refState.hash;
     saveCache(CACHE_PATH, cache);
     console.log('Refreshing env template...');
     runEnvTemplate('docs', '.env.testing.example', srcDir);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activate-core/test-forge",
-  "version": "0.14.0",
+  "version": "0.14.1",
   "description": "AI-powered QA automation. Scans a project, drafts journey docs, and generates seeded Jest/Playwright tests. Works in any project, with any AI CLI.",
   "type": "module",
   "bin": {

package/reporters/jest-reporter.cjs

@@ -27,6 +27,13 @@ module.exports = class TestForgeJestReporter {
       }
     }
     const durationSec = Math.round((results.startTime ? Date.now() - results.startTime : 0) / 1000);
-    await publish(this.category, Object.values(suites), durationSec);
+    const list = Object.values(suites);
+    // Same reasoning as the Playwright reporter: no suites means nothing ran,
+    // and a published 0/0/0 run reads as a clean result.
+    if (!list.length) {
+      console.log('[testforge] nothing ran — no run published');
+      return;
+    }
+    await publish(this.category, list, durationSec);
   }
 };

package/reporters/playwright-reporter.cjs

@@ -26,6 +26,15 @@ module.exports = class TestForgePlaywrightReporter {
   }
   async onEnd(result) {
     const durationSec = Math.round(((result && result.duration) || (Date.now() - this.start)) / 1000);
-    await publish(this.category, Object.values(this.suites), durationSec);
+    const suites = Object.values(this.suites);
+    // A run that collected no suites did not test anything: global setup failed,
+    // or the filter matched nothing. Publishing it records a 0/0/0 result that
+    // reads as "the suite ran and found nothing wrong" — the most misleading
+    // thing this reporter could say. Say nothing instead.
+    if (!suites.length) {
+      console.log('[testforge] nothing ran — no run published');
+      return;
+    }
+    await publish(this.category, suites, durationSec);
   }
 };