@activate-core/test-forge 0.2.0 → 0.3.0

package/README.md

@@ -40,7 +40,7 @@ testforge update     # re-check and refresh only what changed
 | Command              | What it does                                                                                                                       |
 | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
 | `testforge setup`    | One-time setup: runs init, map, and discover in order, with progress.                                                              |
-| `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. First asks what the tests should be based on — the AI-derived docs, or an external reference (see [Source of truth](#source-of-truth)). |
+| `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated. |
 | `testforge update`   | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |
 
 ### Advanced (run a single step)
@@ -75,18 +75,22 @@ change means no work and no AI calls; a change triggers a refresh.
 
 ## Source of truth
 
-By default, testforge's docs are derived from your **code** — the AI reads
-the project and writes `testforge.md` and the journey docs. That is fast,
-but it inherits whatever the code does: if a flow is buggy or a scenario
-is mishandled, the AI faithfully documents the wrong behavior, and the
-tests then lock that wrong behavior in.
+The journey docs in `docs/` are the source of truth for the tests —
+`generate` reads the docs and nothing else. So where the docs come from
+decides everything.
 
-So when you run `testforge generate`, it first asks what the tests should
-be based on:
+By default the docs are derived from your **code** (the AI reads the
+project, writes `testforge.md`, then drafts the docs from it). That is
+fast, but it inherits whatever the code does: if a flow is buggy or a
+scenario is mishandled, the AI faithfully documents the wrong behavior,
+and the tests then lock that wrong behavior in.
+
+So when the docs are drafted — during `testforge setup` (or `testforge
+discover`) — testforge asks what they should be based on:
 
 ```
-What should the tests be based on?
-  [1] testforge.md + docs/ (what the AI derived from your code) (default)
+What should the journey docs be based on?
+  [1] testforge.md (what the AI derived from your code) (default)
   [2] An external reference (a file/PDF in scope, a website, or a Notion link)
 ```
 
@@ -94,27 +98,31 @@ Choose **[2]** to give testforge an authoritative reference of the
 _expected_ behavior — a spec or requirements doc in the repo, a PDF, a
 website, or a Notion page. testforge then:
 
-1. reads the reference and compares it against the current docs,
+1. reads the reference and compares it against `testforge.md`,
 2. writes a full comparison to **`testforge.diff.md`** and prints a short
    summary of the differences,
-3. asks you to confirm, and on confirmation **reconciles the docs** so the
-   reference wins (fixing wrong expectations, adding missing journeys),
-4. generates the tests from those reconciled docs.
+3. asks you to confirm, and on confirmation **drafts the docs from the
+   reference** — expected behavior comes from the reference, while routes,
+   fields, models, and env var names come from the code/map,
+4. `generate` then turns those docs into tests as usual.
 
 The reference becomes the source of truth — expected behavior from a human
-or client, not just what the code happens to do. Your review of
-`testforge.diff.md` stays as an audit of what changed and why.
+or client, not just what the code happens to do. The choice is remembered
+in `.testforge/source.json`, so a later `testforge update` refreshes the
+docs from the same source without asking again. Your review of
+`testforge.diff.md` stays as an audit of what differed and why.
 
 **Non-interactive use (CI, scripts):**
 
 ```bash
-testforge generate --source docs              # skip the prompt, use the docs as-is
-testforge generate --reference ./spec.md      # reconcile against a reference, no prompts
-testforge generate --reference https://…      # a URL or Notion link works too
+testforge setup --source map              # base the docs on testforge.md, no prompt
+testforge setup --reference ./spec.md     # base the docs on a reference, no prompts
+testforge setup --reference https://…     # a URL or Notion link works too
 ```
 
-With no flags and no terminal attached, testforge does not block — it
-falls back to the docs and prints a note.
+The same flags work on `testforge discover`. With no flags and no terminal
+attached, testforge does not block — it falls back to `testforge.md` and
+prints a note.
 
 ## Running the generated tests
 

package/dist/cli/index.js

@@ -3,15 +3,19 @@
  * testforge — command-line entry point.
  *
  * Commands:
- *   testforge generate                 Generate tests for changed journeys (AI).
+ *   testforge setup                    init + map + discover. discover asks
+ *                                      whether the docs come from testforge.md
+ *                                      or an external reference (the source of
+ *                                      truth for every test).
+ *   testforge setup --reference <x>    Base the docs on reference <x> (a file
+ *                                      path, URL, or Notion link) — no prompt.
+ *   testforge setup --source map       Base the docs on testforge.md — no prompt.
+ *   testforge generate                 Generate tests from the journey docs.
  *   testforge generate --basic         Use the built-in basic generator only.
  *   testforge generate --force         Regenerate everything, ignoring the cache.
- *   testforge generate --source docs   Skip the source-of-truth prompt, use docs.
- *   testforge generate --reference <x> Reconcile docs against a reference first,
- *                                      where <x> is a file path, URL, or Notion link.
  */
 import { VERSION } from '../core/index.js';
-import { runGenerateCommand } from '../core/generate-command.js';
+import { runGenerate } from '../core/run-generate.js';
 import { runDiscover } from '../core/discover.js';
 import { runMap } from '../core/map.js';
 import { runInit } from '../core/init.js';
@@ -31,65 +35,102 @@ function flagValue(name) {
         return undefined;
     return value;
 }
-if (command === 'update') {
-    runUpdate('.');
+/** Source-of-truth options shared by setup and discover. */
+function sourceOptions() {
+    return { reference: flagValue('--reference'), source: flagValue('--source') };
 }
-else if (command === 'setup') {
-    runSetup('.');
-}
-else if (command === 'init') {
-    console.log('Inspecting your project to set up testforge...\n');
-    const output = runInit();
-    console.log(output);
-    console.log('\ntestforge.config.json written. Review it, then run `testforge map`.');
-}
-else if (command === 'map') {
-    console.log('Building a high-level map of your project with AI...');
-    console.log('This scans your source code and writes testforge.md\n');
-    const output = runMap('.');
-    console.log(output);
-    console.log('\nProject map written to testforge.md');
-}
-else if (command === 'discover') {
-    console.log('Scanning your project with AI to draft journey files...');
-    console.log('This reads your source code and writes drafts into docs/.\n');
-    const output = runDiscover('.', 'docs');
-    console.log(output);
-    console.log('\nDraft journeys written to docs/. Please review and edit them,');
-    console.log('then run `testforge generate`.');
-}
-else if (command === 'env') {
-    runEnvTemplate('docs', '.env.testing.example');
-}
-else if (command === 'generate') {
-    const useAI = !useBasic;
-    console.log(useAI
-        ? 'Generating tests with AI for changed journeys...\n'
-        : 'Generating tests with the basic generator...\n');
-    runGenerateCommand({
-        docsDir: 'docs',
-        useAI,
-        force,
-        reference: flagValue('--reference'),
-        source: flagValue('--source'),
-    }).catch((err) => {
-        console.error(err instanceof Error ? err.message : String(err));
-        process.exit(1);
-    });
-}
-else {
+function printUsage() {
     console.log(`testforge v${VERSION}`);
     console.log('');
     console.log('Usage:');
-    console.log('  testforge generate           Generate tests for changed journeys');
+    console.log('  testforge setup             Run init + map + discover in order');
+    console.log('  testforge setup --reference <file|url>  Base the docs on a reference');
+    console.log('  testforge setup --source map            Base the docs on testforge.md');
+    console.log('  testforge generate           Generate tests from the journey docs');
     console.log('  testforge generate --basic   Use the basic generator only');
     console.log('  testforge generate --force   Regenerate everything');
-    console.log('  testforge generate --source docs        Skip the source-of-truth prompt');
-    console.log('  testforge generate --reference <file|url>  Reconcile docs against a reference first');
-    console.log('  testforge discover          Scan source code and draft journey files');
+    console.log('  testforge discover          Draft journey docs (asks: map or reference)');
     console.log('  testforge map               Scan project, write journey + data map');
     console.log('  testforge init              Inspect project and write config');
-    console.log('  testforge setup             Run init + map + discover in order');
     console.log('  testforge update            Re-check and update only what changed');
     console.log('  testforge env               Write .env.testing.example from docs');
 }
+async function main() {
+    if (command === 'update') {
+        await runUpdate('.');
+    }
+    else if (command === 'setup') {
+        await runSetup('.', sourceOptions());
+    }
+    else if (command === 'init') {
+        console.log('Inspecting your project to set up testforge...\n');
+        const output = runInit();
+        console.log(output);
+        console.log('\ntestforge.config.json written. Review it, then run `testforge map`.');
+    }
+    else if (command === 'map') {
+        console.log('Building a high-level map of your project with AI...');
+        console.log('This scans your source code and writes testforge.md\n');
+        const output = runMap('.');
+        console.log(output);
+        console.log('\nProject map written to testforge.md');
+    }
+    else if (command === 'discover') {
+        console.log('Drafting journey files with AI...');
+        console.log('This writes drafts into docs/ from your chosen source.\n');
+        const output = await runDiscover('.', 'docs', sourceOptions());
+        console.log(output);
+        console.log('\nDraft journeys written to docs/. Please review and edit them,');
+        console.log('then run `testforge generate`.');
+    }
+    else if (command === 'env') {
+        runEnvTemplate('docs', '.env.testing.example');
+    }
+    else if (command === 'generate') {
+        const useAI = !useBasic;
+        console.log(useAI
+            ? 'Generating tests with AI for changed journeys...\n'
+            : 'Generating tests with the basic generator...\n');
+        const results = runGenerate('docs', useAI, force);
+        if (results.length === 0) {
+            console.log('No journey files found in docs/. Add a .md file and try again.');
+        }
+        else {
+            for (const r of results) {
+                if (r.status === 'skipped') {
+                    console.log(`  [skip] ${r.journeyName} (unchanged)`);
+                }
+                else {
+                    const tag = r.status === 'ai' ? '[AI]' : '[basic]';
+                    console.log(`  ${tag} ${r.journeyName}`);
+                    console.log(`         test: ${r.testPath}`);
+                    if (r.status === 'ai') {
+                        if (r.seedStatus === 'ok') {
+                            console.log(`         seed: ${r.seedPath}`);
+                        }
+                        else if (r.seedStatus === 'failed') {
+                            console.log(`         seed: (not written — generation failed)`);
+                        }
+                    }
+                }
+                for (const w of r.warnings ?? []) {
+                    console.log(`         ! ${w}`);
+                }
+            }
+            const generated = results.filter((r) => r.status !== 'skipped').length;
+            const skipped = results.filter((r) => r.status === 'skipped').length;
+            const seedFailures = results.filter((r) => r.seedStatus === 'failed').length;
+            console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged).`);
+            if (seedFailures > 0) {
+                console.log(`Warning: ${seedFailures} seed file(s) could not be generated — see notes above.`);
+            }
+        }
+    }
+    else {
+        printUsage();
+    }
+}
+main().catch((err) => {
+    console.error(err instanceof Error ? err.message : String(err));
+    process.exit(1);
+});

package/dist/core/discover.js

@@ -16,6 +16,9 @@
  */
 import { execFileSync } from 'node:child_process';
 import { loadConfig, splitCommand, withEditTools } from './config.js';
+import { promptConfirm } from './prompt.js';
+import { resolveDocsSource, saveSourceChoice, } from './source-of-truth.js';
+import { analyzeReference, draftDocsFromReference, DIFF_PATH, } from './reference.js';
 /** Build the discovery prompt. */
 export function buildDiscoverPrompt(srcDir, docsDir) {
     return [
@@ -83,11 +86,10 @@ export function buildDiscoverPrompt(srcDir, docsDir) {
     ].join('\n');
 }
 /**
- * Run discovery. This invokes the Claude CLI with permission to read
- * and write files in the project. Returns the CLI's text output (the
- * summary of what it created).
+ * Draft the docs from the project's own code (the default). Invokes the
+ * AI CLI with permission to read source and write markdown files.
  */
-export function runDiscover(srcDir, docsDir) {
+function runCodeDiscover(srcDir, docsDir) {
     const prompt = buildDiscoverPrompt(srcDir, docsDir);
     // Use the configured AI command. Note: discovery needs the AI to
     // read source files and write markdown files. The flags for that
@@ -101,3 +103,39 @@ export function runDiscover(srcDir, docsDir) {
     });
     return output.trim();
 }
+/**
+ * Run discovery. First decides the source of truth for the docs (the
+ * code-derived map, or an external reference — see source-of-truth.ts),
+ * then drafts the journey docs from it. When a reference is chosen it is
+ * compared against the map first (writing testforge.diff.md), and, for a
+ * fresh interactive choice, confirmed before the docs are rewritten.
+ *
+ * Async because the source-of-truth barrier may prompt the developer.
+ * Returns the AI's text summary of what it created.
+ */
+export async function runDiscover(srcDir, docsDir, opts = {}) {
+    const source = await resolveDocsSource(opts);
+    if (source.mode === 'reference' && source.reference) {
+        console.log(`\nComparing the reference against testforge.md: ${source.reference}`);
+        console.log('(reading the reference and the map — this may take a moment)\n');
+        const summary = analyzeReference(source.reference, docsDir);
+        console.log(summary);
+        console.log(`\nFull comparison written to ${DIFF_PATH}.`);
+        // Only a freshly-typed interactive choice gets a confirmation step;
+        // an explicit flag or a remembered choice proceeds directly.
+        if (source.origin === 'prompt') {
+            const ok = await promptConfirm('\nDraft the docs from this reference (make it the source of truth)?', true);
+            if (!ok) {
+                console.log('Basing docs on testforge.md instead.\n');
+                saveSourceChoice({ mode: 'map' });
+                return runCodeDiscover(srcDir, docsDir);
+            }
+        }
+        console.log('\nDrafting docs from the reference...\n');
+        const out = draftDocsFromReference(srcDir, docsDir, source.reference);
+        saveSourceChoice({ mode: 'reference', reference: source.reference });
+        return out;
+    }
+    saveSourceChoice({ mode: 'map' });
+    return runCodeDiscover(srcDir, docsDir);
+}

package/dist/core/index.js

@@ -4,4 +4,4 @@
  * Currently exposes the package version only.
  * Model, parser, and engines will be added in upcoming steps.
  */
-export const VERSION = '0.2.0';
+export const VERSION = '0.3.0';

package/dist/core/reference.js

@@ -0,0 +1,162 @@
+/**
+ * testforge — external reference support for doc generation.
+ *
+ * The journey docs are the source of truth for the tests: `generate`
+ * reads the docs and nothing else. So WHERE the docs come from decides
+ * everything. By default they are derived from the project's own code
+ * (via testforge.md), which bakes in any bugs the code has.
+ *
+ * This module lets the docs be drafted from an AUTHORITATIVE reference
+ * instead — a spec/requirements doc in the repo, a PDF, a website, or a
+ * Notion page that captures the human/client EXPECTED behavior:
+ *
+ *   - analyzeReference()   compares the reference against the current
+ *                          map (testforge.md) and any existing docs, and
+ *                          writes a comparison to testforge.diff.md.
+ *   - draftDocsFromReference() drafts the journey docs with the reference
+ *                          as the source of truth for behavior, using the
+ *                          map/code only for technical detail.
+ *
+ * Both drive the configured AI CLI with file-read/write (and, for
+ * URLs/Notion, web) access. As with discover/map this is tool-specific
+ * and tuned for the Claude CLI.
+ */
+import { execFileSync } from 'node:child_process';
+import { loadConfig, splitCommand, withEditTools } from './config.js';
+/** Where the comparison report is written. */
+export const DIFF_PATH = 'testforge.diff.md';
+const MAP_PATH = 'testforge.md';
+/** Run the configured AI command with read/write/web access. */
+function runAI(prompt) {
+    const { program, args } = withEditTools(splitCommand(loadConfig().aiCommand));
+    const output = execFileSync(program, args, {
+        input: prompt,
+        encoding: 'utf-8',
+        maxBuffer: 10 * 1024 * 1024,
+    });
+    return output.trim();
+}
+/** Build the prompt for the compare-only (analyze) step. */
+export function buildAnalyzePrompt(reference, docsDir) {
+    return [
+        `You are a senior QA engineer. The developer has given you an`,
+        `AUTHORITATIVE reference describing how this product is EXPECTED to`,
+        `behave. It comes from a human/client, NOT from the code, so where it`,
+        `disagrees with what the code implies, the reference is right.`,
+        ``,
+        `In THIS step you only COMPARE — you do not write any journey docs.`,
+        ``,
+        `Reference provided by the developer:`,
+        `  ${reference}`,
+        ``,
+        `This may be a path to a file in this project, a path to a document`,
+        `(e.g. a PDF), a URL to a website, or a link to a Notion page. Read`,
+        `it with your available tools. If you cannot access it at all, print`,
+        `a single line starting with "ERROR:" explaining why, and stop.`,
+        ``,
+        `Then read the project map at "${MAP_PATH}" (the AI's code-derived view`,
+        `of the journeys and data layer) and any existing docs under`,
+        `"${docsDir}/".`,
+        ``,
+        `Compare what the reference specifies against what the map/docs imply,`,
+        `focusing on things that matter for testing:`,
+        `  - journeys/flows the reference describes that the map is missing`,
+        `  - expected results / acceptance criteria that DIFFER`,
+        `  - fields, routes, validation, or error cases the reference requires`,
+        `    that the code-derived view omits or gets wrong`,
+        `  - anything the code-derived view asserts that the reference contradicts`,
+        ``,
+        `Write a comparison report to "${DIFF_PATH}" at the project root with`,
+        `these sections:`,
+        `  # Source-of-truth comparison`,
+        `  ## Reference        (what you read, and where from)`,
+        `  ## Summary          (a few bullet lines)`,
+        `  ## Differences      (per journey: the code implies X vs the`,
+        `                       reference says Y, and which should win)`,
+        `  ## Missing          (journeys/cases only in the reference)`,
+        `  ## Code-only        (in the code-derived view but not the reference —`,
+        `                       likely assumptions to double-check)`,
+        ``,
+        `Do NOT write any files under "${docsDir}/" in this step — only write`,
+        `"${DIFF_PATH}". When done, print a SHORT plain-text summary (about`,
+        `5-10 lines) of the key differences.`,
+    ].join('\n');
+}
+/**
+ * Build the prompt that drafts the journey docs FROM the reference,
+ * treating it as the source of truth for behavior. Mirrors the code-based
+ * discover format so the docs are interchangeable downstream.
+ */
+export function buildReferenceDiscoverPrompt(srcDir, docsDir, reference) {
+    return [
+        `You are a senior QA engineer creating per-journey documentation so a`,
+        `QA tool can generate tests from it. The developer has chosen an`,
+        `AUTHORITATIVE reference as the SOURCE OF TRUTH for how the product is`,
+        `EXPECTED to behave. The reference wins over the code: where the code`,
+        `does something different, document the reference's expected behavior`,
+        `(the code may be buggy — that is exactly what these tests must catch).`,
+        ``,
+        `Reference: ${reference}`,
+        `Read it with your available tools (read the file, fetch the URL/page).`,
+        `The comparison you need is already in "${DIFF_PATH}" — use it.`,
+        ``,
+        `Use the project map at "${MAP_PATH}" and the source under "${srcDir}"`,
+        `ONLY for TECHNICAL detail the reference does not spell out: routes,`,
+        `field names, models/collections, enums, and env var names. NEVER take`,
+        `expected behavior, acceptance criteria, or negative cases from the`,
+        `code — those come from the reference.`,
+        ``,
+        `For EACH journey the reference describes, create one markdown file at`,
+        `"${docsDir}/<area>/<journey>.md" (area = folder such as api/web/ops,`,
+        `short kebab-case journey name; do NOT put the area in the file name).`,
+        `Also create a doc for any journey that exists in the map but NOT in`,
+        `the reference, drafted from the map, and add a line`,
+        `"# TODO: not covered by the reference — confirm expected behavior"`,
+        `at its top so it is clearly flagged.`,
+        ``,
+        `Each file starts with this header (required):`,
+        ``,
+        `# Journey: <Name>`,
+        ``,
+        `Type: web        (use "web" for a page/UI, "api" for an endpoint)`,
+        `Route: <path>    (the URL path or endpoint, e.g. /login)`,
+        ``,
+        `Coverage:`,
+        `- functional`,
+        `- security   (add more if relevant: performance, accessibility, ...)`,
+        ``,
+        `Then whichever of these sections apply: Fields, Actions, Expected,`,
+        `Negative Cases, Race Conditions, Webhooks, Security Notes, Data. The`,
+        `Expected / Negative Cases / acceptance criteria MUST reflect the`,
+        `reference. For the Data section use the map's data layer.`,
+        ``,
+        `Rules:`,
+        `- Base behavior on the reference and technical detail on real code;`,
+        `  do not invent either with false confidence.`,
+        `- Where the reference is silent on a needed detail, add a "# TODO:"`,
+        `  line describing what the developer must verify.`,
+        `- Never include secrets — refer to env var names only.`,
+        `- Create the area subfolders as needed.`,
+        `- After creating the files, print the list of journeys you documented`,
+        `  and remind the developer to review them.`,
+    ].join('\n');
+}
+/**
+ * Compare-only step. Reads the reference and the current map/docs, writes
+ * testforge.diff.md, and returns the AI's short summary. Throws if the AI
+ * reports it could not read the reference.
+ */
+export function analyzeReference(reference, docsDir) {
+    const summary = runAI(buildAnalyzePrompt(reference, docsDir));
+    if (/^ERROR:/m.test(summary)) {
+        throw new Error(`Could not read the reference "${reference}":\n${summary}`);
+    }
+    return summary;
+}
+/**
+ * Draft the journey docs from the reference. Returns the AI's summary of
+ * the journeys it documented.
+ */
+export function draftDocsFromReference(srcDir, docsDir, reference) {
+    return runAI(buildReferenceDiscoverPrompt(srcDir, docsDir, reference));
+}

package/dist/core/setup.js

@@ -16,7 +16,7 @@ import { runInit } from './init.js';
 import { runMap } from './map.js';
 import { runDiscover } from './discover.js';
 import { runEnvTemplate } from './env-template.js';
-export function runSetup(srcDir) {
+export async function runSetup(srcDir, opts = {}) {
     // If the project already looks set up, don't redo it — point the user
     // to `update` instead, which only refreshes what actually changed.
     const alreadySetUp = existsSync('testforge.config.json') &&
@@ -49,9 +49,10 @@ export function runSetup(srcDir) {
     else {
         console.log('  warning: testforge.md was not created\n');
     }
-    // Step 3 — discover.
+    // Step 3 — discover. This is where the source-of-truth barrier lives:
+    // the docs may be based on testforge.md or on an external reference.
     console.log('Step 3/3: Drafting a doc for each journey...');
-    runDiscover(srcDir, 'docs');
+    await runDiscover(srcDir, 'docs', opts);
     console.log('  done — docs/\n');
     // Remember the source state so the first `update` can detect "no change".
     const cache = loadCache('.testforge/cache.json');

package/dist/core/source-of-truth.js

@@ -1,95 +1,84 @@
 /**
- * testforge — the "source of truth" barrier for `generate`.
+ * testforge — deciding where the journey docs come from.
  *
- * Before generating tests, testforge asks the developer WHAT the tests
- * should be based on. Two answers are possible:
- *
- *   1. testforge.md + docs/  — what the AI derived from the project's own
- *      code. Fast, but only as correct as the code it read.
+ * The docs are the source of truth for the tests, so before drafting
+ * them (in discover / setup) testforge asks WHAT to base them on:
  *
+ *   1. testforge.md  — the AI's code-derived view. Fast, but only as
+ *      correct as the code it read.
  *   2. An external reference — a spec/doc/PDF in the repo, a website, or
- *      a Notion page — that captures the HUMAN/CLIENT expected behavior.
- *      testforge compares it against the current docs (writing a diff for
- *      the developer to inspect), and, on confirmation, reconciles the
- *      docs so the reference wins. Tests are then generated from those
- *      reconciled docs.
+ *      a Notion page capturing the human/client EXPECTED behavior.
  *
- * This keeps the generate pipeline itself unchanged (it still reads
- * docs/) — the barrier just decides whether the docs get reconciled
- * against a reference first.
+ * The choice is remembered in .testforge/source.json so later refreshes
+ * (`update`) keep using the same source without asking again — otherwise
+ * an update would silently regenerate reference-based docs from the code.
  *
- * Non-interactive callers (CI, cron, piped stdin) must not be blocked on
- * a prompt, so:
- *   - `source: 'docs'`        → skip the barrier, use docs as-is.
- *   - `reference: <locator>`  → use that reference, analyze + apply with
- *                               no prompts (implicit confirmation).
- *   - neither, and no TTY     → fall back to docs with a printed note.
+ * Non-interactive callers are never blocked:
+ *   - source: 'map'          → use testforge.md, skip the prompt.
+ *   - reference: <locator>   → use that reference, skip the prompt.
+ *   - noPrompt (e.g. update) → reuse the remembered choice, else the map.
+ *   - no TTY                 → fall back to the map with a printed note.
  */
-import { isInteractive, promptChoice, promptConfirm, promptLine, } from './prompt.js';
-import { analyzeReference, applyReconciliation, DIFF_PATH } from './reconcile.js';
-/** Run the analyze step and reconcile the docs against the reference. */
-function reconcileToReference(reference, docsDir) {
-    console.log(`\nAnalyzing reference against current docs: ${reference}`);
-    console.log('(reading the reference, the map and docs/ — this may take a moment)\n');
-    const summary = analyzeReference(reference, docsDir);
-    console.log(summary);
-    console.log(`\nFull comparison written to ${DIFF_PATH}.`);
-    console.log('\nReconciling docs so the reference is the source of truth...\n');
-    const applied = applyReconciliation(reference, docsDir);
-    console.log(applied);
-    console.log('\nDocs updated. Generating tests from the reconciled docs.\n');
+import { readFileSync, writeFileSync, mkdirSync, existsSync } from 'node:fs';
+import { dirname } from 'node:path';
+import { isInteractive, promptChoice, promptLine } from './prompt.js';
+const SOURCE_PATH = '.testforge/source.json';
+/** Read the remembered source choice, or null if none/invalid. */
+export function loadSourceChoice() {
+    if (!existsSync(SOURCE_PATH))
+        return null;
+    try {
+        const parsed = JSON.parse(readFileSync(SOURCE_PATH, 'utf-8'));
+        if (parsed && (parsed.mode === 'map' || parsed.mode === 'reference')) {
+            return { mode: parsed.mode, reference: parsed.reference };
+        }
+    }
+    catch {
+        // ignore a corrupt file — treat as "no choice remembered"
+    }
+    return null;
+}
+/** Remember the source choice for later refreshes. */
+export function saveSourceChoice(choice) {
+    mkdirSync(dirname(SOURCE_PATH), { recursive: true });
+    writeFileSync(SOURCE_PATH, JSON.stringify(choice, null, 2), 'utf-8');
 }
 /**
- * Resolve the source of truth before generation. May reconcile the docs
- * in place (when a reference is chosen). Returns once docs/ reflects the
- * chosen source; the caller then runs the normal generate pipeline.
+ * Decide where the docs should come from for this run. May prompt the
+ * developer interactively; never blocks in non-interactive contexts.
  */
-export async function resolveSourceOfTruth(opts) {
-    const { docsDir } = opts;
-    // Explicit "use the docs" — nothing to do.
-    if (opts.source === 'docs')
-        return;
-    // Explicit reference via flag: non-interactive analyze + apply.
+export async function resolveDocsSource(opts = {}) {
+    if (opts.source === 'map') {
+        return { mode: 'map', origin: 'flag' };
+    }
     if (opts.reference) {
-        reconcileToReference(opts.reference, docsDir);
-        return;
+        return { mode: 'reference', reference: opts.reference, origin: 'flag' };
+    }
+    // Reuse a previously remembered choice (so `update` stays consistent).
+    const stored = loadSourceChoice();
+    if (stored) {
+        return { ...stored, origin: 'stored' };
     }
-    // No explicit choice and no terminal to ask on — do the safe,
-    // non-blocking thing: use the docs as they are.
-    if (!isInteractive()) {
-        console.log('Using testforge.md + docs/ as the source of truth ' +
-            '(non-interactive; pass --reference <file|url> to use an external reference).');
-        return;
+    // Nothing chosen and nowhere to ask — safe default.
+    if (opts.noPrompt || !isInteractive()) {
+        if (!opts.noPrompt) {
+            console.log('Basing docs on testforge.md (non-interactive; pass ' +
+                '--reference <file|url> to use an external reference).');
+        }
+        return { mode: 'map', origin: 'default' };
     }
-    // Interactive: ask what to base the tests on.
-    const choice = await promptChoice('What should the tests be based on?', [
-        'testforge.md + docs/ (what the AI derived from your code)',
+    // Interactive: ask what the docs should be based on.
+    const choice = await promptChoice('What should the journey docs be based on?', [
+        'testforge.md (what the AI derived from your code)',
         'An external reference (a file/PDF in scope, a website, or a Notion link)',
     ], 1);
     if (choice === 1) {
-        console.log('Using testforge.md + docs/ as the source of truth.\n');
-        return;
+        return { mode: 'map', origin: 'prompt' };
     }
     const reference = await promptLine('\nProvide the reference (a file path, a URL, or a Notion link): ');
     if (!reference) {
-        console.log('No reference given — falling back to testforge.md + docs/.\n');
-        return;
-    }
-    // Analyze first so the developer can see the differences, then confirm
-    // before we rewrite any docs.
-    console.log(`\nAnalyzing reference against current docs: ${reference}`);
-    console.log('(reading the reference, the map and docs/ — this may take a moment)\n');
-    const summary = analyzeReference(reference, docsDir);
-    console.log(summary);
-    console.log(`\nFull comparison written to ${DIFF_PATH}.`);
-    const apply = await promptConfirm('\nApply these changes to docs/ (make the reference the source of truth)?', false);
-    if (!apply) {
-        console.log(`\nKeeping the current docs. The comparison is saved in ${DIFF_PATH} ` +
-            'for you to review. Generating tests from the existing docs.\n');
-        return;
+        console.log('No reference given — basing docs on testforge.md.\n');
+        return { mode: 'map', origin: 'prompt' };
     }
-    console.log('\nReconciling docs so the reference is the source of truth...\n');
-    const applied = applyReconciliation(reference, docsDir);
-    console.log(applied);
-    console.log('\nDocs updated. Generating tests from the reconciled docs.\n');
+    return { mode: 'reference', reference, origin: 'prompt' };
 }

package/dist/core/update.js

@@ -16,7 +16,7 @@ import { hashProjectSource, loadCache, saveCache } from './cache.js';
 import { runEnvTemplate } from './env-template.js';
 const CACHE_PATH = '.testforge/cache.json';
 const SOURCE_KEY = '__source__';
-export function runUpdate(srcDir) {
+export async function runUpdate(srcDir) {
     if (!existsSync('testforge.md') || !existsSync('testforge.config.json')) {
         console.log("This project isn't set up yet.");
         console.log('Run `testforge setup` first.');
@@ -35,8 +35,10 @@ export function runUpdate(srcDir) {
     console.log('Re-mapping the project...');
     runMap(srcDir);
     console.log('  done — testforge.md\n');
+    // Reuse the source of truth chosen at setup (map or a reference)
+    // without prompting — an update is a refresh, not a re-decision.
     console.log('Updating journey docs...');
-    runDiscover(srcDir, 'docs');
+    await runDiscover(srcDir, 'docs', { noPrompt: true });
     console.log('  done — docs/\n');
     console.log('Regenerating tests for changed journeys...');
     const results = runGenerate('docs', true, false);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@activate-core/test-forge",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "AI-powered QA automation. Scans a project, drafts journey docs, and generates seeded Jest/Playwright tests. Works in any project, with any AI CLI.",
   "type": "module",
   "bin": {

package/dist/core/generate-command.js

@@ -1,55 +0,0 @@
-/**
- * testforge — the `generate` command (CLI orchestration).
- *
- * This ties together the source-of-truth barrier and the generate
- * pipeline, and prints results for the terminal. It is async because the
- * barrier may prompt the developer interactively (see source-of-truth.ts).
- *
- * Kept separate from the pure runGenerate() engine so that engine stays
- * synchronous and easy to call from update.ts, while the human-facing
- * command handles prompting and output.
- */
-import { runGenerate } from './run-generate.js';
-import { resolveSourceOfTruth } from './source-of-truth.js';
-export async function runGenerateCommand(opts) {
-    // 1. Decide (and possibly reconcile) the source of truth first.
-    await resolveSourceOfTruth({
-        docsDir: opts.docsDir,
-        reference: opts.reference,
-        source: opts.source,
-    });
-    // 2. Generate from whatever the docs now say.
-    const results = runGenerate(opts.docsDir, opts.useAI, opts.force);
-    if (results.length === 0) {
-        console.log(`No journey files found in ${opts.docsDir}/. Add a .md file and try again.`);
-        return;
-    }
-    for (const r of results) {
-        if (r.status === 'skipped') {
-            console.log(`  [skip] ${r.journeyName} (unchanged)`);
-        }
-        else {
-            const tag = r.status === 'ai' ? '[AI]' : '[basic]';
-            console.log(`  ${tag} ${r.journeyName}`);
-            console.log(`         test: ${r.testPath}`);
-            if (r.status === 'ai') {
-                if (r.seedStatus === 'ok') {
-                    console.log(`         seed: ${r.seedPath}`);
-                }
-                else if (r.seedStatus === 'failed') {
-                    console.log(`         seed: (not written — generation failed)`);
-                }
-            }
-        }
-        for (const w of r.warnings ?? []) {
-            console.log(`         ! ${w}`);
-        }
-    }
-    const generated = results.filter((r) => r.status !== 'skipped').length;
-    const skipped = results.filter((r) => r.status === 'skipped').length;
-    const seedFailures = results.filter((r) => r.seedStatus === 'failed').length;
-    console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged).`);
-    if (seedFailures > 0) {
-        console.log(`Warning: ${seedFailures} seed file(s) could not be generated — see notes above.`);
-    }
-}

package/dist/core/reconcile.js

@@ -1,157 +0,0 @@
-/**
- * testforge — reconcile docs against an external reference.
- *
- * By default testforge's source of truth is what the AI derived from the
- * project's own code (testforge.md + the journey docs under docs/). That
- * is convenient, but it inherits any bugs or wrong assumptions in the
- * code: if the code mishandles a scenario, the AI documents the wrong
- * behavior, and every test then locks that wrong behavior in.
- *
- * This module lets a human point testforge at an AUTHORITATIVE reference
- * of the expected behavior — a spec file in the repo, a document/PDF, a
- * website, or a Notion page — and:
- *
- *   1. analyzeReference() — compares the reference against the current
- *      testforge.md/docs, writes a comparison report to testforge.diff.md,
- *      and prints a short summary. It changes nothing else.
- *
- *   2. applyReconciliation() — once the human confirms, rewrites the
- *      journey docs so they reflect the reference as the source of truth
- *      (fixing wrong expectations, adding missing journeys/cases).
- *
- * Both steps drive the configured AI CLI with file-read/write (and, for
- * URLs/Notion, web) access. As with discover/map, this is tool-specific;
- * it is tuned for the Claude CLI and other CLIs may need their own setup.
- */
-import { execFileSync } from 'node:child_process';
-import { loadConfig, splitCommand, withEditTools } from './config.js';
-/** Where the comparison report is written. */
-export const DIFF_PATH = 'testforge.diff.md';
-const MAP_PATH = 'testforge.md';
-/** Run the configured AI command with read/write/web access. */
-function runAI(prompt) {
-    const { program, args } = withEditTools(splitCommand(loadConfig().aiCommand));
-    const output = execFileSync(program, args, {
-        input: prompt,
-        encoding: 'utf-8',
-        maxBuffer: 10 * 1024 * 1024,
-    });
-    return output.trim();
-}
-/**
- * The journey-doc format, kept in sync with discover.ts so newly created
- * docs match the ones setup produced. Referenced from the apply prompt.
- */
-const DOC_FORMAT = [
-    `# Journey: <Name>`,
-    ``,
-    `Type: web        (use "web" for a page/UI, "api" for an endpoint)`,
-    `Route: <path>    (the URL path or endpoint, e.g. /login)`,
-    ``,
-    `Coverage:`,
-    `- functional`,
-    `- security   (add more if relevant: performance, accessibility, ...)`,
-    ``,
-    `Then whichever sections apply: Fields, Actions, Expected, Negative`,
-    `Cases, Race Conditions, Webhooks, Security Notes, Data.`,
-].join('\n');
-/** Build the prompt for the compare-only (analyze) step. */
-export function buildAnalyzePrompt(reference, docsDir) {
-    return [
-        `You are a senior QA engineer. The developer has given you an`,
-        `AUTHORITATIVE reference describing how this product is EXPECTED to`,
-        `behave. This reference comes from a human/client, NOT from the code,`,
-        `so where it disagrees with the current docs the reference is right`,
-        `and the docs (which were derived from possibly-buggy code) are wrong.`,
-        ``,
-        `In THIS step you only COMPARE. Do not change the docs.`,
-        ``,
-        `Reference provided by the developer:`,
-        `  ${reference}`,
-        ``,
-        `This may be a path to a file in this project, a path to a document`,
-        `(e.g. a PDF), a URL to a website, or a link to a Notion page. Read`,
-        `it with your available tools (read the file, fetch the URL/page). If`,
-        `you cannot access it at all, print a line starting with "ERROR:"`,
-        `explaining why, and stop.`,
-        ``,
-        `Then read:`,
-        `  - the project map at "${MAP_PATH}" (if it exists)`,
-        `  - every journey doc under "${docsDir}/"`,
-        ``,
-        `Compare what the reference specifies against what the map and docs`,
-        `currently say, focusing on things that matter for testing:`,
-        `  - journeys/flows the reference describes that are MISSING from docs`,
-        `  - expected results / acceptance criteria that DIFFER`,
-        `  - fields, routes, validation, or error cases the reference requires`,
-        `    that the docs omit or get wrong`,
-        `  - anything the docs assert that the reference contradicts`,
-        ``,
-        `Write a comparison report to "${DIFF_PATH}" at the project root with`,
-        `these sections:`,
-        `  # Source-of-truth comparison`,
-        `  ## Reference        (what you read, and where from)`,
-        `  ## Summary          (a few bullet lines)`,
-        `  ## Differences      (per journey: docs say X vs reference says Y,`,
-        `                       and which should win)`,
-        `  ## Missing in docs  (journeys/cases only in the reference)`,
-        `  ## Docs-only        (in the docs but not the reference — may be`,
-        `                       code-derived assumptions to double-check)`,
-        ``,
-        `Do NOT modify "${MAP_PATH}" or anything under "${docsDir}/" in this`,
-        `step — only write "${DIFF_PATH}". When done, print a SHORT plain-text`,
-        `summary (about 5-10 lines) of the key differences.`,
-    ].join('\n');
-}
-/** Build the prompt for the apply (reconcile) step. */
-export function buildApplyPrompt(reference, docsDir) {
-    return [
-        `You are a senior QA engineer. The developer has CONFIRMED that the`,
-        `reference below is the SOURCE OF TRUTH for tests, overriding what was`,
-        `previously derived from the code.`,
-        ``,
-        `Reference: ${reference}`,
-        `The comparison has already been computed — read "${DIFF_PATH}" for the`,
-        `differences you must resolve.`,
-        ``,
-        `Update the journey docs under "${docsDir}/" so they reflect the`,
-        `reference as the source of truth:`,
-        `  - Fix expected results / acceptance criteria to match the reference`,
-        `    wherever they differ.`,
-        `  - Add sections or cases the reference requires that are missing.`,
-        `  - For a journey the reference describes that has NO doc yet, create`,
-        `    "${docsDir}/<area>/<journey>.md" (area = folder, short kebab-case`,
-        `    journey name) using this format:`,
-        ``,
-        DOC_FORMAT,
-        ``,
-        `  - Where a doc made an assumption from the code that the reference`,
-        `    does not mention, KEEP it, but if it conflicts with the reference`,
-        `    add a line starting with "# TODO:" describing the conflict.`,
-        `  - If the reference reveals models/fields the map missed, update`,
-        `    "${MAP_PATH}" too.`,
-        ``,
-        `Never include secrets — refer to env var names only. Preserve the`,
-        `existing doc format. When done, print a SHORT summary listing which`,
-        `docs you created or changed.`,
-    ].join('\n');
-}
-/**
- * Compare-only step. Reads the reference and the current docs/map, writes
- * testforge.diff.md, and returns the AI's short summary. Changes nothing
- * else. Throws if the AI reports it could not read the reference.
- */
-export function analyzeReference(reference, docsDir) {
-    const summary = runAI(buildAnalyzePrompt(reference, docsDir));
-    if (/^ERROR:/m.test(summary)) {
-        throw new Error(`Could not read the reference "${reference}":\n${summary}`);
-    }
-    return summary;
-}
-/**
- * Apply step. Rewrites the docs (and map if needed) so they reflect the
- * reference. Returns the AI's short summary of what it changed.
- */
-export function applyReconciliation(reference, docsDir) {
-    return runAI(buildApplyPrompt(reference, docsDir));
-}