@activate-core/test-forge 0.1.0

package/README.md

@@ -0,0 +1,179 @@
+# testforge
+
+AI-powered QA automation. Point it at a project and it scans the code,
+drafts a journey doc for each feature, and generates seeded Jest and
+Playwright tests that match the project's own structure.
+
+Works in any project, with any AI CLI (defaults to Claude).
+
+## Install
+
+```bash
+npm install -D testforge
+```
+
+testforge calls a local AI CLI to do its work. By default it uses
+`claude -p`, so make sure the Claude CLI is installed and logged in.
+To use a different CLI, see [Configuration](#).
+
+## Quick start
+
+From your project root:
+
+```bash
+testforge setup      # one-time: inspect project, map it, draft journey docs
+testforge generate   # generate tests + seed files from the docs
+```
+
+After `setup`, review the drafted docs in `docs/` and fix anything the
+AI got wrong — they are the source of truth for the tests. Then run
+`generate`.
+
+When your code changes later:
+
+```bash
+testforge update     # re-check and refresh only what changed
+```
+
+## Commands
+
+| Command              | What it does                                                                                                                       |
+| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
+| `testforge setup`    | One-time setup: runs init, map, and discover in order, with progress.                                                              |
+| `testforge generate` | Generates a test file and a seed file for each journey doc. Only changed docs are regenerated.                                     |
+| `testforge update`   | Detects source-code changes and refreshes the map, docs, and tests when needed. Skips everything if nothing changed (no AI calls). |
+
+### Advanced (run a single step)
+
+| Command              | What it does                                                |
+| -------------------- | ----------------------------------------------------------- |
+| `testforge init`     | Inspect the project and write `testforge.config.json` only. |
+| `testforge map`      | Rebuild the project map (`testforge.md`) only.              |
+| `testforge discover` | Redraft the per-journey docs in `docs/` only.               |
+| `testforge env`      | Rebuild `.env.testing.example` from the docs only.          |
+
+## How it works
+
+**setup** prepares the project in three stages:
+
+1. **init** — detects the layout (monorepo or single app, where tests
+   live, JS or TS) and writes `testforge.config.json`.
+2. **map** — writes `testforge.md`: a short list of journeys and a
+   summary of the data layer (database, models, key fields). The AI
+   reads this instead of re-scanning all the code each time.
+3. **discover** — writes one rich doc per journey into `docs/` — fields,
+   actions, expected results, edge cases, security notes, and the data a
+   test needs.
+
+**generate** turns each doc into a test. A journey with `Type: api`
+becomes a Jest test (HTTP + database checks); `Type: web` becomes a
+Playwright test. Other areas use the framework set in config. It also
+generates a seed file that provisions the test data each test needs.
+
+**update** hashes your source code and compares it to the last run. No
+change means no work and no AI calls; a change triggers a refresh.
+
+## Running the generated tests
+
+testforge generates the tests and seed files — you run them with your
+own test runner, the way your project already does. testforge does not
+run tests for you; that keeps it out of your test setup and CI.
+
+A typical run:
+
+1. **Provide test credentials.** The tests and seeds read the database
+   URL and any secrets from environment variables — never hardcoded.
+
+   `testforge setup` writes a `.env.testing.example` for you. It lists
+   every env var the journeys need, grouped by area (api / web / ops),
+   and annotates each one with the journeys that use it — so you know
+   exactly which credentials a feature needs, and can skip the ones for
+   features you don't run. For example:
+
+```bash
+   # ========================================
+   # API
+   # ========================================
+   DB_URL=              # used by: (all)
+   TOKEN_SECRET=        # used by: auth, user
+   STRIPE_SECRET_KEY=   # used by: payments, subscription
+
+   # ========================================
+   # WEB
+   # ========================================
+   StripePublicKey=     # used by: payments
+```
+
+Copy it to `.env.testing`, fill in real values, and keep that file
+out of version control (add it to `.gitignore`). Re-running `setup`
+or `update` refreshes the example as your code changes.
+
+Note: some credentials aren't static — e.g. a user access/refresh
+token from a login flow. Those belong in your own test setup (a
+global setup or seed step), not in the env file.
+
+Keep this file out of version control (add it to `.gitignore`). The
+exact variable names come from your own code — the journey docs and
+the project map list the env var names each journey needs.
+
+2. **Run the seeds** to put known test data in the database. Run the
+   seed files in `seedOutput` (see config) with your runtime, e.g.:
+
+```bash
+   npx tsx path/to/seed/<journey>.seed.ts   # or: node <journey>.seed.js
+```
+
+3. **Run the tests** with the project's own command, e.g.:
+
+```bash
+   # api (Jest)
+   NODE_ENV=testing npx jest
+
+   # web (Playwright)
+   npx playwright test
+```
+
+The test runner produces its own pass/fail report.
+
+## Configuration
+
+`testforge setup` writes `testforge.config.json` for you. You can edit it:
+
+```json
+{
+  "aiCommand": "claude -p",
+  "testOutput": {
+    "api": "packages/api/test",
+    "web": "packages/webapp/tests",
+    "ops": "packages/ops/tests"
+  },
+  "framework": {
+    "api": "jest",
+    "web": "playwright",
+    "ops": "playwright"
+  },
+  "seedOutput": "packages/api/test/helpers/seed",
+  "testExtension": ".test.ts"
+}
+```
+
+- **aiCommand** — the CLI used to call the AI. Any tool that reads a
+  prompt from stdin and prints the reply works, so testforge is not tied
+  to one provider.
+- **testOutput** — where tests go, per area. Areas are not fixed — use
+  whatever your project has (`api`, `web`, `ops`, `lambda`, ...).
+- **framework** — which test framework each area uses.
+- **seedOutput** — where seed files go.
+- **testExtension** — `.test.ts` or `.test.js`.
+
+## Notes
+
+- Generated tests and docs are a strong draft, not final. Review them —
+  the AI leaves `TODO` comments where it could not be certain.
+- Secrets are never written to docs, configs, or tests. Connections use
+  environment variable names only; real values come from your
+  environment at run time.
+
+## License
+
+MIT

package/dist/cli/index.js

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+/**
+ * testforge — command-line entry point.
+ *
+ * Commands:
+ *   testforge generate           Generate tests for changed journeys (AI).
+ *   testforge generate --basic   Use the built-in basic generator only.
+ *   testforge generate --force   Regenerate everything, ignoring the cache.
+ */
+import { VERSION } from '../core/index.js';
+import { runGenerate } from '../core/run-generate.js';
+import { runDiscover } from '../core/discover.js';
+import { runMap } from '../core/map.js';
+import { runInit } from '../core/init.js';
+import { runSetup } from '../core/setup.js';
+import { runUpdate } from '../core/update.js';
+import { runEnvTemplate } from '../core/env-template.js';
+const command = process.argv[2];
+const useBasic = process.argv.includes('--basic');
+const force = process.argv.includes('--force');
+if (command === 'update') {
+    runUpdate('.');
+}
+else if (command === 'setup') {
+    runSetup('.');
+}
+else if (command === 'init') {
+    console.log('Inspecting your project to set up testforge...\n');
+    const output = runInit();
+    console.log(output);
+    console.log('\ntestforge.config.json written. Review it, then run `testforge map`.');
+}
+else if (command === 'map') {
+    console.log('Building a high-level map of your project with AI...');
+    console.log('This scans your source code and writes testforge.md\n');
+    const output = runMap('.');
+    console.log(output);
+    console.log('\nProject map written to testforge.md');
+}
+else if (command === 'discover') {
+    console.log('Scanning your project with AI to draft journey files...');
+    console.log('This reads your source code and writes drafts into docs/.\n');
+    const output = runDiscover('.', 'docs');
+    console.log(output);
+    console.log('\nDraft journeys written to docs/. Please review and edit them,');
+    console.log('then run `testforge generate`.');
+}
+else if (command === 'env') {
+    runEnvTemplate('docs', '.env.testing.example');
+}
+else if (command === 'generate') {
+    const useAI = !useBasic;
+    console.log(useAI
+        ? 'Generating tests with AI for changed journeys...\n'
+        : 'Generating tests with the basic generator...\n');
+    const results = runGenerate('docs', useAI, force);
+    if (results.length === 0) {
+        console.log('No journey files found in docs/. Add a .md file and try again.');
+    }
+    else {
+        for (const r of results) {
+            if (r.status === 'skipped') {
+                console.log(`  [skip] ${r.journeyName} (unchanged)`);
+            }
+            else {
+                const tag = r.status === 'ai' ? '[AI]' : '[basic]';
+                console.log(`  ${tag} ${r.journeyName}`);
+                console.log(`         test: ${r.testPath}`);
+                if (r.status === 'ai')
+                    console.log(`         seed: ${r.seedPath}`);
+            }
+        }
+        const generated = results.filter((r) => r.status !== 'skipped').length;
+        const skipped = results.filter((r) => r.status === 'skipped').length;
+        console.log(`\nDone. ${generated} generated, ${skipped} skipped (unchanged).`);
+    }
+}
+else {
+    console.log(`testforge v${VERSION}`);
+    console.log('');
+    console.log('Usage:');
+    console.log('  testforge generate           Generate tests for changed journeys');
+    console.log('  testforge generate --basic   Use the basic generator only');
+    console.log('  testforge generate --force   Regenerate everything');
+    console.log('  testforge discover          Scan source code and draft journey files');
+    console.log('  testforge map               Scan project, write journey + data map');
+    console.log('  testforge init              Inspect project and write config');
+    console.log('  testforge setup             Run init + map + discover in order');
+    console.log('  testforge update            Re-check and update only what changed');
+    console.log('  testforge env               Write .env.testing.example from docs');
+}

package/dist/core/ai-generator.js

@@ -0,0 +1,141 @@
+/**
+ * testforge — AI-powered test generator.
+ *
+ * Instead of using a fixed list of hardcoded scenarios, this asks
+ * the locally installed Claude Code CLI to act as a senior QA
+ * engineer: it reads the journey and decides what to test, then
+ * writes a complete Playwright test file.
+ *
+ * Requirements: the `claude` CLI must be installed and logged in.
+ * We call it in print mode (`claude -p`) and feed the prompt via
+ * stdin so long prompts and special characters are handled safely.
+ */
+import { execFileSync } from 'node:child_process';
+import { readFileSync, existsSync } from 'node:fs';
+import { loadConfig, splitCommand } from './config.js';
+const MAP_PATH = 'testforge.md';
+/** Build the prompt we send to the AI for one journey. */
+export function buildPrompt(journey, map) {
+    const config = loadConfig();
+    const fw = config.framework?.[journey.type];
+    const isApi = journey.type === 'api' || fw === 'jest' || fw === 'supertest';
+    const framework = fw
+        ? `${fw} for a "${journey.type}" test`
+        : isApi
+            ? 'Jest (with supertest for HTTP) for an API/backend test'
+            : 'an appropriate test framework for this kind of journey';
+    const ext = config.testExtension ?? '.test.ts';
+    const isJs = ext.endsWith('.js');
+    const language = isJs
+        ? 'plain JavaScript (CommonJS: use require/module.exports, NOT import; NO TypeScript type annotations)'
+        : 'TypeScript';
+    return [
+        `You are a senior QA engineer. Generate a complete ${framework}`,
+        `for the user journey described below.`,
+        ``,
+        `Think like an experienced tester whose goal is to break the`,
+        `application. Cover, where relevant: happy path, missing/empty`,
+        `required fields, invalid formats and boundary values (very long,`,
+        `very short, special characters, unicode), and — if security`,
+        `coverage is requested — security attacks (XSS, SQL/NoSQL injection,`,
+        `auth bypass, idempotency/duplicate charge, and others you judge`,
+        `useful). For payment/webhook journeys, test idempotency, amount/`,
+        `price tampering, and signature/verification where relevant.`,
+        ``,
+        isApi
+            ? [
+                `Because this is an API journey, write Jest tests that call the`,
+                `endpoint over HTTP, assert status codes, response body and`,
+                `schema, auth/authorization, and invalid payloads.`,
+                `If the journey's Data section names models/collections, VERIFY`,
+                `the database state after calls (e.g. a record was created, no`,
+                `duplicate exists, correct fields) using the database/ORM the`,
+                `project uses. Connect via the env var named in the map (e.g.`,
+                `process.env.DB_URL) — never hardcode a connection string.`,
+            ].join('\n')
+            : [
+                `Because this is a web journey, write Playwright tests that drive`,
+                `the UI. Use placeholder selectors with a TODO comment where the`,
+                `real page structure is unknown — do not invent selectors as if`,
+                `they are real.`,
+            ].join('\n'),
+        ``,
+        `This test relies on seeded test data (a separate seed file provisions`,
+        `the database before tests run). Assume the seed has created the`,
+        `records described in the journey's Data section, using stable test`,
+        `ids prefixed with "qa-"/"QA-TEST". Reference those known test ids`,
+        `rather than creating fresh data inside the test where possible. After`,
+        `a test mutates data, clean up what it created so reruns stay valid.`,
+        ``,
+        `Rules for your output:`,
+        `- Write the test in ${language}.`,
+        `- Output ONLY the raw test code. Your entire response must be valid`,
+        `  ${isJs ? 'JavaScript' : 'TypeScript'} that can be saved directly to a file and run.`,
+        `- The VERY FIRST characters of your response must be the first line of`,
+        `  code. Do NOT write any sentence, preamble, acknowledgement, or`,
+        `  explanation before or after the code.`,
+        `- Do NOT ask for approval or describe what you are doing.`,
+        `- Do NOT wrap the code in markdown fences.`,
+        `- Tag the top with a comment: // @generated by testforge`,
+        `- Where you must assume something uncertain, add a "// TODO:" note`,
+        `  rather than inventing it confidently.`,
+        ``,
+        `--- PROJECT MAP (data layer) ---`,
+        map,
+        ``,
+        `--- JOURNEY ---`,
+        `It may include sections beyond fields — actions, expected results,`,
+        `negative cases, race conditions, webhooks, security notes, and a`,
+        `Data section. Use ALL of it. Respect any "# TODO:" notes.`,
+        ``,
+        journey.rawMarkdown,
+    ].join('\n');
+}
+/**
+ * Call the configured AI command to generate a test file for one
+ * journey. Reads the project map (testforge.md) for data-layer
+ * context. Returns the test file content.
+ */
+export function generateWithAI(journey) {
+    const map = existsSync(MAP_PATH) ? readFileSync(MAP_PATH, 'utf-8') : '';
+    const prompt = buildPrompt(journey, map);
+    const { program, args } = splitCommand(loadConfig().aiCommand);
+    const output = execFileSync(program, args, {
+        input: prompt,
+        encoding: 'utf-8',
+        maxBuffer: 10 * 1024 * 1024,
+    });
+    return cleanOutput(output);
+}
+/**
+ * Extract clean code from the AI's reply.
+ *
+ * 1. If there's a fenced code block, take its contents.
+ * 2. Otherwise, drop any leading prose by starting from the first line
+ *    that looks like code (a comment, import, require, or known
+ *    statement keyword). This rescues cases where the AI adds a
+ *    sentence before the code despite instructions.
+ */
+function cleanOutput(text) {
+    // 1. Prefer a fenced code block if present.
+    const fence = text.match(/```[a-zA-Z]*\n([\s\S]*?)\n```/);
+    if (fence) {
+        return fence[1].trim();
+    }
+    // 2. No fence — find the first line that looks like real code.
+    const lines = text.split('\n');
+    const codeStart = lines.findIndex((line) => {
+        const t = line.trim();
+        return (t.startsWith('//') ||
+            t.startsWith('/*') ||
+            t.startsWith('import ') ||
+            t.startsWith('const ') ||
+            t.startsWith('require(') ||
+            t.startsWith("'use strict'") ||
+            t.startsWith('"use strict"'));
+    });
+    if (codeStart > 0) {
+        return lines.slice(codeStart).join('\n').trim();
+    }
+    return text.trim();
+}

package/dist/core/cache.js

@@ -0,0 +1,105 @@
+/**
+ * testforge — change detection cache.
+ *
+ * Remembers a fingerprint (hash) of each journey file from the last
+ * run. On the next run we compare hashes: if a file's hash is the
+ * same, it has not changed and we can skip regenerating it. If it is
+ * different (edited) or missing (new file), we regenerate.
+ *
+ * The cache is stored as a small JSON file in the project so it
+ * survives between runs.
+ */
+import { createHash } from 'node:crypto';
+import { readFileSync, writeFileSync, existsSync, mkdirSync, readdirSync, statSync, } from 'node:fs';
+import { dirname, join, extname } from 'node:path';
+/** Make a short fingerprint of a file's content. */
+export function hashContent(content) {
+    return createHash('sha256').update(content).digest('hex');
+}
+/** Load the saved cache, or an empty one if it does not exist yet. */
+export function loadCache(cachePath) {
+    if (!existsSync(cachePath)) {
+        return {};
+    }
+    try {
+        return JSON.parse(readFileSync(cachePath, 'utf-8'));
+    }
+    catch {
+        // Corrupted cache — treat as empty so we just regenerate everything.
+        return {};
+    }
+}
+/** Save the cache back to disk. */
+export function saveCache(cachePath, cache) {
+    mkdirSync(dirname(cachePath), { recursive: true });
+    writeFileSync(cachePath, JSON.stringify(cache, null, 2), 'utf-8');
+}
+/**
+ * Build a single hash representing the current state of a project's
+ * source code. Reads source files (by extension), concatenates their
+ * paths + contents, and hashes the result. If any file changes, is
+ * added, or removed, the hash changes — deterministically, with no AI
+ * involved.
+ *
+ * Skips non-source folders (node_modules, .git, dist, etc.) and
+ * testforge's own output, so unrelated churn is ignored.
+ */
+const SOURCE_EXTENSIONS = [
+    '.js',
+    '.jsx',
+    '.ts',
+    '.tsx',
+    '.py',
+    '.go',
+    '.rb',
+    '.php',
+    '.java',
+];
+const SKIP_DIRS = new Set([
+    'node_modules',
+    '.git',
+    'dist',
+    'build',
+    'coverage',
+    '.next',
+    'docs',
+    '.testforge',
+    'generated-tests',
+]);
+export function hashProjectSource(rootDir) {
+    const parts = [];
+    function walk(dir) {
+        let entries;
+        try {
+            entries = readdirSync(dir);
+        }
+        catch {
+            return;
+        }
+        for (const entry of entries.sort()) {
+            const full = join(dir, entry);
+            let stat;
+            try {
+                stat = statSync(full);
+            }
+            catch {
+                continue;
+            }
+            if (stat.isDirectory()) {
+                if (SKIP_DIRS.has(entry) || entry.startsWith('.'))
+                    continue;
+                walk(full);
+            }
+            else if (SOURCE_EXTENSIONS.includes(extname(entry))) {
+                try {
+                    parts.push(full + '\n' + readFileSync(full, 'utf-8'));
+                }
+                catch {
+                    // unreadable file — skip
+                }
+            }
+        }
+    }
+    walk(rootDir);
+    return hashContent(parts.join('\n---\n'));
+}

package/dist/core/config.js

@@ -0,0 +1,100 @@
+/**
+ * testforge — configuration.
+ *
+ * testforge does not depend on any one AI provider. Instead, the user
+ * tells it which command to run to talk to their AI. That command must
+ * read a prompt from stdin and print the AI's reply to stdout.
+ *
+ * Configuration lives in testforge.config.json at the project root,
+ * for example:
+ *
+ *   {
+ *     "aiCommand": "claude -p"
+ *   }
+ *
+ * If no config file exists, we default to "claude -p" so users with
+ * Claude installed work out of the box. Users of other tools (Gemini,
+ * Codex, etc.) set aiCommand to their own CLI.
+ */
+import { readFileSync, existsSync } from 'node:fs';
+import { join } from 'node:path';
+const CONFIG_PATH = 'testforge.config.json';
+const DEFAULT_CONFIG = {
+    aiCommand: 'claude -p',
+    testOutput: {},
+    framework: { api: 'jest', web: 'playwright' },
+    seedOutput: 'generated-tests/seed',
+    testExtension: '.test.ts',
+    testCommand: {},
+    envFile: '.env.testing',
+};
+/** Load config from the project, falling back to defaults. */
+export function loadConfig() {
+    if (!existsSync(CONFIG_PATH)) {
+        return DEFAULT_CONFIG;
+    }
+    try {
+        const parsed = JSON.parse(readFileSync(CONFIG_PATH, 'utf-8'));
+        return { ...DEFAULT_CONFIG, ...parsed };
+    }
+    catch {
+        // Bad config file — fall back to defaults rather than crashing.
+        return DEFAULT_CONFIG;
+    }
+}
+/**
+ * Split an AI command string into a program and its arguments.
+ * Example: "claude -p" -> { program: "claude", args: ["-p"] }
+ */
+export function splitCommand(command) {
+    const parts = command.trim().split(/\s+/);
+    const program = parts[0];
+    const args = parts.slice(1);
+    return { program, args };
+}
+/**
+ * Decide where a journey's test file should go, based on its type
+ * and the config. Falls back to "generated-tests/<area>" when the
+ * type is not configured.
+ *
+ * "area" is the journey type today (web / api). This keeps web and
+ * api tests in separate, configurable locations to match a project's
+ * existing layout.
+ */
+export function testDirForType(config, type) {
+    const configured = config.testOutput?.[type];
+    if (configured) {
+        return configured;
+    }
+    return join('generated-tests', type);
+}
+/**
+ * Parse a simple .env file into a key/value object. Ignores blank
+ * lines and comments. Does not expand or validate — just reads what is
+ * there so we can pass it to a child process. Returns {} if the file
+ * is missing.
+ */
+export function loadEnvFile(path) {
+    if (!existsSync(path)) {
+        return {};
+    }
+    const out = {};
+    const text = readFileSync(path, 'utf-8');
+    for (const line of text.split('\n')) {
+        const trimmed = line.trim();
+        if (!trimmed || trimmed.startsWith('#'))
+            continue;
+        const eq = trimmed.indexOf('=');
+        if (eq === -1)
+            continue;
+        const key = trimmed.slice(0, eq).trim();
+        let value = trimmed.slice(eq + 1).trim();
+        // strip surrounding quotes if present
+        if ((value.startsWith('"') && value.endsWith('"')) ||
+            (value.startsWith("'") && value.endsWith("'"))) {
+            value = value.slice(1, -1);
+        }
+        out[key] = value;
+    }
+    return out;
+}