@haystackeditor/cli 0.15.14 → 0.15.15

package/dist/assets/skills/install-verification.md

@@ -0,0 +1,67 @@
+# Install Haystack Verification
+
+Add a Haystack verification contract to this repository so Haystack (and any
+coding agent) can safely verify risky PRs and produce reviewable evidence.
+
+The deliverable is a small, reviewable PR containing:
+
+- `.haystack/verification.yml` — the verification contract
+- `scripts/haystack-preflight` — proves the app is up and answering
+- `scripts/haystack-smoke` — boots the app, runs a basic logged-in check, captures evidence
+- `.haystack/AGENTS.md` — how agents should use the contract
+
+## Ground rules
+
+1. **No production behavior changes.** Anything you add must be dev/test-only,
+   clearly isolated, and easy for a reviewer to verify as inert in production.
+2. **Prefer existing test utilities** (seed scripts, factories, test logins,
+   fixtures) over new application code.
+3. **Only add dev-only endpoints when necessary**, and gate them so they cannot
+   run in production (`NODE_ENV !== 'production'` or the framework equivalent).
+   If you add one, also add a test proving it is disabled in production mode,
+   and wire that test into `safety.production_disabled_check`.
+4. **Never reference real credentials, real payment providers, real email
+   delivery, or customer data.** Sandbox or mock everything external.
+5. **Keep the integration small.** A working smoke scenario beats ten
+   aspirational ones.
+
+## Steps
+
+1. **Scaffold.** Run `npx @haystackeditor/cli verification init` (or
+   `haystack verification init` if installed). This detects the framework and
+   writes starter files. Review everything it generated.
+
+2. **Inspect the repo.** Identify:
+   - framework, package manager, and how the app starts locally
+   - how the app knows it's "ready" (port, health endpoint, log line)
+   - existing seed/test data and how tests authenticate
+   - existing E2E/smoke tests you can reuse as scenarios
+
+3. **Fix the environment section.** Make `setup`, `start`, and `preflight`
+   real: `setup` must work on a clean clone; `preflight` must exit 0 only when
+   the app is actually usable (prefer a health endpoint over a port check).
+
+4. **Wire personas if a safe login path exists.** Personas are seeded fake
+   users (admin/member/viewer). Use an existing test-login helper if there is
+   one. If none exists and one is genuinely needed, add a dev-only helper per
+   ground rule 3 — otherwise leave personas empty.
+
+5. **Make the smoke scenario real.** It should boot the app, reach a
+   meaningful logged-in (or at least rendered) state, and leave evidence in the
+   artifact directory: server logs, the captured page, and screenshots if a
+   browser runner is available. If part of the flow can't be tested, append a
+   line to `<artifact_dir>/gaps.txt` saying what was skipped.
+
+6. **Validate until clean.** Iterate until both pass:
+   ```bash
+   haystack verification validate
+   haystack verification check-safety
+   haystack verification run smoke
+   ```
+   `validate` must not report `not_ready` or `unsafe`; `run smoke` must end
+   `verified` and write a manifest + review packet under the artifact dir.
+
+7. **Open a reviewable PR.** Keep it focused on the verification contract.
+   In the description, state explicitly: what was added, why each piece is
+   dev-only, and what the smoke scenario proves. Include the review packet
+   from your successful `run smoke` as evidence.

package/dist/assets/skills/submit.md

@@ -151,7 +151,20 @@ This uses GitHub device flow - no tokens to copy/paste.
 - The command uses your current branch as the PR head
 - Labels are applied automatically based on your chosen mode
 - You don't need to push manually - the command handles it
-- Use `haystack triage <pr>` to view full analysis results (rating, findings, agent fix prompts) for any PR at any time
+
+## Close the Loop After Submitting
+
+After `haystack submit` creates the PR, **run `haystack triage <pr>` before ending your turn.**
+It blocks until Haystack's full analysis completes (waking instantly on the completion event),
+then prints the rating, findings, and a per-finding `agentFixPrompt`.
+
+- **Findings marked auto-fixing**: leave them alone — Haystack's fixer is already on them
+- **Actionable findings**: address them (the `agentFixPrompt` is written for you), commit, and
+  push — analysis re-runs on the new commit
+- **Good to merge**: report the rating to your user and stop
+
+Don't end the session right after submit without checking — the verdict usually lands within a
+few minutes, and acting on it immediately is much cheaper than a human bouncing the PR back.
 
 ---
 

package/dist/commands/triage.js

@@ -20,6 +20,7 @@
  *   (default) Rich terminal output
  */
 import chalk from 'chalk';
+import WebSocket from 'ws';
 import { checkAnalysisReady, fetchRatingSynthesis, fetchAutoFixManifest } from '../utils/analysis-api.js';
 import { parseRemoteUrl } from '../utils/git.js';
 import { loadToken } from '../utils/auth.js';
@@ -202,31 +203,111 @@ function printHookOutput(pr, synthesis, manifest, status) {
     }
 }
 // ============================================================================
-// Polling
+// Waiting (event-driven with polling fallback)
 // ============================================================================
 const POLL_INTERVAL_MS = 5_000;
 const POLL_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes
+/** With a live event subscription, checks are event-driven; this slow cycle
+ *  is only a safety net against a silently dead socket. */
+const SUBSCRIBED_RECHECK_MS = 60_000;
+const PR_EVENTS_WS_BASE = process.env.HAYSTACK_PR_EVENTS_WS_URL ?? 'wss://haystackeditor.com/api/pr-events';
+/**
+ * Subscribe to the pr-events worker for this PR — the same real-time channel
+ * the web inbox uses. Any event (analysis_complete, commit_pushed, ...) calls
+ * `onEvent`, and the caller re-checks analysis state immediately instead of
+ * discovering it on the next poll tick. Returns null when a socket can't even
+ * be constructed; connection failures after that surface via isConnected().
+ */
+function subscribeToPrEvents(owner, repo, prNumber, token, onEvent, quiet) {
+    // Diagnostics are dim stderr lines, once per failure mode — polling is the
+    // designed fallback, but WHY live updates are off must stay visible (PR001).
+    // quiet (--hook) mode stays terse: the hook output is machine-parsed.
+    const diag = (msg) => {
+        if (!quiet)
+            process.stderr.write(`\n  ${chalk.dim(`[live updates unavailable: ${msg}; using polling]`)}\n`);
+    };
+    try {
+        const ws = new WebSocket(`${PR_EVENTS_WS_BASE}/${owner}/${repo}/${prNumber}/subscribe`, {
+            headers: { Authorization: `Bearer ${token}` },
+        });
+        let connected = false;
+        let reported = false;
+        ws.on('open', () => { connected = true; });
+        ws.on('message', () => onEvent());
+        ws.on('error', (err) => {
+            connected = false;
+            if (!reported) {
+                reported = true;
+                diag(err.message);
+            }
+        });
+        ws.on('close', () => { connected = false; });
+        const keepalive = setInterval(() => {
+            // A failed send means the socket died between ticks; the 'error'/'close'
+            // handlers above report it and flip isConnected() — nothing extra to say.
+            try {
+                ws.send('ping');
+            }
+            catch { /* surfaced via error/close handlers */ }
+        }, 30_000);
+        keepalive.unref?.();
+        return {
+            isConnected: () => connected,
+            close: () => {
+                clearInterval(keepalive);
+                // Idempotent teardown, not a fallback: close() on an already-dead
+                // socket can throw in ws and there is genuinely nothing to handle.
+                try {
+                    ws.close();
+                }
+                catch { /* already closed */ }
+            },
+        };
+    }
+    catch (err) {
+        diag(err instanceof Error ? err.message : String(err));
+        return null;
+    }
+}
 async function waitForAnalysis(owner, repo, prNumber, timeoutMs, token, quiet) {
     const deadline = Date.now() + timeoutMs;
     let consecutiveErrors = 0;
-    while (Date.now() < deadline) {
-        const result = await checkAnalysisReady(owner, repo, prNumber, token);
-        if (result.status === 'ready')
-            return 'ready';
-        if (result.status === 'error') {
-            consecutiveErrors++;
-            if (consecutiveErrors >= 5)
-                return 'error';
-        }
-        else {
-            consecutiveErrors = 0;
-        }
-        if (!quiet) {
-            process.stderr.write(`\r  ${chalk.dim('Waiting for analysis to complete...')}${'  '.repeat(5)}`);
+    // Event-driven wake: a pr-events message resolves the current sleep so the
+    // re-check happens the moment analysis completes, not on the next tick.
+    let wake = null;
+    const subscription = token
+        ? subscribeToPrEvents(owner, repo, prNumber, token, () => wake?.(), quiet)
+        : null;
+    try {
+        while (Date.now() < deadline) {
+            const result = await checkAnalysisReady(owner, repo, prNumber, token);
+            if (result.status === 'ready')
+                return 'ready';
+            if (result.status === 'error') {
+                consecutiveErrors++;
+                if (consecutiveErrors >= 5)
+                    return 'error';
+            }
+            else {
+                consecutiveErrors = 0;
+            }
+            const live = subscription?.isConnected() ?? false;
+            if (!quiet) {
+                const mode = live ? 'Waiting for analysis (live)...' : 'Waiting for analysis to complete...';
+                process.stderr.write(`\r  ${chalk.dim(mode)}${'  '.repeat(5)}`);
+            }
+            const interval = Math.min(live ? SUBSCRIBED_RECHECK_MS : POLL_INTERVAL_MS, Math.max(deadline - Date.now(), 0));
+            await new Promise((resolve) => {
+                const timer = setTimeout(() => { wake = null; resolve(); }, interval);
+                timer.unref?.();
+                wake = () => { clearTimeout(timer); wake = null; resolve(); };
+            });
         }
-        await new Promise(r => setTimeout(r, POLL_INTERVAL_MS));
+        return 'pending';
+    }
+    finally {
+        subscription?.close();
     }
-    return 'pending';
 }
 // ============================================================================
 // PR resolution

package/dist/commands/verification.d.ts

@@ -0,0 +1,21 @@
+export declare function verificationInitCommand(options: {
+    force?: boolean;
+    skill?: boolean;
+}): Promise<void>;
+export declare function verificationValidateCommand(options: {
+    json?: boolean;
+}): Promise<void>;
+export declare function verificationPreflightCommand(): Promise<void>;
+export declare function verificationRunCommand(scenarioName: string, options: {
+    json?: boolean;
+    skipPreflight?: boolean;
+    setup?: boolean;
+    timeout?: number;
+}): Promise<void>;
+export declare function verificationCollectCommand(options: {
+    scenario?: string;
+    json?: boolean;
+}): Promise<void>;
+export declare function verificationCheckSafetyCommand(options: {
+    json?: boolean;
+}): Promise<void>;

package/dist/commands/verification.js

@@ -0,0 +1,291 @@
+/**
+ * `haystack verification ...` — repo-local verification contract commands.
+ *
+ *   init               scaffold .haystack/verification.yml + scripts + skill
+ *   validate           can Haystack trust this repo's verification setup?
+ *   preflight          is the app up and answering?
+ *   run <scenario>     run a scenario, capture artifacts, write manifest + review packet
+ *   collect-artifacts  rebuild a manifest from artifacts already on disk
+ *   check-safety       prove the verification setup can't hurt anyone
+ *
+ * See docs/VERIFICATION-CONTRACT.md for the product framing and schema.
+ */
+import * as path from 'node:path';
+import chalk from 'chalk';
+import { CONTRACT_RELATIVE_PATH, ensureArtifactDir, loadContract, } from '../verification/contract.js';
+import { initVerification } from '../verification/init.js';
+import { validateContract } from '../verification/validate.js';
+import { checkSafety } from '../verification/safety.js';
+import { runCommand } from '../verification/runner.js';
+import { collectArtifacts, getHeadSha, readGaps, writeManifest, writeReviewPacket, } from '../verification/manifest.js';
+const DEFAULT_SCENARIO_TIMEOUT_S = 900;
+const PREFLIGHT_TIMEOUT_S = 300;
+// ─── init ─────────────────────────────────────────────────────────────────────
+export async function verificationInitCommand(options) {
+    const result = await initVerification(process.cwd(), {
+        force: options.force,
+        skillOnly: options.skill,
+    });
+    for (const file of result.created) {
+        console.log(chalk.green('  created ') + file);
+    }
+    for (const file of result.skipped) {
+        console.log(chalk.yellow('  skipped ') + file + chalk.dim(' (exists, use --force to overwrite)'));
+    }
+    if (result.created.length > 0) {
+        console.log();
+        console.log(chalk.white('Next steps:'));
+        console.log(chalk.cyan('  1.') + ` Review ${CONTRACT_RELATIVE_PATH} — detection is a starting point, not truth`);
+        console.log(chalk.cyan('  2.') + ' haystack verification validate');
+        console.log(chalk.cyan('  3.') + ' haystack verification run smoke');
+        if (options.skill) {
+            console.log();
+            console.log(chalk.dim('Hand .haystack/skills/install-verification/SKILL.md to your coding agent to complete the integration.'));
+        }
+    }
+}
+// ─── validate ─────────────────────────────────────────────────────────────────
+export async function verificationValidateCommand(options) {
+    const report = await validateContract();
+    if (options.json) {
+        console.log(JSON.stringify(report, null, 2));
+    }
+    else {
+        printValidationReport(report);
+    }
+    if (report.readiness === 'unsafe')
+        process.exit(2);
+    if (report.readiness === 'not_ready')
+        process.exit(1);
+}
+function printValidationReport(report) {
+    const styles = {
+        ready: chalk.green,
+        partially_ready: chalk.yellow,
+        not_ready: chalk.red,
+        unsafe: chalk.red,
+    };
+    console.log(chalk.white('Haystack Verification Contract: ') + styles[report.readiness](report.readiness));
+    console.log();
+    const sections = [
+        { title: 'Passed', level: 'pass', bullet: chalk.green('✓') },
+        { title: 'Missing', level: 'warn', bullet: chalk.yellow('•') },
+        { title: 'Broken', level: 'fail', bullet: chalk.red('✗') },
+        { title: 'Unsafe', level: 'unsafe', bullet: chalk.red('⚠') },
+    ];
+    for (const { title, level, bullet } of sections) {
+        const items = report.items.filter((i) => i.level === level);
+        if (items.length === 0)
+            continue;
+        console.log(chalk.white(`${title}:`));
+        for (const item of items) {
+            console.log(`${bullet} ${item.message}`);
+        }
+        console.log();
+    }
+}
+// ─── preflight ────────────────────────────────────────────────────────────────
+export async function verificationPreflightCommand() {
+    const contract = await requireContract();
+    const artifactDir = await ensureArtifactDir(contract);
+    const record = await runCommand(contract.environment.preflight, {
+        cwd: process.cwd(),
+        timeoutMs: PREFLIGHT_TIMEOUT_S * 1000,
+        logFile: path.join(artifactDir, 'logs', 'preflight.log'),
+        env: { ARTIFACT_DIR: artifactDir },
+    });
+    if (record.status === 'passed') {
+        console.log(chalk.green('✓ preflight passed') + chalk.dim(` (${(record.duration_ms / 1000).toFixed(1)}s)`));
+    }
+    else {
+        console.error(chalk.red(`✗ preflight ${record.status}`) + chalk.dim(` — see ${record.log_path}`));
+        process.exit(1);
+    }
+}
+// ─── run <scenario> ───────────────────────────────────────────────────────────
+export async function verificationRunCommand(scenarioName, options) {
+    const rootDir = process.cwd();
+    const contract = await requireContract();
+    const scenario = contract.scenarios[scenarioName];
+    if (!scenario) {
+        console.error(chalk.red(`Unknown scenario "${scenarioName}".`) +
+            ` Available: ${Object.keys(contract.scenarios).join(', ')}`);
+        process.exit(1);
+    }
+    const artifactDir = await ensureArtifactDir(contract);
+    const startedAt = new Date();
+    const commandsRun = [];
+    const env = { ARTIFACT_DIR: artifactDir };
+    // In --json mode stdout must carry ONLY the manifest — send progress to
+    // stderr and keep command streaming off (output still lands in log files).
+    const quiet = Boolean(options.json);
+    const progress = (msg) => (quiet ? console.error(msg) : console.log(msg));
+    const finish = async (status) => {
+        const manifest = {
+            contract_version: '1',
+            status,
+            scenario: scenarioName,
+            commit_sha: getHeadSha(rootDir),
+            started_at: startedAt.toISOString(),
+            completed_at: new Date().toISOString(),
+            commands_run: commandsRun,
+            artifacts: await collectArtifacts({
+                rootDir,
+                artifactDir,
+                extraPaths: contract.environment.log_paths,
+                since: startedAt,
+            }),
+            gaps: await readGaps(artifactDir),
+            safety: {
+                real_email_sent: contract.safety.allow_real_email,
+                real_payment_attempted: contract.safety.allow_real_payments,
+                customer_data_accessed: contract.safety.allow_customer_data,
+            },
+        };
+        const manifestPath = await writeManifest(manifest, artifactDir);
+        const packetPath = await writeReviewPacket(manifest, artifactDir);
+        if (options.json) {
+            console.log(JSON.stringify(manifest, null, 2));
+        }
+        else {
+            printRunSummary(manifest, manifestPath, packetPath, rootDir);
+        }
+        process.exit(status === 'verified' ? 0 : status === 'unsafe_to_verify' ? 2 : 1);
+    };
+    // Safety gate before anything runs.
+    const { safety } = contract;
+    if (safety.allow_real_email || safety.allow_real_payments || safety.allow_customer_data) {
+        console.error(chalk.red('Refusing to run: the safety policy permits real external side effects.') +
+            ' Fix the safety section of the contract first.');
+        await finish('unsafe_to_verify');
+    }
+    if (options.setup && contract.environment.setup) {
+        progress(chalk.dim(`$ ${contract.environment.setup}`));
+        const setup = await runCommand(contract.environment.setup, {
+            cwd: rootDir,
+            timeoutMs: 15 * 60 * 1000,
+            logFile: path.join(artifactDir, 'logs', 'setup.log'),
+            env,
+            quiet,
+        });
+        commandsRun.push(setup);
+        if (setup.status !== 'passed')
+            await finish('environment_not_ready');
+    }
+    if (!options.skipPreflight && scenario.preflight) {
+        progress(chalk.dim(`$ ${contract.environment.preflight}`));
+        const preflight = await runCommand(contract.environment.preflight, {
+            cwd: rootDir,
+            timeoutMs: PREFLIGHT_TIMEOUT_S * 1000,
+            logFile: path.join(artifactDir, 'logs', 'preflight.log'),
+            env,
+            quiet,
+        });
+        commandsRun.push(preflight);
+        if (preflight.status !== 'passed')
+            await finish('environment_not_ready');
+    }
+    const timeoutS = options.timeout ?? scenario.timeout_seconds ?? DEFAULT_SCENARIO_TIMEOUT_S;
+    progress(chalk.dim(`$ ${scenario.command}`));
+    const run = await runCommand(scenario.command, {
+        cwd: rootDir,
+        timeoutMs: timeoutS * 1000,
+        logFile: path.join(artifactDir, 'logs', `scenario-${scenarioName}.log`),
+        env,
+        quiet,
+    });
+    commandsRun.push(run);
+    await finish(run.status === 'passed' ? 'verified' : 'failed');
+}
+function printRunSummary(manifest, manifestPath, packetPath, rootDir) {
+    const statusStyle = {
+        verified: chalk.green,
+        failed: chalk.red,
+        inconclusive: chalk.yellow,
+        environment_not_ready: chalk.yellow,
+        unsafe_to_verify: chalk.red,
+        missing_contract: chalk.red,
+    };
+    console.log();
+    console.log(chalk.white('Scenario: ') + manifest.scenario);
+    console.log(chalk.white('Status:   ') + statusStyle[manifest.status](manifest.status));
+    console.log(chalk.white('Evidence: ') + `${manifest.artifacts.length} artifact(s)`);
+    if (manifest.gaps.length > 0) {
+        console.log(chalk.white('Gaps:     ') + chalk.yellow(`${manifest.gaps.length} declared`));
+    }
+    console.log();
+    console.log(chalk.dim(`Manifest:      ${path.relative(rootDir, manifestPath)}`));
+    console.log(chalk.dim(`Review packet: ${path.relative(rootDir, packetPath)}`));
+}
+// ─── collect-artifacts ────────────────────────────────────────────────────────
+export async function verificationCollectCommand(options) {
+    const rootDir = process.cwd();
+    const contract = await requireContract();
+    const artifactDir = await ensureArtifactDir(contract);
+    const now = new Date().toISOString();
+    const manifest = {
+        contract_version: '1',
+        // Artifacts without a recorded run prove presence, not success.
+        status: 'inconclusive',
+        scenario: options.scenario ?? 'collected',
+        commit_sha: getHeadSha(rootDir),
+        started_at: now,
+        completed_at: now,
+        commands_run: [],
+        artifacts: await collectArtifacts({
+            rootDir,
+            artifactDir,
+            extraPaths: contract.environment.log_paths,
+        }),
+        gaps: await readGaps(artifactDir),
+        safety: {
+            real_email_sent: contract.safety.allow_real_email,
+            real_payment_attempted: contract.safety.allow_real_payments,
+            customer_data_accessed: contract.safety.allow_customer_data,
+        },
+    };
+    const manifestPath = await writeManifest(manifest, artifactDir);
+    const packetPath = await writeReviewPacket(manifest, artifactDir);
+    if (options.json) {
+        console.log(JSON.stringify(manifest, null, 2));
+    }
+    else {
+        printRunSummary(manifest, manifestPath, packetPath, rootDir);
+    }
+}
+// ─── check-safety ─────────────────────────────────────────────────────────────
+export async function verificationCheckSafetyCommand(options) {
+    const report = await checkSafety();
+    if (options.json) {
+        console.log(JSON.stringify(report, null, 2));
+    }
+    else {
+        for (const check of report.checks) {
+            const bullet = check.status === 'pass' ? chalk.green('✓') : check.status === 'warn' ? chalk.yellow('•') : chalk.red('✗');
+            console.log(`${bullet} ${chalk.white(check.name)}: ${check.detail}`);
+        }
+        console.log();
+        console.log(report.ok
+            ? chalk.green('Safety checks passed.')
+            : chalk.red('Safety checks failed — fix the items above before running verification.'));
+    }
+    if (!report.ok)
+        process.exit(1);
+}
+// ─── shared ───────────────────────────────────────────────────────────────────
+async function requireContract() {
+    const loaded = await loadContract();
+    if (loaded.status === 'missing') {
+        console.error(chalk.red(`No verification contract found at ${CONTRACT_RELATIVE_PATH}.`) +
+            ' Run `haystack verification init` to scaffold one.');
+        process.exit(1);
+    }
+    if (loaded.status === 'invalid') {
+        console.error(chalk.red(`Invalid ${CONTRACT_RELATIVE_PATH}:`));
+        for (const error of loaded.errors) {
+            console.error(chalk.red(`  ✗ ${error}`));
+        }
+        process.exit(1);
+    }
+    return loaded.contract;
+}