@haystackeditor/cli 0.8.0 → 0.9.0

package/dist/assets/skills/setup-haystack.md

@@ -0,0 +1,639 @@
+# Haystack Verification Setup
+
+> **PRO Feature**: Verification Flow requires a Haystack PRO subscription to run cloud sandboxes. Free users can still set up the `.haystack.json` configuration — it will be ready when they upgrade. See https://haystackeditor.com/pricing for details.
+
+**Your job**: Help the verification system understand this app so it can visually verify PRs work correctly.
+
+The verification system has two parts:
+1. **Planner** - An AI that explores the codebase to figure out what to test
+2. **Executor** - Takes screenshots based on the plan
+
+Your `.haystack.json` flows feed into the **corpus** - they're hints for the Planner about what routes exist, what selectors to use, and what the core user journey looks like.
+
+---
+
+## Step 0: Configure Dev Server for Cloud Sandboxes
+
+Cloud sandboxes access your dev server via tunnel URLs (`*.modal.host`). Most modern dev servers block non-localhost hosts by default. **You must configure your dev server to allow external hosts.**
+
+### Vite / SvelteKit / Astro / Nuxt 3
+
+```typescript
+// vite.config.ts (or framework equivalent)
+export default defineConfig({
+  server: {
+    allowedHosts: true, // Required for Haystack cloud sandboxes
+  },
+});
+```
+
+### Next.js
+
+Next.js allows all hosts by default in development. No configuration needed.
+
+If you've restricted hosts, add to `next.config.js`:
+```javascript
+module.exports = {
+  allowedDevHosts: ['localhost', '.modal.host'],
+};
+```
+
+### Webpack Dev Server (CRA, Vue CLI, custom)
+
+```javascript
+// webpack.config.js or vue.config.js
+module.exports = {
+  devServer: {
+    allowedHosts: 'all', // or ['.modal.host']
+  },
+};
+```
+
+For Create React App, set environment variable:
+```bash
+DANGEROUSLY_DISABLE_HOST_CHECK=true npm start
+```
+
+### Angular
+
+```json
+// angular.json → serve → options
+{
+  "disableHostCheck": true
+}
+```
+
+Or run with: `ng serve --disable-host-check`
+
+### Remix
+
+Remix dev server allows all hosts by default. No configuration needed.
+
+### Other Frameworks
+
+Look for `allowedHosts`, `disableHostCheck`, or `host` options in your dev server config. The goal is to allow requests from `*.modal.host` domains.
+
+**Skip this step if**: Your dev server already allows external hosts or doesn't have host checking.
+
+---
+
+## Step 1: Generate Base Config
+
+```bash
+npx @haystackeditor/cli init --yes
+```
+
+This detects your dev server command, port, and service dependencies.
+
+---
+
+## Step 2: Understand the App's Core Feature
+
+Before writing any flows, answer this:
+
+**What is the ONE main thing users do in this app?**
+
+Read the codebase to understand the core user journey:
+- E-commerce: browse products → add to cart → checkout
+- Dashboard: view metrics → filter data → export
+- Editor: create document → edit → save
+- Social: view feed → create post → interact
+- SaaS: sign up → configure → use feature
+
+Your flows should describe THIS journey, not just "pages load".
+
+---
+
+## Step 3: Verify ALL Services Have Flows (CRITICAL)
+
+**Every service in `.haystack.json` must have at least one verification flow.** This is the most common mistake - adding services but forgetting to add flows that exercise them.
+
+### Check for Uncovered Services
+
+```bash
+# List all services
+grep -A1 '"services"' .haystack.json | grep -E '^\s+"[^"]+":' | sed 's/.*"\([^"]*\)".*/\1/'
+
+# List all flows
+grep '"name":' .haystack.json
+
+# MANUAL CHECK: Does every service have a flow?
+```
+
+### Types of Services
+
+| Service Type | How to Verify |
+|--------------|---------------|
+| **Web frontend** | UI flows (navigate, wait_for, screenshot) |
+| **API/Worker** | UI flows that hit API endpoints |
+| **CLI/Batch** | Backend flows with golden inputs |
+| **Analysis pipeline** | Backend flows with known PR input |
+
+### Backend Verification (CLI tools, analysis pipelines, batch jobs)
+
+**Not everything is a web page.** For CLI tools and batch processes, you need:
+
+1. **Golden input** - A known-good input that produces predictable output
+2. **Run command** - How to execute the service
+3. **Output assertion** - What to check in the output
+
+```json
+{
+  "name": "Backend pipeline - golden input",
+  "description": "Run pipeline on known input to verify it works",
+  "trigger": "on_change",
+  "watch_patterns": ["packages/my-pipeline/src/**"],
+  "service": "my-pipeline",
+  "type": "backend",
+  "steps": [
+    {
+      "action": "run",
+      "command": "pnpm start",
+      "env": { "INPUT_ID": "known-good-input-123" },
+      "timeout": 120
+    },
+    { "action": "assert_exit_code", "code": 0 },
+    { "action": "assert_output_contains", "pattern": "Processing complete" }
+  ]
+}
+```
+
+---
+
+## STOP - Confirm Golden URLs and Golden Data
+
+**If you identified backend services that need verification, you MUST ask the user with a proposed example:**
+
+First, explore the codebase to find:
+- Example inputs used in tests, scripts, or documentation
+- Default/demo values in config files or environment variables
+- IDs or URLs referenced in comments or READMEs
+
+Then propose a specific example:
+
+> I found these backend services that need verification:
+> - **[service name]**: [what it does]
+>
+> For backend verification, I need **golden test data** - stable inputs that produce predictable output.
+>
+> **Based on what I found in the codebase, here's my suggestion:**
+> - **Golden URL/Input**: `[specific example you found, e.g., a test ID, demo endpoint, or sample file]`
+> - **Command**: `[the command to run with this input]`
+> - **Expected Output**: `[what success looks like based on the code]`
+>
+> Does this look right? Or should I use different test data?
+
+**Wait for the user to confirm or provide alternatives before adding backend flows.**
+
+---
+
+## Step 4: Assess Data Needs
+
+```bash
+# Find API calls
+grep -r "fetch(\|useQuery\|useSWR\|axios" src/ --include="*.tsx" --include="*.ts" | wc -l
+
+# Find dynamic routes
+grep -r "useParams\|router.query\|\[.*\]" src/ --include="*.tsx" | head -10
+
+# Find external domains
+grep -r "https://" src/ --include="*.ts" --include="*.tsx" | grep -v node_modules | head -10
+```
+
+---
+
+## STOP - Ask About Auth Requirements
+
+**You MUST ask the user before proceeding with fixtures or flows:**
+
+> Does this app require signed-in users to access the main features?
+>
+> Looking at the codebase, I see [authentication code, protected routes, user state, etc.].
+>
+> **How should verification handle authentication?**
+>
+> | Strategy | Best For | Complexity |
+> |----------|----------|------------|
+> | **Bypass** | Dev mode skips auth entirely | ⭐ Simplest |
+> | **Fixture** | Mock auth endpoints return fake user | ⭐⭐ Easy |
+> | **Token** | Inject real session token (you provide) | ⭐⭐⭐ Medium |
+> | **Test Account** | Full login flow with test credentials | ⭐⭐⭐⭐ Complex |
+>
+> 1. **Bypass** - Set env var like `SKIP_AUTH=true` (does your app support this?)
+> 2. **Fixture** - I'll create a mock `/auth/status` that returns a fake user
+> 3. **Token** - You provide a session cookie/token value to inject
+> 4. **Test Account** - Automated login with username/password (most realistic)
+
+**Wait for the user's response, then configure auth accordingly.**
+
+### Configuring Auth Based on User Response
+
+**Bypass (simplest):**
+```json
+{
+  "auth": {
+    "strategy": "bypass",
+    "env": { "VITE_SKIP_AUTH": "true" }
+  }
+}
+```
+
+**Fixture (mock auth endpoint):**
+```json
+{
+  "auth": {
+    "strategy": "fixture",
+    "env": { "VITE_SKIP_AUTH": "true" },
+    "fixture": {
+      "pattern": "/auth/status",
+      "source": "file://fixtures/auth-status.json"
+    }
+  }
+}
+```
+
+Create `fixtures/auth-status.json`:
+```json
+{
+  "authenticated": true,
+  "user": {
+    "id": "test-user-123",
+    "email": "test@example.com",
+    "name": "Test User"
+  }
+}
+```
+
+**Token (inject session):**
+```json
+{
+  "auth": {
+    "strategy": "token",
+    "token": {
+      "type": "cookie",
+      "name": "session",
+      "value": "$SESSION_TOKEN"
+    }
+  }
+}
+```
+
+The `$SESSION_TOKEN` is resolved from Haystack secrets. Tell the user:
+> To use token auth, you'll need to add your session value as a secret:
+> ```bash
+> npx @haystackeditor/cli secrets set SESSION_TOKEN "your-session-value"
+> ```
+
+**Test Account (full login flow):**
+```json
+{
+  "auth": {
+    "strategy": "test_account",
+    "test_account": {
+      "username": "$TEST_USERNAME",
+      "password": "$TEST_PASSWORD",
+      "login_url": "/login",
+      "steps": [
+        { "action": "type", "selector": "input[name=email]", "value": "$TEST_USERNAME" },
+        { "action": "type", "selector": "input[name=password]", "value": "$TEST_PASSWORD" },
+        { "action": "click", "selector": "button[type=submit]" }
+      ]
+    }
+  }
+}
+```
+
+---
+
+## STOP - Ask About Database Requirements
+
+**You MUST ask the user if the app needs database access:**
+
+> Does this app need a real database for verification?
+>
+> Looking at the codebase, I see [Prisma/Drizzle/Knex/raw SQL queries/etc.].
+>
+> **How should verification handle database access?**
+>
+> | Strategy | Best For | Complexity |
+> |----------|----------|------------|
+> | **Fixture** | Mock API responses (no real DB needed) | ⭐ Simplest |
+> | **Memory** | In-memory SQLite for read-heavy apps | ⭐⭐ Easy |
+> | **Seed** | File-based DB with seed data | ⭐⭐⭐ Medium |
+> | **Docker** | Spin up Postgres/MySQL/Redis containers | ⭐⭐⭐⭐ Complex |
+> | **Remote** | Connect to read-only staging database | ⭐⭐⭐⭐⭐ Advanced |
+>
+> For most apps, **fixtures are sufficient** — they mock the API layer so no real DB is needed.
+
+**Wait for the user's response, then configure database accordingly.**
+
+### Configuring Database Based on User Response
+
+**Fixture (simplest - mock API responses):**
+```json
+{
+  "fixtures": [
+    { "pattern": "/api/users", "source": "file://fixtures/users.json" },
+    { "pattern": "/api/posts/*", "source": "file://fixtures/posts.json" }
+  ]
+}
+```
+
+**Memory (in-memory SQLite):**
+```json
+{
+  "database": {
+    "strategy": "memory",
+    "env": { "DATABASE_URL": "sqlite::memory:" }
+  }
+}
+```
+
+**Seed (file-based with seed data):**
+```json
+{
+  "database": {
+    "strategy": "seed",
+    "env": { "DATABASE_URL": "file:./test.db" },
+    "seed_command": "pnpm db:seed"
+  }
+}
+```
+
+**Docker (real database containers):**
+```json
+{
+  "database": {
+    "strategy": "docker",
+    "services": [
+      {
+        "image": "postgres:15-alpine",
+        "port": 5432,
+        "env": {
+          "POSTGRES_USER": "test",
+          "POSTGRES_PASSWORD": "test",
+          "POSTGRES_DB": "testdb"
+        },
+        "health_check": "pg_isready -U test",
+        "health_timeout": 30
+      }
+    ],
+    "env": { "DATABASE_URL": "postgresql://test:test@localhost:5432/testdb" },
+    "migrate_command": "pnpm prisma migrate deploy",
+    "seed_command": "pnpm db:seed"
+  }
+}
+```
+
+**Remote (read-only staging database):**
+```json
+{
+  "database": {
+    "strategy": "remote",
+    "env": { "DATABASE_URL": "$STAGING_DATABASE_URL" },
+    "read_only": true
+  }
+}
+```
+
+The `$STAGING_DATABASE_URL` is resolved from Haystack secrets. Tell the user:
+> To use a remote database, add the connection string as a secret:
+> ```bash
+> npx @haystackeditor/cli secrets set STAGING_DATABASE_URL "postgresql://..."
+> ```
+
+**Fallback strategies** (try multiple approaches):
+```json
+{
+  "database": {
+    "strategy": "memory",
+    "env": { "DATABASE_URL": "sqlite::memory:" },
+    "fallback": {
+      "strategy": "seed",
+      "env": { "DATABASE_URL": "file:./test.db" },
+      "seed_command": "pnpm db:seed"
+    }
+  }
+}
+```
+
+---
+
+## STOP - Ask About Test Data Strategy
+
+**You MUST ask the user before proceeding:**
+
+> I analyzed the codebase and found:
+> - X API fetch calls
+> - Y dynamic routes with parameters
+> - These external domains: [list them]
+>
+> **How should I handle test data for verification?**
+>
+> 1. **Passthrough** - Let API calls through to real servers (add domains to `network.allow`)
+> 2. **Staging URL** - Point flows at your staging/demo environment
+> 3. **Local fixtures** - JSON files in `fixtures/` directory
+> 4. **Skip data pages** - Only test static pages that don't need API data
+
+**Wait for the user's response before continuing.**
+
+---
+
+## STOP - Confirm Frontend Golden URLs
+
+**Goldens are URLs with real staging data** - they let verification test pages that need actual content to render properly. You need MULTIPLE goldens to cover different scenarios.
+
+**Before writing flows for dynamic routes, explore the codebase to find example data:**
+
+1. Check test files (`*.test.ts`, `cypress/`, `playwright/`) for example IDs/URLs
+2. Check README/docs for demo links or example usage
+3. Check seed data, fixtures, or mock files
+4. Check `.env.example` or config for staging URLs
+5. Check existing routes to understand the URL format
+
+**Then propose MULTIPLE golden URLs (aim for 3-5) covering different scenarios:**
+
+> I found these dynamic routes that need staging data to render:
+>
+> **[Route Name]** (`/path/:param1/:param2`)
+>
+> | Golden | URL | Why |
+> |--------|-----|-----|
+> | Large dataset | `/path/example/large-123` | Tests performance with many items |
+> | Typical case | `/path/example/medium-456` | Common user scenario |
+> | Edge case | `/path/example/empty-789` | Tests empty/minimal state |
+> | Error state | `/path/example/error-000` | Tests error handling UI |
+>
+> **Evidence:**
+> - Found `large-123` in `cypress/fixtures/test-data.json`
+> - Found `medium-456` in README as demo example
+> - Found `empty-789` in `src/tests/edge-cases.test.ts`
+>
+> Does this look right?
+> 1. ✅ Yes, use these goldens
+> 2. 🔄 Use different URLs (I'll provide them)
+
+**Add goldens to `verification.goldens` in `.haystack.json` using this EXACT format:**
+
+```json
+{
+  "verification": {
+    "goldens": {
+      "frontend": [
+        {
+          "id": "large-dataset",
+          "description": "100+ items - tests rendering performance",
+          "route": "/the/actual/url/with/data"
+        },
+        {
+          "id": "typical-case",
+          "description": "Standard 10-20 items",
+          "route": "/another/url/with/typical/data"
+        },
+        {
+          "id": "empty-state",
+          "description": "No items - tests empty state UI",
+          "route": "/url/that/shows/empty/state"
+        }
+      ],
+      "backend": [
+        {
+          "id": "golden-input-1",
+          "description": "Known-good input for pipeline testing",
+          "input": { "type": "github_pr", "value": "owner/repo#123" }
+        }
+      ]
+    },
+    "commands": [...]
+  }
+}
+```
+
+**DO NOT:**
+- Ask the user what format to use - USE THIS FORMAT
+- Put goldens at the top level - they go inside `verification.goldens`
+- Use a flat object like `{ "goldens": { "name": "url" } }` - use the array format above
+- Embed golden URLs only in flow steps - also add them to the goldens section for discoverability
+
+**CRITICAL**:
+- Propose MULTIPLE goldens (3-5), not just one
+- Cover different scenarios: large, typical, empty, error states
+- Always propose SPECIFIC URLs, never ask the user to freestyle
+- Show WHERE you found the example data (file path, line if possible)
+- Make sure URLs match the app's actual route format (check `<Route path=`)
+- Wait for confirmation before writing flows with these URLs
+
+---
+
+## Step 5: Write Flows
+
+Flows tell the Planner about your app's routes, UI elements, and user journeys.
+
+### Structure
+
+```yaml
+flows:
+  - name: "Descriptive name of what this tests"
+    trigger: always  # or on_change with watch_patterns
+    steps:
+      - action: navigate
+        url: "/"
+      - action: wait_for
+        selector: "[data-testid='specific-element']"
+      - action: click
+        selector: "[data-testid='button']"
+      - action: screenshot
+        name: "result"
+```
+
+### What to Include
+
+**Core journey flow** (trigger: always):
+- The main thing users do in your app
+- Multiple steps with interactions (click, type)
+- Waits for meaningful state changes
+
+**Route coverage flows**:
+- One flow per major route
+- Uses specific selectors the Planner can learn from
+- `watch_patterns` to only run when relevant files change
+
+### Finding Good Selectors
+
+```bash
+# Find data-testid attributes
+grep -r "data-testid" src/ --include="*.tsx" | head -20
+
+# Find aria-labels
+grep -r "aria-label" src/ --include="*.tsx" | head -20
+
+# Find component class names
+grep -r "className=" src/components/ --include="*.tsx" | head -20
+```
+
+Use specific selectors like:
+- `[data-testid='dashboard-chart']`
+- `[aria-label='Submit form']`
+- `.pricing-table`
+- `button[type='submit']`
+
+If good selectors don't exist, add `data-testid` to key components.
+
+### watch_patterns
+
+For flows that only matter when certain files change:
+
+```yaml
+- name: "Settings page works"
+  trigger: on_change
+  watch_patterns:
+    - "src/pages/settings/**"
+    - "src/components/settings/**"
+  steps:
+    - action: navigate
+      url: "/settings"
+    # ...
+```
+
+---
+
+## Step 6: Configure Fixtures (if needed)
+
+Based on user's answer in Step 3:
+
+**Passthrough**:
+```yaml
+network:
+  allow:
+    - "api.example.com"
+    - "cdn.example.com"
+```
+
+**Staging URL**:
+```yaml
+flows:
+  - name: "Dashboard with real data"
+    steps:
+      - action: navigate
+        url: "https://staging.example.com/dashboard"
+```
+
+**Local fixtures**:
+```yaml
+fixtures:
+  - pattern: "/api/user"
+    source: "file://fixtures/user.json"
+  - pattern: "/api/data/*"
+    source: "file://fixtures/data.json"
+```
+
+---
+
+## Step 7: Commit
+
+```bash
+git add .haystack.json fixtures/
+git commit -m "Add Haystack verification"
+```
+
+Done! The Planner will use your flows to understand the app and create verification plans for PRs.