@test-bro/cli 0.1.0

package/README.md

@@ -0,0 +1,855 @@
+# TestBro CLI
+
+Command-line interface for AI-powered browser testing. Generate, run, and manage automated Playwright tests from your terminal.
+
+## Installation
+
+### Global Installation (Recommended)
+
+```bash
+npm install -g testbro-cli
+```
+
+### Local Development
+
+```bash
+cd apps/cli
+pnpm install
+pnpm build
+pnpm link --global
+```
+
+## Quick Start
+
+### 1. Authenticate
+
+```bash
+# Get API token from https://testbro.dev/settings/tokens
+testbro login --token tb_live_xxxxx
+```
+
+### 2. Create Project Configuration
+
+```bash
+# Initialize config in your project directory
+testbro config init
+
+# Edit test-bro.config.json
+{
+  "baseUrl": "https://your-app.com",
+  "projectId": "your-project-id",
+  "mode": "hybrid"
+}
+```
+
+### 3. Run Tests
+
+```bash
+# Run from natural language description
+testbro run --description "Test user login with valid credentials"
+
+# Run from PRD/requirements file
+testbro run --file ./requirements.md
+
+# Run saved test case
+testbro run --test-case tc_abc123
+
+# Run all project test cases
+testbro run --project proj_xyz789
+```
+
+## Commands
+
+### Authentication
+
+#### `testbro init`
+
+Interactive setup wizard for first-time configuration. Recommended for new users.
+
+```bash
+# Interactive mode (recommended)
+testbro init
+
+# Non-interactive mode (CI/CD)
+testbro init --token tb_live_xxxxx --non-interactive
+
+# Non-interactive with project
+testbro init --token tb_live_xxxxx --project-id proj_abc123 --non-interactive
+
+# Skip project setup
+testbro init --token tb_live_xxxxx --skip-project --non-interactive
+```
+
+**Options:**
+- `--token <token>`: API token (skips interactive authentication prompt)
+- `--non-interactive`: Skip all prompts, use defaults (for CI/CD)
+- `--project-id <id>`: Set project ID in non-interactive mode
+- `--skip-project`: Skip project setup entirely
+
+**What it does:**
+
+1. **Global Configuration**: Creates `~/.testbro/config.json` if needed
+2. **Authentication**: Prompts for API token and validates it
+3. **Project Setup**: Offers to create new project, use existing, or skip
+4. **Verification**: Runs health checks on API, token, and project access
+
+**Output (Interactive Mode):**
+```
+Welcome to TestBro!
+
+Step 1/4: Global configuration
+Config directory: ~/.testbro/ (will be created)
+Global config created at ~/.testbro/config.json
+
+Step 2/4: Authentication
+> Enter your API token: tb_live_***********
+Validating token... done
+Logged in as user@example.com
+
+Step 3/4: Project setup
+? Initialize project-level config here? (Y/n): Y
+? Create a new project or use an existing one?
+  > Create new project
+    Use existing project
+    Skip
+
+? Project name: My App
+? Base URL [http://localhost:3000]:
+? Default test mode [hybrid]:
+
+Creating project... done
+Project "My App" created (ID: proj_abc123)
+Created test-bro.config.json
+
+Step 4/4: Verification
+Checking API connection... healthy
+Checking authentication... valid
+Checking project access... ok
+
+Setup complete!
+
+Next steps:
+  testbro run --url http://localhost:3000 -d "Test the login page"
+  testbro generate --file requirements.md --save
+  testbro config show
+```
+
+#### `testbro login`
+
+Authenticate with TestBro API.
+
+```bash
+# Token-based authentication (recommended)
+testbro login --token tb_live_xxxxx
+
+# Custom API URL
+testbro login --token tb_live_xxxxx --url https://api.testbro.dev
+```
+
+**Options:**
+- `-t, --token <token>`: API token from your TestBro account
+- `-u, --url <url>`: API URL (default: http://localhost:3024)
+
+#### `testbro logout`
+
+Remove stored credentials.
+
+```bash
+testbro logout
+```
+
+#### `testbro whoami`
+
+Show current authenticated user and active project.
+
+```bash
+testbro whoami
+# Output:
+# Logged in as: user@example.com
+# Active Project: My Test Project (proj_123)
+```
+
+### Project Management
+
+#### `testbro projects`
+
+List all projects for the authenticated user.
+
+```bash
+testbro projects
+```
+
+**Output:**
+```
+Your Projects:
+  • My Web App (proj_abc123)
+    https://myapp.com
+    Created: 2024-01-15
+
+  • Mobile Site (proj_xyz789)
+    https://mobile.example.com
+    Created: 2024-01-20
+```
+
+#### `testbro projects create`
+
+Create a new project interactively.
+
+```bash
+testbro projects create
+
+# Interactive prompts:
+# ? Project name: My New App
+# ? Description (optional): E-commerce testing suite
+# ? Base URL (optional): https://shop.example.com
+```
+
+#### `testbro use-project`
+
+Set or clear the active project.
+
+```bash
+# Set active project
+testbro use-project proj_abc123
+
+# Clear active project
+testbro use-project --clear
+```
+
+**Options:**
+- `-c, --clear`: Clear the active project
+
+### Test Generation
+
+#### `testbro generate`
+
+Generate test cases from a file or description.
+
+```bash
+# Generate from PRD file
+testbro generate --file ./prd.md
+
+# Generate from text description
+testbro generate --description "Test checkout flow with payment"
+
+# Generate and save to project
+testbro generate --file ./requirements.md --save
+
+# Specify project
+testbro generate --file ./ac.md --save --project proj_123
+```
+
+**Options:**
+- `-f, --file <path>`: Path to requirements/PRD file
+- `-d, --description <text>`: Text description
+- `-s, --save`: Save generated test cases to project
+- `-p, --project <id>`: Project ID (overrides active project)
+
+**Example Output:**
+```
+Generating test cases...
+
+Generated 3 test cases:
+
+1. Test: User Login Flow
+   Steps:
+     1. Navigate to login page
+     2. Fill in email field
+     3. Fill in password field
+     4. Click login button
+     5. Verify redirect to dashboard
+
+2. Test: Invalid Login Attempt
+   Steps: ...
+
+3. Test: Password Reset Flow
+   Steps: ...
+
+Save to project? [y/N] y
+✓ Saved 3 test cases to project
+```
+
+### Test Execution
+
+#### `testbro run`
+
+Run tests against a URL.
+
+```bash
+# Run from description
+testbro run --url https://example.com --description "Test login flow"
+
+# Run from file (with deduplication)
+testbro run --file ./acceptance-criteria.md
+
+# Run specific test case
+testbro run --test-case tc_abc123
+
+# Run all project test cases
+testbro run --project proj_xyz789
+
+# Use different execution modes
+testbro run --mode ai --description "Test search functionality"
+testbro run --mode hybrid --file ./test-plan.md
+testbro run --mode selector --test-case tc_123
+
+# Run in headed mode (visible browser)
+testbro run --headed --description "Debug login issue"
+
+# Learn selectors from AI actions
+testbro run --mode ai --learn-selectors --yes
+```
+
+**Options:**
+
+**Target:**
+- `-u, --url <url>`: Target URL (or use baseUrl from config)
+- `--file <path>`: Generate and run from PRD/AC file
+- `-d, --description <text>`: Feature description to test
+- `-p, --project <id>`: Run all test cases from project
+- `-t, --test-case <id>`: Run specific test case
+
+**Execution:**
+- `-m, --mode <mode>`: Execution mode: `selector` (default), `ai`, or `hybrid`
+- `--headed`: Run browser in headed mode (visible)
+- `--remote`: Run on remote staging environment
+- `--timeout <ms>`: Timeout per step (default: 30000ms)
+- `--sequential`: Keep browser session between test cases (default: true)
+- `--no-sequential`: Start fresh browser session for each test case
+
+**Output:**
+- `-f, --format <format>`: Output format: `pretty` or `json` (default: `pretty`)
+- `-v, --verbose`: Show verbose output
+
+**Recording:**
+- `--record-video`: Record video of test execution (automatically uploads to server)
+- `--record-trace`: Capture Playwright trace for debugging (automatically uploads to server)
+
+**Selector Learning:**
+- `--learn-selectors`: Learn selectors from AI actions
+- `--yes`: Auto-confirm selector learning
+
+**Deduplication (when using --file):**
+- `--no-dedup`: Skip deduplication check
+- `--auto-merge`: Auto-apply smart merge without prompting
+
+**Examples:**
+
+```bash
+# Basic test run
+testbro run --url https://myapp.com --description "Test user registration"
+
+# Run with AI mode and selector learning
+testbro run --mode ai --learn-selectors --description "Test complex form"
+
+# Run from PRD with smart deduplication
+testbro run --file ./prd.md --auto-merge
+
+# Debug test in visible browser
+testbro run --headed --verbose --test-case tc_123
+
+# CI/CD integration (JSON output)
+testbro run --format json --project proj_123 > results.json
+```
+
+**Output (Pretty Format):**
+```
+Running tests...
+
+✓ Test: User Login Flow (2.3s)
+  ✓ Navigate to login page (450ms)
+  ✓ Fill in email field (320ms)
+  ✓ Fill in password field (290ms)
+  ✓ Click login button (180ms)
+  ✓ Verify redirect to dashboard (1.1s)
+
+Summary:
+  Tests: 1 passed, 0 failed, 1 total
+  Duration: 2.3s
+```
+
+**Output (JSON Format):**
+```json
+{
+  "summary": {
+    "totalTestCases": 1,
+    "totalPassed": 1,
+    "totalFailed": 0,
+    "totalDuration": 2340
+  },
+  "testCases": [
+    {
+      "id": "tc_1",
+      "name": "User Login Flow",
+      "status": "passed",
+      "duration": 2340,
+      "steps": [...]
+    }
+  ]
+}
+```
+
+### Configuration
+
+#### `testbro config`
+
+Show current configuration.
+
+```bash
+testbro config
+```
+
+**Output:**
+```
+TestBro Configuration
+
+User Config (~/.testbro/config.json)
+  Path: /Users/name/.testbro/config.json
+  API URL: http://localhost:3024
+  Authenticated: Yes
+  Active Project: proj_abc123
+
+Project Config (test-bro.config.json)
+  Path: /path/to/project/test-bro.config.json
+  Base URL: https://myapp.com
+  Environment: local
+  Mode: hybrid
+  Timeout: 30000ms
+  Project ID: proj_abc123
+
+Authentication
+  Token: tb_l...xyz (from user config)
+  OpenRouter API Key: sk-o...abc (from OPENROUTER_API_KEY env)
+```
+
+#### `testbro config show`
+
+Alias for `testbro config`.
+
+#### `testbro config init`
+
+Create test-bro.config.json template.
+
+```bash
+testbro config init
+```
+
+Note: For interactive setup with authentication and project selection, use `testbro init` instead.
+
+Creates `test-bro.config.json` in the current directory:
+
+```json
+{
+  "baseUrl": "https://your-app.com",
+  "environment": "local",
+  "mode": "hybrid",
+  "timeout": 30000,
+  "headed": false,
+  "projectId": "",
+  "credentials": {
+    "username": "${USERNAME}",
+    "password": "${PASSWORD}"
+  },
+  "deduplication": {
+    "enabled": true,
+    "autoMerge": false
+  }
+}
+```
+
+## Configuration Files
+
+### User Config (~/.testbro/config.json)
+
+Stores user-level settings and authentication.
+
+```json
+{
+  "apiUrl": "http://localhost:3024",
+  "token": "tb_live_xxxxx",
+  "activeProjectId": "proj_abc123"
+}
+```
+
+**Never commit this file to git.**
+
+### Project Config (test-bro.config.json)
+
+Project-specific settings, checked into version control.
+
+```json
+{
+  "baseUrl": "https://staging.myapp.com",
+  "environment": "staging",
+  "mode": "hybrid",
+  "timeout": 30000,
+  "headed": false,
+  "projectId": "proj_xyz789",
+  "credentials": {
+    "username": "${TEST_USERNAME}",
+    "password": "${TEST_PASSWORD}"
+  },
+  "deduplication": {
+    "enabled": true,
+    "autoMerge": false
+  }
+}
+```
+
+**Config Options:**
+
+- `baseUrl` (string): Default URL for tests
+- `environment` ("local" | "staging"): Execution environment
+- `mode` ("selector" | "ai" | "hybrid"): Default execution mode
+- `timeout` (number): Default step timeout in milliseconds
+- `headed` (boolean): Run browser in headed mode
+- `projectId` (string): TestBro project ID
+- `credentials` (object): Environment variable references for auth
+- `deduplication` (object): Test case deduplication settings
+  - `enabled` (boolean): Enable deduplication checks
+  - `autoMerge` (boolean): Automatically apply smart merge
+
+**Can commit to git.** Use environment variables for secrets.
+
+## Environment Variables
+
+### Required for AI Mode
+
+```bash
+OPENROUTER_API_KEY=sk-or-v1-xxxxx
+```
+
+### Optional
+
+```bash
+# Authentication (alternative to testbro login)
+TEST_BRO_TOKEN=tb_live_xxxxx
+
+# Test credentials (referenced in config)
+TEST_USERNAME=testuser@example.com
+TEST_PASSWORD=SecureP@ssw0rd
+
+# API URL override
+TESTBRO_API_URL=https://api.testbro.dev
+```
+
+## Session Continuity
+
+### Sequential Test Execution (Default)
+
+By default, TestBro keeps the browser session alive between test cases when running multiple tests in a single command. This enables sequential flows where later test cases build on the state left by earlier ones.
+
+**Example:**
+```bash
+testbro run --project proj_123
+# Test 1: Navigate to /login, log in, verify dashboard
+# Test 2: Continue from dashboard, click profile, verify profile page
+# Test 3: Continue from profile, edit settings, verify changes
+```
+
+**How it works:**
+- First test case includes a `navigate` step to the starting URL
+- Subsequent test cases skip their leading `navigate` step and continue from the current page
+- Browser context and cookies persist across all test cases in the run
+- If using auto-login (`storageState`), authentication is injected before the test plan starts
+
+**Opt out:**
+```bash
+testbro run --project proj_123 --no-sequential
+# Each test case starts fresh with a new browser session
+```
+
+**When to use `--no-sequential`:**
+- Tests should be fully independent
+- Each test requires a clean browser state
+- Testing logout/login flows that conflict with session persistence
+
+## Execution Modes
+
+### Selector Mode (Default)
+
+Fast execution using learned selectors from previous AI runs.
+
+```bash
+testbro run --mode selector --test-case tc_123
+```
+
+**Pros:**
+- Fast (no AI API calls)
+- Reliable for stable UIs
+- No API costs
+
+**Cons:**
+- Requires learned selectors
+- May fail on UI changes
+
+### AI Mode
+
+Uses GPT-4o-mini vision for every action.
+
+```bash
+testbro run --mode ai --description "Test complex workflow"
+```
+
+**Pros:**
+- Handles dynamic UIs
+- No selectors needed
+- Adapts to changes
+
+**Cons:**
+- Slower execution
+- API costs per run
+- Requires OPENROUTER_API_KEY
+
+### Hybrid Mode (Recommended)
+
+Tries selectors first, falls back to AI if needed.
+
+```bash
+testbro run --mode hybrid --file ./tests.md
+```
+
+**Pros:**
+- Best of both worlds
+- Fast when selectors work
+- Resilient to UI changes
+- Can learn new selectors
+
+**Cons:**
+- Requires OPENROUTER_API_KEY
+- Slight overhead on fallback
+
+## Selector Learning
+
+Automatically learn selectors from AI executions to speed up future runs.
+
+```bash
+# Enable selector learning
+testbro run --mode ai --learn-selectors --description "Test flow"
+
+# Auto-confirm learned selectors
+testbro run --mode hybrid --learn-selectors --yes
+```
+
+**Workflow:**
+1. AI executes actions and records element interactions
+2. Selectors are generated from DOM analysis
+3. Selectors are verified on the page
+4. User confirms selectors to save (or --yes auto-confirms)
+5. Selectors are stored in database
+6. Future runs use learned selectors (selector or hybrid mode)
+
+## Deduplication
+
+When generating tests from files, TestBro can detect and merge similar test cases.
+
+```bash
+# Enable deduplication (default)
+testbro run --file ./requirements.md
+
+# Skip deduplication
+testbro run --file ./requirements.md --no-dedup
+
+# Auto-merge similar tests
+testbro run --file ./requirements.md --auto-merge
+```
+
+**Deduplication Modes:**
+
+- **Smart (default)**: Prompt user for duplicates
+  - Skip test cases >90% similar
+  - Update test cases 75-89% similar
+  - Create new test cases <75% similar
+
+- **Force (--no-dedup)**: Create all test cases
+
+- **Auto-merge (--auto-merge)**: Apply smart mode without prompts
+
+## CI/CD Integration
+
+### GitHub Actions
+
+```yaml
+name: TestBro Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18
+
+      - name: Install TestBro CLI
+        run: npm install -g testbro-cli
+
+      - name: Run Tests
+        env:
+          TEST_BRO_TOKEN: ${{ secrets.TEST_BRO_TOKEN }}
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+        run: |
+          testbro run --project proj_123 --format json > results.json
+
+      - name: Upload Results
+        uses: actions/upload-artifact@v3
+        with:
+          name: test-results
+          path: results.json
+```
+
+### GitLab CI
+
+```yaml
+test:
+  image: node:18
+  script:
+    - npm install -g testbro-cli
+    - testbro run --project $PROJECT_ID --format json > results.json
+  artifacts:
+    paths:
+      - results.json
+  variables:
+    TEST_BRO_TOKEN: $TEST_BRO_TOKEN
+    OPENROUTER_API_KEY: $OPENROUTER_API_KEY
+```
+
+## Error Handling
+
+### Common Errors
+
+**Not authenticated:**
+```
+Error: Not authenticated. Run 'testbro login --token <token>' first.
+```
+
+**Solution:** Run `testbro login --token tb_live_xxxxx`
+
+**Missing OPENROUTER_API_KEY:**
+```
+Error: OPENROUTER_API_KEY environment variable is required for AI mode.
+```
+
+**Solution:** Set environment variable:
+```bash
+export OPENROUTER_API_KEY=sk-or-v1-xxxxx
+```
+
+**No URL specified:**
+```
+Error: No URL specified. Use --url or set baseUrl in test-bro.config.json
+```
+
+**Solution:** Add URL to config or use `--url` flag
+
+### Debug Mode
+
+```bash
+# Verbose output
+testbro run --verbose --test-case tc_123
+
+# Headed mode (see browser)
+testbro run --headed --description "Debug test"
+
+# Longer timeout
+testbro run --timeout 60000 --test-case tc_123
+```
+
+## Best Practices
+
+### 1. Use Project Config for Shared Settings
+
+```json
+{
+  "baseUrl": "https://staging.myapp.com",
+  "mode": "hybrid",
+  "projectId": "proj_123",
+  "deduplication": {
+    "enabled": true,
+    "autoMerge": false
+  }
+}
+```
+
+Commit to git for team consistency.
+
+### 2. Store Secrets in Environment Variables
+
+```bash
+# .env (don't commit)
+TEST_BRO_TOKEN=tb_live_xxxxx
+OPENROUTER_API_KEY=sk-or-v1-xxxxx
+TEST_USERNAME=user@example.com
+TEST_PASSWORD=password123
+```
+
+Reference in config:
+```json
+{
+  "credentials": {
+    "username": "${TEST_USERNAME}",
+    "password": "${TEST_PASSWORD}"
+  }
+}
+```
+
+### 3. Use Hybrid Mode for Best Results
+
+```bash
+testbro run --mode hybrid --learn-selectors
+```
+
+Balances speed and flexibility.
+
+### 4. Enable Selector Learning
+
+```bash
+testbro run --mode ai --learn-selectors --yes
+```
+
+First run uses AI, subsequent runs use selectors.
+
+### 5. Use JSON Output in CI/CD
+
+```bash
+testbro run --format json --project proj_123 > results.json
+```
+
+Parse results programmatically.
+
+## Development
+
+```bash
+# Install dependencies
+pnpm install
+
+# Build
+pnpm build
+
+# Link globally for testing
+pnpm link --global
+
+# Run locally
+pnpm start -- run --help
+
+# Type check
+pnpm typecheck
+
+# Lint
+pnpm lint
+```
+
+## Related Packages
+
+- [@testbro/core](../../packages/core) - Type definitions and schemas
+- [@testbro/runner](../../packages/runner) - Playwright test executor
+- [@testbro/ai-agent](../../packages/ai-agent) - AI-powered testing
+- [apps/web](../web) - TestBro web API
+
+## License
+
+MIT