@bugzy-ai/bugzy 1.5.0 → 1.6.0

package/dist/templates/init/.bugzy/runtime/testing-best-practices.md

@@ -1,5 +1,69 @@
 # Testing Best Practices Reference
 
+## Two-Phase Test Automation Workflow
+
+**Critical Distinction**: Separate test scenario discovery from automation implementation.
+
+### Phase 1: Test Scenario Discovery (WHAT to test)
+
+**Goal**: Understand application behavior and identify what needs testing coverage.
+
+**Activities**:
+- Explore features and user workflows through manual interaction
+- Identify critical user paths and edge cases
+- Document test scenarios in human-readable format
+- Evaluate automation ROI for each scenario
+- Create manual test case documentation
+
+**Output**: Test plan with prioritized scenarios and automation decisions
+
+### Phase 2: Automation Implementation (HOW to automate)
+
+**Goal**: Build robust test automation framework validated with working tests.
+
+**Activities**:
+- Technical exploration to identify correct selectors
+- Create Page Object infrastructure
+- Generate ONE smoke test to validate framework
+- Run and debug until test passes consistently
+- Scale to additional tests only after validation
+
+**Output**: Working test automation with validated Page Objects
+
+### The "Test One First" Validation Loop
+
+**CRITICAL**: Always validate your framework with ONE working test before scaling.
+
+```
+1. Explore app for selectors (use Playwright MCP or codegen)
+2. Create Page Objects with verified selectors
+3. Write ONE critical path test (e.g., login)
+4. Run the test: npx playwright test <test-file>
+5. If fails → Debug and fix → Go to step 4
+6. If passes → Run 3-5 more times to ensure stability
+7. Once stable → Scale to additional tests
+```
+
+**Why this matters**:
+- Catches framework issues early (config, setup, auth)
+- Validates selectors work in real application
+- Prevents generating 50 broken tests
+- Builds confidence in Page Object reliability
+
+**Example validation workflow**:
+```bash
+# Generate ONE test first
+npx playwright test tests/specs/auth/login.spec.ts
+
+# Run multiple times to verify stability
+npx playwright test tests/specs/auth/login.spec.ts --repeat-each=5
+
+# Check for flakiness
+npx playwright test tests/specs/auth/login.spec.ts --workers=1
+
+# Once stable, generate more tests
+```
+
 ## Page Object Model (POM) Architecture
 
 **Core Principle**: Separate locators, actions, and assertions into distinct layers to isolate UI changes from test logic.
@@ -69,6 +133,105 @@ Add `data-testid` attributes for:
 await page.getByTestId('checkout-submit').click();
 ```
 
+## Playwright Codegen for Selector Discovery
+
+**Playwright's built-in codegen is faster and more reliable than manual selector creation.**
+
+### Using Codegen
+
+```bash
+# Start codegen from specific URL
+npx playwright codegen https://your-app.com
+
+# With authentication (loads saved state)
+npx playwright codegen --load-storage=tests/.auth/user.json https://your-app.com
+
+# Target specific browser
+npx playwright codegen --browser=chromium https://your-app.com
+```
+
+**Workflow**:
+1. Run codegen and interact with your application
+2. Playwright generates test code with verified selectors
+3. Copy generated selectors to your Page Objects
+4. Refactor code to follow Page Object Model pattern
+5. Extract reusable logic to fixtures and helpers
+
+### Hybrid Approach: Codegen + AI Refactoring
+
+```
+1. Use Playwright codegen → Generates working test with selectors
+2. Use AI (Claude) → Refactor to Page Objects, extract fixtures, add types
+3. Best of both worlds: Reliability (codegen) + Intelligence (AI)
+```
+
+**Example**:
+```typescript
+// Raw codegen output
+await page.goto('https://example.com/');
+await page.getByLabel('Email').click();
+await page.getByLabel('Email').fill('test@example.com');
+
+// After AI refactoring into Page Object
+class LoginPage {
+  readonly emailInput = this.page.getByLabel('Email');
+
+  async fillEmail(email: string) {
+    await this.emailInput.fill(email);
+  }
+}
+```
+
+## Smoke Test Strategy
+
+**Smoke tests are a minimal suite of critical path tests that validate core functionality.**
+
+### Characteristics
+
+- **Fast**: Target < 5 minutes total execution time
+- **Critical**: Cover must-work features (login, core user flows)
+- **Stable**: High reliability, minimal flakiness
+- **CI/CD**: Run on every commit/pull request
+
+### Tagging Smoke Tests
+
+```typescript
+// tests/specs/auth/login.spec.ts
+test('should login with valid credentials @smoke', async ({ page }) => {
+  // Critical path test
+});
+
+test('should show error with invalid password', async ({ page }) => {
+  // Not tagged - functional test only
+});
+```
+
+### Running Smoke Tests
+
+```bash
+# Run only smoke tests
+npx playwright test --grep @smoke
+
+# In CI/CD pipeline
+npx playwright test --grep @smoke --workers=2
+
+# Smoke tests as gate for full suite
+npx playwright test --grep @smoke && npx playwright test
+```
+
+### Smoke Test Suite Example
+
+```
+@smoke test coverage:
+✓ Login with valid credentials
+✓ Navigate to dashboard
+✓ Create new item (core feature)
+✓ View item details
+✓ Logout
+
+Target: < 5 minutes, 100% pass rate
+```
+
 ## Test Organization
 
 ### File Structure by Feature
@@ -89,7 +252,13 @@ tests/
 └── setup/              # Global setup/teardown
 ```
 
-### Test Structure
+### Test Structure with test.step()
+
+**REQUIRED**: All tests must use `test.step()` to organize actions into high-level logical phases. This enables:
+- Video navigation by step (users can jump to specific phases in test execution videos)
+- Clear test structure and intent
+- Granular error tracking (know exactly which phase failed)
+- Better debugging with step-level timing
 
 ```typescript
 test.describe('Purchase flow', () => {
@@ -98,15 +267,186 @@ test.describe('Purchase flow', () => {
   });
 
   test('should complete purchase with credit card', async ({ page }) => {
-    // Arrange: Set up page objects
     const checkoutPage = new CheckoutPage(page);
 
-    // Act: Perform actions
-    await checkoutPage.fillPaymentInfo({/*...*/});
-    await checkoutPage.submitOrder();
+    await test.step('Add item to cart', async () => {
+      await checkoutPage.addItemToCart('Product A');
+      await expect(checkoutPage.cartCount).toHaveText('1');
+    });
+
+    await test.step('Navigate to checkout', async () => {
+      await checkoutPage.goToCheckout();
+      await expect(page).toHaveURL('/checkout');
+    });
+
+    await test.step('Fill payment information', async () => {
+      await checkoutPage.fillPaymentInfo({
+        cardNumber: '4111111111111111',
+        expiry: '12/25',
+        cvv: '123'
+      });
+    });
+
+    await test.step('Submit order', async () => {
+      await checkoutPage.submitOrder();
+      await expect(page).toHaveURL('/confirmation');
+    });
+
+    await test.step('Verify order confirmation', async () => {
+      await expect(checkoutPage.confirmationMessage).toBeVisible();
+      await expect(checkoutPage.orderNumber).toContain('ORD-');
+    });
+  });
+});
+```
+
+**Step Granularity Guidelines**:
+- Target **3-7 steps per test** for optimal video navigation
+- Each step should represent a logical phase (e.g., "Login", "Navigate to settings", "Update profile")
+- Avoid micro-steps (e.g., "Click button", "Fill field") - group related actions
+- Step titles should be user-friendly and descriptive
+
+## Video-Synchronized Test Steps
+
+**REQUIRED for all tests**: Use `test.step()` API to create video-navigable test execution.
+
+### Why test.step() is Required
+
+Every test generates a video recording with `steps.json` file containing:
+- Step-by-step breakdown of test actions
+- Video timestamps for each step (in seconds from test start)
+- Step status (success/failed)
+- Step duration
+
+This enables users to:
+- Click on a step to jump to that point in the video
+- See exactly when and where a test failed
+- Navigate through test execution like a timeline
+- Debug issues by reviewing specific test phases
+
+### test.step() Best Practices
+
+```typescript
+import { test, expect } from '@playwright/test';
+
+test('user can update profile settings', async ({ page }) => {
+  const settingsPage = new SettingsPage(page);
+  const profilePage = new ProfilePage(page);
+
+  await test.step('Navigate to settings page', async () => {
+    await settingsPage.navigate();
+    await expect(settingsPage.pageHeading).toBeVisible();
+  });
+
+  await test.step('Open profile section', async () => {
+    await settingsPage.clickProfileTab();
+    await expect(profilePage.nameInput).toBeVisible();
+  });
+
+  await test.step('Update profile information', async () => {
+    await profilePage.updateName('John Doe');
+    await profilePage.updateEmail('john@example.com');
+  });
+
+  await test.step('Save changes', async () => {
+    await profilePage.clickSaveButton();
+    await expect(profilePage.successMessage).toBeVisible();
+  });
+
+  await test.step('Verify changes persisted', async () => {
+    await page.reload();
+    await expect(profilePage.nameInput).toHaveValue('John Doe');
+    await expect(profilePage.emailInput).toHaveValue('john@example.com');
+  });
+});
+```
+
+### What Gets Recorded in steps.json
+
+```json
+{
+  "steps": [
+    {
+      "index": 1,
+      "timestamp": "2025-11-17T09:26:22.335Z",
+      "videoTimeSeconds": 0,
+      "action": "Navigate to settings page",
+      "status": "success",
+      "description": "Navigate to settings page - completed successfully",
+      "technicalDetails": "test.step",
+      "duration": 1234
+    },
+    {
+      "index": 2,
+      "timestamp": "2025-11-17T09:26:23.569Z",
+      "videoTimeSeconds": 1,
+      "action": "Open profile section",
+      "status": "success",
+      "description": "Open profile section - completed successfully",
+      "technicalDetails": "test.step",
+      "duration": 856
+    }
+  ],
+  "summary": {
+    "totalSteps": 5,
+    "successfulSteps": 5,
+    "failedSteps": 0,
+    "skippedSteps": 0
+  }
+}
+```
+
+### Step Naming Conventions
 
-    // Assert: Verify outcomes
-    await expect(page).toHaveURL('/confirmation');
+✅ **Good step names** (user-friendly, high-level):
+- "Navigate to login page"
+- "Login with valid credentials"
+- "Add item to cart"
+- "Complete checkout process"
+- "Verify order confirmation"
+
+❌ **Bad step names** (too technical, too granular):
+- "Click the login button"
+- "Fill email field"
+- "Wait for page load"
+- "Assert element visible"
+- "page.goto('/login')"
+
+### Smoke Test Example with test.step()
+
+```typescript
+// tests/specs/auth/login.spec.ts
+test('should login and navigate through all main pages @smoke', async ({ page }) => {
+  const loginPage = new LoginPage(page);
+  const dashboardPage = new DashboardPage(page);
+
+  await test.step('Navigate to login page', async () => {
+    await loginPage.navigate();
+    await expect(loginPage.pageHeading).toBeVisible();
+  });
+
+  await test.step('Login with valid credentials', async () => {
+    await loginPage.login(
+      process.env.TEST_OWNER_EMAIL!,
+      process.env.TEST_OWNER_PASSWORD!
+    );
+    await page.waitForURL(/.*\/dashboard/);
+  });
+
+  await test.step('Navigate to Overview page', async () => {
+    await dashboardPage.navigateToOverview();
+    await expect(dashboardPage.overviewNavLink).toBeVisible();
+  });
+
+  await test.step('Navigate to Settings page', async () => {
+    await dashboardPage.navigateToSettings();
+    await expect(dashboardPage.settingsNavLink).toBeVisible();
+  });
+
+  await test.step('Logout and verify redirect', async () => {
+    await dashboardPage.logout();
+    await page.waitForURL(/.*\/login/);
+    await expect(loginPage.pageHeading).toBeVisible();
   });
 });
 ```
@@ -119,7 +459,7 @@ test.describe('Purchase flow', () => {
 // tests/setup/auth.setup.ts
 import { test as setup } from '@playwright/test';
 
-const authFile = 'playwright/.auth/user.json';
+const authFile = 'tests/.auth/user.json';
 
 setup('authenticate', async ({ page }) => {
   await page.goto('/login');
@@ -139,7 +479,7 @@ projects: [
   { name: 'setup', testMatch: /.*\.setup\.ts/ },
   {
     name: 'chromium',
-    use: { storageState: 'playwright/.auth/user.json' },
+    use: { storageState: 'tests/.auth/user.json' },
     dependencies: ['setup'],
   },
 ]
@@ -264,15 +604,29 @@ export default defineConfig({
 - [ ] Assertions in test files, not Page Objects
 - [ ] Role-based selectors prioritized
 - [ ] No hardcoded credentials
+- [ ] Framework validated with ONE working test before scaling
+- [ ] Smoke tests tagged with @smoke for CI/CD
+- [ ] All tests use `test.step()` for video-navigable execution (3-7 steps per test)
+
+**Test Independence Validation:**
+- [ ] Each test can run in isolation: `npx playwright test <single-test>`
+- [ ] Tests pass in parallel: `npx playwright test --workers=4`
+- [ ] Tests pass in random order: `npx playwright test --shard=1/3` (run multiple shards)
+- [ ] No shared state between tests (each uses fixtures)
+- [ ] Tests cleanup after themselves (via fixtures or API)
 
 **CI/CD:**
-- [ ] Tests run on every pull request
+- [ ] Smoke tests run on every commit (`npx playwright test --grep @smoke`)
+- [ ] Full suite runs on pull requests
 - [ ] Artifacts uploaded (reports, traces)
 - [ ] Failure notifications configured
+- [ ] Test results published to PR comments
 
 ---
 
-**Remember**: The three critical pillars are:
-1. **Page Object Model** - Isolate UI changes from test logic
-2. **Role-based selectors** - Resist breakage
-3. **Authentication state reuse** - Maximize speed
+**Remember**: The five critical pillars are:
+1. **Two-Phase Approach** - Separate WHAT to test from HOW to automate
+2. **Test One First** - Validate framework with ONE working test before scaling
+3. **Page Object Model** - Isolate UI changes from test logic
+4. **Role-based selectors** - Resist breakage with semantic HTML
+5. **Authentication state reuse** - Maximize speed and reliability

package/dist/templates/init/.gitignore-template

@@ -1,4 +1,25 @@
-
-# Bugzy
+# Environment files (keep .env.testdata tracked, .env.example managed externally)
 .env
 .env.local
+
+# Logs and temporary files
+logs/
+tmp/
+
+# Playwright MCP cache
+.playwright-mcp/
+
+# Playwright test results
+test-results/
+playwright-report/
+playwright/.cache/
+tests/.auth/
+
+# Node modules if using any Node.js tooling
+node_modules/
+.DS_Store
+
+# Test result media files
+**/*.webm
+**/*.zip
+**/*.png

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bugzy-ai/bugzy",
-  "version": "1.5.0",
+  "version": "1.6.0",
   "description": "Open-source AI agent configuration for QA automation with Claude Code",
   "publishConfig": {
     "access": "public"

package/templates/init/.bugzy/runtime/templates/test-plan-template.md

@@ -1,25 +1,50 @@
-# Test Plan Template
+---
+version: 1.0.0
+created_at: [DATE]
+updated_at: [DATE]
+status: draft
+---
 
-## Test Scenarios
+# Test Plan: [PROJECT_NAME]
 
-### Scenario 1: [Name]
-**Objective**: [What are we testing?]
+## Overview
 
-**Test Cases**:
-1. [Test case description]
-2. [Test case description]
+[2-3 sentences describing what the application does and the testing focus]
 
-**Expected Results**:
-- [Expected outcome]
+## Features to Test
 
-### Scenario 2: [Name]
-**Objective**: [What are we testing?]
+### [Feature Area 1]
+- [ ] Feature 1.1 - Brief description
+- [ ] Feature 1.2 - Brief description
 
-**Test Cases**:
-1. [Test case description]
+### [Feature Area 2]
+- [ ] Feature 2.1 - Brief description
+- [ ] Feature 2.2 - Brief description
 
-**Expected Results**:
-- [Expected outcome]
+### [Feature Area 3]
+- [ ] Feature 3.1 - Brief description
+
+## Out of Scope
+
+- Item 1 - Reason (e.g., requires native mobile app)
+- Item 2 - Reason (e.g., backend-only, no UI)
+
+## Test Environment
+
+- **URL**: TEST_BASE_URL
+- **User Credentials**: TEST_USER_EMAIL / TEST_USER_PASSWORD
+- **Admin Credentials**: TEST_ADMIN_EMAIL / TEST_ADMIN_PASSWORD (if applicable)
+
+## Automation Priority
+
+| Priority | Criteria |
+|----------|----------|
+| High | Critical user flows, smoke tests, frequent regression areas |
+| Medium | Important features, moderate user impact |
+| Low | Edge cases, rarely used features |
 
 ## Notes
-[Add any additional context or requirements]
+
+- See `./exploration-reports/` for detailed UI element discovery
+- See `.bugzy/runtime/knowledge-base.md` for technical patterns
+- See `.bugzy/runtime/project-context.md` for SDLC and team info

package/templates/init/.env.testdata

@@ -0,0 +1,18 @@
+# Non-Secret Test Data
+# This file contains actual test values (non-sensitive) and IS committed to version control
+# This file will be populated by /onboard-testing command with discovered test data
+#
+# IMPORTANT: Passwords and API keys go in .env (not here!)
+# This file only contains non-sensitive data like URLs and email addresses.
+
+# Application Under Test
+TEST_BASE_URL=https://example.com
+
+# Test User Identifiers (non-secret - emails can be committed)
+# TEST_OWNER_EMAIL=owner@test.example.com
+# TEST_ADMIN_EMAIL=admin@test.example.com
+
+# NOTE: The corresponding passwords/secrets should be added to .env:
+# TEST_OWNER_PASSWORD=<add to .env>
+# TEST_ADMIN_PASSWORD=<add to .env>
+# TEST_API_KEY=<add to .env>