@bugzy-ai/bugzy 1.9.5 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +273 -273
- package/dist/cli/index.cjs +20 -20
- package/dist/cli/index.cjs.map +1 -1
- package/dist/cli/index.js +19 -19
- package/dist/cli/index.js.map +1 -1
- package/dist/index.cjs +17 -17
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +17 -17
- package/dist/index.js.map +1 -1
- package/dist/subagents/index.cjs +1 -1
- package/dist/subagents/index.cjs.map +1 -1
- package/dist/subagents/index.js +1 -1
- package/dist/subagents/index.js.map +1 -1
- package/dist/subagents/metadata.cjs +1 -1
- package/dist/subagents/metadata.cjs.map +1 -1
- package/dist/subagents/metadata.js +1 -1
- package/dist/subagents/metadata.js.map +1 -1
- package/dist/tasks/index.cjs.map +1 -1
- package/dist/tasks/index.js.map +1 -1
- package/package.json +1 -1
- package/templates/init/.bugzy/runtime/knowledge-base.md +61 -61
- package/templates/init/.bugzy/runtime/knowledge-maintenance-guide.md +97 -97
- package/templates/init/.bugzy/runtime/project-context.md +35 -35
- package/templates/init/.bugzy/runtime/subagent-memory-guide.md +87 -87
- package/templates/init/.bugzy/runtime/templates/test-plan-template.md +50 -50
- package/templates/init/.bugzy/runtime/templates/test-result-schema.md +498 -498
- package/templates/init/.bugzy/runtime/test-execution-strategy.md +535 -535
- package/templates/init/.bugzy/runtime/testing-best-practices.md +724 -724
- package/templates/init/.env.testdata +18 -18
- package/templates/init/.gitignore-template +24 -24
- package/templates/init/AGENTS.md +155 -155
- package/templates/init/CLAUDE.md +157 -157
- package/templates/init/test-runs/README.md +45 -45
- package/templates/playwright/BasePage.template.ts +190 -190
- package/templates/playwright/auth.setup.template.ts +89 -89
- package/templates/playwright/dataGenerators.helper.template.ts +148 -148
- package/templates/playwright/dateUtils.helper.template.ts +96 -96
- package/templates/playwright/pages.fixture.template.ts +50 -50
- package/templates/playwright/playwright.config.template.ts +97 -97
- package/templates/playwright/reporters/bugzy-reporter.ts +454 -454
- package/dist/templates/init/.bugzy/runtime/knowledge-base.md +0 -61
- package/dist/templates/init/.bugzy/runtime/knowledge-maintenance-guide.md +0 -97
- package/dist/templates/init/.bugzy/runtime/project-context.md +0 -35
- package/dist/templates/init/.bugzy/runtime/subagent-memory-guide.md +0 -87
- package/dist/templates/init/.bugzy/runtime/templates/test-plan-template.md +0 -50
- package/dist/templates/init/.bugzy/runtime/templates/test-result-schema.md +0 -498
- package/dist/templates/init/.bugzy/runtime/test-execution-strategy.md +0 -535
- package/dist/templates/init/.bugzy/runtime/testing-best-practices.md +0 -632
- package/dist/templates/init/.gitignore-template +0 -25
|
@@ -1,535 +1,535 @@
|
|
|
1
|
-
# Test Execution Strategy
|
|
2
|
-
|
|
3
|
-
## Overview
|
|
4
|
-
|
|
5
|
-
This document defines the test execution strategy for the Bugzy QA system, explaining **which tests to run when** and **why**. All agents and commands executing tests should reference this document to make intelligent decisions about test selection.
|
|
6
|
-
|
|
7
|
-
## Test Architecture Pattern
|
|
8
|
-
|
|
9
|
-
Our test suite follows a **pyramid testing strategy** where test cases can have multiple test implementations:
|
|
10
|
-
|
|
11
|
-
- **Comprehensive E2E Tests**: Full user journey from start to finish (tagged `@smoke`)
|
|
12
|
-
- **Granular Component Tests**: Isolated validation of specific capabilities (tagged `@component` or `@fast`)
|
|
13
|
-
|
|
14
|
-
**Example**: TC-003 (Project Settings Update) has:
|
|
15
|
-
- 1 comprehensive E2E test covering all 20 manual steps
|
|
16
|
-
- 2 granular tests covering navigation (steps 1-8) and field display (steps 1-10)
|
|
17
|
-
|
|
18
|
-
## Test Tiers
|
|
19
|
-
|
|
20
|
-
### Tier 1: Smoke Tests (`@smoke`)
|
|
21
|
-
|
|
22
|
-
**Purpose**: Quick validation of critical user journeys
|
|
23
|
-
|
|
24
|
-
**Coverage**:
|
|
25
|
-
- Comprehensive E2E tests only
|
|
26
|
-
- One test per test case
|
|
27
|
-
- 100% manual test case coverage
|
|
28
|
-
- **Zero redundancy**
|
|
29
|
-
|
|
30
|
-
**Execution Time**: ~2-5 minutes for full smoke suite
|
|
31
|
-
|
|
32
|
-
**When to Use**:
|
|
33
|
-
- **Default for most test runs**
|
|
34
|
-
- Every commit and pull request
|
|
35
|
-
- Quick validation after code changes
|
|
36
|
-
- Before starting new development work
|
|
37
|
-
- Continuous integration (CI) on every push
|
|
38
|
-
- Post-deployment verification
|
|
39
|
-
|
|
40
|
-
**Command**:
|
|
41
|
-
```bash
|
|
42
|
-
npx playwright test --grep "@smoke"
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
**Why This Tier**:
|
|
46
|
-
- Fast feedback loop
|
|
47
|
-
- Full coverage of critical paths
|
|
48
|
-
- No wasted execution time on redundant tests
|
|
49
|
-
- Ideal for frequent validation
|
|
50
|
-
|
|
51
|
-
---
|
|
52
|
-
|
|
53
|
-
### Tier 2: Component Tests (`@component` or `@fast`)
|
|
54
|
-
|
|
55
|
-
**Purpose**: Isolated component validation for debugging
|
|
56
|
-
|
|
57
|
-
**Coverage**:
|
|
58
|
-
- Granular tests for specific capabilities only
|
|
59
|
-
- Navigation, form display, API calls, etc.
|
|
60
|
-
- Subset coverage (e.g., just steps 1-8 for navigation)
|
|
61
|
-
- Skips comprehensive E2E tests
|
|
62
|
-
|
|
63
|
-
**Execution Time**: ~1-3 minutes (faster than smoke)
|
|
64
|
-
|
|
65
|
-
**When to Use**:
|
|
66
|
-
- Debugging specific component failures
|
|
67
|
-
- After refactoring a specific component
|
|
68
|
-
- When you need to verify a single capability without running full E2E
|
|
69
|
-
- Rapid iteration during development
|
|
70
|
-
- Targeted validation after component-level bug fixes
|
|
71
|
-
|
|
72
|
-
**Command**:
|
|
73
|
-
```bash
|
|
74
|
-
npx playwright test --grep "@component"
|
|
75
|
-
# OR
|
|
76
|
-
npx playwright test --grep "@fast"
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
**Why This Tier**:
|
|
80
|
-
- Ultra-fast feedback for specific areas
|
|
81
|
-
- Avoids running unnecessary setup/teardown
|
|
82
|
-
- Ideal for focused development work
|
|
83
|
-
|
|
84
|
-
---
|
|
85
|
-
|
|
86
|
-
### Tier 3: Full Regression (All Tests)
|
|
87
|
-
|
|
88
|
-
**Purpose**: Complete validation including redundant component tests
|
|
89
|
-
|
|
90
|
-
**Coverage**:
|
|
91
|
-
- Smoke tests (comprehensive E2E)
|
|
92
|
-
- Component tests (granular isolation)
|
|
93
|
-
- **Intentional redundancy** for diagnostic value
|
|
94
|
-
|
|
95
|
-
**Execution Time**: ~10-15 minutes
|
|
96
|
-
|
|
97
|
-
**When to Use**:
|
|
98
|
-
- Weekly scheduled CI runs
|
|
99
|
-
- Before major releases or deployments
|
|
100
|
-
- After significant refactoring across multiple areas
|
|
101
|
-
- When comprehensive validation is required
|
|
102
|
-
- Pre-merge validation for large features
|
|
103
|
-
- Monthly regression testing cycles
|
|
104
|
-
|
|
105
|
-
**Command**:
|
|
106
|
-
```bash
|
|
107
|
-
npx playwright test
|
|
108
|
-
# OR explicitly
|
|
109
|
-
npx playwright test --grep ".*"
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
**Why This Tier**:
|
|
113
|
-
- **Fast failure isolation**: If Settings page breaks, component test fails in 15 seconds vs E2E failing after 30 seconds
|
|
114
|
-
- **Component-level confidence**: Proves each capability works independently
|
|
115
|
-
- **Refactoring safety**: If you refactor save logic, component tests prove navigation/display still work
|
|
116
|
-
- **Comprehensive validation**: Catches edge cases that isolated tests might miss
|
|
117
|
-
|
|
118
|
-
**Understanding Redundancy**:
|
|
119
|
-
- Example: TC-003 login steps (1-5) run in all 3 tests
|
|
120
|
-
- **Cost**: Extra 30 seconds execution time
|
|
121
|
-
- **Benefit**: Instant failure location (know exactly which component broke)
|
|
122
|
-
- **Trade-off**: Worth it for regression scenarios, not for daily validation
|
|
123
|
-
|
|
124
|
-
---
|
|
125
|
-
|
|
126
|
-
## Default Behavior
|
|
127
|
-
|
|
128
|
-
### When User Provides No Selector
|
|
129
|
-
|
|
130
|
-
**Default Action**: Run `@smoke` tests
|
|
131
|
-
|
|
132
|
-
**Rationale**:
|
|
133
|
-
- Provides fast feedback (2-5 min vs 10-15 min)
|
|
134
|
-
- Full coverage of critical user journeys
|
|
135
|
-
- Zero redundancy = no wasted time
|
|
136
|
-
- Suitable for 90% of test execution scenarios
|
|
137
|
-
|
|
138
|
-
**User Communication**:
|
|
139
|
-
```
|
|
140
|
-
Running smoke tests (@smoke) for quick validation.
|
|
141
|
-
For comprehensive regression including component tests, use 'all'.
|
|
142
|
-
For specific areas, provide test file pattern or tag.
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
### When User Says "All"
|
|
146
|
-
|
|
147
|
-
**Action**: Run full regression suite (all tests)
|
|
148
|
-
|
|
149
|
-
**User Communication**:
|
|
150
|
-
```
|
|
151
|
-
Running full regression suite including component tests.
|
|
152
|
-
This includes intentional redundancy for diagnostic value and will take longer (~10-15 min).
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
---
|
|
156
|
-
|
|
157
|
-
## Context-Based Selection Logic
|
|
158
|
-
|
|
159
|
-
Parse user intent from their natural language request:
|
|
160
|
-
|
|
161
|
-
### Fast Validation Context
|
|
162
|
-
|
|
163
|
-
**Keywords**: "quick", "validate", "sanity check", "verify", "check"
|
|
164
|
-
|
|
165
|
-
**Action**: Run `@smoke` tests
|
|
166
|
-
|
|
167
|
-
**Examples**:
|
|
168
|
-
- "Quick check if login still works"
|
|
169
|
-
- "Validate the settings page"
|
|
170
|
-
- "Sanity test before I start work"
|
|
171
|
-
|
|
172
|
-
---
|
|
173
|
-
|
|
174
|
-
### Comprehensive Validation Context
|
|
175
|
-
|
|
176
|
-
**Keywords**: "full", "comprehensive", "regression", "all", "everything", "complete"
|
|
177
|
-
|
|
178
|
-
**Action**: Run all tests
|
|
179
|
-
|
|
180
|
-
**Examples**:
|
|
181
|
-
- "Full regression before release"
|
|
182
|
-
- "Comprehensive validation of the system"
|
|
183
|
-
- "Run everything to be sure"
|
|
184
|
-
|
|
185
|
-
---
|
|
186
|
-
|
|
187
|
-
### Debugging/Focused Context
|
|
188
|
-
|
|
189
|
-
**Keywords**: "debug", "check [specific area]", "[component name]", "just [feature]"
|
|
190
|
-
|
|
191
|
-
**Action**: Run specific test file or pattern
|
|
192
|
-
|
|
193
|
-
**Examples**:
|
|
194
|
-
- "Debug the settings page" → Run `TC-003` or pattern `settings`
|
|
195
|
-
- "Check authentication only" → Run pattern `auth` or `TC-001`
|
|
196
|
-
- "Just test the navigation" → Run `@component` tests
|
|
197
|
-
|
|
198
|
-
---
|
|
199
|
-
|
|
200
|
-
### Ambiguous Context
|
|
201
|
-
|
|
202
|
-
**When**: No clear keywords or selector provided
|
|
203
|
-
|
|
204
|
-
**Action**: Default to `@smoke` and inform user of options
|
|
205
|
-
|
|
206
|
-
**User Communication**:
|
|
207
|
-
```
|
|
208
|
-
No test selection specified. Running smoke tests (@smoke) by default.
|
|
209
|
-
|
|
210
|
-
Available options:
|
|
211
|
-
- '@smoke' - Quick validation (2-5 min, default)
|
|
212
|
-
- '@component' - Component tests only (1-3 min)
|
|
213
|
-
- 'all' - Full regression (10-15 min)
|
|
214
|
-
- '[pattern]' - Specific tests matching pattern
|
|
215
|
-
```
|
|
216
|
-
|
|
217
|
-
---
|
|
218
|
-
|
|
219
|
-
## Tag Taxonomy
|
|
220
|
-
|
|
221
|
-
### Primary Tags
|
|
222
|
-
|
|
223
|
-
| Tag | Purpose | Coverage | Speed |
|
|
224
|
-
|-----|---------|----------|-------|
|
|
225
|
-
| `@smoke` | Critical E2E journeys | Comprehensive, 100% manual test coverage | Medium (30-60s/test) |
|
|
226
|
-
| `@component` | Isolated component validation | Partial, specific capability only | Fast (10-20s/test) |
|
|
227
|
-
| `@fast` | Alias for `@component` | Same as @component | Fast |
|
|
228
|
-
| `@regression` | Full regression suite | All tests (smoke + component) | Slow (full suite) |
|
|
229
|
-
|
|
230
|
-
### Domain Tags (for filtering by feature area)
|
|
231
|
-
|
|
232
|
-
| Tag | Feature Area |
|
|
233
|
-
|-----|--------------|
|
|
234
|
-
| `@authentication` | Login, logout, session management |
|
|
235
|
-
| `@project-management` | Project CRUD, settings |
|
|
236
|
-
| `@test-management` | Test cases, test runs, activity feed |
|
|
237
|
-
| `@settings` | Settings pages and persistence |
|
|
238
|
-
| `@navigation` | Navigation and routing |
|
|
239
|
-
| `@data-persistence` | Data saving and loading |
|
|
240
|
-
|
|
241
|
-
### Combining Tags
|
|
242
|
-
|
|
243
|
-
**Multiple tags on same test**:
|
|
244
|
-
```typescript
|
|
245
|
-
test('should update settings and persist @smoke @project-management @settings', ...)
|
|
246
|
-
```
|
|
247
|
-
|
|
248
|
-
**Run tests matching multiple criteria**:
|
|
249
|
-
```bash
|
|
250
|
-
# Smoke tests in project management area
|
|
251
|
-
npx playwright test --grep "@smoke.*@project-management"
|
|
252
|
-
|
|
253
|
-
# All settings-related tests (smoke + component)
|
|
254
|
-
npx playwright test --grep "@settings"
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
---
|
|
258
|
-
|
|
259
|
-
## Time and Coverage Trade-offs
|
|
260
|
-
|
|
261
|
-
### Execution Time Comparison
|
|
262
|
-
|
|
263
|
-
| Tier | Tests Run | Time | Use Frequency |
|
|
264
|
-
|------|-----------|------|---------------|
|
|
265
|
-
| Smoke (`@smoke`) | ~6 tests | 2-5 min | Daily (90%) |
|
|
266
|
-
| Component (`@component`) | ~8 tests | 1-3 min | As needed (5%) |
|
|
267
|
-
| Full Regression (all) | ~15 tests | 10-15 min | Weekly (5%) |
|
|
268
|
-
|
|
269
|
-
### Coverage vs Speed Matrix
|
|
270
|
-
|
|
271
|
-
```
|
|
272
|
-
High Coverage ┐
|
|
273
|
-
│
|
|
274
|
-
│ ┌─ Full Regression
|
|
275
|
-
│ │ (High coverage, slower)
|
|
276
|
-
│ │
|
|
277
|
-
│ ┌────┘
|
|
278
|
-
│ │
|
|
279
|
-
│ │ Smoke Tests
|
|
280
|
-
│ │ (High coverage, faster)
|
|
281
|
-
│ │
|
|
282
|
-
├───┼─────────────────────►
|
|
283
|
-
│ │ Fast
|
|
284
|
-
│ │
|
|
285
|
-
│ └─ Component Tests
|
|
286
|
-
│ (Partial coverage, fastest)
|
|
287
|
-
│
|
|
288
|
-
Low Coverage ┘
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
### Recommendation by Scenario
|
|
292
|
-
|
|
293
|
-
| Scenario | Recommended Tier | Rationale |
|
|
294
|
-
|----------|------------------|-----------|
|
|
295
|
-
| Developer commit | Smoke | Fast feedback, full coverage |
|
|
296
|
-
| Pull request CI | Smoke | Balance speed and confidence |
|
|
297
|
-
| Pre-deployment | Full Regression | Comprehensive validation |
|
|
298
|
-
| Debugging component | Component or specific file | Fastest feedback for focused area |
|
|
299
|
-
| Weekly CI schedule | Full Regression | Diagnostic redundancy value |
|
|
300
|
-
| Post-hotfix | Smoke | Quick validation of critical paths |
|
|
301
|
-
|
|
302
|
-
---
|
|
303
|
-
|
|
304
|
-
## Example Scenarios
|
|
305
|
-
|
|
306
|
-
### Scenario 1: Developer Makes Code Change
|
|
307
|
-
|
|
308
|
-
**Context**: Developer updates Settings page form validation
|
|
309
|
-
|
|
310
|
-
**User Request**: `/run-tests` (no selector)
|
|
311
|
-
|
|
312
|
-
**Agent Decision**:
|
|
313
|
-
- Read this strategy file
|
|
314
|
-
- No selector provided → Default to `@smoke`
|
|
315
|
-
- Run: `npx playwright test --grep "@smoke"`
|
|
316
|
-
- Result: 6 tests in ~3 min, full E2E coverage, zero redundancy
|
|
317
|
-
|
|
318
|
-
**Communication**:
|
|
319
|
-
```
|
|
320
|
-
Running smoke tests (@smoke) for quick validation...
|
|
321
|
-
✓ 6/6 tests passed in 3m 12s
|
|
322
|
-
All critical user journeys validated successfully.
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
---
|
|
326
|
-
|
|
327
|
-
### Scenario 2: Pre-Release Validation
|
|
328
|
-
|
|
329
|
-
**User Request**: `/run-tests all`
|
|
330
|
-
|
|
331
|
-
**Agent Decision**:
|
|
332
|
-
- Explicit "all" keyword → Run full regression
|
|
333
|
-
- Run: `npx playwright test`
|
|
334
|
-
- Result: 15 tests in ~12 min, includes redundant component tests
|
|
335
|
-
|
|
336
|
-
**Communication**:
|
|
337
|
-
```
|
|
338
|
-
Running full regression suite including component tests...
|
|
339
|
-
This includes intentional redundancy for diagnostic value (~10-15 min).
|
|
340
|
-
|
|
341
|
-
✓ 15/15 tests passed in 12m 34s
|
|
342
|
-
Full regression validation complete:
|
|
343
|
-
- 6 comprehensive E2E tests (smoke)
|
|
344
|
-
- 9 component-level tests
|
|
345
|
-
All tests passed.
|
|
346
|
-
```
|
|
347
|
-
|
|
348
|
-
---
|
|
349
|
-
|
|
350
|
-
### Scenario 3: Debugging Settings Page Failure
|
|
351
|
-
|
|
352
|
-
**User Request**: `/run-tests settings`
|
|
353
|
-
|
|
354
|
-
**Agent Decision**:
|
|
355
|
-
- File pattern provided → Match specific tests
|
|
356
|
-
- Run: `npx playwright test settings`
|
|
357
|
-
- Result: Runs TC-003 (3 tests: 1 smoke + 2 component)
|
|
358
|
-
|
|
359
|
-
**Communication**:
|
|
360
|
-
```
|
|
361
|
-
Running tests matching pattern "settings"...
|
|
362
|
-
Found: tests/specs/project-management/TC-003-project-settings-update.spec.ts
|
|
363
|
-
|
|
364
|
-
Results:
|
|
365
|
-
✓ should update project settings and persist changes @smoke
|
|
366
|
-
✗ should navigate to settings page @component
|
|
367
|
-
└─ Error: Navigation timeout
|
|
368
|
-
✓ should display project name and description @component
|
|
369
|
-
|
|
370
|
-
Failure isolated to navigation component. Settings form display works correctly.
|
|
371
|
-
```
|
|
372
|
-
|
|
373
|
-
---
|
|
374
|
-
|
|
375
|
-
### Scenario 4: Quick Validation Request
|
|
376
|
-
|
|
377
|
-
**User Request**: "Can you quickly check if the login flow still works?"
|
|
378
|
-
|
|
379
|
-
**Agent Decision**:
|
|
380
|
-
- Keyword "quickly" → Use `@smoke`
|
|
381
|
-
- Implicit focus on "login" → Could run specific test or full smoke
|
|
382
|
-
- Run: `npx playwright test --grep "@smoke"` (include all smoke, not just login)
|
|
383
|
-
- Result: 6 tests including login in ~3 min
|
|
384
|
-
|
|
385
|
-
**Communication**:
|
|
386
|
-
```
|
|
387
|
-
Running smoke tests (@smoke) for quick validation of all critical flows including login...
|
|
388
|
-
|
|
389
|
-
✓ 6/6 tests passed in 3m 8s
|
|
390
|
-
All critical user journeys validated, including:
|
|
391
|
-
- TC-001: Login and navigation ✓
|
|
392
|
-
- TC-003: Project settings update ✓
|
|
393
|
-
- TC-004: Test cases page access ✓
|
|
394
|
-
- TC-005: Test runs page access ✓
|
|
395
|
-
- TC-006: Session management ✓
|
|
396
|
-
- TC-008: Activity feed access ✓
|
|
397
|
-
```
|
|
398
|
-
|
|
399
|
-
---
|
|
400
|
-
|
|
401
|
-
## Understanding Test Redundancy
|
|
402
|
-
|
|
403
|
-
### What is Redundant Coverage?
|
|
404
|
-
|
|
405
|
-
Some test suites contain multiple tests that exercise the same functionality:
|
|
406
|
-
|
|
407
|
-
**Example: TC-003 (Project Settings Update)**
|
|
408
|
-
|
|
409
|
-
```
|
|
410
|
-
Test 1 (E2E, @smoke): Steps 1-20 (login → navigate → update → verify)
|
|
411
|
-
Test 2 (Component, @component): Steps 1-8 (login → navigate) ← Overlaps with Test 1
|
|
412
|
-
Test 3 (Component, @component): Steps 1-10 (login → navigate → display) ← Overlaps with Test 1
|
|
413
|
-
```
|
|
414
|
-
|
|
415
|
-
**Overlap**: Steps 1-8 run in all 3 tests
|
|
416
|
-
|
|
417
|
-
### Why Allow Redundancy?
|
|
418
|
-
|
|
419
|
-
**Diagnostic Value**:
|
|
420
|
-
- If Test 2 fails but Test 1 passes → Issue is isolated to navigation
|
|
421
|
-
- If Test 3 fails but Tests 1 & 2 pass → Issue is in form display
|
|
422
|
-
- If Test 1 fails but Tests 2 & 3 pass → Issue is in update/save logic (steps 11-20)
|
|
423
|
-
|
|
424
|
-
**Speed Benefit**:
|
|
425
|
-
- Test 2 fails in 15 seconds (just navigation)
|
|
426
|
-
- Test 1 fails in 30 seconds (full E2E including navigation)
|
|
427
|
-
- **Saved time**: 15 seconds to identify failure point
|
|
428
|
-
|
|
429
|
-
### When is Redundancy Worth It?
|
|
430
|
-
|
|
431
|
-
**Smoke Runs**: NOT worth it (90% of executions)
|
|
432
|
-
- Goal: Fast validation
|
|
433
|
-
- Trade-off: Skip redundant tests, accept slower failure diagnosis
|
|
434
|
-
|
|
435
|
-
**Full Regression Runs**: Worth it (10% of executions)
|
|
436
|
-
- Goal: Comprehensive validation with diagnostic value
|
|
437
|
-
- Trade-off: Accept 30 extra seconds for faster failure isolation
|
|
438
|
-
- Context: Pre-release, weekly CI, post-refactoring
|
|
439
|
-
|
|
440
|
-
---
|
|
441
|
-
|
|
442
|
-
## Integration with Other Systems
|
|
443
|
-
|
|
444
|
-
### Test Generation (`/generate-test-cases`)
|
|
445
|
-
|
|
446
|
-
When generating new test cases, the generator should:
|
|
447
|
-
1. Create comprehensive E2E test with `@smoke` tag
|
|
448
|
-
2. Optionally create component tests with `@component` or `@fast` tags
|
|
449
|
-
3. Apply appropriate domain tags (`@authentication`, `@project-management`, etc.)
|
|
450
|
-
|
|
451
|
-
### CI/CD Pipelines
|
|
452
|
-
|
|
453
|
-
Recommended pipeline configuration:
|
|
454
|
-
|
|
455
|
-
```yaml
|
|
456
|
-
# On every PR - SMOKE ONLY
|
|
457
|
-
on_pull_request:
|
|
458
|
-
npx playwright test --grep "@smoke"
|
|
459
|
-
# Fast validation: 2-5 min
|
|
460
|
-
|
|
461
|
-
# Nightly regression - ALL TESTS
|
|
462
|
-
on_schedule:
|
|
463
|
-
npx playwright test
|
|
464
|
-
# Comprehensive validation: 10-15 min
|
|
465
|
-
|
|
466
|
-
# Pre-deployment - SMOKE (fast gate)
|
|
467
|
-
on_deployment:
|
|
468
|
-
npx playwright test --grep "@smoke"
|
|
469
|
-
# Critical path validation: 2-5 min
|
|
470
|
-
```
|
|
471
|
-
|
|
472
|
-
### Issue Tracking
|
|
473
|
-
|
|
474
|
-
When tests fail, triage results should consider:
|
|
475
|
-
- Smoke test failures → High priority (critical path broken)
|
|
476
|
-
- Component test failures (when smoke passes) → Medium priority (isolated issue)
|
|
477
|
-
- Component test failures (when smoke also fails) → Confirms root cause
|
|
478
|
-
|
|
479
|
-
---
|
|
480
|
-
|
|
481
|
-
## Decision Tree for Agents
|
|
482
|
-
|
|
483
|
-
```
|
|
484
|
-
User provides selector?
|
|
485
|
-
├─ Yes
|
|
486
|
-
│ ├─ Selector is "all" or "everything"?
|
|
487
|
-
│ │ └─ Run: npx playwright test (Full Regression)
|
|
488
|
-
│ ├─ Selector is tag (starts with @)?
|
|
489
|
-
│ │ └─ Run: npx playwright test --grep "[tag]"
|
|
490
|
-
│ ├─ Selector is file pattern (e.g., "auth", "settings")?
|
|
491
|
-
│ │ └─ Run: npx playwright test [pattern]
|
|
492
|
-
│ └─ Selector is specific file path?
|
|
493
|
-
│ └─ Run: npx playwright test [file-path]
|
|
494
|
-
└─ No selector provided
|
|
495
|
-
├─ User request contains context keywords?
|
|
496
|
-
│ ├─ "quick", "validate", "sanity" → Run: @smoke
|
|
497
|
-
│ ├─ "full", "comprehensive", "regression" → Run: all tests
|
|
498
|
-
│ └─ "debug [area]", "check [component]" → Run: specific pattern
|
|
499
|
-
└─ No context keywords
|
|
500
|
-
└─ Default: Run @smoke and inform user of options
|
|
501
|
-
```
|
|
502
|
-
|
|
503
|
-
---
|
|
504
|
-
|
|
505
|
-
## Maintenance Notes
|
|
506
|
-
|
|
507
|
-
### When to Update This Strategy
|
|
508
|
-
|
|
509
|
-
This document should be updated when:
|
|
510
|
-
- New test tiers are introduced
|
|
511
|
-
- Tag taxonomy changes
|
|
512
|
-
- Execution time significantly changes
|
|
513
|
-
- New testing patterns emerge
|
|
514
|
-
- CI/CD pipeline requirements change
|
|
515
|
-
|
|
516
|
-
### Related Documents
|
|
517
|
-
|
|
518
|
-
- `.bugzy/runtime/testing-best-practices.md` - How to write tests (test patterns)
|
|
519
|
-
- `.bugzy/runtime/knowledge-base.md` - Accumulated testing insights
|
|
520
|
-
- `.claude/commands/run-tests.md` - Test execution command (mechanics)
|
|
521
|
-
- `playwright.config.ts` - Playwright configuration
|
|
522
|
-
- Test cases in `./test-cases/` - Individual test case documentation
|
|
523
|
-
|
|
524
|
-
---
|
|
525
|
-
|
|
526
|
-
## Summary
|
|
527
|
-
|
|
528
|
-
**Key Principles**:
|
|
529
|
-
1. **Default to smoke tests** for fast validation (90% of runs)
|
|
530
|
-
2. **Use component tests** for focused debugging
|
|
531
|
-
3. **Run full regression** only when diagnostic redundancy is needed (10% of runs)
|
|
532
|
-
4. **Parse user context** to choose the right tier intelligently
|
|
533
|
-
5. **Communicate clearly** about what's running and why
|
|
534
|
-
|
|
535
|
-
**Remember**: The goal is to provide fast feedback most of the time, while maintaining the ability to run comprehensive validation when needed. Redundancy is intentional but conditional.
|
|
1
|
+
# Test Execution Strategy
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
This document defines the test execution strategy for the Bugzy QA system, explaining **which tests to run when** and **why**. All agents and commands executing tests should reference this document to make intelligent decisions about test selection.
|
|
6
|
+
|
|
7
|
+
## Test Architecture Pattern
|
|
8
|
+
|
|
9
|
+
Our test suite follows a **pyramid testing strategy** where test cases can have multiple test implementations:
|
|
10
|
+
|
|
11
|
+
- **Comprehensive E2E Tests**: Full user journey from start to finish (tagged `@smoke`)
|
|
12
|
+
- **Granular Component Tests**: Isolated validation of specific capabilities (tagged `@component` or `@fast`)
|
|
13
|
+
|
|
14
|
+
**Example**: TC-003 (Project Settings Update) has:
|
|
15
|
+
- 1 comprehensive E2E test covering all 20 manual steps
|
|
16
|
+
- 2 granular tests covering navigation (steps 1-8) and field display (steps 1-10)
|
|
17
|
+
|
|
18
|
+
## Test Tiers
|
|
19
|
+
|
|
20
|
+
### Tier 1: Smoke Tests (`@smoke`)
|
|
21
|
+
|
|
22
|
+
**Purpose**: Quick validation of critical user journeys
|
|
23
|
+
|
|
24
|
+
**Coverage**:
|
|
25
|
+
- Comprehensive E2E tests only
|
|
26
|
+
- One test per test case
|
|
27
|
+
- 100% manual test case coverage
|
|
28
|
+
- **Zero redundancy**
|
|
29
|
+
|
|
30
|
+
**Execution Time**: ~2-5 minutes for full smoke suite
|
|
31
|
+
|
|
32
|
+
**When to Use**:
|
|
33
|
+
- **Default for most test runs**
|
|
34
|
+
- Every commit and pull request
|
|
35
|
+
- Quick validation after code changes
|
|
36
|
+
- Before starting new development work
|
|
37
|
+
- Continuous integration (CI) on every push
|
|
38
|
+
- Post-deployment verification
|
|
39
|
+
|
|
40
|
+
**Command**:
|
|
41
|
+
```bash
|
|
42
|
+
npx playwright test --grep "@smoke"
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
**Why This Tier**:
|
|
46
|
+
- Fast feedback loop
|
|
47
|
+
- Full coverage of critical paths
|
|
48
|
+
- No wasted execution time on redundant tests
|
|
49
|
+
- Ideal for frequent validation
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
### Tier 2: Component Tests (`@component` or `@fast`)
|
|
54
|
+
|
|
55
|
+
**Purpose**: Isolated component validation for debugging
|
|
56
|
+
|
|
57
|
+
**Coverage**:
|
|
58
|
+
- Granular tests for specific capabilities only
|
|
59
|
+
- Navigation, form display, API calls, etc.
|
|
60
|
+
- Subset coverage (e.g., just steps 1-8 for navigation)
|
|
61
|
+
- Skips comprehensive E2E tests
|
|
62
|
+
|
|
63
|
+
**Execution Time**: ~1-3 minutes (faster than smoke)
|
|
64
|
+
|
|
65
|
+
**When to Use**:
|
|
66
|
+
- Debugging specific component failures
|
|
67
|
+
- After refactoring a specific component
|
|
68
|
+
- When you need to verify a single capability without running full E2E
|
|
69
|
+
- Rapid iteration during development
|
|
70
|
+
- Targeted validation after component-level bug fixes
|
|
71
|
+
|
|
72
|
+
**Command**:
|
|
73
|
+
```bash
|
|
74
|
+
npx playwright test --grep "@component"
|
|
75
|
+
# OR
|
|
76
|
+
npx playwright test --grep "@fast"
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
**Why This Tier**:
|
|
80
|
+
- Ultra-fast feedback for specific areas
|
|
81
|
+
- Avoids running unnecessary setup/teardown
|
|
82
|
+
- Ideal for focused development work
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
### Tier 3: Full Regression (All Tests)
|
|
87
|
+
|
|
88
|
+
**Purpose**: Complete validation including redundant component tests
|
|
89
|
+
|
|
90
|
+
**Coverage**:
|
|
91
|
+
- Smoke tests (comprehensive E2E)
|
|
92
|
+
- Component tests (granular isolation)
|
|
93
|
+
- **Intentional redundancy** for diagnostic value
|
|
94
|
+
|
|
95
|
+
**Execution Time**: ~10-15 minutes
|
|
96
|
+
|
|
97
|
+
**When to Use**:
|
|
98
|
+
- Weekly scheduled CI runs
|
|
99
|
+
- Before major releases or deployments
|
|
100
|
+
- After significant refactoring across multiple areas
|
|
101
|
+
- When comprehensive validation is required
|
|
102
|
+
- Pre-merge validation for large features
|
|
103
|
+
- Monthly regression testing cycles
|
|
104
|
+
|
|
105
|
+
**Command**:
|
|
106
|
+
```bash
|
|
107
|
+
npx playwright test
|
|
108
|
+
# OR explicitly
|
|
109
|
+
npx playwright test --grep ".*"
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
**Why This Tier**:
|
|
113
|
+
- **Fast failure isolation**: If Settings page breaks, component test fails in 15 seconds vs E2E failing after 30 seconds
|
|
114
|
+
- **Component-level confidence**: Proves each capability works independently
|
|
115
|
+
- **Refactoring safety**: If you refactor save logic, component tests prove navigation/display still work
|
|
116
|
+
- **Comprehensive validation**: Catches edge cases that isolated tests might miss
|
|
117
|
+
|
|
118
|
+
**Understanding Redundancy**:
|
|
119
|
+
- Example: TC-003 login steps (1-5) run in all 3 tests
|
|
120
|
+
- **Cost**: Extra 30 seconds execution time
|
|
121
|
+
- **Benefit**: Instant failure location (know exactly which component broke)
|
|
122
|
+
- **Trade-off**: Worth it for regression scenarios, not for daily validation
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Default Behavior
|
|
127
|
+
|
|
128
|
+
### When User Provides No Selector
|
|
129
|
+
|
|
130
|
+
**Default Action**: Run `@smoke` tests
|
|
131
|
+
|
|
132
|
+
**Rationale**:
|
|
133
|
+
- Provides fast feedback (2-5 min vs 10-15 min)
|
|
134
|
+
- Full coverage of critical user journeys
|
|
135
|
+
- Zero redundancy = no wasted time
|
|
136
|
+
- Suitable for 90% of test execution scenarios
|
|
137
|
+
|
|
138
|
+
**User Communication**:
|
|
139
|
+
```
|
|
140
|
+
Running smoke tests (@smoke) for quick validation.
|
|
141
|
+
For comprehensive regression including component tests, use 'all'.
|
|
142
|
+
For specific areas, provide test file pattern or tag.
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
### When User Says "All"
|
|
146
|
+
|
|
147
|
+
**Action**: Run full regression suite (all tests)
|
|
148
|
+
|
|
149
|
+
**User Communication**:
|
|
150
|
+
```
|
|
151
|
+
Running full regression suite including component tests.
|
|
152
|
+
This includes intentional redundancy for diagnostic value and will take longer (~10-15 min).
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## Context-Based Selection Logic
|
|
158
|
+
|
|
159
|
+
Parse user intent from their natural language request:
|
|
160
|
+
|
|
161
|
+
### Fast Validation Context
|
|
162
|
+
|
|
163
|
+
**Keywords**: "quick", "validate", "sanity check", "verify", "check"
|
|
164
|
+
|
|
165
|
+
**Action**: Run `@smoke` tests
|
|
166
|
+
|
|
167
|
+
**Examples**:
|
|
168
|
+
- "Quick check if login still works"
|
|
169
|
+
- "Validate the settings page"
|
|
170
|
+
- "Sanity test before I start work"
|
|
171
|
+
|
|
172
|
+
---
|
|
173
|
+
|
|
174
|
+
### Comprehensive Validation Context
|
|
175
|
+
|
|
176
|
+
**Keywords**: "full", "comprehensive", "regression", "all", "everything", "complete"
|
|
177
|
+
|
|
178
|
+
**Action**: Run all tests
|
|
179
|
+
|
|
180
|
+
**Examples**:
|
|
181
|
+
- "Full regression before release"
|
|
182
|
+
- "Comprehensive validation of the system"
|
|
183
|
+
- "Run everything to be sure"
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
### Debugging/Focused Context
|
|
188
|
+
|
|
189
|
+
**Keywords**: "debug", "check [specific area]", "[component name]", "just [feature]"
|
|
190
|
+
|
|
191
|
+
**Action**: Run specific test file or pattern
|
|
192
|
+
|
|
193
|
+
**Examples**:
|
|
194
|
+
- "Debug the settings page" → Run `TC-003` or pattern `settings`
|
|
195
|
+
- "Check authentication only" → Run pattern `auth` or `TC-001`
|
|
196
|
+
- "Just test the navigation" → Run `@component` tests
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
### Ambiguous Context
|
|
201
|
+
|
|
202
|
+
**When**: No clear keywords or selector provided
|
|
203
|
+
|
|
204
|
+
**Action**: Default to `@smoke` and inform user of options
|
|
205
|
+
|
|
206
|
+
**User Communication**:
|
|
207
|
+
```
|
|
208
|
+
No test selection specified. Running smoke tests (@smoke) by default.
|
|
209
|
+
|
|
210
|
+
Available options:
|
|
211
|
+
- '@smoke' - Quick validation (2-5 min, default)
|
|
212
|
+
- '@component' - Component tests only (1-3 min)
|
|
213
|
+
- 'all' - Full regression (10-15 min)
|
|
214
|
+
- '[pattern]' - Specific tests matching pattern
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
---
|
|
218
|
+
|
|
219
|
+
## Tag Taxonomy
|
|
220
|
+
|
|
221
|
+
### Primary Tags
|
|
222
|
+
|
|
223
|
+
| Tag | Purpose | Coverage | Speed |
|
|
224
|
+
|-----|---------|----------|-------|
|
|
225
|
+
| `@smoke` | Critical E2E journeys | Comprehensive, 100% manual test coverage | Medium (30-60s/test) |
|
|
226
|
+
| `@component` | Isolated component validation | Partial, specific capability only | Fast (10-20s/test) |
|
|
227
|
+
| `@fast` | Alias for `@component` | Same as @component | Fast |
|
|
228
|
+
| `@regression` | Full regression suite | All tests (smoke + component) | Slow (full suite) |
|
|
229
|
+
|
|
230
|
+
### Domain Tags (for filtering by feature area)
|
|
231
|
+
|
|
232
|
+
| Tag | Feature Area |
|
|
233
|
+
|-----|--------------|
|
|
234
|
+
| `@authentication` | Login, logout, session management |
|
|
235
|
+
| `@project-management` | Project CRUD, settings |
|
|
236
|
+
| `@test-management` | Test cases, test runs, activity feed |
|
|
237
|
+
| `@settings` | Settings pages and persistence |
|
|
238
|
+
| `@navigation` | Navigation and routing |
|
|
239
|
+
| `@data-persistence` | Data saving and loading |
|
|
240
|
+
|
|
241
|
+
### Combining Tags
|
|
242
|
+
|
|
243
|
+
**Multiple tags on same test**:
|
|
244
|
+
```typescript
|
|
245
|
+
test('should update settings and persist @smoke @project-management @settings', ...)
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
**Run tests matching multiple criteria**:
|
|
249
|
+
```bash
|
|
250
|
+
# Smoke tests in project management area
|
|
251
|
+
npx playwright test --grep "@smoke.*@project-management"
|
|
252
|
+
|
|
253
|
+
# All settings-related tests (smoke + component)
|
|
254
|
+
npx playwright test --grep "@settings"
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
---
|
|
258
|
+
|
|
259
|
+
## Time and Coverage Trade-offs
|
|
260
|
+
|
|
261
|
+
### Execution Time Comparison
|
|
262
|
+
|
|
263
|
+
| Tier | Tests Run | Time | Use Frequency |
|
|
264
|
+
|------|-----------|------|---------------|
|
|
265
|
+
| Smoke (`@smoke`) | ~6 tests | 2-5 min | Daily (90%) |
|
|
266
|
+
| Component (`@component`) | ~8 tests | 1-3 min | As needed (5%) |
|
|
267
|
+
| Full Regression (all) | ~15 tests | 10-15 min | Weekly (5%) |
|
|
268
|
+
|
|
269
|
+
### Coverage vs Speed Matrix
|
|
270
|
+
|
|
271
|
+
```
|
|
272
|
+
High Coverage ┐
|
|
273
|
+
│
|
|
274
|
+
│ ┌─ Full Regression
|
|
275
|
+
│ │ (High coverage, slower)
|
|
276
|
+
│ │
|
|
277
|
+
│ ┌────┘
|
|
278
|
+
│ │
|
|
279
|
+
│ │ Smoke Tests
|
|
280
|
+
│ │ (High coverage, faster)
|
|
281
|
+
│ │
|
|
282
|
+
├───┼─────────────────────►
|
|
283
|
+
│ │ Fast
|
|
284
|
+
│ │
|
|
285
|
+
│ └─ Component Tests
|
|
286
|
+
│ (Partial coverage, fastest)
|
|
287
|
+
│
|
|
288
|
+
Low Coverage ┘
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
### Recommendation by Scenario
|
|
292
|
+
|
|
293
|
+
| Scenario | Recommended Tier | Rationale |
|
|
294
|
+
|----------|------------------|-----------|
|
|
295
|
+
| Developer commit | Smoke | Fast feedback, full coverage |
|
|
296
|
+
| Pull request CI | Smoke | Balance speed and confidence |
|
|
297
|
+
| Pre-deployment | Full Regression | Comprehensive validation |
|
|
298
|
+
| Debugging component | Component or specific file | Fastest feedback for focused area |
|
|
299
|
+
| Weekly CI schedule | Full Regression | Diagnostic redundancy value |
|
|
300
|
+
| Post-hotfix | Smoke | Quick validation of critical paths |
|
|
301
|
+
|
|
302
|
+
---
|
|
303
|
+
|
|
304
|
+
## Example Scenarios
|
|
305
|
+
|
|
306
|
+
### Scenario 1: Developer Makes Code Change
|
|
307
|
+
|
|
308
|
+
**Context**: Developer updates Settings page form validation
|
|
309
|
+
|
|
310
|
+
**User Request**: `/run-tests` (no selector)
|
|
311
|
+
|
|
312
|
+
**Agent Decision**:
|
|
313
|
+
- Read this strategy file
|
|
314
|
+
- No selector provided → Default to `@smoke`
|
|
315
|
+
- Run: `npx playwright test --grep "@smoke"`
|
|
316
|
+
- Result: 6 tests in ~3 min, full E2E coverage, zero redundancy
|
|
317
|
+
|
|
318
|
+
**Communication**:
|
|
319
|
+
```
|
|
320
|
+
Running smoke tests (@smoke) for quick validation...
|
|
321
|
+
✓ 6/6 tests passed in 3m 12s
|
|
322
|
+
All critical user journeys validated successfully.
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
### Scenario 2: Pre-Release Validation
|
|
328
|
+
|
|
329
|
+
**User Request**: `/run-tests all`
|
|
330
|
+
|
|
331
|
+
**Agent Decision**:
|
|
332
|
+
- Explicit "all" keyword → Run full regression
|
|
333
|
+
- Run: `npx playwright test`
|
|
334
|
+
- Result: 15 tests in ~12 min, includes redundant component tests
|
|
335
|
+
|
|
336
|
+
**Communication**:
|
|
337
|
+
```
|
|
338
|
+
Running full regression suite including component tests...
|
|
339
|
+
This includes intentional redundancy for diagnostic value (~10-15 min).
|
|
340
|
+
|
|
341
|
+
✓ 15/15 tests passed in 12m 34s
|
|
342
|
+
Full regression validation complete:
|
|
343
|
+
- 6 comprehensive E2E tests (smoke)
|
|
344
|
+
- 9 component-level tests
|
|
345
|
+
All tests passed.
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
---
|
|
349
|
+
|
|
350
|
+
### Scenario 3: Debugging Settings Page Failure
|
|
351
|
+
|
|
352
|
+
**User Request**: `/run-tests settings`
|
|
353
|
+
|
|
354
|
+
**Agent Decision**:
|
|
355
|
+
- File pattern provided → Match specific tests
|
|
356
|
+
- Run: `npx playwright test settings`
|
|
357
|
+
- Result: Runs TC-003 (3 tests: 1 smoke + 2 component)
|
|
358
|
+
|
|
359
|
+
**Communication**:
|
|
360
|
+
```
|
|
361
|
+
Running tests matching pattern "settings"...
|
|
362
|
+
Found: tests/specs/project-management/TC-003-project-settings-update.spec.ts
|
|
363
|
+
|
|
364
|
+
Results:
|
|
365
|
+
✓ should update project settings and persist changes @smoke
|
|
366
|
+
✗ should navigate to settings page @component
|
|
367
|
+
└─ Error: Navigation timeout
|
|
368
|
+
✓ should display project name and description @component
|
|
369
|
+
|
|
370
|
+
Failure isolated to navigation component. Settings form display works correctly.
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
---
|
|
374
|
+
|
|
375
|
+
### Scenario 4: Quick Validation Request
|
|
376
|
+
|
|
377
|
+
**User Request**: "Can you quickly check if the login flow still works?"
|
|
378
|
+
|
|
379
|
+
**Agent Decision**:
|
|
380
|
+
- Keyword "quickly" → Use `@smoke`
|
|
381
|
+
- Implicit focus on "login" → Could run specific test or full smoke
|
|
382
|
+
- Run: `npx playwright test --grep "@smoke"` (include all smoke, not just login)
|
|
383
|
+
- Result: 6 tests including login in ~3 min
|
|
384
|
+
|
|
385
|
+
**Communication**:
|
|
386
|
+
```
|
|
387
|
+
Running smoke tests (@smoke) for quick validation of all critical flows including login...
|
|
388
|
+
|
|
389
|
+
✓ 6/6 tests passed in 3m 8s
|
|
390
|
+
All critical user journeys validated, including:
|
|
391
|
+
- TC-001: Login and navigation ✓
|
|
392
|
+
- TC-003: Project settings update ✓
|
|
393
|
+
- TC-004: Test cases page access ✓
|
|
394
|
+
- TC-005: Test runs page access ✓
|
|
395
|
+
- TC-006: Session management ✓
|
|
396
|
+
- TC-008: Activity feed access ✓
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
---
|
|
400
|
+
|
|
401
|
+
## Understanding Test Redundancy
|
|
402
|
+
|
|
403
|
+
### What is Redundant Coverage?
|
|
404
|
+
|
|
405
|
+
Some test suites contain multiple tests that exercise the same functionality:
|
|
406
|
+
|
|
407
|
+
**Example: TC-003 (Project Settings Update)**
|
|
408
|
+
|
|
409
|
+
```
|
|
410
|
+
Test 1 (E2E, @smoke): Steps 1-20 (login → navigate → update → verify)
|
|
411
|
+
Test 2 (Component, @component): Steps 1-8 (login → navigate) ← Overlaps with Test 1
|
|
412
|
+
Test 3 (Component, @component): Steps 1-10 (login → navigate → display) ← Overlaps with Test 1
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
**Overlap**: Steps 1-8 run in all 3 tests
|
|
416
|
+
|
|
417
|
+
### Why Allow Redundancy?
|
|
418
|
+
|
|
419
|
+
**Diagnostic Value**:
|
|
420
|
+
- If Test 2 fails but Test 1 passes → Issue is isolated to navigation
|
|
421
|
+
- If Test 3 fails but Tests 1 & 2 pass → Issue is in form display
|
|
422
|
+
- If Test 1 fails but Tests 2 & 3 pass → Issue is in update/save logic (steps 11-20)
|
|
423
|
+
|
|
424
|
+
**Speed Benefit**:
|
|
425
|
+
- Test 2 fails in 15 seconds (just navigation)
|
|
426
|
+
- Test 1 fails in 30 seconds (full E2E including navigation)
|
|
427
|
+
- **Saved time**: 15 seconds to identify failure point
|
|
428
|
+
|
|
429
|
+
### When is Redundancy Worth It?
|
|
430
|
+
|
|
431
|
+
**Smoke Runs**: NOT worth it (90% of executions)
|
|
432
|
+
- Goal: Fast validation
|
|
433
|
+
- Trade-off: Skip redundant tests, accept slower failure diagnosis
|
|
434
|
+
|
|
435
|
+
**Full Regression Runs**: Worth it (10% of executions)
|
|
436
|
+
- Goal: Comprehensive validation with diagnostic value
|
|
437
|
+
- Trade-off: Accept 30 extra seconds for faster failure isolation
|
|
438
|
+
- Context: Pre-release, weekly CI, post-refactoring
|
|
439
|
+
|
|
440
|
+
---
|
|
441
|
+
|
|
442
|
+
## Integration with Other Systems
|
|
443
|
+
|
|
444
|
+
### Test Generation (`/generate-test-cases`)
|
|
445
|
+
|
|
446
|
+
When generating new test cases, the generator should:
|
|
447
|
+
1. Create comprehensive E2E test with `@smoke` tag
|
|
448
|
+
2. Optionally create component tests with `@component` or `@fast` tags
|
|
449
|
+
3. Apply appropriate domain tags (`@authentication`, `@project-management`, etc.)
|
|
450
|
+
|
|
451
|
+
### CI/CD Pipelines
|
|
452
|
+
|
|
453
|
+
Recommended pipeline configuration:
|
|
454
|
+
|
|
455
|
+
```yaml
|
|
456
|
+
# On every PR - SMOKE ONLY
|
|
457
|
+
on_pull_request:
|
|
458
|
+
npx playwright test --grep "@smoke"
|
|
459
|
+
# Fast validation: 2-5 min
|
|
460
|
+
|
|
461
|
+
# Nightly regression - ALL TESTS
|
|
462
|
+
on_schedule:
|
|
463
|
+
npx playwright test
|
|
464
|
+
# Comprehensive validation: 10-15 min
|
|
465
|
+
|
|
466
|
+
# Pre-deployment - SMOKE (fast gate)
|
|
467
|
+
on_deployment:
|
|
468
|
+
npx playwright test --grep "@smoke"
|
|
469
|
+
# Critical path validation: 2-5 min
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
### Issue Tracking
|
|
473
|
+
|
|
474
|
+
When tests fail, triage results should consider:
|
|
475
|
+
- Smoke test failures → High priority (critical path broken)
|
|
476
|
+
- Component test failures (when smoke passes) → Medium priority (isolated issue)
|
|
477
|
+
- Component test failures (when smoke also fails) → Confirms root cause
|
|
478
|
+
|
|
479
|
+
---
|
|
480
|
+
|
|
481
|
+
## Decision Tree for Agents
|
|
482
|
+
|
|
483
|
+
```
|
|
484
|
+
User provides selector?
|
|
485
|
+
├─ Yes
|
|
486
|
+
│ ├─ Selector is "all" or "everything"?
|
|
487
|
+
│ │ └─ Run: npx playwright test (Full Regression)
|
|
488
|
+
│ ├─ Selector is tag (starts with @)?
|
|
489
|
+
│ │ └─ Run: npx playwright test --grep "[tag]"
|
|
490
|
+
│ ├─ Selector is file pattern (e.g., "auth", "settings")?
|
|
491
|
+
│ │ └─ Run: npx playwright test [pattern]
|
|
492
|
+
│ └─ Selector is specific file path?
|
|
493
|
+
│ └─ Run: npx playwright test [file-path]
|
|
494
|
+
└─ No selector provided
|
|
495
|
+
├─ User request contains context keywords?
|
|
496
|
+
│ ├─ "quick", "validate", "sanity" → Run: @smoke
|
|
497
|
+
│ ├─ "full", "comprehensive", "regression" → Run: all tests
|
|
498
|
+
│ └─ "debug [area]", "check [component]" → Run: specific pattern
|
|
499
|
+
└─ No context keywords
|
|
500
|
+
└─ Default: Run @smoke and inform user of options
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
---
|
|
504
|
+
|
|
505
|
+
## Maintenance Notes
|
|
506
|
+
|
|
507
|
+
### When to Update This Strategy
|
|
508
|
+
|
|
509
|
+
This document should be updated when:
|
|
510
|
+
- New test tiers are introduced
|
|
511
|
+
- Tag taxonomy changes
|
|
512
|
+
- Execution time significantly changes
|
|
513
|
+
- New testing patterns emerge
|
|
514
|
+
- CI/CD pipeline requirements change
|
|
515
|
+
|
|
516
|
+
### Related Documents
|
|
517
|
+
|
|
518
|
+
- `.bugzy/runtime/testing-best-practices.md` - How to write tests (test patterns)
|
|
519
|
+
- `.bugzy/runtime/knowledge-base.md` - Accumulated testing insights
|
|
520
|
+
- `.claude/commands/run-tests.md` - Test execution command (mechanics)
|
|
521
|
+
- `playwright.config.ts` - Playwright configuration
|
|
522
|
+
- Test cases in `./test-cases/` - Individual test case documentation
|
|
523
|
+
|
|
524
|
+
---
|
|
525
|
+
|
|
526
|
+
## Summary
|
|
527
|
+
|
|
528
|
+
**Key Principles**:
|
|
529
|
+
1. **Default to smoke tests** for fast validation (90% of runs)
|
|
530
|
+
2. **Use component tests** for focused debugging
|
|
531
|
+
3. **Run full regression** only when diagnostic redundancy is needed (10% of runs)
|
|
532
|
+
4. **Parse user context** to choose the right tier intelligently
|
|
533
|
+
5. **Communicate clearly** about what's running and why
|
|
534
|
+
|
|
535
|
+
**Remember**: The goal is to provide fast feedback most of the time, while maintaining the ability to run comprehensive validation when needed. Redundancy is intentional but conditional.
|