bmad-method 6.0.0-alpha.17 → 6.0.0-alpha.18

package/src/modules/bmgd/gametest/knowledge/qa-automation.md

@@ -0,0 +1,190 @@
+# QA Automation for Games
+
+## Overview
+
+Automated testing in games requires different approaches than traditional software. Games have complex state, real-time interactions, and subjective quality measures that challenge automation.
+
+## Testing Pyramid for Games
+
+```
+        /\
+       /  \     Manual Playtesting
+      /----\    (Experience, Feel, Fun)
+     /      \
+    /--------\  Integration Tests
+   /          \ (Systems, Workflows)
+  /------------\
+ /              \ Unit Tests
+/________________\ (Pure Logic, Math, Data)
+```
+
+### Unit Tests (Foundation)
+
+Test pure logic that doesn't depend on engine runtime:
+
+- Math utilities (vectors, transforms, curves)
+- Data validation (save files, configs)
+- State machines (isolated logic)
+- Algorithm correctness
+
+### Integration Tests (Middle Layer)
+
+Test system interactions:
+
+- Combat system + inventory
+- Save/load round-trips
+- Scene transitions
+- Network message handling
+
+### Manual Testing (Top)
+
+What can't be automated:
+
+- "Does this feel good?"
+- "Is this fun?"
+- "Is the difficulty right?"
+
+## Automation Strategies by Engine
+
+### Unity
+
+```csharp
+// Unity Test Framework
+[Test]
+public void DamageCalculation_CriticalHit_DoublesDamage()
+{
+    var baseDamage = 100;
+    var result = DamageCalculator.Calculate(baseDamage, isCritical: true);
+    Assert.AreEqual(200, result);
+}
+
+// Play Mode Tests (runtime)
+[UnityTest]
+public IEnumerator PlayerJump_WhenGrounded_BecomesAirborne()
+{
+    var player = CreateTestPlayer();
+    player.Jump();
+    yield return new WaitForFixedUpdate();
+    Assert.IsFalse(player.IsGrounded);
+}
+```
+
+### Unreal Engine
+
+```cpp
+// Automation Framework
+IMPLEMENT_SIMPLE_AUTOMATION_TEST(FDamageTest, "Game.Combat.Damage",
+    EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter)
+
+bool FDamageTest::RunTest(const FString& Parameters)
+{
+    float BaseDamage = 100.f;
+    float Result = UDamageCalculator::Calculate(BaseDamage, true);
+    TestEqual("Critical hit doubles damage", Result, 200.f);
+    return true;
+}
+```
+
+### Godot
+
+```gdscript
+# GUT Testing Framework
+func test_damage_critical_hit():
+    var base_damage = 100
+    var result = DamageCalculator.calculate(base_damage, true)
+    assert_eq(result, 200, "Critical hit should double damage")
+```
+
+## What to Automate
+
+### High Value Targets
+
+- **Save/Load** - Data integrity is critical
+- **Economy** - Currency, items, progression math
+- **Combat Math** - Damage, stats, modifiers
+- **Localization** - String loading, formatting
+- **Network Serialization** - Message encoding/decoding
+
+### Medium Value Targets
+
+- **State Machines** - Character states, game states
+- **Pathfinding** - Known scenarios
+- **Spawning** - Wave generation, loot tables
+- **UI Data Binding** - Correct values displayed
+
+### Low Value / Avoid
+
+- **Visual Quality** - Screenshots drift, hard to maintain
+- **Input Feel** - Timing-sensitive, needs human judgment
+- **Audio** - Subjective, context-dependent
+- **Fun** - Cannot be automated
+
+## Continuous Integration for Games
+
+### Build Pipeline
+
+1. **Compile** - Build game executable
+2. **Unit Tests** - Fast, isolated tests
+3. **Integration Tests** - Longer, system tests
+4. **Smoke Test** - Can the game launch and reach main menu?
+5. **Nightly** - Extended test suites, performance benchmarks
+
+### CI Gotchas for Games
+
+- **Long build times** - Games take longer than web apps
+- **GPU requirements** - Some tests need graphics hardware
+- **Asset dependencies** - Large files, binary formats
+- **Platform builds** - Multiple targets to maintain
+
+## Regression Testing
+
+### Automated Regression
+
+- Run full test suite on every commit
+- Flag performance regressions (frame time, memory)
+- Track test stability (flaky tests)
+
+### Save File Regression
+
+- Maintain library of save files from previous versions
+- Test that new builds can load old saves
+- Alert on schema changes
+
+## Test Data Management
+
+### Test Fixtures
+
+```
+tests/
+├── fixtures/
+│   ├── save_files/
+│   │   ├── new_game.sav
+│   │   ├── mid_game.sav
+│   │   └── endgame.sav
+│   ├── configs/
+│   │   └── test_balance.json
+│   └── scenarios/
+│       └── boss_fight_setup.scene
+```
+
+### Deterministic Testing
+
+- Seed random number generators
+- Control time/delta time
+- Mock external services
+
+## Metrics and Reporting
+
+### Track Over Time
+
+- Test count (growing is good)
+- Pass rate (should be ~100%)
+- Execution time (catch slow tests)
+- Code coverage (where applicable)
+- Flaky test rate (should be ~0%)
+
+### Alerts
+
+- Immediate: Any test failure on main branch
+- Daily: Coverage drops, new flaky tests
+- Weekly: Trend analysis, slow test growth

package/src/modules/bmgd/gametest/knowledge/regression-testing.md

@@ -0,0 +1,280 @@
+# Regression Testing for Games
+
+## Overview
+
+Regression testing catches bugs introduced by new changes. In games, this includes functional regressions, performance regressions, and design regressions.
+
+## Types of Regression
+
+### Functional Regression
+
+- Features that worked before now break
+- New bugs introduced by unrelated changes
+- Broken integrations between systems
+
+### Performance Regression
+
+- Frame rate drops
+- Memory usage increases
+- Load time increases
+- Battery drain (mobile)
+
+### Design Regression
+
+- Balance changes with unintended side effects
+- UX changes that hurt usability
+- Art changes that break visual consistency
+
+### Save Data Regression
+
+- Old save files no longer load
+- Progression lost or corrupted
+- Achievements/unlocks reset
+
+## Regression Testing Strategy
+
+### Test Suite Layers
+
+```
+High-Frequency (Every Commit)
+├── Unit Tests - Fast, isolated
+├── Smoke Tests - Can game launch and run?
+└── Critical Path - Core gameplay works
+
+Medium-Frequency (Nightly)
+├── Integration Tests - System interactions
+├── Full Playthrough - Automated or manual
+└── Performance Benchmarks - Frame time, memory
+
+Low-Frequency (Release)
+├── Full Matrix - All platforms/configs
+├── Certification Tests - Platform requirements
+└── Localization - All languages
+```
+
+### What to Test
+
+#### Critical Path (Must Not Break)
+
+- Game launches
+- New game starts
+- Save/load works
+- Core gameplay loop completes
+- Main menu navigation
+
+#### High Priority
+
+- All game systems function
+- Progression works end-to-end
+- Multiplayer connects and syncs
+- In-app purchases process
+- Achievements trigger
+
+#### Medium Priority
+
+- Edge cases in systems
+- Optional content accessible
+- Settings persist correctly
+- Localization displays
+
+## Automated Regression Tests
+
+### Smoke Tests
+
+```python
+# Run on every commit
+def test_game_launches():
+    process = launch_game()
+    assert wait_for_main_menu(timeout=30)
+    process.terminate()
+
+def test_new_game_starts():
+    launch_game()
+    click_new_game()
+    assert wait_for_gameplay(timeout=60)
+
+def test_save_load_roundtrip():
+    launch_game()
+    start_new_game()
+    perform_actions()
+    save_game()
+    load_game()
+    assert verify_state_matches()
+```
+
+### Playthrough Bots
+
+```python
+# Automated player that plays through content
+class PlaythroughBot:
+    def run_level(self, level):
+        self.load_level(level)
+        while not self.level_complete:
+            self.perform_action()
+            self.check_for_softlocks()
+            self.record_metrics()
+```
+
+### Visual Regression
+
+```python
+# Compare screenshots against baselines
+def test_main_menu_visual():
+    launch_game()
+    screenshot = capture_screen()
+    assert compare_to_baseline(screenshot, 'main_menu', threshold=0.01)
+```
+
+## Performance Regression Detection
+
+### Metrics to Track
+
+- Average frame time
+- 1% low frame time
+- Memory usage (peak, average)
+- Load times
+- Draw calls
+- Texture memory
+
+### Automated Benchmarks
+
+```yaml
+performance_benchmark:
+  script:
+    - run_benchmark_scene --duration 60s
+    - collect_metrics
+    - compare_to_baseline
+  fail_conditions:
+    - frame_time_avg > baseline * 1.1 # 10% tolerance
+    - memory_peak > baseline * 1.05 # 5% tolerance
+```
+
+### Trend Tracking
+
+- Graph metrics over time
+- Alert on upward trends
+- Identify problematic commits
+
+## Save Compatibility Testing
+
+### Version Matrix
+
+Maintain save files from:
+
+- Previous major version
+- Previous minor version
+- Current development build
+
+### Automated Validation
+
+```python
+def test_save_compatibility():
+    for save_file in LEGACY_SAVES:
+        load_save(save_file)
+        assert no_errors()
+        assert progress_preserved()
+        assert inventory_intact()
+```
+
+### Schema Versioning
+
+- Version your save format
+- Implement upgrade paths
+- Log migration issues
+
+## Regression Bug Workflow
+
+### 1. Detection
+
+- Automated test fails
+- Manual tester finds issue
+- Player report comes in
+
+### 2. Verification
+
+- Confirm it worked before
+- Identify when it broke
+- Find the breaking commit
+
+### 3. Triage
+
+- Assess severity
+- Determine fix urgency
+- Assign to appropriate developer
+
+### 4. Fix and Verify
+
+- Implement fix
+- Add regression test
+- Verify fix doesn't break other things
+
+### 5. Post-Mortem
+
+- Why wasn't this caught?
+- How can we prevent similar issues?
+- Do we need new tests?
+
+## Bisecting Regressions
+
+When a regression is found, identify the breaking commit:
+
+### Git Bisect
+
+```bash
+git bisect start
+git bisect bad HEAD          # Current is broken
+git bisect good v1.2.0       # Known good version
+# Git will checkout commits to test
+# Run test, mark good/bad
+git bisect good/bad
+# Repeat until culprit found
+```
+
+### Automated Bisect
+
+```bash
+git bisect start HEAD v1.2.0
+git bisect run ./run_regression_test.sh
+```
+
+## Regression Testing Checklist
+
+### Per Commit
+
+- [ ] Unit tests pass
+- [ ] Smoke tests pass
+- [ ] Build succeeds on all platforms
+
+### Per Merge to Main
+
+- [ ] Integration tests pass
+- [ ] Performance benchmarks within tolerance
+- [ ] Save compatibility verified
+
+### Per Release
+
+- [ ] Full playthrough completed
+- [ ] All platforms tested
+- [ ] Legacy saves load correctly
+- [ ] No new critical regressions
+- [ ] All previous hotfix issues still resolved
+
+## Building a Regression Suite
+
+### Start Small
+
+1. Add tests for bugs as they're fixed
+2. Cover critical path first
+3. Expand coverage over time
+
+### Maintain Quality
+
+- Delete flaky tests
+- Keep tests fast
+- Update tests with design changes
+
+### Measure Effectiveness
+
+- Track bugs caught by tests
+- Track bugs that slipped through
+- Identify coverage gaps