qa360 2.1.2 → 2.1.4

package/docs/LIMITATIONS.md

@@ -0,0 +1,176 @@
+# QA360 Generation Module - Limitations
+
+**Version**: 1.4.5
+**Date**: 2026-01-01
+**Module**: `core/src/generation`
+
+---
+
+## Purpose
+
+This document explicitly lists what the Generation module **does NOT guarantee**.
+
+Users must review all generated code before deployment.
+
+---
+
+## What is Generated
+
+The Generation module produces **draft test code** based on:
+- OpenAPI/Swagger specifications
+- HAR files (HTTP recordings)
+- URLs (via scraping simulation)
+- Natural language descriptions
+- Source code (for unit tests)
+
+---
+
+## What is NOT Guaranteed
+
+### 1. Code Correctness
+
+**NOT guaranteed**:
+- Generated code will execute without errors
+- API endpoints exist or match the specification
+- Selectors (CSS/XPath) are valid or stable
+- Assertions are logically correct
+- Edge cases are covered
+
+**Example of failure**:
+```typescript
+// Generated code that may fail
+test('user login', async ({ page }) => {
+  await page.click('#login-button');  // Selector may not exist
+  await page.fill('#email', 'test@example.com');  // ID may differ
+  // No actual verification login succeeded
+});
+```
+
+### 2. Test Completeness
+
+**NOT guaranteed**:
+- All user flows are covered
+- Error scenarios are tested
+- Boundary conditions are handled
+- Security implications are considered
+
+### 3. Production Readiness
+
+**NOT guaranteed**:
+- Generated tests are ready for production CI/CD
+- Tests will pass in your environment
+- Authentication/authorization is correctly handled
+- Test data is valid or safe
+
+### 4. Idempotency
+
+**NOT guaranteed**:
+- Running generation twice produces identical output
+- Generated tests can be run multiple times safely
+- Side effects (if any) are handled
+
+---
+
+## Required Human Review
+
+Before committing generated tests, reviewers MUST verify:
+
+1. **Syntax validity**: Code parses without errors
+2. **Selector accuracy**: CSS/XPath selectors match actual DOM
+3. **Assertion correctness**: Tests verify meaningful behavior
+4. **Error handling**: Appropriate error checks exist
+5. **Data safety**: No hardcoded credentials or PII
+6. **Environment compatibility**: Tests work in your infrastructure
+
+---
+
+## LLM-Specific Limitations
+
+### Non-Determinism
+
+The LLM (Ollama) may produce different outputs for the same input due to:
+- Model version changes
+- Temperature settings
+- Random seed variations
+- Context window limitations
+
+### Timeout Behavior
+
+- Default timeout: 30 seconds
+- Configurable via `timeout` option (100ms to 10 minutes)
+- Timeout returns structured error, does not crash
+- Partial results are discarded on timeout
+
+### Resource Requirements
+
+- LLM must be installed and running (`ollama serve`)
+- Minimum 4GB RAM recommended for `deepseek-coder` model
+- Network latency affects generation time
+- Large responses (>1MB) may be truncated
+
+---
+
+## Security Considerations
+
+### NOT Protected Against
+
+- Prompt injection attacks
+- Malicious HAR files
+- Code injection via generated tests
+- Exposure of secrets in prompts
+
+### Best Practices
+
+1. Review HAR files before generation
+2. Sanitize sensitive data from specifications
+3. Run generated tests in isolated environment first
+4. Never commit generated tests without review
+
+---
+
+## Comparison to Manual Testing
+
+| Aspect | Manual Testing | QA360 Generation |
+|--------|---------------|-------------------|
+| Understanding of context | Human learns | LLM guesses |
+| Error handling | Intentional | May be missing |
+| Edge cases | Experienced tester considers | Often overlooked |
+| Maintenance | Developer owns | Requires human update |
+| Speed | Slow | Fast (draft in seconds) |
+
+**Conclusion**: QA360 accelerates test creation, does NOT replace QA expertise.
+
+---
+
+## Support Boundaries
+
+### Supported
+
+- Bugs in generation pipeline
+- Timeout/configuration issues
+- Parsing errors from valid specs
+- Documentation clarifications
+
+### Out of Scope
+
+- "Generated code is wrong" (use review process)
+- "LLM hallucinates" (model limitation)
+- "Tests don't pass in my environment" (configuration required)
+- "How do I write better prompts?" (prompt engineering is your responsibility)
+
+---
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.4.5 | 2026-01-01 | Initial limitations document |
+
+---
+
+## Feedback
+
+If you encounter unclear limitations, please file an issue:
+https://github.com/xyqotech/qa360/issues
+
+Label: `documentation`

package/docs/MIGRATION.md

@@ -0,0 +1,303 @@
+# Migration Guide: Pack v1 → v2
+
+**Version:** 1.6.0
+**Last Updated:** January 3, 2025
+
+---
+
+## Overview
+
+QA360 Pack v2 introduces a more flexible and powerful configuration format while maintaining backward compatibility with v1 packs.
+
+This guide helps you migrate your existing v1 packs to v2 format.
+
+---
+
+## Key Changes
+
+### 1. Gates: Array → Object
+
+**v1 (Array):**
+```yaml
+gates:
+  - api_smoke
+  - ui
+  - perf
+```
+
+**v2 (Object):**
+```yaml
+gates:
+  api:
+    type: api_smoke
+    config: { ... }
+  ui:
+    type: ui
+    config: { ... }
+  perf:
+    type: perf
+    config: { ... }
+```
+
+### 2. Environment → Auth Profiles + Variables
+
+**v1 (Single environment):**
+```yaml
+environment:
+  API_URL: https://api.example.com
+  API_KEY: ${secrets.API_KEY}
+```
+
+**v2 (Auth profiles + variables):**
+```yaml
+auth:
+  default:
+    type: bearer
+    token: ${secrets.API_KEY}
+
+variables:
+  API_URL: https://api.example.com
+```
+
+### 3. Hooks: String → Typed Objects
+
+**v1 (String commands):**
+```yaml
+hooks:
+  beforeAll:
+    - docker compose up -d
+    - npx wait-on http://localhost:3000
+```
+
+**v2 (Typed hook objects):**
+```yaml
+hooks:
+  beforeAll:
+    - run: docker compose up -d
+      timeout: 30000
+      name: Start services
+    - run: npx wait-on http://localhost:3000
+      timeout: 30000
+      name: Wait for API
+```
+
+### 4. Adapters: Explicit Configuration
+
+**v1 (Implicit):**
+```yaml
+gates:
+  - api_smoke
+targets:
+  api:
+    baseUrl: https://api.example.com
+    smoke:
+      - "GET /health -> 200"
+```
+
+**v2 (Explicit adapters):**
+```yaml
+gates:
+  api:
+    type: api_smoke
+    adapter: playwright-native-api
+    config:
+      baseUrl: https://api.example.com
+      tests:
+        - method: GET
+          path: /health
+          expectedStatus: 200
+```
+
+---
+
+## Auto-Migration Tool
+
+QA360 provides an automatic migration tool:
+
+```bash
+# Preview migration (dry-run)
+qa360 pack migrate pack.yml --dry-run
+
+# Perform migration
+qa360 pack migrate pack.yml --output pack-v2.yml
+
+# Validate migrated pack
+qa360 pack validate pack-v2.yml --check-files
+```
+
+---
+
+## Manual Migration Examples
+
+### Example 1: Simple API Smoke Test
+
+**v1 pack.yml:**
+```yaml
+version: 1
+name: "My API Tests"
+gates:
+  - api_smoke
+targets:
+  api:
+    baseUrl: https://api.example.com
+    smoke:
+      - "GET /health -> 200"
+      - "GET /users -> 200"
+      - "POST /users -> 201"
+```
+
+**v2 pack.yml:**
+```yaml
+version: 2
+name: "My API Tests"
+
+gates:
+  api:
+    type: api_smoke
+    adapter: playwright-native-api
+    config:
+      baseUrl: https://api.example.com
+      tests:
+        - method: GET
+          path: /health
+          expectedStatus: 200
+        - method: GET
+          path: /users
+          expectedStatus: 200
+        - method: POST
+          path: /users
+          expectedStatus: 201
+```
+
+### Example 2: UI + API with Auth
+
+**v1 pack.yml:**
+```yaml
+version: 1
+name: "Full Stack Tests"
+gates:
+  - api_smoke
+  - ui
+environment:
+  API_URL: https://api.example.com
+  API_KEY: ${secrets.API_KEY}
+targets:
+  api:
+    baseUrl: ${API_URL}
+    smoke:
+      - "GET /health -> 200"
+  web:
+    baseUrl: https://example.com
+    pages:
+      - https://example.com/login
+      - https://example.com/dashboard
+hooks:
+  beforeAll:
+    - docker compose up -d
+    - npx wait-on ${API_URL}/health
+```
+
+**v2 pack.yml:**
+```yaml
+version: 2
+name: "Full Stack Tests"
+
+auth:
+  default:
+    type: bearer
+    token: ${secrets.API_KEY}
+
+variables:
+  API_URL: https://api.example.com
+
+gates:
+  api:
+    type: api_smoke
+    adapter: playwright-native-api
+    config:
+      baseUrl: ${API_URL}
+      headers:
+        Authorization: Bearer ${auth.default.token}
+      tests:
+        - method: GET
+          path: /health
+          expectedStatus: 200
+
+  ui:
+    type: ui
+    adapter: playwright-ui
+    config:
+      baseUrl: https://example.com
+      pages:
+        - https://example.com/login
+        - https://example.com/dashboard
+
+hooks:
+  beforeAll:
+    - run: docker compose up -d
+      timeout: 30000
+    - run: npx wait-on ${variables.API_URL}/health
+      timeout: 30000
+```
+
+---
+
+## Migration Checklist
+
+- [ ] Backup your existing pack.yml
+- [ ] Run `qa360 pack migrate --dry-run` to preview changes
+- [ ] Run `qa360 pack migrate` to create v2 pack
+- [ ] Review the migrated pack for manual adjustments
+- [ ] Run `qa360 pack validate --check-files` to validate
+- [ ] Test the new pack: `qa360 run pack-v2.yml`
+- [ ] Update CI/CD pipelines to use v2 pack
+- [ ] Remove old v1 pack after confirming v2 works
+
+---
+
+## Breaking Changes
+
+1. **Gates are now objects** - Must specify `type` and `adapter`
+2. **Auth is separate from variables** - Use `auth` section for credentials
+3. **Hooks are typed objects** - Must specify `run`, `timeout`, etc.
+4. **Adapters are explicit** - Must specify which adapter to use
+
+---
+
+## Rollback
+
+If you need to rollback to v1 format, simply restore your backup:
+
+```bash
+# Restore from backup
+cp pack.yml.backup pack.yml
+
+# Or use git to restore
+git checkout HEAD -- pack.yml
+```
+
+QA360 v1.6.0 continues to support v1 packs with auto-detection.
+
+---
+
+## Getting Help
+
+```bash
+# Get migration help
+qa360 pack migrate --help
+
+# Validate your pack
+qa360 pack validate pack.yml
+
+# Auto-fix formatting issues
+qa360 pack lint pack.yml
+```
+
+---
+
+## Additional Resources
+
+- [Architecture Documentation](./ARCHITECTURE.md)
+- [User Guide](./USER_GUIDE.md)
+- [Contributing Guide](./CONTRIBUTING.md)
+- [GitHub Issues](https://github.com/xyqotech/qa360/issues)