qa360 2.1.7 → 2.2.0

package/docs/DEVELOPMENT_HISTORY.md

@@ -0,0 +1,345 @@
+# QA360 Development History
+
+This document provides a historical record of QA360's development phases.
+
+> **Note**: For current project status, see [STATUS.md](./STATUS.md).
+
+---
+
+## Phase 1: Runner CLI "Zero Friction"
+
+**Completed**: October 2024
+
+### Objectives
+- Develop a CLI enabling "1 command • 1 pack • 1 signed proof"
+- Implement anti-friction features for seamless user experience
+
+### Delivered Features
+
+#### Core Commands
+- `qa360 doctor` - System diagnostics with auto-fix capabilities
+- `qa360 ask` - Pack generation from natural language
+- `qa360 run` - Test execution with hooks support
+- `qa360 report --sign` - PDF generation with Ed25519 signatures
+- `qa360 verify` - Cryptographic proof verification
+- `qa360 explain` - Error code explanations
+
+#### Anti-Friction Features
+- **Portable cache** (`.qa360/tools/`) for downloaded tools
+- **Docker fallback** for missing tools
+- **Pedagogical error messages** (QX001-QX007)
+- **Targeted retry** with exponential backoff
+- **Automatic log redaction** for PII and secrets
+
+### Architecture
+- Monorepo structure with CLI and Core separation
+- Ed25519 cryptographic signatures
+- Trust Score calculation
+- PDF report generation with Puppeteer
+
+### Metrics
+- **Tests**: 26/28 passing (93% success)
+- **Commands**: 7 implemented
+- **Error codes**: QX001-QX007 with guidance
+- **Performance**: <1s pack generation, <100ms verification
+
+---
+
+## Phase 2: Pack v1 + Encrypted Secrets
+
+**Completed**: October 2024
+
+### Objectives
+- Standardize Pack v1 schema
+- Implement encrypted secrets management
+- Ensure deterministic proof generation
+
+### Delivered Features
+
+#### Pack Schema v1
+- Official JSON schema (`pack.schema.json`)
+- Required fields: version, name, gates
+- Supported gates: api_smoke, ui, perf, sast, dast, a11y
+- Validation with 15+ business rules (QP### error codes)
+- Automatic migration from v0.9 to v1
+
+#### Pack CLI Commands
+- `qa360 pack validate` - Schema and business rule validation
+- `qa360 pack lint` - Auto-correction of format issues
+- `qa360 pack migrate` - Legacy to v1 migration with backup
+
+#### Secrets Management
+- **Encryption**: AES-256-GCM with PBKDF2 (100k iterations)
+- **Key derivation**: System keychain (macOS/Windows/Linux)
+- **Storage**: `.qa360/secrets.enc` with 600 permissions
+- **Integrity**: SHA-256 checksum + GCM authentication
+
+#### Secrets CLI Commands
+- `qa360 secrets add` - Secure secret addition
+- `qa360 secrets list` - List secret names (values hidden)
+- `qa360 secrets remove` - Delete with confirmation
+- `qa360 secrets doctor` - System diagnostics
+- `qa360 secrets export --redacted` - Masked export for debugging
+
+#### Secret Interpolation
+- Syntax: `${{ secrets.NAME }}` in pack.yml
+- Automatic validation before execution
+- Zero secrets leakage in logs/reports
+
+#### Security Redaction
+- 15+ patterns for masking (passwords, tokens, API keys, JWT, etc.)
+- Applied to logs, reports, JSON objects, errors
+- Length-preserving or fixed-format masking
+
+### Architecture
+- **PackValidator** - Schema + business rule validation
+- **PackMigrator** - Automatic version migration
+- **SecretsManager** - CRUD with secure caching
+- **SecretsCrypto** - AES-256-GCM encryption
+- **SecurityRedactor** - Automatic PII masking
+
+### Metrics
+- **Tests**: 79/87 passing (91% success)
+- **Commands**: 12 new commands (pack + secrets)
+- **Error codes**: 20+ QP### codes
+- **Security**: AES-256-GCM, PBKDF2 100k iterations
+- **Performance**: <100ms validation, <1s encryption
+
+### Impact
+- **Determinism**: Same pack.yml → same proof everywhere
+- **Security**: Secrets encrypted locally, never in VCS
+- **Portability**: Pack v1 works on all environments
+- **Compliance**: Military-grade encryption, strict permissions
+
+---
+
+## Phase 3: Declarative Hooks + Phase3Runner
+
+**Completed**: December 2024
+
+### Objectives
+- Implement Phase3Runner orchestration engine
+- Connect CLI commands to core engine
+- Establish cryptographic proof foundation
+
+### Delivered Features
+
+#### Phase3Runner
+- Gate orchestration with lifecycle management
+- Ed25519 real signatures (production-ready)
+- Evidence Vault with SQLite storage
+- Trust Score calculation
+- Proof bundle generation
+
+#### CLI Integration
+- `qa360 run` connected to Phase3Runner
+- `qa360 verify` connected to Ed25519 verifier
+- Phase3 proof format support
+- Graceful error handling
+
+#### Evidence System
+- SQLite with WAL mode
+- Content-addressable storage (CAS)
+- Immutable audit trail
+- Historical run tracking
+
+### Architecture
+- **Phase3Runner** - Core orchestration engine
+- **ProofGenerator** - Cryptographic proof creation
+- **SignatureManager** - Ed25519 signing/verification
+- **EvidenceVault** - SQLite storage layer
+
+### Metrics
+- **Completion**: Phase3Runner fully operational
+- **Signatures**: Ed25519 real implementation
+- **Storage**: SQLite WAL mode
+- **Integration**: CLI commands connected
+
+---
+
+## Phase 4: OOTB Adapters
+
+**Completed**: December 2024
+
+### Objectives
+- Implement production-ready test adapters
+- Support multiple testing domains (API, UI, performance, security)
+- Enable comprehensive quality gates
+
+### Delivered Adapters (7 total)
+
+1. **playwright-api** - REST/GraphQL API testing
+2. **playwright-ui** - UI + Accessibility testing (axe-core)
+3. **k6-perf** - Load & performance testing
+4. **semgrep-sast** - Static security analysis
+5. **zap-dast** - Dynamic security scanning (Docker/local, **REAL implementation**)
+6. **gitleaks-secrets** - Secret detection in code
+7. **osv-deps** - Dependency vulnerability scanning
+
+### Features
+- Unified adapter interface
+- Docker and local execution support
+- Budget threshold validation
+- Comprehensive error codes
+- Production-ready implementations
+
+### Metrics
+- **Adapters**: 7 production-ready
+- **Coverage**: API, UI, performance, security, accessibility, dependencies
+- **Docker**: Full container support
+- **Local**: Native execution support
+
+---
+
+## Phase 5: Health/Diag/Metrics Endpoints
+
+**Completed**: December 2024
+
+### Objectives
+- Implement observability server
+- Enable CI/CD readiness checks
+- Provide Prometheus metrics
+
+### Delivered Features
+
+#### Observability Server
+- `qa360 serve` command with HTTP server
+- `/health` - Liveness and readiness checks
+- `/diag` - Diagnostic with actionable solutions
+- `/metrics` - OpenMetrics/Prometheus format
+- `/cancel/:runId` - Graceful shutdown
+
+#### Health Checks
+- System validation (tmp, config, disk space)
+- Tool detection (Playwright, k6, ZAP)
+- Secret accessibility checks
+- Readiness scoring
+
+#### Metrics
+- System metrics (uptime, version, runs)
+- Gate metrics (duration, failures by type)
+- Security metrics (findings by severity)
+- Performance metrics (P95, P90, P50, error rate)
+
+### Architecture
+- **QA360Server** - HTTP routing and error handling
+- **HealthChecker** - Liveness/readiness validation
+- **DiagnosticsCollector** - Problem detection + solutions
+- **MetricsCollector** - OpenMetrics/Prometheus export
+- **ProcessManager** - Graceful shutdown management
+
+### Metrics
+- **Endpoints**: 4 operational
+- **CI/CD**: Readiness check with exit 0/1
+- **Monitoring**: Prometheus-compatible
+- **Shutdown**: Graceful with cleanup
+
+---
+
+## Phase 6: Evidence Vault + SQLite WAL History
+
+**Completed**: December 2024
+
+### Objectives
+- Implement persistent evidence storage
+- Enable historical run tracking
+- Provide content-addressable storage
+
+### Delivered Features
+
+#### Evidence Vault
+- SQLite database with WAL mode
+- Content-addressable storage (SHA-256 based)
+- Immutable audit trail
+- Run metadata tracking
+
+#### Features
+- Historical run queries
+- Trend analysis support
+- Artifact deduplication
+- Integrity verification
+
+### Architecture
+- **EvidenceVault** - SQLite storage layer
+- **HistoryManager** - Run tracking
+- **AuditTrail** - Immutable records
+
+---
+
+## Current Status
+
+As of December 2024:
+- **Phases 1-6**: ✅ Complete
+- **Phase 7**: 🚧 In Progress (Complete Proof Bundle - 90%)
+- **Phase 8+**: ⏳ Planned
+
+### Key Metrics
+- **412 tests** passing (100% success rate)
+- **7 adapters** in production
+- **9 CLI commands** fully functional
+- **~92% overall completion**
+
+For detailed current status, see [STATUS.md](./STATUS.md).
+
+---
+
+## Technology Stack Evolution
+
+### Core Technologies
+- **Runtime**: Node.js 18+
+- **Language**: TypeScript 5+
+- **Testing**: Jest
+- **Monorepo**: pnpm workspaces
+
+### Security
+- **Signatures**: Ed25519 (tweetnacl)
+- **Hashing**: SHA-256
+- **Encryption**: AES-256-GCM
+- **Key Derivation**: PBKDF2 (100k iterations)
+- **Validation**: JSON Schema (AJV 2020-12)
+
+### Storage
+- **Database**: SQLite with WAL mode
+- **Content Addressing**: SHA-256 based CAS
+- **Permissions**: Strict 600 for sensitive files
+
+### Adapters
+- **UI Testing**: Playwright
+- **API Testing**: Playwright
+- **Performance**: K6
+- **SAST**: Semgrep
+- **DAST**: OWASP ZAP
+- **Secrets**: Gitleaks
+- **Dependencies**: OSV
+
+---
+
+## Lessons Learned
+
+### What Worked Well
+1. **Incremental phases** - Clear milestones and deliverables
+2. **Security-first** - Built-in encryption and redaction from start
+3. **Local-first** - No cloud dependencies, fully portable
+4. **Developer experience** - Anti-friction features throughout
+
+### Challenges Overcome
+1. **ESM/Jest compatibility** - Resolved with inline schemas and mocks
+2. **Cross-platform secrets** - Unified keychain approach
+3. **Docker integration** - Seamless fallback mechanisms
+4. **Proof determinism** - Canonical JSON and stable hashing
+
+### Future Improvements
+1. Test coverage expansion (55% → 75%+)
+2. Additional adapters (Appium for mobile)
+3. Enhanced TUI/UI dashboard
+4. Multi-language SDK support
+
+---
+
+## References
+
+- [Architecture Overview](./ARCHITECTURE.md)
+- [Current Status](./STATUS.md)
+- [Pack v1 Specification](./pack-v1.md)
+- [Secrets Management](./secrets.md)
+- [Proof Bundles](./proofs.md)

package/docs/LIMITATIONS.md

@@ -0,0 +1,176 @@
+# QA360 Generation Module - Limitations
+
+**Version**: 1.4.5
+**Date**: 2026-01-01
+**Module**: `core/src/generation`
+
+---
+
+## Purpose
+
+This document explicitly lists what the Generation module **does NOT guarantee**.
+
+Users must review all generated code before deployment.
+
+---
+
+## What is Generated
+
+The Generation module produces **draft test code** based on:
+- OpenAPI/Swagger specifications
+- HAR files (HTTP recordings)
+- URLs (via scraping simulation)
+- Natural language descriptions
+- Source code (for unit tests)
+
+---
+
+## What is NOT Guaranteed
+
+### 1. Code Correctness
+
+**NOT guaranteed**:
+- Generated code will execute without errors
+- API endpoints exist or match the specification
+- Selectors (CSS/XPath) are valid or stable
+- Assertions are logically correct
+- Edge cases are covered
+
+**Example of failure**:
+```typescript
+// Generated code that may fail
+test('user login', async ({ page }) => {
+  await page.click('#login-button');  // Selector may not exist
+  await page.fill('#email', 'test@example.com');  // ID may differ
+  // No actual verification login succeeded
+});
+```
+
+### 2. Test Completeness
+
+**NOT guaranteed**:
+- All user flows are covered
+- Error scenarios are tested
+- Boundary conditions are handled
+- Security implications are considered
+
+### 3. Production Readiness
+
+**NOT guaranteed**:
+- Generated tests are ready for production CI/CD
+- Tests will pass in your environment
+- Authentication/authorization is correctly handled
+- Test data is valid or safe
+
+### 4. Idempotency
+
+**NOT guaranteed**:
+- Running generation twice produces identical output
+- Generated tests can be run multiple times safely
+- Side effects (if any) are handled
+
+---
+
+## Required Human Review
+
+Before committing generated tests, reviewers MUST verify:
+
+1. **Syntax validity**: Code parses without errors
+2. **Selector accuracy**: CSS/XPath selectors match actual DOM
+3. **Assertion correctness**: Tests verify meaningful behavior
+4. **Error handling**: Appropriate error checks exist
+5. **Data safety**: No hardcoded credentials or PII
+6. **Environment compatibility**: Tests work in your infrastructure
+
+---
+
+## LLM-Specific Limitations
+
+### Non-Determinism
+
+The LLM (Ollama) may produce different outputs for the same input due to:
+- Model version changes
+- Temperature settings
+- Random seed variations
+- Context window limitations
+
+### Timeout Behavior
+
+- Default timeout: 30 seconds
+- Configurable via `timeout` option (100ms to 10 minutes)
+- Timeout returns structured error, does not crash
+- Partial results are discarded on timeout
+
+### Resource Requirements
+
+- LLM must be installed and running (`ollama serve`)
+- Minimum 4GB RAM recommended for `deepseek-coder` model
+- Network latency affects generation time
+- Large responses (>1MB) may be truncated
+
+---
+
+## Security Considerations
+
+### NOT Protected Against
+
+- Prompt injection attacks
+- Malicious HAR files
+- Code injection via generated tests
+- Exposure of secrets in prompts
+
+### Best Practices
+
+1. Review HAR files before generation
+2. Sanitize sensitive data from specifications
+3. Run generated tests in isolated environment first
+4. Never commit generated tests without review
+
+---
+
+## Comparison to Manual Testing
+
+| Aspect | Manual Testing | QA360 Generation |
+|--------|---------------|-------------------|
+| Understanding of context | Human learns | LLM guesses |
+| Error handling | Intentional | May be missing |
+| Edge cases | Experienced tester considers | Often overlooked |
+| Maintenance | Developer owns | Requires human update |
+| Speed | Slow | Fast (draft in seconds) |
+
+**Conclusion**: QA360 accelerates test creation, does NOT replace QA expertise.
+
+---
+
+## Support Boundaries
+
+### Supported
+
+- Bugs in generation pipeline
+- Timeout/configuration issues
+- Parsing errors from valid specs
+- Documentation clarifications
+
+### Out of Scope
+
+- "Generated code is wrong" (use review process)
+- "LLM hallucinates" (model limitation)
+- "Tests don't pass in my environment" (configuration required)
+- "How do I write better prompts?" (prompt engineering is your responsibility)
+
+---
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.4.5 | 2026-01-01 | Initial limitations document |
+
+---
+
+## Feedback
+
+If you encounter unclear limitations, please file an issue:
+https://github.com/xyqotech/qa360/issues
+
+Label: `documentation`