qa360 2.2.1 → 2.2.13

package/docs/DEVELOPMENT_HISTORY.md

@@ -1,345 +0,0 @@
-# QA360 Development History
-
-This document provides a historical record of QA360's development phases.
-
-> **Note**: For current project status, see [STATUS.md](./STATUS.md).
-
----
-
-## Phase 1: Runner CLI "Zero Friction"
-
-**Completed**: October 2024
-
-### Objectives
-- Develop a CLI enabling "1 command • 1 pack • 1 signed proof"
-- Implement anti-friction features for seamless user experience
-
-### Delivered Features
-
-#### Core Commands
-- `qa360 doctor` - System diagnostics with auto-fix capabilities
-- `qa360 ask` - Pack generation from natural language
-- `qa360 run` - Test execution with hooks support
-- `qa360 report --sign` - PDF generation with Ed25519 signatures
-- `qa360 verify` - Cryptographic proof verification
-- `qa360 explain` - Error code explanations
-
-#### Anti-Friction Features
-- **Portable cache** (`.qa360/tools/`) for downloaded tools
-- **Docker fallback** for missing tools
-- **Pedagogical error messages** (QX001-QX007)
-- **Targeted retry** with exponential backoff
-- **Automatic log redaction** for PII and secrets
-
-### Architecture
-- Monorepo structure with CLI and Core separation
-- Ed25519 cryptographic signatures
-- Trust Score calculation
-- PDF report generation with Puppeteer
-
-### Metrics
-- **Tests**: 26/28 passing (93% success)
-- **Commands**: 7 implemented
-- **Error codes**: QX001-QX007 with guidance
-- **Performance**: <1s pack generation, <100ms verification
-
----
-
-## Phase 2: Pack v1 + Encrypted Secrets
-
-**Completed**: October 2024
-
-### Objectives
-- Standardize Pack v1 schema
-- Implement encrypted secrets management
-- Ensure deterministic proof generation
-
-### Delivered Features
-
-#### Pack Schema v1
-- Official JSON schema (`pack.schema.json`)
-- Required fields: version, name, gates
-- Supported gates: api_smoke, ui, perf, sast, dast, a11y
-- Validation with 15+ business rules (QP### error codes)
-- Automatic migration from v0.9 to v1
-
-#### Pack CLI Commands
-- `qa360 pack validate` - Schema and business rule validation
-- `qa360 pack lint` - Auto-correction of format issues
-- `qa360 pack migrate` - Legacy to v1 migration with backup
-
-#### Secrets Management
-- **Encryption**: AES-256-GCM with PBKDF2 (100k iterations)
-- **Key derivation**: System keychain (macOS/Windows/Linux)
-- **Storage**: `.qa360/secrets.enc` with 600 permissions
-- **Integrity**: SHA-256 checksum + GCM authentication
-
-#### Secrets CLI Commands
-- `qa360 secrets add` - Secure secret addition
-- `qa360 secrets list` - List secret names (values hidden)
-- `qa360 secrets remove` - Delete with confirmation
-- `qa360 secrets doctor` - System diagnostics
-- `qa360 secrets export --redacted` - Masked export for debugging
-
-#### Secret Interpolation
-- Syntax: `${{ secrets.NAME }}` in pack.yml
-- Automatic validation before execution
-- Zero secrets leakage in logs/reports
-
-#### Security Redaction
-- 15+ patterns for masking (passwords, tokens, API keys, JWT, etc.)
-- Applied to logs, reports, JSON objects, errors
-- Length-preserving or fixed-format masking
-
-### Architecture
-- **PackValidator** - Schema + business rule validation
-- **PackMigrator** - Automatic version migration
-- **SecretsManager** - CRUD with secure caching
-- **SecretsCrypto** - AES-256-GCM encryption
-- **SecurityRedactor** - Automatic PII masking
-
-### Metrics
-- **Tests**: 79/87 passing (91% success)
-- **Commands**: 12 new commands (pack + secrets)
-- **Error codes**: 20+ QP### codes
-- **Security**: AES-256-GCM, PBKDF2 100k iterations
-- **Performance**: <100ms validation, <1s encryption
-
-### Impact
-- **Determinism**: Same pack.yml → same proof everywhere
-- **Security**: Secrets encrypted locally, never in VCS
-- **Portability**: Pack v1 works on all environments
-- **Compliance**: Military-grade encryption, strict permissions
-
----
-
-## Phase 3: Declarative Hooks + Phase3Runner
-
-**Completed**: December 2024
-
-### Objectives
-- Implement Phase3Runner orchestration engine
-- Connect CLI commands to core engine
-- Establish cryptographic proof foundation
-
-### Delivered Features
-
-#### Phase3Runner
-- Gate orchestration with lifecycle management
-- Ed25519 real signatures (production-ready)
-- Evidence Vault with SQLite storage
-- Trust Score calculation
-- Proof bundle generation
-
-#### CLI Integration
-- `qa360 run` connected to Phase3Runner
-- `qa360 verify` connected to Ed25519 verifier
-- Phase3 proof format support
-- Graceful error handling
-
-#### Evidence System
-- SQLite with WAL mode
-- Content-addressable storage (CAS)
-- Immutable audit trail
-- Historical run tracking
-
-### Architecture
-- **Phase3Runner** - Core orchestration engine
-- **ProofGenerator** - Cryptographic proof creation
-- **SignatureManager** - Ed25519 signing/verification
-- **EvidenceVault** - SQLite storage layer
-
-### Metrics
-- **Completion**: Phase3Runner fully operational
-- **Signatures**: Ed25519 real implementation
-- **Storage**: SQLite WAL mode
-- **Integration**: CLI commands connected
-
----
-
-## Phase 4: OOTB Adapters
-
-**Completed**: December 2024
-
-### Objectives
-- Implement production-ready test adapters
-- Support multiple testing domains (API, UI, performance, security)
-- Enable comprehensive quality gates
-
-### Delivered Adapters (7 total)
-
-1. **playwright-api** - REST/GraphQL API testing
-2. **playwright-ui** - UI + Accessibility testing (axe-core)
-3. **k6-perf** - Load & performance testing
-4. **semgrep-sast** - Static security analysis
-5. **zap-dast** - Dynamic security scanning (Docker/local, **REAL implementation**)
-6. **gitleaks-secrets** - Secret detection in code
-7. **osv-deps** - Dependency vulnerability scanning
-
-### Features
-- Unified adapter interface
-- Docker and local execution support
-- Budget threshold validation
-- Comprehensive error codes
-- Production-ready implementations
-
-### Metrics
-- **Adapters**: 7 production-ready
-- **Coverage**: API, UI, performance, security, accessibility, dependencies
-- **Docker**: Full container support
-- **Local**: Native execution support
-
----
-
-## Phase 5: Health/Diag/Metrics Endpoints
-
-**Completed**: December 2024
-
-### Objectives
-- Implement observability server
-- Enable CI/CD readiness checks
-- Provide Prometheus metrics
-
-### Delivered Features
-
-#### Observability Server
-- `qa360 serve` command with HTTP server
-- `/health` - Liveness and readiness checks
-- `/diag` - Diagnostic with actionable solutions
-- `/metrics` - OpenMetrics/Prometheus format
-- `/cancel/:runId` - Graceful shutdown
-
-#### Health Checks
-- System validation (tmp, config, disk space)
-- Tool detection (Playwright, k6, ZAP)
-- Secret accessibility checks
-- Readiness scoring
-
-#### Metrics
-- System metrics (uptime, version, runs)
-- Gate metrics (duration, failures by type)
-- Security metrics (findings by severity)
-- Performance metrics (P95, P90, P50, error rate)
-
-### Architecture
-- **QA360Server** - HTTP routing and error handling
-- **HealthChecker** - Liveness/readiness validation
-- **DiagnosticsCollector** - Problem detection + solutions
-- **MetricsCollector** - OpenMetrics/Prometheus export
-- **ProcessManager** - Graceful shutdown management
-
-### Metrics
-- **Endpoints**: 4 operational
-- **CI/CD**: Readiness check with exit 0/1
-- **Monitoring**: Prometheus-compatible
-- **Shutdown**: Graceful with cleanup
-
----
-
-## Phase 6: Evidence Vault + SQLite WAL History
-
-**Completed**: December 2024
-
-### Objectives
-- Implement persistent evidence storage
-- Enable historical run tracking
-- Provide content-addressable storage
-
-### Delivered Features
-
-#### Evidence Vault
-- SQLite database with WAL mode
-- Content-addressable storage (SHA-256 based)
-- Immutable audit trail
-- Run metadata tracking
-
-#### Features
-- Historical run queries
-- Trend analysis support
-- Artifact deduplication
-- Integrity verification
-
-### Architecture
-- **EvidenceVault** - SQLite storage layer
-- **HistoryManager** - Run tracking
-- **AuditTrail** - Immutable records
-
----
-
-## Current Status
-
-As of December 2024:
-- **Phases 1-6**: ✅ Complete
-- **Phase 7**: 🚧 In Progress (Complete Proof Bundle - 90%)
-- **Phase 8+**: ⏳ Planned
-
-### Key Metrics
-- **412 tests** passing (100% success rate)
-- **7 adapters** in production
-- **9 CLI commands** fully functional
-- **~92% overall completion**
-
-For detailed current status, see [STATUS.md](./STATUS.md).
-
----
-
-## Technology Stack Evolution
-
-### Core Technologies
-- **Runtime**: Node.js 18+
-- **Language**: TypeScript 5+
-- **Testing**: Jest
-- **Monorepo**: pnpm workspaces
-
-### Security
-- **Signatures**: Ed25519 (tweetnacl)
-- **Hashing**: SHA-256
-- **Encryption**: AES-256-GCM
-- **Key Derivation**: PBKDF2 (100k iterations)
-- **Validation**: JSON Schema (AJV 2020-12)
-
-### Storage
-- **Database**: SQLite with WAL mode
-- **Content Addressing**: SHA-256 based CAS
-- **Permissions**: Strict 600 for sensitive files
-
-### Adapters
-- **UI Testing**: Playwright
-- **API Testing**: Playwright
-- **Performance**: K6
-- **SAST**: Semgrep
-- **DAST**: OWASP ZAP
-- **Secrets**: Gitleaks
-- **Dependencies**: OSV
-
----
-
-## Lessons Learned
-
-### What Worked Well
-1. **Incremental phases** - Clear milestones and deliverables
-2. **Security-first** - Built-in encryption and redaction from start
-3. **Local-first** - No cloud dependencies, fully portable
-4. **Developer experience** - Anti-friction features throughout
-
-### Challenges Overcome
-1. **ESM/Jest compatibility** - Resolved with inline schemas and mocks
-2. **Cross-platform secrets** - Unified keychain approach
-3. **Docker integration** - Seamless fallback mechanisms
-4. **Proof determinism** - Canonical JSON and stable hashing
-
-### Future Improvements
-1. Test coverage expansion (55% → 75%+)
-2. Additional adapters (Appium for mobile)
-3. Enhanced TUI/UI dashboard
-4. Multi-language SDK support
-
----
-
-## References
-
-- [Architecture Overview](./ARCHITECTURE.md)
-- [Current Status](./STATUS.md)
-- [Pack v1 Specification](./pack-v1.md)
-- [Secrets Management](./secrets.md)
-- [Proof Bundles](./proofs.md)

package/docs/LIMITATIONS.md

@@ -1,176 +0,0 @@
-# QA360 Generation Module - Limitations
-
-**Version**: 1.4.5
-**Date**: 2026-01-01
-**Module**: `core/src/generation`
-
----
-
-## Purpose
-
-This document explicitly lists what the Generation module **does NOT guarantee**.
-
-Users must review all generated code before deployment.
-
----
-
-## What is Generated
-
-The Generation module produces **draft test code** based on:
-- OpenAPI/Swagger specifications
-- HAR files (HTTP recordings)
-- URLs (via scraping simulation)
-- Natural language descriptions
-- Source code (for unit tests)
-
----
-
-## What is NOT Guaranteed
-
-### 1. Code Correctness
-
-**NOT guaranteed**:
-- Generated code will execute without errors
-- API endpoints exist or match the specification
-- Selectors (CSS/XPath) are valid or stable
-- Assertions are logically correct
-- Edge cases are covered
-
-**Example of failure**:
-```typescript
-// Generated code that may fail
-test('user login', async ({ page }) => {
-  await page.click('#login-button');  // Selector may not exist
-  await page.fill('#email', 'test@example.com');  // ID may differ
-  // No actual verification login succeeded
-});
-```
-
-### 2. Test Completeness
-
-**NOT guaranteed**:
-- All user flows are covered
-- Error scenarios are tested
-- Boundary conditions are handled
-- Security implications are considered
-
-### 3. Production Readiness
-
-**NOT guaranteed**:
-- Generated tests are ready for production CI/CD
-- Tests will pass in your environment
-- Authentication/authorization is correctly handled
-- Test data is valid or safe
-
-### 4. Idempotency
-
-**NOT guaranteed**:
-- Running generation twice produces identical output
-- Generated tests can be run multiple times safely
-- Side effects (if any) are handled
-
----
-
-## Required Human Review
-
-Before committing generated tests, reviewers MUST verify:
-
-1. **Syntax validity**: Code parses without errors
-2. **Selector accuracy**: CSS/XPath selectors match actual DOM
-3. **Assertion correctness**: Tests verify meaningful behavior
-4. **Error handling**: Appropriate error checks exist
-5. **Data safety**: No hardcoded credentials or PII
-6. **Environment compatibility**: Tests work in your infrastructure
-
----
-
-## LLM-Specific Limitations
-
-### Non-Determinism
-
-The LLM (Ollama) may produce different outputs for the same input due to:
-- Model version changes
-- Temperature settings
-- Random seed variations
-- Context window limitations
-
-### Timeout Behavior
-
-- Default timeout: 30 seconds
-- Configurable via `timeout` option (100ms to 10 minutes)
-- Timeout returns structured error, does not crash
-- Partial results are discarded on timeout
-
-### Resource Requirements
-
-- LLM must be installed and running (`ollama serve`)
-- Minimum 4GB RAM recommended for `deepseek-coder` model
-- Network latency affects generation time
-- Large responses (>1MB) may be truncated
-
----
-
-## Security Considerations
-
-### NOT Protected Against
-
-- Prompt injection attacks
-- Malicious HAR files
-- Code injection via generated tests
-- Exposure of secrets in prompts
-
-### Best Practices
-
-1. Review HAR files before generation
-2. Sanitize sensitive data from specifications
-3. Run generated tests in isolated environment first
-4. Never commit generated tests without review
-
----
-
-## Comparison to Manual Testing
-
-| Aspect | Manual Testing | QA360 Generation |
-|--------|---------------|-------------------|
-| Understanding of context | Human learns | LLM guesses |
-| Error handling | Intentional | May be missing |
-| Edge cases | Experienced tester considers | Often overlooked |
-| Maintenance | Developer owns | Requires human update |
-| Speed | Slow | Fast (draft in seconds) |
-
-**Conclusion**: QA360 accelerates test creation, does NOT replace QA expertise.
-
----
-
-## Support Boundaries
-
-### Supported
-
-- Bugs in generation pipeline
-- Timeout/configuration issues
-- Parsing errors from valid specs
-- Documentation clarifications
-
-### Out of Scope
-
-- "Generated code is wrong" (use review process)
-- "LLM hallucinates" (model limitation)
-- "Tests don't pass in my environment" (configuration required)
-- "How do I write better prompts?" (prompt engineering is your responsibility)
-
----
-
-## Version History
-
-| Version | Date | Changes |
-|---------|------|---------|
-| 1.4.5 | 2026-01-01 | Initial limitations document |
-
----
-
-## Feedback
-
-If you encounter unclear limitations, please file an issue:
-https://github.com/xyqotech/qa360/issues
-
-Label: `documentation`