qa360 2.1.7 → 2.2.0

package/docs/vault.md

@@ -0,0 +1,491 @@
+# QA360 Evidence Vault
+
+## Overview
+
+The QA360 Evidence Vault is a comprehensive storage system for test execution data, providing:
+
+- **SQLite WAL database** for structured data storage
+- **Content-Addressable Storage (CAS)** for file deduplication
+- **Idempotent runs** via run keys
+- **Cryptographic signatures** for proof integrity
+- **Historical analysis** and trend tracking
+
+## Architecture
+
+### Core Components
+
+1. **EvidenceVault**: Main vault engine with SQLite WAL backend
+2. **ContentAddressableStorage**: File storage with SHA256-based deduplication
+3. **SecurityRedactor**: Automatic PII removal from stored data
+4. **CLI History Commands**: Query and analysis interface
+
+### Database Schema
+
+```sql
+-- Core run records
+runs (
+    id TEXT PRIMARY KEY,           -- UUID v4
+    run_key TEXT UNIQUE,          -- Idempotence key
+    started_at INTEGER,           -- Epoch milliseconds
+    ended_at INTEGER,             -- Epoch milliseconds
+    pack_hash TEXT,               -- SHA256 of normalized pack.yml
+    status TEXT,                  -- running|passed|failed|cancelled|error
+    trust_score INTEGER,          -- 0-100
+    signature_hex TEXT,           -- Ed25519 signature
+    proof_pdf_path TEXT,          -- CAS path to signed proof
+    pinned INTEGER DEFAULT 0      -- GC protection
+)
+
+-- Test gate results
+gates (
+    run_id TEXT,                  -- Foreign key to runs
+    name TEXT,                    -- api_smoke|ui|perf|sast|dast|a11y|deps|secrets
+    status TEXT,                  -- passed|failed|skipped|warn
+    duration_ms INTEGER,          -- Execution time
+    metrics_json TEXT,            -- JSON metrics (p95, a11y score, etc.)
+    budgets_json TEXT             -- JSON budget definitions
+)
+
+-- Security/quality findings
+findings (
+    run_id TEXT,                  -- Foreign key to runs
+    gate TEXT,                    -- Gate that produced finding
+    severity TEXT,                -- info|low|medium|high|critical
+    rule TEXT,                    -- Rule identifier (semgrep:xxx, zap:yyy)
+    location TEXT,                -- File path, URL, or selector
+    message TEXT,                 -- Redacted finding message
+    fingerprint TEXT              -- Unique fingerprint for deduplication
+)
+
+-- File artifacts (CAS metadata)
+artifacts (
+    sha256 TEXT PRIMARY KEY,      -- Content hash
+    mime_type TEXT,               -- MIME type
+    size_bytes INTEGER,           -- File size
+    cas_path TEXT,                -- CAS storage path
+    created_at INTEGER            -- Creation timestamp
+)
+
+-- Run-artifact relationships
+run_artifacts (
+    run_id TEXT,                  -- Foreign key to runs
+    sha256 TEXT,                  -- Foreign key to artifacts
+    label TEXT,                   -- proof_pdf, ui_screenshot_login, etc.
+    PRIMARY KEY (run_id, sha256, label)
+)
+```
+
+## Usage
+
+### Basic Operations
+
+```typescript
+import { EvidenceVault } from '@qa360/core';
+
+// Open vault
+const vault = await EvidenceVault.open('/path/to/.qa360');
+
+// Begin run with idempotence
+const { runId, isReused } = await vault.beginRun({
+  run_key: 'ci-build-123',  // Optional idempotence key
+  pack_path: './pack.yml',
+  pack_hash: 'sha256-of-normalized-pack',
+  weights: { ui: 0.4, api: 0.6 }
+});
+
+// Record gate execution
+await vault.recordGate(runId, {
+  name: 'ui',
+  status: 'passed',
+  duration_ms: 2500,
+  metrics_json: JSON.stringify({
+    interactions: 12,
+    coverage_percent: 85
+  })
+});
+
+// Record security finding
+await vault.recordFinding(runId, {
+  gate: 'sast',
+  severity: 'high',
+  rule: 'semgrep:hardcoded-secret',
+  location: '/src/config.js:42',
+  message: 'Hardcoded API key detected'
+});
+
+// Store proof PDF
+const proofPdf = await fs.readFile('./proof.pdf');
+const artifact = await vault.storeArtifact(
+  runId,
+  proofPdf,
+  'application/pdf',
+  'proof_pdf',
+  'proof.pdf'
+);
+
+// Finish run
+await vault.finishRun(runId, {
+  status: 'passed',
+  trust_score: 87,
+  signature: 'ed25519-signature-hex',
+  proof_pdf_sha: artifact.sha256
+});
+
+// Close vault
+await vault.close();
+```
+
+### CLI Commands
+
+```bash
+# List recent runs
+qa360 history list --limit 10
+
+# Get detailed run information
+qa360 history get <runId>
+
+# Compare two runs
+qa360 history diff <runIdA> <runIdB>
+
+# Show trends over time
+qa360 history trend --gate ui --window 20
+
+# Export run bundle
+qa360 history export <runId> --bundle ./export.zip
+
+# Garbage collection
+qa360 history gc --keep-last 50 --dry-run
+
+# Pin/unpin runs
+qa360 history pin <runId>
+qa360 history unpin <runId>
+```
+
+## Content-Addressable Storage
+
+### File Organization
+
+```
+.qa360/runs/cas/
+├── aa/
+│   ├── bb/
+│   │   ├── aabbcc1234567890abcdef...  # SHA256 filename
+│   │   └── aabbcc9876543210fedcba...
+│   └── cc/
+└── ff/
+    └── ee/
+```
+
+### Benefits
+
+- **Deduplication**: Identical files stored only once
+- **Integrity**: SHA256 verification prevents corruption
+- **Scalability**: Sharded directory structure
+- **Immutability**: Content cannot be modified without changing hash
+
+### Usage
+
+```typescript
+import { ContentAddressableStorage } from '@qa360/core';
+
+const cas = new ContentAddressableStorage('/path/to/.qa360/runs');
+
+// Save file
+const content = Buffer.from('Hello, World!');
+const artifact = await cas.saveArtifact(content, 'text/plain');
+console.log(artifact.sha256); // Content hash
+console.log(artifact.exists);  // false (first save)
+
+// Retrieve file
+const retrieved = await cas.getArtifact(artifact.sha256);
+console.log(retrieved.equals(content)); // true
+
+// Check existence
+console.log(cas.exists(artifact.sha256)); // true
+
+// Get statistics
+const stats = await cas.getStats();
+console.log(`${stats.totalFiles} files, ${stats.totalBytes} bytes`);
+```
+
+## Idempotence
+
+### Run Keys
+
+Run keys enable idempotent test execution:
+
+```typescript
+// First execution
+const result1 = await vault.beginRun({
+  run_key: 'ci-pr-456',
+  pack_path: './pack.yml',
+  pack_hash: 'abc123'
+});
+console.log(result1.isReused); // false
+
+// Subsequent execution with same key
+const result2 = await vault.beginRun({
+  run_key: 'ci-pr-456',
+  pack_path: './pack.yml',
+  pack_hash: 'abc123'
+});
+console.log(result2.isReused); // true
+console.log(result1.runId === result2.runId); // true
+```
+
+### Pack Hash Normalization
+
+Pack hashes ensure consistent idempotence:
+
+```typescript
+const pack1 = `
+name: test
+version: 1.0.0
+# Comment 1
+description: Test pack
+`;
+
+const pack2 = `
+name: test
+version: 1.0.0
+description: Test pack
+# Comment 2
+`;
+
+const hash1 = EvidenceVault.calculatePackHash(pack1);
+const hash2 = EvidenceVault.calculatePackHash(pack2);
+console.log(hash1 === hash2); // true (comments ignored)
+```
+
+## Security & Redaction
+
+### Automatic PII Removal
+
+The vault automatically redacts sensitive information:
+
+```typescript
+// Input finding
+await vault.recordFinding(runId, {
+  gate: 'sast',
+  severity: 'high',
+  rule: 'hardcoded-secret',
+  message: 'API key found: sk_live_1234567890abcdef',
+  raw_output: 'Full tool output with sk_live_1234567890abcdef'
+});
+
+// Stored finding (redacted)
+const findings = await vault.getFindings(runId);
+console.log(findings[0].message); // "API key found: ***"
+```
+
+### Redaction Rules
+
+- **Passwords**: `"password": "value"` → `"password": "***"`
+- **API Keys**: `sk_live_abc123` → `sk_live_***`
+- **Tokens**: `Bearer abc123` → `Bearer ***`
+- **Email**: `user@domain.com` → `***@domain.com`
+- **Credit Cards**: `4111-1111-1111-1111` → `****-****-****-****`
+
+## Performance & Optimization
+
+### SQLite WAL Mode
+
+```sql
+PRAGMA journal_mode = WAL;           -- Write-Ahead Logging
+PRAGMA journal_size_limit = 67108864; -- 64MB limit
+PRAGMA synchronous = NORMAL;         -- Balanced durability/performance
+PRAGMA cache_size = 10000;           -- 10K pages in memory
+PRAGMA foreign_keys = ON;            -- Referential integrity
+```
+
+### Indexing Strategy
+
+```sql
+-- Optimized for common queries
+CREATE INDEX idx_runs_started_at ON runs(started_at DESC);
+CREATE INDEX idx_runs_status ON runs(status);
+CREATE INDEX idx_gates_run_name ON gates(run_id, name);
+CREATE INDEX idx_findings_run_gate ON findings(run_id, gate);
+CREATE INDEX idx_findings_severity ON findings(severity);
+```
+
+### Views for Analytics
+
+```sql
+-- Recent runs with summary
+CREATE VIEW v_recent_runs AS
+SELECT 
+    r.id,
+    r.started_at,
+    r.status,
+    r.trust_score,
+    COUNT(g.id) as gate_count,
+    COUNT(CASE WHEN g.status = 'passed' THEN 1 END) as gates_passed,
+    COUNT(f.id) as finding_count
+FROM runs r
+LEFT JOIN gates g ON r.id = g.run_id
+LEFT JOIN findings f ON r.id = f.run_id
+GROUP BY r.id
+ORDER BY r.started_at DESC;
+```
+
+## Backup & Restore
+
+### Database Backup
+
+```bash
+# Create backup
+sqlite3 .qa360/vault.db ".backup backup.db"
+
+# Restore from backup
+sqlite3 .qa360/vault.db ".restore backup.db"
+```
+
+### CAS Backup
+
+```bash
+# Backup CAS files
+tar -czf cas-backup.tar.gz .qa360/runs/cas/
+
+# Restore CAS files
+tar -xzf cas-backup.tar.gz
+```
+
+### Full Vault Export
+
+```bash
+# Export complete vault
+qa360 history export --all --bundle complete-vault.zip
+
+# Import vault
+qa360 history import --bundle complete-vault.zip
+```
+
+## Monitoring & Maintenance
+
+### Vault Statistics
+
+```typescript
+const stats = await vault.getStats();
+console.log(`
+Total runs: ${stats.totalRuns}
+Total gates: ${stats.totalGates}
+Total findings: ${stats.totalFindings}
+Total artifacts: ${stats.totalArtifacts}
+Vault size: ${ContentAddressableStorage.formatBytes(stats.vaultSizeBytes)}
+Date range: ${stats.oldestRun} - ${stats.newestRun}
+`);
+```
+
+### Garbage Collection
+
+```bash
+# Dry run to see what would be deleted
+qa360 history gc --keep-last 100 --max-bytes 5GB --dry-run
+
+# Actually perform cleanup
+qa360 history gc --keep-last 100 --max-bytes 5GB
+
+# Protect important runs from GC
+qa360 history pin <important-run-id>
+```
+
+### Health Checks
+
+```bash
+# Verify vault integrity
+qa360 doctor --vault
+
+# Check for orphaned artifacts
+qa360 history gc --dry-run --orphans-only
+
+# Verify artifact integrity
+qa360 history verify --all-artifacts
+```
+
+## Integration Examples
+
+### CI/CD Pipeline
+
+```yaml
+# GitHub Actions
+- name: Run QA360 Tests
+  run: |
+    qa360 run --run-key "ci-${{ github.run_id }}" \
+              --pack ./qa360-pack.yml
+    
+- name: Upload Results
+  if: always()
+  uses: actions/upload-artifact@v3
+  with:
+    name: qa360-results
+    path: .qa360/
+```
+
+### Prometheus Metrics
+
+```bash
+# Export metrics for monitoring
+curl http://localhost:4060/metrics | grep qa360_
+```
+
+### Trend Analysis
+
+```typescript
+// Get performance trends
+const runs = await vault.listRuns({ limit: 50 });
+const uiGates = await Promise.all(
+  runs.map(r => vault.getGates(r.id).then(gates => 
+    gates.find(g => g.name === 'ui')
+  ))
+);
+
+const avgDuration = uiGates
+  .filter(g => g && g.duration_ms)
+  .reduce((sum, g) => sum + g.duration_ms!, 0) / uiGates.length;
+
+console.log(`Average UI test duration: ${avgDuration}ms`);
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Database locked**: Ensure proper vault closure
+2. **Disk space**: Monitor CAS growth and run GC
+3. **Performance**: Check SQLite WAL file size
+4. **Corruption**: Use SQLite integrity check
+
+### Diagnostic Commands
+
+```bash
+# Check vault health
+qa360 doctor --vault
+
+# Verify database integrity
+sqlite3 .qa360/vault.db "PRAGMA integrity_check;"
+
+# Check WAL file size
+ls -lh .qa360/vault.db-wal
+
+# Analyze disk usage
+qa360 history stats --detailed
+```
+
+## Best Practices
+
+1. **Regular GC**: Schedule garbage collection
+2. **Pin important runs**: Protect baseline runs from cleanup
+3. **Monitor growth**: Track vault size over time
+4. **Backup strategy**: Regular database and CAS backups
+5. **Run keys**: Use meaningful, unique identifiers
+6. **Artifact labeling**: Use descriptive labels for artifacts
+
+## API Reference
+
+See the TypeScript interfaces in `@qa360/core` for complete API documentation:
+
+- `EvidenceVault`: Main vault interface
+- `ContentAddressableStorage`: File storage interface
+- `RunRecord`, `GateRecord`, `FindingRecord`: Data structures
+- `VaultConfig`, `BeginRunOptions`, `FinishRunOptions`: Configuration types