qa360 2.1.7 → 2.2.0

package/docs/proofs.md

@@ -0,0 +1,670 @@
+# QA360 Cryptographic Proofs
+
+## Overview
+
+QA360 generates cryptographically signed proofs of test execution, providing tamper-evident evidence of quality assurance activities. These proofs combine comprehensive test results with Ed25519 digital signatures to ensure authenticity and integrity.
+
+## Proof Architecture
+
+### Components
+
+1. **Proof Document (PDF)**: Human-readable test report with visual evidence
+2. **Proof Metadata (JSON)**: Machine-readable test data and metrics
+3. **Digital Signature**: Ed25519 cryptographic signature ensuring integrity
+4. **Evidence Vault Link**: Connection to detailed execution data in the vault
+
+### Trust Chain
+
+```
+Pack Configuration → Test Execution → Results Collection → Proof Generation → Digital Signing → Vault Storage
+```
+
+## Proof Generation Process
+
+### 1. Test Execution
+
+```typescript
+// During test execution
+const { runId } = await vault.beginRun({
+  run_key: 'release-v2.1.0',
+  pack_path: './qa360-pack.yml',
+  pack_hash: EvidenceVault.calculatePackHash(packContent)
+});
+
+// Record test results
+await vault.recordGate(runId, {
+  name: 'ui',
+  status: 'passed',
+  duration_ms: 2500,
+  metrics_json: JSON.stringify({ coverage: 85, interactions: 12 })
+});
+```
+
+### 2. Proof Document Creation
+
+```typescript
+// Generate comprehensive proof
+const proofData = {
+  run_id: runId,
+  timestamp: Date.now(),
+  pack_hash: packHash,
+  gates: await vault.getGates(runId),
+  findings: await vault.getFindings(runId),
+  artifacts: await vault.getRunArtifacts(runId),
+  trust_score: calculateTrustScore(gates, findings),
+  metadata: {
+    version: '1.0.0',
+    generator: 'qa360-core',
+    environment: process.env.NODE_ENV
+  }
+};
+
+// Create PDF proof document
+const proofPdf = await generateProofPDF(proofData);
+```
+
+### 3. Digital Signing
+
+```typescript
+import { sign } from 'crypto';
+
+// Load Ed25519 private key
+const privateKey = await loadPrivateKey('.qa360/keys/private.pem');
+
+// Create signature of proof JSON
+const proofJson = JSON.stringify(proofData, null, 2);
+const signature = sign('ed25519', Buffer.from(proofJson), privateKey);
+
+// Store signed proof
+await vault.finishRun(runId, {
+  status: 'passed',
+  trust_score: proofData.trust_score,
+  signature: signature.toString('hex'),
+  proof_pdf_sha: await vault.storeArtifact(
+    runId, 
+    proofPdf, 
+    'application/pdf', 
+    'proof_pdf'
+  ).then(a => a.sha256)
+});
+```
+
+## Proof Structure
+
+### JSON Proof Format
+
+```json
+{
+  "proof_version": "1.0.0",
+  "run_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
+  "timestamp": 1729781445000,
+  "pack": {
+    "path": "./qa360-pack.yml",
+    "hash": "sha256:abc123def456...",
+    "version": "1.2.0"
+  },
+  "execution": {
+    "started_at": 1729781445000,
+    "ended_at": 1729781490000,
+    "duration_ms": 45000,
+    "environment": {
+      "node_version": "18.20.5",
+      "platform": "darwin",
+      "ci": true,
+      "runner": "github-actions"
+    }
+  },
+  "results": {
+    "status": "passed",
+    "trust_score": 87,
+    "gates": [
+      {
+        "name": "ui",
+        "status": "passed",
+        "duration_ms": 2500,
+        "metrics": {
+          "interactions": 12,
+          "coverage_percent": 85,
+          "accessibility_score": 92
+        },
+        "budgets": {
+          "max_duration_ms": 5000,
+          "min_coverage_percent": 80
+        }
+      },
+      {
+        "name": "api",
+        "status": "passed",
+        "duration_ms": 1200,
+        "metrics": {
+          "endpoints_tested": 8,
+          "response_time_p95": 150,
+          "success_rate": 100
+        }
+      },
+      {
+        "name": "sast",
+        "status": "warn",
+        "duration_ms": 800,
+        "metrics": {
+          "files_scanned": 156,
+          "rules_applied": 234
+        }
+      }
+    ],
+    "findings": [
+      {
+        "gate": "sast",
+        "severity": "medium",
+        "rule": "semgrep:hardcoded-url",
+        "location": "/src/config.js:42",
+        "message": "Hardcoded URL detected",
+        "fingerprint": "abc123def456"
+      }
+    ],
+    "coverage": {
+      "ui_interactions": 85,
+      "api_endpoints": 100,
+      "security_rules": 95,
+      "overall": 87
+    }
+  },
+  "artifacts": [
+    {
+      "label": "proof_pdf",
+      "sha256": "def456abc123...",
+      "mime_type": "application/pdf",
+      "size_bytes": 2415616
+    },
+    {
+      "label": "ui_screenshot_final",
+      "sha256": "789abc123def...",
+      "mime_type": "image/png",
+      "size_bytes": 456789
+    }
+  ],
+  "signature": {
+    "algorithm": "ed25519",
+    "public_key": "302a300506032b657003210012345...",
+    "signature": "abcdef123456789...",
+    "signed_data_hash": "sha256:fedcba987654321..."
+  },
+  "verification": {
+    "vault_location": ".qa360/vault.db",
+    "proof_pdf_sha256": "def456abc123...",
+    "qr_code": "https://verify.qa360.ai/proof/a1b2c3d4"
+  }
+}
+```
+
+### PDF Proof Document
+
+The PDF proof contains:
+
+1. **Executive Summary**
+   - Overall test status and trust score
+   - Key metrics and findings summary
+   - Execution timeline and environment
+
+2. **Detailed Results**
+   - Gate-by-gate breakdown
+   - Performance metrics and budgets
+   - Security findings with severity levels
+
+3. **Visual Evidence**
+   - Screenshots of test execution
+   - Charts and graphs of metrics
+   - Error highlights and annotations
+
+4. **Technical Details**
+   - Pack configuration used
+   - Environment and tool versions
+   - Execution logs and traces
+
+5. **Verification Information**
+   - Digital signature details
+   - QR code for online verification
+   - Instructions for proof validation
+
+## Verification Process
+
+### Command Line Verification
+
+```bash
+# Verify proof from run ID
+qa360 verify <runId>
+
+# Verify exported proof bundle
+qa360 verify ./proof-bundle.zip
+
+# Verify PDF proof directly
+qa360 verify ./proof.pdf
+```
+
+### Programmatic Verification
+
+```typescript
+import { verify } from 'crypto';
+import { EvidenceVault } from '@qa360/core';
+
+async function verifyProof(runId: string): Promise<boolean> {
+  const vault = await EvidenceVault.open('.qa360');
+  
+  // Get run and proof data
+  const run = await vault.getRun(runId);
+  if (!run || !run.signature_hex) {
+    throw new Error('Run not found or not signed');
+  }
+  
+  // Reconstruct proof JSON
+  const [gates, findings, artifacts] = await Promise.all([
+    vault.getGates(runId),
+    vault.getFindings(runId),
+    vault.getRunArtifacts(runId)
+  ]);
+  
+  const proofData = {
+    run_id: runId,
+    timestamp: run.started_at,
+    // ... reconstruct complete proof object
+  };
+  
+  // Verify signature
+  const publicKey = await loadPublicKey('.qa360/keys/public.pem');
+  const proofJson = JSON.stringify(proofData, null, 2);
+  const signature = Buffer.from(run.signature_hex, 'hex');
+  
+  return verify('ed25519', Buffer.from(proofJson), publicKey, signature);
+}
+```
+
+### Online Verification
+
+```bash
+# Generate verification URL
+qa360 verify --online <runId>
+# Output: https://verify.qa360.ai/proof/a1b2c3d4-e5f6-7890-abcd-ef1234567890
+
+# Embed QR code in proof
+qa360 generate-proof --qr-code <runId>
+```
+
+## Key Management
+
+### Key Generation
+
+```bash
+# Generate Ed25519 key pair
+qa360 keys generate --algorithm ed25519
+
+# Specify custom location
+qa360 keys generate --output ./custom-keys/
+
+# Generate with passphrase protection
+qa360 keys generate --passphrase
+```
+
+### Key Storage
+
+```
+.qa360/keys/
+├── private.pem          # Ed25519 private key (600 permissions)
+├── public.pem           # Ed25519 public key
+├── key.fingerprint      # Key fingerprint for identification
+└── key.metadata.json    # Key generation metadata
+```
+
+### Key Rotation
+
+```bash
+# Generate new key pair
+qa360 keys rotate
+
+# Backup old keys
+qa360 keys backup --output ./key-backup/
+
+# Update vault with new key
+qa360 keys update-vault
+```
+
+## Trust Score Calculation
+
+### Algorithm
+
+```typescript
+function calculateTrustScore(gates: GateRecord[], findings: FindingRecord[]): number {
+  let score = 100;
+  
+  // Gate failures reduce trust
+  gates.forEach(gate => {
+    switch (gate.status) {
+      case 'failed':
+        score -= 20;
+        break;
+      case 'warn':
+        score -= 5;
+        break;
+      case 'skipped':
+        score -= 2;
+        break;
+    }
+  });
+  
+  // Security findings reduce trust
+  findings.forEach(finding => {
+    switch (finding.severity) {
+      case 'critical':
+        score -= 25;
+        break;
+      case 'high':
+        score -= 15;
+        break;
+      case 'medium':
+        score -= 5;
+        break;
+      case 'low':
+        score -= 1;
+        break;
+    }
+  });
+  
+  // Performance budget violations
+  gates.forEach(gate => {
+    if (gate.budgets_json) {
+      const budgets = JSON.parse(gate.budgets_json);
+      const metrics = JSON.parse(gate.metrics_json || '{}');
+      
+      // Check budget violations
+      if (budgets.max_duration_ms && metrics.duration_ms > budgets.max_duration_ms) {
+        score -= 10;
+      }
+      if (budgets.min_coverage_percent && metrics.coverage_percent < budgets.min_coverage_percent) {
+        score -= 15;
+      }
+    }
+  });
+  
+  return Math.max(0, Math.min(100, score));
+}
+```
+
+### Trust Levels
+
+- **90-100%**: Excellent - Production ready
+- **80-89%**: Good - Minor issues, acceptable for release
+- **70-79%**: Fair - Some concerns, review required
+- **60-69%**: Poor - Significant issues, not recommended
+- **0-59%**: Critical - Major problems, do not release
+
+## Compliance & Auditing
+
+### Regulatory Requirements
+
+QA360 proofs support various compliance frameworks:
+
+- **SOC 2**: Evidence of security controls testing
+- **ISO 27001**: Information security management validation
+- **GDPR**: Data protection impact assessments
+- **HIPAA**: Healthcare application security verification
+- **PCI DSS**: Payment system security compliance
+
+### Audit Trail
+
+```bash
+# Generate audit report
+qa360 audit generate --from "2025-01-01" --to "2025-12-31"
+
+# Export compliance evidence
+qa360 audit export --standard soc2 --output ./compliance/
+
+# Verify audit trail integrity
+qa360 audit verify --all
+```
+
+### Evidence Package
+
+```
+compliance-package.zip
+├── executive-summary.pdf
+├── detailed-results/
+│   ├── security-testing.pdf
+│   ├── performance-testing.pdf
+│   └── accessibility-testing.pdf
+├── proofs/
+│   ├── proof-run-1.pdf
+│   ├── proof-run-2.pdf
+│   └── ...
+├── verification/
+│   ├── public-keys.pem
+│   ├── signatures.txt
+│   └── verification-instructions.md
+└── metadata/
+    ├── test-configurations.json
+    ├── environment-details.json
+    └── compliance-mapping.json
+```
+
+## Integration Examples
+
+### CI/CD Pipeline
+
+```yaml
+# GitHub Actions
+name: QA360 Testing with Proof Generation
+
+on: [push, pull_request]
+
+jobs:
+  qa360-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      
+      - name: Setup QA360
+        run: |
+          npm install -g @qa360/cli
+          qa360 doctor --fix
+      
+      - name: Run Tests with Proof
+        run: |
+          qa360 run --pack ./qa360-pack.yml \
+                    --run-key "ci-${{ github.run_id }}" \
+                    --sign
+        env:
+          QA360_SIGNING_KEY: ${{ secrets.QA360_SIGNING_KEY }}
+      
+      - name: Export Proof
+        if: always()
+        run: |
+          RUN_ID=$(qa360 history list --json --limit 1 | jq -r '.[0].id')
+          qa360 history export $RUN_ID --bundle ./proof-bundle.zip
+      
+      - name: Upload Proof
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: qa360-proof
+          path: proof-bundle.zip
+      
+      - name: Verify Proof
+        run: |
+          RUN_ID=$(qa360 history list --json --limit 1 | jq -r '.[0].id')
+          qa360 verify $RUN_ID
+```
+
+### Release Process
+
+```bash
+#!/bin/bash
+# release-with-proof.sh
+
+VERSION=$1
+RUN_KEY="release-$VERSION"
+
+echo "🚀 Starting QA360 testing for release $VERSION"
+
+# Run comprehensive tests
+qa360 run --pack ./release-pack.yml \
+          --run-key "$RUN_KEY" \
+          --sign
+
+# Get run ID
+RUN_ID=$(qa360 history list --json --limit 1 | jq -r '.[0].id')
+
+# Verify proof
+if qa360 verify $RUN_ID; then
+  echo "✅ Proof verified successfully"
+else
+  echo "❌ Proof verification failed"
+  exit 1
+fi
+
+# Pin release run
+qa360 history pin $RUN_ID
+
+# Export release evidence
+qa360 history export $RUN_ID --bundle "./releases/$VERSION-evidence.zip"
+
+echo "📦 Release $VERSION evidence package created"
+echo "🔗 Run ID: $RUN_ID"
+echo "📄 Proof bundle: ./releases/$VERSION-evidence.zip"
+```
+
+### Compliance Automation
+
+```bash
+#!/bin/bash
+# compliance-check.sh
+
+COMPLIANCE_STANDARD=$1  # soc2, iso27001, etc.
+PERIOD_START=$2
+PERIOD_END=$3
+
+echo "📋 Generating compliance report for $COMPLIANCE_STANDARD"
+
+# Get all runs in period
+qa360 history list --since "$PERIOD_START" \
+                   --json > runs-$PERIOD_START-$PERIOD_END.json
+
+# Verify all proofs
+FAILED_VERIFICATIONS=0
+while read -r run_id; do
+  if ! qa360 verify "$run_id"; then
+    echo "❌ Verification failed for run: $run_id"
+    ((FAILED_VERIFICATIONS++))
+  fi
+done < <(jq -r '.[].id' runs-$PERIOD_START-$PERIOD_END.json)
+
+if [ $FAILED_VERIFICATIONS -eq 0 ]; then
+  echo "✅ All proofs verified successfully"
+  
+  # Generate compliance package
+  qa360 audit generate --standard "$COMPLIANCE_STANDARD" \
+                       --from "$PERIOD_START" \
+                       --to "$PERIOD_END" \
+                       --output "./compliance/$COMPLIANCE_STANDARD-$(date +%Y%m%d).zip"
+  
+  echo "📦 Compliance package generated"
+else
+  echo "❌ $FAILED_VERIFICATIONS proof verification failures"
+  exit 1
+fi
+```
+
+## Security Considerations
+
+### Key Security
+
+- Store private keys with restricted permissions (600)
+- Use hardware security modules (HSM) for production
+- Implement key rotation policies
+- Backup keys securely and separately from proofs
+
+### Proof Integrity
+
+- Verify signatures before trusting proof data
+- Check proof timestamps for replay attacks
+- Validate pack hashes for configuration integrity
+- Cross-reference vault data with proof claims
+
+### Distribution Security
+
+- Use secure channels for proof distribution
+- Implement access controls for proof archives
+- Log all proof access and verification attempts
+- Consider proof encryption for sensitive environments
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Signature verification fails**
+   - Check key pair consistency
+   - Verify proof data integrity
+   - Ensure correct algorithm (Ed25519)
+
+2. **Missing proof artifacts**
+   - Check vault integrity
+   - Verify CAS file existence
+   - Run garbage collection analysis
+
+3. **Invalid trust scores**
+   - Review gate results and findings
+   - Check budget configurations
+   - Validate scoring algorithm
+
+### Diagnostic Commands
+
+```bash
+# Check key integrity
+qa360 keys verify
+
+# Validate proof structure
+qa360 verify --validate-only <runId>
+
+# Debug signature process
+qa360 verify --debug <runId>
+
+# Check vault consistency
+qa360 doctor --vault --proofs
+```
+
+## Best Practices
+
+### Proof Generation
+
+1. **Consistent environments**: Use reproducible test environments
+2. **Comprehensive coverage**: Include all relevant test gates
+3. **Meaningful run keys**: Use descriptive, unique identifiers
+4. **Regular verification**: Verify proofs immediately after generation
+
+### Key Management
+
+1. **Secure storage**: Protect private keys with appropriate access controls
+2. **Regular rotation**: Implement key rotation schedules
+3. **Backup strategy**: Maintain secure key backups
+4. **Access logging**: Log all key usage and access
+
+### Compliance
+
+1. **Documentation**: Maintain clear proof generation procedures
+2. **Retention policies**: Define proof retention and disposal policies
+3. **Regular audits**: Verify proof integrity and compliance
+4. **Training**: Ensure team understands proof verification process
+
+## Future Enhancements
+
+### Planned Features
+
+- **Multi-signature support**: Require multiple signatures for critical proofs
+- **Timestamping service**: RFC 3161 timestamp integration
+- **Blockchain anchoring**: Immutable proof registration
+- **Hardware security**: HSM and smart card integration
+- **Zero-knowledge proofs**: Privacy-preserving verification
+
+### Integration Roadmap
+
+- **PKI integration**: Enterprise certificate authority support
+- **Cloud KMS**: AWS KMS, Azure Key Vault, Google Cloud KMS
+- **Compliance frameworks**: Additional standards support
+- **Verification services**: Hosted verification endpoints