qa360 2.1.7 → 2.2.0

package/docs/mcp-tools.md

@@ -0,0 +1,1131 @@
+# QA360 MCP Tools Reference
+
+## Overview
+
+This document provides detailed specifications for all QA360 MCP tools, including argument schemas, return types, error codes, and usage examples.
+
+## Tool Categories
+
+- [Core Execution](#core-execution-tools)
+- [History & Analytics](#history--analytics-tools)
+- [Pack Management](#pack-management-tools)
+- [Secrets Management](#secrets-management-tools)
+- [Server & Vault](#server--vault-tools)
+- [Triage & Support](#triage--support-tools)
+
+---
+
+## Core Execution Tools
+
+### qa360.run
+
+Execute a QA360 pack and return comprehensive results.
+
+**Permission Required:** `--allow-run`
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "packPath": {
+      "type": "string",
+      "description": "Path to the QA360 pack file",
+      "pattern": "^[^\\0]+\\.(yml|yaml)$"
+    },
+    "runKey": {
+      "type": "string",
+      "description": "Unique key for idempotent runs",
+      "pattern": "^[a-zA-Z0-9:._-]+$",
+      "maxLength": 100
+    },
+    "strict": {
+      "type": "boolean",
+      "description": "Enable strict mode validation",
+      "default": true
+    },
+    "weights": {
+      "type": "object",
+      "description": "Custom weights for trust score calculation",
+      "additionalProperties": {
+        "type": "number",
+        "minimum": 0,
+        "maximum": 1
+      }
+    }
+  },
+  "required": ["packPath"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "runId": "string",
+  "status": "passed|failed|cancelled|error",
+  "trust": {
+    "score": "number (0-100)",
+    "weights": "object"
+  },
+  "proof": {
+    "pdf": {
+      "sha256": "string",
+      "path": "string"
+    }
+  },
+  "gates": [
+    {
+      "name": "string",
+      "status": "string",
+      "duration_ms": "number",
+      "key_metrics": "object"
+    }
+  ]
+}
+```
+
+**Example Usage:**
+```json
+{
+  "packPath": "./security-pack.yml",
+  "runKey": "ci:security:v1.2.0",
+  "strict": true,
+  "weights": {
+    "security": 0.4,
+    "performance": 0.3,
+    "ui": 0.3
+  }
+}
+```
+
+**Error Codes:**
+- `QA360_PERMISSION_DENIED`: Requires --allow-run flag
+- `QA360_PACK_NOT_FOUND`: Pack file doesn't exist
+- `QA360_PACK_INVALID`: Pack validation failed
+- `QA360_EXECUTION_FAILED`: Test execution error
+
+---
+
+### qa360.report.sign
+
+Sign a test report with Ed25519 cryptographic signature.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    }
+  },
+  "required": ["runId"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "signature": "string",
+  "algorithm": "ed25519",
+  "runId": "string"
+}
+```
+
+**Error Codes:**
+- `QA360_RUN_NOT_FOUND`: Run ID doesn't exist
+- `QA360_ALREADY_SIGNED`: Run already has signature
+- `QA360_SIGNING_FAILED`: Cryptographic signing error
+
+---
+
+### qa360.verify
+
+Verify proof integrity and cryptographic signature.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    },
+    "proofPath": {
+      "type": "string",
+      "description": "Path to proof file"
+    }
+  }
+}
+```
+
+**Return Type:**
+```json
+{
+  "verified": "boolean",
+  "algorithm": "string",
+  "message": "string",
+  "runId": "string"
+}
+```
+
+**Error Codes:**
+- `QA360_INVALID_ARGUMENTS`: Neither runId nor proofPath provided
+- `QA360_RUN_NOT_FOUND`: Run doesn't exist
+- `QA360_PROOF_NOT_FOUND`: Proof file doesn't exist
+- `QA360_VERIFICATION_FAILED`: Signature verification failed
+
+---
+
+## History & Analytics Tools
+
+### qa360.history.list
+
+List test runs with comprehensive filtering options.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "limit": {
+      "type": "integer",
+      "minimum": 1,
+      "maximum": 100,
+      "default": 20
+    },
+    "status": {
+      "type": "string",
+      "enum": ["passed", "failed", "error", "cancelled", "any"],
+      "default": "any"
+    },
+    "gate": {
+      "type": "string",
+      "pattern": "^[a-zA-Z0-9_-]+$"
+    },
+    "since": {
+      "type": "string",
+      "format": "date-time"
+    }
+  }
+}
+```
+
+**Return Type:**
+```json
+{
+  "runs": [
+    {
+      "id": "string",
+      "started_at": "number",
+      "status": "string",
+      "trust_score": "number",
+      "gates_summary": "object"
+    }
+  ]
+}
+```
+
+**Example Usage:**
+```json
+{
+  "limit": 10,
+  "status": "failed",
+  "gate": "security",
+  "since": "2025-01-01T00:00:00Z"
+}
+```
+
+---
+
+### qa360.history.get
+
+Get detailed information about a specific test run.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    }
+  },
+  "required": ["runId"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "run": {
+    "id": "string",
+    "started_at": "number",
+    "ended_at": "number",
+    "status": "string",
+    "trust_score": "number",
+    "pack_path": "string",
+    "pack_hash": "string",
+    "run_key": "string"
+  },
+  "gates": [
+    {
+      "name": "string",
+      "status": "string",
+      "duration_ms": "number",
+      "metrics": "object"
+    }
+  ],
+  "findings": [
+    {
+      "gate": "string",
+      "severity": "string",
+      "rule": "string",
+      "location": "string",
+      "message": "string"
+    }
+  ],
+  "artifacts": [
+    {
+      "label": "string",
+      "sha256": "string",
+      "mime_type": "string",
+      "size_bytes": "number"
+    }
+  ]
+}
+```
+
+---
+
+### qa360.history.diff
+
+Compare two test runs to identify changes and regressions.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runA": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    },
+    "runB": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    }
+  },
+  "required": ["runA", "runB"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "delta": {
+    "trust": "number",
+    "gates": {
+      "gate_name": {
+        "status_change": "string",
+        "duration_change": "number"
+      }
+    },
+    "findings": {
+      "added": ["array"],
+      "removed": ["array"],
+      "regressed": ["array"]
+    }
+  }
+}
+```
+
+---
+
+### qa360.history.trend
+
+Analyze trends over time for performance and quality metrics.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "gate": {
+      "type": "string",
+      "enum": ["perf", "ui", "api", "a11y", "sast", "dast", "deps", "secrets"]
+    },
+    "window": {
+      "type": "integer",
+      "minimum": 5,
+      "maximum": 200,
+      "default": 20
+    }
+  }
+}
+```
+
+**Return Type:**
+```json
+{
+  "series": [
+    {
+      "timestamp": "number",
+      "value": "number",
+      "metadata": "object"
+    }
+  ],
+  "summary": {
+    "average": "number",
+    "trend": "up|down|stable",
+    "window_size": "number"
+  }
+}
+```
+
+---
+
+### qa360.history.export
+
+Export a complete run bundle with all artifacts and metadata.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    },
+    "bundlePath": {
+      "type": "string",
+      "pattern": "^[^\\0]+\\.zip$"
+    }
+  },
+  "required": ["runId", "bundlePath"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "bundlePath": "string",
+  "size_bytes": "number",
+  "contents": ["array of strings"]
+}
+```
+
+---
+
+### qa360.history.gc
+
+Garbage collection to manage vault size and remove orphaned data.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "keepLast": {
+      "type": "integer",
+      "minimum": 1,
+      "maximum": 1000,
+      "default": 50
+    },
+    "maxBytes": {
+      "type": "string",
+      "pattern": "^\\d+(\\.\\d+)?(B|KB|MB|GB|TB)$"
+    },
+    "dryRun": {
+      "type": "boolean",
+      "default": false
+    }
+  }
+}
+```
+
+**Return Type:**
+```json
+{
+  "removed": {
+    "runs": "number",
+    "artifacts": "number",
+    "bytesFreed": "number"
+  },
+  "dryRun": "boolean"
+}
+```
+
+---
+
+### qa360.history.pin / qa360.history.unpin
+
+Pin or unpin runs to protect them from garbage collection.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    }
+  },
+  "required": ["runId"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "success": "boolean",
+  "runId": "string",
+  "pinned": "boolean",
+  "message": "string"
+}
+```
+
+---
+
+## Pack Management Tools
+
+### qa360.pack.ask
+
+Generate a QA360 pack from natural language query using AI.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "query": {
+      "type": "string",
+      "minLength": 10,
+      "maxLength": 1000
+    },
+    "context": {
+      "type": "object",
+      "properties": {
+        "targets": "object"
+      }
+    }
+  },
+  "required": ["query"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "pack": "string (YAML)",
+  "hints": {
+    "tools_needed": ["array of strings"],
+    "estimated_duration": "string",
+    "complexity": "simple|medium|complex"
+  }
+}
+```
+
+**Example Usage:**
+```json
+{
+  "query": "Test my e-commerce checkout flow for security vulnerabilities and performance under load",
+  "context": {
+    "targets": {
+      "url": "https://shop.example.com",
+      "checkout_url": "https://shop.example.com/checkout"
+    }
+  }
+}
+```
+
+---
+
+### qa360.pack.validate
+
+Validate a QA360 pack configuration against the schema.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "pack": {
+      "type": "string",
+      "description": "YAML pack content"
+    },
+    "path": {
+      "type": "string",
+      "description": "Path to pack file"
+    }
+  }
+}
+```
+
+**Return Type:**
+```json
+{
+  "valid": "boolean",
+  "errors": [
+    {
+      "code": "string",
+      "message": "string",
+      "line": "number",
+      "column": "number"
+    }
+  ],
+  "warnings": [
+    {
+      "code": "string",
+      "message": "string",
+      "line": "number",
+      "column": "number"
+    }
+  ]
+}
+```
+
+**Validation Error Codes:**
+- `PACK_EMPTY`: Pack file is empty or invalid YAML
+- `PACK_NO_NAME`: Missing required name field
+- `PACK_NO_VERSION`: Missing required version field
+- `PACK_NO_GATES`: Missing or invalid gates configuration
+- `GATE_INVALID`: Gate configuration is not an object
+- `GATE_NO_ADAPTER`: Gate missing adapter specification
+- `GATE_UNKNOWN_ADAPTER`: Unknown adapter type
+- `BUDGET_INVALID_DURATION`: Invalid budget duration value
+
+---
+
+### qa360.pack.lint
+
+Lint a QA360 pack and suggest improvements and best practices.
+
+**Arguments Schema:** Same as `qa360.pack.validate`
+
+**Return Type:** Same as `qa360.pack.validate` plus:
+```json
+{
+  "auto_fixes": ["array of suggested fixes"]
+}
+```
+
+**Lint Warning Codes:**
+- `PACK_NO_DESCRIPTION`: Consider adding description field
+- `PACK_NO_METADATA`: Consider adding metadata section
+- `PACK_TOO_MANY_GATES`: Too many gates, consider splitting
+- `GATE_NAME_SPACES`: Gate names should use underscores/hyphens
+
+---
+
+### qa360.pack.migrate
+
+Migrate a QA360 pack to the latest version format.
+
+**Arguments Schema:** Same as `qa360.pack.validate`
+
+**Return Type:** Same as `qa360.pack.lint`
+
+**Migration Actions:**
+- Update deprecated adapter names
+- Add missing required fields
+- Modernize configuration format
+- Apply current best practices
+
+---
+
+## Secrets Management Tools
+
+**All secrets tools require `--allow-secrets` flag**
+
+### qa360.secrets.add
+
+Add a secret to the encrypted vault.
+
+**Permission Required:** `--allow-secrets`
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "pattern": "^[a-zA-Z0-9_-]+$",
+      "minLength": 1,
+      "maxLength": 100
+    },
+    "value": {
+      "type": "string",
+      "minLength": 1,
+      "maxLength": 10000
+    },
+    "description": {
+      "type": "string",
+      "maxLength": 500
+    }
+  },
+  "required": ["name", "value"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "success": "boolean",
+  "name": "string",
+  "message": "string",
+  "created_at": "number"
+}
+```
+
+---
+
+### qa360.secrets.list
+
+List all secrets with names and metadata (values redacted).
+
+**Permission Required:** `--allow-secrets`
+
+**Return Type:**
+```json
+{
+  "secrets": [
+    {
+      "name": "string",
+      "description": "string",
+      "created_at": "number",
+      "last_used": "number"
+    }
+  ]
+}
+```
+
+---
+
+### qa360.secrets.remove
+
+Remove a secret from the vault.
+
+**Permission Required:** `--allow-secrets`
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "pattern": "^[a-zA-Z0-9_-]+$"
+    }
+  },
+  "required": ["name"]
+}
+```
+
+---
+
+### qa360.secrets.doctor
+
+Check secrets vault health and provide recommendations.
+
+**Permission Required:** `--allow-secrets`
+
+**Return Type:**
+```json
+{
+  "health": "healthy|warning|error",
+  "secret_count": "number",
+  "issues": ["array of issues"],
+  "recommendations": ["array of recommendations"],
+  "vault_location": "string",
+  "encryption": "string"
+}
+```
+
+---
+
+## Server & Vault Tools
+
+### qa360.serve.health
+
+Check QA360 server health status.
+
+**Return Type:**
+```json
+{
+  "status": "healthy|degraded|unhealthy",
+  "server": "running|not_running",
+  "url": "string",
+  "version": "string",
+  "uptime": "number"
+}
+```
+
+---
+
+### qa360.serve.diag
+
+Get detailed server diagnostics and system information.
+
+**Return Type:**
+```json
+{
+  "health": "healthy|degraded|unhealthy",
+  "version": "string",
+  "uptime_ms": "number",
+  "metrics": {
+    "memory": "object",
+    "process": "object",
+    "mcp": "object"
+  },
+  "vault_stats": "object"
+}
+```
+
+---
+
+### qa360.serve.metrics
+
+Get server metrics and performance data.
+
+**Return Type:**
+```json
+{
+  "timestamp": "number",
+  "server_type": "string",
+  "uptime_ms": "number",
+  "memory_usage": "object",
+  "mcp_metrics": {
+    "tools_registered": "number",
+    "total_calls": "number",
+    "success_rate": "number",
+    "top_tools": "array"
+  },
+  "security": "object"
+}
+```
+
+---
+
+### qa360.vault.get_run
+
+Get run data from the evidence vault (read-only).
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "runId": {
+      "type": "string",
+      "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$"
+    }
+  },
+  "required": ["runId"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "run": "object (run details)",
+  "gates": "array",
+  "findings": "array (redacted)",
+  "metadata": {
+    "total_gates": "number",
+    "total_findings": "number",
+    "passed_gates": "number",
+    "failed_gates": "number",
+    "critical_findings": "number",
+    "high_findings": "number"
+  }
+}
+```
+
+---
+
+### qa360.vault.get_artifacts
+
+Get artifacts metadata for a run (content not included for security).
+
+**Arguments Schema:** Same as `qa360.vault.get_run`
+
+**Return Type:**
+```json
+{
+  "run_id": "string",
+  "total_artifacts": "number",
+  "artifacts": [
+    {
+      "label": "string",
+      "sha256": "string",
+      "mime_type": "string",
+      "size_bytes": "number",
+      "original_name": "string",
+      "cas_path": "string",
+      "created_at": "number",
+      "access_note": "string"
+    }
+  ],
+  "summary": {
+    "total_size_bytes": "number",
+    "mime_types": "array",
+    "labels": "array",
+    "oldest_artifact": "number",
+    "newest_artifact": "number"
+  }
+}
+```
+
+---
+
+## Triage & Support Tools
+
+### qa360.triage.explain_code
+
+Explain error codes and provide remediation guidance.
+
+**Arguments Schema:**
+```json
+{
+  "type": "object",
+  "properties": {
+    "code": {
+      "type": "string",
+      "pattern": "^[A-Z]{2,4}\\d{3,4}$",
+      "description": "Error code (e.g., SG001, ZAP002)"
+    },
+    "context": {
+      "type": "object",
+      "description": "Additional context for explanation"
+    }
+  },
+  "required": ["code"]
+}
+```
+
+**Return Type:**
+```json
+{
+  "explanation": {
+    "cause": "string",
+    "severity": "low|medium|high|critical",
+    "category": "string"
+  },
+  "actions": [
+    {
+      "type": "fix|investigate|ignore",
+      "description": "string",
+      "command": "string"
+    }
+  ],
+  "references": [
+    {
+      "title": "string",
+      "url": "string"
+    }
+  ]
+}
+```
+
+**Supported Error Code Prefixes:**
+- `SG`: Semgrep (Static Analysis Security Testing)
+- `ZAP`: OWASP ZAP (Dynamic Application Security Testing)
+- `GL`: Gitleaks (Secret Detection)
+- `OSV`: OSV Scanner (Dependency Vulnerabilities)
+- `PA`: Playwright (UI Testing)
+- `K6`/`PU`: K6/Performance (Load Testing)
+
+**Example Usage:**
+```json
+{
+  "code": "SG001",
+  "context": {
+    "file": "/src/config.js",
+    "line": 42,
+    "rule": "hardcoded-secret"
+  }
+}
+```
+
+---
+
+## Common Patterns
+
+### Error Handling
+
+All tools return errors in this standardized format:
+
+```json
+{
+  "error": {
+    "code": "QA360_ERROR_CODE",
+    "message": "Human readable description",
+    "hint": "Suggested resolution"
+  }
+}
+```
+
+### Pagination
+
+Tools that return lists support pagination:
+
+```json
+{
+  "limit": 20,
+  "offset": 0,
+  "total": 150,
+  "has_more": true
+}
+```
+
+### Filtering
+
+Many tools support filtering with these common patterns:
+
+```json
+{
+  "status": "passed|failed|error|cancelled|any",
+  "since": "ISO 8601 date string",
+  "until": "ISO 8601 date string",
+  "gate": "gate name",
+  "severity": "low|medium|high|critical"
+}
+```
+
+### Redaction
+
+Sensitive data is automatically redacted in responses:
+- Signatures → `***REDACTED***`
+- Secrets → `***`
+- Tokens → `Bearer ***`
+- Passwords → `***`
+
+---
+
+## Rate Limiting
+
+All tools are subject to rate limiting:
+
+- Default: 5 requests per second
+- Configurable via `--rate-limit` flag
+- Heavy operations (run, export, gc) have additional concurrency limits
+- Rate limit headers included in responses
+
+---
+
+## Security Considerations
+
+### Input Validation
+
+- All arguments validated against JSON Schema
+- Path traversal prevention
+- SQL injection protection
+- Command injection prevention
+
+### Output Sanitization
+
+- Automatic PII redaction
+- Sensitive field masking
+- Safe error messages
+- Audit trail logging
+
+### Access Control
+
+- Tool-level permissions (`--allow-run`, `--allow-secrets`)
+- Path sandboxing
+- Concurrency limiting
+- Rate limiting
+
+---
+
+## Performance Notes
+
+### Heavy Operations
+
+These tools may take longer and consume more resources:
+- `qa360.run`: Executes full test suite
+- `qa360.history.export`: Creates ZIP bundles
+- `qa360.history.gc`: Processes large datasets
+- `qa360.pack.ask`: AI-powered generation
+
+### Optimization Tips
+
+- Use `limit` parameters to reduce response size
+- Implement client-side caching for static data
+- Use `dryRun` for preview operations
+- Pin important runs before garbage collection
+
+---
+
+## Integration Examples
+
+### Batch Operations
+
+```javascript
+// Get recent failed runs and explain their errors
+const failedRuns = await mcp.call('qa360.history.list', {
+  status: 'failed',
+  limit: 5
+});
+
+for (const run of failedRuns.runs) {
+  const details = await mcp.call('qa360.history.get', {
+    runId: run.id
+  });
+  
+  for (const finding of details.findings) {
+    if (finding.rule.match(/^[A-Z]{2,4}\d{3,4}$/)) {
+      const explanation = await mcp.call('qa360.triage.explain_code', {
+        code: finding.rule,
+        context: { location: finding.location }
+      });
+      console.log(`${finding.rule}: ${explanation.explanation.cause}`);
+    }
+  }
+}
+```
+
+### Workflow Automation
+
+```javascript
+// Complete testing workflow
+async function runQualityCheck(packPath, runKey) {
+  // 1. Validate pack
+  const validation = await mcp.call('qa360.pack.validate', { 
+    path: packPath 
+  });
+  
+  if (!validation.valid) {
+    throw new Error(`Pack validation failed: ${validation.errors[0].message}`);
+  }
+  
+  // 2. Run tests
+  const result = await mcp.call('qa360.run', { 
+    packPath, 
+    runKey,
+    strict: true 
+  });
+  
+  // 3. Sign results
+  await mcp.call('qa360.report.sign', { 
+    runId: result.runId 
+  });
+  
+  // 4. Export for archival
+  await mcp.call('qa360.history.export', {
+    runId: result.runId,
+    bundlePath: `./reports/${runKey}.zip`
+  });
+  
+  return result;
+}
+```
+
+This comprehensive reference covers all available MCP tools with their complete specifications, usage patterns, and integration examples.