qa360 2.1.6 → 2.2.0

package/docs/mcp-server.md

@@ -0,0 +1,651 @@
+# QA360 MCP Server
+
+## Overview
+
+The QA360 MCP (Model Context Protocol) Server enables seamless integration between QA360 and LLMs/IDEs. It exposes QA360's testing capabilities through a standardized protocol that can be consumed by Claude, VSCode, and other MCP-compatible clients.
+
+## Installation
+
+### Prerequisites
+
+- Node.js 18+ 
+- QA360 Core installed and configured
+- MCP-compatible client (Claude Desktop, VSCode with MCP extension, etc.)
+
+### Install from NPM
+
+```bash
+npm install -g @qa360/mcp-server
+```
+
+### Build from Source
+
+```bash
+git clone https://github.com/qa360/qa360-core
+cd qa360-core/packages/mcp-server-qa360
+npm install
+npm run build
+npm link
+```
+
+## Quick Start
+
+### 1. Start the MCP Server
+
+```bash
+# Basic usage (read-only tools)
+qa360-mcp
+
+# Enable all tools (requires explicit opt-in)
+qa360-mcp --allow-run --allow-secrets
+
+# Custom configuration
+qa360-mcp --allow-run --max-concurrency 5 --rate-limit 10
+```
+
+### 2. Configure Your MCP Client
+
+#### Claude Desktop
+
+Add to `~/.config/claude-desktop/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "qa360": {
+      "command": "qa360-mcp",
+      "args": ["--allow-run"],
+      "env": {
+        "QA360_LOG_LEVEL": "info"
+      }
+    }
+  }
+}
+```
+
+#### VSCode with MCP Extension
+
+Add to VSCode settings:
+
+```json
+{
+  "mcp.servers": [
+    {
+      "name": "qa360",
+      "command": "qa360-mcp",
+      "args": ["--allow-run", "--allow-secrets"],
+      "cwd": "${workspaceFolder}"
+    }
+  ]
+}
+```
+
+### 3. Test the Connection
+
+In your MCP client, try:
+
+```
+List my recent QA360 test runs
+```
+
+or
+
+```
+Generate a QA360 pack to test my web application for security issues
+```
+
+## Available Tools
+
+### Core Execution Tools
+
+#### `qa360.run`
+Execute a QA360 pack and return results.
+
+**Arguments:**
+- `packPath` (required): Path to pack file
+- `runKey` (optional): Unique key for idempotent runs
+- `strict` (optional): Enable strict validation (default: true)
+- `weights` (optional): Custom trust score weights
+
+**Example:**
+```json
+{
+  "packPath": "./my-pack.yml",
+  "runKey": "ci-build-123",
+  "strict": true
+}
+```
+
+**Requires:** `--allow-run` flag
+
+#### `qa360.report.sign`
+Sign a test report with Ed25519 signature.
+
+**Arguments:**
+- `runId` (required): Run ID to sign
+
+#### `qa360.verify`
+Verify proof integrity and signature.
+
+**Arguments:**
+- `runId` OR `proofPath` (required): Run ID or proof file path
+
+### History & Analytics Tools
+
+#### `qa360.history.list`
+List test runs with optional filters.
+
+**Arguments:**
+- `limit` (optional): Max runs to return (default: 20)
+- `status` (optional): Filter by status (passed|failed|error|cancelled|any)
+- `gate` (optional): Filter by gate name
+- `since` (optional): Show runs since date (ISO format)
+
+#### `qa360.history.get`
+Get detailed information about a specific run.
+
+**Arguments:**
+- `runId` (required): Run ID to retrieve
+
+#### `qa360.history.diff`
+Compare two test runs to identify changes.
+
+**Arguments:**
+- `runA` (required): First run ID
+- `runB` (required): Second run ID
+
+#### `qa360.history.trend`
+Analyze trends over time for performance and quality metrics.
+
+**Arguments:**
+- `gate` (optional): Focus on specific gate (perf|ui|api|a11y|sast|dast|deps|secrets)
+- `window` (optional): Number of runs to analyze (default: 20)
+
+#### `qa360.history.export`
+Export a complete run bundle with all artifacts.
+
+**Arguments:**
+- `runId` (required): Run ID to export
+- `bundlePath` (required): Output bundle path (.zip)
+
+#### `qa360.history.gc`
+Garbage collection to manage vault size.
+
+**Arguments:**
+- `keepLast` (optional): Number of recent runs to keep (default: 50)
+- `maxBytes` (optional): Maximum vault size (e.g., "5GB")
+- `dryRun` (optional): Show what would be deleted (default: false)
+
+#### `qa360.history.pin` / `qa360.history.unpin`
+Pin/unpin runs to protect from garbage collection.
+
+**Arguments:**
+- `runId` (required): Run ID to pin/unpin
+
+### Pack Management Tools
+
+#### `qa360.pack.ask`
+Generate a QA360 pack from natural language query.
+
+**Arguments:**
+- `query` (required): Natural language description
+- `context` (optional): Additional context
+
+**Example:**
+```json
+{
+  "query": "Test my web login form and check for security vulnerabilities",
+  "context": {
+    "targets": {
+      "url": "https://myapp.com/login"
+    }
+  }
+}
+```
+
+#### `qa360.pack.validate`
+Validate a QA360 pack configuration.
+
+**Arguments:**
+- `pack` OR `path` (required): YAML content or file path
+
+#### `qa360.pack.lint`
+Lint a QA360 pack and suggest improvements.
+
+**Arguments:**
+- `pack` OR `path` (required): YAML content or file path
+
+#### `qa360.pack.migrate`
+Migrate a QA360 pack to latest version.
+
+**Arguments:**
+- `pack` OR `path` (required): YAML content or file path
+
+### Secrets Management Tools
+
+#### `qa360.secrets.add`
+Add a secret to the secure vault.
+
+**Arguments:**
+- `name` (required): Secret name
+- `value` (required): Secret value
+- `description` (optional): Description
+
+**Requires:** `--allow-secrets` flag
+
+#### `qa360.secrets.list`
+List all secrets (names only, values redacted).
+
+**Requires:** `--allow-secrets` flag
+
+#### `qa360.secrets.remove`
+Remove a secret from the vault.
+
+**Arguments:**
+- `name` (required): Secret name to remove
+
+**Requires:** `--allow-secrets` flag
+
+#### `qa360.secrets.doctor`
+Check secrets vault health and integrity.
+
+**Requires:** `--allow-secrets` flag
+
+### Server & Vault Tools
+
+#### `qa360.serve.health`
+Check QA360 server health status.
+
+#### `qa360.serve.diag`
+Get detailed server diagnostics.
+
+#### `qa360.serve.metrics`
+Get server metrics and performance data.
+
+#### `qa360.vault.get_run`
+Get run data from the evidence vault.
+
+**Arguments:**
+- `runId` (required): Run ID to retrieve
+
+#### `qa360.vault.get_artifacts`
+Get artifacts for a run from the vault.
+
+**Arguments:**
+- `runId` (required): Run ID to get artifacts for
+
+### Triage & Support Tools
+
+#### `qa360.triage.explain_code`
+Explain error codes and suggest remediation actions.
+
+**Arguments:**
+- `code` (required): Error code (e.g., SG001, ZAP002, GL003)
+- `context` (optional): Additional context
+
+**Example:**
+```json
+{
+  "code": "SG001",
+  "context": {
+    "file": "/src/config.js",
+    "line": 42
+  }
+}
+```
+
+## Security & Configuration
+
+### Security Flags
+
+The MCP server implements multiple security layers:
+
+#### `--allow-run`
+Enables the `qa360.run` tool. **Required explicit opt-in** due to potential system impact.
+
+```bash
+qa360-mcp --allow-run
+```
+
+#### `--allow-secrets`
+Enables all `qa360.secrets.*` tools. **Required explicit opt-in** for security.
+
+```bash
+qa360-mcp --allow-secrets
+```
+
+### Rate Limiting
+
+Control request rate to prevent abuse:
+
+```bash
+qa360-mcp --rate-limit 10  # 10 requests per second
+```
+
+### Concurrency Control
+
+Limit concurrent heavy operations:
+
+```bash
+qa360-mcp --max-concurrency 3  # Max 3 concurrent runs
+```
+
+### Audit Logging
+
+All tool calls are logged for security and compliance:
+
+```bash
+qa360-mcp --audit-path ~/.qa360/mcp-audit.log
+```
+
+Disable audit logging:
+
+```bash
+qa360-mcp --no-audit
+```
+
+### Path Sandboxing
+
+The server restricts file access to allowed paths:
+
+- Default: Current working directory
+- Prevents path traversal attacks
+- Validates all file operations
+
+### Data Redaction
+
+Sensitive information is automatically redacted from:
+- Audit logs
+- Tool responses
+- Error messages
+
+Patterns redacted:
+- Bearer tokens
+- API keys
+- Passwords
+- Secrets
+
+## Configuration File
+
+Create `~/.qa360/mcp-config.json` for persistent settings:
+
+```json
+{
+  "allowRun": true,
+  "allowSecrets": false,
+  "maxConcurrency": 2,
+  "rateLimit": {
+    "requests": 5,
+    "window": 1
+  },
+  "sandbox": {
+    "allowedPaths": [
+      "/home/user/projects",
+      "/opt/qa360-packs"
+    ],
+    "requireQA360Dir": true
+  },
+  "audit": {
+    "enabled": true,
+    "logPath": "~/.qa360/mcp-audit.log"
+  }
+}
+```
+
+## Error Handling
+
+### Common Error Codes
+
+- `QA360_PERMISSION_DENIED`: Tool requires explicit permission flag
+- `QA360_RATE_LIMIT_EXCEEDED`: Too many requests
+- `QA360_CONCURRENCY_LIMIT`: Too many concurrent operations
+- `QA360_PATH_ACCESS_DENIED`: File path not allowed
+- `QA360_VAULT_NOT_FOUND`: No QA360 vault initialized
+- `QA360_RUN_NOT_FOUND`: Specified run ID doesn't exist
+- `QA360_INVALID_ARGUMENTS`: Tool arguments validation failed
+
+### Error Response Format
+
+```json
+{
+  "error": {
+    "code": "QA360_PERMISSION_DENIED",
+    "message": "qa360.run tool requires --allow-run flag",
+    "hint": "Restart the MCP server with --allow-run flag"
+  }
+}
+```
+
+## Integration Examples
+
+### CI/CD Pipeline Integration
+
+```yaml
+# GitHub Actions
+name: QA360 MCP Integration
+on: [push]
+
+jobs:
+  qa360-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      
+      - name: Setup QA360 MCP
+        run: |
+          npm install -g @qa360/mcp-server
+          qa360-mcp --allow-run &
+          sleep 2
+      
+      - name: Run Tests via MCP
+        run: |
+          # Use MCP client to trigger tests
+          mcp-call qa360.run '{"packPath": "./qa360-pack.yml", "runKey": "ci-${{ github.run_id }}"}'
+```
+
+### Claude Desktop Workflow
+
+1. **Generate Pack:**
+   ```
+   Create a QA360 pack to test my e-commerce checkout flow for security and performance issues
+   ```
+
+2. **Execute Tests:**
+   ```
+   Run the pack I just created and show me the results
+   ```
+
+3. **Analyze Results:**
+   ```
+   Compare my last two test runs and show me what changed
+   ```
+
+4. **Triage Issues:**
+   ```
+   Explain error code SG001 and tell me how to fix it
+   ```
+
+### VSCode Integration
+
+With the MCP extension installed:
+
+1. Open Command Palette (`Cmd+Shift+P`)
+2. Type "MCP: QA360"
+3. Select desired action
+4. Follow prompts for arguments
+
+## Troubleshooting
+
+### Server Won't Start
+
+```bash
+# Check if port is available
+lsof -i :stdio
+
+# Verify QA360 Core installation
+qa360 --version
+
+# Check permissions
+ls -la ~/.qa360/
+```
+
+### Tool Permission Errors
+
+```bash
+# Enable required permissions
+qa360-mcp --allow-run --allow-secrets
+
+# Check current configuration
+qa360-mcp --dry-run
+```
+
+### Connection Issues
+
+```bash
+# Test MCP server directly
+echo '{"jsonrpc":"2.0","method":"tools/list","id":1}' | qa360-mcp
+
+# Verify client configuration
+cat ~/.config/claude-desktop/claude_desktop_config.json
+```
+
+### Performance Issues
+
+```bash
+# Increase rate limits
+qa360-mcp --rate-limit 20 --max-concurrency 5
+
+# Check audit log size
+ls -lh ~/.qa360/mcp-audit.log
+
+# Rotate logs
+qa360-mcp --rotate-logs
+```
+
+### Vault Issues
+
+```bash
+# Initialize vault if missing
+qa360 run --url https://example.com --pack ./basic-pack.yml
+
+# Check vault health
+qa360-mcp --vault-doctor
+
+# Repair vault
+qa360 doctor --fix
+```
+
+## Best Practices
+
+### Security
+
+1. **Principle of Least Privilege:** Only enable required tools
+2. **Regular Auditing:** Monitor audit logs for suspicious activity
+3. **Path Restrictions:** Use specific allowed paths, not broad directories
+4. **Secrets Management:** Use `qa360.secrets.*` tools instead of environment variables
+5. **Rate Limiting:** Set appropriate limits for your use case
+
+### Performance
+
+1. **Concurrency Limits:** Balance performance vs resource usage
+2. **Vault Maintenance:** Regular garbage collection with `qa360.history.gc`
+3. **Pack Optimization:** Use `qa360.pack.lint` to optimize pack configurations
+4. **Monitoring:** Use `qa360.serve.metrics` to track performance
+
+### Reliability
+
+1. **Error Handling:** Always check tool responses for errors
+2. **Idempotent Operations:** Use `runKey` for repeatable test execution
+3. **Backup Strategy:** Pin important runs with `qa360.history.pin`
+4. **Health Checks:** Regular `qa360.serve.health` monitoring
+
+### Development Workflow
+
+1. **Pack Development:** Use `qa360.pack.ask` for initial generation, then `qa360.pack.validate` and `qa360.pack.lint`
+2. **Testing:** Use `runKey` for consistent test execution during development
+3. **Debugging:** Use `qa360.triage.explain_code` for error resolution
+4. **Analysis:** Use `qa360.history.trend` for performance tracking
+
+## API Reference
+
+### Tool Schema Validation
+
+All tools use JSON Schema validation. Invalid arguments will be rejected with detailed error messages.
+
+### Response Format
+
+All tools return responses in this format:
+
+```json
+{
+  "success": true,
+  "data": {
+    // Tool-specific response data
+  },
+  "metadata": {
+    "timestamp": 1640995200000,
+    "duration_ms": 1500,
+    "tool": "qa360.run"
+  }
+}
+```
+
+### Error Format
+
+```json
+{
+  "success": false,
+  "error": {
+    "code": "QA360_ERROR_CODE",
+    "message": "Human readable error message",
+    "hint": "Suggestion for resolution"
+  }
+}
+```
+
+## Contributing
+
+### Development Setup
+
+```bash
+git clone https://github.com/qa360/qa360-core
+cd qa360-core/packages/mcp-server-qa360
+npm install
+npm run dev
+```
+
+### Testing
+
+```bash
+npm test                    # Unit tests
+npm run test:integration    # Integration tests
+npm run test:coverage       # Coverage report
+```
+
+### Adding New Tools
+
+1. Create tool implementation in `src/tools/`
+2. Add to `ToolRegistry`
+3. Add JSON schema validation
+4. Write tests
+5. Update documentation
+
+## Changelog
+
+### v0.9.0-mcp
+- Initial MCP server implementation
+- 24 tools covering run, history, pack, secrets, serve, vault, and triage
+- Security features: rate limiting, sandboxing, audit logging
+- Comprehensive test suite
+- Full documentation
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Support
+
+- Documentation: https://qa360.ai/docs/mcp
+- Issues: https://github.com/qa360/qa360-core/issues
+- Discord: https://discord.gg/qa360
+- Email: support@qa360.ai