icdev 1.0.0__py3-none-any.whl

icdev/data/docs/features/phase-11-multi-agent-architecture.md

@@ -0,0 +1,272 @@
+# Phase 11 — Multi-Agent Architecture
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 11 |
+| Title | Multi-Agent Architecture |
+| Status | Implemented |
+| Priority | P0 |
+| Dependencies | Phase 1 (GOTCHA Framework), Phase 2 (Tools Layer) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+A single monolithic AI agent cannot effectively manage the full software development lifecycle for Gov/DoD applications. The cognitive complexity of simultaneously handling architecture, code generation, compliance, security, infrastructure, and monitoring exceeds what one agent can reliably accomplish. Each domain requires specialized knowledge, different confidence thresholds, and distinct failure modes. A security agent must be paranoid; a builder agent must be creative; a compliance agent must be pedantic.
+
+Furthermore, Gov/DoD environments demand separation of duties as a core security principle. NIST 800-53 AC-5 (Separation of Duties) requires that no single entity can both create and approve security-critical artifacts. A monolithic agent that generates code and then approves its own security scan violates this control. Multi-agent architecture enforces domain boundaries where the Security Agent can veto the Builder Agent's output and the Compliance Agent can block deployments.
+
+The A2A (Agent-to-Agent) protocol provides a standardized communication mechanism using JSON-RPC 2.0 over mutual TLS, ensuring that inter-agent communication within the K8s cluster is authenticated, encrypted, and auditable. Each agent publishes its capabilities via an Agent Card at `/.well-known/agent.json`, enabling dynamic discovery and routing without hardcoded dependencies.
+
+---
+
+## 2. Goals
+
+1. Deploy 15 specialized agents across 3 tiers (Core, Domain, Support) with clear responsibility boundaries and port assignments (8443-8458)
+2. Implement A2A protocol using JSON-RPC 2.0 over mutual TLS for authenticated, encrypted inter-agent communication within the K8s cluster
+3. Publish Agent Cards at `/.well-known/agent.json` per agent, enabling dynamic discovery of capabilities, accepted input modes, and output modes
+4. Manage the full task lifecycle (submitted, working, input-required, completed, failed) with state transitions recorded in `a2a_tasks` and `a2a_task_history` tables
+5. Route tasks from the Orchestrator to appropriate domain agents based on capability matching, with fallback and retry logic
+6. Enforce domain authority via a YAML-defined authority matrix where Security Agent holds hard veto on code/deps/infra and Compliance Agent holds hard veto on artifacts/deploy
+7. Support multi-agent workflows where the Orchestrator decomposes complex tasks into DAGs executed across multiple agents with artifact passing
+8. Run each agent as a separate K8s Deployment with resource limits, liveness/readiness probes, network policies, and HPA auto-scaling
+
+---
+
+## 3. Architecture
+
+```
++-----------------------------------------------------------+
+|                   CORE TIER                                |
+|                                                           |
+|  +-------------------+  +-------------------+             |
+|  | Orchestrator      |  | Architect         |             |
+|  | Port 8443         |  | Port 8444         |             |
+|  | Task routing,     |  | ATLAS/M-ATLAS,    |             |
+|  | workflow mgmt,    |  | system design     |             |
+|  | DAG execution     |  |                   |             |
+|  +--------+----------+  +-------------------+             |
++-----------|-----------------------------------------------+
+            |  A2A (JSON-RPC 2.0 / mTLS)
+            v
++-----------------------------------------------------------+
+|                   DOMAIN TIER                              |
+|                                                           |
+|  +----------+ +----------+ +----------+ +----------+     |
+|  | Builder  | |Compliance| | Security | |  Infra   |     |
+|  | 8445     | | 8446     | | 8447     | | 8448     |     |
+|  | TDD gen  | | ATO/SSP  | | SAST/    | | Terraform|     |
+|  |          | | POAM/CUI | | audit    | | Ansible  |     |
+|  +----------+ +----------+ +----------+ +----------+     |
+|                                                           |
+|  +----------+ +----------+ +----------+ +----------+     |
+|  |   MBSE   | | Modern.  | | Req.Anl  | | Supply   |     |
+|  |   8451   | | 8452     | | 8453     | | Chain    |     |
+|  | SysML/   | | Legacy   | | Intake/  | | 8454     |     |
+|  | DOORS    | | 7R/migr. | | gap/SAFe | | SBOM/CVE |     |
+|  +----------+ +----------+ +----------+ +----------+     |
+|                                                           |
+|  +----------+ +----------+ +----------+                  |
+|  | Simulate | |DevSecOps | | Gateway  |                  |
+|  |   8455   | | & ZTA    | | 8458     |                  |
+|  | Monte    | | 8457     | | Remote   |                  |
+|  | Carlo    | | Pipeline | | commands |                  |
+|  +----------+ +----------+ +----------+                  |
++-----------------------------------------------------------+
+            |
+            v
++-----------------------------------------------------------+
+|                   SUPPORT TIER                             |
+|                                                           |
+|  +-------------------+  +-------------------+             |
+|  | Knowledge         |  | Monitor           |             |
+|  | Port 8449         |  | Port 8450         |             |
+|  | Self-healing,     |  | Log analysis,     |             |
+|  | patterns, ML,     |  | metrics, alerts,  |             |
+|  | recommendations   |  | health checks     |             |
+|  +-------------------+  +-------------------+             |
++-----------------------------------------------------------+
+```
+
+### A2A Protocol
+
+```json
+{
+  "jsonrpc": "2.0",
+  "method": "tasks/send",
+  "params": {
+    "id": "<uuid>",
+    "message": {
+      "role": "user",
+      "parts": [{"type": "text", "text": "<task description>"}]
+    }
+  }
+}
+```
+
+### Task Lifecycle
+
+```
+submitted --> working --> completed
+                     \--> input-required --> working
+                     \--> failed
+```
+
+---
+
+## 4. Requirements
+
+### 4.1 Agent Registration and Discovery
+
+#### REQ-11-001: Agent Registration
+Each agent SHALL register on startup with agent ID, name, version, capabilities (skills list), endpoint URL, and health check URL, stored in the `agents` table.
+
+#### REQ-11-002: Agent Card Publication
+Each agent SHALL publish an Agent Card at `/.well-known/agent.json` per the A2A specification, declaring capabilities, accepted input modes, and output modes.
+
+#### REQ-11-003: Dynamic Discovery
+The Orchestrator SHALL discover agent capabilities by fetching and caching Agent Cards, enabling routing decisions without hardcoded agent dependencies.
+
+### 4.2 Communication Protocol
+
+#### REQ-11-004: A2A Protocol
+Inter-agent communication SHALL use JSON-RPC 2.0 over HTTPS with mutual TLS, authenticated via service mesh certificates within the K8s cluster.
+
+#### REQ-11-005: Task Lifecycle Management
+The system SHALL track task state transitions (submitted, working, input-required, completed, failed) in the `a2a_tasks` and `a2a_task_history` tables.
+
+#### REQ-11-006: Artifact Passing
+Multi-agent workflows SHALL pass artifacts between agents via the `a2a_task_artifacts` table, enabling sequential and parallel task execution with data dependencies.
+
+### 4.3 Task Routing and Orchestration
+
+#### REQ-11-007: Capability-Based Routing
+The Orchestrator SHALL analyze incoming tasks to determine required agent(s) and route to appropriate agents based on their declared capabilities.
+
+#### REQ-11-008: DAG-Based Workflow Execution
+Complex tasks SHALL be decomposed into directed acyclic graphs (DAGs) using `graphlib.TopologicalSorter` (D40), enabling parallel execution of independent subtasks.
+
+#### REQ-11-009: Domain Authority Vetoes
+The system SHALL enforce domain authority via a YAML-defined matrix (`args/agent_authority.yaml`) where Security Agent holds hard veto on code/deps/infra and Compliance Agent holds hard veto on artifacts/deploy.
+
+### 4.4 Health and Resilience
+
+#### REQ-11-010: Heartbeat Monitoring
+The system SHALL perform periodic heartbeat checks (every 30 seconds) on all agents, marking agents as offline after 3 consecutive failures.
+
+#### REQ-11-011: Circuit Breaker Protection
+The system SHALL implement circuit breaker protection (fail fast after 5 failures in 1 minute) per agent to prevent cascading failures.
+
+#### REQ-11-012: Error Recovery
+When an agent is offline, the system SHALL queue tasks for retry and route to backup agents when available.
+
+### 4.5 Deployment
+
+#### REQ-11-013: K8s Deployment
+Each agent SHALL run as a separate K8s Deployment with resource limits (256Mi-512Mi memory, 250m-500m CPU), liveness/readiness probes, NetworkPolicy isolation, and HPA auto-scaling.
+
+#### REQ-11-014: STIG-Hardened Containers
+All agent containers SHALL use the STIG-hardened base image with read-only root filesystem, dropped ALL capabilities, non-root user (UID 1000), and resource limits enforced.
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `agents` | Agent registry with ID, name, version, capabilities, endpoint, health status, last heartbeat |
+| `a2a_tasks` | Task records with ID, source agent, target agent, status, message, result |
+| `a2a_task_history` | Append-only task state transitions for audit trail |
+| `a2a_task_artifacts` | Artifacts passed between agents during multi-agent workflows |
+| `agent_token_usage` | Token consumption tracking per agent per model per project |
+| `agent_workflows` | DAG-based workflow definitions with subtask dependencies |
+| `agent_subtasks` | Individual subtasks within a workflow with agent assignment and status |
+| `agent_mailbox` | HMAC-SHA256 signed inter-agent message queue (D41) |
+| `agent_vetoes` | Domain authority veto records (append-only) |
+| `agent_memory` | Agent-scoped memory entries by (agent_id, project_id) with team-shared via _team |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/a2a/agent_registry.py` | Agent registration, health monitoring, discovery |
+| `tools/a2a/agent_client.py` | A2A client for sending tasks and fetching agent cards |
+| `tools/a2a/task.py` | A2A task model with lifecycle state management |
+| `tools/agent/team_orchestrator.py` | Task decomposition into DAGs and parallel execution |
+| `tools/agent/skill_router.py` | Route skills to healthy agents based on capability matching |
+| `tools/agent/collaboration.py` | Multi-agent collaboration patterns (reviewer, pair) |
+| `tools/agent/authority.py` | Domain authority checking and veto enforcement |
+| `tools/agent/mailbox.py` | HMAC-SHA256 signed agent inbox/outbox |
+| `tools/agent/agent_memory.py` | Scoped agent memory recall and storage |
+| `tools/agent/bedrock_client.py` | Bedrock LLM invocation with model fallback chain |
+| `tools/agent/token_tracker.py` | Token usage tracking and cost breakdown per agent |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D2 | Stdio for MCP (Claude Code); HTTPS+mTLS for A2A (K8s inter-agent) | MCP uses stdio for local tooling; A2A needs authenticated network transport within K8s |
+| D36 | `boto3 invoke_model()` + `ThreadPoolExecutor` for Bedrock | Matches existing subprocess/sqlite3 patterns; no asyncio needed |
+| D37 | Model fallback chain: Opus 4.6, Sonnet 4.5, Sonnet 3.5 with 30min health probe TTL | Ensures continuity when preferred model is unavailable |
+| D38 | Effort parameter mapped per agent role | Orchestrator=high, Builder=max, Monitor=low optimizes cost/quality per agent |
+| D40 | `graphlib.TopologicalSorter` (stdlib Python 3.9+) for task DAG | Air-gap safe, zero deps, cycle detection built-in |
+| D41 | SQLite-based agent mailbox with HMAC-SHA256 signing | Air-gap safe, append-only for audit, tamper-evident |
+| D42 | Domain authority defined in YAML matrix, vetoes recorded append-only | Configurable without code changes, auditable |
+| D43 | Agent memory scoped by (agent_id, project_id) | Prevents cross-project contamination; team-shared via agent_id='_team' |
+
+---
+
+## 8. Security Gate
+
+**Agent Management Gate:**
+- All inter-agent communication authenticated via mutual TLS within K8s cluster
+- Domain authority vetoes enforced (Security Agent: hard veto on code/deps/infra; Compliance Agent: hard veto on artifacts/deploy)
+- Agent mailbox messages HMAC-SHA256 signed for tamper detection
+- All task lifecycle transitions recorded in append-only audit trail (NIST AC-2, AU-12)
+- NetworkPolicy restricts inter-agent communication to declared dependencies only
+- All containers: read-only rootfs, drop ALL capabilities, non-root (UID 1000)
+
+---
+
+## 9. Commands
+
+```bash
+# Agent health and routing
+python tools/agent/skill_router.py --health
+python tools/agent/skill_router.py --route-skill "ssp_generate"
+
+# Task orchestration
+python tools/agent/team_orchestrator.py --decompose "task description" --project-id "proj-123"
+python tools/agent/team_orchestrator.py --execute --workflow-id "wf-123"
+
+# Domain authority
+python tools/agent/authority.py --check security-agent code_generation
+
+# Agent communication
+python tools/agent/mailbox.py --inbox --agent-id "builder-agent"
+python tools/agent/agent_memory.py --recall --agent-id "builder-agent" --project-id "proj-123"
+
+# Bedrock client
+python tools/agent/bedrock_client.py --probe
+python tools/agent/bedrock_client.py --prompt "text" --model opus --effort high
+python tools/agent/bedrock_client.py --prompt "text" --stream
+
+# Token tracking
+python tools/agent/token_tracker.py --action summary --project-id "proj-123"
+python tools/agent/token_tracker.py --action cost --project-id "proj-123"
+
+# Collaboration patterns
+python tools/agent/collaboration.py --pattern reviewer --project-id "proj-123"
+```
+
+**CUI // SP-CTI**

icdev/data/docs/features/phase-12-integration-testing.md

@@ -0,0 +1,228 @@
+# Phase 12 — Integration Testing
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 12 |
+| Title | Integration Testing |
+| Status | Implemented |
+| Priority | P0 |
+| Dependencies | Phase 5 (TDD Workflow), Phase 7 (Security Scanning), Phase 6 (Compliance Workflow) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+Individual test layers (unit tests, BDD scenarios, security scans) each provide valuable signal, but none alone can validate that an ICDEV-built application is production-ready for Gov/DoD deployment. A project can pass all unit tests while having critical STIG findings. It can pass security scans while missing CUI markings. It can pass compliance checks while having no E2E browser validation that the user interface actually works.
+
+Gov/DoD ATO (Authority to Operate) requires evidence that systems have been tested at multiple levels: unit, integration, system, and acceptance. NIST 800-53 SA-11 (Developer Testing and Evaluation) and SA-15 (Development Process, Standards, and Tools) mandate comprehensive test coverage with documented results. The test pipeline must produce auditable evidence for each layer, not just a pass/fail summary.
+
+The integration testing framework orchestrates a 9-step pipeline that runs every test layer in sequence, evaluates security and compliance gates, retries failures with resolution attempts, and produces a CUI-marked summary report suitable for ATO evidence packages. It adopts key patterns from the ADW (Agent Development Workflow): Pydantic data types for structured results, `parse_json()` for markdown-wrapped JSON, retry-with-resolution logic, fail-fast E2E execution, and `stdin=DEVNULL` for subprocess safety.
+
+---
+
+## 2. Goals
+
+1. Orchestrate a 9-step testing pipeline: py_compile, Ruff linting, pytest unit tests, behave BDD tests, Bandit SAST, Playwright E2E tests, vision validation, acceptance validation, and security+compliance gates
+2. Provide structured test results using Pydantic data types (TestResult, E2ETestResult, CheckResult) with consistent JSON serialization
+3. Implement retry-with-resolution logic (max 4 unit test retries, max 2 E2E retries) that stops early when no progress is detected between retries
+4. Execute E2E tests via native Playwright CLI (preferred) with MCP fallback, capturing screenshots and video recordings for audit evidence
+5. Evaluate security gates (0 HIGH SAST findings, 0 secrets detected) and compliance gates (CUI markings present, 0 CAT1 STIG findings, SBOM current)
+6. Generate CUI-marked test summary reports saved to `.tmp/test_runs/{run_id}/summary.md` with per-layer results
+7. Support optional vision validation using LLM-based screenshot analysis (CUI banner verification, error detection, content verification)
+8. Record all test events in the append-only audit trail satisfying NIST SA-11, SA-15, and CM-3
+
+---
+
+## 3. Architecture
+
+### 9-Step Testing Pipeline
+
+```
+Step 1: py_compile          (Python syntax validation)
+    |
+Step 2: Ruff                (Ultra-fast linting — replaces flake8+isort+black)
+    |
+Step 3: pytest              (Unit/integration tests with coverage)
+    |
+Step 4: behave/Gherkin      (BDD scenario tests for business requirements)
+    |
+Step 5: Bandit              (SAST security scan)
+    |
+Step 6: Playwright E2E      (Browser automation — native CLI or MCP fallback)
+    |
+Step 7: Vision Validation   (Optional LLM-based screenshot analysis)
+    |
+Step 8: Acceptance          (Deterministic acceptance criteria verification)
+    |
+Step 9: Security +          (CUI markings, STIG, secrets, SBOM gates)
+        Compliance Gates
+    |
+    v
+  Summary Report (CUI-marked, saved to .tmp/test_runs/{run_id}/)
+```
+
+### Key ADW Patterns
+
+| Pattern | Purpose |
+|---------|---------|
+| Pydantic data types | Structured, validated TestResult/E2ETestResult/CheckResult models |
+| parse_json() | Handles markdown-wrapped JSON output from Claude Code |
+| Retry with resolution | Max 4 unit / max 2 E2E retries; stop if no progress |
+| Fail-fast E2E | Stop on first E2E failure (sequential execution) |
+| Health check gating | Validate environment before test execution |
+| Dual logging | File (DEBUG) + console (INFO) for audit traceability |
+| Safe subprocess env | Filter environment variables for subprocess security |
+| stdin=DEVNULL | Prevent Claude Code hanging in subprocess |
+
+---
+
+## 4. Requirements
+
+### 4.1 Pipeline Orchestration
+
+#### REQ-12-001: 9-Step Sequential Pipeline
+The system SHALL execute a 9-step testing pipeline in sequence: py_compile, Ruff, pytest, behave, Bandit, Playwright E2E, vision validation, acceptance validation, security+compliance gates.
+
+#### REQ-12-002: Health Check Pre-Validation
+Before executing the test pipeline, the system SHALL validate environment variables, database connectivity, Python dependencies, tool availability, and MCP server configurations.
+
+#### REQ-12-003: Run ID Tracking
+Each test execution SHALL be assigned a unique run ID, with all artifacts (state, logs, screenshots, reports) stored under `.tmp/test_runs/{run_id}/`.
+
+### 4.2 Unit and BDD Testing
+
+#### REQ-12-004: pytest Execution
+The system SHALL run pytest with verbose output and coverage reporting, parsing results into structured `TestResult` objects.
+
+#### REQ-12-005: behave BDD Execution
+The system SHALL run behave with JSON output, parsing Gherkin scenario results into `TestResult` objects with `test_type="bdd"`.
+
+#### REQ-12-006: Retry with Resolution
+The system SHALL retry failed tests up to 4 times (unit) or 2 times (E2E), stopping early if no progress is detected between retries.
+
+### 4.3 E2E Testing
+
+#### REQ-12-007: Dual-Mode E2E Execution
+The system SHALL prefer native Playwright CLI (`npx playwright test`) when available and fall back to Playwright MCP via Claude Code CLI when native execution is not possible.
+
+#### REQ-12-008: Screenshot and Video Capture
+E2E tests SHALL capture screenshots to `.tmp/test_runs/{run_id}/screenshots/` and record video per configured viewport (1920x1080).
+
+#### REQ-12-009: CUI Banner Verification
+E2E tests SHALL verify CUI banners ("CUI // SP-CTI") are present on every tested page.
+
+### 4.4 Vision Validation
+
+#### REQ-12-010: Optional Vision Validation
+The system SHALL optionally validate screenshots using a vision LLM (Ollama LLaVA, Claude, GPT-4o) to verify CUI banners, detect error states, and confirm content rendering.
+
+### 4.5 Security and Compliance Gates
+
+#### REQ-12-011: Security Gate Evaluation
+The system SHALL evaluate security gates: 0 HIGH SAST findings (Bandit), 0 secrets detected, with thresholds configurable via `args/security_gates.yaml`.
+
+#### REQ-12-012: Compliance Gate Evaluation
+The system SHALL evaluate compliance gates: CUI markings present on all source files, 0 CAT1 STIG findings, SBOM current.
+
+#### REQ-12-013: Acceptance Validation
+The system SHALL verify acceptance criteria by mapping plan criteria to test evidence, checking rendered pages for error patterns (500, tracebacks, JS errors).
+
+### 4.6 Reporting
+
+#### REQ-12-014: CUI-Marked Summary Report
+The system SHALL generate a CUI-marked test summary report with per-layer results (pass/fail counts), gate evaluations, screenshots, and overall pass/fail determination.
+
+#### REQ-12-015: Audit Trail Recording
+All test events SHALL be recorded in the append-only audit trail with event_type=test.complete, satisfying NIST SA-11, SA-15, and CM-3.
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `audit_trail` | Append-only test completion events with run_id, counts, gate results |
+| `sast_findings` | Bandit SAST findings per project per scan |
+| `stig_findings` | STIG checklist findings (CAT1/CAT2/CAT3) per project |
+| `sbom_records` | SBOM generation records with currency tracking |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/testing/test_orchestrator.py` | Full 9-step pipeline orchestration with retry and gate evaluation |
+| `tools/testing/e2e_runner.py` | Native Playwright CLI + MCP fallback E2E test execution |
+| `tools/testing/health_check.py` | Pre-pipeline environment validation (7 checks) |
+| `tools/testing/data_types.py` | Pydantic models for TestResult, E2ETestResult, CheckResult |
+| `tools/testing/utils.py` | JSON parsing (parse_json), logging setup, safe subprocess environment |
+| `tools/testing/screenshot_validator.py` | Vision LLM screenshot analysis (CUI banners, errors, content) |
+| `tools/testing/platform_check.py` | Platform compatibility validation (OS, dependencies) |
+| `tools/security/sast_runner.py` | Bandit SAST execution for security gate |
+| `tools/security/secret_detector.py` | Secret detection for security gate |
+| `tools/compliance/stig_checker.py` | STIG findings evaluation for compliance gate |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D82 | Ollama LLaVA for air-gapped vision | Air-gap safe; vision is a message format concern, not a provider architecture concern |
+| D145 | Platform compatibility module centralizes OS detection | Single source of truth for platform-specific behavior in test execution |
+| D155 | Project-root conftest.py with shared fixtures | Centralizes test DB setup; prioritizes security-critical test paths |
+
+---
+
+## 8. Security Gate
+
+**Integration Testing Gate:**
+- All unit tests must pass (0 failures)
+- All BDD scenarios must pass (0 failures)
+- 0 HIGH or CRITICAL SAST findings (Bandit)
+- 0 secrets detected in source code
+- CUI markings present on all source files
+- 0 CAT1 STIG findings
+- SBOM current (generated within last build cycle)
+- Acceptance criteria: 0 failed criteria, 0 pages with error patterns (500, tracebacks, JS errors)
+
+---
+
+## 9. Commands
+
+```bash
+# Full test pipeline
+python tools/testing/test_orchestrator.py --project-dir /path/to/project
+python tools/testing/test_orchestrator.py --project-dir /path --skip-e2e --project-id "proj-123"
+
+# ICDEV platform tests
+pytest tests/ -v --tb=short
+
+# E2E tests
+python tools/testing/e2e_runner.py --discover
+python tools/testing/e2e_runner.py --run-all
+python tools/testing/e2e_runner.py --run-all --validate-screenshots
+python tools/testing/e2e_runner.py --run-all --validate-screenshots --vision-strict
+
+# Screenshot validation
+python tools/testing/screenshot_validator.py --check --json
+python tools/testing/screenshot_validator.py --image screenshot.png --assert "CUI banner is visible" --json
+python tools/testing/screenshot_validator.py --batch-dir .tmp/test_runs/screenshots/ --json
+
+# Health check
+python tools/testing/health_check.py
+python tools/testing/health_check.py --json
+
+# Platform check
+python tools/testing/platform_check.py
+python tools/testing/platform_check.py --json
+```
+
+**CUI // SP-CTI**