icdev 1.0.0__py3-none-any.whl

icdev/data/docs/features/phase-09-monitoring-observability.md

@@ -0,0 +1,230 @@
+# Phase 9 — Monitoring & Observability
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 9 |
+| Title | Monitoring & Observability |
+| Status | Implemented |
+| Priority | P1 |
+| Dependencies | Phase 7 (Security Scanning), Phase 8 (Self-Healing System) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+ICDEV-built applications deploy into Gov/DoD environments where operational visibility is not optional -- it is a compliance requirement. NIST 800-53 controls AU-6 (Audit Record Review), SI-4 (System Monitoring), and IR-5 (Incident Monitoring) mandate continuous monitoring of system health, security events, and performance metrics. Without a unified observability layer, operators are blind to degradation until users report outages.
+
+Gov/DoD environments typically run heterogeneous monitoring stacks. Some agencies use ELK (Elasticsearch, Logstash, Kibana), others use Splunk, and most require Prometheus/Grafana for metric collection. ICDEV must integrate with all of these, providing a single pane of glass that correlates logs, metrics, and alerts across the entire application stack regardless of which monitoring tools are deployed.
+
+The monitoring system also serves as the sensor layer for the Self-Healing System (Phase 8). Without reliable anomaly detection and alert correlation, self-healing cannot trigger. Monitoring must detect issues, correlate them into root causes, and hand off actionable intelligence to the remediation pipeline -- all while maintaining an immutable audit trail of every observation.
+
+---
+
+## 2. Goals
+
+1. Provide health check capabilities that validate HTTP endpoints, database connectivity, dependent service availability, and response time thresholds
+2. Integrate with ELK Stack and Splunk for log aggregation, parsing application logs for error patterns, warning accumulation, performance degradation, and security events
+3. Collect metrics from Prometheus/Grafana including CPU utilization, memory usage, request rate, latency percentiles (p50/p95/p99), error rate, and connection pool utilization
+4. Correlate related alerts into single root cause groups, distinguishing primary from secondary alerts and calculating severity roll-ups
+5. Detect anomalies versus baseline using statistical pattern detection (BM25 + cosine similarity for text matching, frequency analysis, time correlation)
+6. Feed actionable intelligence to the Self-Healing System (Phase 8) when monitoring detects patterns above confidence thresholds
+7. Generate human-readable monitoring reports with health status, log summaries, metric snapshots, and active alert counts
+8. Record all monitoring events in the append-only audit trail satisfying NIST AU-6, SI-4, and IR-5
+
+---
+
+## 3. Architecture
+
+```
++-----------------------------------------------------------+
+|              External Monitoring Stack                     |
+|                                                           |
+|  +--------+  +--------+  +-----------+  +--------+       |
+|  |  ELK   |  | Splunk |  | Prometheus|  | Grafana|       |
+|  | Stack  |  |        |  |           |  |        |       |
+|  +---+----+  +---+----+  +-----+-----+  +---+----+       |
+|      |           |              |            |            |
++------+-----------+--------------+------------+------------+
+       |           |              |            |
+       v           v              v            v
++-----------------------------------------------------------+
+|              ICDEV Monitoring Layer                        |
+|                                                           |
+|  +------------------+  +-------------------+              |
+|  | Log Analyzer     |  | Metric Collector  |              |
+|  | (ELK + Splunk)   |  | (Prometheus)      |              |
+|  +--------+---------+  +---------+---------+              |
+|           |                      |                        |
+|           v                      v                        |
+|  +------------------+  +-------------------+              |
+|  | Alert Correlator |  | Health Checker    |              |
+|  | (root cause)     |  | (HTTP + DB + deps)|              |
+|  +--------+---------+  +---------+---------+              |
+|           |                      |                        |
++-----------|----------------------|------------------------+
+            |                      |
+            v                      v
++-----------------------------------------------------------+
+|              Pattern Detection (Knowledge Base)           |
+|  BM25 + cosine similarity + frequency + time correlation  |
++----------------------------+------------------------------+
+                             |
+               +-------------+-------------+
+               |                           |
+               v                           v
++---------------------+    +---------------------------+
+| Self-Healing System |    | Monitoring Report         |
+| (Phase 8)           |    | Health + Logs + Metrics   |
++---------------------+    | + Alerts + Patterns       |
+                            +---------------------------+
+```
+
+### Alert Thresholds
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| CPU | > 70% | > 90% |
+| Memory | > 75% | > 90% |
+| Error rate | > 1% | > 5% |
+| p95 latency | > 500ms | > 2000ms |
+| Health check | 1 failure | 3 consecutive failures |
+
+---
+
+## 4. Requirements
+
+### 4.1 Health Checks
+
+#### REQ-09-001: HTTP Endpoint Health Checks
+The system SHALL perform HTTP endpoint health checks, expecting 200 OK within a configurable timeout, and record results in the `metric_snapshots` table.
+
+#### REQ-09-002: Dependent Service Checks
+The system SHALL verify database connectivity and dependent service availability as part of each health check cycle.
+
+#### REQ-09-003: Response Time Measurement
+The system SHALL measure response time for each health check endpoint and compare against configured thresholds (warning > 500ms, critical > 2000ms at p95).
+
+### 4.2 Log Analysis
+
+#### REQ-09-004: Multi-Source Log Integration
+The system SHALL connect to both ELK (Elasticsearch) and Splunk log aggregation platforms, with the source configurable via `args/monitoring_config.yaml`.
+
+#### REQ-09-005: Log Pattern Parsing
+The system SHALL parse application logs for error patterns (exceptions, stack traces), warning accumulation, performance degradation indicators, and security events (failed authentication, unusual access patterns).
+
+#### REQ-09-006: Anomaly Detection
+The system SHALL calculate error rate over configurable time windows and detect anomalies versus established baselines using statistical methods.
+
+### 4.3 Metric Collection
+
+#### REQ-09-007: Prometheus Metric Collection
+The system SHALL collect metrics from Prometheus/Grafana including CPU utilization, memory usage, request rate (req/sec), latency percentiles (p50, p95, p99), error rate (4xx, 5xx), database connection pool utilization, and queue depth.
+
+#### REQ-09-008: Metric Snapshot Storage
+The system SHALL store metric snapshots in the `metric_snapshots` table for historical trend analysis and baseline computation.
+
+### 4.4 Alert Correlation
+
+#### REQ-09-009: Root Cause Grouping
+The system SHALL correlate related alerts by grouping them by root cause (e.g., database failure causing cascading 5xx errors), determining primary versus secondary alerts.
+
+#### REQ-09-010: Severity Roll-Up
+The system SHALL calculate severity roll-up for correlated alert groups and record them in the `alerts` table.
+
+### 4.5 Reporting and Compliance
+
+#### REQ-09-011: Monitoring Report Generation
+The system SHALL generate human-readable monitoring reports showing health check results, log analysis summaries, metric snapshots, active alerts, and pattern matches.
+
+#### REQ-09-012: NIST Compliance Audit Trail
+All monitoring events SHALL be recorded in the append-only audit trail with event_type=monitoring.check, satisfying NIST 800-53 controls AU-6, SI-4, and IR-5.
+
+### 4.6 Edge Cases
+
+#### REQ-09-013: Graceful Degradation
+When the monitoring stack is unreachable, the system SHALL use cached data and report data staleness rather than failing completely.
+
+#### REQ-09-014: Metric Flapping Prevention
+The system SHALL require sustained threshold breaches (2+ minutes) before triggering alerts, preventing flapping from momentary spikes.
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `metric_snapshots` | Time-series metric data (CPU, memory, latency, error rate) per project per health check cycle |
+| `alerts` | Correlated alerts with severity, root cause grouping, primary/secondary classification, and status |
+| `knowledge_patterns` | Failure pattern signatures used for anomaly matching (shared with Phase 8) |
+| `audit_trail` | Append-only monitoring event records (NIST AU compliance) |
+| `deployments` | Deployment records used for post-deployment monitoring context |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/monitor/health_checker.py` | HTTP endpoint health checks, database connectivity, dependent service validation |
+| `tools/monitor/log_analyzer.py` | Log analysis via ELK/Splunk — error patterns, anomalies, security events |
+| `tools/monitor/metric_collector.py` | Prometheus/Grafana metric collection — CPU, memory, latency, error rate |
+| `tools/monitor/alert_correlator.py` | Alert correlation — root cause grouping, severity roll-up, primary/secondary classification |
+| `tools/knowledge/pattern_detector.py` | Statistical pattern matching against knowledge base (BM25, frequency, time correlation) |
+| `tools/audit/audit_logger.py` | Append-only audit trail recording for all monitoring events |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D4 | Statistical methods for pattern detection | No GPU available; BM25 + frequency analysis + time correlation are deterministic and air-gap safe |
+| D29 | SSE over WebSocket for dashboard live updates | Flask-native, simpler, no additional dependencies, unidirectional data flow is sufficient for monitoring |
+| D148 | Structured error hierarchy for new monitoring code | ICDevError base with Transient/Permanent subtypes allows monitoring to distinguish retryable from permanent failures |
+
+---
+
+## 8. Security Gate
+
+**Monitoring Gate:**
+- Health check failures exceeding 3 consecutive on any endpoint trigger critical alert
+- Error rate exceeding 5% triggers critical alert and self-healing evaluation
+- All monitoring events recorded in append-only audit trail (NIST AU-6, SI-4, IR-5)
+- Monitoring stack unreachable for > 5 minutes triggers staleness warning
+- Post-deployment: aggressive monitoring for first 30 minutes after each deployment
+
+---
+
+## 9. Commands
+
+```bash
+# Health checks
+python tools/monitor/health_checker.py --target "http://service:8080/health"
+
+# Log analysis
+python tools/monitor/log_analyzer.py --source elk --query "error"
+python tools/monitor/log_analyzer.py --source splunk --query "exception"
+
+# Metric collection
+python tools/monitor/metric_collector.py --source prometheus --project-id "proj-123"
+
+# Pattern detection
+python tools/knowledge/pattern_detector.py --log-data "/path/to/logs"
+
+# Full system health check
+python tools/testing/health_check.py                 # Human output
+python tools/testing/health_check.py --json           # JSON output
+
+# Heartbeat daemon (Phase 29 extension)
+python tools/monitor/heartbeat_daemon.py              # Foreground daemon (7 checks)
+python tools/monitor/heartbeat_daemon.py --once       # Single pass
+python tools/monitor/heartbeat_daemon.py --status --json  # Check statuses
+```
+
+**CUI // SP-CTI**

icdev/data/docs/features/phase-10-dashboard-web-ui.md

@@ -0,0 +1,218 @@
+# Phase 10 — Dashboard Web UI
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 10 |
+| Title | Dashboard Web UI |
+| Status | Implemented |
+| Priority | P1 |
+| Dependencies | Phase 8 (Self-Healing System), Phase 9 (Monitoring & Observability) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+ICDEV operates as a CLI-driven agentic system, which is effective for developers and engineers but inaccessible to program managers, ISSOs, contracting officers, and other stakeholders who need visibility into project status, compliance posture, and security findings. These users should not need to run Python scripts or parse JSON to understand whether a system is ready for ATO.
+
+Gov/DoD programs require audit-ready dashboards that display CUI markings on every page, provide role-based views for different stakeholders, and render entirely server-side to minimize the STIG attack surface. Client-side JavaScript frameworks (React, Angular, Vue) introduce thousands of transitive dependencies, each a potential vulnerability. A server-side rendered Flask application with Jinja2 templates produces auditable HTML with zero NPM dependencies.
+
+The dashboard must be "GI proof" -- meaning it must be usable by personnel with minimal technical background. Glossary tooltips explain compliance jargon, friendly timestamps replace ISO formats, breadcrumbs provide navigation context, and ARIA accessibility attributes ensure Section 508 compliance. The UX translation layer converts raw technical tool output into business-friendly displays without modifying the underlying deterministic tools.
+
+---
+
+## 2. Goals
+
+1. Provide a Flask-based server-side rendered web dashboard on port 5000 with CUI banners on every page, CSRF protection, and Content-Security-Policy headers
+2. Display project status, compliance posture (SSP, POAM, STIG, SBOM), security scan results, deployment history, and agent health in a unified interface
+3. Implement role-based views (admin, pm, developer, isso, co) via query parameter, providing progressive disclosure of information appropriate to each persona
+4. Deliver "GI-proof" UX with glossary tooltips, friendly timestamps, breadcrumbs, ARIA accessibility, skip-to-content links, notification toasts, and keyboard shortcuts
+5. Provide a REST API for programmatic access to all dashboard data
+6. Support SSE (Server-Sent Events) for near-real-time updates with connection status indicators and 3-second debounced batches
+7. Include zero-dependency SVG chart library (sparklines, line, bar, donut, gauge) and auto-enhancing table interactivity (search, sort, filter, CSV export)
+8. Implement API key authentication with SHA-256 hashed keys, Flask signed sessions, and 5-role RBAC (admin, pm, developer, isso, co)
+
+---
+
+## 3. Architecture
+
+```
++-----------------------------------------------------------+
+|                  Flask Dashboard (Port 5000)               |
+|                                                           |
+|  +-----------+  +------------------+  +---------------+   |
+|  | Jinja2    |  | REST API         |  | SSE Endpoint  |   |
+|  | Templates |  | /api/*           |  | /events       |   |
+|  | (SSR)     |  | (JSON)           |  | (live)        |   |
+|  +-----------+  +------------------+  +---------------+   |
+|       |                |                     |            |
+|  +-----------+  +------------------+  +---------------+   |
+|  | Static    |  | Auth Middleware   |  | UX Helpers    |   |
+|  | CSS/JS    |  | API Key + RBAC   |  | Glossary,     |   |
+|  | (minimal) |  | (SHA-256 hash)   |  | Timestamps,   |   |
+|  +-----------+  +------------------+  | Breadcrumbs   |   |
+|                                       +---------------+   |
++----------------------------+------------------------------+
+                             |
+                             v
++-----------------------------------------------------------+
+|              Data Layer (SQLite)                           |
+|                                                           |
+|  data/icdev.db    — Projects, compliance, agents, audit   |
+|  data/platform.db — Dashboard users, API keys, auth log   |
++-----------------------------------------------------------+
+```
+
+### Dashboard Pages
+
+| Route | Page | Description |
+|-------|------|-------------|
+| `/` | Home | Auto-notifications, project overview, compliance summary |
+| `/projects` | Projects | Project listing with friendly timestamps |
+| `/projects/<id>` | Project Detail | Role-based tab visibility (overview, compliance, security, deploy, audit) |
+| `/agents` | Agents | Agent registry with heartbeat age |
+| `/monitoring` | Monitoring | Status icons, accessibility, health checks |
+| `/wizard` | Getting Started | 3-question wizard recommending workflows |
+| `/quick-paths` | Quick Paths | Workflow templates + error recovery reference |
+| `/events` | Events | Real-time SSE event timeline |
+| `/query` | NLQ Query | Natural language compliance queries |
+| `/chat` | Agent Chat | SQLite-based agent chat interface |
+| `/activity` | Activity Feed | Merged audit + hook events feed |
+| `/login` | Login | API key authentication |
+| `/admin/users` | Admin | User/key management (admin role only) |
+
+---
+
+## 4. Requirements
+
+### 4.1 Core Dashboard
+
+#### REQ-10-001: Flask Server-Side Rendering
+The system SHALL use Flask with Jinja2 templates for server-side rendering, producing auditable HTML with no client-side JavaScript frameworks.
+
+#### REQ-10-002: CUI Markings
+Every dashboard page SHALL include top and bottom CUI banners ("CUI // SP-CTI"), configurable via `ICDEV_CUI_BANNER_ENABLED` environment variable.
+
+#### REQ-10-003: Project Status Views
+The system SHALL display project status with compliance posture (SSP, POAM, STIG, SBOM status), security scan results, deployment history, and audit trail scoped to each project.
+
+#### REQ-10-004: Agent Health Display
+The system SHALL display agent health in a grid view showing health status, heartbeat age, task queue depth, recent completions, and error rates per agent.
+
+### 4.2 Authentication and RBAC
+
+#### REQ-10-005: API Key Authentication
+The system SHALL authenticate users via API keys (SHA-256 hashed, stored in `dashboard_api_keys` table), with Flask signed sessions using `ICDEV_DASHBOARD_SECRET` environment variable.
+
+#### REQ-10-006: Role-Based Access Control
+The system SHALL enforce 5-role RBAC (admin, pm, developer, isso, co) with role-based page visibility and tab-level access control.
+
+### 4.3 UX and Accessibility
+
+#### REQ-10-007: GI-Proof UX
+The system SHALL provide glossary tooltips (via `data-glossary` HTML attributes), friendly timestamps, breadcrumb navigation, ARIA accessibility attributes, skip-to-content links, and Section 508 compliance.
+
+#### REQ-10-008: Keyboard Navigation
+The system SHALL support keyboard shortcuts using chord pattern (`g` + key) for page navigation, `?` for help modal, and `/` for search, with a 1.5-second chord window.
+
+### 4.4 Live Updates and Charts
+
+#### REQ-10-009: SSE Live Updates
+The system SHALL use Server-Sent Events for near-real-time dashboard updates, debounced to 3-second batches, with a connection status indicator.
+
+#### REQ-10-010: Zero-Dependency Charts
+The system SHALL provide SVG-based chart rendering (sparkline, line, bar, donut, gauge) with zero external JavaScript dependencies, WCAG accessible with `role="img"` and `aria-label` attributes.
+
+#### REQ-10-011: Table Interactivity
+The system SHALL auto-enhance all `.table-container` tables with search, sort, filter, and CSV export functionality via a self-contained JavaScript module.
+
+### 4.5 Security Hardening
+
+#### REQ-10-012: Security Headers
+The system SHALL set Content-Security-Policy headers, CSRF tokens on all forms, session timeout at 30 minutes, HTTPS-only in production, and rate limiting on API endpoints.
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `dashboard_users` | Dashboard user accounts with name, email, role, and status |
+| `dashboard_api_keys` | SHA-256 hashed API keys linked to users with expiration and scope |
+| `dashboard_auth_log` | Append-only authentication event log (login, logout, failed attempts) |
+| `dashboard_user_llm_keys` | BYOK LLM keys, Fernet AES-256 encrypted (per-user, per-provider) |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/dashboard/app.py` | Flask application factory, route registration, SSE endpoints |
+| `tools/dashboard/config.py` | Configuration loading from args/*.yaml |
+| `tools/dashboard/auth.py` | User creation, API key management, authentication middleware |
+| `tools/dashboard/static/js/charts.js` | Zero-dependency SVG chart library (sparkline, line, bar, donut, gauge) |
+| `tools/dashboard/static/js/tables.js` | Auto-enhancing table interactivity (search, sort, filter, CSV export) |
+| `tools/dashboard/static/js/tour.js` | First-visit onboarding tour with spotlight overlay |
+| `tools/dashboard/static/js/live.js` | SSE auto-refresh with connection status indicator |
+| `tools/dashboard/static/js/shortcuts.js` | Keyboard chord navigation (`g` + key pattern) |
+| `tools/dashboard/static/js/batch.js` | Batch operation workflows (ATO, Security, Compliance, Build) |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D3 | Flask over FastAPI | Simpler, fewer dependencies, auditable SSR, smaller STIG surface |
+| D29 | SSE over WebSocket for live updates | Flask-native, simpler, no additional dependencies, unidirectional sufficient |
+| D88 | UX Translation Layer wraps existing tools | Jinja2 filters + JS modules convert technical output to business-friendly display without modifying tools |
+| D89 | Glossary tooltip system uses `data-glossary` HTML attributes | No backend changes needed to add new terms |
+| D90 | Role-based views via `?role=` query parameter | No authentication required for progressive disclosure, RBAC layered separately |
+| D94 | SVG chart library is zero-dependency | No Chart.js/D3 needed, air-gap safe, WCAG accessible |
+| D95 | Table interactivity auto-enhances on page load | No per-table configuration needed, self-contained module |
+| D98 | Onboarding tour uses localStorage | No server-side user tracking, air-gap safe |
+| D99 | SSE live updates debounce to 3-second batches | Prevents API hammering while keeping dashboard near-real-time |
+| D169 | Dashboard auth is self-contained against icdev.db | Keeps dashboard independently deployable, not coupled to SaaS layer |
+| D171 | Session cookies use Flask built-in signed sessions | Secret from ICDEV_DASHBOARD_SECRET env var or auto-generated |
+| D172 | 5-role RBAC (admin, pm, developer, isso, co) | Maps to existing ROLE_VIEWS for page visibility |
+
+---
+
+## 8. Security Gate
+
+**Dashboard Security Gate:**
+- CUI banners present on every rendered page (configurable via `ICDEV_CUI_BANNER_ENABLED`)
+- CSRF token required on all form submissions
+- Content-Security-Policy headers set on all responses
+- No inline JavaScript permitted
+- Session timeout enforced at 30 minutes
+- API key authentication required for all non-login routes
+- BYOK keys encrypted with AES-256 (Fernet) before storage
+- Admin-only access to user management routes (`/admin/users`)
+
+---
+
+## 9. Commands
+
+```bash
+# Start web dashboard
+python tools/dashboard/app.py
+
+# Dashboard auth management
+python tools/dashboard/auth.py create-admin --email admin@icdev.local --name "Admin"
+python tools/dashboard/auth.py list-users
+
+# Environment variables
+# ICDEV_DASHBOARD_SECRET     — Flask session signing key
+# ICDEV_CUI_BANNER_ENABLED   — Toggle CUI banners (default: true)
+# ICDEV_BYOK_ENABLED          — Enable bring-your-own LLM keys (default: false)
+# ICDEV_BYOK_ENCRYPTION_KEY   — Fernet key for BYOK encryption
+```
+
+**CUI // SP-CTI**