icdev 1.0.0__py3-none-any.whl

icdev/data/goals/continuous_harmonization.md

@@ -0,0 +1,140 @@
+# [TEMPLATE: CUI // SP-CTI]
+# Goal: Continuous Harmonization — CLI Consistency & Code Quality
+
+> Ensure all ICDEV tools follow consistent CLI patterns, naming conventions,
+> and output formats so developers can trust every tool behaves predictably.
+
+---
+
+## Why
+
+ICDEV has 200+ tools built across 45 phases. Without continuous enforcement,
+conventions drift: some tools use `--project`, others `--project-id`; some
+support `--json`, others don't; some hardcode DB paths, others use env vars.
+Every inconsistency is a papercut that slows adoption and erodes trust.
+
+Harmonization is not a one-time task. It is a **continuous quality signal**
+that must be detected, scored, and resolved through the Innovation Engine.
+
+---
+
+## Standards (Enforced)
+
+### 1. CLI Flag Naming
+| Pattern | Standard | Example |
+|---------|----------|---------|
+| Project identifier | `--project-id` | `--project-id "proj-123"` |
+| Project directory | `--project-dir` | `--project-dir /path/to/code` |
+| Project file path | `--project-path` | `--project-path /path/file` |
+| JSON output | `--json` | `--json` (always `action="store_true"`) |
+| Human output | `--human` | `--human` (colored terminal) |
+| Database path | `--db-path` | `--db-path /path/to/db` |
+
+**Backward compat:** When renaming `--project` to `--project-id`, keep
+`--project` as a deprecated alias via `dest="project_id"`.
+
+### 2. DB Path Resolution
+All tools MUST use the centralized utility:
+```python
+from tools.compat.db_utils import get_icdev_db_path
+DB_PATH = get_icdev_db_path()  # env var > explicit > default
+```
+
+Never hardcode `Path(__file__).resolve().parent.parent.parent / "data" / "icdev.db"`.
+
+### 3. JSON Output Contract
+Every CLI tool with `argparse` MUST support `--json`:
+```python
+parser.add_argument("--json", action="store_true", help="JSON output")
+# ...
+if args.json:
+    print(json.dumps(result, indent=2, default=str))
+```
+
+JSON output MUST be a dict with at minimum: `{"status": "ok"|"error", ...}`.
+
+### 4. CUI Headers
+Every `.py` file MUST start with:
+```python
+#!/usr/bin/env python3
+# CUI // SP-CTI
+```
+
+### 5. Error Handling
+CLI tools MUST exit with code 0 on success, non-zero on failure.
+Errors in JSON mode: `{"status": "error", "error": "message"}`.
+
+---
+
+## Detection (Automated)
+
+### Governance Validator
+`python tools/testing/claude_dir_validator.py --json` runs 6+ checks
+including CLI harmonization. Blocked by `claude_config_alignment` gate.
+
+### Introspective Analyzer
+The Innovation Engine's introspective analyzer (`tools/innovation/introspective_analyzer.py`)
+scans for CLI pattern drift every 12 hours and generates innovation signals
+with category `cli_harmonization`.
+
+### Signals Generated
+| Signal | Category | Auto-Triage |
+|--------|----------|-------------|
+| Tool missing --json flag | cli_harmonization | yes |
+| Tool using --project instead of --project-id | cli_harmonization | yes |
+| Tool hardcoding DB path | cli_harmonization | yes |
+| Tool missing CUI header | cli_harmonization | yes |
+
+---
+
+## Resolution Process
+
+1. **Detect** — Governance validator or introspective analyzer flags drift
+2. **Score** — Innovation Engine scores signal (typically 0.6-0.8 feasibility)
+3. **Fix** — Developer or CI/CD auto-fixer applies the standard pattern
+4. **Verify** — Governance validator confirms fix, signal marked resolved
+5. **Prevent** — Pre-tool-use hook blocks new tools that violate standards
+
+---
+
+## Cadence
+
+| Activity | Frequency | Owner |
+|----------|-----------|-------|
+| Governance validator run | Every CI/CD build | Automated |
+| Introspective CLI scan | Every 12 hours | Innovation Engine daemon |
+| Manual harmonization review | Per PI (every 2 weeks) | Tech lead |
+| Standards update | Quarterly or on new phase | Architect agent |
+
+---
+
+## Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/testing/claude_dir_validator.py` | Governance enforcement (6+ checks) |
+| `tools/compat/db_utils.py` | Centralized DB path resolution |
+| `tools/innovation/introspective_analyzer.py` | CLI drift detection signals |
+| `tools/cli/output_formatter.py` | `--human` colored output helper |
+
+---
+
+## Security Gate
+
+The `claude_config_alignment` gate in `args/security_gates.yaml` blocks on:
+- `append_only_table_unprotected` — new append-only table without hook protection
+- `hook_syntax_error` — broken hook file
+- `hook_reference_missing` — settings.json references nonexistent hook
+- `cli_pattern_violation` — tool violates naming/output standards (NEW)
+
+---
+
+## Metrics
+
+| Metric | Target | Current |
+|--------|--------|---------|
+| Tools with --json support | 100% | ~77% (237/306) |
+| Tools using --project-id (not --project) | 100% | ~77% (110/142) |
+| Tools using db_utils.py | 100% | ~66% (80/122 DB tools) |
+| CUI headers present | 100% | ~98% |
+| Governance validator pass rate | 100% | 100% (6/6) |

icdev/data/goals/cross_language_translation.md

@@ -0,0 +1,171 @@
+# [TEMPLATE: CUI // SP-CTI]
+# Cross-Language Translation — Phase 43
+
+> LLM-assisted cross-language code translation across all 30 directional pairs of 6 first-class languages, with configurable model selection (including local/air-gapped), full ATO compliance preservation, and Dashboard + SaaS Portal visibility.
+
+## Purpose
+
+Translate entire codebases between ICDEV's 6 supported languages (Python, Java, JavaScript/TypeScript, Go, Rust, C#) using a 5-phase hybrid pipeline that combines deterministic extraction/assembly with LLM-assisted translation. Preserves ATO compliance (95% NIST 800-53 control coverage), CUI markings, and audit trail throughout.
+
+## When to Use
+
+- Customer modernizing legacy systems (e.g., Python 2 backend → Go microservice, Java monolith → Rust)
+- Cross-language migration as part of 7R "Rearchitect" strategy
+- DoD software modernization per FY25-26 plan (cATO, DevSecOps, presumptive reciprocity)
+- Test suite translation alongside production code
+
+## Prerequisites
+
+1. Source code accessible at a local path
+2. Source and target language identified (must be different, both in VALID_LANGUAGES)
+3. LLM provider configured in `args/llm_config.yaml` (or Ollama for air-gapped)
+4. ICDEV database initialized (`python tools/db/init_icdev_db.py`)
+
+## Workflow (7 Steps)
+
+### Step 1 — Extract (Deterministic)
+**Tool:** `tools/translation/source_extractor.py`
+
+Parse source code into language-agnostic Intermediate Representation (IR) as JSON. Uses Python `ast` module for Python, regex-based extractors for Java/Go/Rust/C#/TypeScript. Detects functions, classes, interfaces, enums, imports, idioms, concurrency patterns, error handling.
+
+```bash
+python tools/translation/source_extractor.py \
+  --source-path /path/to/source \
+  --language python \
+  --output-ir source_ir.json \
+  --project-id "proj-123" --json
+```
+
+### Step 2 — Type-Check (Deterministic)
+**Tool:** `tools/translation/type_checker.py`
+
+Validate type-compatibility of function signatures between source/target type systems BEFORE LLM translation (D253, adopted from Amazon Oxidizer). Catches nullable/non-nullable mismatches, generic type parameter differences, trait/interface incompatibilities early.
+
+### Step 3 — Translate (LLM-Assisted)
+**Tool:** `tools/translation/code_translator.py`
+
+Chunk-based LLM translation with feature mapping rules (D247), pass@k candidate generation (D254). Walks dependency graph post-order (leaf-first, D244). Mock-and-continue on persistent failure (D256).
+
+```bash
+python tools/translation/code_translator.py \
+  --ir-file source_ir.json \
+  --source-language python \
+  --target-language java \
+  --output-dir /path/to/output \
+  --project-id "proj-123" --candidates 3 --json
+```
+
+### Step 4 — Assemble (Deterministic)
+**Tool:** `tools/translation/project_assembler.py`
+
+Scaffold target project (pom.xml/go.mod/Cargo.toml/etc.), write translated files to language-conventional paths, apply CUI headers, generate README with provenance.
+
+### Step 5 — Validate + Repair (Deterministic + LLM)
+**Tool:** `tools/translation/translation_validator.py`
+
+8-check validation suite:
+1. Syntax check (compiler/interpreter)
+2. Lint (language-specific linter)
+3. Round-trip IR consistency (D248)
+4. API surface match (≥90%)
+5. Type coverage (≥85%)
+6. Complexity analysis (≤30% increase)
+7. Compliance (CUI markings present)
+8. Feature mapping (D247 rules applied)
+
+On failure, feeds compiler errors back to LLM for targeted repair (D255, max 3 attempts).
+
+### Step 6 — Compliance Bridge (Optional)
+**Tool:** `tools/modernization/compliance_bridge.py`
+
+Reuses existing compliance bridge for NIST 800-53 control inheritance. 95% coverage threshold applies. Cascades to FedRAMP/CMMC/800-171 via crosswalk engine.
+
+### Step 7 — Dashboard Review
+Navigate to `/translations` in Dashboard or `/portal/translations` in SaaS Portal. Review job status, validation results, unit-level translation status, dependency mappings.
+
+## Full Pipeline (One Command)
+
+```bash
+python tools/translation/translation_manager.py \
+  --source-path /path/to/source \
+  --source-language python \
+  --target-language java \
+  --output-dir /path/to/output \
+  --project-id "proj-123" \
+  --validate --json
+```
+
+### Pipeline Modes
+
+| Flag | Behavior |
+|------|----------|
+| `--dry-run` | Extract IR + type-check only, no LLM calls |
+| `--extract-only` | Phase 1 only, output IR file |
+| `--translate-only` | Skip validation |
+| `--validate-only` | Run Phase 5 on existing translated code |
+| `--translate-tests` | Include test file translation (D250) |
+| `--compliance-bridge` | Run compliance bridge after validation |
+| `--candidates k` | Generate k translation candidates (pass@k, D254) |
+
+## Security Gates
+
+| Gate | Condition | Type |
+|------|-----------|------|
+| Syntax errors in output | Any syntax/compile error | Blocking |
+| API surface below 90% | Less than 90% of public APIs preserved | Blocking |
+| Compliance coverage below 95% | Less than 95% NIST control inheritance | Blocking |
+| Secrets detected | Any secrets in translated code | Blocking |
+| CUI markings missing | Any file without CUI header | Blocking |
+| Round-trip similarity low | Below 80% structural match | Warning |
+| Type coverage low | Below 85% type mapping coverage | Warning |
+| Complexity increase high | Over 30% LOC increase | Warning |
+| Unmapped dependencies | Dependencies without target equivalents | Warning |
+| Stub functions present | Mock-and-continue units in output | Warning |
+| Lint issues | Linter warnings in translated code | Warning |
+
+## Architecture Decisions
+
+- **D242:** Hybrid 5-phase pipeline — deterministic extraction + type-checking + LLM translation + deterministic assembly + validate-repair loop
+- **D243:** IR pivot — source code extracted into language-agnostic JSON IR before translation
+- **D244:** Post-order dependency graph traversal at function/class granularity — translate leaf nodes first
+- **D245:** Non-destructive output (extends D18) — translation output to separate directory
+- **D246:** Declarative dependency mapping tables (D26 pattern) — `context/translation/dependency_mappings.json`
+- **D247:** 3-part feature mapping rules (Amazon Oxidizer) — syntactic pattern + NL description + static validation check
+- **D248:** Round-trip IR consistency check — re-parse translated output, compare to source IR
+- **D249:** Translation compliance bridge — reuses `compliance_bridge.py` for NIST 800-53 control inheritance
+- **D250:** Test translation as separate tool — `test_translator.py` with framework-specific assertion mapping
+- **D251:** Translation DB tables follow existing `migration_plans`/`migration_tasks` pattern
+- **D252:** Dashboard/Portal pages follow existing page patterns (stat-grid, table-container, charts.js)
+- **D253:** Type-compatibility pre-check (Amazon Oxidizer) — validate function signatures before LLM translation
+- **D254:** Pass@k candidate generation (Google) — generate k translations, select best
+- **D255:** Compiler-feedback repair loop (Google/CoTran) — max 3 repair attempts per unit
+- **D256:** Mock-and-continue (Amazon Oxidizer) — generate type-compatible stub on persistent failure
+
+## Edge Cases
+
+- **Empty source directory** — Return error with helpful message
+- **Single file project** — Works fine, creates minimal project scaffold
+- **Circular dependencies** — DFS handles cycles via visited set
+- **Very large functions (>500 lines)** — Split at method boundaries for classes
+- **No LLM available** — `--dry-run` mode works without LLM for IR extraction + type-checking
+- **All units fail translation** — Job marked as `failed`, all units mocked, dashboard shows full failure state
+- **Mixed language source** — Only files matching `--source-language` are extracted
+- **BDD test files** — `.feature` files copied unchanged; only step definitions translated
+
+## Cross-Phase Integration
+
+| Phase | Integration |
+|-------|-------------|
+| Phase 13-16 (RICOAS) | Intake engine detects translation requirements; gap detector flags missing target language expertise |
+| Phase 15 (Simulation) | "What if we translate X to Y?" simulation scenario |
+| Phase 17 (Compliance) | Crosswalk engine cascades translation compliance to FedRAMP/CMMC/800-171 |
+| Phase 18 (MBSE) | `translates_to` digital thread link type |
+| Phase 19 (Modernization) | Translation = "Rearchitect" in 7R taxonomy; compliance bridge reused |
+| Phase 22 (Marketplace) | Feature mapping rules publishable as marketplace assets |
+| Phase 24-25 (DevSecOps/ZTA) | SAST + secret detection on translated output |
+| Phase 29 (Monitoring) | Translation jobs emit heartbeat events |
+| Phase 35 (Innovation) | Translation metrics feed innovation engine |
+| Phase 36 (Evolutionary) | Translation capability tracked as genome dimension |
+| Phase 37 (ATLAS) | Prompt injection scanning on translation prompts |
+| Phase 38 (Cloud) | Multi-cloud LLM selection via router |
+| Phase 39 (Observability) | Translation events logged to audit trail |

icdev/data/goals/dashboard.md

@@ -0,0 +1,142 @@
+# Goal: Web Dashboard
+
+## Purpose
+Provide a web-based dashboard for business users and operators to view project status, compliance posture, security findings, deployment history, and agent health. Server-side rendered with Flask for minimal attack surface and STIG compliance.
+
+## Trigger
+- User navigates to dashboard URL
+- `/icdev-status` skill invoked (CLI equivalent)
+- Automated status report generation
+
+## Inputs
+- ICDEV database (`data/icdev.db`)
+- Args configuration files
+- CUI marking templates (`args/cui_markings.yaml`)
+
+## Architecture
+
+### Technology Stack
+- **Backend:** Flask (Python) — chosen for simplicity, auditability, smaller STIG surface
+- **Frontend:** Server-side rendered Jinja2 templates — no client-side frameworks
+- **Styling:** Custom CSS with gov-themed dark mode
+- **JavaScript:** Minimal vanilla JS for API polling (30s auto-refresh)
+- **Security:** CUI banners on every page, CSRF protection, no inline scripts
+
+### Directory Structure
+```
+tools/dashboard/
+├── app.py              # Flask app factory, routes
+├── config.py           # Configuration from args/*.yaml
+├── api/                # REST API blueprints
+│   ├── projects.py     # /api/projects/*
+│   ├── agents.py       # /api/agents/*
+│   ├── compliance.py   # /api/compliance/*
+│   ├── audit.py        # /api/audit/*
+│   └── metrics.py      # /api/metrics/*
+├── templates/          # Jinja2 HTML templates
+│   ├── base.html       # CUI banners, nav, content block
+│   ├── index.html      # Overview dashboard
+│   ├── projects/       # Project views
+│   ├── agents/         # Agent views
+│   └── monitoring/     # Monitoring views
+└── static/
+    ├── css/style.css   # Gov-themed dark CSS
+    └── js/api.js       # API client + auto-refresh
+```
+
+## Process
+
+### Step 1: Application Setup
+**Tool:** `tools/dashboard/app.py`
+- Create Flask app with factory pattern
+- Register blueprints for API routes
+- Configure CUI markings from `args/cui_markings.yaml`
+- Set up database connection to `data/icdev.db`
+
+### Step 2: Dashboard Pages
+
+#### Overview (/)
+- Project count with status breakdown
+- Compliance score summary
+- Active alerts
+- Recent audit trail entries
+- Agent health overview
+
+#### Projects (/projects, /projects/<id>)
+- Project list with filtering by status
+- Project detail with 5 tabs:
+  - Overview (metadata, health)
+  - Compliance (SSP, POAM, STIG, SBOM status)
+  - Security (scan results, gate status)
+  - Deployments (history, current versions)
+  - Audit Trail (project-scoped events)
+
+#### Compliance (/compliance)
+- Control family coverage matrix
+- STIG findings summary (CAT1/CAT2/CAT3)
+- SSP/POAM document status
+- SBOM currency
+
+#### Security (/security)
+- Vulnerability summary across projects
+- Dependency CVE listing
+- Secret detection alerts
+- Security gate status
+
+#### Agents (/agents)
+- Agent health grid (8 agents)
+- Task queue depths
+- Recent task completions
+- Error rates per agent
+
+#### Monitoring (/monitoring)
+- Health check status
+- Metric charts (via API data)
+- Active alerts with correlation
+- Self-healing event history
+
+#### Audit Trail (/audit)
+- Searchable audit log
+- Filter by event type, project, date range
+- Export capability
+
+### Step 3: REST API
+All dashboard data is also available via REST API:
+- `GET /api/projects` — List projects
+- `GET /api/projects/<id>` — Project detail
+- `GET /api/compliance/<project_id>` — Compliance status
+- `GET /api/agents` — Agent health
+- `GET /api/audit?event_type=&project_id=&limit=` — Audit trail
+- `GET /api/metrics/<project_id>` — Metric snapshots
+
+### Step 4: CUI Compliance
+Every page includes:
+- Top banner: "CUI // SP-CTI"
+- Bottom banner: "CUI // SP-CTI"
+- Designation indicator in page metadata
+- No data exported without CUI markings
+
+### Step 5: Security Hardening
+- CSRF token on all forms
+- Content-Security-Policy headers
+- No inline JavaScript
+- Session timeout (30 minutes)
+- HTTPS-only in production
+- Rate limiting on API endpoints
+
+## Outputs
+- Running Flask web application
+- REST API for programmatic access
+- CUI-marked HTML pages
+- Auto-refreshing status displays
+
+## Edge Cases
+- Database locked: retry with exponential backoff
+- Large audit trail: paginate (50 items per page)
+- Agent unreachable: show last known status with staleness indicator
+- No projects: show onboarding guide
+
+## Related Goals
+- `monitoring.md` — Data sources for monitoring views
+- `compliance_workflow.md` — Compliance data
+- `agent_management.md` — Agent health data