icdev 1.0.0__py3-none-any.whl

icdev/data/docs/features/phase-42-framework-planning.md

@@ -0,0 +1,177 @@
+# Phase 42 — Framework Planning
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 42 |
+| Title | Framework Planning — Language-Specific Build Commands |
+| Status | Implemented |
+| Priority | P2 |
+| Dependencies | Phase 11 (CI/CD Integration), Phase 3 (TDD Workflow) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+ICDEV supports 6 first-class programming languages (Python, Java, Go, Rust, C#, TypeScript), each with distinct project structures, testing frameworks, SAST tools, linters, formatters, BDD engines, dependency management systems, and container hardening requirements. When a developer initiates a new project, they must manually determine the correct technology stack, configure CI/CD pipeline stages, set up STIG-hardened Dockerfiles, establish CUI marking injection points, and map NIST 800-53 controls — all of which vary significantly per language.
+
+Prior to Phase 42, the `/icdev-init` and `/icdev-build` commands provided language-agnostic scaffolding that required manual customization for each language's ecosystem. This led to inconsistent project setups, missed security scanning steps (e.g., forgetting `gosec` for Go or `cargo-audit` for Rust), and incomplete CI/CD pipeline configurations. The lack of language-specific planning meant that the 12 Leverage Points of Agentic Development framework — which describes the 12 tunable dimensions of agent behavior — was not being fully utilized during project initialization.
+
+Phase 42 introduces 6 language-specific planning commands (`/plan_python`, `/plan_java`, `/plan_go`, `/plan_rust`, `/plan_csharp`, `/plan_typescript`) that generate comprehensive, opinionated build plans tailored to each language's ecosystem. These plans serve as the "Plans" leverage point (dimension 10), providing detailed implementation blueprints that the agent follows through the ATLAS workflow.
+
+---
+
+## 2. Goals
+
+1. Provide 6 language-specific planning commands that generate comprehensive build plans tailored to each supported language
+2. Include project structure templates, technology stack recommendations, and STIG-hardened Dockerfiles in each plan
+3. Generate CI/CD pipeline stage definitions with language-appropriate SAST, linting, formatting, and dependency auditing tools
+4. Map CUI marking injection points specific to each language's comment syntax and file structure
+5. Include test file structure covering unit tests, BDD scenarios, and E2E test specifications per language
+6. Codify the 12 Leverage Points of Agentic Development framework as a systematic approach to tuning agent behavior
+
+---
+
+## 3. Architecture
+
+```
+Developer
+    │
+    ↓
+/plan_<language> <app-name>
+    │
+    ↓
+Language Registry (context/languages/language_registry.json)
+    │
+    ├── Framework Patterns (context/languages/framework_patterns.json)
+    │
+    ↓
+Plan Generator
+    │
+    ├── 1. Project Structure Template
+    ├── 2. Technology Stack (framework, ORM, DB, auth)
+    ├── 3. STIG-Hardened Dockerfile
+    ├── 4. CI/CD Pipeline Stages
+    ├── 5. CUI Marking Injection Points
+    ├── 6. Test File Structure (unit + BDD + E2E)
+    ├── 7. Security Scanning Config (SAST, deps, secrets)
+    └── 8. NIST 800-53 Control Mapping Hints
+    │
+    ↓
+Markdown Plan Document
+    │
+    ↓
+/icdev-init (scaffold) → ATLAS Workflow (build)
+```
+
+Each `/plan_<language>` command reads from the language registry and framework patterns to generate an 8-section build plan as a markdown document. The plan is reviewed by the developer, optionally customized, and then executed through `/icdev-init` for scaffolding followed by the ATLAS TDD workflow for implementation.
+
+### 12 Leverage Points Framework
+
+The 12 Leverage Points describe 12 dimensions that can be tuned to improve agent behavior:
+
+**In Agent (Core Four):** Context, Model, Prompt, Tools
+**Through Agent (Multipliers):** Standard Output, Types, Docs, Tests, Architecture, Plans, Templates, Workflows
+
+Phase 42 planning commands embody dimension 10 (Plans) while generating artifacts that activate all other dimensions.
+
+---
+
+## 4. Requirements
+
+### 4.1 Language-Specific Plans
+
+#### REQ-42-001: Python Build Plan
+The `/plan_python` command SHALL generate a build plan using Flask/FastAPI, pytest, behave, bandit, pip-audit, black+isort, and a STIG-hardened Dockerfile.
+
+#### REQ-42-002: Java Build Plan
+The `/plan_java` command SHALL generate a build plan using Spring Boot, Cucumber-JVM, checkstyle/PMD, SpotBugs, OWASP Dependency Check, google-java-format, and a STIG-hardened Dockerfile.
+
+#### REQ-42-003: Go Build Plan
+The `/plan_go` command SHALL generate a build plan using net/http or Gin, godog, golangci-lint, gosec, govulncheck, gofmt, and a STIG-hardened Dockerfile.
+
+#### REQ-42-004: Rust Build Plan
+The `/plan_rust` command SHALL generate a build plan using Actix-web, cucumber-rs, clippy, cargo-audit, rustfmt, and a STIG-hardened Dockerfile.
+
+#### REQ-42-005: C# Build Plan
+The `/plan_csharp` command SHALL generate a build plan using ASP.NET Core, SpecFlow, dotnet analyzers, SecurityCodeScan, dotnet format, and a STIG-hardened Dockerfile.
+
+#### REQ-42-006: TypeScript Build Plan
+The `/plan_typescript` command SHALL generate a build plan using Express, cucumber-js, eslint+tsc, eslint-security, npm audit, prettier, and a STIG-hardened Dockerfile.
+
+### 4.2 Plan Content
+
+#### REQ-42-007: NIST 800-53 Control Mapping
+Each plan SHALL include NIST 800-53 control mapping hints that indicate which controls are addressed by each component of the technology stack.
+
+#### REQ-42-008: CUI Marking Points
+Each plan SHALL identify CUI marking injection points specific to the language's comment syntax (e.g., `# CUI // SP-CTI` for Python, `// CUI // SP-CTI` for Java/Go/Rust/C#/TypeScript).
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| *(No new tables)* | Phase 42 generates plan documents; execution uses existing project, task, and audit tables |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/builder/language_support.py` | Language detection and registry management — detect project languages, list supported languages |
+| `tools/builder/scaffolder.py` | Project scaffolding from plan — creates directory structure, initial files, Dockerfile |
+| `tools/builder/test_writer.py` | Generate test file structure per language (unit + BDD) |
+| `tools/builder/code_generator.py` | TDD code generation from test files |
+| `tools/builder/linter.py` | Language-appropriate linting |
+| `tools/builder/formatter.py` | Language-appropriate formatting |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| — | Language-specific planning commands as Claude Code skills | Each language ecosystem has distinct tooling; generic plans lead to missed steps and misconfiguration |
+| — | 12 Leverage Points framework codified | Provides systematic approach to tuning agent behavior across all 12 dimensions |
+| — | Plans generate markdown documents (not code directly) | Human review step between planning and execution; developer can customize before scaffolding |
+
+---
+
+## 8. Security Gate
+
+**Framework Planning Gate:**
+- Generated plans must include STIG-hardened Dockerfile with non-root user, read-only rootfs, dropped capabilities
+- Plans must include SAST tool configuration appropriate to the language
+- Plans must include dependency audit tool appropriate to the language
+- Plans must include CUI marking injection points
+- Plans must reference applicable NIST 800-53 controls
+
+---
+
+## 9. Commands
+
+```bash
+# Language-specific planning commands (Claude Code skills)
+/plan_python <app-name>       # Flask/FastAPI, pytest, behave, bandit, pip-audit
+/plan_java <app-name>         # Spring Boot, Cucumber-JVM, SpotBugs, OWASP DC
+/plan_go <app-name>           # net/http/Gin, godog, gosec, govulncheck
+/plan_rust <app-name>         # Actix-web, cucumber-rs, clippy, cargo-audit
+/plan_csharp <app-name>       # ASP.NET Core, SpecFlow, SecurityCodeScan
+/plan_typescript <app-name>   # Express, cucumber-js, eslint-security, npm audit
+
+# Language detection
+python tools/builder/language_support.py --detect "/path/to/project"
+python tools/builder/language_support.py --list
+
+# Context files
+# context/languages/language_registry.json — Language detection profiles
+# context/languages/framework_patterns.json — Framework-specific patterns
+```

icdev/data/docs/features/phase-43-cross-language-translation.md

@@ -0,0 +1,225 @@
+# Phase 43 — Cross-Language Translation
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 43 |
+| Title | Cross-Language Translation — LLM-Assisted Hybrid Pipeline |
+| Status | Implemented |
+| Priority | P1 |
+| Dependencies | Phase 38 (Cloud-Agnostic Architecture), Phase 19 (App Modernization), Phase 17 (ATO Acceleration) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+DoD and government organizations face a persistent challenge in legacy software modernization: migrating codebases from one programming language to another while preserving functionality, compliance posture, and ATO status. The FY25-26 DoD software modernization plan emphasizes cATO, DevSecOps, and presumptive reciprocity — all of which require maintaining compliance continuity through language migrations. Manual translation is error-prone, expensive, and does not scale.
+
+ICDEV already supports 6 first-class languages (Python, Java, Go, Rust, C#, TypeScript) with full toolchain coverage (scaffold, lint, format, SAST, dep audit, BDD, code gen). However, translating between these 30 directional language pairs requires more than simple LLM prompting. LLMs excel at translating small code snippets but struggle with project-level translation: maintaining consistent naming, resolving cross-file dependencies, mapping language-specific idioms, preserving type safety, and ensuring the translated output compiles and passes tests.
+
+The GOTCHA principle — that LLMs are probabilistic while business logic must be deterministic — dictates a hybrid approach. Deterministic extraction parses source code into a language-agnostic Intermediate Representation (IR). Deterministic type-checking validates signature compatibility before translation. LLM-assisted translation handles the creative mapping between language idioms. Deterministic assembly scaffolds the target project. A validate-and-repair loop uses compiler feedback to fix translation errors. This 5-phase pipeline maximizes translation quality while maintaining auditability and compliance traceability.
+
+---
+
+## 2. Goals
+
+1. Translate entire codebases between ICDEV's 6 supported languages (30 directional pairs) using a 5-phase hybrid pipeline
+2. Preserve ATO compliance through the translation with 95% NIST 800-53 control coverage via the compliance bridge
+3. Generate multiple translation candidates per unit (pass@k) and select the best, improving translation quality
+4. Handle persistent translation failures gracefully via mock-and-continue, generating type-compatible stubs that allow dependent code to translate
+5. Repair translation errors automatically via compiler-feedback loops (max 3 attempts per unit)
+6. Translate test suites alongside production code, with framework-specific assertion mapping and BDD feature file preservation
+7. Track all translation units, dependency mappings, and validation results in the database for traceability
+8. Provide dashboard and SaaS portal visibility into translation job status, unit-level progress, and validation results
+
+---
+
+## 3. Architecture
+
+```
+Source Code (/path/to/source)
+         │
+    Phase 1: EXTRACT (deterministic)
+    source_extractor.py → Language-Agnostic IR (JSON)
+         │
+    Phase 2: TYPE-CHECK (deterministic)
+    type_checker.py → Signature compatibility validation
+         │
+    Phase 3: TRANSLATE (LLM-assisted)
+    code_translator.py → Post-order dependency traversal
+         │                  ├── pass@k candidates (D254)
+         │                  ├── mock-and-continue (D256)
+         │                  └── feature mapping rules (D247)
+         │
+    Phase 4: ASSEMBLE (deterministic)
+    project_assembler.py → Target project scaffold
+         │                   ├── pom.xml / go.mod / Cargo.toml
+         │                   ├── CUI headers applied
+         │                   └── README with provenance
+         │
+    Phase 5: VALIDATE + REPAIR (deterministic + LLM)
+    translation_validator.py → 8-check validation
+         │                       ├── Syntax (compiler)
+         │                       ├── Lint (language linter)
+         │                       ├── Round-trip IR (D248)
+         │                       ├── API surface (>=90%)
+         │                       ├── Type coverage (>=85%)
+         │                       ├── Complexity (<=30% increase)
+         │                       ├── Compliance (CUI markings)
+         │                       └── Feature mapping (D247)
+         │
+    On failure: Compiler-feedback repair (D255, max 3 attempts)
+         │
+    Phase 6: COMPLIANCE BRIDGE (optional)
+    compliance_bridge.py → NIST 800-53 control inheritance (95%)
+```
+
+The pipeline processes source code through 5 phases. Phase 1 extracts an IR using Python `ast` for Python and regex-based extractors for other languages. Phase 2 validates type-system compatibility before invoking the LLM. Phase 3 translates code units in post-order dependency graph traversal (leaf nodes first), generating k candidates per unit and selecting the best. Phase 4 assembles the target project with language-conventional structure. Phase 5 validates the output with 8 checks and feeds compiler errors back to the LLM for repair. An optional Phase 6 runs the compliance bridge for ATO preservation.
+
+---
+
+## 4. Requirements
+
+### 4.1 Extraction
+
+#### REQ-43-001: Language-Agnostic IR
+The system SHALL extract source code into a JSON Intermediate Representation containing functions, classes, interfaces, enums, imports, idioms, concurrency patterns, and error handling constructs.
+
+#### REQ-43-002: Dependency Graph
+The system SHALL build a dependency graph at function/class granularity and translate in post-order (leaf nodes first) to ensure dependencies are resolved before dependents.
+
+### 4.2 Translation
+
+#### REQ-43-003: Pass@k Candidate Generation
+The system SHALL generate k translation candidates per unit (default k=3 for cloud, k=1 for air-gapped) with varied prompts and select the best based on validation scores.
+
+#### REQ-43-004: Mock-and-Continue
+When a translation unit persistently fails after max repair attempts, the system SHALL generate a type-compatible mock/stub and continue translating dependent units.
+
+#### REQ-43-005: Feature Mapping Rules
+The system SHALL apply 3-part feature mapping rules (syntactic pattern, natural language description, static validation check) for language-specific idiom translation.
+
+#### REQ-43-006: Non-Destructive Output
+Translation output SHALL be written to a separate directory; source code SHALL never be modified.
+
+### 4.3 Validation
+
+#### REQ-43-007: 8-Check Validation Suite
+The system SHALL validate translated output with: syntax check, lint, round-trip IR consistency, API surface match (>=90%), type coverage (>=85%), complexity analysis (<=30% increase), compliance (CUI markings), and feature mapping verification.
+
+#### REQ-43-008: Compiler-Feedback Repair
+On validation failure, the system SHALL feed compiler errors back to the LLM for targeted repair, with a maximum of 3 repair attempts per unit.
+
+### 4.4 Compliance
+
+#### REQ-43-009: Compliance Bridge
+The system SHALL reuse the existing compliance bridge for NIST 800-53 control inheritance with a 95% coverage threshold, cascading to FedRAMP/CMMC/800-171 via the crosswalk engine.
+
+#### REQ-43-010: CUI Marking Preservation
+All translated files SHALL include CUI headers appropriate to the target language's comment syntax.
+
+### 4.5 Test Translation
+
+#### REQ-43-011: Test Suite Translation
+The system SHALL translate test suites alongside production code with framework-specific assertion mapping (pytest to JUnit, behave to Cucumber-JVM, etc.).
+
+#### REQ-43-012: BDD Feature Preservation
+BDD `.feature` files SHALL be copied unchanged; only step definition implementations SHALL be translated.
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `translation_jobs` | Job-level tracking — source/target language, project ID, status, phase progress, timestamps, validation summary |
+| `translation_units` | Unit-level tracking — function/class name, translation status, candidate count, selected candidate, mock flag, repair attempts |
+| `translation_dependency_mappings` | Cross-language dependency equivalents — source package, target package, mapping confidence, manual override flag |
+| `translation_validations` | Validation results per unit — 8 check results, scores, repair history, final status |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/translation/translation_manager.py` | Full pipeline orchestrator — runs all 5 phases with configurable modes (dry-run, extract-only, translate-only, validate-only) |
+| `tools/translation/source_extractor.py` | Phase 1 — parse source code into language-agnostic IR JSON |
+| `tools/translation/type_checker.py` | Phase 2 — validate type-system compatibility between source and target |
+| `tools/translation/code_translator.py` | Phase 3 — LLM-assisted chunk-based translation with pass@k and mock-and-continue |
+| `tools/translation/project_assembler.py` | Phase 4 — scaffold target project with language conventions and CUI headers |
+| `tools/translation/translation_validator.py` | Phase 5 — 8-check validation suite with compiler-feedback repair loop |
+| `tools/translation/dependency_mapper.py` | Cross-language dependency lookup from declarative mapping tables |
+| `tools/translation/test_translator.py` | Test suite translation with framework-specific assertion mapping |
+| `tools/modernization/compliance_bridge.py` | Phase 6 — NIST 800-53 control inheritance for ATO preservation (reused) |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D242 | Hybrid 5-phase pipeline (deterministic + LLM) | Consistent with GOTCHA: LLMs probabilistic, business logic deterministic |
+| D243 | IR pivot — language-agnostic JSON IR | Enables chunk-based translation, round-trip validation, progress tracking |
+| D244 | Post-order dependency graph traversal | Translate leaf nodes first ensures dependencies resolved before dependents |
+| D245 | Non-destructive output (extends D18) | Source never modified; output to separate directory |
+| D246 | Declarative dependency mapping tables (D26 pattern) | Cross-language package equivalents without code changes |
+| D247 | 3-part feature mapping rules (Amazon Oxidizer) | Syntactic pattern + NL description + static validation per language pair |
+| D248 | Round-trip IR consistency check | Re-parse translated output into IR, compare structurally to source IR |
+| D249 | Translation compliance bridge reuses existing tool | 95% NIST 800-53 control coverage threshold; cascades via crosswalk |
+| D250 | Test translation as separate tool | Framework-specific assertion mapping; BDD features preserved |
+| D253 | Type-compatibility pre-check (Amazon Oxidizer) | Validate signatures before LLM translation; catch mismatches early |
+| D254 | Pass@k candidate generation (Google) | Generate k candidates, select best; default k=3 cloud, k=1 air-gapped |
+| D255 | Compiler-feedback repair loop (Google/CoTran) | Feed compiler errors to LLM for targeted repair; max 3 attempts |
+| D256 | Mock-and-continue (Amazon Oxidizer) | Type-compatible stub on persistent failure; unblocks dependents |
+
+---
+
+## 8. Security Gate
+
+**Translation Gate:**
+- **Blocking:** Syntax errors in output, API surface below 90%, compliance coverage below 95%, secrets detected in translated code, CUI markings missing
+- **Warning:** Round-trip similarity below 80%, type coverage below 85%, complexity increase over 30%, unmapped dependencies, stub functions present, lint issues
+
+---
+
+## 9. Commands
+
+```bash
+# Full pipeline
+python tools/translation/translation_manager.py \
+  --source-path /path/to/source --source-language python --target-language java \
+  --output-dir /path/to/output --project-id "proj-123" --validate --json
+
+# Dry run (no LLM calls)
+python tools/translation/translation_manager.py \
+  --source-path /path --source-language python --target-language java \
+  --output-dir /path --project-id "proj-123" --dry-run --json
+
+# Extract IR only
+python tools/translation/source_extractor.py \
+  --source-path /path --language python --output-ir ir.json --project-id "proj-123" --json
+
+# Translate with pass@k candidates
+python tools/translation/code_translator.py \
+  --ir-file ir.json --source-language python --target-language go \
+  --output-dir /path --candidates 3 --json
+
+# Dependency lookup
+python tools/translation/dependency_mapper.py \
+  --source-language python --target-language go --imports "flask,requests" --json
+
+# Translate tests
+python tools/translation/test_translator.py \
+  --source-test-dir /path/tests --source-language python --target-language java \
+  --output-dir /path/output/tests --ir-file ir.json --json
+
+# Configuration
+# args/translation_config.yaml — 30 language pairs, extraction, translation, repair, validation thresholds
+# context/translation/dependency_mappings.json — Cross-language package equivalents
+```

icdev/data/docs/features/phase-44-innovation-adaptation.md

@@ -0,0 +1,227 @@
+# Phase 44 — Innovation Adaptation
+
+**CUI // SP-CTI**
+
+| Field | Value |
+|-------|-------|
+| Phase | 44 |
+| Title | Innovation Adaptation — Agent Zero & InsForge Patterns |
+| Status | Implemented |
+| Priority | P2 |
+| Dependencies | Phase 35 (Innovation Engine), Phase 39 (Observability & Operations), Phase 36 (Evolutionary Intelligence) |
+| Author | ICDEV Architect Agent |
+| Date | 2026-02-23 |
+
+---
+
+## 1. Problem Statement
+
+ICDEV's Innovation Engine (Phase 35) discovers improvement opportunities through web scanning, signal scoring, and compliance triage. However, the internal agent execution model remained single-threaded and stateless — each agent interaction was independent, with no persistent conversation context, no mechanism for mid-stream course correction, and no way for external systems to extend agent behavior at runtime. These limitations prevented ICDEV from adopting proven patterns from cutting-edge agentic AI frameworks.
+
+Agent Zero demonstrated that multi-stream parallel execution with persistent memory consolidation dramatically improves agent effectiveness. InsForge showed that active extension hooks — allowing external code to modify agent behavior at defined hook points — enable ecosystem-level customization without forking the core framework. Neither pattern was available in ICDEV.
+
+Phase 44 adapts 10 capabilities from these frameworks into ICDEV's GOTCHA architecture: multi-stream parallel chat with thread-per-context execution, active extension hooks with behavioral and observational tiers, mid-stream intervention for atomic course correction, dirty-tracking state push for efficient real-time updates, 3-tier history compression for long-running conversations, shared schema enforcement via dataclasses, AI-driven memory consolidation, semantic layer MCP tools for context-aware agent guidance, dangerous pattern detection across 6 languages, and innovation signal registration for external pattern ingestion. Each capability is implemented as a deterministic tool consistent with the GOTCHA separation of concerns.
+
+---
+
+## 2. Goals
+
+1. Enable multi-stream parallel chat with thread-per-context execution (max 5 concurrent per user) for simultaneous agent interactions
+2. Provide 10 active extension hook points with behavioral (modify data) and observational (log only) tiers for runtime agent customization
+3. Support atomic mid-stream intervention with 3-checkpoint verification for safe course correction during agent execution
+4. Implement dirty-tracking state push with SSE debounced at 25ms and HTTP polling at 3s for efficient real-time client updates
+5. Compress conversation history with a 3-tier budget model (current topic 50%, historical 30%, bulk 20%) to maintain context within token limits
+6. Enforce shared schemas across agent outputs using stdlib dataclasses for air-gap-safe type enforcement
+7. Consolidate duplicate and related memory entries using AI-driven similarity detection (Jaccard + LLM) with append-only consolidation logging
+8. Index CLAUDE.md sections semantically and serve context-aware guidance to agents via MCP tools based on agent role
+
+---
+
+## 3. Architecture
+
+```
+Multi-Stream Chat Manager
+  ┌─────────────┬──────────────┬──────────────┐
+  │ Context A   │ Context B    │ Context C    │  (max 5/user)
+  │ (thread)    │ (thread)     │ (thread)     │
+  │ message_q   │ message_q    │ message_q    │
+  │ extensions  │ extensions   │ extensions   │
+  └──────┬──────┴──────┬───────┴──────┬───────┘
+         │             │              │
+    ┌────┴────┐   ┌────┴────┐   ┌────┴────┐
+    │Extension│   │Extension│   │Extension│
+    │ Hooks   │   │ Hooks   │   │ Hooks   │
+    │(10 pts) │   │(10 pts) │   │(10 pts) │
+    └────┬────┘   └────┬────┘   └────┬────┘
+         │             │              │
+    Intervention    State Push    History
+    (3 checkpoints) (SSE 25ms)   Compression
+         │             │         (3-tier)
+         ↓             ↓              ↓
+    ┌──────────────────────────────────────┐
+    │  Shared Schema Enforcement           │
+    │  (dataclasses + validate_output())   │
+    └──────────────────────────────────────┘
+         │
+    ┌────┴─────────────────────────────────┐
+    │  Memory Consolidation                │
+    │  (Jaccard + LLM, append-only log)    │
+    └──────────────────────────────────────┘
+         │
+    ┌────┴─────────────────────────────────┐
+    │  Semantic Layer MCP Tools            │
+    │  (CLAUDE.md indexing, role mapping)  │
+    └──────────────────────────────────────┘
+         │
+    ┌────┴─────────────────────────────────┐
+    │  Code Pattern Scanner + Signal Reg   │
+    │  (6 languages, innovation pipeline)  │
+    └──────────────────────────────────────┘
+```
+
+Chat contexts are scoped to `(user_id, tenant_id)` with a maximum of 5 concurrent contexts per user. Each context runs in its own thread with an independent message queue (`collections.deque`), extension hook chain, and history compression state. Extensions are loaded from numbered Python files following the Agent Zero pattern, with layered override (project > tenant > default) and exception isolation.
+
+---
+
+## 4. Requirements
+
+### 4.1 Multi-Stream Parallel Chat
+
+#### REQ-44-001: Thread-Per-Context Execution
+The system SHALL support multiple concurrent chat contexts per user (max 5), each running in its own thread with an independent message queue.
+
+#### REQ-44-002: Context Scoping
+Chat contexts SHALL be scoped to `(user_id, tenant_id)` and stored in the `chat_contexts` and `chat_messages` database tables.
+
+#### REQ-44-003: Context Independence
+Each context SHALL be independent of intake sessions (Phase 13) and other agent execution channels.
+
+### 4.2 Active Extension Hooks
+
+#### REQ-44-004: 10 Extension Hook Points
+The system SHALL provide 10 hook points at defined stages of agent execution (pre-LLM, post-LLM, pre-tool, post-tool, pre-output, post-output, pre-queue, post-queue, session-start, session-end).
+
+#### REQ-44-005: Behavioral and Observational Tiers
+Extensions SHALL be classified as behavioral (may modify data flowing through the hook) or observational (read-only logging/metrics), with behavioral extensions subject to stricter safety limits.
+
+#### REQ-44-006: Layered Override
+Extension resolution SHALL follow project > tenant > default precedence, with exception isolation ensuring one failing extension cannot crash the agent.
+
+#### REQ-44-007: Safety Limits
+Total handler execution time SHALL not exceed 30 seconds across all extensions at a single hook point.
+
+### 4.3 Mid-Stream Intervention
+
+#### REQ-44-008: Atomic 3-Checkpoint Intervention
+The system SHALL support mid-stream intervention checked at 3 points per loop iteration: pre-LLM, post-LLM, and pre-queue-pop. Intervention messages SHALL be stored as `role='intervention'` in the message history.
+
+### 4.4 State Management
+
+#### REQ-44-009: Dirty-Tracking State Push
+The system SHALL track per-client dirty/pushed version counters with SSE debounced at 25ms and HTTP polling at 3s. Clients SHALL send `?since_version=N` for incremental updates.
+
+#### REQ-44-010: 3-Tier History Compression
+The system SHALL compress conversation history using a 3-tier budget: current topic 50%, historical summaries 30%, bulk archive 20%. Topic boundaries SHALL be detected by time gap (>30 min) or keyword shift (>60%).
+
+### 4.5 Schema and Memory
+
+#### REQ-44-011: Shared Schema Enforcement
+Agent outputs SHALL be validated against shared schemas using stdlib `dataclasses` with optional Pydantic support, backward compatible via `to_dict()` and `validate_output()` methods.
+
+#### REQ-44-012: AI-Driven Memory Consolidation
+The system SHALL optionally consolidate duplicate memory entries using hybrid search (Jaccard keyword fallback + optional LLM) with decisions logged to an append-only `memory_consolidation_log` table.
+
+### 4.6 Semantic Layer and Pattern Detection
+
+#### REQ-44-013: Semantic Layer MCP Tools
+The MCP context server SHALL index CLAUDE.md sections by `##` headers, cache with configurable TTL, and serve role-appropriate sections to agents based on agent-role-to-section mapping.
+
+#### REQ-44-014: Dangerous Pattern Detection
+The code pattern scanner SHALL detect dangerous patterns (eval, exec, os.system, SQL injection, command injection) across 6 languages using declarative YAML-configured regex patterns.
+
+#### REQ-44-015: Innovation Signal Registration
+External patterns and framework analyses SHALL be registered as innovation signals with 5-dimension weighted scoring (novelty, feasibility, compliance_alignment, user_impact, effort).
+
+---
+
+## 5. Database Schema
+
+### Tables
+
+| Table | Purpose |
+|-------|---------|
+| `chat_contexts` | Chat context metadata — user_id, tenant_id, status, created_at, last_active, context_name |
+| `chat_messages` | Chat message storage — context_id, role (user/assistant/intervention), content, timestamp, token_count |
+| `chat_tasks` | Task tracking per chat context — context_id, task description, status, result |
+| `extension_registry` | Registered extensions — name, hook_point, tier (behavioral/observational), file_path, priority, scope |
+| `extension_execution_log` | Extension execution audit — extension_id, hook_point, duration_ms, success, error_message |
+| `memory_consolidation_log` | Append-only log of consolidation decisions — entry_ids, action (MERGE/REPLACE/KEEP_SEPARATE/UPDATE/SKIP), rationale |
+
+---
+
+## 6. Tools
+
+| Tool | Purpose |
+|------|---------|
+| `tools/agent/chat_manager.py` | Multi-stream parallel chat — context lifecycle, message routing, thread management |
+| `tools/agent/extension_manager.py` | Active extension hook system — load, register, execute extensions with safety limits |
+| `tools/agent/state_tracker.py` | Dirty-tracking state push — version counters, SSE dispatch, incremental updates |
+| `tools/agent/history_compressor.py` | 3-tier history compression — topic detection, budget allocation, LLM/truncation fallback |
+| `tools/agent/schemas.py` | Shared schema enforcement — dataclass definitions, validate_output(), strict/non-strict modes |
+| `tools/memory/memory_consolidation.py` | AI-driven memory consolidation — similarity detection, LLM decision, append-only logging |
+| `tools/mcp/context_server.py` | Semantic layer MCP tools — CLAUDE.md indexer, role-based section serving, cache management |
+| `tools/security/code_pattern_scanner.py` | Dangerous pattern detection — 6-language regex scanner with declarative YAML config |
+| `tools/innovation/register_external_patterns.py` | Innovation signal registration — external pattern ingestion into innovation pipeline |
+
+---
+
+## 7. Architecture Decisions
+
+| ID | Decision | Rationale |
+|----|----------|-----------|
+| D257-D260 | Thread-per-context with collections.deque message queues | Independent execution per context, max 5/user, scoped to (user_id, tenant_id) |
+| D261-D264 | Extensions loaded from numbered Python files (Agent Zero pattern) | Behavioral/observational tiers, layered override, exception isolation, 30s safety limit |
+| D265-D267 | Atomic 3-checkpoint intervention | Checked at pre-LLM, post-LLM, pre-queue-pop; checkpoint preservation; role='intervention' messages |
+| D268-D270 | Dirty-tracking with SSE 25ms debounce, HTTP 3s polling | Efficient incremental updates; clients send ?since_version=N |
+| D271-D274 | 3-tier history compression with topic boundary detection | Budget: 50%/30%/20%; time gap >30min or keyword shift >60% for topic boundaries |
+| D275 | Shared schemas via stdlib dataclasses | Air-gap safe, optional Pydantic, backward compatible to_dict()/validate_output() |
+| D276 | AI-driven memory consolidation with Jaccard fallback | Optional --consolidate flag, LLM decides action, append-only consolidation log |
+| D277 | CLAUDE.md section indexing via ## header parsing | Agent-role-to-section mapping, cache TTL, air-gap safe (stdlib only) |
+| D278 | Dangerous pattern detection via declarative YAML | Unified scanner across 6 languages, callable from marketplace/translation/security |
+| D279 | External patterns registered as innovation signals | 5-dimension scoring, feeds Phase 35 pipeline |
+
+---
+
+## 8. Security Gate
+
+**Innovation Adaptation Gate:**
+- Extension execution must not exceed 30 seconds total per hook point
+- Behavioral extensions must be registered with explicit scope (project/tenant/default)
+- Memory consolidation decisions must be logged to append-only audit trail
+- Dangerous pattern scanner must detect eval/exec/os.system patterns with zero false negatives on known test cases
+- Code pattern gate: max_critical=0, max_high=0, max_medium=10
+
+---
+
+## 9. Commands
+
+```bash
+# Code pattern scanning
+python tools/security/code_pattern_scanner.py --project-dir /path --json
+python tools/security/code_pattern_scanner.py --project-dir /path --gate --json
+
+# Memory consolidation
+python tools/memory/memory_consolidation.py --consolidate --json
+python tools/memory/memory_consolidation.py --dry-run --json
+
+# Innovation signal registration
+python tools/innovation/register_external_patterns.py --source "Agent Zero" --patterns patterns.json --json
+
+# Semantic layer context
+# MCP server: icdev-context with tools: fetch_docs, list_sections, get_icdev_metadata, get_project_context, get_agent_context
+
+# Configuration
+# args/extension_config.yaml — 10 hook points, layered override, safety limits
+# args/context_config.yaml — CLAUDE.md indexing, cache TTL, agent-role mapping
+# args/code_pattern_config.yaml — Per-language patterns, scan settings, severity classification
+```