agent-os-kernel 1.1.0__py3-none-any.whl → 1.3.0__py3-none-any.whl

modules/cmvk/docs/DIAGRAMS.md

@@ -0,0 +1,325 @@
+# CMVK Architecture Diagrams
+
+This document contains diagrams for the Cross-Model Verification Kernel paper and documentation.
+Diagrams are provided in multiple formats: Mermaid (for GitHub rendering), ASCII (for terminals), and descriptions for draw.io/Lucidchart export.
+
+## 1. High-Level Architecture
+
+### Mermaid Diagram
+
+```mermaid
+flowchart TB
+    subgraph User["👤 User"]
+        Task[Task/Problem]
+    end
+
+    subgraph Kernel["⚖️ Verification Kernel (Arbiter)"]
+        direction TB
+        Orchestrator[Orchestrator]
+        LoopControl[Loop Controller]
+        DecisionEngine[Decision Engine]
+    end
+
+    subgraph Generator["🔨 Generator (System 1)"]
+        GenModel[GPT-4o / o1]
+        GenPrompt[Creative Prompt]
+    end
+
+    subgraph Verifier["🔍 Verifier (System 2)"]
+        VerModel[Gemini / Claude]
+        VerPrompt[Adversarial Prompt]
+        Prosecutor[Prosecutor Mode]
+    end
+
+    subgraph Graph["📊 Graph of Truth"]
+        StateCache[State Cache]
+        StrategyBan[Strategy Ban List]
+        SolutionCache[Solution Cache]
+    end
+
+    subgraph Sandbox["🔒 Sandbox"]
+        CodeExec[Code Executor]
+        TestRunner[Test Runner]
+    end
+
+    Task --> Orchestrator
+    Orchestrator --> GenModel
+    GenModel --> |Solution| Orchestrator
+    Orchestrator --> VerModel
+    VerModel --> |Critique| DecisionEngine
+    Prosecutor --> |Hostile Tests| Sandbox
+    Sandbox --> |Results| DecisionEngine
+    DecisionEngine --> |Accept/Reject| LoopControl
+    LoopControl --> |Iterate| GenModel
+    LoopControl --> |Cache| Graph
+    Graph --> |History| Orchestrator
+    DecisionEngine --> |Final| User
+```
+
+### ASCII Diagram (for terminals/plain text)
+
+```
+┌──────────────────────────────────────────────────────────────────────┐
+│                        VERIFICATION KERNEL                            │
+│                          (The Arbiter)                                │
+│  ┌─────────────┐    ┌──────────────┐    ┌─────────────────┐          │
+│  │ Orchestrator│───▶│Loop Controller│───▶│ Decision Engine │          │
+│  └─────────────┘    └──────────────┘    └─────────────────┘          │
+└────────┬─────────────────────┬───────────────────┬───────────────────┘
+         │                     │                   │
+         ▼                     ▼                   ▼
+┌────────────────┐    ┌────────────────┐    ┌──────────────┐
+│   GENERATOR    │    │    VERIFIER    │    │GRAPH OF TRUTH│
+│   (System 1)   │    │   (System 2)   │    │              │
+│                │    │                │    │ • State Cache│
+│  ┌──────────┐  │    │  ┌──────────┐  │    │ • Ban List   │
+│  │ GPT-4o   │  │◀──▶│  │ Gemini   │  │    │ • Solutions  │
+│  │ o1       │  │    │  │ Claude   │  │    │              │
+│  └──────────┘  │    │  └──────────┘  │    └──────────────┘
+│                │    │                │           │
+│  High Creativity    │  High Logic    │           │
+│  Fast Builder  │    │  Cynical Judge │           │
+└────────────────┘    └───────┬────────┘           │
+                              │                    │
+                              ▼                    │
+                     ┌────────────────┐            │
+                     │    SANDBOX     │◀───────────┘
+                     │                │
+                     │ • Code Exec    │
+                     │ • Test Runner  │
+                     │ • Isolation    │
+                     └────────────────┘
+```
+
+## 2. Verification Loop Sequence
+
+### Mermaid Sequence Diagram
+
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant K as Kernel
+    participant G as Generator
+    participant V as Verifier
+    participant S as Sandbox
+    participant GoT as Graph of Truth
+
+    U->>K: Submit Task
+    K->>GoT: Check Cache
+    GoT-->>K: Cache Miss
+
+    loop Verification Loop (max 5)
+        K->>G: Generate Solution
+        G-->>K: Solution + Explanation
+        K->>GoT: Create Node
+        K->>V: Verify Solution
+        V->>S: Run Hostile Tests
+        S-->>V: Test Results
+        V-->>K: Verdict + Confidence
+
+        alt Verdict: PASS & Confidence > 0.85
+            K->>GoT: Cache Solution
+            K-->>U: Return Solution ✓
+        else Verdict: FAIL
+            K->>GoT: Ban Strategy
+            K->>GoT: Record Failure
+            Note over K,G: Iterate with feedback
+        else Max Loops Reached
+            K-->>U: Return Best Effort ⚠️
+        end
+    end
+```
+
+### ASCII Sequence
+
+```
+User          Kernel        Generator      Verifier       Sandbox        Graph
+  │             │              │              │              │              │
+  │──Task──────▶│              │              │              │              │
+  │             │──────────────────────────────────────────▶│ Check Cache  │
+  │             │◀─────────────────────────────────────────── Cache Miss   │
+  │             │                                                          │
+  │             │ ╔══════════════════════════════════════════════════════╗ │
+  │             │ ║              VERIFICATION LOOP (1..5)                ║ │
+  │             │ ╚══════════════════════════════════════════════════════╝ │
+  │             │              │              │              │              │
+  │             │──Generate───▶│              │              │              │
+  │             │◀──Solution───│              │              │              │
+  │             │──────────────────────────────────────────▶│ Create Node  │
+  │             │──────────────────Verify────▶│              │              │
+  │             │              │              │──Tests──────▶│              │
+  │             │              │              │◀──Results────│              │
+  │             │◀─────────────────Verdict────│              │              │
+  │             │              │              │              │              │
+  │             │  [PASS + High Confidence]   │              │              │
+  │             │──────────────────────────────────────────▶│ Cache        │
+  │◀──Solution──│              │              │              │              │
+  │             │              │              │              │              │
+  │             │  [FAIL]      │              │              │              │
+  │             │──────────────────────────────────────────▶│ Ban Strategy │
+  │             │──Feedback───▶│ (Iterate)    │              │              │
+  │             │              │              │              │              │
+```
+
+## 3. Graph of Truth State Machine
+
+### Mermaid State Diagram
+
+```mermaid
+stateDiagram-v2
+    [*] --> Pending: New Solution
+
+    Pending --> Verified: PASS + Confidence ≥ 0.85
+    Pending --> Failed: FAIL or Tests Fail
+    Pending --> Uncertain: Low Confidence
+
+    Uncertain --> Verified: Re-verify PASS
+    Uncertain --> Failed: Re-verify FAIL
+
+    Failed --> Banned: Same strategy fails 2x
+
+    Verified --> [*]: Return to User
+    Failed --> Pending: Generate New (Different Strategy)
+    Banned --> Pending: Force New Approach
+
+    note right of Verified
+        Solution cached
+        for future use
+    end note
+
+    note right of Banned
+        Strategy added to
+        forbidden list
+    end note
+```
+
+### ASCII State Diagram
+
+```
+                              ┌─────────────────┐
+                              │    [START]      │
+                              └────────┬────────┘
+                                       │
+                                       ▼
+                              ┌─────────────────┐
+              ┌──────────────▶│    PENDING      │◀──────────────┐
+              │               │  (New Solution) │               │
+              │               └────────┬────────┘               │
+              │                        │                        │
+              │         ┌──────────────┼──────────────┐         │
+              │         ▼              ▼              ▼         │
+              │  ┌───────────┐  ┌───────────┐  ┌───────────┐   │
+              │  │  VERIFIED │  │ UNCERTAIN │  │  FAILED   │   │
+              │  │   ✓       │  │    ?      │  │    ✗      │   │
+              │  └─────┬─────┘  └─────┬─────┘  └─────┬─────┘   │
+              │        │              │              │          │
+              │        ▼              │              ▼          │
+              │  ┌───────────┐        │       ┌───────────┐    │
+              │  │  CACHED   │        │       │  BANNED   │    │
+              │  │  (Done)   │        │       │ (Strategy)│    │
+              │  └───────────┘        │       └─────┬─────┘    │
+              │                       │             │          │
+              │                       └─────────────┼──────────┘
+              │                                     │
+              └─────────────────────────────────────┘
+                        (New attempt with different strategy)
+```
+
+## 4. Blind Spot Reduction Mathematics
+
+### Probability Diagram
+
+```
+┌────────────────────────────────────────────────────────────────────────┐
+│                    BLIND SPOT PROBABILITY MODEL                        │
+├────────────────────────────────────────────────────────────────────────┤
+│                                                                        │
+│  Single Model Self-Verification:                                       │
+│  ┌──────────────────────────────────────┐                              │
+│  │  P(miss) = P(Generator error)        │                              │
+│  │          × P(Verifier misses | same) │                              │
+│  │          ≈ ε × (1 - α)               │  where α ≈ 0.3 (correlated) │
+│  │          ≈ 0.15 × 0.7 = 0.105        │                              │
+│  └──────────────────────────────────────┘                              │
+│                                                                        │
+│  Cross-Model Verification (CMVK):                                      │
+│  ┌──────────────────────────────────────┐                              │
+│  │  P(miss) = P(Generator error)        │                              │
+│  │          × P(Verifier misses | diff) │                              │
+│  │          ≈ ε × (1 - β)               │  where β ≈ 0.7 (independent)│
+│  │          ≈ 0.15 × 0.3 = 0.045        │                              │
+│  └──────────────────────────────────────┘                              │
+│                                                                        │
+│  Risk Reduction Factor: 0.105 / 0.045 ≈ 2.3x                          │
+│                                                                        │
+│  ┌─────────────────────────────────────────────────────────────────┐  │
+│  │                                                                 │  │
+│  │   Same Model:     ████████████████████░░░░░░░  (70% miss rate) │  │
+│  │   Cross Model:    ████████░░░░░░░░░░░░░░░░░░░  (30% miss rate) │  │
+│  │                                                                 │  │
+│  └─────────────────────────────────────────────────────────────────┘  │
+│                                                                        │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+## 5. Results Comparison Chart (Template)
+
+```
+┌────────────────────────────────────────────────────────────────────────┐
+│                    HumanEval Pass@1 Results                            │
+├────────────────────────────────────────────────────────────────────────┤
+│                                                                        │
+│  100% ┤                                                                │
+│       │                                          ┌───┐                 │
+│   95% ┤                              ┌───┐       │███│ 93.1%           │
+│       │                  ┌───┐       │███│ 92.4% │███│                 │
+│   90% ┤                  │███│ 92.0% │███│       │███│                 │
+│       │      ┌───┐       │███│       │███│       │███│                 │
+│   85% ┤      │░░░│ 85.2% │███│       │███│       │███│                 │
+│       │ ┌───┐│░░░│       │███│       │███│       │███│                 │
+│   80% ┤ │░░░││░░░│ 84.1% │███│       │███│       │███│                 │
+│       │ │░░░││░░░│       │███│       │███│       │███│                 │
+│   75% ┼─┴───┴┴───┴───────┴───┴───────┴───┴───────┴───┴─────────────────│
+│         GPT-4o Claude   CMVK       CMVK         CMVK                   │
+│         alone  alone   (GPT+Gem)  (GPT+Claude) (o1+Gem)                │
+│                                                                        │
+│       ░░░ = Baseline (single model)                                    │
+│       ███ = CMVK (cross-model verification)                            │
+│                                                                        │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+## 6. Export Instructions
+
+### For draw.io / diagrams.net
+
+1. Go to [draw.io](https://app.diagrams.net/)
+2. Create new diagram
+3. Import Mermaid: Arrange → Insert → Advanced → Mermaid
+4. Paste the Mermaid code
+5. Export as PNG (300 DPI for papers) or SVG
+
+### For Lucidchart
+
+1. Use the ASCII diagrams as reference
+2. Recreate using Lucidchart shapes
+3. Export as PNG or PDF
+
+### For LaTeX (TikZ)
+
+See `figures/tikz/` directory for TikZ versions of these diagrams.
+
+---
+
+## Color Scheme (for consistent branding)
+
+| Element | Color | Hex |
+|---------|-------|-----|
+| Generator | Blue | #4A90D9 |
+| Verifier | Red/Orange | #E57373 |
+| Kernel | Green | #81C784 |
+| Graph of Truth | Purple | #9575CD |
+| Sandbox | Gray | #90A4AE |
+| Success | Green | #4CAF50 |
+| Failure | Red | #F44336 |
+| Warning | Yellow | #FFC107 |

modules/cmvk/docs/architecture.md

@@ -0,0 +1,345 @@
+# Architecture Documentation
+
+## Overview
+
+The Cross-Model Verification Kernel implements an **adversarial architecture** that uses **model diversity** to reduce the probability of shared blind spots in code generation systems.
+
+## Core Principle
+
+Traditional self-refinement systems suffer from correlated errors because the same model (or model family) both generates and verifies code. Our solution: use **different models** in an **adversarial relationship** where the verifier actively tries to find problems.
+
+## Architecture Components
+
+### 1. Model Interface Layer (`src/models.py`)
+
+```
+┌─────────────────────────────────────┐
+│     BaseModelInterface (ABC)        │
+│  - generate(prompt) → Response      │
+│  - get_model_info() → Dict          │
+└─────────────────────────────────────┘
+                ▲
+                │ implements
+                │
+    ┌───────────┴───────────┐
+    │                       │
+┌───────────────┐  ┌──────────────────┐
+│MockInterface  │  │ RealAPIInterface │
+│(for testing)  │  │(for production)  │
+└───────────────┘  └──────────────────┘
+```
+
+**Key Features:**
+- Abstraction over different LLM providers
+- Supports GPT-4o, Gemini 1.5 Pro, Claude, etc.
+- Mock implementation for testing
+- Extensible for new providers
+
+### 2. Generator Component (`src/generator.py`)
+
+```
+┌─────────────────────────────────────┐
+│          Generator                  │
+├─────────────────────────────────────┤
+│ Config:                             │
+│  - model: ModelProvider             │
+│  - temperature: 0.7 (creative)      │
+│  - max_tokens: 2000                 │
+├─────────────────────────────────────┤
+│ Methods:                            │
+│  - generate_code(task, language)    │
+│  - get_stats()                      │
+└─────────────────────────────────────┘
+```
+
+**Responsibilities:**
+- Generate code based on task descriptions
+- Use ONE specific model (e.g., GPT-4o)
+- Higher temperature for creativity
+- Decoupled from verification
+
+**Design Choice:** Intentionally has NO knowledge of verification to prevent bias.
+
+### 3. Verifier Component (`src/verifier.py`)
+
+```
+┌─────────────────────────────────────┐
+│          Verifier                   │
+├─────────────────────────────────────┤
+│ Config:                             │
+│  - model: ModelProvider (DIFFERENT!)│
+│  - temperature: 0.2 (deterministic) │
+│  - adversarial_mode: True           │
+├─────────────────────────────────────┤
+│ Methods:                            │
+│  - verify_code(code, desc, lang)    │
+│  - _build_adversarial_prompt()      │
+│  - _parse_verification_response()   │
+└─────────────────────────────────────┘
+```
+
+**Responsibilities:**
+- Perform hostile code review
+- Use DIFFERENT model than generator (e.g., Gemini 1.5 Pro)
+- Lower temperature for consistency
+- Find ALL issues, not just obvious ones
+
+**Adversarial Prompt Strategy:**
+```
+You are a HOSTILE code reviewer. Your job is to find ALL
+possible issues, bugs, vulnerabilities, and problems.
+
+Be extremely critical and thorough. Look for:
+- Security vulnerabilities
+- Logic errors and edge cases
+- Performance issues
+- Missing error handling
+- Type safety issues
+- Resource leaks
+- Concurrency problems
+- Input validation gaps
+- Any possible way the code could fail
+
+DO NOT be cooperative or lenient. Find every flaw.
+```
+
+### 4. Verification Kernel (`src/kernel.py`)
+
+```
+┌──────────────────────────────────────────────┐
+│         VerificationKernel                   │
+├──────────────────────────────────────────────┤
+│ Orchestrates:                                │
+│  1. Code Generation (via Generator)          │
+│  2. Adversarial Verification (via Verifier)  │
+│  3. Blind Spot Analysis (mathematical)       │
+├──────────────────────────────────────────────┤
+│ Enforces:                                    │
+│  - Model Diversity (raises error if same)    │
+│  - Different providers preferred             │
+├──────────────────────────────────────────────┤
+│ Returns:                                     │
+│  - Generated Code                            │
+│  - Verification Report                       │
+│  - Blind Spot Analysis                       │
+│  - Risk Reduction Metrics                    │
+└──────────────────────────────────────────────┘
+```
+
+**Model Diversity Enforcement:**
+```python
+if generator_config.model == verifier_config.model:
+    raise ValueError(
+        "Generator and Verifier must use DIFFERENT models"
+    )
+```
+
+## Mathematical Framework
+
+### Blind Spot Probability Model
+
+Let:
+- **p** = probability a single model misses a bug (e.g., 0.15 = 15%)
+- **ρ** = correlation coefficient between models (0 ≤ ρ ≤ 1)
+
+**Single Model (Self-Refinement):**
+```
+P(error) = p ≈ 15%
+```
+
+**Two Different Models (Adversarial):**
+```
+P(both_miss) = p² + ρ·p·(1-p)
+```
+
+**Correlation by Provider:**
+- Different providers (GPT ↔ Gemini): ρ ≈ 0.2
+- Same provider (GPT-4o ↔ GPT-4): ρ ≈ 0.5
+- Identical model: ρ = 1.0 (defeats purpose)
+
+**Example Calculation (p=0.15, ρ=0.2):**
+```
+P(both_miss) = 0.15² + 0.2·0.15·0.85
+             = 0.0225 + 0.0255
+             = 0.048 ≈ 4.8%
+
+Risk Reduction = 15% / 4.8% ≈ 3.1x
+```
+
+### Why This Works
+
+1. **Independent Perspectives**: Different models have different training data, architectures, and biases
+2. **Lower Correlation**: Cross-provider combinations minimize shared blind spots
+3. **Adversarial Stance**: Hostile review mode maximizes issue detection
+4. **Mathematical Guarantee**: Probability of both models missing the same bug is provably lower
+
+## Data Flow
+
+```
+User Task
+   ↓
+┌─────────────────────────────────────┐
+│   VerificationKernel.verify_task()  │
+└─────────────────────────────────────┘
+   ↓
+   ├──→ Generator (Model A: GPT-4o)
+   │       ↓
+   │    Generated Code
+   │       ↓
+   └──→ Verifier (Model B: Gemini)
+           ↓
+       Verification Report
+           ↓
+       Blind Spot Analysis
+           ↓
+       Final Result
+```
+
+## Security Benefits
+
+### Traditional Approach Problems:
+- ❌ Same model generates and checks → blind spots persist
+- ❌ Cooperative tone → misses critical issues
+- ❌ Model confirms its own biases
+
+### Our Adversarial Approach:
+- ✅ Different model with fresh perspective
+- ✅ Hostile review finds hidden issues
+- ✅ Cross-model validation reduces correlated errors
+- ✅ 3-4x better at catching bugs
+
+## Configuration Examples
+
+### Recommended: Cross-Provider
+```python
+# Best configuration for maximum diversity
+generator = GeneratorConfig(model=ModelProvider.GPT4O)
+verifier = VerifierConfig(model=ModelProvider.GEMINI_15_PRO)
+# ρ ≈ 0.2, Risk Reduction: ~3.1x
+```
+
+### Good: Same Provider, Different Models
+```python
+# Acceptable but higher correlation
+generator = GeneratorConfig(model=ModelProvider.GPT4O)
+verifier = VerifierConfig(model=ModelProvider.GPT4_TURBO)
+# ρ ≈ 0.5, Risk Reduction: ~1.7x
+```
+
+### Blocked: Same Model
+```python
+# System will REJECT this configuration
+generator = GeneratorConfig(model=ModelProvider.GPT4O)
+verifier = VerifierConfig(model=ModelProvider.GPT4O)
+# Raises ValueError: "Must use DIFFERENT models"
+```
+
+## Extending the System
+
+### Adding New Model Providers
+
+1. Add to `ModelProvider` enum:
+```python
+class ModelProvider(Enum):
+    YOUR_MODEL = "your-model-name"
+```
+
+2. Implement `BaseModelInterface`:
+```python
+class YourModelInterface(BaseModelInterface):
+    def generate(self, prompt: str, **kwargs):
+        # Your API call here
+        pass
+```
+
+3. Update correlation estimation if needed:
+```python
+def _get_provider_family(self, model: ModelProvider):
+    if model.value.startswith('your-prefix'):
+        return 'your-provider'
+```
+
+### Calibrating Error Probabilities
+
+To calibrate with real data:
+
+1. Run verification on known buggy code
+2. Measure:
+   - Single model miss rate → p
+   - Both models miss rate → actual combined probability
+3. Calculate actual correlation:
+   ```
+   ρ = (P_actual - p²) / (p·(1-p))
+   ```
+4. Update `_estimate_model_correlation()` with empirical values
+
+## Performance Considerations
+
+### API Costs
+- Two model calls per verification (Generator + Verifier)
+- Verifier typically uses more tokens (detailed analysis)
+- Consider batch processing for efficiency
+
+### Latency
+- Sequential: Generation then verification (~2x single model time)
+- Can be parallelized for multiple tasks
+- Trade-off: 2x time for 3-4x better quality
+
+### Optimization Strategies
+1. Cache generated code for similar tasks
+2. Batch multiple verifications
+3. Use faster models (e.g., Gemini Flash) for verifier when appropriate
+4. Implement early exit for obvious issues
+
+## Testing Strategy
+
+Our test suite covers:
+
+1. **Unit Tests** (`tests/test_kernel.py`)
+   - Model provider enumeration
+   - Mock interfaces
+   - Generator functionality
+   - Verifier functionality
+   - Kernel orchestration
+   - Model diversity enforcement
+   - Blind spot calculations
+
+2. **Integration Tests** (examples)
+   - Full pipeline execution
+   - Cross-model verification
+   - Error handling
+
+3. **Validation**
+   - 19 tests, 100% pass rate
+   - Mock models simulate provider differences
+   - Real-world usage patterns
+
+## Future Enhancements
+
+1. **Multi-Round Verification**
+   - Generator fixes issues found by verifier
+   - Multiple adversarial rounds until quality threshold met
+
+2. **Ensemble Verification**
+   - Use 3+ models for even better coverage
+   - Voting mechanism for issue severity
+
+3. **Adaptive Model Selection**
+   - Automatically choose best model pairs based on task type
+   - Learn optimal combinations from historical data
+
+4. **Real-Time Metrics**
+   - Track actual error correlation over time
+   - Refine probability estimates with production data
+
+5. **Specialized Verifiers**
+   - Security-focused verifier for sensitive code
+   - Performance-focused verifier for critical paths
+   - Domain-specific verifiers for specialized tasks
+
+## References
+
+- Adversarial Machine Learning
+- Ensemble Methods in ML
+- Code Review Best Practices
+- LLM Safety and Alignment Research