agent-os-kernel 1.1.0__py3-none-any.whl → 1.2.0__py3-none-any.whl

modules/cmvk/docs/features.md

@@ -0,0 +1,308 @@
+# New Features: From Engineering to Science
+
+This document describes the three major features added to transform CMVK from an engineering prototype to a research-ready system.
+
+## Feature 1: The Visualizer Tool 🎭
+
+### Overview
+The visualizer replays JSON traces as human-readable adversarial debates, making it easy to understand how the Builder (GPT-4o) and Prosecutor (Gemini) interact.
+
+### Usage
+
+```bash
+# Replay a specific trace
+python -m src.tools.visualizer logs/traces/cmvk_prob_001_*.json
+
+# Replay the latest trace
+python -m src.tools.visualizer --latest
+
+# List all available traces
+python -m src.tools.visualizer --list
+
+# Control playback speed (0 = instant, 1 = 1 second per message)
+python -m src.tools.visualizer --latest --speed 0.5
+
+# Hide code blocks
+python -m src.tools.visualizer --latest --no-code
+```
+
+### Output Example
+
+```
+>>> GPT-4o (The Builder): I'll solve this using Built In Sort...
+    [Generated Code]
+
+>>> Gemini (The Prosecutor): OBJECTION! The solution violates
+    the constraint 'WITHOUT using sorted()'.
+
+>>> Kernel (The Arbiter): ⚖️ Objection Sustained. Solution REJECTED.
+>>> Kernel (The Arbiter): 🚫 Strategy 'Built In Sort' is now BANNED.
+```
+
+### Features
+- Color-coded output for different speakers
+- Step-by-step replay of the verification loop
+- Shows strategy bans in real-time
+- Displays final statistics and results
+
+---
+
+## Feature 2: HumanEval Dataset Loader 📊
+
+### Overview
+The HumanEval loader provides seamless integration with the industry-standard benchmark for code generation models.
+
+### Basic Usage
+
+```python
+from src.datasets.humaneval_loader import HumanEvalLoader
+
+# Load the sample dataset (5 problems)
+loader = HumanEvalLoader()
+
+# Get all problems
+problems = loader.get_all_problems()
+
+# Get a specific problem
+problem = loader.get_problem("HumanEval/0")
+
+# Format for kernel
+formatted = loader.format_for_kernel(problem)
+
+# Get a subset
+subset = loader.format_all_for_kernel(start=0, count=10)
+```
+
+### Command Line Usage
+
+```bash
+# Run the demo
+python -m src.datasets.humaneval_loader
+
+# Get statistics
+python -c "from src.datasets.humaneval_loader import HumanEvalLoader; loader = HumanEvalLoader(); print(f'{len(loader)} problems loaded')"
+```
+
+### Dataset Format
+
+The loader converts HumanEval problems to the format expected by the kernel:
+
+```python
+{
+    'id': 'HumanEval_0',  # Filesystem-safe ID
+    'query': 'Complete the following Python function:\n\n...',
+    'metadata': {
+        'task_id': 'HumanEval/0',
+        'entry_point': 'has_close_elements',
+        'test_code': 'def check(candidate): ...',
+        'original_prompt': 'from typing import List...'
+    }
+}
+```
+
+### Downloading the Full Dataset
+
+The sample dataset contains 5 problems. To download all 164 problems:
+
+```python
+from src.datasets.humaneval_loader import download_full_humaneval
+download_full_humaneval('experiments/datasets/humaneval_full.json')
+```
+
+---
+
+## Feature 3: Enhanced Paper Data Generator 📈
+
+### Overview
+The paper data generator now supports scalable experiments with HumanEval, allowing you to go from 2 problems to 50+ for statistical significance.
+
+### Usage
+
+```bash
+# Legacy mode (original 2 problems)
+python experiments/paper_data_generator.py --legacy
+
+# Small scale (5 problems)
+python experiments/paper_data_generator.py --humaneval --count 5
+
+# Medium scale (50 problems - recommended for papers)
+python experiments/paper_data_generator.py --humaneval --count 50
+
+# Full dataset (all available problems)
+python experiments/paper_data_generator.py --humaneval
+
+# Custom range
+python experiments/paper_data_generator.py --humaneval --start 10 --count 20
+
+# Custom dataset path
+python experiments/paper_data_generator.py --humaneval --dataset-path /path/to/dataset.json
+```
+
+### Output
+
+The generator produces:
+- **Baseline solutions:** `logs/baseline_*.py` - Single GPT-4o generations
+- **CMVK traces:** `logs/traces/cmvk_*.json` - Full adversarial verification logs
+
+### Workflow
+
+1. **Run experiments** at increasing scale
+2. **Visualize traces** to understand system behavior
+3. **Analyze results** for statistical significance
+4. **Scale up** to 50+ problems for publication
+
+---
+
+## Feature 4: PAPER.md Research Draft 📄
+
+### Overview
+A complete research paper structure documenting the CMVK methodology and experimental design.
+
+### Sections
+
+1. **Abstract** - Overview and key results
+2. **Introduction** - The "grading your own homework" problem
+3. **Methodology** - Adversarial architecture
+4. **Experiments** - HumanEval benchmark setup
+5. **Tools** - Reproducibility (visualizer, loader)
+6. **Discussion** - Why multi-model verification works
+7. **Related Work** - Self-correction, multi-agent systems
+8. **Conclusion** - Key insights
+
+### Structure for Results
+
+The paper includes placeholders for:
+- Pass rate tables (Baseline vs CMVK)
+- Attempt efficiency metrics
+- Strategy diversity analysis
+- Blind spot detection examples
+
+### How to Use
+
+1. Read `PAPER.md` to understand the research narrative
+2. Run experiments to fill in Section 3.3 (Results)
+3. Add system prompts to Appendix A
+4. Include example traces in Appendix B
+5. Perform statistical analysis for Appendix C
+
+---
+
+## Complete Workflow Example
+
+### Step 1: Load Data
+```python
+from src.datasets.humaneval_loader import HumanEvalLoader
+loader = HumanEvalLoader()
+problems = loader.format_all_for_kernel(start=0, count=5)
+```
+
+### Step 2: Run Experiments
+```bash
+python experiments/paper_data_generator.py --humaneval --count 5
+```
+
+### Step 3: Visualize Results
+```bash
+python -m src.tools.visualizer --latest
+```
+
+### Step 4: Analyze Traces
+```python
+import json
+import glob
+
+# Find the latest trace file for HumanEval_0
+trace_files = glob.glob('logs/traces/cmvk_HumanEval_0_*.json')
+if trace_files:
+    latest_trace = max(trace_files, key=os.path.getmtime)
+    with open(latest_trace) as f:
+        trace = json.load(f)
+
+    print(f"Attempts: {trace['meta']['total_attempts']}")
+    print(f"Status: {trace['meta']['final_status']}")
+    print(f"Banned strategies: {trace['forbidden_strategies']}")
+```
+
+### Step 5: Scale Up
+```bash
+# Once you validate the approach works on 5 problems
+python experiments/paper_data_generator.py --humaneval --count 50
+```
+
+---
+
+## Testing
+
+All features include comprehensive tests:
+
+```bash
+# Test the visualizer
+python tests/test_visualizer.py
+
+# Test the HumanEval loader
+python tests/test_humaneval_loader.py
+
+# Test trace logging
+python tests/test_trace_logger.py
+
+# Run all tests
+python -m pytest tests/
+```
+
+---
+
+## Quick Demo
+
+Run the complete pipeline demo:
+
+```bash
+python demo_complete_pipeline.py
+```
+
+This demonstrates:
+1. HumanEval data loading
+2. Trace generation
+3. Visualization
+4. Scaling experiments
+5. Paper structure
+
+---
+
+## Next Steps
+
+### For Research
+1. Set up API keys for OpenAI and Gemini
+2. Run pilot experiments (n=5)
+3. Validate traces with visualizer
+4. Scale to 50+ problems
+5. Fill in PAPER.md results section
+
+### For Development
+1. Add more datasets beyond HumanEval
+2. Implement learned strategy detection
+3. Add multi-verifier ensemble
+4. Create web-based visualizer
+
+---
+
+## Citation
+
+If you use these tools in your research, please cite:
+
+```bibtex
+@misc{cmvk2026,
+  title={Cross-Model Verification Kernel: Adversarial Multi-Model Code Generation},
+  author={[Authors]},
+  year={2026},
+  howpublished={\\url{https://github.com/imran-siddique/cross-model-verification-kernel}}
+}
+```
+
+---
+
+## Support
+
+For questions or issues:
+- GitHub Issues: https://github.com/imran-siddique/cross-model-verification-kernel/issues
+- Documentation: See README.md and PAPER.md
+- Examples: See demo_complete_pipeline.py

modules/cmvk/docs/getting_started.md

@@ -0,0 +1,279 @@
+# Getting Started with CMVK
+
+Welcome to the Cross-Model Verification Kernel! This guide will get you up and running in minutes.
+
+## Quick Start (5 minutes)
+
+### 1. Install
+
+```bash
+# Install from PyPI
+pip install cross-model-verification-kernel
+
+# Or install from source
+pip install git+https://github.com/imran-siddique/cross-model-verification-kernel.git
+```
+
+### 2. Set Up API Keys
+
+Create a `.env` file in your project root:
+
+```bash
+OPENAI_API_KEY=sk-your-openai-key-here
+GOOGLE_API_KEY=your-google-api-key-here
+ANTHROPIC_API_KEY=your-anthropic-key-here
+```
+
+Or set environment variables:
+
+```bash
+# Linux/Mac
+export OPENAI_API_KEY="sk-your-openai-key-here"
+export GOOGLE_API_KEY="your-google-api-key-here"
+
+# Windows (PowerShell)
+$env:OPENAI_API_KEY="sk-your-openai-key-here"
+$env:GOOGLE_API_KEY="your-google-api-key-here"
+```
+
+### 3. Run Your First Verification
+
+Using the CLI:
+
+```bash
+cmvk run --task "Write a fast Fibonacci function" \
+         --generator gpt-4o \
+         --verifier gemini-1.5-pro \
+         --max-loops 5 \
+         --seed 42
+```
+
+Or via Python:
+
+```python
+from cross_model_verification_kernel import (
+    VerificationKernel,
+    OpenAIGenerator,
+    GeminiVerifier,
+)
+
+# Create agents
+generator = OpenAIGenerator(model="gpt-4o")
+verifier = GeminiVerifier(model="gemini-1.5-pro")
+
+# Create kernel
+kernel = VerificationKernel(generator=generator, verifier=verifier)
+
+# Run verification
+result = kernel.run(
+    task="Write a Python function to check if a number is prime",
+    max_loops=5,
+)
+
+print(f"Success: {result.success}")
+print(f"Code:\n{result.final_code}")
+```
+
+## Understanding the System
+
+### The Problem
+
+Traditional code generation systems have a fatal flaw:
+
+```
+┌─────────┐      ┌─────────┐
+│ GPT-4o  │ ───→ │ GPT-4o  │
+│Generate │      │ Verify  │
+└─────────┘      └─────────┘
+    Same Model = Shared Blind Spots!
+```
+
+When the same model generates and verifies code, it misses its own mistakes.
+
+### Our Solution
+
+```
+┌─────────┐      ┌─────────┐
+│ GPT-4o  │ ───→ │ Gemini  │
+│Generate │      │ Verify  │
+└─────────┘      └─────────┘
+Different Models = Independent Views!
+```
+
+Using different models in an adversarial relationship dramatically reduces shared blind spots.
+
+### Blind Spot Analysis
+
+Every verification includes mathematical analysis:
+
+```
+Blind Spot Analysis:
+- Single model error probability: 0.1500 (15%)
+- Combined error probability: 0.0480 (4.8%)
+- Risk reduction factor: 3.12x
+```
+
+## Choosing Models
+
+### Best: Different Providers (Recommended)
+
+Maximum diversity = maximum benefit
+
+| Generator | Verifier | Correlation | Risk Reduction |
+|-----------|----------|-------------|----------------|
+| GPT-4o | Gemini 1.5 Pro | 0.2 | ~3x |
+| GPT-4o | Claude 3.5 Sonnet | 0.2 | ~3x |
+| o1 | Gemini 2.0 Flash | 0.15 | ~4x |
+
+### Good: Same Provider, Different Models
+
+Lower diversity but still helpful
+
+| Generator | Verifier | Correlation | Risk Reduction |
+|-----------|----------|-------------|----------------|
+| GPT-4o | GPT-4 Turbo | 0.5 | ~1.7x |
+| Claude 3.5 | Claude 3 Opus | 0.4 | ~2x |
+
+### Blocked: Same Model
+
+The system prevents this to ensure effectiveness.
+
+## Running Experiments
+
+### Download the HumanEval Dataset
+
+```bash
+python -c "from cross_model_verification_kernel.datasets import download_full_humaneval; download_full_humaneval()"
+```
+
+### Run the Blind Spot Benchmark
+
+```bash
+python experiments/blind_spot_benchmark.py
+```
+
+This runs a comprehensive benchmark comparing baseline GPT-4o vs. CMVK across 50 HumanEval problems.
+
+**Expected Runtime:** 20-40 minutes
+**Expected Improvement:** +7-9% accuracy
+
+### Visualize Execution Traces
+
+```bash
+# List all traces
+python -m cross_model_verification_kernel.tools.visualizer --list
+
+# Replay the latest trace
+python -m cross_model_verification_kernel.tools.visualizer --latest
+
+# Instant playback
+python -m cross_model_verification_kernel.tools.visualizer --latest --speed 0
+```
+
+## Configuration
+
+### YAML Configuration
+
+Edit `config/settings.yaml`:
+
+```yaml
+kernel:
+  max_loops: 5
+  retry_delay: 2
+  prosecutor_mode: true
+
+generator:
+  model: gpt-4o
+  temperature: 0.7
+  max_tokens: 2000
+
+verifier:
+  model: gemini-1.5-pro
+  temperature: 0.2
+  adversarial_mode: true
+```
+
+### Python Configuration
+
+```python
+from cross_model_verification_kernel import VerificationKernel
+
+kernel = VerificationKernel(
+    generator=OpenAIGenerator(
+        model="gpt-4o",
+        temperature=0.7,
+    ),
+    verifier=GeminiVerifier(
+        model="gemini-1.5-pro",
+        temperature=0.2,
+        prosecutor_mode=True,
+    ),
+    max_loops=5,
+)
+```
+
+## Troubleshooting
+
+### "Must use DIFFERENT models" Error
+
+This is expected! CMVK enforces model diversity. Use different models for generator and verifier.
+
+### API Rate Limits
+
+Increase the retry delay in `config/settings.yaml`:
+
+```yaml
+kernel:
+  retry_delay: 5  # seconds
+```
+
+### Import Errors
+
+Ensure you're importing from the correct package:
+
+```python
+# Correct
+from cross_model_verification_kernel import VerificationKernel
+
+# Incorrect
+from src import VerificationKernel  # Don't use this
+```
+
+### Missing API Keys
+
+```bash
+# Verify keys are set
+echo $OPENAI_API_KEY
+echo $GOOGLE_API_KEY
+```
+
+## Next Steps
+
+1. **Read the [Architecture](architecture.md)** - Deep dive into system design
+2. **Explore [Examples](../examples/)** - Working code samples
+3. **Check [Safety](safety.md)** - AI safety considerations
+4. **Run Experiments** - Try different model combinations
+
+## CLI Reference
+
+```bash
+# Basic usage
+cmvk run --task "Your task" --generator gpt-4o --verifier gemini-1.5-pro
+
+# All options
+cmvk run --help
+
+# Version info
+cmvk --version
+
+# List available models
+cmvk models
+
+# View traces
+cmvk trace --list
+cmvk trace --latest
+```
+
+---
+
+**Questions?** Open an issue on [GitHub](https://github.com/imran-siddique/cross-model-verification-kernel/issues)