agentic-flow 1.9.0 → 1.9.2

package/agentic-llm/docs/FINETUNING_GUIDE.md

@@ -1,483 +0,0 @@
-# MCP Tool Fine-Tuning Guide for Claude Agent SDK
-
-Complete guide for fine-tuning Phi-4 on MCP tools for Claude Code integration.
-
-## 🎯 Overview
-
-This system fine-tunes Microsoft's Phi-4 model specifically for:
-- **MCP Tool Calling** - Read, Write, Edit, Bash, Grep, Glob
-- **Claude Agent SDK Integration** - Optimized for agentic workflows
-- **Structured Output Generation** - Tool names and parameters
-- **Production Deployment** - ONNX export with INT4/INT8 quantization
-
-### Expected Results
-
-**Target Metrics:**
-- ✅ Tool Accuracy: **>95%**
-- ✅ Parameter Accuracy: **>98%**
-- ✅ Tool Detection: **>97%**
-- ✅ Inference Speed: **2.67x-3.67x faster** (after ONNX quantization)
-
----
-
-## 📊 Dataset
-
-### MCP Tool Coverage
-
-The training dataset includes **40+ examples** across:
-
-| Tool | Examples | Complexity |
-|------|----------|------------|
-| **read** | 4 | Easy-Medium |
-| **write** | 3 | Medium |
-| **edit** | 3 | Easy |
-| **bash** | 5 | Easy-Medium |
-| **grep** | 4 | Medium-Hard |
-| **glob** | 4 | Easy-Medium |
-| **multi-step** | 3 | Hard |
-| **claude-flow** | 3 | Medium |
-
-### Dataset Augmentation
-
-Automatically generates variations:
-- Paraphrased queries ("Read" → "Show me", "Display")
-- Different phrasing patterns
-- **Total dataset: 60-100 examples** after augmentation
-
-### Example Format
-
-```json
-{
-  "text": "User: Read the package.json file\nAssistant: I'll read the package.json file.\n<tool_use>read(file_path='package.json')</tool_use>",
-  "tool_name": "read",
-  "difficulty": "easy"
-}
-```
-
----
-
-## ⚙️ Fine-Tuning Configuration
-
-### Optimal Parameters (Based on Research)
-
-```python
-MCPFineTuningConfig(
-    # Model
-    model_name="microsoft/Phi-4",
-
-    # Training - Optimized for tool calling
-    learning_rate=5e-5,        # Higher for tool learning
-    num_train_epochs=5,         # More epochs for tool patterns
-    per_device_train_batch_size=4,
-    gradient_accumulation_steps=4,
-    max_seq_length=1024,        # Shorter for tool calls
-
-    # LoRA - r=32 for optimal tool accuracy
-    lora_r=32,                  # Higher rank = better tool accuracy
-    lora_alpha=64,              # 2 * lora_r
-    lora_dropout=0.05,
-    lora_target_modules=[
-        "q_proj", "v_proj", "k_proj", "o_proj",
-        "gate_proj", "up_proj", "down_proj"
-    ],
-
-    # Overfitting prevention
-    early_stopping_patience=3,
-    validation_split=0.2,       # 20% for validation
-    weight_decay=0.01,
-
-    # MCP specific
-    tool_focused=True,
-    augment_dataset=True,
-    validation_frequency=100
-)
-```
-
-### Why These Parameters?
-
-**LoRA rank=32**:
-- 97%+ tool correctness accuracy (research-backed)
-- Optimal balance vs r=16 (cheaper) or r=64 (expensive)
-- 20% memory overhead (manageable on A100)
-
-**Learning rate=5e-5**:
-- Higher than standard (2e-5) for faster tool pattern learning
-- Validated through empirical testing
-
-**Epochs=5**:
-- Tool calling benefits from repetition
-- Early stopping prevents overfitting
-
----
-
-## 🚀 Quick Start
-
-### Option 1: Fly.io GPU (Recommended)
-
-```bash
-cd /workspaces/flow-cloud/docker/agentic-llm
-
-# Deploy to A100 GPU
-./deployment/deploy.sh
-```
-
-**Cost**: ~$15-20 for complete training (5-6 hours)
-
-### Option 2: Local GPU
-
-```bash
-# Run complete pipeline
-./run_finetuning.sh
-```
-
-**Requirements**:
-- NVIDIA GPU with 24GB+ VRAM (RTX 3090/4090, A100)
-- CUDA 12.1+
-- 50GB disk space
-
-### Option 3: Step-by-Step
-
-```bash
-# 1. Generate dataset
-python3 training/mcp_dataset.py
-
-# 2. Fine-tune
-python3 training/finetune_mcp.py
-
-# 3. Validate
-python3 validation/mcp_validator.py \
-    --model-path /app/checkpoints/mcp_finetuned/final_model
-
-# 4. Benchmark
-python3 benchmarks/finetune_comparison.py \
-    --finetuned /app/checkpoints/mcp_finetuned/final_model
-```
-
----
-
-## 📈 Training Pipeline
-
-### Step 1: Dataset Generation
-
-```bash
-python3 training/mcp_dataset.py
-```
-
-**Output:**
-- `/app/training/data/mcp_tools_dataset.json` (training data)
-- `/app/validation/data/mcp_validation_set.json` (validation set)
-
-**Dataset Stats:**
-- 40+ base examples
-- 60-100 total after augmentation
-- 80/20 train/validation split
-
-### Step 2: Fine-Tuning
-
-```bash
-python3 training/finetune_mcp.py
-```
-
-**Process:**
-1. Load Phi-4 base model
-2. Apply 4-bit quantization
-3. Configure LoRA adapters (r=32, alpha=64)
-4. Train for 5 epochs with early stopping
-5. Save best checkpoint
-
-**Duration**: 2-6 hours on A100 GPU
-
-**Output:**
-- `/app/checkpoints/mcp_finetuned/final_model/` (fine-tuned model)
-- `/app/checkpoints/mcp_finetuned/metrics.json` (training metrics)
-
-**Monitoring:**
-```bash
-# Watch training progress
-tail -f /app/checkpoints/mcp_finetuned/logs/training.log
-
-# Check GPU usage
-watch -n 1 nvidia-smi
-```
-
-### Step 3: Validation
-
-```bash
-python3 validation/mcp_validator.py \
-    --model-path /app/checkpoints/mcp_finetuned/final_model
-```
-
-**Tests:**
-- Tool detection accuracy
-- Tool name correctness
-- Parameter extraction
-- Response format
-
-**Output:**
-- `/app/validation/results/mcp_validation_results.json`
-
-**Success Criteria:**
-- ✅ Tool Accuracy > 95%
-- ✅ Parameter Accuracy > 98%
-- ✅ Tool Detection > 97%
-
-### Step 4: Benchmarking
-
-```bash
-python3 benchmarks/finetune_comparison.py \
-    --finetuned /app/checkpoints/mcp_finetuned/final_model
-```
-
-**Metrics:**
-- Before/after comparison
-- Tool-by-tool accuracy
-- Average inference time
-- Quality tier assessment
-
-**Output:**
-- `/app/benchmarks/finetune_results/finetune_comparison.json`
-
----
-
-## 🎯 Validation Framework
-
-### Test Cases (20 examples)
-
-```python
-[
-    # Basic tool calls
-    {"query": "Read package.json", "expected_tool": "read"},
-    {"query": "Create utils.ts", "expected_tool": "write"},
-
-    # Complex queries
-    {"query": "What's in the auth config", "expected_tool": "read"},
-    {"query": "Execute the linter", "expected_tool": "bash"},
-
-    # Multi-word tools
-    {"query": "Search for TODO comments", "expected_tool": "grep"},
-    {"query": "List all TypeScript files", "expected_tool": "glob"},
-]
-```
-
-### Validation Levels
-
-**Level 1: Tool Detection** (Did model attempt a tool call?)
-- Pattern: `<tool_use>...</tool_use>` present
-
-**Level 2: Tool Parsing** (Can we extract tool name?)
-- Format: `tool_name(param1='value1')`
-
-**Level 3: Tool Correctness** (Is it the right tool?)
-- Expected: `read`, Actual: `read` ✅
-
-**Level 4: Parameter Accuracy** (Are parameters correct?)
-- Expected: `file_path='package.json'`
-- Actual: `file_path='package.json'` ✅
-
----
-
-## 📊 Expected Results
-
-### Training Metrics
-
-```json
-{
-  "eval_loss": 0.45,
-  "eval_perplexity": 1.57,
-  "training_completed": true
-}
-```
-
-**Good**: Perplexity < 2.0
-**Excellent**: Perplexity < 1.5
-
-### Validation Results
-
-```json
-{
-  "total": 20,
-  "correct_tool": 19,
-  "correct_params": 18,
-  "accuracy": 95.0,
-  "param_accuracy": 90.0,
-  "quality_tier": "EXCELLENT - Production Ready"
-}
-```
-
-### Benchmark Comparison
-
-| Metric | Baseline | Fine-tuned | Improvement |
-|--------|----------|------------|-------------|
-| Tool Accuracy | 70-75% | **95-98%** | **+25-28%** |
-| Parameter Accuracy | 60-65% | **90-95%** | **+30%** |
-| Tool Detection | 80-85% | **97-99%** | **+15%** |
-
----
-
-## 🔧 ONNX Export & Quantization
-
-After successful fine-tuning:
-
-### Step 1: Export to ONNX
-
-```bash
-python3 quantization/quantize.py \
-    --model-path /app/checkpoints/mcp_finetuned/final_model \
-    --output-dir /app/models/quantized
-```
-
-### Step 2: Quantization Options
-
-**INT8 (Recommended for production)**:
-- 75% smaller (14GB → 3.5GB)
-- 2.67x faster
-- <1% accuracy loss
-- Tool accuracy: **~94-96%**
-
-**INT4 (Maximum speed)**:
-- 87.5% smaller (14GB → 1.75GB)
-- 3.67x faster
-- <3% accuracy loss
-- Tool accuracy: **~92-94%**
-
-### Step 3: Validate Quantized Model
-
-```bash
-python3 validation/mcp_validator.py \
-    --model-path /app/models/quantized/phi4_int8.onnx
-```
-
-**Target**: Maintain >90% tool accuracy after quantization
-
----
-
-## 🚀 Deployment
-
-### Claude Agent SDK Integration
-
-```python
-from claude_sdk.integration import ClaudeSDKOptimizedInference
-
-# Load fine-tuned + quantized model
-engine = ClaudeSDKOptimizedInference(
-    model_path="/app/models/quantized/phi4_int8.onnx",
-    use_onnx=True,
-    quantization="int8"
-)
-
-# Generate MCP tool call
-result = engine.generate_response("Read the config file")
-
-print(result['tool_call'].tool_name)     # 'read'
-print(result['tool_call'].parameters)    # {'file_path': 'config'}
-print(result['latency_ms'])              # ~45ms (vs 120ms baseline)
-```
-
-### Production Configuration
-
-```yaml
-model:
-  path: /app/models/quantized/phi4_int8.onnx
-  quantization: int8
-  execution_provider: CUDAExecutionProvider
-
-inference:
-  max_tokens: 150
-  temperature: 0.1
-  batch_size: 4
-
-monitoring:
-  track_tool_accuracy: true
-  log_failed_calls: true
-  alert_threshold: 90%
-```
-
----
-
-## 💰 Cost Analysis
-
-### Training Costs (Fly.io A100 GPU)
-
-| Component | Duration | Cost |
-|-----------|----------|------|
-| Dataset generation | 5 min | $0.25 |
-| Fine-tuning (5 epochs) | 4-6 hours | $12-18 |
-| Validation | 10 min | $0.50 |
-| Benchmarking | 10 min | $0.50 |
-| **Total** | **5-7 hours** | **$13.25-19.25** |
-
-### Inference Cost Savings
-
-**Per 1M tokens:**
-- Claude API: $15.00
-- Fine-tuned INT8: $5.63 (**62.5% savings**)
-- Fine-tuned INT4: $4.08 (**72.8% savings**)
-
-**Break-even**: ~1.3M tokens
-
----
-
-## 🐛 Troubleshooting
-
-### Issue: Low Tool Accuracy (<85%)
-
-**Solutions:**
-1. Increase training epochs (5 → 10)
-2. Add more training examples
-3. Increase LoRA rank (32 → 64)
-4. Lower learning rate (5e-5 → 2e-5)
-
-### Issue: Overfitting
-
-**Symptoms**: Train accuracy > Val accuracy by >10%
-
-**Solutions:**
-1. Enable early stopping (already configured)
-2. Increase weight decay (0.01 → 0.05)
-3. Add more validation data (20% → 30%)
-4. Reduce LoRA rank (32 → 16)
-
-### Issue: OOM (Out of Memory)
-
-**Solutions:**
-1. Reduce batch size (4 → 2)
-2. Increase gradient accumulation (4 → 8)
-3. Reduce sequence length (1024 → 512)
-4. Use A100 80GB instead of 40GB
-
-### Issue: Slow Training
-
-**Solutions:**
-1. Enable gradient checkpointing (already enabled)
-2. Use mixed precision (fp16, already enabled)
-3. Reduce evaluation frequency (100 → 200 steps)
-
----
-
-## 📚 Additional Resources
-
-- **Research**: `PHI4_FINETUNING_RESEARCH.md` - Comprehensive optimization research
-- **Optimization**: `OPTIMIZATION_RESULTS.md` - Quantization benchmarks
-- **Deployment**: `DEPLOYMENT_GUIDE.md` - Fly.io GPU setup
-- **Code**: All training scripts in `training/`, validation in `validation/`
-
----
-
-## ✅ Success Checklist
-
-Before deploying to production:
-
-- [ ] Training completed without errors
-- [ ] Validation accuracy >95%
-- [ ] Parameter accuracy >90%
-- [ ] Tool detection >97%
-- [ ] ONNX export successful
-- [ ] Quantized model validated
-- [ ] Benchmarks show improvement
-- [ ] Integration tested with Claude SDK
-- [ ] Monitoring configured
-- [ ] Rollback plan in place
-
----
-
-**Ready to fine-tune? Run `./run_finetuning.sh` to start!**

package/agentic-llm/docs/INDEX.md

@@ -1,43 +0,0 @@
-# Agentic LLM Documentation Index
-
-## Quick Start
-- **[README.md](./README.md)** - Project overview and main documentation
-- **[QUICKSTART.md](./QUICKSTART.md)** - Quick start guide
-
-## Usage & Deployment
-- **[USAGE_GUIDE.md](./USAGE_GUIDE.md)** - Comprehensive usage documentation
-- **[DEPLOYMENT_GUIDE.md](./DEPLOYMENT_GUIDE.md)** - Local and production deployment guide
-- **[CLOUD_RUN_DEPLOYMENT.md](./CLOUD_RUN_DEPLOYMENT.md)** - Google Cloud Run deployment
-- **[DEPLOYMENT_STATUS.md](./DEPLOYMENT_STATUS.md)** - Current deployment status
-- **[DEPLOYMENT_SUMMARY.md](./DEPLOYMENT_SUMMARY.md)** - Deployment summary
-- **[DEPLOYMENT_READY.txt](./DEPLOYMENT_READY.txt)** - Deployment checklist
-
-## Fine-tuning & Training
-- **[FINETUNING_GUIDE.md](./FINETUNING_GUIDE.md)** - Complete fine-tuning guide
-- **[PHI4_FINETUNING_RESEARCH.md](./PHI4_FINETUNING_RESEARCH.md)** - Phi-4 model fine-tuning research
-
-## Optimization & Performance
-- **[OPTIMIZATION_RESULTS.md](./OPTIMIZATION_RESULTS.md)** - Performance optimization results
-- **[BENCHMARK_OPTIMIZATION_GUIDE.md](./BENCHMARK_OPTIMIZATION_GUIDE.md)** - Benchmarking and optimization guide
-
-## Directory Structure
-
-```
-agentic-llm/
-├── docs/              # Documentation (you are here)
-├── scripts/           # Shell scripts for running tasks
-├── deployment/        # Deployment configuration (fly.toml)
-├── configs/           # Model and training configurations
-├── models/            # Model files and checkpoints
-├── training/          # Training scripts and data
-├── validation/        # Validation and testing
-├── benchmarks/        # Benchmark scripts
-├── quantization/      # Quantization tools
-├── claude_sdk/        # Claude SDK integration
-└── cloudrun/          # Cloud Run specific files
-```
-
-## Key Files
-- `requirements.txt` - Python dependencies
-- `Dockerfile` - Container configuration
-- `.dockerignore` - Docker build exclusions