agentic-flow 1.8.15 → 1.9.0

package/agentic-llm/docs/QUICKSTART.md

@@ -0,0 +1,124 @@
+# Quick Start - Agentic LLM Phi-4 Training
+
+Get started in 5 minutes with GPU training on Fly.io.
+
+## Prerequisites (2 minutes)
+
+```bash
+# 1. Install Fly.io CLI
+curl -L https://fly.io/install.sh | sh
+
+# 2. Login to Fly.io
+flyctl auth login
+
+# 3. Get Hugging Face token
+# Visit: https://huggingface.co/settings/tokens
+```
+
+## Deploy & Train (3 minutes)
+
+```bash
+# 1. Navigate to project
+cd docker/agentic-llm
+
+# 2. Run deployment script
+./deployment/deploy.sh
+
+# Enter your Hugging Face token when prompted
+# Enter your WandB API key (optional, press Enter to skip)
+```
+
+That's it! Training will start automatically on an A100 GPU.
+
+## Monitor Progress
+
+```bash
+# View real-time logs
+flyctl logs -a agentic-llm-phi4 -f
+
+# Check GPU utilization
+flyctl ssh console -a agentic-llm-phi4 -C "nvidia-smi"
+```
+
+## What Happens Next?
+
+The system will:
+
+1. **Train** Phi-4 with LoRA (2-6 hours)
+2. **Validate** with overfitting detection
+3. **Save** best checkpoint automatically
+4. **Stop** after completion (if auto-stop enabled)
+
+## Access Results
+
+```bash
+# SSH into instance
+flyctl ssh console -a agentic-llm-phi4
+
+# View training metrics
+cat /app/checkpoints/training_metrics.json
+
+# Download model
+flyctl ssh sftp shell -a agentic-llm-phi4
+get /app/checkpoints/final_model ./
+```
+
+## Next Steps
+
+After training completes:
+
+```bash
+# 1. Quantize model
+python3 quantization/quantize.py \
+  --model-path /app/checkpoints/final_model \
+  --output-dir /app/models/quantized
+
+# 2. Run benchmarks
+python3 benchmarks/benchmark.py \
+  --baseline /app/checkpoints/final_model \
+  --quantized-int8 /app/models/quantized/phi4_int8.onnx
+
+# 3. Deploy to Claude Agent SDK
+cp /app/models/quantized/phi4_int8.onnx \
+   /claude-agent-sdk/models/
+```
+
+## Cost Management
+
+```bash
+# Stop instance (save costs)
+flyctl scale count 0 -a agentic-llm-phi4
+
+# Restart when needed
+flyctl scale count 1 -a agentic-llm-phi4
+```
+
+**Estimated cost**: $6-18 per training run (2-6 hours on A100 40GB)
+
+## Troubleshooting
+
+**Deployment fails?**
+```bash
+flyctl logs -a agentic-llm-phi4
+```
+
+**Need more memory?**
+```bash
+flyctl scale vm a100-80gb -a agentic-llm-phi4
+```
+
+**Training too slow?**
+- Check batch size in `training/train.py`
+- Reduce `per_device_train_batch_size` if OOM
+- Increase `gradient_accumulation_steps` to maintain effective batch size
+
+## Support
+
+- 📖 Full docs: `README.md`
+- 🚀 Deployment guide: `docs/DEPLOYMENT_GUIDE.md`
+- 📚 Usage guide: `docs/USAGE_GUIDE.md`
+- 💬 Issues: Open GitHub issue
+
+---
+
+**Ready? Run `./deployment/deploy.sh` to start!**

package/agentic-llm/docs/README.md

@@ -0,0 +1,389 @@
+# Agentic LLM Training System for Phi-4
+
+GPU-accelerated training, quantization, and deployment system for optimizing Microsoft's Phi-4 model for Claude Agent SDK and MCP tools.
+
+## 🎯 Features
+
+- **GPU Training on Fly.io** - NVIDIA A100 40GB GPU support
+- **Overfitting Prevention** - Early stopping, validation monitoring, learning curves
+- **ONNX Quantization** - INT8/INT4 quantization for efficient inference
+- **MCP Tool Optimization** - Fine-tuned for Claude SDK tool calling
+- **Comprehensive Benchmarking** - Performance, accuracy, and memory metrics
+- **Modular Architecture** - Clean separation of training, validation, and deployment
+
+## 📁 Project Structure
+
+```
+docker/agentic-llm/
+├── training/           # GPU training system
+│   └── train.py       # Main training script with LoRA and overfitting prevention
+├── validation/        # Validation and overfitting detection
+│   └── validator.py   # Comprehensive validation suite
+├── quantization/      # ONNX quantization pipeline
+│   └── quantize.py    # INT8/INT4 quantization
+├── deployment/        # Fly.io deployment scripts
+│   └── deploy.sh      # Automated deployment to GPU instances
+├── benchmarks/        # Performance benchmarking
+│   └── benchmark.py   # Comparative benchmarking system
+├── docs/              # Documentation
+├── configs/           # Configuration files
+├── models/            # Model storage (persistent volume)
+├── Dockerfile         # GPU-enabled container
+└── fly.toml           # Fly.io configuration
+```
+
+## 🚀 Quick Start
+
+### 1. Prerequisites
+
+```bash
+# Install Fly.io CLI
+curl -L https://fly.io/install.sh | sh
+
+# Login to Fly.io
+flyctl auth login
+
+# Verify CUDA availability (for local testing)
+python3 -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
+```
+
+### 2. Deploy to Fly.io
+
+```bash
+cd docker/agentic-llm
+
+# Make deployment script executable
+chmod +x deployment/deploy.sh
+
+# Deploy with GPU
+./deployment/deploy.sh
+```
+
+The script will:
+- ✅ Create A100 GPU instance
+- ✅ Set up persistent volumes (100GB models, 50GB checkpoints)
+- ✅ Configure secrets (HF_TOKEN, WANDB_API_KEY)
+- ✅ Deploy and start training
+
+### 3. Monitor Training
+
+```bash
+# View real-time logs
+flyctl logs -a agentic-llm-phi4
+
+# SSH into container
+flyctl ssh console -a agentic-llm-phi4
+
+# Check status
+flyctl status -a agentic-llm-phi4
+```
+
+## 🎓 Training Pipeline
+
+### Local Training (Development)
+
+```bash
+# Build Docker image
+docker build -t agentic-llm:latest .
+
+# Run training with GPU
+docker run --gpus all \
+  -v $(pwd)/models:/app/models \
+  -v $(pwd)/checkpoints:/app/checkpoints \
+  -e HF_TOKEN=your_token \
+  -e WANDB_API_KEY=your_key \
+  agentic-llm:latest
+```
+
+### Training Configuration
+
+Edit training parameters in `training/train.py`:
+
+```python
+@dataclass
+class AgenticTrainingConfig:
+    # Model
+    model_name: str = "microsoft/Phi-4"
+
+    # Training hyperparameters
+    learning_rate: float = 2e-5
+    num_train_epochs: int = 3
+    per_device_train_batch_size: int = 4
+    gradient_accumulation_steps: int = 4
+
+    # LoRA config
+    lora_r: int = 16
+    lora_alpha: int = 32
+    lora_dropout: float = 0.05
+
+    # Overfitting prevention
+    early_stopping_patience: int = 3
+    validation_split: float = 0.15
+    weight_decay: float = 0.01
+
+    # Quantization
+    use_4bit: bool = True
+```
+
+### Overfitting Prevention
+
+The system includes multiple mechanisms:
+
+1. **Early Stopping** - Monitors validation loss with patience
+2. **Validation Split** - 15% validation, 5% test data
+3. **Weight Decay** - L2 regularization (0.01)
+4. **Dropout** - LoRA dropout (0.05)
+5. **Learning Curves** - Automatic plotting and analysis
+6. **Generalization Gap Monitoring** - Tracks train/val difference
+
+## 🔧 Quantization Pipeline
+
+### Run Quantization
+
+```bash
+# After training completes
+python3 quantization/quantize.py \
+  --model-path /app/checkpoints/final_model \
+  --output-dir /app/models/quantized \
+  --validate
+```
+
+### Quantization Levels
+
+| Format | Size Reduction | Speed Improvement | Accuracy Loss |
+|--------|---------------|-------------------|---------------|
+| FP32   | Baseline      | 1x                | 0%            |
+| INT8   | ~75%          | 2-3x              | <2%           |
+| INT4   | ~87.5%        | 3-4x              | <5%           |
+
+### Output Files
+
+```
+models/quantized/
+├── phi4_fp32.onnx              # Original ONNX
+├── phi4_int8.onnx              # INT8 quantized
+├── phi4_int4.onnx              # INT4 quantized
+└── quantization_results.json   # Benchmark results
+```
+
+## 📊 Benchmarking
+
+### Run Comprehensive Benchmarks
+
+```bash
+python3 benchmarks/benchmark.py \
+  --baseline /app/checkpoints/final_model \
+  --quantized-int8 /app/models/quantized/phi4_int8.onnx \
+  --quantized-int4 /app/models/quantized/phi4_int4.onnx \
+  --output-dir /app/benchmarks/results
+```
+
+### Metrics Measured
+
+- **Performance**
+  - Inference latency (ms)
+  - Throughput (tokens/sec)
+  - Memory usage (MB)
+
+- **Quality**
+  - Perplexity
+  - MCP tool calling accuracy
+  - Response quality
+
+- **Efficiency**
+  - Model size (MB)
+  - GPU utilization
+  - Cost per 1M tokens
+
+### Benchmark Output
+
+```
+benchmarks/results/
+├── benchmark_report.json        # Detailed metrics
+├── benchmark_comparison.png     # Visual comparisons
+└── performance_summary.txt      # Human-readable summary
+```
+
+## 🛡️ Validation System
+
+### Run Validation
+
+```bash
+python3 validation/validator.py \
+  --model-path /app/checkpoints/final_model \
+  --output-dir /app/validation/results
+```
+
+### Validation Outputs
+
+- **Overfitting Detection**
+  - Train vs validation loss curves
+  - Overfitting ratio
+  - Generalization gap
+
+- **Quality Metrics**
+  - Test set perplexity
+  - Tool calling accuracy
+  - Response coherence
+
+- **Visualizations**
+  - Learning curves
+  - Loss distributions
+  - Metric trends
+
+## 🎯 MCP Tool Optimization
+
+### Training Data
+
+The system includes synthetic MCP tool calling examples:
+
+```python
+# Tool calling patterns
+"User: Search for authentication functions\nAssistant: <tool_use>grep(pattern='def.*auth', path='.')</tool_use>"
+
+# Agentic workflow patterns
+"User: Implement API endpoint\nAssistant: Breaking this down: 1) Create file, 2) Implement logic, 3) Write tests..."
+```
+
+### Evaluation
+
+Tool calling accuracy is measured against expected patterns:
+- ✅ Correct tool selection
+- ✅ Proper tool syntax
+- ✅ Appropriate parameters
+- ✅ Multi-step planning
+
+## 💰 Cost Analysis
+
+### Fly.io GPU Costs
+
+| Instance Type | GPU        | vCPU | RAM   | Cost/hour | Cost/day |
+|---------------|------------|------|-------|-----------|----------|
+| a100-40gb     | A100 40GB  | 8    | 32GB  | ~$3.00    | ~$72.00  |
+| a100-80gb     | A100 80GB  | 12   | 48GB  | ~$4.50    | ~$108.00 |
+
+### Training Cost Estimates
+
+- **Phi-4 Fine-tuning**: 2-6 hours = $6-18
+- **Quantization**: 1-2 hours = $3-6
+- **Benchmarking**: 0.5-1 hour = $1.50-3
+- **Total**: ~$10.50-27 per iteration
+
+### Cost Optimization
+
+```bash
+# Scale down when not training
+flyctl scale count 0 -a agentic-llm-phi4
+
+# Scale up for training
+flyctl scale count 1 -a agentic-llm-phi4
+
+# Auto-pause after training
+flyctl apps update --auto-stop-machines=true -a agentic-llm-phi4
+```
+
+## 🔐 Security & Secrets
+
+### Required Secrets
+
+```bash
+# Hugging Face token (for model downloads)
+flyctl secrets set HF_TOKEN=hf_xxxxxxxxxxxxx
+
+# Weights & Biases API key (for experiment tracking)
+flyctl secrets set WANDB_API_KEY=xxxxxxxxxxxxx
+
+# Optional: Custom training config
+flyctl secrets set TRAINING_CONFIG='{"learning_rate": 1e-5}'
+```
+
+### Environment Variables
+
+```bash
+# In fly.toml
+[env]
+  PYTHONUNBUFFERED = "1"
+  CUDA_VISIBLE_DEVICES = "0"
+  TRANSFORMERS_CACHE = "/app/models"
+  HF_HOME = "/app/models"
+```
+
+## 📈 Expected Results
+
+### Performance Targets
+
+| Metric                  | Baseline | INT8    | INT4    |
+|-------------------------|----------|---------|---------|
+| Model Size              | 14GB     | 3.5GB   | 1.75GB  |
+| Inference Latency       | 100ms    | 40ms    | 30ms    |
+| Throughput              | 10 t/s   | 25 t/s  | 33 t/s  |
+| Tool Calling Accuracy   | 85%      | 84%     | 82%     |
+| GPU Memory Usage        | 12GB     | 4GB     | 2.5GB   |
+
+### Quality Preservation
+
+- **INT8**: <2% perplexity increase
+- **INT4**: <5% perplexity increase
+- **Tool accuracy**: >80% for all quantization levels
+
+## 🛠️ Troubleshooting
+
+### Out of Memory (OOM)
+
+```bash
+# Reduce batch size
+# In train.py, decrease:
+per_device_train_batch_size = 2
+gradient_accumulation_steps = 8
+```
+
+### Slow Training
+
+```bash
+# Enable gradient checkpointing
+gradient_checkpointing = True
+
+# Use mixed precision
+fp16 = True
+```
+
+### Failed Quantization
+
+```bash
+# Try INT8 only (more stable)
+python3 quantization/quantize.py --skip-int4
+
+# Increase available memory
+flyctl scale vm a100-80gb -a agentic-llm-phi4
+```
+
+## 📚 Additional Resources
+
+- [Phi-4 Model Card](https://huggingface.co/microsoft/Phi-4)
+- [Phi-4 ONNX](https://huggingface.co/microsoft/Phi-4-mini-instruct-onnx)
+- [Fly.io GPU Docs](https://fly.io/docs/gpus/)
+- [Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk)
+- [ONNX Runtime](https://onnxruntime.ai/)
+
+## 🤝 Contributing
+
+Contributions welcome! Please:
+1. Test locally first
+2. Include benchmarks
+3. Update documentation
+4. Follow existing patterns
+
+## 📄 License
+
+MIT License - see LICENSE file
+
+## 🙏 Acknowledgments
+
+- Microsoft Research for Phi-4
+- Anthropic for Claude Agent SDK
+- Fly.io for GPU infrastructure
+- Hugging Face for model hosting
+
+---
+
+**Ready to optimize your Phi-4 model for Claude Agent SDK? Start with `./deployment/deploy.sh`!**

package/agentic-llm/docs/TRAINING_MONITOR.md

@@ -0,0 +1,77 @@
+# Phi-4 Training Monitor - Active
+
+## 🚀 Deployment Status: RUNNING
+
+**Project**: ruv-dev
+**Service**: phi4-finetuning-gpu
+**Region**: us-central1
+**GPU**: NVIDIA L4
+
+### 📊 Training Pipeline
+
+The system is now running the complete fine-tuning pipeline:
+
+1. **Phase 1: Baseline Benchmarks** (~15 min)
+   - Tests original Phi-4 model
+   - Measures tool calling accuracy
+   - Records performance metrics
+
+2. **Phase 2: Fine-tuning** (~4-6 hours)
+   - LoRA training with r=32, alpha=64
+   - 5 epochs with early stopping
+   - 60-100 training examples
+   - Target: >95% tool accuracy
+
+3. **Phase 3: Validation** (~10 min)
+   - 20 validation test cases
+   - Tool detection, correctness, parameters
+   - Quality tier assessment
+
+4. **Phase 4: Post-Training Benchmarks** (~10 min)
+   - Compare baseline vs fine-tuned
+   - Performance improvements
+   - Tool-by-tool breakdown
+
+5. **Phase 5: Email Notification**
+   - Sends results to: **ruv@ruv.net**
+   - Includes all metrics and next steps
+
+### 📧 Email Will Include:
+
+- ✅ Training duration
+- ✅ Tool accuracy achieved (target: >95%)
+- ✅ Parameter accuracy (target: >90%)
+- ✅ Quality tier (EXCELLENT/GOOD/NEEDS_IMPROVEMENT)
+- ✅ Improvement metrics vs baseline
+- ✅ Next steps (HuggingFace upload if successful)
+
+### 🔍 Monitoring
+
+Background monitor is checking status every 5 minutes and will notify when complete.
+
+You can also manually check:
+```bash
+# View monitor output
+tail -f /workspaces/agentic-flow/agentic-llm/monitor-training.log
+
+# Check Cloud Run logs
+gcloud run logs tail phi4-finetuning-gpu --region us-central1 --project=ruv-dev
+```
+
+### ⏱️ Expected Completion
+
+**Start Time**: ~17:40 UTC
+**Expected End**: ~23:00-01:00 UTC (5-7 hours)
+
+### 💾 After Training
+
+If tool accuracy >95%, upload to HuggingFace:
+```bash
+python3 cloudrun/upload_to_huggingface.py --repo-name phi4-mcp-tools-optimized
+```
+
+Model will be available at: `https://huggingface.co/YOUR_USERNAME/phi4-mcp-tools-optimized`
+
+---
+
+**Status**: Training in progress... Monitor active ✅