@mariozechner/pi 0.2.4 → 0.5.0

package/README.md

@@ -1,6 +1,6 @@
-# GPU Pod Manager
+# pi
 
-Quickly deploy LLMs on GPU pods from [Prime Intellect](https://www.primeintellect.ai/), [Vast.ai](https://vast.ai/), [DataCrunch](datacrunch.io), AWS, etc., for local coding agents and AI assistants.
+Deploy and manage LLMs on GPU pods with automatic vLLM configuration for agentic workloads.
 
 ## Installation
 
@@ -8,406 +8,504 @@ Quickly deploy LLMs on GPU pods from [Prime Intellect](https://www.primeintellec
 npm install -g @mariozechner/pi
 ```
 
-Or run directly with npx:
-```bash
-npx @mariozechner/pi
-```
-
-## What This Is
+## What is pi?
 
-A simple CLI tool that automatically sets up and manages vLLM deployments on GPU pods. Start from a clean Ubuntu pod and have multiple models running in minutes. A GPU pod is defined as an Ubuntu machine with root access, one or more GPUs, and Cuda drivers installed. It is aimed at individuals who are limited by local hardware and want to experiment with large open weight LLMs for their coding assistent workflows.
+`pi` simplifies running large language models on remote GPU pods. It automatically:
+- Sets up vLLM on fresh Ubuntu pods
+- Configures tool calling for agentic models (Qwen, GPT-OSS, GLM, etc.)
+- Manages multiple models on the same pod with "smart" GPU allocation
+- Provides OpenAI-compatible API endpoints for each model
+- Includes an interactive agent with file system tools for testing
 
-**Key Features:**
-- **Zero to LLM in minutes** - Automatically installs vLLM and all dependencies on clean pods
-- **Multi-model management** - Run multiple models concurrently on a single pod
-- **Smart GPU allocation** - Round robin assigns models to available GPUs on multi-GPU pods
-- **Tensor parallelism** - Run large models across multiple GPUs with `--all-gpus`
-- **OpenAI-compatible API** - Drop-in replacement for OpenAI API clients with automatic tool/function calling support
-- **No complex setup** - Just SSH access, no Kubernetes or Docker required
-- **Privacy first** - vLLM telemetry disabled by default
+## Quick Start
 
-**Limitations:**
-- OpenAI endpoints exposed to the public internet (yolo)
-- Requires manual pod creation via Prime Intellect, Vast.ai, AWS, etc.
-- Assumes Ubuntu 22 image when creating pods
+```bash
+# Set required environment variables
+export HF_TOKEN=your_huggingface_token      # Get from https://huggingface.co/settings/tokens
+export PI_API_KEY=your_api_key              # Any string you want for API authentication
 
-## What this is not
-- A provisioning manager for pods. You need to provision the pods on the respective provider themselves.
-- Super optimized LLM deployment infrastructure for absolute best performance. This is for individuals who want to quickly spin up large open weights models for local LLM loads.
+# Setup a DataCrunch pod with NFS storage (models path auto-extracted)
+pi pods setup dc1 "ssh root@1.2.3.4" \
+  --mount "sudo mount -t nfs -o nconnect=16 nfs.fin-02.datacrunch.io:/your-pseudo /mnt/hf-models"
 
-## Requirements
+# Start a model (automatic configuration for known models)
+pi start Qwen/Qwen2.5-Coder-32B-Instruct --name qwen
 
-- **Node.js 14+** - To run the CLI tool on your machine
-- **HuggingFace Token** - Required for downloading models (get one at https://huggingface.co/settings/tokens)
-- **Prime Intellect/DataCrunch/Vast.ai Account**
-- **GPU Pod** - At least one running pod with:
-  - Ubuntu 22+ image (selected when creating pod)
-  - SSH access enabled
-  - Clean state (no manual vLLM installation needed)
-  - **Note**: B200 GPUs require PyTorch nightly with CUDA 12.8+ (automatically installed if detected). However, vLLM may need to be built from source for full compatibility.
+# Send a single message to the model
+pi agent qwen "What is the Fibonacci sequence?"
 
-## Quick Start
+# Interactive chat mode with file system tools
+pi agent qwen -i
 
-```bash
-# 1. Get a GPU pod from Prime Intellect
-#    Visit https://app.primeintellect.ai or https://vast.ai/ or https://datacrunch.io and create a pod (use Ubuntu 22+ image)
-#    Providers usually give you an SSH command with which to log into the machine. Copy that command.
+# Use with any OpenAI-compatible client
+export OPENAI_BASE_URL='http://1.2.3.4:8001/v1'
+export OPENAI_API_KEY=$PI_API_KEY
+```
 
-# 2. On your local machine, run the following to setup the remote pod. The Hugging Face token
-#    is required for model download.
-export HF_TOKEN=your_huggingface_token
-pi setup my-pod-name "ssh root@135.181.71.41 -p 22"
+## Prerequisites
 
-# 3. Start a model (automatically manages GPU assignment)
-pi start microsoft/Phi-3-mini-128k-instruct --name phi3 --memory 20%
+- Node.js 18+
+- HuggingFace token (for model downloads)
+- GPU pod with:
+  - Ubuntu 22.04 or 24.04
+  - SSH root access
+  - NVIDIA drivers installed
+  - Persistent storage for models
 
-# 4. Test the model with a prompt
-pi prompt phi3 "What is 2+2?"
-# Response: The answer is 4.
+## Supported Providers
 
-# 5. Start another model (automatically uses next available GPU on multi-GPU pods)
-pi start Qwen/Qwen2.5-7B-Instruct --name qwen --memory 30%
+### Primary Support
 
-# 6. Check running models
-pi list
+**DataCrunch** - Best for shared model storage
+- NFS volumes sharable across multiple pods in same region
+- Models download once, use everywhere
+- Ideal for teams or multiple experiments
 
-# 7. Use with your coding agent
-export OPENAI_BASE_URL='http://135.181.71.41:8001/v1'  # For first model
-export OPENAI_API_KEY='dummy'
-```
+**RunPod** - Good persistent storage
+- Network volumes persist independently
+- Cannot share between running pods simultaneously
+- Good for single-pod workflows
 
-## How It Works
+### Also Works With
+- Vast.ai (volumes locked to specific machine)
+- Prime Intellect (no persistent storage)
+- AWS EC2 (with EFS setup)
+- Any Ubuntu machine with NVIDIA GPUs, CUDA driver, and SSH
 
-1. **Automatic Setup**: When you run `pi setup`, it:
-   - Connects to your clean Ubuntu pod
-   - Installs Python, CUDA drivers, and vLLM
-   - Configures HuggingFace tokens
-   - Sets up the model manager
+## Commands
 
-2. **Model Management**: Each `pi start` command:
-   - Automatically finds an available GPU (on multi-GPU systems)
-   - Allocates the specified memory fraction
-   - Starts a separate vLLM instance on a unique port accessible via the OpenAI API protocol
-   - Manages logs and process lifecycle
+### Pod Management
 
-3. **Multi-GPU Support**: On pods with multiple GPUs:
-   - Single models automatically distribute across available GPUs
-   - Large models can use tensor parallelism with `--all-gpus`
-   - View GPU assignments with `pi list`
+```bash
+pi pods setup <name> "<ssh>" [options]        # Setup new pod
+  --mount "<mount_command>"                   # Run mount command during setup
+  --models-path <path>                        # Override extracted path (optional)
+  --vllm release|nightly|gpt-oss              # vLLM version (default: release)
+
+pi pods                                       # List all configured pods
+pi pods active <name>                         # Switch active pod
+pi pods remove <name>                         # Remove pod from local config
+pi shell [<name>]                             # SSH into pod
+pi ssh [<name>] "<command>"                   # Run command on pod
+```
 
+**Note**: When using `--mount`, the models path is automatically extracted from the mount command's target directory. You only need `--models-path` if not using `--mount` or to override the extracted path.
 
-## Commands
+#### vLLM Version Options
 
-### Pod Management
+- `release` (default): Stable vLLM release, recommended for most users
+- `nightly`: Latest vLLM features, needed for newest models like GLM-4.5
+- `gpt-oss`: Special build for OpenAI's GPT-OSS models only
 
-The tool supports managing multiple Prime Intellect pods from a single machine. Each pod is identified by a name you choose (e.g., "prod", "dev", "h200"). While all your pods continue running independently, the tool operates on one "active" pod at a time - all model commands (start, stop, list, etc.) are directed to this active pod. You can easily switch which pod is active to manage models on different machines.
+### Model Management
 
 ```bash
-pi setup <pod-name> "<ssh_command>"  # Configure and activate a pod
-pi pods                              # List all pods (active pod marked)
-pi pod <pod-name>                    # Switch active pod
-pi pod remove <pod-name>             # Remove pod from config
-pi shell                             # SSH into active pod
+pi start <model> --name <name> [options]  # Start a model
+  --memory <percent>      # GPU memory: 30%, 50%, 90% (default: 90%)
+  --context <size>        # Context window: 4k, 8k, 16k, 32k, 64k, 128k
+  --gpus <count>          # Number of GPUs to use (predefined models only)
+  --pod <name>            # Target specific pod (overrides active)
+  --vllm <args...>        # Pass custom args directly to vLLM
+
+pi stop [<name>]          # Stop model (or all if no name given)
+pi list                   # List running models with status
+pi logs <name>            # Stream model logs (tail -f)
 ```
 
-#### Working with Multiple Pods
-
-You can manage models on any pod without switching the active pod by using the `--pod` parameter:
+### Agent & Chat Interface
 
 ```bash
-# List models on a specific pod
-pi list --pod prod
+pi agent <name> "<message>"               # Single message to model
+pi agent <name> "<msg1>" "<msg2>"         # Multiple messages in sequence
+pi agent <name> -i                        # Interactive chat mode
+pi agent <name> -i -c                     # Continue previous session
+
+# Standalone OpenAI-compatible agent (works with any API)
+pi-agent --base-url http://localhost:8000/v1 --model llama-3.1 "Hello"
+pi-agent --api-key sk-... "What is 2+2?"  # Uses OpenAI by default
+pi-agent --json "What is 2+2?"            # Output event stream as JSONL
+pi-agent -i                                # Interactive mode
+```
 
-# Start a model on a specific pod
-pi start Qwen/Qwen2.5-7B-Instruct --name qwen --pod dev
+The agent includes tools for file operations (read, list, bash, glob, rg) to test agentic capabilities, particularly useful for code navigation and analysis tasks.
 
-# Stop a model on a specific pod
-pi stop qwen --pod dev
+## Predefined Model Configurations
 
-# View logs from a specific pod
-pi logs qwen --pod dev
+`pi` includes predefined configurations for popular agentic models, so you do not have to specify `--vllm` arguments manually. `pi` will also check if the model you selected can actually run on your pod with respect to the number of GPUs and available VRAM. Run `pi start` without additional arguments to see a list of predefined models that can run on the active pod.
+
+### Qwen Models
+```bash
+# Qwen2.5-Coder-32B - Excellent coding model, fits on single H100/H200
+pi start Qwen/Qwen2.5-Coder-32B-Instruct --name qwen
 
-# Test a model on a specific pod
-pi prompt qwen "Hello!" --pod dev
+# Qwen3-Coder-30B - Advanced reasoning with tool use
+pi start Qwen/Qwen3-Coder-30B-A3B-Instruct --name qwen3
 
-# SSH into a specific pod
-pi shell --pod prod
-pi ssh --pod prod "nvidia-smi"
+# Qwen3-Coder-480B - State-of-the-art on 8xH200 (data-parallel mode)
+pi start Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --name qwen-480b
 ```
 
-This allows you to manage multiple environments (dev, staging, production) from a single machine without constantly switching between them.
+### GPT-OSS Models
+```bash
+# Requires special vLLM build during setup
+pi pods setup gpt-pod "ssh root@1.2.3.4" --models-path /workspace --vllm gpt-oss
 
-### Model Management
+# GPT-OSS-20B - Fits on 16GB+ VRAM
+pi start openai/gpt-oss-20b --name gpt20
 
-Each model runs as a separate vLLM instance with its own port and GPU allocation. The tool automatically manages GPU assignment on multi-GPU systems and ensures models don't conflict. Models are accessed by their short names (either auto-generated or specified with --name).
+# GPT-OSS-120B - Needs 60GB+ VRAM
+pi start openai/gpt-oss-120b --name gpt120
+```
 
+### GLM Models
 ```bash
-pi list                              # List running models on active pod
-pi search <query>                    # Search HuggingFace models
-pi start <model> [options]           # Start a model with options
-  --name <name>                      # Short alias (default: auto-generated)
-  --context <size>                   # Context window: 4k, 8k, 16k, 32k (default: model default)
-  --memory <percent>                 # GPU memory: 30%, 50%, 90% (default: 90%)
-  --all-gpus                         # Use tensor parallelism across all GPUs
-  --pod <pod-name>                   # Run on specific pod (default: active pod)
-  --vllm-args                        # Pass all remaining args directly to vLLM
-pi stop [name]                       # Stop a model (or all if no name)
-pi logs <name>                       # View logs with tail -f
-pi prompt <name> "message"           # Quick test prompt
-pi downloads [--live]                # Check model download progress (--live for continuous monitoring)
+# GLM-4.5 - Requires 8-16 GPUs, includes thinking mode
+pi start zai-org/GLM-4.5 --name glm
+
+# GLM-4.5-Air - Smaller version, 1-2 GPUs
+pi start zai-org/GLM-4.5-Air --name glm-air
 ```
 
-All model management commands support the `--pod` parameter to target a specific pod without switching the active pod.
+### Custom Models with --vllm
 
-## Examples
+For models not in the predefined list, use `--vllm` to pass arguments directly to vLLM:
 
-### Search for models
 ```bash
-pi search codellama
-pi search deepseek
-pi search qwen
+# DeepSeek with custom settings
+pi start deepseek-ai/DeepSeek-V3 --name deepseek --vllm \
+  --tensor-parallel-size 4 --trust-remote-code
+
+# Mistral with pipeline parallelism
+pi start mistralai/Mixtral-8x22B-Instruct-v0.1 --name mixtral --vllm \
+  --tensor-parallel-size 8 --pipeline-parallel-size 2
+
+# Any model with specific tool parser
+pi start some/model --name mymodel --vllm \
+  --tool-call-parser hermes --enable-auto-tool-choice
 ```
 
-**Note**: vLLM does not support formats like GGUF. Read the [docs](https://docs.vllm.ai/en/latest/)
+## DataCrunch Setup
 
-### A100 80GB scenarios
-```bash
-# Small model, high concurrency (~30-50 concurrent requests)
-pi start microsoft/Phi-3-mini-128k-instruct --name phi3 --memory 30%
+DataCrunch offers the best experience with shared NFS storage across pods:
 
-# Medium model, balanced (~10-20 concurrent requests)
-pi start meta-llama/Llama-3.1-8B-Instruct --name llama8b --memory 50%
+### 1. Create Shared Filesystem (SFS)
+- Go to DataCrunch dashboard → Storage → Create SFS
+- Choose size and datacenter
+- Note the mount command (e.g., `sudo mount -t nfs -o nconnect=16 nfs.fin-02.datacrunch.io:/hf-models-fin02-8ac1bab7 /mnt/hf-models-fin02`)
 
-# Large model, limited concurrency (~5-10 concurrent requests)
-pi start meta-llama/Llama-3.1-70B-Instruct --name llama70b --memory 90%
+### 2. Create GPU Instance
+- Create instance in same datacenter as SFS
+- Share the SFS with the instance
+- Get SSH command from dashboard
+
+### 3. Setup with pi
+```bash
+# Get mount command from DataCrunch dashboard
+pi pods setup dc1 "ssh root@instance.datacrunch.io" \
+  --mount "sudo mount -t nfs -o nconnect=16 nfs.fin-02.datacrunch.io:/your-pseudo /mnt/hf-models"
 
-# Run multiple small models
-pi start Qwen/Qwen2.5-Coder-1.5B --name coder1 --memory 15%
-pi start microsoft/Phi-3-mini-128k-instruct --name phi3 --memory 15%
+# Models automatically stored in /mnt/hf-models (extracted from mount command)
 ```
 
-## Understanding Context and Memory
+### 4. Benefits
+- Models persist across instance restarts
+- Share models between multiple instances in same datacenter
+- Download once, use everywhere
+- Pay only for storage, not compute time during downloads
 
-### Context Window vs Output Tokens
-Models are loaded with their default context length. You can use the `context` parameter to specify a lower or higher context length. The `context` parameter sets the **total** token budget for input + output combined:
-- Starting a model with `context=8k` means 8,192 tokens total
-- If your prompt uses 6,000 tokens, you have 2,192 tokens left for the response
-- Each OpenAI API request to the model can specify `max_output_tokens` to control output length within this budget
+## RunPod Setup
 
-Example:
-```bash
-# Start model with 32k total context
-pi start meta-llama/Llama-3.1-8B --name llama --context 32k --memory 50%
+RunPod offers good persistent storage with network volumes:
 
-# When calling the API, you control output length per request:
-# - Send 20k token prompt
-# - Request max_tokens=4000
-# - Total = 24k (fits within 32k context)
-```
+### 1. Create Network Volume (optional)
+- Go to RunPod dashboard → Storage → Create Network Volume
+- Choose size and region
 
-### GPU Memory and Concurrency
-vLLM pre-allocates GPU memory controlled by `gpu_fraction`. This matters for coding agents that spawn sub-agents, as each connection needs memory.
+### 2. Create GPU Pod
+- Select "Network Volume" during pod creation (if using)
+- Attach your volume to `/runpod-volume`
+- Get SSH command from pod details
 
-Example: On an A100 80GB with a 7B model (FP16, ~14GB weights):
-- `gpu_fraction=0.3` (24GB): ~10GB for KV cache → ~30-50 concurrent requests
-- `gpu_fraction=0.5` (40GB): ~26GB for KV cache → ~50-80 concurrent requests
-- `gpu_fraction=0.9` (72GB): ~58GB for KV cache → ~100+ concurrent requests
+### 3. Setup with pi
+```bash
+# With network volume
+pi pods setup runpod "ssh root@pod.runpod.io" --models-path /runpod-volume
+
+# Or use workspace (persists with pod but not shareable)
+pi pods setup runpod "ssh root@pod.runpod.io" --models-path /workspace
+```
 
-Models load in their native precision from HuggingFace (usually FP16/BF16). Check the model card's "Files and versions" tab - look for file sizes: 7B models are ~14GB, 13B are ~26GB, 70B are ~140GB. Quantized models (AWQ, GPTQ) in the name use less memory but may have quality trade-offs.
 
 ## Multi-GPU Support
 
-For pods with multiple GPUs, the tool automatically manages GPU assignment:
+### Automatic GPU Assignment
+When running multiple models, pi automatically assigns them to different GPUs:
+```bash
+pi start model1 --name m1  # Auto-assigns to GPU 0
+pi start model2 --name m2  # Auto-assigns to GPU 1
+pi start model3 --name m3  # Auto-assigns to GPU 2
+```
 
-### Automatic GPU assignment for multiple models
+### Specify GPU Count for Predefined Models
+For predefined models with multiple configurations, use `--gpus` to control GPU usage:
 ```bash
-# Each model automatically uses the next available GPU
-pi start microsoft/Phi-3-mini-128k-instruct --memory 20%  # Auto-assigns to GPU 0
-pi start Qwen/Qwen2.5-7B-Instruct --memory 20%         # Auto-assigns to GPU 1
-pi start meta-llama/Llama-3.1-8B --memory 20%          # Auto-assigns to GPU 2
+# Run Qwen on 1 GPU instead of all available
+pi start Qwen/Qwen2.5-Coder-32B-Instruct --name qwen --gpus 1
 
-# Check which GPU each model is using
-pi list
+# Run GLM-4.5 on 8 GPUs (if it has an 8-GPU config)
+pi start zai-org/GLM-4.5 --name glm --gpus 8
 ```
 
-## Qwen on a single H200
+If the model doesn't have a configuration for the requested GPU count, you'll see available options.
+
+### Tensor Parallelism for Large Models
+For models that don't fit on a single GPU:
 ```bash
-pi start Qwen/Qwen3-Coder-30B-A3B-Instruct qwen3-30b
+# Use all available GPUs
+pi start meta-llama/Llama-3.1-70B-Instruct --name llama70b --vllm \
+  --tensor-parallel-size 4
+
+# Specific GPU count
+pi start Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --name qwen480 --vllm \
+  --data-parallel-size 8 --enable-expert-parallel
 ```
 
-### Run large models across all GPUs
-```bash
-# Use --all-gpus for tensor parallelism across all available GPUs
-pi start meta-llama/Llama-3.1-70B-Instruct --all-gpus
-pi start Qwen/Qwen2.5-72B-Instruct --all-gpus --context 64k
+## API Integration
+
+All models expose OpenAI-compatible endpoints:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://your-pod-ip:8001/v1",
+    api_key="your-pi-api-key"
+)
+
+# Chat completion with tool calling
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-Coder-32B-Instruct",
+    messages=[
+        {"role": "user", "content": "Write a Python function to calculate fibonacci"}
+    ],
+    tools=[{
+        "type": "function",
+        "function": {
+            "name": "execute_code",
+            "description": "Execute Python code",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "code": {"type": "string"}
+                },
+                "required": ["code"]
+            }
+        }
+    }],
+    tool_choice="auto"
+)
 ```
 
-### Advanced: Custom vLLM arguments
+## Standalone Agent CLI
+
+`pi` includes a standalone OpenAI-compatible agent that can work with any API:
+
 ```bash
-# Pass custom arguments directly to vLLM with --vllm-args
-# Everything after --vllm-args is passed to vLLM unchanged
+# Install globally to get pi-agent command
+npm install -g @mariozechner/pi
 
-# Qwen3-Coder 480B on 8xH200 with expert parallelism
-pi start Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --name qwen-coder --vllm-args \
-  --data-parallel-size 8 --enable-expert-parallel \
-  --tool-call-parser qwen3_coder --enable-auto-tool-choice --gpu-memory-utilization 0.95 --max-model-len 200000
+# Use with OpenAI
+pi-agent --api-key sk-... "What is machine learning?"
 
-# DeepSeek with custom quantization
-pi start deepseek-ai/DeepSeek-Coder-V2-Instruct --name deepseek --vllm-args \
-  --tensor-parallel-size 4 --quantization fp8 --trust-remote-code
+# Use with local vLLM
+pi-agent --base-url http://localhost:8000/v1 \
+         --model meta-llama/Llama-3.1-8B-Instruct \
+         --api-key dummy \
+         "Explain quantum computing"
 
-# Mixtral with pipeline parallelism
-pi start mistralai/Mixtral-8x22B-Instruct-v0.1 --name mixtral --vllm-args \
-  --tensor-parallel-size 8 --pipeline-parallel-size 2
+# Interactive mode
+pi-agent -i
+
+# Continue previous session
+pi-agent --continue "Follow up question"
+
+# Custom system prompt
+pi-agent --system-prompt "You are a Python expert" "Write a web scraper"
+
+# Use responses API (for GPT-OSS models)
+pi-agent --api responses --model openai/gpt-oss-20b "Hello"
 ```
 
-**Note on Special Models**: Some models require specific vLLM arguments to run properly:
-- **Qwen3-Coder 480B**: Requires `--enable-expert-parallel` for MoE support
-- **Kimi K2**: May require custom arguments - check the model's documentation
-- **DeepSeek V3**: Often needs `--trust-remote-code` for custom architectures
-- When in doubt, consult the model's HuggingFace page or documentation for recommended vLLM settings
+The agent supports:
+- Session persistence across conversations
+- Interactive TUI mode with syntax highlighting
+- File system tools (read, list, bash, glob, rg) for code navigation
+- Both Chat Completions and Responses API formats
+- Custom system prompts
+
+## Tool Calling Support
 
-### Check GPU usage
+`pi` automatically configures appropriate tool calling parsers for known models:
+
+- **Qwen models**: `hermes` parser (Qwen3-Coder uses `qwen3_coder`)
+- **GLM models**: `glm4_moe` parser with reasoning support
+- **GPT-OSS models**: Uses `/v1/responses` endpoint, as tool calling (function calling in OpenAI parlance) is currently a [WIP with the `v1/chat/completions` endpoint](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#tool-use).
+- **Custom models**: Specify with `--vllm --tool-call-parser <parser> --enable-auto-tool-choice`
+
+To disable tool calling:
 ```bash
-pi ssh "nvidia-smi"
+pi start model --name mymodel --vllm --disable-tool-call-parser
 ```
 
-## Architecture Notes
+## Memory and Context Management
 
-- **Multi-Pod Support**: The tool stores multiple pod configurations in `~/.pi_config` with one active pod at a time.
-- **Port Allocation**: Each model runs on a separate port (8001, 8002, etc.) allowing multiple models on one GPU.
-- **Memory Management**: vLLM uses PagedAttention for efficient memory use with less than 4% waste.
-- **Model Caching**: Models are downloaded once and cached on the pod.
-- **Tool Parser Auto-Detection**: The tool automatically selects the appropriate tool parser based on the model:
-  - Qwen models: `hermes` (Qwen3-Coder: `qwen3_coder` if available)
-  - Mistral models: `mistral` with optimized chat template
-  - Llama models: `llama3_json` or `llama4_pythonic` based on version
-  - InternLM models: `internlm`
-  - Phi models: Tool calling disabled by default (no compatible tokens)
-  - Override with `--vllm-args --tool-call-parser <parser> --enable-auto-tool-choice`
+### GPU Memory Allocation
+Controls how much GPU memory vLLM pre-allocates:
+- `--memory 30%`: High concurrency, limited context
+- `--memory 50%`: Balanced (default)
+- `--memory 90%`: Maximum context, low concurrency
 
+### Context Window
+Sets maximum input + output tokens:
+- `--context 4k`: 4,096 tokens total
+- `--context 32k`: 32,768 tokens total
+- `--context 128k`: 131,072 tokens total
 
-## Tool Calling (Function Calling)
+Example for coding workload:
+```bash
+# Large context for code analysis, moderate concurrency
+pi start Qwen/Qwen2.5-Coder-32B-Instruct --name coder \
+  --context 64k --memory 70%
+```
 
-Tool calling allows LLMs to request the use of external functions/APIs, but it's a complex feature with many caveats:
+**Note**: When using `--vllm`, the `--memory`, `--context`, and `--gpus` parameters are ignored. You'll see a warning if you try to use them together.
 
-### The Reality of Tool Calling
+## Session Persistence
 
-1. **Model Compatibility**: Not all models support tool calling, even if they claim to. Many models lack the special tokens or training needed for reliable tool parsing.
+The interactive agent mode (`-i`) saves sessions for each project directory:
 
-2. **Parser Mismatches**: Different models use different tool calling formats:
-   - Hermes format (XML-like)
-   - Mistral format (specific JSON structure)
-   - Llama format (JSON-based or pythonic)
-   - Custom formats for each model family
+```bash
+# Start new session
+pi agent qwen -i
 
-3. **Common Issues**:
-   - "Could not locate tool call start/end tokens" - Model doesn't have required special tokens
-   - Malformed JSON/XML output - Model wasn't trained for the parser format
-   - Tool calls when you don't want them - Model overeager to use tools
-   - No tool calls when you need them - Model doesn't understand when to use tools
+# Continue previous session (maintains chat history)
+pi agent qwen -i -c
+```
 
-### How We Handle It
+Sessions are stored in `~/.pi/sessions/` organized by project path and include:
+- Complete conversation history
+- Tool call results
+- Token usage statistics
 
-The tool automatically detects the model and tries to use an appropriate parser:
-- **Qwen models**: `hermes` parser (Qwen3-Coder uses `qwen3_coder`)
-- **Mistral models**: `mistral` parser with custom template
-- **Llama models**: `llama3_json` or `llama4_pythonic` based on version
-- **Phi models**: Tool calling disabled (no compatible tokens)
+## Architecture & Event System
 
-### Your Options
+The agent uses a unified event-based architecture where all interactions flow through `AgentEvent` types. This enables:
+- Consistent UI rendering across console and TUI modes
+- Session recording and replay
+- Clean separation between API calls and UI updates
+- JSON output mode for programmatic integration
 
-1. **Let auto-detection handle it** (default):
-   ```bash
-   pi start meta-llama/Llama-3.1-8B-Instruct --name llama
-   ```
+Events are automatically converted to the appropriate API format (Chat Completions or Responses) based on the model type.
 
-2. **Force a specific parser** (if you know better):
-   ```bash
-   pi start model/name --name mymodel --vllm-args \
-     --tool-call-parser mistral --enable-auto-tool-choice
-   ```
+### JSON Output Mode
 
-3. **Disable tool calling entirely** (most reliable):
-   ```bash
-   pi start model/name --name mymodel --vllm-args \
-     --disable-tool-call-parser
-   ```
+Use `--json` flag to output the event stream as JSONL (JSON Lines) for programmatic consumption:
+```bash
+pi-agent --api-key sk-... --json "What is 2+2?"
+```
 
-4. **Handle tools in your application** (recommended for production):
-   - Send regular prompts asking the model to output JSON
-   - Parse the response in your code
-   - More control, more reliable
+Each line is a complete JSON object representing an event:
+```jsonl
+{"type":"user_message","text":"What is 2+2?"}
+{"type":"assistant_start"}
+{"type":"assistant_message","text":"2 + 2 = 4"}
+{"type":"token_usage","inputTokens":10,"outputTokens":5,"totalTokens":15,"cacheReadTokens":0,"cacheWriteTokens":0}
+```
 
-### Best Practices
+## Troubleshooting
+
+### OOM (Out of Memory) Errors
+- Reduce `--memory` percentage
+- Use smaller model or quantized version (FP8)
+- Reduce `--context` size
+
+### Model Won't Start
+```bash
+# Check GPU usage
+pi ssh "nvidia-smi"
 
-- **Test first**: Try a simple tool call to see if it works with your model
-- **Have a fallback**: Be prepared for tool calling to fail
-- **Consider alternatives**: Sometimes a well-crafted prompt works better than tool calling
-- **Read the docs**: Check the model card for tool calling examples
-- **Monitor logs**: Check `~/.vllm_logs/` for parser errors
+# Check if port is in use
+pi list
+
+# Force stop all models
+pi stop
+```
 
-Remember: Tool calling is still an evolving feature in the LLM ecosystem. What works today might break tomorrow with a model update.
+### Tool Calling Issues
+- Not all models support tool calling reliably
+- Try different parser: `--vllm --tool-call-parser mistral`
+- Or disable: `--vllm --disable-tool-call-parser`
 
-## Monitoring Downloads
+### Access Denied for Models
+Some models (Llama, Mistral) require HuggingFace access approval. Visit the model page and click "Request access".
 
-Use `pi downloads` to check the progress of model downloads in the HuggingFace cache:
+### vLLM Build Issues
+If using `--vllm nightly` fails, try:
+- Use `--vllm release` for stable version
+- Check CUDA compatibility with `pi ssh "nvidia-smi"`
 
+### Agent Not Finding Messages
+If the agent shows configuration instead of your message, ensure quotes around messages with special characters:
 ```bash
-pi downloads                         # Check downloads on active pod
-pi downloads --live                  # Live monitoring (updates every 2 seconds)
-pi downloads --pod 8h200            # Check downloads on specific pod
-pi downloads --live --pod 8h200     # Live monitoring on specific pod
+# Good
+pi agent qwen "What is this file about?"
+
+# Bad (shell might interpret special chars)
+pi agent qwen What is this file about?
 ```
 
-The command shows:
-- Model name and current size
-- Download progress (files downloaded / total files)
-- Download status (⏬ Downloading or ⏸ Idle)
-- Estimated total size (if available from HuggingFace)
+## Advanced Usage
+
+### Working with Multiple Pods
+```bash
+# Override active pod for any command
+pi start model --name test --pod dev-pod
+pi list --pod prod-pod
+pi stop test --pod dev-pod
+```
 
-**Tip for large models**: When starting models like Qwen-480B that take time to download, run `pi start` in one terminal and `pi downloads --live` in another to monitor progress. This is especially helpful since the log output during downloads can be minimal.
+### Custom vLLM Arguments
+```bash
+# Pass any vLLM argument after --vllm
+pi start model --name custom --vllm \
+  --quantization awq \
+  --enable-prefix-caching \
+  --max-num-seqs 256 \
+  --gpu-memory-utilization 0.95
+```
 
-**Downloads stalled?** If downloads appear stuck (e.g., at 92%), you can safely stop and restart:
+### Monitoring
 ```bash
-pi stop <model-name>         # Stop the current process
-pi downloads                 # Verify progress (e.g., 45/49 files)
-pi start <same-command>      # Restart with the same command
+# Watch GPU utilization
+pi ssh "watch -n 1 nvidia-smi"
+
+# Check model downloads
+pi ssh "du -sh ~/.cache/huggingface/hub/*"
+
+# View all logs
+pi ssh "ls -la ~/.vllm_logs/"
+
+# Check agent session history
+ls -la ~/.pi/sessions/
 ```
-vLLM will automatically use the already-downloaded files and continue from where it left off. This often resolves network or CDN throttling issues.
 
-## Troubleshooting
+## Environment Variables
+
+- `HF_TOKEN` - HuggingFace token for model downloads
+- `PI_API_KEY` - API key for vLLM endpoints
+- `PI_CONFIG_DIR` - Config directory (default: `~/.pi`)
+- `OPENAI_API_KEY` - Used by `pi-agent` when no `--api-key` provided
+
+## License
 
-- **OOM Errors**: Reduce gpu_fraction or use a smaller model
-- **Slow Inference**: Could be too many concurrent requests, try increasing gpu_fraction
-- **Connection Refused**: Check pod is running and port is correct
-- **HF Token Issues**: Ensure HF_TOKEN is set before running setup
-- **Access Denied**: Some models (like Llama, Mistral) require completing an access request on HuggingFace first. Visit the model page and click "Request access"
-- **Tool Calling Errors**: See the Tool Calling section above - consider disabling it or using a different model
-- **Model Won't Stop**: If `pi stop` fails, force kill all Python processes and verify GPU is free:
-  ```bash
-  pi ssh "killall -9 python3"
-  pi ssh "nvidia-smi"  # Should show no processes using GPU
-  ```
-- **Model Deployment Fails**: Pi currently does not check GPU memory utilization before starting models. If deploying a model fails:
-  1. Check if GPUs are full with other models: `pi ssh "nvidia-smi"`
-  2. If memory is insufficient, make room by stopping running models: `pi stop <model_name>`
-  3. If the error persists with sufficient memory, copy the error output and feed it to an LLM for troubleshooting assistance
-
-## Timing notes
-- 8x B200 on DataCrunch, Spot instance
-   - pi setup
-      - 1:27 min
-   - pi start Qwen/Qwen3-Coder-30B-A3B-Instruct
-      - (cold start incl. HF download, kernel warmup) 7:32m
-      - (warm start, HF model already in cache) 1:02m
-
-- 8x H200 on DataCrunch, Spot instance
-   - pi setup
-      -2:04m
-   - pi start Qwen/Qwen3-Coder-30B-A3B-Instruct
-      - (cold start incl. HF download, kernel warmup) 9:30m
-      - (warm start, HF model already in cache) 1:14m
-   - pi start Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 ...
-      - (cold start incl. HF download, kernel warmup)
-      - (warm start, HF model already in cache)
+MIT