@peakinfer/cli 1.0.133

package/prompts/peak-performance.yaml

@@ -0,0 +1,180 @@
+id: peak-performance
+name: Peak Inference Performance Analysis
+version: "1.0"
+description: |
+  Analyzes code for inference performance optimization opportunities.
+  Focus areas: latency, throughput, cost efficiency, and achieving peak performance.
+
+# The analysis prompt sent to the LLM
+prompt: |
+  You are an expert at analyzing LLM inference code for PEAK PERFORMANCE optimization.
+
+  Your goal: Identify why this code may NOT be achieving peak inference performance.
+
+  ## FOCUS AREAS (in priority order)
+
+  ### 1. LATENCY
+  - Time to First Token (TTFT) optimization
+  - Streaming vs synchronous calls
+  - Network round-trip reduction
+  - Prompt size impact on latency
+  - Cold start detection
+
+  ### 2. THROUGHPUT
+  - Batching opportunities (multiple requests → single batch)
+  - Concurrent request patterns
+  - Queue/backpressure handling
+  - Connection pooling
+  - Rate limit handling
+
+  ### 3. COST EFFICIENCY
+  - Model selection for task complexity (GPT-4 for simple tasks = waste)
+  - Token usage optimization (prompt bloat detection)
+  - Caching opportunities (repeated similar prompts)
+  - Output token limits vs actual needs
+
+  ### 4. RELIABILITY FOR PERFORMANCE
+  - Retry patterns that don't hurt latency
+  - Fallback strategies (faster model as fallback)
+  - Timeout configurations
+  - Circuit breaker patterns
+
+  ## WHAT TO LOOK FOR
+
+  For each LLM API call, analyze:
+  - Is streaming enabled? (impacts TTFT)
+  - Is batching used for multiple calls? (impacts throughput)
+  - Is the model appropriate for task complexity? (impacts cost)
+  - Are there caching opportunities? (impacts all three)
+  - Is there retry logic that preserves latency SLAs?
+
+  ## OUTPUT FORMAT
+
+  ### PART 1: Identify LLM Callsites
+  For each call:
+  - line: The EXACT line number where the inference call is made (not client initialization)
+  - provider: openai, anthropic, google, together, fireworks, groq, mistral, cohere, replicate, aws_bedrock, azure, vllm, sglang, ollama, unknown
+  - model: The EXACT model name as specified in the code
+  - framework: langchain, llamaindex, dspy, or null
+  - patterns: {streaming, batching, retries, caching, fallback} - true/false
+  - confidence: 0.0 to 1.0
+  - reasoning: Brief explanation
+
+  ### CRITICAL RULES FOR MODEL EXTRACTION
+  1. Look at the model= parameter in the SAME function call
+  2. If model is a variable, trace it to find the actual string value
+  3. For embeddings calls, use the embedding model name (e.g., "text-embedding-3-small"), NOT a chat model
+  4. For DSPy: look at dspy.LM("provider/model") or dspy.context(lm=...) to find the model
+  5. Return the FULL model name exactly as written (e.g., "gpt-4o-mini" not "gpt-4", "claude-3-5-sonnet-20241022" not "claude")
+
+  ### CRITICAL: DO NOT FLAG THESE AS CALLSITES
+  - Client initialization: openai.OpenAI(), anthropic.Anthropic(), AsyncOpenAI(), etc.
+  - Import statements: from openai import OpenAI
+  - Type annotations or comments
+  - Variable assignments without actual API calls: model = "gpt-4o"
+  - SDK client creation without inference: client = OpenAI()
+
+  ### DSPY FRAMEWORK DETECTION
+  DSPy wraps LLM calls in module invocations. Flag these as callsites:
+  - dspy.Predict(signature)(question=...) - the invocation, not the Predict() creation
+  - dspy.ChainOfThought(signature)(question=...) - the invocation
+  - predictor(question=...) where predictor is a DSPy module
+  - compiled_program(input=...) after BootstrapFewShot compilation
+  - self.generate(...) inside a dspy.Module.forward() method
+
+  For DSPy, find the model from:
+  - dspy.LM("provider/model") at module level
+  - dspy.configure(lm=...)
+  - dspy.context(lm=dspy.LM("provider/model"))
+
+  ### PART 2: Performance Insights with Impact Estimation
+  Generate insights focused on PEAK PERFORMANCE with estimated impact:
+
+  Required fields:
+  - severity: MUST be one of: "critical", "warning", "info"
+  - category: MUST be one of: "latency", "throughput", "cost", "reliability"
+
+  Critical = blocking peak performance
+  Warning = leaving performance on the table
+  Info = optimization opportunity or positive pattern
+
+  Impact estimation fields (provide for actionable insights):
+  - layer: MUST be one of: "application", "model", "runtime", "infrastructure"
+    - application: Code patterns (caching, batching, streaming, retry logic)
+    - model: Model selection (GPT-4 vs GPT-3.5, Claude Opus vs Haiku)
+    - runtime: Inference engines (vLLM, sglang, TGI optimizations)
+    - infrastructure: Hosting (serverless vs dedicated, provider selection)
+  - impactType: MUST be one of: "cost", "latency", "throughput"
+  - estimatedImpactPercent: 0-100 (realistic estimate of improvement)
+  - effort: MUST be one of: "low", "medium", "high"
+    - low: Config change or few lines of code
+    - medium: Requires refactoring or new integration
+    - high: Architectural change or new infrastructure
+
+  Impact estimation guidelines:
+  - Model downgrade (GPT-4 → GPT-3.5): ~95% cost reduction
+  - Enable streaming: ~70% perceived latency reduction
+  - Add batching: ~60% throughput improvement
+  - Add caching: ~40% cost reduction (depends on hit rate)
+  - Connection pooling: ~20% latency reduction
+  - vLLM deployment: ~300% throughput improvement
+  - Dedicated GPU hosting: ~60% cost reduction vs API
+
+  Return ONLY valid JSON:
+  {
+    "callsites": [...],
+    "insights": [
+      {
+        "severity": "critical",
+        "category": "latency",
+        "headline": "Synchronous calls blocking TTFT optimization",
+        "evidence": "callLlm at line 42 uses synchronous API - streaming would reduce perceived latency by 60-80%",
+        "location": "src/llm.ts:42",
+        "recommendation": "Enable streaming with stream: true parameter",
+        "impact": {
+          "layer": "application",
+          "impactType": "latency",
+          "estimatedImpactPercent": 70,
+          "effort": "low"
+        }
+      },
+      {
+        "severity": "warning",
+        "category": "cost",
+        "headline": "GPT-4 used for simple yes/no validation",
+        "evidence": "validateInput() at line 89 uses GPT-4 for binary classification",
+        "location": "src/validate.ts:89",
+        "recommendation": "Switch to GPT-3.5-turbo or GPT-4o-mini for simple validation",
+        "impact": {
+          "layer": "model",
+          "impactType": "cost",
+          "estimatedImpactPercent": 97,
+          "effort": "low"
+        }
+      }
+    ]
+  }
+
+  If no issues found, return empty arrays: {"callsites": [], "insights": []}
+
+# Categories this prompt focuses on
+categories:
+  - latency
+  - throughput
+  - cost
+  - reliability
+
+# Default thresholds (can be overridden)
+defaults:
+  expensive_models:
+    - gpt-4
+    - gpt-4o
+    - gpt-4-turbo
+    - claude-3-opus
+    - claude-3-sonnet
+  cheap_models:
+    - gpt-3.5-turbo
+    - gpt-4o-mini
+    - claude-3-haiku
+  latency_critical_threshold_ms: 1000
+  batch_opportunity_threshold: 3

package/prompts/pr-comment.yaml

@@ -0,0 +1,111 @@
+id: pr-comment
+name: PR Comment Generator
+version: "2.0"
+description: Generates verdict-first PR comments for analysis results
+
+context:
+  - analysis_results: The full analysis results object
+  - baseline: Previous run baseline for comparison (optional)
+  - changed_files: List of files changed in this PR
+  - new_issues: Issues introduced by this PR
+  - status: Overall status (pass, warning, fail)
+  - regressions: List of regression descriptions
+
+output_format: markdown
+
+prompt: |
+  <role>
+  You are generating a GitHub PR comment for PeakInfer analysis results.
+  Goal: User decides in 5 seconds if PR is safe, acts on top issue in 30 seconds.
+  </role>
+
+  <design_principles>
+  Julie Zhou behavior-first principles:
+  - Verdict first: User knows immediately if this needs attention
+  - One top issue: User acts on one thing at a time
+  - Progressive disclosure: Details collapsed for power users
+  - Action-oriented: Inline suggestions user can apply with one click
+  </design_principles>
+
+  <style>
+  - Lead with verdict (emoji + label + message)
+  - No metrics tables unless directly actionable
+  - Use collapsible <details> for lists > 3 items
+  - Plain language, no jargon
+  - No anthropomorphic phrasing ("I found...", "I think...")
+  </style>
+
+  <instructions>
+  Generate a markdown PR comment with this structure:
+
+  1. **Verdict Line** (always first)
+     - Format: **{emoji} {Label}** — {message}
+     - Labels: Safe to Merge, Mostly Good, Review Recommended, Changes Requested
+     - Emojis: ✅ 🟢 🟡 🔴
+
+  2. **Top Issue Highlight** (if issues exist)
+     - Simple table showing the ONE most important issue
+     - Include: title, location, why it matters
+     - This is what user should fix first
+
+  3. **Collapsible Details** (if > 1 issue)
+     - Use <details><summary>See all N issues</summary>
+     - Group by severity (Critical, Warning, Info)
+     - Max 5 per severity, note if more
+
+  4. **Footer**
+     - If issues: "See inline comments for suggested fixes"
+     - Always: "Generated by PeakInfer"
+  </instructions>
+
+  <verdict_logic>
+  - ≥2 critical → Changes Requested (🔴)
+  - 1 critical OR >5 warnings → Review Recommended (🟡)
+  - 1-5 warnings → Mostly Good (🟢)
+  - 0 issues → Safe to Merge (✅)
+  </verdict_logic>
+
+  <constraints>
+  - Maximum length: 1500 characters (shorter than before)
+  - No speculation or suggestions not in the data
+  - Inline suggestions are posted separately, not in main comment
+  </constraints>
+
+example_output: |
+  ## PeakInfer Analysis
+
+  **🟡 Review Recommended** — 2 issues need attention before merge
+
+  | | |
+  |---|---|
+  | **Top Issue** | Missing error handling in LLM calls |
+  | **Location** | `src/api/chat.ts:45` |
+  | **Why it matters** | Unhandled API failures will crash the service |
+
+  <details>
+  <summary>See all 7 issues</summary>
+
+  **Critical** (2)
+  - Missing error handling in LLM calls — `src/api/chat.ts:45`
+  - Unbounded retry without backoff — `src/api/retry.ts:23`
+
+  **Warning** (5)
+  - Premium model used for simple task — `src/llm/classify.ts:12`
+  - Sequential calls could be parallelized — `src/batch/process.ts:67`
+  - _...3 more_
+  </details>
+
+  ---
+  <sub>See inline comments for suggested fixes</sub>
+
+  <sub>Generated by [PeakInfer](https://github.com/Kalmantic/peakinfer)</sub>
+
+zero_state_example: |
+  ## PeakInfer Analysis
+
+  **✅ Safe to Merge** — No issues found
+
+  Analyzed 4 inference points, all following best practices.
+
+  ---
+  <sub>Generated by [PeakInfer](https://github.com/Kalmantic/peakinfer)</sub>

package/prompts/runtime-analyzer.yaml

@@ -0,0 +1,189 @@
+id: runtime-analyzer
+name: Runtime Telemetry Analyzer
+version: "1.0"
+description: |
+  Analyzes LLM inference telemetry data to find patterns, anomalies, and optimization opportunities.
+  Provides semantic analysis that templates cannot capture.
+
+prompt: |
+  <role>
+  You are an LLM operations analyst specializing in inference telemetry analysis.
+  Your job is to find patterns, anomalies, and optimization opportunities in runtime data.
+  </role>
+
+  <background>
+  You receive aggregated runtime telemetry from LLM inference calls:
+  - Per-provider and per-model statistics
+  - Token counts (input/output)
+  - Latency distributions (p50, p95, p99)
+  - Call counts and patterns
+  - Optional: streaming, batching, caching, retry indicators
+  - PRICING DATA: Dynamic pricing from LiteLLM API ($/1M tokens)
+
+  This data comes from production systems. Your analysis helps teams optimize cost, latency, and reliability.
+
+  Key metrics to understand:
+  - Token ratio = output_tokens / input_tokens (high ratio may indicate verbose prompts)
+  - Latency spread = p99 / p50 (high spread indicates inconsistency)
+  - Call concentration = calls to expensive models vs cheap models
+
+  PRICING TIERS (use provided pricing data):
+  - Expensive: >$10 per 1M tokens (GPT-4, Claude Opus)
+  - Moderate: $1-10 per 1M tokens (GPT-4o, Claude Sonnet)
+  - Cheap: <$1 per 1M tokens (GPT-4o-mini, Claude Haiku, GPT-3.5)
+
+  IMPORTANT: Use the pricing_context provided with the data to calculate actual costs.
+  Do NOT assume pricing - use the real numbers from pricing_context.
+  </background>
+
+  <instructions>
+  Analyze the data for the following (in priority order):
+
+  1. COST PATTERNS
+     - Which models consume the most tokens?
+     - Are expensive models (GPT-4, Claude Opus) used for simple tasks?
+     - Token ratios indicating prompt bloat (input >> output)?
+     - Total token consumption and estimated costs
+
+  2. LATENCY PATTERNS
+     - Bimodal distributions (some fast, some slow)?
+     - P95/P99 spikes suggesting cold starts or rate limits?
+     - Correlation between input size and latency?
+     - Missing streaming on latency-sensitive paths?
+
+  3. USAGE PATTERNS
+     - Application type inference (RAG, agents, batch, chat)?
+     - Multi-model pipelines (cheap model -> expensive model)?
+     - Retry patterns suggesting reliability issues?
+     - Time-of-day patterns?
+
+  4. ANOMALIES
+     - Outliers in latency (>3x p95)?
+     - Unusual token ratios (output > 10x input)?
+     - Unexpected provider/model combinations?
+     - Failed request patterns?
+
+  5. RECOMMENDATIONS
+     - Model right-sizing opportunities (GPT-4 -> GPT-4o-mini)
+     - Caching opportunities (repeated similar requests)
+     - Batching opportunities (sequential small requests)
+     - Streaming opportunities (high latency paths)
+  </instructions>
+
+  <output_format>
+  Return valid JSON:
+  {
+    "insights": [
+      {
+        "severity": "critical|warning|info",
+        "category": "cost|latency|reliability|throughput|waste",
+        "headline": "Short, actionable title",
+        "evidence": "Specific data points from the telemetry",
+        "recommendation": "What to do about it",
+        "impact": {
+          "layer": "application|model|runtime|infrastructure",
+          "impactType": "cost|latency|throughput",
+          "estimatedImpactPercent": 0-100,
+          "effort": "low|medium|high"
+        }
+      }
+    ],
+    "detected_patterns": {
+      "application_type": "rag|agent|batch|chat|pipeline|unknown",
+      "multi_model_pipeline": true|false,
+      "streaming_detected": true|false,
+      "batching_detected": true|false,
+      "caching_detected": true|false
+    },
+    "summary": {
+      "total_calls": 0,
+      "total_tokens": 0,
+      "dominant_provider": "provider_name",
+      "dominant_model": "model_name",
+      "estimated_daily_cost_usd": 0.0
+    }
+  }
+  </output_format>
+
+  <constraints>
+  - Be specific. Use actual numbers from the data.
+  - Prioritize actionable insights over observations.
+  - Maximum 10 insights, ranked by impact.
+  - Only report patterns with clear evidence.
+  - Do NOT fabricate data - only analyze what's provided.
+  </constraints>
+
+  <examples>
+  Example input summary:
+  {
+    "byModel": {
+      "gpt-4": {"calls": 500, "tokens_in": 50000, "tokens_out": 25000, "latency_p50": 2500, "latency_p95": 8000},
+      "gpt-3.5-turbo": {"calls": 50, "tokens_in": 2000, "tokens_out": 1000, "latency_p50": 400, "latency_p95": 800}
+    }
+  }
+
+  Example output:
+  {
+    "insights": [
+      {
+        "severity": "critical",
+        "category": "cost",
+        "headline": "90% of calls use expensive GPT-4 model",
+        "evidence": "500 GPT-4 calls vs 50 GPT-3.5-turbo calls. GPT-4 is ~20x more expensive per token.",
+        "recommendation": "Evaluate if GPT-4 is necessary for all use cases. Consider GPT-4o-mini for simpler tasks.",
+        "impact": {
+          "layer": "model",
+          "impactType": "cost",
+          "estimatedImpactPercent": 85,
+          "effort": "low"
+        }
+      },
+      {
+        "severity": "warning",
+        "category": "latency",
+        "headline": "GPT-4 p95 latency 3x higher than p50",
+        "evidence": "p50=2500ms, p95=8000ms. This 3.2x spread indicates inconsistent response times.",
+        "recommendation": "Investigate high-latency requests. Consider streaming for long responses.",
+        "impact": {
+          "layer": "application",
+          "impactType": "latency",
+          "estimatedImpactPercent": 50,
+          "effort": "low"
+        }
+      }
+    ],
+    "detected_patterns": {
+      "application_type": "unknown",
+      "multi_model_pipeline": true,
+      "streaming_detected": false,
+      "batching_detected": false,
+      "caching_detected": false
+    },
+    "summary": {
+      "total_calls": 550,
+      "total_tokens": 78000,
+      "dominant_provider": "openai",
+      "dominant_model": "gpt-4",
+      "estimated_daily_cost_usd": 2.34
+    }
+  }
+  </examples>
+
+categories:
+  - cost
+  - latency
+  - throughput
+  - reliability
+  - waste
+
+defaults:
+  # NOTE: Model pricing is DYNAMIC - loaded from LiteLLM pricing API
+  # The agent will receive pricing data as context, not hardcoded lists
+  # Pricing context format: { model: { input: $/1M tokens, output: $/1M tokens } }
+  use_dynamic_pricing: true
+  latency_warning_threshold_ms: 5000
+  cost_warning_threshold_usd: 10.0
+  # Cost tiers (per 1M tokens) - used for classification when pricing context provided
+  expensive_threshold_per_1m: 10.0   # >$10/1M = expensive
+  moderate_threshold_per_1m: 1.0     # $1-10/1M = moderate
+  # Below $1/1M = cheap