aidevops 2.52.1 → 2.53.2

package/.agent/tools/context/prompt-optimization.md

@@ -1,338 +0,0 @@
----
-description: Prompt optimization using DSPy and DSPyGround
-mode: subagent
-tools:
-  read: true
-  write: false
-  edit: false
-  bash: true
-  glob: true
-  grep: true
-  webfetch: true
-  task: true
----
-
-# Prompt Optimization with DSPy & DSPyGround
-
-<!-- AI-CONTEXT-START -->
-
-## Quick Reference
-
-- **DSPy**: Programmatic prompt optimization framework
-- **DSPyGround**: Visual playground for interactive prompt refinement
-- **Workflow**: Bootstrap → Refine → Collect Samples → Optimize → Deploy
-- **Optimizers**: BootstrapFewShot, MIPRO, ChainOfThought
-- **Metrics**: Technical accuracy, security awareness, actionability, completeness
-- **Process**: Week 1 Foundation → Week 2 Refinement → Week 3 Specialization → Ongoing Maintenance
-- **Integration**: Export optimized prompts to provider scripts, quality workflows
-<!-- AI-CONTEXT-END -->
-
-## Overview
-
-This guide covers comprehensive prompt optimization strategies using both DSPy (programmatic optimization) and DSPyGround (visual playground) within the AI DevOps Framework.
-
-## Optimization Workflow
-
-### **Phase 1: Initial Development (DSPyGround)**
-
-1. **Bootstrap with Basic Prompt**
-   - Start with simple, clear instructions
-   - Define core functionality and constraints
-   - Test basic scenarios interactively
-
-2. **Interactive Refinement**
-   - Use DSPyGround's chat interface
-   - Collect diverse conversation samples
-   - Identify edge cases and failure modes
-
-3. **Sample Collection**
-   - Save positive examples (good responses)
-   - Mark negative examples (problematic responses)
-   - Organize samples into logical groups
-
-### **Phase 2: Automated Optimization (DSPy)**
-
-1. **Data Preparation**
-   - Export samples from DSPyGround
-   - Convert to DSPy training format
-   - Create validation and test sets
-
-2. **Systematic Optimization**
-   - Apply multiple DSPy optimizers
-   - Compare performance metrics
-   - Select best-performing variants
-
-3. **Production Deployment**
-   - Integrate optimized prompts
-   - Monitor performance metrics
-   - Iterate based on real-world feedback
-
-## Practical Examples
-
-### **DevOps Assistant Optimization**
-
-#### **Initial Prompt (DSPyGround)**
-
-```typescript
-systemPrompt: `You are a DevOps assistant. Help with server management.`
-```
-
-#### **Refined Prompt (After DSPyGround)**
-
-```typescript
-systemPrompt: `You are an expert DevOps engineer with 10+ years of experience.
-
-Your expertise includes:
-- Infrastructure automation and configuration management
-- CI/CD pipeline design and optimization
-- Container orchestration with Docker and Kubernetes
-- Cloud platform management (AWS, Azure, GCP)
-- Monitoring, logging, and observability
-- Security best practices and compliance
-
-Guidelines:
-- Provide specific, actionable solutions
-- Include relevant commands and configurations
-- Explain potential risks and mitigation strategies
-- Suggest best practices and industry standards
-- Ask clarifying questions when requirements are unclear
-
-Always prioritize security, reliability, and maintainability.`
-```
-
-#### **DSPy Optimization Code**
-
-```python
-import dspy
-from dspy.teleprompt import BootstrapFewShot
-
-class DevOpsAssistant(dspy.Signature):
-    """Expert DevOps assistance with practical, secure solutions."""
-    query = dspy.InputField(desc="DevOps question or problem")
-    solution = dspy.OutputField(desc="Detailed, actionable solution")
-
-class DevOpsModule(dspy.Module):
-    def **init**(self):
-        super().**init**()
-        self.generate_solution = dspy.ChainOfThought(DevOpsAssistant)
-
-    def forward(self, query):
-        return self.generate_solution(query=query)
-
-# Training data from DSPyGround samples
-trainset = [
-    dspy.Example(
-        query="How do I deploy a Node.js app with zero downtime?",
-        solution="Use blue-green deployment with load balancer..."
-    ),
-    # More examples from DSPyGround
-]
-
-# Optimize
-teleprompter = BootstrapFewShot(metric=devops_accuracy_metric)
-optimized_assistant = teleprompter.compile(DevOpsModule(), trainset=trainset)
-```
-
-### **Code Review Assistant**
-
-#### **DSPyGround Configuration**
-
-```typescript
-export default {
-  systemPrompt: `You are a senior software engineer conducting code reviews.
-
-  Focus on:
-  - Code quality and maintainability
-  - Security vulnerabilities
-  - Performance implications
-  - Best practices adherence
-
-  Provide constructive feedback with specific suggestions.`,
-
-  tools: {
-    analyzeCode: tool({
-      description: 'Analyze code for issues',
-      parameters: z.object({
-        code: z.string(),
-        language: z.string(),
-      }),
-      execute: async ({ code, language }) => {
-        // Static analysis integration
-        return analyzeCodeQuality(code, language);
-      },
-    }),
-  },
-
-  preferences: {
-    selectedMetrics: ['accuracy', 'tone', 'efficiency'],
-    batchSize: 5,
-    numRollouts: 15,
-  }
-}
-```
-
-#### **DSPy Implementation**
-
-```python
-class CodeReview(dspy.Signature):
-    """Comprehensive code review with actionable feedback."""
-    code = dspy.InputField(desc="Code to review")
-    language = dspy.InputField(desc="Programming language")
-    review = dspy.OutputField(desc="Detailed code review with suggestions")
-
-class CodeReviewModule(dspy.Module):
-    def **init**(self):
-        super().**init**()
-        self.review_code = dspy.ChainOfThought(CodeReview)
-
-    def forward(self, code, language):
-        return self.review_code(code=code, language=language)
-
-# Multi-stage optimization
-from dspy.teleprompt import MIPRO
-
-teleprompter = MIPRO(
-    metric=code_review_quality_metric,
-    num_candidates=20,
-    init_temperature=1.0
-)
-optimized_reviewer = teleprompter.compile(CodeReviewModule(), trainset=code_samples)
-```
-
-## Metrics and Evaluation
-
-### **Custom Metrics for DevOps**
-
-```python
-def devops_accuracy_metric(example, pred, trace=None):
-    """Evaluate DevOps solution accuracy."""
-    # Check for security considerations
-    security_score = check_security_mentions(pred.solution)
-
-    # Verify technical accuracy
-    technical_score = verify_technical_details(pred.solution, example.query)
-
-    # Assess actionability
-    actionability_score = assess_actionability(pred.solution)
-
-    return (security_score + technical_score + actionability_score) / 3
-
-def code_review_quality_metric(example, pred, trace=None):
-    """Evaluate code review quality."""
-    # Check for common issues identification
-    issue_detection = check_issue_detection(pred.review, example.code)
-
-    # Assess suggestion quality
-    suggestion_quality = evaluate_suggestions(pred.review)
-
-    # Verify constructive tone
-    tone_score = assess_constructive_tone(pred.review)
-
-    return (issue_detection + suggestion_quality + tone_score) / 3
-```
-
-### **DSPyGround Metrics Configuration**
-
-```typescript
-metricsPrompt: {
-  evaluation_instructions: `You are an expert evaluator for DevOps AI assistants.
-
-  Evaluate responses across these dimensions:
-  - Technical accuracy and completeness
-  - Security awareness and best practices
-  - Clarity and actionability of instructions
-  - Appropriate level of detail for the context`,
-
-  dimensions: {
-    technical_accuracy: {
-      name: 'Technical Accuracy',
-      description: 'Is the technical information correct and up-to-date?',
-      weight: 1.0
-    },
-    security_awareness: {
-      name: 'Security Awareness',
-      description: 'Does the response consider security implications?',
-      weight: 0.9
-    },
-    actionability: {
-      name: 'Actionability',
-      description: 'Can the user immediately implement the solution?',
-      weight: 0.8
-    },
-    completeness: {
-      name: 'Completeness',
-      description: 'Does the response address all aspects of the question?',
-      weight: 0.7
-    }
-  }
-}
-```
-
-## Iterative Improvement Process
-
-### **Week 1: Foundation**
-
-1. Create basic prompts in DSPyGround
-2. Collect 50+ diverse samples
-3. Run initial GEPA optimization
-4. Deploy improved prompts
-
-### **Week 2: Refinement**
-
-1. Monitor real-world performance
-2. Collect edge cases and failures
-3. Add negative examples to training
-4. Re-optimize with expanded dataset
-
-### **Week 3: Specialization**
-
-1. Create domain-specific variants
-2. Optimize for specific use cases
-3. A/B test different approaches
-4. Measure business impact
-
-### **Ongoing: Maintenance**
-
-1. Regular performance monitoring
-2. Quarterly re-optimization
-3. Adaptation to new requirements
-4. Integration of user feedback
-
-## Best Practices
-
-### **Sample Quality**
-
-- **Diversity**: Cover various scenarios and edge cases
-- **Quality**: Use real-world, high-quality examples
-- **Balance**: Include both positive and negative examples
-- **Context**: Preserve conversation context and nuance
-
-### **Optimization Strategy**
-
-- **Start Simple**: Begin with basic optimizers
-- **Iterate Gradually**: Make incremental improvements
-- **Measure Everything**: Track multiple metrics consistently
-- **Validate Thoroughly**: Test on held-out datasets
-
-### **Production Deployment**
-
-- **Gradual Rollout**: Deploy to small user groups first
-- **Monitor Closely**: Track performance and user satisfaction
-- **Rollback Ready**: Maintain previous versions for quick rollback
-- **Continuous Learning**: Collect feedback for next iteration
-
-## Integration Points
-
-### **With AI DevOps Framework**
-
-- Export optimized prompts to provider scripts
-- Integrate with quality control workflows
-- Use in documentation generation
-- Apply to server management tasks
-
-### **With External Systems**
-
-- CI/CD pipeline integration
-- Monitoring and alerting systems
-- Code review platforms
-- Documentation platforms

package/.agent/tools/context/toon.md

@@ -1,292 +0,0 @@
----
-description: TOON format for token-efficient data serialization
-mode: subagent
-tools:
-  read: true
-  write: true
-  edit: true
-  bash: true
-  glob: true
-  grep: true
-  webfetch: false
-  task: true
----
-
-# TOON Format Integration - AI DevOps Framework
-
-<!-- AI-CONTEXT-START -->
-
-## Quick Reference
-
-- **Purpose**: 20-60% token reduction vs JSON for LLM prompts
-- **CLI**: `npx @toon-format/cli` (no install needed)
-- **Commands**: `toon-helper.sh [encode|decode|compare|validate|batch|stdin-encode|stdin-decode] [input] [output]`
-- **Format**: `users[2]{id,name,role}:` followed by `1,Alice,admin` rows
-- **Delimiters**: comma (default), tab (`\t`), pipe (`|`)
-- **Best for**: Tabular data (60%+ savings), config data, API responses
-- **Config**: `configs/toon-config.json`
-- **Resources**: https://toonformat.dev, https://github.com/toon-format/toon
-<!-- AI-CONTEXT-END -->
-
-**Token-Oriented Object Notation (TOON)** - Compact, human-readable, schema-aware JSON for LLM prompts.
-
-## Overview
-
-TOON is a revolutionary data format designed specifically for Large Language Models (LLMs), offering:
-
-- **20-60% token reduction** compared to JSON
-- **Human-readable tabular format** for structured data
-- **Schema-aware** with explicit array lengths and field headers
-- **Better LLM comprehension** and generation accuracy
-- **Supports nested structures** and mixed data types
-
-## 🚀 **Quick Start**
-
-### **Installation**
-
-TOON CLI is automatically available through npx (no installation required):
-
-```bash
-# Test TOON CLI
-npx @toon-format/cli --help
-
-# Or use the AI DevOps helper
-./.agent/scripts/toon-helper.sh info
-```
-
-### **Basic Usage**
-
-```bash
-# Convert JSON to TOON
-./.agent/scripts/toon-helper.sh encode data.json output.toon
-
-# Convert TOON back to JSON
-./.agent/scripts/toon-helper.sh decode output.toon restored.json
-
-# Show token efficiency comparison
-./.agent/scripts/toon-helper.sh compare large-dataset.json
-
-# Validate TOON format
-./.agent/scripts/toon-helper.sh validate data.toon
-```
-
-## 📊 **Format Examples**
-
-### **Simple Object**
-
-```json
-{"id": 1, "name": "Alice", "active": true}
-```
-
-**TOON:**
-
-```toon
-id: 1
-name: Alice
-active: true
-```
-
-### **Tabular Data (Most Efficient)**
-
-```json
-{
-  "users": [
-    {"id": 1, "name": "Alice", "role": "admin"},
-    {"id": 2, "name": "Bob", "role": "user"}
-  ]
-}
-```
-
-**TOON:**
-
-```toon
-users[2]{id,name,role}:
-  1,Alice,admin
-  2,Bob,user
-```
-
-### **Nested Structures**
-
-```json
-{
-  "project": {
-    "name": "AI DevOps",
-    "metrics": [
-      {"date": "2025-01-01", "users": 100},
-      {"date": "2025-01-02", "users": 150}
-    ]
-  }
-}
-```
-
-**TOON:**
-
-```toon
-project:
-  name: AI DevOps
-  metrics[2]{date,users}:
-    2025-01-01,100
-    2025-01-02,150
-```
-
-## 🛠️ **Helper Script Commands**
-
-### **File Conversion**
-
-```bash
-# Basic conversion
-./.agent/scripts/toon-helper.sh encode input.json output.toon
-
-# With tab delimiter (often more efficient)
-./.agent/scripts/toon-helper.sh encode input.json output.toon '\t' true
-
-# Decode with lenient validation
-./.agent/scripts/toon-helper.sh decode input.toon output.json false
-```
-
-### **Batch Processing**
-
-```bash
-# Convert directory of JSON files to TOON
-./.agent/scripts/toon-helper.sh batch ./json-files ./toon-files json-to-toon
-
-# Convert directory of TOON files to JSON
-./.agent/scripts/toon-helper.sh batch ./toon-files ./json-files toon-to-json '\t'
-```
-
-### **Stream Processing**
-
-```bash
-# Convert from stdin
-cat data.json | ./.agent/scripts/toon-helper.sh stdin-encode
-echo '{"name": "test"}' | ./.agent/scripts/toon-helper.sh stdin-encode '\t' true
-
-# Decode from stdin
-cat data.toon | ./.agent/scripts/toon-helper.sh stdin-decode
-```
-
-## 🎯 **AI DevOps Use Cases**
-
-### **1. Configuration Data**
-
-Perfect for server configurations, deployment settings, and infrastructure data:
-
-```bash
-# Convert server inventory to TOON for AI analysis
-./.agent/scripts/toon-helper.sh encode servers.json servers.toon '\t' true
-```
-
-### **2. API Response Formatting**
-
-Reduce token costs when sending API responses to LLMs:
-
-```bash
-# Convert API responses for efficient LLM processing
-curl -s "https://api.example.com/data" | ./.agent/scripts/toon-helper.sh stdin-encode
-```
-
-### **3. Database Exports**
-
-Efficient format for database query results:
-
-```bash
-# Export database results in TOON format
-mysql -e "SELECT * FROM users" --json | ./.agent/scripts/toon-helper.sh stdin-encode '\t'
-```
-
-### **4. Log Analysis**
-
-Structure log data for AI analysis:
-
-```bash
-# Convert structured logs to TOON
-./.agent/scripts/toon-helper.sh batch ./logs/json ./logs/toon json-to-toon
-```
-
-## 📈 **Token Efficiency**
-
-TOON provides significant token savings, especially for tabular data:
-
-| Data Type | JSON Tokens | TOON Tokens | Savings |
-|-----------|-------------|-------------|---------|
-| Employee Records | 126,860 | 49,831 | 60.7% |
-| Time Series | 22,250 | 9,120 | 59.0% |
-| GitHub Repos | 15,145 | 8,745 | 42.3% |
-| E-commerce Orders | 108,806 | 72,771 | 33.1% |
-
-## 🔧 **Configuration**
-
-Copy and customize the configuration template:
-
-```bash
-cp configs/toon-config.json.txt configs/toon-config.json
-# Edit with your preferences
-```
-
-Key configuration options:
-
-- **default_delimiter**: Choose between `,`, `\t`, or `|`
-- **key_folding**: Enable path compression for nested data
-- **batch_processing**: Configure concurrent conversions
-- **ai_prompts**: Optimize for LLM interactions
-
-## 🤖 **LLM Integration**
-
-### **Sending TOON to LLMs**
-
-```markdown
-Data is in TOON format (2-space indent, arrays show length and fields):
-
-```toon
-users[3]{id,name,role,lastLogin}:
-  1,Alice,admin,2025-01-15T10:30:00Z
-  2,Bob,user,2025-01-14T15:22:00Z
-  3,Charlie,user,2025-01-13T09:45:00Z
-```
-
-Task: Return only users with role "user" as TOON.
-
-```text
-
-```
-
-### **Generating TOON from LLMs**
-
-- Show expected header format: `users[N]{id,name,role}:`
-- Specify rules: 2-space indent, no trailing spaces, [N] matches row count
-- Request code block output only
-
-## 🔍 **Validation & Quality**
-
-```bash
-# Validate TOON format
-./.agent/scripts/toon-helper.sh validate data.toon
-
-# Compare efficiency
-./.agent/scripts/toon-helper.sh compare large-dataset.json
-
-# Show format information
-./.agent/scripts/toon-helper.sh info
-```
-
-## 📚 **Resources**
-
-- **Official Website**: https://toonformat.dev
-- **GitHub Repository**: https://github.com/toon-format/toon
-- **Specification**: https://github.com/toon-format/toon/blob/main/spec.md
-- **Benchmarks**: https://github.com/toon-format/toon#benchmarks
-- **TypeScript SDK**: `npm install @toon-format/toon`
-
-## 🛡️ **Security & Best Practices**
-
-- **Validate input**: Always validate TOON data before processing
-- **Use strict mode**: Enable strict validation for production use
-- **Backup originals**: Keep JSON backups when converting
-- **Test conversions**: Verify round-trip conversion accuracy
-- **Monitor token usage**: Track actual token savings in your use case
-
----
-
-**Integration Status**: ✅ **Fully Integrated** with AI DevOps Framework
-**Maintenance**: Automated updates via npm/npx
-**Support**: Community-driven with active development