aidevops 2.52.1

package/.agent/tools/context/prompt-optimization.md

@@ -0,0 +1,338 @@
+---
+description: Prompt optimization using DSPy and DSPyGround
+mode: subagent
+tools:
+  read: true
+  write: false
+  edit: false
+  bash: true
+  glob: true
+  grep: true
+  webfetch: true
+  task: true
+---
+
+# Prompt Optimization with DSPy & DSPyGround
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **DSPy**: Programmatic prompt optimization framework
+- **DSPyGround**: Visual playground for interactive prompt refinement
+- **Workflow**: Bootstrap → Refine → Collect Samples → Optimize → Deploy
+- **Optimizers**: BootstrapFewShot, MIPRO, ChainOfThought
+- **Metrics**: Technical accuracy, security awareness, actionability, completeness
+- **Process**: Week 1 Foundation → Week 2 Refinement → Week 3 Specialization → Ongoing Maintenance
+- **Integration**: Export optimized prompts to provider scripts, quality workflows
+<!-- AI-CONTEXT-END -->
+
+## Overview
+
+This guide covers comprehensive prompt optimization strategies using both DSPy (programmatic optimization) and DSPyGround (visual playground) within the AI DevOps Framework.
+
+## Optimization Workflow
+
+### **Phase 1: Initial Development (DSPyGround)**
+
+1. **Bootstrap with Basic Prompt**
+   - Start with simple, clear instructions
+   - Define core functionality and constraints
+   - Test basic scenarios interactively
+
+2. **Interactive Refinement**
+   - Use DSPyGround's chat interface
+   - Collect diverse conversation samples
+   - Identify edge cases and failure modes
+
+3. **Sample Collection**
+   - Save positive examples (good responses)
+   - Mark negative examples (problematic responses)
+   - Organize samples into logical groups
+
+### **Phase 2: Automated Optimization (DSPy)**
+
+1. **Data Preparation**
+   - Export samples from DSPyGround
+   - Convert to DSPy training format
+   - Create validation and test sets
+
+2. **Systematic Optimization**
+   - Apply multiple DSPy optimizers
+   - Compare performance metrics
+   - Select best-performing variants
+
+3. **Production Deployment**
+   - Integrate optimized prompts
+   - Monitor performance metrics
+   - Iterate based on real-world feedback
+
+## Practical Examples
+
+### **DevOps Assistant Optimization**
+
+#### **Initial Prompt (DSPyGround)**
+
+```typescript
+systemPrompt: `You are a DevOps assistant. Help with server management.`
+```
+
+#### **Refined Prompt (After DSPyGround)**
+
+```typescript
+systemPrompt: `You are an expert DevOps engineer with 10+ years of experience.
+
+Your expertise includes:
+- Infrastructure automation and configuration management
+- CI/CD pipeline design and optimization
+- Container orchestration with Docker and Kubernetes
+- Cloud platform management (AWS, Azure, GCP)
+- Monitoring, logging, and observability
+- Security best practices and compliance
+
+Guidelines:
+- Provide specific, actionable solutions
+- Include relevant commands and configurations
+- Explain potential risks and mitigation strategies
+- Suggest best practices and industry standards
+- Ask clarifying questions when requirements are unclear
+
+Always prioritize security, reliability, and maintainability.`
+```
+
+#### **DSPy Optimization Code**
+
+```python
+import dspy
+from dspy.teleprompt import BootstrapFewShot
+
+class DevOpsAssistant(dspy.Signature):
+    """Expert DevOps assistance with practical, secure solutions."""
+    query = dspy.InputField(desc="DevOps question or problem")
+    solution = dspy.OutputField(desc="Detailed, actionable solution")
+
+class DevOpsModule(dspy.Module):
+    def **init**(self):
+        super().**init**()
+        self.generate_solution = dspy.ChainOfThought(DevOpsAssistant)
+
+    def forward(self, query):
+        return self.generate_solution(query=query)
+
+# Training data from DSPyGround samples
+trainset = [
+    dspy.Example(
+        query="How do I deploy a Node.js app with zero downtime?",
+        solution="Use blue-green deployment with load balancer..."
+    ),
+    # More examples from DSPyGround
+]
+
+# Optimize
+teleprompter = BootstrapFewShot(metric=devops_accuracy_metric)
+optimized_assistant = teleprompter.compile(DevOpsModule(), trainset=trainset)
+```
+
+### **Code Review Assistant**
+
+#### **DSPyGround Configuration**
+
+```typescript
+export default {
+  systemPrompt: `You are a senior software engineer conducting code reviews.
+
+  Focus on:
+  - Code quality and maintainability
+  - Security vulnerabilities
+  - Performance implications
+  - Best practices adherence
+
+  Provide constructive feedback with specific suggestions.`,
+
+  tools: {
+    analyzeCode: tool({
+      description: 'Analyze code for issues',
+      parameters: z.object({
+        code: z.string(),
+        language: z.string(),
+      }),
+      execute: async ({ code, language }) => {
+        // Static analysis integration
+        return analyzeCodeQuality(code, language);
+      },
+    }),
+  },
+
+  preferences: {
+    selectedMetrics: ['accuracy', 'tone', 'efficiency'],
+    batchSize: 5,
+    numRollouts: 15,
+  }
+}
+```
+
+#### **DSPy Implementation**
+
+```python
+class CodeReview(dspy.Signature):
+    """Comprehensive code review with actionable feedback."""
+    code = dspy.InputField(desc="Code to review")
+    language = dspy.InputField(desc="Programming language")
+    review = dspy.OutputField(desc="Detailed code review with suggestions")
+
+class CodeReviewModule(dspy.Module):
+    def **init**(self):
+        super().**init**()
+        self.review_code = dspy.ChainOfThought(CodeReview)
+
+    def forward(self, code, language):
+        return self.review_code(code=code, language=language)
+
+# Multi-stage optimization
+from dspy.teleprompt import MIPRO
+
+teleprompter = MIPRO(
+    metric=code_review_quality_metric,
+    num_candidates=20,
+    init_temperature=1.0
+)
+optimized_reviewer = teleprompter.compile(CodeReviewModule(), trainset=code_samples)
+```
+
+## Metrics and Evaluation
+
+### **Custom Metrics for DevOps**
+
+```python
+def devops_accuracy_metric(example, pred, trace=None):
+    """Evaluate DevOps solution accuracy."""
+    # Check for security considerations
+    security_score = check_security_mentions(pred.solution)
+
+    # Verify technical accuracy
+    technical_score = verify_technical_details(pred.solution, example.query)
+
+    # Assess actionability
+    actionability_score = assess_actionability(pred.solution)
+
+    return (security_score + technical_score + actionability_score) / 3
+
+def code_review_quality_metric(example, pred, trace=None):
+    """Evaluate code review quality."""
+    # Check for common issues identification
+    issue_detection = check_issue_detection(pred.review, example.code)
+
+    # Assess suggestion quality
+    suggestion_quality = evaluate_suggestions(pred.review)
+
+    # Verify constructive tone
+    tone_score = assess_constructive_tone(pred.review)
+
+    return (issue_detection + suggestion_quality + tone_score) / 3
+```
+
+### **DSPyGround Metrics Configuration**
+
+```typescript
+metricsPrompt: {
+  evaluation_instructions: `You are an expert evaluator for DevOps AI assistants.
+
+  Evaluate responses across these dimensions:
+  - Technical accuracy and completeness
+  - Security awareness and best practices
+  - Clarity and actionability of instructions
+  - Appropriate level of detail for the context`,
+
+  dimensions: {
+    technical_accuracy: {
+      name: 'Technical Accuracy',
+      description: 'Is the technical information correct and up-to-date?',
+      weight: 1.0
+    },
+    security_awareness: {
+      name: 'Security Awareness',
+      description: 'Does the response consider security implications?',
+      weight: 0.9
+    },
+    actionability: {
+      name: 'Actionability',
+      description: 'Can the user immediately implement the solution?',
+      weight: 0.8
+    },
+    completeness: {
+      name: 'Completeness',
+      description: 'Does the response address all aspects of the question?',
+      weight: 0.7
+    }
+  }
+}
+```
+
+## Iterative Improvement Process
+
+### **Week 1: Foundation**
+
+1. Create basic prompts in DSPyGround
+2. Collect 50+ diverse samples
+3. Run initial GEPA optimization
+4. Deploy improved prompts
+
+### **Week 2: Refinement**
+
+1. Monitor real-world performance
+2. Collect edge cases and failures
+3. Add negative examples to training
+4. Re-optimize with expanded dataset
+
+### **Week 3: Specialization**
+
+1. Create domain-specific variants
+2. Optimize for specific use cases
+3. A/B test different approaches
+4. Measure business impact
+
+### **Ongoing: Maintenance**
+
+1. Regular performance monitoring
+2. Quarterly re-optimization
+3. Adaptation to new requirements
+4. Integration of user feedback
+
+## Best Practices
+
+### **Sample Quality**
+
+- **Diversity**: Cover various scenarios and edge cases
+- **Quality**: Use real-world, high-quality examples
+- **Balance**: Include both positive and negative examples
+- **Context**: Preserve conversation context and nuance
+
+### **Optimization Strategy**
+
+- **Start Simple**: Begin with basic optimizers
+- **Iterate Gradually**: Make incremental improvements
+- **Measure Everything**: Track multiple metrics consistently
+- **Validate Thoroughly**: Test on held-out datasets
+
+### **Production Deployment**
+
+- **Gradual Rollout**: Deploy to small user groups first
+- **Monitor Closely**: Track performance and user satisfaction
+- **Rollback Ready**: Maintain previous versions for quick rollback
+- **Continuous Learning**: Collect feedback for next iteration
+
+## Integration Points
+
+### **With AI DevOps Framework**
+
+- Export optimized prompts to provider scripts
+- Integrate with quality control workflows
+- Use in documentation generation
+- Apply to server management tasks
+
+### **With External Systems**
+
+- CI/CD pipeline integration
+- Monitoring and alerting systems
+- Code review platforms
+- Documentation platforms

package/.agent/tools/context/toon.md

@@ -0,0 +1,292 @@
+---
+description: TOON format for token-efficient data serialization
+mode: subagent
+tools:
+  read: true
+  write: true
+  edit: true
+  bash: true
+  glob: true
+  grep: true
+  webfetch: false
+  task: true
+---
+
+# TOON Format Integration - AI DevOps Framework
+
+<!-- AI-CONTEXT-START -->
+
+## Quick Reference
+
+- **Purpose**: 20-60% token reduction vs JSON for LLM prompts
+- **CLI**: `npx @toon-format/cli` (no install needed)
+- **Commands**: `toon-helper.sh [encode|decode|compare|validate|batch|stdin-encode|stdin-decode] [input] [output]`
+- **Format**: `users[2]{id,name,role}:` followed by `1,Alice,admin` rows
+- **Delimiters**: comma (default), tab (`\t`), pipe (`|`)
+- **Best for**: Tabular data (60%+ savings), config data, API responses
+- **Config**: `configs/toon-config.json`
+- **Resources**: https://toonformat.dev, https://github.com/toon-format/toon
+<!-- AI-CONTEXT-END -->
+
+**Token-Oriented Object Notation (TOON)** - Compact, human-readable, schema-aware JSON for LLM prompts.
+
+## Overview
+
+TOON is a revolutionary data format designed specifically for Large Language Models (LLMs), offering:
+
+- **20-60% token reduction** compared to JSON
+- **Human-readable tabular format** for structured data
+- **Schema-aware** with explicit array lengths and field headers
+- **Better LLM comprehension** and generation accuracy
+- **Supports nested structures** and mixed data types
+
+## 🚀 **Quick Start**
+
+### **Installation**
+
+TOON CLI is automatically available through npx (no installation required):
+
+```bash
+# Test TOON CLI
+npx @toon-format/cli --help
+
+# Or use the AI DevOps helper
+./.agent/scripts/toon-helper.sh info
+```
+
+### **Basic Usage**
+
+```bash
+# Convert JSON to TOON
+./.agent/scripts/toon-helper.sh encode data.json output.toon
+
+# Convert TOON back to JSON
+./.agent/scripts/toon-helper.sh decode output.toon restored.json
+
+# Show token efficiency comparison
+./.agent/scripts/toon-helper.sh compare large-dataset.json
+
+# Validate TOON format
+./.agent/scripts/toon-helper.sh validate data.toon
+```
+
+## 📊 **Format Examples**
+
+### **Simple Object**
+
+```json
+{"id": 1, "name": "Alice", "active": true}
+```
+
+**TOON:**
+
+```toon
+id: 1
+name: Alice
+active: true
+```
+
+### **Tabular Data (Most Efficient)**
+
+```json
+{
+  "users": [
+    {"id": 1, "name": "Alice", "role": "admin"},
+    {"id": 2, "name": "Bob", "role": "user"}
+  ]
+}
+```
+
+**TOON:**
+
+```toon
+users[2]{id,name,role}:
+  1,Alice,admin
+  2,Bob,user
+```
+
+### **Nested Structures**
+
+```json
+{
+  "project": {
+    "name": "AI DevOps",
+    "metrics": [
+      {"date": "2025-01-01", "users": 100},
+      {"date": "2025-01-02", "users": 150}
+    ]
+  }
+}
+```
+
+**TOON:**
+
+```toon
+project:
+  name: AI DevOps
+  metrics[2]{date,users}:
+    2025-01-01,100
+    2025-01-02,150
+```
+
+## 🛠️ **Helper Script Commands**
+
+### **File Conversion**
+
+```bash
+# Basic conversion
+./.agent/scripts/toon-helper.sh encode input.json output.toon
+
+# With tab delimiter (often more efficient)
+./.agent/scripts/toon-helper.sh encode input.json output.toon '\t' true
+
+# Decode with lenient validation
+./.agent/scripts/toon-helper.sh decode input.toon output.json false
+```
+
+### **Batch Processing**
+
+```bash
+# Convert directory of JSON files to TOON
+./.agent/scripts/toon-helper.sh batch ./json-files ./toon-files json-to-toon
+
+# Convert directory of TOON files to JSON
+./.agent/scripts/toon-helper.sh batch ./toon-files ./json-files toon-to-json '\t'
+```
+
+### **Stream Processing**
+
+```bash
+# Convert from stdin
+cat data.json | ./.agent/scripts/toon-helper.sh stdin-encode
+echo '{"name": "test"}' | ./.agent/scripts/toon-helper.sh stdin-encode '\t' true
+
+# Decode from stdin
+cat data.toon | ./.agent/scripts/toon-helper.sh stdin-decode
+```
+
+## 🎯 **AI DevOps Use Cases**
+
+### **1. Configuration Data**
+
+Perfect for server configurations, deployment settings, and infrastructure data:
+
+```bash
+# Convert server inventory to TOON for AI analysis
+./.agent/scripts/toon-helper.sh encode servers.json servers.toon '\t' true
+```
+
+### **2. API Response Formatting**
+
+Reduce token costs when sending API responses to LLMs:
+
+```bash
+# Convert API responses for efficient LLM processing
+curl -s "https://api.example.com/data" | ./.agent/scripts/toon-helper.sh stdin-encode
+```
+
+### **3. Database Exports**
+
+Efficient format for database query results:
+
+```bash
+# Export database results in TOON format
+mysql -e "SELECT * FROM users" --json | ./.agent/scripts/toon-helper.sh stdin-encode '\t'
+```
+
+### **4. Log Analysis**
+
+Structure log data for AI analysis:
+
+```bash
+# Convert structured logs to TOON
+./.agent/scripts/toon-helper.sh batch ./logs/json ./logs/toon json-to-toon
+```
+
+## 📈 **Token Efficiency**
+
+TOON provides significant token savings, especially for tabular data:
+
+| Data Type | JSON Tokens | TOON Tokens | Savings |
+|-----------|-------------|-------------|---------|
+| Employee Records | 126,860 | 49,831 | 60.7% |
+| Time Series | 22,250 | 9,120 | 59.0% |
+| GitHub Repos | 15,145 | 8,745 | 42.3% |
+| E-commerce Orders | 108,806 | 72,771 | 33.1% |
+
+## 🔧 **Configuration**
+
+Copy and customize the configuration template:
+
+```bash
+cp configs/toon-config.json.txt configs/toon-config.json
+# Edit with your preferences
+```
+
+Key configuration options:
+
+- **default_delimiter**: Choose between `,`, `\t`, or `|`
+- **key_folding**: Enable path compression for nested data
+- **batch_processing**: Configure concurrent conversions
+- **ai_prompts**: Optimize for LLM interactions
+
+## 🤖 **LLM Integration**
+
+### **Sending TOON to LLMs**
+
+```markdown
+Data is in TOON format (2-space indent, arrays show length and fields):
+
+```toon
+users[3]{id,name,role,lastLogin}:
+  1,Alice,admin,2025-01-15T10:30:00Z
+  2,Bob,user,2025-01-14T15:22:00Z
+  3,Charlie,user,2025-01-13T09:45:00Z
+```
+
+Task: Return only users with role "user" as TOON.
+
+```text
+
+```
+
+### **Generating TOON from LLMs**
+
+- Show expected header format: `users[N]{id,name,role}:`
+- Specify rules: 2-space indent, no trailing spaces, [N] matches row count
+- Request code block output only
+
+## 🔍 **Validation & Quality**
+
+```bash
+# Validate TOON format
+./.agent/scripts/toon-helper.sh validate data.toon
+
+# Compare efficiency
+./.agent/scripts/toon-helper.sh compare large-dataset.json
+
+# Show format information
+./.agent/scripts/toon-helper.sh info
+```
+
+## 📚 **Resources**
+
+- **Official Website**: https://toonformat.dev
+- **GitHub Repository**: https://github.com/toon-format/toon
+- **Specification**: https://github.com/toon-format/toon/blob/main/spec.md
+- **Benchmarks**: https://github.com/toon-format/toon#benchmarks
+- **TypeScript SDK**: `npm install @toon-format/toon`
+
+## 🛡️ **Security & Best Practices**
+
+- **Validate input**: Always validate TOON data before processing
+- **Use strict mode**: Enable strict validation for production use
+- **Backup originals**: Keep JSON backups when converting
+- **Test conversions**: Verify round-trip conversion accuracy
+- **Monitor token usage**: Track actual token savings in your use case
+
+---
+
+**Integration Status**: ✅ **Fully Integrated** with AI DevOps Framework
+**Maintenance**: Automated updates via npm/npx
+**Support**: Community-driven with active development