tech-hub-skills 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +250 -0
- package/bin/cli.js +241 -0
- package/bin/copilot.js +182 -0
- package/bin/postinstall.js +42 -0
- package/package.json +46 -0
- package/tech_hub_skills/roles/ai-engineer/skills/01-prompt-engineering/README.md +252 -0
- package/tech_hub_skills/roles/ai-engineer/skills/02-rag-pipeline/README.md +448 -0
- package/tech_hub_skills/roles/ai-engineer/skills/03-agent-orchestration/README.md +599 -0
- package/tech_hub_skills/roles/ai-engineer/skills/04-llm-guardrails/README.md +735 -0
- package/tech_hub_skills/roles/ai-engineer/skills/05-vector-embeddings/README.md +711 -0
- package/tech_hub_skills/roles/ai-engineer/skills/06-llm-evaluation/README.md +777 -0
- package/tech_hub_skills/roles/azure/skills/01-infrastructure-fundamentals/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/02-data-factory/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/03-synapse-analytics/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/04-databricks/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/05-functions/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/06-kubernetes-service/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/07-openai-service/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/08-machine-learning/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/09-storage-adls/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/10-networking/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/11-sql-cosmos/README.md +264 -0
- package/tech_hub_skills/roles/azure/skills/12-event-hubs/README.md +264 -0
- package/tech_hub_skills/roles/code-review/skills/01-automated-code-review/README.md +394 -0
- package/tech_hub_skills/roles/code-review/skills/02-pr-review-workflow/README.md +427 -0
- package/tech_hub_skills/roles/code-review/skills/03-code-quality-gates/README.md +518 -0
- package/tech_hub_skills/roles/code-review/skills/04-reviewer-assignment/README.md +504 -0
- package/tech_hub_skills/roles/code-review/skills/05-review-analytics/README.md +540 -0
- package/tech_hub_skills/roles/data-engineer/skills/01-lakehouse-architecture/README.md +550 -0
- package/tech_hub_skills/roles/data-engineer/skills/02-etl-pipeline/README.md +580 -0
- package/tech_hub_skills/roles/data-engineer/skills/03-data-quality/README.md +579 -0
- package/tech_hub_skills/roles/data-engineer/skills/04-streaming-pipelines/README.md +608 -0
- package/tech_hub_skills/roles/data-engineer/skills/05-performance-optimization/README.md +547 -0
- package/tech_hub_skills/roles/data-governance/skills/01-data-catalog/README.md +112 -0
- package/tech_hub_skills/roles/data-governance/skills/02-data-lineage/README.md +129 -0
- package/tech_hub_skills/roles/data-governance/skills/03-data-quality-framework/README.md +182 -0
- package/tech_hub_skills/roles/data-governance/skills/04-access-control/README.md +39 -0
- package/tech_hub_skills/roles/data-governance/skills/05-master-data-management/README.md +40 -0
- package/tech_hub_skills/roles/data-governance/skills/06-compliance-privacy/README.md +46 -0
- package/tech_hub_skills/roles/data-scientist/skills/01-eda-automation/README.md +230 -0
- package/tech_hub_skills/roles/data-scientist/skills/02-statistical-modeling/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/03-feature-engineering/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/04-predictive-modeling/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/05-customer-analytics/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/06-campaign-analysis/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/07-experimentation/README.md +264 -0
- package/tech_hub_skills/roles/data-scientist/skills/08-data-visualization/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/01-cicd-pipeline/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/02-container-orchestration/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/03-infrastructure-as-code/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/04-gitops/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/05-environment-management/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/06-automated-testing/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/07-release-management/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/08-monitoring-alerting/README.md +264 -0
- package/tech_hub_skills/roles/devops/skills/09-devsecops/README.md +265 -0
- package/tech_hub_skills/roles/finops/skills/01-cost-visibility/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/02-resource-tagging/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/03-budget-management/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/04-reserved-instances/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/05-spot-optimization/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/06-storage-tiering/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/07-compute-rightsizing/README.md +264 -0
- package/tech_hub_skills/roles/finops/skills/08-chargeback/README.md +264 -0
- package/tech_hub_skills/roles/ml-engineer/skills/01-mlops-pipeline/README.md +566 -0
- package/tech_hub_skills/roles/ml-engineer/skills/02-feature-engineering/README.md +655 -0
- package/tech_hub_skills/roles/ml-engineer/skills/03-model-training/README.md +704 -0
- package/tech_hub_skills/roles/ml-engineer/skills/04-model-serving/README.md +845 -0
- package/tech_hub_skills/roles/ml-engineer/skills/05-model-monitoring/README.md +874 -0
- package/tech_hub_skills/roles/mlops/skills/01-ml-pipeline-orchestration/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/02-experiment-tracking/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/03-model-registry/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/04-feature-store/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/05-model-deployment/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/06-model-observability/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/07-data-versioning/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/08-ab-testing/README.md +264 -0
- package/tech_hub_skills/roles/mlops/skills/09-automated-retraining/README.md +264 -0
- package/tech_hub_skills/roles/platform-engineer/skills/01-internal-developer-platform/README.md +153 -0
- package/tech_hub_skills/roles/platform-engineer/skills/02-self-service-infrastructure/README.md +57 -0
- package/tech_hub_skills/roles/platform-engineer/skills/03-slo-sli-management/README.md +59 -0
- package/tech_hub_skills/roles/platform-engineer/skills/04-developer-experience/README.md +57 -0
- package/tech_hub_skills/roles/platform-engineer/skills/05-incident-management/README.md +73 -0
- package/tech_hub_skills/roles/platform-engineer/skills/06-capacity-management/README.md +59 -0
- package/tech_hub_skills/roles/product-designer/skills/01-requirements-discovery/README.md +407 -0
- package/tech_hub_skills/roles/product-designer/skills/02-user-research/README.md +382 -0
- package/tech_hub_skills/roles/product-designer/skills/03-brainstorming-ideation/README.md +437 -0
- package/tech_hub_skills/roles/product-designer/skills/04-ux-design/README.md +496 -0
- package/tech_hub_skills/roles/product-designer/skills/05-product-market-fit/README.md +376 -0
- package/tech_hub_skills/roles/product-designer/skills/06-stakeholder-management/README.md +412 -0
- package/tech_hub_skills/roles/security-architect/skills/01-pii-detection/README.md +319 -0
- package/tech_hub_skills/roles/security-architect/skills/02-threat-modeling/README.md +264 -0
- package/tech_hub_skills/roles/security-architect/skills/03-infrastructure-security/README.md +264 -0
- package/tech_hub_skills/roles/security-architect/skills/04-iam/README.md +264 -0
- package/tech_hub_skills/roles/security-architect/skills/05-application-security/README.md +264 -0
- package/tech_hub_skills/roles/security-architect/skills/06-secrets-management/README.md +264 -0
- package/tech_hub_skills/roles/security-architect/skills/07-security-monitoring/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/01-architecture-patterns/README.md +337 -0
- package/tech_hub_skills/roles/system-design/skills/02-requirements-engineering/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/03-scalability/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/04-high-availability/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/05-cost-optimization-design/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/06-api-design/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/07-observability-architecture/README.md +264 -0
- package/tech_hub_skills/roles/system-design/skills/08-process-automation/PROCESS_TEMPLATE.md +336 -0
- package/tech_hub_skills/roles/system-design/skills/08-process-automation/README.md +521 -0
- package/tech_hub_skills/skills/README.md +336 -0
- package/tech_hub_skills/skills/ai-engineer.md +104 -0
- package/tech_hub_skills/skills/azure.md +149 -0
- package/tech_hub_skills/skills/code-review.md +399 -0
- package/tech_hub_skills/skills/compliance-automation.md +747 -0
- package/tech_hub_skills/skills/data-engineer.md +113 -0
- package/tech_hub_skills/skills/data-governance.md +102 -0
- package/tech_hub_skills/skills/data-scientist.md +123 -0
- package/tech_hub_skills/skills/devops.md +160 -0
- package/tech_hub_skills/skills/docker.md +160 -0
- package/tech_hub_skills/skills/enterprise-dashboard.md +613 -0
- package/tech_hub_skills/skills/finops.md +184 -0
- package/tech_hub_skills/skills/ml-engineer.md +115 -0
- package/tech_hub_skills/skills/mlops.md +187 -0
- package/tech_hub_skills/skills/optimization-advisor.md +329 -0
- package/tech_hub_skills/skills/orchestrator.md +497 -0
- package/tech_hub_skills/skills/platform-engineer.md +102 -0
- package/tech_hub_skills/skills/process-automation.md +226 -0
- package/tech_hub_skills/skills/process-changelog.md +184 -0
- package/tech_hub_skills/skills/process-documentation.md +484 -0
- package/tech_hub_skills/skills/process-kanban.md +324 -0
- package/tech_hub_skills/skills/process-versioning.md +214 -0
- package/tech_hub_skills/skills/product-designer.md +104 -0
- package/tech_hub_skills/skills/project-starter.md +443 -0
- package/tech_hub_skills/skills/security-architect.md +135 -0
- package/tech_hub_skills/skills/system-design.md +126 -0
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Post-install script for tech-hub-skills npm package
|
|
5
|
+
* Shows installation success message but doesn't auto-install to avoid unwanted side effects
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
const colors = {
|
|
9
|
+
reset: '\x1b[0m',
|
|
10
|
+
bright: '\x1b[1m',
|
|
11
|
+
cyan: '\x1b[36m',
|
|
12
|
+
green: '\x1b[32m',
|
|
13
|
+
yellow: '\x1b[33m',
|
|
14
|
+
};
|
|
15
|
+
|
|
16
|
+
console.log(`
|
|
17
|
+
${colors.cyan}
|
|
18
|
+
╔═══════════════════════════════════════════════════════════╗
|
|
19
|
+
║ TECH HUB SKILLS - Successfully Installed! ║
|
|
20
|
+
╚═══════════════════════════════════════════════════════════╝${colors.reset}
|
|
21
|
+
|
|
22
|
+
${colors.green}✓ Package installed successfully!${colors.reset}
|
|
23
|
+
|
|
24
|
+
${colors.bright}Next Steps:${colors.reset}
|
|
25
|
+
|
|
26
|
+
1. Install skills to your project:
|
|
27
|
+
${colors.cyan}npx tech-hub-skills install${colors.reset}
|
|
28
|
+
|
|
29
|
+
2. Or install globally:
|
|
30
|
+
${colors.cyan}npx tech-hub-skills install --global${colors.reset}
|
|
31
|
+
|
|
32
|
+
3. View all available skills:
|
|
33
|
+
${colors.cyan}npx tech-hub-skills list${colors.reset}
|
|
34
|
+
|
|
35
|
+
${colors.bright}Quick Start:${colors.reset}
|
|
36
|
+
After installing, use in Claude Code:
|
|
37
|
+
${colors.yellow}@orchestrator "Build a customer churn prediction model"${colors.reset}
|
|
38
|
+
|
|
39
|
+
${colors.bright}Documentation:${colors.reset}
|
|
40
|
+
https://github.com/6ogo/tech-hub-skills
|
|
41
|
+
|
|
42
|
+
`);
|
package/package.json
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "tech-hub-skills",
|
|
3
|
+
"version": "1.0.0",
|
|
4
|
+
"description": "110+ production-ready AI agent skills for Claude Code and GitHub Copilot",
|
|
5
|
+
"main": "bin/cli.js",
|
|
6
|
+
"bin": {
|
|
7
|
+
"tech-hub-skills": "./bin/cli.js"
|
|
8
|
+
},
|
|
9
|
+
"scripts": {
|
|
10
|
+
"postinstall": "node bin/postinstall.js"
|
|
11
|
+
},
|
|
12
|
+
"keywords": [
|
|
13
|
+
"claude",
|
|
14
|
+
"claude-code",
|
|
15
|
+
"github-copilot",
|
|
16
|
+
"copilot",
|
|
17
|
+
"ai-agents",
|
|
18
|
+
"skills",
|
|
19
|
+
"llm",
|
|
20
|
+
"ai-engineer",
|
|
21
|
+
"data-engineer",
|
|
22
|
+
"security",
|
|
23
|
+
"enterprise",
|
|
24
|
+
"vscode"
|
|
25
|
+
],
|
|
26
|
+
"author": "6ogo",
|
|
27
|
+
"license": "MIT",
|
|
28
|
+
"repository": {
|
|
29
|
+
"type": "git",
|
|
30
|
+
"url": "https://github.com/6ogo/tech-hub-skills.git"
|
|
31
|
+
},
|
|
32
|
+
"files": [
|
|
33
|
+
"tech_hub_skills/skills/**/*.md",
|
|
34
|
+
"tech_hub_skills/roles/**/*.md",
|
|
35
|
+
"bin/**/*.js",
|
|
36
|
+
"README.md",
|
|
37
|
+
"LICENSE"
|
|
38
|
+
],
|
|
39
|
+
"engines": {
|
|
40
|
+
"node": ">=16.0.0"
|
|
41
|
+
},
|
|
42
|
+
"homepage": "https://github.com/6ogo/tech-hub-skills#readme",
|
|
43
|
+
"bugs": {
|
|
44
|
+
"url": "https://github.com/6ogo/tech-hub-skills/issues"
|
|
45
|
+
}
|
|
46
|
+
}
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
# Skill 1: Prompt Engineering & Optimization
|
|
2
|
+
|
|
3
|
+
## 🎯 Overview
|
|
4
|
+
Master the art and science of crafting, versioning, and optimizing prompts for production LLM applications.
|
|
5
|
+
|
|
6
|
+
## 🔗 Connections
|
|
7
|
+
- **ML Engineer**: Model evaluation and performance metrics (ml-01)
|
|
8
|
+
- **MLOps**: Prompt versioning and experiment tracking (mo-01, mo-03)
|
|
9
|
+
- **Data Scientist**: A/B testing and statistical analysis of prompt variations (ds-08)
|
|
10
|
+
- **System Design**: Cost optimization and latency management (sd-05)
|
|
11
|
+
- **FinOps**: LLM cost optimization, prompt caching strategies (fo-01, fo-03, fo-07)
|
|
12
|
+
- **DevOps**: CI/CD for prompt templates, version control (do-01, do-05)
|
|
13
|
+
- **Security Architect**: Prompt injection prevention, content safety (sa-08)
|
|
14
|
+
|
|
15
|
+
## 🛠️ Tools Included
|
|
16
|
+
|
|
17
|
+
### 1. `prompt_template_manager.py`
|
|
18
|
+
Version-controlled prompt template system with variable injection and inheritance.
|
|
19
|
+
|
|
20
|
+
### 2. `token_cost_estimator.py`
|
|
21
|
+
Calculate costs across providers (OpenAI, Claude, Gemini) with real-time pricing.
|
|
22
|
+
|
|
23
|
+
### 3. `prompt_ab_tester.py`
|
|
24
|
+
A/B testing framework for comparing prompt variations with statistical significance.
|
|
25
|
+
|
|
26
|
+
### 4. `prompt_quality_scorer.py`
|
|
27
|
+
Automated quality scoring for relevance, coherence, and factuality.
|
|
28
|
+
|
|
29
|
+
## 📊 Key Metrics
|
|
30
|
+
- Token efficiency (output quality per token)
|
|
31
|
+
- Cost per query
|
|
32
|
+
- Response latency
|
|
33
|
+
- Quality scores (0-100)
|
|
34
|
+
|
|
35
|
+
## 🚀 Quick Start
|
|
36
|
+
|
|
37
|
+
```python
|
|
38
|
+
from prompt_template_manager import PromptTemplate
|
|
39
|
+
from token_cost_estimator import estimate_cost
|
|
40
|
+
|
|
41
|
+
# Load a template
|
|
42
|
+
template = PromptTemplate.load("marketing_email_generator")
|
|
43
|
+
|
|
44
|
+
# Inject variables
|
|
45
|
+
prompt = template.render(product="AI Course", audience="Data Scientists")
|
|
46
|
+
|
|
47
|
+
# Estimate cost
|
|
48
|
+
cost = estimate_cost(prompt, model="gpt-4", provider="openai")
|
|
49
|
+
print(f"Estimated cost: ${cost:.4f}")
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## 📚 Best Practices
|
|
53
|
+
|
|
54
|
+
### Cost Optimization (FinOps Integration)
|
|
55
|
+
1. **Enable Prompt Caching** - Save up to 90% on costs by caching system prompts and context
|
|
56
|
+
- Cache static system prompts with `cache_control: ephemeral`
|
|
57
|
+
- Cache large knowledge bases and conversation history
|
|
58
|
+
- Monitor cache hit rates and adjust caching strategy
|
|
59
|
+
- Reference: FinOps fo-07 (AI/ML Cost Optimization)
|
|
60
|
+
|
|
61
|
+
2. **Track and Optimize Token Usage**
|
|
62
|
+
- Monitor input/output token ratios
|
|
63
|
+
- Set token budgets per application/user
|
|
64
|
+
- Use smaller models (Haiku) for simple tasks, Sonnet/Opus for complex reasoning
|
|
65
|
+
- Implement token usage alerts and cost dashboards
|
|
66
|
+
- Reference: FinOps fo-01 (Cost Monitoring), fo-03 (Budget Management)
|
|
67
|
+
|
|
68
|
+
3. **Optimize Prompt Length**
|
|
69
|
+
- Remove redundant instructions
|
|
70
|
+
- Use structured prompts with clear sections
|
|
71
|
+
- Implement dynamic context pruning for long conversations
|
|
72
|
+
- Reference: AI Engineer best practices on prompt optimization
|
|
73
|
+
|
|
74
|
+
### Version Control & Deployment (DevOps Integration)
|
|
75
|
+
4. **Version Prompts with Semantic Versioning**
|
|
76
|
+
- Store prompts in Git with version tags (v1.0.0, v1.1.0)
|
|
77
|
+
- Use CI/CD pipelines to deploy prompt changes
|
|
78
|
+
- Implement blue-green deployments for critical prompts
|
|
79
|
+
- Reference: DevOps do-01 (CI/CD), do-05 (GitOps)
|
|
80
|
+
|
|
81
|
+
5. **Automate Prompt Testing**
|
|
82
|
+
- Run automated tests on prompt changes before deployment
|
|
83
|
+
- Use golden datasets for regression testing
|
|
84
|
+
- Implement quality gates in CI/CD pipelines
|
|
85
|
+
- Reference: DevOps do-02 (Testing Automation)
|
|
86
|
+
|
|
87
|
+
### Experimentation & Quality (MLOps Integration)
|
|
88
|
+
6. **Use A/B Testing for Production Changes**
|
|
89
|
+
- Deploy prompt variations to subset of users
|
|
90
|
+
- Track statistical significance before full rollout
|
|
91
|
+
- Use experiment tracking (MLflow, Azure ML)
|
|
92
|
+
- Reference: MLOps mo-01 (Experiment Tracking), Data Scientist ds-08
|
|
93
|
+
|
|
94
|
+
7. **Monitor Quality Metrics Over Time**
|
|
95
|
+
- Track quality score degradation (model drift)
|
|
96
|
+
- Set up alerts for quality drops below thresholds
|
|
97
|
+
- Implement continuous evaluation pipelines
|
|
98
|
+
- Reference: MLOps mo-04 (Monitoring), ML Engineer ml-05
|
|
99
|
+
|
|
100
|
+
### Security & Compliance
|
|
101
|
+
8. **Prevent Prompt Injection Attacks**
|
|
102
|
+
- Validate and sanitize user inputs
|
|
103
|
+
- Use structured prompts with clear delimiters
|
|
104
|
+
- Implement content safety filters
|
|
105
|
+
- Reference: Security Architect sa-08 (LLM Security)
|
|
106
|
+
|
|
107
|
+
9. **Audit Prompt Usage**
|
|
108
|
+
- Log all prompt executions for compliance
|
|
109
|
+
- Track PII in prompts and responses
|
|
110
|
+
- Implement GDPR-compliant data retention
|
|
111
|
+
- Reference: Security Architect sa-01 (PII Detection)
|
|
112
|
+
|
|
113
|
+
### Azure-Specific Best Practices
|
|
114
|
+
10. **Leverage Azure OpenAI Features**
|
|
115
|
+
- Use managed identities for authentication
|
|
116
|
+
- Enable diagnostic logging to Azure Monitor
|
|
117
|
+
- Implement retry logic with exponential backoff
|
|
118
|
+
- Use provisioned throughput for high-volume applications
|
|
119
|
+
- Reference: Azure az-05 (Azure OpenAI Service)
|
|
120
|
+
|
|
121
|
+
## 💰 Cost Optimization Examples
|
|
122
|
+
|
|
123
|
+
### Prompt Caching Implementation (90% Cost Savings)
|
|
124
|
+
```python
|
|
125
|
+
from anthropic import Anthropic
|
|
126
|
+
|
|
127
|
+
client = Anthropic()
|
|
128
|
+
|
|
129
|
+
# Without caching: $0.015 per request
|
|
130
|
+
# With caching: $0.0015 per request (10x cheaper!)
|
|
131
|
+
|
|
132
|
+
response = client.messages.create(
|
|
133
|
+
model="claude-3-5-sonnet-20241022",
|
|
134
|
+
max_tokens=2048,
|
|
135
|
+
system=[
|
|
136
|
+
{
|
|
137
|
+
"type": "text",
|
|
138
|
+
"text": "You are a customer support AI assistant with access to our knowledge base...",
|
|
139
|
+
"cache_control": {"type": "ephemeral"} # Cache this! Saves 90%
|
|
140
|
+
},
|
|
141
|
+
{
|
|
142
|
+
"type": "text",
|
|
143
|
+
"text": LARGE_KNOWLEDGE_BASE, # 50K tokens
|
|
144
|
+
"cache_control": {"type": "ephemeral"} # Cache this too!
|
|
145
|
+
}
|
|
146
|
+
],
|
|
147
|
+
messages=[{"role": "user", "content": user_message}]
|
|
148
|
+
)
|
|
149
|
+
|
|
150
|
+
# Monitor cache performance
|
|
151
|
+
print(f"Cache hit rate: {response.usage.cache_read_input_tokens / response.usage.input_tokens * 100:.1f}%")
|
|
152
|
+
print(f"Cost savings: ${calculate_savings(response.usage):.4f}")
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
### Cost Tracking Dashboard
|
|
156
|
+
```python
|
|
157
|
+
from token_cost_estimator import LLMCostTracker
|
|
158
|
+
|
|
159
|
+
tracker = LLMCostTracker()
|
|
160
|
+
|
|
161
|
+
# Track all requests
|
|
162
|
+
tracker.log_request(
|
|
163
|
+
model="claude-3-5-sonnet",
|
|
164
|
+
input_tokens=1000,
|
|
165
|
+
output_tokens=500,
|
|
166
|
+
cached_tokens=800,
|
|
167
|
+
user_id="team_alpha",
|
|
168
|
+
project="customer_support"
|
|
169
|
+
)
|
|
170
|
+
|
|
171
|
+
# Generate cost reports
|
|
172
|
+
monthly_report = tracker.generate_report(period="monthly")
|
|
173
|
+
print(f"Total cost: ${monthly_report.total_cost}")
|
|
174
|
+
print(f"Cost by team: {monthly_report.cost_by_user}")
|
|
175
|
+
print(f"Savings from caching: ${monthly_report.cache_savings}")
|
|
176
|
+
|
|
177
|
+
# Set budget alerts
|
|
178
|
+
tracker.set_budget_alert(
|
|
179
|
+
project="customer_support",
|
|
180
|
+
monthly_budget=1000.00,
|
|
181
|
+
alert_threshold=0.8 # Alert at 80%
|
|
182
|
+
)
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## 🚀 CI/CD for Prompt Templates
|
|
186
|
+
|
|
187
|
+
### Git-Based Prompt Versioning
|
|
188
|
+
```yaml
|
|
189
|
+
# .github/workflows/prompt-deployment.yml
|
|
190
|
+
name: Deploy Prompts
|
|
191
|
+
|
|
192
|
+
on:
|
|
193
|
+
push:
|
|
194
|
+
paths:
|
|
195
|
+
- 'prompts/**'
|
|
196
|
+
branches:
|
|
197
|
+
- main
|
|
198
|
+
|
|
199
|
+
jobs:
|
|
200
|
+
test-and-deploy:
|
|
201
|
+
runs-on: ubuntu-latest
|
|
202
|
+
steps:
|
|
203
|
+
- name: Run prompt tests
|
|
204
|
+
run: pytest tests/test_prompts.py
|
|
205
|
+
|
|
206
|
+
- name: A/B test new prompts
|
|
207
|
+
run: python scripts/ab_test_prompts.py --canary 10%
|
|
208
|
+
|
|
209
|
+
- name: Deploy to production
|
|
210
|
+
if: success()
|
|
211
|
+
run: python scripts/deploy_prompts.py --env production
|
|
212
|
+
|
|
213
|
+
- name: Monitor quality metrics
|
|
214
|
+
run: python scripts/monitor_quality.py --duration 1h
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
## 📊 Enhanced Metrics
|
|
218
|
+
|
|
219
|
+
| Metric | Description | Target | Monitoring Tool |
|
|
220
|
+
|--------|-------------|--------|-----------------|
|
|
221
|
+
| **Token Efficiency** | Output quality per token | >0.8 | Custom dashboard |
|
|
222
|
+
| **Cost per Query** | Average cost including caching | <$0.01 | Azure Monitor + FinOps dashboard |
|
|
223
|
+
| **Cache Hit Rate** | % of tokens served from cache | >70% | Application Insights |
|
|
224
|
+
| **Response Latency** | P95 latency | <2s | Azure Monitor |
|
|
225
|
+
| **Quality Score** | Automated quality rating | >85/100 | MLOps monitoring |
|
|
226
|
+
| **A/B Test Win Rate** | % of new prompts that beat baseline | >60% | MLflow experiments |
|
|
227
|
+
|
|
228
|
+
## 🔄 Integration with Other Skills
|
|
229
|
+
|
|
230
|
+
### End-to-End Workflow
|
|
231
|
+
```
|
|
232
|
+
1. Develop Prompt (ai-01)
|
|
233
|
+
↓
|
|
234
|
+
2. Version in Git (do-05)
|
|
235
|
+
↓
|
|
236
|
+
3. A/B Test (ds-08, mo-01)
|
|
237
|
+
↓
|
|
238
|
+
4. Monitor Costs (fo-01, fo-07)
|
|
239
|
+
↓
|
|
240
|
+
5. Deploy via CI/CD (do-01)
|
|
241
|
+
↓
|
|
242
|
+
6. Monitor Quality (ml-05, mo-04)
|
|
243
|
+
↓
|
|
244
|
+
7. Optimize Caching (fo-07)
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
## 🎯 Quick Wins
|
|
248
|
+
1. **Enable caching today** - Immediate 70-90% cost reduction for conversational apps
|
|
249
|
+
2. **Set up cost tracking** - Know where your LLM budget is going
|
|
250
|
+
3. **Version prompts in Git** - Enable rollbacks and A/B testing
|
|
251
|
+
4. **Automate testing** - Catch regressions before production
|
|
252
|
+
5. **Monitor quality** - Detect model drift early
|