swe-workflow-skills 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -17
- package/VERSION +1 -1
- package/catalog.json +95 -5
- package/package.json +1 -1
- package/roles.json +58 -7
- package/skills/ai-evaluation/SKILL.md +111 -0
- package/skills/ai-evaluation/evals/evals.json +40 -0
- package/skills/ai-evaluation/references/eval-tooling.md +63 -0
- package/skills/architecture-design/SKILL.md +3 -1
- package/skills/brainstorming/SKILL.md +92 -0
- package/skills/brainstorming/evals/evals.json +40 -0
- package/skills/browser-verification/SKILL.md +95 -0
- package/skills/browser-verification/evals/evals.json +40 -0
- package/skills/build-vs-buy/SKILL.md +109 -0
- package/skills/build-vs-buy/evals/evals.json +40 -0
- package/skills/cicd-pipeline/SKILL.md +1 -0
- package/skills/code-archaeology/SKILL.md +103 -0
- package/skills/code-archaeology/evals/evals.json +41 -0
- package/skills/compliance-privacy/SKILL.md +113 -0
- package/skills/compliance-privacy/evals/evals.json +41 -0
- package/skills/compliance-privacy/references/obligations-map.md +69 -0
- package/skills/data-pipeline-design/SKILL.md +116 -0
- package/skills/data-pipeline-design/evals/evals.json +41 -0
- package/skills/data-pipeline-design/references/orchestration.md +64 -0
- package/skills/data-quality/SKILL.md +107 -0
- package/skills/data-quality/evals/evals.json +39 -0
- package/skills/data-quality/references/tooling.md +45 -0
- package/skills/deployment-checklist/SKILL.md +2 -1
- package/skills/dx-audit/SKILL.md +97 -0
- package/skills/dx-audit/evals/evals.json +40 -0
- package/skills/feature-planning/SKILL.md +5 -1
- package/skills/finops-cost-optimization/SKILL.md +109 -0
- package/skills/finops-cost-optimization/evals/evals.json +40 -0
- package/skills/git-workflow/SKILL.md +10 -0
- package/skills/llm-app-engineering/SKILL.md +102 -0
- package/skills/llm-app-engineering/evals/evals.json +40 -0
- package/skills/llm-app-engineering/references/patterns.md +70 -0
- package/skills/ml-experiment-tracking/SKILL.md +1 -1
- package/skills/ml-pipeline-design/SKILL.md +1 -1
- package/skills/mobile-architecture/SKILL.md +106 -0
- package/skills/mobile-architecture/evals/evals.json +40 -0
- package/skills/mobile-release/SKILL.md +109 -0
- package/skills/mobile-release/evals/evals.json +41 -0
- package/skills/plan-execution/SKILL.md +115 -0
- package/skills/plan-execution/evals/evals.json +55 -0
- package/skills/project-documentation/SKILL.md +4 -0
- package/skills/release-management/SKILL.md +97 -0
- package/skills/release-management/evals/evals.json +56 -0
- package/skills/release-management/references/release-tooling.md +106 -0
- package/skills/resilience-engineering/SKILL.md +121 -0
- package/skills/resilience-engineering/evals/evals.json +40 -0
- package/skills/security-audit/SKILL.md +2 -2
- package/skills/skill-router/SKILL.md +46 -3
- package/skills/subagent-orchestration/SKILL.md +100 -0
- package/skills/subagent-orchestration/evals/evals.json +40 -0
- package/skills/threat-modeling/SKILL.md +101 -0
- package/skills/threat-modeling/evals/evals.json +40 -0
- package/skills/threat-modeling/references/stride-mitigations.md +90 -0
- package/skills/writing-skills/SKILL.md +27 -11
package/README.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# SWE Workflow Skills for Claude Code
|
|
2
2
|
|
|
3
3
|
[](https://github.com/SWEStash/swe-workflow-skills/actions/workflows/roles-check.yml)
|
|
4
|
-

|
|
5
|
+
[](LICENSE)
|
|
6
6
|
|
|
7
|
-
A curated library of **
|
|
7
|
+
A curated library of **62 Claude Code Agent Skills** that walk Claude through the
|
|
8
8
|
software lifecycle the way a disciplined senior engineer would — planning, design,
|
|
9
9
|
TDD, review, security, deployment, incidents, and the project-management work around
|
|
10
10
|
them.
|
|
@@ -17,21 +17,37 @@ LLM-as-judge eval harness.
|
|
|
17
17
|
|
|
18
18
|
## Why this library
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
20
|
+
Not just a pile of skills — an activation architecture, an eval discipline, and
|
|
21
|
+
full-SDLC breadth that popular collections don't cover:
|
|
22
|
+
|
|
23
|
+
- **An orchestrator, not auto-trigger roulette.** A `skill-router` skill maps your
|
|
24
|
+
intent to the right skill(s) and chains them across phases. Activation is *routed and
|
|
25
|
+
deterministic*, not left to fuzzy auto-triggering — the routing eval harness measures
|
|
26
|
+
it (top-1 accuracy 1.00, zero misroutes on the current baseline, regression-gated in
|
|
27
|
+
CI).
|
|
23
28
|
- **Scales past Claude's skill-listing budget.** Claude Code only injects skill
|
|
24
29
|
descriptions up to ~1% of context, so large libraries silently stop triggering past
|
|
25
30
|
~20–40 skills. This library keeps only the router + safety skills "loud" and lists the
|
|
26
|
-
rest **name-only** —
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
+
rest **name-only** — the mechanism Claude Code's own docs now recommend for large
|
|
32
|
+
installs — so the router can invoke every skill by name, with no cropping and without
|
|
33
|
+
making you pre-pick a subset ([how it works](docs/ROLES.md)).
|
|
34
|
+
- **Tested like code, not prose.** Every skill ships 3 evals; two LLM-as-judge
|
|
35
|
+
harnesses (content quality and routing accuracy) replay them with skill loaded vs
|
|
36
|
+
absent and gate regressions in CI. Safety-critical skills (deploys, releases, tests,
|
|
37
|
+
incidents, security) are **hardened**: an Iron Law, a rationalization table distilled
|
|
38
|
+
from real baseline failures, and pressure tests that try to talk the agent out of it.
|
|
39
|
+
- **Full-SDLC breadth.** Planning, architecture/ADRs, API and data design, TDD, review,
|
|
40
|
+
security, releases, deploys, GitOps, observability, incidents, MLOps, LLM apps
|
|
41
|
+
and AI evaluation, data pipelines and data quality, and the PM/strategy work
|
|
42
|
+
around them — a wider slice of the lifecycle than the community collections we
|
|
43
|
+
know of aim for (they tend to go deep on the coding inner loop; see
|
|
44
|
+
[Acknowledgements](#acknowledgements)).
|
|
45
|
+
- **Role-scoped.** `/role backend` (or `frontend`, `devops`, `ml`, `ai`, `data`,
|
|
46
|
+
`security`, `architect`, `em`, `pm`, `strategy`, `qa`, `mobile`, `designer`) promotes
|
|
47
|
+
a working set to auto-trigger; the rest stay one route away.
|
|
31
48
|
- **Cross-platform install.** The installer and SessionStart hook are **pure Node** — the
|
|
32
49
|
one runtime Claude Code already requires — so they run identically on Linux, macOS, and
|
|
33
50
|
Windows (no bash, Python, or `sed`).
|
|
34
|
-
- **Tested.** Every skill ships 3 evals; safety/discipline skills add pressure tests.
|
|
35
51
|
|
|
36
52
|
## Quick Start
|
|
37
53
|
|
|
@@ -46,7 +62,7 @@ web, claude.ai chat, and Cowork):
|
|
|
46
62
|
**Want the whole library with the orchestrator** (CLI) — no clone needed:
|
|
47
63
|
|
|
48
64
|
```bash
|
|
49
|
-
npx swe-workflow-skills install --global # all
|
|
65
|
+
npx swe-workflow-skills install --global # all 62 skills + router + /role + the SessionStart hook
|
|
50
66
|
```
|
|
51
67
|
|
|
52
68
|
Or from a clone: `node install.mjs --global`.
|
|
@@ -104,15 +120,18 @@ review) live in the `skill-router` skill and **[ROLES.md](docs/ROLES.md)**.
|
|
|
104
120
|
|
|
105
121
|
## What's included
|
|
106
122
|
|
|
107
|
-
|
|
123
|
+
62 skills — **[full catalog → SKILLS.md](docs/SKILLS.md)**:
|
|
108
124
|
|
|
109
125
|
| Area | Count | Examples |
|
|
110
126
|
|------|-------|----------|
|
|
111
|
-
| Software Engineering |
|
|
112
|
-
| Project Management |
|
|
113
|
-
| DevOps |
|
|
127
|
+
| Software Engineering | 30 | feature-planning, plan-execution, architecture-design, threat-modeling, compliance-privacy, code-archaeology, tdd-workflow, security-audit, browser-verification, subagent-orchestration |
|
|
128
|
+
| Project Management | 8 | brainstorming, prd-writing, effort-estimation, build-vs-buy, metrics-and-okrs, retrospective, strategic-review |
|
|
129
|
+
| DevOps | 8 | containerization, cicd-pipeline, release-management, gitops-delivery, resilience-engineering, finops-cost-optimization |
|
|
114
130
|
| Design | 3 | ui-ux-design, frontend-architecture, accessibility-design |
|
|
115
131
|
| MLOps | 3 | ml-pipeline-design, ml-experiment-tracking, ml-model-deployment |
|
|
132
|
+
| AI Engineering | 2 | llm-app-engineering, ai-evaluation |
|
|
133
|
+
| Data Engineering | 2 | data-pipeline-design, data-quality |
|
|
134
|
+
| Mobile | 2 | mobile-architecture, mobile-release |
|
|
116
135
|
| Evaluation & Monitoring | 2 | observability-design, test-data-strategy |
|
|
117
136
|
| Meta | 2 | skill-router, writing-skills |
|
|
118
137
|
|
|
@@ -139,6 +158,27 @@ New or improved skills are welcome — start with **[AUTHORING.md](docs/AUTHORIN
|
|
|
139
158
|
install the `writing-skills` skill). The short version: descriptions are everything, keep
|
|
140
159
|
SKILL.md concise with detail in `references/`, and ship exactly 3 evals.
|
|
141
160
|
|
|
161
|
+
## Acknowledgements
|
|
162
|
+
|
|
163
|
+
This library stands on ideas from projects we found genuinely useful:
|
|
164
|
+
|
|
165
|
+
- **[obra/superpowers](https://github.com/obra/superpowers)** — the guardrail pattern
|
|
166
|
+
our hardened safety skills adopt (an Iron Law, a rationalization table distilled from
|
|
167
|
+
real failures, and pressure tests that try to talk the agent out of it) comes from
|
|
168
|
+
Jesse Vincent's work here, as does the idea of a Socratic brainstorming skill as the
|
|
169
|
+
entry point to a feature. Superpowers goes deeper on the coding inner loop than we do;
|
|
170
|
+
if that's what you want, use it.
|
|
171
|
+
- **[anthropics/skills](https://github.com/anthropics/skills)** — Anthropic's official
|
|
172
|
+
skills (and `skill-creator` in particular) are the authoring practice we benchmark
|
|
173
|
+
ours against; the eval-first loop in [AUTHORING.md](docs/AUTHORING.md) follows the
|
|
174
|
+
same instinct.
|
|
175
|
+
- **The [Agent Skills docs](https://code.claude.com/docs/en/skills) and Anthropic's
|
|
176
|
+
engineering posts** — progressive disclosure, the listing budget, and the
|
|
177
|
+
`name-only` + `skillOverrides` activation surface this library's routing model is
|
|
178
|
+
built on.
|
|
179
|
+
- **The awesome-claude-skills community lists** — the ecosystem survey that shaped our
|
|
180
|
+
gap analysis of what a full-SDLC library should cover.
|
|
181
|
+
|
|
142
182
|
## License
|
|
143
183
|
|
|
144
184
|
MIT — see [LICENSE](LICENSE).
|
package/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
0.
|
|
1
|
+
0.2.0
|
package/catalog.json
CHANGED
|
@@ -6,6 +6,11 @@
|
|
|
6
6
|
"description": "Design and implement accessible web/mobile apps to WCAG 2.1/2.2 — semantic HTML, ARIA patterns, keyboard navigation, focus management, color contrast, screen readers, accessible forms. Triggers: accessibility, a11y, WCAG, screen reader, keyboard navigation, ARIA, focus management, color contrast, alt text, semantic HTML, tab order, focus trap, skip link, announce. Build it in, don't bolt it on.",
|
|
7
7
|
"path": "accessibility-design/SKILL.md"
|
|
8
8
|
},
|
|
9
|
+
{
|
|
10
|
+
"name": "ai-evaluation",
|
|
11
|
+
"description": "Define how to evaluate ML models and GenAI/LLM apps — golden datasets, offline metrics, RAG evaluation (faithfulness, answer relevance, context precision/recall) with ragas/deepeval/promptfoo, LLM-as-judge design and calibration, eval regression gates in CI, online A/B and human feedback. Triggers: evaluate the model, eval my chatbot, RAG evals, ragas, LLM as judge, golden dataset, eval suite, is the new prompt better, hallucination rate, benchmark our AI app. Logging/comparing runs → ml-experiment-tracking; building the app → llm-app-engineering.",
|
|
12
|
+
"path": "ai-evaluation/SKILL.md"
|
|
13
|
+
},
|
|
9
14
|
{
|
|
10
15
|
"name": "api-design",
|
|
11
16
|
"description": "Design RESTful and GraphQL APIs — endpoint naming, request/response contracts, error handling, pagination, versioning, auth patterns, OpenAPI specs. Triggers: design the API, API contract, REST API, GraphQL schema, error response format, pagination, API versioning, OpenAPI, swagger, endpoints. Use architecture-design for REST-vs-GraphQL or monolith-vs-microservices decisions.",
|
|
@@ -13,7 +18,7 @@
|
|
|
13
18
|
},
|
|
14
19
|
{
|
|
15
20
|
"name": "architecture-design",
|
|
16
|
-
"description": "Guide architectural and structural decisions using Architecture Decision Records (ADRs) — new services, pattern choices, database selection, component boundaries, state management, costly-to-reverse decisions. Triggers: how should I architect, which pattern, design decision, should I use X or Y, system design, ADR, component design, architectural trade-off.",
|
|
21
|
+
"description": "Guide architectural and structural decisions using Architecture Decision Records (ADRs) — new services, pattern choices, database selection, component boundaries, state management, costly-to-reverse decisions. Triggers: how should I architect, which pattern, design decision, should I use X or Y, system design, ADR, component design, architectural trade-off. Owns internal-structure decisions — build vs buy/adopt a vendor or OSS capability → build-vs-buy.",
|
|
17
22
|
"path": "architecture-design/SKILL.md"
|
|
18
23
|
},
|
|
19
24
|
{
|
|
@@ -21,21 +26,46 @@
|
|
|
21
26
|
"description": "Design and maintain architecture documentation with multi-level diagrams (C4) and docs-as-code. Triggers: architecture diagram, document the architecture, C4 diagram, system context, container diagram, component diagram, architecture docs, docs-as-code, PlantUML, Structurizr, D2, Mermaid architecture, infrastructure diagram, runtime flow. Use architecture-design for ADRs.",
|
|
22
27
|
"path": "architecture-documentation/SKILL.md"
|
|
23
28
|
},
|
|
29
|
+
{
|
|
30
|
+
"name": "brainstorming",
|
|
31
|
+
"description": "Facilitate divergent ideation before committing to a spec — Socratic questioning, widening the option space, challenging assumptions and inherited constraints, then converging on candidates. Triggers: brainstorm, let's explore ideas, what are our options, help me think through, generate ideas, what could we build, not sure what I want, explore approaches, ideate. Once ideas converge, hand off to prd-writing (WHAT/WHY) or feature-planning (breakdown) — this skill opens options; those close them.",
|
|
32
|
+
"path": "brainstorming/SKILL.md"
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"name": "browser-verification",
|
|
36
|
+
"description": "Verify a web app change by driving the real UI in a browser — reproduce the user flow end to end, watch console errors and network failures, exercise loading/error/empty states, capture screenshots as evidence. Use before claiming a UI change works. Triggers: check it in the browser, verify the UI, click through the flow, console errors, test it like a user, smoke test the frontend. Designing persistent test suites → test-suite-design; the generic done-gate → verification-before-completion (this is its browser-specific practice).",
|
|
37
|
+
"path": "browser-verification/SKILL.md"
|
|
38
|
+
},
|
|
24
39
|
{
|
|
25
40
|
"name": "bug-investigating",
|
|
26
41
|
"description": "Systematic debugging and root cause analysis using structured methodology — reproduce, isolate, hypothesize, verify. Triggers: this doesn't work, I'm getting an error, something is broken, bug, debug this, why is this happening, unexpected behavior, regression, root cause, stack trace, error message.",
|
|
27
42
|
"path": "bug-investigating/SKILL.md"
|
|
28
43
|
},
|
|
44
|
+
{
|
|
45
|
+
"name": "build-vs-buy",
|
|
46
|
+
"description": "Decide whether to build in-house, buy a vendor/SaaS product, or adopt OSS — core-vs-commodity framing, multi-year total cost of ownership, integration and maintenance burden, vendor viability, lock-in and exit costs. Triggers: build vs buy, build or buy, vendor evaluation, SaaS vs in-house, should we adopt this open source, TCO, total cost of ownership, lock-in, exit cost, make or buy, off the shelf. Internal-structure decisions (patterns, service boundaries, ADRs for your own code) → architecture-design.",
|
|
47
|
+
"path": "build-vs-buy/SKILL.md"
|
|
48
|
+
},
|
|
29
49
|
{
|
|
30
50
|
"name": "cicd-pipeline",
|
|
31
51
|
"description": "Design CI/CD pipelines for automated build, test, deploy — GitHub Actions, GitLab CI, quality gates. Triggers: CI/CD, pipeline, GitHub Actions, GitLab CI, workflow, automated deployment, build pipeline, continuous integration, continuous deployment, automate tests on PR, deploy automatically, quality gate.",
|
|
32
52
|
"path": "cicd-pipeline/SKILL.md"
|
|
33
53
|
},
|
|
54
|
+
{
|
|
55
|
+
"name": "code-archaeology",
|
|
56
|
+
"description": "Understand unfamiliar or legacy code before changing it — map entry points, mine git history (blame, churn, log -S), trace data flow, recover the why behind odd code, pin current behavior with characterization tests, find seams for safe change. Triggers: understand this codebase, legacy code, inherited this project, what does this code do, why is it written this way, nobody knows how this works, onboard to this repo, is it safe to change this. Prioritizing debt → technical-debt-review; improving code you already understand → refactoring.",
|
|
57
|
+
"path": "code-archaeology/SKILL.md"
|
|
58
|
+
},
|
|
34
59
|
{
|
|
35
60
|
"name": "code-reviewing",
|
|
36
61
|
"description": "Structured code reviews enforcing DRY, KISS, YAGNI, SRP, best practices, and project conventions. Triggers: review this code, code review, check my code, what do you think of this implementation, review this PR, is this code good, feedback on my code, review staged changes before commit.",
|
|
37
62
|
"path": "code-reviewing/SKILL.md"
|
|
38
63
|
},
|
|
64
|
+
{
|
|
65
|
+
"name": "compliance-privacy",
|
|
66
|
+
"description": "Engineer for regulatory and privacy obligations — GDPR/CCPA privacy-by-design, PII data mapping and minimization, retention and deletion (right to erasure), data subject requests, consent, SOC 2 controls (access, change management, audit logging). Triggers: GDPR, CCPA, SOC 2, HIPAA, compliance, privacy review, PII, personal data, data retention, right to be forgotten, DSR, audit requirements, are we compliant. Vulnerabilities in code → security-audit; attack analysis of a design → threat-modeling.",
|
|
67
|
+
"path": "compliance-privacy/SKILL.md"
|
|
68
|
+
},
|
|
39
69
|
{
|
|
40
70
|
"name": "configuration-strategy",
|
|
41
71
|
"description": "Design environment configuration, secrets management, and feature-flag hierarchy for a service or feature. Triggers: config strategy, environment variables, .env, secrets management, feature flag, config hierarchy, config precedence, twelve-factor config, environment-specific settings.",
|
|
@@ -51,6 +81,16 @@
|
|
|
51
81
|
"description": "Design database schemas, relationships, indexes, and migration strategies — relational and document stores. Triggers: data model, schema design, ER diagram, database schema, table design, foreign key, index strategy, normalization, denormalization, migration plan, document model, partition key.",
|
|
52
82
|
"path": "data-modeling/SKILL.md"
|
|
53
83
|
},
|
|
84
|
+
{
|
|
85
|
+
"name": "data-pipeline-design",
|
|
86
|
+
"description": "Design batch and streaming data pipelines — ELT into a warehouse/lakehouse, dbt and analytics engineering (staging/intermediate/marts layers), orchestration with Airflow or Dagster, idempotent loads, incremental models, backfills, CDC ingestion. Triggers: data pipeline, ELT, ETL, dbt models, data warehouse, Snowflake, BigQuery, ingest into the warehouse, Airflow DAG, Dagster, Kafka streaming, backfill, incremental load, nightly job duplicates rows. ML training/feature pipelines → ml-pipeline-design; validating the data → data-quality.",
|
|
87
|
+
"path": "data-pipeline-design/SKILL.md"
|
|
88
|
+
},
|
|
89
|
+
{
|
|
90
|
+
"name": "data-quality",
|
|
91
|
+
"description": "Keep warehouse and pipeline data trustworthy — dbt tests, expectations (Great Expectations/Soda), data contracts with upstream producers, source freshness, schema-drift and volume anomaly detection, lineage and blast radius, severity and alerting. Triggers: data quality, bad data in dashboards, data validation, dbt tests, data contract, schema drift, stale data, freshness check, duplicate rows, nulls in the warehouse, data lineage, data SLA, upstream schema change broke us. Generating test/synthetic data → test-data-strategy; schema design → data-modeling.",
|
|
92
|
+
"path": "data-quality/SKILL.md"
|
|
93
|
+
},
|
|
54
94
|
{
|
|
55
95
|
"name": "dependency-impact-analysis",
|
|
56
96
|
"description": "Map the blast radius of a change before implementing — find dependents, classify breaking vs additive, identify coordination needs, produce a propagation plan. Triggers: what will break if I change this, blast radius, impact analysis, breaking change, who depends on this, downstream impact, safe to rename, shared schema change, dependency graph.",
|
|
@@ -71,6 +111,11 @@
|
|
|
71
111
|
"description": "Design a deployment (GitOps) repo that orchestrates multiple services in a polyrepo — version pinning, environment promotion, compatibility matrices, system config. Triggers: deployment repo, GitOps repo, polyrepo, multi-repo, version pinning, compatibility matrix, environment promotion, multi-service deployment, orchestration repo, which versions work together.",
|
|
72
112
|
"path": "deployment-repo/SKILL.md"
|
|
73
113
|
},
|
|
114
|
+
{
|
|
115
|
+
"name": "dx-audit",
|
|
116
|
+
"description": "Audit and improve developer experience — inner-loop feedback (build/test/reload times), CI wait and queue times, flaky tests, local environment setup, onboarding time-to-first-PR, tooling friction, docs discoverability; produce a prioritized remediation plan. Triggers: developer experience, DX audit, devex, builds are slow, CI takes forever, flaky tests, onboarding takes weeks, tooling friction, developer productivity, engineers are frustrated. Code health/debt hotspots → technical-debt-review; designing one pipeline → cicd-pipeline.",
|
|
117
|
+
"path": "dx-audit/SKILL.md"
|
|
118
|
+
},
|
|
74
119
|
{
|
|
75
120
|
"name": "effort-estimation",
|
|
76
121
|
"description": "Estimate engineering effort with agile techniques — story points, t-shirt sizing, three-point estimation, capacity planning. Triggers: estimate this, how long will this take, story points, t-shirt sizing, effort estimation, capacity planning, sprint planning, budget estimate, forecast, velocity, when will this be done.",
|
|
@@ -78,9 +123,14 @@
|
|
|
78
123
|
},
|
|
79
124
|
{
|
|
80
125
|
"name": "feature-planning",
|
|
81
|
-
"description": "Break features into well-scoped tasks with acceptance criteria, risk assessment, and dependency mapping. Triggers: plan this, break this down, scope this feature, create tasks for, sprint planning, how should I implement this feature, user stories, acceptance criteria, dependency mapping, feature breakdown.",
|
|
126
|
+
"description": "Break features into well-scoped tasks with acceptance criteria, risk assessment, and dependency mapping. Triggers: plan this, break this down, scope this feature, create tasks for, sprint planning, how should I implement this feature, user stories, acceptance criteria, dependency mapping, feature breakdown. Creates the plan — executing an already-approved plan checkpoint by checkpoint → plan-execution.",
|
|
82
127
|
"path": "feature-planning/SKILL.md"
|
|
83
128
|
},
|
|
129
|
+
{
|
|
130
|
+
"name": "finops-cost-optimization",
|
|
131
|
+
"description": "Understand and reduce cloud spend — cost visibility and allocation (tagging, ownership), unit economics, rightsizing, autoscaling, commitment discounts (reserved instances, savings plans), storage/egress traps, architecture cost review, budgets and anomaly alerts. Triggers: cloud costs, AWS bill, cut costs, cost optimization, FinOps, why is our bill so high, rightsizing, reserved instances, savings plan, egress fees, cost per customer. Whether to build or buy a capability → build-vs-buy; making code faster → performance-optimization.",
|
|
132
|
+
"path": "finops-cost-optimization/SKILL.md"
|
|
133
|
+
},
|
|
84
134
|
{
|
|
85
135
|
"name": "frontend-architecture",
|
|
86
136
|
"description": "Design frontend architecture — React component hierarchy, state management, design tokens, data fetching, routing, error boundaries, code organization. Triggers: component architecture, state management, design system, design tokens, component library, React structure, folder structure, Zustand vs Redux, React Query, code splitting, error boundary, compound component.",
|
|
@@ -106,6 +156,11 @@
|
|
|
106
156
|
"description": "Write and review IaC with Terraform, CloudFormation, Pulumi, or CDK — modularity, security, state management. Triggers: Terraform, CloudFormation, Pulumi, infrastructure as code, IaC, provision, cloud resources, terraform module, state management, terraform plan, CDK, drift, remote state.",
|
|
107
157
|
"path": "infrastructure-as-code/SKILL.md"
|
|
108
158
|
},
|
|
159
|
+
{
|
|
160
|
+
"name": "llm-app-engineering",
|
|
161
|
+
"description": "Design and build LLM-powered applications — prompt and context engineering, RAG architecture (chunking, hybrid retrieval, reranking, grounded prompts), agent design (tool surfaces, planning, sub-agents), and memory. Triggers: build a chatbot, LLM app, AI assistant, prompt engineering, system prompt, RAG design, chunking, embeddings, vector database, semantic search, AI agent, tool calling, agent memory, context window, hallucinations. Measuring quality → ai-evaluation; model serving/inference infra → ml-model-deployment.",
|
|
162
|
+
"path": "llm-app-engineering/SKILL.md"
|
|
163
|
+
},
|
|
109
164
|
{
|
|
110
165
|
"name": "metrics-and-okrs",
|
|
111
166
|
"description": "Define OKRs, KPIs, success metrics, and engineering health metrics (DORA). Triggers: OKR, KPI, metrics, success metrics, key results, objectives, measure success, DORA metrics, engineering metrics, velocity, cycle time, deployment frequency, how do we measure, what should we track.",
|
|
@@ -113,7 +168,7 @@
|
|
|
113
168
|
},
|
|
114
169
|
{
|
|
115
170
|
"name": "ml-experiment-tracking",
|
|
116
|
-
"description": "Design reproducible ML experiments — tracking, versioning, run comparison with MLflow, W&B, or DVC. Triggers: experiment tracking, MLflow, wandb, weights and biases, DVC, track experiments, compare models, hyperparameter, reproducibility, model registry, which model is better, experiment results, log metrics.",
|
|
171
|
+
"description": "Design reproducible ML experiments — tracking, versioning, run comparison with MLflow, W&B, or DVC. Triggers: experiment tracking, MLflow, wandb, weights and biases, DVC, track experiments, compare models, hyperparameter, reproducibility, model registry, which model is better, experiment results, log metrics. Defining WHAT to measure (metrics, golden datasets, judges) → ai-evaluation; this skill records and compares the runs.",
|
|
117
172
|
"path": "ml-experiment-tracking/SKILL.md"
|
|
118
173
|
},
|
|
119
174
|
{
|
|
@@ -123,9 +178,19 @@
|
|
|
123
178
|
},
|
|
124
179
|
{
|
|
125
180
|
"name": "ml-pipeline-design",
|
|
126
|
-
"description": "Design reproducible ML training and data pipelines — ingestion, validation, feature engineering, training, evaluation, continuous training orchestration. Triggers: training pipeline, data pipeline, feature engineering, ETL for ML, continuous training, data validation, feature store, preprocessing, notebook to pipeline, orchestrate training,
|
|
181
|
+
"description": "Design reproducible ML training and data pipelines — ingestion, validation, feature engineering, training, evaluation, continuous training orchestration. Triggers: training pipeline, ML data pipeline, feature engineering, ETL for ML, continuous training, data validation, feature store, preprocessing, notebook to pipeline, orchestrate training, Kubeflow, pipeline DAG, point-in-time features. Analytics/BI ELT and dbt warehouse pipelines → data-pipeline-design.",
|
|
127
182
|
"path": "ml-pipeline-design/SKILL.md"
|
|
128
183
|
},
|
|
184
|
+
{
|
|
185
|
+
"name": "mobile-architecture",
|
|
186
|
+
"description": "Design mobile app architecture — native (Swift/Kotlin) vs cross-platform (React Native/Flutter) selection, navigation and deep linking, state management, offline-first design and data sync, local persistence, push notifications, platform constraints (background work, battery, permissions). Triggers: mobile app architecture, iOS app, Android app, React Native or Flutter, offline first, data sync, mobile state management, deep linking, push notifications, app structure. Web SPA component architecture → frontend-architecture.",
|
|
187
|
+
"path": "mobile-architecture/SKILL.md"
|
|
188
|
+
},
|
|
189
|
+
{
|
|
190
|
+
"name": "mobile-release",
|
|
191
|
+
"description": "Ship mobile apps through the app stores — signing and provisioning, store review and rejection handling, phased/staged rollouts with halt criteria, crash monitoring, versioning and build numbers, beta channels (TestFlight, Play tracks), forced updates, and the no-instant-rollback reality of mobile. Triggers: app store release, publish to App Store, Play Store, TestFlight, staged rollout, app review rejected, code signing, provisioning profile, mobile release, hotfix a mobile bug. General semver/registry/library publishing → release-management.",
|
|
192
|
+
"path": "mobile-release/SKILL.md"
|
|
193
|
+
},
|
|
129
194
|
{
|
|
130
195
|
"name": "observability-design",
|
|
131
196
|
"description": "Design production observability — SLIs, SLOs, SLAs, error budgets, OpenTelemetry traces/metrics/logs, structured logging, alerting, dashboards. Triggers: SLO, SLI, SLA, error budget, observability, monitoring, OpenTelemetry, OTel, tracing, distributed tracing, structured logging, alerting, dashboard, metrics, correlation ID, alert fatigue.",
|
|
@@ -136,6 +201,11 @@
|
|
|
136
201
|
"description": "Identify and resolve performance bottlenecks via static analysis — N+1 queries, algorithmic complexity, query optimization, caching, memory leaks, bundle size, connection management. Triggers: this is slow, optimize, performance, N+1, query optimization, caching, bundle size, memory leak, latency, response time, scale, bottleneck.",
|
|
137
202
|
"path": "performance-optimization/SKILL.md"
|
|
138
203
|
},
|
|
204
|
+
{
|
|
205
|
+
"name": "plan-execution",
|
|
206
|
+
"description": "Execute an already-approved plan with discipline — batch tasks into verifiable checkpoints, verify each with fresh evidence before marking it done, log drift, and stop to re-plan when reality diverges from the plan's assumptions. Triggers: execute the plan, implement the approved plan, work through this plan, continue/resume the plan, next phase, checkpoint. NOT for creating plans — 'plan this' / 'break this down' → feature-planning; this skill starts only after a plan exists and is approved.",
|
|
207
|
+
"path": "plan-execution/SKILL.md"
|
|
208
|
+
},
|
|
139
209
|
{
|
|
140
210
|
"name": "prd-writing",
|
|
141
211
|
"description": "Write lightweight agile PRDs and technical RFCs that align stakeholders before implementation. Triggers: PRD, product requirements, RFC, request for comments, technical design doc, design document, tech spec, write requirements, what are we building, spec this out. Produces the WHAT and WHY — feature-planning handles HOW.",
|
|
@@ -161,6 +231,16 @@
|
|
|
161
231
|
"description": "Systematic code improvement via design principles, smell detection, and safe transformation patterns. Triggers: refactor this, clean this up, simplify this code, reduce complexity, extract this, this code is messy, tech debt, improve this code, code smell, rename, extract function, dead code.",
|
|
162
232
|
"path": "refactoring/SKILL.md"
|
|
163
233
|
},
|
|
234
|
+
{
|
|
235
|
+
"name": "release-management",
|
|
236
|
+
"description": "Cut and publish software releases right-sized to the project — semver decisions, changelogs, tagging, automated version bumps (release-please, changesets, semantic-release), publishing gates, npm/PyPI/container registry publishing, pre-release channels, monorepo versioning. Triggers: cut a release, release this, publish to npm, publish to PyPI, publish this package, version bump, what version should this be, semver, changelog, tag a release, release notes, GitHub release, prerelease, release automation, release PR. Deploy-time safety → deployment-checklist; commit/PR hygiene → git-workflow.",
|
|
237
|
+
"path": "release-management/SKILL.md"
|
|
238
|
+
},
|
|
239
|
+
{
|
|
240
|
+
"name": "resilience-engineering",
|
|
241
|
+
"description": "Design systems that survive failure — failure-mode analysis, resilience patterns (timeouts, retries with backoff, circuit breakers, bulkheads, load shedding, graceful degradation), chaos experiments, disaster recovery with RTO/RPO, backup restore testing, game days. Triggers: chaos engineering, disaster recovery, DR plan, what if the database goes down, single point of failure, failover, circuit breaker, RTO, RPO, game day, high availability. An incident happening NOW → incident-response; undoing a bad deploy → rollback-strategy.",
|
|
242
|
+
"path": "resilience-engineering/SKILL.md"
|
|
243
|
+
},
|
|
164
244
|
{
|
|
165
245
|
"name": "retrospective",
|
|
166
246
|
"description": "Facilitate sprint retros, project post-mortems, and incident post-mortems with actionable, owned improvements. Triggers: retrospective, retro, post-mortem, postmortem, lessons learned, what went wrong, what went well, incident review, blameless, action items, sprint review, after action review.",
|
|
@@ -173,7 +253,7 @@
|
|
|
173
253
|
},
|
|
174
254
|
{
|
|
175
255
|
"name": "security-audit",
|
|
176
|
-
"description": "Comprehensive security analysis — OWASP Top 10, auth/authz flows, injection vulnerabilities, data exposure, secrets detection, dependency CVEs, hardening recommendations. Triggers: security audit, vulnerability, is this secure, security review, pentest prep, OWASP, harden this, check for vulnerabilities, injection, XSS, CSRF, auth security.",
|
|
256
|
+
"description": "Comprehensive security analysis — OWASP Top 10, auth/authz flows, injection vulnerabilities, data exposure, secrets detection, dependency CVEs, hardening recommendations. Triggers: security audit, vulnerability, is this secure, security review, pentest prep, OWASP, harden this, check for vulnerabilities, injection, XSS, CSRF, auth security. Reviews EXISTING code/config — design-time analysis of a system not yet built → threat-modeling.",
|
|
177
257
|
"path": "security-audit/SKILL.md"
|
|
178
258
|
},
|
|
179
259
|
{
|
|
@@ -186,6 +266,11 @@
|
|
|
186
266
|
"description": "Review a project's strategic position before going public, launching, or raising — vision, mission, value proposition, scope positioning, the defensible wedge, and a live competitive / market comparative analysis. Triggers: strategic review, positioning, go public, go-to-market, market analysis, competitive landscape, value proposition, is there a moat, who are our competitors, platform absorption risk, market positioning, comparable products. Use project-review for execution/roadmap/implementation health; delegates deep market scans to deep-research.",
|
|
187
267
|
"path": "strategic-review/SKILL.md"
|
|
188
268
|
},
|
|
269
|
+
{
|
|
270
|
+
"name": "subagent-orchestration",
|
|
271
|
+
"description": "Decompose big or parallelizable work across subagents — when to fan out vs stay solo, scoping self-contained sub-tasks, prompting agents that don't share your context, isolating parallel edits (git worktrees), skeptically verifying and synthesizing results you didn't produce. Triggers: use subagents, parallelize this, fan out, orchestrate agents, multi-agent, spawn agents, worktree, delegate to agents, split this across agents, agent team. Sequencing checkpoints of an approved plan → plan-execution (orchestration parallelizes within its batches).",
|
|
272
|
+
"path": "subagent-orchestration/SKILL.md"
|
|
273
|
+
},
|
|
189
274
|
{
|
|
190
275
|
"name": "tdd-workflow",
|
|
191
276
|
"description": "Guide test-driven development with red-green-refactor. Triggers: use tdd, test first, write the tests, red green refactor, test-driven, what should I test, write tests before code, failing test, implement using TDD. For NEW code via test-first — use test-suite-design for adding tests to existing code.",
|
|
@@ -206,6 +291,11 @@
|
|
|
206
291
|
"description": "Design comprehensive test suites for existing code — strategy across unit/integration/e2e, fixtures, factories, helpers. Triggers: add tests, write tests for, increase coverage, test this module, testing strategy, test plan, what tests do I need, test infrastructure, test helpers, this has no tests. Not for TDD — use tdd-workflow.",
|
|
207
292
|
"path": "test-suite-design/SKILL.md"
|
|
208
293
|
},
|
|
294
|
+
{
|
|
295
|
+
"name": "threat-modeling",
|
|
296
|
+
"description": "Design-time security analysis of a system or feature before it's built — trust boundaries, attack surface, STRIDE per element, abuse cases, prioritized mitigations that become security requirements. Triggers: threat model, STRIDE, trust boundary, attack surface, what could go wrong security-wise, security design review, abuse case, secure this design, security requirements. Reviewing EXISTING code/config for vulnerabilities → security-audit; use this skill while the design is still on the whiteboard.",
|
|
297
|
+
"path": "threat-modeling/SKILL.md"
|
|
298
|
+
},
|
|
209
299
|
{
|
|
210
300
|
"name": "ui-ux-design",
|
|
211
301
|
"description": "Design user experiences from an engineer's perspective — user flows, wireframes, interaction patterns, responsive strategy, navigation, loading/error/empty states. Triggers: design the UI, user flow, wireframe, how should this screen look, loading state, error state, empty state, responsive design, navigation, page layout, form design, mockup.",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "swe-workflow-skills",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "A curated library of 44 Claude Code Agent Skills for the software lifecycle, with orchestrator-routed activation that scales past the skill-listing budget.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
package/roles.json
CHANGED
|
@@ -14,11 +14,13 @@
|
|
|
14
14
|
"core": {
|
|
15
15
|
"universal": [
|
|
16
16
|
"skill-router",
|
|
17
|
+
"brainstorming",
|
|
17
18
|
"feature-planning"
|
|
18
19
|
],
|
|
19
20
|
"technical": [
|
|
20
21
|
"skill-router",
|
|
21
22
|
"feature-planning",
|
|
23
|
+
"plan-execution",
|
|
22
24
|
"git-workflow",
|
|
23
25
|
"code-reviewing",
|
|
24
26
|
"verification-before-completion",
|
|
@@ -28,7 +30,8 @@
|
|
|
28
30
|
},
|
|
29
31
|
|
|
30
32
|
"meta_only": [
|
|
31
|
-
"writing-skills"
|
|
33
|
+
"writing-skills",
|
|
34
|
+
"subagent-orchestration"
|
|
32
35
|
],
|
|
33
36
|
|
|
34
37
|
"roles": {
|
|
@@ -62,6 +65,7 @@
|
|
|
62
65
|
"api-design",
|
|
63
66
|
"tdd-workflow",
|
|
64
67
|
"test-suite-design",
|
|
68
|
+
"browser-verification",
|
|
65
69
|
"performance-optimization",
|
|
66
70
|
"refactoring"
|
|
67
71
|
]
|
|
@@ -73,6 +77,7 @@
|
|
|
73
77
|
"skills": [
|
|
74
78
|
"containerization",
|
|
75
79
|
"cicd-pipeline",
|
|
80
|
+
"release-management",
|
|
76
81
|
"infrastructure-as-code",
|
|
77
82
|
"deployment-repo",
|
|
78
83
|
"gitops-delivery",
|
|
@@ -80,18 +85,43 @@
|
|
|
80
85
|
"rollback-strategy",
|
|
81
86
|
"incident-response",
|
|
82
87
|
"observability-design",
|
|
83
|
-
"
|
|
84
|
-
"
|
|
88
|
+
"resilience-engineering",
|
|
89
|
+
"configuration-strategy"
|
|
85
90
|
]
|
|
86
91
|
},
|
|
87
92
|
"ml": {
|
|
88
93
|
"label": "ML Engineer / MLOps",
|
|
89
|
-
"description": "Skills for ML pipelines, experiment tracking, model deployment, and the data/observability around them.",
|
|
94
|
+
"description": "Skills for ML pipelines, experiment tracking, evaluation, model deployment, and the data/observability around them.",
|
|
90
95
|
"core": "technical",
|
|
91
96
|
"skills": [
|
|
92
97
|
"ml-pipeline-design",
|
|
93
98
|
"ml-experiment-tracking",
|
|
94
99
|
"ml-model-deployment",
|
|
100
|
+
"ai-evaluation",
|
|
101
|
+
"data-modeling",
|
|
102
|
+
"test-data-strategy",
|
|
103
|
+
"observability-design"
|
|
104
|
+
]
|
|
105
|
+
},
|
|
106
|
+
"ai": {
|
|
107
|
+
"label": "AI Engineer",
|
|
108
|
+
"description": "Skills for LLM-powered applications: prompt/RAG/agent design, evaluation, experiment tracking, model deployment, and the API around them.",
|
|
109
|
+
"core": "technical",
|
|
110
|
+
"skills": [
|
|
111
|
+
"llm-app-engineering",
|
|
112
|
+
"ai-evaluation",
|
|
113
|
+
"ml-experiment-tracking",
|
|
114
|
+
"ml-model-deployment",
|
|
115
|
+
"api-design"
|
|
116
|
+
]
|
|
117
|
+
},
|
|
118
|
+
"data": {
|
|
119
|
+
"label": "Data Engineer",
|
|
120
|
+
"description": "Skills for data pipelines, analytics engineering, data quality/contracts, schema design, test data, and pipeline observability.",
|
|
121
|
+
"core": "technical",
|
|
122
|
+
"skills": [
|
|
123
|
+
"data-pipeline-design",
|
|
124
|
+
"data-quality",
|
|
95
125
|
"data-modeling",
|
|
96
126
|
"test-data-strategy",
|
|
97
127
|
"observability-design"
|
|
@@ -99,10 +129,12 @@
|
|
|
99
129
|
},
|
|
100
130
|
"security": {
|
|
101
131
|
"label": "Security Engineer",
|
|
102
|
-
"description": "Skills for security audits, dependency/CVE management, blast-radius analysis, and incident handling.",
|
|
132
|
+
"description": "Skills for design-time threat modeling, security audits, dependency/CVE management, blast-radius analysis, and incident handling.",
|
|
103
133
|
"core": "technical",
|
|
104
134
|
"skills": [
|
|
105
135
|
"security-audit",
|
|
136
|
+
"threat-modeling",
|
|
137
|
+
"compliance-privacy",
|
|
106
138
|
"dependency-management",
|
|
107
139
|
"dependency-impact-analysis",
|
|
108
140
|
"incident-response",
|
|
@@ -119,6 +151,7 @@
|
|
|
119
151
|
"architecture-documentation",
|
|
120
152
|
"api-design",
|
|
121
153
|
"data-modeling",
|
|
154
|
+
"code-archaeology",
|
|
122
155
|
"dependency-impact-analysis",
|
|
123
156
|
"technical-debt-review",
|
|
124
157
|
"project-review",
|
|
@@ -140,7 +173,9 @@
|
|
|
140
173
|
"project-review",
|
|
141
174
|
"strategic-review",
|
|
142
175
|
"code-reviewing",
|
|
143
|
-
"incident-response"
|
|
176
|
+
"incident-response",
|
|
177
|
+
"release-management",
|
|
178
|
+
"dx-audit"
|
|
144
179
|
]
|
|
145
180
|
},
|
|
146
181
|
"pm": {
|
|
@@ -158,12 +193,14 @@
|
|
|
158
193
|
},
|
|
159
194
|
"strategy": {
|
|
160
195
|
"label": "Strategy / Founder Review",
|
|
161
|
-
"description": "Skills for pre-public and strategic reviews: vision and positioning, market analysis, execution-vs-roadmap, and go/no-go synthesis.",
|
|
196
|
+
"description": "Skills for pre-public and strategic reviews: vision and positioning, market analysis, build-vs-buy decisions, execution-vs-roadmap, and go/no-go synthesis.",
|
|
162
197
|
"core": "universal",
|
|
163
198
|
"skills": [
|
|
164
199
|
"strategic-review",
|
|
165
200
|
"project-review",
|
|
166
201
|
"project-proposal",
|
|
202
|
+
"build-vs-buy",
|
|
203
|
+
"finops-cost-optimization",
|
|
167
204
|
"metrics-and-okrs"
|
|
168
205
|
]
|
|
169
206
|
},
|
|
@@ -175,10 +212,24 @@
|
|
|
175
212
|
"tdd-workflow",
|
|
176
213
|
"test-suite-design",
|
|
177
214
|
"test-data-strategy",
|
|
215
|
+
"browser-verification",
|
|
178
216
|
"code-reviewing",
|
|
179
217
|
"bug-investigating"
|
|
180
218
|
]
|
|
181
219
|
},
|
|
220
|
+
"mobile": {
|
|
221
|
+
"label": "Mobile Engineer",
|
|
222
|
+
"description": "Skills for native and cross-platform mobile apps: architecture, offline/sync, UI/UX, accessibility, performance, API contracts, and app-store releases.",
|
|
223
|
+
"core": "technical",
|
|
224
|
+
"skills": [
|
|
225
|
+
"mobile-architecture",
|
|
226
|
+
"mobile-release",
|
|
227
|
+
"ui-ux-design",
|
|
228
|
+
"accessibility-design",
|
|
229
|
+
"performance-optimization",
|
|
230
|
+
"api-design"
|
|
231
|
+
]
|
|
232
|
+
},
|
|
182
233
|
"designer": {
|
|
183
234
|
"label": "Designer / UX",
|
|
184
235
|
"description": "Skills for user flows, accessibility, product requirements, and the frontend architecture that realizes them.",
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ai-evaluation
|
|
3
|
+
description: "Define how to evaluate ML models and GenAI/LLM apps — golden datasets, offline metrics, RAG evaluation (faithfulness, answer relevance, context precision/recall) with ragas/deepeval/promptfoo, LLM-as-judge design and calibration, eval regression gates in CI, online A/B and human feedback. Triggers: evaluate the model, eval my chatbot, RAG evals, ragas, LLM as judge, golden dataset, eval suite, is the new prompt better, hallucination rate, benchmark our AI app. Logging/comparing runs → ml-experiment-tracking; building the app → llm-app-engineering."
|
|
4
|
+
model: sonnet
|
|
5
|
+
allowed-tools: Read, Grep, Glob, Write, Edit, Bash
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# AI Evaluation
|
|
9
|
+
|
|
10
|
+
Define **what to measure and how** for ML models and GenAI apps. Without an
|
|
11
|
+
evaluation harness, every prompt tweak, retrieval change, or model swap ships on
|
|
12
|
+
vibes — and quality silently regresses. The boundary with its neighbors:
|
|
13
|
+
`ml-experiment-tracking` *records and compares* runs; this skill *defines the
|
|
14
|
+
metrics, datasets, and judges* those runs report. `llm-app-engineering` designs
|
|
15
|
+
the app; this skill proves whether a change to it helped.
|
|
16
|
+
|
|
17
|
+
## Workflow
|
|
18
|
+
|
|
19
|
+
### Step 1: Pin Down What "Good" Means
|
|
20
|
+
|
|
21
|
+
Before choosing tools, write down 3–5 quality criteria in the product's terms
|
|
22
|
+
(answers grounded in our docs, no fabricated order numbers, resolves the ticket).
|
|
23
|
+
Vague goals ("high quality answers") make every later step unmeasurable. Decide
|
|
24
|
+
the primary metric — the one that decides ship/no-ship — before seeing results,
|
|
25
|
+
to avoid cherry-picking.
|
|
26
|
+
|
|
27
|
+
### Step 2: Build a Golden Dataset
|
|
28
|
+
|
|
29
|
+
Evaluate against a fixed, versioned dataset — not ad-hoc spot checks:
|
|
30
|
+
|
|
31
|
+
- **Source**: real user questions (logs, support tickets) first; synthesize from
|
|
32
|
+
the corpus only to fill gaps, and have a human review synthetic cases.
|
|
33
|
+
- **Size**: 30–50 cases is enough to start; grow with every production failure
|
|
34
|
+
(each becomes a regression case — same principle as a bug's regression test).
|
|
35
|
+
- **Coverage**: include the unhappy paths — questions the corpus can't answer
|
|
36
|
+
(the correct behavior is refusal), ambiguous phrasing, adversarial inputs.
|
|
37
|
+
- **Version it** alongside the code; a metric is only comparable across runs on
|
|
38
|
+
the same dataset version.
|
|
39
|
+
|
|
40
|
+
With zero labeled data, this is still not optional — bootstrap: label a few dozen
|
|
41
|
+
cases by hand (an afternoon), use reference-free metrics (groundedness against
|
|
42
|
+
the source, rubric-based judging) while the labeled set grows.
|
|
43
|
+
|
|
44
|
+
### Step 3: Choose Metrics by System Type
|
|
45
|
+
|
|
46
|
+
- **Classical ML models**: standard offline metrics vs held-out data (accuracy,
|
|
47
|
+
F1, RMSE, AUC), sliced by segment — aggregate numbers hide subgroup failures.
|
|
48
|
+
- **RAG systems** — score the stages separately, or you can't tell whether a bad
|
|
49
|
+
answer is a retrieval or a generation failure:
|
|
50
|
+
- *Retrieval*: context precision / context recall (did the right chunks come back?)
|
|
51
|
+
- *Generation*: faithfulness/groundedness (is every claim supported by the
|
|
52
|
+
retrieved context?) and answer relevance (does it address the question?)
|
|
53
|
+
- **Agents**: end-to-end task success rate plus trajectory checks (right tool,
|
|
54
|
+
no loops, steps within budget).
|
|
55
|
+
- **Reference-free** (no ground truth): groundedness against the source,
|
|
56
|
+
rubric-based judge scores, and pairwise A-vs-B comparison between variants.
|
|
57
|
+
|
|
58
|
+
Pick **one** eval tool and wire it in — ragas (RAG metrics), deepeval
|
|
59
|
+
(pytest-style, broad metric set), or promptfoo (config-driven prompt/model
|
|
60
|
+
comparison). Selection detail: [references/eval-tooling.md](references/eval-tooling.md).
|
|
61
|
+
|
|
62
|
+
### Step 4: Design the Judge Deliberately
|
|
63
|
+
|
|
64
|
+
LLM-as-judge is the workhorse for GenAI evals, and an uncalibrated judge is a
|
|
65
|
+
random-number generator with authority. Treat the judge as a component you
|
|
66
|
+
design and validate:
|
|
67
|
+
|
|
68
|
+
- **Rubric, not vibes**: decompose "good" into the Step-1 criteria, each scored
|
|
69
|
+
separately with a binary or 3-point scale — never a bare 1–10 overall score.
|
|
70
|
+
- **Known biases, known counters**: position bias (swap A/B order and re-judge),
|
|
71
|
+
self-preference (judge with a different model family than the one being
|
|
72
|
+
judged), verbosity bias (instruct the judge to ignore length; cap output length).
|
|
73
|
+
- **Reduce noise**: majority-of-k voting for important comparisons.
|
|
74
|
+
- **Calibrate**: score 20–30 cases with both the judge and a human; if agreement
|
|
75
|
+
is poor, fix the rubric before trusting any judge-derived number.
|
|
76
|
+
|
|
77
|
+
### Step 5: Gate Changes in CI
|
|
78
|
+
|
|
79
|
+
An eval suite that runs once before launch rots immediately. Run it on every
|
|
80
|
+
change to prompts, retrieval, or model version — like a test suite:
|
|
81
|
+
|
|
82
|
+
- Gate on **regression vs the last accepted baseline** (metric dropped), not an
|
|
83
|
+
absolute threshold — absolute judge scores drift; movement is the signal.
|
|
84
|
+
- Keep the CI subset small and fast (the golden set, k=1); run the full suite
|
|
85
|
+
nightly or pre-release with voting.
|
|
86
|
+
- Log eval runs in the experiment tracker (`ml-experiment-tracking`) so results
|
|
87
|
+
are comparable over time.
|
|
88
|
+
|
|
89
|
+
### Step 6: Close the Loop Online
|
|
90
|
+
|
|
91
|
+
Offline evals predict; production confirms. Once users exist: lightweight
|
|
92
|
+
feedback (thumbs up/down + optional reason), A/B test significant changes on the
|
|
93
|
+
primary metric, and mine failures — every confirmed production failure becomes a
|
|
94
|
+
new golden-set case (Step 2).
|
|
95
|
+
|
|
96
|
+
## Principles Applied
|
|
97
|
+
|
|
98
|
+
- **YAGNI**: 30 golden cases + one tool + one judge rubric beats a
|
|
99
|
+
six-metric dashboard nobody trusts. Add metrics when a decision needs them.
|
|
100
|
+
- **KISS**: prefer binary per-criterion judgments over fine-grained scales —
|
|
101
|
+
humans and judges both agree more on yes/no.
|
|
102
|
+
- **Measure movement, not absolutes**: judge scores are comparable only within
|
|
103
|
+
the same dataset + rubric + judge version; gate on deltas.
|
|
104
|
+
|
|
105
|
+
## Cross-Skill References
|
|
106
|
+
|
|
107
|
+
- `llm-app-engineering` — designing the system this skill measures
|
|
108
|
+
- `ml-experiment-tracking` — recording and comparing the eval runs over time
|
|
109
|
+
- `ml-model-deployment` — online monitoring and drift detection in production
|
|
110
|
+
- `cicd-pipeline` — wiring the eval gate into CI
|
|
111
|
+
- `metrics-and-okrs` — tying eval metrics to product-level success metrics
|