swe-workflow-skills 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -17
- package/VERSION +1 -1
- package/catalog.json +95 -5
- package/package.json +1 -1
- package/roles.json +58 -7
- package/skills/ai-evaluation/SKILL.md +111 -0
- package/skills/ai-evaluation/evals/evals.json +40 -0
- package/skills/ai-evaluation/references/eval-tooling.md +63 -0
- package/skills/architecture-design/SKILL.md +3 -1
- package/skills/brainstorming/SKILL.md +92 -0
- package/skills/brainstorming/evals/evals.json +40 -0
- package/skills/browser-verification/SKILL.md +95 -0
- package/skills/browser-verification/evals/evals.json +40 -0
- package/skills/build-vs-buy/SKILL.md +109 -0
- package/skills/build-vs-buy/evals/evals.json +40 -0
- package/skills/cicd-pipeline/SKILL.md +1 -0
- package/skills/code-archaeology/SKILL.md +103 -0
- package/skills/code-archaeology/evals/evals.json +41 -0
- package/skills/compliance-privacy/SKILL.md +113 -0
- package/skills/compliance-privacy/evals/evals.json +41 -0
- package/skills/compliance-privacy/references/obligations-map.md +69 -0
- package/skills/data-pipeline-design/SKILL.md +116 -0
- package/skills/data-pipeline-design/evals/evals.json +41 -0
- package/skills/data-pipeline-design/references/orchestration.md +64 -0
- package/skills/data-quality/SKILL.md +107 -0
- package/skills/data-quality/evals/evals.json +39 -0
- package/skills/data-quality/references/tooling.md +45 -0
- package/skills/deployment-checklist/SKILL.md +2 -1
- package/skills/dx-audit/SKILL.md +97 -0
- package/skills/dx-audit/evals/evals.json +40 -0
- package/skills/feature-planning/SKILL.md +5 -1
- package/skills/finops-cost-optimization/SKILL.md +109 -0
- package/skills/finops-cost-optimization/evals/evals.json +40 -0
- package/skills/git-workflow/SKILL.md +10 -0
- package/skills/llm-app-engineering/SKILL.md +102 -0
- package/skills/llm-app-engineering/evals/evals.json +40 -0
- package/skills/llm-app-engineering/references/patterns.md +70 -0
- package/skills/ml-experiment-tracking/SKILL.md +1 -1
- package/skills/ml-pipeline-design/SKILL.md +1 -1
- package/skills/mobile-architecture/SKILL.md +106 -0
- package/skills/mobile-architecture/evals/evals.json +40 -0
- package/skills/mobile-release/SKILL.md +109 -0
- package/skills/mobile-release/evals/evals.json +41 -0
- package/skills/plan-execution/SKILL.md +115 -0
- package/skills/plan-execution/evals/evals.json +55 -0
- package/skills/project-documentation/SKILL.md +4 -0
- package/skills/release-management/SKILL.md +97 -0
- package/skills/release-management/evals/evals.json +56 -0
- package/skills/release-management/references/release-tooling.md +106 -0
- package/skills/resilience-engineering/SKILL.md +121 -0
- package/skills/resilience-engineering/evals/evals.json +40 -0
- package/skills/security-audit/SKILL.md +2 -2
- package/skills/skill-router/SKILL.md +46 -3
- package/skills/subagent-orchestration/SKILL.md +100 -0
- package/skills/subagent-orchestration/evals/evals.json +40 -0
- package/skills/threat-modeling/SKILL.md +101 -0
- package/skills/threat-modeling/evals/evals.json +40 -0
- package/skills/threat-modeling/references/stride-mitigations.md +90 -0
- package/skills/writing-skills/SKILL.md +27 -11
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: data-quality
|
|
3
|
+
description: "Keep warehouse and pipeline data trustworthy — dbt tests, expectations (Great Expectations/Soda), data contracts with upstream producers, source freshness, schema-drift and volume anomaly detection, lineage and blast radius, severity and alerting. Triggers: data quality, bad data in dashboards, data validation, dbt tests, data contract, schema drift, stale data, freshness check, duplicate rows, nulls in the warehouse, data lineage, data SLA, upstream schema change broke us. Generating test/synthetic data → test-data-strategy; schema design → data-modeling."
|
|
4
|
+
model: sonnet
|
|
5
|
+
allowed-tools: Read, Grep, Glob, Write, Edit, Bash
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Data Quality
|
|
9
|
+
|
|
10
|
+
Make bad data **loud**: caught at the boundary where it enters, blocking what
|
|
11
|
+
must not ship, alerting someone who owns the fix — instead of being discovered
|
|
12
|
+
by a stakeholder in a dashboard. Boundaries: *generating* fake/test data is
|
|
13
|
+
`test-data-strategy`; *designing* the schema is `data-modeling`; this skill
|
|
14
|
+
keeps the real data flowing through pipelines trustworthy.
|
|
15
|
+
|
|
16
|
+
## Workflow
|
|
17
|
+
|
|
18
|
+
### Step 1: Start From Incidents, Not Inventory
|
|
19
|
+
|
|
20
|
+
Don't blanket every column with tests. List the recent data incidents (wrong
|
|
21
|
+
dashboard numbers, silent breakage) and the tables where an error costs real
|
|
22
|
+
money or trust — revenue marts, ML features, anything an exec reads. Each
|
|
23
|
+
known symptom maps to a check: nulls → `not_null`, duplicates → `unique`,
|
|
24
|
+
staleness → freshness, broken joins → `relationships`. Wins from day one, and
|
|
25
|
+
the suite grows the same way tests do after bugs — every new incident adds a
|
|
26
|
+
check that would have caught it.
|
|
27
|
+
|
|
28
|
+
### Step 2: Check at the Right Layer — Boundary First
|
|
29
|
+
|
|
30
|
+
Where a check runs matters as much as what it checks:
|
|
31
|
+
|
|
32
|
+
- **Ingestion boundary (sources)**: schema, freshness, volume, primary-key
|
|
33
|
+
integrity — catch bad inputs *before* transforms consume them. One failed
|
|
34
|
+
source check beats fifty downstream test failures at 6 a.m.
|
|
35
|
+
- **Transform layers (staging → marts)**: uniqueness/not-null on the grain of
|
|
36
|
+
each model, relationship tests on joins, accepted values on enums.
|
|
37
|
+
- **Consumer edge (marts/metrics)**: business-rule assertions — "revenue is
|
|
38
|
+
never negative", "row count within N% of the 7-day average".
|
|
39
|
+
|
|
40
|
+
In dbt this is: source `freshness` + source tests, generic tests on models
|
|
41
|
+
(`unique`, `not_null`, `relationships`, `accepted_values`), singular SQL tests
|
|
42
|
+
for business rules. Dedicated tools (Great Expectations, Soda) earn their keep
|
|
43
|
+
outside dbt-land or when non-engineers must own the rules:
|
|
44
|
+
[references/tooling.md](references/tooling.md).
|
|
45
|
+
|
|
46
|
+
### Step 3: Assign Severity — What Blocks vs What Warns
|
|
47
|
+
|
|
48
|
+
A suite where everything fails the build gets muted within a month; a suite
|
|
49
|
+
where nothing blocks is decoration. Decide per check:
|
|
50
|
+
|
|
51
|
+
- **error** — data is unusable or contaminating (duplicate keys in a revenue
|
|
52
|
+
mart, failed contract at ingestion): the pipeline stops, downstream models
|
|
53
|
+
don't build on poisoned input.
|
|
54
|
+
- **warn** — degraded but usable (slightly stale, volume dip within reason):
|
|
55
|
+
alert, keep flowing.
|
|
56
|
+
|
|
57
|
+
dbt: `severity:` config with `error_if`/`warn_if` thresholds. Revisit
|
|
58
|
+
severities after each incident and each false alarm.
|
|
59
|
+
|
|
60
|
+
### Step 4: Route Failures to an Owner
|
|
61
|
+
|
|
62
|
+
A failing check nobody sees is identical to no check. Every table (or domain)
|
|
63
|
+
has a named owner; alerts go where that owner works (Slack channel per domain,
|
|
64
|
+
pager only for `error`-severity on critical marts), and each alert says what
|
|
65
|
+
failed, on which table, and the blast radius (Step 5). Track
|
|
66
|
+
time-to-detection and time-to-fix for data incidents — that's the metric this
|
|
67
|
+
whole skill improves (`observability-design` for the SLO framing).
|
|
68
|
+
|
|
69
|
+
### Step 5: Use Lineage to Scope Blast Radius
|
|
70
|
+
|
|
71
|
+
When a check fails, the first question is "what downstream is now wrong?" —
|
|
72
|
+
lineage (dbt's DAG, `dbt ls --select my_model+`, or a catalog tool) answers
|
|
73
|
+
which models, dashboards, and consumers sit downstream of the failure. Use it
|
|
74
|
+
to: prioritize fixes by consumer impact, notify affected dashboard owners
|
|
75
|
+
proactively, and decide what to rebuild after the fix. If you can't answer
|
|
76
|
+
"what breaks if this table is wrong?", invest in lineage before more checks.
|
|
77
|
+
|
|
78
|
+
### Step 6: Contract the Upstream Boundary
|
|
79
|
+
|
|
80
|
+
Downstream tests detect what upstream producers break; **contracts prevent
|
|
81
|
+
it.** When an external team owns the source: agree schema + semantics + SLA
|
|
82
|
+
explicitly (a versioned artifact — dbt model contracts, JSON Schema/protobuf in
|
|
83
|
+
a registry), enforce at ingestion so violations fail loudly at entry, and shift
|
|
84
|
+
left — contract validation in the *producer's* CI, so a breaking rename fails
|
|
85
|
+
their build, not your 6 a.m. run. Breaking changes get a version bump and a
|
|
86
|
+
deprecation window, coordinated like an API change
|
|
87
|
+
(`dependency-impact-analysis` thinking, applied to data). Back it with
|
|
88
|
+
schema-drift detection and volume/distribution anomaly checks for what
|
|
89
|
+
enforcement can't see.
|
|
90
|
+
|
|
91
|
+
## Principles Applied
|
|
92
|
+
|
|
93
|
+
- **YAGNI**: checks earn their place by a failure mode you've seen or can't
|
|
94
|
+
afford — blanket-testing every column produces alert fatigue, the failure
|
|
95
|
+
mode that kills quality programs.
|
|
96
|
+
- **Fail loudly at the boundary**: the cost of bad data grows with every layer
|
|
97
|
+
it flows through undetected.
|
|
98
|
+
- **DRY**: one contract at the source replaces N teams' defensive checks
|
|
99
|
+
against the same breakage.
|
|
100
|
+
|
|
101
|
+
## Cross-Skill References
|
|
102
|
+
|
|
103
|
+
- `data-pipeline-design` — the pipeline these checks are embedded in
|
|
104
|
+
- `data-modeling` — constraints and grain the checks assert (unique keys exist by design)
|
|
105
|
+
- `test-data-strategy` — generating synthetic/fixture data for testing pipelines
|
|
106
|
+
- `observability-design` — alerting design, SLOs, avoiding alert fatigue
|
|
107
|
+
- `incident-response` — when bad data has already reached production consumers
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill_name": "data-quality",
|
|
3
|
+
"evals": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"prompt": "Stakeholders keep finding bad numbers in our dashboards — null customer IDs, duplicated orders, and data that's a day stale. We're on dbt + Snowflake. Set up data quality checks.",
|
|
7
|
+
"expected_output": "Maps each symptom to a concrete check (not_null/unique dbt tests, source freshness), places checks at the ingestion boundary first and critical marts second, assigns severity (error blocks, warn alerts), routes failures to an owner, and uses lineage to scope which downstream models and dashboards a failure affects",
|
|
8
|
+
"assertions": [
|
|
9
|
+
"Maps each reported symptom to a concrete check: not_null/unique dbt tests for the IDs and duplicates, and source freshness checks for the staleness",
|
|
10
|
+
"Places checks at the right layers: the source/ingestion boundary first, then critical marts — not only on the final dashboards",
|
|
11
|
+
"Assigns severity levels (error vs warn) and defines what blocks the pipeline versus what only alerts",
|
|
12
|
+
"Sets up alerting and ownership so failures reach someone who acts, instead of failing silently",
|
|
13
|
+
"Uses lineage to scope blast radius — which downstream models or dashboards a failing check affects"
|
|
14
|
+
]
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"id": 2,
|
|
18
|
+
"prompt": "An upstream team owns the events we ingest and keeps renaming columns and changing types without telling us. Our pipeline breaks silently every few weeks. What should we put in place?",
|
|
19
|
+
"expected_output": "Proposes a producer/consumer data contract (schema, types, semantics, SLA) enforced at the ingestion boundary so violations fail loudly at entry; shifts validation left into the producer's CI or publish path; adds versioning/deprecation windows for breaking changes; and backstops enforcement with schema-drift and volume/distribution anomaly detection",
|
|
20
|
+
"assertions": [
|
|
21
|
+
"Proposes an explicit data contract between producer and consumer (schema, types, semantics, SLAs), not just more downstream tests",
|
|
22
|
+
"Enforces the schema at the ingestion boundary so violations fail loudly at entry instead of propagating silently",
|
|
23
|
+
"Shifts checks left: contract validation in the producer's CI or at publish time",
|
|
24
|
+
"Covers change management for breaking changes: contract versioning or a deprecation window",
|
|
25
|
+
"Adds detection for what enforcement misses: schema-drift alerts or anomaly checks on volume/distributions"
|
|
26
|
+
]
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": 3,
|
|
30
|
+
"prompt": "We need realistic but fake customer data to seed staging so QA can test the checkout flow — GDPR means we can't copy production.",
|
|
31
|
+
"expected_output": "Recognizes this as generating synthetic/test data, not validating production data — hands off to test-data-strategy (factories, synthetic generation, anonymization) rather than proposing expectations, contracts, or freshness checks",
|
|
32
|
+
"assertions": [
|
|
33
|
+
"Recognizes this is about generating test/synthetic data, not validating production data quality",
|
|
34
|
+
"Refers to the test-data-strategy skill for factories, synthetic data generation, and GDPR-safe anonymization",
|
|
35
|
+
"Does not answer with data-quality machinery (expectations, contracts, freshness checks) as the solution"
|
|
36
|
+
]
|
|
37
|
+
}
|
|
38
|
+
]
|
|
39
|
+
}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Data Quality Tooling — Selection Notes
|
|
2
|
+
|
|
3
|
+
Tool capabilities shift quickly; verify against current docs. The placement
|
|
4
|
+
logic is stable.
|
|
5
|
+
|
|
6
|
+
## Picking the enforcement layer
|
|
7
|
+
|
|
8
|
+
| Tool | Shape | Choose when |
|
|
9
|
+
|---|---|---|
|
|
10
|
+
| **dbt tests** | YAML generic tests + singular SQL tests, run with the transforms | You're already on dbt. Zero new infrastructure; tests version with the models; severity/thresholds built in. The default. |
|
|
11
|
+
| **dbt model contracts** | Enforced column names/types on a model; breaks the *producer's* build | A dbt model is consumed across team boundaries — the contract fails the producing project's CI, which is the shift-left goal. |
|
|
12
|
+
| **Great Expectations** | Python expectation suites + data docs, runs anywhere | Checks must run outside the warehouse/dbt (files, in-flight DataFrames, non-dbt pipelines), or you need its rich expectation vocabulary and generated documentation. Heavier to operate. |
|
|
13
|
+
| **Soda (Core/Cloud)** | Declarative checks YAML (SodaCL), CLI + SaaS | Non-engineers own quality rules, or checks span multiple stores with one syntax; Cloud adds anomaly detection and incident routing. |
|
|
14
|
+
| **Elementary / observability platforms (Monte Carlo, Metaplane…)** | Monitor the warehouse from metadata: freshness, volume, schema drift, distribution anomalies | You want detection you didn't hand-write — anomaly monitoring across many tables — and (platforms) column-level lineage + incident management. Elementary is the dbt-native OSS entry point. |
|
|
15
|
+
|
|
16
|
+
Layering that works in practice: **dbt tests + source freshness** as the
|
|
17
|
+
foundation; **contracts** at team boundaries; **one anomaly/observability
|
|
18
|
+
layer** for the unknown-unknowns. Adding a second tool in the same layer buys
|
|
19
|
+
maintenance, not coverage.
|
|
20
|
+
|
|
21
|
+
## Contract mechanics by boundary type
|
|
22
|
+
|
|
23
|
+
- **Event streams**: schema registry (Avro/Protobuf/JSON Schema) with
|
|
24
|
+
compatibility mode set (backward at minimum); producers can't publish a
|
|
25
|
+
breaking schema. The strongest enforcement available — use it if events are
|
|
26
|
+
the interface.
|
|
27
|
+
- **Warehouse-to-warehouse (dbt ↔ dbt)**: dbt model contracts + versioned
|
|
28
|
+
models (`v2` with a deprecation window) on the shared models.
|
|
29
|
+
- **API/file drops from another org**: JSON Schema validated at ingestion;
|
|
30
|
+
reject-and-alert (dead-letter the bad batch) rather than best-effort
|
|
31
|
+
parsing.
|
|
32
|
+
- The social half matters as much: a named producer contact, a change-notice
|
|
33
|
+
channel, and the contract in a repo both teams can PR.
|
|
34
|
+
|
|
35
|
+
## Anomaly checks worth having before "ML-powered" anything
|
|
36
|
+
|
|
37
|
+
- Row count vs trailing 7/28-day window (volume drops are the most common
|
|
38
|
+
silent failure).
|
|
39
|
+
- Freshness vs declared SLA per source.
|
|
40
|
+
- Schema diff vs last run (added/removed/retyped columns) with an explicit
|
|
41
|
+
allowlist flow.
|
|
42
|
+
- Null-rate and distinct-count drift on business-critical columns.
|
|
43
|
+
|
|
44
|
+
These four catch the large majority of real incidents; statistical/ML anomaly
|
|
45
|
+
detection is a refinement, not a starting point.
|
|
@@ -69,7 +69,7 @@ Walk through each section. Check items by reading code, running commands, and ve
|
|
|
69
69
|
#### Documentation
|
|
70
70
|
- [ ] README updated if setup steps changed
|
|
71
71
|
- [ ] API documentation updated for new/changed endpoints
|
|
72
|
-
- [ ]
|
|
72
|
+
- [ ] Release cut correctly, if this deploy ships one — version, changelog, tag consistent (see `release-management`)
|
|
73
73
|
- [ ] Breaking changes are clearly communicated
|
|
74
74
|
|
|
75
75
|
### Step 3: Report
|
|
@@ -126,6 +126,7 @@ See [references/pre-deploy-gates.md](references/pre-deploy-gates.md) for environ
|
|
|
126
126
|
|
|
127
127
|
## Cross-Skill References
|
|
128
128
|
|
|
129
|
+
- `release-management` — cutting the versioned release (semver, changelog, tag, publish) that this checklist deploys
|
|
129
130
|
- `rollback-strategy` — design the rollback plan before deploying (required for High-risk changes)
|
|
130
131
|
- `configuration-strategy` — verify all environment variables and feature flags are configured correctly
|
|
131
132
|
- `incident-response` — use if the deployment causes a production incident
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: dx-audit
|
|
3
|
+
description: "Audit and improve developer experience — inner-loop feedback (build/test/reload times), CI wait and queue times, flaky tests, local environment setup, onboarding time-to-first-PR, tooling friction, docs discoverability; produce a prioritized remediation plan. Triggers: developer experience, DX audit, devex, builds are slow, CI takes forever, flaky tests, onboarding takes weeks, tooling friction, developer productivity, engineers are frustrated. Code health/debt hotspots → technical-debt-review; designing one pipeline → cicd-pipeline."
|
|
4
|
+
model: sonnet
|
|
5
|
+
allowed-tools: Read, Grep, Glob, Write, Edit, Bash
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# DX Audit
|
|
9
|
+
|
|
10
|
+
Find and fix the friction that taxes every engineer, every day. Developer
|
|
11
|
+
experience compounds: a 10-minute CI queue on a 20-person team burns weeks of
|
|
12
|
+
engineering time per month, and worse, it changes *behavior* — engineers batch
|
|
13
|
+
changes, skip test runs, and avoid touching areas with slow feedback.
|
|
14
|
+
Boundary: `technical-debt-review` audits the *code's* health; this skill audits
|
|
15
|
+
the *workflow around* the code — loops, tooling, environments, and onboarding.
|
|
16
|
+
|
|
17
|
+
## Workflow
|
|
18
|
+
|
|
19
|
+
### Step 1: Measure the Loops — Numbers Before Anecdotes
|
|
20
|
+
|
|
21
|
+
Quantify the feedback loops before collecting opinions, so the audit ranks by
|
|
22
|
+
data and the improvement is provable later:
|
|
23
|
+
|
|
24
|
+
- **Inner loop**: cold build, incremental build, single-test run, hot-reload —
|
|
25
|
+
time them yourself in the actual repo.
|
|
26
|
+
- **Outer loop**: CI wall-clock (queue + run, from real recent pipelines), PR
|
|
27
|
+
time-to-first-review and time-to-merge, deploy lead time (DORA overlap —
|
|
28
|
+
`metrics-and-okrs`).
|
|
29
|
+
- **Flake rate**: retries and "re-run job" clicks per week; a 5%-flaky suite
|
|
30
|
+
means engineers already ignore red.
|
|
31
|
+
- **Onboarding**: time from laptop to first merged PR for recent joiners.
|
|
32
|
+
|
|
33
|
+
### Step 2: Collect the Friction Log
|
|
34
|
+
|
|
35
|
+
Then ask the engineers — a lightweight survey or five 15-minute conversations:
|
|
36
|
+
"what wastes your time every day?", "what do you avoid doing because it's
|
|
37
|
+
painful?", "what would you fix first?". The second question matters most:
|
|
38
|
+
avoidance signals (never running the full suite locally, dreading a service's
|
|
39
|
+
local setup, batching deploys) reveal costs the timers can't see. Recurring
|
|
40
|
+
themes + Step-1 numbers = the finding list.
|
|
41
|
+
|
|
42
|
+
### Step 3: Audit the Inner Loop
|
|
43
|
+
|
|
44
|
+
The highest-frequency loop gets the deepest look: Is the dev environment
|
|
45
|
+
reproducible in one command (or is setup a wiki page of drift-prone steps)?
|
|
46
|
+
Can you run one test in seconds without the world? Does hot-reload actually
|
|
47
|
+
work? Is the IDE experience functional (types, go-to-def, lint-on-save)? Are
|
|
48
|
+
common tasks (reset DB, seed data, run service X) scripted or tribal
|
|
49
|
+
knowledge? Every manual step here multiplies by invocations-per-day.
|
|
50
|
+
|
|
51
|
+
### Step 4: Audit the Outer Loop
|
|
52
|
+
|
|
53
|
+
- **CI**: queue time vs run time (capacity vs speed problems differ), cache
|
|
54
|
+
hit rates, test parallelization/sharding, whether PRs run only affected
|
|
55
|
+
targets — delegate pipeline redesign to `cicd-pipeline`.
|
|
56
|
+
- **Flakes**: quarantine policy (a flaky test in the merge gate is a tax on
|
|
57
|
+
everyone), ownership for fixing, `test-suite-design` for structural causes.
|
|
58
|
+
- **Review latency**: are PRs waiting on people or on process (required
|
|
59
|
+
approvals, codeowner bottlenecks)?
|
|
60
|
+
- **Docs**: can an engineer answer "how do I X here?" without interrupting a
|
|
61
|
+
senior — README freshness, runbooks, `project-documentation`.
|
|
62
|
+
|
|
63
|
+
### Step 5: Prioritize by Tax, Not by Annoyance
|
|
64
|
+
|
|
65
|
+
Score each finding as **time-cost × frequency × people affected** (daily
|
|
66
|
+
10-second cuts beat monthly 10-minute ones), weighted by behavior distortion
|
|
67
|
+
(anything that makes engineers *skip verification* is top-tier regardless of
|
|
68
|
+
seconds). Produce the remediation list: quick wins first (cache config, test
|
|
69
|
+
sharding, a `make dev` script) to build credibility, structural items
|
|
70
|
+
(environment overhaul, CI capacity) as scheduled work with owners.
|
|
71
|
+
|
|
72
|
+
### Step 6: Prove It and Keep It
|
|
73
|
+
|
|
74
|
+
Re-measure the Step-1 numbers after the fixes — DX work is uniquely easy to
|
|
75
|
+
verify, so verify it. Then keep the loop honest: track 2–3 DX metrics
|
|
76
|
+
continuously (CI p50/p95, flake rate, onboarding time), set a budget
|
|
77
|
+
("CI > 10 min is a defect"), and re-run the friction survey a couple of times
|
|
78
|
+
a year. DX decays by default — every new service adds setup steps and CI
|
|
79
|
+
minutes unless something pushes back.
|
|
80
|
+
|
|
81
|
+
## Principles Applied
|
|
82
|
+
|
|
83
|
+
- **Frequency beats magnitude**: optimize the loop that runs 50×/day before
|
|
84
|
+
the one that runs weekly.
|
|
85
|
+
- **Avoidance is the loudest signal**: what engineers won't do reveals more
|
|
86
|
+
than what they complain about.
|
|
87
|
+
- **Measured before and after**: a DX improvement that can't show the numbers
|
|
88
|
+
moved is an opinion.
|
|
89
|
+
|
|
90
|
+
## Cross-Skill References
|
|
91
|
+
|
|
92
|
+
- `technical-debt-review` — the code-health complement to this workflow audit
|
|
93
|
+
- `cicd-pipeline` — redesigning the pipeline the audit flagged
|
|
94
|
+
- `test-suite-design` — structural fixes for slow or flaky suites
|
|
95
|
+
- `project-documentation` — the docs/onboarding gaps surfaced in Step 4
|
|
96
|
+
- `metrics-and-okrs` — DORA metrics and making DX goals measurable
|
|
97
|
+
- `retrospective` — mining recurring friction from team retros
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill_name": "dx-audit",
|
|
3
|
+
"evals": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"prompt": "Our 25-person engineering team feels slow. CI takes ~35 minutes, people complain about local setup, and new hires seem to take forever to become productive. Run a DX audit and tell us what to fix.",
|
|
7
|
+
"expected_output": "Measures the loops first (inner-loop timings, CI queue vs run time, flake rate, PR cycle time, onboarding time-to-first-merge), collects a friction log from engineers with attention to avoidance behavior, audits inner loop (one-command env, single-test speed) and outer loop (CI caching/sharding, flake quarantine, review latency, docs), prioritizes by time-cost × frequency × people with quick wins first, and commits to re-measuring the same numbers after fixes",
|
|
8
|
+
"assertions": [
|
|
9
|
+
"Quantifies the feedback loops with real measurements (build/test times, CI queue vs run, flake rate, PR cycle, onboarding time) before prescribing fixes",
|
|
10
|
+
"Collects friction from the engineers themselves (survey/interviews) and pays attention to avoidance behavior, not just complaints",
|
|
11
|
+
"Audits both the inner loop (env setup, single-test runs, hot reload, scripted common tasks) and the outer loop (CI caching/parallelization, flaky-test policy, review latency, docs)",
|
|
12
|
+
"Prioritizes findings by frequency × time-cost × people affected, sequencing quick wins before structural work, with owners",
|
|
13
|
+
"Plans to re-measure the baseline numbers after remediation and keep 2–3 DX metrics tracked continuously"
|
|
14
|
+
]
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"id": 2,
|
|
18
|
+
"prompt": "Management says our DX is fine because nobody's complaining, but I notice engineers never run the full test suite locally, deploys pile up on Fridays, and everyone SSHes into a shared staging box to test things. Is there a DX problem here?",
|
|
19
|
+
"expected_output": "Reads the avoidance signals as the loudest DX evidence: skipping the local suite implies it's too slow/flaky (verification is being skipped — top-tier severity), deploy batching implies the deploy loop is painful or risky, shared-staging testing implies local environments can't reproduce reality; quantifies each to confirm, and explains why absence of complaints doesn't mean absence of tax (normalized friction becomes invisible)",
|
|
20
|
+
"assertions": [
|
|
21
|
+
"Identifies the described behaviors as avoidance signals that indicate DX problems despite the absence of complaints",
|
|
22
|
+
"Infers the likely underlying causes: local suite too slow or flaky, painful/risky deploy loop, local environments unable to reproduce reality",
|
|
23
|
+
"Flags that engineers skipping verification (not running tests locally) is especially severe because it distorts quality behavior",
|
|
24
|
+
"Proposes measuring to confirm (suite runtime, flake rate, deploy lead time, env setup) rather than acting on inference alone",
|
|
25
|
+
"Explains why normalized friction stops generating complaints (habituation), so surveys must ask about avoidance, not just satisfaction"
|
|
26
|
+
]
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": 3,
|
|
30
|
+
"prompt": "Our checkout module is a nightmare of spaghetti code — cyclomatic complexity through the roof, no tests, God classes everywhere. Audit it and give us a prioritized cleanup roadmap.",
|
|
31
|
+
"expected_output": "Recognizes this as a code-health/debt assessment, not a workflow-friction audit — refers to technical-debt-review for the hotspot analysis and remediation roadmap, stating the boundary: dx-audit covers the workflow around the code (loops, tooling, environments); technical-debt-review covers the code itself",
|
|
32
|
+
"assertions": [
|
|
33
|
+
"Recognizes the request as a code-health/technical-debt assessment, not a developer-workflow audit",
|
|
34
|
+
"Refers to the technical-debt-review skill for the hotspot analysis and cleanup roadmap",
|
|
35
|
+
"Does not produce a DX workflow audit (CI timings, onboarding, tooling friction) as the answer",
|
|
36
|
+
"States or implies the boundary: dx-audit audits the workflow around the code; technical-debt-review audits the code's health"
|
|
37
|
+
]
|
|
38
|
+
}
|
|
39
|
+
]
|
|
40
|
+
}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: feature-planning
|
|
3
|
-
description: "Break features into well-scoped tasks with acceptance criteria, risk assessment, and dependency mapping. Triggers: plan this, break this down, scope this feature, create tasks for, sprint planning, how should I implement this feature, user stories, acceptance criteria, dependency mapping, feature breakdown."
|
|
3
|
+
description: "Break features into well-scoped tasks with acceptance criteria, risk assessment, and dependency mapping. Triggers: plan this, break this down, scope this feature, create tasks for, sprint planning, how should I implement this feature, user stories, acceptance criteria, dependency mapping, feature breakdown. Creates the plan — executing an already-approved plan checkpoint by checkpoint → plan-execution."
|
|
4
4
|
model: sonnet
|
|
5
5
|
allowed-tools: Read, Grep, Glob, Write, Edit
|
|
6
6
|
---
|
|
@@ -82,3 +82,7 @@ Output the complete plan using the template at [templates/plan.md](templates/pla
|
|
|
82
82
|
If the feature involves architectural decisions (new services, database changes, API design), suggest using the `architecture-design` skill before finalizing the plan.
|
|
83
83
|
|
|
84
84
|
If the feature requires data model changes, suggest using the `data-modeling` skill for that portion.
|
|
85
|
+
|
|
86
|
+
If the idea itself is still fuzzy — the user is exploring what to build rather than how — run `brainstorming` first; this skill converges, that one diverges.
|
|
87
|
+
|
|
88
|
+
Once the plan is approved and execution begins, the `plan-execution` skill takes over: checkpointed batches, verification evidence per checkpoint, and drift-triggered re-planning (which loops back here).
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: finops-cost-optimization
|
|
3
|
+
description: "Understand and reduce cloud spend — cost visibility and allocation (tagging, ownership), unit economics, rightsizing, autoscaling, commitment discounts (reserved instances, savings plans), storage/egress traps, architecture cost review, budgets and anomaly alerts. Triggers: cloud costs, AWS bill, cut costs, cost optimization, FinOps, why is our bill so high, rightsizing, reserved instances, savings plan, egress fees, cost per customer. Whether to build or buy a capability → build-vs-buy; making code faster → performance-optimization."
|
|
4
|
+
model: sonnet
|
|
5
|
+
allowed-tools: Read, Grep, Glob, Write, Edit, Bash, WebFetch, WebSearch
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# FinOps & Cost Optimization
|
|
9
|
+
|
|
10
|
+
Treat cloud cost as an engineering metric — observable, attributed, and
|
|
11
|
+
optimized with the same rigor as latency. The failure mode this skill counters:
|
|
12
|
+
nobody owns the bill, so it only gets attention as a quarterly surprise, and
|
|
13
|
+
the response is a panicked freeze instead of systematic reduction. Boundaries:
|
|
14
|
+
`performance-optimization` makes code faster (sometimes cheaper as a side
|
|
15
|
+
effect); `build-vs-buy` decides whether to run the capability at all; this
|
|
16
|
+
skill optimizes what you *do* run.
|
|
17
|
+
|
|
18
|
+
## Workflow
|
|
19
|
+
|
|
20
|
+
### Step 1: Visibility Before Optimization
|
|
21
|
+
|
|
22
|
+
You can't cut what you can't attribute. Establish first:
|
|
23
|
+
|
|
24
|
+
- **Allocation**: tag/label resources by service, team, and environment;
|
|
25
|
+
measure the untagged remainder and drive it down — an 40%-unattributed bill
|
|
26
|
+
makes every later step guesswork.
|
|
27
|
+
- **The big rocks**: rank spend by service and by resource type. Cloud bills
|
|
28
|
+
follow a power law; the top 5 line items usually carry most of the total.
|
|
29
|
+
Optimizing anything outside them is procrastination with charts.
|
|
30
|
+
- **Trend vs events**: is spend growing with usage (fine, if unit costs hold),
|
|
31
|
+
or did it step-change with a deploy (an incident with a dollar signature)?
|
|
32
|
+
|
|
33
|
+
Pull real billing data (cost explorer exports, `infracost` on IaC diffs, the
|
|
34
|
+
provider CLIs) — never optimize from memory of last quarter's bill.
|
|
35
|
+
|
|
36
|
+
### Step 2: Define Unit Economics
|
|
37
|
+
|
|
38
|
+
Total spend is noise; **cost per unit of value** — per customer, per request,
|
|
39
|
+
per GB processed, per model inference — is signal. A bill that doubles while
|
|
40
|
+
customers triple is a win. Pick 1–3 unit metrics with the team, compute them
|
|
41
|
+
from the Step-1 allocation, and make them the number optimization is judged by.
|
|
42
|
+
This is also what connects engineering to the business conversation
|
|
43
|
+
(`metrics-and-okrs`).
|
|
44
|
+
|
|
45
|
+
### Step 3: Harvest the Waste (no architecture changes)
|
|
46
|
+
|
|
47
|
+
The reliable first wins, in typical order of return:
|
|
48
|
+
|
|
49
|
+
- **Orphans**: unattached volumes/IPs, forgotten environments, snapshots-forever,
|
|
50
|
+
idle dev clusters running nights and weekends (schedule them off).
|
|
51
|
+
- **Rightsizing**: instances/databases provisioned for imagined peak — check
|
|
52
|
+
actual utilization (single-digit CPU on a large instance is a purchase order
|
|
53
|
+
for the cloud provider, not capacity planning).
|
|
54
|
+
- **Storage classes & lifecycle**: hot-tier data nobody has read in a year;
|
|
55
|
+
logs and backups with no expiry — lifecycle policies are one-line fixes.
|
|
56
|
+
- **Non-prod parity**: staging doesn't need production's instance sizes,
|
|
57
|
+
replica counts, or retention.
|
|
58
|
+
|
|
59
|
+
### Step 4: Buy Commitment for the Stable Baseline
|
|
60
|
+
|
|
61
|
+
After rightsizing (never before — you'd lock in the waste): cover the stable,
|
|
62
|
+
predictable baseline with reserved instances / savings plans / committed-use
|
|
63
|
+
discounts, and keep the spiky remainder on-demand or spot (spot for
|
|
64
|
+
interruption-tolerant work: batch, CI, stateless workers). Start conservative
|
|
65
|
+
(cover well under 100% of the baseline) and ratchet up as forecasts prove out —
|
|
66
|
+
over-commitment converts a variable cost into a fixed regret.
|
|
67
|
+
|
|
68
|
+
### Step 5: Review the Architecture's Cost Shape
|
|
69
|
+
|
|
70
|
+
Some spend is structural, and only architecture changes move it:
|
|
71
|
+
|
|
72
|
+
- **Egress and cross-zone/region chatter** — the classic invisible line item;
|
|
73
|
+
co-locate chatty services, cache at the edge, compress.
|
|
74
|
+
- **Per-invocation pricing at scale** — serverless that grew past the
|
|
75
|
+
crossover where instances are cheaper (and vice versa: idle instances that
|
|
76
|
+
should be serverless).
|
|
77
|
+
- **Managed-premium vs ops-burden** — the managed service's markup is often
|
|
78
|
+
worth it (that's a `build-vs-buy` call when it gets big).
|
|
79
|
+
- **Data pipelines** — scan-based pricing (warehouse queries over unpartitioned
|
|
80
|
+
tables) and always-on streaming for hourly-batch needs (`data-pipeline-design`).
|
|
81
|
+
|
|
82
|
+
Cost these with current provider pricing (verify live — pricing changes), and
|
|
83
|
+
record significant trade-offs as ADRs (`architecture-design`).
|
|
84
|
+
|
|
85
|
+
### Step 6: Keep It Continuous
|
|
86
|
+
|
|
87
|
+
- **Budgets + anomaly alerts** per team/service, routed to the owning team —
|
|
88
|
+
a cost regression is a defect, detected in days not quarters.
|
|
89
|
+
- **Cost in the deploy loop**: IaC cost diffs (e.g. infracost) in PRs makes
|
|
90
|
+
cost visible at decision time.
|
|
91
|
+
- **A monthly 30-minute review** of the unit metrics and top movers beats an
|
|
92
|
+
annual cost-cutting project.
|
|
93
|
+
|
|
94
|
+
## Principles Applied
|
|
95
|
+
|
|
96
|
+
- **Attribution before optimization** — unowned costs never stay cut.
|
|
97
|
+
- **Unit economics over totals** — growth should raise the bill; only rising
|
|
98
|
+
*unit* cost is a problem.
|
|
99
|
+
- **Biggest line item first** — a 10% cut of the top item beats a 90% cut of
|
|
100
|
+
the tenth.
|
|
101
|
+
|
|
102
|
+
## Cross-Skill References
|
|
103
|
+
|
|
104
|
+
- `build-vs-buy` — whether to run the capability at all; managed-vs-self-hosted TCO
|
|
105
|
+
- `performance-optimization` — efficiency work that reduces compute per request
|
|
106
|
+
- `architecture-design` — recording cost-driven structural decisions as ADRs
|
|
107
|
+
- `observability-design` — the metrics/alerting machinery cost signals ride on
|
|
108
|
+
- `data-pipeline-design` — cost-shaping batch/streaming and warehouse workloads
|
|
109
|
+
- `metrics-and-okrs` — tying unit economics to business metrics
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill_name": "finops-cost-optimization",
|
|
3
|
+
"evals": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"prompt": "Our AWS bill jumped from $18k to $31k/month over the last two quarters and leadership wants it 'fixed'. Where do we start?",
|
|
7
|
+
"expected_output": "Starts with visibility, not cuts: attribute the spend (tagging/allocation), rank the top line items, and separate usage-driven growth from step-change regressions; defines unit economics (cost per customer/request) as the real judge; then harvests waste in order of return (orphans, rightsizing from actual utilization, storage lifecycle, non-prod schedules), only then commitment discounts on the post-rightsizing baseline, plus budgets/anomaly alerts so the next jump is caught in days",
|
|
8
|
+
"assertions": [
|
|
9
|
+
"Establishes visibility and attribution first (tagging/allocation, ranking top line items from real billing data) before proposing any cuts",
|
|
10
|
+
"Distinguishes usage-driven growth from step-change regressions and/or introduces unit economics (cost per customer/request) rather than treating the total as the metric",
|
|
11
|
+
"Proposes concrete waste harvesting: orphaned resources, rightsizing based on actual utilization, storage lifecycle policies, or scheduling non-prod environments off",
|
|
12
|
+
"Recommends commitment discounts (RIs/savings plans) only after rightsizing, covering a conservative stable baseline",
|
|
13
|
+
"Sets up continuous guardrails: budgets and anomaly alerts routed to owning teams, and/or cost checks in the deploy loop"
|
|
14
|
+
]
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"id": 2,
|
|
18
|
+
"prompt": "Our bill is dominated by data transfer: about 40% is inter-AZ traffic and NAT gateway charges between our microservices, and the rest of the top items are a Kafka cluster and a data warehouse where queries scan whole tables. Rightsizing won't move these. What are our options?",
|
|
19
|
+
"expected_output": "Recognizes structural/architectural cost shape: co-locating chatty services or reworking cross-AZ topology (weighing the availability trade-off), VPC endpoints / NAT alternatives, compression; warehouse scan costs via partitioning/clustering and incremental models; evaluates whether always-on streaming matches the actual latency need; recommends costing options with live pricing and recording the trade-offs as ADRs since these decisions trade resilience/latency for money",
|
|
20
|
+
"assertions": [
|
|
21
|
+
"Recognizes these as architecture-shaped costs that rightsizing cannot fix and addresses the egress/inter-AZ problem specifically (co-location, topology, VPC endpoints, compression)",
|
|
22
|
+
"Notes the availability/resilience trade-off in reducing cross-AZ traffic rather than recommending single-AZ blindly",
|
|
23
|
+
"Addresses the warehouse scan costs structurally (partitioning, clustering, incremental models) rather than only suggesting fewer queries",
|
|
24
|
+
"Questions whether the always-on streaming infrastructure matches the actual latency requirement (batch vs streaming)",
|
|
25
|
+
"Recommends verifying costs against current provider pricing and/or recording significant cost-driven decisions as ADRs"
|
|
26
|
+
]
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": 3,
|
|
30
|
+
"prompt": "Our checkout API's p99 latency is 4 seconds and profiling shows an N+1 query pattern plus a missing index. Can you help optimize it?",
|
|
31
|
+
"expected_output": "Recognizes this as code/query performance work, not cloud-spend optimization — refers to performance-optimization for the N+1 and indexing work, noting the boundary: performance-optimization makes code faster (sometimes cheaper as a side effect); finops-cost-optimization attributes and reduces cloud spend",
|
|
32
|
+
"assertions": [
|
|
33
|
+
"Recognizes the request as application performance optimization, not cloud cost management",
|
|
34
|
+
"Refers to the performance-optimization skill for the N+1/indexing work",
|
|
35
|
+
"Does not produce a FinOps workflow (tagging, rightsizing, commitments) as the answer",
|
|
36
|
+
"States or implies the boundary: performance-optimization owns making code fast; finops-cost-optimization owns understanding and reducing spend"
|
|
37
|
+
]
|
|
38
|
+
}
|
|
39
|
+
]
|
|
40
|
+
}
|
|
@@ -52,6 +52,10 @@ Use the Conventional Commits format — see [references/conventions.md](referenc
|
|
|
52
52
|
- Breaking changes: `BREAKING CHANGE: description`
|
|
53
53
|
- Co-authors: `Co-authored-by: Name <email>`
|
|
54
54
|
|
|
55
|
+
Commit types are not just style — in automated release flows they drive the version:
|
|
56
|
+
`fix` → PATCH, `feat` → MINOR, `feat!`/`BREAKING CHANGE:` → MAJOR. A mislabeled type
|
|
57
|
+
mis-versions the next release (see `release-management`).
|
|
58
|
+
|
|
55
59
|
### Step 3: Validate
|
|
56
60
|
|
|
57
61
|
Before committing, check:
|
|
@@ -125,3 +129,9 @@ See [references/conventions.md](references/conventions.md) for branch naming con
|
|
|
125
129
|
- **KISS**: One commit = one logical change. Don't bundle unrelated changes.
|
|
126
130
|
- **DRY**: If you're writing the same commit message pattern repeatedly, you might be making commits too granular or too broad.
|
|
127
131
|
- **Functional Independence**: Each PR should be independently deployable when possible. Avoid PRs that depend on other unmerged PRs.
|
|
132
|
+
|
|
133
|
+
## Cross-Skill References
|
|
134
|
+
|
|
135
|
+
- `release-management` — cutting the release these commits accumulate into: semver, changelog, tagging, publishing
|
|
136
|
+
- `code-reviewing` — reviewing the PR the description introduces
|
|
137
|
+
- `verification-before-completion` — run the proving commands before pushing or opening the PR
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: llm-app-engineering
|
|
3
|
+
description: "Design and build LLM-powered applications — prompt and context engineering, RAG architecture (chunking, hybrid retrieval, reranking, grounded prompts), agent design (tool surfaces, planning, sub-agents), and memory. Triggers: build a chatbot, LLM app, AI assistant, prompt engineering, system prompt, RAG design, chunking, embeddings, vector database, semantic search, AI agent, tool calling, agent memory, context window, hallucinations. Measuring quality → ai-evaluation; model serving/inference infra → ml-model-deployment."
|
|
4
|
+
model: opus
|
|
5
|
+
allowed-tools: Read, Grep, Glob, Write, Edit, Bash
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# LLM App Engineering
|
|
9
|
+
|
|
10
|
+
Design LLM applications as **systems, not prompts**: what goes into the context
|
|
11
|
+
window, what the model is allowed to do, and what happens when it's wrong. The
|
|
12
|
+
model is the one component you don't control — engineer everything around it.
|
|
13
|
+
This skill designs and builds; proving a change helped is `ai-evaluation`'s job,
|
|
14
|
+
and no design below is "done" until it has an eval harness.
|
|
15
|
+
|
|
16
|
+
## Workflow
|
|
17
|
+
|
|
18
|
+
### Step 1: Choose the Simplest Sufficient Architecture
|
|
19
|
+
|
|
20
|
+
Escalate only when the previous rung measurably fails:
|
|
21
|
+
|
|
22
|
+
1. **Single prompt** — the task fits in one call with instructions + input.
|
|
23
|
+
2. **+ RAG** — the answer needs knowledge that doesn't fit in the prompt or
|
|
24
|
+
changes independently of deploys.
|
|
25
|
+
3. **+ Tools** — the model must act (query, search, write) — not just answer.
|
|
26
|
+
4. **Agent loop** — the path to the goal is genuinely unpredictable in advance.
|
|
27
|
+
Most "agents" that fail are workflows that should have been a fixed pipeline
|
|
28
|
+
with one or two LLM steps.
|
|
29
|
+
|
|
30
|
+
### Step 2: Engineer the Prompt as an Interface
|
|
31
|
+
|
|
32
|
+
- Structure: role and constraints → domain context → task → output format.
|
|
33
|
+
Put stable content first (prompt caching) and the volatile input last.
|
|
34
|
+
- State what to do on failure: "if the context doesn't contain the answer, say
|
|
35
|
+
so" — the unhappy path is where LLM apps lose user trust.
|
|
36
|
+
- Version prompts in git like code; a prompt change is a deploy and goes
|
|
37
|
+
through the same eval gate (`ai-evaluation`).
|
|
38
|
+
|
|
39
|
+
### Step 3: RAG — Design Retrieval Before Generation
|
|
40
|
+
|
|
41
|
+
Most bad RAG answers are retrieval failures wearing a generation costume.
|
|
42
|
+
|
|
43
|
+
- **Chunking**: split on document structure (headings, sections, one
|
|
44
|
+
ticket+resolution per chunk), not fixed character counts; attach metadata
|
|
45
|
+
(source, date, product area) for filtering and citations.
|
|
46
|
+
- **Retrieval**: hybrid (BM25 + vector) as the default — pure vector search
|
|
47
|
+
misses exact identifiers, SKUs, and error codes; filter by metadata, then
|
|
48
|
+
**rerank** the candidate set (retrieve ~20–50, keep the top 5–10).
|
|
49
|
+
- **Context budget** (decide, don't default): how many chunks fit, in what
|
|
50
|
+
order (most relevant near the question), the relevance floor below which a
|
|
51
|
+
chunk is dropped, and the behavior when nothing clears it — say "I don't
|
|
52
|
+
know" rather than pad the context with noise.
|
|
53
|
+
- **Grounding prompt**: answer only from the provided context, cite which
|
|
54
|
+
source supports each claim, refuse when unsupported.
|
|
55
|
+
|
|
56
|
+
### Step 4: Agents — Design the Tool Surface and the Loop
|
|
57
|
+
|
|
58
|
+
- **Tools**: few and sharply distinct beat many and overlapping — every tool
|
|
59
|
+
description competes for the model's attention on every step. Name by
|
|
60
|
+
intent (`search_orders`, not `api_call`); make each tool's failure return a
|
|
61
|
+
message the model can act on, not a stack trace.
|
|
62
|
+
- **Decomposition**: long tasks fail as one monolithic loop with 25 tools;
|
|
63
|
+
split into phases or sub-agents, each with a scoped toolset, and pass
|
|
64
|
+
structured summaries between them — not raw transcripts.
|
|
65
|
+
- **Guardrails**: cap iterations, budget tokens per task, and validate
|
|
66
|
+
side-effecting tool calls (or require confirmation) — an agent retrying a
|
|
67
|
+
failed write in a loop is the classic production incident.
|
|
68
|
+
|
|
69
|
+
### Step 5: Memory — Externalize State
|
|
70
|
+
|
|
71
|
+
The context window is a scarce cache, not a database. For long-horizon work:
|
|
72
|
+
keep a plan/scratchpad file the agent re-reads and updates; summarize or
|
|
73
|
+
compact stale history instead of letting it scroll off; store durable facts
|
|
74
|
+
(user preferences, decisions) outside the window and retrieve them like RAG.
|
|
75
|
+
|
|
76
|
+
### Step 6: Wire In Evaluation Before Shipping
|
|
77
|
+
|
|
78
|
+
Hand the design to `ai-evaluation`: a golden dataset (including unanswerable
|
|
79
|
+
and adversarial cases), RAG-stage metrics (retrieval vs generation scored
|
|
80
|
+
separately), agent task-success rate, and a CI regression gate. "The answers
|
|
81
|
+
feel better" is not evidence; a design without an eval harness is a prototype.
|
|
82
|
+
|
|
83
|
+
Deeper pattern detail (chunking table, reranking, memory patterns, failure
|
|
84
|
+
modes): [references/patterns.md](references/patterns.md).
|
|
85
|
+
|
|
86
|
+
## Principles Applied
|
|
87
|
+
|
|
88
|
+
- **KISS**: every architecture rung you skip is a class of failure you never
|
|
89
|
+
have to debug. Workflows beat agents when the path is predictable.
|
|
90
|
+
- **YAGNI**: no vector DB before checking whether the corpus fits in the
|
|
91
|
+
prompt; no agent framework before a plain loop with 3 tools fails.
|
|
92
|
+
- **SRP for tools**: one tool, one capability — overlapping tools force the
|
|
93
|
+
model to guess, and it will guess differently each run.
|
|
94
|
+
|
|
95
|
+
## Cross-Skill References
|
|
96
|
+
|
|
97
|
+
- `ai-evaluation` — golden datasets, RAG metrics, judges, eval gates (the
|
|
98
|
+
measurement half of every step above)
|
|
99
|
+
- `ml-model-deployment` — serving, monitoring, and drift once the app ships
|
|
100
|
+
- `api-design` — the contract around the LLM feature (streaming, errors, timeouts)
|
|
101
|
+
- `security-audit` — prompt injection, data exfiltration via tools, output handling
|
|
102
|
+
- `architecture-design` — ADRs for costly-to-reverse choices (vector store, framework)
|