swe-workflow-skills 0.1.0 → 0.3.0

package/skills/data-pipeline-design/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: data-pipeline-design
+description: "Design batch and streaming data pipelines — ELT into a warehouse/lakehouse, dbt and analytics engineering (staging/intermediate/marts layers), orchestration with Airflow or Dagster, idempotent loads, incremental models, backfills, CDC ingestion. Triggers: data pipeline, ELT, ETL, dbt models, data warehouse, Snowflake, BigQuery, ingest into the warehouse, Airflow DAG, Dagster, Kafka streaming, backfill, incremental load, nightly job duplicates rows. ML training/feature pipelines → ml-pipeline-design; validating the data → data-quality."
+model: opus
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# Data Pipeline Design
+
+Design pipelines that move and transform data for analytics — and stay correct
+when they fail, rerun, and backfill, because they will. The two design
+properties that separate a pipeline from a script: **idempotency** (any run can
+be repeated safely) and **parameterization** (any time window can be processed
+by the same code). Boundaries: pipelines feeding *model training* (features,
+point-in-time correctness) belong to `ml-pipeline-design`; *validating* the
+data this pipeline moves belongs to `data-quality`.
+
+## Workflow
+
+### Step 1: Map Sources, Consumers, and Freshness
+
+List each source (databases, SaaS APIs, event streams), each consumer
+(dashboards, reverse ETL, downstream teams), and the freshness each consumer
+actually needs. Freshness drives everything: daily batch is an order of
+magnitude cheaper to build and operate than streaming — don't buy streaming
+because "real-time" sounds better; buy it when a consumer decision genuinely
+changes within minutes.
+
+### Step 2: Choose ELT (and Say Why)
+
+For a cloud warehouse (Snowflake, BigQuery, Redshift, Databricks), default to
+**ELT**: land raw data unchanged, transform inside the warehouse with SQL/dbt.
+The reasons are operational, not fashionable — raw history means transform bugs
+are fixed by *re-running transforms*, not re-extracting from sources; the
+warehouse scales the compute; transformations become versioned, testable SQL.
+Classic ETL survives where data must be cleaned/redacted before it may land
+(PII, compliance) or the target isn't a warehouse.
+
+### Step 3: Don't Hand-Roll Extraction
+
+Ingestion is undifferentiated heavy lifting with long tail of pain (API
+pagination, rate limits, schema changes, backpressure):
+
+- **SaaS/DB sources**: managed or open connectors first — Fivetran/Airbyte/dlt.
+  Hand-write only when no connector exists or volume/cost forces it.
+- **Operational DBs at scale**: CDC (Debezium or the warehouse-native
+  equivalent) instead of daily full dumps or fragile `updated_at` queries.
+- **Events**: land the stream (Kafka → object storage/warehouse) raw; process
+  micro-batch unless Step 1 proved sub-minute freshness is needed.
+
+### Step 4: Layer the Transforms (dbt)
+
+- **Staging** — 1:1 with sources: rename, cast, deduplicate. No business logic.
+- **Intermediate** — joins and business logic, composable, not exposed to BI.
+- **Marts** — consumer-facing facts/dimensions, one per business domain.
+
+Each layer only reads from the one below; BI tools read only marts. This is
+SRP for SQL: when a number is wrong, the layer tells you where to look. Schema
+design for the marts themselves is `data-modeling`'s domain.
+
+### Step 5: Make Every Run Idempotent
+
+The Iron Rule of pipelines: **a rerun must produce the same result as one
+run.** Append-only loads fail this and produce the classic duplicate-rows-
+after-retry incident. Choose a write pattern per table:
+
+- **Delete + insert by partition** (or `INSERT OVERWRITE`): simplest, right
+  default for date-partitioned facts.
+- **Merge/upsert on a unique key**: for mutable entities and late updates —
+  requires a true unique key (enforce it via `data-quality` tests).
+- **Full refresh**: fine while small; flip to incremental on cost, not pride.
+
+In dbt: `incremental` materialization with `unique_key` set — and always define
+the full-refresh path (`--full-refresh` must work, or you can't recover from a
+logic bug).
+
+### Step 6: Parameterize Time — Backfill Is a First-Class Case
+
+Every run processes an explicit **logical window** passed in by the
+orchestrator (`WHERE event_date = {{ ds }}`) — never `CURRENT_DATE - 1` inside
+the SQL, which makes yesterday's failure unreproducible today. Then backfill is
+not a special script: it's the same job over a range, natively supported by
+the orchestrator (Airflow `backfill`, Dagster partitioned assets). If
+backfilling means hand-editing dates, Steps 5–6 were skipped.
+
+Handle **late-arriving data** explicitly: reprocess a trailing lookback window
+(e.g. the last 3 days) each run, or track a watermark — pick per source based
+on how late its data actually arrives.
+
+### Step 7: Orchestrate and Operate
+
+Pick **one** orchestrator and make a recommendation, not a brochure: Dagster
+for a dbt-centric greenfield stack (asset/partition model matches warehouse
+thinking, first-class dbt integration); Airflow where the org already runs it
+or needs its ecosystem breadth (comparison and managed options:
+[references/orchestration.md](references/orchestration.md)). Then wire in
+operations: retries with alerting on final failure (`observability-design`),
+quality checks between layers (`data-quality`), and cost visibility per model —
+warehouses make it very easy to spend quietly.
+
+## Principles Applied
+
+- **KISS**: batch over streaming, connectors over custom extractors, SQL over
+  a framework — until a measured requirement says otherwise.
+- **DRY**: one parameterized job for daily runs *and* backfills; logic lives
+  once, in the transform layer, not copied into extraction scripts.
+- **YAGNI**: no lakehouse/streaming/metadata-platform buildout for a stack
+  whose consumers refresh dashboards daily.
+
+## Cross-Skill References
+
+- `data-quality` — tests, contracts, and freshness checks between the layers
+- `data-modeling` — the schema design of the marts this pipeline produces
+- `ml-pipeline-design` — when the consumer is model training, not BI
+- `observability-design` — pipeline SLOs, alerting, run-level monitoring
+- `architecture-design` — ADRs for the costly-to-reverse platform choices

package/skills/data-pipeline-design/evals/evals.json

@@ -0,0 +1,41 @@
+{
+  "skill_name": "data-pipeline-design",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We need to get data from Postgres and Stripe into Snowflake for BI dashboards. Design the pipeline — the team is considering dbt plus Airflow or Dagster.",
+      "expected_output": "Designs an ELT pipeline: extraction/connector choice for both sources, raw load into Snowflake, dbt layering (staging → intermediate → marts), a reasoned Airflow-vs-Dagster recommendation, idempotent incremental models, and a backfill story — not just the initial full load",
+      "assertions": [
+        "Recommends ELT (load raw, transform in-warehouse) and explains why it fits a Snowflake + dbt stack",
+        "Designs dbt layering: raw/staging models, intermediate transformations, and marts for BI consumption",
+        "Makes a reasoned orchestrator recommendation (Airflow vs Dagster) instead of listing both neutrally",
+        "Designs for idempotency: reruns must not duplicate or corrupt data (incremental models with unique keys, merge or delete+insert patterns)",
+        "Covers incremental loading and a backfill strategy (parameterized date ranges or partitions), not just the initial full load",
+        "Addresses extraction from the sources concretely (managed connector such as Fivetran/Airbyte/dlt, or CDC) rather than hand-waving ingestion"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Our nightly job loads yesterday's orders into the warehouse. When it fails midway and we rerun it, we get duplicate rows — and backfilling last month means manually editing the date in the script 30 times. Fix the design.",
+      "expected_output": "Identifies the non-idempotent append-only load as the root cause; converts the job to an idempotent write pattern (delete+insert by partition, merge/upsert on a key, or partition overwrite) parameterized over a logical date, so backfill becomes running the same job over a range via the orchestrator — with a lookback window for late-arriving data",
+      "assertions": [
+        "Identifies the root cause as a non-idempotent, append-only load",
+        "Proposes an idempotent write pattern: delete+insert by partition, merge/upsert on a unique key, or partition-overwrite semantics",
+        "Parameterizes the run over a logical date/window so any day can be run or re-run identically (backfill = the same job over a range)",
+        "Recommends orchestrator-native backfill (or an equivalent parameterized range run) over hand-edited scripts",
+        "Considers late-arriving or corrected data (lookback window, watermark, or reprocessing policy), not just the happy path"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Design the data pipeline that produces training features for our churn model — we need point-in-time correct features and a training set refreshed weekly.",
+      "expected_output": "Recognizes this as an ML training/feature pipeline, not analytics ELT — hands off to ml-pipeline-design, flagging point-in-time correctness/leakage as the ML-specific concern that a BI-oriented warehouse design does not address",
+      "assertions": [
+        "Recognizes this is an ML training/feature pipeline, not an analytics/BI ELT pipeline",
+        "Refers to the ml-pipeline-design skill for feature engineering and training orchestration",
+        "Flags point-in-time correctness / data leakage as the ML-specific concern that distinguishes this from BI work",
+        "Does not answer with a BI-oriented warehouse/dbt marts design as the primary solution"
+      ]
+    }
+  ]
+}

package/skills/data-pipeline-design/references/orchestration.md

@@ -0,0 +1,64 @@
+# Orchestration — Choosing and Using the Scheduler
+
+Versions and managed offerings change; check current docs before committing.
+The selection logic is stable.
+
+## Airflow vs Dagster vs the rest
+
+| | Airflow | Dagster |
+|---|---|---|
+| Mental model | Tasks in a DAG; scheduler runs tasks | Software-defined **assets** (tables/files) with partitions; runs materialize assets |
+| dbt integration | Via operators (Cosmos et al.) — workable | First-class: each dbt model becomes an asset with its own lineage/partitions |
+| Backfill | `airflow backfill` over execution dates | Partitioned assets — backfill = materialize missing partitions, UI-native |
+| Ecosystem | Enormous (every provider has an operator); huge hiring pool | Younger, smaller, growing; strong local dev experience |
+| Managed | MWAA (AWS), Cloud Composer (GCP), Astronomer | Dagster+ |
+
+Recommendation logic:
+
+- **Greenfield, warehouse/dbt-centric** → Dagster. The asset+partition model
+  *is* Steps 5–6 of the skill (idempotent, parameterized, backfillable) encoded
+  in the orchestrator; you fight the framework less.
+- **Org already runs Airflow, or the pipeline is mostly "call these seven
+  systems in order"** → Airflow. Operational familiarity and operator breadth
+  beat elegance; don't run two orchestrators for one team.
+- **All-in on one vendor stack** → the platform-native option (Databricks
+  Workflows, Snowflake Tasks) is acceptable glue; you trade portability.
+- **Cron + a queue** is fine for one or two jobs — adopt an orchestrator when
+  you need dependencies, backfill, and retry visibility, which is usually at
+  the third job.
+
+Whatever the choice: schedules trigger *logical windows*, tasks are
+idempotent, retries are bounded and alert on final failure, and secrets come
+from the platform's secret backend (`configuration-strategy`), never DAG code.
+
+## Ingestion tool notes
+
+- **Fivetran**: managed, broadest connector catalog, per-row pricing that gets
+  real at volume. Buy when engineer time is the scarce resource.
+- **Airbyte**: open-source + cloud; connector quality varies — test the
+  specific connectors you need, especially for incremental correctness.
+- **dlt**: Python library, connectors-as-code, great for API sources a
+  framework doesn't cover; you own the runtime.
+- **CDC (Debezium / native)**: when daily snapshots lose intermediate states
+  or full dumps are too heavy. Operationally nontrivial (snapshotting, schema
+  evolution, tombstones) — budget for it.
+
+## Streaming — only after Step 1 says so
+
+- Micro-batch (5–15 min) covers most "real-time" dashboard asks at batch-like
+  complexity; genuine streaming (Flink, Spark Structured Streaming,
+  Materialize) is for operational decisions made in seconds-to-minutes.
+- Streaming reintroduces every hard problem batch solved for you: exactly-once
+  vs at-least-once delivery, watermarks for late events, state management,
+  replay. Idempotent sinks (merge on key) make at-least-once livable — design
+  the sink first.
+- Keep the raw stream landed to cheap storage regardless; it's your backfill
+  and reprocessing escape hatch.
+
+## Warehouse cost hygiene
+
+- Tag/attribute compute per model or job from day one (dbt does this cheaply).
+- Incremental models exist for cost as much as latency — flip when a full
+  refresh's bill, not its runtime, annoys you.
+- Watch for accidental cross-joins and full scans in staging layers; the
+  warehouse will happily execute them nightly forever.

package/skills/data-quality/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: data-quality
+description: "Keep warehouse and pipeline data trustworthy — dbt tests, expectations (Great Expectations/Soda), data contracts with upstream producers, source freshness, schema-drift and volume anomaly detection, lineage and blast radius, severity and alerting. Triggers: data quality, bad data in dashboards, data validation, dbt tests, data contract, schema drift, stale data, freshness check, duplicate rows, nulls in the warehouse, data lineage, data SLA, upstream schema change broke us. Generating test/synthetic data → test-data-strategy; schema design → data-modeling."
+model: sonnet
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# Data Quality
+
+Make bad data **loud**: caught at the boundary where it enters, blocking what
+must not ship, alerting someone who owns the fix — instead of being discovered
+by a stakeholder in a dashboard. Boundaries: *generating* fake/test data is
+`test-data-strategy`; *designing* the schema is `data-modeling`; this skill
+keeps the real data flowing through pipelines trustworthy.
+
+## Workflow
+
+### Step 1: Start From Incidents, Not Inventory
+
+Don't blanket every column with tests. List the recent data incidents (wrong
+dashboard numbers, silent breakage) and the tables where an error costs real
+money or trust — revenue marts, ML features, anything an exec reads. Each
+known symptom maps to a check: nulls → `not_null`, duplicates → `unique`,
+staleness → freshness, broken joins → `relationships`. Wins from day one, and
+the suite grows the same way tests do after bugs — every new incident adds a
+check that would have caught it.
+
+### Step 2: Check at the Right Layer — Boundary First
+
+Where a check runs matters as much as what it checks:
+
+- **Ingestion boundary (sources)**: schema, freshness, volume, primary-key
+  integrity — catch bad inputs *before* transforms consume them. One failed
+  source check beats fifty downstream test failures at 6 a.m.
+- **Transform layers (staging → marts)**: uniqueness/not-null on the grain of
+  each model, relationship tests on joins, accepted values on enums.
+- **Consumer edge (marts/metrics)**: business-rule assertions — "revenue is
+  never negative", "row count within N% of the 7-day average".
+
+In dbt this is: source `freshness` + source tests, generic tests on models
+(`unique`, `not_null`, `relationships`, `accepted_values`), singular SQL tests
+for business rules. Dedicated tools (Great Expectations, Soda) earn their keep
+outside dbt-land or when non-engineers must own the rules:
+[references/tooling.md](references/tooling.md).
+
+### Step 3: Assign Severity — What Blocks vs What Warns
+
+A suite where everything fails the build gets muted within a month; a suite
+where nothing blocks is decoration. Decide per check:
+
+- **error** — data is unusable or contaminating (duplicate keys in a revenue
+  mart, failed contract at ingestion): the pipeline stops, downstream models
+  don't build on poisoned input.
+- **warn** — degraded but usable (slightly stale, volume dip within reason):
+  alert, keep flowing.
+
+dbt: `severity:` config with `error_if`/`warn_if` thresholds. Revisit
+severities after each incident and each false alarm.
+
+### Step 4: Route Failures to an Owner
+
+A failing check nobody sees is identical to no check. Every table (or domain)
+has a named owner; alerts go where that owner works (Slack channel per domain,
+pager only for `error`-severity on critical marts), and each alert says what
+failed, on which table, and the blast radius (Step 5). Track
+time-to-detection and time-to-fix for data incidents — that's the metric this
+whole skill improves (`observability-design` for the SLO framing).
+
+### Step 5: Use Lineage to Scope Blast Radius
+
+When a check fails, the first question is "what downstream is now wrong?" —
+lineage (dbt's DAG, `dbt ls --select my_model+`, or a catalog tool) answers
+which models, dashboards, and consumers sit downstream of the failure. Use it
+to: prioritize fixes by consumer impact, notify affected dashboard owners
+proactively, and decide what to rebuild after the fix. If you can't answer
+"what breaks if this table is wrong?", invest in lineage before more checks.
+
+### Step 6: Contract the Upstream Boundary
+
+Downstream tests detect what upstream producers break; **contracts prevent
+it.** When an external team owns the source: agree schema + semantics + SLA
+explicitly (a versioned artifact — dbt model contracts, JSON Schema/protobuf in
+a registry), enforce at ingestion so violations fail loudly at entry, and shift
+left — contract validation in the *producer's* CI, so a breaking rename fails
+their build, not your 6 a.m. run. Breaking changes get a version bump and a
+deprecation window, coordinated like an API change
+(`dependency-impact-analysis` thinking, applied to data). Back it with
+schema-drift detection and volume/distribution anomaly checks for what
+enforcement can't see.
+
+## Principles Applied
+
+- **YAGNI**: checks earn their place by a failure mode you've seen or can't
+  afford — blanket-testing every column produces alert fatigue, the failure
+  mode that kills quality programs.
+- **Fail loudly at the boundary**: the cost of bad data grows with every layer
+  it flows through undetected.
+- **DRY**: one contract at the source replaces N teams' defensive checks
+  against the same breakage.
+
+## Cross-Skill References
+
+- `data-pipeline-design` — the pipeline these checks are embedded in
+- `data-modeling` — constraints and grain the checks assert (unique keys exist by design)
+- `test-data-strategy` — generating synthetic/fixture data for testing pipelines
+- `observability-design` — alerting design, SLOs, avoiding alert fatigue
+- `incident-response` — when bad data has already reached production consumers

package/skills/data-quality/evals/evals.json

@@ -0,0 +1,39 @@
+{
+  "skill_name": "data-quality",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Stakeholders keep finding bad numbers in our dashboards — null customer IDs, duplicated orders, and data that's a day stale. We're on dbt + Snowflake. Set up data quality checks.",
+      "expected_output": "Maps each symptom to a concrete check (not_null/unique dbt tests, source freshness), places checks at the ingestion boundary first and critical marts second, assigns severity (error blocks, warn alerts), routes failures to an owner, and uses lineage to scope which downstream models and dashboards a failure affects",
+      "assertions": [
+        "Maps each reported symptom to a concrete check: not_null/unique dbt tests for the IDs and duplicates, and source freshness checks for the staleness",
+        "Places checks at the right layers: the source/ingestion boundary first, then critical marts — not only on the final dashboards",
+        "Assigns severity levels (error vs warn) and defines what blocks the pipeline versus what only alerts",
+        "Sets up alerting and ownership so failures reach someone who acts, instead of failing silently",
+        "Uses lineage to scope blast radius — which downstream models or dashboards a failing check affects"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "An upstream team owns the events we ingest and keeps renaming columns and changing types without telling us. Our pipeline breaks silently every few weeks. What should we put in place?",
+      "expected_output": "Proposes a producer/consumer data contract (schema, types, semantics, SLA) enforced at the ingestion boundary so violations fail loudly at entry; shifts validation left into the producer's CI or publish path; adds versioning/deprecation windows for breaking changes; and backstops enforcement with schema-drift and volume/distribution anomaly detection",
+      "assertions": [
+        "Proposes an explicit data contract between producer and consumer (schema, types, semantics, SLAs), not just more downstream tests",
+        "Enforces the schema at the ingestion boundary so violations fail loudly at entry instead of propagating silently",
+        "Shifts checks left: contract validation in the producer's CI or at publish time",
+        "Covers change management for breaking changes: contract versioning or a deprecation window",
+        "Adds detection for what enforcement misses: schema-drift alerts or anomaly checks on volume/distributions"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "We need realistic but fake customer data to seed staging so QA can test the checkout flow — GDPR means we can't copy production.",
+      "expected_output": "Recognizes this as generating synthetic/test data, not validating production data — hands off to test-data-strategy (factories, synthetic generation, anonymization) rather than proposing expectations, contracts, or freshness checks",
+      "assertions": [
+        "Recognizes this is about generating test/synthetic data, not validating production data quality",
+        "Refers to the test-data-strategy skill for factories, synthetic data generation, and GDPR-safe anonymization",
+        "Does not answer with data-quality machinery (expectations, contracts, freshness checks) as the solution"
+      ]
+    }
+  ]
+}

package/skills/data-quality/references/tooling.md

@@ -0,0 +1,45 @@
+# Data Quality Tooling — Selection Notes
+
+Tool capabilities shift quickly; verify against current docs. The placement
+logic is stable.
+
+## Picking the enforcement layer
+
+| Tool | Shape | Choose when |
+|---|---|---|
+| **dbt tests** | YAML generic tests + singular SQL tests, run with the transforms | You're already on dbt. Zero new infrastructure; tests version with the models; severity/thresholds built in. The default. |
+| **dbt model contracts** | Enforced column names/types on a model; breaks the *producer's* build | A dbt model is consumed across team boundaries — the contract fails the producing project's CI, which is the shift-left goal. |
+| **Great Expectations** | Python expectation suites + data docs, runs anywhere | Checks must run outside the warehouse/dbt (files, in-flight DataFrames, non-dbt pipelines), or you need its rich expectation vocabulary and generated documentation. Heavier to operate. |
+| **Soda (Core/Cloud)** | Declarative checks YAML (SodaCL), CLI + SaaS | Non-engineers own quality rules, or checks span multiple stores with one syntax; Cloud adds anomaly detection and incident routing. |
+| **Elementary / observability platforms (Monte Carlo, Metaplane…)** | Monitor the warehouse from metadata: freshness, volume, schema drift, distribution anomalies | You want detection you didn't hand-write — anomaly monitoring across many tables — and (platforms) column-level lineage + incident management. Elementary is the dbt-native OSS entry point. |
+
+Layering that works in practice: **dbt tests + source freshness** as the
+foundation; **contracts** at team boundaries; **one anomaly/observability
+layer** for the unknown-unknowns. Adding a second tool in the same layer buys
+maintenance, not coverage.
+
+## Contract mechanics by boundary type
+
+- **Event streams**: schema registry (Avro/Protobuf/JSON Schema) with
+  compatibility mode set (backward at minimum); producers can't publish a
+  breaking schema. The strongest enforcement available — use it if events are
+  the interface.
+- **Warehouse-to-warehouse (dbt ↔ dbt)**: dbt model contracts + versioned
+  models (`v2` with a deprecation window) on the shared models.
+- **API/file drops from another org**: JSON Schema validated at ingestion;
+  reject-and-alert (dead-letter the bad batch) rather than best-effort
+  parsing.
+- The social half matters as much: a named producer contact, a change-notice
+  channel, and the contract in a repo both teams can PR.
+
+## Anomaly checks worth having before "ML-powered" anything
+
+- Row count vs trailing 7/28-day window (volume drops are the most common
+  silent failure).
+- Freshness vs declared SLA per source.
+- Schema diff vs last run (added/removed/retyped columns) with an explicit
+  allowlist flow.
+- Null-rate and distinct-count drift on business-critical columns.
+
+These four catch the large majority of real incidents; statistical/ML anomaly
+detection is a refinement, not a starting point.

package/skills/deployment-checklist/SKILL.md

@@ -69,7 +69,7 @@ Walk through each section. Check items by reading code, running commands, and ve
 #### Documentation
 - [ ] README updated if setup steps changed
 - [ ] API documentation updated for new/changed endpoints
-- [ ] CHANGELOG updated (if project uses one)
+- [ ] Release cut correctly, if this deploy ships one — version, changelog, tag consistent (see `release-management`)
 - [ ] Breaking changes are clearly communicated
 
 ### Step 3: Report
@@ -126,6 +126,7 @@ See [references/pre-deploy-gates.md](references/pre-deploy-gates.md) for environ
 
 ## Cross-Skill References
 
+- `release-management` — cutting the versioned release (semver, changelog, tag, publish) that this checklist deploys
 - `rollback-strategy` — design the rollback plan before deploying (required for High-risk changes)
 - `configuration-strategy` — verify all environment variables and feature flags are configured correctly
 - `incident-response` — use if the deployment causes a production incident

package/skills/dx-audit/SKILL.md

@@ -0,0 +1,97 @@
+---
+name: dx-audit
+description: "Audit and improve developer experience — inner-loop feedback (build/test/reload times), CI wait and queue times, flaky tests, local environment setup, onboarding time-to-first-PR, tooling friction, docs discoverability; produce a prioritized remediation plan. Triggers: developer experience, DX audit, devex, builds are slow, CI takes forever, flaky tests, onboarding takes weeks, tooling friction, developer productivity, engineers are frustrated. Code health/debt hotspots → technical-debt-review; designing one pipeline → cicd-pipeline."
+model: sonnet
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# DX Audit
+
+Find and fix the friction that taxes every engineer, every day. Developer
+experience compounds: a 10-minute CI queue on a 20-person team burns weeks of
+engineering time per month, and worse, it changes *behavior* — engineers batch
+changes, skip test runs, and avoid touching areas with slow feedback.
+Boundary: `technical-debt-review` audits the *code's* health; this skill audits
+the *workflow around* the code — loops, tooling, environments, and onboarding.
+
+## Workflow
+
+### Step 1: Measure the Loops — Numbers Before Anecdotes
+
+Quantify the feedback loops before collecting opinions, so the audit ranks by
+data and the improvement is provable later:
+
+- **Inner loop**: cold build, incremental build, single-test run, hot-reload —
+  time them yourself in the actual repo.
+- **Outer loop**: CI wall-clock (queue + run, from real recent pipelines), PR
+  time-to-first-review and time-to-merge, deploy lead time (DORA overlap —
+  `metrics-and-okrs`).
+- **Flake rate**: retries and "re-run job" clicks per week; a 5%-flaky suite
+  means engineers already ignore red.
+- **Onboarding**: time from laptop to first merged PR for recent joiners.
+
+### Step 2: Collect the Friction Log
+
+Then ask the engineers — a lightweight survey or five 15-minute conversations:
+"what wastes your time every day?", "what do you avoid doing because it's
+painful?", "what would you fix first?". The second question matters most:
+avoidance signals (never running the full suite locally, dreading a service's
+local setup, batching deploys) reveal costs the timers can't see. Recurring
+themes + Step-1 numbers = the finding list.
+
+### Step 3: Audit the Inner Loop
+
+The highest-frequency loop gets the deepest look: Is the dev environment
+reproducible in one command (or is setup a wiki page of drift-prone steps)?
+Can you run one test in seconds without the world? Does hot-reload actually
+work? Is the IDE experience functional (types, go-to-def, lint-on-save)? Are
+common tasks (reset DB, seed data, run service X) scripted or tribal
+knowledge? Every manual step here multiplies by invocations-per-day.
+
+### Step 4: Audit the Outer Loop
+
+- **CI**: queue time vs run time (capacity vs speed problems differ), cache
+  hit rates, test parallelization/sharding, whether PRs run only affected
+  targets — delegate pipeline redesign to `cicd-pipeline`.
+- **Flakes**: quarantine policy (a flaky test in the merge gate is a tax on
+  everyone), ownership for fixing, `test-suite-design` for structural causes.
+- **Review latency**: are PRs waiting on people or on process (required
+  approvals, codeowner bottlenecks)?
+- **Docs**: can an engineer answer "how do I X here?" without interrupting a
+  senior — README freshness, runbooks, `project-documentation`.
+
+### Step 5: Prioritize by Tax, Not by Annoyance
+
+Score each finding as **time-cost × frequency × people affected** (daily
+10-second cuts beat monthly 10-minute ones), weighted by behavior distortion
+(anything that makes engineers *skip verification* is top-tier regardless of
+seconds). Produce the remediation list: quick wins first (cache config, test
+sharding, a `make dev` script) to build credibility, structural items
+(environment overhaul, CI capacity) as scheduled work with owners.
+
+### Step 6: Prove It and Keep It
+
+Re-measure the Step-1 numbers after the fixes — DX work is uniquely easy to
+verify, so verify it. Then keep the loop honest: track 2–3 DX metrics
+continuously (CI p50/p95, flake rate, onboarding time), set a budget
+("CI > 10 min is a defect"), and re-run the friction survey a couple of times
+a year. DX decays by default — every new service adds setup steps and CI
+minutes unless something pushes back.
+
+## Principles Applied
+
+- **Frequency beats magnitude**: optimize the loop that runs 50×/day before
+  the one that runs weekly.
+- **Avoidance is the loudest signal**: what engineers won't do reveals more
+  than what they complain about.
+- **Measured before and after**: a DX improvement that can't show the numbers
+  moved is an opinion.
+
+## Cross-Skill References
+
+- `technical-debt-review` — the code-health complement to this workflow audit
+- `cicd-pipeline` — redesigning the pipeline the audit flagged
+- `test-suite-design` — structural fixes for slow or flaky suites
+- `project-documentation` — the docs/onboarding gaps surfaced in Step 4
+- `metrics-and-okrs` — DORA metrics and making DX goals measurable
+- `retrospective` — mining recurring friction from team retros

package/skills/dx-audit/evals/evals.json

@@ -0,0 +1,40 @@
+{
+  "skill_name": "dx-audit",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Our 25-person engineering team feels slow. CI takes ~35 minutes, people complain about local setup, and new hires seem to take forever to become productive. Run a DX audit and tell us what to fix.",
+      "expected_output": "Measures the loops first (inner-loop timings, CI queue vs run time, flake rate, PR cycle time, onboarding time-to-first-merge), collects a friction log from engineers with attention to avoidance behavior, audits inner loop (one-command env, single-test speed) and outer loop (CI caching/sharding, flake quarantine, review latency, docs), prioritizes by time-cost × frequency × people with quick wins first, and commits to re-measuring the same numbers after fixes",
+      "assertions": [
+        "Quantifies the feedback loops with real measurements (build/test times, CI queue vs run, flake rate, PR cycle, onboarding time) before prescribing fixes",
+        "Collects friction from the engineers themselves (survey/interviews) and pays attention to avoidance behavior, not just complaints",
+        "Audits both the inner loop (env setup, single-test runs, hot reload, scripted common tasks) and the outer loop (CI caching/parallelization, flaky-test policy, review latency, docs)",
+        "Prioritizes findings by frequency × time-cost × people affected, sequencing quick wins before structural work, with owners",
+        "Plans to re-measure the baseline numbers after remediation and keep 2–3 DX metrics tracked continuously"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Management says our DX is fine because nobody's complaining, but I notice engineers never run the full test suite locally, deploys pile up on Fridays, and everyone SSHes into a shared staging box to test things. Is there a DX problem here?",
+      "expected_output": "Reads the avoidance signals as the loudest DX evidence: skipping the local suite implies it's too slow/flaky (verification is being skipped — top-tier severity), deploy batching implies the deploy loop is painful or risky, shared-staging testing implies local environments can't reproduce reality; quantifies each to confirm, and explains why absence of complaints doesn't mean absence of tax (normalized friction becomes invisible)",
+      "assertions": [
+        "Identifies the described behaviors as avoidance signals that indicate DX problems despite the absence of complaints",
+        "Infers the likely underlying causes: local suite too slow or flaky, painful/risky deploy loop, local environments unable to reproduce reality",
+        "Flags that engineers skipping verification (not running tests locally) is especially severe because it distorts quality behavior",
+        "Proposes measuring to confirm (suite runtime, flake rate, deploy lead time, env setup) rather than acting on inference alone",
+        "Explains why normalized friction stops generating complaints (habituation), so surveys must ask about avoidance, not just satisfaction"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Our checkout module is a nightmare of spaghetti code — cyclomatic complexity through the roof, no tests, God classes everywhere. Audit it and give us a prioritized cleanup roadmap.",
+      "expected_output": "Recognizes this as a code-health/debt assessment, not a workflow-friction audit — refers to technical-debt-review for the hotspot analysis and remediation roadmap, stating the boundary: dx-audit covers the workflow around the code (loops, tooling, environments); technical-debt-review covers the code itself",
+      "assertions": [
+        "Recognizes the request as a code-health/technical-debt assessment, not a developer-workflow audit",
+        "Refers to the technical-debt-review skill for the hotspot analysis and cleanup roadmap",
+        "Does not produce a DX workflow audit (CI timings, onboarding, tooling friction) as the answer",
+        "States or implies the boundary: dx-audit audits the workflow around the code; technical-debt-review audits the code's health"
+      ]
+    }
+  ]
+}