swe-workflow-skills 0.1.0 → 0.2.0

package/skills/code-archaeology/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: code-archaeology
+description: "Understand unfamiliar or legacy code before changing it — map entry points, mine git history (blame, churn, log -S), trace data flow, recover the why behind odd code, pin current behavior with characterization tests, find seams for safe change. Triggers: understand this codebase, legacy code, inherited this project, what does this code do, why is it written this way, nobody knows how this works, onboard to this repo, is it safe to change this. Prioritizing debt → technical-debt-review; improving code you already understand → refactoring."
+model: sonnet
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# Code Archaeology
+
+Build a working model of code nobody fully understands — *before* changing it.
+The cardinal rule: **the code is the way it is for reasons; find them before
+overriding them.** Chesterton's fence applies — weird code is usually a bug fix
+whose bug report is lost, and "cleaning it up" reintroduces the bug. Boundary
+with siblings: `technical-debt-review` assesses and prioritizes what to fix;
+`refactoring` improves code you already understand; this skill produces the
+understanding both depend on.
+
+## Workflow
+
+### Step 1: Survey Before Digging
+
+Get the shape of the territory without reading line-by-line: entry points
+(main, routes, handlers, cron, consumers), the dependency skeleton (what are
+the load-bearing modules — imported by everything), build/run/test commands
+(does the test suite even pass? that's data), and the directory map with sizes.
+Resist starting at file one and reading forward — comprehension follows the
+call graph, not the alphabet.
+
+### Step 2: Mine the History — the Code's Memory
+
+Git knows what the docs forgot. The high-yield digs:
+
+- `git log --follow -p -- <file>` — a file's life story; the commit that
+  introduced the odd code usually says why (and links a ticket).
+- `git log -S '<string>'` (pickaxe) — when a magic value or weird guard
+  appeared, and what else changed with it.
+- **Churn analysis** — files with the most commits are the active organs
+  (`git log --format= --name-only | sort | uniq -c | sort -rn | head`);
+  high-churn + large = where understanding pays most (same hotspot logic as
+  `technical-debt-review`).
+- `git blame -w -C` (ignore whitespace, follow copies) — who last touched this
+  line *and in what commit context*; blame the commit, then read its message
+  and siblings.
+- Merge/PR references in messages — the review thread often holds the design
+  discussion that never made it into the code.
+
+### Step 3: Trace, Don't Guess
+
+For the specific behavior you need to understand, follow the data: pick one
+concrete input (a request, an event, a row) and trace it end to end — through
+handlers, transformations, side effects, and persistence. Use the debugger or
+strategic log statements over static reading when the control flow is dynamic
+(dispatch tables, DI containers, metaprogramming). Write the trace down as you
+go; a data-flow narrative ("the order enters here, gets enriched here, forks
+here") is worth ten class diagrams.
+
+### Step 4: Interrogate the Oddities
+
+For each "why on earth" you hit, run the checklist before concluding it's
+senseless: What does history say (Step 2)? Does a test encode it as intended
+behavior? Do comments/tickets/ADRs reference it? Does production data depend on
+it (that dead-looking branch may handle the 2019 records)? Only after all four
+come up empty may you *suspect* it's vestigial — and even then you prove it
+(logging/telemetry on the branch) rather than assume it.
+
+### Step 5: Pin Behavior with Characterization Tests
+
+Before changing anything you don't fully understand, write tests that capture
+what the code **currently does** — not what it should do. Feed it
+representative inputs (including the weird ones from production), assert the
+observed outputs, and lock in today's behavior as the baseline. These tests
+are your tripwire: if a "safe cleanup" changes an output, you learn it in CI,
+not in production. Golden-master/snapshot testing works well when outputs are
+large or numerous. (For designing the suite around them long-term:
+`test-suite-design`.)
+
+### Step 6: Leave the Map Better Than You Found It
+
+Record what you learned where the next archaeologist will find it: a short
+architecture note or README-in-the-directory (the data-flow narrative from
+Step 3, the load-bearing oddities from Step 4 and *why they exist*), backfilled
+ADRs for the big recovered decisions (`architecture-design`), and comments only
+on the genuinely non-obvious constraints. Then, with understanding and
+characterization tests in place, changes proceed via `refactoring` (seams,
+safe transformations) or `dependency-impact-analysis` (blast radius) as normal.
+
+## Principles Applied
+
+- **Chesterton's fence**: understand why the fence is there before removing
+  it; lost context is not absent context.
+- **Evidence over inference**: history, traces, and tests beat reading-and-
+  guessing — the code's actual behavior outranks anyone's model of it.
+- **Capture as you go**: understanding that lives in one head (or one session)
+  is re-excavated at full cost next time.
+
+## Cross-Skill References
+
+- `technical-debt-review` — assess and prioritize what to fix once understood
+- `refactoring` — the safe-change patterns applied after comprehension
+- `test-suite-design` — growing characterization tests into a real suite
+- `dependency-impact-analysis` — blast radius before changing a shared piece
+- `architecture-documentation` — recording the recovered architecture (C4, flows)
+- `bug-investigating` — when the goal narrows to one specific misbehavior

package/skills/code-archaeology/evals/evals.json

@@ -0,0 +1,41 @@
+{
+  "skill_name": "code-archaeology",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I just inherited a 8-year-old Django billing service. The original team is gone, there are no docs, and I need to change how proration is calculated. Help me understand it before I touch anything.",
+      "expected_output": "Surveys structure first (entry points, load-bearing modules, does the test suite pass), mines git history for the proration code specifically (log --follow, pickaxe for magic values, blame with commit context, churn hotspots), traces one concrete billing case end to end, interrogates oddities against history/tests/production data before judging them, writes characterization tests pinning current proration outputs before any change, and records the recovered understanding",
+      "assertions": [
+        "Surveys the codebase shape first (entry points, dependency skeleton, whether tests run) instead of reading files linearly",
+        "Uses git history as a primary source — git log --follow, pickaxe (log -S), or blame with commit context — to recover why the proration code is the way it is",
+        "Traces at least one concrete billing input end to end through the code rather than only reading statically",
+        "Writes characterization tests that pin the CURRENT proration behavior (including odd cases) before making any change",
+        "Treats odd code as potentially load-bearing (Chesterton's fence) — checks history, tests, and production data before deeming anything vestigial",
+        "Records the recovered understanding (data-flow narrative, architecture note, or backfilled ADR) for the next person"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "There's a bizarre guard clause in our payment processor: `if amount == 999999: amount = amount - 1`. No comment, no test, git blame just shows a 'fixes' commit from 2019 by someone who left. Everyone's afraid to delete it. Should I?",
+      "expected_output": "Does not say 'yes, delete it' outright: digs into the 2019 commit's full context (message, sibling changes, linked ticket, the PR discussion), searches for related production data or downstream systems that might depend on the cap, and if all sources come up empty, proposes proving it's dead (logging/telemetry on the branch for a period) plus a characterization test — removal only with evidence and a safety net, not on vibes",
+      "assertions": [
+        "Investigates the 2019 commit's full context (message, other files in the same commit, linked ticket/PR discussion) rather than accepting blame's surface result",
+        "Considers that production data or a downstream system may depend on the behavior before judging it vestigial",
+        "Proposes gathering runtime evidence (logging/telemetry on the branch) to prove the code path is dead rather than assuming it",
+        "Recommends a characterization test or equivalent safety net around the behavior before/when removing it",
+        "Does not recommend immediate deletion based on the code looking senseless"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Our codebase is a mess of tech debt — give me a prioritized assessment of what we should fix first and a remediation roadmap for the quarter.",
+      "expected_output": "Recognizes this as a debt assessment/prioritization request — refers to technical-debt-review for the hotspot analysis and remediation roadmap, noting the boundary: code-archaeology builds understanding of unfamiliar code; technical-debt-review assesses health and prioritizes fixes",
+      "assertions": [
+        "Recognizes the request as debt assessment and prioritization, not comprehension of unfamiliar code",
+        "Refers to the technical-debt-review skill for the hotspot analysis and roadmap",
+        "Does not produce an archaeology workflow (history mining, characterization tests) as the primary answer",
+        "States or implies the boundary: code-archaeology produces understanding; technical-debt-review assesses and prioritizes what to fix"
+      ]
+    }
+  ]
+}

package/skills/compliance-privacy/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: compliance-privacy
+description: "Engineer for regulatory and privacy obligations — GDPR/CCPA privacy-by-design, PII data mapping and minimization, retention and deletion (right to erasure), data subject requests, consent, SOC 2 controls (access, change management, audit logging). Triggers: GDPR, CCPA, SOC 2, HIPAA, compliance, privacy review, PII, personal data, data retention, right to be forgotten, DSR, audit requirements, are we compliant. Vulnerabilities in code → security-audit; attack analysis of a design → threat-modeling."
+model: opus
+allowed-tools: Read, Grep, Glob, Write, Edit, WebFetch, WebSearch
+---
+
+# Compliance & Privacy Engineering
+
+Turn regulatory obligations into engineering requirements — before an auditor,
+a regulator, or a deletion request does it for you. This skill covers the
+*engineering* of compliance (what to build and how); it is not legal advice, and
+material decisions (lawful basis, DPAs, breach notification) need counsel —
+flag them explicitly rather than silently deciding. Boundary with its siblings:
+`security-audit` finds technical vulnerabilities, `threat-modeling` analyzes
+attacker paths in a design; this skill handles what regulations *oblige* you to
+do with data regardless of attackers.
+
+## Workflow
+
+### Step 1: Map the Data — You Can't Comply with What You Can't Find
+
+Build the data inventory first; every obligation hangs off it:
+
+- **What personal data exists** — direct identifiers (name, email, phone),
+  indirect (IP, device IDs, location), sensitive categories (health, biometrics,
+  children's data — these carry stricter rules).
+- **Where it lives** — every store, including the ones people forget: logs,
+  analytics, backups, data warehouses, third-party processors, caches, error
+  trackers.
+- **Where it flows** — which services touch it, which vendors receive it
+  (each is a processor needing a DPA), and whether it crosses borders.
+
+Grep the codebase and schemas for it; the inventory that only reflects the
+architecture diagram misses the PII in logs and analytics events.
+
+### Step 2: Determine What Applies
+
+Which regimes govern you follows from the data and the users, not the company's
+location: EU/UK users → GDPR/UK-GDPR; California residents → CCPA/CPRA; health
+data in the US → HIPAA; card data → PCI-DSS; enterprise customers asking for
+audits → SOC 2. Verify specifics against current official sources — regulations
+and thresholds change; don't answer from memory. Then extract the engineering
+obligations (see [references/obligations-map.md](references/obligations-map.md)
+for the regulation → engineering-requirement mapping).
+
+### Step 3: Design for Minimization and Purpose
+
+The cheapest data to protect is data you don't have:
+
+- **Collect less**: challenge every field — what breaks if we don't collect it?
+- **Retain less**: every data class gets a retention period and an *automated*
+  deletion path; "keep forever by default" is a finding, not a policy.
+- **Expose less**: pseudonymize/tokenize where full identity isn't needed
+  (analytics, testing — see `test-data-strategy` for GDPR-safe test data);
+  scope access per purpose.
+- **Log less**: PII in logs inherits none of your database's controls and all
+  of its obligations — scrub at the logging layer, not in post-processing.
+
+### Step 4: Build the Rights Machinery
+
+Data-subject rights are engineering features with deadlines (typically ~30
+days), not support tickets to improvise:
+
+- **Access/export**: produce all of a person's data across every Step-1 store.
+- **Deletion**: erase or anonymize across primary stores, caches, search
+  indexes, analytics, and a documented stance on backups (usually: excluded
+  from immediate erasure, purged on backup expiry — state it in the policy).
+  Deletion that misses a store the inventory forgot is the classic failure.
+- **Consent**: record what was consented to, when, and which version; make
+  withdrawal as easy as granting; gate the relevant processing on it.
+
+Test the deletion path like a feature — run it against a seeded user and verify
+every store — because the first real request is a terrible time to discover it
+doesn't work.
+
+### Step 5: Implement the Control Layer (SOC 2 lens)
+
+The controls auditors verify are mostly good engineering hygiene made
+demonstrable: least-privilege access with a review cadence, change management
+(PRs + reviews + CI — which you likely have; the gap is usually *evidence*),
+append-only audit logging of sensitive-data access and admin actions
+(overlaps `threat-modeling`'s repudiation counters), encryption in transit and
+at rest, offboarding that provably revokes access, and vendor review for every
+processor in the Step-1 flow map. Wire evidence collection into the systems
+themselves — screenshots gathered the week before an audit don't scale.
+
+### Step 6: Make It Continuous
+
+Compliance decays with every feature that adds a data flow. Add a lightweight
+privacy check to the definition of done for features touching personal data
+(new data collected? new vendor? retention set? deletion path covers it?), keep
+the data inventory in version control next to the code, and re-run the Step-1
+mapping when architecture changes — same living-document discipline as a threat
+model.
+
+## Principles Applied
+
+- **Minimization is the master control**: every other obligation scales with
+  how much data you hold.
+- **Rights are features**: access, deletion, and consent need designs, tests,
+  and deadlines — not runbooks of manual SQL.
+- **Evidence or it didn't happen**: an undocumented control fails the audit
+  even when it works.
+
+## Cross-Skill References
+
+- `security-audit` — technical vulnerability review (confidentiality controls overlap)
+- `threat-modeling` — design-time attack analysis; shares the data-classification step
+- `data-modeling` — schema-level retention, soft-delete vs erasure, PII isolation
+- `test-data-strategy` — GDPR-safe test data, anonymization
+- `configuration-strategy` — secrets and access to production data
+- `observability-design` — keeping PII out of logs, traces, and metrics

package/skills/compliance-privacy/evals/evals.json

@@ -0,0 +1,41 @@
+{
+  "skill_name": "compliance-privacy",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We're a B2B SaaS with EU customers and we just got our first GDPR deletion request. We honestly don't know where all the user data lives. What do we need to build to handle this properly?",
+      "expected_output": "Starts with a data inventory across ALL stores (databases, logs, analytics, backups, third-party processors, caches), then designs deletion machinery: erasure/anonymization across every store, a documented stance on backups, the ~30-day deadline, and testing the deletion path against a seeded user; frames rights handling as an engineered feature and adds minimization/retention so the problem shrinks over time",
+      "assertions": [
+        "Begins with a data inventory/mapping across all stores — explicitly including the commonly-missed ones (logs, analytics, backups, third-party processors, caches)",
+        "Designs deletion as an engineered, testable feature spanning every store, not a manual SQL runbook against the primary database",
+        "Addresses backups explicitly with a documented stance (e.g. excluded from immediate erasure, purged on backup expiry)",
+        "Mentions the response deadline for data subject requests (about 30 days) and/or verifying the deletion path against a seeded test user before a real request",
+        "Recommends minimization and retention policies (automated deletion, collect less, PII out of logs) so future obligations shrink",
+        "Flags that material legal decisions (lawful basis, DPAs, breach specifics) need counsel rather than deciding them itself"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "An enterprise prospect just sent us a security questionnaire asking if we're SOC 2 compliant. We're a 10-person startup with decent engineering hygiene (PRs, CI, SSO) but zero formal compliance work. What does getting there actually involve on the engineering side?",
+      "expected_output": "Explains SOC 2 as an audit of controls operating over time (Type I vs II), maps their existing hygiene (PR review, CI, SSO) onto trust-criteria controls and identifies the real gap as demonstrable evidence — access reviews, offboarding proof, audit logging, vendor review — recommending evidence automation over screenshot scrambles, scoping to the Security criterion first, and being realistic about timeline/audit-window",
+      "assertions": [
+        "Explains that SOC 2 audits controls operating over time (Type II observation window), not a one-time checklist",
+        "Maps existing engineering hygiene (PR reviews, CI, SSO) onto SOC 2 controls and identifies evidence/demonstrability as the actual gap",
+        "Covers the core control areas: logical access (least privilege, access reviews, offboarding), change management, audit logging, and vendor management",
+        "Recommends starting with the Security criterion and adding other trust criteria only when customers demand them",
+        "Recommends automating evidence collection rather than gathering screenshots before the audit"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Can you review our Express API code for injection vulnerabilities and check whether our JWT auth implementation is secure?",
+      "expected_output": "Recognizes this as a technical vulnerability review of existing code — refers to security-audit for the injection/auth analysis, stating the boundary: compliance-privacy handles regulatory and privacy obligations (GDPR/SOC 2, data rights, retention); security-audit finds technical vulnerabilities",
+      "assertions": [
+        "Recognizes the request as a technical security review of code, not a regulatory/privacy obligation question",
+        "Refers to the security-audit skill for the injection and auth review",
+        "Does not produce a compliance workflow (data mapping, DSR machinery, SOC 2 controls) as the answer",
+        "States the boundary: compliance-privacy covers regulatory/privacy obligations; security-audit covers technical vulnerabilities"
+      ]
+    }
+  ]
+}

package/skills/compliance-privacy/references/obligations-map.md

@@ -0,0 +1,69 @@
+# Regulation → Engineering Obligation Map
+
+What each regime actually demands from the engineering side. Verify thresholds
+and details against current official sources before relying on them — this map
+orients; it doesn't replace the regulation text or counsel.
+
+## GDPR / UK-GDPR (EU/UK personal data)
+
+| Obligation | Engineering requirement |
+|---|---|
+| Lawful basis per processing purpose (Art. 6) | Record which basis covers each data flow; consent flows need capture + withdrawal machinery |
+| Data minimization & purpose limitation (Art. 5) | Field-level justification; no reuse of data for new purposes without a new basis |
+| Right of access / portability (Art. 15, 20) | Export of all of a subject's data, machine-readable, across every store |
+| Right to erasure (Art. 17) | Deletion/anonymization across primary stores, caches, indexes, analytics; documented backup stance |
+| Privacy by design & default (Art. 25) | Privacy review in the feature workflow; most-protective defaults |
+| Records of processing (Art. 30) | The data inventory (SKILL.md Step 1), kept current |
+| Security of processing (Art. 32) | Encryption in transit/at rest, access control, pseudonymization where feasible |
+| Breach notification (Art. 33) | Detection + escalation path that can meet 72-hour notification — needs `observability-design`-grade alerting |
+| Processors & transfers (Art. 28, Ch. V) | DPA with every vendor receiving personal data; transfer mechanism (SCCs/adequacy) for cross-border flows |
+| DPIA for high-risk processing (Art. 35) | Formal assessment before launching high-risk features (large-scale profiling, sensitive categories) — pairs with `threat-modeling` |
+
+## CCPA / CPRA (California residents)
+
+Similar rights machinery to GDPR (know/access, delete, correct) plus two deltas
+worth engineering attention: **opt-out of sale/sharing** (honor Global Privacy
+Control signals; maintain a "Do Not Sell or Share" path) and **service-provider
+contracts** (the CCPA analog of DPAs). Thresholds determine applicability
+(revenue / volume of consumers' data / share of revenue from selling data) —
+check current numbers.
+
+## SOC 2 (Trust Services Criteria — what enterprise customers ask for)
+
+SOC 2 is an audit of *your controls operating over time* (Type II), not a
+checklist you pass once. The Security criterion (always in scope):
+
+| Control area | Typical evidence engineering owns |
+|---|---|
+| Logical access | SSO/MFA, least-privilege roles, quarterly access reviews, provably-executed offboarding |
+| Change management | PR review required on protected branches, CI gates, deploy audit trail |
+| System operations | Monitoring/alerting, incident-response process with records (`incident-response`, `observability-design`) |
+| Risk & vendor management | Vendor list with security review per processor; risk assessment cadence |
+| Data handling | Encryption at rest/in transit, backup + restore testing (`resilience-engineering`), retention/disposal execution |
+
+Availability, Confidentiality, Processing Integrity, and Privacy criteria are
+opt-in scope — add them when customers demand them, not preemptively.
+
+## HIPAA (US health data)
+
+Applies to covered entities and their **business associates** (a BAA makes you
+one). Engineering core: PHI inventory and isolation, access controls + audit
+logs on every PHI touch, encryption, minimum-necessary access, breach
+notification machinery. PHI in logs/analytics is the recurring violation.
+
+## PCI-DSS (card data)
+
+The dominant engineering decision is **scope reduction**: use a tokenizing
+payment processor (Stripe/Adyen-style) so raw PANs never transit your systems —
+this collapses most requirements to the lowest self-assessment tier. Storing
+raw card data puts your whole environment in scope; treat that as a
+`build-vs-buy` decision with a strong buy bias.
+
+## Choosing what to tackle first
+
+1. Data inventory (everything depends on it).
+2. Kill the involuntary violations: PII in logs, forever-retention, missing
+   deletion path.
+3. Rights machinery (access + deletion) — deadline-bound once requests arrive.
+4. Evidence automation for the controls you already operate.
+5. Formal frameworks (SOC 2 audit, DPIAs) when customers or scale demand.

package/skills/data-pipeline-design/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: data-pipeline-design
+description: "Design batch and streaming data pipelines — ELT into a warehouse/lakehouse, dbt and analytics engineering (staging/intermediate/marts layers), orchestration with Airflow or Dagster, idempotent loads, incremental models, backfills, CDC ingestion. Triggers: data pipeline, ELT, ETL, dbt models, data warehouse, Snowflake, BigQuery, ingest into the warehouse, Airflow DAG, Dagster, Kafka streaming, backfill, incremental load, nightly job duplicates rows. ML training/feature pipelines → ml-pipeline-design; validating the data → data-quality."
+model: opus
+allowed-tools: Read, Grep, Glob, Write, Edit, Bash
+---
+
+# Data Pipeline Design
+
+Design pipelines that move and transform data for analytics — and stay correct
+when they fail, rerun, and backfill, because they will. The two design
+properties that separate a pipeline from a script: **idempotency** (any run can
+be repeated safely) and **parameterization** (any time window can be processed
+by the same code). Boundaries: pipelines feeding *model training* (features,
+point-in-time correctness) belong to `ml-pipeline-design`; *validating* the
+data this pipeline moves belongs to `data-quality`.
+
+## Workflow
+
+### Step 1: Map Sources, Consumers, and Freshness
+
+List each source (databases, SaaS APIs, event streams), each consumer
+(dashboards, reverse ETL, downstream teams), and the freshness each consumer
+actually needs. Freshness drives everything: daily batch is an order of
+magnitude cheaper to build and operate than streaming — don't buy streaming
+because "real-time" sounds better; buy it when a consumer decision genuinely
+changes within minutes.
+
+### Step 2: Choose ELT (and Say Why)
+
+For a cloud warehouse (Snowflake, BigQuery, Redshift, Databricks), default to
+**ELT**: land raw data unchanged, transform inside the warehouse with SQL/dbt.
+The reasons are operational, not fashionable — raw history means transform bugs
+are fixed by *re-running transforms*, not re-extracting from sources; the
+warehouse scales the compute; transformations become versioned, testable SQL.
+Classic ETL survives where data must be cleaned/redacted before it may land
+(PII, compliance) or the target isn't a warehouse.
+
+### Step 3: Don't Hand-Roll Extraction
+
+Ingestion is undifferentiated heavy lifting with long tail of pain (API
+pagination, rate limits, schema changes, backpressure):
+
+- **SaaS/DB sources**: managed or open connectors first — Fivetran/Airbyte/dlt.
+  Hand-write only when no connector exists or volume/cost forces it.
+- **Operational DBs at scale**: CDC (Debezium or the warehouse-native
+  equivalent) instead of daily full dumps or fragile `updated_at` queries.
+- **Events**: land the stream (Kafka → object storage/warehouse) raw; process
+  micro-batch unless Step 1 proved sub-minute freshness is needed.
+
+### Step 4: Layer the Transforms (dbt)
+
+- **Staging** — 1:1 with sources: rename, cast, deduplicate. No business logic.
+- **Intermediate** — joins and business logic, composable, not exposed to BI.
+- **Marts** — consumer-facing facts/dimensions, one per business domain.
+
+Each layer only reads from the one below; BI tools read only marts. This is
+SRP for SQL: when a number is wrong, the layer tells you where to look. Schema
+design for the marts themselves is `data-modeling`'s domain.
+
+### Step 5: Make Every Run Idempotent
+
+The Iron Rule of pipelines: **a rerun must produce the same result as one
+run.** Append-only loads fail this and produce the classic duplicate-rows-
+after-retry incident. Choose a write pattern per table:
+
+- **Delete + insert by partition** (or `INSERT OVERWRITE`): simplest, right
+  default for date-partitioned facts.
+- **Merge/upsert on a unique key**: for mutable entities and late updates —
+  requires a true unique key (enforce it via `data-quality` tests).
+- **Full refresh**: fine while small; flip to incremental on cost, not pride.
+
+In dbt: `incremental` materialization with `unique_key` set — and always define
+the full-refresh path (`--full-refresh` must work, or you can't recover from a
+logic bug).
+
+### Step 6: Parameterize Time — Backfill Is a First-Class Case
+
+Every run processes an explicit **logical window** passed in by the
+orchestrator (`WHERE event_date = {{ ds }}`) — never `CURRENT_DATE - 1` inside
+the SQL, which makes yesterday's failure unreproducible today. Then backfill is
+not a special script: it's the same job over a range, natively supported by
+the orchestrator (Airflow `backfill`, Dagster partitioned assets). If
+backfilling means hand-editing dates, Steps 5–6 were skipped.
+
+Handle **late-arriving data** explicitly: reprocess a trailing lookback window
+(e.g. the last 3 days) each run, or track a watermark — pick per source based
+on how late its data actually arrives.
+
+### Step 7: Orchestrate and Operate
+
+Pick **one** orchestrator and make a recommendation, not a brochure: Dagster
+for a dbt-centric greenfield stack (asset/partition model matches warehouse
+thinking, first-class dbt integration); Airflow where the org already runs it
+or needs its ecosystem breadth (comparison and managed options:
+[references/orchestration.md](references/orchestration.md)). Then wire in
+operations: retries with alerting on final failure (`observability-design`),
+quality checks between layers (`data-quality`), and cost visibility per model —
+warehouses make it very easy to spend quietly.
+
+## Principles Applied
+
+- **KISS**: batch over streaming, connectors over custom extractors, SQL over
+  a framework — until a measured requirement says otherwise.
+- **DRY**: one parameterized job for daily runs *and* backfills; logic lives
+  once, in the transform layer, not copied into extraction scripts.
+- **YAGNI**: no lakehouse/streaming/metadata-platform buildout for a stack
+  whose consumers refresh dashboards daily.
+
+## Cross-Skill References
+
+- `data-quality` — tests, contracts, and freshness checks between the layers
+- `data-modeling` — the schema design of the marts this pipeline produces
+- `ml-pipeline-design` — when the consumer is model training, not BI
+- `observability-design` — pipeline SLOs, alerting, run-level monitoring
+- `architecture-design` — ADRs for the costly-to-reverse platform choices

package/skills/data-pipeline-design/evals/evals.json

@@ -0,0 +1,41 @@
+{
+  "skill_name": "data-pipeline-design",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "We need to get data from Postgres and Stripe into Snowflake for BI dashboards. Design the pipeline — the team is considering dbt plus Airflow or Dagster.",
+      "expected_output": "Designs an ELT pipeline: extraction/connector choice for both sources, raw load into Snowflake, dbt layering (staging → intermediate → marts), a reasoned Airflow-vs-Dagster recommendation, idempotent incremental models, and a backfill story — not just the initial full load",
+      "assertions": [
+        "Recommends ELT (load raw, transform in-warehouse) and explains why it fits a Snowflake + dbt stack",
+        "Designs dbt layering: raw/staging models, intermediate transformations, and marts for BI consumption",
+        "Makes a reasoned orchestrator recommendation (Airflow vs Dagster) instead of listing both neutrally",
+        "Designs for idempotency: reruns must not duplicate or corrupt data (incremental models with unique keys, merge or delete+insert patterns)",
+        "Covers incremental loading and a backfill strategy (parameterized date ranges or partitions), not just the initial full load",
+        "Addresses extraction from the sources concretely (managed connector such as Fivetran/Airbyte/dlt, or CDC) rather than hand-waving ingestion"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Our nightly job loads yesterday's orders into the warehouse. When it fails midway and we rerun it, we get duplicate rows — and backfilling last month means manually editing the date in the script 30 times. Fix the design.",
+      "expected_output": "Identifies the non-idempotent append-only load as the root cause; converts the job to an idempotent write pattern (delete+insert by partition, merge/upsert on a key, or partition overwrite) parameterized over a logical date, so backfill becomes running the same job over a range via the orchestrator — with a lookback window for late-arriving data",
+      "assertions": [
+        "Identifies the root cause as a non-idempotent, append-only load",
+        "Proposes an idempotent write pattern: delete+insert by partition, merge/upsert on a unique key, or partition-overwrite semantics",
+        "Parameterizes the run over a logical date/window so any day can be run or re-run identically (backfill = the same job over a range)",
+        "Recommends orchestrator-native backfill (or an equivalent parameterized range run) over hand-edited scripts",
+        "Considers late-arriving or corrected data (lookback window, watermark, or reprocessing policy), not just the happy path"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Design the data pipeline that produces training features for our churn model — we need point-in-time correct features and a training set refreshed weekly.",
+      "expected_output": "Recognizes this as an ML training/feature pipeline, not analytics ELT — hands off to ml-pipeline-design, flagging point-in-time correctness/leakage as the ML-specific concern that a BI-oriented warehouse design does not address",
+      "assertions": [
+        "Recognizes this is an ML training/feature pipeline, not an analytics/BI ELT pipeline",
+        "Refers to the ml-pipeline-design skill for feature engineering and training orchestration",
+        "Flags point-in-time correctness / data leakage as the ML-specific concern that distinguishes this from BI work",
+        "Does not answer with a BI-oriented warehouse/dbt marts design as the primary solution"
+      ]
+    }
+  ]
+}

package/skills/data-pipeline-design/references/orchestration.md

@@ -0,0 +1,64 @@
+# Orchestration — Choosing and Using the Scheduler
+
+Versions and managed offerings change; check current docs before committing.
+The selection logic is stable.
+
+## Airflow vs Dagster vs the rest
+
+| | Airflow | Dagster |
+|---|---|---|
+| Mental model | Tasks in a DAG; scheduler runs tasks | Software-defined **assets** (tables/files) with partitions; runs materialize assets |
+| dbt integration | Via operators (Cosmos et al.) — workable | First-class: each dbt model becomes an asset with its own lineage/partitions |
+| Backfill | `airflow backfill` over execution dates | Partitioned assets — backfill = materialize missing partitions, UI-native |
+| Ecosystem | Enormous (every provider has an operator); huge hiring pool | Younger, smaller, growing; strong local dev experience |
+| Managed | MWAA (AWS), Cloud Composer (GCP), Astronomer | Dagster+ |
+
+Recommendation logic:
+
+- **Greenfield, warehouse/dbt-centric** → Dagster. The asset+partition model
+  *is* Steps 5–6 of the skill (idempotent, parameterized, backfillable) encoded
+  in the orchestrator; you fight the framework less.
+- **Org already runs Airflow, or the pipeline is mostly "call these seven
+  systems in order"** → Airflow. Operational familiarity and operator breadth
+  beat elegance; don't run two orchestrators for one team.
+- **All-in on one vendor stack** → the platform-native option (Databricks
+  Workflows, Snowflake Tasks) is acceptable glue; you trade portability.
+- **Cron + a queue** is fine for one or two jobs — adopt an orchestrator when
+  you need dependencies, backfill, and retry visibility, which is usually at
+  the third job.
+
+Whatever the choice: schedules trigger *logical windows*, tasks are
+idempotent, retries are bounded and alert on final failure, and secrets come
+from the platform's secret backend (`configuration-strategy`), never DAG code.
+
+## Ingestion tool notes
+
+- **Fivetran**: managed, broadest connector catalog, per-row pricing that gets
+  real at volume. Buy when engineer time is the scarce resource.
+- **Airbyte**: open-source + cloud; connector quality varies — test the
+  specific connectors you need, especially for incremental correctness.
+- **dlt**: Python library, connectors-as-code, great for API sources a
+  framework doesn't cover; you own the runtime.
+- **CDC (Debezium / native)**: when daily snapshots lose intermediate states
+  or full dumps are too heavy. Operationally nontrivial (snapshotting, schema
+  evolution, tombstones) — budget for it.
+
+## Streaming — only after Step 1 says so
+
+- Micro-batch (5–15 min) covers most "real-time" dashboard asks at batch-like
+  complexity; genuine streaming (Flink, Spark Structured Streaming,
+  Materialize) is for operational decisions made in seconds-to-minutes.
+- Streaming reintroduces every hard problem batch solved for you: exactly-once
+  vs at-least-once delivery, watermarks for late events, state management,
+  replay. Idempotent sinks (merge on key) make at-least-once livable — design
+  the sink first.
+- Keep the raw stream landed to cheap storage regardless; it's your backfill
+  and reprocessing escape hatch.
+
+## Warehouse cost hygiene
+
+- Tag/attribute compute per model or job from day one (dbt does this cheaply).
+- Incremental models exist for cost as much as latency — flip when a full
+  refresh's bill, not its runtime, annoys you.
+- Watch for accidental cross-joins and full scans in staging layers; the
+  warehouse will happily execute them nightly forever.