haiku-method 2.2.0

package/studios/data-pipeline/STUDIO.md

@@ -0,0 +1,20 @@
+---
+name: data-pipeline
+description: Data engineering lifecycle for ETL pipelines, data warehouses, and analytics workflows
+stages: [discovery, extraction, transformation, validation, deployment]
+category: engineering
+default_model: sonnet
+---
+
+# Data Pipeline Studio
+
+Lifecycle for data engineering work: ETL/ELT pipelines, data warehouse modeling,
+analytics workflows, and data integration projects. Use this studio when the intent
+involves moving, reshaping, or validating data across systems.
+
+Appropriate for:
+- Building new ETL/ELT pipelines from source to target
+- Data warehouse or lakehouse schema design and implementation
+- Analytics pipeline development (batch or streaming)
+- Data migration between systems or schema versions
+- Data quality framework implementation

package/studios/data-pipeline/examples/salesforce-to-warehouse-sync/intent.md

@@ -0,0 +1,18 @@
+---
+title: salesforce-to-warehouse-sync
+studio: data-pipeline
+stages: [discovery, extraction, transformation, validation, deployment]
+status: complete
+---
+
+# Salesforce → Snowflake Opportunity Sync
+
+Incremental 15-minute sync of Salesforce opportunity data into Snowflake. Land raw into `RAW.SFDC.OPPORTUNITY` and let the existing dbt graph take it from there. Incremental by `LastModifiedDate` — full snapshots previously exhausted the SFDC API quota.
+
+## Goals
+
+- Incremental extraction keyed on `LastModifiedDate` with high-water-mark checkpoint
+- Raw landing in Snowflake `RAW.SFDC.OPPORTUNITY` with audit columns
+- 15-minute orchestration cadence under the SFDC API rate limit
+- dbt handoff validated against the existing opportunity fact model
+- Zero data loss across run boundaries (verified by reconciliation job)

package/studios/data-pipeline/examples/salesforce-to-warehouse-sync/pre-intent.yml

@@ -0,0 +1,17 @@
+related:
+  - software
+  - documentation
+  - security-assessment
+turns:
+  - role: user
+    text: "Pipe Salesforce opportunity data into the warehouse every 15 minutes."
+  - role: agent
+    text: "Incremental by LastModifiedDate or full snapshot each run?"
+  - role: user
+    text: "Incremental. Full snapshots choked the API last time."
+  - role: agent
+    text: "Warehouse target — Snowflake, BigQuery, or something else?"
+  - role: user
+    text: "Snowflake. Land raw into RAW.SFDC then transform in dbt."
+  - role: agent
+    text: "Got it — incremental extract, Snowflake RAW landing, dbt downstream."

package/studios/data-pipeline/fix-hats/reconciler.md

@@ -0,0 +1,10 @@
+---
+agent_type: general-purpose
+---
+**Focus:** Reconcile cross-stage artifacts against studio-wide standards. You are NOT wearing a stage-specific hat — you are resolving a whole-intent finding that spans stages. Your mandate is alignment, not fresh authoring.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** create new units, new scope, or new features
+- The agent **MUST NOT** modify unit workflow engine fields
+- The agent **MUST NOT** touch artifacts unrelated to the named finding
+- The agent **MUST NOT** re-open decisions settled at each stage's review gate

package/studios/data-pipeline/fix-hats/validator.md

@@ -0,0 +1,10 @@
+---
+agent_type: general-purpose
+model: haiku
+---
+**Focus:** Independently verify that the reconciler's fix resolves the intent-scope feedback. You are the terminal hat — the workflow engine trusts your closure decision.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** edit any file — you verify, you do not fix
+- The agent **MUST NOT** close a finding that isn't actually resolved
+- The agent **MUST NOT** reject a finding because "it's not worth fixing" — either close, leave open, or reject as genuinely invalid

package/studios/data-pipeline/operations/backfill-procedure.md

@@ -0,0 +1,22 @@
+---
+name: backfill-procedure
+type: process
+owner: human
+frequency: "as-needed"
+---
+
+**Purpose:** Documented procedure for reprocessing historical data when transformations change or data corrections are needed.
+
+**Procedure:**
+- [ ] Identify the date range and entities requiring backfill
+- [ ] Verify sufficient compute resources for backfill volume
+- [ ] Disable downstream consumers or switch to backfill mode
+- [ ] Execute backfill pipeline with appropriate parameters
+- [ ] Verify row counts and data quality post-backfill
+- [ ] Re-enable downstream consumers
+- [ ] Validate downstream reports/dashboards reflect corrected data
+
+**Signals:**
+- Transformation logic change affecting historical data
+- Data quality issue discovered in historical records
+- Source system retroactive correction

package/studios/data-pipeline/operations/data-freshness-check.md

@@ -0,0 +1,20 @@
+---
+name: data-freshness-check
+type: scheduled
+owner: agent
+schedule: "0 */4 * * *"
+runtime: node
+---
+
+**Purpose:** Verify data is arriving on schedule. Stale data is often worse than no data.
+
+**Procedure:**
+- Check last-updated timestamps for all target tables
+- Compare against expected freshness SLAs
+- Verify row counts are within expected ranges
+- Flag tables that haven't updated within their SLA window
+
+**Signals:**
+- Every 4 hours
+- After source system maintenance windows
+- After pipeline infrastructure changes

package/studios/data-pipeline/operations/schema-drift-detection.md

@@ -0,0 +1,20 @@
+---
+name: schema-drift-detection
+type: reactive
+owner: agent
+trigger: "source schema change detected"
+runtime: node
+---
+
+**Purpose:** Detect and alert on source schema changes before they break extraction or transformation.
+
+**Procedure:**
+- Compare current source schemas against documented baselines
+- Identify added, removed, or modified columns
+- Assess impact on downstream transformations
+- Generate a change report with affected pipelines
+
+**Signals:**
+- Source system deployment or migration
+- Extraction errors indicating schema mismatch
+- New columns appearing in staging that aren't in the model

package/studios/data-pipeline/reflections/data-quality.md

@@ -0,0 +1,17 @@
+---
+name: data-quality
+studio: data-pipeline
+---
+
+**Analyze:** Validation pass rates, data quality check results, schema compliance, business rule violations.
+
+**Look for:**
+- Validation rules that failed frequently (misconfigured or genuine quality issues)
+- Data quality dimensions with the most violations (completeness, accuracy, consistency)
+- Source systems with the worst data quality
+- Whether quality improved through the transformation pipeline or degraded
+
+**Produce:**
+- Data quality scorecard by dimension and source
+- Validation rule effectiveness assessment
+- Recommendations for additional validation rules or source-side fixes

package/studios/data-pipeline/reflections/pipeline-reliability.md

@@ -0,0 +1,17 @@
+---
+name: pipeline-reliability
+studio: data-pipeline
+---
+
+**Analyze:** Pipeline run success rates, failure modes, recovery times, SLA adherence.
+
+**Look for:**
+- Most common failure modes (timeout, schema drift, resource exhaustion, source unavailable)
+- Mean time to recovery after failures
+- Whether retry policies were effective or masked systemic issues
+- SLA breaches and their root causes
+
+**Produce:**
+- Reliability metrics (success rate, MTTR, SLA adherence)
+- Failure mode taxonomy
+- Recommendations for retry policy, alerting, and infrastructure changes

package/studios/data-pipeline/review-agents/cross-stage-consistency.md

@@ -0,0 +1,17 @@
+---
+scope: intent
+interpretation: lens
+---
+**Mandate:** Verify the intent's artifacts are internally consistent across stages. You are the only reviewer that sees the whole intent at once — your job is to catch seams that per-stage reviewers miss.
+
+**Check:**
+- The agent **MUST** verify that each stage's outputs align with what upstream stages specified — no dropped requirements, no silent scope expansion
+- The agent **MUST** verify that naming is consistent across stages — a concept named one thing upstream should carry the same name downstream
+- The agent **MUST** verify that stages' declared outputs exist at the paths their unit frontmatter promised
+- The agent **MUST** verify that the stages collectively deliver the intent's stated goal (read `intent.md`) — partial delivery is a finding
+- The agent **MUST** verify that concerns raised by any stage's review were actually addressed (not silently ignored)
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** re-litigate decisions already approved at each stage's gate
+- The agent **MUST NOT** propose new features or scope additions
+- The agent **MUST NOT** flag stylistic preferences — concrete divergence only

package/studios/data-pipeline/stages/deployment/STAGE.md

@@ -0,0 +1,20 @@
+---
+name: deployment
+description: Deploy pipelines to production with monitoring and alerting
+hats: [pipeline-engineer, sre, verifier]
+fix_hats: [pipeline-engineer, feedback-assessor]
+review: external
+elaboration: autonomous
+inputs:
+  - stage: validation
+    discovery: validation-report
+review-agents-include:
+  - stage: transformation
+    agents: [data-quality]
+  - stage: validation
+    agents: [coverage]
+---
+
+# Deployment
+
+Deploy pipelines to production with monitoring and alerting.

package/studios/data-pipeline/stages/deployment/hats/feedback-assessor.md

@@ -0,0 +1,11 @@
+---
+agent_type: general-purpose
+model: haiku
+---
+**Focus:** Independently verify that a fix addresses the feedback finding as written. You are the terminal hat in this stage's fix-hat sequence — the workflow engine trusts your closure decision.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** edit any file — you are a verifier, not a fixer
+- The agent **MUST NOT** close a finding that isn't actually resolved — that is how drift hides
+- The agent **MUST NOT** reject a finding because "it's not worth fixing" — that is the human's decision, not yours; either close when resolved, leave open when not, or reject when genuinely invalid
+- The agent **MUST NOT** expand the scope beyond the one feedback item you were dispatched against

package/studios/data-pipeline/stages/deployment/hats/pipeline-engineer.md

@@ -0,0 +1,8 @@
+**Focus:** Package and deploy the pipeline to the production orchestrator. Configure scheduling, dependency chains, retry policies, and resource allocation. Ensure the pipeline runs reliably on the target infrastructure with proper logging and observability.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** deploy without configuring retries and timeout policies
+- The agent **MUST NOT** use hardcoded schedules without considering upstream dependency completion
+- The agent **MUST** set resource limits (memory, CPU, parallelism) for pipeline stages
+- The agent **MUST NOT** deploy to production without a rollback plan for the first run
+- The agent **MUST NOT** skip integration testing of the full DAG in a staging environment

package/studios/data-pipeline/stages/deployment/hats/sre.md

@@ -0,0 +1,8 @@
+**Focus:** Verify operational readiness — monitoring, alerting, runbooks, and incident response paths. Ensure the pipeline meets SLA commitments and that the team can diagnose and recover from failures without the original builder.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** approve deployment without verifying alert routing reaches the right on-call channel
+- The agent **MUST NOT** accept monitoring that covers only success cases, not failure and degradation modes
+- The agent **MUST** verify that runbooks are actionable by someone unfamiliar with the pipeline internals
+- The agent **MUST NOT** ignore data freshness monitoring in favor of only pipeline execution monitoring
+- The agent **MUST NOT** treat operational readiness as a checkbox rather than a genuine safety review

package/studios/data-pipeline/stages/deployment/hats/verifier.md

@@ -0,0 +1,26 @@
+**Focus:** Validate the per-unit operational artifact for the deployment stage of data-pipeline. Units here are deployment step — operational steps with concrete preconditions, actions, and post-condition checks. Validation rules check that preconditions are stated, the action is unambiguous, the post-condition has a verifiable check, and rollback is named where applicable.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** read or interpret unit frontmatter for any mechanical purpose. workflow engine territory per architecture §1.1.
+- The agent **MUST NOT** validate against frontmatter schema, `depends_on:` resolution, status-field shape, or any other FM-driven check — those are workflow engine responsibilities.
+- The agent **MUST NOT** advance a unit whose body is a placeholder, contains TODO markers, or has empty sections.
+- The agent **MUST NOT** reject for stylistic preferences. Substantive gaps only.
+- The agent **MUST** name a specific failed criterion in any rejection.
+- The agent **MUST NOT** invent rules not in this mandate. Stage scope is the contract.
+
+## What you check (BODY ONLY)
+
+### 1. Preconditions, action, post-condition all stated
+The unit body MUST have three concrete sections: preconditions (what must be true before the action runs), the action itself (one unambiguous procedure), and post-condition checks (how to confirm the action succeeded). Reject if any of the three is missing or vague.
+
+### 2. Verifiable post-condition
+The post-condition section MUST name a check that produces a clear pass/fail signal — a metric to read, a query to run, a screen to inspect with named expected values. "Verify by eye that things look good" is a reject.
+
+### 3. Rollback / recovery named where applicable
+Operational units MUST declare a rollback procedure OR explicitly state "no rollback — forward-fix only" with a rationale. Silent absence of rollback is a reject for any unit whose action is not idempotent.
+
+### 4. Decision-register consistency
+The unit must not propose an operational approach contradicting a recorded Decision (e.g., blue-green deploy when Decision N chose canary). Cite the Decision ID.
+
+### 5. Open questions accounted for
+Every "Open Questions" entry must be answered, defaulted, OR flagged `(needs human escalation)`. Operational open questions left to runtime are how outages happen.

package/studios/data-pipeline/stages/deployment/outputs/PIPELINE-CONFIG.md

@@ -0,0 +1,25 @@
+---
+name: pipeline-config
+location: (project source tree)
+scope: repo
+format: code
+required: true
+---
+
+# Pipeline Configuration
+
+Production pipeline deployment with monitoring, alerting, and runbook.
+
+## Expected Artifacts
+
+- **Pipeline DAG** -- registered in orchestrator with dependencies, retry policies, and SLA-based alerting
+- **Monitoring setup** -- runtime, row counts, data freshness, and error rate tracking with alert routing
+- **Runbook** -- manual recovery steps for the most likely failure modes
+- **Deployment verification** -- confirmation that pipeline runs successfully in production
+
+## Quality Signals
+
+- Pipeline DAG has correct dependencies and retry policies
+- Monitoring covers all critical pipeline health metrics
+- Runbook documents recovery for at least 3 failure scenarios
+- Alerts route to the correct on-call channel

package/studios/data-pipeline/stages/deployment/phases/ELABORATION.md

@@ -0,0 +1,16 @@
+# Deployment Stage — Elaboration
+
+## Criteria Guidance
+
+### Good criteria — concrete and verifiable
+
+- "Pipeline DAG is registered in the orchestrator with correct dependencies, retry policies, and SLA-based alerting"
+- "Monitoring covers pipeline runtime, row counts per stage, data freshness, and error rates with alerts routed to the on-call channel"
+- "Runbook documents manual recovery steps for the 3 most likely failure modes (source unavailable, schema drift, transformation timeout)"
+
+### Bad criteria — vague (no clear check)
+
+- "Pipeline is deployed"
+- "Monitoring is set up"
+- "Documentation exists"
+

package/studios/data-pipeline/stages/deployment/review-agents/reliability.md

@@ -0,0 +1,10 @@
+---
+interpretation: lens
+---
+**Mandate:** The agent **MUST** verify the deployed pipeline is resilient and observable in production.
+
+**Check:**
+- The agent **MUST** verify that failure recovery is defined: retry policies, dead-letter queues, alerting
+- The agent **MUST** verify that monitoring covers pipeline health, data freshness, and quality metrics
+- The agent **MUST** verify that backfill procedures exist for when historical data needs reprocessing
+- The agent **MUST** verify that resource sizing accounts for peak volumes, not just average load

package/studios/data-pipeline/stages/discovery/STAGE.md

@@ -0,0 +1,13 @@
+---
+name: discovery
+description: Understand data sources, schemas, volumes, and SLAs
+hats: [data-architect, schema-analyst, verifier]
+fix_hats: [data-architect, feedback-assessor]
+review: auto
+elaboration: collaborative
+inputs: []
+---
+
+# Discovery
+
+Understand data sources, schemas, volumes, and SLAs.

package/studios/data-pipeline/stages/discovery/hats/data-architect.md

@@ -0,0 +1,8 @@
+**Focus:** Map the data landscape — sources, targets, volumes, latency requirements, and system constraints. Define the high-level data flow architecture and identify integration patterns (batch, streaming, CDC) appropriate for each source-target pair.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** design the target schema before understanding source constraints
+- The agent **MUST NOT** assume all sources can support real-time extraction without verifying
+- The agent **MUST NOT** ignore volume growth projections and designing only for current scale
+- The agent **MUST NOT** skip SLA negotiation with source system owners
+- The agent **MUST NOT** treat all data sources as equally reliable or consistent

package/studios/data-pipeline/stages/discovery/hats/feedback-assessor.md

@@ -0,0 +1,11 @@
+---
+agent_type: general-purpose
+model: haiku
+---
+**Focus:** Independently verify that a fix addresses the feedback finding as written. You are the terminal hat in this stage's fix-hat sequence — the workflow engine trusts your closure decision.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** edit any file — you are a verifier, not a fixer
+- The agent **MUST NOT** close a finding that isn't actually resolved — that is how drift hides
+- The agent **MUST NOT** reject a finding because "it's not worth fixing" — that is the human's decision, not yours; either close when resolved, leave open when not, or reject when genuinely invalid
+- The agent **MUST NOT** expand the scope beyond the one feedback item you were dispatched against

package/studios/data-pipeline/stages/discovery/hats/schema-analyst.md

@@ -0,0 +1,8 @@
+**Focus:** Profile source schemas in detail — column types, nullability, cardinality, encoding, and semantic meaning. Identify type conflicts, naming inconsistencies, and data quality issues that will affect downstream transformation.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** accept schema documentation at face value without sampling actual data
+- The agent **MUST NOT** ignore edge cases in data types (e.g., timestamps without timezone, numeric precision loss)
+- The agent **MUST** profil for null rates, distinct counts, and value distributions
+- The agent **MUST NOT** treat schema discovery as a one-time activity rather than validating against live data
+- The agent **MUST NOT** miss implicit schemas in semi-structured sources (JSON, XML, CSV without headers)

package/studios/data-pipeline/stages/discovery/hats/verifier.md

@@ -0,0 +1,26 @@
+**Focus:** Validate the per-unit knowledge artifact for the discovery stage of data-pipeline. Units here are source-system knowledge artifact — knowledge artifacts that downstream stages consume. Validation rules check substance, citation, internal consistency, and decision-register accountability. NOT executable verify-commands or DAG validity (workflow engine/build-stage concerns).
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** read or interpret unit frontmatter for any mechanical purpose. workflow engine territory per architecture §1.1.
+- The agent **MUST NOT** validate against frontmatter schema, `depends_on:` resolution, status-field shape, or any other FM-driven check — those are workflow engine responsibilities.
+- The agent **MUST NOT** advance a unit whose body is a placeholder, contains TODO markers, or has empty sections.
+- The agent **MUST NOT** reject for stylistic preferences. Substantive gaps only.
+- The agent **MUST** name a specific failed criterion in any rejection.
+- The agent **MUST NOT** invent rules not in this mandate. Stage scope is the contract.
+
+## What you check (BODY ONLY)
+
+### 1. Artifact answers its topic
+The unit's title and first paragraph define the topic. The remaining body MUST deliver substantive content on that topic. Reject placeholders, content-free outlines, or redirects.
+
+### 2. Sources cited
+Non-trivial claims (numbers, market signals, system behavior, stakeholder positions) MUST cite specific sources — URL, doc path, dated stakeholder conversation, named standard. Reject "industry common knowledge" or unsourced numerical claims.
+
+### 3. Internal consistency
+Title, mission, and body must align. Numerical/categorical claims must be consistent across the body. Recommendations must follow from the evidence presented.
+
+### 4. Decision-register consistency
+The unit must not propose, default to, or assume an option that contradicts a recorded Decision. Cite the Decision ID in any rejection.
+
+### 5. Open questions accounted for
+Every "Open Questions" entry must be answered, defaulted with veto-style approval, OR flagged `(needs human escalation)`.

package/studios/data-pipeline/stages/discovery/outputs/SOURCE-CATALOG.md

@@ -0,0 +1,25 @@
+---
+name: source-catalog
+location: .haiku/intents/{intent-slug}/stages/discovery/artifacts/
+scope: intent
+format: text
+required: true
+---
+
+# Source Catalog
+
+Comprehensive inventory of data sources with schemas, volumes, and SLA requirements.
+
+## Expected Artifacts
+
+- **Source inventory** -- connection details, schema snapshots, volume estimates, and freshness requirements for every source
+- **Schema analysis** -- type conflicts, nullability patterns, and encoding issues across sources
+- **SLA targets** -- latency, completeness, and error tolerance defined per target table
+- **Data lineage map** -- source-to-target lineage for all intended data flows
+
+## Quality Signals
+
+- Every known source has connection details and schema documented
+- SLA requirements are captured with specific thresholds, not vague expectations
+- Schema analysis identifies all type mismatches and encoding inconsistencies
+- Data lineage is mapped from source through to intended target

package/studios/data-pipeline/stages/discovery/phases/ELABORATION.md

@@ -0,0 +1,16 @@
+# Discovery Stage — Elaboration
+
+## Criteria Guidance
+
+### Good criteria — concrete and verifiable
+
+- "Source catalog documents at least all known data sources with connection type, schema, and estimated row counts"
+- "SLA requirements are captured for each target table including freshness, completeness, and acceptable error rates"
+- "Schema analysis identifies all nullable fields, data type mismatches, and encoding inconsistencies across sources"
+
+### Bad criteria — vague (no clear check)
+
+- "Sources are documented"
+- "Schemas are understood"
+- "Requirements are gathered"
+

package/studios/data-pipeline/stages/discovery/review-agents/completeness.md

@@ -0,0 +1,10 @@
+---
+interpretation: lens
+---
+**Mandate:** The agent **MUST** verify all data sources, schemas, and constraints are documented.
+
+**Check:**
+- The agent **MUST** verify that every source system is inventoried with connection details and access patterns
+- The agent **MUST** verify that schema documentation includes all fields, types, nullability, and relationships
+- The agent **MUST** verify that volume estimates and SLAs are based on measurements, not guesses
+- The agent **MUST** verify that data quality issues in sources are documented, not discovered later

package/studios/data-pipeline/stages/extraction/STAGE.md

@@ -0,0 +1,15 @@
+---
+name: extraction
+description: Design and implement data extraction from sources
+hats: [extractor, connector-reviewer]
+fix_hats: [extractor, feedback-assessor]
+review: ask
+elaboration: autonomous
+inputs:
+  - stage: discovery
+    discovery: source-catalog
+---
+
+# Extraction
+
+Design and implement data extraction from sources.

package/studios/data-pipeline/stages/extraction/hats/connector-reviewer.md

@@ -0,0 +1,11 @@
+---
+interpretation: lens
+---
+**Focus:** Review extraction implementations for reliability, idempotency, and operational safety. Verify that connectors handle schema drift, network failures, and partial extractions without data loss or duplication.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** approve extraction logic without verifying idempotency (re-run safety)
+- The agent **MUST** test what happens when a source schema changes mid-extraction
+- The agent **MUST NOT** ignore partial failure scenarios (e.g., network timeout after 80% of records)
+- The agent **MUST NOT** treat retry logic as optional for "reliable" sources
+- The agent **MUST** verify that extraction metadata is sufficient for debugging production issues

package/studios/data-pipeline/stages/extraction/hats/extractor.md

@@ -0,0 +1,8 @@
+**Focus:** Implement extraction logic that reliably moves data from sources to the staging area. Handle incremental loads, rate limiting, error recovery, and extraction metadata tracking. Prioritize correctness and idempotency over speed.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** build only full-loadd extraction when incremental is feasible
+- The agent **MUST NOT** ignore source system rate limits or connection pool constraints
+- The agent **MUST NOT** silently drop records on extraction errors instead of dead-lettering
+- The agent **MUST** track extraction metadata (when, what, how much) for auditability
+- The agent **MUST NOT** hardcode connection strings or credentials instead of using config/secrets management

package/studios/data-pipeline/stages/extraction/hats/feedback-assessor.md

@@ -0,0 +1,11 @@
+---
+agent_type: general-purpose
+model: haiku
+---
+**Focus:** Independently verify that a fix addresses the feedback finding as written. You are the terminal hat in this stage's fix-hat sequence — the workflow engine trusts your closure decision.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** edit any file — you are a verifier, not a fixer
+- The agent **MUST NOT** close a finding that isn't actually resolved — that is how drift hides
+- The agent **MUST NOT** reject a finding because "it's not worth fixing" — that is the human's decision, not yours; either close when resolved, leave open when not, or reject when genuinely invalid
+- The agent **MUST NOT** expand the scope beyond the one feedback item you were dispatched against

package/studios/data-pipeline/stages/extraction/outputs/EXTRACTION-JOBS.md

@@ -0,0 +1,25 @@
+---
+name: extraction-jobs
+location: (project source tree)
+scope: repo
+format: code
+required: true
+---
+
+# Extraction Jobs
+
+Implemented extraction logic for all identified data sources.
+
+## Expected Artifacts
+
+- **Extraction scripts** -- jobs for each source handling full and incremental loads
+- **Error handling** -- retry logic with exponential backoff and dead-letter handling
+- **Schema drift detection** -- alerting for unexpected schema changes
+- **Staging output** -- raw data landed in staging area with extraction metadata (timestamp, source, batch ID)
+
+## Quality Signals
+
+- Extraction jobs exist for all sources identified in discovery
+- Each job handles both full and incremental loads
+- Jobs are idempotent and respect source system rate limits
+- Schema drift raises alerts rather than silently dropping columns

package/studios/data-pipeline/stages/extraction/phases/ELABORATION.md

@@ -0,0 +1,16 @@
+# Extraction Stage — Elaboration
+
+## Criteria Guidance
+
+### Good criteria — concrete and verifiable
+
+- "Extraction logic handles incremental loads using watermark columns identified in discovery"
+- "Connector includes retry logic with exponential backoff and dead-letter handling for failed records"
+- "Schema drift detection raises alerts rather than silently dropping or truncating columns"
+
+### Bad criteria — vague (no clear check)
+
+- "Extraction works"
+- "Data is pulled from sources"
+- "Connectors are configured"
+

package/studios/data-pipeline/stages/extraction/review-agents/correctness.md

@@ -0,0 +1,10 @@
+---
+interpretation: lens
+---
+**Mandate:** The agent **MUST** verify extraction logic faithfully captures source data without loss or corruption.
+
+**Check:**
+- The agent **MUST** verify that all fields from the source schema are accounted for (extracted or explicitly excluded with justification)
+- The agent **MUST** verify that incremental extraction handles late-arriving data and schema evolution
+- The agent **MUST** verify that error handling covers connection failures, timeouts, and malformed records
+- The agent **MUST** verify that extraction does not impose excessive load on source systems

package/studios/data-pipeline/stages/transformation/STAGE.md

@@ -0,0 +1,15 @@
+---
+name: transformation
+description: Transform and model data for the target schema
+hats: [transformer, data-modeler, verifier]
+fix_hats: [transformer, feedback-assessor]
+review: ask
+elaboration: autonomous
+inputs:
+  - stage: extraction
+    discovery: staged-data
+---
+
+# Transformation
+
+Transform and model data for the target schema.

package/studios/data-pipeline/stages/transformation/hats/data-modeler.md

@@ -0,0 +1,8 @@
+**Focus:** Design and validate the target data model — grain definitions, entity relationships, surrogate key strategies, and slowly changing dimension (SCD) types. Ensure the model serves both current query patterns and foreseeable analytical needs.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** define tables without explicitly stating the grain (one row per what?)
+- The agent **MUST NOT** use natural keys as primary keys without considering change scenarios
+- The agent **MUST NOT** over-normalize for OLTP patterns when the target is analytical (OLAP)
+- The agent **MUST** document SCD strategy per dimension (Type 1 overwrite vs Type 2 history)
+- The agent **MUST NOT** design the model without understanding the primary query access patterns

package/studios/data-pipeline/stages/transformation/hats/feedback-assessor.md

@@ -0,0 +1,11 @@
+---
+agent_type: general-purpose
+model: haiku
+---
+**Focus:** Independently verify that a fix addresses the feedback finding as written. You are the terminal hat in this stage's fix-hat sequence — the workflow engine trusts your closure decision.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** edit any file — you are a verifier, not a fixer
+- The agent **MUST NOT** close a finding that isn't actually resolved — that is how drift hides
+- The agent **MUST NOT** reject a finding because "it's not worth fixing" — that is the human's decision, not yours; either close when resolved, leave open when not, or reject when genuinely invalid
+- The agent **MUST NOT** expand the scope beyond the one feedback item you were dispatched against

package/studios/data-pipeline/stages/transformation/hats/transformer.md

@@ -0,0 +1,8 @@
+**Focus:** Implement transformation logic that converts raw staged data into the target schema. Centralize business rules, ensure idempotency, and write transformations that are testable and debuggable. Substance over cleverness — readable SQL/code beats terse one-liners.
+
+**Anti-patterns (RFC 2119):**
+- The agent **MUST NOT** scatter business logic across multiple transformations instead of centralizing
+- The agent **MUST NOT** write non-idempotent transformations that produce duplicates on re-run
+- The agent **MUST NOT** use opaque column aliases without documenting semantic meaning
+- The agent **MUST NOT** perform implicit type coercions without explicit CAST statements
+- The agent **MUST NOT** build deeply nested subqueries instead of named CTEs or intermediate models