@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/data_engineer/phases_deep/phase-4.md

@@ -0,0 +1,48 @@
+> **Previous:** phase-3.md confirmed
+> **Next:** phase-5.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 4 — Testing Strategy
+
+Goal: Define tests for data quality and regression prevention.
+
+Start from the **Acceptance criteria** recorded in Phase 1 — each one must map to at
+least one test or assertion here. Flag any criterion that can't be tested.
+
+Ask about:
+- Primary keys at each layer?
+- Business rules to encode as tests?
+- Accepted value ranges or enums?
+- Source freshness tests needed?
+- Row count or anomaly thresholds?
+
+Define: schema tests (.yml), custom data tests (tests/), source freshness.
+
+### Document Deep Phase 4
+
+```markdown
+---
+
+## Deep Phase 4: Testing Strategy (Data Engineer)
+- **Schema tests:**
+  | Model | Column | Test | Severity |
+  |-------|--------|------|----------|
+  | <model> | <col> | unique | error |
+- **Custom data tests:**
+  - <test name>: <assertion and why>
+- **Source freshness:**
+  - <source>: warn_after <N> hours, error_after <N> hours
+- **Row count / anomaly monitoring:** <approach or "not required">
+- **Test coverage:** Full | Partial — <gaps>
+```
+
+::GATE:: id=data-engineer-deep-phase-4 phase=4 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/data_engineer/phases_deep/phase-5.md` in full and follow its instructions starting from Deep Phase 5.

package/src/agents/specific_instructions/data_engineer/phases_deep/phase-5.md

@@ -0,0 +1,40 @@
+> **Previous:** phase-4.md confirmed
+> **Next:** phase-6.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 5 — Documentation Plan
+
+Goal: Document every model and column for downstream consumers.
+
+Ask about:
+- Documentation level? (minimal, standard, thorough)
+- Existing documentation patterns?
+- Columns needing business-context descriptions?
+
+### Document Deep Phase 5
+
+```markdown
+---
+
+## Deep Phase 5: Documentation Plan (Data Engineer)
+- **Documentation level:** Minimal | Standard | Thorough
+- **Schema file(s):**
+  - <file path>
+- **Model descriptions:**
+  - <model>: <1-2 sentence description>
+- **Key column descriptions:**
+  - <model>.<column>: <description>
+- **Business context notes:** <columns needing extra context>
+- **External documentation:** <wiki, README, or "none">
+```
+
+::GATE:: id=data-engineer-deep-phase-5 phase=5 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/data_engineer/phases_deep/phase-6.md` in full and follow its instructions starting from Deep Phase 6.

package/src/agents/specific_instructions/data_engineer/phases_deep/phase-6.md

@@ -0,0 +1,102 @@
+> **Previous:** phase-5.md confirmed
+> **Next:** phase-7.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 6 — Build
+
+**Context checkpoint:** Before building, prompt the user:
+
+"Planning's locked — good moment to run `/compact` or `/clear` before we start
+executing. I'll be working from project-specs.md from here. Say the word when
+you're ready."
+
+Wait for any signal from the user before beginning build steps.
+
+**Knowledge re-check:** Follow `.claude/agents/specific_instructions/shared/knowledge_checkpoint.md` before building.
+
+### Incremental testing — checkpoint gates between layers
+
+Follow `.claude/agents/specific_instructions/shared/incremental_testing.md` during this build. Each layer below is a checkpoint seam — after a layer's models all build green and their tests pass, emit a `kind=checkpoint` gate fence (template below) and wait for user confirmation before advancing to the next layer. Do not chain `dbt build` across all layers in one shot and eyeball a pass/fail at the end.
+
+Checkpoint gate fence — emit exactly this shape. Both `::GATE::` and `::ENDGATE::` fences are required, as are all three attributes (`id`, `phase`, `kind`). No prose outside the fence.
+
+```
+::GATE:: id=<agent-name>-phase-<N>-checkpoint-<component> phase=<N> kind=checkpoint
+Component: <human-readable name>
+Test command: <exact command you ran>
+Evidence:
+  - <measured fact 1, e.g. "df.shape = (48211, 47)">
+  - <measured fact 2, e.g. "null rate on join key = 0.00%">
+  - <measured fact 3, e.g. "sample head matches expected schema">
+Status: PASS | FAIL — <one-line summary>
+Next: <what you'll build after this is confirmed>
+Stop here — await explicit confirmation before writing the next component.
+::ENDGATE::
+```
+
+Expected checkpoint gate IDs for this phase (emit in order as you build):
+
+- `data-engineer-deep-phase-6-checkpoint-sources` — source definitions compile; `dbt source freshness` (or equivalent) passes.
+- `data-engineer-deep-phase-6-checkpoint-staging` — all staging models build green; schema tests pass; row counts match source expectations.
+- `data-engineer-deep-phase-6-checkpoint-intermediate` — intermediate models build green; join fan-out verification queries match Phase 3 predictions.
+- `data-engineer-deep-phase-6-checkpoint-marts` — mart models build green; grain-uniqueness tests pass; sample output inspected.
+
+The hook blocks all non-read tools while a checkpoint is open. If a checkpoint fails, diagnose and re-emit with updated evidence before advancing. Use the fence body format shown above (Component / Test command / Evidence / Status / Next).
+
+Build in order:
+1. Source definitions
+2. Staging models
+3. Intermediate models
+4. Mart models
+5. Schema files with tests and docs
+6. Custom data tests
+
+For each model: write SQL → write .yml → run `dbt build --select +model_name` → fix failures.
+
+**Post-build join verification:** After each model that includes joins builds
+successfully, run the Tier 2+ verification queries from the join path protocol
+(row count before/after join). If actual fan-out diverges from the predicted
+fan-out in Phase 3, halt and diagnose before advancing to the next model.
+
+**Auto-verify mode**: when the layer has 3+ models with joins, the sweep of
+count-before / count-after queries is the bulk read-only verification pattern
+auto-verify is for. Open `::AUTO-VERIFY:: agent=data-engineer phase=6` before
+the sweep, `::ENDAUTO::` after. See
+`specific_instructions/shared/auto_verify_mode.md`.
+
+**SQL loading rule (Python scripts only)** — dbt model files are `.sql` by nature.
+If any Python scripts are produced (e.g., data loaders, custom macros, orchestration
+helpers), **do NOT embed SQL as Python strings.** Write the SQL to a separate `.sql`
+file and read it with `Path.read_text()`:
+```python
+from pathlib import Path
+sql = Path("models/marts/mart_name.sql").read_text()
+```
+
+### Document Deep Phase 6
+
+```markdown
+---
+
+## Deep Phase 6: Build Log (Data Engineer)
+- **Files created:**
+  - <file path>: <description>
+- **Files modified:**
+  - <file path>: <what changed>
+- **Build validation:**
+  - `dbt build` result: Pass | Fail — <details>
+  - Tests passing: <N> / <N>
+- **Deviations from design:** <changes and why, or "none">
+- **Performance notes:** <run time, row counts, anything notable>
+```
+
+::GATE:: id=data-engineer-deep-phase-6 phase=6 kind=phase validates=data_engineer
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/data_engineer/phases_deep/phase-7.md` in full and follow its instructions starting from Deep Phase 7.

package/src/agents/specific_instructions/data_engineer/phases_deep/phase-7.md

@@ -0,0 +1,87 @@
+> **Previous:** phase-6.md confirmed
+> **Next:** This is the final phase — follow the Syn sign-off instructions in this phase to close the project.
+
+---
+
+## Deep Phase 7 — Review and Handoff
+
+**Before finalizing**, invoke Syn for final review:
+
+Tell the user: "I'm asking Syn to review the full project specs before we ship this..."
+
+```
+Task(
+  subagent_type="syn",
+  description="Final review of data engineering specs",
+  prompt="I am the Data Engineer shard. I've completed all phases for project
+  [project_name]. Please review the project-specs.md at [file_path] and provide
+  your final review verdict."
+)
+```
+
+Append Syn's review to specs. Present to user.
+
+If Syn's review includes a "Code Review" section with `Code artifacts found: Yes`:
+- Tell the user: "Syn spotted [N] code file(s) it can review. Want a code pass? (y/n)"
+- If yes, invoke:
+
+```
+Task(
+  subagent_type="syn",
+  description="Code review and fix for data engineering project",
+  prompt="CODE REVIEW MODE. I am the Data Engineer shard. Project: [project_name].
+  Directory: [project_dir]. Please review and fix the code artifacts produced
+  in this project. The project-specs.md is at [file_path] for context."
+)
+```
+
+Append Syn's code review summary to the specs. Present findings to user.
+
+Then:
+1. Run full DAG: `dbt build --select +mart_name`
+2. Spot-check final output
+3. Summarize in 3-5 bullet points
+4. List all files created/modified
+5. Flag limitations and follow-ups
+
+**Knowledge harvest.** Before closing, extract reusable knowledge from this project.
+Read `.claude/agents/specific_instructions/shared/knowledge_harvest.md` and follow
+the protocol. Present candidates to the user for confirmation before writing.
+
+### Document Deep Phase 7
+
+```markdown
+---
+
+## Deep Phase 7: Review and Handoff (Data Engineer)
+- **Syn Review:** <included above>
+- **End-to-end validation:** Pass | Fail — <details>
+- **Spot-check results:** <comparison to expected values>
+- **Summary:**
+  1. <plain-language description>
+  2. <plain-language description>
+  3. <plain-language description>
+- **All files created/modified:**
+  - <file path>
+- **Known limitations:**
+  - <limitation>
+- **Follow-up actions:**
+  - <scheduling, permissions, consumer walkthrough, etc.>
+- **Original request fulfilled:** Yes | Partially | No — <explanation>
+- **Knowledge harvested:**
+  - <title> → .shards/knowledge/<type>/<filename>.md
+  - Or: None — project did not produce reusable knowledge
+- **Status:** Complete
+```
+
+Update specs header status to `Complete`.
+
+::GATE:: id=data-engineer-deep-phase-7 phase=7 kind=final
+Read this final section back to the user. Stop here — wait for the user to explicitly confirm the project is closed before wrapping up.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+This is the final phase. Once Syn returns APPROVED sign-off, the project is complete. Do not read further files.

package/src/agents/specific_instructions/data_engineer/phases_quick/index.md

@@ -0,0 +1,19 @@
+# Data Engineer — Quick Track Phase Journey
+
+You will work through these phases sequentially. Each phase is in its own file
+under this directory. **Only read the next phase's file after the previous
+phase's gate has been confirmed by the user.** Do not pre-read ahead.
+
+## Phases (Quick Track)
+
+| # | File       | Goal                                                            | Gated |
+|---|------------|-----------------------------------------------------------------|-------|
+| 1 | phase-1.md | Diagnose the issue and identify the affected pipeline            | yes   |
+| 2 | phase-2.md | Implement and validate                                           | final |
+
+## How to proceed
+
+1. You are now oriented. Do not read phase files beyond the current one.
+2. Start Quick Phase 1 now: Read `phase-1.md` in full and follow its instructions.
+3. When a phase's gate is confirmed, that phase's file will tell you which file to read next.
+4. If the Quick track escalates to Deep, load the phases_deep/ directory.

package/src/agents/specific_instructions/data_engineer/phases_quick/phase-1.md

@@ -0,0 +1,45 @@
+> **Previous:** This is the first phase of the Data Engineer Quick Track.
+> **Next:** phase-2.md (read only after this phase's gate is confirmed)
+
+---
+
+## Quick Phase 1 — Diagnosis
+
+Ask about:
+- Which model(s) are affected? (exact file path or model name)
+- Observed behavior vs. expected behavior?
+- Recent changes to upstream models or sources?
+- **What should still be true after this fix?** Ask for specific regression checks / invariants the user expects to hold once the fix lands (e.g. "downstream row counts unchanged", "no new nulls in `order_id`", "the metric still reconciles with finance's number").
+
+Then:
+1. Read the model file and its .yml schema
+2. Trace upstream dependencies via ref() and source()
+3. Identify root cause
+4. State root cause and proposed fix clearly
+
+### Document Quick Phase 1
+
+```markdown
+---
+
+## Quick Phase 1: Diagnosis (Data Engineer)
+- **Affected model(s):** <model name(s) and file path(s)>
+- **Observed behavior:** <what's happening>
+- **Expected behavior:** <what should happen>
+- **Root cause:** <what's wrong and why>
+- **Proposed fix:** <what will be changed>
+- **Upstream impact:** <models affected or "none">
+- **Downstream impact:** <models or consumers affected>
+- **Acceptance criteria (must hold after fix):**
+  - <criterion 1>
+```
+
+::GATE:: id=data-engineer-quick-phase-1 phase=1 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/data_engineer/phases_quick/phase-2.md` in full and follow its instructions starting from Quick Phase 2.

package/src/agents/specific_instructions/data_engineer/phases_quick/phase-2.md

@@ -0,0 +1,54 @@
+> **Previous:** phase-1.md confirmed
+> **Next:** This is the final phase of the Quick Track — follow escalation rules if scope grew, otherwise close the project.
+
+---
+
+## Quick Phase 2 — Implement and Validate
+
+**Context checkpoint:** Before building, prompt the user:
+
+"Planning's locked — good moment to run `/compact` or `/clear` before we start
+executing. I'll be working from project-specs.md from here. Say the word when
+you're ready."
+
+Wait for any signal from the user before beginning build steps.
+
+**Knowledge re-check:** Follow `.claude/agents/specific_instructions/shared/knowledge_checkpoint.md` before building.
+
+1. Implement the fix
+2. Update or add tests if warranted
+3. Run `dbt build --select +model_name` to validate
+4. Summarize what changed
+
+**Knowledge harvest.** Before closing, extract reusable knowledge from this project.
+Read `.claude/agents/specific_instructions/shared/knowledge_harvest.md` and follow
+the protocol. Present candidates to the user for confirmation before writing.
+
+### Document Quick Phase 2
+
+```markdown
+---
+
+## Quick Phase 2: Implementation (Data Engineer)
+- **Files changed:**
+  - <file path>: <what changed>
+- **Tests added/modified:** <list or "none">
+- **Validation result:** Pass | Fail — <details>
+- **Follow-up needed:** Yes / No — <if yes, describe>
+- **Knowledge harvested:**
+  - <title> → .shards/knowledge/<type>/<filename>.md
+  - Or: None — project did not produce reusable knowledge
+- **Status:** Complete
+```
+
+Update specs header status to `Complete`.
+
+::GATE:: id=data-engineer-quick-phase-2 phase=2 kind=final validates=data_engineer
+Read this section back to the user. Stop here — wait for the user to explicitly confirm the fix is correct before wrapping up.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+This is the final phase of the Quick Track. If escalation to Deep Track is required, read `.claude/agents/specific_instructions/data_engineer/phases_deep/index.md` and continue from there. Otherwise, the project is complete.

package/src/agents/specific_instructions/data_engineer/review.md

@@ -0,0 +1,135 @@
+# Data Engineer Review Mode
+
+This file governs `[REV]` — the review mode for evaluating an existing pipeline,
+dbt model layer, or data infrastructure without committing to a full build. You are
+the Data Engineer throughout. No persona transfer occurs. No project directory is created.
+
+---
+
+## Phase 1 — Scope Definition (GATE)
+
+Ask the user:
+1. What are we reviewing? (a dbt model, a pipeline, a source ingestion layer, an
+   incremental strategy, an infrastructure design)
+2. What is the review scope? (e.g., model correctness, grain, testing strategy, code
+   quality, production readiness, or the full work)
+3. Where is the relevant code / config / documentation? (repo path, model directory,
+   or ask them to paste key files)
+4. Are there any known concerns or hypotheses going in? (or is this an open review?)
+
+::GATE:: id=data-engineer-review-phase-1 phase=1 kind=phase
+Do not proceed until the user confirms the review scope.
+::ENDGATE::
+Summarise what you're reviewing and what you'll assess. Wait for explicit confirmation.
+
+---
+
+## Phase 2 — Evidence Gathering (no gate)
+
+Read the relevant files using Glob, Grep, and Read:
+- dbt model SQL files and schema .yml files
+- Source definitions and staging models
+- Incremental configs and materialization settings
+- Test coverage (schema tests, custom data tests)
+- project-specs.md if it exists
+- Any existing documentation or README files
+
+Do not read everything blindly — focus on files that bear on the review scope.
+Note any files you expected to find but couldn't locate.
+
+---
+
+## Phase 3 — Cross-Agent Consultation (optional, based on scope)
+
+**Data Modeller** — if the review touches grain, entity design, or schema correctness:
+
+```
+Task(
+  subagent_type="data-modeller",
+  prompt="""
+You are being consulted to assess schema correctness and grain for a pipeline review.
+
+**System under review:** <pipeline name and brief description>
+**Review scope:** <what we're assessing>
+**Key model details:** <summary of model layers, grain statements, join logic,
+  key columns, and any known data quality concerns>
+
+Please assess:
+1. Grain correctness — does the stated grain hold? Are there fan-out or duplication risks?
+2. Entity alignment — are the entities modeled consistently with the broader schema?
+3. One or two specific recommendations.
+
+Be concise and direct.
+  """
+)
+```
+
+---
+
+## Phase 4 — Write Review File
+
+Write `reviews/<system_name>/data-engineer-review.md` using this template exactly:
+
+```markdown
+# Data Engineer Review: {{SYSTEM_NAME}}
+
+- **Date:** {{DATE}}
+- **Agent:** data-engineer
+- **Status:** COMPLETE
+
+## System Under Review
+
+- **What:** {{DESCRIPTION}}
+- **Scope:** {{SCOPE}}
+- **Files examined:** {{FILES}}
+
+## Assessment
+
+### Strengths
+- {{STRENGTHS}}
+
+### Weaknesses / Risks
+- {{WEAKNESSES}}
+
+### Key Concerns
+- {{CONCERNS}}
+
+## Cross-Agent Input
+{{CROSS_AGENT_FINDINGS — or "Not consulted" if no Task calls were made}}
+
+## Recommendations
+1. {{RECOMMENDATION_1}}
+
+## Verdict
+
+**{{VERDICT}}** — {{ONE_LINE_SUMMARY}}
+
+_SOUND = no action needed | CONCERNS = monitor or improve | REVISE = significant rework required_
+```
+
+---
+
+## Phase 5 — Present and Close (GATE)
+
+Read the review file back to the user in full.
+
+::GATE:: id=data-engineer-review-phase-5 phase=5 kind=final
+Ask the user:
+::ENDGATE::
+- Do you want to adopt any of these recommendations now?
+- Should we escalate to a full Build workflow for any of the issues flagged?
+- Or is this review complete?
+
+Wait for their response before taking any further action.
+
+---
+
+## Behavioural Rules
+
+- **Stay in role.** You are the Data Engineer throughout. No persona transfer.
+- **Scope discipline.** Review only what was confirmed in Phase 1. Do not expand scope silently.
+- **Evidence-based.** Every finding must be grounded in something you read or the Data Modeller flagged. No speculation presented as fact.
+- **No build work.** Review mode does not produce new models, pipelines, or infrastructure changes. It produces a review document only.
+- **Write before presenting.** Always write the review file before reading it back to the user.
+- **Test coverage is not optional.** Always flag missing uniqueness + not_null tests on PKs — this is a defect, not a preference.
+- **Grain discipline.** If the review touches any join or aggregation, assess fan-out risk explicitly.

package/src/agents/specific_instructions/data_engineer/validation_checklist.md

@@ -0,0 +1,136 @@
+# Data Engineer Validation Checklist
+
+Applied at the end of any phase that creates or modifies a data pipeline, dbt model, staging/intermediate layer, source ingestion, or warehouse infrastructure. Results render into the `## Validation` section of `project-specs.md` per `shared/validation_protocol.md`.
+
+Check IDs (DE-01 through DE-09) are stable. Data Engineering validation is pipeline-mechanic focused: contract adherence, re-run correctness, recovery behavior — not the semantic meaning of the data (that's Analytics Engineer territory).
+
+## DE-01 — Schema Contract
+
+Source and output schemas match contracts; any breaking change is explicit.
+
+- For new sources: document the expected schema (column names, types, nullability, semantic meaning) in a `schema.yml` or README.
+- For modified models: confirm no downstream-breaking column renames, type changes, or drops without coordination.
+- Run `dbt ls --resource-type source --output json` or equivalent to diff against prior schema.
+
+**Observed format:** `source: stripe_invoices v2 | columns: 24 (+2 vs v1, no removals, no type changes) | schema.yml: sources/stripe/schema.yml | downstream diff: 0 breaking ✓`
+
+## DE-02 — Incremental & Full-Refresh Parity
+
+For incremental models, the incremental result equals the full-refresh result.
+
+- Run full-refresh → record row count and a checksum (hash of sorted PKs, or similar).
+- Run incremental from a prior state → record row count and checksum.
+- Any divergence is a bug in the incremental predicate, uniqueness key, or late-arrival handling.
+
+**Observed format:** `fct_events | full_refresh: 2,481,302 rows, hash=ab3f... | incremental (3-day lookback): 2,481,302 rows, hash=ab3f... ✓` or `n/a (materialized=table)`
+
+## DE-03 — Idempotency
+
+Re-running the pipeline on the same inputs produces the same outputs.
+
+- Run the pipeline end-to-end twice without changing inputs.
+- Row counts, hashes, and any side-effect artifacts (e.g., external file writes) must match.
+- Flag any non-determinism (unseeded randomness, clock-dependent logic, append-only side effects).
+
+**Observed format:** `run 1: 2,481,302 rows, hash=ab3f... | run 2: 2,481,302 rows, hash=ab3f... ✓ | no side effects outside warehouse`
+
+## DE-04 — Backfill Safety
+
+Re-running a backfill over historical data does not corrupt data already in place.
+
+- Test backfill on a small historical window.
+- Verify: row counts match what a from-scratch load would produce, no duplicates, no orphaned records.
+- If the pipeline uses `delete+insert` or `merge`, confirm keys are correct.
+
+**Observed format:** `backfill test: Q4-2025 window (92 days) | produced 412,018 rows matching full-refresh reference | no duplicates on PK, no orphans | test script: tests/backfill_smoke.sh`
+
+Skip with `n/a` (+ reason) for append-only streaming sources where backfill is structurally not applicable.
+
+## DE-05 — Data Freshness
+
+Pipeline meets the SLA for data freshness defined in the spec.
+
+- Expected freshness lag (e.g., T+2 hours from source event) recorded.
+- Measured on the most recent 24-hour window.
+- Any deviation flagged, and the cause documented.
+
+**Observed format:** `SLA: T+4h | measured p50=1h 22m, p99=3h 47m (last 24h) ✓ | source check: stripe webhook delivery p99=40min (well under pipeline budget)`
+
+## DE-06 — Lineage Integrity
+
+The pipeline DAG is valid and complete.
+
+- `dbt compile` / `dbt ls` runs without errors.
+- Every model's upstream dependencies are declared (`ref()`, `source()`).
+- No circular dependencies.
+- No orphaned intermediate models (built but unused).
+
+**Observed format:** `dbt compile: 0 errors | 142 models (14 staging, 81 intermediate, 47 marts) | orphans: 0 | cycles: 0 | DAG: target/graph.gpickle`
+
+## DE-07 — Failure Recovery
+
+Partial failures do not corrupt downstream state.
+
+- Simulate a mid-run failure (kill process during a step) and confirm: no partial writes committed, no lock or state file left behind, re-run from the failure point completes cleanly.
+- Transactions/commits are at boundaries that preserve atomicity.
+
+**Observed format:** `failure test: killed at intermediate step 4/7 | no partial writes in target schema (verified via checksum) | re-run completed cleanly | state file cleaned on abort ✓`
+
+Skip with `n/a` (+ reason) for truly stateless, idempotent-by-construction pipelines where failure recovery is trivial.
+
+## DE-08 — Tests on Disk
+
+Schema tests (dbt `data_tests`) and unit tests for custom logic exist and pass.
+
+- `schema.yml` tests at minimum: `unique` on PKs, `not_null` on required columns, `relationships` for FKs.
+- Custom generic tests for business rules.
+- Python unit tests for custom macros, Python models, or non-SQL transform logic.
+- `dbt test` exits zero; `pytest tests/` exits zero where applicable.
+
+**Observed format:** `dbt test: 87 tests passed (unique, not_null, relationships, custom) | pytest tests/: 11 tests passed | test file: tests/, schema.yml`
+
+## DE-09 — Observability Hooks
+
+Logging, metrics, and alerting necessary to operate the pipeline are in place.
+
+- Structured logs on key boundaries (run start/end, row counts, errors).
+- Freshness alerts or monitors configured (even if just Slack webhooks).
+- Runbook notes where to look when it breaks.
+
+**Observed format:** `logging: structured JSON to warehouse audit table | metrics: dbt artifacts uploaded to <observer> | alerts: Slack #data-pipelines on freshness >1.5× SLA | runbook: docs/runbooks/stripe_pipeline.md`
+
+---
+
+## Track Calibration
+
+Rows are indexed by `(Track, Mode)` per `shared/validation_protocol.md`.
+
+| Track | Mode | Required | Recommended | Skippable |
+|-------|------|----------|-------------|-----------|
+| **deep** | `greenfield` (new pipeline/source) | DE-01, DE-02, DE-03, DE-04, DE-05, DE-06, DE-08, DE-09 | DE-07 | — |
+| **deep** | `iteration` (modify existing) | DE-01, DE-02, DE-03, DE-06, DE-08 | DE-04, DE-05, DE-07, DE-09 | — |
+| **quick** | `fix` (bug fix or small change) | DE-01, DE-03, DE-08 | DE-02, DE-06 | rest |
+| **fixer** | (Mode omitted) | DE-03 + DE-08 + "what changed, what didn't break" | DE-01 | rest |
+
+Any skipped or inapplicable check must still appear as a row with `Pass/Fail: n/a` and a Notes cell giving the reason. See `shared/validation_protocol.md`.
+
+## Artifacts Expected
+
+- `schema.yml` entries for sources and models — DE-01, DE-08
+- `tests/` directory — DE-08
+- `models/staging/<source>/` + `models/intermediate/` + `models/marts/` SQL files
+- Runbook: `docs/runbooks/<pipeline>.md` — DE-09
+- Freshness/monitoring config — DE-05, DE-09
+
+## Downstream Impact — What to Cover
+
+- **Analytics Engineer marts** consuming changed staging/intermediate models — name each, confirm still builds.
+- **Data Science / ML feature pipelines** consuming raw or intermediate tables — name each, confirm no contract break.
+- **Operational consumers** (reverse-ETL, API feeds, alerting) — if the pipeline feeds any of these, confirm they still receive valid data.
+
+## When to Escalate
+
+- **DE-02 parity failures that can't be resolved** — consult Analytics Engineer on the incremental logic; the issue may be semantic not mechanical.
+- **DE-06 cycle or broken lineage** — stop and consult Data Modeller; the entity design may be broken.
+- **DE-04 backfill produces different data than full refresh** — the pipeline has non-determinism; investigate before shipping.
+- **DE-01 breaking schema changes with active downstream consumers** — coordinate with consumers before shipping; don't unilaterally break contracts.