@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/analytics_engineer/phases_deep/index.md

@@ -0,0 +1,24 @@
+# Analytics Engineer — Deep Track Phase Journey
+
+You will work through these phases sequentially. Each phase is in its own file
+under this directory. **Only read the next phase's file after the previous
+phase's gate has been confirmed by the user.** Do not pre-read ahead.
+
+## Phases (Deep Track)
+
+| # | File       | Goal                                                              | Gated |
+|---|------------|-------------------------------------------------------------------|-------|
+| 1 | phase-1.md | Business requirements — who, what, why                             | yes   |
+| 2 | phase-2.md | Source and staging assessment                                      | yes   |
+| 3 | phase-3.md | Grain and entity design                                            | yes   |
+| 4 | phase-4.md | Model layer architecture                                           | yes   |
+| 5 | phase-5.md | Testing strategy                                                   | yes   |
+| 6 | phase-6.md | Documentation plan                                                 | yes   |
+| 7 | phase-7.md | Build — implement the models, tests, docs                          | yes (validated) |
+| 8 | phase-8.md | Peer review and handoff                                            | final |
+
+## How to proceed
+
+1. You are now oriented. Do not read phase files beyond the current one.
+2. Start Deep Phase 1 now: Read `phase-1.md` in full and follow its instructions.
+3. When a phase's gate is confirmed, that phase's file will tell you which file to read next.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-1.md

@@ -0,0 +1,77 @@
+> **Previous:** This is the first phase of the Analytics Engineer Deep Track.
+> **Next:** phase-2.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 1 — Business Discovery
+
+Goal: Deepen understanding of what the user is building and what questions the data must answer — driven by their intent, not a checklist.
+
+Continue the discovery rhythm from Phase 0 — open by referencing what the user already said. See the Analytics Engineer section in `.claude/agents/specific_instructions/shared/intent_discovery.md` for your domain probes.
+
+Let the conversation flow. Surface these topics naturally when the user's responses lead there:
+- **Consumers:** analyst, dashboard, ML model, reverse ETL, finance report
+- **Grain:** probe for the right level of detail when it becomes relevant
+- **Edge cases / unknowns:** refunds, soft deletes, multi-currency — surface what the user knows about edge cases in their domain
+- **Acceptance criteria:** steer toward domain-specific invariants they expect to hold. Example (a mart measuring recommender effectiveness): "every treated user has ≥1 impression", "CTR ∈ [0,1]", "no user appears in both control and treatment", "row count matches the experiment assignment table".
+- **Where to look:** existing models, intake docs, upstream pipelines, stakeholders to consult
+
+**If the downstream consumer is a BI dashboard:** Note in Phase 4 (Model Layer Architecture) that aggregations and the date spine should be designed with dashboard query patterns in mind — pre-aggregated at the mart level where possible, date dimension at the right granularity for time-series charts, and dimension columns kept at manageable cardinality for filter dropdowns.
+
+**If the user references an `ae-intake.md` file:** Read that file. Check the
+`Originating agent` field to determine the source:
+
+- **If originating agent is "BI Engineer":** Pre-populate Phase 1 business
+  requirements — grain, downstream consumer, business questions, required
+  measures and dimensions, date spine, and refresh cadence. Set
+  `Downstream consumer: Dashboard (BI Engineer)`. Confirm pre-populated
+  values with the user before proceeding.
+
+- **If originating agent is "Data Analyst":** Pre-populate Phase 1 business
+  requirements — grain, business questions the mart must answer, required
+  measures, required dimensions, date spine, and update frequency. Set
+  `Downstream consumer: Direct analyst queries (Data Analyst)`. Also
+  populate analysis context fields (core question, filters, definition of
+  done) from the intake file. Confirm pre-populated values with the user
+  before proceeding. Do not re-ask questions already answered in the intake file.
+
+In both cases: if a required field is missing or unclear, ask only about
+the missing field — not the whole set.
+
+### Document Deep Phase 1
+
+```markdown
+---
+
+## Deep Phase 1: Business Requirements (Analytics Engineer)
+- **Consumer(s):** <who uses this and how>
+- **Downstream consumer:** Dashboard (BI Engineer) | ML feature store | Finance report | Direct analyst queries (Data Analyst) | Other: <describe>
+- **Intake file source:** Not applicable | BI Engineer — dashboards/<project_name>/ae-intake.md | Data Analyst — analysis/<project_name>/ae-intake.md
+- **Analysis context (DA intake only):** <core question from DA intake, or "N/A">
+- **Business questions this mart answers:**
+  - <question 1>
+  - <question 2>
+- **Edge cases / unknowns:** <domain-specific edge cases surfaced>
+- **Where to look:** <additional context sources identified>
+- **Required grain:** <one row per ___>
+- **Refresh cadence:** Real-time | Hourly | Daily | Weekly
+- **SLA / dependency:** <time constraint or "none">
+- **Replaces existing model:** Yes — <which> | No — net new
+- **Key business rules:**
+  - <rule 1: e.g., refunds reduce gross revenue>
+  - <rule 2>
+- **Acceptance criteria (what must be true):**
+  - <criterion 1 — e.g. one row per user-day, no gaps>
+  - <criterion 2 — e.g. CTR ∈ [0,1]>
+- **How each will be verified:** <test/query per criterion, or "designed in Phase 5">
+```
+
+::GATE:: id=analytics-engineer-deep-phase-1 phase=1 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-2.md` in full and follow its instructions starting from Deep Phase 2. Do not pre-read further phase files.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-2.md

@@ -0,0 +1,106 @@
+> **Previous:** phase-1.md confirmed
+> **Next:** phase-3.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 2 — Source and Staging Assessment
+
+Goal: Understand what staging models exist, what's missing, and whether the
+upstream data is sound enough to build on.
+
+Ask about:
+- Which source systems are involved?
+- Are staging models already defined? (stg_ prefix)
+- Known data quality issues upstream?
+- Existing intermediate models relevant to this work?
+- Incremental strategies already in use?
+
+Inspect the transformation project:
+- Glob: `**/*.sql` filtered to staging and intermediate paths; look for project config files
+- Grep for existing source reference calls to find defined sources
+- Check for source definition files (e.g., `sources.yml` or equivalent) for freshness configs
+
+**Consult Data Engineer as the first step of source assessment:**
+
+Tell the user: "Let me check with the Data Engineer shard on the staging layer before we design anything on top of it."
+
+```
+Task(
+  subagent_type="data-engineer",
+  description="Staging layer assessment for [project]",
+  prompt="I am the Analytics Engineer shard working on [project]. I need to assess
+  the staging layer before designing a transformation pipeline. Please explore and
+  return:
+  1. Which staging models exist for [source systems / entities]? List model names
+     and grain (one row per what).
+  2. Are there existing intermediate models I can reuse or build on?
+  3. Are source definitions present in sources.yml? Are freshness configs defined?
+  4. Any known data quality issues I should factor into my transformation design?
+     (late-arriving data, duplicates, schema drift, soft deletes)
+  5. What incremental strategies are already in use? Any patterns I should follow
+     for consistency?
+
+  Keep your response focused on source layer soundness and staging model inventory —
+  not the transformation logic I should build."
+)
+```
+
+**Greenfield handling:** Before proceeding, check whether the Data Engineer's
+response contains "NO DATA ENVIRONMENT DETECTED".
+
+If it does:
+1. Present the Data Engineer's response to the user.
+2. Ask:
+   "The Data Engineer found no existing staging models, source definitions, or
+   dbt project files. For transformation work, I need to know what we're building on:
+   - (a) Staged data exists — tell me the source system and whether it's already
+     modeled in staging. I'll design the transformation layer from there.
+   - (b) Raw data exists but no staging models yet — I'll need to flag that staging
+     models are a prerequisite. I can design them alongside the transformation layer,
+     but the Data Engineer should own the staging work.
+   - (c) No data exists yet — I can do contract-first design: define expected staging
+     models, intermediate shells, and mart stubs ready for when data arrives.
+     Nothing will run until staging data exists.
+   Which situation are we in?"
+3. Wait for the user's response before proceeding.
+   - (a): proceed with provided source info.
+   - (b): note that staging is a prerequisite; scope includes staging design but
+     implementation should be reviewed by Data Engineer.
+   - (c): set Data sufficiency: `Insufficient`, proceed as contract-first design.
+     All models will be flagged [THEORETICAL — UNTESTED].
+
+### Document Deep Phase 2
+
+```markdown
+---
+
+## Deep Phase 2: Source and Staging Assessment (Analytics Engineer)
+- **Source system(s):**
+  - <source>: <description, relevant tables or models>
+- **Existing staging models:**
+  - <stg_model> (<grain>): <brief description>
+  - (or "none found")
+- **Existing intermediate models relevant to this work:**
+  - <int_model>: <what it contains>
+  - (or "none")
+- **Source definitions:** Present | Missing — <details>
+- **Freshness configs:** Defined | Missing — <details>
+- **Known data quality issues:**
+  - <issue or "none identified">
+- **Incremental patterns in use:** <pattern or "none — full refresh only">
+- **Data Engineer consultation:** <summary of findings>
+- **Data sufficiency:** Sufficient | Partial | Insufficient
+- **Decision:** Proceed | Proceed with caveats | Blocked — <rationale>
+- **Data environment:** <not greenfield | Staging exists, transform layer missing | GREENFIELD — no transformation layer detected. Contract-first design only>
+```
+
+::GATE:: id=analytics-engineer-deep-phase-2 phase=2 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+**If Insufficient, do not proceed. Discuss alternatives.**
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-3.md` in full and follow its instructions starting from Deep Phase 3. Do not pre-read further phase files.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-3.md

@@ -0,0 +1,93 @@
+> **Previous:** phase-2.md confirmed
+> **Next:** phase-4.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 3 — Grain and Entity Design
+
+Goal: Define the grain of every model in the planned DAG, and validate entity
+relationships before writing a single CTE.
+
+"What does one row represent?" must be answered for every model before Phase 4.
+
+Ask about:
+- Confirmed grain for the target mart (from Phase 1 — verify it's still right)
+- Grain for each intermediate model needed
+- Many-to-many relationships that need bridge models?
+- Fan-out risks from any planned joins?
+- Conformed dimensions already in use elsewhere?
+
+**Consult Data Modeller for grain confirmation and entity design:**
+
+Tell the user: "Pulling in the Data Modeller — I need grain and entity validation before I commit to a model design."
+
+```
+Task(
+  subagent_type="data-modeller",
+  description="Grain and entity validation for [project] transformation design",
+  prompt="I am the Analytics Engineer shard designing a transformation pipeline
+  for [project]. I've identified the following planned models with their intended
+  grains:
+
+  [list each planned model with intended grain statement]
+
+  Business requirements context:
+  - Consumer(s): [from Phase 1]
+  - Key business questions: [from Phase 1]
+  - Required mart grain: [from Phase 1]
+  - Source staging models: [from Phase 2]
+
+  Please review and return:
+  1. Is the proposed grain correct for each model? Are there grain violations I
+     haven't anticipated?
+  2. Are there many-to-many relationship risks in the planned joins that could
+     cause fan-out? Which joins are highest risk?
+  3. Are there conformed dimensions already in the project I should use instead
+     of defining new ones?
+  4. Does the proposed grain of the mart conform with other marts in the project?
+     Any conformance conflicts?
+  5. Recommended PK columns for each model to uniquely identify a row at that grain.
+
+  Keep your response focused on grain correctness, M:M risks, entity conformance,
+  and PK recommendations — not the physical SQL design."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `data-modeller` row).
+
+### Document Deep Phase 3
+
+```markdown
+---
+
+## Deep Phase 3: Grain and Entity Design (Analytics Engineer)
+- **Confirmed model grains:**
+  | Model | Layer | Grain Statement | PK Column(s) |
+  |-------|-------|-----------------|--------------|
+  | <model> | staging | one row per <source_event_id> | <source_event_id> |
+  | <model> | intermediate | one row per <entity> per <period> | <surrogate_key> |
+  | <model> | mart | one row per <entity> | <entity_id> |
+- **Many-to-many relationship risks identified:**
+  - <risk or "none">
+- **Conformed dimensions in use:**
+  - <dimension model>: <shared across which marts>
+  - (or "none — new grain only")
+- **Data Modeller consultation:**
+  - Verdict: Sound | Concerns | Revise
+  - Tier: Proceed | Proceed with caveats | Halt
+  - Grain assessment: <summary>
+  - Fan-out risks flagged: <list or "none">
+  - Conformance notes: <list or "none">
+  - **Grain design revised:** Yes / No — <if yes, what changed>
+  - Reviewer resolution: Approved | Approved on resubmit | User override — <rationale> | Project stopped
+```
+
+::GATE:: id=analytics-engineer-deep-phase-3 phase=3 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-4.md` in full and follow its instructions starting from Deep Phase 4. Do not pre-read further phase files.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-4.md

@@ -0,0 +1,79 @@
+> **Previous:** phase-3.md confirmed
+> **Next:** phase-5.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 4 — Model Layer Architecture
+
+Goal: Design the full DAG — every model, its layer, materialization, incremental
+config, and key joins. No SQL yet, but the design is complete and confirmed.
+
+Present as a text DAG in chat:
+```
+[source: system_a] → [stg_a_entities] → [int_a_enriched]
+                                                 ↓
+[source: system_b] → [stg_b_events]  → [int_ab_joined] → [fct_target_mart]
+```
+
+**If UI-Aware Mode is active**, also push the DAG as an interactive Mermaid diagram to the browser. Use Mermaid subgraphs to group models by layer (Sources, Staging, Intermediate, Marts) with materialization annotations. Use a stable `--panel-id` (e.g., `dag-<project_name>`) so the DAG can be updated in place during Phase 7:
+```bash
+node .shards/ui/ui-push.js dag \
+  --title "DAG: <project_name>" \
+  --agent "analytics-engineer" \
+  --panel-id "dag-<project_name>" \
+  --data '<mermaid_syntax_string>'
+```
+
+For each model: model name, layer, grain (confirmed in Phase 3), materialization
+strategy, incremental config if applicable, key columns, key joins.
+
+Ask about:
+- Existing naming conventions to follow?
+- Materialization preferences? (views for staging, tables/incremental for marts)
+- Historical backfill needed?
+- Macros to create or reuse?
+
+Materialization guidance (state your reasoning, not just the choice):
+- Staging: view (lightweight, always fresh, no storage cost)
+- Intermediate: view unless large or expensive to compute — then table
+- Mart: table or incremental; incremental when >10M rows or expensive recomputation
+- Snapshots for SCD Type 2
+
+### Document Deep Phase 4
+
+```markdown
+---
+
+## Deep Phase 4: Model Layer Architecture (Analytics Engineer)
+- **DAG:**
+  ```
+  <text DAG diagram>
+  ```
+- **Models to create/modify:**
+  | Model | Layer | Grain | Materialization | New/Modified | File Path |
+  |-------|-------|-------|-----------------|--------------|-----------|
+  | <name> | staging | one per <x> | view | New | models/staging/<name>.sql |
+  | <name> | intermediate | one per <x> per <y> | view | New | models/intermediate/<name>.sql |
+  | <name> | marts | one per <x> | table | New | models/marts/<name>.sql |
+- **Incremental strategy (if applicable):**
+  - Model: <name>
+  - Unique key: <column(s)>
+  - Strategy: append | delete+insert | merge
+  - On schema change: append_new_columns | fail | sync_all_columns
+- **Key joins:**
+  - <model A> JOIN <model B> on <key>: <cardinality — 1:M, 1:1>
+- **Macros to create or reuse:**
+  - <macro_name>: <purpose or "none">
+- **Backfill approach:** full refresh | date-bounded — <details> | N/A
+- **Naming conventions confirmed:** Yes | No — <deviations from project standard>
+```
+
+::GATE:: id=analytics-engineer-deep-phase-4 phase=4 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-5.md` in full and follow its instructions starting from Deep Phase 5. Do not pre-read further phase files.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-5.md

@@ -0,0 +1,61 @@
+> **Previous:** phase-4.md confirmed
+> **Next:** phase-6.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 5 — Testing Strategy
+
+Goal: Define comprehensive test coverage for every model. No model ships untested.
+
+"An untested mart is a rumor, not a fact."
+
+Start from the **Acceptance criteria** recorded in Phase 1 — each one must map to at
+least one test or assertion here. Flag any criterion that can't be tested.
+
+For each model: schema tests (unique, not_null, accepted_values, relationships),
+singular tests, source freshness configs.
+
+Rules — non-negotiable:
+- Every PK gets `unique` + `not_null`. Every single one. No exceptions.
+- Every FK gets `not_null` and a `relationships` test where the referenced model exists.
+- Source freshness configs for every source definition used.
+- Accepted values tests for low-cardinality categorical columns.
+
+Ask about:
+- Business rules to encode as singular tests?
+- Accepted value ranges or enums for specific columns?
+- Row count or anomaly thresholds to monitor?
+- Severity levels — which failures should warn vs. error?
+
+### Document Deep Phase 5
+
+```markdown
+---
+
+## Deep Phase 5: Testing Strategy (Analytics Engineer)
+- **Schema tests:**
+  | Model | Column | Test | Severity |
+  |-------|--------|------|----------|
+  | <model> | <pk_col> | unique | error |
+  | <model> | <pk_col> | not_null | error |
+  | <model> | <fk_col> | not_null | error |
+  | <model> | <fk_col> | relationships(to=ref('<parent>'), field='<col>') | warn |
+  | <model> | <enum_col> | accepted_values(values=[...]) | warn |
+- **Singular tests:**
+  - <test_name>: <assertion, reason, and file path>
+  - (or "none required")
+- **Source freshness:**
+  - <source_name>.<table>: warn_after <N> hours, error_after <N> hours
+- **Row count / anomaly monitoring:** <approach or "not required at this stage">
+- **Test coverage:** Full | Partial — <gaps and rationale>
+```
+
+::GATE:: id=analytics-engineer-deep-phase-5 phase=5 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-6.md` in full and follow its instructions starting from Deep Phase 6. Do not pre-read further phase files.

package/src/agents/specific_instructions/analytics_engineer/phases_deep/phase-6.md

@@ -0,0 +1,45 @@
+> **Previous:** phase-5.md confirmed
+> **Next:** phase-7.md (read only after this phase's gate is confirmed)
+
+---
+
+## Deep Phase 6 — Documentation Plan
+
+Goal: Every model and non-obvious column gets a description.
+
+Ask about:
+- Documentation level? (minimal: model descriptions only; standard: model + key columns;
+  thorough: all columns + business context)
+- Existing documentation patterns in the project?
+- Columns with non-obvious business definitions that need explanation?
+- Business metrics or calculated fields that need formal definitions?
+
+### Document Deep Phase 6
+
+```markdown
+---
+
+## Deep Phase 6: Documentation Plan (Analytics Engineer)
+- **Documentation level:** Minimal | Standard | Thorough
+- **Schema file(s) to create/update:**
+  - <file path>
+- **Model descriptions:**
+  - <model>: <1-2 sentence description — what it represents, who consumes it>
+- **Key column descriptions:**
+  - <model>.<column>: <description — especially for non-obvious columns>
+- **Business metric definitions:**
+  - <metric_name>: <definition — e.g., "gross_revenue: sum of order amounts before
+    refunds, in USD, at time of capture">
+  - (or "none — no metrics layer in scope")
+- **External documentation:** <wiki, README, or "none">
+```
+
+::GATE:: id=analytics-engineer-deep-phase-6 phase=6 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/analytics_engineer/phases_deep/phase-7.md` in full and follow its instructions starting from Deep Phase 7. Do not pre-read further phase files.