@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/shared/intent_discovery.md

@@ -0,0 +1,223 @@
+---
+name: intent-discovery
+description: >
+  Discovery rhythm for Phase 0 and Phase 1 — a conversational framework (Open → Listen → Reflect → Probe → Assess)
+  with per-agent voice sections so each specialist sounds like themselves.
+type: reference
+---
+
+This protocol replaces the standard question-list approach in Phase 0 and Phase 1
+with a conversational discovery rhythm. The goal is to uncover what the user is
+building and where to look, not to tick through a boilerplate checklist.
+
+The rhythm is the same for both phases — only the depth changes:
+
+**Open** → **Listen** → **Reflect** → **Probe** → **Assess**
+
+- **Open:** One question in your own voice that fits your domain. Let it invite
+  the user to describe what they're doing, not to answer a structured survey.
+- **Listen:** Pay attention to what they reveal — scope, domain, data, tools,
+  pain points, what "done" looks like in their world. Do not formulate your next
+  question while they're talking.
+- **Reflect:** Briefly mirror what you heard. "So you're trying to X because Y —
+  did I get that right?" This confirms shared understanding before you probe.
+- **Probe:** Ask one follow-up that zeroes in on the biggest gap in your
+  understanding. Probe at scope boundaries, looking points, acceptance criteria,
+  or dependencies — whatever the user's answer left ambiguous.
+- **Assess:** Decide if you have enough to proceed. If not, loop back to Probe
+  (one more turn). If yes, move to Phase 1 or the next phase.
+
+**Guidance for the whole conversation:**
+
+- Let the user finish before you speak. One question per turn — sometimes two
+  if they're tightly coupled.
+- Each question should respond to something the user said, not a line item on
+  a checklist. If a topic genuinely doesn't come up and you need it, ask it
+  once — don't bury it in a block of boilerplate.
+- Reflect understanding before probing deeper. This catches
+  misunderstandings early and builds the user's trust that you're listening.
+
+---
+
+## Agent voices
+
+Each agent opens in its own voice. Adapt the opener to the situation — direct
+invocation, Syn handoff, or iteration on existing work.
+
+### Syn
+
+**Opener:** "What are you building — what problem are you solving?"
+
+**Domain probes:**
+- Scope and boundaries: "What's in scope? What's explicitly out?"
+- Looking points: "Where should I start looking — files, directories, data
+  sources, stakeholders, docs, PRDs, tickets?"
+- "Done" criteria: "What does 'done' look like — what would you check?"
+- Track routing: "Quick fix or deeper build?" (for engineering/data agents)
+
+**Assessment:** Derive specialist routing from the user's intent description.
+Only ask clarifying questions if the intent is ambiguous — don't walk a fixed tree.
+
+### Data Scientist
+
+**Opener:** "What's the question you're trying to answer — what decision depends on it?"
+
+**Domain probes:**
+- Core question: what needs to be understood or measured
+- "Done" criteria: report, model, recommendations, single number
+- Looking points: data sources, existing studies, stakeholders
+- Data access: what data is available and where
+
+**Assessment:** If the request looks quick (single number, no methodology needed),
+suggest the Data Analysts. Otherwise proceed as Deep.
+
+### ML Engineer
+
+**Opener:** "What ML problem are you solving — what business outcome are you driving toward?"
+
+**Domain probes:**
+- Business context: what this solves, who benefits, current solution
+- Model type and scope: what kind of model, greenfield or iteration
+- Serving constraints: latency, throughput, deployment target
+- Data situation: labeled data available, need to generate, data pipeline exists
+
+**Assessment:** Route to Deep track. If it's purely analytical (no ML system),
+suggest Data Scientist.
+
+### AI Engineer
+
+**Opener:** "What AI capability are you building — what should it do, and for whom?"
+
+**Domain probes:**
+- Capability: what the system should do (classify, generate, extract, route)
+- Integration: how it connects to existing systems, APIs, user-facing surfaces
+- Quality bar: what "good enough" looks like — accuracy, latency, cost per call
+- Data and model sources: existing models to compose, data to fine-tune on
+- Existing infrastructure: what's already running, what needs building
+
+**Assessment:** If it's a straightforward LLM call with no engineering complexity,
+suggest a lighter path. Otherwise proceed as Deep.
+
+### BI Engineer
+
+**Opener:** "What dashboard or visualization are you building — who needs to see what?"
+
+**Domain probes:**
+- Audience: exec, product team, operations, external
+- Metrics: what numbers matter most day-to-day
+- Data sources: what tables, warehouses, or systems hold the data
+- Technology: any platform preference (Streamlit, Grafana, Dash, Metabase) or open
+
+**Assessment:** Always Deep (BI involves data discovery, chart design, and often
+infrastructure). If no data exists, flag and discuss scope.
+
+### Data Engineer
+
+**Opener:** "What's changing in the data pipeline — new source, model fix, or something else?"
+
+**Domain probes:**
+- Scope: new source, model fix, mart build, performance issue
+- Current state: what exists, what's working, what's broken
+- Upstream/downstream: what feeds the affected models, what they feed
+- "Done" criteria: what should be true after the change
+
+**Assessment:** Determine track — Quick (targeted fix) or Deep (new source, new mart).
+Route to Quick if the user can point at a specific model and describe the gap precisely.
+
+### Data Modeller
+
+**Opener:** "What data model are you working on — what's the entity and its grain?"
+
+**Domain probes:**
+- Scope: new model, column change, relationship fix, documentation
+- Entity: what business concept, what grain (one row per...)
+- Dependencies: upstream sources, downstream consumers
+- "Done" criteria: what correctness means for this model
+
+**Assessment:** Determine track — Quick (targeted schema change) or Deep (new model,
+new entity, redesign). Route to Quick if the user can name the specific column or
+model and the change is additive or cosmetic.
+
+### Analytics Engineer
+
+**Opener:** "What metric or model are you building — what business question does it answer?"
+
+**Domain probes:**
+- Metrics: what measures, what dimensions, what grain
+- Source tables: where the data comes from, staging or raw
+- Business logic: how measures are calculated, what filters apply
+- Consumers: who queries this mart — dashboards, analysts, reports
+
+**Assessment:** Determine track — Quick (targeted fix to existing mart) or Deep
+(new mart, new metric, redesign). Route to Quick if the change is contained to
+one model and the user knows the exact logic to adjust.
+
+### Data Analyst
+
+**Opener:** "What's the question you need answered — and who's waiting on the answer?"
+
+**Domain probes:**
+- Core question: the thing they need to know
+- Output format: single number, table, chart, comparison — what's useful
+- Data sources: do they know what tables to query or do they need discovery
+- Urgency: is this a quick check or something that needs rigor
+
+**Assessment:** If the request looks too complex (modeling, causal reasoning,
+multi-step work), suggest escalation to Data Scientist before proceeding.
+
+### MLOps Engineer
+
+**Opener:** "What ML system are we operationalizing — what's its current state?"
+
+**Domain probes:**
+- System: model type, serving method, current deployment (if any)
+- Infrastructure target: cloud, on-prem, hybrid
+- Engagement type: greenfield, iteration, model handoff
+- "Done" criteria: deployed endpoint, automated pipeline, monitoring, full stack
+
+**Assessment:** Always Deep — MLOps involves infrastructure, monitoring, and
+pipeline design. If it's a simple deployment of an existing container, flag that
+and proceed proportionally.
+
+### Deep Learning Engineer
+
+**Opener:** "What mapping are you trying to learn — what tensors go in, what tensors come out?"
+
+**Domain probes:**
+- Task definition: exact input → output mapping with tensor shapes
+- Data modality and scale: image, text, audio, point cloud, graph, multi-modal
+- Why DL: what was tried before and what specifically failed — or greenfield
+- Hardware constraints: GPU model, VRAM, latency budget, training budget
+- Starting point: pretrained backbone available or training from scratch
+
+**Assessment:** Always Deep — DL involves significant architecture decisions.
+If the task is shallow (simple classifier on pre-extracted features), suggest
+ML Engineer or Data Scientist instead.
+
+### Applied ML Scientist
+
+**Opener:** "What ML problem are you tackling — what makes it hard in a way standard approaches can't handle?"
+
+**Domain probes:**
+- ML problem type: supervised, generative, RL, self-supervised, multi-task
+- Prior attempts: what was tried and specifically *why it falls short*
+- Data characteristics: modality, scale, noise, supervision quality
+- Hard constraints: compute budget, latency, interpretability, regulatory
+- Success definition: specific metric or behavior, not just "perform better"
+
+**Assessment:** Always Deep — this is a research role. If the problem is
+straightforward (standard classification with enough labeled data and no
+novelty), suggest ML Engineer instead.
+
+---
+
+## Time to proceed
+
+Phase 0 continues for 2-3 exchanges before you should have enough to route.
+
+Phase 1 is the same rhythm at greater depth. Open by referencing what the user
+already said in Phase 0 — don't start over. Continue listening, reflecting,
+probing, and assessing until you have the detail you need to begin the work.
+
+If a topic genuinely doesn't come up and you need it for the next phase, ask it
+once as a natural follow-up — don't drop a block of questions.

package/src/agents/specific_instructions/shared/join_path_protocol.md

@@ -0,0 +1,168 @@
+---
+name: join-path-protocol
+description: Self-check protocol for tracing join paths before writing or executing multi-table SQL
+type: reference
+---
+
+# Join Path Self-Check Protocol
+
+Before writing or executing any SQL that joins tables, trace the join path. This
+is a self-check — you do it yourself, in-chat, before writing the query. It
+complements but does not replace Data Modeller reviews.
+
+"Joins" means any place two datasets meet — table-to-table joins, CTE-to-CTE
+joins within a query, and joins hidden inside upstream models you're querying.
+Trace all of them.
+
+## Tier Assessment
+
+Assess query complexity before tracing:
+
+| Tier | Trigger | What to do |
+|------|---------|------------|
+| **Tier 1 — Simple** | Single table or single CTE chain with no joins, or two tables joined on PK | State the grain of each table/CTE. One sentence. No verification query needed. |
+| **Tier 2 — Standard** | 2-3 tables/CTEs with FK joins | Full trace: grain per table and per CTE that introduces a join, relationship type per join, predicted output grain. Run one verification query (row count before/after the key join). |
+| **Tier 3 — Complex** | 4+ tables/CTEs with joins, any suspected M:M, fan-out risk, cross-grain aggregation, or querying upstream models whose internal logic you haven't verified | Full trace for every join (including within CTEs). Run verification queries for each join. Predict fan-out multiplier per join. Escalate to Data Modeller if anything is uncertain. |
+
+**Count joins, not just tables.** A query against two tables looks simple, but if
+one table is a model built from five CTEs with internal joins, the real complexity
+is higher. When in doubt, bump the tier up.
+
+## Knowledge Pre-Check
+
+Before beginning the join trace, run a Knowledge Checkpoint for each table in the planned join. Follow `.claude/agents/specific_instructions/shared/knowledge_checkpoint.md` (fast path first, deep path if needed).
+
+If a knowledge entry documents the grain, key type, or semantics of a table involved in the join, incorporate it into the trace and cite it using the citation format in the checkpoint protocol. Grain and key-type mismatches that the ledger already documents should be called out before any SQL is written — not discovered post-execution.
+
+If the ledger has no relevant entries, proceed directly to the trace.
+
+## The Trace
+
+Present the trace in-chat before writing the query. Format:
+
+```
+**Join Path Trace** — Tier <N>
+- <table_a>: grain = one row per <entity>
+- <table_b>: grain = one row per <entity>
+- <table_a> JOIN <table_b> ON <key> — <1:1 | 1:M | M:1 | M:M>
+  Output grain after this join: one row per <entity>
+- Predicted final grain: one row per <entity>
+- Fan-out risk: none | <describe the risk>
+```
+
+For queries with CTEs, trace the CTE chain from source to final:
+
+```
+**Join Path Trace** — Tier <N>
+CTE: source_orders — selects from orders; grain = one row per order_id
+CTE: order_items — selects from line_items; grain = one row per line_item_id
+CTE: enriched — source_orders JOIN order_items ON order_id — 1:M
+  Output grain: one row per line_item_id (fan-out from order to items — expected)
+CTE: aggregated — GROUP BY order_id on enriched
+  Output grain: one row per order_id (fan-out collapsed by aggregation)
+CTE: final — aggregated JOIN dim_customers ON customer_id — M:1
+  Output grain: one row per order_id (no fan-out — customer is a dimension)
+- Predicted final grain: one row per order_id
+- Fan-out risk: none — 1:M fan-out in enriched is collapsed by aggregation before final
+```
+
+For Tier 2+, append verification results after running the query:
+
+```
+- Verification: <left_table> rows = <N> → after join = <N> (<multiplier>x)
+```
+
+### What to check at each join
+
+For every join in the query, answer these three questions:
+
+1. **Grain match:** What is one row in the left table? What is one row in the
+   right table? After the join, what is one row in the result?
+2. **Key semantics:** Do the join keys represent the same entity? Same type?
+   (Not just same column name — same meaning.)
+3. **Cardinality:** Is this 1:1, 1:M, M:1, or M:M? If 1:M, is the fan-out
+   expected and handled (e.g., by a downstream aggregation)?
+
+### Tracing through upstream models
+
+When your query references a model you did not build (a mart, intermediate model,
+or staging model), do not assume its grain — verify it before including it in
+your trace:
+
+1. **Check for schema tests first.** If the model has a `unique` or
+   `unique_combination_of_columns` test on its PK in a `.yml` file, the grain is
+   documented and tested. State it in your trace and move on.
+2. **If no tests exist, read the model's SQL.** Open the `.sql` file and trace
+   its CTEs the same way you trace your own query. Identify where joins happen
+   inside the model, what cardinality they introduce, and what the final CTE's
+   grain is.
+3. **If you can't find the model definition** (e.g., it's a raw table from a
+   source system), run a grain validation query:
+   ```sql
+   select count(*) as total_rows, count(distinct <suspected_pk>) as distinct_pks
+   from <model>
+   ```
+   If `total_rows != distinct_pks`, the column you assumed was the PK is not
+   unique — your grain assumption is wrong. Escalate to the Data Modeller.
+
+**Why this matters:** A query that joins two well-tested marts is Tier 1. A query
+that joins two undocumented intermediate models with internal CTEs and no grain
+tests is Tier 3 — you're inheriting every unvalidated join inside those models.
+
+### Verification query (Tier 2+)
+
+Run before the actual query to confirm your cardinality assumption:
+
+```sql
+select 'before' as stage, count(*) as rows from <left_table>
+union all
+select 'after', count(*)
+from <left_table> join <right_table> on <join_condition>
+```
+
+If after > before: fan-out is occurring. Confirm it is expected. If unexpected,
+diagnose before writing the query.
+
+**Auto-verify**: when a query has 3+ joins and you're running this sweep
+once per join, that's the bulk read-only pattern auto-verify is for. Open
+`::AUTO-VERIFY:: agent=<your-agent> phase=<N>` before the sweep and
+`::ENDAUTO::` after. See `specific_instructions/shared/auto_verify_mode.md`.
+
+## Escalation Triggers
+
+Invoke the Data Modeller via Task instead of self-assessing when:
+
+- Any M:M relationship is detected or suspected
+- Verification query shows unexpected fan-out (predicted 1.0x, observed > 1.0x)
+- You cannot confidently state a table's grain
+- Tier 3 complexity on tables you have not previously explored with the Data Modeller
+- Join keys come from different source systems and semantic match is unclear
+- Upstream model has no grain tests and its internal CTE logic is unclear after reading
+
+Escalation prompt pattern:
+
+> "Before I write this query, I need the Data Modeller to verify the join path.
+> Here is my trace: [include trace]. The concern is: [specific concern]."
+
+## Outcome Documentation
+
+The trace itself lives in-chat only — do not write it to project-specs.md.
+
+If the trace reveals an issue (unexpected fan-out, grain mismatch, M:M risk),
+document it in the phase specs template:
+
+```
+- **Join path issues:** <finding> | none
+```
+
+## SQL Header Convention
+
+When writing `.sql` files, include the output grain in the header comment:
+
+```sql
+-- Output grain: one row per <entity>
+```
+
+This goes alongside existing header fields (Analysis/Dashboard name, Query
+description, Date, Filters/Data sources). It makes grain visible to anyone
+reading the query later.

package/src/agents/specific_instructions/shared/knowledge_checkpoint.md

@@ -0,0 +1,83 @@
+---
+name: knowledge-checkpoint-protocol
+description: Mid-phase knowledge re-grounding protocol — invoked inline during execution phases to re-ground against verified knowledge entries
+type: reference
+---
+
+# Knowledge Checkpoint Protocol
+
+A lightweight re-grounding check invoked inline during execution phases, and as the full build-phase knowledge re-grounding procedure referenced by the `**Knowledge re-check:**` line in each phase file.
+
+---
+
+## When to run / When to skip
+
+**Run the checkpoint** when the current task involves:
+- Writing SQL that references specific tables, joins, or entities
+- Making infrastructure claims (warehouse behaviors, API limits, connection patterns)
+- Asserting grain, key semantics, or column types for any data table
+
+**Skip the checkpoint** when the current task does not involve data tables, join paths, entity semantics, or infrastructure behavior. Knowledge re-checks apply to data-facing decisions, not general code generation (e.g., writing a Python utility, config files, README updates, test scaffolding).
+
+If it's unclear, default to running a fast path check — it costs at most one read.
+
+---
+
+## Fast path
+
+Re-read the `### Knowledge Ledger` subsection already in `project-specs.md`. This subsection was populated during Phase 0 retrieval and reflects what the ledger contained at project start.
+
+If the subsection covers the current question, use it directly — no file I/O required.
+
+---
+
+## Deep path (bounded)
+
+If the fast path is insufficient (the subsection doesn't address the specific entity, table, or system in question), scan `.shards/knowledge/INDEX.md` for rows matching the specific entity, table, or system using keyword match — the same approach as the retrieval protocol.
+
+- Read **up to 3 matching knowledge files** — no more.
+- Do **NOT** read the full ledger.
+- If no INDEX rows match, stop — the ledger has nothing relevant. Do not fall back to reading all entries.
+
+If `.shards/knowledge/` does not exist or INDEX.md is missing, the checkpoint is a no-op — proceed normally.
+
+---
+
+## Cite or flag
+
+### Citations
+
+Record all knowledge-influenced decisions in a `### Knowledge Citations` subsection in `project-specs.md`, appended to the current phase section. Do **not** scatter citations inline inside every SQL block or paragraph.
+
+Format:
+```
+- <decision> (per Knowledge Ledger: "<title>", <confidence>)
+```
+
+One citation per knowledge entry per phase is sufficient — do not repeat citations within the same phase.
+
+### Contradictions
+
+If an observation contradicts a ledger entry, flag it to the user immediately using this exact template:
+
+```
+**Knowledge contradiction:** "<ledger entry title>" claims <X>. Observed: <Y>. Ledger update needed: Yes/No. Resolution: <user decision>.
+```
+
+Ask the user for resolution before proceeding. Once resolved, document the resolution in `project-specs.md` using the template above (fill in the user's decision). Leave `Resolution: <user decision>` blank until the user responds, then update it.
+
+At harvest time, contradiction resolutions become update candidates (see `knowledge_harvest.md` Step 1b).
+
+---
+
+## Build-phase re-grounding (referenced by `**Knowledge re-check:**` lines)
+
+When a phase file instructs: `**Knowledge re-check:** Follow .claude/agents/specific_instructions/shared/knowledge_checkpoint.md before building.`
+
+Execute this sequence before writing any code, SQL, or artifacts:
+
+1. Re-read the `### Knowledge Ledger` subsection in `project-specs.md`.
+2. For each relevant entry in that subsection, keep it in mind as you build.
+3. When a knowledge entry influences a specific decision (query design, join key choice, grain assertion, infrastructure assumption), cite it using the citation format above.
+4. If the ledger subsection is absent or empty, run the deep path against `.shards/knowledge/INDEX.md` using the project's domain keywords as the keyword match.
+5. If the ledger has nothing relevant, proceed without citations — do not fabricate ledger references.