dew-method 0.1.0

package/src/dew-skills/2-plan-workflows/dew-source-validation/steps/step-05-grain-freshness-profile.md

@@ -0,0 +1,71 @@
+# Step 04 — Source Sample and Schema Snapshot
+
+## Mandatory Rules
+
+- Do not assume schema.
+- Do not assume required fields exist.
+- Do not promote to P1 if sample or schema is missing.
+- If evidence is missing, trigger `HALT-18 — No Validation Evidence`.
+
+## Task
+
+Capture or document source sample and schema snapshot.
+
+## Required evidence
+
+For each validation-first source:
+
+- source sample
+- schema snapshot or field list
+- required KPI field mapping
+
+## Sample evidence
+
+Record:
+- sample path
+- captured date
+- row count if applicable
+- format
+- time range if applicable
+- coverage notes
+
+## Schema evidence
+
+Record:
+- schema path
+- field names
+- field types if available
+- nullable/required if known
+- example values
+- related KPI fields
+
+## Field mapping
+
+| KPI Required Field | Source Field | Available? | Evidence | Notes |
+|---|---|---|---|---|
+
+## Missing field handling
+
+If required field is missing:
+
+A. Find substitute field  
+B. Modify KPI formula  
+C. Defer KPI  
+D. Reject source for this KPI  
+
+Ask user if the missing field affects primary KPI.
+
+## Update report
+
+Update:
+- source-validation-report.md
+- evidence index if evidence exists
+
+Update frontmatter:
+
+```yaml
+stepsCompleted:
+  - step-01-init
+  - step-02-load-source-context
+  - step-03-access-check
+  - step-04-sample-and-schema

package/src/dew-skills/2-plan-workflows/dew-source-validation/steps/step-06-source-trust-gate.md

@@ -0,0 +1,61 @@
+# Step 06 — Source Trust Gate
+
+## Mandatory Rules
+
+- Stop at this gate.
+- Present evidence and missing evidence.
+- Recommend status, but do not decide.
+- User must approve final status for validation-first sources.
+- Do not promote to P1 without access/sample/schema evidence.
+
+## Task
+
+Run GATE-08 — Source Trust Gate.
+
+## Status options
+
+For each source, choose one:
+
+A. P1 — Use in MVP  
+B. P2 — Use later  
+C. Exploration only  
+D. Reject  
+E. Blocked  
+
+## Gate explanation
+
+Explain:
+- why source validation matters
+- how unverified sources cause architecture hallucination
+- how source grain affects transformation
+- how source freshness affects ingestion/serving
+- how schema stability affects ingestion and DQ
+- what evidence is still missing
+
+## Required table
+
+| Source ID | Access | Sample | Schema | Grain | Freshness | DQ Risk | Recommended Status |
+|---|---|---|---|---|---|---|---|
+
+## User decision
+
+Ask the user to approve status for each validation-first source.
+
+Do not continue until user chooses.
+
+## Decision log
+
+Record each source decision in `.decision-log.md`.
+
+## Learning log
+
+If learning mode is enabled, record what was learned about source validation.
+
+## Update frontmatter
+
+Update:
+
+```yaml
+decisionGates:
+  GATE-08-source-trust:
+    status: approved / approved-with-caveats / blocked

package/src/dew-skills/2-plan-workflows/dew-source-validation/steps/step-07-finalize.md

@@ -0,0 +1,50 @@
+# Step 07 — Finalize Source Validation Report
+
+## Mandatory Rules
+
+- Do not finalize if validation-first P1 source lacks access/sample/schema evidence unless user explicitly approved caveated continuation.
+- Do not mark architecture ready unless source trust gate is approved or approved-with-caveats.
+- Evidence must be indexed if it exists.
+
+## Task
+
+Finalize source validation report.
+
+## Checklist
+
+Run `checklist.md`.
+
+## Required final outputs
+
+- source-validation-report.md
+- source sample notes
+- schema snapshot notes
+- source profile notes
+- decision log entry
+- learning log entry
+- evidence index update if evidence exists
+
+## Architecture readiness status
+
+Choose one:
+
+A. Architecture can proceed after requirement gate  
+B. Architecture can proceed with source caveats after requirement gate  
+C. Architecture cannot proceed; source validation unresolved  
+D. Reduce MVP scope  
+
+## Final status
+
+Set report frontmatter:
+
+```yaml
+status: final
+updated: {date}
+stepsCompleted:
+  - step-01-init
+  - step-02-load-source-context
+  - step-03-access-check
+  - step-04-sample-and-schema
+  - step-05-grain-freshness-profile
+  - step-06-source-trust-gate
+  - step-07-finalize

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: dew-create-epics-and-stories
+description: Convert approved DEW designs into data engineering epics, story map, dependencies, and implementation backlog.
+---
+
+# DEW Create Epics and Stories
+
+**Goal:** Convert approved DEW planning/design artifacts into implementation-ready epics and a story map.
+
+**Your Role:** You are a data engineering delivery planner.
+
+You translate validated data product design into epics and story candidates.
+
+You may recommend, but you must not decide.
+
+## Workflow Architecture
+
+This workflow uses step-file architecture.
+
+Rules:
+- Read one step file at a time.
+- Read the complete step file before acting.
+- Do not skip steps.
+- Do not load future steps early.
+- Stop at gates.
+- Update frontmatter after each completed step.
+- Do not create implementation stories without design context.
+- Do not invent requirements missing from design artifacts.
+
+## Mandatory Rules
+
+- Do not create epics before DataOps design exists.
+- Do not create implementation backlog if architecture/storage/ingestion/transformation/model/DQ/serving context is missing.
+- Do not create vague epics like “build pipeline” without layer/source/KPI scope.
+- Every epic must map to design artifacts.
+- Every story candidate must have evidence requirements.
+- If implementation readiness is unclear, trigger `HALT-17 — Implementation Not Ready`.
+- If evidence requirements are missing, trigger `HALT-18 — No Validation Evidence`.
+
+## Conventions
+
+- Bare paths resolve from the skill root.
+- `{skill-root}` resolves to this skill's installed directory.
+- `{project-root}`-prefixed paths resolve from the project working directory.
+- `{workflow.<name>}` resolves to fields in `customize.toml`'s `[workflow]` table.
+- `{doc_workspace}` is the run folder for this workflow.
+
+## On Activation
+
+1. Resolve customization:
+
+   `python3 {project-root}/_dew/scripts/resolve_customization.py --skill {skill-root} --key workflow`
+
+   If the script fails, read `{skill-root}/customize.toml` directly and use defaults.
+
+2. Execute each entry in `{workflow.activation_steps_prepend}`.
+
+3. Load persistent facts from `{workflow.persistent_facts}`.
+
+4. Load `{project-root}/_dew/dew/config.yaml` if present.
+
+5. Load:
+   - `{workflow.epics_template}`
+   - `{workflow.story_map_template}`
+   - `{workflow.epic_readiness_rubric}`
+   - `{workflow.data_engineering_epic_patterns}`
+   - `{workflow.dependency_mapping_rubric}`
+
+6. Greet user in configured language.
+
+## Execution
+
+Read fully and follow:
+
+`steps/step-01-init.md`

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/assets/data-engineering-epic-patterns.md

@@ -0,0 +1,78 @@
+# Data Engineering Epic Patterns
+
+## EPIC-01 — Foundation and Project Setup
+
+Typical stories:
+- initialize repo/project structure
+- create config management
+- create evidence folder structure
+- create logging conventions
+- create README/run guide
+
+## EPIC-02 — Source Access and Bronze Ingestion
+
+Typical stories:
+- implement source access
+- implement raw download/API response capture
+- implement Bronze write
+- capture schema snapshot
+- capture ingestion run log
+
+## EPIC-03 — Source Contract and DQ Prechecks
+
+Typical stories:
+- implement required field checks
+- implement schema drift check
+- implement row count check
+- implement freshness check
+
+## EPIC-04 — Silver Transformation
+
+Typical stories:
+- implement source-specific cleaning
+- cast/rename fields
+- standardize units/codes
+- deduplicate
+- add metadata
+- quarantine invalid records
+
+## EPIC-05 — Gold Modeling and KPI Computation
+
+Typical stories:
+- implement Gold fact/mart
+- implement KPI formula
+- implement bridge table
+- implement aggregate table
+- validate sample KPI output
+
+## EPIC-06 — Data Quality Evidence
+
+Typical stories:
+- implement DQ rule catalog
+- implement critical DQ tests
+- write DQ result artifacts
+- fail/quarantine/warn policy
+
+## EPIC-07 — Serving Data Product
+
+Typical stories:
+- create dashboard/mart/API/report
+- expose metric contract
+- add freshness display
+- show limitations/caveats
+
+## EPIC-08 — Governance Documentation
+
+Typical stories:
+- document lineage
+- document access control
+- document trust statement
+- document owners/stewards
+
+## EPIC-09 — DataOps and Observability
+
+Typical stories:
+- implement run logs
+- create observability plan artifacts
+- create incident/backfill runbook
+- add CI/test commands

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/assets/dependency-mapping-rubric.md

@@ -0,0 +1,36 @@
+# Dependency Mapping Rubric
+
+## Hard dependency
+
+A story cannot start until another story is done.
+
+Examples:
+- Silver transform depends on Bronze ingestion.
+- Gold KPI depends on Silver model.
+- Serving dashboard depends on Gold model.
+- DQ rule execution depends on target dataset.
+
+## Soft dependency
+
+A story can start with caveats, but must be revisited.
+
+Examples:
+- documentation can start before final code paths exist.
+- dashboard wireframe can start before final Gold model.
+
+## Evidence dependency
+
+A story depends on validation evidence.
+
+Examples:
+- KPI computation story needs KPI feasibility evidence.
+- source ingestion story needs source access evidence.
+- DQ implementation story needs DQ rule catalog.
+
+## User decision dependency
+
+A story depends on an approved decision gate.
+
+Examples:
+- serving story depends on serving mode decision.
+- privacy/access story depends on privacy posture decision.

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/assets/epic-readiness-rubric.md

@@ -0,0 +1,30 @@
+# Epic Readiness Rubric
+
+## Ready
+
+Use when:
+- epic has clear goal
+- design inputs exist
+- story candidates are scoped
+- dependencies are known
+- evidence requirements are defined
+
+## Ready with caveats
+
+Use when:
+- design inputs exist but caveats remain
+- caveats are carried into stories
+- user approves continuation
+
+## Blocked
+
+Use when:
+- required design artifact is missing
+- KPI/source/model/DQ dependency is unresolved
+- evidence requirement is unclear
+
+## Not applicable
+
+Use when:
+- epic is outside MVP scope
+- user explicitly deferred it

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/assets/epics-template.md

@@ -0,0 +1,171 @@
+# Epics Template
+
+```markdown
+---
+title: "{Project Name} Epics"
+status: draft
+created: {YYYY-MM-DD}
+updated: {YYYY-MM-DD}
+workflow: dew
+phase: "3-solutioning"
+stepsCompleted: []
+decisionGates:
+  GATE-23-implementation-readiness:
+    status: pending
+evidence: []
+assumptions: []
+caveats: []
+---
+
+# Epics: {Project Name}
+
+## 0. Document Purpose
+
+This document converts approved DEW designs into implementation epics.
+
+Each epic must map back to validated design artifacts and define evidence expected from implementation.
+
+## 1. Input Design Artifacts
+
+| Artifact | Path | Loaded? | Status |
+|---|---|---|---|
+| Data Product PRD | | Yes/No | |
+| Requirement Gate | | Yes/No | |
+| Data Architecture | | Yes/No | |
+| Storage Design | | Yes/No | |
+| Ingestion Design | | Yes/No | |
+| Transformation Design | | Yes/No | |
+| Data Model Spec | | Yes/No | |
+| Data Quality Rules | | Yes/No | |
+| Serving Design | | Yes/No | |
+| Governance Security Review | | Yes/No | |
+| DataOps Runbook | | Yes/No | |
+
+## 2. Epic Summary
+
+| Epic ID | Epic Name | Purpose | Depends On | Status |
+|---|---|---|---|---|
+| EPIC-01 | Foundation and Project Setup | | | backlog |
+
+## 3. Epics
+
+## EPIC-01 — Foundation and Project Setup
+
+### Goal
+
+Set up project structure, configuration, environment, and evidence folders needed for data engineering implementation.
+
+### Design Inputs
+
+- Data Architecture:
+- DataOps Runbook:
+- Governance/Security:
+- Storage Design:
+
+### Scope
+
+In:
+- repo/project structure
+- config management
+- environment setup
+- evidence folder setup
+- logging conventions
+
+Out:
+- actual source ingestion
+- actual transformations
+
+### Story Candidates
+
+| Story ID | Story Title | Layer | Depends On | Evidence Required |
+|---|---|---|---|---|
+| 1.1 | Initialize project structure | foundation | | repo structure evidence |
+
+### Acceptance Criteria
+
+- project structure matches design
+- config files exist
+- evidence paths exist
+- README/run instructions exist
+
+### Evidence Expected
+
+- file list
+- setup log
+- config sample
+- run command output
+
+### Risks / Caveats
+
+## EPIC-02 — Source Access and Bronze Ingestion
+
+### Goal
+
+Implement access and ingestion for validated P1 sources into Raw/Bronze storage.
+
+### Design Inputs
+
+- Source Validation:
+- Storage Design:
+- Ingestion Design:
+- Source Contract Tests:
+
+### Story Candidates
+
+| Story ID | Story Title | Source | Depends On | Evidence Required |
+|---|---|---|---|---|
+| 2.1 | Implement source access for SRC-001 | SRC-001 | 1.1 | access log |
+| 2.2 | Ingest SRC-001 into Bronze | SRC-001 | 2.1 | run log, row count, schema snapshot |
+
+## EPIC-03 — Source Contract and DQ Prechecks
+
+### Goal
+
+Implement source contract tests and ingestion-level DQ checks.
+
+## EPIC-04 — Silver Transformation
+
+### Goal
+
+Implement Bronze → Silver transformation rules.
+
+## EPIC-05 — Gold Modeling and KPI Computation
+
+### Goal
+
+Implement Gold models and KPI computation.
+
+## EPIC-06 — Data Quality Evidence
+
+### Goal
+
+Implement DQ rule catalog tests and evidence outputs.
+
+## EPIC-07 — Serving Data Product
+
+### Goal
+
+Implement dashboard/mart/API/report/web app serving layer.
+
+## EPIC-08 — Governance, Documentation, and Trust Statement
+
+### Goal
+
+Implement documentation, access notes, lineage metadata, and user-facing trust/limitations statement.
+
+## EPIC-09 — DataOps, Observability, and Runbook
+
+### Goal
+
+Implement operational runbook, monitoring, logging, incident/backfill process, and evidence generation.
+
+## 4. Dependency Map
+
+## 5. Implementation Risks
+
+## 6. Open Questions
+
+## 7. Next Workflow
+
+`dew-sprint-planning`
+```

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/assets/story-map-template.md

@@ -0,0 +1,34 @@
+# Story Map Template
+
+```markdown
+# Story Map
+
+## Purpose
+
+This file maps story candidates to implementation order and dependencies.
+
+## Story Map Table
+
+| Story ID | Epic ID | Title | Layer | Depends On | Required Design Input | Evidence Required | Status |
+|---|---|---|---|---|---|---|---|
+| 1.1 | EPIC-01 | Initialize project structure | foundation | | dataops-runbook | setup evidence | backlog |
+
+## Layer Order
+
+Recommended order:
+
+1. Foundation
+2. Source access
+3. Bronze ingestion
+4. Source contract tests
+5. Silver transformation
+6. Gold model / KPI computation
+7. DQ tests and evidence
+8. Serving product
+9. Governance documentation
+10. DataOps / observability
+
+## Dependency Notes
+
+## Caveats Carried Into Stories
+```

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/checklist.md

@@ -0,0 +1,19 @@
+# Create Epics and Stories Checklist
+
+- [ ] DataOps design loaded.
+- [ ] Serving design loaded.
+- [ ] Governance design loaded.
+- [ ] DQ design loaded.
+- [ ] Data model design loaded.
+- [ ] Transformation design loaded.
+- [ ] Ingestion design loaded.
+- [ ] Storage design loaded.
+- [ ] Architecture loaded.
+- [ ] Epics created.
+- [ ] Story candidates created.
+- [ ] Story dependencies mapped.
+- [ ] Evidence requirements mapped.
+- [ ] Caveats carried into story map.
+- [ ] Implementation readiness gate prepared.
+- [ ] Decision log updated if decisions made.
+- [ ] Learning log updated if enabled.

package/src/dew-skills/3-solutioning/dew-create-epics-and-stories/customize.toml

@@ -0,0 +1,42 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for dew-create-epics-and-stories.
+
+[workflow]
+
+activation_steps_prepend = []
+activation_steps_append = []
+
+persistent_facts = [
+  "file:{project-root}/**/data-product-prd.md",
+  "file:{project-root}/**/requirement-gate-report.md",
+  "file:{project-root}/**/data-architecture.md",
+  "file:{project-root}/**/storage-design.md",
+  "file:{project-root}/**/ingestion-design.md",
+  "file:{project-root}/**/transformation-design.md",
+  "file:{project-root}/**/data-model-spec.md",
+  "file:{project-root}/**/data-quality-rules.md",
+  "file:{project-root}/**/serving-data-product-spec.md",
+  "file:{project-root}/**/governance-security-review.md",
+  "file:{project-root}/**/dataops-runbook.md",
+  "file:{project-root}/**/operational-readiness-report.md",
+  "file:{project-root}/**/validation-evidence-index.md",
+  "file:{project-root}/**/.decision-log.md",
+  "file:{project-root}/**/.learning-log.md",
+]
+
+on_complete = ""
+
+epics_template = "assets/epics-template.md"
+story_map_template = "assets/story-map-template.md"
+epic_readiness_rubric = "assets/epic-readiness-rubric.md"
+data_engineering_epic_patterns = "assets/data-engineering-epic-patterns.md"
+dependency_mapping_rubric = "assets/dependency-mapping-rubric.md"
+checklist = "checklist.md"
+
+epics_output_path = "{planning_artifacts}/epics-and-stories"
+run_folder_pattern = "epics-stories-{project_name}-{date}"
+
+decision_log_path = "{planning_artifacts}/.decision-log.md"
+learning_log_path = "{learning_artifacts}/.learning-log.md"
+evidence_index_path = "{evidence_artifacts}/validation-evidence-index.md"