dew-method 0.1.0

package/src/dew-skills/3-solutioning/dew-storage-design/assets/storage-decision-record-template.md

@@ -0,0 +1,76 @@
+
+# Storage Decision Record Template
+
+```markdown
+# Storage Decision Records
+
+## SDR-0001 — {Decision Title}
+
+Date:
+
+Status:
+- proposed
+- accepted
+- accepted-with-caveats
+- rejected
+- superseded
+
+## Context
+
+## Decision
+
+## Options Considered
+
+### A. {Option A}
+
+Pros:
+
+Cons:
+
+### B. {Option B}
+
+Pros:
+
+Cons:
+
+### C. {Option C}
+
+Pros:
+
+Cons:
+
+## Selected Option
+
+## Rationale
+
+## Evidence Used
+
+- architecture:
+- source validation:
+- KPI feasibility:
+- requirement gate:
+- evidence index:
+
+## Consequences
+
+Positive:
+
+Negative:
+
+Trade-offs:
+
+## Reversibility
+
+- Easy
+- Medium
+- Hard
+
+## Revisit Trigger
+
+## Downstream Impact
+
+- Ingestion:
+- Transformation:
+- Serving:
+- Governance:
+- Implementation stories:

package/src/dew-skills/3-solutioning/dew-storage-design/assets/storage-design-template.md

@@ -0,0 +1,195 @@
+# Storage Design Template
+
+```markdown
+---
+title: "{Project Name} Storage Design"
+status: draft
+created: {YYYY-MM-DD}
+updated: {YYYY-MM-DD}
+workflow: dew
+phase: "3-solutioning"
+stepsCompleted: []
+decisionGates:
+  GATE-15-storage-layer:
+    status: pending
+  GATE-storage-metadata-standard:
+    status: pending
+  GATE-storage-retention:
+    status: pending
+evidence: []
+assumptions: []
+caveats: []
+---
+
+# Storage Design: {Project Name}
+
+## 0. Document Purpose
+
+This document defines the storage design for the project.
+
+It translates architecture-level storage intent into concrete storage layers, metadata standards, retention expectations, schema snapshot strategy, and source-to-storage mapping.
+
+## 1. Input Context
+
+| Input | Path | Status | Notes |
+|---|---|---|---|
+| Data Architecture | | Loaded/Missing | |
+| Architecture Decisions | | Loaded/Missing | |
+| Architecture Caveats | | Loaded/Missing | |
+| Source Validation | | Loaded/Missing | |
+| KPI Feasibility | | Loaded/Missing | |
+| Evidence Index | | Loaded/Missing | |
+
+## 2. Architecture Carryover
+
+### Selected Architecture Pattern
+
+### Storage Technology Direction
+
+### Caveats Carried Forward
+
+| Caveat ID | Area | Description | Storage Impact |
+|---|---|---|---|
+
+## 3. Storage Layer Strategy
+
+Choose one:
+
+A. Minimal raw + curated outputs  
+B. Bronze/Silver/Gold medallion  
+C. Warehouse staging + marts  
+D. Lakehouse tables with medallion convention  
+E. Local-first files with promotion path  
+
+Selected:
+
+Rationale:
+
+## 4. Layer Responsibilities
+
+| Layer | Purpose | Data Shape | Mutability | Consumers | Owner |
+|---|---|---|---|---|---|
+| Raw / Bronze | | | Append-only/Mutable | | |
+| Silver | | | | | |
+| Gold | | | | | |
+| Evidence | | | | | |
+| Metadata | | | | | |
+| Quarantine / Error | | | | | |
+
+## 5. Raw / Bronze Design
+
+### Purpose
+
+### Accepted Data Forms
+
+- original API response
+- original file
+- extracted rows
+- raw JSON
+- raw CSV
+- raw Parquet
+- manual upload copy
+
+### Raw Retention
+
+Choose one:
+
+A. Keep all raw inputs  
+B. Keep raw inputs for MVP period only  
+C. Keep raw samples and schema snapshots only  
+D. Keep raw externally and store references  
+E. No raw retention, with explicit caveat  
+
+Selected:
+
+### Raw Immutability
+
+### Source File / API Response Preservation
+
+### Caveats
+
+## 6. Silver Design
+
+### Purpose
+
+### Cleaning / Standardization Scope
+
+### Conformance Scope
+
+### Metadata Required
+
+### DQ Expectations
+
+### Caveats
+
+## 7. Gold Design
+
+### Purpose
+
+### KPI / Serving Alignment
+
+### Grain
+
+### Consumers
+
+### Caveats
+
+## 8. Metadata Standard
+
+Required metadata fields:
+
+| Field | Layer | Purpose | Required? |
+|---|---|---|---|
+| dew_source_system | Bronze/Silver/Gold | Source ID | Yes |
+| dew_source_file_path | Bronze/Silver | Raw source traceability | Yes |
+| dew_ingestion_timestamp | Bronze | Ingestion time | Optional/Required |
+| dew_silver_timestamp | Silver | Silver processing time | Optional/Required |
+| dew_silver_run_id | Silver | Processing run ID | Optional/Required |
+| dew_dq_status | Silver/Gold | DQ result status | Optional/Required |
+
+Selected metadata standard:
+
+## 9. Schema Snapshot Strategy
+
+| Source ID | Schema Snapshot Location | Frequency | Drift Handling |
+|---|---|---|---|
+| SRC-001 | | | |
+
+## 10. Partitioning / Organization
+
+### Folder / Table Organization
+
+### Partitioning Direction
+
+### Naming Convention
+
+### Environment Separation
+
+## 11. Source-to-Storage Mapping
+
+| Source ID | Source Name | Raw/Bronze Target | Silver Target | Gold Dependency | Caveat |
+|---|---|---|---|---|---|
+| SRC-001 | | | | | |
+
+## 12. Evidence and Audit Storage
+
+### Evidence Artifacts
+
+### Validation Results
+
+### Decision Logs
+
+### Learning Logs
+
+## 13. Open Questions
+
+## 14. Handoff to Ingestion Design
+
+Key storage decisions ingestion must follow:
+
+- target raw/bronze location:
+- metadata requirements:
+- schema snapshot expectations:
+- quarantine/error location:
+- retention rules:
+- idempotency implications:

package/src/dew-skills/3-solutioning/dew-storage-design/assets/storage-layer-rubric.md

@@ -0,0 +1,55 @@
+# Storage Layer Rubric
+
+## Minimal raw + curated outputs
+
+Use when:
+- project is small
+- learning/portfolio scope
+- source count is low
+- speed matters more than full medallion rigor
+
+Risk:
+- weaker governance
+- harder lineage if not disciplined
+
+## Bronze/Silver/Gold medallion
+
+Use when:
+- source validation evidence exists
+- transformations need clear separation
+- DQ and lineage matter
+- project is internal decision-support or more serious
+
+Risk:
+- more files/tables
+- more design overhead
+
+## Warehouse staging + marts
+
+Use when:
+- structured data dominates
+- SQL analytics is primary
+- serving is BI or data mart
+
+Risk:
+- less flexible for raw files/semi-structured data
+
+## Lakehouse tables with medallion convention
+
+Use when:
+- cloud/lakehouse architecture is chosen
+- scalable storage is needed
+- bronze/silver/gold should be table-based
+
+Risk:
+- higher operational complexity
+
+## Local-first files with promotion path
+
+Use when:
+- project needs fast local iteration
+- later deployment may target Fabric/cloud/lakehouse
+- DuckDB/Parquet/dbt-style development is preferred
+
+Risk:
+- local/cloud drift if conventions are weak

package/src/dew-skills/3-solutioning/dew-storage-design/checklist.md

@@ -0,0 +1,45 @@
+# Storage Design Checklist
+
+## Required inputs
+
+- [ ] Data architecture loaded.
+- [ ] Architecture decision record loaded.
+- [ ] Architecture caveat carryover loaded.
+- [ ] Source validation report loaded.
+- [ ] KPI feasibility report loaded.
+- [ ] Evidence index loaded or caveat recorded.
+
+## Storage layer design
+
+- [ ] Storage layer strategy selected by user.
+- [ ] Layer responsibilities defined.
+- [ ] Raw/Bronze purpose defined.
+- [ ] Silver purpose defined.
+- [ ] Gold purpose defined.
+- [ ] Evidence/metadata storage defined.
+- [ ] Quarantine/error storage direction defined.
+
+## Metadata and retention
+
+- [ ] Metadata standard selected.
+- [ ] Required metadata fields listed.
+- [ ] Raw retention decision made.
+- [ ] Schema snapshot strategy defined.
+- [ ] Caveats documented.
+
+## Mapping
+
+- [ ] Source-to-storage mapping created.
+- [ ] P1 sources mapped to raw/bronze target.
+- [ ] Schema snapshot targets mapped.
+- [ ] Quarantine targets mapped.
+- [ ] Ingestion handoff notes created.
+
+## Decision discipline
+
+- [ ] Decision log updated.
+- [ ] Learning log updated if learning mode is enabled.
+- [ ] Storage decision record created.
+- [ ] User approved storage layer decision.
+- [ ] User approved metadata/retention decisions.
+- [ ] No unresolved HALT-11 remains.

package/src/dew-skills/3-solutioning/dew-storage-design/customize.toml

@@ -0,0 +1,37 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for dew-storage-design.
+
+[workflow]
+
+activation_steps_prepend = []
+activation_steps_append = []
+
+persistent_facts = [
+  "file:{project-root}/**/data-architecture.md",
+  "file:{project-root}/**/architecture-decision-record.md",
+  "file:{project-root}/**/architecture-caveat-carryover.md",
+  "file:{project-root}/**/requirement-gate-report.md",
+  "file:{project-root}/**/source-validation-report.md",
+  "file:{project-root}/**/kpi-feasibility-report.md",
+  "file:{project-root}/**/validation-evidence-index.md",
+  "file:{project-root}/**/.decision-log.md",
+  "file:{project-root}/**/.learning-log.md",
+]
+
+on_complete = ""
+
+storage_design_template = "assets/storage-design-template.md"
+storage_decision_record_template = "assets/storage-decision-record-template.md"
+source_to_storage_mapping_template = "assets/source-to-storage-mapping-template.md"
+storage_layer_rubric = "assets/storage-layer-rubric.md"
+metadata_standard_rubric = "assets/metadata-standard-rubric.md"
+raw_retention_rubric = "assets/raw-retention-rubric.md"
+checklist = "checklist.md"
+
+storage_output_path = "{planning_artifacts}/storage-design"
+run_folder_pattern = "storage-design-{project_name}-{date}"
+
+decision_log_path = "{planning_artifacts}/.decision-log.md"
+learning_log_path = "{learning_artifacts}/.learning-log.md"
+evidence_index_path = "{evidence_artifacts}/validation-evidence-index.md"

package/src/dew-skills/3-solutioning/dew-storage-design/steps/step-01-init.md

@@ -0,0 +1,64 @@
+# Step 01 — Storage Design Initialization
+
+## Mandatory Rules
+
+- Do not design storage before data architecture exists.
+- Do not proceed if architecture is missing or blocked.
+- Read the complete step before acting.
+
+## Task
+
+Initialize Storage Design workflow.
+
+## Sequence
+
+### 1. Detect existing workflow
+
+Look for:
+
+- `{workflow.storage_output_path}/**/storage-design.md`
+- `{planning_artifacts}/**/*storage*design*.md`
+
+If unfinished storage design exists, offer to resume.
+
+### 2. Validate prerequisite
+
+Look for:
+
+- `data-architecture.md`
+- `architecture-decision-record.md`
+- `architecture-caveat-carryover.md`
+
+If missing, trigger:
+
+`HALT-10 — Architecture Before Validation`
+
+### 3. Create workspace
+
+Bind `{doc_workspace}` to:
+
+`{workflow.storage_output_path}/{workflow.run_folder_pattern}/`
+
+Create:
+
+- `storage-design.md`
+- `storage-decision-record.md`
+- `source-to-storage-mapping.md`
+
+Use templates.
+
+### 4. Initialize frontmatter
+
+Set:
+
+```yaml
+status: draft
+stepsCompleted:
+  - step-01-init
+decisionGates:
+  GATE-15-storage-layer:
+    status: pending
+  GATE-storage-metadata-standard:
+    status: pending
+  GATE-storage-retention:
+    status: pending

package/src/dew-skills/3-solutioning/dew-storage-design/steps/step-02-load-architecture-context.md

@@ -0,0 +1,50 @@
+# Step 02 — Load Architecture Context
+
+## Mandatory Rules
+
+- Load architecture, source validation, KPI feasibility, and caveat carryover.
+- Do not drop caveats.
+- Do not invent source targets.
+
+## Task
+
+Extract storage-relevant context.
+
+## Extract from architecture
+
+- selected architecture pattern
+- storage technology direction
+- source/generation summary
+- high-level storage intent
+- caveats
+
+## Extract from source validation
+
+- P1/P2 sources
+- source type
+- access pattern
+- source grain
+- source freshness
+- schema stability
+- validation status
+- caveats
+
+## Extract from KPI feasibility
+
+- KPI grain
+- required fields
+- Gold output needs
+- caveats
+
+## Update documents
+
+Update:
+- `storage-design.md`
+- `source-to-storage-mapping.md`
+
+## Update frontmatter
+
+```yaml
+stepsCompleted:
+  - step-01-init
+  - step-02-load-architecture-context

package/src/dew-skills/3-solutioning/dew-storage-design/steps/step-03-storage-layer-gate.md

@@ -0,0 +1,59 @@
+# Step 03 — Storage Layer Gate
+
+## Mandatory Rules
+
+- Stop at this gate.
+- Recommend, but do not decide.
+- User must approve storage layer strategy.
+- Record decision.
+
+## Task
+
+Run GATE-15 — Storage Layer Gate.
+
+## Options
+
+A. Minimal raw + curated outputs  
+B. Bronze/Silver/Gold medallion  
+C. Warehouse staging + marts  
+D. Lakehouse tables with medallion convention  
+E. Local-first files with promotion path  
+
+## Evaluation criteria
+
+- project type
+- trust expectation
+- source count
+- source variety
+- KPI complexity
+- need for lineage
+- DQ expectations
+- serving mode
+- local vs cloud architecture
+- cost/complexity
+
+## Required comparison table
+
+| Option | Pros | Cons | Fit | Risk |
+|---|---|---|---|---|
+
+## User decision
+
+Ask user to choose A/B/C/D/E.
+
+Do not continue until user chooses.
+
+## Record
+
+Update:
+- storage-design.md
+- storage-decision-record.md
+- .decision-log.md
+- .learning-log.md if enabled
+
+Update frontmatter:
+
+```yaml
+decisionGates:
+  GATE-15-storage-layer:
+    status: approved / approved-with-caveats

package/src/dew-skills/3-solutioning/dew-storage-design/steps/step-04-metadata-retention-schema-gates.md

@@ -0,0 +1,61 @@
+# Step 04 — Metadata, Retention, and Schema Snapshot Gates
+
+## Mandatory Rules
+
+- Stop for user decisions.
+- Metadata standard must match trust expectation.
+- Raw retention must be explicit.
+- Schema snapshot strategy must be explicit.
+- If schema evolution direction is missing, trigger HALT-13.
+
+## Gate 1 — Metadata Standard
+
+Options:
+
+A. Minimal metadata  
+B. Operational metadata  
+C. Full lineage metadata  
+D. Custom project-specific metadata  
+
+Ask user to choose.
+
+## Gate 2 — Raw Retention
+
+Options:
+
+A. Keep all raw inputs  
+B. Keep raw inputs for MVP period only  
+C. Keep raw samples and schema snapshots only  
+D. Keep raw externally and store references  
+E. No raw retention with explicit caveat  
+
+Ask user to choose.
+
+## Gate 3 — Schema Snapshot Strategy
+
+Options:
+
+A. Snapshot schema every ingestion run  
+B. Snapshot schema on source version/change  
+C. Snapshot schema manually during validation only  
+D. Use source contract tests and compare with baseline  
+E. No schema snapshot with explicit caveat  
+
+Ask user to choose.
+
+## Record
+
+Update:
+- storage-design.md
+- storage-decision-record.md
+- .decision-log.md
+- .learning-log.md if enabled
+
+Update frontmatter:
+
+```yaml
+stepsCompleted:
+  - step-01-init
+  - step-02-load-architecture-context
+  - step-03-storage-layer-gate
+  - step-04-metadata-retention-schema-gates

package/src/dew-skills/3-solutioning/dew-storage-design/steps/step-05-source-to-storage-mapping.md

@@ -0,0 +1,46 @@
+# Step 05 — Source-to-Storage Mapping
+
+## Mandatory Rules
+
+- Every P1 source must have a raw/bronze target.
+- Every P1 source must have schema snapshot target.
+- Every source caveat must be carried forward.
+- Do not create ingestion logic here.
+
+## Task
+
+Map sources to storage targets.
+
+## For each source
+
+Record:
+
+| Source ID | Source Name | Raw/Bronze Target | Silver Target | Schema Snapshot Target | Quarantine Target | Metadata Required |
+|---|---|---|---|---|---|---|
+
+## Mapping considerations
+
+- source type
+- access method
+- file/API response shape
+- schema stability
+- grain
+- freshness
+- DQ risk
+- caveats
+
+## Update
+
+Update:
+- storage-design.md
+- source-to-storage-mapping.md
+
+Update frontmatter:
+
+```yaml
+stepsCompleted:
+  - step-01-init
+  - step-02-load-architecture-context
+  - step-03-storage-layer-gate
+  - step-04-metadata-retention-schema-gates
+  - step-05-source-to-storage-mapping