dew-method 0.1.0

package/src/dew-skills/3-solutioning/dew-data-architecture/assets/data-architecture-template.md

@@ -0,0 +1,219 @@
+# Data Architecture Template
+
+```markdown
+---
+title: "{Project Name} Data Architecture"
+status: draft
+created: {YYYY-MM-DD}
+updated: {YYYY-MM-DD}
+workflow: dew
+phase: "3-solutioning"
+stepsCompleted: []
+decisionGates:
+  GATE-13-architecture-pattern:
+    status: pending
+  GATE-14-technology-choice:
+    status: pending
+evidence: []
+assumptions: []
+caveats: []
+architecturePattern:
+  status: pending
+technologyDecisions:
+  status: pending
+---
+
+# Data Architecture: {Project Name}
+
+## 0. Document Purpose
+
+This architecture translates validated/caveated requirements into a data engineering architecture.
+
+It is grounded in:
+- Requirement Gate Report
+- KPI Feasibility Report
+- Source Validation Report
+- Evidence Index
+- Decision Log
+- Caveat Register
+
+This document does not replace detailed storage, ingestion, transformation, serving, or governance designs.
+
+Those are created in later DEW skills.
+
+## 1. Validated Context
+
+### Data Product
+
+### Data Consumers
+
+### Decisions Supported
+
+### Trust Expectation
+
+### MVP Scope
+
+### Explicit Non-Goals
+
+## 2. Evidence-Grounded Inputs
+
+| Input | Status | Evidence | Caveat |
+|---|---|---|---|
+| Primary KPI feasibility | Ready/Caveated/Blocked | | |
+| P1 source validation | Ready/Caveated/Blocked | | |
+| Product grain | Ready/Caveated/Blocked | | |
+| Freshness | Ready/Caveated/Blocked | | |
+| Trust expectation | Ready/Caveated/Blocked | | |
+| Requirement gate | Ready/Caveated/Blocked | | |
+
+## 3. Data Engineering Lifecycle Architecture
+
+### 3.1 Generation / Source Systems
+
+Validated sources:
+
+| Source ID | Source Name | Role | Validation Status | Caveat |
+|---|---|---|---|---|
+
+Source constraints:
+- access:
+- schema stability:
+- grain:
+- freshness:
+- quality:
+- privacy/security:
+
+### 3.2 Storage
+
+Architecture-level storage intent:
+
+- Raw/Bronze:
+- Silver:
+- Gold:
+- Metadata:
+- Evidence retention:
+
+Detailed storage design deferred to:
+
+`dew-storage-design`
+
+### 3.3 Ingestion
+
+Architecture-level ingestion intent:
+
+- ingestion mode:
+- batch/streaming assumption:
+- snapshot/incremental direction:
+- replay/backfill expectation:
+- schema drift direction:
+
+Detailed ingestion design deferred to:
+
+`dew-ingestion-design`
+
+### 3.4 Transformation
+
+Architecture-level transformation intent:
+
+- grain transformations:
+- KPI computation layer:
+- conformance logic:
+- business rule ownership:
+- bridge/mapping needs:
+
+Detailed transformation design deferred to:
+
+`dew-transformation-design`
+
+### 3.5 Serving
+
+Architecture-level serving intent:
+
+- serving mode:
+- consumers:
+- freshness:
+- trust statement:
+- limitations:
+
+Detailed serving design deferred to:
+
+`dew-serving-design`
+
+## 4. Architecture Pattern Decision
+
+Options considered:
+- Local-first
+- Warehouse-first
+- Lakehouse
+- Hybrid
+- Streaming-first
+- Application-integrated
+
+Selected pattern:
+
+Rationale:
+
+Trade-offs:
+
+Rejected alternatives:
+
+Revisit trigger:
+
+## 5. Technology Decision Summary
+
+| Decision Area | Selected Direction | Alternatives | Rationale | Revisit Trigger |
+|---|---|---|---|---|
+| Local development | | | | |
+| Storage format | | | | |
+| Compute engine | | | | |
+| Orchestration | | | | |
+| Transformation tool | | | | |
+| Quality/testing | | | | |
+| Serving layer | | | | |
+| Documentation/catalog | | | | |
+
+## 6. Caveat Carryover
+
+| Caveat ID | Area | Description | Architecture Impact | Later Skill Responsible |
+|---|---|---|---|---|
+| CAV-001 | | | | |
+
+## 7. DataOps, Security, and Governance Undercurrents
+
+### Security
+
+### Privacy
+
+### Access Control
+
+### Observability
+
+### Orchestration
+
+### Data Management
+
+### Software Engineering Practices
+
+### Cost / FinOps
+
+## 8. Architecture Risks
+
+| Risk ID | Risk | Severity | Mitigation | Owner / Later Skill |
+|---|---|---|---|---|
+
+## 9. Reversible vs Hard-to-Reverse Decisions
+
+| Decision | Reversibility | Why | Revisit Trigger |
+|---|---|---|---|
+
+## 10. Open Questions
+
+## 11. Architecture Handoff
+
+Next skills:
+
+- `dew-storage-design`
+- `dew-ingestion-design`
+- `dew-transformation-design`
+- `dew-serving-design`
+- `dew-governance-security-design`

package/src/dew-skills/3-solutioning/dew-data-architecture/assets/data-lifecycle-architecture-guide.md

@@ -0,0 +1,74 @@
+# Data Lifecycle Architecture Guide
+
+DEW architecture follows the data engineering lifecycle.
+
+## 1. Generation / Source Systems
+
+Architecture must record:
+- source type
+- access method
+- ownership/provider
+- grain
+- freshness
+- schema stability
+- source trust status
+- caveats
+
+Do not design ingestion until source validation is clear.
+
+## 2. Storage
+
+Architecture should decide at high level:
+- raw/bronze retention
+- silver conformance purpose
+- gold serving purpose
+- metadata expectations
+- evidence retention
+- file/table direction
+
+Detailed storage design belongs to `dew-storage-design`.
+
+## 3. Ingestion
+
+Architecture should decide at high level:
+- batch vs streaming direction
+- snapshot vs incremental direction
+- replay/backfill expectation
+- schema drift direction
+- failure handling expectation
+
+Detailed ingestion design belongs to `dew-ingestion-design`.
+
+## 4. Transformation
+
+Architecture should decide at high level:
+- where KPI computation happens
+- grain transformation points
+- business rule ownership
+- bridge/mapping needs
+- DQ responsibilities
+
+Detailed transformation design belongs to `dew-transformation-design`.
+
+## 5. Serving
+
+Architecture should decide at high level:
+- serving mode
+- consumer
+- freshness
+- trust statement
+- limitations
+
+Detailed serving design belongs to `dew-serving-design`.
+
+## 6. Undercurrents
+
+Architecture must not forget:
+- security
+- privacy
+- data management
+- DataOps
+- orchestration
+- software engineering
+- observability
+- cost / FinOps

package/src/dew-skills/3-solutioning/dew-data-architecture/assets/technology-decision-rubric.md

@@ -0,0 +1,130 @@
+# Technology Decision Rubric
+
+Technology choices must be justified by requirements and evidence.
+
+Do not choose technology because it is trendy.
+
+## Decision criteria
+
+Evaluate each technology choice using:
+
+- project scope
+- user skill level
+- data volume
+- data variety
+- freshness requirement
+- source access pattern
+- transformation complexity
+- serving needs
+- governance/security posture
+- cost
+- operational complexity
+- portability
+- local development speed
+- deployment target
+- team maintainability
+
+## Required format
+
+Every technology decision must include:
+
+- selected option
+- alternatives considered
+- why selected
+- why alternatives rejected
+- evidence used
+- risk
+- reversibility
+- revisit trigger
+
+## Common decision areas
+
+### Local development
+
+Options:
+- Python scripts
+- notebooks
+- DuckDB
+- dbt-duckdb
+- local Spark
+- containerized dev environment
+
+### Storage
+
+Options:
+- local files
+- Parquet
+- Delta
+- warehouse tables
+- lakehouse tables
+- object storage
+
+### Compute
+
+Options:
+- pandas
+- Polars
+- DuckDB
+- Spark
+- SQL warehouse
+- cloud notebook runtime
+
+### Orchestration
+
+Options:
+- manual run
+- Makefile/task runner
+- Airflow/Dagster/Prefect
+- Fabric Data Pipeline
+- cloud-native scheduler
+
+### Transformation
+
+Options:
+- SQL
+- dbt
+- Python
+- Spark
+- hybrid
+
+### Data quality
+
+Options:
+- custom tests
+- dbt tests
+- Great Expectations/Soda
+- lightweight Python checks
+- source contract tests
+
+### Serving
+
+Options:
+- dashboard
+- Gold mart
+- API
+- notebook/report
+- web app feature
+- ML feature table
+
+## Reversibility guide
+
+### Easy to reverse
+
+- local folder structure
+- naming conventions
+- lightweight scripts
+- dashboard layout
+
+### Medium to reverse
+
+- storage format
+- transformation framework
+- orchestration tool
+
+### Hard to reverse
+
+- cloud platform
+- data model grain
+- source integration pattern
+- security model
+- public API contract

package/src/dew-skills/3-solutioning/dew-data-architecture/checklist.md

@@ -0,0 +1,68 @@
+# Data Architecture Checklist
+
+## Required inputs
+
+- [ ] Requirement Gate Report loaded.
+- [ ] Architecture readiness is approved or approved-with-caveats.
+- [ ] Data Product PRD loaded.
+- [ ] KPI Feasibility Report loaded.
+- [ ] Source Validation Report loaded.
+- [ ] Evidence Index loaded or missing evidence recorded.
+- [ ] Decision Log loaded or initialized.
+- [ ] Learning Log loaded or initialized.
+
+## Validated context
+
+- [ ] Business decision captured.
+- [ ] Data consumer captured.
+- [ ] Trust expectation captured.
+- [ ] MVP scope captured.
+- [ ] Non-goals captured.
+- [ ] Caveats carried over.
+
+## Lifecycle architecture
+
+- [ ] Source/generation architecture summarized.
+- [ ] Storage intent summarized.
+- [ ] Ingestion intent summarized.
+- [ ] Transformation intent summarized.
+- [ ] Serving intent summarized.
+- [ ] Undercurrents summarized.
+
+## Architecture decisions
+
+- [ ] Architecture pattern options compared.
+- [ ] User approved selected architecture pattern.
+- [ ] Technology decision options compared.
+- [ ] User approved technology direction.
+- [ ] ADRs created.
+- [ ] Rejected alternatives documented.
+- [ ] Reversibility documented.
+- [ ] Revisit triggers documented.
+
+## Caveats and risks
+
+- [ ] Requirement Gate caveats carried over.
+- [ ] Architecture risks documented.
+- [ ] Caveat owners / later skills assigned.
+- [ ] No caveat hidden or silently removed.
+
+## Finalization
+
+- [ ] data-architecture.md completed.
+- [ ] architecture-decision-record.md completed.
+- [ ] architecture-caveat-carryover.md completed.
+- [ ] Decision log updated.
+- [ ] Learning log updated if learning mode is enabled.
+- [ ] Next skills identified.
+- [ ] No unresolved hard HALT remains.
+
+## Failure conditions
+
+Fail this workflow if:
+- Requirement Gate is missing.
+- Requirement Gate is blocked.
+- Architecture proceeds without user approval.
+- Unvalidated KPI/source assumptions become architecture facts.
+- Technology is selected without trade-off explanation.
+- Caveats from Requirement Gate are dropped.

package/src/dew-skills/3-solutioning/dew-data-architecture/customize.toml

@@ -0,0 +1,42 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for dew-data-architecture.
+#
+# Override files:
+#   {project-root}/_dew/custom/dew-data-architecture.toml
+#   {project-root}/_dew/custom/dew-data-architecture.user.toml
+
+[workflow]
+
+activation_steps_prepend = []
+activation_steps_append = []
+
+persistent_facts = [
+  "file:{project-root}/**/requirement-gate-report.md",
+  "file:{project-root}/**/architecture-readiness-matrix.md",
+  "file:{project-root}/**/caveat-register.md",
+  "file:{project-root}/**/data-product-prd.md",
+  "file:{project-root}/**/kpi-catalog.md",
+  "file:{project-root}/**/kpi-feasibility-report.md",
+  "file:{project-root}/**/source-validation-report.md",
+  "file:{project-root}/**/validation-evidence-index.md",
+  "file:{project-root}/**/.decision-log.md",
+  "file:{project-root}/**/.learning-log.md",
+]
+
+on_complete = ""
+
+architecture_template = "assets/data-architecture-template.md"
+adr_template = "assets/architecture-decision-record-template.md"
+caveat_carryover_template = "assets/architecture-caveat-carryover-template.md"
+architecture_pattern_rubric = "assets/architecture-pattern-rubric.md"
+technology_decision_rubric = "assets/technology-decision-rubric.md"
+lifecycle_guide = "assets/data-lifecycle-architecture-guide.md"
+checklist = "checklist.md"
+
+architecture_output_path = "{planning_artifacts}/data-architecture"
+run_folder_pattern = "data-architecture-{project_name}-{date}"
+
+decision_log_path = "{planning_artifacts}/.decision-log.md"
+learning_log_path = "{learning_artifacts}/.learning-log.md"
+evidence_index_path = "{evidence_artifacts}/validation-evidence-index.md"

package/src/dew-skills/3-solutioning/dew-data-architecture/steps/step-01-init.md

@@ -0,0 +1,74 @@
+# Step 01 — Data Architecture Initialization
+
+## Mandatory Rules
+
+- Do not create architecture before checking Requirement Gate.
+- Do not proceed if Requirement Gate is missing.
+- Do not proceed if Requirement Gate decision is blocked.
+- Read the complete step before acting.
+- Initialize document state before moving to next step.
+
+## Task
+
+Initialize Data Architecture workflow.
+
+## Sequence
+
+### 1. Detect existing architecture workflow
+
+Look for existing architecture docs in:
+
+- `{workflow.architecture_output_path}/**/data-architecture.md`
+- `{planning_artifacts}/**/*architecture*.md`
+
+If an unfinished architecture exists, offer to resume.
+
+### 2. Validate prerequisite
+
+Look for:
+
+- `requirement-gate-report.md`
+- `architecture-readiness-matrix.md`
+- `caveat-register.md`
+
+The Requirement Gate must have one of:
+
+- Proceed to architecture
+- Proceed to architecture with caveats
+- architectureReadiness.status = ready
+- architectureReadiness.status = ready-with-caveats
+
+If missing or blocked, trigger:
+
+`HALT-10 — Architecture Before Validation`
+
+### 3. Create workspace
+
+If fresh workflow, bind `{doc_workspace}` to:
+
+`{workflow.architecture_output_path}/{workflow.run_folder_pattern}/`
+
+Create:
+
+- `data-architecture.md`
+- `architecture-decision-record.md`
+- `architecture-caveat-carryover.md`
+
+Use:
+- `{workflow.architecture_template}`
+- `{workflow.adr_template}`
+- `{workflow.caveat_carryover_template}`
+
+### 4. Initialize frontmatter
+
+Set:
+
+```yaml
+status: draft
+stepsCompleted:
+  - step-01-init
+decisionGates:
+  GATE-13-architecture-pattern:
+    status: pending
+  GATE-14-technology-choice:
+    status: pending

package/src/dew-skills/3-solutioning/dew-data-architecture/steps/step-02-load-validated-context.md

@@ -0,0 +1,77 @@
+# Step 02 — Load Validated Context
+
+## Mandatory Rules
+
+- Load all validated/caveated context before architecture design.
+- Do not treat assumptions as facts.
+- Do not drop caveats from Requirement Gate.
+- If evidence is missing for a key claim, label it as caveated or blocked.
+
+## Task
+
+Load validated context for architecture.
+
+## Input discovery
+
+Search for:
+
+- `{planning_artifacts}/**/requirement-gate-report.md`
+- `{planning_artifacts}/**/architecture-readiness-matrix.md`
+- `{planning_artifacts}/**/caveat-register.md`
+- `{planning_artifacts}/**/data-product-prd.md`
+- `{planning_artifacts}/**/kpi-feasibility-report.md`
+- `{planning_artifacts}/**/source-validation-report.md`
+- `{evidence_artifacts}/**/validation-evidence-index.md`
+- `{planning_artifacts}/**/.decision-log.md`
+- `{learning_artifacts}/**/.learning-log.md`
+
+## Extract
+
+### From Requirement Gate
+
+- final gate decision
+- trust expectation
+- grain decision
+- freshness decision
+- accepted caveats
+- architecture readiness status
+
+### From KPI Feasibility
+
+- primary KPI statuses
+- KPI caveats
+- grain implications
+- required fields
+- sample computation evidence
+
+### From Source Validation
+
+- P1/P2 sources
+- source access evidence
+- schema evidence
+- source grain
+- source freshness
+- source DQ/profile notes
+- source caveats
+
+### From PRD
+
+- data consumers
+- data product type
+- decisions supported
+- MVP scope
+- non-goals
+
+## Update documents
+
+Update:
+
+- `data-architecture.md`
+- `architecture-caveat-carryover.md`
+
+## Update frontmatter
+
+```yaml
+stepsCompleted:
+  - step-01-init
+  - step-02-load-validated-context