dew-method 0.1.0

package/src/dew-skills/3-solutioning/dew-ingestion-design/SKILL.md

@@ -0,0 +1,69 @@
+---
+name: dew-ingestion-design
+description: Design ingestion patterns, source extraction, frequency, schema evolution, failure handling, idempotency, replay, and evidence requirements.
+---
+
+# DEW Ingestion Design
+
+**Goal:** Create a detailed ingestion design grounded in validated sources and approved storage design.
+
+**Your Role:** You are an ingestion design facilitator.
+
+You translate source validation and storage design into ingestion patterns and operational rules.
+
+You may recommend, but you must not decide.
+
+## Workflow Architecture
+
+This workflow uses step-file architecture.
+
+Rules:
+- Read one step file at a time.
+- Read the complete step file before acting.
+- Do not skip steps.
+- Do not load future steps early.
+- Stop at decision gates.
+- Update frontmatter after each completed step.
+- Preserve caveats from architecture/storage/source validation.
+
+## Mandatory Rules
+
+- Do not design ingestion without source validation.
+- Do not design ingestion without storage targets.
+- Do not assume batch/streaming; ask at gate.
+- Do not assume snapshot/incremental; ask at gate.
+- Do not ignore schema evolution.
+- Do not ignore failure handling, idempotency, replay, or backfill.
+- If ingestion pattern is unclear, trigger `HALT-12 — Ingestion Pattern Unclear`.
+- If schema evolution is undefined, trigger `HALT-13 — Schema Evolution Undefined`.
+- If evidence is missing, trigger `HALT-18 — No Validation Evidence`.
+
+## On Activation
+
+1. Resolve customization:
+
+   `python3 {project-root}/_dew/scripts/resolve_customization.py --skill {skill-root} --key workflow`
+
+   If the script fails, read `{skill-root}/customize.toml` directly and use defaults.
+
+2. Execute `{workflow.activation_steps_prepend}`.
+
+3. Load persistent facts from `{workflow.persistent_facts}`.
+
+4. Load `{project-root}/_dew/dew/config.yaml` if present.
+
+5. Load:
+   - `{workflow.ingestion_design_template}`
+   - `{workflow.ingestion_decision_record_template}`
+   - `{workflow.ingestion_source_plan_template}`
+   - `{workflow.ingestion_pattern_rubric}`
+   - `{workflow.schema_evolution_rubric}`
+   - `{workflow.failure_handling_rubric}`
+
+6. Greet user in configured language.
+
+## Execution
+
+Read fully and follow:
+
+`steps/step-01-init.md`

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/failure-handling-rubric.md

@@ -0,0 +1,48 @@
+# Failure Handling Rubric
+
+## Retry
+
+Use for:
+- transient network errors
+- temporary API failures
+- database timeouts
+
+## Backoff
+
+Use for:
+- rate-limited APIs
+- overloaded sources
+
+## Quarantine
+
+Use for:
+- malformed records
+- invalid schema
+- records failing critical DQ checks
+
+## Dead-letter
+
+Use for:
+- event/message ingestion
+- records that cannot be processed
+
+## Partial load handling
+
+Define whether partial loads:
+- fail whole run
+- load valid records and quarantine invalid records
+- require manual approval
+
+## Idempotency
+
+Every ingestion design must explain:
+- what happens if the same run is executed twice
+- how duplicate files/API pages/records are handled
+- how reruns are detected
+
+## Replay / Backfill
+
+Every ingestion design must explain:
+- whether raw data is retained
+- whether historical source data can be re-requested
+- how to rerun ingestion safely

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/ingestion-decision-record-template.md

@@ -0,0 +1,76 @@
+
+# Ingestion Decision Record Template
+
+```markdown
+# Ingestion Decision Records
+
+## IDR-0001 — {Decision Title}
+
+Date:
+
+Status:
+- proposed
+- accepted
+- accepted-with-caveats
+- rejected
+- superseded
+
+## Context
+
+## Decision
+
+## Options Considered
+
+### A. {Option A}
+
+Pros:
+
+Cons:
+
+### B. {Option B}
+
+Pros:
+
+Cons:
+
+### C. {Option C}
+
+Pros:
+
+Cons:
+
+## Selected Option
+
+## Rationale
+
+## Evidence Used
+
+- source validation:
+- storage design:
+- source contract test:
+- architecture:
+- evidence index:
+
+## Consequences
+
+Positive:
+
+Negative:
+
+Trade-offs:
+
+## Reversibility
+
+- Easy
+- Medium
+- Hard
+
+## Revisit Trigger
+
+## Downstream Impact
+
+- Storage:
+- Transformation:
+- Data quality:
+- Serving:
+- Implementation stories:

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/ingestion-design-template.md

@@ -0,0 +1,210 @@
+# Ingestion Design Template
+
+```markdown
+---
+title: "{Project Name} Ingestion Design"
+status: draft
+created: {YYYY-MM-DD}
+updated: {YYYY-MM-DD}
+workflow: dew
+phase: "3-solutioning"
+stepsCompleted: []
+decisionGates:
+  GATE-16-ingestion-pattern:
+    status: pending
+  GATE-17-batch-vs-streaming:
+    status: pending
+  GATE-ingestion-snapshot-incremental:
+    status: pending
+  GATE-11-schema-evolution:
+    status: pending
+evidence: []
+assumptions: []
+caveats: []
+---
+
+# Ingestion Design: {Project Name}
+
+## 0. Document Purpose
+
+This document defines how validated/caveated sources are ingested into the approved storage design.
+
+It covers:
+- source extraction pattern
+- batch vs streaming
+- snapshot vs incremental
+- frequency
+- schema evolution
+- failure handling
+- idempotency
+- replay/backfill
+- logging/evidence requirements
+
+## 1. Input Context
+
+| Input | Path | Status | Notes |
+|---|---|---|---|
+| Data Architecture | | Loaded/Missing | |
+| Storage Design | | Loaded/Missing | |
+| Source-to-Storage Mapping | | Loaded/Missing | |
+| Source Validation Report | | Loaded/Missing | |
+| Source Contract Tests | | Loaded/Missing | |
+| Evidence Index | | Loaded/Missing | |
+
+## 2. Architecture and Storage Carryover
+
+### Architecture Pattern
+
+### Storage Targets
+
+### Caveats
+
+| Caveat ID | Area | Description | Ingestion Impact |
+|---|---|---|---|
+
+## 3. Source Ingestion Summary
+
+| Source ID | Source Name | Source Type | Access Pattern | Target | Recommended Ingestion Pattern |
+|---|---|---|---|---|---|
+| SRC-001 | | API/File/DB | | | |
+
+## 4. Ingestion Pattern Decision
+
+Options:
+
+A. Pull API  
+B. File download / file drop  
+C. Database query/export  
+D. CDC  
+E. Stream/event ingestion  
+F. Manual upload  
+
+Selected by source:
+
+## 5. Batch vs Streaming Decision
+
+Options:
+
+A. Batch  
+B. Micro-batch  
+C. Streaming  
+D. Event-driven  
+E. Manual/on-demand  
+
+Selected:
+
+Rationale:
+
+## 6. Snapshot vs Incremental Decision
+
+Options:
+
+A. Full snapshot  
+B. Incremental by timestamp  
+C. Incremental by cursor/token  
+D. Append-only events  
+E. Manual versioned files  
+F. Hybrid  
+
+Selected by source:
+
+## 7. Ingestion Frequency
+
+| Source ID | Required Freshness | Source Freshness | Ingestion Frequency | Caveat |
+|---|---|---|---|---|
+| SRC-001 | | | | |
+
+## 8. Schema Evolution Strategy
+
+Options:
+
+A. Strict schema contract  
+B. Schema snapshot + drift detection  
+C. Flexible rescued fields  
+D. Manual review on change  
+E. Source-specific strategy  
+
+Selected by source:
+
+## 9. Idempotency and Deduplication
+
+### Idempotency Key
+
+### Natural Key / Business Key
+
+### File/API response uniqueness
+
+### Duplicate handling
+
+## 10. Failure Handling
+
+### Retry
+
+### Backoff
+
+### Dead-letter / quarantine
+
+### Partial load handling
+
+### Alerting
+
+### Manual recovery
+
+## 11. Replay and Backfill
+
+### Replay Strategy
+
+### Backfill Strategy
+
+### Raw Dependency
+
+### Limitations
+
+## 12. Logging and Evidence
+
+Required logs:
+- ingestion run log
+- source access log
+- schema snapshot
+- row count summary
+- error/quarantine report
+- DQ precheck report
+
+## 13. Source-Specific Ingestion Plans
+
+## SRC-001 — {Source Name}
+
+### Source Type
+
+### Access Pattern
+
+### Target Raw/Bronze Location
+
+### Ingestion Pattern
+
+### Frequency
+
+### Snapshot/Incremental Strategy
+
+### Required Metadata
+
+### Schema Evolution Strategy
+
+### Failure Handling
+
+### Idempotency
+
+### Replay/Backfill
+
+### Caveats
+
+## 14. Handoff to Transformation Design
+
+Transformation must know:
+- raw/bronze target
+- metadata fields
+- schema caveats
+- source grain
+- ingestion frequency
+- idempotency/deduplication assumptions
+- DQ precheck results

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/ingestion-pattern-rubric.md

@@ -0,0 +1,75 @@
+
+# Ingestion Pattern Rubric
+
+## Pull API
+
+Use when:
+- source provides HTTP API
+- data can be requested periodically
+- API response can be saved as raw evidence
+
+Risks:
+- rate limits
+- schema drift
+- auth changes
+- pagination
+
+## File download / file drop
+
+Use when:
+- source publishes CSV/JSON/Parquet/etc.
+- file versioning is possible
+- raw file can be retained
+
+Risks:
+- manual updates
+- file format changes
+- unclear freshness
+
+## Database query/export
+
+Use when:
+- source is a database or warehouse
+- query access is available
+- data is structured
+
+Risks:
+- source load
+- query permissions
+- schema changes
+
+## CDC
+
+Use when:
+- change-level capture is required
+- source supports CDC
+- freshness and auditability justify complexity
+
+Risks:
+- operational complexity
+- ordering issues
+- replay complexity
+
+## Stream/event ingestion
+
+Use when:
+- source is event-driven
+- low-latency is required
+- data is unbounded
+
+Risks:
+- much higher complexity
+- ordering and exactly-once semantics
+- monitoring burden
+
+## Manual upload
+
+Use when:
+- source is human-provided
+- automation is not possible yet
+- learning/MVP accepts manual process
+
+Risks:
+- inconsistent format
+- late delivery
+- weak operational reliability

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/ingestion-source-plan-template.md

@@ -0,0 +1,83 @@
+### `assets/ingestion-source-plan-template.md`
+
+```markdown
+# Ingestion Source Plan Template
+
+```markdown
+# Ingestion Source Plan
+
+## Source
+
+Source ID:
+
+Source Name:
+
+Validation Status:
+
+## Access
+
+Access method:
+
+Access evidence:
+
+## Target
+
+Raw/Bronze target:
+
+Schema snapshot target:
+
+Quarantine target:
+
+## Pattern
+
+Ingestion pattern:
+
+Batch/streaming:
+
+Snapshot/incremental:
+
+Frequency:
+
+## Metadata
+
+Required metadata fields:
+
+## Schema Evolution
+
+Strategy:
+
+Drift detection:
+
+Breaking change behavior:
+
+## Failure Handling
+
+Retry:
+
+Backoff:
+
+Dead-letter/quarantine:
+
+Manual intervention:
+
+## Idempotency
+
+Idempotency key:
+
+Deduplication logic:
+
+## Replay / Backfill
+
+Replay possible?
+
+Backfill possible?
+
+Requirements:
+
+## Evidence to Produce During Implementation
+
+- run log
+- row count summary
+- schema snapshot
+- access log
+- error report

package/src/dew-skills/3-solutioning/dew-ingestion-design/assets/schema-evolution-rubric.md

@@ -0,0 +1,46 @@
+# Schema Evolution Rubric
+
+## Strict schema contract
+
+Use when:
+- source schema must not change silently
+- production trust is high
+- breaking changes should fail fast
+
+Behavior:
+- fail ingestion on unexpected schema
+- require review before accepting changes
+
+## Schema snapshot + drift detection
+
+Use when:
+- source is external
+- no formal contract exists
+- changes must be detected and logged
+
+Behavior:
+- capture schema snapshot
+- compare to previous snapshot
+- warn or fail depending on severity
+
+## Flexible rescued fields
+
+Use when:
+- semi-structured data may contain extra fields
+- you want to avoid hard failure for non-critical changes
+
+Behavior:
+- expected fields validated
+- unexpected fields captured separately
+
+## Manual review on change
+
+Use when:
+- source updates are infrequent
+- project is learning/MVP
+- manual approval is acceptable
+
+Behavior:
+- detect change
+- stop or warn
+- user reviews before proceeding

package/src/dew-skills/3-solutioning/dew-ingestion-design/checklist.md

@@ -0,0 +1,46 @@
+# Ingestion Design Checklist
+
+## Required inputs
+
+- [ ] Data architecture loaded.
+- [ ] Storage design loaded.
+- [ ] Source-to-storage mapping loaded.
+- [ ] Source validation report loaded.
+- [ ] Source contract test results loaded or caveat recorded.
+- [ ] Evidence index loaded or caveat recorded.
+
+## Ingestion pattern
+
+- [ ] Ingestion pattern selected by source.
+- [ ] Batch vs streaming decision made.
+- [ ] Snapshot vs incremental decision made.
+- [ ] Frequency mapped to freshness requirement.
+- [ ] User approved key ingestion decisions.
+
+## Source-specific plans
+
+- [ ] Each P1 source has ingestion source plan.
+- [ ] Raw/Bronze target specified.
+- [ ] Metadata requirements specified.
+- [ ] Schema snapshot target specified.
+- [ ] Quarantine/error handling specified.
+- [ ] Idempotency strategy specified.
+- [ ] Replay/backfill strategy specified.
+
+## Schema and failure handling
+
+- [ ] Schema evolution strategy selected.
+- [ ] Failure handling strategy defined.
+- [ ] Retry/backoff behavior defined if relevant.
+- [ ] Partial load behavior defined.
+- [ ] Evidence/logging requirements defined.
+
+## Finalization
+
+- [ ] ingestion-design.md completed.
+- [ ] ingestion-decision-record.md completed.
+- [ ] ingestion-source-plan.md completed.
+- [ ] Decision log updated.
+- [ ] Learning log updated if learning mode is enabled.
+- [ ] No unresolved HALT-12 remains.
+- [ ] No unresolved HALT-13 remains.

package/src/dew-skills/3-solutioning/dew-ingestion-design/customize.toml

@@ -0,0 +1,38 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for dew-ingestion-design.
+
+[workflow]
+
+activation_steps_prepend = []
+activation_steps_append = []
+
+persistent_facts = [
+  "file:{project-root}/**/data-architecture.md",
+  "file:{project-root}/**/architecture-caveat-carryover.md",
+  "file:{project-root}/**/storage-design.md",
+  "file:{project-root}/**/source-to-storage-mapping.md",
+  "file:{project-root}/**/source-validation-report.md",
+  "file:{project-root}/**/source-contract-test*.md",
+  "file:{project-root}/**/kpi-feasibility-report.md",
+  "file:{project-root}/**/validation-evidence-index.md",
+  "file:{project-root}/**/.decision-log.md",
+  "file:{project-root}/**/.learning-log.md",
+]
+
+on_complete = ""
+
+ingestion_design_template = "assets/ingestion-design-template.md"
+ingestion_decision_record_template = "assets/ingestion-decision-record-template.md"
+ingestion_source_plan_template = "assets/ingestion-source-plan-template.md"
+ingestion_pattern_rubric = "assets/ingestion-pattern-rubric.md"
+schema_evolution_rubric = "assets/schema-evolution-rubric.md"
+failure_handling_rubric = "assets/failure-handling-rubric.md"
+checklist = "checklist.md"
+
+ingestion_output_path = "{planning_artifacts}/ingestion-design"
+run_folder_pattern = "ingestion-design-{project_name}-{date}"
+
+decision_log_path = "{planning_artifacts}/.decision-log.md"
+learning_log_path = "{learning_artifacts}/.learning-log.md"
+evidence_index_path = "{evidence_artifacts}/validation-evidence-index.md"