dew-method 0.1.0

package/src/dew-skills/4-implementation/dew-dev-story/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: dew-dev-story
+description: Implement a ready-for-dev data engineering story while producing required evidence, logs, tests, and story updates.
+---
+
+# DEW Dev Story
+
+**Goal:** Implement one ready-for-dev data engineering story and produce required evidence.
+
+**Your Role:** You are a data engineering implementation agent.
+
+You implement only what the story authorizes.
+
+## Mandatory Rules
+
+- Work on one story at a time.
+- Do not implement a story unless status is `ready-for-dev`.
+- Do not implement missing requirements not present in story/design context.
+- Do not silently change acceptance criteria.
+- Do not silently change KPI formula, grain, DQ rule, or serving contract.
+- If a new dependency is needed, HALT.
+- If required config/secrets/source path is missing, HALT.
+- If validation fails three times, HALT.
+- Do not mark story done. Dev story can only move story to `review`.
+- Required evidence must be produced or story remains blocked/in-progress.
+
+## Workflow Architecture
+
+This workflow uses step-file architecture.
+
+Rules:
+- Read one step file at a time.
+- Do not skip steps.
+- Update story file after implementation.
+- Update sprint-status.yaml.
+- Preserve story caveats.
+
+## On Activation
+
+1. Resolve customization.
+2. Load persistent facts.
+3. Load config if present.
+4. Load:
+   - `{workflow.dev_notes_template}`
+   - `{workflow.evidence_manifest_template}`
+   - `{workflow.file_list_template}`
+   - `{workflow.implementation_task_rubric}`
+   - `{workflow.dev_halt_rubric}`
+5. Greet user in configured language.
+
+## Execution
+
+Read fully and follow:
+
+`steps/step-01-load-ready-story.md`

package/src/dew-skills/4-implementation/dew-dev-story/assets/dev-halt-rubric.md

@@ -0,0 +1,42 @@
+# Dev HALT Rubric
+
+Trigger HALT when:
+
+## New dependency required
+
+Examples:
+- new library not approved
+- new service needed
+- new cloud resource needed
+- new source credential needed
+
+Recommended HALT:
+`HALT-17 — Implementation Not Ready`
+
+## Missing config/secrets/source path
+
+Recommended HALT:
+`HALT-17 — Implementation Not Ready`
+
+## Story conflicts with design
+
+Examples:
+- story says KPI formula A but model design says formula B
+- source path differs from ingestion design
+- DQ severity conflicts with DQ policy
+
+Recommended HALT:
+Return to design or story revision.
+
+## Validation fails repeatedly
+
+If validation fails three times, stop and document:
+- attempts
+- error
+- suspected cause
+- required decision
+
+## Evidence cannot be produced
+
+Recommended HALT:
+`HALT-18 — No Validation Evidence`

package/src/dew-skills/4-implementation/dew-dev-story/assets/dev-notes-template.md

@@ -0,0 +1,64 @@
+# Dev Notes Template
+
+```markdown
+---
+title: "Dev Notes — {Story ID}"
+status: draft
+created: {YYYY-MM-DD}
+updated: {YYYY-MM-DD}
+workflow: dew
+phase: "4-implementation"
+storyId: "{Story ID}"
+---
+
+# Dev Notes: {Story ID} — {Story Title}
+
+## Story
+
+Path:
+
+Status before dev:
+
+Status after dev:
+
+## Implementation Summary
+
+## Tasks Completed
+
+| Task | Status | Notes |
+|---|---|---|
+| | done/incomplete/blocked | |
+
+## Decisions Made During Dev
+
+| Decision | Reason | Approved? |
+|---|---|---|
+
+## Files Created / Modified
+
+| Path | Action | Purpose |
+|---|---|---|
+| | created/modified/deleted | |
+
+## Validation Run
+
+| Command / Method | Result | Evidence |
+|---|---|---|
+| | pass/fail/not-run | |
+
+## Evidence Produced
+
+| Evidence ID | Type | Path | Status |
+|---|---|---|---|
+| | | | |
+
+## DQ Results
+
+## Operational Notes
+
+## Caveats
+
+## Blockers
+
+## Handoff to Review
+```

package/src/dew-skills/4-implementation/dew-dev-story/assets/evidence-manifest-template.md

@@ -0,0 +1,37 @@
+# Evidence Manifest Template
+
+```markdown
+# Evidence Manifest: {Story ID}
+
+## Purpose
+
+Record evidence produced by implementation.
+
+## Evidence Table
+
+| Evidence ID | Type | Path | Required By | Status |
+|---|---|---|---|---|
+| EVD-{Story ID}-001 | run-log | | Story AC | missing/produced/validated |
+
+## Evidence Types
+
+- run-log
+- test-result
+- dq-result
+- schema-snapshot
+- row-count-summary
+- sample-output
+- metric-output
+- freshness-report
+- screenshot
+- api-response
+- notebook-output
+- validation-report
+- README/run-instructions
+
+## Missing Evidence
+
+| Required Evidence | Reason Missing | Impact |
+|---|---|---|
+| | | |
+```

package/src/dew-skills/4-implementation/dew-dev-story/assets/file-list-template.md

@@ -0,0 +1,9 @@
+# File List Template
+
+```markdown
+# File List: {Story ID}
+
+| Path | Action | Purpose | Related AC |
+|---|---|---|---|
+| | created/modified/deleted | | |
+```

package/src/dew-skills/4-implementation/dew-dev-story/assets/implementation-task-rubric.md

@@ -0,0 +1,71 @@
+# Implementation Task Rubric
+
+## Foundation story
+
+Must usually include:
+- project folders
+- config files
+- README/run guide
+- evidence folder
+- environment instructions
+
+## Source access story
+
+Must usually include:
+- source access code/script/notebook
+- access log
+- error handling
+- sample output
+
+## Bronze ingestion story
+
+Must usually include:
+- ingestion script/notebook
+- raw/bronze output
+- metadata fields
+- run log
+- row count
+- schema snapshot
+
+## Source contract story
+
+Must usually include:
+- required field tests
+- schema drift test
+- row count test
+- freshness test if applicable
+- contract test result
+
+## Silver transformation story
+
+Must usually include:
+- transformation code
+- DQ checks
+- output dataset
+- invalid/quarantine handling
+- validation evidence
+
+## Gold/KPI story
+
+Must usually include:
+- KPI computation code
+- Gold model output
+- KPI sample output
+- grain validation
+- DQ evidence
+
+## Serving story
+
+Must usually include:
+- dashboard/API/report/app artifact
+- metric contract reference
+- trust statement display/reference
+- freshness/caveat display if relevant
+
+## DataOps story
+
+Must usually include:
+- runbook update
+- observability output
+- monitoring/logging code or plan
+- incident/backfill instruction

package/src/dew-skills/4-implementation/dew-dev-story/checklist.md

@@ -0,0 +1,17 @@
+# Dev Story Checklist
+
+- [ ] Story loaded.
+- [ ] Story status is ready-for-dev.
+- [ ] Story context pack loaded.
+- [ ] Sprint status loaded.
+- [ ] Dependencies checked.
+- [ ] Implementation plan created.
+- [ ] Tasks implemented.
+- [ ] Required evidence produced or blocker documented.
+- [ ] Validation commands/approach executed or blocker documented.
+- [ ] DQ results produced if relevant.
+- [ ] File list updated.
+- [ ] Dev notes created.
+- [ ] Evidence manifest created.
+- [ ] Story updated.
+- [ ] Sprint status updated to review or blocked.

package/src/dew-skills/4-implementation/dew-dev-story/customize.toml

@@ -0,0 +1,36 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for dew-dev-story.
+
+[workflow]
+
+activation_steps_prepend = []
+activation_steps_append = []
+
+persistent_facts = [
+  "file:{project-root}/**/sprint-status.yaml",
+  "file:{project-root}/**/stories/**/*.md",
+  "file:{project-root}/**/story-context/**/*.md",
+  "file:{project-root}/**/readiness-reviews/**/*.md",
+  "file:{project-root}/**/validation-evidence-index.md",
+  "file:{project-root}/**/.decision-log.md",
+  "file:{project-root}/**/.learning-log.md",
+]
+
+on_complete = ""
+
+dev_notes_template = "assets/dev-notes-template.md"
+evidence_manifest_template = "assets/evidence-manifest-template.md"
+file_list_template = "assets/file-list-template.md"
+implementation_task_rubric = "assets/implementation-task-rubric.md"
+dev_halt_rubric = "assets/dev-halt-rubric.md"
+checklist = "checklist.md"
+
+stories_output_path = "{implementation_artifacts}/stories"
+dev_notes_output_path = "{implementation_artifacts}/dev-notes"
+story_evidence_output_path = "{evidence_artifacts}/story-evidence"
+sprint_status_path = "{implementation_artifacts}/sprint/sprint-status.yaml"
+evidence_index_path = "{evidence_artifacts}/validation-evidence-index.md"
+
+decision_log_path = "{planning_artifacts}/.decision-log.md"
+learning_log_path = "{learning_artifacts}/.learning-log.md"

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-01-load-ready-story.md

@@ -0,0 +1,35 @@
+# Step 01 — Load Ready Story
+
+## Mandatory Rules
+
+- Do not work on story unless status is `ready-for-dev`.
+- Do not proceed if sprint-status.yaml is missing.
+- Do not skip dependency check.
+
+## Task
+
+Load selected story.
+
+## Sequence
+
+1. Load `sprint-status.yaml`.
+2. Select story:
+   - user-specified story, or
+   - first `ready-for-dev` story.
+3. Load story file.
+4. Load story context pack.
+5. Load implementation readiness review if available.
+6. Verify dependencies are done or not required.
+
+## If invalid
+
+- no ready story → recommend `dew-create-data-story`
+- missing context → `HALT-17`
+- missing evidence requirements → `HALT-18`
+- dependency not done → `HALT-17`
+
+## Continue checkpoint
+
+Ask:
+
+`[C] Continue to plan implementation`

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-02-plan-implementation.md

@@ -0,0 +1,40 @@
+# Step 02 — Plan Implementation
+
+## Mandatory Rules
+
+- Plan only what story authorizes.
+- Do not add new scope.
+- Do not change ACs.
+
+## Task
+
+Create implementation plan.
+
+## Extract
+
+From story:
+- acceptance criteria
+- tasks
+- evidence requirements
+- validation approach
+- DQ requirements
+- operational requirements
+- files to create/modify
+
+## Create
+
+- `dev-notes/{story-id}-dev-notes.md`
+- `story-evidence/{story-id}/evidence-manifest.md`
+- `story-evidence/{story-id}/file-list.md`
+
+## Update sprint status
+
+Set story status:
+
+`in-progress`
+
+## Continue checkpoint
+
+Ask:
+
+`[C] Continue to implement tasks`

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-03-implement-tasks.md

@@ -0,0 +1,31 @@
+# Step 03 — Implement Tasks
+
+## Mandatory Rules
+
+- Implement tasks in story order.
+- Do not silently change formulas, grain, DQ rules, or source paths.
+- If new dependency appears, HALT.
+- If task cannot be completed, document blocker.
+
+## Task
+
+Implement story tasks.
+
+## Implementation discipline
+
+For each task:
+- update dev notes
+- update file list
+- preserve caveats
+- create/modify only relevant files
+- avoid unrelated refactors
+
+## Completion
+
+When tasks are implemented or blocked, continue to validation.
+
+## Continue checkpoint
+
+Ask:
+
+`[C] Continue to run validation and evidence`

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-04-run-validation-and-evidence.md

@@ -0,0 +1,45 @@
+# Step 04 — Run Validation and Evidence
+
+## Mandatory Rules
+
+- Evidence must match story requirements.
+- Do not fabricate test results.
+- If validation fails three times, HALT.
+- If required evidence cannot be produced, trigger HALT-18.
+
+## Task
+
+Run validation commands or validation approach defined in story.
+
+## Validate
+
+Check:
+- acceptance criteria
+- DQ requirements
+- output paths
+- row count/schema/metric output if relevant
+- operational logs if relevant
+
+## Produce evidence
+
+Examples:
+- run log
+- DQ result
+- schema snapshot
+- row count summary
+- metric output
+- screenshot/report export
+- validation report
+
+## Update
+
+- evidence manifest
+- validation evidence index
+- dev notes
+- story file evidence table
+
+## Continue checkpoint
+
+Ask:
+
+`[C] Continue to update story and status`

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-05-update-story-and-status.md

@@ -0,0 +1,42 @@
+# Step 05 — Update Story and Status
+
+## Mandatory Rules
+
+- Dev cannot mark story done.
+- Dev can move story to review only if implementation and evidence are complete.
+- If evidence missing, keep story in-progress or blocked.
+
+## Task
+
+Update story.
+
+## Update story sections
+
+- Tasks completed
+- Evidence Requirements
+- Validation Commands / Approach result
+- Files Created / Modified
+- Completion Notes
+- Caveats
+- Status
+
+## Status rules
+
+If implementation complete and evidence produced:
+- story status → Review
+- sprint-status → review
+
+If blocked:
+- story status → Blocked
+- sprint-status → blocked
+- blocker documented
+
+If incomplete:
+- story status → In Progress
+- sprint-status → in-progress
+
+## Continue checkpoint
+
+Ask:
+
+`[C] Continue to finalize dev`

package/src/dew-skills/4-implementation/dew-dev-story/steps/step-06-finalize-dev.md

@@ -0,0 +1,22 @@
+# Step 06 — Finalize Dev
+
+## Mandatory Rules
+
+- Run checklist.
+- Do not mark done.
+- Recommend review next.
+
+## Required outputs
+
+- updated story
+- dev notes
+- evidence manifest
+- file list
+- evidence files
+- updated sprint-status.yaml
+
+## Next workflow
+
+Recommend:
+
+`dew-code-review`

package/src/dew-skills/4-implementation/dew-e2e-validation/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: dew-e2e-validation
+description: Validate the implemented data product from source through storage, ingestion, transformation, DQ, Gold, and serving outputs.
+---
+
+# DEW E2E Validation
+
+**Goal:** Validate that the implemented data product works end-to-end.
+
+**Your Role:** You are an end-to-end data product validation facilitator.
+
+You check the complete path from source to serving. You do not approve release by yourself.
+
+## Mandatory Rules
+
+- Do not run E2E validation before relevant implementation stories are done.
+- Do not mark E2E valid if any required lifecycle stage has no evidence.
+- Do not hide partial validation.
+- Do not treat unit/story validation as full E2E validation.
+- Do not approve release here; release approval belongs to `dew-release-readiness`.
+- If implementation stories are incomplete, trigger `HALT-17 — Implementation Not Ready`.
+- If E2E evidence is missing, trigger `HALT-18 — No Validation Evidence`.
+- If learning capture is missing for completed stories, trigger `HALT-19 — Learning Capture Missing`.
+
+## Workflow Architecture
+
+This workflow uses step-file architecture.
+
+Rules:
+- Read one step file at a time.
+- Do not skip steps.
+- Stop at E2E Validation Gate.
+- Preserve caveats.
+- Update frontmatter after each completed step.
+
+## Conventions
+
+- Bare paths resolve from the skill root.
+- `{skill-root}` resolves to this skill's installed directory.
+- `{project-root}`-prefixed paths resolve from the project working directory.
+- `{workflow.<name>}` resolves to fields in `customize.toml`'s `[workflow]` table.
+- `{doc_workspace}` is the run folder for this workflow.
+
+## On Activation
+
+1. Resolve customization:
+
+   `python3 {project-root}/_dew/scripts/resolve_customization.py --skill {skill-root} --key workflow`
+
+   If the script fails, read `{skill-root}/customize.toml` directly and use defaults.
+
+2. Load persistent facts from `{workflow.persistent_facts}`.
+
+3. Load `{project-root}/_dew/dew/config.yaml` if present.
+
+4. Load:
+   - `{workflow.e2e_validation_report_template}`
+   - `{workflow.e2e_path_rubric}`
+   - `{workflow.e2e_failure_rubric}`
+   - `{workflow.e2e_validation_matrix_template}`
+
+5. Greet user in configured language.
+
+## Execution
+
+Read fully and follow:
+
+`steps/step-01-init.md`

package/src/dew-skills/4-implementation/dew-e2e-validation/assets/e2e-failure-rubric.md

@@ -0,0 +1,41 @@
+# E2E Failure Rubric
+
+## Critical
+
+Blocks release.
+
+Examples:
+- primary source cannot be accessed
+- Bronze/Silver/Gold output missing
+- primary KPI wrong or missing
+- critical DQ rule fails
+- serving output missing
+- trust limitations missing for user-facing product
+
+## High
+
+Usually blocks release unless user explicitly accepts caveat.
+
+Examples:
+- partial source coverage
+- freshness evidence missing
+- schema snapshot missing
+- DQ evidence incomplete
+
+## Medium
+
+Can release with caveat.
+
+Examples:
+- non-critical metric missing
+- documentation incomplete
+- minor operational evidence weak
+
+## Low
+
+Does not block.
+
+Examples:
+- naming issue
+- formatting issue
+- non-critical note

package/src/dew-skills/4-implementation/dew-e2e-validation/assets/e2e-path-rubric.md

@@ -0,0 +1,35 @@
+# E2E Path Rubric
+
+An end-to-end validation path should cover the complete product lifecycle.
+
+## Required lifecycle stages
+
+1. Source access
+2. Raw/Bronze ingestion
+3. Schema snapshot
+4. Silver transformation
+5. Gold model / KPI computation
+6. Data quality checks
+7. Serving output
+8. Trust and limitations statement
+9. Operational logs / observability
+
+## Pass
+
+Use when every required stage has evidence and no blocking caveat remains.
+
+## Pass with caveats
+
+Use when output is usable, but known limitations are documented and accepted.
+
+## Partial
+
+Use when only some lifecycle stages were validated.
+
+## Fail
+
+Use when primary data path, KPI, DQ, or serving output is invalid.
+
+## Not run
+
+Use when no E2E validation command/procedure/evidence exists.