@bhargavvc/sdd-cc 1.30.1 → 1.42.3

package/sdd/workflows/extract-learnings.md

@@ -0,0 +1,242 @@
+<purpose>
+Extract decisions, lessons learned, patterns discovered, and surprises encountered from completed phase artifacts into a structured LEARNINGS.md file. Captures institutional knowledge that would otherwise be lost between phases.
+</purpose>
+
+<required_reading>
+Read all files referenced by the invoking prompt's execution_context before starting.
+</required_reading>
+
+<objective>
+Analyze completed phase artifacts (PLAN.md, SUMMARY.md, VERIFICATION.md, UAT.md, STATE.md) and extract structured learnings into 4 categories: decisions, lessons, patterns, and surprises. Each extracted item includes source attribution. The output is a LEARNINGS.md file with YAML frontmatter containing metadata about the extraction.
+</objective>
+
+<process>
+
+<step name="initialize">
+Parse arguments and load project state:
+
+```bash
+INIT=$(sdd-sdk query init.phase-op "${PHASE_ARG}")
+if [[ "$INIT" == @file:* ]]; then INIT=$(cat "${INIT#@file:}"); fi
+```
+
+Parse from init JSON: `phase_found`, `phase_dir`, `phase_number`, `phase_name`, `padded_phase`.
+
+If phase not found, exit with error: "Phase {PHASE_ARG} not found."
+</step>
+
+<step name="collect_artifacts">
+Read the phase artifacts. PLAN.md and SUMMARY.md are required; VERIFICATION.md, UAT.md, and STATE.md are optional.
+
+**Required artifacts:**
+- `${PHASE_DIR}/*-PLAN.md` — all plan files for the phase
+- `${PHASE_DIR}/*-SUMMARY.md` — all summary files for the phase
+
+If PLAN.md or SUMMARY.md files are not found or missing, exit with error: "Required artifacts missing. PLAN.md and SUMMARY.md are required for learning extraction."
+
+**Optional artifacts (read if available, skip if not found):**
+- `${PHASE_DIR}/*-VERIFICATION.md` — verification results
+- `${PHASE_DIR}/*-UAT.md` — user acceptance test results
+- `.planning/STATE.md` — project state with decisions and blockers
+
+Track which optional artifacts are missing for the `missing_artifacts` frontmatter field.
+</step>
+
+<step name="extract-learnings">
+Analyze all collected artifacts and extract learnings into 4 categories:
+
+### 1. Decisions
+Technical and architectural decisions made during the phase. Look for:
+- Explicit decisions documented in PLAN.md or SUMMARY.md
+- Technology choices and their rationale
+- Trade-offs that were evaluated
+- Design decisions recorded in STATE.md
+
+Each decision entry must include:
+- **What** was decided
+- **Why** it was decided (rationale)
+- **Source:** attribution to the artifact where the decision was found (e.g., "Source: 03-01-PLAN.md")
+
+### 2. Lessons
+Things learned during execution that were not known beforehand. Look for:
+- Unexpected complexity in SUMMARY.md
+- Issues discovered during verification in VERIFICATION.md
+- Failed approaches documented in SUMMARY.md
+- UAT feedback that revealed gaps
+
+Each lesson entry must include:
+- **What** was learned
+- **Context** for the lesson
+- **Source:** attribution to the originating artifact
+
+### 3. Patterns
+Reusable patterns, approaches, or techniques discovered. Look for:
+- Successful implementation patterns in SUMMARY.md
+- Testing patterns from VERIFICATION.md or UAT.md
+- Workflow patterns that worked well
+- Code organization patterns from PLAN.md
+
+Each pattern entry must include:
+- **Pattern** name/description
+- **When to use** it
+- **Source:** attribution to the originating artifact
+
+### 4. Surprises
+Unexpected findings, behaviors, or outcomes. Look for:
+- Things that took longer or shorter than estimated
+- Unexpected dependencies or interactions
+- Edge cases not anticipated in planning
+- Performance or behavior that differed from expectations
+
+Each surprise entry must include:
+- **What** was surprising
+- **Impact** of the surprise
+- **Source:** attribution to the originating artifact
+</step>
+
+<step name="capture_thought_integration">
+**What this step is:** `capture_thought` is an **optional convention**, not a bundled SDD tool. SDD does not ship one and does not require one. The step is a hook for users who run a memory / knowledge-base MCP server (for example ExoCortex-style servers, `claude-mem`, or `mem0`-style servers) that exposes a tool with this exact name. If any MCP server in the current session provides a `capture_thought` tool with the signature below, each extracted learning is routed through it with metadata. If no such tool is present, the step is a silent no-op — `LEARNINGS.md` is always the primary output.
+
+**Detection:** Check whether a tool named `capture_thought` is available in the current session. Do not assume any specific MCP server is connected.
+
+**If available**, call once per extracted learning:
+
+```
+capture_thought({
+  category: "decision" | "lesson" | "pattern" | "surprise",
+  phase: PHASE_NUMBER,
+  content: LEARNING_TEXT,
+  source: ARTIFACT_NAME
+})
+```
+
+**If not available** (no MCP server in the session exposes this tool, or the runtime does not support it), skip the step silently and continue. The workflow must not fail or warn — this is expected behavior for users who do not run a knowledge-base MCP.
+</step>
+
+<step name="write_learnings">
+Write the LEARNINGS.md file to the phase directory. If a previous LEARNINGS.md exists, overwrite it (replace the file entirely).
+
+Output path: `${PHASE_DIR}/${PADDED_PHASE}-LEARNINGS.md`
+
+The file must have YAML frontmatter with these fields:
+```yaml
+---
+phase: {PHASE_NUMBER}
+phase_name: "{PHASE_NAME}"
+project: "{PROJECT_NAME}"
+generated: "{ISO_DATE}"
+counts:
+  decisions: {N}
+  lessons: {N}
+  patterns: {N}
+  surprises: {N}
+missing_artifacts:
+  - "{ARTIFACT_NAME}"
+---
+```
+
+Individual items may carry an optional `graduated:` annotation (added by `graduation.md` when a cluster is promoted):
+```markdown
+**Graduated:** {target-file}:{ISO_DATE}
+```
+This annotation is appended after the item's existing fields and prevents the item from being re-surfaced in future graduation scans. Do not add this field during extraction — it is written only by the graduation workflow.
+
+The body follows this structure:
+```markdown
+# Phase {PHASE_NUMBER} Learnings: {PHASE_NAME}
+
+## Decisions
+
+### {Decision Title}
+{What was decided}
+
+**Rationale:** {Why}
+**Source:** {artifact file}
+
+---
+
+## Lessons
+
+### {Lesson Title}
+{What was learned}
+
+**Context:** {context}
+**Source:** {artifact file}
+
+---
+
+## Patterns
+
+### {Pattern Name}
+{Description}
+
+**When to use:** {applicability}
+**Source:** {artifact file}
+
+---
+
+## Surprises
+
+### {Surprise Title}
+{What was surprising}
+
+**Impact:** {impact description}
+**Source:** {artifact file}
+```
+</step>
+
+<step name="update_state">
+Update STATE.md to reflect the learning extraction:
+
+```bash
+sdd-sdk query state.update "Last Activity" "$(date +%Y-%m-%d)"
+```
+</step>
+
+<step name="report">
+```
+---------------------------------------------------------------
+
+## Learnings Extracted: Phase {X} — {Name}
+
+Decisions:  {N}
+Lessons:    {N}
+Patterns:   {N}
+Surprises:  {N}
+Total:      {N}
+
+Output: {PHASE_DIR}/{PADDED_PHASE}-LEARNINGS.md
+
+Missing artifacts: {list or "none"}
+
+Next steps:
+- Review extracted learnings for accuracy
+- /sdd:progress — see overall project state
+- /sdd:execute-phase {next} — continue to next phase
+
+---------------------------------------------------------------
+```
+</step>
+
+</process>
+
+<success_criteria>
+- [ ] Phase artifacts located and read successfully
+- [ ] All 4 categories extracted: decisions, lessons, patterns, surprises
+- [ ] Each extracted item has source attribution
+- [ ] LEARNINGS.md written with correct YAML frontmatter
+- [ ] Missing optional artifacts tracked in frontmatter
+- [ ] capture_thought integration attempted if tool available
+- [ ] STATE.md updated with extraction activity
+- [ ] User receives summary report
+</success_criteria>
+
+<critical_rules>
+- PLAN.md and SUMMARY.md are required — exit with clear error if missing
+- VERIFICATION.md, UAT.md, and STATE.md are optional — extract from them if present, skip gracefully if not found
+- Every extracted learning must have source attribution back to the originating artifact
+- Running extract-learnings twice on the same phase must overwrite (replace) the previous LEARNINGS.md, not append
+- Do not fabricate learnings — only extract what is explicitly documented in artifacts
+- If capture_thought is unavailable, the workflow must not fail — graceful degradation to file-only output
+- LEARNINGS.md frontmatter must include counts for all 4 categories and list any missing_artifacts
+</critical_rules>

package/sdd/workflows/forensics.md

@@ -115,6 +115,18 @@ For each phase that should be complete:
 - SUMMARY.md missing → phase was not properly closed
 - VERIFICATION.md missing → quality check was skipped
 
+### Partial-plan Drift Detection
+
+**Signal:** commits exist but SUMMARY.md is missing for the current or recently
+active plan.
+
+Run the same comparison as the execute-phase safe-resume verifier: identify the
+active plan from STATE.md/phase artifacts, search git history for that plan id,
+then compare against the expected SUMMARY.md path. If production commits exist
+but SUMMARY.md is missing, flag a high-confidence partial-plan drift anomaly.
+This usually means an executor was interrupted after implementation commits but
+before atomic close-out.
+
 ### Abandoned Work Detection
 
 **Signal:** Large gap between last commit and current time, with STATE.md showing mid-execution.
@@ -244,13 +256,14 @@ If actionable anomalies were found (HIGH or MEDIUM confidence):
 If confirmed:
 ```bash
 # Check if "bug" label exists before using it
-BUG_LABEL=$(gh label list --search "bug" --json name -q '.[0].name' 2>/dev/null)
+BUG_LABEL=$(gh label list --repo bhargavvc/sdd-cc --search "bug" --json name -q '.[0].name' 2>/dev/null)
 LABEL_FLAG=""
 if [ -n "$BUG_LABEL" ]; then
   LABEL_FLAG="--label bug"
 fi
 
 gh issue create \
+  --repo bhargavvc/sdd-cc \
   --title "bug: {concise description from anomaly}" \
   $LABEL_FLAG \
   --body "{formatted findings from report}"
@@ -259,7 +272,7 @@ gh issue create \
 ## Step 8: Update STATE.md
 
 ```bash
-sdd-tools.cjs state record-session \
-  --stopped-at "Forensic investigation complete" \
-  --resume-file ".planning/forensics/report-{timestamp}.md"
+sdd-sdk query state.record-session "" \
+  "Forensic investigation complete" \
+  ".planning/forensics/report-{timestamp}.md"
 ```

package/sdd/workflows/graduation.md

@@ -0,0 +1,195 @@
+# graduation.md — LEARNINGS.md Cross-Phase Graduation Helper
+
+**Invoked by:** `transition.md` step `graduation_scan`. Never invoked directly by users.
+
+This workflow clusters recurring items across the last N phases' LEARNINGS.md files and surfaces promotion candidates to the developer via HITL. No item is promoted without explicit developer approval.
+
+---
+
+## Configuration
+
+Read from project config (`config.json`):
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `features.graduation` | `true` | Master on/off switch. `false` skips silently. |
+| `features.graduation_window` | `5` | How many prior phases to scan |
+| `features.graduation_threshold` | `3` | Minimum cluster size to surface |
+
+---
+
+## Step 1: Guard Checks
+
+```bash
+GRADUATION_ENABLED=$(sdd-sdk query config-get features.graduation 2>/dev/null || echo "true")
+GRADUATION_WINDOW=$(sdd-sdk query config-get features.graduation_window 2>/dev/null || echo "5")
+GRADUATION_THRESHOLD=$(sdd-sdk query config-get features.graduation_threshold 2>/dev/null || echo "3")
+```
+
+**Skip silently (print nothing) if:**
+- `features.graduation` is `false`
+- Fewer than `graduation_threshold` completed prior phases exist (not enough data)
+
+**Skip silently (print nothing) if total items across all LEARNINGS.md files in the window is fewer than 5.**
+
+---
+
+## Step 2: Collect LEARNINGS.md Files
+
+Find LEARNINGS.md files from the last N completed phases (excluding the phase currently completing):
+
+```bash
+find .planning/phases -name "*-LEARNINGS.md" | sort | tail -n "$GRADUATION_WINDOW"
+```
+
+For each file found:
+1. Parse the four category sections: `## Decisions`, `## Lessons`, `## Patterns`, `## Surprises`
+2. Extract each `### Item Title` + body as a single item record: `{ category, title, body, source_phase, source_file }`
+3. **Skip items that already contain `**Graduated:**`** — they have been promoted and must not re-surface
+
+---
+
+## Step 3: Cluster by Lexical Similarity
+
+For each category independently, cluster items using Jaccard similarity on tokenized title+body:
+
+**Tokenization:** lowercase, strip punctuation, split on whitespace, remove stop words (a, an, the, is, was, in, on, at, to, for, of, and, or, but, with, from, that, this, by, as).
+
+**Jaccard similarity:** `|A ∩ B| / |A ∪ B|` where A and B are token sets. Two items are in the same cluster if similarity ≥ 0.25.
+
+**Clustering algorithm:** single-pass greedy — process items in phase order; add to the first cluster whose centroid (union of all cluster tokens) has similarity ≥ 0.25 with the new item; otherwise start a new cluster.
+
+**Cluster size filter:** only surface clusters with distinct source phases ≥ `graduation_threshold` (not just total items — same item repeated in one phase still counts as 1 distinct phase).
+
+---
+
+## Step 4: Check graduation_backlog in STATE.md
+
+Read `.planning/STATE.md` `graduation_backlog` section (if present). Format:
+
+```yaml
+graduation_backlog:
+  - cluster_id: "{sha256-of-cluster-title}"
+    status: "dismissed"   # or "deferred"
+    deferred_until: "phase-N"  # only for deferred entries
+    cluster_title: "{representative title}"
+```
+
+**Skip any cluster whose `cluster_id` matches a `dismissed` entry.**
+
+**Skip any cluster whose `cluster_id` matches a `deferred` entry where `deferred_until` phase has not yet completed.**
+
+---
+
+## Step 5: Surface Promotion Candidates
+
+For each qualifying cluster, determine the suggested target file:
+
+| Category | Suggested Target |
+|----------|-----------------|
+| `decisions` | `PROJECT.md` — append under `## Validated Decisions` (create section if absent) |
+| `patterns` | `PATTERNS.md` — append under the appropriate category section (create file if absent) |
+| `lessons` | `PROJECT.md` — append under `## Invariants` (create section if absent) |
+| `surprises` | Flag for human review — if genuinely surprising 3+ times, something structural is wrong |
+
+Print the graduation report:
+
+```text
+📚 Graduation scan across phases {M}–{N}:
+
+  HIGH RECURRENCE ({K}/{WINDOW} phases)
+  ├─ Cluster: "{representative title}"
+  ├─ Category: {category}
+  ├─ Sources: {list of NN-LEARNINGS filenames}
+  └─ Suggested target: {target file} § {section}
+
+  [repeat for each qualifying cluster, ordered HIGH→LOW recurrence]
+
+For each cluster above, choose an action:
+  P = Promote now   D = Defer (re-surface next transition)   X = Dismiss (never re-surface)   A = Defer all remaining
+```
+
+---
+
+## Step 6: HITL — Process Each Cluster
+
+For each cluster (in order from Step 5), ask the developer:
+
+```text
+Cluster: "{title}" [{category}, {K} phases] → {target}
+Action [P/D/X/A]:
+```
+
+Use `AskUserQuestion` (or equivalent HITL primitive for the current runtime). If `TEXT_MODE` is true, display the cluster question as plain text and accept typed input. Accept single-character input: `P`, `D`, `X`, `A` (case-insensitive).
+
+**On `P` (Promote now):**
+
+1. Read the target file (or create it with a standard header if absent)
+2. Append the cluster entry under the suggested section:
+   ```markdown
+   ### {Cluster representative title}
+   {Merged body — combine unique sentences across cluster items}
+
+   **Sources:** Phase {A}, Phase {B}, Phase {C}
+   **Promoted:** {ISO_DATE}
+   ```
+3. For each source LEARNINGS.md item in the cluster, append `**Graduated:** {target-file}:{ISO_DATE}` after its last existing field
+4. Commit both the target file and all annotated LEARNINGS.md files in a single atomic commit:
+   `docs(learnings): graduate "{cluster title}" to {target-file}`
+
+**On `D` (Defer):**
+
+Write to `.planning/STATE.md` under `graduation_backlog`:
+```yaml
+- cluster_id: "{sha256}"
+  status: "deferred"
+  deferred_until: "phase-{NEXT_PHASE_NUMBER}"
+  cluster_title: "{title}"
+```
+
+**On `X` (Dismiss):**
+
+Write to `.planning/STATE.md` under `graduation_backlog`:
+```yaml
+- cluster_id: "{sha256}"
+  status: "dismissed"
+  cluster_title: "{title}"
+```
+
+**On `A` (Defer all):**
+
+Defer the current cluster (same as `D`) and skip all remaining clusters for this run, deferring each to the next transition. Print:
+```text
+[graduation: deferred all remaining clusters to next transition]
+```
+Then proceed directly to Step 7.
+
+---
+
+## Step 7: Completion Report
+
+After processing all clusters, print:
+
+```text
+Graduation complete: {promoted} promoted, {deferred} deferred, {dismissed} dismissed.
+```
+
+If no clusters qualified (all filtered by backlog or threshold), print:
+```text
+[graduation: no qualifying clusters in phases {M}–{N}]
+```
+
+---
+
+## First-Run Behaviour
+
+On the first transition after upgrading to a version that includes this workflow, all extant LEARNINGS.md files may produce a large batch of candidates at once. A `[Defer all]` shorthand is available: if the developer enters `A` at any cluster prompt, all remaining clusters for this run are deferred to the next transition.
+
+---
+
+## No-Op Conditions (silent skip)
+
+- `features.graduation = false`
+- Fewer than `graduation_threshold` prior phases with LEARNINGS.md
+- Total items < 5 across the window
+- All qualifying clusters are in `graduation_backlog` as dismissed

package/sdd/workflows/health.md

@@ -11,21 +11,60 @@ Read all files referenced by the invoking prompt's execution_context before star
 <step name="parse_args">
 **Parse arguments:**
 
-Check if `--repair` flag is present in the command arguments.
+Check if `--repair`, `--backfill`, or `--context` flags are present in the command arguments.
 
 ```
 REPAIR_FLAG=""
+BACKFILL_FLAG=""
+CONTEXT_MODE=""
 if arguments contain "--repair"; then
   REPAIR_FLAG="--repair"
 fi
+if arguments contain "--backfill"; then
+  BACKFILL_FLAG="--backfill"
+fi
+if arguments contain "--context"; then
+  CONTEXT_MODE="true"
+fi
 ```
+
+If `CONTEXT_MODE` is set, jump to the `context_check` step and skip the
+integrity validation steps. The two modes are orthogonal — context utilization
+has nothing to do with `.planning/` directory health.
+</step>
+
+<step name="context_check">
+**Run only when `--context` is set.**
+
+The model running this workflow self-reports the current session's
+approximate `tokensUsed` and the active model's `contextWindow`. Use the values
+visible in your runtime (Claude Code's `/context` slash command output, or the
+model's own session telemetry). If the runtime exposes neither, prompt the user
+once via AskUserQuestion for both numbers.
+
+**TEXT_MODE fallback:** when `text_mode` is true (config or `--text` flag) the
+runtime is non-Claude (Codex, Gemini, etc.) and `AskUserQuestion` is not
+available — replace the prompt with a plain-text two-question sequence
+("Approximate tokens used? Context window size?") and read the answers as
+plain text from the user's response.
+
+```bash
+sdd-sdk query validate.context \
+  --tokens-used "$TOKENS_USED" \
+  --context-window "$CONTEXT_WINDOW"
+```
+
+The query prints a one-line status (`Context utilization: NN% (state)`) plus
+a recommendation line for the warning and critical states. Print the SDK
+output verbatim and end the workflow — do **not** mix in `.planning/`
+health output, the two modes are independent diagnostics.
 </step>
 
 <step name="run_health_check">
 **Run health validation:**
 
 ```bash
-node "$HOME/.claude/sdd/bin/sdd-tools.cjs" validate health $REPAIR_FLAG
+sdd-sdk query validate.health $REPAIR_FLAG $BACKFILL_FLAG
 ```
 
 Parse JSON output:
@@ -112,7 +151,7 @@ If yes, re-run with --repair flag and display results.
 Re-run health check without --repair to confirm issues are resolved:
 
 ```bash
-node "$HOME/.claude/sdd/bin/sdd-tools.cjs" validate health
+sdd-sdk query validate.health
 ```
 
 Report final status.
@@ -138,6 +177,8 @@ Report final status.
 | W007 | warning | Phase on disk but not in ROADMAP | No |
 | W008 | warning | config.json: workflow.nyquist_validation absent (defaults to enabled but agents may skip) | Yes |
 | W009 | warning | Phase has Validation Architecture in RESEARCH.md but no VALIDATION.md | No |
+| W018 | warning | MILESTONES.md missing entry for archived milestone snapshot | Yes (`--backfill`) |
+| W019 | warning | Unrecognized .planning/ root file — not a canonical SDD artifact | No |
 | I001 | info | Plan without SUMMARY (may be in progress) | No |
 
 </error_codes>
@@ -150,6 +191,7 @@ Report final status.
 | resetConfig | Delete + recreate config.json | Loses custom settings |
 | regenerateState | Create STATE.md from ROADMAP structure when it is missing | Loses session history |
 | addNyquistKey | Add workflow.nyquist_validation: true to config.json | None — matches existing default |
+| backfillMilestones | Synthesize missing MILESTONES.md entries from `.planning/milestones/vX.Y-ROADMAP.md` snapshots | None — additive only; triggered by `--backfill` flag |
 
 **Not repairable (too risky):**
 - PROJECT.md, ROADMAP.md content