@leeovery/claude-technical-workflows 2.0.44 → 2.0.45

package/commands/workflow/start-specification.md

@@ -388,8 +388,41 @@ Present the groupings with FULL status information.
 
 For each grouping, show:
 - The grouping name
-- Whether a specification already exists for this grouping
-- Each discussion in the grouping and whether it has an individual spec
+- Whether a specification already exists for this grouping (and its effective status)
+- Each discussion in the grouping and its incorporation status
+
+### Determining Discussion Status Within a Grouping
+
+For each grouping, first check if a grouped specification exists:
+1. Convert the grouping name to kebab-case (lowercase, spaces to hyphens)
+2. Check if `docs/workflow/specification/{kebab-name}.md` exists
+3. If it exists, get its `sources` array from the discovery output
+
+The sources array uses object format with explicit status tracking:
+```yaml
+sources:
+  - name: topic-a
+    status: incorporated
+  - name: topic-b
+    status: pending
+```
+
+**If a grouped spec exists for the grouping:**
+
+For each discussion in the grouping:
+1. Look up the discussion in the spec's `sources` array (by `name` field)
+2. If found → use the source's `status` field (`incorporated` or `pending`)
+3. If NOT found → status is `"pending"` (new source not yet added to spec)
+
+Calculate the **effective spec status**:
+- If ALL discussions in the grouping have `status: incorporated` → use the spec's actual status from file
+- If ANY discussion has `status: pending` OR is not in sources → effective status is `"needs update"`
+
+**If NO grouped spec exists:**
+
+For each discussion in the grouping:
+- If the discussion has an individual spec (`has_individual_spec: true`) → status is `"spec: {spec_status}"`
+- If the discussion has no spec → status is `"ready"`
 
 **Format:**
 
@@ -398,19 +431,20 @@ For each grouping, show:
 
 Recommended Groupings:
 
-### 1. {Grouping Name} {if spec exists: "(spec: {spec_status})"}
+### 1. {Grouping Name} (spec: {effective_status})
 | Discussion | Status |
 |------------|--------|
-| {topic-a} | discussion only |
-| {topic-b} | spec: {spec_status} |
-| {topic-c} | discussion only |
+| {topic-a} | incorporated |
+| {topic-b} | incorporated |
+| {topic-c} | pending |
 
 Coupling: {explanation}
 
 ### 2. {Another Grouping}
 | Discussion | Status |
 |------------|--------|
-| {topic-d} | discussion only |
+| {topic-d} | ready |
+| {topic-e} | spec: in-progress |
 
 Coupling: {explanation}
 
@@ -429,6 +463,13 @@ How would you like to proceed?
 (Enter 'refresh' to re-analyze)
 ```
 
+**Status Legend:**
+- `incorporated` - Discussion content has been woven into the grouped specification
+- `pending` - Discussion is part of the group but not yet incorporated into the spec
+- `ready` - Discussion has no spec yet (ready to be specified)
+- `spec: {status}` - Discussion has its own individual specification
+- `needs update` - Grouped spec exists but has pending sources to incorporate
+
 **STOP.** Wait for user to choose.
 
 → Based on choice, proceed to **Step 8**.
@@ -636,7 +677,9 @@ Proceed? (y/n)
 ```
 {Creating / Continuing} specification: {topic}
 
-Source: docs/workflow/discussion/{topic}.md
+Sources:
+- docs/workflow/discussion/{topic}.md
+
 Output: docs/workflow/specification/{topic}.md
 
 Proceed? (y/n)
@@ -680,7 +723,9 @@ Invoke the [technical-specification](../../skills/technical-specification/SKILL.
 ```
 Specification session for: {topic}
 
-Source: docs/workflow/discussion/{topic}.md
+Sources:
+- docs/workflow/discussion/{topic}.md
+
 Output: docs/workflow/specification/{topic}.md
 
 Additional context: {summary of user's answers from Step 10}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@leeovery/claude-technical-workflows",
-  "version": "2.0.44",
+  "version": "2.0.45",
   "description": "Technical workflow skills & commands for Claude Code",
   "license": "MIT",
   "author": "Lee Overy <me@leeovery.com>",

package/scripts/discovery-for-specification.sh

@@ -35,9 +35,8 @@ extract_array_field() {
     local file="$1"
     local field="$2"
     local result
-    # Look for field followed by array items (- item), excluding --- delimiters
-    result=$(sed -n '/^---$/,/^---$/p' "$file" 2>/dev/null | \
-        grep -v "^---$" | \
+    # Look for field followed by array items (- item), within frontmatter only
+    result=$(awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null | \
         sed -n "/^${field}:/,/^[a-z_]*:/p" | \
         grep "^[[:space:]]*-" | \
         sed 's/^[[:space:]]*-[[:space:]]*//' | \
@@ -46,6 +45,62 @@ extract_array_field() {
     echo "$result"
 }
 
+# Helper: Extract sources with status from object format
+# Outputs YAML-formatted source entries with name and status
+# Usage: extract_sources_with_status <file>
+#
+# Note: This only handles the object format. Legacy simple array format
+# is converted by migration 004 before discovery runs.
+extract_sources_with_status() {
+    local file="$1"
+    local in_sources=false
+    local current_name=""
+    local current_status=""
+
+    # Read frontmatter and parse sources block
+    while IFS= read -r line; do
+        # Detect start of sources block
+        if [[ "$line" =~ ^sources: ]]; then
+            in_sources=true
+            continue
+        fi
+
+        # Detect end of sources block (next top-level field)
+        if $in_sources && [[ "$line" =~ ^[a-z_]+: ]] && [[ ! "$line" =~ ^[[:space:]] ]]; then
+            # Output last source if pending
+            if [ -n "$current_name" ]; then
+                echo "      - name: \"$current_name\""
+                echo "        status: \"${current_status:-incorporated}\""
+            fi
+            break
+        fi
+
+        if $in_sources; then
+            # Object format: "  - name: value"
+            if [[ "$line" =~ ^[[:space:]]*-[[:space:]]*name:[[:space:]]*(.+)$ ]]; then
+                # Output previous source if exists
+                if [ -n "$current_name" ]; then
+                    echo "      - name: \"$current_name\""
+                    echo "        status: \"${current_status:-incorporated}\""
+                fi
+                current_name="${BASH_REMATCH[1]}"
+                current_name=$(echo "$current_name" | sed 's/^"//' | sed 's/"$//' | xargs)
+                current_status=""
+            # Status line: "    status: value"
+            elif [[ "$line" =~ ^[[:space:]]*status:[[:space:]]*(.+)$ ]]; then
+                current_status="${BASH_REMATCH[1]}"
+                current_status=$(echo "$current_status" | sed 's/^"//' | sed 's/"$//' | xargs)
+            fi
+        fi
+    done < <(awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null)
+
+    # Output last source if pending (end of frontmatter)
+    if [ -n "$current_name" ]; then
+        echo "      - name: \"$current_name\""
+        echo "        status: \"${current_status:-incorporated}\""
+    fi
+}
+
 # Start YAML output
 echo "# Specification Command State Discovery"
 echo "# Generated: $(date -Iseconds)"
@@ -101,7 +156,6 @@ if [ -d "$SPEC_DIR" ] && [ -n "$(ls -A "$SPEC_DIR" 2>/dev/null)" ]; then
         status=${status:-"active"}
 
         superseded_by=$(extract_field "$file" "superseded_by")
-        sources=$(extract_array_field "$file" "sources")
 
         echo "  - name: \"$name\""
         echo "    status: \"$status\""
@@ -110,11 +164,11 @@ if [ -d "$SPEC_DIR" ] && [ -n "$(ls -A "$SPEC_DIR" 2>/dev/null)" ]; then
             echo "    superseded_by: \"$superseded_by\""
         fi
 
-        if [ -n "$sources" ]; then
+        # Extract sources with status (handles both old and new format)
+        sources_output=$(extract_sources_with_status "$file")
+        if [ -n "$sources_output" ]; then
             echo "    sources:"
-            for src in $sources; do
-                echo "      - \"$src\""
-            done
+            echo "$sources_output"
         fi
     done
 else

package/scripts/migrate.sh

@@ -116,8 +116,10 @@ for script in "${MIGRATION_SCRIPTS[@]}"; do
     MIGRATIONS_RUN=$((MIGRATIONS_RUN + 1))
 done
 
-# Only output if files were actually updated
+# Report results
 if [ "$FILES_UPDATED" -gt 0 ]; then
     echo ""
     echo "$FILES_UPDATED file(s) migrated. Review with \`git diff\`, then proceed."
+else
+    echo "[SKIP] No changes needed"
 fi

package/scripts/migrations/003-planning-frontmatter.sh

@@ -84,16 +84,18 @@ for file in "$PLAN_DIR"/*.md; do
     topic_kebab=$(basename "$file" .md)
 
     # Extract format from existing frontmatter (if present)
-    format_value=$(sed -n '/^---$/,/^---$/p' "$file" 2>/dev/null | grep "^format:" | sed 's/^format:[[:space:]]*//' | xargs || echo "")
+    # Use awk to extract only the first frontmatter block (between first pair of --- delimiters)
+    # This avoids matching --- horizontal rules in body content
+    format_value=$(awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null | grep "^format:" | sed 's/^format:[[:space:]]*//' | xargs || echo "")
     if [ -z "$format_value" ]; then
         format_value="MISSING"  # No default - missing format is an error
     fi
 
     # Extract plan_id from existing frontmatter - could be 'epic' (beads) or 'project' (linear/backlog)
     # These are migrated to a unified 'plan_id' field
-    plan_id_value=$(sed -n '/^---$/,/^---$/p' "$file" 2>/dev/null | grep "^epic:" | sed 's/^epic:[[:space:]]*//' | xargs || echo "")
+    plan_id_value=$(awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null | grep "^epic:" | sed 's/^epic:[[:space:]]*//' | xargs || echo "")
     if [ -z "$plan_id_value" ]; then
-        plan_id_value=$(sed -n '/^---$/,/^---$/p' "$file" 2>/dev/null | grep "^project:" | sed 's/^project:[[:space:]]*//' | xargs || echo "")
+        plan_id_value=$(awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null | grep "^project:" | sed 's/^project:[[:space:]]*//' | xargs || echo "")
     fi
 
     # Extract status from **Status**: Value

package/scripts/migrations/004-sources-object-format.sh

@@ -0,0 +1,204 @@
+#!/usr/bin/env bash
+#
+# 004-sources-object-format.sh
+#
+# Migrates specification sources from simple array format to object format
+# with status tracking. Also ensures all specs have a sources field.
+#
+# Previous format (from 002-specification-frontmatter.sh):
+#   sources:
+#     - topic-a
+#     - topic-b
+#
+# New format:
+#   sources:
+#     - name: topic-a
+#       status: incorporated
+#     - name: topic-b
+#       status: incorporated
+#
+# Status values:
+#   - pending: Source selected but content not yet extracted
+#   - incorporated: Source content has been fully woven into the specification
+#
+# For existing sources, we assume "incorporated" since they were part of
+# the specification when it was created/worked on.
+#
+# For specs WITHOUT a sources field:
+#   - If a matching discussion exists (same filename), add it as incorporated
+#   - If no matching discussion, add empty sources: [] and report for user review
+#
+# This script is sourced by migrate.sh and has access to:
+#   - is_migrated "filepath" "migration_id"
+#   - record_migration "filepath" "migration_id"
+#   - report_update "filepath" "description"
+#   - report_skip "filepath"
+#
+
+MIGRATION_ID="004"
+SPEC_DIR="docs/workflow/specification"
+DISCUSSION_DIR="docs/workflow/discussion"
+
+# Skip if no specification directory
+if [ ! -d "$SPEC_DIR" ]; then
+    return 0
+fi
+
+# Helper: Extract ONLY the frontmatter content (between first pair of --- delimiters)
+# Documents may contain --- elsewhere (horizontal rules), so sed range matching
+# can return content beyond frontmatter. Use awk for precise first-block extraction.
+extract_frontmatter() {
+    local file="$1"
+    awk 'BEGIN{c=0} /^---$/{c++; if(c==2) exit; next} c==1{print}' "$file" 2>/dev/null
+}
+
+# Helper: Check if sources are already in object format
+# Returns 0 if already migrated (has "name:" entries), 1 if not
+sources_already_object_format() {
+    local file="$1"
+    # Look for "- name:" pattern within the sources block (frontmatter only)
+    # This indicates the new object format
+    # Using subshell with || false to ensure proper exit code without pipefail issues
+    ( extract_frontmatter "$file" | \
+        sed -n '/^sources:/,/^[a-z_]*:/p' | \
+        grep -q "^[[:space:]]*-[[:space:]]*name:" 2>/dev/null ) || return 1
+    return 0
+}
+
+# Helper: Extract sources array items (simple string format)
+# Returns space-separated list of source names
+extract_simple_sources() {
+    local file="$1"
+    # Extract sources from frontmatter only, then find the sources block
+    extract_frontmatter "$file" | \
+        sed -n '/^sources:/,/^[a-z_]*:/p' | \
+        grep -v "^sources:" | \
+        grep -v "^[a-z_]*:" | \
+        { grep "^[[:space:]]*-[[:space:]]" || true; } | \
+        { grep -v "name:" || true; } | \
+        sed 's/^[[:space:]]*-[[:space:]]*//' | \
+        sed 's/^"//' | \
+        sed 's/"$//' | \
+        tr '\n' ' ' | \
+        sed 's/[[:space:]]*$//' || true
+}
+
+# Process each specification file
+for file in "$SPEC_DIR"/*.md; do
+    [ -f "$file" ] || continue
+
+    # Check if already migrated via tracking
+    if is_migrated "$file" "$MIGRATION_ID"; then
+        report_skip "$file"
+        continue
+    fi
+
+    # Check if file has YAML frontmatter
+    if ! head -1 "$file" 2>/dev/null | grep -q "^---$"; then
+        record_migration "$file" "$MIGRATION_ID"
+        report_skip "$file"
+        continue
+    fi
+
+    # Check if file has sources field at all
+    has_sources_field=false
+    if grep -q "^sources:" "$file" 2>/dev/null; then
+        has_sources_field=true
+    fi
+
+    # If sources field exists, check if already in object format
+    if $has_sources_field && sources_already_object_format "$file"; then
+        record_migration "$file" "$MIGRATION_ID"
+        report_skip "$file"
+        continue
+    fi
+
+    #
+    # Build new sources block in object format
+    #
+    new_sources_block="sources:"
+    sources_added=false
+
+    if $has_sources_field; then
+        # Extract existing sources from simple array format
+        sources=$(extract_simple_sources "$file")
+
+        for src in $sources; do
+            # Clean the source name (trim whitespace, sed avoids xargs quote issues)
+            src=$(echo "$src" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
+            if [ -n "$src" ]; then
+                new_sources_block="${new_sources_block}
+  - name: $src
+    status: incorporated"
+                sources_added=true
+            fi
+        done
+    else
+        # No sources field - check for matching discussion by filename
+        spec_name=$(basename "$file" .md)
+        discussion_file="$DISCUSSION_DIR/${spec_name}.md"
+
+        if [ -f "$discussion_file" ]; then
+            # Matching discussion found - add it as incorporated
+            new_sources_block="${new_sources_block}
+  - name: $spec_name
+    status: incorporated"
+            sources_added=true
+        fi
+    fi
+
+    # If no sources were added, use empty array format
+    if ! $sources_added; then
+        new_sources_block="sources: []"
+        # Echo info for Claude to prompt user about unmatched specs
+        spec_name=$(basename "$file" .md)
+        echo "MIGRATION_INFO: Specification '$spec_name' has no matching discussion. Sources field set to empty - please review and add sources manually."
+    fi
+
+    #
+    # Update sources block in file
+    #
+
+    # Extract frontmatter (only the first block between --- delimiters)
+    frontmatter=$(extract_frontmatter "$file")
+
+    if $has_sources_field; then
+        # Remove old sources block from frontmatter
+        # First, remove lines from "sources:" until the next top-level field or end of frontmatter
+        new_frontmatter=$(echo "$frontmatter" | awk '
+/^sources:/ { skip=1; next }
+/^[a-z_]+:/ && skip { skip=0 }
+skip == 0 { print }
+')
+    else
+        # No existing sources field - use frontmatter as-is
+        new_frontmatter="$frontmatter"
+    fi
+
+    # Add new sources block at the end
+    new_frontmatter="${new_frontmatter}
+${new_sources_block}"
+
+    # Extract content after frontmatter (everything after the second ---)
+    # Uses awk to skip only the first two --- delimiters, preserving any --- in body content
+    content=$(awk '/^---$/ && c<2 {c++; next} c>=2 {print}' "$file")
+
+    # Write new file
+    {
+        echo "---"
+        echo "$new_frontmatter"
+        echo "---"
+        echo "$content"
+    } > "$file"
+
+    record_migration "$file" "$MIGRATION_ID"
+
+    # Report appropriate message based on what was done
+    if $has_sources_field; then
+        report_update "$file" "converted sources to object format"
+    elif $sources_added; then
+        report_update "$file" "added sources field with matching discussion"
+    else
+        report_update "$file" "added empty sources field (no matching discussion found)"
+    fi
+done

package/skills/technical-specification/references/specification-guide.md

@@ -148,6 +148,11 @@ topic: {topic-name}
 status: in-progress
 type: feature
 date: YYYY-MM-DD  # Use today's actual date
+sources:
+  - name: discussion-one
+    status: incorporated
+  - name: discussion-two
+    status: pending
 ---
 
 # Specification: [Topic Name]
@@ -169,6 +174,45 @@ date: YYYY-MM-DD  # Use today's actual date
 - **status**: `in-progress` (building) or `concluded` (complete)
 - **type**: `feature` (something to build) or `cross-cutting` (patterns/policies)
 - **date**: Last updated date
+- **sources**: Array of source discussions with incorporation status (see below)
+
+### Sources and Incorporation Status
+
+**All specifications must track their sources**, even when built from a single discussion. This enables proper tracking when additional discussions are later added to the same grouping.
+
+When a specification is built from discussion(s), track each source with its incorporation status:
+
+```yaml
+sources:
+  - name: auth-flow
+    status: incorporated
+  - name: api-design
+    status: pending
+```
+
+**Status values:**
+- `pending` - Source has been selected for this specification but content extraction is not complete
+- `incorporated` - Source content has been fully extracted and woven into the specification
+
+**When to update source status:**
+
+1. **When creating the specification**: All sources start as `pending`
+2. **After completing exhaustive extraction from a source**: Mark that source as `incorporated`
+3. **When adding a new source to an existing spec**: Add it with `status: pending`
+
+**How to determine if a source is incorporated:**
+
+A source is `incorporated` when you have:
+- Performed exhaustive extraction (reviewed ALL content in the source for relevant material)
+- Presented and logged all relevant content from that source
+- No more content from that source needs to be extracted
+
+**Important**: The specification's overall `status: concluded` should only be set when:
+- All sources are marked as `incorporated`
+- Both review phases are complete
+- User has signed off
+
+If a new source is added to a concluded specification (via grouping analysis), the specification effectively needs updating - even if the file still says `status: concluded`, the presence of `pending` sources indicates work remains.
 
 ## Specification Types