@ktpartners/dgs-platform 2.6.2

package/deliver-great-systems/workflows/import-spec.md

@@ -0,0 +1,381 @@
+<purpose>
+Import an external document and convert it into a structured 9-section PRD using AI-powered restructuring. Reads the source file, transforms its content into standard PRD sections with full content preservation, and presents the result for user review with revision support. This is the core conversion pipeline for the `/dgs:import-spec` command.
+</purpose>
+
+<context_tier>planning</context_tier>
+
+<required_reading>
+Read all files referenced by the invoking prompt's execution_context before starting.
+</required_reading>
+
+<process>
+
+<step name="initialize" priority="first">
+Load planning context:
+
+```bash
+INIT=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs init milestone-op)
+```
+
+Extract: `project_root`, `config_path` as needed by the workflow.
+
+Load planning-tier context files:
+
+```bash
+TIER_FILES=$(node "$HOME/.claude/deliver-great-systems/bin/dgs-tools.cjs" context load-tier planning --raw 2>/dev/null)
+```
+</step>
+
+<step name="parse_arguments">
+Extract the file path and optional flags from `$ARGUMENTS`.
+
+**File path extraction:**
+- The first positional argument (before any `--` flags) is the file path.
+- If no file path is provided, display an error and stop:
+  ```
+  Usage: /dgs:import-spec <file-path> [--ideas IDEA-1 IDEA-2 ...]
+  ```
+- The file path may be absolute or relative to the current working directory.
+- Store the resolved path as `FILE_PATH`.
+
+**Ideas flag parsing:**
+- Scan `$ARGUMENTS` for the `--ideas` flag.
+- If `--ideas` is present with no IDs after it (no arguments follow, or the next argument is another flag starting with `--`), display an error and stop:
+  ```
+  --ideas requires at least one idea ID. Usage: --ideas IDEA-1 IDEA-2
+  ```
+- If `--ideas` is present with IDs, collect all arguments after `--ideas` (space-separated, e.g., `--ideas IDEA-1 IDEA-3 IDEA-7`) as the `IDEA_IDS` list. Stop collecting when another `--` flag is encountered or arguments end.
+- The IDs may be in formats like `IDEA-1`, `IDEA-3`, `1`, `3` -- normalize by extracting the numeric portion.
+- If `--ideas` is not present, set `IDEA_IDS` to empty.
+</step>
+
+<step name="validate_ideas">
+Only runs if `IDEA_IDS` is non-empty. If `IDEA_IDS` is empty, skip this step entirely.
+
+**Validate all idea IDs upfront before conversion starts:**
+
+1. Run:
+   ```bash
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas list --raw
+   ```
+2. Parse the JSON output to get the list of all ideas (the `ideas` array).
+3. For each idea ID in `IDEA_IDS`:
+   - Extract the numeric portion (e.g., `IDEA-3` becomes `3`, `42` stays `42`).
+   - Check if any idea in the returned list has an `id` field matching that number.
+   - If no match is found, add the ID to an `INVALID_IDS` list.
+4. If `INVALID_IDS` is non-empty, display an error and stop:
+   ```
+   Invalid idea IDs: IDEA-99, IDEA-42. Run /dgs:list-ideas to see available ideas.
+   ```
+5. If all IDs are valid, store:
+   - `IDEA_FILENAMES`: comma-separated idea filenames for passing to `specs create --source-ideas` (e.g., `001-my-idea.md,003-other-idea.md`). Extract the `filename` field from each matching idea.
+   - `IDEA_DISPLAY`: list of `IDEA-{id} ({title})` strings for display during review (e.g., `IDEA-1 (Phase 0 Foundation Infrastructure)`).
+</step>
+
+<step name="read_source">
+Use the Read tool to read the file at `FILE_PATH`.
+
+**Error handling:**
+- If the file does not exist, display: `File not found: <FILE_PATH>` and stop execution.
+- If the file cannot be read (permissions, binary, etc.), display: `Cannot read file: <FILE_PATH>` and stop execution.
+
+**Processing:**
+- Store the full file content as `SOURCE_CONTENT`.
+- Extract the filename (without directory path) as `SOURCE_FILENAME`.
+- Handle both absolute paths and relative paths (resolve relative paths against the current working directory).
+
+**Frontmatter handling:**
+- If the source file has YAML frontmatter (delimited by `---`), extract any useful metadata from it (title, status, author, date, tags, etc.) and store separately.
+- Strip the raw frontmatter block from the content before passing to conversion -- the frontmatter metadata will be used to inform the conversion but should not appear as raw YAML in the output.
+- If no frontmatter is present, proceed with the full content as-is.
+</step>
+
+<step name="check_duplicate">
+Check if any existing spec already references the same source filename. This step runs after `read_source` (which sets `SOURCE_FILENAME`) and before `load_context`.
+
+1. Run:
+   ```bash
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs specs list --raw
+   ```
+2. Parse the JSON output to get the list of all specs.
+3. For each spec, check if it has a `source_document` field.
+4. If `source_document` is present, extract the filename portion (the last path segment, e.g., `specs/spec-foo/docs/my-doc.md` becomes `my-doc.md`).
+5. Compare against `SOURCE_FILENAME` (the filename being imported, already set by `read_source`).
+6. If a match is found, display a warning:
+   ```
+   Warning: Spec "{existing-spec-title}" (ID: {existing-id}) was already imported from a file named "{filename}". Proceeding will create a second spec from this source.
+   ```
+7. Use AskUserQuestion: `Continue with import? (yes/no)`
+   - If the user responds "no" or "cancel", display `Import cancelled.` and stop execution.
+   - If the user responds "yes" or any other input, proceed to the next step.
+8. If no duplicate is found, proceed silently.
+</step>
+
+<step name="load_context">
+Load project context to inform the conversion. Two context sources are checked, each with a different loading strategy. When neither source exists, the workflow proceeds silently with empty context -- the conversion step never mentions whether context was available.
+
+**1. Codebase context (selective loading):**
+
+```bash
+ls ${project_root}/codebase/*.md 2>/dev/null
+```
+
+- If the directory does not exist or contains no `.md` files, set `CODEBASE_CONTEXT` to empty and proceed silently.
+- If files are found, read the first ~20 lines of each file using the Read tool (with `limit: 20`) to extract its title and purpose.
+- Score each file's relevance against `SOURCE_CONTENT`:
+  - **Broadly relevant** (almost always load): files about Architecture, Stack, Structure -- these inform any conversion.
+  - **Conditionally relevant** (load only if source mentions the domain): files about Testing, Integrations, Conventions, Concerns, Security, Performance -- load only if keywords from the file's topic appear in `SOURCE_CONTENT`.
+  - Use keyword overlap between the codebase file's title/purpose and the source document's content to decide relevance.
+- Load the full content of relevant files (typically 2-4 files; use judgment to balance thoroughness against token budget).
+- Concatenate the loaded file contents and store as `CODEBASE_CONTEXT`.
+
+**2. Product docs (load all):**
+
+```bash
+ls ${project_root}/docs/product/*.md 2>/dev/null
+```
+
+- If the directory does not exist or contains no `.md` files, set `PRODUCT_CONTEXT` to empty and proceed silently.
+- If files are found, read every `.md` file using the Read tool.
+- Concatenate all file contents and store as `PRODUCT_CONTEXT`.
+
+**Silent fallback:** When both `CODEBASE_CONTEXT` and `PRODUCT_CONTEXT` are empty, proceed to `convert_to_prd` without any warning or output change. The conversion produces the same output as it would without this step.
+</step>
+
+<step name="convert_to_prd">
+This is the core AI conversion step. Restructure `SOURCE_CONTENT` into a 9-section PRD.
+
+**Context incorporation:** If `CODEBASE_CONTEXT` or `PRODUCT_CONTEXT` were loaded by the previous step, use them to inform the conversion throughout. Codebase context grounds Implementation Notes in real files and patterns; product context aligns goals and architecture with the broader product vision. When context is available, weave it naturally into the relevant sections -- do not mention whether context was or was not available in the output. When context is empty, produce the same quality output using inference alone.
+
+**Auto-generate a spec title** from the source content. The title should capture the core topic or feature described in the document. Store as `SPEC_TITLE`.
+
+**Produce a PRD with these exact 9 sections in this order:**
+
+1. `## Problem Statement` -- Extract or infer the core problem the source document addresses. If the source does not have an explicit problem statement, synthesise one from the goals, requirements, and context described.
+
+2. `## Goals` -- Concrete goals derived from the source. Each goal should be specific and actionable. If the source lists objectives, outcomes, or success criteria, translate them into goals.
+
+3. `## Non-Goals` -- Explicit boundaries on what this spec does NOT cover. Derive from what the source deliberately excludes or marks as out of scope. If the source is silent on non-goals, infer likely non-goals from the problem domain and flag each with "(inferred)".
+
+4. `## User Stories` -- User stories in "As a [user], I want [X] so that [Y]" format. Derive from the source's requirements, features, or described workflows. Each distinct capability should have its own user story.
+
+5. `## Requirements` -- All requirements organised by priority:
+   - `### P0 -- Must Have` -- Core requirements that are essential for the feature to function.
+   - `### P1 -- Nice-to-Have` -- Important but not blocking requirements.
+   - `### P2 -- Future Considerations` -- Items explicitly deferred or mentioned as future work.
+   - **Every individual requirement, acceptance criterion, and behavioural detail from the source MUST appear in the appropriate priority tier.** If the source has numbered requirements (e.g., REQ-01, FR-3), each must be carried across with its original identifier preserved.
+   - **Priority classification rules based on explicit language signals:**
+     - **P0:** "must", "critical", "required", "essential", "mandatory", "shall" -> P0
+     - **P1:** "should", "important", "ideally", "recommended", "expected" -> P1
+     - **P2:** "could", "nice to have", "nice-to-have", "future", "later", "eventually", "out of scope for now" -> P2
+   - **When no explicit priority language exists** in the source for a requirement, default it to P1.
+   - **Always normalise the source document's own priority scheme to P0/P1/P2.** For example: High/Medium/Low -> P0/P1/P2; Critical/Major/Minor -> P0/P1/P2; Tier 1/Tier 2/Tier 3 -> P0/P1/P2.
+   - **Honest classification:** Classify based on language signals regardless of the resulting distribution. It is acceptable if most requirements end up in the same tier (e.g., all P1). Do not artificially redistribute to achieve balance.
+
+6. `## Success Metrics` -- Measurable outcomes that indicate the feature is working correctly. Derive from the source's acceptance criteria, KPIs, or success definitions. If the source lacks explicit metrics, infer reasonable ones from the goals and requirements.
+
+7. `## Open Questions` -- Flag three types of issues in one section:
+   - **Missing details:** Information the source document does not address that would be needed for implementation. Flag with "(missing detail)".
+   - **Contradictions:** Places where the source document contradicts itself or gives conflicting guidance. Flag with "(contradiction)".
+   - **Ambiguous language:** Statements that could be interpreted multiple ways, where the intended meaning affects implementation. Flag with "(ambiguous)".
+   - Also include questions or unknowns explicitly raised in the source document itself.
+   - **Suggested answers:** For each question, include a brief suggested answer -- a reasonable default Claude would use if the question is not addressed. Format as: "Suggested: [proposed answer]".
+   - **Context-informed questions:** When `CODEBASE_CONTEXT` is available, generate questions that surface gaps between the source document and the existing codebase (e.g., "This spec mentions caching but the codebase has no caching layer -- should a new caching module be introduced?"). When `PRODUCT_CONTEXT` is available, flag misalignments with product direction.
+   - Quantity is at Claude's discretion -- calibrate based on spec complexity and ambiguity level. A simple, clear spec may have 2-3 questions; a complex, ambiguous one may have 8-10.
+
+8. `## Timeline Considerations` -- Dependencies, sequencing, phasing, and any timeline-related information from the source. If the source describes phases, milestones, or ordering constraints, capture them here. If the source has no timeline information, note key dependencies and suggest a logical ordering.
+
+9. `## Implementation Notes` -- Technical details, architecture notes, code examples, CLI signatures, file path references, API contracts, data models, and any other implementation-specific content from the source. Preserve technical details verbatim where possible. If the source contains code snippets, include them in fenced code blocks.
+   - **Codebase grounding (when `CODEBASE_CONTEXT` is available):** Reference specific files from the codebase (e.g., "Update `bin/dgs-tools.cjs`", "Add new module to `deliver-great-systems/bin/lib/`"). Reference established patterns (e.g., "Follow the existing subagent spawning pattern used in plan-phase", "Use the CommonJS module pattern from `lib/*.cjs`"). Blend file-level and pattern-level references naturally -- do not attribute which notes came from codebase context vs inference.
+   - **Product alignment (when `PRODUCT_CONTEXT` is available):** Use product-level knowledge to inform architectural suggestions. If product docs describe a target architecture, align Implementation Notes with it. If product docs define conventions or constraints, reflect those in the implementation guidance.
+   - **When neither context is available:** Still produce Implementation Notes using generic suggestions inferred from the source document's content and domain. The section should always be substantive.
+
+**Critical conversion rules:**
+- This is a RESTRUCTURING, not a summarisation. Every detail, requirement, acceptance criterion, example, edge case, and technical note from the original MUST appear in the converted output.
+- When content maps to multiple PRD sections, duplicate it in both sections rather than choosing one.
+- Non-text elements (images, diagrams) referenced by path in the source should be noted as "[Referenced: <filename>]" in the relevant sections.
+- Infer aggressively for sections where the source has no explicit content -- fill in Non-Goals, Success Metrics, Open Questions, and Timeline Considerations based on what the source implies and what is missing.
+- Always fully restructure even if the source is already close to PRD format -- this ensures consistent formatting and enrichment.
+- If source frontmatter metadata was extracted (title, status, etc.), use it to inform the conversion but do not include it as a separate section.
+
+Store the full converted PRD as `CONVERTED_PRD`.
+</step>
+
+<step name="present_and_review">
+Present the converted PRD for user review and handle the review loop.
+
+**Compute the slug and attachment path for display:**
+1. Generate the slug from `SPEC_TITLE`: lowercase, replace non-alphanumeric characters with hyphens, trim leading/trailing hyphens.
+2. Compute the slugified source filename: take `SOURCE_FILENAME`, lowercase it, replace non-alphanumeric characters (except dots) with hyphens, trim leading/trailing hyphens, preserve file extension. This matches `docs.slugifyFilename()` behaviour.
+3. Compute the attachment path: `specs/spec-{slug}/docs/{slugified-source-filename}` (relative to the planning root).
+
+**Display the conversion result:**
+
+```
+## Converted: {SPEC_TITLE}
+
+{CONVERTED_PRD}
+
+---
+Original: will be saved to ${project_root}/{attachment_path}
+{If IDEA_IDS is non-empty: "Linking to: " followed by comma-separated IDEA_DISPLAY entries, e.g., "Linking to: IDEA-1 (Phase 0 Foundation Infrastructure), IDEA-3 (Other Idea)"}
+
+**Review options:**
+- **save** -- Save as draft spec
+- **edit** -- Describe changes to apply
+- **regenerate** -- Start fresh conversion (with optional guidance)
+- **cancel** -- Abort the import
+```
+
+**Use AskUserQuestion** to get the user's choice.
+
+**Handle each action:**
+
+**If "save":**
+1. Generate the slug from `SPEC_TITLE` (lowercase, replace non-alphanumeric with hyphens, trim leading/trailing hyphens).
+2. Check if `${project_root}/specs/spec-{slug}.md` already exists. If it does, use AskUserQuestion to prompt: `A spec with slug '{slug}' already exists. Enter a new title:` -- then regenerate the slug from the new title and re-check. Loop until no conflict.
+3. Compute the slugified source filename: take `SOURCE_FILENAME`, lowercase it, replace non-alphanumeric chars (except dots) with hyphens, trim leading/trailing hyphens, preserve file extension. This matches `docs.slugifyFilename()` behaviour.
+4. Compute the attachment path: `specs/spec-{slug}/docs/{slugified-source-filename}` (relative to the planning root).
+5. Call specs create (with --source-ideas when ideas are linked):
+   ```bash
+   # When IDEA_FILENAMES is non-empty:
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs specs create --title "{SPEC_TITLE}" --body "{CONVERTED_PRD}" --source-document "{attachment_path}" --source-ideas "{IDEA_FILENAMES}"
+
+   # When IDEA_FILENAMES is empty (no --ideas flag was provided):
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs specs create --title "{SPEC_TITLE}" --body "{CONVERTED_PRD}" --source-document "{attachment_path}"
+   ```
+   The `--source-ideas` value is the comma-separated list of idea filenames from `IDEA_FILENAMES` (e.g., `001-my-idea.md,003-other-idea.md`).
+   Parse the JSON result to get `id`, `filename`, `path`.
+6. Add the initial Refinement Log entry:
+   ```bash
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs specs add-log-entry --id "$id" --date "$(date +%Y-%m-%d)" --version "1.0" --action Created --summary "Imported from $SOURCE_FILENAME via /dgs:import-spec"
+   ```
+7. Call docs add to copy the original source document:
+   ```bash
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs docs add --source "{FILE_PATH}" --scope spec --scope-id "{id}"
+   ```
+   This copies the file (not moves), slugifies the filename, and updates INDEX.md automatically.
+8. Construct the commit message and auto-commit all import artifacts:
+
+   **Commit message format:** `specs: import {id} - {title} (from {original-filename})`
+   - `{id}` is the spec ID from the specs create result (e.g., `SPEC-005`).
+   - `{title}` is `SPEC_TITLE`, truncated to 50 characters with ellipsis if longer. Truncation: if title length > 50, take first 47 characters + `...`.
+   - `{original-filename}` is `SOURCE_FILENAME` (the original filename, not slugified).
+
+   **Execute the commit:**
+   ```bash
+   node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs commit "specs: import {id} - {title} (from {original-filename})" --files ${project_root}/specs/spec-{slug}.md ${project_root}/{attachment_path} ${project_root}/specs/spec-{slug}/docs/INDEX.md ${project_root}/specs/spec-{slug}/docs/.names.json
+   ```
+   The `dgs-tools.cjs commit` helper with `--files` stages only the specified files and commits them. This ensures other unstaged/uncommitted changes in the working tree are not included. The `.names.json` file is included because `docs add` creates it as internal metadata for INDEX rebuilds.
+
+   **Handle commit failure:**
+   If the commit command fails (non-zero exit code):
+   - Do NOT undo the save (files remain on disk).
+   - Display: `Git commit failed: {error message}. Files have been saved but not committed. You can commit manually.`
+   - Do NOT stop execution -- continue to display post-save feedback.
+   - Set `COMMIT_FAILED = true` for use in the post-save feedback display.
+
+9. Display post-save feedback:
+
+   **On successful commit:**
+   ```
+   Spec created:
+     ID: {id}
+     Title: {SPEC_TITLE}
+     Status: draft
+     File: ${project_root}/{filename path from specs create result}
+     Source: ${project_root}/{attachment_path}
+     Committed: specs: import {id} - {title} (from {original-filename})
+     {If IDEA_IDS is non-empty: "Linked ideas: IDEA-1, IDEA-3 (unchanged state)"}
+
+   Next steps:
+     - Review and refine the draft with /dgs:review-spec
+   ```
+
+   **On commit failure (COMMIT_FAILED is true):**
+   ```
+   Spec created:
+     ID: {id}
+     Title: {SPEC_TITLE}
+     Status: draft
+     File: ${project_root}/{filename path from specs create result}
+     Source: ${project_root}/{attachment_path}
+     {If IDEA_IDS is non-empty: "Linked ideas: IDEA-1, IDEA-3 (unchanged state)"}
+
+   Next steps:
+     - Review and refine the draft with /dgs:review-spec
+     - Commit the new spec and source document to version control
+   ```
+10. Stop execution.
+
+**If "cancel":**
+Display: `Import cancelled.`
+Stop execution.
+
+**If "edit":**
+1. Use AskUserQuestion: `What changes would you like to make?`
+2. Read the user's free-text description of desired changes.
+3. Re-run the AI conversion (the convert_to_prd step logic), incorporating the user's edit instructions as additional guidance overlaid on the original `SOURCE_CONTENT`. The edit request should modify the existing `CONVERTED_PRD` -- it is a targeted revision, not a fresh conversion. The feedback may request changes to:
+   - Section content (e.g., "move X from P1 to P0")
+   - Missing details (e.g., "add a user story about admin access")
+   - Tone or framing (e.g., "make the problem statement more specific")
+   - Title changes (e.g., "rename to X")
+   - Any other adjustments
+4. Update `CONVERTED_PRD` and `SPEC_TITLE` (if the edit affects the title).
+5. Recompute the slug and attachment path from the updated `SPEC_TITLE`.
+6. Re-display the full spec with the same review options. Continue the loop.
+
+**If "regenerate":**
+1. Use AskUserQuestion: `Any guidance for the regeneration? (press enter to skip)`
+2. If the user provides guidance, store it. If empty/enter, proceed without guidance.
+3. Re-run the full conversion from scratch (the convert_to_prd step logic) using `SOURCE_CONTENT` plus any guidance. This is a fresh conversion, not a patch.
+4. Update `CONVERTED_PRD` and `SPEC_TITLE`.
+5. Recompute the slug and attachment path from the updated `SPEC_TITLE`.
+6. Re-display the full spec with the same review options. Continue the loop.
+
+**Any other input:** Treat as an edit request (per convention -- non-standard input is treated as revision feedback). Proceed as if they chose "edit" with the input as feedback.
+
+**No limit on iterations** -- the user can edit/regenerate as many times as they want until they save or cancel.
+</step>
+
+</process>
+
+<success_criteria>
+- [ ] File path argument is parsed from $ARGUMENTS
+- [ ] Missing file path produces clear usage error
+- [ ] Non-existent file produces "File not found" error
+- [ ] Unreadable file produces "Cannot read file" error
+- [ ] Source file content is read and stored
+- [ ] YAML frontmatter is stripped from source before conversion (if present)
+- [ ] --ideas flag parsed from arguments with error on empty IDs
+- [ ] All idea IDs validated upfront via ideas list with clear error for invalid IDs
+- [ ] Duplicate import detection warns when another spec references the same source filename
+- [ ] Codebase context selectively loaded based on relevance scoring (when ${project_root}/codebase/ exists)
+- [ ] Product docs unconditionally loaded (when ${project_root}/docs/product/ exists)
+- [ ] Missing context directories produce no warnings or output changes
+- [ ] Conversion produces all 9 PRD sections in correct order
+- [ ] Every detail from source appears in converted output (restructuring, not summarisation)
+- [ ] Non-goals are inferred when source is silent (flagged as inferred)
+- [ ] Open questions flag missing details, contradictions, and ambiguous language with suggested answers
+- [ ] Requirements use explicit language signal mapping (must/critical -> P0, should/important -> P1, could/nice-to-have -> P2) with P1 default
+- [ ] Requirements preserve original identifiers if numbered in source
+- [ ] Implementation Notes reference real files and patterns when codebase context available
+- [ ] Spec title is auto-generated from source content
+- [ ] Full converted PRD is displayed for review with attachment path shown
+- [ ] Linked ideas displayed in review when --ideas provided
+- [ ] Four review actions available: save, edit, regenerate, cancel
+- [ ] Save creates draft spec via specs create with source_document field populated
+- [ ] Save passes --source-ideas to specs create when ideas are linked
+- [ ] Save copies original source file to spec docs via docs add (handles slugified filename and INDEX.md)
+- [ ] Save produces atomic git commit of spec file, attachment, INDEX.md, and .names.json via dgs-tools.cjs commit --files
+- [ ] Commit message follows format: specs: import {id} - {title} (from {original-filename}) with 50-char title truncation
+- [ ] Commit failure is non-fatal: files preserved, error reported, execution continues
+- [ ] Post-save feedback shows commit confirmation and linked ideas (unchanged state)
+- [ ] Edit collects natural language feedback via AskUserQuestion and re-runs targeted conversion
+- [ ] Regenerate asks for optional guidance and re-runs full conversion from scratch
+- [ ] Slug conflict prompts user for new title via AskUserQuestion
+- [ ] Cancel option stops with clear message
+- [ ] Non-standard review input treated as edit feedback
+- [ ] Unlimited edit/regenerate revision rounds supported
+- [ ] source_document field written to spec YAML frontmatter on save
+</success_criteria>