@ktpartners/dgs-platform 2.6.2

package/deliver-great-systems/workflows/develop-idea.md

@@ -0,0 +1,544 @@
+<purpose>
+Orchestrator workflow that runs structured discussion then research as a single continuous creative session. Discussion flows naturally into research -- the two phases feel like one conversation, not two separate tasks bolted together.
+
+Pure orchestrator: no unique artifacts. The idea file (with Discussion Log and Research Log) and research document are the outputs. Discussion commits before research starts so research reads the freshly-updated idea context.
+
+When re-running on an idea with prior history, the user is offered granular choice: re-do both, just discuss, or just research.
+</purpose>
+
+<context_tier>planning</context_tier>
+
+<required_reading>
+Read all files referenced by the invoking prompt's execution_context before starting.
+</required_reading>
+
+<process>
+
+<step name="initialize" priority="first">
+Load planning context:
+
+```bash
+INIT=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs init milestone-op)
+```
+
+Extract: `project_root` as needed by the workflow.
+
+Load planning-tier context files:
+
+```bash
+TIER_FILES=$(node "$HOME/.claude/deliver-great-systems/bin/dgs-tools.cjs" context load-tier planning --raw 2>/dev/null)
+```
+</step>
+
+<step name="parse_arguments">
+Parse `$ARGUMENTS` for an idea ID.
+
+- First positional argument: treat as idea ID (e.g., `/dgs:develop-idea 5` -> id=5)
+- Named flag: `--id N` (e.g., `/dgs:develop-idea --id 5` -> id=5)
+
+If no ID provided, list pending ideas and prompt for selection:
+
+```bash
+PENDING=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas list --state pending --raw)
+```
+
+Parse the JSON `ideas` array.
+
+**If empty:**
+```
+No pending ideas to develop. Run /dgs:add-idea to capture one.
+```
+Exit workflow.
+
+**If ideas exist:** Display them as a numbered list showing ID, title, and tags. Use AskUserQuestion:
+- header: "Select Idea"
+- question: "Which idea would you like to develop?"
+- Provide the numbered idea options
+
+Extract the selected idea's ID.
+</step>
+
+<step name="load_idea">
+Load the full idea content by ID:
+
+```bash
+LIST=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas list --raw)
+```
+
+Find the idea in the JSON by matching `id`. If not found:
+```
+Idea #${id} not found.
+```
+Exit workflow.
+
+Read the idea file directly using the `path` from the list result to get the full content including Discussion Log, Research Log, Notes, and body.
+
+Set variables from the parsed content:
+- `id` -- idea ID
+- `title` -- idea title
+- `body` -- idea body/problem statement
+- `tags` -- tags array
+- `notes` -- Notes section content (if any)
+- `state` -- current state (pending, done, rejected)
+- `filename` -- file name
+- `path` -- file path
+- `discussionLog` -- Discussion Log section content (if any)
+- `researchLog` -- Research Log section content (if any)
+
+Also check for prior research document:
+```bash
+# Derive slug from filename (strip id prefix and .md suffix)
+SLUG=$(echo "${filename}" | sed 's/^[0-9]*-//' | sed 's/\.md$//')
+RESEARCH_DOC="${project_root}/docs/ideas/pending/${SLUG}-research.md"
+```
+If the research document file exists, read it for prior research context.
+
+Also check for existing spec:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs specs list --raw
+```
+Search for matching idea_id or title in the specs list.
+
+Also check for REPOS.md:
+```bash
+cat ${project_root}/REPOS.md 2>/dev/null
+```
+If REPOS.md exists and has entries, this is a multi-repo project -- research should be partitioned by repo where relevant.
+</step>
+
+<step name="assess_history">
+Determine the idea's development state by checking what exists:
+- `hasDiscussion` -- whether Discussion Log has content
+- `hasResearch` -- whether Research Log has content
+- `hasSpec` -- whether a matching spec was found
+- `discussionCount` -- number of `### Session` entries in Discussion Log
+- `researchCount` -- number of `### Session` entries in Research Log
+
+Set routing variables for later steps:
+- `runDiscussion` -- whether to run the discussion phase (default: true)
+- `runResearch` -- whether to run the research phase (default: true)
+
+**For fresh ideas** (no prior discussion or research):
+Skip status display. Proceed directly to discussion_phase. Both `runDiscussion` and `runResearch` remain true.
+
+**For ideas with prior history:** Display brief status line:
+```
+Idea ${id}: ${title}
+Status: Discussed (${N} sessions), ${researched ? 'Researched' : 'Not researched'}
+```
+
+If a spec exists, mention it: "Note: A spec already exists for this idea. Proceeding with development anyway."
+
+**For ideas previously discussed AND researched:** Use AskUserQuestion to offer choice:
+- "Re-do both discussion and research"
+- "Just discuss (skip research)"
+- "Just research (skip discussion)"
+
+Route based on user choice:
+- "Re-do both": `runDiscussion = true`, `runResearch = true`
+- "Just discuss": `runDiscussion = true`, `runResearch = false`
+- "Just research": `runDiscussion = false`, `runResearch = true`
+
+**For ideas discussed but NOT researched:** Use AskUserQuestion to offer choice:
+- "Discuss more, then research"
+- "Go straight to research"
+
+Route based on user choice:
+- "Discuss more, then research": `runDiscussion = true`, `runResearch = true`
+- "Go straight to research": `runDiscussion = false`, `runResearch = true`
+
+If `runDiscussion` is false, skip to step research_phase.
+If `runResearch` is false, after discussion_commit skip to step completion_summary.
+</step>
+
+<step name="discussion_phase">
+**Only runs if `runDiscussion` is true.**
+
+Print a visual banner to mark the transition into discussion:
+```
+--- Discussion Phase ---
+```
+(Exact format at Claude's discretion -- could be a markdown heading, separator line, or styled banner.)
+
+Execute the full discuss-idea workflow inline. This is a single continuous conversation -- do NOT delegate to the discuss-idea command shim.
+
+**Sub-step: discussion_context**
+Present the idea to yourself (Claude) as working context.
+
+*For a new discussion* (no prior Discussion Log):
+Internalize the idea's title, body, tags, and any notes. This is the raw material for discussion.
+
+*For a re-discussion* (prior Discussion Log exists):
+Review the prior discussion entries. Identify:
+- Open Questions from the most recent entry -- these are the starting point
+- Prior Decisions -- context but not necessarily locked; use discretion on whether to revisit
+- Key Insights from prior sessions -- build on these, don't repeat them
+
+Each re-discussion produces its own fresh Decision tag regardless of prior status.
+
+**Sub-step: understanding_phase**
+*Tone: Thinking partner, collaborative, opinionated -- like a cofounder brainstorming.*
+
+*For a new discussion* (no prior log):
+Restate the idea in your own words to confirm understanding. Then ask clarifying questions about the problem:
+- Who has this problem? How often do they hit it?
+- What's the current workaround?
+- What does success look like?
+
+Challenge vague problem statements and push for specifics. Adapt depth to idea complexity -- simple ideas get quick understanding (1-2 exchanges), complex ideas get deeper probing (3-4 exchanges).
+
+*For a re-discussion* (prior log exists):
+Jump straight to unresolved Open Questions from the last session. Skip restating and re-clarifying unless the prior understanding seems incomplete or wrong.
+
+Use AskUserQuestion for each turn of dialogue. Continue until the problem is well-defined.
+
+**Mid-exit handling:** If the user indicates they want to exit (says "done", "save", "exit", "stop", or changes topic), go to handle_exit sub-step.
+
+**Sub-step: exploration_phase**
+Proactively raise considerations the user may not have thought of:
+- Alternative approaches to the problem
+- Potential conflicts with existing functionality (use Grep/Glob to check the codebase if the idea relates to existing features)
+- Scope concerns -- is this bigger than it seems?
+- Edge cases and failure modes
+- Dependencies or prerequisites
+
+When one option is clearly better, recommend it with reasoning. Otherwise present options neutrally and let the user decide.
+
+Continue the back-and-forth dialogue with AskUserQuestion. The developer steers the conversation -- follow their lead on what to explore deeper.
+
+**Mid-exit handling:** If the user indicates they want to exit, go to handle_exit sub-step.
+
+**Sub-step: refinement_phase**
+Propose a refined version of the idea:
+
+1. **Refined problem statement** -- sharper, more specific than the original
+2. **Updated approach** -- incorporating insights from the discussion
+3. **Open questions** -- questions that emerged during discussion and remain unresolved
+
+Present this as a proposal for user review. Propose a Decision tag -- one of:
+- **"Ready for research"** -- idea is well-defined, needs feasibility/landscape investigation
+- **"Ready for spec"** -- idea is clear enough to write a formal specification
+- **"Needs more thought"** -- idea has merit but unresolved questions remain
+
+Use AskUserQuestion to present the proposal and get approval.
+
+**If user rejects the proposal:** Revise based on their feedback and present again (one revision attempt). If rejected a second time, save with "Needs more thought" as the Decision tag.
+
+**Mid-exit handling:** If the user indicates they want to exit, go to handle_exit sub-step.
+
+**Sub-step: handle_exit**
+When the user indicates they want to exit mid-discussion:
+
+1. **Always save on exit** -- inform the user: "Saving your progress." Do NOT ask permission.
+2. Use **"Needs more thought"** as the Decision tag for partial discussions.
+3. Do **NOT** update the idea body on partial save -- only write the discussion log entry.
+4. Format the partial save entry based on how far the discussion progressed:
+   - Exited during Understanding: capture what was clarified so far
+   - Exited during Exploration: capture clarifications plus alternatives/considerations raised
+   - Exited during Refinement: capture the near-complete refinement with a note that it was not finalized
+
+Proceed to step save_discussion with `partial = true`.
+
+**Sub-step: save_discussion**
+Generate the discussion entry fields:
+
+- `date`: current date in YYYY-MM-DD format
+- `keyInsights`: bullet-pointed list of key insights from the discussion (use `\n- ` separator)
+- `refinedProblem`: the refined problem statement (full discussion) or summary of what was discussed (partial)
+- `refinedApproach`: the refined approach (full discussion) or "Not yet refined" (partial)
+- `openQuestions`: bullet-pointed list of open questions (use `\n- ` separator)
+- `decision`: the agreed Decision tag (e.g., "Ready for research", "Ready for spec", "Needs more thought")
+
+For re-discussions: reference what changed from prior discussion in the keyInsights.
+
+Format as a JSON string and call:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas discuss-save --id "${id}" --entry '${entryJson}'
+```
+
+The `entryJson` should be a JSON object:
+```json
+{
+  "date": "YYYY-MM-DD",
+  "keyInsights": "- insight one\n- insight two",
+  "refinedProblem": "The refined problem statement",
+  "refinedApproach": "The refined approach",
+  "openQuestions": "- question one\n- question two",
+  "decision": "Ready for research"
+}
+```
+
+Parse the JSON result to confirm success.
+
+**For FULL discussions only** (not partial saves): If the idea body or approach changed significantly during refinement, also update the idea body:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas update --id "${id}" --field body --value "${refinedBody}"
+```
+
+Per CONTEXT.md: idea body is NOT updated on partial save.
+</step>
+
+<step name="discussion_commit">
+Commit the updated idea file immediately after discussion:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs commit "docs: discuss idea #${id} — ${title}" --files ${project_root}/ideas/${state}/${filename}
+```
+
+**If `partial = true`** (user exited mid-discussion):
+Print: "Saved progress. Run /dgs:develop-idea ${id} to continue where you left off."
+Exit workflow. Do NOT proceed to research.
+
+If `runResearch` is false (user chose "Just discuss" in assess_history), skip to step completion_summary.
+</step>
+
+<step name="decision_routing">
+Read the Decision tag from the discussion that just completed.
+
+**If Decision is "Not viable" or "Reject":**
+Print: "Discussion concluded the idea is not viable. Skipping research."
+Go to step completion_summary (no research).
+
+**If Decision is "Needs more thought":**
+Claude judges whether the open questions are things research can help answer (e.g., feasibility questions, landscape questions, technical unknowns) or things that need more discussion first (e.g., problem definition unclear, user intent ambiguous, scope undefined).
+- If research can help: continue to research_phase.
+- If not: print "Open questions need more discussion before research can help. Run /dgs:discuss-idea ${id} to continue." Go to step completion_summary.
+
+**If Decision is "Ready for research" or "Ready for spec":**
+Continue to research_phase.
+</step>
+
+<step name="research_phase">
+**Only runs if `runResearch` is true and decision_routing allows it.**
+
+Print a visual banner to mark the transition into research:
+```
+--- Research Phase ---
+```
+(Exact format at Claude's discretion.)
+
+Execute the full research-idea workflow inline. This is part of the same continuous session -- do NOT delegate to the research-idea command shim.
+
+**Sub-step: research_context**
+Assess the idea and decide research strategy:
+
+1. **Announce focus**: Print one line explaining which dimensions matter most for this idea and why.
+2. **Read Discussion Log** -- use discussion insights (refined problem, approach, open questions) to guide research focus areas.
+3. **Read prior research** if present -- note what was found before. On re-research, the document will be overwritten but note what changed.
+4. **Determine repo scope**: If REPOS.md exists, identify which repos are relevant. If no REPOS.md, treat as single-repo research.
+
+**Sub-step: web_search**
+*Dimension 1: Web Search*
+
+Announce: "Searching web for prior art and technical landscape..."
+
+Use WebSearch and WebFetch to investigate:
+- How others have solved similar problems
+- Existing libraries, tools, or frameworks relevant to the idea
+- Common patterns and anti-patterns
+- Recent discussions or blog posts
+
+Adapt depth to what the idea needs -- a well-understood problem gets a quick scan, a novel idea gets broader exploration. Web search is best-effort -- search what's findable, note gaps, move on.
+
+**Sub-step: codebase_analysis**
+*Dimension 2: Codebase Analysis*
+
+Announce: "Analyzing codebase for related implementations..."
+
+Use Grep and Glob to examine:
+- Existing code that relates to the idea's domain
+- Patterns already established that would apply
+- Potential conflicts or integration points
+- Code that would need to change
+
+For multi-repo projects: analyze each relevant repo separately.
+
+**Sub-step: landscape_survey**
+*Dimension 3: Landscape Survey*
+
+Announce: "Surveying technical landscape..."
+
+Build comparison information for relevant tools/libraries/approaches:
+- For obvious choices (one clear winner): brief verdict with reasoning
+- For genuine tradeoffs: table with pros, cons, license, maintenance status, compatibility
+- Per repo stack for multi-repo projects
+
+Recommend when clear -- strong recommendation when one option is clearly better, present options neutrally when it's a toss-up.
+
+**Sub-step: approaches**
+*Dimension 4: Approaches & Patterns*
+
+Announce: "Identifying approaches and patterns..."
+
+Identify how similar problems have been solved:
+- Architectural patterns that apply
+- Implementation strategies
+- Sequencing (what to build first)
+- What to avoid and why
+
+**Sub-step: feasibility**
+*Dimension 5: Feasibility Assessment*
+
+Announce: "Assessing feasibility..."
+
+Assess:
+- **Stack fit**: How well does this fit the existing technology stack?
+- **Effort estimate**: T-shirt size (S/M/L/XL) + approximate phase count
+- **Key risks**: What could go wrong? What are the unknowns?
+- **New dependencies**: Any new libraries, services, or external integrations needed?
+- **Recommendation**: Overall go/no-go/conditional assessment
+
+**Sub-step: write_document**
+Create (or overwrite) the research document.
+
+Ensure directory exists:
+```bash
+mkdir -p ${project_root}/docs/ideas/pending
+```
+
+Write the research document to `${project_root}/docs/ideas/pending/${SLUG}-research.md` using the Write tool.
+
+Document structure:
+```markdown
+---
+type: research
+idea_id: {id}
+idea_title: "{title}"
+date: "{YYYY-MM-DD}"
+repos_analysed: ["{repo1}", "{repo2}"]
+---
+
+# Research: {title}
+
+{If re-research and meaningful changes exist:}
+## Changes from Prior Research
+
+{Bullet list of what's new, updated, or contradicted vs prior findings.}
+
+## Web Search Findings
+
+{Findings from Dimension 1}
+
+## Codebase Analysis
+
+{Findings from Dimension 2. For multi-repo: subsections per repo.}
+
+## Landscape Survey
+
+{Findings from Dimension 3. Comparison tables where applicable.}
+
+## Approaches & Patterns
+
+{Findings from Dimension 4}
+
+## Feasibility Assessment
+
+**Stack Fit:** {assessment}
+**Effort:** {T-shirt size} -- {phase count estimate}
+**Key Risks:**
+{bullet list}
+**New Dependencies:**
+{bullet list or "None"}
+**Recommendation:** {go / no-go / conditional with conditions}
+
+---
+*Researched: {YYYY-MM-DD}*
+```
+
+If repos_analysed is empty (no REPOS.md), omit the field from frontmatter or use an empty array.
+
+**Sub-step: save_research_log**
+Save the Research Log entry to the idea file via the CLI command.
+
+Build the entry JSON:
+```json
+{
+  "date": "YYYY-MM-DD",
+  "summary": "{Actionable summary paragraph}",
+  "keyFindings": "- {finding 1}\n- {finding 2}\n- {finding 3}",
+  "recommendation": "{Strong recommendation or neutral options presentation}",
+  "documentLink": "${project_root}/docs/ideas/pending/${SLUG}-research.md",
+  "outcome": "{Recommended next step: 'Ready for spec', 'Needs more discussion', 'Not feasible', etc.}"
+}
+```
+
+Call:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs ideas research-save --id "${id}" --entry '${entryJson}'
+```
+
+Parse the JSON result to confirm success.
+</step>
+
+<step name="research_commit">
+Commit both the research document and the updated idea file:
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs commit "docs: research idea #${id} -- ${title}" --files ${project_root}/docs/ideas/pending/${SLUG}-research.md ${project_root}/ideas/${state}/${filename}
+```
+</step>
+
+<step name="completion_summary">
+Display a unified summary combining both phases.
+
+**If both discussion and research ran:**
+```
+Development Complete: ${title}
+
+Discussion: ${decision}
+Research: ${recommendation}
+  Key finding: ${keyFinding}
+
+Next step: ${contextAwareNextStep}
+
+Files:
+- ${project_root}/ideas/${state}/${filename}
+- ${project_root}/docs/ideas/pending/${SLUG}-research.md
+```
+
+**If only discussion ran** (research skipped by decision routing, user choice, or partial exit):
+```
+Discussion Complete: ${title}
+
+Decision: ${decision}
+
+Next step: ${contextAwareNextStep}
+
+Files:
+- ${project_root}/ideas/${state}/${filename}
+```
+
+**If only research ran** (user chose "Just research" in assess_history):
+```
+Research Complete: ${title}
+
+Research: ${recommendation}
+  Key finding: ${keyFinding}
+
+Next step: ${contextAwareNextStep}
+
+Files:
+- ${project_root}/ideas/${state}/${filename}
+- ${project_root}/docs/ideas/pending/${SLUG}-research.md
+```
+
+**Context-aware next step logic:**
+- Both discussion and research say "ready" -> suggest `/dgs:write-spec`
+- Research says "not feasible" but discussion was positive -> suggest revisiting the idea
+- Research says "needs more discussion" -> suggest `/dgs:discuss-idea ${id}`
+- Idea was rejected in discussion -> suggest `/dgs:reject-idea ${id}` or rethinking
+- Default (both phases completed successfully) -> suggest `/dgs:write-spec`
+</step>
+
+</process>
+
+<success_criteria>
+- [ ] Idea loaded with full context including prior history
+- [ ] Prior history status displayed (if applicable)
+- [ ] Re-run choice offered (if applicable)
+- [ ] Discussion phase completed (or partial save on exit)
+- [ ] Discussion committed before research starts
+- [ ] Decision routing applied (skip research on negative decisions)
+- [ ] Research phase completed (if applicable)
+- [ ] Research document and idea file committed together
+- [ ] Unified completion summary with context-aware next step
+- [ ] File paths listed in footer
+</success_criteria>