@ktpartners/dgs-platform 2.6.2

package/deliver-great-systems/workflows/map-codebase.md

@@ -0,0 +1,621 @@
+<purpose>
+Orchestrate parallel codebase mapper agents to analyze registered repos and produce per-repo structured documents in ${codebase_dir}/<repo-name>/, then synthesize unified top-level files at ${codebase_dir}/.
+
+Supports two modes:
+- **Refresh mode** (default): Clears all per-repo subdirectories and unified files, then remaps everything from scratch.
+- **Update mode** (`--only <repo-name>`): Clears only the targeted repo's subdirectory, re-maps it, then regenerates all unified files.
+
+Each agent has fresh context, explores a specific focus area for a specific repo, and **writes documents directly**. The orchestrator only receives confirmation + line counts, then writes a summary.
+
+Output: ${codebase_dir}/<repo-name>/ folders, each with 7 structured documents about that repo's codebase state, plus 7 unified top-level files synthesized from per-repo content, plus CROSS-REPO.md with cross-repo comparison tables (when 2+ repos).
+</purpose>
+
+<context_tier>planning</context_tier>
+
+<philosophy>
+**Why dedicated mapper agents:**
+- Fresh context per domain (no token contamination)
+- Agents write documents directly (no context transfer back to orchestrator)
+- Orchestrator only summarizes what was created (minimal context usage)
+- Faster execution (agents run simultaneously across all repos)
+
+**Document quality over length:**
+Include enough detail to be useful as reference. Prioritize practical examples (especially code patterns) over arbitrary brevity.
+
+**Always include file paths:**
+Documents are reference material for Claude when planning/executing. Always include actual file paths formatted with backticks: `src/services/user.ts`.
+</philosophy>
+
+<process>
+
+<step name="init_context" priority="first">
+Load codebase mapping context:
+
+```bash
+INIT=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs init map-codebase)
+```
+
+If `--only <repo-name>` was passed to the slash command, forward it:
+
+```bash
+INIT=$(node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs init map-codebase --only <repo-name>)
+```
+
+Extract from init JSON: `mapper_model`, `commit_docs`, `codebase_dir`, `existing_maps`, `has_maps`, `codebase_dir_exists`, `multi_repo`, `repos`, `valid_repos`, `repo_errors`, `existing_repo_maps`, `mode`, `only_repo`.
+
+Load planning-tier context files:
+
+```bash
+TIER_FILES=$(node "$HOME/.claude/deliver-great-systems/bin/dgs-tools.cjs" context load-tier planning --raw 2>/dev/null)
+```
+
+**Mode determination:**
+- `mode === "refresh"`: Default. Clear everything, remap all repos.
+- `mode === "update"`: `--only` was used. Clear and remap only the targeted repo, then regenerate unified files.
+</step>
+
+<step name="validate_repos" priority="high">
+Check multi-repo readiness from init JSON.
+
+**If `multi_repo` is false** (no valid repos registered):
+
+```
+No repos registered — skipping codebase mapping. Run /dgs:add-repo to register your first repo.
+```
+
+This is a clean exit, not an error. In job mode (non-interactive), report `SUCCESS` — this step succeeds with no changes. End workflow.
+
+**If `repo_errors` is non-empty:**
+
+Show warnings for invalid repos but continue with valid ones:
+```
+Warning: Some repos could not be validated:
+[List each error from repo_errors]
+
+Proceeding with valid repos: [List valid_repos]
+```
+
+**If ALL repos are invalid** (valid_repos is empty but REPOS.md had entries):
+
+```
+No repos registered. Run /dgs:add-repo to register your first repo.
+```
+
+End workflow.
+
+Continue to check_existing.
+</step>
+
+<step name="check_existing">
+**Mode-aware cleanup.** Codebase docs are regeneratable data, so deletion requires no confirmation.
+
+**Refresh mode (mode === "refresh"):**
+- Delete any existing flat codebase docs (top-level .md files in ${codebase_dir}/) silently
+- Delete any existing per-repo subdirectories in ${codebase_dir}/ silently
+
+**Update mode (mode === "update", only_repo is set):**
+- Delete ONLY the targeted repo's subdirectory: `rm -rf ${codebase_dir}/${only_repo}/`
+- Do NOT delete other repos' subdirectories
+- Do NOT delete unified top-level files (they will be regenerated after mapping)
+
+Continue to create_structure.
+</step>
+
+<step name="create_structure">
+Create per-repo subdirectories in ${codebase_dir}/:
+
+**Refresh mode:**
+
+```bash
+# Delete any existing flat codebase docs (migration from old layout)
+rm -f ${codebase_dir}/*.md 2>/dev/null
+
+# Delete any existing per-repo subdirectories
+rm -rf ${codebase_dir}/*/ 2>/dev/null
+
+# Create per-repo subdirectories for all valid repos
+for repo in ${valid_repos}; do
+  mkdir -p ${codebase_dir}/${repo}
+done
+```
+
+**Update mode:**
+
+```bash
+# Delete only the targeted repo's subdirectory
+rm -rf ${codebase_dir}/${only_repo}/ 2>/dev/null
+
+# Create directory for the targeted repo
+mkdir -p ${codebase_dir}/${only_repo}
+```
+
+Each repo subdirectory will contain all 7 documents:
+- STACK.md (from tech mapper)
+- INTEGRATIONS.md (from tech mapper)
+- ARCHITECTURE.md (from arch mapper)
+- STRUCTURE.md (from arch mapper)
+- CONVENTIONS.md (from quality mapper)
+- TESTING.md (from quality mapper)
+- CONCERNS.md (from concerns mapper)
+
+Continue to spawn_agents.
+</step>
+
+<step name="spawn_agents">
+Spawn 4 parallel dgs-codebase-mapper agents **per repo**. All agents across all repos run in parallel (all spawned with `run_in_background=true`).
+
+Use Task tool with `subagent_type="dgs-codebase-mapper"`, `model="{mapper_model}"`, and `run_in_background=true`.
+
+**CRITICAL:** Use the dedicated `dgs-codebase-mapper` agent, NOT `Explore`. The mapper agent writes documents directly.
+
+**Refresh mode:** Spawn agents for all repos in `repos` array.
+
+**Update mode:** Spawn agents only for the targeted repo. Show: `Updating ${only_repo}...`
+
+**For each repo in `repos` array:**
+
+Show: `Mapping ${repo.name}...`
+
+Spawn 4 agents for this repo. Each agent's prompt includes `repo_path` and `repo_name` so it knows WHERE to explore and WHERE to write:
+
+**Agent: Tech Focus (per repo)**
+
+```
+Task(
+  subagent_type="dgs-codebase-mapper",
+  model="{mapper_model}",
+  run_in_background=true,
+  description="Map ${repo.name} tech stack",
+  prompt="Focus: tech
+repo_path: ${repo.path}
+repo_name: ${repo.name}
+
+Analyze the codebase at ${repo.path} for technology stack and external integrations.
+
+Write these documents to ${codebase_dir}/${repo.name}/:
+- STACK.md - Languages, runtime, frameworks, dependencies, configuration
+- INTEGRATIONS.md - External APIs, databases, auth providers, webhooks
+
+Explore ${repo.path} thoroughly. Write documents directly using templates. Return confirmation only."
+)
+```
+
+**Agent: Architecture Focus (per repo)**
+
+```
+Task(
+  subagent_type="dgs-codebase-mapper",
+  model="{mapper_model}",
+  run_in_background=true,
+  description="Map ${repo.name} architecture",
+  prompt="Focus: arch
+repo_path: ${repo.path}
+repo_name: ${repo.name}
+
+Analyze the codebase at ${repo.path} for architecture and directory structure.
+
+Write these documents to ${codebase_dir}/${repo.name}/:
+- ARCHITECTURE.md - Pattern, layers, data flow, abstractions, entry points
+- STRUCTURE.md - Directory layout, key locations, naming conventions
+
+Explore ${repo.path} thoroughly. Write documents directly using templates. Return confirmation only."
+)
+```
+
+**Agent: Quality Focus (per repo)**
+
+```
+Task(
+  subagent_type="dgs-codebase-mapper",
+  model="{mapper_model}",
+  run_in_background=true,
+  description="Map ${repo.name} conventions",
+  prompt="Focus: quality
+repo_path: ${repo.path}
+repo_name: ${repo.name}
+
+Analyze the codebase at ${repo.path} for coding conventions and testing patterns.
+
+Write these documents to ${codebase_dir}/${repo.name}/:
+- CONVENTIONS.md - Code style, naming, patterns, error handling
+- TESTING.md - Framework, structure, mocking, coverage
+
+Explore ${repo.path} thoroughly. Write documents directly using templates. Return confirmation only."
+)
+```
+
+**Agent: Concerns Focus (per repo)**
+
+```
+Task(
+  subagent_type="dgs-codebase-mapper",
+  model="{mapper_model}",
+  run_in_background=true,
+  description="Map ${repo.name} concerns",
+  prompt="Focus: concerns
+repo_path: ${repo.path}
+repo_name: ${repo.name}
+
+Analyze the codebase at ${repo.path} for technical debt, known issues, and areas of concern.
+
+Write this document to ${codebase_dir}/${repo.name}/:
+- CONCERNS.md - Tech debt, bugs, security, performance, fragile areas
+
+Explore ${repo.path} thoroughly. Write document directly using template. Return confirmation only."
+)
+```
+
+Continue to collect_confirmations.
+</step>
+
+<step name="collect_confirmations">
+Wait for all agents across all repos to complete.
+
+Read each agent's output to collect confirmations.
+
+Track per-repo success/failure in a `results` map: `{ repo_name: "success" | "failed" }`.
+
+For each repo, as all 4 of its agents complete:
+- If all agents succeeded: mark repo as `"success"`, show: `Done: ${repo.name}`
+- If any agent failed or repo path was inaccessible: mark repo as `"failed"`, show: `Failed: ${repo.name} -- [error reason]`
+
+Continue with remaining repos regardless of individual failures.
+
+**Expected confirmation format from each agent:**
+```
+## Mapping Complete
+
+**Focus:** {focus}
+**Documents written:**
+- `${codebase_dir}/${repo.name}/{DOC1}.md` ({N} lines)
+- `${codebase_dir}/${repo.name}/{DOC2}.md` ({N} lines)
+
+Ready for orchestrator summary.
+```
+
+If any agent failed, note the failure and continue with successful documents.
+
+Continue to verify_output.
+</step>
+
+<step name="verify_output">
+Verify all documents created successfully for each repo:
+
+```bash
+for repo in ${valid_repos}; do
+  echo "--- ${repo} ---"
+  ls -la ${codebase_dir}/${repo}/
+  wc -l ${codebase_dir}/${repo}/*.md
+done
+```
+
+**Verification checklist (per repo):**
+- All 7 documents exist
+- No empty documents (each should have >20 lines)
+
+If any documents missing or empty, note which agents may have failed.
+
+Continue to synthesize_unified.
+</step>
+
+<step name="synthesize_unified">
+Synthesize unified top-level codebase files from per-repo content.
+
+**If ALL repos failed:** Skip synthesis entirely. Show: `All repos failed. No unified files generated.` Continue to offer_next.
+
+**Determine repo list for synthesis:**
+- **Refresh mode:** Use all successful repos from the results map.
+- **Update mode:** Use ALL repos that have per-repo subdirectories (not just the one being updated). The synthesizer needs content from all repos to produce complete unified files.
+
+```bash
+# Build list of repos with content for synthesis
+SYNTH_REPOS=""
+for dir in ${codebase_dir}/*/; do
+  REPO_NAME=$(basename "$dir")
+  SYNTH_REPOS="$SYNTH_REPOS $REPO_NAME"
+done
+```
+
+Show: `Synthesizing unified codebase files...`
+
+Spawn 7 parallel dgs-codebase-synthesizer agents (one per document type), all with `run_in_background=true`. Each agent produces one unified top-level file.
+
+**Document types:** ARCHITECTURE, STACK, STRUCTURE, CONVENTIONS, TESTING, INTEGRATIONS, CONCERNS
+
+**For each doc_type, spawn:**
+
+```
+Task(
+  subagent_type="dgs-codebase-synthesizer",
+  model="{mapper_model}",
+  run_in_background=true,
+  description="Synthesize unified ${doc_type}",
+  prompt="doc_type: ${doc_type}
+codebase_dir: ${codebase_dir}
+repo_names: ${JSON.stringify(synth_repos)}
+
+Read per-repo ${doc_type}.md files from each repo subdirectory in ${codebase_dir}/ and synthesize a unified ${codebase_dir}/${doc_type}.md.
+
+Include provenance header, conditional cross-repo summary (multi-repo only), and labeled repo sections. Return confirmation only."
+)
+```
+
+Wait for all 7 agents to complete.
+
+Verify all 7 unified files exist:
+
+```bash
+for doc in ARCHITECTURE STACK STRUCTURE CONVENTIONS TESTING INTEGRATIONS CONCERNS; do
+  test -f ${codebase_dir}/${doc}.md && echo "OK: ${doc}.md" || echo "MISSING: ${doc}.md"
+done
+```
+
+Show: `Unified files generated.`
+
+Continue to generate_cross_repo.
+</step>
+
+<step name="generate_cross_repo">
+Generate CROSS-REPO.md with cross-repo analysis (comparison tables for shared dependencies, API boundaries, common/divergent patterns).
+
+**Guard check:** If `synth_repos` has fewer than 2 repos, skip this step entirely. Delete any existing CROSS-REPO.md left from a previous multi-repo run:
+
+```bash
+rm -f ${codebase_dir}/CROSS-REPO.md
+```
+
+**If 2+ repos:** Spawn a single cross-analyzer agent (NOT parallel -- it needs cross-repo context):
+
+```
+Task(
+  subagent_type="dgs-codebase-cross-analyzer",
+  model="{mapper_model}",
+  description="Generate cross-repo analysis",
+  prompt="codebase_dir: ${codebase_dir}
+repo_names: ${JSON.stringify(synth_repos)}
+
+Read per-repo documents from each ${codebase_dir}/<repo>/ subdirectory.
+Compare across repos to identify:
+1. Shared dependencies (exact package matches from STACK.md)
+2. API boundaries (from ARCHITECTURE.md and INTEGRATIONS.md)
+3. Common patterns (from CONVENTIONS.md, ARCHITECTURE.md, TESTING.md)
+4. Divergent patterns (same sources, items that differ)
+
+Write ${codebase_dir}/CROSS-REPO.md with comparison tables.
+Do NOT explore repo paths directly. Work only from per-repo maps."
+)
+```
+
+Wait for agent to complete. Show: `Cross-repo analysis generated.`
+
+Continue to verify_compatibility.
+</step>
+
+<step name="verify_compatibility">
+Verify that unified files are compatible with all consumer workflows (write-spec, refine-spec, import-spec, new-project, plan-phase, find-related-ideas, execute-plan). These consumers expect files at `${codebase_dir}/*.md` -- either by exact path (e.g., `${codebase_dir}/ARCHITECTURE.md`) or by glob (e.g., `ls ${codebase_dir}/*.md`).
+
+**1. Check all 7 unified files exist:**
+
+```bash
+EXPECTED="ARCHITECTURE CONCERNS CONVENTIONS INTEGRATIONS STACK STRUCTURE TESTING"
+MISSING=""
+for doc in $EXPECTED; do
+  test -f ${codebase_dir}/${doc}.md || MISSING="$MISSING $doc"
+done
+if [ -n "$MISSING" ]; then
+  echo "WARNING: Missing unified files:$MISSING"
+fi
+```
+
+**2. Check provenance headers:**
+
+Each unified file must have a provenance comment on the first line (written by the synthesizer agent):
+
+```bash
+for doc in ${codebase_dir}/*.md; do
+  head -1 "$doc" | grep -q "Synthesized:" || echo "WARNING: $doc missing provenance header"
+done
+```
+
+**3. Check minimum viable content:**
+
+Each unified file should have meaningful content (>10 lines):
+
+```bash
+for doc in ${codebase_dir}/*.md; do
+  LINES=$(wc -l < "$doc")
+  [ "$LINES" -lt 10 ] && echo "WARNING: $doc only has $LINES lines"
+done
+```
+
+**4. Check CROSS-REPO.md (conditional):**
+
+If 2+ repos were mapped (synth_repos has 2+ entries), verify CROSS-REPO.md exists:
+
+```bash
+# Verify CROSS-REPO.md exists when 2+ repos were mapped
+if [ ${#synth_repos[@]} -ge 2 ]; then
+  test -f ${codebase_dir}/CROSS-REPO.md || echo "WARNING: CROSS-REPO.md missing (expected for multi-repo)"
+fi
+```
+
+If any warnings were emitted, log them but continue -- these are non-blocking warnings that help diagnose synthesis issues.
+
+Continue to scan_for_secrets.
+</step>
+
+<step name="scan_for_secrets">
+**CRITICAL SECURITY CHECK:** Scan output files for accidentally leaked secrets before committing.
+
+Silent when clean -- only produces output if potential secrets are detected. Warn-only in all modes (never halts the workflow).
+
+```
+Secret pattern categories:
+- API keys: OpenAI (sk-), Stripe (sk_live_, sk_test_)
+- Platform tokens: GitHub (ghp_, gho_), GitLab (glpat-), Slack (xox)
+- Cloud credentials: AWS access keys (AKIA)
+- Cryptographic: Private keys (BEGIN PRIVATE KEY), JWTs (eyJ)
+- Connection strings: PostgreSQL, MongoDB, MySQL, Redis, AMQP with embedded credentials
+- Hardcoded secrets: Password assignments with 8+ char values
+```
+
+```bash
+# Build comprehensive secret detection pattern
+SECRET_PATTERN='(sk-[a-zA-Z0-9]{20,}|sk_live_[a-zA-Z0-9]+|sk_test_[a-zA-Z0-9]+|ghp_[a-zA-Z0-9]{36}|gho_[a-zA-Z0-9]{36}|glpat-[a-zA-Z0-9_-]+|AKIA[A-Z0-9]{16}|xox[baprs]-[a-zA-Z0-9-]+|-----BEGIN.*PRIVATE KEY|eyJ[a-zA-Z0-9_-]+\.eyJ[a-zA-Z0-9_-]+\.|postgresql://[^:]+:[^@]+@|mongodb(\+srv)?://[^:]+:[^@]+@|mysql://[^:]+:[^@]+@|redis://:[^@]+@|amqp://[^:]+:[^@]+@|[pP]assword\s*[:=]\s*['"'"'"][^'"'"'"]{8,})'
+
+SECRETS_OUTPUT=""
+
+# Determine which repos to scan based on mode
+# Refresh mode: scan all valid_repos
+# Update mode: scan only the targeted repo
+if [ "${mode}" = "update" ]; then
+  repos_to_scan="${only_repo}"
+else
+  repos_to_scan="${valid_repos}"
+fi
+
+# Scan per-repo files
+for repo in ${repos_to_scan}; do
+  FOUND=$(grep -nE "$SECRET_PATTERN" ${codebase_dir}/${repo}/*.md 2>/dev/null || true)
+  if [ -n "$FOUND" ]; then
+    SECRETS_OUTPUT="${SECRETS_OUTPUT}\n--- ${repo} ---\n${FOUND}"
+  fi
+done
+
+# Scan unified top-level files (always, regardless of mode)
+FOUND=$(grep -nE "$SECRET_PATTERN" ${codebase_dir}/*.md 2>/dev/null || true)
+if [ -n "$FOUND" ]; then
+  SECRETS_OUTPUT="${SECRETS_OUTPUT}\n--- unified ---\n${FOUND}"
+fi
+
+# Only show output if secrets found (silent when clean)
+if [ -n "$SECRETS_OUTPUT" ]; then
+  echo "WARNING: Potential secrets detected in codebase files:"
+  echo -e "$SECRETS_OUTPUT"
+  echo ""
+  echo "Review these files before committing. Secrets are left as-is -- review and redact manually if needed."
+fi
+```
+
+**Behavior (all modes):**
+- If secrets found: show warning with file + line numbers, continue to commit
+- If no secrets found: produce no output (silent when clean)
+- Never halts the workflow
+
+Continue to commit_codebase_map.
+</step>
+
+<step name="commit_codebase_map">
+Commit the codebase maps for all repos:
+
+```bash
+node ~/.claude/deliver-great-systems/bin/dgs-tools.cjs commit "docs: map codebase for registered repos" --files ${codebase_dir}/
+```
+
+Continue to offer_next.
+</step>
+
+<step name="offer_next">
+Present completion summary and next steps.
+
+**Mapping summary (before file listing):**
+
+```
+Mapping summary:
+- Mode: [refresh | update (only_repo)]
+- Repos mapped: N of M
+  [For each repo: success_mark repo-name or fail_mark repo-name (reason)]
+```
+
+Use checkmarks for success and X marks for failure.
+
+**Get line counts per repo and unified files:**
+```bash
+for repo in ${valid_repos}; do
+  echo "--- ${repo} ---"
+  wc -l ${codebase_dir}/${repo}/*.md
+done
+echo "--- Unified ---"
+wc -l ${codebase_dir}/*.md
+```
+
+**Output format:**
+
+```
+Codebase mapping complete.
+
+Mapping summary:
+- Mode: [refresh | update (only_repo)]
+- Repos mapped: N of M
+  [For each repo: checkmark/X repo-name]
+
+[For each repo:]
+${repo.name} (${codebase_dir}/${repo.name}/):
+- STACK.md ([N] lines) - Technologies and dependencies
+- ARCHITECTURE.md ([N] lines) - System design and patterns
+- STRUCTURE.md ([N] lines) - Directory layout and organization
+- CONVENTIONS.md ([N] lines) - Code style and patterns
+- TESTING.md ([N] lines) - Test structure and practices
+- INTEGRATIONS.md ([N] lines) - External services and APIs
+- CONCERNS.md ([N] lines) - Technical debt and issues
+
+Unified files (${codebase_dir}/):
+- ARCHITECTURE.md ([N] lines) - Synthesized architecture across all repos
+- STACK.md ([N] lines) - Synthesized technology stack across all repos
+- STRUCTURE.md ([N] lines) - Synthesized codebase structure across all repos
+- CONVENTIONS.md ([N] lines) - Synthesized coding conventions across all repos
+- TESTING.md ([N] lines) - Synthesized testing patterns across all repos
+- INTEGRATIONS.md ([N] lines) - Synthesized external integrations across all repos
+- CONCERNS.md ([N] lines) - Synthesized codebase concerns across all repos
+
+[If CROSS-REPO.md exists (2+ repos):]
+Cross-repo analysis (${codebase_dir}/):
+- CROSS-REPO.md ([N] lines) - Shared dependencies, API boundaries, pattern comparison
+
+---
+
+## Next Up
+
+**Initialize project** — use codebase context for planning
+
+`/dgs:new-project`
+
+<sub>`/clear` first -> fresh context window</sub>
+
+---
+
+**Also available:**
+- Re-run mapping: `/dgs:map-codebase`
+- Update a single repo: `/dgs:map-codebase --only <repo-name>`
+- Review specific repo: `cat ${codebase_dir}/<repo>/STACK.md`
+- Review unified files: `cat ${codebase_dir}/ARCHITECTURE.md`
+- Review cross-repo analysis: `cat ${codebase_dir}/CROSS-REPO.md`
+- Edit any document before proceeding
+
+---
+```
+
+End workflow.
+</step>
+
+</process>
+
+<success_criteria>
+- Per-repo subdirectories created at ${codebase_dir}/<repo-name>/
+- 4 parallel dgs-codebase-mapper agents spawned per repo with run_in_background=true
+- Agents explore repo_path (not planning repo) and write documents directly
+- Read agent output files to collect per-repo confirmations
+- All 7 codebase documents exist per repo
+- Per-repo progress announcements shown
+- 7 parallel dgs-codebase-synthesizer agents produce unified top-level files
+- All 7 unified files exist at ${codebase_dir}/*.md
+- Compatibility verification checks file existence, provenance headers, and minimum content
+- Secret scanning covers both per-repo and unified files
+- Commit captures all codebase content (per-repo and unified)
+- Clear completion summary with per-repo and unified file line counts
+- User offered clear next steps in DGS style
+- Error handling: failed repos skipped, remaining repos continue
+- End-of-run summary with success/fail counts
+- Mode-aware cleanup (refresh wipes all, update wipes only target repo)
+- Cross-repo analysis generated for 2+ repos (CROSS-REPO.md)
+</success_criteria>