@leanspec/ui 0.2.7-dev.20251126100750 → 0.2.7

package/.next/standalone/packages/ui/dist/standalone/specs/059-programmatic-spec-management/README.md

@@ -0,0 +1,504 @@
+---
+status: complete
+created: '2025-11-07'
+tags:
+  - context-engineering
+  - automation
+  - tooling
+  - ai-agents
+  - performance
+  - v0.3.0
+priority: critical
+created_at: '2025-11-07T11:28:43.206Z'
+depends_on:
+  - 067-monorepo-core-extraction
+  - 069-token-counting-utils
+related:
+  - 048-spec-complexity-analysis
+  - 049-leanspec-first-principles
+  - 018-spec-validation
+  - 012-sub-spec-files
+  - 066-context-economy-thresholds-refinement
+updated_at: '2025-11-16T13:59:12.743Z'
+transitions:
+  - status: in-progress
+    at: '2025-11-13T10:24:05.467Z'
+  - status: complete
+    at: '2025-11-13T14:21:42.907Z'
+completed_at: '2025-11-13T14:21:42.907Z'
+completed: '2025-11-13'
+---
+
+# Programmatic Spec Management & Context Engineering
+
+> **Status**: ✅ Complete · **Priority**: Critical · **Created**: 2025-11-07 · **Tags**: context-engineering, automation, tooling, ai-agents, performance, v0.3.0
+
+**The Problem**: AI agents manually editing oversized spec files is slow and error-prone. They need clean, mechanical tools to transform specs without direct markdown manipulation.
+
+**The Solution**: Provide programmatic transformation commands that AI agents can orchestrate. AI agents analyze specs and call tools with explicit parameters - tools execute transformations mechanically without LLM calls.
+
+## Overview
+
+### Critical Performance Issue
+
+**Current Reality**:
+- AI agents manually editing 1,166-line markdown files → slow, error-prone
+- Text corruption during large multi-replace operations
+- Context window pollution from oversized specs
+- Manual markdown editing by AI is fundamentally inefficient
+
+**Root Cause**: AI agents lack clean tools to transform specs programmatically - they resort to direct markdown editing.
+
+**Impact**:
+- ❌ Spec 045 (4,800 tokens): AI struggles to edit coherently
+- ❌ Context window waste processing oversized specs
+- ❌ Risk of file corruption during complex transformations
+- ❌ Violation of our own Context Economy principle
+
+### The AI Agent Orchestration Model
+
+**Key Insight**: AI agents should orchestrate transformations, not perform them manually.
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  AI Agent (GitHub Copilot, Claude, etc.)                │
+│  - Reads spec files                                      │
+│  - Detects issues (token count, redundancy, etc.)       │
+│  - Decides transformation strategy                       │
+│  - Calls tools with explicit parameters                  │
+└─────────────────────────────────────────────────────────┘
+                           ↓
+┌─────────────────────────────────────────────────────────┐
+│  LeanSpec CLI Tools (No LLM, Pure Execution)            │
+│  - Parse markdown structure                              │
+│  - Execute mechanical transformations                    │
+│  - Validate results                                      │
+│  - No semantic analysis, no LLM calls                    │
+└─────────────────────────────────────────────────────────┘
+                           ↓
+                  Transformed Specs
+```
+
+**Benefits**:
+- ✅ Fast: No LLM text generation for file operations
+- ✅ Reliable: Deterministic, testable transformations
+- ✅ Clean: AI agents don't touch markdown directly
+- ✅ Composable: Tools are building blocks AI agents orchestrate
+
+### Context Engineering Foundation
+
+Based on research from [Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents), [LangChain](https://blog.langchain.com/context-engineering-for-agents/), and [Drew Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html):
+
+**Four Core Strategies**:
+1. **Partitioning** - Split into sub-specs (what we do in spec 012)
+2. **Compaction** - Remove redundancy, preserve signal
+3. **Compression** - Summarize without losing intent
+4. **Isolation** - Move unrelated concerns to separate specs
+
+**Four Context Failure Modes** (what LeanSpec addresses):
+1. **Context Poisoning** - Hallucinations accumulate in spec history
+2. **Context Distraction** - Spec length overwhelms trained knowledge
+3. **Context Confusion** - Superfluous content influences decisions
+4. **Context Clash** - Conflicting information within same spec
+
+### What We're Building
+
+**Mechanical transformation tools for AI agent orchestration**:
+- ✅ Parse markdown structure (sections, line ranges, tokens)
+- ✅ Analyze complexity algorithmically (metrics, patterns)
+- ✅ Execute transformations mechanically (split, move, merge)
+- ✅ Validate results automatically (structure, references)
+- ⚡ No LLM calls - AI agents provide the intelligence
+
+**AI Agent Workflow**:
+1. Agent reads spec → detects issue (e.g., 4,800 tokens)
+2. Agent decides strategy (e.g., "split by concerns")
+3. Agent calls tool with explicit parameters (e.g., section mappings)
+4. Tool executes transformation mechanically
+5. Agent reviews result and continues or adjusts
+
+**Why This Works**:
+- AI agents already have context understanding
+- Tools just need to execute what AI decides
+- Clean separation: intelligence (AI) vs execution (tools)
+
+## The Vision
+
+```bash
+# AI Agent Workflow Example:
+
+# 1. AI agent detects issue
+$ lean-spec analyze 045 --json
+{
+  "tokens": 4800,
+  "threshold": "warning",
+  "concerns": [
+    {"name": "Overview", "sections": ["Overview", "Background"], "lines": "1-150"},
+    {"name": "Design", "sections": ["Architecture", "Components"], "lines": "151-528"},
+    {"name": "Testing", "sections": ["Test Strategy", "Test Cases"], "lines": "529-710"}
+  ],
+  "recommendation": "split"
+}
+
+# 2. AI agent decides: "I'll split by concerns"
+
+# 3. AI agent calls tool with explicit parameters
+$ lean-spec split 045 \
+  --output=README.md:1-150 \
+  --output=DESIGN.md:151-528 \
+  --output=TESTING.md:529-710 \
+  --update-refs
+
+# Tool executes mechanically (no LLM):
+# ✓ Created README.md (812 tokens / 150 lines)
+# ✓ Created DESIGN.md (1,512 tokens / 378 lines)
+# ✓ Created TESTING.md (728 tokens / 182 lines)
+# ✓ Updated 47 cross-references
+# ✓ Validated all files
+
+# 4. AI agent verifies result
+$ lean-spec tokens 045/*
+# 045-unified-dashboard/README.md: 812 tokens
+# 045-unified-dashboard/DESIGN.md: 1,512 tokens
+# 045-unified-dashboard/TESTING.md: 728 tokens
+# Total: 3,052 tokens (saved 1,748 via compaction)
+```
+
+**Key Difference from Current Approach**:
+- ❌ Old: AI manually rewrites markdown → slow, error-prone
+- ✅ New: AI orchestrates tools → fast, deterministic
+
+## Sub-Specs
+
+This spec is organized using sub-spec files:
+
+- **[CONTEXT-ENGINEERING.md](./CONTEXT-ENGINEERING.md)** - Research: 4 strategies, 4 failure modes, academic synthesis
+- **[ARCHITECTURE.md](./ARCHITECTURE.md)** - System design: AI agent orchestration, mechanical tools, simple parsing
+- **[COMMANDS.md](./COMMANDS.md)** - CLI reference: analyze, split, compact, compress, isolate with AI agent examples
+- **[IMPLEMENTATION.md](./IMPLEMENTATION.md)** - Roadmap: 4-week plan, simplified from original 7-week complexity
+- **[TESTING.md](./TESTING.md)** - Test strategy: unit tests, integration tests, real-world validation
+
+## Quick Reference
+
+### Context Engineering Strategies
+
+| Strategy | Purpose | When to Use | Tool |
+|----------|---------|-------------|------|
+| **Partition** | Split into sub-specs | Spec >3,500 tokens, multiple concerns | `lean-spec split` |
+| **Compact** | Remove redundancy | Verbose, repetitive content | `lean-spec compact` |
+| **Compress** | Summarize sections | Historical context, completed phases | `lean-spec compress` |
+| **Isolate** | Move to separate spec | Unrelated concern, different lifecycle | `lean-spec isolate` |
+
+### Context Failure Detection
+
+| Failure Mode | Symptom | Detection | Mitigation |
+|--------------|---------|-----------|------------|
+| **Poisoning** | AI references non-existent content | Validate references | Remove corrupted sections |
+| **Distraction** | AI ignores training, repeats spec | Track spec token count | Split at 3,500 tokens |
+| **Confusion** | AI uses irrelevant context | Identify superfluous sections | Compact/remove noise |
+| **Clash** | AI contradicts itself | Detect conflicting statements | Resolve or isolate |
+
+### Commands Preview
+
+```bash
+# Analyze spec (returns JSON for AI agent)
+lean-spec analyze <spec> --json
+
+# Transform specs (AI agent provides parameters)
+lean-spec split <spec> --output=FILE:LINES [--output=...]
+lean-spec compact <spec> --remove=LINES [--remove=...]
+lean-spec compress <spec> --replace=LINES:TEXT
+lean-spec isolate <spec> --lines=RANGE --to=NEW_SPEC
+
+# Utilities
+lean-spec diff <spec> --before-after
+lean-spec preview <spec> --split=FILE:LINES
+lean-spec rollback <spec>
+```
+
+## Status
+
+**Current Phase**: 📋 Planning & Design
+
+**Next Steps**:
+1. Complete sub-spec documentation
+2. Review with team
+3. Begin implementation (Phase 1: Parser)
+
+## Key Principles
+
+### Why AI Agent Orchestration Works
+
+**AI Agent Strengths** (provide intelligence):
+- Understanding spec content
+- Detecting issues (oversized, redundant, contradictory)
+- Deciding transformation strategy
+- Determining split points, what to remove
+- Reviewing and verifying results
+
+**Tool Strengths** (provide execution):
+- Fast file operations
+- Deterministic behavior
+- No hallucinations
+- Syntax validation
+- Reference updating
+
+**Clean Separation**:
+```
+AI Agent: "Split this 4,800 token spec at lines 1-150, 151-528, 529-710"
+  ↓
+Tool: [mechanically extracts line ranges, creates files, validates]
+  ↓
+AI Agent: "Verify: all files under 2,000 tokens" → ✓
+```
+
+**Why This is Better**:
+- ✅ AI agents already have context (no need to re-analyze in tool)
+- ✅ Tools are simple and fast (no LLM calls)
+- ✅ Deterministic (same params = same result)
+- ✅ Testable (no AI unpredictability)
+
+### Context Engineering as First Principle
+
+This builds on **Context Economy** (Principle #1 from spec 049):
+- Specs must fit in working memory
+- <2,000 tokens excellent, >3,500 tokens warning, >5,000 tokens should split
+- But splitting shouldn't require 10 minutes of LLM text generation
+
+**Evolution**:
+```
+v0.1.0: Manual spec writing
+v0.2.0: Detection + warnings (lean-spec validate)
+v0.3.0: Programmatic transformation (this spec)
+v0.4.0: Continuous context management (auto-compaction, etc.)
+```
+
+## Plan
+
+### Phase 1: Foundation (Week 1) ✅ COMPLETE
+- [x] Markdown AST parser (unified.js ecosystem)
+- [x] Spec structure analyzer
+- [x] Boundary detection algorithms
+- [x] Core data structures
+
+### Phase 2: Analysis Tools (Week 2) ✅ COMPLETE
+- [x] `lean-spec analyze --complexity`
+- [x] `lean-spec analyze --json` (for AI agents)
+- [x] Visual reports
+
+### Phase 3: Transformation Engine (Week 3) ✅ COMPLETE
+- [x] `lean-spec split` - Partition specs into sub-specs
+- [x] `lean-spec compact` - Remove redundancy
+- [x] `lean-spec compress` - Replace with summaries
+- [x] `lean-spec isolate` - Move to new spec
+
+### Phase 4: Testing & Launch (Week 4) ✅ COMPLETE
+- [x] Test all commands
+- [x] Add comprehensive test coverage
+- [x] CLI integration and polish
+- [x] Documentation and help text
+
+**Implementation Status**: All 5 transformation commands are now available in v0.2.2+
+
+## Usage Examples
+
+### Analyze Spec Complexity
+
+```bash
+# Get structured analysis (JSON output for AI agents)
+lean-spec analyze 059 --json
+
+# Human-readable output with recommendations
+lean-spec analyze 045 --verbose
+```
+
+### Split Spec into Sub-Specs
+
+```bash
+# Split by explicit line ranges (AI agent provides ranges)
+lean-spec split 045 \
+  --output=README.md:1-150 \
+  --output=DESIGN.md:151-528 \
+  --output=TESTING.md:529-710 \
+  --update-refs
+
+# Preview before applying
+lean-spec split 045 --output=README.md:1-150 --dry-run
+```
+
+### Compact Redundant Content
+
+```bash
+# Remove specified line ranges (AI agent identifies redundancy)
+lean-spec compact 045 \
+  --remove=145-153 \
+  --remove=234-256 \
+  --remove=401-415
+
+# Preview what would be removed
+lean-spec compact 045 --remove=145-153 --dry-run
+```
+
+### Compress with Summaries
+
+```bash
+# Replace verbose sections with AI-provided summaries
+lean-spec compress 043 \
+  --replace='142-284:## ✅ Phase 1: Completed
+
+Established first principles. See: specs/049/'
+
+# Preview compression
+lean-spec compress 043 --replace='142-284:Summary here' --dry-run
+```
+
+### Isolate Content to New Spec
+
+```bash
+# Move independent sections to separate specs
+lean-spec isolate 045 \
+  --lines=401-542 \
+  --to=060-velocity-algorithm \
+  --add-reference
+
+# Preview isolation
+lean-spec isolate 045 --lines=401-542 --to=060-new-spec --dry-run
+```
+
+For detailed command documentation, see [COMMANDS.md](./COMMANDS.md).
+
+## Test
+
+### Validation Criteria
+
+**Performance**:
+- [ ] Split 4,800-token spec in <1 second (vs 10+ minutes manual)
+- [ ] Parse/analyze 100 specs in <2 seconds
+- [ ] Zero text corruption (programmatic = deterministic)
+
+**Correctness**:
+- [ ] Preserves all content (no information loss)
+- [ ] Maintains markdown validity
+- [ ] Updates all cross-references correctly
+- [ ] Frontmatter remains valid
+
+**Usability**:
+- [ ] Clear analysis reports
+- [ ] Interactive preview before applying
+- [ ] Undo/rollback capability
+- [ ] Helpful error messages
+
+### Test Approach
+
+**Golden Tests**:
+- Snapshot known-good transformations
+- Regression testing against corpus
+- Compare manual vs programmatic splits
+
+**Dogfooding**:
+- Use tools on our own oversized specs
+- Validate against specs 045, 046, 048 splits
+- Measure time savings vs manual approach
+
+**Edge Cases**:
+- Specs with complex nested structures
+- Specs with many code blocks
+- Specs with tables and diagrams
+- Specs with cross-references
+
+## Success Metrics
+
+### Quantitative
+
+**Speed**:
+- 100x faster than LLM text generation
+- <1s to split any spec <8,000 tokens
+- <2s to analyze entire project
+
+**Quality**:
+- Zero corruption incidents
+- 100% markdown validity preserved
+- 100% frontmatter validity preserved
+- 100% cross-references updated
+
+### Qualitative
+
+**Developer Experience**:
+- "Splitting specs is now instant"
+- "No more babysitting AI rewrites"
+- "Confident transformations won't corrupt"
+- "Can experiment with splits freely"
+
+**Impact**:
+- Enables proactive splitting at 3,500 tokens (warning threshold)
+- Removes friction from Context Economy
+- Makes LeanSpec principles easier to follow
+- Dogfooding our own methodology effectively
+
+## Notes
+
+### Research Synthesis
+
+The external references identified four key insights:
+
+1. **Context is Finite** (Anthropic): Even 1M token windows experience "context rot"—attention degrades with length
+2. **Four Strategies** (LangChain): Write, Select, Compress, Isolate for managing context
+3. **Four Failure Modes** (Breunig): Poisoning, Distraction, Confusion, Clash
+4. **Hybrid Approach**: AI for strategy, code for execution
+
+### Why This Matters
+
+**For LeanSpec**:
+- ✅ Practices our own principles (Context Economy)
+- ✅ Removes major pain point (slow manual splitting)
+- ✅ Enables proactive management (split at 300, not 600)
+- ✅ Makes AI agents more effective (faster, fewer errors)
+
+**For Users**:
+- ✅ Faster workflow (seconds vs minutes)
+- ✅ Higher confidence (deterministic transforms)
+- ✅ Better specs (easy to maintain context limits)
+- ✅ Learning tool (see how specs should be structured)
+
+### Alternatives Considered
+
+**1. Pure AI Approach** (current, rejected):
+- ❌ Too slow (10+ minutes per spec)
+- ❌ Error-prone (context corruption)
+- ❌ Not deterministic (varies by run)
+
+**2. Manual Guidelines Only** (rejected):
+- ❌ Relies on discipline
+- ❌ Still slow when needed
+- ❌ No automation assistance
+
+**3. Hybrid Approach** (chosen):
+- ✅ AI suggests, code executes
+- ✅ Fast (programmatic) + smart (AI)
+- ✅ Best of both worlds
+
+### Open Questions
+
+1. **AST Library**: unified.js (remark) vs custom parser?
+   - Leaning toward unified.js (battle-tested, ecosystem)
+
+2. **LLM Integration**: When to use AI vs pure code?
+   - AI for: Suggesting concerns, reviewing results
+   - Code for: Parsing, moving content, updating refs
+
+3. **Preview UX**: How to show transformation preview?
+   - Interactive diff view? Side-by-side? Git-style?
+
+4. **Undo Mechanism**: Git commits? Custom snapshots?
+   - Probably git-based (user is already in git)
+
+## Related Specs
+
+- **[048-spec-complexity-analysis](../048-spec-complexity-analysis/)** - Identified the problem
+- **[049-leanspec-first-principles](../049-leanspec-first-principles/)** - Context Economy principle
+- **[018-spec-validation](../018-spec-validation/)** - Validation framework
+- **[012-sub-spec-files](../012-sub-spec-files/)** - Sub-spec pattern we're automating
+
+---
+
+**Remember**: Context engineering isn't about bigger windows—it's about smarter curation. Programmatic tools make curation fast and reliable.