@leanspec/ui 0.2.8 → 0.2.9-dev.20251205030455

package/.next/standalone/packages/ui/dist/standalone/specs/059-programmatic-spec-management/README.md

@@ -1,504 +0,0 @@
----
-status: complete
-created: '2025-11-07'
-tags:
-  - context-engineering
-  - automation
-  - tooling
-  - ai-agents
-  - performance
-  - v0.3.0
-priority: critical
-created_at: '2025-11-07T11:28:43.206Z'
-depends_on:
-  - 067-monorepo-core-extraction
-  - 069-token-counting-utils
-related:
-  - 048-spec-complexity-analysis
-  - 049-leanspec-first-principles
-  - 018-spec-validation
-  - 012-sub-spec-files
-  - 066-context-economy-thresholds-refinement
-updated_at: '2025-11-16T13:59:12.743Z'
-transitions:
-  - status: in-progress
-    at: '2025-11-13T10:24:05.467Z'
-  - status: complete
-    at: '2025-11-13T14:21:42.907Z'
-completed_at: '2025-11-13T14:21:42.907Z'
-completed: '2025-11-13'
----
-
-# Programmatic Spec Management & Context Engineering
-
-> **Status**: ✅ Complete · **Priority**: Critical · **Created**: 2025-11-07 · **Tags**: context-engineering, automation, tooling, ai-agents, performance, v0.3.0
-
-**The Problem**: AI agents manually editing oversized spec files is slow and error-prone. They need clean, mechanical tools to transform specs without direct markdown manipulation.
-
-**The Solution**: Provide programmatic transformation commands that AI agents can orchestrate. AI agents analyze specs and call tools with explicit parameters - tools execute transformations mechanically without LLM calls.
-
-## Overview
-
-### Critical Performance Issue
-
-**Current Reality**:
-- AI agents manually editing 1,166-line markdown files → slow, error-prone
-- Text corruption during large multi-replace operations
-- Context window pollution from oversized specs
-- Manual markdown editing by AI is fundamentally inefficient
-
-**Root Cause**: AI agents lack clean tools to transform specs programmatically - they resort to direct markdown editing.
-
-**Impact**:
-- ❌ Spec 045 (4,800 tokens): AI struggles to edit coherently
-- ❌ Context window waste processing oversized specs
-- ❌ Risk of file corruption during complex transformations
-- ❌ Violation of our own Context Economy principle
-
-### The AI Agent Orchestration Model
-
-**Key Insight**: AI agents should orchestrate transformations, not perform them manually.
-
-```
-┌─────────────────────────────────────────────────────────┐
-│  AI Agent (GitHub Copilot, Claude, etc.)                │
-│  - Reads spec files                                      │
-│  - Detects issues (token count, redundancy, etc.)       │
-│  - Decides transformation strategy                       │
-│  - Calls tools with explicit parameters                  │
-└─────────────────────────────────────────────────────────┘
-                           ↓
-┌─────────────────────────────────────────────────────────┐
-│  LeanSpec CLI Tools (No LLM, Pure Execution)            │
-│  - Parse markdown structure                              │
-│  - Execute mechanical transformations                    │
-│  - Validate results                                      │
-│  - No semantic analysis, no LLM calls                    │
-└─────────────────────────────────────────────────────────┘
-                           ↓
-                  Transformed Specs
-```
-
-**Benefits**:
-- ✅ Fast: No LLM text generation for file operations
-- ✅ Reliable: Deterministic, testable transformations
-- ✅ Clean: AI agents don't touch markdown directly
-- ✅ Composable: Tools are building blocks AI agents orchestrate
-
-### Context Engineering Foundation
-
-Based on research from [Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents), [LangChain](https://blog.langchain.com/context-engineering-for-agents/), and [Drew Breunig](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html):
-
-**Four Core Strategies**:
-1. **Partitioning** - Split into sub-specs (what we do in spec 012)
-2. **Compaction** - Remove redundancy, preserve signal
-3. **Compression** - Summarize without losing intent
-4. **Isolation** - Move unrelated concerns to separate specs
-
-**Four Context Failure Modes** (what LeanSpec addresses):
-1. **Context Poisoning** - Hallucinations accumulate in spec history
-2. **Context Distraction** - Spec length overwhelms trained knowledge
-3. **Context Confusion** - Superfluous content influences decisions
-4. **Context Clash** - Conflicting information within same spec
-
-### What We're Building
-
-**Mechanical transformation tools for AI agent orchestration**:
-- ✅ Parse markdown structure (sections, line ranges, tokens)
-- ✅ Analyze complexity algorithmically (metrics, patterns)
-- ✅ Execute transformations mechanically (split, move, merge)
-- ✅ Validate results automatically (structure, references)
-- ⚡ No LLM calls - AI agents provide the intelligence
-
-**AI Agent Workflow**:
-1. Agent reads spec → detects issue (e.g., 4,800 tokens)
-2. Agent decides strategy (e.g., "split by concerns")
-3. Agent calls tool with explicit parameters (e.g., section mappings)
-4. Tool executes transformation mechanically
-5. Agent reviews result and continues or adjusts
-
-**Why This Works**:
-- AI agents already have context understanding
-- Tools just need to execute what AI decides
-- Clean separation: intelligence (AI) vs execution (tools)
-
-## The Vision
-
-```bash
-# AI Agent Workflow Example:
-
-# 1. AI agent detects issue
-$ lean-spec analyze 045 --json
-{
-  "tokens": 4800,
-  "threshold": "warning",
-  "concerns": [
-    {"name": "Overview", "sections": ["Overview", "Background"], "lines": "1-150"},
-    {"name": "Design", "sections": ["Architecture", "Components"], "lines": "151-528"},
-    {"name": "Testing", "sections": ["Test Strategy", "Test Cases"], "lines": "529-710"}
-  ],
-  "recommendation": "split"
-}
-
-# 2. AI agent decides: "I'll split by concerns"
-
-# 3. AI agent calls tool with explicit parameters
-$ lean-spec split 045 \
-  --output=README.md:1-150 \
-  --output=DESIGN.md:151-528 \
-  --output=TESTING.md:529-710 \
-  --update-refs
-
-# Tool executes mechanically (no LLM):
-# ✓ Created README.md (812 tokens / 150 lines)
-# ✓ Created DESIGN.md (1,512 tokens / 378 lines)
-# ✓ Created TESTING.md (728 tokens / 182 lines)
-# ✓ Updated 47 cross-references
-# ✓ Validated all files
-
-# 4. AI agent verifies result
-$ lean-spec tokens 045/*
-# 045-unified-dashboard/README.md: 812 tokens
-# 045-unified-dashboard/DESIGN.md: 1,512 tokens
-# 045-unified-dashboard/TESTING.md: 728 tokens
-# Total: 3,052 tokens (saved 1,748 via compaction)
-```
-
-**Key Difference from Current Approach**:
-- ❌ Old: AI manually rewrites markdown → slow, error-prone
-- ✅ New: AI orchestrates tools → fast, deterministic
-
-## Sub-Specs
-
-This spec is organized using sub-spec files:
-
-- **[CONTEXT-ENGINEERING.md](./CONTEXT-ENGINEERING.md)** - Research: 4 strategies, 4 failure modes, academic synthesis
-- **[ARCHITECTURE.md](./ARCHITECTURE.md)** - System design: AI agent orchestration, mechanical tools, simple parsing
-- **[COMMANDS.md](./COMMANDS.md)** - CLI reference: analyze, split, compact, compress, isolate with AI agent examples
-- **[IMPLEMENTATION.md](./IMPLEMENTATION.md)** - Roadmap: 4-week plan, simplified from original 7-week complexity
-- **[TESTING.md](./TESTING.md)** - Test strategy: unit tests, integration tests, real-world validation
-
-## Quick Reference
-
-### Context Engineering Strategies
-
-| Strategy | Purpose | When to Use | Tool |
-|----------|---------|-------------|------|
-| **Partition** | Split into sub-specs | Spec >3,500 tokens, multiple concerns | `lean-spec split` |
-| **Compact** | Remove redundancy | Verbose, repetitive content | `lean-spec compact` |
-| **Compress** | Summarize sections | Historical context, completed phases | `lean-spec compress` |
-| **Isolate** | Move to separate spec | Unrelated concern, different lifecycle | `lean-spec isolate` |
-
-### Context Failure Detection
-
-| Failure Mode | Symptom | Detection | Mitigation |
-|--------------|---------|-----------|------------|
-| **Poisoning** | AI references non-existent content | Validate references | Remove corrupted sections |
-| **Distraction** | AI ignores training, repeats spec | Track spec token count | Split at 3,500 tokens |
-| **Confusion** | AI uses irrelevant context | Identify superfluous sections | Compact/remove noise |
-| **Clash** | AI contradicts itself | Detect conflicting statements | Resolve or isolate |
-
-### Commands Preview
-
-```bash
-# Analyze spec (returns JSON for AI agent)
-lean-spec analyze <spec> --json
-
-# Transform specs (AI agent provides parameters)
-lean-spec split <spec> --output=FILE:LINES [--output=...]
-lean-spec compact <spec> --remove=LINES [--remove=...]
-lean-spec compress <spec> --replace=LINES:TEXT
-lean-spec isolate <spec> --lines=RANGE --to=NEW_SPEC
-
-# Utilities
-lean-spec diff <spec> --before-after
-lean-spec preview <spec> --split=FILE:LINES
-lean-spec rollback <spec>
-```
-
-## Status
-
-**Current Phase**: 📋 Planning & Design
-
-**Next Steps**:
-1. Complete sub-spec documentation
-2. Review with team
-3. Begin implementation (Phase 1: Parser)
-
-## Key Principles
-
-### Why AI Agent Orchestration Works
-
-**AI Agent Strengths** (provide intelligence):
-- Understanding spec content
-- Detecting issues (oversized, redundant, contradictory)
-- Deciding transformation strategy
-- Determining split points, what to remove
-- Reviewing and verifying results
-
-**Tool Strengths** (provide execution):
-- Fast file operations
-- Deterministic behavior
-- No hallucinations
-- Syntax validation
-- Reference updating
-
-**Clean Separation**:
-```
-AI Agent: "Split this 4,800 token spec at lines 1-150, 151-528, 529-710"
-  ↓
-Tool: [mechanically extracts line ranges, creates files, validates]
-  ↓
-AI Agent: "Verify: all files under 2,000 tokens" → ✓
-```
-
-**Why This is Better**:
-- ✅ AI agents already have context (no need to re-analyze in tool)
-- ✅ Tools are simple and fast (no LLM calls)
-- ✅ Deterministic (same params = same result)
-- ✅ Testable (no AI unpredictability)
-
-### Context Engineering as First Principle
-
-This builds on **Context Economy** (Principle #1 from spec 049):
-- Specs must fit in working memory
-- <2,000 tokens excellent, >3,500 tokens warning, >5,000 tokens should split
-- But splitting shouldn't require 10 minutes of LLM text generation
-
-**Evolution**:
-```
-v0.1.0: Manual spec writing
-v0.2.0: Detection + warnings (lean-spec validate)
-v0.3.0: Programmatic transformation (this spec)
-v0.4.0: Continuous context management (auto-compaction, etc.)
-```
-
-## Plan
-
-### Phase 1: Foundation (Week 1) ✅ COMPLETE
-- [x] Markdown AST parser (unified.js ecosystem)
-- [x] Spec structure analyzer
-- [x] Boundary detection algorithms
-- [x] Core data structures
-
-### Phase 2: Analysis Tools (Week 2) ✅ COMPLETE
-- [x] `lean-spec analyze --complexity`
-- [x] `lean-spec analyze --json` (for AI agents)
-- [x] Visual reports
-
-### Phase 3: Transformation Engine (Week 3) ✅ COMPLETE
-- [x] `lean-spec split` - Partition specs into sub-specs
-- [x] `lean-spec compact` - Remove redundancy
-- [x] `lean-spec compress` - Replace with summaries
-- [x] `lean-spec isolate` - Move to new spec
-
-### Phase 4: Testing & Launch (Week 4) ✅ COMPLETE
-- [x] Test all commands
-- [x] Add comprehensive test coverage
-- [x] CLI integration and polish
-- [x] Documentation and help text
-
-**Implementation Status**: All 5 transformation commands are now available in v0.2.2+
-
-## Usage Examples
-
-### Analyze Spec Complexity
-
-```bash
-# Get structured analysis (JSON output for AI agents)
-lean-spec analyze 059 --json
-
-# Human-readable output with recommendations
-lean-spec analyze 045 --verbose
-```
-
-### Split Spec into Sub-Specs
-
-```bash
-# Split by explicit line ranges (AI agent provides ranges)
-lean-spec split 045 \
-  --output=README.md:1-150 \
-  --output=DESIGN.md:151-528 \
-  --output=TESTING.md:529-710 \
-  --update-refs
-
-# Preview before applying
-lean-spec split 045 --output=README.md:1-150 --dry-run
-```
-
-### Compact Redundant Content
-
-```bash
-# Remove specified line ranges (AI agent identifies redundancy)
-lean-spec compact 045 \
-  --remove=145-153 \
-  --remove=234-256 \
-  --remove=401-415
-
-# Preview what would be removed
-lean-spec compact 045 --remove=145-153 --dry-run
-```
-
-### Compress with Summaries
-
-```bash
-# Replace verbose sections with AI-provided summaries
-lean-spec compress 043 \
-  --replace='142-284:## ✅ Phase 1: Completed
-
-Established first principles. See: specs/049/'
-
-# Preview compression
-lean-spec compress 043 --replace='142-284:Summary here' --dry-run
-```
-
-### Isolate Content to New Spec
-
-```bash
-# Move independent sections to separate specs
-lean-spec isolate 045 \
-  --lines=401-542 \
-  --to=060-velocity-algorithm \
-  --add-reference
-
-# Preview isolation
-lean-spec isolate 045 --lines=401-542 --to=060-new-spec --dry-run
-```
-
-For detailed command documentation, see [COMMANDS.md](./COMMANDS.md).
-
-## Test
-
-### Validation Criteria
-
-**Performance**:
-- [ ] Split 4,800-token spec in <1 second (vs 10+ minutes manual)
-- [ ] Parse/analyze 100 specs in <2 seconds
-- [ ] Zero text corruption (programmatic = deterministic)
-
-**Correctness**:
-- [ ] Preserves all content (no information loss)
-- [ ] Maintains markdown validity
-- [ ] Updates all cross-references correctly
-- [ ] Frontmatter remains valid
-
-**Usability**:
-- [ ] Clear analysis reports
-- [ ] Interactive preview before applying
-- [ ] Undo/rollback capability
-- [ ] Helpful error messages
-
-### Test Approach
-
-**Golden Tests**:
-- Snapshot known-good transformations
-- Regression testing against corpus
-- Compare manual vs programmatic splits
-
-**Dogfooding**:
-- Use tools on our own oversized specs
-- Validate against specs 045, 046, 048 splits
-- Measure time savings vs manual approach
-
-**Edge Cases**:
-- Specs with complex nested structures
-- Specs with many code blocks
-- Specs with tables and diagrams
-- Specs with cross-references
-
-## Success Metrics
-
-### Quantitative
-
-**Speed**:
-- 100x faster than LLM text generation
-- <1s to split any spec <8,000 tokens
-- <2s to analyze entire project
-
-**Quality**:
-- Zero corruption incidents
-- 100% markdown validity preserved
-- 100% frontmatter validity preserved
-- 100% cross-references updated
-
-### Qualitative
-
-**Developer Experience**:
-- "Splitting specs is now instant"
-- "No more babysitting AI rewrites"
-- "Confident transformations won't corrupt"
-- "Can experiment with splits freely"
-
-**Impact**:
-- Enables proactive splitting at 3,500 tokens (warning threshold)
-- Removes friction from Context Economy
-- Makes LeanSpec principles easier to follow
-- Dogfooding our own methodology effectively
-
-## Notes
-
-### Research Synthesis
-
-The external references identified four key insights:
-
-1. **Context is Finite** (Anthropic): Even 1M token windows experience "context rot"—attention degrades with length
-2. **Four Strategies** (LangChain): Write, Select, Compress, Isolate for managing context
-3. **Four Failure Modes** (Breunig): Poisoning, Distraction, Confusion, Clash
-4. **Hybrid Approach**: AI for strategy, code for execution
-
-### Why This Matters
-
-**For LeanSpec**:
-- ✅ Practices our own principles (Context Economy)
-- ✅ Removes major pain point (slow manual splitting)
-- ✅ Enables proactive management (split at 300, not 600)
-- ✅ Makes AI agents more effective (faster, fewer errors)
-
-**For Users**:
-- ✅ Faster workflow (seconds vs minutes)
-- ✅ Higher confidence (deterministic transforms)
-- ✅ Better specs (easy to maintain context limits)
-- ✅ Learning tool (see how specs should be structured)
-
-### Alternatives Considered
-
-**1. Pure AI Approach** (current, rejected):
-- ❌ Too slow (10+ minutes per spec)
-- ❌ Error-prone (context corruption)
-- ❌ Not deterministic (varies by run)
-
-**2. Manual Guidelines Only** (rejected):
-- ❌ Relies on discipline
-- ❌ Still slow when needed
-- ❌ No automation assistance
-
-**3. Hybrid Approach** (chosen):
-- ✅ AI suggests, code executes
-- ✅ Fast (programmatic) + smart (AI)
-- ✅ Best of both worlds
-
-### Open Questions
-
-1. **AST Library**: unified.js (remark) vs custom parser?
-   - Leaning toward unified.js (battle-tested, ecosystem)
-
-2. **LLM Integration**: When to use AI vs pure code?
-   - AI for: Suggesting concerns, reviewing results
-   - Code for: Parsing, moving content, updating refs
-
-3. **Preview UX**: How to show transformation preview?
-   - Interactive diff view? Side-by-side? Git-style?
-
-4. **Undo Mechanism**: Git commits? Custom snapshots?
-   - Probably git-based (user is already in git)
-
-## Related Specs
-
-- **[048-spec-complexity-analysis](../048-spec-complexity-analysis/)** - Identified the problem
-- **[049-leanspec-first-principles](../049-leanspec-first-principles/)** - Context Economy principle
-- **[018-spec-validation](../018-spec-validation/)** - Validation framework
-- **[012-sub-spec-files](../012-sub-spec-files/)** - Sub-spec pattern we're automating
-
----
-
-**Remember**: Context engineering isn't about bigger windows—it's about smarter curation. Programmatic tools make curation fast and reliable.