@leanspec/ui 0.2.7-dev.20251126100750 → 0.2.7

package/.next/standalone/packages/ui/dist/standalone/specs/066-context-economy-thresholds-refinement/README.md

@@ -0,0 +1,842 @@
+---
+status: complete
+created: '2025-11-11'
+tags:
+  - validation
+  - philosophy
+  - context-economy
+  - quality
+priority: high
+related:
+  - 048-spec-complexity-analysis
+  - 049-leanspec-first-principles
+  - 018-spec-validation
+  - 059-programmatic-spec-management
+  - 069-token-counting-utils
+created_at: '2025-11-11T06:58:44.846Z'
+updated_at: '2025-11-11T15:00:26.886Z'
+completed_at: '2025-11-11T15:00:26.886Z'
+completed: '2025-11-11'
+transitions:
+  - status: complete
+    at: '2025-11-11T15:00:26.886Z'
+---
+
+# Context Economy Thresholds Refinement
+
+> **Status**: ✅ Complete · **Priority**: High · **Created**: 2025-11-11 · **Tags**: validation, philosophy, context-economy, quality
+
+## Overview
+
+**The Suspicion**: Hard line thresholds (>300 warning, >400 error) may not accurately reflect spec complexity and readability.
+
+**The Investigation**: Deep dive into existing specs + research on LLM performance reveals that **structure, density, and token count matter more than raw line count**.
+
+**Key Finding**: A well-structured 394-line spec with sub-specs can be **more readable** than a dense 316-line spec with 26 code blocks. **Token count is critical** - research shows 39% performance drop in multi-turn contexts and quality degradation beyond 50K tokens.
+
+## Evidence from Current Specs
+
+### Top 4 Largest Specs (All Near Threshold)
+
+| Spec | Lines | Tokens* | Sections | Code Blocks | Lines/Section | Sub-specs | Readable? |
+|------|-------|---------|----------|-------------|---------------|-----------|-----------|
+| 059-programmatic | 394 | ~2,100 | 32 | 8 | ~12 | 6 files | ✅ Yes |
+| 049-first-principles | 374 | ~1,700 | 38 | 0 | ~9 | 5 files | ✅ Yes |
+| 051-docs-system-prompt | 339 | ~1,600 | 28 | 4 | ~12 | 0 | ✅ Yes |
+| 016-github-action | 315 | ~2,400 | 20 | 26 | ~15 | 0 | ⚠️ Dense |
+
+*Estimated tokens (code is denser: ~3 chars/token vs prose: ~4 chars/token)
+
+### Key Observations
+
+**1. Sub-specs Improve Readability**
+- Spec 059 (394 lines): Has 6 sub-spec files → README is just an overview
+- Spec 049 (374 lines): Has 5 sub-spec files → Progressive disclosure works
+- These are **easier to navigate** than 300-line single-file specs
+
+**2. Section Density Matters More Than Line Count**
+- Spec 049: 9 lines/section → Easy to scan and understand
+- Spec 016: 15 lines/section + 26 code blocks → Cognitively heavier despite fewer lines
+
+**3. Code Block Density is a Complexity Factor**
+- High code density (like spec 016 with 26 blocks) increases cognitive load
+- Code requires more attention than prose
+- Not captured by simple line counting
+
+**4. Structure Trumps Size**
+- A well-organized 400-line spec with clear sections and sub-specs
+- Is MORE readable than a poorly structured 250-line spec
+- Current validation misses this
+
+**5. Token Count Reveals True Cognitive Load**
+- Spec 016: Only 315 lines but **~2,400 tokens** (26 code blocks = dense)
+- Spec 049: 374 lines but only **~1,700 tokens** (pure prose, no code)
+- Research shows: Quality drops beyond 50K tokens, 6x cost difference between 2,000-line vs 300-line specs
+- **Token count better predicts AI performance** than line count
+
+## The Problem with Hard Thresholds
+
+### What Current Validation Checks
+
+```typescript
+// Current: Simple line counting
+if (lines > 400) return ERROR;
+if (lines > 300) return WARNING;
+```
+
+**Issues**:
+1. ❌ Doesn't account for structure (section organization)
+2. ❌ Doesn't account for density (code blocks, lists, tables)
+3. ❌ Doesn't account for sub-specs (progressive disclosure)
+4. ❌ Doesn't account for content type (prose vs. code vs. data)
+5. ❌ **Doesn't account for token count** (true cognitive load for AI)
+6. ❌ False positives: 394-line spec with 6 sub-specs → WARNING (but it's fine!)
+7. ❌ False negatives: 280-line dense spec with no structure → PASS (but it's hard to read!)
+
+### What Actually Affects Readability
+
+**Cognitive Load Factors (in priority order)**:
+
+1. **Cognitive Chunking** - Can you break it into 7±2 concepts?
+   - Well-sectioned spec with 15-30 sections: Easy to chunk
+   - Monolithic wall of text: Hard to process
+
+2. **Information Density** - How much attention does each line require?
+   - Code blocks: High cognitive load
+   - Tables: Medium load
+   - Narrative prose: Lower load
+   - Frontmatter/lists: Scannable
+
+3. **Progressive Disclosure** - Can you defer details?
+   - Spec with sub-specs: Read README for overview, dive into DESIGN.md when needed
+   - Single file: Must read everything to understand
+
+4. **Signal-to-Noise** - How much is fluff vs. decision-critical?
+   - High signal: Every sentence informs decisions
+   - Low signal: Obvious content, verbose explanations
+
+5. **Token Count** - True AI cognitive load (CRITICAL)
+   - Research: 39% performance drop in multi-turn contexts
+   - Quality degrades beyond 50K tokens despite 200K limits
+   - 6x cost difference: 2,000 lines vs 300 lines
+   - Code is denser (~3 chars/token) than prose (~4 chars/token)
+   - **Better predictor of AI effectiveness than line count**
+
+6. **Total Length** - Raw line count (legacy metric)
+   - Yes, it matters, but LESS than the above factors
+   - A necessary but not sufficient condition
+   - Proxy for token count but less accurate
+
+## Proposed Refined Approach
+
+### Multi-Dimensional Complexity Score
+
+Instead of just line count, calculate **Cognitive Load Score**:
+
+```typescript
+type ComplexityMetrics = {
+  lineCount: number;
+  sectionCount: number;
+  codeBlockCount: number;
+  codeBlockChars: number;  // Total characters in code blocks
+  listItemCount: number;
+  tableCount: number;
+  tableChars: number;      // Total characters in tables
+  hasSubSpecs: boolean;
+  subSpecCount: number;
+  averageSectionLength: number;
+  estimatedTokens: number; // Estimated token count for LLM input
+  estimatedReadingTime: number; // minutes
+};
+
+type ComplexityScore = {
+  score: number; // 0-100
+  factors: {
+    tokens: number;      // Primary: token-based score (0-60)
+    structure: number;   // Modifier: structure quality (-30 to +20)
+  };
+  recommendation: 'good' | 'review' | 'split';
+  costMultiplier: number; // vs 300-line baseline
+  aiEffectiveness: number; // 0-100% (hypothesis to validate)
+};
+```
+
+### Scoring Algorithm (Draft)
+
+```typescript
+// Token estimation using tokenx package (https://www.npmjs.com/package/tokenx)
+// 94% accuracy, 2kB size, zero dependencies
+import { estimateTokenCount } from 'tokenx';
+
+function calculateComplexityScore(metrics: ComplexityMetrics): ComplexityScore {
+  // PRIMARY: Token count (research-backed predictor of AI performance)
+  // Thresholds to be validated empirically - these are hypotheses
+  const tokenScore = 
+    metrics.estimatedTokens < 2000 ? 0 :   // Excellent
+    metrics.estimatedTokens < 3500 ? 20 :  // Good
+    metrics.estimatedTokens < 5000 ? 40 :  // Warning
+    60;                                     // Should split
+  
+  // MODIFIERS: Structure quality adjusts token-based score
+  // Sub-specs enable progressive disclosure (big win for Context Economy)
+  // Good sectioning enables cognitive chunking (7±2 rule)
+  const structureModifier = 
+    metrics.hasSubSpecs ? -30 :                           // Progressive disclosure bonus
+    (metrics.sectionCount >= 15 && metrics.sectionCount <= 35) ? -15 : // Good chunking
+    (metrics.sectionCount < 8) ? +20 :                    // Too monolithic
+    0;                                                     // Acceptable
+  
+  const finalScore = Math.max(0, Math.min(100, tokenScore + structureModifier));
+  
+  // Calculate cost multiplier (vs 300-line baseline ≈ 1,200 tokens)
+  const baselineTokens = 1200;
+  const costMultiplier = metrics.estimatedTokens / baselineTokens;
+  
+  // AI effectiveness estimate (to be validated empirically)
+  // Research suggests degradation, but exact thresholds need testing
+  let aiEffectiveness = 100;
+  if (metrics.estimatedTokens > 10000) {
+    aiEffectiveness = 50; // Severe degradation (hypothesis)
+  } else if (metrics.estimatedTokens > 5000) {
+    aiEffectiveness = 65; // Significant degradation (hypothesis)
+  } else if (metrics.estimatedTokens > 3500) {
+    aiEffectiveness = 80; // Noticeable degradation (hypothesis)
+  } else if (metrics.estimatedTokens > 2000) {
+    aiEffectiveness = 90; // Slight degradation (hypothesis)
+  }
+  
+  return {
+    score: finalScore,
+    factors: {
+      tokens: tokenScore,        // Primary factor
+      structure: structureModifier, // Modifier
+    },
+    recommendation: 
+      finalScore <= 25 ? 'good' :
+      finalScore <= 50 ? 'review' :
+      'split',
+    costMultiplier: Math.round(costMultiplier * 10) / 10,
+    aiEffectiveness: Math.round(aiEffectiveness),
+  };
+}
+```
+
+### New Thresholds
+
+Instead of hard line limits:
+
+- **Score 0-30**: ✅ Good - Readable and well-structured
+- **Score 31-60**: ⚠️ Review - Consider simplification or splitting
+- **Score 61-100**: 🔴 Split - Too complex, should split
+
+### Examples Applied to Current Specs
+
+**Spec 059 (394 lines, ~2,100 tokens, 32 sections, 8 code blocks, 6 sub-specs)**:
+- Token score: 20 (~2,100 tokens)
+- Structure modifier: -30 (has sub-specs)
+- **Total: -10 points** → ✅ Excellent | Cost: 1.8x | AI: 90%
+
+**Spec 016 (315 lines, ~2,400 tokens, 20 sections, 26 code blocks, no sub-specs)**:
+- Token score: 20 (~2,400 tokens)
+- Structure modifier: -15 (20 sections, good chunking)
+- **Total: 5 points** → ✅ Good | Cost: 2.0x | AI: 90%
+- **Key insight**: Token count captures code density automatically
+
+**Spec 051 (339 lines, ~1,600 tokens, 28 sections, 4 code blocks, no sub-specs)**:
+- Token score: 0 (~1,600 tokens)
+- Structure modifier: -15 (28 sections, good chunking)
+- **Total: -15 points** → ✅ Excellent | Cost: 1.3x | AI: 100%
+
+**Spec 049 (374 lines, ~1,700 tokens, 38 sections, 0 code blocks, 5 sub-specs)**:
+- Token score: 0 (~1,700 tokens)
+- Structure modifier: -30 (has sub-specs, 38 sections)
+- **Total: -30 points** → ✅ Excellent | Cost: 1.4x | AI: 100%
+- **Key insight**: Sub-specs + good structure = optimal
+
+**Hypothetical: 280 lines, ~1,400 tokens, 5 sections, no code blocks, no sub-specs**:
+- Token score: 0 (~1,400 tokens)
+- Structure modifier: +20 (only 5 sections, poor chunking)
+- **Total: 20 points** → ✅ Good | Cost: 1.2x | AI: 100%
+- **Key insight**: Short with poor structure still acceptable (tokens dominate)
+
+## Validation Changes Needed
+
+### Phase 1: Add Complexity Metrics (v0.3.0)
+
+Enhance validation to collect:
+```typescript
+interface SpecComplexity {
+  lineCount: number;
+  sectionCount: number;
+  codeBlockCount: number;
+  listItemCount: number;
+  tableCount: number;
+  subSpecFiles: string[];
+  averageSectionLength: number;
+  estimatedReadingTime: number;
+}
+```
+
+### Phase 2: Implement Complexity Scoring (v0.3.0)
+
+Add new validator:
+```typescript
+class ComplexityScoreValidator implements ValidationRule {
+  name = 'complexity-score';
+  description = 'Multi-dimensional complexity analysis';
+  
+  validate(spec: SpecInfo, content: string): ValidationResult {
+    const metrics = analyzeComplexity(spec, content);
+    const score = calculateComplexityScore(metrics);
+    
+    if (score.recommendation === 'split') {
+      return {
+        passed: false,
+        errors: [{
+          message: `Spec complexity too high (score: ${score.score}/100)`,
+          suggestion: `Consider splitting. Main issues: ${identifyTopIssues(score.factors)}`,
+        }],
+      };
+    }
+    
+    if (score.recommendation === 'review') {
+      return {
+        passed: true,
+        warnings: [{
+          message: `Spec complexity moderate (score: ${score.score}/100)`,
+          suggestion: `Consider: ${suggestImprovements(metrics, score.factors)}`,
+        }],
+      };
+    }
+    
+    return { passed: true, errors: [], warnings: [] };
+  }
+}
+```
+
+### Phase 3: Keep Line Count as Backstop (v0.3.0)
+
+Don't remove line count validation entirely - use it as a **backstop**:
+
+```typescript
+// Complexity score is primary
+// Line count is secondary safety net
+
+if (complexityScore < 60 && lineCount < 500) {
+  // Good - pass both checks
+} else if (complexityScore < 60 && lineCount >= 500) {
+  // Warning - good structure but very long
+  warning("Well-structured but consider splitting for Context Economy");
+} else if (complexityScore >= 60 && lineCount < 400) {
+  // Error - complex despite being shorter
+  error("Poor structure or high density - needs refactoring");
+} else {
+  // Error - both metrics problematic
+  error("Too complex - split into sub-specs");
+}
+```
+
+### Phase 4: Educate Users (v0.3.0)
+
+Update guidance:
+- AGENTS.md: Explain complexity factors beyond line count
+- README.md: Show examples of good vs. poor structure
+- Validation output: Explain WHY a spec is complex
+- CLI: Add `lean-spec complexity <spec>` command for detailed analysis
+
+## Research Evidence
+
+### 1. Token Count is Critical
+
+**Source**: [AI Agent Performance Blog Post](https://www.lean-spec.dev/blog/ai-agent-performance)
+
+- **Finding**: 2,000-line spec costs **6x more** than 300-line spec
+- **Finding**: Quality degradation happens **even within context limits** (not just at 50K)
+- **Key Quote**: "Quality drops beyond 50K tokens despite 200K limits" - but degradation **starts much earlier**
+- **Why**: Attention dilution (N² complexity), context rot, option overload, premature convergence
+
+### 2. Multi-Turn Performance Degradation
+
+**Source**: [arXiv:2505.06120 - "LLMs Get Lost In Multi-Turn Conversation"](https://arxiv.org/abs/2505.06120)
+
+- **Finding**: **39% average performance drop** across six generation tasks
+- **Root Cause**: LLMs make premature assumptions and can't recover
+- **Key Quote**: "When LLMs take a wrong turn, they get lost and do not recover"
+
+### 3. Function-Calling Performance
+
+**Source**: [Berkeley Function-Calling Leaderboard (BFCL)](https://gorilla.cs.berkeley.edu/leaderboard.html)
+
+- **Finding**: ALL models perform worse with more tools/options
+- **Implication**: More context = more confusion = lower accuracy
+
+### 4. Information Density Matters
+
+**Source**: [arXiv:2407.11963 - "NeedleBench"](https://arxiv.org/abs/2407.11963)
+
+- **Finding**: Models struggle with information-dense scenarios even at shorter context lengths
+- **Phenomenon**: "Under-thinking" - premature reasoning termination
+
+### 5. Long-Context RAG Performance
+
+**Source**: [Databricks Research](https://www.databricks.com/blog/long-context-rag-performance-llms)
+
+- **Finding**: Long-context performance degrades significantly even within theoretical limits
+- **Implication**: Smaller models degrade earlier
+
+### Key Takeaway
+
+**Token count is a better predictor of AI performance than line count** because:
+1. Direct measure of LLM input cost
+2. Accounts for content density (code vs prose)
+3. Backed by research showing **non-linear degradation patterns**
+4. Correlates with actual AI effectiveness
+
+**Degradation Gradient** (based on research):
+- **<2K tokens**: Baseline performance (~100% effectiveness)
+- **2-5K tokens**: Early degradation begins (~85-95% effectiveness)
+- **5-10K tokens**: Noticeable degradation (~65-85% effectiveness)
+- **10-20K tokens**: Moderate degradation (~50-65% effectiveness)  
+- **50K+ tokens**: Severe "cliff" effect (~40% performance drop or worse)
+
+For validation, we use conservative thresholds (5K tokens = severe penalty) to catch specs **before** they reach problematic sizes.
+
+## Implementation Plan
+
+### Phase 1: Research & Metrics ✅ (This spec)
+- [x] Investigate current specs
+- [x] Identify complexity factors
+- [x] Propose scoring algorithm
+- [x] Design empirical validation plan
+- [x] Get feedback on approach
+
+### Phase 2: Core Implementation (v0.3.0 - Next)
+- [ ] Install `tokenx` for token estimation
+- [ ] Implement simplified `calculateComplexityScore()` function
+- [ ] Use **hypothesis thresholds** (2K/3.5K/5K tokens) initially
+- [ ] Create `ComplexityScoreValidator` class
+- [ ] Add tests for edge cases
+- [ ] Integrate with existing validation framework
+
+### Phase 3: CLI Integration (v0.3.0)
+- [ ] Add `lean-spec complexity <spec>` command
+- [ ] Show breakdown: token score, structure modifier
+- [ ] Display cost multiplier and AI effectiveness estimates
+- [ ] Provide actionable suggestions
+- [ ] Update `lean-spec validate` output
+
+### Phase 4: Documentation (v0.3.0)
+- [ ] Update AGENTS.md with complexity guidance
+- [ ] Update README.md with examples
+- [ ] Create "good structure" showcase
+- [ ] Document that thresholds are hypotheses pending validation
+
+### Phase 5: Empirical Validation (Future - v0.4.0+)
+**Deferred until we have:**
+- More real-world usage data from v0.3.0
+- Clear methodology for LLM integration
+- Resources for comprehensive benchmarking
+
+**Tasks when ready:**
+- [ ] Implement benchmark framework (already stubbed in `src/benchmark/`)
+- [ ] Define benchmark tasks for 10+ specs
+- [ ] Run benchmarks to validate thresholds
+- [ ] Refine scoring weights based on data
+- [ ] Publish empirical findings
+
+### Phase 6: Advanced Features (v0.4.0+)
+- [ ] Complexity trends over time
+- [ ] Project-wide complexity dashboard
+- [ ] Automated splitting suggestions
+- [ ] Model-specific thresholds (if empirical data shows need)
+
+## Implementation Details
+
+### Token Estimation: Two Options
+
+We have two viable approaches for token counting, each with different tradeoffs:
+
+#### Option 1: tokenx (Recommended for Validation)
+
+**Fast, lightweight estimation** - Best for validation thresholds where perfect accuracy isn't critical.
+
+**Pros:**
+- ✅ 94% accuracy compared to full tokenizers
+- ✅ Just **2kB** bundle size with **zero dependencies**
+- ✅ Very fast - no tokenization overhead
+- ✅ Multi-language support (English, German, French, Chinese, etc.)
+- ✅ Good enough for validation warnings/errors
+- ✅ 45K+ weekly downloads
+
+**Cons:**
+- ❌ Not 100% accurate (6-12% error margin)
+- ❌ Estimation-based, not true BPE encoding
+
+**Installation:**
+```bash
+npm install tokenx
+```
+
+**Usage:**
+```typescript
+import { estimateTokenCount, isWithinTokenLimit } from 'tokenx';
+
+// Fast estimation for validation
+const tokens = estimateTokenCount(specContent);
+
+// Check if within limit (e.g., 5000 token warning threshold)
+const needsReview = !isWithinTokenLimit(specContent, 5000);
+```
+
+**Accuracy benchmarks:**
+- English prose: 10-12% error margin
+- Code (TypeScript): 6.18% error margin
+- Large text (31K tokens): 12.29% error margin
+
+---
+
+#### Option 2: gpt-tokenizer (For Exact Counts)
+
+**Precise tokenization** - Port of OpenAI's tiktoken with 100% accuracy.
+
+**Pros:**
+- ✅ **100% accurate** - exact BPE encoding
+- ✅ Supports all OpenAI models (GPT-4o, GPT-4, GPT-3.5, etc.)
+- ✅ Fastest full tokenizer on NPM (faster than WASM bindings)
+- ✅ Built-in cost estimation with `estimateCost()`
+- ✅ Chat-specific tokenization with `encodeChat()`
+- ✅ 283K+ weekly downloads, trusted by Microsoft, Elastic
+
+**Cons:**
+- ❌ **53.1 MB** unpacked size (vs 2kB for tokenx)
+- ❌ Slower than estimation (but still fastest full tokenizer)
+- ❌ Model-specific - need to import correct encoding
+
+**Installation:**
+```bash
+npm install gpt-tokenizer
+```
+
+**Usage:**
+```typescript
+import { encode, countTokens, isWithinTokenLimit } from 'gpt-tokenizer';
+// or model-specific: from 'gpt-tokenizer/model/gpt-4o'
+
+// Exact token count
+const tokens = encode(specContent);
+const count = tokens.length;
+
+// Or use helper
+const exactCount = countTokens(specContent);
+
+// Check limit with exact counting
+const needsReview = !isWithinTokenLimit(specContent, 5000);
+```
+
+**Accuracy:**
+- 100% accurate (port of OpenAI's tiktoken)
+- Benchmarked against OpenAI's Python library
+
+---
+
+### Recommendation: Hybrid Approach
+
+**For v0.3.0, use tokenx:**
+- Fast validation during CLI commands
+- 2kB size won't bloat the package
+- 94% accuracy is sufficient for warnings/errors
+- 6-12% margin is acceptable for thresholds
+
+**Future: Offer gpt-tokenizer as optional**
+- Add as peer dependency (optional install)
+- Use if available for exact counts
+- Fall back to tokenx if not installed
+- Display "estimated" vs "exact" in output
+
+**Implementation:**
+```typescript
+// Try exact tokenizer first, fall back to estimation
+let tokenCount: number;
+let isExact = false;
+
+try {
+  const { countTokens } = await import('gpt-tokenizer');
+  tokenCount = countTokens(content);
+  isExact = true;
+} catch {
+  const { estimateTokenCount } = await import('tokenx');
+  tokenCount = estimateTokenCount(content);
+  isExact = false;
+}
+
+// Display in output
+console.log(`Tokens: ${tokenCount} ${isExact ? '(exact)' : '(estimated ±6%)'}`);
+```
+
+## Empirical Validation Plan (Future Work)
+
+**Status**: Deferred to v0.4.0+ - too early for comprehensive benchmarking
+
+**The Problem**: Current thresholds (2K/3.5K/5K tokens) are hypotheses based on research, not validated on LeanSpec's actual use case.
+
+**The Challenge**: Building a proper benchmark suite requires:
+- Clear methodology for LLM integration and evaluation
+- Significant time investment for framework + test data
+- Real-world usage patterns from v0.3.0 to guide validation
+- Resources for running benchmarks across multiple models
+
+**The Pragmatic Approach**:
+1. **v0.3.0**: Ship complexity scoring with research-based thresholds (good enough to start)
+2. **Collect data**: Gather real-world usage patterns, see which specs trigger warnings
+3. **v0.4.0+**: Build benchmark framework when we have clearer requirements
+
+**Validation Framework Stub** (see `src/benchmark/` for implementation):
+- Type definitions and interfaces ready
+- Complexity analysis functions implemented
+- Benchmark task examples defined
+- LLM integration and statistical analysis deferred
+
+**When ready to validate**, the framework will answer:
+- Does token count predict performance better than line count?
+- Where does degradation actually start? (2K? 3K? 5K?)
+- How much do sub-specs improve AI comprehension?
+- What's the real cost multiplier for large specs?
+
+**For now**: Use hypothesis thresholds, document them as such, refine based on user feedback in v0.3.0.
+
+## Open Questions
+
+1. **Token Thresholds**: Are 2K/3.5K/5K correct?
+   - **Current**: Using research-based hypotheses
+   - **To validate (v0.4.0+)**: Run benchmark suite when methodology is clear
+   - **For now**: Gather user feedback in v0.3.0, adjust if obviously wrong
+
+2. **Structure Impact**: How much does it matter?
+   - **Current**: -30 bonus for sub-specs, -15 for good sectioning
+   - **To validate (v0.4.0+)**: Compare monolithic vs sub-spec variants
+   - **For now**: Based on cognitive science (7±2 chunks) and intuition
+
+3. **Section Count Sweet Spot**: Is 15-35 sections right?
+   - **Current**: Based on cognitive load theory
+   - **To validate (v0.4.0+)**: Test specs with varying section counts
+   - **For now**: Seems reasonable, may adjust based on user feedback
+
+4. **Model Differences**: Do thresholds vary by model?
+   - **Current**: Assume similar across Claude/GPT
+   - **To validate (v0.4.0+)**: Test multiple models if data shows divergence
+   - **For now**: Single set of thresholds
+
+5. **Performance**: Can we run this efficiently?
+   - **Current**: tokenx is very fast (2kB, no dependencies)
+   - **To benchmark**: Test on 100+ specs to verify <100ms per spec
+   - **For now**: Should be fine, optimize if issues arise
+
+## Success Criteria
+
+### Phase 2-4: Initial Implementation (v0.3.0)
+- ✅ Spec 059 (394 lines, 6 sub-specs) scores well (≤25 points)
+- ✅ Poorly structured specs flagged even if short
+- ✅ Users understand WHY a spec is complex (clear breakdown)
+- ✅ Validation guides toward better structure, not just length reduction
+- ✅ AI agents make informed splitting decisions based on token count + structure
+- ✅ No false negatives: Truly oversized specs (>600 lines) caught
+- ✅ Thresholds documented as hypotheses, not validated facts
+
+### Phase 5: Empirical Validation (v0.4.0+ - When Ready)
+- ✅ Token count predicts performance better than line count (R² > 0.7 vs < 0.5)
+- ✅ Degradation thresholds validated within ±500 tokens of hypothesis
+- ✅ Sub-specs show measurable quality improvement (>5% accuracy)
+- ✅ Cost multiplier validated against actual API usage
+- ✅ Multi-turn degradation measured and documented
+
+### User Experience (All Phases)
+- ✅ Complexity scores align with user intuition
+- ✅ Suggestions are actionable and specific
+- ✅ AGENTS.md reflects current best practices (hypothesis-based for v0.3.0)
+
+## Related Specs
+
+- **[048-spec-complexity-analysis](../048-spec-complexity-analysis/)** - Identified line count thresholds
+- **[049-leanspec-first-principles](../049-leanspec-first-principles/)** - Context Economy principle
+- **[059-programmatic-spec-management](../059-programmatic-spec-management/)** - Context engineering and programmatic analysis
+- **[018-spec-validation](../018-spec-validation/)** - Current validation framework
+
+## Notes
+
+### Why This Matters
+
+**Current Problem**: False positives and false negatives
+- We're warning about well-structured 394-line specs (false positive)
+- We're missing dense 280-line specs with poor structure (false negative)
+
+**Impact**:
+- Users may ignore warnings if they seem arbitrary
+- AI agents get confused about when to split
+- We're not measuring what we actually care about (readability, not just length)
+
+**Solution**: Measure complexity more holistically
+- Line count remains important but not sufficient
+- Structure, density, and progressive disclosure matter
+- Give users actionable feedback
+
+### The Meta-Learning
+
+This spec itself demonstrates the principle:
+- 410 lines, ~2,200 tokens (includes code examples)
+- Well-structured with clear sections (28 sections)
+- Each section is scannable and focused
+- Tables and lists make information easy to parse
+- References research with clear citations
+
+**Applying the simplified scoring to this spec:**
+- Token score: 20 (~2,200 tokens, in good range)
+- Structure modifier: -15 (28 sections, good chunking, no sub-specs)
+- **Total: 5 points** → ✅ Good | Cost: 1.8x | AI: 90%
+
+**Insight**: Well-structured with clear sections. Could benefit from sub-specs for benchmark details (would get -30 modifier).
+
+Using old rules: "🔴 Error: 410/400 lines - must split!"
+Using new rules: "✅ Good: Score 5/100 - well-structured, token count acceptable, consider sub-specs to reach 'excellent'"
+
+---
+
+## Implementation Status (2025-11-11)
+
+### ✅ What's Done
+
+**Core Implementation Exists**:
+- `ComplexityValidator` class implemented in `packages/core/src/validators/complexity.ts`
+- Token estimation using `tokenx` package (installed and working)
+- Multi-dimensional scoring algorithm implemented
+- Registered in validation pipeline (`packages/cli/src/commands/validate.ts`)
+- All token thresholds and structure modifiers coded as designed
+
+**Build Status**:
+- `@leanspec/core` package builds successfully
+- `@leanspec/cli` package builds successfully
+- `tokenx` dependency properly installed
+
+### 🐛 Issues Found
+
+**1. Sub-Spec Detection Bug (Critical)**
+- **Problem**: `hasSubSpecs` detected by text pattern matching, not actual file existence
+- **Current Code**: `/\b(DESIGN|IMPLEMENTATION|TESTING|CONFIGURATION|API|MIGRATION)\.md\b/.test(content)`
+- **Issue**: Spec 066 mentions "DESIGN.md" in documentation → gets -30 bonus it doesn't deserve
+- **Impact**: False negatives - specs get structure bonuses for merely documenting sub-specs
+
+**2. Silent Warning Issue**
+- **Problem**: Complexity validator produces no output even when it should warn
+- **Actual Score for Spec 066**:
+  - 706 lines
+  - **7,307 tokens** (very high, >5000 threshold)
+  - 45 sections
+  - Token score: 60
+  - Structure modifier: -30 (false positive due to bug #1)
+  - Final score: 30 → "review" recommendation (should show warning)
+- **Expected**: Warning message displayed
+- **Actual**: No output from complexity validator
+- **Hypothesis**: Warning not being formatted/displayed (need to debug formatter or result handling)
+
+**3. Old Line Count Validator Still Active**
+- **Current Behavior**: Shows "Error: Spec exceeds 400 lines (706 lines)"
+- **Expected**: Complexity validator should be primary, line count as backstop
+- **Decision Needed**: Should old validator be:
+  - Disabled entirely?
+  - Adjusted to only warn at 500+ lines?
+  - Kept as-is for redundancy?
+
+### 🔧 Fixes Needed for Next Session
+
+**Priority 1: Fix Sub-Spec Detection**
+```typescript
+// Current (WRONG):
+const hasSubSpecs = /\b(DESIGN|IMPLEMENTATION|TESTING|CONFIGURATION|API|MIGRATION)\.md\b/.test(content);
+
+// Should be (need to check actual files):
+// Option A: Pass file list to validator
+const hasSubSpecs = subSpecFiles.length > 0;
+
+// Option B: Check spec directory for .md files (requires fs access)
+const files = await fs.readdir(path.dirname(spec.filePath));
+const mdFiles = files.filter(f => f.endsWith('.md') && f !== 'README.md');
+const hasSubSpecs = mdFiles.length > 0;
+```
+
+**Location**: `packages/core/src/validators/complexity.ts`, line ~160
+
+**Priority 2: Debug Silent Warning**
+- Add logging to see if validator is running and producing results
+- Check if `ValidationResult` with warnings is being filtered out
+- Verify formatter (`validate-formatter.ts`) handles complexity validator output
+- Test with simpler spec to isolate issue
+
+**Priority 3: Coordinate Line Count Validator**
+- Decide on line count validator role (keep, adjust, or remove)
+- Update thresholds if keeping (suggest 500/600 instead of 300/400)
+- Document relationship between validators in code comments
+
+### 📊 Test Cases for Validation
+
+**Test with Spec 066** (this spec):
+- Expected: Score 60 (no sub-specs) → "split" recommendation → ERROR
+- Currently: Score 30 (false bonus) → "review" → WARNING (but silent)
+
+**Test with Spec 049** (has 5 sub-specs):
+- 374 lines, ~1,700 tokens, 38 sections, 5 sub-spec files
+- Expected: Score -30 → "excellent" → PASS
+- Should verify this works correctly
+
+**Test with Spec 059** (has 6 sub-specs):
+- 394 lines, ~2,100 tokens, 32 sections, 6 sub-spec files
+- Expected: Score -10 → "excellent" → PASS
+- Should verify this works correctly
+
+### 🎯 Next Steps
+
+1. **Fix sub-spec detection** (30 min):
+   - Modify `analyzeComplexity()` to check actual files
+   - May need to pass spec path or file list to validator
+   - Update tests to verify correct detection
+
+2. **Debug warning output** (20 min):
+   - Add temporary console.log in validator
+   - Rebuild and test
+   - Check if result is being produced but not displayed
+
+3. **Verify full pipeline** (10 min):
+   - Run validation on multiple specs
+   - Confirm token counts and scores match expectations
+   - Validate formatter displays all validator results
+
+4. **Update line count validator** (10 min):
+   - Adjust thresholds to 500/600 or disable
+   - Update messages to reference complexity validator
+   - Document as backstop in comments
+
+5. **Test and document** (20 min):
+   - Validate spec 049, 059, 066 with corrected logic
+   - Update AGENTS.md with complexity guidance
+   - Mark spec 066 as fully implemented
+
+**Estimated Total**: ~90 minutes of focused work
+
+### 💡 Design Questions to Resolve
+
+1. **File Detection Approach**: Should we:
+   - A) Pass file list to validator (cleaner, requires API change)
+   - B) Let validator read directory (simpler, but core needs fs access)
+   - C) Pre-compute in CLI and pass as metadata (best separation of concerns)
+
+2. **Line Count Validator**: Should we:
+   - A) Remove it (complexity validator handles everything)
+   - B) Keep with raised thresholds (500/600) as backstop
+   - C) Keep current thresholds for redundancy
+
+3. **Sections Outside 15-35 Range**: Currently gives 0 modifier. Should we:
+   - A) Keep as-is (only penalize <8 sections)
+   - B) Penalize >35 sections (too fragmented)
+   - C) Use sliding scale instead of fixed ranges
+
+---
+
+**Status**: Implementation exists but has bugs. Ready for debugging and fixing in next session.