@leanspec/ui 0.2.5 → 0.2.6-dev.20251125010539

package/.next/standalone/packages/ui/dist/standalone/specs/071-simplified-token-validation/README.md

@@ -1,489 +0,0 @@
----
-status: complete
-created: '2025-11-13'
-tags:
-  - validation
-  - simplification
-  - tokens
-priority: high
-assignee: marvin
-created_at: '2025-11-13T03:11:29.739Z'
-related:
-  - 066-context-economy-thresholds-refinement
-  - 069-token-counting-utils
-updated_at: '2025-11-13T03:54:24.312Z'
-completed_at: '2025-11-13T03:54:24.312Z'
-completed: '2025-11-13'
-transitions:
-  - status: complete
-    at: '2025-11-13T03:54:24.312Z'
----
-
-# Simplified Token-Based Validation
-
-> **Status**: ✅ Complete · **Priority**: High · **Created**: 2025-11-13 · **Tags**: validation, simplification, tokens
-> **Assignee**: marvin · **Reviewer**: TBD
-
-## Overview
-
-**The Problem**: Current complexity validation uses a score-based approach (0-100) that:
-- Uses arbitrary weight ratios (token score 0-60, structure modifier -30 to +20)
-- Produces confusing derived scores (what does "45/100" mean?)
-- Hides actual metrics users care about (5,207 tokens becomes "score 60")
-- Can give misleading results (>5K tokens with sub-specs = "good" due to -30 modifier)
-
-**The Solution**: Use **direct token thresholds** with clear, independent checks for each factor.
-
-**Why Now**: We just removed line-count validator in favor of token-based validation, but the implementation is overly complex and confusing.
-
-## Design
-
-### Core Principle: Direct, Independent Checks
-
-Instead of calculating a composite score, evaluate each factor independently and report clearly.
-
-### Token Thresholds (Primary Check)
-
-```typescript
-interface TokenValidation {
-  tokens: number;
-  level: 'excellent' | 'good' | 'warning' | 'error';
-  message: string;
-}
-
-function validateTokens(tokens: number): TokenValidation {
-  if (tokens > 5000) {
-    return {
-      tokens,
-      level: 'error',
-      message: `Spec has ${tokens} tokens (threshold: 5,000) - should split for Context Economy`
-    };
-  }
-  
-  if (tokens > 3500) {
-    return {
-      tokens,
-      level: 'warning',
-      message: `Spec has ${tokens} tokens (threshold: 3,500) - consider simplification`
-    };
-  }
-  
-  if (tokens > 2000) {
-    return {
-      tokens,
-      level: 'info',
-      message: `Spec has ${tokens} tokens - acceptable, watch for growth`
-    };
-  }
-  
-  return {
-    tokens,
-    level: 'excellent',
-    message: `Spec has ${tokens} tokens - excellent`
-  };
-}
-```
-
-### Structure Checks (Independent Feedback)
-
-Each structural issue gets its own clear message:
-
-```typescript
-interface StructureCheck {
-  passed: boolean;
-  message?: string;
-  suggestion?: string;
-}
-
-function checkStructure(metrics: Metrics): StructureCheck[] {
-  const checks: StructureCheck[] = [];
-  
-  // Sub-specs presence (positive feedback)
-  if (metrics.hasSubSpecs) {
-    checks.push({
-      passed: true,
-      message: `✓ Uses ${metrics.subSpecCount} sub-spec files for progressive disclosure`
-    });
-  } else if (metrics.tokens > 3000) {
-    checks.push({
-      passed: false,
-      message: `Consider using sub-spec files (DESIGN.md, IMPLEMENTATION.md)`,
-      suggestion: `Progressive disclosure reduces cognitive load for large specs`
-    });
-  }
-  
-  // Section organization
-  if (metrics.sectionCount >= 15 && metrics.sectionCount <= 35) {
-    checks.push({
-      passed: true,
-      message: `✓ Good sectioning (${metrics.sectionCount} sections) enables cognitive chunking`
-    });
-  } else if (metrics.sectionCount < 8) {
-    checks.push({
-      passed: false,
-      message: `Only ${metrics.sectionCount} sections - too monolithic`,
-      suggestion: `Break into 15-35 sections for better readability (7±2 cognitive chunks)`
-    });
-  }
-  
-  return checks;
-}
-```
-
-### Line Count (Backstop Only)
-
-Line count becomes a simple backstop check:
-
-```typescript
-function checkLineCount(lines: number): StructureCheck | null {
-  if (lines > 500) {
-    return {
-      passed: false,
-      message: `Spec is very long (${lines} lines)`,
-      suggestion: `Consider splitting even if token count is acceptable`
-    };
-  }
-  return null;
-}
-```
-
-### Output Format
-
-**Example 1: Clean spec (016)**
-```
-✓ 016-github-action passed
-
-Token Analysis:
-  2,004 tokens - acceptable, watch for growth
-  
-Structure:
-  ✓ Good sectioning (20 sections) enables cognitive chunking
-```
-
-**Example 2: Warning spec (049)**
-```
-⚠ 049-first-principles has warnings
-
-Token Analysis:
-  3,413 tokens (threshold: 3,500) - consider simplification
-  
-Structure:
-  ✓ Uses 5 sub-spec files for progressive disclosure
-  ✓ Good sectioning (38 sections) enables cognitive chunking
-```
-
-**Example 3: Error spec (066)**
-```
-✖ 066-context-economy-thresholds-refinement failed
-
-Token Analysis:
-  8,073 tokens (threshold: 5,000) - should split for Context Economy
-  
-Structure:
-  Consider using sub-spec files (DESIGN.md, IMPLEMENTATION.md)
-    → Progressive disclosure reduces cognitive load for large specs
-    
-Line Count:
-  Spec is very long (843 lines)
-    → Consider splitting even if token count is acceptable
-```
-
-### Key Improvements
-
-1. **Clear thresholds**: Users see actual token counts and thresholds
-2. **No derived scores**: No confusing "45/100" numbers
-3. **Independent factors**: Each aspect evaluated separately
-4. **Actionable feedback**: Specific suggestions for each issue
-5. **Positive reinforcement**: Shows what's working well
-6. **Simple logic**: Easy to understand and maintain
-
-### Comparison: Before vs After
-
-**Before (Score-Based):**
-```
-✖ error: Spec complexity too high (score: 60/100, 8073 tokens) and 843 lines
-       → Token count very high - strongly consider splitting; 
-         Use sub-spec files for progressive disclosure
-```
-
-**After (Direct Thresholds):**
-```
-✖ error: Spec has 8,073 tokens (threshold: 5,000) - should split for Context Economy
-⚠ warning: Consider using sub-spec files (DESIGN.md, IMPLEMENTATION.md)
-          → Progressive disclosure reduces cognitive load for large specs
-⚠ warning: Spec is very long (843 lines)
-          → Consider splitting even if token count is acceptable
-```
-
-**Advantages:**
-- ✅ Shows actual token count prominently
-- ✅ Clear threshold (5,000)
-- ✅ Separate, specific suggestions
-- ✅ No confusing score math
-- ✅ Easy to understand what's wrong and how to fix it
-
-## Plan
-
-### Phase 1: Simplify ComplexityValidator ✅
-- [x] Remove score calculation logic
-- [x] Implement direct token threshold checks
-- [x] Implement independent structure checks
-- [x] Update error/warning messages to show actual values
-- [x] Keep line count as simple backstop
-
-### Phase 2: Update SubSpecValidator ✅
-- [x] Apply same simplification to sub-spec validation
-- [x] Remove score calculation from sub-spec checks
-- [x] Use direct token thresholds for sub-specs
-
-### Phase 3: Update Tests ✅
-- [x] Remove tests that check score values
-- [x] Add tests for direct threshold behavior
-- [x] Add tests for structure feedback messages
-- [x] Verify output format
-
-### Phase 4: Update Documentation ✅
-- [x] Update AGENTS.md to remove score references
-- [x] Update spec 066 to document simplified approach
-- [x] Update validation output examples in docs
-
-### Phase 5: Test Against Real Specs ✅
-- [x] Validate against all current specs
-- [x] Verify messages are clear and actionable
-- [x] Confirm no false positives/negatives
-
-## Test
-
-### Validation Behavior Tests
-
-**Test Case 1: Excellent spec (<2K tokens)**
-- Input: Spec with 1,500 tokens, 15 sections, 250 lines
-- Expected: ✓ Pass with positive feedback
-- Output: Shows token count, notes good sectioning
-
-**Test Case 2: Good spec (2-3.5K tokens)**
-- Input: Spec with 2,800 tokens, 20 sections, 350 lines
-- Expected: ✓ Pass with info message
-- Output: "acceptable, watch for growth"
-
-**Test Case 3: Warning spec (3.5-5K tokens)**
-- Input: Spec with 4,200 tokens, 18 sections, 450 lines
-- Expected: ⚠ Warning
-- Output: Shows exact token count and threshold (3,500)
-
-**Test Case 4: Error spec (>5K tokens)**
-- Input: Spec with 8,073 tokens, 45 sections, 843 lines
-- Expected: ✖ Error
-- Output: Shows exact token count and threshold (5,000)
-
-**Test Case 5: Good tokens + sub-specs**
-- Input: Spec with 3,200 tokens, has 4 sub-specs
-- Expected: ✓ Pass with positive feedback
-- Output: Notes progressive disclosure
-
-**Test Case 6: High tokens + poor structure**
-- Input: Spec with 4,500 tokens, only 5 sections
-- Expected: ⚠ Warning for tokens + structure
-- Output: Both token warning and sectioning suggestion
-
-**Test Case 7: Line count backstop**
-- Input: Spec with 2,500 tokens, 600 lines
-- Expected: ⚠ Warning about length
-- Output: Token count OK, but warns about line count
-
-### Output Format Tests
-
-- [ ] Error messages show actual token counts
-- [ ] Warnings show actual thresholds
-- [ ] Structure feedback is separate from token checks
-- [ ] Positive feedback appears for good practices
-- [ ] No score values (0-100) appear anywhere
-- [ ] Messages are actionable with specific suggestions
-
-### Real Spec Validation
-
-Test against actual specs:
-- [ ] 016 (2,004 tokens): Should pass cleanly
-- [ ] 049 (3,413 tokens): Should pass with info/positive feedback
-- [ ] 059 (3,364 tokens): Should pass with positive feedback for sub-specs
-- [ ] 066 (8,073 tokens): Should error with clear token threshold message
-- [ ] 069 (5,207 tokens): Should error with clear token threshold message
-
-## Notes
-
-### Why Simplify?
-
-**Current Problems:**
-1. **Arbitrary math**: `tokenScore (0-60) + structureModifier (-30 to +20) = finalScore` - these ratios are made up
-2. **Confusing abstraction**: Score 45/100 is meaningless to users
-3. **Misleading results**: >5K tokens with sub-specs gets -30 modifier = "good" (but it's still >5K!)
-4. **Hides real data**: Users care about "5,207 tokens" not "score 60"
-5. **Hard to maintain**: Complex score calculation logic
-
-**Simplification Benefits:**
-1. **Direct thresholds**: If tokens > 5000 → error. Simple.
-2. **Clear values**: Show actual tokens and thresholds
-3. **Independent checks**: Each factor evaluated separately
-4. **Easier to understand**: No derived scores to explain
-5. **Easier to maintain**: Straightforward if/else logic
-
-### Design Principles Applied
-
-From First Principles (spec 049):
-
-1. **Context Economy**: Token count is the direct measure of context size - don't abstract it
-2. **Signal-to-Noise**: Show actual numbers users care about, not derived scores
-3. **Intent Over Implementation**: Users want to know "how many tokens?" not "what's the complexity score?"
-
-### Research Basis & Threshold Validation
-
-**From Academic Research (2024-2025):**
-
-1. **NeedleBench (arXiv:2407.11963, Sep 2025)** - Information Density Study:
-   - Models struggle with "information-dense" scenarios where relevant information is continuously distributed
-   - "Under-thinking" phenomenon: Models prematurely conclude reasoning despite available information
-   - **Finding**: Even advanced reasoning models (Deepseek-R1, OpenAI o3) struggle with continuous retrieval
-   - **Implication**: Denser specs (more code blocks, tables) are cognitively heavier
-
-2. **Multi-Turn Degradation (arXiv:2505.06120, May 2025)** - Context Accumulation:
-   - **39% average performance drop** in multi-turn conversations vs single-turn
-   - LLMs make premature assumptions and overly rely on them
-   - **Finding**: "When LLMs take a wrong turn in a conversation, they get lost and do not recover"
-   - **Implication**: Accumulated context across turns compounds degradation
-
-**From Latest Models (Nov 2025):**
-
-**OpenAI GPT-5 & Family:**
-- **GPT-5**: Flagship model with "thinking built in" for complex tasks
-- **GPT-4o**: 128K context window standard
-- **Industry standard**: Most models now support 100K-200K tokens
-- **Key insight**: "Tasks that would typically require hours of human effort to complete may take Claude a few minutes" - latency increases with context
-
-**Anthropic Claude 4.5 & Family:**
-- **Claude Sonnet 4.5**: 200K context standard, **1M tokens (beta)**
-- **Pricing**: $3/MTok input, $15/MTok output
-- **Extended thinking**: Available for complex reasoning
-- **Finding**: "Superior instruction following, tool selection, error correction for long-running agents"
-- **Implication**: While 200K+ is possible, optimal performance is still at lower token counts
-
-**Industry Pricing Signal:**
-- Standard context: Free/low-cost tier
-- Extended context (>50K): Premium pricing
-- **Interpretation**: Providers indicate optimal usage is <50K tokens despite technical capacity
-
-**Our Threshold Analysis:**
-
-Real-world spec data from our corpus:
-- Median spec: ~2,500 tokens (~300 lines)
-- Well-structured: ~3,500 tokens (~400 lines)  
-- Large spec: ~5,000 tokens (~550 lines)
-- Very large: >8,000 tokens (~800+ lines)
-
-**Conservative Thresholds (RECOMMENDED):**
-
-Based on:
-- Multi-turn degradation research (39% drop)
-- Information density effects
-- Real-world spec corpus
-- Industry pricing signals
-
-```
-<2,000 tokens:  ✅ Excellent - Baseline performance
-                   (~1,500 words, ~250 lines)
-                   Fits comfortably with room for conversation
-
-2,000-3,500:    ✅ Good - Slight degradation acceptable
-                   (~2,500 words, ~350 lines)
-                   Well within all model capacities
-
-3,500-5,000:    ⚠️  Warning - Consider simplification
-                   (~4,000 words, ~500 lines)
-                   Approaching cognitive/attention limits
-
->5,000:         🔴 Should split - Significant performance impact
-                   (~4,000+ words, ~600+ lines)
-                   Research shows compound degradation
-```
-
-**Why These Thresholds Remain Valid (Nov 2025):**
-
-1. **Research-backed**: 39% degradation in multi-turn contexts affects all models
-2. **Information density**: Dense content (code, tables) remains cognitively heavier
-3. **Attention constraints**: Human working memory still limited to 7±2 items
-4. **Cost-effective**: Smaller prompts = faster responses + lower costs
-5. **Conservative**: Well below technical limits, optimized for quality
-
-**Validation Against Real Specs:**
-- 016 (2,004 tokens): ✅ Just above baseline - info appropriate
-- 049 (3,413 tokens): ✅ Good range - acceptable
-- 059 (3,364 tokens): ✅ Good range with sub-specs
-- 066 (8,073 tokens): 🔴 Way over 5K - must split
-- 069 (5,207 tokens): 🔴 Just over 5K - should split
-
-**Conclusion**: Despite massive increases in context window sizes (now 200K-1M tokens), our **conservative thresholds (2K/3.5K/5K) remain optimal** for:
-- Best AI performance (avoiding multi-turn degradation)
-- Human readability (Context Economy principle)
-- Cost efficiency (smaller prompts = faster/cheaper)
-- Cognitive load (attention limits haven't changed)
-
-**Context windows got bigger, but optimal usage patterns didn't change.**
-
-### Alternative Considered: Weighted Score
-
-We could keep the score approach but make weights more principled:
-- Use empirically validated weights from benchmarking
-- Make score calculation transparent
-- Show breakdown (token score, structure score, etc.)
-
-**Rejected because:**
-- Still abstracts away the actual metrics
-- Users don't care about scores, they care about token counts
-- Adds complexity without clear benefit
-- "Good enough" now is better than "perfect" later
-
-### Migration Notes
-
-**Breaking Changes:**
-- No more `ComplexityScore.score` field
-- No more `recommendation` field ('excellent', 'good', 'review', 'split')
-- Validation results return direct checks instead of derived scores
-
-**Backward Compatibility:**
-- Can keep deprecated score fields temporarily if needed
-- Migration path: Show both old score and new checks for one release
-- Remove score fields in v0.4.0
-
-**For Now:**
-- Implement new approach as replacement
-- Update all tests
-- Single release, clean break
-
-### Open Questions
-
-1. **Should we show positive feedback?** (e.g., "✓ Uses sub-specs")
-   - Pro: Reinforces good practices
-   - Con: Adds noise to output
-   - **Decision**: Yes, but only in verbose mode or when spec passes
-
-2. **Info level for 2-3.5K tokens?**
-   - Current design: Shows "acceptable, watch for growth"
-   - Alternative: Silent pass (only warn/error)
-   - **Research supports**: Slight degradation in this range, info message is appropriate
-   - **Decision**: Show info message - it's educational and matches research
-
-3. **Line count threshold?**
-   - Current: 500 lines backstop
-   - Research: Token count is primary, line count is proxy
-   - **Decision**: Keep 500 lines as simple backstop for extreme cases
-
-4. **Sub-spec thresholds?**
-   - Should sub-specs use same thresholds (2K/3.5K/5K)?
-   - Or lower since they're typically smaller?
-   - **Hypothesis**: Same thresholds - a sub-spec shouldn't be a dumping ground
-   - **Decision**: Use same thresholds, validate with real sub-spec data
-
-### Related Work
-
-- **Spec 066**: Established token-based validation with research
-- **Spec 069**: Implemented token counting utilities
-- **This spec (071)**: Simplifies the validation logic itself