@leanspec/ui 0.2.8 → 0.2.9-dev.20251205030455

package/.next/standalone/packages/ui/dist/standalone/specs/069-token-counting-utils/README.md

@@ -1,551 +0,0 @@
----
-status: complete
-created: '2025-11-13'
-tags:
-  - core
-  - tooling
-  - context-economy
-  - llm
-  - validation
-priority: high
-assignee: marvin
-created_at: '2025-11-13T02:17:52.074Z'
-depends_on:
-  - 066-context-economy-thresholds-refinement
-related:
-  - 059-programmatic-spec-management
-  - 048-spec-complexity-analysis
-  - 070-mcp-token-counting-tool
-updated_at: '2025-11-13T02:49:45.179Z'
-transitions:
-  - status: in-progress
-    at: '2025-11-13T02:35:30.196Z'
-  - status: complete
-    at: '2025-11-13T02:49:45.179Z'
-completed_at: '2025-11-13T02:49:45.179Z'
-completed: '2025-11-13'
----
-
-# Token Counting Utilities for LLM Context Management
-
-> **Status**: ✅ Complete · **Priority**: High · **Created**: 2025-11-13 · **Tags**: core, tooling, context-economy, llm, validation
-> **Assignee**: marvin · **Reviewer**: TBD
-
-**The Problem**: Token count is the most accurate predictor of LLM context performance, but we lack convenient tools (MCP, CLI, core utilities) to measure it for specs and sub-specs.
-
-**The Solution**: Create a comprehensive token counting utility layer as core infrastructure for LLM context management, enabling both humans and AI agents to measure and optimize spec token usage.
-
-## Overview
-
-### Why Token Counting Matters
-
-From spec 066 research findings:
-
-1. **Token count predicts AI performance better than line count**
-   - 39% average performance drop in multi-turn contexts (arXiv:2505.06120)
-   - Quality degradation starts well before 50K token limits
-   - 6x cost difference: 2,000-line vs 300-line specs
-
-2. **Content density varies significantly**
-   - Code: ~3 chars/token (denser)
-   - Prose: ~4 chars/token (lighter)
-   - Spec 016: Only 315 lines but ~2,400 tokens (26 code blocks)
-   - Spec 049: 374 lines but only ~1,700 tokens (pure prose)
-
-3. **Current validation uses tokenx**
-   - Integrated in `ComplexityValidator` (spec 066 implementation)
-   - Will migrate to tiktoken for exact token counts
-   - No user-facing tools to inspect token counts
-   - No way for AI agents to query token counts programmatically
-
-### What's Missing
-
-**Current State**:
-- ✅ `ComplexityValidator` uses `tokenx` internally (will migrate to tiktoken)
-- ✅ Token thresholds defined (2K/3.5K/5K - hypotheses)
-- ❌ No CLI command to check token counts
-- ❌ No MCP tool for AI agents to query tokens
-- ❌ No utility to count sub-spec tokens
-- ❌ No breakdown by content type (code vs prose vs tables)
-
-**User Pain Points**:
-- Can't answer "How many tokens is this spec?"
-- Can't compare token counts across specs
-- Can't see token breakdown before/after edits
-- AI agents can't make token-aware decisions
-- No way to validate MCP tool context fits in budget
-
-### What We're Building
-
-**Three Layers of Token Counting**:
-
-1. **Core Library** (`@leanspec/core`)
-   - Token counting utilities using `tiktoken`
-   - Sub-spec aggregation
-   - Content type breakdown
-   - Export for reuse across packages
-
-2. **CLI Commands** (`@leanspec/cli`)
-   - `lean-spec tokens <spec>` - Show token count for spec
-   - `lean-spec tokens <spec> --detailed` - Breakdown by file/type
-   - `lean-spec tokens --all` - Compare all specs
-   - Integration with `lean-spec analyze`
-
-3. **MCP Tools** (Future)
-   - `mcp_lean-spec_tokens` - Query token counts
-   - Enable AI agents to make token-aware decisions
-   - Support context budget planning
-
-## Design
-
-### Architecture
-
-```
-┌─────────────────────────────────────────────────────────┐
-│                    User Interfaces                       │
-├──────────────┬──────────────────┬───────────────────────┤
-│  CLI         │   MCP Server     │   Internal (Validator)│
-│              │                  │                       │
-│ lean-spec    │  mcp_lean-spec  │  ComplexityValidator  │
-│   tokens     │    _tokens       │                       │
-└──────┬───────┴────────┬─────────┴──────────┬───────────┘
-       │                │                    │
-       └────────────────┴────────────────────┘
-                        │
-              ┌─────────▼──────────┐
-              │   Core Library     │
-              │  @leanspec/core    │
-              ├────────────────────┤
-              │ TokenCounter       │
-              │  - count()         │
-              │  - analyze()       │
-              │  - breakdown()     │
-              │                    │
-              │ Uses: tiktoken     │
-              └────────────────────┘
-```
-
-### Token Counting Options
-
-Based on spec 066 analysis of token counting packages:
-
-### Using tiktoken for Token Counting
-
-**Official OpenAI tokenizer** for precise token counting and complexity measurement.
-
-**Note**: We use `tiktoken` (official OpenAI package) NOT `gpt-tokenizer` mentioned in spec 066. The `tiktoken` npm package is the official JavaScript port.
-
-**Why tiktoken?**:
-- ✅ Exact BPE encoding used by GPT-4 and similar models
-- ✅ Official OpenAI tokenizer (ported to JS)
-- ✅ Battle-tested and maintained (1.1M+ weekly downloads)
-- ✅ Token count is the primary metric for complexity (not line count)
-- ✅ Easy to install and setup (no complex configuration)
-- ✅ Fast enough for our use case (<50ms per spec)
-
-**Trade-offs**:
-- Bundle size: ~500KB (acceptable for core functionality)
-- Slightly slower than estimation (but negligible: <50ms per spec)
-- Model-specific encoding (we standardize on GPT-4/Claude encoding)
-
-**Package**: https://www.npmjs.com/package/tiktoken
-
-**Usage**:
-```typescript
-import { encoding_for_model } from 'tiktoken';
-
-const enc = encoding_for_model('gpt-4');
-const tokens = enc.encode(specContent);
-const count = tokens.length;
-enc.free(); // Important: free memory
-```
-
-**Decision**: Use tiktoken as the single solution—no fallback needed. Easy to setup, fast enough, and gives us exact token counts.
-
-### Core Utilities
-
-New utilities in `@leanspec/core/src/utils/token-counter.ts`:
-
-```typescript
-export interface TokenCount {
-  total: number;
-  files: {
-    path: string;
-    tokens: number;
-  }[];
-  breakdown?: {
-    code: number;      // Tokens in code blocks
-    prose: number;     // Tokens in prose
-    tables: number;    // Tokens in tables
-    frontmatter: number; // Tokens in frontmatter
-  };
-}
-
-export interface TokenCounterOptions {
-  detailed?: boolean;    // Include breakdown by file and type
-  includeSubSpecs?: boolean; // Count sub-spec files
-}
-
-export class TokenCounter {
-  /**
-   * Count tokens in a single file
-   */
-  async countFile(filePath: string, options?: TokenCounterOptions): Promise<TokenCount>;
-  
-  /**
-   * Count tokens in a spec (including sub-specs if requested)
-   */
-  async countSpec(specPath: string, options?: TokenCounterOptions): Promise<TokenCount>;
-  
-  /**
-   * Analyze token breakdown by content type
-   */
-  async analyzeBreakdown(content: string): Promise<TokenCount['breakdown']>;
-  
-  /**
-   * Check if content fits within token limit
-   */
-  isWithinLimit(count: TokenCount, limit: number): boolean;
-  
-  /**
-   * Format token count for display
-   */
-  formatCount(count: TokenCount, verbose?: boolean): string;
-}
-```
-
-### CLI Commands
-
-#### Basic Command
-
-```bash
-$ lean-spec tokens 059
-Spec: 059-programmatic-spec-management
-Total: 2,100 tokens
-Files:
-  README.md: 394 lines, 2,100 tokens
-```
-
-#### With Sub-Specs
-
-```bash
-$ lean-spec tokens 059 --include-sub-specs
-Spec: 059-programmatic-spec-management
-Total: 8,450 tokens
-Files:
-  README.md:            2,100 tokens (394 lines)
-  ARCHITECTURE.md:      1,850 tokens (411 lines)
-  CONTEXT-ENGINEERING.md: 3,200 tokens (799 lines)
-  COMMANDS.md:          560 tokens (156 lines)
-  ALGORITHMS.md:        240 tokens (62 lines)
-  IMPLEMENTATION.md:    310 tokens (88 lines)
-  TESTING.md:           190 tokens (54 lines)
-```
-
-#### Detailed Breakdown
-
-```bash
-$ lean-spec tokens 066 --detailed
-Spec: 066-context-economy-thresholds-refinement
-Total: 7,307 tokens
-
-Content Breakdown:
-  Prose:       4,200 tokens (57%)
-  Code:        2,100 tokens (29%)
-  Tables:      800 tokens (11%)
-  Frontmatter: 207 tokens (3%)
-
-Performance Indicators:
-  Cost multiplier: 6.1x vs baseline (1,200 tokens)
-  AI effectiveness: ~65% (hypothesis - >5K tokens)
-  Context Economy: ⚠️ Review - elevated token count
-
-Recommendation: Consider splitting or using sub-specs
-```
-
-#### Compare All Specs
-
-```bash
-$ lean-spec tokens --all --sort-by tokens
-╭────────────────────────────────────────────────────────╮
-│ Token Counts (Top 10)                                   │
-├─────────┬──────────────────────────────┬───────────────┤
-│ Spec    │ Name                         │ Tokens        │
-├─────────┼──────────────────────────────┼───────────────┤
-│ 066     │ context-economy-thresholds   │ 7,307 (⚠️)    │
-│ 045     │ unified-dashboard            │ 4,800 (⚠️)    │
-│ 016     │ github-action                │ 2,400         │
-│ 059     │ programmatic-spec-mgmt       │ 2,100         │
-│ 049     │ first-principles             │ 1,700         │
-│ 051     │ docs-system-prompt           │ 1,600         │
-╰─────────┴──────────────────────────────┴───────────────╯
-
-Legend: ⚠️ = >3,500 tokens (review recommended)
-```
-
-### MCP Tool Interface
-
-```json
-{
-  "name": "mcp_lean-spec_tokens",
-  "description": "Count tokens in spec or sub-spec for LLM context management",
-  "parameters": {
-    "specPath": {
-      "type": "string",
-      "description": "Spec name, number, or file path (e.g., '059', 'unified-dashboard', '059/DESIGN.md')"
-    },
-    "includeSubSpecs": {
-      "type": "boolean",
-      "description": "Include all sub-spec files in count (default: false)"
-    },
-    "detailed": {
-      "type": "boolean",
-      "description": "Return breakdown by content type (default: false)"
-    }
-  }
-}
-```
-
-**Example Usage by AI Agent**:
-```
-Agent: "I need to include spec 059 in context. Will it fit?"
-Tool call: mcp_lean-spec_tokens("059", includeSubSpecs=true)
-Response: { total: 8450 }
-Agent: "That's too large. Let me just include README.md"
-Tool call: mcp_lean-spec_tokens("059")
-Response: { total: 2100 }
-Agent: "Perfect, that fits in my context budget."
-```
-
-## Plan
-
-### Phase 1: Core Utilities (v0.3.0 - Week 1) ✅ COMPLETE
-- [x] Install `tiktoken` as dependency
-- [x] Create `TokenCounter` class in `@leanspec/core`
-- [x] Implement `countFile()` using `tiktoken`
-- [x] Implement `countSpec()` with sub-spec support
-- [x] Implement `analyzeBreakdown()` for content type analysis
-- [x] Add unit tests for edge cases (31 tests, all passing)
-- [x] Export utilities from core package
-
-### Phase 2: CLI Integration (v0.3.0 - Week 1-2) ✅ COMPLETE
-- [x] Add `tokens` command to CLI (using tiktoken)
-- [x] Implement `--include-sub-specs` flag
-- [x] Implement `--detailed` flag for breakdown
-- [x] Implement `--all` flag for project-wide view
-- [x] Add `--sort-by` option (tokens, lines, name)
-- [x] Format output with tables and colors
-- [x] Add `--json` flag for structured output
-
-### Phase 3: Integration & Polish (v0.3.0 - Week 2) ✅ COMPLETE
-- [x] **Replace `tokenx` with `tiktoken` in `ComplexityValidator`**
-- [x] **Make token count the PRIMARY complexity metric** (line count secondary)
-- [x] Update validation thresholds based on exact token counts (2K/3.5K/5K)
-- [x] Ensure consistency across validation and CLI
-- [x] Validation tests passing (21 tests in complexity.test.ts)
-- [x] Documentation complete (comprehensive spec with research rationale)
-
-### Phase 4: MCP Tool (Moved to Spec 070)
-- Deferred to separate spec for focused implementation
-- See spec 070-mcp-token-counting-tool for details
-- Infrastructure ready, just needs MCP server integration
-
-### Phase 5: Advanced Features (Future - v0.4.0+)
-- [ ] Add token trends over time (git history)
-- [ ] Add context budget planning (`--budget` flag)
-- [ ] Add "will this fit?" checker for MCP tools
-- [ ] Support for multiple model tokenizers (Claude, Gemini, etc.)
-- [ ] Token cost estimation ($/1M tokens)
-
-## Test
-
-### Unit Tests ✅ COMPLETE
-
-**Core Utilities**:
-- [x] `countFile()` returns correct token counts
-- [x] `countSpec()` aggregates sub-specs correctly
-- [x] `analyzeBreakdown()` categorizes content types
-- [x] `isWithinLimit()` compares correctly
-- [x] `formatCount()` produces readable output
-
-**Edge Cases**:
-- [x] Empty files (0 tokens)
-- [x] Very large files (>10K tokens)
-- [x] Files with only code blocks
-- [x] Files with only frontmatter
-- [x] Specs without sub-specs
-- [x] Invalid file paths
-
-**Test Results**: 31 tests passing in `token-counter.test.ts`
-
-### Integration Tests ✅ COMPLETE
-
-**CLI Commands**:
-- [x] `lean-spec tokens <spec>` shows basic count
-- [x] `--include-sub-specs` aggregates correctly
-- [x] `--detailed` shows breakdown
-- [x] `--all` lists all specs
-- [x] Output format is readable and correct
-- [x] Error handling for invalid specs
-- [x] `--json` flag outputs structured data
-
-**Validation**: Tested on spec 069 itself (4,936 tokens, warning threshold)
-
-### Validation Tests ✅ COMPLETE
-
-**Against Known Specs**:
-- [x] Spec 066: 8,073 tokens (problem threshold, matches validation)
-- [x] Spec 069: 4,936 tokens (warning threshold, matches validation)
-- [x] Spec 059: 3,364 tokens (good range)
-- [x] Spec 049: 3,413 tokens (good range)
-- [x] Spec 016: 2,004 tokens (good range, code-dense)
-
-**Project Stats**: 34 specs, 73,802 total tokens, 2,171 average
-
-### Consistency Tests ✅ COMPLETE
-
-**Validate tiktoken Behavior**:
-- [x] Token counts are consistent across multiple runs
-- [x] Proper memory cleanup (enc.free() called)
-- [x] Works with various content types (code, prose, tables)
-- [x] Handles edge cases (empty files, very large files)
-- [x] Unicode and emoji support verified
-
-## Success Metrics
-
-### Quantitative ✅ ACHIEVED
-
-**Performance**:
-- [x] Token counting takes <50ms per spec (tested: ~40ms average)
-- [x] Aggregate counting (34 specs) takes <500ms (tested: ~407ms)
-- [x] Memory usage minimal with proper cleanup
-
-**Reliability**:
-- [x] Token counts are consistent and reproducible
-- [x] Matches validation thresholds correctly
-- [x] Uses same tokenization as GPT-4/Claude (tiktoken)
-- [x] 31 unit tests + 21 complexity tests all passing
-
-### Qualitative ✅ ACHIEVED
-
-**Developer Experience**:
-- [x] "Now I can see token counts easily" - CLI command working
-- [x] "Helps me understand Context Economy better" - Indicators show cost/effectiveness
-- [x] "Makes token-aware editing decisions" - Validation provides actionable feedback
-- [x] "CLI output is clear and actionable" - Formatted with colors, emojis, recommendations
-
-**AI Agent Experience** (Deferred to Spec 070):
-- [ ] "Can query token counts programmatically" - MCP tool needed
-- [ ] "Makes informed context budget decisions" - MCP tool needed
-- [ ] "Avoids overloading context windows" - MCP tool needed
-- [ ] "Understands which specs fit in context" - MCP tool needed
-
-## Notes
-
-### Why This Spec Exists
-
-**Separated from Spec 059** because:
-1. **Different lifecycle**: Token counting is foundational infrastructure, programmatic spec management builds on it
-2. **Clearer dependency**: Spec 059 *depends on* having token counting utilities
-3. **Reusable utilities**: Token counting is useful beyond just spec management (MCP tools, validation, CLI)
-4. **Context Economy**: Spec 059 is already 394 lines with 6 sub-specs - adding token counting details would violate its own principles
-
-**Dependency Relationship**:
-- Spec 066: Establishes *why* token counting matters (research, thresholds)
-- **Spec 069 (this)**: Provides *how* to count tokens (utilities, tools)
-- Spec 059: Uses token counting for *programmatic transformations*
-
-### Research References
-
-From Spec 066:
-
-1. **Token Count Critical for AI Performance**
-   - [arXiv:2505.06120](https://arxiv.org/abs/2505.06120): 39% performance drop in multi-turn contexts
-   - [Berkeley BFCL](https://gorilla.cs.berkeley.edu/leaderboard.html): All models worse with more tools/options
-   - [Databricks Research](https://www.databricks.com/blog/long-context-rag-performance-llms): Degradation even within limits
-
-2. **Token Count vs Line Count**
-   - Code: ~3 chars/token (denser)
-   - Prose: ~4 chars/token (lighter)
-   - Better predictor than line count for AI effectiveness
-
-3. **Validated Thresholds** (hypothesis, to be tested):
-   - <2K tokens: Baseline performance (~100%)
-   - 2-3.5K tokens: Good range (~90-95%)
-   - 3.5-5K tokens: Warning zone (~80-85%)
-   - >5K tokens: Should split (~65-80%)
-
-### Why tiktoken?
-
-| Feature | tiktoken |
-|---------|----------|
-| Tokenization | Exact BPE encoding (GPT-4) |
-| Size | ~500KB |
-| Speed | Fast (<50ms/spec, <2s for 100 specs) |
-| Dependencies | Some (but well-maintained) |
-| Downloads/week | 1.1M+ |
-| Maintenance | Official OpenAI port |
-| Setup | Easy (`npm install tiktoken`) |
-
-**Decision Rationale**:
-- Token count is the **primary complexity metric** (spec 066 research)
-- Exact token counts are essential for reliable validation thresholds
-- ~500KB bundle cost is justified for core functionality
-- Performance is more than acceptable for our use case
-- Easy to install and setup—no complex configuration needed
-- Line count becomes secondary "backstop" metric only
-- **No fallback needed**—tiktoken is good enough as single solution
-
-### Implementation Notes
-
-**Why tiktoken Over gpt-tokenizer?**
-- Spec 066 mentioned `gpt-tokenizer` (53.1 MB unpacked)
-- Better choice: `tiktoken` (official OpenAI port to JS)
-- Official, well-maintained, reasonable size (~500KB)
-- **v0.3.0 decision: Use tiktoken as single solution**
-
-**Why tiktoken Over tokenx?**
-- Token count is THE metric for complexity (spec 066 research)
-- tokenx estimation has ~10% variance which is too large for validation thresholds
-- Need exact counts to set reliable thresholds and measure effectiveness
-- Bundle size (~500KB) is justified for core functionality
-- Easy to install—no fallback complexity needed
-
-**Migration from tokenx**:
-- `ComplexityValidator` currently uses `tokenx`
-- Phase 3: Replace with `tiktoken` for exact counts
-- Update validation thresholds based on exact counts
-- Remove `tokenx` dependency entirely
-
-### Open Questions
-
-1. **Display Format**: Show tokens always, or only on request?
-   - **Decision**: Show in `lean-spec list` with flag, dedicated `tokens` command for details
-
-2. **Sub-Spec Aggregation**: Default to including sub-specs or not?
-   - **Decision**: Default to README only (most common), `--include-sub-specs` flag for all
-
-3. **MCP Tool Priority**: Build now or defer to v0.4.0?
-   - **Updated**: Build in v0.3.0 (Phase 4) - token counting is foundational for AI agents
-
-4. **Exact Counts**: Install tiktoken now or wait for user feedback?
-   - **RESOLVED**: Install tiktoken in v0.3.0 as single dependency, no fallback
-   - **Rationale**: Token count is primary metric, need exact counts, easy to setup
-
-5. **Integration with Analyze**: Show tokens in `lean-spec analyze`?
-   - **Decision**: Yes, prominently display exact token count in complexity analysis
-
-6. **Line Count Role**: Keep line count validation or remove it?
-   - **Decision**: Keep as backstop only (warn at >500 lines regardless of tokens)
-   - **Primary metric**: Token count thresholds (2K/3.5K/5K from spec 066)
-
-## Related Specs
-
-- **[066-context-economy-thresholds-refinement](../066-context-economy-thresholds-refinement/)** - Research & thresholds (dependency)
-- **[059-programmatic-spec-management](../059-programmatic-spec-management/)** - Programmatic transformations (dependent)
-- **[048-spec-complexity-analysis](../048-spec-complexity-analysis/)** - Initial complexity work
-- **[018-spec-validation](../018-spec-validation/)** - Validation framework
-
----
-
-**Remember**: Token counting is foundational infrastructure for Context Economy. Make it fast, accurate, and easy to use.