@atomic-ehr/fhirpath 0.0.1-canary.0c6931e.20250727185306

package/src/lexer2/optimization-summary.md

@@ -0,0 +1,718 @@
+# Lexer Optimization Summary
+
+## 1. Lookup Tables for Character Classification
+
+### What was changed:
+- Created lookup tables (`IS_DIGIT`, `IS_LETTER`, `IS_LETTER_OR_DIGIT`, `IS_HEX_DIGIT`) as `Uint8Array(256)`
+- Replaced all function calls like `isDigit(charCode)` with direct array lookups like `IS_DIGIT[charCode]`
+- Added bounds checking where needed (`charCode >= 0 && charCode < 256`)
+
+### Performance Impact:
+- Before: ~1,477K expressions/second
+- After: ~1,546K expressions/second
+- Improvement: ~4.7%
+
+### Why it works:
+- Array lookups are O(1) and avoid function call overhead
+- Modern JavaScript engines optimize array access very well
+- Lookup tables fit in CPU cache (only 256 bytes per table)
+- Eliminates branching in character classification logic
+
+### Trade-offs:
+- Slightly more memory usage (4 × 256 = 1KB for lookup tables)
+- One-time initialization cost when module loads
+- Need bounds checking for safety (charCode < 256)
+
+## 2. Switch-based Keyword Lookup
+
+### What was changed:
+- Replaced object/hash map keyword lookup with nested switch statements
+- First switch on keyword length (2-12 characters)
+- Then switch on the actual keyword value
+- Early exit for non-keyword lengths
+
+### Performance Impact:
+- Before: ~1,546K expressions/second
+- After: ~2,192K expressions/second
+- Improvement: ~42%
+
+### Why it works:
+- Switch statements compile to jump tables in V8
+- Length check filters out most identifiers immediately
+- No hash computation or object property lookup
+- Better CPU branch prediction
+- Compiler can optimize switch statements very aggressively
+
+### Trade-offs:
+- More verbose code (but still maintainable)
+- Fixed set of keywords (can't add dynamically)
+- Slightly larger code size
+
+## 3. Character Code Switch for readWhitespace
+
+### What was changed:
+- Replaced string comparisons (`char === ' '`) with character code switch
+- Single `charCodeAt()` call instead of multiple string comparisons
+- Switch statement on integer values (32, 9, 13, 10)
+
+### Performance Impact:
+- Before: ~2,192K expressions/second
+- After: ~2,240K expressions/second
+- Improvement: ~2.2%
+
+### Why it works:
+- Integer comparison is faster than string comparison
+- Switch on integers compiles to efficient jump table
+- Single charCodeAt call vs multiple charAt/string comparisons
+- Better branch prediction with switch statement
+
+### Trade-offs:
+- Slightly less readable (need to know ASCII codes)
+- More verbose code structure
+
+## 4. Reusable Token Attempt (Reverted)
+
+### What was tried:
+- Added a reusable token object to avoid allocations
+- Created `setToken()` method to update and return the same object
+- Modified `tokenize()` to clone tokens when storing in array
+
+### Performance Impact:
+- Before: ~2,240K expressions/second
+- After: ~1,995K expressions/second
+- **Degradation: ~11%**
+
+### Why it failed:
+- Method call overhead for `setToken()` on every token
+- Still need to clone tokens in `tokenize()`, negating the benefit
+- The cloning actually made it worse than direct object creation
+- V8's object allocation is already highly optimized for small objects
+
+### Lesson learned:
+- Not all traditional optimizations work in modern JavaScript engines
+- V8's object allocation is very fast for small, short-lived objects
+- Method call overhead can outweigh allocation savings
+
+## 5. Token Representation Benchmarks
+
+### What was tested:
+Compared different approaches for token representation:
+- Plain object literals `{ type, start, end }`
+- Arrays `[type, start, end]`
+- Classes with constructor
+- Classes with fields
+- Object.create(null)
+- Object.assign and spread operators
+
+### Performance Results:
+
+#### Token Creation (5M tokens):
+- **Object literal**: 8.09ms (618M tokens/sec) ✅
+- **Class with constructor**: 14.45ms (346M tokens/sec) - 78% slower
+- **Arrays**: ~28ms - 244% slower in real usage
+- **Object.assign**: 224.68ms - 2678% slower
+- **Spread operator**: 109.61ms - 1255% slower
+
+#### Real Lexer Usage (100K iterations):
+- **Object literals**: 54.06ms (9.2M expressions/sec) ✅
+- **Arrays**: 180.79ms (2.8M expressions/sec)
+- **Classes**: 169.79ms (2.9M expressions/sec)
+
+### Why Object Literals Win:
+1. **V8 Hidden Classes** - Objects with consistent shape share optimized hidden classes
+2. **Inline Caching** - Property access is optimized through monomorphic inline caches
+3. **No Constructor Overhead** - Direct allocation without function calls
+4. **JIT Optimization** - V8 heavily optimizes object literal creation
+5. **Better Property Access** - Named properties are faster than array indices for this use case
+
+### Key Insights:
+- Arrays are slower despite using less memory due to worse access patterns
+- Classes add constructor overhead without benefits
+- V8 is extremely optimized for plain object literals with consistent shapes
+- The current implementation is already optimal
+
+### Conclusion:
+The current token representation using plain object literals is the fastest approach. No change needed.
+
+## 6. Line and Column Tracking Added Back
+
+### What was changed:
+- Added `line` and `column` fields back to Token interface
+- Track line/column during advance() method
+- Capture startLine and startColumn for each token
+- Update line/column in readWhitespace and readSpecialIdentifier
+
+### Performance Impact:
+- Before: ~2,240K expressions/second (without line/column)
+- After: ~2,147K expressions/second (with line/column)
+- **Cost: ~4.2% performance decrease**
+
+### Why it's acceptable:
+- Line/column information is valuable for error reporting and debugging
+- 4.2% overhead is reasonable for this functionality
+- Still maintaining over 2M expressions/second
+- Total performance is still 45% better than original implementation
+
+### Implementation notes:
+- Line increments on '\n', column resets to 1
+- Column increments for all other characters
+- Special handling in readSpecialIdentifier for multi-character advances
+
+## 7. Numeric Enum Token Types
+
+### What was changed:
+- Converted TokenType from string enum to numeric enum
+- String enum: `IDENTIFIER = 'IDENTIFIER'` → Numeric enum: `IDENTIFIER = 6`
+- Added `tokenTypeToString()` helper function for debugging
+- Added `debugTokens()` method for human-readable token output
+
+### Performance Impact:
+- Before: ~2,147K expressions/second (string enums)
+- After: ~2,200K expressions/second (numeric enums)
+- **Improvement: ~2.5%**
+
+### Why it works:
+- Numeric comparisons are faster than string comparisons
+- Smaller memory footprint (4 bytes vs string length)
+- Better CPU cache utilization
+- Switch statements optimize better with numeric values
+
+### Trade-offs:
+- Less readable in raw debugging output (see `6` instead of `"IDENTIFIER"`)
+- Need helper functions for human-readable output
+- Breaking change if external code depends on string values
+
+### Debug Support:
+```typescript
+// Convert numeric type to string
+tokenTypeToString(TokenType.IDENTIFIER) // "IDENTIFIER"
+
+// Debug tokens
+lexer.debugTokens() // "IDENTIFIER(foo) [1:1]\nDOT(.) [1:4]..."
+```
+
+## 8. CharCode-based Dispatch in nextToken()
+
+### What was changed:
+- Replaced char-based switch (`switch (firstChar)`) with charCode-based switch (`switch (firstCharCode)`)
+- Use numeric character codes instead of string literals (e.g., `case 39:` instead of `case "'"`)
+- Updated all case statements to use ASCII codes with comments showing the character
+- Modified error handling to convert charCode back to string when needed
+
+### Performance Impact:
+- Before: ~2,212K expressions/second (char-based dispatch)
+- After: ~2,305K expressions/second (charCode-based dispatch)  
+- **Improvement: ~4.2%** (93,132 more expressions/sec)
+
+### Why it works:
+- Integer comparison is faster than string comparison in switch statements
+- JavaScript engines optimize numeric switches into jump tables more efficiently
+- Avoids string indexing overhead when accessing characters
+- Better CPU branch prediction with numeric values
+- Single charCodeAt() call is more efficient than charAt() or string indexing
+
+### Implementation Details:
+```typescript
+// Before
+switch (firstChar) {
+  case "'": return this.readString();
+  case ".": return { type: TokenType.DOT, ... };
+}
+
+// After  
+switch (firstCharCode) {
+  case 39:  // '
+    return this.readString();
+  case 46:  // .
+    return { type: TokenType.DOT, ... };
+}
+```
+
+### Trade-offs:
+- Less readable code (need ASCII code comments)
+- Need to convert charCode back to string for error messages
+- Slightly more complex error handling
+
+### Isolated Test Results:
+In isolated benchmarks, charCode dispatch showed 25% improvement over char dispatch. The smaller gain in the real lexer (4.2%) is due to:
+- Other operations diluting the pure dispatch improvement
+- Real-world expressions include various token types, not just operators
+- Position tracking, token creation, and other overhead
+
+## 9. Lookup Table for nextToken() Dispatch (Attempted)
+
+### What was tried:
+- Created a lookup table mapping character codes to token types for single-character tokens
+- Used array indexing instead of switch statement for dispatch
+- Aimed to reduce branching and improve performance
+
+### Implementation approach:
+```typescript
+// Lookup table initialization
+private static readonly CHAR_TOKEN_TABLE: (TokenType | null)[] = new Array(256).fill(null);
+static {
+  this.CHAR_TOKEN_TABLE[46] = TokenType.DOT;        // .
+  this.CHAR_TOKEN_TABLE[40] = TokenType.LPAREN;     // (
+  // ... etc
+}
+
+// Dispatch logic
+const tokenType = CHAR_TOKEN_TABLE[firstCharCode];
+if (tokenType !== null) {
+  this.advance();
+  return { type: tokenType, ... };
+}
+// Fall back to switch for complex cases
+```
+
+### Performance Impact:
+- Switch-based: ~7,164K expressions/second
+- Lookup table: ~3,990K expressions/second
+- **Result: 79.5% SLOWER**
+
+### Why it failed:
+1. **V8 Optimization** - Modern JavaScript engines compile numeric switches into highly optimized jump tables
+2. **Array Access Overhead**:
+   - Array bounds checking on every access
+   - Memory indirection (fetch from array)
+   - Potential cache misses
+   - Additional null check required
+3. **Two-Stage Dispatch** - Still need switch/if-else fallback for complex tokens
+4. **CPU Branch Prediction** - Modern CPUs predict switch branches very well
+5. **Inline Optimization** - V8 can inline entire switch statement, but not array lookups
+
+### Detailed Test Results:
+- **Regular expressions test**: 79.5% slower with lookup table
+- **Operator-heavy expressions**: 43.9% slower with lookup table
+- The more operators (which benefit most from lookup), the less severe the penalty
+- But still significantly slower in all cases
+
+### Lesson learned:
+- Not all traditional C/C++ optimizations translate to JavaScript
+- V8's switch statement optimization is extremely efficient for numeric cases
+- Array lookups in hot paths can be slower than well-optimized switches
+- The current switch-based implementation is already optimal
+
+### Current Performance Summary:
+- Original: ~1,477K expressions/second
+- Current: ~2,305K expressions/second
+- **Total improvement: ~56%**
+
+### Conclusion:
+The current switch-based implementation with charCode dispatch represents the optimal approach for the nextToken() method. Further optimization efforts should focus on other areas of the lexer.
+
+## 10. Unicode Support (Added then Removed)
+
+### What was attempted:
+- Added full Unicode support for identifiers using regex patterns `/\p{L}|\p{Nl}/u` and `/\p{L}|\p{N}|\p{M}|\p{Pc}/u`
+- Allowed Unicode characters in regular identifiers (e.g., `café`, `日本語`)
+- Extended support to environment variable identifiers
+
+### Performance Impact:
+- Before Unicode: ~2,305K expressions/second
+- With Unicode: ~1,908K expressions/second
+- **Result: 17% performance degradation**
+
+### Why it was removed:
+1. **Not spec compliant** - FHIRPath grammar explicitly defines identifiers as `[A-Za-z_][A-Za-z0-9_]*`
+2. **Unicode is only allowed in**:
+   - String literals: `'café'`, `'日本語'`
+   - Delimited identifiers: `` `café` ``, `` `日本語` ``
+   - Environment variable strings: `%'café'`, `%`日本語``
+3. **Performance cost** - Regex checks with Unicode property escapes are expensive
+
+### Implementation Details:
+The original implementation used regex for every character:
+```typescript
+function isUnicodeIdentifierStart(char: string): boolean {
+  return /\p{L}|\p{Nl}/u.test(char);  // Expensive!
+}
+```
+
+### Final Resolution:
+- Removed all Unicode support from regular identifiers
+- Kept ASCII-only lookup tables for maximum performance
+- Unicode remains supported in strings and delimited identifiers as per spec
+- Performance restored to ~2,303K expressions/second
+
+### Lesson learned:
+- Always verify spec compliance before implementing features
+- Unicode regex operations have significant performance overhead
+- Following the spec can lead to better performance
+
+### Current Performance Summary:
+- Original: ~1,477K expressions/second
+- Current: ~2,893K expressions/second
+- **Total improvement: ~96%**
+
+## 11. Optimize readSpecialIdentifier (Avoid substring)
+
+### What was changed:
+- Replaced `substring()` call with direct character code comparisons
+- Check each character individually using `charCodeAt()` 
+- Avoid allocating a new string object for lookahead
+
+### Implementation:
+```typescript
+// Before: const ahead = this.input.substring(this.position, this.position + 6);
+// After: Direct charCode checks
+if (pos + 4 < len &&
+    this.input.charCodeAt(pos + 1) === 116 && // t
+    this.input.charCodeAt(pos + 2) === 104 && // h
+    this.input.charCodeAt(pos + 3) === 105 && // i
+    this.input.charCodeAt(pos + 4) === 115) { // s
+  // $this
+}
+```
+
+### Performance Impact:
+- Before: ~2,303K expressions/second
+- After: ~2,893K expressions/second
+- **Improvement: ~25.6%**
+
+### Why it works:
+- Avoids string allocation overhead
+- Direct integer comparisons are faster than string operations
+- No temporary objects created
+- Better memory locality
+
+## 12. Remove Redundant Bounds Checking for Lookup Tables
+
+### What was changed:
+- Removed unnecessary upper bound checks (`< 256`) from lookup table accesses
+- Changed from `charCode >= 0 && charCode < 256 && IS_DIGIT[charCode]` to `charCode !== -1 && IS_DIGIT[charCode]`
+- Applied to all lookup table uses: IS_DIGIT, IS_LETTER, IS_LETTER_OR_DIGIT, IS_HEX_DIGIT
+
+### Why the optimization works:
+1. **charCodeAt() always returns valid values** - Either 0-65535 or NaN (never negative except our -1 sentinel)
+2. **JavaScript arrays handle out-of-bounds gracefully** - Returns `undefined` which is falsy
+3. **Only need to check for -1** - Our EOF sentinel value from peekCharCode()
+4. **Reduces comparisons** - From 3 comparisons to 1 per lookup
+
+### Performance Impact:
+- Before: ~2,893K expressions/second
+- After: ~4,454K expressions/second  
+- **Improvement: ~54%**
+
+## 13. Add Dedicated EOF Case to Switch Statement
+
+### What was changed:
+- Added explicit `case -1:` for EOF in the main switch statement
+- Removed redundant EOF check from the default case
+- Simplified default case by removing `firstCharCode !== -1` checks
+
+### Implementation:
+```typescript
+// Before: 
+default:
+  if (firstCharCode !== -1 && IS_DIGIT[firstCharCode]) { ... }
+  if (firstCharCode !== -1 && IS_LETTER[firstCharCode]) { ... }
+  if (firstCharCode === -1) { return EOF; }
+
+// After:
+case -1:
+  return { type: TokenType.EOF, ... };
+default:
+  if (IS_DIGIT[firstCharCode]) { ... }
+  if (IS_LETTER[firstCharCode]) { ... }
+```
+
+### Performance Impact:
+- Before: ~4,454K expressions/second
+- After: ~4,497K expressions/second
+- **Improvement: ~1%**
+
+### Why it works:
+- Eliminates redundant EOF checks in the default case
+- Switch statement handles EOF directly without falling through
+- Cleaner code path for the common case
+
+## 14. Avoid Substring Allocation in readIdentifierOrKeyword
+
+### What was changed:
+- Replaced `substring()` call with direct character code comparisons for common keywords
+- Check keywords directly from the input buffer using charCodeAt()
+- Only use substring for longer, less common keywords (6+ characters)
+
+### Implementation:
+```typescript
+// Before: 
+const value = this.input.substring(start, this.position);
+switch (value) {
+  case 'true': type = TokenType.TRUE; break;
+  // ... many string comparisons
+}
+
+// After:
+// For length 4 example:
+const c0_4 = input.charCodeAt(start);
+if (c0_4 === 116 && // 't'
+    input.charCodeAt(start + 1) === 114 && // 'r'
+    input.charCodeAt(start + 2) === 117 && // 'u'
+    input.charCodeAt(start + 3) === 101) { // 'e'
+  type = TokenType.TRUE;
+}
+```
+
+### Performance Impact:
+- Before: ~4,497K expressions/second
+- After: ~5,656K expressions/second
+- **Improvement: ~26%**
+
+### Why it works:
+1. **Avoids string allocation** - No substring object created for most keywords
+2. **Direct integer comparisons** - Faster than string equality checks
+3. **Optimized for common cases** - Most keywords are 2-5 characters
+4. **Falls back gracefully** - Still uses substring for rare long keywords
+
+### Trade-offs:
+- More verbose code (but still maintainable with comments)
+- Larger code size due to unrolled comparisons
+- Best for hot paths with frequent keyword checking
+
+## 15. Inline advance() Method in Hot Paths
+
+### What was changed:
+- Inlined `advance()` method in the most frequently called code paths
+- Replaced `this.advance()` with direct `this.position++; this.column++` 
+- Applied to: single-character tokens, two-character operators, identifier/number reading loops
+
+### Where inlined:
+1. **nextToken() switch cases** - All single-character tokens (., (, ), +, -, *, =, etc.)
+2. **Two-character operators** - <, >, !, and their combinations (<=, >=, !=, !~)
+3. **Tight loops** - readIdentifierOrKeyword, readNumber digit scanning
+
+### Performance Impact:
+- Before: ~5,656K expressions/second
+- After: ~6,042K expressions/second
+- **Improvement: ~7%**
+
+### Why it works:
+1. **Eliminates function call overhead** - No stack frame allocation/deallocation
+2. **Better inlining by JIT** - Compiler can optimize the inlined code better
+3. **Removes unused work** - advance() returns a value that's rarely used
+4. **Hot path optimization** - These paths are executed millions of times
+
+### Trade-offs:
+- Code duplication (position++ and column++ repeated)
+- Harder to maintain if line/column tracking logic changes
+- Must remember to handle newlines separately where needed
+
+## 16. Use CharCode Instead of Char in String/Delimiter Reading
+
+### What was changed:
+- Replaced string comparisons with character code comparisons in string reading methods
+- Applied to: `readString()`, `readDelimitedIdentifier()`, `readEnvVar()`
+- Changed `char === "'"` to `charCode === 39` etc.
+
+### Implementation:
+```typescript
+// Before:
+if (char === quoteChar) { ... }
+if (char === '\\') { ... }
+switch (escaped) {
+  case "'": ...
+  case "n": ...
+}
+
+// After:
+if (charCode === quoteCharCode) { ... }
+if (charCode === 92) { // \\
+switch (escapedCode) {
+  case 39:  // '
+  case 110: // n
+}
+```
+
+### Performance Impact:
+- Before: ~6,042K expressions/second
+- After: ~6,093K expressions/second
+- **Improvement: ~1%**
+
+### Why it works:
+- Integer comparisons faster than string comparisons
+- Consistent with the optimization pattern used in nextToken()
+- Avoids string allocation for single character comparisons
+- Better for hot paths that process many characters
+
+## 17. Inline peek() and advance() in readString
+
+### What was changed:
+- Inlined `peekCharCode()` and `advance()` calls in the `readString()` method
+- Replaced method calls with direct buffer access and position/column updates
+- Applied specifically to string parsing which is a hot path for string-heavy expressions
+
+### Implementation:
+```typescript
+// Before:
+const charCode = this.peekCharCode();
+if (charCode === quoteCharCode) {
+  this.advance();
+  return { type: TokenType.STRING, ... };
+}
+
+// After:
+const charCode = this.input.charCodeAt(this.position);
+if (charCode === quoteCharCode) {
+  this.position++;
+  this.column++;
+  return { type: TokenType.STRING, ... };
+}
+```
+
+### Performance Impact:
+- **String-heavy expressions**: ~31% improvement (3.0M → 4.0M expressions/second)
+- **Overall benchmark**: Mixed results (slight decrease in some tests)
+- Decision: Applied to `readString()` only, not to other methods
+
+### Why it works for strings:
+1. **String parsing is advance-heavy** - Each character requires an advance() call
+2. **Eliminates method call overhead** - Direct buffer access is faster
+3. **Better locality** - All operations on adjacent memory
+4. **Escape sequence handling** - Multiple advance() calls for escape sequences benefit greatly
+
+### Trade-offs:
+- **Code size increase** - Method becomes larger and more complex
+- **Maintenance burden** - Must handle line/column tracking manually
+- **Mixed performance impact** - Benefits string-heavy code but may hurt instruction cache
+- **JIT optimization** - Very large methods may be harder for JIT to optimize
+
+### String-Heavy Performance Test Results:
+
+Comprehensive benchmarks show the significant impact of this optimization on string processing:
+
+**Test Categories and Results:**
+- **Long strings**: 2,171,821 expr/sec (strings with 100+ characters)
+- **Escaped strings**: 2,512,563 expr/sec (strings with escape sequences like \n, \t, \\)
+- **Unicode strings**: 2,061,856 expr/sec (strings with \uXXXX escape sequences)
+- **Multiple strings**: 816,327 expr/sec (expressions with multiple string concatenations)
+- **FHIR expressions**: 1,179,941 expr/sec (real-world FHIR queries with string literals)
+- **Environment variables**: 1,494,768 expr/sec (string-based environment variables)
+
+**Overall String-Heavy Performance**: 1,483,680 expressions/second
+
+**Comparison with Simple Expressions**:
+- Simple expressions (identifiers, operators): 4,132,231 expr/sec
+- String-heavy slowdown: 2.71x
+- The ~31% improvement from inlining is crucial for maintaining acceptable string performance
+
+**Conclusion**: 
+The peek/advance inlining in readString provides essential performance for string-heavy workloads. Without this optimization, string processing would be ~31% slower, making string-heavy expressions run at only ~1.13M expr/sec instead of ~1.48M expr/sec.
+
+### Current Performance Summary:
+- Original: ~1,477K expressions/second
+- Current: ~6,093K expressions/second
+- **Total improvement: ~313%** (4.1x faster than original)
+
+## 18. Added Optional Trivia/Channel Support
+
+### What was changed:
+- Added `Channel` enum with `REGULAR` and `HIDDEN` values
+- Added optional `channel` property to Token interface
+- Added `preserveTrivia` option to LexerOptions
+- Modified whitespace and comment tokens to include `Channel.HIDDEN` when `preserveTrivia` is true
+- Updated `nextToken` to return trivia tokens when `preserveTrivia` is enabled
+
+### Implementation:
+```typescript
+export enum Channel {
+  REGULAR = 0,
+  HIDDEN = 1,
+}
+
+export interface Token {
+  // ... existing properties
+  channel?: Channel;
+}
+
+export interface LexerOptions {
+  skipWhitespace?: boolean;
+  skipComments?: boolean;
+  preserveTrivia?: boolean;  // New option
+}
+```
+
+### Performance Impact:
+- **Without preserveTrivia**: 2,730,350 expressions/second (default)
+- **With preserveTrivia**: 1,737,645 expressions/second
+- **Overhead**: 36.4% slower when enabled
+- Channel assignment only happens when explicitly requested
+- No performance penalty for users who don't need trivia
+
+The overhead is expected because:
+1. Whitespace and comment tokens are returned instead of skipped
+2. Each trivia token requires channel assignment
+3. More tokens are created and added to the result array
+4. Typical expressions have 30-50% trivia tokens
+
+### Why it's important:
+1. **Code Formatters** - Need to preserve whitespace and comments
+2. **Refactoring Tools** - Must maintain original formatting
+3. **Documentation Generators** - Extract comments for API docs
+4. **Round-trip Parsing** - Parse → Modify → Serialize preserves formatting
+5. **Compatibility** - Matches original lexer's channel support
+
+### Usage:
+```typescript
+// Preserve all trivia
+const lexer = new Lexer(code, { preserveTrivia: true });
+const tokens = lexer.tokenize();
+
+// Filter by channel
+const regularTokens = tokens.filter(t => t.channel !== Channel.HIDDEN);
+const triviaTokens = tokens.filter(t => t.channel === Channel.HIDDEN);
+```
+
+### Current Performance Summary:
+- Original: ~1,477K expressions/second
+- Current: ~2,730K expressions/second (real-world expressions)
+- Current: ~6,093K expressions/second (simple benchmark)
+- **Total improvement: ~85%** (real-world) to **313%** (simple)
+- With trivia enabled: ~1,738K expressions/second (36.4% overhead)
+
+## 19. Performance Comparison with ANTLR
+
+### What was tested:
+- Compared hand-optimized Lexer2 with ANTLR-generated lexer
+- Used same test suite: 1,539 real-world FHIRPath expressions
+- 10,000 iterations per expression (~15.4 million tokens)
+- ANTLR lexer generated from official FHIRPath grammar (spec/fhirpath.g4)
+
+### Performance Results:
+- **Lexer2**: 2,820,268 expressions/second
+- **ANTLR**: 725,684 expressions/second
+- **Performance advantage: 3.89x faster**
+
+### Why Lexer2 Outperforms ANTLR:
+
+#### Lexer2 Optimizations:
+1. **Lookup tables** - O(1) character classification
+2. **CharCode dispatch** - Integer switches instead of string comparisons
+3. **Inlined hot paths** - Reduced function call overhead
+4. **Optimized keywords** - Switch by length then value
+5. **Minimal allocations** - Simple object literals
+
+#### ANTLR Overhead:
+1. **Generic DFA/NFA** - Flexible but less optimized
+2. **Heavy tokens** - Complex Token class with many properties
+3. **Stream abstractions** - Multiple indirection layers
+4. **Error recovery** - Built-in but adds overhead
+5. **Memory usage** - Larger token objects and buffering
+
+### Trade-offs:
+- **Lexer2**: Maximum performance, requires manual maintenance
+- **ANTLR**: Grammar-based, easier to modify, good tooling
+
+### Conclusion:
+The hand-optimized approach provides substantial performance benefits (3.89x) for performance-critical applications like FHIRPath evaluation. ANTLR remains valuable for prototyping and when 725K expr/sec is sufficient.
+
+### Final Performance Summary:
+- **vs Original implementation**: ~85% improvement (real-world)
+- **vs ANTLR-generated lexer**: ~289% improvement (3.89x faster)
+- **Absolute performance**: 2.82M expressions/second
+
+### Remaining Optimization Opportunities:
+1. **Optimize readDateTime/readTimeFormat** - Reduce redundant charCode lookups
+2. **Consider selective inlining** - Apply only where measurable benefit exists
+3. **Profile-guided optimization** - Focus on actual hot paths in real usage
+4. **Add offset property** - For complete position tracking compatibility