@atomic-ehr/fhirpath 0.0.1-canary.0c6931e.20250727185306

package/src/lexer2/notes.md

@@ -0,0 +1,173 @@
+# with position tracking
+
+- Total expressions tested: 1539
+- Total iterations: 1,539,000
+- Total time: 2.18s
+- Time per expression: 0.0014ms
+- Throughput: ~707,073 expressions per second
+
+
+# without position tracking
+
+- Total expressions tested: 1539
+- Total iterations: 1,539,000
+- Total time: 1.96s
+- Time per expression: 0.0013ms
+- Throughput: ~783,321 expressions per second
+
+  Performance improvement:
+  - ~14.7% faster overall
+  - ~0.0002ms faster per expression
+  - ~103,102 more expressions per second
+
+# without value building
+
+- Total expressions: 1539
+- Total iterations: 1539000
+- Total time: 1.91s
+- Time per expression: 0.0012ms
+- Expressions per second: 807382
+
+Before (without position tracking but with value strings):
+
+- Time: ~1.94s
+- Throughput: ~793,647 expressions/second
+
+After (with position tracking, no value strings):
+
+- Time: ~1.88s
+- Throughput: ~817,240 expressions/second
+
+Performance improvement: ~3% faster
+
+The improvement is modest because:
+1. We still need to traverse the input character by character for escape sequences
+2. The main benefit is reduced memory allocation, not raw speed
+3. Further optimizations (like character code comparisons instead of regex) would yield bigger gains
+
+
+# without regex 
+
+Total expressions: 1539
+Total iterations: 1539000
+Total time: 1.45s
+Time per expression: 0.0009ms
+Expressions per second: 1061465
+
+1. Initial (with position tracking & value strings):
+- ~705,096 expressions/sec
+2. Without position tracking:
+- ~793,647 expressions/sec (+12.6%)
+3. With position ranges instead of value strings:
+- ~817,240 expressions/sec (+15.9%)
+4. With character code comparisons instead of regex:
+- ~1,062,000 expressions/sec (+50.6%!)
+
+
+## Array optimization ( does not help )
+The results show that:
+  1. String.charCodeAt() is the fastest - about 3% faster than charAt
+  2. Char arrays are actually slower - about 2.7% slower than string access
+  3. Uint16Array is also slower - about 2.4% slower
+
+  This is because:
+  - Modern JavaScript engines heavily optimize string operations
+  - Creating an array adds memory allocation overhead
+  - Array access isn't faster than optimized string access
+  - We lose the benefit of string interning and other V8 optimizations
+
+
+# First-character dispatch
+
+Optimization Progression:
+
+  1. Initial (with position tracking & value strings):
+    - ~705,096 expressions/sec
+  2. Without position tracking:
+    - ~793,647 expressions/sec (+12.6%)
+  3. With position ranges instead of value strings:
+    - ~817,240 expressions/sec (+15.9%)
+  4. With character code comparisons:
+    - ~1,062,000 expressions/sec (+50.6%)
+  5. With switch-based token dispatch:
+    - ~1,363,000 expressions/sec (+93.4%!)
+
+  Latest Optimization Impact:
+
+  - Reduced time from ~1.45s to ~1.13s
+  - Increased throughput by ~300,000 expressions/second
+  - Time per expression dropped from 0.0009ms to 0.0007ms
+
+  Total Performance Improvement: ~93% faster than the original
+
+  The switch-based dispatch provided another significant performance boost (~28% over the previous version) because:
+  1. Direct jumps to the right token handler based on first character
+  2. No sequential checking of multiple token types
+  3. Better CPU branch prediction with switch statements
+  4. Eliminated unnecessary function calls for tokens that wouldn't match
+
+## switch
+
+Total expressions: 1539
+Total iterations: 1539000
+Total time: 1.12s
+Time per expression: 0.0007ms
+Expressions per second: 1375165
+
+
+## Inline-hot functions
+
+Optimization Progression:
+
+  1. Initial: ~705,096 expressions/sec
+  2. Without position tracking: ~793,647 expressions/sec (+12.6%)
+  3. Position ranges instead of values: ~817,240 expressions/sec (+15.9%)
+  4. Character code comparisons: ~1,062,000 expressions/sec (+50.6%)
+  5. Switch-based dispatch: ~1,363,000 expressions/sec (+93.4%)
+  6. Inlined hot functions: ~1,477,000 expressions/sec (+109.5%!)
+
+  Latest Optimization Impact:
+
+  - Reduced time from ~1.13s to ~1.04s
+  - Increased throughput by 114,000 expressions/second (8.4% improvement)
+  - Time per expression remains at 0.0007ms
+
+  Total Performance Improvement: ~109% faster (more than 2x the original speed!)
+
+  The function inlining provided a solid ~8% performance boost by:
+  1. Eliminating function call overhead in tight loops
+  2. Better CPU pipeline efficiency with fewer jumps
+  3. Reduced stack operations
+  4. Improved locality of reference
+
+  The lexer is now processing nearly 1.5 million FHIRPath expressions per second!
+
+  Remaining High-Impact Optimizations:
+
+  1. Lookup tables for character classification (could add 5-10%)
+  2. Optimize readSpecialIdentifier substring (could add 2-3%)
+  3. Perfect hash for keywords (could add 3-5%)
+  4. Switch on character codes instead of chars (could add 2-3%)
+
+  We've achieved excellent performance with clean, maintainable code!
+
+  ## Lookup table optimization
+
+The lookup tables optimization has been successfully implemented, improving performance by ~4.7% to reach ~1,546K expressions per second. All tests pass and the lexer maintains full compatibility.
+
+Total expressions: 1539
+Total iterations: 1539000
+Total time: 0.98s
+Time per expression: 0.0006ms
+Expressions per second: 1569801
+
+## Switch based keyword lookup
+
+The switch-based keyword lookup was incredibly effective! We've achieved a 42% performance improvement with this optimization alone. 
+The total improvement from all optimizations is now from ~1,477K to ~2,192K expressions per second - a 48% overall improvement.
+
+Total expressions: 1539
+Total iterations: 1539000
+Total time: 0.71s
+Time per expression: 0.0005ms
+Expressions per second: 2160611