@atomic-ehr/fhirpath 0.0.1-canary.0c6931e.20250727185306

package/src/parser2/optimization-summary.md

@@ -0,0 +1,176 @@
+# Parser2 Performance Optimization Summary
+
+## Baseline Performance (2024-01-27)
+
+### Test Configuration
+- **Test Suite**: 1,539 FHIRPath expressions from real-world invariants and search parameters
+- **Iterations**: 5,000 per expression (7,695,000 total parser runs)
+- **Hardware**: Bun v1.2.18 runtime
+- **Parser**: Recursive-descent with Pratt operator precedence parsing
+
+### Baseline Metrics
+
+| Metric | Value |
+|--------|-------|
+| **Total time** | 6.51 seconds |
+| **Time per expression** | 0.0008ms (0.8 microseconds) |
+| **Expressions per second** | 1,182,443 |
+| **Average tokens per expression** | 11.2 |
+| **Average AST nodes per expression** | 8.6 |
+| **Time per token** | ~0.08 microseconds |
+
+### Performance by Expression Complexity
+
+| Token Range | Count | Avg Time (ms) | Time/Token (μs) |
+|-------------|-------|---------------|-----------------|
+| 1-5 tokens | 716 | 0.0003 | 0.08 |
+| 6-10 tokens | 410 | 0.0005 | 0.08 |
+| 11-20 tokens | 251 | 0.0010 | 0.07 |
+| 21-50 tokens | 130 | 0.0024 | 0.08 |
+| 50+ tokens | 32 | 0.0093 | 0.08 |
+
+### Key Observations
+
+1. **Linear Scaling**: Performance scales linearly with token count, indicating O(n) complexity
+2. **Consistent Token Processing**: ~0.08μs per token across all complexity ranges
+3. **Fast Baseline**: Already processing over 1 million expressions per second
+4. **Memory Efficiency**: All tokens are pre-loaded, enabling random access without re-parsing
+
+### Slowest Expressions
+
+The most complex expression has 780 tokens and takes 0.0589ms to parse:
+```
+(ActivityDefinition.useContext.value...
+```
+
+This represents a worst-case scenario that's still processed in under 0.06ms.
+
+## Design Strengths
+
+1. **Pratt Parsing**: Eliminates recursive precedence climbing, reducing call stack depth
+2. **Pre-tokenization**: All tokens generated upfront, avoiding interleaved lexing/parsing
+3. **Direct Token Access**: Array-based token storage enables O(1) lookahead
+4. **Minimal Allocations**: AST nodes created only when needed
+5. **Type-safe Node Creation**: Separate factory methods avoid dynamic object construction
+
+## Comparison with Lexer2
+
+| Component | Expressions/sec | Relative Speed |
+|-----------|----------------|----------------|
+| Lexer2 | ~15,000,000 | 1.0x (baseline) |
+| Parser2 | ~1,200,000 | 0.08x |
+
+The parser is ~12x slower than the lexer, which is expected since it:
+- Builds a complete AST structure
+- Performs syntax validation
+- Handles operator precedence
+- Manages recursive structures
+
+## Potential Optimizations
+
+While the baseline performance is already excellent, potential optimizations include:
+
+1. **Token Caching**: Cache frequently accessed token values to avoid repeated string operations
+2. **Specialized Node Pools**: Pre-allocate common node types to reduce allocation overhead
+3. **Inline Small Functions**: Further inline hot-path functions like `peek()` and `check()`
+4. **Precedence Table**: Replace switch statement with lookup table for operator precedence
+5. **String Interning**: Intern common identifier strings to reduce memory and comparison costs
+
+## ANTLR Parser Comparison
+
+To provide additional context, we compared parser2 performance against the ANTLR-generated parser:
+
+### Overall Performance Comparison
+
+| Parser | Expressions/sec | Time per Expression | Relative Performance |
+|--------|----------------|-------------------|---------------------|
+| ANTLR | 143,965 | 0.0069ms | 1.0x (baseline) |
+| Parser2 | 1,030,046 | 0.0010ms | **7.2x faster** |
+
+### Complex Expression Performance
+
+| Expression | ANTLR (ms) | Parser2 (ms) | Speedup |
+|------------|------------|--------------|---------|
+| Patient.name.where(use = 'official').given | 0.009 | 0.001 | 7.7x |
+| (ActivityDefinition.useContext.value... | 0.009 | 0.001 | 7.1x |
+| ($this is Range) implies ((low.empty()... | 0.027 | 0.002 | 11.2x |
+| extension.exists() or (contentType.co... | 0.047 | 0.001 | 33.1x |
+
+
+
+### Key Differences
+
+1. **Architecture**: ANTLR uses table-driven parsing while parser2 uses recursive descent with Pratt parsing
+2. **Overhead**: ANTLR has more runtime overhead due to its generic parsing framework
+3. **Scalability**: Parser2 shows better performance on complex expressions (up to 33x faster)
+4. **Error Recovery**: ANTLR provides better error recovery but at a performance cost
+
+## Optimization: Bit-Packed Precedence (2024-01-27)
+
+### Implementation
+Encoded operator precedence directly into TokenType enum values using bit-packing:
+- High byte: Precedence (0-255)
+- Low byte: Token ID (0-255)
+
+### Results
+| Metric | Before | After | Improvement |
+|--------|--------|-------|-------------|
+| Expressions/sec | 1,182,443 | 1,214,209 | +2.7% |
+| Time per expression | 0.8μs | 0.8μs | - |
+| getPrecedence cycles | ~5-10 | ~1 | -90% |
+
+### Code Simplification
+- Removed 30+ line switch statement
+- Replaced with single bit shift: `type >>> 8`
+- Precedence now co-located with token definitions
+
+### Note
+A direct precedence value approach (without bit-packing) was also tested but resulted in 8.8% performance degradation due to lookup table overhead. The bit-packed approach with bit shift operation provides the best balance of performance and maintainability.
+
+## Additional Optimizations (2024-01-27)
+
+### 1. Method Inlining
+Inlined hot path methods (`peek()`, `advance()`, `isAtEnd()`) to reduce function call overhead in `parseExpressionWithPrecedence()` and `parsePrimary()`.
+
+**Results**: 
+- Baseline: 1,203,640 expressions/sec
+- After inlining: 1,223,027 expressions/sec (+1.6%)
+
+### 2. Node Creation Inlining (Reverted)
+Attempted to inline frequently used node creation functions (`createBinaryNode`, `createLiteralNode`, `createIdentifierNode`) but observed performance regression due to increased code size affecting CPU cache and JIT optimization.
+
+**Results**: Performance decreased to 1,184,463 expressions/sec - optimization was reverted.
+
+### 3. Lookup Tables for Token Type Checking (Abandoned)
+Attempted to use Uint8Array lookup tables for binary operator and keyword checking, but abandoned due to bit-packed token values exceeding array bounds (values > 256).
+
+### 4. Position Tracking Impact Analysis
+Temporarily disabled position tracking to measure its performance cost.
+
+**Results**:
+- With position tracking: 1,226,319 expressions/sec
+- Without position tracking: 1,212,703 expressions/sec (-1.1%)
+
+**Surprising finding**: Removing position tracking actually decreased performance slightly, likely due to:
+- Position tracking overhead is minimal (simple object property assignment)
+- Code changes disrupted JIT compiler optimizations
+- Modern JS engines optimize object creation very well
+
+### Key Findings:
+- Simple method inlining in hot paths provides consistent gains
+- Over-optimization (excessive inlining) can hurt performance
+- JIT compiler already optimizes method calls effectively
+- Position tracking has negligible performance impact
+- Bit-packed TokenType with getPrecedence bit shift remains the best optimization
+- Current optimized performance: **1,223,027 expressions/sec** (1.6% improvement)
+
+## Conclusion
+
+The parser2 performance is exceptional:
+- **7.2x faster** than the ANTLR-generated parser
+- Processing over **1.2 million expressions per second**
+- Consistent **linear O(n) scaling** with token count
+- Sub-microsecond parsing times for typical expressions
+
+
+The combination of recursive-descent parsing with Pratt operator precedence, enhanced with bit-packed precedence lookup, provides an optimal balance of simplicity, maintainability, and performance. Even with already excellent baseline performance, targeted optimizations like bit-packed precedence can provide measurable improvements while simplifying the codebase.

package/src/registry/default-analyzers.ts

@@ -0,0 +1,257 @@
+import type { 
+  Analyzer, 
+  TypeInfo, 
+  TypeRef,
+  Operator, 
+  Function, 
+  Literal,
+  TypeConstraint,
+  TypeInferenceRule,
+  CardinalityInferenceRule
+} from './types';
+import type { ModelProvider } from '../analyzer/types';
+
+// Helper to check if type matches constraint
+export function matchesConstraint(type: TypeRef, constraint: TypeConstraint): boolean {
+  if (constraint.kind === 'any') return true;
+  
+  if (!constraint.types) return true;
+  
+  // For now, simple type name matching
+  // TODO: Handle inheritance and complex types
+  let typeName = typeof type === 'string' ? type : (type as any).type || (type as any).name || 'unknown';
+  
+  // Normalize case - constraint types are capitalized, but actual types might be lowercase
+  const normalizedTypeName = typeName.charAt(0).toUpperCase() + typeName.slice(1);
+  
+  return constraint.types.includes(normalizedTypeName) || constraint.types.includes(typeName);
+}
+
+// Helper to format constraint for error messages
+export function formatConstraint(constraint: TypeConstraint): string {
+  if (constraint.kind === 'any') return 'any type';
+  if (!constraint.types || constraint.types.length === 0) return 'any type';
+  if (constraint.types.length === 1) return constraint.types[0]!;
+  return constraint.types.join(' or ');
+}
+
+// Helper to resolve type inference rule
+export function resolveTypeInferenceRule(
+  rule: TypeInferenceRule,
+  input: TypeRef,
+  args: TypeRef[],
+  analyzer: Analyzer
+): TypeRef {
+  if (typeof rule === 'string') {
+    if (rule === 'preserve-input') return input;
+    if (rule === 'promote-numeric') {
+      // Check if we have numeric types
+      const types = [input, ...args].map(t => {
+        const rawType = typeof t === 'string' ? t : (t as any).type || (t as any).name || 'unknown';
+        // Normalize to uppercase first letter
+        return rawType.charAt(0).toUpperCase() + rawType.slice(1);
+      });
+      if (types.includes('Decimal')) return analyzer.resolveType('Decimal');
+      if (types.includes('Integer')) return analyzer.resolveType('Integer');
+      return input; // fallback
+    }
+    return analyzer.resolveType(rule); // Fixed type name - resolve through analyzer
+  }
+  
+  if (typeof rule === 'function') {
+    // TODO: We need to pass ModelProvider to the rule function
+    // For now, just return input
+    return input;
+  }
+  
+  return input; // fallback
+}
+
+// Helper to resolve cardinality inference rule
+export function resolveCardinalityInferenceRule(
+  rule: CardinalityInferenceRule,
+  inputIsSingleton: boolean,
+  argsAreSingleton: boolean[]
+): boolean {
+  if (rule === 'singleton') return true;
+  if (rule === 'collection') return false;
+  if (rule === 'preserve-input') return inputIsSingleton;
+  if (rule === 'all-singleton') return inputIsSingleton && argsAreSingleton.every(s => s);
+  
+  if (typeof rule === 'function') {
+    return rule(inputIsSingleton, argsAreSingleton);
+  }
+  
+  return inputIsSingleton; // fallback
+}
+
+// Default analyzer for operators
+export function defaultOperatorAnalyze(
+  this: Operator,
+  analyzer: Analyzer, 
+  input: TypeInfo, 
+  args: TypeInfo[]
+): TypeInfo {
+  const { signature } = this;
+  
+  // Validate parameter count
+  const expectedParams = signature.parameters.length;
+  if (args.length !== expectedParams) {
+    analyzer.error(`Operator '${this.name}' expects ${expectedParams} parameter(s) but got ${args.length}`);
+  }
+  
+  // Validate parameter types and cardinality
+  signature.parameters.forEach((param, i) => {
+    const arg = args[i];
+    if (!arg) return;
+    
+    // Check type constraint
+    if (param.types && !matchesConstraint(arg.type, param.types)) {
+      const argTypeName = typeof arg.type === 'string' 
+        ? arg.type 
+        : (arg.type as any).type || (arg.type as any).name || 'unknown';
+      analyzer.error(
+        `Operator '${this.name}' parameter '${param.name}' expects ${formatConstraint(param.types)} but got ${argTypeName}`
+      );
+    }
+    
+    // Check cardinality constraint
+    if (param.cardinality) {
+      if (param.cardinality === 'singleton' && !arg.isSingleton) {
+        analyzer.error(
+          `Operator '${this.name}' parameter '${param.name}' requires singleton value but got collection`
+        );
+      } else if (param.cardinality === 'collection' && arg.isSingleton) {
+        analyzer.error(
+          `Operator '${this.name}' parameter '${param.name}' requires collection but got singleton`
+        );
+      }
+      // 'any' accepts both
+    }
+  });
+  
+  // Determine output type
+  const outputType = resolveTypeInferenceRule(
+    signature.output.type,
+    input.type,
+    args.map(a => a.type),
+    analyzer
+  );
+  
+  // Determine output cardinality
+  const isSingleton = resolveCardinalityInferenceRule(
+    signature.output.cardinality,
+    input.isSingleton,
+    args.map(a => a.isSingleton)
+  );
+  
+  return { type: outputType, isSingleton };
+}
+
+// Default analyzer for functions
+export function defaultFunctionAnalyze(
+  this: Function,
+  analyzer: Analyzer, 
+  input: TypeInfo, 
+  args: TypeInfo[]
+): TypeInfo {
+  const { signature } = this;
+  
+  // Check input constraints if specified
+  if (signature.input) {
+    if (signature.input.types && !matchesConstraint(input.type, signature.input.types)) {
+      const inputTypeName = typeof input.type === 'string' 
+        ? input.type 
+        : (input.type as any).type || (input.type as any).name || 'unknown';
+      analyzer.error(
+        `Function '${this.name}' expects input of type ${formatConstraint(signature.input.types)} but got ${inputTypeName}`
+      );
+    }
+    
+    if (signature.input.cardinality) {
+      if (signature.input.cardinality === 'singleton' && !input.isSingleton) {
+        analyzer.error(`Function '${this.name}' requires singleton input but got collection`);
+      } else if (signature.input.cardinality === 'collection' && input.isSingleton) {
+        analyzer.error(`Function '${this.name}' requires collection input but got singleton`);
+      }
+    }
+  }
+  
+  // Validate parameters
+  const requiredParams = signature.parameters.filter(p => !p.optional);
+  if (args.length < requiredParams.length) {
+    analyzer.error(
+      `Function '${this.name}' requires at least ${requiredParams.length} parameter(s) but got ${args.length}`
+    );
+  }
+  
+  if (args.length > signature.parameters.length) {
+    analyzer.error(
+      `Function '${this.name}' expects at most ${signature.parameters.length} parameter(s) but got ${args.length}`
+    );
+  }
+  
+  // Check each parameter
+  signature.parameters.forEach((param, i) => {
+    const arg = args[i];
+    if (!arg && !param.optional) {
+      analyzer.error(`Function '${this.name}' parameter '${param.name}' is required`);
+      return;
+    }
+    if (!arg) return; // Optional parameter not provided
+    
+    // Check type constraint
+    if (param.types && !matchesConstraint(arg.type, param.types)) {
+      const argTypeName = typeof arg.type === 'string' 
+        ? arg.type 
+        : (arg.type as any).type || (arg.type as any).name || 'unknown';
+      analyzer.error(
+        `Function '${this.name}' parameter '${param.name}' expects ${formatConstraint(param.types)} but got ${argTypeName}`
+      );
+    }
+    
+    // Check cardinality constraint
+    if (param.cardinality) {
+      if (param.cardinality === 'singleton' && !arg.isSingleton) {
+        analyzer.error(
+          `Function '${this.name}' parameter '${param.name}' requires singleton value but got collection`
+        );
+      } else if (param.cardinality === 'collection' && arg.isSingleton) {
+        analyzer.error(
+          `Function '${this.name}' parameter '${param.name}' requires collection but got singleton`
+        );
+      }
+    }
+  });
+  
+  // Determine output type
+  const outputType = resolveTypeInferenceRule(
+    signature.output.type,
+    input.type,
+    args.map(a => a.type),
+    analyzer
+  );
+  
+  // Determine output cardinality
+  const isSingleton = resolveCardinalityInferenceRule(
+    signature.output.cardinality,
+    input.isSingleton,
+    args.map(a => a.isSingleton)
+  );
+  
+  return { type: outputType, isSingleton };
+}
+
+// Default analyzer for literals
+export function defaultLiteralAnalyze(
+  this: Literal,
+  analyzer: Analyzer, 
+  input: TypeInfo, 
+  args: TypeInfo[]
+): TypeInfo {
+  // Literals have fixed output type and cardinality
+  return {
+    type: analyzer.resolveType(this.signature.output.type),
+    isSingleton: true
+  };
+}

package/src/registry/default-compilers.ts

@@ -0,0 +1,31 @@
+import type { Compiler, CompiledExpression } from './types';
+
+export function defaultFunctionCompile(
+  compiler: Compiler,
+  input: CompiledExpression,
+  args: CompiledExpression[]
+): CompiledExpression {
+  // Default implementation returns a simple pass-through
+  return {
+    fn: (ctx) => {
+      throw new Error('Function compile not implemented');
+    },
+    type: compiler.resolveType('Any'),
+    isSingleton: false
+  };
+}
+
+export function defaultOperatorCompile(
+  compiler: Compiler,
+  left: CompiledExpression,
+  right: CompiledExpression[]
+): CompiledExpression {
+  // Default implementation returns a simple pass-through
+  return {
+    fn: (ctx) => {
+      throw new Error('Operator compile not implemented');
+    },
+    type: compiler.resolveType('Any'),
+    isSingleton: false
+  };
+}

package/src/registry/index.ts

@@ -0,0 +1,96 @@
+import { Registry } from './registry';
+import { arithmeticOperators } from './operations/arithmetic';
+import { logicalOperators, logicalFunctions } from './operations/logical';
+import { comparisonOperators } from './operations/comparison';
+import { membershipOperators } from './operations/membership';
+import { typeOperators } from './operations/type-operators';
+import { literals } from './operations/literals';
+import { existenceFunctions } from './operations/existence';
+import { navigationOperators } from './operations/navigation';
+import { 
+  aggregateFunction, 
+  childrenFunction, 
+  descendantsFunction, 
+  iifFunction, 
+  defineVariableFunction, 
+  traceFunction, 
+  checkFunction 
+} from './operations/utility';
+import { typeFunction, isFunction, asFunction } from './operations/type-checking';
+import { absFunction, roundFunction, sqrtFunction } from './operations/math';
+import { whereFunction, selectFunction, ofTypeFunction, repeatFunction } from './operations/filtering';
+import { 
+  containsFunction, lengthFunction, substringFunction, startsWithFunction, 
+  endsWithFunction, upperFunction, lowerFunction, indexOfFunction,
+  replaceFunction, splitFunction, joinFunction, trimFunction, toCharsFunction
+} from './operations/string';
+import { 
+  toStringFunction, toIntegerFunction, toDecimalFunction, 
+  toBooleanFunction, toQuantityFunction 
+} from './operations/type-conversion';
+import { tailFunction, skipFunction, takeFunction } from './operations/subsetting';
+import { unionFunction, combineFunction, intersectFunction, excludeFunction, collectionOperators } from './operations/collection';
+
+// Export types
+export * from './types';
+export { Registry } from './registry';
+export * from './default-analyzers';
+
+// Register all operations on module load
+[
+  ...arithmeticOperators,
+  ...logicalOperators,
+  ...logicalFunctions,
+  ...comparisonOperators,
+  ...membershipOperators,
+  ...typeOperators,
+  ...collectionOperators,
+  ...navigationOperators,
+  ...literals,
+  ...existenceFunctions,
+  aggregateFunction,
+  childrenFunction,
+  descendantsFunction,
+  iifFunction,
+  defineVariableFunction,
+  traceFunction,
+  checkFunction,
+  typeFunction,
+  isFunction,
+  asFunction,
+  absFunction,
+  roundFunction,
+  sqrtFunction,
+  whereFunction,
+  selectFunction,
+  ofTypeFunction,
+  repeatFunction,
+  containsFunction,
+  lengthFunction,
+  substringFunction,
+  startsWithFunction,
+  endsWithFunction,
+  upperFunction,
+  lowerFunction,
+  indexOfFunction,
+  replaceFunction,
+  splitFunction,
+  joinFunction,
+  trimFunction,
+  toCharsFunction,
+  toStringFunction,
+  toIntegerFunction,
+  toDecimalFunction,
+  toBooleanFunction,
+  toQuantityFunction,
+  tailFunction,
+  skipFunction,
+  takeFunction,
+  unionFunction,
+  combineFunction,
+  intersectFunction,
+  excludeFunction
+].forEach(op => Registry.register(op));
+
+// Export default registry instance
+export default Registry;