kumi 0.0.14 → 0.0.16

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec64db5a7b14df1a8656e2cedc569153a0f092d69a99544c5aa103241f61d931
-  data.tar.gz: 84bd4c7360eb74364e47103901fe8ccbcf6282a9cb40111d42b05357a2172c9e
+  metadata.gz: 9f51be8774c472629e599ce3edd4ab02551edfd52fea8676e2ee93eed5198800
+  data.tar.gz: 26532cec84e5c59553031dc86d82abff310d2a79c857776e1d793af2b35e00ea
 SHA512:
-  metadata.gz: da7a9f3135e63779676ea5a802f48288ff2675935ab8dd8dcc751aaf309392275589f08fb92fede4dce351707369b5113e28540e5f4853629a02f4a1945ca75a
-  data.tar.gz: 5a45952056f81d9b4551bc7b48d3fd35c25065eb058e3cd6737879377a5c6e5f61d1cbba5b51955f54ad6ca14a797254e3381c69cd6b9eb145f3929e6e962dae
+  metadata.gz: 3ba28da3acbf5cd430902c29e34e5786562e7121f9c3851a2e9db3a04a13fc8b1a5e9729649f3daadb523ea55f2c97a1e4560139b9809aa636eff21e9716abbf
+  data.tar.gz: 6683358d25786a5e15cba975e785b3178ec0ea13f40a3eeb940e72f25886f5004c499f3e80ddded31fdd23f3981cbe5244db1ab0aec1405b3c52acb68c4b1e60

data/CHANGELOG.md

@@ -1,5 +1,45 @@
 ## [Unreleased]
 
+## [0.0.16] – 2025-08-22
+
+### Performance
+- Input accessor code generation replaces nested lambda chains with compiled Ruby methods
+- Fix cache handling in Runtime - it was being recreated on updates
+- Add early shortcut for Analyzer Passes.
+
+## [0.0.15] – 2025-08-21
+### Added
+- (DX) Schema-aware VM profiling with multi-schema performance analysis
+- DAG-based execution optimization with pre-computed dependency resolution
+
+### Performance
+- Reference operations eliminated as VM bottleneck via O(1) hash lookups
+
+## [0.0.14] – 2025-08-21
+### Added
+- Text schema frontend with `.kumi` file format support
+- `bin/kumi parse` command for schema analysis and golden file testing
+- LoadInputCSE optimization pass to eliminate redundant load operations
+- Runtime accessor caching with precise field-based invalidation
+- VM profiler with wall time, CPU time, and cache hit rate analysis
+- Structured analyzer debug system with state inspection
+- Checkpoint system for capturing and comparing analyzer states
+- State serialization (StateSerde) for golden testing and regression detection
+- Debug object printers with configurable truncation
+- Multi-run averaging for stable performance benchmarking
+
+### Fixed
+- VM targeting for `__vec` twin declarations that were failing to resolve
+- Demand-driven reference resolution with proper name indexing and cycle detection
+- Accessor cache invalidation now uses precise field dependencies instead of clearing all caches
+- StateSerde JSON serialization issues with frozen hashes, Sets, and Symbols
+
+### Performance
+- 14x improvement on update-heavy workloads (1.88k → 26.88k iterations/second)
+- 30-40% reduction in IR module size for schemas with repeated field access
+- Eliminated load_input performance bottleneck that was consuming ~99% of execution time
+- Optional caching system (enabled via KUMI_VM_CACHE=1) for performance-critical scenarios
+
 ## [0.0.13] – 2025-08-14
 ### Added
 - Runtime performance optimizations for interpreter execution

data/README.md

@@ -207,33 +207,6 @@ end
 # ❌ Function arity error: divide expects 2 arguments, got 1
 ```
 
-**Mutual Recursion**: Kumi supports mutual recursion when cascade conditions are mutually exclusive:
-
-```ruby
-trait :is_forward, input.operation == "forward"
-trait :is_reverse, input.operation == "reverse"
-
-# Safe mutual recursion - conditions are mutually exclusive
-value :forward_processor do
-  on is_forward, input.value * 2        # Direct calculation
-  on is_reverse, reveAnalysisrse_processor + 10  # Delegates to reverse (safe)
-  base "invalid operation"
-end
-
-value :reverse_processor do
-  on is_forward, forward_processor - 5   # Delegates to forward (safe) 
-  on is_reverse, input.value / 2         # Direct calculation
-  base "invalid operation"
-end
-
-# Usage examples:
-# operation="forward", value=10  => forward: 20, reverse: 15
-# operation="reverse", value=10  => forward: 15, reverse: 5  
-# operation="unknown", value=10  => both: "invalid operation"
-```
-
-This compiles because `operation` can only be "forward" or "reverse", never both. Each recursion executes one step before hitting a direct calculation.
-
 #### **Runtime Introspection: Debug and Understand**
 
 **Explainability**: Trace exactly how any value is computed, step-by-step. This is invaluable for debugging complex logic and auditing results.

data/docs/dev/vm-profiling.md

@@ -0,0 +1,95 @@
+# VM Profiling with Schema Differentiation
+
+## Overview
+
+Profiles VM operation execution with schema-level differentiation. Tracks operations by schema type for multi-schema performance analysis.
+
+## Core Components
+
+**Profiler**: `lib/kumi/core/ir/execution_engine/profiler.rb`
+- Streams VM operation events with schema identification
+- Supports persistent mode for cross-run analysis
+- JSONL event format with operation metadata
+
+**Profile Aggregator**: `lib/kumi/dev/profile_aggregator.rb`  
+- Analyzes profiling data by schema type
+- Generates summary and detailed performance reports
+- Schema breakdown showing operations and timing per schema
+
+**CLI Integration**: `bin/kumi profile`
+- Processes JSONL profiling data files
+- Multiple output formats: summary, detailed, raw
+
+## Usage
+
+### Basic Profiling
+
+```bash
+# Single schema with operations
+KUMI_PROFILE=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+
+# Persistent mode across multiple runs
+KUMI_PROFILE=1 KUMI_PROFILE_PERSISTENT=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+
+# Streaming mode for real-time analysis  
+KUMI_PROFILE=1 KUMI_PROFILE_STREAM=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+```
+
+### CLI Analysis
+
+```bash
+# Summary report with schema breakdown
+kumi profile profile.jsonl --summary
+
+# Detailed per-operation analysis
+kumi profile profile.jsonl --detailed
+
+# Raw event stream
+kumi profile profile.jsonl --raw
+```
+
+## Environment Variables
+
+**Core**:
+- `KUMI_PROFILE=1` - Enable profiling
+- `KUMI_PROFILE_FILE=path` - Output file (required)
+- `KUMI_PROFILE_OPS=1` - Enable VM operation profiling
+
+**Modes**:
+- `KUMI_PROFILE_PERSISTENT=1` - Append to existing files across runs
+- `KUMI_PROFILE_STREAM=1` - Stream individual events vs batch
+- `KUMI_PROFILE_TRUNCATE=1` - Truncate existing files
+
+## Event Format
+
+JSONL with operation metadata:
+
+```json
+{"event":"vm_operation","schema":"TestSchema","operation":"LoadInput","duration_ms":0.001,"timestamp":"2025-01-20T10:30:45.123Z"}
+{"event":"vm_operation","schema":"TestSchema","operation":"Map","duration_ms":0.002,"timestamp":"2025-01-20T10:30:45.125Z"}
+```
+
+## Schema Differentiation
+
+Tracks operations by schema class name for multi-schema analysis:
+
+**Implementation**:
+- Schema name propagated through compilation pipeline
+- Profiler tags each VM operation with schema identifier  
+- Aggregator groups operations by schema type
+
+**Output Example**:
+```
+Total operations: 24 (0.8746ms)
+Schemas analyzed: SchemaA, SchemaB
+  SchemaA: 12 operations, 0.3242ms
+  SchemaB: 12 operations, 0.0504ms
+```
+
+## Performance Analysis
+
+**Reference Operations**: Typically dominate execution time in complex schemas
+**Map Operations**: Element-wise computations on arrays
+**LoadInput Operations**: Data access operations
+
+Use schema breakdown to identify performance differences between schema types.

data/docs/features/README.md

@@ -9,13 +9,6 @@ Analyzes rule combinations to detect logical impossibilities across dependency c
 - Validates domain constraints
 - Reports multiple errors
 
-### [Cascade Mutual Exclusion](analysis-cascade-mutual-exclusion.md)
-Enables safe mutual recursion when cascade conditions are mutually exclusive.
-
-- Allows mathematically sound recursive patterns
-- Detects mutually exclusive conditions
-- Prevents unsafe cycles while enabling safe ones
-
 ### [Type Inference](analysis-type-inference.md)  
 Determines types from expressions and propagates them through dependencies.
 

data/lib/kumi/analyzer.rb

@@ -3,6 +3,7 @@
 module Kumi
   module Analyzer
     Result = Struct.new(:definitions, :dependency_graph, :leaf_map, :topo_order, :decl_types, :state, keyword_init: true)
+    ERROR_THRESHOLD_PASS = Core::Analyzer::Passes::LowerToIRPass
 
     DEFAULT_PASSES = [
       Core::Analyzer::Passes::NameIndexer,                     # 1. Finds all names and checks for duplicates.
@@ -21,7 +22,10 @@ module Kumi
       Core::Analyzer::Passes::ScopeResolutionPass,             # 15. Plans execution scope and lifting needs for declarations.
       Core::Analyzer::Passes::JoinReducePlanningPass,          # 16. Plans join/reduce operations (Generates IR Structs)
       Core::Analyzer::Passes::LowerToIRPass,                   # 17. Lowers the schema to IR (Generates IR Structs)
-      Core::Analyzer::Passes::LoadInputCSE                     # 18. Eliminates redundant load_input operations
+      Core::Analyzer::Passes::LoadInputCSE,                    # 18. Eliminates redundant load_input operations
+      Core::Analyzer::Passes::IRDependencyPass,                # 19. Extracts IR-level dependencies for VM execution optimization
+      Core::Analyzer::Passes::IRExecutionSchedulePass # 20. Builds a precomputed execution schedule.
+
     ].freeze
 
     def self.analyze!(schema, passes: DEFAULT_PASSES, **opts)
@@ -43,6 +47,8 @@ module Kumi
       skipping   = !!resume_at
 
       passes.each_with_index do |pass_class, idx|
+        raise handle_analysis_errors(errors) if (ERROR_THRESHOLD_PASS == pass_class) && !errors.empty?
+
         pass_name = pass_class.name.split("::").last
 
         if skipping
@@ -58,7 +64,9 @@ module Kumi
         t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
         pass_instance = pass_class.new(schema, state)
         begin
-          state = pass_instance.run(errors)
+          state = Dev::Profiler.phase("analyzer.pass", pass: pass_name) do
+            pass_instance.run(errors)
+          end
         rescue StandardError => e
           # TODO: - GREATLY improve this, need to capture the context of the error
           # and the pass that failed and line number if relevant

data/lib/kumi/compiler.rb

@@ -3,18 +3,19 @@
 module Kumi
   # Compiles an analyzed schema into executable lambdas
   class Compiler < Core::CompilerBase
-    def self.compile(schema, analyzer:)
-      new(schema, analyzer).compile
+    def self.compile(schema, analyzer:, schema_name: nil)
+      new(schema, analyzer, schema_name: schema_name).compile
     end
 
-    def initialize(schema, analyzer)
-      super
+    def initialize(schema, analyzer, schema_name: nil)
+      super(schema, analyzer)
       @bindings = {}
+      @schema_name = schema_name
     end
 
     def compile
       # Switch to LIR: Use the analysis state instead of old compilation
-      Runtime::Executable.from_analysis(@analysis.state)
+      Runtime::Executable.from_analysis(@analysis.state, schema_name: @schema_name)
     end
   end
 end

data/lib/kumi/core/analyzer/passes/ir_dependency_pass.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Kumi
+  module Core
+    module Analyzer
+      module Passes
+        # RESPONSIBILITY: Extract IR-level dependencies for VM execution optimization
+        # DEPENDENCIES: :ir_module from LowerToIRPass
+        # PRODUCES: :ir_dependencies - Hash mapping declaration names to referenced bindings
+        #           :ir_name_index - Hash mapping stored binding names to producing declarations
+        # INTERFACE: new(schema, state).run(errors)
+        #
+        # NOTE: This pass extracts actual IR-level dependencies by analyzing :ref operations
+        # in the generated IR, providing the dependency information needed for optimized VM scheduling.
+        class IRDependencyPass < PassBase
+          def run(errors)
+            ir_module = get_state(:ir_module, required: true)
+
+            ir_dependencies = build_ir_dependency_map(ir_module)
+            ir_name_index = build_ir_name_index(ir_module)
+
+            state.with(:ir_dependencies, ir_dependencies).with(:ir_name_index, ir_name_index)
+          end
+
+          private
+
+          # Build a map of declaration -> [stored_bindings_it_references] from the IR
+          def build_ir_dependency_map(ir_module)
+            deps_map = {}
+
+            ir_module.decls.each do |decl|
+              refs = []
+              decl.ops.each do |op|
+                refs << op.attrs[:name] if op.tag == :ref
+              end
+              deps_map[decl.name] = refs
+            end
+
+            deps_map.freeze
+          end
+
+          # Build name index to map stored binding names to their producing declarations
+          def build_ir_name_index(ir_module)
+            ir_name_index = {}
+
+            ir_module.decls.each do |decl|
+              # Map the primary declaration name
+              ir_name_index[decl.name] = decl
+
+              # Also map any vectorized twin names produced by this declaration
+              decl.ops.each do |op|
+                if op.tag == :store
+                  stored_name = op.attrs[:name]
+                  ir_name_index[stored_name] = decl
+                end
+              end
+            end
+
+            ir_name_index.freeze
+          end
+        end
+      end
+    end
+  end
+end

data/lib/kumi/core/analyzer/passes/ir_execution_schedule_pass.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+module Kumi
+  module Core
+    module Analyzer
+      module Passes
+        # PRODUCES: :execution_schedules => { store_name(Symbol) => [Decl, ...] }
+        class IRExecutionSchedulePass < PassBase
+          def run(errors)
+            ir          = get_state(:ir_module, required: true)
+            deps        = get_state(:ir_dependencies, required: true)    # decl_name => [binding_name, ...]
+            name_index  = get_state(:ir_name_index, required: true)      # binding_name => Decl  (← use IR-specific index)
+
+            by_name     = ir.decls.to_h { |d| [d.name, d] }
+            pos         = ir.decls.each_with_index.to_h                  # for deterministic ordering
+
+            closure_cache = {}
+            visiting      = {}
+
+            visit = lambda do |dn|
+              return closure_cache[dn] if closure_cache.key?(dn)
+
+              raise Kumi::Core::Errors::TypeError, "cycle detected in IR at #{dn.inspect}" if visiting[dn]
+
+              visiting[dn] = true
+
+              # Resolve binding refs -> producing decl names
+              preds = Array(deps[dn]).filter_map { |b| name_index[b]&.name }.uniq
+
+              # Deterministic order: earlier IR decls first
+              preds.sort_by! { |n| pos[n] || Float::INFINITY }
+
+              order = []
+              preds.each do |p|
+                next if p == dn # guard against self-deps; treat as error if you prefer
+
+                order.concat(visit.call(p))
+              end
+              order << dn unless order.last == dn
+
+              visiting.delete(dn)
+              closure_cache[dn] = order.uniq.freeze
+            end
+
+            schedules = {}
+
+            ir.decls.each do |decl|
+              target_names = [decl.name] + decl.ops.select { _1.tag == :store }.map { _1.attrs[:name] }
+
+              seq = visit.call(decl.name).map { |dn| by_name.fetch(dn) }.freeze
+
+              target_names.each do |t|
+                if schedules.key?(t) && schedules[t] != seq
+                  raise Kumi::Core::Errors::TypeError,
+                        "duplicate schedule target #{t.inspect} produced by #{schedules[t].last.name} and #{decl.name}"
+                end
+                schedules[t] = seq
+              end
+            end
+
+            state.with(:ir_execution_schedules, schedules.freeze)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/kumi/core/analyzer/passes/toposorter.rb

@@ -5,8 +5,8 @@ module Kumi
   module Core
     module Analyzer
       module Passes
-        # RESPONSIBILITY: Compute topological ordering of declarations, allowing safe conditional cycles
-        # DEPENDENCIES: :dependencies from DependencyResolver, :declarations from NameIndexer, :cascades from UnsatDetector
+        # RESPONSIBILITY: Compute topological ordering of declarations, blocking all cycles
+        # DEPENDENCIES: :dependencies from DependencyResolver, :declarations from NameIndexer
         # PRODUCES: :evaluation_order - Array of declaration names in evaluation order
         #           :node_index - Hash mapping object_id to node metadata for later passes
         # INTERFACE: new(schema, state).run(errors)
@@ -18,7 +18,7 @@ module Kumi
             # Create node index for later passes to use
             node_index = build_node_index(definitions)
             order = compute_topological_order(dependency_graph, definitions, errors)
-            
+
             state.with(:evaluation_order, order).with(:node_index, node_index)
           end
 
@@ -26,53 +26,45 @@ module Kumi
 
           def build_node_index(definitions)
             index = {}
-            
+
             # Walk all declarations and their expressions to index every node
             definitions.each_value do |decl|
               index_node_recursive(decl, index)
             end
-            
+
             index
           end
-          
+
           def index_node_recursive(node, index)
             return unless node
-            
+
             # Index this node by its object_id
             index[node.object_id] = {
               node: node,
-              type: node.class.name.split('::').last,
+              type: node.class.name.split("::").last,
               metadata: {}
             }
-            
+
             # Use the same approach as the visitor pattern - recursively index all children
-            if node.respond_to?(:children)
-              node.children.each { |child| index_node_recursive(child, index) }
-            end
-            
+            node.children.each { |child| index_node_recursive(child, index) } if node.respond_to?(:children)
+
             # Index expression for declaration nodes
-            if node.respond_to?(:expression)
-              index_node_recursive(node.expression, index)
-            end
+            return unless node.respond_to?(:expression)
+
+            index_node_recursive(node.expression, index)
           end
 
           def compute_topological_order(graph, definitions, errors)
             temp_marks = Set.new
             perm_marks = Set.new
             order = []
-            cascades = get_state(:cascades) || {}
 
             visit_node = lambda do |node, path = []|
               return if perm_marks.include?(node)
 
               if temp_marks.include?(node)
-                # Check if this is a safe conditional cycle
-                cycle_path = path + [node]
-                return if safe_conditional_cycle?(cycle_path, graph, cascades)
-
-                # Allow this cycle - it's safe due to cascade mutual exclusion
+                # Block all cycles - no mutual recursion allowed
                 report_unexpected_cycle(temp_marks, node, errors)
-
                 return
               end
 
@@ -102,33 +94,6 @@ module Kumi
             order.freeze
           end
 
-          def safe_conditional_cycle?(cycle_path, graph, cascades)
-            return false if cycle_path.nil? || cycle_path.size < 2
-
-            # Find where the cycle starts - look for the first occurrence of the repeated node
-            last_node = cycle_path.last
-            return false if last_node.nil?
-
-            cycle_start = cycle_path.index(last_node)
-            return false unless cycle_start && cycle_start < cycle_path.size - 1
-
-            cycle_nodes = cycle_path[cycle_start..]
-
-            # Check if all edges in the cycle are conditional
-            cycle_nodes.each_cons(2) do |from, to|
-              edges = graph[from] || []
-              edge = edges.find { |e| e.to == to }
-
-              return false unless edge&.conditional
-
-              # Check if the cascade has mutually exclusive conditions
-              cascade_meta = cascades[edge.cascade_owner]
-              return false unless cascade_meta&.dig(:all_mutually_exclusive)
-            end
-
-            true
-          end
-
           def report_unexpected_cycle(temp_marks, current_node, errors)
             cycle_path = temp_marks.to_a.join(" → ") + " → #{current_node}"
 

data/lib/kumi/core/compiler/access_builder.rb

@@ -2,18 +2,32 @@ module Kumi
   module Core
     module Compiler
       class AccessBuilder
+        class << self
+          attr_accessor :strategy
+        end
+
+        self.strategy = if defined?(RubyVM::YJIT) && RubyVM::YJIT.enabled?
+                          :interp
+                        else
+                          :codegen
+                        end
+
         def self.build(plans)
           accessors = {}
           plans.each_value do |variants|
             variants.each do |plan|
-              key = plan.respond_to?(:accessor_key) ? plan.accessor_key : "#{plan.path}:#{mode}"
-              accessors[key] = build_proc_for(
-                mode: plan.mode,
-                path_key: plan.path,
-                missing: (plan.on_missing || :error).to_sym,
-                key_policy: (plan.key_policy || :indifferent).to_sym,
-                operations: plan.operations
-              )
+              accessors[plan.accessor_key] =
+                case strategy
+                when :codegen then AccessCodegen.fetch_or_compile(plan)
+                else
+                  build_proc_for(
+                    mode: plan.mode,
+                    path_key: plan.path,
+                    missing: (plan.on_missing || :error).to_sym,
+                    key_policy: (plan.key_policy || :indifferent).to_sym,
+                    operations: plan.operations
+                  )
+                end
             end
           end
           accessors.freeze
@@ -25,7 +39,6 @@ module Kumi
           when :materialize then Accessors::MaterializeAccessor.build(operations, path_key, missing, key_policy)
           when :ravel       then Accessors::RavelAccessor.build(operations, path_key, missing, key_policy)
           when :each_indexed then Accessors::EachIndexedAccessor.build(operations, path_key, missing, key_policy, true)
-          when :each then Accessors::EachAccessor.build(operations, path_key, missing, key_policy)
           else
             raise "Unknown accessor mode: #{mode.inspect}"
           end

data/lib/kumi/core/compiler/access_codegen.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "digest/sha1"
+
+module Kumi
+  module Core
+    module Compiler
+      class AccessCodegen
+        CACHE = {}
+        CACHE_MUTEX = Mutex.new
+
+        def self.fetch_or_compile(plan)
+          key = Digest::SHA1.hexdigest(Marshal.dump([plan.mode, plan.operations, plan.on_missing, plan.key_policy, plan.path]))
+          CACHE_MUTEX.synchronize do
+            CACHE[key] ||= compile(plan).tap(&:freeze)
+          end
+        end
+
+        def self.compile(plan)
+          case plan.mode
+          when :read         then gen_read(plan)
+          when :materialize  then gen_materialize(plan)
+          when :ravel        then gen_ravel(plan)
+          when :each_indexed then gen_each_indexed(plan)
+          else
+            raise "Unknown accessor mode: #{plan.mode.inspect}"
+          end
+        end
+
+        private_class_method def self.gen_read(plan)
+          code = AccessEmit::Read.build(plan)
+          debug_code(code, plan, "READ") if ENV["DEBUG_CODEGEN"]
+          eval(code, TOPLEVEL_BINDING)
+        end
+
+        private_class_method def self.gen_materialize(plan)
+          code = AccessEmit::Materialize.build(plan)
+          debug_code(code, plan, "MATERIALIZE") if ENV["DEBUG_CODEGEN"]
+          eval(code, TOPLEVEL_BINDING)
+        end
+
+        private_class_method def self.gen_ravel(plan)
+          code = AccessEmit::Ravel.build(plan)
+          debug_code(code, plan, "RAVEL") if ENV["DEBUG_CODEGEN"]
+          eval(code, TOPLEVEL_BINDING)
+        end
+
+        private_class_method def self.gen_each_indexed(plan)
+          code = AccessEmit::EachIndexed.build(plan)
+          debug_code(code, plan, "EACH_INDEXED") if ENV["DEBUG_CODEGEN"]
+          eval(code, TOPLEVEL_BINDING)
+        end
+
+        private_class_method def self.debug_code(code, plan, mode_name)
+          puts "=== Generated #{mode_name} code for #{plan.path}:#{plan.mode} ==="
+          puts code
+        end
+      end
+    end
+  end
+end