kumi 0.0.13 → 0.0.15

data/docs/dev/vm-profiling.md

@@ -0,0 +1,95 @@
+# VM Profiling with Schema Differentiation
+
+## Overview
+
+Profiles VM operation execution with schema-level differentiation. Tracks operations by schema type for multi-schema performance analysis.
+
+## Core Components
+
+**Profiler**: `lib/kumi/core/ir/execution_engine/profiler.rb`
+- Streams VM operation events with schema identification
+- Supports persistent mode for cross-run analysis
+- JSONL event format with operation metadata
+
+**Profile Aggregator**: `lib/kumi/dev/profile_aggregator.rb`  
+- Analyzes profiling data by schema type
+- Generates summary and detailed performance reports
+- Schema breakdown showing operations and timing per schema
+
+**CLI Integration**: `bin/kumi profile`
+- Processes JSONL profiling data files
+- Multiple output formats: summary, detailed, raw
+
+## Usage
+
+### Basic Profiling
+
+```bash
+# Single schema with operations
+KUMI_PROFILE=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+
+# Persistent mode across multiple runs
+KUMI_PROFILE=1 KUMI_PROFILE_PERSISTENT=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+
+# Streaming mode for real-time analysis  
+KUMI_PROFILE=1 KUMI_PROFILE_STREAM=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
+```
+
+### CLI Analysis
+
+```bash
+# Summary report with schema breakdown
+kumi profile profile.jsonl --summary
+
+# Detailed per-operation analysis
+kumi profile profile.jsonl --detailed
+
+# Raw event stream
+kumi profile profile.jsonl --raw
+```
+
+## Environment Variables
+
+**Core**:
+- `KUMI_PROFILE=1` - Enable profiling
+- `KUMI_PROFILE_FILE=path` - Output file (required)
+- `KUMI_PROFILE_OPS=1` - Enable VM operation profiling
+
+**Modes**:
+- `KUMI_PROFILE_PERSISTENT=1` - Append to existing files across runs
+- `KUMI_PROFILE_STREAM=1` - Stream individual events vs batch
+- `KUMI_PROFILE_TRUNCATE=1` - Truncate existing files
+
+## Event Format
+
+JSONL with operation metadata:
+
+```json
+{"event":"vm_operation","schema":"TestSchema","operation":"LoadInput","duration_ms":0.001,"timestamp":"2025-01-20T10:30:45.123Z"}
+{"event":"vm_operation","schema":"TestSchema","operation":"Map","duration_ms":0.002,"timestamp":"2025-01-20T10:30:45.125Z"}
+```
+
+## Schema Differentiation
+
+Tracks operations by schema class name for multi-schema analysis:
+
+**Implementation**:
+- Schema name propagated through compilation pipeline
+- Profiler tags each VM operation with schema identifier  
+- Aggregator groups operations by schema type
+
+**Output Example**:
+```
+Total operations: 24 (0.8746ms)
+Schemas analyzed: SchemaA, SchemaB
+  SchemaA: 12 operations, 0.3242ms
+  SchemaB: 12 operations, 0.0504ms
+```
+
+## Performance Analysis
+
+**Reference Operations**: Typically dominate execution time in complex schemas
+**Map Operations**: Element-wise computations on arrays
+**LoadInput Operations**: Data access operations
+
+Use schema breakdown to identify performance differences between schema types.

data/docs/features/README.md

@@ -9,13 +9,6 @@ Analyzes rule combinations to detect logical impossibilities across dependency c
 - Validates domain constraints
 - Reports multiple errors
 
-### [Cascade Mutual Exclusion](analysis-cascade-mutual-exclusion.md)
-Enables safe mutual recursion when cascade conditions are mutually exclusive.
-
-- Allows mathematically sound recursive patterns
-- Detects mutually exclusive conditions
-- Prevents unsafe cycles while enabling safe ones
-
 ### [Type Inference](analysis-type-inference.md)  
 Determines types from expressions and propagates them through dependencies.
 

data/docs/functions/analyzer_integration.md

@@ -0,0 +1,199 @@
+# Function Signature Analysis Integration
+
+This document describes how NEP-20 function signatures integrate with Kumi's multi-pass analyzer architecture.
+
+## Architecture Overview
+
+Function signature resolution is integrated into the analyzer pipeline through a node index system that allows passes to share metadata efficiently.
+
+```
+┌─────────────────┐    ┌──────────────────────┐    ┌────────────────────┐
+│   Toposorter    │───▶│ FunctionSignaturePass │───▶│   LowerToIRPass    │
+│ Creates node    │    │ Resolves signatures   │    │ Validates metadata │
+│ index by ID     │    │ Attaches metadata     │    │ Emits IR ops       │
+└─────────────────┘    └──────────────────────┘    └────────────────────┘
+```
+
+## Node Index System
+
+### Creation (Toposorter)
+
+The `Toposorter` pass creates a comprehensive index of all nodes in the AST:
+
+```ruby
+# State structure after Toposorter
+state[:node_index] = {
+  object_id_1 => {
+    node: CallExpression_instance,
+    type: "CallExpression",
+    metadata: {}
+  },
+  # ... more nodes
+}
+```
+
+### Population (FunctionSignaturePass)
+
+The `FunctionSignaturePass` populates metadata for `CallExpression` nodes:
+
+```ruby
+# After FunctionSignaturePass
+entry[:metadata] = {
+  signature: Signature_object,           # Full signature object
+  result_axes: [:i, :j],                # Output dimension names  
+  join_policy: :zip,                    # Cross-dimensional policy
+  dropped_axes: [:k],                   # Dimensions eliminated by reductions
+  effective_signature: {                # Normalized for lowering
+    in_shapes: [[:i, :k], [:k, :j]],
+    out_shape: [:i, :j], 
+    join_policy: :zip
+  },
+  dim_env: { i: :i, j: :j, k: :k },    # Dimension variable bindings
+  shape_contract: {                     # Simplified for lowering
+    in: [[:i, :k], [:k, :j]],
+    out: [:i, :j],
+    join: :zip
+  },
+  signature_score: 0                    # Match quality (0 = exact)
+}
+```
+
+### Consumption (LowerToIRPass)
+
+The lowering pass validates and uses the signature metadata:
+
+```ruby
+def validate_signature_metadata(expr, entry)
+  node_index = get_state(:node_index)
+  metadata = node_index[expr.object_id][:metadata]
+  
+  # Validate dropped axes for reductions
+  if entry&.reducer && metadata[:dropped_axes]
+    # Assert axis exists in scope
+  end
+  
+  # Warn about join policies not yet implemented
+  if metadata[:join_policy]
+    # Log warning or feature flag check
+  end
+end
+```
+
+## Pass Integration Points
+
+### 1. Signature Resolution
+
+**Location**: After `BroadcastDetector`, before `TypeChecker`
+
+**Purpose**: Resolve NEP-20 signatures for all function calls
+
+**Input**: 
+- Node index from Toposorter
+- Broadcast metadata (optional)
+- Current function registry
+
+**Output**: Rich signature metadata attached to call nodes
+
+### 2. Type Checking Enhancement
+
+**Integration**: `TypeChecker` can now use signature metadata for enhanced validation:
+
+```ruby
+def validate_function_call(node, errors)
+  # Get resolved signature metadata
+  node_entry = node_index[node.object_id]
+  if node_entry && node_entry[:metadata][:signature]
+    # Use NEP-20 signature for validation
+    validate_nep20_signature(node, node_entry[:metadata], errors)
+  else
+    # Fall back to legacy registry-based validation
+    validate_legacy_signature(node, errors)
+  end
+end
+```
+
+### 3. Lowering Integration
+
+**Integration**: `LowerToIRPass` validates signature consistency and emits appropriate IR:
+
+```ruby  
+when Syntax::CallExpression
+  entry = Kumi::Registry.entry(expr.fn_name)
+  validate_signature_metadata(expr, entry)  # Read-only validation
+  
+  # Use shape contract for IR generation
+  if node_entry = node_index[expr.object_id]
+    contract = node_entry[:metadata][:shape_contract]
+    # Use contract to emit appropriate reduce/join ops
+  end
+```
+
+## Metadata Contract
+
+All passes can rely on this metadata structure for `CallExpression` nodes:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `:signature` | `Signature` | Full signature object with dimensions |
+| `:result_axes` | `Array<Symbol>` | Output dimension names |
+| `:join_policy` | `Symbol?` | `:zip`, `:product`, or `nil` |
+| `:dropped_axes` | `Array<Symbol>` | Dimensions eliminated (reductions) |
+| `:effective_signature` | `Hash` | Normalized signature for lowering |
+| `:dim_env` | `Hash` | Dimension variable bindings |
+| `:shape_contract` | `Hash` | Simplified contract for IR generation |
+| `:signature_score` | `Integer` | Match quality (0 = exact) |
+
+## Error Handling
+
+Enhanced error messages include signature information:
+
+```ruby
+# Before
+"operator `multiply` expects 2 args, got 3"
+
+# After  
+"Signature mismatch for `multiply` with args (i,j), (k). 
+ Candidates: (),()->() | (i),(i)->(i). 
+ no matching signature for shapes (i,j), (k)"
+```
+
+## Feature Flags
+
+Control NEP-20 behavior at runtime:
+
+```bash
+export KUMI_ENABLE_FLEX=1    # Enable flexible dimension matching (?)
+export KUMI_ENABLE_BCAST1=1  # Enable broadcastable matching (|1)  
+```
+
+## Future Extensions
+
+The node index architecture supports future enhancements:
+
+### Registry V2 Integration
+```ruby
+# Read signatures from YAML registry
+signatures = RegistryV2.get_function_signatures(node.fn_name)
+```
+
+### Type System Integration
+```ruby
+# Add dtype metadata alongside shape metadata
+metadata[:result_dtype] = infer_result_dtype(args, signature)
+```
+
+### Optimization Metadata
+```ruby
+# Add optimization hints
+metadata[:can_vectorize] = true
+metadata[:memory_access_pattern] = :sequential
+```
+
+## Performance Considerations
+
+- **Node index creation**: O(n) where n = total AST nodes
+- **Signature resolution**: O(k*m) where k = signatures, m = arguments  
+- **Memory overhead**: ~100 bytes per CallExpression node
+- **Pass integration**: Zero performance impact on existing passes
+
+The architecture is designed for extensibility while maintaining backward compatibility with the existing analyzer pipeline.

data/docs/functions/signatures.md

@@ -0,0 +1,171 @@
+# Function Signatures (NEP 20)
+
+Kumi implements **NEP 20 conformant function signatures** for describing generalized universal functions. This system provides precise control over multidimensional operations with support for fixed-size, flexible, and broadcastable dimensions.
+
+## Basic Syntax
+
+Function signatures use the format: `<inputs> -> <outputs>[@policy]`
+
+```ruby
+"(i),(i)->(i)"           # Element-wise operation on vectors
+"(m,n),(n,p)->(m,p)"     # Matrix multiplication  
+"(i,j)->(i)"             # Reduction along j axis
+"(),()->()"              # Scalar operation
+```
+
+## NEP 20 Extensions
+
+### Fixed-Size Dimensions
+
+Use integers to specify exact dimension sizes:
+
+```ruby
+"(3),(3)->(3)"          # Cross product for 3-vectors
+"()->(2)"               # Function returning 2D vector
+"(),()->(3)"            # Two scalars to 3D vector
+```
+
+Fixed-size dimensions must match exactly - no broadcasting allowed.
+
+### Flexible Dimensions (?)
+
+The `?` modifier indicates dimensions that can be omitted if not present in all operands:
+
+```ruby
+"(m?,n),(n,p?)->(m?,p?)" # matmul signature - handles:
+                         # - (m,n),(n,p) -> (m,p)   matrix * matrix
+                         # - (n),(n,p) -> (p)       vector * matrix  
+                         # - (m,n),(n) -> (m)       matrix * vector
+                         # - (n),(n) -> ()          vector * vector
+```
+
+### Broadcastable Dimensions (|1)
+
+The `|1` modifier allows dimensions to broadcast against scalar or size-1 dimensions:
+
+```ruby
+"(n|1),(n|1)->()"       # all_equal - compares vectors or scalars
+"(i|1),(j|1)->(i,j)"    # Outer product with broadcasting
+```
+
+**Constraints:**
+- Only input dimensions can be broadcastable
+- Output dimensions cannot have `|1` modifier
+- Cannot combine `?` and `|1` on same dimension
+
+## Join Policies
+
+Control how different dimension names are combined:
+
+### Default (nil policy)
+All non-scalar arguments must have compatible dimension names:
+```ruby
+"(i),(i)->(i)"     # ✓ Compatible - same dimensions
+"(i),(j)->(i,j)"   # ✗ Incompatible without policy
+```
+
+### @product Policy
+Allows different dimensions, creates Cartesian product:
+```ruby
+"(i),(j)->(i,j)@product"  # Outer product
+```
+
+### @zip Policy  
+Allows different dimensions, pairs them up:
+```ruby
+"(i),(j)->(i)@zip"        # Paired operation
+```
+
+## Examples from NEP 20
+
+| Signature | Use Case | Description |
+|-----------|----------|-------------|
+| `(),()->()` | Addition | Scalar operations |
+| `(i)->()` | Sum | Reduction over last axis |
+| `(i\|1),(i\|1)->()` | all_equal | Equality test with broadcasting |
+| `(i),(i)->()` | dot product | Inner vector product |
+| `(m,n),(n,p)->(m,p)` | matmul | Matrix multiplication |
+| `(3),(3)->(3)` | cross | Cross product for 3-vectors |
+| `(m?,n),(n,p?)->(m?,p?)` | matmul | Universal matrix operations |
+
+## Signature Validation Rules
+
+1. **Arity matching**: Number of arguments must match signature
+2. **Fixed-size consistency**: Integer dimensions must match exactly
+3. **Broadcasting rules**: `|1` dimensions can broadcast to any size
+4. **Flexible resolution**: `?` dimensions resolved based on presence
+5. **Output constraints**: No broadcastable (`|1`) dimensions in outputs
+6. **Policy requirements**: Different dimension names need explicit policy
+
+## Matching Priority
+
+When multiple signatures are available, resolution prefers:
+
+1. **Exact matches** (score: 0) - All dimensions match perfectly
+2. **Fixed-size matches** (score: 0-2) - Integer dimensions match
+3. **Broadcast matches** (score: 1-3) - Scalar broadcasting  
+4. **Flexible matches** (score: 10+) - Dimension omission/addition
+
+Lower scores indicate better matches.
+
+## Implementation Notes
+
+Signatures are parsed into `Dimension` objects that track:
+- Name (Symbol or Integer)
+- Flexible flag (`?`)
+- Broadcastable flag (`|1`)
+
+The resolver handles NEP 20 semantics including:
+- Flexible dimension resolution
+- Broadcastable matching
+- Fixed-size validation
+- Join policy enforcement
+
+## Constraint Solver
+
+Kumi implements a **dimension constraint solver** that validates cross-argument dimension consistency for operations like matrix multiplication:
+
+```ruby
+signature = "(m,n),(n,p)->(m,p)"
+arg_shapes = [[:m, :n], [:n, :p]]
+
+# The solver validates that 'n' dimensions are consistent across arguments
+plan = resolver.choose(signatures: [sig], arg_shapes: arg_shapes)
+# Returns: { env: { m: :m, n: :n, p: :p }, result_axes: [:m, :p] }
+```
+
+The constraint solver:
+- Links repeated dimension names across arguments
+- Validates dimension consistency (same `n` in both positions)
+- Returns dimension environment bindings
+- Enables complex operations without explicit join policies
+
+## Integration with Analyzer
+
+Function signatures integrate with Kumi's analyzer pipeline through:
+
+### FunctionSignaturePass
+Resolves NEP-20 signatures for all function calls and attaches metadata:
+- `:result_axes` - Output dimension names
+- `:join_policy` - Cross-dimensional operation policy  
+- `:dropped_axes` - Dimensions eliminated by reductions
+- `:effective_signature` - Normalized signature for lowering
+- `:dim_env` - Dimension variable bindings
+- `:shape_contract` - Simplified contract for lowering
+
+### Node Index Architecture
+The analyzer creates a node index mapping `object_id → metadata` that passes can use to:
+- Attach signature resolution results
+- Share metadata across analysis passes
+- Validate consistency in lowering
+
+## Feature Flags
+
+Control NEP-20 extensions with environment variables:
+
+```bash
+KUMI_ENABLE_FLEX=1    # Enable flexible dimension matching (?)
+KUMI_ENABLE_BCAST1=1  # Enable broadcastable dimension matching (|1)
+```
+
+For more details see [NEP 20 specification](https://numpy.org/neps/nep-0020-expansion-of-generalized-ufunc-signatures.html).

data/examples/hash_objects_demo.rb

@@ -0,0 +1,138 @@
+# frozen_string_literal: true
+
+# Hash Objects Demo - Demonstrates Kumi's structured input syntax
+# This example shows how to use hash objects for organizing related input fields
+
+require_relative "../lib/kumi"
+
+module HashObjectsDemo
+  extend Kumi::Schema
+
+  schema do
+    input do
+      # Employee information as hash object
+      hash :employee do
+        string :name
+        integer :age, domain: 18..65
+        float :base_salary, domain: 30_000.0..200_000.0
+        boolean :is_manager
+        integer :years_experience, domain: 0..40
+      end
+      
+      # Company configuration as hash object
+      hash :company_config do
+        string :name
+        float :bonus_percentage, domain: 0.0..0.50
+        float :manager_multiplier, domain: 1.0..2.0
+        integer :current_year, domain: 2020..2030
+      end
+      
+      # Benefits configuration as hash object
+      hash :benefits do
+        boolean :health_insurance
+        boolean :dental_coverage
+        float :retirement_match, domain: 0.0..0.10
+        integer :vacation_days, domain: 10..30
+      end
+    end
+
+    # Traits using hash object access
+    trait :is_senior, input.employee.years_experience >= 5
+    trait :eligible_for_bonus, is_senior & input.employee.is_manager
+    trait :has_full_benefits, input.benefits.health_insurance & input.benefits.dental_coverage
+
+    # Salary calculations using structured data
+    value :base_annual_salary, input.employee.base_salary
+    
+    value :bonus_amount do
+      on eligible_for_bonus, base_annual_salary * input.company_config.bonus_percentage
+      base 0.0
+    end
+    
+    trait :is_manager_trait, input.employee.is_manager == true
+    
+    value :manager_adjustment do
+      on is_manager_trait, input.company_config.manager_multiplier
+      base 1.0
+    end
+    
+    value :total_compensation, (base_annual_salary * manager_adjustment) + bonus_amount
+    
+    # Benefits calculations
+    value :retirement_contribution, total_compensation * input.benefits.retirement_match
+    
+    value :benefits_package_value do
+      on has_full_benefits, 5_000.0 + input.benefits.vacation_days * 150.0
+      base input.benefits.vacation_days * 100.0
+    end
+    
+    # Final totals
+    value :total_package_value, total_compensation + retirement_contribution + benefits_package_value
+    
+    # Summary calculations
+    value :years_to_retirement, 65 - input.employee.age
+  end
+end
+
+# Example usage
+if __FILE__ == $0
+  puts "Hash Objects Demo - Employee Compensation Calculator"
+  puts "=" * 55
+
+  # Sample data demonstrating hash objects structure
+  employee_data = {
+    employee: {
+      name: "Alice Johnson",
+      age: 32,
+      base_salary: 85_000.0,
+      is_manager: true,
+      years_experience: 8
+    },
+    company_config: {
+      name: "Tech Solutions Inc",
+      bonus_percentage: 0.15,
+      manager_multiplier: 1.25,
+      current_year: 2024
+    },
+    benefits: {
+      health_insurance: true,
+      dental_coverage: true,
+      retirement_match: 0.06,
+      vacation_days: 25
+    }
+  }
+
+  # Calculate compensation
+  result = HashObjectsDemo.from(employee_data)
+  
+  def format_currency(amount)
+    "$#{amount.round(0).to_s.gsub(/\B(?=(\d{3})+(?!\d))/, ',')}"
+  end
+  
+  puts "\nEmployee Information:"
+  puts "- Name: #{employee_data[:employee][:name]}"
+  puts "- Age: #{employee_data[:employee][:age]}"
+  puts "- Experience: #{employee_data[:employee][:years_experience]} years"
+  puts "- Manager: #{employee_data[:employee][:is_manager] ? 'Yes' : 'No'}"
+  
+  puts "\nCompany: #{employee_data[:company_config][:name]}"
+  puts "- Bonus Rate: #{(employee_data[:company_config][:bonus_percentage] * 100).round(1)}%"
+  puts "- Manager Multiplier: #{employee_data[:company_config][:manager_multiplier]}x"
+  
+  puts "\nBenefits:"
+  puts "- Health Insurance: #{employee_data[:benefits][:health_insurance] ? 'Yes' : 'No'}"
+  puts "- Dental Coverage: #{employee_data[:benefits][:dental_coverage] ? 'Yes' : 'No'}"
+  puts "- Retirement Match: #{(employee_data[:benefits][:retirement_match] * 100).round(1)}%"
+  puts "- Vacation Days: #{employee_data[:benefits][:vacation_days]}"
+
+  puts "\nCompensation Breakdown:"
+  puts "- Base Salary: #{format_currency(result[:base_annual_salary])}"
+  puts "- Manager Adjustment: #{result[:manager_adjustment]}x"
+  puts "- Bonus: #{format_currency(result[:bonus_amount])}"
+  puts "- Total Compensation: #{format_currency(result[:total_compensation])}"
+  puts "- Retirement Contribution: #{format_currency(result[:retirement_contribution])}"
+  puts "- Benefits Package Value: #{format_currency(result[:benefits_package_value])}"
+  
+  puts "\nTotal Package: #{format_currency(result[:total_package_value])}"
+  puts "Years to Retirement: #{result[:years_to_retirement]}"
+end

data/golden/array_operations/schema.kumi

@@ -0,0 +1,17 @@
+schema do
+  input do
+    array :items do
+      float :price
+      integer :quantity
+      string :category
+    end
+    float :tax_rate
+  end
+  
+  value :subtotals, input.items.price * input.items.quantity
+  trait :expensive_items, input.items.price > 100.0
+  trait :electronics, input.items.category == "electronics"
+  
+  value :discounted_price, input.items.price * 0.9
+  value :is_valid_quantity, input.items.quantity > 0
+end

data/golden/cascade_logic/schema.kumi

@@ -0,0 +1,16 @@
+schema do
+  input do
+    integer :x
+    integer :y
+  end
+  
+  trait :x_positive, input.x > 0
+  trait :y_positive, input.y > 0
+  
+  value :status do
+    on x_positive, y_positive, "both positive"
+    on x_positive, "x positive"  
+    on y_positive, "y positive"
+    base "neither positive"
+  end
+end

data/golden/mixed_nesting/schema.kumi

@@ -0,0 +1,42 @@
+schema do
+    input do
+    hash :organization do
+        string :name
+        array :regions do
+        string :region_name
+        hash :headquarters do
+            string :city
+            array :buildings do
+            string :building_name
+            hash :facilities do
+                string :facility_type
+                integer :capacity
+                float :utilization_rate
+            end
+            end
+        end
+        end
+    end
+    end
+    
+    # Deep access across 5 levels
+    value :org_name, input.organization.name
+    value :region_names, input.organization.regions.region_name
+    value :hq_cities, input.organization.regions.headquarters.city
+    value :building_names, input.organization.regions.headquarters.buildings.building_name
+    value :facility_types, input.organization.regions.headquarters.buildings.facilities.facility_type
+    value :capacities, input.organization.regions.headquarters.buildings.facilities.capacity
+    value :utilization_rates, input.organization.regions.headquarters.buildings.facilities.utilization_rate
+    
+    # Traits using deep nesting - avoiding cross-scope issues
+    trait :large_organization, fn(:size, input.organization.regions) > 1
+    
+    # Simple cascade using traits that work within same scope
+    value :org_classification do
+    on large_organization, "Enterprise"
+    base "Standard"
+    end
+    
+    # Aggregations that work properly
+    value :total_capacity, fn(:sum, input.organization.regions.headquarters.buildings.facilities.capacity)
+end