kumi 0.0.9 → 0.0.11

data/docs/features/hierarchical-broadcasting.md

@@ -0,0 +1,349 @@
+# Hierarchical Broadcasting
+
+Automatic vectorization of operations over hierarchical data structures with two complementary access modes for different use cases.
+
+## Overview
+
+The hierarchical broadcasting system enables natural field access syntax that automatically applies operations element-wise across complex nested data structures, with intelligent detection of map vs reduce operations and support for both structured objects and raw multi-dimensional arrays.
+
+## Core Mechanism
+
+The system uses a three-stage pipeline:
+
+1. **Parser** - Creates InputElementReference AST nodes for nested field access
+2. **BroadcastDetector** - Identifies which operations should be vectorized vs scalar
+3. **Compiler** - Generates appropriate map/reduce functions based on usage context
+
+## Access Modes
+
+The system supports two complementary access modes that can be mixed within the same schema:
+
+### Object Access Mode (Default)
+For structured business data with named fields:
+
+```ruby
+input do
+  array :users do
+    string :name
+    integer :age
+  end
+end
+
+value :user_names, input.users.name     # Broadcasts over structured objects
+trait :adults, input.users.age >= 18    # Boolean array from age comparison
+```
+
+### Element Access Mode  
+For multi-dimensional raw arrays using `element` syntax with **progressive path traversal**:
+
+```ruby
+input do
+  array :cube do
+    element :array, :layer do
+      element :array, :row do
+        element :integer, :cell
+      end
+    end
+  end
+end
+
+# Progressive path traversal - each level goes deeper
+value :dimensions, [
+  fn(:size, input.cube),            # Number of layers
+  fn(:size, input.cube.layer),     # Total matrices across all layers  
+  fn(:size, input.cube.layer.row), # Total rows across all matrices
+  fn(:size, input.cube.layer.row.cell) # Total cells (leaf elements)
+]
+
+# Operations at different dimensional levels
+value :layer_data, input.cube.layer              # 3D matrices
+value :row_data, input.cube.layer.row           # 2D rows
+value :cell_data, input.cube.layer.row.cell     # 1D values
+```
+
+**Key Benefits of Progressive Path Traversal:**
+- **Intuitive syntax**: `input.cube.layer.row` naturally represents "rows within layers within cube"
+- **Direct dimensional access**: No need for complex flattening operations for basic dimensional analysis
+- **Ranked polymorphism**: Same operations work across different dimensional arrays
+- **Clean code**: `fn(:size, input.cube.layer.row)` instead of `fn(:size, fn(:flatten_one, input.cube.layer))`
+
+## Business Use Cases
+
+Element access mode is essential for common business scenarios involving simple nested arrays:
+
+```ruby
+# E-commerce: Product availability analysis
+input do
+  array :products do
+    string :name
+    array :daily_sales do        # Simple array: [12, 8, 15, 3]
+      element :integer, :units
+    end
+  end
+end
+
+# Create flags for business rules
+trait :had_slow_days, fn(:any?, input.products.daily_sales.units < 5)
+trait :consistently_strong, fn(:all?, input.products.daily_sales.units >= 10)
+
+# Progressive analysis
+value :sales_summary, [
+  fn(:size, input.products),                    # Number of products
+  fn(:size, input.products.daily_sales.units), # Total sales data points
+  fn(:sum, fn(:flatten, input.products.daily_sales.units)) # Total units sold
+]
+```
+
+### Mixed Access Modes
+Both modes work together seamlessly:
+
+```ruby
+input do
+  array :users do          # Object access
+    string :name
+    integer :age  
+  end
+  array :activity_logs do  # Element access
+    element :integer, :day
+  end
+end
+
+value :user_count, fn(:size, input.users)
+value :total_activity_days, fn(:size, fn(:flatten, input.activity_logs.day))
+```
+
+## Basic Broadcasting (Object Access)
+
+```ruby
+schema do
+  input do
+    array :line_items do
+      float   :price
+      integer :quantity
+      string  :category
+    end
+    float :tax_rate, type: :float
+  end
+
+  # Element-wise computation - broadcasts over each item
+  value :subtotals, input.line_items.price * input.line_items.quantity
+  
+  # Element-wise traits - applied to each item
+  trait :is_taxable, (input.line_items.category != "digital")
+  
+  # Conditional logic - element-wise evaluation
+  value :taxes, fn(:if, is_taxable, subtotals * input.tax_rate, 0.0)
+end
+```
+
+## Aggregation Operations
+
+Operations that consume arrays to produce scalars are automatically detected:
+
+```ruby
+schema do
+  # These aggregate the vectorized results
+  value :total_subtotal, fn(:sum, subtotals)
+  value :total_tax, fn(:sum, taxes)
+  value :grand_total, total_subtotal + total_tax
+  
+  # Statistics over arrays
+  value :avg_price, fn(:avg, input.line_items.price)
+  value :max_quantity, fn(:max, input.line_items.quantity)
+end
+```
+
+## Field Access Nesting
+
+Supports arbitrary depth field access with path building:
+
+```ruby
+schema do
+  input do
+    array :orders do
+      array :items do
+        hash :product do
+          string :name
+          float  :base_price
+        end
+        integer :quantity
+      end
+    end
+  end
+
+  # Deep field access - automatically broadcasts over nested arrays  
+  value :all_product_names, input.orders.items.product.name
+  value :total_values, input.orders.items.product.base_price * input.orders.items.quantity
+end
+```
+
+## Type Inference
+
+The type system automatically infers appropriate types for broadcasted operations:
+
+- `input.items.price` (float array) → inferred as `:float` per element
+- `input.items.price * input.items.quantity` → element-wise `:float` result
+- `fn(:sum, input.items.price)` → scalar `:float` result
+
+## Implementation Details
+
+### Parser Layer
+- **InputFieldProxy** - Handles `input.field.subfield...` with path building
+- **InputElementReference** - AST node representing array field access paths
+
+### Analysis Layer  
+- **BroadcastDetector** - Identifies vectorized vs scalar operations
+- **TypeInferencerPass** - Infers types for array element access patterns
+
+### Compilation Layer
+- **Automatic Dispatch** - Maps element-wise operations to array map functions
+- **Reduction Detection** - Converts aggregation functions to array reduce operations
+
+## Usage Patterns
+
+### Element-wise Operations
+```ruby
+# All of these broadcast element-wise
+value :discounted_prices, input.items.price * 0.9
+trait :expensive, (input.items.price > 100.0)  
+value :categories, input.items.category
+```
+
+### Aggregation Operations
+```ruby
+# These consume arrays to produce scalars
+value :item_count, fn(:size, input.items)
+value :total_price, fn(:sum, input.items.price)
+value :has_expensive, fn(:any?, expensive)
+```
+
+### Mixed Operations
+```ruby
+# Element-wise computation followed by aggregation
+value :line_totals, input.items.price * input.items.quantity
+value :order_total, fn(:sum, line_totals)
+value :avg_line_total, fn(:avg, line_totals)
+```
+
+### Conditional Aggregation Functions
+
+Kumi provides powerful conditional aggregation functions that make working with vectorized traits clean and intuitive:
+
+```ruby
+# Step 1: Create vectorized values and traits
+value :revenues, input.sales.price * input.sales.quantity  # → [3000.0, 2000.0, 1250.0]
+trait :expensive, input.sales.price > 100                   # → [true, true, false]
+trait :bulk_order, input.sales.quantity >= 15              # → [false, false, true]
+
+# Step 2: Use conditional aggregation functions (NEW - clean and readable)
+value :expensive_count, fn(:count_if, expensive)                    # → 2
+value :expensive_total, fn(:sum_if, revenues, expensive)            # → 5000.0
+value :expensive_average, fn(:avg_if, revenues, expensive)          # → 2500.0
+
+# Step 3: Combine conditions for complex filtering
+trait :expensive_bulk, expensive & bulk_order                       # → [false, false, false]
+value :expensive_bulk_total, fn(:sum_if, revenues, expensive_bulk)  # → 0.0
+```
+
+**Old cascade pattern** (verbose, harder to read):
+```ruby
+# OLD WAY - required verbose cascade patterns
+value :expensive_amounts do
+  on expensive, revenues
+  base 0.0
+end
+value :expensive_total, fn(:sum, expensive_amounts)
+
+value :expensive_count_markers do
+  on expensive, 1.0
+  base 0.0
+end
+value :expensive_count, fn(:sum, expensive_count_markers)
+```
+
+**New conditional functions** (clean, direct):
+```ruby
+# NEW WAY - direct and readable
+value :expensive_total, fn(:sum_if, revenues, expensive)
+value :expensive_count, fn(:count_if, expensive)
+value :expensive_average, fn(:avg_if, revenues, expensive)
+```
+
+The conditional aggregation functions are now the idiomatic way to work with boolean arrays in Kumi.
+
+## Common Patterns and Gotchas
+
+### Operator Precedence with Mixed Arithmetic
+
+**Problem**: Complex arithmetic expressions with arrays can fail due to precedence parsing:
+
+```ruby
+# This fails with confusing error message
+value :ones, input.items.price * 0 + 1
+# Error: "no implicit conversion of Integer into Array"
+
+# The expression is parsed as: (input.items.price * 0) + 1
+# Which becomes: [0.0, 0.0, 0.0] + 1 (array + scalar in wrong context)
+```
+
+**Solutions**:
+
+```ruby
+# Use explicit function calls
+value :ones, fn(:add, input.items.price * 0, 1)
+
+# Use cascade pattern (most idiomatic)
+trait :always_true, input.items.price >= 0
+value :ones do
+  on always_true, 1.0
+  base 0.0
+end
+
+# Use separate steps
+value :zeros, input.items.price * 0
+value :ones, zeros + 1
+```
+
+The cascade pattern is preferred as it's more explicit about intent and leverages Kumi's automatic scalar broadcasting.
+
+## Error Handling
+
+### Dimension Mismatch Detection
+
+Array broadcasting operations are only valid within the same array source. Attempting to broadcast across different arrays generates detailed error messages:
+
+```ruby
+schema do
+  input do
+    array :items do
+      string :name
+    end
+    array :logs do  
+      string :user_name
+    end
+  end
+
+  # This will generate a dimension mismatch error
+  trait :same_name, input.items.name == input.logs.user_name
+end
+
+# Error:
+# Cannot broadcast operation across arrays from different sources: items, logs. 
+# Problem: Multiple operands are arrays from different sources:
+#   - Operand 1 resolves to array(string) from array 'items'
+#   - Operand 2 resolves to array(string) from array 'logs'
+# Direct operations on arrays from different sources is ambiguous and not supported. 
+# Vectorized operations can only work on fields from the same array input.
+```
+
+The error messages provide:
+- **Quick Summary**: Identifies the conflicting array sources
+- **Type Information**: Shows the resolved types of each operand  
+- **Clear Explanation**: Why the operation is ambiguous and not supported
+
+## Performance Characteristics
+
+- **Single Pass** - Each array is traversed once per computation chain
+- **Lazy Evaluation** - Operations are composed into pipelines  
+- **Memory Efficient** - No intermediate array allocations for simple operations
+- **Type Safe** - Full compile-time type checking for array element operations

data/docs/features/javascript-transpiler.md

@@ -0,0 +1,148 @@
+# JavaScript Transpiler
+
+Transpiles compiled schemas to standalone JavaScript code.
+
+## Usage
+
+### Export Schema
+
+```ruby
+class TaxCalculator
+  extend Kumi::Schema
+  
+  schema do
+    input do
+      float :income
+      string :filing_status
+    end
+    
+    trait :single, input.filing_status == "single"
+    
+    value :std_deduction do
+      on single, 14_600
+      base 29_200
+    end
+    
+    value :taxable_income, fn(:max, [input.income - std_deduction, 0])
+    value :tax_owed, taxable_income * 0.22
+  end
+end
+
+Kumi::Js.export_to_file(TaxCalculator, "tax-calculator.js")
+```
+
+### Use in JavaScript
+
+```javascript
+const { schema } = require('./tax-calculator.js');
+
+const taxpayer = {
+  income: 75000,
+  filing_status: "single"
+};
+
+const calculator = schema.from(taxpayer);
+console.log(calculator.fetch('tax_owed'));
+
+const results = calculator.slice('taxable_income', 'tax_owed');
+```
+
+## Export Methods
+
+### Command Line
+
+```bash
+bundle exec kumi --export-js output.js SchemaClass
+```
+
+### Programmatic
+
+```ruby
+Kumi::Js.export_to_file(MySchema, "schema.js")
+
+js_code = Kumi::Js.compile(MySchema)
+File.write("output.js", js_code)
+```
+
+## JavaScript API
+
+### schema.from(input)
+
+Creates runner instance.
+
+```javascript
+const runner = schema.from({ income: 50000, status: "single" });
+```
+
+### runner.fetch(key)
+
+Returns computed value. Results are cached.
+
+```javascript
+const tax = runner.fetch('tax_owed');
+```
+
+### runner.slice(...keys)
+
+Returns multiple values.
+
+```javascript
+const results = runner.slice('taxable_income', 'tax_owed');
+// Returns: { taxable_income: 35400, tax_owed: 7788 }
+```
+
+### runner.functionsUsed
+
+Array of functions used by the schema.
+
+```javascript
+console.log(runner.functionsUsed); // ["max", "subtract", "multiply"]
+```
+
+## Function Optimization
+
+The transpiler only includes functions actually used by the schema.
+
+Example schema using 4 functions generates ~3 KB instead of ~8 KB with all 67 functions.
+
+## Browser Compatibility
+
+- ES6+ (Chrome 60+, Firefox 55+, Safari 10+) 
+- Modern bundlers (Webpack, Rollup, Vite)
+- Node.js 12+
+
+## Limitations
+
+- No `explain()` method (Ruby only)
+- Custom Ruby functions need JavaScript equivalents
+
+## Module Formats
+
+Generated JavaScript supports:
+- CommonJS (`require()`)
+- ES Modules (`import`) 
+- Global variables (browser)
+
+## Minification
+
+Use production minifiers like Terser or UglifyJS for smaller bundles.
+
+## Dual Mode Validation
+
+Set `KUMI_DUAL_MODE=true` to automatically execute both Ruby and JavaScript versions and validate they produce identical results:
+
+```bash
+KUMI_DUAL_MODE=true ruby my_script.rb
+```
+
+Every calculation is validated in real-time. Mismatches throw detailed error reports with both results for debugging.
+
+## Error Handling
+
+```javascript
+try {
+  const runner = schema.from({ invalid: "data" });
+} catch (error) {
+  console.error(error.message);
+}
+```

data/docs/features/performance.md

@@ -1,8 +1,6 @@
 # Performance
 
-TODO: Add benchmark data
-
-Processes large schemas with optimized algorithms for analysis, compilation, and execution.
+Analysis, compilation, and execution performance for large schemas.
 
 ## Execution Model
 

data/docs/features/s-expression-printer.md

@@ -42,7 +42,7 @@ The printer produces indented S-expressions that clearly show the hierarchical s
     (InputDeclaration :age :integer)
     (InputDeclaration :name :string)
   ]
-  attributes: [
+  values: [
     (ValueDeclaration :greeting
       (CallExpression :concat
         (Literal "Hello ")
@@ -65,7 +65,7 @@ The printer produces indented S-expressions that clearly show the hierarchical s
 
 The printer handles all Kumi AST node types:
 
-- **Root** - Schema container with inputs, attributes, and traits
+- **Root** - Schema container with inputs, values, and traits
 - **Declarations** - InputDeclaration, ValueDeclaration, TraitDeclaration
 - **Expressions** - CallExpression, ArrayExpression, CascadeExpression, CaseExpression
 - **References** - InputReference, InputElementReference, DeclarationReference

data/docs/schema_metadata.md

@@ -1,6 +1,6 @@
 # Schema Metadata
 
-Kumi's SchemaMetadata interface provides structured access to analyzed schema information for building external tools like form generators, documentation systems, and analysis utilities.
+Kumi's SchemaMetadata interface accesses analyzed schema information for building external tools like form generators, documentation systems, and analysis utilities.
 
 ## Primary Interface
 
@@ -10,7 +10,7 @@ SchemaMetadata is the main interface for extracting metadata from Kumi schemas:
 metadata = MySchema.schema_metadata
 ```
 
-See the comprehensive API documentation in the SchemaMetadata class for detailed method documentation, examples, and usage patterns.
+See the API documentation in the SchemaMetadata class for method documentation, examples, and usage patterns.
 
 ## Processed Metadata (Tool-Friendly)
 
@@ -83,24 +83,24 @@ metadata.values
 # }
 ```
 
-### Clean Public Interface Examples
+### Public Interface Examples
 ```ruby
-# Processed dependency information (clean hashes)
+# Processed dependency information
 metadata.dependencies
 # => { :tax_amount => [{ to: :income, conditional: false }, { to: :tax_rate, conditional: false }] }
 
-# Processed declaration metadata (clean hashes)
+# Processed declaration metadata
 metadata.declarations  
 # => { :adult => { type: :trait, expression: ">=(input.age, 18)" }, :tax_amount => { type: :value, expression: "multiply(input.income, tax_rate)" } }
 
-# Type inference results (clean data)
+# Type inference results
 metadata.inferred_types
 # => { :adult => :boolean, :tax_amount => :float, :item_totals => { array: :float } }
 ```
 
 ### Raw Analyzer State (Advanced Usage)
 ```ruby
-# Complete raw state hash with internal objects (AST nodes, Edge objects)
+# Raw state hash with internal objects (AST nodes, Edge objects)
 metadata.analyzer_state
 # => { declarations: {AST nodes...}, dependencies: {Edge objects...}, ... }
 ```

data/examples/deep_schema_compilation_and_evaluation_benchmark.rb

@@ -86,21 +86,27 @@ puts
 # ------------------------------------------------------------------
 Benchmark.ips do |x|
   schemas.each do |d, schema|
-    runner = schema.from(seed: 0)          # memoised runner
-    x.report("eval #{d}-deep") { runner[:final_result] }
+    # 1) HOT (memoized): expect ~flat, nanosecond-level if cached
+    hot = schema.from(seed: 0)
+    x.report("HOT fetch #{d}-deep") do
+      hot[:final_result]
+    end
+
+    # 2) COLD via UPDATE (no memoized result): change a dependent input each iter
+    upd = schema.from(seed: 0)
+    i = 0
+    x.report("COLD update #{d}-deep") do
+      i += 1
+      upd.update(seed: i)      # invalidates v0..vN; forces recompute
+      upd[:final_result]
+    end
+
+    # 3) COLD new runner (includes construction)
+    prng = Random.new(42)
+    x.report("COLD new #{d}-deep") do
+      r = schema.from(seed: prng.rand(1_000_000))
+      r[:final_result]
+    end
   end
   x.compare!
 end
-# Warming up --------------------------------------
-#         eval 50-deep   222.000 i/100ms
-#        eval 100-deep    57.000 i/100ms
-#        eval 150-deep    26.000 i/100ms
-# Calculating -------------------------------------
-#         eval 50-deep      2.166k (± 1.9%) i/s  (461.70 μs/i) -     10.878k in   5.024320s
-#        eval 100-deep    561.698 (± 1.4%) i/s    (1.78 ms/i) -      2.850k in   5.075057s
-#        eval 150-deep    253.732 (± 0.8%) i/s    (3.94 ms/i) -      1.274k in   5.021499s
-
-# Comparison:
-#         eval 50-deep:     2165.9 i/s
-#        eval 100-deep:      561.7 i/s - 3.86x  slower
-#        eval 150-deep:      253.7 i/s - 8.54x  slower

data/examples/game_of_life.rb

@@ -1,19 +1,17 @@
-$LOAD_PATH.unshift(File.join(__dir__, "..", "lib"))
 require "kumi"
 
-NEIGHBOR_DELTAS = [[-1, -1], [-1, 0], [-1, 1], [0, -1], [0, 1], [1, -1], [1, 0], [1, 1]]
 begin
   # in a block so we dont define this globally
   def neighbor_cells_sum_method(cells, row, col, height, width)
     # Calculate neighbor indices with wraparound
-    NEIGHBOR_DELTAS.map do |dr, dc|
+    [[-1, -1], [-1, 0], [-1, 1], [0, -1], [0, 1], [1, -1], [1, 0], [1, 1]].map do |dr, dc|
       neighbor_row = (row + dr) % height
       neighbor_col = (col + dc) % width
       neighbor_index = (neighbor_row * width) + neighbor_col
       cells[neighbor_index]
     end.sum
   end
-  Kumi::Core::FunctionRegistry.register_with_metadata(:neighbor_cells_sum, method(:neighbor_cells_sum_method),
+  Kumi::Registry.register_with_metadata(:neighbor_cells_sum, method(:neighbor_cells_sum_method),
                                                 return_type: :integer, arity: 5,
                                                 param_types: %i[array integer integer integer integer],
                                                 description: "Get neighbor cells for Conway's Game of Life")