kumi 0.0.8 → 0.0.10

data/docs/FUNCTIONS.md

@@ -10,6 +10,8 @@ Kumi provides a rich library of built-in functions for use within `value` and `t
   * **Usage**: `fn(:and, boolean1, boolean2, ...)` → `boolean`
 * **`any?`**: Check if any element in collection is truthy
   * **Usage**: `fn(:any?, array(any) arg1)` → `boolean`
+* **`cascade_and`**: Element-wise AND for arrays with same nested structure
+  * **Usage**: `fn(:cascade_and, boolean1, boolean2, ...)` → `boolean`
 * **`none?`**: Check if no elements in collection are truthy
   * **Usage**: `fn(:none?, array(any) arg1)` → `boolean`
 * **`not`**: Logical NOT
@@ -52,14 +54,14 @@ Kumi provides a rich library of built-in functions for use within `value` and `t
   * **Usage**: `fn(:modulo, float arg1, float arg2)` → `float`
 * **`multiply`**: Multiply two numbers
   * **Usage**: `fn(:multiply, float arg1, float arg2)` → `float`
+* **`piecewise_sum`**: Accumulate over tiered ranges; returns [sum, marginal_rate]
+  * **Usage**: `fn(:piecewise_sum, float arg1, array(float) arg2, array(float) arg3)` → `array(float)`
 * **`power`**: Raise first number to power of second
   * **Usage**: `fn(:power, float arg1, float arg2)` → `float`
 * **`round`**: Round number to specified precision
   * **Usage**: `fn(:round, float1, float2, ...)` → `float`
 * **`subtract`**: Subtract second number from first
   * **Usage**: `fn(:subtract, float arg1, float arg2)` → `float`
-* **`piecewise_sum`**: Accumulate over tiered ranges; returns [sum, marginal_rate]
-  * **Usage**: `fn(:piecewise_sum, float arg1, array(float) arg2, array(float) arg3)` → `array(float)`
 
 ## String Functions
 
@@ -67,16 +69,22 @@ Kumi provides a rich library of built-in functions for use within `value` and `t
   * **Usage**: `fn(:capitalize, string arg1)` → `string`
 * **`concat`**: Concatenate multiple strings
   * **Usage**: `fn(:concat, string1, string2, ...)` → `string`
+* **`contains?`**: Check if string contains substring
+  * **Usage**: `fn(:contains?, string arg1, string arg2)` → `boolean`
 * **`downcase`**: Convert string to lowercase
   * **Usage**: `fn(:downcase, string arg1)` → `string`
 * **`end_with?`**: Check if string ends with suffix
   * **Usage**: `fn(:end_with?, string arg1, string arg2)` → `boolean`
-* **`include?`**: Check if collection includes element
-  * **Usage**: `fn(:include?, array(any) arg1, any arg2)` → `boolean`
-* **`length`**: Get collection length
-  * **Usage**: `fn(:length, array(any) arg1)` → `integer`
+* **`includes?`**: Check if string contains substring
+  * **Usage**: `fn(:includes?, string arg1, string arg2)` → `boolean`
+* **`length`**: Get string length
+  * **Usage**: `fn(:length, string arg1)` → `integer`
 * **`start_with?`**: Check if string starts with prefix
   * **Usage**: `fn(:start_with?, string arg1, string arg2)` → `boolean`
+* **`string_include?`**: Check if string contains substring
+  * **Usage**: `fn(:string_include?, string arg1, string arg2)` → `boolean`
+* **`string_length`**: Get string length
+  * **Usage**: `fn(:string_length, string arg1)` → `integer`
 * **`strip`**: Remove leading and trailing whitespace
   * **Usage**: `fn(:strip, string arg1)` → `string`
 * **`upcase`**: Convert string to uppercase
@@ -84,20 +92,54 @@ Kumi provides a rich library of built-in functions for use within `value` and `t
 
 ## Collection Functions
 
+* **`all_across`**: Check if all elements are truthy across all nested levels
+  * **Usage**: `fn(:all_across, array(any) arg1)` → `boolean`
+* **`any_across`**: Check if any element is truthy across all nested levels
+  * **Usage**: `fn(:any_across, array(any) arg1)` → `boolean`
+* **`avg_if`**: Average values where corresponding condition is true
+  * **Usage**: `fn(:avg_if, array(float) arg1, array(boolean) arg2)` → `float`
+* **`build_array`**: Build array of given size with index values
+  * **Usage**: `fn(:build_array, integer arg1)` → `array(any)`
+* **`count_across`**: Count total elements across all nested levels
+  * **Usage**: `fn(:count_across, array(any) arg1)` → `integer`
+* **`count_if`**: Count number of true values in boolean array
+  * **Usage**: `fn(:count_if, array(boolean) arg1)` → `integer`
+* **`each_slice`**: Group array elements into subarrays of given size
+  * **Usage**: `fn(:each_slice, array arg1, integer arg2)` → `array(array)`
 * **`empty?`**: Check if collection is empty
   * **Usage**: `fn(:empty?, array(any) arg1)` → `boolean`
 * **`first`**: Get first element of collection
   * **Usage**: `fn(:first, array(any) arg1)` → `any`
+* **`flatten`**: Flatten nested arrays into a single array
+  * **Usage**: `fn(:flatten, array(any) arg1)` → `array(any)`
+* **`flatten_deep`**: Recursively flatten all nested arrays (alias for flatten)
+  * **Usage**: `fn(:flatten_deep, array(any) arg1)` → `array(any)`
+* **`flatten_one`**: Flatten nested arrays by one level only
+  * **Usage**: `fn(:flatten_one, array(any) arg1)` → `array(any)`
 * **`include?`**: Check if collection includes element
   * **Usage**: `fn(:include?, array(any) arg1, any arg2)` → `boolean`
+* **`indices`**: Generate array of indices for the collection
+  * **Usage**: `fn(:indices, array(any) arg1)` → `array(integer)`
+* **`join`**: Join array elements into string with separator
+  * **Usage**: `fn(:join, array arg1, string arg2)` → `string`
 * **`last`**: Get last element of collection
   * **Usage**: `fn(:last, array(any) arg1)` → `any`
-* **`length`**: Get collection length
-  * **Usage**: `fn(:length, array(any) arg1)` → `integer`
+* **`map_add`**: Add value to each element
+  * **Usage**: `fn(:map_add, array(float) arg1, float arg2)` → `array(float)`
+* **`map_conditional`**: Transform elements based on condition: if element == condition_value then true_value else false_value
+  * **Usage**: `fn(:map_conditional, array arg1, any arg2, any arg3, any arg4)` → `array`
+* **`map_join_rows`**: Join 2D array into string with row and column separators
+  * **Usage**: `fn(:map_join_rows, array(array) arg1, string arg2, string arg3)` → `string`
+* **`map_multiply`**: Multiply each element by factor
+  * **Usage**: `fn(:map_multiply, array(float) arg1, float arg2)` → `array(float)`
+* **`map_with_index`**: Map collection elements to [element, index] pairs
+  * **Usage**: `fn(:map_with_index, array(any) arg1)` → `array(any)`
 * **`max`**: Find maximum value in numeric collection
   * **Usage**: `fn(:max, array(float) arg1)` → `float`
 * **`min`**: Find minimum value in numeric collection
   * **Usage**: `fn(:min, array(float) arg1)` → `float`
+* **`range`**: Generate range of integers from start to finish (exclusive)
+  * **Usage**: `fn(:range, integer arg1, integer arg2)` → `array(integer)`
 * **`reverse`**: Reverse collection order
   * **Usage**: `fn(:reverse, array(any) arg1)` → `array(any)`
 * **`size`**: Get collection size
@@ -106,6 +148,8 @@ Kumi provides a rich library of built-in functions for use within `value` and `t
   * **Usage**: `fn(:sort, array(any) arg1)` → `array(any)`
 * **`sum`**: Sum all numeric elements in collection
   * **Usage**: `fn(:sum, array(float) arg1)` → `float`
+* **`sum_if`**: Sum values where corresponding condition is true
+  * **Usage**: `fn(:sum_if, array(float) arg1, array(boolean) arg2)` → `float`
 * **`unique`**: Remove duplicate elements from collection
   * **Usage**: `fn(:unique, array(any) arg1)` → `array(any)`
 

data/docs/compiler_design_principles.md

@@ -0,0 +1,86 @@
+# Compiler Design Principles
+
+## Core Principle: Smart Analyzer, Dumb Compiler
+
+The Kumi compiler follows a strict separation of concerns:
+
+### Analyzer Phase (Smart)
+- **Makes all decisions** about how operations should be executed
+- **Analyzes semantic context** to determine operation modes
+- **Pre-computes execution strategies** and stores in metadata
+- **Resolves complex logic** like nested array broadcasting, reduction flattening, etc.
+- **Produces complete instructions** for the compiler to follow
+
+### Compiler Phase (Dumb)
+- **Follows metadata instructions** without making decisions
+- **No conditional logic** based on data types, function types, or structure analysis
+- **Mechanically executes** the pre-computed strategy from analyzer
+- **Pure translation** from AST + metadata → executable functions
+
+## Examples
+
+### ❌ BAD: Compiler Making Decisions
+```ruby
+def compile_call(expr)
+  if Kumi::Registry.reducer?(expr.fn_name)
+    if nested_array_detected?(values)
+      # Compiler deciding to flatten
+      flatten_and_call(expr.fn_name, values)
+    end
+  end
+end
+```
+
+### ✅ GOOD: Compiler Following Metadata
+```ruby
+def compile_call(expr)
+  # Just read the pre-computed strategy
+  strategy = @analysis.metadata[:call_strategies][expr]
+  execute_strategy(strategy, expr)
+end
+```
+
+### ❌ BAD: Runtime Structure Analysis
+```ruby
+def vectorized_function_call(fn_name, values)
+  # Compiler analyzing structure at runtime
+  if values.any? { |v| deeply_nested?(v) }
+    apply_nested_broadcasting(fn, values)
+  end
+end
+```
+
+### ✅ GOOD: Pre-computed Broadcasting Plan
+```ruby
+def compile_element_field_reference(expr)
+  # Analyzer already determined the strategy
+  metadata = @analysis.state[:broadcasts][:nested_paths][expr.path]
+  traverse_nested_path(ctx, expr.path, metadata[:operation_mode])
+end
+```
+
+## Benefits
+
+1. **Predictable Performance**: No runtime analysis or decision-making
+2. **Easier Testing**: Compiler behavior determined entirely by metadata
+3. **Maintainable**: Complex logic isolated in analyzer passes
+4. **Extensible**: New features added by extending analyzer, not compiler
+5. **Debuggable**: All decisions visible in analyzer metadata
+
+## Implementation Pattern
+
+For any new compiler feature:
+
+1. **Analyzer Pass**: Analyze the requirement and store strategy in metadata
+2. **Metadata Schema**: Define clear data structure for the strategy
+3. **Compiler Method**: Read metadata and execute strategy mechanically
+4. **No Conditionals**: Avoid `if` statements based on runtime data in compiler
+
+## Metadata-Driven Architecture
+
+The compiler should be a pure **metadata interpreter**:
+- Input: AST + Analyzer Metadata
+- Output: Executable Functions
+- Process: Mechanical translation following metadata instructions
+
+This ensures the compiler remains simple, fast, and maintainable as the system grows in complexity.

data/docs/features/README.md

@@ -30,13 +30,33 @@ Defines expected inputs with types and constraints.
 - Domain validation at runtime
 - Separates input metadata from business logic
 
+### [Hierarchical Broadcasting](hierarchical-broadcasting.md)
+Automatic vectorization over hierarchical data structures with dual access modes.
+
+- Object access for structured business data
+- Element access for multi-dimensional arrays
+- Mixed access modes in same schema
+
 ### [Performance](performance.md)
-TODO: Add benchmark data
-Processes large schemas with optimized algorithms.
+Processes large schemas.
 
 - Result caching
 - Selective evaluation
 
+### [S-Expression Printer](s-expression-printer.md)
+Debug and inspect AST structures with readable S-expression notation output.
+
+- Visitor pattern implementation for all node types
+- Proper indentation and hierarchical structure
+- Useful for debugging schema parsing and AST analysis
+
+### [JavaScript Transpiler](javascript-transpiler.md)
+Transpiles compiled schemas to standalone JavaScript code.
+
+- Generates bundles with only required functions
+- Supports CommonJS and browser environments
+- Maintains identical behavior across platforms
+
 ## Integration
 
 - Type inference uses input declarations

data/docs/features/hierarchical-broadcasting.md

@@ -0,0 +1,349 @@
+# Hierarchical Broadcasting
+
+Automatic vectorization of operations over hierarchical data structures with two complementary access modes for different use cases.
+
+## Overview
+
+The hierarchical broadcasting system enables natural field access syntax that automatically applies operations element-wise across complex nested data structures, with intelligent detection of map vs reduce operations and support for both structured objects and raw multi-dimensional arrays.
+
+## Core Mechanism
+
+The system uses a three-stage pipeline:
+
+1. **Parser** - Creates InputElementReference AST nodes for nested field access
+2. **BroadcastDetector** - Identifies which operations should be vectorized vs scalar
+3. **Compiler** - Generates appropriate map/reduce functions based on usage context
+
+## Access Modes
+
+The system supports two complementary access modes that can be mixed within the same schema:
+
+### Object Access Mode (Default)
+For structured business data with named fields:
+
+```ruby
+input do
+  array :users do
+    string :name
+    integer :age
+  end
+end
+
+value :user_names, input.users.name     # Broadcasts over structured objects
+trait :adults, input.users.age >= 18    # Boolean array from age comparison
+```
+
+### Element Access Mode  
+For multi-dimensional raw arrays using `element` syntax with **progressive path traversal**:
+
+```ruby
+input do
+  array :cube do
+    element :array, :layer do
+      element :array, :row do
+        element :integer, :cell
+      end
+    end
+  end
+end
+
+# Progressive path traversal - each level goes deeper
+value :dimensions, [
+  fn(:size, input.cube),            # Number of layers
+  fn(:size, input.cube.layer),     # Total matrices across all layers  
+  fn(:size, input.cube.layer.row), # Total rows across all matrices
+  fn(:size, input.cube.layer.row.cell) # Total cells (leaf elements)
+]
+
+# Operations at different dimensional levels
+value :layer_data, input.cube.layer              # 3D matrices
+value :row_data, input.cube.layer.row           # 2D rows
+value :cell_data, input.cube.layer.row.cell     # 1D values
+```
+
+**Key Benefits of Progressive Path Traversal:**
+- **Intuitive syntax**: `input.cube.layer.row` naturally represents "rows within layers within cube"
+- **Direct dimensional access**: No need for complex flattening operations for basic dimensional analysis
+- **Ranked polymorphism**: Same operations work across different dimensional arrays
+- **Clean code**: `fn(:size, input.cube.layer.row)` instead of `fn(:size, fn(:flatten_one, input.cube.layer))`
+
+## Business Use Cases
+
+Element access mode is essential for common business scenarios involving simple nested arrays:
+
+```ruby
+# E-commerce: Product availability analysis
+input do
+  array :products do
+    string :name
+    array :daily_sales do        # Simple array: [12, 8, 15, 3]
+      element :integer, :units
+    end
+  end
+end
+
+# Create flags for business rules
+trait :had_slow_days, fn(:any?, input.products.daily_sales.units < 5)
+trait :consistently_strong, fn(:all?, input.products.daily_sales.units >= 10)
+
+# Progressive analysis
+value :sales_summary, [
+  fn(:size, input.products),                    # Number of products
+  fn(:size, input.products.daily_sales.units), # Total sales data points
+  fn(:sum, fn(:flatten, input.products.daily_sales.units)) # Total units sold
+]
+```
+
+### Mixed Access Modes
+Both modes work together seamlessly:
+
+```ruby
+input do
+  array :users do          # Object access
+    string :name
+    integer :age  
+  end
+  array :activity_logs do  # Element access
+    element :integer, :day
+  end
+end
+
+value :user_count, fn(:size, input.users)
+value :total_activity_days, fn(:size, fn(:flatten, input.activity_logs.day))
+```
+
+## Basic Broadcasting (Object Access)
+
+```ruby
+schema do
+  input do
+    array :line_items do
+      float   :price
+      integer :quantity
+      string  :category
+    end
+    float :tax_rate, type: :float
+  end
+
+  # Element-wise computation - broadcasts over each item
+  value :subtotals, input.line_items.price * input.line_items.quantity
+  
+  # Element-wise traits - applied to each item
+  trait :is_taxable, (input.line_items.category != "digital")
+  
+  # Conditional logic - element-wise evaluation
+  value :taxes, fn(:if, is_taxable, subtotals * input.tax_rate, 0.0)
+end
+```
+
+## Aggregation Operations
+
+Operations that consume arrays to produce scalars are automatically detected:
+
+```ruby
+schema do
+  # These aggregate the vectorized results
+  value :total_subtotal, fn(:sum, subtotals)
+  value :total_tax, fn(:sum, taxes)
+  value :grand_total, total_subtotal + total_tax
+  
+  # Statistics over arrays
+  value :avg_price, fn(:avg, input.line_items.price)
+  value :max_quantity, fn(:max, input.line_items.quantity)
+end
+```
+
+## Field Access Nesting
+
+Supports arbitrary depth field access with path building:
+
+```ruby
+schema do
+  input do
+    array :orders do
+      array :items do
+        hash :product do
+          string :name
+          float  :base_price
+        end
+        integer :quantity
+      end
+    end
+  end
+
+  # Deep field access - automatically broadcasts over nested arrays  
+  value :all_product_names, input.orders.items.product.name
+  value :total_values, input.orders.items.product.base_price * input.orders.items.quantity
+end
+```
+
+## Type Inference
+
+The type system automatically infers appropriate types for broadcasted operations:
+
+- `input.items.price` (float array) → inferred as `:float` per element
+- `input.items.price * input.items.quantity` → element-wise `:float` result
+- `fn(:sum, input.items.price)` → scalar `:float` result
+
+## Implementation Details
+
+### Parser Layer
+- **InputFieldProxy** - Handles `input.field.subfield...` with path building
+- **InputElementReference** - AST node representing array field access paths
+
+### Analysis Layer  
+- **BroadcastDetector** - Identifies vectorized vs scalar operations
+- **TypeInferencer** - Infers types for array element access patterns
+
+### Compilation Layer
+- **Automatic Dispatch** - Maps element-wise operations to array map functions
+- **Reduction Detection** - Converts aggregation functions to array reduce operations
+
+## Usage Patterns
+
+### Element-wise Operations
+```ruby
+# All of these broadcast element-wise
+value :discounted_prices, input.items.price * 0.9
+trait :expensive, (input.items.price > 100.0)  
+value :categories, input.items.category
+```
+
+### Aggregation Operations
+```ruby
+# These consume arrays to produce scalars
+value :item_count, fn(:size, input.items)
+value :total_price, fn(:sum, input.items.price)
+value :has_expensive, fn(:any?, expensive)
+```
+
+### Mixed Operations
+```ruby
+# Element-wise computation followed by aggregation
+value :line_totals, input.items.price * input.items.quantity
+value :order_total, fn(:sum, line_totals)
+value :avg_line_total, fn(:avg, line_totals)
+```
+
+### Conditional Aggregation Functions
+
+Kumi provides powerful conditional aggregation functions that make working with vectorized traits clean and intuitive:
+
+```ruby
+# Step 1: Create vectorized values and traits
+value :revenues, input.sales.price * input.sales.quantity  # → [3000.0, 2000.0, 1250.0]
+trait :expensive, input.sales.price > 100                   # → [true, true, false]
+trait :bulk_order, input.sales.quantity >= 15              # → [false, false, true]
+
+# Step 2: Use conditional aggregation functions (NEW - clean and readable)
+value :expensive_count, fn(:count_if, expensive)                    # → 2
+value :expensive_total, fn(:sum_if, revenues, expensive)            # → 5000.0
+value :expensive_average, fn(:avg_if, revenues, expensive)          # → 2500.0
+
+# Step 3: Combine conditions for complex filtering
+trait :expensive_bulk, expensive & bulk_order                       # → [false, false, false]
+value :expensive_bulk_total, fn(:sum_if, revenues, expensive_bulk)  # → 0.0
+```
+
+**Old cascade pattern** (verbose, harder to read):
+```ruby
+# OLD WAY - required verbose cascade patterns
+value :expensive_amounts do
+  on expensive, revenues
+  base 0.0
+end
+value :expensive_total, fn(:sum, expensive_amounts)
+
+value :expensive_count_markers do
+  on expensive, 1.0
+  base 0.0
+end
+value :expensive_count, fn(:sum, expensive_count_markers)
+```
+
+**New conditional functions** (clean, direct):
+```ruby
+# NEW WAY - direct and readable
+value :expensive_total, fn(:sum_if, revenues, expensive)
+value :expensive_count, fn(:count_if, expensive)
+value :expensive_average, fn(:avg_if, revenues, expensive)
+```
+
+The conditional aggregation functions are now the idiomatic way to work with boolean arrays in Kumi.
+
+## Common Patterns and Gotchas
+
+### Operator Precedence with Mixed Arithmetic
+
+**Problem**: Complex arithmetic expressions with arrays can fail due to precedence parsing:
+
+```ruby
+# This fails with confusing error message
+value :ones, input.items.price * 0 + 1
+# Error: "no implicit conversion of Integer into Array"
+
+# The expression is parsed as: (input.items.price * 0) + 1
+# Which becomes: [0.0, 0.0, 0.0] + 1 (array + scalar in wrong context)
+```
+
+**Solutions**:
+
+```ruby
+# Use explicit function calls
+value :ones, fn(:add, input.items.price * 0, 1)
+
+# Use cascade pattern (most idiomatic)
+trait :always_true, input.items.price >= 0
+value :ones do
+  on always_true, 1.0
+  base 0.0
+end
+
+# Use separate steps
+value :zeros, input.items.price * 0
+value :ones, zeros + 1
+```
+
+The cascade pattern is preferred as it's more explicit about intent and leverages Kumi's automatic scalar broadcasting.
+
+## Error Handling
+
+### Dimension Mismatch Detection
+
+Array broadcasting operations are only valid within the same array source. Attempting to broadcast across different arrays generates detailed error messages:
+
+```ruby
+schema do
+  input do
+    array :items do
+      string :name
+    end
+    array :logs do  
+      string :user_name
+    end
+  end
+
+  # This will generate a dimension mismatch error
+  trait :same_name, input.items.name == input.logs.user_name
+end
+
+# Error:
+# Cannot broadcast operation across arrays from different sources: items, logs. 
+# Problem: Multiple operands are arrays from different sources:
+#   - Operand 1 resolves to array(string) from array 'items'
+#   - Operand 2 resolves to array(string) from array 'logs'
+# Direct operations on arrays from different sources is ambiguous and not supported. 
+# Vectorized operations can only work on fields from the same array input.
+```
+
+The error messages provide:
+- **Quick Summary**: Identifies the conflicting array sources
+- **Type Information**: Shows the resolved types of each operand  
+- **Clear Explanation**: Why the operation is ambiguous and not supported
+
+## Performance Characteristics
+
+- **Single Pass** - Each array is traversed once per computation chain
+- **Lazy Evaluation** - Operations are composed into pipelines  
+- **Memory Efficient** - No intermediate array allocations for simple operations
+- **Type Safe** - Full compile-time type checking for array element operations