kumi 0.0.18 → 0.0.19

data/docs/features/hierarchical-broadcasting.md

@@ -1,415 +0,0 @@
-# Hierarchical Broadcasting
-
-Automatic vectorization of operations over hierarchical data structures with two complementary access modes for different use cases.
-
-## Overview
-
-The hierarchical broadcasting system enables natural field access syntax that automatically applies operations element-wise across complex nested data structures, with intelligent detection of map vs reduce operations and support for both structured objects and raw multi-dimensional arrays.
-
-## Core Mechanism
-
-The system uses a three-stage pipeline:
-
-1. **Parser** - Creates InputElementReference AST nodes for nested field access
-2. **BroadcastDetector** - Identifies which operations should be vectorized vs scalar
-3. **Compiler** - Generates appropriate map/reduce functions based on usage context
-
-## Access Modes
-
-The system supports two complementary access modes that can be mixed within the same schema:
-
-### Object Access Mode (Default)
-For structured business data with named fields:
-
-```ruby
-input do
-  array :users do
-    string :name
-    integer :age
-  end
-end
-
-value :user_names, input.users.name     # Broadcasts over structured objects
-trait :adults, input.users.age >= 18    # Boolean array from age comparison
-```
-
-### Element Access Mode  
-For multi-dimensional raw arrays using `element` syntax with **progressive path traversal**:
-
-```ruby
-input do
-  array :cube do
-    element :array, :layer do
-      element :array, :row do
-        element :integer, :cell
-      end
-    end
-  end
-end
-
-# Progressive path traversal - each level goes deeper
-value :dimensions, [
-  fn(:size, input.cube),            # Number of layers
-  fn(:size, input.cube.layer),     # Total matrices across all layers  
-  fn(:size, input.cube.layer.row), # Total rows across all matrices
-  fn(:size, input.cube.layer.row.cell) # Total cells (leaf elements)
-]
-
-# Operations at different dimensional levels
-value :layer_data, input.cube.layer              # 3D matrices
-value :row_data, input.cube.layer.row           # 2D rows
-value :cell_data, input.cube.layer.row.cell     # 1D values
-```
-
-**Key Benefits of Progressive Path Traversal:**
-- **Intuitive syntax**: `input.cube.layer.row` naturally represents "rows within layers within cube"
-- **Direct dimensional access**: No need for complex flattening operations for basic dimensional analysis
-- **Ranked polymorphism**: Same operations work across different dimensional arrays
-- **Clean code**: `fn(:size, input.cube.layer.row)` instead of `fn(:size, fn(:flatten_one, input.cube.layer))`
-
-### Dynamic Hash Elements with `element :any`
-
-For arrays containing hash data with flexible or unknown structure, use `element :any` to access dynamic hash content:
-
-```ruby
-input do
-  array :api_responses do
-    element :any, :response_data    # Flexible hash structure
-  end
-  
-  array :user_events do
-    element :any, :event_data       # Dynamic event properties (includes event_type)
-  end
-end
-
-# Access hash fields using fn(:fetch)
-value :response_codes, fn(:fetch, input.api_responses.response_data, "status")
-value :error_messages, fn(:fetch, input.api_responses.response_data, "error")
-value :event_types, fn(:fetch, input.user_events.event_data, "event_type")
-value :user_ids, fn(:fetch, input.user_events.event_data, "user_id")
-value :timestamps, fn(:fetch, input.user_events.event_data, "timestamp")
-
-# Mathematical operations on extracted values
-value :avg_response_time, fn(:mean, fn(:fetch, input.api_responses.response_data, "response_time"))
-value :total_events, fn(:size, input.user_events.event_data)
-
-# Traits and cascades with dynamic data
-trait :has_errors, fn(:any?, fn(:fetch, input.api_responses.response_data, "status") >= 400)
-trait :recent_events, fn(:any?, fn(:fetch, input.user_events.event_data, "timestamp") > 1640995200)
-
-value :system_status do
-  on has_errors, "Error State"
-  on recent_events, "Active"
-  base "Idle"
-end
-```
-
-**When to use `element :any`:**
-- API responses with varying JSON schemas
-- Configuration files with flexible key-value structures
-- Event data where properties vary by event type
-- Legacy systems where data structure may change
-- Prototyping when exact hash structure is unknown
-
-**Comparison with Hash Objects:**
-
-| Approach | Use Case | Flexibility | Type Safety |
-|----------|----------|-------------|-------------|
-| `hash :field do ... end` | Known structure, strong typing | Limited | High |
-| `element :any, :field` | Unknown/flexible structure | High | Low |
-
-```ruby
-# Known structure - use hash objects
-array :orders do
-  hash :customer do
-    string :name
-    string :email
-  end
-end
-
-# Unknown/flexible structure - use element :any
-array :api_calls do
-  element :any, :response    # Could be any JSON structure
-end
-```
-
-## Business Use Cases
-
-Element access mode is essential for common business scenarios involving simple nested arrays:
-
-```ruby
-# E-commerce: Product availability analysis
-input do
-  array :products do
-    string :name
-    array :daily_sales do        # Simple array: [12, 8, 15, 3]
-      element :integer, :units
-    end
-  end
-end
-
-# Create flags for business rules
-trait :had_slow_days, fn(:any?, input.products.daily_sales.units < 5)
-trait :consistently_strong, fn(:all?, input.products.daily_sales.units >= 10)
-
-# Progressive analysis
-value :sales_summary, [
-  fn(:size, input.products),                    # Number of products
-  fn(:size, input.products.daily_sales.units), # Total sales data points
-  fn(:sum, fn(:flatten, input.products.daily_sales.units)) # Total units sold
-]
-```
-
-### Mixed Access Modes
-Both modes work together seamlessly:
-
-```ruby
-input do
-  array :users do          # Object access
-    string :name
-    integer :age  
-  end
-  array :activity_logs do  # Element access
-    element :integer, :day
-  end
-end
-
-value :user_count, fn(:size, input.users)
-value :total_activity_days, fn(:size, fn(:flatten, input.activity_logs.day))
-```
-
-## Basic Broadcasting (Object Access)
-
-```ruby
-schema do
-  input do
-    array :line_items do
-      float   :price
-      integer :quantity
-      string  :category
-    end
-    float :tax_rate, type: :float
-  end
-
-  # Element-wise computation - broadcasts over each item
-  value :subtotals, input.line_items.price * input.line_items.quantity
-  
-  # Element-wise traits - applied to each item
-  trait :is_taxable, (input.line_items.category != "digital")
-  
-  # Conditional logic - element-wise evaluation
-  value :taxes, fn(:if, is_taxable, subtotals * input.tax_rate, 0.0)
-end
-```
-
-## Aggregation Operations
-
-Operations that consume arrays to produce scalars are automatically detected:
-
-```ruby
-schema do
-  # These aggregate the vectorized results
-  value :total_subtotal, fn(:sum, subtotals)
-  value :total_tax, fn(:sum, taxes)
-  value :grand_total, total_subtotal + total_tax
-  
-  # Statistics over arrays
-  value :avg_price, fn(:avg, input.line_items.price)
-  value :max_quantity, fn(:max, input.line_items.quantity)
-end
-```
-
-## Field Access Nesting
-
-Supports arbitrary depth field access with path building:
-
-```ruby
-schema do
-  input do
-    array :orders do
-      array :items do
-        hash :product do
-          string :name
-          float  :base_price
-        end
-        integer :quantity
-      end
-    end
-  end
-
-  # Deep field access - automatically broadcasts over nested arrays  
-  value :all_product_names, input.orders.items.product.name
-  value :total_values, input.orders.items.product.base_price * input.orders.items.quantity
-end
-```
-
-## Type Inference
-
-The type system automatically infers appropriate types for broadcasted operations:
-
-- `input.items.price` (float array) → inferred as `:float` per element
-- `input.items.price * input.items.quantity` → element-wise `:float` result
-- `fn(:sum, input.items.price)` → scalar `:float` result
-
-## Implementation Details
-
-### Parser Layer
-- **InputFieldProxy** - Handles `input.field.subfield...` with path building
-- **InputElementReference** - AST node representing array field access paths
-
-### Analysis Layer  
-- **BroadcastDetector** - Identifies vectorized vs scalar operations
-- **TypeInferencerPass** - Infers types for array element access patterns
-
-### Compilation Layer
-- **Automatic Dispatch** - Maps element-wise operations to array map functions
-- **Reduction Detection** - Converts aggregation functions to array reduce operations
-
-## Usage Patterns
-
-### Element-wise Operations
-```ruby
-# All of these broadcast element-wise
-value :discounted_prices, input.items.price * 0.9
-trait :expensive, (input.items.price > 100.0)  
-value :categories, input.items.category
-```
-
-### Aggregation Operations
-```ruby
-# These consume arrays to produce scalars
-value :item_count, fn(:size, input.items)
-value :total_price, fn(:sum, input.items.price)
-value :has_expensive, fn(:any?, expensive)
-```
-
-### Mixed Operations
-```ruby
-# Element-wise computation followed by aggregation
-value :line_totals, input.items.price * input.items.quantity
-value :order_total, fn(:sum, line_totals)
-value :avg_line_total, fn(:avg, line_totals)
-```
-
-### Conditional Aggregation Functions
-
-Kumi provides powerful conditional aggregation functions that make working with vectorized traits clean and intuitive:
-
-```ruby
-# Step 1: Create vectorized values and traits
-value :revenues, input.sales.price * input.sales.quantity  # → [3000.0, 2000.0, 1250.0]
-trait :expensive, input.sales.price > 100                   # → [true, true, false]
-trait :bulk_order, input.sales.quantity >= 15              # → [false, false, true]
-
-# Step 2: Use conditional aggregation functions (NEW - clean and readable)
-value :expensive_count, fn(:count_if, expensive)                    # → 2
-value :expensive_total, fn(:sum_if, revenues, expensive)            # → 5000.0
-value :expensive_average, fn(:avg_if, revenues, expensive)          # → 2500.0
-
-# Step 3: Combine conditions for complex filtering
-trait :expensive_bulk, expensive & bulk_order                       # → [false, false, false]
-value :expensive_bulk_total, fn(:sum_if, revenues, expensive_bulk)  # → 0.0
-```
-
-**Old cascade pattern** (verbose, harder to read):
-```ruby
-# OLD WAY - required verbose cascade patterns
-value :expensive_amounts do
-  on expensive, revenues
-  base 0.0
-end
-value :expensive_total, fn(:sum, expensive_amounts)
-
-value :expensive_count_markers do
-  on expensive, 1.0
-  base 0.0
-end
-value :expensive_count, fn(:sum, expensive_count_markers)
-```
-
-**New conditional functions** (clean, direct):
-```ruby
-# NEW WAY - direct and readable
-value :expensive_total, fn(:sum_if, revenues, expensive)
-value :expensive_count, fn(:count_if, expensive)
-value :expensive_average, fn(:avg_if, revenues, expensive)
-```
-
-The conditional aggregation functions are now the idiomatic way to work with boolean arrays in Kumi.
-
-## Common Patterns and Gotchas
-
-### Operator Precedence with Mixed Arithmetic
-
-**Problem**: Complex arithmetic expressions with arrays can fail due to precedence parsing:
-
-```ruby
-# This fails with confusing error message
-value :ones, input.items.price * 0 + 1
-# Error: "no implicit conversion of Integer into Array"
-
-# The expression is parsed as: (input.items.price * 0) + 1
-# Which becomes: [0.0, 0.0, 0.0] + 1 (array + scalar in wrong context)
-```
-
-**Solutions**:
-
-```ruby
-# Use explicit function calls
-value :ones, fn(:add, input.items.price * 0, 1)
-
-# Use cascade pattern (most idiomatic)
-trait :always_true, input.items.price >= 0
-value :ones do
-  on always_true, 1.0
-  base 0.0
-end
-
-# Use separate steps
-value :zeros, input.items.price * 0
-value :ones, zeros + 1
-```
-
-The cascade pattern is preferred as it's more explicit about intent and leverages Kumi's automatic scalar broadcasting.
-
-## Error Handling
-
-### Dimension Mismatch Detection
-
-Array broadcasting operations are only valid within the same array source. Attempting to broadcast across different arrays generates detailed error messages:
-
-```ruby
-schema do
-  input do
-    array :items do
-      string :name
-    end
-    array :logs do  
-      string :user_name
-    end
-  end
-
-  # This will generate a dimension mismatch error
-  trait :same_name, input.items.name == input.logs.user_name
-end
-
-# Error:
-# Cannot broadcast operation across arrays from different sources: items, logs. 
-# Problem: Multiple operands are arrays from different sources:
-#   - Operand 1 resolves to array(string) from array 'items'
-#   - Operand 2 resolves to array(string) from array 'logs'
-# Direct operations on arrays from different sources is ambiguous and not supported. 
-# Vectorized operations can only work on fields from the same array input.
-```
-
-The error messages provide:
-- **Quick Summary**: Identifies the conflicting array sources
-- **Type Information**: Shows the resolved types of each operand  
-- **Clear Explanation**: Why the operation is ambiguous and not supported
-
-## Performance Characteristics
-
-- **Single Pass** - Each array is traversed once per computation chain
-- **Lazy Evaluation** - Operations are composed into pipelines  
-- **Memory Efficient** - No intermediate array allocations for simple operations
-- **Type Safe** - Full compile-time type checking for array element operations

data/docs/features/input-declaration-system.md

@@ -1,58 +0,0 @@
-# Input Declarations
-
-Declares expected inputs with types and domain constraints, separating input metadata from business logic.
-
-## Declaration Syntax
-
-```ruby
-schema do
-  input do
-    string  :customer_name
-    integer :age, domain: 18..120
-    float   :balance, domain: 0.0..Float::INFINITY
-    boolean :verified
-    array   :tags, elem: { type: :string }
-    hash    :metadata, key: { type: :string }, val: { type: :any }
-    any     :flexible
-    
-    # Structured arrays with defined fields
-    array :orders do
-      hash :customer do
-        string :name
-        string :email
-      end
-      float :total
-    end
-    
-    # Dynamic arrays with flexible elements
-    array :api_responses do
-      element :any, :response_data    # For unknown/flexible hash structures
-    end
-  end
-  
-  trait :adult, (input.age >= 18)
-  value :status, input.verified ? "verified" : "pending"
-  value :customer_emails, input.orders.customer.email                           # Structured access
-  value :response_codes, fn(:fetch, input.api_responses.response_data, "status") # Dynamic access
-end
-```
-
-## Domain Constraints
-
-**Validation occurs at runtime:**
-```ruby
-schema.from(credit_score: 900)  # Domain: 300..850
-# => InputValidationError: Field :credit_score value 900 is outside domain 300..850
-```
-
-**Constraint types:**
-- Range domains: `domain: 18..120`
-- Array domains: `domain: %w[active inactive]`
-- Regex domains: `domain: /^[a-zA-Z]+$/`
-
-## Validation Process
-
-- Input data validated against declared field metadata
-- Type validation checks value matches declared type
-- Domain validation checks value satisfies constraints
-- Detailed error messages for violations

data/docs/features/performance.md

@@ -1,14 +0,0 @@
-# Performance
-
-Analysis, compilation, and execution performance for large schemas.
-
-## Execution Model
-
-**Compilation:**
-- Each expression compiled to executable lambda
-- Direct function calls for operations
-
-**Runtime:**
-- Result caching to avoid recomputation
-- Selective evaluation: only requested keys computed
-- Direct lambda invocation

data/docs/features/s-expression-printer.md

@@ -1,77 +0,0 @@
-# S-Expression Printer
-
-Debug and inspect Kumi AST structures with readable S-expression notation output.
-
-## Overview
-
-The S-Expression Printer provides a clean, structured way to visualize Kumi's Abstract Syntax Tree (AST) nodes in traditional Lisp-style S-expression format. This is particularly useful for debugging schema parsing, understanding AST structure, and analyzing complex expressions.
-
-## Usage
-
-```ruby
-require 'kumi/support/s_expression_printer'
-
-# Print any AST node
-Kumi::Support::SExpressionPrinter.print(node)
-
-# Print a complete schema AST
-module MySchema
-  extend Kumi::Schema
-  
-  schema do
-    input do
-      integer :age
-      string :name
-    end
-    
-    trait :adult, (input.age >= 18)
-    value :greeting, fn(:concat, "Hello ", input.name)
-  end
-end
-
-puts Kumi::Support::SExpressionPrinter.print(MySchema.__syntax_tree__)
-```
-
-## Output Format
-
-The printer produces indented S-expressions that clearly show the hierarchical structure:
-
-```lisp
-(Root
-  inputs: [
-    (InputDeclaration :age :integer)
-    (InputDeclaration :name :string)
-  ]
-  values: [
-    (ValueDeclaration :greeting
-      (CallExpression :concat
-        (Literal "Hello ")
-        (InputReference :name)
-      )
-    )
-  ]
-  traits: [
-    (TraitDeclaration :adult
-      (CallExpression :>=
-        (InputReference :age)
-        (Literal 18)
-      )
-    )
-  ]
-)
-```
-
-## Supported Node Types
-
-The printer handles all Kumi AST node types:
-
-- **Root** - Schema container with inputs, values, and traits
-- **Declarations** - InputDeclaration, ValueDeclaration, TraitDeclaration
-- **Expressions** - CallExpression, ArrayExpression, CascadeExpression, CaseExpression
-- **References** - InputReference, InputElementReference, DeclarationReference
-- **Literals** - Literal values (strings, numbers, booleans)
-- **Collections** - Arrays and HashExpression nodes
-
-## Implementation
-
-Built as a visitor pattern class that traverses AST nodes recursively, with each node type having its own specialized formatting method. The printer preserves proper indentation and handles nested structures gracefully.

data/docs/schema_metadata/broadcasts.md

@@ -1,53 +0,0 @@
-# Broadcasts Metadata
-
-Array broadcasting operation analysis for vectorized computations.
-
-## Structure
-
-```ruby
-state[:broadcasts] = {
-  array_fields: Hash,
-  vectorized_operations: Hash,
-  reduction_operations: Hash
-}
-```
-
-## Array Fields
-
-```ruby
-array_fields: {
-  :line_items => {
-    element_fields: [:price, :quantity, :name],
-    element_types: { price: :float, quantity: :integer, name: :string }
-  }
-}
-```
-
-## Vectorized Operations
-
-```ruby
-vectorized_operations: {
-  :item_totals => {
-    operation: :multiply,
-    vectorized_args: { 0 => true, 1 => true }
-  }
-}
-```
-
-## Reduction Operations
-
-```ruby
-reduction_operations: {
-  :total_amount => {
-    function: :sum,
-    source: :array_field
-  }
-}
-```
-
-## Usage
-
-- Compiler optimizations
-- Parallel execution
-- Type inference
-- Performance analysis

data/docs/schema_metadata/cascades.md

@@ -1,45 +0,0 @@
-# Cascades Metadata
-
-Cascade mutual exclusion analysis for safe conditional cycles.
-
-## Structure
-
-```ruby
-state[:cascades] = {
-  cascade_name => {
-    condition_traits: Array,
-    condition_count: Integer,
-    all_mutually_exclusive: Boolean,
-    exclusive_pairs: Integer,
-    total_pairs: Integer
-  }
-}
-```
-
-## Example
-
-```ruby
-{
-  :tax_rate => {
-    condition_traits: [:single, :married],
-    condition_count: 2,
-    all_mutually_exclusive: true,
-    exclusive_pairs: 1,
-    total_pairs: 1
-  }
-}
-```
-
-## Fields
-
-- `condition_traits`: Trait names used in cascade conditions
-- `all_mutually_exclusive`: Whether all condition pairs are exclusive
-- `exclusive_pairs`: Count of mutually exclusive pairs
-- `total_pairs`: Total possible pairs
-
-## Usage
-
-- Cycle safety analysis
-- Topological sorting
-- Optimization detection
-- Dependency validation