kumi 0.0.10 → 0.0.12

data/docs/VECTOR_SEMANTICS.md

@@ -0,0 +1,286 @@
+# Kumi Vector Semantics — Short Guide
+
+This note documents how Kumi handles **vectorized traversal** over **arbitrary nested objects**, how **alignment/broadcasting** works, and how **reducers** and **structure functions** behave. It’s intentionally concise but hits all the sharp edges.
+
+---
+
+## Terminology
+
+* **Path** – a dot-separated traversal, e.g. `input.regions.offices.employees.salary`.
+* **Scope (axes)** – the list of array segments encountered along a path.
+  Example: for `regions.offices.employees.salary` the scope is `[:regions, :offices, :employees]`.
+* **Rank** – number of axes = `scope.length`.
+* **Index tuple** – lexicographic coordinates per axis, e.g. `[region_i, office_j, employee_k]`.
+
+**Three Laws (think of them as invariants):**
+
+1. **Enumeration**
+   `each_indexed(path).map(&:first) == ravel(path)`
+
+2. **Reconstruction**
+   `lift(to_scope, each_indexed(path))` regroups by `to_scope` (must be a prefix of `scope(path)`).
+
+3. **Counting**
+   `size(path) == ravel(path).length == each_indexed(path).count`
+
+These laws are the mental model. Everything else is just mechanics.
+
+---
+
+## Access Modes
+
+Kumi’s Access Planner emits low-level ops (`enter_hash`, `enter_array`) and supports three vector modes per path:
+
+### 1) `:materialize`
+
+Return the **original nested structure** down to that path (no enumeration).
+Good for “give me the data shaped like the input.”
+
+```ruby
+# Input (object mode)
+{
+  regions: [
+    { name: "E", offices: [{ employees: [{salary: 100}, {salary: 120}] }] },
+    { name: "D", offices: [{ employees: [{salary: 90}] }] }
+  ]
+}
+
+materialize("regions.offices.employees.salary")
+# => [[ [100,120] ], [ [90] ]]
+```
+
+### 2) `:ravel`
+
+**Enumerate elements at the next array boundary** for that path, i.e., “collect the items at this depth.”
+It is **not** NumPy’s “flatten everything.” It collects the next level.
+
+```ruby
+ravel("regions")                          # => [ {…E…}, {…D…} ]          (enumerate regions)
+ravel("regions.offices")                  # => [ {employees:[…]}, {employees:[…]} ] (each office)
+ravel("regions.offices.employees.salary") # => [ [100,120], [90] ]       (each employee group at that depth)
+```
+
+### 3) `:each_indexed`
+
+Enumerate leaf values **with** their index tuple (authoritative for `lift` and alignment):
+
+```ruby
+each_indexed("regions.offices.employees.salary")
+# => [
+#   [100, [0,0,0]], [120, [0,0,1]],
+#   [ 90, [1,0,0]]
+# ]
+```
+
+---
+
+## Lift (Regroup by prefix)
+
+`lift(to_scope)` turns a vector-of-rows (from `each_indexed`) into a nested array grouped by `to_scope`.
+
+```ruby
+# Given values from each_indexed above:
+lift([:regions],   …) # => [ [100,120], [90] ]
+lift([:regions,:offices], …) # => [ [[100,120]], [[90]] ]
+lift([:regions,:offices,:employees], …) # => [ [[[100,120]]], [[[90]]] ]
+```
+
+* `to_scope` must be a **prefix** of the vector’s `scope`.
+* Depth is derived mechanically from index arity; VM doesn’t guess.
+
+---
+
+## Alignment & Broadcasting
+
+When mapping a function over multiple arguments, Kumi:
+
+1. Picks a **carrier** vector (the one with the longest scope).
+2. **Aligns** other vectors to the carrier if they are **prefix-compatible** (same axes prefix).
+3. **Broadcasts** scalars across the carrier.
+
+If scopes aren’t prefix-compatible, lowering raises:
+`cross-scope map without join: [:a] vs [:b,:c]`
+
+```ruby
+# price, quantity both scope [:items]
+final = price * quantity             # zip by position (same scope)
+
+# Broadcast scalar across [:items]
+discounted = price * 0.9
+
+# Align prefix [:regions] to carrier [:regions,:offices]
+aligned_tax = align_to(offices_subtotals, regions_tax)
+total = offices_subtotals * (1 - aligned_tax)
+```
+
+---
+
+## Structure Functions vs Reducers
+
+* **Reducers** collapse a vector to a **scalar** (e.g., `sum`, `min`, `avg`).
+  Lowering selects a vector argument and emits a `Reduce`.
+
+* **Structure functions** observe or reshape **structure** (e.g., `size`, `flatten`, `count_across`).
+  Lowering usually uses a `:ravel` plan and a plain `Map` (no indices required).
+
+### Laws for `size` and `flatten`
+
+* `size(path) == ravel(path).length` (Counting Law)
+* `flatten(path)` flattens nested arrays (by default all levels; use `flatten_one` for one level).
+
+---
+
+## End-to-End Mini Examples
+
+### A. Simple vector math + reducers (object access)
+
+```ruby
+module Cart
+  extend Kumi::Schema
+  schema do
+    input do
+      array :items do
+        float :price
+        integer :qty
+      end
+      float :shipping_threshold
+    end
+
+    value :subtotals, input.items.price * input.items.qty
+    value :subtotal,  fn(:sum, subtotals)
+    value :shipping,  subtotal > input.shipping_threshold ? 0.0 : 9.99
+    value :total,     subtotal + shipping
+  end
+end
+
+data = {
+  items: [{price: 100.0, qty: 2}, {price: 200.0, qty: 1}],
+  shipping_threshold: 50.0
+}
+
+r = Cart.from(data)
+r[:subtotals] # => [200.0, 200.0]  (vector map)
+r[:subtotal]  # => 400.0           (reducer)
+r[:shipping]  # => 0.0
+r[:total]     # => 400.0
+```
+
+**Internal truths**:
+
+* `each_indexed(input.items.price)` → `[[100.0,[0]],[200.0,[1]]]`
+* `size(input.items)` → `2` because `ravel(input.items)` has length 2.
+
+### B. Mixed scopes + alignment
+
+```ruby
+module Regions
+  extend Kumi::Schema
+  schema do
+    input do
+      array :regions do
+        float :tax
+        array :offices do
+          array :employees do
+            float :salary
+          end
+        end
+      end
+    end
+
+    value :office_payrolls, fn(:sum, input.regions.offices.employees.salary)   # vector reduce per office
+    value :taxed, office_payrolls * (1 - input.regions.tax) # tax (align regions.tax to [:regions,:offices])
+  end
+end
+
+# Alignment rule: regions.tax (scope [:regions]) aligns to office_payrolls (scope [:regions,:offices])
+```
+
+### C. Element access (pure arrays) + structure functions
+
+```ruby
+module Cube
+  extend Kumi::Schema
+  schema do
+    input do
+      array :cube do
+        element :array, :layer do
+          element :array, :row do
+            element :float, :cell
+          end
+        end
+      end
+    end
+
+    value :layers,      fn(:size, input.cube)                 # == ravel(input.cube).length
+    value :matrices,    fn(:size, input.cube.layer)           # enumerate at next depth
+    value :rows,        fn(:size, input.cube.layer.row)
+    value :all_values,  fn(:flatten, input.cube.layer.row.cell)
+    value :total,       fn(:sum, all_values)
+  end
+end
+
+data = { cube: [ [[1,2],[3]], [[4]] ] }
+
+# ravel views (intuition)
+# ravel(cube)                => [ [[1,2],[3]], [[4]] ]
+# ravel(cube.layer)          => [ [1,2], [3], [4] ]
+# ravel(cube.layer.row)      => [ 1, 2, 3, 4 ]
+# ravel(cube.layer.row.cell) => [ 1, 2, 3, 4 ]  (same leaf)
+
+c = Cube.from(data)
+c[:layers]     # => 2
+c[:matrices]   # => 3
+c[:rows]       # => 4
+c[:all_values] # => [1,2,3,4]
+c[:total]      # => 10
+```
+
+---
+
+## Planner & VM: Who does what?
+
+* **Planner**: Emits deterministic `enter_hash`/`enter_array` sequences per path and mode.
+
+  * For element edges (inline array aliases), it **does not** emit `enter_hash`.
+  * For `:each_indexed` / `:ravel`, it appends a terminal `enter_array` **only if** the final node is an array.
+* **Lowerer**: Decides plans (`:ravel`, `:each_indexed`, `:materialize`), inserts `align_to`, emits `lift` at declaration boundary when a vector result should be exposed as a scalar nested array.
+* **VM**: Purely mechanical:
+
+  * `broadcast_scalar` for scalar→vec expansion,
+  * `zip_same_scope` when scopes match,
+  * `align_to` for prefix alignment,
+  * `group_rows` inside `lift` to reconstruct prefixes.
+
+No type sniffing or guesses: the IR is the source of truth.
+
+---
+
+## Jagged & Sparse Arrays
+
+* Ordering is **lexicographic by index tuple** (stable).
+* No padding is introduced; missing branches are just… missing.
+* `align_to(..., on_missing: :error|:nil)` enforces policy.
+
+---
+
+## Error Policies
+
+For missing keys/arrays, accessors obey policy:
+
+* `:error` (default) – raise descriptive error with the path/mode.
+* `:skip` – drop the missing branch (useful in ravels).
+* `:yield_nil` – emit `nil` in place (preserves cardinality).
+
+Document these on any user-facing accessor.
+
+---
+
+## Quick Cheatsheet
+
+* Use **`ravel(path)`** to “list the things at this level.”
+* Use **`each_indexed(path)`** when you need `(value, idx)` pairs for joins/regroup.
+* Use **`lift(to_scope, each_indexed(path))`** to reconstruct nested structure.
+* **Reducers** (e.g., `sum`, `avg`, `min`) consume the raveled view of their argument.
+* **Structure functions** (e.g., `size`, `flatten`, `flatten_one`, `count_across`) operate on structure at that depth and usually compile via `:ravel`.
+
+Keep the three laws in mind and Kumi’s behavior is predictable—even over deeply nested, heterogeneous data.

data/docs/features/hierarchical-broadcasting.md

@@ -67,6 +67,72 @@ value :cell_data, input.cube.layer.row.cell     # 1D values
 - **Ranked polymorphism**: Same operations work across different dimensional arrays
 - **Clean code**: `fn(:size, input.cube.layer.row)` instead of `fn(:size, fn(:flatten_one, input.cube.layer))`
 
+### Dynamic Hash Elements with `element :any`
+
+For arrays containing hash data with flexible or unknown structure, use `element :any` to access dynamic hash content:
+
+```ruby
+input do
+  array :api_responses do
+    element :any, :response_data    # Flexible hash structure
+  end
+  
+  array :user_events do
+    element :any, :event_data       # Dynamic event properties (includes event_type)
+  end
+end
+
+# Access hash fields using fn(:fetch)
+value :response_codes, fn(:fetch, input.api_responses.response_data, "status")
+value :error_messages, fn(:fetch, input.api_responses.response_data, "error")
+value :event_types, fn(:fetch, input.user_events.event_data, "event_type")
+value :user_ids, fn(:fetch, input.user_events.event_data, "user_id")
+value :timestamps, fn(:fetch, input.user_events.event_data, "timestamp")
+
+# Mathematical operations on extracted values
+value :avg_response_time, fn(:mean, fn(:fetch, input.api_responses.response_data, "response_time"))
+value :total_events, fn(:size, input.user_events.event_data)
+
+# Traits and cascades with dynamic data
+trait :has_errors, fn(:any?, fn(:fetch, input.api_responses.response_data, "status") >= 400)
+trait :recent_events, fn(:any?, fn(:fetch, input.user_events.event_data, "timestamp") > 1640995200)
+
+value :system_status do
+  on has_errors, "Error State"
+  on recent_events, "Active"
+  base "Idle"
+end
+```
+
+**When to use `element :any`:**
+- API responses with varying JSON schemas
+- Configuration files with flexible key-value structures
+- Event data where properties vary by event type
+- Legacy systems where data structure may change
+- Prototyping when exact hash structure is unknown
+
+**Comparison with Hash Objects:**
+
+| Approach | Use Case | Flexibility | Type Safety |
+|----------|----------|-------------|-------------|
+| `hash :field do ... end` | Known structure, strong typing | Limited | High |
+| `element :any, :field` | Unknown/flexible structure | High | Low |
+
+```ruby
+# Known structure - use hash objects
+array :orders do
+  hash :customer do
+    string :name
+    string :email
+  end
+end
+
+# Unknown/flexible structure - use element :any
+array :api_calls do
+  element :any, :response    # Could be any JSON structure
+end
+```
+
 ## Business Use Cases
 
 Element access mode is essential for common business scenarios involving simple nested arrays:
@@ -193,7 +259,7 @@ The type system automatically infers appropriate types for broadcasted operation
 
 ### Analysis Layer  
 - **BroadcastDetector** - Identifies vectorized vs scalar operations
-- **TypeInferencer** - Infers types for array element access patterns
+- **TypeInferencerPass** - Infers types for array element access patterns
 
 ### Compilation Layer
 - **Automatic Dispatch** - Maps element-wise operations to array map functions

data/docs/features/input-declaration-system.md

@@ -14,10 +14,26 @@ schema do
     array   :tags, elem: { type: :string }
     hash    :metadata, key: { type: :string }, val: { type: :any }
     any     :flexible
+    
+    # Structured arrays with defined fields
+    array :orders do
+      hash :customer do
+        string :name
+        string :email
+      end
+      float :total
+    end
+    
+    # Dynamic arrays with flexible elements
+    array :api_responses do
+      element :any, :response_data    # For unknown/flexible hash structures
+    end
   end
   
   trait :adult, (input.age >= 18)
   value :status, input.verified ? "verified" : "pending"
+  value :customer_emails, input.orders.customer.email                           # Structured access
+  value :response_codes, fn(:fetch, input.api_responses.response_data, "status") # Dynamic access
 end
 ```
 

data/docs/features/s-expression-printer.md

@@ -42,7 +42,7 @@ The printer produces indented S-expressions that clearly show the hierarchical s
     (InputDeclaration :age :integer)
     (InputDeclaration :name :string)
   ]
-  attributes: [
+  values: [
     (ValueDeclaration :greeting
       (CallExpression :concat
         (Literal "Hello ")
@@ -65,7 +65,7 @@ The printer produces indented S-expressions that clearly show the hierarchical s
 
 The printer handles all Kumi AST node types:
 
-- **Root** - Schema container with inputs, attributes, and traits
+- **Root** - Schema container with inputs, values, and traits
 - **Declarations** - InputDeclaration, ValueDeclaration, TraitDeclaration
 - **Expressions** - CallExpression, ArrayExpression, CascadeExpression, CaseExpression
 - **References** - InputReference, InputElementReference, DeclarationReference

data/lib/kumi/analyzer.rb

@@ -7,15 +7,19 @@ module Kumi
     DEFAULT_PASSES = [
       Core::Analyzer::Passes::NameIndexer,                     # 1. Finds all names and checks for duplicates.
       Core::Analyzer::Passes::InputCollector,                  # 2. Collects field metadata from input declarations.
-      Core::Analyzer::Passes::DeclarationValidator,            # 3. Checks the basic structure of each rule.
-      Core::Analyzer::Passes::SemanticConstraintValidator,     # 4. Validates DSL semantic constraints at AST level.
-      Core::Analyzer::Passes::DependencyResolver,              # 5. Builds the dependency graph with conditional dependencies.
-      Core::Analyzer::Passes::UnsatDetector,                   # 6. Detects unsatisfiable constraints and analyzes cascade mutual exclusion.
-      Core::Analyzer::Passes::Toposorter,                      # 7. Creates the final evaluation order, allowing safe cycles.
-      Core::Analyzer::Passes::BroadcastDetector, # 8. Detects which operations should be broadcast over arrays (must run before type inference).
-      Core::Analyzer::Passes::TypeInferencer,                  # 9. Infers types for all declarations (uses vectorization metadata).
-      Core::Analyzer::Passes::TypeConsistencyChecker,          # 10. Validates declared vs inferred type consistency.
-      Core::Analyzer::Passes::TypeChecker                      # 11. Validates types using inferred information.
+      Core::Analyzer::Passes::DeclarationValidator,            # 4. Checks the basic structure of each rule.
+      Core::Analyzer::Passes::SemanticConstraintValidator,     # 5. Validates DSL semantic constraints at AST level.
+      Core::Analyzer::Passes::DependencyResolver,              # 6. Builds the dependency graph with conditional dependencies.
+      Core::Analyzer::Passes::UnsatDetector,                   # 7. Detects unsatisfiable constraints and analyzes cascade mutual exclusion.
+      Core::Analyzer::Passes::Toposorter,                      # 8. Creates the final evaluation order, allowing safe cycles.
+      Core::Analyzer::Passes::BroadcastDetector,               # 9. Detects which operations should be broadcast over arrays.
+      Core::Analyzer::Passes::TypeInferencerPass,              # 10. Infers types for all declarations (uses vectorization metadata).
+      Core::Analyzer::Passes::TypeConsistencyChecker,          # 11. Validates declared vs inferred type consistency.
+      Core::Analyzer::Passes::TypeChecker,                     # 12. Validates types using inferred information.
+      Core::Analyzer::Passes::InputAccessPlannerPass,          # 13. Plans access strategies for input fields.
+      Core::Analyzer::Passes::ScopeResolutionPass,             # 14. Plans execution scope and lifting needs for declarations.
+      Core::Analyzer::Passes::JoinReducePlanningPass,          # 15. Plans join/reduce operations (Generates IR Structs)
+      Core::Analyzer::Passes::LowerToIRPass # 16. Lowers the schema to IR (Generates IR Structs)
     ].freeze
 
     def self.analyze!(schema, passes: DEFAULT_PASSES, **opts)
@@ -33,7 +37,13 @@ module Kumi
         begin
           state = pass_instance.run(errors)
         rescue StandardError => e
-          errors << Core::ErrorReporter.create_error(e.message, location: nil, type: :semantic)
+          # TODO: - GREATLY improve this, need to capture the context of the error
+          # and the pass that failed and line number if relevant
+          pass_name = pass_class.name.split("::").last
+          message = "Error in Analysis Pass(#{pass_name}): #{e.message}"
+          errors << Core::ErrorReporter.create_error(message, location: nil, type: :semantic, backtrace: e.backtrace)
+
+          raise
         end
       end
       state
@@ -41,11 +51,14 @@ module Kumi
 
     def self.handle_analysis_errors(errors)
       type_errors = errors.select { |e| e.type == :type }
+      semantic_errors = errors.select { |e| e.type == :semantic }
       first_error_location = errors.first.location
 
       raise Errors::TypeError.new(format_errors(errors), first_error_location) if type_errors.any?
 
-      raise Errors::SemanticError.new(format_errors(errors), first_error_location)
+      raise Errors::SemanticError.new(format_errors(errors), first_error_location) if first_error_location || semantic_errors
+
+      raise Errors::AnalysisError.new(format_errors(errors))
     end
 
     def self.create_analysis_result(state)
@@ -63,7 +76,16 @@ module Kumi
     def self.format_errors(errors)
       return "" if errors.empty?
 
-      errors.map(&:to_s).join("\n")
+      backtrace = errors.first.backtrace
+
+      message = errors.map(&:to_s).join("\n")
+
+      message.tap do |msg|
+        if backtrace && !backtrace.empty?
+          msg << "\n\nBacktrace:\n"
+          msg << backtrace[0..10].join("\n") # Limit to first 10 lines for readability
+        end
+      end
     end
   end
 end

data/lib/kumi/compiler.rb

@@ -3,11 +3,6 @@
 module Kumi
   # Compiles an analyzed schema into executable lambdas
   class Compiler < Core::CompilerBase
-    include Kumi::Core::Compiler::ReferenceCompiler
-    include Kumi::Core::Compiler::PathTraversalCompiler
-    include Kumi::Core::Compiler::ExpressionCompiler
-    include Kumi::Core::Compiler::FunctionInvoker
-
     def self.compile(schema, analyzer:)
       new(schema, analyzer).compile
     end
@@ -18,13 +13,8 @@ module Kumi
     end
 
     def compile
-      build_index
-      @analysis.topo_order.each do |name|
-        decl = @index[name] or raise("Unknown binding #{name}")
-        compile_declaration(decl)
-      end
-
-      Core::CompiledSchema.new(@bindings.freeze)
+      # Switch to LIR: Use the analysis state instead of old compilation
+      Runtime::Executable.from_analysis(@analysis.state)
     end
   end
 end