kumi 0.0.18 → 0.0.20

data/docs/VECTOR_SEMANTICS.md

@@ -1,286 +0,0 @@
-# Kumi Vector Semantics — Short Guide
-
-This note documents how Kumi handles **vectorized traversal** over **arbitrary nested objects**, how **alignment/broadcasting** works, and how **reducers** and **structure functions** behave. It’s intentionally concise but hits all the sharp edges.
-
----
-
-## Terminology
-
-* **Path** – a dot-separated traversal, e.g. `input.regions.offices.employees.salary`.
-* **Scope (axes)** – the list of array segments encountered along a path.
-  Example: for `regions.offices.employees.salary` the scope is `[:regions, :offices, :employees]`.
-* **Rank** – number of axes = `scope.length`.
-* **Index tuple** – lexicographic coordinates per axis, e.g. `[region_i, office_j, employee_k]`.
-
-**Three Laws (think of them as invariants):**
-
-1. **Enumeration**
-   `each_indexed(path).map(&:first) == ravel(path)`
-
-2. **Reconstruction**
-   `lift(to_scope, each_indexed(path))` regroups by `to_scope` (must be a prefix of `scope(path)`).
-
-3. **Counting**
-   `size(path) == ravel(path).length == each_indexed(path).count`
-
-These laws are the mental model. Everything else is just mechanics.
-
----
-
-## Access Modes
-
-Kumi’s Access Planner emits low-level ops (`enter_hash`, `enter_array`) and supports three vector modes per path:
-
-### 1) `:materialize`
-
-Return the **original nested structure** down to that path (no enumeration).
-Good for “give me the data shaped like the input.”
-
-```ruby
-# Input (object mode)
-{
-  regions: [
-    { name: "E", offices: [{ employees: [{salary: 100}, {salary: 120}] }] },
-    { name: "D", offices: [{ employees: [{salary: 90}] }] }
-  ]
-}
-
-materialize("regions.offices.employees.salary")
-# => [[ [100,120] ], [ [90] ]]
-```
-
-### 2) `:ravel`
-
-**Enumerate elements at the next array boundary** for that path, i.e., “collect the items at this depth.”
-It is **not** NumPy’s “flatten everything.” It collects the next level.
-
-```ruby
-ravel("regions")                          # => [ {…E…}, {…D…} ]          (enumerate regions)
-ravel("regions.offices")                  # => [ {employees:[…]}, {employees:[…]} ] (each office)
-ravel("regions.offices.employees.salary") # => [ [100,120], [90] ]       (each employee group at that depth)
-```
-
-### 3) `:each_indexed`
-
-Enumerate leaf values **with** their index tuple (authoritative for `lift` and alignment):
-
-```ruby
-each_indexed("regions.offices.employees.salary")
-# => [
-#   [100, [0,0,0]], [120, [0,0,1]],
-#   [ 90, [1,0,0]]
-# ]
-```
-
----
-
-## Lift (Regroup by prefix)
-
-`lift(to_scope)` turns a vector-of-rows (from `each_indexed`) into a nested array grouped by `to_scope`.
-
-```ruby
-# Given values from each_indexed above:
-lift([:regions],   …) # => [ [100,120], [90] ]
-lift([:regions,:offices], …) # => [ [[100,120]], [[90]] ]
-lift([:regions,:offices,:employees], …) # => [ [[[100,120]]], [[[90]]] ]
-```
-
-* `to_scope` must be a **prefix** of the vector’s `scope`.
-* Depth is derived mechanically from index arity; VM doesn’t guess.
-
----
-
-## Alignment & Broadcasting
-
-When mapping a function over multiple arguments, Kumi:
-
-1. Picks a **carrier** vector (the one with the longest scope).
-2. **Aligns** other vectors to the carrier if they are **prefix-compatible** (same axes prefix).
-3. **Broadcasts** scalars across the carrier.
-
-If scopes aren’t prefix-compatible, lowering raises:
-`cross-scope map without join: [:a] vs [:b,:c]`
-
-```ruby
-# price, quantity both scope [:items]
-final = price * quantity             # zip by position (same scope)
-
-# Broadcast scalar across [:items]
-discounted = price * 0.9
-
-# Align prefix [:regions] to carrier [:regions,:offices]
-aligned_tax = align_to(offices_subtotals, regions_tax)
-total = offices_subtotals * (1 - aligned_tax)
-```
-
----
-
-## Structure Functions vs Reducers
-
-* **Reducers** collapse a vector to a **scalar** (e.g., `sum`, `min`, `avg`).
-  Lowering selects a vector argument and emits a `Reduce`.
-
-* **Structure functions** observe or reshape **structure** (e.g., `size`, `flatten`, `count_across`).
-  Lowering usually uses a `:ravel` plan and a plain `Map` (no indices required).
-
-### Laws for `size` and `flatten`
-
-* `size(path) == ravel(path).length` (Counting Law)
-* `flatten(path)` flattens nested arrays (by default all levels; use `flatten_one` for one level).
-
----
-
-## End-to-End Mini Examples
-
-### A. Simple vector math + reducers (object access)
-
-```ruby
-module Cart
-  extend Kumi::Schema
-  schema do
-    input do
-      array :items do
-        float :price
-        integer :qty
-      end
-      float :shipping_threshold
-    end
-
-    value :subtotals, input.items.price * input.items.qty
-    value :subtotal,  fn(:sum, subtotals)
-    value :shipping,  subtotal > input.shipping_threshold ? 0.0 : 9.99
-    value :total,     subtotal + shipping
-  end
-end
-
-data = {
-  items: [{price: 100.0, qty: 2}, {price: 200.0, qty: 1}],
-  shipping_threshold: 50.0
-}
-
-r = Cart.from(data)
-r[:subtotals] # => [200.0, 200.0]  (vector map)
-r[:subtotal]  # => 400.0           (reducer)
-r[:shipping]  # => 0.0
-r[:total]     # => 400.0
-```
-
-**Internal truths**:
-
-* `each_indexed(input.items.price)` → `[[100.0,[0]],[200.0,[1]]]`
-* `size(input.items)` → `2` because `ravel(input.items)` has length 2.
-
-### B. Mixed scopes + alignment
-
-```ruby
-module Regions
-  extend Kumi::Schema
-  schema do
-    input do
-      array :regions do
-        float :tax
-        array :offices do
-          array :employees do
-            float :salary
-          end
-        end
-      end
-    end
-
-    value :office_payrolls, fn(:sum, input.regions.offices.employees.salary)   # vector reduce per office
-    value :taxed, office_payrolls * (1 - input.regions.tax) # tax (align regions.tax to [:regions,:offices])
-  end
-end
-
-# Alignment rule: regions.tax (scope [:regions]) aligns to office_payrolls (scope [:regions,:offices])
-```
-
-### C. Element access (pure arrays) + structure functions
-
-```ruby
-module Cube
-  extend Kumi::Schema
-  schema do
-    input do
-      array :cube do
-        element :array, :layer do
-          element :array, :row do
-            element :float, :cell
-          end
-        end
-      end
-    end
-
-    value :layers,      fn(:size, input.cube)                 # == ravel(input.cube).length
-    value :matrices,    fn(:size, input.cube.layer)           # enumerate at next depth
-    value :rows,        fn(:size, input.cube.layer.row)
-    value :all_values,  fn(:flatten, input.cube.layer.row.cell)
-    value :total,       fn(:sum, all_values)
-  end
-end
-
-data = { cube: [ [[1,2],[3]], [[4]] ] }
-
-# ravel views (intuition)
-# ravel(cube)                => [ [[1,2],[3]], [[4]] ]
-# ravel(cube.layer)          => [ [1,2], [3], [4] ]
-# ravel(cube.layer.row)      => [ 1, 2, 3, 4 ]
-# ravel(cube.layer.row.cell) => [ 1, 2, 3, 4 ]  (same leaf)
-
-c = Cube.from(data)
-c[:layers]     # => 2
-c[:matrices]   # => 3
-c[:rows]       # => 4
-c[:all_values] # => [1,2,3,4]
-c[:total]      # => 10
-```
-
----
-
-## Planner & VM: Who does what?
-
-* **Planner**: Emits deterministic `enter_hash`/`enter_array` sequences per path and mode.
-
-  * For element edges (inline array aliases), it **does not** emit `enter_hash`.
-  * For `:each_indexed` / `:ravel`, it appends a terminal `enter_array` **only if** the final node is an array.
-* **Lowerer**: Decides plans (`:ravel`, `:each_indexed`, `:materialize`), inserts `align_to`, emits `lift` at declaration boundary when a vector result should be exposed as a scalar nested array.
-* **VM**: Purely mechanical:
-
-  * `broadcast_scalar` for scalar→vec expansion,
-  * `zip_same_scope` when scopes match,
-  * `align_to` for prefix alignment,
-  * `group_rows` inside `lift` to reconstruct prefixes.
-
-No type sniffing or guesses: the IR is the source of truth.
-
----
-
-## Jagged & Sparse Arrays
-
-* Ordering is **lexicographic by index tuple** (stable).
-* No padding is introduced; missing branches are just… missing.
-* `align_to(..., on_missing: :error|:nil)` enforces policy.
-
----
-
-## Error Policies
-
-For missing keys/arrays, accessors obey policy:
-
-* `:error` (default) – raise descriptive error with the path/mode.
-* `:skip` – drop the missing branch (useful in ravels).
-* `:yield_nil` – emit `nil` in place (preserves cardinality).
-
-Document these on any user-facing accessor.
-
----
-
-## Quick Cheatsheet
-
-* Use **`ravel(path)`** to “list the things at this level.”
-* Use **`each_indexed(path)`** when you need `(value, idx)` pairs for joins/regroup.
-* Use **`lift(to_scope, each_indexed(path))`** to reconstruct nested structure.
-* **Reducers** (e.g., `sum`, `avg`, `min`) consume the raveled view of their argument.
-* **Structure functions** (e.g., `size`, `flatten`, `flatten_one`, `count_across`) operate on structure at that depth and usually compile via `:ravel`.
-
-Keep the three laws in mind and Kumi’s behavior is predictable—even over deeply nested, heterogeneous data.

data/docs/compiler_design_principles.md

@@ -1,86 +0,0 @@
-# Compiler Design Principles
-
-## Core Principle: Smart Analyzer, Dumb Compiler
-
-The Kumi compiler follows a strict separation of concerns:
-
-### Analyzer Phase (Smart)
-- **Makes all decisions** about how operations should be executed
-- **Analyzes semantic context** to determine operation modes
-- **Pre-computes execution strategies** and stores in metadata
-- **Resolves complex logic** like nested array broadcasting, reduction flattening, etc.
-- **Produces complete instructions** for the compiler to follow
-
-### Compiler Phase (Dumb)
-- **Follows metadata instructions** without making decisions
-- **No conditional logic** based on data types, function types, or structure analysis
-- **Mechanically executes** the pre-computed strategy from analyzer
-- **Pure translation** from AST + metadata → executable functions
-
-## Examples
-
-### ❌ BAD: Compiler Making Decisions
-```ruby
-def compile_call(expr)
-  if Kumi::Registry.reducer?(expr.fn_name)
-    if nested_array_detected?(values)
-      # Compiler deciding to flatten
-      flatten_and_call(expr.fn_name, values)
-    end
-  end
-end
-```
-
-### ✅ GOOD: Compiler Following Metadata
-```ruby
-def compile_call(expr)
-  # Just read the pre-computed strategy
-  strategy = @analysis.metadata[:call_strategies][expr]
-  execute_strategy(strategy, expr)
-end
-```
-
-### ❌ BAD: Runtime Structure Analysis
-```ruby
-def vectorized_function_call(fn_name, values)
-  # Compiler analyzing structure at runtime
-  if values.any? { |v| deeply_nested?(v) }
-    apply_nested_broadcasting(fn, values)
-  end
-end
-```
-
-### ✅ GOOD: Pre-computed Broadcasting Plan
-```ruby
-def compile_element_field_reference(expr)
-  # Analyzer already determined the strategy
-  metadata = @analysis.state[:broadcasts][:nested_paths][expr.path]
-  traverse_nested_path(ctx, expr.path, metadata[:operation_mode])
-end
-```
-
-## Benefits
-
-1. **Predictable Performance**: No runtime analysis or decision-making
-2. **Easier Testing**: Compiler behavior determined entirely by metadata
-3. **Maintainable**: Complex logic isolated in analyzer passes
-4. **Extensible**: New features added by extending analyzer, not compiler
-5. **Debuggable**: All decisions visible in analyzer metadata
-
-## Implementation Pattern
-
-For any new compiler feature:
-
-1. **Analyzer Pass**: Analyze the requirement and store strategy in metadata
-2. **Metadata Schema**: Define clear data structure for the strategy
-3. **Compiler Method**: Read metadata and execute strategy mechanically
-4. **No Conditionals**: Avoid `if` statements based on runtime data in compiler
-
-## Metadata-Driven Architecture
-
-The compiler should be a pure **metadata interpreter**:
-- Input: AST + Analyzer Metadata
-- Output: Executable Functions
-- Process: Mechanical translation following metadata instructions
-
-This ensures the compiler remains simple, fast, and maintainable as the system grows in complexity.

data/docs/dev/analyzer-debug.md

@@ -1,52 +0,0 @@
-# Analyzer Debug System
-
-## Debug Module
-
-**File**: `lib/kumi/core/analyzer/debug.rb`
-
-**Enable**: `KUMI_DEBUG_STATE=1`
-
-**Output**: JSONL to stdout with state diffs and timing per pass
-
-**Configuration**:
-- `KUMI_DEBUG_REQUIRE_FROZEN=1` - Enforce state immutability checks
-
-## Checkpoint System
-
-**Files**: 
-- `lib/kumi/core/analyzer/checkpoint.rb`
-- `lib/kumi/core/analyzer/state_serde.rb`
-
-**Enable**: `KUMI_CHECKPOINT=1`
-
-**Configuration**:
-- `KUMI_CHECKPOINT_DIR=path` - Output directory (default: `/tmp/kumi_checkpoints`)
-- `KUMI_CHECKPOINT_FORMAT=marshal|json|both` - File format (default: `marshal`)
-- `KUMI_CHECKPOINT_PHASE=before|after|both` - When to save (default: `both`)
-
-**Resume/Stop**:
-- `KUMI_RESUME_FROM=file.msh` - Resume from checkpoint file
-- `KUMI_RESUME_AT=PassName` - Skip to specific pass
-- `KUMI_STOP_AFTER=PassName` - Stop after specific pass
-
-## Object Printers
-
-**File**: `spec/support/debug_printers.rb`
-
-Handles clean output for debug logs. Add new object types here when they appear as `#<Object:0x...>` in debug output.
-
-## Usage
-
-```bash
-# Basic debug
-KUMI_DEBUG_STATE=1 bundle exec ruby script.rb
-
-# Debug with checkpoints
-KUMI_DEBUG_STATE=1 KUMI_CHECKPOINT=1 bundle exec ruby script.rb
-
-# Resume from checkpoint
-KUMI_RESUME_FROM=/tmp/kumi_checkpoints/005_TypeChecker_after.msh bundle exec ruby script.rb
-
-# Stop at specific pass for debugging
-KUMI_STOP_AFTER=BroadcastDetector bundle exec ruby script.rb
-```

data/docs/dev/parse-command.md

@@ -1,64 +0,0 @@
-# Parse Command
-
-One-command developer loop for text schemas: parse → analyze → render IR → diff.
-
-## Usage
-
-```bash
-# Default: show diff vs golden file (or print if no golden)
-kumi parse golden/simple_math/schema.kumi
-
-# Write/overwrite golden file  
-kumi parse golden/simple_math/schema.kumi --write
-
-# Update golden only if different
-kumi parse golden/simple_math/schema.kumi --update
-
-# JSON format instead of text
-kumi parse golden/simple_math/schema.kumi --json --write
-
-# Enable state tracing
-kumi parse golden/simple_math/schema.kumi --trace
-
-# Snapshot analysis passes
-kumi parse golden/simple_math/schema.kumi --snap after --snap-dir debug/
-```
-
-## Exit Codes
-
-- `0` - No changes or successful write
-- `1` - Diff mismatch or analysis error
-
-## Output
-
-**No changes:**
-```
-No changes (golden/simple_math/expected/ir.txt)
-```
-
-**Diff mismatch:**
-```
---- expected
-+++ actual
-@@ -6,7 +6,7 @@
-   2: map argc=2 fn=:add [0, 1]
-+  2: map argc=2 fn=:power [0, 1]
-```
-
-**Write mode:**
-```
-Wrote: golden/simple_math/expected/ir.txt
-```
-
-## IR Format
-
-Deterministic text representation:
-```
-IR Module
-decls: 3
-decl[0] value:sum shape=scalar ops=4
-  0: load_input has_idx=false is_scalar=true plan_id=x:read scope=[] []
-  1: load_input has_idx=false is_scalar=true plan_id=y:read scope=[] []
-  2: map argc=2 fn=:add [0, 1]
-  3: store name=:sum [2]
-```

data/docs/dev/vm-profiling.md

@@ -1,95 +0,0 @@
-# VM Profiling with Schema Differentiation
-
-## Overview
-
-Profiles VM operation execution with schema-level differentiation. Tracks operations by schema type for multi-schema performance analysis.
-
-## Core Components
-
-**Profiler**: `lib/kumi/core/ir/execution_engine/profiler.rb`
-- Streams VM operation events with schema identification
-- Supports persistent mode for cross-run analysis
-- JSONL event format with operation metadata
-
-**Profile Aggregator**: `lib/kumi/dev/profile_aggregator.rb`  
-- Analyzes profiling data by schema type
-- Generates summary and detailed performance reports
-- Schema breakdown showing operations and timing per schema
-
-**CLI Integration**: `bin/kumi profile`
-- Processes JSONL profiling data files
-- Multiple output formats: summary, detailed, raw
-
-## Usage
-
-### Basic Profiling
-
-```bash
-# Single schema with operations
-KUMI_PROFILE=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
-
-# Persistent mode across multiple runs
-KUMI_PROFILE=1 KUMI_PROFILE_PERSISTENT=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
-
-# Streaming mode for real-time analysis  
-KUMI_PROFILE=1 KUMI_PROFILE_STREAM=1 KUMI_PROFILE_OPS=1 KUMI_PROFILE_FILE=profile.jsonl ruby script.rb
-```
-
-### CLI Analysis
-
-```bash
-# Summary report with schema breakdown
-kumi profile profile.jsonl --summary
-
-# Detailed per-operation analysis
-kumi profile profile.jsonl --detailed
-
-# Raw event stream
-kumi profile profile.jsonl --raw
-```
-
-## Environment Variables
-
-**Core**:
-- `KUMI_PROFILE=1` - Enable profiling
-- `KUMI_PROFILE_FILE=path` - Output file (required)
-- `KUMI_PROFILE_OPS=1` - Enable VM operation profiling
-
-**Modes**:
-- `KUMI_PROFILE_PERSISTENT=1` - Append to existing files across runs
-- `KUMI_PROFILE_STREAM=1` - Stream individual events vs batch
-- `KUMI_PROFILE_TRUNCATE=1` - Truncate existing files
-
-## Event Format
-
-JSONL with operation metadata:
-
-```json
-{"event":"vm_operation","schema":"TestSchema","operation":"LoadInput","duration_ms":0.001,"timestamp":"2025-01-20T10:30:45.123Z"}
-{"event":"vm_operation","schema":"TestSchema","operation":"Map","duration_ms":0.002,"timestamp":"2025-01-20T10:30:45.125Z"}
-```
-
-## Schema Differentiation
-
-Tracks operations by schema class name for multi-schema analysis:
-
-**Implementation**:
-- Schema name propagated through compilation pipeline
-- Profiler tags each VM operation with schema identifier  
-- Aggregator groups operations by schema type
-
-**Output Example**:
-```
-Total operations: 24 (0.8746ms)
-Schemas analyzed: SchemaA, SchemaB
-  SchemaA: 12 operations, 0.3242ms
-  SchemaB: 12 operations, 0.0504ms
-```
-
-## Performance Analysis
-
-**Reference Operations**: Typically dominate execution time in complex schemas
-**Map Operations**: Element-wise computations on arrays
-**LoadInput Operations**: Data access operations
-
-Use schema breakdown to identify performance differences between schema types.

data/docs/development/README.md

@@ -1,120 +0,0 @@
-# Development Guides
-
-This directory contains detailed guides for developing and maintaining Kumi. These guides complement the high-level information in the main `CLAUDE.md` file.
-
-## Guide Index
-
-### Architecture & Design
-- [Error Reporting Standards](error-reporting.md) - Comprehensive guide to unified error reporting
-- [Analyzer Pass Development](analyzer-passes.md) - How to create new analyzer passes
-- [Type System Integration](type-system.md) - Working with Kumi's type inference and checking
-
-### Code Quality & Standards  
-- [Testing Standards](testing-standards.md) - Testing patterns and requirements
-- [Code Organization](code-organization.md) - File structure and class design patterns
-- [RuboCop Guidelines](rubocop-guidelines.md) - Code style and quality requirements
-
-### Common Tasks
-- [Adding New Functions](adding-functions.md) - Extending the FunctionRegistry
-- [DSL Extension Patterns](dsl-extensions.md) - Adding new DSL constructs
-- [Performance Considerations](performance.md) - Guidelines for maintaining performance
-
-### Integration & Compatibility
-- [Backward Compatibility](backward-compatibility.md) - Maintaining compatibility during changes
-- [Migration Patterns](migration-patterns.md) - Safe patterns for evolving APIs
-- [Zeitwerk Integration](zeitwerk.md) - Autoloading patterns and requirements
-
-## Quick Reference
-
-### Key Principles
-1. **Unified Error Reporting**: All errors must provide clear location information
-2. **Multi-pass Analysis**: Each analyzer pass has single responsibility  
-3. **Backward Compatibility**: Changes maintain existing API compatibility
-4. **Type Safety**: Optional but comprehensive type checking
-5. **Ruby Integration**: Leverage Ruby idioms while maintaining structure
-
-### Common Commands
-```bash
-# Run all tests
-bundle exec rspec
-
-# Run specific test categories
-bundle exec rspec spec/integration/
-bundle exec rspec spec/kumi/analyzer/
-
-# Check code quality
-bundle exec rubocop
-bundle exec rubocop -a
-
-# Validate error reporting
-bundle exec ruby test_location_improvements.rb
-```
-
-### File Templates
-
-**New Analyzer Pass**:
-```ruby
-# frozen_string_literal: true
-
-module Kumi
-  module Analyzer
-    module Passes
-      class MyNewPass < PassBase
-        def run(errors)
-          # Implementation with proper error reporting
-          report_error(errors, "message", location: node.loc, type: :semantic)
-        end
-      end
-    end
-  end
-end
-```
-
-**New Integration Test**:
-```ruby
-# frozen_string_literal: true
-
-RSpec.describe "My Feature Integration" do
-  it "validates the feature works correctly" do
-    schema = build_schema do
-      input { integer :field }
-      value :result, input.field * 2
-    end
-    
-    expect(schema.from(field: 5).fetch(:result)).to eq(10)
-  end
-end
-```
-
-## Contributing Guidelines
-
-### Before Making Changes
-1. Check relevant development guide in this directory
-2. Review `CLAUDE.md` for high-level architecture understanding
-3. Run existing tests to ensure baseline functionality
-4. Consider backward compatibility implications
-
-### After Making Changes  
-1. Update relevant development guides if patterns change
-2. Add or update tests for new functionality
-3. Run full test suite: `bundle exec rspec`
-4. Check code quality: `bundle exec rubocop`
-5. Verify error reporting quality with integration tests
-
-### Adding New Guides
-When adding new development guides:
-1. Create focused, actionable guides for specific development tasks
-2. Include code examples and common patterns
-3. Reference related files and tests
-4. Update this README index
-5. Cross-reference from main `CLAUDE.md` if needed
-
-## Guide Maintenance
-
-These guides should be kept up-to-date as the codebase evolves:
-- **Review quarterly** for accuracy and completeness
-- **Update immediately** when patterns or APIs change significantly  
-- **Expand based on common questions** during development
-- **Consolidate** overlapping or redundant information
-
-The goal is to make Kumi development efficient and consistent while maintaining high code quality.