kumi 0.0.13 → 0.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7af4bdb11ef5da5a25b799cef8387752636a8224634c70656efc203b9d923185
-  data.tar.gz: 4e604513e42dcb8754fab54ead4140d5ca2d8443ae4d0461e62dc3b48bb925d4
+  metadata.gz: 5405d7d0612a81e5154bd1d452fdfc150691b022137fc0ee132c47ede1a58e2e
+  data.tar.gz: '093cf7a6d305c02f92de600b06f62be39f8af90d798a0f93ed3ef59f539ada9b'
 SHA512:
-  metadata.gz: dcbd93268081ff8f0bcf112101603b180932196b1d6fcb59aae2f363d6bbfe4e9ea07cf3cbe811290222dcd15cdf132ea04f409bea34f81599412f729211a5fa
-  data.tar.gz: 3e0c2e91609e742c2852457b2dd61a114a03b988ae983e0cfb23c9bbf46bc5c6c1714636961ed46b4b9ae924c6a77a79dc0aab9013cadbb73e49d5374f8b8dbc
+  metadata.gz: b3ea711bf465e0c11cc95fabb3809dd632ebbfcc8c36297b161fb1f179fffdda5df1e5c033968e837dd8ed3f983639416de08bd371f30be9f2cefd5543efe1ff
+  data.tar.gz: 0a63fe824fb604639b4efb9cfc2ce24a93429110adc75cfd3edc905c58b3501273ad07a3cf66415eabce4e099c5fd4d202bd2c4064875d7a73e0c2a26f0689cb

data/.rspec

@@ -1,3 +1,2 @@
---format documentation
 --color
 --require spec_helper

data/BACKLOG.md

@@ -0,0 +1,34 @@
+# Kumi Development Backlog
+
+## High Priority
+
+### RegistryV2 Migration Cleanup
+- [ ] Remove legacy `lib/kumi/core/function_registry.rb` and related modules after RegistryV2 is fully integrated
+- [ ] Clean up bridge code in `FunctionSignaturePass` once all functions are migrated
+- [ ] Remove `create_basic_signature` fallback logic
+- [ ] Update all tests to use RegistryV2 functions
+
+## Medium Priority
+
+### Function Registry Enhancements
+- [ ] Add YAML linting for function definitions
+- [ ] Implement golden-IR test fixtures for signature resolution
+- [ ] Add Arrow/JS kernel mapping support
+- [ ] Implement type promotion rules from YAML dtypes
+
+### Performance Optimizations
+- [ ] Cache RegistryV2 loading for repeated schema compilations
+- [ ] Optimize signature resolution for large function sets
+- [ ] Implement lazy loading of function kernels
+
+## Low Priority
+
+### Documentation
+- [ ] Auto-generate function reference docs from YAML
+- [ ] Add examples for each function in registry
+- [ ] Document kernel implementation guidelines
+
+### Testing
+- [ ] Add comprehensive NEP-20 compliance test suite
+- [ ] Stress test RegistryV2 with large function sets
+- [ ] Add performance benchmarks for signature resolution

data/CHANGELOG.md

@@ -1,5 +1,38 @@
 ## [Unreleased]
 
+## [0.0.15] – 2025-08-21
+### Added
+- (DX) Schema-aware VM profiling with multi-schema performance analysis
+- DAG-based execution optimization with pre-computed dependency resolution
+
+### Performance
+- Reference operations eliminated as VM bottleneck via O(1) hash lookups
+
+## [0.0.14] – 2025-08-21
+### Added
+- Text schema frontend with `.kumi` file format support
+- `bin/kumi parse` command for schema analysis and golden file testing
+- LoadInputCSE optimization pass to eliminate redundant load operations
+- Runtime accessor caching with precise field-based invalidation
+- VM profiler with wall time, CPU time, and cache hit rate analysis
+- Structured analyzer debug system with state inspection
+- Checkpoint system for capturing and comparing analyzer states
+- State serialization (StateSerde) for golden testing and regression detection
+- Debug object printers with configurable truncation
+- Multi-run averaging for stable performance benchmarking
+
+### Fixed
+- VM targeting for `__vec` twin declarations that were failing to resolve
+- Demand-driven reference resolution with proper name indexing and cycle detection
+- Accessor cache invalidation now uses precise field dependencies instead of clearing all caches
+- StateSerde JSON serialization issues with frozen hashes, Sets, and Symbols
+
+### Performance
+- 14x improvement on update-heavy workloads (1.88k → 26.88k iterations/second)
+- 30-40% reduction in IR module size for schemas with repeated field access
+- Eliminated load_input performance bottleneck that was consuming ~99% of execution time
+- Optional caching system (enabled via KUMI_VM_CACHE=1) for performance-critical scenarios
+
 ## [0.0.13] – 2025-08-14
 ### Added
 - Runtime performance optimizations for interpreter execution

data/CLAUDE.md

@@ -1,11 +1,9 @@
 # CLAUDE.md
 
-!! Important:
-!! Remember, this gem is not on production yet, so no backward compatilibity is necessary. But do not change the public interfaces (e.g. DSL, Schema) without explicitly requested or demanded. 
-!! We are using zeitwerk, i.e.: no requires
-!! Disregard linting or coverage issues unless asked to do so.
-!! Communication style - Write direct, factual statements. Avoid promotional language, unnecessary claims, or marketing speak. State what the system does, not what benefits it provides. Use TODOs for missing information rather than placeholder claims.
-!! See all Available Functions in docs/FUNCTIONS.md
+IMPORTANT!: Use `kumi parse` to see IR or Analysis States Diff - see docs/dev/parse-command.md
+Remember, this gem is not on production yet, so no backward compatilibity is necessary. But do not change the public interfaces (e.g. DSL, Schema) without explicitly requested or demanded. 
+We are using zeitwerk, i.e.: no requires
+See all Available Functions in docs/FUNCTIONS.md
 
 ## Project Overview
 

data/README.md

@@ -207,33 +207,6 @@ end
 # ❌ Function arity error: divide expects 2 arguments, got 1
 ```
 
-**Mutual Recursion**: Kumi supports mutual recursion when cascade conditions are mutually exclusive:
-
-```ruby
-trait :is_forward, input.operation == "forward"
-trait :is_reverse, input.operation == "reverse"
-
-# Safe mutual recursion - conditions are mutually exclusive
-value :forward_processor do
-  on is_forward, input.value * 2        # Direct calculation
-  on is_reverse, reveAnalysisrse_processor + 10  # Delegates to reverse (safe)
-  base "invalid operation"
-end
-
-value :reverse_processor do
-  on is_forward, forward_processor - 5   # Delegates to forward (safe) 
-  on is_reverse, input.value / 2         # Direct calculation
-  base "invalid operation"
-end
-
-# Usage examples:
-# operation="forward", value=10  => forward: 20, reverse: 15
-# operation="reverse", value=10  => forward: 15, reverse: 5  
-# operation="unknown", value=10  => both: "invalid operation"
-```
-
-This compiles because `operation` can only be "forward" or "reverse", never both. Each recursion executes one step before hitting a direct calculation.
-
 #### **Runtime Introspection: Debug and Understand**
 
 **Explainability**: Trace exactly how any value is computed, step-by-step. This is invaluable for debugging complex logic and auditing results.
@@ -436,24 +409,6 @@ runner[:after_tax]     # => 196,844.80 (cached)
 - Sequential procedural workflows  
 - High-frequency processing
 
-## JavaScript Transpiler (V 0.0.10)
-Note: On the current 0.0.11 this is disabled but will be back in later versions. reason: IR/Interpreter update.
-Transpiles compiled schemas to standalone JavaScript code. See [docs/features/javascript-transpiler.md](docs/features/javascript-transpiler.md) for details.
-
-```ruby
-Kumi::Js.export_to_file(FederalTax2024, "federal-tax-2024.js")
-```
-
-```javascript
-const { schema } = require('./federal-tax-2024.js');
-const calculator = schema.from({ income: 100_000, filing_status: "single" });
-console.log(calculator.fetch('total_tax'));   // 21491
-```
-
-Generated JavaScript includes only functions used by the schema.
-
-All tests run in dual mode to verify compiled schemas produce identical results in both Ruby and JavaScript.
-
 ## Performance
 
 Benchmarks on Linux with Ruby 3.3.8 on a Dell Latitude 7450:

data/config/functions.yaml

@@ -0,0 +1,352 @@
+# ==================== CORE - SCALAR (elementwise) ====================
+
+- name: core.add
+  domain: core
+  opset: 1
+  class: scalar
+  summary: Numeric addition with broadcasting
+  signature: ["(),()->()", "(i),(i)->(i)", "(i,j),(i,j)->(i,j)"]
+  type_vars: {T: Numeric, U: Numeric}
+  dtypes: {result: "promote(T,U)", casting: "same_kind"}
+  null_policy: propagate
+  algebra: {associative: true, commutative: true, identity: 0}
+  vectorization: {mask_aware: true}
+  kernels: 
+    - backend: ruby
+      impl: kumi_add
+      priority: 10
+
+- name: core.sub
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)", "(i,j),(i,j)->(i,j)"]
+  type_vars: {T: Numeric, U: Numeric}
+  dtypes: {result: "promote(T,U)", casting: "same_kind"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: kumi_sub
+      priority: 10
+
+- name: core.mul
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)", "(i,j),(i,j)->(i,j)"]
+  type_vars: {T: Numeric, U: Numeric}
+  dtypes: {result: "promote(T,U)", casting: "same_kind"}
+  null_policy: propagate
+  algebra: {associative: true, commutative: true, identity: 1, annihilator: 0}
+  kernels:
+    - backend: ruby
+      impl: kumi_mul
+      priority: 10
+
+- name: core.div
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)", "(i,j),(i,j)->(i,j)"]
+  type_vars: {T: Numeric, U: Numeric}
+  dtypes: {result: "promote_float(T,U)", casting: "same_kind"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: kumi_div
+      priority: 10
+
+- name: core.eq
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)", "(i,j),(i,j)->(i,j)"]
+  type_vars: {T: Any, U: Any}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  algebra: {idempotent: true}
+  kernels:
+    - backend: ruby
+      impl: kumi_eq
+      priority: 10
+
+- name: core.gt
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {T: Orderable, U: Orderable}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: kumi_gt
+      priority: 10
+
+- name: core.gte
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {T: Orderable, U: Orderable}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: kumi_gte
+      priority: 10
+
+- name: core.and
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {B1: Bool, B2: Bool}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  algebra: {associative: true, commutative: true, identity: true, annihilator: false}
+  vectorization: {short_circuit: left_to_right, mask_aware: true}
+  kernels:
+    - backend: ruby
+      impl: kumi_and
+      priority: 10
+
+- name: core.or
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {B1: Bool, B2: Bool}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  algebra: {associative: true, commutative: true, identity: false, annihilator: true}
+  vectorization: {short_circuit: left_to_right, mask_aware: true}
+  kernels:
+    - backend: ruby
+      impl: kumi_or
+      priority: 10
+
+- name: core.not
+  domain: core
+  opset: 1
+  class: scalar
+  signature: ["()->()", "(i)->(i)"]
+  type_vars: {B: Bool}
+  dtypes: {result: "bool"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: kumi_not
+      priority: 10
+
+# ==================== AGGREGATES (reduce last axis in signature) ==============
+
+- name: agg.sum
+  domain: core
+  opset: 1
+  class: aggregate
+  signature: ["(i)->()", "(i,j)->(i)"]
+  type_vars: {T: Numeric}
+  dtypes: {result: "promote(T)"}
+  null_policy: skip
+  options: {min_count: 0}
+  algebra: {associative: true, commutative: true, identity: 0}
+  kernels:
+    - backend: ruby
+      impl: kumi_sum
+      priority: 10
+
+- name: agg.min
+  domain: core
+  opset: 1
+  class: aggregate
+  signature: ["(i)->()", "(i,j)->(i)"]
+  type_vars: {T: Orderable}
+  dtypes: {result: "T"}
+  null_policy: skip
+  kernels:
+    - backend: ruby
+      impl: kumi_min
+      priority: 10
+
+- name: agg.max
+  domain: core
+  opset: 1
+  class: aggregate
+  signature: ["(i)->()", "(i,j)->(i)"]
+  type_vars: {T: Orderable}
+  dtypes: {result: "T"}
+  null_policy: skip
+  kernels:
+    - backend: ruby
+      impl: kumi_max
+      priority: 10
+
+- name: agg.mean
+  domain: core
+  opset: 1
+  class: aggregate
+  signature: ["(i)->()", "(i,j)->(i)"]
+  type_vars: {T: Numeric}
+  dtypes: {result: "float"}
+  null_policy: skip
+  options: {min_count: 0}
+  kernels:
+    - backend: ruby
+      impl: kumi_mean
+      priority: 10
+
+# ==================== STRUCTURE / SHAPE / JOIN ================================
+
+- name: struct.join
+  domain: struct
+  opset: 1
+  class: structure
+  summary: Align inputs by axes using zip or product
+  signature: ["(i),(i)->(i)@zip", "(i),(j)->(i,j)@product"]
+  null_policy: error_on_missing
+  kernels:
+    - backend: ruby
+      impl: join_zip
+      when: {policy: zip}
+      priority: 10
+    - backend: ruby
+      impl: join_product
+      when: {policy: product}
+      priority: 10
+
+- name: struct.align_to
+  domain: struct
+  opset: 1
+  class: structure
+  signature: ["(i)->(i)", "(i,j)->(i,j)"]
+  traits: {reorders: false}
+  null_policy: error_on_missing
+  kernels:
+    - backend: ruby
+      impl: align_to
+      priority: 10
+
+- name: struct.lift
+  domain: struct
+  opset: 1
+  class: structure
+  signature: ["(i)->(i)", "(i,j)->(i,j)"]
+  summary: Group flat rows back to nested collections based on indices
+  kernels:
+    - backend: ruby
+      impl: lift
+      priority: 10
+
+- name: struct.flatten
+  domain: struct
+  opset: 1
+  class: structure
+  signature: ["(i,j)->(k)"]
+  shape_fn: "flatten_axes(i,j)->k"
+  kernels:
+    - backend: ruby
+      impl: flatten
+      priority: 10
+
+- name: struct.take
+  domain: struct
+  opset: 1
+  class: structure
+  signature: ["(i),(i)->(i)"]  # take(values, indices) elemwise
+  type_vars: {T: Any}
+  dtypes: {result: "T"}
+  null_policy: error
+  kernels:
+    - backend: ruby
+      impl: take
+      priority: 10
+
+- name: struct.size
+  domain: struct
+  opset: 1
+  class: vector
+  signature: ["(i)->()", "(i,j)->()"]
+  dtypes: {result: "int"}
+  null_policy: error
+  kernels:
+    - backend: ruby
+      impl: size
+      priority: 10
+
+# ==================== MASK / CONDITIONAL ======================================
+
+- name: mask.where
+  domain: mask
+  opset: 1
+  class: scalar
+  summary: Ternary select with broadcasting of then/else
+  signature:
+    - "(i),(i),(i)->(i)"
+    - "(i),(),()->(i)"
+    - "(),(i),()->(i)"
+    - "(),(),()->()"
+  type_vars: {B: Bool, T: Any, U: Any}
+  dtypes: {result: "unify(T,U)"}
+  null_policy: propagate
+  vectorization: {mask_aware: true, short_circuit: left_to_right}
+  kernels:
+    - backend: ruby
+      impl: where
+      priority: 10
+
+# ==================== STRING (minimal) ========================================
+
+- name: string.concat
+  domain: string
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {S1: String, S2: String}
+  dtypes: {result: "string"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: str_concat
+      priority: 10
+
+- name: string.length
+  domain: string
+  opset: 1
+  class: vector
+  signature: ["(i)->(i)", "()->()"]
+  type_vars: {S: String}
+  dtypes: {result: "int"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: str_length
+      priority: 10
+
+# ==================== DATETIME (minimal) ======================================
+
+- name: datetime.add_days
+  domain: datetime
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {D: DateLike, N: IntLike}
+  dtypes: {result: "datetime"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: dt_add_days
+      priority: 10
+
+- name: datetime.diff_days
+  domain: datetime
+  opset: 1
+  class: scalar
+  signature: ["(),()->()", "(i),(i)->(i)"]
+  type_vars: {D1: DateLike, D2: DateLike}
+  dtypes: {result: "int"}
+  null_policy: propagate
+  kernels:
+    - backend: ruby
+      impl: dt_diff_days
+      priority: 10

data/docs/dev/analyzer-debug.md

@@ -0,0 +1,52 @@
+# Analyzer Debug System
+
+## Debug Module
+
+**File**: `lib/kumi/core/analyzer/debug.rb`
+
+**Enable**: `KUMI_DEBUG_STATE=1`
+
+**Output**: JSONL to stdout with state diffs and timing per pass
+
+**Configuration**:
+- `KUMI_DEBUG_REQUIRE_FROZEN=1` - Enforce state immutability checks
+
+## Checkpoint System
+
+**Files**: 
+- `lib/kumi/core/analyzer/checkpoint.rb`
+- `lib/kumi/core/analyzer/state_serde.rb`
+
+**Enable**: `KUMI_CHECKPOINT=1`
+
+**Configuration**:
+- `KUMI_CHECKPOINT_DIR=path` - Output directory (default: `/tmp/kumi_checkpoints`)
+- `KUMI_CHECKPOINT_FORMAT=marshal|json|both` - File format (default: `marshal`)
+- `KUMI_CHECKPOINT_PHASE=before|after|both` - When to save (default: `both`)
+
+**Resume/Stop**:
+- `KUMI_RESUME_FROM=file.msh` - Resume from checkpoint file
+- `KUMI_RESUME_AT=PassName` - Skip to specific pass
+- `KUMI_STOP_AFTER=PassName` - Stop after specific pass
+
+## Object Printers
+
+**File**: `spec/support/debug_printers.rb`
+
+Handles clean output for debug logs. Add new object types here when they appear as `#<Object:0x...>` in debug output.
+
+## Usage
+
+```bash
+# Basic debug
+KUMI_DEBUG_STATE=1 bundle exec ruby script.rb
+
+# Debug with checkpoints
+KUMI_DEBUG_STATE=1 KUMI_CHECKPOINT=1 bundle exec ruby script.rb
+
+# Resume from checkpoint
+KUMI_RESUME_FROM=/tmp/kumi_checkpoints/005_TypeChecker_after.msh bundle exec ruby script.rb
+
+# Stop at specific pass for debugging
+KUMI_STOP_AFTER=BroadcastDetector bundle exec ruby script.rb
+```

data/docs/dev/parse-command.md

@@ -0,0 +1,64 @@
+# Parse Command
+
+One-command developer loop for text schemas: parse → analyze → render IR → diff.
+
+## Usage
+
+```bash
+# Default: show diff vs golden file (or print if no golden)
+kumi parse golden/simple_math/schema.kumi
+
+# Write/overwrite golden file  
+kumi parse golden/simple_math/schema.kumi --write
+
+# Update golden only if different
+kumi parse golden/simple_math/schema.kumi --update
+
+# JSON format instead of text
+kumi parse golden/simple_math/schema.kumi --json --write
+
+# Enable state tracing
+kumi parse golden/simple_math/schema.kumi --trace
+
+# Snapshot analysis passes
+kumi parse golden/simple_math/schema.kumi --snap after --snap-dir debug/
+```
+
+## Exit Codes
+
+- `0` - No changes or successful write
+- `1` - Diff mismatch or analysis error
+
+## Output
+
+**No changes:**
+```
+No changes (golden/simple_math/expected/ir.txt)
+```
+
+**Diff mismatch:**
+```
+--- expected
++++ actual
+@@ -6,7 +6,7 @@
+   2: map argc=2 fn=:add [0, 1]
++  2: map argc=2 fn=:power [0, 1]
+```
+
+**Write mode:**
+```
+Wrote: golden/simple_math/expected/ir.txt
+```
+
+## IR Format
+
+Deterministic text representation:
+```
+IR Module
+decls: 3
+decl[0] value:sum shape=scalar ops=4
+  0: load_input has_idx=false is_scalar=true plan_id=x:read scope=[] []
+  1: load_input has_idx=false is_scalar=true plan_id=y:read scope=[] []
+  2: map argc=2 fn=:add [0, 1]
+  3: store name=:sum [2]
+```