kumi-parser 0.0.2 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eb3ed4f3c5d534b41a856cb05ecacdafa233c66f97039af6c7f695d7b11e9638
-  data.tar.gz: dae5536d6035f67546136f12ffe1fc52d602bdffc8dd5eeba120fec2313c4cd7
+  metadata.gz: 0067c520152fae1c2285b2628f3b8b5d25eff145ae61780cf3d283310816c336
+  data.tar.gz: 4c604f36cd250a8d4672fef7c2acc667c7de498fe0ee088980bcbd8a744b8d5e
 SHA512:
-  metadata.gz: 1bc09e791d8557b52f2b4b6475ee368f6ac000c4cae9de0efb9e877e86eb7929f1933e6bb247aa6cfa4cdd5412d22217645ab4a873f9881935613cc02bc95dc2
-  data.tar.gz: 8cf8d948c812075b068479e25737a0d43999831f2dea87a018d6d62fe8c1049d96bb27076f88173ed91d4bd4ae1939669d768402ee9e8ce8e5c215c14014d909
+  metadata.gz: c3d3342f43d5bb41b140f40d02d3eaac08abeb88509d4c4f4ce127e38ec3583e7566c68d8a96956d2877a2819b046663e2bf95aa246763bcf7556cf4df380934
+  data.tar.gz: f7891f6e388e224ebfd0da8d027d331c6794325e0d27715f8800f4aa9124dd3421fc36ca3afb1ac10e6f4e8c5db61f610a18e9db0a33d21c9292efbbd5fa45a7

data/.rspec

@@ -0,0 +1,3 @@
+--require spec_helper
+--format documentation
+--color

data/CLAUDE.md

@@ -0,0 +1,120 @@
+# Kumi Parser - Technical Context
+
+## Current Architecture (January 2025)
+
+## Key Files
+
+- `lib/kumi/parser/smart_tokenizer.rb` - Tokenizer with context tracking
+- `lib/kumi/parser/direct_parser.rb` - Parser implementation (renamed from direct_ast_parser.rb)
+- `lib/kumi/parser/token_metadata.rb` - Token types and metadata
+- `lib/kumi/parser/text_parser.rb` - Public API maintaining compatibility
+- `lib/kumi/parser/base.rb` - Core parsing interface
+- `lib/kumi/parser/syntax_validator.rb` - Validation with proper diagnostics
+- `lib/kumi/parser/errors.rb` - Custom error types
+
+## Important Syntax Rules
+
+- **Functions**: `fn(:symbol, args...)` only (no dot notation like `fn.max()`)
+- **Operators**: Standard precedence (*/% > +- > comparisons > & > |)
+- **Array access**: Uses `array[index]` syntax (converted to `:at` function internally)
+- **Equality**: `==` and `!=` operators (converted from `:eq`/`:ne` tokens)
+- **Multi-line expressions**: Parser skips newlines within expressions
+- **Cascade**: `value :name do ... on condition, result ... base result ... end`
+- **Constants**: Text parser cannot resolve Ruby constants - use inline values
+
+## AST Structure & Compatibility
+
+All nodes from `Kumi::Syntax::*` (defined in main kumi gem):
+- `Root(inputs, attributes, traits)`
+- `InputDeclaration(name, domain, type, children)`
+- `ValueDeclaration(name, expression)`
+- `TraitDeclaration(name, expression)`
+- `CallExpression(fn_name, args)`
+- `InputReference(name)` / `InputElementReference(path)`
+- `DeclarationReference(name)`
+- `Literal(value)`
+- `CascadeExpression(cases)` / `CaseExpression(condition, result)`
+- `ArrayExpression(elements)`
+
+**Ruby DSL Compatibility**:
+- Cascade conditions: Simple trait references wrapped in `all?([trait])` function calls
+- Array access: `[index]` becomes `CallExpression(:at, [array, index])`
+- Operators: `:eq` → `:==`, `:ne` → `:!=` for consistency
+- Constants: Ruby constants resolved to values in DSL, remain as `DeclarationReference` in text parser
+
+## Debugging & Testing
+
+**View AST structure**:
+```ruby
+ast = Kumi::Parser::TextParser.parse(schema)
+puts Kumi::Support::SExpressionPrinter.print(ast)
+# => (Root
+#      inputs: [(InputDeclaration :income :float)]
+#      attributes: [(ValueDeclaration :tax (CallExpression :+ ...))]
+#      traits: [(TraitDeclaration :adult (CallExpression :>= ...))])
+```
+
+**Quick validation test**:
+```ruby
+ruby -r./lib/kumi/parser/text_parser -e "p Kumi::Parser::TextParser.valid?('schema do input do float :x end end')"
+```
+
+**Compare with Ruby DSL**:
+```ruby
+# Define schema in Ruby
+module TestSchema
+  extend Kumi::Schema
+  schema do
+    input do
+      float :income
+    end
+    value :tax, fn(:calc, input.income)
+  end
+end
+
+# Parse equivalent text
+text_ast = Kumi::Parser::TextParser.parse(<<~KUMI)
+  schema do
+    input do
+      float :income
+    end
+    value :tax, fn(:calc, input.income)
+  end
+KUMI
+
+# Compare ASTs
+ruby_ast = TestSchema.__syntax_tree__
+text_ast == ruby_ast # Should be true
+```
+
+- Tax schema in `spec/kumi/parser/text_parser_example tax_schema_spec.rb` is canonical test
+- Run all tests: `rspec spec/kumi/parser/` 
+- Integration tests: `rspec spec/kumi/parser/text_parser_integration_spec.rb`
+
+## Error Handling & Validation
+
+- **Parse errors**: `Kumi::Parser::Errors::ParseError` (internal) → `Kumi::Errors::SyntaxError` (public API)
+- **Tokenizer errors**: `Kumi::Parser::Errors::TokenizerError` with location info
+- **Diagnostics**: Use `SyntaxValidator` for detailed error reporting with line/column info
+- **Location tracking**: All tokens and AST nodes include `Kumi::Syntax::Location(file, line, column)`
+
+## Test Status (January 2025)
+
+✅ **All specs passing**: 32 examples, 0 failures, 1 pending
+- ✅ Syntax validation with proper diagnostics
+- ✅ AST compatibility with Ruby DSL (when constants aren't used)
+- ✅ Integration with analyzer and compiler
+- ✅ End-to-end execution testing
+- ✅ Error type compatibility
+
+## Known Limitations
+
+- **Ruby constants**: Text parser cannot resolve Ruby constants like `CONST_NAME` - use inline values instead
+- **Domain specification**: Parsing not fully implemented
+- **Diagnostic APIs**: Monaco/CodeMirror/JSON format methods not implemented
+
+## Performance
+
+- Tokenization: <1ms for typical schemas
+- Parsing: ~4ms for complete tax schema (21 values, 4 traits)
+- Direct AST construction eliminates transformation overhead

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Kumi Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,73 @@
+# Kumi::Parser
+
+Text parser for [Kumi](https://github.com/amuta/kumi) schemas. Direct tokenizer → AST construction with ~4ms parse time.
+
+## Installation
+
+```ruby
+gem 'kumi-parser'
+```
+
+## Usage
+
+```ruby
+require 'kumi/parser'
+
+schema = <<~KUMI
+  schema do
+    input do
+      float :income
+      string :status
+    end
+    
+    trait :adult, input.age >= 18
+    value :tax, fn(:calculate_tax, input.income)
+  end
+KUMI
+
+# Parse to AST
+ast = Kumi::Parser::TextParser.parse(schema)
+
+# Validate
+Kumi::Parser::TextParser.valid?(schema) # => true
+```
+
+## API
+
+- `parse(text)` → AST
+- `valid?(text)` → Boolean  
+- `validate(text)` → Array of error hashes
+
+## Syntax
+
+```
+schema do
+  input do
+    <type> :<name>[, domain: <spec>]
+  end
+  
+  trait :<name>, <expression>
+  
+  value :<name>, <expression>
+  value :<name> do
+    on <condition>, <result>
+    base <result>
+  end
+end
+```
+
+**Function calls**: `fn(:name, arg1, arg2, ...)`  
+**Operators**: `+` `-` `*` `/` `%` `>` `<` `>=` `<=` `==` `!=` `&` `|`  
+**References**: `input.field`, `value_name`, `array[index]`
+
+## Architecture
+
+- `smart_tokenizer.rb` - Context-aware tokenization with embedded metadata
+- `direct_ast_parser.rb` - Recursive descent parser, direct AST construction
+- `token_metadata.rb` - Token types, precedence, and semantic hints
+
+See `docs/` for technical details.
+
+## License
+
+MIT

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require 'rubocop/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/examples/debug_text_parser.rb

@@ -0,0 +1,41 @@
+# Debug the text parser transform
+
+require_relative 'lib/kumi/text_parser'
+
+schema_text = <<~SCHEMA
+  schema do
+    input do
+      integer :age
+    end
+  #{'  '}
+    trait :adult, input.age >= 18
+    value :bonus, 100
+  end
+SCHEMA
+
+puts 'Debugging text parser...'
+
+begin
+  # Test just the grammar parsing first
+  grammar = Kumi::TextParser::Grammar.new
+  parse_tree = grammar.parse(schema_text)
+
+  puts 'Raw parse tree:'
+  puts parse_tree.inspect
+  puts
+
+  # Now test the transform
+  transform = Kumi::TextParser::Transform.new
+  ast = transform.apply(parse_tree)
+
+  puts 'Transformed AST:'
+  puts ast.inspect
+  puts
+
+  puts 'AST structure:'
+  puts "- Values: #{ast.attributes.count} - #{ast.attributes.map(&:name)}"
+  puts "- Traits: #{ast.traits.count} - #{ast.traits.map(&:name)}"
+rescue StandardError => e
+  puts "Error: #{e.message}"
+  puts e.backtrace.first(5)
+end

data/examples/debug_transform_rule.rb

@@ -0,0 +1,26 @@
+# Debug specific transform rule
+
+require_relative 'lib/kumi/text_parser'
+
+# Test just the trait parsing
+trait_text = 'trait :adult, input.age >= 18'
+
+grammar = Kumi::TextParser::Grammar.new
+transform = Kumi::TextParser::Transform.new
+
+begin
+  # Parse just the trait declaration
+  parse_result = grammar.trait_declaration.parse(trait_text)
+  puts 'Trait parse result:'
+  puts parse_result.inspect
+  puts
+
+  # Try to transform it
+  transformed = transform.apply(parse_result)
+  puts 'Transformed result:'
+  puts transformed.inspect
+  puts "Class: #{transformed.class}"
+rescue StandardError => e
+  puts "Error: #{e.message}"
+  puts e.backtrace.first(5)
+end

data/examples/text_parser_comprehensive_test.rb

@@ -0,0 +1,333 @@
+# frozen_string_literal: true
+
+require_relative '../lib/kumi'
+
+# Comprehensive test of all text parser supported features
+# This example tests what the text parser ACTUALLY supports
+# (not what the Ruby DSL supports)
+
+module TextParserComprehensiveTest
+  extend Kumi::Schema
+
+  schema do
+    input do
+      # Basic type declarations
+      integer :age, domain: 18..65
+      float :score, domain: 0.0..100.0
+      string :status, domain: %w[active inactive suspended]
+      boolean :verified
+      any :metadata
+
+      # Nested array declarations
+      array :items do
+        string :name
+        float :price
+        integer :quantity
+      end
+    end
+
+    # ============================================================
+    # ARITHMETIC OPERATIONS - All supported operators
+    # ============================================================
+
+    # Addition and subtraction
+    value :total_price, input.items.price + input.items.quantity
+    value :price_diff, input.items.price - 10.0
+
+    # Multiplication and division
+    value :scaled_score, input.score * 1.5
+    value :average_score, input.score / 2
+
+    # Modulo
+    value :remainder, input.age % 10
+
+    # Complex arithmetic with parentheses
+    value :complex_calc, (input.score * 2.5) + (input.age / 2) - 10
+
+    # ============================================================
+    # COMPARISON OPERATIONS - All comparison operators
+    # ============================================================
+
+    trait :adult, input.age >= 18
+    trait :senior, input.age > 60
+    trait :young_adult, input.age <= 25
+    trait :is_teen, input.age < 20
+    trait :perfect_score, input.score == 100.0
+    trait :not_perfect, input.score != 100.0
+
+    # String comparisons
+    trait :is_active, input.status == 'active'
+    trait :not_suspended, input.status != 'suspended'
+
+    # Boolean comparisons
+    trait :is_verified, input.verified == true
+    trait :not_verified, input.verified == false
+
+    # ============================================================
+    # LOGICAL OPERATIONS - AND and OR
+    # ============================================================
+
+    # AND operations using &
+    trait :eligible, (input.age >= 18) & (input.verified == true) & (input.status == 'active')
+    trait :premium_user, is_verified & is_active & (input.score > 80.0)
+
+    # OR operations using |
+    trait :needs_attention, (input.status == 'suspended') | (input.verified == false)
+    trait :special_case, senior | (input.score >= 95.0)
+
+    # Complex logical expressions
+    trait :complex_logic, (adult & is_verified) | (senior & is_active)
+
+    # ============================================================
+    # FUNCTION CALLS - fn(:name, args) syntax
+    # ============================================================
+
+    # Math functions
+    value :absolute_diff, fn(:abs, fn(:subtract, input.score, 50.0))
+    value :rounded_score, fn(:round, input.score)
+    value :clamped_score, fn(:clamp, input.score, 20.0, 80.0)
+
+    # String functions (note: no method syntax like .length)
+    value :name_length, fn(:string_length, input.status)
+    value :uppercase_status, fn(:upcase, input.status)
+    value :lowercase_status, fn(:downcase, input.status)
+
+    # Collection/aggregation functions on arrays
+    value :total_items, fn(:sum, input.items.quantity)
+    value :item_count, fn(:size, input.items)
+    value :max_price, fn(:max, input.items.price)
+    value :min_price, fn(:min, input.items.price)
+
+    # ============================================================
+    # REFERENCES - Both bare identifiers and ref() NOT supported in text parser
+    # ============================================================
+
+    # Using previously defined values/traits (bare identifiers)
+    trait :super_eligible, eligible & premium_user
+    value :bonus_points, fn(:multiply, total_price, 0.1)
+
+    # ============================================================
+    # CASCADE EXPRESSIONS - on/base syntax
+    # ============================================================
+
+    value :user_tier do
+      on premium_user, 'premium'
+      on eligible, 'standard'
+      on is_verified, 'basic'
+      base 'guest'
+    end
+
+    value :discount_rate do
+      on senior, 0.25
+      on premium_user, 0.15
+      on is_active, 0.05
+      base 0.0
+    end
+
+    # Cascades using complex conditions
+    value :risk_level do
+      on complex_logic, 'low'
+      on needs_attention, 'high'
+      on is_active, 'medium'
+      base 'unknown'
+    end
+
+    # ============================================================
+    # NESTED INPUT REFERENCES - Deep field access
+    # ============================================================
+
+    # Direct nested field access (parsed as multi-part input reference)
+    value :first_item_name, input.items.name
+    value :all_prices, input.items.price
+
+    # Operations on nested fields (broadcasting)
+    value :discounted_prices, input.items.price * 0.9
+
+    # ============================================================
+    # LITERALS - All supported literal types
+    # ============================================================
+
+    # Number literals (integers and floats)
+    value :constant_int, 42
+    value :constant_float, 3.14159
+
+    # String literals (double quotes only)
+    value :greeting, 'Hello, World!'
+
+    # Boolean literals
+    value :always_true, true
+    value :always_false, false
+
+    # ============================================================
+    # EDGE CASES AND LIMITATIONS
+    # ============================================================
+
+    # Complex parenthesized expressions
+    value :nested_parens, ((input.age + 10) * 2) / (input.score - 20)
+
+    # Multiple operators in sequence
+    value :operator_chain, input.age + 10 - 5 + 20 - 3
+
+    # Nested function calls
+    value :nested_functions, fn(:round, fn(:multiply, fn(:add, input.score, 10), 1.5))
+
+    # ============================================================
+    # NOT SUPPORTED (would fail in text parser)
+    # ============================================================
+
+    # These are commented out as they would cause parse errors:
+
+    # Array literals in expressions
+    # value :array_literal, [1, 2, 3, 4, 5]
+    # value :max_of_array, fn(:max, [input.age, 30, 50])
+
+    # Method-style function calls
+    # value :method_style, fn.add(input.age, 10)
+
+    # Sugar method calls on fields
+    # trait :long_status, input.status.length > 5
+    # value :name_chars, input.status.size
+
+    # Ternary/conditional expressions
+    # value :conditional, eligible ? 100 : 0
+
+    # Hash literals
+    # value :options, { min: 0, max: 100 }
+
+    # ref() syntax
+    # trait :ref_example, ref(:eligible) & ref(:premium_user)
+
+    # Symbol literals (except in fn() calls)
+    # value :symbol_value, :active
+
+    # Power operator
+    # value :squared, input.age ** 2
+
+    # Comments within the DSL
+    # value :test, 42 # this would fail
+  end
+end
+
+# Test the schema with the text parser
+if __FILE__ == $0
+  require_relative '../lib/kumi/text_parser'
+
+  # Create a clean schema without comments for text parser testing
+  schema_text = <<~SCHEMA
+    schema do
+      input do
+        integer :age, domain: 18..65
+        float :score, domain: 0.0..100.0
+        string :status, domain: %w[active inactive suspended]
+        boolean :verified
+        any :metadata
+    #{'    '}
+        array :items do
+          string :name
+          float :price
+          integer :quantity
+        end
+      end
+
+      value :total_price, input.items.price + input.items.quantity
+      value :price_diff, input.items.price - 10.0
+      value :scaled_score, input.score * 1.5
+      value :average_score, input.score / 2
+      value :remainder, input.age % 10
+      value :complex_calc, (input.score * 2.5) + (input.age / 2) - 10
+    #{'  '}
+      trait :adult, input.age >= 18
+      trait :senior, input.age > 60
+      trait :young_adult, input.age <= 25
+      trait :is_teen, input.age < 20
+      trait :perfect_score, input.score == 100.0
+      trait :not_perfect, input.score != 100.0
+      trait :is_active, input.status == "active"
+      trait :not_suspended, input.status != "suspended"
+      trait :is_verified, input.verified == true
+      trait :not_verified, input.verified == false
+    #{'  '}
+      trait :eligible, (input.age >= 18) & (input.verified == true) & (input.status == "active")
+      trait :premium_user, is_verified & is_active & (input.score > 80.0)
+      trait :needs_attention, (input.status == "suspended") | (input.verified == false)
+      trait :special_case, senior | (input.score >= 95.0)
+      trait :complex_logic, (adult & is_verified) | (senior & is_active)
+    #{'  '}
+      value :absolute_diff, fn(:abs, fn(:subtract, input.score, 50.0))
+      value :rounded_score, fn(:round, input.score)
+      value :clamped_score, fn(:clamp, input.score, 20.0, 80.0)
+      value :name_length, fn(:string_length, input.status)
+      value :uppercase_status, fn(:upcase, input.status)
+      value :lowercase_status, fn(:downcase, input.status)
+      value :total_items, fn(:sum, input.items.quantity)
+      value :item_count, fn(:size, input.items)
+      value :max_price, fn(:max, input.items.price)
+      value :min_price, fn(:min, input.items.price)
+    #{'  '}
+      trait :super_eligible, eligible & premium_user
+      value :bonus_points, fn(:multiply, total_price, 0.1)
+    #{'  '}
+      value :user_tier do
+        on premium_user, "premium"
+        on eligible, "standard"
+        on is_verified, "basic"
+        base "guest"
+      end
+    #{'  '}
+      value :discount_rate do
+        on senior, 0.25
+        on premium_user, 0.15
+        on is_active, 0.05
+        base 0.0
+      end
+    #{'  '}
+      value :risk_level do
+        on complex_logic, "low"
+        on needs_attention, "high"
+        on is_active, "medium"
+        base "unknown"
+      end
+    #{'  '}
+      value :first_item_name, input.items.name
+      value :all_prices, input.items.price
+      value :discounted_prices, input.items.price * 0.9
+    #{'  '}
+      value :constant_int, 42
+      value :constant_float, 3.14159
+      value :greeting, "Hello, World!"
+      value :always_true, true
+      value :always_false, false
+    #{'  '}
+      value :nested_parens, ((input.age + 10) * 2) / (input.score - 20)
+      value :operator_chain, input.age + 10 - 5 + 20 - 3
+      value :nested_functions, fn(:round, fn(:multiply, fn(:add, input.score, 10), 1.5))
+    end
+  SCHEMA
+
+  puts 'Testing text parser with comprehensive DSL...'
+  puts '=' * 60
+
+  begin
+    # Validate the schema
+    diagnostics = Kumi::TextParser.validate(schema_text)
+
+    if diagnostics.empty?
+      puts '✅ Schema is valid!'
+
+      # Parse and show some info
+      ast = Kumi::TextParser.parse(schema_text)
+      puts "\nParsed successfully!"
+      puts "- Input fields: #{ast.inputs.map(&:name).join(', ')}"
+      puts "- Values: #{ast.attributes.count}"
+      puts "- Traits: #{ast.traits.count}"
+    else
+      puts '❌ Schema has errors:'
+      diagnostics.to_a.each do |diagnostic|
+        puts "  Line #{diagnostic.line}, Column #{diagnostic.column}: #{diagnostic.message}"
+      end
+    end
+  rescue StandardError => e
+    puts "❌ Parser error: #{e.message}"
+    puts e.backtrace.first(5)
+  end
+end