kumi-parser 0.0.3 → 0.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 14750f2db79c5e3125f477a55452f97c53a919dfd14f6731ebef89169b6a25a3
-  data.tar.gz: 2c617dd3c673d580c0ea4dcee97d0a82cc34d69e8d23aefd7e6cc53c9e288fcf
+  metadata.gz: 0067c520152fae1c2285b2628f3b8b5d25eff145ae61780cf3d283310816c336
+  data.tar.gz: 4c604f36cd250a8d4672fef7c2acc667c7de498fe0ee088980bcbd8a744b8d5e
 SHA512:
-  metadata.gz: fe96cecd1682fde3006a890413e82aa2d8963400a160b5a808d4cb841bfe702c72624ab4b3abb9abc3829745e4355fcecb7f096300c5b1f13e2b8cdcd2475f34
-  data.tar.gz: ab9576fa7828bedcd731567a37e1b977691cc5aa4a87ba2b2c1faf3a83299eabe466c4a62165548cef4178dd3648f012167378581b70a58d0e2435206fce5d5f
+  metadata.gz: c3d3342f43d5bb41b140f40d02d3eaac08abeb88509d4c4f4ce127e38ec3583e7566c68d8a96956d2877a2819b046663e2bf95aa246763bcf7556cf4df380934
+  data.tar.gz: f7891f6e388e224ebfd0da8d027d331c6794325e0d27715f8800f4aa9124dd3421fc36ca3afb1ac10e6f4e8c5db61f610a18e9db0a33d21c9292efbbd5fa45a7

data/CLAUDE.md

@@ -0,0 +1,120 @@
+# Kumi Parser - Technical Context
+
+## Current Architecture (January 2025)
+
+## Key Files
+
+- `lib/kumi/parser/smart_tokenizer.rb` - Tokenizer with context tracking
+- `lib/kumi/parser/direct_parser.rb` - Parser implementation (renamed from direct_ast_parser.rb)
+- `lib/kumi/parser/token_metadata.rb` - Token types and metadata
+- `lib/kumi/parser/text_parser.rb` - Public API maintaining compatibility
+- `lib/kumi/parser/base.rb` - Core parsing interface
+- `lib/kumi/parser/syntax_validator.rb` - Validation with proper diagnostics
+- `lib/kumi/parser/errors.rb` - Custom error types
+
+## Important Syntax Rules
+
+- **Functions**: `fn(:symbol, args...)` only (no dot notation like `fn.max()`)
+- **Operators**: Standard precedence (*/% > +- > comparisons > & > |)
+- **Array access**: Uses `array[index]` syntax (converted to `:at` function internally)
+- **Equality**: `==` and `!=` operators (converted from `:eq`/`:ne` tokens)
+- **Multi-line expressions**: Parser skips newlines within expressions
+- **Cascade**: `value :name do ... on condition, result ... base result ... end`
+- **Constants**: Text parser cannot resolve Ruby constants - use inline values
+
+## AST Structure & Compatibility
+
+All nodes from `Kumi::Syntax::*` (defined in main kumi gem):
+- `Root(inputs, attributes, traits)`
+- `InputDeclaration(name, domain, type, children)`
+- `ValueDeclaration(name, expression)`
+- `TraitDeclaration(name, expression)`
+- `CallExpression(fn_name, args)`
+- `InputReference(name)` / `InputElementReference(path)`
+- `DeclarationReference(name)`
+- `Literal(value)`
+- `CascadeExpression(cases)` / `CaseExpression(condition, result)`
+- `ArrayExpression(elements)`
+
+**Ruby DSL Compatibility**:
+- Cascade conditions: Simple trait references wrapped in `all?([trait])` function calls
+- Array access: `[index]` becomes `CallExpression(:at, [array, index])`
+- Operators: `:eq` → `:==`, `:ne` → `:!=` for consistency
+- Constants: Ruby constants resolved to values in DSL, remain as `DeclarationReference` in text parser
+
+## Debugging & Testing
+
+**View AST structure**:
+```ruby
+ast = Kumi::Parser::TextParser.parse(schema)
+puts Kumi::Support::SExpressionPrinter.print(ast)
+# => (Root
+#      inputs: [(InputDeclaration :income :float)]
+#      attributes: [(ValueDeclaration :tax (CallExpression :+ ...))]
+#      traits: [(TraitDeclaration :adult (CallExpression :>= ...))])
+```
+
+**Quick validation test**:
+```ruby
+ruby -r./lib/kumi/parser/text_parser -e "p Kumi::Parser::TextParser.valid?('schema do input do float :x end end')"
+```
+
+**Compare with Ruby DSL**:
+```ruby
+# Define schema in Ruby
+module TestSchema
+  extend Kumi::Schema
+  schema do
+    input do
+      float :income
+    end
+    value :tax, fn(:calc, input.income)
+  end
+end
+
+# Parse equivalent text
+text_ast = Kumi::Parser::TextParser.parse(<<~KUMI)
+  schema do
+    input do
+      float :income
+    end
+    value :tax, fn(:calc, input.income)
+  end
+KUMI
+
+# Compare ASTs
+ruby_ast = TestSchema.__syntax_tree__
+text_ast == ruby_ast # Should be true
+```
+
+- Tax schema in `spec/kumi/parser/text_parser_example tax_schema_spec.rb` is canonical test
+- Run all tests: `rspec spec/kumi/parser/` 
+- Integration tests: `rspec spec/kumi/parser/text_parser_integration_spec.rb`
+
+## Error Handling & Validation
+
+- **Parse errors**: `Kumi::Parser::Errors::ParseError` (internal) → `Kumi::Errors::SyntaxError` (public API)
+- **Tokenizer errors**: `Kumi::Parser::Errors::TokenizerError` with location info
+- **Diagnostics**: Use `SyntaxValidator` for detailed error reporting with line/column info
+- **Location tracking**: All tokens and AST nodes include `Kumi::Syntax::Location(file, line, column)`
+
+## Test Status (January 2025)
+
+✅ **All specs passing**: 32 examples, 0 failures, 1 pending
+- ✅ Syntax validation with proper diagnostics
+- ✅ AST compatibility with Ruby DSL (when constants aren't used)
+- ✅ Integration with analyzer and compiler
+- ✅ End-to-end execution testing
+- ✅ Error type compatibility
+
+## Known Limitations
+
+- **Ruby constants**: Text parser cannot resolve Ruby constants like `CONST_NAME` - use inline values instead
+- **Domain specification**: Parsing not fully implemented
+- **Diagnostic APIs**: Monaco/CodeMirror/JSON format methods not implemented
+
+## Performance
+
+- Tokenization: <1ms for typical schemas
+- Parsing: ~4ms for complete tax schema (21 values, 4 traits)
+- Direct AST construction eliminates transformation overhead

data/README.md

@@ -1,76 +1,73 @@
 # Kumi::Parser
 
-Text parser for [Kumi](https://github.com/amuta/kumi). Allows Kumi schemas to be written as plain text with syntax validation and editor integration.
+Text parser for [Kumi](https://github.com/amuta/kumi) schemas. Direct tokenizer → AST construction with ~4ms parse time.
 
 ## Installation
 
-Add this line to your application's Gemfile:
-
 ```ruby
 gem 'kumi-parser'
 ```
 
-And then execute:
-
-    $ bundle install
-
-Or install it yourself as:
-
-    $ gem install kumi-parser
-
 ## Usage
 
-### Basic Parsing
-
 ```ruby
 require 'kumi/parser'
 
-schema_text = <<~SCHEMA
+schema = <<~KUMI
   schema do
     input do
-      integer :age, domain: 18..65
-      string :status, domain: %w[active inactive]
+      float :income
+      string :status
     end
     
     trait :adult, input.age >= 18
-    value :bonus, 100
+    value :tax, fn(:calculate_tax, input.income)
   end
-SCHEMA
+KUMI
 
-# Parse and get AST
-ast = Kumi::Parser::TextParser.parse(schema_text)
+# Parse to AST
+ast = Kumi::Parser::TextParser.parse(schema)
 
-# Validate syntax
-diagnostics = Kumi::Parser::TextParser.validate(schema_text)
-puts "Valid!" if diagnostics.empty?
+# Validate
+Kumi::Parser::TextParser.valid?(schema) # => true
 ```
 
-### Editor Integration
+## API
 
-```ruby
-# Get diagnostics for Monaco Editor
-monaco_diagnostics = Kumi::Parser::TextParser.diagnostics_for_monaco(schema_text)
+- `parse(text)` → AST
+- `valid?(text)` → Boolean  
+- `validate(text)` → Array of error hashes
 
-# Get diagnostics for CodeMirror
-codemirror_diagnostics = Kumi::Parser::TextParser.diagnostics_for_codemirror(schema_text)
+## Syntax
 
-# Get diagnostics as JSON
-json_diagnostics = Kumi::Parser::TextParser.diagnostics_as_json(schema_text)
+```
+schema do
+  input do
+    <type> :<name>[, domain: <spec>]
+  end
+  
+  trait :<name>, <expression>
+  
+  value :<name>, <expression>
+  value :<name> do
+    on <condition>, <result>
+    base <result>
+  end
+end
 ```
 
-## API Reference
+**Function calls**: `fn(:name, arg1, arg2, ...)`  
+**Operators**: `+` `-` `*` `/` `%` `>` `<` `>=` `<=` `==` `!=` `&` `|`  
+**References**: `input.field`, `value_name`, `array[index]`
 
-- `parse(text)` - Parse schema text and return AST
-- `validate(text)` - Validate syntax and return diagnostics
-- `valid?(text)` - Quick validation check (returns boolean)
-- `diagnostics_for_monaco(text)` - Get Monaco Editor format diagnostics
-- `diagnostics_for_codemirror(text)` - Get CodeMirror format diagnostics  
-- `diagnostics_as_json(text)` - Get JSON format diagnostics
+## Architecture
 
-## Development
+- `smart_tokenizer.rb` - Context-aware tokenization with embedded metadata
+- `direct_ast_parser.rb` - Recursive descent parser, direct AST construction
+- `token_metadata.rb` - Token types, precedence, and semantic hints
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+See `docs/` for technical details.
 
 ## License
 
-MIT License
+MIT

data/lib/kumi/parser/base.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require_relative 'smart_tokenizer'
+require_relative 'direct_parser'
+require_relative 'errors'
+
+module Kumi
+  module Parser
+    # Text parser using tokenizer + direct AST construction
+    class Base
+      def self.parse(source, source_file: '<input>')
+        tokens = SmartTokenizer.new(source, source_file: source_file).tokenize
+        Kumi::Parser::DirectParser.new(tokens).parse
+      end
+
+      def self.valid?(source, source_file: '<input>')
+        parse(source, source_file: source_file)
+        true
+      rescue Errors::TokenizerError, Errors::ParseError
+        false
+      end
+
+      def self.validate(source, source_file: '<input>')
+        parse(source, source_file: source_file)
+        []
+      rescue Errors::TokenizerError, Errors::ParseError => e
+        [create_diagnostic(e, source_file)]
+      end
+
+      private
+
+      def self.create_diagnostic(error, source_file)
+        location = if error.is_a?(Errors::ParseError) && error.token
+                     error.token.location
+                   elsif error.respond_to?(:location)
+                     error.location
+                   else
+                     nil
+                   end
+
+        {
+          line: location&.line || 1,
+          column: location&.column || 1,
+          message: error.message,
+          severity: :error,
+          type: :syntax
+        }
+      end
+    end
+  end
+end