grammy 0.10 → 0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d714a645eea4ccadea369eafd8adf22ced89afc16b114fc80fa913bd5d06b73a
-  data.tar.gz: 480c0b3c9eea43baf7702359ae8b27105c44ae87d8f88d3c86a56101b8c77b16
+  metadata.gz: 62d6469d52259f78e46fdfcea47da7ac7c8a9b5b77e7cbcd3e685902c3410f98
+  data.tar.gz: 3abad7fd1e8c3fbc6f6ed6e9710c5bfb97728ef0efeadd01f15062bd1a38f971
 SHA512:
-  metadata.gz: f1bcba79afd6e4280db5be638510576fdd810feb6628f58ab68fb7666dcb5b58ef8cbeacc03ae03b4104af4a8b6d477ec6df90a5b539b4d2265016b261ae6a0d
-  data.tar.gz: e67c989b6a0759a0d189c782b8cb5b17fb8737e210bd373ce76dd6a9fe219a3c3ac28c276e2937537a7bd0b752dd8f5514c41db6727c2f06b259b193f861d3fd
+  metadata.gz: f301f90f661a08429f5bb956f030bc60b8950e0b108acbf7633eb2b1980d27021476602d640901c4dbc4de159bc651f0ac75bd634b6bc769c68149bf2a3e3721
+  data.tar.gz: e84cd332a524c119d615f7ba814edcd2ccabd2debe91e83f1da133cbd11b9475455d924a62a339706f050741d2a236a0dc8fc014d8cc2c8a6c8f40ca5cba3c6a

data/docs/TODO.md

@@ -1,5 +1,141 @@
 # TODO
 
+## IMMEDIATE (DO NOT COMMIT!) - Check these off below
+
+- [ ] Update gem
+    - Also allow other person to use `preserves` gem name
+
+- [ ] AI thread:
+
+~~~markdown
+Prompt: Allow user-defined combinators to wrap non-terminal rules
+
+## Problem
+
+Currently, user-defined combinators in Grammy can only wrap terminals (matchers), not rules (non-terminals).
+
+Example that DOESN'T work:
+```ruby
+class MyGrammar < Grammy::Grammar
+  start :expr
+  rule(:expr) { parens(number) }
+  rule(:number) { reg(/\d+/) }  # Non-terminal rule
+
+  def parens(exp) = str("(") + exp + str(")")
+end
+```
+
+When `number` is called as a rule, it executes immediately and returns a `ParseTree`, not a `Matcher`. The sequence combinator (`+`) then fails because it can't work with ParseTrees.
+
+Example that DOES work (using a terminal):
+```ruby
+class MyGrammar < Grammy::Grammar
+  start :expr
+  rule(:expr) { parens(number) }
+  terminal(:number, /\d+/)  # Terminal - returns a Matcher
+
+  def parens(exp) = str("(") + exp + str(":")
+end
+```
+
+## Goal
+
+Make it possible to pass rule references (non-terminals) to user-defined combinators, so `parens(number)` works whether `number` is a terminal or a rule.
+
+## Context
+
+- File: `lib/grammy/grammar.rb`
+- When a rule is called within a rule block, it should return something that can be composed with combinators
+- The grammar DSL currently uses `instance_eval` to evaluate rule blocks
+- Test file: `spec/grammar/user_defined_combinator_spec.rb`
+
+## Constraints
+
+- Don't break existing functionality with terminals
+- Maintain the clean DSL syntax
+- Keep the rule execution model intact for normal rule calls
+- Should work with all combinators (`+`, `|`, `[]`, etc.)
+
+## Test Case
+
+```ruby
+class UserDefinedGrammar < Grammy::Grammar
+  start :us
+  rule(:us) { parens(number) }
+  rule(:number) { reg(/\d+/) }  # Should work as a rule now
+
+  def parens(exp) = str("(") + exp + str(")")
+end
+
+# Should parse "(123)" successfully
+```
+~~~
+
+- [ ] COMMIT: BUGFIX: rules are able to reference other nested rules
+    - lib/grammy/grammar.rb and tests
+
+- [ ] Tree.each specs
+    - git fixup where I added `Enumerable`
+    - move specs to proper place
+    - commit message should mention pre-order traversal.
+- [ ] Add AST transformation DSL
+    - I may have already tried to commit it, but missed `lib/ast*` and `spec/ast_spec.rb`
+    - Does `tree.each` traversal help?
+- [ ] AI-assisted refactoring (suggestions, missing, etc)
+    - [x] tests
+    - [ ] code
+        - [ ] Rename Matcher to Primitive or Combinator?
+            - [ ] I suppose `custom.rb` would be the only other file in `combinators`
+    - [ ] README
+    - [ ] TODO
+- [ ] add `Makefile` / `Rakefile` rules for common tasks (AI) (BELOW)
+    - [ ] publish to RubyGems (BELOW)
+        - [ ] check that the `CHANGELOG` is updated
+    - [ ] update version (BELOW)
+    - [ ] update dependencies (BELOW)
+        - [ ] Ruby
+        - [ ] gems
+        - [ ] RuboCop (see if any rules need updated config)
+        - [ ] bun
+
+- [x] START ON STONE 0.10!
+
+- [ ] CHANGELOG
+
+- [ ] https://guides.rubygems.org/security/
+
+- [ ] combinator aliases (BELOW)
+    - [ ] `zero_or_more`
+    - [ ] `one_or_more`
+    - [ ] `zero_or_one`, `optional` (and maybe `_opt`?)
+    - [ ] update README/TODO
+
+- Release VERSION 0.11
+
+- [ ] indentation (BELOW)
+    - [ ] README updates
+- [ ] CHANGELOG
+
+- Release VERSION 0.12
+
+- [ ] line continuation (BELOW)
+    - [ ] README updates
+- [ ] CHANGELOG
+
+- Release VERSION 0.13
+
+- [ ] error handling (BELOW)
+- [ ] CHANGELOG
+
+- Release VERSION 0.14
+
+- [ ] AI prompts
+    - [ ] Copilot
+    - [ ] Claude
+    - [ ] ChatGPT
+- [ ] easier debugging (lib/extensions/debug.rb)
+- [ ] CHANGELOG
+
 Here's a list of things I'd like to complete to make Grammy **great**.
 
 ## Scanner
@@ -37,7 +173,7 @@ Here's a list of things I'd like to complete to make Grammy **great**.
     - [x] tree transformation DSL
     - [ ] AST builder
     - [ ] actions in grammar rules
-- [ ] user-defined combinators
+- [x] user-defined combinators
 - [ ] error handling
     - [ ] error messages
     - [ ] error recovery
@@ -141,7 +277,7 @@ Here's a list of things I'd like to complete to make Grammy **great**.
 
 - [x] publish a gem
     - [x] `gemspec`
-    - [x] upload to RubyGems
+    - [x] publish to RubyGems
 - [ ] CLI
     - [ ] `bin/grammy`
     - [ ] install with the gem

data/lib/disable_ssl_verify.rb

@@ -0,0 +1,5 @@
+# Temporary workaround for SSL certificate verification issues
+# Used by Makefile when downloading remote RuboCop configs.
+require "openssl"
+OpenSSL::SSL.__send__(:remove_const, :VERIFY_PEER)
+OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

data/lib/extensions/array.rb

@@ -0,0 +1,13 @@
+class Array
+  unless self.respond_to?(:wrap)
+    def self.wrap(object)
+      if object.nil?
+        []
+      elsif object.respond_to?(:to_ary)
+        object.to_ary || [object]
+      else
+        [object]
+      end
+    end
+  end
+end

data/lib/grammy/ast/transformer.rb

@@ -1,25 +1,25 @@
-# require "grammy/tree"
+require "grammy/tree"
 
 
-# # The purpose of the Transformer is to provide a simpler DSL within `transform` blocks.
-# # If the `transform` is for a ParseTree, you can use `name`, `children`, and `build(child)`.
-# # If the `transform` is for a Token, you can use `name`, `token`, and `text`.
-# module Grammy
-#   class AST < Tree
-#     class Transformer
+# The purpose of the Transformer is to provide a simpler DSL within `transform` blocks.
+# If the `transform` is for a ParseTree, you can use `name`, `children`, and `build(child)`.
+# If the `transform` is for a Token, you can use `name`, `token`, and `text`.
+module Grammy
+  class AST < Tree
+    class Transformer
 
-#       def initialize(node, node_class)
-#         @node = node
-#         @node_class = node_class
-#       end
+      def initialize(node, node_class)
+        @node = node
+        @node_class = node_class
+      end
 
-#       def build(child) = @node_class.build(child)
+      def build(child) = @node_class.build(child)
 
-#       def name = @node.name.to_sym
-#       def children = @node.children
-#       def token = @node
-#       def text = token.text
+      def name = @node.name.to_sym
+      def children = @node.children
+      def token = @node
+      def text = token.text
 
-#     end
-#   end
-# end
+    end
+  end
+end

data/lib/grammy/ast.rb

@@ -9,12 +9,17 @@ module Grammy
       def transform(name, &blk) = transforms[name.to_sym] = blk
 
       # The `node` will be a ParseTree or a Token. The code can handle other trees and leaves though.
+      # TODO: FIXME: Change this to use `node.each`, checking each node to see if there is a transform.
+      # ... if there isn't, we should treat the node's children as belonging to the node's parent.
       def build(node)
-        blk = transforms[node.name.to_sym]
+        node_name = node.respond_to?(:name) ? node.name.to_sym : nil
+        blk = transforms[node_name]
         if blk
-          Transformer.new(node, self).instance_exec(&blk)
+          Grammy::AST::Transformer.new(node, self).instance_exec(&blk)
         elsif node.respond_to?(:children)
-          node.children.flat_map(&method(:build))
+          # TODO: FIXME: We should pass the children, not ourself.
+          node.with(children: node.children.map{ build(it) })
+          # node
         else
           node
         end

data/lib/grammy/grammar.rb

@@ -2,6 +2,7 @@ require "grammy/combinator/primitives"
 require "grammy/errors"
 require "grammy/scanner"
 require "grammy/parse_tree"
+require_relative "../extensions/array"
 
 
 module Grammy
@@ -13,9 +14,9 @@ module Grammy
       # DSL for defining grammar rules.
       def start(rule_name) = @start_rule = rule_name
       def rule(name, &)
-        rule_proc = lambda { |_|
+        rule_proc = lambda {
           results = instance_eval(&)
-          children = Array(results).flatten.map { |result|
+          children = Array.wrap(results).flatten.map { |result|
             result.is_a?(Grammy::Matcher) ? result.match(@scanner) : result
           }
           Grammy::ParseTree.new(name.to_s, children.flatten)
@@ -59,7 +60,7 @@ module Grammy
     # Primitive combinators will need access to the scanner.
     def initialize(scanner) = @scanner = scanner
 
-    def execute_rule(rule_name) = instance_eval(&rules[rule_name])
+    def execute_rule(rule_name) = public_send(rule_name)
     def rules = self.class.rules
 
   end

data/lib/grammy/grammar.rb.bak

@@ -0,0 +1,66 @@
+require "grammy/combinator/primitives"
+require "grammy/errors"
+require "grammy/scanner"
+require "grammy/parse_tree"
+
+
+module Grammy
+  class Grammar
+
+    include Grammy::Combinator::Primitives
+
+    class << self
+      # DSL for defining grammar rules.
+      def start(rule_name) = @start_rule = rule_name
+      def rule(name, &)
+        rule_proc = lambda {
+          results = instance_eval(&)
+          children = Array(results).flatten.map { |result|
+            result.is_a?(Grammy::Matcher) ? result.match(@scanner) : result
+          }
+          Grammy::ParseTree.new(name.to_s, children.flatten)
+        }
+        define_method(name, &rule_proc)
+        rules[name] = rule_proc
+      end
+
+      # Examples:
+      #   terminal(:number, /\d+/)
+      #   terminal(:number) { /\d+/ }
+      #   terminal(:open_paren, "(")
+      #   terminal(:open_paren) { "(" }
+      def terminal(name, pattern = nil, &block)
+        fail ArgumentError, "may only supply a pattern OR a block to #{__callee__}" if pattern && block
+        pattern ||= yield if block
+        if pattern.is_a?(Regexp)
+          terminal_proc = -> { Grammy::Matcher::Regexp.new(pattern) }
+        else
+          terminal_proc = -> { Grammy::Matcher::String.new(pattern) }
+        end
+        define_method(name, &terminal_proc)
+        rules[name] = terminal_proc
+      end
+      alias token terminal
+
+      # Access to the rules.
+      def start_rule = @start_rule || @rules.first || :start
+      def rules = @rules ||= {}
+
+      # Parse an input using the grammar.
+      def parse(input, start: start_rule)
+        scanner = Grammy::Scanner.new(input)
+        grammar = self.new(scanner)
+        result = grammar.execute_rule(start)
+        fail(Grammy::ParseError, "Parsing failed at location #{scanner.location}") if result.nil? || result.empty? && !scanner.input.empty?
+        result
+      end
+    end
+
+    # Primitive combinators will need access to the scanner.
+    def initialize(scanner) = @scanner = scanner
+
+    def execute_rule(rule_name) = send(rule_name)
+    def rules = self.class.rules
+
+  end
+end

data/lib/grammy/location.rb

@@ -1,25 +1,25 @@
 module Grammy
-  # row/column are 1-based.
-  # index is 0-based, with respect to the full input string.
-  class Location < Data.define(:row, :column, :index)
+  # line/column are 1-based.
+  # offset is 0-based character position in the full input string.
+  class Location < Data.define(:line, :column, :offset)
 
-    def self.new(row = 1, column = 1, index = 0)
+    def self.new(line = 1, column = 1, offset = 0)
       super
     end
 
     def advance(text)
       newline_count = text.count("\n")
-      new_index = index + text.length
-      new_row = row + newline_count
+      new_offset = offset + text.length
+      new_line = line + newline_count
       if newline_count.zero?
         new_column = column + text.length
       else
         new_column = text.length - text.rindex("\n")
       end
-      Location.new(new_row, new_column, new_index)
+      Location.new(new_line, new_column, new_offset)
     end
 
-    def to_s = "(#{row},#{column})"
+    def to_s = "(#{line},#{column})"
     def inspect = to_s
     def pretty_print(pp) = pp.text inspect # For IRB output.
 

data/lib/grammy/matcher/sof.rb

@@ -9,7 +9,7 @@ module Grammy
       def initialize = super(nil)
 
       def match(scanner)
-        scanner.location.index.zero? ? Match.new(nil, scanner.location, scanner.location) : nil
+        scanner.location.offset.zero? ? Match.new(nil, scanner.location, scanner.location) : nil
       end
 
     end

data/lib/grammy/scanner.rb

@@ -20,7 +20,7 @@ module Grammy
     # Returns `nil` if pattern does not match.
     # Otherwise, updates @location and returns a Match object.
     def match_string(pattern)
-      return nil if @location.index >= @input.size
+      return nil if @location.offset >= @input.size
       return nil unless remaining_input.start_with?(pattern)
       match_text(pattern)
     end
@@ -29,7 +29,7 @@ module Grammy
     # Returns `nil` if pattern does not match.
     # Otherwise, updates @location and returns a Match object.
     def match_regexp(pattern)
-      return nil if @location.index >= @input.size
+      return nil if @location.offset >= @input.size
       anchored_regex = Regexp.new("\\A(?:#{pattern.source})", pattern.options)
       match = remaining_input.match(anchored_regex)
       return nil unless match
@@ -54,11 +54,11 @@ module Grammy
     end
 
     def finished?
-      @location.index == @input.size
+      @location.offset == @input.size
     end
 
     private def remaining_input
-      @input[@location.index..]
+      @input[@location.offset..]
     end
 
     # Returns matched text and its location (or nil).

data/lib/grammy/token.rb

@@ -13,6 +13,7 @@ module Grammy
     end
 
     # TODO: Extract this into a Data helper. Better yet, create a Value class for value objects.
+    # UHH: I think Data **already** has `with`!
     def with(value:) = self.class.new(name, match, value)
 
     def text = match.text

data/lib/grammy/tree.rb

@@ -1,26 +1,48 @@
 module Grammy
-  class Tree < Data.define(:name, :children)
+  class Tree
+
     INDENTATION = 4
 
+    include Enumerable
+
+    attr_reader :name
+    attr_reader :children
+
     # Make it easier to create nested trees.
-    def self.new(name, children = [], &block)
+    def initialize(name, children = [], &block)
       children = yield if block
-      super(name:, children: Array(children))
+      @name = name
+      @children = Array(children)
     end
 
-    def empty? = children.flatten.compact.empty?
+    def empty? = leaves.empty?
     def leaves = children.flat_map { it.is_a?(self.class) ? it.leaves : it }.compact
 
     def to_s(level = 0) = ([to_s_base(level)] + children.map{ to_s_child(it, level) }).join("\n")
 
-    def inspect(level = 0) = ([inspect_base(level)] + children.map{ to_s_child(it, level) }).join("\n")
+    # Walk the tree in pre-order, yielding every node (including self).
+    def each(&block)
+      return enum_for(:each) unless block
+      yield self
+      children.each do |child|
+        if child.respond_to?(:children) && child.respond_to?(:each)
+          child.each(&block)
+        else
+          yield child
+        end
+      end
+    end
+
+    def inspect(level = 0) = ([inspect_base(level)] + Array(children&.map{ inspect_child(it, level) })).join("\n")
     def to_h = {name:, children: children.map(&:to_h)}
     def pretty_print(pp) = pp.text inspect # For IRB output.
 
     private def to_s_base(level) = "#{indent(level)}#{name}"
-    private def inspect_base(level) = "#{indent(level)}#<#{class_name} #{name.inspect}>"
     private def to_s_child(child, level) = child.is_a?(self.class) ? child.to_s(level + 1) : to_s_leaf(child, level + 1)
     private def to_s_leaf(leaf, level) = "#{indent(level)}#{leaf}"
+    private def inspect_base(level) = "#{indent(level)}#<#{class_name} #{name.inspect}>"
+    private def inspect_child(child, level) = child.is_a?(self.class) ? child.inspect(level + 1) : inspect_leaf(child, level + 1)
+    private def inspect_leaf(leaf, level) = "#{indent(level)}#{leaf.inspect}"
     private def indent(level) = " " * (level * INDENTATION)
     private def class_name = self.class.name.split("::").last
 

data/lib/grammy.rb

@@ -4,6 +4,6 @@ require "grammy/grammar"
 
 module Grammy
 
-  VERSION = File.read(File.join("..", "VERSION"))
+  VERSION = "0.10"
 
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: grammy
 version: !ruby/object:Gem::Version
-  version: '0.10'
+  version: '0.11'
 platform: ruby
 authors:
 - Craig Buchek
@@ -24,6 +24,8 @@ files:
 - docs/CHANGELOG.md
 - docs/LICENSE.md
 - docs/TODO.md
+- lib/disable_ssl_verify.rb
+- lib/extensions/array.rb
 - lib/extensions/debugging.rb
 - lib/grammy.rb
 - lib/grammy/ast.rb
@@ -31,6 +33,7 @@ files:
 - lib/grammy/combinator/primitives.rb
 - lib/grammy/errors.rb
 - lib/grammy/grammar.rb
+- lib/grammy/grammar.rb.bak
 - lib/grammy/location.rb
 - lib/grammy/match.rb
 - lib/grammy/matcher.rb
@@ -68,7 +71,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.7
+rubygems_version: 3.6.9
 specification_version: 4
 summary: A PEG parsing library with a simple DSL
 test_files: []