grammy 0.10

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d714a645eea4ccadea369eafd8adf22ced89afc16b114fc80fa913bd5d06b73a
+  data.tar.gz: 480c0b3c9eea43baf7702359ae8b27105c44ae87d8f88d3c86a56101b8c77b16
+SHA512:
+  metadata.gz: f1bcba79afd6e4280db5be638510576fdd810feb6628f58ab68fb7666dcb5b58ef8cbeacc03ae03b4104af4a8b6d477ec6df90a5b539b4d2265016b261ae6a0d
+  data.tar.gz: e67c989b6a0759a0d189c782b8cb5b17fb8737e210bd373ce76dd6a9fe219a3c3ac28c276e2937537a7bd0b752dd8f5514c41db6727c2f06b259b193f861d3fd

data/README.md

@@ -0,0 +1,314 @@
+# Grammy
+
+Grammy is a tool for generating parsers.
+You describe the language with a grammar written in a Ruby [DSL], similar to [EBNF].
+Grammy dynamically generates a parser from that description.
+You can then use the parser to parse strings into a [parse tree],
+and create an [AST] from the parse tree.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)
+[![Version](https://img.shields.io/badge/version-v0.1-blue.svg)](https://github.com/stone-lang/stone/blob/master/src/stone/version.rb)
+[![Gem Version](https://img.shields.io/gem/v/grammy)](https://rubygems.org/gems/grammy)
+
+## ToC
+
+- [Features](#features)
+- [Usage](#usage)
+- [Combinators](#combinators)
+- [Parse Tree](#parse-tree)
+- [AST](#ast)
+- [Tests](#tests)
+- [License](#license)
+- [Contributing](#contributing)
+- [Changelog](./docs/CHANGELOG.md)
+- [TODO](./docs/TODO.md)
+
+## Features
+
+Grammy implements a [PEG] (Parsing Expression Grammar) parser.
+But that's an implementation detail, possibly subject to change,
+if something better comes along (LPEG, GLR, etc).
+PEG parsers are quick (using "packrat" caching) and easy to use.
+They easily handle ambiguous grammars, left or right recursion, and infinite lookahead.
+
+You can use Grammy to parse complex languages.
+I'm going to be using it to write a full general-purpose programming language.
+But you can also use it to parse simpler languages, like:
+
+- JSON
+- CSV
+- external DSLs
+- configuration files
+- data formats
+- HTTP headers
+
+Basically, if you have an [EBNF] grammar, it should be easy to use Grammy to parse it.
+
+## Usage
+
+~~~ ruby
+require "grammy"
+require "arithmetic" # Your grammar file, as below.
+
+input = "1+2*3"
+parse_tree = Arithmetic.parse(input)
+ast = parse_tree.ast
+~~~
+
+I'm still experimenting with a few different DSL syntaxes for the grammar.
+
+### Class Methods
+
+NOTE: This is the only syntax currently implemented.
+
+~~~ ruby
+require 'grammy'
+
+class Arithmetic < Grammy::Grammar
+  # Specify which rule to start with.
+  start :expression
+
+  # Define the rules.
+  rule(:expression) { term + (plus + term)[0..] }
+  rule(:term) { factor + (times + factor)[0..] }
+  rule(:factor) { number | parens(expression) }
+  terminal(:number) { /\d+/ }  # You can also use `token` instead of `terminal`.
+  terminal(:plus) { "+" }      # A terminal rule matches a string or regex, without needing `str` or `reg`.
+  terminal(:times) { "*" }     # Other than that, it works like a normal rule.
+
+  # Define any custom combinators.
+  def parens(exp) = str("(") + expression + str(")")
+end
+~~~
+
+### Decorated Methods
+
+NOTE: This syntax has not been implemented yet.
+
+~~~ ruby
+require 'grammy'
+
+class Arithmetic < Grammy::Grammar
+  # Specify which rule to start with.
+  start :expression
+
+  # Define the rules.
+  rule def expression = term + (plus + term)[0..]
+  rule def term = factor + (times + factor)[0..]
+  rule def factor = number | parens(expression)
+  terminal def number = /\d+/
+  terminal def plus = "+"
+  terminal def times = "*"
+
+  # Define any custom combinators.
+  def parens(exp) = str("(") + expression + str(")")
+end
+~~~
+
+### Instance Methods
+
+NOTE: This syntax has not been implemented yet.
+
+~~~ ruby
+require 'grammy'
+
+class Arithmetic < Grammy::Grammar
+  # Specify which rule to start with.
+  start :expression
+
+  # Define the rules.
+  def expression = rule { term + (str("+") + term)[0..] }
+  def term = rule { factor + (str("*") + factor)[0..] }
+  def factor = rule { number | parens(expression) }
+  def number = terminal { /\d+/ }
+
+  # Define any custom combinators.
+  def parens(exp) = str("(") + expression + str(")")
+end
+~~~
+
+## Combinators
+
+Grammy uses combinators to define the grammar.
+Combinators are functions that take one or more parsers as arguments and return a new parser.
+
+Only a few primitive combinators are needed.
+
+### String
+
+The `str` combinator is used to match a string; it has an alias of `lit` (for "literal").
+
+~~~ ruby
+str("return")
+lit("return")
+~~~
+
+### Regex
+
+The `reg` combinator is used to match a regular expression.
+
+~~~ ruby
+reg(/\d+/)
+~~~
+
+### Sequence
+
+You use the `seq` combinator to specify a sequence of what to match.
+
+This example matches a sequence of a term, a plus sign, and another term.
+
+~~~ ruby
+seq(term, str("+"), term)
+~~~
+
+### Alternatives
+
+The `alt` combinator is used to specify alternatives (multiple choices) of what to match.
+
+This example matches either a plus sign or a minus sign.
+
+~~~ ruby
+alt(str("+"), str("-"))
+~~~
+
+> [!NOTE]
+> In practical use, this would most likely be done with a single `reg` call:
+>
+> ~~~ ruby
+> reg(/[+-]/)
+> ~~~
+
+### Repetition
+
+The `rep` combinator is used to specify repetition of what to match.
+You can specify the minimum and maximum number of repetitions, using a range.
+
+This example matches one or more digits.
+
+~~~ ruby
+rep(reg(/\d+/), 1..)
+~~~
+
+### Operator DSL
+
+You can use `+`, `|`, and `[]` operators in place of the named combinators.
+The `+` operator can be used in place of the `seq` combinator.
+The `|` operator can be used in place of the `alt` combinator.
+The `[]` operator can be used in place of the `rep` combinator.
+
+For example, the following two lines are equivalent:
+
+~~~ ruby
+seq(alt(term, expr), str("+"), rep(term, 0..1))
+(term | expr) + lit("+") + term[0..1]
+~~~
+
+### Start/End of Line/File
+
+The `eol` and `sol` combinators match the end of a line and the start of a line.
+The `eof` and `sof` combinators match the end of a file and the start of a file.
+
+## Whitespace
+
+The `wsp` combinator matches any whitespace characters, including tab, newline, carriage return, etc.
+
+## Parse Tree
+
+The internal nodes of the parse tree are ParseTree objects.
+The leaves of the parse tree are Token objects.
+
+The parse tree generated by the example above will look like this:
+
+~~~ ruby
+expected_parse_tree = ParseTree.new("expression", [
+  ParseTree.new("term", [
+    ParseTree.new("factor", [
+      ParseTree.new("number", [
+        Token.new("1")
+      ])
+    ])
+  ]),
+  Token.new("+"),
+  ParseTree.new("term", [
+    ParseTree.new("factor", [
+      ParseTree.new("number", [
+        Token.new("2")
+      ]),
+    Token.new("*"),
+    ParseTree.new("factor", [
+      ParseTree.new("number", [
+        Token.new("3")
+      ])
+    ])
+  ])
+])
+~~~
+
+
+## AST
+
+An AST (Abstract Syntax Tree) can be generated from the parse tree.
+Your AST base class can be a subclass of `Grammy::Tree`,
+and use the tree transformation DSL to derive the AST from the parse tree.
+
+Given the parse tree above, and this tree transformation DSL:
+
+~~~ ruby
+transform(:term) do |node|
+  left, op, right = node.children
+  AST::BinaryOp.new(op.text, [transform(left), transform(right)])
+end
+transform(:number) { |token| token.with(value: token.text.to_i) }
+~~~
+
+You'd end up with the following AST:
+
+~~~ ruby
+expected_ast = AST::Arithmetic.new(
+  AST::BinaryOp.new("+", [
+    AST::Number.new(1),
+    AST::BinaryOp.new("*", [
+      AST::Number.new(2),
+      AST::Number.new(3)
+    ])
+  ])
+)
+~~~
+
+## Tests
+
+~~~ shell
+rspec
+~~~
+
+## License
+
+Copyright (c) 2024-2025 by Craig Buchek and BoochTek, LLC.
+
+This code is licensed under the MIT License.
+See the [LICENSE] for the full details.
+
+## Contributing
+
+[PRs] and [issues] are welcome!
+
+
+## Release
+
+To release a new version, update the version in `lib/VERSION`,
+then build the gem and publish it to RubyGems.
+
+~~~ shell
+make publish
+~~~
+
+---
+
+[DSL]: https://en.wikipedia.org/wiki/Domain-specific_language
+[EBNF]: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form
+[AST]: https://en.wikipedia.org/wiki/Abstract_syntax_tree
+[parse tree]: https://en.wikipedia.org/wiki/Parse_tree
+[PEG]: https://en.wikipedia.org/wiki/Parsing_expression_grammar
+[License]: ./docs/LICENSE.md
+[PRs]: https://github.com/stone-lang/grammy/pulls
+[issues]: https://github.com/stone-lang/grammy/issues

data/docs/CHANGELOG.md

@@ -0,0 +1,15 @@
+# Changelog
+
+## 0.1.0 (TBD)
+
+Initial "release".
+
+### Added
+
+- README, including Usage section
+- CHANGELOG
+- Basic classes: `Grammar`, `Scanner`, `Parser`
+
+## Notes
+
+- Sections for each release (in order): Changed, Added, Removed, Updated, Fixed

data/docs/LICENSE.md

@@ -0,0 +1,21 @@
+# MIT License
+
+Copyright (c) 2025 Craig Buchek and BoochTek, LLC.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/docs/TODO.md

@@ -0,0 +1,157 @@
+# TODO
+
+Here's a list of things I'd like to complete to make Grammy **great**.
+
+## Scanner
+
+- [ ] line continuation
+    - [ ] `\` followed immediately by newline (traditional)
+    - [ ] newline followed immediately by `\` (more visible)
+    - [ ] skip leading whitespace on following line
+    - [ ] optional per grammar
+    - [ ] optional per rule
+- [ ] indentation
+
+## Grammar
+
+- [x] primitive combinators: `str` (AKA `lit`) and `reg`
+- [x] primitive combinators: `seq`, `alt`, `rep`
+- [x] primitive combinators: `eol`/`sol`, `eof`/`sof`
+- [x] primitive combinators: `wsp`
+- [x] add `_` to canonical names (where it makes sense)
+- [ ] combinator aliases
+    - [ ] `zero_or_more` / `_any`
+    - [ ] `one_or_more` / `_some`
+    - [ ] `zero_or_one` / `optional` (and maybe `_opt`?)
+- [x] rules
+    - [x] DSL syntax
+        - `rule(:expression) { term + (str("+") + term)[0..] }`
+    - [ ] try instance methods syntax
+        - `def expression = term + (str("+") + term)[0..]`
+    - [ ] try decorated methods syntax
+        - `rule def expression = term + (str("+") + term)[0..]`
+- [x] start rule
+    - [x] defaults to first defined rule
+- [x] `terminal` (AKA `token`) rules
+- [ ] AST generation
+    - [x] tree transformation DSL
+    - [ ] AST builder
+    - [ ] actions in grammar rules
+- [ ] user-defined combinators
+- [ ] error handling
+    - [ ] error messages
+    - [ ] error recovery
+    - [ ] `fail`/`catch` combinators?
+- [ ] packrat caching/memoization
+- [ ] lookahead predicates
+    - [ ] "and" lookahead predicate
+    - [ ] "not" lookahead predicate
+- [ ] "cut operator"
+    - [ ] combinator?
+    - [ ] automatic insertion
+- [ ] automatic left recursion support
+
+## Setup
+
+- [x] git (or jj)
+    - [ ] root directory only contains worktrees (git --bare; git worktree add)
+        - [ ] and maybe support files (`.gitignore`, `.vscode`, etc)
+        - [ ] add git aliases for `worktree add -b` and `switch`
+- [x] `.gitignore`
+- [x] GitHub project
+- [x] `.tool-versions`
+- [x] `Gemfile`
+- [x] `.rubocop.yml`
+- [x] `.markdownlint.yml`
+- [x] `.rspec`
+- [ ] `.irbrc`
+- [ ] `.pryrc`
+
+## Documentation
+
+- [x] comprehensive `README` (and keep it updated)
+    - [x] intro
+    - [x] table of contents
+    - [x] features
+    - [ ] installation
+    - [x] usage
+    - [x] tests
+    - [x] contributing
+    - [ ] badges (see [shields.io](https://shields.io/))
+        - [ ] version
+        - [x] license
+        - [ ] build status (https://github.com/OWNER/REPO/actions/workflows/WORKFLOW/badge.svg)
+        - [ ] test coverage
+        - [ ] dependencies status
+- [x] `TODO` (and keep it updated)
+- [ ] `CHANGELOG` (and keep it updated)
+- [x] `LICENSE` file (and keep it updated)
+- [ ] code of conduct
+- [ ] FAQ
+
+## Automation
+
+- [x] `Makefile` for "standard" common tasks
+- [x] `Rakefile` for "standard" common tasks for Ruby projects
+    - [ ] update version
+        - [ ] verify that CHANGELOG is updated
+    - [x] upload to RubyGems
+    - [ ] update dependencies
+        - [ ] Ruby
+        - [ ] gems
+        - [ ] RuboCop (see if any rules need updated config)
+        - [ ] bun
+- [x] CI/CD setup (GitHub Actions)
+- [x] linting
+    - [x] RuboCop
+    - [x] markdownlint
+    - [ ] Reek
+    - [ ] bundler-audit
+- [ ] test coverage
+- [ ] code quality metrics
+- [ ] git hooks (pre-commit, pre-push, etc)
+    - [ ] linting
+    - [ ] tests
+    - [ ] security checks (bundle audit, etc)
+- [ ] ISSUES_TEMPLATE
+- [ ] PULL_REQUEST_TEMPLATE
+- [ ] `.editorconfig`
+
+## Specs
+
+- [x] RSpec setup
+- [ ] unit tests
+    - [x] scanner
+    - [ ] combinators
+    - [x] grammar
+    - [x] parse tree
+    - [ ] AST builder
+- [ ] integration tests
+    - [x] parser
+    - [ ] parse tree
+    - [ ] grammar
+    - [ ] AST builder
+    - [ ] user-defined combinators
+    - [ ] rule actions
+- [ ] performance tests
+    - [ ] benchmarks
+- [ ] property (fuzzing) tests
+
+## Gem
+
+- [x] publish a gem
+    - [x] `gemspec`
+    - [x] upload to RubyGems
+- [ ] CLI
+    - [ ] `bin/grammy`
+    - [ ] install with the gem
+    - [ ] `--help`
+    - [ ] `--version`
+    - [ ] `--json` (use JSON nesting for parse tree nesting)
+    - [ ] `<grammar_file>` to output a serialized parse tree, with input from `STDIN`
+
+## Support
+
+- [ ] VS Code configuration
+    - [ ] tasks (`launch.json`)
+    - [ ] recommended extensions

data/lib/extensions/debugging.rb

@@ -0,0 +1,46 @@
+# The idea here is to allow `binding.break` and `binding.pry` to work
+# only when running tests or when the DEBUG environment variable is set,
+# and not have RuboCop whine about it.
+
+
+require "debug/config"
+DEBUGGER__::CONFIG[:skip_path] = Array(DEBUGGER__::CONFIG[:skip_path]) + [binding.source_location.first]
+
+
+class Binding
+
+  def break!
+    return unless should_debug?
+    require "debug"
+    require "binding_of_caller"
+    caller_binding = binding.of_caller(1)
+    # FIXME: This causes the breakpoint to happen **here**, instead of in the caller.
+    # The `skip_path` option SHOULD fix it, but I couldn't get it to work at all.
+    # I've added the pre-command `up` as a work-around.
+    # But it still shows **this** source first, before showing the "correct" location.
+    # TODO: File a bug at https://github.com/ruby/debug/issues, requesting parity with Pry.
+    caller_binding.break(pre: "up ;; list")
+  end
+
+  def pry!
+    return unless should_debug?
+    require "pry"
+    require "binding_of_caller"
+    caller_binding = binding.of_caller(1)
+    caller_binding.pry
+  end
+
+  # TODO: Consider moving these elsewhere.
+  private def should_debug?
+    debug_enabled? || ((defined?(RSpec) || defined?(Minitest)) && !debug_disabled?)
+  end
+
+  private def debug_enabled?
+    ENV.fetch("DEBUG", "0") != "0"
+  end
+
+  private def debug_disabled?
+    ENV.fetch("DEBUG", "1") == "0"
+  end
+
+end

data/lib/grammy/ast/transformer.rb

@@ -0,0 +1,25 @@
+# require "grammy/tree"
+
+
+# # The purpose of the Transformer is to provide a simpler DSL within `transform` blocks.
+# # If the `transform` is for a ParseTree, you can use `name`, `children`, and `build(child)`.
+# # If the `transform` is for a Token, you can use `name`, `token`, and `text`.
+# module Grammy
+#   class AST < Tree
+#     class Transformer
+
+#       def initialize(node, node_class)
+#         @node = node
+#         @node_class = node_class
+#       end
+
+#       def build(child) = @node_class.build(child)
+
+#       def name = @node.name.to_sym
+#       def children = @node.children
+#       def token = @node
+#       def text = token.text
+
+#     end
+#   end
+# end

data/lib/grammy/ast.rb

@@ -0,0 +1,27 @@
+require "grammy/tree"
+require "grammy/ast/transformer"
+
+
+module Grammy
+  class AST < Tree
+
+    class << self
+      def transform(name, &blk) = transforms[name.to_sym] = blk
+
+      # The `node` will be a ParseTree or a Token. The code can handle other trees and leaves though.
+      def build(node)
+        blk = transforms[node.name.to_sym]
+        if blk
+          Transformer.new(node, self).instance_exec(&blk)
+        elsif node.respond_to?(:children)
+          node.children.flat_map(&method(:build))
+        else
+          node
+        end
+      end
+
+      def transforms = @transforms ||= {}
+    end
+
+  end
+end

data/lib/grammy/combinator/primitives.rb

@@ -0,0 +1,41 @@
+require "grammy/matcher/string"
+require "grammy/matcher/regexp"
+require "grammy/matcher/sequence"
+require "grammy/matcher/alternative"
+require "grammy/matcher/repetition"
+
+require "grammy/matcher/eol"
+require "grammy/matcher/sol"
+require "grammy/matcher/eof"
+require "grammy/matcher/sof"
+
+
+module Grammy
+  module Combinator
+    module Primitives
+
+      protected def str(pattern) = Grammy::Matcher::String.new(pattern)
+      protected def reg(pattern) = Grammy::Matcher::Regexp.new(pattern)
+      protected def seq(*matchers) = Grammy::Matcher::Sequence.new(*matchers)
+      protected def alt(*matchers) = Grammy::Matcher::Alternative.new(*matchers)
+      protected def rep(matcher, count_range) = Grammy::Matcher::Repetition.new(matcher, count_range)
+      alias lit str
+
+      protected def eol = Grammy::Matcher::EOL.new
+      protected def sol = Grammy::Matcher::SOL.new
+      protected def eof = Grammy::Matcher::EOF.new
+      protected def sof = Grammy::Matcher::SOF.new
+
+      protected def wsp = reg(/\p{Space}+/u) # If `\p` is not available, try `[[:space:]]+` or `\s+`.
+
+      # Use these aliases if you have naming conflicts with your grammar.
+      alias _seq seq
+      alias _alt alt
+      alias _rep rep
+      alias _str str
+      alias _reg reg
+      alias _lit str
+
+    end
+  end
+end

data/lib/grammy/errors.rb

@@ -0,0 +1,3 @@
+module Grammy
+  class ParseError < StandardError; end
+end

data/lib/grammy/grammar.rb

@@ -0,0 +1,66 @@
+require "grammy/combinator/primitives"
+require "grammy/errors"
+require "grammy/scanner"
+require "grammy/parse_tree"
+
+
+module Grammy
+  class Grammar
+
+    include Grammy::Combinator::Primitives
+
+    class << self
+      # DSL for defining grammar rules.
+      def start(rule_name) = @start_rule = rule_name
+      def rule(name, &)
+        rule_proc = lambda { |_|
+          results = instance_eval(&)
+          children = Array(results).flatten.map { |result|
+            result.is_a?(Grammy::Matcher) ? result.match(@scanner) : result
+          }
+          Grammy::ParseTree.new(name.to_s, children.flatten)
+        }
+        define_method(name, &rule_proc)
+        rules[name] = rule_proc
+      end
+
+      # Examples:
+      #   terminal(:number, /\d+/)
+      #   terminal(:number) { /\d+/ }
+      #   terminal(:open_paren, "(")
+      #   terminal(:open_paren) { "(" }
+      def terminal(name, pattern = nil, &block)
+        fail ArgumentError, "may only supply a pattern OR a block to #{__callee__}" if pattern && block
+        pattern ||= yield if block
+        if pattern.is_a?(Regexp)
+          terminal_proc = -> { Grammy::Matcher::Regexp.new(pattern) }
+        else
+          terminal_proc = -> { Grammy::Matcher::String.new(pattern) }
+        end
+        define_method(name, &terminal_proc)
+        rules[name] = terminal_proc
+      end
+      alias token terminal
+
+      # Access to the rules.
+      def start_rule = @start_rule || @rules.first || :start
+      def rules = @rules ||= {}
+
+      # Parse an input using the grammar.
+      def parse(input, start: start_rule)
+        scanner = Grammy::Scanner.new(input)
+        grammar = self.new(scanner)
+        result = grammar.execute_rule(start)
+        fail(Grammy::ParseError, "Parsing failed at location #{scanner.location}") if result.nil? || result.empty? && !scanner.input.empty?
+        result
+      end
+    end
+
+    # Primitive combinators will need access to the scanner.
+    def initialize(scanner) = @scanner = scanner
+
+    def execute_rule(rule_name) = instance_eval(&rules[rule_name])
+    def rules = self.class.rules
+
+  end
+end

data/lib/grammy/location.rb

@@ -0,0 +1,27 @@
+module Grammy
+  # row/column are 1-based.
+  # index is 0-based, with respect to the full input string.
+  class Location < Data.define(:row, :column, :index)
+
+    def self.new(row = 1, column = 1, index = 0)
+      super
+    end
+
+    def advance(text)
+      newline_count = text.count("\n")
+      new_index = index + text.length
+      new_row = row + newline_count
+      if newline_count.zero?
+        new_column = column + text.length
+      else
+        new_column = text.length - text.rindex("\n")
+      end
+      Location.new(new_row, new_column, new_index)
+    end
+
+    def to_s = "(#{row},#{column})"
+    def inspect = to_s
+    def pretty_print(pp) = pp.text inspect # For IRB output.
+
+  end
+end

data/lib/grammy/match.rb

@@ -0,0 +1,19 @@
+require "grammy/location"
+
+
+# A Match represents a successful match of a pattern in the input string.
+# It contains the matched text and its start and end locations.
+module Grammy
+  # NOTE: start/end are a "closed interval", the locations of the first and last character of the match.
+  class Match < Data.define(:text, :start_location, :end_location)
+
+    def self.new(text, start_location = Grammy::Location.new, end_location = Grammy::Location.new)
+      super
+    end
+
+    def to_s = text
+    def inspect = "#<Match #{text} (#{start_location}..#{end_location})>"
+    def pretty_print(pp) = pp.text inspect # For IRB output.
+
+  end
+end

data/lib/grammy/matcher/alternative.rb

@@ -0,0 +1,22 @@
+require "grammy/matcher"
+
+
+module Grammy
+  class Matcher
+    class Alternative < Matcher
+
+      def initialize(*alternatives) = @alternatives = alternatives
+
+      def match(scanner)
+        @alternatives.each do |matcher|
+          mark = scanner.mark
+          result = matcher.match(scanner)
+          return result if result
+          scanner.backtrack(mark)
+        end
+        nil
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/eof.rb

@@ -0,0 +1,17 @@
+require "grammy/matcher"
+require "grammy/match"
+
+
+module Grammy
+  class Matcher
+    class EOF < Matcher
+
+      def initialize = super(nil)
+
+      def match(scanner)
+        scanner.finished? ? Match.new(nil, scanner.location, scanner.location) : nil
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/eol.rb

@@ -0,0 +1,17 @@
+require "grammy/matcher"
+require "grammy/match"
+
+
+module Grammy
+  class Matcher
+    class EOL < Matcher
+
+      def initialize = super(nil)
+
+      def match(scanner)
+        scanner.match_regexp(/\r?\n/)
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/regexp.rb

@@ -0,0 +1,16 @@
+require "grammy/matcher"
+
+
+module Grammy
+  class Matcher
+    class Regexp < Matcher
+
+      def initialize(pattern) = @pattern = pattern
+
+      def match(scanner)
+        scanner.match_regexp(@pattern)
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/repetition.rb

@@ -0,0 +1,22 @@
+require "grammy/matcher"
+
+module Grammy
+  class Matcher
+    class Repetition < Matcher
+
+      def initialize(submatcher, count_range)
+        @submatcher = submatcher
+        @count_range = count_range
+      end
+
+      def match(scanner)
+        all_matches = (1..@count_range.end).lazy.map { @submatcher.match(scanner) }
+        successful_matches = all_matches.take_while { |match| !match.nil? }.to_a
+        successful_matches.tap do |results|
+          return nil unless @count_range.include?(results.size)
+        end
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/sequence.rb

@@ -0,0 +1,16 @@
+require "grammy/matcher"
+
+
+module Grammy
+  class Matcher
+    class Sequence < Matcher
+
+      def initialize(*submatchers) = @submatchers = submatchers
+
+      def match(scanner)
+        @submatchers.map { |matcher| matcher.match(scanner) || (return nil) }
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/sof.rb

@@ -0,0 +1,17 @@
+require "grammy/matcher"
+require "grammy/match"
+
+
+module Grammy
+  class Matcher
+    class SOF < Matcher
+
+      def initialize = super(nil)
+
+      def match(scanner)
+        scanner.location.index.zero? ? Match.new(nil, scanner.location, scanner.location) : nil
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/sol.rb

@@ -0,0 +1,17 @@
+require "grammy/matcher"
+require "grammy/match"
+
+
+module Grammy
+  class Matcher
+    class SOL < Matcher
+
+      def initialize = super(nil)
+
+      def match(scanner)
+        scanner.location.column == 1 ? Match.new(nil, scanner.location, scanner.location) : nil
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher/string.rb

@@ -0,0 +1,16 @@
+require "grammy/matcher"
+
+
+module Grammy
+  class Matcher
+    class String < Matcher
+
+      def initialize(string) = @string = string
+
+      def match(scanner)
+        scanner.match_string(@string)
+      end
+
+    end
+  end
+end

data/lib/grammy/matcher.rb

@@ -0,0 +1,17 @@
+require "grammy/match"
+
+
+module Grammy
+  class Matcher
+
+    def initialize(pattern) = @pattern = pattern
+
+    def match(scanner) = fail NotImplementedError, "abstract method -- must override in derived classes"
+
+    # DSL for sequence, alternative, and repetition.
+    def +(other) = Matcher::Sequence.new(self, other)
+    def |(other) = Matcher::Alternative.new(self, other)
+    def [](range) = Matcher::Repetition.new(self, range)
+
+  end
+end

data/lib/grammy/parse_tree.rb

@@ -0,0 +1,11 @@
+require "grammy/tree"
+
+
+module Grammy
+  class ParseTree < Tree
+    # Turns out that there's not much special about a ParseTree.
+
+    def tokens = leaves.flatten.reject { it.text.nil? }
+
+  end
+end

data/lib/grammy/scanner.rb

@@ -0,0 +1,75 @@
+require "grammy/location"
+require "grammy/match"
+
+
+# PEG parsers don't need a lexer, so this is basically half a lexer.
+# It needs to implement backtracking for the PEG parser.
+module Grammy
+
+  class Scanner
+
+    attr_reader :location, :input
+
+    def initialize(input)
+      @input = input.is_a?(String) ? input : input.read
+      @location = Location.new(1, 1, 0)
+      @marks = [] # stack of Locations
+    end
+
+    # Try to match given String at current location.
+    # Returns `nil` if pattern does not match.
+    # Otherwise, updates @location and returns a Match object.
+    def match_string(pattern)
+      return nil if @location.index >= @input.size
+      return nil unless remaining_input.start_with?(pattern)
+      match_text(pattern)
+    end
+
+    # Try to match given Regexp at current location.
+    # Returns `nil` if pattern does not match.
+    # Otherwise, updates @location and returns a Match object.
+    def match_regexp(pattern)
+      return nil if @location.index >= @input.size
+      anchored_regex = Regexp.new("\\A(?:#{pattern.source})", pattern.options)
+      match = remaining_input.match(anchored_regex)
+      return nil unless match
+      match_text(match[0])
+    end
+
+    def mark
+      @marks.push(@location)
+      @location
+    end
+
+    def backtrack(mark)
+      fail ArgumentError, "can only backtrack the top mark" unless @marks.last == mark
+      @location = mark
+      @marks.pop
+    end
+
+    # Permanently consume input, confirming we'll never backtrack to mark again.
+    def consume(mark)
+      fail ArgumentError, "can only consume the top mark" unless @marks.last == mark
+      @marks.pop
+    end
+
+    def finished?
+      @location.index == @input.size
+    end
+
+    private def remaining_input
+      @input[@location.index..]
+    end
+
+    # Returns matched text and its location (or nil).
+    # Moves the current location forward to the character *after* the match.
+    private def match_text(text)
+      return nil unless text && !text.empty?
+      start_pos = @location
+      end_pos = start_pos.advance(text[0...-1])
+      @location = end_pos.advance(text[-1])
+      Match.new(text, start_pos, end_pos)
+    end
+
+  end
+end

data/lib/grammy/token.rb

@@ -0,0 +1,28 @@
+require "grammy/match"
+
+
+# A Token is a Match, plus the name of the token rule it matched.
+# It can also have a value, providing a more useful representation of the matched text.
+
+module Grammy
+  class Token < Data.define(:name, :match, :value)
+
+    def self.new(name, match = nil, value = nil)
+      match = Grammy::Match.new(match) unless match.is_a?(Grammy::Match)
+      super
+    end
+
+    # TODO: Extract this into a Data helper. Better yet, create a Value class for value objects.
+    def with(value:) = self.class.new(name, match, value)
+
+    def text = match.text
+    def start_location = match.start_location
+    def end_location = match.end_location
+
+    def to_s = text
+    def inspect = "#<Token #{name}: \"#{text}\"#{inspect_value}>"
+    def inspect_value = value ? " (#{value})" : ""
+    def pretty_print(pp) = pp.text inspect # For IRB output.
+
+  end
+end

data/lib/grammy/tree/transformation.rb

@@ -0,0 +1,31 @@
+module Grammy
+  class Tree
+    module Transformation
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      module ClassMethods
+        def transform_rules
+          @transform_rules ||= {}
+        end
+
+        def transform(name, &block)
+          transform_rules[name.to_sym] = block
+        end
+      end
+
+      def transform(node)
+        rule = self.class.transform_rules[node.name.to_sym]
+        if rule
+          instance_exec(node, &rule)
+        elsif node.respond_to?(:children)
+          children = node.children.map { |child| transform(child) }
+          node.class.new(node.name, children)
+        else
+          node
+        end
+      end
+    end
+  end
+end

data/lib/grammy/tree.rb

@@ -0,0 +1,28 @@
+module Grammy
+  class Tree < Data.define(:name, :children)
+    INDENTATION = 4
+
+    # Make it easier to create nested trees.
+    def self.new(name, children = [], &block)
+      children = yield if block
+      super(name:, children: Array(children))
+    end
+
+    def empty? = children.flatten.compact.empty?
+    def leaves = children.flat_map { it.is_a?(self.class) ? it.leaves : it }.compact
+
+    def to_s(level = 0) = ([to_s_base(level)] + children.map{ to_s_child(it, level) }).join("\n")
+
+    def inspect(level = 0) = ([inspect_base(level)] + children.map{ to_s_child(it, level) }).join("\n")
+    def to_h = {name:, children: children.map(&:to_h)}
+    def pretty_print(pp) = pp.text inspect # For IRB output.
+
+    private def to_s_base(level) = "#{indent(level)}#{name}"
+    private def inspect_base(level) = "#{indent(level)}#<#{class_name} #{name.inspect}>"
+    private def to_s_child(child, level) = child.is_a?(self.class) ? child.to_s(level + 1) : to_s_leaf(child, level + 1)
+    private def to_s_leaf(leaf, level) = "#{indent(level)}#{leaf}"
+    private def indent(level) = " " * (level * INDENTATION)
+    private def class_name = self.class.name.split("::").last
+
+  end
+end

data/lib/grammy.rb

@@ -0,0 +1,9 @@
+require "grammy/scanner"
+require "grammy/parse_tree"
+require "grammy/grammar"
+
+module Grammy
+
+  VERSION = File.read(File.join("..", "VERSION"))
+
+end

metadata

@@ -0,0 +1,74 @@
+--- !ruby/object:Gem::Specification
+name: grammy
+version: !ruby/object:Gem::Version
+  version: '0.10'
+platform: ruby
+authors:
+- Craig Buchek
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: Grammy is a parser with a simple DSL to define the grammar and tree transformations.
+email:
+- craig@boochtek.com
+executables: []
+extensions: []
+extra_rdoc_files:
+- README.md
+- docs/CHANGELOG.md
+- docs/LICENSE.md
+- docs/TODO.md
+files:
+- README.md
+- docs/CHANGELOG.md
+- docs/LICENSE.md
+- docs/TODO.md
+- lib/extensions/debugging.rb
+- lib/grammy.rb
+- lib/grammy/ast.rb
+- lib/grammy/ast/transformer.rb
+- lib/grammy/combinator/primitives.rb
+- lib/grammy/errors.rb
+- lib/grammy/grammar.rb
+- lib/grammy/location.rb
+- lib/grammy/match.rb
+- lib/grammy/matcher.rb
+- lib/grammy/matcher/alternative.rb
+- lib/grammy/matcher/eof.rb
+- lib/grammy/matcher/eol.rb
+- lib/grammy/matcher/regexp.rb
+- lib/grammy/matcher/repetition.rb
+- lib/grammy/matcher/sequence.rb
+- lib/grammy/matcher/sof.rb
+- lib/grammy/matcher/sol.rb
+- lib/grammy/matcher/string.rb
+- lib/grammy/parse_tree.rb
+- lib/grammy/scanner.rb
+- lib/grammy/token.rb
+- lib/grammy/tree.rb
+- lib/grammy/tree/transformation.rb
+homepage: https://github.com/stone-lang/grammy
+licenses:
+- MIT
+metadata:
+  changelog_uri: https://github.com/stone-lang/grammy/blob/main/docs/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.4'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.7
+specification_version: 4
+summary: A PEG parsing library with a simple DSL
+test_files: []