gullah 0.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6c5345d281d3e785973c68f4c0bb05466f5563c45bce2d2e71871b0fcc5d3c4e
+  data.tar.gz: 7640864e5cdc6c5b788798e2364f8d72e6ff43b2eb30d57a6c7e32a549161e29
+SHA512:
+  metadata.gz: 7e5eed3d8ea6bfcc00f5594b636fc74e98c14b1efc3af447c956a869c1137e00ea9e7f2d0804cbffd1c3923e92c58cc319c5e4b6934af7a44d558e461ebef20a
+  data.tar.gz: f47ca345b05f9ab747b1bd2cff6d85be1773979d98bb76e9d5d0ebcb3ada5e599d2cf3c1d8cf4006ab55fc29a6399ba600ec78949b11b5dd34d5890bee5363d1

data/.gitignore

@@ -0,0 +1 @@
+doc/*

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 David Fairchild Houghton
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,87 @@
+# gullah
+
+A simple, fault-tolerant bottom-up parser written in Ruby.
+
+# Synopsis
+
+```ruby
+  class Cat
+    extend Gullah
+
+    rule :S, 'NP VP'
+    rule :NP, 'D NB'
+    rule :NB, 'A* N'
+    rule :VP, 'VP PP'
+    rule :VP, 'V'
+    rule :PP, 'P NP'
+    rule :P, 'prepositions'
+    rule :V, 'verbs'
+    rule :D, 'determiners'
+    rule :N, 'nouns'
+    rule :A, 'adjectives'
+
+    leaf :determiners, /\b(the|a)\b/i
+    leaf :nouns, /\b(cat|mat)\b/i
+    leaf :prepositions, /\b(on|in|around|above|beside)\b/i
+    leaf :verbs, /\b(sat|slept|moped)\b/
+    leaf :adjectives, /\b(big|small|hairy|bald)\b/i
+
+    ignore :whatever, /[^\w\s]+/
+  end
+
+  def test_cat
+    parses = Cat.parse 'The fat cat sat on the mat.'
+    assert_equal 1, parses.length, 'there is only one parse of this sentence'
+    parse = parses.first
+    assert_equal 1, parse.nodes.reject(&:ignorable?).count, 'there is a root node for this parse'
+    root = parse.nodes.first
+    assert_equal :S, root.name, 'the root node is a sentence'
+    verb = root.descendants.find { |d| d.name == :VP }&.descendants&.find { |d| d.name == :V }
+    assert_equal 'sat', verb&.text, 'we have the expected verb'
+  end
+```
+
+# What is this?
+
+A parser takes a string representing some structured data -- a sentence in a natural language, say, or a data structure, or a program in some programming language -- and a set of rules defining the possible structures in this data and it returns an object representing the structured data.
+
+A top-down parser requires some root rule that all data structures obeying these rules will obey. A bottom-up parser says for a given piece of data what rules it may participate in. A top-down parser in effect compiles into a state machine similar to a regular expression that represents all ways a string may obey its rules. It in effect constructs a parsing plan and tries to match this plan to the string. A bottom-up parser begins planning when it sees the data. It looks at the first thing it is given to match and creates a plan for matching it and whatever may follow.
+
+The important difference is that a top-down parser must have a single root element. A bottom-up parser takes what is given and reduces it to a set of root symbols. It need not have a common symbol that must be at the root of all parses.
+
+# Why?
+
+I made Gullah because it seemed like a fun project. I have written several parsing-related things
+for several languages. I have written a [top-down parser](https://github.com/dfhoughton/Grammar) and a [non-recursive top-down parser](https://github.com/dfhoughton/pidgin). I thought I'd try a bottom-up parser. I have no particular use for it, but I often find I want to parse things, so maybe a use will show up.
+
+# Should I use this?
+
+Well, it's pretty easy to use, but if you have a bespoke parser for a particular unambiguous language, that will almost certainly be much faster. An XML parser can parse XML in linear time. Because Gullah is looking for errors and ambiguity it will consider lots of alternative deadend permutations that a SAX parser, say, will skip. Don't write a new JSON parser in Gullah, in other words. But if you want to play with natural language, or you have some toy language or small spec you're working with, Gullah can get you going quickly. Maybe it will suffice for all your needs!
+
+Gullah will give you its best parses of your string even if it is ungrammatical. Also, Gullah makes it easy to add arbitrary conditions on rules that another parser might not. For instance, in Gullah you can specify arbitrary-width lookarounds for a rule -- `foo` must be preceded/followed by `bar` and some number of whitespaces -- and you can define other long-distance dependences -- "runs" must have a singular subject, "viejas" must be modifying a feminine plural noun. For more on this see the documentation of node tests, ancestor tests, and preconditions in the `Gullah` module.
+
+# Name
+
+[Gullah](https://en.wikipedia.org/wiki/Gullah_language) is a [creole](https://en.wikipedia.org/wiki/Gullah_language)
+spoken on the barrier islands off the coast of the Carolinas and Georgia. I wanted to call this gem "creole" because I've
+written a more impoverished parser called [pidgin](https://github.com/dfhoughton/pidgin). A
+[pidgin](https://en.wikipedia.org/wiki/Pidgin) is a somewhat impoverished language created as a minimal medium
+of communication between two groups without a common language. A creole is a complete language created from a pidgin
+when children adopt it as their primary language. The pidgin library I wrote is minimal in that it cannot handle
+recursive structures. I wanted to create a better parser that could handle all the complexity of natural (or artificial)
+languages. Since this was an evolution from pidgin, I wanted to call it creole.
+
+Well, "creole" was taken. So I chose among the names of creoles of I knew of. Gullah is a creole of English and various
+Central and West African languages. I thought the name "Gullah" was cool and I like the way Gullah sounds, so I picked "Gullah".
+
+I hope this causes no offense to speakers of Gullah.
+
+# Future
+
+Because Gullah is designed to handle ambiguous grammars and erroneous data, it can produce many parses for a given string. Right now you can ask for all parses (maybe very slow), or at least `n` equally good parses. This makes the API a little complicated and noisy. I have begun work rewriting it to return a lazy enumeration of parses. Ruby's `Enumerable` magic is nice. I haven't finished this, though, and I might not, but it may be that some future version of this will change the API dramatically.
+
+Another possibility, if I have sufficient spare time and am sufficiently ambitious, is that I may refactor the algorithm to use ractors. Certain parts of the algorithm are a natural fit for parallelization. We shall see.
+
+# Acknowledgements
+
+I would like to thank my family and co-workers for tolerating me saying "Gullah" much more often than any of them expected.

data/Rakefile

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.test_files = FileList['test/*_test.rb']
+end
+
+task default: :test

data/TODO.md

@@ -0,0 +1,2 @@
+- iterator interface
+

data/examples/hat.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+require 'gullah'
+
+# :stopdoc:
+
+class Englishish
+  extend Gullah
+
+  rule :NP,         'NP PP'
+  rule :NP,         'D N'
+  rule :NP,         'Proper'
+  rule :D,          'the | Possessive'
+  rule :PP,         'prep NP'
+  rule :Possessive, 'NP pe'
+
+  leaf :the,    /\bthe\b/i
+  leaf :pe,     /(?<=[a-rt-z])'s|(?<=s)'/i
+  leaf :Proper, /\bE(?i)ngland\b/
+  leaf :N,      /\b(?:queen|hat)\b/i
+  leaf :prep,   /\bof\b/i
+end
+
+Englishish.parse("the queen of England's hat").each_with_index do |parse, i|
+  puts parse.summary
+  Gullah::Dotifier.dot parse, "hat#{i}", make_it: :so
+end

data/examples/trash.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+# :stopdoc:
+
+require 'gullah'
+
+# some random rules to experiment with and where
+# it's easy to make trash -- !@#$%@#$ is trash, for instance
+class Sanitation
+  extend Gullah
+
+  rule :ping, 'foo bar | bar foo'
+  rule :pang, 'bar baz+ | plugh'
+  rule :pong, 'foo{2,} | bar{2,} | baz{2,} | plugh{2,}'
+  rule :peng, 'ping | pang | pong'
+  rule :pung, 'peng+'
+
+  leaf :foo,   /\b\w\b/
+  leaf :bar,   /\b\w{2}\b/
+  leaf :baz,   /\b\w{3}\b/
+  leaf :plugh, /\b\w{4,}\b/
+
+  boundary :stop, /[.!?;:]/
+end
+
+text = <<-PROFUNDITY
+  A Riddle (somewhat German)
+
+  The beginning of Eternity.
+  The end of Time and Space.
+  The beginning of every End.
+  The end of every Place.
+
+  There once was a girl who had a little curl
+    Right in the middle of her forehead.
+  And when she was good, she was very, very good
+    But when she was bad she was horrid!
+PROFUNDITY
+
+poetry = Sanitation.first text
+puts poetry.summary
+Gullah::Dotifier.dot poetry, 'poem', make_it: :so

data/examples/xml.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require 'gullah'
+
+# :stopdoc:
+
+class XMLish
+  extend Gullah
+
+  rule :root,      'element'
+  rule :element,   'full | empty'
+  rule :full,      '"<" tag attribute* ">" content* "</" tag ">"', preconditions: %i[same_tag]
+  rule :empty,     '"<" tag attribute* "/>"'
+  rule :content,   'element | text | entity', tests: %i[has_parent]
+  rule :attribute, 'tag "=" value'
+  rule :value,     'squote | dquote'
+
+  leaf :tag,    /\b[a-z]+\b/
+  leaf :text,   /[^&<>]+/
+  leaf :entity, /&(?:[lg]t|amp|[lr]dquo);/
+  leaf :squote, /'[^']*'/
+  leaf :dquote, /"[^"]*"/
+
+  def same_tag(_name, _s, _e, _text, children)
+    first, last = children.select { |c| c.name == :tag }
+    first.text == last.text
+  end
+
+  def has_parent(_root, _node)
+    :pass
+  end
+end
+
+[
+  '<root/>',
+  '<root>some text &ldquo;thing&rdquo; and more text</root>',
+  '<foo>I have</foo><bar>no root</bar>',
+  '<foo attibutes="!"><and><nested/></and></foo>'
+  # this next one is r-e-a-l-l-y s-l-o-o-o-o-o-w
+  # '<big i="have" some="attributes"><empty/>text<element also="attributes">with<things/>inside</element></big>'
+].each_with_index do |xml, i|
+  parse = XMLish.parse(xml).min_by { |p| [p.length, p.size] }
+  puts parse.summary
+  Gullah::Dotifier.dot parse, "xml#{i}", make_it: :so
+end

data/gullah.gemspec

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+lib = File.expand_path('lib', __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'gullah/version'
+
+Gem::Specification.new do |s|
+  s.name        = 'gullah'
+  s.version     = Gullah::VERSION
+  s.summary     = 'A bottom up parser generator'
+  s.description = <<-DESC.strip.gsub(/\s+/, ' ')
+    Gullah is a bottom-up parser generator than can
+    handle errors, ambiguous syntax, and arbitrary matching
+    conditions.
+  DESC
+  s.authors     = ['David F. Houghton']
+  s.email       = 'dfhoughton@gmail.com'
+  s.homepage    =
+    'https://rubygems.org/gems/gullah'
+  s.license = 'MIT'
+  s.required_ruby_version = '>= 2.6'
+  s.files                 = `git ls-files -z`.split("\x0")
+  s.test_files            = s.files.grep(%r{^(test|spec|features)/})
+  s.require_paths         = ['lib']
+
+  s.add_development_dependency 'bundler', '~> 1.7'
+  s.add_development_dependency 'byebug', '~> 9.1.0'
+  s.add_development_dependency 'json', '~> 2'
+  s.add_development_dependency 'minitest', '~> 5'
+  s.add_development_dependency 'rake', '~> 10.0'
+end

data/lib/gullah/atom.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+module Gullah
+  # a minimal rule fragment; this is where the actual matching occurs
+  class Atom # :nodoc:
+    attr_reader :seeking, :min_repeats, :max_repeats, :parent, :next, :literal
+
+    def initialize(atom, parent)
+      @parent = parent
+      rule, suffix =
+        /\A
+          (
+            (?:[a-zA-Z_]|\\.)(?:\w|\\.)* # decent identifier, maybe with escaped bits
+            |
+            "(?:[^"\\]|\\.)+"        # double-quoted string, maybe with escaped characters
+            |
+            '(?:[^'\\]|\\.)+''       # single-quoted string, maybe with escaped characters
+          )
+          ([?*+!]|\{\d+(?:,\d*)?\})? # optional repetition suffix
+        \z/x
+        .match(atom)&.captures
+      raise Error, "cannot parse #{atom}" unless rule
+
+      @literal = rule[0] =~ /['"]/
+      @seeking = clean(rule).to_sym
+
+      if suffix
+        case suffix[0]
+        when '?'
+          @min_repeats = 0
+          @max_repeats = 1
+        when '+'
+          @min_repeats = 1
+          @max_repeats = Float::INFINITY
+        when '*'
+          @min_repeats = 0
+          @max_repeats = Float::INFINITY
+        else
+          min, comma, max = /(\d+)(?:(,)(\d+)?)?/.match(suffix).captures
+          min = min.to_i
+          @min_repeats = min
+          if comma
+            if max
+              max = max.to_i
+              raise Error, "cannot parse #{atom}: #{min} is greater than #{max}" if max < min
+
+              @max_repeats = max
+            else
+              @max_repeats = Float::INFINITY
+            end
+          else
+            @max_repeats = min
+          end
+        end
+      else
+        @min_repeats = @max_repeats = 1
+      end
+    end
+
+    # whether this atom must match at least once
+    def required?
+      min_repeats.positive?
+    end
+
+    # returns the new offset, or nil if the atom doesn't match
+    def match(nodes, offset)
+      if offset >= nodes.length
+        return min_repeats.zero? ? offset : nil
+      end
+
+      count = 0
+      nodes[offset...nodes.length].each_with_index do |n, i|
+        next if n.ignorable?
+
+        return returnable(nodes, i + offset + 1) if count == max_repeats
+
+        if n.traversible? && n.name == seeking
+          count += 1
+          return returnable(nodes, i + offset + 1) if count == max_repeats
+
+          next
+        end
+
+        return count >= min_repeats ? returnable(nodes, i + offset) : nil
+      end
+      count < min_repeats ? nil : returnable(nodes, nodes.length) # all nodes were consumed
+    end
+
+    # used to order rules so greedier ones go first
+    def max_consumption
+      @max_consumption ||= begin
+        augment = max_repeats == Float::INFINITY ? 10 : max_repeats
+        self.next&.max_consumption.to_i + augment
+      end
+    end
+
+    ## ADVISORILY PRIVATE
+
+    def _next=(nxt)
+      @next = nxt
+    end
+
+    private
+
+    def returnable(nodes, offset)
+      if self.next
+        self.next.match(nodes, offset)
+      else
+        offset
+      end
+    end
+
+    # remove quotes and escapes
+    def clean(str)
+      str = str[1...(str.length - 1)] if literal
+      escaped = false
+      cleaned = ''
+      (0...str.length).each do |i|
+        c = str[i]
+        if escaped
+          cleaned += c
+          escaped = false
+        elsif c == '\\'
+          escaped = true
+        else
+          cleaned += c
+        end
+      end
+      cleaned
+    end
+  end
+end

data/lib/gullah/boundary.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Gullah
+  # a node just for trash
+  class Boundary < Node # :nodoc:
+    # is this node something that cannot be the child of another node?
+    def boundary?
+      true
+    end
+  end
+end

data/lib/gullah/dotifier.rb

@@ -0,0 +1,127 @@
+# frozen_string_literal: true
+
+module Gullah
+  # A little tool to help visualize a parse tree. It generates .dot files
+  # parsable by graphviz. If you have graphviz installed, you may be able
+  # to invoke it like so and generate a .png file
+  #
+  #   Gullah::Dotifier.dot parses.first, "tree", make_it: :so
+  #
+  # This will generate a file called tree.png showing the parse tree. If you
+  # don't have graphviz, or perhaps if you're on a machine which doesn't like
+  # the command this generates -- I suspect Windows doesn't -- you can skip
+  # the named argument and just generate the dot file which you can feed into
+  # graphviz some other way.
+  #
+  # I make no guarantees about this utility. You may want to build your own,
+  # in which case this can serve as a simple prototype.
+  class Dotifier
+    ##
+    # Receives a parse and a file name and generates a graph specification
+    # readable by graphviz. The specification is written to a file with the
+    # specified file name. If +make_it+ is truthy, the +dot+ command will
+    # also be invoked and the graph image generated. By default this will be
+    # a png, though the type is specifiable via the +type+ named argument.
+    def self.dot(parse, file, make_it: false, type: 'png')
+      new.send :dot, parse, file, make_it, type
+    end
+
+    # making the guts private to simplify the API
+
+    private
+
+    def dot(parse, file, make_it, type)
+      @edges = {}
+      File.open file, 'w' do |f|
+        f.puts 'graph {'
+        f.puts "\tnode[shape=none]"
+        f.puts
+        parse.roots.each do |root|
+          tree(root, f)
+        end
+        # put all the leaves in a row at the bottom
+        f.puts
+        f.puts "\tsubgraph {"
+        f.puts "\t\trank=\"same\""
+        parse.roots.flat_map(&:leaves).reject(&:ignorable?).each do |leaf|
+          f.puts "\t\t#{leaf_name(leaf)}"
+        end
+        f.puts "\t}"
+        f.puts '}'
+      end
+      `dot -T#{type} -o#{file}.#{type} #{file}` if make_it
+    end
+
+    def tree(node, f)
+      return if node.ignorable?
+
+      nn = name(node)
+      f.puts "\t#{nn} #{node_attributes(node)}"
+      if node.leaf?
+        ln = leaf_name(node)
+        f.puts "\t#{ln} [label=#{node.text.inspect}]"
+        f.puts "\t#{nn} -- #{ln}"
+      end
+      Array(node.atts[:satisfied_ancestor]).each do |_, loc, *|
+        child = node.find loc
+        add_edge node, child, :success, true
+      end
+      Array(node.atts[:failed_ancestor]).each do |_, loc, *|
+        child = node.find loc
+        add_edge node, child, :error, true
+      end
+      Array(node.children&.reject(&:ignorable?)).each do |child|
+        f.puts "\t#{nn} -- #{name(child)}#{edge_attributes node, child}"
+        tree(child, f)
+      end
+    end
+
+    def add_edge(parent, child, property, value)
+      while parent != child
+        middle = parent.children.find { |c| c.contains? child.start }
+        (@edges[[parent.position, middle.position]] ||= {})[property] = value
+        parent = middle
+      end
+    end
+
+    def edge_attributes(node, child)
+      atts = []
+      if (properties = @edges[[node.position, child.position]])
+        if properties[:error]
+          atts << 'color=red'
+        elsif properties[:success]
+          atts << 'color=green'
+        end
+      end
+      " [#{atts.join(';')}]" if atts.any?
+    end
+
+    def node_attributes(node)
+      atts = ["label=#{node.trash? ? 'trash' : node.name.to_s.inspect}"]
+      if node.trash?
+        atts << 'color=red'
+        atts << 'shape=box'
+      elsif node.boundary?
+        atts << 'color=blue'
+        atts << 'shape=box'
+      elsif node.error?
+        atts << 'color=red'
+        atts << 'shape=oval'
+      elsif node.atts[:satisfied_ancestor] || node.atts[:satisfied_descendant]
+        atts << 'color=green'
+        atts << 'shape=oval'
+      end
+      "[#{atts.join(';')}]"
+    end
+
+    def name(node)
+      offset, height = node.position
+      "n_#{offset}_#{height}"
+    end
+
+    def leaf_name(node)
+      offset, height = node.position
+      "l_#{offset}_#{height}"
+    end
+  end
+end