rley 0.4.04 → 0.4.05

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5100723b47c2572c2ac0b5996413eb4cd8e462ba
-  data.tar.gz: 4d37be9ca7128a36f77505560c7214454e413eb5
+  metadata.gz: 89fbee176719b82e8e1b1676ceb89b837e9ad249
+  data.tar.gz: ff72f2224e98479729b441aa0a38b0da61fd34e4
 SHA512:
-  metadata.gz: c197b1b962408e3a3ce4be5d310c227bdc638028d9a40d26b89c544232f59a8ce994228e4ae3673cbe6147dbcd520ee53c36a0620e3c7871edd3aba46762933f
-  data.tar.gz: 21b59e0b9b5c942be83954b35ee15650e1bc2c2aa2f9a6f1ebba645b94e3f822df510026825354512cf7eea879431feed852f34cef9fd176c5d0b89377504a46
+  metadata.gz: a8eed1075570fac6f89f6ed16164c52026426dd0c497030fd31bbfd861e7873744be3acc14afa046da40f67ddf9d4987301dac76227ac4d0f504e40f46caa862
+  data.tar.gz: 684e37a2a236bc9e34e4bacba4775b61608be5dab58b3e4af414da222cf0c79f3db98631b904969c23b3ed882368ea553c7c25f9099cd09f17d5d5105111ba21

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+### 0.4.05 / 2017-05-06
+* [CHANGE] File `README.md` Added documentation on how to build parse trees and manipulate them.
+* [CHANGE] File `examples\NLP\mini_en_demo.rb` now emits different parse tree representations.
+* [NEW] Directory `www`. Contains a diagram output produced from Rley and fed to online RSyntaxTree tool.
+
+
+
 ### 0.4.04 / 2017-05-01
 * [NEW] `Asciitree` formatter class. Allows parse tree output in simple printable text.
 * [CHANGE] Major enhancements in directory `examples\data_formats\JSON`. The demo command-line tool parses JSON and outputs the parse tree in one of the supported formats.

data/README.md

@@ -1,5 +1,5 @@
 [Rley](https://github.com/famished-tiger/Rley)
-
+====
 [![Linux Build Status](https://img.shields.io/travis/famished-tiger/Rley/master.svg?label=Linux%20build)](https://travis-ci.org/famished-tiger/Rley)
 [![Build status](https://ci.appveyor.com/api/projects/status/l5adgcbfo128rvo9?svg=true)](https://ci.appveyor.com/project/famished-tiger/rley)
 [![Coverage Status](https://img.shields.io/coveralls/famished-tiger/Rley.svg)](https://coveralls.io/r/famished-tiger/Rley?branch=master)
@@ -10,9 +10,6 @@
 
 A Ruby library for constructing general parsers for _any_ context-free language.  
 
-====
-
-
 What is Rley?
 -------------
 __Rley__ uses the [Earley](http://en.wikipedia.org/wiki/Earley_parser)
@@ -28,7 +25,7 @@ encode efficiently all the possible parse trees that result from an ambiguous
 grammar.  
 
 As another distinctive mark, __Rley__ is also the first Ruby implementation of a
-parsing library based on the new _Grammar Flow Graph_ approach [References on GFG](#references-on-gfg).
+parsing library based on the new [Grammar Flow Graph](#grammar-flow-graph) approach .
 
 ### What it can do?
 Maybe parsing algorithms and internal implementation details are of lesser
@@ -81,7 +78,7 @@ The tour is organized as follows:
 3. [Creating a tokenizer](#creating-a-tokenizer)  
 4. [Building the parser](#building-the-parser)  
 5. [Parsing some input](#parsing-some-input)  
-6. [Generating the parse forest](#generating-the-parse-forest)
+6. [Generating the parse tree](#generating-the-parse-tree)
 
 The complete source code of the example used in this tour can be found in the
 [examples](https://github.com/famished-tiger/Rley/tree/master/examples/NLP/mini_en_demo.rb)
@@ -183,11 +180,173 @@ creating a lexicon and tokenizer from scratch. Here are a few Ruby Part-of-Speec
     puts "Parsing successful? #{result.success?}" # => Parsing successful? true
 ```
 
-## Generating the parse forest
+At this stage, we're done with parsing. What we need next are convenient means
+to exploit the parse result. As it is, the `result` variable in the last code snippet
+above is a data structure ("Earley item sets") that is tightly related to the intricate details
+of the Earley's parsing algorithm. Obviously, it contains all the necessary data to exploit
+the parsing results but it is too low-level and inconvenient from a programming viewpoint.
+Therefore, __Rley__ provides out of the box two convenient data structures for
+representing the parse outcome:
+- Parse tree (optimal when the parse is unambiguous)   
+- Parse forest (a more sophisticated data structure that copes with ambiguity)
+
+For our whirlwind tour, we will opt for parse trees.
+
+## Generating the parse tree
+
+```ruby
+    ptree = result.parse_tree
+```  
+OK. Now that we have the parse tree, what we can do with it?
+One option is to manipulate the parse tree and its node directly. For instance,
+one could write code to customize and transform the parse tree. This approach gives
+most the of flexibility needed for advanced applications. The other, more common
+option is to rely on the parse tree visitor instantiated from the `Rley::ParseTreeVisitor` class.
+Such a visitor walks over the parse tree nodes and generates visit events that
+are dispatched to subscribed event listeners. All this may, at first, sound
+complicate but the coming code snippets will show it otherwise.
+
+Let's do it by:  
+- Creating a parse tree visitor  
+- Using one of the built-in visit subscribers specifically created to render the
+ parse tree in a given output format.  
+
+### Creating a parse tree visitor  
+Good news: creating a parse tree visitor for the parse tree `ptree` is just
+an one-liner:
+
+```ruby
+    # Let's create a parse tree visitor
+    visitor = Rley::ParseTreeVisitor.new(ptree)
+```
+
+### Visiting the parse tree
+
+Unsurprisingly, to start the parse tree visit, one calls the `#start` method:
+
+```ruby
+    visitor.start
+```
+
+If you try the above line, no particular result will be visible and for a good reason:
+no object was specified as a visit event subscriber. As a convenience, __Rley__
+bundles a number of [formatter classes](https://github.com/famished-tiger/Rley/tree/master/lib/rley/formatter)
+that were designed to listen to the visit event and then render the parse tree
+in a specific format. To begin with, we'll use the simple formatter
+`Rley::Formatter::Debug` class. Its purpose is just to print out the visit event
+name.
+
+Remove the line with the call to the `#start` method and replace it with the two
+statements:
+```ruby
+    # Let's create a formatter (i.e. visit event listener)
+    renderer = Rley::Formatter::Debug.new($stdout)
+
+    # Subscribe the formatter to the visitor's event and launch the visit
+    renderer.render(visitor)    
+```
+
+These two lines will generate the following output:
+```
+before_ptree
+  before_non_terminal
+    before_subnodes
+      before_non_terminal
+        before_subnodes
+          before_terminal
+          after_terminal
+        after_subnodes
+      after_non_terminal
+      before_non_terminal
+        before_subnodes
+          before_terminal
+          after_terminal
+          before_non_terminal
+            before_subnodes
+              before_terminal
+              after_terminal
+            after_subnodes
+          after_non_terminal
+          before_non_terminal
+            before_subnodes
+              before_terminal
+              after_terminal
+              before_non_terminal
+                before_subnodes
+                  before_terminal
+                  after_terminal
+                  before_terminal
+                  after_terminal
+                after_subnodes
+              after_non_terminal
+            after_subnodes
+          after_non_terminal
+        after_subnodes
+      after_non_terminal
+    after_subnodes
+  after_non_terminal
+after_ptree
+```
+
+At least is something visible: these are the parse tree visit events.
+Note that the indentation of event names depends on the nesting level of
+the tree node being visited.
+
+Not really impressive? So let's use another formatter...
+
+### Visualizing the parse tree structure
+If one replaces the previous formatter by an instance of
+`Rley::Formatter::Asciitree` the output now shows the parse tree structure.
+
 ```ruby
-    pforest = result.parse_forest
+    # Let's create a formatter that will render the parse tree with characters
+    renderer = Rley::Formatter::Asciitree.new($stdout)
+
+    # Subscribe the formatter to the visitor's event and launch the visit
+    renderer.render(visitor)   
+```
+
+The outputs looks like this:
+```
+S
++-- NP
+|   +-- Proper-Noun: 'John'
++-- VP
+    +-- Verb: 'saw'
+    +-- NP
+    |   +-- Proper-Noun: 'Mary'
+    +-- PP
+        +-- Preposition: 'with'
+        +-- NP
+            +-- Determiner: 'a'
+            +-- Noun: 'telescope'
 ```
 
+If you are more inclined for graphical representation, then replace the last formatter
+by yet another one:
+
+```ruby
+    # Let's create a formatter that will render the parse tree in labelled bracket notation
+    renderer = Rley::Formatter::BracketNotation .new($stdout)
+
+    # Subscribe the formatter to the visitor's event and launch the visit
+    renderer.render(visitor)   
+```
+
+This results in the strange-looking output:
+```
+[S [NP [Proper-Noun John]][VP [Verb saw][NP [Proper-Noun Mary]][PP [Preposition with][NP [Determiner a][Noun telescope]]]]]
+```
+
+This output is in a format that is recognized by many NLP software.
+The next diagram was created by copy-pasting the output above in the online tool
+[RSyntaxTree](http://yohasebe.com/rsyntaxtree/).
+By the way, this tool is also a Ruby gem, [rsyntaxtree](https://rubygems.org/gems/rsyntaxtree).
+
+![Sample parse tree diagram](www/sample_parse_tree.png)
+
+
+
 ## Error reporting
 __Rley__ is a non-violent parser, that is, it won't throw an exception when it
 detects a syntax error. Instead, the parse result will be marked as
@@ -277,12 +436,15 @@ Here are a few other ones:
   conforms to the grammar rules or not. As such it cannot build parse trees.  
   The code doesn't seem to be maintained: latest commit dates from Oct. 2011.
 
+## Other interesting Ruby resources
+The extensive resource list not to miss: [Awesome NLP with Ruby](https://github.com/arbox/nlp-with-ruby)
+actively curated by Andrei Beliankou (aka arbox).
 
 ##  Thanks to:
 * Professor Keshav Pingali, one of the creators of the Grammar Flow Graph parsing approach for his encouraging e-mail exchanges.
 
-## References on GFG
-Since the __G__rammar __F__low __G__raph parsing approach is quite new, it has not yet taken a place in
+## Grammar Flow Graph
+Since the Grammar Flow Graph parsing approach is quite new, it has not yet taken a place in
 standard parser textbooks. Here are a few references (and links) of papers on GFG:    
 - K. Pingali, G. Bilardi. [Parsing with Pictures](http://apps.cs.utexas.edu/tech_reports/reports/tr/TR-2102.pdf)
 - K. Pingali, G. Bilardi. [A Graphical Model for Context-Free Grammar Parsing.](https://link.springer.com/chapter/10.1007/978-3-662-46663-6_1)

data/examples/NLP/mini_en_demo.rb

@@ -90,7 +90,21 @@ unless result.success?
 end
 
 ########################################
-# Step 6. Generating the parse forest
-pforest = result.parse_forest
+# Step 6. Generating a parse tree from parse result
+ptree = result.parse_tree
 
-puts "Parsing ambiguous? #{pforest.ambiguous?}" # => Parsing ambiguous? false
+# Let's create a parse tree visitor
+visitor = Rley::ParseTreeVisitor.new(ptree)
+
+# Let's create a formatter (i.e. visit event listener)
+# renderer = Rley::Formatter::Debug.new($stdout)
+
+# Let's create a formatter that will render the parse tree with characters
+renderer = Rley::Formatter::Asciitree.new($stdout)
+
+# Let's create a formatter that will render the parse tree in labelled bracket notation
+# renderer = Rley::Formatter::BracketNotation .new($stdout)
+
+# Subscribe the formatter to the visitor's event and launch the visit
+renderer.render(visitor)
+# End of file

data/examples/general/calc/calc_demo.rb

@@ -27,6 +27,13 @@ unless result.success?
   exit(1)
 end
 
-# Generate a parse forest from the parse result
-pforest = result.parse_forest
+# Generate a parse tree from the parse result
+ptree = result.parse_tree
+
+# Let's create a parse tree visitor
+visitor = Rley::ParseTreeVisitor.new(ptree)
+
+# Now output formatted parse tree
+renderer = Rley::Formatter::Asciitree.new($stdout)
+renderer.render(visitor)
 # End of file

data/lib/rley/constants.rb

@@ -3,7 +3,7 @@
 
 module Rley # Module used as a namespace
   # The version number of the gem.
-  Version = '0.4.04'.freeze
+  Version = '0.4.05'.freeze
 
   # Brief description of the gem.
   Description = "Ruby implementation of the Earley's parsing algorithm".freeze

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rley
 version: !ruby/object:Gem::Version
-  version: 0.4.04
+  version: 0.4.05
 platform: ruby
 authors:
 - Dimitri Geshef
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-05-01 00:00:00.000000000 Z
+date: 2017-05-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake