rley 0.5.14 → 0.6.00

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 331a505963faf201ef03664616805ce831a591e0
-  data.tar.gz: 1e80b03ca806612f72fa97db13dd49b1cc55e218
+  metadata.gz: a7fa26ec5c47c28dd6d0cc68f32e8f023f92ce75
+  data.tar.gz: 73717a9a1c2daa9886952c502020b2fcf5e4bb33
 SHA512:
-  metadata.gz: e070d60d75abd4beace97c65d938a7bc4e81e1c9c4bc0c4d3f0b1bba4acd13662e3196f83113bfd5def31992d2df85367dc6a3fb6ab367480e6e1d51d9d6d98d
-  data.tar.gz: 3816e3675513be5608d731fba3b37e9c08b41ced276403182790dcb8d38235cd88c7908b8a3540bf2c5afc87fec84e3340a1b61f9caf55fa095e9b4f6092774d
+  metadata.gz: 055963ec1637d43b33703e87b454fd9a1590a3993609f2df89f0d3ca94a1b8eba7b39cf102d318e0c1da6fc3e29c51a84bd3929f3000b7c800ee4585260d9b78
+  data.tar.gz: 2b0bc71c159ecac03789644497af1ce789b7975efe2fb6fecc399e41fb346aec26ad67a9d1015ebed1d43a9e9fdd09321d7dfef5f86afa3ce5402b4f04b8f062

data/CHANGELOG.md

@@ -1,4 +1,14 @@
-### 0.5.14 / 2018-XX-XX
+### 0.6.00 / 2018-02-25
+Version bump. Highlights: new programming interface through facade object, improved AST generation. 
+* [NEW] Class `Rley::Engine`: Implementation of Facade design pattern to reach more convenient interface.
+* [NEW] Class `Rley::ParseRep::ASTBaseBuilder` Abstract class that simplifies the creation of custom AST (Abstract Syntax Tree)
+* [NEW] Module `Rley::ParseRep` hosts the classes for building parse representations (parse trees and forests)
+* [CHANGE] File `README.md` updated to reflect the new facade interface.
+* [CHANGE] Almost all the examples have been updated to use the `Rley::Engine` facade and the new AST building.
+* [CHANGE] All the specs have been updated to use the `Rley::Engine` facade and the new AST building.
+* [DEPRECATED] Method `Rley::Parser::GFGParsing#parse_tree`.
+
+### 0.5.14 / 2018-02-10
 * [NEW] Files spec/rley/sppf/token_node_spec.rb` Added RSpec file for testing  `SPPF::TokenNode` class.
 * [CHANGE] Files `lib/rley/sppf` Minor update in the YARD documentation of the SPPF node classes.
 * [FIX] Method `Parser::CSTRawNode#initialize`. Yard warning because of duplicate parameter names in documentation.
@@ -9,7 +19,7 @@
 * [CHANGE] File `examples/general/SRL/ast_builder.rb Code refactoring to take profit of rule naming.
 
 ### 0.5.12 / 2018-02-03
-* [CHANGE] Simple Regex Language is fully supported!...
+* [CHANGE] Simple Regex Language is almost fully supported!...
 * [CHANGE] File `examples/general/SRL/grammar.rb added missing rule productions for Simple Regex Language.
 * [CHANGE] File `examples/general/SRL/ast_builder.rb Added transformation rules missing regular expressions features.
 * [CHANGE] File `examples/general/SRL/spac/integration_spec.rb Added tests for covering SRL.

data/README.md

@@ -74,10 +74,10 @@ Installing the latest stable version is simple:
 The purpose of this section is show how to create a parser for a minimalistic
 English language subset.
 The tour is organized as follows:  
-1. [Defining the language grammar](#defining-the-language-grammar)  
-2. [Creating a lexicon](#creating-a-lexicon)  
-3. [Creating a tokenizer](#creating-a-tokenizer)  
-4. [Building the parser](#building-the-parser)  
+1. [Creating facade object of Rley library](#creating-facade-object-of-rley-library)   
+2. [Defining the language grammar](#defining-the-language-grammar)  
+3. [Creating a lexicon](#creating-a-lexicon)  
+4. [Creating a tokenizer](#creating-a-tokenizer)    
 5. [Parsing some input](#parsing-some-input)  
 6. [Generating the parse tree](#generating-the-parse-tree)
 
@@ -85,14 +85,22 @@ The complete source code of the example used in this tour can be found in the
 [examples](https://github.com/famished-tiger/Rley/tree/master/examples/NLP/mini_en_demo.rb)
 directory
 
-### Defining the language grammar
-The subset of English grammar is based on an example from the NLTK book.
 
+### Creating facade object of Rley library
 ```ruby  
     require 'rley' # Load Rley library
 
-    # Instantiate a builder object that will build the grammar for us
-    builder = Rley::Syntax::GrammarBuilder.new do
+    # Let's create a facade object called 'engine'
+    # It provides a unified, higher-level interface
+    engine = Rley.Engine.new
+```
+
+
+### Defining the language grammar
+The subset of English grammar is based on an example from the NLTK book.
+
+```ruby  
+    engine.build_grammar do
       # Terminal symbols (= word categories in lexicon)
       add_terminals('Noun', 'Proper-Noun', 'Verb')
       add_terminals('Determiner', 'Preposition')
@@ -106,8 +114,6 @@ The subset of English grammar is based on an example from the NLTK book.
       rule 'VP' => %w[Verb NP PP]
       rule 'PP' => %w[Preposition NP]
     end
-    # And now, let's build the grammar...
-    grammar = builder.grammar
 ```  
 
 ### Creating a lexicon
@@ -141,14 +147,14 @@ The subset of English grammar is based on an example from the NLTK book.
 
 ### Creating a tokenizer
 ```ruby
-    # A tokenizer reads the input string and converts it into a sequence of tokens
-    # Highly simplified tokenizer implementation.
-    def tokenizer(aTextToParse, aGrammar)
+    # A tokenizer reads the input string and converts it into a sequence of tokens.
+    # Remark: Rley doesn't provide tokenizer functionality.
+    # Highly simplified tokenizer implementation
+    def tokenizer(aTextToParse)
       tokens = aTextToParse.scan(/\S+/).map do |word|
         term_name = Lexicon[word]
         raise StandardError, "Word '#{word}' not found in lexicon" if term_name.nil?
-        terminal = aGrammar.name2symbol[term_name]
-        Rley::Lexical::Token.new(word, terminal)
+        Rley::Lexical::Token.new(word, term_name)
       end
 
       return tokens
@@ -161,20 +167,12 @@ creating a lexicon and tokenizer from scratch. Here are a few Ruby Part-of-Speec
 * [rbtagger](https://rubygems.org/gems/rbtagger)
 
 
-
-### Building the parser
-```ruby
-  # Easy with Rley...
-  parser = Rley::Parser::GFGEarleyParser.new(grammar)
-```
-
-
 ### Parsing some input
 ```ruby
     input_to_parse = 'John saw Mary with a telescope'
     # Convert input text into a sequence of token objects...
-    tokens = tokenizer(input_to_parse, grammar)
-    result = parser.parse(tokens)
+    tokens = tokenizer(input_to_parse)
+    result = engine.parse(tokens)
 
     puts "Parsing successful? #{result.success?}" # => Parsing successful? true
 ```
@@ -194,7 +192,7 @@ For our whirlwind tour, we will opt for parse trees.
 ### Generating the parse tree
 
 ```ruby
-    ptree = result.parse_tree
+    ptree = engine.convert(result)
 ```  
 OK. Now that we have the parse tree, what we can do with it?
 One option is to manipulate the parse tree and its node directly. For instance,
@@ -216,7 +214,7 @@ an one-liner:
 
 ```ruby
     # Let's create a parse tree visitor
-    visitor = Rley::ParseTreeVisitor.new(ptree)
+    visitor = engine.ptree_visitor(ptree)
 ```
 
 #### Visiting the parse tree
@@ -359,8 +357,8 @@ above and, as an error, we delete the verb `saw` in the sentence to parse.
     # Verb has been removed from the sentence on next line
     input_to_parse = 'John Mary with a telescope'
     # Convert input text into a sequence of token objects...
-    tokens = tokenizer(input_to_parse, grammar)
-    result = parser.parse(tokens)
+    tokens = tokenizer(input_to_parse)
+    result = engine.parse(tokens)
 
     puts "Parsing successful? #{result.success?}" # => Parsing successful? false
     exit(1)
@@ -390,8 +388,8 @@ Let's experiment again with the original sentence but without the word
     # Last word has been removed from the sentence on next line
     input_to_parse = 'John saw Mary with a '
     # Convert input text into a sequence of token objects...
-    tokens = tokenizer(input_to_parse, grammar)
-    result = parser.parse(tokens)
+    tokens = tokenizer(input_to_parse)
+    result = engine.parse(tokens)
 
     puts "Parsing successful? #{result.success?}" # => Parsing successful? false
     unless result.success?

data/examples/NLP/benchmark_pico_en.rb

@@ -1,12 +1,24 @@
-# File: benchmark_mini_en.rb
+# File: benchmark_pico_en.rb
 # Purpose: benchmark the parse speed
 require 'benchmark'
 require 'rley' # Load Rley library
 
-# Instantiate a builder object that will build the grammar for us
-builder = Rley::Syntax::GrammarBuilder.new do
+########################################
+# Step 0. Instantiate facade object of Rley library.
+# It provides a unified, higher-level interface 
+engine = Rley::Engine.new
 
-  add_terminals('Noun', 'Proper-Noun', 'Verb') 
+########################################
+# Step 1. Define a grammar for a pico English-like language
+# based on example from NLTK book (chapter 8 of the book).
+# Bird, Steven, Edward Loper and Ewan Klein: "Natural Language Processing
+# with Python"; 2009, O’Reilly Media Inc., ISBN 978-0596516499
+# It defines the syntax of a sentence in a mini English-like language
+# with a very simplified syntax and vocabulary
+engine.build_grammar do
+  # Next 2 lines we define the terminal symbols
+  # (= word categories in the lexicon)
+  add_terminals('Noun', 'Proper-Noun', 'Verb')
   add_terminals('Determiner', 'Preposition')
 
   # Here we define the productions (= grammar rules)
@@ -17,10 +29,7 @@ builder = Rley::Syntax::GrammarBuilder.new do
   rule 'VP' => %w[Verb NP]
   rule 'VP' => %w[Verb NP PP]
   rule 'PP' => %w[Preposition NP]
-end 
-
-# And now, let's build the grammar...
-grammar = builder.grammar
+end
 
 ########################################
 # Step 2. Creating a lexicon
@@ -31,7 +40,7 @@ Lexicon = {
   'dog' => 'Noun',
   'cat' => 'Noun',
   'telescope' => 'Noun',
-  'park' => 'Noun',  
+  'park' => 'Noun',
   'saw' => 'Verb',
   'ate' => 'Verb',
   'walked' => 'Verb',
@@ -49,44 +58,35 @@ Lexicon = {
 }.freeze
 
 ########################################
-# Step 3. Creating a tokenizer
-# A tokenizer reads the input string and converts it into a sequence of tokens
-# Highly simplified tokenizer implementation.
-def tokenizer(aTextToParse, aGrammar)
+# Step 3. Create a tokenizer
+# A tokenizer reads the input string and converts it into a sequence of tokens.
+# Rley doesn't provide tokenizer functionality.
+# (Highly simplified tokenizer implementation).
+def tokenizer(aTextToParse)
   tokens = aTextToParse.scan(/\S+/).map do |word|
     term_name = Lexicon[word]
     raise StandardError, "Word '#{word}' not found in lexicon" if term_name.nil?
-    terminal = aGrammar.name2symbol[term_name]
-    Rley::Lexical::Token.new(word, terminal)
+    Rley::Lexical::Token.new(word, term_name)
   end
-  
+
   return tokens
 end
 
-########################################
-# Step 4. Create a parser for that grammar
-# Easy with Rley...
-parser = Rley::Parser::GFGEarleyParser.new(grammar)
 
 ########################################
-# Step 5. Parsing the input
+# Step 4. Parse the input
 input_to_parse = 'John saw Mary with a telescope'
-
+# input_to_parse = 'the dog saw a man in the park' # This one is ambiguous
 # Convert input text into a sequence of token objects...
-tokens = tokenizer(input_to_parse, grammar)
+tokens = tokenizer(input_to_parse)
+result = engine.parse(tokens)
 
 # Use Benchmark mix-in
 include Benchmark
 
 bm(6) do |meter|
-  meter.report("Parse 100 times") { 100.times { parser.parse(tokens) } }
-  meter.report("Parse 1000 times") { 1000.times { parser.parse(tokens) } }
-  meter.report("Parse 10000 times") { 10000.times { parser.parse(tokens) } }
-  meter.report("Parse 1000000 times") { 100000.times { parser.parse(tokens) } }
-end
-
-# puts "Parsing successful? #{result.success?}"
-# unless result.success?
-  # puts result.failure_reason.message
-  # exit(1)
-# end
+  meter.report("Parse 100 times") { 100.times { engine.parse(tokens) } }
+  meter.report("Parse 1000 times") { 1000.times { engine.parse(tokens) } }
+  meter.report("Parse 10000 times") { 10000.times { engine.parse(tokens) } }
+  meter.report("Parse 1000000 times") { 100000.times { engine.parse(tokens) } }
+end

data/examples/NLP/engtagger.rb

@@ -153,7 +153,7 @@ def tokenizer(lexicon, grammar, tokens)
   lexicon.each_with_index do |word, i| 
     term_name = tokens[i].last
     terminal = grammar.name2symbol[term_name]
-    rley_tokens << Rley::Tokens::Token.new(word, terminal)
+    rley_tokens << Rley::Lexical::Token.new(word, terminal)
   end
   return rley_tokens
 end

data/examples/NLP/nano_eng/nano_en_demo.rb

@@ -1,19 +1,22 @@
 require 'rley' # Load Rley library
 
+########################################
+# Step 0. Instantiate facade object of Rley library.
+# It provides a unified, higher-level interface
+engine = Rley::Engine.new
+
 ########################################
 # Step 1. Define a grammar for a nano English-like language
 # based on example from Jurafski & Martin book (chapter 8 of the book).
 # Bird, Steven, Edward Loper and Ewan Klein: "Speech and Language Processing";
 # 2009, Pearson Education, Inc., ISBN 978-0135041963
-# It defines the syntax of a sentence in a mini English-like language 
+# It defines the syntax of a sentence in a mini English-like language
 # with a very simplified syntax and vocabulary
-
-# Instantiate a builder object that will build the grammar for us
-builder = Rley::Syntax::GrammarBuilder.new do
-  # Next 2 lines we define the terminal symbols 
+engine.build_grammar do
+  # Next 2 lines we define the terminal symbols
   # (= word categories in the lexicon)
-  add_terminals('Noun', 'Proper-Noun', 'Pronoun', 'Verb') 
-  add_terminals('Aux', 'Det', 'Preposition')
+  add_terminals('Noun', 'Proper-Noun', 'Pronoun', 'Verb')
+  add_terminals('Aux', 'Determiner', 'Preposition')
 
   # Here we define the productions (= grammar rules)
   rule 'Start' => 'S'
@@ -22,7 +25,7 @@ builder = Rley::Syntax::GrammarBuilder.new do
   rule 'S' => 'VP'
   rule 'NP' => 'Pronoun'
   rule 'NP' => 'Proper-Noun'
-  rule 'NP' => %w[Det Nominal]
+  rule 'NP' => %w[Determiner Nominal]
   rule 'Nominal' => %[Noun]
   rule 'Nominal' => %[Nominal Noun]
   rule 'VP' => 'Verb'
@@ -31,10 +34,7 @@ builder = Rley::Syntax::GrammarBuilder.new do
   rule 'VP' => %w[Verb PP]
   rule 'VP' => %w[VP PP]
   rule 'PP' => %w[Preposition NP]
-end 
-
-# And now, let's build the grammar...
-grammar = builder.grammar
+end
 
 ########################################
 # Step 2. Creating a lexicon
@@ -45,7 +45,7 @@ Lexicon = {
   'dog' => 'Noun',
   'cat' => 'Noun',
   'telescope' => 'Noun',
-  'park' => 'Noun',  
+  'park' => 'Noun',
   'saw' => 'Verb',
   'ate' => 'Verb',
   'walked' => 'Verb',
@@ -66,29 +66,24 @@ Lexicon = {
 # Step 3. Creating a tokenizer
 # A tokenizer reads the input string and converts it into a sequence of tokens
 # Highly simplified tokenizer implementation.
-def tokenizer(aTextToParse, aGrammar)
+def tokenizer(aTextToParse)
   tokens = aTextToParse.scan(/\S+/).map do |word|
     term_name = Lexicon[word]
     raise StandardError, "Word '#{word}' not found in lexicon" if term_name.nil?
-    terminal = aGrammar.name2symbol[term_name]
-    Rley::Lexical::Token.new(word, terminal)
+    Rley::Lexical::Token.new(word, term_name)
   end
-  
+
   return tokens
 end
 
-########################################
-# Step 4. Create a parser for that grammar
-# Easy with Rley...
-parser = Rley::Parser::GFGEarleyParser.new(grammar)
-
 ########################################
 # Step 5. Parsing the input
-input_to_parse = 'John saw Mary with a telescope'
+input_to_parse = 'John saw Mary'
+# input_to_parse = 'John saw Mary with a telescope'
 # input_to_parse = 'the dog saw a man in the park' # This one is ambiguous
 # Convert input text into a sequence of token objects...
-tokens = tokenizer(input_to_parse, grammar)
-result = parser.parse(tokens)
+tokens = tokenizer(input_to_parse)
+result = engine.parse(tokens)
 
 puts "Parsing successful? #{result.success?}"
 unless result.success?
@@ -98,10 +93,10 @@ end
 
 ########################################
 # Step 6. Generating a parse tree from parse result
-ptree = result.parse_tree
+ptree = engine.convert(result)
 
 # Let's create a parse tree visitor
-visitor = Rley::ParseTreeVisitor.new(ptree)
+visitor = engine.ptree_visitor(ptree)
 
 # Let's create a formatter (i.e. visit event listener)
 # renderer = Rley::Formatter::Debug.new($stdout)
@@ -109,7 +104,7 @@ visitor = Rley::ParseTreeVisitor.new(ptree)
 # Let's create a formatter that will render the parse tree with characters
 renderer = Rley::Formatter::Asciitree.new($stdout)
 
-# Let's create a formatter that will render the parse tree in labelled 
+# Let's create a formatter that will render the parse tree in labelled
 # bracket notation
 # renderer = Rley::Formatter::BracketNotation.new($stdout)
 

data/examples/NLP/nano_eng/nano_grammar.rb

@@ -28,7 +28,7 @@ builder = Rley::Syntax::GrammarBuilder.new do
   rule 'yes_no_question' => %w[Aux NP VP]
   rule 'wh_subject_question' => %w[Wh_NP NP VP]
   rule 'wh_non_subject_question' => %w[Wh_NP Aux NP VP]
-  rule 'NP' => %[Predeterminer NP]
+  rule 'NP' => %w[Predeterminer NP]
   rule 'NP' => 'Pronoun'
   rule 'NP' => 'Proper-Noun'
   rule 'NP' => %w[Det Card Ord Quant Nominal] 

data/examples/NLP/pico_en_demo.rb

@@ -1,18 +1,22 @@
 require 'rley' # Load Rley library
 
+
+########################################
+# Step 1. Creating facade object of Rley library
+# It provides a unified, higher-level interface
+engine = Rley::Engine.new
+
 ########################################
-# Step 1. Define a grammar for a pico English-like language
+# Step 2. Define a grammar for a pico English-like language
 # based on example from NLTK book (chapter 8 of the book).
-# Bird, Steven, Edward Loper and Ewan Klein: "Natural Language Processing 
+# Bird, Steven, Edward Loper and Ewan Klein: "Natural Language Processing
 # with Python"; 2009, O’Reilly Media Inc., ISBN 978-0596516499
-# It defines the syntax of a sentence in a mini English-like language 
+# It defines the syntax of a sentence in a mini English-like language
 # with a very simplified syntax and vocabulary
-
-# Instantiate a builder object that will build the grammar for us
-builder = Rley::Syntax::GrammarBuilder.new do
-  # Next 2 lines we define the terminal symbols 
+engine.build_grammar do
+  # Next 2 lines we define the terminal symbols
   # (= word categories in the lexicon)
-  add_terminals('Noun', 'Proper-Noun', 'Verb') 
+  add_terminals('Noun', 'Proper-Noun', 'Verb')
   add_terminals('Determiner', 'Preposition')
 
   # Here we define the productions (= grammar rules)
@@ -23,13 +27,10 @@ builder = Rley::Syntax::GrammarBuilder.new do
   rule 'VP' => %w[Verb NP]
   rule 'VP' => %w[Verb NP PP]
   rule 'PP' => %w[Preposition NP]
-end 
-
-# And now, let's build the grammar...
-grammar = builder.grammar
+end
 
 ########################################
-# Step 2. Creating a lexicon
+# Step 3. Creating a lexicon
 # To simplify things, lexicon is implemented as a Hash with pairs of the form:
 # word => terminal symbol name
 Lexicon = {
@@ -37,7 +38,7 @@ Lexicon = {
   'dog' => 'Noun',
   'cat' => 'Noun',
   'telescope' => 'Noun',
-  'park' => 'Noun',  
+  'park' => 'Noun',
   'saw' => 'Verb',
   'ate' => 'Verb',
   'walked' => 'Verb',
@@ -55,32 +56,28 @@ Lexicon = {
 }.freeze
 
 ########################################
-# Step 3. Creating a tokenizer
-# A tokenizer reads the input string and converts it into a sequence of tokens
-# Highly simplified tokenizer implementation.
-def tokenizer(aTextToParse, aGrammar)
+# Step 4. Create a tokenizer
+# A tokenizer reads the input string and converts it into a sequence of tokens.
+# Remark: Rley doesn't provide tokenizer functionality.
+# Highly simplified tokenizer implementation
+def tokenizer(aTextToParse)
   tokens = aTextToParse.scan(/\S+/).map do |word|
     term_name = Lexicon[word]
     raise StandardError, "Word '#{word}' not found in lexicon" if term_name.nil?
-    terminal = aGrammar.name2symbol[term_name]
-    Rley::Lexical::Token.new(word, terminal)
+    Rley::Lexical::Token.new(word, term_name)
   end
-  
+
   return tokens
 end
 
-########################################
-# Step 4. Create a parser for that grammar
-# Easy with Rley...
-parser = Rley::Parser::GFGEarleyParser.new(grammar)
 
 ########################################
-# Step 5. Parsing the input
+# Step 5. Parse the input
 input_to_parse = 'John saw Mary with a telescope'
 # input_to_parse = 'the dog saw a man in the park' # This one is ambiguous
 # Convert input text into a sequence of token objects...
-tokens = tokenizer(input_to_parse, grammar)
-result = parser.parse(tokens)
+tokens = tokenizer(input_to_parse)
+result = engine.parse(tokens)
 
 puts "Parsing successful? #{result.success?}"
 unless result.success?
@@ -90,10 +87,10 @@ end
 
 ########################################
 # Step 6. Generating a parse tree from parse result
-ptree = result.parse_tree
+ptree = engine.to_ptree(result)
 
 # Let's create a parse tree visitor
-visitor = Rley::ParseTreeVisitor.new(ptree)
+visitor = engine.ptree_visitor(ptree)
 
 # Let's create a formatter (i.e. visit event listener)
 # renderer = Rley::Formatter::Debug.new($stdout)
@@ -101,7 +98,7 @@ visitor = Rley::ParseTreeVisitor.new(ptree)
 # Let's create a formatter that will render the parse tree with characters
 renderer = Rley::Formatter::Asciitree.new($stdout)
 
-# Let's create a formatter that will render the parse tree in labelled 
+# Let's create a formatter that will render the parse tree in labelled
 # bracket notation
 # renderer = Rley::Formatter::BracketNotation.new($stdout)