rley 0.4.05 → 0.4.06

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 89fbee176719b82e8e1b1676ceb89b837e9ad249
-  data.tar.gz: ff72f2224e98479729b441aa0a38b0da61fd34e4
+  metadata.gz: 57136a1796f3625b841a2f1fdb0965b4314b6224
+  data.tar.gz: 80fd87d6ad9f68314278d5b4c15ef85a241647c3
 SHA512:
-  metadata.gz: a8eed1075570fac6f89f6ed16164c52026426dd0c497030fd31bbfd861e7873744be3acc14afa046da40f67ddf9d4987301dac76227ac4d0f504e40f46caa862
-  data.tar.gz: 684e37a2a236bc9e34e4bacba4775b61608be5dab58b3e4af414da222cf0c79f3db98631b904969c23b3ed882368ea553c7c25f9099cd09f17d5d5105111ba21
+  metadata.gz: 3922fe824b8d721892b71022e64a79f4117ed0135f0b02fdaf76f5edfef083d345192328f3086a80fc0484e7dac14f7ec188a51c942bab2b14c8a196e4c9ffc4
+  data.tar.gz: 836625169012043daeba92c142fda1e80e6f4eaf085437c2d1bc006e57ca8de99d70619fce22f2d6d8f000468e9c93c65b8e073d80f68084619d7c2070e18197

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+### 0.4.06 / 2017-05-25
+* [FIX] File `formatter\asciitree.rb` fixed inconsistency in comments that caused Yard warnings.
+* [FIX] File `formatter\bracket_notation.rb` fixed inconsistency in comments that caused Yard warnings.
+* [FIX] File `parser\parse_entry_set.rb` fixed inconsistency in comments that caused Yard warnings.
+* [NEW] Method `Grammar#diagnose` performs a number of checks on the grammar. It detects whether:  
+  there are undefined non-terminals (i.e. non-terminals without a rule that define them)
+  there are non-productive non-terminals (i.e. non-terminals that don't derive a sting of terminals)
+  there are nullable productions and non-terminals.
+* [NEW] Method `GrmFlowGraph#traverse_df` performs depth-first traversal of the GFG.
+* [NEW] Method `GrmFlowGraph#diagnose` determines which terminals are reachable from the start symbol.
+* [NEW] Method `GrmSymbol#generative?` inidcates whether the grammar symbol can produce a sequence of terminals.
+* [CHANGE] Class `GrammarBuilder` Improved the API documentation.
+
+
 ### 0.4.05 / 2017-05-06
 * [CHANGE] File `README.md` Added documentation on how to build parse trees and manipulate them.
 * [CHANGE] File `examples\NLP\mini_en_demo.rb` now emits different parse tree representations.

data/README.md

@@ -182,9 +182,9 @@ creating a lexicon and tokenizer from scratch. Here are a few Ruby Part-of-Speec
 
 At this stage, we're done with parsing. What we need next are convenient means
 to exploit the parse result. As it is, the `result` variable in the last code snippet
-above is a data structure ("Earley item sets") that is tightly related to the intricate details
+above is a data structure ("Earley item sets") that is highly depending on the intricate details
 of the Earley's parsing algorithm. Obviously, it contains all the necessary data to exploit
-the parsing results but it is too low-level and inconvenient from a programming viewpoint.
+the parsing results but it is rather low-level and inconvenient from a programming viewpoint.
 Therefore, __Rley__ provides out of the box two convenient data structures for
 representing the parse outcome:
 - Parse tree (optimal when the parse is unambiguous)   
@@ -201,10 +201,10 @@ OK. Now that we have the parse tree, what we can do with it?
 One option is to manipulate the parse tree and its node directly. For instance,
 one could write code to customize and transform the parse tree. This approach gives
 most the of flexibility needed for advanced applications. The other, more common
-option is to rely on the parse tree visitor instantiated from the `Rley::ParseTreeVisitor` class.
+option is to use an `Rley::ParseTreeVisitor` instance.
 Such a visitor walks over the parse tree nodes and generates visit events that
 are dispatched to subscribed event listeners. All this may, at first, sound
-complicate but the coming code snippets will show it otherwise.
+complicated but the coming code snippets show it otherwise.
 
 Let's do it by:  
 - Creating a parse tree visitor  
@@ -327,7 +327,7 @@ by yet another one:
 
 ```ruby
     # Let's create a formatter that will render the parse tree in labelled bracket notation
-    renderer = Rley::Formatter::BracketNotation .new($stdout)
+    renderer = Rley::Formatter::BracketNotation.new($stdout)
 
     # Subscribe the formatter to the visitor's event and launch the visit
     renderer.render(visitor)   
@@ -338,7 +338,7 @@ This results in the strange-looking output:
 [S [NP [Proper-Noun John]][VP [Verb saw][NP [Proper-Noun Mary]][PP [Preposition with][NP [Determiner a][Noun telescope]]]]]
 ```
 
-This output is in a format that is recognized by many NLP software.
+This output is in a format that is recognized by many NLP softwares.
 The next diagram was created by copy-pasting the output above in the online tool
 [RSyntaxTree](http://yohasebe.com/rsyntaxtree/).
 By the way, this tool is also a Ruby gem, [rsyntaxtree](https://rubygems.org/gems/rsyntaxtree).

data/examples/NLP/mini_en_demo.rb

@@ -103,7 +103,7 @@ visitor = Rley::ParseTreeVisitor.new(ptree)
 renderer = Rley::Formatter::Asciitree.new($stdout)
 
 # Let's create a formatter that will render the parse tree in labelled bracket notation
-# renderer = Rley::Formatter::BracketNotation .new($stdout)
+# renderer = Rley::Formatter::BracketNotation.new($stdout)
 
 # Subscribe the formatter to the visitor's event and launch the visit
 renderer.render(visitor)

data/lib/rley/constants.rb

@@ -3,7 +3,7 @@
 
 module Rley # Module used as a namespace
   # The version number of the gem.
-  Version = '0.4.05'.freeze
+  Version = '0.4.06'.freeze
 
   # Brief description of the gem.
   Description = "Ruby implementation of the Earley's parsing algorithm".freeze

data/lib/rley/formatter/asciitree.rb

@@ -36,8 +36,8 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor is about to visit
       # the children of a non-terminal node
-      # @param _parent [NonTerminalNode]
-      # @param _children [Array] array of children nodes
+      # @param parent [NonTerminalNode]
+      # @param children [Array] array of children nodes
       def before_subnodes(parent, children)
         rank_of(parent)
         curr_path << parent
@@ -47,7 +47,7 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor is about to visit
       # a non-terminal node
-      # @param nonterm [NonTerminalNode]
+      # @param aNonTerm [NonTerminalNode]
       def before_non_terminal(aNonTerm)
         emit(aNonTerm)
       end
@@ -56,7 +56,7 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor is about to visit
       # a terminal node
-      # @param _term [TerminalNode]
+      # @param aTerm [TerminalNode]
       def before_terminal(aTerm)
         emit(aTerm, ": '#{aTerm.token.lexeme}'")
       end

data/lib/rley/formatter/bracket_notation.rb

@@ -23,7 +23,7 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor is about to visit
       # a non-terminal node
-      # @param nonterm [NonTerminalNode]
+      # @param aNonTerm [NonTerminalNode]
       def before_non_terminal(aNonTerm)
         write("[#{aNonTerm.symbol.name} ")
       end
@@ -32,7 +32,7 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor is about to visit
       # a terminal node
-      # @param _term [TerminalNode]
+      # @param aTerm [TerminalNode]
       def before_terminal(aTerm)
         write("[#{aTerm.symbol.name} ")
       end
@@ -40,7 +40,7 @@ module Rley # This module is used as a namespace
       # Method called by a ParseTreeVisitor to which the formatter subscribed.
       # Notification of a visit event: the visitor completed the visit of
       # a terminal node.
-      # @param _term [TerminalNode]
+      # @param aTerm [TerminalNode]
       def after_terminal(aTerm)
         # Escape all opening and closing square brackets
         escape_lbrackets = aTerm.token.lexeme.gsub(/\[/, "\\[")

data/lib/rley/gfg/call_edge.rb

@@ -14,7 +14,7 @@ module Rley # This module is used as a namespace
       # Pre-condition: theSuccessor is an StartVertex
       def initialize(thePredecessor, theSuccessor)
         super(thePredecessor, theSuccessor)
-        do_set_key(thePredecessor, theSuccessor)
+        do_set_key(thePredecessor, theSuccessor)      
       end
 
       private

data/lib/rley/gfg/grm_flow_graph.rb

@@ -1,3 +1,4 @@
+require 'set'
 require_relative 'start_vertex'
 require_relative 'end_vertex'
 require_relative 'item_vertex'
@@ -36,6 +37,76 @@ module Rley # This module is used as a namespace
         vertices.find { |a_vertex| a_vertex.label == aVertexLabel }
       end
 
+      # Perform a diagnosis of the grammar elements (symbols and rules)
+      # in order to detect:
+      # If one wants to remove useless rules, then do first:
+      # elimination of non-generating symbols
+      # then elimination of unreachable symbols
+      def diagnose
+        mark_unreachable_symbols
+      end
+
+      Branching = Struct.new(:vertex, :to_visit, :visited) do
+        def initialize(aVertex)
+          super(aVertex)
+          self.to_visit = aVertex.edges.dup
+          self.visited = []
+        end
+
+        def done?
+          self.to_visit.empty?
+        end
+
+        def next_edge
+          next_one = self.to_visit.shift
+          self.visited << next_one.successor unless next_one.nil?
+
+          return next_one
+        end
+      end
+
+      # Walk over all the vertices of the graph that are reachable from a given
+      # start vertex. This is a depth-first graph traversal.
+      # @param aStartVertex [StartVertex] the depth-first traversal begins from here
+      # @param visitAction [Proc] block code called when a new graph vertex is found
+      def traverse_df(aStartVertex, &visitAction)
+        visited = Set.new
+        stack = []
+        visitee = aStartVertex
+
+        begin
+          first_time = !visited.include?(visitee)
+          if first_time
+            visitAction.call(visitee)
+            visited << visitee
+          end
+
+          case visitee
+            when Rley::GFG::StartVertex
+              if first_time
+                stack.push(Branching.new(visitee))
+                curr_edge = stack.last.next_edge
+              else
+                # Skip start and end vertices
+                # Retrieve the corresponding return edge
+                curr_edge = get_matching_return(curr_edge)
+              end
+
+            when Rley::GFG::EndVertex
+              stack.pop if stack.last.done?
+              break if stack.empty?
+              curr_edge = stack.last.next_edge
+
+            else
+              # All other vertex types have only one successor
+              curr_edge = visitee.edges[0]
+          end
+          visitee = curr_edge.successor unless curr_edge.nil?
+        end until stack.empty?
+        # Now process the end vertex matching the initial start vertex
+        visitAction.call(end_vertex_for[aStartVertex.non_terminal])
+      end
+
       private
 
       def add_vertex(aVertex)
@@ -68,7 +139,15 @@ module Rley # This module is used as a namespace
       def build_all_starts_ends(theDottedItems)
         productions_raw = theDottedItems.map(&:production)
         productions = productions_raw.uniq
-        productions.each { |prod| build_start_end_for(prod.lhs) }
+        all_nterms = Set.new
+        productions.each do |prod|
+          all_nterms << prod.lhs
+          nterms_of_rhs = prod.rhs.members.select do |symb|
+            symb.kind_of?(Syntax::NonTerminal)
+          end
+          all_nterms.merge(nterms_of_rhs)
+        end
+        all_nterms.each { |nterm| build_start_end_for(nterm) }
       end
 
       # if there is not yet a start vertex labelled .N in the GFG:
@@ -179,12 +258,43 @@ module Rley # This module is used as a namespace
         # Retrieve corresponding end vertex
         end_vertex = end_vertex_for[nt_symbol]
         # Create an edge end vertex -> return vertex
-        ReturnEdge.new(end_vertex, return_vertex)
+        ReturnEdge.new(end_vertex, return_vertex) if end_vertex
       end
 
       def build_shortcut_edge(fromVertex, toVertex)
         ShortcutEdge.new(fromVertex, toVertex)
       end
+
+
+      # Retrieve the return edge that matches the given
+      # call edge.
+      def get_matching_return(aCallEdge)
+        # Calculate key of return edge from the key of call edge
+        ret_key = aCallEdge.key.sub(/CALL/, 'RET')
+
+        # Retrieve the corresponding end vertex
+        end_vertex = end_vertex_for[aCallEdge.successor.non_terminal]
+
+        # Retrieve the return edge with specified key
+        return_edge = end_vertex.edges.find { |edge| edge.key == ret_key }
+      end
+
+      # Mark non-terminal symbols that cannot be derived from the start symbol.
+      # In a GFG, a non-terminal symbol N is unreachable if there is no path
+      # from the start symbol to the start node .N
+      def mark_unreachable_symbols()
+        # Mark all non-terminals as unreachable
+        start_vertex_for.values.each do |a_vertex|
+          a_vertex.non_terminal.unreachable = true
+        end
+
+        # Now traverse graph from start vertex
+        # and make all visited non-terminals as reachable
+        traverse_df(start_vertex) do |a_vertex|
+          next unless a_vertex.kind_of?(StartVertex)
+          a_vertex.non_terminal.unreachable = false
+        end
+      end
     end # class
   end # module
 end # module

data/lib/rley/parser/parse_entry_set.rb

@@ -39,7 +39,7 @@ module Rley # This module is used as a namespace
       
       # Append the given entry (if it isn't yet in the set)
       # to the list of parse entries
-      # @param aState [ParseEntry] the parse entry to push.
+      # @param anEntry [ParseEntry] the parse entry to push.
       # @return [ParseEntry] the passed parse entry it doesn't   added
       def push_entry(anEntry)
         match = entries.find { |entry| entry == anEntry }

data/lib/rley/rley_error.rb

@@ -1,4 +1,4 @@
-# File: exceptions.rb
+# File: rley_error.rb
 
 module Rley # Module used as a namespace
   # @abstract

data/lib/rley/syntax/grammar.rb

@@ -1,15 +1,16 @@
 require 'set'
+require_relative '../rley_error'
 
 module Rley # This module is used as a namespace
   module Syntax # This module is used as a namespace
     # A grammar specifies the syntax of a language.
-    #   Formally, a grammar has: 
+    #   Formally, a grammar has:
     #   * One start symbol,
     #   * One or more other production rules,
     #   * Each production has a rhs that is a sequence of grammar symbols.
-    #   * Grammar symbols are categorized into 
-    #   -terminal symbols
-    #   -non-terminal symbols
+    #   * Grammar symbols are categorized into:
+    #     -terminal symbols
+    #     -non-terminal symbols
     class Grammar
       # A non-terminal symbol that represents all the possible strings
       # in the language.
@@ -30,15 +31,16 @@ module Rley # This module is used as a namespace
         @symbols = []
         @name2symbol = {}
         valid_productions = validate_productions(theProductions)
+        valid_productions.each { |prod| add_production(prod) }
+        diagnose
+
         # TODO: use topological sorting
         @start_symbol = valid_productions[0].lhs
-        valid_productions.each { |prod| add_production(prod) }
-        compute_nullable
       end
 
       # @return [Array] The list of non-terminals in the grammar.
       def non_terminals()
-        return symbols.select { |s| s.kind_of?(NonTerminal) }
+        @non_terminals ||= symbols.select { |s| s.kind_of?(NonTerminal) }
       end
 
       # @return [Production] The start production of the grammar (i.e.
@@ -61,10 +63,101 @@ module Rley # This module is used as a namespace
         the_lhs = aProduction.lhs
         add_symbol(the_lhs)
 
-        # TODO: remove quadratic execution time
         aProduction.rhs.each { |symb| add_symbol(symb) }
       end
 
+      # Perform some check of the grammar.
+      def diagnose()
+        mark_undefined
+        mark_generative
+        compute_nullable
+      end
+
+      # Check that each non-terminal appears at least once in lhs.
+      # If it is not the case, then mark it as undefined
+      def mark_undefined
+        defined = Set.new
+
+        # Defined non-terminals appear at least once as lhs of a production
+        rules.each { |prod| defined << prod.lhs }
+        defined.each { |n_term| n_term.undefined = false }
+
+        # Retrieve all non-terminals that aren't marked as non-undefined
+        undefined = non_terminals.select { |n_term| n_term.undefined?.nil? }
+
+        undefined.each { |n_term| n_term.undefined = true }
+      end
+
+
+
+      # Mark all non-terminals and production rules as
+      # generative or not.
+      # A production is generative when it can derive a string of terminals.
+      # A production is therefore generative when all its rhs members are
+      # themselves generatives.
+      # A non-terminal is generative if at least one of its defining production
+      # is itself generative.
+      def mark_generative
+        curr_marked = []
+
+        # Iterate until no new rule can be marked.
+        begin
+          prev_marked = curr_marked.dup
+
+          rules.each do |a_rule|
+            next unless a_rule.generative?.nil?
+            if a_rule.empty?
+              a_rule.generative = false
+              curr_marked << a_rule
+              could_mark_nterm_generative(a_rule)
+              next
+            end
+
+            last_considered = nil
+            a_rule.rhs.members.each do |symbol|
+              last_considered = symbol
+              break unless symbol.generative?
+            end
+            next if last_considered.generative?.nil?
+            a_rule.generative = last_considered.generative?
+            curr_marked << a_rule
+            could_mark_nterm_generative(a_rule)
+          end
+        end until prev_marked.size == curr_marked.size
+
+        # The nonterminals that are not marked yet are non-generative
+        non_terminals.each do |nterm|
+          nterm.generative = false if nterm.generative?.nil?
+        end
+      end
+
+      # Given a production rule with given non-terminal
+      # Check whether that non-terminal should be marked
+      # as generative or not.
+      # A non-terminal may be marked as generative if at
+      # least one of its defining production is generative.
+      def could_mark_nterm_generative(aRule)
+        nterm = aRule.lhs
+
+        # non-terminal already marked? If yes, nothing more to do...
+        return unless nterm.generative?.nil?
+
+        defining_rules = rules_for(nterm) # Retrieve all defining productions
+
+        all_false = true
+        defining_rules.each do |prod|
+          if prod.generative?
+            # One generative rule found!
+            nterm.generative = true
+            all_false = false
+            break
+          else
+            all_false = false if prod.generative?.nil?
+          end
+        end
+        nterm.generative = false if all_false
+      end
+
 
       # For each non-terminal determine whether it is nullable or not.
       # A nullable nonterminal is a nonterminal that can match an empty string.
@@ -118,6 +211,12 @@ module Rley # This module is used as a namespace
         @symbols << aSymbol
         @name2symbol[its_name] = aSymbol
       end
+
+      # Retrieve all the production rules that share the same symbol in lhs
+      def rules_for(aNonTerm)
+        rules.select { |a_rule| a_rule.lhs == aNonTerm }
+      end
+
     end # class
   end # module
 end # module

data/lib/rley/syntax/grammar_builder.rb

@@ -1,56 +1,74 @@
+require 'set'
 require_relative 'verbatim_symbol'
 require_relative 'literal'
+require_relative 'terminal'
 require_relative 'non_terminal'
 require_relative 'production'
 require_relative 'grammar'
 
 module Rley # This module is used as a namespace
   module Syntax # This module is used as a namespace
-    # Builder GoF pattern. Builder pattern builds a complex object
-    # (say, a grammar) from simpler objects (terminals and productions)
-    # and using a step by step approach.
+    # Builder GoF pattern. Builder builds a complex object
+    #   (say, a grammar) from simpler objects (terminals and productions)
+    #   and using a step by step approach.
     class GrammarBuilder
-      # The list of symbols of the language.
-      # Grammar symbols are categorized into terminal (symbol)
-      # and non-terminal (symbol).
+      # @return [Hash{String, GrmSymbol}] The mapping of grammar symbol names
+      #   to the matching grammar symbol object.
       attr_reader(:symbols)
 
-      # The list of production rules for the grammar to build
+      # @return [Array<Production>] The list of production rules for 
+      #   the grammar to build.
       attr_reader(:productions)
 
-
+      # Creates a new grammar builder.
+      # @param aBlock [Proc] code block used to build the grammar. 
+      # @example Building a tiny English grammar
+      #   builder = Rley::Syntax::GrammarBuilder.new do
+      #     add_terminals('n', 'v', 'adj', 'det')
+      #     rule 'S' => %w(NP VP)
+      #     rule 'VP' => %w(v NP)
+      #     rule 'NP' => %w(det n)
+      #     rule 'NP' => %w(adj NP)
+      #   end
+      #   tiny_eng = builder.grammar
       def initialize(&aBlock)
         @symbols = {}
         @productions = []
 
         instance_exec(&aBlock) if block_given?
       end
-      
+
       # Retrieve a grammar symbol from its name.
       # Raise an exception if not found.
-      # @param aSymbolName [String] the name of a symbol grammar.
-      # @return [GrmSymbol] the retrieved symbol.
+      # @param aSymbolName [String] the name of a grammar symbol.
+      # @return [GrmSymbol] the retrieved symbol object.
       def [](aSymbolName)
         return symbols[aSymbolName]
       end
 
       # Add the given terminal symbols to the grammar of the language
       # @param terminalSymbols [String or Terminal] 1..* terminal symbols.
+      # @return [void]
       def add_terminals(*terminalSymbols)
         new_symbs = build_symbols(Terminal, terminalSymbols)
         symbols.merge!(new_symbs)
       end
 
 
-      # Add a production rule in the grammar given one 
+      # Add a production rule in the grammar given one
       # key-value pair of the form: String => Array.
-      #   Where the key is the name of the non-terminal appearing in the 
-      #   left side of the rule. 
+      #   Where the key is the name of the non-terminal appearing in the
+      #   left side of the rule.
       #   The value, an Array, is a sequence of grammar symbol names.
       # The rule is created and inserted in the grammar.
-      # Example:
-      #   builder.add_production('A' => ['a', 'A', 'c'])      
-      # @param aProductionRepr [Hash] A Hash-based representation of production 
+      # @example Equivalent call syntaxes
+      #   builder.add_production('A' => ['a', 'A', 'c'])
+      #   builder.rule('A' => ['a', 'A', 'c']) # 'rule' is a synonym
+      #   builder.rule('A' => %w(a A  c)) # Use %w syntax for Array of String
+      #   builder.rule 'A' => %w(a A  c)  # Call parentheses are optional
+      # @param aProductionRepr [Hash{String, Array<String>}] A Hash-based representation 
+      #   of a production.
+      # @return [void]      
       def add_production(aProductionRepr)
         aProductionRepr.each_pair do |(lhs_name, rhs_repr)|
           lhs = get_nonterminal(lhs_name)
@@ -69,26 +87,37 @@ module Rley # This module is used as a namespace
 
       # Given the grammar symbols and productions added to the builder,
       # build the resulting grammar (if not yet done).
+      # @return [Grammar] the created grammar object.      
       def grammar()
         unless @grammar
           raise StandardError, 'No symbol found for grammar' if symbols.empty?
           if productions.empty?
             raise StandardError, 'No production found for grammar'
           end
-          
-          # Check that each non-terminal appears at least once in lhs.
-          all_non_terminals = symbols.values.select { |s| s.is_a?(NonTerminal) }
-          all_non_terminals.each do |n_term|
-            next if productions.any? { |prod| n_term == prod.lhs }
-            raise StandardError, "Nonterminal #{n_term.name} not rewritten"
+
+          # Check that each terminal appears at least in a rhs of a production
+          all_terminals = symbols.values.select do |a_symb|
+            a_symb.kind_of?(Terminal)
+          end
+          in_use = Set.new
+          productions.each do |prod|
+            prod.rhs.members.each do |symb|
+              in_use << symb if symb.kind_of?(Syntax::Terminal)
+            end
+          end
+
+          unused = all_terminals.reject { |a_term| in_use.include?(a_term) }
+          unless unused.empty?
+            suffix = "#{unused.map(&:name).join(', ')}."
+            raise StandardError, 'Useless terminal symbol(s): ' + suffix
           end
 
           @grammar = Grammar.new(productions.dup)
         end
-        
+
         return @grammar
       end
-      
+
       alias rule add_production
 
       private
@@ -125,7 +154,7 @@ module Rley # This module is used as a namespace
 
         return a_symbol
       end
-      
+
       # Retrieve the non-terminal symbol with given name.
       # If it doesn't exist yet, then it is created on the fly.
       # @param aSymbolName [String] the name of the grammar symbol to retrieve
@@ -136,7 +165,7 @@ module Rley # This module is used as a namespace
         end
         return symbols[aSymbolName]
       end
-      
+
     end # class
   end # module
 end # module

data/lib/rley/syntax/grm_symbol.rb

@@ -2,28 +2,37 @@ module Rley # This module is used as a namespace
   module Syntax # This module is used as a namespace
     # Abstract class for grammar symbols.
     # A grammar symbol is an element that appears in grammar rules.
-    class GrmSymbol  
+    class GrmSymbol
       # The name of the grammar symbol
       attr_reader(:name)
 
+      # An indicator that tells whether the grammar symbol can generate a
+      # non-empty string of terminals.
+      attr_writer(:generative)
+
       # Constructor.
       # aName [String] The name of the grammar symbol.
       def initialize(aName)
         @name = aName.dup
       end
-      
-      # Return true iff the symbol is a terminal
-      def terminal?()
-        # Default implementation to override if necessary
-        return false
-      end
-      
+
       # The String representation of the grammar symbol
       # @return [String]
       def to_s()
         return name.to_s
       end
-        
+
+      # @return [Boolean] true iff the symbol is a terminal
+      def terminal?()
+        # Default implementation to override if necessary
+        return false
+      end
+
+      # @return [Boolean] true iff the symbol is generative.
+      def generative?()
+        return @generative
+      end
+
     end # class
   end # module
 end # module

data/lib/rley/syntax/non_terminal.rb

@@ -5,7 +5,16 @@ module Rley # This module is used as a namespace
     # A non-terminal symbol (sometimes called a syntactic variable) represents
     # a composition of terminal or non-terminal symbols
     class NonTerminal < GrmSymbol
+      # A non-terminal symbol is nullable if it can match an empty string.
       attr_writer(:nullable)
+      
+      # A non-terminal symbol is undefined if no production rule in the grammar
+      # has that non-terminal symbol in its left-hand side.
+      attr_writer(:undefined)
+      
+      # A non-terminal symbol is unreachable if it cannot be reached (derived) 
+      # from the start symbol.
+      attr_writer(:unreachable)
 
       # Constructor.
       # @param aName [String] The name of the grammar symbol.
@@ -21,6 +30,18 @@ module Rley # This module is used as a namespace
       def nullable?()
         return @nullable
       end
+      
+      # @return [false/true] Return true if the symbol doesn't appear
+      # on the left-hand side of any production rule.
+      def undefined?()
+        return @undefined
+      end
+      
+      # @return [false/true] Return true if the symbol cannot be derived
+      # from the start symbol.
+      def unreachable?()
+        return @unreachable
+      end      
     end # class
   end # module
 end # module

data/lib/rley/syntax/production.rb

@@ -15,6 +15,10 @@ module Rley # This module is used as a namespace
 
       # The left-hand side of the rule. It must be a non-terminal symbol
       attr_reader(:lhs)
+      
+      # A production is generative when all of its rhs members are generative (that as, they
+      # can each generate/derive a non-empty string of terminals).
+      attr_writer(:generative)      
 
       # Provide common alternate names to lhs and rhs accessors
 
@@ -31,6 +35,14 @@ module Rley # This module is used as a namespace
       def empty?()
         return rhs.empty?
       end
+      
+      # Return true iff the production is generative      
+      def generative?()
+        if @generative.nil?
+        end
+        
+        return @generative
+      end      
 
       private
 

data/lib/rley/syntax/terminal.rb

@@ -10,6 +10,7 @@ module Rley # This module is used as a namespace
       # aName [String] The name of the grammar symbol.
       def initialize(aName)
         super(aName)
+        self.generative = true
       end
       
       # Return true iff the symbol is a terminal

data/spec/rley/gfg/grm_flow_graph_spec.rb

@@ -154,6 +154,102 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           end
         end
       end # context
+      
+      context 'Provided services:' do
+        let(:problematic_grammar) do
+          # Based on grammar example in book
+          # C. Fisher, R. LeBlanc, "Crafting a Compiler"; page 98
+          builder = Rley::Syntax::GrammarBuilder.new
+          builder.add_terminals('a', 'b', 'c')
+          builder.add_production('S' => 'A')
+          builder.add_production('S' => 'B')
+          builder.add_production('A' => 'a')
+          # There is no edge between .B and B => B . b => non-generative
+          builder.add_production('B' => %w(B b))
+          
+          # Non-terminal symbol C is unreachable
+          builder.add_production('C' => 'c')  
+
+          # And now build the grammar...
+          builder.grammar       
+        end
+      
+        it 'should provide depth-first traversal' do
+          result = []
+          subject.traverse_df(subject.start_vertex) do |vertex|
+            result << vertex.label
+          end
+
+          expected = [
+              '.S',
+              'S => . A',
+              '.A',
+              'A => . a A c',
+              'A => a . A c',
+              'A => a A . c',
+              'A => a A c .',
+              'A.',
+              'A => . b',
+              'A => b .',
+              'S.'
+            ]
+          expect(result).to eq(expected)
+        end
+        
+        it 'should perform a diagnosis of a correct grammar' do
+          expect { subject.diagnose }.not_to raise_error
+          grammar_abc.non_terminals.each do |nterm|
+            expect(nterm).not_to be_undefined
+            expect(nterm).not_to be_unreachable
+          end
+        end
+        
+        it 'should detect when a non-terminal is unreachable' do
+          grammar = problematic_grammar
+          items = build_items_for_grammar(grammar)
+
+          graph = GrmFlowGraph.new(items)       
+          expect { graph.diagnose }.not_to raise_error
+          grammar.non_terminals.each do |nterm|
+            expect(nterm).not_to be_undefined
+          end
+          
+          unreachable = grammar.non_terminals.select do |nterm| 
+            nterm.unreachable? 
+          end
+          expect(unreachable.size).to eq(1)
+          expect(unreachable[0].name).to eq('C')
+        end        
+      end # context
+      
+=begin
+      context 'Grammar without undefined symbols:' do
+        it 'should mark all its nonterminals as not undefined' do
+          nonterms = subject.non_terminals
+          nonterms.each do |nterm|
+            expect(nterm).not_to be_undefined
+          end
+        end      
+      end # context
+      
+      context 'Grammar with undefined symbols:' do
+        subject do
+          productions = [prod_S, prod_A1, prod_A2, prod_A3]
+          Grammar.new(productions)
+        end  
+        
+        it 'should detect its nonterminals that are undefined' do
+          nonterms = subject.non_terminals
+          culprits = nonterms.select do |nterm|
+            nterm.undefined?
+          end
+          
+          expect(culprits.size).to eq(1)
+          expect(culprits[0]).to eq(nt_C)
+        end      
+      end # context
+=end      
+      
     end # describe
   end # module
 end # module

data/spec/rley/syntax/grammar_builder_spec.rb

@@ -9,7 +9,7 @@ module Rley # Open this namespace to avoid module qualifier prefixes
       context 'Initialization without argument:' do
         it 'could be created without argument' do
           expect { GrammarBuilder.new }.not_to raise_error
-        end   
+        end
 
         it 'should have no grammar symbols at start' do
             expect(subject.symbols).to be_empty
@@ -19,12 +19,12 @@ module Rley # Open this namespace to avoid module qualifier prefixes
             expect(subject.productions).to be_empty
         end
       end # context
-      
+
       context 'Initialization with argument:' do
         it 'could be created with a block argument' do
           expect do GrammarBuilder.new { nil }
           end.not_to raise_error
-        end        
+        end
 
         it 'could have grammar symbols from block argument' do
           instance = GrammarBuilder.new do
@@ -36,7 +36,7 @@ module Rley # Open this namespace to avoid module qualifier prefixes
         it 'should have no productions at start' do
             expect(subject.productions).to be_empty
         end
-      end # context      
+      end # context
 
       context 'Adding symbols:' do
         it 'should build terminals from their names' do
@@ -122,8 +122,8 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           expect(subject.grammar).to be_kind_of(Grammar)
           grm = subject.grammar
           expect(grm.rules).to eq(subject.productions)
-          
-          # Invoking the factory method again should return 
+
+          # Invoking the factory method again should return
           # the same grammar object
           second_time = subject.grammar
           expect(second_time).to eq(grm)
@@ -144,13 +144,18 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           expect { instance.grammar }.to raise_error(err, msg)
         end
 
-        it 'should complain when non-terminal has no production' do
-          instance = GrammarBuilder.new
-          instance.add_terminals('a', 'b', 'c')
-          instance.add_production('S' => ['A'])
+        it 'should complain if one or more terminals are useless' do
+          # Add one useless terminal symbol
+          subject.add_terminals('d')
+
           err = StandardError
-          msg = 'Nonterminal A not rewritten'
-          expect { instance.grammar }.to raise_error(err, msg)
+          msg = 'Useless terminal symbol(s): d.'
+          expect { subject.grammar }.to raise_error(err, msg)
+          
+          # Add another useless terminal
+          subject.add_terminals('e')
+          msg = 'Useless terminal symbol(s): d, e.'
+          expect { subject.grammar }.to raise_error(err, msg)          
         end
 
         it 'should build a grammar with nullable nonterminals' do

data/spec/rley/syntax/grammar_spec.rb

@@ -70,12 +70,16 @@ module Rley # Open this namespace to avoid module qualifier prefixes
       # A ::= "b".
       let(:nt_S) { NonTerminal.new('S') }
       let(:nt_A) { NonTerminal.new('A') }
+      let(:nt_B) { NonTerminal.new('B') }
+      let(:nt_C) { NonTerminal.new('C') }
+      let(:nt_D) { NonTerminal.new('D') }      
       let(:a_) { VerbatimSymbol.new('a') }
       let(:b_)  { VerbatimSymbol.new('b') }
       let(:c_)  { VerbatimSymbol.new('c') }
       let(:prod_S) { Production.new(nt_S, [nt_A]) }
       let(:prod_A1) { Production.new(nt_A, [a_, nt_A, c_]) }
       let(:prod_A2) { Production.new(nt_A, [b_]) }
+      let(:prod_A3) {Production.new(nt_A, [c_, nt_C] ) }
 
 =begin
       # Non-terminals that specify the lexicon of the language
@@ -137,16 +141,16 @@ module Rley # Open this namespace to avoid module qualifier prefixes
         it 'should know all its symbols' do
           expect(subject.symbols).to eq([nt_S, nt_A, a_, c_, b_])
         end
-        
+
         it 'should know all its non-terminal symbols' do
-          expect(subject.non_terminals).to eq([nt_S, nt_A]) 
+          expect(subject.non_terminals).to eq([nt_S, nt_A])
         end
-        
+
         it 'should know its start production' do
           expect(subject.start_production).to eq(prod_S)
         end
       end # context
-      
+
       context 'Provided services:' do
         it 'should retrieve its symbols from their name' do
           expect(subject.name2symbol['S']).to eq(nt_S)
@@ -156,9 +160,64 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           expect(subject.name2symbol['c']).to eq(c_)
         end
       end # context
-      
+
+      context 'Grammar diagnosis:' do
+        it 'should mark any non-terminal that has no production' do
+          # S ::= A.
+          # S ::= B.
+          # A ::= "a" .
+          # B ::=  C "b". # C doesn't appear on lhs of a rule
+          prod_S1 = Rley::Syntax::Production.new(nt_S, [nt_A])
+          prod_S2 = Rley::Syntax::Production.new(nt_S, [nt_B])
+          prod_A = Rley::Syntax::Production.new(nt_A, [a_])
+          prod_B = Rley::Syntax::Production.new(nt_B, [nt_C, b_]) # C is undefined
+          instance = Grammar.new([prod_S1, prod_S2, prod_A, prod_B])
+          undefineds = instance.non_terminals.select(&:undefined?)
+          expect(undefineds.size).to eq(1)
+          expect(undefineds.first).to eq(nt_C)
+        end
+
+        it 'should mark any non-terminal as generative or not' do
+          # S ::= A.
+          # S ::= B.
+          # A ::= "a" .
+          # B ::=  C "b". # C doesn't appear on lhs of a rule
+          prod_S1 = Rley::Syntax::Production.new(nt_S, [nt_A])
+          prod_S2 = Rley::Syntax::Production.new(nt_S, [nt_B])
+          prod_A = Rley::Syntax::Production.new(nt_A, [a_])
+          prod_B = Rley::Syntax::Production.new(nt_B, [nt_C, b_]) # C is undefined
+          instance = Grammar.new([prod_S1, prod_S2, prod_A, prod_B])
+          partitioning = instance.non_terminals.partition(&:generative?)
+          expect(partitioning[0].size).to eq(2)
+          expect(partitioning[0]).to eq([nt_S, nt_A])
+          expect(partitioning[1]).to eq([nt_B, nt_C])
+        end
+
+        it "should do a diagnosis even for 'loopy' grammars" do
+          # 'S' => 'A'
+          # 'S' => 'B'
+          # 'A' => 'a'
+          # 'B' => 'C'
+          # 'C' => 'D'
+          # 'D' => 'B'
+          prod_S1 = Rley::Syntax::Production.new(nt_S, [nt_A])
+          prod_S2 = Rley::Syntax::Production.new(nt_S, [nt_B])
+          prod_A = Rley::Syntax::Production.new(nt_A, [a_])
+          prod_B = Rley::Syntax::Production.new(nt_B, [nt_C])
+          prod_C = Rley::Syntax::Production.new(nt_C, [nt_D])
+          prod_D = Rley::Syntax::Production.new(nt_D, [nt_B])
+          instance = Grammar.new([prod_S1, prod_S2, prod_A, prod_B, prod_C, prod_D])
+          partitioning = instance.non_terminals.partition(&:generative?)
+          expect(partitioning[0].size).to eq(2)
+          expect(partitioning[0]).to eq([nt_S, nt_A])
+          expect(partitioning[1]).to eq([nt_B, nt_C, nt_D]) 
+          
+          undefined = instance.non_terminals.select(&:undefined?)
+          expect(undefined).to be_empty
+        end
+      end # context
+
       context 'Non-nullable grammar:' do
-        
         it 'should mark all its nonterminals as non-nullable' do
           nonterms = subject.non_terminals
           nonterms.each do |nterm|
@@ -166,14 +225,14 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           end
         end
       end # context
-      
+
       context 'Nullable grammars:' do
         subject do
-          prod_A3 = Production.new(nt_A, [])
-          productions = [prod_S, prod_A1, prod_A2, prod_A3]
+          prod_A4 = Production.new(nt_A, [])
+          productions = [prod_S, prod_A1, prod_A2, prod_A4]
           Grammar.new(productions)
         end
-        
+
         it 'should mark its nullable nonterminals' do
           # In the default grammar, all nonterminals are nullable
           nonterms = subject.non_terminals
@@ -181,8 +240,8 @@ module Rley # Open this namespace to avoid module qualifier prefixes
             expect(nterm).to be_nullable
           end
         end
-        
       end # context
+
     end # describe
   end # module
 end # module

data/spec/rley/syntax/non_terminal_spec.rb

@@ -32,6 +32,22 @@ module Rley # Open this namespace to avoid module qualifier prefixes
           subject.nullable = false
           expect(subject).not_to be_nullable          
         end
+        
+        it 'should know whether it is defined' do
+          expect(subject.undefined?).to be_nil
+          subject.undefined = true
+          expect(subject).to be_undefined
+          subject.undefined = false
+          expect(subject).not_to be_undefined          
+        end
+
+        it 'should know whether it is generative' do
+          expect(subject.generative?).to be_nil
+          subject.generative = true
+          expect(subject).to be_generative
+          subject.generative = false
+          expect(subject).not_to be_generative          
+        end        
       end # context
     end # describe
   end # module

data/spec/rley/syntax/terminal_spec.rb

@@ -25,6 +25,10 @@ module Rley # Open this namespace to avoid module qualifier prefixes
         it "should know that isn't nullable" do
           expect(subject).not_to be_nullable
         end
+        
+        it "should know that it is generative" do
+          expect(subject).to be_generative
+        end        
       end # context
     end # describe
   end # module

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rley
 version: !ruby/object:Gem::Version
-  version: 0.4.05
+  version: 0.4.06
 platform: ruby
 authors:
 - Dimitri Geshef
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-05-06 00:00:00.000000000 Z
+date: 2017-05-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake