sequitur 0.1.10 → 0.1.11

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    ZGMxY2RmOWZlOWI3MzljMjNmZmEyZDFjZDUzZDg2YzQzNzM2MmRhZQ==
+    YjAyNWQ2ZDE4MmE2NGU3MTVjZTMxNDc2YjI4ZmE3NTBlMWRlZWM0MQ==
   data.tar.gz: !binary |-
-    YzNhZGJmNWY4YTg3MWY3ODI5YzE4ZTg5MTUyNTlkMGFmMzhlNjdiYw==
+    NGMzZDRiYjNiYzAwNjE0MzFhOGI0YWQyNjU4YmRjMzNkMDY1ZGEzOA==
 !binary "U0hBNTEy":
   metadata.gz: !binary |-
-    NWY1NDhiZjUzNjJjNTdlMmE2ZjI0ODc3MDNlMjQ2MGM3ZjEzMWQxNzUxZDgz
-    YmE4ZmFlNzI4NDc2NGE0YjQzOTJhNmViYmQ4MThjZTg0YmEyN2JiOTlkMzk4
-    MTc4MWUwMWE3ZWY0NTQ0Y2ZlYjU3MmVkODNiZjUwMzgxZWNlMDQ=
+    NjBhMGRlYWIyNmI1MWJmNmE0NDc5MzU4NTc4MjI1ZDVhYjIwZDA2NjAyYzdj
+    NGMzYmU4NmYzNDQ2NTI4MWJhYTQxYTBkYTU1NGVjODcyNjIwNjQxM2Q5ZTUz
+    NGU3YWQzODk2ODA5OWFkZDY0ZThkNjhmMzMzNTdhMTI2ZmZiYmM=
   data.tar.gz: !binary |-
-    MWI3N2U3ZGU5N2Y4ZTEzNDdmYjQyYjExOWIzYWRjMWE0ZDUxYmM2MjI4YzU0
-    ODZiZTQ3ZDlkMWMxMmY2ZWQwM2JjNmZiNmU1ZDRhMTFhMjY4ZjI5MjJmNWFl
-    MDYxZmU0ZWUwOTdhNDBkODE4NDJjNzQ0MGRmMWQwMDIxMjkwMTI=
+    NDA2OTE2YTAxYmZmODg1NTU3YWFmNTlmNWIwNzU2NjNlMTEzZmE2NGFiMDkw
+    OTYxMTkzMmNmZDJkZGJiZmE2ZDljNzdjY2EzZDk2YmYyMmM4YWJmNDBhNWI2
+    YWJjYWE1NWYwZmE3Y2Q0ZDMzMGYyMjdhMWQzZDI5NzI1MDgyMzc=

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+### 0.1.11 / 2014-10-07
+* [CHANGE] File `README.md`: Added an example showing that Sequitur can work on a sequence of integers.
+* [NEW] Folder `examples` Added a few code sample.
+
 ### 0.1.10 / 2014-10-05
 * [CHANGE] Code refactoring for performance. Impacted classes: `SequiturGrammar`, `SymbolSequence` and `Production`.
 

data/README.md

@@ -15,6 +15,13 @@ The following are good entry points to learn about the algorithm:
 [Sequitur algorithm home](http://sequitur.info/)  
 [Wikipedia](http://en.wikipedia.org/wiki/Sequitur_algorithm)  
 
+### Highlights ###
+* Pure Ruby implementation
+* No runtime dependency with other gems,
+* Test suite with 100%,
+* Documentation: 100% coverage (according to YARD), green badge from inch.io
+* Algorithm works with different input token types (no limited to single character)
+
 ### The theory in a nutshell ###
 Given a sequence of input tokens (say, characters), the Sequitur algorithm 
 will represent that input sequence as a set of rules. As the algorithm detects 
@@ -192,7 +199,57 @@ $[sudo] gem install sequitur
 
 
 
-### TODO: Add more documentation ###
+### Good to know ###
+The above examples might give the impression that the input stream must consist of single 
+character tokens. This is simply not true.  
+This implementation is flexible enough to cope with other kinds of input values.  
+The next example shows how integer values can be correctly processed by Sequitur.  
+Assume that the input is the array of Fixnums *[1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5]*. 
+Then Sequitur algorithm will generate the rule set:  
+```
+start : P1 P2 P3 P3 e.  
+P1 : a b.  
+P2 : P1 c.  
+P3 : P2 d.   
+```
+
+
+```ruby
+require 'sequitur'  # Load the Sequitur library
+
+#
+# Purpose: demo of Sequitur with a stream of integer values
+#
+input_sequence =  [1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5]
+
+# Generate the grammar from the sequence
+grammar = Sequitur.build_from(input_sequence)
+
+
+# Use a formatter to display the grammar rules on the console output
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P1 P2 P3 P3 5.
+# P1 : 1 2.
+# P2 : P1 3.
+# P3 : P2 4.
+
+# Playing a bit with the API
+# Access last symbol from rhs of start production:
+last_symbol_p0 = grammar.start.rhs.symbols[-1]  
+puts last_symbol_p0     # => 5
+
+# Access first symbol from rhs of P1 production:
+first_symbol_p1 = grammar.productions[1].rhs.symbols[0]
+
+puts first_symbol_p1    # => 1
+```
+
+More examples are available in the examples folder.  
 
 
 Copyright

data/Rakefile

@@ -1,31 +1,31 @@
-require 'rubygems'
-require_relative './lib/sequitur/constants'
-
-namespace :gem do
-
-desc 'Push the gem to rubygems.org'
-task :push do
-  system("gem push sequitur-#{Sequitur::Version}.gem")
-end
-
-end # namespace
-
-# Testing-specific tasks
-
-# RSpec as testing tool
-require 'rspec/core/rake_task'
-desc 'Run RSpec'
-RSpec::Core::RakeTask.new do |spec|
-  spec.pattern = 'spec/**/*_spec.rb'
-end
-
-
-# Run RSpec tests
-desc 'Run tests, with RSpec'
-task test: [:spec]
-
-
-# Default rake task
-task default: :test
-
-# End of file
+require 'rubygems'
+require_relative './lib/sequitur/constants'
+
+namespace :gem do
+
+desc 'Push the gem to rubygems.org'
+task :push do
+  system("gem push sequitur-#{Sequitur::Version}.gem")
+end
+
+end # namespace
+
+# Testing-specific tasks
+
+# RSpec as testing tool
+require 'rspec/core/rake_task'
+desc 'Run RSpec'
+RSpec::Core::RakeTask.new do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+end
+
+
+# Run RSpec tests
+desc 'Run tests, with RSpec'
+task test: [:spec]
+
+
+# Default rake task
+task default: :test
+
+# End of file

data/examples/integer_sample.rb

@@ -0,0 +1,33 @@
+require 'sequitur'  # Load the Sequitur library
+
+#
+# Purpose: show how to apply Sequitur on a stream of integer values
+#
+input_sequence =  [1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5]
+
+# Generate the grammar from the sequence
+grammar = Sequitur.build_from(input_sequence)
+
+
+# Use a formatter to display the grammar rules on the console output
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P1 P2 P3 P3 5.
+# P1 : 1 2.
+# P2 : P1 3.
+# P3 : P2 4.
+
+# Playing a bit with the API
+# Access last symbol from rhs of start production:
+last_symbol_p0 = grammar.start.rhs.symbols[-1]  
+puts last_symbol_p0     # => 5
+
+# Access first symbol from rhs of P1 production:
+first_symbol_p1 = grammar.productions[1].rhs.symbols[0]
+
+puts first_symbol_p1    # => 1
+

data/examples/porridge.rb

@@ -0,0 +1,41 @@
+require 'sequitur'  # Load the Sequitur library
+
+
+# Purpose: demo to show that sequitur gem works on example from sequitur.info website
+input_sequence =        
+      input = <<-SNIPPET
+pease porridge hot,
+pease porridge cold,
+pease porridge in the pot,
+nine days old.
+
+some like it hot,
+some like it cold,
+some like it in the pot,
+nine days old.
+SNIPPET
+
+grammar = Sequitur.build_from(input_sequence)
+
+# To display the grammar rules on the console output
+# We use a formatter
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules. 
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P2 P8 P3 P10 P3 P12 P9 P8 P11 P10 P11 P12.
+# P1 : e  .
+# P2 : p e a s P4 r r i d g P1.
+# P3 : P5 P2.
+# P4 : P1 p o.
+# P5 : ,
+# .
+# P6 : i n.
+# P7 : o l d.
+# P8 : h o t.
+# P9 : s o m P1 l i k P1 i t  .
+# P10 : c P7.
+# P11 : P5 P9.
+# P12 : P6   t h P4 t P5 n P6 P1 d a y s   P7 .

data/examples/simple_case.rb

@@ -0,0 +1,27 @@
+require 'sequitur'  # Load the Sequitur library
+
+
+# Purpose: show how to apply Sequitur on a stream of single characters
+input_sequence =  'ababcabcdabcde'  # Let's analyze this string
+
+# The SEQUITUR algorithm will detect the repeated 'ab' pattern
+# and will generate a context-free grammar that represents the input string
+grammar = Sequitur.build_from(input_sequence)
+
+# To display the grammar rules on the console output
+# We use a formatter
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules. Each rule is displayed with the format:
+# rule_id : a_sequence_grammar_symbols.
+# Where: 
+# - rule_id is either 'start' or a name like 'Pxxxx' (xxxx is a sequential number)
+# - a grammar symbol is either a terminal symbol 
+# (i.e. a character from the input) or a rule id
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P1 P2 P3 P3 e.
+# P1 : a b.
+# P2 : P1 c.
+# P3 : P2 d.

data/examples/symbol_sample.rb

@@ -0,0 +1,28 @@
+require 'sequitur'  # Load the Sequitur library
+
+#
+# Purpose: show how to apply Sequitur on a stream of Symbol values
+#
+input_sequence =  [
+  :aa, :bb, :aa, :bb,
+  :cc, :aa, :bb, :cc, 
+  :dd, :aa, :bb, :cc, 
+  :dd, :ee
+]
+
+# Generate the grammar from the sequence
+grammar = Sequitur.build_from(input_sequence)
+
+
+# Use a formatter to display the grammar rules on the console output
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P1 P2 P3 P3 ee.
+# P1 : aa bb.
+# P2 : P1 cc.
+# P3 : P2 dd.
+

data/examples/word_sample.rb

@@ -0,0 +1,30 @@
+require 'sequitur'  # Load the Sequitur library
+
+#
+# Purpose: show how to apply Sequitur on a stream of text words
+#
+
+# Raw input is one String containing repeated sentences...
+raw_input = <<-SNIPPET
+Error: unknown character '?' at position 6
+Error: illegal character '%' at position 20
+Error: unknown character '/' at position 9
+SNIPPET
+
+# Convert into a sequence of words
+input_sequence = raw_input.scan(/\w+/)
+# Generate the grammar from the sequence
+grammar = Sequitur.build_from(input_sequence)
+
+
+# Use a formatter to display the grammar rules on the console output
+formatter = Sequitur::Formatter::BaseText.new(STDOUT)
+
+# Now render the rules
+formatter.render(grammar.visitor)
+
+# Rendered output is:
+# start : P2 6 Error illegal P1 20 P2 9.
+# P1 : character at position.
+# P2 : Error unknown P1.
+

data/lib/sequitur.rb

@@ -22,7 +22,7 @@ module Sequitur
       when Enumerator then tokens
       else tokens.to_enum
     end
-    
+
     return SequiturGrammar.new(input_sequence)
   end
 end # module

data/lib/sequitur/constants.rb

@@ -3,7 +3,7 @@
 
 module Sequitur # Module used as a namespace
   # The version number of the gem.
-  Version = '0.1.10'
+  Version = '0.1.11'
 
   # Brief description of the gem.
   Description = 'Ruby implementation of the Sequitur algorithm'

data/lib/sequitur/digram.rb

@@ -1,52 +1,52 @@
-# File: digram.rb
-
-module Sequitur # Module for classes implementing the Sequitur algorithm
-
-# In linguistics, a digram is a sequence of two letters.
-# In Sequitur, a digram is a sequence of two consecutive symbols that
-# appear in a production rule. Each symbol in a digram 
-# can be a terminal or not.
-class Digram
-  # The sequence of two consecutive grammar symbols.
-  #   The two symbols should respond to the :hash message.
-  attr_reader(:symbols)
-  
-  # An unique hash key of the digram
-  attr_reader(:key)
-  
-  # The production in which the digram occurs
-  attr_reader(:production)
-  
-  # Constructor.
-  # A digram represents a sequence of two symbols 
-  # (that appears in a rhs of a production).
-  # Terminal symbols must respond to the :hash message.
-  # @param symbol1 [StringOrSymbol] First element of the digram
-  # @param symbol2 [StringOrSymbol] Second element of the digram
-  # @param aProduction [Production] Production in which the RHS 
-  #   the sequence symbol1 symbol2 appears.
-  def initialize(symbol1, symbol2, aProduction)
-    @symbols = [symbol1, symbol2]
-    @key = symbol1.hash.to_s(16) + ':' + symbol2.hash.to_s(16) 
-    @production = aProduction
-  end
-  
-  # Equality testing.
-  #   true iff keys of both digrams are equal, false otherwise
-  # @param other [Digram] another to compare with
-  # @return [true/false]
-  def ==(other)
-    return key == other.key
-  end
-  
-  # Does the digram consists of twice the same symbols?
-  # @return [true/false] true when symbols.first == symbols.last
-  def repeating?()
-    return symbols[0] == symbols[1]
-  end
-  
-end # class
-
-end # module
-
-# End of file
+# File: digram.rb
+
+module Sequitur # Module for classes implementing the Sequitur algorithm
+
+# In linguistics, a digram is a sequence of two letters.
+# In Sequitur, a digram is a sequence of two consecutive symbols that
+# appear in a production rule. Each symbol in a digram
+# can be a terminal or not.
+class Digram
+  # The sequence of two consecutive grammar symbols.
+  #   The two symbols should respond to the :hash message.
+  attr_reader(:symbols)
+
+  # An unique hash key of the digram
+  attr_reader(:key)
+
+  # The production in which the digram occurs
+  attr_reader(:production)
+
+  # Constructor.
+  # A digram represents a sequence of two symbols
+  # (that appears in a rhs of a production).
+  # Terminal symbols must respond to the :hash message.
+  # @param symbol1 [StringOrSymbol] First element of the digram
+  # @param symbol2 [StringOrSymbol] Second element of the digram
+  # @param aProduction [Production] Production in which the RHS
+  #   the sequence symbol1 symbol2 appears.
+  def initialize(symbol1, symbol2, aProduction)
+    @symbols = [symbol1, symbol2]
+    @key = symbol1.hash.to_s(16) + ':' + symbol2.hash.to_s(16)
+    @production = aProduction
+  end
+
+  # Equality testing.
+  #   true iff keys of both digrams are equal, false otherwise
+  # @param other [Digram] another to compare with
+  # @return [true/false]
+  def ==(other)
+    return key == other.key
+  end
+
+  # Does the digram consists of twice the same symbols?
+  # @return [true/false] true when symbols.first == symbols.last
+  def repeating?()
+    return symbols[0] == symbols[1]
+  end
+
+end # class
+
+end # module
+
+# End of file