panini 1.0.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,13 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "rspec", "~> 2.3.0"
+  gem "bundler", "~> 1.0.0"
+  gem "jeweler", "~> 1.6.0"
+  gem "rcov", ">= 0"
+end

data/Gemfile.lock

@@ -0,0 +1,28 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    diff-lcs (1.1.2)
+    git (1.2.5)
+    jeweler (1.6.0)
+      bundler (~> 1.0.0)
+      git (>= 1.2.5)
+      rake
+    rake (0.8.7)
+    rcov (0.9.9)
+    rspec (2.3.0)
+      rspec-core (~> 2.3.0)
+      rspec-expectations (~> 2.3.0)
+      rspec-mocks (~> 2.3.0)
+    rspec-core (2.3.1)
+    rspec-expectations (2.3.0)
+      diff-lcs (~> 1.1.2)
+    rspec-mocks (2.3.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 1.0.0)
+  jeweler (~> 1.6.0)
+  rcov
+  rspec (~> 2.3.0)

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Matthew Bellantoni
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,151 @@
+= panini
+
+Panini is a flexible toolkit that enables you to generate sentences from a context-free grammar, also known as a CFG.
+
+== CFG Background
+
+Informally, a context-free grammar consists of a set of productions rules where a _nonterminal_ on the right
+hand side of the produces a string of terminals and nonterminals on the right hand side.  Like this:
+
+  S -> AB
+  A -> a
+  B -> b
+
+I the above example, _S_, _A_, and _B_ are all nonterminals.  _a_ and _b_ are terminals.  Furthermore, the nonterminal _S_ is the _start_ _symbol_ for this CFG.  By applying the productions as follows:
+
+  S     (start symbol)
+  AB    (apply S -> AB)
+  aB    (apply A -> a)
+  ab    (apply B -> b)
+
+The sentence _ab_ is generated.  In fact, this is the only sentence this grammar can produce!  By adding one additional production to the grammar:
+
+  S -> ASB
+
+The grammar may now potentially create an infinite number of sentences.  They will all have the form of _a_<sup>i</sup>_b_<sup>i</sup> where _i_ > 1. Here is one more example derivation:
+
+  S      (start symbol)
+  ASB    (apply S -> ASB)
+  aSB    (apply A -> a)
+  aSb    (apply B -> b)
+  aaSbb  (apply S -> ASB)
+  aaABbb (apply S -> AB)
+  aaaBbb (apply A -> a)
+  aaabbb (apply B -> b)
+
+You learn more about CFGs, you can reference the CFG article[http://en.wikipedia.org/wiki/Context-free_grammar] on Wikipedia</a>.
+
+== Getting Started With Panini
+
+=== Defining a Grammar
+
+Defining a grammar is easy.  Create a grammar object, add some nonterminals and then add the productions to those nonterminals.
+
+Here's how the grammar from above is defined:
+
+  grammar = Panini::Grammar.new
+
+  nt_s = grammar.add_nonterminal
+  nt_a = grammar.add_nonterminal
+  nt_b = grammar.add_nonterminal
+
+  n_s.add_production([n_a, n_b])        # S -> AB
+  n_s.add_production([n_a, n_s, n_b])   # S -> ASB
+  n_a.add_production(['a'])             # A -> 'a'
+  n_b.add_production(['b'])             # A -> 'b'
+
+=== Derivators
+
+Derivators are objects that take a Panini::Grammar and then apply the rules to generate a sentence.  Creating the sentences
+from the grammar can be tricky, and certain derivation strategies may be better for some grammars.
+Currently, the main derivator is the Panini::DerivationStrategy::RandomDampened derivator.
+
+  derivator = Panini::DerivationStrategy::RandomDampened.new(grammar)
+
+=== Generating a Sentence
+
+To generate a sentence, call the derivator's sentence method like thus:
+
+  derivator.sentence -> ['a', 'a', 'b', 'b']
+
+You will get a new sentence (depending on the grammar) with every call:
+
+  derivator.sentence -> ['a', 'a', 'a', 'a', 'b', 'b', 'b', 'b']
+
+
+== Example
+
+In this example, we create a grammar that generates mathematical expressions.
+
+  # ================
+  # = Nonterminals =
+  # ================
+  expression = grammar.add_nonterminal("EXPR")
+  term = grammar.add_nonterminal("TERM")
+  factor = grammar.add_nonterminal("FACT")
+  identifier = grammar.add_nonterminal("ID")
+  number = grammar.add_nonterminal("NUM")
+
+
+  # =============
+  # = Terminals =
+  # =============
+  expression.add_production([term, '+', term])
+  expression.add_production([term, '-', term])
+  expression.add_production([term])
+
+  term.add_production([factor, '*', term])
+  term.add_production([factor, '/', term])
+  term.add_production([factor])
+
+  factor.add_production([identifier])
+  factor.add_production([number])
+  factor.add_production(['(', expression, ')'])
+
+  ('a'..'z').each do |v|
+    identifier.add_production([v])
+  end
+
+  (0..100).each do |n|
+    number.add_production([n])
+  end
+
+  # ===============================================
+  # = Choose a strategy and create some sentences =
+  # ===============================================
+  deriver = Panini::DerivationStrategy::RandomDampened.new(grammar)
+  10.times do
+    puts "#{deriver.sentence.join(' ')}"
+  end
+
+== Contributing to panini
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+== To Do
+
+=== Features
+* Detect invalid grammars
+* Weighted productions.
+* Arbitrary start symbol.
+* Support Enumerator?
+* DSL?
+* Purdom Derivator?
+
+=== Examples
+* Natural language
+* XML
+* JSON
+* Address
+* Tree/Flower (PS?)
+* Simulated user actions
+
+== Copyright
+
+Copyright (c) 2011 Matthew Bellantoni. See LICENSE.txt for further details.
+

data/Rakefile

@@ -0,0 +1,49 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "panini"
+  gem.homepage = "http://github.com/mjbellantoni/panini"
+  gem.license = "MIT"
+  gem.summary = %Q{Create sentences from a context-free grammar (CFG)}
+  gem.description = %Q{Panini allows you to generate sentences from a context-free grammar, also known as a CFG.}
+  gem.email = "mjbellantoni@yahoo.com"
+  gem.authors = ["mjbellantoni"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "panini #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/examples/arithmetic_expression.rb

@@ -0,0 +1,51 @@
+# Clean this up.  Have it assume there's an install?
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+
+require "panini"
+
+grammar = Panini::Grammar.new
+
+
+# ================
+# = Nonterminals =
+# ================
+expression = grammar.add_nonterminal("EXPR")
+term = grammar.add_nonterminal("TERM")
+factor = grammar.add_nonterminal("FACT")
+identifier = grammar.add_nonterminal("ID")
+number = grammar.add_nonterminal("NUM")
+
+
+# =============
+# = Terminals =
+# =============
+expression.add_production([term, '+', term])
+expression.add_production([term, '-', term])
+expression.add_production([term])
+
+term.add_production([factor, '*', term])
+term.add_production([factor, '/', term])
+term.add_production([factor])
+
+factor.add_production([identifier])
+factor.add_production([number])
+factor.add_production(['(', expression, ')'])
+
+('a'..'z').each do |v|
+  identifier.add_production([v])
+end
+
+# It would be cool to have a way to create a random number.
+(0..100).each do |n|
+  number.add_production([n])
+end
+
+
+# ===============================================
+# = Choose a strategy and create some sentences =
+# ===============================================
+deriver = Panini::DerivationStrategy::RandomDampened.new(grammar)
+10.times do
+  puts "#{deriver.sentence.join(' ')}"
+end

data/lib/derivation_strategy/base.rb

@@ -0,0 +1,13 @@
+module Panini
+  module DerivationStrategy
+    
+    class Base
+
+      def initialize(grammar)
+        @grammar = grammar
+      end
+
+    end
+
+  end
+end

data/lib/derivation_strategy/leftmost.rb

@@ -0,0 +1,69 @@
+module Panini
+  module DerivationStrategy
+
+    class RoundRobinProductionChoiceProxy
+
+      def initialize(nonterminal)
+        @nonterminal = nonterminal
+        @round_robin_count = 0
+        @production_count = @nonterminal.productions.count
+      end
+
+      def production
+        i = @round_robin_count % @production_count
+        @round_robin_count += 1
+        @nonterminal.productions[i]
+      end
+
+    end
+
+    # The Leftmost strategy is a naive strategy for deriving sentences from a grammar.  It
+    # will aways substitute for the leftmost nonterminal first.  If a nonterminal has more
+    # than one production, they will be chosen in a round-robin ordering.
+    #
+    # This implementation is slow and will not work on many grammars.
+    #
+    # In other words, don't use this! It's  in place because it is simple to and was used
+    # for early testing.
+    class Leftmost < Base
+
+      def initialize(grammar)
+        build_production_proxies(grammar)
+        super(grammar)
+      end
+
+      def build_production_proxies(grammar)
+        @production_proxies = {}
+        grammar.nonterminals.each do |nonterminal|
+          @production_proxies[nonterminal] = RoundRobinProductionChoiceProxy.new(nonterminal)
+        end
+      end
+      private :build_production_proxies
+
+      # Generates a sentence. 
+      def sentence
+        derived_sentence, substituted = [@grammar.start], false
+        begin
+          derived_sentence, substituted = substitution_pass(derived_sentence)
+        end while substituted
+        derived_sentence
+      end
+
+      def substitution_pass(derived_sentence)
+        substituted = false
+        derived_sentence = derived_sentence.flat_map do |term|
+          if !substituted && (term.class == Nonterminal)
+            substituted = true
+            @production_proxies[term].production
+          else
+            term
+          end
+        end
+        return derived_sentence, substituted
+      end
+      private :substitution_pass
+
+    end
+
+  end
+end

data/lib/derivation_strategy/random_dampened.rb

@@ -0,0 +1,115 @@
+
+
+module Panini
+  module DerivationStrategy
+
+    class DampenedProbabilityProductionChoiceProxy
+
+      def initialize(nonterminal, damping=0.25)
+        @nonterminal = nonterminal
+        @damping = damping
+        @production_counts = @nonterminal.productions.map do
+          0
+        end
+      end
+
+      def initialize_copy(source)  
+        super
+        @production_counts = @production_counts.map do |production_count|
+          production_count
+        end
+      end
+
+      def production
+        i = find_index
+        @production_counts[i] += 1
+        @nonterminal.productions[i]
+      end
+
+      def find_index
+
+        weights = @production_counts.map do |production_count|
+          @damping ** production_count
+        end
+
+        selector = Kernel::rand() * weights.inject(:+)
+
+        weights.each_with_index do |weight, i|
+          selector -= weight
+          if selector < 0
+            return i
+          end
+        end
+
+        raise "You shouldn't be able to get here. #{selector} #{@production_counts}"
+
+      end
+      private :find_index
+
+      def dump_weights
+        puts "production_counts:"
+        @weights.each do |weight|
+          puts "#{weight} "
+        end
+      end
+
+    end
+
+
+    # This derivation strategy uses a dampening factor to reduce the liklihood of hitting either
+    # too-deep or infinite traversals through the grammar.  This is based on material presented
+    # here:
+    #
+    # http://eli.thegreenplace.net/2010/01/28/generating-random-sentences-from-a-context-free-grammar
+    class RandomDampened < Base
+
+      # Initializes the derivator.  The damping factor is a number betweeon 0.0 and 1.0.  In
+      # general, the smaller the number the shorter the senetence generated by the derivator.
+      # If the number is close to 1.0, it is possible that you will encounter stack errors!
+      def initialize(grammar, damping = 0.25)
+        if (damping <= 0.0) || (damping >= 1.0)
+          raise ArgumentError, "The damping factor must be greater than 0.0 and less than 1.0."
+        end
+        build_production_proxies(grammar, damping)
+        super(grammar)
+      end
+
+      def build_production_proxies(grammar, damping)
+        @production_proxies = {}
+        grammar.nonterminals.each do |nonterminal|
+          @production_proxies[nonterminal] = DampenedProbabilityProductionChoiceProxy.new(nonterminal, damping)
+        end
+      end
+      private :build_production_proxies
+
+      # Generates a sentence. 
+      def sentence
+        substitute_nonterminal(@grammar.start, @production_proxies, 0)
+      end
+
+      def substitute_nonterminal(nonterminal, production_proxies, depth)
+
+        # production_proxies_copy = {}
+        #  production_proxies_copy = production_proxies.each do |key, value|
+        #    production_proxies_copy[key] = value.dup
+        #  end
+        #  
+        production_proxies_copy = production_proxies.map do |value|
+          value.dup
+        end
+        
+        production_proxies_copy[nonterminal].production.flat_map do |term|
+          if (term.class == Nonterminal)
+            substitute_nonterminal(term, production_proxies_copy, depth + 1)
+          else
+            term
+          end
+        end
+
+      end
+      private :substitute_nonterminal
+
+    end
+
+  end
+end