classifier 1.3.4 → 1.4.0

data/README.markdown

@@ -1,97 +0,0 @@
-## Welcome to Classifier
-
-Classifier is a general module to allow Bayesian and other types of classifications.
-
-## Download
-
-* https://github.com/cardmagic/classifier
-* gem install classifier
-* git clone https://github.com/cardmagic/classifier.git
-
-## Dependencies
-
-If you install Classifier from source, you'll need to install Roman Shterenzon's fast-stemmer gem with RubyGems as follows:
-
-    gem install fast-stemmer
-
-If you would like to speed up LSI classification by at least 10x, please install the following libraries:
-GNU GSL:: http://www.gnu.org/software/gsl
-rb-gsl:: http://rb-gsl.rubyforge.org
-
-Notice that LSI will work without these libraries, but as soon as they are installed, Classifier will make use of them. No configuration changes are needed, we like to keep things ridiculously easy for you.
-
-## Bayes
-
-A Bayesian classifier by Lucas Carlson. Bayesian Classifiers are accurate, fast, and have modest memory requirements.
-
-### Usage
-
-    require 'classifier'
-    b = Classifier::Bayes.new 'Interesting', 'Uninteresting'
-    b.train_interesting "here are some good words. I hope you love them"
-    b.train_uninteresting "here are some bad words, I hate you"
-    b.classify "I hate bad words and you" # returns 'Uninteresting'
-    
-    require 'madeleine'
-    m = SnapshotMadeleine.new("bayes_data") {
-        Classifier::Bayes.new 'Interesting', 'Uninteresting'
-    }
-    m.system.train_interesting "here are some good words. I hope you love them"
-    m.system.train_uninteresting "here are some bad words, I hate you"
-    m.take_snapshot
-    m.system.classify "I love you" # returns 'Interesting'
-
-Using Madeleine, your application can persist the learned data over time.
-
-### Bayesian Classification
-
-* http://www.process.com/precisemail/bayesian_filtering.htm
-* http://en.wikipedia.org/wiki/Bayesian_filtering
-* http://www.paulgraham.com/spam.html
-
-## LSI
-
-A Latent Semantic Indexer by David Fayram. Latent Semantic Indexing engines
-are not as fast or as small as Bayesian classifiers, but are more flexible, providing 
-fast search and clustering detection as well as semantic analysis of the text that 
-theoretically simulates human learning.
-
-### Usage
-
-    require 'classifier'
-    lsi = Classifier::LSI.new
-    strings = [ ["This text deals with dogs. Dogs.", :dog],
-              ["This text involves dogs too. Dogs! ", :dog],
-              ["This text revolves around cats. Cats.", :cat],
-              ["This text also involves cats. Cats!", :cat],
-              ["This text involves birds. Birds.",:bird ]]
-    strings.each {|x| lsi.add_item x.first, x.last}
-  
-    lsi.search("dog", 3)
-    # returns => ["This text deals with dogs. Dogs.", "This text involves dogs too. Dogs! ", 
-    #             "This text also involves cats. Cats!"]
-
-    lsi.find_related(strings[2], 2)
-    # returns => ["This text revolves around cats. Cats.", "This text also involves cats. Cats!"]
-  
-    lsi.classify "This text is also about dogs!"
-    # returns => :dog
-  
-Please see the Classifier::LSI documentation for more information. It is possible to index, search and classify
-with more than just simple strings. 
-
-### Latent Semantic Indexing
-
-* http://www.c2.com/cgi/wiki?LatentSemanticIndexing
-* http://www.chadfowler.com/index.cgi/Computing/LatentSemanticIndexing.rdoc
-* http://en.wikipedia.org/wiki/Latent_semantic_analysis
-
-## Authors    
-
-* Lucas Carlson  (lucas@rufy.com)
-* David Fayram II (dfayram@gmail.com)
-* Cameron McBride (cameron.mcbride@gmail.com)
-* Ivan Acosta-Rubio (ivan@softwarecriollo.com)
-
-This library is released under the terms of the GNU LGPL. See LICENSE for more details.
-

data/Rakefile

@@ -1,84 +0,0 @@
-require 'rubygems'
-require 'rake'
-require 'rake/testtask'
-require 'rdoc/task'
-require 'rake/contrib/rubyforgepublisher'
-
-desc "Default Task"
-task :default => [ :test ]
-
-# Run the unit tests
-desc "Run all unit tests"
-Rake::TestTask.new("test") { |t|
-  t.libs << "lib"
-  t.pattern = 'test/*/*_test.rb'
-  t.verbose = true
-}
-
-# Make a console, useful when working on tests
-desc "Generate a test console"
-task :console do
-   verbose( false ) { sh "irb -I lib/ -r 'classifier'" }
-end
-
-# Genereate the RDoc documentation
-desc "Create documentation"
-Rake::RDocTask.new("doc") { |rdoc|
-  rdoc.title = "Ruby Classifier - Bayesian and LSI classification library"
-  rdoc.rdoc_dir = 'html'
-  rdoc.rdoc_files.include('README')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-}
-
-# Genereate the package
-spec = Gem::Specification.new do |s|
-
-  #### Basic information.
-
-  s.name = 'classifier'
-  s.version = PKG_VERSION
-  s.summary = <<-EOF
-   A general classifier module to allow Bayesian and other types of classifications.
-  EOF
-  s.description = <<-EOF
-   A general classifier module to allow Bayesian and other types of classifications.
-  EOF
-
-  #### Which files are to be included in this gem?  Everything!  (Except CVS directories.)
-
-  s.files = PKG_FILES
-
-  #### Load-time details: library and application (you will need one or both).
-
-  s.require_path = 'lib'
-  s.autorequire = 'classifier'
-
-  #### Documentation and testing.
-
-  s.has_rdoc = true
-
-  #### Dependencies and requirements.
-
-  s.add_dependency('fast-stemmer', '>= 1.0.0')
-  s.requirements << "A porter-stemmer module to split word stems."
-
-  #### Author and project details.
-  s.author = "Lucas Carlson"
-  s.email = "lucas@rufy.com"
-  s.homepage = "http://classifier.rufy.com/"
-end
-
-desc "Report code statistics (KLOCs, etc) from the application"
-task :stats do
-  require 'code_statistics'
-  CodeStatistics.new(
-    ["Library", "lib"],
-    ["Units", "test"]
-  ).to_s
-end
-
-desc "Publish new documentation"
-task :publish do
-   `ssh rufy update-classifier-doc`
-    Rake::RubyForgePublisher.new('classifier', 'cardmagic').upload
-end

data/test/bayes/bayesian_test.rb

@@ -1,33 +0,0 @@
-require File.dirname(__FILE__) + '/../test_helper'
-class BayesianTest < Test::Unit::TestCase
-	def setup
-		@classifier = Classifier::Bayes.new 'Interesting', 'Uninteresting'
-	end
-	
-	def test_good_training
-		assert_nothing_raised { @classifier.train_interesting "love" }
-	end
-
-	def test_bad_training
-		assert_raise(StandardError) { @classifier.train_no_category "words" }
-	end
-	
-	def test_bad_method
-		assert_raise(NoMethodError) { @classifier.forget_everything_you_know "" }
-	end
-	
-	def test_categories
-		assert_equal ['Interesting', 'Uninteresting'].sort, @classifier.categories.sort
-	end
-
-	def test_add_category
-		@classifier.add_category 'Test'
-		assert_equal ['Test', 'Interesting', 'Uninteresting'].sort, @classifier.categories.sort
-	end
-
-	def test_classification
-		@classifier.train_interesting "here are some good words. I hope you love them"
-		@classifier.train_uninteresting "here are some bad words, I hate you"
-		assert_equal 'Uninteresting', @classifier.classify("I hate bad words and you")
-	end
-end

data/test/extensions/word_hash_test.rb

@@ -1,35 +0,0 @@
-require File.dirname(__FILE__) + '/../test_helper'
-class StringExtensionsTest < Test::Unit::TestCase
-	def test_word_hash
-		hash = {:good=>1, :"!"=>1, :hope=>1, :"'"=>1, :"."=>1, :love=>1, :word=>1, :them=>1, :test=>1}
-		assert_equal hash, "here are some good words of test's. I hope you love them!".word_hash
-	end
-
-   	
-	def test_clean_word_hash
-	   hash = {:good=>1, :word=>1, :hope=>1, :love=>1, :them=>1, :test=>1}
-	   assert_equal hash, "here are some good words of test's. I hope you love them!".clean_word_hash
-	end
-
-end
-
-
-class ArrayExtensionsTest < Test::Unit::TestCase
-
-  def test_plays_nicely_with_any_array
-    assert_equal [Array].sum, Array
-  end
-
-  def test_monkey_path_array_sum
-    assert_equal [1,2,3].sum, 6
-  end
-
-  def test_summing_an_empty_array
-    assert_equal [nil].sum, 0
-  end
-
-  def test_summing_an_empty_array
-    assert_equal Array[].sum, 0
-  end
-
-end

data/test/lsi/lsi_test.rb

@@ -1,123 +0,0 @@
-require File.dirname(__FILE__) + '/../test_helper'
-class LSITest < Test::Unit::TestCase
-	def setup
-	  # we repeat principle words to help weight them. 
-	  # This test is rather delicate, since this system is mostly noise.
-     @str1 = "This text deals with dogs. Dogs."
-	  @str2 = "This text involves dogs too. Dogs! "
-	  @str3 = "This text revolves around cats. Cats."
-	  @str4 = "This text also involves cats. Cats!"
-	  @str5 = "This text involves birds. Birds."
-	end
-	
-	def test_basic_indexing
-	 lsi = Classifier::LSI.new
-	  [@str1, @str2, @str3, @str4, @str5].each { |x| lsi << x }
-	  assert ! lsi.needs_rebuild?
-	  
-	 # note that the closest match to str1 is str2, even though it is not
-	 # the closest text match.
-	 assert_equal [@str2, @str5, @str3], lsi.find_related(@str1, 3)
-	end
-
-	def test_not_auto_rebuild
-	 lsi = Classifier::LSI.new :auto_rebuild => false
-	 lsi.add_item @str1, "Dog"
-	 lsi.add_item @str2, "Dog"
-	 assert lsi.needs_rebuild?
-	 lsi.build_index
-	 assert ! lsi.needs_rebuild?
-	end
-
-	def test_basic_categorizing
-	  lsi = Classifier::LSI.new
-	  lsi.add_item @str2, "Dog"
-	  lsi.add_item @str3, "Cat"
-	  lsi.add_item @str4, "Cat"
-	  lsi.add_item @str5, "Bird"
-	  
-	  assert_equal "Dog", lsi.classify( @str1 )
-	  assert_equal "Cat", lsi.classify( @str3 )
-     assert_equal "Bird", lsi.classify( @str5 )  
-	end
-	
-	def test_external_classifying
-	  lsi = Classifier::LSI.new
-	  bayes = Classifier::Bayes.new 'Dog', 'Cat', 'Bird'
-	  lsi.add_item @str1, "Dog" ; bayes.train_dog @str1
-	  lsi.add_item @str2, "Dog" ; bayes.train_dog @str2
-	  lsi.add_item @str3, "Cat" ; bayes.train_cat @str3
-	  lsi.add_item @str4, "Cat" ; bayes.train_cat @str4
-	  lsi.add_item @str5, "Bird" ; bayes.train_bird @str5
-	  
-	  # We're talking about dogs. Even though the text matches the corpus on 
-	  # cats better.  Dogs have more semantic weight than cats. So bayes
-	  # will fail here, but the LSI recognizes content.
-	  tricky_case = "This text revolves around dogs."
-	  assert_equal "Dog", lsi.classify( tricky_case )
-	  assert_not_equal "Dog", bayes.classify( tricky_case )
-	end 
-	
-	def test_recategorize_interface
-	  lsi = Classifier::LSI.new
-	  lsi.add_item @str1, "Dog"
-	  lsi.add_item @str2, "Dog"
-	  lsi.add_item @str3, "Cat"
-	  lsi.add_item @str4, "Cat"
-	  lsi.add_item @str5, "Bird"
-	  
-	  tricky_case = "This text revolves around dogs."
-	  assert_equal "Dog", lsi.classify( tricky_case )
-	  
-	  # Recategorize as needed.
-	  lsi.categories_for(@str1).clear.push "Cow"
-	  lsi.categories_for(@str2).clear.push "Cow"
-
-	  assert !lsi.needs_rebuild?
-	  assert_equal "Cow", lsi.classify( tricky_case )	  
-	end
-	
-	def test_search
-	  lsi = Classifier::LSI.new
-	  [@str1, @str2, @str3, @str4, @str5].each { |x| lsi << x }
-	  
-	  # Searching by content and text, note that @str2 comes up first, because
-	  # both "dog" and "involve" are present. But, the next match is @str1 instead
-	  # of @str4, because "dog" carries more weight than involves. 
-	  assert_equal( [@str2, @str1, @str4, @str5, @str3],
-	                lsi.search("dog involves", 100) )
-	                
-	  # Keyword search shows how the space is mapped out in relation to 
-	  # dog when magnitude is remove. Note the relations. We move from dog
-	  # through involve and then finally to other words. 
-	  assert_equal( [@str1, @str2, @str4, @str5, @str3],
-	                lsi.search("dog", 5) )
-	end
-	
-	def test_serialize_safe
-    lsi = Classifier::LSI.new
-	  [@str1, @str2, @str3, @str4, @str5].each { |x| lsi << x }
-	  
-	  lsi_md = Marshal.dump lsi
-	  lsi_m = Marshal.load lsi_md
-	  
-	  assert_equal lsi_m.search("cat", 3), lsi.search("cat", 3)
-	  assert_equal lsi_m.find_related(@str1, 3), lsi.find_related(@str1, 3)
-	end
-	
-	def test_keyword_search
-	  lsi = Classifier::LSI.new
-	  lsi.add_item @str1, "Dog"
-	  lsi.add_item @str2, "Dog"
-	  lsi.add_item @str3, "Cat"
-	  lsi.add_item @str4, "Cat"
-	  lsi.add_item @str5, "Bird"
-	
-	  assert_equal [:dog, :text, :deal], lsi.highest_ranked_stems(@str1)
-	end
-	
-	def test_summary
-	   assert_equal "This text involves dogs too [...] This text also involves cats", [@str1, @str2, @str3, @str4, @str5].join.summary(2)
-	end
-	  
-end