stuff-classifier 0.1

data/.gitignore

@@ -0,0 +1,48 @@
+# rcov generated
+coverage
+
+# rdoc generated
+rdoc
+
+# yard generated
+doc
+.yardoc
+
+# bundler
+.bundle
+
+# jeweler generated
+pkg
+
+# Have editor/IDE/OS specific files you need to ignore? Consider using a global gitignore: 
+#
+# * Create a file at ~/.gitignore
+# * Include files you want ignored
+# * Run: git config --global core.excludesfile ~/.gitignore
+#
+# After doing this, these files will be ignored in all your git projects,
+# saving you from having to 'pollute' every project you touch with them
+#
+# Not sure what to needs to be ignored for particular editors/OSes? Here's some ideas to get you started. (Remember, remove the leading # of the line)
+#
+# For MacOS:
+#
+#.DS_Store
+
+# For TextMate
+#*.tmproj
+#tmtags
+
+# For emacs:
+#*~
+#\#*
+#.\#*
+
+# For vim:
+#*.swp
+
+# For redcar:
+#.redcar
+
+# For rubinius:
+#*.rbc

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,46 @@
+PATH
+  remote: .
+  specs:
+    stuff-classifier (0.1)
+      fast-stemmer (>= 1.0)
+
+GEM
+  remote: http://rubygems.org/
+  specs:
+    ansi (1.4.1)
+    archive-tar-minitar (0.5.2)
+    columnize (0.3.4)
+    fast-stemmer (1.0.0)
+    json (1.6.5)
+    linecache19 (0.5.12)
+      ruby_core_source (>= 0.1.4)
+    minitest (2.10.1)
+    rake (0.9.2.2)
+    rcov (0.9.11)
+    rdoc (3.12)
+      json (~> 1.4)
+    ruby-debug-base19 (0.11.25)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby_core_source (>= 0.1.4)
+    ruby-debug19 (0.11.6)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby-debug-base19 (>= 0.11.19)
+    ruby_core_source (0.1.5)
+      archive-tar-minitar (>= 0.5.2)
+    turn (0.8.3)
+      ansi
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler
+  minitest (>= 2.10)
+  rake (>= 0.9.2)
+  rcov (>= 0.9)
+  rdoc (>= 3.1)
+  ruby-debug19
+  stuff-classifier!
+  turn (>= 0.8.3)

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Alexandru Nedelcu
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,103 @@
+# stuff-classifier
+
+A library for classifying text into multiple categories.
+
+Currently provided classifiers:
+
+- a [naive bayes classifier](http://en.wikipedia.org/wiki/Naive_Bayes_classifier)
+- a classifier based on [tf-idf weights](http://en.wikipedia.org/wiki/Tf%E2%80%93idf)
+
+Ran a benchmark of 1345 items that I have previously manually
+classified with multiple categories. Here's the rate over which the 2
+algorithms have correctly detected one of those categories:
+
+- Bayes: 79.26%
+- Tf-Idf: 81.34%
+
+I prefer the Naive Bayes approach, because while having lower stats on
+this benchmark, it seems to make better decisions than I did in many
+cases. For example, an item with title *"Paintball Session, 100 Balls
+and Equipment"* was classified as *"Activities"* by me, but the bayes
+classifier identified it as *"Sports"*, at which point I had an
+intellectual orgasm. Also, the Tf-Idf classifier seems to do better on
+clear-cut cases, but doesn't seem to handle uncertainty so well. Of
+course, these are just quick tests I made and I have no idea which is
+really better.
+
+## Install
+
+```bash
+gem install stuff-classifier
+```
+
+## Usage
+
+You either instantiate one class or the other. Both have the same
+signature:
+
+```ruby
+require 'stuff-classifier'
+
+# for the naive bayes implementation
+cls = StuffClassifier::Bayes.new("Cats or Dogs")
+
+# for the Tf-Idf based implementation
+cls = StuffClassifier::TfIdf.new("Cats or Dogs")
+```
+
+Training the classifier:
+
+```ruby
+cls.train(:dog, "Dogs are awesome, cats too. I love my dog")
+cls.train(:cat, "Cats are more preferred by software developers. I never could stand cats. I have a dog")    
+cls.train(:dog, "My dog's name is Willy. He likes to play with my wife's cat all day long. I love dogs")
+cls.train(:cat, "Cats are difficult animals, unlike dogs, really annoying, I hate them all")
+cls.train(:dog, "So which one should you choose? A dog, definitely.")
+cls.train(:cat, "The favorite food for cats is bird meat, although mice are good, but birds are a delicacy")
+cls.train(:dog, "A dog will eat anything, including birds or whatever meat")
+cls.train(:cat, "My cat's favorite place to purr is on my keyboard")
+cls.train(:dog, "My dog's favorite place to take a leak is the tree in front of our house")
+```
+
+And finally, classifying stuff:
+
+```ruby
+cls.classify("This test is about cats.")
+#=> :cat
+cls.classify("I hate ...")
+#=> :cat
+cls.classify("The most annoying animal on earth.")
+#=> :cat
+cls.classify("The preferred company of software developers.")
+#=> :cat
+cls.classify("My precious, my favorite!")
+#=> :cat
+cls.classify("Kill that bird!")
+#=> :cat
+
+cls.classify("This test is about dogs.")
+#=> :dog
+cls.classify("Cats or Dogs?") 
+#=> :dog
+cls.classify("What pet will I love more?")    
+#=> :dog
+cls.classify("Willy, where the heck are you?")
+#=> :dog
+cls.classify("I like big buts and I cannot lie.") 
+#=> :dog
+cls.classify("Why is the front door of our house open?")
+#=> :dog
+cls.classify("Who is eating my meat?")
+#=> :dog
+```
+
+## TODO
+
+- provide more implementations
+- plugable storage mechanism (in-memory, on disk, database)
+
+## License
+
+MIT Licensed. See LICENSE.txt for details.
+
+

data/Rakefile

@@ -0,0 +1,30 @@
+require 'bundler/setup'
+require 'rake/testtask'
+require 'stuff-classifier'
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+require 'rcov/rcovtask'
+Rcov::RcovTask.new do |test|
+  test.libs << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+  test.rcov_opts << '--exclude "gems/*"'
+end
+
+require 'rdoc/task'
+RDoc::Task.new do |rdoc|
+  version = StuffClassifier::VERSION
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "stuff-classifier #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+task :default => :test
+

data/lib/stuff-classifier.rb

@@ -0,0 +1,8 @@
+module StuffClassifier
+  autoload :VERSION,    'stuff-classifier/version'
+  autoload :STOP_WORDS, 'stuff-classifier/stop_words'
+  autoload :Tokenizer,  'stuff-classifier/tokenizer'
+  autoload :Base,       'stuff-classifier/base'
+  autoload :Bayes,      'stuff-classifier/bayes'
+  autoload :TfIdf,      'stuff-classifier/tf-idf'
+end

data/lib/stuff-classifier/base.rb

@@ -0,0 +1,66 @@
+class StuffClassifier::Base
+  include StuffClassifier::Tokenizer
+  
+  def initialize(name, opts={})
+    @name = name
+    @stemming = opts.key?(:stemming) ? opts[:stemming] : true
+    @wcount = {}
+    @ccount = {}
+    @ignore_words = nil
+  end
+
+  def incr_word(word, category)
+    @wcount[word] ||= {}
+    @wcount[word][category] ||= 0
+    @wcount[word][category] += 1
+  end
+
+  def incr_cat(category)
+    @ccount[category] ||= 0
+    @ccount[category] += 1
+  end
+
+  def word_count(word, category)
+    return 0.0 unless @wcount[word] && @wcount[word][category]
+    @wcount[word][category].to_f
+  end
+  
+  def cat_count(category)
+    @ccount[category] ? @ccount[category].to_f : 0.0
+  end
+
+  def total_count
+    @ccount.values.inject(0){|s,c| s + c}.to_f
+  end
+  
+  def categories
+    @ccount.keys
+  end
+
+  def train(category, text)
+    each_word(text) {|w| incr_word(w, category) }
+    incr_cat(category)
+  end
+
+  def word_prob(word, cat)
+    return 0.0 if cat_count(cat) == 0
+    word_count(word, cat) / cat_count(cat)
+  end
+
+  def word_weighted_average(word, cat, opts={})
+    func = opts[:func]
+    weight = opts[:weight] || 1.0
+    assumed_prob = opts[:assumed_prob] || 0.5
+
+    # calculate current probability
+    basic_prob = func ? func.call(word, cat) 
+      : word_prob(word, cat)
+    
+    # count the number of times this word has appeared in all
+    # categories
+    totals = categories.map{|c| word_count(word, c)}.inject(0){|s,c| s + c}
+    
+    # the final weighted average
+    (weight * assumed_prob + totals * basic_prob) / (weight + totals)
+  end
+end

data/lib/stuff-classifier/bayes.rb

@@ -0,0 +1,55 @@
+
+class StuffClassifier::Bayes < StuffClassifier::Base
+  # http://en.wikipedia.org/wiki/Naive_Bayes_classifier
+
+  def initialize(name, opts={})
+    super(name, opts)
+    @thresholds = {}
+  end
+
+  def doc_prob(text, category)
+    each_word(text).map {|w|
+      word_weighted_average(w, category)
+    }.inject(1) {|p,c| p * c}
+  end
+
+  def text_prob(text, category)
+    cat_prob = cat_count(category) / total_count
+    doc_prob = doc_prob(text, category)
+    cat_prob * doc_prob
+  end
+
+  def cat_scores(text)
+    probs = {}
+    categories.each do |cat|
+      probs[cat] = text_prob(text, cat)
+    end
+    probs.map{|k,v| [k,v]}.sort{|a,b| b[1] <=> a[1]}
+  end
+
+  def classify(text, default=nil)
+    # Find the category with the highest probability
+    max_prob = 0.0
+    best = nil
+    
+    scores = cat_scores(text)
+    scores.each do |score|
+      cat, prob = score
+      if prob > max_prob
+        max_prob = prob
+        best = cat
+      end
+    end
+
+    return default unless best
+    threshold = @thresholds[best] || 1.0
+
+    scores.each do |score|
+      cat, prob = score
+      next if cat == best
+      return default if prob * threshold > max_prob
+    end
+
+    return best
+  end
+end

data/lib/stuff-classifier/stop_words.rb

@@ -0,0 +1,55 @@
+StuffClassifier::STOP_WORDS = Set.new [
+    'a', 'about', 'above', 'across', 'after', 'afterwards', 
+    'again', 'against', 'all', 'almost', 'alone', 'along', 
+    'already', 'also', 'although', 'always', 'am', 'among', 
+    'amongst', 'amoungst', 'amount', 'an', 'and', 'another', 
+    'any', 'anyhow', 'anyone', 'anything', 'anyway', 'anywhere', 
+    'are', 'around', 'as', 'at', 'back', 'be', 
+    'became', 'because', 'become', 'becomes', 'becoming', 'been', 
+    'before', 'beforehand', 'behind', 'being', 'below', 'beside', 
+    'besides', 'between', 'beyond', 'bill', 'both', 'bottom', 
+    'but', 'by', 'call', 'can', 'cannot', 'cant', 'dont',
+    'co', 'computer', 'con', 'could', 'couldnt', 'cry', 
+    'de', 'describe', 'detail', 'do', 'done', 'down', 
+    'due', 'during', 'each', 'eg', 'eight', 'either', 
+    'eleven', 'else', 'elsewhere', 'empty', 'enough', 'etc', 'even', 'ever', 'every', 
+    'everyone', 'everything', 'everywhere', 'except', 'few', 'fifteen', 
+    'fify', 'fill', 'find', 'fire', 'first', 'five', 
+    'for', 'former', 'formerly', 'forty', 'found', 'four', 
+    'from', 'front', 'full', 'further', 'get', 'give', 
+    'go', 'had', 'has', 'hasnt', 'have', 'he', 
+    'hence', 'her', 'here', 'hereafter', 'hereby', 'herein', 
+    'hereupon', 'hers', 'herself', 'him', 'himself', 'his', 
+    'how', 'however', 'hundred', 'i', 'ie', 'if', 
+    'in', 'inc', 'indeed', 'interest', 'into', 'is', 
+    'it', 'its', 'itself', 'keep', 'last', 'latter', 
+    'latterly', 'least', 'less', 'ltd', 'made', 'many', 
+    'may', 'me', 'meanwhile', 'might', 'mill', 'mine', 
+    'more', 'moreover', 'most', 'mostly', 'move', 'much', 
+    'must', 'my', 'myself', 'name', 'namely', 'neither', 
+    'never', 'nevertheless', 'next', 'nine', 'no', 'nobody', 
+    'none', 'noone', 'nor', 'not', 'nothing', 'now', 
+    'nowhere', 'of', 'off', 'often', 'on', 'once', 
+    'one', 'only', 'onto', 'or', 'other', 'others', 
+    'otherwise', 'our', 'ours', 'ourselves', 'out', 'over', 
+    'own', 'part', 'per', 'perhaps', 'please', 'put', 
+    'rather', 're', 'same', 'see', 'seem', 'seemed', 
+    'seeming', 'seems', 'serious', 'several', 'she', 'should', 
+    'show', 'side', 'since', 'sincere', 'six', 'sixty', 
+    'so', 'some', 'somehow', 'someone', 'something', 'sometime', 
+    'sometimes', 'somewhere', 'still', 'such', 'system', 'take', 
+    'ten', 'than', 'that', 'the', 'their', 'them', 
+    'themselves', 'then', 'thence', 'there', 'thereafter', 'thereby', 
+    'therefore', 'therein', 'thereupon', 'these', 'they', 'thick', 
+    'thin', 'third', 'this', 'those', 'though', 'three', 
+    'through', 'throughout', 'thru', 'thus', 'to', 'together', 
+    'too', 'top', 'toward', 'towards', 'twelve', 'twenty', 
+    'two', 'un', 'under', 'until', 'up', 'upon', 
+    'us', 'very', 'via', 'was', 'we', 'well', 
+    'were', 'what', 'whatever', 'when', 'whence', 'whenever', 
+    'where', 'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon', 
+    'wherever', 'whether', 'which', 'while', 'whither', 'who', 
+    'whoever', 'whole', 'whom', 'whose', 'why', 'will', 
+    'with', 'within', 'without', 'would', 'yet', 'you', 'your', 'yours', 
+    'yourself', 'yourselves'
+]

data/lib/stuff-classifier/tf-idf.rb

@@ -0,0 +1,41 @@
+class StuffClassifier::TfIdf < StuffClassifier::Base
+  def tf_idf(word, cat)
+    word_cat_nr = word_count(word, cat)
+    cat_nr = cat_count(cat)
+    tf = 1.0 * word_cat_nr / cat_nr
+    
+    total_categories = categories.length
+    categories_with_word = (@wcount[word] || []).length
+
+    idf = Math.log((total_categories + 2) / (categories_with_word + 1.0), 10)    
+    return tf * idf
+  end
+
+  def text_prob(text, cat)
+    each_word(text).map{|w| tf_idf(w, cat)}.inject(0){|s,p| s + p}
+  end
+
+  def cat_scores(text)
+    probs = {}
+    categories.each do |cat|
+      p = text_prob(text, cat)
+      probs[cat] = p
+    end
+    probs.map{|k,v| [k,v]}.sort{|a,b| b[1] <=> a[1]}
+  end
+
+  def classify(text, default=nil)
+    max_prob = 0.0
+    best = nil
+
+    cat_scores(text).each do |score|
+      cat, prob = score
+      if prob > max_prob
+        max_prob = prob
+        best = cat
+      end
+    end
+
+    max_prob > 0 ? best : default
+  end
+end

data/lib/stuff-classifier/tokenizer.rb

@@ -0,0 +1,48 @@
+require 'fast_stemmer'
+
+module StuffClassifier::Tokenizer
+
+  def ignore_words=(value)
+    @ignore_words = value
+  end
+
+  def ignore_words
+    @ignore_words || StuffClassifier::STOP_WORDS
+  end
+
+  def stemming?
+    defined?(@stemming) ? @stemming : false
+  end
+
+  def stemming=(value)
+    @stemming = value
+  end
+
+  def each_word(string)
+    string = string.strip
+    return if string == ''
+
+    words = []
+    
+    cnt = string.gsub(/['`]/, '')
+    cnt.split("\n").each do |line|
+      line_cnt = line.gsub(/[^a-zA-Z]+/, ' ')
+      line_cnt.split(/\s+/).each do |w|
+        next if w == '' || ignore_words.member?(w.downcase)
+
+        if stemming?
+          w = w.stem.downcase
+          next if ignore_words.member?(w)
+        else
+          w = w.downcase
+        end
+
+        yield w if block_given?
+        words << w
+      end
+    end
+
+    return words
+  end
+
+end

data/lib/stuff-classifier/version.rb

@@ -0,0 +1,3 @@
+module StuffClassifier
+  VERSION = '0.1'
+end

data/stuff-classifier.gemspec

@@ -0,0 +1,29 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "stuff-classifier/version"
+
+Gem::Specification.new do |s|
+  s.name        = "stuff-classifier"
+  s.version     = StuffClassifier::VERSION
+  s.authors     = ["Alexandru Nedelcu"]
+  s.email       = ["me@alexn.org"]
+  s.homepage    = "https://github.com/alexandru/stuff-classifier/"
+  s.summary     = %q{Simple text classifier(s) implemetation}
+  s.description = %q{2 methods are provided for now - (1) naive bayes implementation + (2) tf-idf weights}
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  s.add_runtime_dependency "fast-stemmer", ">= 1.0"
+
+  s.add_development_dependency "bundler"
+  s.add_development_dependency "rake", ">= 0.9.2"
+  s.add_development_dependency "minitest", ">= 2.10"
+  s.add_development_dependency "turn", ">= 0.8.3"
+  s.add_development_dependency "rdoc", ">= 3.1"
+  s.add_development_dependency "rcov", ">= 0.9"
+  s.add_development_dependency "ruby-debug19"
+end
+

data/test/helper.rb

@@ -0,0 +1,42 @@
+require 'minitest/autorun'
+require 'stuff-classifier'
+require 'turn'
+
+Turn.config do |c|
+ # use one of output formats:
+ # :outline  - turn's original case/test outline mode [default]
+ # :progress - indicates progress with progress bar
+ # :dotted   - test/unit's traditional dot-progress mode
+ # :pretty   - new pretty reporter
+ # :marshal  - dump output as YAML (normal run mode only)
+ # :cue      - interactive testing
+ c.format  = :outline
+ # turn on invoke/execute tracing, enable full backtrace
+ c.trace   = true
+ # use humanized test names (works only with :outline format)
+ c.natural = true
+end
+
+class TestBase < MiniTest::Unit::TestCase
+  def self.before(&block)    
+    @on_setup = block if block
+    @on_setup
+  end
+
+  def setup
+    on_setup = self.class.before
+    instance_eval(&on_setup) if on_setup
+  end
+
+  def set_classifier(instance)
+    @classifier = instance
+  end
+
+  def train(category, value)
+    @classifier.train(category, value)
+  end
+
+  def should_be(category, value)
+    assert_equal category, @classifier.classify(value), value
+  end
+end

data/test/test_001_tokenizer.rb

@@ -0,0 +1,39 @@
+require 'helper'
+
+class Test001Tokenizer < TestBase
+  before do
+    tokenizer_cls = Class.new do
+      include StuffClassifier::Tokenizer
+    end
+
+    @tokenizer = tokenizer_cls.new
+  end
+
+  def test_simple_tokens
+    assert_equal ["hello", "world"], 
+      @tokenizer.each_word('Hello world! How are you?')
+  end    
+
+  def test_with_stemming
+    @tokenizer.stemming = true
+    assert_equal(
+       ["lot", "dog", "lot", "cat", "inform", "highwai"], 
+       @tokenizer.each_word('Lots of dogs, lots of cats! This is the information highway')
+    )
+  end
+
+  def test_complicated_tokens 
+    words = @tokenizer.each_word("I don't really get what you want to
+      accomplish. There is a class TestEval2, you can do test_eval2 =
+      TestEval2.new afterwards. And: class A ... end always yields nil, so
+      your output is ok I guess ;-)")
+
+    should_return = [
+      "really", "want", "accomplish", "class",
+      "testeval", "test", "eval", "testeval", "new", "class", "end",
+      "yields", "nil", "output", "ok", "guess"]
+
+    assert_equal should_return, words
+  end  
+
+end

data/test/test_002_naive_bayes.rb

@@ -0,0 +1,37 @@
+require 'helper'
+
+
+class Test002NaiveBayesClassification < TestBase
+  before do
+    set_classifier StuffClassifier::Bayes.new("Cats or Dogs")
+    
+    train :dog, "Dogs are awesome, cats too. I love my dog"
+    train :cat, "Cats are more preferred by software developers. I never could stand cats. I have a dog"    
+    train :dog, "My dog's name is Willy. He likes to play with my wife's cat all day long. I love dogs"
+    train :cat, "Cats are difficult animals, unlike dogs, really annoying, I hate them all"
+    train :dog, "So which one should you choose? A dog, definitely."
+    train :cat, "The favorite food for cats is bird meat, although mice are good, but birds are a delicacy"
+    train :dog, "A dog will eat anything, including birds or whatever meat"
+    train :cat, "My cat's favorite place to purr is on my keyboard"
+    train :dog, "My dog's favorite place to take a leak is the tree in front of our house"
+  end
+
+  def test_for_cats 
+    should_be :cat, "This test is about cats."
+    should_be :cat, "I hate ..."
+    should_be :cat, "The most annoying animal on earth."
+    should_be :cat, "The preferred company of software developers."
+    should_be :cat, "My precious, my favorite!"
+    should_be :cat, "Kill that bird!"
+  end
+
+  def test_for_dogs
+    should_be :dog, "This test is about dogs."
+    should_be :dog, "Cats or Dogs?" 
+    should_be :dog, "What pet will I love more?"    
+    should_be :dog, "Willy, where the heck are you?"
+    should_be :dog, "I like big buts and I cannot lie." 
+    should_be :dog, "Why is the front door of our house open?"
+    should_be :dog, "Who ate my meat?"
+  end
+end

data/test/test_003_tf_idf.rb

@@ -0,0 +1,37 @@
+require 'helper'
+
+
+class Test003TfIdfClassification < TestBase
+  before do
+    set_classifier StuffClassifier::TfIdf.new("Cats or Dogs")
+    
+    train :dog, "Dogs are awesome, cats too. I love my dog"
+    train :cat, "Cats are more preferred by software developers. I never could stand cats. I have a dog"    
+    train :dog, "My dog's name is Willy. He likes to play with my wife's cat all day long. I love dogs"
+    train :cat, "Cats are difficult animals, unlike dogs, really annoying, I hate them all"
+    train :dog, "So which one should you choose? A dog, definitely."
+    train :cat, "The favorite food for cats is bird meat, although mice are good, but birds are a delicacy"
+    train :dog, "A dog will eat anything, including birds or whatever meat"
+    train :cat, "My cat's favorite place to purr is on my keyboard"
+    train :dog, "My dog's favorite place to take a leak is the tree in front of our house"
+  end
+
+  def test_for_cats 
+    should_be :cat, "This test is about cats."
+    should_be :cat, "I hate ..."
+    should_be :cat, "The most annoying animal on earth."
+    should_be :cat, "The preferred company of software developers."
+    should_be :cat, "My precious, my favorite!"
+    should_be :cat, "Kill that bird!"
+  end
+
+  def test_for_dogs
+    should_be :dog, "This test is about dogs."
+    should_be :dog, "Cats or Dogs?" 
+    should_be :dog, "What pet will I love more?"    
+    should_be :dog, "Willy, where the heck are you?"
+    should_be :dog, "I like big buts and I cannot lie." 
+    should_be :dog, "Why is the front door of our house open?"
+    should_be :dog, "Who is eating my meat?"
+  end
+end

metadata

@@ -0,0 +1,152 @@
+--- !ruby/object:Gem::Specification
+name: stuff-classifier
+version: !ruby/object:Gem::Version
+  version: '0.1'
+  prerelease: 
+platform: ruby
+authors:
+- Alexandru Nedelcu
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-01-19 00:00:00.000000000Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fast-stemmer
+  requirement: &71035640 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *71035640
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: &71096090 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *71096090
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &71095820 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 0.9.2
+  type: :development
+  prerelease: false
+  version_requirements: *71095820
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: &71095570 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '2.10'
+  type: :development
+  prerelease: false
+  version_requirements: *71095570
+- !ruby/object:Gem::Dependency
+  name: turn
+  requirement: &71095340 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 0.8.3
+  type: :development
+  prerelease: false
+  version_requirements: *71095340
+- !ruby/object:Gem::Dependency
+  name: rdoc
+  requirement: &71095110 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '3.1'
+  type: :development
+  prerelease: false
+  version_requirements: *71095110
+- !ruby/object:Gem::Dependency
+  name: rcov
+  requirement: &71094880 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0.9'
+  type: :development
+  prerelease: false
+  version_requirements: *71094880
+- !ruby/object:Gem::Dependency
+  name: ruby-debug19
+  requirement: &71094690 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *71094690
+description: 2 methods are provided for now - (1) naive bayes implementation + (2)
+  tf-idf weights
+email:
+- me@alexn.org
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- Gemfile.lock
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/stuff-classifier.rb
+- lib/stuff-classifier/base.rb
+- lib/stuff-classifier/bayes.rb
+- lib/stuff-classifier/stop_words.rb
+- lib/stuff-classifier/tf-idf.rb
+- lib/stuff-classifier/tokenizer.rb
+- lib/stuff-classifier/version.rb
+- stuff-classifier.gemspec
+- test/helper.rb
+- test/test_001_tokenizer.rb
+- test/test_002_naive_bayes.rb
+- test/test_003_tf_idf.rb
+homepage: https://github.com/alexandru/stuff-classifier/
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.6
+signing_key: 
+specification_version: 3
+summary: Simple text classifier(s) implemetation
+test_files: []