stuff-classifier-chinese 0.51

data/.gitignore

@@ -0,0 +1,6 @@
+.rvmrc
+coverage/
+.DS_Store
+*.gem
+utils.rb
+Gemfile.lock

data/Gemfile

@@ -0,0 +1,3 @@
+source "http://rubygems.org"
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Alexandru Nedelcu
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,162 @@
+# stuff-classifier
+
+A library for classifying text into multiple categories.
+
+Currently provided classifiers:
+
+- a [naive bayes classifier](http://en.wikipedia.org/wiki/Naive_Bayes_classifier)
+- a classifier based on [tf-idf weights](http://en.wikipedia.org/wiki/Tf%E2%80%93idf)
+
+Ran a benchmark of 1345 items that I have previously manually
+classified with multiple categories. Here's the rate over which the 2
+algorithms have correctly detected one of those categories:
+
+- Bayes: 79.26%
+- Tf-Idf: 81.34%
+
+I prefer the Naive Bayes approach, because while having lower stats on
+this benchmark, it seems to make better decisions than I did in many
+cases. For example, an item with title *"Paintball Session, 100 Balls
+and Equipment"* was classified as *"Activities"* by me, but the bayes
+classifier identified it as *"Sports"*, at which point I had an
+intellectual orgasm. Also, the Tf-Idf classifier seems to do better on
+clear-cut cases, but doesn't seem to handle uncertainty so well. Of
+course, these are just quick tests I made and I have no idea which is
+really better.
+
+## Install
+
+```bash
+gem install stuff-classifier
+```
+
+## Usage
+
+You either instantiate one class or the other. Both have the same
+signature:
+
+```ruby
+require 'stuff-classifier'
+
+# for the naive bayes implementation
+cls = StuffClassifier::Bayes.new("Cats or Dogs")
+
+# for the Tf-Idf based implementation
+cls = StuffClassifier::TfIdf.new("Cats or Dogs")
+
+# these classifiers use word stemming by default, but if it has weird
+# behavior, then you can disable it on init:
+cls = StuffClassifier::TfIdf.new("Cats or Dogs", :stemming => false)
+
+# also by default, the parsing phase filters out stop words, to
+# disable or to come up with your own list of stop words, on a
+# classifier instance you can do this:
+cls.ignore_words = [ 'the', 'my', 'i', 'dont' ]
+ ```
+
+Training the classifier:
+
+```ruby
+cls.train(:dog, "Dogs are awesome, cats too. I love my dog")
+cls.train(:cat, "Cats are more preferred by software developers. I never could stand cats. I have a dog")    
+cls.train(:dog, "My dog's name is Willy. He likes to play with my wife's cat all day long. I love dogs")
+cls.train(:cat, "Cats are difficult animals, unlike dogs, really annoying, I hate them all")
+cls.train(:dog, "So which one should you choose? A dog, definitely.")
+cls.train(:cat, "The favorite food for cats is bird meat, although mice are good, but birds are a delicacy")
+cls.train(:dog, "A dog will eat anything, including birds or whatever meat")
+cls.train(:cat, "My cat's favorite place to purr is on my keyboard")
+cls.train(:dog, "My dog's favorite place to take a leak is the tree in front of our house")
+```
+
+And finally, classifying stuff:
+
+```ruby
+cls.classify("This test is about cats.")
+#=> :cat
+cls.classify("I hate ...")
+#=> :cat
+cls.classify("The most annoying animal on earth.")
+#=> :cat
+cls.classify("The preferred company of software developers.")
+#=> :cat
+cls.classify("My precious, my favorite!")
+#=> :cat
+cls.classify("Get off my keyboard!")
+#=> :cat
+cls.classify("Kill that bird!")
+#=> :cat
+
+cls.classify("This test is about dogs.")
+#=> :dog
+cls.classify("Cats or Dogs?") 
+#=> :dog
+cls.classify("What pet will I love more?")    
+#=> :dog
+cls.classify("Willy, where the heck are you?")
+#=> :dog
+cls.classify("I like big buts and I cannot lie.") 
+#=> :dog
+cls.classify("Why is the front door of our house open?")
+#=> :dog
+cls.classify("Who is eating my meat?")
+#=> :dog
+```
+
+## Persistency
+
+The following layers for saving the training data between sessions are
+implemented:
+
+- in memory (by default)
+- on disk
+- Redis
+- (coming soon) in a RDBMS
+
+To persist the data in Redis, you can do this:
+```ruby
+# defaults to redis running on localhost on default port
+store = StuffClassifier::RedisStorage.new(@key)
+
+# pass in connection args
+store = StuffClassifier::RedisStorage.new(@key, {host:'my.redis.server.com', port: 4829})
+```
+
+To persist the data on disk, you can do this:
+
+```ruby
+store = StuffClassifier::FileStorage.new(@storage_path)
+
+# global setting
+StuffClassifier::Base.storage = store
+
+# or alternative local setting on instantiation, by means of an
+# optional param ...
+cls = StuffClassifier::Bayes.new("Cats or Dogs", :storage => store)
+
+# after training is done, to persist the data ...
+cls.save_state
+
+# or you could just do this:
+StuffClassifier::Bayes.open("Cats or Dogs") do |cls|
+  # when done, save_state is called on END
+end
+
+# to start fresh, deleting the saved training data for this classifier
+StuffClassifier::Bayes.new("Cats or Dogs", :purge_state => true)
+```
+
+The name you give your classifier is important, as based on it the
+data will get loaded and saved. For instance, following 3 classifiers
+will be stored in different buckets, being independent of each other.
+
+```ruby
+cls1 = StuffClassifier::Bayes.new("Cats or Dogs")
+cls2 = StuffClassifier::Bayes.new("True or False")
+cls3 = StuffClassifier::Bayes.new("Spam or Ham")	
+```
+
+## License
+
+MIT Licensed. See LICENSE.txt for details.
+
+

data/Rakefile

@@ -0,0 +1,12 @@
+require 'bundler/setup'
+require 'rake/testtask'
+require 'stuff-classifier'
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test
+

data/lib/stuff-classifier.rb

@@ -0,0 +1,17 @@
+# -*- encoding : utf-8 -*-
+module StuffClassifier
+  autoload :VERSION,    'stuff-classifier/version'
+
+  autoload :Storage, 'stuff-classifier/storage'
+  autoload :InMemoryStorage, 'stuff-classifier/storage'
+  autoload :FileStorage,     'stuff-classifier/storage'
+  autoload :RedisStorage, 'stuff-classifier/storage'
+
+  autoload :Tokenizer,  'stuff-classifier/tokenizer'
+  autoload :TOKENIZER_PROPERTIES, 'stuff-classifier/tokenizer/tokenizer_properties'
+
+  autoload :Base,       'stuff-classifier/base'
+  autoload :Bayes,      'stuff-classifier/bayes'
+  autoload :TfIdf,      'stuff-classifier/tf-idf'
+
+end

data/lib/stuff-classifier/base.rb

@@ -0,0 +1,190 @@
+# -*- encoding : utf-8 -*-
+
+class StuffClassifier::Base
+  extend StuffClassifier::Storage::ActAsStorable
+  attr_reader :name
+  attr_reader :word_list
+  attr_reader :category_list
+  attr_reader :training_count
+
+  attr_accessor :tokenizer
+  attr_accessor :language
+  
+  attr_accessor :thresholds
+  attr_accessor :min_prob
+
+
+  storable :version,:word_list,:category_list,:training_count,:thresholds,:min_prob
+    
+  # opts :
+  # language
+  # stemming : true | false
+  # weight
+  # assumed_prob
+  # storage
+  # purge_state ?
+
+  def initialize(name, opts={})
+    @version = StuffClassifier::VERSION
+    
+    @name = name
+
+    # This values are nil or are loaded from storage
+    @word_list = {}
+    @category_list = {}
+    @training_count=0
+
+    # storage
+    purge_state = opts[:purge_state]
+    @storage = opts[:storage] || StuffClassifier::Base.storage
+    unless purge_state
+      @storage.load_state(self)
+    else
+      @storage.purge_state(self)
+    end
+
+    # This value can be set during initialization or overrided after load_state
+    @thresholds = opts[:thresholds] || {}
+    @min_prob = opts[:min_prob] || 0.0
+    
+
+    @ignore_words = nil
+    @tokenizer = StuffClassifier::Tokenizer.new(opts)
+    
+  end
+
+  def incr_word(word, category)
+    @word_list[word] ||= {}
+
+    @word_list[word][:categories] ||= {}
+    @word_list[word][:categories][category] ||= 0
+    @word_list[word][:categories][category] += 1
+
+    @word_list[word][:_total_word] ||= 0
+    @word_list[word][:_total_word] += 1
+
+  
+    # words count by categroy
+    @category_list[category] ||= {}
+    @category_list[category][:_total_word] ||= 0
+    @category_list[category][:_total_word] += 1
+
+  end
+
+  def incr_cat(category)
+    @category_list[category] ||= {}
+    @category_list[category][:_count] ||= 0
+    @category_list[category][:_count] += 1
+
+    @training_count ||= 0
+    @training_count += 1 
+
+  end
+
+  # return number of times the word appears in a category
+  def word_count(word, category)
+    return 0.0 unless @word_list[word] && @word_list[word][:categories] && @word_list[word][:categories][category]
+    @word_list[word][:categories][category].to_f
+  end
+
+  # return the number of times the word appears in all categories
+  def total_word_count(word)
+    return 0.0 unless @word_list[word] && @word_list[word][:_total_word]
+    @word_list[word][:_total_word].to_f
+  end
+
+  # return the number of words in a categories
+  def total_word_count_in_cat(cat)
+    return 0.0 unless @category_list[cat] && @category_list[cat][:_total_word]
+    @category_list[cat][:_total_word].to_f
+  end
+
+  # return the number of training item 
+  def total_cat_count
+    @training_count
+  end
+  
+  # return the number of training document for a category
+  def cat_count(category)
+    @category_list[category][:_count] ? @category_list[category][:_count].to_f : 0.0
+  end
+
+  # return the number of time categories in wich a word appear
+  def categories_with_word_count(word)
+    return 0 unless @word_list[word] && @word_list[word][:categories]
+    @word_list[word][:categories].length 
+  end  
+
+  # return the number of categories
+  def total_categories
+    categories.length
+  end
+
+  # return categories list
+  def categories
+    @category_list.keys
+  end
+
+  # train the classifier
+  def train(category, text)
+    @tokenizer.each_word(text) {|w| incr_word(w, category) }
+    incr_cat(category)
+  end
+
+  # classify a text
+  def classify(text, default=nil)
+    # Find the category with the highest probability
+    max_prob = @min_prob
+    best = nil
+
+    scores = cat_scores(text)
+    scores.each do |score|
+      cat, prob = score
+      if prob > max_prob
+        max_prob = prob
+        best = cat
+      end
+    end
+    
+    # Return the default category in case the threshold condition was
+    # not met. For example, if the threshold for :spam is 1.2
+    #
+    #    :spam => 0.73, :ham => 0.40  (OK)
+    #    :spam => 0.80, :ham => 0.70  (Fail, :ham is too close)
+
+    return default unless best
+
+    threshold = @thresholds[best] || 1.0
+
+    scores.each do |score|
+      cat, prob = score
+      next if cat == best
+      return default if prob * threshold > max_prob
+    end
+
+    return best    
+  end
+
+  def save_state
+    @storage.save_state(self)
+  end
+
+  class << self
+    attr_writer :storage
+
+    def storage
+      @storage = StuffClassifier::InMemoryStorage.new unless defined? @storage
+      @storage
+    end
+
+    def open(name)
+      inst = self.new(name)
+      if block_given?
+        yield inst
+        inst.save_state
+      else
+        inst
+      end
+    end
+  end
+end

data/lib/stuff-classifier/bayes.rb

@@ -0,0 +1,81 @@
+# -*- encoding : utf-8 -*-
+
+class StuffClassifier::Bayes < StuffClassifier::Base
+  attr_accessor :weight
+  attr_accessor :assumed_prob
+
+
+  # http://en.wikipedia.org/wiki/Naive_Bayes_classifier
+  extend StuffClassifier::Storage::ActAsStorable
+  storable :weight,:assumed_prob
+
+  def initialize(name, opts={})
+    super(name, opts)
+    @weight = opts[:weight] || 1.0
+    @assumed_prob = opts[:assumed_prob] || 0.1
+  end
+
+  def word_prob(word, cat)
+    total_words_in_cat = total_word_count_in_cat(cat)
+    return 0.0 if total_words_in_cat == 0
+    word_count(word, cat).to_f / total_words_in_cat
+  end
+
+
+  def word_weighted_average(word, cat, opts={})
+    func = opts[:func]
+
+    # calculate current probability
+    basic_prob = func ? func.call(word, cat) : word_prob(word, cat)
+    
+    # count the number of times this word has appeared in all
+    # categories
+    totals = total_word_count(word)
+    
+    # the final weighted average
+    (@weight * @assumed_prob + totals * basic_prob) / (@weight + totals)
+  end
+
+  def doc_prob(text, category)
+    @tokenizer.each_word(text).map {|w|
+      word_weighted_average(w, category)
+    }.inject(1) {|p,c| p * c}
+  end
+
+  def text_prob(text, category)
+    cat_prob = cat_count(category) / total_cat_count
+    doc_prob = doc_prob(text, category)
+    cat_prob * doc_prob
+  end
+
+  def cat_scores(text)
+    probs = {}
+    categories.each do |cat|
+      probs[cat] = text_prob(text, cat)
+    end
+    probs.map{|k,v| [k,v]}.sort{|a,b| b[1] <=> a[1]}
+  end
+
+
+  def word_classification_detail(word)
+
+    p "word_prob"
+    result=self.categories.inject({}) do |h,cat| h[cat]=self.word_prob(word,cat);h end
+    p result
+
+    p "word_weighted_average"
+    result=categories.inject({}) do |h,cat| h[cat]=word_weighted_average(word,cat);h end  
+    p result
+
+    p "doc_prob"
+    result=categories.inject({}) do |h,cat| h[cat]=doc_prob(word,cat);h end  
+    p result
+
+    p "text_prob"
+    result=categories.inject({}) do |h,cat| h[cat]=text_prob(word,cat);h end  
+    p result
+    
+    
+  end
+
+end