green_midget 0.0.2 → 0.0.3

data/.gitignore

@@ -2,3 +2,5 @@
 .bundle
 Gemfile.lock
 pkg/*
+.rvmrc
+.rspec

data/Gemfile

@@ -1,19 +1,4 @@
 # Copyright (c) 2011, SoundCloud Ltd., Nikola Chochkov
 source "http://rubygems.org"
-# Add dependencies required to use your gem here.
-# Example:
-#   gem "activesupport", ">= 2.3.5"
 
-gem "activerecord"
-
-# remove this dependency after testing.
-gem 'jberkel-mysql-ruby',    '= 2.8.1',   :require => 'mysql' # Ruby 1.9 fixes
-
-# Add dependencies to develop your gem here.
-# Include everything needed to run rake, tests, features, etc.
-group :development do
-  gem "rspec", "~> 2.3.0"
-  gem "bundler", "~> 1.0.0"
-  gem "jeweler", "~> 1.5.2"
-  gem "rcov", ">= 0"
-end
+gemspec

data/README.md

@@ -1,6 +1,8 @@
 On Bayesian Classification
 ----------
 
+This project started during an internship at SoundCloud.
+
 Using SoundCloud's private messaging means that you can effectively reach out to everyone on the Cloud. On top of that, you have track commenting, groups posting, forum topics, track sharing - we care about your voice being heard! And read.
 
 I'll put this in some perspective and say that we're now having daily text exchange volume in the order of hundreds of thousands. And it's also rapidly going up.
@@ -23,48 +25,40 @@ to your Gemfile and run
 
     bundle install
 
-then add
-
-    require 'green_midget'
-
-to your Rakefile and run
+after which (so that you get the ActiveRecord backend ready):
 
-    rake green_midget:setup
+    rake green_midget:setup:active_record # creates a green_midget_records table and populate some entried there
 
 You're now done.
 
 How it works
 ----------
 
-GreenMidget
-
-GreenMidget is a learner, so you will only expect effective classification from it only once it has received sufficient training. Training it means providing examples of messages for the categor  
-
-
+GreenMidget learns to classify between two categories, so what you should first do is provide training examples for each of those two categories. See below.
 
 Use it
 ----------
 
-`GreenMidget` exposes two public methods as a start: `GreenMidget#classify_as!` and `GreenMidget#classify`. Let's do a three lines classification session and illustrate them
+`GreenMidget::Classifier` is the interaction class that is there after installation. It exposes two public instance methods as a start: `GreenMidget::Classifier#classify_as!` and `GreenMidget::Classifier#classify`. We'll do a three lines classification session and illustrate them.
 
-We'll start training `GreenMidget` with a spammy example
+We'll start training `GreenMidget` with a spammy example:
 
-    GreenMidget.new(known_spam_text).classify_as! :spam
+    GreenMidget::Classifier.new(known_spam_text).classify_as! :spam
 
 Similarly for legitimate examples
 
-    GreenMidget.new(known_legit_text).classify_as! :ham
+    GreenMidget::Classifier.new(known_legit_text).classify_as! :ham
 
-To get a classification decision we would
+After we've given to it some training data, we can start classifying unknown text:
 
-    decision = GreenMidget.new(new_text).classify
+    decision = GreenMidget::Classifier.new(new_text).classify
 
-`decision` is now one of `[-1, 0, 1]` meaning respectively 'No spam', 'Not enough evidence', 'Spam'.
+`decision` is now in `[ -1, 0, 1 ]` meaning respectively 'No spam', 'Not enough evidence', 'Spam'.
 
 Extend it
 ----------
 
-If the above functionality is not enough for you and you want to add custom logic to GreenMidget you can do that by extending the `GreenMidget::Base` class (check `extensions/sample.b` in the [code][green_midget_github] for an example):
+If the above functionality is not enough for you and you want to add custom logic to GreenMidget you can do that by extending the `GreenMidget::Base` class (check `lib/green_midget/extensions/sample.b` in the [code][green_midget_github] for an example):
 
 * Implement heuristics logic, which will directly classify incoming object as a given category. Example:
 
@@ -99,7 +93,9 @@ If that's not enough too, you're welcome to [browse the code][green_midget_githu
 Benchmarking
 ----------
 
-1. GreenMidget is optimised for classification operations (`classify` method), on which it's very efficient. The results below were obtained from classification on randomly generated messages of length _1 000 words_ (that's _very_ long for SoundCloud). Since GreenMidget runs on a relational database (through ActiveRecord) by default the table size impacts data fetch and write:
+Before moving on, let's say that `GreenMidget` is intended for asynchronous spam checks. Using ActiveRecord as backend has the benefit of wide support and easy setup, but as it also means that the time performance will become progressively worse the more training you provide.
+
+1. GreenMidget is optimised for classification operations (`classify` method), on which it's relatively efficient. The results below were obtained from classification on randomly generated messages of length _1 000 words_ (that's _very_ long for SoundCloud). Since GreenMidget runs on a relational database (through ActiveRecord) by default the table size impacts data fetch and write:
 
 	* on ~ 10 000 table rows = 0.0703 seconds / message
 	* on ~ 100 000 rows = 0.2082 sec / message
@@ -116,13 +112,7 @@ Benchmarking
 Classification Efficiency
 ----------
 
-Trained on s
-
-Benchmarks = > data
-
-Efficiency = > for the sake of this article I ran a small off-production test to show some results on real data - I used 150 000 text items from records which we marked as good and records which we marked as not-the-best!
-
-We'll be next building our own SoundCloud extensions to GreenMidget and use it, so expect to hear more from the student! Meanwhile, I'll be happy to answer everything concerning the project so do feel free to get in touch.
+TODO: give test results; provide a web interface to a trained classifier using some of SoundCloud's spam and legit data; give production experience from DigitaleSeiten.
 
-[green_midget_github]: http://github.com/chochkov/green_midget "Github repository"
+[green_midget_github]: http://github.com/chochkov/GreenMidget "Github repository"
 [guidelines]: http://soundcloud.com/community-guidelines "Community guidelines"

data/lib/green_midget/version.rb

@@ -1,3 +1,3 @@
 module GreenMidget
-  VERSION = '0.0.2'
+  VERSION = '0.0.3'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: green_midget
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -14,7 +14,7 @@ default_executable:
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
-  requirement: &2152860260 !ruby/object:Gem::Requirement
+  requirement: &2153074740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -22,10 +22,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *2152860260
+  version_requirements: *2153074740
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &2152859840 !ruby/object:Gem::Requirement
+  requirement: &2153074320 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -33,10 +33,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2152859840
+  version_requirements: *2153074320
 - !ruby/object:Gem::Dependency
   name: bundler
-  requirement: &2152859420 !ruby/object:Gem::Requirement
+  requirement: &2153073900 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -44,7 +44,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2152859420
+  version_requirements: *2153073900
 description: Naive Bayesian Classifier with customizable features
 email:
 - nikola@howkul.info
@@ -54,17 +54,11 @@ extra_rdoc_files: []
 files:
 - .document
 - .gitignore
-- .rspec
-- .rvmrc
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
 - README.md
 - Rakefile
-- benchmark/benchmark.rb
-- benchmark/test.rb
-- extensions/green_midget_check.rb
-- extensions/sample.rb
 - green_midget.gemspec
 - lib/green_midget.rb
 - lib/green_midget/base.rb

data/.rspec

@@ -1 +0,0 @@
---color

data/.rvmrc

@@ -1 +0,0 @@
-rvm use ruby-1.9.2@green_midget

data/benchmark/benchmark.rb

@@ -1,40 +0,0 @@
-include GreenMidget
-
-TRAININGS               = 90
-CLASSIFICATIONS         = 1
-
-MESSAGE_LENGTH          = 1000
-
-@training_times         = []
-@classification_times   = []
-
-records_count_at_start  = GreenMidgetRecords.count
-
-def generate_text(message_length = 1)
-  message ||= []
-  while message.count < message_length do
-    word = ''
-    (rand(7) + 3).times { word += ('a'..'z').to_a[rand(26)] }
-    message << word unless message.include?(word)
-  end
-  text = message.join(' ')
-end
-
-TRAININGS.times do
-  a = GreenMidgetCheck.new generate_text(MESSAGE_LENGTH)
-  @training_times       << Benchmark.measure{ a.classify_as! [ ALTERNATIVE, NULL ][rand(2)] }.real
-end
-
-CLASSIFICATIONS.times do
-  a = GreenMidgetCheck.new generate_text(MESSAGE_LENGTH)
-  @classification_times << Benchmark.measure{ a.classify }.real
-end
-
-puts " ------------------------------- "
-puts " Average seconds from #{ TRAININGS } trainings and #{ CLASSIFICATIONS } classifications. #{ MESSAGE_LENGTH } words per message:"
-puts " Number of records at start: #{ records_count_at_start } and at the end: #{ GreenMidgetRecords.count }"
-puts " ------------------------------- "
-puts " Training times:                 #{ (@training_times.sum.to_f/TRAININGS).round(4) }"
-puts " ------------------------------- "
-puts " Classification times:           #{ (@classification_times.sum.to_f/CLASSIFICATIONS).round(4) }"
-puts " ------------------------------- "

data/benchmark/test.rb

@@ -1,31 +0,0 @@
-require 'sqlite3'
-
-require File.join(File.dirname(__FILE__), '..', 'spec', 'tester')
-include GreenMidget
-
-ActiveRecord::Base.establish_connection(:adapter => 'sqlite3', :database => '/sc/user_backup/data.db')
-
-@spam = [ 'messages', 'comments', 'posts' ].map { |table| ActiveRecord::Base.connection.execute("select body from #{table} limit 1500").inject([]) { |memo, hash| memo << hash["body"] } }
-
-ActiveRecord::Base.establish_connection(:adapter => 'mysql', :username => 'root', :password => 'root', :database => 'soundcloud_development_temp')
-
-@ham  = [ 'messages', 'comments', 'posts' ].map { |table| GreenMidgetRecords.find_by_sql("select body from #{table} limit 1500").to_a.inject([]) { |memo, hash| memo << hash["body"] } }
-
-ActiveRecord::Base.establish_connection(:adapter => 'mysql', :username => 'root', :password => 'root', :database => 'classifier_development_weird')
-# 
-# # ------ I. PERFORM TRAINING
-# puts Benchmark.measure {
-#   @spam.each { |src|
-#     src.each {|body|
-#       klass = Tester.new(body);klass.classify_as! :spam
-#     }
-#   };true
-# }
-# 
-# puts Benchmark.measure {
-#   @ham.each { |src|
-#     src.each {|body|
-#       klass = Tester.new(body);klass.classify_as! :ham
-#     }
-#   };true
-# }

data/extensions/green_midget_check.rb

@@ -1,8 +0,0 @@
-# Copyright (c) 2011, SoundCloud Ltd., Nikola Chochkov
-class GreenMidgetCheck < GreenMidget::Base
-  attr_accessor :text
-
-  def initialize(text)
-    self.text = text
-  end
-end

data/extensions/sample.rb

@@ -1,19 +0,0 @@
-# Copyright (c) 2011, SoundCloud Ltd., Nikola Chochkov
-class Sample < GreenMidget::Base
-  attr_accessor :user
-
-  def initialize(text, user)
-    @text = text
-    @user = user
-  end
-
-  private
-
-  def features
-    %w(regular_user) + super
-  end
-
-  def regular_user?
-
-  end
-end