classifier-reborn 2.0.3 → 2.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ca32b9ed7e187b4e3ce9ef622828fb0359a36ec3
-  data.tar.gz: 170ada0f042cb00b9aff7668bcc8b1a628b6ed03
+  metadata.gz: 193d6c53d76559337140f192fc69910418015abe
+  data.tar.gz: 745ec353d12ad84aaf74cca6b235401127f00075
 SHA512:
-  metadata.gz: 56f7221cfeb08faf996c0fede890d2d96634d6d39b11f975b8c8b3cc2d581bba335aea01492fada34758f4287ea513d78d39b0fbb7d8297975df26e7d14567e5
-  data.tar.gz: f74e1a9845b5be429e59b6b5fb44701b5db80a606ebd7c271e81cd881428f7b771a00c6582627b3250f583e1cbe8ac16ef5bcff99adf560a2d73fd732ed374d4
+  metadata.gz: c9891b16c6e9fb2ddfffd32a2335f59bfc55a5e97dae675b652acaa9122a2fc268bfc0d4c2be945456fe46397601127dd2a39782a99767718adf7fee373bdae2
+  data.tar.gz: af844b19d90186a6e3866cdcbfa3deb6ba8492e4646298d6712168c1a329f34fbd24c551ff241a561711a8516cf57577dd7fe1a96bb069190e8545a09a570ff0

data/README.markdown

@@ -41,25 +41,140 @@ A Bayesian classifier by Lucas Carlson. Bayesian Classifiers are accurate, fast,
 
 ```ruby
 require 'classifier-reborn'
-b = ClassifierReborn::Bayes.new 'Interesting', 'Uninteresting'
-b.train_interesting "here are some good words. I hope you love them"
-b.train_uninteresting "here are some bad words, I hate you"
-b.classify "I hate bad words and you" # returns 'Uninteresting'
-
-require 'madeleine' # use madeline to persist the data
-m = SnapshotMadeleine.new("bayes_data") {
-  ClassifierReborn::Bayes.new 'Interesting', 'Uninteresting'
-}
-m.system.train_interesting "here are some good words. I hope you love them"
-m.system.train_uninteresting "here are some bad words, I hate you"
-m.take_snapshot
-m.system.classify "I love you" # returns 'Interesting'
+classifier = ClassifierReborn::Bayes.new 'Interesting', 'Uninteresting'
+classifier.train_interesting "here are some good words. I hope you love them"
+classifier.train_uninteresting "here are some bad words, I hate you"
+classifier.classify "I hate bad words and you" # returns 'Uninteresting'
+
+classifier_snapshot = Marshal.dump classifier
+# This is a string of bytes, you can persist it anywhere you like
+
+File.open("classifier.dat", "w") {|f| f.write(classifier_snapshot) }
+# Or Redis.current.save "classifier", classifier_snapshot
+
+# This is now saved to a file, and you can safely restart the application
+data = File.read("classifier.dat")
+# Or data = Redis.current.get "classifier"
+trained_classifier = Marshal.load data
+trained_classifier.classify "I love" # returns 'Interesting'
 ```
 
-Using Madeleine, your application can persist the learned data over time.
+Beyond the basic example, the constructor and trainer can be used in a more
+flexible way to accomidate non-trival applications.  Consider the following
+program:
+
+```ruby
+#!/usr/bin/env ruby
+# classifier_reborn_demo.rb
+
+require 'classifier-reborn'
+
+training_set = DATA.read.split("\n")
+categories   = training_set.shift.split(',').map{|c| c.strip}
+
+classifier = ClassifierReborn::Bayes.new categories
+
+training_set.each do |a_line|
+  next if a_line.empty? || '#' == a_line.strip[0]
+  parts = a_line.strip.split(':')
+  classifier.train(parts.first, parts.last)
+end
+
+puts classifier.classify "I hate bad words and you" #=> 'Uninteresting'
+puts classifier.classify "I hate javascript" #=> 'Uninteresting'
+puts classifier.classify "javascript is bad" #=> 'Uninteresting'
+
+puts classifier.classify "all you need is ruby" #=> 'Interesting'
+puts classifier.classify "i love ruby" #=> 'Interesting'
+
+puts classifier.classify "which is better dogs or cats" #=> 'dog'
+puts classifier.classify "what do I need to kill rats and mice" #=> 'cat'
+
+__END__
+Interesting, Uninteresting
+interesting: here are some good words. I hope you love them
+interesting: all you need is love
+interesting: the love boat, soon we will be taking another ride
+interesting: ruby don't take your love to town
+
+uninteresting: here are some bad words, I hate you
+uninteresting: bad bad leroy brown badest man in the darn town
+uninteresting: the good the bad and the ugly
+uninteresting: java, javascript, css front-end html
+#
+# train categories that were not pre-described
+#
+dog: dog days of summer
+dog: a man's best friend is his dog
+dog: a good hunting dog is a fine thing
+dog: man my dogs are tired
+dog: dogs are better than cats in soooo many ways
+
+cat: the fuzz ball spilt the milk
+cat: got rats or mice get a cat to kill them
+cat: cats never come when you call them
+cat: That dang cat keeps scratching the furniture
+```
+
+#### Knowing the Score
+
+When you ask a bayesian classifier to classify text against a set of trained categories it does so by generating a score (as a Float) for each possible category.  The higher the score the closer the fit your text has with that category.  The category with the highest score is returned as the best matching category.
+
+In *ClassifierReborn* the methods *classifications* and *classify_with_score* give you access to the calculated scores.  The method *classify* only returns the best matching category.
+
+Knowing the score allows you to do some interesting things.  For example if your application is to generate tags for a blog post you could use the *classifications* method to get a hash of the categories and their scores.  You would sort on score and take only the top 3 or 4 categories as your tags for the blog post.
+
+You could within your application establish the smallest acceptable score and only use those categories whose score is greater than or equal to your smallest acceptable score as your tags for the blog post.
+
+But what if you only use the *classify* method?  It does not show you the score of the best category.  How do you know that the best category is really any good?
+
+You can use the threshold.
+
+#### Using the Threshold
+
+Some applications can have only one category.  The application wants to know if the text being classified is of that category or not.  For example consider a list of normal free text responses to some question or maybe a URL string coming to your web application.  You know what a normal response looks like; but, you have no idea how people might mis-use the response.  So what you want to do is create a bayesian classifier that just has one category, for example 'Good' and you want to know wither your text is classified as Good or Not Good.
+
+Or suppose you just want the ability to have multiple categories and a 'None of the Above' as a possibility.
+
+##### Threshold
+
+When you initialize the *ClassifierReborn::Bayes* classifier there are several options which can be set that control threshold processing.
+
+```ruby
+b = ClassifierRebor::Bayes.new(
+        'good',                 # one or more categories
+        enable_threshold: true, # default: false
+        threshold: -10.0        # default: 0.0
+      )
+b.train_good 'good stuff from Dobie Gillis'
+# ...
+text = 'bad junk from Maynard G. Krebs'
+result = b.classify text
+if result.nil?
+  STDERR.puts "ALERT: This is not good: #{text}"
+  let_loose_the_dogs_of_war!  # method definition left to the reader
+end
+
+```
+
+In the *classify* method when the best category for the text has a score that is either less than the established threshold or is Float::INIFINITY, a nil category is returned.  When you see a nil value returned from the *classify* method it means that none of the trained categories (regardless or how many categories were trained) has a score that is above or equal to the established threshold.
+
+#### Other Threshold-related Convience Methods
+
+```ruby
+b.threshold            # get the current threshold
+b.threshold = -10.0    # set the threshold
+b.threshold_enabled?   # Boolean: is the threshold enabled?
+b.threshold_disabled?  # Boolean: is the threshold disabled?
+b.enable_threshold     # enables threshold processing
+b.disable_threshold    # disables threshold processing
+```
+
+Using these convience methods your applications can dynamically adjust threshold processing as required.
 
 ### Bayesian Classification
 
+* https://en.wikipedia.org/wiki/Naive_Bayes_classifier
 * http://www.process.com/precisemail/bayesian_filtering.htm
 * http://en.wikipedia.org/wiki/Bayesian_filtering
 * http://www.paulgraham.com/spam.html
@@ -110,5 +225,6 @@ with more than just simple strings.
 * Cameron McBride (cameron.mcbride@gmail.com)
 * Ivan Acosta-Rubio (ivan@softwarecriollo.com)
 * Parker Moore (email@byparker.com)
+* Chase Gilliam (chase.gilliam@gmail.com)
 
 This library is released under the terms of the GNU LGPL. See LICENSE for more details.

data/data/stopwords/ca

@@ -0,0 +1,126 @@
+de
+es
+i
+a
+o
+un
+una
+unes
+uns
+un
+tot
+també
+altre
+algun
+alguna
+alguns
+algunes
+ser
+és
+soc
+ets
+som
+estic
+està
+estem
+esteu
+estan
+com
+en
+per
+perquè
+per que
+estat
+estava
+ans
+abans
+éssent
+ambdós
+però
+per
+poder
+potser
+puc
+podem
+podeu
+poden
+vaig
+va
+van
+fer
+faig
+fa
+fem
+feu
+fan
+cada
+fi
+inclòs
+primer
+des de
+conseguir
+consegueixo
+consigueix
+consigueixes
+conseguim
+consigueixen
+anar
+haver
+tenir
+tinc
+te
+tenim
+teniu
+tene
+el
+la
+les
+els
+seu
+aquí
+meu
+teu
+ells
+elles
+ens
+nosaltres
+vosaltres
+si
+dins
+sols
+solament
+saber
+saps
+sap
+sabem
+sabeu
+saben
+últim
+llarg
+bastant
+fas
+molts
+aquells
+aquelles
+seus
+llavors
+sota
+dalt
+ús
+molt
+era
+eres
+erem
+eren
+mode
+bé
+quant
+quan
+on
+mentre
+qui
+amb
+entre
+sense
+jo
+aquell

data/data/stopwords/cs

@@ -0,0 +1,138 @@
+dnes
+cz
+timto
+budes
+budem
+byli
+jses
+muj
+svym
+ta
+tomto
+tohle
+tuto
+tyto
+jej
+zda
+proc
+mate
+tato
+kam
+tohoto
+kdo
+kteri
+mi
+nam
+tom
+tomuto
+mit
+nic
+proto
+kterou
+byla
+toho
+protoze
+asi
+ho
+nasi
+napiste
+re
+coz
+tim
+takze
+svych
+jeji
+svymi
+jste
+aj
+tu
+tedy
+teto
+bylo
+kde
+ke
+prave
+ji
+nad
+nejsou
+ci
+pod
+tema
+mezi
+pres
+ty
+pak
+vam
+ani
+kdyz
+vsak
+ne
+jsem
+tento
+clanku
+clanky
+aby
+jsme
+pred
+pta
+jejich
+byl
+jeste
+az
+bez
+take
+pouze
+prvni
+vase
+ktera
+nas
+novy
+tipy
+pokud
+muze
+design
+strana
+jeho
+sve
+jine
+zpravy
+nove
+neni
+vas
+jen
+podle
+zde
+clanek
+uz
+email
+byt
+vice
+bude
+jiz
+nez
+ktery
+by
+ktere
+co
+nebo
+ten
+tak
+ma
+pri
+od
+po
+jsou
+jak
+dalsi
+ale
+si
+ve
+to
+jako
+za
+zpet
+ze
+do
+pro
+je
+na

data/data/stopwords/da

@@ -0,0 +1,101 @@
+af
+alle
+andet
+andre
+at
+begge
+da
+de
+den
+denne
+der
+deres
+det
+dette
+dig
+din
+dog
+du
+ej
+eller
+en
+end
+ene
+eneste
+enhver
+et
+fem
+fire
+flere
+fleste
+for
+fordi
+forrige
+fra
+få
+før
+god
+han
+hans
+har
+hendes
+her
+hun
+hvad
+hvem
+hver
+hvilken
+hvis
+hvor
+hvordan
+hvorfor
+hvornår
+i
+ikke
+ind
+ingen
+intet
+jeg
+jeres
+kan
+kom
+kommer
+lav
+lidt
+lille
+man
+mand
+mange
+med
+meget
+men
+mens
+mere
+mig
+ned
+ni
+nogen
+noget
+ny
+nyt
+nær
+næste
+næsten
+og
+op
+otte
+over
+på
+se
+seks
+ses
+som
+stor
+store
+syv
+ti
+til
+to
+tre
+ud
+var