bayes_on_redis 0.1.3 → 0.1.6

data/datasets/stopwords.txt

@@ -0,0 +1,38 @@
+a able about above abroad according accordingly across actually adj after afterwards again against ago ahead
+ain't all allow allows almost alone along alongside already also although always am amid amidst among amongst
+an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate
+are aren't around as a's aside ask asking associated at available away awfully b back backward backwards be
+became because become becomes becoming been before beforehand begin behind being believe below beside besides
+best better between beyond both brief but by c came can cannot cant can't caption cause causes certain certainly
+changes clearly c'mon co co. com come comes concerning consequently consider considering contain containing
+contains corresponding could couldn't course c's currently d dare daren't definitely described despite did didn't
+different directly do does doesn't doing done don't down downwards during e each edu eg eight eighty either else
+elsewhere end ending enough entirely especially et etc even ever evermore every everybody everyone everything
+everywhere ex exactly example except f fairly far farther few fewer fifth first five followed following follows
+for forever former formerly forth forward found four from further furthermore g get gets getting given gives go
+goes going gone got gotten greetings h had hadn't half happens hardly has hasn't have haven't having he he'd he'll
+hello help hence her here hereafter hereby herein here's hereupon hers herself he's hi him himself his hither
+hopefully how howbeit however hundred i i'd ie if ignored i'll i'm immediate in inasmuch inc inc. indeed indicate
+indicated indicates inner inside insofar instead into inward is isn't it it'd it'll its it's itself i've j just k
+keep keeps kept know known knows l last lately later latter latterly least less lest let let's like liked likely
+likewise little look looking looks low lower ltd m made mainly make makes many may maybe mayn't me mean meantime
+meanwhile merely might mightn't mine minus miss more moreover most mostly mr mrs much must mustn't my myself n name
+namely nd near nearly necessary need needn't needs neither never neverf neverless nevertheless new next nine ninety
+no nobody non none nonetheless noone no-one nor normally not nothing notwithstanding novel now nowhere o obviously
+of off often oh ok okay old on once one ones one's only onto opposite or other others otherwise ought oughtn't our
+ours ourselves out outside over overall own p particular particularly past per perhaps placed please plus possible
+presumably probably provided provides q que quite qv r rather rd re really reasonably recent recently regarding
+regardless regards relatively respectively right round s said same saw say saying says second secondly see seeing
+seem seemed seeming seems seen self selves sensible sent serious seriously seven several shall shan't she she'd
+she'll she's should shouldn't since six so some somebody someday somehow someone something sometime sometimes
+somewhat somewhere soon sorry specified specify specifying still sub such sup sure t take taken taking tell tends
+th than thank thanks thanx that that'll thats that's that've the their theirs them themselves then thence there
+thereafter thereby there'd therefore therein there'll there're theres there's thereupon there've these they they'd
+they'll they're they've thing things think third thirty this thorough thoroughly those though three through
+throughout thru thus till to together too took toward towards tried tries truly try trying t's twice two u un under
+underneath undoing unfortunately unless unlike unlikely until unto up upon upwards us use used useful uses using
+usually v value various versus very via viz vs w want wants was wasn't way we we'd welcome well we'll went were we're
+weren't we've what whatever what'll what's what've when whence whenever where whereafter whereas whereby wherein
+where's whereupon wherever whether which whichever while whilst whither who who'd whoever whole who'll whom whomever
+who's whose why will willing wish with within without wonder won't would wouldn't x y yes yet you you'd you'll your
+you're yours yourself yourselves you've z zero successful greatest began including being all for close but

data/lib/bayes_on_redis.rb

@@ -3,10 +3,14 @@ require "redis"
 
 class BayesOnRedis
   CATEGORIES_KEY = "BayesOnRedis:categories"
-  attr_reader :redis
+  ONE_OR_TWO_WORDS_RE = /\b\w{1,2}\b/mi
+  NON_ALPHANUMERIC_AND_NON_DOT_RE = /[^\w\.]/mi
+
+  attr_reader :redis, :stopwords
 
   def initialize(options)
     @redis = Redis.new(:host => options[:redis_host], :port => options[:redis_port], :db => options[:redis_db])
+    @stopwords = Stopword.new
   end
 
   def flushdb
@@ -69,11 +73,35 @@ class BayesOnRedis
   end
 
   # Incoming text is always downcased
-  def count_occurance(text)
+  def count_occurance(text='')
     raise "input must be instance of String" unless text.is_a?(String)
 
-    text.to_s.downcase.split.inject(Hash.new(0)) do |container, word|
+    text_chunks = text.downcase.gsub(ONE_OR_TWO_WORDS_RE, '').gsub(NON_ALPHANUMERIC_AND_NON_DOT_RE, ' ').gsub(@stopwords.to_re, '').split
+    text_chunks.inject(Hash.new(0)) do |container, word|
       container[word] += 1; container
     end
   end
+
+  def remove_stopwords
+    @redis.smembers(CATEGORIES_KEY).each do |category|
+      @stopwords.to_a.each do |stopword|
+        @redis.hdel(redis_category_key(category), stopword)
+      end
+    end
+  end
+end
+
+
+class Stopword
+  def initialize
+    @stopwords = File.read(File.join("datasets", "stopwords.txt")).split
+  end
+
+  def to_a
+    @stopwords
+  end
+
+  def to_re
+    @to_re ||= /\b(#{@stopwords.join('|')})\b/mi
+  end
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: bayes_on_redis
 version: !ruby/object:Gem::Version 
-  hash: 29
+  hash: 23
   prerelease: false
   segments: 
   - 0
   - 1
-  - 3
-  version: 0.1.3
+  - 6
+  version: 0.1.6
 platform: ruby
 authors: 
 - Didip Kerabat
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-11-24 00:00:00 -08:00
+date: 2010-11-26 00:00:00 -08:00
 default_executable: 
 dependencies: []
 
@@ -30,6 +30,7 @@ extra_rdoc_files: []
 files: 
 - README.markdown
 - lib/bayes_on_redis.rb
+- datasets/stopwords.txt
 has_rdoc: true
 homepage: https://github.com/didip/bayes_on_redis
 licenses: []