groupie 0.3.0 → 0.4.0

data/.document

@@ -1,5 +0,0 @@
-README.rdoc
-lib/**/*.rb
-bin/*
-features/**/*.feature
-LICENSE

data/LICENSE

@@ -1,20 +0,0 @@
-Copyright (c) 2009 Wes Oldenbeuving
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-"Software"), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/VERSION

@@ -1 +0,0 @@
-0.3.0

data/readme.rdoc

@@ -1,24 +0,0 @@
-= Groupie
-
-Groupie is a simple way to group texts and classify new texts as being a likely member of one of the defined groups. Think of bayesian spam filters.
-
-The eventual goal is to have Groupie work as a sort of bayesian spam filter, where you feed it spam and ham (non-spam) and ask it to classify new texts as spam or ham. Applications for this are e-mail spam filtering and blog spam filtering. Other sorts of categorizing might be interesting as well, such as finding suitable tags for a blog post or bookmark.
-
-Started and forgotten in 2009 as a short-lived experiment, in 2010 Groupie got new features when I started using it on a RSS reader project that classified news items into "Interesting" and "Not interesting" categories.
-
-== Current functionality
-
-Current funcionality includes:
-* Tokenize an input text to prepare it for grouping.
-  * Strip XML and HTML tag.
-  * Keep certain infix characters, such as period and comma.
-* Add texts (as an Array of Strings) to any number of groups.
-* Classify a single word to check the likelihood it belongs to each group.
-* Do classification for complete (tokenized) texts.
-* Pick classification strategy to weigh repeat words differently (weigh by sum, square root or log10 of words in group)
-
-== License
-
-As always, the code is licensed under the MIT license.
-
-Wes Oldenbeuving

data/spec/fixtures/ham/email_ham1.txt

@@ -1,13 +0,0 @@
-Re: [ubuntu-art] [Breathe] Network Manager-icons
-Am Sonntag, den 31.05.2009, 17:53 +0200 schrieb Steve Dodier:
-> Hello,
->
-> I think the notify-osd icons have a completely different style, which
-> is looking great within the notification bubbles, but i doubt it'd
-> look great to have the notify-osd wifi icons in the panel. I think the
-> drawing of the notification- wifi icons should be done afterwards, and
-> if they should be based on those of the icon set, they could be made
-> smoother, and possibly desaturated for some of them, to avoid drawing
-> too much attention from the user when popping up.
->
-> Cordially, SD.

data/spec/fixtures/ham/spam.la-44116217.txt

@@ -1,79 +0,0 @@
-From **HIDDEN**@lists.ubuntu.com  Sun May 31 10:22:46 2009
-Return-Path: <**HIDDEN**@lists.ubuntu.com>
-X-Original-To: **HIDDEN**@spam.la
-Delivered-To: **HIDDEN**@speedo.dreamhost.com
-Received: from chlorine.canonical.com (chlorine.canonical.com [91.189.94.204])
-	by speedo.dreamhost.com (Postfix) with ESMTP id 18A4F145730
-	for <**HIDDEN**@spam.la>; Sun, 31 May 2009 10:22:47 -0700 (PDT)
-Received: from localhost ([127.0.0.1] helo=chlorine.canonical.com)
-	by chlorine.canonical.com with esmtp (Exim 4.60)
-	(envelope-from <**HIDDEN**@lists.ubuntu.com>)
-	id 1MAoKV-0000Uv-RB; Sun, 31 May 2009 17:56:15 +0100
-Received: from smtp104.mail.ukl.yahoo.com ([77.238.184.36])
-	by chlorine.canonical.com with smtp (Exim 4.60)
-	(envelope-from <**HIDDEN**@yahoo.de>) id 1MAoKO-0000T5-0J
-	for **HIDDEN**@lists.ubuntu.com; Sun, 31 May 2009 17:56:08 +0100
-Received: (qmail 95693 invoked from network); 31 May 2009 16:56:07 -0000
-Received: from unknown (HELO ?192.168.1.33?) **HIDDEN**@88.5.92.30 with plain)
-	by smtp104.mail.ukl.yahoo.com with SMTP; 31 May 2009 16:56:07 -0000
-X-Yahoo-SMTP: omQsrMiswBC_IZdIQhRgQAA3Gn6tTTc-
-X-YMail-OSG: e1ihGm4VM1ljqtG.6IPGOps5aG8IYZJEPQLptGPSxphH174zk4rRTWYQJmj9MMc2nJwZjNEqUnYAjErWKypElvLWu0n.v8baMMlcOOELQK2IZfFaV5Ij3HUpUDWRbd0n6PCV5iFLHlyruq5CSGsiZvfME6HpngIO0RuAcin3rePXdzWpmPnTlZwuC3qjSE9N8wC4pdBdwfmYHy4EKSKFRCXUNzdy9DPgfwqrjiCTP_tqaWmpeUOqA2Os13l0j5d6acIpgo9DcW8P_1ENNVGjJ2Lk4XbZ0oc51M_BJ2n6DHMxoazT
-X-Yahoo-Newman-Property: ymail-3
-From: Oliver Scholtz 1 <**HIDDEN**@yahoo.de>
-To: Discussion on Ubuntu artwork <**HIDDEN**@lists.ubuntu.com>
-In-Reply-To: <**HIDDEN**@mail.gmail.com>
-References: <**HIDDEN**@yahoo.com>
-	<**HIDDEN**@jws141-laptop> <**HIDDEN**@yahoo.com>
-	<**HIDDEN**@isabel-desktop>
-	<**HIDDEN**@web95411.mail.in2.yahoo.com>
-	<**HIDDEN**@dani-desktop> <**HIDDEN**@yahoo.com>
-	<**HIDDEN**@mail.gmail.com>
-	<**HIDDEN**@mail.gmail.com>
-Date: Sun, 31 May 2009 18:56:05 +0200
-Message-Id: <**HIDDEN**@oliver-ubuntu>
-Mime-Version: 1.0
-X-Mailer: Evolution 2.26.1 
-Subject: Re: [ubuntu-art] [Breathe] Network Manager-icons
-X-BeenThere: **HIDDEN**@lists.ubuntu.com
-X-Mailman-Version: 2.1.8
-Precedence: list
-Reply-To: Discussion on Ubuntu artwork <**HIDDEN**@lists.ubuntu.com>
-List-Id: Discussion on Ubuntu artwork <ubuntu-art.lists.ubuntu.com>
-List-Unsubscribe: <https://lists.ubuntu.com/mailman/listinfo/ubuntu-art>,
-	<**HIDDEN**@lists.ubuntu.com?subject=unsubscribe>
-List-Archive: <https://lists.ubuntu.com/archives/ubuntu-art>
-List-Post: <**HIDDEN**@lists.ubuntu.com>
-List-Help: <**HIDDEN**@lists.ubuntu.com?subject=help>
-List-Subscribe: <https://lists.ubuntu.com/mailman/listinfo/ubuntu-art>,
-	<**HIDDEN**@lists.ubuntu.com?subject=subscribe>
-Content-Type: text/plain; charset="us-ascii"
-Content-Transfer-Encoding: 7bit
-Sender: **HIDDEN**@lists.ubuntu.com
-Errors-To: **HIDDEN**@lists.ubuntu.com
-
-
-
-Am Sonntag, den 31.05.2009, 17:53 +0200 schrieb Steve Dodier:
-> Hello,
-> 
-> I think the notify-osd icons have a completely different style, which
-> is looking great within the notification bubbles, but i doubt it'd
-> look great to have the notify-osd wifi icons in the panel. I think the
-> drawing of the notification- wifi icons should be done afterwards, and
-> if they should be based on those of the icon set, they could be made
-> smoother, and possibly desaturated for some of them, to avoid drawing
-> too much attention from the user when popping up.
-> 
-> Cordially, SD.
-
-+1
----
-And Mac ... much better! Maybe 22 and 16 without pale. ;)
-
-Oliver
-
-
--- 
-ubuntu-art mailing list
-**HIDDEN**@lists.ubuntu.com
-https://lists.ubuntu.com/mailman/listinfo/ubuntu-art

data/spec/fixtures/spam/email_spam1.txt

@@ -1,5 +0,0 @@
-I noticed your flirt
-If you cannot see the pictures and links below, please click here to view them.
-PHARMACY CLUB | UNSUBSCRIBE | YOUR PRIVACY RIGHTS
-Copyright 2009 Zjfqq, all rights reserved
-Customer Service Dept., 87 Hizq Iveox Street, Isahaylo, VS 25270

data/spec/fixtures/spam/email_spam2.txt

@@ -1,7 +0,0 @@
-Re: Your subscribe #976589
-Tell a friend · Download latest version	See this email as a webpage
-Hello!
-Shipped Privately And Discreetly To Your Door!
-We want to put a great big grin on your face in 2009. You'll be to rejoice all year.
-Unsubscribe · Lost Password · Account Settings · Help · Terms of Service · Privacy
-Ottho Heldringstraat 2, 31719 AZ Amsterdam, The Netherlands

data/spec/fixtures/spam/spam.la-44118014.txt

@@ -1,73 +0,0 @@
-From **HIDDEN**@manpoints.net  Sun May 31 10:34:01 2009
-Return-Path: <**HIDDEN**@manpoints.net>
-X-Original-To: **HIDDEN**@spam.la
-Delivered-To: **HIDDEN**@speedo.dreamhost.com
-Received: from 201-40-49-243.bsace702.dsl.brasiltelecom.net.br (201-40-49-243.bsace702.dsl.brasiltelecom.net.br [201.40.49.243])
-	by speedo.dreamhost.com (Postfix) with ESMTP id 4BDC714572F
-	for <**HIDDEN**@spam.la>; Sun, 31 May 2009 10:33:56 -0700 (PDT)
-Message-Id: <**HIDDEN**@201-40-49-243.bsace702.dsl.brasiltelecom.net.br>
-From: "Leskovar L. Golda" <**HIDDEN**@manpoints.net>
-To: **HIDDEN**@spam.la
-Subject: My official mail blocked
-Content-Type: text/html; charset="iso-8859-1"
-Content-Transfer-Encoding: 7bit
-MIME-Version: 1.0
-Date: Sun, 31 May 2009 10:33:56 -0700 (PDT)
-
-
-
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html>
-<head>
-<title></title>
-<meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type">
-</head>
-<body bgcolor="#FFFFFF" topmargin="0" leftmargin="0" marginwidth="0" marginheight="0">
-
-
-<table width="646" cellspacing="0" border="0" align="center" cellpadding="4">
-<tr>
-<td align="center"><font face="Arial" size="1" color="#000000">If you cannot see
-the pictures and links below, please <a href="http://www.qicweman.cn/?abo=0C8C72B3E1C8648676158B">
-click here</a> to view them.<br></font></td>
-</tr>
-
-</table>
-
-<table width="742" cellspacing="0" border="0" align="center" cellpadding="0">
-
-<tr valign="top">
-<td width="475" style="border-left:1px solid #371E96;">
-<a href="http://www.qicweman.cn/?ex=0C8C72B3E1C8648676158B">
-<img alt="click to see the full version" src="http://www.qicweman.cn/d.jpg" style="border-width: 0px" /></a></td>
-
-</tr>
-
-<tr>
-<td><br>
-
-<div style="padding:10px;">
-
-<span style="font-size:10px;color:#666666;font-family:arial;">You may also
-	respond to this email and subscribe to <i>Hizqru</i> by calling
-	1-085-417-9085, Monday-Friday, 8 a.m.-6 p.m. ET. Outside the U.S. and in
-	Canada, please call 1-254-403-7409.<br><br>
-
-	To opt out from receiving any future marketing-related emails from
-	Dqpjnjp, please <a style="color:#666666;" href="http://www.qicweman.cn/?jr=0C8C72B3E1C8648676158B&email=**HIDDEN**@spam.la">
-	click here</a>.<br />
-	Please be assured that we respect the privacy of our subscribers. To view
-	our privacy policy, please <a style="color:#666666;" href="http://www.qicweman.cn/?sj=0C8C72B3E1C8648676158B">
-	click here</a>.<br><br>
-
-	© 2009 Jdqofypy, Inc., 14 Poze Iylqod, 08th Floor, New York, NY 74172.<br></span>
-
-</div>
-</td>
-</tr>
-</table>
-
-
-
-</body>
-</html>

data/spec/groupie/core_ext/string_spec.rb

@@ -1,37 +0,0 @@
-require File.join(File.dirname(__FILE__), %w[.. .. spec_helper])
-
-describe String do
-  context "tokenize" do
-    it 'should split words' do
-      "hello world".tokenize.should == %w[hello world]
-    end
-
-    it 'should downcase words' do
-      "Hello World".tokenize.should == %w[hello world]
-    end
-
-    it 'should strip special characters' do
-      "blah, bla!".tokenize.should == %w[blah bla]
-    end
-
-    it 'should prserve infix hyphens and underscores' do
-      "hyphen-ated under_score".tokenize.should == %w[hyphen-ated under_score]
-    end
-
-    it 'should sanitize html tags' do
-      '<a href="http://example.org">example</a>'.tokenize.should == %w[example]
-    end
-
-    it 'should preserve infix periods' do
-      'example.org rocks. read it...'.tokenize.should == %w[example.org rocks read it]
-    end
-    
-    it "should preserve infix commas" do
-      '$1,000,000.00 or $1.000.000,00'.tokenize.should == %w[1,000,000.00 or 1.000.000,00]
-    end
-    
-    it "should strip quotes around tokens" do
-      '"first last"'.tokenize.should == %w[first last]
-    end
-  end
-end

data/spec/groupie/group_spec.rb

@@ -1,37 +0,0 @@
-require File.join(File.dirname(__FILE__), %w[.. spec_helper])
-require 'yaml'
-
-describe Groupie::Group do
-  describe "add" do
-    before(:each) do
-      @group = Groupie::Group.new("test")
-    end
-
-    it "should accept a single string" do
-      @group.add "bla"
-      @group.words.should == %w[bla]
-    end
-
-    it "should accept an Array of strings" do
-      @group.add ["bla", "bla2"]
-      @group.words.should == %w[bla bla2]
-    end
-
-    it "should accept multiple strings" do
-      @group.add "bla", "bla2"
-      @group.words.should == %w[bla bla2]
-    end
-
-    it "should be aliased as <<" do
-      @group.method(:add).should == @group.method(:<<)
-    end
-  end
-
-  it "can be serialized and loaded through YAML" do
-    group = Groupie::Group.new 'group'
-    group.add %w[buy flowers]
-    loaded_group = YAML.load(group.to_yaml)
-    loaded_group.add %w[buy candy]
-    loaded_group.count("candy").should == 1
-  end
-end

data/spec/groupie_spec.rb

@@ -1,163 +0,0 @@
-require File.join(File.dirname(__FILE__), 'spec_helper')
-
-describe Groupie do
-  describe "classify" do
-    it 'should work when 100% certaint' do
-      g = Groupie.new
-      g[:spam].add %w[viagra]
-      g[:ham].add %w[flowers]
-      g.classify('viagra').should == {:spam => 1.0, :ham => 0.0}
-    end
-
-    it 'should work when split 50/50 between two groups' do
-      g = Groupie.new
-      g[:spam].add %w[buy viagra now]
-      g[:ham].add %w[buy flowers for your mom]
-      g.classify('buy').should == {:spam => 0.5, :ham => 0.5}
-    end
-
-    it 'should work when weighed more towards one group' do
-      g = Groupie.new
-      g[:spam].add %w[buy viagra now]
-      g[:spam].add %w[buy cialis now]
-      g[:ham].add %w[buy flowers for your mom]
-      g.classify('buy').should == {:spam => 2 / 3.0, :ham => 1 / 3.0}
-    end
-
-    it 'should work with more than two groups' do
-      g = Groupie.new
-      g[:weight].add 'pound'
-      g[:currency].add 'pound'
-      g[:phone_key].add 'pound'
-      g.classify('pound').should == {:weight => 1/3.0, :currency => 1/3.0, :phone_key => 1/3.0}
-    end
-
-    it 'should tokenize and classify emails' do
-      email = File.read(File.join(File.dirname(__FILE__), %w[fixtures spam email_spam1.txt]))
-      email2 = File.read(File.join(File.dirname(__FILE__), %w[fixtures spam email_spam2.txt]))
-      email3 = File.read(File.join(File.dirname(__FILE__), %w[fixtures ham email_ham1.txt]))
-      g = Groupie.new
-      g[:spam].add email.tokenize
-      g[:spam].add email2.tokenize
-      g[:ham].add email3.tokenize
-      c = g.classify('discreetly')
-      c[:spam].should > c[:ham]
-      c2 = g.classify('user')
-      c2[:ham].should > c2[:spam]
-    end
-
-    describe "strategies" do
-      describe "sum" do
-        it "should weigh words for the sum of their occurances" do
-          g = Groupie.new
-          g[:spam].add %w[word] * 9
-          g[:ham].add %w[word]
-          g.classify('word', :sum).should == {:spam=>0.9, :ham=>0.1}
-        end
-      end
-
-      describe "sqrt" do
-        it "should weigh words for the square root of the sum of ocurances" do
-          g = Groupie.new
-          g[:spam].add %w[word] * 9
-          g[:ham].add %w[word]
-          g.classify('word', :sqrt).should == {:spam=>0.75, :ham=>0.25}
-        end
-      end
-
-      describe "log" do
-        it "should weigh words for log10 of their sum of occurances" do
-          g = Groupie.new
-          g[:spam].add %w[word] * 1000
-          g[:ham].add %w[word] * 10
-          g.classify('word', :log).should == {:spam=>0.75, :ham=>0.25}
-        end
-      end
-
-      describe "unique" do
-        it "should should behave as sqrt strategy" do
-          g = Groupie.new
-          g[:spam].add %w[buy viagra now]
-          g[:ham].add %w[buy flowers now]
-          g.classify('buy', :unique).should == g.classify('buy', :sqrt)
-          g.classify('flowers', :unique).should == g.classify('flowers', :sqrt)
-        end
-      end
-    end
-  end
-
-  describe "unique_words" do
-    it "should exclude all words in the 4th quintile of all groups" do
-      g = Groupie.new
-      g[:spam].add %w[one two two three three three four four four four]
-      g[:ham].add %w[apple banana pear orange three]
-      g.unique_words.sort.should == %w[one two apple banana pear orange].sort
-    end
-
-    it "should work on an empty word set" do
-      g = Groupie.new
-      g[:spam].add []
-      g[:ham].add []
-      g.unique_words.should == []
-    end
-  end
-
-  context "classify_text" do
-    it 'should tokenized html emails' do
-      g = Groupie.new
-      spam_tokens = File.read(File.join(File.dirname(__FILE__), %w[fixtures spam spam.la-44118014.txt])).tokenize
-      ham_tokens = File.read(File.join(File.dirname(__FILE__), %w[fixtures ham spam.la-44116217.txt])).tokenize
-      g[:spam].add spam_tokens
-      g[:ham].add ham_tokens
-
-      c = g.classify 'user'
-      c[:ham].should > c[:spam]
-
-      c = g.classify_text(spam_tokens)
-      c[:spam].should > c[:ham]
-    end
-
-    it 'should classify a text' do
-      g = Groupie.new
-      g[:spam].add %w[buy viagra now to grow fast]
-      g[:spam].add %w[buy cialis on our website]
-      g[:ham].add %w[buy flowers for your mom]
-      result = g.classify_text "Grow flowers to sell on our website".tokenize
-      result[:spam].should > result[:ham]
-      result2 = g.classify_text "Grow flowers to give to your mom".tokenize
-      result2[:ham].should == result2[:spam]
-    end
-
-    it "should skip unknown tokens" do
-      g = Groupie.new
-      g[:spam].add %w[buy viagra now]
-      g[:ham].add %w[buy flowers now]
-      g.classify_text(%w[buy buckets now]).should == {:spam=>0.5, :ham=>0.5}
-    end
-
-    it "should support the sqrt strategy" do
-      g = Groupie.new
-      g[:spam].add %w[one] * 9
-      g[:ham].add %w[one]
-      g[:spam].add %w[two] * 9
-      g[:ham].add %w[two]
-      g.classify_text(%w[one two three], :sqrt).should == {:spam=>0.75, :ham=>0.25}
-    end
-
-    it "should support the log strategy" do
-      g = Groupie.new
-      g[:spam].add %w[one] * 100
-      g[:ham].add %w[one]
-      g[:spam].add %w[two]
-      g[:ham].add %w[two] * 100
-      g.classify_text(%w[one two three], :log).should == {:spam=>0.5, :ham=>0.5}
-    end
-
-    it "should only rate unique words for the unique strategy" do
-      g = Groupie.new
-      g[:spam].add %w[one two two three three three four four four four]
-      g[:ham].add %w[apple banana pear]
-      g.classify_text(%w[one two three apple banana], :unique).should == {:spam=>0.5, :ham=>0.5}
-    end
-  end
-end