pedantic 0.1.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Pat Allan
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,29 @@
+h1. Pedantic
+
+Pedantic cleans strings of text - stripping out unimportant words and URLs, fixing typos, replacing symbols (like emoticons) with real words, and running the results through a stemmer.
+
+In short - it gives you reliable text to process (but not read).
+
+And if the name didn't give it away, yes this library is opinionated.
+
+h2. Installation
+
+Grab the gem.
+
+<pre><code>gem install pedantic</code></pre>
+
+h2. Usage
+
+<pre><code>Pedantic.fix('my messy string ;)') #=> 'messi string joke'</code></pre>
+
+Note that the stemmer generates imperfect words, but it is reasonably reliable and constant in the output, so you can work with those assumptions in the output.
+
+Also - this library is a work in progress - currently I've aimed for a relatively useful but extremely basic implementation. If you look through the code, you'll see there's few typos and emoticons handled. It's easy enough to extend, though - so please, fork, patch and send a pull request.
+
+h2. Contributing
+
+Fork and patch as you see fit - and please send me a pull request if you think it's useful for others. Don't forget to write specs first, and don't mess with the version numbers please (or at least: only do so in a different branch).
+
+h2. Copyright
+
+Copyright (c) 2010 "Pat Allan":http://freelancing-gods.com, but released under an open licence. Go for your life.

data/lib/pedantic/emoticons.rb

@@ -0,0 +1,20 @@
+module Pedantic::Emoticons
+  def self.included(base)
+    base.processors :replace_emoticons
+  end
+  
+  def replace_emoticons(string)
+    {
+      /(^|\s)\:\)(\s|$)/ => ' smile ',
+      /(^|\s)\:\((\s|$)/ => ' sad ',
+      /(^|\s)\:D(\s|$)/  => ' happy ',
+      /(^|\s)\:S(\s|$)/  => ' unsure ',
+      /(^|\s)\:s(\s|$)/  => ' unsure ',
+      /(^|\s)\;\)(\s|$)/ => ' joke '
+    }.each do |pattern, replacement|
+      string.gsub!(pattern, replacement)
+    end
+    
+    string
+  end
+end

data/lib/pedantic/emphasis.rb

@@ -0,0 +1,18 @@
+module Pedantic::Emphasis
+  def self.included(base)
+    base.processors :fix_emphasis
+  end
+  
+  def fix_emphasis(string)
+    {
+      /\bso+\b/        => 'so',
+      /\bre+a+ll+y\b/i => 'really',
+      /\boka+y\b/i     => 'okay',
+      /\boo+h/i        => 'ooh'
+    }.each { |pattern, replacement|
+      string.gsub!(pattern, replacement)
+    }
+    
+    string
+  end
+end

data/lib/pedantic/html.rb

@@ -0,0 +1,15 @@
+module Pedantic::Html
+  def self.included(base)
+    base.processors :remove_html
+  end
+  
+  def remove_html(string)
+    [
+      /"/
+    ].each { |pattern|
+      string.gsub!(pattern, '')
+    }
+    
+    string
+  end
+end

data/lib/pedantic/punctuation.rb

@@ -0,0 +1,18 @@
+module Pedantic::Punctuation
+  def self.included(base)
+    base.processors :replace_punctuation
+  end
+  
+  def replace_punctuation(string)
+    {
+      /\bcan't\b/ => 'cannot',
+      /'/         => '',
+      /sh\*t/     => 'shit',
+      /f\*[\*c]k/ => 'fuck'
+    }.each { |pattern, replacement|
+      string.gsub!(pattern, replacement)
+    }
+    
+    string
+  end
+end

data/lib/pedantic/stems.rb

@@ -0,0 +1,13 @@
+require 'stemmer'
+
+module Pedantic::Stems
+  def self.included(base)
+    base.processors :reduce_to_stems
+  end
+  
+  def reduce_to_stems(string)
+    string.split(/\W+/).collect { |word|
+      word.stem
+    }.join(' ')
+  end
+end

data/lib/pedantic/txt.rb

@@ -0,0 +1,16 @@
+module Pedantic::Txt
+  def self.included(base)
+    base.processors :replace_txt_words
+  end
+  
+  def replace_txt_words(string)
+    {
+      /\btn?x\b/i  => 'thanks',
+      /\bthnks\b/i => 'thanks'
+    }.each { |pattern, replacement|
+      string.gsub!(pattern, replacement)
+    }
+    
+    string
+  end
+end

data/lib/pedantic/unimportant.rb

@@ -0,0 +1,19 @@
+module Pedantic::Unimportant
+  def self.included(base)
+    base.processors :remove_unimportant_words
+  end
+  
+  def remove_unimportant_words(string)
+    %w(
+      a again all along also an and are as at but by came can cant couldnt did 
+      didn didnt do doesntdont ever first from have her here him how i if in 
+      into is isnt it itll just last least like most my new no not now of on or
+      should sinc so some th than this that the their then those to told too
+      true try until url us were when whether while with within yes you youll
+    ).each { |word|
+      string.gsub!(/\b#{word}\b/, '')
+    }
+    
+    string
+  end
+end

data/lib/pedantic/uris.rb

@@ -0,0 +1,16 @@
+module Pedantic::Uris
+  def self.included(base)
+    base.processors :remove_uris
+  end
+  
+  def remove_uris(string)
+    [
+      /https?:\/\/\S+/,
+      /([\w\.])+\.(com|co|net|org|info)(\.\w+)?/
+    ].each { |pattern|
+      string.gsub!(pattern, '')
+    }
+    
+    string
+  end
+end

data/lib/pedantic.rb

@@ -0,0 +1,51 @@
+class Pedantic
+  @@processors = []
+  
+  def self.fix(string)
+    Pedantic.new(string).to_s
+  end
+  
+  def self.processors(*methods)
+    @@processors += methods
+  end
+  
+  def initialize(string)
+    @original = string
+  end
+  
+  def to_s
+    process
+  end
+  
+  private
+  
+  def process
+    @current = @original.clone
+    
+    @@processors.each do |processor|
+      @current = send(processor, @current)
+    end
+    
+    @current.gsub(/\s+/, ' ').strip
+  end
+end
+
+require 'pedantic/emphasis'
+require 'pedantic/emoticons'
+require 'pedantic/html'
+require 'pedantic/punctuation'
+require 'pedantic/stems'
+require 'pedantic/txt'
+require 'pedantic/unimportant'
+require 'pedantic/uris'
+
+class Pedantic
+  include Pedantic::Uris
+  include Pedantic::Html
+  include Pedantic::Emoticons
+  include Pedantic::Txt
+  include Pedantic::Punctuation
+  include Pedantic::Emphasis
+  include Pedantic::Unimportant
+  include Pedantic::Stems
+end

data/spec/pedantic/emoticons_spec.rb

@@ -0,0 +1,23 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Emoticons do
+  it "replaces :) with smile" do
+    Pedantic.fix('foo :)').should == 'foo smile'
+  end
+  
+  it "replaces :( with sad" do
+    Pedantic.fix('foo :(').should == 'foo sad'
+  end
+  
+  it "replaces :D with happy" do
+    Pedantic.fix('foo :D').should == 'foo happi'
+  end
+  
+  it "replaces :S and :s with unsure" do
+    Pedantic.fix('foo :S :s').should == 'foo unsur unsur'
+  end
+  
+  it "replaces ;) with joke" do
+    Pedantic.fix('foo ;)').should == 'foo joke'
+  end
+end

data/spec/pedantic/emphasis_spec.rb

@@ -0,0 +1,15 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Emphasis do
+  it "replaces reeeaalllly with realy" do
+    Pedantic.fix('Reaaally reeeaalllly').should == 'realli realli'
+  end
+  
+  it "replaces okaaaay with okay" do
+    Pedantic.fix('Okaaaaay okaaay').should == 'okai okai'
+  end
+  
+  it "replaces oooooh with ooh" do
+    Pedantic.fix('OoOoOoh').should == 'ooh'
+  end
+end

data/spec/pedantic/html_spec.rb

@@ -0,0 +1,7 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Html do
+  it "removes HTML quotes" do
+    Pedantic.fix('"air quotes"').should == 'air quot'
+  end
+end

data/spec/pedantic/punctuation_spec.rb

@@ -0,0 +1,19 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Punctuation do
+  it "replaces can't with cannot" do
+    Pedantic.fix("can't foo").should == 'cannot foo'
+  end
+  
+  it "removes other apostrophes" do
+    Pedantic.fix("isn't it foo").should == 'foo'
+  end
+  
+  it "replaces sh*t with shit" do
+    Pedantic.fix('sh*t').should == 'shit'
+  end
+  
+  it "replaces f**k and f*ck with fuck" do
+    Pedantic.fix('f**k f*ck').should == 'fuck fuck'
+  end
+end

data/spec/pedantic/stems_spec.rb

@@ -0,0 +1,11 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Stems do
+  it "reduces words to their core stems" do
+    Pedantic.fix('wandering').should == 'wander'
+  end
+  
+  it "reduces all words to their core stems" do
+    Pedantic.fix('wandering and running').should == 'wander run'
+  end
+end

data/spec/pedantic/txt_spec.rb

@@ -0,0 +1,15 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Txt do
+  it "replaces tx with thank" do
+    Pedantic.fix('tx').should == 'thank'
+  end
+  
+  it "replaces tnx with thank" do
+    Pedantic.fix('tnx').should == 'thank'
+  end
+  
+  it "replaces thnks with thank" do
+    Pedantic.fix('thnks').should == 'thank'
+  end
+end

data/spec/pedantic/unimportant_spec.rb

@@ -0,0 +1,7 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Unimportant do
+  it "removes words like 'the', 'a', 'and'" do
+    Pedantic.fix('a foo and the bar').should == 'foo bar'
+  end
+end

data/spec/pedantic/uris_spec.rb

@@ -0,0 +1,15 @@
+require 'spec/spec_helper'
+
+describe Pedantic::Uris do
+  it "removes explicit web addresses" do
+    Pedantic.fix('foo http://bar.com bar').should == 'foo bar'
+  end
+  
+  it "removes explicit https addresses" do
+    Pedantic.fix('foo https://bar.com bar').should == 'foo bar'
+  end
+  
+  it "removes implicit web addresses" do
+    Pedantic.fix('foo domain.com bar').should == 'foo bar'
+  end
+end

data/spec/pedantic_spec.rb

@@ -0,0 +1,17 @@
+require 'spec/spec_helper'
+
+describe Pedantic do
+  describe '.fix' do
+    it "removes leading and trailing spaces" do
+      Pedantic.fix(' foo ').should == 'foo'
+    end
+    
+    it "replaces multiple spaces with a single space" do
+      Pedantic.fix('foo   bar').should == 'foo bar'
+    end
+    
+    it "replaces new lines with spaces" do
+      Pedantic.fix("foo\nbar").should == 'foo bar'
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,10 @@
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+
+require 'pedantic'
+require 'spec'
+require 'spec/autorun'
+
+Spec::Runner.configure do |config|
+  #
+end

metadata

@@ -0,0 +1,104 @@
+--- !ruby/object:Gem::Specification 
+name: pedantic
+version: !ruby/object:Gem::Version 
+  version: 0.1.0
+platform: ruby
+authors: 
+- Pat Allan
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-02-08 00:00:00 +08:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: stemmer
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.0.1
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.2.9
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: yard
+  type: :development
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+    version: 
+description: Pares text down to the words that matter
+email: pat@freelancing-gods.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.textile
+files: 
+- LICENSE
+- README.textile
+- lib/pedantic.rb
+- lib/pedantic/emoticons.rb
+- lib/pedantic/emphasis.rb
+- lib/pedantic/html.rb
+- lib/pedantic/punctuation.rb
+- lib/pedantic/stems.rb
+- lib/pedantic/txt.rb
+- lib/pedantic/unimportant.rb
+- lib/pedantic/uris.rb
+has_rdoc: true
+homepage: http://github.com/freelancing-god/pedantic
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: Text Cleaner
+test_files: 
+- spec/pedantic/emoticons_spec.rb
+- spec/pedantic/emphasis_spec.rb
+- spec/pedantic/html_spec.rb
+- spec/pedantic/punctuation_spec.rb
+- spec/pedantic/stems_spec.rb
+- spec/pedantic/txt_spec.rb
+- spec/pedantic/unimportant_spec.rb
+- spec/pedantic/uris_spec.rb
+- spec/pedantic_spec.rb
+- spec/spec_helper.rb