highscore 1.0.0 → 1.1.0

data/History.txt

@@ -1,3 +1,10 @@
+== 1.1.0 / 2013-04-
+
+* added support for custom word ignore handlers using lambda functions
+* fixed some performance issues with unnecessary ranking by unused criteria (consonants & vowels)
+* added possibility to use bloomfilters for faster performance instead of Array#include?
+* removed compatibility with Ruby 1.8.x
+
 == 1.0.0 / 2013-02-02
 
 * added per-language support for black- and whitelists (thanks to bobjflong)

data/README.md

@@ -2,7 +2,7 @@
 
 Easily find and rank keywords in long texts.
 
-[![Build Status](https://secure.travis-ci.org/domnikl/highscore.png?branch=develop)](http://travis-ci.org/domnikl/highscore) [![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/domnikl/highscore)
+[![Build Status](https://secure.travis-ci.org/domnikl/highscore.png?branch=develop)](http://travis-ci.org/domnikl/highscore) [![Code Climate](https://codeclimate.com/github/domnikl/highscore.png)](https://codeclimate.com/github/domnikl/highscore)
 
 ## Features
 
@@ -11,9 +11,23 @@ Easily find and rank keywords in long texts.
 * directly get keywords from String objects
 * blacklist words via a plain text file, String or an Array of words (per language if needed)
 * optionally, configure a whitelist and only words from that list will get ranked
-* use word stemming (requires the `fast-stemmer` gem, or the `stemmer` gem)
+* uses word stemming if necessary (requires the `fast-stemmer` or `stemmer` gem)
 * merge together Keywords from multiple sources
 * contains a CLI tool that operates on STDIN/OUT and is configurable via parameters
+* can use `bloomfilter-rb` gem for better performance (optional)
+
+## Installation
+
+* `[sudo] gem install highscore`
+
+For better blacklist perfomance, use the `bloomfilter-rb` gem:
+
+* `[sudo] gem install bloomfilter-rb`
+
+To use word stemming, you need to have the `fast-stemmer` (C extension) or `stemmer` gem installed:
+
+* `[sudo] gem install fast-stemmer`
+* `[sudo] gem install stemmer`
 
 ## Examples
 
@@ -37,7 +51,7 @@ text.keywords.top(50).each do |keyword|
   keyword.weight # => rank weight (float)
 end
 
-# you could just use a string
+# you could also just use a string
 keywords = "Foo bar baz".keywords(blacklist) do
   set :multiplier, 10
 end
@@ -59,9 +73,11 @@ end
 # not 1.0
 ```
 
-Have a look at bin/highscore, you can run highscore on your CLI and feed it with text on STDIN.
+Have a look at `bin/highscore`, you can run highscore on your CLI and feed it with text on STDIN.
 
-## Using a custom blacklist to ignore keywords
+## Blacklisting and Whitelisting
+
+### Using a custom blacklist to ignore keywords
 
 ```ruby
 # setting single words
@@ -82,7 +98,7 @@ blacklist = Highscore::Blacklist.load_default_file
 content = Highscore::Content.new "a string", blacklist
 ```
 
-## Using a whitelist instead of ranking all words
+### Using a whitelist instead of ranking all words
 
 ```ruby
 # construct and inject it just like a blacklist
@@ -90,7 +106,7 @@ whitelist = Highscore::Whitelist.load %w{these are valid keywords}
 content = Highscore::Content.new "invalid words", whitelist
 ```
 
-## Multiple languages
+## I18n
 
 ```ruby
 # Load a default blacklist
@@ -98,34 +114,25 @@ blacklist_default = Highscore::Blacklist.load "mister"
 text = Highscore::Content.new "oui mister interesting", blacklist_default
 text.keywords.top(3).join " "
 
-# Prints "interesting oui"
+# => prints "interesting oui"
 
 # Load a rudimentary blacklist for French
 blacklist_francais = Highscore::Blacklist.load "oui"
-text.add_wordlist blacklist_francais, "fr"
+text.add_wordlist blacklist_francais, :french
 text.keywords(:lang => :fr).top(3).join " "
 
-# Prints "interesting mister"
+# => prints "interesting mister"
 ```
 
-## Install
-
-* `[sudo] gem install highscore`
-
-To use word stemming, you need to have the `fast-stemmer` (C extension) or `stemmer` gem installed:
-
-* `[sudo] gem install fast-stemmer`
-* `[sudo] gem install stemmer`
-
 ## Author
 
-Original author: Dominik Liebler <liebler.dominik@googlemail.com>
+Original author: Dominik Liebler <liebler.dominik@gmail.com>
 
 ## License
 
 (The MIT License)
 
-Copyright (c) 2012 Dominik Liebler
+Copyright (c) 2013 Dominik Liebler
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/Rakefile

@@ -9,18 +9,22 @@ require "highscore"
 gem_name = "highscore-#{Highscore::VERSION}.gem"
 
 namespace :gem do
+    desc "clean previously generated gems"
     task :clean do
       system "rm -f *.gem"
     end
-  
+    
+    desc "build gem"
     task :build => [:clean, :test] do
       system "gem build highscore.gemspec"
     end
 
+    desc "install gem"
     task :install => :build do
       system "gem install #{gem_name}"
     end
 
+    desc "release to rubygems.org"
     task :release => :build do
       system "gem push #{gem_name}"
     end

data/lib/highscore.rb

@@ -1,5 +1,3 @@
-require 'rubygems' if RUBY_VERSION !~ /^1\.9/
-
 module Highscore
   # :stopdoc:
   LIBPATH = ::File.expand_path('..', __FILE__) + ::File::SEPARATOR

data/lib/highscore/blacklist.rb

@@ -14,5 +14,17 @@ module Highscore
       file_path = File.join(File.dirname(__FILE__), %w{.. blacklist.txt})
       self.load_file(file_path)
     end
+    
+    # filters a given keywords array
+    #
+    # @param Array keywords
+    # @return Array
+    def filter(keywords)
+      keywords.delete_if do |key, value|
+        include?(key.downcase)
+      end
+    end
+    
+    
   end
 end

data/lib/highscore/content.rb

@@ -1,9 +1,6 @@
 $:.unshift(File.dirname(__FILE__))
 require 'keywords'
 
-# external gems
-require 'rubygems'
-
 module Highscore
   class Content
     attr_reader :content
@@ -33,13 +30,10 @@ module Highscore
         :consonants => 0,
         :ignore_short_words => true,
         :ignore_case => false,
+        :ignore => nil,
         :word_pattern => /\p{Word}+/u,
         :stemming => false
       }
-      
-      if RUBY_VERSION =~ /^1\.8/
-        @emphasis[:word_pattern] = /\w+/
-      end
     end
 
     # configure ranking
@@ -71,6 +65,7 @@ module Highscore
     # @return Highscore::Keywords
     def keywords(opts = {})
       used_wordlist = nil
+
       if opts[:lang]
         used_wordlist = language_wordlists[opts[:lang].to_sym]
       else
@@ -80,16 +75,9 @@ module Highscore
       @emphasis[:stemming] = use_stemming?
 
       keywords = Keywords.new
-
-      Keywords.find_keywords(processed_content, used_wordlist, word_pattern).each do |text|
-        text = text.to_s
-        text = text.stem if @emphasis[:stemming]
-
-        if not (text.match(/^[\d]+(\.[\d]+){0,1}$/) or text.length <= 2)
-          keywords << Highscore::Keyword.new(text, weight(text))
-        elsif allow_short_words
-          keywords << Highscore::Keyword.new(text, weight(text))
-        end
+      Keywords.find_keywords(processed_content, used_wordlist, word_pattern).each do |word|
+        keyword = extract_keyword(word)
+        keywords << keyword unless keyword.nil?
       end
 
       keywords
@@ -99,10 +87,10 @@ module Highscore
     #
     # @return Highscore::Wordlist
     def wordlist
-      unless @whitelist.nil?
-        @whitelist
-      else
+      if @whitelist.nil?
         @blacklist
+      else
+        @whitelist
       end
     end
 
@@ -116,6 +104,35 @@ module Highscore
 
     private
 
+    # extracts a single keyword from a single word
+    # 
+    # @return Highscore::Keyword
+    def extract_keyword word
+      word = word.to_s
+      word = word.stem if @emphasis[:stemming]
+
+      unless ignore?(word)
+        Highscore::Keyword.new(word, weight(word))
+      end
+    end
+
+    # Internal: should the word be ignored or not?
+    #
+    # @return TrueClass FalseClass
+    def ignore? word
+      ignore = word.short?
+
+      # exception: allow short words
+      ignore = (not allow_short_words?) if ignore
+
+      # exception: custom handler
+      unless @emphasis[:ignore].nil?
+        ignore = @emphasis[:ignore].call(word)
+      end
+
+      ignore
+    end
+
     # processes the text content applying any necessary transformations
     #
     # @return String
@@ -128,8 +145,8 @@ module Highscore
 
     # allow short words to be rated
     #
-    # @return TrueClass FalseClass
-    def allow_short_words
+    # @return TrueClass|FalseClass
+    def allow_short_words?
       not @emphasis[:ignore_short_words]
     end
 
@@ -156,8 +173,7 @@ module Highscore
       end
 
       weight += vowels(text)
-      weight += consonants(text)
-      weight
+      weight + consonants(text)
     end
 
     # weight the vowels on a text
@@ -165,6 +181,8 @@ module Highscore
     # @param text String
     # @return Float
     def vowels(text)
+      return 0 if @emphasis[:vowels] == 0
+      
       percent = text.vowels.length / text.length.to_f
       percent * @emphasis[:vowels]
     end
@@ -174,28 +192,31 @@ module Highscore
     # @param text String
     # @return Float
     def consonants(text)
+      return 0 if @emphasis[:consonants] == 0
+      
       percent = text.consonants.length / text.length.to_f
       percent * @emphasis[:consonants]
     end
 
-    # using stemming is only possible, if fast-stemmer is installed
-    # doesn't work for JRuby
+    # Internal: using stemming is only possible if fast-stemmer is installed
+    #           doesn't work for JRuby
+    #
+    # @return TrueClass|FalseClass
     def use_stemming?
-      if @emphasis[:stemming]
+      return false unless @emphasis[:stemming]
+
+      gems = %w(fast_stemmer stemmer)
+
+      gems.each do |gem|
         begin
-          require 'fast_stemmer'
-          true
+          require gem
+          return true
         rescue LoadError
-          begin
-            require 'stemmer'
-            true
-          rescue LoadError
-            false
-          end
+          false
         end
-      else
-        false
       end
     end
+
+
   end
-end
+end

data/lib/highscore/keyword.rb

@@ -3,7 +3,7 @@ module Highscore
   # keywords read from the content
   #
   class Keyword
-    attr_accessor :weight, :text
+    attr_accessor :weight, :text, :percent
 
     # init a keyword
     #
@@ -14,6 +14,13 @@ module Highscore
       @weight = weight.to_f
     end
 
+    # sets a percent value (in the keywords context)
+    #
+    # @param percent Float
+    def percent=(percent)
+      @percent = percent.to_f
+    end
+
     # sort keywords
     #
     # @param other Highscore::Keyword

data/lib/highscore/keywords.rb

@@ -1,6 +1,3 @@
-# external
-require 'digest/sha1'
-
 module Highscore
 
   # keywords that were found in content
@@ -16,14 +13,11 @@ module Highscore
     # @return Highscore::Keywords
     def self.find_keywords content, wordlist, pattern=/\w+/
       keywords = content.to_s.scan(pattern).flatten
-      keywords.delete_if do |key, value|
-        if wordlist.kind_of? Highscore::Blacklist
-          wordlist.include?(key.downcase)
-        elsif wordlist.kind_of? Highscore::Whitelist
-          not wordlist.include?(key.downcase)
-        end
+      
+      if not wordlist.nil? and wordlist.respond_to? :filter
+        keywords = wordlist.filter(keywords)
       end
-
+      
       keywords.sort
     end
 
@@ -33,8 +27,7 @@ module Highscore
       @keywords = {}
     end
 
-    # ranks the keywords and removes keywords that have a low ranking
-    # or are blacklisted
+    # ranks the keywords
     #
     # @return Array
     def rank
@@ -46,20 +39,26 @@ module Highscore
     # @param n Fixnum
     # @return Array
     def top n = 10
-      rank[0..(n - 1)]
+      tops = rank[0..(n - 1)]
+      sum_all = sum(n)
+      
+      # set percentage values
+      tops.each do |keyword|
+        keyword.percent = keyword.weight * 100 / sum_all
+      end
+      
+      tops
     end
 
     # add new keywords
     #
-    # @param keyword String
+    # @param keyword Highscore::Keyword
     # @return Highscore::Keywords
     def <<(keyword)
-      key = Digest::SHA1.hexdigest(keyword.text)
-
-      if @keywords.has_key?(key)
-        @keywords[key].weight += keyword.weight
+      if @keywords.has_key?(keyword.text)
+        @keywords[keyword.text].weight += keyword.weight
       else
-        @keywords[key] = keyword
+        @keywords[keyword.text] = keyword
       end
 
       @keywords
@@ -102,6 +101,14 @@ module Highscore
       sort.reverse.first
     end
 
+    # Returns the sum of the weight of the top n keywords
+    #
+    # @return Float
+    def sum(n)
+      top = rank[0..(n - 1)]
+      top.map(&:weight).inject { |sum,weight| sum + weight }
+    end
+
     # merge in another keyword list, operates on self
     #
     # @return Highscore::Keywords

data/lib/highscore/string.rb

@@ -34,4 +34,11 @@ class String
   def consonants
     gsub(/[aeiou]/, '')
   end
+
+  # is this a short word?
+  #
+  # @return TrueClass|FalseClass
+  def short?
+    match(/^[\d]+(\.[\d]+){0,1}$/) or length <= 2
+  end
 end

data/lib/highscore/whitelist.rb

@@ -5,6 +5,15 @@ module Highscore
   # whitelisted words
   #
   class Whitelist < Wordlist
-
+    # filters a given keywords array
+    #
+    # @param Array keywords
+    # @return Array
+    def filter(keywords)
+      keywords.delete_if do |key, value|
+        not include?(key.downcase)
+      end
+    end
+    
   end
 end

data/lib/highscore/wordlist.rb

@@ -8,7 +8,7 @@ module Highscore
     #
     # @param file_path String
     # @return Highscore::Wordlist
-    def self.load_file file_path
+    def self.load_file(file_path, use_bloom_filter = true)
       words = File.read(file_path).split(' ')
       self.load(words)
     end
@@ -34,8 +34,11 @@ module Highscore
     attr_reader :words
 
     # @param words Array
-    def initialize(words = [])
+    def initialize(words = [], use_bloom_filter = true)
       @words = words
+      @bloom_filter = nil
+      
+      init_bloom_filter
     end
 
     # iterate over words
@@ -62,8 +65,12 @@ module Highscore
     #
     # @param keyword String
     # @return true/false
-    def include? keyword
-      @words.include? keyword
+    def include?(keyword)
+      unless @bloom_filter.nil?
+        @bloom_filter.include? keyword
+      else
+        @words.include? keyword
+      end
     end
 
     # add a new word to the blacklist
@@ -71,6 +78,40 @@ module Highscore
     # @param word String
     def <<(word)
       @words << word
+      @bloom_filter << word unless @bloom_filter.nil?
+    end
+    
+  private
+    
+    # determine whether bloom filters should be used
+    def use_bloom_filter
+      begin
+        require 'bloomfilter-rb'
+        true
+      rescue LoadError
+        false
+      end
+    end
+    
+    # build a bloom filter out of this wordlist to determine faster
+    # if words should be black- or whitelisted
+    #
+    def init_bloom_filter
+      return unless use_bloom_filter
+      
+      n = length # number of filter elements
+    
+      if n > 0
+        b = 4  # bits per bucket
+        m = n * b * 10 # number of filter buckets
+
+        k = (0.7 * (m / n)).to_i # number of hash functions
+        k = 1 if k <= 1
+
+        @bloom_filter = BloomFilter::Native.new(:size => m, :bucket => b, :raise => true, :hashes => k)
+        each { |w| @bloom_filter.insert(w) }
+      end
     end
+    
   end
 end

data/test/highscore/test_content.rb

@@ -37,13 +37,7 @@ class TestContent < Highscore::TestCase
     content = 'Schöne Grüße, caractères, русский'
 
     content = Highscore::Content.new content
-    
-    if RUBY_VERSION =~ /^1\.8/
-      # Ruby 1.8 doesn't support correct tokenization
-      assert_equal 3, content.keywords.length
-    else
-      assert_equal 4, content.keywords.length
-    end
+    assert_equal 4, content.keywords.length
   end
 
   def test_vowels_and_consonants
@@ -66,6 +60,14 @@ class TestContent < Highscore::TestCase
     assert_equal 4, keywords.length
   end
 
+  def test_ignore_custom
+    keywords = 'foobar a3832'.keywords do
+      set :ignore, lambda { |w| w.gsub(/[^0-9]/, '').length > 2 }
+    end
+
+    assert_equal 1, keywords.length
+  end
+
   def test_word_pattern
     keywords = 'foo Ruby foo Ruby'.keywords do
       set :word_pattern, /(?=(\b\w+\s\w+\b))/

data/test/highscore/test_keyword.rb

@@ -30,4 +30,12 @@ class TestKeyword < Highscore::TestCase
     @keyword.weight = 10.123
     assert_equal 10.123, @keyword.weight
   end
+  
+  def test_percent
+    # per default, percent is not used => nil
+    assert_nil @keyword.percent
+    
+    @keyword.percent = 50.1
+    assert_equal 50.1, @keyword.percent
+  end
 end

data/test/highscore/test_keywords.rb

@@ -51,8 +51,24 @@ class TestKeywords < Highscore::TestCase
   def test_top
     top = @keywords.top(1)
 
+    assert_equal(1, top.length)
     assert_equal('the', top[0].text)
     assert_equal(10.0, top[0].weight)
+    assert_equal(100.0, top[0].percent)
+  end
+  
+  def test_percent
+    top = @keywords.top(10)
+    
+    assert_equal 4, top.length
+    assert_equal 62.5, top[0].percent
+    assert_equal 18.75, top[1].percent
+    assert_equal 12.5, top[2].percent
+    assert_equal 6.25, top[3].percent
+  end
+  
+  def test_sum
+    assert_equal(16, @keywords.sum(50))
   end
 
   def test_top_empty

data/test/highscore/test_string.rb

@@ -41,4 +41,9 @@ class TestString < Highscore::TestCase
     assert_equal 90, keywords.first.weight
     assert_equal 30, keywords.last.weight
   end
+
+  def test_short?
+    assert "to".short?
+    assert (not "this ain't no short word".short?)
+  end
 end

data/test/highscore/test_wordlist.rb

@@ -54,4 +54,12 @@ class TestBlacklist < Highscore::TestCase
     assert blacklist.include?("foobar")
     assert !blacklist.include?("bla")
   end
+  
+  def test_each
+    blacklist = Highscore::Wordlist.load "foo bar baz"
+    
+    a = []
+    blacklist.each { |x| a << x }
+    assert_equal ['foo', 'bar', 'baz'], a
+  end
 end

data/version.txt

@@ -1 +1 @@
-1.0.0
+1.1.0

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: highscore
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -13,7 +13,7 @@ date: 2012-12-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: simplecov
-  requirement: &70317557633680 !ruby/object:Gem::Requirement
+  requirement: &70314466936760 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: 0.6.4
   type: :development
   prerelease: false
-  version_requirements: *70317557633680
+  version_requirements: *70314466936760
 - !ruby/object:Gem::Dependency
   name: whatlanguage
-  requirement: &70317557633200 !ruby/object:Gem::Requirement
+  requirement: &70314466936180 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,7 +32,7 @@ dependencies:
         version: 1.0.0
   type: :runtime
   prerelease: false
-  version_requirements: *70317557633200
+  version_requirements: *70314466936180
 description: Find and rank keywords in text.
 email: liebler.dominik@googlemail.com
 executables:
@@ -66,7 +66,7 @@ files:
 - History.txt
 - Rakefile
 - version.txt
-homepage: https://domnikl.github.com/highscore
+homepage: http://domnikl.github.com/highscore
 licenses: []
 post_install_message: 
 rdoc_options: