tf-idf-similarity 0.0.1 → 0.0.2

data/README.md

@@ -1,5 +1,8 @@
 # Ruby Vector Space Model (VSM) with tf*idf weights
 
+[![Dependency Status](https://gemnasium.com/opennorth/tf-idf-similarity.png)](https://gemnasium.com/opennorth/tf-idf-similarity)
+[![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/opennorth/tf-idf-similarity)
+
 Calculates the similarity between texts using a [bag-of-words](http://en.wikipedia.org/wiki/Bag_of_words_model) [Vector Space Model](http://en.wikipedia.org/wiki/Vector_space_model) with [Term Frequency-Inverse Document Frequency](http://en.wikipedia.org/wiki/Tf*idf) weights. If your use case demands performance, use [Lucene](http://lucene.apache.org/core/) (or similar), which also implements other information retrieval functions like [BM 25](http://en.wikipedia.org/wiki/Okapi_BM25).
 
 ## Usage
@@ -17,6 +20,10 @@ This gem will use the [gsl gem](http://rb-gsl.rubyforge.org/) if available, for
 
 ## Optimizations
 
+### [NArray](http://narray.rubyforge.org/)
+
+    gem install narray
+
 ### [GNU Scientific Library (GSL)](http://www.gnu.org/software/gsl/)
 
 The latest `gsl` gem (`1.14.7`) is [not compatible](http://bretthard.in/2012/03/getting-related_posts-lsi-and-gsl-to-work-in-jekyll/) with the `gsl` package (`1.15`) in Homebrew:
@@ -32,22 +39,11 @@ gem install gsl
 
 ### [Automatically Tuned Linear Algebra Software (ATLAS)](http://math-atlas.sourceforge.net/)
 
-You may know this software through [Linear Algebra PACKage (LAPACK)](http://www.netlib.org/lapack/) or [Basic Linear Algebra Subprograms (BLAS)](http://www.netlib.org/blas/).
-
-The `nmatrix` gem (`0.0.1`) can't find the `cblas.h` and `clapack.h` header files. Either [set the C_INCLUDE_PATH](https://github.com/SciRuby/nmatrix#synopsis):
-
-    export C_INCLUDE_PATH=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current/Headers/
-
-Or [create links](https://github.com/SciRuby/nmatrix/issues/21) before installing the gem:
-
-    sudo ln -s /System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current/Headers/cblas.h /usr/include/cblas.h
-    sudo ln -s /System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current/Headers/clapack.h /usr/include/clapack.h
-
-Version `0.0.2` [doesn't compile on Mac OS X Lion](https://github.com/SciRuby/nmatrix/issues/34).
+You may know this software through [Linear Algebra PACKage (LAPACK)](http://www.netlib.org/lapack/) or [Basic Linear Algebra Subprograms (BLAS)](http://www.netlib.org/blas/). You can use it through version `0.0.2` of the [nmatrix gem](https://github.com/SciRuby/nmatrix). As of writing, `0.0.2` is not released, so follow [these instructions](https://github.com/SciRuby/nmatrix#synopsis) to install it. You may need [additional instructions for Mac OS X Lion](https://github.com/SciRuby/nmatrix/wiki/NMatrix-Installation).
 
 ### Other Considerations
 
-The [narray](http://narray.rubyforge.org/) and [nmatrix](http://sciruby.com/nmatrix/) gems have no method to calculate the magnitude of a vector. [Ruby-LAPACK](http://ruby.gfd-dennou.org/products/ruby-lapack/) is a very thin wrapper around LAPACK, which has an opaque Fortran-style naming scheme. [Linalg](https://github.com/quix/linalg) and [RNum](http://rnum.rubyforge.org/) and old and not available as gems.
+The [nmatrix](http://sciruby.com/nmatrix/) gem has no easy way to normalize all columns to unit vectors. [Ruby-LAPACK](http://ruby.gfd-dennou.org/products/ruby-lapack/) is a very thin wrapper around LAPACK, which has an opaque Fortran-style naming scheme. [Linalg](https://github.com/quix/linalg) and [RNum](http://rnum.rubyforge.org/) are old and not available as gems.
 
 ## Extras
 
@@ -58,6 +54,10 @@ You can access more term frequency, document frequency, and normalization formul
 
 The default tf*idf formula follows the [Lucene Conceptual Scoring Formula](http://lucene.apache.org/core/4_0_0-BETA/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html).
 
+## Why?
+
+The [treat](https://github.com/louismullie/treat), [tf-idf](https://github.com/reddavis/TF-IDF), [similarity](https://github.com/bbcrd/Similarity) and [rsimilarity](https://github.com/josephwilk/rsemantic) gems normalize the frequency of a term in a document to the number of terms in that document (which, as far as I can tell, never occurs in the academic literature) and have no normalization component. [vss](https://github.com/mkdynamic/vss) uses plain term and document frequencies, with no damping or normalization.
+
 ## Reference
 
 * [G. Salton and C. Buckley. "Term Weighting Approaches in Automatic Text Retrieval."" Technical Report. Cornell University, Ithaca, NY, USA. 1987.](http://www.cs.odu.edu/~jbollen/IR04/readings/article1-29-03.pdf)

data/lib/tf-idf-similarity/collection.rb

@@ -1,5 +1,8 @@
+# @todo Do speed comparison between these gsl and narray, to load fastest first.
 begin
   require 'gsl'
+rescue LoadError
+  require 'narray'
 rescue LoadError
   require 'matrix'
 end
@@ -45,11 +48,9 @@ class TfIdfSimilarity::Collection
       term_document_matrix = if gsl?
         GSL::Matrix.alloc terms.size, documents.size
       elsif narray?
-        NMatrix.float documents.size, terms.size
+        NArray.float documents.size, terms.size
       elsif nmatrix?
-        # The nmatrix gem's sparse matrices are unfortunately buggy.
-        # @see https://github.com/SciRuby/nmatrix/issues/35
-        NMatrix.new([terms.size, documents.size], :float64)
+        NMatrix.new(:list, [terms.size, documents.size], :float64)
       end
 
       terms.each_with_index do |term,i|
@@ -95,14 +96,11 @@ class TfIdfSimilarity::Collection
     if gsl?
       matrix.each_col(&:normalize!)
     elsif narray?
-      # @todo NArray doesn't have a method to normalize a vector.
-      # 0.upto(matrix.shape[0] - 1).each do |j|
-      #   matrix[j, true] # Normalize this column somehow.
-      # end
-      matrix
+      # @see https://github.com/masa16/narray/issues/21
+      NMatrix.refer matrix / NMath.sqrt((matrix ** 2).sum(1).reshape(5,1))
     elsif nmatrix?
-      # @todo NMatrix doesn't have a method to normalize a vector.
-      matrix
+      # @todo NMatrix has no way to retrieve a column, besides iteration.
+      matrix.cast :yale, :float64
     else
       Matrix.columns matrix.column_vectors.map(&:normalize)
     end

data/lib/tf-idf-similarity/version.rb

@@ -1,3 +1,3 @@
 module TfIdfSimilarity
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tf-idf-similarity
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -95,7 +95,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 697007281194730821
+      hash: -1570138910816303214
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -104,7 +104,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 697007281194730821
+      hash: -1570138910816303214
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.24