tfidf 0.0.0

data/examples/demo_tf.rb

@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+require 'tfidf'
+require 'pp'
+require 'ruby-debug'
+Debugger.settings[:autoeval] = true
+Debugger.start
+Debugger.post_mortem
+
+corpus = ["The quick brown fox jumps over the lazy dog",
+          %Q_Fox is a common name for many species of omnivorous mammals belonging to the Canidae family. Foxes are small to medium-sized canids (slightly smaller than the medium-sized domestic dog), characterized by possessing a long narrow snout, and a bushy tail (or brush).
+Members of about 37 species are referred to as foxes, of which only 12 species actually belong to the Vulpes genus of "true foxes". By far the most common and widespread species of fox is the red fox (Vulpes vulpes), although various species are found on almost every continent. The presence of fox-like carnivores all over the globe, together with their widespread reputation for cunning, has contributed to their appearance in popular culture and folklore in many societies around the world (see also Foxes in culture)._,
+          %Q_On March 13, 2008, a YouTube user named RANDYPETERS1, a 9-year-old boy from Chicago, submitted a handdrawn animated video about Octocat, a red cat head with eight long legs looking for his parents. The videos featured crude MS Paint animation and a loud, highpitched, child-like voice narrating. On September 7, the fifth, final episode was released, but featured an unexpected twist - about 20 seconds into it, the crude sketchy animation switched to intricately crafted 3D with an orchestral soundtrack; the whole Octocat story (and as such, the Randy Peters persona) was revealed to be by David O'Reilly [8]. In an interview he joked "I wanted to try experimenting with the Youtube audience and Microsoft Paint. The story for Octocat came to me by reading the bible word-for-word backwards".[9]_ ,
+         "Master's thesis
+Students are required to complete a master's thesis, which is a research assignment with a workload corresponding to 30 credits. The thesis is written on a topic related to the student's major and agreed upon between the student and a professor who specialises in the topic of the thesis. The supervisor of the thesis must be a professor in the University, whereas the instructor(s) must have at least a master’s degree.
+Topic application
+
+The master's thesis process begins by contacting a professor in the student's field of interest, i.e. major, and agreeing on the topic of the thesis. For well founded reasons, the thesis may also be written on a topic related to the student's minor (if a minor is included in the degree).
+
+Once a topic, a supervisor, an instructor and a timetable for the thesis have been determined, an official topic application must be submitted to the Student Services Office. Topic applications are accepted once a month. The Degree Programme Committee confirms the topic and appoints the supervisor and the instructor for the thesis.
+
+A topic for the thesis may be applied for when the Bachelor's degree and at least 45 credits of the Master's degree have been completed. Once confirmed, the topic is valid for one year.
+AaltoELEC's master's thesis instructions
+
+    An introduction to what a master's thesis is and what it's requirements are: What is a Master's Thesis?
+    A guide book written by the School's former professor gives general instructions about thesis work and its academic requirements: How to write a diploma thesis
+    School of Electrical Engineering thesis template (gzipped tarball), provides a LaTeX-template for your thesis writing process and gives detailed instructions on the form and style of the Master's thesis. Make sure to change the language into \"English\" when using this template.  Example Master's Thesis title page and abstract produced with thesis template.
+
+Other useful links:
+
+    Electronic master's thesis database - read completed theses from previous years
+    I\x2415nformation resources
+    Making a bibliography (Otaniemi main library's instructions)
+    How to avoid plagiarism 1 & 2
+    Aalto University Code of Academic Integrity and Handling Violations Thereof
+"         ,
+         %q_The tf*idf weight (term frequency–inverse document frequency) is a numerical statistic which reflects how important a word is to a document in a collection or corpus. It is often used as a weighting factor in information retrieval and text mining. The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to control for the fact that some words are generally more common than others.
+
+Variations of the tf*idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. tf*idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.[1]
+
+One of the simplest ranking functions is computed by summing the tf*idf for each query term; many more sophisticated ranking functions are variants of this simple model._         ,
+         %q_By Melissa Healy, Los Angeles Times
+
+March 27, 2012, 5:45 p.m.
+When roasted at 475 degrees, coffee beans are sometimes described as rich and full-bodied. But for the full-bodied person who is not so rich, unroasted coffee beans — green as the day they were picked — may hold the key to cheap and effective weight loss, new research suggests.
+
+In a study presented Tuesday at the American Chemical Society's spring national meeting in San Diego, 16 overweight young adults took, by turns, a low dose of green coffee bean extract, a high dose of the supplement, and a placebo. Though the study was small, the results were striking: Subjects lost an average of 17.5 pounds in 22 weeks and reduced their overall body weight by 10.5%.
+
+If green coffee extract were a medication seeking approval from the Food and Drug Administration, these results would make it a viable candidate — more than 35% of subjects lost more than 5% of their body weight, and weight loss appeared to be greater while subjects were taking the pills than when they were on the placebo.
+
+But as a dietary supplement, green coffee extract does not require the FDA's blessing. In fact, it is already available as a naturopathic medicine and antioxidant.
+
+Joe Vinson, the University of Scranton chemist who conducted the pilot study, said the findings should pave the way for more rigorous research on coffee bean extract's effects. A larger trial involving 60 people is being planned.
+
+Vinson, whose research focuses on plant polyphenols and their effects on human health, said it appears that green coffee bean extract may work by reducing the absorption of fat and glucose in the gut; it may also reduce insulin levels, which would improve metabolic function. There were no signs of ill effects on any subjects, Vinson reported Tuesday.
+
+The study used a "cross-over" design, which allowed each subject to serve as his or her own comparison group. For six weeks, volunteers swallowed capsules three times a day, ingesting either 700 or 1,050 milligrams of green coffee extract a day or taking a placebo. After a two-week break, they moved, round-robin style, to another arm of the trial.
+
+Subjects did not change their calorie intake over the course of the trial. But the more extract they consumed, the more weight and fat they lost. Altogether, they reduced their body fat by 16%, on average.
+
+Of the 16 volunteers, six wound up with a body mass index in the healthful range.
+
+One downside is that the extract is "extremely bitter." It would be difficult to take without a lot of water, Vinson reported.
+
+At roughly $20 per month, however, green coffee extract is much less expensive than any of the weight-loss medications available over the counter or by prescription.
+
+The trial was conducted in India and paid for by Applied Food Sciences Inc. of Austin, Tex., a manufacturer of green coffee bean extract.
+
+The pilot study drew strong cautions from several scientists who weren't involved in the research.
+
+"This is certainly a provocative study," said Dr. Gerald Weissmann, a physician and biochemist at New York University. But he said nutrition experts would want assurances that green coffee beans do not cause "malabsorption" within the human gut — a condition that would lead to weight loss as well as malnutrition, heart arrhythmias and other problems because vitamins and minerals are not passing through the intestine.
+
+Dr. Arthur Grollman, a pharmacologist at the State University of New York at Stony Brook, said coffee beans contain about 250 different chemicals — some with positive and others with negative effects on human health. Though Vinson identified polyphenols and chlorogenic acid as the agents that appear to promote weight loss, Grollman said that claim needed further study. In the meantime, he said, consuming an extract that contains both good and bad chemicals in dense concentration seems an unwise thing to do._,
+%q_By Andrea Mustain
+OurAmazingPlanet
+updated 3/27/2012 6:34:52 PM ET
+
+    Print
+    Font:
+
+James Cameron's record-setting dive to Earth's deepest spot has sparked a wave of excitement among many in the science community, who are not only heralding the new technology produced by the Hollywood veteran but lauding the renewed focus the project has put on the deep ocean.
+
+"It's wonderful, absolutely wonderful," said Robert J. Stern, a geoscientist at the University of Texas at Dallas. He is one of several deep-sea researchers who said they'd been closely following Cameron's bid to return human observers to the Challenger Deep, a trough within the Mariana Trench more than 35,000 feet (10,700 meters) below the ocean surface.
+
+Cameron's roundtrip earlier this week to the deepest place on Earth lasted just under seven hours. The only previous time humans visited this spot was in 1960.
+
+News of Cameron's successful solo dive "gave me goose bumps," said Cindy Lee Van Dover, director of the marine laboratory at Duke University's Nicholas School of the Environment.
+
+"I think it's a really good thing," said Bruce Robison, a senior scientist at the Monterey Bay Aquarium Research Institute in California. [ Infographic: James Cameron's Mariana Trench Dive ]
+
+Opening the deep
+All three scientists have spent many hours aboard some of the few deep-diving research submersibles on Earth and said they hoped Cameron's technology eventually will prove a boon to researchers wishing to collect samples and perhaps even conduct experiments in the deepest reaches of the sea, a place that until now has been off-limits to humans.
+
+"This establishes that the technology exists to allow that to happen, so we shouldn't constrain our thinking about approaching work in that extreme habitat," Robison told OurAmazingPlanet. "This is a technological breakthrough and a huge accomplishment on Cameron's part, and I'm very pleased that he's done it; but let's hope it opens the door for more."
+
+Cameron's team hasn't confirmed what (if any) samples the filmmaker and explorer retrieved from the Challenger Deep during his three-hour seafloor sojourn. The sub is equipped with a sampling arm, among other research tools. However, Cameron did describe a bleak view through the windows of his lime-green submersible.
+
+    More science news from msnbc.com
+        Image: Brain regions
+        Yamada et al. / Nature Comm.
+        Scientists take a look inside a jury's brains
+
+        Science editor Alan Boyle's blog: Jurors show a characteristic pattern of brain activity when they decide to be lenient on a criminal, and the strength of that pattern can vary from juror to juror, researchers say.
+        'Invisibility cloak' can serve as heat shield
+        Dolphin society adopts freewheeling lifestyle
+        Ancient stone monolith likely marked seasons
+
+"It looked like the moon," Cameron told National Geographic reporters upon his return to the surface world. [ See photos from Cameron's historic dive ]
+
+For scientists, such an assessment was hardly discouraging.
+
+"I was rooting for him to land and find strange-looking animals," Van Dover said, but she added she wasn't surprised. She said the seafloor is vast, conditions are harsh, and life is likely sparsely spread.
+
+"Three hours is just a drop in the bucket, and with more hours I think he's going to discover cool things," Van Dover said. "How many years have we been studying the ocean? And it took until 1977 to discover hydrothermal vents." Van Dover specializes in researching the strange creatures that congregate around the seafloor vents, which spew super-heated water laced with trace chemicals that sustain the animals.
+
+Robison, a veteran deep-sea ecologist, said that whatever does live in the trench, which is nearly a mile deeper than Mount Everest is tall, will be of great interest to scientists.
+
+"Anything that has adapted to thrive in that habitat is going to have some really remarkable adaptations," he said. "But most of the animals in the ocean don't live on the bottom, so there's an enormous potential for discovery up off the bottom as well."
+
+Much of the deepest ocean is unreachable via state-owned submersibles, which at this point can dive no more than 21,000 feet (6,500 m). Only Japan's Shinkai 6500 has reached such depths. The United States is refurbishing Alvin, its deepest-diving craft, to be able to reach 21,000 feet within th_         ]
+
+tfidf = TFIDF.new corpus
+
+puts "Documents in the corpus:"
+tfidf.docs.each {|k,v| puts "Document ID: #{k} => term: #{v}"}
+
+puts "Terms in the corpus"
+tfidf.terms.each {|k,v| puts "Term ID: #{k} => term: #{v}"}
+
+puts "Document-Term Matrix, sparse List of lists(LIL)"
+tfidf.sparse_matrix_doc_idx.each {|e| puts e}
+
+puts "Term-Document Matrix, sparse LIL"
+tfidf.sparse_matrix_term_idx.each {|e| puts e}
+
+puts "Term Frequency of word: video in document"
+puts tfidf.tf
+
+puts "Inverse Document Frequency of word: octocat"
+puts tfidf.idf("octocat")
+
+puts "TF-IDF of word : octocats in document 7e38fa195cee92d2e7d834095d6938a89b5fdd58"
+puts tfidf.tfidf("octocat","7e38fa195cee92d2e7d834095d6938a89b5fdd58")

data/lib/tfidf.rb

@@ -0,0 +1,186 @@
+require 'set'
+require 'fast_stemmer'
+require 'digest'
+
+
+class TFIDF
+
+  #Regex pattern of delimiters for splitting text
+  @@split_pattern = /[\W]/
+  
+  #Hash function used for generating id for documents as well as terms
+  def hash_func(obj)
+    return Digest::SHA1.hexdigest obj
+  end
+
+  #=Arguments
+  #  corpus: an array of strings, one string per document
+  #=Returns
+  #  self
+  #Example:
+  #tfidf = TFIDF.new(["This is a document...",
+  #"Far, far away...",
+  #"The quick brown fox jumps over the lazy dog"])
+  #tfidf.tf("fox","2fd4e1c67a2d28fced849ee1bb76e7391b93eb12") #=> 1
+  #See examples/demo_tf.rb for more
+  def initialize(corpus)
+    @cardinality = 0
+    @docs = {}
+    @terms = {}
+    @sparse_matrix_term_idx = {}
+    @sparse_matrix_doc_idx = {}
+    @idf = {}
+
+    #not in use
+    #TODO:
+    @dense_matrix = nil
+
+    if corpus.is_a? String
+      @cardinality = 1
+      corpus = [corpus]
+    else
+      @cardinality = corpus.length
+    end
+    memo = corpus.reduce({:terms => {}, :docs => {}, :sparse_matrix_doc_idx => {}, :sparse_matrix_term_idx => {}}) do |memo, doc|
+      doc_id = hash_func doc
+      memo[:docs][doc_id] = doc
+      tf_single_doc = TFIDF.tf_single(doc)
+      memo[:sparse_matrix_doc_idx][doc_id] = tf_single_doc
+      tf_single_doc.each do |keyvalue| 
+        term, freq = keyvalue
+        term_id = hash_func term
+        lambda {|x|
+          if !x.has_key?(term_id)
+            x[term_id] = term
+          end}.call memo[:terms]
+        lambda {|x|
+          if x[term] != nil
+            x[term][doc_id] = freq
+          else
+            x[term] = {doc_id => freq}
+          end
+        }.call memo[:sparse_matrix_term_idx]
+      end
+      memo
+    end
+    @docs = memo[:docs]
+    @terms = memo[:terms]
+    @sparse_matrix_term_idx = memo[:sparse_matrix_term_idx]
+    @sparse_matrix_doc_idx = memo[:sparse_matrix_doc_idx]
+    @sparse_matrix_term_idx.each {|k, v|
+      @idf[k] = TFIDF.idf(v.size, @cardinality)}
+  end
+
+  #Build a TF vector out of a single document(String)
+  #=Argument:
+  #  String valued document
+  #=Returns:
+  #  A hash as in {"term" => frequency, ...}
+  def self.tf_single(str)
+    if str == nil
+      return nil
+    else
+      dict = str.split(pattern=@@split_pattern).reduce({}) {|dict, key|
+        key = key.stem.downcase
+        unless TFIDF.should_be_ignored_in_TF?(key)
+          if dict[key] != nil
+            dict[key] += 1
+          else
+            dict[key] = 1
+          end
+        end
+        dict}
+      if block_given?
+        yield dict.keys
+      end
+      dict
+    end
+  end
+
+  #Cardinality, or number of documents in corpus
+  def cardinality
+    return @cardinality
+  end
+  
+  #Documents, in a hash, as in: {"doc_id" => "this is a document...", ...}
+  def docs
+    return @docs
+  end
+  
+  #Terms, stored in a similar way as documents
+  def terms
+    return @terms
+  end
+  
+  #Aka DTM, in sparse List of lists(LIL)
+  def sparse_matrix_doc_idx
+    return @sparse_matrix_doc_idx
+  end
+
+  #Aka TDM, in sparse List of lists(LIL)
+  def sparse_matrix_term_idx
+    return @sparse_matrix_term_idx
+  end
+
+  #=Arguments
+  #  t: Term
+  #  d: Document ID
+  #=Returns
+  #  tf(t,d)
+  #
+  #Alternatively:
+  #=Arguments
+  #  d: Document ID
+  #  t: nil(or unspecified)
+  #=Returns
+  #  Hash which contains non-zero tf of all terms
+  #
+  #Yet another alternative:
+  #=Arguments
+  #  d: nil
+  #  t: nil
+  #=Returns
+  #  Everything
+  def tf(term=nil, doc=nil)
+    if term == nil || doc == nil
+      return @sparse_matrix_doc_idx
+    elsif term == nil
+      return @sparse_matrix_doc_idx[doc]
+    else
+      return lambda {|x| (x == nil)?0:x}.call(@sparse_matrix_doc_idx[doc][term])
+    end
+  end
+
+  #=Arguments
+  #  (Optional)term: Term
+  #=Returns
+  #  IDF of Term
+  def idf(term = nil)
+    if term == nil
+      return @idf
+    else
+      return lambda {|x| (x==nil)?0:x}.call(@idf[term])
+    end
+  end
+
+  #tf*idf(t,d)
+  def tfidf(term, doc)
+    return tf(term,doc) * idf(term)
+  end
+    
+  #Simply the formula for tf*idf
+  def self.idf(x,cardinality)
+    return Math.log2(cardinality.to_f/(x+1).to_f)
+  end
+
+  #If a string is too short or contains non-alphanumeric characters, dump it
+  def self.should_be_ignored_in_TF?(str)
+    if str.length <= 3
+      true
+    elsif (/[^[[:alnum:]]]/ =~ str) != nil
+           true
+    else false
+    end
+  end
+
+end

data/test/test_TFIDF.rb

@@ -0,0 +1,200 @@
+# -*- coding: utf-8 -*-
+require 'test/unit'
+require 'tfidf'
+=begin
+require 'ruby-debug'
+Debugger.start(:post_mortem => true)
+Debugger.settings[:autoeval] = true
+=end
+
+class TFIDFTest < Test::Unit::TestCase
+  def setup
+    @tfidf = TFIDF.new @@corpus
+  end
+
+
+  def test_arbitrary_text
+    #Just an arbitrary test on a single text, the number isn't definitive
+    tfidf = TFIDF.new @@text1
+    pp tfidf.terms.size > 50
+  end
+
+
+  def test_arbitrary_array_of_texts
+    #YA arbitrary test, the number isn't definitive
+    assert @tfidf.terms.size > 300
+  end
+
+
+  def test_should_ignore_short_words
+    assert TFIDF.should_be_ignored_in_TF? "any"
+  end
+
+  def test_should_ignore_weirdos
+    assert TFIDF.should_be_ignored_in_TF? "\x0081\x0081\x0081"
+  end
+  
+  def test_idf_function
+    assert TFIDF.idf(1,2) == 0
+    assert TFIDF.idf(0,2) == 1
+  end
+  
+  def teardown
+    #I don't do nothing
+  end
+
+
+  @@text = "Master's thesis
+Students are required to complete a master's thesis, which is a research assignment with a workload corresponding to 30 credits. The thesis is written on a topic related to the student's major and agreed upon between the student and a professor who specialises in the topic of the thesis. The supervisor of the thesis must be a professor in the University, whereas the instructor(s) must have at least a master’s degree.
+Topic application
+
+The master's thesis process begins by contacting a professor in the student's field of interest, i.e. major, and agreeing on the topic of the thesis. For well founded reasons, the thesis may also be written on a topic related to the student's minor (if a minor is included in the degree).
+
+Once a topic, a supervisor, an instructor and a timetable for the thesis have been determined, an official topic application must be submitted to the Student Services Office. Topic applications are accepted once a month. The Degree Programme Committee confirms the topic and appoints the supervisor and the instructor for the thesis.
+
+A topic for the thesis may be applied for when the Bachelor's degree and at least 45 credits of the Master's degree have been completed. Once confirmed, the topic is valid for one year.
+AaltoELEC's master's thesis instructions
+
+    An introduction to what a master's thesis is and what it's requirements are: What is a Master's Thesis?
+    A guide book written by the School's former professor gives general instructions about thesis work and its academic requirements: How to write a diploma thesis
+    School of Electrical Engineering thesis template (gzipped tarball), provides a LaTeX-template for your thesis writing process and gives detailed instructions on the form and style of the Master's thesis. Make sure to change the language into \"English\" when using this template.  Example Master's Thesis title page and abstract produced with thesis template.
+
+Other useful links:
+
+    Electronic master's thesis database - read completed theses from previous years
+    I\x2415nformation resources
+    Making a bibliography (Otaniemi main library's instructions)
+    How to avoid plagiarism 1 & 2
+    Aalto University Code of Academic Integrity and Handling Violations Thereof
+"
+  @@text1 = %q_The tf*idf weight (term frequency–inverse document frequency) is a numerical statistic which reflects how important a word is to a document in a collection or corpus. It is often used as a weighting factor in information retrieval and text mining. The tf-idf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to control for the fact that some words are generally more common than others.
+
+Variations of the tf*idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. tf*idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.[1]
+
+One of the simplest ranking functions is computed by summing the tf*idf for each query term; many more sophisticated ranking functions are variants of this simple model._
+
+  @@text2 = %q_David O'Reilly (1985, Kilkenny, Ireland) is an Irish film director and artist based in Los Angeles, California, USA. He is known for creating animated short films with a stripped down aesthetic.
+Contents
+ [hide] 
+
+    1 Work
+    2 Octocat Adventure
+    3 Short films
+    4 Music videos
+    5 References
+    6 External links
+
+[edit] Work
+
+Aside from a 1-minute film entitled Ident from which he draws his logo, the earliest work available on his website is WOFL2106[1]. This short draws equally on original designs and popular internet memes, such as Brian Peppers, to create a disturbing landscape of serenity juxtaposed with chaos. This film sets the tone for his entire ouvre, though the direct inclusion of outside memes disappears in his later work.
+
+His short film, Please Say Something, was awarded the Golden Bear at the 2009 Berlin International Film Festival, Best Narrative Short at the 2009 Ottawa International Animation Festival[2] and several other awards.[3]
+
+He created several animation sequences and props for the 2007 film Son of Rambow.[4] As well as animation for the "guide" sequences in Hitchhiker's Guide to the Galaxy, with Shynola.[5]
+
+He created the first video for Irish rock band U2's single "I'll Go Crazy If I Don't Go Crazy Tonight."[6] The video was released on U2.com on July 21, 2009.
+
+His latest short film, The External World, premiered at the 67th Venice Film Festival and the 2011 Sundance Film Festival, and has since won over twenty awards on its festival circuit.[7]
+[edit] Octocat Adventure
+
+On March 13, 2008, a YouTube user named RANDYPETERS1, a 9-year-old boy from Chicago, submitted a handdrawn animated video about Octocat, a red cat head with eight long legs looking for his parents. The videos featured crude MS Paint animation and a loud, highpitched, child-like voice narrating. On September 7, the fifth, final episode was released, but featured an unexpected twist - about 20 seconds into it, the crude sketchy animation switched to intricately crafted 3D with an orchestral soundtrack; the whole Octocat story (and as such, the Randy Peters persona) was revealed to be by David O'Reilly [8]. In an interview he joked \"I wanted to try experimenting with the Youtube audience and Microsoft Paint. The story for Octocat came to me by reading the bible word-for-word backwards\".[9]
+[edit] Short films
+
+    The External World (2010, 15 min)
+    Please Say Something (2009, 10 min)
+    Octocat Adventure (2008, 6 min)[10][11]
+    Serial Entoptics (2008, 10 min)
+    RGBXYZ (2007, 12 min)[12]
+    Wofl2106 (2006, 4 min)
+
+[edit] Music videos
+
+    Szamar Madar (Venetian Snares, 2005, 4 min)
+    I'll Go Crazy If I Don't Go Crazy Tonight (U2, 2009, 4 min)[6]_
+
+  @@text3 = %q_By Melissa Healy, Los Angeles Times
+
+March 27, 2012, 5:45 p.m.
+When roasted at 475 degrees, coffee beans are sometimes described as rich and full-bodied. But for the full-bodied person who is not so rich, unroasted coffee beans — green as the day they were picked — may hold the key to cheap and effective weight loss, new research suggests.
+
+In a study presented Tuesday at the American Chemical Society's spring national meeting in San Diego, 16 overweight young adults took, by turns, a low dose of green coffee bean extract, a high dose of the supplement, and a placebo. Though the study was small, the results were striking: Subjects lost an average of 17.5 pounds in 22 weeks and reduced their overall body weight by 10.5%.
+
+If green coffee extract were a medication seeking approval from the Food and Drug Administration, these results would make it a viable candidate — more than 35% of subjects lost more than 5% of their body weight, and weight loss appeared to be greater while subjects were taking the pills than when they were on the placebo.
+
+But as a dietary supplement, green coffee extract does not require the FDA's blessing. In fact, it is already available as a naturopathic medicine and antioxidant.
+
+Joe Vinson, the University of Scranton chemist who conducted the pilot study, said the findings should pave the way for more rigorous research on coffee bean extract's effects. A larger trial involving 60 people is being planned.
+
+Vinson, whose research focuses on plant polyphenols and their effects on human health, said it appears that green coffee bean extract may work by reducing the absorption of fat and glucose in the gut; it may also reduce insulin levels, which would improve metabolic function. There were no signs of ill effects on any subjects, Vinson reported Tuesday.
+
+The study used a "cross-over" design, which allowed each subject to serve as his or her own comparison group. For six weeks, volunteers swallowed capsules three times a day, ingesting either 700 or 1,050 milligrams of green coffee extract a day or taking a placebo. After a two-week break, they moved, round-robin style, to another arm of the trial.
+
+Subjects did not change their calorie intake over the course of the trial. But the more extract they consumed, the more weight and fat they lost. Altogether, they reduced their body fat by 16%, on average.
+
+Of the 16 volunteers, six wound up with a body mass index in the healthful range.
+
+One downside is that the extract is "extremely bitter." It would be difficult to take without a lot of water, Vinson reported.
+
+At roughly $20 per month, however, green coffee extract is much less expensive than any of the weight-loss medications available over the counter or by prescription.
+
+The trial was conducted in India and paid for by Applied Food Sciences Inc. of Austin, Tex., a manufacturer of green coffee bean extract.
+
+The pilot study drew strong cautions from several scientists who weren't involved in the research.
+
+"This is certainly a provocative study," said Dr. Gerald Weissmann, a physician and biochemist at New York University. But he said nutrition experts would want assurances that green coffee beans do not cause "malabsorption" within the human gut — a condition that would lead to weight loss as well as malnutrition, heart arrhythmias and other problems because vitamins and minerals are not passing through the intestine.
+
+Dr. Arthur Grollman, a pharmacologist at the State University of New York at Stony Brook, said coffee beans contain about 250 different chemicals — some with positive and others with negative effects on human health. Though Vinson identified polyphenols and chlorogenic acid as the agents that appear to promote weight loss, Grollman said that claim needed further study. In the meantime, he said, consuming an extract that contains both good and bad chemicals in dense concentration seems an unwise thing to do._
+
+  @@text4 = %q_By Andrea Mustain
+OurAmazingPlanet
+updated 3/27/2012 6:34:52 PM ET
+
+    Print
+    Font:
+
+James Cameron's record-setting dive to Earth's deepest spot has sparked a wave of excitement among many in the science community, who are not only heralding the new technology produced by the Hollywood veteran but lauding the renewed focus the project has put on the deep ocean.
+
+"It's wonderful, absolutely wonderful," said Robert J. Stern, a geoscientist at the University of Texas at Dallas. He is one of several deep-sea researchers who said they'd been closely following Cameron's bid to return human observers to the Challenger Deep, a trough within the Mariana Trench more than 35,000 feet (10,700 meters) below the ocean surface.
+
+Cameron's roundtrip earlier this week to the deepest place on Earth lasted just under seven hours. The only previous time humans visited this spot was in 1960.
+
+News of Cameron's successful solo dive "gave me goose bumps," said Cindy Lee Van Dover, director of the marine laboratory at Duke University's Nicholas School of the Environment.
+
+"I think it's a really good thing," said Bruce Robison, a senior scientist at the Monterey Bay Aquarium Research Institute in California. [ Infographic: James Cameron's Mariana Trench Dive ]
+
+Opening the deep
+All three scientists have spent many hours aboard some of the few deep-diving research submersibles on Earth and said they hoped Cameron's technology eventually will prove a boon to researchers wishing to collect samples and perhaps even conduct experiments in the deepest reaches of the sea, a place that until now has been off-limits to humans.
+
+"This establishes that the technology exists to allow that to happen, so we shouldn't constrain our thinking about approaching work in that extreme habitat," Robison told OurAmazingPlanet. "This is a technological breakthrough and a huge accomplishment on Cameron's part, and I'm very pleased that he's done it; but let's hope it opens the door for more."
+
+Cameron's team hasn't confirmed what (if any) samples the filmmaker and explorer retrieved from the Challenger Deep during his three-hour seafloor sojourn. The sub is equipped with a sampling arm, among other research tools. However, Cameron did describe a bleak view through the windows of his lime-green submersible.
+
+    More science news from msnbc.com
+        Image: Brain regions
+        Yamada et al. / Nature Comm.
+        Scientists take a look inside a jury's brains
+
+        Science editor Alan Boyle's blog: Jurors show a characteristic pattern of brain activity when they decide to be lenient on a criminal, and the strength of that pattern can vary from juror to juror, researchers say.
+        'Invisibility cloak' can serve as heat shield
+        Dolphin society adopts freewheeling lifestyle
+        Ancient stone monolith likely marked seasons
+
+"It looked like the moon," Cameron told National Geographic reporters upon his return to the surface world. [ See photos from Cameron's historic dive ]
+
+For scientists, such an assessment was hardly discouraging.
+
+"I was rooting for him to land and find strange-looking animals," Van Dover said, but she added she wasn't surprised. She said the seafloor is vast, conditions are harsh, and life is likely sparsely spread.
+
+"Three hours is just a drop in the bucket, and with more hours I think he's going to discover cool things," Van Dover said. "How many years have we been studying the ocean? And it took until 1977 to discover hydrothermal vents." Van Dover specializes in researching the strange creatures that congregate around the seafloor vents, which spew super-heated water laced with trace chemicals that sustain the animals.
+
+Robison, a veteran deep-sea ecologist, said that whatever does live in the trench, which is nearly a mile deeper than Mount Everest is tall, will be of great interest to scientists.
+
+"Anything that has adapted to thrive in that habitat is going to have some really remarkable adaptations," he said. "But most of the animals in the ocean don't live on the bottom, so there's an enormous potential for discovery up off the bottom as well."
+
+Much of the deepest ocean is unreachable via state-owned submersibles, which at this point can dive no more than 21,000 feet (6,500 m). Only Japan's Shinkai 6500 has reached such depths. The United States is refurbishing Alvin, its deepest-diving craft, to be able to reach 21,000 feet within th_
+
+
+  @@corpus = [@@text,@@text1,@@text2,@@text3,@@text4]
+  
+
+end

metadata

@@ -0,0 +1,60 @@
+--- !ruby/object:Gem::Specification
+name: tfidf
+version: !ruby/object:Gem::Version
+  version: 0.0.0
+  prerelease: 
+platform: ruby
+authors:
+- Yu Shen
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-03-17 00:00:00.000000000Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fast-stemmer
+  requirement: &16818040 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *16818040
+description: Calculate TF-IDF out of a text, resulting in a hash with term as key,
+  frequency as value. Sorry for taking the convenient name for myself! See examples/demo_tf.rb
+  for usage
+email: yushen83@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/tfidf.rb
+- examples/demo_tf.rb
+- test/test_TFIDF.rb
+homepage: https://github.com/yushen
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.10
+signing_key: 
+specification_version: 3
+summary: A W.I.P implementation of TF-IDF
+test_files: []