chinese_vocab 0.8.0

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in chinese.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Stefan Rohlfing
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,103 @@
+# Chinese::Vocab
+
+`Chinese::Vocab` is meant to make live easier for any Chinese language student who:
+
+* Prefers to learn vocabulary from Chinese sentences.
+* Needs to memorize a lot of words on a __tight time schedule__.
+* Uses the spaced repetition flashcard program [Anki](http://ankisrs.net/).
+
+`Chinese::Vocab` addresses all of the above requirements by downloading sentences for each word and selecting the __minimum required number of Chinese sentences__ (and English translations) to __represent all words__.
+
+You can then export the sentences as well as additional tags provided by `Chinese::Vocab` to Anki.
+
+## Features
+
+* Downloads sentences for each word in a Chinese vocabulary list and selects the __minimum required number of sentences__ to represent all words.
+* With the option key `:compact` set to `true` on initialization, all single character words that also appear in at least one multi character word are removed. The reason behind this option is to __remove redundancy in meaning__ and focus on learning distinct words. Example: (["看", "看书"] => [看书])
+* Adds additional __tags__ to every sentence that can be used in *Anki*:
+ * __Pinyin__: By default the pinyin representation is added to each sentence. Example: "除了这张大钞以外，我没有其他零票了。" => "chú le zhè zhāng dà chāo yĭ wài ，wŏ méi yŏu qí tā líng piào le 。"
+ * __Number of target words__: The number of words from the vocabulary that are covered by a sentence. Example: "除了这张大钞以外，我没有其他零票了。" => "3_words"
+ * __List of target words__: A list of the words from the vocabulary that are covered by a sentence. Example: "除了这张大钞以外，我没有其他零票了。" => "[我, 他, 除了 以外]"
+* Export data to csv for easy import from *Anki*.
+
+
+## Real World Example (using the Traditional HSK word list)
+
+```` ruby
+# Import words from source.
+# First argument:  path to file
+# Second argument: column number of word column (counting starts at 1)
+words = Chinese::Vocab.parse_words('../old_hsk_level_8828_chars_1_word_edited.csv', 4)
+# Sample output:
+words.take(6)
+# => ["啊", "啊", "矮", "爱", "爱人", "安静"]
+
+
+# Initialize an object.
+# First argument:  word list as an array of strings.
+# Options:
+# :compact (defaults to false)
+anki = Chinese::Vocab.new(words, :compact => true)
+
+# List all words
+p anki.words.take(6)
+# => ["啊", "啊", "矮", "爱", "爱人", "安静"]
+p anki.words.size
+# => 7251
+
+# Options:
+# :source (defaults to :nciku)
+# :size   (defaults to :short)
+# :with_pinyin (defaults to true)
+anki.min_sentences(:thread_count => 10)
+
+p anki.stored_sentences.take(2)
+# [{:word=>"吧", :chinese=>"放心吧，他做事向来把牢。",
+#   :pinyin=>"fàng xīn ba ，tā zuò shì xiàng lái bă láo 。",
+#   :english=>"Take it easy. You can always count on him."},
+#  {:word=>"喝", :chinese=>"喝酒挂红的人一般都很能喝。",
+#   :pinyin=>"hē jiŭ guà hóng de rén yī bān dōu hĕn néng hē 。",
+#   :english=>"Those whose face turn red after drinking are normally heavy drinkers."}]
+
+# words not found
+p anki.not_found
+# ["来回来去", "来看来讲", "深美"]
+
+# Number of unique characters in the selected sentences
+p anki.sentences_unique_chars.size
+# => 3290
+
+# Save data to csv.
+# First parameter: path to file
+# Options:
+# Any supported option of Ruby's CSV libary
+anki.to_csv('in_the_wild_test.csv')
+# Sample output (2 sentences/lines out of 4511):
+
+# 舞台上正在上演的是吕剧。,wŭ tái shàng zhèng zài shàng yăn de shì lǚ jù 。,
+# What is being performed on the stage is Lv opera (a local opera of Shandong Province).
+# ,2_words,"[正在, 舞台]"
+# 古代官员上朝都要穿朝靴。,gŭ dài guān yuán shàng cháo dōu yào chuān cháo xuē 。,
+# "In ancient times, all courtiers had to wear special boots to enter the court.",
+# 2_words,"[古代, 官员]"
+
+````
+
+## Documentation
+* [parse_words](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab.parse_words) - How to read in the Chinese words and correctly set the column number, Options:
+ * The [supported options](http://ruby-doc.org/stdlib-1.9.3/libdoc/csv/rdoc/CSV.html#method-c-new) of Ruby's CSV library as well as the `:encoding` parameter. __Note__: `:encoding` is always set to `utf-8` and `:skip_blanks` to `true` internally.
+* [initialize](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab:initialize) - How to write composite expressions such as "除了。。以外", Options:
+ * `:compress` (`Boolean`): Whether or not to remove all single character words that
+also appear in at least one multi character word. Example: (["看", "看书"] => [看书]). The reason behind this option is to remove redundancy in meaning and focus on learning distinct words.
+* [words](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab:words) - Learn how words are edited internally.
+* [min_sentences](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab:min_sentences) - Options:
+ * `:source` (`Symbol`): The online dictionary to download the sentences from, either [:nciku](http://www.nciku.com) or [:jukuu](http://www.jukuu.com). Defaults to `:nciku`. __Note__: Despite the download source chosen (by using the default or setting the `:source` options), if a word was not found on the first site, the second site is used as an alternative.
+ *  `:with_pinyin` (`Boolean`): Whether or not to return the pinyin representation of a sentence. Defaults to `true`.
+ *  `:size` (`Symbol`): The size of the sentence to return from a possible set of several sentences. Supports the values `:short`, `:average`, and `:long`. Defaults to `:short`.
+ * `:thread_count` (`Integer`): The number of threads used to download the sentences. Defaults to `8`.
+* [sentences_unique_chars](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab:sentences_unique_chars) - List of unique Chinese *characters* (single character words) are found in the selected sentences.
+* [to_csv](http://rubydoc.info/github/bytesource/chinese_vocab/master/Chinese/Vocab:to_csv) - Options:
+ * All [supported options](http://ruby-doc.org/stdlib-1.9.3/libdoc/csv/rdoc/CSV.html#method-c-new) of Ruby's CSV library.
+
+
+

data/Rakefile

@@ -0,0 +1,22 @@
+# require 'spec/rake/spectask' # depreciated
+require 'rspec/core/rake_task'
+# require 'rake/gempackagetask' # depreciated
+require 'rubygems/package_task'
+require 'rdoc/task'
+
+# Build gem: rake gem
+# Push gem:  rake push
+
+task :default => [ :spec, :gem ]
+
+RSpec::Core::RakeTask.new {:spec}
+
+gem_spec = eval(File.read('chinese_vocab.gemspec'))
+
+Gem::PackageTask.new( gem_spec ) do |t|
+  t.need_zip = true
+end
+
+task :push => :gem do |t|
+  sh "gem push -v pkg/#{gem_spec.name}-#{gem_spec.version}.gem"
+end

data/lib/chinese.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+require 'chinese/vocab'
+require 'chinese/scraper'
+require 'chinese/version'
+require 'chinese/core_ext/array'
+require 'chinese/core_ext/hash'
+require 'chinese/core_ext/queue'
+require 'chinese/modules/helper_methods'
+
+module Chinese
+end

data/lib/chinese/core_ext/array.rb

@@ -0,0 +1,14 @@
+# encoding: utf-8
+
+class Array
+
+  # Input:  [1,2,3,4,5]
+  # Output: [[1, 2], [2, 3], [3, 4], [4, 5]]
+  def overlap_pairs
+    second = self.dup.drop(1)
+    self.each_with_index.inject([]) {|acc,(item,i)|
+      acc << [item,second[i]]  unless second[i].nil?
+      acc
+    }
+  end
+end

data/lib/chinese/core_ext/hash.rb

@@ -0,0 +1,37 @@
+# encoding: utf-8
+
+class Hash
+
+   # Returns a copy of self with *keys removed.
+   def delete_keys(*keys)
+     hash = self.dup
+
+     keys.each do |key|
+       hash.delete(key)
+     end
+     hash
+   end
+
+   # Remove *keys from self
+   def delete_keys!(*keys)
+     keys.each do |key|
+       self.delete(key)
+     end
+   end
+
+   # Creates a sub-hash from `self` with the keys from `keys`
+   # @note keys in `keys` not present in `self` are silently ignored.
+   # @return [Hash] a copy of `self`.
+   def slice(*keys)
+     self.select { |k,v| keys.include?(k) }
+   end
+
+   def slice!(*keys)
+     sub_hash = self.select { |k,v| keys.include?(k) }
+     # Remove 'keys' form self:
+     self.delete_keys!(*sub_hash.keys)
+     sub_hash
+   end
+end
+
+

data/lib/chinese/core_ext/queue.rb

@@ -0,0 +1,25 @@
+# encoding: utf-8
+
+require 'thread'
+
+class Queue
+
+  def to_a
+    @que
+  end
+
+  # Return nil if queue is empty.
+  def pop!
+    pop(non_block = true)
+  rescue ThreadError => e
+    case e.message
+    when /queue empty/
+      nil
+    else
+      raise
+    end
+  end
+
+end
+
+

data/lib/chinese/modules/helper_methods.rb

@@ -0,0 +1,38 @@
+# encoding: utf-8
+
+module Chinese
+  module HelperMethods
+
+    def self.included(klass)
+      klass.extend(self)
+    end
+
+    def is_unicode?(word)
+      # Remove all non-ascii and non-unicode word characters
+      word = distinct_words(word).join
+      # English text at this point only contains characters that are mathed by \w
+      # Chinese text at this point contains mostly/only unicode word characters that are not matched by \w.
+      # In case of Chinese text the size of 'char_arr' therefore has to be smaller than the size of 'word'
+      char_arr = word.scan(/\w/)
+      char_arr.size < word.size
+    end
+
+    # Input: "除了。。。 以外。。。"
+    # Outout: ["除了", "以外"]
+    def distinct_words(word)
+      # http://stackoverflow.com/a/3976004
+      # Alternative: /[[:word:]]+/
+      word.scan(/\p{Word}+/)      # Returns an array of characters that belong together.
+    end
+
+    # Return true if every distince word (as defined by #distinct_words)
+    # can be found in the given sentence.
+    def include_every_char?(word, sentence)
+      characters = distinct_words(word)
+      characters.all? {|char| sentence.include?(char) }
+    end
+
+
+  end
+end
+

data/lib/chinese/scraper.rb

@@ -0,0 +1,143 @@
+# encoding: utf-8
+require 'cgi'
+require 'open-uri'
+require 'nokogiri'
+require 'timeout'
+require 'chinese/core_ext/array'
+require 'with_validations'
+require 'chinese/modules/helper_methods'
+
+module Chinese
+  class Scraper
+    include WithValidations
+    include HelperMethods
+
+    attr_reader   :source, :word
+    attr_accessor :sentences
+
+    Sources = {
+    nciku:
+    {:url         => "http://www.nciku.com/search/all/examples/",
+     :parent_sel  => "div.examples_box > dl",
+     :cn_sel      => "//dt/span[1]",
+     :en_sel      => "//dd/span[@class='tc_sub']",
+                     # Only cn/en sentence pairs where the second node has a class 'tc_sub' belong together.
+     :select_pair => lambda { |node1,node2| node1['class'] != "tc_sub" && node2['class'] == "tc_sub" },
+                     # Just return the text stored in the node. :text_sel is mainly intended for jukuu (see below)
+     :text_sel    => "text()",
+                     # We want cn first, en second, but nciku does not return cn/en sentence pairs in a strict order.
+     :reorder     => lambda { |text1,text2| if is_unicode?(text2) then [text2,text1] else [text1,text2] end }},
+     jukuu:
+     {:url         => "http://www.jukuu.com/search.php?q=",
+      :parent_sel  => "table#Table1 table[width = '680']",
+      :cn_sel      => "//tr[@class='c']",
+      :en_sel      => "//tr[@class='e']",
+                     # Only cn/en sentence pairs where the first node has a class 'e' belong together.
+      :select_pair => lambda { |node1,node2| node1['class'] == "e" && node2['class'] != "e" },
+      :text_sel    => "td[2]",
+      :reorder     => lambda { |text1,text2| [text2,text1] }}
+  }
+
+  OPTIONS =  {:source =>  [:nciku,  lambda {|value| Sources.keys.include?(value) }],
+              :size   =>  [:average, lambda {|value| [:short, :average, :long].include?(value) }]}
+
+
+    # Options:
+    # size => [:short, :average, :long], default = :average
+    def self.sentences(word, options={})
+      download_source = validate { :source }
+
+      source = Sources[download_source]
+
+      CGI.accept_charset = 'UTF-8'
+      # Note: Use + because << changes the object on its left hand side, but + doesn't:
+      # http://stackoverflow.com/questions/377768/string-concatenation-and-ruby/378258#378258
+      url       = source[:url] + CGI.escape(word)
+      # http://ruby-doc.org/stdlib-1.9.2/libdoc/timeout/rdoc/Timeout.html#method-c-timeout
+      content   = Timeout.timeout(20) { open(url) }
+      main_node = Nokogiri::HTML(content).css(source[:parent_sel]) # Returns a single node.
+      return []  if main_node.to_a.empty?
+
+      # CSS selector:   Returns the tags in the order they are specified
+      # XPath selector: Return the tags in the order they appear in the document (that's what we want here).
+      # Source:         http://stackoverflow.com/questions/5825136/nokogiri-and-finding-element-by-name/5845985#5845985
+      target_nodes = main_node.search("#{source[:cn_sel]} | #{source[:en_sel]}")
+      return [] if target_nodes.to_a.empty?
+
+      # In order to make sure we only return text that also has a translation,
+      # we need to first group each target node with Array#overlap_pairs like this:
+      # Input:  [cn1, cn2, en2, cn3, en3, cn4]
+      # Output: [[cn1,cn2],[cn2,en2],[en2,cn3],[cn3,en3],[en3,cn4]]
+      # and then select the correct pairs: [[cn2,en2],[cn3,en3]].
+      # Regarding #to_a: Nokogiri::XML::NodeSet => Array
+      sentence_pairs = target_nodes.to_a.overlap_pairs.select {|(node1,node2)| source[:select_pair].call(node1,node2) }
+      sentence_pairs = sentence_pairs.reduce([]) do |acc,(cn_node,en_node)|
+        cn   = cn_node.css(source[:text_sel]).text.strip  # 'text' returns an empty string when 'css' returns an empty array.
+        en   = en_node.css(source[:text_sel]).text.strip
+        pair = [cn,en]
+        # Ensure that both the chinese and english selector have text.
+        # (sometimes they don't).
+        acc << pair unless pair_with_empty_string?(pair)
+        acc
+      end
+      # Switch position of each pair if the first entry is the translation,
+      # as we always return an array of [cn_sentence,en_sentence] pairs.
+      # The following step is necessary because:
+      # 1) Jukuu returns sentences in the order English first, Chinese second
+      # 2) Nciku mostly returns sentences in the order Chinese first, English second
+      #    (but sometimes it is the other way round.)
+      sentence_pairs = sentence_pairs.map {|node1,node2| source[:reorder].call(node1,node2) }
+      # Only select Chinese sentences that don't separate words, e.g., skip all sentences like the following:
+      # 北边 => 树林边的河流向北方
+      sentence_pairs = sentence_pairs.select { |cn, _| include_every_char?(word, cn) }
+
+      sentence_pairs
+    end
+
+    def self.sentence(word, options={})
+      value = validate { :size }
+
+      scraped_sentences = sentences(word, options)
+      return [] if scraped_sentences.empty?
+
+      case value
+      when :short
+        shortest_size(scraped_sentences)
+      when :average
+        average_size(scraped_sentences)
+      when :long
+        longest_size(scraped_sentences)
+      end
+    end
+
+
+    # ===================
+    # Helper methods
+    # ===================
+
+    def self.pair_with_empty_string?(pair)
+      pair[0].empty? || pair[1].empty?
+    end
+
+    # Despite its name returns the SECOND shortest sentence,
+    # as the shortest result often is not a real sentence,
+    # but a definition.
+    def self.shortest_size(sentence_pairs)
+      sentence_pairs.sort_by {|(cn,_)| cn.length }.take(2).last
+    end
+
+    def self.longest_size(sentence_pairs)
+      sentence_pairs.sort_by {|(cn,_)| cn.length }.last
+    end
+
+    def self.average_size(sentence_pairs)
+      sorted = sentence_pairs.sort_by {|(cn,_)| cn.length }
+      length = sorted.length
+      sorted.find {|(cn,_)| cn.size >= length/2 }
+    end
+
+
+
+  end
+end
+

data/lib/chinese/version.rb

@@ -0,0 +1,3 @@
+module Chinese
+  VERSION = "0.8.0"
+end

data/lib/chinese/vocab.rb

@@ -0,0 +1,595 @@
+# encoding: utf-8
+require 'thread'
+require 'open-uri'
+require 'nokogiri'
+require 'cgi'
+require 'csv'
+require 'with_validations'
+require 'string_to_pinyin'
+require 'chinese/scraper'
+require 'chinese/modules/helper_methods'
+require 'chinese/core_ext/hash'
+require 'chinese/core_ext/queue'
+
+module Chinese
+  class Vocab
+    include WithValidations
+    include HelperMethods
+
+    # The list of Chinese words after calling {#edit_vocab}. Editing includes:
+    #
+    #  * Removing parentheses (with the content inside each parenthesis).
+    #  * Removing any slash (/) and only keeping the longest part.
+    #  * Removing '儿' for any word longer than two characters.
+    #  * Removing non-word characters such as points and commas.
+    #  * Removing and duplicate words.
+    #@return [Array<String>]
+    attr_reader :words
+    #@return [Boolean] the value of the _:compact_ options key.
+    attr_reader :compact
+    #@return [Array<String>] holds those Chinese words from {#words} that could not be found in any
+    # of the supported online dictionaries during a call to either {#sentences} or {#min_sentences}.
+    # Defaults to `[]`.
+    attr_reader :not_found
+    #@return [Boolean] the value of the `:with_pinyin` option key.
+    attr_reader :with_pinyin
+    #@return [Array<Hash>] holds the return value of either {#sentences} or {#min_sentences},
+    #  whichever was called last. Defaults to `[]`.
+    attr_reader :stored_sentences
+
+    # Mandatory constant for the [WithValidations](http://rubydoc.info/github/bytesource/with_validations/file/README.md) module. Each key-value pair is of the following type:
+    #  `option_key => [default_value, validation]`
+    OPTIONS = {:compact      => [false, lambda {|value| is_boolean?(value) }],
+               :with_pinyin  => [true,  lambda {|value| is_boolean?(value) }],
+               :thread_count => [8,     lambda {|value| value.kind_of?(Integer) }]}
+
+    # Intializes an object.
+    # @note Words that are composite expressions must be written with a least one non-word
+    #   character (such as whitespace) between each sub-expression. Example: "除了 以外" or
+    #   "除了。。以外" instead of "除了以外".
+    # @overload initialize(word_array, options)
+    #  @param [Array<String>] word_array An array of Chinese words that is stored in {#words} after
+    #   all non-ascii, non-unicode characters have been stripped and double entries removed.
+    #  @param [Hash] options The options to customize the following feature.
+    #  @option options [Boolean] :compact Whether or not to remove all single character words that
+    #   also appear in at least one multi character word. Example: (["看", "看书"] => [看书])
+    #   The reason behind this option is to remove redundancy in meaning and focus on learning distinct words.
+    #   Defaults to `false`.
+    # @overload initialize(word_array)
+    # @param [Array<String>] word_array An array of Chinese words that is stored in {#words} after
+    #  all non-ascii, non-unicode characters have been stripped and double entries removed.
+    # @example (see #sentences_unique_chars)
+    def initialize(word_array, options={})
+      @compact = validate { :compact }
+      @words    = edit_vocab(word_array)
+      @words    = remove_redundant_single_char_words(@words)  if @compact
+      @chinese  = is_unicode?(@words[0])
+      @not_found        = []
+      @stored_sentences = []
+    end
+
+
+    # Extracts the vocabulary column from a CSV file as an array of strings. The array is
+    #   normally provided as an argument to {#initialize}
+    # @note (see #initialize)
+    # @overload parse_words(path_to_csv, word_col, options)
+    #  @param [String] path_to_csv The relative or full path to the CSV file.
+    #  @param [Integer] word_col The column number of the vocabulary column (counting starts at 1).
+    #  @param [Hash] options The [supported options](http://ruby-doc.org/stdlib-1.9.3/libdoc/csv/rdoc/CSV.html#method-c-new) of Ruby's CSV library as well as the `:encoding` parameter.
+    #    Exceptions: `:encoding` is always set to `utf-8` and `:skip_blanks` to `true` internally.
+    # @overload parse_words(path_to_csv, word_col)
+    #  @param [String] path_to_csv The relative or full path to the CSV file.
+    #  @param [Integer] word_col The column number of the vocabulary column (counting starts at 1).
+    # @return [Array<String>] The vocabluary (Chinese words)
+    # @example (see #sentences_unique_chars)
+    def self.parse_words(path_to_csv, word_col, options={})
+      # Enforced options:
+      # encoding: utf-8 (necessary for parsing Chinese characters)
+      # skip_blanks: true
+      options.merge!({:encoding => 'utf-8', :skip_blanks => true})
+      csv = CSV.read(path_to_csv, options)
+
+      raise ArgumentError, "Column number (#{word_col}) out of range."  unless within_range?(word_col, csv[0])
+      # 'word_col counting starts at 1, but CSV.read returns an array,
+      # where counting starts at 0.
+      col = word_col-1
+      csv.reduce([]) {|words, row|
+        word = row[col]
+        # If word_col contains no data, CSV::read returns nil.
+        # We also want to skip empty strings or strings that only contain whitespace.
+        words << word  unless word.nil? || word.strip.empty?
+        words
+      }
+    end
+
+
+    # For every Chinese word in {#words} fetches a Chinese sentence and its English translation
+    # from an online dictionary,
+    # @note (Normally you only call this method directly if you really need one sentence
+    #  per Chinese word (even if these words might appear in more than one of the sentences.).
+    # @note (see #min_sentences)
+    # @overload sentences(options)
+    #  @param [Hash] options The options to customize the following features.
+    #  @option options [Symbol] :source The online dictionary to download the sentences from,
+    #    either [:nciku](http://www.nciku.com) or [:jukuu](http://www.jukuu.com).
+    #    Defaults to *:nciku*.
+    #  @option options [Symbol] :size The size of the sentence to return from a possible set of
+    #    several sentences. Supports the values *:short*, *:average*, and *:long*.
+    #    Defaults to *:short*.
+    #  @option options [Boolean] :with_pinyin Whether or not to return the pinyin representation
+    #    of a sentence.
+    #    Defaults to `true`.
+    #  @option options [Integer] :thread_count The number of threads used to download the sentences.
+    #    Defaults to `8`.
+    # @return [Hash] By default each hash holds the following key-value pairs (The return value is also stored in {#stored_sentences}.):
+    #
+    #    * :chinese => Chinese sentence
+    #    * :english => English translation
+    #    * :pinyin  => Pinyin
+    #   The return value is also stored in {#stored_sentences}.
+    # @example
+    #  require 'chinese_vocab'
+    #
+    #  # Extract the Chinese words from a CSV file.
+    #  words = Chinese::Vocab.parse_words('path/to/file/hsk.csv', 4)
+    #
+    #  # Initialize Chinese::Vocab with word array
+    #  # :compact => true means single character words are that also appear in multi-character
+    #  # words are removed from the word array (["看", "看书"] => [看书])
+    #  vocabulary = Chinese::Vocab.new(words, :compact => true)
+    #
+    #  # Return a sentence for each word
+    #  vocabulary.sentences(:size => small)
+    def sentences(options={})
+      puts "Fetching sentences..."
+      # Always run this method.
+
+      # We assign all options to a variable here (also those that are passed on)
+      # as we need them in order to calculate the id.
+      @with_pinyin = validate { :with_pinyin }
+      thread_count = validate { :thread_count }
+      id           = make_hash(@words, options.to_a.sort)
+      words        = @words
+
+      from_queue  = Queue.new
+      to_queue    = Queue.new
+      file_name   = id
+
+      if File.exist?(file_name)
+        puts "Examining file..."
+        words, sentences, not_found = File.open(file_name) { |f| f.readlines }
+        words = convert(words)
+        convert(sentences).each { |s| to_queue << s }
+        @not_found = convert(not_found)
+        size_a = words.size
+        size_b = to_queue.size
+        # puts "Size(words)       = #{size_a}"
+        # puts "Size(to_queue)    = #{size_b}"
+        # puts "Size(words+queue) = #{size_a+size_b}"
+
+        # Remove file
+        File.unlink(file_name)
+      end
+
+      words.each {|word| from_queue << word }
+      result = []
+
+      Thread.abort_on_exception = false
+
+      1.upto(thread_count).map {
+        Thread.new do
+
+          while(word = from_queue.pop!) do
+
+            begin
+              local_result = select_sentence(word, options)
+              puts "Processing word: #{word}"
+              # rescue SocketError, Timeout::Error, Errno::ETIMEDOUT,
+              # Errno::ECONNREFUSED, Errno::ECONNRESET, EOFError => e
+            rescue Exception => e
+              puts " #{e.message}."
+              puts "Please DO NOT abort, but wait for either the program to continue or all threads"
+              puts "to terminate (in which case the data will be saved to disk for fast retrieval on the next run.)"
+              puts "Number of running threads: #{Thread.list.size - 1}."
+              raise
+
+            ensure
+              from_queue << word  if $!
+              puts "Wrote '#{word}' back to queue"  if $!
+            end
+
+            to_queue << local_result  unless local_result.nil?
+
+          end
+        end
+      }.each {|thread| thread.join }
+
+      @stored_sentences = to_queue.to_a
+      @stored_sentences
+
+    ensure
+      if $!
+        while(Thread.list.size > 1) do # Wait for all child threads to terminate.
+          sleep 5
+        end
+
+        File.open(file_name, 'w') do |f|
+          p "============================="
+          p "Writing data to file..."
+          f.write from_queue.to_a
+          f.puts
+          f.write to_queue.to_a
+          f.puts
+          f.write @not_found
+          puts "Finished writing data."
+          puts "Please run the program again after solving the (connection) problem."
+        end
+      end
+    end
+
+
+    # For every Chinese word in {#words} fetches a Chinese sentence and its English translation
+    # from an online dictionary, then calculates and the minimum number of sentences
+    # necessary to cover every word in {#words} at least once.
+    # The calculation is based on the fact that many words occur in more than one sentence.
+    #
+    # @note In case of a network error during dowloading the sentences the data fetched
+    #  so far is automatically copied to a file after several retries. This data is read and
+    #  processed on the next run to reduce the time spend with downloading the sentences
+    #  (which is by far the most time-consuming part).
+    # @note Despite the download source chosen (by using the default or setting the `:source` options), if a word was not found on the first site, the second site is used as an alternative.
+    # @overload min_sentences(options)
+    #  @param [Hash] options The options to customize the following features.
+    #  @option options [Symbol] :source The online dictionary to download the sentences from,
+    #    either [:nciku](http://www.nciku.com) or [:jukuu](http://www.jukuu.com).
+    #    Defaults to `:nciku`.
+    #  @option options [Symbol] :size The size of the sentence to return from a possible set of
+    #    several sentences. Supports the values `:short`, `:average`, and `:long`.
+    #    Defaults to `:short`.
+    #  @option options [Boolean] :with_pinyin Whether or not to return the pinyin representation
+    #    of a sentence.
+    #    Defaults to `true`.
+    #  @option options [Integer] :thread_count The number of threads used to download the sentences.
+    #    Defaults to `8`.
+    # @return [Array<Hash>, []] By default each hash holds the following key-value pairs (The return value is also stored in {#stored_sentences}.):
+    #
+    #    * :chinese => Chinese sentence
+    #    * :english => English translation
+    #    * :pinyin  => Pinyin
+    #    * :uwc     => Unique words count tag (String) of the form "x_words",
+    #      where *x* denotes the number of unique words from {#words} found in the sentence.
+    #    * :uws     => Unique words string tag (String) of the form "[词语1，词语2，...]",
+    #      where *词语* denotes the actual word(s) from {#words} found in the sentence.
+    #   The return value is also stored in {#stored_sentences}.
+    # @example (see #sentences_unique_chars)
+    def min_sentences(options = {})
+      @with_pinyin = validate { :with_pinyin }
+      # Always run this method.
+      thread_count = validate { :thread_count }
+      sentences    = sentences(options)
+
+      minimum_sentences = select_minimum_necessary_sentences(sentences)
+      # :uwc = 'unique words count'
+      with_uwc_tag      = add_key(minimum_sentences, :uwc) {|row| uwc_tag(row[:target_words]) }
+      # :uws = 'unique words string'
+      with_uwc_uws_tags = add_key(with_uwc_tag, :uws) do |row|
+        words = row[:target_words].sort.join(', ')
+        "[" + words + "]"
+      end
+      # Remove those keys we don't need anymore
+      result            = remove_keys(with_uwc_uws_tags, :target_words, :word)
+      @stored_sentences = result
+      @stored_sentences
+    end
+
+
+    # Finds the unique Chinese characters from either the data in {#stored_sentences} or an
+    # array of Chinese sentences passed as an argument.
+    # @overload sentences_unique_chars(sentences)
+    #  @param [Array<String>, Array<Hash>] sentences An array of chinese sentences or an array of hashes with the key `:chinese`.
+    # @note If no argument is passed, the data from {#stored_sentences} is used as input
+    # @return [Array<String>] The unique Chinese characters
+    # @example
+    #  require 'chinese_vocab'
+    #
+    #  # Extract the Chinese words from a CSV file.
+    #  words = Chinese::Vocab.parse_words('path/to/file/hsk.csv', 4)
+    #
+    #  # Initialize Chinese::Vocab with word array
+    #  # :compact => true means single character words are that also appear in multi-character
+    #  # words are removed from the word array (["看", "看书"] => [看书])
+    #  vocabulary = Chinese::Vocab.new(words, :compact => true)
+    #
+    #  # Return minimum necessary sentences.
+    #  vocabulary.min_sentences(:size => small)
+    #
+    #  # See how what are the unique characters in all these sentences.
+    #  vocabulary.sentences_unique_chars(my_sentences)
+    #  # => ["我", "们", "跟", "他", "是", "好", "朋", "友", ...]
+    #
+    #  # Save to file
+    #  vocabulary.to_csv('path/to_file/vocab_sentences.csv')
+    def sentences_unique_chars(sentences = stored_sentences)
+      # If the argument is an array of hashes, then it must be the data from @stored_sentences
+      sentences = sentences.map { |hash| hash[:chinese] }  if sentences[0].kind_of?(Hash)
+
+      sentences.reduce([]) do |acc, row|
+        acc = acc | row.scan(/\p{Word}/) # only return characters, skip punctuation marks
+        acc
+      end
+    end
+
+
+    # Saves the data stored in {#stored_sentences} to disk.
+    # @overload to_csv(path_to_file, options)
+    #  @param [String] path_to_file file name and path of where to save the file.
+    #  @param [Hash] options The [supported options](http://ruby-doc.org/stdlib-1.9.3/libdoc/csv/rdoc/CSV.html#method-c-new) of Ruby's CSV library.
+    # @overload to_csv(path_to_file)
+    #  @param [String] path_to_file file name and path of where to save the file.
+    # @return [void]
+    # @example (see #sentences_unique_chars)
+    def to_csv(path_to_file, options = {})
+
+      CSV.open(path_to_file, "w", options) do |csv|
+        @stored_sentences.each do |row|
+          csv << row.values
+        end
+      end
+    end
+
+
+    # Helper functions
+    # -----------------
+    def remove_parens(word)
+      # 1) Remove all ASCII parens and all data in between.
+      # 2) Remove all Chinese parens and all data in between.
+      word.gsub(/\(.*?\)/, '').gsub(/（.*?）/, '')
+    end
+
+
+    def is_boolean?(value)
+      # Only true for either 'false' or 'true'
+      !!value == value
+    end
+
+
+    # Remove all non-word characters
+    def edit_vocab(word_array)
+
+      word_array.map {|word|
+        edited = remove_parens(word)
+        edited = remove_slash(edited)
+        edited = remove_er_character_from_end(edited)
+        distinct_words(edited).join(' ')
+      }.uniq
+    end
+
+
+    def remove_er_character_from_end(word)
+      if word.size > 2
+      word.gsub(/儿$/, '')
+      else # Don't remove "儿" form words like 女儿
+        word
+      end
+    end
+
+
+    def remove_slash(word)
+      if word.match(/\//)
+        word.split(/\//).sort_by { |w| w.size }.last
+      else
+        word
+      end
+    end
+
+
+    def make_hash(*data)
+      require 'digest'
+      data = data.reduce("") { |acc, item| acc << item.to_s }
+      Digest::SHA2.hexdigest(data)[0..6]
+    end
+
+
+    # Input: ["看", "书", "看书"]
+    # Output: ["看书"]
+    def remove_redundant_single_char_words(words)
+      puts "Removing redundant single character words from the vocabulary..."
+
+      single_char_words, multi_char_words = words.partition {|word| word.length == 1 }
+      return single_char_words  if multi_char_words.empty?
+
+      non_redundant_single_char_words = single_char_words.reduce([]) do |acc, single_c|
+
+        already_found = multi_char_words.find do |multi_c|
+          multi_c.include?(single_c)
+        end
+        # Add single char word to array if it is not part of any of the multi char words.
+        acc << single_c  unless already_found
+        acc
+      end
+
+      non_redundant_single_char_words + multi_char_words
+    end
+
+
+    # Uses options passed from #sentences
+    def select_sentence(word, options)
+      sentence_pair = Scraper.sentence(word, options)
+
+      sources = Scraper::Sources.keys
+      sentence_pair = try_alternate_download_sources(sources, word, options)  if sentence_pair.empty?
+
+      if sentence_pair.empty?
+        @not_found << word
+        return nil
+      else
+        chinese, english = sentence_pair
+
+        result = Hash.new
+        result.merge!(word:    word)
+        result.merge!(chinese: chinese)
+        result.merge!(pinyin:  chinese.to_pinyin)  if @with_pinyin
+        result.merge!(english: english)
+      end
+    end
+
+
+    def try_alternate_download_sources(alternate_sources, word, options)
+      sources = alternate_sources.dup
+      sources.delete(options[:source])
+
+      result = sources.find do |s|
+        options  = options.merge(:source => s)
+        sentence = Scraper.sentence(word, options)
+        sentence.empty? ? nil : sentence
+      end
+
+      if result
+        optins = options.merge(:source => result)
+        Scraper.sentence(word, options)
+      else
+        []
+      end
+    end
+
+
+    def convert(text)
+      eval(text.chomp)
+    end
+
+
+    def add_target_words(hash_array)
+      from_queue  = Queue.new
+      to_queue    = Queue.new
+      # semaphore = Mutex.new
+      result      = []
+      words       = @words
+      hash_array.each {|hash| from_queue << hash}
+
+      10.times.map {
+        Thread.new(words) do
+
+          while(row = from_queue.pop!)
+            sentence     = row[:chinese]
+            target_words = target_words_per_sentence(sentence, words)
+
+            to_queue << row.merge(:target_words => target_words)
+
+          end
+        end
+      }.map {|thread| thread.join}
+
+      to_queue.to_a
+
+    end
+
+
+    def target_words_per_sentence(sentence, words)
+       words.select {|w| include_every_char?(w, sentence) }
+    end
+
+
+    def sort_by_target_word_count(with_target_words)
+
+      # First sort by size of unique word array (from large to short)
+      # If the unique word count is equal, sort by the length of the sentence (from small to large)
+      with_target_words.sort_by {|row|
+        [-row[:target_words].size, row[:chinese].size] }
+
+        #  The above is the same as:
+        #   with_target_words.sort {|a,b|
+        #     first = -(a[:target_words].size <=> b[:target_words].size)
+        #     first.nonzero? || (a[:chinese].size <=> b[:chinese].size) }
+    end
+
+
+    def select_minimum_necessary_sentences(sentences)
+      with_target_words = add_target_words(sentences)
+      rows              = sort_by_target_word_count(with_target_words)
+
+      selected_rows   = []
+      unmatched_words = @words.dup
+      matched_words   = []
+
+      rows.each do |row|
+        words = row[:target_words].dup
+        # Delete all words from 'words' that have already been encoutered
+        # (and are included in 'matched_words').
+        words = words - matched_words
+
+        if words.size > 0  # Words that where not deleted above have to be part of 'unmatched_words'.
+          selected_rows << row  # Select this row.
+
+          # When a row is selected, its 'words' are no longer unmatched but matched.
+          unmatched_words = unmatched_words - words
+          matched_words   = matched_words + words
+        end
+      end
+      selected_rows
+    end
+
+
+    def remove_keys(hash_array, *keys)
+      hash_array.map { |row| row.delete_keys(*keys) }
+    end
+
+
+    def add_key(hash_array, key, &block)
+      hash_array.map do |row|
+        if block
+          row.merge({key => block.call(row)})
+        else
+          row
+        end
+      end
+    end
+
+
+    def uwc_tag(string)
+      size = string.length
+      case size
+      when 1
+        "1_word"
+      else
+        "#{size}_words"
+      end
+    end
+
+
+    def contains_all_target_words?(selected_rows, sentence_key)
+
+      matched_words = @words.reduce([]) do |acc, word|
+
+        result = selected_rows.find do |row|
+          sentence = row[sentence_key]
+          include_every_char?(word, sentence)
+        end
+
+        if result
+          acc << word
+        end
+
+        acc
+      end
+
+      matched_words.size == @words.size
+    end
+
+
+    # Input:
+    # column: word column number (counting from 1)
+    # row   : Array of the processed CSV data that contains our word column.
+    def self.within_range?(column, row)
+      no_of_cols = row.size
+      column >= 1 && column <= no_of_cols
+    end
+
+
+    def alternate_source(sources, selection)
+      sources = sources.dup
+      sources.delete(selection)
+      sources.pop
+    end
+
+  end
+end

metadata

@@ -0,0 +1,120 @@
+--- !ruby/object:Gem::Specification
+name: chinese_vocab
+version: !ruby/object:Gem::Version
+  version: 0.8.0
+  prerelease: 
+platform: ruby
+authors:
+- Stefan Rohlfing
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-04-13 00:00:00.000000000Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: with_validations
+  requirement: &13638320 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *13638320
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: &13637840 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *13637840
+- !ruby/object:Gem::Dependency
+  name: string_to_pinyin
+  requirement: &13637400 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: *13637400
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: &13636980 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *13636980
+description: ! '=== Chinese::Vocab
+
+  This gem is meant to make live easier for any Chinese language student who:
+
+  * Prefers to learn vocabulary from Chinese sentences.
+
+  * Needs to memorize a lot of words on a _tight_ _time_ _schedule_.
+
+  * Uses the spaced repetition flashcard program {Anki}[http://ankisrs.net/].
+
+
+  Chinese::Vocab addresses all of the above requirements by downloading sentences
+  for each word and
+
+  selecting the *minimum* *required* *number* *of* *Chinese* *sentences* (and English
+  translations)
+
+  to *represent* *all* *words*.
+
+'
+email: stefan.rohlfing@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/chinese.rb
+- lib/chinese/version.rb
+- lib/chinese/core_ext/hash.rb
+- lib/chinese/core_ext/array.rb
+- lib/chinese/core_ext/queue.rb
+- lib/chinese/modules/helper_methods.rb
+- lib/chinese/vocab.rb
+- lib/chinese/scraper.rb
+- README.md
+- Rakefile
+- LICENSE
+- Gemfile
+homepage: http://github.com/bytesource/chinese_vocab
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.9.1
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: chinese_vocab
+rubygems_version: 1.8.15
+signing_key: 
+specification_version: 3
+summary: Chinese::Vocab - Downloading and selecting the minimum required number of
+  sentences to your Chinese vocabulary list
+test_files: []