wenlin_db_scanner 0.2.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/Gemfile

@@ -0,0 +1,13 @@
+source "http://rubygems.org"
+# Add dependencies required to use your gem here.
+# Example:
+#   gem "activesupport", ">= 2.3.5"
+
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "yard", ">= 0.8.2.1"
+  gem "rdoc", ">= 3.12"
+  gem "bundler", ">= 1.2.0"
+  gem "jeweler", ">= 1.8.4"
+end

data/Gemfile.lock

@@ -0,0 +1,23 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    git (1.2.5)
+    jeweler (1.8.4)
+      bundler (~> 1.0)
+      git (>= 1.2.5)
+      rake
+      rdoc
+    json (1.7.5)
+    rake (0.9.2.2)
+    rdoc (3.12)
+      json (~> 1.4)
+    yard (0.8.2.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (>= 1.2.0)
+  jeweler (>= 1.8.4)
+  rdoc (>= 3.12)
+  yard (>= 0.8.2.1)

data/LICENSE.txt

@@ -0,0 +1,122 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.
+

data/README.md

@@ -0,0 +1,102 @@
+# wenlin_db_scanner
+
+Extracts the data from the Wenlin dictionary program.
+
+The Wenlin Dictionary contains two great databases, the
+[ABC English<->Chinese Dictionary](http://www.wenlin.com/abc.htm) and the
+[Character Description Language](http://www.wenlin.com/cdl/) (CDL).
+
+Unfortunately, this great data is wrapped by a less-than-great UI. This code is
+intended to be useful to Chinese language students who wish to interact with
+the data on their own terms.
+
+
+## Installation
+
+The tool ships as a Ruby gem, and the standard installation process applies.
+The code relies on Ruby 1.9 syntax and String encoding. It was tested to work
+with MRI 1.9.3.
+
+```bash
+gem install wenlin_db_server
+```
+
+
+## Command-Line Usage
+
+The following commands assume that the current directory of your `Terminal` /
+`Command Prompt` is the Wenlin application's main directory. If your current
+directory contains a `W4DB` directory, you're probably in the right place.
+
+### wenlin_dict
+
+Parses a dictionary database into a file containing one JSON line per entry.
+
+```bash
+wenlin_dict W4DB/ en-zh > en_zh.json
+wenlin_dict W4DB/ zh-en > zh_en.json
+wenlin_dict W4DB/ hz-en > hz_en.json
+```
+
+### wenlin_hanzi
+
+Parses the database that breaks down hanzi (Chinese characters) into
+components.
+
+```bash
+wenlin_hanzi W4DB > hanzi.json
+```
+
+### wenlin_parts
+
+Parses a parts-of-speech database into a file containing one JSON line per part
+of speech.
+
+The parts of speech are referenced by the word defintion databases, which use
+their abbreviations.
+
+```bash
+wenlin_parts W4DB/ en > en_parts.json
+wenlin_parts W4DB/ zh > zh_parts.json
+```
+
+### wenlin_dbdump
+
+Extracts the raw text entries in a .db file. Useful for debugging and
+understanding the record format.
+
+```bash
+wenlin_dbdumb W4DB/abc_ce.db
+```
+
+
+## API Usage
+
+The scripts in the `bin` directory are thin wrappers over the API. Read them if
+you want to use the Ruby API directly.
+
+It is very likely that you'll get your job done faster by using the output of
+the CLI tools.
+
+
+## Testing
+
+I test this code by runing the tools inside `bin` against the Wenlin databases,
+and by spot-checking the output.
+
+
+## Contributing
+
+This tool works fairly well on the Wenlin 4 data files. Bugfixes and support
+for new .db file formats are welcome,  other features are most likely outside
+the project's scope.
+
+Note that this tool is designed to help moving the data into another program,
+so it only supports full table scans. Support for random access using the
+B-tree indexes is outside the scope of this project.
+
+
+## Copyright
+
+This code is licensed under the
+[CC0 Public Domain](http://creativecommons.org/publicdomain/zero/1.0/) license.

data/Rakefile

@@ -0,0 +1,36 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "wenlin_db_scanner"
+  gem.homepage = "http://github.com/pwnall/wenlin_db_scanner"
+  gem.license = "CC0"
+  gem.summary = %Q{Extracts the data from the Wenlin dictionary}
+  gem.description = <<END
+The Wenlin dictionary contains two great databases, the ABC English<->Chinese
+dictionary, and the Character Description Language (CDL). Unfortunately, this
+data is wrapped by a less-than-great UI. This gem lets you extract the data so
+you can build your own UI for it.
+END
+  gem.email = "victor@costan.us"
+  gem.authors = ["Victor Costan"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+task :default => :install
+
+require 'yard'
+YARD::Rake::YardocTask.new

data/VERSION

@@ -0,0 +1 @@
+0.2.0

data/bin/wenlin_dbdump

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+# Requires Ruby 1.9, tested on MRI 1.9.3.
+
+require 'wenlin_db_scanner'
+
+unless ARGV.length == 1
+  STDERR.puts "Usage: #{$0} path-to-db-file"
+  exit 1
+end
+
+db = WenlinDbScanner::Db.new ARGV[0]
+db.records.each do |record|
+  puts "---------- record tag: #{record.tag} = 0b#{'%b' % record.tag}"
+
+  if record.binary?
+    puts "---------- binary record, size: #{record.size}"
+    next
+  end
+
+  puts record.text
+  puts "---------- record end"
+end
+db.close
+

data/bin/wenlin_dict

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+# Requires Ruby 1.9, tested on MRI 1.9.3.
+
+require 'json'
+require 'wenlin_db_scanner'
+
+unless ARGV.length == 2
+  STDERR.puts "Usage: #{$0} path-to-db-dir en-zh|zh-en|hz-en"
+  exit 1
+end
+
+case ARGV[1]
+when 'en-zh'
+  entries = WenlinDbScanner::Dicts.en_zh ARGV[0]
+when 'zh-en'
+  entries = WenlinDbScanner::Dicts.zh_en ARGV[0]
+when 'hz-en'
+  entries = WenlinDbScanner::Chars.hz_en ARGV[0]
+else
+  STDERR.puts "Unknown dictionary #{ARGV[1]}\nUse en-zh, zh-en, or hz-en\n"
+  exit 1
+end
+
+entries.each { |entry| puts entry.to_hash.to_json }

data/bin/wenlin_hanzi

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+# Requires Ruby 1.9, tested on MRI 1.9.3.
+
+require 'json'
+require 'wenlin_db_scanner'
+
+unless ARGV.length == 1
+  puts "Usage: #{$0} path-to-db-dir"
+end
+
+chars = WenlinDbScanner::Chars.hanzi ARGV[0]
+
+chars.each { |char| puts char.to_hash.to_json }

data/bin/wenlin_parts

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+# Requires Ruby 1.9, tested on MRI 1.9.3.
+
+require 'json'
+require 'wenlin_db_scanner'
+
+unless ARGV.length == 2
+  STDERR.puts "Usage: #{$0} path-to-db-dir en|zh"
+  exit 1
+end
+
+case ARGV[1]
+when 'en'
+  parts = WenlinDbScanner::SpeechParts.en ARGV[0]
+when 'zh'
+  parts = WenlinDbScanner::SpeechParts.zh ARGV[0]
+else
+  STDERR.puts "Unknown language #{ARGV[1]}\nUse en or zh"
+  exit 1
+end
+
+parts.each { |part| puts part.to_hash.to_json }
+

data/lib/wenlin_db_scanner.rb

@@ -0,0 +1,13 @@
+# Namespace for the Db scanner classes.
+#
+# The awfully long name was chosen on purpose, to save the good names for more
+# useful libraries.
+module WenlinDbScanner
+end
+
+require 'wenlin_db_scanner/chars.rb'
+require 'wenlin_db_scanner/db.rb'
+require 'wenlin_db_scanner/db_record.rb'
+require 'wenlin_db_scanner/dict.rb'
+require 'wenlin_db_scanner/speech_parts.rb'
+

data/lib/wenlin_db_scanner/chars.rb

@@ -0,0 +1,210 @@
+# coding: utf-8
+
+require 'rexml/document'
+
+module WenlinDbScanner
+
+# Parses the data in the character (hanzi) databases.
+module Chars
+  # The entries in the database that breaks down hanzi into components.
+  #
+  # @param [String] db_root the directory containing the .db files
+  # @return [Enumerator<Hash>]
+  def self.hanzi(db_root)
+    _hanzi File.join(db_root, 'cdl.db')
+  end
+
+  # Decoder for a CDL database.
+  #
+  # @param [String] db_file path to the .db file containing CDL data
+  # @return [Enumerator<Hash>]
+  def self._hanzi(db_file)
+    Enumerator.new do |yielder|
+      db = Db.new db_file
+      db.records.each do |record|
+        next if record.binary?
+        xml = REXML::Document.new record.text
+
+        entry = {}
+        xml.root.attributes.each do |name, raw_value|
+          key = name.to_sym
+          entry[key] = cdl_attribute_value key, raw_value
+        end
+
+        entry[:parts] = xml.root.elements.map do |element|
+          part = { part: element.name.to_sym }
+          element.attributes.each do |name, raw_value|
+            key = name.to_sym
+            part[key] = cdl_attribute_value key, raw_value
+          end
+          part
+        end
+
+        yielder << entry
+      end
+    end
+  end
+
+  # Decodes known attributes for CDL XML elements.
+  #
+  # @param [Symbol] key the attribute's name, symbolized
+  # @param [String] value the attribute's value
+  # @return [Integer, Array, String] a more programmer-friendly value
+  def self.cdl_attribute_value(key, raw_value)
+    case key
+    when :points  # coordinates
+      raw_value.split(' ').map do |pair|
+        pair.split(',').map { |coord| coord.strip.to_i }
+      end
+    when :radical  # dictionary radicals?
+      raw_value.strip.split(' ').map(&:strip)
+    when :type  # stroke type
+      raw_value.strip.to_sym
+    when :uni  # unicode value
+      raw_value.strip.to_i(16)
+    else
+      raw_value.strip
+    end
+  end
+
+  # The entries in the hanzi -> English meaning dictionary.
+  #
+  # @param [String] db_root the directory containing the .db files
+  # @return [Enumerator<CharMeaning>]
+  def self.hz_en(db_root)
+    _hz_en File.join(db_root, 'zidian.db')
+  end
+
+  # Decodeder for a database of hanzi -> English meaning entries.
+  #
+  # @param [String] db_file path to the .db file containing dictionary data
+  # @return [Enumerator<DictEntry>]
+  def self._hz_en(db_file)
+    Enumerator.new do |yielder|
+      db = Db.new db_file
+      db.records.each do |record|
+        next if record.binary?
+        lines = record.text.split("\n").map(&:strip).reject(&:empty?)
+
+        header = lines[0]
+
+        entry = CharMeaning.new
+        entry.char = header[0, 1]
+        header = header[1..-1]
+
+        entry.pinyin = header.scan(/\[([^\]]*)\]/).
+                              map { |match| match.first.strip }
+        entry.latin_pinyin =
+            entry.pinyin.map { |pinyin| pinyin_to_latin pinyin }
+        header.gsub!(/\[[^\]]*\]/, '')
+        header.strip!
+
+        header.scan(/\([^\)]+\)/).each do |aside|
+          aside_text = aside[1...-1]
+          case aside_text[0]
+          when '='
+            entry.variants = aside_text[1..-1].chars.to_a
+            header.gsub! aside, ''
+          when '!', '?'
+            entry.related ||= []
+            entry.related += aside_text[1..-1].chars.to_a
+            header.gsub! aside, ''
+          when 'F'
+            entry.complex_forms = aside_text[1..-1].chars.to_a
+            header.gsub! aside, ''
+          when 'S'
+            entry.simplified_forms = aside_text[1..-1].chars.to_a
+            header.gsub! aside, ''
+          when 'u', 'U'
+            if /^Unihan/i =~ aside_text
+              header.gsub! aside, ''
+            end
+          end
+        end
+        header.strip!
+        # Many definitions start with a (note).
+        if note_match = /^\(([^\)]*)\)/.match(header)
+          entry.note = note_match[1]
+          header = header[note_match[0].length..-1].strip
+        end
+        entry.meaning = header.gsub(/\s*<hr\s*\/?>\s*/, "\n")
+
+        lines[1..-1].each do |line|
+          unless line[0] == ?#
+            if entry.note
+              entry.note << "/ #{line}"
+            else
+              entry.note = line
+            end
+            next
+          end
+
+          tag, data = line[1], line[2..-1].strip
+          case 'tag'
+          when 'c'
+            entry.components = data.chars.to_a
+          when 'r'
+            # NOTE: skipping remarks
+          when 'y'
+            entry.cantonese = data
+          end
+        end
+
+        yielder << entry
+      end
+    end
+  end
+
+  # Removes the accents from a pinyin string.
+  #
+  # This computes the closest Latin alphabet string matching the given pinyin
+  # string. It is what users will most likely type to refer to the character,
+  # word or phrase inside the pinyin-spelling string.
+  #
+  # @param [String] pinyin a string that uses pinyin spelling
+  # @return [String] the closest approximation to the given string that only
+  #     uses Latin characters
+  def self.pinyin_to_latin(pinyin)
+    pinyin.tr 'āēīōūǖĀĒĪŌŪǕáéíóúǘÁÉÍÓÚǗǎěǐǒǔǚǍĚǏǑǓǙàèìòùǜÀÈÌÒÙǛüÜ',
+              'aeiouvAEIOUVaeiouvAEIOUVaeiouvAEIOUVaeiouvAEIOUVvV'
+  end
+end  # module WenlinDbScanner::Dicts
+
+# Wraps a record in a dictionary database
+class CharMeaning < Struct.new(:char, :meaning, :note, :pinyin, :variants,
+                               :complex_forms, :simplified_forms,
+                               :components, :cantonese, :related,
+                               :latin_pinyin)
+  # @!attribute [r] char
+  #   @return [String] 1-character string containing the defined character
+  # @!attribute [r] meaning
+  #   @return [String] the character's definition, in English
+  # @!attribute [r] note
+  #   @return [String] e.g., "same as X" or
+  # @!attribute [r] pinyin
+  #   @return [Array<String>] pinyin pronunciation(s) of the character
+  # @!attribute [r] latin_pinyin
+  #   @return [Array<String>] pinyin pronunciation(s) of the character, with
+  #       with the accents removed; this is what users type to get the
+  #       character
+  # @!attribute [r] variants
+  #   @return [Array<String>] other variants of the character
+  # @!attribute [r] related
+  #   @return [Array<String>] characters that are somehow related
+  # @!attribute [r] simplified_forms
+  #   @return [Array<String>] simplified variants of the character
+  # @!attribute [r] complex_forms
+  #   @return [Array<String>] this character is a simplified variant of them
+  # @!attribute [r] components
+  #   @return [Array<String>] 1-character strings with characters that are
+  #       contained in this character's image
+  # @!attribute [r] cantonese
+  #   @return [Array<String>] character's pronunciation in Cantonese
+
+  # @return [Hash]
+  def to_hash
+    Hash[each_pair.reject { |k, v| v.nil? }.to_a]
+  end
+end  # class WenlinDbScanner::DictEntry
+
+end  # namespace WenlinDbScanner