ting 0.3.0 → 0.9.0

data/{README.rdoc → README.md}

@@ -1,4 +1,6 @@
-= Ting
+[![Build Status](https://travis-ci.org/arnebrasseur/ting.png)](https://travis-ci.org/arnebrasseur/ting) [![Code Climate](https://codeclimate.com/github/arnebrasseur/ting.png)](https://codeclimate.com/github/arnebrasseur/ting)
+
+# Ting
 
 Ting can convert between various systems for phonetically
 writing Mandarin Chinese. It can also handle various representation
@@ -8,88 +10,94 @@ to pinyin with tones.
 Hanyu Pinyin, Bopomofo, Wade-Giles, Tongyong Pinyin
 and International Phonetic Alphabet (IPA) are supported.
 
-== SYNOPSIS
+## SYNOPSIS
 
-To parse your strings create a +Reader+ object. Ting.reader() takes two
+To parse your strings create a `Reader` object. Ting.reader() takes two
 parameters : the transliteration format, and the way that tones are represented.
 
 To some extent these can be mixed and matched.
 
-To generate pinyin/wade-giles/etc. create a +Writer+ object. Use Ting.writer()
+To generate pinyin/wade-giles/etc. create a `Writer` object. Use Ting.writer()
 
-=== Formats
+### Formats
 
-* +:hanyu+ Hanyu Pinyin
-* +:zhuyin+ Zhuyin Fuhao (a.k.a. Bopomofo)
-* +:wadegiles+ Wade Giles
-* +:ipa+ International Phonetic Alphabet
-* +:tongyong+ Tongyong Pinyin
+* `:hanyu` Hanyu Pinyin
+* `:zhuyin` Zhuyin Fuhao (a.k.a. Bopomofo)
+* `:wadegiles` Wade Giles
+* `:ipa` International Phonetic Alphabet
+* `:tongyong` Tongyong Pinyin
 
-=== Tones
+### Tones
 
-* +:numbers+ Simply put a number after the syllable, easy to type
-* +:accents+ Use diacritics, follows the Hanyu Pinyin rules, there needs to be at least one vowel to apply this to, not usable with IPA or Bopomofo
-* +:supernum+ Superscript numerals, typically used for Wade-Giles
-* +:marks+ Tone mark after the syllable, typically used for Bopomofo
-* +:ipa+ IPA tone marks
-* +:no_tones+ Use no tones
+* `:numbers` Simply put a number after the syllable, easy to type
+* `:accents` Use diacritics, follows the Hanyu Pinyin rules, there needs to be at least one vowel to apply this to, not usable with IPA or Bopomofo
+* `:supernum` Superscript numerals, typically used for Wade-Giles
+* `:marks` Tone mark after the syllable, typically used for Bopomofo
+* `:ipa` IPA tone marks
+* `:no_tones` Use no tones
 
-== Examples
+## Examples
 
 Parse Hanyu Pinyin
 
+````ruby
    require 'ting'
 
    reader = Ting.reader(:hanyu, :numbers)
-   reader << "wo3 ai4 ni3"
-    # => [<Ting::Syllable <initial=Empty, final=Uo, tone=3>>, 
-    #     <Ting::Syllable <initial=Empty, final=Ai, tone=4>>, 
+   reader.( "wo3 ai4 ni3" )
+    # => [<Ting::Syllable <initial=Empty, final=Uo, tone=3>>,
+    #     <Ting::Syllable <initial=Empty, final=Ai, tone=4>>,
     #     <Ting::Syllable <initial=Ne, final=I, tone=3>>]
+````
 
 Generate Bopomofo
 
+````ruby
    zhuyin = Ting.writer(:zhuyin, :marks)
-   zhuyin << (reader << "wo3 ai4 ni3")
+   zhuyin.(reader.("wo3 ai4 ni3"))
    # => "ㄨㄛˇ ㄞˋ ㄋㄧˇ"
+````
 
 Generate Wade-Giles
 
+````ruby
    wadegiles = Ting.writer(:wadegiles, :supernum)
-   wadegiles << (reader << "qing2 kuang4 ru2 he2")
+   wadegiles.(reader.("qing2 kuang4 ru2 he2"))
    # => "ch`ing² k`uang⁴ ju² ho²"
+````
 
 Generate IPA
 
+````ruby
    ipa = Ting.writer.new(:ipa, :ipa)
-   ipa << (reader << "you3 peng2 zi4 yuan2 fang1 lai2")
+   ipa.(reader.("you3 peng2 zi4 yuan2 fang1 lai2"))
    # => "iou˧˩˧ pʰeŋ˧˥ ts˥˩ yɛn˧˥ faŋ˥˥ lai˧˥"
+````
 
 Since this is such a common use case, a convenience method to add diacritics to pinyin.
 
-   require 'ting/string'
-
-   "wo3 ai4 ni3".pretty_tones
+````ruby
+   Ting.pretty_tones "wo3 ai4 ni3"
    # => "wǒ ài nǐ"
+````
 
 Note that syllables need to be separated by spaces, feeding "peng2you3" to the parser
-does not work. The String#pretty_tones method does handle these things a bit more gracefully.
+does not work. The `String#pretty_tones` method does handle these things a bit more gracefully.
 
 If you need to parse input that does not conform, consider using a regexp to scan for valid
-syllables, then feed the syllables to the parser one by one. Have a look at #pretty_tones for
+syllables, then feed the syllables to the parser one by one. Have a look at `#pretty_tones` for
 an example of how to do this.
 
-== REQUIREMENTS
+## REQUIREMENTS
 
 * none, Ting uses nothing but Ruby
-* $KCODE should probably be set to "UTF8", or you might run into problems.
 
-== INSTALL
+## INSTALL
 
 * gem install ting
 
-== LICENSE
+## LICENSE
 
-Copyright (c) 2004-2010, Arne Brasseur. (http://www.arnebrasseur.net)
+Copyright (c) 2007-2014, Arne Brasseur. (http://www.arnebrasseur.net)
 
 Available as Free Software under the GPLv3 License, see LICENSE.txt for details
-

data/Rakefile

@@ -1,15 +1,28 @@
-require 'rubygems'
-
-require 'rake'
-require 'rake/testtask'
-
-task :default => [:"test:ting"]
-
-namespace "test" do
-  Rake::TestTask.new("ting") do |t|
-    $: << File.dirname(__FILE__) + '/lib'
-    t.pattern = 'test/*.rb'
-    t.verbose = true
-    t.warning = true
-  end
-end
+require 'rubygems'
+require 'pathname'
+
+require 'rake'
+require 'rake/testtask'
+require 'rspec/core/rake_task'
+require 'rubygems/package_task'
+task :default   => :all_tests
+task :all_tests => [:test, :spec]
+
+Rake::TestTask.new(:test) do |t|
+  $: << File.dirname(__FILE__) + '/lib'
+  t.pattern = 'test/*.rb'
+  #t.verbose = true
+  #t.warning = true
+end
+
+RSpec::Core::RakeTask.new(:spec)
+
+spec = Gem::Specification.load(Pathname.glob('*.gemspec').first.to_s)
+Gem::PackageTask.new(spec).define
+
+desc "Push gem to rubygems.org"
+task :push => :gem do
+  sh "git tag v#{Ting::VERSION}"
+  sh "git push --tags"
+  sh "gem push pkg/hexp-#{Ting::VERSION}.gem"
+end

data/TODO

@@ -1,15 +1,16 @@
-- Additional transcription systems
-  - MSP2 (or how do you call that)
-  - Palladiy (To make things interesting)
-  - Gwoyueh
-  - Yale
-
-- Research some rare pinyin syllables : lo, yo ^e, yai
-- Get a definitive answer about ong/ueng/weng
-- Add a README to the data/ directory with info on sources, contents and purposes
-- More tests
-- Add remembering of parameters to cgiform example, other examples
-
-The core lib basically does translation on the syllable level. It can handle strings with syllables nicely seperated by spaces. Successive layers should make it possible to convert a sentence with interpunction into a different system. It should be possible to write compound words together in Hanyu, and have the syllables seperated by dashes when converting to WG. For instance:
-
-Wǒ de péngyǒu, shì dàifu.  =>  Wǒ te p`éng-yǔ, shih tài-fu.
+This used to be a checklist style TODO, but with the pace that this thing evolves, I think it's more interesting to add some reflections.
+
+I wrote this Gem back in 2006. When I look at the code now there are plenty of things I would do different. The code's biggest vice is that it's too 'clever', i.e. opaque. But it gets the job done.
+
+- Additional transcription systems that could be added
+  - MPS2 (or how do you call that)
+  - Gwoyueh
+  - Yale
+
+One of the most interesting things about this undertaking has been learning the fine details of Mandarin romanization, and phonology.
+
+E.g. is 'lo' valid pinyin, or should it be 'luo'? CEDICT has entries for both, with the first being used only for exlamations. But most tables on pinyin won't list 'lo'. How about 'ng'? 'mm'? ...
+
+The core lib basically does translation on the syllable level. It can handle strings with syllables nicely seperated by spaces. Successive layers should make it possible to convert a sentence with interpunction into a different system. It should be possible to write compound words together in Hanyu, and have the syllables seperated by dashes when converting to WG. For instance:
+
+Wǒ de péngyǒu, shì dàifu.  =>  Wǒ te p`éng-yǔ, shih tài-fu.

data/examples/hello.rb

@@ -1,12 +1,12 @@
-$: << File.join(File.dirname(__FILE__), '../lib')
-
-require 'pinyin'
-
-conv1 = Pinyin::Converter.new(:hanyu, :numbers, :wadegiles, :accents)
-conv2 = Pinyin::Converter.new(:wadegiles, :accents, :zhuyin, :marks)
-
-pinyin    = 'wo3 de2 peng2 you3 shi4 dai4 fu'
-wadegiles = conv1 << pinyin
-zhuyin    = conv2 << wadegiles
-
-puts pinyin, wadegiles, zhuyin
+$: << File.join(File.dirname(__FILE__), '../lib')
+
+require 'pinyin'
+
+conv1 = Pinyin::Converter.new(:hanyu, :numbers, :wadegiles, :accents)
+conv2 = Pinyin::Converter.new(:wadegiles, :accents, :zhuyin, :marks)
+
+pinyin    = 'wo3 de2 peng2 you3 shi4 dai4 fu'
+wadegiles = conv1 << pinyin
+zhuyin    = conv2 << wadegiles
+
+puts pinyin, wadegiles, zhuyin

data/lib/ting.rb

@@ -1,13 +1,12 @@
+# -*- coding: utf-8 -*-
+
 # Handle several romanization systems for Mandarin Chinese
 #
 # Author::     Arne Brasseur (arne@arnebrasseur.net)
-# Copyright::  Copyright (c) 2007-2010, Arne Brasseur
+# Copyright::  Copyright (c) 2007-2014, Arne Brasseur
 # Licence::    GNU General Public License, v3
 
-$: << File.dirname(__FILE__)
-
 require 'ting/version'
-require 'ting/support'
 require 'ting/groundwork'
 require 'ting/exception'
 
@@ -16,77 +15,53 @@ require 'ting/conversion'
 require 'ting/conversions'
 require 'ting/conversions/hanyu'
 
+require 'ting/procable'
+require 'ting/reader'
+require 'ting/writer'
+require 'ting/converter'
+
 module Ting
-  class Reader
-    def initialize(conv, tone)
-      @conv = conv.to_s
-      @tone = Tones.const_get tone.to_s.camelcase
-      @cache = {}
-    end
+  class << self
 
-    def parse(str)
-      return @cache[str] ||= Conversions.tokenize(str).map do |s, pos|
-        tone,syll = @tone.pop_tone(s)
-        tsyll = Conversions.parse(@conv,syll)
-        ini, fin = tsyll.initial, tsyll.final
-        unless tone && fin && ini
-          raise ParseError.new(s,pos),"Illegal syllable <#{s}> in input <#{str}> at position #{pos}." 
-        end
-        Syllable.new(ini, fin, tone)
-      end
-    rescue Object => e
-      raise ParseError.new(str,0), "Parsing of #{str.inspect} failed : #{e}"
+    def reader(format, tones)
+      Reader.new(format,tones)
     end
 
-    alias :<< :parse
-  end
-
-  class Writer
-    def initialize(conv, tone)
-      @conv = conv.to_s
-      @tone = Tones.const_get tone.to_s.camelcase
-      @cache = {}
+    def writer(format, tones)
+      Writer.new(format,tones)
     end
 
-    def generate(py)
-      conv=lambda {|syll| @tone.add_tone(Conversions.unparse(@conv,syll),syll.tone)}
-      return @cache[py] ||= if py.respond_to? :map
-        py.map(&conv).join(' ')
-      else
-        conv.call(py)
-      end
+    def from(from, from_tone)
+      Converter.new(from, from_tone, nil, nil)
     end
 
-    alias :<< :generate
-    alias :unparse :generate
-  end
-
-  class Converter
-    def initialize(from, from_tone, to, to_tone)
-      @reader = Reader.new(from, from_tone)
-      @writer = Writer.new(to, to_tone)
+    def camelize(str)
+      str = str.dup
+      str.gsub!(/(?:_+|-+)([a-z])/){ $1.upcase }
+      str.gsub!(/(\A|\s)([a-z])/){ $1 + $2.upcase }
+      str
     end
 
-    def convert(str)
-      @writer.unparse @reader.parse(str)
+    SYLLABLE_CACHE = Hash.new do |hsh, syll|
+      hsh[syll] = Ting.writer(:hanyu, :accents).(
+        Ting.reader(:hanyu, :numbers).(syll.downcase)
+      )
     end
 
-    alias :<< :convert
-  end
-  
-  class <<self
-    READERS={}
-    WRITERS={}
 
-    def reader(format, tones)
-      return READERS[[format, tones]] ||= Reader.new(format,tones)
+    def pretty_tones(string)
+      string.gsub('u:','ü').gsub(/[A-Za-züÜ]{1,5}\d/) do |syll|
+        SYLLABLE_CACHE[syll]
+      end
     end
-    def writer(format, tones)
-      return WRITERS[[format, tones]] ||= Writer.new(format,tones)
+
+    def bpmf(string)
+      string.gsub('u:','ü').scan(/[A-Za-züÜ]{1,5}\d/).map do |m|
+        Ting.writer(:zhuyin, :marks).(
+          Ting.reader(:hanyu, :numbers).(m.downcase)
+        )
+      end.join(' ')
     end
-  end
 
+  end
 end
-
-
-Pinyin = Ting #legacy support

data/lib/ting/conversion.rb

@@ -1,7 +1,9 @@
+# coding: utf-8
+
 module Ting
-  
+
   #
-  # Base class for conversions like Hanyu pinyin, 
+  # Base class for conversions like Hanyu pinyin,
   # Wade-Giles, etc.
   #
   class Conversion
@@ -10,7 +12,7 @@ module Ting
     # For Wade-Giles this is a dash, Hanyu pinyin
     # uses a single quote in certain situations
     attr_reader :syllable_separator
-    
+
     # The tone handling object
     attr_reader :tones
 
@@ -30,7 +32,7 @@ module Ting
       if Tone === tone
         @tone = tone
       else
-        @tone = Ting::Tones.const_get(tone.to_s.camelcase)
+        @tone = Ting::Tones.const_get(Ting.camelize(tone.to_s))
       end
 
       @name = self.class.name.underscore
@@ -48,4 +50,3 @@ module Ting
 
   end
 end
-    

data/lib/ting/conversions.rb

@@ -1,80 +1,88 @@
-require 'csv'
-require 'yaml'
-
-module Ting
-  module Conversions
-    All=[]
-
-    DATA_DIR=File.dirname(__FILE__)+'/data/'
-
-    #Load various representations for initials and finals
-    %w(Initial Final).each do |c|
-      klazz=Ting.const_get c
-      begin
-        CSV.open(DATA_DIR+c.downcase+'.csv', 'r').each do |name, *values|
-          next if name == "name"
-          All << name.to_s unless All.include?(name) || name =~ /standalone/i
-          klazz.class_eval {attr_accessor name.to_sym}
-          values.each_with_index do |v,i|
-            klazz::All[i].send(name+'=', v)
-          end
-        end
-      rescue
-        puts "Bad data in #{c.downcase}.csv : " + $!
-        raise
-      end
-      
-    end
-
-    #Substitution rules 
-    @@rules=YAML::load(IO.read(DATA_DIR+'rules.yaml'))
-
-    def self.parse(type, string)
-      if (final = Final::All.find {|f| f.respond_to?("#{type}_standalone") && f.send("#{type}_standalone") == string})
-        TonelessSyllable.new(Initial::Empty, final)
-      else
-        finals = Final::All.dup
-        finals.unshift(finals.delete(Final::Uo)) #hack : move Uo to the front
-                                                 #otherwise wadegiles parses 'lo' as Le+O rather than Le+Uo
-                                                 #probably better to add a hardcoded 'overrule' table for these cases
-        Initial::All.each do |ini|
-          finals.each do |fin|
-            next                                  if TonelessSyllable.illegal?(ini,fin)
-            return TonelessSyllable.new(ini,fin)  if apply_rules(type, (ini.send(type)||'') + (fin.send(type)||'')) == string
-          end
-        end
-      end
-    end
-    
-    def self.unparse(type, tsyll)
-      if tsyll.initial.send(type)
-        apply_rules(type, tsyll.initial.send(type) + (tsyll.final.send(type) || ''))
-      elsif tsyll.final.respond_to?(type.to_s+'_standalone') && standalone = tsyll.final.send(type.to_s+'_standalone')
-        standalone
-      else
-        apply_rules(type, tsyll.final.send(type))
-      end
-    end
-
-    def self.tokenize(str)
-      returning [] do |ary|
-        str,pos = str.dup, 0
-        while s=str.slice!(/[^' ]*/) and s != ""
-          ary << [s.strip, pos]
-          pos+=s.length
-          str.slice!(/[' ]/)
-        end
-      end
-    end
-    
-    private
-      def self.apply_rules(type, string)
-        returning string.dup do |s|
-          @@rules[type] && @@rules[type].each do |rule|
-            s.gsub!(Regexp.new(rule['match']),rule['subst'])
-          end
-        end
-      end
-
-  end
-end
+# coding: utf-8
+
+require 'csv'
+require 'yaml'
+
+module Ting
+  module Conversions
+    All=[]
+
+    DATA_DIR=File.dirname(__FILE__)+'/data/'
+
+    #Load various representations for initials and finals
+    %w(Initial Final).each do |c|
+      klazz=Ting.const_get c
+      begin
+        CSV.open(DATA_DIR+c.downcase+'.csv', 'r:utf-8').each do |name, *values|
+          next if name == "name"
+          All << name.to_s unless All.include?(name) || name =~ /standalone/i
+          klazz.class_eval {attr_accessor name.to_sym}
+          values.each_with_index do |v,i|
+            klazz::All[i].send(name+'=', v && v.force_encoding('UTF-8'))
+          end
+        end
+      rescue
+        STDERR << "Bad data in #{c.downcase}.csv : #{$!}"
+        raise
+      end
+
+    end
+
+    #Substitution rules
+    @@rules=YAML::load(IO.read(DATA_DIR+'rules.yaml'))
+
+    def self.parse(type, string)
+      capitalized = (string.downcase != string && string.downcase.capitalize == string)
+      string = string.to_s.downcase
+      if (final = Final::All.find {|f| f.respond_to?("#{type}_standalone") && f.send("#{type}_standalone") == string})
+        Syllable.new(Initial::Empty, final, nil, capitalized)
+      else
+        finals = Final::All.dup
+        finals.unshift(finals.delete(Final::Uo)) #hack : move Uo to the front
+                                                 #otherwise wadegiles parses 'lo' as Le+O rather than Le+Uo
+                                                 #probably better to add a hardcoded 'overrule' table for these cases
+        Initial.each do |ini|
+          finals.each do |fin|
+            next if Syllable.illegal?(ini,fin)
+            if string == apply_rules(type, (ini.send(type)||'') + (fin.send(type)||''))
+              return Syllable.new(ini, fin, nil, capitalized)
+            end
+          end
+        end
+        raise "Can't parse `#{string.inspect}'"
+      end
+    end
+
+    def self.unparse(type, tsyll)
+      str = if tsyll.initial.send(type)
+              apply_rules(type, tsyll.initial.send(type) + (tsyll.final.send(type) || ''))
+            elsif tsyll.final.respond_to?(type.to_s+'_standalone') && standalone = tsyll.final.send(type.to_s+'_standalone')
+              standalone
+            else
+              apply_rules(type, tsyll.final.send(type))
+            end
+      (tsyll.capitalized? ? str.capitalize : str).force_encoding('UTF-8')
+    end
+
+    def self.tokenize(str)
+      [].tap do |tokens|
+        str,pos = str.dup, 0
+        while str && token = str[/[^' ]*/]
+          tokens << [token.strip, pos]
+          pos += token.length
+          str = str[/[' ]+(.*)/, 1]
+        end
+      end
+    end
+
+    private
+      def self.apply_rules(type, string)
+        string.dup.tap do |s|
+          @@rules[type] && @@rules[type].each do |rule|
+            s.gsub!(Regexp.new(rule['match']), rule['subst'])
+          end
+        end
+      end
+
+  end
+end