talentbox-unidecoder 1.1.2

data/Changelog.md

@@ -0,0 +1,16 @@
+# Unidecoder Changelog
+
+## 1.1.1 2010-08-11
+
+* Added some missing transliterations.
+* Fixed some other incorrect transliterations.
+
+## 1.1.0 2010-06-23
+
+* Ability to override transliterations.
+* Better Unicode validation.
+* Cleanups and refactoring.
+
+## 1.0.0 2009-02-20
+
+* Unidecoder extracted from Stringex and released as independent gem.

data/README.md

@@ -0,0 +1,51 @@
+# Unidecoder
+
+This library provides methods to transliterate Unicode characters to an ASCII
+approximation.
+
+The functionality in this library was originally written by [Russel Norris](http://github.com/rsl)
+for his [Stringex library](http://github.com/rsl/stringex). This gem is an
+extraction of the Unicode transliteration functionality from Stringex into a
+separate library with some added functionality.
+
+The Unidecoder component of Stringex is itself a port of Sean M. Burke's
+[Unidecode](http://search.cpan.org/dist/Text-Unidecode/lib/Text/Unidecode.pm)
+Perl module.
+
+## Installation
+
+    gem install unidecoder
+
+## Usage
+
+    "olá, mundo!".to_ascii                 #=> "ola, mundo!"
+    "你好".to_ascii                        #=> "Ni Hao "
+    "Jürgen Müller".to_ascii               #=> "Jurgen Muller"
+    "Jürgen Müller".to_ascii("ü" => "ue")  #=> "Juergen Mueller"
+
+## Extra stuff
+
+If you also install either the [Unicode](http://github.com/blackwinter/unicode)
+**or** [Active Support](http://github.com/rails/rails) gems, Unidecoder will
+also perform Unicode normalization before attempting to transliterate strings
+to ASCII.
+
+## Warnings
+
+While this is a neat trick, in practice many transliterations end up being
+fairly useless. For example, all Chinese characters are transliterated to
+Mandarin Chinese. Since Japanese uses Chinese characters writing, but
+pronounces them differently from Mandarin, this makes the transliteration of
+Japanese with this library useless.
+
+Some languages, like Russian, would most correctly transliterate some letters
+based on context, rather than a 1-1 mapping with ASCII. This library does not
+do that.
+
+Other languages, like Hebrew and Arabic, don't write vowels, but assume them
+from context, so the ASCII representation of these langages given by this
+library will look fairly ugly to native speakers.
+
+Basically, your milage may vary. I don't speak every language used by this
+library, so there are certain to be limitations and errors. Your feedback is
+most appreciated!

data/Rakefile

@@ -0,0 +1,28 @@
+require "rake"
+require "rake/testtask"
+require "rake/gempackagetask"
+require "rake/clean"
+
+task :default => :test
+
+CLEAN << "pkg" << "doc" << "coverage" << ".yardoc"
+Rake::GemPackageTask.new(eval(File.read("unidecoder.gemspec"))) { |pkg| }
+Rake::TestTask.new(:test) { |t| t.pattern = "test/**/*_test.rb" }
+
+begin
+  require "yard"
+  YARD::Rake::YardocTask.new do |t|
+    t.options = ["--output-dir=doc"]
+  end
+rescue LoadError
+end
+
+begin
+  require "rcov/rcovtask"
+  Rcov::RcovTask.new do |r|
+    r.test_files = FileList["test/**/*_test.rb"]
+    r.verbose = true
+    r.rcov_opts << "--exclude gems/*"
+  end
+rescue LoadError
+end

data/lib/unidecoder.rb

@@ -0,0 +1,98 @@
+require "yaml"
+
+# Utilities for transliterating UTF-8 strings to ASCII.
+module Unidecoder
+
+  # Contains Unicode codepoints, loading as needed from YAML files.
+  CODEPOINTS = Hash.new { |h, k|
+    h[k] = YAML::load_file(File.expand_path("../unidecoder/data/#{k}.yml", __FILE__))
+  } unless defined?(CODEPOINTS)
+
+  module StringExtensions
+    # Returns string with its UTF-8 characters transliterated to ASCII ones. Example:
+    #
+    #   "⠋⠗⠁⠝⠉⠑".to_ascii #=> "braille"
+    def to_ascii(*args)
+      Unidecoder.decode(self, *args)
+    end
+  end
+
+  extend self
+  # Transliterates UTF-8 characters to ASCII.
+  #
+  # @param [#to_s] string The string or string-like object to transliterate.
+  # @param [Hash] overrides A Hash of UTF-8 to ASCII characters to use in
+  #   place of the defaults.  This can be used for language-specific
+  #   transliterations.
+  #
+  # @example
+  #
+  #     Unidecoder.decode("你好")                        #=> "Ni Hao"
+  #     Unidecoder.decode("Jürgen Müller", "ü" => "ue")  #=> "Juergen Mueller"
+  #     Unidecoder.decode("feliz año", "ñ" => "ni") #=>  #=> "feliz anio"
+  #
+  # @return [String] The transliterated string.
+  def decode(string, overrides = nil)
+    validate_utf8!(string)
+    normalize(string.to_s).gsub(/[^\x00-\x7f]/u) do |char|
+      begin
+        decode_overridden(char, overrides) or decode_char(char)
+      rescue
+        "?"
+      end
+    end
+  end
+
+  def validate_utf8!(string)
+    string.unpack("U*")
+  end
+
+  # Returns a UTF-8 character for the given UTF-8 codepoint
+  def encode(codepoint)
+    [codepoint.to_i(16)].pack("U")
+  end
+
+  # Returns string indicating which file (and line) contains the
+  # transliteration value for the character. This is useful only for
+  # development.
+  def in_yaml_file(character)
+    unpacked = character.unpack("U")[0]
+    "#{code_group(unpacked)}.yml (line #{grouped_point(unpacked) + 2})"
+  end
+
+  def define_normalize(library = nil, &block)
+    return if method_defined? :normalize
+    begin
+      require library if library
+      define_method(:normalize, &block)
+    rescue LoadError
+    end
+  end
+
+  define_normalize("unicode") {|str| Unicode.normalize_C(str)}
+  define_normalize("active_support") {|str| ActiveSupport::Multibyte::Chars.new(str).normalize(:c).to_s}
+  define_normalize {|str| str}
+
+  def decode_char(char)
+    unpacked = char.unpack("U")[0]
+    CODEPOINTS[code_group(unpacked)][grouped_point(unpacked)]
+  end
+
+  def decode_overridden(char, overrides)
+    overrides[char] if overrides
+  end
+
+  # Returns the Unicode codepoint grouping for the given character
+  def code_group(unpacked_character)
+    "x%02x" % (unpacked_character >> 8)
+  end
+
+  # Returns the index of the given character in the YAML file for its codepoint group
+  def grouped_point(unpacked_character)
+    unpacked_character & 255
+  end
+end
+
+class String
+  include Unidecoder::StringExtensions
+end

data/lib/unidecoder/data/x00.yml

@@ -0,0 +1,257 @@
+---
+- "\x00"
+- "\x01"
+- "\x02"
+- "\x03"
+- "\x04"
+- "\x05"
+- "\x06"
+- "\a"
+- "\x08"
+- '  '
+- "\n"
+- "\v"
+- "\f"
+- "\r"
+- "\x0e"
+- "\x0f"
+- "\x10"
+- "\x11"
+- "\x12"
+- "\x13"
+- "\x14"
+- "\x15"
+- "\x16"
+- "\x17"
+- "\x18"
+- "\x19"
+- "\x1a"
+- "\e"
+- "\x1c"
+- "\x1d"
+- "\x1e"
+- "\x1f"
+- ' '
+- '!'
+- '"'
+- '#'
+- $
+- '%'
+- '&'
+- "'"
+- (
+- )
+- '*'
+- +
+- ','
+- '-'
+- .
+- /
+- 0
+- 1
+- 2
+- 3
+- 4
+- 5
+- 6
+- 7
+- 8
+- 9
+- ':'
+- ;
+- <
+- '='
+- '>'
+- '?'
+- '@'
+- A
+- B
+- C
+- D
+- E
+- F
+- G
+- H
+- I
+- J
+- K
+- L
+- M
+- N
+- O
+- P
+- Q
+- R
+- S
+- T
+- U
+- V
+- W
+- X
+- Y
+- Z
+- ']'
+- \
+- ']'
+- '^'
+- _
+- '`'
+- a
+- b
+- c
+- d
+- e
+- f
+- g
+- h
+- i
+- j
+- k
+- l
+- m
+- n
+- o
+- p
+- q
+- r
+- s
+- t
+- u
+- v
+- w
+- x
+- y
+- z
+- '{'
+- '|'
+- '}'
+- '~'
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- "'"
+- "'"
+- '"'
+- '"'
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ''
+- ' '
+- ''
+- C/
+- PS
+- $?
+- Y=
+- '|'
+- SS
+- '"'
+- (c)
+- a
+- '<<'
+- '!'
+- ''
+- (r)
+- '-'
+- deg
+- +-
+- 2
+- 3
+- "'"
+- u
+- P
+- ';'
+- ','
+- 1
+- o
+- '>>'
+- 1/4
+- 1/2
+- 3/4
+- ''
+- A
+- A
+- A
+- A
+- A
+- A
+- AE
+- C
+- E
+- E
+- E
+- E
+- I
+- I
+- I
+- I
+- D
+- N
+- O
+- O
+- O
+- O
+- O
+- x
+- O
+- U
+- U
+- U
+- U
+- Y
+- Th
+- ss
+- a
+- a
+- a
+- a
+- a
+- a
+- ae
+- c
+- e
+- e
+- e
+- e
+- i
+- i
+- i
+- i
+- d
+- n
+- o
+- o
+- o
+- o
+- o
+- /
+- o
+- u
+- u
+- u
+- u
+- y
+- th
+- y