twitter_cldr 1.8.1 → 1.9.0

data/Gemfile

@@ -8,7 +8,7 @@ end
 
 group :development do
   gem 'nokogiri'
-  gem 'ruby-cldr', :github => 'KL-7/ruby-cldr', :branch => 'for-twitter-cldr'
+  gem 'ruby-cldr', :github => 'svenfuchs/ruby-cldr'
 end
 
 group :test do
@@ -19,4 +19,5 @@ group :test do
   platform :mri_18 do
     gem 'rcov'
   end
-end
+end
+

data/History.txt

@@ -1,6 +1,12 @@
+=== 1.9.0
+
+* Included Unicode-safe YAML dumping support via an adaptation of the ya2yaml gem.
+* Implemented the Unicode Bidirectional Algorithm to help reorder mixed right-to-left and left-to-right text.
+* Added list formatting support.
+
 === 1.8.1
 
-* Improved, more accurate Japanese collation support.
+* Improved, more accurate Finnish and Chinese collation support.
 * Moved JavaScript build environment to twitter-cldr-js.
 
 === 1.8.0

data/README.md

@@ -154,6 +154,27 @@ ts.to_s                         # Vor 1 Tag
 ts.to_s(:unit => :hour)         # Vor 24 Stunden
 ```
 
+### Lists
+
+TwitterCLDR supports formatting lists of strings as you might do in English by using commas, eg: "Apples, cherries, and oranges".  Use the `localize` method on an array followed by a call to `to_sentence`:
+
+```ruby
+["apples", "cherries", "oranges"].localize.to_sentence  # "apples, cherries, and oranges"
+["apples", "cherries", "oranges"].localize(:es).to_sentence  # "apples, cherries y oranges"
+```
+
+Behind the scenes, these convenience methods are creating instances of `ListFormatter`.  You can do the same thing if you're feeling adventurous:
+
+```ruby
+f = TwitterCldr::Formatters::ListFormatter.new(:locale => :en)
+f.format(["Larry", "Curly", "Moe"])  # "Larry, Curly, and Moe"
+
+f = TwitterCldr::Formatters::ListFormatter.new(:locale => :es)
+f.format(["Larry", "Curly", "Moe"])  # "Larry, Curly y Moe"
+```
+
+The TwitterCLDR `ListFormatter` class is smart enough to handle right-to-left (RTL) text and will format the list "backwards" in these cases (note that what looks backwards to English speakers looks frontwards for RTL speakers).  See the section on handling bidirectional text below for more information.
+
 ### Plural Rules
 
 Some languages, like English, have "countable" nouns.  You probably know this concept better as "plural" and "singular", i.e. the difference between "strawberry" and "strawberries".  Other languages, like Russian, have three plural forms: one (numbers ending in 1), few (numbers ending in 2, 3, or 4), and many (everything else).  Still other languages like Japanese don't use countable nouns at all.
@@ -475,6 +496,40 @@ collator.get_sort_key("Älg")             # [39, 61, 51, 1, 134, 157, 6, 1, 143,
 
 **Note**: The TwitterCLDR collator does not currently pass all the collation tests provided by Unicode, but for some strange reasons.  See the [summary](https://gist.github.com/f4ee3bd280a2257c5641) of these discrepancies if you're curious.
 
+### Handling Bidirectional Text
+
+When it comes to displaying text written in both right-to-left (RTL) and left-to-right (LTR) languages, most display systems run into problems.  The trouble is that Arabic or Hebrew text and English text (for example) often get scrambled visually and are therefore difficult to read.  It's not usually the basic ASCII characters like A-Z that get scrambled - it's most often punctuation marks and the like that are confusingly mixed up (they are considered "weak" types by Unicode).
+
+To mitigate this problem, Unicode supports special invisible characters that force visual reordering so that mixed RTL and LTR (called "bidirectional") text renders naturally on the screen.  The Unicode Consortium has developed an algorithm (The Unicode Bidirectional Algorithm, or UBA) that intelligently inserts these control characters where appropriate.  You can make use of the UBA implementation in TwitterCLDR by creating a new instance of `TwitterCldr::Shared::Bidi` using the `from_string` static method, and manipulating it like so:
+
+```ruby
+bidi = TwitterCldr::Shared::Bidi.from_string("hello نزوة world", :direction => :RTL)
+bidi.reorder_visually!
+bidi.to_s
+```
+
+**Disclaimer**: Google Translate tells me the Arabic in the example above means "fancy", but my confidence is not very high, especially since all the letters are unattached. Apologies to any native speakers :)
+
+### Unicode YAML Support
+
+Ruby 1.8 does not come with great Unicode support, and nowhere is this more apparent then when dumping Unicode characters in YAML.  The Psych gem by @tenderlove is a good replacement and is the default in Ruby 1.9, but requires libyaml and still doesn't handle Unicode characters perfectly.  To mitigate this problem (especially in Ruby 1.8), TwitterCLDR contains an adaptation of the [ya2yaml](https://github.com/afunai/ya2yaml) gem by Akira Funai.  Our changes specifically add better dumping of Ruby symbols.  If you can get Mr. Funai's attention, please gently remind him to merge @camertron's pull request so we can use his gem and not have to maintain a separate version :)  Fortunately, YAML parsing can still be done with the usual `YAML.load` or `YAML.load_file`.
+
+You can make use of TwitterCLDR's YAML dumper by calling `localize` and then `to_yaml` on an `Array`, `Hash`, or `String`:
+
+```ruby
+{ :hello => "world" }.localize.to_yaml
+["hello", "world"].localize.to_yaml
+"hello, world".localize.to_yaml
+```
+
+Behind the scenes, these convenience methods are using the `TwitterCldr::Shared::YAML` class.  You can do the same thing if you're feeling adventurous:
+
+```ruby
+TwitterCldr::Shared::YAML.dump({ :hello => "world" })
+TwitterCldr::Shared::YAML.dump(["hello", "world"])
+TwitterCldr::Shared::YAML.dump("hello, world")
+```
+
 ## About Twitter-specific Locales
 
 Twitter tries to always use BCP-47 language codes.  Data from the CLDR doesn't always match those codes however, so TwitterCLDR provides a `convert_locale` method to convert between the two.  All functionality throughout the entire gem defers to `convert_locale` before retrieving CLDR data.  `convert_locale` supports Twitter-supported BCP-47 language codes as well as CLDR locale codes, so you don't have to guess which one to use.  Here are a few examples:

data/lib/twitter_cldr/core_ext.rb

@@ -7,6 +7,6 @@
   TwitterCldr::Localized::LocalizedNumber.localize(klass)
 end
 
-[Array, Date, DateTime, String, Symbol, Time].each do |klass|
+[Array, Date, DateTime, String, Symbol, Time, Hash].each do |klass|
   TwitterCldr::Localized::const_get("Localized#{klass}").localize(klass)
 end

data/lib/twitter_cldr/formatters/list_formatter.rb

@@ -0,0 +1,64 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Formatters
+    class ListFormatter < Base
+
+      attr_accessor :locale
+
+      def initialize(options = {})
+        @locale = TwitterCldr.convert_locale(extract_locale(options))
+      end
+
+      def format(list)
+        if resource.include?(list.size)
+          compose(resource[list.size], list)
+        else
+          compose_list(list)
+        end
+      end
+
+      protected
+
+      def compose_list(list)
+        result = compose(resource[:end] || resource[:middle] || "", [list[-2], list[-1]])
+
+        # Ruby ranges don't support subtraction for some reason (eg. -3..-5).
+        # Instead, we use a positive counter and negate it on array access.
+        (3..list.size).each do |i|
+          format_sym = i == list.size ? :start : :middle
+          format_sym = :middle unless resource.include?(format_sym)
+          result = compose(resource[format_sym] || "", [list[-i], result])
+        end
+
+        result
+      end
+
+      def compose(format, elements)
+        elements.compact!
+
+        if elements.size > 1
+          result = format.dup
+          result.gsub!(/\{(\d+)\}/) { $1 }  # unfortunately "{" and "}" are weak types - must replace
+
+          if TwitterCldr::Shared::Languages.is_rtl?(@locale)
+            result = result.localize.to_reordered_s(:direction => :RTL)
+          end
+
+          result.gsub!(/(\d+)/) { elements[$1.to_i] }
+          result
+        else
+          elements[0] || ""
+        end
+      end
+
+      def resource
+        @resource ||= TwitterCldr.get_locale_resource(@locale, :lists)[@locale][:lists]
+      end
+
+    end
+  end
+end

data/lib/twitter_cldr/formatters.rb

@@ -18,7 +18,9 @@ module TwitterCldr
     autoload :CurrencyFormatter, 'twitter_cldr/formatters/numbers/currency_formatter'
     autoload :PercentFormatter,  'twitter_cldr/formatters/numbers/percent_formatter'
 
-    autoload :Plurals,         'twitter_cldr/formatters/plurals'
-    autoload :PluralFormatter, 'twitter_cldr/formatters/plurals/plural_formatter'
+    autoload :Plurals,           'twitter_cldr/formatters/plurals'
+    autoload :PluralFormatter,   'twitter_cldr/formatters/plurals/plural_formatter'
+
+    autoload :ListFormatter,     'twitter_cldr/formatters/list_formatter'
   end
 end

data/lib/twitter_cldr/localized/localized_array.rb

@@ -7,6 +7,8 @@ module TwitterCldr
   module Localized
 
     class LocalizedArray < LocalizedObject
+      include Enumerable
+
       def code_points_to_string
         TwitterCldr::Utils::CodePoints.to_string(base_obj)
       end
@@ -27,7 +29,23 @@ module TwitterCldr
       def to_a
         @base_obj.dup
       end
+
+      def to_sentence
+        TwitterCldr::Formatters::ListFormatter.new(:locale => locale).format(base_obj)
+      end
+
+      def each
+        if block_given?
+          @base_obj.each { |val| yield val }
+        else
+          @base_obj.to_enum
+        end
+      end
+
+      def to_yaml(options = {})
+        TwitterCldr::Utils::YAML.dump(@base_obj, options)
+      end
     end
 
   end
-end
+end

data/lib/twitter_cldr/localized/localized_hash.rb

@@ -0,0 +1,20 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Localized
+
+    class LocalizedHash < LocalizedObject
+      def to_yaml(options = {})
+        TwitterCldr::Utils::YAML.dump(@base_obj, options)
+      end
+
+      def formatter_const
+        nil
+      end
+    end
+
+  end
+end

data/lib/twitter_cldr/localized/localized_string.rb

@@ -7,6 +7,7 @@ module TwitterCldr
   module Localized
 
     class LocalizedString < LocalizedObject
+      include Enumerable
 
       # Uses wrapped string object as a format specification and returns the result of applying it to +args+ (see
       # +TwitterCldr::Utils.interpolate+ method for interpolation syntax).
@@ -35,7 +36,51 @@ module TwitterCldr
         @base_obj.dup
       end
 
+      def size
+        code_points.size
+      end
+
+      alias :length :size
+
+      def bytesize
+        # bytesize method available in ruby 1.9 only
+        @base_obj.respond_to?(:bytesize) ? @base_obj.bytesize : @base_obj.size
+      end
+
+      def [](index)
+        if index.is_a?(Range)
+          TwitterCldr::Utils::CodePoints.to_string(code_points[index])
+        else
+          TwitterCldr::Utils::CodePoints.to_char(code_points[index])
+        end
+      end
+
+      def each_char
+        if block_given?
+          code_points.each do |code_point|
+            yield TwitterCldr::Utils::CodePoints.to_char(code_point)
+          end
+          @base_obj
+        else
+          code_points.map { |code_point| TwitterCldr::Utils::CodePoints.to_char(code_point) }.to_enum
+        end
+      end
+
+      alias :each :each_char
+
+      def to_yaml(options = {})
+        TwitterCldr::Utils::YAML.dump(@base_obj, options)
+      end
+
+      def to_bidi(options = {})
+        TwitterCldr::Shared::Bidi.from_string(@base_obj, options)
+      end
+
+      def to_reordered_s(options = {})
+        to_bidi(options).reorder_visually!.to_s
+      end
+
     end
 
   end
-end
+end

data/lib/twitter_cldr/localized/localized_symbol.rb

@@ -11,6 +11,10 @@ module TwitterCldr
         TwitterCldr::Shared::Languages.from_code_for_locale(@base_obj, @locale)
       end
 
+      def is_rtl?
+        TwitterCldr::Shared::Languages.is_rtl?(@base_obj)
+      end
+
       def formatter_const
         nil
       end

data/lib/twitter_cldr/localized.rb

@@ -6,6 +6,7 @@
 module TwitterCldr
   module Localized
     autoload :LocalizedArray,    'twitter_cldr/localized/localized_array'
+    autoload :LocalizedHash,    'twitter_cldr/localized/localized_hash'
     autoload :LocalizedDate,     'twitter_cldr/localized/localized_date'
     autoload :LocalizedDateTime, 'twitter_cldr/localized/localized_datetime'
     autoload :LocalizedNumber,   'twitter_cldr/localized/localized_number'

data/lib/twitter_cldr/resources/bidi_test_importer.rb

@@ -0,0 +1,118 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+require 'java'
+java_import "java.lang.Character"
+java_import "classpath.Bidi"
+
+require 'twitter_cldr/resources/download'
+
+module TwitterCldr
+  module Resources
+    # This class should be used with JRuby in 1.9 mode
+
+    class BidiTestImporter
+
+      BIDI_TEST_URL = 'http://unicode.org/Public/6.1.0/ucd/BidiTest.txt'  # this file is about 3.4 MB
+      OUT_FILE = "classpath_bidi_test.txt"
+      DIRECTIONS = [nil, :LTR, :RTL]
+
+      def initialize(working_dir)
+        @working_dir = working_dir
+      end
+
+      def import
+        TwitterCldr::Resources.download_if_necessary(File.join(@working_dir, File.basename(BIDI_TEST_URL)), BIDI_TEST_URL)
+        generate_test
+      end
+
+      protected
+
+      def generate_test
+        run_hash = {}
+
+        File.open(File.join(@working_dir, File.basename(BIDI_TEST_URL)), "r").each_line do |ln|
+          cur_line = ln.strip
+
+          case cur_line[0]
+            when "#", "@"
+              next
+            else
+              input, bitset = cur_line.split("; ")
+
+              expand_bitset_str(bitset).each_with_index do |check, index|
+                if check
+                  types = input.split(" ")
+                  direction = get_java_direction(DIRECTIONS[index])
+                  bidi = Java::Classpath::Bidi.new(types_to_string(types), direction)
+                  levels = types.each_with_index.map { |_, idx| bidi.getLevelAt(idx) }
+                  reorder_arr = Java::Classpath::Bidi.reorderVisually(levels.dup, 0, (0...types.size).to_a, 0, types.size)
+
+                  key = "#{levels.join(" ")} | #{reorder_arr.join(" ")}"
+                  run_hash[key] ||= {}
+                  run_hash[key][input] ||= 0
+                  run_hash[key][input] |= (2 ** index)
+                end
+              end
+          end
+        end
+
+        File.open(File.join(@working_dir, OUT_FILE), "w+") do |out|
+          run_hash.each_pair do |levels_and_reorders, inputs|
+            levels, reorders = levels_and_reorders.split(" | ")
+            out.write("@Levels: #{levels}\n")
+            out.write("@Reorder: #{reorders}\n")
+            inputs.each_pair do |input, bitset|
+              out.write("#{input}; #{bitset}\n")
+            end
+          end
+        end
+      end
+
+      def expand_bitset_str(bitset)
+        bitset.to_i.to_s(2).rjust(3, "0").chars.to_a.map { |i| i == "1" }.reverse
+      end
+
+      def get_java_direction(dir)
+        case dir
+          when :RTL
+            Java::Classpath::Bidi::DIRECTION_RIGHT_TO_LEFT
+          when :LTR
+            Java::Classpath::Bidi::DIRECTION_LEFT_TO_RIGHT
+          else
+            Java::Classpath::Bidi::DIRECTION_DEFAULT_LEFT_TO_RIGHT
+        end
+      end
+
+      def types_to_string(types)
+        @utf_map ||= {
+          "L"   => "\u0041",
+          "LRE" => "\u202a",
+          "LRO" => "\u202d",
+          "R"   => "\u05be",
+          "AL"  => "\u0626",
+          "RLE" => "\u202b",
+          "RLO" => "\u202e",
+          "PDF" => "\u202c",
+          "EN"  => "\u0030",
+          "ET"  => "\u0023",
+          "AN"  => "\u0667",
+          "CS"  => "\u002c",
+          "NSM" => "\u0300",
+          "BN"  => "\u0000",
+          "B"   => "\u0085",
+          "S"   => "\u0009",
+          "WS"  => "\u000c",
+          "ON"  => "\u0021"
+        }
+
+        # java 1.6 and 1.7 report different representative characters for the "ES" bidi class
+        @utf_map["ES"] = (Character.getDirectionality(0x002b) == Character::DIRECTIONALITY_EUROPEAN_NUMBER_SEPARATOR) ? "\u002b" : "\u002f"
+        types.inject("") { |ret, type| ret << @utf_map[type]; ret }
+      end
+
+    end
+  end
+end

data/lib/twitter_cldr/resources/locales_resources_importer.rb

@@ -12,7 +12,7 @@ module TwitterCldr
 
     class LocalesResourcesImporter
 
-      COMPONENTS = %w[calendars languages numbers units plurals]
+      COMPONENTS = %w[calendars languages numbers units plurals lists layout]
 
       # Arguments:
       #

data/lib/twitter_cldr/resources.rb

@@ -16,5 +16,6 @@ module TwitterCldr
     autoload :PostalCodesImporter,            'twitter_cldr/resources/postal_codes_importer'
     autoload :TailoringImporter,              'twitter_cldr/resources/tailoring_importer'
     autoload :UnicodeDataImporter,            'twitter_cldr/resources/unicode_data_importer'
+    autoload :BidiTestImporter,               'twitter_cldr/resources/bidi_test_importer'
   end
 end