twitter_cldr 1.2.0 → 1.3.0

data/README.md

@@ -48,9 +48,9 @@ Fixnum, Bignum, and Float objects are supported.  Here are some examples:
 1337.localize(:es).to_s                                    # 1.337
 
 # currencies, default USD
-1337.localize(:es).to_currency.to_s                        # $1.337,00
-1337.localize(:es).to_currency.to_s(:currency => "EUR")    # € 1.337,00
-1337.localize(:es).to_currency.to_s(:currency => "Peru")   # S/. 1.337,00
+1337.localize(:es).to_currency.to_s                        # 1.337,00 $
+1337.localize(:es).to_currency.to_s(:currency => "EUR")    # 1.337,00 €
+1337.localize(:es).to_currency.to_s(:currency => "Peru")   # 1.337,00 S/.
 
 # percentages
 1337.localize(:es).to_percent.to_s                         # 1.337%
@@ -92,10 +92,10 @@ TwitterCldr::Shared::Currencies.for_code("CAD")            # { :currency => "Dol
 Date, Time, and DateTime objects are supported:
 
 ```ruby
-DateTime.now.localize(:es).to_full_s                    # "21:44:57 UTC -0800 lunes 12 de diciembre de 2011"
-DateTime.now.localize(:es).to_long_s                    # "21:45:42 -08:00 12 de diciembre de 2011"
-DateTime.now.localize(:es).to_medium_s                  # "21:46:09 12/12/2011"
-DateTime.now.localize(:es).to_short_s                   # "21:47 12/12/11"
+DateTime.now.localize(:es).to_full_s                    # "lunes, 12 de diciembre de 2011 21:44:57 UTC -0800"
+DateTime.now.localize(:es).to_long_s                    # "12 de diciembre de 2011 21:44:57 -08:00"
+DateTime.now.localize(:es).to_medium_s                  # "12/12/2011 21:44:57"
+DateTime.now.localize(:es).to_short_s                   # "12/12/11 21:44"
 
 Date.today.localize(:es).to_full_s                      # "lunes 12 de diciembre de 2011"
 Date.today.localize(:es).to_long_s                      # "12 de diciembre de 2011"
@@ -103,9 +103,9 @@ Date.today.localize(:es).to_medium_s                    # "12/12/2011"
 Date.today.localize(:es).to_short_s                     # "12/12/11"
 
 Time.now.localize(:es).to_full_s                        # "21:44:57 UTC -0800"
-Time.now.localize(:es).to_long_s                        # "21:45:42 -08:00"
-Time.now.localize(:es).to_medium_s                      # "21:46:09"
-Time.now.localize(:es).to_short_s                       # "21:47"
+Time.now.localize(:es).to_long_s                        # "21:44:57 UTC"
+Time.now.localize(:es).to_medium_s                      # "21:44:57"
+Time.now.localize(:es).to_short_s                       # "21:44"
 ```
 
 The CLDR data set only includes 4 specific date formats, full, long, medium, and short, so you'll have to choose amongst them for the one that best fits your needs.  Yes, it's limiting, but the 4 formats get the job done most of the time :)
@@ -183,7 +183,7 @@ The `LocalizedString` class supports all forms of interpolation and combines sup
 "five euros plus %.3f in tax" % (13.25 * 0.087)
 
 # Ruby 1.9
-"five euros plus %3.f in tax" % (13.25 * 0.087)
+"five euros plus %.3f in tax" % (13.25 * 0.087)
 "there are %{count} horses in the barn" % { :count => "5" }
 
 # with TwitterCLDR
@@ -251,34 +251,50 @@ code_point.combining_class  # "0"
 Convert characters to code points:
 
 ```ruby
-TwitterCldr::Normalizers::Base.char_to_code_point("¿")  # "00BF"
+TwitterCldr::Utils::CodePoints.from_string("¿")  # ["00BF"]
 ```
 
 Convert code points to characters:
 
 ```ruby
-TwitterCldr::Normalizers::Base.code_point_to_char("00BF")  # "¿"
+TwitterCldr::Utils::CodePoints.to_string(["00BF"])  # "¿"
 ```
 
-Normalize/decompose a Unicode string (NFD only for now).  Note that the normalized/decomposed string will almost always look the same as the original string because most character display systems automatically combine decomposed characters.
+Normalize/decompose a Unicode string (NFD, NFKD implementations available).  Note that the normalized string will almost always look the same as the original string because most character display systems automatically combine decomposed characters.
 
 ```ruby
 TwitterCldr::Normalizers::NFD.normalize("français")  # "français"
-TwitterCldr::Normalizers::NFD.decompose("español")   # "español"
 ```
 
-Normalization and decomposition are easier to see in hex:
+Normalization is easier to see in hex:
 
 ```ruby
 # ["0065", "0073", "0070", "0061", "00F1", "006F", "006C"]
-code_points = "español".split('').map { |char| TwitterCldr::Normalizers::NFD.char_to_code_point(char) }
+TwitterCldr::Utils::CodePoints.from_string("español")
 
 # ["0065", "0073", "0070", "0061", "006E", "0303", "006F", "006C"]
-TwitterCldr::Normalizers::NFD.normalize_code_points(code_points)
+TwitterCldr::Utils::CodePoints.from_string(TwitterCldr::Normalizers::NFD.normalize("español"))
 ```
 
 Notice in the example above that the letter "ñ" was transformed from `00F1` to `006E 0303`, which represent the "n" and the "˜" respectively.
 
+A few convenience methods also exist for `String` that make it easy to normalize and get code points for strings:
+
+```ruby
+# ["0065", "0073", "0070", "0061", "00F1", "006F", "006C"]
+"español".localize.code_points
+
+# ["0065", "0073", "0070", "0061", "006E", "0303", "006F", "006C"]
+"español".localize.normalize.code_points
+```
+
+Specify a specific normalization algorithm via the :using option.  Currently, only NFD and NFKD are supported (default is NFD):
+
+```ruby
+# ["0065", "0073", "0070", "0061", "006E", "0303", "006F", "006C"]
+"español".localize.normalize(:using => :NFKD).code_points
+```
+
 ## About Twitter-specific Locales
 
 Twitter tries to always use BCP-47 language codes.  Data from the CLDR doesn't always match those codes, so TwitterCLDR provides a `convert_locale` method to convert between the two.  All functionality throughout the entire gem defers to `convert_locale` before retrieving CLDR data.  `convert_locale` supports Twitter-supported BCP-47 language codes as well as CLDR locale codes, so you don't have to guess which one to use.  Here are a few examples:

data/lib/twitter_cldr.rb

@@ -24,27 +24,28 @@ module TwitterCldr
 
   extend SingleForwardable
 
+  # version of CLDR that was used for generating YAML files in the resources/ directory
+  CLDR_VERSION = '21.0' # release date: 2012-02-10
+
   DEFAULT_LOCALE = :en
   DEFAULT_CALENDAR_TYPE = :gregorian
-  RESOURCE_DIR = File.join(File.dirname(File.dirname(File.expand_path(__FILE__))), "resources")
 
-  # maps twitter locales to cldr locales
-  TWITTER_LOCALE_MAP = { :msa     => :ms,
-                         :'zh-cn' => :zh,
-                         :'zh-tw' => :'zh-Hant' }
+  RESOURCES_DIR = File.join(File.dirname(File.dirname(File.expand_path(__FILE__))), 'resources')
 
-  @@resources = TwitterCldr::Shared::Resources.new
+  # maps twitter locales to cldr locales
+  TWITTER_LOCALE_MAP = {
+      :msa     => :ms,
+      :'zh-cn' => :zh,
+      :'zh-tw' => :'zh-Hant'
+  }
 
-  def_delegator :resources, :resource_for, :get_resource
+  def_delegator :resources, :get_resource
+  def_delegator :resources, :get_locale_resource
 
   class << self
 
-    def get_resource_file(locale, resource)
-      File.join(RESOURCE_DIR, convert_locale(locale).to_s, "#{resource}.yml")
-    end
-
     def resources
-      @@resources
+      @resources ||= TwitterCldr::Shared::Resources.new
     end
 
     def get_locale
@@ -60,18 +61,11 @@ module TwitterCldr
 
     def convert_locale(locale)
       locale = locale.to_sym
-      TWITTER_LOCALE_MAP.include?(locale) ? TWITTER_LOCALE_MAP[locale] : locale
+      TWITTER_LOCALE_MAP.fetch(locale, locale)
     end
 
     def supported_locales
-      unless defined?(@@supported_locales)
-        rejectable = [:shared]
-        @@supported_locales = Dir.glob(File.join(File.dirname(File.dirname(__FILE__)), "resources/*")).map do |file|
-          File.basename(file).to_sym
-        end.reject { |file| rejectable.include?(file) }
-      end
-
-      @@supported_locales
+      @supported_locales ||= Dir.glob(File.join(RESOURCES_DIR, 'locales', '*')).map { |f| File.basename(f).to_sym }
     end
 
     def supported_locale?(locale)

data/lib/twitter_cldr/core_ext.rb

@@ -7,4 +7,5 @@ require 'twitter_cldr/core_ext/numbers/bignum'
 require 'twitter_cldr/core_ext/numbers/fixnum'
 require 'twitter_cldr/core_ext/numbers/float'
 require 'twitter_cldr/core_ext/symbol'
-require 'twitter_cldr/core_ext/string'
+require 'twitter_cldr/core_ext/string'
+require 'twitter_cldr/core_ext/array'

data/lib/twitter_cldr/core_ext/array.rb

@@ -0,0 +1,22 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+class Array
+  def localize(locale = TwitterCldr.get_locale)
+    TwitterCldr::LocalizedArray.new(self, locale)
+  end
+end
+
+module TwitterCldr
+  class LocalizedArray < LocalizedObject
+    def code_points_to_string
+      TwitterCldr::Utils::CodePoints.to_string(self.base_obj)
+    end
+
+    def formatter_const
+      nil
+    end
+  end
+end

data/lib/twitter_cldr/core_ext/numbers/localized_number.rb

@@ -33,7 +33,6 @@ module TwitterCldr
     end
 
     def to_s(options = {})
-      options = { :precision => 0 }.merge(options) if @base_obj.is_a?(Fixnum)
       @formatter.format(@base_obj, options)
     end
 

data/lib/twitter_cldr/core_ext/string.rb

@@ -11,6 +11,7 @@ end
 
 module TwitterCldr
   class LocalizedString < LocalizedObject
+    VALID_NORMALIZERS = [:NFD, :NFKD]
 
     # Uses wrapped string object as a format specification and returns the result of applying it to +args+ (see
     # +TwitterCldr::Utils.interpolate+ method for interpolation syntax).
@@ -27,8 +28,15 @@ module TwitterCldr
       TwitterCldr::Formatters::PluralFormatter
     end
 
-    def normalize
-      LocalizedString.new(TwitterCldr::Normalizers::NFD.normalize(@base_obj), @locale)
+    def normalize(options = {})
+      options[:using] ||= :NFD
+
+      if VALID_NORMALIZERS.include?(options[:using])
+        normalizer_const = TwitterCldr::Normalizers.const_get(options[:using])
+        LocalizedString.new(normalizer_const.normalize(@base_obj), @locale)
+      else
+        raise ArgumentError.new("Invalid normalization form specified with :using option.  Choices are [#{VALID_NORMALIZERS.map(&:to_s).join(", ")}]")
+      end
     end
 
     def code_points

data/lib/twitter_cldr/formatters/calendars/datetime_formatter.rb

@@ -169,7 +169,10 @@ module TwitterCldr
       end
 
       def period(time, pattern, length)
-        @tokenizer.calendar[:periods][time.strftime('%p').downcase.to_sym]
+        # Always use :wide form. Day-period design was changed in CLDR -
+        # http://cldr.unicode.org/development/development-process/design-proposals/day-period-design that means some
+        # major changes are required for a full support of day periods.
+        @tokenizer.calendar[:periods][:format][:wide][time.strftime('%p').downcase.to_sym]
       end
 
       def hour(time, pattern, length)

data/lib/twitter_cldr/formatters/numbers/currency_formatter.rb

@@ -6,8 +6,8 @@
 module TwitterCldr
   module Formatters
     class CurrencyFormatter < NumberFormatter
-      DEFAULT_FORMAT_OPTIONS = { :precision => 2 }
       DEFAULT_CURRENCY_SYMBOL = "$"
+      DEFAULT_PRECISION = 2
 
       def initialize(options = {})
         @tokenizer = TwitterCldr::Tokenizers::NumberTokenizer.new(:locale => self.extract_locale(options), :type => :currency)
@@ -23,7 +23,12 @@ module TwitterCldr
           currency = { :symbol => DEFAULT_CURRENCY_SYMBOL }
         end
 
-        super(number, DEFAULT_FORMAT_OPTIONS.merge(options)).gsub('¤', currency[:symbol])
+        super(number, options).gsub('¤', currency[:symbol])
+      end
+
+      def default_format_options_for(number)
+        precision = precision_from(number)
+        { :precision => precision == 0 ? DEFAULT_PRECISION : precision }
       end
     end
   end

data/lib/twitter_cldr/formatters/numbers/decimal_formatter.rb

@@ -12,11 +12,15 @@ module TwitterCldr
       end
 
       def format(number, options = {})
-        super(Float(number), options)
+        super(number, options)
       rescue TypeError, ArgumentError
         number
       end
 
+      def default_format_options_for(number)
+        { :precision => precision_from(number) }
+      end
+
       protected
 
       def get_tokens(obj, options = {})

data/lib/twitter_cldr/formatters/numbers/number_formatter.rb

@@ -15,7 +15,7 @@ module TwitterCldr
       end
 
       def format(number, options = {})
-        opts = { :precision => precision_from(number) }.merge(options)
+        opts = self.default_format_options_for(number).merge(options)
         prefix, suffix, integer_format, fraction_format = *partition_tokens(self.get_tokens(number, opts))
         int, fraction = parse_number(number, opts)
 

data/lib/twitter_cldr/formatters/numbers/percent_formatter.rb

@@ -7,7 +7,6 @@ module TwitterCldr
   module Formatters
     class PercentFormatter < NumberFormatter
       DEFAULT_PERCENT_SIGN = "%"
-      DEFAULT_FORMAT_OPTIONS = { :precision => 0 }
 
       def initialize(options = {})
         @tokenizer = TwitterCldr::Tokenizers::NumberTokenizer.new(:locale => self.extract_locale(options), :type => :percent)
@@ -15,8 +14,11 @@ module TwitterCldr
       end
 
       def format(number, options = {})
-        opts = DEFAULT_FORMAT_OPTIONS.merge(options)
-        super(number, opts).gsub('¤', @tokenizer.symbols[:percent_sign] || DEFAULT_PERCENT_SIGN)
+        super(number, options).gsub('¤', @tokenizer.symbols[:percent_sign] || DEFAULT_PERCENT_SIGN)
+      end
+
+      def default_format_options_for(number)
+        { :precision => 0 }
       end
     end
   end

data/lib/twitter_cldr/formatters/plurals/rules.rb

@@ -6,20 +6,22 @@
 module TwitterCldr
   module Formatters
     module Plurals
-      class Rules
+      module Rules
+
         class << self
+
           def all
-            all_for(TwitterCldr::get_locale)
+            all_for(TwitterCldr.get_locale)
           end
 
           def all_for(locale)
             locale = TwitterCldr.convert_locale(locale.to_sym)
             get_resource(locale)[locale][:i18n][:plural][:keys]
           rescue
-            []
+            nil
           end
 
-          def rule_for(number, locale = TwitterCldr::get_locale)
+          def rule_for(number, locale = TwitterCldr.get_locale)
             locale = TwitterCldr.convert_locale(locale.to_sym)
             get_resource(locale)[locale][:i18n][:plural][:rule].call(number)
           rescue
@@ -30,9 +32,11 @@ module TwitterCldr
 
           def get_resource(locale)
             locale = TwitterCldr.convert_locale(locale)
-            eval(TwitterCldr.get_resource(locale, "plurals")[locale])
+            eval(TwitterCldr.get_locale_resource(locale, :plurals)[locale])
           end
+
         end
+
       end
     end
   end

data/lib/twitter_cldr/normalizers/nfkd.rb

@@ -114,7 +114,9 @@ module TwitterCldr
           result.map { |cp_with_cc| cp_with_cc[0] }
         end
 
-        # Performs stable sorting of a sequence of [code_point, combining_class] pairs.
+        # Performs stable sorting of a sequence of [code_point, combining_class] pairs. For sorting a regular bubble
+        # sort is used (with a small optimization that stops the algorithm if none of the elements were swapped during
+        # the iteration).
         #
         def stable_sort(code_points_with_cc)
           n = code_points_with_cc.size - 2

data/lib/twitter_cldr/shared.rb

@@ -5,8 +5,10 @@
 
 module TwitterCldr
   module Shared
+    autoload :Calendar,    'twitter_cldr/shared/calendar'
     autoload :Currencies,  'twitter_cldr/shared/currencies'
     autoload :Languages,   'twitter_cldr/shared/languages'
+    autoload :Numbers,     'twitter_cldr/shared/numbers'
     autoload :Resources,   'twitter_cldr/shared/resources'
     autoload :UnicodeData, 'twitter_cldr/shared/unicode_data'
   end

data/lib/twitter_cldr/shared/calendar.rb

@@ -0,0 +1,64 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Shared
+    class Calendar
+
+      DEFAULT_FORMAT = :'stand-alone'
+
+      NAMES_FORMS = [:wide, :narrow, :abbreviated]
+
+      attr_reader :locale, :calendar_type
+
+      def initialize(locale = TwitterCldr.get_locale, calendar_type = TwitterCldr::DEFAULT_CALENDAR_TYPE)
+        @locale = TwitterCldr.convert_locale(locale)
+        @calendar_type = calendar_type
+      end
+
+      def months(names_form = :wide)
+        data = get_with_names_form(:months, names_form)
+        data && data.sort_by{ |m| m.first }.map { |m| m.last }
+      end
+
+      def weekdays(names_form = :wide)
+        get_with_names_form(:days, names_form)
+      end
+
+      private
+
+      def get_with_names_form(data_type, names_form)
+        get_data(data_type, DEFAULT_FORMAT, names_form) if NAMES_FORMS.include?(names_form.to_sym)
+      end
+
+      def get_data(*path)
+        data = traverse_path(calendar_data, path)
+        redirect = parse_redirect(data)
+        redirect ? get_data(*redirect) : data
+      end
+
+      def traverse_path(tree, path)
+        !path.empty? && tree.is_a?(Hash) ? traverse_path(tree[path.first], path[1..-1]) : tree
+      end
+
+      def parse_redirect(data)
+        $1.split('.').map(&:to_sym) if data.is_a?(Symbol) && data.to_s =~ redirect_regexp
+      end
+
+      def redirect_regexp
+        Regexp.new("^calendars\.#{calendar_type}\.(.*)$")
+      end
+
+      def calendar_data
+        @calendar_data ||= traverse_path(resource, [locale, :calendars, calendar_type])
+      end
+
+      def resource
+        TwitterCldr.get_locale_resource(@locale, :calendars)
+      end
+
+    end
+  end
+end