twitter_cldr 1.4.1 → 1.5.0

data/NOTICE

@@ -2,7 +2,7 @@ twitter-cldr-rb is a Ruby implementation of the Common Locale Data Repository
 Copyright (C) 2012 Twitter, Inc.
 
 
-Portions of this gem were borrowed from Sven Fuchs' ruby-cldr gem.  Here is
+Portions of this gem were borrowed from Sven Fuchs' ruby-cldr gem. Here is
 the license that accompanied Mr. Fuchs' code:
 
 Copyright (c) 2009 Sven Fuchs
@@ -117,4 +117,38 @@ the software.
   6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
      IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
      WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-     PURPOSE.
+     PURPOSE.
+
+
+TwitterCldr::Collation::ImplicitCollationElements module was ported from
+the ICU4J library (http://site.icu-project.org). Here is its license:
+
+ICU License - ICU 1.8.1 and later
+
+COPYRIGHT AND PERMISSION NOTICE
+
+Copyright (c) 1995-2012 International Business Machines Corporation and others
+
+All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, provided that
+the above copyright notice(s) and this permission notice appear in all copies
+of the Software and that both the above copyright notice(s) and this permission
+notice appear in supporting documentation.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL
+THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM,
+OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER
+RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
+NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE
+OR PERFORMANCE OF THIS SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall not be
+used in advertising or otherwise to promote the sale, use or other dealings in
+this Software without prior written authorization of the copyright holder.

data/README.md

@@ -300,7 +300,7 @@ TwitterCldr::Utils::CodePoints.to_string(["00BF"])  # "¿"
 Normalize/decompose a Unicode string (NFD, NFKD, NFC, and NFKC implementations available).  Note that the normalized string will almost always look the same as the original string because most character display systems automatically combine decomposed characters.
 
 ```ruby
-TwitterCldr::Normalizers::NFD.normalize("français")  # "français"
+TwitterCldr::Normalization::NFD.normalize("français")  # "français"
 ```
 
 Normalization is easier to see in hex:
@@ -310,7 +310,7 @@ Normalization is easier to see in hex:
 TwitterCldr::Utils::CodePoints.from_string("español")
 
 # ["0065", "0073", "0070", "0061", "006E", "0303", "006F", "006C"]
-TwitterCldr::Utils::CodePoints.from_string(TwitterCldr::Normalizers::NFD.normalize("español"))
+TwitterCldr::Utils::CodePoints.from_string(TwitterCldr::Normalization::NFD.normalize("español"))
 ```
 
 Notice in the example above that the letter "ñ" was transformed from `00F1` to `006E 0303`, which represent the "n" and the "˜" respectively.

data/lib/twitter_cldr/collation/collator.rb

@@ -0,0 +1,143 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Collation
+
+    # Collator uses fractional collation elements table form CLDR to generate sort keys for Unicode strings as well as
+    # compare and sort such strings by generated sort keys.
+    #
+    class Collator
+
+      FRACTIONAL_UCA_SHORT_RESOURCE = 'collation/FractionalUCA_SHORT.txt'
+
+      def sort(strings)
+        strings.map{ |s| [s, comparison_key(s)] }.sort{ |a, b| compare_keys(a[1], b[1]) }.map(&:first)
+      end
+
+      def compare(string_a, string_b)
+        string_a == string_b ? 0 : compare_keys(comparison_key(string_a), comparison_key(string_b))
+      end
+
+      def sort_key(string_or_code_points)
+        sort_key_for_code_points(get_code_points(string_or_code_points))
+      end
+
+      def trie
+        @trie ||= self.class.trie
+      end
+
+      def self.trie
+        @trie ||= TwitterCldr::Collation::TrieBuilder.load_trie(FRACTIONAL_UCA_SHORT_RESOURCE)
+      end
+
+      private
+
+      def comparison_key(string)
+        code_points = TwitterCldr::Utils::CodePoints.from_string(string)
+        { :code_points => code_points, :sort_key => sort_key(code_points) }
+      end
+
+      def compare_keys(a, b)
+        (a[:sort_key] <=> b[:sort_key]).nonzero? || get_integer_code_points(a[:code_points]) <=> get_integer_code_points(b[:code_points])
+      end
+
+      def sort_key_for_code_points(integer_code_points)
+        TwitterCldr::Collation::SortKey.build(get_collation_elements(integer_code_points))
+      end
+
+      def get_integer_code_points(code_points)
+        code_points.map { |code_point| code_point.to_i(16) }
+      end
+
+      def get_collation_elements(integer_code_points)
+        result = []
+        result.concat(code_point_collation_elements(integer_code_points)) until integer_code_points.empty?
+        result
+      end
+
+      def get_code_points(str_or_code_points)
+        code_points = str_or_code_points.is_a?(String) ? TwitterCldr::Utils::CodePoints.from_string(str_or_code_points) : str_or_code_points
+
+        # Normalization makes the collation process significantly slower (like seven times slower on the UCA
+        # non-ignorable test from CollationTest_NON_IGNORABLE.txt). ICU uses some optimizations to apply normalization
+        # only in special, rare cases. We need to investigate possible solutions and do normalization cleverly too.
+        code_points = TwitterCldr::Normalization::NFD.normalize_code_points(code_points)
+
+        get_integer_code_points(code_points)
+      end
+
+      # Returns the first sequence of fractional collation elements for an array of integer code points. Returned value
+      # is an array of well formed (including weights for all significant levels) integer arrays.
+      #
+      # NOTE (side-effect): all used code points are removed from the input array.
+      #
+      def code_point_collation_elements(integer_code_points)
+        explicit_collation_elements(integer_code_points) || implicit_collation_elements(integer_code_points)
+      end
+
+      # Tries to build explicit collation elements array for the longest code points prefix in the given sequence. When
+      # possible, combines this prefix with (not necessarily subsequent) non-starters that follow it in the sequence.
+      # That's necessary because canonical ordering (that is performed during normalization) can break contractions
+      # that existed in the original, de-normalized string.
+      #
+      # NOTE (side-effect): all used code points are removed from the input array.
+      #
+      # For more information see section '4.2 Produce Array' of the main algorithm at http://www.unicode.org/reports/tr10/#Main_Algorithm
+      #
+      def explicit_collation_elements(integer_code_points)
+        # find the longest prefix in the trie
+        collation_elements, suffixes, prefix_size = trie.find_prefix(integer_code_points)
+
+        return unless collation_elements
+
+        # remove prefix from the code points sequence
+        integer_code_points.shift(prefix_size)
+
+        non_starter_pos = 0
+
+        used_combining_classes = {}
+
+        while non_starter_pos < integer_code_points.size && !suffixes.empty?
+          # create a trie from a hash of suffixes available for the chosen prefix
+          subtrie = TwitterCldr::Collation::Trie.new(suffixes)
+
+          # get next code point (possibly non-starter)
+          non_starter_code_point = integer_code_points[non_starter_pos]
+          combining_class        = TwitterCldr::Normalization::Base.combining_class_for(non_starter_code_point.to_s(16))
+
+          # code point is a starter or combining class has been already used (non-starter is 'blocked' from the prefix)
+          break if combining_class == 0 || used_combining_classes[combining_class]
+
+          used_combining_classes[combining_class] = true
+
+          # Try to find collation elements for [prefix + non-starter] code points sequence. As the subtrie contains
+          # suffixes (without prefix) we pass only non-starter itself.
+          new_collation_elements, new_suffixes = subtrie.find_prefix([non_starter_code_point]).first(2)
+
+          if new_collation_elements
+            # non-starter with a collation elements sequence corresponding to [prefix + non-starter] accepted
+            collation_elements = new_collation_elements
+            suffixes           = new_suffixes
+
+            # Remove non-starter from its position in the sequence. Then we can move further from the same position.
+            integer_code_points.delete_at(non_starter_pos)
+          else
+            # move to the next code point
+            non_starter_pos += 1
+          end
+        end
+
+        collation_elements
+      end
+
+      def implicit_collation_elements(integer_code_points)
+        TwitterCldr::Collation::ImplicitCollationElements.for_code_point(integer_code_points.shift)
+      end
+
+    end
+
+  end
+end

data/lib/twitter_cldr/collation/implicit_collation_elements.rb

@@ -0,0 +1,188 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Collation
+
+    # ImplicitCollationElements generates implicit collation elements for code points (including some CJK characters),
+    # that are not explicitly mentioned in the collation elements table.
+    #
+    # This module was ported from the ICU4J library (ImplicitCEGenerator class). See NOTICE file for license information.
+    #
+    module ImplicitCollationElements
+
+      DEFAULT_SECONDARY_AND_TERTIARY = 5
+
+      class << self
+
+        def for_code_point(code_point)
+          [[primary_weight(swapCJK(code_point) + 1), DEFAULT_SECONDARY_AND_TERTIARY, DEFAULT_SECONDARY_AND_TERTIARY]]
+        end
+
+        private
+
+        # Generates the primary weight of the implicit CE for a given code point.
+        #
+        def primary_weight(code_point)
+          byte0 = code_point - MIN_4_BOUNDARY
+
+          if byte0 < 0
+            byte1 = code_point / FINAL_3_COUNT
+            byte0 = code_point % FINAL_3_COUNT
+
+            byte2 = byte1 / MEDIAL_COUNT
+            byte1 %= MEDIAL_COUNT
+
+            # spread out, leaving gap at start
+            byte0 = MIN_TRAIL + byte0 * FINAL_3_MULTIPLIER
+
+            # offset
+            byte1 += MIN_TRAIL
+            byte2 += MIN_PRIMARY
+
+            (byte2 << 16) + (byte1 << 8) + byte0
+          else
+            byte1 = byte0 / FINAL_4_COUNT
+            byte0 %= FINAL_4_COUNT
+
+            byte2 = byte1 / MEDIAL_COUNT
+            byte1 %= MEDIAL_COUNT
+
+            byte3 = byte2 / MEDIAL_COUNT
+            byte2 %= MEDIAL_COUNT
+
+            # spread out, leaving gap at start
+            byte0 = MIN_TRAIL + byte0 * FINAL_4_MULTIPLIER
+
+            # offset
+            byte1 += MIN_TRAIL
+            byte2 += MIN_TRAIL
+            byte3 += MIN_4_PRIMARY
+
+            (byte3 << 24) + (byte2 << 16) + (byte1 << 8) + byte0
+          end
+        end
+
+        # Method used to:
+        #   a) collapse two different Han ranges from UCA into one (in the right order)
+        #   b) bump any non-CJK characters by NON_CJK_OFFSET.
+        #
+        # The relevant blocks are:
+        # A:  4E00..9FFF; CJK Unified Ideographs
+        #     F900..FAFF; CJK Compatibility Ideographs
+        # B:  3400..4DBF; CJK Unified Ideographs Extension A
+        #     20000..XX;  CJK Unified Ideographs Extension B (and others later on)
+        #
+        # As long as
+        #   no new B characters are allocated between 4E00 and FAFF, and
+        #   no new A characters are outside of this range,
+        # (very high probability) this simple code will work.
+        #
+        # The reordered blocks are:
+        #   Block1 is CJK
+        #   Block2 is CJK_COMPAT_USED
+        #   Block3 is CJK_A
+        #   (all contiguous)
+        #
+        # Any other CJK gets its normal code point.
+        #
+        # When we reorder Block1, we make sure that it is at the very start, so that it will use a 3-byte form.
+        #
+        def swapCJK(code_point)
+          if code_point >= CJK_BASE
+            return code_point - CJK_BASE                                      if code_point < CJK_LIMIT
+            return code_point + NON_CJK_OFFSET                                if code_point < CJK_COMPAT_USED_BASE
+            return code_point - CJK_COMPAT_USED_BASE + (CJK_LIMIT - CJK_BASE) if code_point < CJK_COMPAT_USED_LIMIT
+            return code_point + NON_CJK_OFFSET                                if code_point < CJK_B_BASE
+            return code_point                                                 if code_point < CJK_B_LIMIT # non-BMP-CJK
+            return code_point + NON_CJK_OFFSET                                if code_point < CJK_C_BASE
+            return code_point                                                 if code_point < CJK_C_LIMIT # non-BMP-CJK
+            return code_point + NON_CJK_OFFSET                                if code_point < CJK_D_BASE
+            return code_point                                                 if code_point < CJK_D_LIMIT # non-BMP-CJK
+
+            return code_point + NON_CJK_OFFSET # non-CJK
+          end
+
+          return code_point + NON_CJK_OFFSET if code_point < CJK_A_BASE
+          return code_point - CJK_A_BASE + (CJK_LIMIT - CJK_BASE) + (CJK_COMPAT_USED_LIMIT - CJK_COMPAT_USED_BASE) if code_point < CJK_A_LIMIT
+
+          code_point + NON_CJK_OFFSET # non-CJK
+        end
+
+      end
+
+      # primary value
+      MIN_PRIMARY = 0xE0
+      MAX_PRIMARY = 0xe4
+
+      # final byte
+      MIN_TRAIL = 0x04
+      MAX_TRAIL = 0xFE
+
+      # gap for tailoring of 3-byte forms
+      GAP_3 = 1
+
+      # number of 3-byte primaries that can be used
+      PRIMARIES_3_COUNT = 1
+
+      # 2 * [Unicode range] + 2
+      MAX_INPUT = 0x220001
+
+      # medials can use full range
+      MEDIAL_COUNT       = MAX_TRAIL - MIN_TRAIL + 1
+
+      # number of values we can use in trailing bytes
+      # leave room for empty values between AND above, e.g., if gap = 2
+      #   range 3..7 => +3 -4 -5 -6 -7: so 1 value
+      #   range 3..8 => +3 -4 -5 +6 -7 -8: so 2 values
+      #   range 3..9 => +3 -4 -5 +6 -7 -8 -9: so 2 values
+      FINAL_3_MULTIPLIER = GAP_3 + 1
+      FINAL_3_COUNT      = MEDIAL_COUNT / FINAL_3_MULTIPLIER
+
+      # find out how many values fit in each form
+      THREE_BYTE_COUNT = MEDIAL_COUNT * FINAL_3_COUNT
+
+      # now determine where the 3/4 boundary is
+      # we use 3 bytes below the boundary, and 4 above
+      PRIMARIES_AVAILABLE = MAX_PRIMARY - MIN_PRIMARY + 1
+      PRIMARIES_4_COUNT   = PRIMARIES_AVAILABLE - PRIMARIES_3_COUNT
+      MIN_4_PRIMARY       = MIN_PRIMARY + PRIMARIES_3_COUNT
+      MIN_4_BOUNDARY      = PRIMARIES_3_COUNT * THREE_BYTE_COUNT
+
+      TOTAL_NEEDED            = MAX_INPUT - MIN_4_BOUNDARY
+      NEEDED_PER_PRIMARY_BYTE = (TOTAL_NEEDED - 1) / PRIMARIES_4_COUNT + 1
+      NEEDED_PER_FINAL_BYTE   = (NEEDED_PER_PRIMARY_BYTE - 1) / (MEDIAL_COUNT * MEDIAL_COUNT) + 1
+
+      GAP_4 = (MAX_TRAIL - MIN_TRAIL - 1) / NEEDED_PER_FINAL_BYTE
+
+      FINAL_4_MULTIPLIER = GAP_4 + 1
+      FINAL_4_COUNT      = NEEDED_PER_FINAL_BYTE
+
+      # CJK constants
+
+      NON_CJK_OFFSET = 0x110000
+
+      CJK_COMPAT_USED_BASE  = 0xFA0E
+      CJK_COMPAT_USED_LIMIT = 0xFA2F + 1
+
+      CJK_BASE    = 0x4E00        # 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
+      CJK_LIMIT   = 0x9FCC + 1    # 9FCC;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;
+
+      CJK_A_BASE  = 0x3400        # 3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
+      CJK_A_LIMIT = 0x4DB5 + 1    # 4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
+
+      CJK_B_BASE  = 0x20000       # 20000;<CJK Ideograph Extension B, First>;Lo;0;L;;;;;N;;;;;
+      CJK_B_LIMIT = 0x2A6D6 + 1   # 2A6D6;<CJK Ideograph Extension B, Last>;Lo;0;L;;;;;N;;;;;
+
+      CJK_C_BASE  = 0x2A700       # 2A700;<CJK Ideograph Extension C, First>;Lo;0;L;;;;;N;;;;;
+      CJK_C_LIMIT = 0x2B734 + 1   # 2B734;<CJK Ideograph Extension C, Last>;Lo;0;L;;;;;N;;;;;
+
+      CJK_D_BASE  = 0x2B740       # 2B740;<CJK Ideograph Extension D, First>;Lo;0;L;;;;;N;;;;;
+      CJK_D_LIMIT = 0x2B81D + 1   # 2B81D;<CJK Ideograph Extension D, Last>;Lo;0;L;;;;;N;;;;;
+
+    end
+
+  end
+end

data/lib/twitter_cldr/collation/sort_key.rb

@@ -0,0 +1,199 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Collation
+
+    # SortKey builds a collation sort key from an array of collation elements.
+    #
+    class SortKey
+
+      PRIMARY_LEVEL, SECONDARY_LEVEL, TERTIARY_LEVEL = 0, 1, 2
+
+      LEVEL_SEPARATOR = 1 # separate levels in a sort key '01' bytes
+
+      TERTIARY_LEVEL_MASK = 0x3F # mask for removing case bits from tertiary weight ('CC' bits in 'CC00 0000')
+
+      attr_reader :collation_elements
+
+      # Returns a sort key as an array of bytes.
+      #
+      # Arguments:
+      #
+      #   collation_elements - an array of collation elements, represented as arrays of integer weights.
+      #
+      # An instance of the class is created only to prevent passing of @collation_elements and @bytes_array from one
+      # method into another while forming the sort key.
+      #
+      def self.build(collation_elements)
+        new(collation_elements).bytes_array
+      end
+
+      # Arguments:
+      #
+      #   collation_elements - an array of collation elements, represented as arrays of integer weights.
+      #
+      def initialize(collation_elements)
+        @collation_elements = collation_elements
+      end
+
+      def bytes_array
+        @bytes_array ||= build_bytes_array
+      end
+
+      private
+
+      def build_bytes_array
+        @bytes_array = []
+
+        append_primary_bytes
+        append_secondary_bytes
+        append_tertiary_bytes
+
+        @bytes_array
+      end
+
+      def append_primary_bytes
+        @collation_elements.each do |collation_element|
+          append_weight(level_weight(collation_element, PRIMARY_LEVEL))
+        end
+      end
+
+      def append_secondary_bytes
+        @bytes_array << LEVEL_SEPARATOR
+
+        @common_count = 0
+
+        @collation_elements.each do |collation_element|
+          fixnum_to_bytes_array(level_weight(collation_element, SECONDARY_LEVEL)).each do |byte|
+            append_secondary_byte(byte)
+          end
+        end
+
+        # append compressed trailing common bytes
+        append_common_bytes(SECONDARY_BOTTOM, SECONDARY_BOTTOM_COUNT, false) if @common_count > 0
+      end
+
+      def append_tertiary_bytes
+        @bytes_array << LEVEL_SEPARATOR
+
+        @common_count = 0
+
+        @collation_elements.each do |collation_element|
+          fixnum_to_bytes_array(tertiary_weight(collation_element)).each do |byte|
+            append_tertiary_byte(byte)
+          end
+        end
+
+        # append compressed trailing common bytes
+        append_common_bytes(TERTIARY_BOTTOM, TERTIARY_BOTTOM_COUNT, false) if @common_count > 0
+      end
+
+      def append_secondary_byte(secondary)
+        if secondary == SECONDARY_COMMON
+          @common_count += 1
+        else
+          append_with_common_bytes(secondary, SECONDARY_COMMON_SPACE)
+        end
+      end
+
+      def append_tertiary_byte(tertiary)
+        if tertiary == TERTIARY_COMMON
+          @common_count += 1
+        else
+          tertiary += TERTIARY_TOP_ADDITION if tertiary > TERTIARY_COMMON # create a gap above TERTIARY_COMMON
+          append_with_common_bytes(tertiary, TERTIARY_COMMON_SPACE)
+        end
+      end
+
+      def append_with_common_bytes(byte, options)
+        if @common_count > 0
+          if byte < options[:common]
+            append_common_bytes(options[:bottom], options[:bottom_count], false)
+          else
+            append_common_bytes(options[:top], options[:top_count], true)
+          end
+        end
+
+        @bytes_array << byte
+      end
+
+      def append_common_bytes(boundary, count_limit, top)
+        sign = top ? -1 : +1
+
+        while @common_count > count_limit
+          @bytes_array << boundary + sign * count_limit
+          @common_count -= count_limit
+        end
+
+        @bytes_array << boundary + sign * (@common_count - 1)
+        @common_count = 0
+      end
+
+      def tertiary_weight(collation_element)
+        level_weight(collation_element, TERTIARY_LEVEL) & TERTIARY_LEVEL_MASK
+      end
+
+      def append_weight(weight)
+        @bytes_array.concat(fixnum_to_bytes_array(weight))
+      end
+
+      def level_weight(collation_element, level)
+        collation_element[level] || 0
+      end
+
+      def fixnum_to_bytes_array(number)
+        bytes = []
+
+        while number > 0
+          bytes.unshift(number & 0xFF)
+          number >>= 8
+        end
+
+        bytes
+      end
+
+      # Secondary level compression constants
+
+      SECONDARY_BOTTOM       = 0x05
+      SECONDARY_TOP          = 0x86
+      SECONDARY_PROPORTION   = 0.5
+      SECONDARY_COMMON       = SECONDARY_BOTTOM
+      SECONDARY_TOTAL_COUNT  = SECONDARY_TOP - SECONDARY_BOTTOM - 1
+      SECONDARY_TOP_COUNT    = (SECONDARY_PROPORTION * SECONDARY_TOTAL_COUNT).to_i
+      SECONDARY_BOTTOM_COUNT = SECONDARY_TOTAL_COUNT - SECONDARY_TOP_COUNT
+
+      SECONDARY_COMMON_SPACE = {
+          :common       => SECONDARY_COMMON,
+          :bottom       => SECONDARY_BOTTOM,
+          :bottom_count => SECONDARY_BOTTOM_COUNT,
+          :top          => SECONDARY_TOP,
+          :top_count    => SECONDARY_TOP_COUNT
+      }
+
+      # Tertiary level compression constants
+
+      TERTIARY_TOP_ADDITION = 0x80
+
+      TERTIARY_BOTTOM       = 0x05
+      TERTIARY_TOP          = 0x85
+      TERTIARY_PROPORTION   = 0.667
+      TERTIARY_COMMON       = TERTIARY_BOTTOM
+      TERTIARY_TOTAL_COUNT  = TERTIARY_TOP - TERTIARY_BOTTOM - 1
+      TERTIARY_TOP_COUNT    = (TERTIARY_PROPORTION * TERTIARY_TOTAL_COUNT).to_i
+      TERTIARY_BOTTOM_COUNT = TERTIARY_TOTAL_COUNT - TERTIARY_TOP_COUNT
+
+      TERTIARY_COMMON_SPACE = {
+          :common       => TERTIARY_COMMON,
+          :bottom       => TERTIARY_BOTTOM,
+          :bottom_count => TERTIARY_BOTTOM_COUNT,
+          :top          => TERTIARY_TOP,
+          :top_count    => TERTIARY_TOP_COUNT
+      }
+
+    end
+
+  end
+end

data/lib/twitter_cldr/collation/trie.rb

@@ -0,0 +1,73 @@
+# encoding: UTF-8
+
+# Copyright 2012 Twitter, Inc
+# http://www.apache.org/licenses/LICENSE-2.0
+
+module TwitterCldr
+  module Collation
+
+    # This class represents a trie - a tree data structure, also known as a prefix tree.
+    #
+    # Every node corresponds to a single character of the key. To find the value by key one goes down the trie
+    # starting from the root and descending one character at a time. If at some level current node doesn't have a
+    # child corresponding to the next character of the key, then the trie doesn't contain a value with the given key.
+    # Otherwise, the final node, corresponding to the last character of the key, should contain the value. If it's
+    # nil, then the trie doesn't contain a value with the given key (or the value itself is nil).
+    #
+    class Trie
+
+      # Initializes a new trie. If `trie_hash` value is passed it's used as the initial data for the trie. Usually,
+      # `trie_hash` is extracted from other trie and represents its sub-trie.
+      #
+      def initialize(trie_hash = {})
+        @root = [nil, trie_hash]
+      end
+
+      def add(key, value)
+        final = key.inject(@root) do |node, key_element|
+          node[1][key_element] ||= [nil, {}]
+        end
+
+        final[0] = value
+      end
+
+      def get(key)
+        final = key.inject(@root) do |node, key_element|
+          subtree = node[1][key_element]
+          return unless subtree
+          subtree
+        end
+
+        final[0]
+      end
+
+      # Finds the longest substring of the `key` that matches, as a key, a node in the trie.
+      #
+      # Returns a three elements array:
+      #
+      #   1. value in the last node that was visited
+      #   2. sub-trie of this node (as a hash)
+      #   3. size of the `key` prefix that matches this node
+      #
+      def find_prefix(key)
+        prefix_size = 0
+        node = @root
+
+        key.each do |key_element|
+          subtree = node[1][key_element]
+
+          if subtree
+            prefix_size += 1
+            node = subtree
+          else
+            break
+          end
+        end
+
+        node + [prefix_size]
+      end
+
+    end
+
+  end
+end