camertron-eprun 1.1.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 748bd124f6bde8d5560dd6bbf4ff0a5067efb3eb
-  data.tar.gz: b9d6f465088a8d6def6d0adb58f6e4fcbb826786
+SHA256:
+  metadata.gz: 26933843a3512031c7c7d634c1fe12df78d39ee78639f673d1fb175b8df50c21
+  data.tar.gz: 8ef34886023216ee7b39aee4d93b9fe1374c855528af2e5885be4cc4d7dd8ade
 SHA512:
-  metadata.gz: b4fbe32f3ad1084398f0564aba0aedf027e689ece3400f2d19541d5f8be18524bdf68ed67e1d3aae05454244820d7e534fb9d50e44816ee4aeb1f46d16aac8ca
-  data.tar.gz: ad7d2746a641c8e89be10f4e4c1f7072b98e9a47a8bd50079bba9a22577971e4a132cec554e435dbacc70b306dea56d1c1d145f4727d6b1f3558e0f4719fdebc
+  metadata.gz: ce0baa5b5f798e53d8ab831f7e1865665a0aeaf673cf32cf1223dadeef983cd2cf99fc3be5415c2253370e6ae4da7a4546ba403ebb49df70a32b8e794d2aff54
+  data.tar.gz: 06a218b4865a111ebf9696f9bd08b52bdab15a8b1eda0ea0d2250a36ab14856400dd8e11032dbd5f8cacd896238a3f99f507365e0f058165d9bfb5106907e4ea

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+== 2.0.0
+* Removed MRI 1.8 compatibility.
+* Big internal refactor, but same API.
+* Support multiple versions of the Unicode Character Database (UCD), including the latest (v18.0.0).
+* Switch CI from Travis to GitHub Actions.
+* Simplify benchmarks
+  - All the 3rd-party gems used for comparison are very old and use old versions of the UCD.
+  - Now we test against Ruby's normalization implementation, which is actually also written in Ruby and was itself upstreamed from the original eprun by @duerst.
+
+== 1.0.0
+* Repo converted into gem.
+* Added MRI 1.8 compatibility.

data/Gemfile

@@ -4,14 +4,10 @@ gemspec
 
 group :development do
   gem "rake"
-  gem "pry-nav"
-  gem 'unicode'
-  gem 'unf'
-  # gem 'unicode_utils'
-  gem 'activesupport'
+  gem "debug"
+  gem "benchmark-ips"
 end
 
 group :test do
-  gem "rspec"
-  gem "rr"
+  gem "test-unit"
 end

data/README.md

@@ -1,63 +1,90 @@
 Efficient Pure Ruby Unicode Normalization (eprun)
-=================================================
-
-(pronounced e-prune)
-
-The Talk
---------
-
-Please see the
-[Internationalization & Unicode Conference 37](http://www.unicodeconference.org/)
-talk on
-[Implementing Normalization in Pure Ruby - the Fast and Easy Way](http://www.sw.it.aoyama.ac.jp/2013/pub/RubyNorm/).
-
-Directories and Files
----------------------
-
-*   lib/normalize.rb: The core normalization code.
-*   lib/string_normalize.rm: String#normalize.
-*   lib/generate.rb: Generation script, generates lib/normalize_tables.rb
-    from data/UnicodeData.txt and data/CompositionExclusions.txt.
-    This needs to be run only once when updating to a new Unicode version.
-*   lib/normalize_tables.rb: Data used for normalization,
-    automatically generated by lib/generate.rb.
-*   data/: All three files in this directory are downloaded from the
-    [Unicode Character Database](http://www.unicode.org/Public/UCD/latest/ucd/).
-    They are currently at Unicode version 6.3. They need to be updated for
-    a newer Unicode version (happens about once a year).
-*   test/test_normalize.rb: Tests for lib/string_normalize.rb,
-    using data/NormalizationTest.txt.
-*   benchmark/benchmark.rb: Runs the benchmark with example text files.
-    Automatically checks for existing gems/libraries; if e.g. the unicode_util
-    gem is not available, that part of the benchmark is skipped.
-    This also applies to eprun, which will not be run on Ruby 1.8.
-*   benchmark/Deutsch_.txt, Japanese_.txt, Korean_.txt, Vietnamese_.txt:
-    example texts extracted from random Wikipedia pages
-    (see http://en.wikipedia.org/wiki/Wikipedia:Random).
-    The languages are choosen based on number of characters affected
-    by normalization (Deutsch < Japanese < Vietnamese < Korean).
-    These files have somewhat differing lengths,
-    so the results cannot directly be compared across languages.
-    Adding other files with ending "_.txt" will include them in
-    the benchmark.
-*   benchmark/benchmark_results.rb:
-    Results of benchmark for eprun, unicode_utils,
-    ActiveSupport::Multibyte (version 3.0.0), twitter_cldr, and the unicode gem.
-    Eprun, unicode_utils, and unicode normalizations are run 100 times each,
-    ActiveSupport::Multibyte is run 10 times each, and
-    twitter_cldr is run only 1 time (didn't want to wait any longer).
-*   benchmark/benchmark_results_jruby.txt:
-    Results of benchmark when using jruby (excludes unicode gem),
-    version 1.7.4 (1.9.3p392, 2013-05-16 2390d3b on Java HotSpot(TM) Client VM 1.7.0_07-b10 [Windows 7-x86]).
-*   benchmark/benchmark.pl: Runs the benchmark using Perl, both with
-    xsub (i.e. C) version (run 100 times) and pure Perl version
-    (run 10 times).
-*   benchmark/benchmark_results_pl.txt: Results of Perl benchmarks.
-
-TODOs and Ideas
----------------
-*   Publish as a gem, or several gems.
-*   Deal better with encodings other than UTF-8.
-*   Add methods such as String#nfc, String#nfd,...
-*   Add methods for normalization variants.
-*   See [talk](http://www.sw.it.aoyama.ac.jp/2013/pub/RubyNorm/) for more.
+===
+
+(pronounced ee-prune)
+
+## Introduction
+
+This library was originally developed by Martin J. Dürst and Ayumu Nojima, a professor and graduate student at Aoyama Gakuin university in Japan. In 2013, Dürst presented eprun, a fast and efficient pure-ruby Unicode normalization algorithm, at the [International Unicode Conference 37](https://www.unicodeconference.org/iuc37/Conference-Program.pdf) in Santa Clara, California. A textual version of the presentation can be found [here](http://www.sw.it.aoyama.ac.jp/2013/pub/RubyNorm/).
+
+Shorty after @camertron met Dürst at the conference, @camertron packaged up eprun into a Rubygem, published it to rubygems.org, and replaced the original normalization algorithm in [TwitterCLDR](https://github.com/ruby-i18n/twitter-cldr-rb) with eprun.
+
+Dürst, an active member of the Ruby community back in those days, was eventually able to upstream a version of eprun into Ruby itself. The `String#unicode_normalize` method uses a modified version of eprun to perform normalization work.
+
+## Purpose
+
+The version of eprun built into Ruby only supports a single version of the Unicode Character Database (UCD), which evolves over time. TwitterCLDR uses a specific version of the UCD, and needs eprun to use the same one for consistency and compatibility. Other libraries dependent on eprun may also have the same need. For this reason, eprun supports the last 5 UCD versions.
+
+## Usage
+
+First, require the library:
+
+```ruby
+require "eprun"
+```
+
+### Normalizing
+
+Eprun's normalization API consists of two methods. To normalize a string:
+
+```ruby
+Eprun.normalize("abc")
+```
+
+`Eprun.normalize` accepts the normalization form as the second argument. The valid normalization forms are `:nfc` (the default), `:nfd`, `:nfkc`, and `:nfkd`. For more information on Unicode normalization forms, see the [Unicode Standard Annex #15](https://www.unicode.org/reports/tr15/).
+
+### Checking normalization
+
+The second API method tests if a string is already normalized:
+
+```ruby
+Eprun.normalized?("abc")
+```
+
+## Using Different UCD Versions
+
+The currently supported UCD versions can be inspected via the `Eprun::UCD_VERSIONS` constant, which is an array of strings of the form "1.2.3."
+
+To change the version eprun will use during normalization operations, set `Eprun.current_version`.
+
+```ruby
+Eprun.current_version = "17.0.0"
+
+# normalizes using UCD v17
+Eprun.normalize("abc")
+```
+
+There's also a `.with_version` method that changes the current version for the duration of the given block:
+
+```ruby
+Eprun.with_version("17.0.0") do
+  # normalizes using UCD v17
+  Eprun.normalize("abc")
+end
+```
+
+## Running Tests
+
+Run tests by executing:
+
+```shell
+bundle exec rake
+```
+
+## Re-generating Normalization Data
+
+To download the necessary Unicode files, first run:
+
+```shell
+bundle exec rake update_data
+```
+
+Then, to re-generate the normalization tables, regexes, etc, run:
+
+```shell
+bundle exec rake generate_tables
+```
+
+## License
+
+Licensed under the terms of the original eprun library, which is licensed under the same terms as Ruby.

data/Rakefile

@@ -4,37 +4,75 @@
 # available under the same licence as Ruby itself
 # (see http://www.ruby-lang.org/en/LICENSE.txt)
 
-ROOT_DIR = Pathname.new(File.join(File.dirname(__FILE__)))
+ROOT_DIR = Pathname.new(__dir__)
 $:.push(ROOT_DIR.to_s)
 
-require 'eprun/helpers'
-require 'tasks/erb_template'
-require 'tasks/tables_generator'
-
-require 'rubygems/package_task'
+require "eprun"
+require "rubygems/package_task"
 
 task :default => :test
 Bundler::GemHelper.install_tasks
 
 task :test do
-  require 'test/unit'
+  require "test/unit"
+
   files = Dir.glob("./test/test_*.rb")
   runner = Test::Unit::AutoRunner.new(true)
   runner.process_args(files)
-  runner.run
+  succeeded = runner.run
+  exit(succeeded ? 0 : 1)
+end
+
+task :update_data do
+  require "open-uri"
+  require "fileutils"
+
+  files = ["UnicodeData.txt", "NormalizationTest.txt", "CompositionExclusions.txt"]
+
+  Eprun::UCD_VERSIONS.each do |ucd_version_string|
+    base_url = "https://www.unicode.org/Public/#{ucd_version_string}/ucd"
+    ucd_version = Eprun::UcdVersion.get(ucd_version_string)
+
+    files.each do |file|
+      url = "#{base_url}/#{file}"
+      FileUtils.mkdir_p(ucd_version.data_path)
+      File.write(File.join(ucd_version.data_path, file), URI.open(url).read)
+    end
+  end
 end
 
 task :generate_tables do
-  EprunTasks::TablesGenerator.new(
-    ROOT_DIR.join("data").to_s,
-    ROOT_DIR.join("lib", Eprun.require_path).to_s
-  ).generate
+  require "tasks/tables_generator"
+
+  Eprun::UCD_VERSIONS.each do |ucd_version_string|
+    ucd_version = Eprun::UcdVersion.get(ucd_version_string)
+    generator = EprunTasks::TablesGenerator.new(
+      ROOT_DIR.join("data").to_s,
+      ROOT_DIR.join("lib", "eprun").to_s,
+      ucd_version,
+    )
+
+    generator.generate
+  end
 end
 
 task :benchmark do
-  require 'eprun'
-  require ROOT_DIR.join("benchmark/benchmark").to_s
+  require "benchmark/ips"
+
+  deutsch = File.read(File.join(ROOT_DIR, "benchmark", "Deutsch.txt"))
+  japanese = File.read(File.join(ROOT_DIR, "benchmark", "Japanese.txt"))
 
-  Eprun.enable_core_extensions!
-  EprunTasks::Benchmarks.new(ROOT_DIR.join("benchmark".to_s)).run
-end
+  Benchmark.ips do |x|
+    x.report("eprun Deutsch") { Eprun.normalize(deutsch, :nfc) }
+    x.report("ruby Deutsch")  { deutsch.unicode_normalize(:nfc) }
+    x.compare!
+  end
+
+  puts
+
+  Benchmark.ips do |x|
+    x.report("eprun Japanese") { Eprun.normalize(japanese, :nfc) }
+    x.report("ruby Japanese")  { japanese.unicode_normalize(:nfc) }
+    x.compare!
+  end
+end

data/lib/eprun/normalizer.rb

@@ -0,0 +1,215 @@
+# encoding: utf-8
+
+# Copyright 2010-2013 Ayumu Nojima (野島 歩) and Martin J. Dürst (duerst@it.aoyama.ac.jp)
+# available under the same licence as Ruby itself
+# (see http://www.ruby-lang.org/en/LICENSE.txt)
+
+module Eprun
+  class Normalizer
+    ## Constant for max hash capacity to avoid DoS attack
+    MAX_HASH_LENGTH = 18000 # enough for all test cases, otherwise tests get slow
+
+    ## Constants For Hangul
+    # for details such as the meaning of the identifiers below, please see
+    # http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf, pp. 144/145
+    SBASE = 0xAC00
+    LBASE = 0x1100
+    VBASE = 0x1161
+    TBASE = 0x11A7
+    LCOUNT = 19
+    VCOUNT = 21
+    TCOUNT = 28
+    NCOUNT = VCOUNT * TCOUNT
+    SCOUNT = LCOUNT * NCOUNT
+
+    # Unicode-based encodings (except UTF-8)
+    UNICODE_ENCODINGS = [
+      Encoding::UTF_16BE,
+      Encoding::UTF_16LE,
+      Encoding::UTF_32BE,
+      Encoding::UTF_32LE,
+      Encoding::GB18030,
+      Encoding::UCS_2BE,
+      Encoding::UCS_4BE
+    ]
+
+    class << self
+      def get(version = Eprun.current_version)
+        version = Eprun::UcdVersion.get(version)
+        cache[version.string] ||= new(version)
+      end
+
+      private
+
+      def cache
+        @cache ||= {}
+      end
+    end
+
+    attr_reader :version
+
+    def initialize(version)
+      @version = version
+
+      tables = version.namespace::Tables
+
+      @class_table = tables.class_table
+      @composition_table = tables.composition_table
+      @decomposition_table = tables.decomposition_table
+      @kompatible_table = tables.kompatible_table
+
+      ## Regular Expressions and Hash Constants
+      @regexp_d = Regexp.compile(tables.regexp_d_string, Regexp::EXTENDED)
+      @regexp_c = Regexp.compile(tables.regexp_c_string, Regexp::EXTENDED)
+      @regexp_k = Regexp.compile(tables.regexp_k_string, Regexp::EXTENDED)
+      @nf_hash_d = Hash.new do |hash, key|
+        hash.shift if hash.length > MAX_HASH_LENGTH # prevent DoS attack
+        hash[key] = nfd_one(key)
+      end
+
+      @nf_hash_c = Hash.new do |hash, key|
+        hash.shift if hash.length > MAX_HASH_LENGTH # prevent DoS attack
+        hash[key] = nfc_one(key)
+      end
+    end
+
+    ## Hangul Algorithm
+    def hangul_decomp_one(target)
+      syllable_index = target.ord - SBASE
+      return target if syllable_index < 0 || syllable_index >= SCOUNT
+
+      l = LBASE + syllable_index / NCOUNT
+      v = VBASE + (syllable_index % NCOUNT) / TCOUNT
+      t = TBASE + syllable_index % TCOUNT
+      (t == TBASE ? [l, v] : [l, v, t]).pack("U*") + target[1..-1]
+    end
+
+    def hangul_comp_one(string)
+      length = string.length
+
+      if length > 1 && 0 <= (lead = string[0].ord - LBASE) && lead < LCOUNT && 0 <= (vowel = string[1].ord - VBASE) && vowel < VCOUNT
+        lead_vowel = SBASE + (lead * VCOUNT + vowel) * TCOUNT
+        if length > 2 && 0 < (trail = string[2].ord - TBASE) && trail < TCOUNT
+          (lead_vowel + trail).chr(Encoding::UTF_8) + string[3..-1]
+        else
+          lead_vowel.chr(Encoding::UTF_8) + string[2..-1]
+        end
+      else
+        string
+      end
+    end
+
+    ## Canonical Ordering
+    def canonical_ordering_one(string)
+      result = +""
+      unordered = []
+      chars = string.chars
+      n = chars.size
+
+      chars.each_with_index do |char, i|
+        ccc = @class_table[char]
+
+        if ccc == 0
+          unordered.sort!.each { |elem| result << chars[elem % n] }
+          unordered.clear
+          result << char
+        else
+          unordered << ccc * n + i
+        end
+      end
+
+      unordered.sort!.each { |elem| result << chars[elem % n] }
+      result
+    end
+
+    ## Normalization Forms for Patterns (not whole Strings)
+    def nfd_one(string)
+      string = string.chars.map! { |c| @decomposition_table[c] || c }.join("")
+      canonical_ordering_one(hangul_decomp_one(string))
+    end
+
+    def nfc_one(string)
+      nfd_string = nfd_one(string)
+      start = nfd_string[0]
+      last_class = @class_table[start]-1
+      accents = +""
+      result = +""
+
+      nfd_string[1..-1].each_char do |accent|
+        accent_class = @class_table[accent]
+
+        if last_class < accent_class && composite = @composition_table[start + accent]
+          start = composite
+        elsif accent_class == 0
+          result << start << accents
+          start = accent
+          accents = ""
+          last_class = -1
+        else
+          accents << accent
+          last_class = accent_class
+        end
+      end
+
+      hangul_comp_one(result + start + accents)
+    end
+
+    def normalize(string, form = :nfc)
+      encoding = string.encoding
+
+      case encoding
+        when Encoding::UTF_8
+          case form
+          when :nfc
+            string.gsub(@regexp_c, @nf_hash_c)
+          when :nfd
+            string.gsub(@regexp_d, @nf_hash_d)
+          when :nfkc
+            string.gsub(@regexp_k, @kompatible_table).gsub(@regexp_c, @nf_hash_c)
+          when :nfkd
+            string.gsub(@regexp_k, @kompatible_table).gsub(@regexp_d, @nf_hash_d)
+          else
+            raise ArgumentError, "Invalid normalization form #{form}."
+          end
+        when Encoding::US_ASCII
+          string
+        when *UNICODE_ENCODINGS
+          normalize(string.encode(Encoding::UTF_8), form).encode(encoding)
+        else
+          raise Encoding::CompatibilityError, "Unicode Normalization not appropriate for #{encoding}"
+      end
+    end
+
+    def normalized?(string, form = :nfc)
+      encoding = string.encoding
+
+      case encoding
+        when Encoding::UTF_8
+          case form
+          when :nfc
+            string.scan(@regexp_c) do |match|
+              return false if @nf_hash_c[match] != match
+            end
+            true
+          when :nfd
+            string.scan(@regexp_d) do |match|
+              return false if @nf_hash_d[match] != match
+            end
+            true
+          when :nfkc
+            normalized?(string, :nfc) && string !~ @regexp_k
+          when :nfkd
+            normalized?(string, :nfd) && string !~ @regexp_k
+          else
+            raise ArgumentError, "Invalid normalization form #{form}."
+          end
+        when Encoding::US_ASCII
+          true
+        when *UNICODE_ENCODINGS
+          normalized?(string.encode(Encoding::UTF_8), form)
+        else
+          raise Encoding::CompatibilityError, "Unicode Normalization not appropriate for #{encoding}"
+      end
+    end
+  end
+end

data/lib/eprun/ucd_version.rb

@@ -0,0 +1,83 @@
+# encoding: utf-8
+
+# Copyright 2010-2013 Ayumu Nojima (野島 歩) and Martin J. Dürst (duerst@it.aoyama.ac.jp)
+# available under the same licence as Ruby itself
+# (see http://www.ruby-lang.org/en/LICENSE.txt)
+
+module Eprun
+  class UcdVersion
+    class << self
+      def get(str_or_version)
+        return str_or_version if str_or_version.is_a?(self)
+
+        m = str_or_version.match(/v?(\d+)\.(\d+)\.(\d+)/)
+        raise(ArgumentError, "#{str_or_version.inspect} is not a valid UCD version") unless m
+
+        parsed_version = m.captures.join(".")
+
+        if !Eprun::UCD_VERSIONS.include?(parsed_version)
+          raise ArgumentError, "Version #{parsed_version.inspect} is unsupported; please provide a version from `Eprun::UCD_VERSIONS`"
+        end
+
+        numbers = m.captures.map(&:to_i)
+        cache[numbers] ||= new(*numbers)
+      end
+
+      def latest
+        @latest ||= get(::Eprun::UCD_VERSIONS.first)
+      end
+
+      private
+
+      def cache
+        @cache ||= {}
+      end
+    end
+
+    attr_reader :major, :minor, :patch
+
+    def initialize(major, minor, patch)
+      @major = major
+      @minor = minor
+      @patch = patch
+    end
+
+    def data_path
+      @data_path ||= File.expand_path(File.join("..", "..", "data", path_string), __dir__)
+    end
+
+    def composition_exclusions_path
+      @composition_exclusions_path ||= File.join(data_path, "CompositionExclusions.txt")
+    end
+
+    def normalization_test_path
+      @normalization_test_path ||= File.join(data_path, "NormalizationTest.txt")
+    end
+
+    def unicode_data_path
+      @unicode_data_path ||= File.join(data_path, "UnicodeData.txt")
+    end
+
+    def string
+      @string ||= "v#{bare_string}"
+    end
+
+    def bare_string
+      @bare_string ||= "#{major}.#{minor}.#{patch}"
+    end
+
+    def path_string
+      @path_string ||= "v#{major}_#{minor}_#{patch}"
+    end
+
+    alias underscored_string path_string
+
+    def module_string
+      @module_string ||= "V#{major}_#{minor}_#{patch}"
+    end
+
+    def namespace
+      @namespace ||= ::Eprun.const_get(module_string)
+    end
+  end
+end