utf8_utils 1.0.0 → 2.0.0

data/README.md

@@ -2,10 +2,12 @@
 
 This library provides a means of cleaning UTF8 strings with invalid characters.
 
-It provides functionality very similar to [ActiveSupport's `tidy_bytes`
+It provides functionality that replaces [ActiveSupport's `tidy_bytes`
 method](http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/Chars.html#M000977),
-but works for Ruby 1.8.6 - 1.9.x. Once I sort out any potentially embarrassing
-issues with it, I'll probably try patching it into ActiveSupport.
+with a faster algorithm that works on 1.8.6 - 1.9.x.
+
+I will be sending this as a patch to ActiveSupport; in the mean time you can
+access at [its home on Github](github.com/norman/utf8_utils).
 
 ## The Problem
 
@@ -19,35 +21,22 @@ Here's what happens when you try to access a string with invalid UTF-8 character
 
 ## The Solution
 
-    ruby-1.9.1-p378 > "my messed up \x92 string".to_utf8_chars.tidy_bytes.to_s.split(//u)
+    ruby-1.9.1-p378 > "my messed up \x92 string".tidy_bytes.split(//u)
      => ["m", "y", " ", "m", "e", "s", "s", "e", "d", " ", "u", "p", " ", "’", " ", "s", "t", "r", "i", "n", "g"]
 
-Amazing in its brevity and elegance, huh? Ok, maybe not really but if you have
-some badly encoded data you need to clean up, it can save you from ripping out
-your hair.
-
 Note that like ActiveSupport, it naively assumes if you have invalid UTF8
 characters, they are either Windows CP1251 or ISO8859-1. In practice this isn't
 a bad assumption, but may not always work.
 
-Unlike ActiveSupport, however, the performance of this library is **very** poor
-right now.  Since my intention is for this to be used mostly for very short
-strings, it should, however, be good enough for many kinds of applications.
-
-How poor is "very poor?" Have a look:
-
+This library's `tidy_bytes` method is a little less than twice as fast as the
+one provided by ActiveSupport:
 
                                | ACTIVE_SUPPORT | UTF8_UTILS |
     ----------------------------------------------------------
-    tidy bytes           x2000 |          0.087 |      1.225 |
+    tidy bytes          x20000 |          1.008 |      0.650 |
     ==========================================================
-    Total                      |          0.087 |      1.225 |
-
+    Total                      |          1.008 |      0.650 |
 
-This will improve quite a bit soon, as I'm pretty well aware of where the
-slowness is coming from. If performance is important for you now though, by all
-means use another library (if you can find one) until I've made a few more
-releases.
 
 ## Getting it
 
@@ -59,13 +48,11 @@ releases.
     # encoding: utf-8
     require "utf8_utils"
 
-    # Iterate over multibyte characters
-    "hello ーチエンジンの日本".to_utf8_chars.each_char do |char|
-        puts char.valid?
-     end
+    # tidy bytes
+    good_string = bad_string.tidy_bytes
 
-     # tidy bytes
-     good_string = bad_string.to_utf8_chars.tidy_bytes.to_s
+    # tidy bytes in-place
+    string.tidy_bytes!
 
 ## API Docs
 
@@ -73,8 +60,6 @@ releases.
 
 ## Credits
 
-Created by Norman Clarke. Some code was taken from
-[ActiveRecord](http://github.com/rails/rails/tree/master/activesupport/), as
-indicated in the source code.
+Created by Norman Clarke.
 
-Copyright (c) 2010, released under the MIT license.
+Copyright (c) 2010, released under the MIT license.

data/lib/utf8_utils.rb

@@ -1,14 +1,8 @@
-require File.expand_path("../utf8_utils/byte",  __FILE__)
-require File.expand_path("../utf8_utils/char",  __FILE__)
-require File.expand_path("../utf8_utils/chars", __FILE__)
-
-# Wraps a string as an array of bytes and allows some naive cleanup operations
-# as a workaround for Ruby 1.9's crappy encoding support that throws exceptions
-# when attempting to access UTF8 strings with invalid characters.
+# Utilities for cleaning up UTF-8 strings with invalid characters.
 module UTF8Utils
 
-  # CP1251 decimal byte => UTF-8 approximation as an array of bytes
-  CP1251 = {
+  # CP1252 decimal byte => UTF-8 approximation as an array of bytes
+  CP1252 = {
     128 => [226, 130, 172],
     129 => nil,
     130 => [226, 128, 154],
@@ -43,11 +37,80 @@ module UTF8Utils
     159 => [197, 184]
   }
 
-end
+  # A mixin to Ruby's String class to add the {#tidy_bytes} and {#tidy_bytes!}
+  # methods.
+  module StringExt
 
-# Get an array of UTF8 charsfrom a string.
-class String
-  def to_utf8_chars
-    UTF8Utils::Chars.new self
+    # Attempt to replace invalid UTF-8 bytes with valid ones. This method
+    # naively assumes if you have invalid UTF8 bytes, they are either Windows
+    # CP1251 or ISO8859-1. In practice this isn't a bad assumption, but may not
+    # always work.
+    def tidy_bytes
+
+      bytes = unpack("C*")
+      continuation_bytes_expected = 0
+
+      bytes.each_index do |index|
+
+        byte = bytes[index]
+
+        is_continuation_byte = byte[7] == 1 && byte[6] == 0
+        ascii_byte = byte[7] == 0
+        leading_byte = byte[7] == 1 && byte[6] == 1
+
+        if is_continuation_byte
+          if continuation_bytes_expected > 0
+            continuation_bytes_expected = continuation_bytes_expected - 1
+          else
+            # Not expecting a continuation, so clean it
+            bytes[index] = tidy_byte(byte)
+          end
+        # ASCII byte
+        elsif ascii_byte
+          if continuation_bytes_expected > 0
+            # Expected continuation, got ASCII, so clean previous
+            bytes[index - 1] = tidy_byte(bytes[index - 1])
+            continuation_bytes_expected = 0
+          end
+        elsif leading_byte
+          if continuation_bytes_expected > 0
+            # Expected continuation, got leading, so clean previous
+            bytes[index - 1] = tidy_byte(bytes[index - 1])
+            continuation_bytes_expected = 0
+          end
+          continuation_bytes_expected =
+            if    byte[5] == 0 then 1
+            elsif byte[4] == 0 then 2
+            elsif byte[3] == 0 then 3
+          end
+        end
+        # Don't allow the string to terminate with a leading byte
+        if leading_byte && index == bytes.length - 1
+          bytes[index] = tidy_byte(bytes.last)
+        end
+      end
+      bytes.empty? ? "" : bytes.flatten.compact.pack("C*").unpack("U*").pack("U*")
+    end
+
+    # Tidy bytes in-place.
+    def tidy_bytes!
+      replace tidy_bytes
+    end
+
+    private
+
+    def tidy_byte(byte)
+      if UTF8Utils::CP1252.key? byte
+        UTF8Utils::CP1252[byte]
+      elsif byte < 192
+        [194, byte]
+      else
+        [195, byte - 64]
+      end
+    end
   end
 end
+
+class String
+  include UTF8Utils::StringExt
+end

data/lib/utf8_utils/version.rb

@@ -1,6 +1,6 @@
 module UTF8Utils
   module Version
-    MAJOR = 1
+    MAJOR = 2
     MINOR = 0
     TINY  = 0
     STRING = [MAJOR, MINOR, TINY].join('.')

data/test/utf8_utils_test.rb

@@ -1,70 +1,19 @@
 # encoding: utf-8
-
-require "rubygems"
 require "test/unit"
-require "mocha"
 require File.expand_path("../../lib/utf8_utils", __FILE__)
 
-module UTF8ByteTest
-
-  def test_leading_1_bits
-    [0, 128, 194, 224, 240].each_with_index do |n, i|
-      byte = UTF8Utils::Byte.new(n)
-      assert_equal i, byte.leading_1_bits
-    end
-  end
-
-  def test_invalid_bytes
-    [192, 193, 245, 255].each do |n|
-      assert !UTF8Utils::Byte.new(n).valid?
-    end
-  end
-
-  def test_continuation
-    assert UTF8Utils::Byte.new(130).continuation?
-  end
-
-end
-
 class UTF8UtilsTest < Test::Unit::TestCase
 
-  include UTF8ByteTest
-
-  def test_entries_should_be_one_byte_for_ascii_char
-    assert_equal 1, "a".to_utf8_chars.first.length
-  end
-
-  def test_entries_should_be_two_bytes_for_latin_char_with_diacritics
-    assert_equal 2, "¡".to_utf8_chars.first.length
-  end
-
-  def test_entries_should_be_three_bytes_for_basic_multilingual_char
-    assert_equal 3, "आ".to_utf8_chars.first.length
-  end
+  CASES = {
+    "Sim\xF3n Bol\xEDvar" => "Simón Bolívar", # utf-8 leading bytes followed by an ascii char (fix as CP1252)
+    "\xBFhola?" => "¿hola?", # iso-8859-1 inverted question mark
+    "\xFF" => "something"
+  }
 
-  def test_entries_should_be_four_bytes_for_other_chars
-    u = UTF8Utils::Chars.new("")
-    # Editors tend to freak out with chars in this plane, so just stub the
-    # chars field instead. This char is U+10404, DESERET CAPITAL LETTER LONG O.
-    u.stubs(:bytes).returns([240, 144, 144, 132].map { |b| UTF8Utils::Byte.new(b)})
-    assert_equal 4, u.first.length
-  end
-
-  def test_should_detect_valid_chars
-    "cañón आ".to_utf8_chars.each_char {|c| assert c.valid? }
-  end
-
-  def test_should_detect_invalid_chars
-    "\x92".to_utf8_chars.each_char {|c| assert c.invalid? }
-  end
-
-  def test_should_split_correctly_with_invalid_chars
-    assert_equal 3, "a\x92a".to_utf8_chars.entries.length
-  end
-
-  def test_should_tidy_bytes
-    assert_equal "a’a", "a\x92a".to_utf8_chars.tidy_bytes.to_s
-    assert_equal "Simón Bolívar", "Sim\xF3n Bol\xEDvar".to_utf8_chars.tidy_bytes.to_s
+  def test_tidy_bytes
+    CASES.each do |bad, good|
+      assert_equal good, bad.tidy_bytes
+    end
   end
 
-end
+end

metadata

@@ -3,10 +3,10 @@ name: utf8_utils
 version: !ruby/object:Gem::Version 
   prerelease: false
   segments: 
-  - 1
+  - 2
   - 0
   - 0
-  version: 1.0.0
+  version: 2.0.0
 platform: ruby
 authors: 
 - Norman Clarke
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-04-07 00:00:00 -03:00
+date: 2010-04-08 00:00:00 -03:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -38,9 +38,6 @@ extensions: []
 extra_rdoc_files: []
 
 files: 
-- lib/utf8_utils/byte.rb
-- lib/utf8_utils/char.rb
-- lib/utf8_utils/chars.rb
 - lib/utf8_utils/version.rb
 - lib/utf8_utils.rb
 - README.md

data/lib/utf8_utils/byte.rb

@@ -1,86 +0,0 @@
-module UTF8Utils
-
-  # A single UTF-8 byte.
-  class Byte
-
-    attr_reader :byte
-
-    def initialize(byte)
-      @byte = byte
-    end
-
-    def codepoint_mask
-      case leading_1_bits
-      when 0 then 0
-      when 1 then 0b1000_0000
-      when 2 then 0b1100_0000
-      when 3 then 0b1110_0000
-      when 4 then 0b1111_0000
-      end
-    end
-
-    # Is this a continuation byte?
-    def continuation?
-      leading_1_bits == 1
-    end
-
-    # How many continuation bytes should follow this byte?
-    def continuations
-      bits = leading_1_bits
-      bits < 2 ? 0 : bits - 1
-    end
-
-    def invalid?
-      !valid?
-    end
-
-    # From Wikipedia's entry on UTF-8:
-    #
-    # The UTF-8 encoding is variable-width, with each character represented by 1
-    # to 4 bytes. Each byte has 0–4 leading consecutive 1 bits followed by a zero bit
-    # to indicate its type. N 1 bits indicates the first byte in a N-byte sequence,
-    # with the exception that zero 1 bits indicates a one-byte sequence while one 1
-    # bit indicates a continuation byte in a multi-byte sequence (this was done for
-    # ASCII compatibility).
-    # @see http://en.wikipedia.org/wiki/Utf-8
-    def leading_1_bits
-      nibble = byte >> 4
-      if    nibble < 0b1000 then 0 # single-byte chars
-      elsif nibble < 0b1100 then 1 # continuation byte
-      elsif nibble < 0b1110 then 2 # start of 2-byte char
-      elsif nibble < 0b1111 then 3 # 3-byte char
-      else                       4 # 4-byte char
-      end
-    end
-
-    # Start of a 2-byte sequence, but code point ≤ 127
-    # @see http://tools.ietf.org/html/rfc3629
-    def overlong?
-      (192..193) === byte
-    end
-
-    # RFC 3629 reserves 245-253 for the leading bytes of 4-6 byte sequences.
-    # @see http://tools.ietf.org/html/rfc3629
-    def restricted?
-      (245..253) === byte
-    end
-
-    def to_i
-      byte
-    end
-
-    # Bytes 254 and  255 are not defined by the original UTF-8 spec.
-    def undefined?
-      (254..255) === byte
-    end
-
-    def valid?
-      !(overlong? or restricted? or undefined?)
-    end
-
-    def codepoint_bits
-      byte ^ codepoint_mask
-    end
-
-  end
-end

data/lib/utf8_utils/char.rb

@@ -1,52 +0,0 @@
-module UTF8Utils
-
-  class Char < Array
-
-    # Given the first byte, how many bytes long should this character be?
-    def expected_length
-      (first.continuations rescue 0) + 1
-    end
-
-    # Is the character invalid?
-    def invalid?
-      !valid?
-    end
-
-    # Attempt to rescue a valid UTF-8 character from a malformed character. It
-    # will first attempt to convert from CP1251, and if this isn't possible, it
-    # prepends a valid leading byte, treating the character as the last byte in
-    # a two-byte character.  Note that much of the logic here is taken from
-    # ActiveSupport; the difference is that this works for Ruby 1.8.6 - 1.9.1.
-    def tidy
-      return self if valid?
-      byte = first.to_i
-      if UTF8Utils::CP1251.key? byte
-        self.class.new [UTF8Utils::CP1251[byte]]
-      elsif byte < 192
-        self.class.new [194, byte]
-      else
-        self.class.new [195, byte - 64]
-      end
-    end
-
-    # Get a multibyte character from the bytes.
-    def to_s
-      flatten.map {|b| b.to_i }.pack("C*").unpack("U*").pack("U*")
-    end
-
-    def to_codepoint
-      flatten.map {|b| b.to_i }.pack("C*").unpack("U*")[0]
-    end
-
-    def valid?
-      return false if length != expected_length
-      each_with_index do |byte, index|
-        return false if byte.invalid?
-        return false if index == 0 and byte.continuation?
-        return false if index > 0 and !byte.continuation?
-      end
-      true
-    end
-
-  end
-end

data/lib/utf8_utils/chars.rb

@@ -1,59 +0,0 @@
-module UTF8Utils
-  class Chars
-
-    attr :bytes
-    attr :position
-
-    include Enumerable
-
-    def initialize(string)
-      @position = 0
-      begin
-        # Create an array of bytes without raising an ArgumentError in 1.9.x
-        # when the string contains invalid UTF-8 characters
-        @bytes = string.each_byte.map {|b| Byte.new(b)}
-      rescue LocalJumpError
-        # 1.8.6's `each_byte` does not return an Enumerable
-        @bytes = []
-        string.each_byte { |b| @bytes << Byte.new(b) }
-      end
-    end
-
-    # Attempt to clean up malformed characters.
-    def tidy_bytes
-      Chars.new(entries.map {|c| c.tidy.to_s}.compact.join)
-    end
-
-    # Cast to string.
-    def to_s
-      entries.flatten.map {|b| b.to_i }.pack("C*").unpack("U*").pack("U*")
-    end
-
-    def first
-      entries.first
-    end
-
-    private
-
-    def each(&block)
-      while char = next_char
-        yield char
-      end
-      @position = 0
-    end
-
-    alias :each_char :each
-    public :each_char
-
-    def next_char
-      return if !bytes[position]
-      char = Char.new(bytes.slice(position, bytes[position].continuations + 1))
-      if char.invalid?
-        char = Char.new(bytes.slice(position, 1))
-      end
-      @position = position + char.size
-      char unless char.empty?
-    end
-
-  end
-end