gigo 1.2.0 → 1.3.0

data/README.md

@@ -5,9 +5,7 @@ Or better yet, Garbage In, Gold Out! - The GIGO gem aims to fix ruby string enco
 
 The GIGO gem is not likely the proper solutions. If you have bad encodings in your database, you should fix them and write consistent encodings. That said, if you have no other choice, GIGO can help.
 
-This gem depends on one of the many public forks of `CharDet` for ruby. Since `CharDet` is not a public gem and following proper semantic versioning, we have decided to vendor the [kirillrdy/rchardet](http://github.com/kirillrdy/rchardet) repo. We have even made sure that our vendored version stays in our namesacpe by using `GIGO::CharDet`. So if you have another version bundled, feel confident that the two will not conflict.
-
-We use `GIGO::CharDet` to do the grunt work of finding the proper encoding of an untrusted string. Once found, we use the [EnsureValidEncoding](http://github.com/jrochkind/ensure_valid_encoding) gem to either force an encoding while removing any non-convertable characters.
+This gem depends on a series of transcoders including `ActiveSupport::Multibyte#tidy_bytes` along with one of the many public forks of `CharDet` for ruby. Since `CharDet` is not a public gem and following proper semantic versioning, we have decided to vendor the [kirillrdy/rchardet](http://github.com/kirillrdy/rchardet) repo. We have even made sure that our vendored version stays in our namesacpe by using `GIGO::CharDet`. So if you have another version bundled, feel confident that the two will not conflict.
 
 
 ## Usage
@@ -26,6 +24,19 @@ def comments
 end
 ```
 
+GIGO's encoding can be configured using the `GIGO.encoding` accessor. By default this is either `Encoding.default_internal` with a fallback to `Encoding::UTF_8`.
+
+
+## Transcoders
+
+GIGO transcoders can be any module or class that implements the `transcode` method. This method takes one argument, the string to transcode and can hook into the `GIGO.encoding` if needed. The default list of transcoders is.
+
+* GIGO::Transcoders::ActiveSupport
+* GIGO::Transcoders::CharDet
+* GIGO::Transcoders::Blind
+
+GIGO attempts to use each in that order. Upon successful transcoding, we use the [EnsureValidEncoding](http://github.com/jrochkind/ensure_valid_encoding) gem to force an encoding to match the `GIGO.encoding` while removing any non-convertable characters.
+
 
 ## Toe Dough List
 
@@ -45,6 +56,6 @@ $ bundle exec rake appraisal test
 We use the [appraisal](https://github.com/thoughtbot/appraisal) gem from Thoughtbot to help us generate the individual gemfiles for each ActiveSupport version and to run the tests locally against each generated Gemfile. The `rake appraisal test` command actually runs our test suite against all Rails versions in our `Appraisal` file. If you want to run the tests for a specific Rails version, use `rake -T` for a list. For example, the following command will run the tests for Rails 3.2 only.
 
 ```shell
-$ bundle exec rake appraisal:rails32 test
+$ bundle exec rake appraisal:activesupport32 test
 ```
 

data/gemfiles/activesupport30.gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: /Users/kencollins/Repositories/customink/gigo
   specs:
-    gigo (1.1.0)
+    gigo (1.2.0)
       activesupport (>= 3.0)
       ensure_valid_encoding (~> 0.5.3)
 
@@ -9,12 +9,12 @@ GEM
   remote: https://rubygems.org/
   specs:
     activesupport (3.0.20)
-    appraisal (0.5.1)
+    appraisal (0.5.2)
       bundler
       rake
     ensure_valid_encoding (0.5.3)
-    minitest (4.7.0)
-    minitest-emoji (1.0.0)
+    i18n (0.6.4)
+    minitest (5.0.1)
     rake (10.0.4)
 
 PLATFORMS
@@ -24,6 +24,6 @@ DEPENDENCIES
   activesupport (~> 3.0.0)
   appraisal
   gigo!
-  minitest
-  minitest-emoji
+  i18n
+  minitest (~> 5.0)
   rake

data/gemfiles/activesupport31.gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: /Users/kencollins/Repositories/customink/gigo
   specs:
-    gigo (1.1.0)
+    gigo (1.2.0)
       activesupport (>= 3.0)
       ensure_valid_encoding (~> 0.5.3)
 
@@ -10,12 +10,12 @@ GEM
   specs:
     activesupport (3.1.10)
       multi_json (>= 1.0, < 1.3)
-    appraisal (0.5.1)
+    appraisal (0.5.2)
       bundler
       rake
     ensure_valid_encoding (0.5.3)
-    minitest (4.7.0)
-    minitest-emoji (1.0.0)
+    i18n (0.6.4)
+    minitest (5.0.1)
     multi_json (1.2.0)
     rake (10.0.4)
 
@@ -26,6 +26,6 @@ DEPENDENCIES
   activesupport (~> 3.1.0)
   appraisal
   gigo!
-  minitest
-  minitest-emoji
+  i18n
+  minitest (~> 5.0)
   rake

data/gemfiles/activesupport32.gemfile.lock

@@ -1,24 +1,23 @@
 PATH
   remote: /Users/kencollins/Repositories/customink/gigo
   specs:
-    gigo (1.1.0)
+    gigo (1.2.0)
       activesupport (>= 3.0)
       ensure_valid_encoding (~> 0.5.3)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (3.2.13)
-      i18n (= 0.6.1)
+    activesupport (3.2.12)
+      i18n (~> 0.6)
       multi_json (~> 1.0)
-    appraisal (0.5.1)
+    appraisal (0.5.2)
       bundler
       rake
     ensure_valid_encoding (0.5.3)
-    i18n (0.6.1)
-    minitest (4.7.0)
-    minitest-emoji (1.0.0)
-    multi_json (1.7.2)
+    i18n (0.6.4)
+    minitest (5.0.1)
+    multi_json (1.7.3)
     rake (10.0.4)
 
 PLATFORMS
@@ -28,6 +27,6 @@ DEPENDENCIES
   activesupport (~> 3.2.0)
   appraisal
   gigo!
-  minitest
-  minitest-emoji
+  i18n
+  minitest (~> 5.0)
   rake

data/gemfiles/activesupport40.gemfile.lock

@@ -1,33 +1,32 @@
 GIT
   remote: git://github.com/rails/rails.git
-  revision: b6e5971e53746a79abc92ce9acea85ca14fab0b3
+  revision: d3d8cfd5689188f48714f49ad000a1c1fbd9edcd
   specs:
-    activesupport (4.0.0.beta1)
+    activesupport (4.1.0.beta)
       i18n (~> 0.6, >= 0.6.4)
-      minitest (~> 4.2)
-      multi_json (~> 1.3)
+      json (~> 1.7)
+      minitest (~> 5.0)
       thread_safe (~> 0.1)
       tzinfo (~> 0.3.37)
 
 PATH
   remote: /Users/kencollins/Repositories/customink/gigo
   specs:
-    gigo (1.1.0)
+    gigo (1.2.0)
       activesupport (>= 3.0)
       ensure_valid_encoding (~> 0.5.3)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    appraisal (0.5.1)
+    appraisal (0.5.2)
       bundler
       rake
-    atomic (1.0.1)
+    atomic (1.1.9)
     ensure_valid_encoding (0.5.3)
     i18n (0.6.4)
-    minitest (4.7.0)
-    minitest-emoji (1.0.0)
-    multi_json (1.7.2)
+    json (1.8.0)
+    minitest (5.0.1)
     rake (10.0.4)
     thread_safe (0.1.0)
       atomic
@@ -40,6 +39,6 @@ DEPENDENCIES
   activesupport!
   appraisal
   gigo!
-  minitest
-  minitest-emoji
+  i18n
+  minitest (~> 5.0)
   rake

data/gigo.gemspec

@@ -18,7 +18,7 @@ Gem::Specification.new do |gem|
   gem.add_runtime_dependency     'activesupport', '>= 3.0'
   gem.add_runtime_dependency     'ensure_valid_encoding', '~> 0.5.3'
   gem.add_development_dependency 'appraisal'
+  gem.add_development_dependency 'i18n' # Older ActiveSupport does not have a proper dep.
   gem.add_development_dependency 'rake'
-  gem.add_development_dependency 'minitest'
-  gem.add_development_dependency 'minitest-emoji'
+  gem.add_development_dependency 'minitest', '~> 5.0'
 end

data/lib/gigo.rb

@@ -1,40 +1,38 @@
-require 'active_support/multibyte'
-require 'active_support/core_ext/object/acts_like'
-require 'active_support/core_ext/string/behavior'
+require 'active_support/all'
 require 'ensure_valid_encoding'
-require 'gigo/rchardet'
+require 'gigo/transcoders'
+require 'gigo/transcoders/active_support'
+require 'gigo/transcoders/rchardet'
+require 'gigo/transcoders/blind'
 require 'gigo/version'
 
 module GIGO
 
-  def self.load(data)
+  mattr_accessor :encoding
+  self.encoding = Encoding.default_internal || Encoding::UTF_8
+
+  def self.load(data, options = {})
     return data if data.nil? || !data.acts_like?(:string)
-    encoded_string = safe_detect_and_encoder(data)
-    return data if data.encoding == forced_encoding && data == encoded_string
+    tcoders = options[:transcoders] || transcoders
+    encoded_string = transcode(data, tcoders)
+    return data if data.encoding == GIGO.encoding && data == encoded_string
     encoded_string
   end
 
 
   protected
 
-  def self.safe_detect_and_encoder(data)
+  def self.transcode(data, tcoders)
     string = data
-    begin
-      string = ActiveSupport::Multibyte.proxy_class.new(string).tidy_bytes
-    rescue Exception => e
+    tcoders.detect do |t|
       begin
-        encoding = CharDet.detect(string.dup)['encoding'] || string.encoding || Encoding.default_internal || forced_encoding
-        string = string.force_encoding(encoding).encode forced_encoding, :undef => :replace, :invalid => :replace
+        string = t.transcode(string)
       rescue Exception => e
-        string = string.encode forced_encoding, :undef => :replace, :invalid => :replace
+        false
       end
     end
     string = EnsureValidEncoding.ensure_valid_encoding string, invalid: :replace, replace: "?"
-    string.to_s
+    string
   end
 
-  def self.forced_encoding
-    Encoding.default_internal || Encoding::UTF_8
-  end
-  
 end

data/lib/gigo/transcoders.rb

@@ -0,0 +1,7 @@
+module GIGO
+  module Transcoders
+  
+  end
+  mattr_accessor :transcoders
+  self.transcoders = []
+end

data/lib/gigo/transcoders/active_support.rb

@@ -0,0 +1,13 @@
+module GIGO
+  module Transcoders
+    module ActiveSupport
+
+      GIGO.transcoders << self
+
+      def self.transcode(data)
+        ::ActiveSupport::Multibyte.proxy_class.new(data).tidy_bytes.to_s
+      end
+      
+    end
+  end
+end

data/lib/gigo/transcoders/blind.rb

@@ -0,0 +1,13 @@
+module GIGO
+  module Transcoders
+    module Blind
+
+      GIGO.transcoders << self
+
+      def self.transcode(data)
+        data.encode GIGO.encoding, :undef => :replace, :invalid => :replace
+      end
+      
+    end
+  end
+end

data/lib/gigo/transcoders/rchardet.rb

@@ -0,0 +1,22 @@
+require 'gigo/rchardet'
+
+module GIGO
+  module Transcoders
+    module CharDet
+
+      GIGO.transcoders << self
+    
+      def self.transcode(data)
+        source_encoding = detect_encoding(data) || data.encoding || Encoding.default_internal || Encoding::UTF_8
+        data.force_encoding(source_encoding).encode GIGO.encoding, :undef => :replace, :invalid => :replace
+      end
+
+      private
+
+      def self.detect_encoding(data)
+        CharDet.detect(data.dup)['encoding']
+      end
+      
+    end
+  end
+end

data/lib/gigo/version.rb

@@ -1,3 +1,3 @@
 module GIGO
-  VERSION = "1.2.0"
+  VERSION = "1.3.0"
 end

data/test/cases/gigo_test.rb

@@ -4,8 +4,6 @@ require 'test_helper'
 module GIGO
   class BaseTest < TestCase
 
-    include ERB::Util
-
     let(:data_utf8_emoji)   { "💖" }
     let(:data_utf8)         { "€20 – “Woohoo”" }
     let(:data_bad_readin)   { "�20 � �Woohoo�" }
@@ -14,6 +12,19 @@ module GIGO
     let(:data_really_bad)   { "ed.Ã\u0083Ã\u0083\xC3" }
 
 
+    describe '.encoding' do
+
+      it 'defaults to UTF-8 encoding' do
+        GIGO.encoding.must_equal Encoding::UTF_8
+      end
+
+      it 'can be set to any encoding' do
+        GIGO.encoding = Encoding::CP1252
+        GIGO.encoding.must_equal Encoding::CP1252
+      end
+
+    end
+
     describe '.load' do
       
       it 'ignores if string is not present' do
@@ -61,7 +72,17 @@ module GIGO
 
     end
 
-    
+    describe '.transcoders' do
+
+      it 'is an array of default transcoders' do
+        GIGO.transcoders.must_equal [
+          GIGO::Transcoders::ActiveSupport,
+          GIGO::Transcoders::CharDet,
+          GIGO::Transcoders::Blind
+        ]
+      end
+
+    end
     
   end
 end

data/test/test_helper.rb

@@ -1,14 +1,26 @@
-require 'bundler'
-require 'minitest/autorun'
-Bundler.require :development, :test
+require 'bundler' ; Bundler.require :development, :test
 require 'gigo'
-require 'support/minitest'
+require 'minitest/autorun'
 require 'erb'
 
 module GIGO
   class TestCase < MiniTest::Spec
 
+    include ERB::Util
+
+    before { setup_gigo }
+    after  { teardown_gigo }
+
+
+    private
+
+    def setup_gigo
+      @_default_gigo_encoding = GIGO.encoding
+    end
 
+    def teardown_gigo
+      GIGO.encoding = @_default_gigo_encoding
+    end
     
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: gigo
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-04-11 00:00:00.000000000 Z
+date: 2013-05-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -60,7 +60,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: i18n
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
@@ -76,7 +76,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: minitest
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
@@ -92,21 +92,21 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: minitest-emoji
+  name: minitest
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '5.0'
 description: Garbage in, garbage out. Fix ruby encoded strings at all costs.
 email:
 - kcollins@customink.com
@@ -165,9 +165,12 @@ files:
 - lib/gigo/rchardet/sjisprober.rb
 - lib/gigo/rchardet/universaldetector.rb
 - lib/gigo/rchardet/utf8prober.rb
+- lib/gigo/transcoders.rb
+- lib/gigo/transcoders/active_support.rb
+- lib/gigo/transcoders/blind.rb
+- lib/gigo/transcoders/rchardet.rb
 - lib/gigo/version.rb
 - test/cases/gigo_test.rb
-- test/support/minitest.rb
 - test/test_helper.rb
 homepage: http://github.com/customink/gigo
 licenses: []
@@ -183,7 +186,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 4476278558512394603
+      hash: -1915512776307670961
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -192,7 +195,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 4476278558512394603
+      hash: -1915512776307670961
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.25
@@ -203,5 +206,4 @@ summary: The gigo gem aims to solve bad data, likely from a legacy database. It
   put in and take out of your data stores.
 test_files:
 - test/cases/gigo_test.rb
-- test/support/minitest.rb
 - test/test_helper.rb

data/test/support/minitest.rb

@@ -1,7 +0,0 @@
-require 'minitest/emoji'
-
-if ENV['CI']
-  MiniTest::Emoji::DEFAULT.merge! '.' => ".", 'F' => "F", 'E' => "E", 'S' => "S"
-else
-  MiniTest::Emoji::DEFAULT.merge! '.' => "\u{1f49A} ", 'F' => "\u{1f494} ", 'E' => "\u{1f480} ", 'S' => "\u{1f49B} "
-end