ensure_valid_encoding 0.5.0

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in ensure_valid_encoding.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Jonathan Rochkind
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,120 @@
+# EnsureValidEncoding
+
+For ruby 1.9 strings, replace bad bytes in given encoding with replacement strings, _or_ fail quickly on invalid encodings --  _without_ a transcode to a different encoding. 
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'ensure_valid_encoding'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install ensure_valid_encoding
+
+## Usage
+
+~~~ruby
+# \xE9 is not valid UTF-8
+bad_utf8 = "M\xE9xico".force_encoding("UTF-8")  
+
+EnsureValidEncoding.ensure_valid_encoding(bad_utf8)
+# => raises a Encoding::InvalidByteSequenceError
+#    Note well, sadly, for performance and pain-in-the-neck reasons,
+#    this will not be filled out with byte number, preceeding or succeeding
+#    bytes, or any other metadata normally included with an InvalidByteSequenceError
+#    from stdlib. 
+~~~~
+
+Uses the same options as String#encode, `:invalid => :replace`, possibly
+combined with `:replace => custom_replace_string` (which can be empty
+string if you like). 
+
+~~~ruby
+fixed = EnsureValidEncoding.ensure_valid_encoding(bad_utf8, :invalid => :replace)
+# => Replaces invalid bytes with default replacement char. 
+#    For unicode encodings, that's unicode replacement code, "\uFFFD",
+#    otherwise, '?'
+
+fixed = EnsureValidEncoding.ensure_valid_encoding(bad_utf8, :invalid => :replace, :replace => "*")
+# => "M*xico"
+~~~
+
+Mutate a string in-place with replacement chars? No problem, use the bang
+version. 
+
+~~~ruby
+EnsureValidEncoding.ensure_valid_encoding!(bad_utf8, :invalid => :replace)
+# bad_utf8 has been mutated
+~~~
+
+For convenience to save you some typing, methods defined as module instance
+methods too:
+
+~~~ruby
+include EnsureValidEncoding
+fixed = ensure_valid_encoding(bad_str)
+~~~
+
+## Rationale
+
+You are taking textual input from some external source. Could be user input, 
+could be a user-uploaded file of some kind, could be a a third party API, could
+be anything. 
+
+You know what character encoding the textual data _claims_ to be, what it
+_should_ be, and what it _usually_ is.  But occasionally it may have bad bytes
+in it, due to data corruption, due to mistakes, due to mis-advertised encoding, 
+due to bugs upstream, due to anything. 
+
+What do you do?  If you do nothing, in cases of such corruption, then 
+eventually your code will _probably_ (but not neccesarily) do something 
+that causes some kind of exception to be raised, could be an 
+Encoding::InvalidByteSequenceError, could be something else from somewhere else. 
+May be hard to predict exactly when, if, and what will be raised. 
+But when it does happen, if you're not rescue'ing the exception, 
+your application dies hard. 
+
+Okay, so maybe you manage to catch the exceptions. Or more 
+conveniently you guard by testing `input_str.valid_encoding?` instead of waiting
+for an exception to be raised. Then what? You could ignore this particular 
+file/stream of input, and have your application go on it's merry way. 
+
+But what if you want to do what most _every other_ mature application that
+displays textual streams does in the case of bad bytes? Display the parts of the
+string that _can_ be displayed, replace the other parts with a replacement
+string of some sort. 
+
+String#encode gives you an API for using a replacement char when converting/transcoding
+from one encoding to another, but that's not where we are. We know what encoding
+the string is supposed to be, we don't know any other better encoding to 
+transcode it to -- we just want to do the best we can with it, substituting
+any illegal bytes. It's what `bash` does. It's what `vim` does.  It is
+surprisingly tricky to do with the ruby 1.9.3 stdlib, or even with 'iconv'. 
+
+So there you go, now you can do it with this gem. Maybe not as performant 
+as if it were implemented in C like stdlib char encoding routines, but, hey. 
+
+**Note well:** A buncha [people on reddit](http://www.reddit.com/r/ruby/comments/sfceq) don't see why you'd ever want to do
+this. If you agree, then, well, don't do it. 
+
+**Also:** I have [filed a feature request](https://bugs.ruby-lang.org/issues/6321) for ruby stdlib. Developer from 
+ruby core team also thinks it's an unusually odd thing to do. shrug. 
+
+## Developing/Contributing
+
+Some tests written with minitest/spec. Run with `rake test`. 
+
+Gem built with bundler rake tests. `rake build`, `rake install`, `rake release`. 
+
+Suggestions/improvements welcome. 
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,10 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['test/*_test.rb']
+  t.verbose = true
+end

data/ensure_valid_encoding.gemspec

@@ -0,0 +1,17 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/ensure_valid_encoding/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.authors       = ["Jonathan Rochkind"]
+  gem.email         = ["jonathan@dnil.net"]
+  gem.summary   = %q{For ruby 1.9 strings, replace bad bytes in given encoding with replacement strings, _or_ fail quickly on invalid encodings --  _without_ a transcode to a different encoding. 
+}
+  gem.homepage      = "https://github.com/jrochkind/ensure_valid_encoding"
+
+  gem.files         = `git ls-files`.split($\)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = "ensure_valid_encoding"
+  gem.require_paths = ["lib"]
+  gem.version       = EnsureValidEncoding::VERSION
+end

data/lib/ensure_valid_encoding.rb

@@ -0,0 +1,91 @@
+require "ensure_valid_encoding/version"
+
+module EnsureValidEncoding
+  
+  # Pass in a string, this method promises the return string
+  # will be #valid_encoding? for the input's existing #encoding, or an exception
+  # will be raised. 
+  #
+  # With no arguments, an Encoding::InvalidByteSequenceError will be raised
+  # unless str.valid_encoding?  Unfortunately, unlike InvalidByteSequenceErrors
+  # raised by stdlib, there will be _no_ line number or preceeding/succeeding
+  # char info included in the exception though, sorry. 
+  #
+  # Or, just like String#encode, pass in :invalid => :replace to replace
+  # invalid bytes with a replacement string. 
+  #
+  # Just like String#encode, the default replacement string is Unicode
+  # replacement char for Unicode encodings or ascii "?" otherwise. 
+  #
+  # Just like String#encode, you can set your own replacement string (including
+  # the empty string) with `:replace => your_string`
+  #
+  # Under ruby 1.8.x (or any ruby without String#encoding), 
+  # this method no-ops and just returns it's input.
+  #
+  #     EnsureValidEncoding.ensure_valid_encoding( some_string )
+  #
+  #     include EnsureValidEncoding  
+  #     ensure_valid_encoding( some_string, :invalid => :replace)
+  #     ensure_valid_encoding( some_string, :invalid => :replace, :replace => '')
+  #     ensure_valid_encoding( some_string, :invalid => :replace, :replace => "*")
+  def self.ensure_valid_encoding(str, options = {})
+      # Can do nothing in ruby 1.8.x
+      return str unless str.respond_to?(:encoding)
+      
+      # We believe it's fastest to use built in #valid_encoding?
+      # with it's C implementation, and bail out immediately if we need
+      # to do nothing more, rather than stepping through byte by byte
+      # in cases where the string was valid in the first place. 
+      if str.valid_encoding?
+        return str       
+      elsif options[:invalid] != :replace
+        # If we're not replacing, just raise right away without going through
+        # chars for performance. 
+        #
+        # That does mean we're not able to say exactly what byte was bad though.
+        # And the exception isn't filled out with all it's usual attributes,
+        # which would be hard even we were going through all the chars/bytes. 
+        raise  Encoding::InvalidByteSequenceError.new("invalid byte in string for source encoding #{str.encoding.name}")
+      else   
+        # :replace => :invalid, 
+        # actually need to go through chars to replace bad ones
+        return str.chars.collect do |c|
+          if c.valid_encoding?
+            c
+          else
+            options[:replace] || (
+             # surely there's a better way to tell if
+             # an encoding is a 'Unicode encoding form'
+             # than this? What's wrong with you ruby 1.9?
+             str.encoding.name.start_with?('UTF') ?
+               "\uFFFD".force_encoding(str.encoding) :
+               "?".force_encoding(str.encoding) )
+          end
+        end.join
+      end
+    end
+    
+    # just like #ensure_valid_encoding, but actually mutates
+    # the input string if neccesary to ensure validity (using String#replace), 
+    # rather than returning the valid string. 
+    #
+    # ensure_valid_encoding!( some_string, :invalid => :replace )
+    def self.ensure_valid_encoding!(str, options = {})
+      str.replace(  ensure_valid_encoding(str, options) )
+    end
+    
+    # instance version, so you can type less. 
+    # 
+    #    include EnsureValidEncoding
+    #    ensure_valid_encoding(bad_str)
+    def ensure_valid_encoding(*args)
+      EnsureValidEncoding.ensure_valid_encoding(*args)
+    end
+    
+    def ensure_valid_encoding!(*args)
+      EnsureValidEncoding.ensure_valid_encoding!(*args)
+    end
+  
+  
+end

data/lib/ensure_valid_encoding/version.rb

@@ -0,0 +1,3 @@
+module EnsureValidEncoding
+  VERSION = "0.5.0"
+end

data/test/ensure_valid_encoding_test.rb

@@ -0,0 +1,80 @@
+require 'minitest/spec'
+require 'minitest/autorun'
+
+require 'ensure_valid_encoding'
+
+describe EnsureValidEncoding do
+  before do
+    @bad_bytes_utf8   = "M\xE9xico".force_encoding("UTF-8")  
+    @bad_bytes_ascii  = "M\xA1xico".force_encoding("ASCII")
+  end
+  
+  it "raises on invalid bytes" do
+    proc do
+      EnsureValidEncoding.ensure_valid_encoding( @bad_bytes_utf8 )
+    end.must_raise Encoding::InvalidByteSequenceError
+  end
+  
+  it "replaces with unicode replacement string" do
+    EnsureValidEncoding.ensure_valid_encoding(@bad_bytes_utf8, 
+        :invalid => :replace).
+      must_equal("M\uFFFDxico")
+  end
+  
+  it "replaces with chosen replacement string" do
+    EnsureValidEncoding.ensure_valid_encoding(@bad_bytes_utf8, 
+        :invalid => :replace, :replace => "*").
+      must_equal("M*xico")
+  end
+  
+  it "replaces with empty string" do
+    EnsureValidEncoding.ensure_valid_encoding(@bad_bytes_utf8, 
+        :invalid => :replace, :replace => '').
+      must_equal("Mxico")
+  end
+  
+  it "replaces non-unicode encoding with ? replacement str" do
+    EnsureValidEncoding.ensure_valid_encoding(@bad_bytes_ascii,
+        :invalid => :replace).
+    must_equal("M?xico")    
+  end
+  
+  describe "edge cases" do
+    it "works with first byte bad" do
+      str = "\xE9xico".force_encoding("UTF-8")
+      EnsureValidEncoding.ensure_valid_encoding(str, 
+          :invalid => :replace,
+          :replace => "?").must_equal("?xico")          
+    end
+    
+    it "works with last bad byte" do
+      str = "Mexico\xE9".force_encoding("UTF-8")
+      EnsureValidEncoding.ensure_valid_encoding(str, 
+          :invalid => :replace,
+          :replace => "?").must_equal("Mexico?")
+    end
+  end
+  
+  it "mutates" do
+    EnsureValidEncoding.ensure_valid_encoding!(@bad_bytes_utf8, :invalid => :replace)
+    @bad_bytes_utf8.must_equal("M\uFFFDxico")
+  end
+  
+  describe "instance method versions" do
+    before do
+      @klass = Class.new do
+        include EnsureValidEncoding
+      end
+    end
+    
+    it "ensure_valid_encoding" do
+      @klass.new.ensure_valid_encoding("foo", :invalid => :replace)
+    end
+    
+    it "ensure_valid_encoding!" do
+      @klass.new.ensure_valid_encoding!("foo", :invalid => :replace) 
+    end
+    
+  end
+  
+end

metadata

@@ -0,0 +1,57 @@
+--- !ruby/object:Gem::Specification
+name: ensure_valid_encoding
+version: !ruby/object:Gem::Version
+  version: 0.5.0
+  prerelease: 
+platform: ruby
+authors:
+- Jonathan Rochkind
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-07-10 00:00:00.000000000 Z
+dependencies: []
+description: 
+email:
+- jonathan@dnil.net
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- ensure_valid_encoding.gemspec
+- lib/ensure_valid_encoding.rb
+- lib/ensure_valid_encoding/version.rb
+- test/ensure_valid_encoding_test.rb
+homepage: https://github.com/jrochkind/ensure_valid_encoding
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: For ruby 1.9 strings, replace bad bytes in given encoding with replacement
+  strings, _or_ fail quickly on invalid encodings --  _without_ a transcode to a different
+  encoding.
+test_files:
+- test/ensure_valid_encoding_test.rb