ensure-encoding 0.1

data/LICENSE

@@ -0,0 +1,18 @@
+Copyright (c) 2009 Manfred Stienstra, Fingertips <manfred@fngtps.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,95 @@
+= Ensure Encoding
+
+Experimental project to find the best way to ensure a preferred encoding in
+Strings coming from untrusted sources.
+
+== Algorithms
+
+The most sane way of dealing with character data is choosing an internal
+representation for your application and convert all incoming and outgoing data
+when necessary.
+
+To ensure that our internal encoding is always at least valid we can choose to
+do one of two things.
+
+1. Throw an exception when we receive invalid character data. (ie. Our internal
+   encoding is UTF-8 and we receive Latin-1 data or invalid UTF-8)
+2. Accept whatever we get and try to mold it in such a way that it becomes
+   usable in our application.
+
+It is generally accepted to go for the second option, most of the times the
+end-user has no way of solving these problems because a vendor made a mistake.
+It's not very nice to shut them out.
+
+There are a number of techniques when molding the character data to our needs.
+Sniffing encoding, transcoding, dropping invalid characters are just a few
+examples. We implement a number of these techniques so you can easily protect
+your application from bad data.
+
+== Ensure encoding
+
+We've crammed at lot of functionality into the ensure_encoding method because
+we want to keep the number of new methods on String to a minimum. We'll walk
+through an example to show how it works.
+
+  example = 'Café'
+  example.encoding => #<Encoding:ISO-8859-1>
+  
+After ensuring the encoding of a string you can at least assume that you can
+concatenate the string to another string with the same encoding. In other
+words, it contains data valid for the specified encoding.
+
+  example.ensure_encoding('UTF-8')
+  example.encoding => #<Encoding:UTF-8>
+
+Beyond this you can specify a number of options to perform more operations
+to make sure the data in the string didn't become unreadable garbage. Let's
+look at a number of situations.
+
+=== Untrusted source with known encoding
+
+For instance, when you're excepting data from browsers you can be pretty sure
+the character data is properly encoded. When someone does send bad data it's
+probably a hacker and you can discard the request.
+
+  example.ensure_encoding('UTF-8',
+    :external_encoding  => 'UTF-8,
+    :invalid_characters => :raise
+  )
+
+=== Friendly source with known encoding
+
+In this scenario you're connecting to web API through a ReST library and you
+know the encoding of the source data because it's in the headers. However
+you're not sure the encoding of the strings is valid.
+
+  example.ensure_encoding(Encoding::UTF_8
+    :external_encoding  => Encoding::UTF_8,
+    :invalid_characters => :drop
+  )
+
+=== Untrusted source with variable encoding
+
+Assume we have a legacy database and some of the fields contain Shift JIS,
+while some of the newer fields contain UTF-8 because someone screwed up the
+server configuration. You're not even sure the encoding property on the
+strings you got make any sense because your database adapter is confused too.
+
+  example.ensure_encoding('UTF-8',
+    :external_encoding  => [Encoding::Shift_JIS, Encoding::UTF_8],
+    :invalid_characters => :transcode
+  )
+
+=== Untrusted source with unknown encoding
+
+As a last resort you're trying to read some random files from disk and you
+have no idea what the external encoding is. You've just read them as binary
+and are hoping to make some sense from the data.
+
+  example.ensure_encoding('UTF-8',
+    :external_encoding  => :sniff,
+    :invalid_characters => :transcode
+  )
+
+Note that the encoding sniffer is currently very naive and might not always be
+of any help.

data/lib/ensure.rb

@@ -0,0 +1,3 @@
+module Ensure
+  autoload :Encoding, 'ensure/encoding'
+end

data/lib/ensure/encoding.rb

@@ -0,0 +1,132 @@
+# encoding: utf-8
+
+module Ensure
+  module Encoding
+    BYTE_ORDER_MARKS = {
+      ::Encoding::UTF_16BE => [0xfe, 0xff],
+      ::Encoding::UTF_16LE => [0xff, 0xfe],
+      ::Encoding::UTF_8    => [0xef, 0xbb, 0xbf]
+    }
+    
+    # Tries to guess the encoding of the string and returns the most likely
+    # encoding.
+    def self.sniff_encoding(string)
+      first_bytes = string.unpack('C3')
+      BYTE_ORDER_MARKS.each do |encoding, bytes|
+        if first_bytes[0...bytes.length] == bytes
+          return encoding
+        end
+      end
+      ::Encoding::UTF_8
+    end
+    
+    # Checks the encodings in +guesses+ from front to back and returns the
+    # first encoding in which the character data is a valid sequence.
+    def self.guess_encoding(string, guesses)
+      original_encoding = string.encoding
+      guessed_encoding = nil
+      
+      guesses.each do |guess|
+        string.force_encoding(guess)
+        if string.valid_encoding?
+          guessed_encoding = string.encoding
+          break
+        end
+      end
+      
+      string.force_encoding(original_encoding)
+      guessed_encoding
+    end
+    
+    # Forces the encoding of +string+ to +target_encoding+ and using a number
+    # of smart tricks. See String#ensure_encoding for more details.
+    def self.force_encoding(string, target_encoding, options={})
+      target_string = string.dup
+      force_encoding!(target_string, target_encoding, options)
+      target_string
+    end
+    
+    # Performs just like +force_encoding+, only it changes the string
+    # in place instead of returning it.
+    def self.force_encoding!(string, target_encoding, options={})
+      if options[:external_encoding] == :sniff
+        external_encoding = sniff_encoding(string)
+      else
+        external_encoding = options[:external_encoding] || [target_encoding, string.encoding]
+      end
+      
+      if external_encoding.respond_to?(:each)
+        external_encoding = guess_encoding(string, external_encoding) || target_encoding
+      end
+      
+      if options[:invalid_characters] == :raise
+        string.force_encoding(target_encoding)
+        raise ::Encoding::InvalidByteSequenceError, "String is not encoded as `#{target_encoding}'" unless string.valid_encoding?
+      else
+        filters = (options[:invalid_characters] == :drop) ? { :replace => '', :undef => :replace, :invalid => :replace } : {}
+        string.encode!(target_encoding, external_encoding, filters)
+      end
+    end
+    
+    module String
+      # Ensures the character encoding in a string. It employs a number of
+      # techniques to detect and transcode characters to make sure they end
+      # up in a usuable form in the encoding you need.
+      #
+      # == Arguments
+      #
+      # +target_encoding+
+      #   The character encoding you want to ensure; this is usually the
+      #   internal encoding of your application. Accepts both string
+      #   constants and encoding constants. (ie. 'UTF-8' or Encoding::UTF_8)
+      # +options+
+      #   Options to trigger activate certain algorithms.
+      #
+      # === Options
+      #
+      # :external_encoding
+      #   Specifies both your certainty about the external encoding and what
+      #   you think it might be. Valid options are :sniff, an array of
+      #   encodings, or a single encoding. When you specify :sniff, we will
+      #   sniff around in the data to guess which encoding it is. When you
+      #   supply a list of possible encodings we will check them from begin
+      #   to end if one of them matches the data. Finally, when you specify
+      #   a specific encoding we assume you know which it is and we will use
+      #   that. By default we use :external_encoding => [target_encoding,
+      #   self.encoding].
+      # :invalid_characters
+      #   Specifies what to do with invalid characters. There are three valid
+      #   values: :raise, :drop, and :transcode. The first raises and exception
+      #   on an invalid character. The second will strip all invalid characters
+      #   the last will try to transcode them to the wanted encoding. By default
+      #   we transcode.
+      #
+      # == Example
+      #
+      #   response = REST.get('http://www.google.com')
+      #
+      #   if match = /charset=([^;]*)/.match(response.content_type)
+      #     encoding = match[1]
+      #   else
+      #     encoding = 'UTF-8'
+      #   end
+      #
+      #   body = response.body.ensure_encoding('UTF-8',
+      #     :external_encoding => encoding,
+      #     :invalid_characters => :drop)
+      def ensure_encoding(target_encoding, options={})
+        Ensure::Encoding.force_encoding(self, target_encoding, options)
+      end
+      
+      # Performs just like String#ensure_encoding, only it changes the string
+      # in place instead of returning it.
+      def ensure_encoding!(target_encoding, options={})
+        Ensure::Encoding.force_encoding!(self, target_encoding, options)
+      end
+    end
+  end
+end
+
+class String
+  include Ensure::Encoding::String
+end

metadata

@@ -0,0 +1,61 @@
+--- !ruby/object:Gem::Specification 
+name: ensure-encoding
+version: !ruby/object:Gem::Version 
+  version: "0.1"
+platform: ruby
+authors: 
+- Manfred Stienstra
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-01-11 00:00:00 +01:00
+default_executable: 
+dependencies: []
+
+description: "    Ensure the character encoding in Strings coming from untrusted sources.\n"
+email: manfred@fngtps.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+- LICENSE
+files: 
+- README.rdoc
+- LICENSE
+- lib/ensure/encoding.rb
+- lib/ensure.rb
+has_rdoc: true
+homepage: 
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --all
+- --charset
+- utf-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: Ensure the character encoding in Strings coming from untrusted sources.
+test_files: []
+