mojibake 1.1.0 → 1.1.1

data/History.rdoc

@@ -1,4 +1,14 @@
-=== 1.1.0 (2011-6-31)
+=== 1.1.1 (2013-5-28)
+* Fix Ruby 2.0.0 compatibility (#3, Maciej Hitsu Kowalski)
+* Drop EncodingSupport for jruby 1.7.x, as 1.7.4 continues to exhibit
+  Unicode conformance bugs vs MRI and Rubinius. Note that
+  table.json config backed recovery still works on jruby 1.7.x.
+* Avoid warning by not using 'U' lang on 1.9 Regexp.new.
+* Upgrade to tarpit ~> 2.0, bundler Gemfile, gemspec.
+* Consolidate (j)ruby version checks to VersionSupport and mojibake.rb.
+* Add Travis CI setup and add increased test findings to README.
+
+=== 1.1.0 (2011-10-30)
 * Add table.json output support as more convenient language
   independent format. Include default version (as well as table.txt)
   as part of gem.

data/README.rdoc

@@ -2,6 +2,7 @@
 = MojiBake
 
 * http://github.com/dekellum/mojibake
+* {<img src="https://travis-ci.org/dekellum/mojibake.png" />}[https://travis-ci.org/dekellum/mojibake]
 
 == Description
 
@@ -15,10 +16,16 @@ Windows-1252 is in the wild, should also benefit.
 
 == Dependencies
 
-Requires the String Encoding support in ruby 1.9 as provided by:
+Mojibake mappings generation requires the String encoding support in
+ruby 1.9 as provided by:
 
-*  ruby 1.9.2+ (tested 1.9.2p180, Linux)
-* jruby 1.6.5+ (tested 1.6.5, Linux)
+*  ruby >= 1.9.2 (tested 1.9.2p290, 1.9.3p392, 2.0.0p0 Linux)
+* jruby ~> 1.6.5 (tested 1.6.5, 1.6.8, Linux)
+* rubinius >= 2.0.0 (tested via Travis CI)
+
+Recovery of text with default settings is supported by using the
+pre-generated table.json when in ruby 1.8 mode, or for jruby 1.7.x
+(with contains bugs vs MRI ruby 1.9.x, 2.0.x behavior).
 
 == Synopsis
 
@@ -42,7 +49,7 @@ Recover from a text file:
 
 == License
 
-Copyright (c) 2011 David Kellum
+Copyright (c) 2011-2013 David Kellum
 
 Licensed under the Apache License, Version 2.0 (the "License"); you
 may not use this file except in compliance with the License.  You

data/Rakefile

@@ -1,45 +1,16 @@
 # -*- ruby -*-
 
-$LOAD_PATH << './lib'
-
 require 'rubygems'
-gem     'rjack-tarpit', '~> 1.4'
+require 'bundler/setup'
 require 'rjack-tarpit'
 
-require 'mojibake/base'
-
-t = RJack::TarPit.new( 'mojibake', MojiBake::VERSION )
-
-t.specify do |h|
-  h.developer( 'David Kellum', 'dek-oss@gravitext.com' )
-
-  h.testlib = :minitest
-  h.extra_deps     += [ [ 'json',     '~> 1.6.1' ] ]
-  h.extra_dev_deps += [ [ 'minitest', '~> 2.3'   ] ]
-
-  h.url = 'http://github.com/dekellum/mojibake'
-end
-
-# Version/date consistency checks:
-
-task :check_history_version do
-  t.test_line_match( 'History.rdoc', /^==/, / #{ t.version } / )
-end
-task :check_history_date do
-  t.test_line_match( 'History.rdoc', /^==/, /\([0-9\-]+\)$/ )
-end
-
-task :gem  => [ :check_history_version  ]
-task :tag  => [ :check_history_version, :check_history_date ]
-task :push => [ :check_history_version, :check_history_date ]
-
-t.define_tasks
+RJack::TarPit.new( 'mojibake' ).define_tasks
 
 desc "(Re-)generate config output files (requires 1.9)"
 task :generate_config do
-  if ( RUBY_VERSION.split( '.' ).map { |d| d.to_i } <=> [ 1, 9 ] ) >= 0
-    require 'mojibake'
-    mapper = MojiBake::Mapper.new
+  require 'mojibake'
+  mapper = MojiBake::Mapper.new
+  if defined?( mapper.table )
     open( "config/table.txt",  'w' ) { |fout| fout.puts( mapper.table ) }
     open( "config/table.json", 'w' ) { |fout| fout.puts( mapper.json  ) }
   else

data/bin/mojibake

@@ -3,7 +3,7 @@
 #.hashdot.profile += jruby-shortlived
 
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You
@@ -36,7 +36,7 @@ module Script
       exit 1
     end
 
-    if ( RUBY_VERSION.split( '.' ).map { |d| d.to_i } <=> [ 1, 9 ] ) >= 0
+    if defined?( mapper.char_tree )
 
       opts.on( "--no-windows-1252",
                "Don't include miscodings from Windows-1252" ) do

data/config/table.json

@@ -1,5 +1,5 @@
 {
-  "mojibake": "1.1.0",
+  "mojibake": "1.1.1",
   "url": "https://github.com/dekellum/mojibake",
   "regexp": "Â[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u008C\u008D\u008E\u008F\u0090\u0091\u0092\u0093\u0094\u0095\u0096\u0097\u0098\u0099\u009A\u009B\u009C\u009D\u009E\u009F\u00A0¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ŒœŠšŸŽžƒˆ˜–—‘’‚“”„†‡•…‰‹›€™\uFFFD]|Ã[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u008C\u008D\u008E\u008F\u0090\u0091\u0092\u0093\u0094\u0095\u0096\u0097\u0098\u0099\u009A\u009B\u009C\u009D\u009E\u009F\u00A0¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ŒœŠšŸŽžƒˆ˜–—‘’‚“”„†‡•…‰‹›€™\uFFFD]|Å[\u0092\u0093\u00A0¡¸½¾’“]|Æ[\u0092’]|Ë[\u0086\u009Cœ†]|â(\u0080[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u0093\u0094\u0098\u0099\u009A\u009C\u009D\u009E\u00A0¡¢¦°¹º]|\u0081\u00A0|\u0082¬|\u0084¢|‚¬|„¢|€[\u0081\u009D\u00A0¡¢¦°¹ºœŠšžƒˆ˜‚“”„†‡…‰‹€™\uFFFD]|\uFFFD\u00A0)|ï(»¿|¿[½¾])",
   "moji": {

data/config/table.txt

@@ -1,4 +1,4 @@
-# -*- coding: utf-8 -*- mojibake: 1.1.0
+# -*- coding: utf-8 -*- mojibake: 1.1.1
 /Â[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u008C\u008D\u008E\u008F\u0090\u0091\u0092\u0093\u0094\u0095\u0096\u0097\u0098\u0099\u009A\u009B\u009C\u009D\u009E\u009F\u00A0¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ŒœŠšŸŽžƒˆ˜–—‘’‚“”„†‡•…‰‹›€™\uFFFD]|Ã[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u008C\u008D\u008E\u008F\u0090\u0091\u0092\u0093\u0094\u0095\u0096\u0097\u0098\u0099\u009A\u009B\u009C\u009D\u009E\u009F\u00A0¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ŒœŠšŸŽžƒˆ˜–—‘’‚“”„†‡•…‰‹›€™\uFFFD]|Å[\u0092\u0093\u00A0¡¸½¾’“]|Æ[\u0092’]|Ë[\u0086\u009Cœ†]|â(\u0080[\u0080\u0081\u0082\u0083\u0084\u0085\u0086\u0087\u0088\u0089\u008A\u008B\u0093\u0094\u0098\u0099\u009A\u009C\u009D\u009E\u00A0¡¢¦°¹º]|\u0081\u00A0|\u0082¬|\u0084¢|‚¬|„¢|€[\u0081\u009D\u00A0¡¢¦°¹ºœŠšžƒˆ˜‚“”„†‡…‰‹€™\uFFFD]|\uFFFD\u00A0)|ï(»¿|¿[½¾])/
 
 Moji	UNICODE  	Org	CODE

data/lib/mojibake.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You may
@@ -22,9 +22,13 @@ module MojiBake
 
   # Supports recovering Mojibake characters to the original text.
   class Mapper
+    extend VersionSupport
     include JSONSupport
 
-    if ( RUBY_VERSION.split( '.' ).map { |d| d.to_i } <=> [ 1, 9 ] ) >= 0
+    if ( ruby_version_a <=> [1,9] ) >= 0 &&
+        ( !jruby_version_a ||
+          ( ( ( jruby_version_a <=> [1,6,5] ) >= 0 ) &&
+            ( ( jruby_version_a <=> [1,7]   )  < 0 ) ) )
       require 'mojibake/encoding'
       include EncodingSupport
     end

data/lib/mojibake/base.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You may
@@ -15,5 +15,19 @@
 #++
 
 module MojiBake
-  VERSION = "1.1.0"
+  VERSION = "1.1.1"
+
+  module VersionSupport
+    def ruby_version_a
+      ver_to_a( RUBY_VERSION )
+    end
+
+    def jruby_version_a
+      ver_to_a( JRUBY_VERSION ) if defined?( JRUBY_VERSION )
+    end
+
+    def ver_to_a( ver )
+      ver.split( '.' ).map { |d| d.to_i }
+    end
+  end
 end

data/lib/mojibake/encoding.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You may
@@ -161,7 +161,7 @@ module MojiBake
     end
 
     def regex_encode( c )
-      i = c.codepoints.next #only one
+      i = c.each_codepoint.next #only one
       if INTEREST_CODEPOINTS.include?( i )
         sprintf( '\u%04X', i )
       else

data/lib/mojibake/json.rb

@@ -1,5 +1,5 @@
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You may
@@ -20,6 +20,7 @@ require 'json'
 module MojiBake
 
   module JSONSupport
+    include VersionSupport
 
     JSON_CONFIG = File.join( File.dirname( __FILE__ ),
                              '..', '..', 'config', 'table.json' )
@@ -37,8 +38,13 @@ module MojiBake
     end
 
     def regexp
-      # Note use of Unicode mode for ruby 1.8's
-      @regexp ||= Regexp.new( config[ 'regexp' ], 0, 'U' )
+      @regexp ||= create_regexp
+    end
+
+    def create_regexp
+      # Use (U)nicode mode on ruby 1.8 only
+      lang = ( ruby_version_a <=> [1,9] ) >= 0 ? nil : 'U'
+      Regexp.new( config[ 'regexp' ], 0, lang )
     end
 
     # table as self contained json-ready Hash

data/test/test_encoding.rb

@@ -1,10 +1,9 @@
 #!/usr/bin/env ruby
 # -*- coding: utf-8 -*-
-#.hashdot.args.pre = --1.9
 #.hashdot.profile += jruby-shortlived
 
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You
@@ -19,10 +18,9 @@
 # permissions and limitations under the License.
 #++
 
-ldir = File.join( File.dirname( __FILE__ ), "..", "lib" )
-$LOAD_PATH.unshift( ldir ) unless $LOAD_PATH.include?( ldir )
-
 require 'rubygems'
+require 'bundler/setup'
+
 require 'minitest/unit'
 require 'minitest/autorun'
 
@@ -31,16 +29,14 @@ require 'mojibake'
 class TestEncoding < MiniTest::Unit::TestCase
   include MojiBake
 
-  def setup
-    @mapper = Mapper.new
-  end
+  MAPPER = Mapper.new
 
   TEST_TREE = { "a" => { "b" => { "c" => {},
                                   "d" => {} } },
                 "d" => { "b" => { "f" => {} } } }
 
   # These only test with Ruby 1.9 support
-  if ( RUBY_VERSION.split( '.' ).map { |d| d.to_i } <=> [ 1, 9 ] ) >= 0
+  if defined?( MAPPER.char_tree )
 
     def test_init_options
       assert_equal( true, Mapper.new.map_iso_8859_1 )
@@ -50,16 +46,16 @@ class TestEncoding < MiniTest::Unit::TestCase
 
     def test_char_tree
       assert_equal( TEST_TREE,
-                    @mapper.char_tree( [ "abc", "abd", "dbf" ] ) )
+                    MAPPER.char_tree( [ "abc", "abd", "dbf" ] ) )
     end
 
     def test_tree_flaten
       assert_equal( "ab[cd]|dbf",
-                    @mapper.tree_flatten( TEST_TREE ) )
+                    MAPPER.tree_flatten( TEST_TREE ) )
     end
 
     def test_regexp
-      re = Regexp.new( @mapper.tree_flatten( TEST_TREE ) )
+      re = Regexp.new( MAPPER.tree_flatten( TEST_TREE ) )
       assert_match( re, "abc" )
       assert_match( re, "abd" )
       assert_match( re, "dbf" )

data/test/test_mapper.rb

@@ -1,10 +1,9 @@
 #!/usr/bin/env ruby
 # -*- coding: utf-8 -*-
-#.hashdot.args.pre = --1.9
 #.hashdot.profile += jruby-shortlived
 
 #--
-# Copyright (c) 2011 David Kellum
+# Copyright (c) 2011-2013 David Kellum
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you
 # may not use this file except in compliance with the License.  You
@@ -19,10 +18,9 @@
 # permissions and limitations under the License.
 #++
 
-ldir = File.join( File.dirname( __FILE__ ), "..", "lib" )
-$LOAD_PATH.unshift( ldir ) unless $LOAD_PATH.include?( ldir )
-
 require 'rubygems'
+require 'bundler/setup'
+
 require 'minitest/unit'
 require 'minitest/autorun'
 

metadata

@@ -1,68 +1,80 @@
---- !ruby/object:Gem::Specification 
+--- !ruby/object:Gem::Specification
 name: mojibake
-version: !ruby/object:Gem::Version 
-  prerelease: 
-  version: 1.1.0
+version: !ruby/object:Gem::Version
+  prerelease:
+  version: 1.1.1
 platform: ruby
-authors: 
+authors:
 - David Kellum
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-
-date: 2011-10-31 00:00:00 Z
-dependencies: 
-- !ruby/object:Gem::Dependency 
+date: 2013-05-28 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
   name: json
-  prerelease: false
-  requirement: &id001 !ruby/object:Gem::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.7.5
+    - - <
+      - !ruby/object:Gem::Version
+        version: '1.9'
     none: false
-    requirements: 
-    - - ~>
-      - !ruby/object:Gem::Version 
-        version: 1.6.1
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.7.5
+    - - <
+      - !ruby/object:Gem::Version
+        version: '1.9'
+    none: false
+  prerelease: false
   type: :runtime
-  version_requirements: *id001
-- !ruby/object:Gem::Dependency 
+- !ruby/object:Gem::Dependency
   name: minitest
-  prerelease: false
-  requirement: &id002 !ruby/object:Gem::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '3.2'
     none: false
-    requirements: 
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        version: "2.3"
+      - !ruby/object:Gem::Version
+        version: '3.2'
+    none: false
+  prerelease: false
   type: :development
-  version_requirements: *id002
-- !ruby/object:Gem::Dependency 
+- !ruby/object:Gem::Dependency
   name: rjack-tarpit
-  prerelease: false
-  requirement: &id003 !ruby/object:Gem::Requirement 
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.0'
     none: false
-    requirements: 
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        version: 1.4.0
+      - !ruby/object:Gem::Version
+        version: '2.0'
+    none: false
+  prerelease: false
   type: :development
-  version_requirements: *id003
-description: "Mojibake occurs in English most frequently due to misinterpreting and\n\
-  bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8. This\n\
-  module provides a mojibake sequence to original character mapping\n\
-  table, and utility to recover mojibake\xE2\x80\x99d text.\n\n\
-  Testing has been with English but other Latin based languages, where\n\
-  Windows-1252 is in the wild, should also benefit."
-email: 
+description: Mojibake occurs in English most frequently due to misinterpreting and bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8. This module provides a mojibake sequence to original character mapping table, and utility to recover mojibake’d text.
+email:
 - dek-oss@gravitext.com
-executables: 
+executables:
 - mojibake
 extensions: []
-
-extra_rdoc_files: 
-- Manifest.txt
-- config/table.txt
+extra_rdoc_files:
 - History.rdoc
 - README.rdoc
-files: 
+files:
 - History.rdoc
 - Manifest.txt
 - README.rdoc
@@ -77,35 +89,37 @@ files:
 - test/test.txt
 - test/test_encoding.rb
 - test/test_mapper.rb
-- .gemtest
 homepage: http://github.com/dekellum/mojibake
 licenses: []
-
-post_install_message: 
-rdoc_options: 
+post_install_message:
+rdoc_options:
 - --main
 - README.rdoc
-require_paths: 
+require_paths:
 - lib
-required_ruby_version: !ruby/object:Gem::Requirement 
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+      segments:
+      - 0
+      hash: 2
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      version: "0"
-required_rubygems_version: !ruby/object:Gem::Requirement 
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+      segments:
+      - 0
+      hash: 2
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      version: "0"
 requirements: []
-
-rubyforge_project: mojibake
-rubygems_version: 1.8.11
-signing_key: 
+rubyforge_project:
+rubygems_version: 1.8.24
+signing_key:
 specification_version: 3
-summary: Mojibake occurs in English most frequently due to misinterpreting and bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8
-test_files: 
-- test/test_encoding.rb
-- test/test_mapper.rb
+summary: Mojibake occurs in English most frequently due to misinterpreting and bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8.
+test_files: []
+...