unibits 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0405b9099735bcf25bc05f82e91d9c5629071ee1
-  data.tar.gz: 85d9bde5ddc1e90ac70cf288e7753ddc8511fc87
+  metadata.gz: 3a765c7b1a6edfd62f8fed06b683d0d4ff2cacae
+  data.tar.gz: c66cb699f9a4425fc169fd6562dd4b758b94c584
 SHA512:
-  metadata.gz: a8d944dfcf50869cf1a64a7237e8145ee8b8fcb57a801db9a475684ac0e7bd4865be2b2d372f943ddeebb6d86f632a3eb9655d147cc6e3b79bc6e0c6dc09fa66
-  data.tar.gz: be6685c1bfc9b9ecf073e9899be4ded5d590ccdd90b527dfdfdfeb1ac654016cac4116033905183f988e249cabedcc767cc8fa3a4d04b3681650ad7852035330
+  metadata.gz: a516728fe08185386fc1902fa341784517521d0a85e494e35a7e2700c1c6d7fc3dfc8f86bb20d4268de6d69df591ab431aa3ab5ee02d96f6798fcd97f2426a8e
+  data.tar.gz: cb2fbfea68be359661ad75cc6e203bd4235ccd81431a37f74f965e90f6116e1a12d8cb04094a213f537e2a032c0c25775edd08378cd7c8ae613503dbe5cc3e0e

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 ## CHANGELOG
 
+### 1.1.0
+
+* Support (and highlight) invalid encodings \o/
+* Improve character symbolification
+* Fix that the Kernel method would not take keyword arguments
+* New option for setting a custom output width to use
+* New option for activating wide ambiguous characters
+
 ### 1.0.0
 
 * Initial release

data/README.md

@@ -1,18 +1,24 @@
-# unibits [![[version]](https://badge.fury.io/rb/unibits.svg)](http://badge.fury.io/rb/unibits)  [![[travis]](https://travis-ci.org/janlelis/unibits.svg)](https://travis-ci.org/janlelis/unibits)
+# unibits | Reveal the Unicode [![[version]](https://badge.fury.io/rb/unibits.svg)](http://badge.fury.io/rb/unibits)  [![[travis]](https://travis-ci.org/janlelis/unibits.svg)](https://travis-ci.org/janlelis/unibits)
 
-Ruby library and CLI command that visualizes various Unicode and ASCII encodings in the terminal.
+Ruby library and CLI command that visualizes various Unicode and ASCII encodings in the terminal:
 
-Helps you debugging and learning about Unicode encodings.
-
-Supported encodings: **UTF-8**, **UTF-16LE**, **UTF-16BE**, **UTF-32LE**, **UTF-32BE**, **BINARY**, **ASCII**
+- Makes analyzing encodings easier
+- Helps you with debugging strings
+- Supports **UTF-8**, **UTF-16LE**/**UTF-16BE**, **UTF-32LE**/**UTF-32BE**, arbitrary **BINARY** data, and **ASCII**
+- Highlights invalid encodings
 
 ## Setup
 
+Make sure you have Ruby installed and installing gems works properly. Then do:
+
 ```
 $ gem install unibits
 ```
 
 ## Usage
+
+Pass the string to debug to unibits:
+
 ### From CLI
 
 ```
@@ -26,17 +32,17 @@ require 'unibits/kernel_method'
 unibits "🌫 Idiosyncrätic ℜսᖯʏ"
 ```
 
-### Options
+### Advanced Options
 
-`unibits` takes three optional options:
+`unibits` takes some optional options:
 
 - *encoding (e)*: The encoding of the given string (uses your default encoding if none given)
 - *convert (c)*: An encoding the string should be converted to before visualizing it
 - *stats*: Whether to show a short stats header (default: `true`), you can deactivate on the CLI with `--no-stats`
+- *wide-ambiguous*: Treat characters of ambiguous width as 2 spaces instead of 1 ([more info](https://github.com/janlelis/unicode-display_width))
+- *width (w)*: Set a custom column width, if not set, *unibits* will retrieve it from the terminal or just use 80
 
-**Please note**: This uses Ruby's built-in encoding support. Currently, only strings with valid encodings are supported.
-
-## Output for Different Encodings
+## Output of Different Valid Encodings
 ### UTF-8
 
 CLI: `$ unibits -e utf-8 -c utf-8 "🌫 Idiosyncrätic ℜսᖯʏ"`
@@ -47,33 +53,33 @@ Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert:
 
 ### UTF-16LE
 
-CLI: `$ unibits -e utf-8 -c utf-8 "🌫 Idiosyncrätic ℜսᖯʏ"`
+CLI: `$ unibits -e utf-8 -c utf-16le "🌫 Idiosyncrätic ℜսᖯʏ"`
 
-Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-8'`
+Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-16le'`
 
 ![Screenshot UTF-16LE](/screenshots/utf-16le.png?raw=true "UTF-16LE")
 
 ### UTF-16BE
 
-CLI: `$ unibits -e utf-8 -c utf-8 "🌫 Idiosyncrätic ℜսᖯʏ"`
+CLI: `$ unibits -e utf-8 -c utf-16be "🌫 Idiosyncrätic ℜսᖯʏ"`
 
-Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-8'`
+Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-16be'`
 
 ![Screenshot UTF-16BE](/screenshots/utf-16be.png?raw=true "UTF-16BE")
 
 ### UTF-32LE
 
-CLI: `$ unibits -e utf-8 -c utf-8 "🌫 Idiosyncrätic ℜսᖯʏ"`
+CLI: `$ unibits -e utf-8 -c utf-32le "🌫 Idiosyncrätic ℜսᖯʏ"`
 
-Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-8'`
+Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-32le'`
 
 ![Screenshot UTF-32LE](/screenshots/utf-32le.png?raw=true "UTF-32LE")
 
 ### UTF-32BE
 
-CLI: `$ unibits -e utf-8 -c utf-8 "🌫 Idiosyncrätic ℜսᖯʏ"`
+CLI: `$ unibits -e utf-8 -c utf-32be "🌫 Idiosyncrätic ℜսᖯʏ"`
 
-Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-8'`
+Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'utf-8', convert: 'utf-32be'`
 
 ![Screenshot UTF-32BE](/screenshots/utf-32be.png?raw=true "UTF-32BE")
 
@@ -93,6 +99,23 @@ Ruby: `unibits "ASCII String", encoding: 'utf-8', convert: 'ascii'`
 
 ![Screenshot ASCII](/screenshots/ascii.png?raw=true "ASCII")
 
+## Invalid Encodings
+### UTF-8
+
+Example in Ruby: `unibits "unexpected \x80 | not enough \xF0\x9F\x8C | overlong \xE0\x81\x81 | surrogate \xED\xA0\x80 | too large \xF5\x8F\xBF\xBF"`
+
+![Screenshot invalid UTF-8](/screenshots/utf-8.invalid.png?raw=true "Invalid UTF-8")
+
+### ASCII
+
+Example in Ruby: `unibits "🌫 Idiosyncrätic ℜսᖯʏ", encoding: 'ascii'`
+
+![Screenshot invalid ASCII](/screenshots/ascii.invalid.png?raw=true "Invalid ASCII")
+
+### BINARY
+
+(not possible to produce invalid binary strings)
+
 ## Notes
 
 Also see
@@ -101,6 +124,7 @@ Also see
 - [UTF-16 (Wikipedia)](https://en.wikipedia.org/wiki/UTF-16#Description)
 - [UTF-32 (Wikipedia)](https://en.wikipedia.org/wiki/UTF-32)
 - [Ruby's Encoding class](https://ruby-doc.org/core/Encoding.html)
+- [Difference between BINARY and ASCII](http://idiosyncratic-ruby.com/56-us-ascii-8bit.html)
 - [Unicode Micro Libraries for Ruby](https://github.com/janlelis/unicode-x)
 
 Lots of thanks to @damienklinnert for the motivation and inspiration required to build this! 🎆

data/Rakefile

@@ -24,7 +24,7 @@ end
 
 desc "#{gemspec.name} | IRB"
 task :irb do
-  sh "irb -I ./lib -r #{gemspec.name.gsub '-','/'}"
+  sh "irb -I ./lib -r #{gemspec.name.gsub '-','/'}/kernel_method"
 end
 
 # # #

data/bin/unibits

@@ -7,14 +7,16 @@ argv = Rationalist.parse(
   ARGV,
   string: '_',
   alias: {
-    e: 'encoding',
     c: 'convert',
+    e: 'encoding',
     v: 'version',
+    w: 'width',
   },
   boolean: [
     'help',
+    'stats',
     'version',
-    'stats'
+    'wide-ambiguous',
   ],
   default: {
     stats: true
@@ -22,21 +24,54 @@ argv = Rationalist.parse(
 )
 
 if argv[:version]
-  puts Unibits::VERSION
+  puts "unibits #{Unibits::VERSION} by #{Paint["J-_-L", :bold]} <https://github.com/janlelis/unibits>"
   exit(0)
 end
 
 if argv[:help]
   puts <<-HELP
 
+  #{Paint["DESCRIPTION", :underline]}
+
+  Visualizes ANSI and variouus Unicode encodings in the terminal.
+
   #{Paint["USAGE", :underline]}
 
-  #{Paint["unibits", :bold]} [--encoding <encoding>] [--convert <encoding>] [--no-stats] data
+  #{Paint["unibits", :bold]} [options] data
+
+  --encoding <encoding> | -e | which encoding to use for given data
+  --convert <encoding>  | -c | which encoding to convert to (if possible)
+  --width <n>           | -w | force a specific number of terminal columns
+  --no-stats            |    | no stats header with length info
+  --version             |    | displays version of unibits
+  --wide-ambiguous      |    | ambiguous characters
+
+  #{Paint["ENCODINGS", :underline]}
+
+  #{Unibits::SUPPORTED_ENCODINGS.join(', ')}
+
+  #{Paint["STATS", :underline]}
+
+  ( bytes / codepoints / glyphs / expected terminal width )
+
+  #{Paint["INVALID BYTES", :underline]}
+
+  UTF-8
+
+  n.e.con. | not enough continuation bytes
+  unexp.c. | unexpected continuation byte
+  toolarge | codepoint value exceeds maximum allowed (U+10FFFF)
+  sur.gate | codepoint value would be a surrogate half
+  overlong | unnecessary null padding
+
+  UTF-16
+
+  incompl. | not enough bytes left to finish codepoint
+  hlf.srg. | other half of surrogate missing
 
-  Supported encodings: #{Unibits::SUPPORTED_ENCODINGS.join(', ')}
-  Explanation of stats: bytes / codepoints / glyphs / expected terminal width
+  #{Paint["MORE INFO", :underline]}
 
-  More info and examples at: https://github.com/janlelis/unibits
+  https://github.com/janlelis/unibits
 
   HELP
   exit(0)
@@ -51,7 +86,14 @@ else
 end
 
 begin
-  Unibits.of(data, encoding: argv[:encoding], convert: argv[:convert], stats: argv[:stats])
+  Unibits.of(
+    data,
+    encoding: argv[:encoding],
+    convert: argv[:convert],
+    stats: argv[:stats],
+    wide_ambiguous: argv[:'wide-ambiguous'],
+    width: argv[:width],
+  )
 rescue ArgumentError
   $stderr.puts Paint[$!.message, :red]
   exit(1)

data/lib/unibits.rb

@@ -1,4 +1,5 @@
 require_relative "unibits/version"
+require_relative "unibits/symbolify"
 
 require "io/console"
 require "paint"
@@ -14,8 +15,9 @@ module Unibits
     'ASCII-8BIT',
     'US-ASCII',
   ].freeze
+  DEFAULT_TERMINAL_WIDTH = 80
 
-  def self.of(string, encoding: nil, convert: nil, stats: true)
+  def self.of(string, encoding: nil, convert: nil, stats: true, wide_ambiguous: false, width: nil)
     if !string || string.empty?
       raise ArgumentError, "no data given to unibits"
     end
@@ -25,12 +27,8 @@ module Unibits
 
     case string.encoding.name
     when *SUPPORTED_ENCODINGS
-      unless string.valid_encoding?
-        raise ArgumentError, "the data given is not valid #{string.encoding.name}"
-      end
-
-      puts stats(string) if stats
-      puts visualize(string)
+      puts stats(string, wide_ambiguous: wide_ambiguous) if stats
+      puts visualize(string, wide_ambiguous: wide_ambiguous, width: width)
     when 'UTF-16', 'UTF-32'
       raise ArgumentError, "unibits only supports #{string.encoding.name} with specified endianess, please use #{string.encoding.name}LE or #{string.encoding.name}BE"
     else
@@ -38,27 +36,36 @@ module Unibits
     end
   end
 
-  def self.stats(string)
-    "\n  " \
-    "#{Paint[string.encoding.name, :bold]} (" \
-    "#{string.bytesize}/" \
-    "#{string.size}/" \
-    "#{string.scan(Regexp.compile('\X'.encode(string.encoding))).size}/" \
-    "#{Unicode::DisplayWidth.of(string)})"
+  def self.stats(string, wide_ambiguous: false)
+    valid      = string.valid_encoding?
+    bytes      = string.bytesize rescue "?"
+    codepoints = string.size rescue "?"
+    glyphs     = string.scan(Regexp.compile('\X'.encode(string.encoding))).size rescue "?"
+    width      = Unicode::DisplayWidth.of(string, wide_ambiguous ? 2 : 1) rescue "?"
+
+    "\n  #{valid ? '' : Paint["Invalid ", :bold, :red]}#{Paint[string.encoding.name, :bold]} (#{bytes}/#{codepoints}/#{glyphs}/#{width})"
   end
 
-  def self.visualize(string)
-    cols = determine_terminal_cols
+  def self.visualize(string, wide_ambiguous: false, width: nil)
+    cols = width || determine_terminal_cols
 
     cp_buffer  = ["  "]
     enc_buffer = ["  "]
     hex_buffer = ["  "]
     bin_buffer = ["  "]
     separator  = ["  "]
+    current_encoding_error = nil
 
     puts
     string.each_char{ |char|
-      current_color = random_color
+      if char.valid_encoding?
+        char_valid = true
+        current_color = random_color
+        current_encoding_error = nil
+      else
+        char_valid = false
+        current_color = :red
+      end
 
       char.each_byte.with_index{ |byte, index|
         if Paint.unpaint(hex_buffer[-1]).bytesize > cols - 12
@@ -70,12 +77,116 @@ module Unibits
         end
 
         if index == 0
+          if char_valid
+            codepoint = "U+%04X" % char.ord
+          else
+            case string.encoding.name
+            when "US-ASCII"
+              codepoint = "invalid"
+            when "UTF-8"
+              # this tries to detect what is wrong with this utf-8 encoded string
+              # sorry for this mess
+              case char.unpack("B*")[0]
+              when /^110.{5}$/
+                current_encoding_error = [:nec, 1, 1]
+                codepoint = "n.e.con."
+              when /^1110(.{4})$/
+                if $1 == "1101"
+                  current_encoding_error = [:nec, 2, 2, :maybe_surrogate]
+                else
+                  current_encoding_error = [:nec, 2, 2]
+                end
+                codepoint = "n.e.con."
+              when /^11110(.{3})$/
+                case $1
+                when "100"
+                  current_encoding_error = [:nec, 3, 3, :leading_at_max]
+                when "101", "110", "111"
+                  current_encoding_error = [:nec, 3, 3, :too_large]
+                else
+                  current_encoding_error = [:nec, 3, 3]
+                end
+                codepoint = "n.e.con."
+              when /^11111.{3}$/
+                codepoint = "toolarge"
+              when /^10(.{2}).{4}$/
+                # uglyhack to fixup that it is not n.e.c, but something different
+                if current_encoding_error && current_encoding_error[0] == :nec
+                  if current_encoding_error[3] == :leading_at_max
+                    if $1 != "00"
+                      current_encoding_error[3] = :too_large
+                    else
+                      current_encoding_error[3] = nil
+                    end
+                  elsif current_encoding_error[3] == :maybe_surrogate
+                    if $1[0] == "1"
+                      current_encoding_error[3] = :surrogate
+                    else
+                      current_encoding_error[3] = nil
+                    end
+                  end
+
+                  if current_encoding_error[1] > 1
+                    current_encoding_error[1] -= 1
+                    codepoint = "n.e.con."
+                  else
+                    case current_encoding_error[3]
+                    when :too_large
+                      actual_error = "toolarge"
+                    when :surrogate
+                      actual_error = "sur.gate"
+                    else
+                      actual_error = "overlong"
+                    end
+                    current_cp_buffer_index = -1
+                    (current_encoding_error[2]).times{
+                      if index = cp_buffer[current_cp_buffer_index].rindex("n.e.con.")
+                        cp_buffer[current_cp_buffer_index][index..-1] = cp_buffer[current_cp_buffer_index][index..-1].sub("n.e.con.", actual_error)
+                      else
+                        current_cp_buffer_index -= 1
+                        index = cp_buffer[current_cp_buffer_index].rindex("n.e.con.")
+                        cp_buffer[current_cp_buffer_index][index..-1] = cp_buffer[current_cp_buffer_index][index..-1].sub("n.e.con.", actual_error)
+                      end
+                      current_encoding_error = [:overlong]
+                      codepoint = actual_error
+                    }
+                  end
+                else
+                  current_encoding_error = [:unexp]
+                  codepoint = "unexp.c."
+                end
+              else
+                current_encoding_error = [:invallid]
+                codepoint = "invalid"
+              end
+            when 'UTF-16LE', 'UTF-16BE'
+              if char.bytesize.odd?
+                codepoint = "incompl."
+              elsif char.b[string.encoding.name == 'UTF-16LE' ? 1 : 0].unpack("B*")[0][0, 5] == "11011"
+                codepoint = "hlf.srg."
+              else
+                codepoint = "invalid"
+              end
+            when 'UTF-32LE', 'UTF-32BE'
+              if char.bytesize != "4"
+                codepoint = "incompl."
+              else
+                codepoint = "invalid"
+              end
+            end
+          end
+
           cp_buffer[-1] << Paint[
-            ("U+%04X" % char.ord).ljust(10), current_color, :bold
+            codepoint.ljust(10), current_color, :bold
           ]
 
-          symbolified_char = symbolify(char)
-          padding = 10 - Unicode::DisplayWidth.of(symbolified_char)
+          if char_valid
+            symbolified_char = symbolify(char)
+          else
+            symbolified_char = "�"
+          end
+
+          padding = 10 - Unicode::DisplayWidth.of(symbolified_char, wide_ambiguous ? 2 : 1)
 
           enc_buffer[-1] << Paint[
             symbolified_char, current_color
@@ -92,54 +203,62 @@ module Unibits
 
         bin_byte_complete = byte.to_s(2).rjust(8, "0")
 
-        case string.encoding.name
-        when 'US-ASCII'
-          bin_byte_1 = bin_byte_complete[0...1]
-          bin_byte_2 = bin_byte_complete[1...8]
-        when 'ASCII-8BIT'
-          bin_byte_1 = ""
-          bin_byte_2 = bin_byte_complete
-        when 'UTF-8'
-          if index == 0
-            if bin_byte_complete =~ /^(0|1{2,4}0)([01]+)$/
+        if !char_valid
+          bin_byte_1 = bin_byte_complete
+          bin_byte_2 = ""
+        else
+          case string.encoding.name
+          when 'US-ASCII'
+            bin_byte_1 = bin_byte_complete[0...1]
+            bin_byte_2 = bin_byte_complete[1...8]
+          when 'ASCII-8BIT'
+            bin_byte_1 = ""
+            bin_byte_2 = bin_byte_complete
+          when 'UTF-8'
+            if index == 0
+              if bin_byte_complete =~ /^(0|1{2,4}0)([01]+)$/
+                bin_byte_1 = $1
+                bin_byte_2 = $2
+              else
+                bin_byte_1 = ""
+                bin_byte_2 = bin_byte_complete
+              end
+            else
+              bin_byte_1 = bin_byte_complete[0...2]
+              bin_byte_2 = bin_byte_complete[2...8]
+            end
+          when 'UTF-16LE'
+            if char.ord <= 0xFFFF || index == 0 || index == 2
+              bin_byte_1 = ""
+              bin_byte_2 = bin_byte_complete
+            else
+              bin_byte_complete =~ /^(11011[01])([01]+)$/
               bin_byte_1 = $1
               bin_byte_2 = $2
-            else
+            end
+          when 'UTF-16BE'
+            if char.ord <= 0xFFFF || index == 1 || index == 3
               bin_byte_1 = ""
               bin_byte_2 = bin_byte_complete
+            else
+              bin_byte_complete =~ /^(11011[01])([01]+)$/
+              bin_byte_1 = $1
+              bin_byte_2 = $2
             end
-          else
-            bin_byte_1 = bin_byte_complete[0...2]
-            bin_byte_2 = bin_byte_complete[2...8]
-          end
-        when 'UTF-16LE'
-          if char.ord <= 0xFFFF || index == 0 || index == 2
+          when 'UTF-32LE', 'UTF-32BE'
             bin_byte_1 = ""
             bin_byte_2 = bin_byte_complete
-          else
-            bin_byte_complete =~ /^(11011[01])([01]+)$/
-            bin_byte_1 = $1
-            bin_byte_2 = $2
-          end
-        when 'UTF-16BE'
-          if char.ord <= 0xFFFF || index == 1 || index == 3
-            bin_byte_1 = ""
-            bin_byte_2 = bin_byte_complete
-          else
-            bin_byte_complete =~ /^(11011[01])([01]+)$/
-            bin_byte_1 = $1
-            bin_byte_2 = $2
           end
-        when 'UTF-32LE', 'UTF-32BE'
-          bin_byte_1 = ""
-          bin_byte_2 = bin_byte_complete
         end
+
         bin_buffer[-1] << Paint[
           bin_byte_1, current_color
         ] unless !bin_byte_1 || bin_byte_1.empty?
+
         bin_buffer[-1] << Paint[
           bin_byte_2, current_color, :underline
         ] unless !bin_byte_2 || bin_byte_2.empty?
+
         bin_buffer[-1] << "  "
       }
     }
@@ -157,18 +276,12 @@ module Unibits
 
   def self.symbolify(char)
     return char.inspect unless char.encoding.name[0, 3] == "UTF"
-    char
-      .tr("\x00-\x1F".encode(char.encoding), "\u{2400}-\u{241F}".encode(char.encoding))
-      .gsub(
-        Regexp.compile('[\p{Space}᠎​‌‍⁠]'.encode(char.encoding)),
-        ']\0['.encode(char.encoding)
-      )
-      .encode('UTF-8')
+    Symbolify.symbolify(char).encode('UTF-8')
   end
 
   def self.determine_terminal_cols
-    STDIN.winsize[1] || 80
+    STDIN.winsize[1] || DEFAULT_TERMINAL_WIDTH
   rescue Errno::ENOTTY
-    return 80
+    return DEFAULT_TERMINAL_WIDTH
   end
 end

data/lib/unibits/kernel_method.rb

@@ -3,7 +3,7 @@ require_relative '../unibits'
 module Kernel
   private
 
-  def unibits(string)
-    Unibits.of(string)
+  def unibits(string, **kwargs)
+    Unibits.of(string, **kwargs)
   end
 end

data/lib/unibits/symbolify.rb

@@ -0,0 +1,108 @@
+module Unibits
+  module Symbolify
+    ASCII_CONTROL_CODEPOINTS = "\x00-\x1F\x7F".freeze
+    ASCII_CONTROL_SYMBOLS = "\u{2400}-\u{241F}\u{2421}".freeze
+    ASCII_CHARS = "\x20-\x7E".freeze
+    TAG_START = "\u{E0001}".freeze
+    TAG_START_SYMBOL = "LANG TAG".freeze
+    TAG_SPACE = "\u{E0020}".freeze
+    TAG_SPACE_SYMBOL = "TAG ␠".freeze
+    TAGS = "\u{E0021}-\u{E007E}".freeze
+    TAG_DELETE = "\u{E007F}".freeze
+    TAG_DELETE_SYMBOL = "TAG ␡".freeze
+    INTERESTING_CODEPOINTS = {
+      "\u{0080}" => "PAD",
+      "\u{0081}" => "HOP",
+      "\u{0082}" => "BPH",
+      "\u{0083}" => "NBH",
+      "\u{0084}" => "IND",
+      "\u{0085}" => "NEL",
+      "\u{0086}" => "SSA",
+      "\u{0087}" => "ESA",
+      "\u{0088}" => "HTS",
+      "\u{0089}" => "HTJ",
+      "\u{008A}" => "VTS",
+      "\u{008B}" => "PLD",
+      "\u{008C}" => "PLU",
+      "\u{008D}" => "RI",
+      "\u{008E}" => "SS2",
+      "\u{008F}" => "SS3",
+      "\u{0090}" => "DCS",
+      "\u{0091}" => "PU1",
+      "\u{0092}" => "PU2",
+      "\u{0093}" => "STS",
+      "\u{0094}" => "CCH",
+      "\u{0095}" => "MW",
+      "\u{0096}" => "SPA",
+      "\u{0097}" => "EPA",
+      "\u{0098}" => "SOS",
+      "\u{0099}" => "SGC",
+      "\u{009A}" => "SCI",
+      "\u{009B}" => "CSI",
+      "\u{009C}" => "ST",
+      "\u{009D}" => "OSC",
+      "\u{009E}" => "PM",
+      "\u{009F}" => "APC",
+
+      "\u{200E}" => "LRM",
+      "\u{200F}" => "RLM",
+      "\u{202A}" => "LRE",
+      "\u{202B}" => "RLE",
+      "\u{202C}" => "PDF",
+      "\u{202D}" => "LRO",
+      "\u{202E}" => "RLO",
+      "\u{2066}" => "LRI",
+      "\u{2067}" => "RLI",
+      "\u{2068}" => "FSI",
+      "\u{2069}" => "PDI",
+
+      "\u{FE00}" => "VS1",
+      "\u{FE01}" => "VS2",
+      "\u{FE02}" => "VS3",
+      "\u{FE03}" => "VS4",
+      "\u{FE04}" => "VS5",
+      "\u{FE05}" => "VS6",
+      "\u{FE06}" => "VS7",
+      "\u{FE07}" => "VS8",
+      "\u{FE08}" => "VS9",
+      "\u{FE09}" => "VS10",
+      "\u{FE0A}" => "VS11",
+      "\u{FE0B}" => "VS12",
+      "\u{FE0C}" => "VS13",
+      "\u{FE0D}" => "VS14",
+      "\u{FE0E}" => "VS15",
+      "\u{FE0F}" => "VS16",
+    }.freeze
+    COULD_BE_WHITESPACE = '[\p{Space}­᠎​‌‍⁠⁡⁢⁣⁤⠀𛲠𛲡𛲢𛲣𝅳𝅴𝅵𝅶𝅷𝅸𝅹𝅺]'.freeze
+    # UNASSIGNED = '\p{Cn}'.freeze
+
+    def self.symbolify(char, encoding = char.encoding)
+      char.tr!(
+        ASCII_CONTROL_CODEPOINTS.encode(encoding),
+        ASCII_CONTROL_SYMBOLS.encode(encoding)
+      )
+      char.gsub!(
+        Regexp.compile(COULD_BE_WHITESPACE.encode(encoding)),
+        ']\0['.encode(encoding)
+      )
+      # char.gsub!(
+      #   Regexp.compile(UNASSIGNED.encode(encoding)),
+      #   'n/a'.encode(encoding)
+      # )
+      INTERESTING_CODEPOINTS.each{ |cp, desc|
+        char.gsub! Regexp.compile(cp.encode(encoding)), desc.encode(encoding)
+      }
+      char.gsub! TAG_START.encode(encoding), TAG_START_SYMBOL.encode(encoding)
+      char.gsub! TAG_SPACE.encode(encoding), TAG_SPACE_SYMBOL.encode(encoding)
+      char.gsub! TAG_DELETE.encode(encoding), TAG_DELETE_SYMBOL.encode(encoding)
+
+      ord = char.ord
+      if ord > 917536 && ord < 917631
+        char.tr!(TAGS.encode(encoding), ASCII_CHARS.encode(encoding))
+        char = "TAG ".encode(encoding) + char
+      end
+
+      char
+    end
+  end
+end

data/lib/unibits/version.rb

@@ -1,3 +1,3 @@
 module Unibits
-  VERSION = "1.0.0".freeze
+  VERSION = "1.1.0".freeze
 end

data/spec/unibits_spec.rb

@@ -66,5 +66,171 @@ describe Unibits do
       result.must_match "43"
       result.must_match "01000011"
     end
+
+    describe "invalid UTF-8 encodings" do
+      it "- unexpected continuation byte (1/2)" do
+        string = "abc\x80efg"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "unexp.c."
+        result.must_match /e.*f.*g/m
+      end
+
+      it "- unexpected continuation byte (2/2)" do
+        string = "🌫\x81efg"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "unexp.c."
+        result.must_match /e.*f.*g/m
+      end
+
+      it "- not enough continuation bytes" do
+        string = "\xF0\x9F\x8CABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "n.e.con."
+        result.must_match /A.*B.*C/m
+      end
+
+      it "- overlong padding (1/2)" do
+        string = "\xE0\x81\x81ABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match /overlong.*overlong.*overlong/m
+        result.must_match /A.*B.*C/m
+      end
+
+      it "- overlong padding (2/2)" do
+        string = "\xC0\x80no double null"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match /overlong.*overlong/m
+      end
+
+      it "- too large codepoint (1/2)" do
+        string = "\xF5\x8F\xBF\xBFABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match /toolarge.*toolarge.*toolarge.*toolarge/m
+        result.must_match /A.*B.*C/m
+      end
+
+      it "- too large codepoint (2/2)" do
+        string = "\xF4\xAF\xBF\xBFABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match /toolarge.*toolarge.*toolarge.*toolarge/m
+        result.must_match /A.*B.*C/m
+      end
+
+      it "- too large byte" do
+        string = "\xFF"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "toolarge"
+      end
+
+      it "- has surrogate (1/2)" do
+        string = "\xED\xA0\x80ABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "sur.gate"
+        result.must_match /A.*B.*C/m
+      end
+
+      it "- has surrogate (2/2)" do
+        string = "\xED\xBF\xBFABC"
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match "sur.gate"
+        result.must_match /A.*B.*C/m
+      end
+    end
+
+    describe "invalid UTF-16 encodings" do
+      it "- incomplete number of bytes (1/2)" do
+        string = "a".b.force_encoding("UTF-16LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "incompl."
+        result.must_match "�"
+      end
+
+      it "- incomplete number of bytes (2/2)" do
+        string = "🌫".b[0..-2].force_encoding("UTF-16LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "incompl."
+        result.must_match "�"
+      end
+
+      it "- only lower half surrogate" do
+        string = "\x3C\xD8\x2Ba".force_encoding("UTF-16LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "hlf.srg."
+        result.must_match "�"
+      end
+
+      it "- only higher half surrogate" do
+        string = "\x3Ca\x2B\xDF".force_encoding("UTF-16LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "hlf.srg."
+        result.must_match "�"
+      end
+    end
+
+    describe "invalid UTF-32 encodings" do
+      # please note, currently, too large codepoints and encoded utf16 surrogates are treated as valid encodings
+
+      it "- incomplete number of bytes (1/3)" do
+        string = "a".b.force_encoding("UTF-32LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "incompl."
+        result.must_match "�"
+      end
+
+      it "- incomplete number of bytes (2/3)" do
+        string = "🌫".b[0..-2].force_encoding("UTF-32LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "incompl."
+        result.must_match "�"
+      end
+
+      it "- incomplete number of bytes (3/3)" do
+        string = "🌫".b[0..-2].force_encoding("UTF-32LE")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "incompl."
+        result.must_match "�"
+      end
+    end
+
+    describe "invalid ASCII encodings" do
+      it "- contains bytes with 8th bit set" do
+        string = "abc\x80efg".force_encoding("ASCII")
+        result = Paint.unpaint(Unibits.visualize(string))
+        result.must_match "�"
+        result.must_match /e.*f.*g/m
+      end
+    end
+  end
+
+  describe "wide_ambiguous: option" do
+    it "- default is 1" do
+      string = "⚀······"
+      result = Unibits.stats(string)
+      result.wont_match "13"
+    end
+
+    it "- default is 2" do
+      string = "⚀······"
+      result = Unibits.stats(string, wide_ambiguous: true)
+      result.must_match "13"
+    end
+  end
+
+  describe "width: option" do
+    it "sets a custom column width" do
+      string = "bla" * 99
+      result = Paint.unpaint(Unibits.visualize(string, width: 50))
+      (result[/^.*$/].size <= 50).must_equal true
+    end
   end
 end

data/unibits.gemspec

@@ -17,7 +17,7 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
-  gem.add_dependency 'paint', '~> 2.0'
+  gem.add_dependency 'paint', '>= 0.9', '< 3.0'
   gem.add_dependency 'unicode-display_width', '~> 1.1'
   gem.add_dependency 'rationalist', '~> 2.0'
 

metadata

@@ -1,29 +1,35 @@
 --- !ruby/object:Gem::Specification
 name: unibits
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Jan Lelis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-03-05 00:00:00.000000000 Z
+date: 2017-03-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: paint
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '0.9'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '2.0'
+        version: '0.9'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: unicode-display_width
   requirement: !ruby/object:Gem::Requirement
@@ -74,13 +80,16 @@ files:
 - bin/unibits
 - lib/unibits.rb
 - lib/unibits/kernel_method.rb
+- lib/unibits/symbolify.rb
 - lib/unibits/version.rb
+- screenshots/ascii.invalid.png
 - screenshots/ascii.png
 - screenshots/binary.png
 - screenshots/utf-16be.png
 - screenshots/utf-16le.png
 - screenshots/utf-32be.png
 - screenshots/utf-32le.png
+- screenshots/utf-8.invalid.png
 - screenshots/utf-8.png
 - spec/unibits_spec.rb
 - unibits.gemspec