characteristics 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 95c672e25105f6a9281caee75f58cc242e50bee5
-  data.tar.gz: 17fe74e2842c0175b284e9a0b170561aba80c7af
+  metadata.gz: 8bd09e16bed3587eaa18057790766a6e720e3d25
+  data.tar.gz: 0bb97a8df68aa447fce5b04750573ef9da8a7373
 SHA512:
-  metadata.gz: f12e4d9b7b471019a606060166629a99e950b1e5fc3d44f933da70239f2cbc5ed4ac8023b9bb79998709a6bed14c3681dd141f1894c11ddec14649e31c68d841
-  data.tar.gz: 5110369bc3619f9e3ddc23c9bc6dd5abd7ff0e8ffdfd46889fbc2b04aea1d7611de607852ed0dfab04befa330e8c3e8c3afe5fa680ff0cb16ace232a5a968f6d
+  metadata.gz: '0519adb1524a00105cb7bc54b329aadac50ad2fd350bbe0fda9356474e038a52796ad6548cd3a33865b943f1fd66ae22c5c6bf0673c8528b31cfdf2b5296626e'
+  data.tar.gz: 4a10cc4ec190ca4ed5fdf075449ce8d655182a70cc09faeb75e3805a02a0c7798b762901e6ff881723170f5a7bf51bc119c4af18c44be51c8fb803722db956f7

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 ## CHANGELOG
 
+### 0.7.0
+
+* Add more Unicode properties
+  * variation_selector?
+  * tag?
+  * ignorable?
+  * noncharacter?
+
 ### 0.6.0
 
 * Add separator? property

data/README.md

@@ -1,14 +1,17 @@
 # Characteristics [![[version]](https://badge.fury.io/rb/characteristics.svg)](http://badge.fury.io/rb/characteristics)  [![[travis]](https://travis-ci.org/janlelis/characteristics.svg)](https://travis-ci.org/janlelis/characteristics)
 
-A Ruby library which provides some basic information about how characters behave in different encodings:
+A Ruby library that provides additional info about characters:¹
 
-- Is a character valid according to its encoding?
+- Could a character be invisible (blank)?
 - Is a character assigned?
 - Is a character a special control character?
-- Could a character be invisible (blank)?
+
+Extra data is available for Unicode characters (see below).
 
 The [unibits](https://github.com/janlelis/unibits) and [uniscribe](https://github.com/janlelis/uniscribe) gems makes use of this data to visualize it accordingliy.
 
+¹ in the sense of [codepoints](https://en.wikipedia.org/wiki/Codepoint)
+
 ## Setup
 
 Add to your `Gemfile`:
@@ -20,6 +23,7 @@ gem 'characteristics'
 ## Usage
 
 ```ruby
+# All supported encodings
 char_info = Characteristics.create(character)
 char_info.valid? # => true / false
 char_info.unicode? # => true / false
@@ -28,6 +32,13 @@ char_info.control? # => true / false
 char_info.blank? # => true / false
 char_info.separator? # => true / false
 char_info.format? # => true / false
+
+# Unicode characters
+char_info = Characteristics.create(character)
+char_info.variation_selector? # => true / false
+char_info.tag? # => true / false
+char_info.ignorable? # => true / false
+char_info.noncharacter? # => true / false
 ```
 
 ## Types of Encodings
@@ -43,44 +54,68 @@ This library knows of four different kinds of encodings:
 - **:binary** Arbitrary string
   - *ASCII-8BIT*
 
-Other encodings are not supported, yet.
+Other encodings are currently not supported.
+
+## Properties
 
-## Predicates
+### General
 
-### `valid?`
+#### `valid?`
 
 Validness is determined by Ruby's `String#valid_encoding?`
 
-### `unicode?`
+#### `unicode?`
 
-`true` for Unicode encodings (`UTF-X`)
+**true** for Unicode encodings (`UTF-X`)
 
-### `control?`
+#### `control?`
 
 Control characters are codepoints in the is [C0, delete or C1 control character range](https://en.wikipedia.org/wiki/C0_and_C1_control_codes). Characters in this range of [IBM codepage 437](https://en.wikipedia.org/wiki/Code_page_437) based encodings are always treated as control characters.
 
-### `assigned?`
+#### `assigned?`
 
 - All valid ASCII and BINARY characters are considered assigned
 - For other byte based encodings, a character is considered assigned if it is not on the exception list included in this library. C0 control characters (and `\x7F`) are always considered assigned. C1 control characters are treated as assigned, if the encoding generally does not assign characters in the C1 region.
 - For Unicode, the general category is considered
 
-### `blank?`
+#### `blank?`
 
 The library includes a list of characters that might not be rendered visually. This list does not include unassigned codepoints, control characters (except for `\t`, `\n`, `\v`, `\f`, `\r`, and `\u{85}` in Unicode), or special formatting characters (right-to-left markers, variation selectors, etc).
 
-### `separator?`
+#### `separator?`
 
 Returns true if character is considered a separator. All separators also return true for the `blank?` check. In Unicode, the following characters are separators: `\n`, `\v`, `\f`, `\r`, `\u{85}` (next line), `\u{2028}` (line separator), and `\u{2029}` (paragraph separator)
 
-### `format?`
+#### `format?`
+
+This flag is *true* only for special formatting characters, which are not control characters, like right-to-left marks. In Unicode, this means codepoints with the General Category of **Cf**.
+
+### Additional Unicode Properties
 
-This flag is `true` only for special formatting characters, which are not control characters, like Right-to-left marks. In Unicode, this means codepoints with the General Category of **Cf**.
+#### `variation_selector?`
+
+**true** for [variation selectors](https://en.wikipedia.org/wiki/Variation_Selector).
+
+#### `tag?`
+
+**true** for [tags](https://en.wikipedia.org/wiki/Tags_(Unicode_block)).
+
+#### `ignorable?`
+
+**true** for characters which might not be implemented, and thus, might render no visible glyph.
+
+#### `noncharacter?`
+
+**true** if codepoint will never be assigned in a future standard of Unicode.
 
 ## Todo
 
 - Support all non-dummy encodings that Ruby supports
 
+## Also See
+
+- [Symbolify](https://github.com/janlelis/symbolify)
+
 ## MIT License
 
 Copyright (C) 2017 Jan Lelis <http://janlelis.com>. Released under the MIT license.

data/lib/characteristics/unicode.rb

@@ -91,6 +91,76 @@ class UnicodeCharacteristics < Characteristics
     0x2069,
   ].freeze
 
+  VARIATION_SELECTORS = [
+    *0x180B..0x180D,
+    *0xFE00..0xFE0F,
+    *0xE0100..0xE01EF,
+  ].freeze
+
+  TAGS = [
+    0xE0001,
+    *0xE0020..0xE007F,
+  ].freeze
+
+  NONCHARACTERS = [
+    *0xFDD0..0xFDEF,
+    0xFFFE,   0xFFFF,
+    0x1FFFE,  0x1FFFF,
+    0x2FFFE,  0x2FFFF,
+    0x3FFFE,  0x3FFFF,
+    0x4FFFE,  0x4FFFF,
+    0x5FFFE,  0x5FFFF,
+    0x6FFFE,  0x6FFFF,
+    0x7FFFE,  0x7FFFF,
+    0x8FFFE,  0x8FFFF,
+    0x9FFFE,  0x9FFFF,
+    0xAFFFE,  0xAFFFF,
+    0xBFFFE,  0xBFFFF,
+    0xCFFFE,  0xCFFFF,
+    0xDFFFE,  0xDFFFF,
+    0xEFFFE,  0xEFFFF,
+    0xFFFFE,  0xFFFFF,
+    0x10FFFE, 0x10FFFF,
+  ].freeze
+
+  IGNORABLE = [
+    0x00AD,
+    0x034F,
+    0x061C,
+    *0x115F..0x1160,
+    *0x17B4..0x17B5,
+    *0x180B..0x180E,
+    *0x200B..0x200F,
+    *0x202A..0x202E,
+    *0x2060..0x206F,
+    0x3164,
+    *0xFE00..0xFE0F,
+    0xFEFF,
+    0xFFA0,
+    *0xFFF0..0xFFF8,
+    *0x1BCA0..0x1BCA3,
+    *0x1D173..0x1D17A,
+    *0xE0000..0xE0FFF,
+  ].freeze
+
+  KDDI = [
+    *0xE468..0xE5DF,
+    *0xEA80..0xEB8E,
+  ].freeze
+
+  SOFTBANK = [
+    *0xE001..0xE05A,
+    *0xE101..0xE15A,
+    *0xE201..0xE25A,
+    *0xE301..0xE34D,
+    *0xE401..0xE44C,
+    *0xE501..0xE53E,
+  ].freeze
+
+  DOCOMO = [
+    *0xE63E..0xE757,
+  ].freeze
+
   attr_reader :category
 
   def initialize(char)
@@ -142,28 +212,42 @@ class UnicodeCharacteristics < Characteristics
     @is_valid && BIDI_CONTROL.include?(@ord)
   end
 
+  # unicode specific
+
+  def variation_selector?
+    @is_valid && VARIATION_SELECTORS.include?(@ord)
+  end
+
+  def tag?
+    @is_valid && TAGS.include?(@ord)
+  end
+
+  def noncharacter?
+    @is_valid && NONCHARACTERS.include?(@ord)
+  end
+
+  def ignorable?
+    @is_valid && IGNORABLE.include?(@ord)
+  end
+
+  # emoji
+
   def kddi?
     @is_valid &&
     encoding_has_kddi? &&
-    ( @ord >= 0xE468 && @ord <= 0xE5DF ||
-      @ord >= 0xEA80 && @ord <= 0xEB8E )
+    KDDI.include?(@ord)
   end
 
   def softbank?
     @is_valid &&
     encoding_has_softbank? &&
-    ( @ord >= 0xE001 && @ord <= 0xE05A ||
-      @ord >= 0xE101 && @ord <= 0xE15A ||
-      @ord >= 0xE201 && @ord <= 0xE25A ||
-      @ord >= 0xE301 && @ord <= 0xE34D ||
-      @ord >= 0xE401 && @ord <= 0xE44C ||
-      @ord >= 0xE501 && @ord <= 0xE53E )
+    SOFTBANK.include?(@ord)
   end
 
   def docomo?
     @is_valid &&
     encoding_has_docomo? &&
-    ( @ord >= 0xE63E && @ord <= 0xE757 )
+    DOCOMO.include?(@ord)
   end
 
   private

data/lib/characteristics/version.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
 class Characteristics
-  VERSION = "0.6.0".freeze
+  VERSION = "0.7.0".freeze
   UNICODE_VERSION = "9.0.0".freeze
 end

data/spec/characteristics_spec.rb

@@ -41,13 +41,13 @@ describe Characteristics do
 
       it "is assigned or not" do
         assert assigned? "\x21"
-        refute assigned? "\uFFEF"
+        refute assigned? "\u{FFEF}"
       end
 
       it "is control or not" do
         assert control? "\x1E"
         assert control? "\x7F"
-        assert control? "\u0080"
+        assert control? "\u{0080}"
         refute control? "\x67"
       end
 
@@ -62,32 +62,55 @@ describe Characteristics do
       end
 
       it "is format or not" do
-        assert format? "\uFFF9"
+        assert format? "\u{FFF9}"
         refute format? "\x21"
       end
 
       it "is bidi_control or not" do
-        assert bidi_control? "\u202D"
+        assert bidi_control? "\u{202D}"
         refute bidi_control? "\x21"
       end
     end
 
+    describe "Unicode Properties" do
+      it "is variation_selector or not" do
+        assert Characteristics.create("\u{FE00}").variation_selector?
+        refute Characteristics.create("a").variation_selector?
+      end
+
+      it "is tag or not" do
+        assert Characteristics.create("\u{E0020}").tag?
+        refute Characteristics.create("a").tag?
+      end
+
+      it "is noncharacter or not" do
+        assert Characteristics.create("\u{10FFFF}").noncharacter?
+        refute Characteristics.create("a").noncharacter?
+      end
+
+      it "is ignorable or not" do
+        assert Characteristics.create("\u{AD}").ignorable?
+        assert Characteristics.create("\u{E0000}").ignorable?
+        refute Characteristics.create(" ").ignorable?
+      end
+    end
+
     describe "Japanese Emojis" do
       it "can be a KDDI emoji" do
         encoding = "UTF8-KDDI"
-        assert Characteristics.create("\uE468".force_encoding(encoding)).kddi?
+        assert Characteristics.create("\u{E468}".force_encoding(encoding)).kddi?
         refute Characteristics.create("A".force_encoding(encoding)).kddi?
       end
 
       it "can be a SoftBank emoji" do
         encoding = "UTF8-SoftBank"
-        assert Characteristics.create("\uE001".force_encoding(encoding)).softbank?
+        assert Characteristics.create("\u{E001}".force_encoding(encoding)).softbank?
         refute Characteristics.create("A".force_encoding(encoding)).softbank?
       end
 
       it "can be a DoCoMo emoji" do
         encoding = "UTF8-DoCoMo"
-        assert Characteristics.create("\uE63E".force_encoding(encoding)).docomo?
+        assert Characteristics.create("\u{E63E}".force_encoding(encoding)).docomo?
         refute Characteristics.create("A".force_encoding(encoding)).docomo?
       end
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: characteristics
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Jan Lelis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-03-30 00:00:00.000000000 Z
+date: 2017-03-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: unicode-categories