unicode-display_width 2.1.0 → 2.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 83f868860aad325832499e0994c9b727daac6bc8f2f2059a2c3762d947372c80
-  data.tar.gz: f7e6c78087266834cd0edd4e4728b0742455309cc1f83dd6562efbfcd62dac92
+  metadata.gz: e0bcd900f031999ffa43dd6ef091b07b45d425b1ab04e6559f8c8e2c54e08710
+  data.tar.gz: ec3daad5e92107072f8f590d5f2217fd1213d7b25d6491bb3b20ee103f7a2087
 SHA512:
-  metadata.gz: 1c765285a14f5b45fed8add2fa4d35e144465de2e1943d0d4e8aaab54e1f225fca88858408e24ac7ad110ec9b71fb14d4ea1165282e3f9aae733b34962cacb4d
-  data.tar.gz: 585579fa0a392c9005a8ef43aaba4c6f69ca2282c5474f873f5287d39604cf042eaa1448edc818de5f4bacd94a1522a3f9dfbd3b7c9b2630aa30d696fe82c2d4
+  metadata.gz: da2098a3b5d56518129b453aa88e113583403fefb37bb23eeb3f2ff3213426ea6a7076ab0a2f4c778f91a6cfbe3f98f59ea8dc6a4e37ff4bb6a1499536a5a4b9
+  data.tar.gz: 8559483daad47ca76757cf8701f09060bf66f99017a32d62b9c6f739794e1fcc15f2a4b9aae754252d7c8176571f18b590555ed3aa83748836e8e6a681bc7e10

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 # CHANGELOG
 
+## 2.4.2
+
+More performance improvements:
+
+- Optimize lookup of first 4096 codepoints
+- Avoid overwrite lookup if no overwrites are set
+
+## 2.4.1
+
+- Improve general performance!
+- Further improve performance for ASCII strings
+
+*You should really upgrade - it's much faster now!*
+
+## 2.4.0
+- Improve performance for ASCII-only strings, by @fatkodima
+- Require Ruby 2.4
+
+## 2.3.0
+
+- Unicode 15.0
+
+## 2.2.0
+
+- Add *Hangul Jamo Extended-B* block to zero-width chars, thanks @ninjalj #22
+
 ## 2.1.0
 
 - Unicode 14.0

data/MIT-LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT LICENSE
 
-Copyright (c) 2011, 2015-2021 Jan Lelis
+Copyright (c) 2011, 2015-2023 Jan Lelis
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,12 +1,20 @@
 ## Unicode::DisplayWidth [![[version]](https://badge.fury.io/rb/unicode-display_width.svg)](https://badge.fury.io/rb/unicode-display_width) [<img src="https://github.com/janlelis/unicode-display_width/workflows/Test/badge.svg" />](https://github.com/janlelis/unicode-display_width/actions?query=workflow%3ATest)
 
-Determines the monospace display width of a string in Ruby. Implementation based on [EastAsianWidth.txt](https://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt) and other data, 100% in Ruby. It does not rely on the OS vendor (like [wcwidth()](https://github.com/janlelis/wcswidth-ruby)) to provide an up-to-date method for measuring string width.
+Determines the monospace display width of a string in Ruby. Useful for all kinds of terminal-based applications. Implementation based on [EastAsianWidth.txt](https://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt) and other data, 100% in Ruby. It does not rely on the OS vendor (like [wcwidth()](https://github.com/janlelis/wcswidth-ruby)) to provide an up-to-date method for measuring string width.
 
-Unicode version: **14.0.0** (September 2021)
+Unicode version: **15.0.0** (September 2022)
 
-Supported Rubies: **3.0**, **2.7**
+Supported Rubies: **3.1**, **3.0**, **2.7**
 
-Old Rubies which might still work: **2.6**, **2.5**, **2.4**, **2.3**, **2.2**, **2.1**, **2.0**, **1.9**
+Old Rubies which might still work: **2.6**, **2.5**, **2.4**
+
+For even older Rubies, use version 2.3.0 of this gem: **2.3**, **2.2**, **2.1**, **2.0**, **1.9**
+
+## Version 2.4.2 — Performance Updates
+
+**If you use this gem, you should really upgrade to 2.4.2. It's often 100x faster, sometimes even 1000x and more!**
+
+This is possible because the gem now detects if you use very basic (and common) characters, like ASCII characters. Furthermore, the charachter width lookup code has been optimized, so even when full-width characters are involved, the gem is much faster now.
 
 ## Version 2.0 — Breaking Changes
 
@@ -39,12 +47,12 @@ Width  | Characters                   | Comment
 -------|------------------------------|--------------------------------------------------
 X      | (user defined)               | Overwrites any other values
 -1     | `"\b"`                       | Backspace (total width never below 0)
-0      | `"\0"`, `"\x05"`, `"\a"`, `"\n"`, `"\v"`, `"\f"`, `"\r"`, `"\x0E"`, `"\x0F"` | [C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_.28ASCII_and_derivatives.29) that do not change horizontal width
+0      | `"\0"`, `"\x05"`, `"\a"`, `"\n"`, `"\v"`, `"\f"`, `"\r"`, `"\x0E"`, `"\x0F"` | [C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_.28ASCII_and_derivatives.29) which do not change horizontal width
 1      | `"\u{00AD}"`                 | SOFT HYPHEN
 2      | `"\u{2E3A}"`                 | TWO-EM DASH
 3      | `"\u{2E3B}"`                 | THREE-EM DASH
 0      | General Categories: Mn, Me, Cf (non-arabic) | Excludes ARABIC format characters
-0      | `"\u{1160}".."\u{11FF}"`     | HANGUL JUNGSEONG
+0      | `"\u{1160}".."\u{11FF}"`, `"\u{D7B0}".."\u{D7FF}"`     | HANGUL JUNGSEONG
 0      | `"\u{2060}".."\u{206F}"`, `"\u{FFF0}".."\u{FFF8}"`, `"\u{E0000}".."\u{E0FFF}"` | Ignorable ranges
 2      | East Asian Width: F, W       | Full-width characters
 2      | `"\u{3400}".."\u{4DBF}"`, `"\u{4E00}".."\u{9FFF}"`, `"\u{F900}".."\u{FAFF}"`, `"\u{20000}".."\u{2FFFD}"`, `"\u{30000}".."\u{3FFFD}"` | Full-width ranges
@@ -89,6 +97,9 @@ You can overwrite how to handle specific code points by passing a hash (or even
 Unicode::DisplayWidth.of("a\tb", 1, "\t".ord => 10)) # => tab counted as 10, so result is 12
 ```
 
+Please note that using overwrites disables some perfomance optimizations of this gem.
+
+
 #### Emoji Support
 
 Emoji width support is included, but in must be activated manually. It will adjust the string's size for modifier and zero-width joiner sequences. You also need to add the [unicode-emoji](https://github.com/janlelis/unicode-emoji) gem to your Gemfile:
@@ -148,12 +159,13 @@ Replace "一" with the actual string to measure
 - JavaScript: https://github.com/mycoboco/wcwidth.js
 - C: https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
 - C for Julia: https://github.com/JuliaLang/utf8proc/issues/2
+- Golang: https://github.com/rivo/uniseg
 
 See [unicode-x](https://github.com/janlelis/unicode-x) for more Unicode related micro libraries.
 
 ## Copyright & Info
 
-- Copyright (c) 2011, 2015-2021 Jan Lelis, https://janlelis.com, released under the MIT
+- Copyright (c) 2011, 2015-2023 Jan Lelis, https://janlelis.com, released under the MIT
 license
 - Early versions based on runpaint's unicode-data interface: Copyright (c) 2009 Run Paint Run Run
 - Unicode data: https://www.unicode.org/copyright.html#Exhibit1

data/lib/unicode/display_width/constants.rb

@@ -2,8 +2,8 @@
 
 module Unicode
   class DisplayWidth
-    VERSION = "2.1.0"
-    UNICODE_VERSION = "14.0.0"
+    VERSION = "2.4.2"
+    UNICODE_VERSION = "15.0.0"
     DATA_DIRECTORY = File.expand_path(File.dirname(__FILE__) + "/../../../data/")
     INDEX_FILENAME = DATA_DIRECTORY + "/display_width.marshal.gz"
   end

data/lib/unicode/display_width/index.rb

@@ -10,5 +10,25 @@ module Unicode
       serialized_data.force_encoding Encoding::BINARY
       INDEX = Marshal.load(serialized_data)
     end
+
+    def self.decompress_index(index, level)
+      index.flat_map{ |value|
+        if level > 0
+          if value.instance_of?(Array)
+            value[15] ||= nil
+            decompress_index(value, level - 1)
+          else
+            decompress_index([value] * 16, level - 1)
+          end
+        else
+          if value.instance_of?(Array)
+            value[15] ||= nil
+            value
+          else
+            [value] * 16
+          end
+        end
+      }
+    end
   end
 end

data/lib/unicode/display_width.rb

@@ -5,26 +5,84 @@ require_relative "display_width/index"
 
 module Unicode
   class DisplayWidth
-    DEPTHS = [0x10000, 0x1000, 0x100, 0x10].freeze
+    INITIAL_DEPTH = 0x10000
+    ASCII_NON_ZERO_REGEX = /[\0\x05\a\b\n\v\f\r\x0E\x0F]/
+    FIRST_4096 = decompress_index(INDEX[0][0], 1)
 
     def self.of(string, ambiguous = 1, overwrite = {}, options = {})
-      res = string.codepoints.inject(0){ |total_width, codepoint|
-        index_or_value = INDEX
-        codepoint_depth_offset = codepoint
-        DEPTHS.each{ |depth|
-          index_or_value         = index_or_value[codepoint_depth_offset / depth]
-          codepoint_depth_offset = codepoint_depth_offset % depth
-          break unless index_or_value.is_a? Array
-        }
-        width = index_or_value.is_a?(Array) ? index_or_value[codepoint_depth_offset] : index_or_value
-        width = ambiguous if width == :A
-        total_width + (overwrite[codepoint] || width || 1)
+      if overwrite.empty?
+        # Optimization for ASCII-only strings without certain control symbols
+        if string.ascii_only?
+          if string.match?(ASCII_NON_ZERO_REGEX)
+            res = string.gsub(ASCII_NON_ZERO_REGEX, "").size - string.count("\b")
+            res < 0 ? 0 : res
+          else
+            string.size
+          end
+        else
+          width_no_overwrite(string, ambiguous, options)
+        end
+      else
+        width_all_features(string, ambiguous, overwrite, options)
+      end
+    end
+
+    def self.width_no_overwrite(string, ambiguous, options = {})
+      # Sum of all chars widths
+      res = string.codepoints.sum{ |codepoint|
+        if codepoint > 15 && codepoint < 161 # very common
+          next 1
+        elsif codepoint < 0x1001
+          width = FIRST_4096[codepoint]
+        else
+          width = INDEX
+          depth = INITIAL_DEPTH
+          while (width = width[codepoint / depth]).instance_of? Array
+            codepoint %= depth
+            depth /= 16
+          end
+        end
+
+        width == :A ? ambiguous : (width || 1)
+      }
+
+      # Substract emoji error
+      res -= emoji_extra_width_of(string, ambiguous) if options[:emoji]
+
+      # Return result + prevent negative lengths
+      res < 0 ? 0 : res
+    end
+
+    # Same as .width_no_overwrite - but with applying overwrites for each char
+    def self.width_all_features(string, ambiguous, overwrite, options)
+      # Sum of all chars widths
+      res = string.codepoints.sum{ |codepoint|
+        next overwrite[codepoint] if overwrite[codepoint]
+
+        if codepoint > 15 && codepoint < 161 # very common
+          next 1
+        elsif codepoint < 0x1001
+          width = FIRST_4096[codepoint]
+        else
+          width = INDEX
+          depth = INITIAL_DEPTH
+          while (width = width[codepoint / depth]).instance_of? Array
+            codepoint %= depth
+            depth /= 16
+          end
+        end
+
+        width == :A ? ambiguous : (width || 1)
       }
 
+      # Substract emoji error
       res -= emoji_extra_width_of(string, ambiguous, overwrite) if options[:emoji]
+
+      # Return result + prevent negative lengths
       res < 0 ? 0 : res
     end
 
+
     def self.emoji_extra_width_of(string, ambiguous = 1, overwrite = {}, _ = {})
       require "unicode/emoji"
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: unicode-display_width
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 2.4.2
 platform: ruby
 authors:
 - Jan Lelis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-09-15 00:00:00.000000000 Z
+date: 2023-01-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -38,7 +38,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '13.0'
-description: "[Unicode 14.0.0] Determines the monospace display width of a string
+description: "[Unicode 15.0.0] Determines the monospace display width of a string
   using EastAsianWidth.txt, Unicode general category, and other data."
 email:
 - hi@ruby.consulting
@@ -62,9 +62,10 @@ homepage: https://github.com/janlelis/unicode-display_width
 licenses:
 - MIT
 metadata:
-  changelog_uri: https://github.com/janlelis/unicode-display_width/blob/master/CHANGELOG.md
+  changelog_uri: https://github.com/janlelis/unicode-display_width/blob/main/CHANGELOG.md
   source_code_uri: https://github.com/janlelis/unicode-display_width
   bug_tracker_uri: https://github.com/janlelis/unicode-display_width/issues
+  rubygems_mfa_required: 'true'
 post_install_message: 
 rdoc_options: []
 require_paths:
@@ -73,14 +74,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 1.9.3
+      version: 2.4.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.3
+rubygems_version: 3.4.1
 signing_key: 
 specification_version: 4
 summary: Determines the monospace display width of a string in Ruby.