unicode-display_width 2.3.0 → 2.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1fce5b196a16187b338dee6aaba7f4bf499bfc12eee213085c29296851671bc4
-  data.tar.gz: 4554e159dbc6242ab9e8e285282a6bb4361d715997df21ef81b1d9c90665b4cd
+  metadata.gz: 27acdfc12bbc4c8455095846bcd520ae1a816d6927ba8fa607d93c0925127c70
+  data.tar.gz: 5c65efc8de3821260f0080f582d7ece1085fdb762f501b5c2c0306fd95dbf6f2
 SHA512:
-  metadata.gz: 57824a99da0e7db191264802c8cd01e47319b1387643f4d2ea269a5ab86cfdd2c3d188bcdddc71d56c3cecfd1538940bc68b7d870f424e39cf1d9849cf8123d1
-  data.tar.gz: 655e5048a23f2576e076511d267cf5df7e382c7531e9f1e954c05e3aea5fbbf3975c1af6b4b7fadc4c433283a96afeb02e5621ca925af4bda605d9a6e3ffafeb
+  metadata.gz: 00026a3cf58828846a8f1429f818c900f0998c672b0ad637c402fec46cd63b4470751d6ac35168e7d092a33b966de608ebce84a4e5c2fb936610e9866c1ef7fb
+  data.tar.gz: 21f6105c2084a3b4f4f5ad0f6eb12840e0bf8adc408b073dc69f919acfa451bca8adb80e1c813e9a887728467420dcad4198faa30333cc87d63f994b2e396337

data/CHANGELOG.md

@@ -1,5 +1,27 @@
 # CHANGELOG
 
+## 2.5.0
+
+- Unicode 15.1
+
+## 2.4.2
+
+More performance improvements:
+
+- Optimize lookup of first 4096 codepoints
+- Avoid overwrite lookup if no overwrites are set
+
+## 2.4.1
+
+- Improve general performance!
+- Further improve performance for ASCII strings
+
+*You should really upgrade - it's much faster now!*
+
+## 2.4.0
+- Improve performance for ASCII-only strings, by @fatkodima
+- Require Ruby 2.4
+
 ## 2.3.0
 
 - Unicode 15.0

data/MIT-LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT LICENSE
 
-Copyright (c) 2011, 2015-2022 Jan Lelis
+Copyright (c) 2011, 2015-2023 Jan Lelis
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,12 +1,20 @@
 ## Unicode::DisplayWidth [![[version]](https://badge.fury.io/rb/unicode-display_width.svg)](https://badge.fury.io/rb/unicode-display_width) [<img src="https://github.com/janlelis/unicode-display_width/workflows/Test/badge.svg" />](https://github.com/janlelis/unicode-display_width/actions?query=workflow%3ATest)
 
-Determines the monospace display width of a string in Ruby. Implementation based on [EastAsianWidth.txt](https://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt) and other data, 100% in Ruby. It does not rely on the OS vendor (like [wcwidth()](https://github.com/janlelis/wcswidth-ruby)) to provide an up-to-date method for measuring string width.
+Determines the monospace display width of a string in Ruby. Useful for all kinds of terminal-based applications. Implementation based on [EastAsianWidth.txt](https://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt) and other data, 100% in Ruby. It does not rely on the OS vendor (like [wcwidth()](https://github.com/janlelis/wcswidth-ruby)) to provide an up-to-date method for measuring string width.
 
-Unicode version: **15.0.0** (September 2022)
+Unicode version: **15.1.0** (September 2023)
 
-Supported Rubies: **3.1**, **3.0**, **2.7**
+Supported Rubies: **3.2**,  **3.1**, **3.0**, **2.7**
 
-Old Rubies which might still work: **2.6**, **2.5**, **2.4**, **2.3**, **2.2**, **2.1**, **2.0**, **1.9**
+Old Rubies which might still work: **2.6**, **2.5**, **2.4**
+
+For even older Rubies, use version 2.3.0 of this gem: **2.3**, **2.2**, **2.1**, **2.0**, **1.9**
+
+## Version 2.4.2 — Performance Updates
+
+**If you use this gem, you should really upgrade to 2.4.2 or newer. It's often 100x faster, sometimes even 1000x and more!**
+
+This is possible because the gem now detects if you use very basic (and common) characters, like ASCII characters. Furthermore, the charachter width lookup code has been optimized, so even when full-width characters are involved, the gem is much faster now.
 
 ## Version 2.0 — Breaking Changes
 
@@ -39,7 +47,7 @@ Width  | Characters                   | Comment
 -------|------------------------------|--------------------------------------------------
 X      | (user defined)               | Overwrites any other values
 -1     | `"\b"`                       | Backspace (total width never below 0)
-0      | `"\0"`, `"\x05"`, `"\a"`, `"\n"`, `"\v"`, `"\f"`, `"\r"`, `"\x0E"`, `"\x0F"` | [C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_.28ASCII_and_derivatives.29) that do not change horizontal width
+0      | `"\0"`, `"\x05"`, `"\a"`, `"\n"`, `"\v"`, `"\f"`, `"\r"`, `"\x0E"`, `"\x0F"` | [C0 control codes](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_.28ASCII_and_derivatives.29) which do not change horizontal width
 1      | `"\u{00AD}"`                 | SOFT HYPHEN
 2      | `"\u{2E3A}"`                 | TWO-EM DASH
 3      | `"\u{2E3B}"`                 | THREE-EM DASH
@@ -89,6 +97,9 @@ You can overwrite how to handle specific code points by passing a hash (or even
 Unicode::DisplayWidth.of("a\tb", 1, "\t".ord => 10)) # => tab counted as 10, so result is 12
 ```
 
+Please note that using overwrites disables some perfomance optimizations of this gem.
+
+
 #### Emoji Support
 
 Emoji width support is included, but in must be activated manually. It will adjust the string's size for modifier and zero-width joiner sequences. You also need to add the [unicode-emoji](https://github.com/janlelis/unicode-emoji) gem to your Gemfile:
@@ -154,7 +165,7 @@ See [unicode-x](https://github.com/janlelis/unicode-x) for more Unicode related
 
 ## Copyright & Info
 
-- Copyright (c) 2011, 2015-2022 Jan Lelis, https://janlelis.com, released under the MIT
+- Copyright (c) 2011, 2015-2023 Jan Lelis, https://janlelis.com, released under the MIT
 license
 - Early versions based on runpaint's unicode-data interface: Copyright (c) 2009 Run Paint Run Run
 - Unicode data: https://www.unicode.org/copyright.html#Exhibit1

data/lib/unicode/display_width/constants.rb

@@ -2,8 +2,8 @@
 
 module Unicode
   class DisplayWidth
-    VERSION = "2.3.0"
-    UNICODE_VERSION = "15.0.0"
+    VERSION = "2.5.0"
+    UNICODE_VERSION = "15.1.0"
     DATA_DIRECTORY = File.expand_path(File.dirname(__FILE__) + "/../../../data/")
     INDEX_FILENAME = DATA_DIRECTORY + "/display_width.marshal.gz"
   end

data/lib/unicode/display_width/index.rb

@@ -10,5 +10,25 @@ module Unicode
       serialized_data.force_encoding Encoding::BINARY
       INDEX = Marshal.load(serialized_data)
     end
+
+    def self.decompress_index(index, level)
+      index.flat_map{ |value|
+        if level > 0
+          if value.instance_of?(Array)
+            value[15] ||= nil
+            decompress_index(value, level - 1)
+          else
+            decompress_index([value] * 16, level - 1)
+          end
+        else
+          if value.instance_of?(Array)
+            value[15] ||= nil
+            value
+          else
+            [value] * 16
+          end
+        end
+      }
+    end
   end
 end

data/lib/unicode/display_width.rb

@@ -5,26 +5,84 @@ require_relative "display_width/index"
 
 module Unicode
   class DisplayWidth
-    DEPTHS = [0x10000, 0x1000, 0x100, 0x10].freeze
+    INITIAL_DEPTH = 0x10000
+    ASCII_NON_ZERO_REGEX = /[\0\x05\a\b\n\v\f\r\x0E\x0F]/
+    FIRST_4096 = decompress_index(INDEX[0][0], 1)
 
     def self.of(string, ambiguous = 1, overwrite = {}, options = {})
-      res = string.codepoints.inject(0){ |total_width, codepoint|
-        index_or_value = INDEX
-        codepoint_depth_offset = codepoint
-        DEPTHS.each{ |depth|
-          index_or_value         = index_or_value[codepoint_depth_offset / depth]
-          codepoint_depth_offset = codepoint_depth_offset % depth
-          break unless index_or_value.is_a? Array
-        }
-        width = index_or_value.is_a?(Array) ? index_or_value[codepoint_depth_offset] : index_or_value
-        width = ambiguous if width == :A
-        total_width + (overwrite[codepoint] || width || 1)
+      if overwrite.empty?
+        # Optimization for ASCII-only strings without certain control symbols
+        if string.ascii_only?
+          if string.match?(ASCII_NON_ZERO_REGEX)
+            res = string.gsub(ASCII_NON_ZERO_REGEX, "").size - string.count("\b")
+            res < 0 ? 0 : res
+          else
+            string.size
+          end
+        else
+          width_no_overwrite(string, ambiguous, options)
+        end
+      else
+        width_all_features(string, ambiguous, overwrite, options)
+      end
+    end
+
+    def self.width_no_overwrite(string, ambiguous, options = {})
+      # Sum of all chars widths
+      res = string.codepoints.sum{ |codepoint|
+        if codepoint > 15 && codepoint < 161 # very common
+          next 1
+        elsif codepoint < 0x1001
+          width = FIRST_4096[codepoint]
+        else
+          width = INDEX
+          depth = INITIAL_DEPTH
+          while (width = width[codepoint / depth]).instance_of? Array
+            codepoint %= depth
+            depth /= 16
+          end
+        end
+
+        width == :A ? ambiguous : (width || 1)
+      }
+
+      # Substract emoji error
+      res -= emoji_extra_width_of(string, ambiguous) if options[:emoji]
+
+      # Return result + prevent negative lengths
+      res < 0 ? 0 : res
+    end
+
+    # Same as .width_no_overwrite - but with applying overwrites for each char
+    def self.width_all_features(string, ambiguous, overwrite, options)
+      # Sum of all chars widths
+      res = string.codepoints.sum{ |codepoint|
+        next overwrite[codepoint] if overwrite[codepoint]
+
+        if codepoint > 15 && codepoint < 161 # very common
+          next 1
+        elsif codepoint < 0x1001
+          width = FIRST_4096[codepoint]
+        else
+          width = INDEX
+          depth = INITIAL_DEPTH
+          while (width = width[codepoint / depth]).instance_of? Array
+            codepoint %= depth
+            depth /= 16
+          end
+        end
+
+        width == :A ? ambiguous : (width || 1)
       }
 
+      # Substract emoji error
       res -= emoji_extra_width_of(string, ambiguous, overwrite) if options[:emoji]
+
+      # Return result + prevent negative lengths
       res < 0 ? 0 : res
     end
 
+
     def self.emoji_extra_width_of(string, ambiguous = 1, overwrite = {}, _ = {})
       require "unicode/emoji"
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: unicode-display_width
 version: !ruby/object:Gem::Version
-  version: 2.3.0
+  version: 2.5.0
 platform: ruby
 authors:
 - Jan Lelis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-09-14 00:00:00.000000000 Z
+date: 2023-10-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -38,7 +38,7 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '13.0'
-description: "[Unicode 15.0.0] Determines the monospace display width of a string
+description: "[Unicode 15.1.0] Determines the monospace display width of a string
   using EastAsianWidth.txt, Unicode general category, and other data."
 email:
 - hi@ruby.consulting
@@ -62,7 +62,7 @@ homepage: https://github.com/janlelis/unicode-display_width
 licenses:
 - MIT
 metadata:
-  changelog_uri: https://github.com/janlelis/unicode-display_width/blob/master/CHANGELOG.md
+  changelog_uri: https://github.com/janlelis/unicode-display_width/blob/main/CHANGELOG.md
   source_code_uri: https://github.com/janlelis/unicode-display_width
   bug_tracker_uri: https://github.com/janlelis/unicode-display_width/issues
   rubygems_mfa_required: 'true'
@@ -74,14 +74,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 1.9.3
+      version: 2.4.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.7
+rubygems_version: 3.4.4
 signing_key: 
 specification_version: 4
 summary: Determines the monospace display width of a string in Ruby.