unicode-display_width 3.0.1 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2d2f36aeac48ce4e9e4fb84f578bbf652d59ee81cca87f19608d302a0cf175bc
-  data.tar.gz: d7df0d9fe9d2cc82a615aaa2ceb0d149606d0992aeb9bc886f1ec565ed19ae5c
+  metadata.gz: 01657362aaf60cf79bb03c63bb96e01914139c7bb965dc9bed18e7988b8c6709
+  data.tar.gz: 297cc1ab03e72a02e9f33eb4eec2dea2006f23987818083c4bc12aa168e437c3
 SHA512:
-  metadata.gz: 5d5ec1bea71952674bb8d868d06eaf0d634869acbd57cf3bb2119e7b0b24b719ee193ac43cdc7b36146c35398b13e0e6567eea120ec7bc6abd91c68192786532
-  data.tar.gz: f6f94791603e6eb719e02e0054eb8a7b539747859ab460ecee61b4e383107594d1b9dc2aacae947542c4a8c8826e47e07659e3883e2e32051a81054f3bc8c75e
+  metadata.gz: a3878d504a273e44268762fca4857bf26a9322e0e54c0afc437d953dca675822262c9aec54cb5de3d23390b4b778403b36ce0f73ba5b0f1d2c8554a1f796d210
+  data.tar.gz: 00de0d22f3b245f16de15b3b4864ff754da04ea94eafdeaf06c0e38fec8cfb2559fbeaafc17f165534e70a386e154f70d7b071f5a226c9f64d7088bbb408cabb

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # CHANGELOG
 
+## 3.1.0
+
+**Improve Emoji support:**
+
+- Emoji modes: Differentiate between well-formed Emoji (`:possible`) and any
+  ZWJ/modifier sequence (`:all`). The latter is more common and more efficient
+  to implement.
+- Unify `rgi_{fqe,mqe,uqe}` options to just `:rgi` to keep things simpler (corresponds to
+  the former `:rgi_uqe` option). Most terminals that want to support the RGI set
+  will probably want to catch Emoji sequences with missing VS16s.
+- Add new `:all_no_vs16` and `:rgi_at` modes to be able to support some terminals
+  that needs these quirks
+- Add alias `emoji: :auto` for `emoji: true` and `emoji: :none` for `emoji: false`
+- `:auto` mode: Only consider terminal cells when recommending Emoji support level
+  (Emoji themselves might display differently)
+- `:auto` mode: Set default Emoji mode for unknown/unsupported terminals to `:none`
+- Rename `:basic` mode to `:vs16`
+
 ## 3.0.1
 
 - Add WezTerm and foot as good Emoji terminals

data/README.md

@@ -4,7 +4,7 @@ Determines the monospace display width of a string in Ruby, which is useful for
 
 Unicode version: **16.0.0** (September 2024)
 
-## Gem Version 3.0 — Improved Emoji Support
+## Gem Version 3 — Improved Emoji Support
 
 **Emoji support is now enabled by default.** See below for description and configuration possibilities.
 
@@ -81,58 +81,53 @@ Unicode::DisplayWidth.of("a\tb", 1, overwrite: { "\t".ord => 10 })) # => TAB cou
 
 Please note that using overwrites disables some perfomance optimizations of this gem.
 
-### Emoji Option
+### Emoji
 
-The gem detects Emoji and Emoji sequences and adjusts the width of the measured string. This can be disabled by passing `emoji: false` as an argument:
+If your terminal supports it, the gem detects Emoji and Emoji sequences and adjusts the width of the measured string. This can be disabled by passing `emoji: false` as an argument:
 
 ```ruby
-Unicode::DisplayWidth.of "🤾🏽‍♀️" # => 2
+Unicode::DisplayWidth.of "🤾🏽‍♀️", emoji: :all # => 2
 Unicode::DisplayWidth.of "🤾🏽‍♀️", emoji: false # => 5
 ```
 
-Disabling Emoji support yields wrong results, as illustrated in the example above, but increases performance of display width calculation. You can configure [the Emoji set to match for](https://www.unicode.org/reports/tr51/#def_rgi_set) by passing a symbol as value:
-
-```ruby
-Unicode::DisplayWidth.of "🐻‍❄", emoji: :rgi_mqe # => 3
-Unicode::DisplayWidth.of "🐻‍❄", emoji: :rgi_uqe # => 2
-```
-
 #### How this Library Handles Emoji Width
 
 There are many Emoji which get constructed by combining other Emoji in a sequence. This makes measuring the width complicated, since terminals might either display the combined Emoji or the separate parts of the Emoji individually.
 
+Another aspect where terminals disagree is whether Emoji characters which have a text presentation by default (width 1) should be turned into full-width (width 2) when combined with Variation Selector 16 (*U+FEOF*).
+
 Emoji Type  | Width / Comment
 ------------|----------------
-Basic/Single Emoji character without Variation Selector | No special handling, uses mechanism from table above
-Basic/Single Emoji character with VS15 (Text)           | No special handling, uses mechanism from table above
-Basic/Single Emoji character with VS16 (Emoji)          | 2
-Emoji Sequence                                          | 2 (only if sequence belongs to configured Emoji set)
-
-The `emoji:` option can be used to configure which type of Emoji should be considered to have a width of 2. Other sequences are treated as non-combined Emoji, so the widths of all partial Emoji add up (e.g. width of one basic Emoji + one skin tone modifier + another basic Emoji). The following Emoji sets can be used:
+Basic/Single Emoji character without Variation Selector | No special handling
+Basic/Single Emoji character with VS15 (Text)           | No special handling
+Basic/Single Emoji character with VS16 (Emoji)          | 2 or East Asian Width (see table below)
+Emoji Sequence                                          | 2 if Emoji belongs to configured Emoji set (see table below)
 
-Option | Descriptions
--------|-------------
-`emoji: true`     | Use recommended Emoji set on your platform, see section below
-`emoji: :basic`   | No width adjustments for Emoji sequences: all partial Emoji treated separately
-`emoji: :rgi_fqe` | All fully-qualified RGI Emoji sequences are considered to have a width of 2
-`emoji: :rgi_mqe` | All fully- and minimally-qualified RGI Emoji sequences are considered to have a width of 2
-`emoji: :rgi_uqe` | All RGI Emoji sequences, regardless of qualification status are considered to have a width of 2
-`emoji: :all`     | All possible/well-formed Emoji sequences are considered to have a width of 2
-`emoji: false`    | No Emoji adjustments, Emoji characters with VS16 not handled
+#### Emoji Modes
 
-*RGI Emoji:* Emoji Recommended for General Interchange
+The `emoji:` option can be used to configure which type of Emoji should be considered to have a width of 2 and if VS16-Emoji should be widened. Other sequences are treated as non-combined Emoji, so the widths of all partial Emoji add up (e.g. width of one basic Emoji + one skin tone modifier + another basic Emoji). The following Emoji settings can be used:
 
-*Qualification:* Whether an Emoji sequence has all required VS16 codepoints
+`emoji:` Option | VS16-Emoji Width | Emoji Sequences Width / Comment | Example Terminals
+----------------|------------------|---------------------------------|------------------
+`true` or `:auto`  | - | Automatically use recommended Emoji setting for your terminal | -
+`:all`     | 2                | 2 for all ZWJ/modifier/keycap sequences, even if they are not well-formed Emoji sequences | iTerm, foot
+`:all_no_vs16` | EAW (1 or 2) | 2 for all ZWJ/modifier/keycap sequences, even if they are not well-formed Emoji sequences | WezTerm
+`:possible`| 2                | 2 for all possible/well-formed Emoji sequences | ?
+`:rgi`     | 2                | 2 for all [RGI Emoji](https://www.unicode.org/reports/tr51/#def_rgi_set) sequences | ?
+`:rgi_at`  | EAW (1 or 2)     | 1 or 2: Like `:rgi`, but Emoji sequences starting with a default-text Emoji have width 1 | Apple Terminal
+`:vs16`    | 2                | 2 * number of partial Emoji (sequences never considered to represent a combined Emoji) | kitty?
+`false` or  `:none` | EAW (1 or 2) | No Emoji adjustments | gnome-terminal, many older terminals
 
-See [emoji-test.txt](https://www.unicode.org/Public/emoji/16.0/emoji-test.txt), the [unicode-emoji gem](https://github.com/janlelis/unicode-emoji) and [UTS-51](https://www.unicode.org/reports/tr51/#def_qualified_emoji_character) for more details about qualified and unqualified Emoji sequences.
+- *RGI Emoji:* Emoji Recommended for General Interchange
+- *ZWJ:* Zero-width Joiner: Codepoint `U+200D`,used in many Emoji sequences
 
 #### Emoji Support in Terminals
 
-Unfortunately, the level of Emoji support varies a lot between terminals. While some of them are able to display (almost) all Emoji sequences correctly, others fall back to displaying sequences of basic Emoji. When `emoji: true` is used, the gem will attempt to set the best fitting Emoji set for you (e.g. `:rgi_uqe` on "Apple_Terminal" or `:basic` on Gnome's terminal widget).
+Unfortunately, the level of Emoji support varies a lot between terminals. While some of them are able to display (almost) all Emoji sequences correctly, others fall back to displaying sequences of basic Emoji. When `emoji: true` or `emoji: :auto` is used, the gem will attempt to set the best fitting Emoji setting for you (e.g. `:rgi_at` on "Apple_Terminal" or `false` on Gnome's terminal widget).
 
-Please [open an issue](https://github.com/janlelis/unicode-display_width/issues/new) if you notice your terminal application could use a better default value.
+Please note that Emoji display and number of terminal columns used might differs a lot. For example, it might be the case that a terminal does not understand which Emoji to display, but still manages to calculate the proper amount of terminal cells. The automatic Emoji support level per terminal only considers the latter (cursor position), not the actual Emoji image(s) displayed. Please [open an issue](https://github.com/janlelis/unicode-display_width/issues/new) if you notice your terminal application could use a better default value. Also see the [ucs-detect project](https://ucs-detect.readthedocs.io/results.html), which is a great resource that compares various terminal's Unicode/Emoji capabilities.
 
-You are encouraged to give your users the option to configure the level of Emoji support in your library or application and for the best developer experience in their terminals. (same is true for ambigouos width).
+**To terminal implementors reading this:** Although the practice of giving all Emoji/ZWJ sequences a width of 2 (`:all` mode described above) has some advantages, it does not lead to a particularly good developer experience. Since there is always the possibility of well-formed Emoji that are currently not supported (non-RGI / future Unicode) appearing, those sequences will take more cells. Instead of overflowing, cutting off sequences or displaying placeholder-Emoji, could it be worthwile to implement the `:rgi` option (only known Emoji get width 2) and give those unknown Emoji the space they need? This would support the idea that the meaning of an unknown Emoji sequence can still be conveyed (without messing up the terminal at the same time). Just a thought…
 
 ### Usage with String Extension
 

data/lib/unicode/display_width/constants.rb

@@ -2,7 +2,7 @@
 
 module Unicode
   class DisplayWidth
-    VERSION = "3.0.1"
+    VERSION = "3.1.0"
     UNICODE_VERSION = "16.0.0"
     DATA_DIRECTORY = File.expand_path(File.dirname(__FILE__) + "/../../../data/")
     INDEX_FILENAME = DATA_DIRECTORY + "/display_width.marshal.gz"

data/lib/unicode/display_width/emoji_support.rb

@@ -7,30 +7,39 @@ module Unicode
       # Tries to find out which terminal emulator is used to
       # set emoji: config to best suiting value
       #
-      # Please note: Many terminals do not set any ENV vars
+      # Please also see section in README.md and
+      # misc/terminal-emoji-width.rb
+      #
+      # Please note: Many terminals do not set any ENV vars,
+      # maybe CSI queries can help?
       def self.recommended
         if ENV["CI"]
-          return :rqi_uqe
+          return :rqi
         end
 
         case ENV["TERM_PROGRAM"]
-        when "iTerm.app", "WezTerm"
+        when "iTerm.app"
           return :all
         when "Apple_Terminal"
-          return :rgi_uqe
+          return :rgi_at
+        when "WezTerm"
+          return :all_no_vs16
         end
 
         case ENV["TERM"]
-        when "foot"
+        when "contour","foot"
+          # konsole: all, how to detect?
           return :all
-        when "contour"
-          return :rgi_uqe
         when /kitty/
-          return :rgi_fqe
+          return :vs16
+        end
+
+        if ENV["WT_SESSION"] # Windows Terminal
+          return :vs16
         end
 
-        # As of last time checked: gnome-terminal, vscode, alacritty, konsole
-        :basic
+        # As of last time checked: gnome-terminal, vscode, alacritty
+        :none
       end
 
       # Maybe: Implement something like https://github.com/jquast/ucs-detect

data/lib/unicode/display_width.rb

@@ -8,6 +8,7 @@ require_relative "display_width/emoji_support"
 
 module Unicode
   class DisplayWidth
+    DEFAULT_AMBIGUOUS = 1
     INITIAL_DEPTH = 0x10000
     ASCII_NON_ZERO_REGEX = /[\0\x05\a\b\n\v\f\r\x0E\x0F]/
     ASCII_NON_ZERO_STRING = "\0\x05\a\b\n\v\f\r\x0E\x0F"
@@ -25,13 +26,13 @@ module Unicode
       WIDTH_TWO: decompress_index(INDEX[:WIDTH_TWO][0][0], 1),
     }
     EMOJI_SEQUENCES_REGEX_MAPPING = {
-      rgi_fqe: :REGEX,
-      rgi_mqe: :REGEX_INCLUDE_MQE,
-      rgi_uqe: :REGEX_INCLUDE_MQE_UQE,
-      all:     :REGEX_WELL_FORMED,
+      rgi: :REGEX_INCLUDE_MQE_UQE,
+      rgi_at: :REGEX_INCLUDE_MQE_UQE,
+      possible: :REGEX_WELL_FORMED,
     }
-    EMOJI_NOT_POSSIBLE = /\A[#*0-9]\z/
     REGEX_EMOJI_BASIC_OR_KEYCAP = Regexp.union(Unicode::Emoji::REGEX_BASIC, Unicode::Emoji::REGEX_EMOJI_KEYCAP)
+    REGEX_EMOJI_ALL_SEQUENCES = Regexp.union(/.[\u{1F3FB}-\u{1F3FF}\u{FE0F}]?(\u{200D}.[\u{1F3FB}-\u{1F3FF}\u{FE0F}]?)+/, Unicode::Emoji::REGEX_EMOJI_KEYCAP)
+    REGEX_EMOJI_NOT_POSSIBLE = /\A[#*0-9]\z/
 
     # Returns monospace display width of string
     def self.of(string, ambiguous = nil, overwrite = nil, old_options = {}, **options)
@@ -41,7 +42,7 @@ module Unicode
       end
 
       options[:ambiguous] = ambiguous if ambiguous
-      options[:ambiguous] ||= 1
+      options[:ambiguous] ||= DEFAULT_AMBIGUOUS
 
       if options[:ambiguous] != 1 && options[:ambiguous] != 2
         raise ArgumentError, "Unicode::DisplayWidth: Ambiguous width must be 1 or 2"
@@ -53,7 +54,7 @@ module Unicode
       end
       options[:overwrite] ||= {}
 
-      if options[:emoji] == nil || options[:emoji] == true
+      if [nil, true, :auto].include?(options[:emoji])
         options[:emoji] = EmojiSupport.recommended
       end
 
@@ -87,12 +88,13 @@ module Unicode
 
     def self.width_frame(string, options)
       # Retrieve Emoji width
-      if !options[:emoji]
+      if options[:emoji] == false || options[:emoji] == :none
         res = 0
-      else options[:emoji]
+      else
         res, string = emoji_width(
           string,
           options[:emoji],
+          options[:ambiguous],
         )
       end
 
@@ -163,32 +165,81 @@ module Unicode
     end
 
 
-    def self.emoji_width(string, sequences = :rgi_fqe)
+    def self.emoji_width(string, mode = :all, ambiguous = DEFAULT_AMBIGUOUS)
       res = 0
 
-      if regex = EMOJI_SEQUENCES_REGEX_MAPPING[sequences]
-        emoji_sequence_regex = Unicode::Emoji.const_get(regex)
-      else # sequences == :basic
-        emoji_sequence_regex = nil
+      string = string.encode(Encoding::UTF_8) unless string.encoding.name == "utf-8"
+
+      if emoji_set_regex = EMOJI_SEQUENCES_REGEX_MAPPING[mode]
+        emoji_width_via_possible(
+          string,
+          Unicode::Emoji.const_get(emoji_set_regex),
+          mode == :rgi_at,
+          ambiguous,
+        )
+      elsif mode == :all_no_vs16
+        emoji_width_all(string)
+      elsif mode == :vs16
+        emoji_width_basic(string)
+      elsif mode == :all
+        res_all, string = emoji_width_all(string)
+        res_basic, string = emoji_width_basic(string)
+        [res_all + res_basic, string]
+      else
+        [0, string]
       end
+    end
 
-      # Make sure we have UTF-8
-      string = string.encode(Encoding::UTF_8) unless string.encoding.name == "utf-8"
+    # Ensure all explicit VS16 sequences have width 2
+    def self.emoji_width_basic(string)
+      res = 0
 
-      if emoji_sequence_regex
-        # For each string possibly an emoji
-        no_emoji_string = string.gsub(Unicode::Emoji::REGEX_POSSIBLE){ |emoji_candidate|
-          # Skip notorious false positives
-          if EMOJI_NOT_POSSIBLE.match?(emoji_candidate)
-            emoji_candidate
+      no_emoji_string = string.gsub(REGEX_EMOJI_BASIC_OR_KEYCAP){ |basic_emoji|
+        if basic_emoji.size >= 2 # VS16 present
+          res += 2
+          ""
+        else
+          basic_emoji
+        end
+      }
 
-          # Check if we have a combined Emoji with width 2
-          elsif emoji_candidate == emoji_candidate[emoji_sequence_regex]
-            res += 2
-            ""
+      [res, no_emoji_string]
+    end
+
+    # Use simplistic ZWJ/modifier/kecap sequence matching
+    def self.emoji_width_all(string)
+      res = 0
+
+      no_emoji_string = string.gsub(REGEX_EMOJI_ALL_SEQUENCES){
+        res += 2
+        ""
+      }
+
+      [res, no_emoji_string]
+    end
+
+    # Match possible Emoji first, then refine
+    def self.emoji_width_via_possible(string, emoji_set_regex, strict_eaw = false, ambiguous = DEFAULT_AMBIGUOUS)
+      res = 0
+
+      # For each string possibly an emoji
+      no_emoji_string = string.gsub(Unicode::Emoji::REGEX_POSSIBLE){ |emoji_candidate|
+        # Skip notorious false positives
+        if REGEX_EMOJI_NOT_POSSIBLE.match?(emoji_candidate)
+          emoji_candidate
 
-          # We are dealing with a default text presentation emoji or a well-formed sequence not matching the above Emoji set
+        # Check if we have a combined Emoji with width 2 (or EAW an Apple Terminal)
+        elsif emoji_candidate == emoji_candidate[emoji_set_regex]
+          if strict_eaw
+            res += self.of(emoji_candidate[0], ambiguous, emoji: false)
           else
+            res += 2
+          end
+          ""
+
+        # We are dealing with a default text presentation emoji or a well-formed sequence not matching the above Emoji set
+        else
+          if !strict_eaw
             # Ensure all explicit VS16 sequences have width 2
             emoji_candidate.gsub!(Unicode::Emoji::REGEX_BASIC){ |basic_emoji|
               if basic_emoji.size == 2 # VS16 present
@@ -198,28 +249,16 @@ module Unicode
                 basic_emoji
               end
             }
-
-            emoji_candidate
           end
-        }
-      else
-        # Only consider basic emoji
 
-        # Ensure all explicit VS16 sequences have width 2
-        no_emoji_string = string.gsub(REGEX_EMOJI_BASIC_OR_KEYCAP){ |basic_emoji|
-          if basic_emoji.size >= 2 # VS16 present
-            res += 2
-            ""
-          else
-            basic_emoji
-          end
-        }
-      end
+          emoji_candidate
+        end
+      }
 
       [res, no_emoji_string]
     end
 
-    def initialize(ambiguous: 1, overwrite: {}, emoji: true)
+    def initialize(ambiguous: DEFAULT_AMBIGUOUS, overwrite: {}, emoji: true)
       @ambiguous = ambiguous
       @overwrite = overwrite
       @emoji     = emoji

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: unicode-display_width
 version: !ruby/object:Gem::Version
-  version: 3.0.1
+  version: 3.1.0
 platform: ruby
 authors:
 - Jan Lelis
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-11-13 00:00:00.000000000 Z
+date: 2024-11-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: unicode-emoji