rsyntaxtree 2.2.0 → 2.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fe4cd7a8b28ffa968f99ff228e3a182a64d420e0405cd27e120b5f4ff02fe48e
-  data.tar.gz: 23a64934f448707de2b55cad4c206d65fab18aa7d6c1cd3aab0b1871efd9be1e
+  metadata.gz: 2cab83f32f9b3adab3a84ef896406e8683cf3a9abcecfe132540bda198475a10
+  data.tar.gz: 29d0804c7d76c1661103e9dcbc922ea7b916ba503f3fb0c9a8f2d78c8e8008d6
 SHA512:
-  metadata.gz: 2dfd53e10a62187649fb3d7285fa38189ef25cf56d6af7ceea27fba3cc7358b1f7d528db6466607061120fab61a2ff7bd9b84fa0b989e4313faaac306af42235
-  data.tar.gz: 7399787ea179bdd17910fa9695cc292a18cb5b8e9b3948e943bf32fd52fd05f89e6ec0e2792138197d56c839b9960bab678489a976590667eca9b4dd80679029
+  metadata.gz: 22ddead5114613c3a1c82ce9774911e4c414721495b66ed152f904f55ba8aef3d8dde957a473dc563c764635f2892f58f7f2d0d74dd6142427e3de279e34d5c4
+  data.tar.gz: 4be4281b2e805ac45c0c01a0cc4d67b837fac67118c807812e88038ecf08fef63952897ba8fab3c66d2ec2b5a66d84428de4e038d1409ee766ab371c17fa907c

data/CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## [2.4.0] - 2026-09-02
+
+### Added
+- `RSyntaxTree.escape(text, as:, hyphen:, apostrophe:)` returns notation that
+  draws as the given text, for a program writing notation from strings it did
+  not choose. The context (`:word`, `:phrase`, `:label`, `:cell`) decides what
+  whitespace means; the hyphen mode has to be given because no one spelling of
+  a hyphen serves both; `apostrophe: :keep` keeps a straight apostrophe. The
+  test draws every result and reads the text back, so the rules are exercised
+  rather than restated. The one text with no spelling under `hyphen: literal`
+  — a line of nothing but hyphens, which is the rule — raises `ArgumentError`.
+- `\'` keeps a straight apostrophe, which the notation otherwise sets as a
+  curly one. `\@` is an at sign.
+
+### Fixed
+- `\+` before digits is a plus sign. The path markers at the end of a label
+  (`+1`, `+>2`) were detected without regard to a backslash, so no label could
+  contain "+1" at all: every "C++11" and "2+2" was read as one end of a
+  movement path.
+- A label containing a backslash could lose characters in the drawing. The
+  text was placed into the SVG as a replacement string, which Ruby reads for
+  backreferences: `\+` and `\'` were two of them.
+- An escaped character the tokenizer did not know (`\'`, `\@`) lost its
+  backslash before the markup was read.
+
+## [2.3.0] - 2026-09-01
+
+### Added
+- `RSTError::CODES` lists every error code the library can report. A program
+  keyed by code — one that translates the diagnosis, or tallies what kinds of
+  mistake a writer makes — needs the whole set rather than the codes it has
+  happened to see, and until now that set could only be recovered by reading
+  five files and the repair table. The list only grows: a published code is
+  not renamed or removed.
+
+### Fixed
+- Whitespace alone is an empty input rather than a defect in the library. It
+  reached the drawing, which walked off the end of an empty tree and reported
+  `internal_error`, telling the caller that their own fixable input was the
+  library's fault.
+- A bracketless input of a single character draws. The tokenizer counted its
+  cursor as exhausted one character early, so `A`, and `<>` once it is one
+  whitespace block, produced no tokens at all — though the manual has always
+  said a label on its own draws as one leaf.
+
 ## [2.2.0] - 2026-09-01
 
 ### Changed
@@ -97,9 +142,8 @@ looking at what was left. Every figure in the gallery draws exactly as it did.
 
 ### Added
 - `shear`: the drawn figure tilts by the given angle (degrees, positive
-  leaning the top to the right) and lies on a plane drawn behind it, so the
-  lean reads as a surface seen at an angle rather than as a mistake. The
-  whole picture shears as one affine piece — layout untouched, region
+  leaning the top to the right) and lies on a plane drawn behind it, which
+  shows the tilt as a surface. The whole picture shears as one affine piece — layout untouched, region
   shades coming out parallelograms on their own, and nothing able to newly
   touch or cross. The plane is fill without an edge — a region shade is
   bounded because it marks one part of a figure off from the rest, and a
@@ -245,9 +289,8 @@ looking at what was left. Every figure in the gallery draws exactly as it did.
 - Derivations. A derivation puts the words first and the result last, and
   joins what each step combines with one rule drawn across all of it rather
   than with a line to each daughter. `derivation` draws the rules and
-  `direction: btt` turns the tree over; together they give the format
-  categorial grammar is written in, and `derivation` on its own marks the
-  spans of an ordinary tree. The name of each step rides in the label after a
+  `direction: btt` turns the tree over; together they draw a CCG derivation,
+  and `derivation` on its own marks the spans of an ordinary tree. The name of each step rides in the label after a
   column break and is set beside the end of its rule. A derivation runs down
   the page and is drawn with its rules, so `direction: ltr` and hiding the
   default connectors are refused rather than approximated.
@@ -379,15 +422,15 @@ RMagick dependency.
   rule that work never uses.
 - A matrix nested in a label, written between `#(` and `#)`. The value of an
   attribute can be another attribute-value matrix, to any depth, which is what
-  a feature path such as SYNSEM | LOCAL | CATEGORY | HEAD needs and what HPSG,
-  SBCG and LFG are written in. The nested matrix draws its own brackets and
+  a feature path such as SYNSEM | LOCAL | CATEGORY | HEAD needs, and what
+  HPSG and LFG are written in. The nested matrix draws its own brackets and
   lays out its own columns, and the rows after it clear its full height.
 - `\t` in a label cuts the line into cells. Every line is cut at the same
   points and each column is drawn at the width of its widest cell, so the
   parts line up down the label. Together with the bracket enclosure and the
   horizontal rule this gives attribute-value matrices — the feature structures
-  of HPSG, SBCG and LFG — without spacing each row by hand, which is how the
-  gallery's HPSG example used to do it.
+  of HPSG, LFG and the like — without spacing each row by hand, which is how
+  the gallery's HPSG example used to do it.
 - `color: gray`, a scheme that keeps node and leaf labels black and draws the
   connectors, triangles and movement paths in grey. It is for diagrams whose
   links outnumber their labels — an ontology, a network of constructions —

data/lib/rsyntaxtree/element.rb

@@ -33,11 +33,8 @@ module RSyntaxTree
       @vertical_indent = 0     # Drawing offset
       content = content.strip
 
-      @path = if /.+?\^?((?:\+-?>?<?\d+)+)\^?\z/m =~ content
-                $1.sub(/\A\+/, "").split("+")
-              else
-                []
-              end
+      _body, path_text = Element.split_path(content)
+      @path = path_text.delete_prefix("^").delete_suffix("^").sub(/\A\+/, "").split("+")
 
       @fontset = fontset
       @fontsize = fontsize
@@ -64,7 +61,7 @@ module RSyntaxTree
         end
       end
 
-      @raw_content = content.sub(/\^?(?:\+-?>?<?\d+)+\^?\z/, '')
+      @raw_content = Element.split_path(content).first
 
       parsed = Markup.parse(prepare_markup(content))
 
@@ -125,9 +122,33 @@ module RSyntaxTree
     # something else: a `---` rule line of its own, and the `+-2` markers
     # at the end. Shared by swap_hyphen_markup and escape_hyphens.
     def self.hyphen_safe_lines(text)
-      path = text[/\^?(?:\+-?>?<?\d+)+\^?\z/]
-      body = path ? text[0...-path.length] : text
-      [body.split('\n', -1), path.to_s]
+      body, path = split_path(text)
+      [body.split('\n', -1), path]
+    end
+
+    # The run of path markers a label ends in: `+1`, `+>2`, `+-3`, several of
+    # them, an optional `^` on either side. A marker's `+` is a marker only
+    # when the backslashes before it are even in number: `x\+1` is a plus
+    # sign and a digit, `x\\+1` a backslash and a marker. Detection used to
+    # ignore the backslash, so a label could not contain "+1" at all — every
+    # "C++11" and "2+2" was read as one end of a movement path.
+    PATH_MARKERS = /(?:\+-?>?<?\d+)+/
+
+    # Splits a label into what is drawn and its trailing path text, the
+    # latter empty when there is none. A label that is nothing but markers
+    # has no path: there would be nothing to draw them from.
+    def self.split_path(text)
+      i = 0
+      while (i = text.index("+", i))
+        tail = text[i..]
+        if i.positive? && tail.match?(/\A#{PATH_MARKERS}\^?\z/) &&
+           text[0...i][/\\*\z/].length.even?
+          i -= 1 if i > 1 && text[i - 1] == "^"
+          return [text[0...i], text[i..]]
+        end
+        i += 1
+      end
+      [text, ""]
     end
 
     def swap_hyphen_markup(text)
@@ -366,7 +387,7 @@ module RSyntaxTree
             # as a curly apostrophe (U+2019) for smarter typography, e.g. the
             # X-bar prime in "T'". Applied before metrics so the measured glyph
             # matches the rendered one.
-            text = text.gsub("'", "’")
+            text = text.gsub("'", "’").gsub(Markup::KEPT_APOSTROPHE, "'")
             e[:text] = text.gsub(" ", WHITESPACE_BLOCK)
                           .gsub(">", '&#62;')
                           .gsub("<", '&#60;')

data/lib/rsyntaxtree/escape.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+#==========================
+# escape.rb
+#==========================
+#
+# Turns arbitrary text into notation that draws as that text.
+# Copyright (c) 2007-2026 Yoichiro Hasebe <yohasebe@gmail.com>
+
+module RSyntaxTree
+  # The characters the notation reads as markup, each written as itself by
+  # a backslash. The backslash comes first so that the others are escaped
+  # once, not twice. A yen sign is here because the reader treats it as a
+  # backslash for keyboards that have no other.
+  ESCAPED_CHARACTERS = %w[\\ [ ] < > ^ + * _ = ~ | # { } % @ ¥].freeze
+
+  ESCAPE_CONTEXTS = %i[word phrase label cell].freeze
+
+  # The web interface's transport spellings of characters a form cannot
+  # carry, read out of the input before anything else. Under hyphen: markup
+  # the hyphens are escaped and nothing matches; under :literal they would
+  # be read, so a backslash goes in after the first hyphen — an escape the
+  # reader does not know, whose backslash it drops — and the text survives.
+  TRANSPORT_PLACEHOLDERS = %w[AMP PERCENT PRIME SCOLON OABRACKET CABRACKET].freeze
+
+  # A line of three or more hyphens and nothing else is the horizontal rule.
+  RULE_LINE = /\A-{3,}\z/
+
+  # Notation that draws as +text+, for a program writing notation from
+  # strings it did not choose — a tagger's tokens, a corpus's words.
+  # Written here rather than in every such program, because a copy of the
+  # escaping rules kept elsewhere would drift from the notation as it
+  # changes, and the drift would show as a different figure drawn without
+  # complaint. The test for this method draws each result and checks that
+  # the text came out, so the rules are never restated, only exercised.
+  #
+  # +as+ says where the text will stand, which decides what whitespace
+  # means: a :word is one leaf or one label (a space inside it becomes <>),
+  # a :phrase is a leaf of several words (spaces stay, so the leaf gets a
+  # triangle), a :label is a node label (like a word, with a newline kept
+  # as a line break), and a :cell is one cell of a column-aligned or
+  # matrix label (tabs and newlines become the column and row breaks).
+  #
+  # +hyphen+ must match the option the figure is drawn with. Under :markup
+  # a hyphen is escaped; under :literal a bare hyphen is already itself,
+  # and the escaped form means an underline instead — so there is no
+  # notation that reads as a hyphen under both, and the caller has to say.
+  #
+  # +apostrophe+ :curly leaves a straight apostrophe to be set as a curly
+  # one, which is what the notation does; :keep escapes it so it stays
+  # straight, for text that must come out exactly as given.
+  #
+  # One text has no spelling under hyphen: :literal — a line of nothing but
+  # three or more hyphens, which is the horizontal rule and draws as one —
+  # and asking for it raises ArgumentError rather than returning notation
+  # that draws an empty leaf. Empty and all-whitespace text is returned as
+  # the whitespace it holds, which draws as a blank; a caller building a
+  # tree from tokens will usually want to drop such tokens first.
+  def self.escape(text, as: :word, hyphen: :markup, apostrophe: :curly)
+    unless ESCAPE_CONTEXTS.include?(as)
+      raise ArgumentError, "as: must be one of #{ESCAPE_CONTEXTS.join(', ')}"
+    end
+    raise ArgumentError, "hyphen: must be :markup or :literal" unless %i[markup literal].include?(hyphen)
+    raise ArgumentError, "apostrophe: must be :curly or :keep" unless %i[curly keep].include?(apostrophe)
+
+    s = text.to_s
+    if as == :phrase
+      words = s.split(/\s+/).reject(&:empty?)
+      # One word alone is the whole leaf; with company, hyphens are text.
+      refuse_rule_line(s, hyphen) if hyphen == :literal && words.length == 1 && words.first.match?(RULE_LINE)
+      return words.map { |w| escape_text(w, :word, hyphen, apostrophe) }.join(" ")
+    end
+
+    if hyphen == :literal
+      # A newline is a line break in a label or a cell, and a space in a word.
+      lines = %i[label cell].include?(as) ? s.split(/\r\n?|\n/, -1) : [s]
+      lines.each { |line| refuse_rule_line(s, hyphen) if line.match?(RULE_LINE) }
+    end
+    escape_text(s, as, hyphen, apostrophe)
+  end
+
+  # The character work, once the text has been accepted.
+  def self.escape_text(s, as, hyphen, apostrophe)
+    # Every replacement is a block: as a replacement string a backslash is
+    # read for backreferences, and `\\'` and `\\+` are two of them.
+    out = s.gsub(/[#{Regexp.escape(ESCAPED_CHARACTERS.join)}]/) { |c| "\\#{c}" }
+    out = out.gsub("-") { "\\-" } if hyphen == :markup
+    if hyphen == :literal
+      out = out.gsub(/-(#{TRANSPORT_PLACEHOLDERS.join('|')})-/) { "-\\#{Regexp.last_match(1)}-" }
+    end
+    out = out.gsub("'") { "\\'" } if apostrophe == :keep
+    out = out.gsub(/\r\n?|\n/) { "\\n" } if %i[label cell].include?(as)
+    out = out.gsub("\t") { "\\t" } if as == :cell
+    # A CRLF is one break, so one space, as it is one break in a label.
+    out.gsub(/\r\n|\s/) { "<>" }
+  end
+  private_class_method :escape_text
+
+  def self.refuse_rule_line(text, hyphen)
+    raise ArgumentError,
+          "no notation draws #{text.inspect} under hyphen: #{hyphen.inspect}: a line of " \
+          "nothing but hyphens is the horizontal rule. Draw with hyphen: :markup."
+  end
+  private_class_method :refuse_rule_line
+end

data/lib/rsyntaxtree/markup_parser.rb

@@ -39,9 +39,13 @@ class MarkupParser < Parslet::Parser
 
   rule(:path) { (str('+') >> str('-').maybe >> (str('>') | str('<')).maybe >> match('\d').repeat(1)).as(:path) }
   # rule(:escaped) { str('\\') >> match('[#<>{}\\^+*_=~\|\n\-]').as(:chr) }
-  rule(:escaped) { str('\\') >> match('[#<>{}\\\\^+*_=~\\|\\n\\-\\[\\]%]').as(:chr) }
+  rule(:escaped) { str('\\') >> match('[#<>{}\\\\^+*_=~\\|\\n\\-\\[\\]%@]').as(:chr) }
+  # A straight apostrophe is set as a curly one; `\'` asks for the straight
+  # one itself. It comes through as a placeholder so the substitution, which
+  # runs later on the joined text, can tell the two apart.
+  rule(:kept_apostrophe) { str("\\'").as(:kept_apostrophe) }
   rule(:non_escaped) { ((match('[#<>{}\\^+*_=~\|\-]') | str('\\n') | str('\\t')).absent? >> any).as(:chr) }
-  rule(:text) { (escaped | non_escaped).repeat(1).as(:text) }
+  rule(:text) { (kept_apostrophe | escaped | non_escaped).repeat(1).as(:text) }
 
   # Column separator. Every line of a label is cut at these into cells, and
   # each column is laid out at the width of its widest cell, which is what an
@@ -87,7 +91,7 @@ class MarkupParser < Parslet::Parser
   rule(:matrix_line) { (rule_line | double_rule_line | matrix_markup.repeat(1).as(:line)) >> (cr | str('#)').present?) }
   rule(:matrix_markup) { (matrix | tabstop | matrix_text | decoration | shape | bstroke) }
   # Text inside a matrix stops at the closing delimiter as well.
-  rule(:matrix_text) { (escaped | (str('#)').absent? >> non_escaped)).repeat(1).as(:text) }
+  rule(:matrix_text) { (kept_apostrophe | escaped | (str('#)').absent? >> non_escaped)).repeat(1).as(:text) }
 
   rule(:markup) { (matrix | tabstop | text | decoration | shape | bstroke) }
 
@@ -107,10 +111,15 @@ class MarkupParser < Parslet::Parser
 end
 
 module Markup
+  # Stands in for an escaped apostrophe until the curly substitution has
+  # run. A private-use character no label carries.
+  KEPT_APOSTROPHE = "\uE000"
+
   @parser = MarkupParser.new
 
   @evaluator = Parslet::Transform.new do
     rule(chr: simple(:chr)) { chr.to_s }
+    rule(kept_apostrophe: simple(:kept)) { KEPT_APOSTROPHE }
     rule(text: sequence(:text)) { { text: text.join(""), decoration: [] } }
 
     rule(tabstop: subtree(:empty)) {

data/lib/rsyntaxtree/notation_core.md

@@ -24,9 +24,16 @@ figure that was meant.
   `\n` or `\t` in it: there it can split a value that carries markup, and it
   breaks a nested #( ... #) matrix. Inside any label with columns, write a
   space as `<>`: 'a<>toy', not 'a toy'.
-- A straight apostrophe is typeset as the curly ’ (U+2019).
+- A straight apostrophe is typeset as the curly ’ (U+2019); \' keeps it
+  straight.
+- A `+` followed by digits at the end of a label is a movement path (+1, +>2).
+  \+ is a plus sign, so C\+\+11 and 2\+2 draw as written.
 - A backslash takes the character after it, whatever it is: \q draws q, and
   C:\path draws C:path. A backslash itself is \\.
+- A program building labels from text it did not choose should not copy
+  these rules: RSyntaxTree.escape(text, as: :word) returns notation that
+  draws as the text (as: :phrase, :label or :cell for the other places;
+  hyphen: to match the hyphen option; apostrophe: :keep to keep it straight).
 - The prefixes of a label compose in one order, and only this one:
   `^` → `#`/`##`/`###` → `%` → `@color:`. So ^#%@red:NP is read whole, while
   @red:%NP leaves the % as a literal character and silently drops the shade.

data/lib/rsyntaxtree/string_parser.rb

@@ -64,7 +64,12 @@ module RSyntaxTree
     end
 
     def self.valid?(data)
-      raise RSTError.new(+"Error: input text is empty", code: :empty_input, retryable: false) if data.empty?
+      # Whitespace alone is an empty input, not a tree with one blank label:
+      # nothing survives the parse, and what used to happen is that the
+      # drawing walked off the end of an empty element list and reported a
+      # defect in the library for what the caller could see and fix. A space
+      # meant as a label is written <>, which is not whitespace here.
+      raise RSTError.new(+"Error: input text is empty", code: :empty_input, retryable: false) if data.strip.empty?
 
       if /\[\s*\]/m =~ data
         raise RSTError.new(+"Error: inside the brackets is empty", code: :empty_brackets,
@@ -204,7 +209,11 @@ module RSyntaxTree
       token = ""
       i = 0
 
-      return "" if (@pos + 1) >= data.length
+      # The cursor is exhausted when it reaches the end, not one before it:
+      # counting from @pos + 1 skipped the last character, so an input of a
+      # single character — "A", or "<>" once it is one whitespace block —
+      # produced no tokens at all and drew from an empty element list.
+      return "" if @pos >= data.length
 
       escape = false
       while ((@pos + i) < data.length) && !gottoken
@@ -247,7 +256,7 @@ module RSyntaxTree
         # `\#` arrived at the grammar as a bare '#', which opens an enclosure,
         # so a label written with a hash in it lost the hash and everything
         # after it went inside brackets instead.
-        when /[nt{}<>^+*_=~|%\-#]/
+        when /[nt{}<>^+*_=~|%\-#@']/
           if escape
             token += '\\' + ch
             escape = false

data/lib/rsyntaxtree/svg_graph.rb

@@ -623,7 +623,11 @@ module RSyntaxTree
         element.ink_top = first_baseline - f_m.ink_above
         element.ink_bottom = text_y + (l_m.ink_height - l_m.ink_above)
       end
-      @tree_data += text_data.sub(/CONTENT/, new_text)
+      # The block form, so the label goes in as it is. As a replacement
+      # string it was read for backreferences, and `\+` — the last matched
+      # group, of which there is none — took a backslash and a plus sign out
+      # of any label that carried them.
+      @tree_data += text_data.sub(/CONTENT/) { new_text }
     end
 
     # Draws one run of a label — a stretch of text with its decorations, or the

data/lib/rsyntaxtree/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RSyntaxTree
-  VERSION = "2.2.0"
+  VERSION = "2.4.0"
 end

data/lib/rsyntaxtree.rb

@@ -133,6 +133,50 @@ class RSTError < StandardError
     super(msg.gsub(WHITESPACE_BLOCK, "<>"))
   end
 
+  # Every code an error from this library can carry. Consumers dispatch on
+  # these — a translation keyed by code, a harness counting what kinds of
+  # mistake occur — and both need to know the whole set, not only the codes
+  # they have happened to see: a kind of mistake nobody made is not the same
+  # as a kind that cannot happen. Until this list existed the set was only
+  # discoverable by reading five files and a repair table.
+  #
+  # Written out rather than computed, because it is a contract and a contract
+  # should be readable at a glance and visible in a diff. A test keeps it
+  # honest by finding the codes the library actually raises and comparing.
+  #
+  # The list grows and does not churn: a code once published is not renamed
+  # or removed, so a consumer keyed by code — a translation table, a tally of
+  # what kinds of mistake a writer makes — only ever has entries to add.
+  #
+  # Eight are named by the repair table, which diagnoses a label by rewriting
+  # it and asking the parser whether the rewrite reads: angle_brackets,
+  # bare_hyphen, incomplete_path, invalid_color, rule_name_without_derivation,
+  # stray_triangle, unclosed_markup, unclosed_matrix. invalid_markup is where
+  # a label lands whose mistake no single rewrite fits, which in practice
+  # means more than one. unknown_color and label_split are mistakes inside a
+  # label too, found before the repair table is reached.
+  CODES = %i[
+    angle_brackets
+    bare_hyphen
+    empty_brackets
+    empty_input
+    incomplete_path
+    internal_error
+    invalid_color
+    invalid_markup
+    invalid_option
+    label_split
+    path_multiple_ends
+    path_single_end
+    result_too_big
+    rule_name_without_derivation
+    stray_triangle
+    unbalanced_brackets
+    unclosed_markup
+    unclosed_matrix
+    unknown_color
+  ].freeze
+
   # Where the notation is written down. A hint repairs the mistake in front of
   # it and says nothing about the rest, which is enough for a reader who knows
   # the notation and not enough for one who was guessing at it. Given once, at
@@ -166,6 +210,7 @@ require_relative 'rsyntaxtree/tikz_generator'
 require_relative 'rsyntaxtree/version'
 require_relative 'rsyntaxtree/string_parser'
 require_relative 'rsyntaxtree/format_converter'
+require_relative 'rsyntaxtree/escape'
 
 require 'cgi'
 require 'rsvg2'
@@ -435,8 +480,21 @@ module RSyntaxTree
       !OFF.include?(value.to_s.strip.downcase)
     end
 
+    # Whether the input is nothing to draw. Whitespace alone counts: it
+    # leaves no label behind, and reporting it as a defect in the library
+    # told the caller their own fixable input was our fault. Asked without
+    # strip when the bytes are not valid UTF-8, because strip raises there
+    # and that input is not blank — it is malformed, which the caller
+    # learns further down.
+    def self.blank?(text)
+      s = text.to_s
+      return true if s.empty?
+
+      s.valid_encoding? && s.strip.empty?
+    end
+
     def self.check_data(text, params = {})
-      raise RSTError.new(+"Error: input text is empty", code: :empty_input, retryable: false) if text.to_s == ""
+      raise RSTError.new(+"Error: input text is empty", code: :empty_input, retryable: false) if blank?(text)
 
       begin
         StringParser.valid?(text)
@@ -510,7 +568,7 @@ module RSyntaxTree
     def self.diagnose(text, params = {})
       errors = []
       note = nil
-      if text.to_s == ""
+      if blank?(text)
         errors << RSTError.new(+"Error: input text is empty", code: :empty_input, retryable: false)
       else
         begin

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rsyntaxtree
 version: !ruby/object:Gem::Version
-  version: 2.2.0
+  version: 2.4.0
 platform: ruby
 authors:
 - Yoichiro Hasebe
@@ -128,6 +128,7 @@ files:
 - lib/rsyntaxtree/color_names.rb
 - lib/rsyntaxtree/element.rb
 - lib/rsyntaxtree/elementlist.rb
+- lib/rsyntaxtree/escape.rb
 - lib/rsyntaxtree/format_converter.rb
 - lib/rsyntaxtree/json_graph.rb
 - lib/rsyntaxtree/markup_parser.rb