rouge-carve 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c4006805a5023dc8b016745f1ab3dc73bf29fa24de27acc46c19696658a9ee36
-  data.tar.gz: 85965e83d476add300683d7e5ce7464d6affb784b067b3797318ef45575d5c78
+  metadata.gz: 4b5dd23d4f12415b4c19ed7333a5bd1555ae21e4f547b4f0a8f52b7829f8ad94
+  data.tar.gz: 1f9a6066c9c55b5457eda908909ff6b865d84fedaebbe18a37fdbe4f9e62a76d
 SHA512:
-  metadata.gz: eb1fb40ca27583ebb19f1af32bf208efcdd5d21c2f880f5e400f4c06562d4a79a977988008ec054d448d1e7ac9c0020acdce83db8b2cd19d0bcb6912878896dd
-  data.tar.gz: 7bf48fae5c12930737941c834ec4347dcd5f1a30a77e4cbcbdf99e8fa20c4d90c39d9f45bfaa9d1be3a2edea3d1025ef1f31f5ffa3ecbfade9212e0a4a4811ff
+  metadata.gz: bdc875e2a50141600798ff0efcd8741e751516dbb3de632f55039a62e881f4edbe4561942e8b39228e6603c1e6cef428d5320323b2617213d5f8ceed1053e550
+  data.tar.gz: 8d1983500c85ecef17ac25bac9aad4ebf76b477bf75ee721d1343ef09a82cf424c86c4bf9cb65b948361183bf97cce2bffe90630f20fa519368d5253fed1f13c

data/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## [Unreleased]
+
+## [0.1.3] - 2026-10-14
+
+0.1.2 was tagged without its `RougeCarve::VERSION` bump, so it never reached RubyGems. 0.1.3 is the first published release with its change.
+
+- An include selector whose id starts with a digit is lexed as a selector. `{{ ch.crv #2024-plan }}` used to fall through to plain text (#15, #16).
+- A footnote or abbreviation definition ends with its line. It used to turn the rest of the file into plain text, so no later heading, list or fence was recognized (#22).
+- An inline footnote `^[` or extension `:name[` with no closing `]` in its block stays text instead of swallowing the blocks after it. A `]` inside a code span does not count, and a heading, caption or table cell keeps the body on its line (#22, #24).
+- A blockquote line can open a block: `> # Heading`, `> - item`, and a fence whose lines carry the `>` prefix (#22).
+- A `>` marks a quote only when a space or the line end follows it, so `>> x` is paragraph text and `> > x` nests, as in the engine (#22).
+- A comment fence closes only on a `%` run of its own length, which may be followed by text. An opener with no such closer is a one-line comment, as in the engine (#22).
+- A standalone attribute block no longer runs across a blank line (#22).
+- Bare delimiters follow Carve's flanking rules, so `snake_case_name`, `/usr/local/bin`, `x=1 and y=2`, `2*3*4` and the text between `==>` arrows stay text (#23).
+- Emphasis content is lexed inline: an escape, a code span, a link or another kind of emphasis inside a run keeps its own token, and `*a \* b*` closes where the engine closes it (#23). A run opened inside another must close before it, so `*a /b* c/` is strong `a /b` (#24).
+- A list marker or a caption under a paragraph line stays text, as in the engine; headings, quotes, tables, containers and closed fences still interrupt it. A wrapped definition term keeps its definition (#23).
+- A table row needs its closing pipe, `+` is no longer read as a bullet, and an alphabetic ordered marker is one letter or a roman numeral, so `Body.` is a word (#23).
+- Lexing a long line of unclosed openers (`*a *a *a ...`, `{{ a {{ a ...`) is linear in its length; it used to be quadratic (#23).
+- The gem now requires Rouge 3.30 or later. Rouge 3.0, which the old constraint admitted, cannot load the lexer (#23).
+
+## [0.1.2] - 2026-10-08
+
+- An include selector whose id starts with a digit is lexed as a selector. `{{ ch.crv #2024-plan }}` used to fall through to plain text, because the lexer required a letter or underscore first while the spec's `explicit_identifier` has admitted a leading digit since carve 0.1.8 (#15).
+
 ## [0.1.1] - 2026-09-21
 
 - The reserved include directive `{{ path #section @key:value }}` (PART 9

data/README.md

@@ -14,6 +14,8 @@ Rouge::Lexer.find('carve')                    # => Rouge::Lexers::Carve
 Rouge::Lexer.guess(filename: 'notes.crv')     # => Rouge::Lexers::Carve
 ```
 
+It needs Rouge 3.30 or later and is tested against Rouge 3, 4 and 5.
+
 Requiring the gem registers the lexer. Anything that already highlights with
 Rouge - GitLab, Jekyll, Redcarpet, most Ruby static site generators - then
 handles `.crv` and `.carve` files, and ` ```carve ` fenced blocks, without

data/lib/rouge/lexers/carve.rb

@@ -3,7 +3,8 @@
 
 module Rouge
   module Lexers
-    # Carve is a post-Markdown lightweight markup language. Its inline
+    # Carve is a lightweight markup language for documents and the web. Its
+    # inline
     # delimiters deliberately differ from Markdown's, which is why lexing a
     # Carve document as Markdown produces actively WRONG output rather than
     # merely plain text:
@@ -48,7 +49,7 @@ module Rouge
     # Spec: https://markup-carve.github.io/carve/
     class Carve < RegexLexer
       title 'Carve'
-      desc 'Carve, a post-Markdown lightweight markup language'
+      desc 'Carve, a lightweight markup language for documents and the web'
 
       tag 'carve'
       aliases 'crv'
@@ -62,28 +63,37 @@ module Rouge
       # One attribute block, brace to brace. Quoted values may contain a brace
       # and an escaped quote, so the value alternatives come before the
       # bare-character one; a single nested brace pair is allowed for a braced
-      # span written inside.
-      ATTRS = %r/
-        \{(?=[.:}'"]|\#[\w-]+[\s}]|[A-Za-z][\w-]*(?:[=\s}]|$))(?:
+      # span written inside. The tail is the block after its `{`.
+      ATTRS_TAIL = %r/
+        (?=[.:}'"]|\#[\w-]+[\s}]|[A-Za-z][\w-]*(?:[=\s}]|$))(?:
           "(?:[^"\\\n]|\\.)*"
           |'(?:[^'\\\n]|\\.)*'
           |\{[^{}\n]*\}
           |[^{}\n]
         )*\}
       /x.freeze
+      ATTRS = /\{#{ATTRS_TAIL}/.freeze
 
-      # The same block as a STANDALONE ATTRIBUTE LINE, which may span lines.
-      # The inline form deliberately cannot, because an unclosed inline `{`
-      # would otherwise swallow the rest of the document.
+      # The same block as a STANDALONE ATTRIBUTE LINE, which may span lines
+      # but not a blank one: a blank line ends the block it would attach to.
+      # The inline form cannot span lines at all, because an unclosed inline
+      # `{` would otherwise swallow the rest of the document.
+      LINE_BREAK = '\n(?![ \t]*(?:\n|\z))'
       ATTRS_LINE = %r/
         \{(?=[.:}'"]|\#[\w-]+[\s}]|[A-Za-z][\w-]*(?:[=\s}]|$))(?:
-          "(?:[^"\\]|\\.)*"
-          |'(?:[^'\\]|\\.)*'
-          |\{[^{}]*\}
-          |[^{}]
+          "(?:[^"\\\n]|\\.|#{LINE_BREAK})*"
+          |'(?:[^'\\\n]|\\.|#{LINE_BREAK})*'
+          |\{(?:[^{}\n]|#{LINE_BREAK})*\}
+          |[^{}\n]
+          |#{LINE_BREAK}
         )*\}
       /x.freeze
 
+      # The prefix of a line inside a blockquote, nested markers included.
+      # Each `>` needs a space or the line end after it: `>foo` and `>> x`
+      # are paragraph text.
+      QUOTE_PREFIX = /(?:[ \t]*>(?=[ \t]|$))+[ \t]?/.freeze
+
       # A bracketed label that may itself contain three levels of brackets. A
       # regex cannot match arbitrarily balanced brackets, and a link label in
       # practice nests a level or two (`[t[z]](/u)`). Bounding the nesting keeps
@@ -96,6 +106,69 @@ module Rouge
         Regexp.new("\\[#{pattern}\\]").freeze
       end
 
+      # An ordered marker: a number, one letter, or a roman numeral. `Body.`
+      # is a word, not a marker.
+      ORDERED = '(?:\\d+|[A-Za-z]|[ivxlcdm]+|[IVXLCDM]+)[.)]'
+
+      # A list marker at the start of a line, as the list rules below read it.
+      LIST_MARKER = "(?:[-*]|#{ORDERED}|\\.)(?:#{ATTRS})?(?:[ \\t]|$)"
+
+      # A table row needs its closing pipe: `| not a table` is a paragraph.
+      ROW_END = /(?=(?:\\.|[^\\\n])*\|[ \t]*(?:#{ATTRS})?[ \t]*$)/.freeze
+
+      # The block openers that interrupt a paragraph. A list marker and a
+      # caption do not, and a fence does only when it is closed - that one is
+      # checked in code, see the :paraline state.
+      INTERRUPT = %r/
+        \#{1,6}[ \t] | >(?:[ \t]|$) | ::: | ::[ \t] | %%
+        | (?:(?:\*[ \t]*){3,}|(?:-[ \t]*){3,}|(?:_[ \t]*){3,})$
+        | \|=?[<>^v~]*#{ROW_END}
+        | (?:\[\^|\*\[|\[)[^\]\n]+\]:[ ]
+        | #{ATTRS_LINE}[ \t]*$
+      /x.freeze
+
+      # Where a bracketed inline body must close, by the block it sits in: a
+      # paragraph ends at a blank line or an interrupting block, a list item
+      # also at the next marker, and a heading or table row at its line end.
+      # A fence ends either only when it is closed, which is checked in code.
+      BODY_ENDS = {
+        para: /\n(?=[ \t]*(?:\n|\z)|#{MARGIN}(?:#{INTERRUPT}))/,
+        item: /\n(?=[ \t]*(?:\n|\z)|#{MARGIN}(?:#{INTERRUPT}|#{LIST_MARKER}|:[ \t]))/,
+        line: /\n/
+      }.freeze
+      FENCE_NEXT = /\n(?=#{MARGIN}(`{3,}|~{3,}))/.freeze
+
+      # The block states that decide which of those ends applies.
+      BODY_SCOPE = {
+        paragraph: :para, listitem: :item, dlterm: :item, heading: :line,
+        caption: :line, quoteline: :line, tablerow: :line, defline: :line
+      }.freeze
+
+      # A paragraph that is a lone image or display math: a caption under it
+      # is that block's.
+      IMAGE_LINE = %r/
+        #{MARGIN}(?:!#{LABEL}(?:\([^)\n]*\)|#{LABEL})?|\$\$(`+)[^\n]*?\1)(?:#{ATTRS})?[ \t]*$
+      /x.freeze
+
+      # The braced inline families: delimiter, content token, state.
+      BRACED = {
+        '*' => [Generic::Strong, :forcedstrong], '/' => [Generic::Emph, :forcedemph],
+        '_' => [Generic::Emph, :forcedunderline], '^' => [Generic::Emph, :superscript],
+        ',' => [Generic::Emph, :subscript], '=' => [Generic::Inserted, :bracedmark],
+        '+' => [Generic::Inserted, :insertion], '-' => [Generic::Deleted, :deletion]
+      }.freeze
+
+      # The bare emphasis delimiters: state, content token, and the characters
+      # that may not stand before an opener or after it. A closer needs a
+      # non-space before it and no letter or digit after it.
+      BARE = {
+        '*' => [:strong, Generic::Strong, /[\w*]/, /[\s*]/],
+        '/' => [:emph, Generic::Emph, /[\w\/]/, /[\s\/]/],
+        '_' => [:underline, Generic::Emph, /[\w_\/]/, /[\s_]/],
+        '~' => [:strike, Generic::Deleted, /[\w~]/, /[\s~]/],
+        '=' => [:mark, Generic::Inserted, /[\w=<>!]/, /[\s>=]/]
+      }.freeze
+
       # Characters that can begin an inline construct. A run of anything else is
       # ordinary content and is emitted as ONE token - without this every
       # content character becomes its own token, which is both noisy and
@@ -118,6 +191,147 @@ module Rouge
       # ------------------------------------------------------------------
       # Block level
       # ------------------------------------------------------------------
+
+      # The openers a blockquote marker can introduce as well as a line start.
+      # They are written once and instantiated twice: anchored at a line start
+      # for the document, and unanchored for the rest of a line after `>`,
+      # where the scanner already sits where the block would begin. Each one
+      # enters its state through #open_block, which knows which of the two it
+      # is running in.
+      def self.block_openers(bol)
+        proc do
+          # A raw block: the `=FORMAT` info string routes the payload to that
+          # output format verbatim. Emitted as one token including the `=`,
+          # because the format word without its sigil is not the construct.
+          # The payload is never a language, so it is never delegated.
+          rule %r/#{bol}(#{MARGIN})(`{3,}|~{3,})([ \t]*)(=[a-zA-Z][\w+.-]*)([^\n]*)/ do |m|
+            groups Text, Punctuation, Text, Keyword::Type
+            token Text, m[5]
+            push_fence(m[2], nil)
+          end
+
+          # A code fence. The first word of the info string names a language,
+          # and a lexer for it highlights the payload - the one place this
+          # lexer looks past Carve, because a reader of a `.crv` file expects
+          # an embedded PHP sample to look like PHP.
+          rule %r/#{bol}(#{MARGIN})(`{3,}|~{3,})([ \t]*)([a-zA-Z][\w+#.-]*)?([^\n]*)/ do |m|
+            groups Text, Punctuation, Text, Name::Builtin
+            lex_info_string(m[5])
+            push_fence(m[2], m[4])
+          end
+
+          # Container divs. A reserved kind word (note, tip, figure, ...)
+          # names a known container; `:::` followed by `|` is the layout form.
+          rule %r/#{bol}(#{MARGIN})(:{3,})([ \t]*)(\|)/ do
+            groups Text, Punctuation, Text, Operator
+            open_block
+          end
+          rule %r/#{bol}(#{MARGIN})(:{3,})([ \t]*)([a-zA-Z][\w-]*)?([^\n]*)/ do |m|
+            groups Text, Punctuation, Text, Keyword::Namespace
+            lex_info_string(m[5])
+            open_block
+          end
+
+          # A caption line attaches to the block above or below it.
+          rule %r/#{bol}(#{MARGIN})(\^)([ \t]+)/ do
+            groups Text, Punctuation, Text
+            open_block :caption
+          end
+
+          # Headings. Carve has no setext form, so a `#` run is the only
+          # spelling and a trailing `{...}` is NOT an attribute block here.
+          rule %r/#{bol}(#{MARGIN})(\#{1,6})([ \t]+)/ do
+            groups Text, Punctuation, Text
+            open_block :heading
+          end
+
+          # Thematic breaks, before the list rules so `---` is not a bullet.
+          rule %r/#{bol}#{MARGIN}(?:(?:\*[ \t]*){3,}|(?:-[ \t]*){3,}|(?:_[ \t]*){3,})$/ do
+            token Punctuation
+            open_block
+          end
+
+          # Definition markers: footnote, link reference, abbreviation. The
+          # separator after the colon must START WITH A LITERAL SPACE - a
+          # tab-first separator makes the line an ordinary paragraph. Each
+          # definition ends with its line.
+          rule %r/#{bol}(#{MARGIN})(\[\^)([^\]\n]+)(\]:)( )/ do
+            groups Text, Punctuation, Name::Label, Punctuation, Text
+            open_block :defline
+          end
+          rule %r/#{bol}(#{MARGIN})(\*\[)([^\]\n]+)(\]:)( )/ do
+            groups Text, Punctuation, Name::Entity, Punctuation, Text
+            open_block :defline
+          end
+          rule %r/#{bol}(#{MARGIN})(\[)([^\]\n]+)(\]:)( )/ do
+            groups Text, Punctuation, Name::Label, Punctuation, Text
+            open_block :linkdest
+          end
+
+          # A definition-list term (`::`) and its definition (`:`). A term
+          # may wrap onto lazy lines, and a definition's text is an item's.
+          rule %r/#{bol}(#{MARGIN})(::)([ \t]+)/ do
+            groups Text, Punctuation, Text
+            open_block :dlterm, :heading
+          end
+          rule %r/#{bol}(#{MARGIN})(:)(?=[ \t])/ do
+            groups Text, Punctuation
+            open_block :listitem, :quoteline
+          end
+
+          # A blockquote marker must be followed by a space or end the line;
+          # `>foo` and `>> x` are paragraphs, and `> > x` nests. What follows
+          # a marker may open a block of its own, so the openers are tried
+          # again there.
+          rule %r/#{bol}(#{MARGIN})(>)(?=[ \t]|$)/ do
+            groups Text, Punctuation
+            state?(:quotestart) ? goto(:quotestart) : push(:quotestart)
+          end
+
+          # Task items before plain bullets, so the state marker is its own
+          # token. The state is any single character, not only a space or an
+          # x: `[>]` is deferred and `[-]` is dropped. A `+` is not a bullet in
+          # Carve; it continues a container.
+          rule %r/#{bol}(#{MARGIN})((?:[-*]|#{ORDERED})(?:#{ATTRS})?)([ \t]+)(\[[^\]\n]\])/ do
+            groups Text, Punctuation, Text, Name::Constant
+            open_block :listitem, :quoteline
+          end
+
+          # Bullets. A run of markers on one line opens nested lists at once
+          # (`- - A`), and attributes may be glued straight onto the marker.
+          rule %r/#{bol}(#{MARGIN})((?:[-*][ \t]+)*[-*](?:#{ATTRS})?)(?=[ \t]|$)/ do
+            groups Text, Punctuation
+            open_block :listitem, :quoteline
+          end
+
+          # Ordered markers: numeric, alphabetic, roman, and the bare `.` that
+          # continues the enclosing sequence.
+          rule %r/#{bol}(#{MARGIN})(#{ORDERED}(?:#{ATTRS})?)(?=[ \t]|$)/ do
+            groups Text, Num::Integer
+            open_block :listitem, :quoteline
+          end
+          rule %r/#{bol}(#{MARGIN})(\.(?:#{ATTRS})?)(?=[ \t]|$)/ do
+            groups Text, Num::Integer
+            open_block :listitem, :quoteline
+          end
+
+          # Tables. The header marker, the alignment run and the separator row
+          # are their own tokens; cell content is lexed inline. A row needs its
+          # closing pipe.
+          rule %r/#{bol}(#{MARGIN})(\|=[<>^v~]*)#{ROW_END}/ do
+            groups Text, Operator
+            open_block :tablerow
+          end
+          rule %r/#{bol}(#{MARGIN})(\|)#{ROW_END}/ do
+            groups Text, Punctuation
+            open_block :tablerow
+          end
+        end
+      end
+
+      state(:blockopeners, &block_openers('^'))
+      state(:quoteopeners, &block_openers(''))
+
       state :block do
         # Front matter, only at the very start of the document. \A is what
         # keeps a `---yaml` line mid-document from opening one.
@@ -127,134 +341,97 @@ module Rouge
         end
 
         # A comment fence (%%% or longer) versus a one-line comment. The fence
-        # has to be tried first: %%% also matches the one-line form.
-        rule %r/^(#{MARGIN})(%%%+)([^\n]*)(\n)/ do
+        # has to be tried first: %%% also matches the one-line form. Only a
+        # run of the SAME length closes it, so the width is closed over the
+        # way #push_fence does it, and an opener with no such closer below it
+        # is the one-line form. Text after either run is part of the comment.
+        rule %r/^(#{MARGIN})(%%%+)([^\n]*)(\n)/ do |m|
+          closer = %r/^([ \t]*)(%{#{m[2].length}})(?!%)([^\n]*)$/
+          unless closer_below?(closer, m[2].length)
+            # The tokens of the one-line rule below. Rouge 3 has no
+            # fallthrough! to hand the line on to it.
+            token Text, m[1]
+            token Comment::Preproc, '%%'
+            token Comment, m[2][2..] + m[3]
+            token Text, m[4]
+            next
+          end
+
           groups Text, Comment::Preproc, Comment, Text
-          push :commentfence
+          push do
+            rule closer do
+              groups Text, Comment::Preproc, Comment
+              pop!
+            end
+            rule %r/\n/, Comment
+            rule %r/[^\n]+\n?/, Comment
+          end
         end
         rule %r/^(#{MARGIN})(%%)([^\n]*)$/ do
           groups Text, Comment::Preproc, Comment
         end
 
-        # A raw block: the `=FORMAT` info string routes the payload to that
-        # output format verbatim. Emitted as one token including the `=`,
-        # because the format word without its sigil is not the construct. The
-        # payload is never a language, so it is never delegated.
-        rule %r/^(#{MARGIN})(`{3,}|~{3,})([ \t]*)(=[a-zA-Z][\w+.-]*)([^\n]*)/ do |m|
-          groups Text, Punctuation, Text, Keyword::Type
-          token Text, m[5]
-          push_fence(m[2], nil)
-        end
-
-        # A code fence. The first word of the info string names a language, and
-        # a lexer for it highlights the payload - the one place this lexer
-        # looks past Carve, because a reader of a `.crv` file expects an
-        # embedded PHP sample to look like PHP.
-        rule %r/^(#{MARGIN})(`{3,}|~{3,})([ \t]*)([a-zA-Z][\w+#.-]*)?([^\n]*)/ do |m|
-          groups Text, Punctuation, Text, Name::Builtin
-          lex_info_string(m[5])
-          push_fence(m[2], m[4])
-        end
-
-        # Container divs. A reserved kind word (note, tip, figure, ...) names a
-        # known container; `:::` followed by `|` is the layout form.
-        rule %r/^(#{MARGIN})(:{3,})([ \t]*)(\|)/ do
-          groups Text, Punctuation, Text, Operator
-        end
-        rule %r/^(#{MARGIN})(:{3,})([ \t]*)([a-zA-Z][\w-]*)?([^\n]*)/ do |m|
-          groups Text, Punctuation, Text, Keyword::Namespace
-          lex_info_string(m[5])
-        end
-
-        # A caption line attaches to the block above or below it.
-        rule %r/^(#{MARGIN})(\^)([ \t]+)/ do
-          groups Text, Punctuation, Text
-          push :caption
-        end
-
-        # Headings. Carve has no setext form, so a `#` run is the only spelling
-        # and a trailing `{...}` is NOT an attribute block here.
-        rule %r/^(#{MARGIN})(\#{1,6})([ \t]+)/ do
-          groups Text, Punctuation, Text
-          push :heading
-        end
-
-        # Thematic breaks, before the list rules so `---` is not a bullet.
-        rule %r/^#{MARGIN}(?:\*[ \t]*){3,}$/, Punctuation
-        rule %r/^#{MARGIN}(?:-[ \t]*){3,}$/, Punctuation
-        rule %r/^#{MARGIN}(?:_[ \t]*){3,}$/, Punctuation
+        mixin :blockopeners
 
-        # Definition markers: footnote, link reference, abbreviation. The
-        # separator after the colon must START WITH A LITERAL SPACE - a
-        # tab-first separator makes the line an ordinary paragraph.
-        rule %r/^(#{MARGIN})(\[\^)([^\]\n]+)(\]:)( )/ do
-          groups Text, Punctuation, Name::Label, Punctuation, Text
-          push :inline
-        end
-        rule %r/^(#{MARGIN})(\*\[)([^\]\n]+)(\]:)( )/ do
-          groups Text, Punctuation, Name::Entity, Punctuation, Text
-          push :inline
-        end
-        rule %r/^(#{MARGIN})(\[)([^\]\n]+)(\]:)( )/ do
-          groups Text, Punctuation, Name::Label, Punctuation, Text
-          push :linkdest
-        end
-
-        # A definition-list term (`::`) and its definition (`:`).
-        rule %r/^(#{MARGIN})(::)([ \t]+)/ do
-          groups Text, Punctuation, Text
-          push :heading
-        end
-        rule %r/^(#{MARGIN})(:)(?=[ \t])/ do
-          groups Text, Punctuation
+        # A standalone attribute block, which may span lines.
+        rule %r/^(#{MARGIN})(#{ATTRS_LINE})/ do
+          groups Text, Name::Attribute
         end
 
-        # A blockquote marker must be followed by a space or end the line;
-        # `>foo` is a paragraph.
-        rule %r/^(#{MARGIN})(>+)(?=[ \t]|$)/ do
-          groups Text, Punctuation
-          push :quoteline
+        # Any other line with content starts a paragraph.
+        rule %r/^(?=#{MARGIN}\S)/ do
+          @lone_image = !@current_stream.check(IMAGE_LINE).nil?
+          @indented = @current_stream.check(/[ \t]/) ? true : false
+          push :paragraph
         end
 
-        # Task items before plain bullets, so the state marker is its own
-        # token. The state is any single character, not only a space or an x:
-        # `[>]` is deferred and `[-]` is dropped.
-        rule %r/^(#{MARGIN})((?:[-*+]|\d+[.)]|[A-Za-z]+[.)])(?:#{ATTRS})?)([ \t]+)(\[[^\]\n]\])/ do
-          groups Text, Punctuation, Text, Name::Constant
-        end
+        mixin :inline
+      end
 
-        # Bullets. A run of markers on one line opens nested lists at once
-        # (`- - A`), and attributes may be glued straight onto the marker.
-        rule %r/^(#{MARGIN})((?:[-*+][ \t]+)*[-*+](?:#{ATTRS})?)(?=[ \t]|$)/ do
-          groups Text, Punctuation
-        end
+      # A paragraph, and the start of each line after its first: a line that
+      # opens a block interrupting it ends it, anything else continues it, so
+      # a list marker or a caption under prose is prose.
+      state :paragraph do
+        rule(/\n/) { token Text; goto :paraline }
+        mixin :inlinecontent
+        rule PLAIN, Text
+        rule %r/./, Text
+      end
 
-        # Ordered markers: numeric, alphabetic, roman, and the bare `.` that
-        # continues the enclosing sequence.
-        rule %r/^(#{MARGIN})((?:\d+|[A-Za-z]+)[.)](?:#{ATTRS})?)(?=[ \t]|$)/ do
-          groups Text, Num::Integer
-        end
-        rule %r/^(#{MARGIN})(\.(?:#{ATTRS})?)(?=[ \t]|$)/ do
-          groups Text, Num::Integer
-        end
+      state :paraline do
+        rule(%r/^(?=[ \t]*(?:\n|\z))/) { pop! }
+        rule(%r/^(?=#{MARGIN}(?:#{INTERRUPT}))/) { pop! }
+        rule(%r/^(?=#{MARGIN}(`{3,}|~{3,}))/) { |m| fence_closed_below?(m[1]) ? pop! : continue_paragraph(:paragraph) }
+        rule(%r/^(?=#{MARGIN}\^[ \t])/) { @lone_image ? pop! : continue_paragraph(:paragraph) }
+        # An indented paragraph sits in a container, so a list marker below it
+        # opens a sublist or the next item rather than continuing it.
+        rule(%r/^(?=#{MARGIN}#{LIST_MARKER})/) { @indented ? pop! : continue_paragraph(:paragraph) }
+        rule(//) { continue_paragraph(:paragraph) }
+      end
 
-        # Tables. The header marker, the alignment run and the separator row
-        # are their own tokens; cell content is lexed inline.
-        rule %r/^(#{MARGIN})(\|=[<>^v~]*)/ do
-          groups Text, Operator
-          push :tablerow
-        end
-        rule %r/^(#{MARGIN})(\|)/ do
-          groups Text, Punctuation
-          push :tablerow
+      # The text of a list item, a definition or a definition term. The line
+      # after it may also start the next item or definition.
+      { listitem: Text, dlterm: Generic::Heading }.each do |name, tok|
+        line = :"#{name}line"
+        state name do
+          rule(/\n/) { token Text; goto line }
+          mixin :inlinecontent
+          rule PLAIN, tok
+          rule(/./) { token plain_token }
         end
 
-        # A standalone attribute block, which may span lines.
-        rule %r/^(#{MARGIN})(#{ATTRS_LINE})/ do
-          groups Text, Name::Attribute
+        state line do
+          rule(%r/^(?=[ \t]*(?:\n|\z))/) { pop! }
+          rule(%r/^(?=#{MARGIN}(?:#{INTERRUPT}|#{LIST_MARKER}|:[ \t]))/) { pop! }
+          rule(%r/^(?=#{MARGIN}(`{3,}|~{3,}))/) { |m| fence_closed_below?(m[1]) ? pop! : goto(name) }
+          rule(//) { goto name }
         end
+      end
 
-        mixin :inline
+      # Right after a blockquote marker: a block opener, or quoted text.
+      state :quotestart do
+        mixin :quoteopeners
+        rule(//) { goto :quoteline }
       end
 
       state :frontmatter do
@@ -266,17 +443,16 @@ module Rouge
         rule %r/[^\n]+\n?/, Comment::Special
       end
 
-      state :commentfence do
-        rule %r/^([ \t]*)(%%%+)([ \t]*)$/ do
-          groups Text, Comment::Preproc, Text
-          pop!
-        end
-        rule %r/\n/, Comment
-        rule %r/[^\n]+\n?/, Comment
+      state :defline do
+        rule %r/$/, Text, :pop!
+        mixin :inlinecontent
+        rule PLAIN, Text
+        rule %r/[^\n]/, Text
       end
 
       state :heading do
         rule %r/$/, Text, :pop!
+        mixin :lineverbatim
         mixin :inlinecontent
         rule PLAIN, Generic::Heading
         rule %r/[^\n]/, Generic::Heading
@@ -284,6 +460,7 @@ module Rouge
 
       state :quoteline do
         rule %r/$/, Text, :pop!
+        mixin :lineverbatim
         mixin :inlinecontent
         rule PLAIN, Generic::Emph
         rule %r/[^\n]/, Generic::Emph
@@ -291,6 +468,7 @@ module Rouge
 
       state :caption do
         rule %r/$/, Text, :pop!
+        mixin :lineverbatim
         mixin :inlinecontent
         rule PLAIN, Generic::Subheading
         rule %r/[^\n]/, Generic::Subheading
@@ -330,6 +508,18 @@ module Rouge
         rule %r/./, Text
       end
 
+      state :lineverbatim do
+        rule %r/(!)(`+)(?!`)((?:(?!(?<!`)\2(?!`))[^\n])*)((?<!`)\2(?!`)|$)/ do
+          groups Operator, Punctuation, Literal, Punctuation
+        end
+        rule %r/(\$\$?)(`+)(?!`)((?:(?!(?<!`)\2(?!`))[^\n])*)((?<!`)\2(?!`)|$)/ do
+          groups Operator, Punctuation, Str::Other, Punctuation
+        end
+        rule %r/(`+)(?!`)((?:(?!(?<!`)\1(?!`))[^\n])*)((?<!`)\1(?!`)|$)/ do
+          groups Punctuation, Str::Backtick, Punctuation
+        end
+      end
+
       state :inlinecontent do
         # An escape wins over every delimiter that follows it. A backslash at
         # END OF LINE escapes nothing - it is a hard break, and needs its own
@@ -370,74 +560,95 @@ module Rouge
         # as a hashtag inside a path. After the verbatim family, so a directive
         # in a code span stays literal; before every other brace rule, none of
         # which can spell `{{`.
-        # THE CLOSER IS IN THE LOOKAHEAD, so an unterminated `{{` never opens
-        # the state at all and stays the ordinary text the processor treats it
-        # as. Without it the opener scoped and only the newline rule below
-        # stopped the damage.
-        rule %r/(\{\{)(?=[ \t]+[^\n]*?[ \t]+\}\})([ \t]+)((?:"(?:\\.|[^"\\])*"|[^#@}\s"][^#@}\s]*))/ do
-          groups Punctuation, Text, Name::Namespace
-          push :includeparts
+        # An unterminated `{{` never opens the state and stays the ordinary
+        # text the processor treats it as.
+        rule %r/(\{\{)([ \t]+)((?:"(?:\\.|[^"\\])*"|[^#@}\s"][^#@}\s]*))/ do |m|
+          if closer_ahead?(:include, /[ \t]\}\}/)
+            groups Punctuation, Text, Name::Namespace
+            push :includeparts
+          else
+            token plain_token, m[0]
+          end
         end
 
-        # CriticMarkup substitution and comment, before the forced family:
-        # `{~old~>new~}` also matches the forced-strike shape.
-        rule %r/(\{~)([^\n]*?)(~>)([^\n]*?)(~\})/ do
-          groups Punctuation, Generic::Deleted, Operator, Generic::Inserted, Punctuation
-        end
-        rule %r/(\{\#)([^\n]*?)(\#\})/ do
-          groups Punctuation, Comment, Punctuation
-        end
+        # The braced families. Each opener consumes only its `{` here, so an
+        # unclosed one leaves the character after it to the rules below, and
+        # its closer must be on the line - looked up in a per-line cache, so
+        # a run of unclosed openers stays linear.
+        #
+        # CriticMarkup substitution and forced strike share `{~`; the first of
+        # `~>` and `~}` decides which one it is.
+        rule %r/\{(?=~(?!~\}))/ do
+          next token(plain_token) unless closer_ahead?(:tilde, /~\}/)
 
-        # The braced FORCED emphasis family. Braces make a delimiter apply
-        # where the bare form would not - `my{_path_}name` is underline inside
-        # a word - so the content is emphasis and the braces are its
-        # delimiters, not an attribute block.
-        rule %r/(\{\*)([^\n]+?)(\*\})/ do
-          groups Punctuation, Generic::Strong, Punctuation
-        end
-        rule %r/(\{\/)([^\n]+?)(\/\})/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(\{_)([^\n]+?)(_\})/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(\{~)([^\n]+?)(~\})/ do
-          groups Punctuation, Generic::Deleted, Punctuation
+          enter_braced(:tildebrace)
         end
 
-        # Braced inline families. Sup and sub are braced-only in Carve: a bare
-        # `^x^` or `,x,` is literal text.
-        rule %r/(\{\^)([^\n]+?)(\^\})/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(\{,)([^\n]+?)(,\})/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(\{=)([^\n]+?)(=\})/ do
-          groups Punctuation, Generic::Inserted, Punctuation
-        end
-        rule %r/(\{\+)([^\n]+?)(\+\})/ do
-          groups Punctuation, Generic::Inserted, Punctuation
+        # A CriticMarkup comment. Without its closer, `{#id}` is an attribute
+        # block.
+        # The scanner's match is read before #check replaces it.
+        rule %r/\{(?=\#)/ do |m|
+          brace = m[0]
+          if closer_ahead?(:criticcomment, /\#\}/, @current_stream.charpos + 1)
+            enter_braced(:criticcomment)
+          elsif (tail = @current_stream.check(ATTRS_TAIL))
+            token Name::Attribute, brace
+            consume_as(Name::Attribute, tail)
+          else
+            token plain_token, brace
+          end
         end
-        rule %r/(\{-)([^\n]+?)(-\})/ do
-          groups Punctuation, Generic::Deleted, Punctuation
+
+        # The FORCED emphasis family and the other braced inlines. Braces make
+        # a delimiter apply where the bare form would not - `my{_path_}name`
+        # is underline inside a word - so the content is emphasis and the
+        # braces are its delimiters, not an attribute block. Sup and sub are
+        # braced-only in Carve: a bare `^x^` or `,x,` is literal text.
+        BRACED.each do |char, (_, name)|
+          delimiter = Regexp.escape(char)
+          rule %r/\{(?=#{delimiter}(?!#{delimiter}\}))/ do
+            if closer_ahead?(name, /#{delimiter}\}/, @current_stream.charpos + 2)
+              enter_braced(name)
+            else
+              token plain_token
+            end
+          end
         end
-        rule %r/(\{>>)([^\n]+?)(<<\})/ do
-          groups Punctuation, Comment, Punctuation
+        rule %r/\{(?=>>)/ do
+          if closer_ahead?(:criticnote, /<<\}/, @current_stream.charpos + 3)
+            enter_braced(:criticnote, 2)
+          else
+            token plain_token
+          end
         end
-        rule %r/(\{%)(.*?)(%\})/ do
-          groups Comment::Preproc, Comment, Comment::Preproc
+        rule %r/\{(?=%)/ do
+          if closer_ahead?(:bracedcomment, /%\}/, @current_stream.charpos + 1)
+            token Comment::Preproc
+            push :bracedcomment
+            push :preprocchar
+          else
+            token plain_token
+          end
         end
 
         # An inline footnote carries content; a reference carries a label.
-        rule %r/\^\[/, Punctuation, :inlinefootnote
-        rule %r/(\[\^)([^\]\n]+)(\])/ do
+        # Both bracketed bodies need a closing `]` before the paragraph ends;
+        # an opener without one is text, as it is in the engine.
+        rule %r/\^\[/ do
+          if (scope = bracket_ahead?)
+            token Punctuation
+            push :"inlinefootnote_#{scope}"
+          else
+            token plain_token
+          end
+        end
+        rule %r/(\[\^)([^\[\]\n]+)(\])/ do
           groups Punctuation, Name::Label, Punctuation
         end
 
         # A citation before a link: both open with `[`, and `[@key]` or
         # `[+@key]` would otherwise be read as a link label.
-        rule %r/(\[)(\+?@[^\]\n]+)(\])/ do
+        rule %r/(\[)(\+?@[^\[\]\n]+)(\])/ do
           groups Punctuation, Name::Variable, Punctuation
         end
 
@@ -468,8 +679,12 @@ module Rouge
 
         # The `:name[...]` extension form.
         rule %r/(:)([a-zA-Z][\w-]*)(\[)/ do
-          groups Punctuation, Name::Function, Punctuation
-          push :rolebody
+          if (scope = bracket_ahead?)
+            groups Punctuation, Name::Function, Punctuation
+            push :"rolebody_#{scope}"
+          else
+            token plain_token
+          end
         end
 
         # A code callout marker, and a symbol shortcode - whose name may start
@@ -480,28 +695,12 @@ module Rouge
         # An attribute block attached to the construct before it.
         rule ATTRS, Name::Attribute
 
-        # Bare emphasis delimiters. Carve's bare set is / * _ ~ = and each
-        # needs a non-space inner boundary so `a / b` stays literal.
-        rule %r/(\/\*)([^\n]+?)(\*\/)/ do
-          groups Punctuation, Generic::Strong, Punctuation
-        end
-        rule %r/(\*\/)([^\n]+?)(\/\*)/ do
-          groups Punctuation, Generic::Strong, Punctuation
-        end
-        rule %r/(\*)(\S(?:[^\n]*?\S)?)(\*)/ do
-          groups Punctuation, Generic::Strong, Punctuation
-        end
-        rule %r/(\/)(\S(?:[^\n]*?\S)?)(\/)/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(_)(\S(?:[^\n]*?\S)?)(_)/ do
-          groups Punctuation, Generic::Emph, Punctuation
-        end
-        rule %r/(~)(\S(?:[^\n]*?\S)?)(~)/ do
-          groups Punctuation, Generic::Deleted, Punctuation
-        end
-        rule %r/(=)(\S(?:[^\n]*?\S)?)(=)/ do
-          groups Punctuation, Generic::Inserted, Punctuation
+        # Bare emphasis delimiters, Carve's / * _ ~ = set. Whether one opens is
+        # decided in #open_bare, from the characters on both sides and from
+        # whether a closer follows on the line; what it encloses is lexed as
+        # inline content. `==>` is left whole for the typographic rule.
+        rule %r/[*\/_~]|=(?!=>)/ do |m|
+          open_bare(m[0])
         end
 
         # A mention and a tag, each one token: the sigil is part of the name
@@ -528,7 +727,7 @@ module Rouge
       state :includeparts do
         rule %r/[ \t]+/, Text
         rule %r/\}\}/, Punctuation, :pop!
-        rule %r/#[A-Za-z_][\w-]*/, Name::Label
+        rule %r/#[A-Za-z0-9_][\w-]*/, Name::Label
         rule %r/(@[A-Za-z_][\w-]*)(:)([^\s}]+)/ do
           groups Name::Attribute, Punctuation, Literal
         end
@@ -537,24 +736,375 @@ module Rouge
         rule %r/\n/, Text, :pop!
       end
 
-      state :inlinefootnote do
-        rule %r/\]/, Punctuation, :pop!
-        mixin :inlinecontent
-        rule PLAIN_NO_BRACKET, Generic::Emph
-        rule %r/[^\]\n]/, Generic::Emph
-        rule %r/\n/, Text
+      # The body of a bare emphasis run. A same-kind delimiter that cannot
+      # close is content (`*a *b* c*` closes at `b*`), and the run ends with
+      # its line in case a construct inside took the closer the opener saw.
+      #
+      # NO LOOKBEHIND IN A RULE. Rouge before 5 scans without a fixed anchor,
+      # so a lookbehind cannot see what precedes the scan position; the
+      # character before a closer is read in #close_bare instead.
+      BARE.each do |char, (name, _, _, _)|
+        state name do
+          rule(%r/#{Regexp.escape(char)}(?![\p{L}\p{N}])/) { close_bare(/\s/) }
+          rule(/$/) { pop! }
+          rule(/#{Regexp.escape(char)}/) { token plain_token }
+          mixin :inlinecontent
+          rule(PLAIN) { token plain_token }
+          rule(/./) { token plain_token }
+        end
       end
 
-      state :rolebody do
-        rule %r/\]/, Punctuation, :pop!
-        mixin :inlinecontent
-        rule PLAIN_NO_BRACKET, Name::Function
-        rule %r/[^\]\n]/, Name::Function
-        rule %r/\n/, Text
+      # Bold italic in one delimiter pair, in both nesting orders. Its own
+      # delimiters inside it are content unless they close it.
+      # A comment inside it ends at its closer.
+      {
+        bolditalic: ['\*\/', /\s/, '(?<=\S)\*\/'],
+        italicbold: ['\/\*(?!\w)', /[\s*]/, '(?<![\s*])\/\*(?!\w)']
+      }.each do |name, (closer, refused_before, closer_in_line)|
+        state name do
+          rule(/#{closer}/) { close_bare(refused_before) }
+          rule(/$/) { pop! }
+          rule %r/(%%)((?:(?!#{closer_in_line})[^\n])*)/ do
+            groups Comment::Preproc, Comment
+          end
+          rule(%r/[*\/]/) { token plain_token }
+          mixin :inlinecontent
+          rule(PLAIN) { token plain_token }
+          rule(/./) { token plain_token }
+        end
+      end
+
+      # The rest of an opener whose `{` was already emitted.
+      state :punctchar do
+        rule(/[^\n]/) { token Punctuation; pop! }
+      end
+      state :preprocchar do
+        rule(/[^\n]/) { token Comment::Preproc; pop! }
+      end
+
+      # Braced bodies, flat up to their closer.
+      BRACED.each do |char, (tok, name)|
+        closer = /#{Regexp.escape(char)}\}/
+        state name do
+          rule closer, Punctuation, :pop!
+          rule(/$/) { pop! }
+          rule %r/(?:(?!#{closer})[^\n])+/, tok
+        end
+      end
+
+      state :tildebrace do
+        rule %r/~>/, Operator, :tildenew
+        rule %r/~\}/, Punctuation, :pop!
+        rule(/$/) { pop! }
+        rule %r/(?:(?!~[>}])[^\n])+/, Generic::Deleted
+      end
+
+      # The new side of a substitution. Its closer ends the whole construct.
+      state :tildenew do
+        rule(/~\}/) { token Punctuation; pop!(2) }
+        rule(/$/) { pop!(2) }
+        rule %r/(?:(?!~\})[^\n])+/, Generic::Inserted
+      end
+
+      state :criticcomment do
+        rule %r/\#\}/, Punctuation, :pop!
+        rule(/$/) { pop! }
+        rule %r/(?:(?!\#\})[^\n])+/, Comment
+      end
+
+      state :criticnote do
+        rule %r/<<\}/, Punctuation, :pop!
+        rule(/$/) { pop! }
+        rule %r/(?:(?!<<\})[^\n])+/, Comment
+      end
+
+      state :bracedcomment do
+        rule %r/%\}/, Comment::Preproc, :pop!
+        rule(/$/) { pop! }
+        rule %r/(?:(?!%\})[^\n])+/, Comment
+      end
+
+      # Both bodies end at their bracket, or where their block ends when a
+      # construct inside consumed the bracket the opener saw. The newline is
+      # left to the block, which decides what the next line opens.
+      { inlinefootnote: Generic::Emph, rolebody: Name::Function }.each do |body, tok|
+        BODY_ENDS.each do |scope, ends|
+          state :"#{body}_#{scope}" do
+            rule %r/\]/, Punctuation, :pop!
+            rule(%r/(?=#{ends.source}|\n?\z)/) { pop! }
+            # The newline is handed back so the block reads the fence line.
+            rule(FENCE_NEXT) do |m|
+              if fence_closed_below?(m[1])
+                m.pos -= 1
+                pop!
+              else
+                token Text
+              end
+            end
+            mixin :inlinecontent
+            rule PLAIN_NO_BRACKET, tok
+            rule %r/[^\]\n]/, tok
+            rule %r/\n/, Text
+          end
+        end
       end
 
       private
 
+      # The token a state gives a character that opens nothing.
+      PLAIN_TOKENS = {
+        heading: Generic::Heading, quoteline: Generic::Emph,
+        caption: Generic::Subheading, dlterm: Generic::Heading, bolditalic: Generic::Strong,
+        italicbold: Generic::Strong
+      }.merge(BARE.values.to_h { |name, tok, _, _| [name, tok] })
+        .merge(BODY_ENDS.keys.flat_map do |scope|
+          [[:"inlinefootnote_#{scope}", Generic::Emph], [:"rolebody_#{scope}", Name::Function]]
+        end.to_h).freeze
+
+      # Italic or underline inside strong, or the reverse, reads as strong,
+      # the token the combined delimiters have always had.
+      def plain_token
+        name = state.name
+        if %i[emph underline].include?(name) && in_state?(:strong)
+          Generic::Strong
+        else
+          PLAIN_TOKENS.fetch(name, Text)
+        end
+      end
+
+      # Decide whether the bare delimiter +char+, just consumed, opens a run.
+      # It opens when the characters around it allow it and a closer follows
+      # on its line; otherwise it is plain text. `/*` and `*/` open bold
+      # italic first, with the engine's guards for each order.
+      def open_bare(char)
+        text = @current_stream.string
+        pos = @current_stream.charpos
+        before = pos > 1 ? text[pos - 2] : nil
+        before = nil if before == "\n"
+        after = text[pos]
+        name, _, refused_before, refused_after = BARE[char]
+
+        limit = enclosing_closer(pos)
+
+        if char == '/' && after == '*' && text[pos + 1].to_s.match?(/\S/) &&
+           closer_ahead?(:bolditalic, COMBINED_CLOSERS[:bolditalic], pos + 2, masked: :combined, before: limit)
+          token Punctuation
+          push :bolditalic
+          return push(:punctchar)
+        end
+
+        if char == '*' && after == '/' && !before.to_s.match?(/[\w*\/]/) &&
+           text[pos + 1].to_s.match?(/\S/) && !(text[pos + 1] == '*' && text[pos + 2].to_s.match?(/[\s*]/)) &&
+           closer_ahead?(:italicbold, COMBINED_CLOSERS[:italicbold], pos + 2, masked: :combined, before: limit)
+          token Punctuation
+          push :italicbold
+          return push(:punctchar)
+        end
+
+        if after && !after.match?(refused_after) && !(before && before.match?(refused_before)) &&
+           closer_ahead?(name, BARE_CLOSERS[name], pos, masked: :bare, before: limit)
+          token Punctuation
+          return push(name)
+        end
+
+        token plain_token
+      end
+
+      BARE_CLOSERS = BARE.to_h do |char, (name, _, _, _)|
+        [name, %r/(?<=\S)#{Regexp.escape(char)}(?![\p{L}\p{N}])/]
+      end.freeze
+      COMBINED_CLOSERS = {
+        bolditalic: %r/(?<=\S)\*\//, italicbold: %r/(?<![\s*])\/\*(?!\w)/
+      }.freeze
+
+      # Where the run the scanner is in closes, if it is in one: a run opened
+      # inside it must close first, so `*a /b* c/` is strong `a /b`.
+      def enclosing_closer(pos)
+        name = state.name
+        if BARE_CLOSERS.key?(name)
+          closer_at(name, BARE_CLOSERS[name], pos, masked: :bare)
+        elsif COMBINED_CLOSERS.key?(name)
+          closer_at(name, COMBINED_CLOSERS[name], pos, masked: :combined)
+        end
+      end
+
+      # A closer candidate inside a bare run, just consumed: it closes unless
+      # the character before it is one +refused_before+ matches.
+      def close_bare(refused_before)
+        text = @current_stream.string
+        at = @current_stream.charpos - @current_stream[0].length
+        if at.positive? && !text[at - 1].match?(refused_before)
+          token Punctuation
+          pop!
+        else
+          token plain_token
+        end
+      end
+
+      # Emit the `{` of a braced opener and enter +name+, whose remaining
+      # opener characters come next.
+      def enter_braced(name, rest = 1)
+        token Punctuation
+        push name
+        rest.times { push :punctchar }
+      end
+
+      # Emit the next +text+ of the input as one +tok+ token.
+      def consume_as(tok, text)
+        literal = Regexp.new(Regexp.escape(text))
+        push do
+          rule(literal) { token tok; pop! }
+        end
+      end
+
+      # Whether +pattern+ matches on the current line at or after +from+.
+      # The matches of one line are found once per pattern and reused, which
+      # keeps a line full of unclosed openers linear instead of quadratic.
+      # A masked line hides what a bare closer cannot reach into: escapes,
+      # code spans, braced inlines, link destinations and autolinks. A bare
+      # run's line also ends at a trailing comment, which takes its closer.
+      def closer_ahead?(key, pattern, from = @current_stream.charpos, masked: nil, before: nil)
+        at = closer_at(key, pattern, from, masked: masked)
+        !at.nil? && (before.nil? || at < before)
+      end
+
+      # The position of the first match of +pattern+ on the current line at or
+      # after +from+, or nil.
+      def closer_at(key, pattern, from = @current_stream.charpos, masked: nil)
+        stream = @current_stream
+        unless @closers_for.equal?(stream)
+          @closers_for = stream
+          @closers = {}
+        end
+        pos = stream.charpos
+        entry = @closers[key]
+        unless entry && pos >= entry[0] && pos <= entry[1]
+          text = stream.string
+          first = pos.zero? ? 0 : (text.rindex("\n", pos - 1) || -1) + 1
+          last = text.index("\n", pos) || text.length
+          line = text[first...last]
+          line = mask_opaque(line) if masked
+          line = line[0...line.index('%%')] if masked == :bare && line.include?('%%')
+          hits = []
+          line.scan(pattern) { hits << first + Regexp.last_match.begin(0) }
+          entry = @closers[key] = [first, last, hits]
+        end
+        entry[2].bsearch { |at| at >= from }
+      end
+
+      OPAQUE = /\\.|(?<!`)(`+)(?!`).*?(?<!`)\1(?!`)|\]\([^)\n]*\)|<[^<>\s]+>/.freeze
+
+      def mask_opaque(line)
+        line = line.gsub(OPAQUE) { |span| "\u0001" * span.length }
+        # Braced inlines pair up in code: a regex scanning from every unclosed
+        # `{*` to the end of the line is quadratic. Once a kind has no closer
+        # left, no later opener of that kind can have one either.
+        '*/_^,=+~#%-'.each_char do |kind|
+          at = 0
+          while (open = line.index("{#{kind}", at))
+            close = line.index("#{kind}}", open + 3)
+            break unless close
+
+            line[open...close + 2] = "\u0001" * (close + 2 - open)
+            at = close + 2
+          end
+        end
+        line
+      end
+
+      # The scope of the block a bracketed body opens in, when a `]` outside
+      # escapes and code spans follows before that block ends; nil otherwise.
+      # The brackets of one block are found in one pass and reused, so a run
+      # of unclosed openers does not rescan it once each.
+      def bracket_ahead?
+        stream = @current_stream
+        pos = stream.charpos
+        scope = body_scope
+        @brackets = {} unless @brackets_for.equal?(stream)
+        @brackets_for = stream
+        entry = @brackets[scope]
+        unless entry && pos < entry[0]
+          text = stream.string
+          last = body_end(text, pos, scope)
+          hits = []
+          text[pos...last].scan(BRACKET_SCAN) { hits << pos + $~.begin(0) if $~[0] == ']' }
+          entry = @brackets[scope] = [last, hits]
+        end
+        entry[1].bsearch { |at| at >= pos } ? scope : nil
+      end
+
+      # Where a body opened at +pos+ must end: the first block end, or the
+      # first closed fence, after it.
+      def body_end(text, pos, scope)
+        ends = BODY_ENDS[scope]
+        last = text.index(ends, pos) || text.length
+        return last if scope == :line
+
+        at = pos
+        while (fence = text.index(FENCE_NEXT, at)) && fence < last
+          return fence if fence_closed_below?($~[1], fence + 1)
+
+          at = fence + 1
+        end
+        last
+      end
+
+      BRACKET_SCAN = /\\.|(?<!`)(`+)(?!`).*?(?<!`)\1(?!`)|\]/.freeze
+
+      # The scope of the nearest enclosing block state. An indented paragraph
+      # sits in a container, so the next list marker ends it as it ends an item.
+      def body_scope
+        stack.reverse_each do |s|
+          scope = BODY_SCOPE[s.name]
+          next unless scope
+          return :item if scope == :para && @indented
+
+          return scope
+        end
+        :para
+      end
+
+      # Whether +closer+ matches a line below the current position. A miss
+      # is remembered per +key+, because every later opener of the same kind
+      # would miss too - without that, a run of unclosed openers rescans the
+      # rest of the document once each.
+      def closer_below?(closer, key, from = @current_stream.charpos)
+        stream = @current_stream
+        @closer_misses = {} unless @closer_misses_for.equal?(stream)
+        @closer_misses_for = stream
+        return false if @closer_misses[key]
+
+        found = stream.string.index(closer, from)
+        @closer_misses[key] = true unless found
+        !found.nil?
+      end
+
+      # Enter the state of a block whose opener just matched, or none for an
+      # opener that is a single token. After a blockquote marker the opener
+      # replaces the quote's start state, and a single-token opener hands the
+      # rest of the line to quoted text.
+      def open_block(name = nil, quoted = name || :quoteline)
+        if state?(:quotestart)
+          goto quoted
+        elsif name
+          push name
+        end
+      end
+
+      def continue_paragraph(name)
+        @lone_image = false
+        goto name
+      end
+
+      # Whether a fence opened by +run+ on the current line is closed further
+      # down. Only a closed fence interrupts a paragraph.
+      def fence_closed_below?(run, line_start = @current_stream.charpos)
+        text = @current_stream.string
+        line_end = text.index("\n", line_start) || text.length
+        closer = %r/^[ \t]*#{Regexp.escape(run[0])}{#{run.length},}[ \t]*$/
+        closer_below?(closer, run, line_end)
+      end
+
       # Lex the tail of a fence or container opener line without leaving the
       # current state: a quoted title, a bracketed label and an attribute
       # block, each as its own token.
@@ -590,6 +1140,8 @@ module Rouge
         sublexer = fence_lexer(language)
         sublexer&.reset!
 
+        return push_quoted_fence(char, width, sublexer) if state?(:quotestart)
+
         push do
           rule closer, Punctuation, :pop!
           if sublexer
@@ -611,6 +1163,35 @@ module Rouge
         end
       end
 
+      # A fence opened after a blockquote marker. Every payload line carries
+      # the quote prefix at the opener's depth, which is markup rather than
+      # payload; a line with fewer markers leaves that quote and so ends the
+      # fence too, while a deeper marker is payload.
+      def push_quoted_fence(char, width, sublexer)
+        pop!
+        text = @current_stream.string
+        pos = @current_stream.charpos
+        line = text[((text.rindex("\n", pos - 1) || -1) + 1)...pos]
+        depth = line[/\A#{QUOTE_PREFIX}/].to_s.count('>')
+        prefix = %r/(?:[ \t]*>(?=[ \t]|$)){#{depth}}[ \t]?/
+
+        push do
+          rule %r/^(#{prefix})([ \t]*#{char}{#{width},})([ \t]*)$/ do
+            groups Punctuation, Punctuation, Text
+            pop!
+          end
+          rule(%r/^(?!#{prefix})/) { pop! }
+          rule %r/^#{prefix}/, Punctuation
+          if sublexer
+            rule %r/\n/, Text
+            rule(%r/[^\n]+\n?/) { |m| delegate sublexer, m[0] }
+          else
+            rule %r/<\d+>/, Name::Constant
+            rule %r/[^\n<]+|<|\n/, Str::Backtick
+          end
+        end
+      end
+
       # A lexer for the fence's declared language, or nil when there is no
       # language, none is known, or the name is ambiguous. Guessing is
       # deliberately not attempted: a wrong guess colours a sample as the wrong

data/lib/rouge-carve/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+# Kept apart from the entry file so the gemspec can read it without loading
+# Rouge, which is not installed yet when Bundler evaluates the gemspec.
+module RougeCarve
+  VERSION = '0.1.3'
+end

data/lib/rouge-carve.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'rouge'
+require_relative 'rouge-carve/version'
 require_relative 'rouge/lexers/carve'
 
 # A Rouge lexer for the Carve markup language.
@@ -9,5 +10,4 @@ require_relative 'rouge/lexers/carve'
 # registers it, so `Rouge::Lexer.find('carve')` and
 # `Rouge::Lexer.guess(filename: 'x.crv')` work from that point on.
 module RougeCarve
-  VERSION = '0.1.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rouge-carve
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - Mark Scherer
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-22 00:00:00.000000000 Z
+date: 2026-10-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rouge
@@ -16,7 +16,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.30'
     - - "<"
       - !ruby/object:Gem::Version
         version: '6.0'
@@ -26,7 +26,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.30'
     - - "<"
       - !ruby/object:Gem::Version
         version: '6.0'
@@ -44,6 +44,7 @@ files:
 - LICENSE
 - README.md
 - lib/rouge-carve.rb
+- lib/rouge-carve/version.rb
 - lib/rouge/demos/carve
 - lib/rouge/lexers/carve.rb
 homepage: https://github.com/markup-carve/rouge-carve