prosemirror 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1006b6a2ecb95f5a4d603d5d9321bbdab419f26f882365ff14d1692f27970ce7
-  data.tar.gz: 22734518a2e81ecaba0ff64ed5057f40705dfb55dcd919567673f88e7cbf7e45
+  metadata.gz: 9a11c83675790681267f7d5282882f967154892b408d8965e528b1c34d166876
+  data.tar.gz: 14ae8422a61cbe0be055d6ef7590a10981014430978089e14b2805ebf3100be2
 SHA512:
-  metadata.gz: 0f4b24b76e78e49523f068aa8520304c955428d4ff295c0e0f918dfad07ad0bc5b7ee391b8dacaeabf258f871d99f2dcfba15e4ebad4bec961e97529cdd24f34
-  data.tar.gz: be218156ef3b417342de7d393d219b5019fe6f270a9c865e5cce9b5f9a2436008f59d5ada72d0111e09152586a06e21a7e98653ce4002923049eb0e70331daa0
+  metadata.gz: 02ac22023a65ba9631752cf6c42b9895cfb4be99acc4d2b1922d0a9742e5d61c7d084455b0c6aec2f0c80400531870113b2ceba519f0a7a4af701de7e192c1f0
+  data.tar.gz: 8adae01f43c5ea57367db4d77605ceb28b4d606829b905440a716c64dc483c82a105898878349bb79f1fd73df4459651e9172d67546ee1d23968fd8cd3538801

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 ## [Unreleased]
 
+## [0.3.0] - 2026-10-24
+
+- Import Markdown with `Prosemirror.from_markdown` (or `schema.parse_markdown`). Needs the [`commonmarker`](https://github.com/gjtorikian/commonmarker) gem, which isn't installed with this gem. GitHub Flavored Markdown tables, strikethrough and autolinks are supported, and the schema's parse rules apply, so custom types can match Markdown too. Raw HTML goes through HTML import when the herb gem is installed. YAML front matter is dropped.
+- Import plain text with `Prosemirror.from_text` (or `schema.parse_text`): blank lines separate paragraphs, and other line breaks become hard breaks.
+- HTML, Markdown and text import accept nil and binary strings, and replace invalid UTF-8 with U+FFFD, as browsers do. HTML import raised on invalid UTF-8.
+
 ## [0.2.0] - 2026-10-23
 
 - Import HTML with `Prosemirror.from_html` (or `schema.parse_html`), as ProseMirror's `DOMParser` does. Needs the [`herb`](https://herb-tools.dev) gem, which isn't installed with this gem.

data/README.md

@@ -1,13 +1,13 @@
 # ProseMirror
 
-Render [ProseMirror](https://prosemirror.net) and [Tiptap](https://tiptap.dev) documents to HTML and text, and import HTML into them, in Ruby.
+Render [ProseMirror](https://prosemirror.net) and [Tiptap](https://tiptap.dev) documents to HTML and text, and import HTML, Markdown and text into them, in Ruby.
 
 - HTML is built with [Phlex](https://www.phlex.fun) and escaped as it is written.
 - Attributes are typed with [Literal](https://literal.fun).
 - Link and image URLs must use an allowed scheme.
 - No Rails dependency.
 
-This gem replaces [prosemirror_to_html](https://github.com/etaminstudio/prosemirror_to_html) and [html_to_prosemirror](https://github.com/etaminstudio/html_to_prosemirror). Markdown import is planned: see the [roadmap](ROADMAP.md).
+This gem replaces [prosemirror_to_html](https://github.com/etaminstudio/prosemirror_to_html) and [html_to_prosemirror](https://github.com/etaminstudio/html_to_prosemirror). See the [roadmap](ROADMAP.md) for what comes next.
 
 ## Installation
 
@@ -70,6 +70,35 @@ It works like ProseMirror's `DOMParser`, and for the built-in types gives the sa
 - entities are decoded, and ERB tags (`<%= … %>`) are kept as text;
 - invalid attributes are fixed and reported, as in [Invalid documents](#invalid-documents).
 
+## Markdown import
+
+Markdown import uses [commonmarker](https://github.com/gjtorikian/commonmarker)'s parser, written in Rust. Add it to your Gemfile:
+
+```sh
+bundle add commonmarker
+```
+
+```ruby
+doc = Prosemirror.from_markdown("## Hello\n\nSome **bold** text.")
+```
+
+It reads GitHub Flavored Markdown: tables, strikethrough and autolinks. Each Markdown node becomes the element commonmarker renders for it, and the schema's parse rules apply as in HTML import, so custom types can match Markdown too: a type that matches `img` also gets Markdown images. For the built-in types, it gives the same JSON as rendering the Markdown to HTML and importing it in Tiptap, except that:
+
+- code blocks lose their final newline, as in prosemirror-markdown;
+- task list items (`- [ ] todo`) stay as list items starting with `[ ] ` or `[x] `.
+
+Raw HTML in Markdown goes through HTML import when the herb gem is installed, and stays as text otherwise.
+
+YAML front matter, between `---` lines at the start, is dropped. Footnotes aren't supported yet: `[^1]: Note` reads as a link definition, as in CommonMark. Markdown nested more than 1,000 levels deep raises `Prosemirror::Error`.
+
+## Text import
+
+```ruby
+doc = Prosemirror.from_text("First line\nsecond line\n\nNew paragraph")
+```
+
+Blank lines separate paragraphs, and other line breaks become hard breaks. Spaces are kept.
+
 ## Schemas
 
 Two schemas are included:
@@ -163,7 +192,15 @@ Prosemirror::Schema.tiptap(
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests, and `rake` to also run [Standard](https://github.com/standardrb/standard). You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+To release a new version, update the version number in `version.rb` and the changelog, and merge them into main. Then tag the version and push the tag:
+
+```sh
+git tag v0.3.0 && git push origin v0.3.0
+```
+
+The [Push gem](.github/workflows/push_gem.yml) workflow publishes the gem to [rubygems.org](https://rubygems.org) with [trusted publishing](https://guides.rubygems.org/trusted-publishing/), after approval in the `release` environment.
 
 ## Contributing
 

data/ROADMAP.md

@@ -169,11 +169,11 @@ They differ in more than names:
 - Old HTML mapped to the schema: `<b>` and `<i>` to `bold` and `italic`, `<strike>` to `strike`, `<tt>` to `code`. `<font>` and `<center>` are unwrapped.
 - Tests: HTML from Tiptap gives the same JSON as Tiptap.
 
-### 0.3: Markdown and text import
+### 0.3: Markdown and text import (released)
 
-- `Prosemirror.from_markdown`, built from [`commonmarker`](https://github.com/gjtorikian/commonmarker)'s syntax tree, without going through HTML (like `prosemirror-markdown`). commonmarker's parser is written in Rust.
-- GitHub Flavored Markdown: tables, strikethrough, autolinks, task lists.
-- Custom node types can define their Markdown mapping.
+- `Prosemirror.from_markdown`, built from [`commonmarker`](https://github.com/gjtorikian/commonmarker)'s syntax tree. commonmarker's parser is written in Rust. Each Markdown node becomes the element commonmarker renders for it, without going through an HTML string, and HTML import's parse rules build the document.
+- GitHub Flavored Markdown: tables, strikethrough, autolinks. Task list items stay as text until task list nodes (0.6).
+- Custom node types match Markdown with their parse rules.
 - Raw HTML in Markdown goes through the HTML importer if Herb is installed.
 - `Prosemirror.from_text`: blank lines make paragraphs, line breaks make hard breaks.
 

data/TYPES.md

@@ -26,6 +26,7 @@ A type is a [Phlex](https://www.phlex.fun) component, and its attributes are [Li
 - [Attributes](#attributes)
 - [Rendering](#rendering)
 - [Parse rules](#parse-rules)
+- [Markdown](#markdown)
 - [Subclassing built-in types](#subclassing-built-in-types)
 
 ## Names
@@ -186,6 +187,22 @@ The element passed to rules is a `Prosemirror::HTML::Element`, with lowercase na
 | `matches?("p.lead")` | Whether the element matches a selector. |
 | `parent` | The parent element. |
 
+## Markdown
+
+Markdown import builds the elements commonmarker renders for each Markdown node, and applies the same parse rules. To match Markdown, a type matches these elements:
+
+| Markdown | Element |
+|---|---|
+| Paragraph, heading, blockquote, thematic break | `p`, `h1`…`h6`, `blockquote`, `hr` |
+| Bullet list, ordered list, list item | `ul`, `ol` (with `start` when it isn't 1), `li` |
+| Code block | `pre` holding a `code` with a `language-…` class from the info string |
+| Table | `table`, `tr`, `th` for the first row, `td`, with a `text-align` style for aligned columns |
+| Emphasis, strong, strikethrough, inline code | `em`, `strong`, `s`, `code` |
+| Link, image | `a` with `href` and `title`, `img` with `src`, `alt` and `title` |
+| Hard line break | `br` |
+
+Raw HTML in Markdown goes through HTML import when the herb gem is installed, and stays as text otherwise.
+
 ## Subclassing built-in types
 
 A subclass keeps the name, options, attributes and parse rules of its parent, and can change any of them. Register it with `Schema#with`, which replaces the built-in type with the same name:

data/lib/prosemirror/html/builder.rb

@@ -22,11 +22,19 @@ module Prosemirror
         raise Error, 'HTML import needs the herb gem: add `gem "herb"` to your Gemfile'
       end
 
+      # Whether the herb gem is installed.
+      def self.herb?
+        load_herb
+        true
+      rescue Error
+        false
+      end
+
       # A root Element holding the parsed content.
       def call(html)
         self.class.load_herb
         root = Element.new("#document")
-        Splitter.new(PIECE_SIZE).call(html.to_s).each do |piece|
+        Splitter.new(PIECE_SIZE).call(UTF8.string(html)).each do |piece|
           # Without `timeout: 0`, Herb stops after a second and returns what it parsed so far.
           build(Herb.parse(piece, track_locations: false, strict: false, timeout: 0).value.children, root)
         end

data/lib/prosemirror/html_parser.rb

@@ -41,8 +41,13 @@ module Prosemirror
 
     # The document parsed from the HTML, as its top node.
     def parse(html)
+      parse_element(HTML::Builder.new.call(html))
+    end
+
+    # The document for an Element's content, like the one HTML::Builder gives.
+    def parse_element(root)
       context = Context.new(self)
-      context.add_all(HTML::Builder.new.call(html), [])
+      context.add_all(root, [])
       context.finish
     end
 

data/lib/prosemirror/markdown/builder.rb

@@ -0,0 +1,200 @@
+# frozen_string_literal: true
+
+module Prosemirror
+  module Markdown
+    # Builds the Elements and Texts of HTML import from Markdown, with commonmarker's syntax tree
+    # (https://github.com/gjtorikian/commonmarker), so that the schema's parse rules apply to Markdown too.
+    # Each Markdown node becomes the element commonmarker renders for it, without going through HTML.
+    #
+    # - GitHub Flavored Markdown: tables, strikethrough and autolinks. Task list items stay as text (`[ ] `).
+    # - Raw HTML can span several Markdown blocks, like `<del>` around a paragraph: with the herb gem, a
+    #   document holding raw HTML is rendered to HTML by commonmarker and imported. Without it, raw HTML
+    #   stays as text.
+    # - Code blocks lose their final newline, as in prosemirror-markdown.
+    # - YAML front matter, between `---` lines at the start, is dropped.
+    class Builder
+      MAX_DEPTH = HTML::Builder::MAX_DEPTH
+      EXTENSIONS = {
+        strikethrough: true, table: true, autolink: true, tasklist: false, shortcodes: false, header_ids: nil,
+        front_matter_delimiter: "---"
+      }.freeze
+      RENDER_OPTIONS = {extension: EXTENSIONS, render: {unsafe: true, hardbreaks: false, github_pre_lang: false}}.freeze
+      TAGS = {
+        paragraph: "p", block_quote: "blockquote", item: "li", thematic_break: "hr", table_row: "tr",
+        emph: "em", strong: "strong", strikethrough: "s", linebreak: "br"
+      }.freeze
+      # A table's delimiter row, like `| :-- | --: |`, at the end of its line.
+      DELIMITER_ROW = /\|?(?:\s*:?-+:?\s*\|)*\s*:?-+:?\s*\|?\s*\z/
+      # The bytes commonmarker escapes in URLs, as comrak's `escape_href`: all but these, and `%` when it
+      # doesn't start an escape. `&` and `'` are kept: comrak writes them as HTML references.
+      URL_UNSAFE = %r{%(?!\h\h)|[^-_.+!*(),#@?=;:/$~A-Za-z0-9&'%]}n
+      # An IPv6 host at the start of a URL, like `http://[::1]`, kept as it is.
+      IPV6_HOST = %r{\A[A-Za-z][A-Za-z0-9+.-]*://\[[\h:.]+\]}
+
+      def self.load_commonmarker
+        require "commonmarker"
+      rescue LoadError
+        raise Error, 'Markdown import needs the commonmarker gem: add `gem "commonmarker"` to your Gemfile'
+      end
+
+      # A root Element holding the content.
+      def call(markdown)
+        self.class.load_commonmarker
+        markdown = UTF8.string(markdown)
+        document = parse(markdown)
+        return from_html(document) if raw_html?(document) && HTML::Builder.herb?
+
+        @lines = markdown.lines
+        root = HTML::Element.new("#document")
+        build(document, root)
+        root
+      end
+
+      private
+
+      def parse(markdown)
+        Commonmarker.parse(markdown, options: {extension: EXTENSIONS})
+      rescue Commonmarker::NestingDepthExceededError => error
+        raise Error, "Markdown nested too deeply: #{error.message}"
+      end
+
+      # Walks the tree without recursion. Each item of the stack is a node, the element to add it to, and
+      # the depth of that element. Elements nested deeper than MAX_DEPTH are flattened.
+      def build(document, root)
+        stack = document.to_a.reverse.map { |node| [node, root, 0] }
+        until stack.empty?
+          node, target, depth = stack.pop
+          element = add(node, target)
+          next unless element
+
+          if depth >= MAX_DEPTH
+            element = target
+          else
+            target.append(element)
+          end
+          node.to_a.reverse_each { |child| stack << [child, element, element.equal?(target) ? depth : depth + 1] }
+        end
+      end
+
+      # Adds a node to the target. Returns the element to add its children to, when they need adding.
+      def add(node, target)
+        case node.type
+        when *TAGS.keys then HTML::Element.new(TAGS[node.type])
+        when :heading then HTML::Element.new("h#{node.header_level}")
+        when :list then list(node)
+        when :code_block then code_block(node, target)
+        when :table then table(node)
+        when :table_cell then table_cell(node)
+        when :link then HTML::Element.new("a", {"href" => escape_url(node.url), "title" => presence(node.title)}.compact)
+        when :image then image(node, target)
+        when :code then text(target, node.string_content, "code")
+        when :text then text(target, node.string_content)
+        when :softbreak then text(target, "\n")
+        when :html_block then text(target.append(HTML::Element.new("p")), node.literal.chomp)
+        when :html_inline then text(target, node.literal)
+        when :frontmatter then nil
+        else HTML::Element.new("span") # Unwrapped by HTML import, like escaped characters (`\*`).
+        end
+      end
+
+      def list(node)
+        return HTML::Element.new("ul") unless node.list_type == :ordered
+
+        HTML::Element.new("ol", (node.list_start == 1) ? {} : {"start" => node.list_start.to_s})
+      end
+
+      def code_block(node, target)
+        language = node.fence_info.to_s.split.first
+        code = HTML::Element.new("code", language ? {"class" => "language-#{language}"} : {})
+        code.append(HTML::Text.new(node.string_content.chomp))
+        target.append(HTML::Element.new("pre")).append(code)
+        nil
+      end
+
+      def table(node)
+        @alignments = alignments(node)
+        @row = 0
+        HTML::Element.new("table")
+      end
+
+      # The first row holds header cells.
+      def table_cell(node)
+        index = cell_index(node)
+        @row += 1 if index.zero?
+        align = @alignments[index]
+        HTML::Element.new((@row == 1) ? "th" : "td", align ? {"style" => "text-align: #{align}"} : {})
+      end
+
+      def cell_index(node)
+        index = 0
+        index += 1 while (node = node.previous_sibling)
+        index
+      end
+
+      # commonmarker doesn't give the alignments of a table's columns: they are read from its delimiter row,
+      # the line after its first line.
+      def alignments(node)
+        row = @lines[node.source_position[:start_line]].to_s.chomp[DELIMITER_ROW].to_s
+        row.strip.delete_prefix("|").delete_suffix("|").split("|").map do |cell|
+          cell = cell.strip
+          if cell.start_with?(":") && cell.end_with?(":") then "center"
+          elsif cell.end_with?(":") then "right"
+          elsif cell.start_with?(":") then "left"
+          end
+        end
+      end
+
+      def image(node, target)
+        attributes = {"src" => escape_url(node.url), "alt" => plain_text(node), "title" => presence(node.title)}.compact
+        target.append(HTML::Element.new("img", attributes))
+        nil
+      end
+
+      def text(target, value, tag = nil)
+        target = target.append(HTML::Element.new(tag)) if tag
+        target.append(HTML::Text.new(value))
+        nil
+      end
+
+      def raw_html?(document)
+        document.walk { |node| return true if node.type == :html_block || node.type == :html_inline }
+        false
+      end
+
+      def from_html(document)
+        root = HTML::Builder.new.call(document.to_html(options: RENDER_OPTIONS, plugins: {syntax_highlighter: nil}))
+        chomp_code_blocks(root)
+        root
+      end
+
+      # The code blocks commonmarker renders, `<pre><code>`, not the `<pre>` elements of raw HTML.
+      def chomp_code_blocks(element)
+        element.children.each do |child|
+          next unless child.element?
+          next chomp_code_blocks(child) unless child.name == "pre"
+
+          code = child.children.first
+          next unless code&.element? && code.name == "code" && child.children.size == 1
+
+          last = code.children.last
+          code.children[-1] = HTML::Text.new(last.value.chomp).tap { |text| text.parent = code } if last.is_a?(HTML::Text)
+        end
+      end
+
+      # The URL as commonmarker renders it, with the characters URLs can't hold percent-encoded.
+      def escape_url(url)
+        host = url[IPV6_HOST].to_s
+        rest = url.delete_prefix(host).b.gsub(URL_UNSAFE) { |byte| byte.ord.zero? ? "%EF%BF%BD" : format("%%%02X", byte.ord) }
+        host + rest.force_encoding(Encoding::UTF_8)
+      end
+
+      def plain_text(node)
+        node.to_a.map { |child| %i[text code].include?(child.type) ? child.string_content : plain_text(child) }.join
+      end
+
+      def presence(value)
+        value unless value.nil? || value.empty?
+      end
+    end
+  end
+end

data/lib/prosemirror/schema.rb

@@ -117,6 +117,27 @@ module Prosemirror
       Document.new(HTMLParser.new(self).parse(html), schema: self)
     end
 
+    # Build a document from Markdown, with the parse rules of the schema's types. Needs the `commonmarker` gem.
+    def parse_markdown(markdown)
+      Document.new(HTMLParser.new(self).parse_element(Markdown::Builder.new.call(markdown)), schema: self)
+    end
+
+    # Build a document from plain text: blank lines separate paragraphs, and other line breaks become the
+    # schema's line break node, like `hard_break`, or spaces without one. Spaces are kept.
+    def parse_text(text)
+      paragraph = node_type(:paragraph) || @nodes.find { |type| textblock?(type) }
+      linebreak = @nodes.find(&:linebreak_replacement?)
+      separator = linebreak ? {"type" => linebreak.type_name.to_s} : {"type" => "text", "text" => " "}
+      blocks = UTF8.string(text).gsub(/\r\n?/, "\n").split(/\n[ \t]*\n/).map { |block| block.gsub(/\A\n+|\n+\z/, "") }
+      content = blocks.reject { |block| block.strip.empty? }.map do |block|
+        inline = block.split("\n").each_with_index.flat_map do |line, index|
+          [(separator if index.positive?), ({"type" => "text", "text" => line} unless line.empty?)].compact
+        end
+        {"type" => paragraph.type_name.to_s, "content" => inline}
+      end
+      parse({"type" => @top_node.to_s, "content" => content.empty? ? [{"type" => paragraph.type_name.to_s}] : content})
+    end
+
     # The type name as written in JSON.
     def json_type_name(name)
       (@case == :camel) ? Naming.camel(name) : name.to_s

data/lib/prosemirror/utf8.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Prosemirror
+  # Imported HTML, Markdown and text, as valid UTF-8.
+  module UTF8
+    module_function
+
+    # The value as a UTF-8 String: nil becomes empty, other encodings are read as UTF-8 (a binary string
+    # from a file or an HTTP body), and invalid bytes become U+FFFD, as browsers do.
+    def string(value)
+      string = value.to_s
+      string = string.dup.force_encoding(Encoding::UTF_8) unless string.encoding == Encoding::UTF_8
+      string.valid_encoding? ? string : string.scrub
+    end
+  end
+end

data/lib/prosemirror/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Prosemirror
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/lib/prosemirror.rb

@@ -10,7 +10,8 @@ require "zeitwerk"
 
 loader = Zeitwerk::Loader.for_gem
 loader.inflector.inflect(
-  "html" => "HTML", "html_parser" => "HTMLParser", "html_renderer" => "HTMLRenderer", "json_parser" => "JSONParser"
+  "html" => "HTML", "html_parser" => "HTMLParser", "html_renderer" => "HTMLRenderer", "json_parser" => "JSONParser",
+  "utf8" => "UTF8"
 )
 loader.setup
 
@@ -45,5 +46,15 @@ module Prosemirror
     def from_html(html, schema: self.schema)
       schema.parse_html(html)
     end
+
+    # Build a document from Markdown, with the schema's parse rules. Needs the `commonmarker` gem.
+    def from_markdown(markdown, schema: self.schema)
+      schema.parse_markdown(markdown)
+    end
+
+    # Build a document from plain text: blank lines separate paragraphs, other line breaks become hard breaks.
+    def from_text(text, schema: self.schema)
+      schema.parse_text(text)
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: prosemirror
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Hans Lemuet
@@ -82,6 +82,7 @@ files:
 - lib/prosemirror/json_parser.rb
 - lib/prosemirror/mark.rb
 - lib/prosemirror/mark_type.rb
+- lib/prosemirror/markdown/builder.rb
 - lib/prosemirror/marks/basic/code.rb
 - lib/prosemirror/marks/basic/em.rb
 - lib/prosemirror/marks/basic/link.rb
@@ -122,6 +123,7 @@ files:
 - lib/prosemirror/schema.rb
 - lib/prosemirror/text_serializer.rb
 - lib/prosemirror/type_definition.rb
+- lib/prosemirror/utf8.rb
 - lib/prosemirror/version.rb
 - sig/prosemirror.rbs
 homepage: https://github.com/etaminstudio/prosemirror