commonmarker 0.23.7.pre1 → 1.0.0.pre.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f41f7e434edcc989547b4cc9d02007ae0eed60c393bd6f7ee8a288f139c6b537
-  data.tar.gz: 1c92f108fb98a076cf403c9e643f8a10b96d8c7ef359293686a27173e1e70436
+  metadata.gz: 50518c47228bb2c8fa7316a72874bd80441a7461e6763e7c267138a958b8f5bd
+  data.tar.gz: 20df9a9139bef07fbd0748c5b0ceeee9217c61ba521dca4aca9e7e71a5ba9ed7
 SHA512:
-  metadata.gz: ee9ef9c3ff7bcee7d054e1961ce402ddc067a082ea969752346b9642510473547f1fa8aea5e534b5113b7035ea8ddacdbb67c50e2490585bc3d1197d888ea4ac
-  data.tar.gz: 74b8df6b443d787cb9ce7110e7cabc34d5a0233a865b8cb67839b9bf1b11b636c02cd107a119e4fc02ca1122d3e736b84fc3fba0a674f5bd17928e25e3c4e8cf
+  metadata.gz: 1c64b798bbfb45a53c70e2a07f7a5bd279ff49ebb7518d62d404af71c3d46a1a6f929f33f03b0bf4819bb28a5d7df83962626ca5cef431e07ba4a8d8dea08ef0
+  data.tar.gz: 5d5dbae1cb19fcafbdf9f83ae263ea60bcb0ad59adc2802b52c80af676b755900f875a6cf217f81e9124e519ae45d7b07f4e07e7dcba2cfdd5d64c3957fac846

data/README.md

@@ -1,9 +1,11 @@
-# CommonMarker
+# Commonmarker
 
-![Build Status](https://github.com/gjtorikian/commonmarker/workflows/CI/badge.svg) [![Gem Version](https://badge.fury.io/rb/commonmarker.svg)](http://badge.fury.io/rb/commonmarker)
+> **Note**
+> This README refers to the behavior in the new 1.0.0.pre gem.
 
-Ruby wrapper for [libcmark-gfm](https://github.com/github/cmark),
-GitHub's fork of the reference parser for CommonMark. It passes all of the C tests, and is therefore spec-complete. It also includes extensions to the CommonMark spec as documented in the [GitHub Flavored Markdown spec](http://github.github.com/gfm/), such as support for tables, strikethroughs, and autolinking.
+Ruby wrapper for Rust's [comrak](https://github.com/kivikakk/comrak) crate.
+
+It passes all of the CommonMark test suite, and is therefore spec-complete. It also includes extensions to the CommonMark spec as documented in the [GitHub Flavored Markdown spec](http://github.github.com/gfm/), such as support for tables, strikethroughs, and autolinking.
 
 For more information on available extensions, see [the documentation below](#extensions).
 
@@ -27,232 +29,83 @@ Or install it yourself as:
 
 Call `render_html` on a string to convert it to HTML:
 
-``` ruby
-require 'commonmarker'
-CommonMarker.render_html('Hi *there*', :DEFAULT)
-# <p>Hi <em>there</em></p>\n
-```
-
-The second argument is optional--[see below](#options) for more information.
-
-### Generating a document
-
-You can also parse a string to receive a `Document` node. You can then print that node to HTML, iterate over the children, and other fun node stuff. For example:
-
-``` ruby
+```ruby
 require 'commonmarker'
-
-doc = CommonMarker.render_doc('*Hello* world', :DEFAULT)
-puts(doc.to_html) # <p>Hi <em>there</em></p>\n
-
-doc.walk do |node|
-  puts node.type # [:document, :paragraph, :text, :emph, :text]
-end
+Commonmarker.to_html('"Hi *there*"', options: {
+    parse: { smart: true }
+})
+# <p>“Hi <em>there</em>”</p>\n
 ```
 
 The second argument is optional--[see below](#options) for more information.
 
-#### Example: walking the AST
+## Parse and Render Options
 
-You can use `walk` or `each` to iterate over nodes:
+Commonmarker accepts the same options that comrak does, as a hash dictionary with symbol keys:
 
-- `walk` will iterate on a node and recursively iterate on a node's children.
-- `each` will iterate on a node and its children, but no further.
-
-``` ruby
-require 'commonmarker'
-
-# parse the files specified on the command line
-doc = CommonMarker.render_doc("# The site\n\n [GitHub](https://www.github.com)")
-
-# Walk tree and print out URLs for links
-doc.walk do |node|
-  if node.type == :link
-    printf("URL = %s\n", node.url)
-  end
-end
-
-# Capitalize all regular text in headers
-doc.walk do |node|
-  if node.type == :header
-    node.each do |subnode|
-      if subnode.type == :text
-        subnode.string_content = subnode.string_content.upcase
-      end
-    end
-  end
-end
-
-# Transform links to regular text
-doc.walk do |node|
-  if node.type == :link
-    node.insert_before(node.first_child)
-    node.delete
-  end
-end
-```
-
-### Creating a custom renderer
-
-You can also derive a class from CommonMarker's `HtmlRenderer` class. This produces slower output, but is far more customizable. For example:
-
-``` ruby
-class MyHtmlRenderer < CommonMarker::HtmlRenderer
-  def initialize
-    super
-    @headerid = 1
-  end
-
-  def header(node)
-    block do
-      out("<h", node.header_level, " id=\"", @headerid, "\">",
-               :children, "</h", node.header_level, ">")
-      @headerid += 1
-    end
-  end
-end
-
-myrenderer = MyHtmlRenderer.new
-puts myrenderer.render(doc)
-
-# Print any warnings to STDERR
-renderer.warnings.each do |w|
-  STDERR.write("#{w}\n")
-end
+```ruby
+Commonmarker.to_html('"Hi *there*"', options:{
+  parse: { smart: true },
+  render: { hardbreaks: false}
+})
 ```
 
-## Options
-
-CommonMarker accepts the same options that CMark does, as symbols. Note that there is a distinction in CMark for "parse" options and "render" options, which are represented in the tables below.
+Note that there is a distinction in comrak for "parse" options and "render" options, which are represented in the tables below.
 
 ### Parse options
 
-| Name                          | Description
-| ----------------------------- | -----------
-| `:DEFAULT`                    | The default parsing system.
-| `:SOURCEPOS`                  | Include source position in nodes
-| `:UNSAFE`                     | Allow raw/custom HTML and unsafe links.
-| `:VALIDATE_UTF8`              | Replace illegal sequences with the replacement character `U+FFFD`.
-| `:SMART`                      | Use smart punctuation (curly quotes, etc.).
-| `:LIBERAL_HTML_TAG`           | Support liberal parsing of inline HTML tags.
-| `:FOOTNOTES`                  | Parse footnotes.
-| `:STRIKETHROUGH_DOUBLE_TILDE` | Parse strikethroughs by double tildes (compatibility with [redcarpet](https://github.com/vmg/redcarpet))
+| Name                  | Description                                                                          | Default |
+| --------------------- | ------------------------------------------------------------------------------------ | ------- |
+| `smart`               | Punctuation (quotes, full-stops and hyphens) are converted into 'smart' punctuation. | `false` |
+| `default_info_string` | The default info string for fenced code blocks.                                      | `""`    |
 
 ### Render options
 
-| Name                             | Description                                                     |
-| ------------------               | -----------                                                     |
-| `:DEFAULT`                       | The default rendering system.                                   |
-| `:SOURCEPOS`                     | Include source position in rendered HTML.                       |
-| `:HARDBREAKS`                    | Treat `\n` as hardbreaks (by adding `<br/>`).                   |
-| `:UNSAFE`                        | Allow raw/custom HTML and unsafe links.                         |
-| `:NOBREAKS`                      | Translate `\n` in the source to a single whitespace.            |
-| `:VALIDATE_UTF8`                 | Replace illegal sequences with the replacement character `U+FFFD`. |
-| `:SMART`                         | Use smart punctuation (curly quotes, etc.).                     |
-| `:GITHUB_PRE_LANG`               | Use GitHub-style `<pre lang>` for fenced code blocks.           |
-| `:LIBERAL_HTML_TAG`              | Support liberal parsing of inline HTML tags.                    |
-| `:FOOTNOTES`                     | Render footnotes.                                               |
-| `:STRIKETHROUGH_DOUBLE_TILDE`    | Parse strikethroughs by double tildes (compatibility with [redcarpet](https://github.com/vmg/redcarpet)) |
-| `:TABLE_PREFER_STYLE_ATTRIBUTES` | Use `style` insted of `align` for table cells.                  |
-| `:FULL_INFO_STRING`              | Include full info strings of code blocks in separate attribute. |
-
-### Passing options
-
-To apply a single option, pass it in as a symbol argument:
-
-``` ruby
-CommonMarker.render_doc("\"Hello,\" said the spider.", :SMART)
-# <p>“Hello,” said the spider.</p>\n
-```
+| Name              | Description                                                                                            | Default |
+| ----------------- | ------------------------------------------------------------------------------------------------------ | ------- |
+| `hardbreaks`      | [Soft line breaks](http://spec.commonmark.org/0.27/#soft-line-breaks) translate into hard line breaks. | `true`  |
+| `github_pre_lang` | GitHub-style `<pre lang="xyz">` is used for fenced code blocks with info tags.                         | `true`  |
+| `width`           | The wrap column when outputting CommonMark.                                                            | `80`    |
+| `unsafe_`         | Allow rendering of raw HTML and potentially dangerous links.                                           | `false` |
+| `escape`          | Escape raw HTML instead of clobbering it.                                                              | `false` |
 
-To have multiple options applied, pass in an array of symbols:
+As well, there are several extensions which you can toggle in the same manner:
 
-``` ruby
-CommonMarker.render_html("\"'Shelob' is my name.\"", [:HARDBREAKS, :SOURCEPOS])
+```ruby
+Commonmarker.to_html('"Hi *there*"', options: {
+    extensions: { footnotes: true, description_lists: true },
+    render: { hardbreaks: false}
+})
 ```
 
-For more information on these options, see [the CMark documentation](https://git.io/v7nh1).
-
-## Extensions
-
-Both `render_html` and `render_doc` take an optional third argument defining the extensions you want enabled as your CommonMark document is being processed. The documentation for these extensions are [defined in this spec](https://github.github.com/gfm/), and the rationale is provided [in this blog post](https://githubengineering.com/a-formal-spec-for-github-markdown/).
+### Extension options
 
-The available extensions are:
+| Name                     | Description                                                                                                         | Default |
+| ------------------------ | ------------------------------------------------------------------------------------------------------------------- | ------- |
+| `strikethrough`          | Enables the [strikethrough extension](https://github.github.com/gfm/#strikethrough-extension-) from the GFM spec.   | `true`  |
+| `tagfilter`              | Enables the [tagfilter extension](https://github.github.com/gfm/#disallowed-raw-html-extension-) from the GFM spec. | `true`  |
+| `table`                  | Enables the [table extension](https://github.github.com/gfm/#tables-extension-) from the GFM spec.                  | `true`  |
+| `autolink`               | Enables the [autolink extension](https://github.github.com/gfm/#autolinks-extension-) from the GFM spec.            | `true`  |
+| `tasklist`               | Enables the [task list extension](https://github.github.com/gfm/#task-list-items-extension-) from the GFM spec.     | `true`  |
+| `superscript`            | Enables the superscript Comrak extension.                                                                           | `false` |
+| `header_ids`             | Enables the header IDs Comrak extension. from the GFM spec.                                                         | `""`    |
+| `footnotes`              | Enables the footnotes extension per `cmark-gfm`.                                                                    | `false` |
+| `description_lists`      | Enables the description lists extension..                                                                           | `false` |
+| `front_matter_delimiter` | Enables the front matter extension.                                                                                 | `""`    |
 
-* `:table` - This provides support for tables.
-* `:tasklist` - This provides support for task list items.
-* `:strikethrough` - This provides support for strikethroughs.
-* `:autolink` - This provides support for automatically converting URLs to anchor tags.
-* `:tagfilter` - This escapes [several "unsafe" HTML tags](https://github.github.com/gfm/#disallowed-raw-html-extension-), causing them to not have any effect.
+For more information on these options, see [the comrak documentation](https://github.com/kivikakk/comrak#usage).
 
 ## Output formats
 
-Like CMark, CommonMarker can generate output in several formats: HTML, XML, plaintext, and commonmark are currently supported.
+Commonmarker can currently only generate output in one format: HTML.
 
 ### HTML
 
-The default output format, HTML, will be generated when calling `to_html` or using `--to=html` on the command line.
-
-```ruby
-doc = CommonMarker.render_doc('*Hello* world!', :DEFAULT)
-puts(doc.to_html)
-
-<p><em>Hello</em> world!</p>
-```
-
-### XML
-
-XML will be generated when calling `to_xml` or using `--to=xml` on the command line.
-
-```ruby
-doc = CommonMarker.render_doc('*Hello* world!', :DEFAULT)
-puts(doc.to_xml)
-
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE document SYSTEM "CommonMark.dtd">
-<document xmlns="http://commonmark.org/xml/1.0">
-  <paragraph>
-    <emph>
-      <text xml:space="preserve">Hello</text>
-    </emph>
-    <text xml:space="preserve"> world!</text>
-  </paragraph>
-</document>
-```
-
-### Plaintext
-
-Plaintext will be generated when calling `to_plaintext` or using `--to=plaintext` on the command line.
-
 ```ruby
-doc = CommonMarker.render_doc('*Hello* world!', :DEFAULT)
-puts(doc.to_plaintext)
+html = CommonMarker.to_html('*Hello* world!', :DEFAULT)
+puts(html)
 
-Hello world!
-```
-
-### Commonmark
-
-Commonmark will be generated when calling `to_commonmark` or using `--to=commonmark` on the command line.
-
-``` ruby
-text = <<-TEXT
-1. I am a numeric list.
-2. I continue the list.
-* Suddenly, an unordered list!
-* What fun!
-TEXT
-
-doc = CommonMarker.render_doc(text, :DEFAULT)
-puts(doc.to_commonmark)
-
-1.  I am a numeric list.
-2.  I continue the list.
-
-<!-- end list -->
-
-  - Suddenly, an unordered list\!
-  - What fun\!
+# <p><em>Hello</em> world!</p>
 ```
 
 ## Developing locally
@@ -264,7 +117,7 @@ script/bootstrap
 bundle exec rake compile
 ```
 
-If there were no errors, you're done! Otherwise, make sure to follow the CMark dependency instructions.
+If there were no errors, you're done! Otherwise, make sure to follow the comrak dependency instructions.
 
 ## Benchmarks
 
@@ -273,16 +126,21 @@ Some rough benchmarks:
 ```
 $ bundle exec rake benchmark
 
-input size = 11063727 bytes
+input size = 11064832 bytes
 
-redcarpet
-  0.070000   0.020000   0.090000 (  0.079641)
-github-markdown
-  0.070000   0.010000   0.080000 (  0.083535)
+Warming up --------------------------------------
+           redcarpet     2.000  i/100ms
 commonmarker with to_html
-  0.100000   0.010000   0.110000 (  0.111947)
-commonmarker with ruby HtmlRenderer
-  1.830000   0.030000   1.860000 (  1.866203)
-kramdown
-  4.610000   0.070000   4.680000 (  4.678398)
+                         1.000  i/100ms
+            kramdown     1.000  i/100ms
+Calculating -------------------------------------
+           redcarpet     22.317  (± 4.5%) i/s -    112.000  in   5.036374s
+commonmarker with to_html
+                          5.815  (± 0.0%) i/s -     30.000  in   5.168869s
+            kramdown      0.327  (± 0.0%) i/s -      2.000  in   6.121486s
+
+Comparison:
+           redcarpet:       22.3 i/s
+commonmarker with to_html:        5.8 i/s - 3.84x  (± 0.00) slower
+            kramdown:        0.3 i/s - 68.30x  (± 0.00) slower
 ```

data/commonmarker.gemspec

@@ -4,35 +4,38 @@ lib = File.expand_path("lib", __dir__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require "commonmarker/version"
 
-Gem::Specification.new do |s|
-  s.name = "commonmarker"
-  s.version = CommonMarker::VERSION
-  s.summary = "CommonMark parser and renderer. Written in C, wrapped in Ruby."
-  s.description = "A fast, safe, extensible parser for CommonMark. This wraps the official libcmark library."
-  s.authors = ["Garen Torikian", "Ashe Connor"]
-  s.homepage = "https://github.com/gjtorikian/commonmarker"
-  s.license = "MIT"
-
-  s.files         = ["LICENSE.txt", "README.md", "Rakefile", "commonmarker.gemspec", "bin/commonmarker"]
-  s.files        += Dir.glob("lib/**/*.rb")
-  s.files        += Dir.glob("ext/commonmarker/*.*")
-  s.extensions    = ["ext/commonmarker/extconf.rb"]
-
-  s.executables = ["commonmarker"]
-  s.require_paths = ["lib", "ext"]
-  s.required_ruby_version = [">= 2.6", "< 4.0"]
-
-  s.metadata["rubygems_mfa_required"] = "true"
-
-  s.rdoc_options += ["-x", "ext/commonmarker/cmark/.*"]
-
-  s.add_development_dependency("awesome_print")
-  s.add_development_dependency("json", "~> 2.3")
-  s.add_development_dependency("minitest", "~> 5.6")
-  s.add_development_dependency("minitest-focus", "~> 1.1")
-  s.add_development_dependency("rake")
-  s.add_development_dependency("rake-compiler", "~> 0.9")
-  s.add_development_dependency("rdoc", "~> 6.2")
-  s.add_development_dependency("rubocop")
-  s.add_development_dependency("rubocop-standard")
+Gem::Specification.new do |spec|
+  spec.name = "commonmarker"
+  spec.version = Commonmarker::VERSION
+  spec.summary = "CommonMark parser and renderer. Written in Rust, wrapped in Ruby."
+  spec.description = "A fast, safe, extensible parser for CommonMark. This wraps the comrak Rust crate."
+  spec.authors = ["Garen Torikian", "Ashe Connor"]
+  spec.license = "MIT"
+  spec.homepage = "https://github.com/gjtorikian/commonmarker"
+
+  spec.required_ruby_version = "~> 3.1"
+  # https://github.com/rubygems/rubygems/pull/5852#issuecomment-1231118509
+  spec.required_rubygems_version = ">= 3.3.22"
+
+  spec.files = ["LICENSE.txt", "README.md", "commonmarker.gemspec"]
+  spec.files += Dir.glob("lib/**/*.rb")
+  spec.files += Dir.glob("ext/**/*.{rs,toml,lock,rb}")
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+
+  spec.require_paths = ["lib"]
+  spec.extensions = ["ext/commonmarker/Cargo.toml"]
+
+  spec.metadata = {
+    "allowed_push_host" => "https://rubygems.org",
+    "funding_uri" => "https://github.com/sponsors/gjtorikian/",
+    "source_code_uri" => "https://github.com/gjtorikian/commonmarker",
+    "rubygems_mfa_required" => "true",
+  }
+
+  spec.add_dependency("rb_sys", "~> 0.9")
+
+  spec.add_development_dependency("rake", "~> 13.0")
+  spec.add_development_dependency("rake-compiler", "~> 1.2")
+  spec.add_development_dependency("rake-compiler-dock", "~> 1.2")
 end

data/ext/commonmarker/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "commonmarker"
+version = "1.0.0"
+edition = "2021"
+
+[dependencies]
+magnus = { git = "https://github.com/gjtorikian/magnus", branch = "main" } # waiting for release with full rb-sys backend
+comrak = "0.15.0"
+
+[lib]
+name = "commonmarker"
+crate-type = ["cdylib"]

data/ext/commonmarker/_util.rb

@@ -0,0 +1,102 @@
+# frozen_string_literal: true
+
+RUBY_MAJOR, RUBY_MINOR = RUBY_VERSION.split(".").collect(&:to_i)
+
+PACKAGE_ROOT_DIR = File.expand_path(File.join(File.dirname(__FILE__), "..", ".."))
+PACKAGE_EXT_DIR = File.join(PACKAGE_ROOT_DIR, "ext", "commonmarker")
+
+OS = case os = RbConfig::CONFIG["host_os"].downcase
+when /linux/
+  # The official ruby-alpine Docker containers pre-build Ruby. As a result,
+  # Ruby doesn't know that it's on a musl-based platform. `ldd` is the
+  # a more reliable way to detect musl.
+  # See https://github.com/skylightio/skylight-ruby/issues/92
+  if ENV["SKYLIGHT_MUSL"] || %x(ldd --version 2>&1).include?("musl")
+    "linux-musl"
+  else
+    "linux"
+  end
+when /darwin/
+  "darwin"
+when /freebsd/
+  "freebsd"
+when /netbsd/
+  "netbsd"
+when /openbsd/
+  "openbsd"
+when /sunos|solaris/
+  "solaris"
+when /mingw|mswin/
+  "windows"
+else
+  os
+end
+
+# Normalize the platform CPU
+ARCH = case cpu = RbConfig::CONFIG["host_cpu"].downcase
+when /amd64|x86_64|x64/
+  "x86_64"
+when /i?86|x86|i86pc/
+  "x86"
+when /ppc|powerpc/
+  "powerpc"
+when /^aarch/
+  "aarch"
+when /^arm/
+  "arm"
+else
+  cpu
+end
+
+def windows?
+  OS == "windows"
+end
+
+def solaris?
+  OS == solaries
+end
+
+def darwin?
+  OS == "darwin"
+end
+
+def macos?
+  darwin? || OS == "macos"
+end
+
+def openbsd?
+  OS == "openbsd"
+end
+
+def aix?
+  OS == "aix"
+end
+
+def nix?
+  !(windows? || solaris? || darwin?)
+end
+
+def x86_64?
+  ARCH == "x86_64"
+end
+
+def x86?
+  ARCH == "x86"
+end
+
+def abs_path(path)
+  File.join(PACKAGE_EXT_DIR, path)
+end
+
+def find_header_or_abort(header, *paths)
+  find_header(header, *paths) || abort("#{header} was expected in `#{paths.join(", ")}`, but it is missing.")
+end
+
+def find_library_or_abort(lib, func, *paths)
+  find_library(lib, func, *paths) || abort("#{lib} was expected in `#{paths.join(", ")}`, but it is missing.")
+end
+
+def concat_flags(*args)
+  args.compact.join(" ")
+end
+

data/ext/commonmarker/extconf.rb

@@ -1,7 +1,6 @@
-# frozen_string_literal: true
+require "mkmf"
+require "rb_sys/mkmf"
 
-require 'mkmf'
+require_relative "_util"
 
-$CFLAGS << ' -std=c99'
-
-create_makefile('commonmarker/commonmarker')
+create_rust_makefile("commonmarker")