commonmarker 1.0.0.pre-x86_64-linux → 1.0.0.pre7-x86_64-linux

data/README.md

@@ -7,7 +7,7 @@ Ruby wrapper for Rust's [comrak](https://github.com/kivikakk/comrak) crate.
 
 It passes all of the CommonMark test suite, and is therefore spec-complete. It also includes extensions to the CommonMark spec as documented in the [GitHub Flavored Markdown spec](http://github.github.com/gfm/), such as support for tables, strikethroughs, and autolinking.
 
-For more information on available extensions, see [the documentation below](#extensions).
+For more information on available extensions, see [the documentation below](#extension-options).
 
 ## Installation
 
@@ -27,7 +27,7 @@ Or install it yourself as:
 
 ### Converting to HTML
 
-Call `render_html` on a string to convert it to HTML:
+Call `to_html` on a string to convert it to HTML:
 
 ```ruby
 require 'commonmarker'
@@ -39,9 +39,11 @@ Commonmarker.to_html('"Hi *there*"', options: {
 
 The second argument is optional--[see below](#options) for more information.
 
-## Parse and Render Options
+## Options and plugins
 
-Commonmarker accepts the same options that comrak does, as a hash dictionary with symbol keys:
+### Options
+
+Commonmarker accepts the same parse, render, and extensions options that comrak does, as a hash dictionary with symbol keys:
 
 ```ruby
 Commonmarker.to_html('"Hi *there*"', options:{
@@ -66,14 +68,14 @@ Note that there is a distinction in comrak for "parse" options and "render" opti
 | `hardbreaks`      | [Soft line breaks](http://spec.commonmark.org/0.27/#soft-line-breaks) translate into hard line breaks. | `true`  |
 | `github_pre_lang` | GitHub-style `<pre lang="xyz">` is used for fenced code blocks with info tags.                         | `true`  |
 | `width`           | The wrap column when outputting CommonMark.                                                            | `80`    |
-| `unsafe_`         | Allow rendering of raw HTML and potentially dangerous links.                                           | `false` |
+| `unsafe`          | Allow rendering of raw HTML and potentially dangerous links.                                           | `false` |
 | `escape`          | Escape raw HTML instead of clobbering it.                                                              | `false` |
 
 As well, there are several extensions which you can toggle in the same manner:
 
 ```ruby
 Commonmarker.to_html('"Hi *there*"', options: {
-    extensions: { footnotes: true, description_lists: true },
+    extension: { footnotes: true, description_lists: true },
     render: { hardbreaks: false}
 })
 ```
@@ -90,11 +92,57 @@ Commonmarker.to_html('"Hi *there*"', options: {
 | `superscript`            | Enables the superscript Comrak extension.                                                                           | `false` |
 | `header_ids`             | Enables the header IDs Comrak extension. from the GFM spec.                                                         | `""`    |
 | `footnotes`              | Enables the footnotes extension per `cmark-gfm`.                                                                    | `false` |
-| `description_lists`      | Enables the description lists extension..                                                                           | `false` |
+| `description_lists`      | Enables the description lists extension.                                                                            | `false` |
 | `front_matter_delimiter` | Enables the front matter extension.                                                                                 | `""`    |
+| `shortcodes`             | Enables the shortcodes extension.                                                                                   | `true`  |
 
 For more information on these options, see [the comrak documentation](https://github.com/kivikakk/comrak#usage).
 
+### Plugins
+
+In addition to the possibilities provided by generic CommonMark rendering, Commonmarker also supports plugins as a means of
+providing further niceties.
+
+#### Syntax Highlighter Plugin
+
+````ruby
+code = <<~CODE
+  ```ruby
+  def hello
+  puts "hello"
+  end
+CODE
+
+puts Commonmarker.to_html(code, plugins: { syntax_highlighter: { theme: "InspiredGitHub" } })
+
+# <pre style="background-color:#ffffff;" lang="ruby"><code>
+# <span style="font-weight:bold;color:#a71d5d;">def </span><span style="font-weight:bold;color:#795da3;">hello
+# </span><span style="color:#62a35c;">puts </span><span style="color:#183691;">&quot;hello&quot;
+# </span><span style="font-weight:bold;color:#a71d5d;">end
+# </span>
+# </code></pre>
+````
+
+To disable this plugin, pass `nil`:
+
+```ruby
+Commonmarker.to_html(code, plugins: { syntax_highlighter: nil })
+# or
+Commonmarker.to_html(code, plugins: { syntax_highlighter: { theme: nil } })
+```
+
+##### Available themes
+
+Here's [a list of available themes](https://docs.rs/syntect/5.0.0/syntect/highlighting/struct.ThemeSet.html#implementations):
+
+- `"base16-ocean.dark"`
+- `"base16-eighties.dark"`
+- `"base16-mocha.dark"`
+- `"base16-ocean.light"`
+- `"InspiredGitHub"`
+- `"Solarized (dark)"`
+- `"Solarized (light)"`
+
 ## Output formats
 
 Commonmarker can currently only generate output in one format: HTML.
@@ -102,8 +150,7 @@ Commonmarker can currently only generate output in one format: HTML.
 ### HTML
 
 ```ruby
-html = CommonMarker.to_html('*Hello* world!', :DEFAULT)
-puts(html)
+puts Commonmarker.to_html('*Hello* world!')
 
 # <p><em>Hello</em> world!</p>
 ```

data/ext/commonmarker/Cargo.toml

@@ -4,8 +4,8 @@ version = "1.0.0"
 edition = "2021"
 
 [dependencies]
-magnus = { git = "https://github.com/gjtorikian/magnus", branch = "main" } # waiting for release with full rb-sys backend
-comrak = "0.14.0"
+magnus =  "0.4"
+comrak = { version = "0.16", features = ["shortcodes"] }
 
 [lib]
 name = "commonmarker"

data/ext/commonmarker/extconf.rb

@@ -3,4 +3,4 @@ require "rb_sys/mkmf"
 
 require_relative "_util"
 
-create_rust_makefile("commonmarker")
+create_rust_makefile("commonmarker/commonmarker")

data/ext/commonmarker/src/lib.rs

@@ -1,27 +1,85 @@
 extern crate core;
 
-use comrak::{markdown_to_html, ComrakOptions};
-use magnus::{define_module, function, r_hash::ForEach, Error, RHash, Symbol};
+use comrak::{
+    adapters::SyntaxHighlighterAdapter, markdown_to_html, markdown_to_html_with_plugins,
+    plugins::syntect::SyntectAdapter, ComrakOptions, ComrakPlugins,
+};
+use magnus::{define_module, function, r_hash::ForEach, scan_args, Error, RHash, Symbol, Value};
 
-mod comrak_options;
-use comrak_options::iterate_options_hash;
+mod options;
+use options::iterate_options_hash;
+
+mod plugins;
+use plugins::{
+    syntax_highlighting::{
+        fetch_syntax_highlighter_theme, SYNTAX_HIGHLIGHTER_PLUGIN_DEFAULT_THEME,
+    },
+    SYNTAX_HIGHLIGHTER_PLUGIN,
+};
+
+mod utils;
+
+pub const EMPTY_STR: &str = "";
+
+fn commonmark_to_html<'a>(args: &[Value]) -> Result<String, magnus::Error> {
+    let args = scan_args::scan_args(args)?;
+    let (rb_commonmark,): (String,) = args.required;
+    let _: () = args.optional;
+    let _: () = args.splat;
+    let _: () = args.trailing;
+    let _: () = args.block;
+
+    let kwargs = scan_args::get_kwargs::<_, (), (Option<RHash>, Option<RHash>), ()>(
+        args.keywords,
+        &[],
+        &["options", "plugins"],
+    )?;
+    let (rb_options, rb_plugins) = kwargs.optional;
 
-fn commonmark_to_html(rb_commonmark: String, rb_options: magnus::RHash) -> String {
     let mut comrak_options = ComrakOptions::default();
 
-    rb_options.foreach(|key: Symbol, value: RHash| {
-        iterate_options_hash(&mut comrak_options, key, value);
-        Ok(ForEach::Continue)
-    });
+    if let Some(rb_options) = rb_options {
+        rb_options.foreach(|key: Symbol, value: RHash| {
+            iterate_options_hash(&mut comrak_options, key, value)?;
+            Ok(ForEach::Continue)
+        })?;
+    }
+
+    if let Some(rb_plugins) = rb_plugins {
+        let mut comrak_plugins = ComrakPlugins::default();
+
+        let syntax_highlighter: Option<&dyn SyntaxHighlighterAdapter>;
+        let adapter: SyntectAdapter;
+
+        let theme = match rb_plugins.get(Symbol::new(SYNTAX_HIGHLIGHTER_PLUGIN)) {
+            Some(theme_val) => fetch_syntax_highlighter_theme(theme_val)?,
+            None => SYNTAX_HIGHLIGHTER_PLUGIN_DEFAULT_THEME.to_string(), // no `syntax_highlighter:` defined
+        };
+
+        if theme.is_empty() || theme == "none" {
+            syntax_highlighter = None;
+        } else {
+            adapter = SyntectAdapter::new(&theme);
+            syntax_highlighter = Some(&adapter);
+        }
+
+        comrak_plugins.render.codefence_syntax_highlighter = syntax_highlighter;
 
-    markdown_to_html(&rb_commonmark, &comrak_options)
+        Ok(markdown_to_html_with_plugins(
+            &rb_commonmark,
+            &comrak_options,
+            &comrak_plugins,
+        ))
+    } else {
+        Ok(markdown_to_html(&rb_commonmark, &comrak_options))
+    }
 }
 
 #[magnus::init]
 fn init() -> Result<(), Error> {
     let module = define_module("Commonmarker")?;
 
-    module.define_module_function("commonmark_to_html", function!(commonmark_to_html, 2))?;
+    module.define_module_function("commonmark_to_html", function!(commonmark_to_html, -1))?;
 
     Ok(())
 }

data/ext/commonmarker/src/options.rs

@@ -0,0 +1,134 @@
+use std::borrow::Cow;
+
+use comrak::ComrakOptions;
+
+use magnus::{class, r_hash::ForEach, Error, RHash, Symbol, Value};
+
+use crate::utils::try_convert_string;
+
+const PARSE_SMART: &str = "smart";
+const PARSE_DEFAULT_INFO_STRING: &str = "default_info_string";
+
+fn iterate_parse_options(comrak_options: &mut ComrakOptions, options_hash: RHash) {
+    options_hash
+        .foreach(|key: Symbol, value: Value| {
+            match key.name() {
+                Ok(Cow::Borrowed(PARSE_SMART)) => {
+                    comrak_options.parse.smart = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(PARSE_DEFAULT_INFO_STRING)) => {
+                    comrak_options.parse.default_info_string = try_convert_string(value);
+                }
+                _ => {}
+            }
+            Ok(ForEach::Continue)
+        })
+        .unwrap();
+}
+
+const RENDER_HARDBREAKS: &str = "hardbreaks";
+const RENDER_GITHUB_PRE_LANG: &str = "github_pre_lang";
+const RENDER_WIDTH: &str = "width";
+const RENDER_UNSAFE: &str = "unsafe";
+const RENDER_ESCAPE: &str = "escape";
+
+fn iterate_render_options(comrak_options: &mut ComrakOptions, options_hash: RHash) {
+    options_hash
+        .foreach(|key: Symbol, value: Value| {
+            match key.name() {
+                Ok(Cow::Borrowed(RENDER_HARDBREAKS)) => {
+                    comrak_options.render.hardbreaks = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(RENDER_GITHUB_PRE_LANG)) => {
+                    comrak_options.render.github_pre_lang = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(RENDER_WIDTH)) => {
+                    comrak_options.render.width = value.try_convert::<usize>()?;
+                }
+                Ok(Cow::Borrowed(RENDER_UNSAFE)) => {
+                    comrak_options.render.unsafe_ = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(RENDER_ESCAPE)) => {
+                    comrak_options.render.escape = value.try_convert::<bool>()?;
+                }
+                _ => {}
+            }
+            Ok(ForEach::Continue)
+        })
+        .unwrap();
+}
+
+const EXTENSION_STRIKETHROUGH: &str = "strikethrough";
+const EXTENSION_TAGFILTER: &str = "tagfilter";
+const EXTENSION_TABLE: &str = "table";
+const EXTENSION_AUTOLINK: &str = "autolink";
+const EXTENSION_TASKLIST: &str = "tasklist";
+const EXTENSION_SUPERSCRIPT: &str = "superscript";
+const EXTENSION_HEADER_IDS: &str = "header_ids";
+const EXTENSION_FOOTNOTES: &str = "footnotes";
+const EXTENSION_DESCRIPTION_LISTS: &str = "description_lists";
+const EXTENSION_FRONT_MATTER_DELIMITER: &str = "front_matter_delimiter";
+const EXTENSION_SHORTCODES: &str = "shortcodes";
+
+fn iterate_extension_options(comrak_options: &mut ComrakOptions, options_hash: RHash) {
+    options_hash
+        .foreach(|key: Symbol, value: Value| {
+            match key.name() {
+                Ok(Cow::Borrowed(EXTENSION_STRIKETHROUGH)) => {
+                    comrak_options.extension.strikethrough = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_TAGFILTER)) => {
+                    comrak_options.extension.tagfilter = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_TABLE)) => {
+                    comrak_options.extension.table = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_AUTOLINK)) => {
+                    comrak_options.extension.autolink = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_TASKLIST)) => {
+                    comrak_options.extension.tasklist = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_SUPERSCRIPT)) => {
+                    comrak_options.extension.superscript = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_HEADER_IDS)) => {
+                    comrak_options.extension.header_ids = try_convert_string(value);
+                }
+                Ok(Cow::Borrowed(EXTENSION_FOOTNOTES)) => {
+                    comrak_options.extension.footnotes = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_DESCRIPTION_LISTS)) => {
+                    comrak_options.extension.description_lists = value.try_convert::<bool>()?;
+                }
+                Ok(Cow::Borrowed(EXTENSION_FRONT_MATTER_DELIMITER)) => {
+                    comrak_options.extension.front_matter_delimiter = try_convert_string(value);
+                }
+                Ok(Cow::Borrowed(EXTENSION_SHORTCODES)) => {
+                    comrak_options.extension.shortcodes = value.try_convert::<bool>()?;
+                }
+                _ => {}
+            }
+            Ok(ForEach::Continue)
+        })
+        .unwrap();
+}
+
+pub fn iterate_options_hash(
+    comrak_options: &mut ComrakOptions,
+    key: Symbol,
+    value: RHash,
+) -> Result<ForEach, Error> {
+    assert!(value.is_kind_of(class::hash()));
+
+    if key.name().unwrap() == "parse" {
+        iterate_parse_options(comrak_options, value);
+    }
+    if key.name().unwrap() == "render" {
+        iterate_render_options(comrak_options, value);
+    }
+    if key.name().unwrap() == "extension" {
+        iterate_extension_options(comrak_options, value);
+    }
+    Ok(ForEach::Continue)
+}

data/ext/commonmarker/src/plugins/syntax_highlighting.rs

@@ -0,0 +1,30 @@
+use magnus::{RHash, Symbol, Value};
+
+use crate::EMPTY_STR;
+
+pub const SYNTAX_HIGHLIGHTER_PLUGIN_THEME_KEY: &str = "theme";
+pub const SYNTAX_HIGHLIGHTER_PLUGIN_DEFAULT_THEME: &str = "base16-ocean.dark";
+
+pub fn fetch_syntax_highlighter_theme(value: Value) -> Result<String, magnus::Error> {
+    if value.is_nil() {
+        // `syntax_highlighter: nil`
+        return Ok(EMPTY_STR.to_string());
+    }
+
+    let syntax_highlighter_plugin = value.try_convert::<RHash>()?;
+    let theme_key = Symbol::new(SYNTAX_HIGHLIGHTER_PLUGIN_THEME_KEY);
+
+    match syntax_highlighter_plugin.get(theme_key) {
+        Some(theme) => {
+            if theme.is_nil() {
+                // `syntax_highlighter: { theme: nil }`
+                return Ok(EMPTY_STR.to_string());
+            }
+            Ok(theme.try_convert::<String>()?)
+        }
+        None => {
+            // `syntax_highlighter: {  }`
+            Ok(EMPTY_STR.to_string())
+        }
+    }
+}

data/ext/commonmarker/src/plugins.rs

@@ -0,0 +1,21 @@
+// use comrak::ComrakPlugins;
+// use magnus::{class, r_hash::ForEach, RHash, Symbol, Value};
+
+// use crate::plugins::syntax_highlighting::fetch_syntax_highlighter_theme;
+
+pub mod syntax_highlighting;
+
+pub const SYNTAX_HIGHLIGHTER_PLUGIN: &str = "syntax_highlighter";
+
+// pub fn iterate_plugins_hash(
+//     comrak_plugins: &mut ComrakPlugins,
+//     mut theme: String,
+//     key: Symbol,
+//     value: Value,
+// ) -> Result<ForEach, magnus::Error> {
+//     if key.name().unwrap() == SYNTAX_HIGHLIGHTER_PLUGIN {
+//         theme = fetch_syntax_highlighter_theme(value)?;
+//     }
+
+//     Ok(ForEach::Continue)
+// }

data/ext/commonmarker/src/utils.rs

@@ -0,0 +1,8 @@
+use magnus::Value;
+
+pub fn try_convert_string(value: Value) -> Option<String> {
+    match value.try_convert::<String>() {
+        Ok(s) => Some(s),
+        Err(_) => None,
+    }
+}

data/lib/commonmarker/config.rb

@@ -4,7 +4,7 @@ module Commonmarker
   module Config
     # For details, see
     # https://github.com/kivikakk/comrak/blob/162ef9354deb2c9b4a4e05be495aa372ba5bb696/src/main.rs#L201
-    OPTS = {
+    OPTIONS = {
       parse: {
         smart: false,
         default_info_string: "",
@@ -13,7 +13,7 @@ module Commonmarker
         hardbreaks: true,
         github_pre_lang: true,
         width: 80,
-        unsafe_: false,
+        unsafe: false,
         escape: false,
       }.freeze,
       extension: {
@@ -26,14 +26,23 @@ module Commonmarker
         header_ids: "",
         footnotes: false,
         description_lists: false,
-        front_matter_delimiter: "",
+        front_matter_delimiter: nil,
+        shortcodes: true,
       },
       format: [:html].freeze,
     }.freeze
 
+    PLUGINS = {
+      syntax_highlighter: {
+        theme: "base16-ocean.dark",
+      },
+    }
+
     class << self
+      include Commonmarker::Utils
+
       def merged_with_defaults(options)
-        Commonmarker::Config::OPTS.merge(process_options(options))
+        Commonmarker::Config::OPTIONS.merge(process_options(options))
       end
 
       def process_options(options)
@@ -43,29 +52,44 @@ module Commonmarker
           extension: process_extension_options(options[:extension]),
         }
       end
-  end
 
-    BOOLS = [true, false]
-    ["parse", "render", "extension"].each do |type|
-      define_singleton_method :"process_#{type}_options" do |options|
-        Commonmarker::Config::OPTS[type.to_sym].each_with_object({}) do |(key, value), hash|
-          if options.nil? # option not provided, go for the default
+      def process_plugins(plugins)
+        {
+          syntax_highlighter: process_syntax_highlighter_plugin(plugins&.fetch(:syntax_highlighter, nil)),
+        }
+      end
+    end
+
+    [:parse, :render, :extension].each do |type|
+      define_singleton_method :"process_#{type}_options" do |option|
+        Commonmarker::Config::OPTIONS[type].each_with_object({}) do |(key, value), hash|
+          if option.nil? # option not provided, go for the default
             hash[key] = value
             next
           end
 
           # option explicitly not included, remove it
-          next if options[key].nil?
+          next if option[key].nil?
 
-          value_klass = value.class
-          if BOOLS.include?(value) && BOOLS.include?(options[key])
-            hash[key] = options[key]
-          elsif options[key].is_a?(value_klass)
-            hash[key] = options[key]
-          else
-            expected_type = BOOLS.include?(value) ? "Boolean" : value_klass.to_s
-            raise TypeError, "#{type}_options[:#{key}] must be a #{expected_type}; got #{options[key].class}"
+          hash[key] = fetch_kv(option, key, value, type)
+        end
+      end
+    end
+
+    [:syntax_highlighter].each do |type|
+      define_singleton_method :"process_#{type}_plugin" do |plugin|
+        return nil if plugin.nil? # plugin explicitly nil, remove it
+
+        Commonmarker::Config::PLUGINS[type].each_with_object({}) do |(key, value), hash|
+          if plugin.nil? # option not provided, go for the default
+            hash[key] = value
+            next
           end
+
+          # option explicitly not included, remove it
+          next if plugin[key].nil?
+
+          hash[key] = fetch_kv(plugin, key, value, type)
         end
       end
     end

data/lib/commonmarker/constants.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Commonmarker
+  module Constants
+    BOOLS = [true, false].freeze
+  end
+end

data/lib/commonmarker/extension.rb

@@ -3,7 +3,7 @@
 begin
   # native precompiled gems package shared libraries in <gem_dir>/lib/commonmarker/<ruby_version>
   # load the precompiled extension file
-  ruby_version = /\d+\.\d+/.match(::RUBY_VERSION)
+  ruby_version = /\d+\.\d+/.match(RUBY_VERSION)
   require_relative "#{ruby_version}/commonmarker"
 rescue LoadError
   # fall back to the extension compiled upon installation.

data/lib/commonmarker/utils.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "commonmarker/constants"
+
+module Commonmarker
+  module Utils
+    include Commonmarker::Constants
+
+    def fetch_kv(option, key, value, type)
+      value_klass = value.class
+
+      if Constants::BOOLS.include?(value) && BOOLS.include?(option[key])
+        option[key]
+      elsif option[key].is_a?(value_klass)
+        option[key]
+      else
+        expected_type = Constants::BOOLS.include?(value) ? "Boolean" : value_klass.to_s
+        raise TypeError, "#{type} option `:#{key}` must be #{expected_type}; got #{option[key].class}"
+      end
+    end
+  end
+end

data/lib/commonmarker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Commonmarker
-  VERSION = "1.0.0.pre"
+  VERSION = "1.0.0.pre7"
 end

data/lib/commonmarker.rb

@@ -2,6 +2,7 @@
 
 require_relative "commonmarker/extension"
 
+require "commonmarker/utils"
 require "commonmarker/config"
 require "commonmarker/renderer"
 require "commonmarker/version"
@@ -16,15 +17,19 @@ module Commonmarker
     # Public: Parses a CommonMark string into an HTML string.
     #
     # text - A {String} of text
-    # option - A {Hash} of render, parse, and extension options to transform the text.
+    # options - A {Hash} of render, parse, and extension options to transform the text.
+    # plugins - A {Hash} of additional plugins.
     #
     # Returns a {String} of converted HTML.
-    def to_html(text, options: Commonmarker::Config::OPTS)
+    def to_html(text, options: Commonmarker::Config::OPTIONS, plugins: Commonmarker::Config::PLUGINS)
       raise TypeError, "text must be a String; got a #{text.class}!" unless text.is_a?(String)
+      raise TypeError, "text must be UTF-8 encoded; got #{text.encoding}!" unless text.encoding.name == "UTF-8"
       raise TypeError, "options must be a Hash; got a #{options.class}!" unless options.is_a?(Hash)
 
       opts = Config.process_options(options)
-      commonmark_to_html(text.encode("UTF-8"), opts)
+      plugins = Config.process_plugins(plugins)
+
+      commonmark_to_html(text, options: opts, plugins: plugins)
     end
   end
 end