coradoc 0.2.0 → 0.3.0

data/lib/coradoc/reverse_adoc/README.adoc

@@ -0,0 +1,302 @@
+= AsciiDoc from HTML and Microsoft Word: reverse_adoc
+
+https://github.com/metanorma/reverse_adoc[reverse_adoc] image:https://img.shields.io/gem/v/reverse_adoc.svg["Gem Version", link="https://rubygems.org/gems/reverse_adoc"]::
+image:https://github.com/metanorma/reverse_adoc/workflows/rake/badge.svg["Build Status", link="https://github.com/metanorma/reverse_adoc/actions?workflow=rake"]
+image:https://codeclimate.com/github/metanorma/reverse_adoc/badges/gpa.svg["Code Climate", link="https://codeclimate.com/github/metanorma/reverse_adoc"]
+image:https://img.shields.io/github/issues-pr-raw/metanorma/reverse_adoc.svg["Pull Requests", link="https://github.com/metanorma/reverse_adoc/pulls"]
+image:https://img.shields.io/github/commits-since/metanorma/reverse_adoc/latest.svg["Commits since latest",link="https://github.com/metanorma/reverse_adoc/releases"]
+
+== Purpose
+
+Transforms HTML and Microsoft Word into AsciiDoc.
+
+Based on https://github.com/xijo/reverse_markdown
+
+
+== Installation
+
+Install the gem:
+
+[source,console]
+----
+[sudo] gem install reverse_adoc
+----
+
+or add it to your `Gemfile`:
+
+[source,ruby]
+----
+gem 'reverse_adoc'
+----
+
+
+== Command-line usage
+
+=== HTML to AsciiDoc: `reverse_adoc`
+
+Convert HTML files to AsciiDoc:
+
+[source,console]
+----
+$ reverse_adoc file.html > file.adoc
+$ cat file.html | reverse_adoc > file.adoc
+----
+
+
+=== Microsoft Word to AsciiDoc: `w2a`
+
+Convert Word `.doc` or `.docx` files to AsciiDoc:
+
+[source,console]
+----
+$ w2a file.docx > file.adoc
+----
+
+[source,console]
+----
+$ w2a input.docx -o output.adoc
+----
+
+Help:
+
+[source,console]
+----
+$ w2a -h
+Usage: w2a [options] <file>
+    -a, --mathml2asciimath           Convert MathML to AsciiMath
+    -o, --output=FILENAME            Output file to write to
+    -e, --external-images            Export images if data URI
+    -v, --version                    Version information
+    -h, --help                       Prints this help
+----
+
+
+NOTE: `w2a` requires LibreOffice to be installed. It uses LibreOffice's
+export to XHTML. LibreOffice's export of XHTML is superior to the native Microsoft Word export
+to HTML: it exports lists (which Word keeps as paragraphs), and it exports OOMML into MathML.
+On the other hand, the LibreOffice export relies on default styling being used in the
+document, and it may not cope with ordered lists or headings with customised appearance.
+For best results, reset the styles in the document you're converting to those in
+the default `Normal.dot` template.
+
+NOTE: `w2a` requires the command-line version of LibreOffice, `soffice`. As it turns out,
+LibreOffice v6 appears to render formulae in HTML as images instead of MathML expressions;
+use LibreOffice v5. If you have both LibreOffice v5 and LibreOffice v6 installed, make sure
+that your OS path searches for the LibreOffice v5 version of `soffice` first; e.g. on Mac,
+include something like `/Applications/LibreOffice5.4.7.2.app/Contents/MacOS` in your PATH
+environment.
+
+NOTE: Some information in OOMML is not preserved in the export to MathML from LibreOffice;
+in particular, font shifts such as double-struck fonts.
+The LibreOffice exporter does seem to drop some text (possibly associated with
+MathML); use with caution.
+
+NOTE: Adapted from `w2m` of
+https://github.com/benbalter/word-to-markdown[Ben Balter's word-to-markdown]
+
+
+=== Common options
+
+
+==== MathML to AsciiMath conversion
+
+If you wish to convert the MathML in the document to AsciiMath, run the script with the
+`--mathml2asciimath` option:
+
+[source,console]
+----
+$ w2a --mathml2asciimath document.docx > document.adoc
+----
+
+
+==== Extracting images
+
+Images referred by the HTML can be extracted into the destination output folder by using:
+
+[source,console]
+----
+$ reverse_adoc input.docx -o output/file.adoc -e
+$ reverse_adoc input.docx --output output/file.adoc --external-images
+----
+
+
+Word embedded images can be extracted into the destination output folder by using:
+
+[source,console]
+----
+$ w2a input.docx -o output/file.adoc -e
+$ w2a input.docx --output output/file.adoc --external-images
+----
+
+
+==== Handling unknown HTML tags
+
+The `--unknown_tags` option allows you to specify how to handle unknown tags
+(default `pass_through`).
+
+Valid options are:
+
+* `pass_through` - Include the unknown tag completely into the result
+* `drop` - Drop the unknown tag and its content
+* `bypass` - Ignore the unknown tag but try to convert its content
+* `raise` - Raise an error to let you know
+
+
+==== Tagging of borders
+
+Specify how to handle tag borders with the option `--tag_border` (default `' '`).
+
+Valid options are:
+
+* `' '` - Add whitespace if there is none at tag borders.
+* `''` - Do not not add whitespace.
+
+
+== Features
+
+=== General
+
+`reverse_adoc` shares features as a port of `reverse_markdown`:
+
+* Module based -- if you miss a tag, just add it
+* Can deal with nested lists
+* Inline and block code is supported
+* Supports blockquote
+
+It supports the following HTML tags (these are supported by `reverse_markdown`):
+
+* `a`
+* `blockquote`
+* `br`
+* `code`, `tt` (added: `kbd`, `samp`, `var`)
+* `div`, `article`
+* `em`, `i` (added: `cite`)
+* `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `hr`
+* `img`
+* `li`, `ol`, `ul` (added: `dir`)
+* `p`, `pre`
+* `strong`, `b`
+* `table`, `td`, `th`, `tr`
+
+[NOTE]
+====
+* reverse_adoc does *not* support `del` or `strike`, because Asciidoctor does not out of the box.
+* As with reverse_markdown, `pre` is only treated as sourcecode if it is contained in a `div@class = highlight-` element, or has a `@brush` attribute naming the language (Confluence).
+* The gem does not support `p@align`, because Asciidoctor doesn't
+====
+
+In addition, it supports:
+
+* `aside`
+* `audio`, `video` (with `@src` attributes)
+* `figure`, `figcaption`
+* `mark`
+* `q`
+* `sub`, `sup`
+* `@id` anchors
+* `blockquote@cite`
+* `img/@width`, `img/@height`
+* `ol/@style`, `ol/@start`, `ol/@reversed`, `ul/@type`
+* `td/@colspan`, `td/@rowspan`, `td@/align`, `td@/valign`
+* `table/caption`, `table/@width`, `table/@frame` (partial), `table/@rules` (partial)
+* Lists and paragraphs within cells
+** Not tables within cells: Asciidoctor cannot deal with nested tables
+
+The gem does not support:
+
+* `col`, `colgroup`
+* `source`, `picture`
+* `bdi`, `bdo`, `ruby`, `rt`, `rp`, `wbr`
+* `frame`, `frameset`, `iframe`, `noframes`, `noscript`, `script`, `input`, `output`, `progress`
+* `map`, `canvas`, `dialog`, `embed`, `object`, `param`, `svg`, `track`
+* `fieldset`, `button`, `datalist`, `form`, `label`, `legend`, `menu`, `menulist`, `optgroup`, `option`, `select`, `textarea`
+* `big`, `dfn`, `font`, `s`, `small`, `span`, `strike`, `u`
+* `center`
+* `data`, `meter`
+* `del`, `ins`
+* `footer`, `header`, `main`, `nav`, `details`, `section`, `summary`, `template`
+
+
+=== MathML support
+
+If you are using this gem in the context of https://www.metanorma.com[Metanorma],
+Metanorma AsciiDoc accepts MathML as a native mathematical format. So you do not need
+to convert the MathML to AsciiMath.
+
+The gem will optionally invoke the https://github.com/metanorma/mathml2asciimath
+gem, to convert MathML to AsciiMath. The conversion is not perfect, and will need to be
+post-edited; but it's a lot better than nothing.
+
+NOTE: Asciidoctor does not support MathML input. HTML uses MathML.
+The gem will recognize MathML expressions in HTML, and will wrap them in Asciidoctor
+`stem:[ ]` macros. The result of this gem is not actually legal Asciidoctor for `stem`:
+Asciidoctor will presumably
+think this is AsciiMath in the `stem:[ ]` macro, try to pass it into MathJax as
+AsciiMath, and fail. But of course, MathJax has no problem with MathML, and some postprocessing
+on the Asciidoctor output can ensure that the MathML is treated by MathJax (or whatever else
+uses the output) as such; so this is still much better than nothing for stem processing.
+
+=== Word cleanup
+
+This gem is routinely used in the Metanorma project to export Word documents to AsciiDoc.
+The HTML export from Word that the gem uses, from LibreOffice, is much cleaner than the
+native HTML 4 export from Word; but it has some infelicities which this gem cleans up:
+
+* The HTML export has trouble with subscripts, and routinely exports them as headings; the `w2a`
+script tries to clean them up.
+* The `w2a` cleans up spaces, but it does not strip them.
+* Spaces are removed from anchors and cross-references.
+* Double underscores are removed from anchors and cross-references.
+* Cross-references to `_GoBack` and to `_Toc` followed by numbers (used to construct tables of contents) are ignored.
+
+== Ruby library usage
+
+=== General
+
+Simple to use.
+
+[source,ruby]
+----
+require 'coradoc/reverse_adoc'
+
+result = Coradoc::ReverseAdoc.convert input
+result.inspect # " *feelings* "
+----
+
+=== Configure with options
+
+Just pass your chosen configuration options in after the input. The given options will last for this operation only.
+
+[source,ruby]
+----
+require 'coradoc/reverse_adoc'
+
+Coradoc::ReverseAdoc.convert(input, unknown_tags: :raise, mathml2asciimath: true)
+----
+
+
+=== Preconfigure using an initializer
+
+Or configure it block style on a initializer level. These configurations will last for all conversions until they are set to something different.
+
+[source,ruby]
+----
+require 'coradoc/reverse_adoc'
+
+Coradoc::ReverseAdoc.config do |config|
+  config.unknown_tags      = :bypass
+  config.mathml2asciimath  = true
+  config.tag_border  = ''
+end
+----
+
+
+== Related stuff
+
+* https://github.com/xijo/reverse_markdown[Xijo's original reverse_markdown gem]
+* https://github.com/xijo/reverse_markdown/wiki/Write-your-own-converter[Write custom converters] - Wiki entry about how to write your own converter
+* https://github.com/harlantwood/html_massage[html_massage] - A gem by Harlan T. Wood to convert regular sites into markdown using reverse_markdown
+* https://github.com/benbalter/word-to-markdown[word-to-markdown] - Convert word docs into markdown while using reverse_markdown, by Ben Balter
+* https://github.com/asciidocfx/HtmlToAsciidoc[HtmlToAsciidoc] - Javascript regexp-based converter of HTML to Asciidoctor
+* https://asciidoctor.org/docs/user-manual/[The Asciidoctor User Manual]
+

data/lib/coradoc/reverse_adoc/cleaner.rb

@@ -0,0 +1,113 @@
+module Coradoc::ReverseAdoc
+  class Cleaner
+    def tidy(string)
+      result = remove_inner_whitespaces(String.new(string))
+      result = remove_newlines(result)
+      result = remove_leading_newlines(result)
+      result = clean_tag_borders(result)
+      clean_punctuation_characters(result)
+    end
+
+    def remove_newlines(string)
+      string.gsub(/\n{3,}/, "\n\n")
+    end
+
+    def remove_leading_newlines(string)
+      string.gsub(/\A\n+/, "")
+    end
+
+    def remove_inner_whitespaces(string)
+      unless string.nil?
+        string.gsub!(/\n stem:\[/, "\nstem:[")
+        string.gsub!(/(stem:\[([^\]]|\\\])*\])\n(?=\S)/, "\\1 ")
+        string.gsub!(/(stem:\[([^\]]|\\\])*\])\s+(?=[\^-])/, "\\1")
+      end
+      string.each_line.inject("") do |memo, line|
+        memo + preserve_border_whitespaces(line) do
+          line.strip.gsub(/[ \t]{2,}/, " ")
+        end
+      end
+    end
+
+    # Find non-asterisk content that is enclosed by two or
+    # more asterisks. Ensure that only one whitespace occurs
+    # in the border area.
+    # Same for underscores and brackets.
+    def clean_tag_borders(string)
+      # result = string.gsub(/\s?\*{2,}.*?\*{2,}\s?/) do |match|
+      # preserve_border_whitespaces(match, default_border: Coradoc::ReverseAdoc.config.tag_border) do
+      #   match.strip.sub("** ", "**").sub(" **", "**")
+      # end
+      # end
+
+      # result = string.gsub(/\s?_{2,}.*?_{2,}\s?/) do |match|
+      #   preserve_border_whitespaces(match, default_border: Coradoc::ReverseAdoc.config.tag_border) do
+      #     match.strip.sub("__ ", "__").sub(" __", "__")
+      #   end
+      # end
+
+      result = string.gsub(/\s?~{2,}.*?~{2,}\s?/) do |match|
+        preserve_border_whitespaces(match,
+                                    default_border: Coradoc::ReverseAdoc.config.tag_border) do
+          match.strip.sub("~~ ", "~~").sub(" ~~", "~~")
+        end
+      end
+
+      result.gsub(/\s?\[.*?\]\s?/) do |match|
+        preserve_border_whitespaces(match) do
+          match.strip.sub("[ ", "[").sub(" ]", "]")
+        end
+      end
+    end
+
+    def clean_punctuation_characters(string)
+      string.gsub(/(\*\*|~~|__)\s([.!?'"])/, "#{'\\1'.strip}\\2")
+    end
+
+    # preprocesses HTML, rather than postprocessing it
+    def preprocess_word_html(string)
+      clean_headings(scrub_whitespace(string.dup))
+    end
+
+    def scrub_whitespace(string)
+      string.gsub!(/ | |\u00a0/i, " ") # HTML encoded spaces
+      string.sub!(/^\A[[:space:]]+/m, "") # document leading whitespace
+      string.sub!(/[[:space:]]+\z$/m, "") # document trailing whitespace
+      string.gsub!(/( +)$/, " ") # line trailing whitespace
+      string.gsub!(/\n\n\n\n/, "\n\n") # Quadruple line breaks
+      # string.delete!('?| ')               # Unicode non-breaking spaces, injected as tabs
+      string
+    end
+
+    # following added by me
+    def clean_headings(string)
+      string.gsub!(%r{<h([1-9])[^>]*></h\1>}, " ")
+      # I don't know why Libre Office is inserting them, but they need to go
+      string.gsub!(%r{<h([1-9])[^>]* style="vertical-align: super;[^>]*>(.+?)</h\1>},
+                   "<sup>\\2</sup>")
+      # I absolutely don't know why Libre Office is rendering superscripts as h1
+      string
+    end
+
+    private
+
+    def preserve_border_whitespaces(string, options = {})
+      return string if /\A\s*\Z/.match?(string)
+
+      default_border = options.fetch(:default_border, "")
+      # If the string contains part of a link so the characters [,],(,)
+      # then don't add any extra spaces
+      default_border = "" if /[\[(\])]/.match?(string)
+      string_start   = present_or_default(string[/\A\s*/], default_border)
+      string_end     = present_or_default(string[/\s*\Z/], default_border)
+      result         = yield
+      string_start + result + string_end
+    end
+
+    def present_or_default(string, default)
+      return default if string.nil? || string.empty?
+
+      string
+    end
+  end
+end

data/lib/coradoc/reverse_adoc/config.rb

@@ -0,0 +1,54 @@
+require "tmpdir"
+
+module Coradoc::ReverseAdoc
+  class Config
+    attr_accessor :unknown_tags, :tag_border, :mathml2asciimath, :external_images,
+                  :destination, :sourcedir, :image_counter, :image_counter_pattern, :input_format
+
+    def initialize
+      @unknown_tags     = :pass_through
+      @input_format     = :html
+      @mathml2asciimath = false
+      @external_images  = false
+
+      # Destination to save file and images
+      @destination      = nil
+
+      # Source of HTML
+      # @sourcedir        = nil
+
+      # Image counter, assuming there are max 999 images
+      @image_counter = 1
+      # pad with 0s
+      @image_counter_pattern = "%03d"
+
+      @em_delimiter     = "_".freeze
+      @strong_delimiter = "*".freeze
+      @inline_options   = {}
+      @tag_border       = " ".freeze
+    end
+
+    def with(options = {})
+      @inline_options = options
+      result = yield
+      @inline_options = {}
+      result
+    end
+
+    def unknown_tags
+      @inline_options[:unknown_tags] || @unknown_tags
+    end
+
+    def mathml2asciimath
+      @inline_options[:mathml2asciimath] || @mathml2asciimath
+    end
+
+    def external_images
+      @inline_options[:external_images] || @external_images
+    end
+
+    def tag_border
+      @inline_options[:tag_border] || @tag_border
+    end
+  end
+end

data/lib/coradoc/reverse_adoc/converters/a.rb

@@ -0,0 +1,42 @@
+require "coradoc"
+
+module Coradoc::ReverseAdoc
+  module Converters
+    class A < Base
+      def to_coradoc(node, state = {})
+        name  = treat_children(node, state)
+
+        href  = node["href"]
+        title = extract_title(node)
+        id = node["id"] || node["name"]
+
+        id = id&.gsub(/\s/, "")&.gsub(/__+/, "_")
+
+        return "" if /^_Toc\d+$|^_GoBack$/.match?(id)
+
+        if !id.nil? && !id.empty?
+          return Coradoc::Element::Inline::Anchor.new(id)
+        end
+
+        if href.to_s.start_with?("#")
+          href = href.sub(/^#/, "").gsub(/\s/, "").gsub(/__+/, "_")
+          return Coradoc::Element::Inline::CrossReference.new(href, name)
+        end
+
+        if href.to_s.empty?
+          return name
+        end
+
+        Coradoc::Element::Inline::Link.new(path: href,
+                                           name: name,
+                                           title: title)
+      end
+
+      def convert(node, state = {})
+        Coradoc::Generator.gen_adoc(to_coradoc(node, state))
+      end
+    end
+
+    register :a, A.new
+  end
+end

data/lib/coradoc/reverse_adoc/converters/aside.rb

@@ -0,0 +1,16 @@
+module Coradoc::ReverseAdoc
+  module Converters
+    class Aside < Base
+      def to_coradoc(node, state = {})
+        content = treat_children(node, state)
+        Coradoc::Element::Block::Side.new(lines: content.lines)
+      end
+
+      def convert(node, state = {})
+        Coradoc::Generator.gen_adoc(to_coradoc(node, state))
+      end
+    end
+
+    register :aside, Aside.new
+  end
+end

data/lib/coradoc/reverse_adoc/converters/audio.rb

@@ -0,0 +1,29 @@
+module Coradoc::ReverseAdoc
+  module Converters
+    class Audio < Base
+      def to_coradoc(node, _state = {})
+        src = node["src"]
+        id = node["id"]
+        title = extract_title(node)
+        attributes = Coradoc::Element::AttributeList.new
+        options = options(node)
+        attributes.add_named("options", options) if options.any?
+        Coradoc::Element::Audio.new(title, id: id, src: src,
+                                           attributes: attributes)
+      end
+
+      def convert(node, state = {})
+        Coradoc::Generator.gen_adoc(to_coradoc(node, state))
+      end
+
+      def options(node)
+        autoplay = node["autoplay"]
+        loop_attr = node["loop"]
+        controls = node["controls"]
+        [autoplay, loop_attr, controls].compact
+      end
+    end
+
+    register :audio, Audio.new
+  end
+end

data/lib/coradoc/reverse_adoc/converters/base.rb

@@ -0,0 +1,100 @@
+module Coradoc::ReverseAdoc
+  module Converters
+    class Base
+      def treat_children(node, state)
+        node.children.inject("") do |memo, child|
+          memo << treat(child, state)
+        end
+      end
+
+      def treat(node, state)
+        Coradoc::ReverseAdoc::Converters.lookup(node.name).convert(node, state)
+      end
+
+      def treat_children_coradoc(node, state)
+        node.children.inject([]) do |memo, child|
+          memo << treat_coradoc(child, state)
+        end.flatten.reject { |x| x == "" || x.nil? }
+      end
+
+      def treat_coradoc(node, state)
+        Coradoc::ReverseAdoc::Converters.lookup(node.name).to_coradoc(node, state)
+      end
+
+      def escape_keychars(string)
+        subs = { "*" => '\*', "_" => '\_' }
+        string
+          .gsub(/((?<=\s)[\*_]+)|[\*_]+(?=\s)/) do |n|
+          n.chars.map do |char|
+            subs[char]
+          end.join
+        end
+      end
+
+      def extract_title(node)
+        title = escape_keychars(node["title"].to_s)
+        title.empty? ? "" : %[ #{title}]
+      end
+
+      def node_has_ancestor?(node, name)
+        case name
+        when String
+          node.ancestors.map(&:name).include?(name)
+        when Array
+          (node.ancestors.map(&:name) & name).any?
+        end
+      end
+
+      def textnode_before_end_with?(node, str)
+        return nil if !str.is_a?(String) || str.empty?
+
+        node2 = node.at_xpath("preceding-sibling::node()[1]")
+        node2.respond_to?(:text) && node2.text.end_with?(str)
+      end
+
+      def unconstrained_before?(node)
+        before = node.at_xpath("preceding::node()[1]")
+
+        before &&
+          !before.text.strip.empty? &&
+          before.text[-1]&.match?(/\w/)
+      end
+
+      # TODO: This logic ought to be cleaned up.
+      def unconstrained_after?(node)
+        after = node.at_xpath("following::node()[1]")
+
+        after && !after.text.strip.empty? &&
+          after.text[0]&.match?(/\w|,|;|"|\.\?!/)
+      end
+
+      # def trailing_whitespace?(node)
+
+      # TODO: This logic ought to be cleaned up.
+      def constrained?(node)
+        before = node.at_xpath("preceding::node()[1]").to_s[-1]
+        before = if before
+                   before&.match?(/\s/) ? true : false
+                 else
+                   true
+                 end
+
+        if !before && (node.to_s[0] =~ /\s/)
+          before = true
+        end
+
+        after = node.at_xpath("following::node()[1]").to_s[0]
+        after = if after
+                  after&.match?(/\s|,|;|"|\.\?!/) ? true : false
+                else
+                  true
+                end
+        if !after && (node.to_s[-1] =~ /\s/)
+          after = true
+        end
+
+        before && after
+      end
+    end
+  end
+end

data/lib/coradoc/reverse_adoc/converters/blockquote.rb

@@ -0,0 +1,27 @@
+module Coradoc::ReverseAdoc
+  module Converters
+    class Blockquote < Base
+      def to_coradoc(node, state = {})
+        node["id"]
+        cite = node["cite"]
+        attributes = if cite.nil?
+                       nil
+                     else
+                       Coradoc::Element::AttributeList.new(
+                         "quote", cite
+                       )
+                     end
+        content = treat_children(node, state).strip
+        content = Coradoc::ReverseAdoc.cleaner.remove_newlines(content)
+        Coradoc::Element::Block::Quote.new(nil, lines: content,
+                                                attributes: attributes)
+      end
+
+      def convert(node, state = {})
+        Coradoc::Generator.gen_adoc(to_coradoc(node, state))
+      end
+    end
+
+    register :blockquote, Blockquote.new
+  end
+end