word-to-markdown 1.1.8 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3febb4398acdc4eacedcc62e09f4beeaee625858043a27e6df7e597fee1e0d17
-  data.tar.gz: 7a76057aeca2db8f321282bc309a835f282ea921343b28c8bce6d83cd0fc4582
+  metadata.gz: d5fff6778e90bf76409d47c5177fd2400a14308cf40be3ea2e3ee118917945b0
+  data.tar.gz: 18062303949a156da4c940daeb79854c820036fe623cdb9c4c07b1bce9dca242
 SHA512:
-  metadata.gz: ee2340688c2d5f3f21c7e47f85220bebc88201e28200a682146041fa7f5a47e89c56d687b4f6565a95d7d3e7f4c70fb1551894ad297620e7bd49cef520975e18
-  data.tar.gz: 44d405387990ee9a09cb33572a1d7843461c0817789c92a08df6d3542082ffc837bbd39d544a2e6a734a0772c67da3afed77049dc96b0ace0d562473515816e1
+  metadata.gz: f041232dbf6ae3aee4738d9562fab3f820a492dfd8d71fdd257397b278ca03550a27a59a0d0b5c3514254bab49367ba067970f7551ea11c3a7b90b8eda434499
+  data.tar.gz: e8c267ccd410ccfc978ef873805696299b1a658d1894dccae2f63d39f8afd12595ced23fbce6c8f4173188cc0b1e6fa44957d37c29c32b8ea18c08fb9a8745e1

data/README.md

@@ -1,24 +1,29 @@
 # Word to Markdown converter
 
-A Ruby gem to liberate content from [the jail that is Word documents](http://ben.balter.com/2012/10/19/we-ve-been-trained-to-make-paper/#jailbreaking-content)
+> [!IMPORTANT]
+> **Looking for the latest and greatest?** Check out [**word-to-markdown-js**](https://github.com/benbalter/word-to-markdown-js), the newer, better successor to this project. It's a modern, actively maintained rewrite and is recommended for new projects. This Ruby gem remains available for existing users.
 
-[![Build Status](https://travis-ci.org/benbalter/word-to-markdown.svg?branch=master)](https://travis-ci.org/benbalter/word-to-markdown) [![Gem Version](https://badge.fury.io/rb/word-to-markdown.png)](http://badge.fury.io/rb/word-to-markdown) [![Inline docs](http://inch-ci.org/github/benbalter/word-to-markdown.png)](http://inch-ci.org/github/benbalter/word-to-markdown) [![Build status](https://ci.appveyor.com/api/projects/status/x2gnsfvli3q47a2e/branch/master?svg=true)](https://ci.appveyor.com/project/benbalter/word-to-markdown/branch/master)
+A Ruby gem to liberate content from [the jail that is Word documents](https://ben.balter.com/2012/10/19/we-ve-been-trained-to-make-paper/#jailbreaking-content)
+
+[![CI](https://github.com/benbalter/word-to-markdown/actions/workflows/ci.yml/badge.svg)](https://github.com/benbalter/word-to-markdown/actions/workflows/ci.yml) [![Gem Version](https://img.shields.io/gem/v/word-to-markdown)](https://rubygems.org/gems/word-to-markdown)
 
 ## The problem
 
-> Our default content publishing workflow is terribly broken. [We've all been trained to make paper](http://ben.balter.com/2012/10/19/we-ve-been-trained-to-make-paper/), yet today, content authored once is more commonly consumed in multiple formats, and rarely, if ever, does it embody physical form. Put another way, our go-to content authoring workflow remains relatively unchanged since it was conceived in the early 80s.
+> Our default content publishing workflow is terribly broken. [We've all been trained to make paper](https://ben.balter.com/2012/10/19/we-ve-been-trained-to-make-paper/), yet today, content authored once is more commonly consumed in multiple formats, and rarely, if ever, does it embody physical form. Put another way, our go-to content authoring workflow remains relatively unchanged since it was conceived in the early 80s.
 >
-> I'm asked regularly by government employees — knowledge workers who fire up a desktop word processor as the first step to any project — for an automated pipeline to convert Microsoft Word documents to [Markdown](http://guides.github.com/overviews/mastering-markdown/), the *lingua franca* of the internet, but as my recent foray into building [just such a converter](http://word-to-markdown.herokuapp.com/) proves, it's not that simple.
+> I'm asked regularly by government employees — knowledge workers who fire up a desktop word processor as the first step to any project — for an automated pipeline to convert Microsoft Word documents to [Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax), the *lingua franca* of the internet, but as my recent foray into building [just such a converter](https://word2md.com/) proves, it's not that simple.
 >
 > Markdown isn't just an alternative format. Markdown forces you to write for the web.
 
-**[Read more](http://ben.balter.com/2014/03/31/word-versus-markdown-more-than-mere-semantics/)**
+**[Read more](https://ben.balter.com/2014/03/31/word-versus-markdown-more-than-mere-semantics/)**
+
+## Just want to convert a Microsoft Word (or Google) document to Markdown?
 
-**[Demo](http://word-to-markdown.herokuapp.com/)**
+You can use this **[hosted service](https://word2md.com/)** (or check out [its source](https://github.com/benbalter/word-to-markdown-server)).
 
 ## Install
 
-You'll need to install [LibreOffice](http://www.libreoffice.org/). Then:
+You'll need to install [LibreOffice](https://www.libreoffice.org/). Then:
 
 ```bash
 gem install word-to-markdown
@@ -65,14 +70,30 @@ $ w2m path/to/document.docx
 
 Word-to-markdown requires `soffice` a command line interface to LibreOffice that works on Linux, Mac, and Windows. To install soffice, see [the LibreOffice documentation](https://www.libreoffice.org/get-help/install-howto/).
 
+Word-to-markdown only accepts Word documents (`.docx` and `.doc`), identified by their contents rather than their file extension. Other files, including those LibreOffice could otherwise open, such as HTML, ODT, or RTF, raise `WordToMarkdown::Document::UnsupportedFormatError`.
+
+LibreOffice is killed if a conversion takes longer than 60 seconds, raising `WordToMarkdown::TimeoutError`. To change the limit, set `WordToMarkdown.timeout = 120` or the `WORD_TO_MARKDOWN_TIMEOUT` environment variable.
+
+### Converting untrusted documents
+
+Word documents can link to remote resources, such as images, which LibreOffice fetches while converting the document. If you convert documents from untrusted sources (for example, files uploaded to a web service), run the conversion without network access, such as in a container or sandbox with no outbound network, so that a document can't make requests to internal services or other hosts on your behalf.
+
 ## Testing
 
 ```
 script/cibuild
 ```
 
-## Server
+## Docker
+
+Everything you need to run the executable locally:
+
+```
+docker compose build
+docker compose run --rm app bundle exec w2m --help
+docker compose run --rm app bundle exec w2m test/fixtures/em.docx
+```
 
-[Word-to-markdown-demo](https://github.com/benbalter/word-to-markdown-demo) contains a lightweight server for converting Word Documents as a service.
+## Hosted service
 
-A live version runs at [word-to-markdown.herokuapp.com](http://word-to-markdown.herokuapp.com).
+[Word-to-markdown-server](https://github.com/benbalter/word-to-markdown-server) contains a lightweight server for converting Word Documents as a service. A live version runs at [word2md.com](https://word2md.com).

data/bin/w2m

@@ -1,17 +1,35 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+require 'optparse'
 require 'word-to-markdown'
 
-if ARGV.size != 1 || ARGV[0] == '--help'
-  puts 'Usage: bundle exec w2m path/to/document.docx'
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: w2m path/to/document.docx'
+
+  opts.on('-h', '--help', 'Show this help') do
+    puts opts
+    exit
+  end
+
+  opts.on('-v', '--version', 'Show the WordToMarkdown and LibreOffice versions') do
+    puts "WordToMarkdown v#{WordToMarkdown::VERSION}"
+    puts "LibreOffice v#{WordToMarkdown.soffice.version}" unless Gem.win_platform?
+    exit
+  end
+end
+
+begin
+  parser.parse!
+rescue OptionParser::ParseError => e
+  warn e.message
+  warn parser
   exit 1
 end
 
-if ARGV[0] == '--version'
-  puts "WordToMarkdown v#{WordToMarkdown::VERSION}"
-  puts "LibreOffice v#{WordToMarkdown.soffice.version}" unless Gem.win_platform?
-else
-  doc = WordToMarkdown.new ARGV[0]
-  puts doc.to_s
+if ARGV.size != 1
+  warn parser
+  exit 1
 end
+
+puts WordToMarkdown.new(ARGV[0])

data/lib/cliver/dependency_ext.rb

@@ -24,14 +24,15 @@ module Cliver
 
     # Returns the version of the resolved dependency
     def version
-      return @detected_version if defined? @detected_version
+      return @version if defined? @version
       return if Gem.win_platform?
+
       version = installed_versions.find { |p, _v| p == path }
-      @detected_version = version.nil? ? nil : version[1]
+      @version = version.nil? ? nil : version[1]
     end
 
     def major_version
-      version.split('.').first if version
+      version&.split('.')&.first
     end
   end
 end

data/lib/word-to-markdown/converter.rb

@@ -7,14 +7,22 @@ class WordToMarkdown
     # Number of headings to guess, e.g., h6
     HEADING_DEPTH = 6
 
-    # Percentile step for eaceh eheading
+    # Percentile step for each heading
     HEADING_STEP = 100 / HEADING_DEPTH
 
     # Minimum heading size
     MIN_HEADING_SIZE = 20
 
     # Unicode bullets to strip when processing
-    UNICODE_BULLETS = ['○', 'o', '●', "\u2022", '\\p{C}'].freeze
+    UNICODE_BULLETS = ['○', '●', "\u2022", '\\p{C}'].freeze
+
+    # Leading bullets to strip from list items. A plain "o" only counts as a
+    # bullet when followed by whitespace, so words like "orange" survive.
+    BULLET_REGEX = /\A(?:[#{UNICODE_BULLETS.join}]|o(?=[[:space:]]))+[[:space:]]*/
+
+    # Leading list numbering to strip, e.g., "1.", "a.", or "iv.", along with
+    # any whitespace (including non-breaking spaces) that follows it
+    NUMBERING_REGEX = /\A(?:\d+|[a-zA-Z]|[ivxlcdm]+|[IVXLCDM]+)\.(?:[[:space:]]+|\z)/
 
     # @param document [WordToMarkdown::Document] The document to convert
     def initialize(document)
@@ -64,10 +72,11 @@ class WordToMarkdown
 
     # Given a Nokogiri node, guess what heading it represents, if any
     #
-    # @param node [Nokigiri::Node] the nokigiri node
+    # @param node [Nokogiri::Node] the nokogiri node
     # @return [String, nil] the heading tag (e.g., H1), or nil
     def guess_heading(node)
       return nil if node.font_size.nil?
+
       [*1...HEADING_DEPTH].each do |heading|
         return "h#{heading}" if node.font_size >= h(heading)
       end
@@ -81,7 +90,23 @@ class WordToMarkdown
     #
     # @return [Integer] the minimum font size
     def h(num)
-      font_sizes.percentile(((HEADING_DEPTH - 1) - num) * HEADING_STEP)
+      self.class.percentile(font_sizes, ((HEADING_DEPTH - 1) - num) * HEADING_STEP)
+    end
+
+    # Linearly interpolated percentile, matching the algorithm used by the
+    # descriptive_statistics gem (v2.5.1) that this replaces
+    #
+    # @param values [Array<Numeric>] the values
+    # @param pct [Numeric] the percentile, from 0 to 100
+    #
+    # @return [Float, nil] the percentile, or nil if values is empty
+    def self.percentile(values, pct)
+      sorted = values.map(&:to_f).sort
+      rank = pct / 100.0 * (sorted.size - 1)
+      lower, upper = sorted[rank.floor, 2]
+      return lower if upper.nil? # empty or single-element collection, or the 100th percentile
+
+      lower + ((upper - lower) * (rank - rank.floor))
     end
 
     # Convert span-based font styles to `strong`s and `em`s
@@ -109,7 +134,7 @@ class WordToMarkdown
     def remove_unicode_bullets_from_list_items!
       path = WordToMarkdown.soffice.major_version == '5' ? 'li span span' : 'li span'
       @document.tree.search(path).each do |span|
-        span.inner_html = span.inner_html.gsub(/^([#{UNICODE_BULLETS.join("")}]+)/, '')
+        span.inner_html = span.inner_html.sub(BULLET_REGEX, '')
       end
     end
 
@@ -117,21 +142,21 @@ class WordToMarkdown
     def remove_numbering_from_list_items!
       path = WordToMarkdown.soffice.major_version == '5' ? 'li span span' : 'li span'
       @document.tree.search(path).each do |span|
-        span.inner_html = span.inner_html.gsub(/^[a-zA-Z0-9]+\./m, '')
+        span.inner_html = span.inner_html.sub(NUMBERING_REGEX, '')
       end
     end
 
-    # Remvoe whitespace from list items
+    # Remove whitespace from list items
     def remove_whitespace_from_list_items!
-      @document.tree.search('li span').each { |span| span.inner_html.strip! }
+      @document.tree.search('li span').each { |span| span.inner_html = span.inner_html.strip }
     end
 
-    # Convert table headers to `th`s2
+    # Convert table headers to `th`s
     def semanticize_table_headers!
       @document.tree.search('table tr:first td').each { |node| node.node_name = 'th' }
     end
 
-    # Try to guess heading where implicit bassed on font size
+    # Try to guess heading where implicit based on font size
     def semanticize_headings!
       implicit_headings.each do |element|
         heading = guess_heading element

data/lib/word-to-markdown/document.rb

@@ -3,16 +3,24 @@
 class WordToMarkdown
   class Document
     class NotFoundError < StandardError; end
+
     class ConversionError < StandardError; end
 
-    attr_reader :path, :tmpdir
+    class UnsupportedFormatError < StandardError; end
+
+    attr_reader :path, :tmpdir, :import_filter
 
     # @param path [string] Path to the Word document
     # @param tmpdir [string] Path to a working directory to use
     def initialize(path, tmpdir = nil)
       @path = File.expand_path path, Dir.pwd
-      @tmpdir = tmpdir || Dir.mktmpdir
       raise NotFoundError, "File #{@path} does not exist" unless File.exist?(@path)
+
+      @import_filter = InputFormat.filter_for(@path)
+      raise UnsupportedFormatError, "File #{@path} is not a Word document (.docx or .doc)" if @import_filter.nil?
+
+      @own_tmpdir = tmpdir.nil?
+      @tmpdir = tmpdir || Dir.mktmpdir
     end
 
     # @return [String] the document's extension
@@ -20,7 +28,7 @@ class WordToMarkdown
       File.extname path
     end
 
-    # @return [Nokigiri::Document]
+    # @return [Nokogiri::Document]
     def tree
       @tree ||= begin
         tree = Nokogiri::HTML(normalized_html)
@@ -31,6 +39,7 @@ class WordToMarkdown
 
     # @return [String] the html representation of the document
     def html
+      UrlScrubber.scrub!(tree)
       tree.to_html.gsub("</li>\n", '</li>')
     end
 
@@ -44,7 +53,7 @@ class WordToMarkdown
     #
     # @return [String] the encoding, defaulting to "UTF-8"
     def encoding
-      match = raw_html.encode('UTF-8', invalid: :replace, replace: '').match(/charset=([^\"]+)/)
+      match = raw_html.encode('UTF-8', invalid: :replace, replace: '').match(/charset=([^"]+)/)
       if match
         match[1].sub('macintosh', 'MacRoman')
       else
@@ -78,30 +87,40 @@ class WordToMarkdown
       string.gsub!('&nbsp;', ' ')       # HTML encoded spaces
       string.sub!(/\A[[:space:]]+/, '') # document leading whitespace
       string.sub!(/[[:space:]]+\z/, '') # document trailing whitespace
-      string.gsub!(/([ ]+)$/, '')       # line trailing whitespace
-      string.gsub!(/\n\n\n\n/, "\n\n")  # Quadruple line breaks
+      string.gsub!(/( +)$/, '')         # line trailing whitespace
+      string.gsub!("\n\n\n\n", "\n\n")  # Quadruple line breaks
       string.delete!(' ')               # Unicode non-breaking spaces, injected as tabs
+      string.gsub!(/\*\*\ +(?!\*|_)([[:punct:]])/, '**\1') # Remove extra space after bold
       string
     end
 
     # @return [String] the path to the intermediary HTML document
     def dest_path
-      dest_filename = File.basename(path).gsub(/#{Regexp.escape(extension)}$/, '.html')
-      File.expand_path(dest_filename, tmpdir)
+      File.expand_path("#{File.basename(path, '.*')}.html", tmpdir)
     end
 
     # @return [String] the unnormalized HTML representation
     def raw_html
       @raw_html ||= begin
-        WordToMarkdown.run_command '--headless', '--convert-to', filter, path, '--outdir', tmpdir
+        WordToMarkdown.run_command '--headless', "--infilter=#{import_filter}", '--convert-to', filter, path, '--outdir', tmpdir
         raise ConversionError, "Failed to convert #{path}" unless File.exist?(dest_path)
+
         html = File.read dest_path
         File.delete dest_path
         html
+      ensure
+        remove_tmpdir
       end
     end
 
-    # @return [String] the LibreOffice filter to use for conversion
+    # Remove the working directory if we created it and nothing else is in it.
+    # Non-empty directories are kept, since LibreOffice may have written
+    # images there that the markdown references.
+    def remove_tmpdir
+      Dir.rmdir(tmpdir) if @own_tmpdir && Dir.empty?(tmpdir)
+    end
+
+    # @return [String] the LibreOffice filter to use for export
     def filter
       if WordToMarkdown.soffice.major_version == '5'
         'html:XHTML Writer File:UTF8'

data/lib/word-to-markdown/input_format.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+class WordToMarkdown
+  # Identifies Word documents by their content, rather than their extension,
+  # so that other formats LibreOffice would otherwise sniff and import (e.g.,
+  # HTML, which can reference remote resources) are never passed to it
+  module InputFormat
+    # Signature of an OLE compound file, used by .doc files
+    OLE_SIGNATURE = "\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1".b.freeze
+
+    # Signature of a ZIP local file header, used by .docx files
+    ZIP_SIGNATURE = "PK\x03\x04".b.freeze
+
+    # Signature of a ZIP end of central directory record
+    ZIP_EOCD_SIGNATURE = "PK\x05\x06".b.freeze
+
+    # Size of the end of central directory record, excluding the comment
+    ZIP_EOCD_SIZE = 22
+
+    # Maximum length of a ZIP archive comment
+    ZIP_MAX_COMMENT_SIZE = 0xFFFF
+
+    # The main document part every Word OOXML package contains
+    OOXML_DOCUMENT_PART = 'word/document.xml'.b.freeze
+
+    # LibreOffice import filters, by format
+    FILTERS = {
+      ooxml: 'MS Word 2007 XML',
+      ole: 'MS Word 97'
+    }.freeze
+
+    class << self
+      # @param path [String] path to the file
+      # @return [String, nil] the LibreOffice import filter for the file, or nil if it isn't a Word document
+      def filter_for(path)
+        FILTERS[detect(path)]
+      end
+
+      # @param path [String] path to the file
+      # @return [Symbol, nil] :ooxml for .docx, :ole for .doc, or nil if it isn't a Word document
+      def detect(path)
+        File.open(path, 'rb') do |file|
+          signature = file.read(OLE_SIGNATURE.bytesize).to_s
+          if signature == OLE_SIGNATURE
+            :ole
+          elsif signature.start_with?(ZIP_SIGNATURE) && ooxml_document?(file)
+            :ooxml
+          end
+        end
+      end
+
+      private
+
+      # @param file [File] an open ZIP file
+      # @return [Boolean] true if the ZIP's central directory lists the Word main document part
+      def ooxml_document?(file)
+        directory = central_directory(file)
+        !directory.nil? && directory.include?(OOXML_DOCUMENT_PART)
+      end
+
+      # Read the ZIP central directory, whose entries include each file name, uncompressed
+      #
+      # @param file [File] an open ZIP file
+      # @return [String, nil] the raw central directory, or nil if it can't be found
+      def central_directory(file)
+        size, offset = central_directory_location(file)
+        return if size.nil? || offset + size > file.size
+
+        file.seek(offset)
+        file.read(size)
+      end
+
+      # @param file [File] an open ZIP file
+      # @return [Array<Integer>, nil] the central directory's size and offset, or nil if not found
+      def central_directory_location(file)
+        tail_size = [file.size, ZIP_EOCD_SIZE + ZIP_MAX_COMMENT_SIZE].min
+        file.seek(-tail_size, IO::SEEK_END)
+        tail = file.read(tail_size)
+        index = tail.rindex(ZIP_EOCD_SIGNATURE)
+        return if index.nil? || tail.bytesize - index < ZIP_EOCD_SIZE
+
+        tail.byteslice(index + 12, 8).unpack('VV')
+      end
+    end
+  end
+end

data/lib/word-to-markdown/url_scrubber.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+class WordToMarkdown
+  # Removes links and images with unsafe URL schemes (e.g., javascript: or
+  # vbscript:) so that they don't survive into the markdown output
+  module UrlScrubber
+    # URL schemes permitted in link targets. Links with any other scheme are
+    # unwrapped to their text. Relative URLs and fragments are always permitted.
+    SAFE_LINK_SCHEMES = %w[http https mailto].freeze
+
+    # URL schemes permitted in image sources, in addition to data:image/ URIs.
+    # Images with any other scheme are removed.
+    SAFE_IMAGE_SCHEMES = %w[http https].freeze
+
+    # Browsers remove ASCII tabs and newlines anywhere in a URL, and C0
+    # control characters and spaces at either end, before parsing the scheme
+    STRIPPED_CHARS = /[\t\n\r]/
+    EDGE_CHARS = /\A[\x00-\x20]+|[\x00-\x20]+\z/
+
+    # Matches a URL scheme, e.g., "https:"
+    SCHEME_REGEX = /\A([a-z][a-z0-9+.-]*):/i
+
+    # Matches a URL without a recognizable scheme whose first segment contains
+    # characters a Markdown renderer or browser may decode into a scheme
+    # separator, e.g., "javascript:", "javascript:", or "javascript\:"
+    AMBIGUOUS_REGEX = %r{\A[^/?#]*[:&\\]}
+
+    # Characters that could end or alter a Markdown link destination, which
+    # are percent-encoded in the URLs that are kept
+    DESTINATION_UNSAFE_CHARS = /[\x00-\x20\x7F()<>\[\]\\`"]/
+
+    # Matches a data URI for an image
+    DATA_IMAGE_REGEX = %r{\Adata:image/}i
+
+    class << self
+      # Unwrap links and remove images whose URL scheme isn't permitted
+      #
+      # @param tree [Nokogiri::HTML::Document] the document to scrub, in place
+      # @return [Nokogiri::HTML::Document] the scrubbed document
+      def scrub!(tree)
+        tree.css('a[href]').each do |node|
+          safe_link?(node['href']) ? escape_attributes!(node, 'href') : node.replace(node.children)
+        end
+
+        tree.css('img[src]').each do |node|
+          safe_image?(node['src']) ? escape_attributes!(node, 'src') : node.remove
+        end
+
+        tree
+      end
+
+      # Percent-encode characters in a URL that could end or alter a Markdown link destination
+      #
+      # @param url [String] the URL
+      # @return [String] the escaped URL
+      def escape_destination(url)
+        url.gsub(DESTINATION_UNSAFE_CHARS) { |char| format('%%%02X', char.ord) }
+      end
+
+      # @param url [String] a link target
+      # @return [Boolean] true if the URL is relative, a fragment, or has a permitted scheme
+      def safe_link?(url)
+        scheme = scheme(url)
+        scheme.nil? ? relative?(url) : SAFE_LINK_SCHEMES.include?(scheme)
+      end
+
+      # @param url [String] an image source
+      # @return [Boolean] true if the URL is relative, a data:image/ URI, or has a permitted scheme
+      def safe_image?(url)
+        scheme = scheme(url)
+        return relative?(url) if scheme.nil?
+
+        SAFE_IMAGE_SCHEMES.include?(scheme) || normalize(url).match?(DATA_IMAGE_REGEX)
+      end
+
+      # @param url [String] the URL
+      # @return [String, nil] the URL's lowercased scheme, or nil if it has none
+      def scheme(url)
+        match = normalize(url).match(SCHEME_REGEX)
+        match && match[1].downcase
+      end
+
+      private
+
+      # Escape a kept link or image's URL and title so they can't break out
+      # of the Markdown link that ReverseMarkdown writes for them
+      #
+      # @param node [Nokogiri::XML::Element] the link or image
+      # @param attribute [String] the name of the URL attribute
+      def escape_attributes!(node, attribute)
+        node[attribute] = escape_destination(node[attribute])
+        node['title'] = node['title'].tr('"', "'") if node['title']
+      end
+
+      # @param url [String] a URL without a scheme
+      # @return [Boolean] true if the URL is a relative path or fragment that can't be decoded into one with a scheme
+      def relative?(url)
+        !normalize(url).match?(AMBIGUOUS_REGEX)
+      end
+
+      # Normalize a URL the way a browser does before parsing its scheme
+      #
+      # @param url [String] the URL
+      # @return [String] the normalized URL
+      def normalize(url)
+        url.to_s.gsub(STRIPPED_CHARS, '').gsub(EDGE_CHARS, '')
+      end
+    end
+  end
+end

data/lib/word-to-markdown/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class WordToMarkdown
-  VERSION = '1.1.8'.freeze
+  VERSION = '1.2.0'
 end

data/lib/word-to-markdown.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-require 'descriptive_statistics'
 require 'reverse_markdown'
 require 'nokogiri-styles'
 require 'premailer'
@@ -12,28 +11,38 @@ require 'cliver'
 require 'open3'
 
 require_relative 'word-to-markdown/version'
+require_relative 'word-to-markdown/input_format'
+require_relative 'word-to-markdown/url_scrubber'
 require_relative 'word-to-markdown/document'
 require_relative 'word-to-markdown/converter'
 require_relative 'nokogiri/xml/element'
 require_relative 'cliver/dependency_ext'
 
 class WordToMarkdown
+  class TimeoutError < StandardError; end
+
   attr_reader :document, :converter
 
   # Options to be passed to Reverse Markdown
   REVERSE_MARKDOWN_OPTIONS = {
-    unknown_tags:    :bypass,
+    unknown_tags: :bypass,
     github_flavored: true
   }.freeze
 
+  # Default number of seconds to wait for LibreOffice to convert a document
+  # before giving up. Can be overridden with the WORD_TO_MARKDOWN_TIMEOUT
+  # environment variable, or by setting WordToMarkdown.timeout
+  DEFAULT_TIMEOUT = 60
+
   # Minimum version of LibreOffice Required
-  SOFFICE_VERSION_REQUIREMENT = '> 4.0'.freeze
+  SOFFICE_VERSION_REQUIREMENT = '> 4.0'
 
   # Paths to look for LibreOffice, in order of preference
   PATHS = [
     '*', # Sub'd for ENV["PATH"]
     '~/Applications/LibreOffice.app/Contents/MacOS',
     '/Applications/LibreOffice.app/Contents/MacOS',
+    '/Program Files/LibreOffice/program',
     '/Program Files/LibreOffice 5/program',
     '/Program Files (x86)/LibreOffice 4/program'
   ].freeze
@@ -56,19 +65,46 @@ class WordToMarkdown
   end
 
   class << self
+    attr_writer :timeout
+
+    # @return [Numeric] seconds to wait for LibreOffice before giving up
+    def timeout
+      @timeout ||= Float(ENV.fetch('WORD_TO_MARKDOWN_TIMEOUT', DEFAULT_TIMEOUT))
+    end
+
     # Run an soffice command
     #
-    # @param args [string] one or more arguments to pass to the sofice command
+    # @param args [string] one or more arguments to pass to the soffice command
     # @return [string] the command output
     def run_command(*args)
       raise 'LibreOffice already running' if soffice.open?
 
-      output, status = Open3.capture2e(soffice.path, *args)
+      output, status = capture_with_timeout(soffice.path, *args, timeout: timeout)
       logger.debug output
       raise "Command `#{soffice.path} #{args.join(' ')}` failed: #{output}" if status.exitstatus != 0
+
       output
     end
 
+    # Run a command, capturing its combined output, and kill it (along with
+    # any child processes) if it runs longer than the timeout
+    #
+    # @param command [Array<String>] the command and its arguments
+    # @param timeout [Numeric] seconds to wait before killing the command
+    # @return [Array(String, Process::Status)] the command output and status
+    def capture_with_timeout(*command, timeout:)
+      Open3.popen2e(*command, process_group_option) do |stdin, output, waiter|
+        stdin.close
+        reader = read_in_background(output)
+        unless waiter.join(timeout)
+          kill_process_group(waiter.pid)
+          raise TimeoutError, "Command `#{command.join(' ')}` timed out after #{timeout} seconds"
+        end
+
+        [reader.value, waiter.value]
+      end
+    end
+
     # Returns a Cliver::Dependency object representing our soffice dependency
     #
     # Attempts to resolve by looking at PATH followed by paths in the PATHS constant
@@ -85,7 +121,7 @@ class WordToMarkdown
     # @return Logger instance
     def logger
       @logger ||= begin
-        logger = Logger.new(STDOUT)
+        logger = Logger.new($stdout)
         logger.level = Logger::ERROR unless ENV['DEBUG']
         logger
       end
@@ -93,13 +129,38 @@ class WordToMarkdown
 
     private
 
+    # Start commands in their own process group, so that on timeout any
+    # processes they spawn (e.g., soffice.bin) are killed too
+    def process_group_option
+      Gem.win_platform? ? { new_pgroup: true } : { pgroup: true }
+    end
+
+    # Read a stream in a separate thread, so the command can't block on a full pipe
+    #
+    # @param io [IO] the stream to read
+    # @return [Thread] a thread whose value is the stream's contents
+    def read_in_background(io)
+      Thread.new do
+        io.read
+      rescue IOError
+        '' # The stream was closed after the command timed out
+      end
+    end
+
+    # @param pid [Integer] the pid of a process group leader
+    def kill_process_group(pid)
+      Process.kill('KILL', Gem.win_platform? ? pid : -pid)
+    rescue Errno::ESRCH
+      nil
+    end
+
     # Workaround for two upstream bugs:
-    # 1. `soffice.exe --version` on windows opens a popup and retuns a null string when manually closed
+    # 1. `soffice.exe --version` on windows opens a popup and returns a null string when manually closed
     # 2. Even if the second argument to Cliver is nil, Cliver thinks there's a requirement
     #    and will shell out to `soffice.exe --version`
     # In order to support Windows, don't pass *any* version requirement to Cliver
     def soffice_dependency_args
-      args = [path: PATHS.join(File::PATH_SEPARATOR)]
+      args = [{ path: PATHS.join(File::PATH_SEPARATOR) }]
       if Gem.win_platform?
         args
       else

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: word-to-markdown
 version: !ruby/object:Gem::Version
-  version: 1.1.8
+  version: 1.2.0
 platform: ruby
 authors:
 - Ben Balter
-autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-08-01 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: cliver
@@ -25,19 +24,19 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0.3'
 - !ruby/object:Gem::Dependency
-  name: descriptive_statistics
+  name: logger
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.5'
+        version: '1.4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.5'
+        version: '1.4'
 - !ruby/object:Gem::Dependency
   name: nokogiri-styles
   requirement: !ruby/object:Gem::Requirement
@@ -70,128 +69,160 @@ dependencies:
   name: reverse_markdown
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
 - !ruby/object:Gem::Dependency
   name: sys-proctable
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.0'
+        version: '1.3'
 - !ruby/object:Gem::Dependency
-  name: bundler
+  name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '1.6'
+        version: '7'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5'
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '1.6'
+        version: '7'
 - !ruby/object:Gem::Dependency
-  name: minitest
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '4'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '4'
+- !ruby/object:Gem::Dependency
+  name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
+        version: '0.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
+        version: '0.10'
 - !ruby/object:Gem::Dependency
-  name: mocha
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.1'
+        version: '13.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.1'
+        version: '13.0'
 - !ruby/object:Gem::Dependency
-  name: pry
+  name: rubocop
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.10'
+        version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.10'
+        version: '1.0'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: rubocop-minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.4'
+        version: '0.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.4'
+        version: '0.3'
 - !ruby/object:Gem::Dependency
-  name: rubocop
+  name: rubocop-performance
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.49'
+        version: '1.5'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.49'
+        version: '1.5'
 - !ruby/object:Gem::Dependency
   name: shoulda
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.5'
+        version: '4.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.5'
+        version: '4.0'
 description: Ruby Gem to convert Word documents to markdown.
 email: ben.balter@github.com
 executables:
@@ -207,12 +238,17 @@ files:
 - lib/word-to-markdown.rb
 - lib/word-to-markdown/converter.rb
 - lib/word-to-markdown/document.rb
+- lib/word-to-markdown/input_format.rb
+- lib/word-to-markdown/url_scrubber.rb
 - lib/word-to-markdown/version.rb
 homepage: https://github.com/benbalter/word-to-markdown
 licenses:
 - MIT
-metadata: {}
-post_install_message: 
+metadata:
+  rubygems_mfa_required: 'true'
+  source_code_uri: https://github.com/benbalter/word-to-markdown
+  bug_tracker_uri: https://github.com/benbalter/word-to-markdown/issues
+  changelog_uri: https://github.com/benbalter/word-to-markdown/releases
 rdoc_options: []
 require_paths:
 - lib
@@ -220,16 +256,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '3.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6
-signing_key: 
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Ruby Gem to convert Word documents to markdown
 test_files: []