loofah 2.2.3 → 2.21.1

data/README.md

@@ -1,81 +1,74 @@
 # Loofah
 
 * https://github.com/flavorjones/loofah
-* Docs: http://rubydoc.info/github/flavorjones/loofah/master/frames
+* Docs: http://rubydoc.info/github/flavorjones/loofah/main/frames
 * Mailing list: [loofah-talk@googlegroups.com](https://groups.google.com/forum/#!forum/loofah-talk)
 
 ## Status
 
-|System|Status|
-|--|--|
-| Concourse | [![Concourse CI](https://ci.nokogiri.org/api/v1/teams/nokogiri-core/pipelines/loofah/jobs/ruby-2.5/badge)](https://ci.nokogiri.org/teams/nokogiri-core/pipelines/loofah?groups=master) |
-| Code Climate | [![Code Climate](https://codeclimate.com/github/flavorjones/loofah.svg)](https://codeclimate.com/github/flavorjones/loofah) |
-| Version Eye | [![Version Eye](https://www.versioneye.com/ruby/loofah/badge.png)](https://www.versioneye.com/ruby/loofah) |
+[![ci](https://github.com/flavorjones/loofah/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/flavorjones/loofah/actions/workflows/ci.yml)
+[![Tidelift dependencies](https://tidelift.com/badges/package/rubygems/loofah)](https://tidelift.com/subscription/pkg/rubygems-loofah?utm_source=rubygems-loofah&utm_medium=referral&utm_campaign=readme)
 
 
 ## Description
 
-Loofah is a general library for manipulating and transforming HTML/XML
-documents and fragments. It's built on top of Nokogiri and libxml2, so
-it's fast and has a nice API.
+Loofah is a general library for manipulating and transforming HTML/XML documents and fragments, built on top of Nokogiri.
 
-Loofah excels at HTML sanitization (XSS prevention). It includes some
-nice HTML sanitizers, which are based on HTML5lib's whitelist, so it
-most likely won't make your codes less secure. (These statements have
-not been evaluated by Netexperts.)
+Loofah also includes some HTML sanitizers based on `html5lib`'s safelist, which are a specific application of the general transformation functionality.
 
-ActiveRecord extensions for sanitization are available in the
-[`loofah-activerecord` gem](https://github.com/flavorjones/loofah-activerecord).
+Active Record extensions for HTML sanitization are available in the [`loofah-activerecord` gem](https://github.com/flavorjones/loofah-activerecord).
 
 
 ## Features
 
-* Easily write custom scrubbers for HTML/XML leveraging the sweetness of Nokogiri (and HTML5lib's whitelists).
-* Common HTML sanitizing tasks are built-in:
+* Easily write custom transformations for HTML and XML
+* Common HTML sanitizing transformations are built-in:
   * _Strip_ unsafe tags, leaving behind only the inner text.
   * _Prune_ unsafe tags and their subtrees, removing all traces that they ever existed.
   * _Escape_ unsafe tags and their subtrees, leaving behind lots of <tt>&lt;</tt> and <tt>&gt;</tt> entities.
   * _Whitewash_ the markup, removing all attributes and namespaced nodes.
-* Common HTML transformation tasks are built-in:
+* Other common HTML transformations are built-in:
   * Add the _nofollow_ attribute to all hyperlinks.
-* Format markup as plain text, with or without sensible whitespace handling around block elements.
+  * Remove _unprintable_ characters from text nodes.
+* Format markup as plain text, with (or without) sensible whitespace handling around block elements.
 * Replace Rails's `strip_tags` and `sanitize` view helper methods.
 
 
 ## Compare and Contrast
 
-Loofah is one of two known Ruby XSS/sanitization solutions that
-guarantees well-formed and valid markup (the other is Sanitize, which
-also uses Nokogiri).
+Loofah is both:
 
-Loofah works on XML, XHTML and HTML documents.
+- a general framework for transforming XML, XHTML, and HTML documents
+- a specific toolkit for HTML sanitization
 
-Also, it's pretty fast. Here is a benchmark comparing Loofah to other
-commonly-used libraries (ActionView, Sanitize, HTML5lib and HTMLfilter):
+### General document transformation
 
-* https://gist.github.com/170193
+Loofah tries to make it easy to write your own custom scrubbers for whatever document transformation you need. You don't like the built-in scrubbers? Build your own, like a boss.
 
-Lastly, Loofah is extensible. It's super-easy to write your own custom
-scrubbers for whatever document manipulation you need. You don't like
-the built-in scrubbers? Build your own, like a boss.
+
+### HTML sanitization
+
+Another Ruby library that provides HTML sanitization is [`rgrove/sanitize`](https://github.com/rgrove/sanitize), another library built on top of Nokogiri, which provides a bit more flexibility on the tags and attributes being scrubbed.
+
+You may also want to look at [`rails/rails-html-sanitizer`](https://github.com/rails/rails-html-sanitizer) which is built on top of Loofah and provides some useful extensions and additional flexibility in the HTML sanitization.
 
 
 ## The Basics
 
-Loofah wraps [Nokogiri](http://nokogiri.org) in a loving
-embrace. Nokogiri is an excellent HTML/XML parser. If you don't know
-how Nokogiri works, you might want to pause for a moment and go check
-it out. I'll wait.
+Loofah wraps [Nokogiri](http://nokogiri.org) in a loving embrace. Nokogiri is a stable, well-maintained parser for XML, HTML4, and HTML5.
 
-Loofah presents the following classes:
+Loofah implements the following classes:
 
-* `Loofah::HTML::Document` and `Loofah::HTML::DocumentFragment`
-* `Loofah::XML::Document` and `Loofah::XML::DocumentFragment`
-* `Loofah::Scrubber`
+* `Loofah::HTML5::Document`
+* `Loofah::HTML5::DocumentFragment`
+* `Loofah::HTML4::Document` (aliased as `Loofah::HTML::Document` for now)
+* `Loofah::HTML4::DocumentFragment` (aliased as `Loofah::HTML::DocumentFragment` for now)
+* `Loofah::XML::Document`
+* `Loofah::XML::DocumentFragment`
 
-The documents and fragments are subclasses of the similar Nokogiri classes.
+These document and fragment classes are subclasses of the similarly-named Nokogiri classes `Nokogiri::HTML5::Document` et al.
 
-The Scrubber represents the document manipulation, either by wrapping
+Loofah also implements `Loofah::Scrubber`, which represents the document transformation, either by wrapping
 a block,
 
 ``` ruby
@@ -89,50 +82,49 @@ or by implementing a method.
 
 ### Side Note: Fragments vs Documents
 
-Generally speaking, unless you expect to have a DOCTYPE and a single
-root node, you don't have a *document*, you have a *fragment*. For
-HTML, another rule of thumb is that *documents* have `html` and `body`
-tags, and *fragments* usually do not.
+Generally speaking, unless you expect to have a DOCTYPE and a single root node, you don't have a *document*, you have a *fragment*. For HTML, another rule of thumb is that *documents* have `html` and `body` tags, and *fragments* usually do not.
+
+**HTML fragments** should be parsed with `Loofah.html5_fragment` or `Loofah.html4_fragment`. The result won't be wrapped in `html` or `body` tags, won't have a DOCTYPE declaration, `head` elements will be silently ignored, and multiple root nodes are allowed.
+
+**HTML documents** should be parsed with `Loofah.html5_document` or `Loofah.html4_document`. The result will have a DOCTYPE declaration, along with `html`, `head` and `body` tags.
+
+**XML fragments** should be parsed with `Loofah.xml_fragment`. The result won't have a DOCTYPE declaration, and multiple root nodes are allowed.
 
-HTML fragments should be parsed with Loofah.fragment. The result won't
-be wrapped in `html` or `body` tags, won't have a DOCTYPE declaration,
-`head` elements will be silently ignored, and multiple root nodes are
-allowed.
+**XML documents** should be parsed with `Loofah.xml_document`. The result will have a DOCTYPE declaration and a single root node.
 
-XML fragments should be parsed with Loofah.xml_fragment. The result
-won't have a DOCTYPE declaration, and multiple root nodes are allowed.
 
-HTML documents should be parsed with Loofah.document. The result will
-have a DOCTYPE declaration, along with `html`, `head` and `body` tags.
+### Side Note: HTML4 vs HTML5
 
-XML documents should be parsed with Loofah.xml_document. The result
-will have a DOCTYPE declaration and a single root node.
+⚠ _HTML5 functionality is not available on JRuby, or with versions of Nokogiri `< 1.14.0`._
 
+Currently, Loofah's methods `Loofah.document` and `Loofah.fragment` are aliases to `.html4_document` and `.html4_fragment`, which use Nokogiri's HTML4 parser. (Similarly, `Loofah::HTML::Document` and `Loofah::HTML::DocumentFragment` are aliased to `Loofah::HTML4::Document` and `Loofah::HTML4::DocumentFragment`.)
 
-### Loofah::HTML::Document and Loofah::HTML::DocumentFragment
+**Please note** that in a future version of Loofah, these methods and classes may switch to using Nokogiri's HTML5 parser and classes on platforms that support it [1].
 
-These classes are subclasses of Nokogiri::HTML::Document and
-Nokogiri::HTML::DocumentFragment, so you get all the markup
-fixer-uppery and API goodness of Nokogiri.
+**We strongly recommend that you explicitly use `.html5_document` or `.html5_fragment`** unless you know of a compelling reason not to. If you are sure that you need to use the HTML4 parser, you should explicitly call `.html4_document` or `.html4_fragment` to avoid breakage in a future version.
 
-The module methods Loofah.document and Loofah.fragment will parse an
-HTML document and an HTML fragment, respectively.
+  [1]: [[feature request] HTML5 parser for JRuby implementation · Issue #2227 · sparklemotion/nokogiri](https://github.com/sparklemotion/nokogiri/issues/2227)
+
+
+### `Loofah::HTML5::Document` and `Loofah::HTML5::DocumentFragment`
+
+These classes are subclasses of `Nokogiri::HTML5::Document` and `Nokogiri::HTML5::DocumentFragment`.
+
+The module methods `Loofah.html5_document` and `Loofah.html5_fragment` will parse either an HTML document and an HTML fragment, respectively.
 
 ``` ruby
-Loofah.document(unsafe_html).is_a?(Nokogiri::HTML::Document)         # => true
-Loofah.fragment(unsafe_html).is_a?(Nokogiri::HTML::DocumentFragment) # => true
+Loofah.html5_document(unsafe_html).is_a?(Nokogiri::HTML5::Document)         # => true
+Loofah.html5_fragment(unsafe_html).is_a?(Nokogiri::HTML5::DocumentFragment) # => true
 ```
 
-Loofah injects a `scrub!` method, which takes either a symbol (for
-built-in scrubbers) or a Loofah::Scrubber object (for custom
-scrubbers), and modifies the document in-place.
+Loofah injects a `scrub!` method, which takes either a symbol (for built-in scrubbers) or a `Loofah::Scrubber` object (for custom scrubbers), and modifies the document in-place.
 
 Loofah overrides `to_s` to return HTML:
 
 ``` ruby
 unsafe_html = "ohai! <div>div is safe</div> <script>but script is not</script>"
 
-doc = Loofah.fragment(unsafe_html).scrub!(:prune)
+doc = Loofah.html5_fragment(unsafe_html).scrub!(:prune)
 doc.to_s    # => "ohai! <div>div is safe</div> "
 ```
 
@@ -142,36 +134,41 @@ and `text` to return plain text:
 doc.text    # => "ohai! div is safe "
 ```
 
-Also, `to_text` is available, which does the right thing with
-whitespace around block-level elements.
+Also, `to_text` is available, which does the right thing with whitespace around block-level and line break elements.
+
+``` ruby
+doc = Loofah.html5_fragment("<h1>Title</h1><div>Content<br>Next line</div>")
+doc.text    # => "TitleContentNext line"            # probably not what you want
+doc.to_text # => "\nTitle\n\nContent\nNext line\n"  # better
+```
+
+### `Loofah::HTML4::Document` and `Loofah::HTML4::DocumentFragment`
+
+These classes are subclasses of `Nokogiri::HTML4::Document` and `Nokogiri::HTML4::DocumentFragment`.
+
+The module methods `Loofah.html4_document` and `Loofah.html4_fragment` will parse either an HTML document and an HTML fragment, respectively.
 
 ``` ruby
-doc = Loofah.fragment("<h1>Title</h1><div>Content</div>")
-doc.text    # => "TitleContent"           # probably not what you want
-doc.to_text # => "\nTitle\n\nContent\n"   # better
+Loofah.html4_document(unsafe_html).is_a?(Nokogiri::HTML4::Document)         # => true
+Loofah.html4_fragment(unsafe_html).is_a?(Nokogiri::HTML4::DocumentFragment) # => true
 ```
 
-### Loofah::XML::Document and Loofah::XML::DocumentFragment
+### `Loofah::XML::Document` and `Loofah::XML::DocumentFragment`
 
-These classes are subclasses of Nokogiri::XML::Document and
-Nokogiri::XML::DocumentFragment, so you get all the markup
-fixer-uppery and API goodness of Nokogiri.
+These classes are subclasses of `Nokogiri::XML::Document` and `Nokogiri::XML::DocumentFragment`.
 
-The module methods Loofah.xml_document and Loofah.xml_fragment will
-parse an XML document and an XML fragment, respectively.
+The module methods `Loofah.xml_document` and `Loofah.xml_fragment` will parse an XML document and an XML fragment, respectively.
 
 ``` ruby
 Loofah.xml_document(bad_xml).is_a?(Nokogiri::XML::Document)         # => true
 Loofah.xml_fragment(bad_xml).is_a?(Nokogiri::XML::DocumentFragment) # => true
 ```
 
-### Nodes and NodeSets
+### Nodes and Node Sets
 
-Nokogiri::XML::Node and Nokogiri::XML::NodeSet also get a `scrub!`
-method, which makes it easy to scrub subtrees.
+Nokogiri's `Node` and `NodeSet` classes also get a `scrub!` method, which makes it easy to scrub subtrees.
 
-The following code will apply the `employee_scrubber` only to the
-`employee` nodes (and their subtrees) in the document:
+The following code will apply the `employee_scrubber` only to the `employee` nodes (and their subtrees) in the document:
 
 ``` ruby
 Loofah.xml_document(bad_xml).xpath("//employee").scrub!(employee_scrubber)
@@ -183,7 +180,7 @@ And this code will only scrub the first `employee` node and its subtree:
 Loofah.xml_document(bad_xml).at_xpath("//employee").scrub!(employee_scrubber)
 ```
 
-### Loofah::Scrubber
+### `Loofah::Scrubber`
 
 A Scrubber wraps up a block (or method) that is run on a document node:
 
@@ -197,14 +194,11 @@ end
 This can then be run on a document:
 
 ``` ruby
-Loofah.fragment("<span>foo</span><p>bar</p>").scrub!(span2div).to_s
+Loofah.html5_fragment("<span>foo</span><p>bar</p>").scrub!(span2div).to_s
 # => "<div>foo</div><p>bar</p>"
 ```
 
-Scrubbers can be run on a document in either a top-down traversal (the
-default) or bottom-up. Top-down scrubbers can optionally return
-Scrubber::STOP to terminate the traversal of a subtree. Read below and
-in the Loofah::Scrubber class for more detailed usage.
+Scrubbers can be run on a document in either a top-down traversal (the default) or bottom-up. Top-down scrubbers can optionally return `Scrubber::STOP` to terminate the traversal of a subtree. Read below and in the `Loofah::Scrubber` class for more detailed usage.
 
 Here's an XML example:
 
@@ -219,12 +213,12 @@ end
 Loofah.xml_document(File.read('plague.xml')).scrub!(bring_out_your_dead)
 ```
 
-=== Built-In HTML Scrubbers
+### Built-In HTML Scrubbers
 
-Loofah comes with a set of sanitizing scrubbers that use HTML5lib's
-whitelist algorithm:
+Loofah comes with a set of sanitizing scrubbers that use `html5lib`'s safelist algorithm:
 
 ``` ruby
+doc = Loofah.html5_document(input)
 doc.scrub!(:strip)       # replaces unknown/unsafe tags with their inner text
 doc.scrub!(:prune)       #  removes unknown/unsafe tags and their children
 doc.scrub!(:escape)      #  escapes unknown/unsafe tags, like this: &lt;script&gt;
@@ -232,14 +226,14 @@ doc.scrub!(:whitewash)   #  removes unknown/unsafe/namespaced tags and their chi
                          #          and strips all node attributes
 ```
 
-Loofah also comes with some common transformation tasks: 
+Loofah also comes with some common transformation tasks:
 
 ``` ruby
 doc.scrub!(:nofollow)    #     adds rel="nofollow" attribute to links
 doc.scrub!(:unprintable) #  removes unprintable characters from text nodes
 ```
 
-See Loofah::Scrubbers for more details and example usage.
+See `Loofah::Scrubbers` for more details and example usage.
 
 
 ### Chaining Scrubbers
@@ -247,7 +241,7 @@ See Loofah::Scrubbers for more details and example usage.
 You can chain scrubbers:
 
 ``` ruby
-Loofah.fragment("<span>hello</span> <script>alert('OHAI')</script>") \
+Loofah.html5_fragment("<span>hello</span> <script>alert('OHAI')</script>") \
       .scrub!(:prune) \
       .scrub!(span2div).to_s
 # => "<div>hello</div> "
@@ -255,21 +249,26 @@ Loofah.fragment("<span>hello</span> <script>alert('OHAI')</script>") \
 
 ### Shorthand
 
-The class methods Loofah.scrub_fragment and Loofah.scrub_document are
-shorthand.
+The class methods `Loofah.scrub_html5_fragment` and `Loofah.scrub_html5_document` (and the corresponding HTML4 methods) are shorthand.
+
+These methods:
 
 ``` ruby
-Loofah.scrub_fragment(unsafe_html, :prune)
-Loofah.scrub_document(unsafe_html, :prune)
+Loofah.scrub_html5_fragment(unsafe_html, :prune)
+Loofah.scrub_html5_document(unsafe_html, :prune)
+Loofah.scrub_html4_fragment(unsafe_html, :prune)
+Loofah.scrub_html4_document(unsafe_html, :prune)
 Loofah.scrub_xml_fragment(bad_xml, custom_scrubber)
 Loofah.scrub_xml_document(bad_xml, custom_scrubber)
 ```
 
-are the same thing as (and arguably semantically clearer than):
+do the same thing as (and arguably semantically clearer than):
 
 ``` ruby
-Loofah.fragment(unsafe_html).scrub!(:prune)
-Loofah.document(unsafe_html).scrub!(:prune)
+Loofah.html5_fragment(unsafe_html).scrub!(:prune)
+Loofah.html5_document(unsafe_html).scrub!(:prune)
+Loofah.html4_fragment(unsafe_html).scrub!(:prune)
+Loofah.html4_document(unsafe_html).scrub!(:prune)
 Loofah.xml_fragment(bad_xml).scrub!(custom_scrubber)
 Loofah.xml_document(bad_xml).scrub!(custom_scrubber)
 ```
@@ -277,10 +276,9 @@ Loofah.xml_document(bad_xml).scrub!(custom_scrubber)
 
 ### View Helpers
 
-Loofah has two "view helpers": Loofah::Helpers.sanitize and
-Loofah::Helpers.strip_tags, both of which are drop-in replacements for
-the Rails ActionView helpers of the same name.
-These are no longer required automatically. You must require `loofah/helpers`. 
+Loofah has two "view helpers": `Loofah::Helpers.sanitize` and `Loofah::Helpers.strip_tags`, both of which are drop-in replacements for the Rails Action View helpers of the same name.
+
+These are not required automatically. You must require `loofah/helpers` to use them.
 
 
 ## Requirements
@@ -306,7 +304,9 @@ And the mailing list is on Google Groups:
 * Mail: loofah-talk@googlegroups.com
 * Archive: https://groups.google.com/forum/#!forum/loofah-talk
 
-And the IRC channel is \#loofah on freenode.
+Consider subscribing to [Tidelift][tidelift] which provides license assurances and timely security notifications for your open source dependencies, including Loofah. [Tidelift][tidelift] subscriptions also help the Loofah maintainers fund our [automated testing](https://ci.nokogiri.org) which in turn allows us to ship releases, bugfixes, and security updates more often.
+
+  [tidelift]: https://tidelift.com/subscription/pkg/rubygems-loofah?utm_source=undefined&utm_medium=referral&utm_campaign=enterprise
 
 
 ## Security
@@ -314,26 +314,12 @@ And the IRC channel is \#loofah on freenode.
 See [`SECURITY.md`](SECURITY.md) for vulnerability reporting details.
 
 
-### "Secure by Default"
-
-Some tools may incorrectly report Loofah as a potential security
-vulnerability.
-
-Loofah depends on Nokogiri, and it's _possible_ to use Nokogiri in a
-dangerous way (by enabling its DTDLOAD option and disabling its NONET
-option). This specifically allows the opportunity for an XML External
-Entity (XXE) vulnerability if the XML data is untrusted.
-
-However, Loofah __never enables this Nokogiri configuration__; Loofah
-never enables DTDLOAD, and it never disables NONET, thereby protecting
-you by default from this XXE vulnerability.
-
-
 ## Related Links
 
+* loofah-activerecord: https://github.com/flavorjones/loofah-activerecord
 * Nokogiri: http://nokogiri.org
 * libxml2: http://xmlsoft.org
-* html5lib: https://code.google.com/p/html5lib
+* html5lib: https://github.com/html5lib/
 
 
 ## Authors
@@ -354,15 +340,14 @@ And a big shout-out to Corey Innis for the name, and feedback on the API.
 
 ## Thank You
 
-The following people have generously donated via the [Pledgie](http://pledgie.com) badge on the [Loofah github page](https://github.com/flavorjones/loofah):
+The following people have generously funded Loofah:
 
 * Bill Harding
 
 
 ## Historical Note
 
-This library was formerly known as Dryopteris, which was a very bad
-name that nobody could spell properly.
+This library was once named "Dryopteris", which was a very bad name that nobody could spell properly.
 
 
 ## License

data/lib/loofah/concerns.rb

@@ -0,0 +1,207 @@
+# frozen_string_literal: true
+
+module Loofah
+  #
+  #  Mixes +scrub!+ into Document, DocumentFragment, Node and NodeSet.
+  #
+  #  Traverse the document or fragment, invoking the +scrubber+ on each node.
+  #
+  #  +scrubber+ must either be one of the symbols representing the built-in scrubbers (see
+  #  Scrubbers), or a Scrubber instance.
+  #
+  #    span2div = Loofah::Scrubber.new do |node|
+  #      node.name = "div" if node.name == "span"
+  #    end
+  #    Loofah.html5_fragment("<span>foo</span><p>bar</p>").scrub!(span2div).to_s
+  #    # => "<div>foo</div><p>bar</p>"
+  #
+  #  or
+  #
+  #    unsafe_html = "ohai! <div>div is safe</div> <script>but script is not</script>"
+  #    Loofah.html5_fragment(unsafe_html).scrub!(:strip).to_s
+  #    # => "ohai! <div>div is safe</div> "
+  #
+  #  Note that this method is called implicitly from the shortcuts Loofah.scrub_html5_fragment et
+  #  al.
+  #
+  #  Please see Scrubber for more information on implementation and traversal, and README.rdoc for
+  #  more example usage.
+  #
+  module ScrubBehavior
+    module Node # :nodoc:
+      def scrub!(scrubber)
+        #
+        #  yes. this should be three separate methods. but nokogiri decorates (or not) based on
+        #  whether the module name has already been included. and since documents get decorated just
+        #  like their constituent nodes, we need to jam all the logic into a single module.
+        #
+        scrubber = ScrubBehavior.resolve_scrubber(scrubber)
+        case self
+        when Nokogiri::XML::Document
+          scrubber.traverse(root) if root
+        when Nokogiri::XML::DocumentFragment
+          children.scrub!(scrubber)
+        else
+          scrubber.traverse(self)
+        end
+        self
+      end
+    end
+
+    module NodeSet # :nodoc:
+      def scrub!(scrubber)
+        each { |node| node.scrub!(scrubber) }
+        self
+      end
+    end
+
+    class << self
+      def resolve_scrubber(scrubber) # :nodoc:
+        scrubber = Scrubbers::MAP[scrubber].new if Scrubbers::MAP[scrubber]
+        unless scrubber.is_a?(Loofah::Scrubber)
+          raise Loofah::ScrubberNotFound, "not a Scrubber or a scrubber name: #{scrubber.inspect}"
+        end
+
+        scrubber
+      end
+    end
+  end
+
+  #
+  #  Overrides +text+ in Document and DocumentFragment classes, and mixes in +to_text+.
+  #
+  module TextBehavior
+    #
+    #  Returns a plain-text version of the markup contained by the document, with HTML entities
+    #  encoded.
+    #
+    #  This method is significantly faster than #to_text, but isn't clever about whitespace around
+    #  block elements.
+    #
+    #    Loofah.html5_document("<h1>Title</h1><div>Content</div>").text
+    #    # => "TitleContent"
+    #
+    #  By default, the returned text will have HTML entities escaped. If you want unescaped
+    #  entities, and you understand that the result is unsafe to render in a browser, then you can
+    #  pass an argument as shown:
+    #
+    #    frag = Loofah.html5_fragment("&lt;script&gt;alert('EVIL');&lt;/script&gt;")
+    #    # ok for browser:
+    #    frag.text                                 # => "&lt;script&gt;alert('EVIL');&lt;/script&gt;"
+    #    # decidedly not ok for browser:
+    #    frag.text(:encode_special_chars => false) # => "<script>alert('EVIL');</script>"
+    #
+    def text(options = {})
+      result = if serialize_root
+        serialize_root.children.reject(&:comment?).map(&:inner_text).join("")
+      else
+        ""
+      end
+      if options[:encode_special_chars] == false
+        result # possibly dangerous if rendered in a browser
+      else
+        encode_special_chars(result)
+      end
+    end
+
+    alias_method :inner_text, :text
+    alias_method :to_str, :text
+
+    #
+    #  Returns a plain-text version of the markup contained by the fragment, with HTML entities
+    #  encoded.
+    #
+    #  This method is slower than #text, but is clever about whitespace around block elements and
+    #  line break elements.
+    #
+    #    Loofah.html5_document("<h1>Title</h1><div>Content<br>Next line</div>").to_text
+    #    # => "\nTitle\n\nContent\nNext line\n"
+    #
+    def to_text(options = {})
+      Loofah.remove_extraneous_whitespace(dup.scrub!(:newline_block_elements).text(options))
+    end
+  end
+
+  module DocumentDecorator # :nodoc:
+    def initialize(*args, &block)
+      super
+      decorators(Nokogiri::XML::Node) << ScrubBehavior::Node
+      decorators(Nokogiri::XML::NodeSet) << ScrubBehavior::NodeSet
+    end
+  end
+
+  module HtmlDocumentBehavior # :nodoc:
+    module ClassMethods
+      def parse(*args, &block)
+        remove_comments_before_html_element(super)
+      end
+
+      private
+
+      # remove comments that exist outside of the HTML element.
+      #
+      # these comments are allowed by the HTML spec:
+      #
+      #    https://www.w3.org/TR/html401/struct/global.html#h-7.1
+      #
+      # but are not scrubbed by Loofah because these nodes don't meet
+      # the contract that scrubbers expect of a node (e.g., it can be
+      # replaced, sibling and children nodes can be created).
+      def remove_comments_before_html_element(doc)
+        doc.children.each do |child|
+          child.unlink if child.comment?
+        end
+        doc
+      end
+    end
+
+    class << self
+      def included(base)
+        base.extend(ClassMethods)
+      end
+    end
+
+    def serialize_root
+      at_xpath("/html/body")
+    end
+  end
+
+  module HtmlFragmentBehavior # :nodoc:
+    module ClassMethods
+      def parse(tags, encoding = nil)
+        doc = document_klass.new
+
+        encoding ||= tags.respond_to?(:encoding) ? tags.encoding.name : "UTF-8"
+        doc.encoding = encoding
+
+        new(doc, tags)
+      end
+
+      def document_klass
+        @document_klass ||= if Loofah.html5_support? && self == Loofah::HTML5::DocumentFragment
+          Loofah::HTML5::Document
+        elsif self == Loofah::HTML4::DocumentFragment
+          Loofah::HTML4::Document
+        else
+          raise ArgumentError, "unexpected class: #{self}"
+        end
+      end
+    end
+
+    class << self
+      def included(base)
+        base.extend(ClassMethods)
+      end
+    end
+
+    def to_s
+      serialize_root.children.to_s
+    end
+
+    alias_method :serialize, :to_s
+
+    def serialize_root
+      at_xpath("./body") || self
+    end
+  end
+end