sanitize 4.6.4 → 5.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ac9e4e6beb6025350578007019bf48458a64b82ef26cd11e4547aee35b72625c
-  data.tar.gz: 000a0cd02b3a2690589f042f3d27fb0dd8fe34d150cc1ba073793dcfb2eb0a92
+  metadata.gz: 4f01a992746ecc3f28e9c1fd14c08c99456fb98a59c0b7ba6a8c6f01d0ab07cf
+  data.tar.gz: 4f379538b26db4d239078ea7e54fea3b106e7801d093ed7407e9b71282f6c4d3
 SHA512:
-  metadata.gz: 127b1656fa575c9ba793db8d6026a37b240a884172b18b05bb2776c80b2cda6fc2982938e3a98272b46d5741b05c8c05f3238ebb61e439a9f7ded36615854c4d
-  data.tar.gz: adb7c8b6bf29118b82094dd2cb7109dfaa6286ca57b73e98ec457b8eb433e91c76efd987fe2c6d3b6dac1ed554331f7d067240c4a96e7522d7122da5c5b925b1
+  metadata.gz: 52d96c5f73eea8d738fe23d816d5aec856f9f37ca37cf88d88d385fcffbf242605d13494ab531b517af7bdea44bfae2569f27bc2d5fb005dbeee85a54211d674
+  data.tar.gz: 897e95c05448509cfeb455bb4ec156ff7557495987e1d058ff63b888f9c0069a821a9b3684e0fe0463f78e4f28faf9fe2089760ad59bbbd1b5a5390fe9632154

data/HISTORY.md

@@ -1,5 +1,94 @@
 # Sanitize History
 
+## 5.2.0 (2020-06-06)
+
+### Changes
+
+* The term "whitelist" has been replaced with "allowlist" throughout Sanitize's
+  source and documentation.
+
+  While the etymology of "whitelist" may not be explicitly racist in origin or
+  intent, there are inherent racial connotations in the implication that white
+  is good and black (as in "blacklist") is not.
+
+  This is a change I should have made long ago, and I apologize for not making
+  it sooner.
+
+* In transformer input, the `:is_whitelisted` and `:node_whitelist` keys are now
+  deprecated. New `:is_allowlisted` and `:node_allowlist` keys have been added.
+  The old keys will continue to work in order to avoid breaking existing code,
+  but they are no longer documented and may be removed in a future semver major
+  release.
+
+## 5.1.0 (2019-09-07)
+
+### Features
+
+* Added a `:parser_options` config hash, which makes it possible to pass custom
+  parsing options to Nokogumbo. [@austin-wang - #194][194]
+
+### Bug Fixes
+
+* Non-characters and non-whitespace control characters are now stripped from
+  HTML input before parsing to comply with the HTML Standard's [preprocessing
+  guidelines][html-preprocessing]. Prior to this Sanitize had adhered to [older
+  W3C guidelines][unicode-xml] that have since been withdrawn. [#179][179]
+
+[179]:https://github.com/rgrove/sanitize/issues/179
+[194]:https://github.com/rgrove/sanitize/pull/194
+[html-preprocessing]:https://html.spec.whatwg.org/multipage/parsing.html#preprocessing-the-input-stream
+[unicode-xml]:https://www.w3.org/TR/unicode-xml/
+
+## 5.0.0 (2018-10-14)
+
+For most users, upgrading from 4.x shouldn't require any changes. However, the
+minimum required Ruby version has changed, and Sanitize 5.x's HTML output may
+differ in some small ways from 4.x's output. If this matters to you, please
+review the changes below carefully.
+
+### Potentially Breaking Changes
+
+* Ruby 2.3.0 is now the oldest officially supported Ruby version. Sanitize may
+  work in older 2.x Rubies, but they aren't actively tested. Sanitize definitely
+  no longer works in Ruby 1.9.x.
+
+* Upgraded to Nokogumbo 2.x, which fixes various bugs and adds
+  standard-compliant HTML serialization. [@stevecheckoway - #189][189]
+
+* Children of the following elements are now removed by default when these
+  elements are removed, rather than being preserved and escaped:
+
+  - `iframe`
+  - `noembed`
+  - `noframes`
+  - `noscript`
+  - `script`
+  - `style`
+
+* Children of allowlisted `iframe` elements are now always removed. In modern
+  HTML, `iframe` elements should never have children. In HTML 4 and earlier
+  `iframe` elements were allowed to contain fallback content for legacy
+  browsers, but it's been almost two decades since that was useful.
+
+* Fixed a bug that caused `:remove_contents` to behave as if it were set to
+  `true` when it was actually an Array.
+
+[189]:https://github.com/rgrove/sanitize/pull/189
+
+## 4.6.6 (2018-07-23)
+
+* Improved performance and memory usage by optimizing `Sanitize#transform_node!`
+  [@stanhu - #183][183]
+
+[183]:https://github.com/rgrove/sanitize/pull/183
+
+## 4.6.5 (2018-05-16)
+
+* Improved performance slightly by tweaking the order of built-in transformers.
+  [@rafbm - #180][180]
+
+[180]:https://github.com/rgrove/sanitize/pull/180
+
 ## 4.6.4 (2018-03-20)
 
 * Fixed: A change introduced in 4.6.2 broke certain transformers that relied on
@@ -15,7 +104,7 @@
 
   When Sanitize <= 4.6.2 is used in combination with libxml2 >= 2.9.2, a
   specially crafted HTML fragment can cause libxml2 to generate improperly
-  escaped output, allowing non-whitelisted attributes to be used on whitelisted
+  escaped output, allowing non-allowlisted attributes to be used on allowlisted
   elements.
 
   Sanitize now performs additional escaping on affected attributes to prevent
@@ -59,7 +148,7 @@
 
 ## 4.4.0 (2016-09-29)
 
-* Added `srcset` to the attribute whitelist for `img` elements in the relaxed
+* Added `srcset` to the attribute allowlist for `img` elements in the relaxed
   config. [@ejtttje - #156][156]
 
 [156]:https://github.com/rgrove/sanitize/pull/156
@@ -180,7 +269,7 @@
 ## 3.0.4 (2014-12-12)
 
 * Fixed: Harmless whitespace preceding a URL protocol (such as " http://")
-  caused the URL to be removed even when the protocol was whitelisted.
+  caused the URL to be removed even when the protocol was allowlisted.
   [@benubois - #126][126]
 
 [126]:https://github.com/rgrove/sanitize/pull/126
@@ -189,7 +278,7 @@
 ## 3.0.3 (2014-10-29)
 
 * Fixed: Some CSS selectors weren't parsed correctly inside the body of a
-  `@media` block, causing them to be removed even when whitelist rules should
+  `@media` block, causing them to be removed even when allowlist rules should
   have allowed them to remain. [#121][121]
 
 [121]:https://github.com/rgrove/sanitize/issues/121
@@ -254,7 +343,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 * The `clean_node!` method was renamed to `node!`.
 
 * The `document` method now raises a `Sanitize::Error` if the `<html>` element
-  isn't whitelisted, rather than a `RuntimeError`. This error is also now raised
+  isn't allowlisted, rather than a `RuntimeError`. This error is also now raised
   regardless of the `:remove_contents` config setting.
 
 * The `:output` config has been removed. Output is now always HTML, not XHTML.
@@ -265,7 +354,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 
 * Added advanced CSS sanitization support using [Crass][crass], which is fully
   compliant with the CSS Syntax Module Level 3 parsing spec. The contents of
-  whitelisted `<style>` elements and `style` attributes in HTML will be
+  allowlisted `<style>` elements and `style` attributes in HTML will be
   sanitized as CSS, or you can use the `Sanitize::CSS` class to manually
   sanitize CSS stylesheets or properties.
 
@@ -310,9 +399,29 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 [n1008]:https://github.com/sparklemotion/nokogiri/issues/1008
 
 
+## 2.1.1 (2018-09-30)
+
+* [CVE-2018-3740][176]: Fixed an HTML injection vulnerability that could allow
+  XSS (backported from Sanitize 4.6.3). [@dometto - #188][188]
+
+  When Sanitize <= 2.1.0 is used in combination with libxml2 >= 2.9.2, a
+  specially crafted HTML fragment can cause libxml2 to generate improperly
+  escaped output, allowing non-allowlisted attributes to be used on allowlisted
+  elements.
+
+  Sanitize now performs additional escaping on affected attributes to prevent
+  this.
+
+  Many thanks to the Shopify Application Security Team for responsibly reporting
+  this issue.
+
+[176]:https://github.com/rgrove/sanitize/issues/176
+[188]:https://github.com/rgrove/sanitize/pull/188
+
+
 ## 2.1.0 (2014-01-13)
 
-* Added support for whitelisting arbitrary HTML5 `data-*` attributes. Use the
+* Added support for allowlisting arbitrary HTML5 `data-*` attributes. Use the
   symbol `:data` instead of an attribute name in the `:attributes` config to
   indicate that arbitrary data attributes should be allowed on an element.
 
@@ -393,12 +502,12 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
   the default depth-first mode.
 
 * Added the `abbr`, `dfn`, `kbd`, `mark`, `s`, `samp`, `time`, and `var`
-  elements to the whitelists for the basic and relaxed configs.
+  elements to the allowlists for the basic and relaxed configs.
 
 * Added the `bdo`, `del`, `figcaption`, `figure`, `hgroup`, `ins`, `rp`, `rt`,
-  `ruby`, and `wbr` elements to the whitelist for the relaxed config.
+  `ruby`, and `wbr` elements to the allowlist for the relaxed config.
 
-* The `dir`, `lang`, and `title` attributes are now whitelisted for all
+* The `dir`, `lang`, and `title` attributes are now allowlisted for all
   elements in the relaxed config.
 
 * Bumped minimum Nokogiri version to 1.4.4 to avoid a bug in 1.4.2+
@@ -409,7 +518,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 ## 1.2.1 (2010-04-20)
 
 * Added a `:remove_contents` config setting. If set to `true`, Sanitize will
-  remove the contents of all non-whitelisted elements in addition to the
+  remove the contents of all non-allowlisted elements in addition to the
   elements themselves. If set to an array of element names, Sanitize will
   remove the contents of only those elements (when filtered), and leave the
   contents of other filtered elements. [Thanks to Rafael Souza for the array
@@ -437,7 +546,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 * Added `Sanitize.clean_node!`, which sanitizes a `Nokogiri::XML::Node` and
   all its children.
 
-* Added elements `<h1>` through `<h6>` to the Relaxed whitelist. [Suggested by
+* Added elements `<h1>` through `<h6>` to the Relaxed allowlist. [Suggested by
   David Reese]
 
 
@@ -457,7 +566,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 
 * Added a workaround for an Hpricot bug that prevents attribute names from
   being downcased in recent versions of Hpricot. This was exploitable to
-  prevent non-whitelisted protocols from being cleaned. [Reported by Ben
+  prevent non-allowlisted protocols from being cleaned. [Reported by Ben
   Wanicur]
 
 
@@ -487,7 +596,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 
 ## 1.0.5 (2009-02-05)
 
-* Fixed a bug introduced in version 1.0.3 that prevented non-whitelisted
+* Fixed a bug introduced in version 1.0.3 that prevented non-allowlisted
   protocols from being cleaned when relative URLs were allowed. [Reported by
   Dev Purkayastha]
 
@@ -497,7 +606,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 
 ## 1.0.4 (2009-01-16)
 
-* Fixed a bug that made it possible to sneak a non-whitelisted element through
+* Fixed a bug that made it possible to sneak a non-allowlisted element through
   by repeating it several times in a row. All versions of Sanitize prior to
   1.0.4 are vulnerable. [Reported by Cristobal]
 
@@ -505,7 +614,7 @@ Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
 ## 1.0.3 (2009-01-15)
 
 * Fixed a bug whereby incomplete Unicode or hex entities could be used to
-  prevent non-whitelisted protocols from being cleaned. Since IE6 and Opera
+  prevent non-allowlisted protocols from being cleaned. Since IE6 and Opera
   still decode the incomplete entities, users of those browsers may be
   vulnerable to malicious script injection on websites using versions of
   Sanitize prior to 1.0.3.

data/README.md

@@ -1,20 +1,19 @@
 Sanitize
 ========
 
-Sanitize is a whitelist-based HTML and CSS sanitizer. Given a list of acceptable
-elements, attributes, and CSS properties, Sanitize will remove all unacceptable
-HTML and/or CSS from a string.
+Sanitize is an allowlist-based HTML and CSS sanitizer. It removes all HTML
+and/or CSS from a string except the elements, attributes, and properties you
+choose to allow.
 
 Using a simple configuration syntax, you can tell Sanitize to allow certain HTML
 elements, certain attributes within those elements, and even certain URL
-protocols within attributes that contain URLs. You can also whitelist CSS
-properties, @ rules, and URL protocols you wish to allow in elements or
-attributes containing CSS. Any HTML or CSS that you don't explicitly allow will
-be removed.
+protocols within attributes that contain URLs. You can also allow specific CSS
+properties, @ rules, and URL protocols in elements or attributes containing CSS.
+Any HTML or CSS that you don't explicitly allow will be removed.
 
 Sanitize is based on [Google's Gumbo HTML5 parser][gumbo], which parses HTML
 exactly the same way modern browsers do, and [Crass][crass], which parses CSS
-exactly the same way modern browsers do. As long as your whitelist config only
+exactly the same way modern browsers do. As long as your allowlist config only
 allows safe markup and CSS, even the most malformed or malicious input will be
 transformed into safe output.
 
@@ -88,7 +87,7 @@ Sanitize.fragment(html)
 # => 'foo'
 ```
 
-To keep certain elements, add them to the element whitelist.
+To keep certain elements, add them to the element allowlist.
 
 ```ruby
 Sanitize.fragment(html, :elements => ['b'])
@@ -97,7 +96,7 @@ Sanitize.fragment(html, :elements => ['b'])
 
 ### HTML Documents
 
-When sanitizing a document, the `<html>` element must be whitelisted. You can
+When sanitizing a document, the `<html>` element must be allowlisted. You can
 also set `:allow_doctype` to `true` to allow well-formed document type
 definitions.
 
@@ -123,8 +122,8 @@ Sanitize.document(html,
 
 ### CSS in HTML
 
-To sanitize CSS in an HTML fragment or document, first whitelist the `<style>`
-element and/or the `style` attribute. Then whitelist the CSS properties,
+To sanitize CSS in an HTML fragment or document, first allowlist the `<style>`
+element and/or the `style` attribute. Then allowlist the CSS properties,
 @ rules, and URL protocols you wish to allow. You can also choose whether to
 allow CSS comments or browser compatibility hacks.
 
@@ -267,7 +266,7 @@ new copy using `Sanitize::Config.merge()`, like so:
 
 ```ruby
 # Create a customized copy of the Basic config, adding <div> and <table> to the
-# existing whitelisted elements.
+# existing allowlisted elements.
 Sanitize.fragment(html, Sanitize::Config.merge(Sanitize::Config::BASIC,
   :elements        => Sanitize::Config::BASIC[:elements] + ['div', 'table'],
   :remove_contents => true
@@ -395,8 +394,7 @@ Proc.new { |url| url.start_with?("https://fonts.googleapis.com") }
 
 ##### :css => :properties (Array or Set)
 
-Whitelist of CSS property names to allow. Names should be specified in
-lowercase.
+List of CSS property names to allow. Names should be specified in lowercase.
 
 ##### :css => :protocols (Array or Set)
 
@@ -417,6 +415,17 @@ elements not in this array will be removed.
 ]
 ```
 
+#### :parser_options (Hash)
+
+[Parsing options](https://github.com/rubys/nokogumbo/tree/v2.0.1#parsing-options) supplied to `nokogumbo`.
+
+```ruby
+:parser_options => {
+  max_errors: -1,
+  max_tree_depth: -1
+}
+```
+
 #### :protocols (Hash)
 
 URL protocols to allow in specific attributes. If an attribute is listed here
@@ -441,13 +450,13 @@ include the symbol `:relative` in the protocol array:
 
 #### :remove_contents (boolean or Array or Set)
 
-If set to `true`, Sanitize will remove the contents of any non-whitelisted
+If this is `true`, Sanitize will remove the contents of any non-allowlisted
 elements in addition to the elements themselves. By default, Sanitize leaves the
 safe parts of an element's contents behind when the element is removed.
 
-If set to an array of element names, then only the contents of the specified
-elements (when filtered) will be removed, and the contents of all other filtered
-elements will be left behind.
+If this is an Array or Set of element names, then only the contents of the
+specified elements (when filtered) will be removed, and the contents of all
+other filtered elements will be left behind.
 
 The default value is `false`.
 
@@ -474,6 +483,15 @@ children, in which case it will be inserted after those children.
 }
 ```
 
+The default elements with whitespace added before and after are:
+
+```
+address article aside blockquote br dd div dl dt
+footer h1 h2 h3 h4 h5 h6 header hgroup hr li nav
+ol p pre section ul
+
+```
+
 ## Transformers
 
 Transformers allow you to filter and modify HTML nodes using your own custom
@@ -498,33 +516,33 @@ argument a Hash that contains the following items:
 
   * **:config** - The current Sanitize configuration Hash.
 
-  * **:is_whitelisted** - `true` if the current node has been whitelisted by a
+  * **:is_allowlisted** - `true` if the current node has been allowlisted by a
     previous transformer, `false` otherwise. It's generally bad form to remove
-    a node that a previous transformer has whitelisted.
+    a node that a previous transformer has allowlisted.
 
   * **:node** - A `Nokogiri::XML::Node` object representing an HTML node. The
     node may be an element, a text node, a comment, a CDATA node, or a document
     fragment. Use Nokogiri's inspection methods (`element?`, `text?`, etc.) to
     selectively ignore node types you aren't interested in.
 
+  * **:node_allowlist** - Set of `Nokogiri::XML::Node` objects in the current
+    document that have been allowlisted by previous transformers, if any. It's
+    generally bad form to remove a node that a previous transformer has
+    allowlisted.
+
   * **:node_name** - The name of the current HTML node, always lowercase (e.g.
     "div" or "span"). For non-element nodes, the name will be something like
     "text", "comment", "#cdata-section", "#document-fragment", etc.
 
-  * **:node_whitelist** - Set of `Nokogiri::XML::Node` objects in the current
-    document that have been whitelisted by previous transformers, if any. It's
-    generally bad form to remove a node that a previous transformer has
-    whitelisted.
-
 ### Output
 
 A transformer doesn't have to return anything, but may optionally return a Hash,
 which may contain the following items:
 
-  * **:node_whitelist** -  Array or Set of specific Nokogiri::XML::Node objects
-    to add to the document's whitelist, bypassing the current Sanitize config.
-    These specific nodes and all their attributes will be whitelisted, but
-    their children will not be.
+  * **:node_allowlist** -  Array or Set of specific `Nokogiri::XML::Node`
+    objects to add to the document's allowlist, bypassing the current Sanitize
+    config. These specific nodes and all their attributes will be allowlisted,
+    but their children will not be.
 
 If a transformer returns anything other than a Hash, the return value will be
 ignored.
@@ -567,16 +585,16 @@ Transformers have a tremendous amount of power, including the power to
 completely bypass Sanitize's built-in filtering. Be careful! Your safety is in
 your own hands.
 
-### Example: Transformer to whitelist image URLs by domain
+### Example: Transformer to allow image URLs by domain
 
 The following example demonstrates how to remove image elements unless they use
 a relative URL or are hosted on a specific domain. It assumes that the `<img>`
-element and its `src` attribute are already whitelisted.
+element and its `src` attribute are already allowlisted.
 
 ```ruby
 require 'uri'
 
-image_whitelist_transformer = lambda do |env|
+image_allowlist_transformer = lambda do |env|
   # Ignore everything except <img> elements.
   return unless env[:node_name] == 'img'
 
@@ -592,20 +610,20 @@ image_whitelist_transformer = lambda do |env|
 end
 ```
 
-### Example: Transformer to whitelist YouTube video embeds
+### Example: Transformer to allow YouTube video embeds
 
 The following example demonstrates how to create a transformer that will safely
-whitelist valid YouTube video embeds without having to blindly allow other kinds
-of embedded content, which would be the case if you tried to do this by just
-whitelisting all `<iframe>` elements:
+allow valid YouTube video embeds without having to allow other kinds of embedded
+content, which would be the case if you tried to do this by just allowing all
+`<iframe>` elements:
 
 ```ruby
 youtube_transformer = lambda do |env|
   node      = env[:node]
   node_name = env[:node_name]
 
-  # Don't continue if this node is already whitelisted or is not an element.
-  return if env[:is_whitelisted] || !node.element?
+  # Don't continue if this node is already allowlisted or is not an element.
+  return if env[:is_allowlisted] || !node.element?
 
   # Don't continue unless the node is an iframe.
   return unless node_name == 'iframe'
@@ -626,8 +644,8 @@ youtube_transformer = lambda do |env|
 
   # Now that we're sure that this is a valid YouTube embed and that there are
   # no unwanted elements or attributes hidden inside it, we can tell Sanitize
-  # to whitelist the current node.
-  {:node_whitelist => [node]}
+  # to allowlist the current node.
+  {:node_allowlist => [node]}
 end
 
 html = %[