loofah 0.3.1 → 0.4.0

data/CHANGELOG.rdoc

@@ -1,5 +1,14 @@
 = Changelog
 
+== 0.4.0
+
+Enhancements:
+
+  * Scrubber class introduced, allowing development of custom scrubbers.
+  * Added support for XML documents and fragments.
+  * Added :nofollow HTML scrubber (thanks Luke Melia!)
+  * Built-in scrubbing methods refactored to use Scrubber.
+
 == 0.3.1 (2009-10-12)
 
 Bug fixes:

data/Manifest.txt

@@ -17,14 +17,16 @@ lib/loofah/html/document.rb
 lib/loofah/html/document_fragment.rb
 lib/loofah/html5/scrub.rb
 lib/loofah/html5/whitelist.rb
+lib/loofah/instance_methods.rb
 lib/loofah/scrubber.rb
+lib/loofah/scrubbers.rb
 lib/loofah/xss_foliate.rb
 test/helper.rb
 test/html5/test_sanitizer.rb
-test/html5/testdata/tests1.dat
 test/test_active_record.rb
 test/test_ad_hoc.rb
 test/test_api.rb
 test/test_helpers.rb
 test/test_scrubber.rb
+test/test_scrubbers.rb
 test/test_xss_foliate.rb

data/README.rdoc

@@ -4,139 +4,253 @@
 * http://rubyforge.org/projects/loofah
 * http://github.com/flavorjones/loofah
 
-== DESCRIPTION
+== Description
+
+Loofah is a general library for manipulating HTML/XML documents and
+fragments. It's built on top of Nokogiri and libxml2, so it's fast and
+has a nice API.
+
+Loofah excels at HTML sanitization (XSS prevention). It includes some
+nice HTML sanitizers, which are based on HTML5lib's whitelist, so it
+most likely won't make your codes less secure. (These statements have
+not been evaluated by Netexperts.)
+
+== Features
+
+* Easily write custom scrubbers for HTML/XML leveraging the sweetness of Nokogiri (and HTML5lib's whitelists).
+* Common HTML sanitizing tasks are built-in:
+  * _Strip_ unsafe tags, leaving behind only the inner text.
+  * _Prune_ unsafe tags and their subtrees, removing all traces that they ever existed.
+  * _Escape_ unsafe tags and their subtrees, leaving behind lots of <tt>&lt;</tt> and <tt>&gt;</tt> entities.
+  * _Whitewash_ the markup, removing all attributes and namespaced nodes.
+* Common HTML transformation tasks are built-in:
+  * Add the _nofollow_ attribute to all hyperlinks.
+* Format markup as plain text.
+* Replace Rails's +strip_tags+ and +sanitize+ helper methods.
+* Two ActiveRecord extensions:
+  * Loofah::XssFoliate, an XssTerminate[http://github.com/look/xss_terminate/tree/master] drop-in replacement, is an *opt-out* sanitizer. By default all models and attributes are sanitized.
+  * Loofah::ActiveRecordExtension is an *opt-in* sanitizer. You must explicitly declare attributes to be sanitized.
+
+== Compare and Contrast
+
+Loofah is the only Ruby XSS/sanitization solution that guarantees
+well-formed and valid markup.
 
-Loofah is an HTML sanitizer. It will always fix broken markup, but
-can also sanitize unsafe tags in a few different ways, and transform
-the markup for storage or display.
+Loofah works fine on XML, XHTML and HTML documents.
 
-It's built on top of Nokogiri and libxml2, so it's fast. And it uses
-html5lib's whitelist, so it most likely won't make your codes less
-secure. \*
+Also, it's pretty fast. Here is a benchmark comparing Loofah to other
+commonly-used libraries (ActionView, Sanitize and HTML5lib):
 
-\* These statements have not been evaluated by Netexperts.
+* http://gist.github.com/170193
 
-== FEATURES
+Lastly, Loofah is extensible. It's super-easy to write your own custom
+scrubbers for whatever document manipulation you need. You don't like
+the built-in scrubbers? Build your own, like a boss.
 
-* _Strip_ unsafe tags, leaving behind only the inner text.
-* _Prune_ unsafe tags and their subtrees, removing all traces that they ever existed.
-* _Escape_ unsafe tags and their subtrees, leaving behind lots of <tt>&lt;</tt> and <tt>&gt;</tt> entities.
-* _Whitewash_ the markup, removing all attributes and namespaced nodes.
-* Format the markup as plain text.
-* Replacements for Rails's +strip_tags+ and +sanitize+ helper methods.
-* TWO! Count them, TWO! ActiveRecord extensions:
-  * Loofah::XssFoliate (an XssTerminate[http://github.com/look/xss_terminate/tree/master] drop-in replacement) is an *opt-out* sanitizer; by default all models and attributes are sanitized.
-  * Loofah::ActiveRecordExtension is an *opt-in* sanitizer; you must explicitly declare attributes to be sanitized.
-* 99 44/100 % pure
+== The Basics
 
-== COMPARE AND CONTRAST
+Loofah wraps Nokogiri[http://nokogiri.org] in a loving
+embrace. Nokogiri[http://nokogiri.org] is an excellent HTML/XML
+parser. If you don't know how Nokogiri[http://nokogiri.org] works, you
+might want to pause for a moment and go check it out. I'll wait.
 
-Loofah is the only ruby XSS/sanitization library that guarantees
-well-formed and valid markup.
+Loofah presents the following classes:
 
-Also, it's pretty fast. Here is a benchmark comparing Loofah to other
-commonly-used libraries:
+* Loofah::HTML::Document and Loofah::HTML::DocumentFragment
+* Loofah::XML::Document and Loofah::XML::DocumentFragment
+* Loofah::Scrubber
 
-* http://gist.github.com/170193
+The documents and fragments are subclasses of the similar Nokogiri classes.
 
-== SYNOPSIS
+The Scrubber represents the document manipulation, either by wrapping
+a block,
 
-For a full explanation, see the documentation for Loofah.
+  span2div = Loofah::Scrubber.new do |node|
+    node.name = "div" if node.name == "span"
+  end
 
-  require 'loofah'
+or by implementing a method.
 
-  unsafe_html = "ohai! <div>a div is safe</div> <script>but script is not</script>"
+=== Side Note: Fragments vs Documents
 
-  Loofah.scrub_fragment(unsafe_html, :prune).to_s  # => "ohai! <div>div is safe</div> "
+Generally speaking, unless you expect to have a DOCTYPE and a single
+root node, you don't have a *document*, you have a *fragment*. For
+HTML, another rule of thumb is that *documents* have \&lt;html\&gt;
+and \&lt;body\&gt; tags, and *fragments* usually do not.
 
-OR
+HTML fragments should be parsed with Loofah.fragment. Loofah won't
+wrap the result in +html+ and +body+ tags, won't add a DOCTYPE
+declaration, and will ignore +head+ elements.
 
-  doc = Loofah.fragment(unsafe_html)  # returns a Nokogiri document ...
-  doc.scrub!(:prune)                  # ... with one extra method
-  doc.to_s                            # => "ohai! <div>div is safe</div> "
-  doc.text                            # => "ohai! div is safe "
+XML fragments should be parsed with Loofah.xml_fragment. Loofah won't
+add a DOCTYPE declaration and will allow multiple root nodes.
 
-=== ACTIVERECORD EXTENSION \#1: OPT-IN
+HTML documents should be parsed with Loofah.document, which will add
+the DOCTYPE declaration, and properly handle +head+ and +body+
+elements.
 
-See Loofah::ActiveRecordExtension for more documentation. The methods
-mixed into ActiveRecord are:
+XML documents should be parsed with Loofah.xml_document. Loofah will
+make sure there's a DOCTYPE declaration and a single root node.
 
-* Loofah::ActiveRecordExtension.html_document
-* Loofah::ActiveRecordExtension.html_fragment
+=== Loofah::HTML::Document and Loofah::HTML::DocumentFragment
 
-Example:
+These classes are subclasses of Nokogiri::HTML::Document and
+Nokogiri::HTML::DocumentFragment, so you get all the markup
+fixer-uppery and API goodness of Nokogiri.
 
-  # config/environment.rb
-  Rails::Initializer.run do |config|
-    config.gem 'loofah'
-  end
+The module methods Loofah.document and Loofah.fragment will parse an
+HTML document and an HTML fragment, respectively.
 
-  # db/schema.rb
-  create_table "posts" do |t|
-    t.string  "title"
-    t.text    "body"
-    t.string  "author"
-  end
+  Loofah.document(unsafe_html).is_a?(Nokogiri::HTML::Document)         # => true
+  Loofah.fragment(unsafe_html).is_a?(Nokogiri::HTML::DocumentFragment) # => true
 
-  # app/model/post.rb
-  class Post < ActiveRecord::Base
-    html_fragment :body, :scrub => :prune  # scrubs 'body' in a before_validation
-  end
+Loofah injects a +scrub!+ method, which takes either a symbol (for
+built-in scrubbers) or a Loofah::Scrubber object (for custom
+scrubbers), and modifies the document in-place.
 
-=== ACTIVERECORD EXTENSION \#2: OPT-OUT (XSS_TERMINATE DROP-IN REPLACEMENT)
+Loofah overrides +to_s+ to return HTML:
 
-See Loofah::XssFoliate::ClassMethods for more documentation. The methods mixed into ActiveRecord are:
+  unsafe_html = "ohai! <div>div is safe</div> <script>but script is not</script>"
 
-* Loofah::XssFoliate::ClassMethods.xss_foliate
-* Loofah::XssFoliate::ClassMethods.xss_foliated?
+  doc = Loofah.fragment(unsafe_html).scrub!(:strip)
+  doc.to_s    # => "ohai! <div>div is safe</div> "
 
-If the constant LOOFAH_XSS_FOLIATE_ALL_MODELS is set, then all models
-inheriting from ActiveRecord::Base will sanitize their string and text
-attributes in a before_validate. Otherwise, the xss_foliate method is
-available for opting in to sanitization.
+and +text+ to return plain text:
 
-Example:
+  doc.text    # => "ohai! div is safe "
 
-  # config/environment
-  LOOFAH_XSS_FOLIATE_ALL_MODELS = true
-  Rails::Initializer.run do |config|
-    config.gem "loofah"
-  end
+=== Loofah::XML::Document and Loofah::XML::DocumentFragment
+
+These classes are subclasses of Nokogiri::XML::Document and
+Nokogiri::XML::DocumentFragment, so you get all the markup
+fixer-uppery and API goodness of Nokogiri.
+
+The module methods Loofah.xml_document and Loofah.xml_fragment will
+parse an XML document and an XML fragment, respectively.
+
+  Loofah.xml_document(bad_xml).is_a?(Nokogiri::XML::Document)         # => true
+  Loofah.xml_fragment(bad_xml).is_a?(Nokogiri::XML::DocumentFragment) # => true
+
+=== Loofah::Scrubber
 
-  # db/schema.rb
-  create_table "posts" do |t|
-    t.string  "title"
-    t.text    "body"
-    t.string  "author"
+A Scrubber wraps up a block (or method) that is run on a document node:
+
+  # change all <span> tags to <div> tags
+  span2div = Loofah::Scrubber.new do |node|
+    node.name = "div" if node.name == "span"
   end
 
-  # app/model/post.rb
-  class Post < ActiveRecord::Base
-    # by default, 'title', 'body' and 'author' will all be sanitized in a before_validation,
-    # without additional declarations.
+This can then be run on a document:
+
+  Loofah.fragment("<span>foo</span><p>bar</p>").scrub!(span2div).to_s
+  # => "<div>foo</div><p>bar</p>"
+
+Scrubbers can be run on a document in either a top-down traversal (the
+default) or bottom-up. Top-down scrubbers can optionally return
+Scrubber::STOP to terminate the traversal of a subtree. Read below and
+in the Loofah::Scrubber class for more detailed usage.
+
+Here's an XML example:
+
+  # remove all <employee> tags that have a "deceased" attribute set to true
+  bring_out_your_dead = Loofah::Scrubber.new do |node|
+    if node.name == "employee" and node["deceased"] == "true"
+      node.remove
+      Loofah::Scrubber::STOP # don't bother with the rest of the subtree
+    end
   end
+  Loofah.xml_document(File.read('plague.xml')).scrub!(bring_out_your_dead)
 
-OR
+=== Built-In HTML Scrubbers
+
+Loofah comes with a set of sanitizing scrubbers that use HTML5lib's
+whitelist algorithm:
+
+  doc.scrub!(:strip)       # replaces unknown/unsafe tags with their inner text
+  doc.scrub!(:prune)       #  removes unknown/unsafe tags and their children
+  doc.scrub!(:escape)      #  escapes unknown/unsafe tags, like this: &lt;script&gt;
+  doc.scrub!(:whitewash)   #  removes unknown/unsafe/namespaced tags and their children,
+                           #          and strips all node attributes
+
+Loofah also comes with some common transformation tasks: 
+
+  doc.scrub!(:nofollow)    #     adds rel="nofollow" attribute to links
+
+See Loofah::Scrubbers for more details and example usage.
+
+=== Chaining Scrubbers
+
+You can chain scrubbers:
+
+  Loofah.fragment("<span>hello</span> <script>alert('OHAI')</script>") \
+        .scrub!(:prune) \
+        .scrub!(span2div).to_s
+  # => "<div>hello</div> "
+
+=== Shorthand
+
+The class methods Loofah.scrub_fragment and Loofah.scrub_document are
+shorthand.
+
+  Loofah.scrub_fragment(unsafe_html, :prune)
+  Loofah.scrub_document(unsafe_html, :prune)
+  Loofah.scrub_xml_fragment(bad_xml, custom_scrubber)
+  Loofah.scrub_xml_document(bad_xml, custom_scrubber)
+
+are the same thing as (and arguably semantically clearer than):
+
+  Loofah.fragment(unsafe_html).scrub!(:prune)
+  Loofah.document(unsafe_html).scrub!(:prune)
+  Loofah.xml_fragment(bad_xml).scrub!(custom_scrubber)
+  Loofah.xml_document(bad_xml).scrub!(custom_scrubber)
+
+=== ActiveRecord Extension \#1: Opt-In
+
+See Loofah::ActiveRecordExtension for full documentation. The methods
+mixed into ActiveRecord are:
+
+* Loofah::ActiveRecordExtension.html_document
+* Loofah::ActiveRecordExtension.html_fragment
+
+which are used to declare how specific string and text attributes
+should be scrubbed at +before_validation+.
 
   # app/model/post.rb
   class Post < ActiveRecord::Base
-    # opt-out and/or modify the sanitization method used
-    xss_foliate :except => [:title, :author], :prune => :body
+    html_fragment :body, :scrub => :prune  # scrubs 'body' at before_validation
   end
 
-== REQUIREMENTS
+=== ActiveRecord Extension \#2: Opt-Out
+
+See Loofah::XssFoliate::ClassMethods for more documentation. The methods mixed into ActiveRecord are:
+
+* Loofah::XssFoliate::ClassMethods.xss_foliate
+* Loofah::XssFoliate::ClassMethods.xss_foliated?
+
+which are used to declare how specific string and text attributes
+should be scrubbed at +before_validation+.
+
+Attributes are stripped by default, unless another scrubber is
+specified or the attribute is present in an +:except+ clause.
+
+=== View Helpers
+
+Loofah has two "view helpers": Loofah::Helpers.sanitize and
+Loofah::Helpers.strip_tags, both of which are drop-in replacements for
+the ActionView helpers of the same name.
+
+== Requirements
 
 * Nokogiri >= 1.3.3
-* ruby 1.8.6, 1.8.7 or 1.9
-* Rails 2.3, 2.2, 2.1, 2.0 or 1.2 (for ActiveRecord extensions)
+* Rails 2.3, 2.2, 2.1, 2.0 or 1.2 (if you're using the ActiveRecord extensions)
 
-== INSTALLATION
+== Installation
 
 Unsurprisingly:
 
 * gem install loofah
 
-== SUPPORT
+== Support
 
 The bug tracker is available here:
 
@@ -148,16 +262,16 @@ For now, we're piggybacking on the Nokogiri mailing list:
 
 And the IRC channel is #nokogiri on freenode.
 
-== RELATED LINKS
+== Related Links
 
 * Nokogiri: http://nokogiri.org
 * libxml2: http://xmlsoft.org
 * html5lib: http://code.google.com/p/html5lib
 * XssTerminate: http://github.com/look/xss_terminate/tree/master
 
-== AUTHORS
+== Authors
 
-* {Mike Dalessio}[mailto:mike.dalessio@gmail.com]
+* {Mike Dalessio}[mailto:mike.dalessio@gmail.com] (@flavorjones)
 * {Bryan Helmkamp}[mailto:bryan@brynary.com]
 
 Featuring code contributed by:
@@ -167,18 +281,35 @@ Featuring code contributed by:
 * Josh Owens
 * Paul Dix
 * Josh Nichols
+* Luke Melia
 
 And a big shout-out to Corey Innis for the name, and feedback on the API.
 
-== HISTORICAL NOTE
+== Historical Note
 
 This library was formerly known as Dryopteris, which was a very bad
 name that nobody could spell properly.
 
-== LICENSE
+== License
 
 The MIT License
 
 Copyright (c) 2009 Mike Dalessio, Bryan Helmkamp
 
-See MIT-LICENSE.txt in this directory.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/Rakefile

@@ -13,6 +13,9 @@ Hoe.spec "loofah" do
   self.readme_file      = "README.rdoc"
 
   extra_deps << ["nokogiri", ">= 1.3.3"]
+  extra_dev_deps << ["mocha", ">=0.9"]
+  extra_dev_deps << ["thoughtbot-shoulda", ">=2.10"]
+  extra_dev_deps << ["acts_as_fu", ">=0.0.5"]
 
   # note: .hoerc should have the following line to omit rails tests and tmp
   #   exclude: !ruby/regexp /\/tmp\/|\/rails_tests\/|CVS|TAGS|\.(svn|git|DS_Store)/
@@ -44,10 +47,15 @@ task :fix_css do
   margin-top: .5em ;
 }
 
-div#main ul {
-  list-style-type: disc ;
-  list-style-position: inside ;
+#main ul, div#documentation ul {
+  list-style-type: disc ! IMPORTANT ;
+  list-style-position: inside ! IMPORTANT ;
 }
+
+h2 + ul {
+  margin-top: 1em;
+}
+
 EOT
   puts "* fixing css"
   File.open("doc/rdoc.css", "a") { |f| f.write better_css }