loofah 0.2.2 → 0.3.0

data/CHANGELOG.rdoc

@@ -1,5 +1,19 @@
 = Changelog
 
+== 0.3.0 (2009-10-06)
+
+Enhancements:
+
+* New ActiveRecord extension `xss_foliate`, a drop-in replacement for xss_terminate[http://github.com/look/xss_terminate/tree/master].
+* Replacement methods for Rails's helpers, Loofah::Rails.sanitize and Loofah::Rails.strip_tags.
+* Official support (and test coverage) for Rails versions 2.3, 2.2, 2.1, 2.0 and 1.2.
+
+Deprecations:
+
+* The methods strip_tags, whitewash, whitewash_document, sanitize, and
+  sanitize_document have been deprecated. See DEPRECATED.rdoc for
+  details on the equivalent calls with the post-0.2 API.
+
 == 0.2.2 (2009-09-30)
 
 Enhancements:

data/DEPRECATED.rdoc

@@ -0,0 +1,12 @@
+= Deprecations
+
+In Loofah 0.3.0, some methods have been deprecated. The following
+lists the equivalent calls with the post-0.2 API:
+
+* <tt>strip_tags(string_or_io)</tt> is now <tt>scrub_document(string_or_io, :prune).text</tt>
+* <tt>whitewash(string_or_io)</tt> is now <tt>scrub_fragment(string_or_io, :whitewash).to_s</tt>
+* <tt>whitewash_document(string_or_io)</tt> is now <tt>scrub_document(string_or_io, :whitewash).to_s</tt>
+* <tt>sanitize(string_or_io)</tt> is now <tt>scrub_fragment(string_or_io, :escape).to_xml</tt>
+* <tt>sanitize_document(string_or_io)</tt> is now <tt>scrub_document(string_or_io, :escape).to_xml</tt>
+
+Have a nice day.

data/Manifest.txt

@@ -1,4 +1,5 @@
 CHANGELOG.rdoc
+DEPRECATED.rdoc
 MIT-LICENSE.txt
 Manifest.txt
 README.rdoc
@@ -11,19 +12,19 @@ benchmark/www.slashdot.com.html
 init.rb
 lib/loofah.rb
 lib/loofah/active_record.rb
-lib/loofah/deprecated.rb
+lib/loofah/helpers.rb
 lib/loofah/html/document.rb
 lib/loofah/html/document_fragment.rb
 lib/loofah/html5/scrub.rb
 lib/loofah/html5/whitelist.rb
 lib/loofah/scrubber.rb
+lib/loofah/xss_foliate.rb
 test/helper.rb
-test/html5/test_deprecated_sanitizer.rb
 test/html5/test_sanitizer.rb
 test/html5/testdata/tests1.dat
 test/test_active_record.rb
+test/test_ad_hoc.rb
 test/test_api.rb
-test/test_deprecated_basic.rb
-test/test_microsofty.rb
+test/test_helpers.rb
 test/test_scrubber.rb
-test/test_strip_tags.rb
+test/test_xss_foliate.rb

data/README.rdoc

@@ -1,34 +1,41 @@
 = Loofah
 
-* http://loofah.rubyforge.org/
+* http://loofah.rubyforge.org
 * http://rubyforge.org/projects/loofah
 * http://github.com/flavorjones/loofah
 
 == DESCRIPTION
 
-Loofah is an HTML sanitizer. It will *always* fix broken markup, but
+Loofah is an HTML sanitizer. It will always fix broken markup, but
 can also sanitize unsafe tags in a few different ways, and transform
 the markup for storage or display.
 
 It's built on top of Nokogiri and libxml2, so it's fast. And it uses
 html5lib's whitelist, so it most likely won't make your codes less
-secure.
+secure. \*
 
-(These statements have not been evaluated by Internet Experts.)
-
-This library was formerly known as Dryopteris.
+\* These statements have not been evaluated by Netexperts.
 
 == FEATURES
 
-* Strip unsafe tags, leaving behind only the inner text.
-* Prune unsafe tags and their subtrees, removing all traces that they ever existed.
-* Escape unsafe tags and their subtrees, leaving behind lots of <tt>&lt;</tt> and <tt>&gt;</tt> entities.
-* Whitewash the markup, removing all attributes and namespaced nodes.
+* _Strip_ unsafe tags, leaving behind only the inner text.
+* _Prune_ unsafe tags and their subtrees, removing all traces that they ever existed.
+* _Escape_ unsafe tags and their subtrees, leaving behind lots of <tt>&lt;</tt> and <tt>&gt;</tt> entities.
+* _Whitewash_ the markup, removing all attributes and namespaced nodes.
 * Format the markup as plain text.
-* ActiveRecord extension.
-* 99 44/100 % Tenderlove-free!
+* Replacements for Rails's +strip_tags+ and +sanitize+ helper methods.
+* TWO! Count them, TWO! ActiveRecord extensions:
+  * Loofah::XssFoliate (an XssTerminate[http://github.com/look/xss_terminate/tree/master] drop-in replacement) is an *opt-out* sanitizer; by default all models and attributes are sanitized.
+  * Loofah::ActiveRecordExtension is an *opt-in* sanitizer; you must explicitly declare attributes to be sanitized.
+* 99 44/100 % pure
+
+== COMPARE AND CONTRAST
 
-Here is a speed test comparing Loofah to other commonly-used sanitization libraries:
+Loofah is the only ruby XSS/sanitization library that guarantees
+well-formed and valid markup.
+
+Also, it's pretty fast. Here is a benchmark comparing Loofah to other
+commonly-used libraries:
 
 * http://gist.github.com/170193
 
@@ -49,7 +56,15 @@ OR
   doc.to_s                            # => "ohai! <div>div is safe</div> "
   doc.text                            # => "ohai! div is safe "
 
-=== ACTIVERECORD EXTENSION
+=== ACTIVERECORD EXTENSION \#1: OPT-IN
+
+See Loofah::ActiveRecordExtension for more documentation. The methods
+mixed into ActiveRecord are:
+
+* Loofah::ActiveRecordExtension.html_document
+* Loofah::ActiveRecordExtension.html_fragment
+
+Example:
 
   # config/environment.rb
   Rails::Initializer.run do |config|
@@ -59,7 +74,8 @@ OR
   # db/schema.rb
   create_table "posts" do |t|
     t.string  "title"
-    t.string  "body"
+    t.text    "body"
+    t.string  "author"
   end
 
   # app/model/post.rb
@@ -67,10 +83,52 @@ OR
     html_fragment :body, :scrub => :prune  # scrubs 'body' in a before_validation
   end
 
+=== ACTIVERECORD EXTENSION \#2: OPT-OUT (XSS_TERMINATE DROP-IN REPLACEMENT)
+
+See Loofah::XssFoliate::ClassMethods for more documentation. The methods mixed into ActiveRecord are:
+
+* Loofah::XssFoliate::ClassMethods.xss_foliate
+* Loofah::XssFoliate::ClassMethods.xss_foliated?
+
+If the constant LOOFAH_XSS_FOLIATE_ALL_MODELS is set, then all models
+inheriting from ActiveRecord::Base will sanitize their string and text
+attributes in a before_validate. Otherwise, the xss_foliate method is
+available for opting in to sanitization.
+
+Example:
+
+  # config/environment
+  LOOFAH_XSS_FOLIATE_ALL_MODELS = true
+  Rails::Initializer.run do |config|
+    config.gem "loofah"
+  end
+
+  # db/schema.rb
+  create_table "posts" do |t|
+    t.string  "title"
+    t.text    "body"
+    t.string  "author"
+  end
+
+  # app/model/post.rb
+  class Post < ActiveRecord::Base
+    # by default, 'title', 'body' and 'author' will all be sanitized in a before_validation,
+    # without additional declarations.
+  end
+
+OR
+
+  # app/model/post.rb
+  class Post < ActiveRecord::Base
+    # opt-out and/or modify the sanitization method used
+    xss_foliate :except => [:title, :author], :prune => :body
+  end
+
 == REQUIREMENTS
 
-* ruby 1.8 or 1.9
 * Nokogiri >= 1.3.3
+* ruby 1.8.6, 1.8.7 or 1.9
+* Rails 2.3, 2.2, 2.1, 2.0 or 1.2 (for ActiveRecord extensions)
 
 == INSTALLATION
 
@@ -84,7 +142,7 @@ The bug tracker is available here:
 
 * http://github.com/flavorjones/loofah/issues
 
-You can also try the Nokogiri mailing list:
+For now, we're piggybacking on the Nokogiri mailing list:
 
 * http://groups.google.com/group/nokogiri-talk
 
@@ -92,9 +150,10 @@ And the IRC channel is #nokogiri on freenode.
 
 == RELATED LINKS
 
-* Nokogiri: http://nokogiri.org/
-* libxml2: http://xmlsoft.org/
-* html5lib: http://code.google.com/p/html5lib/
+* Nokogiri: http://nokogiri.org
+* libxml2: http://xmlsoft.org
+* html5lib: http://code.google.com/p/html5lib
+* XssTerminate: http://github.com/look/xss_terminate/tree/master
 
 == AUTHORS
 
@@ -109,6 +168,13 @@ Featuring code contributed by:
 * Paul Dix
 * Josh Nichols
 
+And a big shout-out to Corey Innis for the name, and feedback on the API.
+
+== HISTORICAL NOTE
+
+This library was formerly known as Dryopteris, which was a very bad
+name that nobody could spell properly.
+
 == LICENSE
 
 The MIT License

data/Rakefile

@@ -13,4 +13,42 @@ Hoe.spec "loofah" do
   self.readme_file      = "README.rdoc"
 
   extra_deps << ["nokogiri", ">= 1.3.3"]
+
+  # note: .hoerc should have the following line to omit rails tests and tmp
+  #   exclude: !ruby/regexp /\/tmp\/|\/rails_tests\/|CVS|TAGS|\.(svn|git|DS_Store)/
+end
+
+if File.exist?("rails_test/Rakefile")
+  load "rails_test/Rakefile"
+else
+  task :test do
+    puts "----------"
+    puts "-- NOTE: An additional Rails regression test suite is available in source repository"
+    puts "----------"
+  end
 end
+
+task :redocs => :fix_css
+task :fix_css do
+  better_css = <<-EOT
+.method-description pre {
+  margin: 1em 0 ;
+}
+
+.method-description ul {
+  padding: .5em 0 .5em 2em ;
+}
+
+.method-description p {
+  margin-top: .5em ;
+}
+
+div#main ul {
+  list-style-type: disc ;
+  list-style-position: inside ;
+}
+EOT
+  puts "* fixing css"
+  File.open("doc/rdoc.css", "a") { |f| f.write better_css }
+end
+

data/benchmark/benchmark.rb

@@ -1,49 +1,129 @@
 #!/usr/bin/env ruby
 require "#{File.dirname(__FILE__)}/helper.rb"
 
-BIG_FILE = File.read(File.join(File.dirname(__FILE__), "www.slashdot.com.html"))
-FRAGMENT = File.read(File.join(File.dirname(__FILE__), "fragment.html"))
-SNIPPET = "This is typical form field input in <b>length and content."
-
-def bench(content, ntimes, fragment_p)
-  Benchmark.bm(15) do |x|
-    x.report('Loofah') do
-      ntimes.times do
-        if fragment_p
-          Loofah.scrub_fragment(content, :escape)
-        else
-          Loofah.scrub_document(content, :escape)
-        end
-      end
+def compare_scrub_methods
+  snip = "<div>foo</div><foo>fuxx <b>quux</b></foo><script>i have a chair</script>"
+  puts "starting with: #{snip}"
+  puts
+  puts RailsSanitize.new.sanitize(snip) # => Rails.sanitize / scrub!(:prune).to_s
+  puts Loofah::Helpers.sanitize(snip)
+  puts "--"
+  puts RailsSanitize.new.strip_tags(snip) # => Rails.strip_tags / parse().text
+  puts Loofah::Helpers.strip_tags(snip)
+  puts "--"
+  puts Sanitize.clean(snip, Sanitize::Config::RELAXED) # => scrub!(:strip).to_s
+  puts Loofah.scrub_fragment(snip, :strip).to_s
+  puts "--"
+  puts HTML5libSanitize.new.sanitize(snip) # => scrub!(:escape).to_s
+  puts Loofah.scrub_fragment(snip, :escape).to_s
+  puts "--"
+end
+
+module TestSet
+  def test_set options={}
+    scale = options[:rehearse] ? 10 : 1
+    puts self.class.name
+
+    n = 100 / scale
+    puts "  Large document, #{BIG_FILE.length} bytes (x#{n})"
+    bench BIG_FILE, n, false
+    puts
+
+    n = 1000 / scale
+    puts "  Small fragment, #{FRAGMENT.length} bytes (x#{n})"
+    bench FRAGMENT, n, true
+    puts
+
+    n = 10_000 / scale
+    puts "  Text snippet, #{SNIPPET.length} bytes (x#{n})"
+    bench SNIPPET, n, true
+    puts
+  end
+end
+
+class HeadToHead < Measure
+end
+
+class HeadToHeadRailsSanitize < Measure
+  include TestSet
+  def bench(content, ntimes, fragment_p)
+    clear_measure
+
+    measure "Loofah::Helpers.sanitize", ntimes do
+      Loofah::Helpers.sanitize content
     end
-    
-    x.report('ActionView') do
-      sanitizer = RailsSanitize.new
-      
-      ntimes.times do
-        sanitizer.sanitize(content)
-      end
+
+    sanitizer = RailsSanitize.new
+    measure "ActionView sanitize", ntimes do
+      sanitizer.sanitize(content)
+    end
+  end
+end
+
+class HeadToHeadRailsStripTags < Measure
+  include TestSet
+  def bench(content, ntimes, fragment_p)
+    clear_measure
+
+    measure "Loofah::Helpers.strip_tags", ntimes do
+      Loofah::Helpers.strip_tags content
     end
-    
-    x.report('Sanitize') do
-      ntimes.times do
-        Sanitize.clean(content, Sanitize::Config::RELAXED)
+
+    sanitizer = RailsSanitize.new
+    measure "ActionView strip_tags", ntimes do
+      sanitizer.strip_tags(content)
+    end
+  end
+end
+
+class HeadToHeadSanitizerSanitize < Measure
+  include TestSet
+  def bench(content, ntimes, fragment_p)
+    clear_measure
+
+    measure "Loofah :strip", ntimes do
+      if fragment_p
+        Loofah.scrub_fragment(content, :strip).to_s
+      else
+        Loofah.scrub_document(content, :strip).to_s
       end
     end
-    
-    x.report('HTML5lib') do
-      sanitizer = HTML5libSanitize.new
-      
-      ntimes.times do
-        sanitizer.sanitize(content)
+
+    measure "Sanitize.clean", ntimes do
+      Sanitize.clean(content, Sanitize::Config::RELAXED)
+    end
+  end
+end
+
+class HeadToHeadHtml5LibSanitize < Measure
+  include TestSet
+  def bench(content, ntimes, fragment_p)
+    clear_measure
+
+    measure "Loofah :escape", ntimes do
+      if fragment_p
+        Loofah.scrub_fragment(content, :escape).to_s
+      else
+        Loofah.scrub_document(content, :escape).to_s
       end
     end
+
+    html5_sanitizer = HTML5libSanitize.new
+    measure "HTML5lib.sanitize", ntimes do
+      html5_sanitizer.sanitize(content)
+    end
   end
 end
 
-puts "Large document, #{BIG_FILE.length} bytes (x100)"
-bench BIG_FILE, 100, false
-puts "Small fragment, #{FRAGMENT.length} bytes (x1000)"
-bench FRAGMENT, 1000, true
-puts "Text snippet, #{SNIPPET.length} bytes (x10000)"
-bench SNIPPET, 10000, true
+puts "Nokogiri version: #{Nokogiri::VERSION_INFO.inspect}"
+puts "Loofah version: #{Loofah::VERSION.inspect}"
+
+benches = []
+benches << HeadToHeadRailsSanitize.new
+benches << HeadToHeadRailsStripTags.new
+benches << HeadToHeadSanitizerSanitize.new
+benches << HeadToHeadHtml5LibSanitize.new
+puts "---------- rehearsal ----------"
+benches.each { |bench| bench.test_set :rehearse => true }
+puts "---------- realsies ----------"
+benches.each { |bench| bench.test_set }

data/benchmark/helper.rb

@@ -6,6 +6,7 @@ require 'benchmark'
 require "action_view"
 require "action_controller/vendor/html-scanner"
 require "sanitize"
+require 'hitimes'
 
 class RailsSanitize
   include ActionView::Helpers::SanitizeHelper
@@ -30,3 +31,38 @@ class HTML5libSanitize
     }).to_s
   end
 end
+
+BIG_FILE = File.read(File.join(File.dirname(__FILE__), "www.slashdot.com.html"))
+FRAGMENT = File.read(File.join(File.dirname(__FILE__), "fragment.html"))
+SNIPPET = "This is typical form field input in <b>length and content."
+
+class Measure
+  def initialize
+    clear_measure
+  end
+
+  def clear_measure
+    @first_time = true
+    @baseline = nil
+  end
+
+  def measure(name, ntimes)
+    if @first_time
+      printf "  %-30s %7s  %8s  %5s\n", "", "total", "single", "rel"
+      @first_time = false
+    end
+    timer = Hitimes::TimedMetric.new(name)
+    timer.start
+    ntimes.times do |j|
+      yield
+    end
+    timer.stop
+    if @baseline
+      printf "  %30s %7.3f (%8.6f) %5.2fx\n", timer.name, timer.sum, timer.sum / ntimes, timer.sum / @baseline
+    else
+      @baseline = timer.sum
+      printf "  %30s %7.3f (%8.6f) %5s\n", timer.name, timer.sum, timer.sum / ntimes, "-"
+    end
+    timer.sum
+  end
+end