html-pipeline 0.0.4 → 0.0.5

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+# CHANGELOG
+
+## 0.0.5 (unreleased)
+
+  * fix li xss vulnerability in sanitization filter: vmg #31
+  * gemspec cleanup: nbibler #23, jbarnette #24
+  * doc updates: jch #16, pborreli #17, wickedshimmy #18, benubois #19, blackerby #21
+  * loosen gemoji dependency: josh #15
+
+## 0.0.4
+
+  * initial public release

data/README.md

@@ -53,7 +53,9 @@ pipeline = HTML::Pipeline.new [
 result = pipeline.call <<CODE
 This is *great*:
 
-    some_code(:first)
+``` ruby
+some_code(:first)
+```
 
 CODE
 result[:output].to_s
@@ -83,19 +85,108 @@ filter.call
 ## Filters
 
 * `MentionFilter` - replace `@user` mentions with links
-* `AutoLinkFilter` - auto_linking urls in HTML
-* `CamoFilter` - replace http image urls with [camo-fied](https://github.com/github/camo) https versions
+* `AutolinkFilter` - auto_linking urls in HTML
+* `CamoFilter` - replace http image urls with [camo-fied](https://github.com/atmos/camo) https versions
 * `EmailReplyFilter` - util filter for working with emails
 * `EmojiFilter` - everyone loves [emoji](http://www.emoji-cheat-sheet.com/)!
+* `HttpsFilter` - HTML Filter for replacing http github urls with https versions.
 * `ImageMaxWidthFilter` - link to full size image for large images
 * `MarkdownFilter` - convert markdown to html
 * `PlainTextInputFilter` - html escape text and wrap the result in a div
-* `SanitizationFilter` - whitelist santize user markup
+* `SanitizationFilter` - whitelist sanitize user markup
 * `SyntaxHighlightFilter` - code syntax highlighter with [linguist](https://github.com/github/linguist)
 * `TextileFilter` - convert textile to html
 * `TableOfContentsFilter` - anchor headings with name attributes
 
-## Development Setup
+## Examples
+
+We define different pipelines for different parts of our app. Here are a few
+paraphrased snippets to get you started:
+
+```ruby
+# The context hash is how you pass options between different filters.
+# See individual filter source for explanation of options.
+context = {
+  :asset_root => "http://your-domain.com/where/your/images/live/icons",
+  :base_url   => "http://your-domain.com"
+}
+
+# Pipeline providing sanitization and image hijacking but no mention
+# related features.
+SimplePipeline = Pipeline.new [
+  SanitizationFilter,
+  TableOfContentsFilter, # add 'name' anchors to all headers
+  CamoFilter,
+  ImageMaxWidthFilter,
+  SyntaxHighlightFilter,
+  EmojiFilter,
+  AutolinkFilter
+], context, {}
+
+# Pipeline used for user provided content on the web
+MarkdownPipeline = Pipeline.new [
+  MarkdownFilter,
+  SanitizationFilter,
+  CamoFilter,
+  ImageMaxWidthFilter,
+  HttpsFilter,
+  MentionFilter,
+  EmojiFilter,
+  SyntaxHighlightFilter
+], context.merge(:gfm => true), {}  # enable github formatted markdown
+
+
+# Define a pipeline based on another pipeline's filters
+NonGFMMarkdownPipeline = Pipeline.new(MarkdownPipeline.filters,
+  context.merge(:gfm => false), {})
+
+# Pipelines aren't limited to the web. You can use them for email
+# processing also.
+HtmlEmailPipeline = Pipeline.new [
+  ImageMaxWidthFilter
+], {}, {}
+
+# Just emoji.
+EmojiPipeline = Pipeline.new [
+  HTMLInputFilter,
+  EmojiFilter
+], context, {}
+```
+
+## Extending
+To write a custom filter, you need a class with a `call` method that inherits
+from `HTML::Pipeline::Filter`.
+
+For example this filter adds a base url to images that are root relative:
+
+```ruby
+require 'uri'
+
+class RootRelativeFilter < HTML::Pipeline::Filter
+
+  def call
+    doc.search("img").each do |img| 
+      next if img['src'].nil?
+      src = img['src'].strip
+      if src.start_with? '/'
+        img["src"] = URI.join(context[:base_url], src).to_s
+      end
+    end
+    doc
+  end
+
+end
+```
+
+Now this filter can be used in a pipeline:
+
+```ruby
+Pipeline.new [ RootRelativeFilter ], { :base_url => 'http://somehost.com' }
+```
+
+## Development
+
+To see what has changed in recent versions, see the [CHANGELOG](https://github.com/jch/html-pipeline/blob/master/CHANGELOG.md).
 
 ```sh
 bundle
@@ -104,11 +195,11 @@ rake test
 
 ## Contributing
 
-1. Fork it
+1. [Fork it](https://help.github.com/articles/fork-a-repo)
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Added some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+5. Create new [Pull Request](https://help.github.com/articles/using-pull-requests)
 
 
 ## TODO
@@ -126,3 +217,5 @@ rake test
 * [Simon Rozet](mailto:simon@rozet.name)
 * [Vicent Martí](mailto:tanoku@gmail.com)
 * [Risk :danger: Olson](mailto:technoweenie@gmail.com)
+
+Project is a member of the [OSS Manifesto](http://ossmanifesto.org/).

data/html-pipeline.gemspec

@@ -1,9 +1,10 @@
 # -*- encoding: utf-8 -*-
-require File.expand_path('../lib/html/pipeline/version', __FILE__)
+require File.expand_path("../lib/html/pipeline/version", __FILE__)
 
 Gem::Specification.new do |gem|
   gem.name          = "html-pipeline"
   gem.version       = HTML::Pipeline::VERSION
+  gem.license       = "MIT"
   gem.authors       = ["Ryan Tomayko", "Jerry Cheung"]
   gem.email         = ["ryan@github.com", "jerry@github.com"]
   gem.description   = %q{GitHub HTML processing filters and utilities}
@@ -14,12 +15,12 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^test})
   gem.require_paths = ["lib"]
 
-  gem.add_dependency 'gemoji',          '~> 1.1.1'
-  gem.add_dependency 'nokogiri',        '~> 1.4'
-  gem.add_dependency 'github-markdown', '~> 0.5'
-  gem.add_dependency 'sanitize',        '~> 2.0'
-  gem.add_dependency 'github-linguist', '~> 2.1'
-  gem.add_dependency 'rinku',           '~> 1.7'
-  gem.add_dependency 'escape_utils',    '~> 0.2'
-  gem.add_dependency 'activesupport',   '>= 2'
+  gem.add_dependency "gemoji",          "~> 1.0"
+  gem.add_dependency "nokogiri",        "~> 1.4"
+  gem.add_dependency "github-markdown", "~> 0.5"
+  gem.add_dependency "sanitize",        "~> 2.0"
+  gem.add_dependency "github-linguist", "~> 2.1"
+  gem.add_dependency "rinku",           "~> 1.7"
+  gem.add_dependency "escape_utils",    "~> 0.2"
+  gem.add_dependency "activesupport",   ">= 2"
 end

data/lib/html/pipeline.rb

@@ -9,8 +9,8 @@ module HTML
   #
   # See HTML::Pipeline::Filter for information on building filters.
   #
-  # Contruct a Pipeline for running multiple HTML filters.  A pipeline is created once
-  # with one to many filters, and is then can be `call`ed many times over the course
+  # Construct a Pipeline for running multiple HTML filters.  A pipeline is created once
+  # with one to many filters, and it then can be `call`ed many times over the course
   # of its lifetime with input.
   #
   # filters         - Array of Filter objects. Each must respond to call(doc,
@@ -22,7 +22,7 @@ module HTML
   #                   nil.  Default: empty Hash.
   # result_class    - The default Class of the result object for individual
   #                   calls.  Default: Hash.  Protip:  Pass in a Struct to get
-  #                   some semblence of type safety.
+  #                   some semblance of type safety.
   class Pipeline
     autoload :VERSION,               'html/pipeline/version'
     autoload :Pipeline,              'html/pipeline/pipeline'

data/lib/html/pipeline/camo_filter.rb

@@ -11,8 +11,8 @@ module HTML
     # in browser clients.
     #
     # Context options:
-    #   :asset_proxy - Base URL for constructed asset proxy URLs.
-    #   :asset_proxy_secret_key - The shared secret used to encode URLs.
+    #   :asset_proxy (required) - Base URL for constructed asset proxy URLs.
+    #   :asset_proxy_secret_key (required) - The shared secret used to encode URLs.
     #
     # This filter does not write additional information to the context.
     class CamoFilter < Filter
@@ -33,6 +33,12 @@ module HTML
         end
         doc
       end
+      
+      # Implementation of validate hook.
+      # Errors should raise exceptions or use an existing validator.
+      def validate
+        needs :asset_proxy, :asset_proxy_secret_key
+      end
 
       # The camouflaged URL for a given image URL.
       def asset_proxy_url(url)
@@ -47,11 +53,11 @@ module HTML
 
       # Private: the hostname to use for generated asset proxied URLs.
       def asset_proxy_host
-        context[:asset_proxy] or raise "Missing context :asset_proxy"
+        context[:asset_proxy]
       end
 
       def asset_proxy_secret_key
-        context[:asset_proxy_secret_key] or raise "Missing context :asset_proxy_secret_key"
+        context[:asset_proxy_secret_key]
       end
 
       # Private: helper to hexencode a string. Each byte ends up encoded into

data/lib/html/pipeline/emoji_filter.rb

@@ -5,7 +5,7 @@ module HTML
     # HTML filter that replaces :emoji: with images.
     #
     # Context:
-    #   :asset_root - base url to link to emoji sprite
+    #   :asset_root (required) - base url to link to emoji sprite
     class EmojiFilter < Filter
       # Build a regexp that matches all valid :emoji: names.
       EmojiPattern = /:(#{Emoji.names.map { |name| Regexp.escape(name) }.join('|')}):/
@@ -21,6 +21,12 @@ module HTML
         end
         doc
       end
+      
+      # Implementation of validate hook.
+      # Errors should raise exceptions or use an existing validator.
+      def validate
+        needs :asset_root
+      end
 
       # Replace :emoji: with corresponding images.
       #
@@ -41,7 +47,7 @@ module HTML
       # Raises ArgumentError if context option has not been provided.
       # Returns the context's asset_root.
       def asset_root
-        context[:asset_root] or raise ArgumentError, "Missing context :asset_root"
+        context[:asset_root]
       end
     end
   end

data/lib/html/pipeline/filter.rb

@@ -39,8 +39,9 @@ module HTML
         end
         @context = context || {}
         @result = result || {}
+        validate
       end
-
+      
       # Public: Returns a simple Hash used to pass extra information into filters
       # and also to allow filters to make extracted information available to the
       # caller.
@@ -73,6 +74,10 @@ module HTML
       def call
         raise NotImplementedError
       end
+      
+      # Make sure the context has everything we need. Noop: Subclasses can override.
+      def validate
+      end
 
       # The Repository object provided in the context hash, or nil when no
       # :repository was specified.
@@ -153,6 +158,21 @@ module HTML
           output.to_s
         end
       end
+      
+      # Validator for required context. This will check that anything passed in 
+      # contexts exists in @contexts
+      # 
+      # If any errors are found an ArgumentError will be raised with a
+      # message listing all the missing contexts and the filters that
+      # require them.
+      def needs(*keys)
+        missing = keys.reject { |key| context.include? key }
+
+        if missing.any?
+          raise ArgumentError,
+            "Missing context keys for #{self.class.name}: #{missing.map(&:inspect).join ', '}"
+        end
+      end
     end
   end
 end

data/lib/html/pipeline/image_max_width_filter.rb

@@ -9,11 +9,11 @@ module HTML
     class ImageMaxWidthFilter < Filter
       def call
         doc.search('img').each do |element|
-          # Skip if theres already a style attribute. Not sure how this
+          # Skip if there's already a style attribute. Not sure how this
           # would happen but we can reconsider it in the future.
           next if element['style']
 
-          # Bail out if src doesn't look like a valid http url. tryna avoid weird
+          # Bail out if src doesn't look like a valid http url. trying to avoid weird
           # js injection via javascript: urls.
           next if element['src'].to_s.strip =~ /\Ajavascript/i
 

data/lib/html/pipeline/sanitization_filter.rb

@@ -33,7 +33,7 @@ module HTML
         :elements => %w(
           h1 h2 h3 h4 h5 h6 h7 h8 br b i strong em a pre code img tt
           div ins del sup sub p ol ul table blockquote dl dt dd
-          kbd q samp var hr ruby rt rp
+          kbd q samp var hr ruby rt rp li tr td th
         ),
         :attributes => {
           'a' => ['href'],
@@ -62,22 +62,20 @@ module HTML
           'img' => {'src'  => ['http', 'https', :relative]}
         },
         :transformers => [
-          # whitelist only <li> elements that are descended from a <ul> or <ol>.
-          # top-level <li> elements are removed because they can break out of
+          # Top-level <li> elements are removed because they can break out of
           # containing markup.
           lambda { |env|
             name, node = env[:node_name], env[:node]
-            if name == LIST_ITEM && node.ancestors.any?{ |n| LISTS.include?(n.name) }
-              {:node_whitelist => [node]}
+            if name == LIST_ITEM && !node.ancestors.any?{ |n| LISTS.include?(n.name) }
+              node.replace(node.children)
             end
           },
 
-          # Whitelist only table child elements that are descended from a <table>.
           # Table child elements that are not contained by a <table> are removed.
           lambda { |env|
             name, node = env[:node_name], env[:node]
-            if TABLE_ITEMS.include?(name) && node.ancestors.any? { |n| n.name == TABLE }
-              { :node_whitelist => [node] }
+            if TABLE_ITEMS.include?(name) && !node.ancestors.any? { |n| n.name == TABLE }
+              node.replace(node.children)
             end
           }
         ]

data/lib/html/pipeline/version.rb

@@ -1,5 +1,5 @@
 module HTML
   class Pipeline
-    VERSION = "0.0.4"
+    VERSION = "0.0.5"
   end
 end

data/test/html/pipeline/camo_filter_test.rb

@@ -36,4 +36,12 @@ class HTML::Pipeline::CamoFilterTest < Test::Unit::TestCase
       CamoFilter.call(orig, @options).to_s
     end
   end
+    
+  def test_required_context_validation
+    exception = assert_raise(ArgumentError) { 
+      CamoFilter.call("", {}) 
+    }
+    assert_match /:asset_proxy[^_]/, exception.message
+    assert_match /:asset_proxy_secret_key/, exception.message
+  end
 end

data/test/html/pipeline/emoji_filter_test.rb

@@ -1,16 +1,18 @@
 require 'test_helper'
 
 class HTML::Pipeline::EmojiFilterTest < Test::Unit::TestCase
+  EmojiFilter = HTML::Pipeline::EmojiFilter
+  
   def test_emojify
-    filter = HTML::Pipeline::EmojiFilter.new("<p>:shipit:</p>", {:asset_root => 'https://foo.com'})
+    filter = EmojiFilter.new("<p>:shipit:</p>", {:asset_root => 'https://foo.com'})
     doc = filter.call
     assert_match "https://foo.com/emoji/shipit.png", doc.search('img').attr('src').value
   end
-
-  def test_missing_context
-    filter = HTML::Pipeline::EmojiFilter.new("<p>:shipit:</p>", {})
-    assert_raises ArgumentError do
-      filter.call
-    end
+  
+  def test_required_context_validation
+    exception = assert_raise(ArgumentError) { 
+      EmojiFilter.call("", {}) 
+    }
+    assert_match /:asset_root/, exception.message
   end
 end

data/test/html/pipeline/sanitization_filter_test.rb

@@ -32,7 +32,7 @@ class HTML::Pipeline::SanitizationFilterTest < Test::Unit::TestCase
   def test_sanitizes_li_elements_not_contained_in_ul_or_ol
     stuff = "a\n<li>b</li>\nc"
     html  = SanitizationFilter.call(stuff).to_s
-    assert_equal "a\n b \nc", html
+    assert_equal "a\nb\nc", html
   end
 
   def test_does_not_sanitize_li_elements_contained_in_ul_or_ol

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: html-pipeline
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-11-07 00:00:00.000000000 Z
+date: 2012-12-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gemoji
@@ -19,7 +19,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.1.1
+        version: '1.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -27,7 +27,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.1.1
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: nokogiri
   requirement: !ruby/object:Gem::Requirement
@@ -150,6 +150,7 @@ extra_rdoc_files: []
 files:
 - .gitignore
 - .travis.yml
+- CHANGELOG.md
 - Gemfile
 - LICENSE
 - README.md
@@ -184,7 +185,8 @@ files:
 - test/html/pipeline/toc_filter_test.rb
 - test/test_helper.rb
 homepage: https://github.com/jch/html-pipeline
-licenses: []
+licenses:
+- MIT
 post_install_message: 
 rdoc_options: []
 require_paths: