deadweight 0.1.3 → 0.2.0

data/README.md

@@ -0,0 +1,123 @@
+Deadweight
+==========
+
+Deadweight is a CSS coverage tool. Given a set of stylesheets and a set of URLs, it determines which selectors are actually used and reports which can be "safely" deleted.
+
+Screencast!
+-----------
+
+Ryan Bates has worked his magic once again. [Head over here for an excellent introduction to Deadweight](http://railscasts.com/episodes/180-finding-unused-css).
+
+How to Use It
+-------------
+
+There are multiple ways to use Deadweight. It's designed to be completely agnostic of whatever framework (or, indeed language) your website is built on, but optimised for Ruby and Rails.
+
+I'm going to tell you about the coolest way to use it first.
+
+### The Coolest Way to Use It ###
+
+Deadweight can hijack your Rails integration tests (both the spartan Test::Unit type and the refreshing Cucumber variety), capturing every page that is returned by your app during testing and saving you the trouble of manually specifying a ton of URLs and login processses. THIS IS PRETTY COOL YOU SHOULD TRY IT. (It's also somewhat new and untested, but you're the kind of person who just _loves_ it out here on the edge, I can tell.)
+
+Here's how it works. First, put this in your `Gemfile`:
+
+    group :test do
+      gem 'colored' # optional, in the same way that dressing respectably when you leave the house is 'optional'
+      gem 'deadweight', :require => 'deadweight/hijack/rails'
+    end
+
+Then, run your integration tests with the environment variable `DEADWEIGHT` set to `true`:
+
+    rake test DEADWEIGHT=true
+    rake cucumber DEADWEIGHT=true
+
+Let me know how it goes. It's not terribly customisable at the moment (you can't specify what exact stylesheets to look at, or what selectors to ignore). I'm looking for your feedback on how you'd like to be able to do that.
+
+### Or Make a Rake Task ###
+
+    # lib/tasks/deadweight.rake
+  
+    require 'deadweight'
+  
+    Deadweight::RakeTask.new do |dw|
+      dw.stylesheets = %w( /stylesheets/style.css )
+      dw.pages = %w( / /page/1 /about )
+    end
+
+Running `rake deadweight` will output all unused rules, one per line. Note that it looks at `http://localhost:3000` by default, so you'll need to have `script/server` (or whatever your server command looks like) running.
+
+### Or Run it From the Command Line ###
+
+    $ deadweight -s styles.css -s ie.css index.html about.html
+    $ deadweight -s http://www.tigerbloodwins.com/index.css http://www.tigerbloodwins.com/
+    $ deadweight --root http://kottke.org/ -s '/templates/2009/css.php?p=mac' / /everfresh /about
+
+You can pipe in CSS rules from STDIN:
+
+    $ cat styles.css | deadweight index.html
+
+And you can use it as an HTTP proxy:
+
+    $ deadweight -l deadweight.log -s styles.css -w http://github.com/ -P
+
+### Or Call it Directly ###
+
+    require 'deadweight'
+  
+    dw = Deadweight.new
+    dw.stylesheets = %w( /stylesheets/style.css )
+    dw.pages = %w( / /page/1 /about )
+    puts dw.run
+
+Setting the Root URL
+--------------------
+
+By default, Deadweight uses `http://localhost:3000` as the base URL for all paths. To change it, set `root`:
+
+    dw.root = "http://staging.example.com"      # staging server
+    dw.root = "http://example.com/staging-area" # urls can have paths in
+    dw.root = "/path/to/some/html"              # local paths work too
+
+What About Stuff Added by Javascript?
+-------------------------------------
+
+Deadweight is completely dumb about any classes, IDs or tags that are only added by your Javascript layer, but you can filter them out by setting `ignore_selectors`:
+
+    dw.ignore_selectors = /hover|lightbox|superimposed_kittens/
+
+The command-line tool also has basic support for [Lyndon](http://github.com/defunkt/lyndon) with the `-L` flag, which simply pipes all HTML through the `lyndon` executable.
+
+You Can Use Mechanize for Complex Stuff
+---------------------------------------
+
+Set `mechanize` to `true` and add a Proc to `pages` (rather than a String), and Deadweight will execute it using [Mechanize](http://mechanize.rubyforge.org/mechanize):
+
+    dw.mechanize = true
+
+    # go through the login form to get to a protected URL
+    dw.pages << proc {
+      fetch('/login')
+      form = agent.page.forms.first
+      form.username = 'username'
+      form.password = 'password'
+      agent.submit(form)
+      fetch('/secret-page')
+    }
+
+    # use HTTP basic auth
+    dw.pages << proc {
+      agent.auth('username', 'password')
+      fetch('/other-secret-page')
+    }
+
+The `agent` method returns the Mechanize instance. The `fetch` method is a wrapper around `agent.get` that will abort in the event of an HTTP error status.
+
+If You Install `colored`, It'll Look Nicer
+-------------------------------------------------
+
+    gem install colored
+
+Copyright
+---------
+
+Copyright (c) 2009 Aanand Prasad. See LICENSE for details.

data/lib/deadweight.rb

@@ -1,7 +1,11 @@
+require 'rubygems'
+require 'bundler'
+Bundler.setup
+
 $LOAD_PATH.concat Dir.glob(File.expand_path('../../vendor/gems/*/lib', __FILE__))
 
 require 'css_parser'
-require 'hpricot'
+require 'nokogiri'
 require 'open-uri'
 
 begin
@@ -30,12 +34,16 @@ class Deadweight
   end
 
   def analyze(html)
-    doc = Hpricot(html)
+    doc = Nokogiri::HTML(html)
 
     @unused_selectors.collect do |selector, declarations|
       # We test against the selector stripped of any pseudo classes,
       # but we report on the selector with its pseudo classes.
-      unless doc.search(strip(selector)).empty?
+      stripped_selector = strip(selector)
+
+      next if stripped_selector.empty?
+
+      if doc.search(stripped_selector).any?
         log.puts("  #{selector.green}")
         selector
       end
@@ -62,8 +70,7 @@ class Deadweight
     selector_count
   end
 
-  # Find all unused CSS selectors and return them as an array.
-  def run
+  def reset!
     @parsed_rules     = {}
     @unused_selectors = []
 
@@ -78,7 +85,18 @@ class Deadweight
       log.puts("Added #{new_selector_count} extra selectors".yellow)
     end
 
-    total_selectors = @unused_selectors.size
+    @total_selectors = @unused_selectors.size
+  end
+
+  def report
+    log.puts
+    log.puts "found #{@unused_selectors.size} unused selectors out of #{@total_selectors} total".yellow
+    log.puts
+  end
+
+  # Find all unused CSS selectors and return them as an array.
+  def run
+    reset!
 
     pages.each do |page|
       log.puts
@@ -106,9 +124,7 @@ class Deadweight
       process!(html)
     end
 
-    log.puts
-    log.puts "found #{@unused_selectors.size} unused selectors out of #{total_selectors} total".yellow
-    log.puts
+    report
 
     @unused_selectors
   end
@@ -139,7 +155,7 @@ class Deadweight
 
       begin
         page = agent.get(loc)
-      rescue WWW::Mechanize::ResponseCodeError => e
+      rescue Mechanize::ResponseCodeError => e
         raise FetchError.new("#{loc} returned a response code of #{e.response_code}")
       end
 
@@ -178,7 +194,17 @@ private
   def initialize_agent
     begin
       require 'mechanize'
-      return WWW::Mechanize.new
+
+      unless defined?(Mechanize::VERSION) and Mechanize::VERSION >= "1.0.0"
+        log.puts %{
+          =================================================================
+          A mechanize version of 1.0.0 or above is required.
+          Install it like so: gem install mechanize
+          =================================================================
+        }
+      end
+
+      return Mechanize.new
     rescue LoadError
       log.puts %{
         =================================================================

data/lib/deadweight/hijack.rb

@@ -0,0 +1,13 @@
+class Deadweight
+  module Hijack
+    def self.redirect_output(log_file_prefix)
+      original_stdout, original_stderr = STDOUT.clone, STDERR.clone
+
+      STDOUT.reopen(File.open("#{log_file_prefix}stdout.log", 'w'))
+      STDERR.reopen(File.open("#{log_file_prefix}stderr.log", 'w'))
+
+      [original_stdout, original_stderr]
+    end
+  end
+end
+

data/lib/deadweight/hijack/rails.rb

@@ -0,0 +1,35 @@
+if ENV['DEADWEIGHT'] == 'true'
+  require 'deadweight'
+  require 'deadweight/hijack'
+  require 'deadweight/rack/capturing_middleware'
+
+  class Deadweight
+    module Hijack
+      module Rails
+        class Railtie < ::Rails::Railtie
+          initializer "deadweight.hijack" do |app|
+            root = ::Rails.root
+
+            original_stdout, original_stderr = Deadweight::Hijack.redirect_output(root + 'log/test_')
+
+            dw = Deadweight.new
+
+            dw.root        = root + 'public'
+            dw.stylesheets = Dir.chdir(dw.root) { Dir.glob("stylesheets/*.css") }
+            dw.log_file    = original_stderr
+
+            dw.reset!
+
+            at_exit do
+              dw.report
+              dw.dump(original_stdout)
+            end
+
+            app.middleware.insert(0, Deadweight::Rack::CapturingMiddleware, dw)
+          end
+        end
+      end
+    end
+  end
+end
+

data/lib/deadweight/rack/capturing_middleware.rb

@@ -0,0 +1,26 @@
+class Deadweight
+  module Rack
+    class CapturingMiddleware
+      def initialize(app, dw)
+        @app = app
+        @dw  = dw
+      end
+
+      def call(env)
+        response = @app.call(env)
+        process(response)
+        response
+      end
+
+      def process(rack_response)
+        status, headers, response = rack_response
+
+        if response.respond_to?(:body)
+          html = response.body
+          @dw.process!(html)
+        end
+      end
+    end
+  end
+end
+

data/vendor/gems/{css_parser-1.0.0 → css_parser-1.1.5}/lib/css_parser.rb

@@ -1,13 +1,14 @@
-$:.unshift File.dirname(__FILE__)
 require 'uri'
+require 'net/https'
+require 'open-uri'
 require 'digest/md5'
 require 'zlib'
+require 'stringio'
 require 'iconv'
-require 'css_parser/rule_set'
-require 'css_parser/regexps'
-require 'css_parser/parser'
 
 module CssParser
+  VERSION = '1.1.5'
+
   # Merge multiple CSS RuleSets by cascading according to the CSS 2.1 cascading rules 
   # (http://www.w3.org/TR/REC-CSS2/cascade.html#cascading-order).
   #
@@ -74,20 +75,28 @@ module CssParser
 
       rule_set.each_declaration do |property, value, is_important|
         # Add the property to the list to be folded per http://www.w3.org/TR/CSS21/cascade.html#cascading-order
-        if not properties.has_key?(property) or
-               is_important or # step 2
-               properties[property][:specificity] < specificity or # step 3
-               properties[property][:specificity] == specificity # step 4    
-          properties[property] = {:value => value, :specificity => specificity, :is_important => is_important}            
+        if not properties.has_key?(property)
+          properties[property] = {:value => value, :specificity => specificity, :is_important => is_important}
+        elsif properties[property][:specificity] < specificity or properties[property][:specificity] == specificity
+          unless properties[property][:is_important]
+            properties[property] = {:value => value, :specificity => specificity, :is_important => is_important}            
+          end
         end
+
+        if is_important
+            properties[property] = {:value => value, :specificity => specificity, :is_important => is_important}            
+        end        
       end
     end
 
     merged = RuleSet.new(nil, nil)
 
-    # TODO: what about important
     properties.each do |property, details|
-      merged[property.strip] = details[:value].strip
+      if details[:is_important]
+        merged[property.strip] = details[:value].strip.gsub(/\;\Z/, '') + '!important' 
+      else
+        merged[property.strip] = details[:value].strip
+      end
     end
 
     merged.create_shorthand!
@@ -146,4 +155,8 @@ module CssParser
     end
     out
   end
-end
+end
+
+require File.dirname(__FILE__) + '/css_parser/rule_set'
+require File.dirname(__FILE__) + '/css_parser/regexps'
+require File.dirname(__FILE__) + '/css_parser/parser'

data/vendor/gems/{css_parser-1.0.0 → css_parser-1.1.5}/lib/css_parser/parser.rb

@@ -15,19 +15,15 @@ module CssParser
   # [<tt>import</tt>] Follow <tt>@import</tt> rules. Boolean, default is <tt>true</tt>.
   # [<tt>io_exceptions</tt>] Throw an exception if a link can not be found. Boolean, default is <tt>true</tt>.
   class Parser
-    USER_AGENT   = "Ruby CSS Parser/#{RUBY_VERSION} (http://code.dunae.ca/css_parser/)"
+    USER_AGENT   = "Ruby CSS Parser/#{CssParser::VERSION} (http://github.com/alexdunae/css_parser)"
 
     STRIP_CSS_COMMENTS_RX = /\/\*.*?\*\//m
     STRIP_HTML_COMMENTS_RX = /\<\!\-\-|\-\-\>/m
 
     # Initial parsing
-    RE_AT_IMPORT_RULE = /\@import[\s]+(url\()?["']+(.[^'"]*)["']\)?([\w\s\,]*);?/i
+    RE_AT_IMPORT_RULE = /\@import\s*(?:url\s*)?(?:\()?(?:\s*)["']?([^'"\s\)]*)["']?\)?([\w\s\,^\])]*)\)?[;\n]?/
 
-    #--
-    # RE_AT_IMPORT_RULE = Regexp.new('@import[\s]*(' + RE_STRING.to_s + ')([\w\s\,]*)[;]?', Regexp::IGNORECASE) -- should handle url() even though it is not allowed
-    #++
-
-    # Array of CSS files that have been loaded.
+     # Array of CSS files that have been loaded.
     attr_reader   :loaded_uris
 
     #attr_reader   :rules
@@ -82,6 +78,13 @@ module CssParser
 
     # Add a raw block of CSS.
     #
+    # In order to follow +@import+ rules you must supply either a
+    # +:base_dir+ or +:base_uri+ option.
+    #
+    # Use the +:media_types+ option to set the media type(s) for this block.  Takes an array of symbols.
+    #
+    # Use the +:only_media_types+ option to selectively follow +@import+ rules.  Takes an array of symbols.
+    #
     # ==== Example
     #   css = <<-EOT
     #     body { font-size: 10pt }
@@ -92,21 +95,43 @@ module CssParser
     #   EOT
     #
     #   parser = CssParser::Parser.new
-    #   parser.load_css!(css)
-    #--
-    # TODO: add media_type
-    #++
+    #   parser.add_block!(css)
     def add_block!(block, options = {})
-      options = {:base_uri => nil, :charset => nil, :media_types => :all}.merge(options)
-      
+      options = {:base_uri => nil, :base_dir => nil, :charset => nil, :media_types => :all, :only_media_types => :all}.merge(options)
+      options[:media_types] = [options[:media_types]].flatten
+      options[:only_media_types] = [options[:only_media_types]].flatten
+
       block = cleanup_block(block)
 
       if options[:base_uri] and @options[:absolute_paths]
         block = CssParser.convert_uris(block, options[:base_uri])
       end
+
+      # Load @imported CSS
+      block.scan(RE_AT_IMPORT_RULE).each do |import_rule|    
+        media_types = []
+        if media_string = import_rule[-1]
+          media_string.split(/\s|\,/).each do |t|
+            media_types << t.to_sym unless t.empty?
+          end
+        end
+              
+        next unless options[:only_media_types].include?(:all) or media_types.length < 1 or (media_types & options[:only_media_types]).length > 0
+
+        import_path = import_rule[0].to_s.gsub(/['"]*/, '').strip
+
+        if options[:base_uri]
+          import_uri = URI.parse(options[:base_uri].to_s).merge(import_path)
+          load_uri!(import_uri, options[:base_uri], media_types)
+        elsif options[:base_dir]
+          load_file!(import_path, options[:base_dir], media_types)
+        end     
+      end
+
+      # Remove @import declarations
+      block.gsub!(RE_AT_IMPORT_RULE, '')
       
       parse_block_into_rule_sets!(block, options)
-      
     end
 
     # Add a CSS rule by setting the +selectors+, +declarations+ and +media_types+.
@@ -175,7 +200,7 @@ module CssParser
       options = {:media_types => :all}.merge(options)
       media_types = options[:media_types]
 
-      in_declarations = false
+      in_declarations = 0
 
       block_depth = 0
 
@@ -195,13 +220,25 @@ module CssParser
           in_string = !in_string
         end       
 
-        if in_declarations
+        if in_declarations > 0
+
+          # too deep, malformed declaration block
+          if in_declarations > 1
+            in_declarations -= 1 if token =~ /\}/
+            next
+          end
+          
+          if token =~ /\{/
+            in_declarations += 1
+            next
+          end
+        
           current_declarations += token
 
           if token =~ /\}/ and not in_string
             current_declarations.gsub!(/\}[\s]*$/, '')
             
-            in_declarations = false
+            in_declarations -= 1
 
             unless current_declarations.strip.empty?
               #puts "SAVING #{current_selectors} -> #{current_declarations}"
@@ -233,7 +270,7 @@ module CssParser
             if token =~ /\{/ and not in_string
               current_selectors.gsub!(/^[\s]*/, '')
               current_selectors.gsub!(/[\s]*$/, '')
-              in_declarations = true
+              in_declarations += 1
             else
               current_selectors += token
             end
@@ -243,35 +280,35 @@ module CssParser
     end
 
     # Load a remote CSS file.
+    #
+    # You can also pass in file://test.css
     def load_uri!(uri, base_uri = nil, media_types = :all)
+      uri = URI.parse(uri) unless uri.respond_to? :scheme
+      if uri.scheme == 'file' or uri.scheme.nil?
+        uri.path = File.expand_path(uri.path)
+        uri.scheme = 'file'
+      end
       base_uri = uri if base_uri.nil?
-      src, charset = read_remote_file(uri)
-
-      # Load @imported CSS
-      src.scan(RE_AT_IMPORT_RULE).each do |import_rule|        
-        import_path = import_rule[1].to_s.gsub(/['"]*/, '').strip
-        import_uri = URI.parse(base_uri.to_s).merge(import_path)
-        #puts import_uri.to_s
 
-        media_types = []
-        if media_string = import_rule[import_rule.length-1]
-          media_string.split(/\s|\,/).each do |t|
-            media_types << t.to_sym unless t.empty?
-          end
-        end
+      src, charset = read_remote_file(uri)
 
-        # Recurse
-        load_uri!(import_uri, nil, media_types)
+      if src
+        add_block!(src, {:media_types => media_types, :base_uri => base_uri})
       end
+    end
+    
+    # Load a local CSS file.
+    def load_file!(file_name, base_dir = nil, media_types = :all)
+      file_name = File.expand_path(file_name, base_dir)
+      return unless File.readable?(file_name)
 
-      # Remove @import declarations
-      src.gsub!(RE_AT_IMPORT_RULE, '')
-
-      # Relative paths need to be converted here
-      src = CssParser.convert_uris(src, base_uri) if base_uri and @options[:absolute_paths]
+      src = IO.read(file_name)
+      base_dir = File.dirname(file_name)
 
-      add_block!(src, {:media_types => media_types})
+      add_block!(src, {:media_types => media_types, :base_dir => base_dir})
     end
+    
+    
 
   protected
     # Strip comments and clean up blank lines from a block of CSS.
@@ -298,30 +335,59 @@ module CssParser
     # TODO: add option to fail silently or throw and exception on a 404
     #++
     def read_remote_file(uri) # :nodoc:
-      raise CircularReferenceError, "can't load #{uri.to_s} more than once" if @loaded_uris.include?(uri.to_s)
+      if @loaded_uris.include?(uri.to_s)
+        raise CircularReferenceError, "can't load #{uri.to_s} more than once" if @options[:io_exceptions]
+        return '', nil
+      end
+
       @loaded_uris << uri.to_s
 
-      begin
-      #fh = open(uri, 'rb')
-        fh = open(uri, 'rb', 'User-Agent' => USER_AGENT, 'Accept-Encoding' => 'gzip')
+      src = '', charset = nil
 
-        if fh.content_encoding.include?('gzip')
-          remote_src = Zlib::GzipReader.new(fh).read
+      begin
+        uri = URI.parse(uri.to_s)    
+        http = Net::HTTP.new(uri.host, uri.port)
+
+        if uri.scheme == 'file'
+          # local file
+          fh = open(uri.path, 'rb')
+          src = fh.read
+          fh.close
         else
-          remote_src = fh.read
-        end
+          # remote file
+          if uri.scheme == 'https'
+            http.use_ssl = true 
+            http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+          end
 
-        #puts "reading #{uri} (#{fh.charset})"
+          res, src = http.get(uri.path, {'User-Agent' => USER_AGENT, 'Accept-Encoding' => 'gzip'})
+          charset = fh.respond_to?(:charset) ? fh.charset : 'utf-8'
 
-        ic = Iconv.new('UTF-8//IGNORE', fh.charset)
-        src = ic.iconv(remote_src)
+          if res.code.to_i >= 400
+            raise RemoteFileError if @options[:io_exceptions]
+            return '', nil
+          end
+
+          case res['content-encoding']
+            when 'gzip'
+              io = Zlib::GzipReader.new(StringIO.new(res.body))
+              src = io.read
+            when 'deflate'
+              io = Zlib::Inflate.new
+              src = io.inflate(res.body)
+          end
+        end
 
-        fh.close
-        return src, fh.charset
+        if charset
+          ic = Iconv.new('UTF-8//IGNORE', charset)
+          src = ic.iconv(src)
+        end
       rescue
         raise RemoteFileError if @options[:io_exceptions]
-        return '', nil
+        return nil, nil
       end
+
+      return src, charset  
     end
 
   private
@@ -342,4 +408,4 @@ module CssParser
       @css_warnings = []
     end
   end
-end
+end