varnisher 1.0.beta.1 → 1.0.beta.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 89bc9abf14d10e978e9d63af4652deebd7912205
-  data.tar.gz: 3074562679934cc26c5b413a281fe9a4aef31f23
+  metadata.gz: 4a9d710584e6d43f0f925cd4894f536fa1630551
+  data.tar.gz: 9d22ce540f64d36683db840c395c03c1096a88cd
 SHA512:
-  metadata.gz: ec489165fc9a6b70f8824cfef3b962657467ad319dc584a4478497d76eedf8c0adf74d6af8b1b4201935038a57affc156d1e33ec41927f1eb39d5cac6147a112
-  data.tar.gz: b41896764dfab0db49743b7d30236de8f6ea30810b72f6e135d8b252ac8bd9b58c91c65fd9c10a61c7e2cb16bc7f7a869a3999d2f2e2dacdfd57abe59852bca6
+  metadata.gz: 7afaedc98a7557689c4908da0ba13e54f674d70ec3fd48543f0bf4181b016d61d05f532196683f3dfd97ff8e44ddde417afb4e116bfdf941af73068664531327
+  data.tar.gz: 55cd0e503a1152418c84e3a1876ee75a38b2c6450ea5e9495b0cd2d2d55bf608d8ef07939e84f170cfc19d2c665f8863cafb5e6aeb21241a71e8fc71caa706f2

data/README.md

@@ -1,23 +1,53 @@
-# Varnish Toolkit
+# varnisher
 
-Administering Varnish is generally a breeze, but sometimes you want to do one of the few things that aren't painless out of the box. Hopefully, that's where this toolbox comes in.
+Administering Varnish is generally a breeze, but sometimes you want to
+do one of the few things that aren't painless out of the box. Hopefully,
+that's where this toolbox comes in.
 
-Varnish Toolkit relies on the wonderful libraries [hpricot](http://hpricot.com/), by why the lucky stiff, and [Parallel](https://github.com/grosser/parallel), by Michael Grosser. If you don't have them, install them with:
+Varnisher lets you do things like:
 
-    $ sudo gem install hpricot parallel
+* Purge a webpage and everything (e.g. images, JavaScript files, CSS
+  files) referenced on that page
+* Spider an entire domain — useful for priming a cache
+* Purge an entire domain, including optionally re-spidering it
+  afterwards to keep the cache warm
+
+## Installation
+
+Varnish requires Ruby >1.9.3 to run. If you've got a recent Ruby
+installed, then Varnisher can be installed by running:
+
+	gem install varnisher
 
 ## Usage
 
-    Usage: varnish.rb [options] action target
-        -h, --help                       Display this help
-        -v, --verbose                    Output more information
-        -H, --hostname HOSTNAME          Hostname/IP address of your Varnish server. Default is localhost
-        -p, --port PORT                  Port your Varnish server is listening on. Default is 80
-        -n, --num-pages NUM              Number of pages to crawl when in spider mode. -1 will crawl all pages
-        -#, --hashes                     If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode
-        -q, --ignore-query-string        If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode
+	NAME
+	varnisher
+
+	SYNOPSIS
+	varnisher (purge|spider) [options]+
+
+	DESCRIPTION
+	Varnisher is a set of tools for working with the Varnish HTTP cache.
+
+	PARAMETERS
+	--verbose, -v
+		If given, Varnisher will be noisier about what it's up to.
+	--hostname=hostname, -H (0 ~> hostname=localhost)
+		The hostname/IP address of your Varnish server.
+	--port=port, -p (0 ~> int(port=80))
+		The port Varnish is listening on.
+	--help, -h
+
+	EXAMPLES
+	varnisher purge http://example.com
+	varnisher spider example.com
+	varnisher purge --reindex example.com
 
-If you find yourself typing certain parameters every time you use the script, you can specify them in an RC file called `.varnishrc` in your home directory. The file format is YAML and the default options are, if you want to paste and override them:
+If you find yourself typing certain parameters every time you use the
+script, you can specify them in an RC file called `.varnishrc` in your
+home directory. The file format is YAML and the default options are, if
+you want to paste and override them:
 
     verbose: false
     hostname: localhost
@@ -30,17 +60,25 @@ If you find yourself typing certain parameters every time you use the script, yo
 
 ### Purging a page and all the resources on it
 
-Quite often, it's necessary redevelop a page on a website in a way that involves changes not only to the page but also to CSS files, images, JavaScript files, etc. Purging pages in this instance can be a painful process, or at least one that requires a few `ban` commands in `varnishadm`. No longer!
+Quite often, it's necessary redevelop a page on a website in a way that
+involves changes not only to the page but also to CSS files, images,
+JavaScript files, etc. Purging pages in this instance can be a painful
+process, or at least one that requires a few `ban` commands in
+`varnishadm`. No longer!
 
 Just enter:
 
-	$ varnish.rb purge http://www.example.com/path/to/page
+	$ varnisher purge http://www.example.com/path/to/page
 
-...and `/path/to/page`, along with all its images, CSS files, JavaScript files, and other external accoutrements, will be purged from Varnish's cache. 
+...and `/path/to/page`, along with all its images, CSS files, JavaScript
+files, and other external accoutrements, will be purged from Varnish's
+cache.
 
-As a bonus, this action is multithreaded, meaning even resource-heavy pages should purge quickly and evenly.
+As a bonus, this action is multithreaded, meaning even resource-heavy
+pages should purge quickly and evenly.
 
-This action requires your VCL to have something like the following, which is fairly standard:
+This action requires your VCL to have something like the following,
+which is fairly standard:
 
 	if (req.request == "PURGE") {
         if ( client.ip ~ auth ) {
@@ -49,7 +87,9 @@ This action requires your VCL to have something like the following, which is fai
         }
     }
 
-(For an explanation of just what `obj.http.x-url` means, and why you should use it rather than `req.url`, see [this page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
+(For an explanation of just what `obj.http.x-url` means, and why you
+should use it rather than `req.url`, see [this
+page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
 
 ### Purging an entire domain
 
@@ -62,25 +102,31 @@ Provided your VCL has something akin to the following in it:
             }
     }
 
-...then you should be able to quickly purge an entire domain's worth of pages and resources by simply issuing the command:
+...then you should be able to quickly purge an entire domain's worth of
+pages and resources by simply issuing the command:
 
-	$ varnish.rb purge www.example.com
+	$ varnisher purge www.example.com
 
 ### Repopulating the cache
 
-If you've purged a whole domain, and particularly if your backend is slow, you might want to quickly repopulate the cache so that users never see your slow misses. Well, you can! Use the `spider` action:
+If you've purged a whole domain, and particularly if your backend is
+slow, you might want to quickly repopulate the cache so that users never
+see your slow misses. Well, you can! Use the `spider` action:
 
-	$ varnish.rb spider www.example.com
+	$ varnisher spider www.example.com
 
-`spider` accepts either a hostname or a URL as its starting point, and will only fetch pages on the same domain as its origin. You can limit the number of pages it will process using the `-n` parameter:
+`spider` accepts either a hostname or a URL as its starting point, and
+will only fetch pages on the same domain as its origin. You can limit
+the number of pages it will process using the `-n` parameter:
 
-	$ varnish.rb -n 500 spider www.example.com
+	$ varnisher -n 500 spider www.example.com
 
-If you'd like to combine purging and spidering, you can use the `reindex` action:
+If you'd like to combine purging and spidering, you can use the
+`reindex` option:
 
-	$ varnish.rb reindex www.example.com
+	$ varnisher purge --reindex www.example.com
 
 …which is functionally equivalent to:
 
-	$ varnish.rb purge www.example.com
-	$ varnish.rb spider www.example.com
+	$ varnisher purge www.example.com
+	$ varnisher spider www.example.com

data/bin/varnisher

@@ -1,96 +1,107 @@
 #!/usr/bin/env ruby
 
-require 'optparse'
+require 'bundler'
+Bundler.setup
+
+require 'main'
 require 'yaml'
 
 require 'varnisher'
 
-$options = {
-  :verbose             => false,
-  :hostname            => 'localhost',
-  :port                => 80,
-  :num_pages           => 100,
-  :ignore_hash         => true,
-  :spider_threads      => 16,
-  :ignore_query_string => false
-}
+Main {
+  examples "varnisher purge http://example.com", "varnisher spider example.com", "varnisher purge --reindex example.com"
 
-rcfile = File.expand_path("~/.varnishrc")
-if FileTest.readable? rcfile
-  rc = YAML::load(File.open(rcfile))
-  # Convert to symbols
-  rc = rc.inject({}){ |memo,(k,v)| memo[k.to_sym] = v; memo }
-  $options.merge!(rc)
-end
+  description "Varnisher is a set of tools for working with the Varnish HTTP cache."
 
-optparse = OptionParser.new do |opts|
-  opts.banner = 'Usage: varnish.rb [options] action target'
+  argument 'target'
 
-  opts.on('-h', '--help', 'Display this help') do
-    puts opts
-  end
+  option('v', 'verbose') {
+    description "If given, Varnisher will be noisier about what it's up to."
+  }
 
-  opts.on('-v', '--verbose', 'Output more information') do
-    $options[:verbose] = true
-  end
+  option('H', 'hostname') {
+    argument :required
+    description "The hostname/IP address of your Varnish server."
+    default "localhost"
+  }
 
-  opts.on('-H', '--hostname HOSTNAME', 'Hostname/IP address of your Varnish server. Default is localhost') do |hostname|
-    $options[:hostname] = hostname
-  end
+  option('p', 'port') {
+    argument :required
+    cast :int
+    description "The port Varnish is listening on."
+    default 80
+  }
 
-  opts.on('-p', '--port PORT', 'Port your Varnish server is listening on. Default is 80') do |port|
-    $options[:port] = port
+  def before_run
+    load_config
   end
 
-  opts.on('-n', '--num-pages NUM', 'Number of pages to crawl when in spider mode. -1 will crawl all pages') do |num|
-    $options[:num_pages] = num.to_i
-  end
+  mode "purge" do
+    argument('target') { description "The URL or hostname to purge" }
 
-  opts.on('-t', '--spider-threads NUM', 'Number of threads to use when spidering. Default is 16') do |num|
-    $options[:spider_threads] = num.to_i
-  end
+    option('reindex') {
+      description "If you specify a hostname to purge, this option will respider that hostname after the purging is complete. This will keep your cache as warm as possible."
+    }
 
-  opts.on('-#', '--hashes', 'If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode') do
-    $options[:ignore_hash] = false 
-  end
+    def run
+      target = params['target'].value
+
+      # If target is a valid URL, then assume we're purging a page and its contents.
+      if target =~ /^[a-z]+:\/\//
+        Varnisher::PagePurger.new target
+      end
 
-  opts.on('-q', '--ignore-query-string', 'If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode') do
-    $options[:ignore_query_string] = true
+      # If target is a hostname, assume we want to purge an entire domain.
+      if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
+        Varnisher::DomainPurger.new target
+
+        if params['reindex'].given?
+          Varnisher::Spider.new "http://#{target}/"
+        end
+      end
+    end
   end
-end
 
-optparse.parse!
+  mode "spider" do
+    argument('target') { description "The URL to begin spidering from." }
 
-# All our libs use these constants.
-PROXY_HOSTNAME = $options[:hostname]
-PROXY_PORT = $options[:port].to_i
+    option('n', 'num-pages') {
+      argument :required
+      cast :int
+      description "Maximum number of pages to crawl. Setting this to -1 (the default) will impose no limit."
+      default -1
+    }
 
-if ( ARGV.length < 2 )
-  puts "You must specify both an action and a target."
-end
+    option('t', 'threads') {
+      argument :required
+      cast :int
+      description "Spidering is done in parallel; this variable controls how many threads will be used."
+      default 16
+    }
 
-action = ARGV[0]
-target = ARGV[1]
+    option('#', 'ignore-hashes') {
+      description "When given, /foo#foo and /foo#bar will be treated as separate URLs; the default is to treat them as the same resource."
+    }
 
-case action
-  when "purge"
-    # If target is a valid URL, then assume we're purging a page and its contents.
-    if target =~ /^[a-z]+:\/\//
-      Varnisher::PagePurger.new target
-    end
+    option('q', 'ignore-query-strings') {
+      description "When given, /foo?foo=bar and /foo?foo=baz will be treated as the same resource."
+    }
 
-    # If target is a hostname, assume we want to purge an entire domain.
-    if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
-      Varnisher::DomainPurger.new target
+    def run
+      target = params['target'].value
+
+      Varnisher::Spider.new target
     end
-  
-  when "spider"
-    Varnisher::Spider.new target
+  end
 
-  when "reindex"
-    Varnisher::DomainPurger.new target
-    Varnisher::Spider.new target
+  def load_config
+    $options = params.to_options
+
+    rcfile = File.expand_path("~/.varnishrc")
+    if FileTest.readable? rcfile
+      rc = YAML::load(File.open(rcfile))
+      $options.merge!(rc)
+    end
+  end
+}
 
-  else
-    puts "Invalid action."
-end

data/lib/varnisher/domainpurger.rb

@@ -12,7 +12,7 @@ require 'net/http'
 module Varnisher
   class DomainPurger
     def initialize(domain)
-      s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
+      s = TCPSocket.open($options['hostname'], $options['port'])
       s.print("DOMAINPURGE / HTTP/1.1\r\nHost: #{domain}\r\n\r\n")
 
       if s.read =~ /HTTP\/1\.1 200 Purged\./

data/lib/varnisher/pagepurger.rb

@@ -19,13 +19,13 @@ module Varnisher
       # Then, do a fresh GET of the page and queue any resources we find on it.
       puts "Looking for external resources on #{@url}..."
 
-      if $options[:verbose]
+      if $options["verbose"]
         puts "\n\n"
       end
 
       fetch_page(@url)
 
-      if $options[:verbose]
+      if $options["verbose"]
         puts "\n"
       end
 
@@ -44,13 +44,13 @@ module Varnisher
       # Now, purge all of the resources we just queued.
       puts "Purging resources..."
 
-      if $options[:verbose]
+      if $options["verbose"]
         puts "\n\n"
       end
 
       purge_queue
 
-      if $options[:verbose]
+      if $options["verbose"]
         puts "\n"
       end
       
@@ -66,10 +66,10 @@ module Varnisher
         return
       end
 
-      s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
+      s = TCPSocket.open($options['hostname'], $options['port'])
       s.print("PURGE #{uri.path} HTTP/1.1\r\nHost: #{uri.host}\r\n\r\n")
 
-      if $options[:verbose]
+      if $options["verbose"]
         if s.read =~ /HTTP\/1\.1 200 Purged\./
           puts "Purged  #{url}"
         else
@@ -103,7 +103,7 @@ module Varnisher
       end
 
       find_resources(doc) do |resource|
-        if $options[:verbose]
+        if $options["verbose"]
             puts "Found #{resource}"
           end
         queue_resource(resource)
@@ -169,7 +169,7 @@ module Varnisher
     # Processes the queue of URLs, sending a purge request for each of them.
     def purge_queue()
       Parallel.map(@urls) { |url|
-        if $options[:verbose]
+        if $options["verbose"]
           puts "Purging #{url}..."
         end
 

data/lib/varnisher/spider.rb

@@ -63,7 +63,7 @@ module Varnisher
 
       @pages_hit += 1
 
-      if $options[:verbose]
+      if $options["verbose"]
         puts "Fetched #{url}..."
       end
 
@@ -123,12 +123,12 @@ module Varnisher
           # its original format.
 
           # Strip hash links
-          if ( $options[:ignore_hash] )
+          if ( $options["ignore-hashes"] )
             href.gsub!(/(#.*?)$/, '')
           end
 
           # Strip query strings
-          if ( $options[:ignore_query_string] )
+          if ( $options["ignore-query-strings"] )
             href.gsub!(/(\?.*?)$/, '')
           end
 
@@ -147,9 +147,12 @@ module Varnisher
     end
 
     def spider
-      Parallel.in_threads($options[:spider_threads]) { |thread_number|
+      threads = $options["threads"] || 16
+      num_pages = $options["num-pages"] || -1
+
+      Parallel.in_threads(threads) { |thread_number|
           # We've crawled too many pages
-          next if @pages_hit > $options[:num_pages] && $options[:num_pages] >= 0
+          next if @pages_hit > num_pages && num_pages >= 0
 
           while @to_visit.length > 0 do
             begin

data/lib/varnisher/version.rb

@@ -1,3 +1,3 @@
 module Varnisher
-  VERSION = "1.0.beta.1"
+  VERSION = "1.0.beta.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: varnisher
 version: !ruby/object:Gem::Version
-  version: 1.0.beta.1
+  version: 1.0.beta.2
 platform: ruby
 authors:
 - Rob Miller
@@ -10,6 +10,20 @@ bindir: bin
 cert_chain: []
 date: 2013-08-11 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: main
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 5.2.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 5.2.0
 - !ruby/object:Gem::Dependency
   name: hpricot
   requirement: !ruby/object:Gem::Requirement
@@ -38,6 +52,20 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.7.1
+- !ruby/object:Gem::Dependency
+  name: letters
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.4.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.4.1
 description: Some tools that make working with the Varnish HTTP cache easier, including
   things like doing mass purges of entire domains.
 email: rob@bigfish.co.uk