varnisher 1.0.beta.1 → 1.0.beta.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 89bc9abf14d10e978e9d63af4652deebd7912205
4
- data.tar.gz: 3074562679934cc26c5b413a281fe9a4aef31f23
3
+ metadata.gz: 4a9d710584e6d43f0f925cd4894f536fa1630551
4
+ data.tar.gz: 9d22ce540f64d36683db840c395c03c1096a88cd
5
5
  SHA512:
6
- metadata.gz: ec489165fc9a6b70f8824cfef3b962657467ad319dc584a4478497d76eedf8c0adf74d6af8b1b4201935038a57affc156d1e33ec41927f1eb39d5cac6147a112
7
- data.tar.gz: b41896764dfab0db49743b7d30236de8f6ea30810b72f6e135d8b252ac8bd9b58c91c65fd9c10a61c7e2cb16bc7f7a869a3999d2f2e2dacdfd57abe59852bca6
6
+ metadata.gz: 7afaedc98a7557689c4908da0ba13e54f674d70ec3fd48543f0bf4181b016d61d05f532196683f3dfd97ff8e44ddde417afb4e116bfdf941af73068664531327
7
+ data.tar.gz: 55cd0e503a1152418c84e3a1876ee75a38b2c6450ea5e9495b0cd2d2d55bf608d8ef07939e84f170cfc19d2c665f8863cafb5e6aeb21241a71e8fc71caa706f2
data/README.md CHANGED
@@ -1,23 +1,53 @@
1
- # Varnish Toolkit
1
+ # varnisher
2
2
 
3
- Administering Varnish is generally a breeze, but sometimes you want to do one of the few things that aren't painless out of the box. Hopefully, that's where this toolbox comes in.
3
+ Administering Varnish is generally a breeze, but sometimes you want to
4
+ do one of the few things that aren't painless out of the box. Hopefully,
5
+ that's where this toolbox comes in.
4
6
 
5
- Varnish Toolkit relies on the wonderful libraries [hpricot](http://hpricot.com/), by why the lucky stiff, and [Parallel](https://github.com/grosser/parallel), by Michael Grosser. If you don't have them, install them with:
7
+ Varnisher lets you do things like:
6
8
 
7
- $ sudo gem install hpricot parallel
9
+ * Purge a webpage and everything (e.g. images, JavaScript files, CSS
10
+ files) referenced on that page
11
+ * Spider an entire domain — useful for priming a cache
12
+ * Purge an entire domain, including optionally re-spidering it
13
+ afterwards to keep the cache warm
14
+
15
+ ## Installation
16
+
17
+ Varnish requires Ruby >1.9.3 to run. If you've got a recent Ruby
18
+ installed, then Varnisher can be installed by running:
19
+
20
+ gem install varnisher
8
21
 
9
22
  ## Usage
10
23
 
11
- Usage: varnish.rb [options] action target
12
- -h, --help Display this help
13
- -v, --verbose Output more information
14
- -H, --hostname HOSTNAME Hostname/IP address of your Varnish server. Default is localhost
15
- -p, --port PORT Port your Varnish server is listening on. Default is 80
16
- -n, --num-pages NUM Number of pages to crawl when in spider mode. -1 will crawl all pages
17
- -#, --hashes If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode
18
- -q, --ignore-query-string If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode
24
+ NAME
25
+ varnisher
26
+
27
+ SYNOPSIS
28
+ varnisher (purge|spider) [options]+
29
+
30
+ DESCRIPTION
31
+ Varnisher is a set of tools for working with the Varnish HTTP cache.
32
+
33
+ PARAMETERS
34
+ --verbose, -v
35
+ If given, Varnisher will be noisier about what it's up to.
36
+ --hostname=hostname, -H (0 ~> hostname=localhost)
37
+ The hostname/IP address of your Varnish server.
38
+ --port=port, -p (0 ~> int(port=80))
39
+ The port Varnish is listening on.
40
+ --help, -h
41
+
42
+ EXAMPLES
43
+ varnisher purge http://example.com
44
+ varnisher spider example.com
45
+ varnisher purge --reindex example.com
19
46
 
20
- If you find yourself typing certain parameters every time you use the script, you can specify them in an RC file called `.varnishrc` in your home directory. The file format is YAML and the default options are, if you want to paste and override them:
47
+ If you find yourself typing certain parameters every time you use the
48
+ script, you can specify them in an RC file called `.varnishrc` in your
49
+ home directory. The file format is YAML and the default options are, if
50
+ you want to paste and override them:
21
51
 
22
52
  verbose: false
23
53
  hostname: localhost
@@ -30,17 +60,25 @@ If you find yourself typing certain parameters every time you use the script, yo
30
60
 
31
61
  ### Purging a page and all the resources on it
32
62
 
33
- Quite often, it's necessary redevelop a page on a website in a way that involves changes not only to the page but also to CSS files, images, JavaScript files, etc. Purging pages in this instance can be a painful process, or at least one that requires a few `ban` commands in `varnishadm`. No longer!
63
+ Quite often, it's necessary redevelop a page on a website in a way that
64
+ involves changes not only to the page but also to CSS files, images,
65
+ JavaScript files, etc. Purging pages in this instance can be a painful
66
+ process, or at least one that requires a few `ban` commands in
67
+ `varnishadm`. No longer!
34
68
 
35
69
  Just enter:
36
70
 
37
- $ varnish.rb purge http://www.example.com/path/to/page
71
+ $ varnisher purge http://www.example.com/path/to/page
38
72
 
39
- ...and `/path/to/page`, along with all its images, CSS files, JavaScript files, and other external accoutrements, will be purged from Varnish's cache.
73
+ ...and `/path/to/page`, along with all its images, CSS files, JavaScript
74
+ files, and other external accoutrements, will be purged from Varnish's
75
+ cache.
40
76
 
41
- As a bonus, this action is multithreaded, meaning even resource-heavy pages should purge quickly and evenly.
77
+ As a bonus, this action is multithreaded, meaning even resource-heavy
78
+ pages should purge quickly and evenly.
42
79
 
43
- This action requires your VCL to have something like the following, which is fairly standard:
80
+ This action requires your VCL to have something like the following,
81
+ which is fairly standard:
44
82
 
45
83
  if (req.request == "PURGE") {
46
84
  if ( client.ip ~ auth ) {
@@ -49,7 +87,9 @@ This action requires your VCL to have something like the following, which is fai
49
87
  }
50
88
  }
51
89
 
52
- (For an explanation of just what `obj.http.x-url` means, and why you should use it rather than `req.url`, see [this page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
90
+ (For an explanation of just what `obj.http.x-url` means, and why you
91
+ should use it rather than `req.url`, see [this
92
+ page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
53
93
 
54
94
  ### Purging an entire domain
55
95
 
@@ -62,25 +102,31 @@ Provided your VCL has something akin to the following in it:
62
102
  }
63
103
  }
64
104
 
65
- ...then you should be able to quickly purge an entire domain's worth of pages and resources by simply issuing the command:
105
+ ...then you should be able to quickly purge an entire domain's worth of
106
+ pages and resources by simply issuing the command:
66
107
 
67
- $ varnish.rb purge www.example.com
108
+ $ varnisher purge www.example.com
68
109
 
69
110
  ### Repopulating the cache
70
111
 
71
- If you've purged a whole domain, and particularly if your backend is slow, you might want to quickly repopulate the cache so that users never see your slow misses. Well, you can! Use the `spider` action:
112
+ If you've purged a whole domain, and particularly if your backend is
113
+ slow, you might want to quickly repopulate the cache so that users never
114
+ see your slow misses. Well, you can! Use the `spider` action:
72
115
 
73
- $ varnish.rb spider www.example.com
116
+ $ varnisher spider www.example.com
74
117
 
75
- `spider` accepts either a hostname or a URL as its starting point, and will only fetch pages on the same domain as its origin. You can limit the number of pages it will process using the `-n` parameter:
118
+ `spider` accepts either a hostname or a URL as its starting point, and
119
+ will only fetch pages on the same domain as its origin. You can limit
120
+ the number of pages it will process using the `-n` parameter:
76
121
 
77
- $ varnish.rb -n 500 spider www.example.com
122
+ $ varnisher -n 500 spider www.example.com
78
123
 
79
- If you'd like to combine purging and spidering, you can use the `reindex` action:
124
+ If you'd like to combine purging and spidering, you can use the
125
+ `reindex` option:
80
126
 
81
- $ varnish.rb reindex www.example.com
127
+ $ varnisher purge --reindex www.example.com
82
128
 
83
129
  …which is functionally equivalent to:
84
130
 
85
- $ varnish.rb purge www.example.com
86
- $ varnish.rb spider www.example.com
131
+ $ varnisher purge www.example.com
132
+ $ varnisher spider www.example.com
data/bin/varnisher CHANGED
@@ -1,96 +1,107 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- require 'optparse'
3
+ require 'bundler'
4
+ Bundler.setup
5
+
6
+ require 'main'
4
7
  require 'yaml'
5
8
 
6
9
  require 'varnisher'
7
10
 
8
- $options = {
9
- :verbose => false,
10
- :hostname => 'localhost',
11
- :port => 80,
12
- :num_pages => 100,
13
- :ignore_hash => true,
14
- :spider_threads => 16,
15
- :ignore_query_string => false
16
- }
11
+ Main {
12
+ examples "varnisher purge http://example.com", "varnisher spider example.com", "varnisher purge --reindex example.com"
17
13
 
18
- rcfile = File.expand_path("~/.varnishrc")
19
- if FileTest.readable? rcfile
20
- rc = YAML::load(File.open(rcfile))
21
- # Convert to symbols
22
- rc = rc.inject({}){ |memo,(k,v)| memo[k.to_sym] = v; memo }
23
- $options.merge!(rc)
24
- end
14
+ description "Varnisher is a set of tools for working with the Varnish HTTP cache."
25
15
 
26
- optparse = OptionParser.new do |opts|
27
- opts.banner = 'Usage: varnish.rb [options] action target'
16
+ argument 'target'
28
17
 
29
- opts.on('-h', '--help', 'Display this help') do
30
- puts opts
31
- end
18
+ option('v', 'verbose') {
19
+ description "If given, Varnisher will be noisier about what it's up to."
20
+ }
32
21
 
33
- opts.on('-v', '--verbose', 'Output more information') do
34
- $options[:verbose] = true
35
- end
22
+ option('H', 'hostname') {
23
+ argument :required
24
+ description "The hostname/IP address of your Varnish server."
25
+ default "localhost"
26
+ }
36
27
 
37
- opts.on('-H', '--hostname HOSTNAME', 'Hostname/IP address of your Varnish server. Default is localhost') do |hostname|
38
- $options[:hostname] = hostname
39
- end
28
+ option('p', 'port') {
29
+ argument :required
30
+ cast :int
31
+ description "The port Varnish is listening on."
32
+ default 80
33
+ }
40
34
 
41
- opts.on('-p', '--port PORT', 'Port your Varnish server is listening on. Default is 80') do |port|
42
- $options[:port] = port
35
+ def before_run
36
+ load_config
43
37
  end
44
38
 
45
- opts.on('-n', '--num-pages NUM', 'Number of pages to crawl when in spider mode. -1 will crawl all pages') do |num|
46
- $options[:num_pages] = num.to_i
47
- end
39
+ mode "purge" do
40
+ argument('target') { description "The URL or hostname to purge" }
48
41
 
49
- opts.on('-t', '--spider-threads NUM', 'Number of threads to use when spidering. Default is 16') do |num|
50
- $options[:spider_threads] = num.to_i
51
- end
42
+ option('reindex') {
43
+ description "If you specify a hostname to purge, this option will respider that hostname after the purging is complete. This will keep your cache as warm as possible."
44
+ }
52
45
 
53
- opts.on('-#', '--hashes', 'If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode') do
54
- $options[:ignore_hash] = false
55
- end
46
+ def run
47
+ target = params['target'].value
48
+
49
+ # If target is a valid URL, then assume we're purging a page and its contents.
50
+ if target =~ /^[a-z]+:\/\//
51
+ Varnisher::PagePurger.new target
52
+ end
56
53
 
57
- opts.on('-q', '--ignore-query-string', 'If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode') do
58
- $options[:ignore_query_string] = true
54
+ # If target is a hostname, assume we want to purge an entire domain.
55
+ if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
56
+ Varnisher::DomainPurger.new target
57
+
58
+ if params['reindex'].given?
59
+ Varnisher::Spider.new "http://#{target}/"
60
+ end
61
+ end
62
+ end
59
63
  end
60
- end
61
64
 
62
- optparse.parse!
65
+ mode "spider" do
66
+ argument('target') { description "The URL to begin spidering from." }
63
67
 
64
- # All our libs use these constants.
65
- PROXY_HOSTNAME = $options[:hostname]
66
- PROXY_PORT = $options[:port].to_i
68
+ option('n', 'num-pages') {
69
+ argument :required
70
+ cast :int
71
+ description "Maximum number of pages to crawl. Setting this to -1 (the default) will impose no limit."
72
+ default -1
73
+ }
67
74
 
68
- if ( ARGV.length < 2 )
69
- puts "You must specify both an action and a target."
70
- end
75
+ option('t', 'threads') {
76
+ argument :required
77
+ cast :int
78
+ description "Spidering is done in parallel; this variable controls how many threads will be used."
79
+ default 16
80
+ }
71
81
 
72
- action = ARGV[0]
73
- target = ARGV[1]
82
+ option('#', 'ignore-hashes') {
83
+ description "When given, /foo#foo and /foo#bar will be treated as separate URLs; the default is to treat them as the same resource."
84
+ }
74
85
 
75
- case action
76
- when "purge"
77
- # If target is a valid URL, then assume we're purging a page and its contents.
78
- if target =~ /^[a-z]+:\/\//
79
- Varnisher::PagePurger.new target
80
- end
86
+ option('q', 'ignore-query-strings') {
87
+ description "When given, /foo?foo=bar and /foo?foo=baz will be treated as the same resource."
88
+ }
81
89
 
82
- # If target is a hostname, assume we want to purge an entire domain.
83
- if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
84
- Varnisher::DomainPurger.new target
90
+ def run
91
+ target = params['target'].value
92
+
93
+ Varnisher::Spider.new target
85
94
  end
86
-
87
- when "spider"
88
- Varnisher::Spider.new target
95
+ end
89
96
 
90
- when "reindex"
91
- Varnisher::DomainPurger.new target
92
- Varnisher::Spider.new target
97
+ def load_config
98
+ $options = params.to_options
99
+
100
+ rcfile = File.expand_path("~/.varnishrc")
101
+ if FileTest.readable? rcfile
102
+ rc = YAML::load(File.open(rcfile))
103
+ $options.merge!(rc)
104
+ end
105
+ end
106
+ }
93
107
 
94
- else
95
- puts "Invalid action."
96
- end
@@ -12,7 +12,7 @@ require 'net/http'
12
12
  module Varnisher
13
13
  class DomainPurger
14
14
  def initialize(domain)
15
- s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
15
+ s = TCPSocket.open($options['hostname'], $options['port'])
16
16
  s.print("DOMAINPURGE / HTTP/1.1\r\nHost: #{domain}\r\n\r\n")
17
17
 
18
18
  if s.read =~ /HTTP\/1\.1 200 Purged\./
@@ -19,13 +19,13 @@ module Varnisher
19
19
  # Then, do a fresh GET of the page and queue any resources we find on it.
20
20
  puts "Looking for external resources on #{@url}..."
21
21
 
22
- if $options[:verbose]
22
+ if $options["verbose"]
23
23
  puts "\n\n"
24
24
  end
25
25
 
26
26
  fetch_page(@url)
27
27
 
28
- if $options[:verbose]
28
+ if $options["verbose"]
29
29
  puts "\n"
30
30
  end
31
31
 
@@ -44,13 +44,13 @@ module Varnisher
44
44
  # Now, purge all of the resources we just queued.
45
45
  puts "Purging resources..."
46
46
 
47
- if $options[:verbose]
47
+ if $options["verbose"]
48
48
  puts "\n\n"
49
49
  end
50
50
 
51
51
  purge_queue
52
52
 
53
- if $options[:verbose]
53
+ if $options["verbose"]
54
54
  puts "\n"
55
55
  end
56
56
 
@@ -66,10 +66,10 @@ module Varnisher
66
66
  return
67
67
  end
68
68
 
69
- s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
69
+ s = TCPSocket.open($options['hostname'], $options['port'])
70
70
  s.print("PURGE #{uri.path} HTTP/1.1\r\nHost: #{uri.host}\r\n\r\n")
71
71
 
72
- if $options[:verbose]
72
+ if $options["verbose"]
73
73
  if s.read =~ /HTTP\/1\.1 200 Purged\./
74
74
  puts "Purged #{url}"
75
75
  else
@@ -103,7 +103,7 @@ module Varnisher
103
103
  end
104
104
 
105
105
  find_resources(doc) do |resource|
106
- if $options[:verbose]
106
+ if $options["verbose"]
107
107
  puts "Found #{resource}"
108
108
  end
109
109
  queue_resource(resource)
@@ -169,7 +169,7 @@ module Varnisher
169
169
  # Processes the queue of URLs, sending a purge request for each of them.
170
170
  def purge_queue()
171
171
  Parallel.map(@urls) { |url|
172
- if $options[:verbose]
172
+ if $options["verbose"]
173
173
  puts "Purging #{url}..."
174
174
  end
175
175
 
@@ -63,7 +63,7 @@ module Varnisher
63
63
 
64
64
  @pages_hit += 1
65
65
 
66
- if $options[:verbose]
66
+ if $options["verbose"]
67
67
  puts "Fetched #{url}..."
68
68
  end
69
69
 
@@ -123,12 +123,12 @@ module Varnisher
123
123
  # its original format.
124
124
 
125
125
  # Strip hash links
126
- if ( $options[:ignore_hash] )
126
+ if ( $options["ignore-hashes"] )
127
127
  href.gsub!(/(#.*?)$/, '')
128
128
  end
129
129
 
130
130
  # Strip query strings
131
- if ( $options[:ignore_query_string] )
131
+ if ( $options["ignore-query-strings"] )
132
132
  href.gsub!(/(\?.*?)$/, '')
133
133
  end
134
134
 
@@ -147,9 +147,12 @@ module Varnisher
147
147
  end
148
148
 
149
149
  def spider
150
- Parallel.in_threads($options[:spider_threads]) { |thread_number|
150
+ threads = $options["threads"] || 16
151
+ num_pages = $options["num-pages"] || -1
152
+
153
+ Parallel.in_threads(threads) { |thread_number|
151
154
  # We've crawled too many pages
152
- next if @pages_hit > $options[:num_pages] && $options[:num_pages] >= 0
155
+ next if @pages_hit > num_pages && num_pages >= 0
153
156
 
154
157
  while @to_visit.length > 0 do
155
158
  begin
@@ -1,3 +1,3 @@
1
1
  module Varnisher
2
- VERSION = "1.0.beta.1"
2
+ VERSION = "1.0.beta.2"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: varnisher
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.beta.1
4
+ version: 1.0.beta.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Rob Miller
@@ -10,6 +10,20 @@ bindir: bin
10
10
  cert_chain: []
11
11
  date: 2013-08-11 00:00:00.000000000 Z
12
12
  dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: main
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ~>
18
+ - !ruby/object:Gem::Version
19
+ version: 5.2.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ~>
25
+ - !ruby/object:Gem::Version
26
+ version: 5.2.0
13
27
  - !ruby/object:Gem::Dependency
14
28
  name: hpricot
15
29
  requirement: !ruby/object:Gem::Requirement
@@ -38,6 +52,20 @@ dependencies:
38
52
  - - ~>
39
53
  - !ruby/object:Gem::Version
40
54
  version: 0.7.1
55
+ - !ruby/object:Gem::Dependency
56
+ name: letters
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ~>
60
+ - !ruby/object:Gem::Version
61
+ version: 0.4.1
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ~>
67
+ - !ruby/object:Gem::Version
68
+ version: 0.4.1
41
69
  description: Some tools that make working with the Varnish HTTP cache easier, including
42
70
  things like doing mass purges of entire domains.
43
71
  email: rob@bigfish.co.uk