varnisher 1.0.beta.1 → 1.0.beta.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 89bc9abf14d10e978e9d63af4652deebd7912205
4
- data.tar.gz: 3074562679934cc26c5b413a281fe9a4aef31f23
3
+ metadata.gz: 4a9d710584e6d43f0f925cd4894f536fa1630551
4
+ data.tar.gz: 9d22ce540f64d36683db840c395c03c1096a88cd
5
5
  SHA512:
6
- metadata.gz: ec489165fc9a6b70f8824cfef3b962657467ad319dc584a4478497d76eedf8c0adf74d6af8b1b4201935038a57affc156d1e33ec41927f1eb39d5cac6147a112
7
- data.tar.gz: b41896764dfab0db49743b7d30236de8f6ea30810b72f6e135d8b252ac8bd9b58c91c65fd9c10a61c7e2cb16bc7f7a869a3999d2f2e2dacdfd57abe59852bca6
6
+ metadata.gz: 7afaedc98a7557689c4908da0ba13e54f674d70ec3fd48543f0bf4181b016d61d05f532196683f3dfd97ff8e44ddde417afb4e116bfdf941af73068664531327
7
+ data.tar.gz: 55cd0e503a1152418c84e3a1876ee75a38b2c6450ea5e9495b0cd2d2d55bf608d8ef07939e84f170cfc19d2c665f8863cafb5e6aeb21241a71e8fc71caa706f2
data/README.md CHANGED
@@ -1,23 +1,53 @@
1
- # Varnish Toolkit
1
+ # varnisher
2
2
 
3
- Administering Varnish is generally a breeze, but sometimes you want to do one of the few things that aren't painless out of the box. Hopefully, that's where this toolbox comes in.
3
+ Administering Varnish is generally a breeze, but sometimes you want to
4
+ do one of the few things that aren't painless out of the box. Hopefully,
5
+ that's where this toolbox comes in.
4
6
 
5
- Varnish Toolkit relies on the wonderful libraries [hpricot](http://hpricot.com/), by why the lucky stiff, and [Parallel](https://github.com/grosser/parallel), by Michael Grosser. If you don't have them, install them with:
7
+ Varnisher lets you do things like:
6
8
 
7
- $ sudo gem install hpricot parallel
9
+ * Purge a webpage and everything (e.g. images, JavaScript files, CSS
10
+ files) referenced on that page
11
+ * Spider an entire domain — useful for priming a cache
12
+ * Purge an entire domain, including optionally re-spidering it
13
+ afterwards to keep the cache warm
14
+
15
+ ## Installation
16
+
17
+ Varnish requires Ruby >1.9.3 to run. If you've got a recent Ruby
18
+ installed, then Varnisher can be installed by running:
19
+
20
+ gem install varnisher
8
21
 
9
22
  ## Usage
10
23
 
11
- Usage: varnish.rb [options] action target
12
- -h, --help Display this help
13
- -v, --verbose Output more information
14
- -H, --hostname HOSTNAME Hostname/IP address of your Varnish server. Default is localhost
15
- -p, --port PORT Port your Varnish server is listening on. Default is 80
16
- -n, --num-pages NUM Number of pages to crawl when in spider mode. -1 will crawl all pages
17
- -#, --hashes If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode
18
- -q, --ignore-query-string If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode
24
+ NAME
25
+ varnisher
26
+
27
+ SYNOPSIS
28
+ varnisher (purge|spider) [options]+
29
+
30
+ DESCRIPTION
31
+ Varnisher is a set of tools for working with the Varnish HTTP cache.
32
+
33
+ PARAMETERS
34
+ --verbose, -v
35
+ If given, Varnisher will be noisier about what it's up to.
36
+ --hostname=hostname, -H (0 ~> hostname=localhost)
37
+ The hostname/IP address of your Varnish server.
38
+ --port=port, -p (0 ~> int(port=80))
39
+ The port Varnish is listening on.
40
+ --help, -h
41
+
42
+ EXAMPLES
43
+ varnisher purge http://example.com
44
+ varnisher spider example.com
45
+ varnisher purge --reindex example.com
19
46
 
20
- If you find yourself typing certain parameters every time you use the script, you can specify them in an RC file called `.varnishrc` in your home directory. The file format is YAML and the default options are, if you want to paste and override them:
47
+ If you find yourself typing certain parameters every time you use the
48
+ script, you can specify them in an RC file called `.varnishrc` in your
49
+ home directory. The file format is YAML and the default options are, if
50
+ you want to paste and override them:
21
51
 
22
52
  verbose: false
23
53
  hostname: localhost
@@ -30,17 +60,25 @@ If you find yourself typing certain parameters every time you use the script, yo
30
60
 
31
61
  ### Purging a page and all the resources on it
32
62
 
33
- Quite often, it's necessary redevelop a page on a website in a way that involves changes not only to the page but also to CSS files, images, JavaScript files, etc. Purging pages in this instance can be a painful process, or at least one that requires a few `ban` commands in `varnishadm`. No longer!
63
+ Quite often, it's necessary redevelop a page on a website in a way that
64
+ involves changes not only to the page but also to CSS files, images,
65
+ JavaScript files, etc. Purging pages in this instance can be a painful
66
+ process, or at least one that requires a few `ban` commands in
67
+ `varnishadm`. No longer!
34
68
 
35
69
  Just enter:
36
70
 
37
- $ varnish.rb purge http://www.example.com/path/to/page
71
+ $ varnisher purge http://www.example.com/path/to/page
38
72
 
39
- ...and `/path/to/page`, along with all its images, CSS files, JavaScript files, and other external accoutrements, will be purged from Varnish's cache.
73
+ ...and `/path/to/page`, along with all its images, CSS files, JavaScript
74
+ files, and other external accoutrements, will be purged from Varnish's
75
+ cache.
40
76
 
41
- As a bonus, this action is multithreaded, meaning even resource-heavy pages should purge quickly and evenly.
77
+ As a bonus, this action is multithreaded, meaning even resource-heavy
78
+ pages should purge quickly and evenly.
42
79
 
43
- This action requires your VCL to have something like the following, which is fairly standard:
80
+ This action requires your VCL to have something like the following,
81
+ which is fairly standard:
44
82
 
45
83
  if (req.request == "PURGE") {
46
84
  if ( client.ip ~ auth ) {
@@ -49,7 +87,9 @@ This action requires your VCL to have something like the following, which is fai
49
87
  }
50
88
  }
51
89
 
52
- (For an explanation of just what `obj.http.x-url` means, and why you should use it rather than `req.url`, see [this page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
90
+ (For an explanation of just what `obj.http.x-url` means, and why you
91
+ should use it rather than `req.url`, see [this
92
+ page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
53
93
 
54
94
  ### Purging an entire domain
55
95
 
@@ -62,25 +102,31 @@ Provided your VCL has something akin to the following in it:
62
102
  }
63
103
  }
64
104
 
65
- ...then you should be able to quickly purge an entire domain's worth of pages and resources by simply issuing the command:
105
+ ...then you should be able to quickly purge an entire domain's worth of
106
+ pages and resources by simply issuing the command:
66
107
 
67
- $ varnish.rb purge www.example.com
108
+ $ varnisher purge www.example.com
68
109
 
69
110
  ### Repopulating the cache
70
111
 
71
- If you've purged a whole domain, and particularly if your backend is slow, you might want to quickly repopulate the cache so that users never see your slow misses. Well, you can! Use the `spider` action:
112
+ If you've purged a whole domain, and particularly if your backend is
113
+ slow, you might want to quickly repopulate the cache so that users never
114
+ see your slow misses. Well, you can! Use the `spider` action:
72
115
 
73
- $ varnish.rb spider www.example.com
116
+ $ varnisher spider www.example.com
74
117
 
75
- `spider` accepts either a hostname or a URL as its starting point, and will only fetch pages on the same domain as its origin. You can limit the number of pages it will process using the `-n` parameter:
118
+ `spider` accepts either a hostname or a URL as its starting point, and
119
+ will only fetch pages on the same domain as its origin. You can limit
120
+ the number of pages it will process using the `-n` parameter:
76
121
 
77
- $ varnish.rb -n 500 spider www.example.com
122
+ $ varnisher -n 500 spider www.example.com
78
123
 
79
- If you'd like to combine purging and spidering, you can use the `reindex` action:
124
+ If you'd like to combine purging and spidering, you can use the
125
+ `reindex` option:
80
126
 
81
- $ varnish.rb reindex www.example.com
127
+ $ varnisher purge --reindex www.example.com
82
128
 
83
129
  …which is functionally equivalent to:
84
130
 
85
- $ varnish.rb purge www.example.com
86
- $ varnish.rb spider www.example.com
131
+ $ varnisher purge www.example.com
132
+ $ varnisher spider www.example.com
data/bin/varnisher CHANGED
@@ -1,96 +1,107 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- require 'optparse'
3
+ require 'bundler'
4
+ Bundler.setup
5
+
6
+ require 'main'
4
7
  require 'yaml'
5
8
 
6
9
  require 'varnisher'
7
10
 
8
- $options = {
9
- :verbose => false,
10
- :hostname => 'localhost',
11
- :port => 80,
12
- :num_pages => 100,
13
- :ignore_hash => true,
14
- :spider_threads => 16,
15
- :ignore_query_string => false
16
- }
11
+ Main {
12
+ examples "varnisher purge http://example.com", "varnisher spider example.com", "varnisher purge --reindex example.com"
17
13
 
18
- rcfile = File.expand_path("~/.varnishrc")
19
- if FileTest.readable? rcfile
20
- rc = YAML::load(File.open(rcfile))
21
- # Convert to symbols
22
- rc = rc.inject({}){ |memo,(k,v)| memo[k.to_sym] = v; memo }
23
- $options.merge!(rc)
24
- end
14
+ description "Varnisher is a set of tools for working with the Varnish HTTP cache."
25
15
 
26
- optparse = OptionParser.new do |opts|
27
- opts.banner = 'Usage: varnish.rb [options] action target'
16
+ argument 'target'
28
17
 
29
- opts.on('-h', '--help', 'Display this help') do
30
- puts opts
31
- end
18
+ option('v', 'verbose') {
19
+ description "If given, Varnisher will be noisier about what it's up to."
20
+ }
32
21
 
33
- opts.on('-v', '--verbose', 'Output more information') do
34
- $options[:verbose] = true
35
- end
22
+ option('H', 'hostname') {
23
+ argument :required
24
+ description "The hostname/IP address of your Varnish server."
25
+ default "localhost"
26
+ }
36
27
 
37
- opts.on('-H', '--hostname HOSTNAME', 'Hostname/IP address of your Varnish server. Default is localhost') do |hostname|
38
- $options[:hostname] = hostname
39
- end
28
+ option('p', 'port') {
29
+ argument :required
30
+ cast :int
31
+ description "The port Varnish is listening on."
32
+ default 80
33
+ }
40
34
 
41
- opts.on('-p', '--port PORT', 'Port your Varnish server is listening on. Default is 80') do |port|
42
- $options[:port] = port
35
+ def before_run
36
+ load_config
43
37
  end
44
38
 
45
- opts.on('-n', '--num-pages NUM', 'Number of pages to crawl when in spider mode. -1 will crawl all pages') do |num|
46
- $options[:num_pages] = num.to_i
47
- end
39
+ mode "purge" do
40
+ argument('target') { description "The URL or hostname to purge" }
48
41
 
49
- opts.on('-t', '--spider-threads NUM', 'Number of threads to use when spidering. Default is 16') do |num|
50
- $options[:spider_threads] = num.to_i
51
- end
42
+ option('reindex') {
43
+ description "If you specify a hostname to purge, this option will respider that hostname after the purging is complete. This will keep your cache as warm as possible."
44
+ }
52
45
 
53
- opts.on('-#', '--hashes', 'If true, /foo.html#foo and /foo.html#bar will be seen as different in spider mode') do
54
- $options[:ignore_hash] = false
55
- end
46
+ def run
47
+ target = params['target'].value
48
+
49
+ # If target is a valid URL, then assume we're purging a page and its contents.
50
+ if target =~ /^[a-z]+:\/\//
51
+ Varnisher::PagePurger.new target
52
+ end
56
53
 
57
- opts.on('-q', '--ignore-query-string', 'If true, /foo?foo=bar and /foo?foo=baz will be seen as the same in spider mode') do
58
- $options[:ignore_query_string] = true
54
+ # If target is a hostname, assume we want to purge an entire domain.
55
+ if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
56
+ Varnisher::DomainPurger.new target
57
+
58
+ if params['reindex'].given?
59
+ Varnisher::Spider.new "http://#{target}/"
60
+ end
61
+ end
62
+ end
59
63
  end
60
- end
61
64
 
62
- optparse.parse!
65
+ mode "spider" do
66
+ argument('target') { description "The URL to begin spidering from." }
63
67
 
64
- # All our libs use these constants.
65
- PROXY_HOSTNAME = $options[:hostname]
66
- PROXY_PORT = $options[:port].to_i
68
+ option('n', 'num-pages') {
69
+ argument :required
70
+ cast :int
71
+ description "Maximum number of pages to crawl. Setting this to -1 (the default) will impose no limit."
72
+ default -1
73
+ }
67
74
 
68
- if ( ARGV.length < 2 )
69
- puts "You must specify both an action and a target."
70
- end
75
+ option('t', 'threads') {
76
+ argument :required
77
+ cast :int
78
+ description "Spidering is done in parallel; this variable controls how many threads will be used."
79
+ default 16
80
+ }
71
81
 
72
- action = ARGV[0]
73
- target = ARGV[1]
82
+ option('#', 'ignore-hashes') {
83
+ description "When given, /foo#foo and /foo#bar will be treated as separate URLs; the default is to treat them as the same resource."
84
+ }
74
85
 
75
- case action
76
- when "purge"
77
- # If target is a valid URL, then assume we're purging a page and its contents.
78
- if target =~ /^[a-z]+:\/\//
79
- Varnisher::PagePurger.new target
80
- end
86
+ option('q', 'ignore-query-strings') {
87
+ description "When given, /foo?foo=bar and /foo?foo=baz will be treated as the same resource."
88
+ }
81
89
 
82
- # If target is a hostname, assume we want to purge an entire domain.
83
- if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
84
- Varnisher::DomainPurger.new target
90
+ def run
91
+ target = params['target'].value
92
+
93
+ Varnisher::Spider.new target
85
94
  end
86
-
87
- when "spider"
88
- Varnisher::Spider.new target
95
+ end
89
96
 
90
- when "reindex"
91
- Varnisher::DomainPurger.new target
92
- Varnisher::Spider.new target
97
+ def load_config
98
+ $options = params.to_options
99
+
100
+ rcfile = File.expand_path("~/.varnishrc")
101
+ if FileTest.readable? rcfile
102
+ rc = YAML::load(File.open(rcfile))
103
+ $options.merge!(rc)
104
+ end
105
+ end
106
+ }
93
107
 
94
- else
95
- puts "Invalid action."
96
- end
@@ -12,7 +12,7 @@ require 'net/http'
12
12
  module Varnisher
13
13
  class DomainPurger
14
14
  def initialize(domain)
15
- s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
15
+ s = TCPSocket.open($options['hostname'], $options['port'])
16
16
  s.print("DOMAINPURGE / HTTP/1.1\r\nHost: #{domain}\r\n\r\n")
17
17
 
18
18
  if s.read =~ /HTTP\/1\.1 200 Purged\./
@@ -19,13 +19,13 @@ module Varnisher
19
19
  # Then, do a fresh GET of the page and queue any resources we find on it.
20
20
  puts "Looking for external resources on #{@url}..."
21
21
 
22
- if $options[:verbose]
22
+ if $options["verbose"]
23
23
  puts "\n\n"
24
24
  end
25
25
 
26
26
  fetch_page(@url)
27
27
 
28
- if $options[:verbose]
28
+ if $options["verbose"]
29
29
  puts "\n"
30
30
  end
31
31
 
@@ -44,13 +44,13 @@ module Varnisher
44
44
  # Now, purge all of the resources we just queued.
45
45
  puts "Purging resources..."
46
46
 
47
- if $options[:verbose]
47
+ if $options["verbose"]
48
48
  puts "\n\n"
49
49
  end
50
50
 
51
51
  purge_queue
52
52
 
53
- if $options[:verbose]
53
+ if $options["verbose"]
54
54
  puts "\n"
55
55
  end
56
56
 
@@ -66,10 +66,10 @@ module Varnisher
66
66
  return
67
67
  end
68
68
 
69
- s = TCPSocket.open(PROXY_HOSTNAME, PROXY_PORT)
69
+ s = TCPSocket.open($options['hostname'], $options['port'])
70
70
  s.print("PURGE #{uri.path} HTTP/1.1\r\nHost: #{uri.host}\r\n\r\n")
71
71
 
72
- if $options[:verbose]
72
+ if $options["verbose"]
73
73
  if s.read =~ /HTTP\/1\.1 200 Purged\./
74
74
  puts "Purged #{url}"
75
75
  else
@@ -103,7 +103,7 @@ module Varnisher
103
103
  end
104
104
 
105
105
  find_resources(doc) do |resource|
106
- if $options[:verbose]
106
+ if $options["verbose"]
107
107
  puts "Found #{resource}"
108
108
  end
109
109
  queue_resource(resource)
@@ -169,7 +169,7 @@ module Varnisher
169
169
  # Processes the queue of URLs, sending a purge request for each of them.
170
170
  def purge_queue()
171
171
  Parallel.map(@urls) { |url|
172
- if $options[:verbose]
172
+ if $options["verbose"]
173
173
  puts "Purging #{url}..."
174
174
  end
175
175
 
@@ -63,7 +63,7 @@ module Varnisher
63
63
 
64
64
  @pages_hit += 1
65
65
 
66
- if $options[:verbose]
66
+ if $options["verbose"]
67
67
  puts "Fetched #{url}..."
68
68
  end
69
69
 
@@ -123,12 +123,12 @@ module Varnisher
123
123
  # its original format.
124
124
 
125
125
  # Strip hash links
126
- if ( $options[:ignore_hash] )
126
+ if ( $options["ignore-hashes"] )
127
127
  href.gsub!(/(#.*?)$/, '')
128
128
  end
129
129
 
130
130
  # Strip query strings
131
- if ( $options[:ignore_query_string] )
131
+ if ( $options["ignore-query-strings"] )
132
132
  href.gsub!(/(\?.*?)$/, '')
133
133
  end
134
134
 
@@ -147,9 +147,12 @@ module Varnisher
147
147
  end
148
148
 
149
149
  def spider
150
- Parallel.in_threads($options[:spider_threads]) { |thread_number|
150
+ threads = $options["threads"] || 16
151
+ num_pages = $options["num-pages"] || -1
152
+
153
+ Parallel.in_threads(threads) { |thread_number|
151
154
  # We've crawled too many pages
152
- next if @pages_hit > $options[:num_pages] && $options[:num_pages] >= 0
155
+ next if @pages_hit > num_pages && num_pages >= 0
153
156
 
154
157
  while @to_visit.length > 0 do
155
158
  begin
@@ -1,3 +1,3 @@
1
1
  module Varnisher
2
- VERSION = "1.0.beta.1"
2
+ VERSION = "1.0.beta.2"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: varnisher
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.beta.1
4
+ version: 1.0.beta.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Rob Miller
@@ -10,6 +10,20 @@ bindir: bin
10
10
  cert_chain: []
11
11
  date: 2013-08-11 00:00:00.000000000 Z
12
12
  dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: main
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ~>
18
+ - !ruby/object:Gem::Version
19
+ version: 5.2.0
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ~>
25
+ - !ruby/object:Gem::Version
26
+ version: 5.2.0
13
27
  - !ruby/object:Gem::Dependency
14
28
  name: hpricot
15
29
  requirement: !ruby/object:Gem::Requirement
@@ -38,6 +52,20 @@ dependencies:
38
52
  - - ~>
39
53
  - !ruby/object:Gem::Version
40
54
  version: 0.7.1
55
+ - !ruby/object:Gem::Dependency
56
+ name: letters
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ~>
60
+ - !ruby/object:Gem::Version
61
+ version: 0.4.1
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ~>
67
+ - !ruby/object:Gem::Version
68
+ version: 0.4.1
41
69
  description: Some tools that make working with the Varnish HTTP cache easier, including
42
70
  things like doing mass purges of entire domains.
43
71
  email: rob@bigfish.co.uk