varnisher 1.0.beta.1 → 1.0.beta.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +75 -29
- data/bin/varnisher +80 -69
- data/lib/varnisher/domainpurger.rb +1 -1
- data/lib/varnisher/pagepurger.rb +8 -8
- data/lib/varnisher/spider.rb +8 -5
- data/lib/varnisher/version.rb +1 -1
- metadata +29 -1
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4a9d710584e6d43f0f925cd4894f536fa1630551
|
4
|
+
data.tar.gz: 9d22ce540f64d36683db840c395c03c1096a88cd
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7afaedc98a7557689c4908da0ba13e54f674d70ec3fd48543f0bf4181b016d61d05f532196683f3dfd97ff8e44ddde417afb4e116bfdf941af73068664531327
|
7
|
+
data.tar.gz: 55cd0e503a1152418c84e3a1876ee75a38b2c6450ea5e9495b0cd2d2d55bf608d8ef07939e84f170cfc19d2c665f8863cafb5e6aeb21241a71e8fc71caa706f2
|
data/README.md
CHANGED
@@ -1,23 +1,53 @@
|
|
1
|
-
#
|
1
|
+
# varnisher
|
2
2
|
|
3
|
-
Administering Varnish is generally a breeze, but sometimes you want to
|
3
|
+
Administering Varnish is generally a breeze, but sometimes you want to
|
4
|
+
do one of the few things that aren't painless out of the box. Hopefully,
|
5
|
+
that's where this toolbox comes in.
|
4
6
|
|
5
|
-
|
7
|
+
Varnisher lets you do things like:
|
6
8
|
|
7
|
-
|
9
|
+
* Purge a webpage and everything (e.g. images, JavaScript files, CSS
|
10
|
+
files) referenced on that page
|
11
|
+
* Spider an entire domain — useful for priming a cache
|
12
|
+
* Purge an entire domain, including optionally re-spidering it
|
13
|
+
afterwards to keep the cache warm
|
14
|
+
|
15
|
+
## Installation
|
16
|
+
|
17
|
+
Varnish requires Ruby >1.9.3 to run. If you've got a recent Ruby
|
18
|
+
installed, then Varnisher can be installed by running:
|
19
|
+
|
20
|
+
gem install varnisher
|
8
21
|
|
9
22
|
## Usage
|
10
23
|
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
24
|
+
NAME
|
25
|
+
varnisher
|
26
|
+
|
27
|
+
SYNOPSIS
|
28
|
+
varnisher (purge|spider) [options]+
|
29
|
+
|
30
|
+
DESCRIPTION
|
31
|
+
Varnisher is a set of tools for working with the Varnish HTTP cache.
|
32
|
+
|
33
|
+
PARAMETERS
|
34
|
+
--verbose, -v
|
35
|
+
If given, Varnisher will be noisier about what it's up to.
|
36
|
+
--hostname=hostname, -H (0 ~> hostname=localhost)
|
37
|
+
The hostname/IP address of your Varnish server.
|
38
|
+
--port=port, -p (0 ~> int(port=80))
|
39
|
+
The port Varnish is listening on.
|
40
|
+
--help, -h
|
41
|
+
|
42
|
+
EXAMPLES
|
43
|
+
varnisher purge http://example.com
|
44
|
+
varnisher spider example.com
|
45
|
+
varnisher purge --reindex example.com
|
19
46
|
|
20
|
-
If you find yourself typing certain parameters every time you use the
|
47
|
+
If you find yourself typing certain parameters every time you use the
|
48
|
+
script, you can specify them in an RC file called `.varnishrc` in your
|
49
|
+
home directory. The file format is YAML and the default options are, if
|
50
|
+
you want to paste and override them:
|
21
51
|
|
22
52
|
verbose: false
|
23
53
|
hostname: localhost
|
@@ -30,17 +60,25 @@ If you find yourself typing certain parameters every time you use the script, yo
|
|
30
60
|
|
31
61
|
### Purging a page and all the resources on it
|
32
62
|
|
33
|
-
Quite often, it's necessary redevelop a page on a website in a way that
|
63
|
+
Quite often, it's necessary redevelop a page on a website in a way that
|
64
|
+
involves changes not only to the page but also to CSS files, images,
|
65
|
+
JavaScript files, etc. Purging pages in this instance can be a painful
|
66
|
+
process, or at least one that requires a few `ban` commands in
|
67
|
+
`varnishadm`. No longer!
|
34
68
|
|
35
69
|
Just enter:
|
36
70
|
|
37
|
-
$
|
71
|
+
$ varnisher purge http://www.example.com/path/to/page
|
38
72
|
|
39
|
-
...and `/path/to/page`, along with all its images, CSS files, JavaScript
|
73
|
+
...and `/path/to/page`, along with all its images, CSS files, JavaScript
|
74
|
+
files, and other external accoutrements, will be purged from Varnish's
|
75
|
+
cache.
|
40
76
|
|
41
|
-
As a bonus, this action is multithreaded, meaning even resource-heavy
|
77
|
+
As a bonus, this action is multithreaded, meaning even resource-heavy
|
78
|
+
pages should purge quickly and evenly.
|
42
79
|
|
43
|
-
This action requires your VCL to have something like the following,
|
80
|
+
This action requires your VCL to have something like the following,
|
81
|
+
which is fairly standard:
|
44
82
|
|
45
83
|
if (req.request == "PURGE") {
|
46
84
|
if ( client.ip ~ auth ) {
|
@@ -49,7 +87,9 @@ This action requires your VCL to have something like the following, which is fai
|
|
49
87
|
}
|
50
88
|
}
|
51
89
|
|
52
|
-
(For an explanation of just what `obj.http.x-url` means, and why you
|
90
|
+
(For an explanation of just what `obj.http.x-url` means, and why you
|
91
|
+
should use it rather than `req.url`, see [this
|
92
|
+
page](http://kristianlyng.wordpress.com/2010/07/28/smart-bans-with-varnish/).)
|
53
93
|
|
54
94
|
### Purging an entire domain
|
55
95
|
|
@@ -62,25 +102,31 @@ Provided your VCL has something akin to the following in it:
|
|
62
102
|
}
|
63
103
|
}
|
64
104
|
|
65
|
-
...then you should be able to quickly purge an entire domain's worth of
|
105
|
+
...then you should be able to quickly purge an entire domain's worth of
|
106
|
+
pages and resources by simply issuing the command:
|
66
107
|
|
67
|
-
$
|
108
|
+
$ varnisher purge www.example.com
|
68
109
|
|
69
110
|
### Repopulating the cache
|
70
111
|
|
71
|
-
If you've purged a whole domain, and particularly if your backend is
|
112
|
+
If you've purged a whole domain, and particularly if your backend is
|
113
|
+
slow, you might want to quickly repopulate the cache so that users never
|
114
|
+
see your slow misses. Well, you can! Use the `spider` action:
|
72
115
|
|
73
|
-
$
|
116
|
+
$ varnisher spider www.example.com
|
74
117
|
|
75
|
-
`spider` accepts either a hostname or a URL as its starting point, and
|
118
|
+
`spider` accepts either a hostname or a URL as its starting point, and
|
119
|
+
will only fetch pages on the same domain as its origin. You can limit
|
120
|
+
the number of pages it will process using the `-n` parameter:
|
76
121
|
|
77
|
-
$
|
122
|
+
$ varnisher -n 500 spider www.example.com
|
78
123
|
|
79
|
-
If you'd like to combine purging and spidering, you can use the
|
124
|
+
If you'd like to combine purging and spidering, you can use the
|
125
|
+
`reindex` option:
|
80
126
|
|
81
|
-
$
|
127
|
+
$ varnisher purge --reindex www.example.com
|
82
128
|
|
83
129
|
…which is functionally equivalent to:
|
84
130
|
|
85
|
-
$
|
86
|
-
$
|
131
|
+
$ varnisher purge www.example.com
|
132
|
+
$ varnisher spider www.example.com
|
data/bin/varnisher
CHANGED
@@ -1,96 +1,107 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
require '
|
3
|
+
require 'bundler'
|
4
|
+
Bundler.setup
|
5
|
+
|
6
|
+
require 'main'
|
4
7
|
require 'yaml'
|
5
8
|
|
6
9
|
require 'varnisher'
|
7
10
|
|
8
|
-
|
9
|
-
|
10
|
-
:hostname => 'localhost',
|
11
|
-
:port => 80,
|
12
|
-
:num_pages => 100,
|
13
|
-
:ignore_hash => true,
|
14
|
-
:spider_threads => 16,
|
15
|
-
:ignore_query_string => false
|
16
|
-
}
|
11
|
+
Main {
|
12
|
+
examples "varnisher purge http://example.com", "varnisher spider example.com", "varnisher purge --reindex example.com"
|
17
13
|
|
18
|
-
|
19
|
-
if FileTest.readable? rcfile
|
20
|
-
rc = YAML::load(File.open(rcfile))
|
21
|
-
# Convert to symbols
|
22
|
-
rc = rc.inject({}){ |memo,(k,v)| memo[k.to_sym] = v; memo }
|
23
|
-
$options.merge!(rc)
|
24
|
-
end
|
14
|
+
description "Varnisher is a set of tools for working with the Varnish HTTP cache."
|
25
15
|
|
26
|
-
|
27
|
-
opts.banner = 'Usage: varnish.rb [options] action target'
|
16
|
+
argument 'target'
|
28
17
|
|
29
|
-
|
30
|
-
|
31
|
-
|
18
|
+
option('v', 'verbose') {
|
19
|
+
description "If given, Varnisher will be noisier about what it's up to."
|
20
|
+
}
|
32
21
|
|
33
|
-
|
34
|
-
|
35
|
-
|
22
|
+
option('H', 'hostname') {
|
23
|
+
argument :required
|
24
|
+
description "The hostname/IP address of your Varnish server."
|
25
|
+
default "localhost"
|
26
|
+
}
|
36
27
|
|
37
|
-
|
38
|
-
|
39
|
-
|
28
|
+
option('p', 'port') {
|
29
|
+
argument :required
|
30
|
+
cast :int
|
31
|
+
description "The port Varnish is listening on."
|
32
|
+
default 80
|
33
|
+
}
|
40
34
|
|
41
|
-
|
42
|
-
|
35
|
+
def before_run
|
36
|
+
load_config
|
43
37
|
end
|
44
38
|
|
45
|
-
|
46
|
-
|
47
|
-
end
|
39
|
+
mode "purge" do
|
40
|
+
argument('target') { description "The URL or hostname to purge" }
|
48
41
|
|
49
|
-
|
50
|
-
|
51
|
-
|
42
|
+
option('reindex') {
|
43
|
+
description "If you specify a hostname to purge, this option will respider that hostname after the purging is complete. This will keep your cache as warm as possible."
|
44
|
+
}
|
52
45
|
|
53
|
-
|
54
|
-
|
55
|
-
|
46
|
+
def run
|
47
|
+
target = params['target'].value
|
48
|
+
|
49
|
+
# If target is a valid URL, then assume we're purging a page and its contents.
|
50
|
+
if target =~ /^[a-z]+:\/\//
|
51
|
+
Varnisher::PagePurger.new target
|
52
|
+
end
|
56
53
|
|
57
|
-
|
58
|
-
|
54
|
+
# If target is a hostname, assume we want to purge an entire domain.
|
55
|
+
if target =~ /^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\-]*[A-Za-z0-9])$/
|
56
|
+
Varnisher::DomainPurger.new target
|
57
|
+
|
58
|
+
if params['reindex'].given?
|
59
|
+
Varnisher::Spider.new "http://#{target}/"
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
59
63
|
end
|
60
|
-
end
|
61
64
|
|
62
|
-
|
65
|
+
mode "spider" do
|
66
|
+
argument('target') { description "The URL to begin spidering from." }
|
63
67
|
|
64
|
-
|
65
|
-
|
66
|
-
|
68
|
+
option('n', 'num-pages') {
|
69
|
+
argument :required
|
70
|
+
cast :int
|
71
|
+
description "Maximum number of pages to crawl. Setting this to -1 (the default) will impose no limit."
|
72
|
+
default -1
|
73
|
+
}
|
67
74
|
|
68
|
-
|
69
|
-
|
70
|
-
|
75
|
+
option('t', 'threads') {
|
76
|
+
argument :required
|
77
|
+
cast :int
|
78
|
+
description "Spidering is done in parallel; this variable controls how many threads will be used."
|
79
|
+
default 16
|
80
|
+
}
|
71
81
|
|
72
|
-
|
73
|
-
|
82
|
+
option('#', 'ignore-hashes') {
|
83
|
+
description "When given, /foo#foo and /foo#bar will be treated as separate URLs; the default is to treat them as the same resource."
|
84
|
+
}
|
74
85
|
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
if target =~ /^[a-z]+:\/\//
|
79
|
-
Varnisher::PagePurger.new target
|
80
|
-
end
|
86
|
+
option('q', 'ignore-query-strings') {
|
87
|
+
description "When given, /foo?foo=bar and /foo?foo=baz will be treated as the same resource."
|
88
|
+
}
|
81
89
|
|
82
|
-
|
83
|
-
|
84
|
-
|
90
|
+
def run
|
91
|
+
target = params['target'].value
|
92
|
+
|
93
|
+
Varnisher::Spider.new target
|
85
94
|
end
|
86
|
-
|
87
|
-
when "spider"
|
88
|
-
Varnisher::Spider.new target
|
95
|
+
end
|
89
96
|
|
90
|
-
|
91
|
-
|
92
|
-
|
97
|
+
def load_config
|
98
|
+
$options = params.to_options
|
99
|
+
|
100
|
+
rcfile = File.expand_path("~/.varnishrc")
|
101
|
+
if FileTest.readable? rcfile
|
102
|
+
rc = YAML::load(File.open(rcfile))
|
103
|
+
$options.merge!(rc)
|
104
|
+
end
|
105
|
+
end
|
106
|
+
}
|
93
107
|
|
94
|
-
else
|
95
|
-
puts "Invalid action."
|
96
|
-
end
|
@@ -12,7 +12,7 @@ require 'net/http'
|
|
12
12
|
module Varnisher
|
13
13
|
class DomainPurger
|
14
14
|
def initialize(domain)
|
15
|
-
s = TCPSocket.open(
|
15
|
+
s = TCPSocket.open($options['hostname'], $options['port'])
|
16
16
|
s.print("DOMAINPURGE / HTTP/1.1\r\nHost: #{domain}\r\n\r\n")
|
17
17
|
|
18
18
|
if s.read =~ /HTTP\/1\.1 200 Purged\./
|
data/lib/varnisher/pagepurger.rb
CHANGED
@@ -19,13 +19,13 @@ module Varnisher
|
|
19
19
|
# Then, do a fresh GET of the page and queue any resources we find on it.
|
20
20
|
puts "Looking for external resources on #{@url}..."
|
21
21
|
|
22
|
-
if $options[
|
22
|
+
if $options["verbose"]
|
23
23
|
puts "\n\n"
|
24
24
|
end
|
25
25
|
|
26
26
|
fetch_page(@url)
|
27
27
|
|
28
|
-
if $options[
|
28
|
+
if $options["verbose"]
|
29
29
|
puts "\n"
|
30
30
|
end
|
31
31
|
|
@@ -44,13 +44,13 @@ module Varnisher
|
|
44
44
|
# Now, purge all of the resources we just queued.
|
45
45
|
puts "Purging resources..."
|
46
46
|
|
47
|
-
if $options[
|
47
|
+
if $options["verbose"]
|
48
48
|
puts "\n\n"
|
49
49
|
end
|
50
50
|
|
51
51
|
purge_queue
|
52
52
|
|
53
|
-
if $options[
|
53
|
+
if $options["verbose"]
|
54
54
|
puts "\n"
|
55
55
|
end
|
56
56
|
|
@@ -66,10 +66,10 @@ module Varnisher
|
|
66
66
|
return
|
67
67
|
end
|
68
68
|
|
69
|
-
s = TCPSocket.open(
|
69
|
+
s = TCPSocket.open($options['hostname'], $options['port'])
|
70
70
|
s.print("PURGE #{uri.path} HTTP/1.1\r\nHost: #{uri.host}\r\n\r\n")
|
71
71
|
|
72
|
-
if $options[
|
72
|
+
if $options["verbose"]
|
73
73
|
if s.read =~ /HTTP\/1\.1 200 Purged\./
|
74
74
|
puts "Purged #{url}"
|
75
75
|
else
|
@@ -103,7 +103,7 @@ module Varnisher
|
|
103
103
|
end
|
104
104
|
|
105
105
|
find_resources(doc) do |resource|
|
106
|
-
if $options[
|
106
|
+
if $options["verbose"]
|
107
107
|
puts "Found #{resource}"
|
108
108
|
end
|
109
109
|
queue_resource(resource)
|
@@ -169,7 +169,7 @@ module Varnisher
|
|
169
169
|
# Processes the queue of URLs, sending a purge request for each of them.
|
170
170
|
def purge_queue()
|
171
171
|
Parallel.map(@urls) { |url|
|
172
|
-
if $options[
|
172
|
+
if $options["verbose"]
|
173
173
|
puts "Purging #{url}..."
|
174
174
|
end
|
175
175
|
|
data/lib/varnisher/spider.rb
CHANGED
@@ -63,7 +63,7 @@ module Varnisher
|
|
63
63
|
|
64
64
|
@pages_hit += 1
|
65
65
|
|
66
|
-
if $options[
|
66
|
+
if $options["verbose"]
|
67
67
|
puts "Fetched #{url}..."
|
68
68
|
end
|
69
69
|
|
@@ -123,12 +123,12 @@ module Varnisher
|
|
123
123
|
# its original format.
|
124
124
|
|
125
125
|
# Strip hash links
|
126
|
-
if ( $options[
|
126
|
+
if ( $options["ignore-hashes"] )
|
127
127
|
href.gsub!(/(#.*?)$/, '')
|
128
128
|
end
|
129
129
|
|
130
130
|
# Strip query strings
|
131
|
-
if ( $options[
|
131
|
+
if ( $options["ignore-query-strings"] )
|
132
132
|
href.gsub!(/(\?.*?)$/, '')
|
133
133
|
end
|
134
134
|
|
@@ -147,9 +147,12 @@ module Varnisher
|
|
147
147
|
end
|
148
148
|
|
149
149
|
def spider
|
150
|
-
|
150
|
+
threads = $options["threads"] || 16
|
151
|
+
num_pages = $options["num-pages"] || -1
|
152
|
+
|
153
|
+
Parallel.in_threads(threads) { |thread_number|
|
151
154
|
# We've crawled too many pages
|
152
|
-
next if @pages_hit >
|
155
|
+
next if @pages_hit > num_pages && num_pages >= 0
|
153
156
|
|
154
157
|
while @to_visit.length > 0 do
|
155
158
|
begin
|
data/lib/varnisher/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: varnisher
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.beta.
|
4
|
+
version: 1.0.beta.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Rob Miller
|
@@ -10,6 +10,20 @@ bindir: bin
|
|
10
10
|
cert_chain: []
|
11
11
|
date: 2013-08-11 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: main
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ~>
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 5.2.0
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ~>
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: 5.2.0
|
13
27
|
- !ruby/object:Gem::Dependency
|
14
28
|
name: hpricot
|
15
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -38,6 +52,20 @@ dependencies:
|
|
38
52
|
- - ~>
|
39
53
|
- !ruby/object:Gem::Version
|
40
54
|
version: 0.7.1
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: letters
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ~>
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 0.4.1
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ~>
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: 0.4.1
|
41
69
|
description: Some tools that make working with the Varnish HTTP cache easier, including
|
42
70
|
things like doing mass purges of entire domains.
|
43
71
|
email: rob@bigfish.co.uk
|