middleman-s3_sync 3.0.7 → 3.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +82 -1
- data/lib/middleman-s3_sync.rb +39 -42
- data/lib/middleman-s3_sync/commands.rb +6 -6
- data/lib/middleman-s3_sync/extension.rb +59 -2
- data/lib/middleman-s3_sync/version.rb +1 -1
- data/middleman-s3_sync.gemspec +3 -0
- metadata +52 -4
data/README.md
CHANGED
@@ -41,7 +41,85 @@ activate :s3_sync do |s3_sync|
|
|
41
41
|
end
|
42
42
|
```
|
43
43
|
|
44
|
-
You can then start synchronizing files with S3 through ```middleman s3_sync```.
|
44
|
+
You can then start synchronizing files with S3 through ```middleman s3_sync```.
|
45
|
+
|
46
|
+
## Push All Content to S3
|
47
|
+
|
48
|
+
There are situations where you might need to push the files to S3. In
|
49
|
+
such case, you can pass the ```--force``` option:
|
50
|
+
|
51
|
+
$ middleman s3_sync --force
|
52
|
+
|
53
|
+
## HTTP Caching
|
54
|
+
|
55
|
+
By default, ```middleman-s3_sync``` does not set caching headers. In
|
56
|
+
general, the default settings are sufficient. However, there are
|
57
|
+
situations where you might want to set a different HTTP caching policy.
|
58
|
+
This may be very helpful if you are using the ```asset_hash```
|
59
|
+
extension.
|
60
|
+
|
61
|
+
### Setting a policy based on the mime-type of a file
|
62
|
+
|
63
|
+
You can set a caching policy for every files that match a certain
|
64
|
+
mime-type. For example, setting max-age to 0 and kindly asking the
|
65
|
+
browser to revalidate the content for HTML files would take the
|
66
|
+
following form:
|
67
|
+
|
68
|
+
```ruby
|
69
|
+
caching_policy 'text/html', max_age: 0, must_revalidate: true
|
70
|
+
```
|
71
|
+
|
72
|
+
As a result, the following ```Cache-Control``` header would be set to ```max-age:0, must-revalidate```
|
73
|
+
|
74
|
+
### Setting a Default Policy
|
75
|
+
|
76
|
+
You can set the default policy by passing an options hash to ```default_caching_policy``` in your ```config.rb``` file:
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
default_caching_policy max_age:(60 * 60 * 24 * 365)
|
80
|
+
```
|
81
|
+
|
82
|
+
This will apply the policy to any file that do not have a mime-type
|
83
|
+
specific policy.
|
84
|
+
|
85
|
+
### Caching Policies
|
86
|
+
|
87
|
+
The [Caching Tutorial](http://www.mnot.net/cache_docs/) is a great
|
88
|
+
introduction to HTTP caching. The caching policy code in this gem is
|
89
|
+
based on it.
|
90
|
+
|
91
|
+
The following keys can be set:
|
92
|
+
|
93
|
+
| Key | Value | Header | Description |
|
94
|
+
| --- | ---- | ------ | ----------- |
|
95
|
+
| `max_age` | seconds | `max-age` | Specifies the maximum amount of time that a representation will be considered fresh. This value is relative to the time of the request |
|
96
|
+
| `s_maxage` | seconds | `s-maxage` | Only applies to shared (proxies) caches |
|
97
|
+
| `public` | boolean | `public` | Marks authenticated responses as cacheable. |
|
98
|
+
| `private` | boolean | `private` | Allows caches that are specific to one user to store the response. Shared caches (proxies) may not. |
|
99
|
+
| `no_cache` | boolean | `no-cache` | Forces caches to submit the request to the origin server for validation before releasing a cached copy, every time. |
|
100
|
+
| `no_store` | boolean | `no-store` | Instructs caches not to keep a copy of the representation under any conditions. |
|
101
|
+
| `must_revalidate` | boolean | `must-revalidate` | Tells the caches that they must obey any freshness information you give them about a representation. |
|
102
|
+
| `proxy_revalidate` | boolean | `proxy-revalidate` | Similar as `must-revalidate`, but only for proxies. |
|
103
|
+
|
104
|
+
### Setting `Expires` Header
|
105
|
+
|
106
|
+
You can pass the `expires` key to the `caching_policy` and
|
107
|
+
`default_caching_policy` methods if you insist on setting the expires
|
108
|
+
header on a results. You will need to pass it a Time object indicating
|
109
|
+
when the resourse is set to expire.
|
110
|
+
|
111
|
+
> Note that the `Cache-Control` header will take precedence over the
|
112
|
+
> `Expires` header if both are present.
|
113
|
+
|
114
|
+
### A Note About Browser Caching
|
115
|
+
|
116
|
+
Browser caching is well specified. It hasn't always been the case.
|
117
|
+
Still, even modern browsers have different behaviors if it suits it's
|
118
|
+
developers or their employers. Specs are meant to be ignored and so they
|
119
|
+
are (I'm looking at you Chrome!). Setting the `Cache-Control` or
|
120
|
+
`Expires` headers are not a guarrantie that the browsers and the proxies
|
121
|
+
that stand between them and your content will behave the way you want
|
122
|
+
them to. YMMV.
|
45
123
|
|
46
124
|
## A Debt of Gratitude
|
47
125
|
|
@@ -50,6 +128,9 @@ The code is well structured and easy to understand and it was easy to
|
|
50
128
|
extend it to add my synchronization code. My gratitude goes to @karlfreeman
|
51
129
|
and is work on Middleman sync.
|
52
130
|
|
131
|
+
Many thanks to [Gnip](http://gnip.com) and [dojo4](http://dojo4.com) for
|
132
|
+
supporting and sponsoring work on middleman-s3_sync.
|
133
|
+
|
53
134
|
## Contributing
|
54
135
|
|
55
136
|
1. Fork it
|
data/lib/middleman-s3_sync.rb
CHANGED
@@ -13,52 +13,53 @@ end
|
|
13
13
|
module Middleman
|
14
14
|
module S3Sync
|
15
15
|
class << self
|
16
|
-
LONG_TIME = 315360000 # 10 years
|
17
|
-
DONT_CACHE = ['html'] # types of files to exclude from browser cache
|
18
|
-
|
19
16
|
def sync
|
20
17
|
puts "Gathering local files."
|
21
18
|
|
22
19
|
local_files = (Dir[options.build_dir + "/**/*"] + Dir[options.build_dir + "/**/.*"])
|
23
20
|
.reject { |f| File.directory?(f) }
|
24
21
|
.map { |f| f.gsub(/^#{options.build_dir}\//, '') }
|
25
|
-
puts "Gathering remote files."
|
22
|
+
puts "Gathering remote files from #{options.bucket}"
|
26
23
|
remote_files = bucket.files.map { |f| f.key }
|
27
24
|
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
puts "Determine
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
files_to_reject = []
|
40
|
-
Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
|
41
|
-
print '.'
|
42
|
-
local_mtime = File.mtime("#{options.build_dir}/#{f}")
|
43
|
-
remote_mtime = s3_files.get(f).last_modified
|
44
|
-
files_to_reject << f if remote_mtime >= local_mtime
|
45
|
-
end
|
46
|
-
|
47
|
-
files_to_evaluate = files_to_evaluate - files_to_reject
|
25
|
+
if options.force
|
26
|
+
files_to_push = local_files
|
27
|
+
else
|
28
|
+
# First pass on the set of files to work with.
|
29
|
+
puts "Determine files to add to #{options.bucket}."
|
30
|
+
files_to_push = local_files - remote_files
|
31
|
+
if options.delete
|
32
|
+
puts "Determine which files to delete from #{options.bucket}"
|
33
|
+
files_to_delete = remote_files - local_files
|
34
|
+
end
|
35
|
+
files_to_evaluate = local_files - files_to_push
|
48
36
|
|
49
|
-
|
50
|
-
|
51
|
-
|
37
|
+
# No need to evaluate the files that are newer on S3 than the local files.
|
38
|
+
puts "Determine which local files are newer than their S3 counterparts"
|
39
|
+
files_to_reject = []
|
52
40
|
Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
|
53
41
|
print '.'
|
54
|
-
|
55
|
-
|
56
|
-
|
42
|
+
local_mtime = File.mtime("#{options.build_dir}/#{f}")
|
43
|
+
remote_mtime = s3_files.get(f).last_modified
|
44
|
+
files_to_reject << f if remote_mtime >= local_mtime
|
45
|
+
end
|
46
|
+
|
47
|
+
files_to_evaluate = files_to_evaluate - files_to_reject
|
48
|
+
|
49
|
+
# Are the files different? Use MD5 to see
|
50
|
+
if (files_to_evaluate.size > 0)
|
51
|
+
puts "\n\nDetermine which remaining files are actually different than their S3 counterpart."
|
52
|
+
Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
|
53
|
+
print '.'
|
54
|
+
local_md5 = Digest::MD5.hexdigest(File.read("#{options.build_dir}/#{f}"))
|
55
|
+
remote_md5 = s3_files.get(f).etag
|
56
|
+
files_to_push << f if local_md5 != remote_md5
|
57
|
+
end
|
57
58
|
end
|
58
59
|
end
|
59
60
|
|
60
61
|
if files_to_push.size > 0
|
61
|
-
puts "\n\nReady to apply updates to
|
62
|
+
puts "\n\nReady to apply updates to #{options.bucket}."
|
62
63
|
files_to_push.each do |f|
|
63
64
|
if remote_files.include?(f)
|
64
65
|
puts "Updating #{f}"
|
@@ -66,11 +67,9 @@ module Middleman
|
|
66
67
|
file.body = File.open("#{options.build_dir}/#{f}")
|
67
68
|
file.public = true
|
68
69
|
file.content_type = MIME::Types.of(f).first
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
file.cache_control = "public, max-age=#{LONG_TIME}"
|
73
|
-
file.expires = CGI.rfc1123_date(Time.now + LONG_TIME)
|
70
|
+
if policy = options.caching_policy_for(file.content_type)
|
71
|
+
file.cache_control = policy.cache_control if policy.cache_control
|
72
|
+
file.expires = policy.expires if policy.expires
|
74
73
|
end
|
75
74
|
|
76
75
|
file.save
|
@@ -85,13 +84,11 @@ module Middleman
|
|
85
84
|
}
|
86
85
|
|
87
86
|
# Add cache-control headers
|
88
|
-
|
89
|
-
|
90
|
-
file_hash.
|
91
|
-
:cache_control => "public, max-age=#{LONG_TIME}",
|
92
|
-
:expires => CGI.rfc1123_date(Time.now + LONG_TIME)
|
93
|
-
})
|
87
|
+
if policy = options.caching_policy_for(file_hash[:content_type])
|
88
|
+
file_hash[:cache_control] = policy.cache_control if policy.cache_control
|
89
|
+
file_hash[:expires] = policy.expires if policy.expires
|
94
90
|
end
|
91
|
+
|
95
92
|
file = bucket.files.create(file_hash)
|
96
93
|
end
|
97
94
|
end
|
@@ -5,9 +5,6 @@ module Middleman
|
|
5
5
|
module Cli
|
6
6
|
class S3Sync < Thor
|
7
7
|
include Thor::Actions
|
8
|
-
|
9
|
-
check_unknown_options!
|
10
|
-
|
11
8
|
namespace :s3_sync
|
12
9
|
|
13
10
|
def self.exit_on_failure?
|
@@ -15,17 +12,20 @@ module Middleman
|
|
15
12
|
end
|
16
13
|
|
17
14
|
desc "s3_sync", "Builds and push the minimum set of files needed to S3"
|
15
|
+
option :force, type: :boolean,
|
16
|
+
desc: "Push all local files to the server",
|
17
|
+
aliases: :f
|
18
18
|
def s3_sync
|
19
19
|
shared_inst = ::Middleman::Application.server.inst
|
20
20
|
bucket = shared_inst.options.bucket
|
21
21
|
if (!bucket)
|
22
22
|
raise Thor::Error.new "You need to activate this extension."
|
23
23
|
end
|
24
|
-
|
25
|
-
::Middleman::S3Sync.sync
|
26
24
|
|
27
|
-
|
25
|
+
shared_inst.options.force = options[:force] if options[:force]
|
28
26
|
|
27
|
+
::Middleman::S3Sync.sync
|
28
|
+
end
|
29
29
|
end
|
30
30
|
end
|
31
31
|
end
|
@@ -11,8 +11,57 @@ module Middleman
|
|
11
11
|
:after_build,
|
12
12
|
:delete,
|
13
13
|
:existing_remote_file,
|
14
|
-
:build_dir
|
14
|
+
:build_dir,
|
15
|
+
:force
|
15
16
|
)
|
17
|
+
|
18
|
+
def add_caching_policy(content_type, options)
|
19
|
+
caching_policies[content_type.to_s] = BrowserCachePolicy.new(options)
|
20
|
+
end
|
21
|
+
|
22
|
+
def caching_policy_for(content_type)
|
23
|
+
caching_policies.fetch(content_type.to_s, caching_policies[:default])
|
24
|
+
end
|
25
|
+
|
26
|
+
def default_caching_policy
|
27
|
+
caching_policies[:default]
|
28
|
+
end
|
29
|
+
|
30
|
+
def caching_policies
|
31
|
+
@caching_policies ||= Map.new
|
32
|
+
end
|
33
|
+
|
34
|
+
protected
|
35
|
+
class BrowserCachePolicy
|
36
|
+
attr_accessor :policies
|
37
|
+
|
38
|
+
def initialize(options)
|
39
|
+
@policies = Map.from_hash(options)
|
40
|
+
end
|
41
|
+
|
42
|
+
def cache_control
|
43
|
+
policy = []
|
44
|
+
policy << "max-age=#{policies.max_age}" if policies.has_key?(:max_age)
|
45
|
+
policy << "s-maxage=#{s_maxage}" if policies.has_key?(:s_maxage)
|
46
|
+
policy << "public" if policies.fetch(:public, false)
|
47
|
+
policy << "private" if policies.fetch(:private, false)
|
48
|
+
policy << "no-cache" if policies.fetch(:no_cache, false)
|
49
|
+
policy << "no-store" if policies.fetch(:no_store, false)
|
50
|
+
policy << "must-revalidate" if policies.fetch(:must_revalidate, false)
|
51
|
+
policy << "proxy-revalidate" if policies.fetch(:proxy_revalidate, false)
|
52
|
+
if policy.empty?
|
53
|
+
nil
|
54
|
+
else
|
55
|
+
policy.join(", ")
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
def expires
|
60
|
+
if expiration = policies.fetch(:expires, nil)
|
61
|
+
CGI.rfc1123_date(expiration)
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
16
65
|
end
|
17
66
|
|
18
67
|
class << self
|
@@ -29,7 +78,7 @@ module Middleman
|
|
29
78
|
app.send :include, Helpers
|
30
79
|
|
31
80
|
app.after_configuration do |config|
|
32
|
-
options.build_dir
|
81
|
+
options.build_dir ||= build_dir
|
33
82
|
end
|
34
83
|
|
35
84
|
app.after_build do |builder|
|
@@ -43,6 +92,14 @@ module Middleman
|
|
43
92
|
def options
|
44
93
|
::Middleman::S3Sync.options
|
45
94
|
end
|
95
|
+
|
96
|
+
def default_caching_policy(policy = {})
|
97
|
+
options.add_caching_policy(:default, policy)
|
98
|
+
end
|
99
|
+
|
100
|
+
def caching_policy(content_type, policy = {})
|
101
|
+
options.add_caching_policy(content_type, policy)
|
102
|
+
end
|
46
103
|
end
|
47
104
|
end
|
48
105
|
end
|
data/middleman-s3_sync.gemspec
CHANGED
@@ -20,6 +20,9 @@ Gem::Specification.new do |gem|
|
|
20
20
|
gem.add_runtime_dependency 'middleman-core', '>= 3.0.0'
|
21
21
|
gem.add_runtime_dependency 'fog'
|
22
22
|
gem.add_runtime_dependency 'parallel'
|
23
|
+
gem.add_runtime_dependency 'map'
|
23
24
|
|
24
25
|
gem.add_development_dependency 'rake'
|
26
|
+
gem.add_development_dependency 'pry'
|
27
|
+
gem.add_development_dependency 'pry-nav'
|
25
28
|
end
|
metadata
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
name: middleman-s3_sync
|
3
3
|
version: !ruby/object:Gem::Version
|
4
4
|
prerelease:
|
5
|
-
version: 3.0.
|
5
|
+
version: 3.0.8
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
8
|
- Frederic Jean
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2013-03-
|
13
|
+
date: 2013-03-29 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -60,6 +60,22 @@ dependencies:
|
|
60
60
|
- !ruby/object:Gem::Version
|
61
61
|
version: '0'
|
62
62
|
type: :runtime
|
63
|
+
- !ruby/object:Gem::Dependency
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
none: false
|
66
|
+
requirements:
|
67
|
+
- - ! '>='
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '0'
|
70
|
+
name: map
|
71
|
+
prerelease: false
|
72
|
+
requirement: !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ! '>='
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: '0'
|
78
|
+
type: :runtime
|
63
79
|
- !ruby/object:Gem::Dependency
|
64
80
|
version_requirements: !ruby/object:Gem::Requirement
|
65
81
|
none: false
|
@@ -76,6 +92,38 @@ dependencies:
|
|
76
92
|
- !ruby/object:Gem::Version
|
77
93
|
version: '0'
|
78
94
|
type: :development
|
95
|
+
- !ruby/object:Gem::Dependency
|
96
|
+
version_requirements: !ruby/object:Gem::Requirement
|
97
|
+
none: false
|
98
|
+
requirements:
|
99
|
+
- - ! '>='
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
name: pry
|
103
|
+
prerelease: false
|
104
|
+
requirement: !ruby/object:Gem::Requirement
|
105
|
+
none: false
|
106
|
+
requirements:
|
107
|
+
- - ! '>='
|
108
|
+
- !ruby/object:Gem::Version
|
109
|
+
version: '0'
|
110
|
+
type: :development
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
version_requirements: !ruby/object:Gem::Requirement
|
113
|
+
none: false
|
114
|
+
requirements:
|
115
|
+
- - ! '>='
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
name: pry-nav
|
119
|
+
prerelease: false
|
120
|
+
requirement: !ruby/object:Gem::Requirement
|
121
|
+
none: false
|
122
|
+
requirements:
|
123
|
+
- - ! '>='
|
124
|
+
- !ruby/object:Gem::Version
|
125
|
+
version: '0'
|
126
|
+
type: :development
|
79
127
|
description: Only syncs files that have been updated to S3.
|
80
128
|
email:
|
81
129
|
- fred@fredjean.net
|
@@ -107,7 +155,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
107
155
|
- !ruby/object:Gem::Version
|
108
156
|
segments:
|
109
157
|
- 0
|
110
|
-
hash: -
|
158
|
+
hash: -1761006957673169609
|
111
159
|
version: '0'
|
112
160
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
113
161
|
none: false
|
@@ -116,7 +164,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
116
164
|
- !ruby/object:Gem::Version
|
117
165
|
segments:
|
118
166
|
- 0
|
119
|
-
hash: -
|
167
|
+
hash: -1761006957673169609
|
120
168
|
version: '0'
|
121
169
|
requirements: []
|
122
170
|
rubyforge_project:
|