middleman-s3_sync 3.0.7 → 3.0.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -41,7 +41,85 @@ activate :s3_sync do |s3_sync|
41
41
  end
42
42
  ```
43
43
 
44
- You can then start synchronizing files with S3 through ```middleman s3_sync```.
44
+ You can then start synchronizing files with S3 through ```middleman s3_sync```.
45
+
46
+ ## Push All Content to S3
47
+
48
+ There are situations where you might need to push the files to S3. In
49
+ such case, you can pass the ```--force``` option:
50
+
51
+ $ middleman s3_sync --force
52
+
53
+ ## HTTP Caching
54
+
55
+ By default, ```middleman-s3_sync``` does not set caching headers. In
56
+ general, the default settings are sufficient. However, there are
57
+ situations where you might want to set a different HTTP caching policy.
58
+ This may be very helpful if you are using the ```asset_hash```
59
+ extension.
60
+
61
+ ### Setting a policy based on the mime-type of a file
62
+
63
+ You can set a caching policy for every files that match a certain
64
+ mime-type. For example, setting max-age to 0 and kindly asking the
65
+ browser to revalidate the content for HTML files would take the
66
+ following form:
67
+
68
+ ```ruby
69
+ caching_policy 'text/html', max_age: 0, must_revalidate: true
70
+ ```
71
+
72
+ As a result, the following ```Cache-Control``` header would be set to ```max-age:0, must-revalidate```
73
+
74
+ ### Setting a Default Policy
75
+
76
+ You can set the default policy by passing an options hash to ```default_caching_policy``` in your ```config.rb``` file:
77
+
78
+ ```ruby
79
+ default_caching_policy max_age:(60 * 60 * 24 * 365)
80
+ ```
81
+
82
+ This will apply the policy to any file that do not have a mime-type
83
+ specific policy.
84
+
85
+ ### Caching Policies
86
+
87
+ The [Caching Tutorial](http://www.mnot.net/cache_docs/) is a great
88
+ introduction to HTTP caching. The caching policy code in this gem is
89
+ based on it.
90
+
91
+ The following keys can be set:
92
+
93
+ | Key | Value | Header | Description |
94
+ | --- | ---- | ------ | ----------- |
95
+ | `max_age` | seconds | `max-age` | Specifies the maximum amount of time that a representation will be considered fresh. This value is relative to the time of the request |
96
+ | `s_maxage` | seconds | `s-maxage` | Only applies to shared (proxies) caches |
97
+ | `public` | boolean | `public` | Marks authenticated responses as cacheable. |
98
+ | `private` | boolean | `private` | Allows caches that are specific to one user to store the response. Shared caches (proxies) may not. |
99
+ | `no_cache` | boolean | `no-cache` | Forces caches to submit the request to the origin server for validation before releasing a cached copy, every time. |
100
+ | `no_store` | boolean | `no-store` | Instructs caches not to keep a copy of the representation under any conditions. |
101
+ | `must_revalidate` | boolean | `must-revalidate` | Tells the caches that they must obey any freshness information you give them about a representation. |
102
+ | `proxy_revalidate` | boolean | `proxy-revalidate` | Similar as `must-revalidate`, but only for proxies. |
103
+
104
+ ### Setting `Expires` Header
105
+
106
+ You can pass the `expires` key to the `caching_policy` and
107
+ `default_caching_policy` methods if you insist on setting the expires
108
+ header on a results. You will need to pass it a Time object indicating
109
+ when the resourse is set to expire.
110
+
111
+ > Note that the `Cache-Control` header will take precedence over the
112
+ > `Expires` header if both are present.
113
+
114
+ ### A Note About Browser Caching
115
+
116
+ Browser caching is well specified. It hasn't always been the case.
117
+ Still, even modern browsers have different behaviors if it suits it's
118
+ developers or their employers. Specs are meant to be ignored and so they
119
+ are (I'm looking at you Chrome!). Setting the `Cache-Control` or
120
+ `Expires` headers are not a guarrantie that the browsers and the proxies
121
+ that stand between them and your content will behave the way you want
122
+ them to. YMMV.
45
123
 
46
124
  ## A Debt of Gratitude
47
125
 
@@ -50,6 +128,9 @@ The code is well structured and easy to understand and it was easy to
50
128
  extend it to add my synchronization code. My gratitude goes to @karlfreeman
51
129
  and is work on Middleman sync.
52
130
 
131
+ Many thanks to [Gnip](http://gnip.com) and [dojo4](http://dojo4.com) for
132
+ supporting and sponsoring work on middleman-s3_sync.
133
+
53
134
  ## Contributing
54
135
 
55
136
  1. Fork it
@@ -13,52 +13,53 @@ end
13
13
  module Middleman
14
14
  module S3Sync
15
15
  class << self
16
- LONG_TIME = 315360000 # 10 years
17
- DONT_CACHE = ['html'] # types of files to exclude from browser cache
18
-
19
16
  def sync
20
17
  puts "Gathering local files."
21
18
 
22
19
  local_files = (Dir[options.build_dir + "/**/*"] + Dir[options.build_dir + "/**/.*"])
23
20
  .reject { |f| File.directory?(f) }
24
21
  .map { |f| f.gsub(/^#{options.build_dir}\//, '') }
25
- puts "Gathering remote files."
22
+ puts "Gathering remote files from #{options.bucket}"
26
23
  remote_files = bucket.files.map { |f| f.key }
27
24
 
28
- # First pass on the set of files to work with.
29
- puts "Determine files to add to S3."
30
- files_to_push = local_files - remote_files
31
- if options.delete
32
- puts "Determine which files to delete from S3"
33
- files_to_delete = remote_files - local_files
34
- end
35
- files_to_evaluate = local_files - files_to_push
36
-
37
- # No need to evaluate the files that are newer on S3 than the local files.
38
- puts "Determine which local files are newer than their S3 counterparts"
39
- files_to_reject = []
40
- Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
41
- print '.'
42
- local_mtime = File.mtime("#{options.build_dir}/#{f}")
43
- remote_mtime = s3_files.get(f).last_modified
44
- files_to_reject << f if remote_mtime >= local_mtime
45
- end
46
-
47
- files_to_evaluate = files_to_evaluate - files_to_reject
25
+ if options.force
26
+ files_to_push = local_files
27
+ else
28
+ # First pass on the set of files to work with.
29
+ puts "Determine files to add to #{options.bucket}."
30
+ files_to_push = local_files - remote_files
31
+ if options.delete
32
+ puts "Determine which files to delete from #{options.bucket}"
33
+ files_to_delete = remote_files - local_files
34
+ end
35
+ files_to_evaluate = local_files - files_to_push
48
36
 
49
- # Are the files different? Use MD5 to see
50
- if (files_to_evaluate.size > 0)
51
- puts "\n\nDetermine which remaining files are actually different than their S3 counterpart."
37
+ # No need to evaluate the files that are newer on S3 than the local files.
38
+ puts "Determine which local files are newer than their S3 counterparts"
39
+ files_to_reject = []
52
40
  Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
53
41
  print '.'
54
- local_md5 = Digest::MD5.hexdigest(File.read("#{options.build_dir}/#{f}"))
55
- remote_md5 = s3_files.get(f).etag
56
- files_to_push << f if local_md5 != remote_md5
42
+ local_mtime = File.mtime("#{options.build_dir}/#{f}")
43
+ remote_mtime = s3_files.get(f).last_modified
44
+ files_to_reject << f if remote_mtime >= local_mtime
45
+ end
46
+
47
+ files_to_evaluate = files_to_evaluate - files_to_reject
48
+
49
+ # Are the files different? Use MD5 to see
50
+ if (files_to_evaluate.size > 0)
51
+ puts "\n\nDetermine which remaining files are actually different than their S3 counterpart."
52
+ Parallel.each(files_to_evaluate, :in_threads => 4) do |f|
53
+ print '.'
54
+ local_md5 = Digest::MD5.hexdigest(File.read("#{options.build_dir}/#{f}"))
55
+ remote_md5 = s3_files.get(f).etag
56
+ files_to_push << f if local_md5 != remote_md5
57
+ end
57
58
  end
58
59
  end
59
60
 
60
61
  if files_to_push.size > 0
61
- puts "\n\nReady to apply updates to S3."
62
+ puts "\n\nReady to apply updates to #{options.bucket}."
62
63
  files_to_push.each do |f|
63
64
  if remote_files.include?(f)
64
65
  puts "Updating #{f}"
@@ -66,11 +67,9 @@ module Middleman
66
67
  file.body = File.open("#{options.build_dir}/#{f}")
67
68
  file.public = true
68
69
  file.content_type = MIME::Types.of(f).first
69
- ext = File.extname(f)[1..-1]
70
-
71
- unless DONT_CACHE.include? ext
72
- file.cache_control = "public, max-age=#{LONG_TIME}"
73
- file.expires = CGI.rfc1123_date(Time.now + LONG_TIME)
70
+ if policy = options.caching_policy_for(file.content_type)
71
+ file.cache_control = policy.cache_control if policy.cache_control
72
+ file.expires = policy.expires if policy.expires
74
73
  end
75
74
 
76
75
  file.save
@@ -85,13 +84,11 @@ module Middleman
85
84
  }
86
85
 
87
86
  # Add cache-control headers
88
- ext = File.extname(f)[1..-1]
89
- unless DONT_CACHE.include? ext
90
- file_hash.merge!({
91
- :cache_control => "public, max-age=#{LONG_TIME}",
92
- :expires => CGI.rfc1123_date(Time.now + LONG_TIME)
93
- })
87
+ if policy = options.caching_policy_for(file_hash[:content_type])
88
+ file_hash[:cache_control] = policy.cache_control if policy.cache_control
89
+ file_hash[:expires] = policy.expires if policy.expires
94
90
  end
91
+
95
92
  file = bucket.files.create(file_hash)
96
93
  end
97
94
  end
@@ -5,9 +5,6 @@ module Middleman
5
5
  module Cli
6
6
  class S3Sync < Thor
7
7
  include Thor::Actions
8
-
9
- check_unknown_options!
10
-
11
8
  namespace :s3_sync
12
9
 
13
10
  def self.exit_on_failure?
@@ -15,17 +12,20 @@ module Middleman
15
12
  end
16
13
 
17
14
  desc "s3_sync", "Builds and push the minimum set of files needed to S3"
15
+ option :force, type: :boolean,
16
+ desc: "Push all local files to the server",
17
+ aliases: :f
18
18
  def s3_sync
19
19
  shared_inst = ::Middleman::Application.server.inst
20
20
  bucket = shared_inst.options.bucket
21
21
  if (!bucket)
22
22
  raise Thor::Error.new "You need to activate this extension."
23
23
  end
24
-
25
- ::Middleman::S3Sync.sync
26
24
 
27
- end
25
+ shared_inst.options.force = options[:force] if options[:force]
28
26
 
27
+ ::Middleman::S3Sync.sync
28
+ end
29
29
  end
30
30
  end
31
31
  end
@@ -11,8 +11,57 @@ module Middleman
11
11
  :after_build,
12
12
  :delete,
13
13
  :existing_remote_file,
14
- :build_dir
14
+ :build_dir,
15
+ :force
15
16
  )
17
+
18
+ def add_caching_policy(content_type, options)
19
+ caching_policies[content_type.to_s] = BrowserCachePolicy.new(options)
20
+ end
21
+
22
+ def caching_policy_for(content_type)
23
+ caching_policies.fetch(content_type.to_s, caching_policies[:default])
24
+ end
25
+
26
+ def default_caching_policy
27
+ caching_policies[:default]
28
+ end
29
+
30
+ def caching_policies
31
+ @caching_policies ||= Map.new
32
+ end
33
+
34
+ protected
35
+ class BrowserCachePolicy
36
+ attr_accessor :policies
37
+
38
+ def initialize(options)
39
+ @policies = Map.from_hash(options)
40
+ end
41
+
42
+ def cache_control
43
+ policy = []
44
+ policy << "max-age=#{policies.max_age}" if policies.has_key?(:max_age)
45
+ policy << "s-maxage=#{s_maxage}" if policies.has_key?(:s_maxage)
46
+ policy << "public" if policies.fetch(:public, false)
47
+ policy << "private" if policies.fetch(:private, false)
48
+ policy << "no-cache" if policies.fetch(:no_cache, false)
49
+ policy << "no-store" if policies.fetch(:no_store, false)
50
+ policy << "must-revalidate" if policies.fetch(:must_revalidate, false)
51
+ policy << "proxy-revalidate" if policies.fetch(:proxy_revalidate, false)
52
+ if policy.empty?
53
+ nil
54
+ else
55
+ policy.join(", ")
56
+ end
57
+ end
58
+
59
+ def expires
60
+ if expiration = policies.fetch(:expires, nil)
61
+ CGI.rfc1123_date(expiration)
62
+ end
63
+ end
64
+ end
16
65
  end
17
66
 
18
67
  class << self
@@ -29,7 +78,7 @@ module Middleman
29
78
  app.send :include, Helpers
30
79
 
31
80
  app.after_configuration do |config|
32
- options.build_dir = build_dir
81
+ options.build_dir ||= build_dir
33
82
  end
34
83
 
35
84
  app.after_build do |builder|
@@ -43,6 +92,14 @@ module Middleman
43
92
  def options
44
93
  ::Middleman::S3Sync.options
45
94
  end
95
+
96
+ def default_caching_policy(policy = {})
97
+ options.add_caching_policy(:default, policy)
98
+ end
99
+
100
+ def caching_policy(content_type, policy = {})
101
+ options.add_caching_policy(content_type, policy)
102
+ end
46
103
  end
47
104
  end
48
105
  end
@@ -1,5 +1,5 @@
1
1
  module Middleman
2
2
  module S3Sync
3
- VERSION = "3.0.7"
3
+ VERSION = "3.0.8"
4
4
  end
5
5
  end
@@ -20,6 +20,9 @@ Gem::Specification.new do |gem|
20
20
  gem.add_runtime_dependency 'middleman-core', '>= 3.0.0'
21
21
  gem.add_runtime_dependency 'fog'
22
22
  gem.add_runtime_dependency 'parallel'
23
+ gem.add_runtime_dependency 'map'
23
24
 
24
25
  gem.add_development_dependency 'rake'
26
+ gem.add_development_dependency 'pry'
27
+ gem.add_development_dependency 'pry-nav'
25
28
  end
metadata CHANGED
@@ -2,7 +2,7 @@
2
2
  name: middleman-s3_sync
3
3
  version: !ruby/object:Gem::Version
4
4
  prerelease:
5
- version: 3.0.7
5
+ version: 3.0.8
6
6
  platform: ruby
7
7
  authors:
8
8
  - Frederic Jean
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: bin
12
12
  cert_chain: []
13
- date: 2013-03-21 00:00:00.000000000 Z
13
+ date: 2013-03-29 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  version_requirements: !ruby/object:Gem::Requirement
@@ -60,6 +60,22 @@ dependencies:
60
60
  - !ruby/object:Gem::Version
61
61
  version: '0'
62
62
  type: :runtime
63
+ - !ruby/object:Gem::Dependency
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ none: false
66
+ requirements:
67
+ - - ! '>='
68
+ - !ruby/object:Gem::Version
69
+ version: '0'
70
+ name: map
71
+ prerelease: false
72
+ requirement: !ruby/object:Gem::Requirement
73
+ none: false
74
+ requirements:
75
+ - - ! '>='
76
+ - !ruby/object:Gem::Version
77
+ version: '0'
78
+ type: :runtime
63
79
  - !ruby/object:Gem::Dependency
64
80
  version_requirements: !ruby/object:Gem::Requirement
65
81
  none: false
@@ -76,6 +92,38 @@ dependencies:
76
92
  - !ruby/object:Gem::Version
77
93
  version: '0'
78
94
  type: :development
95
+ - !ruby/object:Gem::Dependency
96
+ version_requirements: !ruby/object:Gem::Requirement
97
+ none: false
98
+ requirements:
99
+ - - ! '>='
100
+ - !ruby/object:Gem::Version
101
+ version: '0'
102
+ name: pry
103
+ prerelease: false
104
+ requirement: !ruby/object:Gem::Requirement
105
+ none: false
106
+ requirements:
107
+ - - ! '>='
108
+ - !ruby/object:Gem::Version
109
+ version: '0'
110
+ type: :development
111
+ - !ruby/object:Gem::Dependency
112
+ version_requirements: !ruby/object:Gem::Requirement
113
+ none: false
114
+ requirements:
115
+ - - ! '>='
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ name: pry-nav
119
+ prerelease: false
120
+ requirement: !ruby/object:Gem::Requirement
121
+ none: false
122
+ requirements:
123
+ - - ! '>='
124
+ - !ruby/object:Gem::Version
125
+ version: '0'
126
+ type: :development
79
127
  description: Only syncs files that have been updated to S3.
80
128
  email:
81
129
  - fred@fredjean.net
@@ -107,7 +155,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
107
155
  - !ruby/object:Gem::Version
108
156
  segments:
109
157
  - 0
110
- hash: -892703647475452173
158
+ hash: -1761006957673169609
111
159
  version: '0'
112
160
  required_rubygems_version: !ruby/object:Gem::Requirement
113
161
  none: false
@@ -116,7 +164,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
116
164
  - !ruby/object:Gem::Version
117
165
  segments:
118
166
  - 0
119
- hash: -892703647475452173
167
+ hash: -1761006957673169609
120
168
  version: '0'
121
169
  requirements: []
122
170
  rubyforge_project: