middleman-s3_sync 4.6.1 → 4.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.envrc +5 -0
- data/.gitignore +1 -0
- data/Changelog.md +9 -0
- data/WARP.md +112 -0
- data/lib/middleman/s3_sync/resource.rb +4 -6
- data/lib/middleman/s3_sync/version.rb +1 -1
- data/lib/middleman/s3_sync.rb +3 -3
- data/spec/aws_sdk_parameters_spec.rb +415 -0
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 61870e8a5164a26db62b7463da7be2c74272800ab76cd6eab3fe83ec68e58081
|
4
|
+
data.tar.gz: d901b5c020cc4fe8ddc695ceeb2fde34a3555f4645aca7ec04cea7d513a0e37c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 40ff859c4ae0b722c2132b3192bd14ab454988b612afb18f442645d7fbfb894b48e7d5ba502a1aae36439ec751c8f2f03e8496df255c736c5c2718770f69a7be
|
7
|
+
data.tar.gz: 0a1fe315c71b79ae80041efaf2beb206cdc4caa1db49a7635ede51c8e53c8bbb1cd95a2b0a28cf2e9eb2dcc4c98101e5a6847c3792fc567d6f2b2d26c4bd92f4
|
data/.envrc
ADDED
data/.gitignore
CHANGED
data/Changelog.md
CHANGED
@@ -2,6 +2,15 @@
|
|
2
2
|
|
3
3
|
The gem that tries really hard not to push files to S3.
|
4
4
|
|
5
|
+
## v4.6.2
|
6
|
+
|
7
|
+
* Fix AWS SDK parameter format issues from Fog migration
|
8
|
+
* Fix website configuration parameters to use symbol keys instead of strings
|
9
|
+
* Fix S3 object metadata parameter format to use correct key suffixes
|
10
|
+
* Remove obsolete Fog-style constants (CONTENT_MD5_KEY, REDIRECT_KEY)
|
11
|
+
* Add comprehensive test suite for AWS SDK parameter validation (18 new tests)
|
12
|
+
* Improve compatibility with AWS SDK v3 to prevent API errors
|
13
|
+
|
5
14
|
## v4.6.1
|
6
15
|
|
7
16
|
* Add CloudFront rate limit handling with exponential backoff retry logic
|
data/WARP.md
ADDED
@@ -0,0 +1,112 @@
|
|
1
|
+
# WARP.md
|
2
|
+
|
3
|
+
This file provides guidance to WARP (warp.dev) when working with code in this repository.
|
4
|
+
|
5
|
+
## Project Overview
|
6
|
+
|
7
|
+
middleman-s3_sync is a Ruby gem that provides intelligent S3 synchronization for Middleman static sites. Unlike other sync tools, it only transfers files that have been added, updated, or deleted, making deployments more efficient. The gem also supports CloudFront cache invalidation and advanced caching policies.
|
8
|
+
|
9
|
+
## Key Commands
|
10
|
+
|
11
|
+
### Development
|
12
|
+
```bash
|
13
|
+
# Install dependencies
|
14
|
+
bundle install
|
15
|
+
|
16
|
+
# Run tests
|
17
|
+
bundle exec rspec
|
18
|
+
|
19
|
+
# Run specific test file
|
20
|
+
bundle exec rspec spec/resource_spec.rb
|
21
|
+
|
22
|
+
# Run tests with verbose output
|
23
|
+
bundle exec rspec -fd
|
24
|
+
|
25
|
+
# Build the gem
|
26
|
+
bundle exec rake build
|
27
|
+
|
28
|
+
# Install the gem locally for testing
|
29
|
+
bundle exec rake install
|
30
|
+
```
|
31
|
+
|
32
|
+
### Testing & Quality
|
33
|
+
```bash
|
34
|
+
# Run all specs
|
35
|
+
bundle exec rake spec # or just `rake` (default task)
|
36
|
+
|
37
|
+
# Run specific test patterns
|
38
|
+
bundle exec rspec spec/*_spec.rb --pattern "*policy*"
|
39
|
+
|
40
|
+
# Check gem build without installing
|
41
|
+
gem build middleman-s3_sync.gemspec
|
42
|
+
```
|
43
|
+
|
44
|
+
## Architecture
|
45
|
+
|
46
|
+
### Core Components
|
47
|
+
|
48
|
+
**Main Sync Engine** (`lib/middleman/s3_sync.rb`)
|
49
|
+
- Central orchestration of sync operations
|
50
|
+
- Thread-safe S3 operations using mutexes
|
51
|
+
- Parallel processing (8 threads by default) for file operations
|
52
|
+
- Tracks CloudFront invalidation paths during sync
|
53
|
+
|
54
|
+
**Extension Integration** (`lib/middleman-s3_sync/extension.rb`)
|
55
|
+
- Middleman extension that hooks into the build process
|
56
|
+
- Configurable options with environment variable fallbacks
|
57
|
+
- Resource list manipulation to prepare files for sync
|
58
|
+
|
59
|
+
**CLI Commands** (`lib/middleman-s3_sync/commands.rb`)
|
60
|
+
- Thor-based command-line interface
|
61
|
+
- Extensive option parsing for CloudFront, AWS credentials, and sync behavior
|
62
|
+
- Support for dry-run mode and build-then-sync workflows
|
63
|
+
|
64
|
+
**Resource Management** (`lib/middleman/s3_sync/resource.rb`)
|
65
|
+
- Individual file resource handling and status determination
|
66
|
+
- MD5-based change detection and caching policy application
|
67
|
+
|
68
|
+
**CloudFront Integration** (`lib/middleman/s3_sync/cloudfront.rb`)
|
69
|
+
- Intelligent cache invalidation with batch processing
|
70
|
+
- Rate limit handling and retry logic
|
71
|
+
- Path optimization and wildcard support
|
72
|
+
|
73
|
+
### Key Design Patterns
|
74
|
+
|
75
|
+
- **Thread Safety**: Uses mutexes for bucket and bucket_files operations
|
76
|
+
- **Parallel Processing**: Leverages `parallel` gem for concurrent S3 operations
|
77
|
+
- **Status-Based Operations**: Resources maintain state (create, update, delete, ignore)
|
78
|
+
- **Configuration Cascade**: CLI options override config.rb options override .s3_sync file options override environment variables
|
79
|
+
|
80
|
+
### File Status Logic
|
81
|
+
|
82
|
+
The gem determines what to do with each file by comparing:
|
83
|
+
- Local file MD5 hashes vs S3 ETags
|
84
|
+
- Presence in local build vs S3 bucket
|
85
|
+
- Caching policies and content encoding preferences
|
86
|
+
|
87
|
+
## Testing Strategy
|
88
|
+
|
89
|
+
**RSpec Structure**:
|
90
|
+
- `caching_policy_spec.rb`: HTTP caching header generation
|
91
|
+
- `cloudfront_spec.rb`: CloudFront invalidation logic (comprehensive, 18k lines)
|
92
|
+
- `resource_spec.rb`: Individual file resource operations
|
93
|
+
- `s3_sync_integration_spec.rb`: End-to-end sync workflows
|
94
|
+
|
95
|
+
**Mock Strategy**: Uses AWS SDK stub responses and custom S3 object mocks for isolated testing without real AWS calls.
|
96
|
+
|
97
|
+
## Configuration Files
|
98
|
+
|
99
|
+
- **`.s3_sync`**: YAML configuration file for credentials and options (should be gitignored)
|
100
|
+
- **`config.rb`**: Middleman configuration with `activate :s3_sync` block
|
101
|
+
- **Environment Variables**: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET, etc.
|
102
|
+
|
103
|
+
## Common Development Patterns
|
104
|
+
|
105
|
+
When adding new functionality:
|
106
|
+
1. Add option to `extension.rb` with appropriate defaults
|
107
|
+
2. Add CLI flag to `commands.rb` if user-facing
|
108
|
+
3. Implement core logic in main `s3_sync.rb` module
|
109
|
+
4. Add comprehensive specs following existing patterns
|
110
|
+
5. Update README.md with new configuration options
|
111
|
+
|
112
|
+
The codebase emphasizes security (credential handling), efficiency (parallel operations), and reliability (comprehensive error handling and dry-run support).
|
@@ -3,8 +3,6 @@ module Middleman
|
|
3
3
|
class Resource
|
4
4
|
attr_accessor :path, :resource, :partial_s3_resource, :full_s3_resource, :content_type, :gzipped, :options
|
5
5
|
|
6
|
-
CONTENT_MD5_KEY = 'x-amz-meta-content-md5'
|
7
|
-
REDIRECT_KEY = 'x-amz-website-redirect-location'
|
8
6
|
|
9
7
|
include Status
|
10
8
|
|
@@ -59,7 +57,7 @@ module Middleman
|
|
59
57
|
:key => key,
|
60
58
|
:acl => options.acl,
|
61
59
|
:content_type => content_type,
|
62
|
-
|
60
|
+
'content-md5' => local_content_md5
|
63
61
|
}
|
64
62
|
|
65
63
|
if caching_policy
|
@@ -80,7 +78,7 @@ module Middleman
|
|
80
78
|
end
|
81
79
|
|
82
80
|
if redirect?
|
83
|
-
attributes[
|
81
|
+
attributes['website-redirect-location'] = redirect_url
|
84
82
|
end
|
85
83
|
|
86
84
|
attributes
|
@@ -125,7 +123,7 @@ module Middleman
|
|
125
123
|
|
126
124
|
# Add metadata if present
|
127
125
|
if local_content_md5
|
128
|
-
upload_options[:metadata] = {
|
126
|
+
upload_options[:metadata] = { 'content-md5' => local_content_md5 }
|
129
127
|
end
|
130
128
|
|
131
129
|
# Add redirect if present
|
@@ -292,7 +290,7 @@ module Middleman
|
|
292
290
|
|
293
291
|
def remote_content_md5
|
294
292
|
if full_s3_resource && full_s3_resource.metadata
|
295
|
-
full_s3_resource.metadata[
|
293
|
+
full_s3_resource.metadata['content-md5']
|
296
294
|
end
|
297
295
|
end
|
298
296
|
|
data/lib/middleman/s3_sync.rb
CHANGED
@@ -107,10 +107,10 @@ module Middleman
|
|
107
107
|
|
108
108
|
def update_bucket_website
|
109
109
|
opts = {}
|
110
|
-
opts[:
|
111
|
-
opts[:
|
110
|
+
opts[:index_document] = { suffix: s3_sync_options.index_document } if s3_sync_options.index_document
|
111
|
+
opts[:error_document] = { key: s3_sync_options.error_document } if s3_sync_options.error_document
|
112
112
|
|
113
|
-
if opts[:
|
113
|
+
if opts[:error_document] && !opts[:index_document]
|
114
114
|
raise 'S3 requires `index_document` if `error_document` is specified'
|
115
115
|
end
|
116
116
|
|
@@ -0,0 +1,415 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe 'AWS SDK Parameter Validation' do
|
4
|
+
let(:options) { Middleman::S3Sync::Options.new }
|
5
|
+
let(:s3_client) { instance_double(Aws::S3::Client) }
|
6
|
+
let(:s3_resource) { instance_double(Aws::S3::Resource) }
|
7
|
+
let(:bucket) { instance_double(Aws::S3::Bucket) }
|
8
|
+
let(:s3_object) { instance_double(Aws::S3::Object) }
|
9
|
+
|
10
|
+
before do
|
11
|
+
Middleman::S3Sync.s3_sync_options = options
|
12
|
+
options.build_dir = "build"
|
13
|
+
options.bucket = "test-bucket"
|
14
|
+
options.acl = "public-read"
|
15
|
+
options.index_document = "index.html"
|
16
|
+
options.error_document = "404.html"
|
17
|
+
options.version_bucket = true
|
18
|
+
|
19
|
+
allow(Aws::S3::Client).to receive(:new).and_return(s3_client)
|
20
|
+
allow(Aws::S3::Resource).to receive(:new).and_return(s3_resource)
|
21
|
+
allow(s3_resource).to receive(:bucket).and_return(bucket)
|
22
|
+
allow(bucket).to receive(:exists?).and_return(true)
|
23
|
+
allow(bucket).to receive(:object).and_return(s3_object)
|
24
|
+
allow(s3_object).to receive(:put).and_return(true)
|
25
|
+
|
26
|
+
# Allow Middleman::S3Sync to use our mocked client/bucket
|
27
|
+
allow(Middleman::S3Sync).to receive(:s3_client).and_return(s3_client)
|
28
|
+
allow(Middleman::S3Sync).to receive(:bucket).and_return(bucket)
|
29
|
+
allow(Middleman::S3Sync).to receive(:say_status)
|
30
|
+
end
|
31
|
+
|
32
|
+
describe 'put_bucket_website parameters' do
|
33
|
+
it 'uses symbol keys for index_document and error_document' do
|
34
|
+
expect(s3_client).to receive(:put_bucket_website) do |params|
|
35
|
+
expect(params[:bucket]).to eq("test-bucket")
|
36
|
+
expect(params[:website_configuration]).to have_key(:index_document)
|
37
|
+
expect(params[:website_configuration]).to have_key(:error_document)
|
38
|
+
|
39
|
+
# Verify the nested structure uses symbols, not strings
|
40
|
+
expect(params[:website_configuration][:index_document]).to have_key(:suffix)
|
41
|
+
expect(params[:website_configuration][:error_document]).to have_key(:key)
|
42
|
+
|
43
|
+
# Verify the values are correct
|
44
|
+
expect(params[:website_configuration][:index_document][:suffix]).to eq("index.html")
|
45
|
+
expect(params[:website_configuration][:error_document][:key]).to eq("404.html")
|
46
|
+
end
|
47
|
+
|
48
|
+
Middleman::S3Sync.send(:update_bucket_website)
|
49
|
+
end
|
50
|
+
|
51
|
+
context 'when only index_document is set' do
|
52
|
+
before do
|
53
|
+
options.error_document = nil
|
54
|
+
end
|
55
|
+
|
56
|
+
it 'only includes index_document in website configuration' do
|
57
|
+
expect(s3_client).to receive(:put_bucket_website) do |params|
|
58
|
+
expect(params[:website_configuration]).to have_key(:index_document)
|
59
|
+
expect(params[:website_configuration]).not_to have_key(:error_document)
|
60
|
+
end
|
61
|
+
|
62
|
+
Middleman::S3Sync.send(:update_bucket_website)
|
63
|
+
end
|
64
|
+
end
|
65
|
+
|
66
|
+
context 'when neither document is set' do
|
67
|
+
before do
|
68
|
+
options.index_document = nil
|
69
|
+
options.error_document = nil
|
70
|
+
end
|
71
|
+
|
72
|
+
it 'does not call put_bucket_website' do
|
73
|
+
expect(s3_client).not_to receive(:put_bucket_website)
|
74
|
+
|
75
|
+
Middleman::S3Sync.send(:update_bucket_website)
|
76
|
+
end
|
77
|
+
end
|
78
|
+
|
79
|
+
context 'when only error_document is set' do
|
80
|
+
before do
|
81
|
+
options.index_document = nil
|
82
|
+
end
|
83
|
+
|
84
|
+
it 'raises an error because S3 requires index_document if error_document is specified' do
|
85
|
+
expect {
|
86
|
+
Middleman::S3Sync.send(:update_bucket_website)
|
87
|
+
}.to raise_error('S3 requires `index_document` if `error_document` is specified')
|
88
|
+
end
|
89
|
+
end
|
90
|
+
end
|
91
|
+
|
92
|
+
describe 'put_bucket_versioning parameters' do
|
93
|
+
it 'uses correct parameter structure' do
|
94
|
+
expect(s3_client).to receive(:put_bucket_versioning) do |params|
|
95
|
+
expect(params[:bucket]).to eq("test-bucket")
|
96
|
+
expect(params[:versioning_configuration]).to be_a(Hash)
|
97
|
+
expect(params[:versioning_configuration][:status]).to eq("Enabled")
|
98
|
+
end
|
99
|
+
|
100
|
+
Middleman::S3Sync.send(:update_bucket_versioning)
|
101
|
+
end
|
102
|
+
|
103
|
+
context 'when version_bucket is false' do
|
104
|
+
before do
|
105
|
+
options.version_bucket = false
|
106
|
+
end
|
107
|
+
|
108
|
+
it 'does not call put_bucket_versioning' do
|
109
|
+
expect(s3_client).not_to receive(:put_bucket_versioning)
|
110
|
+
|
111
|
+
Middleman::S3Sync.send(:update_bucket_versioning)
|
112
|
+
end
|
113
|
+
end
|
114
|
+
end
|
115
|
+
|
116
|
+
describe 'S3 object upload parameters' do
|
117
|
+
let(:mm_resource) do
|
118
|
+
double(
|
119
|
+
destination_path: 'test/file.html',
|
120
|
+
content_type: 'text/html'
|
121
|
+
)
|
122
|
+
end
|
123
|
+
|
124
|
+
let(:resource) { Middleman::S3Sync::Resource.new(mm_resource, nil) }
|
125
|
+
|
126
|
+
before do
|
127
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
128
|
+
allow(File).to receive(:exist?).with('build/test/file.html.gz').and_return(false)
|
129
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('test content')
|
130
|
+
allow(File).to receive(:directory?).with('build/test/file.html').and_return(false)
|
131
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
132
|
+
options.dry_run = false
|
133
|
+
end
|
134
|
+
|
135
|
+
it 'uses correct metadata key format' do
|
136
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
137
|
+
# Verify basic parameters
|
138
|
+
expect(upload_options[:body]).to eq('test content')
|
139
|
+
expect(upload_options[:content_type]).to eq('text/html')
|
140
|
+
expect(upload_options[:acl]).to eq('public-read')
|
141
|
+
|
142
|
+
# Verify metadata uses correct key format (suffix only, not full header name)
|
143
|
+
expect(upload_options[:metadata]).to be_a(Hash)
|
144
|
+
expect(upload_options[:metadata]).to have_key('content-md5')
|
145
|
+
expect(upload_options[:metadata]).not_to have_key('x-amz-meta-content-md5')
|
146
|
+
|
147
|
+
# Verify metadata value is the MD5 hash
|
148
|
+
expected_md5 = Digest::MD5.hexdigest('test content')
|
149
|
+
expect(upload_options[:metadata]['content-md5']).to eq(expected_md5)
|
150
|
+
end
|
151
|
+
|
152
|
+
resource.upload!
|
153
|
+
end
|
154
|
+
|
155
|
+
context 'when gzip is enabled' do
|
156
|
+
before do
|
157
|
+
options.prefer_gzip = true
|
158
|
+
allow(File).to receive(:exist?).with('build/test/file.html.gz').and_return(true)
|
159
|
+
allow(File).to receive(:read).with('build/test/file.html.gz').and_return('gzipped content')
|
160
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
161
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('original content')
|
162
|
+
|
163
|
+
# Mock the HEAD response to avoid calling it during redirect?
|
164
|
+
head_response = double(
|
165
|
+
metadata: {},
|
166
|
+
etag: '"abc123"',
|
167
|
+
content_encoding: nil,
|
168
|
+
cache_control: nil,
|
169
|
+
website_redirect_location: nil
|
170
|
+
)
|
171
|
+
allow(s3_object).to receive(:head).and_return(head_response)
|
172
|
+
resource.instance_variable_set(:@full_s3_resource, head_response)
|
173
|
+
end
|
174
|
+
|
175
|
+
it 'includes content_encoding parameter' do
|
176
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
177
|
+
expect(upload_options[:content_encoding]).to eq('gzip')
|
178
|
+
end
|
179
|
+
|
180
|
+
resource.upload!
|
181
|
+
end
|
182
|
+
end
|
183
|
+
|
184
|
+
context 'when reduced redundancy storage is enabled' do
|
185
|
+
before do
|
186
|
+
options.reduced_redundancy_storage = true
|
187
|
+
end
|
188
|
+
|
189
|
+
it 'includes storage_class parameter' do
|
190
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
191
|
+
expect(upload_options[:storage_class]).to eq('REDUCED_REDUNDANCY')
|
192
|
+
end
|
193
|
+
|
194
|
+
resource.upload!
|
195
|
+
end
|
196
|
+
end
|
197
|
+
|
198
|
+
context 'when encryption is enabled' do
|
199
|
+
before do
|
200
|
+
options.encryption = true
|
201
|
+
end
|
202
|
+
|
203
|
+
it 'includes server_side_encryption parameter' do
|
204
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
205
|
+
expect(upload_options[:server_side_encryption]).to eq('AES256')
|
206
|
+
end
|
207
|
+
|
208
|
+
resource.upload!
|
209
|
+
end
|
210
|
+
end
|
211
|
+
|
212
|
+
context 'when resource has a redirect' do
|
213
|
+
let(:mm_resource) do
|
214
|
+
double(
|
215
|
+
destination_path: 'redirect/file.html',
|
216
|
+
content_type: 'text/html',
|
217
|
+
redirect?: true,
|
218
|
+
target_url: 'https://example.com/new-location'
|
219
|
+
)
|
220
|
+
end
|
221
|
+
|
222
|
+
before do
|
223
|
+
allow(File).to receive(:exist?).with('build/redirect/file.html').and_return(true)
|
224
|
+
allow(File).to receive(:exist?).with('build/redirect/file.html.gz').and_return(false)
|
225
|
+
allow(File).to receive(:read).with('build/redirect/file.html').and_return('redirect content')
|
226
|
+
allow(File).to receive(:directory?).with('build/redirect/file.html').and_return(false)
|
227
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
228
|
+
allow(resource).to receive(:redirect?).and_return(true)
|
229
|
+
allow(resource).to receive(:redirect_url).and_return('https://example.com/new-location')
|
230
|
+
end
|
231
|
+
|
232
|
+
it 'includes website_redirect_location parameter' do
|
233
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
234
|
+
expect(upload_options[:website_redirect_location]).to eq('https://example.com/new-location')
|
235
|
+
end
|
236
|
+
|
237
|
+
resource.upload!
|
238
|
+
end
|
239
|
+
end
|
240
|
+
end
|
241
|
+
|
242
|
+
describe 'S3 object metadata retrieval' do
|
243
|
+
let(:mm_resource) do
|
244
|
+
double(
|
245
|
+
destination_path: 'test/file.html',
|
246
|
+
content_type: 'text/html'
|
247
|
+
)
|
248
|
+
end
|
249
|
+
|
250
|
+
let(:resource) { Middleman::S3Sync::Resource.new(mm_resource, nil) }
|
251
|
+
let(:head_response) do
|
252
|
+
double(
|
253
|
+
metadata: { 'content-md5' => 'abc123def456' },
|
254
|
+
etag: '"def456abc123"',
|
255
|
+
content_encoding: nil,
|
256
|
+
cache_control: nil,
|
257
|
+
website_redirect_location: nil
|
258
|
+
)
|
259
|
+
end
|
260
|
+
|
261
|
+
before do
|
262
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
263
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('test content')
|
264
|
+
allow(File).to receive(:directory?).with('build/test/file.html').and_return(false)
|
265
|
+
allow(s3_object).to receive(:head).and_return(head_response)
|
266
|
+
resource.instance_variable_set(:@full_s3_resource, head_response)
|
267
|
+
end
|
268
|
+
|
269
|
+
it 'reads metadata using correct key format' do
|
270
|
+
expect(resource.remote_content_md5).to eq('abc123def456')
|
271
|
+
end
|
272
|
+
|
273
|
+
it 'does not try to read metadata with old header format' do
|
274
|
+
# Ensure it's not looking for the full header name
|
275
|
+
expect(head_response.metadata).not_to receive(:[]).with('x-amz-meta-content-md5')
|
276
|
+
|
277
|
+
resource.remote_content_md5
|
278
|
+
end
|
279
|
+
end
|
280
|
+
|
281
|
+
describe 'to_h method for legacy compatibility' do
|
282
|
+
let(:mm_resource) do
|
283
|
+
double(
|
284
|
+
destination_path: 'test/file.html',
|
285
|
+
content_type: 'text/html'
|
286
|
+
)
|
287
|
+
end
|
288
|
+
|
289
|
+
let(:resource) { Middleman::S3Sync::Resource.new(mm_resource, nil) }
|
290
|
+
|
291
|
+
before do
|
292
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
293
|
+
allow(File).to receive(:exist?).with('build/test/file.html.gz').and_return(false)
|
294
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('test content')
|
295
|
+
allow(File).to receive(:directory?).with('build/test/file.html').and_return(false)
|
296
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
297
|
+
end
|
298
|
+
|
299
|
+
it 'returns attributes with correct key formats' do
|
300
|
+
attributes = resource.to_h
|
301
|
+
|
302
|
+
expect(attributes[:key]).to eq('test/file.html')
|
303
|
+
expect(attributes[:acl]).to eq('public-read')
|
304
|
+
expect(attributes[:content_type]).to eq('text/html')
|
305
|
+
expect(attributes['content-md5']).to be_a(String)
|
306
|
+
expect(attributes).not_to have_key('x-amz-meta-content-md5')
|
307
|
+
end
|
308
|
+
|
309
|
+
context 'when resource has a redirect' do
|
310
|
+
let(:mm_resource) do
|
311
|
+
double(
|
312
|
+
destination_path: 'redirect/file.html',
|
313
|
+
content_type: 'text/html',
|
314
|
+
redirect?: true,
|
315
|
+
target_url: 'https://example.com/new-location'
|
316
|
+
)
|
317
|
+
end
|
318
|
+
|
319
|
+
before do
|
320
|
+
allow(File).to receive(:exist?).with('build/redirect/file.html').and_return(true)
|
321
|
+
allow(File).to receive(:exist?).with('build/redirect/file.html.gz').and_return(false)
|
322
|
+
allow(File).to receive(:read).with('build/redirect/file.html').and_return('redirect content')
|
323
|
+
allow(File).to receive(:directory?).with('build/redirect/file.html').and_return(false)
|
324
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
325
|
+
allow(resource).to receive(:redirect?).and_return(true)
|
326
|
+
allow(resource).to receive(:redirect_url).and_return('https://example.com/new-location')
|
327
|
+
end
|
328
|
+
|
329
|
+
it 'includes redirect with correct key format' do
|
330
|
+
attributes = resource.to_h
|
331
|
+
|
332
|
+
expect(attributes['website-redirect-location']).to eq('https://example.com/new-location')
|
333
|
+
expect(attributes).not_to have_key('x-amz-website-redirect-location')
|
334
|
+
end
|
335
|
+
end
|
336
|
+
end
|
337
|
+
|
338
|
+
describe 'Regression tests for Fog-style parameter issues' do
|
339
|
+
# These tests validate that we've fixed the old Fog-style parameter formatting
|
340
|
+
# and demonstrate what would fail if we reverted to the old style
|
341
|
+
|
342
|
+
it 'does not use string keys for website configuration (old Fog style)' do
|
343
|
+
# This would fail if we reverted to the old format:
|
344
|
+
# opts[:index_document] = { "suffix" => s3_sync_options.index_document }
|
345
|
+
|
346
|
+
expect(s3_client).to receive(:put_bucket_website) do |params|
|
347
|
+
config = params[:website_configuration]
|
348
|
+
|
349
|
+
# Ensure we're not using string keys (old Fog style)
|
350
|
+
expect(config[:index_document]).not_to have_key("suffix")
|
351
|
+
expect(config[:error_document]).not_to have_key("key")
|
352
|
+
|
353
|
+
# Ensure we ARE using symbol keys (correct AWS SDK style)
|
354
|
+
expect(config[:index_document]).to have_key(:suffix)
|
355
|
+
expect(config[:error_document]).to have_key(:key)
|
356
|
+
end
|
357
|
+
|
358
|
+
Middleman::S3Sync.send(:update_bucket_website)
|
359
|
+
end
|
360
|
+
|
361
|
+
it 'does not use full header names in metadata (old style)' do
|
362
|
+
mm_resource = double(
|
363
|
+
destination_path: 'test/file.html',
|
364
|
+
content_type: 'text/html'
|
365
|
+
)
|
366
|
+
resource = Middleman::S3Sync::Resource.new(mm_resource, nil)
|
367
|
+
|
368
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
369
|
+
allow(File).to receive(:exist?).with('build/test/file.html.gz').and_return(false)
|
370
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('test content')
|
371
|
+
allow(File).to receive(:directory?).with('build/test/file.html').and_return(false)
|
372
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
373
|
+
options.dry_run = false
|
374
|
+
|
375
|
+
expect(s3_object).to receive(:put) do |upload_options|
|
376
|
+
# Ensure we're not using the old full header format
|
377
|
+
expect(upload_options[:metadata]).not_to have_key('x-amz-meta-content-md5')
|
378
|
+
|
379
|
+
# Ensure we ARE using the correct suffix-only format
|
380
|
+
expect(upload_options[:metadata]).to have_key('content-md5')
|
381
|
+
end
|
382
|
+
|
383
|
+
resource.upload!
|
384
|
+
end
|
385
|
+
|
386
|
+
it 'validates that old constants are no longer used' do
|
387
|
+
# This test ensures the old constants were removed/changed
|
388
|
+
# If they still existed, this would be a sign we didn't clean up properly
|
389
|
+
|
390
|
+
mm_resource = double(
|
391
|
+
destination_path: 'test/file.html',
|
392
|
+
content_type: 'text/html'
|
393
|
+
)
|
394
|
+
resource = Middleman::S3Sync::Resource.new(mm_resource, nil)
|
395
|
+
|
396
|
+
allow(File).to receive(:exist?).with('build/test/file.html').and_return(true)
|
397
|
+
allow(File).to receive(:exist?).with('build/test/file.html.gz').and_return(false)
|
398
|
+
allow(File).to receive(:read).with('build/test/file.html').and_return('test content')
|
399
|
+
allow(File).to receive(:directory?).with('build/test/file.html').and_return(false)
|
400
|
+
allow(s3_object).to receive(:head).and_return(nil)
|
401
|
+
allow(resource).to receive(:redirect?).and_return(true)
|
402
|
+
allow(resource).to receive(:redirect_url).and_return('https://example.com/redirect')
|
403
|
+
|
404
|
+
attributes = resource.to_h
|
405
|
+
|
406
|
+
# Validate that the old constant values are not used
|
407
|
+
expect(attributes).not_to have_key('x-amz-meta-content-md5')
|
408
|
+
expect(attributes).not_to have_key('x-amz-website-redirect-location')
|
409
|
+
|
410
|
+
# Validate that the correct formats are used
|
411
|
+
expect(attributes).to have_key('content-md5')
|
412
|
+
expect(attributes).to have_key('website-redirect-location')
|
413
|
+
end
|
414
|
+
end
|
415
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: middleman-s3_sync
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 4.6.
|
4
|
+
version: 4.6.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Frederic Jean
|
8
8
|
- Will Koehler
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2025-
|
11
|
+
date: 2025-09-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: middleman-core
|
@@ -311,6 +311,7 @@ executables: []
|
|
311
311
|
extensions: []
|
312
312
|
extra_rdoc_files: []
|
313
313
|
files:
|
314
|
+
- ".envrc"
|
314
315
|
- ".gitignore"
|
315
316
|
- ".rspec"
|
316
317
|
- ".s3_sync.sample"
|
@@ -320,6 +321,7 @@ files:
|
|
320
321
|
- LICENSE.txt
|
321
322
|
- README.md
|
322
323
|
- Rakefile
|
324
|
+
- WARP.md
|
323
325
|
- lib/middleman-s3_sync.rb
|
324
326
|
- lib/middleman-s3_sync/commands.rb
|
325
327
|
- lib/middleman-s3_sync/extension.rb
|
@@ -333,6 +335,7 @@ files:
|
|
333
335
|
- lib/middleman/s3_sync/version.rb
|
334
336
|
- lib/middleman_extension.rb
|
335
337
|
- middleman-s3_sync.gemspec
|
338
|
+
- spec/aws_sdk_parameters_spec.rb
|
336
339
|
- spec/caching_policy_spec.rb
|
337
340
|
- spec/cloudfront_spec.rb
|
338
341
|
- spec/resource_spec.rb
|
@@ -360,6 +363,7 @@ rubygems_version: 3.6.2
|
|
360
363
|
specification_version: 4
|
361
364
|
summary: Tries really, really hard not to push files to S3.
|
362
365
|
test_files:
|
366
|
+
- spec/aws_sdk_parameters_spec.rb
|
363
367
|
- spec/caching_policy_spec.rb
|
364
368
|
- spec/cloudfront_spec.rb
|
365
369
|
- spec/resource_spec.rb
|