middleman-s3_sync 4.6.1 → 4.6.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ad34e26fb6ea816a0426ef33e543c4eabc3c5323151f1c40512a95183ae6ed7e
-  data.tar.gz: a5b875195ed49d7b311eb87a6672a67aba69773e9d85f50fc1022c185238cf41
+  metadata.gz: 575e3838865eccc6e585b35829b3891f2a86a77502df0da9e411634ad40312a1
+  data.tar.gz: bf897e1d7725bf536759469ef486dc81e51410c67f20b801322a1ad2403a8445
 SHA512:
-  metadata.gz: c8f1e1678fd3e45350cdc4efefdd3b2520ad90abd075478dbfcfabb85d8c6294aef1c78acffc7451a9f19cb69b890fd90f6049f35a3000960c87906c7188059f
-  data.tar.gz: 8b062b097077d16019a1aef109338ab3603941dc4a57d06ea0da23374fc5a947ac0f93b0bbedb08eac1f87e3fd1df38fd7db5f3c116d6a04f4c8bf7f79ee1c39
+  metadata.gz: 1ef461a263b9ec005dcca856b34d3b02509550d2ba6d94904920bdf448268aacc547cbdf412406616f4ff409ce70e425ad013cd3e5e0de8a75315c1134d518e9
+  data.tar.gz: a00ae49ff314f27e8a3d6878ae00709a67c5759563675b64d72fdc5a61bdc057a19191ab04dea2795f80cfd75c86c00d0e99d707be382227c6c6eeabefeff2c6

data/.envrc.backup

@@ -0,0 +1,5 @@
+# Adds bin to the path.
+if ! test -d bin || test Gemfile.lock -nt bin; then
+  bundle binstubs --all
+fi
+PATH_add bin

data/.gitignore

@@ -16,3 +16,4 @@ test/tmp
 test/version_tmp
 tmp
 .aider*
+/bin

data/.mise.toml

@@ -0,0 +1,17 @@
+[tools]
+ruby = "3.4"
+
+[env]
+_.file = ".env"
+_.path = ["./bin"]
+
+[tasks.binstubs]
+description = "Generate binstubs for all gems"
+run = "bundle binstubs --all"
+
+[tasks.setup]
+description = "Setup project dependencies and binstubs"
+run = [
+  "bundle install",
+  "mise run binstubs"
+]

data/Changelog.md

@@ -2,6 +2,18 @@
 
 The gem that tries really hard not to push files to S3.
 
+## v4.6.3
+* Restrict incompatible map 8.x installation
+
+## v4.6.2
+
+* Fix AWS SDK parameter format issues from Fog migration
+* Fix website configuration parameters to use symbol keys instead of strings
+* Fix S3 object metadata parameter format to use correct key suffixes
+* Remove obsolete Fog-style constants (CONTENT_MD5_KEY, REDIRECT_KEY)
+* Add comprehensive test suite for AWS SDK parameter validation (18 new tests)
+* Improve compatibility with AWS SDK v3 to prevent API errors
+
 ## v4.6.1
 
 * Add CloudFront rate limit handling with exponential backoff retry logic

data/WARP.md

@@ -0,0 +1,112 @@
+# WARP.md
+
+This file provides guidance to WARP (warp.dev) when working with code in this repository.
+
+## Project Overview
+
+middleman-s3_sync is a Ruby gem that provides intelligent S3 synchronization for Middleman static sites. Unlike other sync tools, it only transfers files that have been added, updated, or deleted, making deployments more efficient. The gem also supports CloudFront cache invalidation and advanced caching policies.
+
+## Key Commands
+
+### Development
+```bash
+# Install dependencies
+bundle install
+
+# Run tests
+bundle exec rspec
+
+# Run specific test file
+bundle exec rspec spec/resource_spec.rb
+
+# Run tests with verbose output
+bundle exec rspec -fd
+
+# Build the gem
+bundle exec rake build
+
+# Install the gem locally for testing
+bundle exec rake install
+```
+
+### Testing & Quality
+```bash
+# Run all specs
+bundle exec rake spec  # or just `rake` (default task)
+
+# Run specific test patterns
+bundle exec rspec spec/*_spec.rb --pattern "*policy*"
+
+# Check gem build without installing
+gem build middleman-s3_sync.gemspec
+```
+
+## Architecture
+
+### Core Components
+
+**Main Sync Engine** (`lib/middleman/s3_sync.rb`)
+- Central orchestration of sync operations
+- Thread-safe S3 operations using mutexes
+- Parallel processing (8 threads by default) for file operations
+- Tracks CloudFront invalidation paths during sync
+
+**Extension Integration** (`lib/middleman-s3_sync/extension.rb`)
+- Middleman extension that hooks into the build process
+- Configurable options with environment variable fallbacks
+- Resource list manipulation to prepare files for sync
+
+**CLI Commands** (`lib/middleman-s3_sync/commands.rb`)
+- Thor-based command-line interface
+- Extensive option parsing for CloudFront, AWS credentials, and sync behavior
+- Support for dry-run mode and build-then-sync workflows
+
+**Resource Management** (`lib/middleman/s3_sync/resource.rb`)
+- Individual file resource handling and status determination
+- MD5-based change detection and caching policy application
+
+**CloudFront Integration** (`lib/middleman/s3_sync/cloudfront.rb`)
+- Intelligent cache invalidation with batch processing
+- Rate limit handling and retry logic
+- Path optimization and wildcard support
+
+### Key Design Patterns
+
+- **Thread Safety**: Uses mutexes for bucket and bucket_files operations
+- **Parallel Processing**: Leverages `parallel` gem for concurrent S3 operations
+- **Status-Based Operations**: Resources maintain state (create, update, delete, ignore)
+- **Configuration Cascade**: CLI options override config.rb options override .s3_sync file options override environment variables
+
+### File Status Logic
+
+The gem determines what to do with each file by comparing:
+- Local file MD5 hashes vs S3 ETags
+- Presence in local build vs S3 bucket
+- Caching policies and content encoding preferences
+
+## Testing Strategy
+
+**RSpec Structure**:
+- `caching_policy_spec.rb`: HTTP caching header generation
+- `cloudfront_spec.rb`: CloudFront invalidation logic (comprehensive, 18k lines)
+- `resource_spec.rb`: Individual file resource operations
+- `s3_sync_integration_spec.rb`: End-to-end sync workflows
+
+**Mock Strategy**: Uses AWS SDK stub responses and custom S3 object mocks for isolated testing without real AWS calls.
+
+## Configuration Files
+
+- **`.s3_sync`**: YAML configuration file for credentials and options (should be gitignored)
+- **`config.rb`**: Middleman configuration with `activate :s3_sync` block
+- **Environment Variables**: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET, etc.
+
+## Common Development Patterns
+
+When adding new functionality:
+1. Add option to `extension.rb` with appropriate defaults
+2. Add CLI flag to `commands.rb` if user-facing
+3. Implement core logic in main `s3_sync.rb` module
+4. Add comprehensive specs following existing patterns
+5. Update README.md with new configuration options
+
+The codebase emphasizes security (credential handling), efficiency (parallel operations), and reliability (comprehensive error handling and dry-run support).

data/lib/middleman/s3_sync/options.rb

@@ -10,10 +10,10 @@ module Middleman
         :region,
         :aws_access_key_id,
         :aws_secret_access_key,
+        :aws_session_token,
         :after_build,
         :delete,
         :encryption,
-        :existing_remote_file,
         :build_dir,
         :force,
         :prefer_gzip,
@@ -25,12 +25,35 @@ module Middleman
         :content_types,
         :ignore_paths,
         :index_document,
-        :error_document
+        :error_document,
+        :cloudfront_distribution_id,
+        :cloudfront_invalidate,
+        :cloudfront_invalidate_all,
+        :cloudfront_invalidation_batch_size,
+        :cloudfront_invalidation_max_retries,
+        :cloudfront_invalidation_batch_delay,
+        :cloudfront_wait
       ]
       attr_accessor *OPTIONS
 
       def acl
-        @acl || 'public-read'
+        # If @acl is explicitly set to empty string or false, return nil (for buckets with ACLs disabled)
+        # If @acl is nil and was never set, return default 'public-read'
+        # Otherwise return the set value
+        return nil if @acl == '' || @acl == false
+        @acl_explicitly_set ? @acl : (@acl || 'public-read')
+      end
+
+      def acl=(value)
+        @acl_explicitly_set = true
+        @acl = value
+      end
+
+      def acl_enabled?
+        # ACLs are disabled if explicitly set to nil, empty string, or false
+        return false if @acl_explicitly_set && (@acl.nil? || @acl == '' || @acl == false)
+        # Otherwise ACLs are enabled (using default or explicit value)
+        true
       end
 
       def aws_access_key_id=(aws_access_key_id)

data/lib/middleman/s3_sync/resource.rb

@@ -3,8 +3,6 @@ module Middleman
     class Resource
       attr_accessor :path, :resource, :partial_s3_resource, :full_s3_resource, :content_type, :gzipped, :options
 
-      CONTENT_MD5_KEY = 'x-amz-meta-content-md5'
-      REDIRECT_KEY = 'x-amz-website-redirect-location'
 
       include Status
 
@@ -57,10 +55,11 @@ module Middleman
       def to_h
         attributes = {
           :key => key,
-          :acl => options.acl,
           :content_type => content_type,
-          CONTENT_MD5_KEY => local_content_md5
+          'content-md5' => local_content_md5
         }
+        # Only add ACL if enabled (not for buckets with ACLs disabled)
+        attributes[:acl] = options.acl if options.acl_enabled?
 
         if caching_policy
           attributes[:cache_control] = caching_policy.cache_control
@@ -80,7 +79,7 @@ module Middleman
         end
 
         if redirect?
-          attributes[REDIRECT_KEY] = redirect_url
+          attributes['website-redirect-location'] = redirect_url
         end
 
         attributes
@@ -117,40 +116,22 @@ module Middleman
 
       def upload!
         object = bucket.object(remote_path.sub(/^\//, ''))
-        upload_options = {
-          body: local_content,
-          content_type: content_type,
-          acl: options.acl
-        }
-
-        # Add metadata if present
-        if local_content_md5
-          upload_options[:metadata] = { CONTENT_MD5_KEY => local_content_md5 }
-        end
-
-        # Add redirect if present
-        upload_options[:website_redirect_location] = redirect_url if redirect?
-
-        # Add content encoding if present
-        upload_options[:content_encoding] = "gzip" if options.prefer_gzip && gzipped
-
-        # Add cache control and expires if present
-        if caching_policy
-          upload_options[:cache_control] = caching_policy.cache_control
-          upload_options[:expires] = caching_policy.expires
-        end
-
-        # Add storage class if needed
-        if options.reduced_redundancy_storage
-          upload_options[:storage_class] = 'REDUCED_REDUNDANCY'
-        end
-
-        # Add encryption if needed
-        if options.encryption
-          upload_options[:server_side_encryption] = 'AES256'
+        upload_options = build_upload_options
+
+        begin
+          object.put(upload_options)
+        rescue Aws::S3::Errors::AccessControlListNotSupported => e
+          # Bucket has ACLs disabled - retry without ACL
+          if upload_options.key?(:acl)
+            say_status "#{ANSI.yellow{"Note"}} Bucket does not support ACLs, retrying without ACL parameter"
+            # Automatically disable ACLs for this bucket going forward
+            options.acl = ''
+            upload_options.delete(:acl)
+            retry
+          else
+            raise e
+          end
         end
-
-        object.put(upload_options)
       end
 
       def ignore!
@@ -292,7 +273,7 @@ module Middleman
 
       def remote_content_md5
         if full_s3_resource && full_s3_resource.metadata
-          full_s3_resource.metadata[CONTENT_MD5_KEY]
+          full_s3_resource.metadata['content-md5']
         end
       end
 
@@ -332,6 +313,45 @@ module Middleman
       end
 
       protected
+      
+      def build_upload_options
+        upload_options = {
+          body: local_content,
+          content_type: content_type
+        }
+        # Only add ACL if enabled (not for buckets with ACLs disabled)
+        upload_options[:acl] = options.acl if options.acl_enabled?
+
+        # Add metadata if present
+        if local_content_md5
+          upload_options[:metadata] = { 'content-md5' => local_content_md5 }
+        end
+
+        # Add redirect if present
+        upload_options[:website_redirect_location] = redirect_url if redirect?
+
+        # Add content encoding if present
+        upload_options[:content_encoding] = "gzip" if options.prefer_gzip && gzipped
+
+        # Add cache control and expires if present
+        if caching_policy
+          upload_options[:cache_control] = caching_policy.cache_control
+          upload_options[:expires] = caching_policy.expires
+        end
+
+        # Add storage class if needed
+        if options.reduced_redundancy_storage
+          upload_options[:storage_class] = 'REDUCED_REDUNDANCY'
+        end
+
+        # Add encryption if needed
+        if options.encryption
+          upload_options[:server_side_encryption] = 'AES256'
+        end
+
+        upload_options
+      end
+
       def bucket
         Middleman::S3Sync.bucket
       end

data/lib/middleman/s3_sync/version.rb

@@ -1,5 +1,5 @@
 module Middleman
   module S3Sync
-    VERSION = "4.6.1"
+    VERSION = "4.6.3"
   end
 end

data/lib/middleman/s3_sync.rb

@@ -107,10 +107,10 @@ module Middleman
 
       def update_bucket_website
         opts = {}
-        opts[:IndexDocument] = s3_sync_options.index_document if s3_sync_options.index_document
-        opts[:ErrorDocument] = s3_sync_options.error_document if s3_sync_options.error_document
+        opts[:index_document] = { suffix: s3_sync_options.index_document } if s3_sync_options.index_document
+        opts[:error_document] = { key: s3_sync_options.error_document } if s3_sync_options.error_document
 
-        if opts[:ErrorDocument] && !opts[:IndexDocument]
+        if opts[:error_document] && !opts[:index_document]
           raise 'S3 requires `index_document` if `error_document` is specified'
         end
 

data/lib/middleman-s3_sync/extension.rb

@@ -77,7 +77,28 @@ module Middleman
     end
 
     def s3_sync_options
-      options
+      self
+    end
+
+    def acl_enabled?
+      # ACLs are disabled if acl is explicitly set to nil, empty string, or false
+      acl_value = options.acl
+      return false if acl_value.nil? || acl_value == '' || acl_value == false
+      # Otherwise ACLs are enabled (using default or explicit value)
+      true
+    end
+
+    # Delegate option readers to the options object
+    def method_missing(method, *args, &block)
+      if options.respond_to?(method)
+        options.send(method, *args, &block)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(method, include_private = false)
+      options.respond_to?(method) || super
     end
 
     # Read config options from an IO stream and set them on `self`. Defaults

data/middleman-s3_sync.gemspec

@@ -20,15 +20,13 @@ Gem::Specification.new do |gem|
 
   gem.add_runtime_dependency 'middleman-core'
   gem.add_runtime_dependency 'middleman-cli'
-  gem.add_runtime_dependency 'unf'
-  gem.add_runtime_dependency 'aws-sdk-s3'
+  gem.add_runtime_dependency 'aws-sdk-s3', '>= 1.187.0'
   gem.add_runtime_dependency 'aws-sdk-cloudfront'
-  gem.add_runtime_dependency 'map'
+  gem.add_runtime_dependency 'map', '6.6.0'
   gem.add_runtime_dependency 'parallel'
   gem.add_runtime_dependency 'ruby-progressbar'
   gem.add_runtime_dependency 'ansi', '~> 1.5.0'
   gem.add_runtime_dependency 'mime-types', '~> 3.1'
-  gem.add_runtime_dependency 'base64'
   gem.add_runtime_dependency 'nokogiri', '>= 1.18.4'
 
   gem.add_development_dependency 'rake'