aws-sdk-s3 1.136.0 → 1.176.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3a563e0602ed3c1393406bb2158324b9571f7c579924f3f349fe237333d218d1
-  data.tar.gz: 4559f4a20a3abd9dcddde15fd814012e3874813d2e5cc15cd0066125aea8348f
+  metadata.gz: 6549eea72ce7643897ce882d77e1c563d04d8bc64c31195eea0692eba98a9771
+  data.tar.gz: 7662c9e60f2f3cc42f03c2dfd300426a0c57f199ca2e8d0a88239c95a6b7fc99
 SHA512:
-  metadata.gz: 03eda70ab440bdf32208be9a6f8bbe5435d77090ef6198129b9cb06b91d2aba9fb5bb3820d53380af6863f4538344d6833d507d5c46b3609d648f2c52a26561c
-  data.tar.gz: 43cd7b5022e8602f9590463478de5adf88de7cc31d1d13179551b9434a21fd6150c61009b98796eddc91a772c70e47698e58720f79733ad03ea638c27e052653
+  metadata.gz: 1bb6e5e889aa5031deaf66f795f0ad211295064bb850464944ad88c33d4f4660db8abf42fdd10816dee17fc2d99f3c221334ff7dcf87a3efb34073f4090a8ac7
+  data.tar.gz: 6f76df44f96c1ab0fd5c5a4ee372ed5c659ed326362cfdce7cd7ead1919df5dffcf6ff947c4595e94064a3fdc20a51247da6c03b98203d0e95ecf74cd1ec9aae

data/CHANGELOG.md

@@ -1,6 +1,254 @@
 Unreleased Changes
 ------------------
 
+1.176.1 (2024-12-12)
+------------------
+
+* Issue - Do not normalize object keys when calling `presigned_url` or `presigned_request`.
+
+1.176.0 (2024-12-03)
+------------------
+
+* Feature - Amazon S3 Metadata stores object metadata in read-only, fully managed Apache Iceberg metadata tables that you can query. You can create metadata table configurations for S3 general purpose buckets.
+
+1.175.0 (2024-12-02)
+------------------
+
+* Feature - Amazon S3 introduces support for AWS Dedicated Local Zones
+
+1.174.0 (2024-11-25)
+------------------
+
+* Feature - Amazon Simple Storage Service / Features: Add support for ETag based conditional writes in PutObject and CompleteMultiPartUpload APIs to prevent unintended object modifications.
+
+1.173.0 (2024-11-21)
+------------------
+
+* Feature - Add support for conditional deletes for the S3 DeleteObject and DeleteObjects APIs. Add support for write offset bytes option used to append to objects with the S3 PutObject API.
+
+1.172.0 (2024-11-18)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.171.0 (2024-11-14)
+------------------
+
+* Feature - This release updates the ListBuckets API Reference documentation in support of the new 10,000 general purpose bucket default quota on all AWS accounts. To increase your bucket quota from 10,000 to up to 1 million buckets, simply request a quota increase via Service Quotas.
+
+1.170.1 (2024-11-11)
+------------------
+
+* Issue - Tighten regex used to check for S3 200 errors.
+
+1.170.0 (2024-11-06)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.169.0 (2024-10-18)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.168.0 (2024-10-16)
+------------------
+
+* Feature - Add support for the new optional bucket-region and prefix query parameters in the ListBuckets API. For ListBuckets requests that express pagination, Amazon S3 will now return both the bucket names and associated AWS regions in the response.
+
+1.167.0 (2024-10-02)
+------------------
+
+* Feature - This release introduces a header representing the minimum object size limit for Lifecycle transitions.
+
+1.166.0 (2024-09-24)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.165.0 (2024-09-23)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.164.0 (2024-09-20)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.163.0 (2024-09-18)
+------------------
+
+* Feature - Added SSE-KMS support for directory buckets.
+
+1.162.0 (2024-09-11)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.161.0 (2024-09-10)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.160.0 (2024-09-03)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.159.0 (2024-08-20)
+------------------
+
+* Feature - Amazon Simple Storage Service / Features : Add support for conditional writes for PutObject and CompleteMultipartUpload APIs.
+
+1.158.0 (2024-08-15)
+------------------
+
+* Feature - Amazon Simple Storage Service / Features  : Adds support for pagination in the S3 ListBuckets API.
+
+1.157.0 (2024-08-01)
+------------------
+
+* Feature - Support `head_bucket`, `get_object_attributes`, `delete_objects`, and `copy_object` for Access Grants.
+
+1.156.0 (2024-07-02)
+------------------
+
+* Feature - Added response overrides to Head Object requests.
+
+1.155.0 (2024-06-28)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.154.0 (2024-06-25)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.153.0 (2024-06-24)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.152.3 (2024-06-13)
+------------------
+
+* Issue - Handle 200 errors for all S3 operations that do not have streaming responses.
+
+1.152.2 (2024-06-12)
+------------------
+
+* Issue - Revert Handling of 200 errors for all S3 operations.
+
+1.152.1 (2024-06-10)
+------------------
+
+* Issue - Handle 200 errors for all S3 operations that do not have streaming responses.
+
+1.152.0 (2024-06-05)
+------------------
+
+* Feature - Added new params copySource and key to copyObject API for supporting S3 Access Grants plugin. These changes will not change any of the existing S3 API functionality.
+
+1.151.0 (2024-05-14)
+------------------
+
+* Feature - Updated a few x-id in the http uri traits
+
+1.150.0 (2024-05-13)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.149.1 (2024-05-06)
+------------------
+
+* Issue - Fix bug where destination bucket default encryption was inadvertently overridden by source object encryption.
+
+1.149.0 (2024-04-30)
+------------------
+
+* Feature - Support S3 Access Grants authentication. Access Grants can be enabled with the `access_grants` option, and custom options can be passed into the `access_grants_credentials_provider` option. This feature requires `aws-sdk-s3control` to be installed.
+
+* Feature - Add RBS signatures for customizations of S3.
+
+1.148.0 (2024-04-25)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.147.0 (2024-04-16)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+* Issue - Omit `ContentType` plugin when generating presigned url.
+
+1.146.1 (2024-03-28)
+------------------
+
+* Issue - Fix bug where thread_count option was not being respected for multipart uploads.
+
+1.146.0 (2024-03-18)
+------------------
+
+* Feature - Fix two issues with response root node names.
+
+1.145.0 (2024-03-15)
+------------------
+
+* Feature - Documentation updates for Amazon S3.
+
+1.144.0 (2024-03-13)
+------------------
+
+* Feature - This release makes the default option for S3 on Outposts request signing to use the SigV4A algorithm when using AWS Common Runtime (CRT).
+
+1.143.1 (2024-03-12)
+------------------
+
+* Issue - Include original part errors in message when aborting multipart upload fails (#2990).
+
+1.143.0 (2024-01-26)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.142.0 (2023-12-22)
+------------------
+
+* Feature - Added additional examples for some operations.
+
+1.141.0 (2023-11-28)
+------------------
+
+* Feature - Adds support for S3 Express One Zone.
+
+* Feature - Support S3 Express authentication and endpoints. Express session auth can be disabled with the `disable_s3_express_session_auth` Client option, the `AWS_S3_DISABLE_EXPRESS_SESSION_AUTH` environment variable, and the `s3_disable_express_session_auth` shared config option. A custom `express_credentials_provider` can be configured onto the Client.
+
+1.140.0 (2023-11-27)
+------------------
+
+* Feature - Adding new params - Key and Prefix, to S3 API operations for supporting S3 Access Grants. Note - These updates will not change any of the existing S3 API functionality.
+
+* Issue - Fix thread interruptions in multipart `download_file`, `file_uploader` and `stream_uploader` (#2944).
+
+1.139.0 (2023-11-22)
+------------------
+
+* Feature - Code Generated Changes, see `./build_tools` or `aws-sdk-core`'s CHANGELOG.md for details.
+
+1.138.0 (2023-11-21)
+------------------
+
+* Feature - Add support for automatic date based partitioning in S3 Server Access Logs.
+
+1.137.0 (2023-11-17)
+------------------
+
+* Feature - Removes all default 0 values for numbers and false values for booleans
+
 1.136.0 (2023-09-26)
 ------------------
 
@@ -46,6 +294,7 @@ Unreleased Changes
 * Feature - S3 Inventory now supports Object Access Control List and Object Owner as available object metadata fields in inventory reports.
 
 * Feature - Allow Object multipart copy API to work when requiring a checksum algorithm.
+
 * Feature - Allow Object multipart copy API to optionally copy parts as they exist on the source object if it has parts, instead of generating new part ranges, when specifying `use_source_parts: true`.
 
 1.129.0 (2023-07-11)
@@ -83,7 +332,7 @@ Unreleased Changes
 1.123.2 (2023-06-12)
 ------------------
 
-* Issue - Fix issue when decrypting noncurrent versions of objects when using client side encryption (#2866). 
+* Issue - Fix issue when decrypting noncurrent versions of objects when using client side encryption (#2866).
 
 1.123.1 (2023-06-02)
 ------------------

data/VERSION

@@ -1 +1 @@
-1.136.0
+1.176.1

data/lib/aws-sdk-s3/access_grants_credentials.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require 'set'
+
+module Aws
+  module S3
+    # @api private
+    class AccessGrantsCredentials
+      include CredentialProvider
+      include RefreshingCredentials
+
+      def initialize(options = {})
+        @client = options[:client]
+        @get_data_access_params = {}
+        options.each_pair do |key, value|
+          if self.class.get_data_access_options.include?(key)
+            @get_data_access_params[key] = value
+          end
+        end
+        @async_refresh = true
+        super
+      end
+
+      # @return [S3Control::Client]
+      attr_reader :client
+
+      # @return [String]
+      attr_reader :matched_grant_target
+
+      private
+
+      def refresh
+        c = @client.get_data_access(@get_data_access_params)
+        credentials = c.credentials
+        @matched_grant_target = c.matched_grant_target
+        @credentials = Credentials.new(
+          credentials.access_key_id,
+          credentials.secret_access_key,
+          credentials.session_token
+        )
+        @expiration = credentials.expiration
+      end
+
+      class << self
+
+        # @api private
+        def get_data_access_options
+          @gdao ||= begin
+            input = Aws::S3Control::Client.api.operation(:get_data_access).input
+            Set.new(input.shape.member_names)
+          end
+        end
+
+      end
+    end
+  end
+end

data/lib/aws-sdk-s3/access_grants_credentials_provider.rb

@@ -0,0 +1,250 @@
+# frozen_string_literal: true
+
+module Aws
+  module S3
+    # @api private
+    def self.access_grants_credentials_cache
+      @access_grants_credentials_cache ||= LRUCache.new(max_entries: 100)
+    end
+
+    # @api private
+    def self.access_grants_account_id_cache
+      @access_grants_account_id_cache ||= LRUCache.new(
+        max_entries: 100,
+        expiration: 60 * 10
+      )
+    end
+
+    # Returns Credentials class for S3 Access Grants. Accepts GetDataAccess
+    # params and other configuration as options. See
+    # {Aws::S3Control::Client#get_data_access} for details.
+    class AccessGrantsCredentialsProvider
+      # @param [Hash] options
+      # @option options [Hash] :s3_control_client_options The S3 Control
+      #  client options used to create regional S3 Control clients to
+      #  create the session. Region will be set to the region of the
+      #  bucket.
+      # @option options [Aws::STS::Client] :sts_client The STS client used for
+      #  fetching the Account ID for the credentials if credentials do not
+      #  include an Account ID.
+      # @option options [Aws::S3::Client] :s3_client The S3 client used for
+      #  fetching the location of the bucket so that a regional S3 Control
+      #  client can be created. Defaults to the S3 client from the access
+      #  grants plugin.
+      # @option options [String] :privilege ('Default') The privilege to use
+      #  when requesting credentials. (see: {Aws::S3Control::Client#get_data_access})
+      # @option options [Boolean] :fallback (false) When true, if access is
+      #  denied, the provider will fall back to the configured credentials.
+      # @option options [Boolean] :caching (true) When true, credentials and
+      #  bucket account ids will be cached.
+      # @option options [Callable] :before_refresh Proc called before
+      #  credentials are refreshed.
+      def initialize(options = {})
+        @s3_control_options = options.delete(:s3_control_client_options) || {}
+        @s3_client = options.delete(:s3_client)
+        @sts_client = options.delete(:sts_client)
+        @fallback = options.delete(:fallback) || false
+        @caching = options.delete(:caching) != false
+        @s3_control_clients = {}
+        @bucket_region_cache = Aws::S3.bucket_region_cache
+        @head_bucket_mutex = Mutex.new
+        @head_bucket_call = false
+        return unless @caching
+
+        @credentials_cache = Aws::S3.access_grants_credentials_cache
+        @account_id_cache = Aws::S3.access_grants_account_id_cache
+      end
+
+      def access_grants_credentials_for(options = {})
+        target = target_prefix(
+          options[:bucket],
+          options[:key],
+          options[:prefix]
+        )
+        credentials = s3_client.config.credentials.credentials # resolves
+
+        if @caching
+          cached_credentials_for(target, options[:permission], credentials)
+        else
+          new_credentials_for(target, options[:permission], credentials)
+        end
+      rescue Aws::S3Control::Errors::AccessDenied
+        raise unless @fallback
+
+        warn 'Access denied for S3 Access Grants. Falling back to ' \
+             'configured credentials.'
+        s3_client.config.credentials
+      end
+
+      attr_accessor :s3_client
+
+      private
+
+      def s3_control_client(bucket_region)
+        @s3_control_clients[bucket_region] ||= begin
+          credentials = s3_client.config.credentials
+          config = { credentials: credentials }.merge(@s3_control_options)
+          Aws::S3Control::Client.new(config.merge(
+            region: bucket_region,
+            use_fips_endpoint: s3_client.config.use_fips_endpoint,
+            use_dualstack_endpoint: s3_client.config.use_dualstack_endpoint
+          ))
+        end
+      end
+
+      def cached_credentials_for(target, permission, credentials)
+        cached_creds = broad_search_credentials_cache_prefix(target, permission, credentials)
+        return cached_creds if cached_creds
+
+        if %w[READ WRITE].include?(permission)
+          cached_creds = broad_search_credentials_cache_prefix(target, 'READWRITE', credentials)
+          return cached_creds if cached_creds
+        end
+
+        cached_creds = broad_search_credentials_cache_characters(target, permission, credentials)
+        return cached_creds if cached_creds
+
+        if %w[READ WRITE].include?(permission)
+          cached_creds = broad_search_credentials_cache_characters(target, 'READWRITE', credentials)
+          return cached_creds if cached_creds
+        end
+
+        creds = new_credentials_for(target, permission, credentials)
+        if creds.matched_grant_target.end_with?('*')
+          # remove /* from the end of the target
+          key = credentials_cache_key(creds.matched_grant_target[0...-2], permission, credentials)
+          @credentials_cache[key] = creds
+        end
+
+        creds
+      end
+
+      def broad_search_credentials_cache_prefix(target, permission, credentials)
+        prefix = target
+        while prefix != 's3:'
+          key = credentials_cache_key(prefix, permission, credentials)
+          return @credentials_cache[key] if @credentials_cache.key?(key)
+
+          prefix = prefix.split('/', -1)[0..-2].join('/')
+        end
+        nil
+      end
+
+      def broad_search_credentials_cache_characters(target, permission, credentials)
+        prefix = target
+        while prefix != 's3://'
+          key = credentials_cache_key("#{prefix}*", permission, credentials)
+          return @credentials_cache[key] if @credentials_cache.key?(key)
+
+          prefix = prefix[0..-2]
+        end
+        nil
+      end
+
+      def new_credentials_for(target, permission, credentials)
+        bucket_region = bucket_region_for_access_grants(target)
+        client = s3_control_client(bucket_region)
+
+        AccessGrantsCredentials.new(
+          target: target,
+          account_id: account_id_for_access_grants(target, credentials),
+          permission: permission,
+          client: client
+        )
+      end
+
+      def account_id_for_access_grants(target, credentials)
+        if @caching
+          cached_account_id_for(target, credentials)
+        else
+          new_account_id_for(target, credentials)
+        end
+      end
+
+      def cached_account_id_for(target, credentials)
+        bucket = bucket_name_from(target)
+
+        if @account_id_cache.key?(bucket)
+          @account_id_cache[bucket]
+        else
+          @account_id_cache[bucket] = new_account_id_for(target, credentials)
+        end
+      end
+
+      # returns the account id associated with the access grants instance
+      def new_account_id_for(target, credentials)
+        bucket_region = bucket_region_for_access_grants(target)
+        s3_control_client = s3_control_client(bucket_region)
+        resp = s3_control_client.get_access_grants_instance_for_prefix(
+          s3_prefix: target,
+          account_id: account_id_for_credentials(bucket_region, credentials)
+        )
+        ARNParser.parse(resp.access_grants_instance_arn).account_id
+      end
+
+      def bucket_region_for_access_grants(target)
+        bucket = bucket_name_from(target)
+        # regardless of caching option, bucket region cache is always shared
+        cached_bucket_region_for(bucket)
+      end
+
+      def cached_bucket_region_for(bucket)
+        if @bucket_region_cache.key?(bucket)
+          @bucket_region_cache[bucket]
+        else
+          @bucket_region_cache[bucket] = new_bucket_region_for(bucket)
+        end
+      end
+
+      def new_bucket_region_for(bucket)
+        @head_bucket_mutex.synchronize do
+          begin
+            @head_bucket_call = true
+            @s3_client.head_bucket(bucket: bucket).bucket_region
+          rescue Aws::S3::Errors::Http301Error => e
+            e.data.region
+          ensure
+            @head_bucket_call = false
+          end
+        end
+      end
+
+      # returns the account id for the configured credentials
+      def account_id_for_credentials(region, credentials)
+        # use resolved credentials to check for account id
+        if credentials.respond_to?(:account_id) && credentials.account_id &&
+           !credentials.account_id.empty?
+          credentials.account_id
+        else
+          @sts_client ||= Aws::STS::Client.new(
+            credentials: s3_client.config.credentials,
+            region: region,
+            use_fips_endpoint: s3_client.config.use_fips_endpoint,
+            use_dualstack_endpoint: s3_client.config.use_dualstack_endpoint
+          )
+          @sts_client.get_caller_identity.account
+        end
+      end
+
+      def target_prefix(bucket, key, prefix)
+        if key && !key.empty?
+          "s3://#{bucket}/#{key}"
+        elsif prefix && !prefix.empty?
+          "s3://#{bucket}/#{prefix}"
+        else
+          "s3://#{bucket}/*"
+        end
+      end
+
+      def credentials_cache_key(target, permission, credentials)
+        "#{credentials.access_key_id}-#{credentials.secret_access_key}" \
+        "-#{permission}-#{target}"
+      end
+
+      # extracts bucket name from target prefix
+      def bucket_name_from(target)
+        URI(target).host
+      end
+    end
+  end
+end