fog-aws 3.11.0 → 3.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db78eac28d74ef1522933e294d0528ecf4f99d072594629f6befeb75927f63e6
-  data.tar.gz: 9496a31776f748834e82525bebf26a157208d7fec58f273d21508bc1c6d25062
+  metadata.gz: c8af574874772df5b7a7e51819b23f554543472c6cef0b4ff86addac3b0c24b7
+  data.tar.gz: f5171df32eec2c204d27ce7ac7ef361d5b7bc9c0761b6b95efd347b977deb97c
 SHA512:
-  metadata.gz: 0a6c79d3d5598348a49a3c1bff6b4e89804900d851a9a6e1cd328f98e2877b6ff6a66fe1703e5276f46f434d39adc3ad8d00dcef6905308168f34993278c52a5
-  data.tar.gz: d7a32f8b0df5f0f439662d8917e1eaf7ef5cc74433f48653162d74a6ae725d3fdf9a2bfe64e81f59abf8d3470c1cb44631476cd514d145249b99ca95d09615ad
+  metadata.gz: 0526c83e1825435460136d7afa5f2467e88de9171ed64fcbe0fc75871ffd2564435b310f54585de0e24e2dbb38d36fa969952c0da98787e6d43fb8128950ad36
+  data.tar.gz: b9282f04b1b59446ab5781154932461d9230e6ef58dde2d573dd8bc488722faa02e54b77a376bda9e1ab8391a673c44bd975f24be9e1ade9ec7b30f1b817c70f

data/CHANGELOG.md

@@ -1,8 +1,58 @@
 # Changelog
 
-## [Unreleased](https://github.com/fog/fog-aws/tree/HEAD)
+## [v3.14.0](https://github.com/fog/fog-aws/tree/v3.14.0) (2022-05-09)
 
-[Full Changelog](https://github.com/fog/fog-aws/compare/v3.10.0...HEAD)
+[Full Changelog](https://github.com/fog/fog-aws/compare/v3.13.0...HEAD)
+
+**Closed issues:**
+
+- Add a special note to the documentation around the danger of using directory.get [\#633](https://github.com/fog/fog-aws/issues/633)
+
+**Merged pull requests:**
+
+- RDD tags issue in AWS GovCloud Account regions. [\#643](https://github.com/fog/fog-aws/pull/643) ([svavhal](https://github.com/svavhal))
+- Create codeql.yml [\#641](https://github.com/fog/fog-aws/pull/641) ([naveensrinivasan](https://github.com/naveensrinivasan))
+- chore\(deps\): Included dependency review [\#640](https://github.com/fog/fog-aws/pull/640) ([naveensrinivasan](https://github.com/naveensrinivasan))
+- Bump actions/stale from 4 to 5 [\#639](https://github.com/fog/fog-aws/pull/639) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Set permissions for GitHub actions [\#638](https://github.com/fog/fog-aws/pull/638) ([naveensrinivasan](https://github.com/naveensrinivasan))
+- Add option to control IAM credential refresh [\#637](https://github.com/fog/fog-aws/pull/637) ([gl-gh-hchouraria](https://github.com/gl-gh-hchouraria))
+- Add warning messages around directories.get [\#636](https://github.com/fog/fog-aws/pull/636) ([orrin-naylor-instacart](https://github.com/orrin-naylor-instacart))
+- Bump actions/checkout from 2.4.0 to 3 [\#632](https://github.com/fog/fog-aws/pull/632) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Add Ruby 3.1 to the CI matrix [\#631](https://github.com/fog/fog-aws/pull/631) ([petergoldstein](https://github.com/petergoldstein))
+
+## [v3.13.0](https://github.com/fog/fog-aws/tree/v3.13.0) (2022-02-13)
+
+[Full Changelog](https://github.com/fog/fog-aws/compare/v3.12.0...v3.13.0)
+
+**Closed issues:**
+
+- Please include all files required for running tests in the gem [\#625](https://github.com/fog/fog-aws/issues/625)
+- Using Hitachi compatible S3 and still see the AWS S3 host  [\#624](https://github.com/fog/fog-aws/issues/624)
+- Spawn compute node with Elastic Inference [\#617](https://github.com/fog/fog-aws/issues/617)
+
+**Merged pull requests:**
+
+- Drop ipaddress dependency in favor of built in ipaddr [\#630](https://github.com/fog/fog-aws/pull/630) ([ekohl](https://github.com/ekohl))
+- Exclude test files from gem [\#629](https://github.com/fog/fog-aws/pull/629) ([ursm](https://github.com/ursm))
+- Add Truffleruby head to CI [\#628](https://github.com/fog/fog-aws/pull/628) ([gogainda](https://github.com/gogainda))
+- Bump actions/checkout from 2.3.5 to 2.4.0 [\#627](https://github.com/fog/fog-aws/pull/627) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Update Fog::AWS::Storage::File\#body [\#626](https://github.com/fog/fog-aws/pull/626) ([10io](https://github.com/10io))
+- Bump actions/checkout from 2.3.4 to 2.3.5 [\#623](https://github.com/fog/fog-aws/pull/623) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Add json files to the gem file [\#622](https://github.com/fog/fog-aws/pull/622) ([acant](https://github.com/acant))
+
+## [v3.12.0](https://github.com/fog/fog-aws/tree/v3.12.0) (2021-08-23)
+
+[Full Changelog](https://github.com/fog/fog-aws/compare/v3.11.0...v3.12.0)
+
+**Merged pull requests:**
+
+-  Add support for regional STS endpoints [\#620](https://github.com/fog/fog-aws/pull/620) ([stanhu](https://github.com/stanhu))
+- Add IPv6 support for Ingress Security Groups [\#619](https://github.com/fog/fog-aws/pull/619) ([p8](https://github.com/p8))
+- Separate CHANGELOG entry for 3.11.0 [\#618](https://github.com/fog/fog-aws/pull/618) ([sunny](https://github.com/sunny))
+
+## [v3.11.0](https://github.com/fog/fog-aws/tree/v3.11.0) (2021-08-05)
+
+[Full Changelog](https://github.com/fog/fog-aws/compare/v3.10.0...v3.11.0)
 
 **Closed issues:**
 
@@ -136,7 +186,7 @@
 
 **Merged pull requests:**
 
-- Handle multiple versions of objects in multiple delete request [\#600](https://github.com/fog/fog-aws/pull/600) ([shanu-idrive](https://github.com/shanu-idrive))
+- Handle multiple versions of objects in multiple delete request [\#600](https://github.com/fog/fog-aws/pull/600) ([shanu-kr](https://github.com/shanu-kr))
 - Add Truffleruby head to CI [\#596](https://github.com/fog/fog-aws/pull/596) ([gogainda](https://github.com/gogainda))
 - Fixes domain name duplication in url [\#593](https://github.com/fog/fog-aws/pull/593) ([midhunkrishna](https://github.com/midhunkrishna))
 

data/README.md

@@ -98,6 +98,7 @@ file = directory.files.create(key: 'user/1/Gemfile', body: File.open('Gemfile'),
 directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
 directory.files
 ```
+**Warning!** `s3.directories.get` retrieves and caches meta data for the first 10,000 objects in the bucket, which can be very expensive. When possible use `s3.directories.new`.
 
 #### Generating a URL for a file:
 
@@ -105,6 +106,48 @@ directory.files
 directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
 ```
 
+##### Controlling credential refresh time with IAM authentication
+
+When using IAM authentication with
+[temporary security credentials](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html),
+generated S3 pre-signed URLs
+[only last as long as the temporary credential](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html).
+
+Generating the URLs in the following manner will return a URL
+that will not last as long as its requested expiration time if
+the remainder of the authentication token lifetime was shorter.
+
+```ruby
+s3 = Fog::Storage.new(provider: 'AWS', use_iam_auth: true)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
+By default the temporary credentials in use are refreshed only within the last
+15 seconds of its expiration time. The URL requested with 60 seconds lifetime
+using the above example will only remain valid for 15 seconds in the worst case.
+
+The problem can be avoided by refreshing the token early and often,
+by setting configuration `aws_credentials_refresh_threshold_seconds` (default: 15)
+which controls the time when the refresh must occur. It is expressed in seconds
+before the temporary credential's expiration time.
+
+The following example can ensure pre-signed URLs last as long as 60 seconds
+by automatically refreshing the credentials when its remainder lifetime
+is lower than 60 seconds:
+
+```ruby
+s3 = Fog::Storage.new(
+  provider: 'AWS',
+  use_iam_auth: true,
+  aws_credentials_refresh_threshold_seconds: 60
+)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
 #### Copying a file
 
 ```ruby

data/fog-aws.gemspec

@@ -14,10 +14,10 @@ Gem::Specification.new do |spec|
   spec.homepage      = "https://github.com/fog/fog-aws"
   spec.license       = "MIT"
 
-  spec.files         = Dir['lib/**/*.rb', 'tests/**/*', 'CHANGELOG.md', 'CONTRIBUTING.md',
-                           'CONTRIBUTORS.md', 'LICENSE.md', 'README.md', 'fog-aws.gemspec',]
+  spec.files         = Dir['lib/**/*.{rb,json}',
+                           'CHANGELOG.md', 'CONTRIBUTING.md', 'CONTRIBUTORS.md',
+                           'LICENSE.md', 'README.md', 'fog-aws.gemspec',]
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
   spec.required_ruby_version = '>= 2.0.0'
@@ -31,5 +31,4 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'fog-core',  '~> 2.1'
   spec.add_dependency 'fog-json',  '~> 1.1'
   spec.add_dependency 'fog-xml',   '~> 0.1'
-  spec.add_dependency 'ipaddress', '~> 0.8'
 end

data/lib/fog/aws/compute.rb

@@ -233,21 +233,24 @@ module Fog
                         'fromPort'    => -1,
                         'toPort'      => -1,
                         'ipProtocol'  => 'icmp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       },
                       {
                         'groups'      => [{'groupName' => 'default', 'userId' => owner_id, 'groupId' => security_group_id}],
                         'fromPort'    => 0,
                         'toPort'      => 65535,
                         'ipProtocol'  => 'tcp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       },
                       {
                         'groups'      => [{'groupName' => 'default', 'userId' => owner_id, 'groupId' => security_group_id}],
                         'fromPort'    => 0,
                         'toPort'      => 65535,
                         'ipProtocol'  => 'udp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       }
                     ],
                     'ownerId'             => owner_id

data/lib/fog/aws/credential_fetcher.rb

@@ -13,8 +13,6 @@ module Fog
 
       CONTAINER_CREDENTIALS_HOST = "http://169.254.170.2"
 
-      STS_GLOBAL_ENDPOINT = "https://sts.amazonaws.com"
-
       module ServiceMethods
         def fetch_credentials(options)
           if options[:use_iam_profile] && Fog.mocking?
@@ -23,7 +21,7 @@ module Fog
           if options[:use_iam_profile]
             begin
               role_data = nil
-              region = options[:region]
+              region = options[:region] || ENV["AWS_DEFAULT_REGION"]
 
               if ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
                 connection = options[:connection] || Excon.new(CONTAINER_CREDENTIALS_HOST)
@@ -44,7 +42,15 @@ module Fog
                   :WebIdentityToken => File.read(options[:aws_web_identity_token_file] || ENV.fetch("AWS_WEB_IDENTITY_TOKEN_FILE")),
                   :Version => "2011-06-15",
                 }
-                connection = options[:connection] || Excon.new(STS_GLOBAL_ENDPOINT, :query => params)
+
+                sts_endpoint =
+                  if ENV["AWS_STS_REGIONAL_ENDPOINTS"] == "regional" && region
+                    "https://sts.#{region}.amazonaws.com"
+                  else
+                    "https://sts.amazonaws.com"
+                  end
+
+                connection = options[:connection] || Excon.new(sts_endpoint, :query => params)
                 document = Nokogiri::XML(connection.get(:idempotent => true, :expects => 200).body)
 
                 session = {
@@ -65,18 +71,19 @@ module Fog
                 role_name = connection.get(:path => INSTANCE_METADATA_PATH, :idempotent => true, :expects => 200, :headers => token_header).body
                 role_data = connection.get(:path => INSTANCE_METADATA_PATH+role_name, :idempotent => true, :expects => 200, :headers => token_header).body
                 session = Fog::JSON.decode(role_data)
-                
+
                 region ||= connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200, :headers => token_header).body[0..-2]
               end
-              
+
               credentials = {}
               credentials[:aws_access_key_id] = session['AccessKeyId']
               credentials[:aws_secret_access_key] = session['SecretAccessKey']
               credentials[:aws_session_token] = session['Token']
               credentials[:aws_credentials_expire_at] = Time.xmlschema session['Expiration']
-              
+
               # set region by default to the one the instance is in.
               credentials[:region] = region
+              credentials[:sts_endpoint] = sts_endpoint if sts_endpoint
               #these indicate the metadata service is unavailable or has no profile setup
               credentials
             rescue Excon::Error => e
@@ -116,10 +123,17 @@ module Fog
 
         private
 
+        # When defined, 'aws_credentials_refresh_threshold_seconds' controls
+        # when the credential needs to be refreshed, expressed in seconds before
+        # the current credential's expiration time
+        def credentials_refresh_threshold
+          @aws_credentials_refresh_threshold_seconds || 15
+        end
+
         def credentials_expired?
           @use_iam_profile &&
             (!@aws_credentials_expire_at ||
-             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - 15)) #new credentials become available from around 5 minutes before expiration time
+             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - credentials_refresh_threshold)) #new credentials become available from around 5 minutes before expiration time
         end
 
         def refresh_credentials