fog-aws 3.5.2 → 3.19.0

data/LICENSE.md

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2014-2017 [CONTRIBUTORS.md](https://github.com/fog/fog-aws/blob/master/CONTRIBUTORS.md)
+Copyright (c) 2014-2019 [CONTRIBUTORS.md](https://github.com/fog/fog-aws/blob/master/CONTRIBUTORS.md)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

data/README.md

@@ -1,12 +1,10 @@
 # Fog::Aws
 
 ![Gem Version](https://badge.fury.io/rb/fog-aws.svg)
-[![Build Status](https://travis-ci.org/fog/fog-aws.svg?branch=master)](https://travis-ci.org/fog/fog-aws)
-[![Dependency Status](https://gemnasium.com/fog/fog-aws.svg)](https://gemnasium.com/fog/fog-aws)
+[![Build Status](https://github.com/fog/fog-aws/actions/workflows/ruby.yml/badge.svg)](https://github.com/fog/fog-aws/actions/workflows/ruby.yml)
 [![Test Coverage](https://codeclimate.com/github/fog/fog-aws/badges/coverage.svg)](https://codeclimate.com/github/fog/fog-aws)
 [![Code Climate](https://codeclimate.com/github/fog/fog-aws.svg)](https://codeclimate.com/github/fog/fog-aws)
 
-
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -39,7 +37,10 @@ default:
   aws_secret_access_key: <YOUR_SECRET_ACCESS_KEY>
 ```
 
-### Connecting to EC2 service
+### EC2
+
+#### Connecting to the EC2 Service:
+
 ```ruby
 ec2 = Fog::Compute.new :provider => 'AWS', :region => 'us-west-2'
 ```
@@ -50,7 +51,7 @@ You can review all the requests available with this service using ```#requests``
 ec2.requests # => [:allocate_address, :assign_private_ip_addresses, :associate_address, ...]
 ```
 
-### Launch an EC2 on-demand instance:
+#### Launch an EC2 on-demand instance:
 
 ```ruby
 response = ec2.run_instances(
@@ -67,19 +68,128 @@ instance.wait_for { ready? }
 puts instance.public_ip_address # => "356.300.501.20"
 ```
 
-### Terminate an EC2 instance:
+#### Terminate an EC2 instance:
 
 ```ruby
 instance = ec2.servers.get("i-02db5af4")
 instance.destroy
 ```
 
-Fog::AWS is more than EC2 since it supports many services provided by AWS. The best way to learn and to know about how many services are supported is to take a look at the source code. To review the tests directory and to play with the library in ```irb``` can be very helpful resources as well.
+`Fog::AWS` is more than EC2 since it supports many services provided by AWS. The best way to learn and to know about how many services are supported is to take a look at the source code. To review the tests directory and to play with the library in ```bin/console``` can be very helpful resources as well.
+
+### S3
+
+#### Connecting to the S3 Service:
+
+```ruby
+s3 = Fog::Storage.new(provider: 'AWS', region: 'eu-central-1')
+```
+
+#### Creating a file:
+
+```ruby
+directory = s3.directories.new(key: 'gaudi-portal-dev')
+file = directory.files.create(key: 'user/1/Gemfile', body: File.open('Gemfile'), tags: 'Org-Id=1&Service-Name=My-Service')
+```
+
+#### Listing files:
+
+```ruby
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+directory.files
+```
+**Warning!** `s3.directories.get` retrieves and caches meta data for the first 10,000 objects in the bucket, which can be very expensive. When possible use `s3.directories.new`.
+
+#### Generating a URL for a file:
+
+```ruby
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
+##### Generate download URL
+You should pass an option argument that contains the `query` key with `response-content-disposition` inside indicating that is an attachment and the filename to be used when downloaded.
+
+```ruby
+options = {
+  query: {
+    'response-content-disposition' => "attachment; filename=#{key}"
+  }
+}
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60, options)
+```
+
+
+##### Controlling credential refresh time with IAM authentication
+
+When using IAM authentication with
+[temporary security credentials](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html),
+generated S3 pre-signed URLs
+[only last as long as the temporary credential](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html).
+
+Generating the URLs in the following manner will return a URL
+that will not last as long as its requested expiration time if
+the remainder of the authentication token lifetime was shorter.
+
+```ruby
+s3 = Fog::Storage.new(provider: 'AWS', use_iam_profile: true)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
+By default the temporary credentials in use are refreshed only within the last
+15 seconds of its expiration time. The URL requested with 60 seconds lifetime
+using the above example will only remain valid for 15 seconds in the worst case.
+
+The problem can be avoided by refreshing the token early and often,
+by setting configuration `aws_credentials_refresh_threshold_seconds` (default: 15)
+which controls the time when the refresh must occur. It is expressed in seconds
+before the temporary credential's expiration time.
+
+The following example can ensure pre-signed URLs last as long as 60 seconds
+by automatically refreshing the credentials when its remainder lifetime
+is lower than 60 seconds:
+
+```ruby
+s3 = Fog::Storage.new(
+  provider: 'AWS',
+  use_iam_profile: true,
+  aws_credentials_refresh_threshold_seconds: 60
+)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
+#### Copying a file
+
+```ruby
+directory = s3.directories.new(key: 'gaudi-portal-dev')
+file = directory.files.get('user/1/Gemfile')
+file.copy("target-bucket", "user/2/Gemfile.copy")
+```
+
+To speed transfers of large files, the `concurrency` option can be used
+to spawn multiple threads. Note that the file must be at least 5 MB for
+multipart uploads to work. For example:
+
+```ruby
+directory = s3.directories.new(key: 'gaudi-portal-dev')
+file = directory.files.get('user/1/Gemfile')
+file.multipart_chunk_size = 10 * 1024 * 1024
+file.concurrency = 10
+file.copy("target-bucket", "user/2/Gemfile.copy")
+```
 
 ## Documentation
 
 See the [online documentation](http://www.rubydoc.info/github/fog/fog-aws) for a complete API reference.
 
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
 ## Contributing
 
 1. Fork it ( https://github.com/fog/fog-aws/fork )

data/fog-aws.gemspec

@@ -14,21 +14,21 @@ Gem::Specification.new do |spec|
   spec.homepage      = "https://github.com/fog/fog-aws"
   spec.license       = "MIT"
 
-  spec.files         = `git ls-files -z`.split("\x0")
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
-  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.files         = Dir['lib/**/*.{rb,json}',
+                           'CHANGELOG.md', 'CONTRIBUTING.md', 'CONTRIBUTORS.md',
+                           'LICENSE.md', 'README.md', 'fog-aws.gemspec',]
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
   spec.required_ruby_version = '>= 2.0.0'
 
-  spec.add_development_dependency 'bundler', '~> 2.0'
-  spec.add_development_dependency 'github_changelog_generator', '~> 1.14'
-  spec.add_development_dependency 'rake',    '~> 10.0'
-  spec.add_development_dependency 'rubyzip', '~> 1.2.1'
+  spec.add_development_dependency 'bundler'
+  spec.add_development_dependency 'github_changelog_generator', '~> 1.16'
+  spec.add_development_dependency 'rake',    '>= 12.3.3'
+  spec.add_development_dependency 'rubyzip', '~> 2.3.0'
   spec.add_development_dependency 'shindo',  '~> 0.3'
 
   spec.add_dependency 'fog-core',  '~> 2.1'
   spec.add_dependency 'fog-json',  '~> 1.1'
   spec.add_dependency 'fog-xml',   '~> 0.1'
-  spec.add_dependency 'ipaddress', '~> 0.8'
 end

data/lib/fog/aws/compute.rb

@@ -233,21 +233,24 @@ module Fog
                         'fromPort'    => -1,
                         'toPort'      => -1,
                         'ipProtocol'  => 'icmp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       },
                       {
                         'groups'      => [{'groupName' => 'default', 'userId' => owner_id, 'groupId' => security_group_id}],
                         'fromPort'    => 0,
                         'toPort'      => 65535,
                         'ipProtocol'  => 'tcp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       },
                       {
                         'groups'      => [{'groupName' => 'default', 'userId' => owner_id, 'groupId' => security_group_id}],
                         'fromPort'    => 0,
                         'toPort'      => 65535,
                         'ipProtocol'  => 'udp',
-                        'ipRanges'    => []
+                        'ipRanges'    => [],
+                        'ipv6Ranges'  => []
                       }
                     ],
                     'ownerId'             => owner_id

data/lib/fog/aws/credential_fetcher.rb

@@ -1,8 +1,13 @@
+# frozen_string_literal: true
+
+require 'securerandom'
+
 module Fog
   module AWS
     module CredentialFetcher
 
       INSTANCE_METADATA_HOST = "http://169.254.169.254"
+      INSTANCE_METADATA_TOKEN = "/latest/api/token"
       INSTANCE_METADATA_PATH = "/latest/meta-data/iam/security-credentials/"
       INSTANCE_METADATA_AZ = "/latest/meta-data/placement/availability-zone/"
 
@@ -16,25 +21,60 @@ module Fog
           if options[:use_iam_profile]
             begin
               role_data = nil
-              region = options[:region]
+              region = options[:region] || ENV["AWS_DEFAULT_REGION"]
 
               if ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
                 connection = options[:connection] || Excon.new(CONTAINER_CREDENTIALS_HOST)
                 credential_path = options[:credential_path] || ENV["AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"]
                 role_data = connection.get(:path => credential_path, :idempotent => true, :expects => 200).body
+                session = Fog::JSON.decode(role_data)
+
+                if region.nil?
+                  connection = options[:metadata_connection] || Excon.new(INSTANCE_METADATA_HOST)
+                  token_header = fetch_credentials_token_header(connection, options[:disable_imds_v2])
+                  region = connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200, :headers => token_header).body[0..-2]
+                end
+              elsif ENV["AWS_WEB_IDENTITY_TOKEN_FILE"]
+                params = {
+                  :Action => "AssumeRoleWithWebIdentity",
+                  :RoleArn => options[:role_arn] || ENV.fetch("AWS_ROLE_ARN"),
+                  :RoleSessionName => options[:role_session_name] || ENV["AWS_ROLE_SESSION_NAME"] || "fog-aws-#{SecureRandom.hex}",
+                  :WebIdentityToken => File.read(options[:aws_web_identity_token_file] || ENV.fetch("AWS_WEB_IDENTITY_TOKEN_FILE")),
+                  :Version => "2011-06-15",
+                }
+
+                sts_endpoint =
+                  if ENV["AWS_STS_REGIONAL_ENDPOINTS"] == "regional" && region
+                    "https://sts.#{region}.amazonaws.com"
+                  else
+                    "https://sts.amazonaws.com"
+                  end
+
+                connection = options[:connection] || Excon.new(sts_endpoint, :query => params)
+                document = Nokogiri::XML(connection.get(:idempotent => true, :expects => 200).body)
+
+                session = {
+                  "AccessKeyId" => document.css("AccessKeyId").children.text,
+                  "SecretAccessKey" => document.css("SecretAccessKey").children.text,
+                  "Token" => document.css("SessionToken").children.text,
+                  "Expiration" => document.css("Expiration").children.text,
+                }
 
                 if region.nil?
                   connection = options[:metadata_connection] || Excon.new(INSTANCE_METADATA_HOST)
-                  region = connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200).body[0..-2]
+                  token_header = fetch_credentials_token_header(connection, options[:disable_imds_v2])
+                  region = connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200, :headers => token_header).body[0..-2]
                 end
               else
                 connection = options[:connection] || Excon.new(INSTANCE_METADATA_HOST)
-                role_name = connection.get(:path => INSTANCE_METADATA_PATH, :idempotent => true, :expects => 200).body
-                role_data = connection.get(:path => INSTANCE_METADATA_PATH+role_name, :idempotent => true, :expects => 200).body
-                region ||= connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200).body[0..-2]
+                token_header = fetch_credentials_token_header(connection, options[:disable_imds_v2])
+                role_name = connection.get(:path => INSTANCE_METADATA_PATH, :idempotent => true, :expects => 200, :headers => token_header).body
+                role_data = connection.get(:path => INSTANCE_METADATA_PATH+role_name, :idempotent => true, :expects => 200, :headers => token_header).body
+                session = Fog::JSON.decode(role_data)
+
+                region ||= connection.get(:path => INSTANCE_METADATA_AZ, :idempotent => true, :expects => 200, :headers => token_header).body[0..-2]
               end
 
-              session = Fog::JSON.decode(role_data)
               credentials = {}
               credentials[:aws_access_key_id] = session['AccessKeyId']
               credentials[:aws_secret_access_key] = session['SecretAccessKey']
@@ -43,6 +83,7 @@ module Fog
 
               # set region by default to the one the instance is in.
               credentials[:region] = region
+              credentials[:sts_endpoint] = sts_endpoint if sts_endpoint
               #these indicate the metadata service is unavailable or has no profile setup
               credentials
             rescue Excon::Error => e
@@ -53,6 +94,26 @@ module Fog
             super
           end
         end
+
+        def fetch_credentials_token_header(connection, disable_imds_v2)
+          return nil if disable_imds_v2
+
+          token = connection.put(
+            :path => INSTANCE_METADATA_TOKEN,
+            :idempotent => true,
+            :expects => 200,
+            :retry_interval => 1,
+            :retry_limit => 3,
+            :read_timeout => 1,
+            :write_timeout => 1,
+            :connect_timeout => 1,
+            :headers => { "X-aws-ec2-metadata-token-ttl-seconds" => "300" }
+          ).body
+
+          { "X-aws-ec2-metadata-token" => token }
+        rescue Excon::Error
+          nil
+        end
       end
 
       module ConnectionMethods
@@ -62,10 +123,17 @@ module Fog
 
         private
 
+        # When defined, 'aws_credentials_refresh_threshold_seconds' controls
+        # when the credential needs to be refreshed, expressed in seconds before
+        # the current credential's expiration time
+        def credentials_refresh_threshold
+          @aws_credentials_refresh_threshold_seconds || 15
+        end
+
         def credentials_expired?
           @use_iam_profile &&
             (!@aws_credentials_expire_at ||
-             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - 15)) #new credentials become available from around 5 minutes before expiration time
+             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - credentials_refresh_threshold)) #new credentials become available from around 5 minutes before expiration time
         end
 
         def refresh_credentials

data/lib/fog/aws/elasticache.rb

@@ -171,8 +171,10 @@ module Fog
           @aws_credentials_expire_at = Time::now + 20
           setup_credentials(options)
           @region = options[:region] || 'us-east-1'
-          unless ['ap-northeast-1', 'ap-southeast-1', 'eu-west-1', 'us-east-1',
-                  'us-west-1', 'us-west-2', 'sa-east-1'].include?(@region)
+          unless ['ap-south-1', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3', 'ap-southeast-1', 'ap-southeast-2', 
+                  'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-central-1',
+                  'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2', 'ca-central-1',
+                  'sa-east-1', 'cn-north-1', 'cn-northwest-1', 'ap-east-1', 'us-gov-west-1'].include?(@region)
             raise ArgumentError, "Unknown region: #{@region.inspect}"
           end
         end

data/lib/fog/aws/elb.rb

@@ -142,7 +142,7 @@ module Fog
           @port       = options[:port]        || 443
           @scheme     = options[:scheme]      || 'https'
           @connection = Fog::XML::Connection.new("#{@scheme}://#{@host}:#{@port}#{@path}", @persistent, @connection_options)
-          @version    = options[:version] || '2012-06-01'
+          @version ||= options[:version] || '2012-06-01'
 
           setup_credentials(options)
         end

data/lib/fog/aws/elbv2.rb

@@ -0,0 +1,72 @@
+module Fog
+  module AWS
+    class ELBV2 < ELB
+      requires :aws_access_key_id, :aws_secret_access_key
+      recognizes :region, :host, :path, :port, :scheme, :persistent, :use_iam_profile, :aws_session_token, :aws_credentials_expire_at, :version, :instrumentor, :instrumentor_name
+
+      request_path 'fog/aws/requests/elbv2'
+      request :add_tags
+      request :create_load_balancer
+      request :describe_tags
+      request :remove_tags
+      request :describe_load_balancers
+      request :describe_listeners
+
+      class Real < ELB::Real
+        def initialize(options={})
+          @version = '2015-12-01'
+
+          super(options)
+        end
+      end
+
+      class Mock
+        def self.data
+          @data ||= Hash.new do |hash, region|
+            owner_id = Fog::AWS::Mock.owner_id
+            hash[region] = Hash.new do |region_hash, key|
+              region_hash[key] = {
+                :owner_id => owner_id,
+                :load_balancers_v2 => {}
+              }
+            end
+          end
+        end
+
+        def self.dns_name(name, region)
+          "#{name}-#{Fog::Mock.random_hex(8)}.#{region}.elb.amazonaws.com"
+        end
+
+        def self.reset
+          @data = nil
+        end
+
+        attr_reader :region
+
+        def initialize(options={})
+          @use_iam_profile = options[:use_iam_profile]
+
+          @region = options[:region] || 'us-east-1'
+          setup_credentials(options)
+
+          Fog::AWS.validate_region!(@region)
+        end
+
+        def setup_credentials(options)
+          @aws_access_key_id     = options[:aws_access_key_id]
+          @aws_secret_access_key = options[:aws_secret_access_key]
+
+          @signer = Fog::AWS::SignatureV4.new(@aws_access_key_id, @aws_secret_access_key,@region,'elasticloadbalancing')
+        end
+
+        def data
+          self.class.data[@region][@aws_access_key_id]
+        end
+
+        def reset_data
+          self.class.data[@region].delete(@aws_access_key_id)
+        end
+      end
+    end
+  end
+end

data/lib/fog/aws/kinesis.rb

@@ -125,23 +125,26 @@ module Fog
       end
 
       class Mock
-        def self.mutex
-          @mutex ||= Mutex.new
-        end
-        def mutex; self.class.mutex; end
+        @mutex = Mutex.new
 
         def self.data
-          @data ||= Hash.new do |hash, region|
-            hash[region] = Hash.new do |region_hash, key|
-              region_hash[key] = {
-                :kinesis_streams => {}
-              }
+          @mutex.synchronize do
+            @data ||= Hash.new do |hash, region|
+              hash[region] = Hash.new do |region_hash, key|
+                region_hash[key] = {
+                  :kinesis_streams => {}
+                }
+              end
             end
+            
+            yield @data if block_given?
           end
         end
 
         def self.reset
-          @data = nil
+          @mutex.synchronize do
+            @data = nil
+          end
         end
 
         def initialize(options={})
@@ -153,31 +156,36 @@ module Fog
         end
 
         def data
-          self.class.data[@region][@aws_access_key_id]
+          self.class.data do |data|
+            data[@region][@aws_access_key_id]
+          end
         end
 
         def reset_data
-          self.class.data[@region].delete(@aws_access_key_id)
+          self.class.data do |data|
+            data[@region].delete(@aws_access_key_id)
+          end
         end
 
         def self.next_sequence_number
-          mutex.synchronize do
+          @mutex.synchronize do
             @sequence_number ||= -1
             @sequence_number += 1
             @sequence_number.to_s
           end
         end
+        
         def next_sequence_number; self.class.next_sequence_number; end
 
         def self.next_shard_id
-          mutex.synchronize do
+          @mutex.synchronize do
             @shard_id ||= -1
             @shard_id += 1
             "shardId-#{@shard_id.to_s.rjust(12, "0")}"
           end
         end
+        
         def next_shard_id; self.class.next_shard_id; end
-
       end
 
     end