fog-google 1.8.2 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f5891afe294dbc055e5e3f3ff46a2dab00692cb3
-  data.tar.gz: f81a710b9e7774d9ece9c8b49d02c9fe54a37fac
+  metadata.gz: 870ce94387132895473e52fcee3adc9cbce052c5
+  data.tar.gz: aa4f7954f7c441b6bcb4f4d32c49e8ab8ff78ac5
 SHA512:
-  metadata.gz: 47b3b0b0a4687ec054180bda94aa7e45a65d3e65a6c7145a574d4f3f9fbbc433da67b60e61de5ceff72664fd5839f20ccc5694ae61daae606885d3e259b4c714
-  data.tar.gz: 44ed811c6cc6246667aed5742d2b43ca312cf2864b19a8c477d1592ca9b11066d421e0ee4be21e2db9356098e794ac2800357946f75052313a8c4af0aad82788
+  metadata.gz: 6426e7e238fe7592575516799ea209c4019d2807519efb91830a694fda2ccf733432e9c28b6ae91f3311b28dd80a97015663d1a9fa0552bc67f15095c4f2fd86
+  data.tar.gz: f19bbc71aa3bba9bb6e859cb82767e64c501a5638e962a159fa6a6baa3bed9a25bd3ed69de084773dece607021eb139777fcdf30cbe0d2d137c712de888849f3

data/.fog.example

@@ -15,10 +15,8 @@
 # START GOOGLE CONFIG
 my_google_credentials:
     google_project: my-project-id
-    google_client_email: xxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com
     google_json_key_location: /path/to/my-project-xxxxxxxxxxxx.json
-    # You can also provide service account credentials with `google_json_key_string` or 
-    # with `google_key_location` and `google_key_string` for P12 private keys.
+    # You can also provide service account credentials with `google_json_key_string`
     # If so, uncomment the two following lines.
     # HMAC credentials follow a similar format:
 	#google_storage_access_key_id: GOOGXXXXXXXXXXXXXXXX

data/CHANGELOG.md

@@ -6,6 +6,35 @@ This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html
 
 ## Next
 
+## 1.9.0
+
+### User-facing
+
+#### Added
+
+- \#442 Add support for Application Default credentials [mavin]
+  - This change allows the use of Application Default Credentials so that end 
+    users can authenticate without a service account for development, testing,
+    and one-off interactions by using `:google_application_default`client 
+    option. See README for more details.
+
+### Fixed
+
+- \#444 Remove deprecated `google_client_email` option from client parameters 
+  [temikus]
+- \#446 Updating service parameters to avoid "unrecognised parameter" warnings
+  when initializing Fog client with application default auth [temikus]
+
+### Development changes
+
+#### Fixed
+
+- \#441 Update CI pipeline to Concourse V4 [temikus]
+- \#444 Rework client authentication workflow [temikus]
+  - Separate different auth streams into private helper methods
+  - Add a fallback auth option - Google Application Default credentials
+  - Minor fixes and performance optimizations
+
 ## 1.8.2
 
 ### User-facing

data/CONTRIBUTING.md

@@ -94,7 +94,6 @@ make sure you have a `:test` credential in `~/.fog`, something like:
 ```
 test:
   google_project: my-project
-  google_client_email: xxxxxxxxxxxxx-xxxxxxxxxxxxx@developer.gserviceaccount.com
   google_json_key_location: /path/to/my-project-xxxxxxxxxxxxx.json
 ```
 

data/README.md

@@ -94,6 +94,20 @@ cat .fog.example >> ~/.fog # appends the sample configuration
 vim ~/.fog                 # edit file with yout config
 ```
 
+As of `1.9.0` fog-google supports Google [application default credentials (ADC)](https://cloud.google.com/docs/authentication/production) 
+The auth method uses [Google::Auth.get_application_default](https://www.rubydoc.info/gems/googleauth/0.6.7/Google%2FAuth.get_application_default)
+under the hood.
+
+Example workflow for a GCE instance with [service account scopes](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances)
+defined: 
+
+```
+> connection = Fog::Compute::Google.new(:google_project => "my-project", :google_application_default => true)
+=> #<Fog::Compute::Google::Real:32157700...
+> connection.servers
+=> [  <Fog::Compute::Google::Server ...  ]
+```
+
 #### SSH-ing into instances
 
 If you want to be able to bootstrap SSH-able instances, (using `servers.bootstrap`,) be sure you have a key in `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`

data/ci/README.md

@@ -9,8 +9,6 @@ PR when the test completes.
 
 In order to run the fog-google Concourse Pipeline you must have an existing
 [Concourse](http://concourse.ci) environment.
-See [Deploying Concourse on Google Compute Engine](https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/blob/master/docs/concourse/README.md)
-for instructions.
 
 To deploy the pipeline:
 
@@ -28,13 +26,13 @@ file. See [Credentials Requirements](#credentials-requirements) for specific ins
 * Set the fog-google pipeline:
 
 ```
-fly -t fog-ci set-pipeline -p fog-google -c pipeline.yml -l credentials.yml
+fly -t fog-ci set-pipeline -p pr-integration -c pipeline.yml -l credentials.yml
 ```
 
 * Unpause the fog-google pipeline:
 
 ```
-fly -t fog-ci unpause-pipeline -p fog-google
+fly -t fog-ci unpause-pipeline -p pr-integration
 ```
 
 ## Credentials Requirements
@@ -42,18 +40,19 @@ fly -t fog-ci unpause-pipeline -p fog-google
 Several external pieces of authentication are needed for credentials.yml
 
 1. A JSON Service Account File for a service account with at least Editor access to the project.
-    * To get a Service Account File, see [here](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount) 
+    * To get a Service Account File, see [here](https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount)
       and create using the Project/Editor role.
 
 1. A [Github Access Token](https://github.com/blog/1509-personal-api-tokens) with at least
 `repo:status` access and a private key with push access to the `fog/fog-google` repositoy.
 
+1. A [Codecov.io](https://codecov.io/) token for tracking test coverage.
+
 These items are equivalent to your login credentials for their resources.
 
 ## Login Gotchas
 
-* If your Concourse deployment is using a self-signed certificate, use `--insecure` to
-trust the provided certificate.
+* Our concourse pipeline is using GitHub auth, providing access to manipulate the pipeline to the members of the [fog-google team](https://github.com/orgs/fog/teams/fog-google), managed through the OAuth application managed by [fog-google bot](https://github.com/fog-google-bot).
 
 * If logging into a specific team, ie `fog-google`, use `--team-name fog-google` to specify that.
 

data/ci/credentials.yml.template

@@ -3,8 +3,6 @@
 codecov_token:
 # Google Cloud Platform project to run under
 google_project:
-# Google Compute Engine Service Account email
-google_client_email:
 # Google Compute Engine Service Account JSON
 google_json_key_data: |
   {

data/ci/pipeline.yml

@@ -16,7 +16,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -40,7 +39,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -64,7 +62,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -88,7 +85,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -112,7 +108,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -136,7 +131,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -160,7 +154,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -184,7 +177,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:
@@ -208,7 +200,6 @@ jobs:
           codecov_token: {{codecov_token}}
           google_project: {{google_project}}
           google_json_key_data: {{google_json_key_data}}
-          google_client_email: {{google_client_email}}
         on_failure:
           put: pull-request
           params:

data/ci/tasks/run-int.sh

@@ -13,7 +13,6 @@ source ci/tasks/utils.sh
 popd > /dev/null
 
 check_param google_project
-check_param google_client_email
 check_param google_json_key_data
 check_param rake_task
 check_param codecov_token
@@ -23,7 +22,6 @@ echo $google_json_key_data > `pwd`/service_account_key.json
 cat >~/.fog <<EOL
 test:
   google_project: ${google_project}
-  google_client_email: ${google_client_email}
   google_json_key_location: `pwd`/service_account_key.json
 EOL
 

data/ci/tasks/run-int.yml

@@ -10,7 +10,7 @@ run:
   path: src/fog-google/ci/tasks/run-int.sh
 params:
   rake_task: replace-me
+  codecov_token: replace-me
   google_project: replace-me
-  google_client_email: replace-me
   google_json_key_data: |
     replace-me

data/lib/fog/compute/google.rb

@@ -10,7 +10,6 @@ module Fog
         :app_version,
         :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_extra_global_projects,
         :google_key_location,

data/lib/fog/dns/google.rb

@@ -8,9 +8,9 @@ module Fog
       recognizes(
         :app_name,
         :app_version,
+        :google_application_default,
         :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_key_location,
         :google_key_string,

data/lib/fog/google/monitoring.rb

@@ -8,8 +8,9 @@ module Fog
       recognizes(
         :app_name,
         :app_version,
+        :google_application_default,
+        :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_key_location,
         :google_key_string,

data/lib/fog/google/pubsub.rb

@@ -8,9 +8,9 @@ module Fog
       recognizes(
         :app_name,
         :app_version,
+        :google_application_default,
         :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_json_key_location,
         :google_json_key_string,

data/lib/fog/google/shared.rb

@@ -20,16 +20,13 @@ module Fog
       # Initializes the Google API Client
       #
       # @param [Hash] options Google API options
+      # @option options [Bool]   :google_application_default Explicitly use application default credentials
       # @option options [Google::Auth|Signet] :google_auth Manually created authorization to use
-      # @option options [String] :google_client_email A @developer.gserviceaccount.com email address to use
-      # @option options [String] :google_key_location The location of a pkcs12 key file
-      # @option options [String] :google_key_string The content of the pkcs12 key file
       # @option options [String] :google_json_key_location The location of a JSON key file
       # @option options [String] :google_json_key_string The content of the JSON key file
       # @option options [String] :google_api_scope_url The access scope URLs
       # @option options [String] :app_name The app name to set in the user agent
       # @option options [String] :app_version The app version to set in the user agent
-      # @option options [Google::APIClient] :google_client Existing Google API Client
       # @option options [Hash] :google_client_options A hash to send additional options to Google API Client
       # @return [Google::APIClient] Google API Client
       # @raises [ArgumentError] If there is any missing argument
@@ -50,25 +47,7 @@ module Fog
           raise error
         end
 
-        # Users can no longer provide their own clients due to rewrite of auth
-        # in https://github.com/google/google-api-ruby-client/ version 0.9.
-        if options[:google_client]
-          raise ArgumentError.new("Deprecated argument no longer works: google_client")
-        end
-
-        # They can also no longer use pkcs12 files, because Google's new auth
-        # library doesn't support them either.
-        if options[:google_key_location]
-          raise ArgumentError.new("Deprecated argument no longer works: google_key_location")
-        end
-        if options[:google_key_string]
-          raise ArgumentError.new("Deprecated argument no longer works: google_key_string")
-        end
-
-        # Validate required arguments
-        unless options[:google_api_scope_url]
-          raise ArgumentError.new("Missing required arguments: google_api_scope_url")
-        end
+        validate_client_options(options)
 
         application_name = "fog"
         unless options[:app_name].nil?
@@ -78,41 +57,21 @@ module Fog
         ::Google::Apis::ClientOptions.default.application_name = application_name
         ::Google::Apis::ClientOptions.default.application_version = Fog::Google::VERSION
 
-        auth = nil
-        if options[:google_json_key_location] || options[:google_json_key_string]
-          if options[:google_json_key_location]
-            json_key_location = File.expand_path(options[:google_json_key_location])
-            json_key = File.open(json_key_location, "r", &:read)
-          else
-            json_key = options[:google_json_key_string]
-          end
-
-          json_key_hash = Fog::JSON.decode(json_key)
-          unless json_key_hash.key?("client_email") || json_key_hash.key?("private_key")
-            raise ArgumentError.new("Invalid Google JSON key")
-          end
-
-          options[:google_client_email] = json_key_hash["client_email"]
-          unless options[:google_client_email]
-            raise ArgumentError.new("Missing required arguments: google_client_email")
-          end
+        if ENV["DEBUG"]
+          ::Google::Apis.logger = ::Logger.new(::STDERR)
+          ::Google::Apis.logger.level = ::Logger::DEBUG
+        end
 
-          if ENV["DEBUG"]
-            ::Google::Apis.logger = ::Logger.new(::STDERR)
-            ::Google::Apis.logger.level = ::Logger::DEBUG
-          end
+        auth = nil
 
-          auth = ::Google::Auth::ServiceAccountCredentials.make_creds(
-            :json_key_io => StringIO.new(json_key_hash.to_json),
-            :scope => options[:google_api_scope_url]
-          )
+        if options[:google_json_key_location] || options[:google_json_key_string]
+          auth = process_key_auth(options)
         elsif options[:google_auth]
           auth = options[:google_auth]
+        elsif options[:google_application_default]
+          auth = process_application_default_auth(options)
         else
-          raise ArgumentError.new(
-            "Missing required arguments: google_json_key_location, "\
-            "google_json_key_string or google_auth"
-          )
+          auth = process_fallback_auth(options)
         end
 
         ::Google::Apis::RequestOptions.default.authorization = auth
@@ -188,6 +147,102 @@ module Fog
 
         response
       end
+
+      private
+
+      # Helper method to process application default authentication
+      #
+      # @param [Hash]  options - client options hash
+      # @return [Google::Auth::DefaultCredentials] - google auth object
+      def process_application_default_auth(options)
+        ::Google::Auth.get_application_default(options[:google_api_scope_url])
+      end
+
+      # Helper method to process fallback authentication
+      # Current fallback is application default authentication
+      #
+      # @param [Hash]  options - client options hash
+      # @return [Google::Auth::DefaultCredentials] - google auth object
+      def process_fallback_auth(options)
+        Fog::Logger.warning(
+          "Didn't detect any client auth settings, " \
+          "trying to fall back to application default credentials..."
+        )
+        begin
+          return process_application_default_auth(options)
+        rescue
+          raise Fog::Errors::Error.new(
+            "Fallback auth failed, could not configure authentication for Fog client.\n" \
+              "Check your auth options, must be one of:\n" \
+              "- :google_json_key_location,\n" \
+              "- :google_json_key_string,\n" \
+              "- :google_auth,\n" \
+              "- :google_application_default,\n" \
+              "If credentials are valid - please, file a bug to fog-google." \
+          )
+        end
+      end
+
+      # Helper method to process key authentication
+      #
+      # @param [Hash]  options - client options hash
+      # @return [Google::Auth::ServiceAccountCredentials] - google auth object
+      def process_key_auth(options)
+        if options[:google_json_key_location]
+          json_key = File.read(File.expand_path(options[:google_json_key_location]))
+        elsif options[:google_json_key_string]
+          json_key = options[:google_json_key_string]
+        end
+
+        validate_json_credentials(json_key)
+
+        ::Google::Auth::ServiceAccountCredentials.make_creds(
+          :json_key_io => StringIO.new(json_key),
+          :scope => options[:google_api_scope_url]
+        )
+      end
+
+      # Helper method to sort out deprecated and missing auth options
+      #
+      # @param [Hash]  options - client options hash
+      def validate_client_options(options)
+        # Users can no longer provide their own clients due to rewrite of auth
+        # in https://github.com/google/google-api-ruby-client/ version 0.9.
+        if options[:google_client]
+          raise ArgumentError.new("Deprecated argument no longer works: google_client")
+        end
+
+        # They can also no longer use pkcs12 files, because Google's new auth
+        # library doesn't support them either.
+        if options[:google_key_location]
+          raise ArgumentError.new("Deprecated auth method no longer works: google_key_location")
+        end
+        if options[:google_key_string]
+          raise ArgumentError.new("Deprecated auth method no longer works: google_key_string")
+        end
+
+        # Google client email option is no longer needed
+        if options[:google_client_email]
+          Fog::Logger.deprecation("Argument no longer needed for auth: google_client_email")
+        end
+
+        # Validate required arguments
+        unless options[:google_api_scope_url]
+          raise ArgumentError.new("Missing required arguments: google_api_scope_url")
+        end
+      end
+
+      # Helper method to checks whether the necessary fields are present in
+      # JSON key credentials
+      #
+      # @param [String]  json_key - Google json auth key string
+      def validate_json_credentials(json_key)
+        json_key_hash = Fog::JSON.decode(json_key)
+
+        unless json_key_hash.key?("client_email") || json_key_hash.key?("private_key")
+          raise ArgumentError.new("Invalid Google JSON key")
+        end
+      end
     end
   end
 end

data/lib/fog/google/sql.rb

@@ -8,9 +8,9 @@ module Fog
       recognizes(
         :app_name,
         :app_version,
+        :google_application_default,
         :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_key_location,
         :google_key_string,

data/lib/fog/google/version.rb

@@ -1,5 +1,5 @@
 module Fog
   module Google
-    VERSION = "1.8.2".freeze
+    VERSION = "1.9.0".freeze
   end
 end

data/lib/fog/storage/google_json.rb

@@ -9,9 +9,9 @@ module Fog
       recognizes(
         :app_name,
         :app_version,
+        :google_application_default,
         :google_auth,
         :google_client,
-        :google_client_email,
         :google_client_options,
         :google_key_location,
         :google_key_string,

data/test/integration/test_authentication.rb

@@ -27,13 +27,6 @@ class TestAuthentication < FogIntegrationTest
     assert_raises(ArgumentError) { Fog::Compute::Google.new(:google_project => nil) }
   end
 
-  def test_raises_argument_error_when_google_client_email_is_missing
-    assert_raises(ArgumentError) do
-      Fog::Compute::Google.new(:google_client_email => nil,
-                               :google_json_key_location => nil)
-    end # JSON key overrides google_client_email
-  end
-
   def test_raises_argument_error_when_google_keys_are_given
     assert_raises(ArgumentError) do
       Fog::Compute::Google.new(:google_key_location => nil,

data/test/unit/storage/test_json_requests.rb

@@ -4,7 +4,6 @@ class UnitTestJsonRequests < MiniTest::Test
   def setup
     Fog.mock!
     @client = Fog::Storage.new(provider: "google",
-                               google_client_email: "",
                                google_project: "",
                                google_json_key_location: "")
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fog-google
 version: !ruby/object:Gem::Version
-  version: 1.8.2
+  version: 1.9.0
 platform: ruby
 authors:
 - Nat Welch
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-01-23 00:00:00.000000000 Z
+date: 2019-03-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog-core
@@ -758,7 +758,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.14
+rubygems_version: 2.6.14.3
 signing_key: 
 specification_version: 4
 summary: Module for the 'fog' gem to support Google.