hybrid_platforms_conductor 33.3.0 → 33.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c0707386d30d5e671d5ceac5e1fec9d971cdccd2b7057c6ed3a75cb5d8bb6d85
-  data.tar.gz: e9e5c023662e7aa9283905f4ae54f4b8995d14f61bea1d844ef4fef32cba68dc
+  metadata.gz: f5f1fa4755dba3c7830397e4b43204517131487eaa5942bd0492f64bb835def5
+  data.tar.gz: 3d6d5de92054abbcfebc902f523466ba79167da40e33c25defc413ed28d2cc28
 SHA512:
-  metadata.gz: 1b70dedf6605d48081ead37771b8f94e29d26334fb1c0f7f15ed8ed70cabc24809b145bd681af9e829cd0c968accd18cf87886987391709a7252908a00104096
-  data.tar.gz: cbe9033432aae4b83b4153501efad2330651696ba0fc8b487d21f5927430d481661e5d3bf2c3617ffb20c2ba7e254f82328d4225aeba4f98295873c55fe4cca9
+  metadata.gz: 944569bc9c74fbbbeff909f6d73ae3c2f8064cb3c6853c4f29c74e1a3e906d446f7db4600b3c66d64bc8642bdd173a694dc337fa78a411cb9da30107e14c7643
+  data.tar.gz: 13acf6ece0db85b0a92caf752aa56b6510f3cfa3f82091c38f0cbd9dc0048056c9f6e9995e5db4dd97141fe0a1bea7e7db75db954b8f114c25acaadf11a7c228

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+# [v33.4.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.3.0...v33.4.0) (2021-07-05 13:24:27)
+
+### Features
+
+* [[Feature] [#83] Credentials can now be given through config DSL for better security and control](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/9dd40e82b4c71bea9686abd828adb4359f5bebb2)
+
 # [v33.3.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.2.4...v33.3.0) (2021-07-02 17:20:58)
 
 ## Global changes

data/README.md

@@ -304,12 +304,41 @@ See [the executables list](docs/executables.md) for more details.
 # Credentials
 
 Some tools or tests require authentication using user/password to an external resource. Examples of such tools are Bitbucket, Thycotic, Confluence...
-Credentials can be given using either environment variables or by parsing the user's `.netrc` file.
+Credentials can be given 3 different ways:
+* from the configuration file `hpc_config.rb`,
+* from environment variables,
+* from the user's `.netrc` file.
 
-In case a process needs a credential that has not been set, a warning message will be output so that the user knows which credential is missing, and eventually for which URL.
+In case a process needs a credential that has not been set, a warning message will be output so that the user knows which credential is missing, and eventually for which resource (URL, file...).
 
 Following sub-sections explain the different ways of setting such credentials.
 
+## Configuration
+
+The [`credentials_for` config DSL method](docs/config_dsl.md#credentials_for) can be used to define the credentials for a given credential ID and eventual resource.
+The way to do it is to provide a callback that will be called only when a credential is needed, and the credentials are given to a requester object. This way the life-cycle of the secret can be completely controlled, and clean-up can be done to ensure no vulnerabilities are staying after usage.
+
+Example:
+```ruby
+# Simple case
+credentials_for(:bitbucket) do |resource, requester|
+  requester.call 'my_bitbucket_name', 'my_bitbucket_PaSsWoRd'
+end
+
+# More secure case, handling user input and memory clean-up after usage
+credentials_for(:bitbucket) do |resource, requester|
+  puts 'Input Bitbucket password...'
+  password = ''
+  $stdin.noecho { |io| io.sysread(256, password) }
+  begin
+    password.chomp!
+    requester.call 'my_bitbucket_name', password
+  ensure
+    SecretString.erase(password)
+  end
+end
+```
+
 ## Environment variables
 
 Environment variables used for credentials are always named following this convention: `hpc_user_for_<credential_id>` and `hpc_password_for_<credential_id>`.

data/docs/config_dsl.md

@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
   * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
   * [`tests_provisioner`](#tests_provisioner)
   * [`expect_tests_to_fail`](#expect_tests_to_fail)
+  * [`credentials_for`](#credentials_for)
   * [`read_secrets_from`](#read_secrets_from)
   * [`send_logs_to`](#send_logs_to)
   * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
@@ -202,6 +203,48 @@ for_nodes('/tst/') do
 end
 ```
 
+<a name="credentials_for"></a>
+## `credentials_for`
+
+Set the credentials to be used (user, password) for a given credential ID and an optional resources selection.
+
+Credentials can be used by any plugin (using the [`with_credentials_for`](/lib/hybrid_platforms_conductor/credentials.rb) method) and are used by various processes in Hybrid Platforms Conductor. See each [plugin's documentation](plugins.md) to know which plugin uses which credential.
+
+It takes the following parameters:
+* the credential ID (as a Symbol),
+* an optional resource specification (can be a String for a complete resource name, or a Regexp matching resources' names),
+* a code block that will be called back by any process needing credentials matching the ID and resource specification.
+The code block will be given both the resource name being accessed, and a requester object that needs to be given the corresponding user and password. When the requester finishes running, the credentials are not needed anymore and should be cleaned from memory to avoid vulnerabilities.
+
+Examples:
+```ruby
+# Using an environment variable as a password
+credentials_for(:github) do |resource, requester|
+  requester.call 'MyUserName', ENV['MY_GITHUB_PASSWORD']
+end
+
+# Using user input and cleaning memory
+credentials_for(:github) do |resource, requester|
+  puts 'Input Github password...'
+  password = ''
+  $stdin.noecho { |io| io.sysread(256, password) }
+  begin
+    password.chomp!
+    requester.call 'MyUserName', password
+  ensure
+    SecretString.erase(password)
+  end
+end
+
+# Defining different credentials based on the resource being accessed
+credentials_for(:github, resource: %r{github.com/my-projects}) do |resource, requester|
+  requester.call 'MyUserName', 'MyPassword'
+end
+credentials_for(:github, resource: %r{github.com/company}) do |resource, requester|
+  requester.call 'CompanyUserName', 'CompanyPassword'
+end
+```
+
 <a name="read_secrets_from"></a>
 ## `read_secrets_from`
 

data/lib/hybrid_platforms_conductor/bitbucket.rb

@@ -7,116 +7,160 @@ require 'hybrid_platforms_conductor/logger_helpers'
 
 module HybridPlatformsConductor
 
-  # Object used to access Bitbucket API
-  class Bitbucket
+  # Mixin used to access Bitbucket API
+  module Bitbucket
 
-    include LoggerHelpers
+    include Credentials
 
     # Provide a Bitbucket connector, and make sure the password is being cleaned when exiting.
     #
     # Parameters::
     # * *bitbucket_url* (String): The Bitbucket URL
-    # * *logger* (Logger): Logger to be used
-    # * *logger_stderr* (Logger): Logger to be used for stderr
     # * Proc: Code called with the Bitbucket instance.
-    #   * *bitbucket* (Bitbucket): The Bitbucket instance to use.
-    def self.with_bitbucket(bitbucket_url, logger, logger_stderr)
-      Credentials.with_credentials_for(:bitbucket, logger, logger_stderr, url: bitbucket_url) do |bitbucket_user, bitbucket_password|
-        yield Bitbucket.new(bitbucket_url, bitbucket_user, bitbucket_password, logger: logger, logger_stderr: logger_stderr)
+    #   * *bitbucket* (BitbucketApi): The Bitbucket instance to use.
+    def with_bitbucket(bitbucket_url)
+      with_credentials_for(:bitbucket, resource: bitbucket_url) do |bitbucket_user, bitbucket_password|
+        yield BitbucketApi.new(bitbucket_url, bitbucket_user, bitbucket_password, logger: @logger, logger_stderr: @logger_stderr)
       end
     end
 
-    # The Bitbucket URL
-    # String
-    attr_reader :bitbucket_url
-
-    # Constructor
+    # Iterate over each Bitbucket repository
     #
     # Parameters::
-    # * *bitbucket_url* (String): The Bitbucket URL
-    # * *bitbucket_user_name* (String): Bitbucket user name to be used when querying the API
-    # * *bitbucket_password* (String): Bitbucket password to be used when querying the API
-    # * *logger* (Logger): Logger to be used [default = Logger.new(STDOUT)]
-    # * *logger_stderr* (Logger): Logger to be used for stderr [default = Logger.new(STDERR)]
-    def initialize(bitbucket_url, bitbucket_user_name, bitbucket_password, logger: Logger.new($stdout), logger_stderr: Logger.new($stderr))
-      init_loggers(logger, logger_stderr)
-      @bitbucket_url = bitbucket_url
-      @bitbucket_user_name = bitbucket_user_name
-      @bitbucket_password = bitbucket_password
+    # * Proc: Code called for each Bitbucket repository:
+    #   * Parameters::
+    #     * *bitbucket* (Bitbucket): The Bitbucket instance used to query the API for this repository
+    #     * *repo_info* (Hash<Symbol, Object>): The repository info:
+    #       * *name* (String): Repository name.
+    #       * *project* (String): Project name.
+    #       * *url* (String): Project Git URL.
+    #       * *jenkins_ci_url* (String or nil): Corresponding Jenkins CI URL, or nil if none.
+    #       * *checks* (Hash<Symbol, Object>): Checks to be performed on this repository:
+    #         * *branch_permissions* (Array< Hash<Symbol, Object> >): List of branch permissions to check [optional]
+    #           * *type* (String): Type of branch permissions to check. Examples of values are 'fast-forward-only', 'no-deletes', 'pull-request-only'.
+    #           * *branch* (String): Branch on which those permissions apply.
+    #           * *exempted_users* (Array<String>): List of exempted users for this permission [default: []]
+    #           * *exempted_groups* (Array<String>): List of exempted groups for this permission [default: []]
+    #           * *exempted_keys* (Array<String>): List of exempted access keys for this permission [default: []]
+    #         * *pr_settings* (Hash<Symbol, Object>): PR specific settings to check [optional]
+    #           * *required_approvers* (Integer): Number of required approvers [optional]
+    #           * *required_builds* (Integer): Number of required successful builds [optional]
+    #           * *default_merge_strategy* (String): Name of the default merge strategy. Example: 'rebase-no-ff' [optional]
+    #           * *mandatory_default_reviewers* (Array<String>): List of mandatory reviewers to check [default: []]
+    def for_each_bitbucket_repo
+      @config.known_bitbucket_repos.each do |bitbucket_repo_info|
+        with_bitbucket(bitbucket_repo_info[:url]) do |bitbucket|
+          (bitbucket_repo_info[:repos] == :all ? bitbucket.repos(bitbucket_repo_info[:project])['values'].map { |repo_info| repo_info['slug'] } : bitbucket_repo_info[:repos]).each do |name|
+            yield bitbucket, {
+              name: name,
+              project: bitbucket_repo_info[:project],
+              url: "#{bitbucket_repo_info[:url]}/scm/#{bitbucket_repo_info[:project].downcase}/#{name}.git",
+              jenkins_ci_url: bitbucket_repo_info[:jenkins_ci_url].nil? ? nil : "#{bitbucket_repo_info[:jenkins_ci_url]}/job/#{name}",
+              checks: bitbucket_repo_info[:checks]
+            }
+          end
+        end
+      end
     end
 
-    # Get the repositories of a given project.
-    # Limit to 1000 results max.
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # Result::
-    # * Object: Corresponding JSON
-    def repos(project)
-      get_api("projects/#{project}/repos?limit=1000")
-    end
+    # Provide an API to Bitbucket
+    class BitbucketApi
 
-    # Get the PR settings of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def settings_pr(project, repo)
-      get_api("projects/#{project}/repos/#{repo}/settings/pull-requests")
-    end
+      include LoggerHelpers
 
-    # Get the default reviewers of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def default_reviewers(project, repo)
-      get_api("projects/#{project}/repos/#{repo}/conditions", api_domain: 'default-reviewers')
-    end
+      # The Bitbucket URL
+      # String
+      attr_reader :bitbucket_url
 
-    # Get the branch permissions of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def branch_permissions(project, repo)
-      # Put 3 retries here as the Bitbucket installation has a very unstable API 2.0 and often returns random 401 errors.
-      get_api("projects/#{project}/repos/#{repo}/restrictions", api_domain: 'branch-permissions', api_version: '2.0', retries: 3)
-    end
+      # Constructor
+      #
+      # Parameters::
+      # * *bitbucket_url* (String): The Bitbucket URL
+      # * *bitbucket_user_name* (String): Bitbucket user name to be used when querying the API
+      # * *bitbucket_password* (String): Bitbucket password to be used when querying the API
+      # * *logger* (Logger): Logger to be used [default = Logger.new(STDOUT)]
+      # * *logger_stderr* (Logger): Logger to be used for stderr [default = Logger.new(STDERR)]
+      def initialize(bitbucket_url, bitbucket_user_name, bitbucket_password, logger: Logger.new($stdout), logger_stderr: Logger.new($stderr))
+        init_loggers(logger, logger_stderr)
+        @bitbucket_url = bitbucket_url
+        @bitbucket_user_name = bitbucket_user_name
+        @bitbucket_password = bitbucket_password
+      end
 
-    # Issue an HTTP get on the API.
-    # Handle authentication.
-    #
-    # Parameters::
-    # * *path* (String): API path to access
-    # * *api_domain* (String): API domain to access [default: 'api']
-    # * *api_version* (String): API version to access [default: '1.0']
-    # * *retries* (Integer): Number of retries in case of failures [default: 0]
-    # Result::
-    # * Object: Returned JSON
-    def get_api(path, api_domain: 'api', api_version: '1.0', retries: 0)
-      api_url = "#{@bitbucket_url}/rest/#{api_domain}/#{api_version}/#{path}"
-      log_debug "Call Bitbucket API #{@bitbucket_user_name}@#{api_url}..."
-      http_response = nil
-      loop do
-        begin
-          http_response = URI.parse(api_url).open(http_basic_authentication: [@bitbucket_user_name, @bitbucket_password])
-        rescue
-          raise if retries.zero?
-
-          log_warn "Got error #{$ERROR_INFO} on #{@bitbucket_user_name}@#{api_url}. Will retry #{retries} times..."
-          retries -= 1
-          sleep 1
+      # Get the repositories of a given project.
+      # Limit to 1000 results max.
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # Result::
+      # * Object: Corresponding JSON
+      def repos(project)
+        get_api("projects/#{project}/repos?limit=1000")
+      end
+
+      # Get the PR settings of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def settings_pr(project, repo)
+        get_api("projects/#{project}/repos/#{repo}/settings/pull-requests")
+      end
+
+      # Get the default reviewers of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def default_reviewers(project, repo)
+        get_api("projects/#{project}/repos/#{repo}/conditions", api_domain: 'default-reviewers')
+      end
+
+      # Get the branch permissions of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def branch_permissions(project, repo)
+        # Put 3 retries here as the Bitbucket installation has a very unstable API 2.0 and often returns random 401 errors.
+        get_api("projects/#{project}/repos/#{repo}/restrictions", api_domain: 'branch-permissions', api_version: '2.0', retries: 3)
+      end
+
+      # Issue an HTTP get on the API.
+      # Handle authentication.
+      #
+      # Parameters::
+      # * *path* (String): API path to access
+      # * *api_domain* (String): API domain to access [default: 'api']
+      # * *api_version* (String): API version to access [default: '1.0']
+      # * *retries* (Integer): Number of retries in case of failures [default: 0]
+      # Result::
+      # * Object: Returned JSON
+      def get_api(path, api_domain: 'api', api_version: '1.0', retries: 0)
+        api_url = "#{@bitbucket_url}/rest/#{api_domain}/#{api_version}/#{path}"
+        log_debug "Call Bitbucket API #{@bitbucket_user_name}@#{api_url}..."
+        http_response = nil
+        loop do
+          begin
+            http_response = URI.parse(api_url).open(http_basic_authentication: [@bitbucket_user_name, @bitbucket_password])
+          rescue
+            raise if retries.zero?
+
+            log_warn "Got error #{$ERROR_INFO} on #{@bitbucket_user_name}@#{api_url}. Will retry #{retries} times..."
+            retries -= 1
+            sleep 1
+          end
+          break unless http_response.nil?
         end
-        break unless http_response.nil?
+        JSON.parse(http_response.read)
       end
-      JSON.parse(http_response.read)
+
     end
 
   end

data/lib/hybrid_platforms_conductor/common_config_dsl/bitbucket.rb

@@ -1,5 +1,3 @@
-require 'hybrid_platforms_conductor/bitbucket'
-
 module HybridPlatformsConductor
 
   module CommonConfigDsl
@@ -7,12 +5,21 @@ module HybridPlatformsConductor
     # Add common Bitbucket config DSL to declare known Bitbucket repositories
     module Bitbucket
 
+      # List of known Bitbucket repos
+      # Array< Hash<Symbol, Object> >
+      # * *url* (String): URL to the Bitbucket server
+      # * *project* (String): Project name from the Bitbucket server, storing repositories
+      # * *repos* (Array<String> or Symbol): List of repository names from this project, or :all for all
+      # * *jenkins_ci_url* (String or nil): Corresponding Jenkins CI URL, or nil if none
+      # * *checks* (Hash<Symbol, Object>): Checks definition to be perform on those repositories (see the #for_each_bitbucket_repo to know the structure)
+      attr_reader :known_bitbucket_repos
+
       # Initialize the DSL
       def init_bitbucket
         # List of Bitbucket repositories definitions
         # Array< Hash<Symbol, Object> >
-        # Each definition is just mapping the signature of #bitbucket_repos
-        @bitbucket_repos = []
+        # Each definition is just mapping the signature of #known_bitbucket_repos
+        @known_bitbucket_repos = []
       end
 
       # Register new Bitbucket repositories
@@ -24,7 +31,7 @@ module HybridPlatformsConductor
       # * *jenkins_ci_url* (String or nil): Corresponding Jenkins CI URL, or nil if none [default: nil]
       # * *checks* (Hash<Symbol, Object>): Checks definition to be perform on those repositories (see the #for_each_bitbucket_repo to know the structure) [default: {}]
       def bitbucket_repos(url:, project:, repos: :all, jenkins_ci_url: nil, checks: {})
-        @bitbucket_repos << {
+        @known_bitbucket_repos << {
           url: url,
           project: project,
           repos: repos,
@@ -33,45 +40,6 @@ module HybridPlatformsConductor
         }
       end
 
-      # Iterate over each Bitbucket repository
-      #
-      # Parameters::
-      # * Proc: Code called for each Bitbucket repository:
-      #   * Parameters::
-      #     * *bitbucket* (Bitbucket): The Bitbucket instance used to query the API for this repository
-      #     * *repo_info* (Hash<Symbol, Object>): The repository info:
-      #       * *name* (String): Repository name.
-      #       * *project* (String): Project name.
-      #       * *url* (String): Project Git URL.
-      #       * *jenkins_ci_url* (String or nil): Corresponding Jenkins CI URL, or nil if none.
-      #       * *checks* (Hash<Symbol, Object>): Checks to be performed on this repository:
-      #         * *branch_permissions* (Array< Hash<Symbol, Object> >): List of branch permissions to check [optional]
-      #           * *type* (String): Type of branch permissions to check. Examples of values are 'fast-forward-only', 'no-deletes', 'pull-request-only'.
-      #           * *branch* (String): Branch on which those permissions apply.
-      #           * *exempted_users* (Array<String>): List of exempted users for this permission [default: []]
-      #           * *exempted_groups* (Array<String>): List of exempted groups for this permission [default: []]
-      #           * *exempted_keys* (Array<String>): List of exempted access keys for this permission [default: []]
-      #         * *pr_settings* (Hash<Symbol, Object>): PR specific settings to check [optional]
-      #           * *required_approvers* (Integer): Number of required approvers [optional]
-      #           * *required_builds* (Integer): Number of required successful builds [optional]
-      #           * *default_merge_strategy* (String): Name of the default merge strategy. Example: 'rebase-no-ff' [optional]
-      #           * *mandatory_default_reviewers* (Array<String>): List of mandatory reviewers to check [default: []]
-      def for_each_bitbucket_repo
-        @bitbucket_repos.each do |bitbucket_repo_info|
-          HybridPlatformsConductor::Bitbucket.with_bitbucket(bitbucket_repo_info[:url], @logger, @logger_stderr) do |bitbucket|
-            (bitbucket_repo_info[:repos] == :all ? bitbucket.repos(bitbucket_repo_info[:project])['values'].map { |repo_info| repo_info['slug'] } : bitbucket_repo_info[:repos]).each do |name|
-              yield bitbucket, {
-                name: name,
-                project: bitbucket_repo_info[:project],
-                url: "#{bitbucket_repo_info[:url]}/scm/#{bitbucket_repo_info[:project].downcase}/#{name}.git",
-                jenkins_ci_url: bitbucket_repo_info[:jenkins_ci_url].nil? ? nil : "#{bitbucket_repo_info[:jenkins_ci_url]}/job/#{name}",
-                checks: bitbucket_repo_info[:checks]
-              }
-            end
-          end
-        end
-      end
-
     end
 
   end