hybrid_platforms_conductor 33.2.2 → 33.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 90e31431abb4e4924342d8057fbb2d2f54d780975ef7079498045a7897262067
-  data.tar.gz: 3bfcb046a30115a681d407559f9395766439ada582a0d57070d5bae067776292
+  metadata.gz: daf46892dd3b5a79ff0e54f16d6881b80519e8fe926e9952ad7faafd1706c31e
+  data.tar.gz: 84f9ee06afb8141f140a8754d0961217ceb14b3f8d2d1d2577792be420d42aa2
 SHA512:
-  metadata.gz: c532fc150a6ac25db61a70de51b88332728cb93b27f1275fda6cf15b0fce03d2fce9b90f7ea1c65134c0475e10708d075235e9abc2ec5710bf8e4823a98dc2e7
-  data.tar.gz: d081b32a30c854f0445f98669ba417ee80256f65558b1d274e476b20bd6a0c35133f0f711adae65c9e118a027136caaffce91cd179a2202ae32623b2469d6b54
+  metadata.gz: 7b1355694d9c7dbc5b2d2f9f6555972c25033d11bd07b14dfc728c94b2965ac1a852c91cb6ed4f20a26bc41cceb12f437a8b599177331cbf544e38ebc387cd4a
+  data.tar.gz: 8636507ef5724bd9f0b3ffbb1f08c3efd1a48d39fe7161a4421350b4a03a9afff279b5522175418742ba87f7daca4dcdb4382e10f3283c39f9816eceba04bdc3

data/CHANGELOG.md

@@ -1,3 +1,52 @@
+# [v33.5.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.4.0...v33.5.0) (2021-07-07 11:03:01)
+
+### Features
+
+* [[Feature] [Security] [#80] Use SecretString for credentials to avoid passwords and secrets accidental leakage](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/0db5f529e539b81f86b9f618a63e62bdbd58cb38)
+
+# [v33.4.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.3.0...v33.4.0) (2021-07-05 13:24:27)
+
+### Features
+
+* [[Feature] [#83] Credentials can now be given through config DSL for better security and control](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/9dd40e82b4c71bea9686abd828adb4359f5bebb2)
+
+# [v33.3.0](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.2.4...v33.3.0) (2021-07-02 17:20:58)
+
+## Global changes
+### Patches
+
+* [[Feature(secrets_reader_keepass)] [#79] Add Keepass secrets reader plugin to get secrets from KeePass databases](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/7ace48653a74f03ed535ee6b41b21242a6454ff3)
+
+## Changes for secrets_reader_keepass
+### Features
+
+* [[Feature(secrets_reader_keepass)] [#79] Add Keepass secrets reader plugin to get secrets from KeePass databases](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/7ace48653a74f03ed535ee6b41b21242a6454ff3)
+
+# [v33.2.4](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.2.3...v33.2.4) (2021-06-23 15:14:20)
+
+## Global changes
+### Patches
+
+* [[Hotfix(platform_handler_serverless_chef)] Forward environment in sudo commands](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/fd3b58875665c29dd071b5af2055eab0c45c0974)
+* [[Hotfix] Fixed unbundled environment not cleaned + Moved deployer config DSL in deployer.rb](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/2bce6dbc31d98b27f196ed646eb9aa669b0f9a86)
+
+## Changes for platform_handler_serverless_chef
+### Patches
+
+* [[Hotfix(platform_handler_serverless_chef)] Forward environment in sudo commands](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/fd3b58875665c29dd071b5af2055eab0c45c0974)
+
+# [v33.2.3](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.2.2...v33.2.3) (2021-06-23 13:45:56)
+
+## Global changes
+### Patches
+
+* [[Hotfix(provisioner_proxmox)] Add missing require in synchronization script](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/7a6c71595789c9180e1d95323fc5cf5051f2e2cd)
+
+## Changes for provisioner_proxmox
+### Patches
+
+* [[Hotfix(provisioner_proxmox)] Add missing require in synchronization script](https://github.com/sweet-delights/hybrid-platforms-conductor/commit/7a6c71595789c9180e1d95323fc5cf5051f2e2cd)
+
 # [v33.2.2](https://github.com/sweet-delights/hybrid-platforms-conductor/compare/v33.2.1...v33.2.2) (2021-06-21 12:41:35)
 
 ## Global changes

data/README.md

@@ -304,12 +304,39 @@ See [the executables list](docs/executables.md) for more details.
 # Credentials
 
 Some tools or tests require authentication using user/password to an external resource. Examples of such tools are Bitbucket, Thycotic, Confluence...
-Credentials can be given using either environment variables or by parsing the user's `.netrc` file.
+Credentials can be given 3 different ways:
+* from the configuration file `hpc_config.rb`,
+* from environment variables,
+* from the user's `.netrc` file.
 
-In case a process needs a credential that has not been set, a warning message will be output so that the user knows which credential is missing, and eventually for which URL.
+In case a process needs a credential that has not been set, a warning message will be output so that the user knows which credential is missing, and eventually for which resource (URL, file...).
 
 Following sub-sections explain the different ways of setting such credentials.
 
+## Configuration
+
+The [`credentials_for` config DSL method](docs/config_dsl.md#credentials_for) can be used to define the credentials for a given credential ID and eventual resource.
+The way to do it is to provide a callback that will be called only when a credential is needed, and the credentials are given to a requester object. This way the life-cycle of the secret can be completely controlled, and clean-up can be done to ensure no vulnerabilities are staying after usage.
+
+Example:
+```ruby
+# Simple case
+credentials_for(:bitbucket) do |resource, requester|
+  requester.call 'my_bitbucket_name', 'my_bitbucket_PaSsWoRd'
+end
+
+# More secure case, handling user input and memory clean-up after usage
+credentials_for(:bitbucket) do |resource, requester|
+  puts 'Input Bitbucket password...'
+  password = ''
+  $stdin.noecho { |io| io.sysread(256, password) }
+  password.chomp!
+  SecretString.protect(password) do |secret_password|
+    requester.call 'my_bitbucket_name', secret_password
+  end
+end
+```
+
 ## Environment variables
 
 Environment variables used for credentials are always named following this convention: `hpc_user_for_<credential_id>` and `hpc_password_for_<credential_id>`.

data/docs/config_dsl.md

@@ -13,6 +13,7 @@ This DSL can also be completed by plugins. Check [the plugins documentations](pl
   * [`hybrid_platforms_dir`](#hybrid_platforms_dir)
   * [`tests_provisioner`](#tests_provisioner)
   * [`expect_tests_to_fail`](#expect_tests_to_fail)
+  * [`credentials_for`](#credentials_for)
   * [`read_secrets_from`](#read_secrets_from)
   * [`send_logs_to`](#send_logs_to)
   * [`retry_deploy_for_errors_on_stdout`](#retry_deploy_for_errors_on_stdout)
@@ -202,6 +203,50 @@ for_nodes('/tst/') do
 end
 ```
 
+<a name="credentials_for"></a>
+## `credentials_for`
+
+Set the credentials to be used (user, password) for a given credential ID and an optional resources selection.
+
+Credentials can be used by any plugin (using the [`with_credentials_for`](/lib/hybrid_platforms_conductor/credentials.rb) method) and are used by various processes in Hybrid Platforms Conductor. See each [plugin's documentation](plugins.md) to know which plugin uses which credential.
+
+It takes the following parameters:
+* the credential ID (as a Symbol),
+* an optional resource specification (can be a String for a complete resource name, or a Regexp matching resources' names),
+* a code block that will be called back by any process needing credentials matching the ID and resource specification.
+The code block will be given both the resource name being accessed, and a requester object that needs to be given the corresponding user and password. When the requester finishes running, the credentials are not needed anymore and should be cleaned from memory to avoid vulnerabilities.
+
+A secure way to five the password is to use a [`SecretString`](https://github.com/Muriel-Salvan/secret_string) class that will give the following guarantees:
+* Avoid the password to leak inadvertently in logs, on-screen, files...
+* Clear the password from memory as soon as it is not needed anymore.
+
+Examples:
+```ruby
+# Using an environment variable as a password
+credentials_for(:github) do |resource, requester|
+  requester.call 'MyUserName', ENV['MY_GITHUB_PASSWORD']
+end
+
+# Using user input and cleaning memory
+credentials_for(:github) do |resource, requester|
+  puts 'Input Github password...'
+  password = ''
+  $stdin.noecho { |io| io.sysread(256, password) }
+  password.chomp!
+  SecretString.protect(password) do |secret_password|
+    requester.call 'MyUserName', secret_password
+  end
+end
+
+# Defining different credentials based on the resource being accessed
+credentials_for(:github, resource: %r{github.com/my-projects}) do |resource, requester|
+  requester.call 'MyUserName', 'MyPassword'
+end
+credentials_for(:github, resource: %r{github.com/company}) do |resource, requester|
+  requester.call 'CompanyUserName', 'CompanyPassword'
+end
+```
+
 <a name="read_secrets_from"></a>
 ## `read_secrets_from`
 

data/docs/plugins.md

@@ -196,6 +196,7 @@ Check the [sample plugin file](../lib/hybrid_platforms_conductor/hpc_plugins/sec
 
 Plugins shipped by default:
 * [`cli`](plugins/secrets_reader/cli.md)
+* [`keepass`](plugins/secrets_reader/keepass.md)
 * [`thycotic`](plugins/secrets_reader/thycotic.md)
 
 <a name="test"></a>

data/docs/plugins/secrets_reader/keepass.md

@@ -0,0 +1,62 @@
+# Secrets reader plugin: `keepass`
+
+The `keepass` secrets reader plugin retrieves secrets from [KeePass](https://keepass.info/) databases, using an actual KeePass installation with the [KPScript plugin](https://keepass.info/plugins.html#kpscript).
+
+It is configured by giving the KPScript command-line (using `use_kpscript_from` config DSL method), the KeePass databases to be read (using `secrets_from_keepass` config DSL method) and uses the `keepass` credential ID to authenticate along with extra environment variables for eventual key files or encrypted passwords.
+
+## Config DSL extension
+
+### `use_kpscript_from`
+
+Provide the KPScript command-line to be used. If KPScript is already in your path, using `KPScript.exe` or `kpscript` should be enough, otherwise the full path to the command-line will be needed. On Windows it is needed to also include double quotes if the path contains spaces (like `"C:\Program Files\KeePass\KPScript.exe"`).
+
+It takes a simple `String` as parameter to get the command line.
+
+Example:
+```ruby
+use_kpscript_from '/path/to/kpscript'
+```
+
+### `secrets_from_keepass`
+
+Define a KeePass database to read secrets from.
+A base group path of the KeePass database can also be specified to only read secrets from this group path.
+
+All entries, attachments and sub-groups from the base group will be read as secrets.
+
+Can be applied to subset of nodes using the [`for_nodes` DSL method](/docs/config_dsl.md#for_nodes).
+
+It takes the following parameters:
+* **database** (`String`): Database file path.
+* **group_path** (`Array<String>`): Group path to extract from [default: `[]`].
+
+Example:
+```ruby
+secrets_from_keepass(
+  database: '/path/to/database.kdbx',
+  group_path: %w[Secrets Automation]
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `keepass` | Used to get the password to the database. No need to be set if the database opens without password. |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+| `hpc_key_file_for_keepass` | Optional path to the key file needed to open the database |
+| `hpc_password_enc_for_keepass` | Optional encrypted password needed to open the database |
+
+## External tools dependencies
+
+* [KeePass](https://keepass.info/) to open databases.
+* [KPScript KeePass plugin](https://keepass.info/plugins.html#kpscript) to query KeePass API.

data/lib/hybrid_platforms_conductor/bitbucket.rb

@@ -7,116 +7,160 @@ require 'hybrid_platforms_conductor/logger_helpers'
 
 module HybridPlatformsConductor
 
-  # Object used to access Bitbucket API
-  class Bitbucket
+  # Mixin used to access Bitbucket API
+  module Bitbucket
 
-    include LoggerHelpers
+    include Credentials
 
     # Provide a Bitbucket connector, and make sure the password is being cleaned when exiting.
     #
     # Parameters::
     # * *bitbucket_url* (String): The Bitbucket URL
-    # * *logger* (Logger): Logger to be used
-    # * *logger_stderr* (Logger): Logger to be used for stderr
     # * Proc: Code called with the Bitbucket instance.
-    #   * *bitbucket* (Bitbucket): The Bitbucket instance to use.
-    def self.with_bitbucket(bitbucket_url, logger, logger_stderr)
-      Credentials.with_credentials_for(:bitbucket, logger, logger_stderr, url: bitbucket_url) do |bitbucket_user, bitbucket_password|
-        yield Bitbucket.new(bitbucket_url, bitbucket_user, bitbucket_password, logger: logger, logger_stderr: logger_stderr)
+    #   * *bitbucket* (BitbucketApi): The Bitbucket instance to use.
+    def with_bitbucket(bitbucket_url)
+      with_credentials_for(:bitbucket, resource: bitbucket_url) do |bitbucket_user, bitbucket_password|
+        yield BitbucketApi.new(bitbucket_url, bitbucket_user, bitbucket_password, logger: @logger, logger_stderr: @logger_stderr)
       end
     end
 
-    # The Bitbucket URL
-    # String
-    attr_reader :bitbucket_url
-
-    # Constructor
+    # Iterate over each Bitbucket repository
     #
     # Parameters::
-    # * *bitbucket_url* (String): The Bitbucket URL
-    # * *bitbucket_user_name* (String): Bitbucket user name to be used when querying the API
-    # * *bitbucket_password* (String): Bitbucket password to be used when querying the API
-    # * *logger* (Logger): Logger to be used [default = Logger.new(STDOUT)]
-    # * *logger_stderr* (Logger): Logger to be used for stderr [default = Logger.new(STDERR)]
-    def initialize(bitbucket_url, bitbucket_user_name, bitbucket_password, logger: Logger.new($stdout), logger_stderr: Logger.new($stderr))
-      init_loggers(logger, logger_stderr)
-      @bitbucket_url = bitbucket_url
-      @bitbucket_user_name = bitbucket_user_name
-      @bitbucket_password = bitbucket_password
+    # * Proc: Code called for each Bitbucket repository:
+    #   * Parameters::
+    #     * *bitbucket* (Bitbucket): The Bitbucket instance used to query the API for this repository
+    #     * *repo_info* (Hash<Symbol, Object>): The repository info:
+    #       * *name* (String): Repository name.
+    #       * *project* (String): Project name.
+    #       * *url* (String): Project Git URL.
+    #       * *jenkins_ci_url* (String or nil): Corresponding Jenkins CI URL, or nil if none.
+    #       * *checks* (Hash<Symbol, Object>): Checks to be performed on this repository:
+    #         * *branch_permissions* (Array< Hash<Symbol, Object> >): List of branch permissions to check [optional]
+    #           * *type* (String): Type of branch permissions to check. Examples of values are 'fast-forward-only', 'no-deletes', 'pull-request-only'.
+    #           * *branch* (String): Branch on which those permissions apply.
+    #           * *exempted_users* (Array<String>): List of exempted users for this permission [default: []]
+    #           * *exempted_groups* (Array<String>): List of exempted groups for this permission [default: []]
+    #           * *exempted_keys* (Array<String>): List of exempted access keys for this permission [default: []]
+    #         * *pr_settings* (Hash<Symbol, Object>): PR specific settings to check [optional]
+    #           * *required_approvers* (Integer): Number of required approvers [optional]
+    #           * *required_builds* (Integer): Number of required successful builds [optional]
+    #           * *default_merge_strategy* (String): Name of the default merge strategy. Example: 'rebase-no-ff' [optional]
+    #           * *mandatory_default_reviewers* (Array<String>): List of mandatory reviewers to check [default: []]
+    def for_each_bitbucket_repo
+      @config.known_bitbucket_repos.each do |bitbucket_repo_info|
+        with_bitbucket(bitbucket_repo_info[:url]) do |bitbucket|
+          (bitbucket_repo_info[:repos] == :all ? bitbucket.repos(bitbucket_repo_info[:project])['values'].map { |repo_info| repo_info['slug'] } : bitbucket_repo_info[:repos]).each do |name|
+            yield bitbucket, {
+              name: name,
+              project: bitbucket_repo_info[:project],
+              url: "#{bitbucket_repo_info[:url]}/scm/#{bitbucket_repo_info[:project].downcase}/#{name}.git",
+              jenkins_ci_url: bitbucket_repo_info[:jenkins_ci_url].nil? ? nil : "#{bitbucket_repo_info[:jenkins_ci_url]}/job/#{name}",
+              checks: bitbucket_repo_info[:checks]
+            }
+          end
+        end
+      end
     end
 
-    # Get the repositories of a given project.
-    # Limit to 1000 results max.
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # Result::
-    # * Object: Corresponding JSON
-    def repos(project)
-      get_api("projects/#{project}/repos?limit=1000")
-    end
+    # Provide an API to Bitbucket
+    class BitbucketApi
 
-    # Get the PR settings of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def settings_pr(project, repo)
-      get_api("projects/#{project}/repos/#{repo}/settings/pull-requests")
-    end
+      include LoggerHelpers
 
-    # Get the default reviewers of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def default_reviewers(project, repo)
-      get_api("projects/#{project}/repos/#{repo}/conditions", api_domain: 'default-reviewers')
-    end
+      # The Bitbucket URL
+      # String
+      attr_reader :bitbucket_url
 
-    # Get the branch permissions of a given repository
-    #
-    # Parameters::
-    # * *project* (String): Project name
-    # * *repo* (String): Repository name
-    # Result::
-    # * Object: Corresponding JSON
-    def branch_permissions(project, repo)
-      # Put 3 retries here as the Bitbucket installation has a very unstable API 2.0 and often returns random 401 errors.
-      get_api("projects/#{project}/repos/#{repo}/restrictions", api_domain: 'branch-permissions', api_version: '2.0', retries: 3)
-    end
+      # Constructor
+      #
+      # Parameters::
+      # * *bitbucket_url* (String): The Bitbucket URL
+      # * *bitbucket_user_name* (String): Bitbucket user name to be used when querying the API
+      # * *bitbucket_password* (SecretString): Bitbucket password to be used when querying the API
+      # * *logger* (Logger): Logger to be used [default = Logger.new(STDOUT)]
+      # * *logger_stderr* (Logger): Logger to be used for stderr [default = Logger.new(STDERR)]
+      def initialize(bitbucket_url, bitbucket_user_name, bitbucket_password, logger: Logger.new($stdout), logger_stderr: Logger.new($stderr))
+        init_loggers(logger, logger_stderr)
+        @bitbucket_url = bitbucket_url
+        @bitbucket_user_name = bitbucket_user_name
+        @bitbucket_password = bitbucket_password
+      end
 
-    # Issue an HTTP get on the API.
-    # Handle authentication.
-    #
-    # Parameters::
-    # * *path* (String): API path to access
-    # * *api_domain* (String): API domain to access [default: 'api']
-    # * *api_version* (String): API version to access [default: '1.0']
-    # * *retries* (Integer): Number of retries in case of failures [default: 0]
-    # Result::
-    # * Object: Returned JSON
-    def get_api(path, api_domain: 'api', api_version: '1.0', retries: 0)
-      api_url = "#{@bitbucket_url}/rest/#{api_domain}/#{api_version}/#{path}"
-      log_debug "Call Bitbucket API #{@bitbucket_user_name}@#{api_url}..."
-      http_response = nil
-      loop do
-        begin
-          http_response = URI.parse(api_url).open(http_basic_authentication: [@bitbucket_user_name, @bitbucket_password])
-        rescue
-          raise if retries.zero?
-
-          log_warn "Got error #{$ERROR_INFO} on #{@bitbucket_user_name}@#{api_url}. Will retry #{retries} times..."
-          retries -= 1
-          sleep 1
+      # Get the repositories of a given project.
+      # Limit to 1000 results max.
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # Result::
+      # * Object: Corresponding JSON
+      def repos(project)
+        get_api("projects/#{project}/repos?limit=1000")
+      end
+
+      # Get the PR settings of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def settings_pr(project, repo)
+        get_api("projects/#{project}/repos/#{repo}/settings/pull-requests")
+      end
+
+      # Get the default reviewers of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def default_reviewers(project, repo)
+        get_api("projects/#{project}/repos/#{repo}/conditions", api_domain: 'default-reviewers')
+      end
+
+      # Get the branch permissions of a given repository
+      #
+      # Parameters::
+      # * *project* (String): Project name
+      # * *repo* (String): Repository name
+      # Result::
+      # * Object: Corresponding JSON
+      def branch_permissions(project, repo)
+        # Put 3 retries here as the Bitbucket installation has a very unstable API 2.0 and often returns random 401 errors.
+        get_api("projects/#{project}/repos/#{repo}/restrictions", api_domain: 'branch-permissions', api_version: '2.0', retries: 3)
+      end
+
+      # Issue an HTTP get on the API.
+      # Handle authentication.
+      #
+      # Parameters::
+      # * *path* (String): API path to access
+      # * *api_domain* (String): API domain to access [default: 'api']
+      # * *api_version* (String): API version to access [default: '1.0']
+      # * *retries* (Integer): Number of retries in case of failures [default: 0]
+      # Result::
+      # * Object: Returned JSON
+      def get_api(path, api_domain: 'api', api_version: '1.0', retries: 0)
+        api_url = "#{@bitbucket_url}/rest/#{api_domain}/#{api_version}/#{path}"
+        log_debug "Call Bitbucket API #{@bitbucket_user_name}@#{api_url}..."
+        http_response = nil
+        loop do
+          begin
+            http_response = URI.parse(api_url).open(http_basic_authentication: [@bitbucket_user_name, @bitbucket_password&.to_unprotected])
+          rescue
+            raise if retries.zero?
+
+            log_warn "Got error #{$ERROR_INFO} on #{@bitbucket_user_name}@#{api_url}. Will retry #{retries} times..."
+            retries -= 1
+            sleep 1
+          end
+          break unless http_response.nil?
         end
-        break unless http_response.nil?
+        JSON.parse(http_response.read)
       end
-      JSON.parse(http_response.read)
+
     end
 
   end