hybrid_platforms_conductor 32.11.2 → 32.13.3

data/docs/executables/test.md

@@ -0,0 +1,213 @@
+# `test`
+
+The `test` executable runs various tests and displays the eventual errors that have occurred.
+Errors are being displayed at the end of the execution, along with a summary of the failed tests and nodes.
+
+This `test` executable is using [test plugins](../plugins.md#test) to be able to validate various tests (at global level, on each node, or on the check-node output), and uses [test report plugins](../plugins.md#test_report) to publish test reports on various mediums.
+
+This executable is perfectly suited to be integrated in a continuous integration workflow.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/test.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/test --all
+participant CMDB as CMDB
+participant PlatformHandler as Platform Handler
+participant Connector as Connector
+participant Node as Node
+participant GlobalTest as Global Test
+participant PlatformTest as Platform Test
+participant NodeTest as Node Test
+participant CheckNodeTest as Deployment Test
+participant TestReport as Test Report
+
+Main->>+CMDB: Get the list of platforms, services and nodes
+CMDB-->>-Main: List of platforms, services and nodes
+Main->>+PlatformHandler: Get actions to check services on nodes
+PlatformHandler-->>-Main: Actions to check services on nodes
+Main->>+NodeTest: Get actions to be executed on each node
+NodeTest-->>-Main: Test actions to be executed
+Main->>+Connector: Connect to nodes to execute node and deployment tests
+Connector->>+Node: Execute node test actions
+Node-->>-Connector: Node test logs
+Connector->>+Node: Execute deployment check actions
+Node-->>-Connector: Deployment check logs
+Connector-->>-Main: Node and deployment test logs
+Main->>+GlobalTest: Run each global test
+GlobalTest-->>-Main: Global tests results
+Main->>+PlatformTest: Run each platform test per platform
+PlatformTest-->>-Main: Platform tests results
+Main->>+NodeTest: Validate node tests based on test logs
+NodeTest-->>-Main: Node tests results
+Main->>+CheckNodeTest: Validate deployment tests based on deployment check logs
+CheckNodeTest-->>-Main: Deployment tests results
+Main->>+TestReport: Send all tests results to the Test Report
+TestReport-->>-Main: Test report published
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/test [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+        --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+    -j, --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+
+Deployer options:
+    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
+                                     * Local path to a JSON file
+                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
+        --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+Tests runner options:
+    -i, --tests-list FILE_NAME       Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
+    -k, --skip-run                   Skip running the check-node commands for real, and just analyze existing run logs.
+    -r, --report REPORT              Specify a report name. Can be used several times. Can be all for all reports. Possible values: confluence, stdout (defaults to stdout).
+    -t, --test TEST                  Specify a test name. Can be used several times. Can be all for all tests. Possible values: ansible_repo_molecule_cdh_admins, ansible_repo_molecule_cdh_datanodes, ansible_repo_molecule_cdh_db, ansible_repo_molecule_cdh_gateways, ansible_repo_molecule_cdh_services, ansible_repo_molecule_common, ansible_repo_molecule_data_gateway, ansible_repo_molecule_dev_servers, ansible_repo_molecule_ds_servers, ansible_repo_molecule_dsnodes, ansible_repo_molecule_import_gateway, ansible_repo_molecule_notebooks, ansible_repo_molecule_tnz_data_gateway, bitbucket_conf, can_be_checked, check_from_scratch, chef_executables, chef_success, chef_woulds, connection, deploy_freshness, deploy_from_scratch, deploy_removes_root_access, executables, food_critic, group_ids, hostname, idempotence, ip, jenkins_ci_conf, jenkins_ci_masters_ok, linear_strategy, obsolete_home_dirs, obsolete_users, orphan_files, private_ips, public_ips, rubocop, spectre, unused_files, unused_node_attributes, unused_recipes, unused_templates, unused_roles, unused_users, user_ids, users_without_roles, veids (defaults to all).
+        --max-threads-connections NBR_THREADS
+                                     Specify the max number of threads to parallelize tests connecting on nodes (defaults to 64).
+        --max-threads-nodes NBR_THREADS
+                                     Specify the max number of threads to parallelize tests at node level (defaults to 8).
+        --max-threads-platforms NBR_THREADS
+                                     Specify the max number of threads to parallelize tests at platform level (defaults to 8).
+```
+
+## Examples
+
+```bash
+# Execute all tests on all nodes
+./bin/test --all-nodes
+
+# Execute only the tests named hostname and ip on all nodes whose names contain xae
+./bin/test --test hostname --test ip --node /xae/
+
+# Execute all tests on all nodes, but reuse the content of run_logs instead of why-run deployments
+./bin/test --all-nodes --skip-run
+
+# Execute the check_from_scratch test on all nodes impacted by changes made on the branch my_branch
+./bin/test --test check_from_scratch --nodes-git-impact chef-repo::my_branch
+```
+
+Here is an example of output:
+```
+========== Error report of 6 tests run on 694 nodes
+
+======= By test:
+
+===== configuration_test found 2 nodes having errors:
+  * [ nodehst-nn3 ] - 3 errors:
+    - Failed to execute command "hostname -I"
+    - Failed to execute command "hostname -s"
+    - Failed to execute command "echo 'Test connection - ok'"
+  * [ project-pinger ] - 1 errors:
+    - Private IP outside
+
+
+======= By node:
+
+===== [ node45 ] - 1 failing tests:
+  * Test configuration_test - 3 errors:
+    - Failed to execute command "hostname -I"
+    - Failed to execute command "hostname -s"
+    - Failed to execute command "echo 'Test connection - ok'"
+
+===== [ node12had41 ] - 1 failing tests:
+  * Test configuration_test - 1 errors:
+    - Failed to connect
+
+===== [ node237 ] - 1 failing tests:
+  * Test configuration_test - 1 errors:
+    - Not handled by Chef
+
+===== [ project-pinger ] - 1 failing tests:
+  * Test configuration_test - 1 errors:
+    - Private IP outside
+
+
+========== Stats by hosts list:
+
++--------------------+----------+-----------+
+| List name          | % tested | % success |
++--------------------+----------+-----------+
+| hosts_with_secrets | 100 %    | 71 %      |
+| node12had          | 100 %    | 1 %       |
+| xaebhs5had         | 100 %    | 90 %      |
+| xaebhsone          | 100 %    | 0 %       |
+| xaerbx5had         | 100 %    | 0 %       |
+| xaerbxcas          | 100 %    | 0 %       |
+| xaerbxhad          | 100 %    | 0 %       |
+| xaesbg1cas         | 100 %    | 66 %      |
+| xaesbg1had         | 100 %    | 0 %       |
+| xaesbg2had         | 100 %    | 0 %       |
+| xaesbghad          | 100 %    | 0 %       |
+| xaesbgkfk          | 100 %    | 100 %     |
+| xaesbgzk           | 100 %    | 100 %     |
+| xaetirb1pdnc       | 100 %    | 0 %       |
+| xaetirb6tdnc       | 100 %    | 0 %       |
+| xaetisb3sdnc       | 100 %    | 0 %       |
+| No list            | 100 %    | 18 %      |
++--------------------+----------+-----------+
+
+===== Some errors were found. Check output. =====
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `host_ip` | `String` | IP address on which a node can be reachable |
+| `image` | `String` | OS image to be used for tests |
+| `services` | `Array<String>` | List of services to check for in tests |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/topograph.md

@@ -0,0 +1,139 @@
+# `topograph`
+
+The `topograph` executable will dump the topology graph of a set of nodes.
+This is useful to have a visualization of the network of nodes and their relations.
+It dumps all the links and groups between a source set of nodes to a destination set of nodes, recursively (the sets can be "all nodes" too).
+It uses the nodes' metadata, as well as the complete nodes JSON dumped by the `dump_nodes_json` executable to get links between nodes.
+
+Prerequisites before running `topograph`:
+* If the `svg` output format is used, then the `dot` utility should be installed in the system.
+
+***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
+
+## Process
+
+TODO
+
+## Usage
+
+```
+Usage: ./bin/topograph [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+        --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+
+Deployer options:
+    -e, --secrets SECRETS_LOCATION   Specify a secrets location. Can be specified several times. Location can be:
+                                     * Local path to a JSON file
+                                     * URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
+    -t, --timeout SECS               Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
+        --retries-on-error NBR       Number of retries in case of non-deterministic errors (defaults to 0)
+
+JSON dump options:
+    -j, --json-dir DIRECTORY         Specify the output directory in which JSON files are being written. Defaults to nodes_json.
+
+Topographer options:
+    -F, --from HOSTS_OPTIONS         Specify options for the set of nodes to start from (enclose them with ""). Default: all nodes. HOSTS_OPTIONS follows the following:
+                                         -a, --all-nodes                  Select all nodes
+                                         -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+                                         -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+                                         -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+                                         -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+                                             --nodes-git-impact GIT_IMPACT
+                                                                          Select nodes impacted by a git diff from a platform (can be used several times).
+                                                                          GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                                                          * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                                                          * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                                                          * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                                                          * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                                                            - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+    -k, --skip-run                   Skip the actual gathering of JSON node files. If set, the current files in nodes_json will be used.
+    -p, --output FORMAT:FILE_NAME    Specify a format and file name. Can be used several times. FORMAT can be one of graphviz, json, svg. Ex.: graphviz:graph.gv
+    -T, --to HOSTS_OPTIONS           Specify options for the set of nodes to get to (enclose them with ""). Default: all nodes. HOSTS_OPTIONS follows the following:
+                                         -a, --all-nodes                  Select all nodes
+                                         -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+                                         -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+                                         -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+                                         -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+                                             --nodes-git-impact GIT_IMPACT
+                                                                          Select nodes impacted by a git diff from a platform (can be used several times).
+                                                                          GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                                                          * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                                                          * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                                                          * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                                                          * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                                                            - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+```
+
+## Examples
+
+```bash
+# Dump the whole network in JSON format
+./bin/topograph --output json:graph.json
+
+# Dump the whole network in JSON and SVG format
+./bin/topograph --output json:graph.json --output svg:graph.svg
+
+# Dump the network starting from any node belonging to the node12had hosts list
+./bin/topograph --output json:graph.json --from "--nodes-list node12had"
+
+# Dump the network getting to nodes xaeprjcttlbd01 and xaeprjctplbd01
+./bin/topograph --output json:graph.json --to "--node xaeprjcttlbd01 --node xaeprjctplbd01"
+
+# Dump the network getting from any node belonging to the node12had hosts list and to nodes xaeprjcttlbd01 and xaeprjctplbd01
+./bin/topograph --output json:graph.json --from "--nodes-list node12had" --to "--node xaeprjcttlbd01 --node xaeprjctplbd01"
+
+# Dump the whole network in JSON format, reusing existing JSON files from nodes_json (won't call dump_nodes_json)
+./bin/topograph --output json:graph.json --skip-run
+```
+
+Example of output:
+```
+=> ./bin/topograph --skip-run --output graphviz:graph.gv
+===== Compute graph...
+!!! Missing JSON file nodes_json/node12hst-nn2.json
+===== Add hosts lists clusters...
+===== Define IP 24 clusters...
+===== Select path...
+===== Filter only nodes 172.16.0.0/12, 172.16.0.0/24, 172.16.1.0/24, 172.16.10.0/24, 172.16.110.0/24, xaetisb3sdnc21, xaetisb3sdnc22, xaetisb3sdnc23, xaetisb3sdnc24, xaetisb3sdnc25, xaetisb3sdnc3, xaetisb3sdnc4, xaetisb3sdnc5, xaetisb3sdnc6, xaetisb3sdnc7, xaetisb3sdnc8, xaetisb3sdnc9, xaetisb3sgwc01, xaetisb3snnc01, xaetisb3snnc02, xaetisbgpnsd01, xaetisqlpwbd01, xaetisqlcid01, xaetitanpwsd01, xaetitanuwsd01...
+===== Collapse hosts lists...
+===== Remove self references...
+===== Remove empty clusters...
+===== Write outputs...
+===== Write graphviz file graph.gv...
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/install.md

@@ -0,0 +1,161 @@
+# Requirements
+
+For a bare usage (no plugins), the only requirement of Hybrid Platforms Conductor is **Ruby**.
+
+Then depending on the plugins being used, external tools might need to be installed (see below).
+Commands in this documentation are taken from a Debian-based environment, but they can be easily translated into other Linuxes.
+
+## Install Ruby
+
+Here are some ways to install it.
+
+### Compiling it from scratch.
+
+```bash
+mkdir ruby
+cd ruby
+wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.3.tar.gz
+tar xvzf ruby-2.7.3.tar.gz
+cd ruby-2.7.3
+sudo apt install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev libgdbm-compat-dev bison
+./configure
+make
+sudo make install
+cd ../..
+```
+
+### Using RVM
+
+```bash
+sudo apt-get install dirmngr curl
+gpg --keyserver hkp://keys.gnupg.net:80 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
+curl -sSL https://get.rvm.io | bash -s stable
+rvm install 2.5.0
+rvm use 2.5.0
+```
+
+### Using Debian (>= Stretch) package manager
+
+```bash
+sudo apt-get install ruby-dev libffi-dev zlib1g-dev
+```
+
+## Install the `hybrid_platforms_conductor` rubygem
+
+This can be done either in your Ruby system directories, or defined as a dependency of a Ruby project
+
+### As a system-wide tool
+
+```bash
+sudo gem install hybrid_platforms_conductor
+```
+
+Then the tools can be used directly from the terminal (they should be part of the PATH).
+
+### As a Ruby project
+
+This needs `bundler` to be installed as well (see below).
+
+1. In a new directory, create a file named `Gemfile`, and declare the dependency on the `hybrid_platforms_conductor` rubygem:
+
+`Gemfile` content:
+```ruby
+source 'http://rubygems.org'
+
+# Orchestrate all the platforms with Hybrid Platforms Conductor
+gem 'hybrid_platforms_conductor'
+```
+
+2. Install the dependencies of your Ruby project
+
+```bash
+bundle config set --local path vendor/bundle
+bundle install
+bundle binstubs hybrid_platforms_conductor
+```
+
+Then the tools can be used directly from the Ruby project directory, inside the `./bin` folder.
+
+## Create the Hybrid Platforms Conductor main configuration file
+
+As a minimum requirement, the current directory from which the tools are being called should have a file named `hpc_config.rb`.
+The file can be empty, and directives can be used to define the various platforms and configuration parameters.
+
+## Check installation
+
+A correct Hybrid Platforms Conductor installation can be checked by running the `run --help` command.
+
+The output should look like this:
+
+```
+Usage: run [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+    -c, --command CMD                Command to execute (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
+    -f, --commands-file FILE_NAME    Execute commands taken from a file (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
+    -i, --interactive                Run an interactive SSH session instead of executing a command (can't be used with --command or --commands-file)
+    -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
+    -t, --timeout SECS               Timeout in seconds to wait for each command (defaults to no timeout)
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+    -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are:
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Actions Executor options:
+    -m, --max-threads NBR            Set the number of threads to use for concurrent queries (defaults to 16)
+```
+
+### For a system-wide installation
+
+```bash
+run --help
+```
+
+### For a Ruby project installation
+
+```bash
+./bin/run --help
+```
+
+## Other dependencies
+
+The following dependencies are not needed for a minimum installation, but are required for some of the plugins provided by default.
+
+## Git
+
+```bash
+sudo apt install git
+git config --global user.email "<your_email>"
+git config --global user.name "<your_user_name>"
+```
+
+## SSH client
+
+```bash
+sudo apt install openssh-client
+```
+
+## Bundler
+
+```bash
+sudo gem install bundler
+```