hybrid_platforms_conductor 32.11.1 → 32.13.2

data/docs/executables/free_ips.md

@@ -0,0 +1,93 @@
+# `free_ips`
+
+The `free_ips` executable will output all free IP ranges for any used range.
+Useful to assign new IPs.
+
+***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/free_ips.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/free_ips
+participant CMDB as CMDB
+
+Main->>+CMDB: Get private IP metadata for nodes
+CMDB-->>-Main: Private IPs
+Main->>Main: Display free private IPs
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/free_ips [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+```
+
+## Examples
+
+```bash
+./bin/free_ips
+```
+
+Example of output:
+```
+=> ./bin/free_ips
+Free IPs for 172.16.0: [11, 20, 23..29, 31, 34, 37..40, 42..45, 48, 51..58, 60, 63..72, 76, 79..80, 82..83, 87, 90..95, 97..101, 103..104, 107..109, 111..113, 115, 117, 119..120, 123..124, 127, 129, 131, 139, 142..149, 153..169, 171..180, 182..189, 191, 193..209, 211..221, 223..251, 253..255]
+Free IPs for 172.16.1: [21..100, 102, 106, 108, 110..112, 114, 116..119, 121..124, 126, 132..177, 179..200, 205..209, 211..221, 223..252, 254..255]
+Free IPs for 172.16.2: [102, 110..111, 113, 120..121, 127, 133, 142..150, 194..200, 204..209, 214..221, 223..251, 253..255]
+Free IPs for 172.16.3: [11..99, 102, 106..199, 201..255]
+Free IPs for 172.16.4: [12..100, 102, 106, 109, 112..255]
+Free IPs for 172.16.5: [102, 105..255]
+Free IPs for 172.16.6: [41..64, 85..102, 104..109, 116..128, 134..255]
+Free IPs for 172.16.7: [28..102, 104..106, 113..255]
+Free IPs for 172.16.8: [18..100, 102, 104, 106, 113..119, 121..122, 124..159, 166..170, 196..200, 204..255]
+Free IPs for 172.16.9: [86..102, 104..106, 113..122, 125, 128..255]
+Free IPs for 172.16.10: [5..104, 106..255]
+Free IPs for 172.16.16: [8..19, 23..25, 28, 30, 32..33, 35..36, 39, 41..42, 44..47, 49..51, 54..62, 64..69, 74, 78, 81, 83, 93, 96..98, 104, 110, 114..116, 118, 125..128, 131..136, 138..152, 156..169, 171..177, 179, 181, 183..185, 187, 189, 191, 193..209, 212..255]
+Free IPs for 172.16.110: [43..54, 56..57, 60..82, 84..105, 108..118, 120..203, 208..255]
+Free IPs for 172.16.111: [22..23, 25..26, 28..30, 32..46, 48..49, 54..255]
+Free IPs for 172.16.132: [16..47, 49, 51..181, 183..187, 189..216, 218..225, 227..229, 231..252, 255]
+Free IPs for 172.16.133: [18..50, 52, 54..95, 97, 99..120, 122..123, 125..128, 130..134, 136..155, 157..163, 166..252, 254..255]
+Free IPs for 172.16.134: [2, 4..54, 56..65, 67..210, 212..222, 224..228, 231..255]
+Free IPs for 172.16.135: [61, 63, 65, 67, 72..89, 93, 95..96, 98..104, 107..110, 115..122, 124..126, 131..255]
+Free IPs for 172.16.139: [99..255]
+Free IPs for 172.30.14: [227..255]
+Free IPs for 192.168.0: [3..255]
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `private_ips` | `Array<String>` | List of private IPs |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/free_veids.md

@@ -0,0 +1,73 @@
+# `free_veids`
+
+The `free_veids` executable will output all free VEIDs (smaller than 10000).
+Useful to assign unused VEIDs to new VMs to be created.
+
+***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/free_veids.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/free_veids
+participant CMDB as CMDB
+
+Main->>+CMDB: Get VEID metadata for nodes
+CMDB-->>-Main: VEIDs
+Main->>Main: Display free VEIDs
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/free_veids [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+```
+
+## Examples
+
+```bash
+./bin/free_veids
+```
+
+Example of output:
+```
+=> ./bin/free_veids
+Free VEIDs: [420, 426, 428, 430, 434, 437..438, 445..446, 449..450, 453, 456..457, 459, 464, 466..467, 471, 475..476, 484, 488, 490, 493, 500..502, 504..513, 523, 525, 536, 544, 546, 554..555, 560..566, 578, 589, 594, 642..659, 668..9999]
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `veid` | `String` or `Integer` | VEID associated to the node |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/get_impacted_nodes.md

@@ -0,0 +1,94 @@
+# `get_impacted_nodes`
+
+The `get_impacted_nodes` executable reports nodes impacted by a git diff in one of the platforms.
+This is especially useful to know which nodes have to be tested against a given PR or local code difference.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/get_impacted_nodes.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/get_impacted_nodes
+participant PlatformHandler as Platform Handler
+participant PlatformRepo as Platform repository
+
+Main->>+PlatformHandler: Get the list of nodes and services impacted by a git diff
+PlatformHandler->>+PlatformRepo: Perform git diff
+PlatformRepo-->>-PlatformHandler: Git diff details
+PlatformHandler-->>-Main: List of nodes and services impacted by a git diff
+Main->>Main: Display the list of impacted nodes and services
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/get_impacted_nodes [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+    -f, --from-commit COMMIT_ID      Specify the GIT commit from which we look for diffs. Defaults to master.
+    -p, --platform PLATFORM_NAME     Specify the repository on which to perform the diff. Possible values are chef-repo, ansible-repo
+        --smallest-test-sample       Display the minimal set of nodes to check that would validate all modifications.
+    -t, --to-commit COMMIT_ID        Specify the GIT commit to which we look for diffs. Defaults to current checked out files.
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+```
+
+## Examples
+
+```bash
+# Get nodes impacted by current modifications in the chef-repo repository
+./bin/get_impacted_nodes --platform chef-repo
+
+# Get nodes impacted by modifications between 2 commit IDs in the chef-repo repository
+./bin/get_impacted_nodes --platform chef-repo --from-commit 6e45ff --to-commit aac411
+
+# Get the smallest set of nodes to be tested to validate changes from a branch compared to master
+./bin/get_impacted_nodes --platform chef-repo --to-commit my_branch --smallest-test-sample
+```
+
+Here is an example of output:
+```
+
+* 1 impacted services:
+websql
+
+* 2 impacted nodes (directly):
+node12hst-nn6
+node12hst-nn2
+
+* 3 impacted nodes (total):
+node12hst-nn6
+node12hst-nn1
+node12hst-nn2
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/last_deploys.md

@@ -0,0 +1,114 @@
+# `last_deploys`
+
+The `last_deploys` executable will fetch the last deployments information for a given list of nodes and display it.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/last_deploys.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/last_deploys
+participant Connector as Connector
+participant Node as Node
+
+Main->>+Connector: Connect to nodes to read deployment logs
+Connector->>+Node: Read deployment logs
+Node-->>-Connector: Deployment logs
+Connector-->>-Main: Close connection
+Main->>Main: Display deployment info
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/last_deploys [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+        --sort-by SORT               Specify a sort. Possible values are: commit_comment, date, node, repo_name, user. Each value can append _desc to specify a reverse sorting. Defaults to node.
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+    -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Actions Executor options:
+    -m, --max-threads NBR            Set the number of threads to use for concurrent queries (defaults to 64)
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+    -j, --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+```
+
+## Examples
+
+```bash
+# Check deployments for all nodes
+./bin/last_deploys --all-nodes
+
+# Check deployments for all nodes, sorted by date descending
+./bin/last_deploys --all-nodes --sort-by date_desc
+```
+
+Example of output:
+```
++----------------+---------------------+-----------+---------------+------------------+-------------+----------------------------+-------+
+| Hostname       | Date                | Admin     | Git artefact  | Git branch       | Chef commit | Chef comment               | Error |
++----------------+---------------------+-----------+---------------+------------------+-------------+----------------------------+-------+
+| node10         | 2017-11-22 09:50:47 | a_usernme | 172.16.0.46   | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12had43    | 2017-11-22 10:07:37 | a_usernme | 172.16.110.42 | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12hst-nn6  | 2017-11-22 10:07:35 | a_usernme | 172.16.110.42 | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12hst-nn9  | 2017-11-23 18:08:59 | root      | 172.16.110.42 | v20171123T190837 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12hst-nn2  | 2017-11-22 10:07:37 | a_usernme | 172.16.110.42 | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12hst-nn3  | 2017-11-22 10:07:37 | a_usernme | 172.16.110.42 | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| node12lnx10    | 2017-11-22 11:07:33 | a_usernme | 172.16.110.42 | v20171122T110551 | 73c2017a2a8 | Added sorting capabilities |       |
+| xaeprjcttlbd01 | 2017-11-23 18:43:01 | a_usernme | 172.16.0.46   | v20171123T194235 | 73c2017a2a8 | Added sorting capabilities |       |
++----------------+---------------------+-----------+---------------+------------------+-------------+----------------------------+-------+
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/nodes_to_deploy.md

@@ -0,0 +1,139 @@
+# `nodes_to_deploy`
+
+The `nodes_to_deploy` executable is used to know which nodes need to be deployed, considering:
+* the deployment schedule allocated to the nodes,
+* the last level of code that has been deployed on the node.
+The deployment schedule is defined by the configuration DSL, with the [`deployment_schedule` method](../config_dsl.md).
+The list of nodes is given in standard output.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/nodes_to_deploy.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/nodes_to_deploy
+participant CMDB as CMDB
+participant PlatformHandler as Platform Handler
+participant PlatformRepo as Platform repository
+participant Connector as Connector
+participant Node as Node
+
+Main->>+CMDB: Get services to be deployed on nodes
+CMDB->>+PlatformHandler: Get metadata from the platform
+PlatformHandler->>+PlatformRepo: Read platform info
+PlatformRepo-->>-PlatformHandler: Platform info
+PlatformHandler-->>-CMDB: Platform metadata, including services
+CMDB-->>-Main: Services to be deployed
+Main->>+Connector: Connect to nodes to read deployment logs
+Connector->>+Node: Read deployment logs
+Node-->>-Connector: Deployment logs
+Connector-->>-Main: Close connection
+Main->>Main: Display nodes to be deployed that have divergent services
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/nodes_to_deploy [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+        --deployment-time DATETIME   Set the deployment time to be considered while matching the schedules. Defaults to now.
+        --ignore-deployed-info       Ignore the current deployed information.
+        --ignore-schedule            Ignore the deployment schedules.
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+    -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Actions Executor options:
+    -m, --max-threads NBR            Set the number of threads to use for concurrent queries (defaults to 16)
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+    -j, --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+```
+
+## Examples
+
+```bash
+# Get nodes to be deployed now
+./bin/nodes_to_deploy
+
+# Get nodes to be deployed now among a given set of nodes
+./bin/nodes_to_deploy --node /node12.+/
+
+# Get nodes to be deployed now, ignoring their current deployed information
+./bin/nodes_to_deploy --ignore-deployed-info
+
+# Get nodes to be deployed only considering their deployed information
+./bin/nodes_to_deploy --ignore-schedule
+
+# Get nodes that would be deployed on October 1st 2020 at 10:00:00 UTC
+./bin/nodes_to_deploy --deployment-time "2020-10-01 10:00:00"
+```
+
+Example of output:
+```
+===== Nodes to deploy =====
+node12had50
+node12had51
+node12had52
+node12had53
+node12had54
+node12had55
+node12had57
+node12had58
+node12had59
+node12hst-nn5
+node12hst-nn3
+node12lnx18
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None