hybrid_platforms_conductor 32.12.0 → 32.13.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1103 -0
- data/LICENSE.md +31 -0
- data/README.md +395 -0
- data/bin/setup +1 -1
- data/docs/api.md +349 -0
- data/docs/config_dsl.md +315 -0
- data/docs/executables.md +226 -0
- data/docs/executables/check-node.md +155 -0
- data/docs/executables/deploy.md +198 -0
- data/docs/executables/dump_nodes_json.md +110 -0
- data/docs/executables/free_ips.md +93 -0
- data/docs/executables/free_veids.md +73 -0
- data/docs/executables/get_impacted_nodes.md +94 -0
- data/docs/executables/last_deploys.md +114 -0
- data/docs/executables/nodes_to_deploy.md +139 -0
- data/docs/executables/report.md +159 -0
- data/docs/executables/run.md +126 -0
- data/docs/executables/setup.md +92 -0
- data/docs/executables/ssh_config.md +151 -0
- data/docs/executables/test.md +213 -0
- data/docs/executables/topograph.md +139 -0
- data/docs/gen/mermaid/README.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/check-node.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_ips.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_veids.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/get_impacted_nodes.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/last_deploys.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/nodes_to_deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/report.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/run.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/setup.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/ssh_config.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/test.md-0.png +0 -0
- data/docs/install.md +161 -0
- data/docs/plugins.md +215 -0
- data/docs/plugins/action/bash.md +37 -0
- data/docs/plugins/action/interactive.md +37 -0
- data/docs/plugins/action/remote_bash.md +67 -0
- data/docs/plugins/action/ruby.md +69 -0
- data/docs/plugins/action/scp.md +61 -0
- data/docs/plugins/cmdb/config.md +46 -0
- data/docs/plugins/cmdb/host_ip.md +33 -0
- data/docs/plugins/cmdb/host_keys.md +33 -0
- data/docs/plugins/cmdb/platform_handlers.md +33 -0
- data/docs/plugins/connector/local.md +28 -0
- data/docs/plugins/connector/ssh.md +95 -0
- data/docs/plugins/platform_handler/yaml_inventory.md +105 -0
- data/docs/plugins/provisioner/docker.md +27 -0
- data/docs/plugins/provisioner/podman.md +27 -0
- data/docs/plugins/provisioner/proxmox.md +115 -0
- data/docs/plugins/report/confluence.md +49 -0
- data/docs/plugins/report/mediawiki.md +28 -0
- data/docs/plugins/report/stdout.md +32 -0
- data/docs/plugins/test/bitbucket_conf.md +97 -0
- data/docs/plugins/test/can_be_checked.md +27 -0
- data/docs/plugins/test/check_deploy_and_idempotence.md +61 -0
- data/docs/plugins/test/check_from_scratch.md +28 -0
- data/docs/plugins/test/connection.md +27 -0
- data/docs/plugins/test/deploy_freshness.md +27 -0
- data/docs/plugins/test/deploy_from_scratch.md +28 -0
- data/docs/plugins/test/deploy_removes_root_access.md +29 -0
- data/docs/plugins/test/divergence.md +41 -0
- data/docs/plugins/test/executables.md +26 -0
- data/docs/plugins/test/file_system.md +49 -0
- data/docs/plugins/test/file_system_hdfs.md +65 -0
- data/docs/plugins/test/hostname.md +27 -0
- data/docs/plugins/test/idempotence.md +56 -0
- data/docs/plugins/test/ip.md +28 -0
- data/docs/plugins/test/jenkins_ci_conf.md +54 -0
- data/docs/plugins/test/jenkins_ci_masters_ok.md +54 -0
- data/docs/plugins/test/linear_strategy.md +26 -0
- data/docs/plugins/test/local_users.md +48 -0
- data/docs/plugins/test/mounts.md +55 -0
- data/docs/plugins/test/orphan_files.md +38 -0
- data/docs/plugins/test/ports.md +50 -0
- data/docs/plugins/test/private_ips.md +27 -0
- data/docs/plugins/test/public_ips.md +27 -0
- data/docs/plugins/test/spectre.md +26 -0
- data/docs/plugins/test/veids.md +27 -0
- data/docs/plugins/test/vulnerabilities.md +65 -0
- data/docs/plugins/test_report/confluence.md +43 -0
- data/docs/plugins/test_report/stdout.md +26 -0
- data/docs/plugins_create.md +135 -0
- data/docs/tutorial.md +57 -0
- data/docs/tutorial/01_installation.md +129 -0
- data/docs/tutorial/02_first_node.md +466 -0
- data/docs/tutorial/03_scale.md +876 -0
- data/docs/tutorial/04_test.md +965 -0
- data/docs/tutorial/05_extend_with_plugins.md +1132 -0
- data/examples/bare/Gemfile +4 -0
- data/examples/bare/hpc_config.rb +2 -0
- data/examples/localhost/Gemfile +4 -0
- data/examples/localhost/hpc_config.rb +2 -0
- data/examples/localhost/inventory.yaml +4 -0
- data/lib/hybrid_platforms_conductor/actions_executor.rb +1 -0
- data/lib/hybrid_platforms_conductor/deployer.rb +3 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/remote_bash.rb +29 -13
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/scp.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/local.rb +98 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/my_connector.rb.sample +2 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb +7 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/platform_handler_plugin.rb.sample +5 -5
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/yaml_inventory.rb +140 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/provisioner/proxmox.rb +5 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/bitbucket_conf.rb +4 -4
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_freshness.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_removes_root_access.rb +19 -17
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/divergence.rb +3 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/hostname.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/ip.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/local_users.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/mounts.rb +4 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/orphan_files.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/spectre.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/vulnerabilities.rb +8 -7
- data/lib/hybrid_platforms_conductor/hpc_plugins/test_report/confluence.rb +1 -1
- data/lib/hybrid_platforms_conductor/json_dumper.rb +1 -1
- data/lib/hybrid_platforms_conductor/platform_handler.rb +1 -1
- data/lib/hybrid_platforms_conductor/services_handler.rb +18 -16
- data/lib/hybrid_platforms_conductor/tests_runner.rb +0 -1
- data/lib/hybrid_platforms_conductor/topographer.rb +0 -1
- data/lib/hybrid_platforms_conductor/version.rb +1 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/actions/remote_bash_spec.rb +16 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/connectable_nodes_spec.rb +30 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/remote_actions_spec.rb +113 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/cli_options_spec.rb +6 -2
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/global_helpers_spec.rb +38 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/remote_actions_spec.rb +8 -8
- data/spec/hybrid_platforms_conductor_test/docs_spec.rb +10 -0
- data/tools/check_md +89 -0
- data/tools/generate_mermaid +75 -0
- metadata +207 -12
@@ -0,0 +1,155 @@
|
|
1
|
+
# `check-node`
|
2
|
+
|
3
|
+
The `check-node` executable checks the differences of configuration between what is stored in the platform repositories and what is currently configured on a node.
|
4
|
+
It will:
|
5
|
+
1. package the configuration,
|
6
|
+
2. optionally upload the packaged configuration on an artefact repository, or on the node directly (depends on the node's configuration),
|
7
|
+
3. run a deployment in why-run mode (harmless) on the node,
|
8
|
+
4. display the result on screen.
|
9
|
+
|
10
|
+
## Process
|
11
|
+
|
12
|
+
<!-- Mermaid generator - Section start -->
|
13
|
+
![Mermaid diagram](/docs/gen/mermaid/docs/executables/check-node.md-0.png)
|
14
|
+
<details>
|
15
|
+
<summary>See diagram Mermaid code</summary>
|
16
|
+
|
17
|
+
```mermaid
|
18
|
+
sequenceDiagram
|
19
|
+
participant Main as ./bin/check-node --node my_node
|
20
|
+
participant CMDB as CMDB
|
21
|
+
participant PlatformHandler as Platform Handler
|
22
|
+
participant PlatformRepo as Platform repository
|
23
|
+
participant Connector as Connector
|
24
|
+
participant Node as Provisioned node (my_node)
|
25
|
+
|
26
|
+
Main->>+CMDB: Get services to be deployed on my_node
|
27
|
+
CMDB->>+PlatformHandler: Get my_node metadata from the platform
|
28
|
+
PlatformHandler->>+PlatformRepo: Read platform inventory files
|
29
|
+
PlatformRepo-->>-PlatformHandler: Platform inventory
|
30
|
+
PlatformHandler-->>-CMDB: Services metadata containing my_web_app
|
31
|
+
CMDB-->>-Main: my_node has service my_web_app
|
32
|
+
Main->>+PlatformHandler: Get actions to check my_web_app
|
33
|
+
PlatformHandler-->>-Main: Actions to check my_web_app
|
34
|
+
Main->>+Connector: Connect to my_node to execute actions
|
35
|
+
Connector->>+Node: Execute actions to check my_web_app on my_node
|
36
|
+
Node-->>-Connector: Result of my_web_app check on my_node
|
37
|
+
Connector-->>-Main: Close connection
|
38
|
+
```
|
39
|
+
</details>
|
40
|
+
<!-- Mermaid generator - Section end -->
|
41
|
+
|
42
|
+
## Usage
|
43
|
+
|
44
|
+
```
|
45
|
+
Usage: ./bin/check-node [options]
|
46
|
+
|
47
|
+
Main options:
|
48
|
+
-d, --debug Activate debug mode
|
49
|
+
-h, --help Display help and exit
|
50
|
+
-n, --node NODE Run the command on a specific node.
|
51
|
+
|
52
|
+
Nodes handler options:
|
53
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
54
|
+
|
55
|
+
Command runner options:
|
56
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
57
|
+
|
58
|
+
Connector ssh options:
|
59
|
+
-g, --ssh-gateway-user USER Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
|
60
|
+
-j, --ssh-no-control-master If used, don't create SSH control masters for connections.
|
61
|
+
-q, --ssh-no-host-key-checking If used, don't check for SSH host keys.
|
62
|
+
-u, --ssh-user USER Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
|
63
|
+
-w, --password If used, then expect SSH connections to ask for a password.
|
64
|
+
-y GATEWAYS_CONF, Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
|
65
|
+
--ssh-gateways-conf
|
66
|
+
|
67
|
+
Deployer options:
|
68
|
+
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
69
|
+
* Local path to a JSON file
|
70
|
+
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
71
|
+
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
72
|
+
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
73
|
+
```
|
74
|
+
|
75
|
+
## Examples
|
76
|
+
|
77
|
+
```bash
|
78
|
+
# Test on node23hst-nn1
|
79
|
+
./bin/check-node --node node23hst-nn1
|
80
|
+
|
81
|
+
# Test on node23hst-nn1 using a secrets file
|
82
|
+
./bin/check-node --node node23hst-nn1 --secrets passwords.json
|
83
|
+
```
|
84
|
+
|
85
|
+
Example of output:
|
86
|
+
```
|
87
|
+
=> ./bin/check-node --node node12had01
|
88
|
+
Actions Executor configuration used:
|
89
|
+
* User: a_usernme
|
90
|
+
* Dry run: false
|
91
|
+
* Max threads used: 16
|
92
|
+
* Gateways configuration: madrid
|
93
|
+
* Gateway user: ubradm
|
94
|
+
* Debug mode: false
|
95
|
+
|
96
|
+
===== Packaging current repository ===== Begin... =====
|
97
|
+
cd ../chef-repo && rm -rf dist Berksfile.lock && ./bin/thor solo:bundle
|
98
|
+
Resolving cookbook dependencies...
|
99
|
+
Fetching 'project' from source at site-cookbooks/project
|
100
|
+
[...]
|
101
|
+
create data_bag/.gitkeep
|
102
|
+
create .gitignore
|
103
|
+
create .branch
|
104
|
+
create .chef_commit
|
105
|
+
===== Packaging current repository ===== ...End =====
|
106
|
+
|
107
|
+
===== Delivering on artefacts repositories ===== Begin... =====
|
108
|
+
cd ../chef-repo && ./bin/thor solo:deploy -r git@hpc.172.16.110.42:chef-repo/chef-dist.git -y
|
109
|
+
Warning: no 'deploy' tag found
|
110
|
+
Change log for branch v20180326T104601:
|
111
|
+
<empty>
|
112
|
+
Done
|
113
|
+
===== Delivering on artefacts repositories ===== ...End =====
|
114
|
+
|
115
|
+
===== Checking on 1 hosts ===== Begin... =====
|
116
|
+
+ [[ v20180326T104601 == '' ]]
|
117
|
+
+ [[ http://172.16.110.42/chef-repo/chef-dist.git == '' ]]
|
118
|
+
[...]
|
119
|
+
Converging 51 resources
|
120
|
+
Recipe: site_hadoop::default
|
121
|
+
* execute[centos::yum-update] action run
|
122
|
+
- Would execute yum -y update
|
123
|
+
Recipe: ssh::server
|
124
|
+
* yum_package[openssh-server] action install (up to date)
|
125
|
+
* service[ssh] action enable (up to date)
|
126
|
+
* service[ssh] action start (up to date)
|
127
|
+
* template[/etc/ssh/sshd_config] action create (up to date)
|
128
|
+
[...]
|
129
|
+
Chef Client finished, 3/133 resources would have been updated
|
130
|
+
===== Checking on 1 hosts ===== ...End =====
|
131
|
+
```
|
132
|
+
|
133
|
+
## Used credentials
|
134
|
+
|
135
|
+
| Credential | Usage
|
136
|
+
| --- | --- |
|
137
|
+
| `thycotic` | Used to connect to the Thycotic secret server if used within the `--secrets` option |
|
138
|
+
|
139
|
+
## Used Metadata
|
140
|
+
|
141
|
+
| Metadata | Type | Usage
|
142
|
+
| --- | --- | --- |
|
143
|
+
| `image` | `String` | OS image corresponding to the node |
|
144
|
+
| `services` | `Array<String>` | List of services to check for |
|
145
|
+
|
146
|
+
## Used environment variables
|
147
|
+
|
148
|
+
| Variable | Usage
|
149
|
+
| --- | --- |
|
150
|
+
| `hpc_certificates` | Directory containing certificates to be deployed on nodes by default. If ommitted, then no certificate is deployed. |
|
151
|
+
| `hpc_domain_for_thycotic` | Logon domain used to connect to the Thycotic secret server if used within the `--secrets` option |
|
152
|
+
|
153
|
+
## External tools dependencies
|
154
|
+
|
155
|
+
None
|
@@ -0,0 +1,198 @@
|
|
1
|
+
# `deploy`
|
2
|
+
|
3
|
+
The `deploy` executable will deploy the `master` branch on a node or list of nodes.
|
4
|
+
It will:
|
5
|
+
1. package the configuration,
|
6
|
+
2. optionally upload the packaged configuration on all needed artefact repositories, or on the nodes directly (depends on the nodes' configuration),
|
7
|
+
3. run deployments on all specified nodes,
|
8
|
+
4. display the result on screen, or in local log files (in case of parallel executions).
|
9
|
+
|
10
|
+
## Process
|
11
|
+
|
12
|
+
<!-- Mermaid generator - Section start -->
|
13
|
+
![Mermaid diagram](/docs/gen/mermaid/docs/executables/deploy.md-0.png)
|
14
|
+
<details>
|
15
|
+
<summary>See diagram Mermaid code</summary>
|
16
|
+
|
17
|
+
```mermaid
|
18
|
+
sequenceDiagram
|
19
|
+
participant Main as ./bin/deploy --node my_node
|
20
|
+
participant CMDB as CMDB
|
21
|
+
participant PlatformHandler as Platform Handler
|
22
|
+
participant PlatformRepo as Platform repository
|
23
|
+
participant Connector as Connector
|
24
|
+
participant Node as Provisioned node (my_node)
|
25
|
+
|
26
|
+
Main->>+CMDB: Get services to be deployed on my_node
|
27
|
+
CMDB->>+PlatformHandler: Get my_node metadata from the platform
|
28
|
+
PlatformHandler->>+PlatformRepo: Read platform inventory files
|
29
|
+
PlatformRepo-->>-PlatformHandler: Platform inventory
|
30
|
+
PlatformHandler-->>-CMDB: Services metadata containing my_web_app
|
31
|
+
CMDB-->>-Main: my_node has service my_web_app
|
32
|
+
Main->>+PlatformHandler: Get actions to deploy my_web_app
|
33
|
+
PlatformHandler-->>-Main: Actions to deploy my_web_app
|
34
|
+
Main->>+Connector: Connect to my_node to execute actions
|
35
|
+
Connector->>+Node: Execute actions to deploy my_web_app on my_node
|
36
|
+
Node-->>-Connector: Result of my_web_app deployment on my_node
|
37
|
+
Connector-->>-Main: Close connection
|
38
|
+
```
|
39
|
+
</details>
|
40
|
+
<!-- Mermaid generator - Section end -->
|
41
|
+
|
42
|
+
## Usage
|
43
|
+
|
44
|
+
```
|
45
|
+
Usage: ./bin/deploy [options]
|
46
|
+
|
47
|
+
Main options:
|
48
|
+
-d, --debug Activate debug mode
|
49
|
+
-h, --help Display help and exit
|
50
|
+
|
51
|
+
Nodes handler options:
|
52
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
53
|
+
|
54
|
+
Nodes selection options:
|
55
|
+
-a, --all-nodes Select all nodes
|
56
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
|
57
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
58
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
59
|
+
--nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
60
|
+
--nodes-git-impact GIT_IMPACT
|
61
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
62
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
63
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
|
64
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
65
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
66
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
67
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
68
|
+
|
69
|
+
Command runner options:
|
70
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
71
|
+
|
72
|
+
Actions Executor options:
|
73
|
+
-m, --max-threads NBR Set the number of threads to use for concurrent queries (defaults to 16)
|
74
|
+
|
75
|
+
Connector ssh options:
|
76
|
+
-g, --ssh-gateway-user USER Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
|
77
|
+
-j, --ssh-no-control-master If used, don't create SSH control masters for connections.
|
78
|
+
-q, --ssh-no-host-key-checking If used, don't check for SSH host keys.
|
79
|
+
-u, --ssh-user USER Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
|
80
|
+
-w, --password If used, then expect SSH connections to ask for a password.
|
81
|
+
-y GATEWAYS_CONF, Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
|
82
|
+
--ssh-gateways-conf
|
83
|
+
|
84
|
+
Deployer options:
|
85
|
+
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
86
|
+
* Local path to a JSON file
|
87
|
+
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
88
|
+
-p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
|
89
|
+
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to no timeout)
|
90
|
+
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
91
|
+
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
92
|
+
```
|
93
|
+
|
94
|
+
## Examples
|
95
|
+
|
96
|
+
```bash
|
97
|
+
# Deploy master on node23hst-nn1
|
98
|
+
./bin/deploy --node node23hst-nn1
|
99
|
+
|
100
|
+
# Check in "why run" mode the deployment of master on node23hst-nn1 (equivalent to ./bin/check-node --node node23hst-nn1)
|
101
|
+
./bin/deploy --node node23hst-nn1 --why-run
|
102
|
+
|
103
|
+
# Check in "why run" mode the deployment of master on node23hst-nn1 with a timeout of 1 minute
|
104
|
+
./bin/deploy --node node23hst-nn1 --why-run --timeout 60
|
105
|
+
|
106
|
+
# Deploy master using a file containing secrets on node23hst-nn1
|
107
|
+
./bin/deploy --node node23hst-nn1 --secrets passwords.json
|
108
|
+
|
109
|
+
# Deploy master on all nodes containing xae in their name
|
110
|
+
./bin/deploy --node /xae/
|
111
|
+
|
112
|
+
# Deploy master on all nodes containing xae in their name in parallel (and send each standard output in log files in ./run_logs/*.stdout)
|
113
|
+
./bin/deploy --node /xae/ --parallel
|
114
|
+
|
115
|
+
# Deploy master on all nodes containing xae in their name in parallel and using 32 threads in parallel
|
116
|
+
./bin/deploy --node /xae/ --parallel --max-threads 32
|
117
|
+
|
118
|
+
# Deploy master on all nodes defined in the list xaebhsone (from ./hosts_lists/xaebhsone)
|
119
|
+
./bin/deploy --nodes-list xaebhsone
|
120
|
+
|
121
|
+
# Deploy master on all nodes defined in the list xaebhsone and also node12hst-nn1 and node12hst-nn2
|
122
|
+
./bin/deploy --nodes-list xaebhsone --node node12hst-nn1 --node node12hst-nn2
|
123
|
+
|
124
|
+
# Deploy master on all nodes
|
125
|
+
./bin/deploy --all-nodes
|
126
|
+
```
|
127
|
+
|
128
|
+
Example of output:
|
129
|
+
```
|
130
|
+
=> ./bin/deploy --node node12had01 --why-run
|
131
|
+
Actions Executor configuration used:
|
132
|
+
* User: a_usernme
|
133
|
+
* Dry run: false
|
134
|
+
* Max threads used: 16
|
135
|
+
* Gateways configuration: madrid
|
136
|
+
* Gateway user: ubradm
|
137
|
+
* Debug mode: false
|
138
|
+
|
139
|
+
===== Packaging current repository ===== Begin... =====
|
140
|
+
cd ../chef-repo && rm -rf dist Berksfile.lock && ./bin/thor solo:bundle
|
141
|
+
Resolving cookbook dependencies...
|
142
|
+
Fetching 'project' from source at site-cookbooks/project
|
143
|
+
[...]
|
144
|
+
create data_bag/.gitkeep
|
145
|
+
create .gitignore
|
146
|
+
create .branch
|
147
|
+
create .chef_commit
|
148
|
+
===== Packaging current repository ===== ...End =====
|
149
|
+
|
150
|
+
===== Delivering on artefacts repositories ===== Begin... =====
|
151
|
+
cd ../chef-repo && ./bin/thor solo:deploy -r git@hpc.172.16.110.42:chef-repo/chef-dist.git -y
|
152
|
+
Warning: no 'deploy' tag found
|
153
|
+
Change log for branch v20180326T104601:
|
154
|
+
<empty>
|
155
|
+
Done
|
156
|
+
===== Delivering on artefacts repositories ===== ...End =====
|
157
|
+
|
158
|
+
===== Checking on 1 hosts ===== Begin... =====
|
159
|
+
+ [[ v20180326T104601 == '' ]]
|
160
|
+
+ [[ http://172.16.110.42/chef-repo/chef-dist.git == '' ]]
|
161
|
+
[...]
|
162
|
+
Converging 51 resources
|
163
|
+
Recipe: site_hadoop::default
|
164
|
+
* execute[centos::yum-update] action run
|
165
|
+
- Would execute yum -y update
|
166
|
+
Recipe: ssh::server
|
167
|
+
* yum_package[openssh-server] action install (up to date)
|
168
|
+
* service[ssh] action enable (up to date)
|
169
|
+
* service[ssh] action start (up to date)
|
170
|
+
* template[/etc/ssh/sshd_config] action create (up to date)
|
171
|
+
[...]
|
172
|
+
Chef Client finished, 3/133 resources would have been updated
|
173
|
+
===== Checking on 1 hosts ===== ...End =====
|
174
|
+
```
|
175
|
+
|
176
|
+
## Used credentials
|
177
|
+
|
178
|
+
| Credential | Usage
|
179
|
+
| --- | --- |
|
180
|
+
| `thycotic` | Used to connect to the Thycotic secret server if used within the `--secrets` option |
|
181
|
+
|
182
|
+
## Used Metadata
|
183
|
+
|
184
|
+
| Metadata | Type | Usage
|
185
|
+
| --- | --- | --- |
|
186
|
+
| `image` | `String` | OS image corresponding to the node |
|
187
|
+
| `services` | `Array<String>` | List of services to deploy |
|
188
|
+
|
189
|
+
## Used environment variables
|
190
|
+
|
191
|
+
| Variable | Usage
|
192
|
+
| --- | --- |
|
193
|
+
| `hpc_certificates` | Directory containing certificates to be deployed on nodes by default. If ommitted, then no certificate is deployed. |
|
194
|
+
| `hpc_domain_for_thycotic` | Logon domain used to connect to the Thycotic secret server if used within the `--secrets` option |
|
195
|
+
|
196
|
+
## External tools dependencies
|
197
|
+
|
198
|
+
None
|
@@ -0,0 +1,110 @@
|
|
1
|
+
# `dump_nodes_json`
|
2
|
+
|
3
|
+
The `dump_nodes_json` executable will dump the complete JSON node configurations and metadata as seen during a deployment in a JSON file.
|
4
|
+
The JSON dumped are in the directory `./nodes_json`.
|
5
|
+
It does so by running a special why-run deployment on the node itself.
|
6
|
+
Those JSON files can then be used for several purposes:
|
7
|
+
* Checking that differences are valid between 2 runs (involving code changes or manual updates).
|
8
|
+
* Get a complete node configuration easy to read and parse, for other tools.
|
9
|
+
* Extract plenty of useful information from the node itself directly from the JSON.
|
10
|
+
|
11
|
+
***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
|
12
|
+
|
13
|
+
## Process
|
14
|
+
|
15
|
+
TODO
|
16
|
+
|
17
|
+
## Usage
|
18
|
+
|
19
|
+
```
|
20
|
+
Usage: ./bin/dump_nodes_json [options]
|
21
|
+
|
22
|
+
Main options:
|
23
|
+
-d, --debug Activate debug mode
|
24
|
+
-h, --help Display help and exit
|
25
|
+
|
26
|
+
Nodes handler options:
|
27
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
28
|
+
|
29
|
+
Nodes selection options:
|
30
|
+
-a, --all-nodes Select all nodes
|
31
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
|
32
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
33
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
34
|
+
--nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
35
|
+
--nodes-git-impact GIT_IMPACT
|
36
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
37
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
38
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
|
39
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
40
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
41
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
42
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
43
|
+
|
44
|
+
Command runner options:
|
45
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
46
|
+
|
47
|
+
Connector ssh options:
|
48
|
+
-g, --ssh-gateway-user USER Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
|
49
|
+
--ssh-no-control-master If used, don't create SSH control masters for connections.
|
50
|
+
-q, --ssh-no-host-key-checking If used, don't check for SSH host keys.
|
51
|
+
-u, --ssh-user USER Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
|
52
|
+
-w, --password If used, then expect SSH connections to ask for a password.
|
53
|
+
-y GATEWAYS_CONF, Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
|
54
|
+
--ssh-gateways-conf
|
55
|
+
|
56
|
+
Deployer options:
|
57
|
+
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
58
|
+
* Local path to a JSON file
|
59
|
+
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
60
|
+
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
|
61
|
+
-W, --why-run Use the why-run mode to see what would be the result of the deploy instead of deploying it for real.
|
62
|
+
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
63
|
+
|
64
|
+
JSON dump options:
|
65
|
+
-k, --skip-run Skip the actual gathering of dumps in run_logs. If set, the current run_logs content will be used.
|
66
|
+
-j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
|
67
|
+
```
|
68
|
+
|
69
|
+
## Examples
|
70
|
+
|
71
|
+
```bash
|
72
|
+
# Dump JSON for the node named xaeprjcttlbd01
|
73
|
+
./bin/dump_nodes_json --node xaeprjcttlbd01
|
74
|
+
|
75
|
+
# Dump JSON for the node named xaeprjcttlbd01, but reuse the actual files in run_logs.
|
76
|
+
./bin/dump_nodes_json --node xaeprjcttlbd01 --skip-run
|
77
|
+
```
|
78
|
+
|
79
|
+
Example of output:
|
80
|
+
```
|
81
|
+
[ node23hst-nn80 ] - OK. Check nodes_json/node23hst-nn80.json
|
82
|
+
[ node23hst-nn81 ] - OK. Check nodes_json/node23hst-nn81.json
|
83
|
+
[ node23hst-nn82 ] - OK. Check nodes_json/node23hst-nn82.json
|
84
|
+
[ node23hst-nn84 ] - OK. Check nodes_json/node23hst-nn84.json
|
85
|
+
[ node23hst-nn85 ] - OK. Check nodes_json/node23hst-nn85.json
|
86
|
+
[ node23hst-nn86 ] - Error while dumping JSON. Check run_logs/node23hst-nn86.stdout
|
87
|
+
[ node23hst-nn87 ] - OK. Check nodes_json/node23hst-nn87.json
|
88
|
+
[ node23hst-nn88 ] - OK. Check nodes_json/node23hst-nn88.json
|
89
|
+
[ node23hst-nn90 ] - OK. Check nodes_json/node23hst-nn90.json
|
90
|
+
[ node23hst-nn8 ] - OK. Check nodes_json/node23hst-nn8.json
|
91
|
+
```
|
92
|
+
|
93
|
+
## Used credentials
|
94
|
+
|
95
|
+
| Credential | Usage
|
96
|
+
| --- | --- |
|
97
|
+
|
98
|
+
## Used Metadata
|
99
|
+
|
100
|
+
| Metadata | Type | Usage
|
101
|
+
| --- | --- | --- |
|
102
|
+
|
103
|
+
## Used environment variables
|
104
|
+
|
105
|
+
| Variable | Usage
|
106
|
+
| --- | --- |
|
107
|
+
|
108
|
+
## External tools dependencies
|
109
|
+
|
110
|
+
None
|