hybrid_platforms_conductor 32.12.0 → 32.13.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1103 -0
- data/LICENSE.md +31 -0
- data/README.md +395 -0
- data/bin/setup +1 -1
- data/docs/api.md +349 -0
- data/docs/config_dsl.md +315 -0
- data/docs/executables.md +226 -0
- data/docs/executables/check-node.md +155 -0
- data/docs/executables/deploy.md +198 -0
- data/docs/executables/dump_nodes_json.md +110 -0
- data/docs/executables/free_ips.md +93 -0
- data/docs/executables/free_veids.md +73 -0
- data/docs/executables/get_impacted_nodes.md +94 -0
- data/docs/executables/last_deploys.md +114 -0
- data/docs/executables/nodes_to_deploy.md +139 -0
- data/docs/executables/report.md +159 -0
- data/docs/executables/run.md +126 -0
- data/docs/executables/setup.md +92 -0
- data/docs/executables/ssh_config.md +151 -0
- data/docs/executables/test.md +213 -0
- data/docs/executables/topograph.md +139 -0
- data/docs/gen/mermaid/README.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/check-node.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_ips.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_veids.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/get_impacted_nodes.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/last_deploys.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/nodes_to_deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/report.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/run.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/setup.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/ssh_config.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/test.md-0.png +0 -0
- data/docs/install.md +161 -0
- data/docs/plugins.md +215 -0
- data/docs/plugins/action/bash.md +37 -0
- data/docs/plugins/action/interactive.md +37 -0
- data/docs/plugins/action/remote_bash.md +67 -0
- data/docs/plugins/action/ruby.md +69 -0
- data/docs/plugins/action/scp.md +61 -0
- data/docs/plugins/cmdb/config.md +46 -0
- data/docs/plugins/cmdb/host_ip.md +33 -0
- data/docs/plugins/cmdb/host_keys.md +33 -0
- data/docs/plugins/cmdb/platform_handlers.md +33 -0
- data/docs/plugins/connector/local.md +28 -0
- data/docs/plugins/connector/ssh.md +95 -0
- data/docs/plugins/platform_handler/yaml_inventory.md +105 -0
- data/docs/plugins/provisioner/docker.md +27 -0
- data/docs/plugins/provisioner/podman.md +27 -0
- data/docs/plugins/provisioner/proxmox.md +115 -0
- data/docs/plugins/report/confluence.md +49 -0
- data/docs/plugins/report/mediawiki.md +28 -0
- data/docs/plugins/report/stdout.md +32 -0
- data/docs/plugins/test/bitbucket_conf.md +97 -0
- data/docs/plugins/test/can_be_checked.md +27 -0
- data/docs/plugins/test/check_deploy_and_idempotence.md +61 -0
- data/docs/plugins/test/check_from_scratch.md +28 -0
- data/docs/plugins/test/connection.md +27 -0
- data/docs/plugins/test/deploy_freshness.md +27 -0
- data/docs/plugins/test/deploy_from_scratch.md +28 -0
- data/docs/plugins/test/deploy_removes_root_access.md +29 -0
- data/docs/plugins/test/divergence.md +41 -0
- data/docs/plugins/test/executables.md +26 -0
- data/docs/plugins/test/file_system.md +49 -0
- data/docs/plugins/test/file_system_hdfs.md +65 -0
- data/docs/plugins/test/hostname.md +27 -0
- data/docs/plugins/test/idempotence.md +56 -0
- data/docs/plugins/test/ip.md +28 -0
- data/docs/plugins/test/jenkins_ci_conf.md +54 -0
- data/docs/plugins/test/jenkins_ci_masters_ok.md +54 -0
- data/docs/plugins/test/linear_strategy.md +26 -0
- data/docs/plugins/test/local_users.md +48 -0
- data/docs/plugins/test/mounts.md +55 -0
- data/docs/plugins/test/orphan_files.md +38 -0
- data/docs/plugins/test/ports.md +50 -0
- data/docs/plugins/test/private_ips.md +27 -0
- data/docs/plugins/test/public_ips.md +27 -0
- data/docs/plugins/test/spectre.md +26 -0
- data/docs/plugins/test/veids.md +27 -0
- data/docs/plugins/test/vulnerabilities.md +65 -0
- data/docs/plugins/test_report/confluence.md +43 -0
- data/docs/plugins/test_report/stdout.md +26 -0
- data/docs/plugins_create.md +135 -0
- data/docs/tutorial.md +57 -0
- data/docs/tutorial/01_installation.md +129 -0
- data/docs/tutorial/02_first_node.md +466 -0
- data/docs/tutorial/03_scale.md +876 -0
- data/docs/tutorial/04_test.md +965 -0
- data/docs/tutorial/05_extend_with_plugins.md +1132 -0
- data/examples/bare/Gemfile +4 -0
- data/examples/bare/hpc_config.rb +2 -0
- data/examples/localhost/Gemfile +4 -0
- data/examples/localhost/hpc_config.rb +2 -0
- data/examples/localhost/inventory.yaml +4 -0
- data/lib/hybrid_platforms_conductor/actions_executor.rb +1 -0
- data/lib/hybrid_platforms_conductor/deployer.rb +3 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/remote_bash.rb +29 -13
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/scp.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/local.rb +98 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/my_connector.rb.sample +2 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb +7 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/platform_handler_plugin.rb.sample +5 -5
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/yaml_inventory.rb +140 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/provisioner/proxmox.rb +5 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/bitbucket_conf.rb +4 -4
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_freshness.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_removes_root_access.rb +19 -17
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/divergence.rb +3 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/hostname.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/ip.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/local_users.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/mounts.rb +4 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/orphan_files.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/spectre.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/vulnerabilities.rb +8 -7
- data/lib/hybrid_platforms_conductor/hpc_plugins/test_report/confluence.rb +1 -1
- data/lib/hybrid_platforms_conductor/json_dumper.rb +1 -1
- data/lib/hybrid_platforms_conductor/platform_handler.rb +1 -1
- data/lib/hybrid_platforms_conductor/services_handler.rb +18 -16
- data/lib/hybrid_platforms_conductor/tests_runner.rb +0 -1
- data/lib/hybrid_platforms_conductor/topographer.rb +0 -1
- data/lib/hybrid_platforms_conductor/version.rb +1 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/actions/remote_bash_spec.rb +16 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/connectable_nodes_spec.rb +30 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/remote_actions_spec.rb +113 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/cli_options_spec.rb +6 -2
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/global_helpers_spec.rb +38 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/remote_actions_spec.rb +8 -8
- data/spec/hybrid_platforms_conductor_test/docs_spec.rb +10 -0
- data/tools/check_md +89 -0
- data/tools/generate_mermaid +75 -0
- metadata +207 -12
@@ -0,0 +1,213 @@
|
|
1
|
+
# `test`
|
2
|
+
|
3
|
+
The `test` executable runs various tests and displays the eventual errors that have occurred.
|
4
|
+
Errors are being displayed at the end of the execution, along with a summary of the failed tests and nodes.
|
5
|
+
|
6
|
+
This `test` executable is using [test plugins](../plugins.md#test) to be able to validate various tests (at global level, on each node, or on the check-node output), and uses [test report plugins](../plugins.md#test_report) to publish test reports on various mediums.
|
7
|
+
|
8
|
+
This executable is perfectly suited to be integrated in a continuous integration workflow.
|
9
|
+
|
10
|
+
## Process
|
11
|
+
|
12
|
+
<!-- Mermaid generator - Section start -->
|
13
|
+
![Mermaid diagram](/docs/gen/mermaid/docs/executables/test.md-0.png)
|
14
|
+
<details>
|
15
|
+
<summary>See diagram Mermaid code</summary>
|
16
|
+
|
17
|
+
```mermaid
|
18
|
+
sequenceDiagram
|
19
|
+
participant Main as ./bin/test --all
|
20
|
+
participant CMDB as CMDB
|
21
|
+
participant PlatformHandler as Platform Handler
|
22
|
+
participant Connector as Connector
|
23
|
+
participant Node as Node
|
24
|
+
participant GlobalTest as Global Test
|
25
|
+
participant PlatformTest as Platform Test
|
26
|
+
participant NodeTest as Node Test
|
27
|
+
participant CheckNodeTest as Deployment Test
|
28
|
+
participant TestReport as Test Report
|
29
|
+
|
30
|
+
Main->>+CMDB: Get the list of platforms, services and nodes
|
31
|
+
CMDB-->>-Main: List of platforms, services and nodes
|
32
|
+
Main->>+PlatformHandler: Get actions to check services on nodes
|
33
|
+
PlatformHandler-->>-Main: Actions to check services on nodes
|
34
|
+
Main->>+NodeTest: Get actions to be executed on each node
|
35
|
+
NodeTest-->>-Main: Test actions to be executed
|
36
|
+
Main->>+Connector: Connect to nodes to execute node and deployment tests
|
37
|
+
Connector->>+Node: Execute node test actions
|
38
|
+
Node-->>-Connector: Node test logs
|
39
|
+
Connector->>+Node: Execute deployment check actions
|
40
|
+
Node-->>-Connector: Deployment check logs
|
41
|
+
Connector-->>-Main: Node and deployment test logs
|
42
|
+
Main->>+GlobalTest: Run each global test
|
43
|
+
GlobalTest-->>-Main: Global tests results
|
44
|
+
Main->>+PlatformTest: Run each platform test per platform
|
45
|
+
PlatformTest-->>-Main: Platform tests results
|
46
|
+
Main->>+NodeTest: Validate node tests based on test logs
|
47
|
+
NodeTest-->>-Main: Node tests results
|
48
|
+
Main->>+CheckNodeTest: Validate deployment tests based on deployment check logs
|
49
|
+
CheckNodeTest-->>-Main: Deployment tests results
|
50
|
+
Main->>+TestReport: Send all tests results to the Test Report
|
51
|
+
TestReport-->>-Main: Test report published
|
52
|
+
```
|
53
|
+
</details>
|
54
|
+
<!-- Mermaid generator - Section end -->
|
55
|
+
|
56
|
+
## Usage
|
57
|
+
|
58
|
+
```
|
59
|
+
Usage: ./bin/test [options]
|
60
|
+
|
61
|
+
Main options:
|
62
|
+
-d, --debug Activate debug mode
|
63
|
+
-h, --help Display help and exit
|
64
|
+
|
65
|
+
Nodes handler options:
|
66
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
67
|
+
|
68
|
+
Nodes selection options:
|
69
|
+
-a, --all-nodes Select all nodes
|
70
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
|
71
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
72
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
73
|
+
--nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
74
|
+
--nodes-git-impact GIT_IMPACT
|
75
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
76
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
77
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
|
78
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
79
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
80
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
81
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
82
|
+
|
83
|
+
Command runner options:
|
84
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
85
|
+
|
86
|
+
Connector ssh options:
|
87
|
+
-g, --ssh-gateway-user USER Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
|
88
|
+
-j, --ssh-no-control-master If used, don't create SSH control masters for connections.
|
89
|
+
-q, --ssh-no-host-key-checking If used, don't check for SSH host keys.
|
90
|
+
-u, --ssh-user USER Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
|
91
|
+
-w, --password If used, then expect SSH connections to ask for a password.
|
92
|
+
-y GATEWAYS_CONF, Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
|
93
|
+
--ssh-gateways-conf
|
94
|
+
|
95
|
+
Deployer options:
|
96
|
+
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
97
|
+
* Local path to a JSON file
|
98
|
+
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
99
|
+
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
100
|
+
|
101
|
+
Tests runner options:
|
102
|
+
-i, --tests-list FILE_NAME Specify a tests file name. The file should contain a list of tests name (1 per line). Can be used several times.
|
103
|
+
-k, --skip-run Skip running the check-node commands for real, and just analyze existing run logs.
|
104
|
+
-r, --report REPORT Specify a report name. Can be used several times. Can be all for all reports. Possible values: confluence, stdout (defaults to stdout).
|
105
|
+
-t, --test TEST Specify a test name. Can be used several times. Can be all for all tests. Possible values: ansible_repo_molecule_cdh_admins, ansible_repo_molecule_cdh_datanodes, ansible_repo_molecule_cdh_db, ansible_repo_molecule_cdh_gateways, ansible_repo_molecule_cdh_services, ansible_repo_molecule_common, ansible_repo_molecule_data_gateway, ansible_repo_molecule_dev_servers, ansible_repo_molecule_ds_servers, ansible_repo_molecule_dsnodes, ansible_repo_molecule_import_gateway, ansible_repo_molecule_notebooks, ansible_repo_molecule_tnz_data_gateway, bitbucket_conf, can_be_checked, check_from_scratch, chef_executables, chef_success, chef_woulds, connection, deploy_freshness, deploy_from_scratch, deploy_removes_root_access, executables, food_critic, group_ids, hostname, idempotence, ip, jenkins_ci_conf, jenkins_ci_masters_ok, linear_strategy, obsolete_home_dirs, obsolete_users, orphan_files, private_ips, public_ips, rubocop, spectre, unused_files, unused_node_attributes, unused_recipes, unused_templates, unused_roles, unused_users, user_ids, users_without_roles, veids (defaults to all).
|
106
|
+
--max-threads-connections NBR_THREADS
|
107
|
+
Specify the max number of threads to parallelize tests connecting on nodes (defaults to 64).
|
108
|
+
--max-threads-nodes NBR_THREADS
|
109
|
+
Specify the max number of threads to parallelize tests at node level (defaults to 8).
|
110
|
+
--max-threads-platforms NBR_THREADS
|
111
|
+
Specify the max number of threads to parallelize tests at platform level (defaults to 8).
|
112
|
+
```
|
113
|
+
|
114
|
+
## Examples
|
115
|
+
|
116
|
+
```bash
|
117
|
+
# Execute all tests on all nodes
|
118
|
+
./bin/test --all-nodes
|
119
|
+
|
120
|
+
# Execute only the tests named hostname and ip on all nodes whose names contain xae
|
121
|
+
./bin/test --test hostname --test ip --node /xae/
|
122
|
+
|
123
|
+
# Execute all tests on all nodes, but reuse the content of run_logs instead of why-run deployments
|
124
|
+
./bin/test --all-nodes --skip-run
|
125
|
+
|
126
|
+
# Execute the check_from_scratch test on all nodes impacted by changes made on the branch my_branch
|
127
|
+
./bin/test --test check_from_scratch --nodes-git-impact chef-repo::my_branch
|
128
|
+
```
|
129
|
+
|
130
|
+
Here is an example of output:
|
131
|
+
```
|
132
|
+
========== Error report of 6 tests run on 694 nodes
|
133
|
+
|
134
|
+
======= By test:
|
135
|
+
|
136
|
+
===== configuration_test found 2 nodes having errors:
|
137
|
+
* [ nodehst-nn3 ] - 3 errors:
|
138
|
+
- Failed to execute command "hostname -I"
|
139
|
+
- Failed to execute command "hostname -s"
|
140
|
+
- Failed to execute command "echo 'Test connection - ok'"
|
141
|
+
* [ project-pinger ] - 1 errors:
|
142
|
+
- Private IP outside
|
143
|
+
|
144
|
+
|
145
|
+
======= By node:
|
146
|
+
|
147
|
+
===== [ node45 ] - 1 failing tests:
|
148
|
+
* Test configuration_test - 3 errors:
|
149
|
+
- Failed to execute command "hostname -I"
|
150
|
+
- Failed to execute command "hostname -s"
|
151
|
+
- Failed to execute command "echo 'Test connection - ok'"
|
152
|
+
|
153
|
+
===== [ node12had41 ] - 1 failing tests:
|
154
|
+
* Test configuration_test - 1 errors:
|
155
|
+
- Failed to connect
|
156
|
+
|
157
|
+
===== [ node237 ] - 1 failing tests:
|
158
|
+
* Test configuration_test - 1 errors:
|
159
|
+
- Not handled by Chef
|
160
|
+
|
161
|
+
===== [ project-pinger ] - 1 failing tests:
|
162
|
+
* Test configuration_test - 1 errors:
|
163
|
+
- Private IP outside
|
164
|
+
|
165
|
+
|
166
|
+
========== Stats by hosts list:
|
167
|
+
|
168
|
+
+--------------------+----------+-----------+
|
169
|
+
| List name | % tested | % success |
|
170
|
+
+--------------------+----------+-----------+
|
171
|
+
| hosts_with_secrets | 100 % | 71 % |
|
172
|
+
| node12had | 100 % | 1 % |
|
173
|
+
| xaebhs5had | 100 % | 90 % |
|
174
|
+
| xaebhsone | 100 % | 0 % |
|
175
|
+
| xaerbx5had | 100 % | 0 % |
|
176
|
+
| xaerbxcas | 100 % | 0 % |
|
177
|
+
| xaerbxhad | 100 % | 0 % |
|
178
|
+
| xaesbg1cas | 100 % | 66 % |
|
179
|
+
| xaesbg1had | 100 % | 0 % |
|
180
|
+
| xaesbg2had | 100 % | 0 % |
|
181
|
+
| xaesbghad | 100 % | 0 % |
|
182
|
+
| xaesbgkfk | 100 % | 100 % |
|
183
|
+
| xaesbgzk | 100 % | 100 % |
|
184
|
+
| xaetirb1pdnc | 100 % | 0 % |
|
185
|
+
| xaetirb6tdnc | 100 % | 0 % |
|
186
|
+
| xaetisb3sdnc | 100 % | 0 % |
|
187
|
+
| No list | 100 % | 18 % |
|
188
|
+
+--------------------+----------+-----------+
|
189
|
+
|
190
|
+
===== Some errors were found. Check output. =====
|
191
|
+
```
|
192
|
+
|
193
|
+
## Used credentials
|
194
|
+
|
195
|
+
| Credential | Usage
|
196
|
+
| --- | --- |
|
197
|
+
|
198
|
+
## Used Metadata
|
199
|
+
|
200
|
+
| Metadata | Type | Usage
|
201
|
+
| --- | --- | --- |
|
202
|
+
| `host_ip` | `String` | IP address on which a node can be reachable |
|
203
|
+
| `image` | `String` | OS image to be used for tests |
|
204
|
+
| `services` | `Array<String>` | List of services to check for in tests |
|
205
|
+
|
206
|
+
## Used environment variables
|
207
|
+
|
208
|
+
| Variable | Usage
|
209
|
+
| --- | --- |
|
210
|
+
|
211
|
+
## External tools dependencies
|
212
|
+
|
213
|
+
None
|
@@ -0,0 +1,139 @@
|
|
1
|
+
# `topograph`
|
2
|
+
|
3
|
+
The `topograph` executable will dump the topology graph of a set of nodes.
|
4
|
+
This is useful to have a visualization of the network of nodes and their relations.
|
5
|
+
It dumps all the links and groups between a source set of nodes to a destination set of nodes, recursively (the sets can be "all nodes" too).
|
6
|
+
It uses the nodes' metadata, as well as the complete nodes JSON dumped by the `dump_nodes_json` executable to get links between nodes.
|
7
|
+
|
8
|
+
Prerequisites before running `topograph`:
|
9
|
+
* If the `svg` output format is used, then the `dot` utility should be installed in the system.
|
10
|
+
|
11
|
+
***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
|
12
|
+
|
13
|
+
## Process
|
14
|
+
|
15
|
+
TODO
|
16
|
+
|
17
|
+
## Usage
|
18
|
+
|
19
|
+
```
|
20
|
+
Usage: ./bin/topograph [options]
|
21
|
+
|
22
|
+
Main options:
|
23
|
+
-d, --debug Activate debug mode
|
24
|
+
-h, --help Display help and exit
|
25
|
+
|
26
|
+
Nodes handler options:
|
27
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
28
|
+
|
29
|
+
Command runner options:
|
30
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
31
|
+
|
32
|
+
Connector ssh options:
|
33
|
+
-g, --ssh-gateway-user USER Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
|
34
|
+
--ssh-no-control-master If used, don't create SSH control masters for connections.
|
35
|
+
-q, --ssh-no-host-key-checking If used, don't check for SSH host keys.
|
36
|
+
-u, --ssh-user USER Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
|
37
|
+
-w, --password If used, then expect SSH connections to ask for a password.
|
38
|
+
-y GATEWAYS_CONF, Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
|
39
|
+
--ssh-gateways-conf
|
40
|
+
|
41
|
+
Deployer options:
|
42
|
+
-e, --secrets SECRETS_LOCATION Specify a secrets location. Can be specified several times. Location can be:
|
43
|
+
* Local path to a JSON file
|
44
|
+
* URL of the form http[s]://<url>:<secret_id> to get a secret JSON file from a Thycotic Secret Server at the given URL.
|
45
|
+
-t, --timeout SECS Timeout in seconds to wait for each chef run. Only used in why-run mode. (defaults to 30)
|
46
|
+
--retries-on-error NBR Number of retries in case of non-deterministic errors (defaults to 0)
|
47
|
+
|
48
|
+
JSON dump options:
|
49
|
+
-j, --json-dir DIRECTORY Specify the output directory in which JSON files are being written. Defaults to nodes_json.
|
50
|
+
|
51
|
+
Topographer options:
|
52
|
+
-F, --from HOSTS_OPTIONS Specify options for the set of nodes to start from (enclose them with ""). Default: all nodes. HOSTS_OPTIONS follows the following:
|
53
|
+
-a, --all-nodes Select all nodes
|
54
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
|
55
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
56
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
57
|
+
-r, --nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
58
|
+
--nodes-git-impact GIT_IMPACT
|
59
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
60
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
61
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
|
62
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
63
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
64
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
65
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
66
|
+
-k, --skip-run Skip the actual gathering of JSON node files. If set, the current files in nodes_json will be used.
|
67
|
+
-p, --output FORMAT:FILE_NAME Specify a format and file name. Can be used several times. FORMAT can be one of graphviz, json, svg. Ex.: graphviz:graph.gv
|
68
|
+
-T, --to HOSTS_OPTIONS Specify options for the set of nodes to get to (enclose them with ""). Default: all nodes. HOSTS_OPTIONS follows the following:
|
69
|
+
-a, --all-nodes Select all nodes
|
70
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
|
71
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
72
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
73
|
+
-r, --nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
74
|
+
--nodes-git-impact GIT_IMPACT
|
75
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
76
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
77
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
|
78
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
79
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
80
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
81
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
82
|
+
```
|
83
|
+
|
84
|
+
## Examples
|
85
|
+
|
86
|
+
```bash
|
87
|
+
# Dump the whole network in JSON format
|
88
|
+
./bin/topograph --output json:graph.json
|
89
|
+
|
90
|
+
# Dump the whole network in JSON and SVG format
|
91
|
+
./bin/topograph --output json:graph.json --output svg:graph.svg
|
92
|
+
|
93
|
+
# Dump the network starting from any node belonging to the node12had hosts list
|
94
|
+
./bin/topograph --output json:graph.json --from "--nodes-list node12had"
|
95
|
+
|
96
|
+
# Dump the network getting to nodes xaeprjcttlbd01 and xaeprjctplbd01
|
97
|
+
./bin/topograph --output json:graph.json --to "--node xaeprjcttlbd01 --node xaeprjctplbd01"
|
98
|
+
|
99
|
+
# Dump the network getting from any node belonging to the node12had hosts list and to nodes xaeprjcttlbd01 and xaeprjctplbd01
|
100
|
+
./bin/topograph --output json:graph.json --from "--nodes-list node12had" --to "--node xaeprjcttlbd01 --node xaeprjctplbd01"
|
101
|
+
|
102
|
+
# Dump the whole network in JSON format, reusing existing JSON files from nodes_json (won't call dump_nodes_json)
|
103
|
+
./bin/topograph --output json:graph.json --skip-run
|
104
|
+
```
|
105
|
+
|
106
|
+
Example of output:
|
107
|
+
```
|
108
|
+
=> ./bin/topograph --skip-run --output graphviz:graph.gv
|
109
|
+
===== Compute graph...
|
110
|
+
!!! Missing JSON file nodes_json/node12hst-nn2.json
|
111
|
+
===== Add hosts lists clusters...
|
112
|
+
===== Define IP 24 clusters...
|
113
|
+
===== Select path...
|
114
|
+
===== Filter only nodes 172.16.0.0/12, 172.16.0.0/24, 172.16.1.0/24, 172.16.10.0/24, 172.16.110.0/24, xaetisb3sdnc21, xaetisb3sdnc22, xaetisb3sdnc23, xaetisb3sdnc24, xaetisb3sdnc25, xaetisb3sdnc3, xaetisb3sdnc4, xaetisb3sdnc5, xaetisb3sdnc6, xaetisb3sdnc7, xaetisb3sdnc8, xaetisb3sdnc9, xaetisb3sgwc01, xaetisb3snnc01, xaetisb3snnc02, xaetisbgpnsd01, xaetisqlpwbd01, xaetisqlcid01, xaetitanpwsd01, xaetitanuwsd01...
|
115
|
+
===== Collapse hosts lists...
|
116
|
+
===== Remove self references...
|
117
|
+
===== Remove empty clusters...
|
118
|
+
===== Write outputs...
|
119
|
+
===== Write graphviz file graph.gv...
|
120
|
+
```
|
121
|
+
|
122
|
+
## Used credentials
|
123
|
+
|
124
|
+
| Credential | Usage
|
125
|
+
| --- | --- |
|
126
|
+
|
127
|
+
## Used Metadata
|
128
|
+
|
129
|
+
| Metadata | Type | Usage
|
130
|
+
| --- | --- | --- |
|
131
|
+
|
132
|
+
## Used environment variables
|
133
|
+
|
134
|
+
| Variable | Usage
|
135
|
+
| --- | --- |
|
136
|
+
|
137
|
+
## External tools dependencies
|
138
|
+
|
139
|
+
None
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
Binary file
|
data/docs/install.md
ADDED
@@ -0,0 +1,161 @@
|
|
1
|
+
# Requirements
|
2
|
+
|
3
|
+
For a bare usage (no plugins), the only requirement of Hybrid Platforms Conductor is **Ruby**.
|
4
|
+
|
5
|
+
Then depending on the plugins being used, external tools might need to be installed (see below).
|
6
|
+
Commands in this documentation are taken from a Debian-based environment, but they can be easily translated into other Linuxes.
|
7
|
+
|
8
|
+
## Install Ruby
|
9
|
+
|
10
|
+
Here are some ways to install it.
|
11
|
+
|
12
|
+
### Compiling it from scratch.
|
13
|
+
|
14
|
+
```bash
|
15
|
+
mkdir ruby
|
16
|
+
cd ruby
|
17
|
+
wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.3.tar.gz
|
18
|
+
tar xvzf ruby-2.7.3.tar.gz
|
19
|
+
cd ruby-2.7.3
|
20
|
+
sudo apt install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev libgdbm-compat-dev bison
|
21
|
+
./configure
|
22
|
+
make
|
23
|
+
sudo make install
|
24
|
+
cd ../..
|
25
|
+
```
|
26
|
+
|
27
|
+
### Using RVM
|
28
|
+
|
29
|
+
```bash
|
30
|
+
sudo apt-get install dirmngr curl
|
31
|
+
gpg --keyserver hkp://keys.gnupg.net:80 --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
|
32
|
+
curl -sSL https://get.rvm.io | bash -s stable
|
33
|
+
rvm install 2.5.0
|
34
|
+
rvm use 2.5.0
|
35
|
+
```
|
36
|
+
|
37
|
+
### Using Debian (>= Stretch) package manager
|
38
|
+
|
39
|
+
```bash
|
40
|
+
sudo apt-get install ruby-dev libffi-dev zlib1g-dev
|
41
|
+
```
|
42
|
+
|
43
|
+
## Install the `hybrid_platforms_conductor` rubygem
|
44
|
+
|
45
|
+
This can be done either in your Ruby system directories, or defined as a dependency of a Ruby project
|
46
|
+
|
47
|
+
### As a system-wide tool
|
48
|
+
|
49
|
+
```bash
|
50
|
+
sudo gem install hybrid_platforms_conductor
|
51
|
+
```
|
52
|
+
|
53
|
+
Then the tools can be used directly from the terminal (they should be part of the PATH).
|
54
|
+
|
55
|
+
### As a Ruby project
|
56
|
+
|
57
|
+
This needs `bundler` to be installed as well (see below).
|
58
|
+
|
59
|
+
1. In a new directory, create a file named `Gemfile`, and declare the dependency on the `hybrid_platforms_conductor` rubygem:
|
60
|
+
|
61
|
+
`Gemfile` content:
|
62
|
+
```ruby
|
63
|
+
source 'http://rubygems.org'
|
64
|
+
|
65
|
+
# Orchestrate all the platforms with Hybrid Platforms Conductor
|
66
|
+
gem 'hybrid_platforms_conductor'
|
67
|
+
```
|
68
|
+
|
69
|
+
2. Install the dependencies of your Ruby project
|
70
|
+
|
71
|
+
```bash
|
72
|
+
bundle config set --local path vendor/bundle
|
73
|
+
bundle install
|
74
|
+
bundle binstubs hybrid_platforms_conductor
|
75
|
+
```
|
76
|
+
|
77
|
+
Then the tools can be used directly from the Ruby project directory, inside the `./bin` folder.
|
78
|
+
|
79
|
+
## Create the Hybrid Platforms Conductor main configuration file
|
80
|
+
|
81
|
+
As a minimum requirement, the current directory from which the tools are being called should have a file named `hpc_config.rb`.
|
82
|
+
The file can be empty, and directives can be used to define the various platforms and configuration parameters.
|
83
|
+
|
84
|
+
## Check installation
|
85
|
+
|
86
|
+
A correct Hybrid Platforms Conductor installation can be checked by running the `run --help` command.
|
87
|
+
|
88
|
+
The output should look like this:
|
89
|
+
|
90
|
+
```
|
91
|
+
Usage: run [options]
|
92
|
+
|
93
|
+
Main options:
|
94
|
+
-d, --debug Activate debug mode
|
95
|
+
-h, --help Display help and exit
|
96
|
+
-c, --command CMD Command to execute (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
|
97
|
+
-f, --commands-file FILE_NAME Execute commands taken from a file (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
|
98
|
+
-i, --interactive Run an interactive SSH session instead of executing a command (can't be used with --command or --commands-file)
|
99
|
+
-p, --parallel Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
|
100
|
+
-t, --timeout SECS Timeout in seconds to wait for each command (defaults to no timeout)
|
101
|
+
|
102
|
+
Nodes handler options:
|
103
|
+
-o, --show-nodes Display the list of possible nodes and exit
|
104
|
+
|
105
|
+
Nodes selection options:
|
106
|
+
-a, --all-nodes Select all nodes
|
107
|
+
-b, --nodes-platform PLATFORM Select nodes belonging to a given platform name. Available platforms are: (can be used several times)
|
108
|
+
-l, --nodes-list LIST Select nodes defined in a nodes list (can be used several times)
|
109
|
+
-n, --node NODE Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
|
110
|
+
-r, --nodes-service SERVICE Select nodes implementing a given service (can be used several times)
|
111
|
+
--nodes-git-impact GIT_IMPACT
|
112
|
+
Select nodes impacted by a git diff from a platform (can be used several times).
|
113
|
+
GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
|
114
|
+
* PLATFORM: Name of the platform to check git diff from. Available platforms are:
|
115
|
+
* FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
|
116
|
+
* TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
|
117
|
+
* FLAGS: Extra comma-separated flags. The following flags are supported:
|
118
|
+
- min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
|
119
|
+
|
120
|
+
Command runner options:
|
121
|
+
-s, --show-commands Display the commands that would be run instead of running them
|
122
|
+
|
123
|
+
Actions Executor options:
|
124
|
+
-m, --max-threads NBR Set the number of threads to use for concurrent queries (defaults to 16)
|
125
|
+
```
|
126
|
+
|
127
|
+
### For a system-wide installation
|
128
|
+
|
129
|
+
```bash
|
130
|
+
run --help
|
131
|
+
```
|
132
|
+
|
133
|
+
### For a Ruby project installation
|
134
|
+
|
135
|
+
```bash
|
136
|
+
./bin/run --help
|
137
|
+
```
|
138
|
+
|
139
|
+
## Other dependencies
|
140
|
+
|
141
|
+
The following dependencies are not needed for a minimum installation, but are required for some of the plugins provided by default.
|
142
|
+
|
143
|
+
## Git
|
144
|
+
|
145
|
+
```bash
|
146
|
+
sudo apt install git
|
147
|
+
git config --global user.email "<your_email>"
|
148
|
+
git config --global user.name "<your_user_name>"
|
149
|
+
```
|
150
|
+
|
151
|
+
## SSH client
|
152
|
+
|
153
|
+
```bash
|
154
|
+
sudo apt install openssh-client
|
155
|
+
```
|
156
|
+
|
157
|
+
## Bundler
|
158
|
+
|
159
|
+
```bash
|
160
|
+
sudo gem install bundler
|
161
|
+
```
|