hybrid_platforms_conductor 32.12.0 → 32.13.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1103 -0
- data/LICENSE.md +31 -0
- data/README.md +395 -0
- data/bin/setup +1 -1
- data/docs/api.md +349 -0
- data/docs/config_dsl.md +315 -0
- data/docs/executables.md +226 -0
- data/docs/executables/check-node.md +155 -0
- data/docs/executables/deploy.md +198 -0
- data/docs/executables/dump_nodes_json.md +110 -0
- data/docs/executables/free_ips.md +93 -0
- data/docs/executables/free_veids.md +73 -0
- data/docs/executables/get_impacted_nodes.md +94 -0
- data/docs/executables/last_deploys.md +114 -0
- data/docs/executables/nodes_to_deploy.md +139 -0
- data/docs/executables/report.md +159 -0
- data/docs/executables/run.md +126 -0
- data/docs/executables/setup.md +92 -0
- data/docs/executables/ssh_config.md +151 -0
- data/docs/executables/test.md +213 -0
- data/docs/executables/topograph.md +139 -0
- data/docs/gen/mermaid/README.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/check-node.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_ips.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/free_veids.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/get_impacted_nodes.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/last_deploys.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/nodes_to_deploy.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/report.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/run.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/setup.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/ssh_config.md-0.png +0 -0
- data/docs/gen/mermaid/docs/executables/test.md-0.png +0 -0
- data/docs/install.md +161 -0
- data/docs/plugins.md +215 -0
- data/docs/plugins/action/bash.md +37 -0
- data/docs/plugins/action/interactive.md +37 -0
- data/docs/plugins/action/remote_bash.md +67 -0
- data/docs/plugins/action/ruby.md +69 -0
- data/docs/plugins/action/scp.md +61 -0
- data/docs/plugins/cmdb/config.md +46 -0
- data/docs/plugins/cmdb/host_ip.md +33 -0
- data/docs/plugins/cmdb/host_keys.md +33 -0
- data/docs/plugins/cmdb/platform_handlers.md +33 -0
- data/docs/plugins/connector/local.md +28 -0
- data/docs/plugins/connector/ssh.md +95 -0
- data/docs/plugins/platform_handler/yaml_inventory.md +105 -0
- data/docs/plugins/provisioner/docker.md +27 -0
- data/docs/plugins/provisioner/podman.md +27 -0
- data/docs/plugins/provisioner/proxmox.md +115 -0
- data/docs/plugins/report/confluence.md +49 -0
- data/docs/plugins/report/mediawiki.md +28 -0
- data/docs/plugins/report/stdout.md +32 -0
- data/docs/plugins/test/bitbucket_conf.md +97 -0
- data/docs/plugins/test/can_be_checked.md +27 -0
- data/docs/plugins/test/check_deploy_and_idempotence.md +61 -0
- data/docs/plugins/test/check_from_scratch.md +28 -0
- data/docs/plugins/test/connection.md +27 -0
- data/docs/plugins/test/deploy_freshness.md +27 -0
- data/docs/plugins/test/deploy_from_scratch.md +28 -0
- data/docs/plugins/test/deploy_removes_root_access.md +29 -0
- data/docs/plugins/test/divergence.md +41 -0
- data/docs/plugins/test/executables.md +26 -0
- data/docs/plugins/test/file_system.md +49 -0
- data/docs/plugins/test/file_system_hdfs.md +65 -0
- data/docs/plugins/test/hostname.md +27 -0
- data/docs/plugins/test/idempotence.md +56 -0
- data/docs/plugins/test/ip.md +28 -0
- data/docs/plugins/test/jenkins_ci_conf.md +54 -0
- data/docs/plugins/test/jenkins_ci_masters_ok.md +54 -0
- data/docs/plugins/test/linear_strategy.md +26 -0
- data/docs/plugins/test/local_users.md +48 -0
- data/docs/plugins/test/mounts.md +55 -0
- data/docs/plugins/test/orphan_files.md +38 -0
- data/docs/plugins/test/ports.md +50 -0
- data/docs/plugins/test/private_ips.md +27 -0
- data/docs/plugins/test/public_ips.md +27 -0
- data/docs/plugins/test/spectre.md +26 -0
- data/docs/plugins/test/veids.md +27 -0
- data/docs/plugins/test/vulnerabilities.md +65 -0
- data/docs/plugins/test_report/confluence.md +43 -0
- data/docs/plugins/test_report/stdout.md +26 -0
- data/docs/plugins_create.md +135 -0
- data/docs/tutorial.md +57 -0
- data/docs/tutorial/01_installation.md +129 -0
- data/docs/tutorial/02_first_node.md +466 -0
- data/docs/tutorial/03_scale.md +876 -0
- data/docs/tutorial/04_test.md +965 -0
- data/docs/tutorial/05_extend_with_plugins.md +1132 -0
- data/examples/bare/Gemfile +4 -0
- data/examples/bare/hpc_config.rb +2 -0
- data/examples/localhost/Gemfile +4 -0
- data/examples/localhost/hpc_config.rb +2 -0
- data/examples/localhost/inventory.yaml +4 -0
- data/lib/hybrid_platforms_conductor/actions_executor.rb +1 -0
- data/lib/hybrid_platforms_conductor/deployer.rb +3 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/remote_bash.rb +29 -13
- data/lib/hybrid_platforms_conductor/hpc_plugins/action/scp.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/local.rb +98 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/my_connector.rb.sample +2 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/connector/ssh.rb +7 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/platform_handler_plugin.rb.sample +5 -5
- data/lib/hybrid_platforms_conductor/hpc_plugins/platform_handler/yaml_inventory.rb +140 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/provisioner/proxmox.rb +5 -2
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/bitbucket_conf.rb +4 -4
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_freshness.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/deploy_removes_root_access.rb +19 -17
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/divergence.rb +3 -0
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/hostname.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/ip.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/local_users.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/mounts.rb +4 -3
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/orphan_files.rb +2 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/spectre.rb +1 -1
- data/lib/hybrid_platforms_conductor/hpc_plugins/test/vulnerabilities.rb +8 -7
- data/lib/hybrid_platforms_conductor/hpc_plugins/test_report/confluence.rb +1 -1
- data/lib/hybrid_platforms_conductor/json_dumper.rb +1 -1
- data/lib/hybrid_platforms_conductor/platform_handler.rb +1 -1
- data/lib/hybrid_platforms_conductor/services_handler.rb +18 -16
- data/lib/hybrid_platforms_conductor/tests_runner.rb +0 -1
- data/lib/hybrid_platforms_conductor/topographer.rb +0 -1
- data/lib/hybrid_platforms_conductor/version.rb +1 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/actions/remote_bash_spec.rb +16 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/connectable_nodes_spec.rb +30 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/local/remote_actions_spec.rb +113 -0
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/cli_options_spec.rb +6 -2
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/global_helpers_spec.rb +38 -1
- data/spec/hybrid_platforms_conductor_test/api/actions_executor/connectors/ssh/remote_actions_spec.rb +8 -8
- data/spec/hybrid_platforms_conductor_test/docs_spec.rb +10 -0
- data/tools/check_md +89 -0
- data/tools/generate_mermaid +75 -0
- metadata +207 -12
|
@@ -0,0 +1,965 @@
|
|
|
1
|
+
|
|
2
|
+
---
|
|
3
|
+
**<p style="text-align: center;">Tutorial navigation</p>**
|
|
4
|
+
|
|
5
|
+
| <sub>[Introduction](/docs/tutorial.md)</sub> | <sub>[1. Installation and first-time setup](/docs/tutorial/01_installation.md)</sub> | <sub>[2. Deploy and check a first node](/docs/tutorial/02_first_node.md)</sub> | <sub>[3. Scale your processes](/docs/tutorial/03_scale.md)</sub> | <nobr><sub><sub>👇You are here👇</sub></sub></nobr><br><sub>[4. Testing your processes and platforms](/docs/tutorial/04_test.md)</sub> | <sub>[5. Extend Hybrid Platforms Conductor with your own requirements](/docs/tutorial/05_extend_with_plugins.md)</sub> |
|
|
6
|
+
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
|
|
7
|
+
| <sub><sub>**[Use-case](/docs/tutorial.md#use-case)**</sub></sub> | <sub><sub>**[Dependencies installation](/docs/tutorial/01_installation.md#hpc-dependencies)**</sub></sub> | <sub><sub>**[Add your first node and its platform repository](/docs/tutorial/02_first_node.md#add-first-node)**</sub></sub> | <sub><sub>**[Provision our web services platform](/docs/tutorial/03_scale.md#provision)**</sub></sub> | <sub><sub>**[Hello test framework](/docs/tutorial/04_test.md#framework)**</sub></sub> | <sub><sub>**[Create your plugins' repository](/docs/tutorial/05_extend_with_plugins.md#plugins-repo)**</sub></sub> |
|
|
8
|
+
| <sub><sub>**[Prerequisites](/docs/tutorial.md#prerequisites)**</sub></sub> | <sub><sub>**[Our platforms' main repository](/docs/tutorial/01_installation.md#main-repo)**</sub></sub> | <sub><sub>**[Check and deploy services on this node](/docs/tutorial/02_first_node.md#check-deploy)**</sub></sub> | <sub><sub>**[Run commands on our new web services](/docs/tutorial/03_scale.md#run)**</sub></sub> | <sub><sub>**[Testing your nodes](/docs/tutorial/04_test.md#nodes-tests)**</sub></sub> | <sub><sub>**[Your own platform handler](/docs/tutorial/05_extend_with_plugins.md#platform-handler)**</sub></sub> |
|
|
9
|
+
| <sub><sub>**[Tutorial setup](/docs/tutorial.md#tutorial-setup)**</sub></sub> | | <sub><sub>**[Updating the configuration](/docs/tutorial/02_first_node.md#update)**</sub></sub> | <sub><sub>**[Check and deploy our web services on several nodes at once](/docs/tutorial/03_scale.md#check-deploy)**</sub></sub> | <sub><sub>**[Testing your platforms' configuration](/docs/tutorial/04_test.md#platforms-tests)**</sub></sub> | <sub><sub>**[Write your own tests](/docs/tutorial/05_extend_with_plugins.md#test)**</sub></sub> |
|
|
10
|
+
| | | | | <sub><sub>**[Other kinds of tests](/docs/tutorial/04_test.md#other-tests)**</sub></sub> | <sub><sub>**[Enough of stdout, we want to report to other tools](/docs/tutorial/05_extend_with_plugins.md#report)**</sub></sub> |
|
|
11
|
+
| | | | | | <sub><sub>**[What next?](/docs/tutorial/05_extend_with_plugins.md#what-next)**</sub></sub> |
|
|
12
|
+
|
|
13
|
+
# 4. Testing your processes and platforms
|
|
14
|
+
|
|
15
|
+
Hybrid Platforms Conductor comes with a bunch of [test plugins](/docs/plugins.md#test) that cover both your processes and your platforms.
|
|
16
|
+
This section will show you some of the most important tests you can use and automate.
|
|
17
|
+
|
|
18
|
+
All tests are run using the [`test` executable](/docs/executables/test.md).
|
|
19
|
+
|
|
20
|
+
<a name="framework"></a>
|
|
21
|
+
## Hello test framework
|
|
22
|
+
|
|
23
|
+
One of simplest tests provided is to check whether your nodes are reachable or not by Hybrid Platforms Conductor.
|
|
24
|
+
That means whether your processes have a [connector plugin](/docs/plugins.md#connector) able to connect to them or not.
|
|
25
|
+
Having such a connector is what enables your processes to use executables like [`run`](/docs/executables/run.md), [`check-node`](/docs/executables/check-node.md) or [`deploy`](/docs/executables/deploy.md) on your nodes.
|
|
26
|
+
Therefore it is important that this is tested and failures be reported.
|
|
27
|
+
The test plugin responsible for such tests is the [`connection` test plugin](/docs/plugins/test/connection.md).
|
|
28
|
+
|
|
29
|
+
Let's invoke it:
|
|
30
|
+
```bash
|
|
31
|
+
./bin/test --all --test connection
|
|
32
|
+
# =>
|
|
33
|
+
# ===== Run 11 connected tests ==== Begin...
|
|
34
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== Begin...
|
|
35
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 11 - Total: 11 ]
|
|
36
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== ...End
|
|
37
|
+
#
|
|
38
|
+
# [ 2021-04-29 08:34:43 ] - [ Node local ] - [ connection ] - Start test...
|
|
39
|
+
# [ 2021-04-29 08:34:43 ] - [ Node local ] - [ connection ] - Test finished in 3.2988e-05 seconds.
|
|
40
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web1 ] - [ connection ] - Start test...
|
|
41
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web1 ] - [ connection ] - Test finished in 1.8718e-05 seconds.
|
|
42
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web10 ] - [ connection ] - Start test...
|
|
43
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web10 ] - [ connection ] - Test finished in 1.812e-05 seconds.
|
|
44
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web2 ] - [ connection ] - Start test...
|
|
45
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web2 ] - [ connection ] - Test finished in 2.8482e-05 seconds.
|
|
46
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web3 ] - [ connection ] - Start test...
|
|
47
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web3 ] - [ connection ] - Test finished in 1.6661e-05 seconds.
|
|
48
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web4 ] - [ connection ] - Start test...
|
|
49
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web4 ] - [ connection ] - Test finished in 1.6589e-05 seconds.
|
|
50
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web5 ] - [ connection ] - Start test...
|
|
51
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web5 ] - [ connection ] - Test finished in 1.8892e-05 seconds.
|
|
52
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web6 ] - [ connection ] - Start test...
|
|
53
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web6 ] - [ connection ] - Test finished in 2.11e-05 seconds.
|
|
54
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web7 ] - [ connection ] - Start test...
|
|
55
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web7 ] - [ connection ] - Test finished in 1.5781e-05 seconds.
|
|
56
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web8 ] - [ connection ] - Start test...
|
|
57
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web8 ] - [ connection ] - Test finished in 1.603e-05 seconds.
|
|
58
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web9 ] - [ connection ] - Start test...
|
|
59
|
+
# [ 2021-04-29 08:34:43 ] - [ Node web9 ] - [ connection ] - Test finished in 1.7352e-05 seconds.
|
|
60
|
+
# ===== Run 11 connected tests ==== ...End
|
|
61
|
+
#
|
|
62
|
+
#
|
|
63
|
+
# ========== Error report of 11 tests run on 11 nodes
|
|
64
|
+
#
|
|
65
|
+
# ======= 0 unexpected failing global tests:
|
|
66
|
+
#
|
|
67
|
+
#
|
|
68
|
+
# ======= 0 unexpected failing platform tests:
|
|
69
|
+
#
|
|
70
|
+
#
|
|
71
|
+
# ======= 0 unexpected failing node tests:
|
|
72
|
+
#
|
|
73
|
+
#
|
|
74
|
+
# ======= 0 unexpected failing platforms:
|
|
75
|
+
#
|
|
76
|
+
#
|
|
77
|
+
# ======= 0 unexpected failing nodes:
|
|
78
|
+
#
|
|
79
|
+
#
|
|
80
|
+
# ========== Stats by nodes list:
|
|
81
|
+
#
|
|
82
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
83
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
84
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
85
|
+
# | No list | 11 | 100 % | 100 % | 100 % | ========================================= |
|
|
86
|
+
# | All | 11 | 100 % | 100 % | 100 % | ========================================= |
|
|
87
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
88
|
+
#
|
|
89
|
+
# ===== No unexpected errors =====
|
|
90
|
+
```
|
|
91
|
+
Here we see that the connection test has reported a success rate of 100 % on a total of 11 nodes (our `local` node and the 10 `webN` nodes).
|
|
92
|
+
All is green.
|
|
93
|
+
|
|
94
|
+
Let's see what happens when problems occur: we will stop some of our web services on purpose, and restart the stest:
|
|
95
|
+
```bash
|
|
96
|
+
# Stop some containers
|
|
97
|
+
docker container stop web1 web3 web5
|
|
98
|
+
|
|
99
|
+
# Re-run connection tests
|
|
100
|
+
./bin/test --all --test connection
|
|
101
|
+
# =>
|
|
102
|
+
# ===== Run 11 connected tests ==== Begin...
|
|
103
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== Begin...
|
|
104
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51240)] ERROR - [ CmdRunner ] - Command 'getent hosts web1.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
105
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51240)] WARN - [ HostIp ] - Host web1.hpc_tutorial.org has no IP.
|
|
106
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51260)] ERROR - [ CmdRunner ] - Command 'getent hosts web3.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
107
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51260)] WARN - [ HostIp ] - Host web3.hpc_tutorial.org has no IP.
|
|
108
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51280)] ERROR - [ CmdRunner ] - Command 'getent hosts web5.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
109
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51280)] WARN - [ HostIp ] - Host web5.hpc_tutorial.org has no IP.
|
|
110
|
+
# [2021-04-29 08:37:32 (PID 1229 / TID 51300)] WARN - [ ActionsExecutor ] - The following nodes have no possible connector to them: web1, web3, web5
|
|
111
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 8 - Total: 8 ]
|
|
112
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== ...End
|
|
113
|
+
#
|
|
114
|
+
# [ 2021-04-29 08:37:35 ] - [ Node local ] - [ connection ] - Start test...
|
|
115
|
+
# [ 2021-04-29 08:37:35 ] - [ Node local ] - [ connection ] - Test finished in 0.000189158 seconds.
|
|
116
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web1 ] - [ connection ] - Start test...
|
|
117
|
+
# [2021-04-29 08:37:35 (PID 1229 / TID 51300)] ERROR - [ Connection ] - [ #< Test connection - Node web1 > ] - Error while executing tests: no_connector: Unable to get a connector to web1
|
|
118
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web1 ] - [ connection ] - Test finished in 0.000381365 seconds.
|
|
119
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web10 ] - [ connection ] - Start test...
|
|
120
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web10 ] - [ connection ] - Test finished in 0.000116228 seconds.
|
|
121
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web2 ] - [ connection ] - Start test...
|
|
122
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web2 ] - [ connection ] - Test finished in 0.000160162 seconds.
|
|
123
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web3 ] - [ connection ] - Start test...
|
|
124
|
+
# [2021-04-29 08:37:35 (PID 1229 / TID 51300)] ERROR - [ Connection ] - [ #< Test connection - Node web3 > ] - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
125
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web3 ] - [ connection ] - Test finished in 0.000344236 seconds.
|
|
126
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web4 ] - [ connection ] - Start test...
|
|
127
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web4 ] - [ connection ] - Test finished in 0.000159634 seconds.
|
|
128
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web5 ] - [ connection ] - Start test...
|
|
129
|
+
# [2021-04-29 08:37:35 (PID 1229 / TID 51300)] ERROR - [ Connection ] - [ #< Test connection - Node web5 > ] - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
130
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web5 ] - [ connection ] - Test finished in 0.000260947 seconds.
|
|
131
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web6 ] - [ connection ] - Start test...
|
|
132
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web6 ] - [ connection ] - Test finished in 0.000120757 seconds.
|
|
133
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web7 ] - [ connection ] - Start test...
|
|
134
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web7 ] - [ connection ] - Test finished in 0.000150549 seconds.
|
|
135
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web8 ] - [ connection ] - Start test...
|
|
136
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web8 ] - [ connection ] - Test finished in 0.000109725 seconds.
|
|
137
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web9 ] - [ connection ] - Start test...
|
|
138
|
+
# [ 2021-04-29 08:37:35 ] - [ Node web9 ] - [ connection ] - Test finished in 0.000140073 seconds.
|
|
139
|
+
# ===== Run 11 connected tests ==== ...End
|
|
140
|
+
#
|
|
141
|
+
#
|
|
142
|
+
# ========== Error report of 11 tests run on 11 nodes
|
|
143
|
+
#
|
|
144
|
+
# ======= 0 unexpected failing global tests:
|
|
145
|
+
#
|
|
146
|
+
#
|
|
147
|
+
# ======= 0 unexpected failing platform tests:
|
|
148
|
+
#
|
|
149
|
+
#
|
|
150
|
+
# ======= 1 unexpected failing node tests:
|
|
151
|
+
#
|
|
152
|
+
# ===== connection found 3 nodes having errors:
|
|
153
|
+
# * [ web1 ] - 1 errors:
|
|
154
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web1
|
|
155
|
+
# * [ web3 ] - 1 errors:
|
|
156
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
157
|
+
# * [ web5 ] - 1 errors:
|
|
158
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
159
|
+
#
|
|
160
|
+
#
|
|
161
|
+
# ======= 0 unexpected failing platforms:
|
|
162
|
+
#
|
|
163
|
+
#
|
|
164
|
+
# ======= 3 unexpected failing nodes:
|
|
165
|
+
#
|
|
166
|
+
# ===== web1 has 1 failing tests:
|
|
167
|
+
# * [ connection ] - 1 errors:
|
|
168
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web1
|
|
169
|
+
#
|
|
170
|
+
# ===== web3 has 1 failing tests:
|
|
171
|
+
# * [ connection ] - 1 errors:
|
|
172
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
173
|
+
#
|
|
174
|
+
# ===== web5 has 1 failing tests:
|
|
175
|
+
# * [ connection ] - 1 errors:
|
|
176
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
177
|
+
#
|
|
178
|
+
#
|
|
179
|
+
# ========== Stats by nodes list:
|
|
180
|
+
#
|
|
181
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
182
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
183
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
184
|
+
# | No list | 11 | 100 % | 100 % | 72 % | ========================================= |
|
|
185
|
+
# | All | 11 | 100 % | 100 % | 72 % | ========================================= |
|
|
186
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
187
|
+
#
|
|
188
|
+
# ===== Some errors were found. Check output. =====
|
|
189
|
+
|
|
190
|
+
# Check exit code
|
|
191
|
+
echo $?
|
|
192
|
+
# => 1
|
|
193
|
+
```
|
|
194
|
+
Here you see that 3 nodes were reported failing the test: success rate is down to 72 %, the command exit code is 1 (useful to integrate such command in third-party tools), and you have summaries of the failures, both per test and per node.
|
|
195
|
+
|
|
196
|
+
When your platforms are evolving and scaling, you'll face situations when some tests are expected to fail, but you want to ignore those failures (temporary decomissioning, accumulating technical debt...).
|
|
197
|
+
For those cases Hybrid Platforms Conductor has the concept of expected failures: you can register some tests as expected failures in your platforms' configuration (`hpc_config.rb`) and the tests will still run those tests but ignore the failures.
|
|
198
|
+
However it will report and error if an expected failure is passing successfully: this way it encourages you to keep your list of expected failures clean and minimal.
|
|
199
|
+
|
|
200
|
+
Let's try that: we don't want to bring back web1, so we will add it as an expected failure.
|
|
201
|
+
An expected failure is always accompanied with a descriptive reason for the expected failure, so that anybody running tests understands why this is expected to fail.
|
|
202
|
+
This is done in `hpc_config.rb` using the [`expect_tests_to_fail` config method](/docs/config_dsl.md#expect_tests_to_fail):
|
|
203
|
+
```bash
|
|
204
|
+
cat <<EOF >>hpc_config.rb
|
|
205
|
+
for_nodes('web1') do
|
|
206
|
+
expect_tests_to_fail %i[connection], 'web1 is temporarily down - will bring it up later'
|
|
207
|
+
end
|
|
208
|
+
EOF
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
And now we try again the tests:
|
|
212
|
+
```bash
|
|
213
|
+
./bin/test --all --test connection
|
|
214
|
+
# =>
|
|
215
|
+
# ===== Run 11 connected tests ==== Begin...
|
|
216
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== Begin...
|
|
217
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51240)] ERROR - [ CmdRunner ] - Command 'getent hosts web1.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
218
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51240)] WARN - [ HostIp ] - Host web1.hpc_tutorial.org has no IP.
|
|
219
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51260)] ERROR - [ CmdRunner ] - Command 'getent hosts web3.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
220
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51280)] ERROR - [ CmdRunner ] - Command 'getent hosts web5.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
221
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51280)] WARN - [ HostIp ] - Host web5.hpc_tutorial.org has no IP.
|
|
222
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51260)] WARN - [ HostIp ] - Host web3.hpc_tutorial.org has no IP.
|
|
223
|
+
# [2021-04-29 08:47:52 (PID 1397 / TID 51300)] WARN - [ ActionsExecutor ] - The following nodes have no possible connector to them: web1, web3, web5
|
|
224
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 8 - Total: 8 ]
|
|
225
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== ...End
|
|
226
|
+
#
|
|
227
|
+
# [ 2021-04-29 08:47:54 ] - [ Node local ] - [ connection ] - Start test...
|
|
228
|
+
# [ 2021-04-29 08:47:54 ] - [ Node local ] - [ connection ] - Test finished in 4.9585e-05 seconds.
|
|
229
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web1 ] - [ connection ] - Start test...
|
|
230
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web1 ] - [ connection ] - Test finished in 1.3001e-05 seconds.
|
|
231
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web10 ] - [ connection ] - Start test...
|
|
232
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web10 ] - [ connection ] - Test finished in 5.9226e-05 seconds.
|
|
233
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web2 ] - [ connection ] - Start test...
|
|
234
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web2 ] - [ connection ] - Test finished in 5.7535e-05 seconds.
|
|
235
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web3 ] - [ connection ] - Start test...
|
|
236
|
+
# [2021-04-29 08:47:54 (PID 1397 / TID 51300)] ERROR - [ Connection ] - [ #< Test connection - Node web3 > ] - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
237
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web3 ] - [ connection ] - Test finished in 0.000447342 seconds.
|
|
238
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web4 ] - [ connection ] - Start test...
|
|
239
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web4 ] - [ connection ] - Test finished in 6.0953e-05 seconds.
|
|
240
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web5 ] - [ connection ] - Start test...
|
|
241
|
+
# [2021-04-29 08:47:54 (PID 1397 / TID 51300)] ERROR - [ Connection ] - [ #< Test connection - Node web5 > ] - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
242
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web5 ] - [ connection ] - Test finished in 0.000421333 seconds.
|
|
243
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web6 ] - [ connection ] - Start test...
|
|
244
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web6 ] - [ connection ] - Test finished in 2.5037e-05 seconds.
|
|
245
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web7 ] - [ connection ] - Start test...
|
|
246
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web7 ] - [ connection ] - Test finished in 2.4091e-05 seconds.
|
|
247
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web8 ] - [ connection ] - Start test...
|
|
248
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web8 ] - [ connection ] - Test finished in 1.9962e-05 seconds.
|
|
249
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web9 ] - [ connection ] - Start test...
|
|
250
|
+
# [ 2021-04-29 08:47:54 ] - [ Node web9 ] - [ connection ] - Test finished in 2.5893e-05 seconds.
|
|
251
|
+
# ===== Run 11 connected tests ==== ...End
|
|
252
|
+
#
|
|
253
|
+
# Expected failure for #< Test connection - Node web1 > (web1 is temporarily down - will bring it up later):
|
|
254
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web1
|
|
255
|
+
#
|
|
256
|
+
# ========== Error report of 11 tests run on 11 nodes
|
|
257
|
+
#
|
|
258
|
+
# ======= 0 unexpected failing global tests:
|
|
259
|
+
#
|
|
260
|
+
#
|
|
261
|
+
# ======= 0 unexpected failing platform tests:
|
|
262
|
+
#
|
|
263
|
+
#
|
|
264
|
+
# ======= 1 unexpected failing node tests:
|
|
265
|
+
#
|
|
266
|
+
# ===== connection found 2 nodes having errors:
|
|
267
|
+
# * [ web3 ] - 1 errors:
|
|
268
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
269
|
+
# * [ web5 ] - 1 errors:
|
|
270
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
271
|
+
#
|
|
272
|
+
#
|
|
273
|
+
# ======= 0 unexpected failing platforms:
|
|
274
|
+
#
|
|
275
|
+
#
|
|
276
|
+
# ======= 2 unexpected failing nodes:
|
|
277
|
+
#
|
|
278
|
+
# ===== web3 has 1 failing tests:
|
|
279
|
+
# * [ connection ] - 1 errors:
|
|
280
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web3
|
|
281
|
+
#
|
|
282
|
+
# ===== web5 has 1 failing tests:
|
|
283
|
+
# * [ connection ] - 1 errors:
|
|
284
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web5
|
|
285
|
+
#
|
|
286
|
+
#
|
|
287
|
+
# ========== Stats by nodes list:
|
|
288
|
+
#
|
|
289
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
290
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
291
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
292
|
+
# | No list | 11 | 100 % | 90 % | 72 % | ========================================= |
|
|
293
|
+
# | All | 11 | 100 % | 90 % | 72 % | ========================================= |
|
|
294
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
295
|
+
#
|
|
296
|
+
# ===== Some errors were found. Check output. =====
|
|
297
|
+
|
|
298
|
+
```
|
|
299
|
+
Here we see that 3 nodes failed, but 1 of them is expected to fail, and is not counted in the failures summaries.
|
|
300
|
+
Expected success is now down to 90 %.
|
|
301
|
+
|
|
302
|
+
Let's bring back the 2 nodes that are expected to succeed and check tests again:
|
|
303
|
+
```bash
|
|
304
|
+
docker container start web3 web5
|
|
305
|
+
|
|
306
|
+
./bin/test --all --test connection
|
|
307
|
+
# ===== Run 11 connected tests ==== Begin...
|
|
308
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== Begin...
|
|
309
|
+
# [2021-04-29 09:03:54 (PID 1568 / TID 51240)] ERROR - [ CmdRunner ] - Command 'getent hosts web1.hpc_tutorial.org' returned error code 2 (expected 0).
|
|
310
|
+
# [2021-04-29 09:03:54 (PID 1568 / TID 51240)] WARN - [ HostIp ] - Host web1.hpc_tutorial.org has no IP.
|
|
311
|
+
# [2021-04-29 09:03:54 (PID 1568 / TID 51260)] WARN - [ ActionsExecutor ] - The following nodes have no possible connector to them: web1
|
|
312
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 10 - Total: 10 ]
|
|
313
|
+
# ===== Run test commands on 11 connected nodes (timeout to 25 secs) ==== ...End
|
|
314
|
+
#
|
|
315
|
+
# [ 2021-04-29 09:03:57 ] - [ Node local ] - [ connection ] - Start test...
|
|
316
|
+
# [ 2021-04-29 09:03:57 ] - [ Node local ] - [ connection ] - Test finished in 6.731e-05 seconds.
|
|
317
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web1 ] - [ connection ] - Start test...
|
|
318
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web1 ] - [ connection ] - Test finished in 1.7436e-05 seconds.
|
|
319
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web10 ] - [ connection ] - Start test...
|
|
320
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web10 ] - [ connection ] - Test finished in 4.1223e-05 seconds.
|
|
321
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web2 ] - [ connection ] - Start test...
|
|
322
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web2 ] - [ connection ] - Test finished in 3.9455e-05 seconds.
|
|
323
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web3 ] - [ connection ] - Start test...
|
|
324
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web3 ] - [ connection ] - Test finished in 4.8024e-05 seconds.
|
|
325
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web4 ] - [ connection ] - Start test...
|
|
326
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web4 ] - [ connection ] - Test finished in 3.7838e-05 seconds.
|
|
327
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web5 ] - [ connection ] - Start test...
|
|
328
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web5 ] - [ connection ] - Test finished in 5.2596e-05 seconds.
|
|
329
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web6 ] - [ connection ] - Start test...
|
|
330
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web6 ] - [ connection ] - Test finished in 3.6374e-05 seconds.
|
|
331
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web7 ] - [ connection ] - Start test...
|
|
332
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web7 ] - [ connection ] - Test finished in 4.7406e-05 seconds.
|
|
333
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web8 ] - [ connection ] - Start test...
|
|
334
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web8 ] - [ connection ] - Test finished in 3.3352e-05 seconds.
|
|
335
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web9 ] - [ connection ] - Start test...
|
|
336
|
+
# [ 2021-04-29 09:03:57 ] - [ Node web9 ] - [ connection ] - Test finished in 3.9451e-05 seconds.
|
|
337
|
+
# ===== Run 11 connected tests ==== ...End
|
|
338
|
+
#
|
|
339
|
+
# Expected failure for #< Test connection - Node web1 > (web1 is temporarily down - will bring it up later):
|
|
340
|
+
# - Error while executing tests: no_connector: Unable to get a connector to web1
|
|
341
|
+
#
|
|
342
|
+
# ========== Error report of 11 tests run on 11 nodes
|
|
343
|
+
#
|
|
344
|
+
# ======= 0 unexpected failing global tests:
|
|
345
|
+
#
|
|
346
|
+
#
|
|
347
|
+
# ======= 0 unexpected failing platform tests:
|
|
348
|
+
#
|
|
349
|
+
#
|
|
350
|
+
# ======= 0 unexpected failing node tests:
|
|
351
|
+
#
|
|
352
|
+
#
|
|
353
|
+
# ======= 0 unexpected failing platforms:
|
|
354
|
+
#
|
|
355
|
+
#
|
|
356
|
+
# ======= 0 unexpected failing nodes:
|
|
357
|
+
#
|
|
358
|
+
#
|
|
359
|
+
# ========== Stats by nodes list:
|
|
360
|
+
#
|
|
361
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
362
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
363
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
364
|
+
# | No list | 11 | 100 % | 90 % | 90 % | ========================================= |
|
|
365
|
+
# | All | 11 | 100 % | 90 % | 90 % | ========================================= |
|
|
366
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
367
|
+
#
|
|
368
|
+
# ===== No unexpected errors =====
|
|
369
|
+
|
|
370
|
+
# Check exit code
|
|
371
|
+
echo $?
|
|
372
|
+
# => 0
|
|
373
|
+
```
|
|
374
|
+
We see that now only the expected failure is failing, so success rate equals the expected success rate (90 %), and as a consequence the exit code is 0.
|
|
375
|
+
Everything is running as expected.
|
|
376
|
+
|
|
377
|
+
<a name="nodes-tests"></a>
|
|
378
|
+
## Testing your nodes
|
|
379
|
+
|
|
380
|
+
We just saw how to test connectivity on your nodes.
|
|
381
|
+
Let's go further and test if we can perform configuration checks on your node.
|
|
382
|
+
The [`can_be_checked` test plugin](/docs/plugins/test/can_be_checked.md) does exactly that: it will run a check on the node and check that it succeeds.
|
|
383
|
+
As web1 is supposedly down, we will also filter on which nodes we run this test.
|
|
384
|
+
|
|
385
|
+
```bash
|
|
386
|
+
./bin/test --node /web\[2-5\]/ --test can_be_checked
|
|
387
|
+
# ===== Run 4 check-node tests ==== Begin...
|
|
388
|
+
# ===== Packaging deployment ==== Begin...
|
|
389
|
+
# ===== Packaging deployment ==== ...End
|
|
390
|
+
#
|
|
391
|
+
# ===== Checking on 4 nodes ==== Begin...
|
|
392
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 4 - Total: 4 ]
|
|
393
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 4 - Total: 4 ]
|
|
394
|
+
# ===== Checking on 4 nodes ==== ...End
|
|
395
|
+
#
|
|
396
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web2 ] - [ can_be_checked ] - Start test...
|
|
397
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web2 ] - [ can_be_checked ] - Test finished in 0.000372264 seconds.
|
|
398
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web3 ] - [ can_be_checked ] - Start test...
|
|
399
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web3 ] - [ can_be_checked ] - Test finished in 2.1605e-05 seconds.
|
|
400
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web4 ] - [ can_be_checked ] - Start test...
|
|
401
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web4 ] - [ can_be_checked ] - Test finished in 0.000222523 seconds.
|
|
402
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web5 ] - [ can_be_checked ] - Start test...
|
|
403
|
+
# [ 2021-04-29 09:32:40 ] - [ Node web5 ] - [ can_be_checked ] - Test finished in 1.5508e-05 seconds.
|
|
404
|
+
# ===== Run 4 check-node tests ==== ...End
|
|
405
|
+
#
|
|
406
|
+
#
|
|
407
|
+
# ========== Error report of 4 tests run on 4 nodes
|
|
408
|
+
#
|
|
409
|
+
# ======= 0 unexpected failing global tests:
|
|
410
|
+
#
|
|
411
|
+
#
|
|
412
|
+
# ======= 0 unexpected failing platform tests:
|
|
413
|
+
#
|
|
414
|
+
#
|
|
415
|
+
# ======= 0 unexpected failing node tests:
|
|
416
|
+
#
|
|
417
|
+
#
|
|
418
|
+
# ======= 0 unexpected failing platforms:
|
|
419
|
+
#
|
|
420
|
+
#
|
|
421
|
+
# ======= 0 unexpected failing nodes:
|
|
422
|
+
#
|
|
423
|
+
#
|
|
424
|
+
# ========== Stats by nodes list:
|
|
425
|
+
#
|
|
426
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
427
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
428
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
429
|
+
# | No list | 11 | 36 % | 100 % | 100 % | ========================================= |
|
|
430
|
+
# | All | 11 | 36 % | 100 % | 100 % | ========================================= |
|
|
431
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
432
|
+
#
|
|
433
|
+
# ===== No unexpected errors =====
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
If you want to be sure what is really run by your test, try the `--debug` flag.
|
|
437
|
+
It will get verbose ;-)
|
|
438
|
+
So better to use it when testing 1 node only (after all it is meant for debugging).
|
|
439
|
+
|
|
440
|
+
Here is an highlight of the most interesting parts of debug logs with such a test.
|
|
441
|
+
```bash
|
|
442
|
+
./bin/test --node web2 --test can_be_checked --debug
|
|
443
|
+
# =>
|
|
444
|
+
# [...]
|
|
445
|
+
# [2021-04-29 09:35:01 (PID 1946 / TID 3340)] DEBUG - [ HostIp ] - Get IPs of 1 hosts...
|
|
446
|
+
# [2021-04-29 09:35:01 (PID 1946 / TID 50280)] DEBUG - [ CmdRunner ] - [ Timeout 30 ] - getent hosts web2.hpc_tutorial.org--------------------------------------------------------------------| - [ Initializing... ]
|
|
447
|
+
# 172.17.0.5 web2.hpc_tutorial.org
|
|
448
|
+
# [2021-04-29 09:35:01 (PID 1946 / TID 50280)] DEBUG - [ CmdRunner ] - Finished in 0.236363836 seconds with exit status 0 (success)
|
|
449
|
+
# [...]
|
|
450
|
+
# [2021-04-29 09:35:02 (PID 1946 / TID 50740)] DEBUG - [ CmdRunner ] - /tmp/hpc_ssh/platforms_ssh_5040020210429-1946-19tz6ia/ssh -o BatchMode=yes -o ControlMaster=yes -o ControlPersist=yes hpc.web2 true
|
|
451
|
+
# [2021-04-29 09:35:02 (PID 1946 / TID 50740)] DEBUG - [ CmdRunner ] - Finished in 0.205245432 seconds with exit status 0 (success)
|
|
452
|
+
# Getting SSH ControlMasters [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 1 - Total: 1 ]
|
|
453
|
+
# [2021-04-29 09:35:02 (PID 1946 / TID 50740)] DEBUG - [ Ssh ] - [ ControlMaster - hpc.web2 ] - ControlMaster created
|
|
454
|
+
# [...]
|
|
455
|
+
# [2021-04-29 09:35:04 (PID 1946 / TID 3340)] DEBUG - [ CmdRunner ] - [ Timeout 1799.7834007259999 ] - /tmp/hpc_ssh/platforms_ssh_5040020210429-1946-fbj9g0/ssh hpc.web2 /bin/bash <<'HPC_EOF'
|
|
456
|
+
# echo 'Hello Venus from web2.hpc_tutorial.org (172.17.0.5)' >/tmp/hello_world.txt.wanted
|
|
457
|
+
# echo Diffs on hello_world.txt:
|
|
458
|
+
# if test -f /root/hello_world.txt; then
|
|
459
|
+
# diff /root/hello_world.txt /tmp/hello_world.txt.wanted || true
|
|
460
|
+
# else
|
|
461
|
+
# echo "Create hello_world.txt from scratch"
|
|
462
|
+
# cat /tmp/hello_world.txt.wanted
|
|
463
|
+
# fi
|
|
464
|
+
#
|
|
465
|
+
# HPC_EOF
|
|
466
|
+
# ===== [ web2 / web-hello ] - HPC Service Check ===== Begin
|
|
467
|
+
# ===== [ web2 / web-hello ] - HPC Service Check ===== Begin
|
|
468
|
+
# Diffs on hello_world.txt:
|
|
469
|
+
# [2021-04-29 09:35:04 (PID 1946 / TID 3340)] DEBUG - [ CmdRunner ] - Finished in 0.204418606 seconds with exit status 0 (success)
|
|
470
|
+
# [...]
|
|
471
|
+
# [2021-04-29 09:35:04 (PID 1946 / TID 3340)] DEBUG - [ CmdRunner ] - [ Timeout 1799.5784322069999 ] - /tmp/hpc_ssh/platforms_ssh_5040020210429-1946-fbj9g0/ssh hpc.web2 /bin/bash <<'HPC_EOF'
|
|
472
|
+
# cat <<EOF >/tmp/my-service.conf.wanted
|
|
473
|
+
# service-port: 1202
|
|
474
|
+
# service-timeout: 60
|
|
475
|
+
# service-logs: stdout
|
|
476
|
+
#
|
|
477
|
+
# EOF
|
|
478
|
+
# echo Diffs on my-service.conf:
|
|
479
|
+
# if test -f ~/hpc_tutorial/node/my-service.conf; then
|
|
480
|
+
# diff ~/hpc_tutorial/node/my-service.conf /tmp/my-service.conf.wanted || true
|
|
481
|
+
# else
|
|
482
|
+
# echo "Create file from scratch"
|
|
483
|
+
# cat /tmp/my-service.conf.wanted
|
|
484
|
+
# fi
|
|
485
|
+
#
|
|
486
|
+
# HPC_EOF
|
|
487
|
+
# ===== [ web2 / web-hello ] - HPC Service Check ===== End
|
|
488
|
+
# ===== [ web2 / my-service ] - HPC Service Check ===== Begin
|
|
489
|
+
# ===== [ web2 / web-hello ] - HPC Service Check ===== End
|
|
490
|
+
# ===== [ web2 / my-service ] - HPC Service Check ===== Begin
|
|
491
|
+
# Diffs on my-service.conf:
|
|
492
|
+
# [2021-04-29 09:35:04 (PID 1946 / TID 3340)] DEBUG - [ CmdRunner ] - Finished in 0.20695705 seconds with exit status 0 (success)
|
|
493
|
+
# [...]
|
|
494
|
+
# ========== Stats by nodes list:
|
|
495
|
+
#
|
|
496
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
497
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
498
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
499
|
+
# | No list | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
500
|
+
# | All | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
501
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
502
|
+
#
|
|
503
|
+
# ===== No unexpected errors =====
|
|
504
|
+
```
|
|
505
|
+
You see:
|
|
506
|
+
* how IP address is being discovered by the [`host_ip` CMDB plugin](/docs/plugins/cmdb/host_ip.md),
|
|
507
|
+
* how the [`ssh` connector plugin](/docs/plugins/connector/ssh.md) connects to the node using an SSH ControlMaster,
|
|
508
|
+
* how the configuration checks are being performed using the bash commands we defined in our configuration.
|
|
509
|
+
|
|
510
|
+
So here we are sure that checking nodes is working.
|
|
511
|
+
That's an important part of the stability of your platforms, as it guarantees that you can anytime check for manual divergences of your nodes and re-align them at will.
|
|
512
|
+
Agility derives from such guarantees.
|
|
513
|
+
|
|
514
|
+
Other tests of interest for nodes:
|
|
515
|
+
* [`hostname`](/docs/plugins/test/hostname.md) checks that the hostname reported by the node corresponds to the node's name. Useful to check for wrong IP assignments for example (if the node web1 is assigned the IP of web2, then this check will detect that web1's hostname is web2 and thus will fail).
|
|
516
|
+
* [`local_users`](/docs/plugins/test/local_users.md) checks that only allowed local users have an account on your nodes. This plugin needs configuration from `hpc_config.rb` (see below).
|
|
517
|
+
* [`spectre`](/docs/plugins/test/spectre.md) checks if your node is vulnerable to the [Spectre and Meltdown variants](https://meltdownattack.com/).
|
|
518
|
+
|
|
519
|
+
We will run those tests, but first we must configure the [`local_users`](/docs/plugins/test/local_users.md) test plugin so that it checks some users rules.
|
|
520
|
+
This is done in `hpc_config.rb` by using the `check_local_users_do_exist` and `check_local_users_do_not_exist` config methods:
|
|
521
|
+
```bash
|
|
522
|
+
cat <<EOF >>hpc_config.rb
|
|
523
|
+
# Select only the nodes implementing our web-hello service (that is all the webN nodes)
|
|
524
|
+
for_nodes [{ service: 'web-hello' }] do
|
|
525
|
+
# On our web servers we should have users used by our services
|
|
526
|
+
check_local_users_do_exist %w[sshd www-data]
|
|
527
|
+
# Make sure we have no leftovers of obsolete users
|
|
528
|
+
check_local_users_do_not_exist %w[dangerous_user obsolete_user]
|
|
529
|
+
end
|
|
530
|
+
EOF
|
|
531
|
+
```
|
|
532
|
+
|
|
533
|
+
And now we run all the tests:
|
|
534
|
+
```bash
|
|
535
|
+
./bin/test --node /web\[2-5\]/ --test connection --test can_be_checked --test hostname --test local_users --test spectre
|
|
536
|
+
# =>
|
|
537
|
+
# ===== Run 16 connected tests ==== Begin...
|
|
538
|
+
# ===== Run test commands on 4 connected nodes (timeout to 65 secs) ==== Begin...
|
|
539
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 4 - Total: 4 ]
|
|
540
|
+
# ===== Run test commands on 4 connected nodes (timeout to 65 secs) ==== ...End
|
|
541
|
+
#
|
|
542
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ connection ] - Start test...
|
|
543
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ connection ] - Test finished in 0.000267197 seconds.
|
|
544
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ connection ] - Start test...
|
|
545
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ connection ] - Test finished in 9.9341e-05 seconds.
|
|
546
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ connection ] - Start test...
|
|
547
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ connection ] - Test finished in 0.00021584 seconds.
|
|
548
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ connection ] - Start test...
|
|
549
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ connection ] - Test finished in 0.000134206 seconds.
|
|
550
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ hostname ] - Start test...
|
|
551
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ hostname ] - Test finished in 0.000140542 seconds.
|
|
552
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ hostname ] - Start test...
|
|
553
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ hostname ] - Test finished in 0.000131584 seconds.
|
|
554
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ hostname ] - Start test...
|
|
555
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ hostname ] - Test finished in 0.00012591 seconds.
|
|
556
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ hostname ] - Start test...
|
|
557
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ hostname ] - Test finished in 0.000170961 seconds.
|
|
558
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ local_users ] - Start test...
|
|
559
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ local_users ] - Test finished in 0.00045222 seconds.
|
|
560
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ local_users ] - Start test...
|
|
561
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ local_users ] - Test finished in 0.000246202 seconds.
|
|
562
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ local_users ] - Start test...
|
|
563
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ local_users ] - Test finished in 0.000202314 seconds.
|
|
564
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ local_users ] - Start test...
|
|
565
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ local_users ] - Test finished in 0.000221657 seconds.
|
|
566
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ spectre ] - Start test...
|
|
567
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web2 ] - [ spectre ] - Test finished in 0.000232288 seconds.
|
|
568
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ spectre ] - Start test...
|
|
569
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web3 ] - [ spectre ] - Test finished in 0.000190466 seconds.
|
|
570
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ spectre ] - Start test...
|
|
571
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web4 ] - [ spectre ] - Test finished in 0.00022884 seconds.
|
|
572
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ spectre ] - Start test...
|
|
573
|
+
# [ 2021-04-29 10:14:45 ] - [ Node web5 ] - [ spectre ] - Test finished in 0.000213272 seconds.
|
|
574
|
+
# ===== Run 16 connected tests ==== ...End
|
|
575
|
+
#
|
|
576
|
+
# ===== Run 4 check-node tests ==== Begin...
|
|
577
|
+
# ===== Packaging deployment ==== Begin...
|
|
578
|
+
# ===== Packaging deployment ==== ...End
|
|
579
|
+
#
|
|
580
|
+
# ===== Checking on 4 nodes ==== Begin...
|
|
581
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 4 - Total: 4 ]
|
|
582
|
+
# Executing actions [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 4 - Total: 4 ]
|
|
583
|
+
# ===== Checking on 4 nodes ==== ...End
|
|
584
|
+
#
|
|
585
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web2 ] - [ can_be_checked ] - Start test...
|
|
586
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web2 ] - [ can_be_checked ] - Test finished in 5.6037e-05 seconds.
|
|
587
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web3 ] - [ can_be_checked ] - Start test...
|
|
588
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web3 ] - [ can_be_checked ] - Test finished in 2.1895e-05 seconds.
|
|
589
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web4 ] - [ can_be_checked ] - Start test...
|
|
590
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web4 ] - [ can_be_checked ] - Test finished in 2.2014e-05 seconds.
|
|
591
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web5 ] - [ can_be_checked ] - Start test...
|
|
592
|
+
# [ 2021-04-29 10:14:48 ] - [ Node web5 ] - [ can_be_checked ] - Test finished in 0.000130932 seconds.
|
|
593
|
+
# ===== Run 4 check-node tests ==== ...End
|
|
594
|
+
#
|
|
595
|
+
#
|
|
596
|
+
# ========== Error report of 20 tests run on 4 nodes
|
|
597
|
+
#
|
|
598
|
+
# ======= 0 unexpected failing global tests:
|
|
599
|
+
#
|
|
600
|
+
#
|
|
601
|
+
# ======= 0 unexpected failing platform tests:
|
|
602
|
+
#
|
|
603
|
+
#
|
|
604
|
+
# ======= 0 unexpected failing node tests:
|
|
605
|
+
#
|
|
606
|
+
#
|
|
607
|
+
# ======= 0 unexpected failing platforms:
|
|
608
|
+
#
|
|
609
|
+
#
|
|
610
|
+
# ======= 0 unexpected failing nodes:
|
|
611
|
+
#
|
|
612
|
+
#
|
|
613
|
+
# ========== Stats by nodes list:
|
|
614
|
+
#
|
|
615
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
616
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
617
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
618
|
+
# | No list | 11 | 36 % | 100 % | 100 % | ========================================= |
|
|
619
|
+
# | All | 11 | 36 % | 100 % | 100 % | ========================================= |
|
|
620
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
621
|
+
#
|
|
622
|
+
# ===== No unexpected errors =====
|
|
623
|
+
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
All tests are green!
|
|
627
|
+
|
|
628
|
+
Before going further, let's bring back `web1` online:
|
|
629
|
+
```bash
|
|
630
|
+
docker container start web1
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
We'll see later how easy to add you own test plugins to complement those, but now it's time to see other kind of tests.
|
|
634
|
+
|
|
635
|
+
<a name="platforms-tests"></a>
|
|
636
|
+
## Testing your platforms' configuration
|
|
637
|
+
|
|
638
|
+
As a DevOps team member, you maintain a lot of configuration repositories, used by many tools (Chef, Terraform, Ansible, Puppet...).
|
|
639
|
+
By integrating those repositories into the Hybrid Platforms Conductor's processes, you can then benefit from testing your configuration as well, without relying on your real nodes.
|
|
640
|
+
Those kind of tests validate that your configuration is useable on your nodes without error, and that they are well written.
|
|
641
|
+
They include tests like linters, coding guidelines checks, checking test nodes, deploying test nodes, checking that a deployed configuration does not detect wrong divergences (idempotence)...
|
|
642
|
+
|
|
643
|
+
There are test plugins that will provision test nodes to check and deploy your configuration on them.
|
|
644
|
+
Those test plugins use [provisioner plugins](/docs/plugins.md#provisioner) to provision test nodes.
|
|
645
|
+
By default the [`docker` provisioner plugin](/docs/plugins/provisioner/docker.md) is used, which is very handy in our case as Docker is already setup.
|
|
646
|
+
|
|
647
|
+
An example of such test is the [`linear_strategy` test plugin](/docs/plugins/test/linear_strategy.md) that checks if the git repositories of your platforms are following a [linear git history](https://www.bitsnbites.eu/a-tidy-linear-git-history/), as some teams like to abide to such strategy.
|
|
648
|
+
This test will be executed on the platform repository itself.
|
|
649
|
+
```bash
|
|
650
|
+
./bin/test --test linear_strategy
|
|
651
|
+
# =>
|
|
652
|
+
# ===== Run 1 platform tests ==== Begin...
|
|
653
|
+
# [ 2021-04-29 10:32:23 ] - [ Platform my-service-conf-repo ] - [ linear_strategy ] - Start test...
|
|
654
|
+
# [ 2021-04-29 10:32:23 ] - [ Platform my-service-conf-repo ] - [ linear_strategy ] - Test finished in 0.10645739 seconds.
|
|
655
|
+
# Run platform tests [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 1 - Total: 1 ]
|
|
656
|
+
# ===== Run 1 platform tests ==== ...End
|
|
657
|
+
#
|
|
658
|
+
#
|
|
659
|
+
# ========== Error report of 1 tests run on 0 nodes
|
|
660
|
+
#
|
|
661
|
+
# ======= 0 unexpected failing global tests:
|
|
662
|
+
#
|
|
663
|
+
#
|
|
664
|
+
# ======= 0 unexpected failing platform tests:
|
|
665
|
+
#
|
|
666
|
+
#
|
|
667
|
+
# ======= 0 unexpected failing node tests:
|
|
668
|
+
#
|
|
669
|
+
#
|
|
670
|
+
# ======= 0 unexpected failing platforms:
|
|
671
|
+
#
|
|
672
|
+
#
|
|
673
|
+
# ======= 0 unexpected failing nodes:
|
|
674
|
+
#
|
|
675
|
+
#
|
|
676
|
+
# ========== Stats by nodes list:
|
|
677
|
+
#
|
|
678
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
679
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
680
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
681
|
+
# | No list | 11 | 0 % | | | ========================================= |
|
|
682
|
+
# | All | 11 | 0 % | | | ========================================= |
|
|
683
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
684
|
+
#
|
|
685
|
+
# ===== No unexpected errors =====
|
|
686
|
+
```
|
|
687
|
+
We see here that no node has been tested, but a platform test has been done, and resulted successful.
|
|
688
|
+
|
|
689
|
+
Now let's use a test that will provision a test node to check our configuration on it, without impacting any of our existing (production) nodes.
|
|
690
|
+
We will use the [`check_from_scratch`](/docs/plugins/test/check_from_scratch.md) test that will:
|
|
691
|
+
1. provision a test node,
|
|
692
|
+
2. run a check (the same way [`check-node`](/docs/executables/check-node.md) does) on this test node,
|
|
693
|
+
3. check that the run is successful.
|
|
694
|
+
|
|
695
|
+
In order to provision a test node, Hybrid Platforms Conductor needs to know which OS is supposedly installed on such node.
|
|
696
|
+
This is done by setting the `image` metadata that points to an OS image id for which our configuration (`hpc_config.rb`) will define a Dockerfile provisioning a test image for any node using this OS image id.
|
|
697
|
+
Test images should always have a default `root` account with the `root_pwd` password setup.
|
|
698
|
+
In our case, web services are running on a Debian buster, so let's define the `debian_10` OS image id and associate a Dockerfile to it:
|
|
699
|
+
```bash
|
|
700
|
+
# Define the debian_10 image id
|
|
701
|
+
cat <<EOF >>hpc_config.rb
|
|
702
|
+
os_image :debian_10, "#{hybrid_platforms_dir}/images/debian_10"
|
|
703
|
+
EOF
|
|
704
|
+
|
|
705
|
+
# Create the associated Dockerfile
|
|
706
|
+
mkdir -p images/debian_10
|
|
707
|
+
cat <<EOF >images/debian_10/Dockerfile
|
|
708
|
+
# syntax=docker/dockerfile:1
|
|
709
|
+
FROM debian:buster
|
|
710
|
+
|
|
711
|
+
RUN apt-get update && apt-get install -y openssh-server
|
|
712
|
+
RUN mkdir /var/run/sshd
|
|
713
|
+
# Activate root login with test password
|
|
714
|
+
RUN echo 'root:root_pwd' | chpasswd
|
|
715
|
+
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
|
|
716
|
+
# Speed-up considerably ssh performance and avoid huge lags and timeouts without DNS
|
|
717
|
+
RUN sed -i 's/#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config
|
|
718
|
+
EXPOSE 22
|
|
719
|
+
|
|
720
|
+
CMD ["/usr/sbin/sshd", "-D"]
|
|
721
|
+
EOF
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
Now we add the OS image id to our web nodes:
|
|
725
|
+
```bash
|
|
726
|
+
sed -i '/description: Web service.*/a \ image: debian_10' ~/hpc_tutorial/my-service-conf-repo/inventory.yaml
|
|
727
|
+
|
|
728
|
+
# Check it
|
|
729
|
+
cat ~/hpc_tutorial/my-service-conf-repo/inventory.yaml
|
|
730
|
+
# =>
|
|
731
|
+
# [...]
|
|
732
|
+
# web1:
|
|
733
|
+
# metadata:
|
|
734
|
+
# description: Web service nbr 1
|
|
735
|
+
# image: debian_10
|
|
736
|
+
# hostname: web1.hpc_tutorial.org
|
|
737
|
+
# planet: Mercury
|
|
738
|
+
# service_port: 1201
|
|
739
|
+
# service_timeout: 60
|
|
740
|
+
# services:
|
|
741
|
+
# - web-hello
|
|
742
|
+
# - my-service
|
|
743
|
+
# [...]
|
|
744
|
+
```
|
|
745
|
+
|
|
746
|
+
One last dependency when Hybrid Platform Conductor processes need to authenticate using SSH passwords, the `sshpass` program has to be installed (this makes processes automatizable even when SSH connections require passwords).
|
|
747
|
+
Let's do it:
|
|
748
|
+
```bash
|
|
749
|
+
apt install sshpass
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
Then we are ready to execute the [`check_from_scratch`](/docs/plugins/test/check_from_scratch.md) test.
|
|
753
|
+
You can add log debugs to see more into details the different parts of this process:
|
|
754
|
+
```bash
|
|
755
|
+
./bin/test --node web1 --test check_from_scratch --debug
|
|
756
|
+
# =>
|
|
757
|
+
# [...]
|
|
758
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Create instance...
|
|
759
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ NodesHandler ] - [CMDB Config.others] - Query property image for 1 nodes (web1...) => Found metadata for 0 nodes.
|
|
760
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ NodesHandler ] - [CMDB PlatformHandlers.others] - Query property image for 1 nodes (web1...) => Found metadata for 1 nodes.
|
|
761
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Creating Docker container hpc_docker_container_web1_root_check_from_scratch...
|
|
762
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Wait for instance to be in state running, created, exited (timeout 60)...
|
|
763
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Instance is in state created
|
|
764
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Start instance...
|
|
765
|
+
# [2021-04-29 11:15:28 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Start Docker Container hpc_docker_container_web1_root_check_from_scratch ...
|
|
766
|
+
# [2021-04-29 11:15:29 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Wait for instance to be in state running (timeout 60)...
|
|
767
|
+
# [2021-04-29 11:15:29 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Instance is in state running
|
|
768
|
+
# [...]
|
|
769
|
+
# [2021-04-29 11:15:29 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Set host_ip to 172.17.0.8.
|
|
770
|
+
# [2021-04-29 11:15:29 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Wait for 172.17.0.8:22 to be opened (timeout 60)...
|
|
771
|
+
# [2021-04-29 11:15:29 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - 172.17.0.8:22 is opened.
|
|
772
|
+
# [...]
|
|
773
|
+
# [2021-04-29 11:15:31 (PID 3470 / TID 50940)] DEBUG - [ Ssh ] - [ ControlMaster - hpc.web1 ] - Creating SSH ControlMaster...
|
|
774
|
+
# [2021-04-29 11:15:31 (PID 3470 / TID 50940)] DEBUG - [ CmdRunner ] - /tmp/hpc_ssh/platforms_ssh_5050020210429-3470-75bo6v/ssh -o ControlMaster=yes -o ControlPersist=yes hpc.web1 true
|
|
775
|
+
# [2021-04-29 11:15:31 (PID 3470 / TID 50940)] DEBUG - [ CmdRunner ] - Finished in 0.205104293 seconds with exit status 0 (success)
|
|
776
|
+
# Getting SSH ControlMasters [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 1 - Total: 1 ]
|
|
777
|
+
# [2021-04-29 11:15:31 (PID 3470 / TID 50940)] DEBUG - [ Ssh ] - [ ControlMaster - hpc.web1 ] - ControlMaster created
|
|
778
|
+
# [...]
|
|
779
|
+
# [2021-04-29 11:15:32 (PID 3470 / TID 3340)] DEBUG - [ CmdRunner ] - /tmp/hpc_ssh/platforms_ssh_5050020210429-3470-14247kw/ssh hpc.web1 /bin/bash <<'HPC_EOF'
|
|
780
|
+
# echo 'Hello Mercury from web1.hpc_tutorial.org (172.17.0.8)' >/tmp/hello_world.txt.wanted
|
|
781
|
+
# echo Diffs on hello_world.txt:
|
|
782
|
+
# if test -f /root/hello_world.txt; then
|
|
783
|
+
# diff /root/hello_world.txt /tmp/hello_world.txt.wanted || true
|
|
784
|
+
# else
|
|
785
|
+
# echo "Create hello_world.txt from scratch"
|
|
786
|
+
# cat /tmp/hello_world.txt.wanted
|
|
787
|
+
# fi
|
|
788
|
+
#
|
|
789
|
+
# HPC_EOF
|
|
790
|
+
# ===== [ web1 / web-hello ] - HPC Service Check ===== Begin
|
|
791
|
+
# ===== [ web1 / web-hello ] - HPC Service Check ===== Begin
|
|
792
|
+
# Diffs on hello_world.txt:
|
|
793
|
+
# Create hello_world.txt from scratch
|
|
794
|
+
# Hello Mercury from web1.hpc_tutorial.org (172.17.0.8)
|
|
795
|
+
# [2021-04-29 11:15:32 (PID 3470 / TID 3340)] DEBUG - [ CmdRunner ] - Finished in 0.22602969 seconds with exit status 0 (success)
|
|
796
|
+
# [...]
|
|
797
|
+
# [2021-04-29 11:15:32 (PID 3470 / TID 3340)] DEBUG - [ CmdRunner ] - /tmp/hpc_ssh/platforms_ssh_5050020210429-3470-14247kw/ssh hpc.web1 /bin/bash <<'HPC_EOF'
|
|
798
|
+
# cat <<EOF >/tmp/my-service.conf.wanted
|
|
799
|
+
# service-port: 1201
|
|
800
|
+
# service-timeout: 60
|
|
801
|
+
# service-logs: stdout
|
|
802
|
+
#
|
|
803
|
+
# EOF
|
|
804
|
+
# echo Diffs on my-service.conf:
|
|
805
|
+
# if test -f ~/hpc_tutorial/node/my-service.conf; then
|
|
806
|
+
# diff ~/hpc_tutorial/node/my-service.conf /tmp/my-service.conf.wanted || true
|
|
807
|
+
# else
|
|
808
|
+
# echo "Create file from scratch"
|
|
809
|
+
# cat /tmp/my-service.conf.wanted
|
|
810
|
+
# fi
|
|
811
|
+
#
|
|
812
|
+
# HPC_EOF
|
|
813
|
+
# ===== [ web1 / web-hello ] - HPC Service Check ===== End
|
|
814
|
+
# ===== [ web1 / my-service ] - HPC Service Check ===== Begin
|
|
815
|
+
# ===== [ web1 / web-hello ] - HPC Service Check ===== End
|
|
816
|
+
# ===== [ web1 / my-service ] - HPC Service Check ===== Begin
|
|
817
|
+
# Diffs on my-service.conf:
|
|
818
|
+
# Create file from scratch
|
|
819
|
+
# service-port: 1201
|
|
820
|
+
# service-timeout: 60
|
|
821
|
+
# service-logs: stdout
|
|
822
|
+
#
|
|
823
|
+
# [2021-04-29 11:15:32 (PID 3470 / TID 3340)] DEBUG - [ CmdRunner ] - Finished in 0.23506667 seconds with exit status 0 (success)
|
|
824
|
+
# [...]
|
|
825
|
+
# [2021-04-29 11:15:34 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Stop instance...
|
|
826
|
+
# [2021-04-29 11:15:34 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Stop Docker Container hpc_docker_container_web1_root_check_from_scratch ...
|
|
827
|
+
# [2021-04-29 11:15:34 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Wait for instance to be in state exited (timeout 60)...
|
|
828
|
+
# [2021-04-29 11:15:34 (PID 3470 / TID 3340)] DEBUG - [ Docker ] - [ web1/root_check_from_scratch ] - Instance is in state exited
|
|
829
|
+
# [...]
|
|
830
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
831
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
832
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
833
|
+
# | No list | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
834
|
+
# | All | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
835
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
836
|
+
#
|
|
837
|
+
# ===== No unexpected errors =====
|
|
838
|
+
```
|
|
839
|
+
We see that:
|
|
840
|
+
1. the Docker provisioner provisions a new test container on IP 172.17.0.8 for the web1 node,
|
|
841
|
+
2. the test framework connects to this test instance,
|
|
842
|
+
3. it runs the checks of the `web-hello` service (it reports `Create hello_world.txt from scratch` - normal as the test node is bare),
|
|
843
|
+
4. it runs the checks of the `my-service` service (it reports `Create file from scratch` - normal as the test node is bare),
|
|
844
|
+
5. it stops the Docker container (it would have removed it without the `--debug` switch - debugging keeps test containers accessible for later investigation if needed),
|
|
845
|
+
6. it ends successfully as no error was raised.
|
|
846
|
+
|
|
847
|
+
This test is really validating a lot regarding your configuration already.
|
|
848
|
+
|
|
849
|
+
There is another similar test that test a deployment from scratch of your configuration on test nodes: the [`deploy_from_scratch` test](/docs/plugins/test/deploy_from_scratch.md).
|
|
850
|
+
Let's try it:
|
|
851
|
+
```bash
|
|
852
|
+
./bin/test --node web1 --test deploy_from_scratch
|
|
853
|
+
# =>
|
|
854
|
+
# ===== Run 1 node tests ==== Begin...
|
|
855
|
+
# [ 2021-04-29 14:50:28 ] - [ Node web1 ] - [ deploy_from_scratch ] - Start test...
|
|
856
|
+
# [ 2021-04-29 14:50:35 ] - [ Node web1 ] - [ deploy_from_scratch ] - Test finished in 7.111104222 seconds.
|
|
857
|
+
# Run node tests [100%] - | C| - [ Queue: 0 - Processing: 0 - Done: 1 - Total: 1 ]
|
|
858
|
+
# ===== Run 1 node tests ==== ...End
|
|
859
|
+
#
|
|
860
|
+
#
|
|
861
|
+
# ========== Error report of 1 tests run on 1 nodes
|
|
862
|
+
#
|
|
863
|
+
# ======= 0 unexpected failing global tests:
|
|
864
|
+
#
|
|
865
|
+
#
|
|
866
|
+
# ======= 0 unexpected failing platform tests:
|
|
867
|
+
#
|
|
868
|
+
#
|
|
869
|
+
# ======= 0 unexpected failing node tests:
|
|
870
|
+
#
|
|
871
|
+
#
|
|
872
|
+
# ======= 0 unexpected failing platforms:
|
|
873
|
+
#
|
|
874
|
+
#
|
|
875
|
+
# ======= 0 unexpected failing nodes:
|
|
876
|
+
#
|
|
877
|
+
#
|
|
878
|
+
# ========== Stats by nodes list:
|
|
879
|
+
#
|
|
880
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
881
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
882
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
883
|
+
# | No list | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
884
|
+
# | All | 11 | 9 % | 100 % | 100 % | ========================================= |
|
|
885
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
886
|
+
#
|
|
887
|
+
# ===== No unexpected errors =====
|
|
888
|
+
```
|
|
889
|
+
|
|
890
|
+
Now you have great tools to ensure that your configuration is testable, runs correctly and follows the guidelines you want it to follow.
|
|
891
|
+
|
|
892
|
+
<a name="other-tests"></a>
|
|
893
|
+
## Other kinds of tests
|
|
894
|
+
|
|
895
|
+
Hybrid Platforms Conductor can also execute tests that are not linked particularly to a platform, services or nodes.
|
|
896
|
+
We call them global tests.
|
|
897
|
+
|
|
898
|
+
They are mainly used to:
|
|
899
|
+
* check all the platforms as a whole (for example to detect global IP conflicts),
|
|
900
|
+
* check other components of your platforms, like third-party services you are not responsible for (for example external connectivity to remote repositories),
|
|
901
|
+
* check the environment.
|
|
902
|
+
|
|
903
|
+
One of them is the [`executables` test plugin](/docs/plugins/test/executables.md) that makes sure all [executables](/docs/executables.md) of Hybrid Platforms Conductor are accessible in your environment.
|
|
904
|
+
Another one is the [`private_ips` test plugin](/docs/plugins/test/private_ips.md) that checks for private IPs conflicts among your nodes' metadata.
|
|
905
|
+
|
|
906
|
+
```bash
|
|
907
|
+
./bin/test --test executables --test private_ips
|
|
908
|
+
# =>
|
|
909
|
+
# ===== Run 2 global tests ==== Begin...
|
|
910
|
+
# [ 2021-04-29 15:01:46 ] - [ Global ] - [ executables ] - Start test...
|
|
911
|
+
# [ 2021-04-29 15:02:05 ] - [ Global ] - [ executables ] - Test finished in 19.767006383 seconds.
|
|
912
|
+
# [ 2021-04-29 15:02:05 ] - [ Global ] - [ private_ips ] - Start test...
|
|
913
|
+
# [ 2021-04-29 15:02:05 ] - [ Global ] - [ private_ips ] - Test finished in 0.000962033 seconds.
|
|
914
|
+
# ===== Run 2 global tests ==== ...End
|
|
915
|
+
#
|
|
916
|
+
#
|
|
917
|
+
# ========== Error report of 2 tests run on 0 nodes
|
|
918
|
+
#
|
|
919
|
+
# ======= 0 unexpected failing global tests:
|
|
920
|
+
#
|
|
921
|
+
#
|
|
922
|
+
# ======= 0 unexpected failing platform tests:
|
|
923
|
+
#
|
|
924
|
+
#
|
|
925
|
+
# ======= 0 unexpected failing node tests:
|
|
926
|
+
#
|
|
927
|
+
#
|
|
928
|
+
# ======= 0 unexpected failing platforms:
|
|
929
|
+
#
|
|
930
|
+
#
|
|
931
|
+
# ======= 0 unexpected failing nodes:
|
|
932
|
+
#
|
|
933
|
+
#
|
|
934
|
+
# ========== Stats by nodes list:
|
|
935
|
+
#
|
|
936
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
937
|
+
# | List name | # nodes | % tested | % expected success | % success | [Expected] [Error] [Success] [Non tested] |
|
|
938
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
939
|
+
# | No list | 11 | 0 % | | | ========================================= |
|
|
940
|
+
# | All | 11 | 0 % | | | ========================================= |
|
|
941
|
+
# +-----------+---------+----------+--------------------+-----------+-------------------------------------------+
|
|
942
|
+
#
|
|
943
|
+
# ===== No unexpected errors =====
|
|
944
|
+
```
|
|
945
|
+
|
|
946
|
+
Now you have simple ways (again, 1-liner command lines) to test a lot of your platforms, environment, configuration and nodes!
|
|
947
|
+
Those tools can easily be embedded in a CI/CD.
|
|
948
|
+
|
|
949
|
+
Now is the time to check how you can adapt all those processes to your own specific technologies.
|
|
950
|
+
The goal of Hybrid Platforms Conductor is to be fully adaptable to your environment, and it has to do so easily.
|
|
951
|
+
|
|
952
|
+
**Let's extend its functionnality with your own plugins!**
|
|
953
|
+
|
|
954
|
+
**[Next >> Write your own plugins](/docs/tutorial/05_extend_with_plugins.md)**
|
|
955
|
+
|
|
956
|
+
---
|
|
957
|
+
**<p style="text-align: center;">Tutorial navigation</p>**
|
|
958
|
+
|
|
959
|
+
| <sub>[Introduction](/docs/tutorial.md)</sub> | <sub>[1. Installation and first-time setup](/docs/tutorial/01_installation.md)</sub> | <sub>[2. Deploy and check a first node](/docs/tutorial/02_first_node.md)</sub> | <sub>[3. Scale your processes](/docs/tutorial/03_scale.md)</sub> | <nobr><sub><sub>👇You are here👇</sub></sub></nobr><br><sub>[4. Testing your processes and platforms](/docs/tutorial/04_test.md)</sub> | <sub>[5. Extend Hybrid Platforms Conductor with your own requirements](/docs/tutorial/05_extend_with_plugins.md)</sub> |
|
|
960
|
+
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
|
|
961
|
+
| <sub><sub>**[Use-case](/docs/tutorial.md#use-case)**</sub></sub> | <sub><sub>**[Dependencies installation](/docs/tutorial/01_installation.md#hpc-dependencies)**</sub></sub> | <sub><sub>**[Add your first node and its platform repository](/docs/tutorial/02_first_node.md#add-first-node)**</sub></sub> | <sub><sub>**[Provision our web services platform](/docs/tutorial/03_scale.md#provision)**</sub></sub> | <sub><sub>**[Hello test framework](/docs/tutorial/04_test.md#framework)**</sub></sub> | <sub><sub>**[Create your plugins' repository](/docs/tutorial/05_extend_with_plugins.md#plugins-repo)**</sub></sub> |
|
|
962
|
+
| <sub><sub>**[Prerequisites](/docs/tutorial.md#prerequisites)**</sub></sub> | <sub><sub>**[Our platforms' main repository](/docs/tutorial/01_installation.md#main-repo)**</sub></sub> | <sub><sub>**[Check and deploy services on this node](/docs/tutorial/02_first_node.md#check-deploy)**</sub></sub> | <sub><sub>**[Run commands on our new web services](/docs/tutorial/03_scale.md#run)**</sub></sub> | <sub><sub>**[Testing your nodes](/docs/tutorial/04_test.md#nodes-tests)**</sub></sub> | <sub><sub>**[Your own platform handler](/docs/tutorial/05_extend_with_plugins.md#platform-handler)**</sub></sub> |
|
|
963
|
+
| <sub><sub>**[Tutorial setup](/docs/tutorial.md#tutorial-setup)**</sub></sub> | | <sub><sub>**[Updating the configuration](/docs/tutorial/02_first_node.md#update)**</sub></sub> | <sub><sub>**[Check and deploy our web services on several nodes at once](/docs/tutorial/03_scale.md#check-deploy)**</sub></sub> | <sub><sub>**[Testing your platforms' configuration](/docs/tutorial/04_test.md#platforms-tests)**</sub></sub> | <sub><sub>**[Write your own tests](/docs/tutorial/05_extend_with_plugins.md#test)**</sub></sub> |
|
|
964
|
+
| | | | | <sub><sub>**[Other kinds of tests](/docs/tutorial/04_test.md#other-tests)**</sub></sub> | <sub><sub>**[Enough of stdout, we want to report to other tools](/docs/tutorial/05_extend_with_plugins.md#report)**</sub></sub> |
|
|
965
|
+
| | | | | | <sub><sub>**[What next?](/docs/tutorial/05_extend_with_plugins.md#what-next)**</sub></sub> |
|