hybrid_platforms_conductor 32.11.2 → 32.13.3

data/docs/executables/report.md

@@ -0,0 +1,159 @@
+# `report`
+
+The `report` executable will produce some reports for a list of nodes, using a given format and locale.
+This executable is using [report plugins](../plugins.md#report) so that the tool is easily extensible to any format or locale needed (think of CSV, Excel, DNS configuration files, other configuration management tools...).
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/report.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/report
+participant Report as Report
+participant CMDB as CMDB
+
+Main->>+Report: Produce a report for selected nodes
+Report->>+CMDB: Get metadata to be reported for nodes
+CMDB-->>-Report: Nodes' metadata
+Report->>-Main: Report produced
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/report [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+    -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Reports handler options:
+    -c, --locale LOCALE_CODE         Generate the report in the given format. Possible codes are formats specific. [confluence: en], [stdout: en], [mediawiki: en]
+    -f, --format FORMAT              Generate the report in the given format. Possible formats are confluence, mediawiki, stdout. Default: stdout.
+```
+
+## Examples
+
+```bash
+# Output all nodes info using mediawiki format
+./bin/report --format mediawiki
+
+# Output all nodes info using mediawiki format in en locale
+./bin/report --format mediawiki --locale en
+
+# Output all nodes containing /xae/ in their names using mediawiki format
+./bin/report --node /xae/ --format mediawiki
+```
+
+Example of output:
+```
+=> ./bin/report --format mediawiki
+Back to the [[Hadoop]] / [[Impala]] / [[XAE_Network_Topology]] portal pages
+
+This page has been generated using <code>./helpers/report --format mediawiki</code> on 2018-03-26 08:58:55 UTC.
+
+= Physical nodes =
+
+== Independent nodes ==
+
+=== 172.16.0/24 ===
+
+* '''WinNode''' - 172.16.0.140 - AD of QlikSense Server (primary AD of Production) - WinNode
+: Handled by Chef: No
+: Server type: Virtual Machine on node456.my_domain.com.
+
+
+* '''WinNode''' - 172.16.0.141 - AD of QlikSense Server (secondary AD of Production) - WinNode
+: Handled by Chef: No
+: Server type: Virtual Machine on node456.my_domain.com.
+
+
+* '''node237''' - 172.16.0.9 - Gateway to Dedicated Cloud (Former Tableau 8) - node237.my_domain.com
+: Handled by Chef: No
+: Location: RBX
+: OS: Windows Server 2008
+: XAE IP: 192.168.255.159
+: Public IPs: 
+::* 192.168.255.159
+
+[...]
+
+=== 172.16.139/24 ===
+
+* '''node12lnx09''' - 172.16.139.98 - Data Processing (Gurobi, GPU, RStudio)
+: OS: Debian 7
+
+
+=== 172.30.14/24 ===
+
+* '''node''' - 172.30.14.226 - ADP gateway in my_platform IaaS
+: OS: RHEL 7 ADP Stadard
+
+
+=== 192.168.0/24 ===
+
+* '''project-pinger''' - 192.168.0.2 - Product availability tester
+: Connection settings: 
+::* ip: 192.168.0.77
+: Direct deployment: Yes
+: Public IPs: 
+::* 192.168.0.77
+
+
+Back to the [[Hadoop]] / [[Impala]] / [[XAE_Network_Topology]] portal pages
+
+[[Category:My Project]]
+[[Category:Hadoop]]
+[[Category:NoSQL]]
+[[Category:Hosting]]
+[[Category:XAE]]
+[[Category:Server]]
+[[Category:Configuration]]
+[[Category:Chef]]
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/run.md

@@ -0,0 +1,126 @@
+# `run`
+
+The `run` executable will run any Bash command (or interactive session) remotely on a node (or list of nodes).
+It will use connectors (SSH...) to do so.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/run.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/run --node my_node --command "echo Hello"
+participant Connector as Connector
+participant Node as Provisioned node (my_node)
+
+Main->>+Connector: Connect to my_node to execute actions
+Connector->>+Node: Execute bash command on my_node
+Node-->>-Connector: stdout of bash execution
+Connector-->>-Main: Close connection
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/run [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+    -c, --command CMD                Command to execute (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
+    -f, --commands-file FILE_NAME    Execute commands taken from a file (can't be used with --interactive) (can be used several times, commands will be executed sequentially)
+    -i, --interactive                Run an interactive SSH session instead of executing a command (can't be used with --command or --commands-file)
+    -p, --parallel                   Execute the commands in parallel (put the standard output in files <hybrid-platforms-dir>/run_logs/*.stdout)
+    -t, --timeout SECS               Timeout in seconds to wait for each command (defaults to no timeout)
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Nodes selection options:
+    -a, --all-nodes                  Select all nodes
+    -b, --nodes-platform PLATFORM    Select nodes belonging to a given platform name. Available platforms are: ansible-repo, chef-repo (can be used several times)
+    -l, --nodes-list LIST            Select nodes defined in a nodes list (can be used several times)
+    -n, --node NODE                  Select a specific node. Can be a regular expression to select several nodes if used with enclosing "/" characters. (can be used several times).
+    -r, --nodes-service SERVICE      Select nodes implementing a given service (can be used several times)
+        --nodes-git-impact GIT_IMPACT
+                                     Select nodes impacted by a git diff from a platform (can be used several times).
+                                     GIT_IMPACT has the format PLATFORM:FROM_COMMIT:TO_COMMIT:FLAGS
+                                     * PLATFORM: Name of the platform to check git diff from. Available platforms are: ansible-repo, chef-repo
+                                     * FROM_COMMIT: Commit ID or refspec from which we perform the diff. If ommitted, defaults to master
+                                     * TO_COMMIT: Commit ID ot refspec to which we perform the diff. If ommitted, defaults to the currently checked-out files
+                                     * FLAGS: Extra comma-separated flags. The following flags are supported:
+                                       - min: If specified then each impacted service will select only 1 node implementing this service. If not specified then all nodes implementing the impacted services will be selected.
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Actions Executor options:
+    -m, --max-threads NBR            Set the number of threads to use for concurrent queries (defaults to 16)
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+    -j, --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+```
+
+## Examples
+
+```bash
+# Display the possible nodes we can run commands on (also outputs the possible nodes lists, services...)
+./bin/run --show-nodes
+
+# Run an interactive SSH session on node23hst-nn1
+./bin/run --node node23hst-nn1 --interactive
+
+# Run the hostname command on node23hst-nn1
+./bin/run --node node23hst-nn1 --command hostname
+
+# Run the hostname and ls commands on node23hst-nn1
+./bin/run --node node23hst-nn1 --command hostname --command ls
+
+# Run a list of commands (taken from the file cmds.list) on node23hst-nn1
+./bin/run --node node23hst-nn1 --commands-file cmds.list
+
+# Run a list of commands (taken from the file cmds.list) and the hostname command on node23hst-nn1
+./bin/run --node node23hst-nn1 --commands-file cmds.list --command hostname
+
+# Run the hostname command on node23hst-nn1 with a timeout of 5 seconds that would interrupt the command if it does not end before
+./bin/run --node node23hst-nn1 --command hostname --timeout 5
+
+# Run the hostname command on all nodes containing xae in parallel (and send each standard output in log files in ./run_logs/*.stdout)
+./bin/run --node /xae/ --command hostname --parallel
+```
+
+Example of output:
+```
+=> ./bin/run --node node12had01 --command hostname
+node12host.site.my_company.net
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/setup.md

@@ -0,0 +1,92 @@
+# `setup`
+
+The `setup` executable installs all dependencies needed for a platform to be operated by Hybrid Platforms Conductor.
+It is intended to be run only for the initial setup or when such dependencies change (for example if a `Gemfile` of a `chef` platform changes).
+
+***This executable is still in alpha version: not properly tested, no clear process, no stable interface. Pending [this ticket](https://github.com/sweet-delights/hybrid-platforms-conductor/issues/45).***
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/setup.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/setup
+participant PlatformHandler as Platform Handler
+
+Main->>+PlatformHandler: Install dependencies for each platform
+PlatformHandler-->>-Main: Dependencies installed
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/setup [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+```
+
+## Examples
+
+```bash
+# Setup all declared platforms
+./bin/setup
+```
+
+Here is an example of output:
+```
+=> ./bin/setup
+cd ../chef-repo && rm -rf Gemfile.lock vendor && bundle install --path vendor/bundle --binstubs
+Fetching gem metadata from http://rubygems.org/........
+Fetching gem metadata from http://rubygems.org/.
+Resolving dependencies....
+Fetching rake 12.3.1
+Installing rake 12.3.1
+[...]
+Bundle complete! 12 Gemfile dependencies, 101 gems now installed.
+Bundled gems are installed into `./vendor/bundle`
+Post-install message from minitar:
+The `minitar` executable is no longer bundled with `minitar`. If you are
+expecting this executable, make sure you also install `minitar-cli`.
+cd ./cloned_platforms/xae-chef-repo && rm -rf Gemfile.lock vendor && bundle install --path vendor/bundle --binstubs
+Fetching gem metadata from http://rubygems.org/........
+Fetching gem metadata from http://rubygems.org/.
+Resolving dependencies....
+Fetching rake 12.3.1
+Installing rake 12.3.1
+[...]
+Bundle complete! 9 Gemfile dependencies, 98 gems now installed.
+Bundled gems are installed into `./vendor/bundle`
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/executables/ssh_config.md

@@ -0,0 +1,151 @@
+# `ssh_config`
+
+The `ssh_config` executable will output (in standard output) an SSH config file ready to be used to address any node accessible using the [`ssh` connector](../plugins/connector/ssh.md).
+The configuration also includes any proxy configuration needed.
+The generated file can also be tuned by specifying the gateway user names to be used, and a path to a different ssh executable.
+
+This executable is also used internally by other tools of Hybrid Platforms Conductor to prepare the SSH environment before executing SSH commands, so it's a good way to see and debug the SSH configuration used internally by the Hybrid Platform Conductor's processes.
+
+## Process
+
+<!-- Mermaid generator - Section start -->
+![Mermaid diagram](/docs/gen/mermaid/docs/executables/ssh_config.md-0.png)
+<details>
+<summary><sub><sup>Diagram code</sup></sub></summary>
+
+```mermaid
+sequenceDiagram
+participant Main as ./bin/ssh_config
+participant CMDB as CMDB
+participant Connector as Connector
+participant ConnectorSSH as Connector SSH
+
+Main->>+CMDB: Get metadata of all nodes
+CMDB->>-Main: Metadata
+Main->>+Connector: Select nodes that are SSH-connectable from their metadata
+Connector->>-Main: List of SSH-connectable nodes
+Main->>+ConnectorSSH: Get the SSH config for selected nodes
+ConnectorSSH-->>-Main: Display SSH config
+```
+</details>
+<!-- Mermaid generator - Section end -->
+
+## Usage
+
+```
+Usage: ./bin/ssh_config [options]
+
+Main options:
+    -d, --debug                      Activate debug mode
+    -h, --help                       Display help and exit
+    -x, --ssh-exec FILE_PATH         Path to the SSH executable to be used. Useful to give default options (especially with GIT_SSH). Defaults to ssh.
+
+Nodes handler options:
+    -o, --show-nodes                 Display the list of possible nodes and exit
+
+Command runner options:
+    -s, --show-commands              Display the commands that would be run instead of running them
+
+Connector ssh options:
+    -g, --ssh-gateway-user USER      Name of the gateway user to be used by the gateways. Can also be set from environment variable hpc_ssh_gateway_user. Defaults to ubradm.
+    -j, --ssh-no-control-master      If used, don't create SSH control masters for connections.
+    -q, --ssh-no-host-key-checking   If used, don't check for SSH host keys.
+    -u, --ssh-user USER              Name of user to be used in SSH connections (defaults to hpc_ssh_user or USER environment variables)
+    -w, --password                   If used, then expect SSH connections to ask for a password.
+    -y GATEWAYS_CONF,                Name of the gateways configuration to be used. Can also be set from environment variable hpc_ssh_gateways_conf.
+        --ssh-gateways-conf
+```
+
+## Examples
+
+```bash
+# Dump in stdout
+./bin/ssh_config
+
+# Use it to overwrite directly the SSH config file
+./bin/ssh_config >~/.ssh/config ; chmod 600 ~/.ssh/config
+
+# Use it to generate a separate included config file (for OpenSSH version >= 7.3p1)
+# Need to add "Include platforms_config" in the existing ~/.ssh/config file.
+./bin/ssh_config >~/.ssh/platforms_config
+
+# Dump in stdout, using hadcli as gateway user
+./bin/ssh_config --ssh-gateway-user hadcli
+
+# Dump in stdout, using /my/other/ssh instead of ssh
+./bin/ssh_config --ssh-exec /my/other/ssh
+
+# Dump in stdout, using the madrid SSH gateways configuration
+./bin/ssh_config --ssh-gateways-conf madrid
+```
+
+Example of output:
+```
+=> ./bin/ssh_config
+
+############
+# GATEWAYS #
+############
+
+# Gateway Nice (when connecting from other sites)
+Host my.gateway.com
+  User sitegw
+  Hostname node12hst-nn5.site.my_company.net
+
+# DMZ Gateway
+Host gw.dmz.be
+  HostName dmz.my_domain.com
+  ProxyCommand ssh -q -W %h:%p my.gateway.com
+
+# Data Gateway
+Host gw.data.be
+  HostName fr-had.my_domain.com
+  ProxyCommand ssh -q -W %h:%p datagw@gw.dmz.be
+
+
+#############
+# ENDPOINTS #
+#############
+
+Host *
+  User a_usernme
+  # Default control socket path to be used when multiplexing SSH connections
+  ControlPath /tmp/actions_executor_mux_%h_%p_%r
+  PubkeyAcceptedKeyTypes +ssh-dss
+
+# AD_Win2012_NP0 - 172.16.16.105 - ./cloned_platforms/xae-chef-repo - AD of QlikSense Server (primary AD of Non-production) - AD_Win2012_NP0
+Host hpc.172.16.16.105 hpc.16.105 hpc.AD_Win2012_NP0
+  Hostname 172.16.16.105
+  ProxyCommand ssh -q -W %h:%p ubradm@gw.dmz.be
+
+[...]
+
+# xaetitanuwsd01 - 172.16.16.89 - ./cloned_platforms/xae-chef-repo - Traffic Analytics WS (UAT/jessie)
+Host hpc.172.16.16.89 hpc.16.89 hpc.xaetitanuwsd01
+  Hostname 172.16.16.89
+  ProxyCommand ssh -q -W %h:%p ubradm@gw.dmz.be
+
+# project-pinger - 192.168.0.2 - ../chef-repo - Product availability tester
+Host hpc.192.168.0.2 hpc.project-pinger
+  Hostname 192.168.0.77
+
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None