hybrid_platforms_conductor 32.11.2 → 32.13.3

data/docs/plugins/cmdb/config.md

@@ -0,0 +1,46 @@
+# CMDB plugin: `config`
+
+The `config` CMDB plugin sets metadata from the Hybrid Platforms Conductor's configuration.
+
+## Metadata set by this plugin
+
+| Metadata | Type | Dependent metadata | Usage
+| --- | --- | --- |
+| * | Any | None | Any metadata can be set through the `set_metadata` config DSL method |
+
+## Config DSL extension
+
+### `set_metadata`
+
+Set metadata for a set of nodes.
+It takes the metadata as a `Hash<Symbol,Object>`.
+
+Example:
+```ruby
+# Make sure all test nodes have the environment set correctly and run under CentOS 7.
+for_nodes('/tst.*/') do
+  set_metadata(
+    environment: 'test',
+    image: 'centos_7'
+  )
+end
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/cmdb/host_ip.md

@@ -0,0 +1,33 @@
+# CMDB plugin: `host_ip`
+
+The `host_ip` CMDB plugin discovers the `host_ip` metadata by querying DNS records using the `hostname` metadata if it is set.
+
+## Metadata set by this plugin
+
+| Metadata | Type | Dependent metadata | Usage
+| --- | --- | --- | --- |
+| `host_ip` | `String` | `hostname` | The node's IP address as returned by a DNS lookup using the `hostname` metadata |
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `hostname` | `String` | Used to query the IP from DNS records |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+* `getent`: Used to query DNS.

data/docs/plugins/cmdb/host_keys.md

@@ -0,0 +1,33 @@
+# CMDB plugin: `host_keys`
+
+The `host_keys` CMDB plugin discovers the SSH host keys based the IP or hostname of nodes (using either `host_ip` or `hostname` metadata).
+
+## Metadata set by this plugin
+
+| Metadata | Type | Dependent metadata | Usage
+| --- | --- | --- | --- |
+| `host_keys` | `Array<String>` | `hostname`, `host_ip` | The list of SSH host keys discovered using `ssh-keyscan` |
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `hostname` | `String` | Used to query the IP from DNS records |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+* `ssh-keyscan`: Used to discover the host keys.

data/docs/plugins/cmdb/platform_handlers.md

@@ -0,0 +1,33 @@
+# CMDB plugin: `platform_handlers`
+
+The `platform_handlers` CMDB plugin sets metadata by querying [`platform_handler`](../platform_handler) plugins.
+
+## Metadata set by this plugin
+
+| Metadata | Type | Dependent metadata | Usage
+| --- | --- | --- |
+| `services` | `Array<String>` | None | List of services that should be present in a node |
+| * | Any | None | Any metadata can be set by the platform handlers |
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/connector/local.md

@@ -0,0 +1,28 @@
+# Connector plugin: `local`
+
+The `local` connector plugin allows remote actions to be executed on localhost, in a dedicated workspace inside `/tmp/hpc_local_workspaces`.
+This connector should only be used for nodes deploying services on localhost.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `local_node` | `Boolean` | If set to true, then consider the node to be handled by this connector |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/connector/ssh.md

@@ -0,0 +1,95 @@
+# Connector plugin: `ssh`
+
+The `ssh` connector plugin allows remote actions to be executed on nodes having an SSH access.
+It supports different ways to retrieve the SSH connection details, from configuration, metadata and environment variables.
+
+## Config DSL extension
+
+### `gateway`
+
+Declare a new SSH gateway, with 2 parameters: its name (as a Symbol) and its SSH configuration (as a String).
+This is used directly in any SSH configuration file used to connect to nodes.
+Any node can then reference this gateway by using the `gateway` metadata.
+
+The gateway definition is an ERB template can use the following variables:
+* `@user` (String): The SSH user name
+* `@ssh_exec` (String): Path to the SSH executable to be used. Always use this variable instead of `ssh` (for example in proxy commands) as the connector might use a different ssh executable to encapsulate the configuration without polluting the system ssh.
+
+Examples:
+```ruby
+gateway :prod_gw, <<~EOS
+Host prod.gateway.com
+  User gateway_<%= @user %>
+  ProxyCommand <%= @ssh_exec %> -q -W %h:%p all.gateway.com
+EOS
+```
+
+### `transform_ssh_connection`
+
+Provide a code block transforing the SSH connection details for nodes.
+The code block has the following signature:
+
+*Parameters*:
+* **node** (`String`): Node for which we transform the SSH connection
+* **connection** (`String` or `nil`): The connection host or IP, or nil if none
+* **connection_user** (`String`): The connection user
+* **gateway** (`String` or `nil`): The gateway name, or nil if none
+* **gateway_user** (`String` or `nil`): The gateway user, or nil if none
+*Result*:
+* `String`: The transformed connection host or IP, or nil if none
+* `String`: The transformed connection user
+* `String` or `nil`: The transformed gateway name, or nil if none
+* `String` or `nil`: The transformed gateway user, or nil if none
+
+Examples:
+```ruby
+# Test nodes have to use the test gateway with hostname in the gateway user name
+for_nodes('/tst/') do
+  transform_ssh_connection do |node, connection, connection_user, gateway, gateway_user|
+    [
+      'test_gateway.tst.my_domain.com',
+      "#{connection_user}@#{connection}"
+    ]
+  end
+end
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `description` | `String` | Nodes description added in generated SSH configs |
+| `gateway_user` | `String` | Name of the gateway user to be used in the SSH config used by the connector. |
+| `gateway` | `String` | Name of the gateway to be used in the SSH config used by the connector. |
+| `host_ip` | `String` | The node's IP address to connect to using SSH. If this metadata is not set, then the node is considered as not connectable using the `ssh` connector. |
+| `host_keys` | `Array<String>` | The node's host keys used to generate a `known_hosts` file with those to avoid user confirmations when connecting. |
+| `hostname` | `String` | Host name used to connect in case no IP address can be found in metadata. |
+| `private_ips` | `Array<String>` | IP list to connect in case `host_ip` is not defined in metadata. |
+| `ssh_session_exec` | `String` | If set to the string `false`, then consider that the node does not have any SSH SessionExec capabilities. This will make sure that remote command executions is done using stdin piping on interactive sessions instead of SSH commands execution. |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+| `hpc_interactive` | If set to `false`, then interactive SSH sessions will fail with an error. Useful to not try interactive mode in non-interactive environments like CI/CD. |
+| `hpc_ssh_gateway_user` | Default gateway user to be used (can be overriden by the `gateway_user` metadata). |
+| `hpc_ssh_gateways_conf` | Gateways configuration name to be used in the SSH configuration. The name should match one of the names declared in the configuration (see the `gateway` config DSL extension). |
+| `hpc_ssh_user` | Name of the user to be used in SSH connections. |
+| `USER` | Name of the user to be used in SSH connections (only used if the env variable `hpc_ssh_user` is not set). |
+
+## External tools dependencies
+
+* `cat`: Used to pipe commands on SSH connections not having SessionExec capabilities.
+* `env`: Used to set shebangs in bash scripts.
+* `gzip`: Used to transfer files on SSH connections having SessionExec capabilities.
+* `scp`: Used to transfer files on SSH connections not having SessionExec capabilities.
+* `ssh`: Used to run SSH commands or interactive sessions.
+* `sshpass`: Used when the SSH connections is done using a password that needs to be set automatically (using the `passwords` accessor from the connector).
+* `tar`: Used to transfer files on SSH connections having SessionExec capabilities.
+* `whoami`: Used to get ssh user name when environment variables `hpc_ssh_user` and `USER` are not set.
+* `xterm`: Used to initiate an interactive ControlMaster on SSH connections not having SessionExec capabilities.

data/docs/plugins/platform_handler/yaml_inventory.md

@@ -0,0 +1,105 @@
+# PlatformHandler plugin: `yaml_inventory`
+
+The `yaml_inventory` platform handler is just a minimalistic handler supporting an inventory definition from a file named `inventory.yaml`, and services to be deployed using simple Ruby methods defined in files named `service_<service_name>.rb`.
+It provides an out-of-the-box solution that can be used to define an inventory in case there are no existing repositories to start with.
+
+## Inventory
+
+The structure of the `inventory.yaml` file is a hash of `<node_name> => <node_info_hash>`, with `<node_info_hash>` having the following properties:
+* **metadata** (`Hash<String,Object>`): The node's metadata
+* **services** (`Array<String>`): The node's services
+
+Example:
+```yaml
+---
+prod_node:
+  metadata:
+    environment: production
+    image: centos_7
+  services:
+    - firewall
+
+test_node:
+  metadata:
+    environment: test
+    image: centos_7
+  services:
+    - web_frontend
+    - firewall
+```
+
+## Services
+
+Each file named `service_<service_name>.rb` defines 2 methods: `check` and `deploy` that return [actions](../../plugins.md#action) to execute in order to respectively check and deploy the service named `<service_name>` on a node.
+Those methods have both the following signature:
+* Parameters:
+  * **node** (`String`): The node for which we check/deploy the service.
+* Result:
+  * `Array< Hash<Symbol,Object> >`: The list of actions to execute to check/deploy the service on the node.
+The code of those methods can use standard logging and the following API components:
+* **`@config`**: The Config API.
+* **`@nodes_handler`**: The NodesHandler API.
+* **`@cmd_runner`**: The CmdRunner API.
+* **`@platform_handler`**: The platform handler for which this service is being checked/deployed.
+
+Example of a service file checking for a file's presence on the remote node:
+```ruby
+# Check if the service is installed on a node
+#
+# Parameters::
+# * *node* (String): Node for which we check the service installation
+# Result::
+# * Array< Hash<Symbol,Object> >: List of actions to execute to check the service
+def check(node)
+  [
+    {
+      remote_bash: <<~EOS
+        if test -f ~/my-file.txt; then
+          echo "[ SUCCESS ] - File exists."
+        else
+          echo "[ FAILURE ] - File does not exist."
+        fi
+      EOS
+    }
+  ]
+end
+
+# Deploy the on a node
+#
+# Parameters::
+# * *node* (String): Node for which we deploy the service
+# Result::
+# * Array< Hash<Symbol,Object> >: List of actions to execute to deploy the service
+def deploy(node)
+  [
+    {
+      remote_bash: <<~EOS
+        touch ~/my-file.txt
+      EOS
+    }
+  ]
+end
+```
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/provisioner/docker.md

@@ -0,0 +1,27 @@
+# Provisioner plugin: `docker`
+
+The `docker` provisioner plugin is handling a local Docker installation to provision nodes.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `image` | `String` | The name of the OS image to be used. The [configuration](../../config_dsl.md) should define the image and point it to a directory containing a `Dockerfile` that will be used to provision the Docker container. |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+* Docker local installation.

data/docs/plugins/provisioner/podman.md

@@ -0,0 +1,27 @@
+# Provisioner plugin: `podman`
+
+The `podman` provisioner plugin is handling a local Podman installation to provision nodes.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `image` | `String` | The name of the OS image to be used. The [configuration](../../config_dsl.md) should define the image and point it to a directory containing a `Dockerfile` that will be used to provision the Podman container. |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+* `podman`: Used to handle Podman containers.

data/docs/plugins/provisioner/proxmox.md

@@ -0,0 +1,115 @@
+# Provisioner plugin: `proxmox`
+
+The `proxmox` provisioner plugin is using a Proxmox cluster to provision nodes.
+
+## Config DSL extension
+
+### `proxmox`
+
+Define a Proxmox cluster configuration.
+
+It takes `Hash<Symbol,Object>` as parameter, defining the following properties:
+* **api_url** (`String`): The Proxmox API URL
+* **api_max_retries** (`Integer`): Max number of API retries
+* **api_wait_between_retries_secs** (`Integer`): Number of seconds to wait between API retries
+* **sync_node** (`String`): Node to be used to synchronize Proxmox resources acquisition
+* **test_config** (`Hash<Symbol,Object>`): The test configuration, as a hash of properties:
+  * **pve_nodes** (`Array<String>`): List of PVE nodes allowed to spawn new containers [default: all]
+  * **vm_ips_list** (`Array<String>`): The list of IPs that are available for the Proxomx containers.
+  * **vm_ids_range** (`[Integer, Integer`]): Minimum and maximum reservable VM ID
+  * **coeff_ram_consumption** (`Integer`): Importance coefficient to assign to the RAM consumption when selecting available PVE nodes
+  * **coeff_disk_consumption** (`Integer`): Importance coefficient to assign to the disk consumption when selecting available PVE nodes
+  * **expiration_period_secs** (`Integer`): Number of seconds defining the expiration period
+  * **expire_stopped_vm_timeout_secs** (`Integer`): Number of seconds before defining stopped VMs as expired
+  * **limits** (`Hash`): Limits to be taken into account while reserving resources. Each property is optional and no property means no limit.
+    * **nbr_vms_max** (`Integer`): Max number of VMs we can reserve.
+    * **cpu_loads_thresholds** (`[Float, Float, Float]`): CPU load thresholds from which a PVE node should not be used (as soon as 1 of the value is greater than those thresholds, discard the node).
+    * **ram_percent_used_max** (`Float`): Max percentage (between 0 and 1) of RAM that can be reserved on a PVE node.
+    * **disk_percent_used_max** (`Float`): Max percentage (between 0 and 1) of disk that can be reserved on a PVE node.
+* **vm_config** (`Hash<Symbol,Object>`): Extra configuration of a created container:
+  * **vm_dns_servers** (`Array<String>`): List of DNS servers
+  * **vm_search_domain** (`String`): Default search domain
+  * **vm_gateway** (`String`): Gateway hostname or IP
+* **default_timeout** (`Integer`): The default timeout to be applied when starting/stopping containers [default: 3600].
+
+Example:
+```ruby
+proxmox(
+  # Entry point API
+  api_url: 'https://my_proxmox.my_domain.com:8006',
+  # This node is used to synchronize all VMs operations
+  sync_node: 'pve_node_1',
+  # Retry in case of API failures
+  api_max_retries: 10,
+  api_wait_between_retries_secs: 20,
+  # When provisioning test containers, make sure we limit their config
+  test_config: {
+    pve_nodes: %w[
+      pve_node_1
+      pve_node_2
+      pve_node_3
+    ],
+    vm_ips_list: %w[
+      172.16.110.1
+      172.16.110.2
+      172.16.110.3
+      172.16.110.4
+      172.16.110.5
+    ],
+    vm_ids_range: [1000, 1100],
+    # Specify limits above which test containers should not be provisioned to not alter other important VMs
+    coeff_ram_consumption: 10,
+    coeff_disk_consumption: 1,
+    limits: {
+      nbr_vms_max: 20,
+      cpu_loads_thresholds: [10, 10, 10],
+      ram_percent_used_max: 0.75,
+      disk_percent_used_max: 0.75
+    },
+    # Test containers are considered expired after 1 day, or when they are stopped for more than 30 secs
+    expiration_period_secs: 24 * 60 * 60,
+    expire_stopped_vm_timeout_secs: 30
+  },
+  # Any provisioned container should have some common network config
+  vm_config: {
+    vm_dns_servers: ['172.16.110.100', '172.16.110.101'],
+    vm_search_domain: 'my_domain.com',
+    vm_gateway: '172.16.110.200'
+  },
+  # Some containers might take a lot of time to be started/stopped
+  default_timeout: 3600
+)
+```
+
+When a node is provisioned on a Proxmox cluster, the OS to be provisioned is driven by the `image` metadata. This metadata references an image through configuration that is linked to a path containing a file named `proxmox.json`, that contains image-specific configuration:
+* **template** (`String`): The path to the template to be used for this image on the Proxmox cluster.
+
+Example for a CentOS 7 image:
+```json
+{
+  "template": "Storage:vztmpl/centos-7-ssh_amd64.tar.gz"
+}
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `proxmox` | Used to connect to the Proxmox API |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `deploy_resources_min` | `Hash<Symbol, Integer>` | A hash of resources to allocate to a container for a node. Properties are `cpus`, `ram_mb` and `disk_gb`, and set the number of CPUs, MB of RAM and GB of disk to allocate to the container. Defaults are 2 cpus, 1024 MB of RAM and 10 GB of disk. |
+| `image` | `String` | The name of the OS image to be used. The [configuration](../../config_dsl.md) should define the image and point it to a directory containing a `proxmox.json` that will contain Proxmox-specific configuration (see above). |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+| `hpc_realm_for_proxmox` | Realm to be used with the `proxmox` credentials to connect to the Proxmox API. Defaults to `pam`. |
+
+## External tools dependencies
+
+None