hybrid_platforms_conductor 32.11.2 → 32.13.3

data/docs/plugins/test/mounts.md

@@ -0,0 +1,55 @@
+# Test plugin: `mounts`
+
+The `mounts` test plugin checks that mounted file systems on nodes are setup correctly.
+
+## Config DSL extension
+
+### `check_mounts_do_include`
+
+Check that a given list of mounts are indeed mounted on a given set of nodes.
+Takes as parameter a Hash of source => destination mounts to be checked. Each source and destination can be an exact String, or a Regexp for pattern matching.
+
+Example:
+```ruby
+# Make sure our cluster are mounted correctly
+for_nodes('/datanode-.+/') do
+  check_mounts_do_include(
+    # Local first disk should always be the root one
+    '/dev/sda1' => '/',
+    # Any sdb disk should be mounted somewhere in /mnt
+    /^\/dev\/sdb.+$/ => /^\/mnt\/.*/
+  )
+end
+```
+
+### `check_mounts_do_not_include`
+
+Check that a given list of mounts are not mounted on a given set of nodes.
+Takes as parameter a Hash of source => destination mounts to be checked. Each source and destination can be an exact String, or a Regexp for pattern matching.
+
+Example:
+```ruby
+# Make sure our data lake is never mounted on test nodes, in any place
+for_nodes('/tst.+/') do
+  check_mounts_do_not_include(/^datalake\.my_domain\/com:/ => /.*/)
+end
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/orphan_files.md

@@ -0,0 +1,38 @@
+# Test plugin: `orphan_files`
+
+The `orphan_files` test plugin checks that nodes don't have any file belonging to non-existing users or groups.
+
+## Config DSL extension
+
+### `ignore_orphan_files_from`
+
+Give a list of paths to be ignored while checking for orphan files.
+Useful when some paths are mounted file systems having files belonging to users that are not recognized on some nodes (like remote users).
+Takes a list of paths (or a single path) as parameter.
+
+Example:
+```ruby
+# Don't check mounted data files
+for_nodes('/prod.*/') do
+  ignore_orphan_files_from '/datalake'
+end
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/ports.md

@@ -0,0 +1,50 @@
+# Test plugin: `ports`
+
+The `ports` test plugin checks that nodes' ports are setup correctly (opened or closed).
+Useful to check for firewall deployments and general network/security settings.
+
+## Config DSL extension
+
+### `check_opened_ports`
+
+Check that a given list of ports are opened.
+Takes as parameter a list of (or single) port numbers.
+
+Example:
+```ruby
+# Check that our web services are listing on https
+for_nodes('/.*web.*/') do
+  check_opened_ports 443
+end
+```
+
+### `check_closed_ports`
+
+Check that a given list of ports are closed.
+Takes as parameter a list of (or single) port numbers.
+
+Example:
+```ruby
+# Check that smtp and pop3 are closed on all nodes
+check_closed_ports 25, 110
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `host_ip` | `String` | Host IP address to be tested for port listening |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/private_ips.md

@@ -0,0 +1,27 @@
+# Test plugin: `private_ips`
+
+The `private_ips` test plugin checks that there are no private IP address conflicts among nodes' metadata. The test will fail if at least 2 nodes declare having a common private IP address.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `private_ips` | `Array<String>` | List of private IPs to be checked |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/public_ips.md

@@ -0,0 +1,27 @@
+# Test plugin: `public_ips`
+
+The `public_ips` test plugin checks that there are no public IP address conflicts among nodes' metadata. The test will fail if at least 2 nodes declare having a common public IP address.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `public_ips` | `Array<String>` | List of private IPs to be checked |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/spectre.md

@@ -0,0 +1,26 @@
+# Test plugin: `spectre`
+
+The `spectre` test plugin checks that nodes are immune to the Spectre variants 1 and 2, and Meltdown vulnerabilities.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/veids.md

@@ -0,0 +1,27 @@
+# Test plugin: `veids`
+
+The `veids` test plugin checks that there are no VEID conflicts among nodes' metadata. The test will fail if at least 2 nodes declare having the same VEID.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `veid` | `String` or `Integer` | The node's veid |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test/vulnerabilities.md

@@ -0,0 +1,65 @@
+# Test plugin: `vulnerabilities`
+
+The `vulnerabilities` test plugin checks that a node has no known vulnerabilities, as published by its vendor's [OVAL files](https://oval.cisecurity.org/).
+
+This plugin uses a `oval.json` file storing OVAL files URLs to be fetched, in the image OS directory.
+Here is the structure of the `oval.json` file:
+* **repo_urls** (`Array<String>`): List of URLs to fetch OVAL files from. The URL can be:
+  * A direct URL to the OVAL `.xml` file.
+  * A direct URL to a compressed (`.gz` or `.bz2`) OVAL file.
+  * Any other URL that will be then crawled to follow the last link recursively unless it finds a URL to a direct OVAL file (compressed or not). This is useful to give URL of browsable repositories adding OVAL files every day, and always grab the last one.
+* **reported_severities** (`Array<String>`): List of OVAL item severities to be reported in case nodes are vulnerable to them.
+
+Example of `oval.json`:
+```json
+{
+  "repo_urls": [
+    // Take the most recent OVAL file from our repository
+    "https://my_oval.my_domain.com/oval-definitions"
+  ],
+  "reported_severities": [
+    "Critical",
+    "Important",
+    "Unknown"
+  ]
+}
+```
+
+Example of `oval.json` for Debian 10:
+```json
+{
+  "urls": [
+    // Use directly the published OVAL file from Debian
+    "https://www.debian.org/security/oval/oval-definitions-buster.xml"
+  ],
+  "reported_severities": [
+    "Critical",
+    "Important",
+    "Unknown"
+  ]
+}
+```
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+| `image` | `String` | The name of the OS image to be used. The [configuration](../../config_dsl.md) should define the image and point it to a directory containing a `oval.json` that will contain definition of OVAL files to be checked for this OS(see above). |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test_report/confluence.md

@@ -0,0 +1,43 @@
+# Test Report plugin: `confluence`
+
+The `confluence` test report plugin is publishing test results in a Confluence page.
+
+## Config DSL extension
+
+### `confluence`
+
+Define a Confluence installation to be targeted.
+
+It takes the following parameters:
+* **url** (`String`): URL to the Confluence server
+* **tests_report_page_id** (`String` or `nil`): Confluence page id used for tests reports, or nil if none [default: nil]
+
+Example:
+```ruby
+# Confluence configuration
+confluence(
+  url: 'https://my_confluence.my_domain.com/confluence',
+  # Tests report page ID
+  tests_report_page_id: '12345678'
+)
+```
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+| `confluence` | Used to connect to the Confluence API |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins/test_report/stdout.md

@@ -0,0 +1,26 @@
+# Test Report plugin: `stdout`
+
+The `stdout` test report plugin is publishing test results on stdout.
+
+## Config DSL extension
+
+None
+
+## Used credentials
+
+| Credential | Usage
+| --- | --- |
+
+## Used Metadata
+
+| Metadata | Type | Usage
+| --- | --- | --- |
+
+## Used environment variables
+
+| Variable | Usage
+| --- | --- |
+
+## External tools dependencies
+
+None

data/docs/plugins_create.md

@@ -0,0 +1,135 @@
+# How to create your own plugins
+
+This section explains how plugins work in Hybrid Platforms Conductor, and how to extend them by creating your own plugin.
+
+Plugins are auto-discovered from any Rubygem that is part of a project, by parsing files named `hpc_plugins/<plugin_type>/<plugin_id>.rb`, wherever they are located in the included Rubygem. Those files then declare plugin classes that inherit from the plugin type's base class, named `HybridPlatformsConductor::<PluginType>`.
+
+Having such simple plugins engine allow projects to adapt their plugins' organization among different repositories or Rubygems the way they see fit.
+Default plugins are shipped with the `hybrid_platforms_conductor` gem. Check [the plugins' list](./plugins.md) for details.
+
+Plugins code can use [Hybrid Platforms Conductor's API components](api.md) to use various features and access platforms' information.
+
+The following sub-sections explain how to install a plugin and the different plugin types that are supported.
+
+## Example of plugin integration from a repository
+
+As an example, we will create a test plugin, named `my_hpc_test`, whose code is defined in a Rubygem named `my_hpc_plugin` in another repository.
+
+### 1. Create the other repository as a Rubygem with your plugin
+
+```
+my_hpc_plugin/ (repository root)
+|-- Gemfile
+|-- my_hpc_plugin.gemspec
+`-- lib/
+    `-- my_hpc_plugin/
+        `-- hpc_plugins/
+            `-- test/
+                `-- my_hpc_test.rb
+```
+
+#### Gemfile
+
+The `Gemfile` file should have this simple content:
+
+```ruby
+source 'https://rubygems.org'
+
+gemspec
+```
+
+#### my_hpc_plugin.gemspec
+
+The gemspec file should contain the Rubygem declaration, with all needed dependencies.
+
+A basic working example of such a file is this:
+
+```ruby
+require 'date'
+
+Gem::Specification.new do |s|
+  s.name = 'my_hpc_plugin'
+  s.version = '0.0.1'
+  s.date = Date.today.to_s
+  s.authors     = ['<Your Name>']
+  s.email       = ['<your_email>@domain.com']
+  s.summary     = 'Plugin for Hybrid Platforms Conductor adding test my_hpc_test'
+  s.description = 'Hybrid Platforms Conductor Plugin to test great things'
+  s.homepage    = 'http://my_domain.com'
+  s.license     = 'Proprietary'
+
+  s.files = Dir['{bin,lib,spec}/**/*']
+  Dir['bin/**/*'].each do |exec_name|
+    s.executables << File.basename(exec_name)
+  end
+
+  # Dependencies
+  # Add here all the needed Rubygem dependencies for your plugin
+  # s.add_runtime_dependency 'my_awesome_rubygem_lib'
+end
+```
+
+#### lib/my_hpc_plugin/hpc_plugins/test/my_hpc_test.rb
+
+This file declares the test plugin and implement all the methods that Hybrid Platforms Conductor need to pilot a platform of this type.
+
+In our example we'll just check a dummy assertion.
+
+```ruby
+module MyHpcPlugin
+
+  module HpcPlugins
+
+    module Test
+
+      # Simple test plugin.
+      # Make sure it inherits the correct HybridPlatformsConductor base class.
+      # Make sure this file is in a hpc_plugins/<plugin_type> directory.
+      class MyHpcTest < HybridPlatformsConductor::Test
+
+        # Check my_test_plugin.rb.sample documentation for signature details.
+        def test
+          assert_equal 2 + 2, 4, 'If you see this message you have a serious problem with your CPU'
+        end
+
+      end
+
+    end
+
+  end
+
+end
+```
+
+### 2. Reference this new repository in your application's Gemfile
+
+This is done in the `Gemfile` of the project that is already using Hybrid Platforms Conductor.
+
+Adding this line to the file is enough:
+```ruby
+gem 'my_hpc_plugin', path: '/path/to/my_hpc_plugin'
+```
+
+Later when your Rubygem is part of a Git repository you may change it to:
+```ruby
+gem 'my_hpc_plugin', git: '<GIT URL for my_hpc_plugin.git>'
+```
+
+Even later when your Rubygem is packaged and deployed as a Rubygem on rubygems.org you may change it to:
+```ruby
+gem 'my_hpc_plugin'
+```
+
+Once this Gemfile is modified, don't forget to fetch the new dependency:
+```bash
+bundle install
+```
+In case the plugin is referenced using a local path, there is no need to re-issue `bundle install` when the plugin files change (good to develop locally your plugin).
+
+### 3. Your plugin is ready to use
+
+Your test plugin can now be used directly from Hybrid Platforms Conductor.
+
+```bash
+./bin/test --test my_hpc_test
+```