dopi 0.17.0

data/doc/getting_started_examples/002_connecting_over_ssh.yaml

@@ -0,0 +1,35 @@
+name: 'hello_world'
+
+credentials:
+  'test-credentials':
+    type: 'ssh_key'
+    username: 'myuser'
+    private_key: '/home/myuser/.ssh/id_rsa'
+
+infrastructures:
+  'test':
+    type: 'baremetal'
+    networks:
+      'localhost':
+        ip_pool:
+          from: '127.0.0.2'
+          to:   '127.0.0.250'
+        ip_netmask: '255.255.255.0'
+        ip_defgw: '127.0.0.1'
+
+nodes:
+  'testlinux.example.com':
+    infrastructure: 'test'
+    interfaces:
+      'eth0':
+        network: 'localhost'
+        ip: '127.0.0.2'
+
+steps:
+  - name: 'write hello world'
+    nodes: 'all'
+    command:
+      plugin: 'ssh/custom'
+      credentials: 'test-credentials'
+      exec: 'echo'
+      arguments: '"hello world"'

data/doc/plugins/custom.md

@@ -0,0 +1,88 @@
+# DOPi Command Plugin: Custom Command
+
+This plugin executes custom commands on the the machine DOPi is running on.
+
+DOPi will execute a customized command on the node DOPi is running on.
+It will run the command once per node in the step and export the node
+fqdn in the environment variable DOP_NODE_FQDN
+
+## Plugin Settings:
+
+### exec (required)
+
+The command the plugin should execute for every node.
+
+### arguments (optional)
+
+`default: ""`
+
+The arguments for the command. This can be set by a string as an array or
+as a hash. All the elements of the hash and the array will be flattened
+and joined with a space.
+
+### env (optional)
+
+`default: { DOP_NODE_FQDN => fqdn_of_node }`
+
+The environment variables that should be set. Will be merged with the defalut hash
+
+### expect_exit_codes (optional)
+
+`default: 0`
+
+The exit codes DOPi should expect if the program terminates. It the program
+exits with an exit code not listed here, DOPi will mark the run as failed.
+The values can be a number, an array of numbers or :all for all possible exit
+codes. Will replace the current default.
+
+### parse_output (optional)
+
+`default: {}`
+
+Here you can define patterns that match against the output of the command plugin
+and flag certain lines as errors or warnings. The parse_output key should contain
+a hash with two keys, 'error' and 'warning' which each can contain an array of
+patterns.
+
+### fail_on_warning
+
+`default: false`
+
+Set this to true if you want to threat warnings in the output as errors. This is
+useful if the plugin already specifies the parsing patterns but your need to stop
+the run on warnings.
+
+## Examples:
+
+### Simple Example
+
+```YAML
+    - name "Say Hello World on the machine DOPi is running on"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'custom'
+        exec: 'echo'
+        arguments: "Hello World from ${DOP_NODE_FQDN}"
+```
+
+### Complete Example
+
+```YAML
+    - name "Install an RPM on the machine DOPi is running on"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'custom'
+        exec: 'yum'
+        arguments: 'install -y puppet'
+        expect_exit_codes: 0
+        fail_on_warning: False
+        parse_output:
+          error:
+            - '^No package puppet available'
+            - 'Some other Error pattern'
+          warning:
+            - '^Warning:'
+```
+

data/doc/plugins/mco/rpc.md

@@ -0,0 +1,82 @@
+# DOPi Command Plugin: MCollective RPC Command
+
+This DOPi Plugin will trigger an action of a MCollective agent on the
+nodes in the Step. The plugin can call any agent and any action and will
+mark the step as failed according to the status in the reply.
+
+However, do to the generic nature of this plugin it will not be able to
+parse the data structure returned by the agent as this is agent specific.
+Dopi will warn about this when am agent is called with this plugin because
+some errors that are not reflected in the status code may not be caught
+by this plugin.
+
+## Plugin Settings:
+
+### agent (required)
+
+The MCollective agent to be called. This agent has to be installed on
+the nodes and the machine DOPi is running on. The DOPi validator will
+check if the agent is present on the machine where you try to run the
+plan.
+
+### action (required)
+
+The action we call on the specified agent. Consult the documentation of
+the MCollective agent in question for a list of valid actions. The DOPi
+validator will check if the agent supports such an action and fail if it
+doesn't.
+
+### options (optional)
+
+`default: {}`
+
+The options hash passed to the MCollective client. Check the output of
+"mco help rpc" to get a list of valid options. However not all of this options
+will make sense in this context. For example, you should never specify a filter
+because DOPi already takes care of that via the node list of the step.
+
+### arguments (optional)
+
+`default: {}`
+
+The arguments passed to the MCollective agent. Check the agent documentation to
+get a list of required and optional arguments. DOPi will use the validation methods
+provided by MCollective agents and fail the plan validation if the arguments are not
+correct or the values are not valid.
+
+### expect_exit_codes (optional)
+
+`default: 0`
+
+The exit codes DOPi should expect in the status code returned by the MCollective
+agent. It the status code is an exit code not listed here, DOPi will mark the run
+as failed. The values can be a number, an array of numbers or :all for all possible exit
+codes. Will replace the current default.
+
+
+## Examples:
+
+### Simple Example
+
+    - name "Get the inventory of a node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'mco/rpc'
+        agent: 'rpcutil'
+        action: 'inventory'
+
+### Complete Example
+
+    - name "Get a fact of a node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'mco/rpc'
+        agent: 'rpcutil'
+        action: 'get_fact'
+        arguments:
+          :fact: 'osfamily'
+        options:
+          :timeout: 30
+          :ttl: 60

data/doc/plugins/ssh/custom.md

@@ -0,0 +1,141 @@
+# DOPi Command Plugin: Custom SSH Command
+
+This plugin executes custom commands on every node in the step over ssh.
+
+By default the ssh custom command plugin will skip the host key checks. This
+means you will be potentially vulnerable to man in the middle attacks. Since
+DOPi is designed for the provisioning phase you are working with new nodes and
+this will usually not be an issue, however changing keys because nodes get
+new provisioned will be. To change this set **ssh_check_host_key** to true.
+
+## credentials for login
+
+The ssh/custom plugin and all the plugins that inherit from it can use
+credentials from the credentials hash in the plan to login.
+
+    credentials:
+      'linux_staging_login':
+        type: :username_password
+        username: 'root'
+        password: 'foo'
+
+    steps:
+      - name: 'set ssh login credentials'
+        command:
+          plugin: 'ssh/custom'
+          credentials: 'linux_staging_login'
+          exec: 'env'
+
+You can also specify multiple credentials and the ssh plugin will try each one
+of them in turn to login to the node.
+
+    credentials:
+      'linux_staging_login':
+        type: :username_password
+        username: 'root'
+        password: 'foo'
+      'linux_prod_login':
+        type: :ssh_key
+        private_key: '/home/root/.ssh/id_dsa'
+
+    steps:
+      - name: 'set ssh login credentials'
+        command:
+          plugin: 'ssh/custom'
+          credentials:
+            - 'linux_staging_login'
+            - 'linux_prod_login'
+          exec: 'env'
+
+This can be set in each plugin (don't forget the validator plugins) or via
+the set_plugin_defaults hash. The plugin defaults will be preserved for
+subsequent steps until it is altered or deleted.
+
+    credentials:
+      'linux_staging_login':
+        type: :username_password
+        username: 'root'
+        password: 'foo'
+      'linux_prod_login':
+        type: :ssh_key
+        private_key: '/home/root/.ssh/id_dsa'
+
+    steps:
+      - name: 'set ssh login credentials'
+        set_plugin_defaults:
+          - plugins: '/^ssh/'
+            :credentials:
+              - 'linux_staging_login'
+              - 'linux_prod_login'
+        command:
+          plugin: 'ssh/custom'
+          exec: 'env'
+
+The login via password will need the program sshpass to be installed on the
+machine DOPi is running on. DOPi will display a warning if you specify a
+username password credential but sshpass is not installed.
+
+## ssh_root_pass auth (DEPRECATED >=0.4)
+
+DOPi will connect to the node via ssh and execute the specified command.
+It will use the configured ssh key **(ssh_key)** with the configured user
+**(ssh_user)** unless **ssh_pass_auth** is set and a password for the node
+is configured.
+
+The login via password will need the program sshpass to be installed on the
+machine DOPi is running on. The password can be specified as a global option
+in the plan or be overwritten via hieradata.
+
+    plan:
+      ssh_root_pass: 'mypass'
+
+    configuration:
+      node:
+        'web01.example.com':
+           ssh_root_pass: 'myotherpass'
+
+## Plugin Settings:
+
+The ssh custom command plugin is based on the [custom command plugin](doc/plugins/custom.md)
+and inherits all its parameters.
+
+### quiet (optional)
+
+`default: true`
+
+By default ssh will run in quiet mode and stuff like login banners will not
+appear in the output. See the documentation about the "-q" flag in the ssh
+man page for more information about this.
+
+### port (optional)
+
+`default: 22`
+
+The port to which ssh should connect
+
+## Examples:
+
+### Simple Example
+
+    - name "Say Hello World on a node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/custom'
+        exec: 'echo'
+        arguments: "Hello World"
+
+### Complete Example
+
+    - name "Install an RPM on the node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/custom'
+        exec: 'yum'
+        arguments: 'install -y puppet'
+        expect_exit_codes: 0
+        fail_on_warning: False
+        parse_output:
+          error:
+            - '^No package puppet available'

data/doc/plugins/ssh/file_contains.md

@@ -0,0 +1,37 @@
+# DOPi Command Plugin: File Contains SSH Command
+
+This DOPi Plugin will check if the specified file on the node contains a
+specific pattern. This plugin is usually used as a verify command to check
+if a command has to be executed on a node.
+
+## Plugin Settings:
+
+The ssh/file_contains command plugin is based on the
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+### file (required)
+
+The file to check
+
+### pattern (required)
+
+The regular expression to check against
+
+## Example:
+
+    - name "Run puppet on a node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/puppet_agent_run'
+        verify_commands:
+          - plugin: 'ssh/file_contains'
+            file: '/etc/puppet/puppet.conf'
+            pattern: 'puppetmaster.example.com'
+        arguments:
+          '--server': 'puppetmaster.example.com'

data/doc/plugins/ssh/file_deploy.md

@@ -0,0 +1,52 @@
+# DOPi Command Plugin: File Deploy SSH Command
+
+This DOPi Plugin will deploy a file to a remote node which can be specified
+inline in the plan or via file or output of an executable.
+
+## Plugin Settings:
+
+The ssh/file_deploy command plugin is based on the
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+### file (required)
+
+The target file to be deployed
+
+### content (required)
+
+The content of the file. This can be specified directly as a string in the
+yaml file or from a file if a hash is specified with a file source.
+
+## Example:
+
+    - name "Deploy file with inline content"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/file_deploy'
+        file: '/tmp/resolv.conf'
+        content: |
+          domain example.com
+          nameserver 1.2.3.4
+          nameserver 4.3.2.1
+
+    - name "Deploy file with content from a local file"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/file_deploy'
+        file: '/tmp/resolv.conf'
+        content: { file: './some/local/resolv.conf' }
+
+    - name "Deploy file with content from a program output"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/file_deploy'
+        file: '/tmp/resolv.conf'
+        content: { exec: ['/bin/exec', 'hello world'] }

data/doc/plugins/ssh/file_exists.md

@@ -0,0 +1,31 @@
+# DOPi Command Plugin: File Contains SSH Command
+
+This DOPi Plugin will check if the specified file on the node exists.
+This plugin is usually used as a verify command to check if a command
+has to be executed on a node.
+
+## Plugin Settings:
+
+The ssh/file_exists command plugin is based on the
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+### file (required)
+
+The file to check
+
+## Example:
+
+    - name: 'Install Puppet'
+      nodes: all
+      command:
+        plugin: 'ssh/custom'
+        verify_commands:
+          - plugin: 'ssh/file_exists'
+            file: '/usr/bin/puppet'
+      exec: 'yum'
+      arguments: 'install -y puppet'

data/doc/plugins/ssh/file_replace.md

@@ -0,0 +1,37 @@
+# DOPi Command Plugin: File Replace SSH Command
+
+This DOPi Plugin to replace a search pattern with a replacement string
+in a file on a node over ssh. This plugin uses sed to replace the string,
+keep that in mind if you define the pattern String.
+
+## Plugin Settings:
+
+The ssh/file_replace command plugin is based on the
+[ssh/file_contains](doc/plugins/ssh/file_contains.md),
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+### replacement (required)
+
+This the replacement string the that will be substituted for the found pattern.
+
+### global (optional)
+
+`default: false`
+
+If false it will only replace the first match, if true it will replace them all.
+
+## Example:
+
+    - name "replace the server in the Puppet config"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/file_replace'
+        file: '/etc/puppet/puppet.conf'
+        pattern: 'puppet.example.com'
+        replacement: 'somethingelse.example.com'

data/doc/plugins/ssh/puppet_agent_run.md

@@ -0,0 +1,50 @@
+# DOPi Command Plugin: SSH Puppet Agent run Command
+
+This DOPi Plugin will run puppet on the nodes in the step.
+
+## Plugin Settings:
+
+The ssh/puppet_agent_run command plugin is based on the
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+You may want to set a high **plugin_timeout** here to make sure it waits
+long enough for all the nodes to come up if you provision the nodes while
+waiting.
+
+This plugin overwrites the defaults for the **expect_exit_codes** parameter.
+Puppet will return an exit code of 2 if there where changes. Since this will
+be what we are looking we have to expect this and mark it as success
+
+`default overwrite for expect_exit_codes : [ 0, 2 ]`
+
+If you are sure that there will be changes you should overwrite this with 2
+to make sure there where changes. This may help you catching problems where
+nodes don't actually get a configuration or are in the wrong environment.
+
+The ssh/puppet_agent_run plugin has no additional parameters.
+
+## Examples:
+
+### Simple Example
+
+    - name "Run puppet on a node"
+      nodes:
+        - 'web01.example.com'
+      command: 'ssh/puppet_agent_run'
+
+### Complete Example
+
+    - name "Run puppet with parameters on a node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/puppet_agent_run'
+        plugin_timeout: 300
+        arguments:
+          '--server': 'puppetmaster.example.com'
+          '--environment': 'development'

data/doc/plugins/ssh/reboot.md

@@ -0,0 +1,22 @@
+# DOPi Command Plugin: Reboot
+
+This plugin will send the reboot command to a node. Then it will check if the
+node actually rebooted by checking until it is unavailable and then available
+again. The command will finish if DOPi is again able to login after the reboot.
+
+## Plugin Settings:
+
+### reboot_cmd (optional)
+
+`default: shutdown -r now`
+
+The command which gets executed to reboot the machine
+
+
+## Example
+
+    - name "Wait until we can successfully login to the node"
+      nodes:
+        - 'web01.example.com'
+      command: 'ssh/reboot'
+

data/doc/plugins/ssh/wait_for_login.md

@@ -0,0 +1,53 @@
+# DOPi Command Plugin: Wait for SSH login Command
+
+This DOPi Plugin will try to connect to the node until a successful login
+is possible or until a timeout is reached.
+
+## Plugin Settings:
+
+The ssh/wait_for_login command plugin is based on the
+[ssh custom command plugin](doc/plugins/ssh/custom.md) and the
+[custom command plugin](doc/plugins/custom.md) and inherits all their
+parameters.
+
+It will however overwrite the **exec** parameter, so it is not possible to
+set a custom command in this plugin.
+
+You may want to set a high **plugin_timeout** here to make sure it waits
+long enough for all the nodes to come up if you provision the nodes while
+waiting.
+
+### connect_timeout (optional)
+
+`default: 0`
+
+Amount of seconds to wait while connecting until giving up.
+
+### interval (optional)
+
+`default: 10`
+
+Amount of seconds to wait between login attempts.
+
+## Examples:
+
+### Simple Example
+
+    - name "Wait until we can successfully login to the node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/wait_for_login'
+        plugin_timeout: 300
+
+
+### Complete Example
+
+    - name "Wait until we can successfully login to the node"
+      nodes:
+        - 'web01.example.com'
+      command:
+        plugin: 'ssh/wait_for_login'
+        plugin_timeout: 300
+        connect_timeout: 5
+        interval: 30