oxidized 0.20.0 → 0.28.0

data/docs/Hooks.md

@@ -0,0 +1,274 @@
+# Hooks
+
+You can define an arbitrary number of hooks that subscribe to different events. The hook system is modular and different kind of hook types can be enabled.
+
+## Configuration
+
+Following configuration keys need to be defined for all hooks:
+
+* `events`: which events to subscribe. Needs to be an array. See below for the list of available events.
+* `type`: what hook class to use. See below for the list of available hook types.
+
+## Events
+
+* `node_success`: triggered when configuration is successfully pulled from a node and right before storing the configuration.
+* `node_fail`: triggered after `retries` amount of failed node pulls.
+* `post_store`: triggered after node configuration is stored (this is executed only when the configuration has changed).
+* `nodes_done`: triggered after finished fetching all nodes.
+
+## Hook type: exec
+
+The `exec` hook type allows users to run an arbitrary shell command or a binary when triggered.
+
+The command is executed on a separate child process either in synchronous or asynchronous fashion. Non-zero exit values cause errors to be logged. STDOUT and STDERR are currently not collected.
+
+Command is executed with the following environment:
+
+```text
+OX_EVENT
+OX_NODE_NAME
+OX_NODE_IP
+OX_NODE_FROM
+OX_NODE_MSG
+OX_NODE_GROUP
+OX_NODE_MODEL
+OX_JOB_STATUS
+OX_JOB_TIME
+OX_REPO_COMMITREF
+OX_REPO_NAME
+```
+
+Exec hook recognizes the following configuration keys:
+
+* `timeout`: hard timeout (in seconds) for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: `60`
+* `async`: Execute the command in an asynchronous fashion. The main thread by default will wait for the hook command execution to complete. Set this to `true` for long running commands so node configuration pulls are not blocked. Default: `false`
+* `cmd`: command to run.
+
+### exec hook configuration example
+
+```yaml
+hooks:
+  name_for_example_hook1:
+    type: exec
+    events: [node_success]
+    cmd: 'echo "Node success $OX_NODE_NAME" >> /tmp/ox_node_success.log'
+  name_for_example_hook2:
+    type: exec
+    events: [post_store, node_fail]
+    cmd: 'echo "Doing long running stuff for $OX_NODE_NAME" >> /tmp/ox_node_stuff.log; sleep 60'
+    async: true
+    timeout: 120
+```
+
+## Hook type: githubrepo
+
+The `githubrepo` hook executes a `git push` to a configured `remote_repo` when the specified event is triggered.
+
+Several authentication methods are supported:
+
+* Provide a `password` for username + password authentication
+* Provide both a `publickey` and a `privatekey` for ssh key-based authentication
+* Don't provide any credentials for ssh-agent authentication
+
+The username will be set to the relevant part of the `remote_repo` URI, with a fallback to `git`. It is also possible to provide one by setting the `username` configuration key.
+
+For ssh key-based authentication, it is possible to set the environment variable `OXIDIZED_SSH_PASSPHRASE` to a passphrase if the private key requires it.
+
+`githubrepo` hook recognizes the following configuration keys:
+
+* `remote_repo`: the remote repository to be pushed to.
+* `username`: username for repository auth.
+* `password`: password for repository auth.
+* `publickey`: public key file path for repository auth.
+* `privatekey`: private key file path for repository auth.
+
+When using groups, each group must have a unique entry in the `remote_repo` config.
+
+```yaml
+hooks:
+  push_to_remote:
+    remote_repo:
+      routers: git@git.intranet:oxidized/routers.git
+      switches: git@git.intranet:oxidized/switches.git
+      firewalls: git@git.intranet:oxidized/firewalls.git
+```
+
+### githubrepo hook configuration example
+
+Authenticate with a username and a password without groups in use:
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    username: user
+    password: pass
+```
+
+Authenticate with the username `git` and an ssh key:
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    publickey: /root/.ssh/id_rsa.pub
+    privatekey: /root/.ssh/id_rsa
+```
+
+## Hook type: awssns
+
+The `awssns` hook publishes messages to AWS SNS topics. This allows you to notify other systems of device configuration changes, for example a config orchestration pipeline. Multiple services can subscribe to the same AWS topic.
+
+Fields sent in the message:
+
+* `event`: Event type (e.g. `node_success`)
+* `group`: Group name
+* `model`: Model name (e.g. `eos`)
+* `node`: Device hostname
+
+The AWS SNS hook requires the following configuration keys:
+
+* `region`: AWS Region name
+* `topic_arn`: ASN Topic reference
+
+### awssns hook configuration example
+
+```yaml
+hooks:
+  hook_script:
+    type: awssns
+    events: [node_fail,node_success,post_store]
+    region: us-east-1
+    topic_arn: arn:aws:sns:us-east-1:1234567:oxidized-test-backup_events
+```
+
+Your AWS credentials should be stored in `~/.aws/credentials`.
+
+## Hook type: slackdiff
+
+The `slackdiff` hook posts colorized config diffs to a [Slack](http://www.slack.com) channel of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `slack-api` gem on your system:
+
+```shell
+gem install slack-api
+```
+
+### slackdiff hook configuration example
+
+```yaml
+hooks:
+  slack:
+    type: slackdiff
+    events: [post_store]
+    token: SLACK_BOT_TOKEN
+    channel: "#network-changes"
+```
+
+The token parameter is a "legacy token" and is generated [Here](https://api.slack.com/custom-integrations/legacy-tokens).
+
+Optionally you can disable snippets and post a formatted message, for instance linking to a commit in a git repo. Named parameters `%{node}`, `%{group}`, `%{model}` and `%{commitref}` are available.
+
+```yaml
+hooks:
+  slack:
+    type: slackdiff
+    events: [post_store]
+    token: SLACK_BOT_TOKEN
+    channel: "#network-changes"
+    diff: false
+    message: "%{node} %{group} %{model} updated https://git.intranet/network-changes/commit/%{commitref}"
+```
+
+Note the channel name must be in quotes.
+
+A proxy can optionally be specified if needed to reach the Slack API endpoint.
+
+```yaml
+hooks:
+  slack:
+    type: slackdiff
+    events: [post_store]
+    token: SLACK_BOT_TOKEN
+    channel: "#network-changes"
+    proxy: http://myproxy:8080
+```
+
+## Hook type: ciscosparkdiff
+
+The `ciscosparkdiff` hook posts config diffs to a [Cisco Spark](https://www.ciscospark.com/) space of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `cisco_spark` gem on your system (see [cisco_spark-ruby](https://github.com/NGMarmaduke/cisco_spark-ruby)) and generate either a [Bot or OAUTH access key](https://developer.ciscospark.com/apps.html), and retrieve the [Spark Space ID](https://developer.ciscospark.com/endpoint-rooms-get.html)
+
+```shell
+gem install cisco_spark
+```
+
+### ciscosparkdiff hook configuration example
+
+```yaml
+hooks:
+  ciscospark:
+    type: ciscosparkdiff
+    events: [post_store]
+    accesskey: SPARK_BOT_API_OR_OAUTH_KEY
+    space: SPARK_SPACE_ID
+    diff: true
+```
+
+Optionally you can disable snippets and post a formatted message, for instance linking to a commit in a git repo. Named parameters `%{node}`, `%{group}`, `%{model}` and `%{commitref}` are available.
+
+```yaml
+hooks:
+  ciscospark:
+    type: ciscosparkdiff
+    events: [post_store]
+    accesskey: SPARK_BOT_API_OR_OAUTH_KEY
+    space: SPARK_SPACE_ID
+    diff: false
+    message: "%{node} %{group} %{model} updated https://git.intranet/network-changes/commit/%{commitref}"
+```
+
+Note the space and access tokens must be in quotes.
+
+A proxy can optionally be specified if needed to reach the Spark API endpoint.
+
+```yaml
+hooks:
+  ciscospark:
+    type: ciscosparkdiff
+    events: [post_store]
+    accesskey: SPARK_BOT_API_OR_OAUTH_KEY
+    space: SPARK_SPACE_ID
+    diff: true
+    proxy: http://myproxy:8080
+```
+
+## Hook type: xmppdiff
+
+The `xmppdiff` hook posts config diffs to a [XMPP](https://en.wikipedia.org/wiki/XMPP) chatroom of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `xmpp4r` gem on your system:
+
+```shell
+gem install xmpp4r
+```
+
+### xmppdiff hook configuration example
+
+```yaml
+hooks:
+  xmpp:
+    type: xmppdiff
+    events: [post_store]
+    jid: "user@server.tld/resource"
+    password: "password"
+    channel: "room@server.tld"
+    nick: "nickname"
+```
+
+Note the channel name must be in quotes.

data/docs/Model-Notes/AireOS.md

@@ -0,0 +1,11 @@
+# Cisco WLC Configuration
+
+Create a user with read-write privilege:
+
+```text
+mgmtuser add oxidized **** read-write
+```
+
+Oxidized needs read-write privilege in order to execute 'config paging disable'.
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/ArbOS.md

@@ -0,0 +1,11 @@
+# Arbor Networks ArbOS notes
+
+If you are running ArbOS version 7 or lower then you may need to update the model to remove `exec true`:
+
+```ruby
+  cfg :ssh do
+    pre_logout 'exit'
+  end
+```
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/Comware.md

@@ -0,0 +1,13 @@
+# Comware Configuration
+
+If you find 3Com Comware devices aren't being backed up this may be due to prompt detection not matching because a previous login message is disabled after the first prompt.
+
+You can disable this on the devices themselves by running this command:
+
+```text
+info-center source default channel 1 log state off debug state off
+```
+
+[Reference](https://github.com/ytti/oxidized/issues/1171)
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/Cumulus.md

@@ -0,0 +1,40 @@
+# Cumulus Linux
+
+## Routing Daemon
+
+With the release of Cumulus Linux 3.4.0 the platform moved the routing daemon to a fork of `Quagga` named `FRRouting`. See the below link for the release notes.
+
+[https://support.cumulusnetworks.com/hc/en-us/articles/115011217808-Cumulus-Linux-3-4-0-Release-Notes](https://support.cumulusnetworks.com/hc/en-us/articles/115011217808-Cumulus-Linux-3-4-0-Release-Notes)
+
+A variable has been added to enable users running Cumulus Linux > 3.4.0 to target the new `frr` routing daemon.
+
+### Example usage
+
+```yaml
+vars:
+  cumulus_routing_daemon: frr
+```
+
+Alternatively map a column for the  `cumulus_routing_daemon` variable.
+
+```yaml
+source:
+  csv:
+    map:
+      name: 0
+      ip: 1
+      model: 2
+      group: 3
+    vars_map:
+      cumulus_routing_daemon: 4
+```
+
+And set the `cumulus_routing_daemon` variable in the `router.db` file.
+
+```text
+cumulus1:192.168.121.134:cumulus:cumulus:frr
+```
+
+The default variable is `quagga` so existing installations continue to operate without interruption.
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/EOS.md

@@ -0,0 +1,12 @@
+# Arista EOS Configuration
+
+By default, EOS requires the `keyboard-interactive` SSH authentication method for a successful SSH login. To add support for this method to your Oxidized configuration, see the [SSH Auth Methods](../Configuration.md#ssh-auth-methods) directive.
+
+It is also possible to modify the EOS configuration to accept the `password` method which Oxidized presents by default. To do so, the following configuration statement can be used:
+
+```text
+management ssh
+   authentication mode password
+```
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/IOS.md

@@ -0,0 +1,29 @@
+# Cisco IOS Switches
+
+## Include unsaved changes done on a device (commented) with each configuration
+
+Create the file `~/.config/oxidized/model/ios.rb` with the following contents to extend the IOS model:
+
+```ruby
+require 'oxidized/model/ios.rb'
+
+class IOS
+
+  cmd 'show archive config diff' do |cfg|
+    # Print diff unless ntp period change or ssl-cert read from file
+    cfg.gsub! /^\n/, '' # Remove empty line
+    cfg.gsub! /^!\n/, '' # Remove line with only !
+    cfg.gsub! /.*ntp clock-period \d+\n/, '' # Remove line with only "ntp clock-period blabla"
+    cfg.gsub! /\n/, "\\n" # Escape newline
+    cfg.gsub! /crypto pki certificate chain.*certificate .*\.cer\\n/, '' # Remove ssl-cert in start config, as it is read from file, this always differ in running if used.
+    cfg.gsub! /crypto pki certificate chain.*-\s*quit\\n/, '' # Remove ssl-cert from running
+    cfg.gsub! /\\n/, "\n" # Set newline back
+    unless cfg == "!Contextual Config Diffs:\n" # Do not print if only something above was changed
+      comment cfg
+    end
+  end
+
+end
+```
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/JunOS.md

@@ -0,0 +1,33 @@
+# JunOS Configuration
+
+Create login class cfg-view
+
+```text
+set system login class cfg-view permissions view-configuration
+set system login class cfg-view allow-commands "(show)|(set cli screen-length)|(set cli screen-width)"
+set system login class cfg-view deny-commands "(clear)|(file)|(file show)|(help)|(load)|(monitor)|(op)|(request)|(save)|(set)|(start)|(test)"
+set system login class cfg-view deny-configuration all
+```
+
+Create a user with cfg-view class
+
+```text
+set system login user oxidized class cfg-view
+set system login user oxidized authentication plain-text-password "verysecret"
+```
+
+The commands Oxidized executes are:
+
+1. set cli screen-length 0
+2. set cli screen-width 0
+3. show version
+4. show chassis hardware
+5. show system license
+6. show system license keys (ex22|ex33|ex4|ex8|qfx only)
+7. show virtual-chassis (MX960 only)
+8. show chassis fabric reachability
+9. show configuration
+
+Oxidized can now retrieve your configuration!
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/LinuxGeneric.md

@@ -0,0 +1,24 @@
+# LinuxGeneric model notes
+
+To expand the usage of this model for more specific needs you can create a file in `~/.config/oxidized/model/linuxgeneric.rb`
+
+```ruby
+require 'oxidized/model/linuxgeneric.rb'
+
+class LinuxGeneric
+  
+  cmd :secret, clear: true do |cfg|
+    cfg.gsub! /^(default (\S+).* (expires) ).*/, '\\1 <redacted>'
+    cfg
+  end
+
+  post do
+    cfg = add_comment 'THE MONKEY PATCH'
+    cfg += cmd 'firewall-cmd --list-all --zone=public'
+  end
+end
+```
+
+See [Extending-Model](https://github.com/ytti/oxidized/blob/master/docs/Creating-Models.md#creating-and-extending-models)
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/Netgear.md

@@ -0,0 +1,87 @@
+# Netgear Configuration
+
+There are several models available with CLI management via telnet (port 60000). To enable telnet configure device with web interface and set 'Maintenance > Troubleshooting > Remote Diagnostics' to 'enable'. All devices behave like one of the following:
+
+## Older models
+
+```text
+Connected to 192.168.3.201.
+
+(GS748Tv4)
+Applying Interface configuration, please wait ...admin
+Password:********
+(GS748Tv4) >enable
+Password:
+
+(GS748Tv4) #terminal length 0
+
+(GS748Tv4) #show running-config
+```
+
+## Newer models
+
+```text
+Connected to 172.0.3.203.
+
+User:admin
+Password:********
+(GS724Tv4) >enable
+
+(GS724Tv4) #terminal length 0
+
+(GS724Tv4) #show running-config
+```
+
+The main differences are:
+
+* the prompt for username is different (looks quite strange for older models)
+* enable password
+  * the older model prompts for enable password and it expects empty string
+  * the newer model does not prompt for enable password at all
+
+Configuration for older/newer models: make sure you have defined variable 'enable':
+
+* `'true'` for newer models
+* `''` empty string: for older models
+
+One possible configuration:
+
+## oxidized config
+
+```yaml
+source:
+  default: csv
+  csv:
+    file: "/home/oxidized/.config/oxidized/router.db"
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    vars_map:
+      enable: 4
+      telnet_port: 5
+```
+
+## router.db
+
+```text
+switchOldFW:netgear:admin:adminpw::60000
+switchNewFW:netgear:admin:adminpw:true:60000
+```
+
+Another approach to set parameters:
+
+## oxidized config
+
+```yaml
+  netgear:
+    vars:
+      enable: true
+      telnet_port: 60000
+```
+
+[Reference](https://github.com/ytti/oxidized/pull/1268)
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/Nokia.md

@@ -0,0 +1,9 @@
+# Nokia
+
+## Nokia ISAM and SSH keepalives
+
+Nokia ISAM might require disabling SSH keepalives.
+
+[Reference](https://github.com/ytti/oxidized/issues/1482)
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/README.md

@@ -0,0 +1,24 @@
+# Model Notes
+
+This directory contains implementation notes and caveats to assist you in your oxidized deployment.
+
+Use the table below for more information on the Vendor/Model caveats.
+
+Vendor          | Model           |Updated
+----------------|-----------------|----------------
+3COM|[Comware](Comware.md)|15 Feb 2018
+AireOS|[AireOS](AireOS.md)|29 Nov 2017
+Arbor Networks|[ArbOS](ArbOS.md)|27 Feb 2018
+Arista|[EOS](EOS.md)|05 Feb 2018
+Cumulus|[Cumulus](Cumulus.md)|11 Jun 2018
+Huawei|[VRP](VRP-Huawei.md)|17 Nov 2017
+Huawei|[SmartAX series](SmartAX-Huawei.md)|21 Jan 2019
+Cisco IOS|[IOS](IOS.md)|29 Mar 2019
+Juniper|[MX/QFX/EX/SRX/J Series](JunOS.md)|18 Jan 2018
+Netgear|[Netgear](Netgear.md)|11 Apr 2018
+Nokia|[Nokia ISAM](Nokia.md)|22 Aug 2018
+Viptela|[Viptela](Viptela.md)|1 Jul 2018
+Zyxel|[XGS4600 Series](XGS4600-Zyxel.md)|1 Feb 2018
+Linux|[LinuxGeneric](LinuxGeneric.md)|10 Jun 2019
+
+If you discover additional caveats or problems please make sure to consult the [GitHub issues for oxidized](https://github.com/ytti/oxidized/issues) known issues.

data/docs/Model-Notes/SmartAX-Huawei.md

@@ -0,0 +1,35 @@
+# Huawei SmartAX GPON/EPON/DOCSIS network access devices
+
+It is necessary to disable SSH keepalives in Oxidized for configuration retrieval via SSH to work properly.
+
+To disable SSH keepalives globally edit the config's vars section and add:
+
+```yaml
+vars:
+  ssh_no_keepalive: true
+```
+
+To disable SSH keepalives per device edit the config's source section and map ssh_no_keepalive to a column inside router.db file.
+
+```yaml
+source:
+  default: csv
+  csv:
+    file: ~/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+      username: 2
+      password: 3
+    vars_map:
+      ssh_no_keepalive: 4
+```
+
+```text
+# router.db
+10.0.0.1:smartax:someusername:somepassword:true
+10.0.0.2:ios:someusername:somepassword:false
+```
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/VRP-Huawei.md

@@ -0,0 +1,34 @@
+# Huawei VRP Configuration
+
+Create a user with no privileges
+
+```text
+    <HUAWEI> system-view
+    [~HUAWEI] aaa
+    [~HUAWEI-aaa] local-user oxidized password irreversible-cipher verysecret
+    [*HUAWEI-aaa] local-user oxidized level 1
+    [*HUAWEI-aaa] local-user oxidized service-type terminal ssh
+    [*HUAWEI-aaa] commit
+```
+
+The commands Oxidized executes are:
+
+1. screen-length 0 temporary
+2. display version
+3. display device
+4. display current-configuration all
+
+Command 2 and 3 can be executed without issues, but 1 and 4 are only available for higher level users. Instead of making Oxidized a read/write user on your device, lower the privilege-level for commands 1 and 4:
+
+```text
+    <HUAWEI> system-view
+    [~HUAWEI] command-privilege level 1 view global display current-configuration all
+    [*HUAWEI] command-privilege level 1 view shell screen-length
+    [*HUAWEI] commit
+```
+
+Oxidized can now retrieve your configuration!
+
+Caveat: Some versions of VRP default to appending a timestamp prior to the output of each `display` command, which will lead to superfluous updates. The configuration statement `timestamp disable` can be used to disable this functionality. (Issue #1218)
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/Viptela.md

@@ -0,0 +1,12 @@
+# Viptela
+
+This model collects running config and other desired commands from Viptela devices.
+
+Pagination is disabled post login.
+
+## Supported Commands
+
+- show running-config
+- show version
+
+Back to [Model-Notes](README.md)