oxidized 0.21.0 → 0.22.0

data/Rakefile

@@ -1,9 +1,22 @@
 require 'bundler/gem_tasks'
 require 'rake/testtask'
+require_relative 'lib/oxidized/version'
 
 gemspec = eval(File.read(Dir['*.gemspec'].first))
 file    = [gemspec.name, gemspec.version].join('-') + '.gem'
 
+# Integrate Rubocop if available
+begin
+  require 'rubocop/rake_task'
+
+  RuboCop::RakeTask.new
+  task(:default).prerequisites << task(:rubocop)
+rescue LoadError
+  task :rubocop do
+    puts 'Install rubocop to run its rake tasks'
+  end
+end
+
 desc 'Validate gemspec'
 task :gemspec do
   gemspec.validate
@@ -19,6 +32,12 @@ task :test do
   end
 end
 
+task :build => :version_set
+task :version_set do
+  Oxidized.version_set
+  Bundler::GemHelper.instance.gemspec.version = Oxidized::VERSION
+end
+
 ## desc 'Install gem'
 ## task :install => :build do
 ##   system "sudo -Es sh -c \'umask 022; gem install gems/#{file}\'"

data/TODO.md

@@ -1,23 +1,29 @@
-# refactor core
-  * move state from memory to disk, sqlite probably
-  * allows us to retain stats etc over restart
-  * simplifies code
-  * keep only running nodes in memory
-  * negligible I/O cost, as majority is I/O wait getting config
-
-# separate login to owon package
-  * oxidized-script is not only use-case
-  * it would be good to have minimal package used to login to routers
-  * oxidized just one consumer of that functionality
-  * what to do with models, we need model to know how to login. Should models be separated to another package? oxidized-core, oxidized-models and oxidized-login?
-  * how can we allow interactive login in oxidized-login? With functional VTY etc? REPL loop in input/ssh and input/telnet?
-
-# thread number
-  * think about algo
-  * if job ended later than now-iteration have rand(node.size) == 0 to add thread
-  * if now is less than job_ended+iteration same chance to remove thread?
-  * should we try to avoid max threads from being hit? (like maybe non-success thread is pulling average?)
-
-# docs, testing
-  * yard docs
-  * minitest tests
+# To Do
+
+## refactor core
+
+* move state from memory to disk, sqlite probably
+* allows us to retain stats etc over restart
+* simplifies code
+* keep only running nodes in memory
+* negligible I/O cost, as majority is I/O wait getting config
+
+## separate login to own package
+
+* oxidized-script is not only use-case
+* it would be good to have minimal package used to login to routers
+* oxidized just one consumer of that functionality
+* what to do with models, we need model to know how to login. Should models be separated to another package? oxidized-core, oxidized-models and oxidized-login?
+* how can we allow interactive login in oxidized-login? With functional VTY etc? REPL loop in input/ssh and input/telnet?
+
+## thread number
+
+* think about algo
+* if job ended later than now-iteration have rand(node.size) == 0 to add thread
+* if now is less than job_ended+iteration same chance to remove thread?
+* should we try to avoid max threads from being hit? (like maybe non-success thread is pulling average?)
+
+## docs, testing
+
+* yard docs
+* minitest tests

data/bin/oxidized

@@ -1,6 +1,5 @@
 #!/usr/bin/env ruby
 
-
 # FIX ME, killing oxidized needs -9
 trap("INT") { exit } # sinatra will otherwise steal this from us
 
@@ -9,6 +8,6 @@ begin
   Oxidized::CLI.new.run
 rescue => error
   warn "#{error}"
-  debug = Oxidied.config.debug rescue true
+  debug = Oxidized.config.debug rescue true
   raise if debug
 end

data/docs/Configuration.md

@@ -1,10 +1,12 @@
-## Configuration
-### Debugging
+# Configuration
+
+## Debugging
+
 In case a model plugin doesn't work correctly (ios, procurve, etc.), you can enable live debugging of SSH/Telnet sessions. Just add a `debug` option containing the value true to the `input` section. The log files will be created depending on the parent directory of the logfile option.
 
-The following example will log an active ssh/telnet session `/home/oxidized/.config/oxidized/log/<IP-Adress>-<PROTOCOL>`. The file will be truncated on each consecutive ssh/telnet session, so you need to put a `tailf` or `tail -f` on that file!
+The following example will log an active ssh/telnet session `/home/oxidized/.config/oxidized/log/<IP-Address>-<PROTOCOL>`. The file will be truncated on each consecutive ssh/telnet session, so you need to put a `tailf` or `tail -f` on that file!
 
-```
+```yaml
 log: /home/oxidized/.config/oxidized/log
 
 ...
@@ -16,54 +18,68 @@ input:
     secure: false
 ```
 
-### Privileged mode
+## Privileged mode
 
 To start privileged mode before pulling the configuration, Oxidized needs to send the enable command. You can globally enable this, by adding the following snippet to the global section of the configuration file.
 
-```
+```yaml
 vars:
    enable: S3cre7
 ```
 
-### Removing secrets
+## Removing secrets
 
-To strip out secrets from configurations before storing them, Oxidized needs the the remove_secrets flag. You can globally enable this by adding the following snippet to the global sections of the configuration file.
+To strip out secrets from configurations before storing them, Oxidized needs the `remove_secret` flag. You can globally enable this by adding the following snippet to the global section of the configuration file.
 
-```
+```yaml
 vars:
   remove_secret: true
 ```
 
-Device models can contain substitution filters to remove potentially sensitive data from configs.
+Device models that contain substitution filters to remove sensitive data will now be run on any fetched configuration.
 
 As a partial example from ios.rb:
 
-```
+```ruby
   cmd :secret do |cfg|
     cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>'
     (...)
     cfg
   end
 ```
+
 The above strips out snmp community strings from your saved configs.
 
 **NOTE:** Removing secrets reduces the usefulness as a full configuration backup, but it may make sharing configs easier.
 
-### Disabling SSH exec channels
+## Disabling SSH exec channels
 
-Oxidized uses exec channels to make information extraction simpler, but there are some situations where this doesn't work well, e.g. configuring devices.  This feature can be turned off by setting the `ssh_no_exec`
+Oxidized uses exec channels to make information extraction simpler, but there are some situations where this doesn't work well, e.g. configuring devices. This feature can be turned off by setting the `ssh_no_exec`
 variable.
 
-```
+```yaml
 vars:
   ssh_no_exec: true
 ```
 
-### SSH Proxy Command
+## SSH Auth Methods
 
-Oxidized can `ssh` through a proxy as well. To do so we just need to set `ssh_proxy` variable.
+By default, Oxidized registers the following auth methods: `none`, `publickey` and `password`. However you can configure this globally, by groups, models or nodes.
 
 ```
+vars:
+    auth_methods: none, publickey, password, keyboard-interactive
+```
+
+## SSH Proxy Command
+
+Oxidized can `ssh` through a proxy as well. To do so we just need to set `ssh_proxy` variable with the proxy host information.
+
+This can be provided on a per-node basis by mapping the proper fields from your source.
+
+An example for a `csv` input source that maps the 4th field as the `ssh_proxy` value.
+
+```yaml
 ...
 map:
   name: 0
@@ -74,16 +90,26 @@ vars_map:
 ...
 ```
 
-### Advanced Configuration
+## FTP Passive Mode
 
-Below is an advanced example configuration. You will be able to (optionally) override options per device. The router.db format used is `hostname:model:username:password:enable_password`. Hostname and model will be the only required options, all others override the global configuration sections.
+Oxidized uses ftp passive mode by default. Some devices require passive mode to be disabled. To do so, we can set `input.ftp.passive` to false - this will make use of FTP active mode.
 
+```yaml
+input:
+  ftp:
+    passive: false
 ```
+
+## Advanced Configuration
+
+Below is an advanced example configuration. You will be able to (optionally) override options per device. The router.db format used is `hostname:model:username:password:enable_password`. Hostname and model will be the only required options, all others override the global configuration sections.
+
+```yaml
 ---
 username: oxidized
 password: S3cr3tx
 model: junos
-interval: 3600
+interval: 3600 #interval in seconds
 log: ~/.config/oxidized/log
 debug: false
 threads: 30
@@ -121,14 +147,13 @@ source:
 model_map:
   cisco: ios
   juniper: junos
-
 ```
 
-### Advanced Group Configuration
+## Advanced Group Configuration
 
 For group specific credentials
 
-```
+```yaml
 groups:
   mikrotik:
     username: admin
@@ -137,16 +162,19 @@ groups:
     username: ubnt
     password: ubnt
 ```
+
 and add group mapping
-```
+
+```yaml
 map:
   model: 0
   name: 1
   group: 2
 ```
+
 For model specific credentials
 
-```
+```yaml
 models:
   junos:
     username: admin
@@ -161,26 +189,38 @@ models:
     password: password
 ```
 
-### RESTful API and Web Interface
+## RESTful API and Web Interface
 
 The RESTful API and Web Interface is enabled by configuring the `rest:` parameter in the config file.  This parameter can optionally contain a relative URI.
 
-```
+```yaml
 # Listen on http://127.0.0.1:8888/
 rest: 127.0.0.1:8888
 ```
 
-```
+```yaml
 # Listen on http://10.0.0.1:8000/oxidized/
 rest: 10.0.0.1:8000/oxidized
 ```
 
-### Triggered backups
+## Triggered backups
 
-A node can be moved to head-of-queue via the REST API `GET/POST /node/next/[NODE]`.
+A node can be moved to head-of-queue via the REST API `GET/POST /node/next/[NODE]`. This can be useful to immediately schedule a fetch of the configuration after some other event such as a syslog message indicating a configuration update on the device.
 
-In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added:
+In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added automatically:
 
-```
+```yaml
 next_adds_job: true
 ```
+
+This will allow for a more timely fetch of the device configuration.
+
+## Disabling DNS resolution
+
+In some instances it might not be desirable to attempt to resolve names of nodes. One such use case is when nodes are accessed through an SSH proxy, where the remote end resolves the names differently than the host on which Oxidized runs would.
+
+Names can instead be passed verbatim to the input:
+
+```yaml
+resolve_dns: false
+```

data/docs/Creating-Models.md

@@ -0,0 +1,78 @@
+# Creating and Extending Models
+
+Oxidized supports a growing list of [operating system types](Supported-OS-Types.md). Out of the box, most model implementations collect configuration data. Some implementations also include a conservative set of additional commands that collect basic device information (device make and model, software version, licensing information, ...) which are appended to the configuration as comments.
+
+A user may wish to extend an existing model to collect the output of additional commands. Oxidized offers smart loading of models in order to facilitate this with ease, without the need to introduce changes to the upstream source code.
+
+This methodology allows local site changes to be preserved during Oxidized version updates / gem updates.
+
+## Extending an existing model with a new command
+
+The example below can be used to extend the `JunOS` model to collect the output of `show interfaces diagnostics optics` and append the output to the configuration file as a comment. This command retrieves DOM information on pluggable optics present in a `JunOS`-powered chassis.
+
+Create the file `~/.config/oxidized/model/junos.rb` with the following contents:
+
+```ruby
+require 'oxidized/model/junos.rb'
+
+
+class JunOS
+
+
+    cmd 'show interfaces diagnostics optics' do |cfg|
+        comment cfg
+    end
+
+
+end
+```
+
+Due to smart loading, the user-supplied `~/.config/oxidized/model/junos.rb` file will take precedence over the model with the same name included in the Oxidized distribution.
+
+The code then uses `require` to initially load the Oxidized-supplied model, and extends the class defined therein with an additional command.
+
+Intuitively, it is also possible to:
+
+* Completely re-define an existing model by creating a file in `~/.config/oxidized/model/` with the same name as an existing model, but not `require`-ing the upstream model file.
+* Create a named variation of an existing model, by creating a file with a new name (such as `~/.config/oxidized/model/junos-extra.rb`), Then `require` the original model and extend its base class as in the above example. The named variation can then be specified as an OS type for specific devices that can benefit from the extra functionality. This allows for preservation of the base functionality for the default model types.
+* Create a completely new model, with a new name, for a new operating system type.
+* Testing/validation of an updated model from the [Oxidized GitHub repo models](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model) by placing an updated model in the proper location without disrupting the gem-supplied model files.
+
+## Advanced features
+
+The loosely-coupled architecture of Oxidized allows for easy extensibility in more advanced use cases as well.
+
+The example below extends the functionality of the `JunOS` model further to collect `display set` formatted configuration from the device, and utilizes the multi-output functionality of the `git` output to place the returned configuration in a separate file within a git repository.
+
+It is possible to configure the `git` output to create new subdirectories under an existing repository instead of creating new repositories for each new defined output type (the default) by including the following configuration in the `~/.config/oxidized/config` file:
+
+```yaml
+output:
+    git:
+        type_as_directory: true
+```
+
+Then, `~/.config/oxidized/model/junos.rb` is adapted as following:
+
+```ruby
+require 'oxidized/model/junos.rb'
+
+
+class JunOS
+
+
+    cmd 'show interface diagnostic optics' do |cfg|
+        comment cfg
+    end
+
+    cmd 'show configuration | display set' do |cfg|
+        cfg.type = "junos-set"
+        cfg.name = "set"
+        cfg
+    end
+
+
+end
+```
+
+The output of the `show configuration | display set` command is marked with a new arbitrary alternative output type, `junos-set`.  The `git` output will use the output type to create a new subdirectory by the same name. In this subdirectory, the `git` output will create files with the name `<devicename>--set` that will contain the output of this command for each device.

data/docs/Hooks.md

@@ -1,46 +1,52 @@
 # Hooks
-You can define arbitrary number of hooks that subscribe different events. The hook system is modular and different kind of hook types can be enabled.
+
+You can define an arbitrary number of hooks that subscribe to different events. The hook system is modular and different kind of hook types can be enabled.
 
 ## Configuration
+
 Following configuration keys need to be defined for all hooks:
 
-  * `events`: which events to subscribe. Needs to be an array. See below for the list of available events.
-  * `type`: what hook class to use. See below for the list of available hook types.
+* `events`: which events to subscribe. Needs to be an array. See below for the list of available events.
+* `type`: what hook class to use. See below for the list of available hook types.
 
-### Events
-  * `node_success`: triggered when configuration is succesfully pulled from a node and right before storing the configuration.
-  * `node_fail`: triggered after `retries` amount of failed node pulls.
-  * `post_store`: triggered after node configuration is stored (this is executed only when the configuration has changed).
-  * `nodes_done`: triggered after finished fetching all nodes.
+## Events
+
+* `node_success`: triggered when configuration is successfully pulled from a node and right before storing the configuration.
+* `node_fail`: triggered after `retries` amount of failed node pulls.
+* `post_store`: triggered after node configuration is stored (this is executed only when the configuration has changed).
+* `nodes_done`: triggered after finished fetching all nodes.
 
 ## Hook type: exec
+
 The `exec` hook type allows users to run an arbitrary shell command or a binary when triggered.
 
 The command is executed on a separate child process either in synchronous or asynchronous fashion. Non-zero exit values cause errors to be logged. STDOUT and STDERR are currently not collected.
 
 Command is executed with the following environment:
-```
+
+```text
 OX_EVENT
 OX_NODE_NAME
 OX_NODE_IP
 OX_NODE_FROM
 OX_NODE_MSG
 OX_NODE_GROUP
+OX_NODE_MODEL
 OX_JOB_STATUS
 OX_JOB_TIME
 OX_REPO_COMMITREF
 OX_REPO_NAME
 ```
 
-Exec hook recognizes following configuration keys:
+Exec hook recognizes the following configuration keys:
 
-  * `timeout`: hard timeout for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: 60
-  * `async`: influences whether main thread will wait for the command execution. Set this true for long running commands so node pull is not blocked. Default: false
-  * `cmd`: command to run.
+* `timeout`: hard timeout (in seconds) for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: `60`
+* `async`: Execute the command in an asynchronous fashion. The main thread by default will wait for the hook command execution to complete. Set this to `true` for long running commands so node configuration pulls are not blocked. Default: `false`
+* `cmd`: command to run.
 
+### exec hook configuration example
 
-## Hook configuration example
-```
+```yaml
 hooks:
   name_for_example_hook1:
     type: exec
@@ -54,21 +60,31 @@ hooks:
     timeout: 120
 ```
 
-### githubrepo
+## Hook type: githubrepo
+
+The `githubrepo` hook executes a `git push` to a configured `remote_repo` when the specified event is triggered.
 
-This hook configures the repository `remote` and _push_ the code when the specified event is triggerd. If the `username` and `password` are not provided, the `Rugged::Credentials::SshKeyFromAgent` will be used.
+Several authentication methods are supported:
 
-`githubrepo` hook recognizes following configuration keys:
+* Provide a `password` for username + password authentication
+* Provide both a `publickey` and a `privatekey` for ssh key-based authentication
+* Don't provide any credentials for ssh-agent authentication
 
-  * `remote_repo`: the remote repository to be pushed to.
-  * `username`: username for repository auth.
-  * `password`: password for repository auth.
-  * `publickey`: publickey for repository auth.
-  * `privatekey`: privatekey for repository auth.
+The username will be set to the relevant part of the `remote_repo` URI, with a fallback to `git`. It is also possible to provide one by setting the `username` configuration key.
 
-When using groups repositories, each group must have its own `remote` in the `remote_repo` config.
+For ssh key-based authentication, it is possible to set the environment variable `OXIDIZED_SSH_PASSPHRASE` to a passphrase if the private key requires it.
 
-``` yaml
+`githubrepo` hook recognizes the following configuration keys:
+
+* `remote_repo`: the remote repository to be pushed to.
+* `username`: username for repository auth.
+* `password`: password for repository auth.
+* `publickey`: public key file path for repository auth.
+* `privatekey`: private key file path for repository auth.
+
+When using groups, each group must have a unique entry in the `remote_repo` config.
+
+```yaml
 hooks:
   push_to_remote:
     remote_repo:
@@ -77,10 +93,11 @@ hooks:
       firewalls: git@git.intranet:oxidized/firewalls.git
 ```
 
+### githubrepo hook configuration example
 
-## Hook configuration example
+Authenticate with a username and a password without groups in use:
 
-``` yaml
+```yaml
 hooks:
   push_to_remote:
     type: githubrepo
@@ -90,20 +107,37 @@ hooks:
     password: pass
 ```
 
+Authenticate with the username `git` and an ssh key:
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    publickey: /root/.ssh/id_rsa.pub
+    privatekey: /root/.ssh/id_rsa
+```
+
 ## Hook type: awssns
 
 The `awssns` hook publishes messages to AWS SNS topics. This allows you to notify other systems of device configuration changes, for example a config orchestration pipeline. Multiple services can subscribe to the same AWS topic.
 
 Fields sent in the message:
 
-  * `event`: Event type (e.g. `node_success`)
-  * `group`: Group name
-  * `model`: Model name (e.g. `eos`)
-  * `node`: Device hostname
+* `event`: Event type (e.g. `node_success`)
+* `group`: Group name
+* `model`: Model name (e.g. `eos`)
+* `node`: Device hostname
+
 
-Configuration example:
+The AWS SNS hook requires the following configuration keys:
 
-``` yaml
+* `region`: AWS Region name
+* `topic_arn`: ASN Topic reference
+### awssns hook configuration example
+
+```yaml
 hooks:
   hook_script:
     type: awssns
@@ -112,11 +146,6 @@ hooks:
     topic_arn: arn:aws:sns:us-east-1:1234567:oxidized-test-backup_events
 ```
 
-AWS SNS hook requires the following configuration keys:
-
-  * `region`: AWS Region name
-  * `topic_arn`: ASN Topic reference
-
 Your AWS credentials should be stored in `~/.aws/credentials`.
 
 ## Hook type: slackdiff
@@ -125,13 +154,13 @@ The `slackdiff` hook posts colorized config diffs to a [Slack](http://www.slack.
 
 You will need to manually install the `slack-api` gem on your system:
 
-```
+```shell
 gem install slack-api
 ```
 
-Configuration example:
+### slackdiff hook configuration example
 
-``` yaml
+```yaml
 hooks:
   slack:
     type: slackdiff
@@ -140,4 +169,79 @@ hooks:
     channel: "#network-changes"
 ```
 
+Optionally you can disable snippets and post a formatted message, for instance linking to a commit in a git repo. Named parameters `%{node}`, `%{group}`, `%{model}` and `%{commitref}` are available.
+
+```yaml
+hooks:
+  slack:
+    type: slackdiff
+    events: [post_store]
+    token: SLACK_BOT_TOKEN
+    channel: "#network-changes"
+    diff: false
+    message: "%{node} %{group} %{model} updated https://git.intranet/network-changes/commit/%{commitref}"
+```
+
+Note the channel name must be in quotes.
+
+## Hook type: ciscosparkdiff
+
+The `ciscosparkdiff` hook posts config diffs to a [Cisco Spark](https://www.ciscospark.com/) space of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `cisco_spark` gem on your system (see [cisco_spark-ruby](https://github.com/NGMarmaduke/cisco_spark-ruby)) and generate either a [Bot or OAUTH access key](https://developer.ciscospark.com/apps.html), and retrieve the [Spark Space ID](https://developer.ciscospark.com/endpoint-rooms-get.html)
+
+```shell
+gem install cisco_spark
+```
+
+### ciscosparkdiff hook configuration example
+
+```yaml
+hooks:
+  ciscospark:
+    type: ciscosparkdiff
+    events: [post_store]
+    accesskey: SPARK_BOT_API_OR_OAUTH_KEY
+    space: SPARK_SPACE_ID
+    diff: true
+```
+
+Optionally you can disable snippets and post a formatted message, for instance linking to a commit in a git repo. Named parameters `%{node}`, `%{group}`, `%{model}` and `%{commitref}` are available.
+
+```yaml
+hooks:
+  ciscospark:
+    type: ciscosparkdiff
+    events: [post_store]
+    accesskey: SPARK_BOT_API_OR_OAUTH_KEY
+    space: SPARK_SPACE_ID
+    diff: false
+    message: "%{node} %{group} %{model} updated https://git.intranet/network-changes/commit/%{commitref}"
+```
+
+Note the space and access tokens must be in quotes.
+
+## Hook type: xmppdiff
+
+The `xmppdiff` hook posts config diffs to a [XMPP](https://en.wikipedia.org/wiki/XMPP) chatroom of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `xmpp4r` gem on your system:
+
+```shell
+gem install xmpp4r
+```
+
+### xmppdiff hook configuration example
+
+```yaml
+hooks:
+  xmpp:
+    type: xmppdiff
+    events: [post_store]
+    jid: "user@server.tld/resource"
+    password: "password"
+    channel: "room@server.tld"
+    nick: "nickname"
+```
+
 Note the channel name must be in quotes.