oxidized 0.20.0 → 0.28.0

data/docs/Model-Notes/XGS4600-Zyxel.md

@@ -0,0 +1,36 @@
+# ZynOS Configuration
+
+## FTP
+
+FTP access is only possible as admin, other users can login but cannot pull the files.
+For the XGS4600 series the config file is _config_ and not _config-0_
+
+The following line in _oxidized/lib/oxidized/model/zynos.rb_ will need changing
+
+```text
+  cmd 'config-0'
+```
+
+The inclusion of an extra ftp option is also require. Within _input_ add the following
+
+```yaml
+input:
+  ftp:
+    passive: false
+```
+
+## SSH/TelNet
+
+Below is the table from the XGS4600 CLI Reference Guide (Version 3.79~4.50 Edition 1, 07/2017)
+Take this table with a pinch of salt, level 3 will not allow _show running-config_!
+
+Privilege Level | Types of commands at this privilege level
+----------------|-------------------------------------------
+0|Display basic system information.
+3|Display configuration or status.
+13|Configure features except for login accounts, SNMP user accounts, the authentication method sequence and authorization settings, multiple logins, administrator and enable passwords, and configuration information display.
+14|Configure login accounts, SNMP user accounts, the authentication method sequence and authorization settings, multiple logins, and administrator and enable passwords, and display configuration information.
+
+Oxidized can now retrieve your configuration!
+
+Back to [Model-Notes](README.md)

data/docs/Outputs.md

@@ -0,0 +1,190 @@
+# Outputs
+
+## Output: File
+
+Parent directory needs to be created manually, one file per device, with most recent running config.
+
+```yaml
+output:
+  file:
+    directory: /var/lib/oxidized/configs
+```
+
+## Output: Git
+
+This uses the rugged/libgit2 interface. So you should remember that normal Git hooks will not be executed.
+
+For a single repository containing all devices:
+
+```yaml
+output:
+  default: git
+  git:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/devices.git"
+```
+
+And for group-based repositories:
+
+```yaml
+output:
+  default: git
+  git:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/git-repos/default.git"
+```
+
+Oxidized will create a repository for each group in the same directory as the `default.git`. For
+example:
+
+```csv
+host1:ios:first
+host2:nxos:second
+```
+
+This will generate the following repositories:
+
+```bash
+$ ls /var/lib/oxidized/git-repos
+
+default.git first.git second.git
+```
+
+If you would like to use groups and a single repository, you can force this with the `single_repo` config.
+
+```yaml
+output:
+  default: git
+  git:
+    single_repo: true
+    repo: "/var/lib/oxidized/devices.git"
+
+```
+
+## Output: Git-Crypt
+
+This uses the gem git and system git-crypt interfaces. Have a look at [GIT-Crypt](https://www.agwa.name/projects/git-crypt/) documentation to know how to install it.
+Additionally to user and email informations, you have to provide the users ID that can be a key ID, a full fingerprint, an email address, or anything else that uniquely identifies a public key to GPG (see "HOW TO SPECIFY A USER ID" in the gpg man page).
+
+For a single repository containing all devices:
+
+```yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/devices"
+    users:
+      - "0x0123456789ABCDEF"
+      - "<user@example.com>"
+```
+
+And for group-based repositories:
+
+```yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/git-repos/default"
+    users:
+      - "0xABCDEF0123456789"
+      - "0x0123456789ABCDEF"
+```
+
+Oxidized will create a repository for each group in the same directory as the `default`. For
+example:
+
+```csv
+host1:ios:first
+host2:nxos:second
+```
+
+This will generate the following repositories:
+
+```bash
+$ ls /var/lib/oxidized/git-repos
+
+default.git first.git second.git
+```
+
+If you would like to use groups and a single repository, you can force this with the `single_repo` config.
+
+```yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    single_repo: true
+    repo: "/var/lib/oxidized/devices"
+    users:
+      - "0xABCDEF0123456789"
+      - "0x0123456789ABCDEF"
+
+```
+
+Please note that user list is only updated once at creation.
+
+## Output: Http
+
+The HTTP output will POST a config to the specified HTTP URL. Basic username/password authentication is supported.
+
+Example HTTP output configuration:
+
+```yaml
+output:
+  default: http
+  http:
+    user: admin
+    password: changeit
+    url: "http://192.168.162.50:8080/db/coll"
+```
+
+## Output types
+
+If you prefer to have different outputs in different files and/or directories, you can easily do this by modifying the corresponding model. To change the behaviour for IOS, you would edit `lib/oxidized/model/ios.rb` (run `gem contents oxidized` to find out the full file path).
+
+For example, let's say you want to split out `show version` and `show inventory` into separate files in a directory called `nodiff` which your tools will not send automated diffstats for. You can apply a patch along the lines of
+
+```text
+-  cmd 'show version' do |cfg|
+-    comment cfg.lines.first
++  cmd 'show version' do |state|
++    state.type = 'nodiff'
++    state
+
+-  cmd 'show inventory' do |cfg|
+-    comment cfg
++  cmd 'show inventory' do |state|
++    state.type = 'nodiff'
++    state
++  end
+
+-  cmd 'show running-config' do |cfg|
+-    cfg = cfg.each_line.to_a[3..-1].join
+-    cfg.gsub! /^Current configuration : [^\n]*\n/, ''
+-    cfg.sub! /^(ntp clock-period).*/, '! \1'
+-    cfg.gsub! /^\ tunnel\ mpls\ traffic-eng\ bandwidth[^\n]*\n*(
++  cmd 'show running-config' do |state|
++    state = state.each_line.to_a[3..-1].join
++    state.gsub! /^Current configuration : [^\n]*\n/, ''
++    state.sub! /^(ntp clock-period).*/, '! \1'
++    state.gsub! /^\ tunnel\ mpls\ traffic-eng\ bandwidth[^\n]*\n*(
+                   (?:\ [^\n]*\n*)*
+                   tunnel\ mpls\ traffic-eng\ auto-bw)/mx, '\1'
+-    cfg
++    state = Oxidized::String.new state
++    state.type = 'nodiff'
++    state
+```
+
+which will result in the following layout
+
+```text
+diff/$FQDN--show_running_config
+nodiff/$FQDN--show_version
+nodiff/$FQDN--show_inventory
+```

data/docs/Ruby-API.md

@@ -0,0 +1,199 @@
+# Ruby API
+
+The following objects exist in Oxidized.
+
+## Input
+
+* gets config from nodes
+* must implement 'connect', 'get', 'cmd'
+* 'ssh', 'telnet', 'ftp', 'tftp', 'http' implemented
+
+#### http
+ * Communicates with a device over http/https
+ * Configurable variables from within model @username, @password, @headers.
+ * @username,@password are used in a Basic Authentication method.
+ * @headers is a Hash of key value pairs of headers to pass along with the request.
+ * Within the sources config under input you define a YAML stanza like the below, this will tell Oxidized to validate certificates on the request
+```yaml
+input:
+   http:
+     ssl_verify: true
+```
+
+## Output
+
+* stores config
+* must implement 'store' (may implement 'fetch')
+* 'git' and 'file' (store as flat ascii) implemented
+
+## Source
+
+* gets list of nodes to poll
+* must implement 'load'
+* source can have 'name', 'model', 'group', 'username', 'password', 'input', 'output', 'prompt' for each device.
+  * `name` - name of the device
+  * `model` - model to use ('ios', 'junos', etc).The model is loaded dynamically by the first node of that model type. (Also default in config file)
+  * `input` - method to acquire config, loaded dynamically as needed (Also default in config file)
+  * `output` - method to store config, loaded dynamically as needed (Also default in config file)
+  * `prompt` - prompt used for node (Also default in config file, can be specified in model too)
+* 'sql', 'csv' and 'http' (supports any format with single entry per line, like router.db)
+
+## Model
+
+### At the top level
+
+A model may use several methods at the top level in the class. `cfg` is
+executed in input/output/source context. `cmd` is executed within an instance
+of the model.
+
+#### `cfg`
+
+`cfg` may be called with a list of methods (`:ssh`, `:telnet`) and a block with
+zero parameters.  Calling `cfg` registers the given access methods and calling
+it at least once is required for a model to work.
+
+The block may contain commands to change some behaviour for the given methods
+(e.g. calling `post_login` to disable the pager).
+
+Supports [monkey patching](#monkey-patching).
+
+#### `cmd`
+
+Is used to specify commands that should be executed on a model in order to
+gather its configuration. It can be called with:
+
+* Just a string
+* A string and a block
+* `:all` and a block
+* `:secret` and a block
+
+The block takes a single parameter `cfg` containing the output of the command
+being processed.
+
+Calling `cmd` with just a string will emit the output of the command given in
+that string as configuration.
+
+Calling `cmd` with a string and a block will pass the output of the given
+command to the block, then emit its return value (that must be a string) as
+configuration.
+
+Calling `cmd` with `:all` and a block will pass all command output through this
+block before emitting it. This is useful if some cleanup is required of the
+output of all commands.
+
+Calling `cmd` with `:secret` and a block will pass all configuration to the
+given block before emitting it to hide secrets if secret hiding is enabled. The
+block should replace any secrets with `'<hidden>'` and return the resulting
+string.
+
+Execution order is `:all`, `:secret`, and lastly the command specific block, if
+given.
+
+Supports [monkey patching](#monkey-patching).
+
+#### `comment`
+
+Called with a single string containing the string to prepend for comments in
+emitted configuration for this model.
+
+If not specified the default of `'# '` will be used (note the trailing space).
+
+#### `prompt`
+
+Is called with a regular expression that is used to detect when command output
+ends after a command has been executed.
+
+If not specified, a default of `/^([\w.@-]+[#>]\s?)$/` is used.
+
+#### `expect`
+
+Called with a regular expression and a block. The block takes two parameters:
+the regular expression, and the data containing the match.
+
+The passed data is replaced by the return value of the block.
+
+`expect` can be used to, for example, strip escape sequences from output before
+it's further processed.
+
+Supports [monkey patching](#monkey-patching).
+
+### At the second level
+
+The following methods are available:
+
+#### `comment`
+
+Used inside `cmd` invocations. Comments out every line in the passed string and
+returns the result.
+
+#### `password`
+
+Used inside `cfg` invocations to specify the regular expression used to detect
+the password prompt. If not specified, the default of `/^Password/` is used.
+
+#### `post_login`
+
+Used inside `cfg` invocations to specify commands to run once Oxidized has
+logged in to the device. Takes one argument that is either a block (taking zero
+parameters) or a string containing a command to execute.
+
+This allows `post_login` to be used for any model-specific items prior to
+running the regular commands. This could include disabling the output pager
+or timestamp outputs that would cause constant differences.
+
+Supports [monkey patching](#monkey-patching).
+
+#### `pre_logout`
+
+Used to specify commands to run before Oxidized closes the connection to the
+device. Takes one argument that is either a block (taking zero parameters) or a
+string containing a command to execute.
+
+This allows `pre_logout` to be used to 'undo' any changes that may have been
+needed via `post_login` (restore pager output, etc.)
+
+Supports [monkey patching](#monkey-patching).
+
+#### `send`
+
+Usually used inside `expect` or blocks passed to `post_login`/`pre_logout`.
+Takes a single parameter: a string to be sent to the device.
+
+### Monkey patching
+
+Several model blocks accept behavior-modifying arguments that make monkey
+patching existing blocks easier. This is primarily useful when a user-supplied
+model aims to override or extend existing behavior of a model included in Oxidized.
+
+This functionality is supported by `cfg`, `cmd`, `pre_*`, `post_*`, and `expect`
+blocks.
+
+#### `clear: true`
+
+Resets the existing block, allowing the user to completely override its contents.
+
+#### `prepend: true`
+
+Ensures that the contents of the block are prepended, rather than appended (the
+default) to an existing block.
+
+### `String` convenience methods
+
+Since configuration processing tasks are occasionally similar across models,
+Oxidized provides an extended [`String`](/lib/oxidized/string.rb) class with the
+intention of providing convenience methods and eliminating code duplication.
+
+#### `cut_tail`
+
+Returns a multi-line string without the last line, or an empty string if only a
+single line was present.
+
+#### `cut_head`
+
+Returns a multi-line string without the first line, or an empty string if only a
+single line was present.
+
+#### `cut_both`
+
+Returns a multi-line string without the first and last lines, or an empty string
+if fewer than three lines were present.

data/docs/Sources.md

@@ -0,0 +1,171 @@
+# Sources
+
+## Source: CSV
+
+One line per device, colon separated. If `ip` isn't present, a DNS lookup will be done against `name`.  For large installations, setting `ip` will dramatically reduce startup time.
+
+```yaml
+source:
+  default: csv
+  csv:
+    file: /var/lib/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      ip: 1
+      model: 2
+      username: 3
+      password: 4
+    vars_map:
+      enable: 5
+```
+
+Example csv `/var/lib/oxidized/router.db`:
+
+```text
+rtr01.local:192.168.1.1:ios:oxidized:5uP3R53cR3T:T0p53cR3t
+```
+
+If you would like to use a GPG encrypted file as the source then you can use the following example:
+
+```yaml
+source:
+  default: csv
+  csv:
+    file: ~/.config/oxidized/router.db
+    delimiter: !ruby/regexp /:/
+    gpg: true
+    gpg_password: 'password'
+    map:
+      name: 0
+      model: 1
+```
+
+Please note, if you are running GPG v2 then you will be prompted for your gpg password on start up, if you use GPG >= 2.1 then you can add the following config to stop that behaviour:
+
+Within `~/.gnupg/gpg-agent.conf`
+
+```text
+allow-loopback-pinentry
+```
+
+and within: `~/.gnupg/gpg.conf`
+
+```text
+pinentry-mode loopback
+```
+
+## Source: SQL
+
+ Oxidized uses the `sequel` ruby gem. You can use a variety of databases that aren't explicitly listed. For more information visit https://github.com/jeremyevans/sequel Make sure you have the correct adapter!
+
+**NOTE** - Many database engines have reserved keywords that may conflict with Oxidized configuration field names (such as 'name', 'group', etc). Pay attention to any names that are used and observed proper quoting methods to avoid errors or unpredictable results.
+
+## Source: MYSQL
+
+`sudo apt-get install libmysqlclient-dev`
+
+The values correspond to your fields in the DB such that ip, model, etc are field names in the DB
+
+```yaml
+source:
+  default: sql
+  sql:
+    adapter: mysql2
+    database: oxidized
+    table: nodes
+    user: root
+    password: rootpass
+    map:
+      name: ip
+      model: model
+      username: username
+      password: password
+    vars_map:
+      enable: enable
+```
+
+## Source: SQLite
+
+One row per device, filtered by hostname.
+
+```yaml
+source:
+  default: sql
+  sql:
+    adapter: sqlite
+    database: "/var/lib/oxidized/nodes.db"
+    table: nodes
+    map:
+      name: fqdn
+      model: model
+      username: username
+      password: password
+    vars_map:
+      enable: enable
+```
+
+## Custom SQL Query Support
+
+You may also implement a custom SQL query to retrieve the nodelist using  SQL syntax with the `query:` configuration parameter under the `sql:` stanza.
+
+### Custom SQL Query Examples
+
+You may have a table named `nodes` which contains a boolean to indicate if the nodes should be enabled (fetched via oxidized). This can be used in the custom SQL query to avoid fetching from known impacted nodes.
+
+In your configuration, you would add the `query:` parameter and specify the SQL query. Make sure to put this within the `sql:` configuration section.
+
+```sql
+query: "SELECT * FROM nodes WHERE enabled = True"
+```
+
+Since this is an SQL query, you can also provide a more advanced query to assist in more complicated oxidized deployments. The exact deployment is up to you on how you design your database and oxidized fetchers.
+
+In this example we limit the nodes to two "POPs" of `mypop1` and `mypop2`. We also require the nodes to have the `enabled` boolean set to `True`.
+
+```sql
+query: "SELECT * FROM nodes WHERE pop IN ('mypop1','mypop2') AND enabled = True"
+```
+
+The order of the nodes returned will influence the order that nodes are fetched by oxidized. You can use standard SQL `ORDER BY` clauses to influence the node order.
+
+You should always test your SQL query before using it in the oxidized configuration as there is no syntax or error checking performed before sending it to the database engine.
+
+Consult your database documentation for more information on query language and table optimization.
+
+## Source: HTTP
+
+One object per device.
+
+HTTP Supports basic auth, configure the user and pass you want to use under the http: section.
+
+```yaml
+source:
+  default: http
+  http:
+    url: https://url/api
+    scheme: https
+    delimiter: !ruby/regexp /:/
+    user: username
+    pass: password
+    map:
+      name: hostname
+      model: os
+      username: username
+      password: password
+    vars_map:
+      enable: enable
+    headers:
+      X-Auth-Token: 'somerandomstring'
+```
+
+You can also pass `secure: false` if you want to disable ssl certificate verification:
+
+```yaml
+source:
+  default: http
+  http:
+    url: https://url/api
+    scheme: https
+    secure: false
+```