oxidized 0.29.1 → 0.37.0

data/docs/Creating-Models.md

@@ -1,33 +1,96 @@
 # Creating and Extending Models
 
-Oxidized supports a growing list of [operating system types](Supported-OS-Types.md). Out of the box, most model implementations collect configuration data. Some implementations also include a conservative set of additional commands that collect basic device information (device make and model, software version, licensing information, ...) which are appended to the configuration as comments.
+Oxidized supports a growing list of
+[operating system types](Supported-OS-Types.md). Out of the box, most model
+implementations collect configuration data. Some implementations also include a
+conservative set of additional commands that collect basic device information
+(device make and model, software version, licensing information, ...) which are
+appended to the configuration as comments.
+
+A user may wish to extend an existing model to collect the output of additional
+commands. Oxidized offers smart loading of models in order to facilitate this
+with ease, without the need to introduce changes to the upstream source code.
+
+This methodology allows local site changes to be preserved during Oxidized
+version updates / gem updates. It also enables convenient local development
+of new models.
+
+## Index
+- [Creating a new model](#creating-a-new-model)
+- [Typical Tasks and Solutions](#typical-tasks-and-solutions)
+  - [Handling 'enable' mode](#handling-enable-mode)
+  - [Remove ANSI escape codes](#remove-ansi-escape-codes)
+  - [Conditional commands](#conditional-commands)
+- [Extending an existing model with a new command](#extending-an-existing-model-with-a-new-command)
+- [Create unit tests for the model](#create-unit-tests-for-the-model)
+- [Advanced features](#advanced-features)
+- [Monkey-patching blocks in existing models](#monkey-patching-blocks-in-existing-models)
+- [Help](#help)
 
-A user may wish to extend an existing model to collect the output of additional commands. Oxidized offers smart loading of models in order to facilitate this with ease, without the need to introduce changes to the upstream source code.
+## Creating a new model
+### Use the defacto model
+Before you create a new model, check the
+[defacto model](/lib/oxidized/model/defacto.rb). It aims to work with a lot of
+devices copying the defacto standard set by the cisco CLI:
+- login per ssh or telnet
+- disable pager
+- execute `show running-config`
+- run both `exit` and `logout`
+
+If you want to extend the defacto model, you can define your own model file,
+and inherit from `Defacto`:
+```ruby
+require 'oxidized/model/defacto'
 
-This methodology allows local site changes to be preserved during Oxidized version updates / gem updates. It also enables convenient local development of new models.
+class OxiWare < Defacto
+  # Redefine how to process show running-config
+  def process_config(cfg)
+    cfg.gsub(/.*service timestamps.*/, '')
+  end
 
-## Creating a new model
+  # run more commands. They will be run before 'show running-config'
+  cmd "show cdp neighbors" do |cfg|
+    comment cfg
+  end
 
+  # Send "enable" password if defined in vars("enable")
+  macro :enable
+end
+```
+
+If your model doesn't support "show running-config", or if you prefer to avoid
+the dependency of the defacto model, you will need to write a model "from
+scratch" as explained in the next section.
+
+### Create a new model from scratch
 An Oxidized model, at minimum, requires just three elements:
 
 * A model file, this file should be placed in the ~/.config/oxidized/model directory and named after the target OS type.
 * A class defined within this file with the same name as the file itself that inherits from `Oxidized::Model`, the base model class.
 * At least one command that will be executed and the output of which will be collected by Oxidized.
 
-A bare-bone example for a fictional model running the OS type `rootware` could be introduced by creating the file `~/.config/oxidized/model/rootware.rb`, with the following content:
+A bare-bone example for a fictional model running the OS type `rootware` could
+be introduced by creating the file `~/.config/oxidized/model/rootware.rb`, with
+the following content:
 
 ```ruby
 class RootWare < Oxidized::Model
   using Refinements
   
   cmd 'show complete-config'
+
+  cfg :ssh do
+    pre_logout 'exit'
+  end
+end
 ```
 
 This model, as-is will:
 
-* Log into the device and expect the default prompt.
+* Log into the device with ssh and expect the default prompt.
 * Upon matching it, execute the command `show complete-config`
 * Collect the output.
+* Logout with the command `exit`
 
 It is often useful to, at minimum, define the following additional elements for any newly introduced module:
 
@@ -40,6 +103,184 @@ The API documentation contains a list of [methods](https://github.com/ytti/oxidi
 
 A more fleshed out example can be found in the `IOS` and `JunOS` models.
 
+## Typical Tasks and Solutions
+
+### Keep or Remove Lines Returned from a Command
+To make command output cleaner, you can remove unwanted lines or keep only
+specific ones.
+
+Most devices echo the executed command on the first line and display a
+prompt on the last line. To remove these for all commands, use
+[cut_both](Ruby-API.md#cut_both):
+```ruby
+  cmd :all do |cfg|
+    cfg.cut_both
+  end
+```
+
+You can also use the macro `clean :cut`, which does the same:
+```ruby
+  clean :cut
+```
+
+If you want to keep only relevant lines, use
+[keep_lines](Ruby-API.md#keep_lines):
+```ruby
+  cmd 'show interfaces transceiver' do |cfg|
+    cfg = cfg.keep_lines [
+      'SFP Information',
+      /Vendor (Name|Serial Number)/
+    ]
+    comment cfg + "\n"
+  end
+```
+
+If you want to suppress specific lines,
+use [reject_lines](Ruby-API.md#reject_lines):
+```ruby
+  cmd 'show running-config' do |cfg|
+    cfg.reject_lines [
+      'System Up Time',
+      /Current .* Time:/
+    ]
+  end
+```
+
+### Handling 'enable' mode
+Some devices need to send an 'enable' command and an enable password.
+
+You can use the `macro :enable` command to implement this:
+```ruby
+class IOS < Oxidized::Model
+  using Refinements
+  # ... Code ...
+  macro :enable
+end
+```
+
+`macro :enable` takes options:
+- `regex`; the regex to match the password prompt (default: `/password/i`)
+- `inputs`: a symbol or a list of symbols for which inputs enable should be activated (default: %i[telnet ssh])
+- `command`: the command needed to access privileged mode (default: `enable`)
+
+If one would want to access a german linux box as root, a minimal model would be:
+```ruby
+class GermanLinux < Oxidized::Model
+  using Refinements
+
+  prompt /^(\w.*|\W.*)[:#$] /
+  comment '# '
+
+  cmd "id"
+
+  cfg :ssh do
+    pre_logout 'exit'
+    pre_logout 'exit'
+  end
+
+  macro :enable, inputs: :ssh, command: "su -", regex: /Passwort: /
+end
+```
+
+The macro (with defaults) implements following code:
+```ruby
+  cfg :telnet, :ssh do
+    post_login do
+      if vars(:enable) == true
+        cmd "enable"
+      elsif vars(:enable)
+        cmd "enable", /password/i
+        cmd vars(:enable)
+      end
+    end
+  end
+```
+
+
+### Remove ANSI Escape Codes
+Some devices produce [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Control_Sequence_Introducer_commands)
+to enhance the appearance of their output.
+However, this can make prompt matching difficult and some of these ANSI escape
+codes might end up in the resulting configuration.
+
+You can remove most ANSI escape codes by inserting the following line in your
+model:
+```ruby
+  clean :escape_codes
+```
+
+When using clean `:escape_codes`, you don't have to worry about escape codes 
+in your prompt regexp, as they will be removed before the prompt detection runs.
+
+If it doesn't work for your model, please open an issue and provide a
+[device simulation file](/docs/DeviceSimulation.md) so that we can adapt the
+code.
+
+### Conditional commands
+Some times, you have to run commands depending on the output of the device or
+a configured variable. For this, there are at least three solutions.
+
+#### Nested `cmd`
+You can nest `cmd` inside [`cmd` blocks](Ruby-API.md#cmd), the following example
+is taken from [nxos.rb](/lib/oxidized/model/nxos.rb):
+```ruby
+  cmd 'show inventory all' do |cfg|
+    if cfg.match? /^% Invalid .* at '\^' marker\./
+      # 'show inventory all' isn't supported on older versions (See Issues #3657, #3779)
+      cfg = cmd 'show inventory'
+    end
+    comment cfg
+  end
+```
+
+#### pre/post blocks
+After all the [`cmd` blocks](Ruby-API.md#cmd) have been run, the [`pre`
+and `post` blocks](Ruby-API.md#pre--post) are run. The following example is
+taken from [junos.rb](/lib/oxidized/model/junos.rb):
+```ruby
+  post do
+    out = String.new
+    case @model
+    when 'mx960'
+      out << cmd('show chassis fabric reachability') { |cfg| comment cfg }
+    when /^(ex22|ex3[34]|ex4|ex8|qfx)/
+      out << cmd('show virtual-chassis') { |cfg| comment cfg }
+    when /^srx/
+      out << cmd('show chassis cluster status') do |cfg|
+        cfg.lines.count <= 1 && cfg.include?("error:") ? String.new : comment(cfg)
+      end
+    end
+    out
+  end
+```
+
+In [pre/post blocks](Ruby-API.md#pre--post), you can also use dynamic generated
+commands, for example in [eatonnetwok.rb](/lib/oxidized/model/eatonnetwork.rb):
+```ruby
+  post do
+    cmd "save_configuration -p #{@node.auth[:password]}"
+  end
+```
+
+#### Conditional `cmd`
+The `cmd "string"` method for accepts a lambda function via the `:if` argument
+to execute the command only when the lambda evaluates to true.
+The lambda function is evaluated at runtime in the instance context.
+
+```ruby
+  cmd 'conditional command', if: lambda {
+    # Use lambda when multiple lines are needed
+    vars("condition")
+  } do |cfg|
+    @run_second_command = "go"
+    comment cfg
+  end
+
+  cmd 'second command', if: -> { @run_second_command == "go" } do |cfg|
+    comment cfg
+  end
+```
+
 ## Extending an existing model with a new command
 
 The example below can be used to extend the `JunOS` model to collect the output of `show interfaces diagnostics optics` and append the output to the configuration file as a comment. This command retrieves DOM information on pluggable optics present in a `JunOS`-powered chassis.
@@ -72,7 +313,18 @@ Intuitively, it is also possible to:
 * Create a completely new model, with a new name, for a new operating system type.
 * Testing/validation of an updated model from the [Oxidized GitHub repo models](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model) by placing an updated model in the proper location without disrupting the gem-supplied model files.
 
-## Advanced features
+## Create Unit Tests for the Model
+If you want the model to be integrated into Oxidized, you can
+[submit a pull request on GitHub](https://github.com/ytti/oxidized/pulls).
+This is a greatly appreciated submission, as there are probably other users
+using the same network device as you are.
+
+A good (and optional) practice for submissions is to provide a
+[unit test for your model](/docs/ModelUnitTests.md). This reduces the risk that
+further developments could break it, and facilitates debugging issues without
+having access to a physical network device for the model.
+
+## Advanced feature: output type
 
 The loosely-coupled architecture of Oxidized allows for easy extensibility in more advanced use cases as well.
 
@@ -124,19 +376,19 @@ Examples:
 
 ```ruby
 cmd :secret, clear: true do
-  ... "(new code for secret removal which replaces the existing :secret definition in the model)" ...
+  # ... "(new code for secret removal which replaces the existing :secret definition in the model)" ...
 end
 ```
 
 ```ruby
 cmd 'show version', clear: true do |cfg|
-  ... "(new code for parsing 'show version', replaces the existing definition in the model)" ...
+  # ... "(new code for parsing 'show version', replaces the existing definition in the model)" ...
 end
 ```
 
 ```ruby
 cmd :ssh, prepend: true do
-  ... "(code that should run first, before any code in the existing :ssh definition in the model)" ...
+  # ... "(code that should run first, before any code in the existing :ssh definition in the model)" ...
 end
 ```
 

data/docs/DeviceSimulation.md

@@ -0,0 +1,196 @@
+# Device Simulation
+Oxidized supports [150+ devices](/docs/Supported-OS-Types.md).
+
+No developer has access to all of these devices, which makes the task of
+maintaining Oxidized difficult:
+
+- Issues can't be resolved because the developer has no access to the device.
+- Further developments can produce regressions.
+
+In order to address this, we can simulate the devices. An example of a
+simulation is the [model unit tests](/spec/model), but one could also simulate a
+device within an SSH server.
+
+The simulation of devices is currently focused on SSH-based devices. This may be
+extended to other inputs like Telnet or FTP in the future.
+
+## YAML Simulation Data
+The underlying data for the simulation is a [YAML](https://yaml.org/) file in
+which we store all relevant information about the device. The most important
+information is the responses to the commands used in the Oxidized models.
+
+The YAML simulation files are stored under
+[/spec/model/data/](/spec/model/data/), with the naming convention
+`<model>#<description>#simulation.yaml`, where `<model>` is the lowercase name
+of the Oxidized model and `<description>` is the name of the test case.
+`<description>` is generally formatted as `<hardware>_<software>` or
+`<hardware>_<software>_<information>`.
+
+### Creating a YAML Simulation File with device2yaml.rb
+A device does not only output the ASCII text we can see in the console.
+It adds ANSI escape codes for nice colors, bold and underline, \r, and so on.
+These are key factors in prompt issues, so they must be represented in the YAML
+file. We use the Ruby string format with interpolations like \r, \e, and so on.
+Another important point is trailing spaces at the end of lines. Some text
+editors automatically remove trailing spaces, so we code them with \x20.
+
+Although a YAML file could be written by hand, this is quite a tedious task to
+catch all the extra codes and code them into YAML. This can be automated with
+the Ruby script [extra/device2yaml.rb](/extra/device2yaml.rb).
+
+`device2yaml.rb` needs Ruby and the gem
+[net-ssh](https://rubygems.org/gems/net-ssh/) to run. On Debian, you can install
+them with `sudo apt install ruby-net-ssh`.
+
+Run `extra/device2yaml.rb`, the online help tells you the options.
+```
+oxidized$ extra/device2yaml.rb
+Missing a host to connect to...
+
+Usages:
+- device2yaml.rb [user@]host -i file [options]
+- device2yaml.rb [user@]host -c "command1
+  command2
+  command3" [options]
+
+-i and -c are mutualy exclusive, one must be specified
+
+[options]:
+    -c, --commands "command list"    specify the commands to be run
+    -i, --input file                 Specify an input file for commands to be run
+    -o, --output file                Specify an output YAML-file
+    -t, --timeout value              Specify the idle timeout beween commands (default: 5 seconds)
+    -e, --exec-mode                  Run ssh in exec mode (without tty)
+    -u, --unordered                  The YAML simulation should not enforce an order of the commands
+    -h, --help                       Print this help
+```
+
+- `[user@]host` specifies the user and host to connect to the device. The
+password will be prompted interactively by the script. If you do not specify a
+user, it will use the user executing the script.
+- The commands that will be run on the device must be defined in
+`deviceyaml.rb`. You can give the commands online with `-c` or read them from a
+file (one line per command) with `-i`. The commands should match exactly the
+ones of the model (no abbreviations) and include the commands of the
+`post_login` and `pre_logout` sections. When using `-c` and editing the shell
+command line, `CTRL-V CTRL-J` is very useful to add a line break.
+- `device2yaml.rb` waits an idle timeout after the last received data
+before sending the next command. The default is 5 seconds. If your device makes
+a longer pause than 5 seconds before or within a command, you will see that the
+output of the command is shortened or slips into the next command in the YAML
+file. You will have to change the idle timeout to a greater value to address
+this.
+- When run without the output argument, `device2yaml.rb` will only print the SSH
+output to the standard output. You must use `-o <model#HW_SW#simulation.yaml>`
+to store the collected data in a YAML file.
+- If your Oxidized model uses SSH exec mode (look for `exec true` in the model),
+you will have to use the option `-e` to run `device2yaml.rb` in SSH exec mode.
+- The default behavior is to create a YAML file in which the commands must
+  appear in the order used in the Oxidized model. This is useful for simulating
+  devices that paginate output. To allow any order or include more commands than
+  the model uses, use the option `-u`. Note that the `unordered` mode may not
+  produce a useful YAML file when combined with user input (see
+  [Interactive Mode](#interactive-mode) below).
+
+Note that `device2yaml.rb` takes some time to run because of the idle timeout of
+(default) 5 seconds between each command. You can press the "Escape" key if you
+know there is no more data to come for the current command (when you see the
+prompt for the next command), and the script will stop waiting and directly
+process the next command.
+
+
+Running the script against an ios device would look like:
+```shell
+extra/device2yaml.rb oxidized@r61 -c "terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit" -o spec/model/data/ios#C8200L_16.12.1#simulation.yaml
+```
+### Publishing the YAML Simulation File to Oxidized
+Publishing the YAML simulation file of your device helps maintain Oxidized. This
+task may take some time, and we are very grateful that you take this time for
+the community!
+
+You should pay attention to removing or replacing anything you don't want to
+share with the rest of the world, for example:
+
+- Passwords
+- IP Addresses
+- Serial numbers
+
+You can also shorten the configuration if you want - we don't need 48 times the
+same configuration for each interface, but it doesn't hurt either.
+
+Take your time, this is an important task: after you have uploaded your file on
+GitHub, it may be impossible to remove it.
+You can use search/replace to make consistent and faster changes, for example
+change the hostname everywhere.
+
+The YAML simulation files are stored under
+[/spec/model/data/](/spec/model/data/), with the naming convention
+`<model>#<description>#simulation.yaml`, where `<model>` is the lowercase name
+of the Oxidized model and `<description>` is the name of the test case.
+`<description>` is generally formatted as `<hardware>_<software>` or
+`<hardware>_<software>_<information>`.
+
+Using a correct name for the file is important to ensure it is included in
+automatic model unit tests.
+
+Examples:
+
+- spec/model/data/aoscx#R0X25A-6410_FL.10.10.1100#simulation.yaml
+- spec/model/data/asa#5512_9.12-4-67_single-context#simulation.yaml
+- spec/model/data/ios#C9200L-24P-4G_17.09.04a#simulation.yaml
+
+When you are finished, commit and push to your forked repository on GitHub, and
+submit a Pull Request. Thank you for your help!
+
+### Interactive Mode
+The `device2yaml.rb` script is basic and sometimes needs some help, especially
+when dealing with a device that sends its output page by page and requires you
+to press space for the next page. `device2yaml.rb` does not know how to handle
+this.
+
+While `device2yaml.rb` is running, you can type anything on the keyboard, and it
+will be sent to the remote device. So you can press space or 'n' to get the next
+page.
+
+You can also use this to enter an enable password.
+
+Every key press will be recorded in the YAML file, so that it can be used
+in the simulation afterwards, especialy for devices that paginate output. You
+may need to clean the YAML file manually if you don't want some input (such
+as passwords) to be included.
+
+If you press the "Esc" key, `device2yaml.rb` will not wait for the idle timeout
+and will process the next command right away.
+
+### YAML Format
+The YAML file has two sections:
+- init_prompt: describing the lines sent by the device before we can send a
+command. It usually includes MOTD banners and must include the first prompt.
+- commands: the commands the Oxidized model sends to the network device and
+their outputs.
+
+The outputs are multiline and use YAML block scalars (`|`), with the trailing \n
+removed (`-` after `|`). The outputs include the echo of the given command and
+the next prompt. Escape characters are coded in Ruby style (\n, \r...).
+
+Here is a shortened example of a YAML file:
+```yaml
+---
+init_prompt: |-
+  \e[4m\rLAB-R1234_Garderos#\e[m\x20
+commands:
+  show system version: |-
+    show system version
+    grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
+    \e[4m\rLAB-R1234_Garderos#\e[m\x20
+# ...
+  exit: ""
+```
+
+