oxidized 0.30.1 → 0.31.0

data/examples/device-simulation/README.md

@@ -0,0 +1,173 @@
+# Device simulation
+Oxidized supports [150+ devices](/docs/Supported-OS-Types.md).
+No developer has access to all of these devices, which makes the task of
+maintaining Oxidized difficult:
+
+- issues can't be resolved because the developer has no access to the device.
+- further developments can produce regressions.
+
+In order to address this, we can simulate the devices. An example for a
+simulation are the [model unit tests](/spec/model) but one could also simulate
+a device within a ssh server.
+
+The simulation of devices is currently focused on ssh-based devices. This may
+be extended to other inputs like telnet or ftp in the future.
+
+## YAML Simulation Data
+The underlying data for the simulation is a [YAML](https://yaml.org/) file in
+which we store all relevant information about the device. The most important
+information is the responses to the commands used in the oxidized models.
+
+The YAML simulation files are stored under
+[/examples/device-simulation/yaml/](/examples/device-simulation/yaml/).
+
+### Creating a YAML file with device2yaml.rb
+A device does not only output the ASCII text we can see in the console.
+It adds ANSI-escape code for nice colors, bold and underline, \r and so on.
+These are key factors in prompt issues so they must be represented in the YAML
+file. We use the ruby string format with interpolations like \r \e and so on.
+Another important point is trailing spaces at the end of lines. Some text
+editors automatically remove trailing spaces, so we code them with \x20.
+
+Although a YAML file could be written by hand, this is quite a tedious task to
+catch all the extra codes and code them into YAML. This can be
+automated with the ruby script
+[device2yaml.rb](/examples/device-simulation/device2yaml.rb).
+
+`device2yaml.rb` needs ruby and the gem
+[net-ssh](https://rubygems.org/gems/net-ssh/) to run. On debian, you can install
+them with `sudo apt install ruby-net-ssh`
+
+Run `device2yaml.rb` in the directory `/examples/device-simulation/`, the
+online help tells you the options.
+```
+device-simulation$ ./device2yaml.rb
+Missing a host to connect to...
+
+Usage: device2yaml.rb [user@]host [options]
+    -c, --cmdset file                Mandatory: specify the commands to be run
+    -o, --output file                Specify an output YAML-file
+    -t, --timeout value              Specify the idle timeout beween commands (default: 5 seconds)
+    -e, --exec-mode                  Run ssh in exec mode (without tty)
+    -h, --help                       Print this help
+```
+
+- `[user@]host` specifies the user and host to connect to the device. The
+password will be prompted interactively by the script. If you do not specify a
+user, it will use the user executing the script.
+- You must list the commands you want to run on the device in a file. Just
+enter one command per line. It is important that you enter exactly the commands
+used by the oxidized model, and no abbreviation like `sh run`. Do not forget
+to insert the `post_login` commands at the beginning if the model has some and
+also the `pre_logout`commands at the end.
+Predefined command sets for some models are stored in
+`/examples/device-simulation/cmdsets`.
+- `device2yaml.rb` waits an idle timeout after the last received data before
+sending the next command. The default is 5 seconds. If your device makes a
+longer pause than 5 seconds before or within a command, you will see that the
+output of the command is shortened or slips into the next command in the yaml
+file. You will have to change the idle timeout to a greater value to address
+this.
+- When run without the output argument, `device2yaml.rb` will only print the ssh
+output to the standard output. You must use `-o <model_HW_SW.yaml>` to store the
+collected data in a YAML file.
+- If your oxidized model uses ssh exec mode (look for `exec true` in the model),
+you will have to use the option `-e` to run device2yaml in ssh exec mode.
+
+Note that `device2yaml.rb` takes some time to run because of the idle
+timeout of (default) 5 seconds between each command. You can press the "Escape"
+key if you know there is no more data to come for the current command (when you
+see the prompt for the next command), and the script will stop waiting and
+directly process the next command.
+
+Here are two examples of how to run the script:
+```shell
+./device2yaml.rb OX-SW123.sample.domain -c cmdsets/aoscx -o yaml/aoscx_R8N85A-C6000-48G-CL4_PL.10.08.1010.yaml
+./device2yaml.rb admin@r7 -c cmdsets/routeros -e -o yaml/routeros_CHR_7.10.1.yaml
+```
+
+### Publishing the YAML simulation file to oxidized
+Publishing the YAML simulation file of your device helps maintain oxidized.
+This task may take some time, and we are very grateful that you take this time
+for the community!
+
+You should pay attention to removing or replacing anything you don't want to
+share with the rest of the world, for example:
+
+- Passwords
+- IP Adresses
+- Serial numbers
+
+You can also shorten the configuration if you want - we don't need 48 times the
+same config for each interface, but it doesn't hurt either.
+
+Take your time, this is an important task: after you have
+uploaded your file on github, it may be impossible to remove it. You can use
+search/replace to make consistent and faster changes (change the hostname).
+
+You can leave the section `oxidized_output` unchanged, it is only used for
+[model unit tests](/spec/model). You will find an explanation of how to produce
+the `oxidized_output`-section in the README.md there.
+
+The YAML simulation file should be stored under
+[/examples/device-simulation/yaml/](/examples/device-simulation/yaml/. It
+should be named so that it can be easily recognized: model, hardware type,
+software version and optionally a description if you need to differentiate two
+YAML files:
+
+- #model_#hardware_#software.yaml
+- #model_#hardware_#software_#description.yaml
+
+Examples:
+
+- garderos_R7709_003_006_068.yaml
+- iosxe_C9200L-24P-4G_17.09.04a.yaml
+- asa_5512_9.12-4-67_single-context.yaml
+
+### Interactive mode
+The `device2yaml.rb` script is a little dumb and needs some help, especially
+when having a device sending its output page by page and requiring you to press
+space for the next page. `device2yaml.rb` does not know how to handle this.
+
+While `device2yaml.rb` is running, you can type anything to the keyboard, it
+will be send to the remote device. So you can press space or 'n' to get the
+next page.
+
+You can also use this to enter an enable password.
+
+If you press the "Esc" key, `device2yaml.rb` will not wait for the idle timeout
+and will process the next command right away.
+
+### YAML Format
+The yaml file has three sections:
+- init_prompt: describing the lines send by the device before we can send a
+command. It usually includes MOTD banners, and must include the first prompt.
+- commands: the commands the oxidized model sends to the network device and the
+expected output.
+- oxidized_output: the expected output of oxidized, so that you can compare it
+to the output generated by the unit test. This is optional and only used for
+unit tests.
+
+The outputs are multiline and use YAML block scalars (`|`), with the trailing \n
+removed (`-` after `|`). The outputs include the echo of the given command and
+the next prompt. Escape characters are coded in Ruby style (\n, \r...).
+
+Here is a shortened example of a YAML file:
+```yaml
+---
+init_prompt: |-
+  \e[4m\rLAB-R1234_Garderos#\e[m\x20
+commands:
+  show system version: |-
+    show system version
+    grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
+    \e[4m\rLAB-R1234_Garderos#\e[m\x20
+# ...
+  exit: ""
+oxidized_output: |
+  # grs-gwuz-armel/003_005_068 (Garderos; 2021-04-30 16:19:35)
+  #\x20
+# ...
+```
+
+

data/examples/device-simulation/cmdsets/aoscx

@@ -0,0 +1,9 @@
+no page
+show version
+show environment
+show module
+show interface transceiver
+show system | exclude "Up Time" | exclude "CPU" | exclude "Memory" | exclude "Pkts .x" | exclude "Lowest" | exclude "Missed"
+show running-config
+show system
+exit

data/examples/device-simulation/cmdsets/arubainstant

@@ -0,0 +1,5 @@
+show version
+show activate status
+show aps
+show running-config no-encrypt
+exit

data/examples/device-simulation/cmdsets/asa

@@ -0,0 +1,7 @@
+enable
+terminal pager 0
+show mode
+show version
+show inventory
+more system:running-config
+exit

data/examples/device-simulation/cmdsets/ios

@@ -0,0 +1,7 @@
+terminal length 0
+terminal width 0
+show version
+show vtp status
+show inventory
+show running-config
+exit

data/examples/device-simulation/cmdsets/nxos

@@ -0,0 +1,5 @@
+terminal length 0
+show version
+show inventory
+show running-config
+exit

data/examples/device-simulation/cmdsets/routeros

@@ -0,0 +1,5 @@
+/system resource print
+/system package update print
+/system history print without-paging
+/export show-sensitive
+quit

data/examples/device-simulation/cmdsets/srosmd

@@ -0,0 +1,11 @@
+environment more false
+show system information
+show card state
+show chassis
+file show bootlog.txt
+admin show configuration debug full-context
+file show config.dbg
+admin show configuration configure | match persistent-indices post-lines 10000
+admin show configuration bof full-context
+admin show configuration configure full-context
+logout

data/examples/device-simulation/device2yaml.rb

@@ -0,0 +1,225 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'net/ssh'
+require 'optparse'
+require 'etc'
+require 'timeout'
+
+# This scripts logs in a network device and outputs a yaml file that can be
+# used for model unit tests in spec/model/
+
+# This script is quick & dirty - it grew with the time an could be a project
+# for its own. It works, and that should be enough ;-)
+
+################# Methods
+# Runs cmd in the ssh session, either im exec mode or with a tty
+# saves the output to @output
+def ssh_exec(cmd)
+  puts "\n### Sending #{cmd}..."
+  @output&.puts "  #{cmd}: |-"
+
+  if @exec_mode
+    @ssh_output = @ssh.exec! cmd + "\n"
+  else
+    @ses.send_data cmd + "\n"
+    shell_wait
+  end
+  yaml_output('    ')
+end
+
+# Wait for the ssh command to be executed, with an idle timout @idle_timeout
+# Pressing CTRL-C exits the script
+# Pressing ESC termiates the idle timeout
+def shell_wait
+  @ssh_output = ''
+  # ssh_output gets appended by chanel.on-data (below)
+  # We store the current length of @ssh_output in @ssh_output_length
+  # if @ssh_output.length is bigger than @ssh_output_length, we got new data
+  @ssh_output_length = 0
+
+  # Keep track of time for idle timeout
+  start_time = Time.now
+
+  # Loop & wait for @idle_timeout seconds after last output
+  # 0.1 means that the loop should run at least once per 0.1 second
+  @ssh.loop(0.1) do
+    # if @ssh_output is longer than our saved length, we got new output
+    if @ssh_output_length < @ssh_output.length
+      # reset the timer and save the new output length
+      start_time = Time.now
+      @ssh_output_length = @ssh_output.length
+    end
+
+    # We wait for 0.1 seconds if a key was pressed
+    begin
+      Timeout.timeout(0.1) do
+        # Get input // this is a blocking call
+        char = $stdin.getch
+        # If ctrl-c is pressed, exit the script
+        if char == "\u0003"
+          puts '### CTRL-C pressed, exiting'
+          cleanup
+          exit
+        end
+        # If escape is pressed, terminate idle timeout
+        if char == "\e"
+          puts "\n### ESC pressed, skipping idle timeout"
+          return false
+        else
+          # if not, send the char through ssh
+          @ses.send_data char
+        end
+      end
+    rescue Timeout::Error
+      # No key pressed
+    end
+
+    # exit the loop when the @idle_timeout has been reached (false = exit)
+    Time.now - start_time < @idle_timeout
+  end
+end
+
+def yaml_output(prepend = '')
+  # Now print the collected output to @output
+  firstline = true
+
+  # as we want to prepend 'prepend' to each line, we need each_line and chomp
+  # chomp removes the trainling \n
+  @ssh_output.each_line(chomp: true) do |line|
+    # encode line and remove the first and the trailing double quote
+    line = line.dump[1..-2]
+    if firstline
+      # Make sure the leading space of the first line (if present)
+      # is coded with \0x20 or YAML block scalars won't work
+      line.sub!(/^\A /, '\x20')
+      firstline = false
+    end
+    # Make sure trailing white spaces are coded with \0x20
+    line.gsub!(/ $/, '\x20')
+    # prepend white spaces for the yaml block scalar
+    line = prepend + line
+    @output&.puts line
+  end
+end
+
+def cleanup
+  (@ssh.close rescue true) unless @ssh.closed?
+  @output&.close
+end
+
+################# Main loop
+
+# Define options
+options = {}
+optparse = OptionParser.new do |opts|
+  opts.banner = "Usage: device2yaml.rb [user@]host [options]"
+
+  opts.on('-c', '--cmdset file', 'Mandatory: specify the commands to be run') do |file|
+    options[:cmdset] = file
+  end
+  opts.on('-o', '--output file', 'Specify an output YAML-file') do |file|
+    options[:output] = file
+  end
+  opts.on('-t', '--timeout value', Integer, 'Specify the idle timeout beween commands (default: 5 seconds)') do |timeout|
+    options[:timeout] = timeout
+  end
+  opts.on('-e', '--exec-mode', 'Run ssh in exec mode (without tty)') { @exec_mode = true }
+  opts.on '-h', '--help', 'Print this help' do
+    puts opts
+    exit
+  end
+end
+
+# Catch and parse the first argument
+if ARGV[0] && ARGV[0][0] != '-'
+  argument = ARGV.shift
+  if argument.include?('@')
+    ssh_user, ssh_host = argument.split('@')
+  else
+    ssh_user = Etc.getlogin
+    ssh_host = argument
+  end
+else
+  puts 'Missing a host to connect to...'
+  puts
+  puts optparse
+  exit 1
+end
+
+# Parse the options
+optparse.parse!
+
+# Get the commands to be run against ssh_host
+unless options[:cmdset]
+  puts 'Missing a command set, use option -c'
+  puts
+  puts optparse
+  exit 1
+end
+# make an array of commands to send, ignore empty lines
+ssh_commands = File.read(options[:cmdset]).split(/\n+|\r+/)
+
+# Defaut idle timeout: 5 seconds, as tests showed that 2 seconds is too short
+@idle_timeout = options[:timeout] || 5
+
+# We will use safe navifation (&.) to call the methods on @output only
+# if @output is not nil
+@output = options[:output] ? File.open(options[:output], 'w') : nil
+
+@ssh = Net::SSH.start(ssh_host,
+                      ssh_user,
+                      { timeout:                         10,
+                        append_all_supported_algorithms: true })
+
+@ssh_output = ''
+
+unless @exec_mode
+  @ses = @ssh.open_channel do |ch|
+    ch.on_data do |_ch, data|
+      @ssh_output += data
+      # Output the data to stdout for interactive control
+      # remove ANSI escape codes, as they can produce problems
+      # The code will be printed as '\e[123m' in the output
+      print data.gsub("\e", '\e')
+    end
+    ch.request_pty(term: 'vt100') do |_ch, success_pty|
+      raise "Can't get PTY" unless success_pty
+
+      ch.send_channel_request 'shell' do |_ch, success_shell|
+        raise "Can't get shell" unless success_shell
+      end
+    end
+    ch.on_extended_data do |_ch, _type, data|
+      $stderr.print "Error: #{data}\n"
+    end
+  end
+end
+
+# YAML begin of file
+@output&.puts '---'
+
+if @exec_mode
+  # init prompt does not exist and is empty in exec mode
+  @output&.puts 'init_prompt:'
+else
+  # get motd and first prompt
+  @output&.puts 'init_prompt: |-'
+  shell_wait
+  yaml_output '  '
+end
+
+@output&.puts "commands:"
+
+begin
+  ssh_commands.each do |cmd|
+    ssh_exec cmd
+  end
+rescue Errno::ECONNRESET, Net::SSH::Disconnect, IOError => e
+  puts "### Connection closed with message: #{e.message}"
+end
+
+@output&.puts 'oxidized_output: |'
+@output&.puts '  !! needs to be written by hand or copy & paste from model output'
+
+cleanup