oxidized 0.29.1 → 0.37.0

data/README.md

@@ -1,29 +1,26 @@
 # Oxidized
 
 [![Build Status](https://github.com/ytti/oxidized/actions/workflows/ruby.yml/badge.svg)](https://github.com/ytti/oxidized/actions/workflows/ruby.yml)
-[![codecov.io](https://codecov.io/gh/ytti/oxidized/coverage.svg?branch=master)](https://codecov.io/gh/ytti/oxidized?branch=master)
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/5a90cb22db6a4d5ea23ad0dfb53fe03a)](https://www.codacy.com/app/ytti/oxidized?utm_source=github.com&utm_medium=referral&utm_content=ytti/oxidized&utm_campaign=Badge_Grade)
-[![Code Climate](https://codeclimate.com/github/ytti/oxidized/badges/gpa.svg)](https://codeclimate.com/github/ytti/oxidized)
 [![Gem Version](https://badge.fury.io/rb/oxidized.svg)](http://badge.fury.io/rb/oxidized)
 [![Join the chat at https://gitter.im/oxidized/Lobby](https://badges.gitter.im/oxidized/Lobby.svg)](https://gitter.im/oxidized/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 Oxidized is a network device configuration backup tool. It's a RANCID replacement!
 
-Light and extensible, Oxidized supports over 130 operating system types.
+It is light and extensible and supports over 130 operating system types.
 
 Feature highlights:
 
 * Automatically adds/removes threads to meet configured retrieval interval
-* Restful API to a move node immediately to head-of-queue (GET/POST /node/next/[NODE])
+* Restful API to a move node immediately to head-of-queue (GET/PUT /node/next/[NODE])
 * Syslog udp+file example to catch config change events (IOS/JunOS) and trigger a config fetch
   * Will signal which IOS/JunOS user made the change, can then be used by output modules (via POST)
-  * The `git` output module uses this info - 'git blame' will show who changed each line, and when
+  * The `git` output module uses this info - 'git blame' will show who changed each line
 * Restful API to reload list of nodes (GET /reload)
 * Restful API to fetch configurations (/node/fetch/[NODE] or /node/fetch/group/[NODE])
 * Restful API to show list of nodes (GET /nodes)
 * Restful API to show list of version for a node (/node/version[NODE]) and diffs
 
-Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h) video on YouTube!
+Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU?t=3h) video on YouTube!
 
 > :warning: [Maintainer Wanted!](#help-needed) :warning:
 >
@@ -34,11 +31,10 @@ Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h
 1. [Supported OS Types](docs/Supported-OS-Types.md)
 2. [Installation](#installation)
     * [Debian and Ubuntu](#debian-and-ubuntu)
-    * [CentOS, Oracle Linux, Red Hat Linux](#centos-oracle-linux-red-hat-linux)
+    * [Rocky Linux, Red Hat Enterprise Linux](#rocky-linux-red-hat-enterprise-linux)
     * [FreeBSD](#freebsd)
     * [Build from Git](#build-from-git)
-    * [Docker](#running-with-docker)
-    * [Installing Ruby 2.3 using RVM](#installing-ruby-23-using-rvm)
+    * [Docker & Podman](docs/Docker.md)
 3. [Initial Configuration](#configuration)
 4. [Configuration](docs/Configuration.md)
     * [Debugging](docs/Configuration.md#debugging)
@@ -67,7 +63,8 @@ Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h
       * [Hook: ciscosparkdiff](docs/Hooks.md#hook-type-ciscosparkdiff)
 5. [Creating and Extending Models](docs/Creating-Models.md)
 6. [Help](#help)
-7. [Ruby API](docs/Ruby-API.md#ruby-api)
+7. [Help Needed](#help-needed)
+8. [Ruby API](docs/Ruby-API.md#ruby-api)
     * [Input](docs/Ruby-API.md#input)
     * [Output](docs/Ruby-API.md#output)
     * [Source](docs/Ruby-API.md#source)
@@ -76,8 +73,7 @@ Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h
 ## Installation
 
 ### Debian and Ubuntu
-
-Debian "buster" or newer and Ubuntu 17.10 (artful) or newer are recommended. On Ubuntu, begin by enabling the `universe`
+Debian 12 "bookworm" or newer and Ubuntu 22.04 (Jammy Jellyfish) or newer are recommended. On Ubuntu, begin by enabling the `universe`
 repository (required for libssh2-1-dev):
 
 ```shell
@@ -87,197 +83,96 @@ add-apt-repository universe
 Install the dependencies:
 
 ```shell
-apt-get install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++
+apt install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++ libyaml-dev libzstd-dev
 ```
 
-Finally, install the gems:
+Finally, install Oxidized:
 
 ```shell
 gem install oxidized
-gem install oxidized-script oxidized-web # If you don't install oxidized-web, ensure "rest" is removed from your Oxidized config.
 ```
 
-### CentOS, Oracle Linux, Red Hat Linux
-
-On CentOS 6 and 7 / RHEL 6 and 7, begin by installing Ruby 2.3 or greater. This can be accomplished in one of several ways:
-
-Install Ruby 2.3 from [SCL](https://www.softwarecollections.org/en/scls/rhscl/rh-ruby23/):
-
+You can also install one or both of the optional gems. They are not required
+to run Oxidized:
 ```shell
-yum install centos-release-scl
-yum install rh-ruby30 rh-ruby30-ruby-devel
-scl enable rh-ruby30 bash
+gem install oxidized-web    # Web interface and rest API
+gem install oxidized-script # Script-based input/output extensions
 ```
 
-The following additional packages will be required to build the dependencies:
+### Rocky Linux, Red Hat Enterprise Linux
+These instructions has been verified on Rocky Linux 9, Rocky Linux 10 and Fedora.
 
+On Rocky Linux 9, you need to install/enable EPEL, CRB and Ruby 3.1:
 ```shell
-yum install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++
+dnf install epel-release
+dnf config-manager --set-enabled crb
+dnf module enable ruby:3.1
 ```
 
-Alternatively, install Ruby 2.6 via RVM by following the instructions:
-
-Make sure you dont have any leftover ruby:
-```yum erase ruby```
-
-Then, install gpg key and rvm
-
+Then you need the required packages for oxidized:
 ```shell
-sudo gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
-curl -sSL https://get.rvm.io | bash -s stable
-source /etc/profile.d/rvm.sh
-rvm requirements run
-rvm install 3.0
-rvm use 3.0
+dnf -y install ruby ruby-devel sqlite-devel openssl-devel pkgconf-pkg-config  cmake libssh-devel libicu-devel zlib-devel gcc-c++ libyaml-devel which
 ```
 
-Install oxidized requirements:
-```yum install make cmake which sqlite-devel openssl-devel libssh2-devel gcc libicu-devel gcc-c++```
-
-Install the gems:
-```gem install oxidized oxidized-web```
-
-You need to wrap the gem and reference the wrap in the systemctl service file:
-```rvm wrapper oxidized```
-
-You can see where the wrapped gem is via
-```rvm wrapper show oxidized```
-Use that path in the oxidized.service file, restart the systemctl daemon, run oxidized by hand once, edit config file, start service.
-
-### FreeBSD
-
-[Use RVM to install Ruby v2.3](#installing-ruby-23-using-rvm), then install all required packages and gems:
+Finally, install Oxidized:
 
 ```shell
-pkg install cmake pkgconf
 gem install oxidized
-gem install oxidized-script oxidized-web
-```
-
-Oxidized is also available via [FreeBSD ports](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203374):
-
-```shell
-pkg install rubygem-oxidized rubygem-oxidized-script rubygem-oxidized-web
-```
-
-### Build from Git
-
-```shell
-git clone https://github.com/ytti/oxidized.git
-cd oxidized/
-gem install bundler
-rake install
-```
-
-### Running with Docker
-
-Currently, Docker Hub automatically builds the master branch as [oxidized/oxidized](https://hub.docker.com/r/oxidized/oxidized/), you can make use of this container or build your own.
-
-To build your own, clone git repo:
-
-```shell
-git clone https://github.com/ytti/oxidized
-```
-
-Then, build the container locally (requires docker 17.05.0-ce or higher):
-
-```shell
-docker build -q -t oxidized/oxidized:latest oxidized/
 ```
 
-Once you've built the container (or chosen to make use of the automatically built container in Docker Hub, which will be downloaded for you by docker on the first `run` command had you not built it), proceed as follows:
-
-Create a configuration directory in the host system:
-
+You can also install one or both of the optional gems. They are not required
+to run Oxidized:
 ```shell
-mkdir /etc/oxidized
+gem install oxidized-web    # Web interface and rest API
+gem install oxidized-script # Script-based input/output extensions
 ```
 
-Run the container for the first time to initialize the config:
-
-_Note: this step in only required for creating the Oxidized configuration file and can be skipped if you already have one._
-
-```shell
-docker run --rm -v /etc/oxidized:/home/oxidized/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized
-```
-
-If the RESTful API and Web Interface are enabled, on the docker host running the container
-edit `/etc/oxidized/config` and modify `rest: 127.0.0.1:8888` to `rest: 0.0.0.0:8888`. This will bind port 8888 to all interfaces, and expose the port so that it could be accessed externally. [(Issue #445)](https://github.com/ytti/oxidized/issues/445)
-
-Alternatively, you can use docker-compose to launch the oxidized container:
-
-```yaml
-# docker-compose.yml
-# docker-compose file example for oxidized that will start along with docker daemon
----
-version: "3"
-services:
-  oxidized:
-    restart: always
-    image: oxidized/oxidized:latest
-    ports:
-      - 8888:8888/tcp
-    environment:
-      CONFIG_RELOAD_INTERVAL: 600
-    volumes:
-       - config:/home/oxidized/.config/oxidized/
-volumes:
-  config:
-```
-
-Create the `/etc/oxidized/router.db` (see [CSV Source](docs/Sources.md#source-csv) for further info):
+### FreeBSD
+> :warning: These installation instructions have been tested on FreeBSD 14.2, but
+oxidized itself has not been tested on it.
 
+First install ruby and rubyXX-gems (Find out the name of the package with `pkg search gems`):
 ```shell
-vim /etc/oxidized/router.db
+pkg instal ruby
+pkg instal ruby32-gems
 ```
 
-Run container again to start oxidized with your configuration:
-
+Then install the dependencies of oxidized an oxidized-web:
 ```shell
-docker run -v /etc/oxidized:/home/oxidized/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest
-oxidized[1]: Oxidized starting, running as pid 1
-oxidized[1]: Loaded 1 nodes
-Puma 2.13.4 starting...
-* Min threads: 0, max threads: 16
-* Environment: development
-* Listening on tcp://0.0.0.0:8888
+pkg install ruby ruby-gems git sqlite3 libssh2 cmake pkgconf gmake
+pkg install libyaml icu   # Dependencies for oxidized-web
 ```
 
-If you want to have the config automatically reloaded (e.g. when using a http source that changes):
+Finally, install Oxidized:
 
 ```shell
-docker run -v /etc/oxidized:/home/oxidized/.config/oxidized -p 8888:8888/tcp -e CONFIG_RELOAD_INTERVAL=3600 -t oxidized/oxidized:latest
+gem install oxidized
 ```
 
-If you need to use an internal CA (e.g. to connect to an private github instance):
-
+You can also install one or both of the optional gems. They are not required
+to run Oxidized:
 ```shell
-docker run -v /etc/oxidized:/home/oxidized/.config/oxidized -v /path/to/MY-CA.crt:/usr/local/share/ca-certificates/MY-CA.crt -p 8888:8888/tcp -e UPDATE_CA_CERTIFICATES=true -t oxidized/oxidized:latest
+gem install oxidized-web    # Web interface and rest API
+gem install oxidized-script # Script-based input/output extensions
 ```
 
-### Installing Ruby 2.3 using RVM
-
-Install Ruby 2.3 build dependencies
+Oxidized is also available via [FreeBSD ports](https://ports.freebsd.org/cgi/ports.cgi?query=oxidized):
 
 ```shell
-yum install curl gcc-c++ patch readline readline-devel zlib zlib-devel
-yum install libyaml-devel libffi-devel openssl-devel make cmake
-yum install bzip2 autoconf automake libtool bison iconv-devel libssh2-devel libicu-devel
+pkg install rubygem-oxidized rubygem-oxidized-script rubygem-oxidized-web
 ```
 
-Install RVM
+### Build from Git
 
 ```shell
-curl -L get.rvm.io | bash -s stable
+git clone https://github.com/ytti/oxidized.git
+cd oxidized/
+gem install bundler
+rake install
 ```
 
-Setup RVM environment and compile and install Ruby 2.3 and set it as default
-
-```shell
-source /etc/profile.d/rvm.sh
-rvm install 2.3
-rvm use --default 2.3
-```
+### Running with Docker or Podman
+See [docs/Docker.md](docs/Docker.md)
 
 ## Configuration
 
@@ -289,7 +184,7 @@ It is recommended practice to run Oxidized using its own username.  This usernam
 useradd -s /bin/bash -m oxidized
 ```
 
-> It is recommended __not__ to run Oxidized as root.
+> It is recommended __not__ to run Oxidized as root. After creating a dedicated user, switch to the oxidized user using su oxidized to ensure that Oxidized is run under the correct user context.
 
 To initialize a default configuration in your home directory `~/.config/oxidized/config`, simply run `oxidized` once. If you don't further configure anything from the output and source sections, it'll extend the examples on a subsequent `oxidized` execution. This is useful to see what options for a specific source or output backend are available.
 
@@ -391,38 +286,14 @@ If you need help with Oxidized then we have a few methods you can use to get in
 
 ## Help Needed
 
-As things stand right now, `oxidized` is maintained by a single person. A great
-many [contributors](https://github.com/ytti/oxidized/graphs/contributors) have
-helped further the software, however contributions are not the same as ongoing
-owner- and maintainer-ship. It appears that many companies use the software to
-manage their network infrastructure, this is great news! But without additional
-help to maintain the software and put out releases, the future of oxidized
-might be less bright. The current pace of development and the much needed
-refactoring simply are not sustainable if they are to be driven by a single
-person.
-
-## Maintainers
-
-If you would like to be a maintainer for Oxidized then please read through the below and see if it's something you would like to help with. It's not a requirement that you can tick all the boxes below but it helps :)
-
-* Triage on issues, review pull requests and help answer any questions from users.
-* Above average knowledge of the Ruby programming language.
-* Professional experience with both oxidized and some other config backup tool (like rancid).
-* Ability to keep a cool head, and enjoy interaction with end users! :)
-* A desire and passion to help drive `oxidized` towards its `1.x.x` stage of life
-  * Help refactor the code
-  * Rework the core infrastructure
-* Permission from your employer to contribute to open source projects
-
-## YES, I WANT TO HELP
-
-Awesome! Simply send an e-mail to Saku Ytti at <saku@ytti.fi>.
+As things stand right now, `oxidized` is maintained by very few people.
+We would appreciate more individuals and companies getting involved in Oxidized.
 
-## Further reading
+Beyond software development, documentation or maintenance of Oxidized, you could
+become a model maintainer, which can be done with little burden and would be a
+big help to the community.
 
-Brian Anderson (from Rust fame) wrote an [excellent
-post](http://brson.github.io/2017/04/05/minimally-nice-maintainer) on what it
-means to be a maintainer.
+Interested? Have a look at [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## License and Copyright
 

data/Rakefile

@@ -1,8 +1,9 @@
 require 'bundler/gem_tasks'
 require 'rake/testtask'
+require 'time'
 require_relative 'lib/oxidized/version'
 
-gemspec = eval(File.read(Dir['*.gemspec'].first))
+gemspec = Gem::Specification.load(Dir['*.gemspec'].first)
 gemfile = [gemspec.name, gemspec.version].join('-') + '.gem'
 
 # Integrate Rubocop if available
@@ -10,7 +11,6 @@ begin
   require 'rubocop/rake_task'
 
   RuboCop::RakeTask.new
-  task(:default).prerequisites << task(:rubocop)
 rescue LoadError
   task :rubocop do
     puts 'Install rubocop to run its rake tasks'
@@ -27,8 +27,10 @@ task :test do
   Rake::TestTask.new do |t|
     t.libs << 'spec'
     t.test_files = FileList['spec/**/*_spec.rb']
-    t.warning = true
-    t.verbose = true
+    t.ruby_opts = ['-W:deprecated']
+    # Don't display ambiguity warning between regexp and division in models
+    t.warning = false
+    t.verbose = false
   end
 end
 
@@ -40,11 +42,6 @@ task :version_set do
   Bundler::GemHelper.instance.gemspec.version = Oxidized::VERSION
 end
 
-# desc 'Install gem'
-# task install: :build do
-#    system "sudo -Es sh -c \'umask 022; gem install gems/#{gemfile}\'"
-# end
-
 desc 'Remove gems'
 task :clean do
   FileUtils.rm_rf 'pkg'
@@ -52,7 +49,7 @@ end
 
 desc 'Tag the release'
 task :tag do
-  system "git tag #{gemspec.version}"
+  system "git tag #{gemspec.version} -m 'Release #{gemspec.version}'"
 end
 
 desc 'Push to rubygems'
@@ -73,13 +70,62 @@ task :chmod do
     extra/oxidized.runit
     extra/syslog.rb
     extra/update-ca-certificates.runit
+    extra/device2yaml.rb
   ]
   dirs = []
-  %x(git ls-files -z).split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }.each do |file|
+  %x(git ls-files -z).split("\x0").reject { |f| f.match(/^(test|spec|features)\//) }.each do |file|
     dirs.push(File.dirname(file))
     xbit.include?(file) ? File.chmod(0o0755, file) : File.chmod(0o0644, file)
   end
   dirs.sort.uniq.each { |dir| File.chmod(0o0755, dir) }
 end
 
-task default: :test
+# Build the container image with docker or podman
+def command_available?(command)
+  system("which #{command} > /dev/null 2>&1")
+end
+
+def docker_needs_root?
+  !system('docker info > /dev/null 2>&1')
+end
+
+desc 'Build the container image with docker or podman'
+task :build_container do
+  branch_name = %x(git rev-parse --abbrev-ref HEAD).chop.gsub '/', '_'
+  sha_hash = %x(git rev-parse --short HEAD).chop
+  sha_hash_long = %x(git rev-parse HEAD).chop
+  image_tag = "#{branch_name}-#{sha_hash}"
+  created_time = Time.now.iso8601
+
+  # Build-Args for consistent labels
+  build_args = [
+    "--label org.opencontainers.image.title=oxidized",
+    "--label org.opencontainers.image.description='Local build of Oxidized'",
+    "--label org.opencontainers.image.url=https://github.com/ytti/oxidized",
+    "--label org.opencontainers.image.source=https://github.com/ytti/oxidized",
+    "--label org.opencontainers.image.created=#{created_time}",
+    "--label org.opencontainers.image.ref.name=#{image_tag}",
+    "--label org.opencontainers.image.licenses=Apache-2.0",
+    "--label org.opencontainers.image.version=#{image_tag}",
+    "--label org.opencontainers.image.revision=#{sha_hash_long}",
+    "-t oxidized:#{image_tag}",
+    "-t oxidized:latest"
+  ].join(' ')
+
+  # Prefer podman if available as it runs rootless
+  if command_available?('podman')
+    sh "podman build #{build_args} ."
+  elsif command_available?('docker')
+    if docker_needs_root?
+      puts 'docker needs root to build the image. Using sudo...'
+      sh "sudo docker build #{build_args} ."
+    else
+      sh "docker build #{build_args} ."
+    end
+  else
+    puts 'You need Podman or Docker to build the container image.'
+    exit 1
+  end
+end
+
+task default: %i[rubocop test]

data/bin/oxidized

@@ -1,13 +1,10 @@
 #!/usr/bin/env ruby
 
-# FIX ME, killing oxidized needs -9
-trap("INT") { exit } # sinatra will otherwise steal this from us
-
 begin
   require_relative '../lib/oxidized/cli'
   Oxidized::CLI.new.run
-rescue StandardError => error
-  warn error.to_s
+rescue StandardError => e
+  warn e
   debug = Oxidized.config.debug rescue true
   raise if debug