oxidized 0.29.1 → 0.30.1

data/README.md

@@ -9,7 +9,7 @@
 
 Oxidized is a network device configuration backup tool. It's a RANCID replacement!
 
-Light and extensible, Oxidized supports over 130 operating system types.
+It is light and extensible and supports over 130 operating system types.
 
 Feature highlights:
 
@@ -17,13 +17,13 @@ Feature highlights:
 * Restful API to a move node immediately to head-of-queue (GET/POST /node/next/[NODE])
 * Syslog udp+file example to catch config change events (IOS/JunOS) and trigger a config fetch
   * Will signal which IOS/JunOS user made the change, can then be used by output modules (via POST)
-  * The `git` output module uses this info - 'git blame' will show who changed each line, and when
+  * The `git` output module uses this info - 'git blame' will show who changed each line
 * Restful API to reload list of nodes (GET /reload)
 * Restful API to fetch configurations (/node/fetch/[NODE] or /node/fetch/group/[NODE])
 * Restful API to show list of nodes (GET /nodes)
 * Restful API to show list of version for a node (/node/version[NODE]) and diffs
 
-Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h) video on YouTube!
+Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU?t=3h) video on YouTube!
 
 > :warning: [Maintainer Wanted!](#help-needed) :warning:
 >
@@ -38,6 +38,7 @@ Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h
     * [FreeBSD](#freebsd)
     * [Build from Git](#build-from-git)
     * [Docker](#running-with-docker)
+    * [Podman-Compose](#running-with-podman-compose)
     * [Installing Ruby 2.3 using RVM](#installing-ruby-23-using-rvm)
 3. [Initial Configuration](#configuration)
 4. [Configuration](docs/Configuration.md)
@@ -67,7 +68,8 @@ Check out the [Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h
       * [Hook: ciscosparkdiff](docs/Hooks.md#hook-type-ciscosparkdiff)
 5. [Creating and Extending Models](docs/Creating-Models.md)
 6. [Help](#help)
-7. [Ruby API](docs/Ruby-API.md#ruby-api)
+7. [Help Needed](#help-needed)
+8. [Ruby API](docs/Ruby-API.md#ruby-api)
     * [Input](docs/Ruby-API.md#input)
     * [Output](docs/Ruby-API.md#output)
     * [Source](docs/Ruby-API.md#source)
@@ -87,7 +89,7 @@ add-apt-repository universe
 Install the dependencies:
 
 ```shell
-apt-get install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++
+apt-get install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++ libyaml-dev
 ```
 
 Finally, install the gems:
@@ -171,7 +173,7 @@ rake install
 
 ### Running with Docker
 
-Currently, Docker Hub automatically builds the master branch as [oxidized/oxidized](https://hub.docker.com/r/oxidized/oxidized/), you can make use of this container or build your own.
+Currently, Docker Hub automatically builds the master branch for linux/amd64 and linux/arm64 platforms as [oxidized/oxidized](https://hub.docker.com/r/oxidized/oxidized/), you can make use of this container or build your own.
 
 To build your own, clone git repo:
 
@@ -255,6 +257,10 @@ If you need to use an internal CA (e.g. to connect to an private github instance
 docker run -v /etc/oxidized:/home/oxidized/.config/oxidized -v /path/to/MY-CA.crt:/usr/local/share/ca-certificates/MY-CA.crt -p 8888:8888/tcp -e UPDATE_CA_CERTIFICATES=true -t oxidized/oxidized:latest
 ```
 
+### Running with podman-compose
+Under [examples/podman-compose](examples/podman-compose), you will find a complete
+example of how to integrate the container into a docker-compose.yml file.
+
 ### Installing Ruby 2.3 using RVM
 
 Install Ruby 2.3 build dependencies
@@ -391,38 +397,14 @@ If you need help with Oxidized then we have a few methods you can use to get in
 
 ## Help Needed
 
-As things stand right now, `oxidized` is maintained by a single person. A great
-many [contributors](https://github.com/ytti/oxidized/graphs/contributors) have
-helped further the software, however contributions are not the same as ongoing
-owner- and maintainer-ship. It appears that many companies use the software to
-manage their network infrastructure, this is great news! But without additional
-help to maintain the software and put out releases, the future of oxidized
-might be less bright. The current pace of development and the much needed
-refactoring simply are not sustainable if they are to be driven by a single
-person.
-
-## Maintainers
-
-If you would like to be a maintainer for Oxidized then please read through the below and see if it's something you would like to help with. It's not a requirement that you can tick all the boxes below but it helps :)
-
-* Triage on issues, review pull requests and help answer any questions from users.
-* Above average knowledge of the Ruby programming language.
-* Professional experience with both oxidized and some other config backup tool (like rancid).
-* Ability to keep a cool head, and enjoy interaction with end users! :)
-* A desire and passion to help drive `oxidized` towards its `1.x.x` stage of life
-  * Help refactor the code
-  * Rework the core infrastructure
-* Permission from your employer to contribute to open source projects
-
-## YES, I WANT TO HELP
-
-Awesome! Simply send an e-mail to Saku Ytti at <saku@ytti.fi>.
+As things stand right now, `oxidized` is maintained by very few people.
+We would appreciate more individuals and companies getting involved in Oxidized.
 
-## Further reading
+Beyond software development, documentation or maintenance of Oxidized, you could
+become a model maintainer, which can be done with little burden and would be a
+big help to the community.
 
-Brian Anderson (from Rust fame) wrote an [excellent
-post](http://brson.github.io/2017/04/05/minimally-nice-maintainer) on what it
-means to be a maintainer.
+Interested? Have a look at [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## License and Copyright
 

data/bin/oxidized

@@ -1,13 +1,10 @@
 #!/usr/bin/env ruby
 
-# FIX ME, killing oxidized needs -9
-trap("INT") { exit } # sinatra will otherwise steal this from us
-
 begin
   require_relative '../lib/oxidized/cli'
   Oxidized::CLI.new.run
-rescue StandardError => error
-  warn error.to_s
+rescue StandardError => e
+  warn e
   debug = Oxidized.config.debug rescue true
   raise if debug
 

data/docs/Hooks.md

@@ -46,7 +46,7 @@ Exec hook recognizes the following configuration keys:
 * `async`: Execute the command in an asynchronous fashion. The main thread by default will wait for the hook command execution to complete. Set this to `true` for long running commands so node configuration pulls are not blocked. Default: `false`
 * `cmd`: command to run.
 
-### exec hook configuration example
+### Exec Hook configuration example
 
 ```yaml
 hooks:
@@ -62,6 +62,42 @@ hooks:
     timeout: 120
 ```
 
+### Exec Hook configuration example to send mail
+
+To send mail you need the package `msmtp` (It is pre-installed with the docker container) 
+
+You then need to update the `~/.msmtprc` file to contain your SMTP credentials like this:
+
+*Note: In the docker container the file is in /home/oxidized/.config/oxidized/.msmtprc so you can create the file if it doesn't exist in your oxidized config folder.*
+
+```cfg
+# Default settings
+defaults
+auth    on
+tls    on
+# Outlook SMTP
+account    mainaccount
+host       smtp.office365.com
+port       587
+from       user@domain.com
+user       user@domain.com
+password   edit-password
+
+account default : mainaccount
+```
+
+For non docker users this file should have the 600 permission, using: `chmod 600 .msmtprc` and the owner of the file should be the owner of oxidized `chown oxidized:oxidized .msmtprc`
+
+Then, you can configure Hooks to send mail like this:
+
+```yaml
+hooks:
+  send_mail_hook:
+    type: exec
+    events: [node_fail]
+    cmd: '/usr/bin/echo -e "Subject: [Oxidized] Error on node $OX_NODE_NAME \n\nThe device $OX_NODE_NAME has not been backed-up, reason: \n\n$OX_EVENT: $OX_ERR_REASON" | msmtp destination@domain.com'
+```
+
 ## Hook type: githubrepo
 
 Note: You must not use the same name as any local repo configured under output. Make sure your 'git' output has a unique name that does not match your remote_repo.

data/docs/Model-Notes/APC_AOS.md

@@ -0,0 +1,52 @@
+# APC AOS Configuration
+
+Currently, the configuration of APC Network Management Cards can be downloaded with FTP only.
+
+A download of the configuration with SCP is [work in progress](https://github.com/ytti/oxidized/issues/1802).
+As the APC has an unusual behavior (the connection is closed without an exit-status), this has to be
+[fixed](https://github.com/net-ssh/net-scp/pull/71) upstream in [Net::SCP](https://github.com/net-ssh/net-scp).
+As soon as there is a release of Net::SCP supporting the behavior of APC OS, we will activate SCP in oxidized.
+
+## Can I collect more information than just the configuration?
+APC OS does not have the ability to show the config.ini within an SSH-session. As oxidized can only get the
+configuration with one input type at a time, it is not possible to fetch config.ini via FTP/SCP and get the output of
+some commands via SSH at the same time.
+
+A ticket has been opened with APC support in order to support "cat config.ini" within an SSH-session, but
+the chances it will be supported at some time are not very good, and older versions will still not support it.
+
+## How do I activate FTP input?
+In order to download the configuration with FTP (and in the future with SCP), you have to activate it as an
+input in the oxidized configuration. If you do not activate the input, oxidized will fail for the node with
+a rather unspecific error (`WARN -- : /apc status fail, retry attempt 1`).
+
+The configuration can be done either globally or only for the model apc_aos.
+
+The global configuration would look like this. Note that Oxidized will try every input type in the given order
+until it succeeds, or it will report a failure.
+```yaml
+input:
+  default: ssh, ftp, scp
+```
+
+Configuration for activating the FTP input for apc_aos only:
+```yaml
+input:
+  default: ssh
+models:
+  apc_aos:
+    input: ftp
+```
+
+You can also set specific username and password for apc_aos only:
+```yaml
+username: default-user
+password: default-password
+input:
+  default: ssh
+models:
+  apc_aos:
+    username: apc-user
+    password: apc-password
+    input: ftp
+```

data/docs/Model-Notes/FSOS.md

@@ -0,0 +1,11 @@
+# Fiberstore (fs.com) FSOS notes
+
+This has been tested against the following models and OS versions
+
+|Model               |OS Version and Build          |
+|--------------------|------------------------------|
+|S3400-48T4SP        |Version 2.0.2J Build 81736    |
+|S3400-48T4SP        |Version 2.0.2J Build 95262    |
+|S3400-48T6SP        |Version 2.2.0F Build 109661   |
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/FortiOS.md

@@ -0,0 +1,28 @@
+# FortiOS Configuration
+
+Create user oxidized with ED25519 public key
+
+```text
+config system admin
+edit oxidized
+set trusthost1 192.0.2.1 255.255.255.255
+set accprofile "super_admin_readonly"
+set ssh-public-key1 "ssh-ed25519 AAAAThisIsJustAnExmapleKey_UseYourOxidizedPUBLICKEY oxidized@librenms"
+end
+```
+
+
+Fortigate procdues a lot of config changes. I recommend filtering using
+
+```yaml
+models:
+  fortios:
+    vars:
+      remove_secret: true
+```
+
+
+
+Oxidized can now retrieve your configuration!
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/README.md

@@ -2,25 +2,6 @@
 
 This directory contains implementation notes and caveats to assist you in your oxidized deployment.
 
-Use the table below for more information on the Vendor/Model caveats.
-
-Vendor          | Model           |Updated
-----------------|-----------------|----------------
-3COM|[Comware](Comware.md)|15 Feb 2018
-AireOS|[AireOS](AireOS.md)|29 Nov 2017
-Arbor Networks|[ArbOS](ArbOS.md)|27 Feb 2018
-Arista|[EOS](EOS.md)|05 Feb 2018
-Cumulus|[Cumulus](Cumulus.md)|11 Jun 2018
-Huawei|[VRP](VRP-Huawei.md)|17 Nov 2017
-Huawei|[SmartAX series](SmartAX-Huawei.md)|21 Jan 2019
-Cisco IOS|[IOS](IOS.md)|29 Mar 2019
-Juniper|[MX/QFX/EX/SRX/J Series](JunOS.md)|18 Jan 2018
-Netgear|[Netgear](Netgear.md)|11 Apr 2018
-Nokia|[Nokia ISAM](Nokia.md)|22 Aug 2018
-Dell/EMC|[Dell EMC Networking OS10](OS10.md)|07 Dec 2021
-Viptela|[Viptela](Viptela.md)|1 Jul 2018
-Zyxel|[XGS4600 Series](XGS4600-Zyxel.md)|1 Feb 2018
-Linux|[LinuxGeneric](LinuxGeneric.md)|10 Jun 2019
-Lenovo|[Lenovo Network OS](LenovoNOS.md)|5 Apr 2022
+The model notes are linked from the [list of supported OS types](/docs/Supported-OS-Types.md).
 
 If you discover additional caveats or problems please make sure to consult the [GitHub issues for oxidized](https://github.com/ytti/oxidized/issues) known issues.

data/docs/Model-Notes/XGS4600-Zyxel.md

@@ -5,6 +5,11 @@
 FTP access is only possible as admin, other users can login but cannot pull the files.
 For the XGS4600 series the config file is _config_ and not _config-0_
 
+To enable FTP backup, uncomment the following line in _oxidized/lib/oxidized/model/zynos.rb_
+```text
+  # cmd 'config-0'
+```
+
 The following line in _oxidized/lib/oxidized/model/zynos.rb_ will need changing
 
 ```text

data/docs/Release.md

@@ -0,0 +1,62 @@
+# How to release a new version of Oxidized?
+This document is targeted at oxidized maintainers. It describes the release process.
+
+## Review changes
+Run `git diff 0.30.0..master` (where `0.30.0` is to be changed to the last release) and review
+all the changes that have been done. Have a specific look at changes you don't understand.
+
+## Test, test test!
+Test the git code and the container against as much device types an environments as you can.
+
+Do not integrate late PRs into master if they do not fix issues for the release. The must wait for the next release.
+
+## Version numbering
+Oxidized versions are nummered like 0.major.minor
+- major is incremented when releasing new features. minor is then set to 0
+- minor is incremented when releasing fixes only, just after a major version.
+
+## Release
+1. Checkout the master branch of oxidized. Make sure you are up to date with origin.
+2. Change the version in lib/oxidized/version.rb
+3. Change CHANGELOG.md to replace [Unreleased] with [0.xx.yy – 202Y-MM-DD]
+4. Run `git diff` to check your changes
+5. Commit the changes to the local git repository with a commit message “chore(release): release version 0.xx.yy”
+6. Tag the commit with `git tag -a 0.xx.yy -m "Release 0.xx.yy"`
+7. Push the change and the tag to github:
+```
+git push
+git push origin 0.xx.yy
+```
+
+## Release in github
+Make a release from the tag in github
+- Thank the contributors
+- Only describe major changes, and refer to CHANGELOG.md
+- List new contributors (generated automatically)
+
+## Release in rubygems
+1. Build the gem with ‘rake build’
+2. Install an test the gem locally
+```
+gem install --user-install pkg/oxidized-0.30.0.gem
+~/.local/share/gem/ruby/3.1.0/bin/oxidized
+```
+3. Push the gem with ‘rake push’
+
+You need an account at rubygems which is allowed to push oxidized
+
+## Release in docker.io
+The OCI-Containter is automatically build and pushed to docker.io by github
+
+## Update CHANGELOG.md for next release
+Add
+```
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Fixed
+
+```

data/docs/Ruby-API.md

@@ -6,7 +6,7 @@ The following objects exist in Oxidized.
 
 * gets config from nodes
 * must implement 'connect', 'get', 'cmd'
-* 'ssh', 'telnet', 'ftp', 'tftp', 'http' implemented
+* 'ssh', 'telnet', 'ftp', 'tftp', 'scp', 'http' implemented
 
 ### http
 

data/docs/Sources.md

@@ -187,3 +187,16 @@ source:
     scheme: https
     secure: false
 ```
+
+HTTP source also supports pagination. Two settings must be enabled. (`pagination` as a bool and `pagination_key_name` as a string)
+The `pagination_key_name` setting is the key name that an api returns to find the url of the next page.
+
+**Disclaimer**: currently only tested with netbox as the source
+
+```yaml
+source:
+  default: http
+  http:
+    pagination: true
+    pagination_key_name: 'next'
+```