oxidized 0.19.0 → 0.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bd75468c104dc960d991ac69be65520c3af0ff6f
-  data.tar.gz: 5189f65c44b78a3f76916417126a11526f53dca3
+  metadata.gz: 5aa4c172ab2a33d6e75ee497b106687064f5f9d9
+  data.tar.gz: b0f9cf4fecb5837934a6ecaf622f12d6fc1646e8
 SHA512:
-  metadata.gz: 1d3c1ff69768064f89d7a5bfeb4d3ea1ff99cb7df051394e428f2b8168b3a7b60dc2e97cc0886c79d664f88c542ca3d3144db66d0c1a363fc187ab264012df30
-  data.tar.gz: e935f661d461388807ab7bcc83d1b90b1e13b9c4d0aef509efcdf3684357aa4c55d959c2a2177bd7abfa408e9ad9f599702122f0118a012c415c6e8391bd67ea
+  metadata.gz: 69d6052e266f2726179040a5e3598f107e8a74874c5b80aec3b1755d3da8ab826b4fd6d51d7be51cf833910bf98dc78e6c21a3658188f502a81a5428fcfc7b34
+  data.tar.gz: b828b07fc03b4684680268e8844cacd9b777b78d7889f4cfc05a9b86b32b20f210e0cfd23183fd27394ff5391c37ee659696e1240a43299d4bdfb31776f6d60b

data/CHANGELOG.md

@@ -1,3 +1,20 @@
+# 0.20.0
+- FEATURE: gpg support for CSV source (@elmobp)
+- FEATURE: slackdiff (@natm)
+- FEATURE: gitcrypt output model (@clement-parisot)
+- FEATURE: model specific credentials (@davromaniak)
+- FEATURE: hierarchical json in http source model
+- FEATURE: next-adds-job config toggle (to add new job when ever /next is called)
+- FEATURE: netgear model (@aschaber1)
+- FEATURE: zhone model (@rfdrake)
+- FEATURE: tplink model (@mediumo)
+- FEATURE: oneos model (@crami)
+- FEATURE: cisco NGA model (@udhos)
+- FEATURE: voltaire model (@clement-parisot)
+- FEATURE: siklu model (@bdg-robert)
+- FEATURE: voss model (@ospfbgp)
+- BUGFIX: ios, cumulus, ironware, nxos, fiberdiver, aosw, fortios, comware, procurve, opengear, timos, routeros, junos, asa, aireos, mlnxos, pfsense, saos, powerconnect, firewareos, quantaos
+
 # 0.19.0
 - FEATURE: allow setting ssh_keys (not relying on openssh config) (@denvera)
 - FEATURE: fujitsupy model (@stokbaek)

data/Dockerfile

@@ -3,9 +3,23 @@ MAINTAINER Samer Abdel-Hafez <sam@arahant.net>
 
 RUN add-apt-repository ppa:brightbox/ruby-ng && \
 	apt-get update && \
-  apt-get install -y ruby2.3 ruby2.3-dev libsqlite3-dev libssl-dev pkg-config make cmake libssh2-1-dev
+  apt-get install -y ruby2.3 ruby2.3-dev libsqlite3-dev libssl-dev pkg-config make cmake libssh2-1-dev git g++
 
-RUN gem install oxidized oxidized-web --no-ri --no-rdoc
+RUN mkdir -p /tmp/oxidized
+COPY . /tmp/oxidized/
+WORKDIR /tmp/oxidized
+
+RUN gem build oxidized.gemspec
+RUN gem install oxidized-*.gem
+
+# web interface
+RUN gem install oxidized-web --no-ri --no-rdoc
+
+# dependencies for hooks
+RUN gem install aws-sdk
+RUN gem install slack-api
+
+RUN rm -rf /tmp/oxidized
 
 RUN apt-get remove -y ruby-dev pkg-config make cmake
 
@@ -13,6 +27,7 @@ RUN apt-get -y autoremove
 
 ADD extra/oxidized.runit /etc/service/oxidized/run
 ADD extra/auto-reload-config.runit /etc/service/auto-reload-config/run
+ADD extra/update-ca-certificates.runit /etc/service/update-ca-certificates/run
 
 VOLUME ["/root/.config/oxidized"]
 EXPOSE 8888/tcp

data/Gemfile

@@ -1,3 +1,2 @@
 source 'https://rubygems.org'
-
 gemspec

data/Gemfile.lock

@@ -1,10 +1,10 @@
 PATH
   remote: .
   specs:
-    oxidized (0.19.0)
+    oxidized (0.20.0)
       asetus (~> 0.1)
       net-ssh (~> 3.0.2)
-      net-telnet
+      net-telnet (~> 0)
       rugged (~> 0.21, >= 0.21.4)
       slop (~> 3.5)
 
@@ -12,20 +12,20 @@ GEM
   remote: https://rubygems.org/
   specs:
     asetus (0.3.0)
-    coderay (1.1.0)
+    coderay (1.1.1)
+    git (1.3.0)
     metaclass (0.0.4)
     method_source (0.8.2)
-    minitest (5.9.0)
-    mocha (1.1.0)
+    minitest (5.10.1)
+    mocha (1.2.1)
       metaclass (~> 0.0.1)
     net-ssh (3.0.2)
     net-telnet (0.1.1)
-    pry (0.10.3)
+    pry (0.11.0.pre2)
       coderay (~> 1.1.0)
       method_source (~> 0.8.1)
-      slop (~> 3.4)
     rake (10.5.0)
-    rugged (0.23.3)
+    rugged (0.25.1.1)
     slop (3.6.0)
 
 PLATFORMS
@@ -33,6 +33,7 @@ PLATFORMS
 
 DEPENDENCIES
   bundler (~> 1.10)
+  git (~> 1)
   minitest (~> 5.8)
   mocha (~> 1.1)
   oxidized!
@@ -40,4 +41,4 @@ DEPENDENCIES
   rake (~> 10.0)
 
 BUNDLED WITH
-   1.11.2
+   1.14.6

data/README.md

@@ -19,6 +19,7 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
 2. [Installation](#installation)
     * [Debian](#debian)
     * [CentOS, Oracle Linux, Red Hat Linux](#centos-oracle-linux-red-hat-linux)
+    * [BSD](#freebsd)
 3. [Initial Configuration](#configuration)
 4. [Installing Ruby 2.1.2 using RVM](#installing-ruby-2.1.2-using-rvm)
 5. [Running with Docker](#running-with-docker)
@@ -32,6 +33,7 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
       * [Source: Mysql](#source-mysql)
     * [Source: HTTP](#source-http)
     * [Output: GIT](#output-git)
+    * [Output: GIT-Crypt](#output-git-crypt)
     * [Output: HTTP](#output-http)
     * [Output: File](#output-file)
     * [Output types](#output-types)
@@ -64,6 +66,8 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
    * [C4CMTS](lib/oxidized/model/c4cmts.rb)
  * Aruba
    * [AOSW](lib/oxidized/model/aosw.rb)
+ * Avaya
+   * [VOSS](lib/oxidized/model/voss.rb)
  * Brocade
    * [FabricOS](lib/oxidized/model/fabricos.rb)
    * [Ironware](lib/oxidized/model/ironware.rb)
@@ -75,13 +79,14 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
  * Check Point
    * [GaiaOS](lib/oxidized/model/gaiaos.rb)
  * Ciena
-   * [SOAS](lib/oxidized/model/saos.rb)
+   * [SAOS](lib/oxidized/model/saos.rb)
  * Cisco
    * [AireOS](lib/oxidized/model/aireos.rb)
    * [ASA](lib/oxidized/model/asa.rb)
    * [CatOS](lib/oxidized/model/catos.rb)
    * [IOS](lib/oxidized/model/ios.rb)
    * [IOSXR](lib/oxidized/model/iosxr.rb)
+   * [NGA](lib/oxidized/model/cisconga.rb)
    * [NXOS](lib/oxidized/model/nxos.rb)
    * [SMB (Nikola series)](lib/oxidized/model/ciscosmb.rb)
  * Citrix
@@ -125,6 +130,7 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
    * [ScreenOS (Netscreen)](lib/oxidized/model/screenos.rb)
  * Mellanox
    * [MLNX-OS](lib/oxidized/model/mlnxos.rb)
+   * [Voltaire](lib/oxidized/model/voltaire.rb)
  * Mikrotik
    * [RouterOS](lib/oxidized/model/routeros.rb)
  * Motorola
@@ -132,10 +138,14 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
  * MRV
    * [MasterOS](lib/oxidized/model/masteros.rb)
    * [FiberDriver](lib/oxidized/model/fiberdriver.rb)
+ * Netgear
+   * [Netgear](lib/oxidized/model/netgear.rb)
  * Netonix
    * [WISP Switch (As Netonix)](lib/oxidized/model/netonix.rb)
  * Nokia (formerly TiMetra, Alcatel, Alcatel-Lucent)
    * [SR OS (TiMOS)](lib/oxidized/model/timos.rb)
+ * OneAccess
+   * [OneOS](lib/oxidized/model/oneos.rb)
  * Opengear
    * [Opengear](lib/oxidized/model/opengear.rb)
  * Palo Alto
@@ -144,16 +154,22 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacemen
  * [pfSense](lib/oxidized/model/pfsense.rb)
  * Quanta
    * [Quanta / VxWorks 6.6 (1.1.0.8)](lib/oxidized/model/quantaos.rb)
+ * Siklu
+   * [EtherHaul](lib/oxidized/model/siklu.rb)
  * Supermicro
    * [Supermicro](lib/oxidized/model/supermicro.rb)
  * Trango Systems
    * [Trango](lib/oxidized/model/trango.rb)
+ * TPLink
+   * [TPLink](lib/oxidized/model/tplink.rb)
  * Ubiquiti
    * [AirOS](lib/oxidized/model/airos.rb)
    * [Edgeos](lib/oxidized/model/edgeos.rb)
    * [EdgeSwitch](lib/oxidized/model/edgeswitch.rb)
  * Watchguard
    * [Fireware OS](lib/oxidized/model/firewareos.rb)
+ * Zhone
+   * [Zhone (OLT and MX)](lib/oxidized/model/zhoneolt.rb)
  * Zyxel
    * [ZyNOS](lib/oxidized/model/zynos.rb)
 
@@ -186,6 +202,27 @@ gem install oxidized
 gem install oxidized-script oxidized-web
 ```
 
+## FreeBSD
+Use RVM to install Ruby v2.1.2
+
+Install all required packages and gems.
+
+```shell
+pkg install cmake pkgconf
+gem install oxidized
+gem install oxidized-script oxidized-web
+```
+
+
+
+## Build from Git
+```shell
+git clone https://github.com/ytti/oxidized.git
+cd oxidized/
+gem build *.gemspec
+gem install pkg/*.gem
+```
+
 # Configuration
 
 Oxidized configuration is in YAML format. Configuration files are subsequently sourced from ```/etc/oxidized/config``` then ```~/.config/oxidized/config```. The hashes will be merged, this might be useful for storing source information in a system wide file and  user specific configuration in the home directory (to only include a staff specific username and password). Eg. if many users are using ```oxs```, see [Oxidized::Script](https://github.com/ytti/oxidized-script).
@@ -221,7 +258,7 @@ Oxidized supports ```CSV```, ```SQLite``` and ```HTTP``` as source backends. The
 
 ## Outputs
 
-Possible outputs are either ```file``` or ```git```. The file backend takes a destination directory as argument and will keep a file per device, with most recent running version of a device. The GIT backend (recommended) will initialize an empty GIT repository in the specified path and create a new commit on every configuration change. Take a look at the [Cookbook](#cookbook) for more details.
+Possible outputs are either ```file```, ```git``` or ```git-crypt```. The file backend takes a destination directory as argument and will keep a file per device, with most recent running version of a device. The GIT backend (recommended) will initialize an empty GIT repository in the specified path and create a new commit on every configuration change. The GIT-Crypt backend will also initialize a GIT repository but every configuration push to it will be encrypted on the fly by using ```git-crypt``` tool. Take a look at the [Cookbook](#cookbook) for more details.
 
 Maps define how to map a model's fields to model [model fields](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model). Most of the settings should be self explanatory, log is ignored if `use_syslog`(requires Ruby >= 2.0) is set to `true`.
 
@@ -233,12 +270,15 @@ oxidized
 
 Now tell Oxidized where it finds a list of network devices to backup configuration from. You can either use CSV or SQLite as source. To create a CSV source add the following snippet:
 
+Note: If gpg is set to anything other than false it will attempt to decrypt the file contents
 ```
 source:
   default: csv
   csv:
     file: ~/.config/oxidized/router.db
     delimiter: !ruby/regexp /:/
+    gpg: false
+    gpg_password: 'password'
     map:
       name: 0
       model: 1
@@ -344,16 +384,26 @@ If you want to have the config automatically reloaded (e.g. when using a http so
 docker run -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -e CONFIG_RELOAD_INTERVAL=3600 -t oxidized/oxidized:latest
 ```
 
+If you need to use an internal CA (e.g. to connect to an private github instance)
+
+```
+docker run -v /etc/oxidized:/root/.config/oxidized -v /path/to/MY-CA.crt:/usr/local/share/ca-certificates/MY-CA.crt -p 8888:8888/tcp -e UPDATE_CA_CERTIFICATES=true -t oxidized/oxidized:latest
+```
+
 ## Cookbook
 ### Debugging
-In case a model plugin doesn't work correctly (ios, procurve, etc.), you can enable live debugging of SSH/Telnet sessions. Just add a ```debug``` option, specifying a log file destination to the ```input``` section.
+In case a model plugin doesn't work correctly (ios, procurve, etc.), you can enable live debugging of SSH/Telnet sessions. Just add a ```debug``` option containing the value true to the ```input``` section. The log files will be created depending on the parent directory of the logfile option.
 
-The following example will log an active ssh session to ```/home/fisakytt/.config/oxidized/log_input-ssh``` and telnet to ```log_input-telnet```. The file will be truncated on each consecutive ssh/telnet session, so you need to put a ```tailf``` or ```tail -f``` on that file!
+The following example will log an active ssh/telnet session ```/home/oxidized/.config/oxidized/log/<IP-Adress>-<PROTOCOL>```. The file will be truncated on each consecutive ssh/telnet session, so you need to put a ```tailf``` or ```tail -f``` on that file!
 
 ```
+log: /home/oxidized/.config/oxidized/log
+
+...
+
 input:
   default: ssh, telnet
-  debug: /tmp/oxidized_log_input
+  debug: true
   ssh:
     secure: false
 ```
@@ -403,7 +453,7 @@ vars:
 
 ### Source: CSV
 
-One line per device, colon seperated.
+One line per device, colon seperated. If `ip` isn't present, a DNS lookup will be done against `name`.  For large installations, setting `ip` will dramatically reduce startup time.
 
 ```
 source:
@@ -413,11 +463,12 @@ source:
     delimiter: !ruby/regexp /:/
     map:
       name: 0
-      model: 1
-      username: 2
-      password: 3
+      ip: 1
+      model: 2
+      username: 3
+      password: 4
     vars_map:
-      enable: 4
+      enable: 5
 ```
 
 ### SSH Proxy Command
@@ -581,6 +632,72 @@ output:
 
 ```
 
+### Output: Git-Crypt
+
+This uses the gem git and system git-crypt interfaces. Have a look at [GIT-Crypt](https://www.agwa.name/projects/git-crypt/) documentation to know how to install it.
+Additionally to user and email informations, you have to provide the users ID that can be a key ID, a full fingerprint, an email address, or anything else that uniquely identifies a public key to GPG (see "HOW TO SPECIFY A USER ID" in the gpg man page).
+
+
+For a single repositories for all devices:
+
+``` yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/devices"
+    users:
+      - "0x0123456789ABCDEF"
+      - "<user@example.com>"
+```
+
+And for groups repositories:
+
+``` yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    user: Oxidized
+    email: o@example.com
+    repo: "/var/lib/oxidized/git-repos/default"
+    users:
+      - "0xABCDEF0123456789"
+      - "0x0123456789ABCDEF"
+```
+
+Oxidized will create a repository for each group in the same directory as the `default`. For
+example:
+
+``` csv
+host1:ios:first
+host2:nxos:second
+```
+
+This will generate the following repositories:
+
+``` bash
+$ ls /var/lib/oxidized/git-repos
+
+default.git first.git second.git
+```
+
+If you would like to use groups and a single repository, you can force this with the `single_repo` config.
+
+``` yaml
+output:
+  default: gitcrypt
+  gitcrypt:
+    single_repo: true
+    repo: "/var/lib/oxidized/devices"
+    users:
+      - "0xABCDEF0123456789"
+      - "0x0123456789ABCDEF"
+
+```
+
+Please note that user list is only updated once at creation.
+
 ### Output: Http
 
 POST a config to the specified URL
@@ -724,6 +841,32 @@ map:
   name: 1
   group: 2
 ```
+For model specific credentials
+
+```
+models:
+  junos:
+    username: admin
+    password: password
+  ironware:
+    username: admin
+    password: password
+    vars: 
+      enable: enablepassword
+  apc_aos:
+    username: apc
+    password: password
+```
+
+### Triggered backups
+
+A node can be moved to head-of-queue via the REST API `GET/POST /node/next/[NODE]`.
+
+In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added:
+
+```
+next_adds_job: true
+```
 
 # Hooks
 You can define arbitrary number of hooks that subscribe different events. The hook system is modular and different kind of hook types can be enabled.
@@ -748,6 +891,7 @@ Command is executed with the following environment:
 ```
 OX_EVENT
 OX_NODE_NAME
+OX_NODE_IP
 OX_NODE_FROM
 OX_NODE_MSG
 OX_NODE_GROUP
@@ -844,6 +988,57 @@ AWS SNS hook requires the following configuration keys:
 
 Your AWS credentials should be stored in `~/.aws/credentials`.
 
+## Hook type: slackdiff
+
+The `slackdiff` hook posts colorized config diffs to a [Slack](http://www.slack.com) channel of your choice. It only triggers for `post_store` events.
+
+You will need to manually install the `slack-api` gem on your system:
+
+```
+gem install slack-api
+```
+
+Configuration example:
+
+``` yaml
+hooks:
+  slack:
+    type: slackdiff
+    events: [post_store]
+    token: SLACK_BOT_TOKEN
+    channel: "#network-changes"
+```
+
+# Extra
+
+## Ubuntu SystemV init setup
+
+The init script assumes that you have a used named 'oxidized' and that oxidized is in one of the following paths:
+
+```
+/sbin
+/bin
+/usr/sbin
+/usr/bin
+/usr/local/bin
+```
+
+1.)Copy init script from extra/ folder to /etc/init.d/oxidized
+2.)Setup /var/run/
+
+```
+mkdir /var/run/oxidized
+chown oxidized:oxidized /var/run/oxidized
+```
+
+3.)Make oxidized start on boot
+
+```
+update-rc.d oxidized deafults
+```
+
+Note the channel name must be in quotes.
+
 # Ruby API
 
 The following objects exist in Oxidized.