oxidized 0.35.0 → 0.37.0

data/docs/Hooks.md

@@ -109,50 +109,102 @@ hooks:
 
 ## Hook type: githubrepo
 
-Note: You must not use the same name as any local repo configured under output. Make sure your 'git' output has a unique name that does not match your remote_repo.
+The `githubrepo` hook executes a `git push` to a configured `remote_repo` when
+the specified event is triggered.
 
-The `githubrepo` hook executes a `git push` to a configured `remote_repo` when the specified event is triggered.
+### Configuration keys
 
-Several authentication methods are supported:
+| Key           | Description |
+|---------------|-------------|
+| `remote_repo` | The remote repository to push to. Use a URL string (no groups) or a group dictionary (see [Using groups](#using-groups)). |
+| `username`    | Username for authentication. Defaults to the user part of the `remote_repo` URI, falling back to `git`. |
+| `password`    | Password for username/password authentication. |
+| `privatekey`  | Path to the private key file. Must be in legacy PEM format (see note below). |
+| `publickey`   | Path to the public key file (optional — inferred from `privatekey` + `.pub` if omitted). |
 
-* Provide a `password` for username + password authentication
-* Provide both a `publickey` and a `privatekey` for ssh key-based authentication
-* Provide only a `privatekey` (public key filename is assumed to be `privatekey` + "`.pub`"
-* Don't provide any credentials for ssh-agent authentication
+Notes:
+- `remote_repo` must not match the name of any local `git` output repo configured under `output`. Use unique names for each.
+- If using SSH key authentication with a passphrase-protected private key, provide the passphrase with the `OXIDIZED_SSH_PASSPHRASE` environment variable.
+- The `privatekey` must be in the legacy PEM format (`BEGIN RSA PRIVATE KEY`), not the newer OpenSSH format (`BEGIN OPENSSH PRIVATE KEY`). See [#1877](https://github.com/ytti/oxidized/issues/1877) and [#2324](https://github.com/ytti/oxidized/issues/2324).
+- To convert an existing key to PEM format, run:
+  ```shell
+  ssh-keygen -p -m PEM -f $MY_KEY_HERE
+  ```
 
-The username will be set to the relevant part of the `remote_repo` URI, with a fallback to `git`. It is also possible to provide one by setting the `username` configuration key.
+### Authentication methods
 
-For ssh key-based authentication, it is possible to set the environment variable `OXIDIZED_SSH_PASSPHRASE` to a passphrase if the private key requires it.
+Choose one of the following methods:
 
-`githubrepo` hook recognizes the following configuration keys:
+| Method                        | Required keys |
+|-------------------------------|---------------|
+| Username + password           | `username` (optional), `password` |
+| SSH key pair                  | `privatekey`, `publickey` (optional - assumed to be at `privatekey` + `.pub`) |
+| SSH agent                     | no credentials needed |
 
-* `remote_repo`: the remote repository to be pushed to.
-* `username`: username for repository auth.
-* `password`: password for repository auth.
-* `publickey`: public key file path for repository auth. (optional)
-* `privatekey`: private key file path for repository auth.
-  * NOTE: this key needs to be in the legacy PEM format, not the newer OpenSSL format [#1877](https://github.com/ytti/oxidized/issues/1877), [#2324](https://github.com/ytti/oxidized/issues/2324)
-    * To convert a key beginning with `BEGIN OPENSSH PRIVATE KEY` to the legacy PEM format, run this command:
-      `ssh-keygen -p -m PEM -f $MY_KEY_HERE`
+### Configuration examples
 
-When using groups, `remote_repo` must be a dictionary of groups that the hook should apply to. If a group is missing from the dictionary, no action will be taken.
+**Username and password:**
 
-The dictionary entry can either be a url alone:
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    username: user
+    password: pass
+```
+
+**SSH key pair:**
 
 ```yaml
 hooks:
   push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    publickey: /root/.ssh/id_rsa.pub
+    privatekey: /root/.ssh/id_rsa
+```
+
+
+**SSH agent:**
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+```
+
+### Using groups
+
+When using groups and `single_repo` is set to `true` (default) in the
+configuration section output/git, `remote_repo` must be a dictionary mapping
+group names to remote repositories. Groups not listed in the dictionary are
+silently skipped.
+
+Each entry can be either a plain URL string:
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
     remote_repo:
       routers: git@git.intranet:oxidized/routers.git
       switches: git@git.intranet:oxidized/switches.git
       firewalls: git@git.intranet:oxidized/firewalls.git
 ```
 
-... or it can be a dictionary with `url` and `privatekey` specified:
+...or a dictionary with `url` and `privatekey` to use a per-group SSH key:
 
 ```yaml
 hooks:
   push_to_remote:
+    type: githubrepo
+    events: [post_store]
     remote_repo:
       routers:
         url: git@git.intranet:oxidized/routers.git
@@ -165,58 +217,31 @@ hooks:
         privatekey: /root/.ssh/id_rsa_firewalls
 ```
 
-Both forms can be mixed and matched.
+Both forms can be mixed within the same configuration.
 
-### githubrepo hook configuration example
+### Custom branch name
 
-Authenticate with a username and a password without groups in use:
+The `githubrepo` hook uses the branch name from the
+[git output](Outputs.md#output-git) as the remote branch name. When the
+repository is first created, Oxidized uses the default branch name from
+`git config --global init.defaultBranch`. The default is `master`.
 
-```yaml
-hooks:
-  push_to_remote:
-    type: githubrepo
-    events: [post_store]
-    remote_repo: git@git.intranet:oxidized/test.git
-    username: user
-    password: pass
-```
+You can manually rename the branch after Oxidized has already created the
+repository. Be aware that you may break things, so make backups.
 
-Authenticate with the username `git` and an ssh key:
+To rename the branch after Oxidized has already created the repository:
 
-```yaml
-hooks:
-  push_to_remote:
-    type: githubrepo
-    events: [post_store]
-    remote_repo: git@git.intranet:oxidized/test.git
-    publickey: /root/.ssh/id_rsa.pub
-    privatekey: /root/.ssh/id_rsa
-```
+1. Stop oxidized.
+2. Back up your oxidized git repository.
+3. Change to your oxidized git repository directory.
+4. Inspect the current branches: `git branch -avv`
+5. Rename the local branch: `git branch -m <NewName>`
+6. Remove the stale remote-tracking reference: `git branch -r -d origin/<OldName>`
+7. Verify the result: `git branch -avv`
+8. Restart oxidized.
 
-### Custom branch name
-Githubrepo will use the branch name used in the
-[git output](Outputs.md#output-git) as a remote branch name. When creating the
-git repository for the first time, Oxidized uses the default branch name
-configured in git with `git config --global init.defaultBranch <Name>`. The
-default is `master`.
-
-If you need to rename the branch name after Oxidized has created it, you may do
-it manually. Be aware that you may break things. Make backups and do not
-complain if something goes wrong!
-
-1. Stop oxidized (no one should access the git repository while doing the
-following steps)
-2. Make a backup of your oxidized data, especially the git repository
-3. Change directory to your oxidized git repository (as configured in oxidized
-configuration file)
-4. Inspect the current branches with `git branch -avv`
-5. Rename the default branch with `git branch -m <NewName>`
-6. Remove the reference to the old remote branch  with
-   `git branch -r -d origin/<OldName>`
-6. Inspect the change with `git branch -avv`
-7. Restart oxidized - you're done!
-
-Note that you will also have to clean your remote git repository.
+Oxidized will push to a new remote branch. When everything is fine, you can
+remove the old branch from the remote repository.
 
 ## Hook type: awssns
 

data/docs/Inputs.md

@@ -177,27 +177,59 @@ input:
     passive: false
 ```
 
+## HTTP
+### Supported HTTP Methods
 
-## Debugging
+The HTTP input supports the following HTTP methods:
+- `:get`  - for GET requests
+- `:post` - for POST requests
 
-In case a model plugin doesn't work correctly (ios, procurve, etc.), you can
-enable live debugging of SSH/Telnet sessions. Just add a `debug` option
-containing the value true to the `input` section. The log files will be created
-depending on the parent directory of the logfile option.
+These methods are used internally by models that require HTTP-based 
+configuration retrieval. Models can use `get_http()` and `post_http()` methods 
+provided by the HTTP input.
+
+Example usage in a model:
+
+```ruby
+cfg :http do
+  post_response = post_http('/some/path', payload, 'Some-Extra-Header' => 'value')
+  get_response  = get_http('/some/path')
+end
+```
 
-The following example will log an active ssh/telnet session
-`/home/oxidized/.config/oxidized/log/<IP-Address>-<PROTOCOL>`. The file will be
-truncated on each consecutive ssh/telnet session, so you need to put a `tailf`
-or `tail -f` on that file!
+HTTP input can be enabled by adding this block to the configuration file:
 
 ```yaml
-log: /home/oxidized/.config/oxidized/log
+input:
+  http:
+    scheme: https
+    ssl_verify: true
+    timeout: 30
+```
 
-# ...
+## Debugging
+In case a model plugin doesn't work correctly (ios, procurve, etc.), you can
+enable live debugging of SSH and Telnet sessions with the `debug` option of
+the `input` section.
 
+Starting with version 0.37.0, `debug` can take different values:
+- `text`: log input and output to a text file (ssh, telnet)
+- `yaml`: produce a yaml simulation file (ssh, scp)
+- `library`: activate debug logging of the underlying library
+- a combination of the options above (`text, yaml`)
+- `true`; activate all debugging options (Only option for versions prior 0.37.0)
+
+The log files will be created in `~/.config/oxidized/logs/` (or `$OXIDIZED_LOGS/logs/`).
+
+The following example will log an active ssh/telnet session to
+`~/.config/oxidized/logs/<IP-Address>-<PROTOCOL>-<timestamp>.txt` and for ssh
+`~/.config/oxidized/logs/<IP-Address>-<PROTOCOL>-<timestamp>.yaml`. A new file
+is created for each session.
+
+```yaml
 input:
   default: ssh, telnet
-  debug: true
+  debug: yaml, text
   ssh:
     secure: false
   http:

data/docs/Model-Notes/APC.md

@@ -0,0 +1,72 @@
+# APC Configuration
+The configuration of APC Network Management Cards can be downloaded using FTP
+and SCP. You can retrieve serial numbers and OS version information through
+an SSH connection.
+
+APC OS does not have the ability to display the config.ini within an SSH shell.
+A ticket was opened with APC support to enable "cat config.ini"
+within an SSH shell, but APC declined to implement this feature.
+
+To overcome this limitation, a capability to run against multiple inputs (SSH + SCP)
+has been implemented in Oxidized and in the [model ApcAos](/lib/oxidized/model/apcaos.rb).
+
+The old model apc_aos (SCP/FTP only) is deprecated and will be removed in a
+future release. Migrate to ApcAos.
+
+## How do I activate FTP/SCP input?
+To download the configuration with FTP or SCP, you must activate it
+as an input in the Oxidized configuration. If you don't activate the input,
+Oxidized will fail for the node with an error.
+
+You probably also need to increase the default timeout to something about 60
+seconds, as the APC are really slow, and need about 30 seconds to complete.
+
+The configuration can be done either globally or only for the ApcAos model.
+
+### Global Configuration
+The global configuration would look like this. Note that Oxidized will try every
+input type in the given order until it succeeds, or it will report a failure.
+```yaml
+timeout: 60
+input:
+  default: ssh, ftp, scp
+```
+The order in the configuration is relevant. With this configuration, the ApcAos
+model will run SSH first, then it will try FTP, and if FTP fails SCP.
+
+### Model-Specific Configuration
+
+Configuration for activating only the SCP input for ApcAos only:
+```yaml
+input:
+  default: ssh
+models:
+  apcaos:
+    input: ssh, scp
+    timeout: 60
+```
+
+### Setting Specific Credentials
+You can also set a specific username and password for ApcAos only:
+```yaml
+username: default-user
+password: default-password
+input:
+  default: ssh
+models:
+  ApcAos:
+    username: apc-user
+    password: apc-password
+    input: ssh, scp
+    timeout: 60
+```
+
+## Why do I partially get CR + LF?
+The config.ini file has a DOS-Format (CR + LF), and is saved without
+modifications, so that it can be uploaded to the device.
+
+Outputs from ssh are stored without CR, so the first part of the file is
+without CR and config.ini with CR + LF.
+
+This is expected behavior and should not affect the functionality of the backup
+or restore process.

data/docs/Model-Notes/ExaLink.md

@@ -0,0 +1,43 @@
+# Cisco Nexus 3550-F (ExaLink Fusion)
+
+The Cisco Nexus 3550-F (formerly Exablaze ExaLink Fusion) is an ultra-low-latency
+Layer 1/2 switch platform based on FPGA technology, primarily used in high-frequency
+trading and HPC environments. It runs a custom Linux-based OS with a proprietary CLI
+and JSON RPC API.
+
+## Device Configuration
+
+Create a read-only user for Oxidized on the device:
+
+```
+admin@N3550-F> configure user oxidized password <password>
+admin@N3550-F> configure user oxidized privilege read-only
+```
+
+## Oxidized Configuration
+
+```yaml
+source:
+  default: csv
+  csv:
+    file: "/home/oxidized/.config/oxidized/router.db"
+    delimiter: !ruby/regexp /:/
+    map:
+      name: 0
+      model: 1
+```
+
+Example `router.db` entry:
+
+```bash
+myswitch.example.com:exalink
+```
+
+## Notes
+
+- Both SSH and Telnet are supported. SSH is recommended.
+- The model collects `show version` (excluding uptime to avoid noisy diffs),
+  `show port`, and `show running-config`.
+- Timestamps (`!Time:`) are stripped from the running config to avoid noisy diffs.
+- The device prompt format is `hostname#` or `hostname>`.
+- This model was developed and tested against software version 1.16.0.

data/docs/Model-Notes/Fortinet.md

@@ -0,0 +1,75 @@
+# Fortinet models
+There are two models for Fortinet devices:
+- fortigate: for the FortiGate firewalls
+- fortios: for VM-Based appliances (FortiManager, FortiADC, FortiAnalyzer...)
+
+# Notes for both models
+## Configuration changes / hiding passwords
+Fortigate and Fortios re-encrypt their passwords every time the configuration is shown.
+This results in a lot of apparent configuration changes on every pull.
+
+To avoid this, you have two options:
+- remove secrets
+- save significant changes only
+
+### Remove secrets
+If you don't want to have a new version every time the configuration is
+downloaded, you can hide all secrets. Beware that you won't have a full backup, as all passwords will be replaced with <configuration removed>
+
+```yaml
+models:
+  fortigate:
+    vars:
+      remove_secret: true
+```
+
+### Save significant changes only
+You can [store the configuration only on significant changes](/docs/Configuration.md#store-configuration-only-on-significant-changes)
+by setting the [variable](#options-credentials-vars-etc-precedence)
+`output_store_mode` to `on_significant`. On FortiGate and FortiOS, this
+prevents Oxidized from saving a configuration when there were only changes to
+the encrypted passwords. Beware that you won't have the last backup if you only
+changed a password.
+
+```yaml
+vars:
+  output_store_mode: on_significant
+```
+
+# Notes for the FortiGate model
+## Create user oxidized with ED25519 public key
+You can use a user/password for retrieving the configuration or use a SSH public key:
+
+```text
+config system admin
+edit oxidized
+set trusthost1 192.0.2.1 255.255.255.255
+set accprofile "super_admin_readonly"
+set ssh-public-key1 "ssh-ed25519 AAAAThisIsJustAnExampleKey_UseYourOxidizedPUBLICKEY oxidized@librenms"
+end
+```
+
+## config vs. full config
+On FortiGate, you can get a configuration without default values (`show`) or
+including all default values (`show full-configuration`).
+
+The full configuration can be long and may cause timeouts.
+Starting with with oxidized 0.30.1, the default is to get the short configuration.
+
+If you need the full configuration, you can activate it in oxidized config file:
+```yaml
+models:
+  fortigate:
+    vars:
+      fullconfig: true
+```
+## Autoupdate
+You can get the result of `diagnose autoupdate version` by setting the [variable](#options-credentials-vars-etc-precedence) `fortigate_autoupdate` to `true`:
+
+```yaml
+vars:
+  fortigate_autoupdate: true
+```
+
+Note that the variable `fortios_autoupdate` is deprecated and will be removed
+in a future Version of Oxidized. Use `fortigate_autoupdate` instead.

data/docs/Model-Notes/GrandstreamHT8xx.md

@@ -0,0 +1,8 @@
+# Grandstream HT8xx
+
+You need to have enabled access to device by SSH.
+Connection using user/password for retrieve the configuration containing XML file with all params stored in device memory.
+
+Tested on hardware: v1 and software version: 1.0.61.2.
+
+Back to [Model-Notes](README.md)

data/docs/Model-Notes/IvantiConnectSecure.md

@@ -0,0 +1,59 @@
+### Ivanti Connect Secure (ICS)
+
+#### Overview
+
+This model provides support for Ivanti Connect Secure (ICS) appliances using REST API ([official documentation](https://help.ivanti.com/ps/help/en_US/ICS/22.x/22.7R2/22.xICSAG.pdf)).
+ICS stores its configuration as a binary ZIP archive (with `system.cfg` and `user.cfg` files) which is retrieved using the `/api/v1/system/binary-configuration` endpoint.
+
+The model performs an initial authentication against `/api/v1/realm_auth` using Basic Auth (`username`/`password`) and retrieves a temporary `api_key`.
+This key is then used for all further API requests during the Oxidized collection cycle.
+
+The model is designed to work with standard ICS deployments without requiring command-line access to the device.
+
+#### How Configuration Is Retrieved
+
+1. Oxidized authenticates using:
+
+```bash
+POST /api/v1/realm_auth
+```
+
+with:
+- Basic Auth: `username` + `password`
+- JSON body `{"realm": "<realm>"}`
+
+
+2. ICS returns a temporary:
+
+```json
+{ "api_key": "<token>" }
+```
+
+
+3. The configuration is fetched from:
+
+```bash
+GET /api/v1/system/binary-configuration
+```
+
+with:
+- `api_key` as `username`
+- `''` as `password`
+
+ICS responds with a BASE64-encoded ZIP archive containing the device configuration.
+The model stores this BASE64 value as a single uninterrupted line.
+
+
+#### Required Node Configuration
+
+In source (CSV, HTTP, SQL, etc.), simply define:
+
+```yaml
+model: ivanti
+username: <your username>
+password: <your password>
+vars:
+  realm: <your realm>   # Optional, default = "Users"
+```
+
+The model will automatically handle authentication and obtain the API key as stated above.

data/docs/Model-Notes/RouterOS.md

@@ -12,4 +12,17 @@ and attach the public key.
 
 Oxidized can now retrieve your configuration!
 
+## Save significant changes only
+
+You can [store the configuration only on significant changes](/docs/Configuration.md#store-configuration-only-on-significant-changes)
+by setting the [variable](/docs/Configuration.md#options-credentials-vars-etc-precedence)
+`output_store_mode` to `on_significant`. On RouterOS, this prevents Oxidized from saving a
+new configuration version when only the system history has changed without any actual
+configuration change.
+
+```yaml
+vars:
+  output_store_mode: on_significant
+```
+
 Back to [Model-Notes](README.md)

data/docs/Model-Notes/TrueNAS.md

@@ -0,0 +1,23 @@
+# TrueNAS
+
+This should support both older TrueNAS CORE (FreeBSD-based) and newer
+TrueNAS SCALE (Linux-based) devices.
+
+## Authentication
+
+Ensure that the user configured for oxidized to login to your device has the
+permissions to read the configuration database. On older CORE instances, this
+would just work without sudo. On newer devices, the `/data/freenas-v1.db` file
+can only be read by the root user.
+
+On SCALE devices with Apps support, it's also necessary to add some privileges
+to read the container configurations for any apps you have installed, which can
+be found under `/mnt/.ix-apps`.
+
+You can make sure that the user that oxidized uses to login (`oxidized` in this
+example) can dump the configuration using `sudo` by adding something like this
+to your `/etc/sudoers` file:
+
+```
+oxidized ALL=(ALL) NOPASSWD: /usr/bin/find /mnt/.ix-apps/app_configs *, /usr/bin/sqlite3 -readonly file\:/data/freenas-v1.db *
+```

data/docs/ModelUnitTests.md

@@ -4,6 +4,7 @@ effort to use. There are three different default unit tests for models:
 - [Device Simulation](ModelUnitTests.md#device-simulation)
 - [Device Prompt](ModelUnitTests.md#device-prompt)
 - [Secrets](ModelUnitTests.md#secrets)
+- [Significant Changes](ModelUnitTests.md#significant-changes)
 
 You only need to provide test files under [/spec/model/data](/spec/model/data),
 and the tests will be run automatically with `rake test`. See
@@ -187,6 +188,28 @@ pass:
   - 'hash-mgmt-user rocks password hash <secret removed> usertype read-only'
 ```
 
+## Significant Changes
+You can test if the model correctly detects significant changes from a YAML
+simulation file (`#simulation.yaml`) when run with variable
+`output_store_mode` set to `on_significant`.
+
+The output is checked against a file with the same 
+prefix as the yaml simulation file, but with the suffix
+`#significant_changes.yaml`.
+
+The `#significant_changes.yaml` file contains two sections with a list of
+strings or regular expressions to test:
+- pass: the test passes only if the output contains these strings (significant changes).
+- fail: the test fails if the output contain these strings (non-significant changes).
+
+```yaml
+pass:
+ - "! Processor ID: FCL2XXXXXXX"
+fail:
+ - "! Last configuration change at 13:57:08 CET Wed Mar 13 2024"
+ - "! NVRAM config last updated at 15:26:39 CET Wed Mar 13 2024 by oxidized"
+```
+
 ## Custom tests
 When you write custom tests for your models, please do not use the filenames
 mentioned above, as it will interfere with the standard tests. If you need to

data/docs/Outputs.md

@@ -211,9 +211,9 @@ output:
 
 Please note that user list is only updated once at creation.
 
-## Output: Http
+## Output: HTTP
 
-The HTTP output will POST a config to the specified HTTP URL. Basic username/password authentication is supported.
+The HTTP output will POST a config as JSON to the specified HTTP URL. It supports HTTP Basic Authentication, custom headers, and SSL/TLS verification control.
 
 Example HTTP output configuration:
 
@@ -221,11 +221,25 @@ Example HTTP output configuration:
 output:
   default: http
   http:
-    user: admin
-    password: changeit
     url: "http://192.168.162.50:8080/db/coll"
+    user: admin             # Optional - for HTTP basic auth
+    password: changeit      # Optional - for HTTP basic auth
+    ssl_verify: false       # Optional - verify SSL certs (default: false)
+    headers:                # Optional - custom HTTP headers
+      X-Custom-Header: "value"
+      X-API-Key: "secret"
 ```
 
+### Configuration Options
+
+| Option       | Required | Description                                             |
+|--------------|----------|---------------------------------------------------------|
+| `url`        | Yes      | Full HTTP/HTTPS URL to POST the config to               |
+| `user`       | No       | Username for HTTP Basic Authentication                  |
+| `password`   | No       | Password for HTTP Basic Authentication                  |
+| `ssl_verify` | No       | When `true`, verify SSL certificates (default: `false`) |
+| `headers`    | No       | Hash of custom HTTP headers to include in the request   |
+
 ## Output types
 
 If you prefer to have different outputs in different files and/or directories, you can easily do this by modifying the corresponding model. To change the behaviour for IOS, you would edit `lib/oxidized/model/ios.rb` (run `gem contents oxidized` to find out the full file path).