oxidized 0.29.1 → 0.37.0

data/docs/Docker.md

@@ -0,0 +1,246 @@
+# Running oxidized within an OCI container (docker, podman...)
+
+## Docker image
+The official Docker image is automatically built and pushed to hub.docker.com
+as [oxidized/oxidized](https://hub.docker.com/r/oxidized/oxidized/) with a
+[GitHub CI](/.github/workflows/publishdocker.yml).
+
+There are three different types of tags:
+- Each commit to the master branch will be published with the tag
+  `master-(git sha oid)`
+- Each release will be published with the full version as a tag (e.g. `0.35.0`)
+- Each release will also be published with the major version as a tag (e.g. `0`)
+- Latest is the latest release, either from a commit or a release tag
+
+Currently, Docker Hub automatically builds the master branch for linux/amd64 and
+linux/arm64 platforms as
+[oxidized/oxidized](https://hub.docker.com/r/oxidized/oxidized/). You can make
+use of this container or build your own.
+
+## Choose a container running environment
+There are many options to run containers. Two main options are
+[docker](https://www.docker.com/) and [podman](https://podman.io/). A main
+difference is that docker requires root rights to run, and podman can be run
+by a local user. Both work with oxidized, so the choice is up to you.
+
+Oxidized has also been reported to work with
+[Portainer](https://www.portainer.io/).
+
+## File rights in the container userspace and host userspace
+As oxidized runs under the user "oxidized" (UID: 30000) in the container
+userspace, docker and podman will map this UID in the shared volumes, producing
+weird UIDs in the host userspace.
+
+### docker
+When docker runs the container as root, the mapping to the UIDs in the host
+userspace will be the same as in the container, so the files produced by the
+oxidized user in the container will have UID 30000 in the host.
+
+If you map a volume between the host and the container and need it to be
+accessed by the oxidized user, you need to fix the UIDs:
+```
+sudo chown 30000:30000 ~/oxidized-config
+```
+
+### podman
+When podman is run as a user, the mapping of UIDs between the container and the
+linux host will depend on your UID on the host.
+
+If you map a volume between the host and the container and need it to be
+accessed by the oxidized user, you need to fix the UIDs:
+
+```
+podman unshare chown 30000:30000 ~/oxidized-config
+```
+
+If you need to access the files from the linux host, you can do this by
+prefixing `podman unshare` to your shell commands.
+
+## Build you own container image
+To build your own container image, clone the git repository:
+
+```shell
+git clone https://github.com/ytti/oxidized
+```
+
+Then, build the container locally:
+
+```shell
+sudo docker build -q -t oxidized/oxidized:latest oxidized/
+```
+
+- `-q` stands for quiet; remove it if you want to see the build process.
+- `-t oxidized/oxidized:latest` tags the image as `oxidized/oxidized:latest`
+
+You can also build with podman:
+```
+podman build -t oxidized:latest oxidized/
+```
+
+Within the oxidized repository, using `rake build_container` will automatically
+build the container (with podman or docker), name it `localhost/oxidized` and
+give it the tags `latest` and `<branchname>-<sha-tag>`, for example
+`localhost/oxidized:master-65baab9`.
+
+## Set up an environment for the container
+Once you've built the container (or chosen to make use of the automatically
+built container in Docker Hub, which will be downloaded for you by docker on the
+first `run` command had you not built it), you need to set up an environment
+for the container.
+
+First, you need a configuration directory in the host system that you can map
+in the container. You can choose any directory you want, we'll take
+`~/oxidized-config` in our example. Don't forget to adjust the permissions as
+explained above.
+
+If you already have a configuration for oxidized (`config`), you can skip this
+step. Just save it under `~/oxidized-config` and run the container (see below).
+
+If you don't have a configuration, you can make oxidized produce one for you, so
+that you just have to adapt it to your needs.
+
+```shell
+sudo docker run --rm -v ~/oxidized-config:/home/oxidized/.config/oxidized docker.io/oxidized/oxidized:latest su - oxidized -c oxidized
+```
+```shell
+podman run --rm -v ~/oxidized-config:/home/oxidized/.config/oxidized docker.io/oxidized/oxidized:latest su - oxidized -c oxidized
+```
+
+- `--rm` tells docker to automatically remove the container when he exits
+- `-v ~/oxidized-config:/home/oxidized/.config/oxidized` maps your local
+  `~/oxidized-config` into `/home/oxidized/.config/oxidized`in the container
+  environment.
+- `su - oxidized -c oxidized` runs oxidized under the user oxidized, so that it
+  can produce a configuration under `/home/oxidized/.config/oxidized`
+
+
+This will return `edit /home/oxidized/.config/oxidized/config`, which is the
+path in the container context. Now you can edit `~/oxidized-config/config` to
+fit your needs.
+
+You can reiterate this process a few times, until oxidized is happy with the
+config, an then you're finished with setting up the environment.
+
+
+You also need to create the `router.db` file under
+`~/oxidized-config/config/router.db` (see
+[CSV Source](/docs/Sources.md#source-csv) for further info) or configure another
+source to suit your needs. Don't forget to set the file permissions (owner)
+properly!
+
+
+
+## Run the container
+Now you can run the container without specifying an entry point. It will
+automatically start oxidized and every other process needed.
+```shell
+sudo docker run --rm -v ~/oxidized-config:/home/oxidized/.config/oxidized -p 8888:8888/tcp docker.io/oxidized/oxidized:latest
+```
+```shell
+podman run --rm -v ~/oxidized-config:/home/oxidized/.config/oxidized -p 8888:8888/tcp docker.io/oxidized/oxidized:latest
+```
+
+`-p 8888:8888/tcp` maps the TCP port 8888 in the container with the port
+8888 on the host, so that you can access the RESTful API and Web Interface
+from the host.
+If the RESTful API and Web Interface should be enabled, edit the
+configuration (in our example `~/oxidized-config/config`) and modify
+`rest: 127.0.0.1:8888` to `rest: 0.0.0.0:8888`. This will bind port 8888 to all
+interfaces, and expose the port so that it can be accessed externally.
+[(Issue #445)](https://github.com/ytti/oxidized/issues/445)
+
+
+## Run with with docker-compose / podman-compose
+Alternatively, you can use docker-compose or podman-compose to run the
+container:
+
+```yaml
+# docker-compose.yml
+# docker-compose file example for oxidized that will start along with docker daemon
+---
+version: "3"
+services:
+  oxidized:
+    restart: always
+    image: docker.io/oxidized/oxidized:latest
+    ports:
+      - 8888:8888/tcp
+    environment:
+      # Reload hosts list once per day
+      CONFIG_RELOAD_INTERVAL: 86400
+    volumes:
+       - ~/oxidized-config/config:/home/oxidized/.config/oxidized/
+```
+
+To start the pod, use `docker-compose up` or `podman-compose down`.
+
+## Special configurations of the official container
+### Reload the configuration
+If you want to have the config automatically reloaded (e.g. when using a http
+source that changes), you need to set the environment variable
+CONFIG_RELOAD_INTERVAL. This can be done in `docker-compose.yml` (see above) or
+on the command line:
+
+```shell
+sudo docker run -v ~/oxidized-config:/home/oxidized/.config/oxidized -p 8888:8888/tcp -e CONFIG_RELOAD_INTERVAL=3600 docker.io/oxidized/oxidized:latest
+```
+### Use an internal CA
+If you need to use an internal CA (e.g. to connect to an private github instance):
+
+```shell
+docker run -v /etc/oxidized:/home/oxidized/.config/oxidized -v /path/to/MY-CA.crt:/usr/local/share/ca-certificates/MY-CA.crt -p 8888:8888/tcp -e UPDATE_CA_CERTIFICATES=true -t oxidized/oxidized:latest
+```
+
+### Pass the ssh passphrase for a remote git
+If you don't want to authenticate with user & password but with a ssh-key, you
+can set the ssh passphrase with the environment variable
+`OXIDIZED_SSH_PASSPHRASE`
+
+## Tipps & tricks
+### podman & Debian Bookworm
+To install podman in Debian Bookwork, you need following packages:
+```shell
+sudo apt install podman containers-storage podman-compose
+```
+
+Ensure Podman is using the overlay driver for image storage.
+Without this driver, Podman may save every container layer separately rather
+than only the changes, which can quickly consume disk space.
+
+This issue can occur if podman was run before installing the
+`container-storage`  package.
+
+```shell
+podman info | grep graphDriverName
+```
+
+You should get this reply
+```shell
+  graphDriverName: overlay
+```
+
+If not, a quick way to solve it is to delete `~/.local/share/containers/`.
+Beware - this will delete **all** your containers!
+
+### Store the ssh keys a remote git repository
+When you use the githubrepo hook to upload your configs to a remote git
+repository, you have to store your ssh-key and the public keys of the remote
+server. Create a directory `~/oxidized-ssh` and map it to `/home/oxidized/.ssh`.
+
+
+To generate an ssh-key, run:
+```shell
+ssh-keygen -q -t ed25519 -C "Oxidized Push Key@`hostname`" -N "YOURPASSPHRASE" -m PEM -f ~/oxidized-ssh/oxidized-key
+```
+
+You also need to store the public keys of the remote git server in known_hosts.
+If you don't store the keys, oxidized will refuse to push to the remote Git with
+the error
+`#<Rugged::SshError: invalid or unknown remote ssh hostkey>`, see Issue #2753.
+
+```shell
+ssh-keyscan git-server.example.com > ~/oxidized-ssh/known_hosts
+```
+
+Don't forget to set the permission (owner) of the files for the user oxidized
+inside the container, or this will not work!

data/docs/Hooks.md

@@ -2,6 +2,15 @@
 
 You can define an arbitrary number of hooks that subscribe to different events. The hook system is modular and different kind of hook types can be enabled.
 
+1. [Events](#events)
+2. Hook types
+ * [exec](#hook-type-exec)
+ * [githubrepo](#hook-type-githubrepo)
+ * [awssns](#hook-type-awssns)
+ * [slackdiff](#hook-type-slackdiff)
+ * [ciscosparkdiff](#ciscosparkdiff)
+ * [xmppdiff](#hook-type-xmppdiff)
+
 ## Configuration
 
 Following configuration keys need to be defined for all hooks:
@@ -46,7 +55,7 @@ Exec hook recognizes the following configuration keys:
 * `async`: Execute the command in an asynchronous fashion. The main thread by default will wait for the hook command execution to complete. Set this to `true` for long running commands so node configuration pulls are not blocked. Default: `false`
 * `cmd`: command to run.
 
-### exec hook configuration example
+### Exec Hook configuration example
 
 ```yaml
 hooks:
@@ -62,52 +71,140 @@ hooks:
     timeout: 120
 ```
 
+### Exec Hook configuration example to send mail
+
+To send mail you need the package `msmtp` (It is pre-installed with the docker container) 
+
+You then need to update the `~/.msmtprc` file to contain your SMTP credentials like this:
+
+*Note: In the docker container the file is in /home/oxidized/.config/oxidized/.msmtprc so you can create the file if it doesn't exist in your oxidized config folder.*
+
+```cfg
+# Default settings
+defaults
+auth    on
+tls    on
+# Outlook SMTP
+account    mainaccount
+host       smtp.office365.com
+port       587
+from       user@domain.com
+user       user@domain.com
+password   edit-password
+
+account default : mainaccount
+```
+
+For non docker users this file should have the 600 permission, using: `chmod 600 .msmtprc` and the owner of the file should be the owner of oxidized `chown oxidized:oxidized .msmtprc`
+
+Then, you can configure Hooks to send mail like this:
+
+```yaml
+hooks:
+  send_mail_hook:
+    type: exec
+    events: [node_fail]
+    cmd: '/usr/bin/echo -e "Subject: [Oxidized] Error on node $OX_NODE_NAME \n\nThe device $OX_NODE_NAME has not been backed-up, reason: \n\n$OX_EVENT: $OX_ERR_REASON" | msmtp destination@domain.com'
+```
+
 ## Hook type: githubrepo
 
-Note: You must not use the same name as any local repo configured under output. Make sure your 'git' output has a unique name that does not match your remote_repo.
+The `githubrepo` hook executes a `git push` to a configured `remote_repo` when
+the specified event is triggered.
+
+### Configuration keys
+
+| Key           | Description |
+|---------------|-------------|
+| `remote_repo` | The remote repository to push to. Use a URL string (no groups) or a group dictionary (see [Using groups](#using-groups)). |
+| `username`    | Username for authentication. Defaults to the user part of the `remote_repo` URI, falling back to `git`. |
+| `password`    | Password for username/password authentication. |
+| `privatekey`  | Path to the private key file. Must be in legacy PEM format (see note below). |
+| `publickey`   | Path to the public key file (optional — inferred from `privatekey` + `.pub` if omitted). |
+
+Notes:
+- `remote_repo` must not match the name of any local `git` output repo configured under `output`. Use unique names for each.
+- If using SSH key authentication with a passphrase-protected private key, provide the passphrase with the `OXIDIZED_SSH_PASSPHRASE` environment variable.
+- The `privatekey` must be in the legacy PEM format (`BEGIN RSA PRIVATE KEY`), not the newer OpenSSH format (`BEGIN OPENSSH PRIVATE KEY`). See [#1877](https://github.com/ytti/oxidized/issues/1877) and [#2324](https://github.com/ytti/oxidized/issues/2324).
+- To convert an existing key to PEM format, run:
+  ```shell
+  ssh-keygen -p -m PEM -f $MY_KEY_HERE
+  ```
+
+### Authentication methods
 
-The `githubrepo` hook executes a `git push` to a configured `remote_repo` when the specified event is triggered.
+Choose one of the following methods:
 
-Several authentication methods are supported:
+| Method                        | Required keys |
+|-------------------------------|---------------|
+| Username + password           | `username` (optional), `password` |
+| SSH key pair                  | `privatekey`, `publickey` (optional - assumed to be at `privatekey` + `.pub`) |
+| SSH agent                     | no credentials needed |
 
-* Provide a `password` for username + password authentication
-* Provide both a `publickey` and a `privatekey` for ssh key-based authentication
-* Provide only a `privatekey` (public key filename is assumed to be `privatekey` + "`.pub`"
-* Don't provide any credentials for ssh-agent authentication
+### Configuration examples
+
+**Username and password:**
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    username: user
+    password: pass
+```
+
+**SSH key pair:**
+
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+    publickey: /root/.ssh/id_rsa.pub
+    privatekey: /root/.ssh/id_rsa
+```
 
-The username will be set to the relevant part of the `remote_repo` URI, with a fallback to `git`. It is also possible to provide one by setting the `username` configuration key.
 
-For ssh key-based authentication, it is possible to set the environment variable `OXIDIZED_SSH_PASSPHRASE` to a passphrase if the private key requires it.
+**SSH agent:**
 
-`githubrepo` hook recognizes the following configuration keys:
+```yaml
+hooks:
+  push_to_remote:
+    type: githubrepo
+    events: [post_store]
+    remote_repo: git@git.intranet:oxidized/test.git
+```
 
-* `remote_repo`: the remote repository to be pushed to.
-* `username`: username for repository auth.
-* `password`: password for repository auth.
-* `publickey`: public key file path for repository auth. (optional)
-* `privatekey`: private key file path for repository auth.
-  * NOTE: this key needs to be in the legacy PEM format, not the newer OpenSSL format [#1877](https://github.com/ytti/oxidized/issues/1877), [#2324](https://github.com/ytti/oxidized/issues/2324)
-    * To convert a key beginning with `BEGIN OPENSSH PRIVATE KEY` to the legacy PEM format, run this command:
-      `ssh-keygen -p -m PEM -f $MY_KEY_HERE`
+### Using groups
 
-When using groups, `remote_repo` must be a dictionary of groups that the hook should apply to. If a group is missing from the dictionary, no action will be taken.
+When using groups and `single_repo` is set to `true` (default) in the
+configuration section output/git, `remote_repo` must be a dictionary mapping
+group names to remote repositories. Groups not listed in the dictionary are
+silently skipped.
 
-The dictionary entry can either be a url alone:
+Each entry can be either a plain URL string:
 
 ```yaml
 hooks:
   push_to_remote:
+    type: githubrepo
+    events: [post_store]
     remote_repo:
       routers: git@git.intranet:oxidized/routers.git
       switches: git@git.intranet:oxidized/switches.git
       firewalls: git@git.intranet:oxidized/firewalls.git
 ```
 
-... or it can be a dictionary with `url` and `privatekey` specified:
+...or a dictionary with `url` and `privatekey` to use a per-group SSH key:
 
 ```yaml
 hooks:
   push_to_remote:
+    type: githubrepo
+    events: [post_store]
     remote_repo:
       routers:
         url: git@git.intranet:oxidized/routers.git
@@ -120,33 +217,31 @@ hooks:
         privatekey: /root/.ssh/id_rsa_firewalls
 ```
 
-Both forms can be mixed and matched.
+Both forms can be mixed within the same configuration.
 
-### githubrepo hook configuration example
+### Custom branch name
 
-Authenticate with a username and a password without groups in use:
+The `githubrepo` hook uses the branch name from the
+[git output](Outputs.md#output-git) as the remote branch name. When the
+repository is first created, Oxidized uses the default branch name from
+`git config --global init.defaultBranch`. The default is `master`.
 
-```yaml
-hooks:
-  push_to_remote:
-    type: githubrepo
-    events: [post_store]
-    remote_repo: git@git.intranet:oxidized/test.git
-    username: user
-    password: pass
-```
+You can manually rename the branch after Oxidized has already created the
+repository. Be aware that you may break things, so make backups.
 
-Authenticate with the username `git` and an ssh key:
+To rename the branch after Oxidized has already created the repository:
 
-```yaml
-hooks:
-  push_to_remote:
-    type: githubrepo
-    events: [post_store]
-    remote_repo: git@git.intranet:oxidized/test.git
-    publickey: /root/.ssh/id_rsa.pub
-    privatekey: /root/.ssh/id_rsa
-```
+1. Stop oxidized.
+2. Back up your oxidized git repository.
+3. Change to your oxidized git repository directory.
+4. Inspect the current branches: `git branch -avv`
+5. Rename the local branch: `git branch -m <NewName>`
+6. Remove the stale remote-tracking reference: `git branch -r -d origin/<OldName>`
+7. Verify the result: `git branch -avv`
+8. Restart oxidized.
+
+Oxidized will push to a new remote branch. When everything is fine, you can
+remove the old branch from the remote repository.
 
 ## Hook type: awssns
 
@@ -179,7 +274,7 @@ Your AWS credentials should be stored in `~/.aws/credentials`.
 
 ## Hook type: slackdiff
 
-The `slackdiff` hook posts colorized config diffs to a [Slack](https://www.slack.com) channel of your choice. It only triggers for `post_store` events.
+The `slackdiff` hook posts colorized config diffs to a [Slack](https://www.slack.com) channel of your choice. It only triggers for `post_store` events. The used output must be capable of generating a diff. E.g. file output is not usable while git/gitcrypt will work.
 
 You will need to manually install the `slack-ruby-client` gem on your system:
 
@@ -189,16 +284,18 @@ gem install slack-ruby-client
 
 ### slackdiff hook configuration example
 
+> Please note that the channel needs to be your Slack channel ID.
+
 ```yaml
 hooks:
   slack:
     type: slackdiff
     events: [post_store]
     token: SLACK_BOT_TOKEN
-    channel: "#network-changes"
+    channel: "CHANNEL_ID"
 ```
 
-The token parameter is a Slack API token that can be generated following [this tutorial](https://api.slack.com/tutorials/tracks/getting-a-token).  Until Slack stops supporting them, legacy tokens can also be used.
+The token parameter is a Slack API token that can be generated following [this tutorial](https://api.slack.com/tutorials/tracks/getting-a-token).  Until Slack stops supporting them, legacy tokens can also be used. If the token has channels:join permission, the bot will attempt to automatically join the configured channel if necessary.
 
 Optionally you can disable snippets and post a formatted message, for instance linking to a commit in a git repo. Named parameters `%{node}`, `%{group}`, `%{model}` and `%{commitref}` are available.
 
@@ -208,13 +305,11 @@ hooks:
     type: slackdiff
     events: [post_store]
     token: SLACK_BOT_TOKEN
-    channel: "#network-changes"
+    channel: "CHANNEL_ID"
     diff: false
     message: "%{node} %{group} %{model} updated https://git.intranet/network-changes/commit/%{commitref}"
 ```
 
-Note the channel name must be in quotes.
-
 A proxy can optionally be specified if needed to reach the Slack API endpoint.
 
 ```yaml
@@ -223,7 +318,7 @@ hooks:
     type: slackdiff
     events: [post_store]
     token: SLACK_BOT_TOKEN
-    channel: "#network-changes"
+    channel: "#CHANNEL_ID"
     proxy: http://myproxy:8080
 ```