netbox-plugin-prometheus-sd 2.0.0__tar.gz → 2.2.0__tar.gz

netbox_plugin_prometheus_sd-2.2.0/PKG-INFO

@@ -0,0 +1,297 @@
+Metadata-Version: 2.4
+Name: netbox-plugin-prometheus-sd
+Version: 2.2.0
+Summary: A Netbox plugin to provide Netbox entires to Prometheus HTTP service discovery
+License: MIT
+License-File: LICENSE
+Author: Felix Peters
+Author-email: felix.peters@breuninger.de
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Description-Content-Type: text/markdown
+
+# netbox-plugin-prometheus-sd
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![CI](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/workflows/CI/badge.svg?event=push)](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/actions?query=workflow%3ACI)
+[![PyPI](https://img.shields.io/pypi/v/netbox-plugin-prometheus-sd)](https://pypi.org/project/netbox-plugin-prometheus-sd/)
+
+[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/flxpeters)
+
+Provide Prometheus `http_sd` compatible API Endpoint with data from Netbox.
+
+HTTP SD is a feature since Prometheus 2.28.0 that allows hosts to be found via a URL instead of just files.
+This plugin implements API endpoints in Netbox to make devices, services, IPs and virtual machines available to Prometheus.
+
+## Compatibility
+
+We aim to support the latest major versions of Netbox.
+
+| Plugin | Netbox |
+|---|---|
+| `2.x` | `>= 4.0` |
+| `1.x` | `3.x` (no longer maintained) |
+
+Check the `.github/workflows/ci.yml` pipeline for the current tested builds.
+Other versions may work, but we do not test them explicitly. All relevant target versions are tested in CI.
+
+Plugin `2.0` also fixed a set of N+1 queries that made the endpoints very slow on
+larger installations ([#265](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/issues/265)).
+If service discovery is putting noticeable load on your Netbox database, upgrading
+is worthwhile.
+
+## Installation
+
+The plugin is available as a Python package in pypi and can be installed with pip
+
+```bash
+pip install netbox-plugin-prometheus-sd
+```
+
+Enable the plugin in /opt/netbox/netbox/netbox/configuration.py:
+
+```python
+    PLUGINS = ['netbox_prometheus_sd']
+```
+
+The plugin has not further plugin configuration.
+
+## Usage
+
+The plugin only provides a new API endpoint on the Netbox API. There is no further action required after installation.
+
+### API
+
+The plugin reuses Netbox API view sets with new serializers for Prometheus.
+This means that all filters that can be used on the Netbox API can also be used to filter Prometheus targets.
+Paging is disabled because Prometheus does not support paged results.
+
+The plugin also reuses the Netbox authentication and permission model.
+Depending on the Netbox configuration, a token with valid object permissions must be passed to Netbox.
+
+```
+GET        /api/plugins/prometheus-sd/devices/              Get a list of devices in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/virtual-machines/     Get a list of vms in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/services/             Get a list of services in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/ip-addresses/         Get a list of ip in a prometheus compatible format
+```
+
+#### Extended services filters
+
+Apart from standard Netbox filters, services endpoint also supports `tenant=<slug>` or `tenant_id=<id>`, and
+`site=<slug>` or `site_id=<id>` parameters. The lookup is only executed against the `tenant`/`site` attribute of the
+object associated with the service.
+
+#### Filtering and response size
+
+Because paging is disabled, one request serializes every object the token is
+allowed to see. Filtering with the standard Netbox query parameters is the
+supported way to keep responses small, and it is usually what you want anyway —
+Prometheus should not be told about decommissioned hardware:
+
+```
+/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring&site=dc1
+```
+
+### Labels
+
+Every target carries `__meta_netbox_*` labels describing the Netbox object.
+**Prometheus discards labels starting with `__` after service discovery**, so
+they have to be copied into real labels with `relabel_configs` (see below) if you
+want them on your metrics.
+
+A label is only present when the underlying field is populated — a device with no
+tenant has no `__meta_netbox_tenant`.
+
+| Label | Devices | VMs | Services | IPs |
+|---|:-:|:-:|:-:|:-:|
+| `id`, `name`, `status`, `model` | ✓ | ✓ | id/name only | ✓ (no name) |
+| `primary_ip`, `primary_ip4`, `primary_ip6` | ✓ | ✓ | from parent | |
+| `oob_ip` | ✓ | | from parent | |
+| `ip` | | | | ✓ |
+| `role`, `role_slug` | ✓ | ✓ | | `role` only |
+| `platform`, `platform_slug` | ✓ | ✓ | | |
+| `device_type`, `device_type_slug` | ✓ | | | |
+| `site`, `site_slug` | ✓ | ✓ | from parent | |
+| `scope`, `scope_slug` | | ✓ ¹ | | |
+| `location`, `location_slug` | ✓ | | | |
+| `rack`, `rack_u_position` | ✓ | | | |
+| `cluster`, `cluster_group`, `cluster_type` | ✓ ² | ✓ | | |
+| `tenant`, `tenant_slug` | ✓ | ✓ | from parent | ✓ |
+| `tenant_group`, `tenant_group_slug` | ✓ | ✓ | from parent | ✓ |
+| `tags`, `tag_slugs` | ✓ | ✓ | ✓ | ✓ |
+| `services` | ✓ | ✓ | | |
+| `contact_<priority>_{name,email,comments,role}` | ✓ | ✓ | from parent | |
+| `custom_field_<name>` | ✓ | ✓ | ✓ | ✓ |
+| `description` | ✓ | | | |
+| `parent`, `display`, `ports`, `ipaddresses` | | | ✓ | |
+
+¹ Netbox 4.2 replaced the cluster's site with a generic `scope`. On 4.2+ a VM
+emits `scope`/`scope_slug` for the cluster, and `site`/`site_slug` for its own
+site; below 4.2 the cluster's site is reported as `site`.
+² Only when the device is assigned to a cluster.
+
+Config context can additionally set `__metrics_path__` and `__scheme__`, which
+Prometheus consumes directly (see [Config context](#config-context)).
+
+### Relabeling
+
+The `__meta_netbox_*` labels are dropped unless you map them. A typical device job
+scraping node_exporter on the primary IP:
+
+```yaml
+scrape_configs:
+  - job_name: netbox-devices
+    http_sd_configs:
+      - url: http://netbox:8080/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring
+        refresh_interval: 60s
+        authorization:
+          type: Token
+          credentials: "<your-netbox-api-token>"
+
+    relabel_configs:
+      # Skip anything without a primary IPv4, otherwise the address below is empty.
+      - source_labels: [__meta_netbox_primary_ip4]
+        regex: ^$
+        action: drop
+
+      # Scrape the primary IP instead of the device name, which may not resolve.
+      - source_labels: [__meta_netbox_primary_ip4]
+        target_label: __address__
+        replacement: "$1:9100"
+
+      # Keep the Netbox name as the instance label rather than the IP.
+      - source_labels: [__meta_netbox_name]
+        target_label: instance
+
+      # Promote the dimensions worth alerting and grouping on.
+      - source_labels: [__meta_netbox_site_slug]
+        target_label: site
+      - source_labels: [__meta_netbox_role_slug]
+        target_label: role
+      - source_labels: [__meta_netbox_tenant_slug]
+        target_label: tenant
+```
+
+Two things worth knowing:
+
+- `tags` and `tag_slugs` are comma-joined, so match them with `.*,?value,?.*`
+  rather than `=`.
+- Devices and VMs use the same label names, so one set of `relabel_configs` can
+  be reused across both jobs.
+
+### Config context
+
+The plugin can also discover extra config to inject in the HTTP SD JSON from the config context of the devices/virtual machines.
+If you have a `prometheus-plugin-prometheus-sd` entry in your config context with the following schema it will be automatically picked up:
+
+```yaml
+prometheus-plugin-prometheus-sd:
+  - metrics_path: /not/metrics
+    port: 4242
+    scheme: https
+  - port: 4243
+```
+
+This allow you to configure those values directly into netbox instead of doing that inside the Prometheus
+config and filtering each scenario by a specific tag for instance.
+
+If there is only one entry you can also use this form:
+
+```yaml
+prometheus-plugin-prometheus-sd:
+  metrics_path: /not/metrics
+  port: 4242
+  scheme: https
+```
+
+### Example
+
+[`example/prometheus.yml`](example/prometheus.yml) is a complete Prometheus
+configuration covering all four endpoints, with `relabel_configs` mapping the
+`__meta_netbox_*` labels onto real ones.
+
+It is not a snippet that happens to be in the repository: the test suite runs
+Prometheus against this exact file and asserts that the expected targets and
+labels are discovered, so it cannot quietly stop working.
+
+To see it running, with a Netbox that has demo data already loaded:
+
+```bash
+poetry run invoke build-dev
+```
+
+- Netbox: <http://localhost:8000> (`admin` / `admin`)
+- Prometheus: <http://localhost:9090> — discovered targets are under
+  Status → Target health, and expanding one shows the raw `__meta_netbox_*`
+  labels before relabeling.
+
+The demo data is the unit-test fixtures, so it also exercises config context
+(the VMs get two targets each, on different ports), services, contacts and tags.
+
+## Development
+
+We use [Poetry](https://python-poetry.org/) for dependency management and [invoke](https://www.pyinvoke.org/) as task runner.
+To test the plugin in an isolated environment, we use [testcontainers](https://testcontainers.com/?language=python)
+which creates "throwaway, lightweight" Netbox Docker containers.
+
+Install the dependencies with `poetry install`, then run the tasks through
+`poetry run` (this works on every Poetry version, whereas `poetry shell` was moved
+into a separate plugin in Poetry 2.0):
+
+```bash
+# Unit tests plus the Prometheus end-to-end check
+poetry run invoke test
+
+# Test against a specific Netbox release (default: latest)
+NETBOX_VER=v4.6.5 poetry run invoke test
+
+# Either half on its own
+poetry run invoke unittest
+poetry run invoke test-prometheus
+```
+
+The Netbox image is built from the `Dockerfile` in the repository root.
+
+Testing has two layers:
+
+- **Unit tests** — plain Django tests under `netbox_prometheus_sd/tests/`,
+  executed inside the Netbox container. These cover the serializers and the
+  label output.
+- **Prometheus end-to-end** — Netbox serving real HTTP with seeded data, and a
+  real Prometheus configured from `example/prometheus.yml`. It asserts that the
+  expected jobs discover targets and that relabeling produced the expected
+  labels.
+
+The second layer exists because "valid JSON with targets and labels" is not the
+same as "Prometheus accepts this as an `http_sd` source". A response that
+Prometheus rejects would pass every unit test in this repository. It also keeps
+the documented example honest, since it is the file under test.
+
+Features should be covered by a test, but sometimes it is easier to develop
+against a running system:
+
+```bash
+# Netbox + Prometheus with demo data, left running until Ctrl+C
+poetry run invoke build-dev
+```
+
+Netbox is on <http://localhost:8000> (`admin` / `admin`) and Prometheus on
+<http://localhost:9090>, already scraping it.
+
+API endpoints for testing can be found at http://localhost:8000/api/plugins/prometheus-sd/
+
+## Conventional Commits
+
+This repository follows the Conventional Commits specification for versioning and changelog generation.
+Conventional Commits provide a standardized way of writing commit messages to convey semantic meaning
+about the changes made. Each commit message follows a defined format that includes a type,
+an optional scope, and a message. The types typically include features, fixes, documentation, and more.
+By adhering to this convention, we ensure clear and automated versioning, release notes, and changelog generation.
+

netbox_plugin_prometheus_sd-2.2.0/README.md

@@ -0,0 +1,278 @@
+# netbox-plugin-prometheus-sd
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![CI](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/workflows/CI/badge.svg?event=push)](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/actions?query=workflow%3ACI)
+[![PyPI](https://img.shields.io/pypi/v/netbox-plugin-prometheus-sd)](https://pypi.org/project/netbox-plugin-prometheus-sd/)
+
+[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/flxpeters)
+
+Provide Prometheus `http_sd` compatible API Endpoint with data from Netbox.
+
+HTTP SD is a feature since Prometheus 2.28.0 that allows hosts to be found via a URL instead of just files.
+This plugin implements API endpoints in Netbox to make devices, services, IPs and virtual machines available to Prometheus.
+
+## Compatibility
+
+We aim to support the latest major versions of Netbox.
+
+| Plugin | Netbox |
+|---|---|
+| `2.x` | `>= 4.0` |
+| `1.x` | `3.x` (no longer maintained) |
+
+Check the `.github/workflows/ci.yml` pipeline for the current tested builds.
+Other versions may work, but we do not test them explicitly. All relevant target versions are tested in CI.
+
+Plugin `2.0` also fixed a set of N+1 queries that made the endpoints very slow on
+larger installations ([#265](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/issues/265)).
+If service discovery is putting noticeable load on your Netbox database, upgrading
+is worthwhile.
+
+## Installation
+
+The plugin is available as a Python package in pypi and can be installed with pip
+
+```bash
+pip install netbox-plugin-prometheus-sd
+```
+
+Enable the plugin in /opt/netbox/netbox/netbox/configuration.py:
+
+```python
+    PLUGINS = ['netbox_prometheus_sd']
+```
+
+The plugin has not further plugin configuration.
+
+## Usage
+
+The plugin only provides a new API endpoint on the Netbox API. There is no further action required after installation.
+
+### API
+
+The plugin reuses Netbox API view sets with new serializers for Prometheus.
+This means that all filters that can be used on the Netbox API can also be used to filter Prometheus targets.
+Paging is disabled because Prometheus does not support paged results.
+
+The plugin also reuses the Netbox authentication and permission model.
+Depending on the Netbox configuration, a token with valid object permissions must be passed to Netbox.
+
+```
+GET        /api/plugins/prometheus-sd/devices/              Get a list of devices in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/virtual-machines/     Get a list of vms in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/services/             Get a list of services in a prometheus compatible format
+GET        /api/plugins/prometheus-sd/ip-addresses/         Get a list of ip in a prometheus compatible format
+```
+
+#### Extended services filters
+
+Apart from standard Netbox filters, services endpoint also supports `tenant=<slug>` or `tenant_id=<id>`, and
+`site=<slug>` or `site_id=<id>` parameters. The lookup is only executed against the `tenant`/`site` attribute of the
+object associated with the service.
+
+#### Filtering and response size
+
+Because paging is disabled, one request serializes every object the token is
+allowed to see. Filtering with the standard Netbox query parameters is the
+supported way to keep responses small, and it is usually what you want anyway —
+Prometheus should not be told about decommissioned hardware:
+
+```
+/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring&site=dc1
+```
+
+### Labels
+
+Every target carries `__meta_netbox_*` labels describing the Netbox object.
+**Prometheus discards labels starting with `__` after service discovery**, so
+they have to be copied into real labels with `relabel_configs` (see below) if you
+want them on your metrics.
+
+A label is only present when the underlying field is populated — a device with no
+tenant has no `__meta_netbox_tenant`.
+
+| Label | Devices | VMs | Services | IPs |
+|---|:-:|:-:|:-:|:-:|
+| `id`, `name`, `status`, `model` | ✓ | ✓ | id/name only | ✓ (no name) |
+| `primary_ip`, `primary_ip4`, `primary_ip6` | ✓ | ✓ | from parent | |
+| `oob_ip` | ✓ | | from parent | |
+| `ip` | | | | ✓ |
+| `role`, `role_slug` | ✓ | ✓ | | `role` only |
+| `platform`, `platform_slug` | ✓ | ✓ | | |
+| `device_type`, `device_type_slug` | ✓ | | | |
+| `site`, `site_slug` | ✓ | ✓ | from parent | |
+| `scope`, `scope_slug` | | ✓ ¹ | | |
+| `location`, `location_slug` | ✓ | | | |
+| `rack`, `rack_u_position` | ✓ | | | |
+| `cluster`, `cluster_group`, `cluster_type` | ✓ ² | ✓ | | |
+| `tenant`, `tenant_slug` | ✓ | ✓ | from parent | ✓ |
+| `tenant_group`, `tenant_group_slug` | ✓ | ✓ | from parent | ✓ |
+| `tags`, `tag_slugs` | ✓ | ✓ | ✓ | ✓ |
+| `services` | ✓ | ✓ | | |
+| `contact_<priority>_{name,email,comments,role}` | ✓ | ✓ | from parent | |
+| `custom_field_<name>` | ✓ | ✓ | ✓ | ✓ |
+| `description` | ✓ | | | |
+| `parent`, `display`, `ports`, `ipaddresses` | | | ✓ | |
+
+¹ Netbox 4.2 replaced the cluster's site with a generic `scope`. On 4.2+ a VM
+emits `scope`/`scope_slug` for the cluster, and `site`/`site_slug` for its own
+site; below 4.2 the cluster's site is reported as `site`.
+² Only when the device is assigned to a cluster.
+
+Config context can additionally set `__metrics_path__` and `__scheme__`, which
+Prometheus consumes directly (see [Config context](#config-context)).
+
+### Relabeling
+
+The `__meta_netbox_*` labels are dropped unless you map them. A typical device job
+scraping node_exporter on the primary IP:
+
+```yaml
+scrape_configs:
+  - job_name: netbox-devices
+    http_sd_configs:
+      - url: http://netbox:8080/api/plugins/prometheus-sd/devices/?status=active&tag=monitoring
+        refresh_interval: 60s
+        authorization:
+          type: Token
+          credentials: "<your-netbox-api-token>"
+
+    relabel_configs:
+      # Skip anything without a primary IPv4, otherwise the address below is empty.
+      - source_labels: [__meta_netbox_primary_ip4]
+        regex: ^$
+        action: drop
+
+      # Scrape the primary IP instead of the device name, which may not resolve.
+      - source_labels: [__meta_netbox_primary_ip4]
+        target_label: __address__
+        replacement: "$1:9100"
+
+      # Keep the Netbox name as the instance label rather than the IP.
+      - source_labels: [__meta_netbox_name]
+        target_label: instance
+
+      # Promote the dimensions worth alerting and grouping on.
+      - source_labels: [__meta_netbox_site_slug]
+        target_label: site
+      - source_labels: [__meta_netbox_role_slug]
+        target_label: role
+      - source_labels: [__meta_netbox_tenant_slug]
+        target_label: tenant
+```
+
+Two things worth knowing:
+
+- `tags` and `tag_slugs` are comma-joined, so match them with `.*,?value,?.*`
+  rather than `=`.
+- Devices and VMs use the same label names, so one set of `relabel_configs` can
+  be reused across both jobs.
+
+### Config context
+
+The plugin can also discover extra config to inject in the HTTP SD JSON from the config context of the devices/virtual machines.
+If you have a `prometheus-plugin-prometheus-sd` entry in your config context with the following schema it will be automatically picked up:
+
+```yaml
+prometheus-plugin-prometheus-sd:
+  - metrics_path: /not/metrics
+    port: 4242
+    scheme: https
+  - port: 4243
+```
+
+This allow you to configure those values directly into netbox instead of doing that inside the Prometheus
+config and filtering each scenario by a specific tag for instance.
+
+If there is only one entry you can also use this form:
+
+```yaml
+prometheus-plugin-prometheus-sd:
+  metrics_path: /not/metrics
+  port: 4242
+  scheme: https
+```
+
+### Example
+
+[`example/prometheus.yml`](example/prometheus.yml) is a complete Prometheus
+configuration covering all four endpoints, with `relabel_configs` mapping the
+`__meta_netbox_*` labels onto real ones.
+
+It is not a snippet that happens to be in the repository: the test suite runs
+Prometheus against this exact file and asserts that the expected targets and
+labels are discovered, so it cannot quietly stop working.
+
+To see it running, with a Netbox that has demo data already loaded:
+
+```bash
+poetry run invoke build-dev
+```
+
+- Netbox: <http://localhost:8000> (`admin` / `admin`)
+- Prometheus: <http://localhost:9090> — discovered targets are under
+  Status → Target health, and expanding one shows the raw `__meta_netbox_*`
+  labels before relabeling.
+
+The demo data is the unit-test fixtures, so it also exercises config context
+(the VMs get two targets each, on different ports), services, contacts and tags.
+
+## Development
+
+We use [Poetry](https://python-poetry.org/) for dependency management and [invoke](https://www.pyinvoke.org/) as task runner.
+To test the plugin in an isolated environment, we use [testcontainers](https://testcontainers.com/?language=python)
+which creates "throwaway, lightweight" Netbox Docker containers.
+
+Install the dependencies with `poetry install`, then run the tasks through
+`poetry run` (this works on every Poetry version, whereas `poetry shell` was moved
+into a separate plugin in Poetry 2.0):
+
+```bash
+# Unit tests plus the Prometheus end-to-end check
+poetry run invoke test
+
+# Test against a specific Netbox release (default: latest)
+NETBOX_VER=v4.6.5 poetry run invoke test
+
+# Either half on its own
+poetry run invoke unittest
+poetry run invoke test-prometheus
+```
+
+The Netbox image is built from the `Dockerfile` in the repository root.
+
+Testing has two layers:
+
+- **Unit tests** — plain Django tests under `netbox_prometheus_sd/tests/`,
+  executed inside the Netbox container. These cover the serializers and the
+  label output.
+- **Prometheus end-to-end** — Netbox serving real HTTP with seeded data, and a
+  real Prometheus configured from `example/prometheus.yml`. It asserts that the
+  expected jobs discover targets and that relabeling produced the expected
+  labels.
+
+The second layer exists because "valid JSON with targets and labels" is not the
+same as "Prometheus accepts this as an `http_sd` source". A response that
+Prometheus rejects would pass every unit test in this repository. It also keeps
+the documented example honest, since it is the file under test.
+
+Features should be covered by a test, but sometimes it is easier to develop
+against a running system:
+
+```bash
+# Netbox + Prometheus with demo data, left running until Ctrl+C
+poetry run invoke build-dev
+```
+
+Netbox is on <http://localhost:8000> (`admin` / `admin`) and Prometheus on
+<http://localhost:9090>, already scraping it.
+
+API endpoints for testing can be found at http://localhost:8000/api/plugins/prometheus-sd/
+
+## Conventional Commits
+
+This repository follows the Conventional Commits specification for versioning and changelog generation.
+Conventional Commits provide a standardized way of writing commit messages to convey semantic meaning
+about the changes made. Each commit message follows a defined format that includes a type,
+an optional scope, and a message. The types typically include features, fixes, documentation, and more.
+By adhering to this convention, we ensure clear and automated versioning, release notes, and changelog generation.

{netbox_plugin_prometheus_sd-2.0.0 → netbox_plugin_prometheus_sd-2.2.0}/netbox_prometheus_sd/__init__.py

@@ -1,7 +1,7 @@
 from netbox.plugins import PluginConfig
 
 # Placeholder for semantic release
-__VERSION__ = "2.0.0"
+__VERSION__ = "2.2.0"
 
 
 class PrometheusSD(PluginConfig):

{netbox_plugin_prometheus_sd-2.0.0 → netbox_plugin_prometheus_sd-2.2.0}/netbox_prometheus_sd/filtersets.py

@@ -8,14 +8,19 @@ from utilities.filters import (
 )
 
 from ipam.filtersets import ServiceFilterSet as NetboxServiceFilterSet
+from ipam.models import Service
+
+# Netbox 4.7 replaced the Service.ports ArrayField with `port_mappings`, and
+# its filterset metaclass rejects a filter on a field that no longer exists.
+SERVICE_HAS_PORTS_FIELD = any(f.name == "ports" for f in Service._meta.get_fields())
 
 
 class ServiceFilterSet(NetboxServiceFilterSet):
-    """Filter set to support tenancy over the device/VM foreign key.
+    """Filter set to support tenancy and site filtering over the device/VM foreign key.
 
-    Tenancy in Netbox is very incosistent and the relationship on its own is defined across many different models. This
+    Tenancy in Netbox is very inconsistent and the relationship on its own is defined across many different models. This
     means that supporting all layers is nearly impossible without a stronger upstream support. For this reason only the
-    "first level" tenancy is supported by this filter set.
+    "first level" tenancy is supported by the tenant filter set.
     """
 
     tenant_id = MultiValueNumberFilter(
@@ -28,9 +33,21 @@ class ServiceFilterSet(NetboxServiceFilterSet):
         label=_("Tenant (slug)"),
     )
 
-    # fix to make the test_missing_filters pass
+    site_id = MultiValueNumberFilter(
+        method="filter_by_site_id",
+        label=_("Site (ID)"),
+    )
+
+    site = MultiValueCharFilter(
+        method="filter_by_site_slug",
+        label=_("Site (slug)"),
+    )
+
+    # Netbox < 4.7 only exposes the ports array as `port`, but Netbox's
+    # test_missing_filters wants a filter named after the model field.
     # see: https://github.com/netbox-community/netbox/blob/master/netbox/utilities/testing/filtersets.py#L98
-    ports = NumericArrayFilter(field_name="ports", lookup_expr="contains")
+    if SERVICE_HAS_PORTS_FIELD:
+        ports = NumericArrayFilter(field_name="ports", lookup_expr="contains")
 
     def filter_by_cluster_tenant_id(self, queryset, name, value):
         return queryset.filter(
@@ -54,3 +71,14 @@ class ServiceFilterSet(NetboxServiceFilterSet):
             Q(device__tenant__slug__in=value)
             | Q(virtual_machine__tenant__slug__in=value)
         )
+
+    def filter_by_site_id(self, queryset, name, value):
+        return queryset.filter(
+            Q(device__site_id__in=value) | Q(virtual_machine__site_id__in=value)
+        )
+
+    def filter_by_site_slug(self, queryset, name, value):
+        return queryset.filter(
+            Q(device__site__slug__in=value)
+            | Q(virtual_machine__site__slug__in=value)
+        )

netbox_plugin_prometheus_sd-2.2.0/netbox_prometheus_sd/tests/test_filtersets.py

@@ -0,0 +1,63 @@
+from django.test import TestCase
+
+from dcim.models.sites import Site
+from ipam.models import Service
+from tenancy.models import Tenant
+
+try:  # NetBox 4.7+
+    from utilities.testing import ChangeLoggedFilterSetTestMixin
+except ImportError:  # NetBox <4.7
+    from utilities.testing import (
+        ChangeLoggedFilterSetTests as ChangeLoggedFilterSetTestMixin,
+    )
+
+from . import utils
+from ..filtersets import ServiceFilterSet
+
+
+class ServiceTestCase(TestCase, ChangeLoggedFilterSetTestMixin):
+    queryset = Service.objects.all()
+    filterset = ServiceFilterSet
+
+    @classmethod
+    def setUpTestData(cls):
+        """Netbox requires us to define test data in this method, otherwise the ORM won't pick them."""
+        for i in range(1, 4):
+            utils.build_device_full(f"firewall-full-0{i}", i)
+            utils.build_vm_full(f"vm-full-0{i}.example.com", i)
+
+    def test_device_tenant(self):
+        tenant = Tenant.objects.all()[0]
+
+        params = {"tenant_id": [tenant.pk]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
+        params = {"tenant": [tenant.slug]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
+
+    def test_vm_tenant(self):
+        tenant = Tenant.objects.all()[0]
+
+        params = {"tenant_id": [tenant.pk]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
+        params = {"tenant": [tenant.slug]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
+
+    def test_device_site(self):
+        # build_device_full() assigns devices directly to the "site" site.
+        site = Site.objects.get(slug="site")
+
+        params = {"site_id": [site.pk]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 3)
+        params = {"site": [site.slug]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 3)
+
+    def test_vm_site(self):
+        # build_vm_full() assigns VMs directly to the "campus-a" site, distinct
+        # from the devices' site, so this also verifies the filter doesn't match
+        # across device/VM boundaries.
+        site = Site.objects.get(slug="campus-a")
+
+        params = {"site_id": [site.pk]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 3)
+        params = {"site": [site.slug]}
+        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 3)

{netbox_plugin_prometheus_sd-2.0.0 → netbox_plugin_prometheus_sd-2.2.0}/netbox_prometheus_sd/tests/test_serializers.py

@@ -9,6 +9,7 @@ from . import utils
 
 from ..api.utils import NETBOX_RELEASE_CURRENT, NETBOX_RELEASE_41
 
+
 class PrometheusVirtualMachineSerializerTests(TestCase):
     def test_vm_minimal_to_target(self):
 
@@ -498,7 +499,12 @@ class PrometheusServiceSerializerTests(TestCase):
             )
             self.assertTrue(
                 utils.dictContainsSubset(
-                    {"__meta_netbox_display": "ssh (TCP/22)"}, data["labels"]
+                    {
+                        "__meta_netbox_display": utils.expected_service_display(
+                            "ssh", "TCP", 22
+                        )
+                    },
+                    data["labels"],
                 )
             )
             self.assertTrue(
@@ -562,7 +568,12 @@ class PrometheusServiceSerializerTests(TestCase):
             )
             self.assertTrue(
                 utils.dictContainsSubset(
-                    {"__meta_netbox_display": "ssh (TCP/22)"}, data["labels"]
+                    {
+                        "__meta_netbox_display": utils.expected_service_display(
+                            "ssh", "TCP", 22
+                        )
+                    },
+                    data["labels"],
                 )
             )
             self.assertTrue(

{netbox_plugin_prometheus_sd-2.0.0 → netbox_plugin_prometheus_sd-2.2.0}/netbox_prometheus_sd/tests/utils.py

@@ -29,7 +29,7 @@ def dictContainsSubset(subset, fullset):
 
 
 def build_cluster():
-    try: # NetBox 4.2+
+    try:  # NetBox 4.2+
         scope_type = ContentType.objects.get_for_model(Site)
         return Cluster.objects.get_or_create(
             name="DC1",
@@ -38,7 +38,7 @@ def build_cluster():
             scope_type=scope_type,
             scope_id=Site.objects.get_or_create(name="Campus A", slug="campus-a")[0].id,
         )[0]
-    except FieldError: # NetBox <4.2
+    except FieldError:  # NetBox <4.2
         return Cluster.objects.get_or_create(
             name="DC1",
             group=ClusterGroup.objects.get_or_create(name="VMware")[0],
@@ -119,6 +119,10 @@ def build_vm_full(name, ip_octet=1):
     )[0]
 
     vm.tenant = build_tenant()
+    # Direct site assignment, independent of the cluster's own site/scope -- Netbox
+    # supports both, and the plugin's ServiceFilterSet only filters on this direct
+    # relation (see filtersets.py).
+    vm.site = Site.objects.get_or_create(name="Campus A", slug="campus-a")[0]
     vm.custom_field_data = build_custom_fields()
     vm.role = DeviceRole.objects.get_or_create(name="VM", slug="vm", vm_role=True)[0]
     vm.primary_ip4 = IPAddress.objects.get_or_create(
@@ -137,15 +141,29 @@ def build_vm_full(name, ip_octet=1):
     return vm
 
 
+def expected_service_display(name, protocol, port):
+    """Netbox 4.7 reduced Service.__str__ to the bare name; older releases
+    append the protocol and port list."""
+    if hasattr(Service, "port_mappings"):
+        return name
+    return f"{name} ({protocol}/{port})"
+
+
 def build_service_for(parent, **kwargs):
     """Create a service bound to parent, with the parent's primary IPv4 attached.
 
     Netbox 4.3 replaced Service.device/Service.virtual_machine with a single
     generic `parent` relation, which is read-only on older releases.
+
+    Netbox 4.7 replaced Service.protocol/Service.ports with `port_mappings`
+    and rejects the legacy pair at the ORM level.
     """
-    try: # NetBox 4.3+
+    if hasattr(Service, "port_mappings") and "ports" in kwargs:
+        protocol = kwargs.pop("protocol", "tcp")
+        kwargs["port_mappings"] = [f"{protocol}/{port}" for port in kwargs.pop("ports")]
+    try:  # NetBox 4.3+
         service = Service.objects.create(parent=parent, **kwargs)
-    except AttributeError: # NetBox <4.3
+    except AttributeError:  # NetBox <4.3
         field = "device" if isinstance(parent, Device) else "virtual_machine"
         service = Service.objects.create(**{field: parent}, **kwargs)
 

{netbox_plugin_prometheus_sd-2.0.0 → netbox_plugin_prometheus_sd-2.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "netbox-plugin-prometheus-sd"
-version = "v2.0.0" # placeholder
+version = "2.2.0" # placeholder
 description = "A Netbox plugin to provide Netbox entires to Prometheus HTTP service discovery"
 authors = ["Felix Peters <felix.peters@breuninger.de>"]
 license = "MIT"

netbox_plugin_prometheus_sd-2.0.0/PKG-INFO

@@ -1,158 +0,0 @@
-Metadata-Version: 2.1
-Name: netbox-plugin-prometheus-sd
-Version: 2.0.0
-Summary: A Netbox plugin to provide Netbox entires to Prometheus HTTP service discovery
-License: MIT
-Author: Felix Peters
-Author-email: felix.peters@breuninger.de
-Requires-Python: >=3.10,<4.0
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Description-Content-Type: text/markdown
-
-# netbox-plugin-prometheus-sd
-
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![CI](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/workflows/CI/badge.svg?event=push)](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/actions?query=workflow%3ACI)
-[![PyPI](https://img.shields.io/pypi/v/netbox-plugin-prometheus-sd)](https://pypi.org/project/netbox-plugin-prometheus-sd/)
-
-[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/flxpeters)
-
-Provide Prometheus `http_sd` compatible API Endpoint with data from Netbox.
-
-HTTP SD is a feature since Prometheus 2.28.0 that allows hosts to be found via a URL instead of just files.
-This plugin implements API endpoints in Netbox to make devices, services, IPs and virtual machines available to Prometheus.
-
-## Compatibility
-
-We aim to support the latest major versions of Netbox.
-For now we support Netbox `>= 4.0` including bugfix versions. Older versions may work, but without any guarantee.
-
-Check the `.github/workflows/ci.yml` pipeline for the current tested builds.
-Other versions may work, but we do not test them explicitly. All relevant target versions are tested in CI.
-
-## Installation
-
-The plugin is available as a Python package in pypi and can be installed with pip
-
-```bash
-pip install netbox-plugin-prometheus-sd
-```
-
-Enable the plugin in /opt/netbox/netbox/netbox/configuration.py:
-
-```python
-    PLUGINS = ['netbox_prometheus_sd']
-```
-
-The plugin has not further plugin configuration.
-
-## Usage
-
-The plugin only provides a new API endpoint on the Netbox API. There is no further action required after installation.
-
-### API
-
-The plugin reuses Netbox API view sets with new serializers for Prometheus.
-This means that all filters that can be used on the Netbox API can also be used to filter Prometheus targets.
-Paging is disabled because Prometheus does not support paged results.
-
-The plugin also reuses the Netbox authentication and permission model.
-Depending on the Netbox configuration, a token with valid object permissions must be passed to Netbox.
-
-```
-GET        /api/plugins/prometheus-sd/devices/              Get a list of devices in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/virtual-machines/     Get a list of vms in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/services/             Get a list of services in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/ip-addresses/         Get a list of ip in a prometheus compatible format
-```
-
-#### Extended services filters
-
-Apart from standard Netbox filters, services endpoint also supports `tenant=<slug>` or `tenant_id=<id>` parameters.
-The lookup is only executed against the `tenant` attribute of the object associated with the service.
-
-### Config context
-
-The plugin can also discover extra config to inject in the HTTP SD JSON from the config context of the devices/virtual machines.
-If you have a `prometheus-plugin-prometheus-sd` entry in your config context with the following schema it will be automatically picked up:
-
-```yaml
-prometheus-plugin-prometheus-sd:
-  - metrics_path: /not/metrics
-    port: 4242
-    scheme: https
-  - port: 4243
-```
-
-This allow you to configure those values directly into netbox instead of doing that inside the Prometheus
-config and filtering each scenario by a specific tag for instance.
-
-If there is only one entry you can also use this form:
-
-```yaml
-prometheus-plugin-prometheus-sd:
-  metrics_path: /not/metrics
-  port: 4242
-  scheme: https
-```
-
-### Example
-
-A working example on how to use this plugin with Prometheus is located at the `example` folder.
-Netbox content is created by using Netbox docker initializers.
-
-The demo data doesn't make sense, but they are good enough for demonstrating how to configure Prometheus
-and get demo data to Prometheus service discovery.
-
-Go to the `example` folder and run `docker-compose up --build`. Prometheus should get available on `http://localhost:9090`.
-
-Push some example devices and objects to Netbox using the initializers:
-
-```
-docker-compose exec  netbox /opt/netbox/netbox/manage.py load_initializer_data --path /opt/netbox/initializers
-```
-
-Netbox content should then be available in the service discovery tab.
-
-## Development
-
-We use [Poetry](https://python-poetry.org/) for dependency management and [invoke](https://www.pyinvoke.org/) as task runner.
-To test the plugin in an isolated environment, we use [testcontainers](https://testcontainers.com/?language=python)
-which creates "throwaway, lightweight" Netbox Docker containers.
-
-All code to run in docker is located under `develop`.
-To start a virtual env managed by poetry run `poetry shell`.
-All following commands are started inside this environment.
-
-In order to run tests invoke the tests:
-
-```bash
-# Build the containers and execute all tests
-invoke test
-```
-
-Features should be covered by a unit test, but some times it's easier to develop on an running system.
-
-```bash
-# Start a development environment
-invoke build_dev
-```
-
-Visit http://localhost:8000 and log in with the default admin credentials.
-You can now define Netbox entities and test around.
-
-API endpoints for testing can be found at http://localhost:8000/api/plugins/prometheus-sd/
-
-## Conventional Commits
-
-This repository follows the Conventional Commits specification for versioning and changelog generation.
-Conventional Commits provide a standardized way of writing commit messages to convey semantic meaning
-about the changes made. Each commit message follows a defined format that includes a type,
-an optional scope, and a message. The types typically include features, fixes, documentation, and more.
-By adhering to this convention, we ensure clear and automated versioning, release notes, and changelog generation.
-

netbox_plugin_prometheus_sd-2.0.0/README.md

@@ -1,141 +0,0 @@
-# netbox-plugin-prometheus-sd
-
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![CI](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/workflows/CI/badge.svg?event=push)](https://github.com/FlxPeters/netbox-plugin-prometheus-sd/actions?query=workflow%3ACI)
-[![PyPI](https://img.shields.io/pypi/v/netbox-plugin-prometheus-sd)](https://pypi.org/project/netbox-plugin-prometheus-sd/)
-
-[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/flxpeters)
-
-Provide Prometheus `http_sd` compatible API Endpoint with data from Netbox.
-
-HTTP SD is a feature since Prometheus 2.28.0 that allows hosts to be found via a URL instead of just files.
-This plugin implements API endpoints in Netbox to make devices, services, IPs and virtual machines available to Prometheus.
-
-## Compatibility
-
-We aim to support the latest major versions of Netbox.
-For now we support Netbox `>= 4.0` including bugfix versions. Older versions may work, but without any guarantee.
-
-Check the `.github/workflows/ci.yml` pipeline for the current tested builds.
-Other versions may work, but we do not test them explicitly. All relevant target versions are tested in CI.
-
-## Installation
-
-The plugin is available as a Python package in pypi and can be installed with pip
-
-```bash
-pip install netbox-plugin-prometheus-sd
-```
-
-Enable the plugin in /opt/netbox/netbox/netbox/configuration.py:
-
-```python
-    PLUGINS = ['netbox_prometheus_sd']
-```
-
-The plugin has not further plugin configuration.
-
-## Usage
-
-The plugin only provides a new API endpoint on the Netbox API. There is no further action required after installation.
-
-### API
-
-The plugin reuses Netbox API view sets with new serializers for Prometheus.
-This means that all filters that can be used on the Netbox API can also be used to filter Prometheus targets.
-Paging is disabled because Prometheus does not support paged results.
-
-The plugin also reuses the Netbox authentication and permission model.
-Depending on the Netbox configuration, a token with valid object permissions must be passed to Netbox.
-
-```
-GET        /api/plugins/prometheus-sd/devices/              Get a list of devices in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/virtual-machines/     Get a list of vms in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/services/             Get a list of services in a prometheus compatible format
-GET        /api/plugins/prometheus-sd/ip-addresses/         Get a list of ip in a prometheus compatible format
-```
-
-#### Extended services filters
-
-Apart from standard Netbox filters, services endpoint also supports `tenant=<slug>` or `tenant_id=<id>` parameters.
-The lookup is only executed against the `tenant` attribute of the object associated with the service.
-
-### Config context
-
-The plugin can also discover extra config to inject in the HTTP SD JSON from the config context of the devices/virtual machines.
-If you have a `prometheus-plugin-prometheus-sd` entry in your config context with the following schema it will be automatically picked up:
-
-```yaml
-prometheus-plugin-prometheus-sd:
-  - metrics_path: /not/metrics
-    port: 4242
-    scheme: https
-  - port: 4243
-```
-
-This allow you to configure those values directly into netbox instead of doing that inside the Prometheus
-config and filtering each scenario by a specific tag for instance.
-
-If there is only one entry you can also use this form:
-
-```yaml
-prometheus-plugin-prometheus-sd:
-  metrics_path: /not/metrics
-  port: 4242
-  scheme: https
-```
-
-### Example
-
-A working example on how to use this plugin with Prometheus is located at the `example` folder.
-Netbox content is created by using Netbox docker initializers.
-
-The demo data doesn't make sense, but they are good enough for demonstrating how to configure Prometheus
-and get demo data to Prometheus service discovery.
-
-Go to the `example` folder and run `docker-compose up --build`. Prometheus should get available on `http://localhost:9090`.
-
-Push some example devices and objects to Netbox using the initializers:
-
-```
-docker-compose exec  netbox /opt/netbox/netbox/manage.py load_initializer_data --path /opt/netbox/initializers
-```
-
-Netbox content should then be available in the service discovery tab.
-
-## Development
-
-We use [Poetry](https://python-poetry.org/) for dependency management and [invoke](https://www.pyinvoke.org/) as task runner.
-To test the plugin in an isolated environment, we use [testcontainers](https://testcontainers.com/?language=python)
-which creates "throwaway, lightweight" Netbox Docker containers.
-
-All code to run in docker is located under `develop`.
-To start a virtual env managed by poetry run `poetry shell`.
-All following commands are started inside this environment.
-
-In order to run tests invoke the tests:
-
-```bash
-# Build the containers and execute all tests
-invoke test
-```
-
-Features should be covered by a unit test, but some times it's easier to develop on an running system.
-
-```bash
-# Start a development environment
-invoke build_dev
-```
-
-Visit http://localhost:8000 and log in with the default admin credentials.
-You can now define Netbox entities and test around.
-
-API endpoints for testing can be found at http://localhost:8000/api/plugins/prometheus-sd/
-
-## Conventional Commits
-
-This repository follows the Conventional Commits specification for versioning and changelog generation.
-Conventional Commits provide a standardized way of writing commit messages to convey semantic meaning
-about the changes made. Each commit message follows a defined format that includes a type,
-an optional scope, and a message. The types typically include features, fixes, documentation, and more.
-By adhering to this convention, we ensure clear and automated versioning, release notes, and changelog generation.

netbox_plugin_prometheus_sd-2.0.0/netbox_prometheus_sd/tests/test_filtersets.py

@@ -1,36 +0,0 @@
-from django.test import TestCase
-
-from ipam.models import Service
-from tenancy.models import Tenant
-from utilities.testing import ChangeLoggedFilterSetTests
-
-from . import utils
-from ..filtersets import ServiceFilterSet
-
-
-class ServiceTestCase(TestCase, ChangeLoggedFilterSetTests):
-    queryset = Service.objects.all()
-    filterset = ServiceFilterSet
-
-    @classmethod
-    def setUpTestData(cls):
-        """Netbox requires us to define test data in this method, otherwise the ORM won't pick them."""
-        for i in range(1, 4):
-            utils.build_device_full(f"firewall-full-0{i}", i)
-            utils.build_vm_full(f"vm-full-0{i}.example.com", i)
-
-    def test_device_tenant(self):
-        tenant = Tenant.objects.all()[0]
-
-        params = {"tenant_id": [tenant.pk]}
-        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
-        params = {"tenant": [tenant.slug]}
-        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
-
-    def test_vm_tenant(self):
-        tenant = Tenant.objects.all()[0]
-
-        params = {"tenant_id": [tenant.pk]}
-        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)
-        params = {"tenant": [tenant.slug]}
-        self.assertEqual(self.filterset(params, self.queryset).qs.count(), 6)