venus-os-observability 0.1.11__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- venus_os_observability-0.1.11/LICENSE +21 -0
- venus_os_observability-0.1.11/PKG-INFO +532 -0
- venus_os_observability-0.1.11/README.md +480 -0
- venus_os_observability-0.1.11/pyproject.toml +120 -0
- venus_os_observability-0.1.11/setup.cfg +4 -0
- venus_os_observability-0.1.11/setup.py +6 -0
- venus_os_observability-0.1.11/src/venus_observability/__init__.py +39 -0
- venus_os_observability-0.1.11/src/venus_observability/__main__.py +286 -0
- venus_os_observability-0.1.11/src/venus_observability/correlation.py +340 -0
- venus_os_observability-0.1.11/src/venus_observability/dbus_listener.py +365 -0
- venus_os_observability-0.1.11/src/venus_observability/dbus_owners.py +221 -0
- venus_os_observability-0.1.11/src/venus_observability/metrics.py +367 -0
- venus_os_observability-0.1.11/src/venus_observability/otlp.py +242 -0
- venus_os_observability-0.1.11/src/venus_observability/py.typed +0 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/PKG-INFO +532 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/SOURCES.txt +32 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/dependency_links.txt +1 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/entry_points.txt +2 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/requires.txt +42 -0
- venus_os_observability-0.1.11/src/venus_os_observability.egg-info/top_level.txt +1 -0
- venus_os_observability-0.1.11/tests/test_alert_bridge.py +219 -0
- venus_os_observability-0.1.11/tests/test_correlation.py +352 -0
- venus_os_observability-0.1.11/tests/test_dbus_listener.py +421 -0
- venus_os_observability-0.1.11/tests/test_installer.py +83 -0
- venus_os_observability-0.1.11/tests/test_invalid_dbus_batch.py +43 -0
- venus_os_observability-0.1.11/tests/test_main.py +414 -0
- venus_os_observability-0.1.11/tests/test_metrics.py +193 -0
- venus_os_observability-0.1.11/tests/test_otlp.py +167 -0
- venus_os_observability-0.1.11/tests/test_otlp_tls.py +688 -0
- venus_os_observability-0.1.11/tests/test_owner_lifecycle.py +323 -0
- venus_os_observability-0.1.11/tests/test_owner_metric_invalidation.py +203 -0
- venus_os_observability-0.1.11/tests/test_relay_tls.py +429 -0
- venus_os_observability-0.1.11/tests/test_setuphelper_archive.py +234 -0
- venus_os_observability-0.1.11/tests/test_smtp_tls.py +201 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 victron-venus
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,532 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: venus-os-observability
|
|
3
|
+
Version: 0.1.11
|
|
4
|
+
Summary: OpenTelemetry/Prometheus observability for Venus OS - D-Bus event tracing, inverter metrics, distributed tracing
|
|
5
|
+
License: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/victron-venus/venus-os-observability
|
|
7
|
+
Project-URL: Repository, https://github.com/victron-venus/venus-os-observability
|
|
8
|
+
Project-URL: Issues, https://github.com/victron-venus/venus-os-observability/issues
|
|
9
|
+
Requires-Python: <3.13,>=3.12
|
|
10
|
+
Description-Content-Type: text/markdown
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Requires-Dist: opentelemetry-api<1.29,>=1.28
|
|
13
|
+
Requires-Dist: opentelemetry-sdk<1.29,>=1.28
|
|
14
|
+
Requires-Dist: opentelemetry-exporter-prometheus<0.57b0,>=0.49b2
|
|
15
|
+
Requires-Dist: opentelemetry-instrumentation<0.50,>=0.49b0
|
|
16
|
+
Requires-Dist: prometheus-client>=0.19
|
|
17
|
+
Requires-Dist: paho-mqtt>=2.0
|
|
18
|
+
Requires-Dist: dbus-next>=0.2
|
|
19
|
+
Requires-Dist: pyyaml>=6.0
|
|
20
|
+
Requires-Dist: structlog>=24.0
|
|
21
|
+
Requires-Dist: click>=8.1
|
|
22
|
+
Provides-Extra: dev
|
|
23
|
+
Requires-Dist: types-atheris>=3.1.0.20260630; extra == "dev"
|
|
24
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "dev"
|
|
25
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "dev"
|
|
26
|
+
Requires-Dist: requests>=2.34.2; extra == "dev"
|
|
27
|
+
Requires-Dist: cryptography>=50.0.2; extra == "dev"
|
|
28
|
+
Requires-Dist: pytest>=8.0; extra == "dev"
|
|
29
|
+
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
|
|
30
|
+
Requires-Dist: pytest-cov>=5.0; extra == "dev"
|
|
31
|
+
Requires-Dist: ruff>=0.5; extra == "dev"
|
|
32
|
+
Requires-Dist: mypy>=1.10; extra == "dev"
|
|
33
|
+
Requires-Dist: pre-commit>=3.7; extra == "dev"
|
|
34
|
+
Requires-Dist: types-setuptools>=75; extra == "dev"
|
|
35
|
+
Provides-Extra: tempo
|
|
36
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "tempo"
|
|
37
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "tempo"
|
|
38
|
+
Requires-Dist: requests>=2.34.2; extra == "tempo"
|
|
39
|
+
Requires-Dist: cryptography>=50.0.2; extra == "tempo"
|
|
40
|
+
Provides-Extra: all
|
|
41
|
+
Requires-Dist: requests>=2.34.2; extra == "all"
|
|
42
|
+
Requires-Dist: cryptography>=50.0.2; extra == "all"
|
|
43
|
+
Requires-Dist: pytest>=8.0; extra == "all"
|
|
44
|
+
Requires-Dist: pytest-asyncio>=0.23; extra == "all"
|
|
45
|
+
Requires-Dist: pytest-cov>=5.0; extra == "all"
|
|
46
|
+
Requires-Dist: ruff>=0.5; extra == "all"
|
|
47
|
+
Requires-Dist: mypy>=1.10; extra == "all"
|
|
48
|
+
Requires-Dist: pre-commit>=3.7; extra == "all"
|
|
49
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "all"
|
|
50
|
+
Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "all"
|
|
51
|
+
Dynamic: license-file
|
|
52
|
+
|
|
53
|
+
# Venus OS Observability
|
|
54
|
+
|
|
55
|
+
[](docs/fuzzing.md)
|
|
56
|
+
[](https://www.bestpractices.dev/projects/15457)
|
|
57
|
+
|
|
58
|
+
OpenTelemetry/Prometheus observability for Victron Venus OS: D-Bus signal tracing, inverter metrics export, optional application correlation helpers, and a separate Grafana alert relay.
|
|
59
|
+
|
|
60
|
+
[](docs/images/venus-demo.jpg)
|
|
61
|
+
|
|
62
|
+
*Grafana dashboard layout with synthetic demonstration data; this is not live installation telemetry. Click for the original screenshot.*
|
|
63
|
+
|
|
64
|
+
[Image source and regeneration](docs/images/README.md).
|
|
65
|
+
|
|
66
|
+
## Python runtime
|
|
67
|
+
|
|
68
|
+
Native Venus OS packages target **Python 3.12.x**. The audited Raspberry Pi 3
|
|
69
|
+
running Venus OS v3.75 reports Python **3.12.13**; the [official Venus OS v3.79 manifest](https://updates.victronenergy.com/feeds/venus/release/sdk/venus-scarthgap-x86_64-arm-cortexa8hf-neon-toolchain-v3.79.target.manifest)
|
|
70
|
+
also ships 3.12.13. Local development and CI use `.python-version` / Python
|
|
71
|
+
3.12.13. Package metadata accepts 3.12 patch updates and rejects other minor
|
|
72
|
+
versions until they have been validated. Use the firmware's system interpreter
|
|
73
|
+
and its matching D-Bus/GI libraries on the device; do not replace the OS Python.
|
|
74
|
+
|
|
75
|
+
<!-- ci-release-process:start -->
|
|
76
|
+
## Release process
|
|
77
|
+
|
|
78
|
+
See the [release strategy](RELEASING.md) for validation, nightly, beta, RC and stable promotion rules, and the [operator runbook](docs/release-workflow.md) for local commands.
|
|
79
|
+
<!-- ci-release-process:end -->
|
|
80
|
+
|
|
81
|
+
## Architecture
|
|
82
|
+
|
|
83
|
+
```mermaid
|
|
84
|
+
graph LR
|
|
85
|
+
A[Venus D-Bus services] -->|ItemsChanged| B[D-Bus Signal Listener]
|
|
86
|
+
B --> C[Metrics and optional spans]
|
|
87
|
+
C --> D[Prometheus Exporter :9090/metrics]
|
|
88
|
+
C -->|Configured OTLP endpoint| E[Collector / Grafana Tempo]
|
|
89
|
+
D --> P[Prometheus / Grafana]
|
|
90
|
+
P -->|Configured alert webhook| R[Separate alert-mqtt-bridge]
|
|
91
|
+
R --> M[MQTT banners / SMTP / Telegram]
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
D-Bus observations feed metrics and optional spans. Prometheus reads the metrics; configured collectors receive spans. The separate relay receives Grafana alerts and forwards notifications to MQTT and optional email/Telegram destinations. The arrows above describe observation and notification, not physical control.
|
|
95
|
+
|
|
96
|
+
## Features
|
|
97
|
+
|
|
98
|
+
- **D-Bus Signal Tracing**: Observe Victron `ItemsChanged` signals; record spans when tracing is configured
|
|
99
|
+
- **Prometheus Metrics**: Export inverter state (SOC, power, grid, battery) as Prometheus metrics
|
|
100
|
+
- **Correlation Helpers**: Explicit MQTT/application instrumentation helpers; installing the agent does not instrument other services or propagate context across native D-Bus automatically
|
|
101
|
+
- **Grafana Tempo Integration**: Visualize trace timelines in Grafana
|
|
102
|
+
- **Native Venus OS Service**: Supports native supervision; offline pip installation requires wheels matching the target Python version and architecture
|
|
103
|
+
|
|
104
|
+
Explicitly unavailable numeric D-Bus values, including Venus empty arrays, are
|
|
105
|
+
exported as `NaN` in both metric backends. They replace the previous reading and
|
|
106
|
+
do not discard other items in the same signal. A later valid zero is retained as
|
|
107
|
+
zero. Text-only updates do not replace numeric measurements. This handles value
|
|
108
|
+
invalidation; it does not independently detect every silent source outage.
|
|
109
|
+
|
|
110
|
+
## Deployment evidence
|
|
111
|
+
|
|
112
|
+
The [verification record and acceptance checklist](docs/deployment-evidence.md)
|
|
113
|
+
separate historical runtime observations from source tests and current deployment
|
|
114
|
+
checks. A release does not establish that its artifact is installed or that every
|
|
115
|
+
configured alert channel reaches a recipient.
|
|
116
|
+
|
|
117
|
+
## Installation
|
|
118
|
+
|
|
119
|
+
### Option 1: SetupHelper / PackageManager (Recommended)
|
|
120
|
+
|
|
121
|
+
**Preferred install path on Cerbo GX / Venus OS** — same as `dbus-ev` and `inverter-control`. Requires [SetupHelper](https://github.com/kwindrem/SetupHelper) on the GX.
|
|
122
|
+
|
|
123
|
+
The easiest way to install is via SetupHelper PackageManager. The `setup` script is PackageManager-compatible and handles service creation, `/data/rc.local` boot persistence, and restarts.
|
|
124
|
+
|
|
125
|
+
Package files needed at `/data/venus-os-observability`: `version`, `setup`, `gitHubInfo`, `update.sh`, `src/`, `services/`.
|
|
126
|
+
|
|
127
|
+
1. **Add package via GUI** (GUI v1):
|
|
128
|
+
- Settings → PackageManager → Inactive packages → **new**
|
|
129
|
+
- Package name: `venus-os-observability`
|
|
130
|
+
- GitHub user: `victron-venus`
|
|
131
|
+
- Branch / tag: `latest` (matches `gitHubInfo`)
|
|
132
|
+
- Proceed → Download → Install
|
|
133
|
+
|
|
134
|
+
2. **Or on the device**:
|
|
135
|
+
|
|
136
|
+
```sh
|
|
137
|
+
/data/venus-os-observability/setup install
|
|
138
|
+
/data/venus-os-observability/setup uninstall
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
`gitHubInfo` is `victron-venus:latest`.
|
|
142
|
+
|
|
143
|
+
`update.sh` (also called by `setup install`):
|
|
144
|
+
|
|
145
|
+
- Installs into **`/data/venus-os-observability`** (SetupHelper convention).
|
|
146
|
+
- Copies daemontools service to `INSTALL_DIR/service/venus-os-observability/` and symlinks `/service/venus-os-observability`.
|
|
147
|
+
- Writes an idempotent **`/data/rc.local`** marker block so the symlink is recreated after reboot (`/service` is tmpfs). Venus OS does **not** run `/data/rc/S99*`.
|
|
148
|
+
- Migrates `.venv2` from a legacy `/data/opt/venus-os-observability` install when present.
|
|
149
|
+
- Removes the dead `/data/rc/S99venus-os-observability.sh` hook if found.
|
|
150
|
+
|
|
151
|
+
Manual refresh from a checkout on the Mac:
|
|
152
|
+
|
|
153
|
+
```sh
|
|
154
|
+
# stream repo → device, run update.sh (same pattern as dbus-ev deploy.sh)
|
|
155
|
+
COPYFILE_DISABLE=1 tar --no-xattrs \
|
|
156
|
+
--exclude='.git' --exclude='.venv' --exclude='__pycache__' \
|
|
157
|
+
--exclude='.pytest_cache' --exclude='.ruff_cache' --exclude='logs' \
|
|
158
|
+
--exclude='*.egg-info' --exclude='build' --exclude='dist' --exclude='wheels' \
|
|
159
|
+
-czf - -C /path/to/venus-os-observability . \
|
|
160
|
+
| ssh root@cerbo 'set -e; rm -rf /data/.vos-obs-deploy; mkdir -p /data/.vos-obs-deploy; \
|
|
161
|
+
tar -xz -C /data/.vos-obs-deploy; sh /data/.vos-obs-deploy/update.sh; \
|
|
162
|
+
rm -rf /data/.vos-obs-deploy; svstat /service/venus-os-observability'
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Runtime prefers `.venv2/bin/python`, then `.venv/bin/python`, with
|
|
166
|
+
`PYTHONPATH=/data/venus-os-observability/src` so source updates take effect.
|
|
167
|
+
Installation checks imports before stopping the service, preserves supervisor
|
|
168
|
+
directories, and never kills processes merely because their working directory
|
|
169
|
+
is the package directory. Only `/data/venus-os-observability` is supported.
|
|
170
|
+
|
|
171
|
+
The boot hook is inserted before a conventional `exit 0` in `/data/rc.local`.
|
|
172
|
+
`/service` is volatile; `/data/rc/S99*` is not a supported boot hook.
|
|
173
|
+
Logs use `multilog t s25000 n4` under `/var/log`, approximately 125 KB including
|
|
174
|
+
the current file, rather than an unbounded file on persistent flash.
|
|
175
|
+
|
|
176
|
+
#### Offline wheel bootstrap (first installation)
|
|
177
|
+
|
|
178
|
+
Check `python3 --version` and `uname -m` on the target first. The audited Cerbo
|
|
179
|
+
uses Python 3.12 and ARMv7; Raspberry Pi images may differ. Build the wheel bundle
|
|
180
|
+
with `scripts/build_wheels.sh` on a matching Linux architecture/Python ABI.
|
|
181
|
+
It resolves versions from `pyproject.toml`, requires binary dependency wheels,
|
|
182
|
+
and includes a pip wheel. It is not an ARM cross-compiler. If a native wheel is
|
|
183
|
+
unavailable, build it in a matching build environment before deployment.
|
|
184
|
+
|
|
185
|
+
Venus OS provides `dbus-python` and GLib. Preserve access to these native bindings
|
|
186
|
+
using `--system-site-packages`; `dbus-next` cannot replace them for this agent.
|
|
187
|
+
Do not install replacement bindings over the firmware's Python packages.
|
|
188
|
+
|
|
189
|
+
Copy and extract the bundle under `/data/venus-os-observability/wheels`, then:
|
|
190
|
+
|
|
191
|
+
```sh
|
|
192
|
+
cd /data/venus-os-observability
|
|
193
|
+
python3 -m venv --system-site-packages --without-pip .venv2
|
|
194
|
+
# pip can run directly from its wheel, even when ensurepip is absent.
|
|
195
|
+
for wheel in wheels/pip-*.whl; do
|
|
196
|
+
PYTHONPATH="$wheel" .venv2/bin/python -m pip install \
|
|
197
|
+
--no-index --find-links=wheels venus-os-observability
|
|
198
|
+
done
|
|
199
|
+
sh update.sh
|
|
200
|
+
svstat /service/venus-os-observability /service/venus-os-observability/log
|
|
201
|
+
tail -n 60 /var/log/venus-os-observability/current
|
|
202
|
+
wget -qO - http://127.0.0.1:9090/metrics
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
If the image lacks `venv` itself, prepare a compatible offline virtualenv
|
|
206
|
+
bootstrap first; the installer does not silently fetch packages from the
|
|
207
|
+
internet. Repeat the bootstrap after firmware changes the Python ABI.
|
|
208
|
+
|
|
209
|
+
Tracing is optional: install the `tempo` extra from a matching offline bundle
|
|
210
|
+
and configure `OTEL_EXPORTER_OTLP_ENDPOINT` only when the device has headroom.
|
|
211
|
+
Without an endpoint, the sampler drops spans and avoids D-Bus payload serialization;
|
|
212
|
+
metrics remain active. D-Bus string subclasses are converted to native strings
|
|
213
|
+
before recorded span attributes are validated.
|
|
214
|
+
|
|
215
|
+
Local plaintext gRPC remains the default. For HTTPS collectors, select
|
|
216
|
+
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` and configure the collector's
|
|
217
|
+
OTLP/HTTP endpoint; HTTPS gRPC is rejected because its native TLS backend does
|
|
218
|
+
not enforce the required minimum certificate key strengths. See
|
|
219
|
+
[OTLP transport and migration](docs/otlp-transport.md) before changing an
|
|
220
|
+
existing deployment. No protocol, port or TLS configuration is changed silently.
|
|
221
|
+
|
|
222
|
+
Metric labels use stable Venus service names. The exporter subscribes to
|
|
223
|
+
`NameOwnerChanged` before asynchronous startup discovery, removes superseded
|
|
224
|
+
owners and invalidates that service's previously published gauges on loss or
|
|
225
|
+
replacement. Fresh measurements restore those gauges; stable counters remain
|
|
226
|
+
cumulative. A reconnect of the same well-known service does not create new
|
|
227
|
+
per-process metric series. Adding genuinely new service names or item paths can
|
|
228
|
+
still create new series; this is not a global registry quota.
|
|
229
|
+
|
|
230
|
+
Signals received before their sender is resolved are replayed in arrival order
|
|
231
|
+
once the Venus service is known. Unresolved input is limited to 256 batches and
|
|
232
|
+
4,096 items, with a 10-second monotonic expiry. Overflow removes the oldest
|
|
233
|
+
unresolved batches; an oversized single batch is discarded. The fixed-reason
|
|
234
|
+
`victron_dbus_unresolved_signals_dropped_total` counter exposes overflow/expiry,
|
|
235
|
+
and warnings are limited to one per minute. Known services bypass this queue.
|
|
236
|
+
Owner discovery uses asynchronous calls with five-second deadlines; failed
|
|
237
|
+
discovery can retry after five seconds, while other unknown-sender refreshes
|
|
238
|
+
are limited to once per minute. It does not perform a blocking scan per signal.
|
|
239
|
+
|
|
240
|
+
The historical IPK Makefile was an OpenWrt/systemd recipe, not a working Venus
|
|
241
|
+
OS package. It now fails with an explicit migration message. The `systemd/`
|
|
242
|
+
example is for separate Linux hosts only; native GX deployment uses SetupHelper.
|
|
243
|
+
|
|
244
|
+
Release candidates provide `venus-os-observability-VERSION.tar.gz` and
|
|
245
|
+
`SHA256SUMS`. This archive contains the native installer, all runtime modules,
|
|
246
|
+
service scripts, example configuration and Python package metadata. CI validates
|
|
247
|
+
its contents, executable modes and matching package versions before publication.
|
|
248
|
+
Extract its single top-level directory into a staging directory and run its
|
|
249
|
+
`update.sh` to update the standard `/data/venus-os-observability` installation.
|
|
250
|
+
Device configuration, virtual environments and live supervisor directories are
|
|
251
|
+
preserved. Dependency wheels are a separate prerequisite; neither this source
|
|
252
|
+
archive nor the Python wheel is an IPK package.
|
|
253
|
+
|
|
254
|
+
Wheel and source distributions accompany the verified GitHub release assets.
|
|
255
|
+
PyPI publication requires a separate explicit promotion of those same verified
|
|
256
|
+
distributions, as described in [RELEASING.md](RELEASING.md). A GitHub release alone
|
|
257
|
+
does not imply that the same version is available from PyPI.
|
|
258
|
+
|
|
259
|
+
### Option 2: Docker Compose
|
|
260
|
+
|
|
261
|
+
For non-GX hosts or a containerized stack (not the preferred Cerbo path):
|
|
262
|
+
|
|
263
|
+
```yaml
|
|
264
|
+
services:
|
|
265
|
+
venus-observability:
|
|
266
|
+
image: ghcr.io/victron-venus/venus-os-observability:latest
|
|
267
|
+
environment:
|
|
268
|
+
- DBUS_SYSTEM_BUS_ADDRESS=unix:path=/run/dbus/system_bus_socket
|
|
269
|
+
- OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317
|
|
270
|
+
- PROMETHEUS_PORT=9090
|
|
271
|
+
volumes:
|
|
272
|
+
- /run/dbus:/run/dbus
|
|
273
|
+
ports:
|
|
274
|
+
- "9090:9090"
|
|
275
|
+
restart: unless-stopped
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
## Metrics Exported
|
|
279
|
+
|
|
280
|
+
| Metric | Type | Description |
|
|
281
|
+
|--------|------|-------------|
|
|
282
|
+
| `victron_battery_soc` | Gauge | Battery state of charge % |
|
|
283
|
+
| `victron_battery_power` | Gauge | Battery charge/discharge power (W) |
|
|
284
|
+
| `victron_pv_power` | Gauge | Total PV production (W) |
|
|
285
|
+
| `victron_grid_power` | Gauge | Grid import/export (W) |
|
|
286
|
+
| `victron_ac_loads` | Gauge | AC loads consumption (W) |
|
|
287
|
+
| `victron_inverter_state` | Gauge | Inverter state machine (0=off,1=on,2=charging,3=inverting) |
|
|
288
|
+
|
|
289
|
+
## Alert Delivery Setup
|
|
290
|
+
|
|
291
|
+
Configure Grafana alert rules such as `venus-agent-down`, `venus-dbus-errors` and
|
|
292
|
+
`venus-signals-stale`, then route their grouped webhooks to a separately deployed
|
|
293
|
+
`alert-mqtt-bridge`. The agent does not create Grafana rules, contact points or
|
|
294
|
+
recipient credentials automatically.
|
|
295
|
+
|
|
296
|
+
| Channel | Path | Installation prerequisite |
|
|
297
|
+
|---|---|---|
|
|
298
|
+
| MQTT banners | Grafana → `alert-mqtt-bridge` → `inverter/notifications` | Broker access, contact point and dashboard consumer |
|
|
299
|
+
| Email | Grafana → `alert-mqtt-bridge` → SMTP relay with STARTTLS | Relay account and verified sender/recipient configuration |
|
|
300
|
+
| Telegram | Grafana → `alert-mqtt-bridge` → Bot API | Bot token and an authorized chat |
|
|
301
|
+
|
|
302
|
+
These paths describe supported integrations, not a current delivery-status
|
|
303
|
+
monitor. The installation notes originally recorded successful email and
|
|
304
|
+
Telegram delivery on 2026-08-24; repeat firing and resolved delivery checks after
|
|
305
|
+
changes. See [evidence and limits](docs/deployment-evidence.md).
|
|
306
|
+
|
|
307
|
+
> **Secrets rule:** real tokens and passwords never go into this repository.
|
|
308
|
+
> README examples keep `<placeholders>`; actual values live in
|
|
309
|
+
> a protected deployment secret store or mode-600 environment file outside Git.
|
|
310
|
+
> For a local working copy use a `*.local` file — already covered by `.gitignore`.
|
|
311
|
+
|
|
312
|
+
### Grouped delivery and query failures
|
|
313
|
+
|
|
314
|
+
The bridge preserves Grafana webhook grouping: one email and one Telegram
|
|
315
|
+
message per payload, with individual MQTT banners for independent problems.
|
|
316
|
+
`DatasourceError` and `DatasourceNoData` instances from the same source and
|
|
317
|
+
status become one notification naming the data source, affected checks, and
|
|
318
|
+
query error. They no longer reuse the underlying rules' failure summaries.
|
|
319
|
+
Recovery notifications explicitly say `Resolved`; a mixed group still includes
|
|
320
|
+
both active and recovered problems.
|
|
321
|
+
|
|
322
|
+
Notification routing and Prometheus endpoints are deployment-specific. Keep site
|
|
323
|
+
configuration and operational evidence outside this public repository. Configure
|
|
324
|
+
Grafana grouping, pending periods and reminder intervals for the installation,
|
|
325
|
+
and keep query errors separate from application alerts. Do not change an error
|
|
326
|
+
state to Normal merely to silence a broken monitoring pipeline.
|
|
327
|
+
The bridge does not impose an additional cooldown that could hide a new incident.
|
|
328
|
+
|
|
329
|
+
### Channel 1: MQTT banners
|
|
330
|
+
|
|
331
|
+
Source: [`alert-mqtt-bridge/`](alert-mqtt-bridge/). Deploy the relay, configure its
|
|
332
|
+
broker connection, and create a Grafana webhook contact point for it. Attach that
|
|
333
|
+
contact point to the intended notification policies. Verify both firing and
|
|
334
|
+
resolved messages in the dashboard; rules outside those policies are not routed
|
|
335
|
+
automatically.
|
|
336
|
+
|
|
337
|
+
### Channel 2: Telegram
|
|
338
|
+
|
|
339
|
+
#### 2.1 Create the bot with @BotFather
|
|
340
|
+
|
|
341
|
+
1. In Telegram, open **@BotFather** (verified badge) and send `/newbot`.
|
|
342
|
+
2. Answer two prompts:
|
|
343
|
+
- **name** — display name, anything, e.g. `Venus Alerts`
|
|
344
|
+
- **username** — choose an available username ending in `bot`
|
|
345
|
+
3. BotFather replies with the **token**, format:
|
|
346
|
+
|
|
347
|
+
```
|
|
348
|
+
1234567890:AAHfJ8xQm3k9...your-token-here
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
Keep it secret: anyone with this token controls your bot.
|
|
352
|
+
|
|
353
|
+
Useful extra commands later: `/mybots` → edit name/description,
|
|
354
|
+
`/revoke` → invalidate and reissue a token, `/deletebot`.
|
|
355
|
+
|
|
356
|
+
#### 2.2 Get your chat id
|
|
357
|
+
|
|
358
|
+
A bot cannot message you first — start the chat yourself:
|
|
359
|
+
|
|
360
|
+
1. Open your new bot in Telegram and press **Start** (send any message).
|
|
361
|
+
2. Then fetch recent updates (run anywhere with internet):
|
|
362
|
+
|
|
363
|
+
```bash
|
|
364
|
+
curl -s https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
3. Find your id in the response:
|
|
368
|
+
|
|
369
|
+
```json
|
|
370
|
+
{"result":[{"message":{"chat":{"id":987654321,"first_name":"..."}}}]}
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
That `id` number is your **chat id**. Alternative: send any message to
|
|
374
|
+
**@userinfobot** and it replies with your id.
|
|
375
|
+
|
|
376
|
+
For a **group chat**: add the bot to the group, post one message there, then call
|
|
377
|
+
`getUpdates` again — the group's `chat.id` is negative (e.g. `-1001234567890`);
|
|
378
|
+
use it as-is.
|
|
379
|
+
|
|
380
|
+
#### 2.3 Wire the bridge
|
|
381
|
+
|
|
382
|
+
Configure Telegram delivery in the bridge environment file. Keep that file
|
|
383
|
+
outside Git with mode 600:
|
|
384
|
+
|
|
385
|
+
```
|
|
386
|
+
TG_BOT_TOKEN=<your-bot-token>
|
|
387
|
+
TG_CHAT_ID=<your-chat-id>
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
(`TG_CHAT_ID` accepts a comma-separated list for group chats.) Then rebuild
|
|
391
|
+
and recreate the container per **Channel 3 → 3.2**. Each webhook group goes to
|
|
392
|
+
MQTT banners plus one email/Telegram digest,
|
|
393
|
+
including firing and recovery states.
|
|
394
|
+
|
|
395
|
+
#### 2.5 Verify end-to-end
|
|
396
|
+
|
|
397
|
+
First use a Grafana test notification to check the configured route without
|
|
398
|
+
interrupting monitoring. Confirm both the recipient and dashboard receive it;
|
|
399
|
+
this tests delivery, not the agent-down rule. To test that rule during an agreed
|
|
400
|
+
maintenance window, temporarily stop the agent on the intended Venus host:
|
|
401
|
+
`venus-agent-down` should fire after its configured `for` window. Resume the agent
|
|
402
|
+
and verify the resolved message even if delivery fails:
|
|
403
|
+
|
|
404
|
+
```bash
|
|
405
|
+
ssh root@cerbo 'svc -d /service/venus-os-observability' # pause
|
|
406
|
+
# wait ~2-3 min: Telegram message + dashboard banner appear
|
|
407
|
+
ssh root@cerbo 'svc -u /service/venus-os-observability' # resume -> resolved message
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
### Channel 3: Email via alert-mqtt-bridge + external SMTP relay
|
|
411
|
+
|
|
412
|
+
In the reference installation's 2026-08-24 notes, outbound port 25 was blocked,
|
|
413
|
+
a usable PTR record was unavailable, and its Grafana-native email contact point
|
|
414
|
+
produced no SMTP attempt or provider event. That installation used the bridge
|
|
415
|
+
with an external STARTTLS relay. These are historical site observations, not a
|
|
416
|
+
claim about every ISP, Grafana version or current deployment. The configured path is:
|
|
417
|
+
|
|
418
|
+
```mermaid
|
|
419
|
+
flowchart LR
|
|
420
|
+
G[Grafana rule fires] -->|webhook| B[alert-mqtt-bridge]
|
|
421
|
+
B -->|MQTT| N["inverter/notifications<br/>desktop banners"]
|
|
422
|
+
B -->|SMTP :587| M[Brevo → mailbox]
|
|
423
|
+
B -->|Bot API| T[Telegram]
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
#### 3.1 Provider setup (Brevo, one-time)
|
|
427
|
+
|
|
428
|
+
1. Create an account at <https://www.brevo.com> or use another SMTP relay
|
|
429
|
+
that supports STARTTLS. Check the provider's current limits and requirements.
|
|
430
|
+
2. Verify a sender address or a domain you control using the provider's
|
|
431
|
+
instructions. Complete any required DNS authentication before sending alerts.
|
|
432
|
+
3. Create SMTP credentials and record the relay hostname, port and login in
|
|
433
|
+
the private bridge environment file.
|
|
434
|
+
|
|
435
|
+
#### 3.2 Bridge configuration
|
|
436
|
+
|
|
437
|
+
Create a private environment file outside the repository (mode 600). The
|
|
438
|
+
following values illustrate a Brevo relay; use your provider's settings:
|
|
439
|
+
|
|
440
|
+
```
|
|
441
|
+
SMTP_HOST=smtp-relay.brevo.com
|
|
442
|
+
SMTP_PORT=587
|
|
443
|
+
SMTP_USER=<login>@smtp-brevo.com
|
|
444
|
+
SMTP_PASSWORD=<secret-key>
|
|
445
|
+
SMTP_FROM=<verified-sender@example.com>
|
|
446
|
+
SMTP_TO=<recipient@example.com>
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
`SMTP_HOST` or `SMTP_TO` empty ⇒ email channel disabled, MQTT-only.
|
|
450
|
+
|
|
451
|
+
Email requires STARTTLS and a certificate valid for `SMTP_HOST`, signed by a
|
|
452
|
+
trusted CA. The bridge verifies the server before sending SMTP credentials or
|
|
453
|
+
the alert. A missing STARTTLS extension, untrusted certificate or hostname
|
|
454
|
+
mismatch fails this delivery attempt and is logged; it never falls back to
|
|
455
|
+
plaintext SMTP. If using a private CA, mount its CA bundle and set
|
|
456
|
+
`SSL_CERT_FILE` to that file inside the container. Existing relays with invalid
|
|
457
|
+
certificates must be corrected or given their proper CA trust before upgrading.
|
|
458
|
+
|
|
459
|
+
Build from the repository root. Set `BRIDGE_ENV_FILE` to the private file,
|
|
460
|
+
`GRAFANA_NETWORK` to a Docker network shared with Grafana, and `MQTT_HOST` to
|
|
461
|
+
the broker reachable from that network. For an update, preserve the existing
|
|
462
|
+
image/configuration, then stop and remove the old container before starting another
|
|
463
|
+
with the same name. A stopped container still reserves its name.
|
|
464
|
+
|
|
465
|
+
```bash
|
|
466
|
+
: "${BRIDGE_ENV_FILE:?Set the path to the private environment file}"
|
|
467
|
+
: "${GRAFANA_NETWORK:?Set the Docker network shared with Grafana}"
|
|
468
|
+
: "${MQTT_HOST:?Set the MQTT broker hostname}"
|
|
469
|
+
sudo docker build -t venus-alert-bridge:latest ./alert-mqtt-bridge
|
|
470
|
+
sudo docker run -d --name venus-alert-bridge --restart unless-stopped \
|
|
471
|
+
--network "$GRAFANA_NETWORK" --network-alias venus-alert-bridge \
|
|
472
|
+
-e MQTT_HOST="$MQTT_HOST" --env-file "$BRIDGE_ENV_FILE" venus-alert-bridge:latest
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
Configure a Grafana webhook contact point for the bridge and route the intended
|
|
476
|
+
notification policies to it. This route does not need a separate Grafana Email
|
|
477
|
+
contact point; it sends email through the bridge's SMTP configuration.
|
|
478
|
+
|
|
479
|
+
#### 3.3 Verify
|
|
480
|
+
|
|
481
|
+
1. Check a test notification first; test actual firing and recovery during a
|
|
482
|
+
maintenance window as described in channel 2.5.
|
|
483
|
+
2. Bridge log shows `Email sent: <alertname>`; check inbox AND spam folder.
|
|
484
|
+
3. Check the selected SMTP provider's delivery-events page, when available, for
|
|
485
|
+
per-message status (delivered / blocked / rejected).
|
|
486
|
+
4. Optional scoring: send a test via the same relay to the address shown at
|
|
487
|
+
<https://www.mail-tester.com>; aim for ≥9/10.
|
|
488
|
+
|
|
489
|
+
## Development
|
|
490
|
+
|
|
491
|
+
```bash
|
|
492
|
+
# Create a locked, isolated check environment
|
|
493
|
+
bash scripts/ci.sh --install
|
|
494
|
+
|
|
495
|
+
# Lint, types and tests
|
|
496
|
+
bash scripts/ci.sh
|
|
497
|
+
```
|
|
498
|
+
|
|
499
|
+
## License
|
|
500
|
+
|
|
501
|
+
MIT
|
|
502
|
+
|
|
503
|
+
---
|
|
504
|
+
|
|
505
|
+
## Related Projects
|
|
506
|
+
|
|
507
|
+
| Project | Scope | When to Use |
|
|
508
|
+
|---------|-------|-------------|
|
|
509
|
+
| [mqtt-observability-opentelemetry](https://github.com/4alvit/mqtt-observability-opentelemetry) | **Generic** — Works with ANY MQTT broker. No Venus OS dependency. | Generic MQTT/IoT observability, any broker, any device types |
|
|
510
|
+
| **venus-os-observability** (this) | **Venus OS specific** — Depends on D-Bus, Victron protocols. | Victron Venus OS only: D-Bus event tracing, inverter metrics, Cerbo GX integration |
|
|
511
|
+
|
|
512
|
+
**Choose mqtt-observability-opentelemetry if:** You need MQTT observability for any IoT system (industrial, home automation, custom devices).
|
|
513
|
+
|
|
514
|
+
**Choose this repo if:** You are running Victron Venus OS (Cerbo GX, Raspberry Pi with Venus OS) and need D-Bus integration, inverter-specific metrics, and Venus OS native deployment.
|
|
515
|
+
|
|
516
|
+
## Contributing and security
|
|
517
|
+
|
|
518
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for reports, development checks and pull requests,
|
|
519
|
+
[SECURITY.md](SECURITY.md) for private vulnerability reporting and deployment trust boundaries,
|
|
520
|
+
[GOVERNANCE.md](GOVERNANCE.md) for responsibilities and continuity limits,
|
|
521
|
+
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for participation,
|
|
522
|
+
[ROADMAP.md](ROADMAP.md) for priorities,
|
|
523
|
+
and the [OpenSSF evidence index](docs/openssf-evidence.md) for assessment references and remaining verification.
|
|
524
|
+
|
|
525
|
+
The alert relay verifies exact certificate key sizes for SMTP STARTTLS and Telegram HTTPS.
|
|
526
|
+
See [alert relay TLS policy](docs/alert-relay-tls.md) for trusted CA, proxy and migration requirements.
|
|
527
|
+
|
|
528
|
+
## Release-candidate fuzzing
|
|
529
|
+
|
|
530
|
+
The real correlation-context and header propagation functions receive synthetic identifiers, including Unicode and punctuation. Header extraction must preserve explicitly supplied IDs and context tokens must restore prior state. No trace exporter, D-Bus subscription or network connection starts.
|
|
531
|
+
|
|
532
|
+
[RC-only fuzzing policy, local reproduction and failure handling](docs/fuzzing.md).
|