venus-os-observability 0.1.11__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (34) hide show
  1. venus_os_observability-0.1.11/LICENSE +21 -0
  2. venus_os_observability-0.1.11/PKG-INFO +532 -0
  3. venus_os_observability-0.1.11/README.md +480 -0
  4. venus_os_observability-0.1.11/pyproject.toml +120 -0
  5. venus_os_observability-0.1.11/setup.cfg +4 -0
  6. venus_os_observability-0.1.11/setup.py +6 -0
  7. venus_os_observability-0.1.11/src/venus_observability/__init__.py +39 -0
  8. venus_os_observability-0.1.11/src/venus_observability/__main__.py +286 -0
  9. venus_os_observability-0.1.11/src/venus_observability/correlation.py +340 -0
  10. venus_os_observability-0.1.11/src/venus_observability/dbus_listener.py +365 -0
  11. venus_os_observability-0.1.11/src/venus_observability/dbus_owners.py +221 -0
  12. venus_os_observability-0.1.11/src/venus_observability/metrics.py +367 -0
  13. venus_os_observability-0.1.11/src/venus_observability/otlp.py +242 -0
  14. venus_os_observability-0.1.11/src/venus_observability/py.typed +0 -0
  15. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/PKG-INFO +532 -0
  16. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/SOURCES.txt +32 -0
  17. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/dependency_links.txt +1 -0
  18. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/entry_points.txt +2 -0
  19. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/requires.txt +42 -0
  20. venus_os_observability-0.1.11/src/venus_os_observability.egg-info/top_level.txt +1 -0
  21. venus_os_observability-0.1.11/tests/test_alert_bridge.py +219 -0
  22. venus_os_observability-0.1.11/tests/test_correlation.py +352 -0
  23. venus_os_observability-0.1.11/tests/test_dbus_listener.py +421 -0
  24. venus_os_observability-0.1.11/tests/test_installer.py +83 -0
  25. venus_os_observability-0.1.11/tests/test_invalid_dbus_batch.py +43 -0
  26. venus_os_observability-0.1.11/tests/test_main.py +414 -0
  27. venus_os_observability-0.1.11/tests/test_metrics.py +193 -0
  28. venus_os_observability-0.1.11/tests/test_otlp.py +167 -0
  29. venus_os_observability-0.1.11/tests/test_otlp_tls.py +688 -0
  30. venus_os_observability-0.1.11/tests/test_owner_lifecycle.py +323 -0
  31. venus_os_observability-0.1.11/tests/test_owner_metric_invalidation.py +203 -0
  32. venus_os_observability-0.1.11/tests/test_relay_tls.py +429 -0
  33. venus_os_observability-0.1.11/tests/test_setuphelper_archive.py +234 -0
  34. venus_os_observability-0.1.11/tests/test_smtp_tls.py +201 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 victron-venus
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,532 @@
1
+ Metadata-Version: 2.4
2
+ Name: venus-os-observability
3
+ Version: 0.1.11
4
+ Summary: OpenTelemetry/Prometheus observability for Venus OS - D-Bus event tracing, inverter metrics, distributed tracing
5
+ License: MIT
6
+ Project-URL: Homepage, https://github.com/victron-venus/venus-os-observability
7
+ Project-URL: Repository, https://github.com/victron-venus/venus-os-observability
8
+ Project-URL: Issues, https://github.com/victron-venus/venus-os-observability/issues
9
+ Requires-Python: <3.13,>=3.12
10
+ Description-Content-Type: text/markdown
11
+ License-File: LICENSE
12
+ Requires-Dist: opentelemetry-api<1.29,>=1.28
13
+ Requires-Dist: opentelemetry-sdk<1.29,>=1.28
14
+ Requires-Dist: opentelemetry-exporter-prometheus<0.57b0,>=0.49b2
15
+ Requires-Dist: opentelemetry-instrumentation<0.50,>=0.49b0
16
+ Requires-Dist: prometheus-client>=0.19
17
+ Requires-Dist: paho-mqtt>=2.0
18
+ Requires-Dist: dbus-next>=0.2
19
+ Requires-Dist: pyyaml>=6.0
20
+ Requires-Dist: structlog>=24.0
21
+ Requires-Dist: click>=8.1
22
+ Provides-Extra: dev
23
+ Requires-Dist: types-atheris>=3.1.0.20260630; extra == "dev"
24
+ Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "dev"
25
+ Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "dev"
26
+ Requires-Dist: requests>=2.34.2; extra == "dev"
27
+ Requires-Dist: cryptography>=50.0.2; extra == "dev"
28
+ Requires-Dist: pytest>=8.0; extra == "dev"
29
+ Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
30
+ Requires-Dist: pytest-cov>=5.0; extra == "dev"
31
+ Requires-Dist: ruff>=0.5; extra == "dev"
32
+ Requires-Dist: mypy>=1.10; extra == "dev"
33
+ Requires-Dist: pre-commit>=3.7; extra == "dev"
34
+ Requires-Dist: types-setuptools>=75; extra == "dev"
35
+ Provides-Extra: tempo
36
+ Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "tempo"
37
+ Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "tempo"
38
+ Requires-Dist: requests>=2.34.2; extra == "tempo"
39
+ Requires-Dist: cryptography>=50.0.2; extra == "tempo"
40
+ Provides-Extra: all
41
+ Requires-Dist: requests>=2.34.2; extra == "all"
42
+ Requires-Dist: cryptography>=50.0.2; extra == "all"
43
+ Requires-Dist: pytest>=8.0; extra == "all"
44
+ Requires-Dist: pytest-asyncio>=0.23; extra == "all"
45
+ Requires-Dist: pytest-cov>=5.0; extra == "all"
46
+ Requires-Dist: ruff>=0.5; extra == "all"
47
+ Requires-Dist: mypy>=1.10; extra == "all"
48
+ Requires-Dist: pre-commit>=3.7; extra == "all"
49
+ Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<1.29,>=1.28; extra == "all"
50
+ Requires-Dist: opentelemetry-exporter-otlp-proto-http<1.29,>=1.28; extra == "all"
51
+ Dynamic: license-file
52
+
53
+ # Venus OS Observability
54
+
55
+ [![Fuzzing: RC only](https://img.shields.io/badge/Fuzzing-RC%20only-blue)](docs/fuzzing.md)
56
+ [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/15457/badge)](https://www.bestpractices.dev/projects/15457)
57
+
58
+ OpenTelemetry/Prometheus observability for Victron Venus OS: D-Bus signal tracing, inverter metrics export, optional application correlation helpers, and a separate Grafana alert relay.
59
+
60
+ [![Grafana dashboard layout with synthetic demonstration data; this is not live installation telemetry. Click for the original screenshot.](docs/images/app-hero.png)](docs/images/venus-demo.jpg)
61
+
62
+ *Grafana dashboard layout with synthetic demonstration data; this is not live installation telemetry. Click for the original screenshot.*
63
+
64
+ [Image source and regeneration](docs/images/README.md).
65
+
66
+ ## Python runtime
67
+
68
+ Native Venus OS packages target **Python 3.12.x**. The audited Raspberry Pi 3
69
+ running Venus OS v3.75 reports Python **3.12.13**; the [official Venus OS v3.79 manifest](https://updates.victronenergy.com/feeds/venus/release/sdk/venus-scarthgap-x86_64-arm-cortexa8hf-neon-toolchain-v3.79.target.manifest)
70
+ also ships 3.12.13. Local development and CI use `.python-version` / Python
71
+ 3.12.13. Package metadata accepts 3.12 patch updates and rejects other minor
72
+ versions until they have been validated. Use the firmware's system interpreter
73
+ and its matching D-Bus/GI libraries on the device; do not replace the OS Python.
74
+
75
+ <!-- ci-release-process:start -->
76
+ ## Release process
77
+
78
+ See the [release strategy](RELEASING.md) for validation, nightly, beta, RC and stable promotion rules, and the [operator runbook](docs/release-workflow.md) for local commands.
79
+ <!-- ci-release-process:end -->
80
+
81
+ ## Architecture
82
+
83
+ ```mermaid
84
+ graph LR
85
+ A[Venus D-Bus services] -->|ItemsChanged| B[D-Bus Signal Listener]
86
+ B --> C[Metrics and optional spans]
87
+ C --> D[Prometheus Exporter :9090/metrics]
88
+ C -->|Configured OTLP endpoint| E[Collector / Grafana Tempo]
89
+ D --> P[Prometheus / Grafana]
90
+ P -->|Configured alert webhook| R[Separate alert-mqtt-bridge]
91
+ R --> M[MQTT banners / SMTP / Telegram]
92
+ ```
93
+
94
+ D-Bus observations feed metrics and optional spans. Prometheus reads the metrics; configured collectors receive spans. The separate relay receives Grafana alerts and forwards notifications to MQTT and optional email/Telegram destinations. The arrows above describe observation and notification, not physical control.
95
+
96
+ ## Features
97
+
98
+ - **D-Bus Signal Tracing**: Observe Victron `ItemsChanged` signals; record spans when tracing is configured
99
+ - **Prometheus Metrics**: Export inverter state (SOC, power, grid, battery) as Prometheus metrics
100
+ - **Correlation Helpers**: Explicit MQTT/application instrumentation helpers; installing the agent does not instrument other services or propagate context across native D-Bus automatically
101
+ - **Grafana Tempo Integration**: Visualize trace timelines in Grafana
102
+ - **Native Venus OS Service**: Supports native supervision; offline pip installation requires wheels matching the target Python version and architecture
103
+
104
+ Explicitly unavailable numeric D-Bus values, including Venus empty arrays, are
105
+ exported as `NaN` in both metric backends. They replace the previous reading and
106
+ do not discard other items in the same signal. A later valid zero is retained as
107
+ zero. Text-only updates do not replace numeric measurements. This handles value
108
+ invalidation; it does not independently detect every silent source outage.
109
+
110
+ ## Deployment evidence
111
+
112
+ The [verification record and acceptance checklist](docs/deployment-evidence.md)
113
+ separate historical runtime observations from source tests and current deployment
114
+ checks. A release does not establish that its artifact is installed or that every
115
+ configured alert channel reaches a recipient.
116
+
117
+ ## Installation
118
+
119
+ ### Option 1: SetupHelper / PackageManager (Recommended)
120
+
121
+ **Preferred install path on Cerbo GX / Venus OS** — same as `dbus-ev` and `inverter-control`. Requires [SetupHelper](https://github.com/kwindrem/SetupHelper) on the GX.
122
+
123
+ The easiest way to install is via SetupHelper PackageManager. The `setup` script is PackageManager-compatible and handles service creation, `/data/rc.local` boot persistence, and restarts.
124
+
125
+ Package files needed at `/data/venus-os-observability`: `version`, `setup`, `gitHubInfo`, `update.sh`, `src/`, `services/`.
126
+
127
+ 1. **Add package via GUI** (GUI v1):
128
+ - Settings → PackageManager → Inactive packages → **new**
129
+ - Package name: `venus-os-observability`
130
+ - GitHub user: `victron-venus`
131
+ - Branch / tag: `latest` (matches `gitHubInfo`)
132
+ - Proceed → Download → Install
133
+
134
+ 2. **Or on the device**:
135
+
136
+ ```sh
137
+ /data/venus-os-observability/setup install
138
+ /data/venus-os-observability/setup uninstall
139
+ ```
140
+
141
+ `gitHubInfo` is `victron-venus:latest`.
142
+
143
+ `update.sh` (also called by `setup install`):
144
+
145
+ - Installs into **`/data/venus-os-observability`** (SetupHelper convention).
146
+ - Copies daemontools service to `INSTALL_DIR/service/venus-os-observability/` and symlinks `/service/venus-os-observability`.
147
+ - Writes an idempotent **`/data/rc.local`** marker block so the symlink is recreated after reboot (`/service` is tmpfs). Venus OS does **not** run `/data/rc/S99*`.
148
+ - Migrates `.venv2` from a legacy `/data/opt/venus-os-observability` install when present.
149
+ - Removes the dead `/data/rc/S99venus-os-observability.sh` hook if found.
150
+
151
+ Manual refresh from a checkout on the Mac:
152
+
153
+ ```sh
154
+ # stream repo → device, run update.sh (same pattern as dbus-ev deploy.sh)
155
+ COPYFILE_DISABLE=1 tar --no-xattrs \
156
+ --exclude='.git' --exclude='.venv' --exclude='__pycache__' \
157
+ --exclude='.pytest_cache' --exclude='.ruff_cache' --exclude='logs' \
158
+ --exclude='*.egg-info' --exclude='build' --exclude='dist' --exclude='wheels' \
159
+ -czf - -C /path/to/venus-os-observability . \
160
+ | ssh root@cerbo 'set -e; rm -rf /data/.vos-obs-deploy; mkdir -p /data/.vos-obs-deploy; \
161
+ tar -xz -C /data/.vos-obs-deploy; sh /data/.vos-obs-deploy/update.sh; \
162
+ rm -rf /data/.vos-obs-deploy; svstat /service/venus-os-observability'
163
+ ```
164
+
165
+ Runtime prefers `.venv2/bin/python`, then `.venv/bin/python`, with
166
+ `PYTHONPATH=/data/venus-os-observability/src` so source updates take effect.
167
+ Installation checks imports before stopping the service, preserves supervisor
168
+ directories, and never kills processes merely because their working directory
169
+ is the package directory. Only `/data/venus-os-observability` is supported.
170
+
171
+ The boot hook is inserted before a conventional `exit 0` in `/data/rc.local`.
172
+ `/service` is volatile; `/data/rc/S99*` is not a supported boot hook.
173
+ Logs use `multilog t s25000 n4` under `/var/log`, approximately 125 KB including
174
+ the current file, rather than an unbounded file on persistent flash.
175
+
176
+ #### Offline wheel bootstrap (first installation)
177
+
178
+ Check `python3 --version` and `uname -m` on the target first. The audited Cerbo
179
+ uses Python 3.12 and ARMv7; Raspberry Pi images may differ. Build the wheel bundle
180
+ with `scripts/build_wheels.sh` on a matching Linux architecture/Python ABI.
181
+ It resolves versions from `pyproject.toml`, requires binary dependency wheels,
182
+ and includes a pip wheel. It is not an ARM cross-compiler. If a native wheel is
183
+ unavailable, build it in a matching build environment before deployment.
184
+
185
+ Venus OS provides `dbus-python` and GLib. Preserve access to these native bindings
186
+ using `--system-site-packages`; `dbus-next` cannot replace them for this agent.
187
+ Do not install replacement bindings over the firmware's Python packages.
188
+
189
+ Copy and extract the bundle under `/data/venus-os-observability/wheels`, then:
190
+
191
+ ```sh
192
+ cd /data/venus-os-observability
193
+ python3 -m venv --system-site-packages --without-pip .venv2
194
+ # pip can run directly from its wheel, even when ensurepip is absent.
195
+ for wheel in wheels/pip-*.whl; do
196
+ PYTHONPATH="$wheel" .venv2/bin/python -m pip install \
197
+ --no-index --find-links=wheels venus-os-observability
198
+ done
199
+ sh update.sh
200
+ svstat /service/venus-os-observability /service/venus-os-observability/log
201
+ tail -n 60 /var/log/venus-os-observability/current
202
+ wget -qO - http://127.0.0.1:9090/metrics
203
+ ```
204
+
205
+ If the image lacks `venv` itself, prepare a compatible offline virtualenv
206
+ bootstrap first; the installer does not silently fetch packages from the
207
+ internet. Repeat the bootstrap after firmware changes the Python ABI.
208
+
209
+ Tracing is optional: install the `tempo` extra from a matching offline bundle
210
+ and configure `OTEL_EXPORTER_OTLP_ENDPOINT` only when the device has headroom.
211
+ Without an endpoint, the sampler drops spans and avoids D-Bus payload serialization;
212
+ metrics remain active. D-Bus string subclasses are converted to native strings
213
+ before recorded span attributes are validated.
214
+
215
+ Local plaintext gRPC remains the default. For HTTPS collectors, select
216
+ `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` and configure the collector's
217
+ OTLP/HTTP endpoint; HTTPS gRPC is rejected because its native TLS backend does
218
+ not enforce the required minimum certificate key strengths. See
219
+ [OTLP transport and migration](docs/otlp-transport.md) before changing an
220
+ existing deployment. No protocol, port or TLS configuration is changed silently.
221
+
222
+ Metric labels use stable Venus service names. The exporter subscribes to
223
+ `NameOwnerChanged` before asynchronous startup discovery, removes superseded
224
+ owners and invalidates that service's previously published gauges on loss or
225
+ replacement. Fresh measurements restore those gauges; stable counters remain
226
+ cumulative. A reconnect of the same well-known service does not create new
227
+ per-process metric series. Adding genuinely new service names or item paths can
228
+ still create new series; this is not a global registry quota.
229
+
230
+ Signals received before their sender is resolved are replayed in arrival order
231
+ once the Venus service is known. Unresolved input is limited to 256 batches and
232
+ 4,096 items, with a 10-second monotonic expiry. Overflow removes the oldest
233
+ unresolved batches; an oversized single batch is discarded. The fixed-reason
234
+ `victron_dbus_unresolved_signals_dropped_total` counter exposes overflow/expiry,
235
+ and warnings are limited to one per minute. Known services bypass this queue.
236
+ Owner discovery uses asynchronous calls with five-second deadlines; failed
237
+ discovery can retry after five seconds, while other unknown-sender refreshes
238
+ are limited to once per minute. It does not perform a blocking scan per signal.
239
+
240
+ The historical IPK Makefile was an OpenWrt/systemd recipe, not a working Venus
241
+ OS package. It now fails with an explicit migration message. The `systemd/`
242
+ example is for separate Linux hosts only; native GX deployment uses SetupHelper.
243
+
244
+ Release candidates provide `venus-os-observability-VERSION.tar.gz` and
245
+ `SHA256SUMS`. This archive contains the native installer, all runtime modules,
246
+ service scripts, example configuration and Python package metadata. CI validates
247
+ its contents, executable modes and matching package versions before publication.
248
+ Extract its single top-level directory into a staging directory and run its
249
+ `update.sh` to update the standard `/data/venus-os-observability` installation.
250
+ Device configuration, virtual environments and live supervisor directories are
251
+ preserved. Dependency wheels are a separate prerequisite; neither this source
252
+ archive nor the Python wheel is an IPK package.
253
+
254
+ Wheel and source distributions accompany the verified GitHub release assets.
255
+ PyPI publication requires a separate explicit promotion of those same verified
256
+ distributions, as described in [RELEASING.md](RELEASING.md). A GitHub release alone
257
+ does not imply that the same version is available from PyPI.
258
+
259
+ ### Option 2: Docker Compose
260
+
261
+ For non-GX hosts or a containerized stack (not the preferred Cerbo path):
262
+
263
+ ```yaml
264
+ services:
265
+ venus-observability:
266
+ image: ghcr.io/victron-venus/venus-os-observability:latest
267
+ environment:
268
+ - DBUS_SYSTEM_BUS_ADDRESS=unix:path=/run/dbus/system_bus_socket
269
+ - OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317
270
+ - PROMETHEUS_PORT=9090
271
+ volumes:
272
+ - /run/dbus:/run/dbus
273
+ ports:
274
+ - "9090:9090"
275
+ restart: unless-stopped
276
+ ```
277
+
278
+ ## Metrics Exported
279
+
280
+ | Metric | Type | Description |
281
+ |--------|------|-------------|
282
+ | `victron_battery_soc` | Gauge | Battery state of charge % |
283
+ | `victron_battery_power` | Gauge | Battery charge/discharge power (W) |
284
+ | `victron_pv_power` | Gauge | Total PV production (W) |
285
+ | `victron_grid_power` | Gauge | Grid import/export (W) |
286
+ | `victron_ac_loads` | Gauge | AC loads consumption (W) |
287
+ | `victron_inverter_state` | Gauge | Inverter state machine (0=off,1=on,2=charging,3=inverting) |
288
+
289
+ ## Alert Delivery Setup
290
+
291
+ Configure Grafana alert rules such as `venus-agent-down`, `venus-dbus-errors` and
292
+ `venus-signals-stale`, then route their grouped webhooks to a separately deployed
293
+ `alert-mqtt-bridge`. The agent does not create Grafana rules, contact points or
294
+ recipient credentials automatically.
295
+
296
+ | Channel | Path | Installation prerequisite |
297
+ |---|---|---|
298
+ | MQTT banners | Grafana → `alert-mqtt-bridge` → `inverter/notifications` | Broker access, contact point and dashboard consumer |
299
+ | Email | Grafana → `alert-mqtt-bridge` → SMTP relay with STARTTLS | Relay account and verified sender/recipient configuration |
300
+ | Telegram | Grafana → `alert-mqtt-bridge` → Bot API | Bot token and an authorized chat |
301
+
302
+ These paths describe supported integrations, not a current delivery-status
303
+ monitor. The installation notes originally recorded successful email and
304
+ Telegram delivery on 2026-08-24; repeat firing and resolved delivery checks after
305
+ changes. See [evidence and limits](docs/deployment-evidence.md).
306
+
307
+ > **Secrets rule:** real tokens and passwords never go into this repository.
308
+ > README examples keep `<placeholders>`; actual values live in
309
+ > a protected deployment secret store or mode-600 environment file outside Git.
310
+ > For a local working copy use a `*.local` file — already covered by `.gitignore`.
311
+
312
+ ### Grouped delivery and query failures
313
+
314
+ The bridge preserves Grafana webhook grouping: one email and one Telegram
315
+ message per payload, with individual MQTT banners for independent problems.
316
+ `DatasourceError` and `DatasourceNoData` instances from the same source and
317
+ status become one notification naming the data source, affected checks, and
318
+ query error. They no longer reuse the underlying rules' failure summaries.
319
+ Recovery notifications explicitly say `Resolved`; a mixed group still includes
320
+ both active and recovered problems.
321
+
322
+ Notification routing and Prometheus endpoints are deployment-specific. Keep site
323
+ configuration and operational evidence outside this public repository. Configure
324
+ Grafana grouping, pending periods and reminder intervals for the installation,
325
+ and keep query errors separate from application alerts. Do not change an error
326
+ state to Normal merely to silence a broken monitoring pipeline.
327
+ The bridge does not impose an additional cooldown that could hide a new incident.
328
+
329
+ ### Channel 1: MQTT banners
330
+
331
+ Source: [`alert-mqtt-bridge/`](alert-mqtt-bridge/). Deploy the relay, configure its
332
+ broker connection, and create a Grafana webhook contact point for it. Attach that
333
+ contact point to the intended notification policies. Verify both firing and
334
+ resolved messages in the dashboard; rules outside those policies are not routed
335
+ automatically.
336
+
337
+ ### Channel 2: Telegram
338
+
339
+ #### 2.1 Create the bot with @BotFather
340
+
341
+ 1. In Telegram, open **@BotFather** (verified badge) and send `/newbot`.
342
+ 2. Answer two prompts:
343
+ - **name** — display name, anything, e.g. `Venus Alerts`
344
+ - **username** — choose an available username ending in `bot`
345
+ 3. BotFather replies with the **token**, format:
346
+
347
+ ```
348
+ 1234567890:AAHfJ8xQm3k9...your-token-here
349
+ ```
350
+
351
+ Keep it secret: anyone with this token controls your bot.
352
+
353
+ Useful extra commands later: `/mybots` → edit name/description,
354
+ `/revoke` → invalidate and reissue a token, `/deletebot`.
355
+
356
+ #### 2.2 Get your chat id
357
+
358
+ A bot cannot message you first — start the chat yourself:
359
+
360
+ 1. Open your new bot in Telegram and press **Start** (send any message).
361
+ 2. Then fetch recent updates (run anywhere with internet):
362
+
363
+ ```bash
364
+ curl -s https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates
365
+ ```
366
+
367
+ 3. Find your id in the response:
368
+
369
+ ```json
370
+ {"result":[{"message":{"chat":{"id":987654321,"first_name":"..."}}}]}
371
+ ```
372
+
373
+ That `id` number is your **chat id**. Alternative: send any message to
374
+ **@userinfobot** and it replies with your id.
375
+
376
+ For a **group chat**: add the bot to the group, post one message there, then call
377
+ `getUpdates` again — the group's `chat.id` is negative (e.g. `-1001234567890`);
378
+ use it as-is.
379
+
380
+ #### 2.3 Wire the bridge
381
+
382
+ Configure Telegram delivery in the bridge environment file. Keep that file
383
+ outside Git with mode 600:
384
+
385
+ ```
386
+ TG_BOT_TOKEN=<your-bot-token>
387
+ TG_CHAT_ID=<your-chat-id>
388
+ ```
389
+
390
+ (`TG_CHAT_ID` accepts a comma-separated list for group chats.) Then rebuild
391
+ and recreate the container per **Channel 3 → 3.2**. Each webhook group goes to
392
+ MQTT banners plus one email/Telegram digest,
393
+ including firing and recovery states.
394
+
395
+ #### 2.5 Verify end-to-end
396
+
397
+ First use a Grafana test notification to check the configured route without
398
+ interrupting monitoring. Confirm both the recipient and dashboard receive it;
399
+ this tests delivery, not the agent-down rule. To test that rule during an agreed
400
+ maintenance window, temporarily stop the agent on the intended Venus host:
401
+ `venus-agent-down` should fire after its configured `for` window. Resume the agent
402
+ and verify the resolved message even if delivery fails:
403
+
404
+ ```bash
405
+ ssh root@cerbo 'svc -d /service/venus-os-observability' # pause
406
+ # wait ~2-3 min: Telegram message + dashboard banner appear
407
+ ssh root@cerbo 'svc -u /service/venus-os-observability' # resume -> resolved message
408
+ ```
409
+
410
+ ### Channel 3: Email via alert-mqtt-bridge + external SMTP relay
411
+
412
+ In the reference installation's 2026-08-24 notes, outbound port 25 was blocked,
413
+ a usable PTR record was unavailable, and its Grafana-native email contact point
414
+ produced no SMTP attempt or provider event. That installation used the bridge
415
+ with an external STARTTLS relay. These are historical site observations, not a
416
+ claim about every ISP, Grafana version or current deployment. The configured path is:
417
+
418
+ ```mermaid
419
+ flowchart LR
420
+ G[Grafana rule fires] -->|webhook| B[alert-mqtt-bridge]
421
+ B -->|MQTT| N["inverter/notifications<br/>desktop banners"]
422
+ B -->|SMTP :587| M[Brevo → mailbox]
423
+ B -->|Bot API| T[Telegram]
424
+ ```
425
+
426
+ #### 3.1 Provider setup (Brevo, one-time)
427
+
428
+ 1. Create an account at <https://www.brevo.com> or use another SMTP relay
429
+ that supports STARTTLS. Check the provider's current limits and requirements.
430
+ 2. Verify a sender address or a domain you control using the provider's
431
+ instructions. Complete any required DNS authentication before sending alerts.
432
+ 3. Create SMTP credentials and record the relay hostname, port and login in
433
+ the private bridge environment file.
434
+
435
+ #### 3.2 Bridge configuration
436
+
437
+ Create a private environment file outside the repository (mode 600). The
438
+ following values illustrate a Brevo relay; use your provider's settings:
439
+
440
+ ```
441
+ SMTP_HOST=smtp-relay.brevo.com
442
+ SMTP_PORT=587
443
+ SMTP_USER=<login>@smtp-brevo.com
444
+ SMTP_PASSWORD=<secret-key>
445
+ SMTP_FROM=<verified-sender@example.com>
446
+ SMTP_TO=<recipient@example.com>
447
+ ```
448
+
449
+ `SMTP_HOST` or `SMTP_TO` empty ⇒ email channel disabled, MQTT-only.
450
+
451
+ Email requires STARTTLS and a certificate valid for `SMTP_HOST`, signed by a
452
+ trusted CA. The bridge verifies the server before sending SMTP credentials or
453
+ the alert. A missing STARTTLS extension, untrusted certificate or hostname
454
+ mismatch fails this delivery attempt and is logged; it never falls back to
455
+ plaintext SMTP. If using a private CA, mount its CA bundle and set
456
+ `SSL_CERT_FILE` to that file inside the container. Existing relays with invalid
457
+ certificates must be corrected or given their proper CA trust before upgrading.
458
+
459
+ Build from the repository root. Set `BRIDGE_ENV_FILE` to the private file,
460
+ `GRAFANA_NETWORK` to a Docker network shared with Grafana, and `MQTT_HOST` to
461
+ the broker reachable from that network. For an update, preserve the existing
462
+ image/configuration, then stop and remove the old container before starting another
463
+ with the same name. A stopped container still reserves its name.
464
+
465
+ ```bash
466
+ : "${BRIDGE_ENV_FILE:?Set the path to the private environment file}"
467
+ : "${GRAFANA_NETWORK:?Set the Docker network shared with Grafana}"
468
+ : "${MQTT_HOST:?Set the MQTT broker hostname}"
469
+ sudo docker build -t venus-alert-bridge:latest ./alert-mqtt-bridge
470
+ sudo docker run -d --name venus-alert-bridge --restart unless-stopped \
471
+ --network "$GRAFANA_NETWORK" --network-alias venus-alert-bridge \
472
+ -e MQTT_HOST="$MQTT_HOST" --env-file "$BRIDGE_ENV_FILE" venus-alert-bridge:latest
473
+ ```
474
+
475
+ Configure a Grafana webhook contact point for the bridge and route the intended
476
+ notification policies to it. This route does not need a separate Grafana Email
477
+ contact point; it sends email through the bridge's SMTP configuration.
478
+
479
+ #### 3.3 Verify
480
+
481
+ 1. Check a test notification first; test actual firing and recovery during a
482
+ maintenance window as described in channel 2.5.
483
+ 2. Bridge log shows `Email sent: <alertname>`; check inbox AND spam folder.
484
+ 3. Check the selected SMTP provider's delivery-events page, when available, for
485
+ per-message status (delivered / blocked / rejected).
486
+ 4. Optional scoring: send a test via the same relay to the address shown at
487
+ <https://www.mail-tester.com>; aim for ≥9/10.
488
+
489
+ ## Development
490
+
491
+ ```bash
492
+ # Create a locked, isolated check environment
493
+ bash scripts/ci.sh --install
494
+
495
+ # Lint, types and tests
496
+ bash scripts/ci.sh
497
+ ```
498
+
499
+ ## License
500
+
501
+ MIT
502
+
503
+ ---
504
+
505
+ ## Related Projects
506
+
507
+ | Project | Scope | When to Use |
508
+ |---------|-------|-------------|
509
+ | [mqtt-observability-opentelemetry](https://github.com/4alvit/mqtt-observability-opentelemetry) | **Generic** — Works with ANY MQTT broker. No Venus OS dependency. | Generic MQTT/IoT observability, any broker, any device types |
510
+ | **venus-os-observability** (this) | **Venus OS specific** — Depends on D-Bus, Victron protocols. | Victron Venus OS only: D-Bus event tracing, inverter metrics, Cerbo GX integration |
511
+
512
+ **Choose mqtt-observability-opentelemetry if:** You need MQTT observability for any IoT system (industrial, home automation, custom devices).
513
+
514
+ **Choose this repo if:** You are running Victron Venus OS (Cerbo GX, Raspberry Pi with Venus OS) and need D-Bus integration, inverter-specific metrics, and Venus OS native deployment.
515
+
516
+ ## Contributing and security
517
+
518
+ See [CONTRIBUTING.md](CONTRIBUTING.md) for reports, development checks and pull requests,
519
+ [SECURITY.md](SECURITY.md) for private vulnerability reporting and deployment trust boundaries,
520
+ [GOVERNANCE.md](GOVERNANCE.md) for responsibilities and continuity limits,
521
+ [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for participation,
522
+ [ROADMAP.md](ROADMAP.md) for priorities,
523
+ and the [OpenSSF evidence index](docs/openssf-evidence.md) for assessment references and remaining verification.
524
+
525
+ The alert relay verifies exact certificate key sizes for SMTP STARTTLS and Telegram HTTPS.
526
+ See [alert relay TLS policy](docs/alert-relay-tls.md) for trusted CA, proxy and migration requirements.
527
+
528
+ ## Release-candidate fuzzing
529
+
530
+ The real correlation-context and header propagation functions receive synthetic identifiers, including Unicode and punctuation. Header extraction must preserve explicitly supplied IDs and context tokens must restore prior state. No trace exporter, D-Bus subscription or network connection starts.
531
+
532
+ [RC-only fuzzing policy, local reproduction and failure handling](docs/fuzzing.md).