pyalert-mail 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pyalert_mail-0.1.1/LICENSE +21 -0
- pyalert_mail-0.1.1/PKG-INFO +335 -0
- pyalert_mail-0.1.1/README.md +296 -0
- pyalert_mail-0.1.1/pyalert/__init__.py +49 -0
- pyalert_mail-0.1.1/pyalert/config.py +427 -0
- pyalert_mail-0.1.1/pyalert/monitor.py +397 -0
- pyalert_mail-0.1.1/pyalert/notifier.py +741 -0
- pyalert_mail-0.1.1/pyalert/py.typed +0 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/PKG-INFO +335 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/SOURCES.txt +14 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/dependency_links.txt +1 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/entry_points.txt +2 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/requires.txt +8 -0
- pyalert_mail-0.1.1/pyalert_mail.egg-info/top_level.txt +1 -0
- pyalert_mail-0.1.1/pyproject.toml +68 -0
- pyalert_mail-0.1.1/setup.cfg +4 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sudipta Majumder
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,335 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pyalert-mail
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Zero-credential-leak email alerting for long-running ML/HPC jobs, powered by a user-owned Gmail Apps Script bridge.
|
|
5
|
+
Author: pyalert contributors
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/sudipta-on/pyalert
|
|
8
|
+
Project-URL: Repository, https://github.com/sudipta-on/pyalert
|
|
9
|
+
Project-URL: Issues, https://github.com/sudipta-on/pyalert/issues
|
|
10
|
+
Project-URL: Changelog, https://github.com/sudipta-on/pyalert/blob/main/CHANGELOG.md
|
|
11
|
+
Keywords: monitoring,alerting,email,gmail,machine-learning,hpc,training,gpu,nvml,notifications,apps-script
|
|
12
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
24
|
+
Classifier: Topic :: System :: Monitoring
|
|
25
|
+
Classifier: Topic :: System :: Distributed Computing
|
|
26
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
27
|
+
Classifier: Typing :: Typed
|
|
28
|
+
Requires-Python: >=3.8
|
|
29
|
+
Description-Content-Type: text/markdown
|
|
30
|
+
License-File: LICENSE
|
|
31
|
+
Requires-Dist: psutil>=5.9.0
|
|
32
|
+
Provides-Extra: dev
|
|
33
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
34
|
+
Requires-Dist: pytest-cov>=4.0; extra == "dev"
|
|
35
|
+
Requires-Dist: mypy>=1.5; extra == "dev"
|
|
36
|
+
Requires-Dist: build>=1.0; extra == "dev"
|
|
37
|
+
Requires-Dist: twine>=4.0; extra == "dev"
|
|
38
|
+
Dynamic: license-file
|
|
39
|
+
|
|
40
|
+
# pyalert
|
|
41
|
+
|
|
42
|
+
[](https://pypi.org/project/pyalert/)
|
|
43
|
+
[](https://pypi.org/project/pyalert/)
|
|
44
|
+
[](LICENSE)
|
|
45
|
+
[](https://pypi.org/project/pyalert/)
|
|
46
|
+
|
|
47
|
+
**Get emailed when your training run finishes, stalls, or crashes โ without ever handing an email password or a paid API key to a Python package.**
|
|
48
|
+
|
|
49
|
+
`pyalert` is a lightweight alerting library for researchers, ML engineers, and anyone running long background jobs (training loops, simulations, HPC batch jobs). It buffers your progress checkpoints into a clean HTML digest, sends immediate alerts on crashes with a full traceback, and does it all through a tiny Google Apps Script bridge that runs **in your own Google account** โ pyalert itself never touches your Gmail credentials.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Why pyalert?
|
|
54
|
+
|
|
55
|
+
| | |
|
|
56
|
+
|---|---|
|
|
57
|
+
| ๐ **Zero-credential-leak** | Sends mail via a Google Apps Script Web App deployed under **your** Google account. No SMTP password, no Gmail "app password", no third-party API key ever stored or transmitted by pyalert. |
|
|
58
|
+
| ๐ชถ **Lightweight** | Pure standard library except for one hard dependency: `psutil`. NVIDIA GPU stats are read via raw `ctypes` bindings to NVML โ no `pynvml`, no `torch` required. |
|
|
59
|
+
| ๐งต **Non-blocking** | Emails dispatch on a background daemon thread by default; your training loop never waits on the network. |
|
|
60
|
+
| ๐งบ **Smart batching** | Frequent `checkpoint()` calls in tight loops are buffered and merged into one digest email per cooldown window (default 60s), instead of spamming your inbox. |
|
|
61
|
+
| ๐จ **Instant crash alerts** | Errors and exceptions bypass the cooldown entirely and are sent immediately with a full stack trace. |
|
|
62
|
+
| ๐ **Attachments** | Attach plots, CSVs, or log files โ base64-encoded and delivered as real Gmail attachments. |
|
|
63
|
+
| ๐ฅ๏ธ **Cross-platform** | Linux, macOS, and Windows, with correct config paths and permissions on each. |
|
|
64
|
+
| ๐งฏ **Fails safe** | Network errors, missing GPU drivers, or bad file paths are caught and logged to stderr โ pyalert will never crash or block your actual computation. |
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## Architecture
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
YOUR MACHINE (laptop / HPC node / cloud VM)
|
|
72
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
73
|
+
โ โ
|
|
74
|
+
โ your_script.py โ
|
|
75
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
76
|
+
โ โ from pyalert import PyAlert โ โ
|
|
77
|
+
โ โ alert = PyAlert(project_name="training-run") โ โ
|
|
78
|
+
โ โ โ โ
|
|
79
|
+
โ โ โโ โ โ
|
|
80
|
+
โ โ def train(): ... โ decorator / context manager / โ โ
|
|
81
|
+
โ โ โ manual .checkpoint() calls โ โ
|
|
82
|
+
โ โ with alert.track_block()โ โ โ
|
|
83
|
+
โ โ alert.checkpoint(...) โโ โ โ
|
|
84
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
85
|
+
โ โ โ
|
|
86
|
+
โ โผ โ
|
|
87
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
88
|
+
โ โ pyalert.notifier.PyAlert โ โ
|
|
89
|
+
โ โ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโ โ โ
|
|
90
|
+
โ โ โ Rate limiter/ โ โ HTML digest โ โ Attachmentโ โ โ
|
|
91
|
+
โ โ โ event buffer โโโโถโ renderer โโโโถโ base64 โ โ โ
|
|
92
|
+
โ โ โ (cooldown, โ โ (inline-CSS, โ โ encoder โ โ โ
|
|
93
|
+
โ โ โ atexit flush) โ โ mobile-safe) โ โ โ โ โ
|
|
94
|
+
โ โ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโ โ โ
|
|
95
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
96
|
+
โ โ background thread โ
|
|
97
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
|
|
98
|
+
โ โ pyalert.monitor โ โ โ
|
|
99
|
+
โ โ SystemMonitor โ โ โ
|
|
100
|
+
โ โ โข psutil: CPU/RAM/disk โ โ โ
|
|
101
|
+
โ โ โข ctypes โ NVML: GPU โ โ โ
|
|
102
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
|
|
103
|
+
โ โผ โ
|
|
104
|
+
โ HTTPS POST (JSON payload) โ
|
|
105
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
106
|
+
โ
|
|
107
|
+
โผ
|
|
108
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
109
|
+
โ Google Apps Script Web App (Code.gs) โ
|
|
110
|
+
โ deployed under YOUR Google account โ
|
|
111
|
+
โ โข validates optional shared secret โ
|
|
112
|
+
โ โข decodes attachments โ
|
|
113
|
+
โ โข GmailApp.sendEmail(...) โ โ runs as YOU,
|
|
114
|
+
โโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโ OAuth-authorized
|
|
115
|
+
โ by Google, no
|
|
116
|
+
โผ password ever
|
|
117
|
+
๐ง Styled HTML email in your inbox passed around
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Because the Apps Script runs *inside* Google's infrastructure under your own account, Gmail sending is authorized via Google's own OAuth consent screen when you deploy it โ pyalert (the Python client) only ever sees a webhook URL that you control, and can be revoked or redeployed at any time from `script.google.com`.
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## Installation
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
pip install pyalert
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Requires Python 3.8+. The only hard runtime dependency is `psutil`.
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Quickstart
|
|
135
|
+
|
|
136
|
+
### 1. Deploy your personal email bridge (one-time, ~2 minutes)
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
pyalert-setup --generate-script
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
This writes a ready-to-paste `Code.gs` file and prints deployment instructions:
|
|
143
|
+
|
|
144
|
+
1. Go to [script.google.com](https://script.google.com/) โ **New project**.
|
|
145
|
+
2. Delete the boilerplate, paste in the generated `Code.gs`.
|
|
146
|
+
3. **Deploy โ New deployment โ Web app**
|
|
147
|
+
- Execute as: **Me**
|
|
148
|
+
- Who has access: **Only myself** (recommended)
|
|
149
|
+
4. Click **Deploy**, authorize the requested Gmail permission (this is Google's own OAuth screen โ pyalert never sees this token).
|
|
150
|
+
5. Copy the **Web app URL**.
|
|
151
|
+
|
|
152
|
+
### 2. Run the setup wizard
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
pyalert-setup
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Paste in the Web App URL, your recipient email, and (optionally) a shared secret โ this saves `~/.config/pyalert/config.json`. Verify it works:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
pyalert-setup --test
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### 3. Use it in your code
|
|
165
|
+
|
|
166
|
+
```python
|
|
167
|
+
from pyalert import PyAlert
|
|
168
|
+
|
|
169
|
+
alert = PyAlert(project_name="resnet50-imagenet")
|
|
170
|
+
|
|
171
|
+
def train_model():
|
|
172
|
+
for epoch in range(100):
|
|
173
|
+
loss = run_epoch()
|
|
174
|
+
alert.checkpoint(
|
|
175
|
+
f"epoch {epoch} complete",
|
|
176
|
+
extra={"loss": round(loss, 4), "epoch": epoch},
|
|
177
|
+
)
|
|
178
|
+
|
|
179
|
+
train_model() # โถ/โ notifications + a final digest email; a crash alert
|
|
180
|
+
# with full traceback fires immediately if anything raises
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Usage patterns
|
|
186
|
+
|
|
187
|
+
### Decorator
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
@alert.watch(project="data-pipeline", capture_result=True)
|
|
191
|
+
def build_dataset():
|
|
192
|
+
...
|
|
193
|
+
return dataset_stats
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
### Context manager
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
with alert.track_block("hyperparameter-search", attachments=["sweep_results.csv"]):
|
|
200
|
+
run_sweep()
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
### Manual checkpoints (buffered + throttled)
|
|
204
|
+
|
|
205
|
+
```python
|
|
206
|
+
alert.checkpoint("validation accuracy improved", extra={"val_acc": 0.94})
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
### Manual crash reporting
|
|
210
|
+
|
|
211
|
+
```python
|
|
212
|
+
try:
|
|
213
|
+
risky_operation()
|
|
214
|
+
except Exception:
|
|
215
|
+
alert.report_exception(context="risky_operation failed")
|
|
216
|
+
raise
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
### Attachments
|
|
220
|
+
|
|
221
|
+
```python
|
|
222
|
+
alert.checkpoint(
|
|
223
|
+
"training complete โ see attached loss curve",
|
|
224
|
+
attachments=["loss_curve.png", "metrics.csv"],
|
|
225
|
+
)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Attachments over `max_attachment_mb` (default 20MB, configurable) are skipped with a warning rather than failing the send.
|
|
229
|
+
|
|
230
|
+
### `logging` integration
|
|
231
|
+
|
|
232
|
+
```python
|
|
233
|
+
import logging
|
|
234
|
+
from pyalert import PyAlert, PyAlertLogHandler
|
|
235
|
+
|
|
236
|
+
alert = PyAlert(project_name="etl-job")
|
|
237
|
+
logging.getLogger().addHandler(PyAlertLogHandler(alert, level=logging.WARNING))
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
Any `logging.warning()`/`error()`/`critical()` call anywhere in your codebase now also feeds pyalert's digest/crash pipeline โ no code changes needed elsewhere.
|
|
241
|
+
|
|
242
|
+
### Graceful SIGINT/SIGTERM handling (opt-in)
|
|
243
|
+
|
|
244
|
+
```python
|
|
245
|
+
alert = PyAlert(project_name="hpc-job", catch_signals=True)
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
On `Ctrl+C` or a scheduler-issued `SIGTERM`, pyalert flushes a final "interrupted" alert before letting the signal proceed normally.
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
|
|
252
|
+
## Configuration
|
|
253
|
+
|
|
254
|
+
`pyalert-setup` writes `~/.config/pyalert/config.json`. Every field can also be overridden with an environment variable, useful for CI/HPC job schedulers:
|
|
255
|
+
|
|
256
|
+
| Config field | Environment variable | Default |
|
|
257
|
+
|---|---|---|
|
|
258
|
+
| `webhook_url` | `PYALERT_WEBHOOK_URL` | โ |
|
|
259
|
+
| `recipient_email` | `PYALERT_RECIPIENT` | โ |
|
|
260
|
+
| `sender_name` | `PYALERT_SENDER_NAME` | `PyAlert` |
|
|
261
|
+
| `shared_secret` | `PYALERT_SHARED_SECRET` | โ |
|
|
262
|
+
| `default_cooldown_seconds` | `PYALERT_COOLDOWN` | `60` |
|
|
263
|
+
| `dry_run` | `PYALERT_DRY_RUN` | `false` |
|
|
264
|
+
| โ | `PYALERT_CONFIG_DIR` | `~/.config/pyalert` |
|
|
265
|
+
|
|
266
|
+
`PYALERT_DRY_RUN=1` makes pyalert print what it *would* send to stderr instead of making a network call โ handy for testing pipelines without spamming your inbox.
|
|
267
|
+
|
|
268
|
+
### CLI reference
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
pyalert-setup # interactive wizard
|
|
272
|
+
pyalert-setup --generate-script # write Code.gs
|
|
273
|
+
pyalert-setup --test # send a real test email with current config
|
|
274
|
+
pyalert-setup --show # print current config (secret redacted)
|
|
275
|
+
pyalert-setup --non-interactive --shared-secret XYZ # scriptable/CI use
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
---
|
|
279
|
+
|
|
280
|
+
## What's in the digest email
|
|
281
|
+
|
|
282
|
+
Every digest includes:
|
|
283
|
+
|
|
284
|
+
- **Event cards** โ one per checkpoint, color-coded by level (`INFO`, `SUCCESS`, `WARNING`, `ERROR`, `CRITICAL`), with your message, any `extra={}` metrics in a monospace table, and (for crashes) the full formatted traceback.
|
|
285
|
+
- **System snapshot** โ host name, platform, per-core CPU%, load average, system RAM, aggregate process RSS (including child processes), disk usage, cumulative network I/O, host uptime, and per-GPU utilization/memory/temperature/power for every detected NVIDIA GPU.
|
|
286
|
+
- Responsive, inline-CSS HTML that renders correctly in Gmail (web + mobile), Apple Mail, and Outlook.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## GPU monitoring details
|
|
291
|
+
|
|
292
|
+
`pyalert.monitor.SystemMonitor` loads `libnvidia-ml.so.1` (Linux) or `nvml.dll` (Windows) directly via `ctypes` โ the same library `nvidia-smi` uses โ and calls a minimal subset of the NVML C API (`nvmlDeviceGetCount_v2`, `nvmlDeviceGetHandleByIndex_v2`, `nvmlDeviceGetMemoryInfo`, `nvmlDeviceGetUtilizationRates`, `nvmlDeviceGetTemperature`, `nvmlDeviceGetPowerUsage`). If the library isn't found (no NVIDIA GPU, no driver, or macOS, which has no NVML), pyalert logs a one-time informational note to stderr and simply omits GPU data โ your job is never interrupted.
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
## Security notes
|
|
297
|
+
|
|
298
|
+
- pyalert **never** asks for, stores, or transmits a Gmail password or app-specific password.
|
|
299
|
+
- The only secret pyalert stores locally is an optional **shared secret** you generate yourself, used to stop random requests to your Apps Script URL from sending mail on your behalf. `config.json` is written with owner-only file permissions (`0600`) on POSIX systems.
|
|
300
|
+
- Set the Apps Script deployment's access to **"Only myself"** for the strongest guarantee โ only requests carrying your Google session (impossible for pyalert to forge) or the URL + shared secret can trigger a send.
|
|
301
|
+
- All payloads are sent over HTTPS to `script.google.com`, Google's own domain.
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
305
|
+
## Project layout
|
|
306
|
+
|
|
307
|
+
```
|
|
308
|
+
pyalert-runner/
|
|
309
|
+
โโโ pyproject.toml
|
|
310
|
+
โโโ LICENSE
|
|
311
|
+
โโโ README.md
|
|
312
|
+
โโโ pyalert/
|
|
313
|
+
โโโ __init__.py # public API surface
|
|
314
|
+
โโโ config.py # Config model, wizard, Code.gs generator, CLI (no cyclic imports)
|
|
315
|
+
โโโ monitor.py # ctypes NVML GPU tracker + psutil CPU/RAM/disk/network sampler
|
|
316
|
+
โโโ notifier.py # PyAlert engine: buffering, HTML rendering, dispatch, decorator/CM
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
Import graph is strictly acyclic: `config.py` has zero imports from `monitor.py` or `notifier.py`; `monitor.py` has zero imports from `notifier.py`; `notifier.py` imports from both. `pyalert-setup` (โ `config.main`) only pulls in `notifier` lazily, inside the `--test` code path, so the CLI stays fast and dependency-light for the common config-only case.
|
|
320
|
+
|
|
321
|
+
---
|
|
322
|
+
|
|
323
|
+
## Contributing
|
|
324
|
+
|
|
325
|
+
Issues and PRs welcome. Run the test suite with:
|
|
326
|
+
|
|
327
|
+
```bash
|
|
328
|
+
pip install -e ".[dev]"
|
|
329
|
+
pytest
|
|
330
|
+
mypy pyalert
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
## License
|
|
334
|
+
|
|
335
|
+
MIT โ see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,296 @@
|
|
|
1
|
+
# pyalert
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/pyalert/)
|
|
4
|
+
[](https://pypi.org/project/pyalert/)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
[](https://pypi.org/project/pyalert/)
|
|
7
|
+
|
|
8
|
+
**Get emailed when your training run finishes, stalls, or crashes โ without ever handing an email password or a paid API key to a Python package.**
|
|
9
|
+
|
|
10
|
+
`pyalert` is a lightweight alerting library for researchers, ML engineers, and anyone running long background jobs (training loops, simulations, HPC batch jobs). It buffers your progress checkpoints into a clean HTML digest, sends immediate alerts on crashes with a full traceback, and does it all through a tiny Google Apps Script bridge that runs **in your own Google account** โ pyalert itself never touches your Gmail credentials.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Why pyalert?
|
|
15
|
+
|
|
16
|
+
| | |
|
|
17
|
+
|---|---|
|
|
18
|
+
| ๐ **Zero-credential-leak** | Sends mail via a Google Apps Script Web App deployed under **your** Google account. No SMTP password, no Gmail "app password", no third-party API key ever stored or transmitted by pyalert. |
|
|
19
|
+
| ๐ชถ **Lightweight** | Pure standard library except for one hard dependency: `psutil`. NVIDIA GPU stats are read via raw `ctypes` bindings to NVML โ no `pynvml`, no `torch` required. |
|
|
20
|
+
| ๐งต **Non-blocking** | Emails dispatch on a background daemon thread by default; your training loop never waits on the network. |
|
|
21
|
+
| ๐งบ **Smart batching** | Frequent `checkpoint()` calls in tight loops are buffered and merged into one digest email per cooldown window (default 60s), instead of spamming your inbox. |
|
|
22
|
+
| ๐จ **Instant crash alerts** | Errors and exceptions bypass the cooldown entirely and are sent immediately with a full stack trace. |
|
|
23
|
+
| ๐ **Attachments** | Attach plots, CSVs, or log files โ base64-encoded and delivered as real Gmail attachments. |
|
|
24
|
+
| ๐ฅ๏ธ **Cross-platform** | Linux, macOS, and Windows, with correct config paths and permissions on each. |
|
|
25
|
+
| ๐งฏ **Fails safe** | Network errors, missing GPU drivers, or bad file paths are caught and logged to stderr โ pyalert will never crash or block your actual computation. |
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## Architecture
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
YOUR MACHINE (laptop / HPC node / cloud VM)
|
|
33
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
34
|
+
โ โ
|
|
35
|
+
โ your_script.py โ
|
|
36
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
37
|
+
โ โ from pyalert import PyAlert โ โ
|
|
38
|
+
โ โ alert = PyAlert(project_name="training-run") โ โ
|
|
39
|
+
โ โ โ โ
|
|
40
|
+
โ โ โโ โ โ
|
|
41
|
+
โ โ def train(): ... โ decorator / context manager / โ โ
|
|
42
|
+
โ โ โ manual .checkpoint() calls โ โ
|
|
43
|
+
โ โ with alert.track_block()โ โ โ
|
|
44
|
+
โ โ alert.checkpoint(...) โโ โ โ
|
|
45
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
46
|
+
โ โ โ
|
|
47
|
+
โ โผ โ
|
|
48
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
49
|
+
โ โ pyalert.notifier.PyAlert โ โ
|
|
50
|
+
โ โ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโ โ โ
|
|
51
|
+
โ โ โ Rate limiter/ โ โ HTML digest โ โ Attachmentโ โ โ
|
|
52
|
+
โ โ โ event buffer โโโโถโ renderer โโโโถโ base64 โ โ โ
|
|
53
|
+
โ โ โ (cooldown, โ โ (inline-CSS, โ โ encoder โ โ โ
|
|
54
|
+
โ โ โ atexit flush) โ โ mobile-safe) โ โ โ โ โ
|
|
55
|
+
โ โ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโ โ โ
|
|
56
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
|
|
57
|
+
โ โ background thread โ
|
|
58
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
|
|
59
|
+
โ โ pyalert.monitor โ โ โ
|
|
60
|
+
โ โ SystemMonitor โ โ โ
|
|
61
|
+
โ โ โข psutil: CPU/RAM/disk โ โ โ
|
|
62
|
+
โ โ โข ctypes โ NVML: GPU โ โ โ
|
|
63
|
+
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
|
|
64
|
+
โ โผ โ
|
|
65
|
+
โ HTTPS POST (JSON payload) โ
|
|
66
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
67
|
+
โ
|
|
68
|
+
โผ
|
|
69
|
+
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
70
|
+
โ Google Apps Script Web App (Code.gs) โ
|
|
71
|
+
โ deployed under YOUR Google account โ
|
|
72
|
+
โ โข validates optional shared secret โ
|
|
73
|
+
โ โข decodes attachments โ
|
|
74
|
+
โ โข GmailApp.sendEmail(...) โ โ runs as YOU,
|
|
75
|
+
โโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโ OAuth-authorized
|
|
76
|
+
โ by Google, no
|
|
77
|
+
โผ password ever
|
|
78
|
+
๐ง Styled HTML email in your inbox passed around
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Because the Apps Script runs *inside* Google's infrastructure under your own account, Gmail sending is authorized via Google's own OAuth consent screen when you deploy it โ pyalert (the Python client) only ever sees a webhook URL that you control, and can be revoked or redeployed at any time from `script.google.com`.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Installation
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
pip install pyalert
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Requires Python 3.8+. The only hard runtime dependency is `psutil`.
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## Quickstart
|
|
96
|
+
|
|
97
|
+
### 1. Deploy your personal email bridge (one-time, ~2 minutes)
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
pyalert-setup --generate-script
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
This writes a ready-to-paste `Code.gs` file and prints deployment instructions:
|
|
104
|
+
|
|
105
|
+
1. Go to [script.google.com](https://script.google.com/) โ **New project**.
|
|
106
|
+
2. Delete the boilerplate, paste in the generated `Code.gs`.
|
|
107
|
+
3. **Deploy โ New deployment โ Web app**
|
|
108
|
+
- Execute as: **Me**
|
|
109
|
+
- Who has access: **Only myself** (recommended)
|
|
110
|
+
4. Click **Deploy**, authorize the requested Gmail permission (this is Google's own OAuth screen โ pyalert never sees this token).
|
|
111
|
+
5. Copy the **Web app URL**.
|
|
112
|
+
|
|
113
|
+
### 2. Run the setup wizard
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
pyalert-setup
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Paste in the Web App URL, your recipient email, and (optionally) a shared secret โ this saves `~/.config/pyalert/config.json`. Verify it works:
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
pyalert-setup --test
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### 3. Use it in your code
|
|
126
|
+
|
|
127
|
+
```python
|
|
128
|
+
from pyalert import PyAlert
|
|
129
|
+
|
|
130
|
+
alert = PyAlert(project_name="resnet50-imagenet")
|
|
131
|
+
|
|
132
|
+
def train_model():
|
|
133
|
+
for epoch in range(100):
|
|
134
|
+
loss = run_epoch()
|
|
135
|
+
alert.checkpoint(
|
|
136
|
+
f"epoch {epoch} complete",
|
|
137
|
+
extra={"loss": round(loss, 4), "epoch": epoch},
|
|
138
|
+
)
|
|
139
|
+
|
|
140
|
+
train_model() # โถ/โ notifications + a final digest email; a crash alert
|
|
141
|
+
# with full traceback fires immediately if anything raises
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
---
|
|
145
|
+
|
|
146
|
+
## Usage patterns
|
|
147
|
+
|
|
148
|
+
### Decorator
|
|
149
|
+
|
|
150
|
+
```python
|
|
151
|
+
@alert.watch(project="data-pipeline", capture_result=True)
|
|
152
|
+
def build_dataset():
|
|
153
|
+
...
|
|
154
|
+
return dataset_stats
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
### Context manager
|
|
158
|
+
|
|
159
|
+
```python
|
|
160
|
+
with alert.track_block("hyperparameter-search", attachments=["sweep_results.csv"]):
|
|
161
|
+
run_sweep()
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### Manual checkpoints (buffered + throttled)
|
|
165
|
+
|
|
166
|
+
```python
|
|
167
|
+
alert.checkpoint("validation accuracy improved", extra={"val_acc": 0.94})
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### Manual crash reporting
|
|
171
|
+
|
|
172
|
+
```python
|
|
173
|
+
try:
|
|
174
|
+
risky_operation()
|
|
175
|
+
except Exception:
|
|
176
|
+
alert.report_exception(context="risky_operation failed")
|
|
177
|
+
raise
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
### Attachments
|
|
181
|
+
|
|
182
|
+
```python
|
|
183
|
+
alert.checkpoint(
|
|
184
|
+
"training complete โ see attached loss curve",
|
|
185
|
+
attachments=["loss_curve.png", "metrics.csv"],
|
|
186
|
+
)
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Attachments over `max_attachment_mb` (default 20MB, configurable) are skipped with a warning rather than failing the send.
|
|
190
|
+
|
|
191
|
+
### `logging` integration
|
|
192
|
+
|
|
193
|
+
```python
|
|
194
|
+
import logging
|
|
195
|
+
from pyalert import PyAlert, PyAlertLogHandler
|
|
196
|
+
|
|
197
|
+
alert = PyAlert(project_name="etl-job")
|
|
198
|
+
logging.getLogger().addHandler(PyAlertLogHandler(alert, level=logging.WARNING))
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Any `logging.warning()`/`error()`/`critical()` call anywhere in your codebase now also feeds pyalert's digest/crash pipeline โ no code changes needed elsewhere.
|
|
202
|
+
|
|
203
|
+
### Graceful SIGINT/SIGTERM handling (opt-in)
|
|
204
|
+
|
|
205
|
+
```python
|
|
206
|
+
alert = PyAlert(project_name="hpc-job", catch_signals=True)
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
On `Ctrl+C` or a scheduler-issued `SIGTERM`, pyalert flushes a final "interrupted" alert before letting the signal proceed normally.
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## Configuration
|
|
214
|
+
|
|
215
|
+
`pyalert-setup` writes `~/.config/pyalert/config.json`. Every field can also be overridden with an environment variable, useful for CI/HPC job schedulers:
|
|
216
|
+
|
|
217
|
+
| Config field | Environment variable | Default |
|
|
218
|
+
|---|---|---|
|
|
219
|
+
| `webhook_url` | `PYALERT_WEBHOOK_URL` | โ |
|
|
220
|
+
| `recipient_email` | `PYALERT_RECIPIENT` | โ |
|
|
221
|
+
| `sender_name` | `PYALERT_SENDER_NAME` | `PyAlert` |
|
|
222
|
+
| `shared_secret` | `PYALERT_SHARED_SECRET` | โ |
|
|
223
|
+
| `default_cooldown_seconds` | `PYALERT_COOLDOWN` | `60` |
|
|
224
|
+
| `dry_run` | `PYALERT_DRY_RUN` | `false` |
|
|
225
|
+
| โ | `PYALERT_CONFIG_DIR` | `~/.config/pyalert` |
|
|
226
|
+
|
|
227
|
+
`PYALERT_DRY_RUN=1` makes pyalert print what it *would* send to stderr instead of making a network call โ handy for testing pipelines without spamming your inbox.
|
|
228
|
+
|
|
229
|
+
### CLI reference
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
pyalert-setup # interactive wizard
|
|
233
|
+
pyalert-setup --generate-script # write Code.gs
|
|
234
|
+
pyalert-setup --test # send a real test email with current config
|
|
235
|
+
pyalert-setup --show # print current config (secret redacted)
|
|
236
|
+
pyalert-setup --non-interactive --shared-secret XYZ # scriptable/CI use
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
---
|
|
240
|
+
|
|
241
|
+
## What's in the digest email
|
|
242
|
+
|
|
243
|
+
Every digest includes:
|
|
244
|
+
|
|
245
|
+
- **Event cards** โ one per checkpoint, color-coded by level (`INFO`, `SUCCESS`, `WARNING`, `ERROR`, `CRITICAL`), with your message, any `extra={}` metrics in a monospace table, and (for crashes) the full formatted traceback.
|
|
246
|
+
- **System snapshot** โ host name, platform, per-core CPU%, load average, system RAM, aggregate process RSS (including child processes), disk usage, cumulative network I/O, host uptime, and per-GPU utilization/memory/temperature/power for every detected NVIDIA GPU.
|
|
247
|
+
- Responsive, inline-CSS HTML that renders correctly in Gmail (web + mobile), Apple Mail, and Outlook.
|
|
248
|
+
|
|
249
|
+
---
|
|
250
|
+
|
|
251
|
+
## GPU monitoring details
|
|
252
|
+
|
|
253
|
+
`pyalert.monitor.SystemMonitor` loads `libnvidia-ml.so.1` (Linux) or `nvml.dll` (Windows) directly via `ctypes` โ the same library `nvidia-smi` uses โ and calls a minimal subset of the NVML C API (`nvmlDeviceGetCount_v2`, `nvmlDeviceGetHandleByIndex_v2`, `nvmlDeviceGetMemoryInfo`, `nvmlDeviceGetUtilizationRates`, `nvmlDeviceGetTemperature`, `nvmlDeviceGetPowerUsage`). If the library isn't found (no NVIDIA GPU, no driver, or macOS, which has no NVML), pyalert logs a one-time informational note to stderr and simply omits GPU data โ your job is never interrupted.
|
|
254
|
+
|
|
255
|
+
---
|
|
256
|
+
|
|
257
|
+
## Security notes
|
|
258
|
+
|
|
259
|
+
- pyalert **never** asks for, stores, or transmits a Gmail password or app-specific password.
|
|
260
|
+
- The only secret pyalert stores locally is an optional **shared secret** you generate yourself, used to stop random requests to your Apps Script URL from sending mail on your behalf. `config.json` is written with owner-only file permissions (`0600`) on POSIX systems.
|
|
261
|
+
- Set the Apps Script deployment's access to **"Only myself"** for the strongest guarantee โ only requests carrying your Google session (impossible for pyalert to forge) or the URL + shared secret can trigger a send.
|
|
262
|
+
- All payloads are sent over HTTPS to `script.google.com`, Google's own domain.
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## Project layout
|
|
267
|
+
|
|
268
|
+
```
|
|
269
|
+
pyalert-runner/
|
|
270
|
+
โโโ pyproject.toml
|
|
271
|
+
โโโ LICENSE
|
|
272
|
+
โโโ README.md
|
|
273
|
+
โโโ pyalert/
|
|
274
|
+
โโโ __init__.py # public API surface
|
|
275
|
+
โโโ config.py # Config model, wizard, Code.gs generator, CLI (no cyclic imports)
|
|
276
|
+
โโโ monitor.py # ctypes NVML GPU tracker + psutil CPU/RAM/disk/network sampler
|
|
277
|
+
โโโ notifier.py # PyAlert engine: buffering, HTML rendering, dispatch, decorator/CM
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
Import graph is strictly acyclic: `config.py` has zero imports from `monitor.py` or `notifier.py`; `monitor.py` has zero imports from `notifier.py`; `notifier.py` imports from both. `pyalert-setup` (โ `config.main`) only pulls in `notifier` lazily, inside the `--test` code path, so the CLI stays fast and dependency-light for the common config-only case.
|
|
281
|
+
|
|
282
|
+
---
|
|
283
|
+
|
|
284
|
+
## Contributing
|
|
285
|
+
|
|
286
|
+
Issues and PRs welcome. Run the test suite with:
|
|
287
|
+
|
|
288
|
+
```bash
|
|
289
|
+
pip install -e ".[dev]"
|
|
290
|
+
pytest
|
|
291
|
+
mypy pyalert
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
## License
|
|
295
|
+
|
|
296
|
+
MIT โ see [LICENSE](LICENSE).
|