airon 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. airon-0.1.0/.gitignore +37 -0
  2. airon-0.1.0/CHANGELOG.md +21 -0
  3. airon-0.1.0/LICENSE +21 -0
  4. airon-0.1.0/PKG-INFO +212 -0
  5. airon-0.1.0/README.md +154 -0
  6. airon-0.1.0/docs/protocol.md +186 -0
  7. airon-0.1.0/docs/releasing.md +37 -0
  8. airon-0.1.0/docs/usage.md +156 -0
  9. airon-0.1.0/examples/custom_loop.py +83 -0
  10. airon-0.1.0/examples/pytorch_loop.py +70 -0
  11. airon-0.1.0/pyproject.toml +72 -0
  12. airon-0.1.0/src/airon/__init__.py +30 -0
  13. airon-0.1.0/src/airon/__main__.py +5 -0
  14. airon-0.1.0/src/airon/_config.py +138 -0
  15. airon-0.1.0/src/airon/_console.py +125 -0
  16. airon-0.1.0/src/airon/_env.py +163 -0
  17. airon-0.1.0/src/airon/_hooks.py +116 -0
  18. airon-0.1.0/src/airon/_log.py +54 -0
  19. airon-0.1.0/src/airon/_names.py +81 -0
  20. airon-0.1.0/src/airon/_run.py +672 -0
  21. airon-0.1.0/src/airon/_sender.py +454 -0
  22. airon-0.1.0/src/airon/_spool.py +199 -0
  23. airon-0.1.0/src/airon/_sync.py +110 -0
  24. airon-0.1.0/src/airon/_system.py +302 -0
  25. airon-0.1.0/src/airon/_transport.py +242 -0
  26. airon-0.1.0/src/airon/_values.py +132 -0
  27. airon-0.1.0/src/airon/_version.py +4 -0
  28. airon-0.1.0/src/airon/cli.py +330 -0
  29. airon-0.1.0/src/airon/py.typed +0 -0
  30. airon-0.1.0/tests/conftest.py +84 -0
  31. airon-0.1.0/tests/fake_server.py +339 -0
  32. airon-0.1.0/tests/test_batching.py +81 -0
  33. airon-0.1.0/tests/test_cli.py +251 -0
  34. airon-0.1.0/tests/test_config.py +349 -0
  35. airon-0.1.0/tests/test_console.py +195 -0
  36. airon-0.1.0/tests/test_happy_path.py +240 -0
  37. airon-0.1.0/tests/test_lifecycle.py +315 -0
  38. airon-0.1.0/tests/test_resilience.py +202 -0
  39. airon-0.1.0/tests/test_retries.py +165 -0
  40. airon-0.1.0/tests/test_spool.py +261 -0
  41. airon-0.1.0/tests/test_system.py +355 -0
  42. airon-0.1.0/tests/test_timing.py +53 -0
  43. airon-0.1.0/tests/test_values.py +122 -0
airon-0.1.0/.gitignore ADDED
@@ -0,0 +1,37 @@
1
+ # Secrets and local configuration
2
+ .env
3
+ .env.*
4
+ !.env.example
5
+ *.pem
6
+ *.key
7
+
8
+ # Python
9
+ __pycache__/
10
+ *.py[cod]
11
+ .venv/
12
+ venv/
13
+ *.egg-info/
14
+ build/
15
+ dist/
16
+ .pytest_cache/
17
+ .mypy_cache/
18
+ .ruff_cache/
19
+ .coverage
20
+ htmlcov/
21
+ .python-version
22
+
23
+ # The offline spool and local credentials must never be committed
24
+ .airon/
25
+ credentials.json
26
+
27
+ # Editors and OS
28
+ .DS_Store
29
+ .idea/
30
+ .vscode/
31
+ *.swp
32
+
33
+ # Local tool state
34
+ .claude/
35
+
36
+ # uv resolves the dev environment itself; the SDK is a library
37
+ uv.lock
@@ -0,0 +1,21 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
5
+ [semantic versioning](https://semver.org/).
6
+
7
+ ## [0.1.0] - unreleased
8
+
9
+ First version, implementing wire protocol 1.
10
+
11
+ ### Added
12
+
13
+ - `airon.init`, `run.log`, `run.progress`, `run.config`, `run.summary`, `run.checkpoint`,
14
+ `run.finish`, the `with` form and the module-level `airon.log` and `airon.finish`.
15
+ - Non-blocking logging through a bounded queue and a background sender with batching, gzip,
16
+ sequence numbers, retries with backoff and `Retry-After`, and the protocol's drop rules.
17
+ - Offline spool, automatic redelivery from the running process, and `airon sync`.
18
+ - Failure capture (`failed` with traceback, `killed` on `SIGTERM`) that chains existing hooks.
19
+ - Console tail capture that keeps carriage-return progress bars intact.
20
+ - Hardware sampling (NVML or `nvidia-smi`, `/proc` or `psutil`) and environment capture.
21
+ - `airon login`, `logout`, `doctor`, `sync` and `version`.
airon-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Manifestro
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
airon-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,212 @@
1
+ Metadata-Version: 2.5
2
+ Name: airon
3
+ Version: 0.1.0
4
+ Summary: Track machine learning runs: metrics, progress and hardware load, sent to an Airon server without ever slowing training down.
5
+ Project-URL: Homepage, https://github.com/Manifestro/airon-metrics
6
+ Project-URL: Issues, https://github.com/Manifestro/airon-metrics/issues
7
+ Project-URL: Changelog, https://github.com/Manifestro/airon-metrics/blob/main/CHANGELOG.md
8
+ Author: Manifestro
9
+ License: MIT License
10
+
11
+ Copyright (c) 2026 Manifestro
12
+
13
+ Permission is hereby granted, free of charge, to any person obtaining a copy
14
+ of this software and associated documentation files (the "Software"), to deal
15
+ in the Software without restriction, including without limitation the rights
16
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
17
+ copies of the Software, and to permit persons to whom the Software is
18
+ furnished to do so, subject to the following conditions:
19
+
20
+ The above copyright notice and this permission notice shall be included in all
21
+ copies or substantial portions of the Software.
22
+
23
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
24
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
25
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
26
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
27
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
28
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
29
+ SOFTWARE.
30
+ License-File: LICENSE
31
+ Keywords: experiment-tracking,machine-learning,metrics,mlops,monitoring
32
+ Classifier: Development Status :: 4 - Beta
33
+ Classifier: Intended Audience :: Science/Research
34
+ Classifier: License :: OSI Approved :: MIT License
35
+ Classifier: Operating System :: MacOS
36
+ Classifier: Operating System :: POSIX :: Linux
37
+ Classifier: Programming Language :: Python :: 3
38
+ Classifier: Programming Language :: Python :: 3.9
39
+ Classifier: Programming Language :: Python :: 3.10
40
+ Classifier: Programming Language :: Python :: 3.11
41
+ Classifier: Programming Language :: Python :: 3.12
42
+ Classifier: Programming Language :: Python :: 3.13
43
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
44
+ Classifier: Typing :: Typed
45
+ Requires-Python: >=3.9
46
+ Provides-Extra: dev
47
+ Requires-Dist: build; extra == 'dev'
48
+ Requires-Dist: mypy; extra == 'dev'
49
+ Requires-Dist: pytest-timeout; extra == 'dev'
50
+ Requires-Dist: pytest>=7; extra == 'dev'
51
+ Requires-Dist: ruff; extra == 'dev'
52
+ Requires-Dist: twine; extra == 'dev'
53
+ Provides-Extra: gpu
54
+ Requires-Dist: nvidia-ml-py; extra == 'gpu'
55
+ Provides-Extra: system
56
+ Requires-Dist: psutil; extra == 'system'
57
+ Description-Content-Type: text/markdown
58
+
59
+ # airon
60
+
61
+ Track machine learning runs from a plain Python training loop. `airon` sends metrics, progress and
62
+ hardware load to an [Airon](https://github.com/Manifestro/airon) server, where people watch loss,
63
+ ETA and GPU usage live.
64
+
65
+ - **One install, two variables, three lines.**
66
+ - **Never in the way of training.** Logging is a queue append; the network, the server and the disk
67
+ cannot slow down or break your loop (see [the guarantees](#it-will-never-break-your-training)).
68
+ - **Built for rented GPUs.** Flaky networks, spot preemption and pods you cannot SSH into are the
69
+ normal case: data that cannot be sent is kept on disk and uploaded later.
70
+ - **No required dependencies.** Python 3.9 to 3.13, standard library only.
71
+
72
+ ## Install
73
+
74
+ ```bash
75
+ pip install airon
76
+ ```
77
+
78
+ Optional extras give better hardware readings: `pip install "airon[gpu]"` (NVML through
79
+ `nvidia-ml-py`) and `pip install "airon[system]"` (`psutil`, which also enables CPU and RAM on
80
+ macOS). Without them the SDK falls back to `nvidia-smi` and `/proc`.
81
+
82
+ ## Set up
83
+
84
+ ```bash
85
+ export AIRON_URL=https://airon.example.com
86
+ export AIRON_API_KEY=... # a key that may log runs
87
+ ```
88
+
89
+ Or run `airon login` once on a machine, which asks for both and saves them in
90
+ `~/.config/airon/credentials.json` (readable only by you). `airon doctor` checks the setup and tells
91
+ you what to fix. There is no default server: if nothing is configured, `airon.init` says so and
92
+ saves the run on disk instead of failing.
93
+
94
+ ## Use
95
+
96
+ ```python
97
+ import airon
98
+
99
+ with airon.init(
100
+ project="memory-layer",
101
+ name="baseline-v2",
102
+ config={"lr": 3e-5, "batch_size": 32},
103
+ total_steps=50_000,
104
+ total_epochs=5,
105
+ ) as run:
106
+ for step, batch in enumerate(loader):
107
+ loss = train_step(batch)
108
+ run.log(
109
+ {"train/loss": loss, "lr": scheduler.get_last_lr()[0]},
110
+ step=step,
111
+ epoch=step / steps_per_epoch,
112
+ samples=32,
113
+ tokens=32 * 512,
114
+ )
115
+ if step % 1000 == 0:
116
+ run.log({"eval/loss": evaluate()}, step=step)
117
+ ```
118
+
119
+ Leaving the `with` block normally marks the run `finished`; an exception marks it `failed` with the
120
+ traceback and is raised again. Without `with`, call `run.finish()` (or do nothing: a normal exit
121
+ finishes the run too). Metric values can be Python numbers, numpy or torch scalars and tensors with
122
+ one element; `NaN` and infinity are recorded, not dropped.
123
+
124
+ More patterns, such as eval loops, resuming, checkpoint pointers and disabling, are in
125
+ [docs/usage.md](docs/usage.md). Runnable examples: [`examples/custom_loop.py`](examples/custom_loop.py)
126
+ (no dependencies) and [`examples/pytorch_loop.py`](examples/pytorch_loop.py).
127
+
128
+ ## What is collected automatically
129
+
130
+ | What | Details |
131
+ |---|---|
132
+ | Environment | Python and platform, host name, GPU names and memory, git commit, branch, dirty flag and remote (credentials in a remote URL are removed), versions of well-known libraries that your program has **already imported** (airon never imports torch itself), and the cloud when it can tell (`RUNPOD_POD_ID` gives `runpod`; override with `AIRON_CLOUD`) |
133
+ | Hardware, every 10 s | `gpu{i}/util_pct`, `mem_used_mb`, `mem_total_mb`, `temp_c`, `power_w`; `cpu/util_pct`; `ram/used_mb`, `ram/total_mb`; `disk/used_gb`, `disk/total_gb` for the working directory. Inside a container the RAM figures follow the container's memory limit |
134
+ | Progress | `steps_per_sec` (smoothed, from step deltas over time, so logging every 100 steps is fine), and `samples_per_sec` and `tokens_per_sec` when you pass `samples` and `tokens`. The server computes the ETA |
135
+ | Console | The last lines of stdout and stderr, with `\r` progress bars (tqdm) collapsed to their final state. Output is never swallowed. Turn off with `console=False` |
136
+ | Heartbeat | Every 30 s when nothing else was sent, so the server can tell a quiet run from a dead one |
137
+
138
+ Environment variables are never sent wholesale, and config values whose key looks like a secret
139
+ (`token`, `password`, `api_key`, ...) are replaced with `***`.
140
+
141
+ ## Offline behaviour
142
+
143
+ Whatever cannot be delivered goes to JSON lines under `~/.cache/airon/runs/<run id>/`
144
+ (`$AIRON_DIR` changes the location). While the server is unreachable the running process keeps
145
+ trying and uploads the backlog, in order, as soon as it answers. After a crash or a long outage run
146
+ `airon sync` to upload what is left. `AIRON_MODE=offline` writes only to disk, for pods without
147
+ internet.
148
+
149
+ ## Distributed training
150
+
151
+ With `torchrun` or similar only rank zero logs by default: processes with `RANK` or `LOCAL_RANK`
152
+ other than 0 receive a run object whose methods do nothing, so the same script works everywhere.
153
+ Pass `rank_zero_only=False` to log from every process.
154
+
155
+ ## It will never break your training
156
+
157
+ The first requirement of this SDK, and each point has tests:
158
+
159
+ - `log()` is an append to a bounded in-memory queue: a few microseconds (median about 4 us on a
160
+ laptop) under a lock that is held only for the append. No network or disk work happens in your
161
+ thread; a background thread sends.
162
+ - Nothing raises into your code because of the network, the server or the disk: down, slow, wrong
163
+ address, DNS failure, TLS errors, garbage replies, a read-only or full disk, and 401/403/409/5xx
164
+ answers are all handled. Bad arguments to `init` or `log` produce at most one warning.
165
+ - At most one warning per kind of problem is printed, and the API key is removed from every message.
166
+ - Memory is bounded: at 100,000 queued events the oldest hardware samples go first, then the oldest
167
+ metrics, with one warning.
168
+ - Retries use exponential backoff with jitter (1 s to 60 s) and honour `Retry-After`; retrying is
169
+ safe because the server ignores sequence numbers it already has.
170
+ - At exit, flushing waits at most 10 seconds; whatever is left goes to disk.
171
+ - Your exception hook and `SIGTERM` handler are chained, never replaced: `failed` (with traceback) is
172
+ reported after your hook has printed the error, and `killed` after your own handler has run.
173
+ - Threads are daemon threads; a forked child never reports for its parent.
174
+ - `AIRON_MODE=disabled` makes every call a no-op.
175
+
176
+ ## Environment variables
177
+
178
+ | Variable | Meaning |
179
+ |---|---|
180
+ | `AIRON_URL` | Server address, for example `https://airon.example.com` |
181
+ | `AIRON_API_KEY` | The API key |
182
+ | `AIRON_PROJECT` | Default project (otherwise the name of the working directory) |
183
+ | `AIRON_RUN_ID` | Run to resume or attach to |
184
+ | `AIRON_MODE` | `online` (default), `offline` or `disabled` |
185
+ | `AIRON_DIR` | Where the spool and the credentials file live (default `~/.cache/airon` and `~/.config/airon`) |
186
+ | `AIRON_CLOUD` | Overrides the detected cloud name |
187
+ | `AIRON_QUIET` | `1` hides the start and finish lines (warnings still appear) |
188
+
189
+ Arguments to `airon.init` beat environment variables, which beat the credentials file.
190
+
191
+ ## Command line
192
+
193
+ | Command | |
194
+ |---|---|
195
+ | `airon login` | Ask for the address and key, check them with the server, save them |
196
+ | `airon logout` | Delete the saved credentials |
197
+ | `airon doctor` | Check address, key, reachability and protocol version, write a tiny test run and print its link, with a fix for each failure |
198
+ | `airon sync` | Upload runs that were saved on disk |
199
+ | `airon version` | Print the version |
200
+
201
+ Exit codes: 0 success, 1 a check or an upload failed, 2 wrong usage or nothing configured.
202
+
203
+ ## Links
204
+
205
+ - [Usage guide](docs/usage.md)
206
+ - [Wire protocol](docs/protocol.md), the contract with the server
207
+ - [Releasing](docs/releasing.md), [Contributing](CONTRIBUTING.md), [Security](SECURITY.md), [Changelog](CHANGELOG.md)
208
+ - Server: [Manifestro/airon](https://github.com/Manifestro/airon)
209
+
210
+ ## License
211
+
212
+ [MIT](LICENSE)
airon-0.1.0/README.md ADDED
@@ -0,0 +1,154 @@
1
+ # airon
2
+
3
+ Track machine learning runs from a plain Python training loop. `airon` sends metrics, progress and
4
+ hardware load to an [Airon](https://github.com/Manifestro/airon) server, where people watch loss,
5
+ ETA and GPU usage live.
6
+
7
+ - **One install, two variables, three lines.**
8
+ - **Never in the way of training.** Logging is a queue append; the network, the server and the disk
9
+ cannot slow down or break your loop (see [the guarantees](#it-will-never-break-your-training)).
10
+ - **Built for rented GPUs.** Flaky networks, spot preemption and pods you cannot SSH into are the
11
+ normal case: data that cannot be sent is kept on disk and uploaded later.
12
+ - **No required dependencies.** Python 3.9 to 3.13, standard library only.
13
+
14
+ ## Install
15
+
16
+ ```bash
17
+ pip install airon
18
+ ```
19
+
20
+ Optional extras give better hardware readings: `pip install "airon[gpu]"` (NVML through
21
+ `nvidia-ml-py`) and `pip install "airon[system]"` (`psutil`, which also enables CPU and RAM on
22
+ macOS). Without them the SDK falls back to `nvidia-smi` and `/proc`.
23
+
24
+ ## Set up
25
+
26
+ ```bash
27
+ export AIRON_URL=https://airon.example.com
28
+ export AIRON_API_KEY=... # a key that may log runs
29
+ ```
30
+
31
+ Or run `airon login` once on a machine, which asks for both and saves them in
32
+ `~/.config/airon/credentials.json` (readable only by you). `airon doctor` checks the setup and tells
33
+ you what to fix. There is no default server: if nothing is configured, `airon.init` says so and
34
+ saves the run on disk instead of failing.
35
+
36
+ ## Use
37
+
38
+ ```python
39
+ import airon
40
+
41
+ with airon.init(
42
+ project="memory-layer",
43
+ name="baseline-v2",
44
+ config={"lr": 3e-5, "batch_size": 32},
45
+ total_steps=50_000,
46
+ total_epochs=5,
47
+ ) as run:
48
+ for step, batch in enumerate(loader):
49
+ loss = train_step(batch)
50
+ run.log(
51
+ {"train/loss": loss, "lr": scheduler.get_last_lr()[0]},
52
+ step=step,
53
+ epoch=step / steps_per_epoch,
54
+ samples=32,
55
+ tokens=32 * 512,
56
+ )
57
+ if step % 1000 == 0:
58
+ run.log({"eval/loss": evaluate()}, step=step)
59
+ ```
60
+
61
+ Leaving the `with` block normally marks the run `finished`; an exception marks it `failed` with the
62
+ traceback and is raised again. Without `with`, call `run.finish()` (or do nothing: a normal exit
63
+ finishes the run too). Metric values can be Python numbers, numpy or torch scalars and tensors with
64
+ one element; `NaN` and infinity are recorded, not dropped.
65
+
66
+ More patterns, such as eval loops, resuming, checkpoint pointers and disabling, are in
67
+ [docs/usage.md](docs/usage.md). Runnable examples: [`examples/custom_loop.py`](examples/custom_loop.py)
68
+ (no dependencies) and [`examples/pytorch_loop.py`](examples/pytorch_loop.py).
69
+
70
+ ## What is collected automatically
71
+
72
+ | What | Details |
73
+ |---|---|
74
+ | Environment | Python and platform, host name, GPU names and memory, git commit, branch, dirty flag and remote (credentials in a remote URL are removed), versions of well-known libraries that your program has **already imported** (airon never imports torch itself), and the cloud when it can tell (`RUNPOD_POD_ID` gives `runpod`; override with `AIRON_CLOUD`) |
75
+ | Hardware, every 10 s | `gpu{i}/util_pct`, `mem_used_mb`, `mem_total_mb`, `temp_c`, `power_w`; `cpu/util_pct`; `ram/used_mb`, `ram/total_mb`; `disk/used_gb`, `disk/total_gb` for the working directory. Inside a container the RAM figures follow the container's memory limit |
76
+ | Progress | `steps_per_sec` (smoothed, from step deltas over time, so logging every 100 steps is fine), and `samples_per_sec` and `tokens_per_sec` when you pass `samples` and `tokens`. The server computes the ETA |
77
+ | Console | The last lines of stdout and stderr, with `\r` progress bars (tqdm) collapsed to their final state. Output is never swallowed. Turn off with `console=False` |
78
+ | Heartbeat | Every 30 s when nothing else was sent, so the server can tell a quiet run from a dead one |
79
+
80
+ Environment variables are never sent wholesale, and config values whose key looks like a secret
81
+ (`token`, `password`, `api_key`, ...) are replaced with `***`.
82
+
83
+ ## Offline behaviour
84
+
85
+ Whatever cannot be delivered goes to JSON lines under `~/.cache/airon/runs/<run id>/`
86
+ (`$AIRON_DIR` changes the location). While the server is unreachable the running process keeps
87
+ trying and uploads the backlog, in order, as soon as it answers. After a crash or a long outage run
88
+ `airon sync` to upload what is left. `AIRON_MODE=offline` writes only to disk, for pods without
89
+ internet.
90
+
91
+ ## Distributed training
92
+
93
+ With `torchrun` or similar only rank zero logs by default: processes with `RANK` or `LOCAL_RANK`
94
+ other than 0 receive a run object whose methods do nothing, so the same script works everywhere.
95
+ Pass `rank_zero_only=False` to log from every process.
96
+
97
+ ## It will never break your training
98
+
99
+ The first requirement of this SDK, and each point has tests:
100
+
101
+ - `log()` is an append to a bounded in-memory queue: a few microseconds (median about 4 us on a
102
+ laptop) under a lock that is held only for the append. No network or disk work happens in your
103
+ thread; a background thread sends.
104
+ - Nothing raises into your code because of the network, the server or the disk: down, slow, wrong
105
+ address, DNS failure, TLS errors, garbage replies, a read-only or full disk, and 401/403/409/5xx
106
+ answers are all handled. Bad arguments to `init` or `log` produce at most one warning.
107
+ - At most one warning per kind of problem is printed, and the API key is removed from every message.
108
+ - Memory is bounded: at 100,000 queued events the oldest hardware samples go first, then the oldest
109
+ metrics, with one warning.
110
+ - Retries use exponential backoff with jitter (1 s to 60 s) and honour `Retry-After`; retrying is
111
+ safe because the server ignores sequence numbers it already has.
112
+ - At exit, flushing waits at most 10 seconds; whatever is left goes to disk.
113
+ - Your exception hook and `SIGTERM` handler are chained, never replaced: `failed` (with traceback) is
114
+ reported after your hook has printed the error, and `killed` after your own handler has run.
115
+ - Threads are daemon threads; a forked child never reports for its parent.
116
+ - `AIRON_MODE=disabled` makes every call a no-op.
117
+
118
+ ## Environment variables
119
+
120
+ | Variable | Meaning |
121
+ |---|---|
122
+ | `AIRON_URL` | Server address, for example `https://airon.example.com` |
123
+ | `AIRON_API_KEY` | The API key |
124
+ | `AIRON_PROJECT` | Default project (otherwise the name of the working directory) |
125
+ | `AIRON_RUN_ID` | Run to resume or attach to |
126
+ | `AIRON_MODE` | `online` (default), `offline` or `disabled` |
127
+ | `AIRON_DIR` | Where the spool and the credentials file live (default `~/.cache/airon` and `~/.config/airon`) |
128
+ | `AIRON_CLOUD` | Overrides the detected cloud name |
129
+ | `AIRON_QUIET` | `1` hides the start and finish lines (warnings still appear) |
130
+
131
+ Arguments to `airon.init` beat environment variables, which beat the credentials file.
132
+
133
+ ## Command line
134
+
135
+ | Command | |
136
+ |---|---|
137
+ | `airon login` | Ask for the address and key, check them with the server, save them |
138
+ | `airon logout` | Delete the saved credentials |
139
+ | `airon doctor` | Check address, key, reachability and protocol version, write a tiny test run and print its link, with a fix for each failure |
140
+ | `airon sync` | Upload runs that were saved on disk |
141
+ | `airon version` | Print the version |
142
+
143
+ Exit codes: 0 success, 1 a check or an upload failed, 2 wrong usage or nothing configured.
144
+
145
+ ## Links
146
+
147
+ - [Usage guide](docs/usage.md)
148
+ - [Wire protocol](docs/protocol.md), the contract with the server
149
+ - [Releasing](docs/releasing.md), [Contributing](CONTRIBUTING.md), [Security](SECURITY.md), [Changelog](CHANGELOG.md)
150
+ - Server: [Manifestro/airon](https://github.com/Manifestro/airon)
151
+
152
+ ## License
153
+
154
+ [MIT](LICENSE)
@@ -0,0 +1,186 @@
1
+ # Wire protocol, version 1
2
+
3
+ This document is the contract between the `airon` Python SDK (this repository) and an Airon server.
4
+ Both sides follow it. Where the two disagree, this document wins; open an issue or a pull request to
5
+ change it.
6
+
7
+ Conventions
8
+
9
+ - Bodies are JSON, UTF-8. Request bodies larger than 1 KiB should be gzip-compressed with
10
+ `Content-Encoding: gzip`.
11
+ - Timestamps are Unix seconds as floats, UTC.
12
+ - Unknown fields must be ignored by both sides, so the protocol can grow without breaking old clients.
13
+ - Metric names are hierarchical with `/`, at most 200 characters, for example `train/loss`.
14
+ - JSON cannot carry `NaN` or infinity. A metric value that is not finite is sent as the string
15
+ `"NaN"`, `"Infinity"` or `"-Infinity"`. The server records it (it is a signal worth showing) and
16
+ leaves it out of charts.
17
+ - The examples use `https://airon.example.com`. The SDK has **no default server**: the address always
18
+ comes from `AIRON_URL` or `airon login`.
19
+
20
+ ## Authentication and versioning
21
+
22
+ - `Authorization: Bearer <api key>` on every request. Keys never appear in URLs or bodies.
23
+ - `401` invalid or revoked key. `403` the key may not log runs (read-only or wrong scope).
24
+ - Clients send `X-Airon-Protocol: 1` and `User-Agent: airon/<sdk version> python/<python version>`.
25
+ Servers answer with `X-Airon-Protocol: 1`. A server that cannot speak the client's version answers `426`.
26
+
27
+ ## GET /v1/ping
28
+
29
+ Used by `airon doctor` and `airon login` to check the address and the key.
30
+
31
+ ```json
32
+ {"ok": true, "protocol": 1, "key": {"name": "pod-h100", "scope": "tracker"}, "server_time": 1730402880.1}
33
+ ```
34
+
35
+ ## POST /v1/runs: create or reopen a run
36
+
37
+ Idempotent by `run_id`, so a retried request is harmless. The client generates `run_id` (a UUID4 string).
38
+
39
+ ```json
40
+ {
41
+ "run_id": "6f1c1c0e-3f55-4d66-9a8e-2f0f0d8f0b11",
42
+ "project": "memory-layer",
43
+ "name": "baseline-v2",
44
+ "experiment": "Baseline v2",
45
+ "config": {"lr": 3e-5, "batch_size": 32, "model": {"name": "small-encoder"}},
46
+ "env": {
47
+ "python": "3.11.6",
48
+ "platform": "Linux-5.15-x86_64",
49
+ "host": "pod-8c1f",
50
+ "cloud": "runpod",
51
+ "gpus": [{"index": 0, "name": "NVIDIA H100 80GB HBM3", "mem_total_mb": 81559}],
52
+ "git": {"commit": "5d1e0aa", "branch": "main", "dirty": false, "remote": "git@example.com:acme/model-code.git"},
53
+ "libs": {"torch": "2.5.0", "numpy": "2.1.0"}
54
+ },
55
+ "total_steps": 50000,
56
+ "total_epochs": 5,
57
+ "tags": ["baseline"],
58
+ "notes": "",
59
+ "resume": "never",
60
+ "started_at": 1730398080.0
61
+ }
62
+ ```
63
+
64
+ Only `run_id` and `project` are required. Everything else is optional. `name` is generated by the server
65
+ when missing. `experiment` is the title of an Airon experiment to link the run to; if there is none, the
66
+ server ignores it and says so in `warnings`. `resume` is `"never"` (default), `"allow"` or `"must"`.
67
+
68
+ | Situation | Result |
69
+ |---|---|
70
+ | `run_id` is new, `resume` is not `"must"` | `201`, a new run |
71
+ | `run_id` exists and the run is still running | `200`, an idempotent update (covers retries) |
72
+ | `run_id` exists and is finished, failed, killed or crashed, `resume` is `"allow"` or `"must"` | `200`, reopened, `resumed: true` |
73
+ | `run_id` exists and is over, `resume` is `"never"` | `409` |
74
+ | `run_id` is new and `resume` is `"must"` | `404` |
75
+
76
+ ```json
77
+ {"run_id": "6f1c…", "name": "baseline-v2", "url": "https://airon.example.com/runs/6f1c…",
78
+ "status": "running", "resumed": false, "last_step": null, "last_seq": 0, "warnings": [],
79
+ "server_time": 1730402880.2}
80
+ ```
81
+
82
+ `last_step` and `last_seq` let a resumed run continue where the server stopped: the client starts `seq`
83
+ after `last_seq`.
84
+
85
+ ## POST /v1/runs/{run_id}/events: send a batch
86
+
87
+ ```json
88
+ {"events": [
89
+ {"seq": 41, "t": 1730402880.0, "step": 12400, "epoch": 1.98,
90
+ "metrics": {"train/loss": 0.412, "lr": 2.4e-5}},
91
+ {"seq": 42, "t": 1730402880.0, "system": {"gpu0/util_pct": 96, "gpu0/mem_used_mb": 71234, "cpu/util_pct": 38}},
92
+ {"seq": 43, "t": 1730402881.0,
93
+ "progress": {"step": 12400, "total_steps": 50000, "epoch": 1.98, "total_epochs": 5,
94
+ "steps_per_sec": 2.4, "samples_per_sec": 76.8, "tokens_per_sec": 18200}}
95
+ ]}
96
+ ```
97
+
98
+ At most 500 events and 1 MiB (uncompressed) per request.
99
+
100
+ Every event has `seq` (integer, increasing by 1 per run, assigned by the client) and `t`. The server
101
+ ignores an event whose `seq` it already has, which makes retries safe. An event can carry any of these
102
+ keys, several at once:
103
+
104
+ | Key | Content |
105
+ |---|---|
106
+ | `metrics` | `{name: number}` at an optional `step` (integer) and `epoch` (float, may be fractional) |
107
+ | `system` | `{name: number}` hardware samples, kept off the step axis (see well-known names) |
108
+ | `progress` | `step`, `total_steps`, `epoch`, `total_epochs`, `steps_per_sec`, `samples_per_sec`, `tokens_per_sec`, all optional |
109
+ | `log` | `[{"stream": "stdout" or "stderr", "text": "one line"}]` console lines |
110
+ | `checkpoint` | `{"uri": "hf://org/repo@rev", "step": 12000, "label": "best", "metrics": {}}` a pointer, never the file |
111
+ | `summary` | `{name: any JSON}` values set explicitly by the user |
112
+ | `config` | `{...}` merged into the run's config, key by key |
113
+
114
+ An event with only `seq` and `t` is a heartbeat. The server derives these without help: the ETA (from
115
+ the recent progress rate, so pauses for evaluation and checkpoints are included), the best and last
116
+ value of each metric, and the `crashed` state (no event for 5 minutes while still `running`).
117
+
118
+ Response `200`, even when some events were refused, so one bad event cannot block the queue:
119
+
120
+ ```json
121
+ {"accepted": 3, "duplicates": 0, "last_seq": 43, "rejected": []}
122
+ ```
123
+
124
+ A rejected event looks like `{"seq": 44, "reason": "metrics.train/loss is not a number"}`. The client
125
+ must not resend rejected events.
126
+
127
+ Errors: `401`, `403`, `404` unknown run, `413` body too large (the client splits the batch and retries),
128
+ `422` malformed request (not retried), `429` with `Retry-After` (seconds), `5xx`.
129
+
130
+ ## POST /v1/runs/{run_id}/finish
131
+
132
+ ```json
133
+ {"status": "failed", "t": 1730412345.0, "exit_code": 1,
134
+ "error": {"type": "torch.OutOfMemoryError", "message": "CUDA out of memory…", "traceback": "Traceback (most recent call last):\n…"},
135
+ "summary": {"eval/loss.best": 0.431}}
136
+ ```
137
+
138
+ `status` is `"finished"`, `"failed"` or `"killed"` (for example a terminated pod). Idempotent. Events that
139
+ arrive after it are still recorded; the status does not change. `"crashed"` is never sent by a client:
140
+ the server sets it when a running run goes silent.
141
+
142
+ ## Well-known names
143
+
144
+ These get special treatment in the interface. Everything else is shown too.
145
+
146
+ - Metrics: `train/loss`, `eval/loss`, `eval/*`, `train/*`, `lr` (or `train/lr`), `train/grad_norm`.
147
+ - System: `gpu{i}/util_pct`, `gpu{i}/mem_used_mb`, `gpu{i}/mem_total_mb`, `gpu{i}/temp_c`,
148
+ `gpu{i}/power_w`, `cpu/util_pct`, `ram/used_mb`, `ram/total_mb`, `disk/used_gb`, `disk/total_gb`.
149
+
150
+ ## Client behaviour
151
+
152
+ The SDK must follow these rules. They are what lets a team adopt it without worrying about their training.
153
+
154
+ 1. **Never break training.** `log()` only appends to an in-memory queue and returns. Nothing in the
155
+ SDK raises into user code because of the network, the server or the disk. At most one warning is
156
+ printed per kind of problem.
157
+ 2. **Bounded memory.** The queue holds at most 100,000 events. When full, drop the oldest `system`
158
+ events first, then the oldest `metrics`, and warn once.
159
+ 3. **Batching.** A background thread flushes every 2 seconds or 200 events, whichever comes first,
160
+ sends a heartbeat every 30 seconds, and samples hardware every 10 seconds. All configurable.
161
+ 4. **Retries.** Exponential backoff from 1 s to 60 s with jitter, honouring `Retry-After`. Retry on
162
+ network errors, `408`, `425`, `429` and `5xx`. Do not retry other `4xx`: drop the batch and warn. On
163
+ `401` or `403` stop sending, keep everything in the offline spool, and warn once.
164
+ 5. **Offline spool.** Events that cannot be delivered are written to JSON lines under `$AIRON_DIR`
165
+ (default `~/.cache/airon`), one directory per run. `airon sync` uploads what is there, in order.
166
+ `AIRON_MODE=offline` writes only to the spool; `AIRON_MODE=disabled` makes every call a no-op.
167
+ 6. **Exit.** Flush on normal exit and on an unhandled exception, waiting at most 10 seconds. An unhandled
168
+ exception inside `with airon.init(...)` is reported as `failed` with its traceback and re-raised.
169
+ `SIGTERM` is reported as `killed` after the program's own handler has run: chain existing handlers,
170
+ never replace them.
171
+ 7. **Console.** Capture the tail of stdout and stderr (default 200 lines) without breaking progress bars
172
+ that use carriage returns. It can be turned off with `console=False`.
173
+
174
+ ## Environment variables
175
+
176
+ | Variable | Meaning |
177
+ |---|---|
178
+ | `AIRON_URL` | Server address, for example `https://airon.example.com` |
179
+ | `AIRON_API_KEY` | The key |
180
+ | `AIRON_PROJECT` | Default project |
181
+ | `AIRON_RUN_ID` | Run to resume or attach to |
182
+ | `AIRON_MODE` | `online` (default), `offline` or `disabled` |
183
+ | `AIRON_DIR` | Where the spool and credentials live |
184
+
185
+ `airon login` stores the address and the key in `~/.config/airon/credentials.json`, readable only by the
186
+ owner. Environment variables take precedence over the file.
@@ -0,0 +1,37 @@
1
+ # Releasing
2
+
3
+ Releases are built and published by GitHub Actions when a tag `v*` is pushed. Publishing uses PyPI
4
+ [trusted publishing](https://docs.pypi.org/trusted-publishers/), so no token is stored anywhere.
5
+
6
+ ## One-time setup (the repository owner)
7
+
8
+ 1. On PyPI, create the project `airon`, or add a *pending publisher* if it does not exist yet:
9
+ **Your account > Publishing > Add a new pending publisher**.
10
+ 2. Fill in:
11
+ - PyPI project name: `airon`
12
+ - Owner: `Manifestro`
13
+ - Repository name: `airon-metrics`
14
+ - Workflow name: `release.yml`
15
+ - Environment name: `pypi`
16
+ 3. In the GitHub repository, **Settings > Environments > New environment** named `pypi`. Add
17
+ *required reviewers* if every release should need an approval.
18
+
19
+ To rehearse on TestPyPI, repeat step 1 and 2 on test.pypi.org and temporarily point the workflow's
20
+ `repository-url` at `https://test.pypi.org/legacy/`.
21
+
22
+ ## Making a release
23
+
24
+ 1. Update `src/airon/_version.py` and `CHANGELOG.md` (move *unreleased* to the new version and date).
25
+ 2. Commit and merge to `main`; wait for CI to pass.
26
+ 3. Tag and push:
27
+
28
+ ```bash
29
+ git tag v0.1.0
30
+ git push origin v0.1.0
31
+ ```
32
+
33
+ 4. The `Release` workflow builds the sdist and wheel, checks the tag matches the package version,
34
+ runs `twine check`, and publishes (after approval, if you set required reviewers).
35
+ 5. Verify in a clean environment: `pip install airon==0.1.0 && airon version`.
36
+
37
+ A mistake in a published version cannot be replaced: yank it on PyPI and release a new patch version.