runlab-client 0.2.1__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {runlab_client-0.2.1/src/runlab_client.egg-info → runlab_client-0.3.0}/PKG-INFO +59 -67
- {runlab_client-0.2.1 → runlab_client-0.3.0}/README.md +58 -66
- {runlab_client-0.2.1 → runlab_client-0.3.0}/pyproject.toml +1 -1
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client/__init__.py +74 -5
- {runlab_client-0.2.1 → runlab_client-0.3.0/src/runlab_client.egg-info}/PKG-INFO +59 -67
- {runlab_client-0.2.1 → runlab_client-0.3.0}/LICENSE +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/setup.cfg +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client/__main__.py +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client.egg-info/SOURCES.txt +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client.egg-info/dependency_links.txt +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client.egg-info/entry_points.txt +0 -0
- {runlab_client-0.2.1 → runlab_client-0.3.0}/src/runlab_client.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: runlab-client
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
|
|
5
5
|
Author: omer-nachman
|
|
6
6
|
License-Expression: MIT
|
|
@@ -12,6 +12,10 @@ Dynamic: license-file
|
|
|
12
12
|
|
|
13
13
|
# runlab-client
|
|
14
14
|
|
|
15
|
+
[](https://pypi.org/project/runlab-client/)
|
|
16
|
+

|
|
17
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
18
|
+
|
|
15
19
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
16
20
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
17
21
|
|
|
@@ -19,9 +23,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
19
23
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
20
24
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
21
25
|
|
|
22
|
-
The RunLab Python client gives access to the RunLab API using only the standard library,
|
|
23
|
-
on Python 3.10+.
|
|
24
|
-
|
|
25
26
|
## Documentation
|
|
26
27
|
|
|
27
28
|
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
@@ -30,36 +31,30 @@ on Python 3.10+.
|
|
|
30
31
|
## Get started
|
|
31
32
|
|
|
32
33
|
```bash
|
|
33
|
-
pip install runlab-client
|
|
34
|
-
runlab
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
34
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
35
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
39
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
43
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
44
|
+
...
|
|
45
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
46
|
+
The verdict page: https://runlab.tools/v/...
|
|
39
47
|
```
|
|
40
48
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
`runlab-client --version` names the version.
|
|
44
|
-
- `runlab login` opens a browser when this machine has a display. On an SSH session or a
|
|
45
|
-
headless GPU box, open the printed link on any device (your laptop or phone).
|
|
46
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
47
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
48
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
49
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
50
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
51
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
52
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
53
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
54
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
49
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
50
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
55
51
|
|
|
56
52
|
## Quickstart
|
|
57
53
|
|
|
58
54
|
```python
|
|
59
55
|
from runlab_client import RunLab
|
|
60
56
|
|
|
61
|
-
rl = RunLab() #
|
|
62
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
57
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
63
58
|
|
|
64
59
|
# 1. Pre-register, before any seed runs.
|
|
65
60
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -83,11 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
83
78
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
84
79
|
```
|
|
85
80
|
|
|
86
|
-
`train` is your
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
81
|
+
`train` is your training run.
|
|
82
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
83
|
+
runs as is, with a simulated `train`;
|
|
84
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
85
|
+
is the same loop over plain HTTP.
|
|
91
86
|
|
|
92
87
|
## The loop
|
|
93
88
|
|
|
@@ -100,20 +95,6 @@ does the same loop over plain HTTP.
|
|
|
100
95
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
101
96
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
102
97
|
|
|
103
|
-
### Each user sees only their projects
|
|
104
|
-
|
|
105
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
106
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
107
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
108
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
109
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
110
|
-
|
|
111
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
112
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
113
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
114
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
115
|
-
`better="lower"` to override.
|
|
116
|
-
|
|
117
98
|
### Results
|
|
118
99
|
|
|
119
100
|
RunLab stores each result as one line:
|
|
@@ -130,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
130
111
|
`record()` as is.
|
|
131
112
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
132
113
|
a 400.
|
|
133
|
-
- Project names are case-insensitive
|
|
114
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
134
115
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
135
116
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
136
117
|
in configurations is refused (`refused_by: equal_budget`).
|
|
137
118
|
|
|
138
119
|
### Verdicts
|
|
139
120
|
|
|
121
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
122
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
123
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
124
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
125
|
+
|
|
140
126
|
| `verdict` | Meaning |
|
|
141
127
|
|---|---|
|
|
142
128
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -146,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
146
132
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
147
133
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
148
134
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
- `ok`;
|
|
153
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
154
|
-
after more than 3 seeds had been seen;
|
|
155
|
-
- `missing`: no plan;
|
|
156
|
-
- `tampered`.
|
|
157
|
-
- `warnings`, e.g. `peeking`.
|
|
158
|
-
- `report()`, before you quote anything.
|
|
135
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
136
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
137
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
159
138
|
|
|
160
139
|
### What keeps a verdict honest
|
|
161
140
|
|
|
@@ -169,8 +148,7 @@ Read these fields next to the verdict:
|
|
|
169
148
|
|
|
170
149
|
## Studies: let RunLab run the seeds
|
|
171
150
|
|
|
172
|
-
RunLab
|
|
173
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
151
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
174
152
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
175
153
|
|
|
176
154
|
```python
|
|
@@ -186,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
186
164
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
187
165
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
188
166
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
189
|
-
- A study never spends more than `max_usd`.
|
|
190
167
|
- `state` is one of:
|
|
191
168
|
|
|
192
169
|
| `state` | Meaning |
|
|
@@ -194,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
194
171
|
| `running` | Still running seeds. |
|
|
195
172
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
196
173
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
197
|
-
| `budget` |
|
|
198
|
-
| `killed` |
|
|
174
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
175
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
199
176
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
200
177
|
| `lost` | Stopped reporting long past its deadline. |
|
|
201
178
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -208,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
208
185
|
params' names are filled in (no shell runs it).
|
|
209
186
|
|
|
210
187
|
```bash
|
|
211
|
-
pip install runlab-client
|
|
212
|
-
runlab login
|
|
213
188
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
214
189
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
215
190
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -227,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
227
202
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
228
203
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
229
204
|
|
|
205
|
+
## Signing in
|
|
206
|
+
|
|
207
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
208
|
+
|
|
209
|
+
| Question | Answer |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
212
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
213
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
214
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
215
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
216
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
217
|
+
|
|
230
218
|
## Errors
|
|
231
219
|
|
|
232
220
|
Every failure raises `RunLabError` with:
|
|
@@ -239,20 +227,19 @@ Every failure raises `RunLabError` with:
|
|
|
239
227
|
|---|---|
|
|
240
228
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
241
229
|
| 401 | Missing or wrong token. |
|
|
242
|
-
| 403 |
|
|
230
|
+
| 403 | This key may not do this; the message says why. |
|
|
243
231
|
| 404 | No such verdict or study. |
|
|
244
232
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
245
233
|
|
|
246
234
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
247
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
248
|
-
harmless anyway.
|
|
235
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
249
236
|
|
|
250
237
|
## Client reference
|
|
251
238
|
|
|
252
239
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
253
240
|
the decoded JSON of one operation in
|
|
254
241
|
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
255
|
-
With no arguments it uses the key `runlab login` saved (see [
|
|
242
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
256
243
|
|
|
257
244
|
| Method | Operation | Returns |
|
|
258
245
|
|---|---|---|
|
|
@@ -275,6 +262,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
|
|
|
275
262
|
|
|
276
263
|
| Command | What it does |
|
|
277
264
|
|---|---|
|
|
265
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
278
266
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
279
267
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
280
268
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
@@ -289,6 +277,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
|
289
277
|
is the short version for Claude Code, Codex and other agents.
|
|
290
278
|
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
291
279
|
|
|
280
|
+
## Support
|
|
281
|
+
|
|
282
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
283
|
+
|
|
292
284
|
## License
|
|
293
285
|
|
|
294
286
|
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# runlab-client
|
|
2
2
|
|
|
3
|
+
[](https://pypi.org/project/runlab-client/)
|
|
4
|
+

|
|
5
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
6
|
+
|
|
3
7
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
4
8
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
5
9
|
|
|
@@ -7,9 +11,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
7
11
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
8
12
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
9
13
|
|
|
10
|
-
The RunLab Python client gives access to the RunLab API using only the standard library,
|
|
11
|
-
on Python 3.10+.
|
|
12
|
-
|
|
13
14
|
## Documentation
|
|
14
15
|
|
|
15
16
|
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
@@ -18,36 +19,30 @@ on Python 3.10+.
|
|
|
18
19
|
## Get started
|
|
19
20
|
|
|
20
21
|
```bash
|
|
21
|
-
pip install runlab-client
|
|
22
|
-
runlab
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
22
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
23
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
27
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
31
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
32
|
+
...
|
|
33
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
34
|
+
The verdict page: https://runlab.tools/v/...
|
|
27
35
|
```
|
|
28
36
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
`runlab-client --version` names the version.
|
|
32
|
-
- `runlab login` opens a browser when this machine has a display. On an SSH session or a
|
|
33
|
-
headless GPU box, open the printed link on any device (your laptop or phone).
|
|
34
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
35
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
36
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
37
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
38
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
39
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
40
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
41
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
42
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
37
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
38
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
43
39
|
|
|
44
40
|
## Quickstart
|
|
45
41
|
|
|
46
42
|
```python
|
|
47
43
|
from runlab_client import RunLab
|
|
48
44
|
|
|
49
|
-
rl = RunLab() #
|
|
50
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
45
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
51
46
|
|
|
52
47
|
# 1. Pre-register, before any seed runs.
|
|
53
48
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -71,11 +66,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
71
66
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
72
67
|
```
|
|
73
68
|
|
|
74
|
-
`train` is your
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
69
|
+
`train` is your training run.
|
|
70
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
71
|
+
runs as is, with a simulated `train`;
|
|
72
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
73
|
+
is the same loop over plain HTTP.
|
|
79
74
|
|
|
80
75
|
## The loop
|
|
81
76
|
|
|
@@ -88,20 +83,6 @@ does the same loop over plain HTTP.
|
|
|
88
83
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
89
84
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
90
85
|
|
|
91
|
-
### Each user sees only their projects
|
|
92
|
-
|
|
93
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
94
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
95
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
96
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
97
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
98
|
-
|
|
99
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
100
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
101
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
102
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
103
|
-
`better="lower"` to override.
|
|
104
|
-
|
|
105
86
|
### Results
|
|
106
87
|
|
|
107
88
|
RunLab stores each result as one line:
|
|
@@ -118,13 +99,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
118
99
|
`record()` as is.
|
|
119
100
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
120
101
|
a 400.
|
|
121
|
-
- Project names are case-insensitive
|
|
102
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
122
103
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
123
104
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
124
105
|
in configurations is refused (`refused_by: equal_budget`).
|
|
125
106
|
|
|
126
107
|
### Verdicts
|
|
127
108
|
|
|
109
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
110
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
111
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
112
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
113
|
+
|
|
128
114
|
| `verdict` | Meaning |
|
|
129
115
|
|---|---|
|
|
130
116
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -134,16 +120,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
134
120
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
135
121
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
136
122
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
- `ok`;
|
|
141
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
142
|
-
after more than 3 seeds had been seen;
|
|
143
|
-
- `missing`: no plan;
|
|
144
|
-
- `tampered`.
|
|
145
|
-
- `warnings`, e.g. `peeking`.
|
|
146
|
-
- `report()`, before you quote anything.
|
|
123
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
124
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
125
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
147
126
|
|
|
148
127
|
### What keeps a verdict honest
|
|
149
128
|
|
|
@@ -157,8 +136,7 @@ Read these fields next to the verdict:
|
|
|
157
136
|
|
|
158
137
|
## Studies: let RunLab run the seeds
|
|
159
138
|
|
|
160
|
-
RunLab
|
|
161
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
139
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
162
140
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
163
141
|
|
|
164
142
|
```python
|
|
@@ -174,7 +152,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
174
152
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
175
153
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
176
154
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
177
|
-
- A study never spends more than `max_usd`.
|
|
178
155
|
- `state` is one of:
|
|
179
156
|
|
|
180
157
|
| `state` | Meaning |
|
|
@@ -182,8 +159,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
182
159
|
| `running` | Still running seeds. |
|
|
183
160
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
184
161
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
185
|
-
| `budget` |
|
|
186
|
-
| `killed` |
|
|
162
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
163
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
187
164
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
188
165
|
| `lost` | Stopped reporting long past its deadline. |
|
|
189
166
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -196,8 +173,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
196
173
|
params' names are filled in (no shell runs it).
|
|
197
174
|
|
|
198
175
|
```bash
|
|
199
|
-
pip install runlab-client
|
|
200
|
-
runlab login
|
|
201
176
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
202
177
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
203
178
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -215,6 +190,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
215
190
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
216
191
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
217
192
|
|
|
193
|
+
## Signing in
|
|
194
|
+
|
|
195
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
196
|
+
|
|
197
|
+
| Question | Answer |
|
|
198
|
+
|---|---|
|
|
199
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
200
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
201
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
202
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
203
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
204
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
205
|
+
|
|
218
206
|
## Errors
|
|
219
207
|
|
|
220
208
|
Every failure raises `RunLabError` with:
|
|
@@ -227,20 +215,19 @@ Every failure raises `RunLabError` with:
|
|
|
227
215
|
|---|---|
|
|
228
216
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
229
217
|
| 401 | Missing or wrong token. |
|
|
230
|
-
| 403 |
|
|
218
|
+
| 403 | This key may not do this; the message says why. |
|
|
231
219
|
| 404 | No such verdict or study. |
|
|
232
220
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
233
221
|
|
|
234
222
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
235
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
236
|
-
harmless anyway.
|
|
223
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
237
224
|
|
|
238
225
|
## Client reference
|
|
239
226
|
|
|
240
227
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
241
228
|
the decoded JSON of one operation in
|
|
242
229
|
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
243
|
-
With no arguments it uses the key `runlab login` saved (see [
|
|
230
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
244
231
|
|
|
245
232
|
| Method | Operation | Returns |
|
|
246
233
|
|---|---|---|
|
|
@@ -263,6 +250,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
|
|
|
263
250
|
|
|
264
251
|
| Command | What it does |
|
|
265
252
|
|---|---|
|
|
253
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
266
254
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
267
255
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
268
256
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
@@ -277,6 +265,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
|
277
265
|
is the short version for Claude Code, Codex and other agents.
|
|
278
266
|
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
279
267
|
|
|
268
|
+
## Support
|
|
269
|
+
|
|
270
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
271
|
+
|
|
280
272
|
## License
|
|
281
273
|
|
|
282
274
|
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "runlab-client"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.3.0"
|
|
8
8
|
description = "Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.10"
|
|
@@ -1,12 +1,13 @@
|
|
|
1
1
|
"""runlab-client: Python client for the RunLab research API.
|
|
2
2
|
|
|
3
3
|
$ pip install runlab-client
|
|
4
|
-
$ runlab
|
|
4
|
+
$ runlab demo # GitHub sign-in, then a first verdict on simulated numbers
|
|
5
|
+
$ runlab login # sign in only; the key stays on this machine
|
|
5
6
|
$ runlab-client login # the same, when another `runlab` comes first on PATH
|
|
6
7
|
|
|
7
8
|
from runlab_client import RunLab
|
|
8
9
|
|
|
9
|
-
rl = RunLab() #
|
|
10
|
+
rl = RunLab() # the key `runlab demo` / `login` saved, or RUNLAB_TOKEN
|
|
10
11
|
rl.plan("demo", "treatment", "baseline", "acc", hypothesis="the change helps", n_max=10,
|
|
11
12
|
stopping="fixed")
|
|
12
13
|
rl.result("demo", "treatment", acc=0.91, seed=0)
|
|
@@ -36,6 +37,7 @@ from __future__ import annotations
|
|
|
36
37
|
|
|
37
38
|
import json
|
|
38
39
|
import os
|
|
40
|
+
import random
|
|
39
41
|
import re
|
|
40
42
|
import shlex
|
|
41
43
|
import socket
|
|
@@ -51,7 +53,7 @@ from pathlib import Path
|
|
|
51
53
|
from typing import Any
|
|
52
54
|
|
|
53
55
|
__all__ = ["RunLab", "RunLabError", "login", "result_line"]
|
|
54
|
-
__version__ = "0.
|
|
56
|
+
__version__ = "0.3.0"
|
|
55
57
|
|
|
56
58
|
DEFAULT_URL = "https://runlab.tools"
|
|
57
59
|
|
|
@@ -615,6 +617,64 @@ def _cmd_logout(args: Any) -> int:
|
|
|
615
617
|
return 0
|
|
616
618
|
|
|
617
619
|
|
|
620
|
+
DEMO_SEEDS = 10
|
|
621
|
+
QUICKSTART = ("https://github.com/omer-nachman/runlab-python/blob/main/examples/"
|
|
622
|
+
"quickstart.py")
|
|
623
|
+
|
|
624
|
+
|
|
625
|
+
def _demo_train(arm: str, seed: int) -> float:
|
|
626
|
+
"""The demo's simulated training run, as in examples/quickstart.py: accuracy 0.80 with
|
|
627
|
+
seed noise (sd 0.01), and the change ("treatment") adds +0.02."""
|
|
628
|
+
rng = random.Random(f"{arm}-{seed}")
|
|
629
|
+
return 0.80 + rng.gauss(0, 0.01) + (0.02 if arm == "treatment" else 0.0)
|
|
630
|
+
|
|
631
|
+
|
|
632
|
+
def _cmd_demo(args: Any) -> int:
|
|
633
|
+
"""Sign in if needed, then the whole loop on simulated numbers: plan, record 10 seeds per
|
|
634
|
+
arm, compare, check it is quotable; prints the verdict and where to go next."""
|
|
635
|
+
def say(line: str) -> None:
|
|
636
|
+
print(line, file=sys.stderr)
|
|
637
|
+
|
|
638
|
+
t0, project = time.monotonic(), args.project
|
|
639
|
+
try:
|
|
640
|
+
rl = RunLab(args.url)
|
|
641
|
+
if not rl.token:
|
|
642
|
+
say(f"Not logged in to {rl.url}: signing in first.")
|
|
643
|
+
say(f"Logged in as {login(args.url, browser=not args.no_browser)['user']}")
|
|
644
|
+
rl = RunLab(args.url)
|
|
645
|
+
say(f"1/4 plan: treatment vs baseline on acc, {DEMO_SEEDS} seeds, one test at the end")
|
|
646
|
+
rl.plan(project, "treatment", "baseline", "acc", n_max=DEMO_SEEDS, stopping="fixed",
|
|
647
|
+
hypothesis="the change raises accuracy")
|
|
648
|
+
say(f"2/4 record: {DEMO_SEEDS} seeds per arm (simulated: the change adds +0.02)")
|
|
649
|
+
rl.record(project, [result_line(arm, "acc", round(_demo_train(arm, seed), 6), seed=seed)
|
|
650
|
+
for seed in range(DEMO_SEEDS) for arm in ("treatment", "baseline")])
|
|
651
|
+
say("3/4 compare: paired test over the shared seeds")
|
|
652
|
+
v = rl.compare(project, "treatment", "baseline", "acc")
|
|
653
|
+
say("4/4 report: may a write-up quote it?")
|
|
654
|
+
claims = rl.report(project, arm="treatment", metric="acc").get("claims") or []
|
|
655
|
+
listed = rl.verdicts(project, limit=1)
|
|
656
|
+
rows = listed.get("verdicts") if isinstance(listed, dict) else listed
|
|
657
|
+
except RunLabError as exc:
|
|
658
|
+
print(f"error: {exc}", file=sys.stderr)
|
|
659
|
+
return 1
|
|
660
|
+
except ValueError as exc: # a project name RunLab cannot store
|
|
661
|
+
print(f"error: {exc}", file=sys.stderr)
|
|
662
|
+
return 2
|
|
663
|
+
except KeyboardInterrupt:
|
|
664
|
+
print("stopped", file=sys.stderr)
|
|
665
|
+
return 130
|
|
666
|
+
print()
|
|
667
|
+
print(v.get("text") or v.get("verdict"))
|
|
668
|
+
for c in claims:
|
|
669
|
+
print(f"{c.get('status')}: {c.get('quote')}")
|
|
670
|
+
vid = rows[0].get("id") if rows and isinstance(rows[0], dict) else None
|
|
671
|
+
if vid is not None:
|
|
672
|
+
print(f"\nThe verdict page: {rl.url}/v/{vid}")
|
|
673
|
+
print(f"\nDone in {time.monotonic() - t0:.0f} s, in your project `{project}`. The numbers "
|
|
674
|
+
"were simulated.\nNext, your own runs: replace train() in " + QUICKSTART)
|
|
675
|
+
return 0
|
|
676
|
+
|
|
677
|
+
|
|
618
678
|
# ---------------------------------------------------------------- command line
|
|
619
679
|
|
|
620
680
|
def main(argv: list[str] | None = None, prog: str | None = None) -> int:
|
|
@@ -640,6 +700,15 @@ def main(argv: list[str] | None = None, prog: str | None = None) -> int:
|
|
|
640
700
|
who = sub.add_parser("whoami", help="who this machine is signed in as")
|
|
641
701
|
who.add_argument("--url", help="your RunLab URL (default: the one you logged in to)")
|
|
642
702
|
sub.add_parser("logout", help="revoke the saved key and delete it")
|
|
703
|
+
dm = sub.add_parser("demo", help="try RunLab in one command: sign in if needed, then a "
|
|
704
|
+
"10-seed comparison on simulated numbers, ending in a verdict",
|
|
705
|
+
description="Sign in if this machine is not yet, then plan, record 10 "
|
|
706
|
+
"simulated seeds per arm, compare and report, in the project --project.")
|
|
707
|
+
dm.add_argument("--project", default="runlab-demo",
|
|
708
|
+
help="the project to use (default: runlab-demo)")
|
|
709
|
+
dm.add_argument("--url", help="your RunLab URL (default: env RUNLAB_URL, else the one you "
|
|
710
|
+
f"logged in to, else {DEFAULT_URL})")
|
|
711
|
+
dm.add_argument("--no-browser", action="store_true", help="only print the sign-in link")
|
|
643
712
|
w = sub.add_parser("worker", help="run a study's seed pairs on this machine",
|
|
644
713
|
description="Run seed pairs of a planned comparison here and post the "
|
|
645
714
|
"numbers to RunLab until its sequential test decides, n_max seeds ran "
|
|
@@ -670,8 +739,8 @@ def main(argv: list[str] | None = None, prog: str | None = None) -> int:
|
|
|
670
739
|
w.add_argument("--cwd", help="where the command runs (default: here)")
|
|
671
740
|
args = ap.parse_args(argv)
|
|
672
741
|
if args.command != "worker":
|
|
673
|
-
return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout
|
|
674
|
-
|
|
742
|
+
return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout,
|
|
743
|
+
"demo": _cmd_demo}[args.command](args)
|
|
675
744
|
try:
|
|
676
745
|
rl = RunLab(args.url, args.token)
|
|
677
746
|
sid = args.study
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: runlab-client
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
|
|
5
5
|
Author: omer-nachman
|
|
6
6
|
License-Expression: MIT
|
|
@@ -12,6 +12,10 @@ Dynamic: license-file
|
|
|
12
12
|
|
|
13
13
|
# runlab-client
|
|
14
14
|
|
|
15
|
+
[](https://pypi.org/project/runlab-client/)
|
|
16
|
+

|
|
17
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
18
|
+
|
|
15
19
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
16
20
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
17
21
|
|
|
@@ -19,9 +23,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
19
23
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
20
24
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
21
25
|
|
|
22
|
-
The RunLab Python client gives access to the RunLab API using only the standard library,
|
|
23
|
-
on Python 3.10+.
|
|
24
|
-
|
|
25
26
|
## Documentation
|
|
26
27
|
|
|
27
28
|
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
@@ -30,36 +31,30 @@ on Python 3.10+.
|
|
|
30
31
|
## Get started
|
|
31
32
|
|
|
32
33
|
```bash
|
|
33
|
-
pip install runlab-client
|
|
34
|
-
runlab
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
34
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
35
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
39
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
43
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
44
|
+
...
|
|
45
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
46
|
+
The verdict page: https://runlab.tools/v/...
|
|
39
47
|
```
|
|
40
48
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
`runlab-client --version` names the version.
|
|
44
|
-
- `runlab login` opens a browser when this machine has a display. On an SSH session or a
|
|
45
|
-
headless GPU box, open the printed link on any device (your laptop or phone).
|
|
46
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
47
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
48
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
49
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
50
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
51
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
52
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
53
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
54
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
49
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
50
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
55
51
|
|
|
56
52
|
## Quickstart
|
|
57
53
|
|
|
58
54
|
```python
|
|
59
55
|
from runlab_client import RunLab
|
|
60
56
|
|
|
61
|
-
rl = RunLab() #
|
|
62
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
57
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
63
58
|
|
|
64
59
|
# 1. Pre-register, before any seed runs.
|
|
65
60
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -83,11 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
83
78
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
84
79
|
```
|
|
85
80
|
|
|
86
|
-
`train` is your
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
81
|
+
`train` is your training run.
|
|
82
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
83
|
+
runs as is, with a simulated `train`;
|
|
84
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
85
|
+
is the same loop over plain HTTP.
|
|
91
86
|
|
|
92
87
|
## The loop
|
|
93
88
|
|
|
@@ -100,20 +95,6 @@ does the same loop over plain HTTP.
|
|
|
100
95
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
101
96
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
102
97
|
|
|
103
|
-
### Each user sees only their projects
|
|
104
|
-
|
|
105
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
106
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
107
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
108
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
109
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
110
|
-
|
|
111
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
112
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
113
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
114
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
115
|
-
`better="lower"` to override.
|
|
116
|
-
|
|
117
98
|
### Results
|
|
118
99
|
|
|
119
100
|
RunLab stores each result as one line:
|
|
@@ -130,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
130
111
|
`record()` as is.
|
|
131
112
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
132
113
|
a 400.
|
|
133
|
-
- Project names are case-insensitive
|
|
114
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
134
115
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
135
116
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
136
117
|
in configurations is refused (`refused_by: equal_budget`).
|
|
137
118
|
|
|
138
119
|
### Verdicts
|
|
139
120
|
|
|
121
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
122
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
123
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
124
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
125
|
+
|
|
140
126
|
| `verdict` | Meaning |
|
|
141
127
|
|---|---|
|
|
142
128
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -146,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
146
132
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
147
133
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
148
134
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
- `ok`;
|
|
153
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
154
|
-
after more than 3 seeds had been seen;
|
|
155
|
-
- `missing`: no plan;
|
|
156
|
-
- `tampered`.
|
|
157
|
-
- `warnings`, e.g. `peeking`.
|
|
158
|
-
- `report()`, before you quote anything.
|
|
135
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
136
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
137
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
159
138
|
|
|
160
139
|
### What keeps a verdict honest
|
|
161
140
|
|
|
@@ -169,8 +148,7 @@ Read these fields next to the verdict:
|
|
|
169
148
|
|
|
170
149
|
## Studies: let RunLab run the seeds
|
|
171
150
|
|
|
172
|
-
RunLab
|
|
173
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
151
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
174
152
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
175
153
|
|
|
176
154
|
```python
|
|
@@ -186,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
186
164
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
187
165
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
188
166
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
189
|
-
- A study never spends more than `max_usd`.
|
|
190
167
|
- `state` is one of:
|
|
191
168
|
|
|
192
169
|
| `state` | Meaning |
|
|
@@ -194,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
194
171
|
| `running` | Still running seeds. |
|
|
195
172
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
196
173
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
197
|
-
| `budget` |
|
|
198
|
-
| `killed` |
|
|
174
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
175
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
199
176
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
200
177
|
| `lost` | Stopped reporting long past its deadline. |
|
|
201
178
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -208,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
208
185
|
params' names are filled in (no shell runs it).
|
|
209
186
|
|
|
210
187
|
```bash
|
|
211
|
-
pip install runlab-client
|
|
212
|
-
runlab login
|
|
213
188
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
214
189
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
215
190
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -227,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
227
202
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
228
203
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
229
204
|
|
|
205
|
+
## Signing in
|
|
206
|
+
|
|
207
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
208
|
+
|
|
209
|
+
| Question | Answer |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
212
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
213
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
214
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
215
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
216
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
217
|
+
|
|
230
218
|
## Errors
|
|
231
219
|
|
|
232
220
|
Every failure raises `RunLabError` with:
|
|
@@ -239,20 +227,19 @@ Every failure raises `RunLabError` with:
|
|
|
239
227
|
|---|---|
|
|
240
228
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
241
229
|
| 401 | Missing or wrong token. |
|
|
242
|
-
| 403 |
|
|
230
|
+
| 403 | This key may not do this; the message says why. |
|
|
243
231
|
| 404 | No such verdict or study. |
|
|
244
232
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
245
233
|
|
|
246
234
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
247
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
248
|
-
harmless anyway.
|
|
235
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
249
236
|
|
|
250
237
|
## Client reference
|
|
251
238
|
|
|
252
239
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
253
240
|
the decoded JSON of one operation in
|
|
254
241
|
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
255
|
-
With no arguments it uses the key `runlab login` saved (see [
|
|
242
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
256
243
|
|
|
257
244
|
| Method | Operation | Returns |
|
|
258
245
|
|---|---|---|
|
|
@@ -275,6 +262,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
|
|
|
275
262
|
|
|
276
263
|
| Command | What it does |
|
|
277
264
|
|---|---|
|
|
265
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
278
266
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
279
267
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
280
268
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
@@ -289,6 +277,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
|
289
277
|
is the short version for Claude Code, Codex and other agents.
|
|
290
278
|
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
291
279
|
|
|
280
|
+
## Support
|
|
281
|
+
|
|
282
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
283
|
+
|
|
292
284
|
## License
|
|
293
285
|
|
|
294
286
|
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|