runlab-client 0.2.0__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {runlab_client-0.2.0/src/runlab_client.egg-info → runlab_client-0.3.0}/PKG-INFO +72 -65
- {runlab_client-0.2.0 → runlab_client-0.3.0}/README.md +71 -64
- {runlab_client-0.2.0 → runlab_client-0.3.0}/pyproject.toml +3 -1
- {runlab_client-0.2.0 → runlab_client-0.3.0}/src/runlab_client/__init__.py +88 -11
- runlab_client-0.3.0/src/runlab_client/__main__.py +6 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0/src/runlab_client.egg-info}/PKG-INFO +72 -65
- {runlab_client-0.2.0 → runlab_client-0.3.0}/src/runlab_client.egg-info/SOURCES.txt +1 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0}/src/runlab_client.egg-info/entry_points.txt +1 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0}/LICENSE +0 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0}/setup.cfg +0 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0}/src/runlab_client.egg-info/dependency_links.txt +0 -0
- {runlab_client-0.2.0 → runlab_client-0.3.0}/src/runlab_client.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: runlab-client
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
|
|
5
5
|
Author: omer-nachman
|
|
6
6
|
License-Expression: MIT
|
|
@@ -12,6 +12,10 @@ Dynamic: license-file
|
|
|
12
12
|
|
|
13
13
|
# runlab-client
|
|
14
14
|
|
|
15
|
+
[](https://pypi.org/project/runlab-client/)
|
|
16
|
+

|
|
17
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
18
|
+
|
|
15
19
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
16
20
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
17
21
|
|
|
@@ -19,39 +23,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
19
23
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
20
24
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
21
25
|
|
|
22
|
-
|
|
23
|
-
|
|
26
|
+
## Documentation
|
|
27
|
+
|
|
28
|
+
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
29
|
+
- [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
|
|
24
30
|
|
|
25
31
|
## Get started
|
|
26
32
|
|
|
27
33
|
```bash
|
|
28
|
-
pip install runlab-client
|
|
29
|
-
runlab
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
35
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
39
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
43
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
44
|
+
...
|
|
45
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
46
|
+
The verdict page: https://runlab.tools/v/...
|
|
34
47
|
```
|
|
35
48
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
39
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
40
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
41
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
42
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
43
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
44
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
45
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
46
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
49
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
50
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
47
51
|
|
|
48
52
|
## Quickstart
|
|
49
53
|
|
|
50
54
|
```python
|
|
51
55
|
from runlab_client import RunLab
|
|
52
56
|
|
|
53
|
-
rl = RunLab() #
|
|
54
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
57
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
55
58
|
|
|
56
59
|
# 1. Pre-register, before any seed runs.
|
|
57
60
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -75,9 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
75
78
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
76
79
|
```
|
|
77
80
|
|
|
78
|
-
`train` is your
|
|
79
|
-
|
|
80
|
-
|
|
81
|
+
`train` is your training run.
|
|
82
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
83
|
+
runs as is, with a simulated `train`;
|
|
84
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
85
|
+
is the same loop over plain HTTP.
|
|
81
86
|
|
|
82
87
|
## The loop
|
|
83
88
|
|
|
@@ -90,20 +95,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
|
|
|
90
95
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
91
96
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
92
97
|
|
|
93
|
-
### Each user sees only their projects
|
|
94
|
-
|
|
95
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
96
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
97
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
98
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
99
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
100
|
-
|
|
101
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
102
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
103
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
104
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
105
|
-
`better="lower"` to override.
|
|
106
|
-
|
|
107
98
|
### Results
|
|
108
99
|
|
|
109
100
|
RunLab stores each result as one line:
|
|
@@ -120,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
120
111
|
`record()` as is.
|
|
121
112
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
122
113
|
a 400.
|
|
123
|
-
- Project names are case-insensitive
|
|
114
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
124
115
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
125
116
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
126
117
|
in configurations is refused (`refused_by: equal_budget`).
|
|
127
118
|
|
|
128
119
|
### Verdicts
|
|
129
120
|
|
|
121
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
122
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
123
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
124
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
125
|
+
|
|
130
126
|
| `verdict` | Meaning |
|
|
131
127
|
|---|---|
|
|
132
128
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -136,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
136
132
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
137
133
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
138
134
|
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
- `ok`;
|
|
143
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
144
|
-
after more than 3 seeds had been seen;
|
|
145
|
-
- `missing`: no plan;
|
|
146
|
-
- `tampered`.
|
|
147
|
-
- `warnings`, e.g. `peeking`.
|
|
148
|
-
- `report()`, before you quote anything.
|
|
135
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
136
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
137
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
149
138
|
|
|
150
139
|
### What keeps a verdict honest
|
|
151
140
|
|
|
@@ -159,8 +148,7 @@ Read these fields next to the verdict:
|
|
|
159
148
|
|
|
160
149
|
## Studies: let RunLab run the seeds
|
|
161
150
|
|
|
162
|
-
RunLab
|
|
163
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
151
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
164
152
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
165
153
|
|
|
166
154
|
```python
|
|
@@ -176,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
176
164
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
177
165
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
178
166
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
179
|
-
- A study never spends more than `max_usd`.
|
|
180
167
|
- `state` is one of:
|
|
181
168
|
|
|
182
169
|
| `state` | Meaning |
|
|
@@ -184,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
184
171
|
| `running` | Still running seeds. |
|
|
185
172
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
186
173
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
187
|
-
| `budget` |
|
|
188
|
-
| `killed` |
|
|
174
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
175
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
189
176
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
190
177
|
| `lost` | Stopped reporting long past its deadline. |
|
|
191
178
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -198,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
198
185
|
params' names are filled in (no shell runs it).
|
|
199
186
|
|
|
200
187
|
```bash
|
|
201
|
-
pip install runlab-client
|
|
202
|
-
runlab login
|
|
203
188
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
204
189
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
205
190
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -217,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
217
202
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
218
203
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
219
204
|
|
|
205
|
+
## Signing in
|
|
206
|
+
|
|
207
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
208
|
+
|
|
209
|
+
| Question | Answer |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
212
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
213
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
214
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
215
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
216
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
217
|
+
|
|
220
218
|
## Errors
|
|
221
219
|
|
|
222
220
|
Every failure raises `RunLabError` with:
|
|
@@ -229,19 +227,19 @@ Every failure raises `RunLabError` with:
|
|
|
229
227
|
|---|---|
|
|
230
228
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
231
229
|
| 401 | Missing or wrong token. |
|
|
232
|
-
| 403 |
|
|
230
|
+
| 403 | This key may not do this; the message says why. |
|
|
233
231
|
| 404 | No such verdict or study. |
|
|
234
232
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
235
233
|
|
|
236
234
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
237
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
238
|
-
harmless anyway.
|
|
235
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
239
236
|
|
|
240
237
|
## Client reference
|
|
241
238
|
|
|
242
239
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
243
|
-
the decoded JSON of one operation in
|
|
244
|
-
|
|
240
|
+
the decoded JSON of one operation in
|
|
241
|
+
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
242
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
245
243
|
|
|
246
244
|
| Method | Operation | Returns |
|
|
247
245
|
|---|---|---|
|
|
@@ -264,16 +262,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
|
|
|
264
262
|
|
|
265
263
|
| Command | What it does |
|
|
266
264
|
|---|---|
|
|
265
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
267
266
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
268
267
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
269
268
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
270
269
|
| `runlab worker ...` | Run a study's seed pairs here (see above). |
|
|
270
|
+
| `runlab --version` | The client's version. |
|
|
271
|
+
|
|
272
|
+
Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
271
273
|
|
|
272
274
|
## Coding agents
|
|
273
275
|
|
|
274
|
-
[AGENTS.md](AGENTS.md)
|
|
275
|
-
|
|
276
|
+
[AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
|
|
277
|
+
is the short version for Claude Code, Codex and other agents.
|
|
278
|
+
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
279
|
+
|
|
280
|
+
## Support
|
|
281
|
+
|
|
282
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
276
283
|
|
|
277
284
|
## License
|
|
278
285
|
|
|
279
|
-
MIT ([LICENSE](LICENSE)).
|
|
286
|
+
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# runlab-client
|
|
2
2
|
|
|
3
|
+
[](https://pypi.org/project/runlab-client/)
|
|
4
|
+

|
|
5
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
6
|
+
|
|
3
7
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
4
8
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
5
9
|
|
|
@@ -7,39 +11,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
7
11
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
8
12
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
9
13
|
|
|
10
|
-
|
|
11
|
-
|
|
14
|
+
## Documentation
|
|
15
|
+
|
|
16
|
+
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
17
|
+
- [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
|
|
12
18
|
|
|
13
19
|
## Get started
|
|
14
20
|
|
|
15
21
|
```bash
|
|
16
|
-
pip install runlab-client
|
|
17
|
-
runlab
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
23
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
27
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
31
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
32
|
+
...
|
|
33
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
34
|
+
The verdict page: https://runlab.tools/v/...
|
|
22
35
|
```
|
|
23
36
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
27
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
28
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
29
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
30
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
31
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
32
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
33
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
34
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
37
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
38
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
35
39
|
|
|
36
40
|
## Quickstart
|
|
37
41
|
|
|
38
42
|
```python
|
|
39
43
|
from runlab_client import RunLab
|
|
40
44
|
|
|
41
|
-
rl = RunLab() #
|
|
42
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
45
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
43
46
|
|
|
44
47
|
# 1. Pre-register, before any seed runs.
|
|
45
48
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -63,9 +66,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
63
66
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
64
67
|
```
|
|
65
68
|
|
|
66
|
-
`train` is your
|
|
67
|
-
|
|
68
|
-
|
|
69
|
+
`train` is your training run.
|
|
70
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
71
|
+
runs as is, with a simulated `train`;
|
|
72
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
73
|
+
is the same loop over plain HTTP.
|
|
69
74
|
|
|
70
75
|
## The loop
|
|
71
76
|
|
|
@@ -78,20 +83,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
|
|
|
78
83
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
79
84
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
80
85
|
|
|
81
|
-
### Each user sees only their projects
|
|
82
|
-
|
|
83
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
84
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
85
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
86
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
87
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
88
|
-
|
|
89
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
90
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
91
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
92
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
93
|
-
`better="lower"` to override.
|
|
94
|
-
|
|
95
86
|
### Results
|
|
96
87
|
|
|
97
88
|
RunLab stores each result as one line:
|
|
@@ -108,13 +99,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
108
99
|
`record()` as is.
|
|
109
100
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
110
101
|
a 400.
|
|
111
|
-
- Project names are case-insensitive
|
|
102
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
112
103
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
113
104
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
114
105
|
in configurations is refused (`refused_by: equal_budget`).
|
|
115
106
|
|
|
116
107
|
### Verdicts
|
|
117
108
|
|
|
109
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
110
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
111
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
112
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
113
|
+
|
|
118
114
|
| `verdict` | Meaning |
|
|
119
115
|
|---|---|
|
|
120
116
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -124,16 +120,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
124
120
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
125
121
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
126
122
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
- `ok`;
|
|
131
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
132
|
-
after more than 3 seeds had been seen;
|
|
133
|
-
- `missing`: no plan;
|
|
134
|
-
- `tampered`.
|
|
135
|
-
- `warnings`, e.g. `peeking`.
|
|
136
|
-
- `report()`, before you quote anything.
|
|
123
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
124
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
125
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
137
126
|
|
|
138
127
|
### What keeps a verdict honest
|
|
139
128
|
|
|
@@ -147,8 +136,7 @@ Read these fields next to the verdict:
|
|
|
147
136
|
|
|
148
137
|
## Studies: let RunLab run the seeds
|
|
149
138
|
|
|
150
|
-
RunLab
|
|
151
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
139
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
152
140
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
153
141
|
|
|
154
142
|
```python
|
|
@@ -164,7 +152,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
164
152
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
165
153
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
166
154
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
167
|
-
- A study never spends more than `max_usd`.
|
|
168
155
|
- `state` is one of:
|
|
169
156
|
|
|
170
157
|
| `state` | Meaning |
|
|
@@ -172,8 +159,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
172
159
|
| `running` | Still running seeds. |
|
|
173
160
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
174
161
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
175
|
-
| `budget` |
|
|
176
|
-
| `killed` |
|
|
162
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
163
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
177
164
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
178
165
|
| `lost` | Stopped reporting long past its deadline. |
|
|
179
166
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -186,8 +173,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
186
173
|
params' names are filled in (no shell runs it).
|
|
187
174
|
|
|
188
175
|
```bash
|
|
189
|
-
pip install runlab-client
|
|
190
|
-
runlab login
|
|
191
176
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
192
177
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
193
178
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -205,6 +190,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
205
190
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
206
191
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
207
192
|
|
|
193
|
+
## Signing in
|
|
194
|
+
|
|
195
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
196
|
+
|
|
197
|
+
| Question | Answer |
|
|
198
|
+
|---|---|
|
|
199
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
200
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
201
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
202
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
203
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
204
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
205
|
+
|
|
208
206
|
## Errors
|
|
209
207
|
|
|
210
208
|
Every failure raises `RunLabError` with:
|
|
@@ -217,19 +215,19 @@ Every failure raises `RunLabError` with:
|
|
|
217
215
|
|---|---|
|
|
218
216
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
219
217
|
| 401 | Missing or wrong token. |
|
|
220
|
-
| 403 |
|
|
218
|
+
| 403 | This key may not do this; the message says why. |
|
|
221
219
|
| 404 | No such verdict or study. |
|
|
222
220
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
223
221
|
|
|
224
222
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
225
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
226
|
-
harmless anyway.
|
|
223
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
227
224
|
|
|
228
225
|
## Client reference
|
|
229
226
|
|
|
230
227
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
231
|
-
the decoded JSON of one operation in
|
|
232
|
-
|
|
228
|
+
the decoded JSON of one operation in
|
|
229
|
+
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
230
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
233
231
|
|
|
234
232
|
| Method | Operation | Returns |
|
|
235
233
|
|---|---|---|
|
|
@@ -252,16 +250,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
|
|
|
252
250
|
|
|
253
251
|
| Command | What it does |
|
|
254
252
|
|---|---|
|
|
253
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
255
254
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
256
255
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
257
256
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
258
257
|
| `runlab worker ...` | Run a study's seed pairs here (see above). |
|
|
258
|
+
| `runlab --version` | The client's version. |
|
|
259
|
+
|
|
260
|
+
Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
259
261
|
|
|
260
262
|
## Coding agents
|
|
261
263
|
|
|
262
|
-
[AGENTS.md](AGENTS.md)
|
|
263
|
-
|
|
264
|
+
[AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
|
|
265
|
+
is the short version for Claude Code, Codex and other agents.
|
|
266
|
+
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
267
|
+
|
|
268
|
+
## Support
|
|
269
|
+
|
|
270
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
264
271
|
|
|
265
272
|
## License
|
|
266
273
|
|
|
267
|
-
MIT ([LICENSE](LICENSE)).
|
|
274
|
+
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "runlab-client"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.3.0"
|
|
8
8
|
description = "Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.10"
|
|
@@ -14,6 +14,8 @@ dependencies = []
|
|
|
14
14
|
|
|
15
15
|
[project.scripts]
|
|
16
16
|
runlab = "runlab_client:main"
|
|
17
|
+
# the same commands under a name no other `runlab` program on PATH takes
|
|
18
|
+
runlab-client = "runlab_client:main"
|
|
17
19
|
|
|
18
20
|
[project.urls]
|
|
19
21
|
Homepage = "https://github.com/omer-nachman/runlab-python"
|
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
"""runlab-client: Python client for the RunLab research API.
|
|
2
2
|
|
|
3
3
|
$ pip install runlab-client
|
|
4
|
-
$ runlab
|
|
4
|
+
$ runlab demo # GitHub sign-in, then a first verdict on simulated numbers
|
|
5
|
+
$ runlab login # sign in only; the key stays on this machine
|
|
6
|
+
$ runlab-client login # the same, when another `runlab` comes first on PATH
|
|
5
7
|
|
|
6
8
|
from runlab_client import RunLab
|
|
7
9
|
|
|
8
|
-
rl = RunLab() #
|
|
10
|
+
rl = RunLab() # the key `runlab demo` / `login` saved, or RUNLAB_TOKEN
|
|
9
11
|
rl.plan("demo", "treatment", "baseline", "acc", hypothesis="the change helps", n_max=10,
|
|
10
12
|
stopping="fixed")
|
|
11
13
|
rl.result("demo", "treatment", acc=0.91, seed=0)
|
|
@@ -35,6 +37,7 @@ from __future__ import annotations
|
|
|
35
37
|
|
|
36
38
|
import json
|
|
37
39
|
import os
|
|
40
|
+
import random
|
|
38
41
|
import re
|
|
39
42
|
import shlex
|
|
40
43
|
import socket
|
|
@@ -50,7 +53,7 @@ from pathlib import Path
|
|
|
50
53
|
from typing import Any
|
|
51
54
|
|
|
52
55
|
__all__ = ["RunLab", "RunLabError", "login", "result_line"]
|
|
53
|
-
__version__ = "0.
|
|
56
|
+
__version__ = "0.3.0"
|
|
54
57
|
|
|
55
58
|
DEFAULT_URL = "https://runlab.tools"
|
|
56
59
|
|
|
@@ -580,12 +583,12 @@ def _cmd_login(args: Any) -> int:
|
|
|
580
583
|
def _cmd_whoami(args: Any) -> int:
|
|
581
584
|
rl = RunLab(args.url)
|
|
582
585
|
if not rl.token:
|
|
583
|
-
print(f"not logged in to {rl.url}: run `
|
|
586
|
+
print(f"not logged in to {rl.url}: run `{args.prog} login`", file=sys.stderr)
|
|
584
587
|
return 1
|
|
585
588
|
try:
|
|
586
589
|
me = rl.me()
|
|
587
590
|
except RunLabError as exc:
|
|
588
|
-
hint = ": run `
|
|
591
|
+
hint = f": run `{args.prog} login`" if exc.status == 401 else ""
|
|
589
592
|
print(f"error: {exc}{hint}", file=sys.stderr)
|
|
590
593
|
return 1
|
|
591
594
|
print(f"{me.get('name')} ({me.get('role')}) at {rl.url}")
|
|
@@ -603,7 +606,7 @@ def _cmd_logout(args: Any) -> int:
|
|
|
603
606
|
except RunLabError as exc:
|
|
604
607
|
if exc.status != 401:
|
|
605
608
|
print(f"error: cannot revoke the key at {url}: {exc.message}. It stays saved: run "
|
|
606
|
-
f"`
|
|
609
|
+
f"`{args.prog} logout` again, or delete {config_dir() / 'token'} to forget it "
|
|
607
610
|
"here only", file=sys.stderr)
|
|
608
611
|
return 1
|
|
609
612
|
how = "the key had already stopped working"
|
|
@@ -614,13 +617,78 @@ def _cmd_logout(args: Any) -> int:
|
|
|
614
617
|
return 0
|
|
615
618
|
|
|
616
619
|
|
|
620
|
+
DEMO_SEEDS = 10
|
|
621
|
+
QUICKSTART = ("https://github.com/omer-nachman/runlab-python/blob/main/examples/"
|
|
622
|
+
"quickstart.py")
|
|
623
|
+
|
|
624
|
+
|
|
625
|
+
def _demo_train(arm: str, seed: int) -> float:
|
|
626
|
+
"""The demo's simulated training run, as in examples/quickstart.py: accuracy 0.80 with
|
|
627
|
+
seed noise (sd 0.01), and the change ("treatment") adds +0.02."""
|
|
628
|
+
rng = random.Random(f"{arm}-{seed}")
|
|
629
|
+
return 0.80 + rng.gauss(0, 0.01) + (0.02 if arm == "treatment" else 0.0)
|
|
630
|
+
|
|
631
|
+
|
|
632
|
+
def _cmd_demo(args: Any) -> int:
|
|
633
|
+
"""Sign in if needed, then the whole loop on simulated numbers: plan, record 10 seeds per
|
|
634
|
+
arm, compare, check it is quotable; prints the verdict and where to go next."""
|
|
635
|
+
def say(line: str) -> None:
|
|
636
|
+
print(line, file=sys.stderr)
|
|
637
|
+
|
|
638
|
+
t0, project = time.monotonic(), args.project
|
|
639
|
+
try:
|
|
640
|
+
rl = RunLab(args.url)
|
|
641
|
+
if not rl.token:
|
|
642
|
+
say(f"Not logged in to {rl.url}: signing in first.")
|
|
643
|
+
say(f"Logged in as {login(args.url, browser=not args.no_browser)['user']}")
|
|
644
|
+
rl = RunLab(args.url)
|
|
645
|
+
say(f"1/4 plan: treatment vs baseline on acc, {DEMO_SEEDS} seeds, one test at the end")
|
|
646
|
+
rl.plan(project, "treatment", "baseline", "acc", n_max=DEMO_SEEDS, stopping="fixed",
|
|
647
|
+
hypothesis="the change raises accuracy")
|
|
648
|
+
say(f"2/4 record: {DEMO_SEEDS} seeds per arm (simulated: the change adds +0.02)")
|
|
649
|
+
rl.record(project, [result_line(arm, "acc", round(_demo_train(arm, seed), 6), seed=seed)
|
|
650
|
+
for seed in range(DEMO_SEEDS) for arm in ("treatment", "baseline")])
|
|
651
|
+
say("3/4 compare: paired test over the shared seeds")
|
|
652
|
+
v = rl.compare(project, "treatment", "baseline", "acc")
|
|
653
|
+
say("4/4 report: may a write-up quote it?")
|
|
654
|
+
claims = rl.report(project, arm="treatment", metric="acc").get("claims") or []
|
|
655
|
+
listed = rl.verdicts(project, limit=1)
|
|
656
|
+
rows = listed.get("verdicts") if isinstance(listed, dict) else listed
|
|
657
|
+
except RunLabError as exc:
|
|
658
|
+
print(f"error: {exc}", file=sys.stderr)
|
|
659
|
+
return 1
|
|
660
|
+
except ValueError as exc: # a project name RunLab cannot store
|
|
661
|
+
print(f"error: {exc}", file=sys.stderr)
|
|
662
|
+
return 2
|
|
663
|
+
except KeyboardInterrupt:
|
|
664
|
+
print("stopped", file=sys.stderr)
|
|
665
|
+
return 130
|
|
666
|
+
print()
|
|
667
|
+
print(v.get("text") or v.get("verdict"))
|
|
668
|
+
for c in claims:
|
|
669
|
+
print(f"{c.get('status')}: {c.get('quote')}")
|
|
670
|
+
vid = rows[0].get("id") if rows and isinstance(rows[0], dict) else None
|
|
671
|
+
if vid is not None:
|
|
672
|
+
print(f"\nThe verdict page: {rl.url}/v/{vid}")
|
|
673
|
+
print(f"\nDone in {time.monotonic() - t0:.0f} s, in your project `{project}`. The numbers "
|
|
674
|
+
"were simulated.\nNext, your own runs: replace train() in " + QUICKSTART)
|
|
675
|
+
return 0
|
|
676
|
+
|
|
677
|
+
|
|
617
678
|
# ---------------------------------------------------------------- command line
|
|
618
679
|
|
|
619
|
-
def main(argv: list[str] | None = None) -> int:
|
|
620
|
-
"""`runlab login | whoami | logout | worker ...` (see the module doc)
|
|
680
|
+
def main(argv: list[str] | None = None, prog: str | None = None) -> int:
|
|
681
|
+
"""`runlab login | whoami | logout | worker ...` (see the module doc); also installed as
|
|
682
|
+
`runlab-client`, and run by `python -m runlab_client`. Every hint names `prog`, the
|
|
683
|
+
command that was run (default: `runlab-client` when run under that name, else `runlab`),
|
|
684
|
+
so following it reaches this client and not another `runlab` on PATH."""
|
|
621
685
|
import argparse
|
|
622
686
|
|
|
623
|
-
|
|
687
|
+
if prog is None:
|
|
688
|
+
prog = "runlab-client" if Path(sys.argv[0]).name.startswith("runlab-client") else "runlab"
|
|
689
|
+
ap = argparse.ArgumentParser(prog=prog, description="RunLab from the command line.")
|
|
690
|
+
ap.add_argument("--version", action="version", version=f"runlab-client {__version__}")
|
|
691
|
+
ap.set_defaults(prog=prog)
|
|
624
692
|
sub = ap.add_subparsers(dest="command", required=True)
|
|
625
693
|
lg = sub.add_parser("login", help="sign this machine in with GitHub",
|
|
626
694
|
description="Print a link and a code, open the link when this machine "
|
|
@@ -632,6 +700,15 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
632
700
|
who = sub.add_parser("whoami", help="who this machine is signed in as")
|
|
633
701
|
who.add_argument("--url", help="your RunLab URL (default: the one you logged in to)")
|
|
634
702
|
sub.add_parser("logout", help="revoke the saved key and delete it")
|
|
703
|
+
dm = sub.add_parser("demo", help="try RunLab in one command: sign in if needed, then a "
|
|
704
|
+
"10-seed comparison on simulated numbers, ending in a verdict",
|
|
705
|
+
description="Sign in if this machine is not yet, then plan, record 10 "
|
|
706
|
+
"simulated seeds per arm, compare and report, in the project --project.")
|
|
707
|
+
dm.add_argument("--project", default="runlab-demo",
|
|
708
|
+
help="the project to use (default: runlab-demo)")
|
|
709
|
+
dm.add_argument("--url", help="your RunLab URL (default: env RUNLAB_URL, else the one you "
|
|
710
|
+
f"logged in to, else {DEFAULT_URL})")
|
|
711
|
+
dm.add_argument("--no-browser", action="store_true", help="only print the sign-in link")
|
|
635
712
|
w = sub.add_parser("worker", help="run a study's seed pairs on this machine",
|
|
636
713
|
description="Run seed pairs of a planned comparison here and post the "
|
|
637
714
|
"numbers to RunLab until its sequential test decides, n_max seeds ran "
|
|
@@ -662,8 +739,8 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
662
739
|
w.add_argument("--cwd", help="where the command runs (default: here)")
|
|
663
740
|
args = ap.parse_args(argv)
|
|
664
741
|
if args.command != "worker":
|
|
665
|
-
return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout
|
|
666
|
-
|
|
742
|
+
return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout,
|
|
743
|
+
"demo": _cmd_demo}[args.command](args)
|
|
667
744
|
try:
|
|
668
745
|
rl = RunLab(args.url, args.token)
|
|
669
746
|
sid = args.study
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: runlab-client
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
|
|
5
5
|
Author: omer-nachman
|
|
6
6
|
License-Expression: MIT
|
|
@@ -12,6 +12,10 @@ Dynamic: license-file
|
|
|
12
12
|
|
|
13
13
|
# runlab-client
|
|
14
14
|
|
|
15
|
+
[](https://pypi.org/project/runlab-client/)
|
|
16
|
+

|
|
17
|
+
[](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
|
|
18
|
+
|
|
15
19
|
**Did my change actually help?** You record each seed's metric for two arms: your change and
|
|
16
20
|
its baseline. RunLab gives back a verdict you can quote, built from:
|
|
17
21
|
|
|
@@ -19,39 +23,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
|
|
|
19
23
|
- a paired test across seeds, with a 95 % confidence interval;
|
|
20
24
|
- checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
|
|
21
25
|
|
|
22
|
-
|
|
23
|
-
|
|
26
|
+
## Documentation
|
|
27
|
+
|
|
28
|
+
- [API reference](https://omer-nachman.github.io/runlab-python/api.html)
|
|
29
|
+
- [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
|
|
24
30
|
|
|
25
31
|
## Get started
|
|
26
32
|
|
|
27
33
|
```bash
|
|
28
|
-
pip install runlab-client
|
|
29
|
-
runlab
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
+
pip install runlab-client # Python 3.10+, no dependencies
|
|
35
|
+
runlab demo # sign in with GitHub, then a first verdict on simulated numbers
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
|
|
39
|
+
Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
treatment vs baseline on acc (higher is better): treatment is better
|
|
43
|
+
paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
|
|
44
|
+
...
|
|
45
|
+
quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
|
|
46
|
+
The verdict page: https://runlab.tools/v/...
|
|
34
47
|
```
|
|
35
48
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
- The page asks "Authorize this terminal?" and shows the code, the address and host name
|
|
39
|
-
that asked. Approve only if the code matches the one in your terminal.
|
|
40
|
-
- The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
|
|
41
|
-
`0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
|
|
42
|
-
until you log out. `runlab logout` revokes it and deletes the file.
|
|
43
|
-
- After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
|
|
44
|
-
the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
|
|
45
|
-
`https://runlab.tools`. The saved key is only sent to the URL that issued it.
|
|
46
|
-
- In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
|
|
49
|
+
Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
|
|
50
|
+
the simulated one. Questions about signing in: [Signing in](#signing-in).
|
|
47
51
|
|
|
48
52
|
## Quickstart
|
|
49
53
|
|
|
50
54
|
```python
|
|
51
55
|
from runlab_client import RunLab
|
|
52
56
|
|
|
53
|
-
rl = RunLab() #
|
|
54
|
-
rl.me() # {'name': 'ada', 'role': 'agent', ...}
|
|
57
|
+
rl = RunLab() # the key `runlab demo` / `login` saved
|
|
55
58
|
|
|
56
59
|
# 1. Pre-register, before any seed runs.
|
|
57
60
|
rl.plan("resnet-aug", "mixup", "baseline", "acc",
|
|
@@ -75,9 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
|
|
|
75
78
|
print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
|
|
76
79
|
```
|
|
77
80
|
|
|
78
|
-
`train` is your
|
|
79
|
-
|
|
80
|
-
|
|
81
|
+
`train` is your training run.
|
|
82
|
+
[examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
|
|
83
|
+
runs as is, with a simulated `train`;
|
|
84
|
+
[examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
|
|
85
|
+
is the same loop over plain HTTP.
|
|
81
86
|
|
|
82
87
|
## The loop
|
|
83
88
|
|
|
@@ -90,20 +95,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
|
|
|
90
95
|
| Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
|
|
91
96
|
| Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
|
|
92
97
|
|
|
93
|
-
### Each user sees only their projects
|
|
94
|
-
|
|
95
|
-
On a RunLab where each user has their own data, a project name is yours alone: your `demo`
|
|
96
|
-
and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
|
|
97
|
-
other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
|
|
98
|
-
study id that is not yours answers 404, the same as one that does not exist, and an unknown
|
|
99
|
-
project's closest match is one of yours. Project names may not contain `~`.
|
|
100
|
-
|
|
101
|
-
`a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
|
|
102
|
-
a scored higher. Whether higher is better comes from the metric name: lower for names
|
|
103
|
-
containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
|
|
104
|
-
`nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
|
|
105
|
-
`better="lower"` to override.
|
|
106
|
-
|
|
107
98
|
### Results
|
|
108
99
|
|
|
109
100
|
RunLab stores each result as one line:
|
|
@@ -120,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
120
111
|
`record()` as is.
|
|
121
112
|
- If one RESULT line is malformed, or there is none, the call stores nothing and fails with
|
|
122
113
|
a 400.
|
|
123
|
-
- Project names are case-insensitive
|
|
114
|
+
- Project names are case-insensitive and may not contain `~`.
|
|
124
115
|
- Params are optional; keep an arm's params the same on every seed. Each distinct set of
|
|
125
116
|
params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
|
|
126
117
|
in configurations is refused (`refused_by: equal_budget`).
|
|
127
118
|
|
|
128
119
|
### Verdicts
|
|
129
120
|
|
|
121
|
+
`a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
|
|
122
|
+
scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
|
|
123
|
+
`perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
|
|
124
|
+
higher otherwise; `better="higher"` or `better="lower"` overrides.
|
|
125
|
+
|
|
130
126
|
| `verdict` | Meaning |
|
|
131
127
|
|---|---|
|
|
132
128
|
| `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
|
|
@@ -136,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
|
|
|
136
132
|
| `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
|
|
137
133
|
| `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
|
|
138
134
|
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
- `ok`;
|
|
143
|
-
- `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
|
|
144
|
-
after more than 3 seeds had been seen;
|
|
145
|
-
- `missing`: no plan;
|
|
146
|
-
- `tampered`.
|
|
147
|
-
- `warnings`, e.g. `peeking`.
|
|
148
|
-
- `report()`, before you quote anything.
|
|
135
|
+
Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
|
|
136
|
+
before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
|
|
137
|
+
`tampered`. Then read `warnings`, e.g. `peeking`.
|
|
149
138
|
|
|
150
139
|
### What keeps a verdict honest
|
|
151
140
|
|
|
@@ -159,8 +148,7 @@ Read these fields next to the verdict:
|
|
|
159
148
|
|
|
160
149
|
## Studies: let RunLab run the seeds
|
|
161
150
|
|
|
162
|
-
RunLab
|
|
163
|
-
It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
151
|
+
RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
|
|
164
152
|
(mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
|
|
165
153
|
|
|
166
154
|
```python
|
|
@@ -176,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
176
164
|
placeholders, using values of letters, digits and `_.:+=-` only.
|
|
177
165
|
- Every check runs before anything starts. A refusal raises `RunLabError` with status 400
|
|
178
166
|
and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
|
|
179
|
-
- A study never spends more than `max_usd`.
|
|
180
167
|
- `state` is one of:
|
|
181
168
|
|
|
182
169
|
| `state` | Meaning |
|
|
@@ -184,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
|
|
|
184
171
|
| `running` | Still running seeds. |
|
|
185
172
|
| `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
|
|
186
173
|
| `undecided` | `n_max` seeds ran without a decision. |
|
|
187
|
-
| `budget` |
|
|
188
|
-
| `killed` |
|
|
174
|
+
| `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
|
|
175
|
+
| `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
|
|
189
176
|
| `failed` | The command failed or printed no result; see `detail`. |
|
|
190
177
|
| `lost` | Stopped reporting long past its deadline. |
|
|
191
178
|
| `refused` | Refused at the start (that call raised a 400); listed for the record. |
|
|
@@ -198,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
|
|
|
198
185
|
params' names are filled in (no shell runs it).
|
|
199
186
|
|
|
200
187
|
```bash
|
|
201
|
-
pip install runlab-client
|
|
202
|
-
runlab login
|
|
203
188
|
runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
204
189
|
--max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
|
|
205
190
|
--cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
|
|
@@ -217,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
|
|
|
217
202
|
- Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
|
|
218
203
|
- From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
|
|
219
204
|
|
|
205
|
+
## Signing in
|
|
206
|
+
|
|
207
|
+
`runlab login` signs a machine in; `runlab demo` does it too on first use.
|
|
208
|
+
|
|
209
|
+
| Question | Answer |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
|
|
212
|
+
| No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
|
|
213
|
+
| What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
|
|
214
|
+
| Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
|
|
215
|
+
| What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
|
|
216
|
+
| CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
|
|
217
|
+
|
|
220
218
|
## Errors
|
|
221
219
|
|
|
222
220
|
Every failure raises `RunLabError` with:
|
|
@@ -229,19 +227,19 @@ Every failure raises `RunLabError` with:
|
|
|
229
227
|
|---|---|
|
|
230
228
|
| 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
|
|
231
229
|
| 401 | Missing or wrong token. |
|
|
232
|
-
| 403 |
|
|
230
|
+
| 403 | This key may not do this; the message says why. |
|
|
233
231
|
| 404 | No such verdict or study. |
|
|
234
232
|
| 413 | Body over 64 KiB; send results in several calls. |
|
|
235
233
|
|
|
236
234
|
The client retries GET requests 3 times with exponential backoff, on connection errors and on
|
|
237
|
-
429, 502, 503 and 504. It never retries POST requests.
|
|
238
|
-
harmless anyway.
|
|
235
|
+
429, 502, 503 and 504. It never retries POST requests.
|
|
239
236
|
|
|
240
237
|
## Client reference
|
|
241
238
|
|
|
242
239
|
`RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
|
|
243
|
-
the decoded JSON of one operation in
|
|
244
|
-
|
|
240
|
+
the decoded JSON of one operation in
|
|
241
|
+
[openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
|
|
242
|
+
With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
|
|
245
243
|
|
|
246
244
|
| Method | Operation | Returns |
|
|
247
245
|
|---|---|---|
|
|
@@ -264,16 +262,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
|
|
|
264
262
|
|
|
265
263
|
| Command | What it does |
|
|
266
264
|
|---|---|
|
|
265
|
+
| `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
|
|
267
266
|
| `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
|
|
268
267
|
| `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
|
|
269
268
|
| `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
|
|
270
269
|
| `runlab worker ...` | Run a study's seed pairs here (see above). |
|
|
270
|
+
| `runlab --version` | The client's version. |
|
|
271
|
+
|
|
272
|
+
Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
|
|
271
273
|
|
|
272
274
|
## Coding agents
|
|
273
275
|
|
|
274
|
-
[AGENTS.md](AGENTS.md)
|
|
275
|
-
|
|
276
|
+
[AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
|
|
277
|
+
is the short version for Claude Code, Codex and other agents.
|
|
278
|
+
[llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
|
|
279
|
+
|
|
280
|
+
## Support
|
|
281
|
+
|
|
282
|
+
Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
|
|
276
283
|
|
|
277
284
|
## License
|
|
278
285
|
|
|
279
|
-
MIT ([LICENSE](LICENSE)).
|
|
286
|
+
MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|