runlab-client 0.2.1__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: runlab-client
3
- Version: 0.2.1
3
+ Version: 0.3.0
4
4
  Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
5
5
  Author: omer-nachman
6
6
  License-Expression: MIT
@@ -12,6 +12,10 @@ Dynamic: license-file
12
12
 
13
13
  # runlab-client
14
14
 
15
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
16
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
17
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
18
+
15
19
  **Did my change actually help?** You record each seed's metric for two arms: your change and
16
20
  its baseline. RunLab gives back a verdict you can quote, built from:
17
21
 
@@ -19,9 +23,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
19
23
  - a paired test across seeds, with a 95 % confidence interval;
20
24
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
21
25
 
22
- The RunLab Python client gives access to the RunLab API using only the standard library,
23
- on Python 3.10+.
24
-
25
26
  ## Documentation
26
27
 
27
28
  - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
@@ -30,36 +31,30 @@ on Python 3.10+.
30
31
  ## Get started
31
32
 
32
33
  ```bash
33
- pip install runlab-client # Python 3.10+, no dependencies
34
- runlab login # sign in with GitHub, approve this terminal
35
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
36
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
37
- # Logged in as ada
38
- runlab whoami # ada (agent) at https://runlab.tools
34
+ pip install runlab-client # Python 3.10+, no dependencies
35
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
36
+ ```
37
+
38
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
39
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
40
+
41
+ ```text
42
+ treatment vs baseline on acc (higher is better): treatment is better
43
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
44
+ ...
45
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
46
+ The verdict page: https://runlab.tools/v/...
39
47
  ```
40
48
 
41
- - Another program called `runlab` first on your PATH? Every command here also runs as
42
- `runlab-client login | whoami | logout | worker` (or `python -m runlab_client ...`), and
43
- `runlab-client --version` names the version.
44
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
45
- headless GPU box, open the printed link on any device (your laptop or phone).
46
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
47
- that asked. Approve only if the code matches the one in your terminal.
48
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
49
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
50
- until you log out. `runlab logout` revokes it and deletes the file.
51
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
52
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
53
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
54
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
49
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
50
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
55
51
 
56
52
  ## Quickstart
57
53
 
58
54
  ```python
59
55
  from runlab_client import RunLab
60
56
 
61
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
62
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
57
+ rl = RunLab() # the key `runlab demo` / `login` saved
63
58
 
64
59
  # 1. Pre-register, before any seed runs.
65
60
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -83,11 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
83
78
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
84
79
  ```
85
80
 
86
- `train` is your own training run. The numbers above are an illustration. A runnable version
87
- with a simulated `train` is
88
- [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py),
89
- and [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
90
- does the same loop over plain HTTP.
81
+ `train` is your training run.
82
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
83
+ runs as is, with a simulated `train`;
84
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
85
+ is the same loop over plain HTTP.
91
86
 
92
87
  ## The loop
93
88
 
@@ -100,20 +95,6 @@ does the same loop over plain HTTP.
100
95
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
101
96
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
102
97
 
103
- ### Each user sees only their projects
104
-
105
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
106
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
107
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
108
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
109
- project's closest match is one of yours. Project names may not contain `~`.
110
-
111
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
112
- a scored higher. Whether higher is better comes from the metric name: lower for names
113
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
114
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
115
- `better="lower"` to override.
116
-
117
98
  ### Results
118
99
 
119
100
  RunLab stores each result as one line:
@@ -130,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
130
111
  `record()` as is.
131
112
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
132
113
  a 400.
133
- - Project names are case-insensitive.
114
+ - Project names are case-insensitive and may not contain `~`.
134
115
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
135
116
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
136
117
  in configurations is refused (`refused_by: equal_budget`).
137
118
 
138
119
  ### Verdicts
139
120
 
121
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
122
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
123
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
124
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
125
+
140
126
  | `verdict` | Meaning |
141
127
  |---|---|
142
128
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -146,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
146
132
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
147
133
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
148
134
 
149
- Read these fields next to the verdict:
150
-
151
- - `prereg.status`:
152
- - `ok`;
153
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
154
- after more than 3 seeds had been seen;
155
- - `missing`: no plan;
156
- - `tampered`.
157
- - `warnings`, e.g. `peeking`.
158
- - `report()`, before you quote anything.
135
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
136
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
137
+ `tampered`. Then read `warnings`, e.g. `peeking`.
159
138
 
160
139
  ### What keeps a verdict honest
161
140
 
@@ -169,8 +148,7 @@ Read these fields next to the verdict:
169
148
 
170
149
  ## Studies: let RunLab run the seeds
171
150
 
172
- RunLab can also run the seeds for you.
173
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
151
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
174
152
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
175
153
 
176
154
  ```python
@@ -186,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
186
164
  placeholders, using values of letters, digits and `_.:+=-` only.
187
165
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
188
166
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
189
- - A study never spends more than `max_usd`.
190
167
  - `state` is one of:
191
168
 
192
169
  | `state` | Meaning |
@@ -194,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
194
171
  | `running` | Still running seeds. |
195
172
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
196
173
  | `undecided` | `n_max` seeds ran without a decision. |
197
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
198
- | `killed` | Stopped at the dollar cap. |
174
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
175
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
199
176
  | `failed` | The command failed or printed no result; see `detail`. |
200
177
  | `lost` | Stopped reporting long past its deadline. |
201
178
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -208,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
208
185
  params' names are filled in (no shell runs it).
209
186
 
210
187
  ```bash
211
- pip install runlab-client
212
- runlab login
213
188
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
214
189
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
215
190
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -227,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
227
202
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
228
203
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
229
204
 
205
+ ## Signing in
206
+
207
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
208
+
209
+ | Question | Answer |
210
+ |---|---|
211
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
212
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
213
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
214
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
215
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
216
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
217
+
230
218
  ## Errors
231
219
 
232
220
  Every failure raises `RunLabError` with:
@@ -239,20 +227,19 @@ Every failure raises `RunLabError` with:
239
227
  |---|---|
240
228
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
241
229
  | 401 | Missing or wrong token. |
242
- | 403 | Your role cannot do this: writes need `agent` or above. |
230
+ | 403 | This key may not do this; the message says why. |
243
231
  | 404 | No such verdict or study. |
244
232
  | 413 | Body over 64 KiB; send results in several calls. |
245
233
 
246
234
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
247
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
248
- harmless anyway.
235
+ 429, 502, 503 and 504. It never retries POST requests.
249
236
 
250
237
  ## Client reference
251
238
 
252
239
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
253
240
  the decoded JSON of one operation in
254
241
  [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
255
- With no arguments it uses the key `runlab login` saved (see [Get started](#get-started)).
242
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
256
243
 
257
244
  | Method | Operation | Returns |
258
245
  |---|---|---|
@@ -275,6 +262,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
275
262
 
276
263
  | Command | What it does |
277
264
  |---|---|
265
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
278
266
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
279
267
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
280
268
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
@@ -289,6 +277,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
289
277
  is the short version for Claude Code, Codex and other agents.
290
278
  [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
291
279
 
280
+ ## Support
281
+
282
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
283
+
292
284
  ## License
293
285
 
294
286
  MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
@@ -1,5 +1,9 @@
1
1
  # runlab-client
2
2
 
3
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
4
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
5
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
6
+
3
7
  **Did my change actually help?** You record each seed's metric for two arms: your change and
4
8
  its baseline. RunLab gives back a verdict you can quote, built from:
5
9
 
@@ -7,9 +11,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
7
11
  - a paired test across seeds, with a 95 % confidence interval;
8
12
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
9
13
 
10
- The RunLab Python client gives access to the RunLab API using only the standard library,
11
- on Python 3.10+.
12
-
13
14
  ## Documentation
14
15
 
15
16
  - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
@@ -18,36 +19,30 @@ on Python 3.10+.
18
19
  ## Get started
19
20
 
20
21
  ```bash
21
- pip install runlab-client # Python 3.10+, no dependencies
22
- runlab login # sign in with GitHub, approve this terminal
23
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
24
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
25
- # Logged in as ada
26
- runlab whoami # ada (agent) at https://runlab.tools
22
+ pip install runlab-client # Python 3.10+, no dependencies
23
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
24
+ ```
25
+
26
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
27
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
28
+
29
+ ```text
30
+ treatment vs baseline on acc (higher is better): treatment is better
31
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
32
+ ...
33
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
34
+ The verdict page: https://runlab.tools/v/...
27
35
  ```
28
36
 
29
- - Another program called `runlab` first on your PATH? Every command here also runs as
30
- `runlab-client login | whoami | logout | worker` (or `python -m runlab_client ...`), and
31
- `runlab-client --version` names the version.
32
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
33
- headless GPU box, open the printed link on any device (your laptop or phone).
34
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
35
- that asked. Approve only if the code matches the one in your terminal.
36
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
37
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
38
- until you log out. `runlab logout` revokes it and deletes the file.
39
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
40
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
41
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
42
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
37
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
38
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
43
39
 
44
40
  ## Quickstart
45
41
 
46
42
  ```python
47
43
  from runlab_client import RunLab
48
44
 
49
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
50
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
45
+ rl = RunLab() # the key `runlab demo` / `login` saved
51
46
 
52
47
  # 1. Pre-register, before any seed runs.
53
48
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -71,11 +66,11 @@ for c in rl.report("resnet-aug")["claims"]:
71
66
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
72
67
  ```
73
68
 
74
- `train` is your own training run. The numbers above are an illustration. A runnable version
75
- with a simulated `train` is
76
- [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py),
77
- and [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
78
- does the same loop over plain HTTP.
69
+ `train` is your training run.
70
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
71
+ runs as is, with a simulated `train`;
72
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
73
+ is the same loop over plain HTTP.
79
74
 
80
75
  ## The loop
81
76
 
@@ -88,20 +83,6 @@ does the same loop over plain HTTP.
88
83
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
89
84
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
90
85
 
91
- ### Each user sees only their projects
92
-
93
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
94
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
95
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
96
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
97
- project's closest match is one of yours. Project names may not contain `~`.
98
-
99
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
100
- a scored higher. Whether higher is better comes from the metric name: lower for names
101
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
102
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
103
- `better="lower"` to override.
104
-
105
86
  ### Results
106
87
 
107
88
  RunLab stores each result as one line:
@@ -118,13 +99,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
118
99
  `record()` as is.
119
100
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
120
101
  a 400.
121
- - Project names are case-insensitive.
102
+ - Project names are case-insensitive and may not contain `~`.
122
103
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
123
104
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
124
105
  in configurations is refused (`refused_by: equal_budget`).
125
106
 
126
107
  ### Verdicts
127
108
 
109
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
110
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
111
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
112
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
113
+
128
114
  | `verdict` | Meaning |
129
115
  |---|---|
130
116
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -134,16 +120,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
134
120
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
135
121
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
136
122
 
137
- Read these fields next to the verdict:
138
-
139
- - `prereg.status`:
140
- - `ok`;
141
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
142
- after more than 3 seeds had been seen;
143
- - `missing`: no plan;
144
- - `tampered`.
145
- - `warnings`, e.g. `peeking`.
146
- - `report()`, before you quote anything.
123
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
124
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
125
+ `tampered`. Then read `warnings`, e.g. `peeking`.
147
126
 
148
127
  ### What keeps a verdict honest
149
128
 
@@ -157,8 +136,7 @@ Read these fields next to the verdict:
157
136
 
158
137
  ## Studies: let RunLab run the seeds
159
138
 
160
- RunLab can also run the seeds for you.
161
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
139
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
162
140
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
163
141
 
164
142
  ```python
@@ -174,7 +152,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
174
152
  placeholders, using values of letters, digits and `_.:+=-` only.
175
153
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
176
154
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
177
- - A study never spends more than `max_usd`.
178
155
  - `state` is one of:
179
156
 
180
157
  | `state` | Meaning |
@@ -182,8 +159,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
182
159
  | `running` | Still running seeds. |
183
160
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
184
161
  | `undecided` | `n_max` seeds ran without a decision. |
185
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
186
- | `killed` | Stopped at the dollar cap. |
162
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
163
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
187
164
  | `failed` | The command failed or printed no result; see `detail`. |
188
165
  | `lost` | Stopped reporting long past its deadline. |
189
166
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -196,8 +173,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
196
173
  params' names are filled in (no shell runs it).
197
174
 
198
175
  ```bash
199
- pip install runlab-client
200
- runlab login
201
176
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
202
177
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
203
178
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -215,6 +190,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
215
190
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
216
191
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
217
192
 
193
+ ## Signing in
194
+
195
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
196
+
197
+ | Question | Answer |
198
+ |---|---|
199
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
200
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
201
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
202
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
203
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
204
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
205
+
218
206
  ## Errors
219
207
 
220
208
  Every failure raises `RunLabError` with:
@@ -227,20 +215,19 @@ Every failure raises `RunLabError` with:
227
215
  |---|---|
228
216
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
229
217
  | 401 | Missing or wrong token. |
230
- | 403 | Your role cannot do this: writes need `agent` or above. |
218
+ | 403 | This key may not do this; the message says why. |
231
219
  | 404 | No such verdict or study. |
232
220
  | 413 | Body over 64 KiB; send results in several calls. |
233
221
 
234
222
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
235
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
236
- harmless anyway.
223
+ 429, 502, 503 and 504. It never retries POST requests.
237
224
 
238
225
  ## Client reference
239
226
 
240
227
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
241
228
  the decoded JSON of one operation in
242
229
  [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
243
- With no arguments it uses the key `runlab login` saved (see [Get started](#get-started)).
230
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
244
231
 
245
232
  | Method | Operation | Returns |
246
233
  |---|---|---|
@@ -263,6 +250,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
263
250
 
264
251
  | Command | What it does |
265
252
  |---|---|
253
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
266
254
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
267
255
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
268
256
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
@@ -277,6 +265,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
277
265
  is the short version for Claude Code, Codex and other agents.
278
266
  [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
279
267
 
268
+ ## Support
269
+
270
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
271
+
280
272
  ## License
281
273
 
282
274
  MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "runlab-client"
7
- version = "0.2.1"
7
+ version = "0.3.0"
8
8
  description = "Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -1,12 +1,13 @@
1
1
  """runlab-client: Python client for the RunLab research API.
2
2
 
3
3
  $ pip install runlab-client
4
- $ runlab login # sign in with GitHub; the key stays on this machine
4
+ $ runlab demo # GitHub sign-in, then a first verdict on simulated numbers
5
+ $ runlab login # sign in only; the key stays on this machine
5
6
  $ runlab-client login # the same, when another `runlab` comes first on PATH
6
7
 
7
8
  from runlab_client import RunLab
8
9
 
9
- rl = RunLab() # after `runlab login`, or with RUNLAB_URL / RUNLAB_TOKEN
10
+ rl = RunLab() # the key `runlab demo` / `login` saved, or RUNLAB_TOKEN
10
11
  rl.plan("demo", "treatment", "baseline", "acc", hypothesis="the change helps", n_max=10,
11
12
  stopping="fixed")
12
13
  rl.result("demo", "treatment", acc=0.91, seed=0)
@@ -36,6 +37,7 @@ from __future__ import annotations
36
37
 
37
38
  import json
38
39
  import os
40
+ import random
39
41
  import re
40
42
  import shlex
41
43
  import socket
@@ -51,7 +53,7 @@ from pathlib import Path
51
53
  from typing import Any
52
54
 
53
55
  __all__ = ["RunLab", "RunLabError", "login", "result_line"]
54
- __version__ = "0.2.1"
56
+ __version__ = "0.3.0"
55
57
 
56
58
  DEFAULT_URL = "https://runlab.tools"
57
59
 
@@ -615,6 +617,64 @@ def _cmd_logout(args: Any) -> int:
615
617
  return 0
616
618
 
617
619
 
620
+ DEMO_SEEDS = 10
621
+ QUICKSTART = ("https://github.com/omer-nachman/runlab-python/blob/main/examples/"
622
+ "quickstart.py")
623
+
624
+
625
+ def _demo_train(arm: str, seed: int) -> float:
626
+ """The demo's simulated training run, as in examples/quickstart.py: accuracy 0.80 with
627
+ seed noise (sd 0.01), and the change ("treatment") adds +0.02."""
628
+ rng = random.Random(f"{arm}-{seed}")
629
+ return 0.80 + rng.gauss(0, 0.01) + (0.02 if arm == "treatment" else 0.0)
630
+
631
+
632
+ def _cmd_demo(args: Any) -> int:
633
+ """Sign in if needed, then the whole loop on simulated numbers: plan, record 10 seeds per
634
+ arm, compare, check it is quotable; prints the verdict and where to go next."""
635
+ def say(line: str) -> None:
636
+ print(line, file=sys.stderr)
637
+
638
+ t0, project = time.monotonic(), args.project
639
+ try:
640
+ rl = RunLab(args.url)
641
+ if not rl.token:
642
+ say(f"Not logged in to {rl.url}: signing in first.")
643
+ say(f"Logged in as {login(args.url, browser=not args.no_browser)['user']}")
644
+ rl = RunLab(args.url)
645
+ say(f"1/4 plan: treatment vs baseline on acc, {DEMO_SEEDS} seeds, one test at the end")
646
+ rl.plan(project, "treatment", "baseline", "acc", n_max=DEMO_SEEDS, stopping="fixed",
647
+ hypothesis="the change raises accuracy")
648
+ say(f"2/4 record: {DEMO_SEEDS} seeds per arm (simulated: the change adds +0.02)")
649
+ rl.record(project, [result_line(arm, "acc", round(_demo_train(arm, seed), 6), seed=seed)
650
+ for seed in range(DEMO_SEEDS) for arm in ("treatment", "baseline")])
651
+ say("3/4 compare: paired test over the shared seeds")
652
+ v = rl.compare(project, "treatment", "baseline", "acc")
653
+ say("4/4 report: may a write-up quote it?")
654
+ claims = rl.report(project, arm="treatment", metric="acc").get("claims") or []
655
+ listed = rl.verdicts(project, limit=1)
656
+ rows = listed.get("verdicts") if isinstance(listed, dict) else listed
657
+ except RunLabError as exc:
658
+ print(f"error: {exc}", file=sys.stderr)
659
+ return 1
660
+ except ValueError as exc: # a project name RunLab cannot store
661
+ print(f"error: {exc}", file=sys.stderr)
662
+ return 2
663
+ except KeyboardInterrupt:
664
+ print("stopped", file=sys.stderr)
665
+ return 130
666
+ print()
667
+ print(v.get("text") or v.get("verdict"))
668
+ for c in claims:
669
+ print(f"{c.get('status')}: {c.get('quote')}")
670
+ vid = rows[0].get("id") if rows and isinstance(rows[0], dict) else None
671
+ if vid is not None:
672
+ print(f"\nThe verdict page: {rl.url}/v/{vid}")
673
+ print(f"\nDone in {time.monotonic() - t0:.0f} s, in your project `{project}`. The numbers "
674
+ "were simulated.\nNext, your own runs: replace train() in " + QUICKSTART)
675
+ return 0
676
+
677
+
618
678
  # ---------------------------------------------------------------- command line
619
679
 
620
680
  def main(argv: list[str] | None = None, prog: str | None = None) -> int:
@@ -640,6 +700,15 @@ def main(argv: list[str] | None = None, prog: str | None = None) -> int:
640
700
  who = sub.add_parser("whoami", help="who this machine is signed in as")
641
701
  who.add_argument("--url", help="your RunLab URL (default: the one you logged in to)")
642
702
  sub.add_parser("logout", help="revoke the saved key and delete it")
703
+ dm = sub.add_parser("demo", help="try RunLab in one command: sign in if needed, then a "
704
+ "10-seed comparison on simulated numbers, ending in a verdict",
705
+ description="Sign in if this machine is not yet, then plan, record 10 "
706
+ "simulated seeds per arm, compare and report, in the project --project.")
707
+ dm.add_argument("--project", default="runlab-demo",
708
+ help="the project to use (default: runlab-demo)")
709
+ dm.add_argument("--url", help="your RunLab URL (default: env RUNLAB_URL, else the one you "
710
+ f"logged in to, else {DEFAULT_URL})")
711
+ dm.add_argument("--no-browser", action="store_true", help="only print the sign-in link")
643
712
  w = sub.add_parser("worker", help="run a study's seed pairs on this machine",
644
713
  description="Run seed pairs of a planned comparison here and post the "
645
714
  "numbers to RunLab until its sequential test decides, n_max seeds ran "
@@ -670,8 +739,8 @@ def main(argv: list[str] | None = None, prog: str | None = None) -> int:
670
739
  w.add_argument("--cwd", help="where the command runs (default: here)")
671
740
  args = ap.parse_args(argv)
672
741
  if args.command != "worker":
673
- return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout}[
674
- args.command](args)
742
+ return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout,
743
+ "demo": _cmd_demo}[args.command](args)
675
744
  try:
676
745
  rl = RunLab(args.url, args.token)
677
746
  sid = args.study
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: runlab-client
3
- Version: 0.2.1
3
+ Version: 0.3.0
4
4
  Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
5
5
  Author: omer-nachman
6
6
  License-Expression: MIT
@@ -12,6 +12,10 @@ Dynamic: license-file
12
12
 
13
13
  # runlab-client
14
14
 
15
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
16
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
17
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
18
+
15
19
  **Did my change actually help?** You record each seed's metric for two arms: your change and
16
20
  its baseline. RunLab gives back a verdict you can quote, built from:
17
21
 
@@ -19,9 +23,6 @@ its baseline. RunLab gives back a verdict you can quote, built from:
19
23
  - a paired test across seeds, with a 95 % confidence interval;
20
24
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
21
25
 
22
- The RunLab Python client gives access to the RunLab API using only the standard library,
23
- on Python 3.10+.
24
-
25
26
  ## Documentation
26
27
 
27
28
  - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
@@ -30,36 +31,30 @@ on Python 3.10+.
30
31
  ## Get started
31
32
 
32
33
  ```bash
33
- pip install runlab-client # Python 3.10+, no dependencies
34
- runlab login # sign in with GitHub, approve this terminal
35
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
36
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
37
- # Logged in as ada
38
- runlab whoami # ada (agent) at https://runlab.tools
34
+ pip install runlab-client # Python 3.10+, no dependencies
35
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
36
+ ```
37
+
38
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
39
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
40
+
41
+ ```text
42
+ treatment vs baseline on acc (higher is better): treatment is better
43
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
44
+ ...
45
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
46
+ The verdict page: https://runlab.tools/v/...
39
47
  ```
40
48
 
41
- - Another program called `runlab` first on your PATH? Every command here also runs as
42
- `runlab-client login | whoami | logout | worker` (or `python -m runlab_client ...`), and
43
- `runlab-client --version` names the version.
44
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
45
- headless GPU box, open the printed link on any device (your laptop or phone).
46
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
47
- that asked. Approve only if the code matches the one in your terminal.
48
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
49
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
50
- until you log out. `runlab logout` revokes it and deletes the file.
51
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
52
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
53
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
54
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
49
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
50
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
55
51
 
56
52
  ## Quickstart
57
53
 
58
54
  ```python
59
55
  from runlab_client import RunLab
60
56
 
61
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
62
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
57
+ rl = RunLab() # the key `runlab demo` / `login` saved
63
58
 
64
59
  # 1. Pre-register, before any seed runs.
65
60
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -83,11 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
83
78
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
84
79
  ```
85
80
 
86
- `train` is your own training run. The numbers above are an illustration. A runnable version
87
- with a simulated `train` is
88
- [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py),
89
- and [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
90
- does the same loop over plain HTTP.
81
+ `train` is your training run.
82
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
83
+ runs as is, with a simulated `train`;
84
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
85
+ is the same loop over plain HTTP.
91
86
 
92
87
  ## The loop
93
88
 
@@ -100,20 +95,6 @@ does the same loop over plain HTTP.
100
95
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
101
96
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
102
97
 
103
- ### Each user sees only their projects
104
-
105
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
106
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
107
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
108
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
109
- project's closest match is one of yours. Project names may not contain `~`.
110
-
111
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
112
- a scored higher. Whether higher is better comes from the metric name: lower for names
113
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
114
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
115
- `better="lower"` to override.
116
-
117
98
  ### Results
118
99
 
119
100
  RunLab stores each result as one line:
@@ -130,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
130
111
  `record()` as is.
131
112
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
132
113
  a 400.
133
- - Project names are case-insensitive.
114
+ - Project names are case-insensitive and may not contain `~`.
134
115
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
135
116
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
136
117
  in configurations is refused (`refused_by: equal_budget`).
137
118
 
138
119
  ### Verdicts
139
120
 
121
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
122
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
123
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
124
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
125
+
140
126
  | `verdict` | Meaning |
141
127
  |---|---|
142
128
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -146,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
146
132
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
147
133
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
148
134
 
149
- Read these fields next to the verdict:
150
-
151
- - `prereg.status`:
152
- - `ok`;
153
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
154
- after more than 3 seeds had been seen;
155
- - `missing`: no plan;
156
- - `tampered`.
157
- - `warnings`, e.g. `peeking`.
158
- - `report()`, before you quote anything.
135
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
136
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
137
+ `tampered`. Then read `warnings`, e.g. `peeking`.
159
138
 
160
139
  ### What keeps a verdict honest
161
140
 
@@ -169,8 +148,7 @@ Read these fields next to the verdict:
169
148
 
170
149
  ## Studies: let RunLab run the seeds
171
150
 
172
- RunLab can also run the seeds for you.
173
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
151
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
174
152
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
175
153
 
176
154
  ```python
@@ -186,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
186
164
  placeholders, using values of letters, digits and `_.:+=-` only.
187
165
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
188
166
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
189
- - A study never spends more than `max_usd`.
190
167
  - `state` is one of:
191
168
 
192
169
  | `state` | Meaning |
@@ -194,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
194
171
  | `running` | Still running seeds. |
195
172
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
196
173
  | `undecided` | `n_max` seeds ran without a decision. |
197
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
198
- | `killed` | Stopped at the dollar cap. |
174
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
175
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
199
176
  | `failed` | The command failed or printed no result; see `detail`. |
200
177
  | `lost` | Stopped reporting long past its deadline. |
201
178
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -208,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
208
185
  params' names are filled in (no shell runs it).
209
186
 
210
187
  ```bash
211
- pip install runlab-client
212
- runlab login
213
188
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
214
189
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
215
190
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -227,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
227
202
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
228
203
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
229
204
 
205
+ ## Signing in
206
+
207
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
208
+
209
+ | Question | Answer |
210
+ |---|---|
211
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
212
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
213
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
214
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
215
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
216
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
217
+
230
218
  ## Errors
231
219
 
232
220
  Every failure raises `RunLabError` with:
@@ -239,20 +227,19 @@ Every failure raises `RunLabError` with:
239
227
  |---|---|
240
228
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
241
229
  | 401 | Missing or wrong token. |
242
- | 403 | Your role cannot do this: writes need `agent` or above. |
230
+ | 403 | This key may not do this; the message says why. |
243
231
  | 404 | No such verdict or study. |
244
232
  | 413 | Body over 64 KiB; send results in several calls. |
245
233
 
246
234
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
247
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
248
- harmless anyway.
235
+ 429, 502, 503 and 504. It never retries POST requests.
249
236
 
250
237
  ## Client reference
251
238
 
252
239
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
253
240
  the decoded JSON of one operation in
254
241
  [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
255
- With no arguments it uses the key `runlab login` saved (see [Get started](#get-started)).
242
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
256
243
 
257
244
  | Method | Operation | Returns |
258
245
  |---|---|---|
@@ -275,6 +262,7 @@ With no arguments it uses the key `runlab login` saved (see [Get started](#get-s
275
262
 
276
263
  | Command | What it does |
277
264
  |---|---|
265
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
278
266
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
279
267
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
280
268
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
@@ -289,6 +277,10 @@ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
289
277
  is the short version for Claude Code, Codex and other agents.
290
278
  [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
291
279
 
280
+ ## Support
281
+
282
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
283
+
292
284
  ## License
293
285
 
294
286
  MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
File without changes
File without changes