runlab-client 0.2.0__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: runlab-client
3
- Version: 0.2.0
3
+ Version: 0.3.0
4
4
  Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
5
5
  Author: omer-nachman
6
6
  License-Expression: MIT
@@ -12,6 +12,10 @@ Dynamic: license-file
12
12
 
13
13
  # runlab-client
14
14
 
15
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
16
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
17
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
18
+
15
19
  **Did my change actually help?** You record each seed's metric for two arms: your change and
16
20
  its baseline. RunLab gives back a verdict you can quote, built from:
17
21
 
@@ -19,39 +23,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
19
23
  - a paired test across seeds, with a 95 % confidence interval;
20
24
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
21
25
 
22
- This repo has the Python client (standard library only), the
23
- [OpenAPI spec](openapi.json) ([rendered](api.html)) and examples.
26
+ ## Documentation
27
+
28
+ - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
29
+ - [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
24
30
 
25
31
  ## Get started
26
32
 
27
33
  ```bash
28
- pip install runlab-client # Python 3.10+, no dependencies
29
- runlab login # sign in with GitHub, approve this terminal
30
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
31
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
32
- # Logged in as ada
33
- runlab whoami # ada (agent) at https://runlab.tools
34
+ pip install runlab-client # Python 3.10+, no dependencies
35
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
36
+ ```
37
+
38
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
39
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
40
+
41
+ ```text
42
+ treatment vs baseline on acc (higher is better): treatment is better
43
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
44
+ ...
45
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
46
+ The verdict page: https://runlab.tools/v/...
34
47
  ```
35
48
 
36
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
37
- headless GPU box, open the printed link on any device (your laptop or phone).
38
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
39
- that asked. Approve only if the code matches the one in your terminal.
40
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
41
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
42
- until you log out. `runlab logout` revokes it and deletes the file.
43
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
44
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
45
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
46
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
49
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
50
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
47
51
 
48
52
  ## Quickstart
49
53
 
50
54
  ```python
51
55
  from runlab_client import RunLab
52
56
 
53
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
54
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
57
+ rl = RunLab() # the key `runlab demo` / `login` saved
55
58
 
56
59
  # 1. Pre-register, before any seed runs.
57
60
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -75,9 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
75
78
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
76
79
  ```
77
80
 
78
- `train` is your own training run. The numbers above are an illustration. A runnable version
79
- with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), and
80
- [examples/curl.sh](examples/curl.sh) does the same loop over plain HTTP.
81
+ `train` is your training run.
82
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
83
+ runs as is, with a simulated `train`;
84
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
85
+ is the same loop over plain HTTP.
81
86
 
82
87
  ## The loop
83
88
 
@@ -90,20 +95,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
90
95
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
91
96
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
92
97
 
93
- ### Each user sees only their projects
94
-
95
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
96
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
97
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
98
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
99
- project's closest match is one of yours. Project names may not contain `~`.
100
-
101
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
102
- a scored higher. Whether higher is better comes from the metric name: lower for names
103
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
104
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
105
- `better="lower"` to override.
106
-
107
98
  ### Results
108
99
 
109
100
  RunLab stores each result as one line:
@@ -120,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
120
111
  `record()` as is.
121
112
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
122
113
  a 400.
123
- - Project names are case-insensitive.
114
+ - Project names are case-insensitive and may not contain `~`.
124
115
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
125
116
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
126
117
  in configurations is refused (`refused_by: equal_budget`).
127
118
 
128
119
  ### Verdicts
129
120
 
121
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
122
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
123
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
124
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
125
+
130
126
  | `verdict` | Meaning |
131
127
  |---|---|
132
128
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -136,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
136
132
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
137
133
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
138
134
 
139
- Read these fields next to the verdict:
140
-
141
- - `prereg.status`:
142
- - `ok`;
143
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
144
- after more than 3 seeds had been seen;
145
- - `missing`: no plan;
146
- - `tampered`.
147
- - `warnings`, e.g. `peeking`.
148
- - `report()`, before you quote anything.
135
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
136
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
137
+ `tampered`. Then read `warnings`, e.g. `peeking`.
149
138
 
150
139
  ### What keeps a verdict honest
151
140
 
@@ -159,8 +148,7 @@ Read these fields next to the verdict:
159
148
 
160
149
  ## Studies: let RunLab run the seeds
161
150
 
162
- RunLab can also run the seeds for you.
163
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
151
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
164
152
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
165
153
 
166
154
  ```python
@@ -176,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
176
164
  placeholders, using values of letters, digits and `_.:+=-` only.
177
165
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
178
166
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
179
- - A study never spends more than `max_usd`.
180
167
  - `state` is one of:
181
168
 
182
169
  | `state` | Meaning |
@@ -184,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
184
171
  | `running` | Still running seeds. |
185
172
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
186
173
  | `undecided` | `n_max` seeds ran without a decision. |
187
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
188
- | `killed` | Stopped at the dollar cap. |
174
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
175
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
189
176
  | `failed` | The command failed or printed no result; see `detail`. |
190
177
  | `lost` | Stopped reporting long past its deadline. |
191
178
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -198,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
198
185
  params' names are filled in (no shell runs it).
199
186
 
200
187
  ```bash
201
- pip install runlab-client
202
- runlab login
203
188
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
204
189
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
205
190
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -217,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
217
202
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
218
203
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
219
204
 
205
+ ## Signing in
206
+
207
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
208
+
209
+ | Question | Answer |
210
+ |---|---|
211
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
212
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
213
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
214
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
215
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
216
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
217
+
220
218
  ## Errors
221
219
 
222
220
  Every failure raises `RunLabError` with:
@@ -229,19 +227,19 @@ Every failure raises `RunLabError` with:
229
227
  |---|---|
230
228
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
231
229
  | 401 | Missing or wrong token. |
232
- | 403 | Your role cannot do this: writes need `agent` or above. |
230
+ | 403 | This key may not do this; the message says why. |
233
231
  | 404 | No such verdict or study. |
234
232
  | 413 | Body over 64 KiB; send results in several calls. |
235
233
 
236
234
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
237
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
238
- harmless anyway.
235
+ 429, 502, 503 and 504. It never retries POST requests.
239
236
 
240
237
  ## Client reference
241
238
 
242
239
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
243
- the decoded JSON of one operation in [openapi.json](openapi.json). With no arguments it uses
244
- the key `runlab login` saved (see [Get started](#get-started)).
240
+ the decoded JSON of one operation in
241
+ [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
242
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
245
243
 
246
244
  | Method | Operation | Returns |
247
245
  |---|---|---|
@@ -264,16 +262,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
264
262
 
265
263
  | Command | What it does |
266
264
  |---|---|
265
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
267
266
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
268
267
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
269
268
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
270
269
  | `runlab worker ...` | Run a study's seed pairs here (see above). |
270
+ | `runlab --version` | The client's version. |
271
+
272
+ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
271
273
 
272
274
  ## Coding agents
273
275
 
274
- [AGENTS.md](AGENTS.md) is the short version for Claude Code, Codex and other agents.
275
- [llms.txt](llms.txt) indexes this repo for LLM tools.
276
+ [AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
277
+ is the short version for Claude Code, Codex and other agents.
278
+ [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
279
+
280
+ ## Support
281
+
282
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
276
283
 
277
284
  ## License
278
285
 
279
- MIT ([LICENSE](LICENSE)).
286
+ MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
@@ -1,5 +1,9 @@
1
1
  # runlab-client
2
2
 
3
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
4
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
5
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
6
+
3
7
  **Did my change actually help?** You record each seed's metric for two arms: your change and
4
8
  its baseline. RunLab gives back a verdict you can quote, built from:
5
9
 
@@ -7,39 +11,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
7
11
  - a paired test across seeds, with a 95 % confidence interval;
8
12
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
9
13
 
10
- This repo has the Python client (standard library only), the
11
- [OpenAPI spec](openapi.json) ([rendered](api.html)) and examples.
14
+ ## Documentation
15
+
16
+ - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
17
+ - [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
12
18
 
13
19
  ## Get started
14
20
 
15
21
  ```bash
16
- pip install runlab-client # Python 3.10+, no dependencies
17
- runlab login # sign in with GitHub, approve this terminal
18
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
19
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
20
- # Logged in as ada
21
- runlab whoami # ada (agent) at https://runlab.tools
22
+ pip install runlab-client # Python 3.10+, no dependencies
23
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
24
+ ```
25
+
26
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
27
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
28
+
29
+ ```text
30
+ treatment vs baseline on acc (higher is better): treatment is better
31
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
32
+ ...
33
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
34
+ The verdict page: https://runlab.tools/v/...
22
35
  ```
23
36
 
24
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
25
- headless GPU box, open the printed link on any device (your laptop or phone).
26
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
27
- that asked. Approve only if the code matches the one in your terminal.
28
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
29
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
30
- until you log out. `runlab logout` revokes it and deletes the file.
31
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
32
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
33
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
34
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
37
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
38
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
35
39
 
36
40
  ## Quickstart
37
41
 
38
42
  ```python
39
43
  from runlab_client import RunLab
40
44
 
41
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
42
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
45
+ rl = RunLab() # the key `runlab demo` / `login` saved
43
46
 
44
47
  # 1. Pre-register, before any seed runs.
45
48
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -63,9 +66,11 @@ for c in rl.report("resnet-aug")["claims"]:
63
66
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
64
67
  ```
65
68
 
66
- `train` is your own training run. The numbers above are an illustration. A runnable version
67
- with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), and
68
- [examples/curl.sh](examples/curl.sh) does the same loop over plain HTTP.
69
+ `train` is your training run.
70
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
71
+ runs as is, with a simulated `train`;
72
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
73
+ is the same loop over plain HTTP.
69
74
 
70
75
  ## The loop
71
76
 
@@ -78,20 +83,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
78
83
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
79
84
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
80
85
 
81
- ### Each user sees only their projects
82
-
83
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
84
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
85
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
86
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
87
- project's closest match is one of yours. Project names may not contain `~`.
88
-
89
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
90
- a scored higher. Whether higher is better comes from the metric name: lower for names
91
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
92
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
93
- `better="lower"` to override.
94
-
95
86
  ### Results
96
87
 
97
88
  RunLab stores each result as one line:
@@ -108,13 +99,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
108
99
  `record()` as is.
109
100
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
110
101
  a 400.
111
- - Project names are case-insensitive.
102
+ - Project names are case-insensitive and may not contain `~`.
112
103
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
113
104
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
114
105
  in configurations is refused (`refused_by: equal_budget`).
115
106
 
116
107
  ### Verdicts
117
108
 
109
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
110
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
111
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
112
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
113
+
118
114
  | `verdict` | Meaning |
119
115
  |---|---|
120
116
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -124,16 +120,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
124
120
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
125
121
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
126
122
 
127
- Read these fields next to the verdict:
128
-
129
- - `prereg.status`:
130
- - `ok`;
131
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
132
- after more than 3 seeds had been seen;
133
- - `missing`: no plan;
134
- - `tampered`.
135
- - `warnings`, e.g. `peeking`.
136
- - `report()`, before you quote anything.
123
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
124
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
125
+ `tampered`. Then read `warnings`, e.g. `peeking`.
137
126
 
138
127
  ### What keeps a verdict honest
139
128
 
@@ -147,8 +136,7 @@ Read these fields next to the verdict:
147
136
 
148
137
  ## Studies: let RunLab run the seeds
149
138
 
150
- RunLab can also run the seeds for you.
151
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
139
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
152
140
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
153
141
 
154
142
  ```python
@@ -164,7 +152,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
164
152
  placeholders, using values of letters, digits and `_.:+=-` only.
165
153
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
166
154
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
167
- - A study never spends more than `max_usd`.
168
155
  - `state` is one of:
169
156
 
170
157
  | `state` | Meaning |
@@ -172,8 +159,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
172
159
  | `running` | Still running seeds. |
173
160
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
174
161
  | `undecided` | `n_max` seeds ran without a decision. |
175
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
176
- | `killed` | Stopped at the dollar cap. |
162
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
163
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
177
164
  | `failed` | The command failed or printed no result; see `detail`. |
178
165
  | `lost` | Stopped reporting long past its deadline. |
179
166
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -186,8 +173,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
186
173
  params' names are filled in (no shell runs it).
187
174
 
188
175
  ```bash
189
- pip install runlab-client
190
- runlab login
191
176
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
192
177
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
193
178
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -205,6 +190,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
205
190
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
206
191
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
207
192
 
193
+ ## Signing in
194
+
195
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
196
+
197
+ | Question | Answer |
198
+ |---|---|
199
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
200
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
201
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
202
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
203
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
204
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
205
+
208
206
  ## Errors
209
207
 
210
208
  Every failure raises `RunLabError` with:
@@ -217,19 +215,19 @@ Every failure raises `RunLabError` with:
217
215
  |---|---|
218
216
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
219
217
  | 401 | Missing or wrong token. |
220
- | 403 | Your role cannot do this: writes need `agent` or above. |
218
+ | 403 | This key may not do this; the message says why. |
221
219
  | 404 | No such verdict or study. |
222
220
  | 413 | Body over 64 KiB; send results in several calls. |
223
221
 
224
222
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
225
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
226
- harmless anyway.
223
+ 429, 502, 503 and 504. It never retries POST requests.
227
224
 
228
225
  ## Client reference
229
226
 
230
227
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
231
- the decoded JSON of one operation in [openapi.json](openapi.json). With no arguments it uses
232
- the key `runlab login` saved (see [Get started](#get-started)).
228
+ the decoded JSON of one operation in
229
+ [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
230
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
233
231
 
234
232
  | Method | Operation | Returns |
235
233
  |---|---|---|
@@ -252,16 +250,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
252
250
 
253
251
  | Command | What it does |
254
252
  |---|---|
253
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
255
254
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
256
255
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
257
256
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
258
257
  | `runlab worker ...` | Run a study's seed pairs here (see above). |
258
+ | `runlab --version` | The client's version. |
259
+
260
+ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
259
261
 
260
262
  ## Coding agents
261
263
 
262
- [AGENTS.md](AGENTS.md) is the short version for Claude Code, Codex and other agents.
263
- [llms.txt](llms.txt) indexes this repo for LLM tools.
264
+ [AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
265
+ is the short version for Claude Code, Codex and other agents.
266
+ [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
267
+
268
+ ## Support
269
+
270
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
264
271
 
265
272
  ## License
266
273
 
267
- MIT ([LICENSE](LICENSE)).
274
+ MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "runlab-client"
7
- version = "0.2.0"
7
+ version = "0.3.0"
8
8
  description = "Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -14,6 +14,8 @@ dependencies = []
14
14
 
15
15
  [project.scripts]
16
16
  runlab = "runlab_client:main"
17
+ # the same commands under a name no other `runlab` program on PATH takes
18
+ runlab-client = "runlab_client:main"
17
19
 
18
20
  [project.urls]
19
21
  Homepage = "https://github.com/omer-nachman/runlab-python"
@@ -1,11 +1,13 @@
1
1
  """runlab-client: Python client for the RunLab research API.
2
2
 
3
3
  $ pip install runlab-client
4
- $ runlab login # sign in with GitHub; the key stays on this machine
4
+ $ runlab demo # GitHub sign-in, then a first verdict on simulated numbers
5
+ $ runlab login # sign in only; the key stays on this machine
6
+ $ runlab-client login # the same, when another `runlab` comes first on PATH
5
7
 
6
8
  from runlab_client import RunLab
7
9
 
8
- rl = RunLab() # after `runlab login`, or with RUNLAB_URL / RUNLAB_TOKEN
10
+ rl = RunLab() # the key `runlab demo` / `login` saved, or RUNLAB_TOKEN
9
11
  rl.plan("demo", "treatment", "baseline", "acc", hypothesis="the change helps", n_max=10,
10
12
  stopping="fixed")
11
13
  rl.result("demo", "treatment", acc=0.91, seed=0)
@@ -35,6 +37,7 @@ from __future__ import annotations
35
37
 
36
38
  import json
37
39
  import os
40
+ import random
38
41
  import re
39
42
  import shlex
40
43
  import socket
@@ -50,7 +53,7 @@ from pathlib import Path
50
53
  from typing import Any
51
54
 
52
55
  __all__ = ["RunLab", "RunLabError", "login", "result_line"]
53
- __version__ = "0.2.0"
56
+ __version__ = "0.3.0"
54
57
 
55
58
  DEFAULT_URL = "https://runlab.tools"
56
59
 
@@ -580,12 +583,12 @@ def _cmd_login(args: Any) -> int:
580
583
  def _cmd_whoami(args: Any) -> int:
581
584
  rl = RunLab(args.url)
582
585
  if not rl.token:
583
- print(f"not logged in to {rl.url}: run `runlab login`", file=sys.stderr)
586
+ print(f"not logged in to {rl.url}: run `{args.prog} login`", file=sys.stderr)
584
587
  return 1
585
588
  try:
586
589
  me = rl.me()
587
590
  except RunLabError as exc:
588
- hint = ": run `runlab login`" if exc.status == 401 else ""
591
+ hint = f": run `{args.prog} login`" if exc.status == 401 else ""
589
592
  print(f"error: {exc}{hint}", file=sys.stderr)
590
593
  return 1
591
594
  print(f"{me.get('name')} ({me.get('role')}) at {rl.url}")
@@ -603,7 +606,7 @@ def _cmd_logout(args: Any) -> int:
603
606
  except RunLabError as exc:
604
607
  if exc.status != 401:
605
608
  print(f"error: cannot revoke the key at {url}: {exc.message}. It stays saved: run "
606
- f"`runlab logout` again, or delete {config_dir() / 'token'} to forget it "
609
+ f"`{args.prog} logout` again, or delete {config_dir() / 'token'} to forget it "
607
610
  "here only", file=sys.stderr)
608
611
  return 1
609
612
  how = "the key had already stopped working"
@@ -614,13 +617,78 @@ def _cmd_logout(args: Any) -> int:
614
617
  return 0
615
618
 
616
619
 
620
+ DEMO_SEEDS = 10
621
+ QUICKSTART = ("https://github.com/omer-nachman/runlab-python/blob/main/examples/"
622
+ "quickstart.py")
623
+
624
+
625
+ def _demo_train(arm: str, seed: int) -> float:
626
+ """The demo's simulated training run, as in examples/quickstart.py: accuracy 0.80 with
627
+ seed noise (sd 0.01), and the change ("treatment") adds +0.02."""
628
+ rng = random.Random(f"{arm}-{seed}")
629
+ return 0.80 + rng.gauss(0, 0.01) + (0.02 if arm == "treatment" else 0.0)
630
+
631
+
632
+ def _cmd_demo(args: Any) -> int:
633
+ """Sign in if needed, then the whole loop on simulated numbers: plan, record 10 seeds per
634
+ arm, compare, check it is quotable; prints the verdict and where to go next."""
635
+ def say(line: str) -> None:
636
+ print(line, file=sys.stderr)
637
+
638
+ t0, project = time.monotonic(), args.project
639
+ try:
640
+ rl = RunLab(args.url)
641
+ if not rl.token:
642
+ say(f"Not logged in to {rl.url}: signing in first.")
643
+ say(f"Logged in as {login(args.url, browser=not args.no_browser)['user']}")
644
+ rl = RunLab(args.url)
645
+ say(f"1/4 plan: treatment vs baseline on acc, {DEMO_SEEDS} seeds, one test at the end")
646
+ rl.plan(project, "treatment", "baseline", "acc", n_max=DEMO_SEEDS, stopping="fixed",
647
+ hypothesis="the change raises accuracy")
648
+ say(f"2/4 record: {DEMO_SEEDS} seeds per arm (simulated: the change adds +0.02)")
649
+ rl.record(project, [result_line(arm, "acc", round(_demo_train(arm, seed), 6), seed=seed)
650
+ for seed in range(DEMO_SEEDS) for arm in ("treatment", "baseline")])
651
+ say("3/4 compare: paired test over the shared seeds")
652
+ v = rl.compare(project, "treatment", "baseline", "acc")
653
+ say("4/4 report: may a write-up quote it?")
654
+ claims = rl.report(project, arm="treatment", metric="acc").get("claims") or []
655
+ listed = rl.verdicts(project, limit=1)
656
+ rows = listed.get("verdicts") if isinstance(listed, dict) else listed
657
+ except RunLabError as exc:
658
+ print(f"error: {exc}", file=sys.stderr)
659
+ return 1
660
+ except ValueError as exc: # a project name RunLab cannot store
661
+ print(f"error: {exc}", file=sys.stderr)
662
+ return 2
663
+ except KeyboardInterrupt:
664
+ print("stopped", file=sys.stderr)
665
+ return 130
666
+ print()
667
+ print(v.get("text") or v.get("verdict"))
668
+ for c in claims:
669
+ print(f"{c.get('status')}: {c.get('quote')}")
670
+ vid = rows[0].get("id") if rows and isinstance(rows[0], dict) else None
671
+ if vid is not None:
672
+ print(f"\nThe verdict page: {rl.url}/v/{vid}")
673
+ print(f"\nDone in {time.monotonic() - t0:.0f} s, in your project `{project}`. The numbers "
674
+ "were simulated.\nNext, your own runs: replace train() in " + QUICKSTART)
675
+ return 0
676
+
677
+
617
678
  # ---------------------------------------------------------------- command line
618
679
 
619
- def main(argv: list[str] | None = None) -> int:
620
- """`runlab login | whoami | logout | worker ...` (see the module doc)."""
680
+ def main(argv: list[str] | None = None, prog: str | None = None) -> int:
681
+ """`runlab login | whoami | logout | worker ...` (see the module doc); also installed as
682
+ `runlab-client`, and run by `python -m runlab_client`. Every hint names `prog`, the
683
+ command that was run (default: `runlab-client` when run under that name, else `runlab`),
684
+ so following it reaches this client and not another `runlab` on PATH."""
621
685
  import argparse
622
686
 
623
- ap = argparse.ArgumentParser(prog="runlab", description="RunLab from the command line.")
687
+ if prog is None:
688
+ prog = "runlab-client" if Path(sys.argv[0]).name.startswith("runlab-client") else "runlab"
689
+ ap = argparse.ArgumentParser(prog=prog, description="RunLab from the command line.")
690
+ ap.add_argument("--version", action="version", version=f"runlab-client {__version__}")
691
+ ap.set_defaults(prog=prog)
624
692
  sub = ap.add_subparsers(dest="command", required=True)
625
693
  lg = sub.add_parser("login", help="sign this machine in with GitHub",
626
694
  description="Print a link and a code, open the link when this machine "
@@ -632,6 +700,15 @@ def main(argv: list[str] | None = None) -> int:
632
700
  who = sub.add_parser("whoami", help="who this machine is signed in as")
633
701
  who.add_argument("--url", help="your RunLab URL (default: the one you logged in to)")
634
702
  sub.add_parser("logout", help="revoke the saved key and delete it")
703
+ dm = sub.add_parser("demo", help="try RunLab in one command: sign in if needed, then a "
704
+ "10-seed comparison on simulated numbers, ending in a verdict",
705
+ description="Sign in if this machine is not yet, then plan, record 10 "
706
+ "simulated seeds per arm, compare and report, in the project --project.")
707
+ dm.add_argument("--project", default="runlab-demo",
708
+ help="the project to use (default: runlab-demo)")
709
+ dm.add_argument("--url", help="your RunLab URL (default: env RUNLAB_URL, else the one you "
710
+ f"logged in to, else {DEFAULT_URL})")
711
+ dm.add_argument("--no-browser", action="store_true", help="only print the sign-in link")
635
712
  w = sub.add_parser("worker", help="run a study's seed pairs on this machine",
636
713
  description="Run seed pairs of a planned comparison here and post the "
637
714
  "numbers to RunLab until its sequential test decides, n_max seeds ran "
@@ -662,8 +739,8 @@ def main(argv: list[str] | None = None) -> int:
662
739
  w.add_argument("--cwd", help="where the command runs (default: here)")
663
740
  args = ap.parse_args(argv)
664
741
  if args.command != "worker":
665
- return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout}[
666
- args.command](args)
742
+ return {"login": _cmd_login, "whoami": _cmd_whoami, "logout": _cmd_logout,
743
+ "demo": _cmd_demo}[args.command](args)
667
744
  try:
668
745
  rl = RunLab(args.url, args.token)
669
746
  sid = args.study
@@ -0,0 +1,6 @@
1
+ """`python -m runlab_client login | whoami | logout | worker ...`: the `runlab` command, for
2
+ when neither `runlab` nor `runlab-client` is on PATH."""
3
+
4
+ from runlab_client import main
5
+
6
+ raise SystemExit(main(prog="python -m runlab_client"))
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: runlab-client
3
- Version: 0.2.0
3
+ Version: 0.3.0
4
4
  Summary: Python client for the RunLab research API: pre-registered A/B comparisons over seeds, verdicts with confidence intervals. Standard library only.
5
5
  Author: omer-nachman
6
6
  License-Expression: MIT
@@ -12,6 +12,10 @@ Dynamic: license-file
12
12
 
13
13
  # runlab-client
14
14
 
15
+ [![PyPI](https://img.shields.io/pypi/v/runlab-client)](https://pypi.org/project/runlab-client/)
16
+ ![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)
17
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)
18
+
15
19
  **Did my change actually help?** You record each seed's metric for two arms: your change and
16
20
  its baseline. RunLab gives back a verdict you can quote, built from:
17
21
 
@@ -19,39 +23,38 @@ its baseline. RunLab gives back a verdict you can quote, built from:
19
23
  - a paired test across seeds, with a 95 % confidence interval;
20
24
  - checks that refuse a comparison that would mislead, e.g. arms tuned with unequal budgets.
21
25
 
22
- This repo has the Python client (standard library only), the
23
- [OpenAPI spec](openapi.json) ([rendered](api.html)) and examples.
26
+ ## Documentation
27
+
28
+ - [API reference](https://omer-nachman.github.io/runlab-python/api.html)
29
+ - [Examples](https://github.com/omer-nachman/runlab-python/tree/main/examples): Python and curl
24
30
 
25
31
  ## Get started
26
32
 
27
33
  ```bash
28
- pip install runlab-client # Python 3.10+, no dependencies
29
- runlab login # sign in with GitHub, approve this terminal
30
- # Sign in to RunLab: open https://runlab.tools/login?code=BCDF-GHJK
31
- # and check that the page shows the code BCDF-GHJK (it works for 10 minutes; ...)
32
- # Logged in as ada
33
- runlab whoami # ada (agent) at https://runlab.tools
34
+ pip install runlab-client # Python 3.10+, no dependencies
35
+ runlab demo # sign in with GitHub, then a first verdict on simulated numbers
36
+ ```
37
+
38
+ The first time, a browser opens: sign in with GitHub and approve the code your terminal shows.
39
+ Then the demo plans, records 10 seeds per arm, compares and reports, in seconds:
40
+
41
+ ```text
42
+ treatment vs baseline on acc (higher is better): treatment is better
43
+ paired_t: diff +0.01868 [95% CI +0.01371, +0.02365], p=1.37e-05, n=10 (A 10, B 10)
44
+ ...
45
+ quotable: treatment − baseline on acc = +0.01868 (95 % CI [+0.01371, +0.02365]; ...)
46
+ The verdict page: https://runlab.tools/v/...
34
47
  ```
35
48
 
36
- - `runlab login` opens a browser when this machine has a display. On an SSH session or a
37
- headless GPU box, open the printed link on any device (your laptop or phone).
38
- - The page asks "Authorize this terminal?" and shows the code, the address and host name
39
- that asked. Approve only if the code matches the one in your terminal.
40
- - The key is saved to `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected): the folder is
41
- `0700`, the file `0600`. It is a personal API key named `cli <host> <date>` that works
42
- until you log out. `runlab logout` revokes it and deletes the file.
43
- - After that, `RunLab()` needs no settings. Its order: `token=` argument, env `RUNLAB_TOKEN`,
44
- the saved key; `url=` argument, env `RUNLAB_URL`, the URL you logged in to,
45
- `https://runlab.tools`. The saved key is only sent to the URL that issued it.
46
- - In CI or other non-interactive jobs, set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead.
49
+ Next, your own runs: the [Quickstart](#quickstart) below, with your training run in place of
50
+ the simulated one. Questions about signing in: [Signing in](#signing-in).
47
51
 
48
52
  ## Quickstart
49
53
 
50
54
  ```python
51
55
  from runlab_client import RunLab
52
56
 
53
- rl = RunLab() # after `runlab login` (or RUNLAB_TOKEN)
54
- rl.me() # {'name': 'ada', 'role': 'agent', ...}
57
+ rl = RunLab() # the key `runlab demo` / `login` saved
55
58
 
56
59
  # 1. Pre-register, before any seed runs.
57
60
  rl.plan("resnet-aug", "mixup", "baseline", "acc",
@@ -75,9 +78,11 @@ for c in rl.report("resnet-aug")["claims"]:
75
78
  print(c["status"], c["quote"]) # quotable mixup − baseline on acc = ...
76
79
  ```
77
80
 
78
- `train` is your own training run. The numbers above are an illustration. A runnable version
79
- with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), and
80
- [examples/curl.sh](examples/curl.sh) does the same loop over plain HTTP.
81
+ `train` is your training run.
82
+ [examples/quickstart.py](https://github.com/omer-nachman/runlab-python/blob/main/examples/quickstart.py)
83
+ runs as is, with a simulated `train`;
84
+ [examples/curl.sh](https://github.com/omer-nachman/runlab-python/blob/main/examples/curl.sh)
85
+ is the same loop over plain HTTP.
81
86
 
82
87
  ## The loop
83
88
 
@@ -90,20 +95,6 @@ with a simulated `train` is [examples/quickstart.py](examples/quickstart.py), an
90
95
  | Audit | `verdicts(project)`, `verdict(id)` | `GET /api/v1/research/verdicts`, `GET /api/v1/verdicts/{id}` | Shows every recorded verdict, each seed's values, and whether it still holds on today's results. |
91
96
  | Delegate | `start_study(...)`, `wait_study(id)` | `POST /api/v1/research/studies` | Runs the seeds itself under a dollar cap and stops once the evidence decides (see below). |
92
97
 
93
- ### Each user sees only their projects
94
-
95
- On a RunLab where each user has their own data, a project name is yours alone: your `demo`
96
- and a colleague's `demo` are two projects, never merged, deduplicated or compared with each
97
- other. Lists (`verdicts()`, `studies()`, `report("")`) show only your projects. A verdict or
98
- study id that is not yours answers 404, the same as one that does not exist, and an unknown
99
- project's closest match is one of yours. Project names may not contain `~`.
100
-
101
- `a` is the treatment and `b` the baseline. `diff` is the mean of a − b, so it is positive when
102
- a scored higher. Whether higher is better comes from the metric name: lower for names
103
- containing `loss`, `error`, `err`, `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`,
104
- `nll`, `latency`, `fid` or `regret`, higher otherwise. Pass `better="higher"` or
105
- `better="lower"` to override.
106
-
107
98
  ### Results
108
99
 
109
100
  RunLab stores each result as one line:
@@ -120,13 +111,18 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
120
111
  `record()` as is.
121
112
  - If one RESULT line is malformed, or there is none, the call stores nothing and fails with
122
113
  a 400.
123
- - Project names are case-insensitive.
114
+ - Project names are case-insensitive and may not contain `~`.
124
115
  - Params are optional; keep an arm's params the same on every seed. Each distinct set of
125
116
  params counts as one tuned configuration. A compare whose arms differ by more than 1.25x
126
117
  in configurations is refused (`refused_by: equal_budget`).
127
118
 
128
119
  ### Verdicts
129
120
 
121
+ `a` is the treatment, `b` the baseline; `diff` is the mean of a − b, so it is positive when a
122
+ scored higher. Lower is better for metric names containing `loss`, `error`, `err`,
123
+ `perplexity`, `ppl`, `wer`, `cer`, `mse`, `mae`, `rmse`, `nll`, `latency`, `fid` or `regret`,
124
+ higher otherwise; `better="higher"` or `better="lower"` overrides.
125
+
130
126
  | `verdict` | Meaning |
131
127
  |---|---|
132
128
  | `a_better` / `b_better` | p < alpha (default 0.05) and the difference favours that arm. |
@@ -136,16 +132,9 @@ prints them (send its log to `record()`) or when you call the HTTP API directly.
136
132
  | `insufficient` | An arm has fewer than 2 seeds, so there is no test. |
137
133
  | `refused` | A check found that any verdict would mislead. `refused_by` names the check and `why` says what to fix. |
138
134
 
139
- Read these fields next to the verdict:
140
-
141
- - `prereg.status`:
142
- - `ok`;
143
- - `deviation`, with `issues`, e.g. compared before `n_max` seeds, or the plan was registered
144
- after more than 3 seeds had been seen;
145
- - `missing`: no plan;
146
- - `tampered`.
147
- - `warnings`, e.g. `peeking`.
148
- - `report()`, before you quote anything.
135
+ Next to `verdict`, read `prereg.status`: `ok`; `deviation`, with `issues` (e.g. compared
136
+ before `n_max` seeds, or planned after more than 3 seeds were seen); `missing` (no plan); or
137
+ `tampered`. Then read `warnings`, e.g. `peeking`.
149
138
 
150
139
  ### What keeps a verdict honest
151
140
 
@@ -159,8 +148,7 @@ Read these fields next to the verdict:
159
148
 
160
149
  ## Studies: let RunLab run the seeds
161
150
 
162
- RunLab can also run the seeds for you.
163
- It runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
151
+ RunLab runs both arms one seed pair at a time and stops as soon as an always-valid sequential test
164
152
  (mSPRT) decides. It also stops at `n_max`, or at your dollar cap, whichever comes first.
165
153
 
166
154
  ```python
@@ -176,7 +164,6 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
176
164
  placeholders, using values of letters, digits and `_.:+=-` only.
177
165
  - Every check runs before anything starts. A refusal raises `RunLabError` with status 400
178
166
  and costs nothing, e.g. an unknown command, no msprt plan, or a cap that is too high.
179
- - A study never spends more than `max_usd`.
180
167
  - `state` is one of:
181
168
 
182
169
  | `state` | Meaning |
@@ -184,8 +171,8 @@ print(s["state"], s["winner"], s["pairs"], s["usd"]) # e.g. dec
184
171
  | `running` | Still running seeds. |
185
172
  | `decided` | The sequential test stopped. `decision` is `A` or `B` (`winner` names the arm), or `futile` (no detectable difference at this budget) or `equivalent` (`winner` is null). |
186
173
  | `undecided` | `n_max` seeds ran without a decision. |
187
- | `budget` | The compute budget was spent before a decision (your own machine: the next pair would pass `max_usd`). |
188
- | `killed` | Stopped at the dollar cap. |
174
+ | `budget` | Stopped undecided: the next pair would have passed `max_usd`. |
175
+ | `killed` | Still running when the time `max_usd` buys ran out; stopped hard. |
189
176
  | `failed` | The command failed or printed no result; see `detail`. |
190
177
  | `lost` | Stopped reporting long past its deadline. |
191
178
  | `refused` | Refused at the start (that call raised a 400); listed for the record. |
@@ -198,8 +185,6 @@ your dollar cap. Your training command prints `<metric>=<value>`; `{arm}`, `{see
198
185
  params' names are filled in (no shell runs it).
199
186
 
200
187
  ```bash
201
- pip install runlab-client
202
- runlab login
203
188
  runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
204
189
  --max-usd 5 --usd-per-hour 0.8 --param epochs=10 \
205
190
  --cmd "python train.py --arm {arm} --seed {seed} --epochs {epochs}"
@@ -217,6 +202,19 @@ runlab worker --project resnet-aug --a mixup --b baseline --metric acc \
217
202
  - Each run's time is recorded next to its metric, as `wall_s` and `usd` of its arm and seed.
218
203
  - From Python: `s = rl.start_worker_study(...)`, then `rl.run_worker(s["id"], cmd)`.
219
204
 
205
+ ## Signing in
206
+
207
+ `runlab login` signs a machine in; `runlab demo` does it too on first use.
208
+
209
+ | Question | Answer |
210
+ |---|---|
211
+ | Another program called `runlab` first on your PATH? | Run `runlab-client <command>` or `python -m runlab_client <command>` instead. |
212
+ | No browser here (SSH, a headless GPU box)? | Open the printed link on any device: your laptop or phone. |
213
+ | What do I approve? | The page asks "Authorize this terminal?" and shows the code, address and host name that asked. Approve only if the code matches your terminal's. |
214
+ | Where is the key? | `~/.config/runlab/token` (`$XDG_CONFIG_HOME` respected; folder `0700`, file `0600`): a personal API key named `cli <host> <date>`, valid until `runlab logout` revokes it and deletes the file. `runlab whoami` shows who it signs in as. |
215
+ | What does `RunLab()` use? | Token: `token=` argument, env `RUNLAB_TOKEN`, the saved key. URL: `url=` argument, env `RUNLAB_URL`, the URL you logged in to, `https://runlab.tools`. The saved key is only sent to the URL that issued it. |
216
+ | CI or other non-interactive jobs? | Set `RUNLAB_TOKEN` (and `RUNLAB_URL`) instead of logging in. |
217
+
220
218
  ## Errors
221
219
 
222
220
  Every failure raises `RunLabError` with:
@@ -229,19 +227,19 @@ Every failure raises `RunLabError` with:
229
227
  |---|---|
230
228
  | 400 | Bad input or a refused action. The message says what to fix; an unknown arm, metric or project names the closest match. |
231
229
  | 401 | Missing or wrong token. |
232
- | 403 | Your role cannot do this: writes need `agent` or above. |
230
+ | 403 | This key may not do this; the message says why. |
233
231
  | 404 | No such verdict or study. |
234
232
  | 413 | Body over 64 KiB; send results in several calls. |
235
233
 
236
234
  The client retries GET requests 3 times with exponential backoff, on connection errors and on
237
- 429, 502, 503 and 504. It never retries POST requests. Recording the same result line twice is
238
- harmless anyway.
235
+ 429, 502, 503 and 504. It never retries POST requests.
239
236
 
240
237
  ## Client reference
241
238
 
242
239
  `RunLab(url=None, token=None, *, timeout=30, retries=3, backoff_s=0.5)`. Every method returns
243
- the decoded JSON of one operation in [openapi.json](openapi.json). With no arguments it uses
244
- the key `runlab login` saved (see [Get started](#get-started)).
240
+ the decoded JSON of one operation in
241
+ [openapi.json](https://github.com/omer-nachman/runlab-python/blob/main/openapi.json).
242
+ With no arguments it uses the key `runlab login` saved (see [Signing in](#signing-in)).
245
243
 
246
244
  | Method | Operation | Returns |
247
245
  |---|---|---|
@@ -264,16 +262,25 @@ the key `runlab login` saved (see [Get started](#get-started)).
264
262
 
265
263
  | Command | What it does |
266
264
  |---|---|
265
+ | `runlab demo [--project P] [--url URL] [--no-browser]` | Sign in if needed, then plan, record 10 simulated seeds per arm, compare and report, in project `runlab-demo`. |
267
266
  | `runlab login [--url URL] [--no-browser]` | Sign this machine in with GitHub and save a personal key. |
268
267
  | `runlab whoami [--url URL]` | Who the key signs in as, and where. Exit 1 when not logged in. |
269
268
  | `runlab logout` | Revoke the saved key and delete it. If RunLab cannot be reached, the key stays saved and the exit code is 1. |
270
269
  | `runlab worker ...` | Run a study's seed pairs here (see above). |
270
+ | `runlab --version` | The client's version. |
271
+
272
+ Each command also runs as `runlab-client ...` and `python -m runlab_client ...`.
271
273
 
272
274
  ## Coding agents
273
275
 
274
- [AGENTS.md](AGENTS.md) is the short version for Claude Code, Codex and other agents.
275
- [llms.txt](llms.txt) indexes this repo for LLM tools.
276
+ [AGENTS.md](https://github.com/omer-nachman/runlab-python/blob/main/AGENTS.md)
277
+ is the short version for Claude Code, Codex and other agents.
278
+ [llms.txt](https://github.com/omer-nachman/runlab-python/blob/main/llms.txt) indexes this repo for LLM tools.
279
+
280
+ ## Support
281
+
282
+ Questions and bug reports: [GitHub Issues](https://github.com/omer-nachman/runlab-python/issues).
276
283
 
277
284
  ## License
278
285
 
279
- MIT ([LICENSE](LICENSE)).
286
+ MIT ([LICENSE](https://github.com/omer-nachman/runlab-python/blob/main/LICENSE)).
@@ -2,6 +2,7 @@ LICENSE
2
2
  README.md
3
3
  pyproject.toml
4
4
  src/runlab_client/__init__.py
5
+ src/runlab_client/__main__.py
5
6
  src/runlab_client.egg-info/PKG-INFO
6
7
  src/runlab_client.egg-info/SOURCES.txt
7
8
  src/runlab_client.egg-info/dependency_links.txt
@@ -1,2 +1,3 @@
1
1
  [console_scripts]
2
2
  runlab = runlab_client:main
3
+ runlab-client = runlab_client:main
File without changes
File without changes