runledger-cli 0.1.2__tar.gz → 0.1.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- runledger_cli-0.1.3/PKG-INFO +90 -0
- runledger_cli-0.1.3/README.md +81 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/ledger.py +27 -7
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_data_features.py +71 -2
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_integration.py +2 -2
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/pyproject.toml +1 -1
- runledger_cli-0.1.3/runledger_cli.egg-info/PKG-INFO +90 -0
- runledger_cli-0.1.2/PKG-INFO +0 -198
- runledger_cli-0.1.2/README.md +0 -189
- runledger_cli-0.1.2/runledger_cli.egg-info/PKG-INFO +0 -198
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/__init__.py +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/docker-compose.yml +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/guided.py +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/__init__.py +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_ledger.py +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_real_results.py +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/SOURCES.txt +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/dependency_links.txt +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/entry_points.txt +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/requires.txt +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/top_level.txt +0 -0
- {runledger_cli-0.1.2 → runledger_cli-0.1.3}/setup.cfg +0 -0
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: runledger-cli
|
|
3
|
+
Version: 0.1.3
|
|
4
|
+
Summary: Zero‑instrumentation experiment registry CLI
|
|
5
|
+
Requires-Python: >=3.10
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: requests
|
|
8
|
+
Requires-Dist: rich
|
|
9
|
+
|
|
10
|
+
Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# RunLedger
|
|
15
|
+
|
|
16
|
+
**A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
|
|
17
|
+
|
|
18
|
+
Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
|
|
19
|
+
|
|
20
|
+
Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
|
|
21
|
+
|
|
22
|
+
## Install
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
pip install runledger-cli
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
|
|
29
|
+
|
|
30
|
+
### Troubleshooting: `runledger` command not found (Windows)
|
|
31
|
+
|
|
32
|
+
On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
|
|
33
|
+
|
|
34
|
+
1. **Add the Scripts folder to your PATH permanently** (recommended):
|
|
35
|
+
- Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
|
|
36
|
+
- Navigate to that folder, then up one level to the `Scripts` directory.
|
|
37
|
+
- Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
|
|
38
|
+
- Restart your terminal.
|
|
39
|
+
|
|
40
|
+
2. **Use the Python module fallback** (works immediately, no PATH changes):
|
|
41
|
+
```bash
|
|
42
|
+
python -m cli.ledger
|
|
43
|
+
```
|
|
44
|
+
All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
|
|
45
|
+
|
|
46
|
+
Once the PATH is configured, `runledger` will work as shown in the examples below.
|
|
47
|
+
|
|
48
|
+
<details>
|
|
49
|
+
<summary>Installing from source</summary>
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
git clone https://github.com/yourusername/run-ledger.git
|
|
53
|
+
cd run-ledger
|
|
54
|
+
docker compose up -d
|
|
55
|
+
pip install -e .
|
|
56
|
+
```
|
|
57
|
+
</details>
|
|
58
|
+
|
|
59
|
+
## Quick start
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
runledger scan ~/experiments --batch my-sweep
|
|
63
|
+
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
64
|
+
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
|
|
68
|
+
|
|
69
|
+
## What you get
|
|
70
|
+
|
|
71
|
+
- **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
|
|
72
|
+
- **Compound AND/OR** across multiple conditions in one query.
|
|
73
|
+
- **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
|
|
74
|
+
- **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
|
|
75
|
+
- **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
|
|
76
|
+
- **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
|
|
77
|
+
- **Self-hosted, local-first** — your data never leaves your machine.
|
|
78
|
+
|
|
79
|
+
## Metric paths
|
|
80
|
+
|
|
81
|
+
Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
|
|
82
|
+
|
|
83
|
+
## Docs
|
|
84
|
+
|
|
85
|
+
- [API reference](docs/api.md)
|
|
86
|
+
- [Usage examples](docs/examples.md)
|
|
87
|
+
|
|
88
|
+
## License
|
|
89
|
+
|
|
90
|
+
MIT
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
|
|
2
|
+
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# RunLedger
|
|
6
|
+
|
|
7
|
+
**A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
|
|
8
|
+
|
|
9
|
+
Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
|
|
10
|
+
|
|
11
|
+
Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
|
|
12
|
+
|
|
13
|
+
## Install
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
pip install runledger-cli
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
|
|
20
|
+
|
|
21
|
+
### Troubleshooting: `runledger` command not found (Windows)
|
|
22
|
+
|
|
23
|
+
On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
|
|
24
|
+
|
|
25
|
+
1. **Add the Scripts folder to your PATH permanently** (recommended):
|
|
26
|
+
- Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
|
|
27
|
+
- Navigate to that folder, then up one level to the `Scripts` directory.
|
|
28
|
+
- Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
|
|
29
|
+
- Restart your terminal.
|
|
30
|
+
|
|
31
|
+
2. **Use the Python module fallback** (works immediately, no PATH changes):
|
|
32
|
+
```bash
|
|
33
|
+
python -m cli.ledger
|
|
34
|
+
```
|
|
35
|
+
All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
|
|
36
|
+
|
|
37
|
+
Once the PATH is configured, `runledger` will work as shown in the examples below.
|
|
38
|
+
|
|
39
|
+
<details>
|
|
40
|
+
<summary>Installing from source</summary>
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
git clone https://github.com/yourusername/run-ledger.git
|
|
44
|
+
cd run-ledger
|
|
45
|
+
docker compose up -d
|
|
46
|
+
pip install -e .
|
|
47
|
+
```
|
|
48
|
+
</details>
|
|
49
|
+
|
|
50
|
+
## Quick start
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
runledger scan ~/experiments --batch my-sweep
|
|
54
|
+
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
55
|
+
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
|
|
59
|
+
|
|
60
|
+
## What you get
|
|
61
|
+
|
|
62
|
+
- **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
|
|
63
|
+
- **Compound AND/OR** across multiple conditions in one query.
|
|
64
|
+
- **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
|
|
65
|
+
- **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
|
|
66
|
+
- **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
|
|
67
|
+
- **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
|
|
68
|
+
- **Self-hosted, local-first** — your data never leaves your machine.
|
|
69
|
+
|
|
70
|
+
## Metric paths
|
|
71
|
+
|
|
72
|
+
Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
|
|
73
|
+
|
|
74
|
+
## Docs
|
|
75
|
+
|
|
76
|
+
- [API reference](docs/api.md)
|
|
77
|
+
- [Usage examples](docs/examples.md)
|
|
78
|
+
|
|
79
|
+
## License
|
|
80
|
+
|
|
81
|
+
MIT
|
|
@@ -333,11 +333,13 @@ def navigate_json_pointer(doc, pointer: str):
|
|
|
333
333
|
|
|
334
334
|
def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
|
|
335
335
|
payload = run.get("payload", {})
|
|
336
|
+
# Use experiment name, then source file, then run id
|
|
336
337
|
if isinstance(payload, dict):
|
|
337
338
|
source_file = payload.get("_source", {}).get("file") or payload.get("experiment") or f"run-{run['id']}"
|
|
338
339
|
else:
|
|
339
340
|
source_file = f"run-{run['id']}"
|
|
340
341
|
|
|
342
|
+
# Show matched pointers if available
|
|
341
343
|
matched = run.get("matched")
|
|
342
344
|
if matched:
|
|
343
345
|
console.print(" Matches:")
|
|
@@ -359,6 +361,7 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
|
|
|
359
361
|
console.print(Panel(pretty, title=f"Run {run['id']} · {source_file}"))
|
|
360
362
|
return
|
|
361
363
|
|
|
364
|
+
# Strip "[]" so JSON Pointer navigation works
|
|
362
365
|
parts = [seg.replace("[]", "") for seg in dot_path.split(".")]
|
|
363
366
|
max_block = len(parts) - 1
|
|
364
367
|
|
|
@@ -372,6 +375,7 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
|
|
|
372
375
|
|
|
373
376
|
for block in sorted(levels, reverse=True):
|
|
374
377
|
if block == 0 and matched:
|
|
378
|
+
# For block=0, show each matched snippet individually
|
|
375
379
|
for i, m in enumerate(matched):
|
|
376
380
|
snippet = m.get("snippet")
|
|
377
381
|
try:
|
|
@@ -381,9 +385,19 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
|
|
|
381
385
|
title = f"Run {run['id']} · {source_file} | metric: {dot_path} | depth: 0 (match {i+1}/{len(matched)})"
|
|
382
386
|
console.print(Panel(pretty, title=title))
|
|
383
387
|
else:
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
388
|
+
if block > 0 and matched:
|
|
389
|
+
# Use the first match's concrete pointer for precise navigation
|
|
390
|
+
# Example: "clients[1].sweep[0].fin_asr" → "/clients/1/sweep"
|
|
391
|
+
concrete = matched[0]['pointer']
|
|
392
|
+
segments = concrete.replace('[', '.').replace(']', '').split('.')
|
|
393
|
+
ancestor_segments = segments[:len(segments) - (block + 1)]
|
|
394
|
+
pointer = "/" + "/".join(ancestor_segments) if ancestor_segments else ""
|
|
395
|
+
node = navigate_json_pointer(payload, pointer) if pointer else payload
|
|
396
|
+
else:
|
|
397
|
+
ancestor_parts = parts[:len(parts) - (block + 1)]
|
|
398
|
+
pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
|
|
399
|
+
node = navigate_json_pointer(payload, pointer) if pointer else payload
|
|
400
|
+
|
|
387
401
|
depth_label = ""
|
|
388
402
|
if block == max_block:
|
|
389
403
|
depth_label = " (full run)"
|
|
@@ -394,7 +408,6 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
|
|
|
394
408
|
except Exception:
|
|
395
409
|
pretty = str(node)
|
|
396
410
|
console.print(Panel(pretty, title=f"Run {run['id']} · {source_file} | metric: {dot_path} | depth: {block}{depth_label}"))
|
|
397
|
-
|
|
398
411
|
# ------------------------------------------------------------
|
|
399
412
|
# Export helpers
|
|
400
413
|
# ------------------------------------------------------------
|
|
@@ -444,9 +457,16 @@ def build_export_blocks(content, dot_path, block_levels, include_pointers=True):
|
|
|
444
457
|
"value": m.get("value")
|
|
445
458
|
})
|
|
446
459
|
else:
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
460
|
+
if block > 0 and matched:
|
|
461
|
+
concrete = matched[0]['pointer']
|
|
462
|
+
segments = concrete.replace('[', '.').replace(']', '').split('.')
|
|
463
|
+
ancestor_segments = segments[:len(segments) - (block + 1)]
|
|
464
|
+
pointer = "/" + "/".join(ancestor_segments) if ancestor_segments else ""
|
|
465
|
+
node = navigate_json_pointer(payload, pointer) if pointer else payload
|
|
466
|
+
else:
|
|
467
|
+
ancestor_parts = parts[:len(parts) - (block + 1)]
|
|
468
|
+
pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
|
|
469
|
+
node = navigate_json_pointer(payload, pointer) if pointer else payload
|
|
450
470
|
blocks.append({
|
|
451
471
|
"id": run["id"],
|
|
452
472
|
"sourceFile": source_file,
|
|
@@ -57,7 +57,7 @@ def setup_data():
|
|
|
57
57
|
# Ingest sample runs into the fresh batch
|
|
58
58
|
result = run_cli("scan", str(SAMPLE_FOLDER), "--batch", BATCH_NAME)
|
|
59
59
|
assert result.returncode == 0, result.stderr
|
|
60
|
-
assert "Successfully ingested
|
|
60
|
+
assert "Successfully ingested 10 run(s)" in result.stdout
|
|
61
61
|
|
|
62
62
|
# ------------------------------------------------------------
|
|
63
63
|
# v1 – Basic metric block search
|
|
@@ -251,4 +251,73 @@ def test_jsonl_ingestion(setup_data):
|
|
|
251
251
|
assert payload.get(k) == v, f"Mismatch at line {idx}: {k} {v} != {payload.get(k)}"
|
|
252
252
|
|
|
253
253
|
# Clean up the temporary batch
|
|
254
|
-
requests.delete(f"{API_BASE}/batch/{batch}") # if this endpoint exists, otherwise manual cleanup
|
|
254
|
+
requests.delete(f"{API_BASE}/batch/{batch}") # if this endpoint exists, otherwise manual cleanup
|
|
255
|
+
|
|
256
|
+
def test_nested_array_block_depth():
|
|
257
|
+
"""Verify block‑depth on nested arrays using direct API calls."""
|
|
258
|
+
import uuid
|
|
259
|
+
import requests
|
|
260
|
+
|
|
261
|
+
batch = f"nest-{uuid.uuid4().hex[:6]}"
|
|
262
|
+
|
|
263
|
+
# 1. Ingest the nested file directly via the API
|
|
264
|
+
payload = {
|
|
265
|
+
"experiment": "nested_arrays_block_test",
|
|
266
|
+
"clients": [
|
|
267
|
+
{
|
|
268
|
+
"client_name": "clean_client",
|
|
269
|
+
"sweep": [
|
|
270
|
+
{"w_base": 0.5, "fin_asr": 0.12, "sst2_asr": 0.07},
|
|
271
|
+
{"w_base": 0.75, "fin_asr": 0.09, "sst2_asr": 0.06}
|
|
272
|
+
]
|
|
273
|
+
},
|
|
274
|
+
{
|
|
275
|
+
"client_name": "poisoned_client",
|
|
276
|
+
"sweep": [
|
|
277
|
+
{"w_base": 0.5, "fin_asr": 0.45, "sst2_asr": 0.88},
|
|
278
|
+
{"w_base": 1.0, "fin_asr": 0.28, "sst2_asr": 0.72}
|
|
279
|
+
]
|
|
280
|
+
}
|
|
281
|
+
]
|
|
282
|
+
}
|
|
283
|
+
resp = requests.post(API_BASE, json={"payload": payload, "batch": batch})
|
|
284
|
+
assert resp.status_code == 201, f"ingest failed: {resp.text}"
|
|
285
|
+
|
|
286
|
+
# 2. Search for fin_asr < 0.3 with block=0
|
|
287
|
+
params = {"metric": "clients[].sweep[].fin_asr", "op": "lt", "value": "0.3",
|
|
288
|
+
"batch": batch, "block": "0", "size": "1", "pointers": "true"}
|
|
289
|
+
resp = requests.get(API_BASE, params=params)
|
|
290
|
+
assert resp.status_code == 200, resp.text
|
|
291
|
+
data = resp.json()
|
|
292
|
+
assert data["totalElements"] == 1
|
|
293
|
+
run = data["content"][0]
|
|
294
|
+
|
|
295
|
+
# 3. Verify matched pointers exist
|
|
296
|
+
matched = run.get("matched")
|
|
297
|
+
assert matched is not None, "matched pointers missing"
|
|
298
|
+
assert len(matched) == 3, f"expected 3 matches, got {len(matched)}"
|
|
299
|
+
|
|
300
|
+
# 4. block=1 should return the sweep array (a list)
|
|
301
|
+
params["block"] = "1"
|
|
302
|
+
resp = requests.get(API_BASE, params=params)
|
|
303
|
+
assert resp.status_code == 200, resp.text
|
|
304
|
+
data1 = resp.json()
|
|
305
|
+
block1_payload = data1["content"][0]["payload"]
|
|
306
|
+
assert isinstance(block1_payload, list), f"block=1 payload should be a list, got {type(block1_payload)}"
|
|
307
|
+
assert any("fin_asr" in json.dumps(elem) for elem in block1_payload)
|
|
308
|
+
|
|
309
|
+
# 5. block=2 should return the client object
|
|
310
|
+
params["block"] = "2"
|
|
311
|
+
resp = requests.get(API_BASE, params=params)
|
|
312
|
+
assert resp.status_code == 200, resp.text
|
|
313
|
+
data2 = resp.json()
|
|
314
|
+
block2_payload = data2["content"][0]["payload"]
|
|
315
|
+
assert isinstance(block2_payload, dict) and block2_payload.get("client_name") == "clean_client"
|
|
316
|
+
|
|
317
|
+
# 6. block=3 should return the clients array
|
|
318
|
+
params["block"] = "3"
|
|
319
|
+
resp = requests.get(API_BASE, params=params)
|
|
320
|
+
assert resp.status_code == 200, resp.text
|
|
321
|
+
data3 = resp.json()
|
|
322
|
+
block3_payload = data3["content"][0]["payload"]
|
|
323
|
+
assert isinstance(block3_payload, list) and len(block3_payload) > 0
|
|
@@ -10,8 +10,8 @@ import pytest
|
|
|
10
10
|
import requests
|
|
11
11
|
|
|
12
12
|
PROJECT_ROOT = Path(__file__).parent.parent.parent
|
|
13
|
-
COMPOSE_FILE = PROJECT_ROOT / "docker-compose.yml"
|
|
14
|
-
HEALTH_URL = "http://localhost:
|
|
13
|
+
COMPOSE_FILE = PROJECT_ROOT / "docker-compose.test.yml"
|
|
14
|
+
HEALTH_URL = "http://localhost:8081/actuator/health"
|
|
15
15
|
API_BASE = "http://localhost:8080/api/runs"
|
|
16
16
|
|
|
17
17
|
def wait_for_health(timeout=120):
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: runledger-cli
|
|
3
|
+
Version: 0.1.3
|
|
4
|
+
Summary: Zero‑instrumentation experiment registry CLI
|
|
5
|
+
Requires-Python: >=3.10
|
|
6
|
+
Description-Content-Type: text/markdown
|
|
7
|
+
Requires-Dist: requests
|
|
8
|
+
Requires-Dist: rich
|
|
9
|
+
|
|
10
|
+
Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
# RunLedger
|
|
15
|
+
|
|
16
|
+
**A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
|
|
17
|
+
|
|
18
|
+
Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
|
|
19
|
+
|
|
20
|
+
Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
|
|
21
|
+
|
|
22
|
+
## Install
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
pip install runledger-cli
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
|
|
29
|
+
|
|
30
|
+
### Troubleshooting: `runledger` command not found (Windows)
|
|
31
|
+
|
|
32
|
+
On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
|
|
33
|
+
|
|
34
|
+
1. **Add the Scripts folder to your PATH permanently** (recommended):
|
|
35
|
+
- Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
|
|
36
|
+
- Navigate to that folder, then up one level to the `Scripts` directory.
|
|
37
|
+
- Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
|
|
38
|
+
- Restart your terminal.
|
|
39
|
+
|
|
40
|
+
2. **Use the Python module fallback** (works immediately, no PATH changes):
|
|
41
|
+
```bash
|
|
42
|
+
python -m cli.ledger
|
|
43
|
+
```
|
|
44
|
+
All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
|
|
45
|
+
|
|
46
|
+
Once the PATH is configured, `runledger` will work as shown in the examples below.
|
|
47
|
+
|
|
48
|
+
<details>
|
|
49
|
+
<summary>Installing from source</summary>
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
git clone https://github.com/yourusername/run-ledger.git
|
|
53
|
+
cd run-ledger
|
|
54
|
+
docker compose up -d
|
|
55
|
+
pip install -e .
|
|
56
|
+
```
|
|
57
|
+
</details>
|
|
58
|
+
|
|
59
|
+
## Quick start
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
runledger scan ~/experiments --batch my-sweep
|
|
63
|
+
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
64
|
+
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
|
|
68
|
+
|
|
69
|
+
## What you get
|
|
70
|
+
|
|
71
|
+
- **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
|
|
72
|
+
- **Compound AND/OR** across multiple conditions in one query.
|
|
73
|
+
- **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
|
|
74
|
+
- **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
|
|
75
|
+
- **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
|
|
76
|
+
- **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
|
|
77
|
+
- **Self-hosted, local-first** — your data never leaves your machine.
|
|
78
|
+
|
|
79
|
+
## Metric paths
|
|
80
|
+
|
|
81
|
+
Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
|
|
82
|
+
|
|
83
|
+
## Docs
|
|
84
|
+
|
|
85
|
+
- [API reference](docs/api.md)
|
|
86
|
+
- [Usage examples](docs/examples.md)
|
|
87
|
+
|
|
88
|
+
## License
|
|
89
|
+
|
|
90
|
+
MIT
|
runledger_cli-0.1.2/PKG-INFO
DELETED
|
@@ -1,198 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: runledger-cli
|
|
3
|
-
Version: 0.1.2
|
|
4
|
-
Summary: Zero‑instrumentation experiment registry CLI
|
|
5
|
-
Requires-Python: >=3.10
|
|
6
|
-
Description-Content-Type: text/markdown
|
|
7
|
-
Requires-Dist: requests
|
|
8
|
-
Requires-Dist: rich
|
|
9
|
-
|
|
10
|
-
# RunLedger
|
|
11
|
-
|
|
12
|
-
**Zero-instrumentation experiment registry and search engine.**
|
|
13
|
-
Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
|
|
14
|
-
|
|
15
|
-
## Quick Start
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
# 1. Start the backend
|
|
19
|
-
docker compose up -d
|
|
20
|
-
|
|
21
|
-
# 2. Ingest your experiment files
|
|
22
|
-
runledger scan ~/experiments --batch my-sweep
|
|
23
|
-
|
|
24
|
-
# 3. Search
|
|
25
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
26
|
-
|
|
27
|
-
# 4. Aggregate
|
|
28
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
New to the tool? Try the guided wizard:
|
|
32
|
-
|
|
33
|
-
```bash
|
|
34
|
-
runledger guided
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
## Key Features
|
|
38
|
-
|
|
39
|
-
### 🔍 Search any JSON field, no schema required
|
|
40
|
-
- Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
|
|
41
|
-
- Full-text phrase search with stemming.
|
|
42
|
-
- Fuzzy trigram search for typos.
|
|
43
|
-
- Compound AND/OR queries: combine multiple conditions.
|
|
44
|
-
- Batch isolation: group runs by folder and filter within a batch.
|
|
45
|
-
|
|
46
|
-
### 📊 Aggregates & group-by
|
|
47
|
-
Answer research questions directly:
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
|
|
54
|
-
|
|
55
|
-
### 🎯 Exact match pointers (“JSON line numbers”)
|
|
56
|
-
For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
|
|
57
|
-
|
|
58
|
-
```text
|
|
59
|
-
Matches:
|
|
60
|
-
• results[0].sst2_cacc = 0.924312
|
|
61
|
-
• results[3].sst2_cacc = 0.933486
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
Works for scalar, array, and compound conditions.
|
|
65
|
-
|
|
66
|
-
### 🔬 Block-depth control
|
|
67
|
-
Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
|
|
68
|
-
|
|
69
|
-
### 📁 JSONL ingestion
|
|
70
|
-
Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
|
|
71
|
-
|
|
72
|
-
### 🔎 Run diff
|
|
73
|
-
Compare any two runs side-by-side:
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
runledger diff 12 15
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
Or see every version of a re-scanned run with `--history`.
|
|
80
|
-
|
|
81
|
-
### 💾 Saved searches
|
|
82
|
-
Save frequent queries and re-run them with a single command:
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
86
|
-
runledger search --saved best-accuracy
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
### 🧭 Interactive & guided modes
|
|
90
|
-
- Filter-as-you-type interactive search (`runledger interactive`).
|
|
91
|
-
- Step-by-step wizard for first-time users (`runledger guided`).
|
|
92
|
-
|
|
93
|
-
### 🔒 Self-hosted, local-first
|
|
94
|
-
Runs in Docker with a single `docker compose up -d`.
|
|
95
|
-
All data stays on your machine. No internet, no accounts, no vendor lock-in.
|
|
96
|
-
|
|
97
|
-
## Installation
|
|
98
|
-
|
|
99
|
-
### From PyPI (CLI only)
|
|
100
|
-
|
|
101
|
-
```bash
|
|
102
|
-
pip install runledger-cli
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
### From source
|
|
106
|
-
|
|
107
|
-
```bash
|
|
108
|
-
git clone https://github.com/yourusername/run-ledger.git
|
|
109
|
-
cd run-ledger
|
|
110
|
-
docker compose up -d # starts PostgreSQL + Spring Boot
|
|
111
|
-
pip install -e . # installs the CLI
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
## Quick Tour
|
|
115
|
-
|
|
116
|
-
### Scan & Discover
|
|
117
|
-
|
|
118
|
-
```bash
|
|
119
|
-
runledger scan experiments/ --batch my-sweep
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
|
|
123
|
-
|
|
124
|
-
### Search
|
|
125
|
-
|
|
126
|
-
```bash
|
|
127
|
-
# Single condition
|
|
128
|
-
runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
|
|
129
|
-
|
|
130
|
-
# Compound AND
|
|
131
|
-
runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
|
|
132
|
-
|
|
133
|
-
# Full-text
|
|
134
|
-
runledger search --q "weighted averaging"
|
|
135
|
-
|
|
136
|
-
# Fuzzy
|
|
137
|
-
runledger search --q "weighted avaraging" --fuzzy
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
### Aggregate
|
|
141
|
-
|
|
142
|
-
```bash
|
|
143
|
-
runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
### Diff
|
|
147
|
-
|
|
148
|
-
```bash
|
|
149
|
-
runledger diff 12 15
|
|
150
|
-
runledger diff --history 12 --batch my-sweep
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
### Export
|
|
154
|
-
|
|
155
|
-
```bash
|
|
156
|
-
runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
### Saved searches
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
163
|
-
runledger search --saved best-accuracy
|
|
164
|
-
runledger saved list
|
|
165
|
-
runledger saved delete best-accuracy
|
|
166
|
-
```
|
|
167
|
-
|
|
168
|
-
## Metrics-Path Convention
|
|
169
|
-
|
|
170
|
-
RunLedger uses **dot-separated paths** to address any value in a JSON payload.
|
|
171
|
-
|
|
172
|
-
| Example | Meaning |
|
|
173
|
-
| -------- | ------- |
|
|
174
|
-
| `accuracy` | Top-level field |
|
|
175
|
-
| `metrics.loss` | Nested field |
|
|
176
|
-
| `results[].sst2_cacc` | Field inside each element of an array |
|
|
177
|
-
| `phases[].metrics.loss` | Nested field inside array elements |
|
|
178
|
-
|
|
179
|
-
**Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
|
|
180
|
-
|
|
181
|
-
Discover available metrics with:
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
runledger metrics --batch my-sweep
|
|
185
|
-
runledger metrics --batch my-sweep --verbose # shows full paths
|
|
186
|
-
```
|
|
187
|
-
|
|
188
|
-
## API
|
|
189
|
-
|
|
190
|
-
See [`docs/api.md`](docs/api.md) for the complete REST API reference.
|
|
191
|
-
|
|
192
|
-
## Usage Examples
|
|
193
|
-
|
|
194
|
-
See [`docs/examples.md`](docs/examples.md) for real-world workflows.
|
|
195
|
-
|
|
196
|
-
## License
|
|
197
|
-
|
|
198
|
-
MIT
|
runledger_cli-0.1.2/README.md
DELETED
|
@@ -1,189 +0,0 @@
|
|
|
1
|
-
# RunLedger
|
|
2
|
-
|
|
3
|
-
**Zero-instrumentation experiment registry and search engine.**
|
|
4
|
-
Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
|
|
5
|
-
|
|
6
|
-
## Quick Start
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
# 1. Start the backend
|
|
10
|
-
docker compose up -d
|
|
11
|
-
|
|
12
|
-
# 2. Ingest your experiment files
|
|
13
|
-
runledger scan ~/experiments --batch my-sweep
|
|
14
|
-
|
|
15
|
-
# 3. Search
|
|
16
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
17
|
-
|
|
18
|
-
# 4. Aggregate
|
|
19
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
New to the tool? Try the guided wizard:
|
|
23
|
-
|
|
24
|
-
```bash
|
|
25
|
-
runledger guided
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
## Key Features
|
|
29
|
-
|
|
30
|
-
### 🔍 Search any JSON field, no schema required
|
|
31
|
-
- Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
|
|
32
|
-
- Full-text phrase search with stemming.
|
|
33
|
-
- Fuzzy trigram search for typos.
|
|
34
|
-
- Compound AND/OR queries: combine multiple conditions.
|
|
35
|
-
- Batch isolation: group runs by folder and filter within a batch.
|
|
36
|
-
|
|
37
|
-
### 📊 Aggregates & group-by
|
|
38
|
-
Answer research questions directly:
|
|
39
|
-
|
|
40
|
-
```bash
|
|
41
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
|
|
45
|
-
|
|
46
|
-
### 🎯 Exact match pointers (“JSON line numbers”)
|
|
47
|
-
For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
|
|
48
|
-
|
|
49
|
-
```text
|
|
50
|
-
Matches:
|
|
51
|
-
• results[0].sst2_cacc = 0.924312
|
|
52
|
-
• results[3].sst2_cacc = 0.933486
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
Works for scalar, array, and compound conditions.
|
|
56
|
-
|
|
57
|
-
### 🔬 Block-depth control
|
|
58
|
-
Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
|
|
59
|
-
|
|
60
|
-
### 📁 JSONL ingestion
|
|
61
|
-
Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
|
|
62
|
-
|
|
63
|
-
### 🔎 Run diff
|
|
64
|
-
Compare any two runs side-by-side:
|
|
65
|
-
|
|
66
|
-
```bash
|
|
67
|
-
runledger diff 12 15
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
Or see every version of a re-scanned run with `--history`.
|
|
71
|
-
|
|
72
|
-
### 💾 Saved searches
|
|
73
|
-
Save frequent queries and re-run them with a single command:
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
77
|
-
runledger search --saved best-accuracy
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
### 🧭 Interactive & guided modes
|
|
81
|
-
- Filter-as-you-type interactive search (`runledger interactive`).
|
|
82
|
-
- Step-by-step wizard for first-time users (`runledger guided`).
|
|
83
|
-
|
|
84
|
-
### 🔒 Self-hosted, local-first
|
|
85
|
-
Runs in Docker with a single `docker compose up -d`.
|
|
86
|
-
All data stays on your machine. No internet, no accounts, no vendor lock-in.
|
|
87
|
-
|
|
88
|
-
## Installation
|
|
89
|
-
|
|
90
|
-
### From PyPI (CLI only)
|
|
91
|
-
|
|
92
|
-
```bash
|
|
93
|
-
pip install runledger-cli
|
|
94
|
-
```
|
|
95
|
-
|
|
96
|
-
### From source
|
|
97
|
-
|
|
98
|
-
```bash
|
|
99
|
-
git clone https://github.com/yourusername/run-ledger.git
|
|
100
|
-
cd run-ledger
|
|
101
|
-
docker compose up -d # starts PostgreSQL + Spring Boot
|
|
102
|
-
pip install -e . # installs the CLI
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
## Quick Tour
|
|
106
|
-
|
|
107
|
-
### Scan & Discover
|
|
108
|
-
|
|
109
|
-
```bash
|
|
110
|
-
runledger scan experiments/ --batch my-sweep
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
|
|
114
|
-
|
|
115
|
-
### Search
|
|
116
|
-
|
|
117
|
-
```bash
|
|
118
|
-
# Single condition
|
|
119
|
-
runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
|
|
120
|
-
|
|
121
|
-
# Compound AND
|
|
122
|
-
runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
|
|
123
|
-
|
|
124
|
-
# Full-text
|
|
125
|
-
runledger search --q "weighted averaging"
|
|
126
|
-
|
|
127
|
-
# Fuzzy
|
|
128
|
-
runledger search --q "weighted avaraging" --fuzzy
|
|
129
|
-
```
|
|
130
|
-
|
|
131
|
-
### Aggregate
|
|
132
|
-
|
|
133
|
-
```bash
|
|
134
|
-
runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
### Diff
|
|
138
|
-
|
|
139
|
-
```bash
|
|
140
|
-
runledger diff 12 15
|
|
141
|
-
runledger diff --history 12 --batch my-sweep
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
### Export
|
|
145
|
-
|
|
146
|
-
```bash
|
|
147
|
-
runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
|
|
148
|
-
```
|
|
149
|
-
|
|
150
|
-
### Saved searches
|
|
151
|
-
|
|
152
|
-
```bash
|
|
153
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
154
|
-
runledger search --saved best-accuracy
|
|
155
|
-
runledger saved list
|
|
156
|
-
runledger saved delete best-accuracy
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
## Metrics-Path Convention
|
|
160
|
-
|
|
161
|
-
RunLedger uses **dot-separated paths** to address any value in a JSON payload.
|
|
162
|
-
|
|
163
|
-
| Example | Meaning |
|
|
164
|
-
| -------- | ------- |
|
|
165
|
-
| `accuracy` | Top-level field |
|
|
166
|
-
| `metrics.loss` | Nested field |
|
|
167
|
-
| `results[].sst2_cacc` | Field inside each element of an array |
|
|
168
|
-
| `phases[].metrics.loss` | Nested field inside array elements |
|
|
169
|
-
|
|
170
|
-
**Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
|
|
171
|
-
|
|
172
|
-
Discover available metrics with:
|
|
173
|
-
|
|
174
|
-
```bash
|
|
175
|
-
runledger metrics --batch my-sweep
|
|
176
|
-
runledger metrics --batch my-sweep --verbose # shows full paths
|
|
177
|
-
```
|
|
178
|
-
|
|
179
|
-
## API
|
|
180
|
-
|
|
181
|
-
See [`docs/api.md`](docs/api.md) for the complete REST API reference.
|
|
182
|
-
|
|
183
|
-
## Usage Examples
|
|
184
|
-
|
|
185
|
-
See [`docs/examples.md`](docs/examples.md) for real-world workflows.
|
|
186
|
-
|
|
187
|
-
## License
|
|
188
|
-
|
|
189
|
-
MIT
|
|
@@ -1,198 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: runledger-cli
|
|
3
|
-
Version: 0.1.2
|
|
4
|
-
Summary: Zero‑instrumentation experiment registry CLI
|
|
5
|
-
Requires-Python: >=3.10
|
|
6
|
-
Description-Content-Type: text/markdown
|
|
7
|
-
Requires-Dist: requests
|
|
8
|
-
Requires-Dist: rich
|
|
9
|
-
|
|
10
|
-
# RunLedger
|
|
11
|
-
|
|
12
|
-
**Zero-instrumentation experiment registry and search engine.**
|
|
13
|
-
Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
|
|
14
|
-
|
|
15
|
-
## Quick Start
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
# 1. Start the backend
|
|
19
|
-
docker compose up -d
|
|
20
|
-
|
|
21
|
-
# 2. Ingest your experiment files
|
|
22
|
-
runledger scan ~/experiments --batch my-sweep
|
|
23
|
-
|
|
24
|
-
# 3. Search
|
|
25
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
|
|
26
|
-
|
|
27
|
-
# 4. Aggregate
|
|
28
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
New to the tool? Try the guided wizard:
|
|
32
|
-
|
|
33
|
-
```bash
|
|
34
|
-
runledger guided
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
## Key Features
|
|
38
|
-
|
|
39
|
-
### 🔍 Search any JSON field, no schema required
|
|
40
|
-
- Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
|
|
41
|
-
- Full-text phrase search with stemming.
|
|
42
|
-
- Fuzzy trigram search for typos.
|
|
43
|
-
- Compound AND/OR queries: combine multiple conditions.
|
|
44
|
-
- Batch isolation: group runs by folder and filter within a batch.
|
|
45
|
-
|
|
46
|
-
### 📊 Aggregates & group-by
|
|
47
|
-
Answer research questions directly:
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
|
|
54
|
-
|
|
55
|
-
### 🎯 Exact match pointers (“JSON line numbers”)
|
|
56
|
-
For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
|
|
57
|
-
|
|
58
|
-
```text
|
|
59
|
-
Matches:
|
|
60
|
-
• results[0].sst2_cacc = 0.924312
|
|
61
|
-
• results[3].sst2_cacc = 0.933486
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
Works for scalar, array, and compound conditions.
|
|
65
|
-
|
|
66
|
-
### 🔬 Block-depth control
|
|
67
|
-
Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
|
|
68
|
-
|
|
69
|
-
### 📁 JSONL ingestion
|
|
70
|
-
Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
|
|
71
|
-
|
|
72
|
-
### 🔎 Run diff
|
|
73
|
-
Compare any two runs side-by-side:
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
runledger diff 12 15
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
Or see every version of a re-scanned run with `--history`.
|
|
80
|
-
|
|
81
|
-
### 💾 Saved searches
|
|
82
|
-
Save frequent queries and re-run them with a single command:
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
86
|
-
runledger search --saved best-accuracy
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
### 🧭 Interactive & guided modes
|
|
90
|
-
- Filter-as-you-type interactive search (`runledger interactive`).
|
|
91
|
-
- Step-by-step wizard for first-time users (`runledger guided`).
|
|
92
|
-
|
|
93
|
-
### 🔒 Self-hosted, local-first
|
|
94
|
-
Runs in Docker with a single `docker compose up -d`.
|
|
95
|
-
All data stays on your machine. No internet, no accounts, no vendor lock-in.
|
|
96
|
-
|
|
97
|
-
## Installation
|
|
98
|
-
|
|
99
|
-
### From PyPI (CLI only)
|
|
100
|
-
|
|
101
|
-
```bash
|
|
102
|
-
pip install runledger-cli
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
### From source
|
|
106
|
-
|
|
107
|
-
```bash
|
|
108
|
-
git clone https://github.com/yourusername/run-ledger.git
|
|
109
|
-
cd run-ledger
|
|
110
|
-
docker compose up -d # starts PostgreSQL + Spring Boot
|
|
111
|
-
pip install -e . # installs the CLI
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
## Quick Tour
|
|
115
|
-
|
|
116
|
-
### Scan & Discover
|
|
117
|
-
|
|
118
|
-
```bash
|
|
119
|
-
runledger scan experiments/ --batch my-sweep
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
|
|
123
|
-
|
|
124
|
-
### Search
|
|
125
|
-
|
|
126
|
-
```bash
|
|
127
|
-
# Single condition
|
|
128
|
-
runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
|
|
129
|
-
|
|
130
|
-
# Compound AND
|
|
131
|
-
runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
|
|
132
|
-
|
|
133
|
-
# Full-text
|
|
134
|
-
runledger search --q "weighted averaging"
|
|
135
|
-
|
|
136
|
-
# Fuzzy
|
|
137
|
-
runledger search --q "weighted avaraging" --fuzzy
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
### Aggregate
|
|
141
|
-
|
|
142
|
-
```bash
|
|
143
|
-
runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
### Diff
|
|
147
|
-
|
|
148
|
-
```bash
|
|
149
|
-
runledger diff 12 15
|
|
150
|
-
runledger diff --history 12 --batch my-sweep
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
### Export
|
|
154
|
-
|
|
155
|
-
```bash
|
|
156
|
-
runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
### Saved searches
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
|
|
163
|
-
runledger search --saved best-accuracy
|
|
164
|
-
runledger saved list
|
|
165
|
-
runledger saved delete best-accuracy
|
|
166
|
-
```
|
|
167
|
-
|
|
168
|
-
## Metrics-Path Convention
|
|
169
|
-
|
|
170
|
-
RunLedger uses **dot-separated paths** to address any value in a JSON payload.
|
|
171
|
-
|
|
172
|
-
| Example | Meaning |
|
|
173
|
-
| -------- | ------- |
|
|
174
|
-
| `accuracy` | Top-level field |
|
|
175
|
-
| `metrics.loss` | Nested field |
|
|
176
|
-
| `results[].sst2_cacc` | Field inside each element of an array |
|
|
177
|
-
| `phases[].metrics.loss` | Nested field inside array elements |
|
|
178
|
-
|
|
179
|
-
**Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
|
|
180
|
-
|
|
181
|
-
Discover available metrics with:
|
|
182
|
-
|
|
183
|
-
```bash
|
|
184
|
-
runledger metrics --batch my-sweep
|
|
185
|
-
runledger metrics --batch my-sweep --verbose # shows full paths
|
|
186
|
-
```
|
|
187
|
-
|
|
188
|
-
## API
|
|
189
|
-
|
|
190
|
-
See [`docs/api.md`](docs/api.md) for the complete REST API reference.
|
|
191
|
-
|
|
192
|
-
## Usage Examples
|
|
193
|
-
|
|
194
|
-
See [`docs/examples.md`](docs/examples.md) for real-world workflows.
|
|
195
|
-
|
|
196
|
-
## License
|
|
197
|
-
|
|
198
|
-
MIT
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|