runledger-cli 0.1.2__tar.gz → 0.1.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (22) hide show
  1. runledger_cli-0.1.3/PKG-INFO +90 -0
  2. runledger_cli-0.1.3/README.md +81 -0
  3. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/ledger.py +27 -7
  4. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_data_features.py +71 -2
  5. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_integration.py +2 -2
  6. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/pyproject.toml +1 -1
  7. runledger_cli-0.1.3/runledger_cli.egg-info/PKG-INFO +90 -0
  8. runledger_cli-0.1.2/PKG-INFO +0 -198
  9. runledger_cli-0.1.2/README.md +0 -189
  10. runledger_cli-0.1.2/runledger_cli.egg-info/PKG-INFO +0 -198
  11. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/__init__.py +0 -0
  12. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/docker-compose.yml +0 -0
  13. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/guided.py +0 -0
  14. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/__init__.py +0 -0
  15. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_ledger.py +0 -0
  16. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/cli/tests/test_real_results.py +0 -0
  17. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/SOURCES.txt +0 -0
  18. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/dependency_links.txt +0 -0
  19. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/entry_points.txt +0 -0
  20. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/requires.txt +0 -0
  21. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/runledger_cli.egg-info/top_level.txt +0 -0
  22. {runledger_cli-0.1.2 → runledger_cli-0.1.3}/setup.cfg +0 -0
@@ -0,0 +1,90 @@
1
+ Metadata-Version: 2.4
2
+ Name: runledger-cli
3
+ Version: 0.1.3
4
+ Summary: Zero‑instrumentation experiment registry CLI
5
+ Requires-Python: >=3.10
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: requests
8
+ Requires-Dist: rich
9
+
10
+ Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
11
+
12
+ ---
13
+
14
+ # RunLedger
15
+
16
+ **A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
17
+
18
+ Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
19
+
20
+ Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
21
+
22
+ ## Install
23
+
24
+ ```bash
25
+ pip install runledger-cli
26
+ ```
27
+
28
+ Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
29
+
30
+ ### Troubleshooting: `runledger` command not found (Windows)
31
+
32
+ On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
33
+
34
+ 1. **Add the Scripts folder to your PATH permanently** (recommended):
35
+ - Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
36
+ - Navigate to that folder, then up one level to the `Scripts` directory.
37
+ - Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
38
+ - Restart your terminal.
39
+
40
+ 2. **Use the Python module fallback** (works immediately, no PATH changes):
41
+ ```bash
42
+ python -m cli.ledger
43
+ ```
44
+ All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
45
+
46
+ Once the PATH is configured, `runledger` will work as shown in the examples below.
47
+
48
+ <details>
49
+ <summary>Installing from source</summary>
50
+
51
+ ```bash
52
+ git clone https://github.com/yourusername/run-ledger.git
53
+ cd run-ledger
54
+ docker compose up -d
55
+ pip install -e .
56
+ ```
57
+ </details>
58
+
59
+ ## Quick start
60
+
61
+ ```bash
62
+ runledger scan ~/experiments --batch my-sweep
63
+ runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
64
+ runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
65
+ ```
66
+
67
+ New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
68
+
69
+ ## What you get
70
+
71
+ - **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
72
+ - **Compound AND/OR** across multiple conditions in one query.
73
+ - **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
74
+ - **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
75
+ - **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
76
+ - **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
77
+ - **Self-hosted, local-first** — your data never leaves your machine.
78
+
79
+ ## Metric paths
80
+
81
+ Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
82
+
83
+ ## Docs
84
+
85
+ - [API reference](docs/api.md)
86
+ - [Usage examples](docs/examples.md)
87
+
88
+ ## License
89
+
90
+ MIT
@@ -0,0 +1,81 @@
1
+ Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
2
+
3
+ ---
4
+
5
+ # RunLedger
6
+
7
+ **A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
8
+
9
+ Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
10
+
11
+ Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
12
+
13
+ ## Install
14
+
15
+ ```bash
16
+ pip install runledger-cli
17
+ ```
18
+
19
+ Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
20
+
21
+ ### Troubleshooting: `runledger` command not found (Windows)
22
+
23
+ On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
24
+
25
+ 1. **Add the Scripts folder to your PATH permanently** (recommended):
26
+ - Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
27
+ - Navigate to that folder, then up one level to the `Scripts` directory.
28
+ - Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
29
+ - Restart your terminal.
30
+
31
+ 2. **Use the Python module fallback** (works immediately, no PATH changes):
32
+ ```bash
33
+ python -m cli.ledger
34
+ ```
35
+ All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
36
+
37
+ Once the PATH is configured, `runledger` will work as shown in the examples below.
38
+
39
+ <details>
40
+ <summary>Installing from source</summary>
41
+
42
+ ```bash
43
+ git clone https://github.com/yourusername/run-ledger.git
44
+ cd run-ledger
45
+ docker compose up -d
46
+ pip install -e .
47
+ ```
48
+ </details>
49
+
50
+ ## Quick start
51
+
52
+ ```bash
53
+ runledger scan ~/experiments --batch my-sweep
54
+ runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
55
+ runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
56
+ ```
57
+
58
+ New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
59
+
60
+ ## What you get
61
+
62
+ - **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
63
+ - **Compound AND/OR** across multiple conditions in one query.
64
+ - **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
65
+ - **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
66
+ - **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
67
+ - **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
68
+ - **Self-hosted, local-first** — your data never leaves your machine.
69
+
70
+ ## Metric paths
71
+
72
+ Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
73
+
74
+ ## Docs
75
+
76
+ - [API reference](docs/api.md)
77
+ - [Usage examples](docs/examples.md)
78
+
79
+ ## License
80
+
81
+ MIT
@@ -333,11 +333,13 @@ def navigate_json_pointer(doc, pointer: str):
333
333
 
334
334
  def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
335
335
  payload = run.get("payload", {})
336
+ # Use experiment name, then source file, then run id
336
337
  if isinstance(payload, dict):
337
338
  source_file = payload.get("_source", {}).get("file") or payload.get("experiment") or f"run-{run['id']}"
338
339
  else:
339
340
  source_file = f"run-{run['id']}"
340
341
 
342
+ # Show matched pointers if available
341
343
  matched = run.get("matched")
342
344
  if matched:
343
345
  console.print(" Matches:")
@@ -359,6 +361,7 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
359
361
  console.print(Panel(pretty, title=f"Run {run['id']} · {source_file}"))
360
362
  return
361
363
 
364
+ # Strip "[]" so JSON Pointer navigation works
362
365
  parts = [seg.replace("[]", "") for seg in dot_path.split(".")]
363
366
  max_block = len(parts) - 1
364
367
 
@@ -372,6 +375,7 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
372
375
 
373
376
  for block in sorted(levels, reverse=True):
374
377
  if block == 0 and matched:
378
+ # For block=0, show each matched snippet individually
375
379
  for i, m in enumerate(matched):
376
380
  snippet = m.get("snippet")
377
381
  try:
@@ -381,9 +385,19 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
381
385
  title = f"Run {run['id']} · {source_file} | metric: {dot_path} | depth: 0 (match {i+1}/{len(matched)})"
382
386
  console.print(Panel(pretty, title=title))
383
387
  else:
384
- ancestor_parts = parts[:len(parts) - (block + 1)]
385
- pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
386
- node = navigate_json_pointer(payload, pointer) if pointer else payload
388
+ if block > 0 and matched:
389
+ # Use the first match's concrete pointer for precise navigation
390
+ # Example: "clients[1].sweep[0].fin_asr" → "/clients/1/sweep"
391
+ concrete = matched[0]['pointer']
392
+ segments = concrete.replace('[', '.').replace(']', '').split('.')
393
+ ancestor_segments = segments[:len(segments) - (block + 1)]
394
+ pointer = "/" + "/".join(ancestor_segments) if ancestor_segments else ""
395
+ node = navigate_json_pointer(payload, pointer) if pointer else payload
396
+ else:
397
+ ancestor_parts = parts[:len(parts) - (block + 1)]
398
+ pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
399
+ node = navigate_json_pointer(payload, pointer) if pointer else payload
400
+
387
401
  depth_label = ""
388
402
  if block == max_block:
389
403
  depth_label = " (full run)"
@@ -394,7 +408,6 @@ def show_blocks(run: dict, dot_path: str = None, block_levels: list = None):
394
408
  except Exception:
395
409
  pretty = str(node)
396
410
  console.print(Panel(pretty, title=f"Run {run['id']} · {source_file} | metric: {dot_path} | depth: {block}{depth_label}"))
397
-
398
411
  # ------------------------------------------------------------
399
412
  # Export helpers
400
413
  # ------------------------------------------------------------
@@ -444,9 +457,16 @@ def build_export_blocks(content, dot_path, block_levels, include_pointers=True):
444
457
  "value": m.get("value")
445
458
  })
446
459
  else:
447
- ancestor_parts = parts[:len(parts) - (block + 1)]
448
- pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
449
- node = navigate_json_pointer(payload, pointer) if pointer else payload
460
+ if block > 0 and matched:
461
+ concrete = matched[0]['pointer']
462
+ segments = concrete.replace('[', '.').replace(']', '').split('.')
463
+ ancestor_segments = segments[:len(segments) - (block + 1)]
464
+ pointer = "/" + "/".join(ancestor_segments) if ancestor_segments else ""
465
+ node = navigate_json_pointer(payload, pointer) if pointer else payload
466
+ else:
467
+ ancestor_parts = parts[:len(parts) - (block + 1)]
468
+ pointer = "/" + "/".join(ancestor_parts) if ancestor_parts else ""
469
+ node = navigate_json_pointer(payload, pointer) if pointer else payload
450
470
  blocks.append({
451
471
  "id": run["id"],
452
472
  "sourceFile": source_file,
@@ -57,7 +57,7 @@ def setup_data():
57
57
  # Ingest sample runs into the fresh batch
58
58
  result = run_cli("scan", str(SAMPLE_FOLDER), "--batch", BATCH_NAME)
59
59
  assert result.returncode == 0, result.stderr
60
- assert "Successfully ingested 9 run(s)" in result.stdout
60
+ assert "Successfully ingested 10 run(s)" in result.stdout
61
61
 
62
62
  # ------------------------------------------------------------
63
63
  # v1 – Basic metric block search
@@ -251,4 +251,73 @@ def test_jsonl_ingestion(setup_data):
251
251
  assert payload.get(k) == v, f"Mismatch at line {idx}: {k} {v} != {payload.get(k)}"
252
252
 
253
253
  # Clean up the temporary batch
254
- requests.delete(f"{API_BASE}/batch/{batch}") # if this endpoint exists, otherwise manual cleanup
254
+ requests.delete(f"{API_BASE}/batch/{batch}") # if this endpoint exists, otherwise manual cleanup
255
+
256
+ def test_nested_array_block_depth():
257
+ """Verify block‑depth on nested arrays using direct API calls."""
258
+ import uuid
259
+ import requests
260
+
261
+ batch = f"nest-{uuid.uuid4().hex[:6]}"
262
+
263
+ # 1. Ingest the nested file directly via the API
264
+ payload = {
265
+ "experiment": "nested_arrays_block_test",
266
+ "clients": [
267
+ {
268
+ "client_name": "clean_client",
269
+ "sweep": [
270
+ {"w_base": 0.5, "fin_asr": 0.12, "sst2_asr": 0.07},
271
+ {"w_base": 0.75, "fin_asr": 0.09, "sst2_asr": 0.06}
272
+ ]
273
+ },
274
+ {
275
+ "client_name": "poisoned_client",
276
+ "sweep": [
277
+ {"w_base": 0.5, "fin_asr": 0.45, "sst2_asr": 0.88},
278
+ {"w_base": 1.0, "fin_asr": 0.28, "sst2_asr": 0.72}
279
+ ]
280
+ }
281
+ ]
282
+ }
283
+ resp = requests.post(API_BASE, json={"payload": payload, "batch": batch})
284
+ assert resp.status_code == 201, f"ingest failed: {resp.text}"
285
+
286
+ # 2. Search for fin_asr < 0.3 with block=0
287
+ params = {"metric": "clients[].sweep[].fin_asr", "op": "lt", "value": "0.3",
288
+ "batch": batch, "block": "0", "size": "1", "pointers": "true"}
289
+ resp = requests.get(API_BASE, params=params)
290
+ assert resp.status_code == 200, resp.text
291
+ data = resp.json()
292
+ assert data["totalElements"] == 1
293
+ run = data["content"][0]
294
+
295
+ # 3. Verify matched pointers exist
296
+ matched = run.get("matched")
297
+ assert matched is not None, "matched pointers missing"
298
+ assert len(matched) == 3, f"expected 3 matches, got {len(matched)}"
299
+
300
+ # 4. block=1 should return the sweep array (a list)
301
+ params["block"] = "1"
302
+ resp = requests.get(API_BASE, params=params)
303
+ assert resp.status_code == 200, resp.text
304
+ data1 = resp.json()
305
+ block1_payload = data1["content"][0]["payload"]
306
+ assert isinstance(block1_payload, list), f"block=1 payload should be a list, got {type(block1_payload)}"
307
+ assert any("fin_asr" in json.dumps(elem) for elem in block1_payload)
308
+
309
+ # 5. block=2 should return the client object
310
+ params["block"] = "2"
311
+ resp = requests.get(API_BASE, params=params)
312
+ assert resp.status_code == 200, resp.text
313
+ data2 = resp.json()
314
+ block2_payload = data2["content"][0]["payload"]
315
+ assert isinstance(block2_payload, dict) and block2_payload.get("client_name") == "clean_client"
316
+
317
+ # 6. block=3 should return the clients array
318
+ params["block"] = "3"
319
+ resp = requests.get(API_BASE, params=params)
320
+ assert resp.status_code == 200, resp.text
321
+ data3 = resp.json()
322
+ block3_payload = data3["content"][0]["payload"]
323
+ assert isinstance(block3_payload, list) and len(block3_payload) > 0
@@ -10,8 +10,8 @@ import pytest
10
10
  import requests
11
11
 
12
12
  PROJECT_ROOT = Path(__file__).parent.parent.parent
13
- COMPOSE_FILE = PROJECT_ROOT / "docker-compose.yml"
14
- HEALTH_URL = "http://localhost:8080/actuator/health"
13
+ COMPOSE_FILE = PROJECT_ROOT / "docker-compose.test.yml"
14
+ HEALTH_URL = "http://localhost:8081/actuator/health"
15
15
  API_BASE = "http://localhost:8080/api/runs"
16
16
 
17
17
  def wait_for_health(timeout=120):
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "runledger-cli"
7
- version = "0.1.2"
7
+ version = "0.1.3"
8
8
  description = "Zero‑instrumentation experiment registry CLI"
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.10"
@@ -0,0 +1,90 @@
1
+ Metadata-Version: 2.4
2
+ Name: runledger-cli
3
+ Version: 0.1.3
4
+ Summary: Zero‑instrumentation experiment registry CLI
5
+ Requires-Python: >=3.10
6
+ Description-Content-Type: text/markdown
7
+ Requires-Dist: requests
8
+ Requires-Dist: rich
9
+
10
+ Here’s the updated README with the troubleshooting note added right after the installation instructions. The primary command remains `runledger`, but users who hit the Windows PATH issue now have a clear fallback.
11
+
12
+ ---
13
+
14
+ # RunLedger
15
+
16
+ **A zero-instrumentation experiment registry.** Point RunLedger at a folder of JSON experiment files and search, filter, aggregate, diff, and export the results; no SDK, no code changes, no cloud.
17
+
18
+ Most experiment trackers make you instrument your training code before they can help you. RunLedger reads the JSON you're already writing, no `wandb.init()`, no `mlflow.log_metric()`, and indexes it in real Postgres, giving you full relational search: structured comparisons, full-text, fuzzy matching, and exact array-element lookups, from one CLI command.
19
+
20
+ Backed by Postgres, served by a Spring Boot API, and fronted by a Python CLI.
21
+
22
+ ## Install
23
+
24
+ ```bash
25
+ pip install runledger-cli
26
+ ```
27
+
28
+ Requires Docker. If it's not installed, `runledger` detects that on first run and links you straight to the installer; everything else (Postgres, the backend) is managed for you.
29
+
30
+ ### Troubleshooting: `runledger` command not found (Windows)
31
+
32
+ On some Windows systems, the Python Scripts folder is not automatically added to your PATH. If `runledger` is not recognized after installation, you have two options:
33
+
34
+ 1. **Add the Scripts folder to your PATH permanently** (recommended):
35
+ - Find where `runledger` was installed: `pip show runledger-cli | findstr Location`
36
+ - Navigate to that folder, then up one level to the `Scripts` directory.
37
+ - Add that `Scripts` path to your system PATH via **System Properties > Environment Variables**.
38
+ - Restart your terminal.
39
+
40
+ 2. **Use the Python module fallback** (works immediately, no PATH changes):
41
+ ```bash
42
+ python -m cli.ledger
43
+ ```
44
+ All subcommands and the guided wizard work exactly the same way; just prefix every command with `python -m cli.ledger`.
45
+
46
+ Once the PATH is configured, `runledger` will work as shown in the examples below.
47
+
48
+ <details>
49
+ <summary>Installing from source</summary>
50
+
51
+ ```bash
52
+ git clone https://github.com/yourusername/run-ledger.git
53
+ cd run-ledger
54
+ docker compose up -d
55
+ pip install -e .
56
+ ```
57
+ </details>
58
+
59
+ ## Quick start
60
+
61
+ ```bash
62
+ runledger scan ~/experiments --batch my-sweep
63
+ runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
64
+ runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
65
+ ```
66
+
67
+ New here? Run `runledger` with no arguments — the guided wizard walks you through your first scan and search.
68
+
69
+ ## What you get
70
+
71
+ - **Search any field, no schema required** — structured comparisons (`gt`, `lt`, `eq`...), full-text with stemming, and fuzzy matching for typos, all on arbitrary dot-paths, including fields nested inside arrays (`results[].sst2_cacc`).
72
+ - **Compound AND/OR** across multiple conditions in one query.
73
+ - **Exact match pointers** — array searches tell you exactly which element(s) matched (`results[3].sst2_cacc = 0.933`), not just that the run did.
74
+ - **Aggregates & group-by** — `AVG` / `MAX` / `MIN` / `SUM` / `COUNT`, straight from the CLI.
75
+ - **Safe re-scanning** — ingestion is idempotent; a changed file gets a new version, and `diff --history` shows you what changed and when.
76
+ - **Run diff, saved searches, JSONL ingestion, block-depth control** — see [docs/examples.md](docs/examples.md) for each in action.
77
+ - **Self-hosted, local-first** — your data never leaves your machine.
78
+
79
+ ## Metric paths
80
+
81
+ Address any value with a dot-path — `accuracy`, `metrics.loss`, `results[].sst2_cacc`. Shorthand (`cacc`) resolves automatically when it's unambiguous. Run `runledger metrics --batch <batch>` to see everything discovered in a scan, or add `--verbose` for full paths.
82
+
83
+ ## Docs
84
+
85
+ - [API reference](docs/api.md)
86
+ - [Usage examples](docs/examples.md)
87
+
88
+ ## License
89
+
90
+ MIT
@@ -1,198 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: runledger-cli
3
- Version: 0.1.2
4
- Summary: Zero‑instrumentation experiment registry CLI
5
- Requires-Python: >=3.10
6
- Description-Content-Type: text/markdown
7
- Requires-Dist: requests
8
- Requires-Dist: rich
9
-
10
- # RunLedger
11
-
12
- **Zero-instrumentation experiment registry and search engine.**
13
- Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
14
-
15
- ## Quick Start
16
-
17
- ```bash
18
- # 1. Start the backend
19
- docker compose up -d
20
-
21
- # 2. Ingest your experiment files
22
- runledger scan ~/experiments --batch my-sweep
23
-
24
- # 3. Search
25
- runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
26
-
27
- # 4. Aggregate
28
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
29
- ```
30
-
31
- New to the tool? Try the guided wizard:
32
-
33
- ```bash
34
- runledger guided
35
- ```
36
-
37
- ## Key Features
38
-
39
- ### 🔍 Search any JSON field, no schema required
40
- - Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
41
- - Full-text phrase search with stemming.
42
- - Fuzzy trigram search for typos.
43
- - Compound AND/OR queries: combine multiple conditions.
44
- - Batch isolation: group runs by folder and filter within a batch.
45
-
46
- ### 📊 Aggregates & group-by
47
- Answer research questions directly:
48
-
49
- ```bash
50
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
51
- ```
52
-
53
- Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
54
-
55
- ### 🎯 Exact match pointers (“JSON line numbers”)
56
- For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
57
-
58
- ```text
59
- Matches:
60
- • results[0].sst2_cacc = 0.924312
61
- • results[3].sst2_cacc = 0.933486
62
- ```
63
-
64
- Works for scalar, array, and compound conditions.
65
-
66
- ### 🔬 Block-depth control
67
- Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
68
-
69
- ### 📁 JSONL ingestion
70
- Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
71
-
72
- ### 🔎 Run diff
73
- Compare any two runs side-by-side:
74
-
75
- ```bash
76
- runledger diff 12 15
77
- ```
78
-
79
- Or see every version of a re-scanned run with `--history`.
80
-
81
- ### 💾 Saved searches
82
- Save frequent queries and re-run them with a single command:
83
-
84
- ```bash
85
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
86
- runledger search --saved best-accuracy
87
- ```
88
-
89
- ### 🧭 Interactive & guided modes
90
- - Filter-as-you-type interactive search (`runledger interactive`).
91
- - Step-by-step wizard for first-time users (`runledger guided`).
92
-
93
- ### 🔒 Self-hosted, local-first
94
- Runs in Docker with a single `docker compose up -d`.
95
- All data stays on your machine. No internet, no accounts, no vendor lock-in.
96
-
97
- ## Installation
98
-
99
- ### From PyPI (CLI only)
100
-
101
- ```bash
102
- pip install runledger-cli
103
- ```
104
-
105
- ### From source
106
-
107
- ```bash
108
- git clone https://github.com/yourusername/run-ledger.git
109
- cd run-ledger
110
- docker compose up -d # starts PostgreSQL + Spring Boot
111
- pip install -e . # installs the CLI
112
- ```
113
-
114
- ## Quick Tour
115
-
116
- ### Scan & Discover
117
-
118
- ```bash
119
- runledger scan experiments/ --batch my-sweep
120
- ```
121
-
122
- The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
123
-
124
- ### Search
125
-
126
- ```bash
127
- # Single condition
128
- runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
129
-
130
- # Compound AND
131
- runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
132
-
133
- # Full-text
134
- runledger search --q "weighted averaging"
135
-
136
- # Fuzzy
137
- runledger search --q "weighted avaraging" --fuzzy
138
- ```
139
-
140
- ### Aggregate
141
-
142
- ```bash
143
- runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
144
- ```
145
-
146
- ### Diff
147
-
148
- ```bash
149
- runledger diff 12 15
150
- runledger diff --history 12 --batch my-sweep
151
- ```
152
-
153
- ### Export
154
-
155
- ```bash
156
- runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
157
- ```
158
-
159
- ### Saved searches
160
-
161
- ```bash
162
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
163
- runledger search --saved best-accuracy
164
- runledger saved list
165
- runledger saved delete best-accuracy
166
- ```
167
-
168
- ## Metrics-Path Convention
169
-
170
- RunLedger uses **dot-separated paths** to address any value in a JSON payload.
171
-
172
- | Example | Meaning |
173
- | -------- | ------- |
174
- | `accuracy` | Top-level field |
175
- | `metrics.loss` | Nested field |
176
- | `results[].sst2_cacc` | Field inside each element of an array |
177
- | `phases[].metrics.loss` | Nested field inside array elements |
178
-
179
- **Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
180
-
181
- Discover available metrics with:
182
-
183
- ```bash
184
- runledger metrics --batch my-sweep
185
- runledger metrics --batch my-sweep --verbose # shows full paths
186
- ```
187
-
188
- ## API
189
-
190
- See [`docs/api.md`](docs/api.md) for the complete REST API reference.
191
-
192
- ## Usage Examples
193
-
194
- See [`docs/examples.md`](docs/examples.md) for real-world workflows.
195
-
196
- ## License
197
-
198
- MIT
@@ -1,189 +0,0 @@
1
- # RunLedger
2
-
3
- **Zero-instrumentation experiment registry and search engine.**
4
- Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
5
-
6
- ## Quick Start
7
-
8
- ```bash
9
- # 1. Start the backend
10
- docker compose up -d
11
-
12
- # 2. Ingest your experiment files
13
- runledger scan ~/experiments --batch my-sweep
14
-
15
- # 3. Search
16
- runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
17
-
18
- # 4. Aggregate
19
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
20
- ```
21
-
22
- New to the tool? Try the guided wizard:
23
-
24
- ```bash
25
- runledger guided
26
- ```
27
-
28
- ## Key Features
29
-
30
- ### 🔍 Search any JSON field, no schema required
31
- - Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
32
- - Full-text phrase search with stemming.
33
- - Fuzzy trigram search for typos.
34
- - Compound AND/OR queries: combine multiple conditions.
35
- - Batch isolation: group runs by folder and filter within a batch.
36
-
37
- ### 📊 Aggregates & group-by
38
- Answer research questions directly:
39
-
40
- ```bash
41
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
42
- ```
43
-
44
- Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
45
-
46
- ### 🎯 Exact match pointers (“JSON line numbers”)
47
- For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
48
-
49
- ```text
50
- Matches:
51
- • results[0].sst2_cacc = 0.924312
52
- • results[3].sst2_cacc = 0.933486
53
- ```
54
-
55
- Works for scalar, array, and compound conditions.
56
-
57
- ### 🔬 Block-depth control
58
- Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
59
-
60
- ### 📁 JSONL ingestion
61
- Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
62
-
63
- ### 🔎 Run diff
64
- Compare any two runs side-by-side:
65
-
66
- ```bash
67
- runledger diff 12 15
68
- ```
69
-
70
- Or see every version of a re-scanned run with `--history`.
71
-
72
- ### 💾 Saved searches
73
- Save frequent queries and re-run them with a single command:
74
-
75
- ```bash
76
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
77
- runledger search --saved best-accuracy
78
- ```
79
-
80
- ### 🧭 Interactive & guided modes
81
- - Filter-as-you-type interactive search (`runledger interactive`).
82
- - Step-by-step wizard for first-time users (`runledger guided`).
83
-
84
- ### 🔒 Self-hosted, local-first
85
- Runs in Docker with a single `docker compose up -d`.
86
- All data stays on your machine. No internet, no accounts, no vendor lock-in.
87
-
88
- ## Installation
89
-
90
- ### From PyPI (CLI only)
91
-
92
- ```bash
93
- pip install runledger-cli
94
- ```
95
-
96
- ### From source
97
-
98
- ```bash
99
- git clone https://github.com/yourusername/run-ledger.git
100
- cd run-ledger
101
- docker compose up -d # starts PostgreSQL + Spring Boot
102
- pip install -e . # installs the CLI
103
- ```
104
-
105
- ## Quick Tour
106
-
107
- ### Scan & Discover
108
-
109
- ```bash
110
- runledger scan experiments/ --batch my-sweep
111
- ```
112
-
113
- The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
114
-
115
- ### Search
116
-
117
- ```bash
118
- # Single condition
119
- runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
120
-
121
- # Compound AND
122
- runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
123
-
124
- # Full-text
125
- runledger search --q "weighted averaging"
126
-
127
- # Fuzzy
128
- runledger search --q "weighted avaraging" --fuzzy
129
- ```
130
-
131
- ### Aggregate
132
-
133
- ```bash
134
- runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
135
- ```
136
-
137
- ### Diff
138
-
139
- ```bash
140
- runledger diff 12 15
141
- runledger diff --history 12 --batch my-sweep
142
- ```
143
-
144
- ### Export
145
-
146
- ```bash
147
- runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
148
- ```
149
-
150
- ### Saved searches
151
-
152
- ```bash
153
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
154
- runledger search --saved best-accuracy
155
- runledger saved list
156
- runledger saved delete best-accuracy
157
- ```
158
-
159
- ## Metrics-Path Convention
160
-
161
- RunLedger uses **dot-separated paths** to address any value in a JSON payload.
162
-
163
- | Example | Meaning |
164
- | -------- | ------- |
165
- | `accuracy` | Top-level field |
166
- | `metrics.loss` | Nested field |
167
- | `results[].sst2_cacc` | Field inside each element of an array |
168
- | `phases[].metrics.loss` | Nested field inside array elements |
169
-
170
- **Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
171
-
172
- Discover available metrics with:
173
-
174
- ```bash
175
- runledger metrics --batch my-sweep
176
- runledger metrics --batch my-sweep --verbose # shows full paths
177
- ```
178
-
179
- ## API
180
-
181
- See [`docs/api.md`](docs/api.md) for the complete REST API reference.
182
-
183
- ## Usage Examples
184
-
185
- See [`docs/examples.md`](docs/examples.md) for real-world workflows.
186
-
187
- ## License
188
-
189
- MIT
@@ -1,198 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: runledger-cli
3
- Version: 0.1.2
4
- Summary: Zero‑instrumentation experiment registry CLI
5
- Requires-Python: >=3.10
6
- Description-Content-Type: text/markdown
7
- Requires-Dist: requests
8
- Requires-Dist: rich
9
-
10
- # RunLedger
11
-
12
- **Zero-instrumentation experiment registry and search engine.**
13
- Point to a folder of JSON experiment files and instantly search, filter, aggregate, diff, and export your results—no SDK, no code changes.
14
-
15
- ## Quick Start
16
-
17
- ```bash
18
- # 1. Start the backend
19
- docker compose up -d
20
-
21
- # 2. Ingest your experiment files
22
- runledger scan ~/experiments --batch my-sweep
23
-
24
- # 3. Search
25
- runledger search --metric accuracy --op gt --value 0.9 --batch my-sweep
26
-
27
- # 4. Aggregate
28
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch my-sweep
29
- ```
30
-
31
- New to the tool? Try the guided wizard:
32
-
33
- ```bash
34
- runledger guided
35
- ```
36
-
37
- ## Key Features
38
-
39
- ### 🔍 Search any JSON field, no schema required
40
- - Structured numeric/text comparisons on arbitrary dot-paths (`metrics.accuracy`, `config.optimizer.learning_rate`).
41
- - Full-text phrase search with stemming.
42
- - Fuzzy trigram search for typos.
43
- - Compound AND/OR queries: combine multiple conditions.
44
- - Batch isolation: group runs by folder and filter within a batch.
45
-
46
- ### 📊 Aggregates & group-by
47
- Answer research questions directly:
48
-
49
- ```bash
50
- runledger aggregate --metric accuracy --agg AVG --group-by experiment --batch sweep
51
- ```
52
-
53
- Supported functions: `AVG`, `MAX`, `MIN`, `SUM`, `COUNT`. Group by any scalar field.
54
-
55
- ### 🎯 Exact match pointers (“JSON line numbers”)
56
- For array searches (`results[].sst2_cacc < 0.8`), the CLI shows exactly which array elements matched:
57
-
58
- ```text
59
- Matches:
60
- • results[0].sst2_cacc = 0.924312
61
- • results[3].sst2_cacc = 0.933486
62
- ```
63
-
64
- Works for scalar, array, and compound conditions.
65
-
66
- ### 🔬 Block-depth control
67
- Limit output to the relevant part of a large JSON payload with `--block 0` (tightest match), `--block 1` (parent), etc.
68
-
69
- ### 📁 JSONL ingestion
70
- Ingest one-JSON-object-per-line logs (common in training scripts) with `--jsonl`.
71
-
72
- ### 🔎 Run diff
73
- Compare any two runs side-by-side:
74
-
75
- ```bash
76
- runledger diff 12 15
77
- ```
78
-
79
- Or see every version of a re-scanned run with `--history`.
80
-
81
- ### 💾 Saved searches
82
- Save frequent queries and re-run them with a single command:
83
-
84
- ```bash
85
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
86
- runledger search --saved best-accuracy
87
- ```
88
-
89
- ### 🧭 Interactive & guided modes
90
- - Filter-as-you-type interactive search (`runledger interactive`).
91
- - Step-by-step wizard for first-time users (`runledger guided`).
92
-
93
- ### 🔒 Self-hosted, local-first
94
- Runs in Docker with a single `docker compose up -d`.
95
- All data stays on your machine. No internet, no accounts, no vendor lock-in.
96
-
97
- ## Installation
98
-
99
- ### From PyPI (CLI only)
100
-
101
- ```bash
102
- pip install runledger-cli
103
- ```
104
-
105
- ### From source
106
-
107
- ```bash
108
- git clone https://github.com/yourusername/run-ledger.git
109
- cd run-ledger
110
- docker compose up -d # starts PostgreSQL + Spring Boot
111
- pip install -e . # installs the CLI
112
- ```
113
-
114
- ## Quick Tour
115
-
116
- ### Scan & Discover
117
-
118
- ```bash
119
- runledger scan experiments/ --batch my-sweep
120
- ```
121
-
122
- The CLI prints a schema summary showing every metric found, its full dot-path, and depth. Array keys are shown with `[]` notation (e.g., `results[].cacc`).
123
-
124
- ### Search
125
-
126
- ```bash
127
- # Single condition
128
- runledger search --metric loss --op lt --value 0.2 --batch my-sweep --summary
129
-
130
- # Compound AND
131
- runledger search --metric accuracy --op gt --value 0.9 --metric loss --op lt --value 0.2 --combine and
132
-
133
- # Full-text
134
- runledger search --q "weighted averaging"
135
-
136
- # Fuzzy
137
- runledger search --q "weighted avaraging" --fuzzy
138
- ```
139
-
140
- ### Aggregate
141
-
142
- ```bash
143
- runledger aggregate --metric accuracy --agg AVG --group-by config.learning_rate --batch my-sweep
144
- ```
145
-
146
- ### Diff
147
-
148
- ```bash
149
- runledger diff 12 15
150
- runledger diff --history 12 --batch my-sweep
151
- ```
152
-
153
- ### Export
154
-
155
- ```bash
156
- runledger export --metric accuracy --op gt --value 0.9 --batch my-sweep --block 0 --output results.txt
157
- ```
158
-
159
- ### Saved searches
160
-
161
- ```bash
162
- runledger search --metric accuracy --op gt --value 0.9 --batch sweep --save best-accuracy
163
- runledger search --saved best-accuracy
164
- runledger saved list
165
- runledger saved delete best-accuracy
166
- ```
167
-
168
- ## Metrics-Path Convention
169
-
170
- RunLedger uses **dot-separated paths** to address any value in a JSON payload.
171
-
172
- | Example | Meaning |
173
- | -------- | ------- |
174
- | `accuracy` | Top-level field |
175
- | `metrics.loss` | Nested field |
176
- | `results[].sst2_cacc` | Field inside each element of an array |
177
- | `phases[].metrics.loss` | Nested field inside array elements |
178
-
179
- **Shorthand keys:** After scanning a batch, the CLI shows a shorthand key for every leaf field. Typing `cacc` will automatically resolve to `client.results[].cacc` if that’s the only (or shallowest) occurrence. Use the full path when there’s ambiguity.
180
-
181
- Discover available metrics with:
182
-
183
- ```bash
184
- runledger metrics --batch my-sweep
185
- runledger metrics --batch my-sweep --verbose # shows full paths
186
- ```
187
-
188
- ## API
189
-
190
- See [`docs/api.md`](docs/api.md) for the complete REST API reference.
191
-
192
- ## Usage Examples
193
-
194
- See [`docs/examples.md`](docs/examples.md) for real-world workflows.
195
-
196
- ## License
197
-
198
- MIT
File without changes