syvain-metrics 0.0.360 → 0.0.367
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +63 -25
- package/package.json +7 -7
package/README.md
CHANGED
|
@@ -1,7 +1,9 @@
|
|
|
1
1
|
# syvain-metrics
|
|
2
2
|
|
|
3
|
-
Command line
|
|
4
|
-
|
|
3
|
+
Command line tool for [Syvain Metrics](https://metrics.syvain.com/). It is built
|
|
4
|
+
for agents and scripts: every command prints newline-delimited JSON on stdout,
|
|
5
|
+
errors go to stderr with exit code 1, and experiments and folders can be
|
|
6
|
+
addressed by slug or path instead of ids.
|
|
5
7
|
|
|
6
8
|
## Install
|
|
7
9
|
|
|
@@ -13,40 +15,76 @@ The package installs the `syvain-metrics` binary.
|
|
|
13
15
|
|
|
14
16
|
## Authenticate
|
|
15
17
|
|
|
16
|
-
Create an API key
|
|
18
|
+
Create an organization API key in the Syvain Metrics dashboard, then either save
|
|
19
|
+
it with a login or export it for the current shell:
|
|
17
20
|
|
|
18
21
|
```sh
|
|
19
|
-
syvain-metrics auth login
|
|
20
|
-
syvain-metrics auth
|
|
22
|
+
syvain-metrics auth login # prompts for the key
|
|
23
|
+
printf '%s' "$KEY" | syvain-metrics auth login --stdin
|
|
24
|
+
syvain-metrics auth status
|
|
25
|
+
|
|
26
|
+
export SYVAIN_METRICS_API_KEY=ak_... # skips the saved login
|
|
27
|
+
export SYVAIN_METRICS_HOST=https://metrics.syvain.com # optional override
|
|
21
28
|
```
|
|
22
29
|
|
|
23
|
-
|
|
30
|
+
The CLI talks to `https://metrics.syvain.com` unless `--host` or
|
|
31
|
+
`SYVAIN_METRICS_HOST` says otherwise.
|
|
32
|
+
|
|
33
|
+
## Addressing things
|
|
34
|
+
|
|
35
|
+
- An **experiment** is its id or its slug: `experiment get mamba-run-001`.
|
|
36
|
+
- A **folder** is its absolute path or its id: `/models/mamba`. The root is `/`.
|
|
37
|
+
Paths are derived from the organization tree, so names with spaces work when
|
|
38
|
+
quoted.
|
|
39
|
+
|
|
40
|
+
## Folders
|
|
24
41
|
|
|
25
42
|
```sh
|
|
26
|
-
|
|
43
|
+
syvain-metrics folder list # every folder with its path
|
|
44
|
+
syvain-metrics folder list /models # direct children of one folder
|
|
45
|
+
syvain-metrics folder create /models/mamba # creates missing parents; idempotent
|
|
46
|
+
syvain-metrics folder move /scratch/run-1 /models
|
|
47
|
+
syvain-metrics folder move /scratch/run-1 / # to the root
|
|
48
|
+
syvain-metrics folder rename /models/mamba mamba-v2
|
|
27
49
|
```
|
|
28
50
|
|
|
29
|
-
|
|
51
|
+
`folder create` prints the leaf folder with `created: true` when it was made and
|
|
52
|
+
`created: false` when the path already existed.
|
|
53
|
+
|
|
54
|
+
## Experiments
|
|
55
|
+
|
|
56
|
+
```sh
|
|
57
|
+
syvain-metrics experiment list --search mamba # id, slug, folderId, folderPath
|
|
58
|
+
syvain-metrics experiment list --folder /models # direct members only
|
|
59
|
+
syvain-metrics experiment get mamba-run-001 # status, timestamps, meta, folderPath, url
|
|
60
|
+
syvain-metrics experiment move mamba-run-001 /models/mamba
|
|
61
|
+
syvain-metrics experiment catalog mamba-run-001 # series names with metadata keys and values
|
|
62
|
+
syvain-metrics experiment annotations mamba-run-001 --limit 20
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Series
|
|
66
|
+
|
|
67
|
+
`series query` prints one line per experiment, series, and metadata partition:
|
|
68
|
+
`{experimentId, seriesName, metadata, data: {step, timestampMs, value}}`.
|
|
69
|
+
Without `--name` it reads every series in the experiment's catalog.
|
|
70
|
+
|
|
71
|
+
```sh
|
|
72
|
+
syvain-metrics series query mamba-run-001 --name loss --filter split=valid
|
|
73
|
+
syvain-metrics series query mamba-run-001 > run-001.jsonl
|
|
74
|
+
syvain-metrics series query run-a run-b --name loss --x-axis timestamp
|
|
75
|
+
```
|
|
30
76
|
|
|
31
|
-
|
|
77
|
+
For analysis across many experiments prefer the
|
|
78
|
+
[DuckDB extension](../../libs/metrics_duckdb/README.md); the CLI is the quick
|
|
79
|
+
way to pipe one experiment into a file.
|
|
32
80
|
|
|
33
|
-
|
|
34
|
-
|
|
81
|
+
`series render` charts the same selections as a PNG through the public Metrics
|
|
82
|
+
Renderer. Every experiment, series, and metadata partition becomes one line, so
|
|
83
|
+
use `--filter` to narrow what is drawn.
|
|
35
84
|
|
|
36
85
|
```sh
|
|
37
|
-
syvain-metrics
|
|
38
|
-
syvain-metrics
|
|
39
|
-
syvain-metrics experiments list --limit 20 --human
|
|
40
|
-
syvain-metrics experiments get --slug dendro-mamba-run-001
|
|
41
|
-
syvain-metrics experiments metrics --id exp_123 --metric-name loss
|
|
42
|
-
syvain-metrics experiments query metric-series \
|
|
43
|
-
--series '{"experimentId":"exp_123","metricName":"loss","metadataFilter":[{"key":"split","value":"\"train\""}]}'
|
|
44
|
-
syvain-metrics experiments render metric-series \
|
|
45
|
-
--series '{"experimentId":"exp_123","metricName":"loss","metadataFilter":[{"key":"model","value":"\"small\""}],"splitKeys":["data_split","curriculum_split"]}' \
|
|
46
|
-
--output metrics.png
|
|
47
|
-
syvain-metrics experiments query metric-catalog --experiment-ids exp_123,exp_456
|
|
48
|
-
syvain-metrics experiments annotate --id exp_123 --annotation "Good validation run"
|
|
86
|
+
syvain-metrics series render run-a run-b --name loss --filter split=valid -o loss.png
|
|
87
|
+
syvain-metrics series render run-a --name loss --display-mode subplots -o loss.png
|
|
49
88
|
```
|
|
50
89
|
|
|
51
|
-
Run `syvain-metrics --help` or `syvain-metrics <command> --help` for
|
|
52
|
-
command list and options.
|
|
90
|
+
Run `syvain-metrics --help` or `syvain-metrics <command> --help` for every flag.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "syvain-metrics",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.367",
|
|
4
4
|
"description": "Syvain Metrics command line interface.",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -15,11 +15,11 @@
|
|
|
15
15
|
"syvain-metrics": "bin/syvain-metrics.js"
|
|
16
16
|
},
|
|
17
17
|
"optionalDependencies": {
|
|
18
|
-
"syvain-metrics-linux-x64": "0.0.
|
|
19
|
-
"syvain-metrics-linux-arm64": "0.0.
|
|
20
|
-
"syvain-metrics-darwin-x64": "0.0.
|
|
21
|
-
"syvain-metrics-darwin-arm64": "0.0.
|
|
22
|
-
"syvain-metrics-windows-x64": "0.0.
|
|
23
|
-
"syvain-metrics-windows-arm64": "0.0.
|
|
18
|
+
"syvain-metrics-linux-x64": "0.0.367",
|
|
19
|
+
"syvain-metrics-linux-arm64": "0.0.367",
|
|
20
|
+
"syvain-metrics-darwin-x64": "0.0.367",
|
|
21
|
+
"syvain-metrics-darwin-arm64": "0.0.367",
|
|
22
|
+
"syvain-metrics-windows-x64": "0.0.367",
|
|
23
|
+
"syvain-metrics-windows-arm64": "0.0.367"
|
|
24
24
|
}
|
|
25
25
|
}
|