dsh-runnpu 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +16 -0
- package/compute-mode/preset/agent.cordis.yml +368 -0
- package/compute-mode/preset/preset.yml +3 -0
- package/compute-mode/preset/skills/runnpu/SKILL.md +233 -0
- package/compute-mode/preset/skills/runnpu/VENDOR.txt +11 -0
- package/compute-mode/preset/skills/runnpu/examples/01-dev-environment.sh +26 -0
- package/compute-mode/preset/skills/runnpu/examples/02-training-job.sh +26 -0
- package/compute-mode/preset/skills/runnpu/examples/03-inference-service.sh +24 -0
- package/compute-mode/preset/skills/runnpu/examples/04-fractional-vnpu.sh +14 -0
- package/compute-mode/preset/skills/runnpu/examples/05-troubleshoot.sh +17 -0
- package/compute-mode/preset/skills/runnpu/references/admin.md +60 -0
- package/compute-mode/preset/skills/runnpu/references/data-sources.md +35 -0
- package/compute-mode/preset/skills/runnpu/references/templates-assets.md +61 -0
- package/compute-mode/preset/skills/runnpu/references/training-runs.md +56 -0
- package/compute-mode/preset/skills/runnpu/references/troubleshooting.md +75 -0
- package/compute-mode/preset/skills/runnpu/references/workload-types.md +47 -0
- package/cordis.patch.yml +80 -0
- package/dist/chunk-JLZMOSQC.mjs +1 -0
- package/dist/chunk-JUBIWGKW.mjs +4 -0
- package/dist/chunk-JZXLCA2E.mjs +1 -0
- package/dist/chunk-KQOFBYGE.mjs +9 -0
- package/dist/chunk-XMZG2NP7.mjs +2 -0
- package/dist/chunk-XQPU3AJN.mjs +1 -0
- package/dist/chunk-YKASEMP3.mjs +1 -0
- package/dist/chunk-YPL4VUYF.mjs +4 -0
- package/dist/compute-node.mjs +1 -0
- package/dist/compute-tools.mjs +41 -0
- package/dist/connectors.mjs +3 -0
- package/dist/dsh-runnpu.mjs +43 -0
- package/dist/json.mjs +1 -0
- package/dist/pins.mjs +1 -0
- package/dist/series.mjs +1 -0
- package/dist/training-node.mjs +2 -0
- package/dist/training-tools.mjs +30 -0
- package/dist/ui-host.mjs +1 -0
- package/lib/client.js +189 -0
- package/package.json +78 -0
- package/training-mode/preset/agent.cordis.yml +343 -0
- package/training-mode/preset/preset.yml +3 -0
- package/training-mode/preset/skills/runnpu/SKILL.md +233 -0
- package/training-mode/preset/skills/runnpu/VENDOR.txt +11 -0
- package/training-mode/preset/skills/runnpu/examples/01-dev-environment.sh +26 -0
- package/training-mode/preset/skills/runnpu/examples/02-training-job.sh +26 -0
- package/training-mode/preset/skills/runnpu/examples/03-inference-service.sh +24 -0
- package/training-mode/preset/skills/runnpu/examples/04-fractional-vnpu.sh +14 -0
- package/training-mode/preset/skills/runnpu/examples/05-troubleshoot.sh +17 -0
- package/training-mode/preset/skills/runnpu/references/admin.md +60 -0
- package/training-mode/preset/skills/runnpu/references/data-sources.md +35 -0
- package/training-mode/preset/skills/runnpu/references/templates-assets.md +61 -0
- package/training-mode/preset/skills/runnpu/references/training-runs.md +56 -0
- package/training-mode/preset/skills/runnpu/references/troubleshooting.md +75 -0
- package/training-mode/preset/skills/runnpu/references/workload-types.md +47 -0
- package/training-mode/preset/skills/swanlab-skill/LICENSE +21 -0
- package/training-mode/preset/skills/swanlab-skill/SKILL.md +163 -0
- package/training-mode/preset/skills/swanlab-skill/VENDOR.txt +9 -0
- package/training-mode/preset/skills/swanlab-skill/references/ANALYSIS_GUIDE.md +83 -0
- package/training-mode/preset/skills/swanlab-skill/references/CLI_REFERENCE.md +510 -0
- package/training-mode/preset/skills/swanlab-skill/references/SDK_QUICKSTART.md +341 -0
- package/training-mode/preset/skills/swanlab-skill/references/SWANLAB_CONCEPTS.md +523 -0
- package/training-mode/preset/skills/swanlab-skill/scripts/plot_metrics.py +353 -0
- package/training-mode/preset/skills/swanlab-skill/scripts/runs_benchmark.py +550 -0
- package/training-mode/preset/skills/training-research/SKILL.md +33 -0
- package/training-mode/preset/skills/training-research/references/pipeline.md +45 -0
- package/training-mode/preset/skills/training-research/references/research.md +76 -0
|
@@ -0,0 +1,523 @@
|
|
|
1
|
+
# SwanLab Concepts & Data Model
|
|
2
|
+
|
|
3
|
+
Reference for understanding SwanLab's entity hierarchy, terminology, and data model. Read this when interpreting user requests or deciding which CLI command to use.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Product Overview
|
|
8
|
+
|
|
9
|
+
SwanLab is an AI training experiment tracking platform. The Python SDK records metrics, configs, media, console output, and hardware stats during training. Data is persisted locally and optionally synced to the SwanLab cloud (or self-hosted instance).
|
|
10
|
+
|
|
11
|
+
The CLI (`swanlab api`) provides read-only query access to all tracked data from the command line.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Instances, Hosts & Credentials
|
|
16
|
+
|
|
17
|
+
SwanLab exists as multiple **independent instances**:
|
|
18
|
+
|
|
19
|
+
- **Public cloud**: `https://swanlab.cn` (the default)
|
|
20
|
+
- **Self-hosted instances**: private deployments on custom domains (e.g. `https://dev.example.com`)
|
|
21
|
+
|
|
22
|
+
**Credentials are per-instance.** An API key issued on one instance is rejected by every other instance (`401 Unauthorized`). Being logged in to instance A grants no access to instance B.
|
|
23
|
+
|
|
24
|
+
**A `404 Not_Found` from a project-scoped query (`project info`, `run list`, `run info`) almost always means you are querying the wrong host — not that the project is missing.** Before concluding a project doesn't exist (or paging through `project list` hunting for it), check the host reported by `swanlab verify` and re-issue the query against the correct instance with `--host` / `--api-key`.
|
|
25
|
+
|
|
26
|
+
To work against an instance other than the logged-in one (without re-login), pass per-command overrides:
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
swanlab api run list user/project --host https://swanlab.cn --api-key <KEY>
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
For a stretch of commands against a non-default host, export `SWANLAB_API_HOST` / `SWANLAB_API_KEY` (see Key Environment Variables) once instead of repeating the flags — this also keeps the key out of repeated terminal echo.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Core Entity Hierarchy
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
User (authenticated account)
|
|
40
|
+
+-- Workspace (personal or team namespace, identified by username)
|
|
41
|
+
| +-- Project (groups related experiments)
|
|
42
|
+
| | +-- Experiment / Run (a single training execution)
|
|
43
|
+
| | | +-- Config (input hyperparameters, mutable during the run via `swanlab.config`)
|
|
44
|
+
| | | +-- Column (a metric definition — e.g. "loss", "acc", "image")
|
|
45
|
+
| | | | +-- Scalar Metrics (time-series numeric data)
|
|
46
|
+
| | | | +-- Media Metrics (images, audio, video, molecules, etc.)
|
|
47
|
+
| | | +-- Console Logs (captured stdout/stderr output)
|
|
48
|
+
| | | +-- Profile (metadata: config, requirements, conda, hardware)
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### Entity Reference
|
|
52
|
+
|
|
53
|
+
| Entity | Identified by | Description |
|
|
54
|
+
| -------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
|
55
|
+
| **User** | implicit (logged-in) | The authenticated account. Has `username`, `name`, `bio`, `email`, etc. |
|
|
56
|
+
| **Workspace** | `username` | A personal or team namespace. Type is `PERSON` or `TEAM`. Every project belongs to a workspace. |
|
|
57
|
+
| **Project** | `username/project_name` (path) | Groups related experiments. Has visibility (`PUBLIC` / `PRIVATE`), description, labels. |
|
|
58
|
+
| **Experiment (Run)** | `username/project_name/run_id` (path) | The atomic unit — one execution of training code. Created by `swanlab.init()`. Has state, config, metrics, logs. |
|
|
59
|
+
| **Column** | `key` within an experiment | A metric definition. Belongs to either `CUSTOM` (user-defined) or `SYSTEM` (auto-collected). Has a data type. |
|
|
60
|
+
| **Metric** | queried via column key | The actual time-series data for a column. Three categories: Scalar, Media, Log. |
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## Key Terminology
|
|
65
|
+
|
|
66
|
+
### Experiment States
|
|
67
|
+
|
|
68
|
+
An experiment (run) is always in one of these states:
|
|
69
|
+
|
|
70
|
+
| State | Meaning |
|
|
71
|
+
| ---------- | --------------------------------------- |
|
|
72
|
+
| `RUNNING` | Currently executing |
|
|
73
|
+
| `FINISHED` | Completed successfully |
|
|
74
|
+
| `CRASHED` | Terminated due to an error |
|
|
75
|
+
| `ABORTED` | Manually stopped by the user |
|
|
76
|
+
| `OFFLINE` | Created in offline mode, not yet synced |
|
|
77
|
+
|
|
78
|
+
### Column Classes
|
|
79
|
+
|
|
80
|
+
| Class | Description |
|
|
81
|
+
| -------- | --------------------------------------------------------- |
|
|
82
|
+
| `CUSTOM` | User-defined metrics logged via `swanlab.log()` (default) |
|
|
83
|
+
| `SYSTEM` | Auto-collected system metrics (CPU, GPU, memory, etc.) |
|
|
84
|
+
|
|
85
|
+
> **SDK ≥ 0.9.0**: metric keys are discovered via `swanlab api run series`, whose `--class` option takes the lowercase values `custom` / `system` mapping to these classes. The `column` / `columns` commands (and the data-type filters below) are deprecated for multi-view experiments.
|
|
86
|
+
|
|
87
|
+
### Column Data Types
|
|
88
|
+
|
|
89
|
+
| Type | Category | Description |
|
|
90
|
+
| ---------- | -------- | ----------------------------------- |
|
|
91
|
+
| `FLOAT` | Scalar | Floating-point numeric values |
|
|
92
|
+
| `BOOLEAN` | Scalar | True/false values |
|
|
93
|
+
| `STRING` | Scalar | Text values |
|
|
94
|
+
| `IMAGE` | Media | Image files (PNG, JPG, etc.) |
|
|
95
|
+
| `AUDIO` | Media | Audio files |
|
|
96
|
+
| `VIDEO` | Media | Video files |
|
|
97
|
+
| `OBJECT3D` | Media | 3D point cloud data (JSON) |
|
|
98
|
+
| `MOLECULE` | Media | Biochemical molecule structures |
|
|
99
|
+
| `ECHARTS` | Media | Custom chart configurations (JS/TS) |
|
|
100
|
+
| `TABLE` | Media | Tabular data |
|
|
101
|
+
| `TEXT` | Media | Rich text content |
|
|
102
|
+
|
|
103
|
+
### Project Visibility
|
|
104
|
+
|
|
105
|
+
| Visibility | Description |
|
|
106
|
+
| ---------- | ---------------------------------------------- |
|
|
107
|
+
| `PRIVATE` | Only accessible to workspace members (default) |
|
|
108
|
+
| `PUBLIC` | Visible to anyone |
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Path Convention
|
|
113
|
+
|
|
114
|
+
The SwanLab API uses a hierarchical path format to address entities:
|
|
115
|
+
|
|
116
|
+
```
|
|
117
|
+
username → Workspace
|
|
118
|
+
username/project_name → Project
|
|
119
|
+
username/project_name/run_id → Experiment
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
- `username`: The workspace's unique identifier.
|
|
123
|
+
- `project_name`: Human-readable project name (1-100 chars, `0-9a-zA-Z-_.+`).
|
|
124
|
+
- `run_id`: A unique identifier for each experiment. Auto-generated as a short experiment_id/run_id by default, not acutally CUID, or user-supplied custom ID (1–512 chars, no `<>:"/\|?*#%` or control chars).
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Run Object Schema (`run list` / `run info`)
|
|
129
|
+
|
|
130
|
+
`run list` and `run info` return the same run-object shape — `run list` wraps it in pagination (`data.list[]` plus `data.total` / `data.page` / `data.size` / `data.pages`), `run info` returns a single object in `data`. **All field names are snake_case.**
|
|
131
|
+
|
|
132
|
+
| Field | Description |
|
|
133
|
+
| ----------------------------- | ------------------------------------------------------------------------------------- |
|
|
134
|
+
| `run_id` | Unique experiment identifier — use this in `username/project_name/run_id` paths |
|
|
135
|
+
| `name` | Display name. Free-text, **not necessarily unique** (several runs may share one name) |
|
|
136
|
+
| `state` | `RUNNING` / `FINISHED` / `CRASHED` / `ABORTED` / `OFFLINE` |
|
|
137
|
+
| `description` | Free-text description — often used for variant/ablation labels |
|
|
138
|
+
| `created_at` / `finished_at` | ISO 8601 UTC timestamps. The difference is the run's wall-clock duration |
|
|
139
|
+
| `created_at_ts` | `created_at` as Unix seconds |
|
|
140
|
+
| `group` / `job_type` | Experiment group / distributed job type (empty when unused) |
|
|
141
|
+
| `labels` | Tag list attached to the run |
|
|
142
|
+
| `show` | Whether the run is visible in the UI |
|
|
143
|
+
| `type` | Run type (e.g. `CHAPTER`) |
|
|
144
|
+
| `url` | Full web URL of the run's chart page |
|
|
145
|
+
| `user` | Owner object: `{username, name, avatar, status}` |
|
|
146
|
+
| `project_id` | Internal project id |
|
|
147
|
+
| `root_exp_id` / `root_pro_id` | Resume-chain ids (empty when the run was not resumed) |
|
|
148
|
+
| `profile` | Config + environment profile — see Experiment Profile below |
|
|
149
|
+
|
|
150
|
+
> **Note**: `run list` embeds the **full `profile`** (including the bulky `requirements` / `conda` strings) in every item, so list responses can be hundreds of KB. Extract only the fields you need (pipe to `jq` / `python`) instead of dumping raw output into context.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## Three Metric Categories
|
|
155
|
+
|
|
156
|
+
Understanding which metric type to query is critical for answering "where's my data?":
|
|
157
|
+
|
|
158
|
+
### Scalar Metrics (e.g. loss, accuracy)
|
|
159
|
+
|
|
160
|
+
Numeric time-series data logged at each training step.
|
|
161
|
+
|
|
162
|
+
Structure per data point:
|
|
163
|
+
|
|
164
|
+
```
|
|
165
|
+
{
|
|
166
|
+
"index": 0, // step number or custom x-axis value
|
|
167
|
+
"data": 0.523, // the metric value (can be int, float, or "NaN"/"INF"/"-INF")
|
|
168
|
+
"timestamp": 1714368000 // Unix timestamp
|
|
169
|
+
}
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Full response structure (SDK `Experiment.metrics()` / CLI `run metrics`; CLI wraps it in `{"ok", "errmsg", "data"}`):
|
|
173
|
+
|
|
174
|
+
```
|
|
175
|
+
{
|
|
176
|
+
"keys": ["loss", "acc"],
|
|
177
|
+
"metricType": "SCALAR",
|
|
178
|
+
"list": [
|
|
179
|
+
{
|
|
180
|
+
"key": "loss",
|
|
181
|
+
"metrics": [ { "index": 0, "data": 0.523, "timestamp": 1714368000 }, ... ],
|
|
182
|
+
"min": { "index": 12, "data": 0.498 }, // server-side statistics,
|
|
183
|
+
"max": { "index": 0, "data": 0.523 }, // each stat is a point object
|
|
184
|
+
"avg": { "index": 0, "data": 0.510 }, // whose "data" holds the stat value
|
|
185
|
+
"median": { "index": 0, "data": 0.509 },
|
|
186
|
+
"latest": { "index": 99, "data": 0.501 }
|
|
187
|
+
}
|
|
188
|
+
]
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
**Statistics** are computed server-side per scalar column: `min`, `max`, `avg`, `median`, `latest` — read `data.list[<entry>].<stat>.data` for a quick aggregate without parsing all points. `stdDev` is **not** included here; it is only available via `run summary` (see Scalar Summary below).
|
|
193
|
+
|
|
194
|
+
> **Tip**: When metric data is large, use `--save` to persist JSON to file, then visualize with `scripts/plot_metrics.py --data file.json -k loss`. For quick stats without full time-series, prefer `run summary`.
|
|
195
|
+
|
|
196
|
+
**Sampling**: By default, `--sample 1500` returns downsampled data for visualization. Use `--all` to fetch all data points without sampling limit.
|
|
197
|
+
|
|
198
|
+
**Range Query** (client-side CSV filtering): When you need only a subset of scalar data points, use range query options to download the full CSV and filter client-side:
|
|
199
|
+
|
|
200
|
+
| Option | Default | Description |
|
|
201
|
+
| --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
202
|
+
| `--range-type` | `step` | Filter axis: `step` (by step number) or `timestamp` (by Unix timestamp in **milliseconds**) |
|
|
203
|
+
| `--range-start` | none | Start value, inclusive (int ≥ 0) |
|
|
204
|
+
| `--range-end` | none | End value, inclusive (int ≥ 0) |
|
|
205
|
+
| `--range-head` | none | First N data points (int ≥ 1). Mutually exclusive with `--range-tail` |
|
|
206
|
+
| `--range-tail` | none | Last N data points (int ≥ 1). Mutually exclusive with `--range-head` |
|
|
207
|
+
| `--range-last` | none | Last N milliseconds of data (int > 0). Mutually exclusive with `--range-start`/`--range-end`. Can combine with `--range-head`/`--range-tail`. |
|
|
208
|
+
|
|
209
|
+
- `--range-head` and `--range-tail` are mutually exclusive, acted as post-sampling.
|
|
210
|
+
- `--range-last` is mutually exclusive with `--range-start`/`--range-end`.
|
|
211
|
+
- `--range-head`/`--range-tail` can be combined with `--range-last` or `--range-start`/`--range-end`.
|
|
212
|
+
- `--range-start` must be ≤ `--range-end`.
|
|
213
|
+
- When `--range-type timestamp` is used, rows missing a timestamp column are skipped.
|
|
214
|
+
- Range query bypasses the sampling API entirely — it streams and filters the CSV export directly. Statistics (min/max/avg/median/latest) come from a separate value-stats endpoint (fetched concurrently with the CSV export) and are **not** affected by the range filter.
|
|
215
|
+
|
|
216
|
+
### Scalar Summary
|
|
217
|
+
|
|
218
|
+
A lightweight aggregate view of scalar metrics, returning per-key statistics without the full time-series. Useful for comparing experiments at a glance.
|
|
219
|
+
|
|
220
|
+
```
|
|
221
|
+
Structure per key:
|
|
222
|
+
{
|
|
223
|
+
"step": 222, // last step number
|
|
224
|
+
"value": 0.523, // last step value
|
|
225
|
+
"minMax": [0.1, 0.9], // [min, max] across all steps
|
|
226
|
+
"min": {...}, // min scalar point
|
|
227
|
+
"max": {...}, // max scalar point
|
|
228
|
+
"avg": 0.45, // average across all steps
|
|
229
|
+
"median": 0.44, // median across all steps
|
|
230
|
+
"stdDev": 0.12 // standard deviation across all steps
|
|
231
|
+
}
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
- `step` and `value` are from the **last** step.
|
|
235
|
+
- `min`, `max`, `avg`, `median`, `stdDev` are aggregated across **all** steps.
|
|
236
|
+
- If a value is `NaN`, `Inf`, or `-Inf`, it is returned as a **string**.
|
|
237
|
+
- Omit `--keys` to query all scalar keys for the experiment.
|
|
238
|
+
|
|
239
|
+
### Media Metrics (e.g. images, audio)
|
|
240
|
+
|
|
241
|
+
File-based data stored per step. Each media item has a presigned download URL.
|
|
242
|
+
|
|
243
|
+
```
|
|
244
|
+
Structure:
|
|
245
|
+
{
|
|
246
|
+
"steps": [0, 1, 2], // available steps
|
|
247
|
+
"step": 0, // current step
|
|
248
|
+
"metrics": [
|
|
249
|
+
{
|
|
250
|
+
"index": 0,
|
|
251
|
+
"items": [
|
|
252
|
+
{ "url": "https://..." } // presigned download URL
|
|
253
|
+
]
|
|
254
|
+
}
|
|
255
|
+
]
|
|
256
|
+
}
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
**Step-based**: Media is retrieved one step at a time by default (`--step N`). Use `--all` to fetch all steps.
|
|
260
|
+
|
|
261
|
+
### Console Logs
|
|
262
|
+
|
|
263
|
+
Captured stdout/stderr output from the training process.
|
|
264
|
+
|
|
265
|
+
```
|
|
266
|
+
Structure per log entry:
|
|
267
|
+
{
|
|
268
|
+
"epoch": 0, // shard index (logs are paginated by shards)
|
|
269
|
+
"level": "INFO", // DEBUG | INFO | WARN | ERROR
|
|
270
|
+
"message": "Epoch 1/10",
|
|
271
|
+
"tag": "", // optional tag
|
|
272
|
+
"timestamp": "..." // ISO timestamp
|
|
273
|
+
}
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
**Shard-based pagination**: Logs are fetched by `--offset` (shard index), not page number. Increase offset to get later shards.
|
|
277
|
+
|
|
278
|
+
### Log Export
|
|
279
|
+
|
|
280
|
+
Console logs can be bulk-exported as a downloadable `.log` file via a presigned URL. This is distinct from shard-based log viewing — export retrieves raw log rows by row index range.
|
|
281
|
+
|
|
282
|
+
| Parameter | Default | Description |
|
|
283
|
+
| --------- | ------- | -------------------------------------- |
|
|
284
|
+
| `start` | 0 | Start row index (0-based) |
|
|
285
|
+
| `rows` | 500,000 | Number of rows to export (max 500,000) |
|
|
286
|
+
|
|
287
|
+
The API returns a presigned download URL (`{"url": "https://..."}`) that is valid for a limited time. No file content is downloaded by the CLI — only the URL is returned.
|
|
288
|
+
|
|
289
|
+
---
|
|
290
|
+
|
|
291
|
+
## Experiment Profile
|
|
292
|
+
|
|
293
|
+
Each experiment carries a `profile` object containing metadata about the run:
|
|
294
|
+
|
|
295
|
+
```
|
|
296
|
+
{
|
|
297
|
+
"config": { ... }, // Hyperparameters set at swanlab.init(config={...})
|
|
298
|
+
"metadata": { ... }, // Hardware & environment info (auto-collected)
|
|
299
|
+
"requirements": "...", // pip requirements string
|
|
300
|
+
"conda": "..." // conda environment YAML
|
|
301
|
+
}
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
- **Config** = user inputs. Initialized at `swanlab.init(config={...})`; mutable during the run — `swanlab.config["k"] = v` / `.update()` take effect immediately and are uploaded.
|
|
305
|
+
- **Metadata** = auto-collected system info (Python version, GPU model, OS, etc.).
|
|
306
|
+
- **Requirements / Conda** = captured from the active Python environment.
|
|
307
|
+
|
|
308
|
+
**Config entries are nested objects, not plain values.** Each config key maps to `{value, desc, sort}`:
|
|
309
|
+
|
|
310
|
+
```
|
|
311
|
+
{
|
|
312
|
+
"learning_rate": { "value": 0.0001, "desc": "", "sort": 0 },
|
|
313
|
+
"batch_size": { "value": 512, "desc": "", "sort": 1 },
|
|
314
|
+
...
|
|
315
|
+
}
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Always read hyperparameters as `profile.config.<key>.value`. When analyzing runs, **trust config values over run names** — names are free-text and can drift out of sync with the actual config.
|
|
319
|
+
|
|
320
|
+
**Metadata keys** (auto-collected): `os`, `cpu` (`{brand, cores}`), `gpu` (vendor-keyed, e.g. `nvidia: {type[], memory[], cores, cuda, driver}`), `memory`, `python`, `python_verbose`, `executable`, `command` (the training command line), `cwd`, `pid`, `hostname`, `git_info` (`[branch, commit]`), `git_remote`, `swanlab` (`{version, logdir, _monitor}`).
|
|
321
|
+
|
|
322
|
+
> Metadata key names vary with the SDK version that recorded the run — the list above matches what existing experiments typically carry. Newer SDKs collect the same information under different names, e.g. `python_version` / `python_executable` instead of `python` / `executable`, a single `git` object (`{remote_url, branch, commit}`) instead of `git_info` / `git_remote`, and `swanlab.run_dir` instead of `swanlab.logdir`.
|
|
323
|
+
|
|
324
|
+
---
|
|
325
|
+
|
|
326
|
+
## Self-Hosted Instance
|
|
327
|
+
|
|
328
|
+
> **Self-hosted commands are only available for self-hosted (private) deployments.** Only proceed with these commands when the resolved host points to a self-hosted deployment.
|
|
329
|
+
|
|
330
|
+
SwanLab can be deployed as a self-hosted instance (private deployment). **Use `swanlab api self-hosted` subcommands exclusively on self-hosted deployments. Always verify the host points to a private server before invoking these commands.**
|
|
331
|
+
|
|
332
|
+
**Host detection rule**: Before using any self-hosted command, check where requests will be sent:
|
|
333
|
+
|
|
334
|
+
1. If the user passes `--host` explicitly → use that value.
|
|
335
|
+
2. Otherwise, check `SWANLAB_API_HOST` / `SWANLAB_WEB_HOST` environment variables.
|
|
336
|
+
3. Otherwise, check `.netrc` or SwanLab config (`~/.swanlab`).
|
|
337
|
+
|
|
338
|
+
Self-hosted commands only succeed on self-hosted deployments. Always confirm the resolved host points to a private server before proceeding.
|
|
339
|
+
|
|
340
|
+
Self-hosted-specific features:
|
|
341
|
+
|
|
342
|
+
| Property | Description |
|
|
343
|
+
| --------- | ------------------------------------------ |
|
|
344
|
+
| `enabled` | Whether this is a self-hosted deployment |
|
|
345
|
+
| `expired` | Whether the license has expired |
|
|
346
|
+
| `root` | Whether the current user is a root (admin) |
|
|
347
|
+
| `plan` | License plan: `free` or `commercial` |
|
|
348
|
+
| `seats` | Number of allowed user seats |
|
|
349
|
+
|
|
350
|
+
Management operations (create user, list users, list projects, list workspaces, summary) all require root privileges. If a non-root user attempts any of these, the API returns a permission error.
|
|
351
|
+
|
|
352
|
+
### Instance-Level Management
|
|
353
|
+
|
|
354
|
+
On a self-hosted deployment, the root (admin) user can query across the entire instance, not just within a single workspace:
|
|
355
|
+
|
|
356
|
+
| Operation | Root Required | Description |
|
|
357
|
+
| --------------- | ------------- | --------------------------------------------------------------------------------------------- |
|
|
358
|
+
| Create User | Yes | Add a new user to the instance |
|
|
359
|
+
| List Users | Yes | All registered users across the instance |
|
|
360
|
+
| List Projects | Yes | All projects across all workspaces, with optional filters for creator, workspace, and keyword |
|
|
361
|
+
| List Workspaces | Yes | All workspaces (personal + team) on the instance, with optional keyword search |
|
|
362
|
+
| Usage Summary | Yes | Aggregate statistics: total users, projects, experiments, storage, etc. |
|
|
363
|
+
|
|
364
|
+
This differs from the regular `project list` / `workspace info` commands, which are scoped to a single workspace. Instance-level queries have a global view.
|
|
365
|
+
|
|
366
|
+
### Quick Disambiguation (Self-Hosted)
|
|
367
|
+
|
|
368
|
+
| User says... | They probably mean... | CLI command |
|
|
369
|
+
| ---------------------------- | ------------------------------- | ----------------------------------------- |
|
|
370
|
+
| "all projects on the server" | Instance-wide project listing | `swanlab api self-hosted list-projects` |
|
|
371
|
+
| "all workspaces" | Instance-wide workspace listing | `swanlab api self-hosted list-workspaces` |
|
|
372
|
+
| "server usage / disk usage" | Usage summary | `swanlab api self-hosted summary` |
|
|
373
|
+
| "all users on the server" | Instance-wide user listing | `swanlab api self-hosted list-users` |
|
|
374
|
+
|
|
375
|
+
---
|
|
376
|
+
|
|
377
|
+
---
|
|
378
|
+
|
|
379
|
+
## Filter Query (Experiment Filtering)
|
|
380
|
+
|
|
381
|
+
Experiments can be filtered by submitting a JSON array of filter objects to the runs/shows API. Each filter describes a condition on an experiment field.
|
|
382
|
+
|
|
383
|
+
### Filter Object Structure
|
|
384
|
+
|
|
385
|
+
Each filter object must contain four fields:
|
|
386
|
+
|
|
387
|
+
```json
|
|
388
|
+
{
|
|
389
|
+
"key": "state",
|
|
390
|
+
"type": "STABLE",
|
|
391
|
+
"op": "EQ",
|
|
392
|
+
"value": ["FINISHED"]
|
|
393
|
+
}
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
| Field | Type | Required | Description |
|
|
397
|
+
| ------- | ------ | -------- | --------------------------------------------- |
|
|
398
|
+
| `key` | string | yes | The field name to filter on |
|
|
399
|
+
| `type` | string | yes | Sidebar type: `STABLE`, `CONFIG`, or `SCALAR` |
|
|
400
|
+
| `op` | string | yes | Filter operator |
|
|
401
|
+
| `value` | array | yes | Values to compare against (always a list) |
|
|
402
|
+
|
|
403
|
+
### Filter Types (`type`)
|
|
404
|
+
|
|
405
|
+
| Type | Description | Valid Keys |
|
|
406
|
+
| -------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
407
|
+
| `STABLE` | Built-in experiment fields (see table below) | `state`, `name`, `description`, `show`, `pin`, `baseline`, `colors`, `cluster`, `job`, `createdAt`, `updatedAt`, `finishedAt`, `pinnedAt`, `labels` |
|
|
408
|
+
| `CONFIG` | Hyperparameters from `swanlab.init(config={...})` | Any key from the config dict, e.g. `learning_rate`, `batch_size` |
|
|
409
|
+
| `SCALAR` | Scalar metrics from `swanlab.log(...)` | Any metric key, e.g. `train/loss`, `val/acc` |
|
|
410
|
+
|
|
411
|
+
### STABLE Keys
|
|
412
|
+
|
|
413
|
+
| Key | Type | Description |
|
|
414
|
+
| ------------- | -------- | ------------------------------------------------------------------------ |
|
|
415
|
+
| `state` | enum | Experiment state: `RUNNING`, `FINISHED`, `CRASHED`, `ABORTED`, `OFFLINE` |
|
|
416
|
+
| `name` | string | Experiment display name |
|
|
417
|
+
| `description` | string | Experiment description text |
|
|
418
|
+
| `show` | boolean | Whether the experiment is visible in the UI |
|
|
419
|
+
| `pin` | boolean | Whether the experiment is pinned to the top |
|
|
420
|
+
| `baseline` | boolean | Whether this experiment is marked as a baseline for comparison |
|
|
421
|
+
| `colors` | string | Assigned color identifier |
|
|
422
|
+
| `cluster` | string | Experiment group/cluster name |
|
|
423
|
+
| `job` | string | Distributed job type (e.g. `worker`, `master`) |
|
|
424
|
+
| `createdAt` | datetime | Time the experiment was created (UTC+8 on filter, UTC in response) |
|
|
425
|
+
| `updatedAt` | datetime | Time the experiment was last updated (UTC+8 on filter, UTC in response) |
|
|
426
|
+
| `finishedAt` | datetime | Time the experiment finished (UTC+8 on filter, UTC in response) |
|
|
427
|
+
| `pinnedAt` | datetime | Time the experiment was pinned (UTC+8 on filter, UTC in response) |
|
|
428
|
+
| `labels` | array | Tags/labels attached to the experiment |
|
|
429
|
+
|
|
430
|
+
### Filter Operators (`op`)
|
|
431
|
+
|
|
432
|
+
| Operator | Meaning | Notes |
|
|
433
|
+
| --------- | --------------------- | ---------------------------------- |
|
|
434
|
+
| `EQ` | Equals | Single value |
|
|
435
|
+
| `NEQ` | Not equals | Single value |
|
|
436
|
+
| `GTE` | Greater than or equal | Numeric / date / string comparison |
|
|
437
|
+
| `LTE` | Less than or equal | Numeric / date / string comparison |
|
|
438
|
+
| `IN` | In list | Value is an array of options |
|
|
439
|
+
| `NOT IN` | Not in list | Value is an array of options |
|
|
440
|
+
| `CONTAIN` | Fuzzy contains | Substring / partial match |
|
|
441
|
+
|
|
442
|
+
### Time Fields & Range Queries
|
|
443
|
+
|
|
444
|
+
Time-type fields (`createdAt`, `updatedAt`, `finishedAt`, `pinnedAt`) are **range-queried** using `GTE` and `LTE` together to define a time interval.
|
|
445
|
+
|
|
446
|
+
**Important timezone note:**
|
|
447
|
+
|
|
448
|
+
- **Filter values** use **UTC+8** (Asia/Shanghai). Input times are interpreted as Beijing time.
|
|
449
|
+
- **Response timestamps** are in **UTC**. The returned experiment fields (`createdAt`, `finishedAt`, etc.) are UTC ISO strings.
|
|
450
|
+
|
|
451
|
+
To filter experiments within a time range, combine `GTE` and `LTE` filters on the same key:
|
|
452
|
+
|
|
453
|
+
```json
|
|
454
|
+
[
|
|
455
|
+
{ "key": "createdAt", "type": "STABLE", "op": "GTE", "value": ["2026-01-01T00:00:00"] },
|
|
456
|
+
{ "key": "createdAt", "type": "STABLE", "op": "LTE", "value": ["2026-04-29T23:59:59"] }
|
|
457
|
+
]
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
- **Value format**: ISO 8601 string (e.g. `"2026-01-01T00:00:00"`). The server interprets filter values as UTC+8 and compares as date objects.
|
|
461
|
+
- When reading results, remember the returned timestamps are UTC — they will be 8 hours behind the filter input times.
|
|
462
|
+
|
|
463
|
+
### Operator Constraints
|
|
464
|
+
|
|
465
|
+
- Array-type fields (e.g. `labels`) only support `EQ`, `NEQ`, `IN`, `NOT IN`, `CONTAIN`.
|
|
466
|
+
- Date-type fields use range queries with `GTE`/`LTE` (see above).
|
|
467
|
+
- Numeric fields prefer numeric comparison, falling back to string comparison.
|
|
468
|
+
|
|
469
|
+
### Examples
|
|
470
|
+
|
|
471
|
+
Filter by state:
|
|
472
|
+
|
|
473
|
+
```json
|
|
474
|
+
[{ "key": "state", "type": "STABLE", "op": "IN", "value": ["FINISHED", "RUNNING"] }]
|
|
475
|
+
```
|
|
476
|
+
|
|
477
|
+
Filter by config value:
|
|
478
|
+
|
|
479
|
+
```json
|
|
480
|
+
[{ "key": "learning_rate", "type": "CONFIG", "op": "EQ", "value": ["0.01"] }]
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
Filter by scalar metric:
|
|
484
|
+
|
|
485
|
+
```json
|
|
486
|
+
[{ "key": "train/loss", "type": "SCALAR", "op": "LTE", "value": ["0.5"] }]
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
Filter by time range (filter values in UTC+8, response in UTC):
|
|
490
|
+
|
|
491
|
+
```json
|
|
492
|
+
[{ "key": "finishedAt", "type": "STABLE", "op": "GTE", "value": ["2026-04-01T00:00:00"] }]
|
|
493
|
+
```
|
|
494
|
+
|
|
495
|
+
Multiple filters combined:
|
|
496
|
+
|
|
497
|
+
```json
|
|
498
|
+
[
|
|
499
|
+
{ "key": "state", "type": "STABLE", "op": "EQ", "value": ["FINISHED"] },
|
|
500
|
+
{ "key": "learning_rate", "type": "CONFIG", "op": "GTE", "value": ["0.001"] },
|
|
501
|
+
{ "key": "train/loss", "type": "SCALAR", "op": "LTE", "value": ["0.5"] }
|
|
502
|
+
]
|
|
503
|
+
```
|
|
504
|
+
|
|
505
|
+
---
|
|
506
|
+
|
|
507
|
+
## Key Environment Variables
|
|
508
|
+
|
|
509
|
+
| Variable | Purpose |
|
|
510
|
+
| ------------------ | ------------------------------------------------- |
|
|
511
|
+
| `SWANLAB_API_KEY` | API authentication key |
|
|
512
|
+
| `SWANLAB_API_HOST` | API server URL (default `https://api.swanlab.cn`) |
|
|
513
|
+
| `SWANLAB_WEB_HOST` | Web dashboard URL (default `https://swanlab.cn`) |
|
|
514
|
+
| `SWANLAB_LOG_DIR` | Custom local log directory (default `./swanlog`) |
|
|
515
|
+
| `SWANLAB_ROOT` | Root dir for SwanLab files (default `~/.swanlab`) |
|
|
516
|
+
|
|
517
|
+
> Legacy aliases still honored for backward compatibility: `SWANLAB_LOGDIR` → `SWANLAB_LOG_DIR`, `SWANLAB_SAVE_DIR` → `SWANLAB_ROOT`.
|
|
518
|
+
|
|
519
|
+
---
|
|
520
|
+
|
|
521
|
+
## Run Modes
|
|
522
|
+
|
|
523
|
+
See `SKILL.md > Run Modes` for the mode table. CONCEPTS focuses on data model, not SDK configuration.
|