cometx 3.6.3__tar.gz → 3.6.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {cometx-3.6.3 → cometx-3.6.5}/PKG-INFO +44 -5
- {cometx-3.6.3 → cometx-3.6.5}/README.md +43 -4
- {cometx-3.6.3 → cometx-3.6.5}/cometx/_version.py +1 -1
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/__init__.py +3 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_gpu_app.py +143 -1
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/copy.py +40 -4
- cometx-3.6.5/cometx/cli/migrate_users.py +393 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/PKG-INFO +44 -5
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/SOURCES.txt +1 -0
- {cometx-3.6.3 → cometx-3.6.5}/LICENSE +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/__init__.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/_typing.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/_ui.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/api.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/audio.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_app.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_gpu_report.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_optimizer_report.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_usage_report.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/config.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/copy_utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/count.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/delete_assets.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/download.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/list_command.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/log.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/rename_duplicates.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/reproduce.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/smoke_test.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/update.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/upload_optimizer_experiments.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/__init__.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/comet/__init__.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/comet/download_manager.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/neptune.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/wandb.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/generate_utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/magics.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/optimizer.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/panel_utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/__init__.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/dataset.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/pointcloud.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx/utils.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/dependency_links.txt +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/entry_points.txt +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/requires.txt +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/top_level.txt +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/pyproject.toml +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/setup.cfg +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/setup.py +0 -0
- {cometx-3.6.3 → cometx-3.6.5}/tests/testlib.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: cometx
|
|
3
|
-
Version: 3.6.
|
|
3
|
+
Version: 3.6.5
|
|
4
4
|
Summary: Python tools for Comet
|
|
5
5
|
Home-page: https://github.com/comet-ml/cometx/
|
|
6
6
|
Author: cometx development team
|
|
@@ -102,6 +102,7 @@ for ML management.
|
|
|
102
102
|
* [cometx download](#cometx-download)
|
|
103
103
|
* [cometx list](#cometx-list)
|
|
104
104
|
* [cometx log](#cometx-log)
|
|
105
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
105
106
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
106
107
|
* [cometx reproduce](#cometx-reproduce)
|
|
107
108
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -324,6 +325,7 @@ Not all combinations are possible:
|
|
|
324
325
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
325
326
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
326
327
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
328
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
327
329
|
|
|
328
330
|
### Using --path
|
|
329
331
|
|
|
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
|
|
|
385
387
|
|
|
386
388
|
For more information, `cometx log --help`
|
|
387
389
|
|
|
390
|
+
## cometx migrate-users
|
|
391
|
+
|
|
392
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
393
|
+
|
|
394
|
+
```
|
|
395
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
396
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
**Arguments:**
|
|
400
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
401
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
402
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
403
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
404
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
405
|
+
|
|
406
|
+
### Flags
|
|
407
|
+
|
|
408
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
409
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
410
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
411
|
+
|
|
412
|
+
### Examples
|
|
413
|
+
|
|
414
|
+
```bash
|
|
415
|
+
# Dry run — preview what would happen
|
|
416
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
417
|
+
|
|
418
|
+
# Execute the migration
|
|
419
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
420
|
+
|
|
421
|
+
# Use a local chargeback report file
|
|
422
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
423
|
+
|
|
424
|
+
# Self-hosted environments — provide explicit URLs
|
|
425
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
426
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
For more information, `cometx migrate-users --help`
|
|
430
|
+
|
|
388
431
|
## cometx rename-duplicates
|
|
389
432
|
|
|
390
433
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
535
578
|
**Examples:**
|
|
536
579
|
```
|
|
537
580
|
cometx admin chargeback-report
|
|
538
|
-
<<<<<<< Updated upstream
|
|
539
|
-
cometx admin usage-report
|
|
540
|
-
=======
|
|
541
581
|
cometx admin chargeback-report 2024-09
|
|
542
582
|
```
|
|
543
|
-
>>>>>>> Stashed changes
|
|
544
583
|
|
|
545
584
|
#### usage-report
|
|
546
585
|
|
|
@@ -46,6 +46,7 @@ for ML management.
|
|
|
46
46
|
* [cometx download](#cometx-download)
|
|
47
47
|
* [cometx list](#cometx-list)
|
|
48
48
|
* [cometx log](#cometx-log)
|
|
49
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
49
50
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
50
51
|
* [cometx reproduce](#cometx-reproduce)
|
|
51
52
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -268,6 +269,7 @@ Not all combinations are possible:
|
|
|
268
269
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
269
270
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
270
271
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
272
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
271
273
|
|
|
272
274
|
### Using --path
|
|
273
275
|
|
|
@@ -329,6 +331,47 @@ Where TYPE is one of the following names:
|
|
|
329
331
|
|
|
330
332
|
For more information, `cometx log --help`
|
|
331
333
|
|
|
334
|
+
## cometx migrate-users
|
|
335
|
+
|
|
336
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
337
|
+
|
|
338
|
+
```
|
|
339
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
340
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
**Arguments:**
|
|
344
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
345
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
346
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
347
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
348
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
349
|
+
|
|
350
|
+
### Flags
|
|
351
|
+
|
|
352
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
353
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
354
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
355
|
+
|
|
356
|
+
### Examples
|
|
357
|
+
|
|
358
|
+
```bash
|
|
359
|
+
# Dry run — preview what would happen
|
|
360
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
361
|
+
|
|
362
|
+
# Execute the migration
|
|
363
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
364
|
+
|
|
365
|
+
# Use a local chargeback report file
|
|
366
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
367
|
+
|
|
368
|
+
# Self-hosted environments — provide explicit URLs
|
|
369
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
370
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
For more information, `cometx migrate-users --help`
|
|
374
|
+
|
|
332
375
|
## cometx rename-duplicates
|
|
333
376
|
|
|
334
377
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -479,12 +522,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
479
522
|
**Examples:**
|
|
480
523
|
```
|
|
481
524
|
cometx admin chargeback-report
|
|
482
|
-
<<<<<<< Updated upstream
|
|
483
|
-
cometx admin usage-report
|
|
484
|
-
=======
|
|
485
525
|
cometx admin chargeback-report 2024-09
|
|
486
526
|
```
|
|
487
|
-
>>>>>>> Stashed changes
|
|
488
527
|
|
|
489
528
|
#### usage-report
|
|
490
529
|
|
|
@@ -26,6 +26,7 @@ Examples:
|
|
|
26
26
|
cometx smoke-test
|
|
27
27
|
cometx update
|
|
28
28
|
cometx admin
|
|
29
|
+
cometx migrate-users
|
|
29
30
|
|
|
30
31
|
For more information:
|
|
31
32
|
cometx COMMAND --help
|
|
@@ -47,6 +48,7 @@ from . import (
|
|
|
47
48
|
download,
|
|
48
49
|
list_command,
|
|
49
50
|
log,
|
|
51
|
+
migrate_users,
|
|
50
52
|
rename_duplicates,
|
|
51
53
|
reproduce,
|
|
52
54
|
smoke_test,
|
|
@@ -108,6 +110,7 @@ def main(raw_args=sys.argv[1:]):
|
|
|
108
110
|
add_subparser(subparsers, config, "config")
|
|
109
111
|
add_subparser(subparsers, rename_duplicates, "rename-duplicates")
|
|
110
112
|
add_subparser(subparsers, smoke_test, "smoke-test")
|
|
113
|
+
add_subparser(subparsers, migrate_users, "migrate-users")
|
|
111
114
|
add_subparser(
|
|
112
115
|
subparsers, upload_optimizer_experiments, "upload-optimizer-experiments"
|
|
113
116
|
)
|
|
@@ -119,6 +119,7 @@ def process_metrics_with_aggregation(
|
|
|
119
119
|
start_date: str = None,
|
|
120
120
|
end_date: str = None,
|
|
121
121
|
aggregate_metric: str = "avg",
|
|
122
|
+
selected_projects: List[str] = None,
|
|
122
123
|
) -> Tuple[Dict, Dict]:
|
|
123
124
|
"""
|
|
124
125
|
Process metric data with specified aggregation and time unit.
|
|
@@ -132,6 +133,7 @@ def process_metrics_with_aggregation(
|
|
|
132
133
|
start_date: Optional start date filter (YYYY-MM-DD)
|
|
133
134
|
end_date: Optional end date filter (YYYY-MM-DD)
|
|
134
135
|
aggregate_metric: Aggregation method for time series - "max" or "avg" (default: "avg")
|
|
136
|
+
selected_projects: Optional list of "workspace/project" strings to filter by
|
|
135
137
|
|
|
136
138
|
Returns:
|
|
137
139
|
tuple: (avg_data, time_series_data)
|
|
@@ -165,6 +167,15 @@ def process_metrics_with_aggregation(
|
|
|
165
167
|
continue
|
|
166
168
|
|
|
167
169
|
exp_info = experiment_map[exp_key]
|
|
170
|
+
|
|
171
|
+
# Apply project filter
|
|
172
|
+
if selected_projects is not None:
|
|
173
|
+
ws = exp_info.get("workspace", "Unknown")
|
|
174
|
+
proj = exp_info.get("project_name", "Unknown")
|
|
175
|
+
project_key = f"{ws}/{proj}"
|
|
176
|
+
if project_key not in selected_projects:
|
|
177
|
+
continue
|
|
178
|
+
|
|
168
179
|
server_timestamp = exp_info.get("server_timestamp")
|
|
169
180
|
|
|
170
181
|
# Apply date filter
|
|
@@ -271,13 +282,19 @@ def create_aggregation_avg_chart(
|
|
|
271
282
|
keys = [k for k, _ in sorted_keys]
|
|
272
283
|
values = [v for _, v in sorted_keys]
|
|
273
284
|
|
|
285
|
+
# Assign colors by alphabetical order to match the time series line chart
|
|
286
|
+
all_keys_sorted = sorted(data.keys())
|
|
287
|
+
colors_list = get_distinct_colors(len(all_keys_sorted))
|
|
288
|
+
color_map = {k: colors_list[i] for i, k in enumerate(all_keys_sorted)}
|
|
289
|
+
bar_colors = [color_map[k] for k in keys]
|
|
290
|
+
|
|
274
291
|
# Create bar chart
|
|
275
292
|
fig, ax = plt.subplots(figsize=(14, 8))
|
|
276
293
|
|
|
277
294
|
ax.bar(
|
|
278
295
|
range(len(keys)),
|
|
279
296
|
values,
|
|
280
|
-
color=
|
|
297
|
+
color=bar_colors,
|
|
281
298
|
edgecolor="navy",
|
|
282
299
|
alpha=0.7,
|
|
283
300
|
width=0.8,
|
|
@@ -449,6 +466,21 @@ def create_time_series_chart(
|
|
|
449
466
|
return fig
|
|
450
467
|
|
|
451
468
|
|
|
469
|
+
def get_available_workspaces_and_projects(data: Dict) -> Dict[str, List[str]]:
|
|
470
|
+
"""Extract unique workspaces and their projects from experiment_map.
|
|
471
|
+
|
|
472
|
+
Returns:
|
|
473
|
+
Dict mapping workspace names to sorted list of project names.
|
|
474
|
+
"""
|
|
475
|
+
experiment_map = data.get("experiment_map", {})
|
|
476
|
+
workspace_projects = defaultdict(set)
|
|
477
|
+
for exp_info in experiment_map.values():
|
|
478
|
+
ws = exp_info.get("workspace", "Unknown")
|
|
479
|
+
proj = exp_info.get("project_name", "Unknown")
|
|
480
|
+
workspace_projects[ws].add(proj)
|
|
481
|
+
return {ws: sorted(list(projects)) for ws, projects in sorted(workspace_projects.items())}
|
|
482
|
+
|
|
483
|
+
|
|
452
484
|
def get_available_dates(data: Dict) -> Tuple[List[str], str, str]:
|
|
453
485
|
"""Extract available date range from the data."""
|
|
454
486
|
experiment_map = data.get("experiment_map", {})
|
|
@@ -496,6 +528,19 @@ def get_available_dates(data: Dict) -> Tuple[List[str], str, str]:
|
|
|
496
528
|
|
|
497
529
|
def main():
|
|
498
530
|
"""Main Streamlit app entry point."""
|
|
531
|
+
# Set theme primary color to blue (overrides default red/coral)
|
|
532
|
+
# This must be done before any Streamlit elements are rendered
|
|
533
|
+
try:
|
|
534
|
+
import streamlit.config as _config
|
|
535
|
+
_config.set_option("theme.primaryColor", "#1f77b4")
|
|
536
|
+
except Exception:
|
|
537
|
+
pass # Fallback gracefully if internal API changes
|
|
538
|
+
|
|
539
|
+
# Force rerun on first load to ensure theme takes effect
|
|
540
|
+
if "theme_initialized" not in st.session_state:
|
|
541
|
+
st.session_state.theme_initialized = True
|
|
542
|
+
st.rerun()
|
|
543
|
+
|
|
499
544
|
st.set_page_config(
|
|
500
545
|
page_title="Comet GPU Report",
|
|
501
546
|
page_icon="🖥️",
|
|
@@ -552,11 +597,107 @@ def main():
|
|
|
552
597
|
# Get available dates
|
|
553
598
|
available_dates, min_date, max_date = get_available_dates(data)
|
|
554
599
|
|
|
600
|
+
# Get available workspaces and projects for filtering
|
|
601
|
+
workspace_projects_map = get_available_workspaces_and_projects(data)
|
|
602
|
+
all_workspaces = list(workspace_projects_map.keys())
|
|
603
|
+
|
|
604
|
+
# Custom CSS for professional styling (override Streamlit's default red/pink accent)
|
|
605
|
+
st.markdown("""
|
|
606
|
+
<style>
|
|
607
|
+
/* Override checkbox checked color - target the role=checkbox div */
|
|
608
|
+
div[role="checkbox"][aria-checked="true"] {
|
|
609
|
+
background-color: #1f77b4 !important;
|
|
610
|
+
border-color: #1f77b4 !important;
|
|
611
|
+
}
|
|
612
|
+
/* Also target via data-baseweb attribute */
|
|
613
|
+
[data-baseweb="checkbox"] div[aria-checked="true"] {
|
|
614
|
+
background-color: #1f77b4 !important;
|
|
615
|
+
border-color: #1f77b4 !important;
|
|
616
|
+
}
|
|
617
|
+
/* Style multi-select pills */
|
|
618
|
+
span[data-baseweb="tag"] {
|
|
619
|
+
background-color: #1f77b4 !important;
|
|
620
|
+
}
|
|
621
|
+
span[data-baseweb="tag"] span {
|
|
622
|
+
color: white !important;
|
|
623
|
+
}
|
|
624
|
+
</style>
|
|
625
|
+
""", unsafe_allow_html=True)
|
|
626
|
+
|
|
555
627
|
# Sidebar configuration
|
|
556
628
|
with st.sidebar:
|
|
557
629
|
st.divider()
|
|
558
630
|
st.header("Configuration")
|
|
559
631
|
|
|
632
|
+
# Workspace/Project Filter Section
|
|
633
|
+
st.subheader("Data Filter")
|
|
634
|
+
|
|
635
|
+
# Get all available projects across all workspaces
|
|
636
|
+
all_projects = []
|
|
637
|
+
for ws in all_workspaces:
|
|
638
|
+
for proj in workspace_projects_map.get(ws, []):
|
|
639
|
+
all_projects.append(f"{ws}/{proj}")
|
|
640
|
+
all_projects = sorted(all_projects)
|
|
641
|
+
|
|
642
|
+
# Initialize session state for project selection
|
|
643
|
+
# Use individual checkbox keys as the source of truth
|
|
644
|
+
if "projects_initialized" not in st.session_state:
|
|
645
|
+
for proj_path in all_projects:
|
|
646
|
+
st.session_state[f"proj_{proj_path}"] = True
|
|
647
|
+
st.session_state.projects_initialized = True
|
|
648
|
+
|
|
649
|
+
# Track whether the expander should stay open
|
|
650
|
+
if "projects_expander_open" not in st.session_state:
|
|
651
|
+
st.session_state.projects_expander_open = True
|
|
652
|
+
|
|
653
|
+
# Select All / Clear All buttons
|
|
654
|
+
btn_col1, btn_col2 = st.columns(2)
|
|
655
|
+
with btn_col1:
|
|
656
|
+
if st.button("Select All", use_container_width=True):
|
|
657
|
+
for proj_path in all_projects:
|
|
658
|
+
st.session_state[f"proj_{proj_path}"] = True
|
|
659
|
+
st.session_state.projects_expander_open = True
|
|
660
|
+
st.rerun()
|
|
661
|
+
with btn_col2:
|
|
662
|
+
if st.button("Clear All", use_container_width=True):
|
|
663
|
+
for proj_path in all_projects:
|
|
664
|
+
st.session_state[f"proj_{proj_path}"] = False
|
|
665
|
+
st.session_state.projects_expander_open = True
|
|
666
|
+
st.rerun()
|
|
667
|
+
|
|
668
|
+
# Expandable project selector for fine-grained control
|
|
669
|
+
with st.expander("Select specific projects", expanded=st.session_state.projects_expander_open):
|
|
670
|
+
# Group projects by workspace for cleaner display
|
|
671
|
+
for ws in all_workspaces:
|
|
672
|
+
ws_projects = [f"{ws}/{p}" for p in workspace_projects_map.get(ws, [])]
|
|
673
|
+
if not ws_projects:
|
|
674
|
+
continue
|
|
675
|
+
|
|
676
|
+
st.markdown(f"**{ws}**")
|
|
677
|
+
|
|
678
|
+
for proj_path in sorted(ws_projects):
|
|
679
|
+
proj_name = proj_path.split("/")[-1]
|
|
680
|
+
st.checkbox(
|
|
681
|
+
proj_name,
|
|
682
|
+
key=f"proj_{proj_path}",
|
|
683
|
+
)
|
|
684
|
+
|
|
685
|
+
# Build selected projects list from checkbox states
|
|
686
|
+
selected_projects_set = set()
|
|
687
|
+
for proj_path in all_projects:
|
|
688
|
+
if st.session_state.get(f"proj_{proj_path}", True):
|
|
689
|
+
selected_projects_set.add(proj_path)
|
|
690
|
+
|
|
691
|
+
# Show project count
|
|
692
|
+
num_selected = len(selected_projects_set)
|
|
693
|
+
num_total = len(all_projects)
|
|
694
|
+
st.caption(f"**{num_selected}** of **{num_total}** projects selected")
|
|
695
|
+
|
|
696
|
+
# Convert set to list for processing
|
|
697
|
+
selected_projects = sorted(selected_projects_set)
|
|
698
|
+
|
|
699
|
+
st.divider()
|
|
700
|
+
|
|
560
701
|
# Aggregation selection
|
|
561
702
|
st.subheader("Aggregation")
|
|
562
703
|
aggregation = st.selectbox(
|
|
@@ -640,6 +781,7 @@ def main():
|
|
|
640
781
|
start_date_str,
|
|
641
782
|
end_date_str,
|
|
642
783
|
aggregate_metric=aggregate_metric,
|
|
784
|
+
selected_projects=selected_projects,
|
|
643
785
|
)
|
|
644
786
|
|
|
645
787
|
# Display statistics
|
|
@@ -78,6 +78,8 @@ import threading
|
|
|
78
78
|
import time
|
|
79
79
|
import urllib.parse
|
|
80
80
|
import zipfile
|
|
81
|
+
|
|
82
|
+
import requests
|
|
81
83
|
from datetime import datetime, timedelta
|
|
82
84
|
|
|
83
85
|
from comet_ml import APIExperiment, Artifact, Experiment, OfflineExperiment
|
|
@@ -102,6 +104,7 @@ from ..api import API
|
|
|
102
104
|
from ..framework.comet.download_manager import sanitize_filename
|
|
103
105
|
from ..utils import remove_extra_slashes
|
|
104
106
|
from .copy_utils import upload_single_offline_experiment
|
|
107
|
+
from .migrate_users import _create_workspace
|
|
105
108
|
|
|
106
109
|
ADDITIONAL_ARGS = False
|
|
107
110
|
|
|
@@ -197,6 +200,12 @@ def get_parser_arguments(parser):
|
|
|
197
200
|
type=str,
|
|
198
201
|
default=None,
|
|
199
202
|
)
|
|
203
|
+
parser.add_argument(
|
|
204
|
+
"--create-workspaces",
|
|
205
|
+
help="Attempt to create the destination workspace if it does not exist",
|
|
206
|
+
default=False,
|
|
207
|
+
action="store_true",
|
|
208
|
+
)
|
|
200
209
|
|
|
201
210
|
|
|
202
211
|
def copy(parsed_args, remaining=None):
|
|
@@ -214,6 +223,7 @@ def copy(parsed_args, remaining=None):
|
|
|
214
223
|
parsed_args.ignore,
|
|
215
224
|
parsed_args.debug,
|
|
216
225
|
parsed_args.sync,
|
|
226
|
+
parsed_args.create_workspaces,
|
|
217
227
|
)
|
|
218
228
|
|
|
219
229
|
# Wait for all uploads to complete
|
|
@@ -790,7 +800,7 @@ class CopyManager:
|
|
|
790
800
|
# Restore default signal handler
|
|
791
801
|
signal.signal(signal.SIGINT, signal.default_int_handler)
|
|
792
802
|
|
|
793
|
-
def copy(self, source, destination, symlink, ignore, debug, sync):
|
|
803
|
+
def copy(self, source, destination, symlink, ignore, debug, sync, create_workspaces=False):
|
|
794
804
|
""" """
|
|
795
805
|
self.ignore = ignore
|
|
796
806
|
self.debug = debug
|
|
@@ -823,9 +833,35 @@ class CopyManager:
|
|
|
823
833
|
# First check to make sure workspace_dst exists:
|
|
824
834
|
workspaces = self.api.get_workspaces()
|
|
825
835
|
if workspace_dst not in workspaces:
|
|
826
|
-
|
|
827
|
-
|
|
828
|
-
|
|
836
|
+
if create_workspaces:
|
|
837
|
+
print(
|
|
838
|
+
f"Workspace {workspace_dst!r} does not exist, attempting to create it..."
|
|
839
|
+
)
|
|
840
|
+
dest_url = self.api._get_url_server()
|
|
841
|
+
headers = {
|
|
842
|
+
"Authorization": self.api.api_key,
|
|
843
|
+
"Content-Type": "application/json",
|
|
844
|
+
}
|
|
845
|
+
try:
|
|
846
|
+
_create_workspace(dest_url, headers, workspace_dst)
|
|
847
|
+
print(f"Workspace {workspace_dst!r} created successfully.")
|
|
848
|
+
except requests.exceptions.HTTPError as exc:
|
|
849
|
+
raise Exception(
|
|
850
|
+
f"Workspace {workspace_dst!r} does not exist and could not be "
|
|
851
|
+
f"created automatically (HTTP {exc.response.status_code}). "
|
|
852
|
+
f"Please create it via the Comet UI and try again."
|
|
853
|
+
) from exc
|
|
854
|
+
except requests.exceptions.RequestException as exc:
|
|
855
|
+
raise Exception(
|
|
856
|
+
f"Workspace {workspace_dst!r} does not exist and could not be "
|
|
857
|
+
f"created automatically: {exc}. "
|
|
858
|
+
f"Please create it via the Comet UI and try again."
|
|
859
|
+
) from exc
|
|
860
|
+
else:
|
|
861
|
+
raise Exception(
|
|
862
|
+
f"{workspace_dst} does not exist; use --create-workspaces to "
|
|
863
|
+
f"create it automatically, or create it via the Comet UI"
|
|
864
|
+
)
|
|
829
865
|
|
|
830
866
|
if project_src == "panels":
|
|
831
867
|
# experiment_src may be "*" or filename
|
|
@@ -0,0 +1,393 @@
|
|
|
1
|
+
#!/usr/bin/env python
|
|
2
|
+
# -*- coding: utf-8 -*-
|
|
3
|
+
# ****************************************
|
|
4
|
+
# __
|
|
5
|
+
# _________ ____ ___ ___ / /__ __
|
|
6
|
+
# / ___/ __ \/ __ `__ \/ _ \/ __/ |/_/
|
|
7
|
+
# / /__/ /_/ / / / / / / __/ /__> <
|
|
8
|
+
# \___/\____/_/ /_/ /_/\___/\__/_/|_|
|
|
9
|
+
#
|
|
10
|
+
#
|
|
11
|
+
# Copyright (c) 2022 Cometx Development
|
|
12
|
+
# Team. All rights reserved.
|
|
13
|
+
# ****************************************
|
|
14
|
+
"""
|
|
15
|
+
Migrate users into workspaces from a source environment to a destination
|
|
16
|
+
environment based on a chargeback report.
|
|
17
|
+
|
|
18
|
+
The chargeback report is fetched automatically from the source environment's
|
|
19
|
+
admin API, or you can provide a local JSON file with --chargeback-report.
|
|
20
|
+
|
|
21
|
+
Examples:
|
|
22
|
+
|
|
23
|
+
$ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
24
|
+
$ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
25
|
+
$ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json
|
|
26
|
+
"""
|
|
27
|
+
|
|
28
|
+
import argparse
|
|
29
|
+
import base64
|
|
30
|
+
import json
|
|
31
|
+
import os
|
|
32
|
+
import sys
|
|
33
|
+
import urllib.parse
|
|
34
|
+
|
|
35
|
+
import requests
|
|
36
|
+
|
|
37
|
+
ADDITIONAL_ARGS = False
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def get_parser_arguments(parser):
|
|
41
|
+
parser.add_argument(
|
|
42
|
+
"--api-key",
|
|
43
|
+
help="API key for the destination environment (used to add workspace members). "
|
|
44
|
+
"Falls back to COMET_API_KEY environment variable if not provided.",
|
|
45
|
+
type=str,
|
|
46
|
+
default=None,
|
|
47
|
+
)
|
|
48
|
+
parser.add_argument(
|
|
49
|
+
"--url",
|
|
50
|
+
help="Base URL of the destination Comet environment (e.g. https://comet.example.com). "
|
|
51
|
+
"Required for self-hosted instances when the API key does not encode the server URL.",
|
|
52
|
+
type=str,
|
|
53
|
+
default=None,
|
|
54
|
+
)
|
|
55
|
+
parser.add_argument(
|
|
56
|
+
"--source-api-key",
|
|
57
|
+
help="API key for the source environment (used to fetch the chargeback report). "
|
|
58
|
+
"Required unless --chargeback-report is given.",
|
|
59
|
+
type=str,
|
|
60
|
+
default=None,
|
|
61
|
+
)
|
|
62
|
+
parser.add_argument(
|
|
63
|
+
"--source-url",
|
|
64
|
+
help="Base URL of the source Comet environment. "
|
|
65
|
+
"Required for self-hosted instances when the source API key does not encode the server URL.",
|
|
66
|
+
type=str,
|
|
67
|
+
default=None,
|
|
68
|
+
)
|
|
69
|
+
parser.add_argument(
|
|
70
|
+
"--chargeback-report",
|
|
71
|
+
help="Path to a local chargeback report JSON file (skips fetching from source environment)",
|
|
72
|
+
type=str,
|
|
73
|
+
default=None,
|
|
74
|
+
)
|
|
75
|
+
parser.add_argument(
|
|
76
|
+
"--create-workspaces",
|
|
77
|
+
help="Create workspaces on the destination environment if they don't already exist",
|
|
78
|
+
default=False,
|
|
79
|
+
action="store_true",
|
|
80
|
+
)
|
|
81
|
+
parser.add_argument(
|
|
82
|
+
"--dry-run",
|
|
83
|
+
help="Print what would happen without making any changes",
|
|
84
|
+
default=False,
|
|
85
|
+
action="store_true",
|
|
86
|
+
)
|
|
87
|
+
parser.add_argument(
|
|
88
|
+
"--failures-output",
|
|
89
|
+
help="Path to write failed operations JSON (default: bulk_add_failures_by_email.json)",
|
|
90
|
+
type=str,
|
|
91
|
+
default="bulk_add_failures_by_email.json",
|
|
92
|
+
)
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
def _resolve_server_url(api_key, explicit_url=None):
|
|
96
|
+
"""Return the server base URL.
|
|
97
|
+
|
|
98
|
+
Priority:
|
|
99
|
+
1. ``explicit_url`` (from --url / --source-url), if provided.
|
|
100
|
+
2. URL encoded inside the API key (new-style keys contain ``*<base64>``).
|
|
101
|
+
3. Error — no silent fallback to cloud URL.
|
|
102
|
+
"""
|
|
103
|
+
if explicit_url:
|
|
104
|
+
parsed = urllib.parse.urlparse(explicit_url)
|
|
105
|
+
if parsed.scheme != "https":
|
|
106
|
+
print("[ERROR] --url/--source-url must use https://.")
|
|
107
|
+
sys.exit(1)
|
|
108
|
+
return explicit_url.rstrip("/")
|
|
109
|
+
|
|
110
|
+
if "*" in api_key:
|
|
111
|
+
try:
|
|
112
|
+
_, encoded = api_key.split("*", 1)
|
|
113
|
+
# base64 padding may be missing
|
|
114
|
+
padding = (4 - len(encoded) % 4) % 4
|
|
115
|
+
payload = json.loads(base64.b64decode(encoded + "=" * padding))
|
|
116
|
+
return payload["baseUrl"].rstrip("/")
|
|
117
|
+
except Exception:
|
|
118
|
+
pass # Fall through to error below
|
|
119
|
+
|
|
120
|
+
print(
|
|
121
|
+
"[ERROR] Cannot determine the server URL from the API key. "
|
|
122
|
+
"Pass --url (destination) or --source-url (source) explicitly, "
|
|
123
|
+
"e.g. --url https://comet.example.com"
|
|
124
|
+
)
|
|
125
|
+
sys.exit(1)
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
def _fetch_chargeback_report(server_url, source_api_key):
|
|
129
|
+
url = f"{server_url}/api/admin/chargeback/report"
|
|
130
|
+
print(f"Fetching chargeback report from {url}...")
|
|
131
|
+
resp = requests.get(
|
|
132
|
+
url,
|
|
133
|
+
headers={"Authorization": source_api_key},
|
|
134
|
+
timeout=30,
|
|
135
|
+
)
|
|
136
|
+
resp.raise_for_status()
|
|
137
|
+
return resp.json()
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
def _load_chargeback_report(path):
|
|
141
|
+
print(f"Loading chargeback report from {path}...")
|
|
142
|
+
with open(path, "r") as f:
|
|
143
|
+
return json.load(f)
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
def _get_existing_workspaces(dest_url, headers):
|
|
147
|
+
url = f"{dest_url}/api/rest/v2/workspaces"
|
|
148
|
+
resp = requests.get(url, headers=headers, timeout=15)
|
|
149
|
+
resp.raise_for_status()
|
|
150
|
+
data = resp.json()
|
|
151
|
+
# Handle {"workspaceNames": [...]} dict shape
|
|
152
|
+
if isinstance(data, dict):
|
|
153
|
+
names = data.get("workspaceNames", [])
|
|
154
|
+
if names and isinstance(names[0], dict):
|
|
155
|
+
return {ws["name"] for ws in names}
|
|
156
|
+
return set(names)
|
|
157
|
+
# Handle list of dicts with a "name" key
|
|
158
|
+
if data and isinstance(data[0], dict):
|
|
159
|
+
return {ws["name"] for ws in data}
|
|
160
|
+
# Handle flat list of strings
|
|
161
|
+
return set(data)
|
|
162
|
+
|
|
163
|
+
|
|
164
|
+
def _create_workspace(dest_url, headers, workspace_name):
|
|
165
|
+
url = f"{dest_url}/api/rest/v2/write/workspace/new"
|
|
166
|
+
resp = requests.post(
|
|
167
|
+
url,
|
|
168
|
+
headers=headers,
|
|
169
|
+
json={"name": workspace_name},
|
|
170
|
+
timeout=15,
|
|
171
|
+
)
|
|
172
|
+
resp.raise_for_status()
|
|
173
|
+
|
|
174
|
+
|
|
175
|
+
def _add_member(url, headers, email, workspace_name):
|
|
176
|
+
"""Attempt to add a single member. Returns (status, error_info_or_None)."""
|
|
177
|
+
payload = {
|
|
178
|
+
"userEmail": email,
|
|
179
|
+
"workspaceName": workspace_name,
|
|
180
|
+
"admin": True,
|
|
181
|
+
}
|
|
182
|
+
try:
|
|
183
|
+
response = requests.post(
|
|
184
|
+
url,
|
|
185
|
+
headers=headers,
|
|
186
|
+
json=payload,
|
|
187
|
+
timeout=15,
|
|
188
|
+
)
|
|
189
|
+
|
|
190
|
+
if response.status_code == 200:
|
|
191
|
+
return "added", None
|
|
192
|
+
|
|
193
|
+
try:
|
|
194
|
+
error_msg = response.json()
|
|
195
|
+
except ValueError:
|
|
196
|
+
error_msg = response.text
|
|
197
|
+
|
|
198
|
+
is_already_member = (
|
|
199
|
+
isinstance(error_msg, dict)
|
|
200
|
+
and "already member of" in error_msg.get("msg", "")
|
|
201
|
+
)
|
|
202
|
+
if is_already_member:
|
|
203
|
+
return "already_member", None
|
|
204
|
+
|
|
205
|
+
return "failed", {
|
|
206
|
+
"status": response.status_code,
|
|
207
|
+
"error": error_msg,
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
except requests.exceptions.RequestException as e:
|
|
211
|
+
return "failed", {
|
|
212
|
+
"status": "exception",
|
|
213
|
+
"error": str(e),
|
|
214
|
+
}
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
def migrate_users(parsed_args):
|
|
218
|
+
api_key = parsed_args.api_key or os.environ.get("COMET_API_KEY")
|
|
219
|
+
if not api_key:
|
|
220
|
+
print("[ERROR] No API key found. Set COMET_API_KEY or pass --api-key.")
|
|
221
|
+
sys.exit(1)
|
|
222
|
+
|
|
223
|
+
source_api_key = parsed_args.source_api_key
|
|
224
|
+
create_workspaces = parsed_args.create_workspaces
|
|
225
|
+
dry_run = parsed_args.dry_run
|
|
226
|
+
failures_output = parsed_args.failures_output
|
|
227
|
+
|
|
228
|
+
if not parsed_args.chargeback_report and not source_api_key:
|
|
229
|
+
print(
|
|
230
|
+
"[ERROR] --source-api-key is required when --chargeback-report is not provided."
|
|
231
|
+
)
|
|
232
|
+
sys.exit(1)
|
|
233
|
+
|
|
234
|
+
dest_url = _resolve_server_url(api_key, parsed_args.url)
|
|
235
|
+
print(f"Destination URL: {dest_url}")
|
|
236
|
+
|
|
237
|
+
if parsed_args.chargeback_report:
|
|
238
|
+
data = _load_chargeback_report(parsed_args.chargeback_report)
|
|
239
|
+
else:
|
|
240
|
+
source_url = _resolve_server_url(source_api_key, parsed_args.source_url)
|
|
241
|
+
print(f"Source URL: {source_url}")
|
|
242
|
+
if source_url == dest_url and source_api_key == api_key:
|
|
243
|
+
print(
|
|
244
|
+
"[WARNING] Source and destination URL and API key are identical. "
|
|
245
|
+
"Are you sure you want to migrate users to the same environment?"
|
|
246
|
+
)
|
|
247
|
+
try:
|
|
248
|
+
data = _fetch_chargeback_report(source_url, source_api_key)
|
|
249
|
+
except requests.exceptions.RequestException as e:
|
|
250
|
+
print(f"[ERROR] Failed to fetch chargeback report: {e}")
|
|
251
|
+
sys.exit(1)
|
|
252
|
+
|
|
253
|
+
workspaces = data.get("workspaces", [])
|
|
254
|
+
print(f"Found {len(workspaces)} workspaces\n")
|
|
255
|
+
|
|
256
|
+
headers = {
|
|
257
|
+
"Authorization": api_key,
|
|
258
|
+
"Content-Type": "application/json",
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
if create_workspaces and not dry_run:
|
|
262
|
+
print("Checking existing workspaces on destination...")
|
|
263
|
+
try:
|
|
264
|
+
existing = _get_existing_workspaces(dest_url, headers)
|
|
265
|
+
except requests.exceptions.RequestException as e:
|
|
266
|
+
print(f"[ERROR] Failed to list destination workspaces: {e}")
|
|
267
|
+
sys.exit(1)
|
|
268
|
+
|
|
269
|
+
ws_created = 0
|
|
270
|
+
for workspace in workspaces:
|
|
271
|
+
ws_name = workspace["name"]
|
|
272
|
+
if ws_name not in existing:
|
|
273
|
+
try:
|
|
274
|
+
_create_workspace(dest_url, headers, ws_name)
|
|
275
|
+
print(f" [CREATED] Workspace '{ws_name}'")
|
|
276
|
+
ws_created += 1
|
|
277
|
+
except requests.exceptions.RequestException as e:
|
|
278
|
+
print(f" [ERROR] Failed to create workspace '{ws_name}': {e}")
|
|
279
|
+
print(f"Created {ws_created} new workspace(s)\n")
|
|
280
|
+
elif create_workspaces and dry_run:
|
|
281
|
+
print("[DRY RUN] Would check/create workspaces on destination\n")
|
|
282
|
+
|
|
283
|
+
add_member_url = f"{dest_url}/api/rest/v2/write/add-workspace-member"
|
|
284
|
+
|
|
285
|
+
total_added = 0
|
|
286
|
+
total_skipped = 0
|
|
287
|
+
total_no_email = 0
|
|
288
|
+
total_already_member = 0
|
|
289
|
+
total_failed = 0
|
|
290
|
+
failures = []
|
|
291
|
+
|
|
292
|
+
for workspace in workspaces:
|
|
293
|
+
ws_name = workspace["name"]
|
|
294
|
+
members = workspace.get("members", [])
|
|
295
|
+
|
|
296
|
+
if not members:
|
|
297
|
+
print(f"[SKIP] Workspace '{ws_name}' — no members, skipping")
|
|
298
|
+
total_skipped += 1
|
|
299
|
+
continue
|
|
300
|
+
|
|
301
|
+
print(f"\n{'=' * 60}")
|
|
302
|
+
print(f"Workspace: {ws_name} ({len(members)} members)")
|
|
303
|
+
print(f"{'=' * 60}")
|
|
304
|
+
|
|
305
|
+
ws_added = 0
|
|
306
|
+
ws_no_email = 0
|
|
307
|
+
ws_already_member = 0
|
|
308
|
+
ws_failed = 0
|
|
309
|
+
|
|
310
|
+
for member in members:
|
|
311
|
+
email = member.get("email")
|
|
312
|
+
|
|
313
|
+
if not email:
|
|
314
|
+
ws_no_email += 1
|
|
315
|
+
total_no_email += 1
|
|
316
|
+
print(f" [SKIP] '{member['userName']}' has no email, skipping")
|
|
317
|
+
continue
|
|
318
|
+
|
|
319
|
+
if dry_run:
|
|
320
|
+
print(f" [DRY RUN] Would add '{email}' to '{ws_name}'")
|
|
321
|
+
ws_added += 1
|
|
322
|
+
total_added += 1
|
|
323
|
+
continue
|
|
324
|
+
|
|
325
|
+
status, error_info = _add_member(add_member_url, headers, email, ws_name)
|
|
326
|
+
|
|
327
|
+
if status == "added":
|
|
328
|
+
ws_added += 1
|
|
329
|
+
total_added += 1
|
|
330
|
+
elif status == "already_member":
|
|
331
|
+
ws_already_member += 1
|
|
332
|
+
total_already_member += 1
|
|
333
|
+
else:
|
|
334
|
+
print(
|
|
335
|
+
f" [FAIL] '{email}' -> '{ws_name}' "
|
|
336
|
+
f"— {error_info['status']}: {error_info['error']}"
|
|
337
|
+
)
|
|
338
|
+
ws_failed += 1
|
|
339
|
+
total_failed += 1
|
|
340
|
+
failures.append(
|
|
341
|
+
{
|
|
342
|
+
"workspace": ws_name,
|
|
343
|
+
"email": email,
|
|
344
|
+
**error_info,
|
|
345
|
+
}
|
|
346
|
+
)
|
|
347
|
+
|
|
348
|
+
if dry_run:
|
|
349
|
+
print(f" [DRY RUN] Total: would add {ws_added}/{len(members)} users")
|
|
350
|
+
else:
|
|
351
|
+
parts = [f" Added {ws_added}/{len(members)} users successfully"]
|
|
352
|
+
if ws_already_member:
|
|
353
|
+
parts.append(f"{ws_already_member} already a member")
|
|
354
|
+
if ws_no_email:
|
|
355
|
+
parts.append(f"{ws_no_email} skipped (no email)")
|
|
356
|
+
if ws_failed:
|
|
357
|
+
parts.append(f"{ws_failed} failed")
|
|
358
|
+
print(
|
|
359
|
+
parts[0]
|
|
360
|
+
+ (" (" + ", ".join(parts[1:]) + ")" if len(parts) > 1 else "")
|
|
361
|
+
)
|
|
362
|
+
|
|
363
|
+
print(f"\n{'=' * 60}")
|
|
364
|
+
print("SUMMARY")
|
|
365
|
+
print(f"{'=' * 60}")
|
|
366
|
+
print(f" Workspaces processed : {len(workspaces)}")
|
|
367
|
+
print(f" Workspaces skipped : {total_skipped} (no members)")
|
|
368
|
+
print(f" Users added : {total_added}")
|
|
369
|
+
print(f" Already a member : {total_already_member}")
|
|
370
|
+
print(f" Skipped (no email) : {total_no_email}")
|
|
371
|
+
print(f" Failures : {total_failed}")
|
|
372
|
+
|
|
373
|
+
if dry_run:
|
|
374
|
+
print(f"\n *** DRY RUN MODE — no API calls were made ***")
|
|
375
|
+
print(f" *** Remove --dry-run to execute for real ***")
|
|
376
|
+
|
|
377
|
+
if failures:
|
|
378
|
+
with open(failures_output, "w") as f:
|
|
379
|
+
json.dump(failures, f, indent=2)
|
|
380
|
+
print(f"\n Failed operations saved to {failures_output}")
|
|
381
|
+
|
|
382
|
+
|
|
383
|
+
def main(args):
|
|
384
|
+
parser = argparse.ArgumentParser(
|
|
385
|
+
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
|
386
|
+
)
|
|
387
|
+
get_parser_arguments(parser)
|
|
388
|
+
parsed_args = parser.parse_args(args)
|
|
389
|
+
migrate_users(parsed_args)
|
|
390
|
+
|
|
391
|
+
|
|
392
|
+
if __name__ == "__main__":
|
|
393
|
+
main(sys.argv[1:])
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: cometx
|
|
3
|
-
Version: 3.6.
|
|
3
|
+
Version: 3.6.5
|
|
4
4
|
Summary: Python tools for Comet
|
|
5
5
|
Home-page: https://github.com/comet-ml/cometx/
|
|
6
6
|
Author: cometx development team
|
|
@@ -102,6 +102,7 @@ for ML management.
|
|
|
102
102
|
* [cometx download](#cometx-download)
|
|
103
103
|
* [cometx list](#cometx-list)
|
|
104
104
|
* [cometx log](#cometx-log)
|
|
105
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
105
106
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
106
107
|
* [cometx reproduce](#cometx-reproduce)
|
|
107
108
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -324,6 +325,7 @@ Not all combinations are possible:
|
|
|
324
325
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
325
326
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
326
327
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
328
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
327
329
|
|
|
328
330
|
### Using --path
|
|
329
331
|
|
|
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
|
|
|
385
387
|
|
|
386
388
|
For more information, `cometx log --help`
|
|
387
389
|
|
|
390
|
+
## cometx migrate-users
|
|
391
|
+
|
|
392
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
393
|
+
|
|
394
|
+
```
|
|
395
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
396
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
**Arguments:**
|
|
400
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
401
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
402
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
403
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
404
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
405
|
+
|
|
406
|
+
### Flags
|
|
407
|
+
|
|
408
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
409
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
410
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
411
|
+
|
|
412
|
+
### Examples
|
|
413
|
+
|
|
414
|
+
```bash
|
|
415
|
+
# Dry run — preview what would happen
|
|
416
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
417
|
+
|
|
418
|
+
# Execute the migration
|
|
419
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
420
|
+
|
|
421
|
+
# Use a local chargeback report file
|
|
422
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
423
|
+
|
|
424
|
+
# Self-hosted environments — provide explicit URLs
|
|
425
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
426
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
For more information, `cometx migrate-users --help`
|
|
430
|
+
|
|
388
431
|
## cometx rename-duplicates
|
|
389
432
|
|
|
390
433
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
535
578
|
**Examples:**
|
|
536
579
|
```
|
|
537
580
|
cometx admin chargeback-report
|
|
538
|
-
<<<<<<< Updated upstream
|
|
539
|
-
cometx admin usage-report
|
|
540
|
-
=======
|
|
541
581
|
cometx admin chargeback-report 2024-09
|
|
542
582
|
```
|
|
543
|
-
>>>>>>> Stashed changes
|
|
544
583
|
|
|
545
584
|
#### usage-report
|
|
546
585
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|