cometx 3.6.4__tar.gz → 3.6.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {cometx-3.6.4 → cometx-3.6.5}/PKG-INFO +44 -5
- {cometx-3.6.4 → cometx-3.6.5}/README.md +43 -4
- {cometx-3.6.4 → cometx-3.6.5}/cometx/_version.py +1 -1
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/__init__.py +3 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/copy.py +40 -4
- cometx-3.6.5/cometx/cli/migrate_users.py +393 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/PKG-INFO +44 -5
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/SOURCES.txt +1 -0
- {cometx-3.6.4 → cometx-3.6.5}/LICENSE +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/__init__.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/_typing.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/_ui.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/api.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/audio.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_app.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_gpu_app.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_gpu_report.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_optimizer_report.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_usage_report.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/admin_utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/config.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/copy_utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/count.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/delete_assets.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/download.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/list_command.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/log.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/rename_duplicates.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/reproduce.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/smoke_test.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/update.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/upload_optimizer_experiments.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/cli/utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/framework/__init__.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/framework/comet/__init__.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/framework/comet/download_manager.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/framework/neptune.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/framework/wandb.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/generate_utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/magics.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/optimizer.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/panel_utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/tools/__init__.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/tools/dataset.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/tools/pointcloud.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx/utils.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/dependency_links.txt +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/entry_points.txt +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/requires.txt +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/cometx.egg-info/top_level.txt +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/pyproject.toml +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/setup.cfg +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/setup.py +0 -0
- {cometx-3.6.4 → cometx-3.6.5}/tests/testlib.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: cometx
|
|
3
|
-
Version: 3.6.
|
|
3
|
+
Version: 3.6.5
|
|
4
4
|
Summary: Python tools for Comet
|
|
5
5
|
Home-page: https://github.com/comet-ml/cometx/
|
|
6
6
|
Author: cometx development team
|
|
@@ -102,6 +102,7 @@ for ML management.
|
|
|
102
102
|
* [cometx download](#cometx-download)
|
|
103
103
|
* [cometx list](#cometx-list)
|
|
104
104
|
* [cometx log](#cometx-log)
|
|
105
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
105
106
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
106
107
|
* [cometx reproduce](#cometx-reproduce)
|
|
107
108
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -324,6 +325,7 @@ Not all combinations are possible:
|
|
|
324
325
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
325
326
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
326
327
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
328
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
327
329
|
|
|
328
330
|
### Using --path
|
|
329
331
|
|
|
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
|
|
|
385
387
|
|
|
386
388
|
For more information, `cometx log --help`
|
|
387
389
|
|
|
390
|
+
## cometx migrate-users
|
|
391
|
+
|
|
392
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
393
|
+
|
|
394
|
+
```
|
|
395
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
396
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
**Arguments:**
|
|
400
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
401
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
402
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
403
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
404
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
405
|
+
|
|
406
|
+
### Flags
|
|
407
|
+
|
|
408
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
409
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
410
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
411
|
+
|
|
412
|
+
### Examples
|
|
413
|
+
|
|
414
|
+
```bash
|
|
415
|
+
# Dry run — preview what would happen
|
|
416
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
417
|
+
|
|
418
|
+
# Execute the migration
|
|
419
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
420
|
+
|
|
421
|
+
# Use a local chargeback report file
|
|
422
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
423
|
+
|
|
424
|
+
# Self-hosted environments — provide explicit URLs
|
|
425
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
426
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
For more information, `cometx migrate-users --help`
|
|
430
|
+
|
|
388
431
|
## cometx rename-duplicates
|
|
389
432
|
|
|
390
433
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
535
578
|
**Examples:**
|
|
536
579
|
```
|
|
537
580
|
cometx admin chargeback-report
|
|
538
|
-
<<<<<<< Updated upstream
|
|
539
|
-
cometx admin usage-report
|
|
540
|
-
=======
|
|
541
581
|
cometx admin chargeback-report 2024-09
|
|
542
582
|
```
|
|
543
|
-
>>>>>>> Stashed changes
|
|
544
583
|
|
|
545
584
|
#### usage-report
|
|
546
585
|
|
|
@@ -46,6 +46,7 @@ for ML management.
|
|
|
46
46
|
* [cometx download](#cometx-download)
|
|
47
47
|
* [cometx list](#cometx-list)
|
|
48
48
|
* [cometx log](#cometx-log)
|
|
49
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
49
50
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
50
51
|
* [cometx reproduce](#cometx-reproduce)
|
|
51
52
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -268,6 +269,7 @@ Not all combinations are possible:
|
|
|
268
269
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
269
270
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
270
271
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
272
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
271
273
|
|
|
272
274
|
### Using --path
|
|
273
275
|
|
|
@@ -329,6 +331,47 @@ Where TYPE is one of the following names:
|
|
|
329
331
|
|
|
330
332
|
For more information, `cometx log --help`
|
|
331
333
|
|
|
334
|
+
## cometx migrate-users
|
|
335
|
+
|
|
336
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
337
|
+
|
|
338
|
+
```
|
|
339
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
340
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
**Arguments:**
|
|
344
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
345
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
346
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
347
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
348
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
349
|
+
|
|
350
|
+
### Flags
|
|
351
|
+
|
|
352
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
353
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
354
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
355
|
+
|
|
356
|
+
### Examples
|
|
357
|
+
|
|
358
|
+
```bash
|
|
359
|
+
# Dry run — preview what would happen
|
|
360
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
361
|
+
|
|
362
|
+
# Execute the migration
|
|
363
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
364
|
+
|
|
365
|
+
# Use a local chargeback report file
|
|
366
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
367
|
+
|
|
368
|
+
# Self-hosted environments — provide explicit URLs
|
|
369
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
370
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
For more information, `cometx migrate-users --help`
|
|
374
|
+
|
|
332
375
|
## cometx rename-duplicates
|
|
333
376
|
|
|
334
377
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -479,12 +522,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
479
522
|
**Examples:**
|
|
480
523
|
```
|
|
481
524
|
cometx admin chargeback-report
|
|
482
|
-
<<<<<<< Updated upstream
|
|
483
|
-
cometx admin usage-report
|
|
484
|
-
=======
|
|
485
525
|
cometx admin chargeback-report 2024-09
|
|
486
526
|
```
|
|
487
|
-
>>>>>>> Stashed changes
|
|
488
527
|
|
|
489
528
|
#### usage-report
|
|
490
529
|
|
|
@@ -26,6 +26,7 @@ Examples:
|
|
|
26
26
|
cometx smoke-test
|
|
27
27
|
cometx update
|
|
28
28
|
cometx admin
|
|
29
|
+
cometx migrate-users
|
|
29
30
|
|
|
30
31
|
For more information:
|
|
31
32
|
cometx COMMAND --help
|
|
@@ -47,6 +48,7 @@ from . import (
|
|
|
47
48
|
download,
|
|
48
49
|
list_command,
|
|
49
50
|
log,
|
|
51
|
+
migrate_users,
|
|
50
52
|
rename_duplicates,
|
|
51
53
|
reproduce,
|
|
52
54
|
smoke_test,
|
|
@@ -108,6 +110,7 @@ def main(raw_args=sys.argv[1:]):
|
|
|
108
110
|
add_subparser(subparsers, config, "config")
|
|
109
111
|
add_subparser(subparsers, rename_duplicates, "rename-duplicates")
|
|
110
112
|
add_subparser(subparsers, smoke_test, "smoke-test")
|
|
113
|
+
add_subparser(subparsers, migrate_users, "migrate-users")
|
|
111
114
|
add_subparser(
|
|
112
115
|
subparsers, upload_optimizer_experiments, "upload-optimizer-experiments"
|
|
113
116
|
)
|
|
@@ -78,6 +78,8 @@ import threading
|
|
|
78
78
|
import time
|
|
79
79
|
import urllib.parse
|
|
80
80
|
import zipfile
|
|
81
|
+
|
|
82
|
+
import requests
|
|
81
83
|
from datetime import datetime, timedelta
|
|
82
84
|
|
|
83
85
|
from comet_ml import APIExperiment, Artifact, Experiment, OfflineExperiment
|
|
@@ -102,6 +104,7 @@ from ..api import API
|
|
|
102
104
|
from ..framework.comet.download_manager import sanitize_filename
|
|
103
105
|
from ..utils import remove_extra_slashes
|
|
104
106
|
from .copy_utils import upload_single_offline_experiment
|
|
107
|
+
from .migrate_users import _create_workspace
|
|
105
108
|
|
|
106
109
|
ADDITIONAL_ARGS = False
|
|
107
110
|
|
|
@@ -197,6 +200,12 @@ def get_parser_arguments(parser):
|
|
|
197
200
|
type=str,
|
|
198
201
|
default=None,
|
|
199
202
|
)
|
|
203
|
+
parser.add_argument(
|
|
204
|
+
"--create-workspaces",
|
|
205
|
+
help="Attempt to create the destination workspace if it does not exist",
|
|
206
|
+
default=False,
|
|
207
|
+
action="store_true",
|
|
208
|
+
)
|
|
200
209
|
|
|
201
210
|
|
|
202
211
|
def copy(parsed_args, remaining=None):
|
|
@@ -214,6 +223,7 @@ def copy(parsed_args, remaining=None):
|
|
|
214
223
|
parsed_args.ignore,
|
|
215
224
|
parsed_args.debug,
|
|
216
225
|
parsed_args.sync,
|
|
226
|
+
parsed_args.create_workspaces,
|
|
217
227
|
)
|
|
218
228
|
|
|
219
229
|
# Wait for all uploads to complete
|
|
@@ -790,7 +800,7 @@ class CopyManager:
|
|
|
790
800
|
# Restore default signal handler
|
|
791
801
|
signal.signal(signal.SIGINT, signal.default_int_handler)
|
|
792
802
|
|
|
793
|
-
def copy(self, source, destination, symlink, ignore, debug, sync):
|
|
803
|
+
def copy(self, source, destination, symlink, ignore, debug, sync, create_workspaces=False):
|
|
794
804
|
""" """
|
|
795
805
|
self.ignore = ignore
|
|
796
806
|
self.debug = debug
|
|
@@ -823,9 +833,35 @@ class CopyManager:
|
|
|
823
833
|
# First check to make sure workspace_dst exists:
|
|
824
834
|
workspaces = self.api.get_workspaces()
|
|
825
835
|
if workspace_dst not in workspaces:
|
|
826
|
-
|
|
827
|
-
|
|
828
|
-
|
|
836
|
+
if create_workspaces:
|
|
837
|
+
print(
|
|
838
|
+
f"Workspace {workspace_dst!r} does not exist, attempting to create it..."
|
|
839
|
+
)
|
|
840
|
+
dest_url = self.api._get_url_server()
|
|
841
|
+
headers = {
|
|
842
|
+
"Authorization": self.api.api_key,
|
|
843
|
+
"Content-Type": "application/json",
|
|
844
|
+
}
|
|
845
|
+
try:
|
|
846
|
+
_create_workspace(dest_url, headers, workspace_dst)
|
|
847
|
+
print(f"Workspace {workspace_dst!r} created successfully.")
|
|
848
|
+
except requests.exceptions.HTTPError as exc:
|
|
849
|
+
raise Exception(
|
|
850
|
+
f"Workspace {workspace_dst!r} does not exist and could not be "
|
|
851
|
+
f"created automatically (HTTP {exc.response.status_code}). "
|
|
852
|
+
f"Please create it via the Comet UI and try again."
|
|
853
|
+
) from exc
|
|
854
|
+
except requests.exceptions.RequestException as exc:
|
|
855
|
+
raise Exception(
|
|
856
|
+
f"Workspace {workspace_dst!r} does not exist and could not be "
|
|
857
|
+
f"created automatically: {exc}. "
|
|
858
|
+
f"Please create it via the Comet UI and try again."
|
|
859
|
+
) from exc
|
|
860
|
+
else:
|
|
861
|
+
raise Exception(
|
|
862
|
+
f"{workspace_dst} does not exist; use --create-workspaces to "
|
|
863
|
+
f"create it automatically, or create it via the Comet UI"
|
|
864
|
+
)
|
|
829
865
|
|
|
830
866
|
if project_src == "panels":
|
|
831
867
|
# experiment_src may be "*" or filename
|
|
@@ -0,0 +1,393 @@
|
|
|
1
|
+
#!/usr/bin/env python
|
|
2
|
+
# -*- coding: utf-8 -*-
|
|
3
|
+
# ****************************************
|
|
4
|
+
# __
|
|
5
|
+
# _________ ____ ___ ___ / /__ __
|
|
6
|
+
# / ___/ __ \/ __ `__ \/ _ \/ __/ |/_/
|
|
7
|
+
# / /__/ /_/ / / / / / / __/ /__> <
|
|
8
|
+
# \___/\____/_/ /_/ /_/\___/\__/_/|_|
|
|
9
|
+
#
|
|
10
|
+
#
|
|
11
|
+
# Copyright (c) 2022 Cometx Development
|
|
12
|
+
# Team. All rights reserved.
|
|
13
|
+
# ****************************************
|
|
14
|
+
"""
|
|
15
|
+
Migrate users into workspaces from a source environment to a destination
|
|
16
|
+
environment based on a chargeback report.
|
|
17
|
+
|
|
18
|
+
The chargeback report is fetched automatically from the source environment's
|
|
19
|
+
admin API, or you can provide a local JSON file with --chargeback-report.
|
|
20
|
+
|
|
21
|
+
Examples:
|
|
22
|
+
|
|
23
|
+
$ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
24
|
+
$ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
25
|
+
$ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json
|
|
26
|
+
"""
|
|
27
|
+
|
|
28
|
+
import argparse
|
|
29
|
+
import base64
|
|
30
|
+
import json
|
|
31
|
+
import os
|
|
32
|
+
import sys
|
|
33
|
+
import urllib.parse
|
|
34
|
+
|
|
35
|
+
import requests
|
|
36
|
+
|
|
37
|
+
ADDITIONAL_ARGS = False
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def get_parser_arguments(parser):
|
|
41
|
+
parser.add_argument(
|
|
42
|
+
"--api-key",
|
|
43
|
+
help="API key for the destination environment (used to add workspace members). "
|
|
44
|
+
"Falls back to COMET_API_KEY environment variable if not provided.",
|
|
45
|
+
type=str,
|
|
46
|
+
default=None,
|
|
47
|
+
)
|
|
48
|
+
parser.add_argument(
|
|
49
|
+
"--url",
|
|
50
|
+
help="Base URL of the destination Comet environment (e.g. https://comet.example.com). "
|
|
51
|
+
"Required for self-hosted instances when the API key does not encode the server URL.",
|
|
52
|
+
type=str,
|
|
53
|
+
default=None,
|
|
54
|
+
)
|
|
55
|
+
parser.add_argument(
|
|
56
|
+
"--source-api-key",
|
|
57
|
+
help="API key for the source environment (used to fetch the chargeback report). "
|
|
58
|
+
"Required unless --chargeback-report is given.",
|
|
59
|
+
type=str,
|
|
60
|
+
default=None,
|
|
61
|
+
)
|
|
62
|
+
parser.add_argument(
|
|
63
|
+
"--source-url",
|
|
64
|
+
help="Base URL of the source Comet environment. "
|
|
65
|
+
"Required for self-hosted instances when the source API key does not encode the server URL.",
|
|
66
|
+
type=str,
|
|
67
|
+
default=None,
|
|
68
|
+
)
|
|
69
|
+
parser.add_argument(
|
|
70
|
+
"--chargeback-report",
|
|
71
|
+
help="Path to a local chargeback report JSON file (skips fetching from source environment)",
|
|
72
|
+
type=str,
|
|
73
|
+
default=None,
|
|
74
|
+
)
|
|
75
|
+
parser.add_argument(
|
|
76
|
+
"--create-workspaces",
|
|
77
|
+
help="Create workspaces on the destination environment if they don't already exist",
|
|
78
|
+
default=False,
|
|
79
|
+
action="store_true",
|
|
80
|
+
)
|
|
81
|
+
parser.add_argument(
|
|
82
|
+
"--dry-run",
|
|
83
|
+
help="Print what would happen without making any changes",
|
|
84
|
+
default=False,
|
|
85
|
+
action="store_true",
|
|
86
|
+
)
|
|
87
|
+
parser.add_argument(
|
|
88
|
+
"--failures-output",
|
|
89
|
+
help="Path to write failed operations JSON (default: bulk_add_failures_by_email.json)",
|
|
90
|
+
type=str,
|
|
91
|
+
default="bulk_add_failures_by_email.json",
|
|
92
|
+
)
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
def _resolve_server_url(api_key, explicit_url=None):
|
|
96
|
+
"""Return the server base URL.
|
|
97
|
+
|
|
98
|
+
Priority:
|
|
99
|
+
1. ``explicit_url`` (from --url / --source-url), if provided.
|
|
100
|
+
2. URL encoded inside the API key (new-style keys contain ``*<base64>``).
|
|
101
|
+
3. Error — no silent fallback to cloud URL.
|
|
102
|
+
"""
|
|
103
|
+
if explicit_url:
|
|
104
|
+
parsed = urllib.parse.urlparse(explicit_url)
|
|
105
|
+
if parsed.scheme != "https":
|
|
106
|
+
print("[ERROR] --url/--source-url must use https://.")
|
|
107
|
+
sys.exit(1)
|
|
108
|
+
return explicit_url.rstrip("/")
|
|
109
|
+
|
|
110
|
+
if "*" in api_key:
|
|
111
|
+
try:
|
|
112
|
+
_, encoded = api_key.split("*", 1)
|
|
113
|
+
# base64 padding may be missing
|
|
114
|
+
padding = (4 - len(encoded) % 4) % 4
|
|
115
|
+
payload = json.loads(base64.b64decode(encoded + "=" * padding))
|
|
116
|
+
return payload["baseUrl"].rstrip("/")
|
|
117
|
+
except Exception:
|
|
118
|
+
pass # Fall through to error below
|
|
119
|
+
|
|
120
|
+
print(
|
|
121
|
+
"[ERROR] Cannot determine the server URL from the API key. "
|
|
122
|
+
"Pass --url (destination) or --source-url (source) explicitly, "
|
|
123
|
+
"e.g. --url https://comet.example.com"
|
|
124
|
+
)
|
|
125
|
+
sys.exit(1)
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
def _fetch_chargeback_report(server_url, source_api_key):
|
|
129
|
+
url = f"{server_url}/api/admin/chargeback/report"
|
|
130
|
+
print(f"Fetching chargeback report from {url}...")
|
|
131
|
+
resp = requests.get(
|
|
132
|
+
url,
|
|
133
|
+
headers={"Authorization": source_api_key},
|
|
134
|
+
timeout=30,
|
|
135
|
+
)
|
|
136
|
+
resp.raise_for_status()
|
|
137
|
+
return resp.json()
|
|
138
|
+
|
|
139
|
+
|
|
140
|
+
def _load_chargeback_report(path):
|
|
141
|
+
print(f"Loading chargeback report from {path}...")
|
|
142
|
+
with open(path, "r") as f:
|
|
143
|
+
return json.load(f)
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
def _get_existing_workspaces(dest_url, headers):
|
|
147
|
+
url = f"{dest_url}/api/rest/v2/workspaces"
|
|
148
|
+
resp = requests.get(url, headers=headers, timeout=15)
|
|
149
|
+
resp.raise_for_status()
|
|
150
|
+
data = resp.json()
|
|
151
|
+
# Handle {"workspaceNames": [...]} dict shape
|
|
152
|
+
if isinstance(data, dict):
|
|
153
|
+
names = data.get("workspaceNames", [])
|
|
154
|
+
if names and isinstance(names[0], dict):
|
|
155
|
+
return {ws["name"] for ws in names}
|
|
156
|
+
return set(names)
|
|
157
|
+
# Handle list of dicts with a "name" key
|
|
158
|
+
if data and isinstance(data[0], dict):
|
|
159
|
+
return {ws["name"] for ws in data}
|
|
160
|
+
# Handle flat list of strings
|
|
161
|
+
return set(data)
|
|
162
|
+
|
|
163
|
+
|
|
164
|
+
def _create_workspace(dest_url, headers, workspace_name):
|
|
165
|
+
url = f"{dest_url}/api/rest/v2/write/workspace/new"
|
|
166
|
+
resp = requests.post(
|
|
167
|
+
url,
|
|
168
|
+
headers=headers,
|
|
169
|
+
json={"name": workspace_name},
|
|
170
|
+
timeout=15,
|
|
171
|
+
)
|
|
172
|
+
resp.raise_for_status()
|
|
173
|
+
|
|
174
|
+
|
|
175
|
+
def _add_member(url, headers, email, workspace_name):
|
|
176
|
+
"""Attempt to add a single member. Returns (status, error_info_or_None)."""
|
|
177
|
+
payload = {
|
|
178
|
+
"userEmail": email,
|
|
179
|
+
"workspaceName": workspace_name,
|
|
180
|
+
"admin": True,
|
|
181
|
+
}
|
|
182
|
+
try:
|
|
183
|
+
response = requests.post(
|
|
184
|
+
url,
|
|
185
|
+
headers=headers,
|
|
186
|
+
json=payload,
|
|
187
|
+
timeout=15,
|
|
188
|
+
)
|
|
189
|
+
|
|
190
|
+
if response.status_code == 200:
|
|
191
|
+
return "added", None
|
|
192
|
+
|
|
193
|
+
try:
|
|
194
|
+
error_msg = response.json()
|
|
195
|
+
except ValueError:
|
|
196
|
+
error_msg = response.text
|
|
197
|
+
|
|
198
|
+
is_already_member = (
|
|
199
|
+
isinstance(error_msg, dict)
|
|
200
|
+
and "already member of" in error_msg.get("msg", "")
|
|
201
|
+
)
|
|
202
|
+
if is_already_member:
|
|
203
|
+
return "already_member", None
|
|
204
|
+
|
|
205
|
+
return "failed", {
|
|
206
|
+
"status": response.status_code,
|
|
207
|
+
"error": error_msg,
|
|
208
|
+
}
|
|
209
|
+
|
|
210
|
+
except requests.exceptions.RequestException as e:
|
|
211
|
+
return "failed", {
|
|
212
|
+
"status": "exception",
|
|
213
|
+
"error": str(e),
|
|
214
|
+
}
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
def migrate_users(parsed_args):
|
|
218
|
+
api_key = parsed_args.api_key or os.environ.get("COMET_API_KEY")
|
|
219
|
+
if not api_key:
|
|
220
|
+
print("[ERROR] No API key found. Set COMET_API_KEY or pass --api-key.")
|
|
221
|
+
sys.exit(1)
|
|
222
|
+
|
|
223
|
+
source_api_key = parsed_args.source_api_key
|
|
224
|
+
create_workspaces = parsed_args.create_workspaces
|
|
225
|
+
dry_run = parsed_args.dry_run
|
|
226
|
+
failures_output = parsed_args.failures_output
|
|
227
|
+
|
|
228
|
+
if not parsed_args.chargeback_report and not source_api_key:
|
|
229
|
+
print(
|
|
230
|
+
"[ERROR] --source-api-key is required when --chargeback-report is not provided."
|
|
231
|
+
)
|
|
232
|
+
sys.exit(1)
|
|
233
|
+
|
|
234
|
+
dest_url = _resolve_server_url(api_key, parsed_args.url)
|
|
235
|
+
print(f"Destination URL: {dest_url}")
|
|
236
|
+
|
|
237
|
+
if parsed_args.chargeback_report:
|
|
238
|
+
data = _load_chargeback_report(parsed_args.chargeback_report)
|
|
239
|
+
else:
|
|
240
|
+
source_url = _resolve_server_url(source_api_key, parsed_args.source_url)
|
|
241
|
+
print(f"Source URL: {source_url}")
|
|
242
|
+
if source_url == dest_url and source_api_key == api_key:
|
|
243
|
+
print(
|
|
244
|
+
"[WARNING] Source and destination URL and API key are identical. "
|
|
245
|
+
"Are you sure you want to migrate users to the same environment?"
|
|
246
|
+
)
|
|
247
|
+
try:
|
|
248
|
+
data = _fetch_chargeback_report(source_url, source_api_key)
|
|
249
|
+
except requests.exceptions.RequestException as e:
|
|
250
|
+
print(f"[ERROR] Failed to fetch chargeback report: {e}")
|
|
251
|
+
sys.exit(1)
|
|
252
|
+
|
|
253
|
+
workspaces = data.get("workspaces", [])
|
|
254
|
+
print(f"Found {len(workspaces)} workspaces\n")
|
|
255
|
+
|
|
256
|
+
headers = {
|
|
257
|
+
"Authorization": api_key,
|
|
258
|
+
"Content-Type": "application/json",
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
if create_workspaces and not dry_run:
|
|
262
|
+
print("Checking existing workspaces on destination...")
|
|
263
|
+
try:
|
|
264
|
+
existing = _get_existing_workspaces(dest_url, headers)
|
|
265
|
+
except requests.exceptions.RequestException as e:
|
|
266
|
+
print(f"[ERROR] Failed to list destination workspaces: {e}")
|
|
267
|
+
sys.exit(1)
|
|
268
|
+
|
|
269
|
+
ws_created = 0
|
|
270
|
+
for workspace in workspaces:
|
|
271
|
+
ws_name = workspace["name"]
|
|
272
|
+
if ws_name not in existing:
|
|
273
|
+
try:
|
|
274
|
+
_create_workspace(dest_url, headers, ws_name)
|
|
275
|
+
print(f" [CREATED] Workspace '{ws_name}'")
|
|
276
|
+
ws_created += 1
|
|
277
|
+
except requests.exceptions.RequestException as e:
|
|
278
|
+
print(f" [ERROR] Failed to create workspace '{ws_name}': {e}")
|
|
279
|
+
print(f"Created {ws_created} new workspace(s)\n")
|
|
280
|
+
elif create_workspaces and dry_run:
|
|
281
|
+
print("[DRY RUN] Would check/create workspaces on destination\n")
|
|
282
|
+
|
|
283
|
+
add_member_url = f"{dest_url}/api/rest/v2/write/add-workspace-member"
|
|
284
|
+
|
|
285
|
+
total_added = 0
|
|
286
|
+
total_skipped = 0
|
|
287
|
+
total_no_email = 0
|
|
288
|
+
total_already_member = 0
|
|
289
|
+
total_failed = 0
|
|
290
|
+
failures = []
|
|
291
|
+
|
|
292
|
+
for workspace in workspaces:
|
|
293
|
+
ws_name = workspace["name"]
|
|
294
|
+
members = workspace.get("members", [])
|
|
295
|
+
|
|
296
|
+
if not members:
|
|
297
|
+
print(f"[SKIP] Workspace '{ws_name}' — no members, skipping")
|
|
298
|
+
total_skipped += 1
|
|
299
|
+
continue
|
|
300
|
+
|
|
301
|
+
print(f"\n{'=' * 60}")
|
|
302
|
+
print(f"Workspace: {ws_name} ({len(members)} members)")
|
|
303
|
+
print(f"{'=' * 60}")
|
|
304
|
+
|
|
305
|
+
ws_added = 0
|
|
306
|
+
ws_no_email = 0
|
|
307
|
+
ws_already_member = 0
|
|
308
|
+
ws_failed = 0
|
|
309
|
+
|
|
310
|
+
for member in members:
|
|
311
|
+
email = member.get("email")
|
|
312
|
+
|
|
313
|
+
if not email:
|
|
314
|
+
ws_no_email += 1
|
|
315
|
+
total_no_email += 1
|
|
316
|
+
print(f" [SKIP] '{member['userName']}' has no email, skipping")
|
|
317
|
+
continue
|
|
318
|
+
|
|
319
|
+
if dry_run:
|
|
320
|
+
print(f" [DRY RUN] Would add '{email}' to '{ws_name}'")
|
|
321
|
+
ws_added += 1
|
|
322
|
+
total_added += 1
|
|
323
|
+
continue
|
|
324
|
+
|
|
325
|
+
status, error_info = _add_member(add_member_url, headers, email, ws_name)
|
|
326
|
+
|
|
327
|
+
if status == "added":
|
|
328
|
+
ws_added += 1
|
|
329
|
+
total_added += 1
|
|
330
|
+
elif status == "already_member":
|
|
331
|
+
ws_already_member += 1
|
|
332
|
+
total_already_member += 1
|
|
333
|
+
else:
|
|
334
|
+
print(
|
|
335
|
+
f" [FAIL] '{email}' -> '{ws_name}' "
|
|
336
|
+
f"— {error_info['status']}: {error_info['error']}"
|
|
337
|
+
)
|
|
338
|
+
ws_failed += 1
|
|
339
|
+
total_failed += 1
|
|
340
|
+
failures.append(
|
|
341
|
+
{
|
|
342
|
+
"workspace": ws_name,
|
|
343
|
+
"email": email,
|
|
344
|
+
**error_info,
|
|
345
|
+
}
|
|
346
|
+
)
|
|
347
|
+
|
|
348
|
+
if dry_run:
|
|
349
|
+
print(f" [DRY RUN] Total: would add {ws_added}/{len(members)} users")
|
|
350
|
+
else:
|
|
351
|
+
parts = [f" Added {ws_added}/{len(members)} users successfully"]
|
|
352
|
+
if ws_already_member:
|
|
353
|
+
parts.append(f"{ws_already_member} already a member")
|
|
354
|
+
if ws_no_email:
|
|
355
|
+
parts.append(f"{ws_no_email} skipped (no email)")
|
|
356
|
+
if ws_failed:
|
|
357
|
+
parts.append(f"{ws_failed} failed")
|
|
358
|
+
print(
|
|
359
|
+
parts[0]
|
|
360
|
+
+ (" (" + ", ".join(parts[1:]) + ")" if len(parts) > 1 else "")
|
|
361
|
+
)
|
|
362
|
+
|
|
363
|
+
print(f"\n{'=' * 60}")
|
|
364
|
+
print("SUMMARY")
|
|
365
|
+
print(f"{'=' * 60}")
|
|
366
|
+
print(f" Workspaces processed : {len(workspaces)}")
|
|
367
|
+
print(f" Workspaces skipped : {total_skipped} (no members)")
|
|
368
|
+
print(f" Users added : {total_added}")
|
|
369
|
+
print(f" Already a member : {total_already_member}")
|
|
370
|
+
print(f" Skipped (no email) : {total_no_email}")
|
|
371
|
+
print(f" Failures : {total_failed}")
|
|
372
|
+
|
|
373
|
+
if dry_run:
|
|
374
|
+
print(f"\n *** DRY RUN MODE — no API calls were made ***")
|
|
375
|
+
print(f" *** Remove --dry-run to execute for real ***")
|
|
376
|
+
|
|
377
|
+
if failures:
|
|
378
|
+
with open(failures_output, "w") as f:
|
|
379
|
+
json.dump(failures, f, indent=2)
|
|
380
|
+
print(f"\n Failed operations saved to {failures_output}")
|
|
381
|
+
|
|
382
|
+
|
|
383
|
+
def main(args):
|
|
384
|
+
parser = argparse.ArgumentParser(
|
|
385
|
+
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
|
386
|
+
)
|
|
387
|
+
get_parser_arguments(parser)
|
|
388
|
+
parsed_args = parser.parse_args(args)
|
|
389
|
+
migrate_users(parsed_args)
|
|
390
|
+
|
|
391
|
+
|
|
392
|
+
if __name__ == "__main__":
|
|
393
|
+
main(sys.argv[1:])
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: cometx
|
|
3
|
-
Version: 3.6.
|
|
3
|
+
Version: 3.6.5
|
|
4
4
|
Summary: Python tools for Comet
|
|
5
5
|
Home-page: https://github.com/comet-ml/cometx/
|
|
6
6
|
Author: cometx development team
|
|
@@ -102,6 +102,7 @@ for ML management.
|
|
|
102
102
|
* [cometx download](#cometx-download)
|
|
103
103
|
* [cometx list](#cometx-list)
|
|
104
104
|
* [cometx log](#cometx-log)
|
|
105
|
+
* [cometx migrate-users](#cometx-migrate-users)
|
|
105
106
|
* [cometx rename-duplicates](#cometx-rename-duplicates)
|
|
106
107
|
* [cometx reproduce](#cometx-reproduce)
|
|
107
108
|
* [cometx smoke-test](#cometx-smoke-test)
|
|
@@ -324,6 +325,7 @@ Not all combinations are possible:
|
|
|
324
325
|
* `--symlink` - Instead of copying, create a link to an experiment in a project
|
|
325
326
|
* `--sync` - Check to see if experiment name has been created first; if so, skip
|
|
326
327
|
* `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
|
|
328
|
+
* `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
|
|
327
329
|
|
|
328
330
|
### Using --path
|
|
329
331
|
|
|
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
|
|
|
385
387
|
|
|
386
388
|
For more information, `cometx log --help`
|
|
387
389
|
|
|
390
|
+
## cometx migrate-users
|
|
391
|
+
|
|
392
|
+
This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
|
|
393
|
+
|
|
394
|
+
```
|
|
395
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
|
|
396
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
**Arguments:**
|
|
400
|
+
* `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
|
|
401
|
+
* `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
|
|
402
|
+
* `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
|
|
403
|
+
* `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
|
|
404
|
+
* `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
|
|
405
|
+
|
|
406
|
+
### Flags
|
|
407
|
+
|
|
408
|
+
* `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
|
|
409
|
+
* `--dry-run` - Print what would happen without making any changes
|
|
410
|
+
* `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
|
|
411
|
+
|
|
412
|
+
### Examples
|
|
413
|
+
|
|
414
|
+
```bash
|
|
415
|
+
# Dry run — preview what would happen
|
|
416
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
|
|
417
|
+
|
|
418
|
+
# Execute the migration
|
|
419
|
+
cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
|
|
420
|
+
|
|
421
|
+
# Use a local chargeback report file
|
|
422
|
+
cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
|
|
423
|
+
|
|
424
|
+
# Self-hosted environments — provide explicit URLs
|
|
425
|
+
cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
|
|
426
|
+
--source-api-key SOURCE_KEY --source-url https://comet.src.example.com
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
For more information, `cometx migrate-users --help`
|
|
430
|
+
|
|
388
431
|
## cometx rename-duplicates
|
|
389
432
|
|
|
390
433
|
This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
|
|
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
|
|
|
535
578
|
**Examples:**
|
|
536
579
|
```
|
|
537
580
|
cometx admin chargeback-report
|
|
538
|
-
<<<<<<< Updated upstream
|
|
539
|
-
cometx admin usage-report
|
|
540
|
-
=======
|
|
541
581
|
cometx admin chargeback-report 2024-09
|
|
542
582
|
```
|
|
543
|
-
>>>>>>> Stashed changes
|
|
544
583
|
|
|
545
584
|
#### usage-report
|
|
546
585
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|