cometx 3.6.3__tar.gz → 3.6.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. {cometx-3.6.3 → cometx-3.6.5}/PKG-INFO +44 -5
  2. {cometx-3.6.3 → cometx-3.6.5}/README.md +43 -4
  3. {cometx-3.6.3 → cometx-3.6.5}/cometx/_version.py +1 -1
  4. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/__init__.py +3 -0
  5. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_gpu_app.py +143 -1
  6. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/copy.py +40 -4
  7. cometx-3.6.5/cometx/cli/migrate_users.py +393 -0
  8. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/PKG-INFO +44 -5
  9. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/SOURCES.txt +1 -0
  10. {cometx-3.6.3 → cometx-3.6.5}/LICENSE +0 -0
  11. {cometx-3.6.3 → cometx-3.6.5}/cometx/__init__.py +0 -0
  12. {cometx-3.6.3 → cometx-3.6.5}/cometx/_typing.py +0 -0
  13. {cometx-3.6.3 → cometx-3.6.5}/cometx/_ui.py +0 -0
  14. {cometx-3.6.3 → cometx-3.6.5}/cometx/api.py +0 -0
  15. {cometx-3.6.3 → cometx-3.6.5}/cometx/audio.py +0 -0
  16. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin.py +0 -0
  17. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_app.py +0 -0
  18. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_gpu_report.py +0 -0
  19. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_optimizer_report.py +0 -0
  20. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_usage_report.py +0 -0
  21. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/admin_utils.py +0 -0
  22. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/config.py +0 -0
  23. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/copy_utils.py +0 -0
  24. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/count.py +0 -0
  25. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/delete_assets.py +0 -0
  26. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/download.py +0 -0
  27. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/list_command.py +0 -0
  28. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/log.py +0 -0
  29. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/rename_duplicates.py +0 -0
  30. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/reproduce.py +0 -0
  31. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/smoke_test.py +0 -0
  32. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/update.py +0 -0
  33. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/upload_optimizer_experiments.py +0 -0
  34. {cometx-3.6.3 → cometx-3.6.5}/cometx/cli/utils.py +0 -0
  35. {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/__init__.py +0 -0
  36. {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/comet/__init__.py +0 -0
  37. {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/comet/download_manager.py +0 -0
  38. {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/neptune.py +0 -0
  39. {cometx-3.6.3 → cometx-3.6.5}/cometx/framework/wandb.py +0 -0
  40. {cometx-3.6.3 → cometx-3.6.5}/cometx/generate_utils.py +0 -0
  41. {cometx-3.6.3 → cometx-3.6.5}/cometx/magics.py +0 -0
  42. {cometx-3.6.3 → cometx-3.6.5}/cometx/optimizer.py +0 -0
  43. {cometx-3.6.3 → cometx-3.6.5}/cometx/panel_utils.py +0 -0
  44. {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/__init__.py +0 -0
  45. {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/dataset.py +0 -0
  46. {cometx-3.6.3 → cometx-3.6.5}/cometx/tools/pointcloud.py +0 -0
  47. {cometx-3.6.3 → cometx-3.6.5}/cometx/utils.py +0 -0
  48. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/dependency_links.txt +0 -0
  49. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/entry_points.txt +0 -0
  50. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/requires.txt +0 -0
  51. {cometx-3.6.3 → cometx-3.6.5}/cometx.egg-info/top_level.txt +0 -0
  52. {cometx-3.6.3 → cometx-3.6.5}/pyproject.toml +0 -0
  53. {cometx-3.6.3 → cometx-3.6.5}/setup.cfg +0 -0
  54. {cometx-3.6.3 → cometx-3.6.5}/setup.py +0 -0
  55. {cometx-3.6.3 → cometx-3.6.5}/tests/testlib.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: cometx
3
- Version: 3.6.3
3
+ Version: 3.6.5
4
4
  Summary: Python tools for Comet
5
5
  Home-page: https://github.com/comet-ml/cometx/
6
6
  Author: cometx development team
@@ -102,6 +102,7 @@ for ML management.
102
102
  * [cometx download](#cometx-download)
103
103
  * [cometx list](#cometx-list)
104
104
  * [cometx log](#cometx-log)
105
+ * [cometx migrate-users](#cometx-migrate-users)
105
106
  * [cometx rename-duplicates](#cometx-rename-duplicates)
106
107
  * [cometx reproduce](#cometx-reproduce)
107
108
  * [cometx smoke-test](#cometx-smoke-test)
@@ -324,6 +325,7 @@ Not all combinations are possible:
324
325
  * `--symlink` - Instead of copying, create a link to an experiment in a project
325
326
  * `--sync` - Check to see if experiment name has been created first; if so, skip
326
327
  * `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
328
+ * `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
327
329
 
328
330
  ### Using --path
329
331
 
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
385
387
 
386
388
  For more information, `cometx log --help`
387
389
 
390
+ ## cometx migrate-users
391
+
392
+ This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
393
+
394
+ ```
395
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
396
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
397
+ ```
398
+
399
+ **Arguments:**
400
+ * `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
401
+ * `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
402
+ * `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
403
+ * `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
404
+ * `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
405
+
406
+ ### Flags
407
+
408
+ * `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
409
+ * `--dry-run` - Print what would happen without making any changes
410
+ * `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
411
+
412
+ ### Examples
413
+
414
+ ```bash
415
+ # Dry run — preview what would happen
416
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
417
+
418
+ # Execute the migration
419
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
420
+
421
+ # Use a local chargeback report file
422
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
423
+
424
+ # Self-hosted environments — provide explicit URLs
425
+ cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
426
+ --source-api-key SOURCE_KEY --source-url https://comet.src.example.com
427
+ ```
428
+
429
+ For more information, `cometx migrate-users --help`
430
+
388
431
  ## cometx rename-duplicates
389
432
 
390
433
  This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
535
578
  **Examples:**
536
579
  ```
537
580
  cometx admin chargeback-report
538
- <<<<<<< Updated upstream
539
- cometx admin usage-report
540
- =======
541
581
  cometx admin chargeback-report 2024-09
542
582
  ```
543
- >>>>>>> Stashed changes
544
583
 
545
584
  #### usage-report
546
585
 
@@ -46,6 +46,7 @@ for ML management.
46
46
  * [cometx download](#cometx-download)
47
47
  * [cometx list](#cometx-list)
48
48
  * [cometx log](#cometx-log)
49
+ * [cometx migrate-users](#cometx-migrate-users)
49
50
  * [cometx rename-duplicates](#cometx-rename-duplicates)
50
51
  * [cometx reproduce](#cometx-reproduce)
51
52
  * [cometx smoke-test](#cometx-smoke-test)
@@ -268,6 +269,7 @@ Not all combinations are possible:
268
269
  * `--symlink` - Instead of copying, create a link to an experiment in a project
269
270
  * `--sync` - Check to see if experiment name has been created first; if so, skip
270
271
  * `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
272
+ * `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
271
273
 
272
274
  ### Using --path
273
275
 
@@ -329,6 +331,47 @@ Where TYPE is one of the following names:
329
331
 
330
332
  For more information, `cometx log --help`
331
333
 
334
+ ## cometx migrate-users
335
+
336
+ This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
337
+
338
+ ```
339
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
340
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
341
+ ```
342
+
343
+ **Arguments:**
344
+ * `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
345
+ * `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
346
+ * `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
347
+ * `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
348
+ * `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
349
+
350
+ ### Flags
351
+
352
+ * `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
353
+ * `--dry-run` - Print what would happen without making any changes
354
+ * `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
355
+
356
+ ### Examples
357
+
358
+ ```bash
359
+ # Dry run — preview what would happen
360
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
361
+
362
+ # Execute the migration
363
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
364
+
365
+ # Use a local chargeback report file
366
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
367
+
368
+ # Self-hosted environments — provide explicit URLs
369
+ cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
370
+ --source-api-key SOURCE_KEY --source-url https://comet.src.example.com
371
+ ```
372
+
373
+ For more information, `cometx migrate-users --help`
374
+
332
375
  ## cometx rename-duplicates
333
376
 
334
377
  This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
@@ -479,12 +522,8 @@ cometx admin chargeback-report [YEAR-MONTH]
479
522
  **Examples:**
480
523
  ```
481
524
  cometx admin chargeback-report
482
- <<<<<<< Updated upstream
483
- cometx admin usage-report
484
- =======
485
525
  cometx admin chargeback-report 2024-09
486
526
  ```
487
- >>>>>>> Stashed changes
488
527
 
489
528
  #### usage-report
490
529
 
@@ -12,5 +12,5 @@
12
12
  # the express permission of Comet ML Inc.
13
13
  # *******************************************************
14
14
 
15
- version_info = (3, 6, 3)
15
+ version_info = (3, 6, 5)
16
16
  __version__ = ".".join(map(str, version_info))
@@ -26,6 +26,7 @@ Examples:
26
26
  cometx smoke-test
27
27
  cometx update
28
28
  cometx admin
29
+ cometx migrate-users
29
30
 
30
31
  For more information:
31
32
  cometx COMMAND --help
@@ -47,6 +48,7 @@ from . import (
47
48
  download,
48
49
  list_command,
49
50
  log,
51
+ migrate_users,
50
52
  rename_duplicates,
51
53
  reproduce,
52
54
  smoke_test,
@@ -108,6 +110,7 @@ def main(raw_args=sys.argv[1:]):
108
110
  add_subparser(subparsers, config, "config")
109
111
  add_subparser(subparsers, rename_duplicates, "rename-duplicates")
110
112
  add_subparser(subparsers, smoke_test, "smoke-test")
113
+ add_subparser(subparsers, migrate_users, "migrate-users")
111
114
  add_subparser(
112
115
  subparsers, upload_optimizer_experiments, "upload-optimizer-experiments"
113
116
  )
@@ -119,6 +119,7 @@ def process_metrics_with_aggregation(
119
119
  start_date: str = None,
120
120
  end_date: str = None,
121
121
  aggregate_metric: str = "avg",
122
+ selected_projects: List[str] = None,
122
123
  ) -> Tuple[Dict, Dict]:
123
124
  """
124
125
  Process metric data with specified aggregation and time unit.
@@ -132,6 +133,7 @@ def process_metrics_with_aggregation(
132
133
  start_date: Optional start date filter (YYYY-MM-DD)
133
134
  end_date: Optional end date filter (YYYY-MM-DD)
134
135
  aggregate_metric: Aggregation method for time series - "max" or "avg" (default: "avg")
136
+ selected_projects: Optional list of "workspace/project" strings to filter by
135
137
 
136
138
  Returns:
137
139
  tuple: (avg_data, time_series_data)
@@ -165,6 +167,15 @@ def process_metrics_with_aggregation(
165
167
  continue
166
168
 
167
169
  exp_info = experiment_map[exp_key]
170
+
171
+ # Apply project filter
172
+ if selected_projects is not None:
173
+ ws = exp_info.get("workspace", "Unknown")
174
+ proj = exp_info.get("project_name", "Unknown")
175
+ project_key = f"{ws}/{proj}"
176
+ if project_key not in selected_projects:
177
+ continue
178
+
168
179
  server_timestamp = exp_info.get("server_timestamp")
169
180
 
170
181
  # Apply date filter
@@ -271,13 +282,19 @@ def create_aggregation_avg_chart(
271
282
  keys = [k for k, _ in sorted_keys]
272
283
  values = [v for _, v in sorted_keys]
273
284
 
285
+ # Assign colors by alphabetical order to match the time series line chart
286
+ all_keys_sorted = sorted(data.keys())
287
+ colors_list = get_distinct_colors(len(all_keys_sorted))
288
+ color_map = {k: colors_list[i] for i, k in enumerate(all_keys_sorted)}
289
+ bar_colors = [color_map[k] for k in keys]
290
+
274
291
  # Create bar chart
275
292
  fig, ax = plt.subplots(figsize=(14, 8))
276
293
 
277
294
  ax.bar(
278
295
  range(len(keys)),
279
296
  values,
280
- color="steelblue",
297
+ color=bar_colors,
281
298
  edgecolor="navy",
282
299
  alpha=0.7,
283
300
  width=0.8,
@@ -449,6 +466,21 @@ def create_time_series_chart(
449
466
  return fig
450
467
 
451
468
 
469
+ def get_available_workspaces_and_projects(data: Dict) -> Dict[str, List[str]]:
470
+ """Extract unique workspaces and their projects from experiment_map.
471
+
472
+ Returns:
473
+ Dict mapping workspace names to sorted list of project names.
474
+ """
475
+ experiment_map = data.get("experiment_map", {})
476
+ workspace_projects = defaultdict(set)
477
+ for exp_info in experiment_map.values():
478
+ ws = exp_info.get("workspace", "Unknown")
479
+ proj = exp_info.get("project_name", "Unknown")
480
+ workspace_projects[ws].add(proj)
481
+ return {ws: sorted(list(projects)) for ws, projects in sorted(workspace_projects.items())}
482
+
483
+
452
484
  def get_available_dates(data: Dict) -> Tuple[List[str], str, str]:
453
485
  """Extract available date range from the data."""
454
486
  experiment_map = data.get("experiment_map", {})
@@ -496,6 +528,19 @@ def get_available_dates(data: Dict) -> Tuple[List[str], str, str]:
496
528
 
497
529
  def main():
498
530
  """Main Streamlit app entry point."""
531
+ # Set theme primary color to blue (overrides default red/coral)
532
+ # This must be done before any Streamlit elements are rendered
533
+ try:
534
+ import streamlit.config as _config
535
+ _config.set_option("theme.primaryColor", "#1f77b4")
536
+ except Exception:
537
+ pass # Fallback gracefully if internal API changes
538
+
539
+ # Force rerun on first load to ensure theme takes effect
540
+ if "theme_initialized" not in st.session_state:
541
+ st.session_state.theme_initialized = True
542
+ st.rerun()
543
+
499
544
  st.set_page_config(
500
545
  page_title="Comet GPU Report",
501
546
  page_icon="🖥️",
@@ -552,11 +597,107 @@ def main():
552
597
  # Get available dates
553
598
  available_dates, min_date, max_date = get_available_dates(data)
554
599
 
600
+ # Get available workspaces and projects for filtering
601
+ workspace_projects_map = get_available_workspaces_and_projects(data)
602
+ all_workspaces = list(workspace_projects_map.keys())
603
+
604
+ # Custom CSS for professional styling (override Streamlit's default red/pink accent)
605
+ st.markdown("""
606
+ <style>
607
+ /* Override checkbox checked color - target the role=checkbox div */
608
+ div[role="checkbox"][aria-checked="true"] {
609
+ background-color: #1f77b4 !important;
610
+ border-color: #1f77b4 !important;
611
+ }
612
+ /* Also target via data-baseweb attribute */
613
+ [data-baseweb="checkbox"] div[aria-checked="true"] {
614
+ background-color: #1f77b4 !important;
615
+ border-color: #1f77b4 !important;
616
+ }
617
+ /* Style multi-select pills */
618
+ span[data-baseweb="tag"] {
619
+ background-color: #1f77b4 !important;
620
+ }
621
+ span[data-baseweb="tag"] span {
622
+ color: white !important;
623
+ }
624
+ </style>
625
+ """, unsafe_allow_html=True)
626
+
555
627
  # Sidebar configuration
556
628
  with st.sidebar:
557
629
  st.divider()
558
630
  st.header("Configuration")
559
631
 
632
+ # Workspace/Project Filter Section
633
+ st.subheader("Data Filter")
634
+
635
+ # Get all available projects across all workspaces
636
+ all_projects = []
637
+ for ws in all_workspaces:
638
+ for proj in workspace_projects_map.get(ws, []):
639
+ all_projects.append(f"{ws}/{proj}")
640
+ all_projects = sorted(all_projects)
641
+
642
+ # Initialize session state for project selection
643
+ # Use individual checkbox keys as the source of truth
644
+ if "projects_initialized" not in st.session_state:
645
+ for proj_path in all_projects:
646
+ st.session_state[f"proj_{proj_path}"] = True
647
+ st.session_state.projects_initialized = True
648
+
649
+ # Track whether the expander should stay open
650
+ if "projects_expander_open" not in st.session_state:
651
+ st.session_state.projects_expander_open = True
652
+
653
+ # Select All / Clear All buttons
654
+ btn_col1, btn_col2 = st.columns(2)
655
+ with btn_col1:
656
+ if st.button("Select All", use_container_width=True):
657
+ for proj_path in all_projects:
658
+ st.session_state[f"proj_{proj_path}"] = True
659
+ st.session_state.projects_expander_open = True
660
+ st.rerun()
661
+ with btn_col2:
662
+ if st.button("Clear All", use_container_width=True):
663
+ for proj_path in all_projects:
664
+ st.session_state[f"proj_{proj_path}"] = False
665
+ st.session_state.projects_expander_open = True
666
+ st.rerun()
667
+
668
+ # Expandable project selector for fine-grained control
669
+ with st.expander("Select specific projects", expanded=st.session_state.projects_expander_open):
670
+ # Group projects by workspace for cleaner display
671
+ for ws in all_workspaces:
672
+ ws_projects = [f"{ws}/{p}" for p in workspace_projects_map.get(ws, [])]
673
+ if not ws_projects:
674
+ continue
675
+
676
+ st.markdown(f"**{ws}**")
677
+
678
+ for proj_path in sorted(ws_projects):
679
+ proj_name = proj_path.split("/")[-1]
680
+ st.checkbox(
681
+ proj_name,
682
+ key=f"proj_{proj_path}",
683
+ )
684
+
685
+ # Build selected projects list from checkbox states
686
+ selected_projects_set = set()
687
+ for proj_path in all_projects:
688
+ if st.session_state.get(f"proj_{proj_path}", True):
689
+ selected_projects_set.add(proj_path)
690
+
691
+ # Show project count
692
+ num_selected = len(selected_projects_set)
693
+ num_total = len(all_projects)
694
+ st.caption(f"**{num_selected}** of **{num_total}** projects selected")
695
+
696
+ # Convert set to list for processing
697
+ selected_projects = sorted(selected_projects_set)
698
+
699
+ st.divider()
700
+
560
701
  # Aggregation selection
561
702
  st.subheader("Aggregation")
562
703
  aggregation = st.selectbox(
@@ -640,6 +781,7 @@ def main():
640
781
  start_date_str,
641
782
  end_date_str,
642
783
  aggregate_metric=aggregate_metric,
784
+ selected_projects=selected_projects,
643
785
  )
644
786
 
645
787
  # Display statistics
@@ -78,6 +78,8 @@ import threading
78
78
  import time
79
79
  import urllib.parse
80
80
  import zipfile
81
+
82
+ import requests
81
83
  from datetime import datetime, timedelta
82
84
 
83
85
  from comet_ml import APIExperiment, Artifact, Experiment, OfflineExperiment
@@ -102,6 +104,7 @@ from ..api import API
102
104
  from ..framework.comet.download_manager import sanitize_filename
103
105
  from ..utils import remove_extra_slashes
104
106
  from .copy_utils import upload_single_offline_experiment
107
+ from .migrate_users import _create_workspace
105
108
 
106
109
  ADDITIONAL_ARGS = False
107
110
 
@@ -197,6 +200,12 @@ def get_parser_arguments(parser):
197
200
  type=str,
198
201
  default=None,
199
202
  )
203
+ parser.add_argument(
204
+ "--create-workspaces",
205
+ help="Attempt to create the destination workspace if it does not exist",
206
+ default=False,
207
+ action="store_true",
208
+ )
200
209
 
201
210
 
202
211
  def copy(parsed_args, remaining=None):
@@ -214,6 +223,7 @@ def copy(parsed_args, remaining=None):
214
223
  parsed_args.ignore,
215
224
  parsed_args.debug,
216
225
  parsed_args.sync,
226
+ parsed_args.create_workspaces,
217
227
  )
218
228
 
219
229
  # Wait for all uploads to complete
@@ -790,7 +800,7 @@ class CopyManager:
790
800
  # Restore default signal handler
791
801
  signal.signal(signal.SIGINT, signal.default_int_handler)
792
802
 
793
- def copy(self, source, destination, symlink, ignore, debug, sync):
803
+ def copy(self, source, destination, symlink, ignore, debug, sync, create_workspaces=False):
794
804
  """ """
795
805
  self.ignore = ignore
796
806
  self.debug = debug
@@ -823,9 +833,35 @@ class CopyManager:
823
833
  # First check to make sure workspace_dst exists:
824
834
  workspaces = self.api.get_workspaces()
825
835
  if workspace_dst not in workspaces:
826
- raise Exception(
827
- f"{workspace_dst} does not exist; use the Comet UI to create it"
828
- )
836
+ if create_workspaces:
837
+ print(
838
+ f"Workspace {workspace_dst!r} does not exist, attempting to create it..."
839
+ )
840
+ dest_url = self.api._get_url_server()
841
+ headers = {
842
+ "Authorization": self.api.api_key,
843
+ "Content-Type": "application/json",
844
+ }
845
+ try:
846
+ _create_workspace(dest_url, headers, workspace_dst)
847
+ print(f"Workspace {workspace_dst!r} created successfully.")
848
+ except requests.exceptions.HTTPError as exc:
849
+ raise Exception(
850
+ f"Workspace {workspace_dst!r} does not exist and could not be "
851
+ f"created automatically (HTTP {exc.response.status_code}). "
852
+ f"Please create it via the Comet UI and try again."
853
+ ) from exc
854
+ except requests.exceptions.RequestException as exc:
855
+ raise Exception(
856
+ f"Workspace {workspace_dst!r} does not exist and could not be "
857
+ f"created automatically: {exc}. "
858
+ f"Please create it via the Comet UI and try again."
859
+ ) from exc
860
+ else:
861
+ raise Exception(
862
+ f"{workspace_dst} does not exist; use --create-workspaces to "
863
+ f"create it automatically, or create it via the Comet UI"
864
+ )
829
865
 
830
866
  if project_src == "panels":
831
867
  # experiment_src may be "*" or filename
@@ -0,0 +1,393 @@
1
+ #!/usr/bin/env python
2
+ # -*- coding: utf-8 -*-
3
+ # ****************************************
4
+ # __
5
+ # _________ ____ ___ ___ / /__ __
6
+ # / ___/ __ \/ __ `__ \/ _ \/ __/ |/_/
7
+ # / /__/ /_/ / / / / / / __/ /__> <
8
+ # \___/\____/_/ /_/ /_/\___/\__/_/|_|
9
+ #
10
+ #
11
+ # Copyright (c) 2022 Cometx Development
12
+ # Team. All rights reserved.
13
+ # ****************************************
14
+ """
15
+ Migrate users into workspaces from a source environment to a destination
16
+ environment based on a chargeback report.
17
+
18
+ The chargeback report is fetched automatically from the source environment's
19
+ admin API, or you can provide a local JSON file with --chargeback-report.
20
+
21
+ Examples:
22
+
23
+ $ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
24
+ $ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
25
+ $ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json
26
+ """
27
+
28
+ import argparse
29
+ import base64
30
+ import json
31
+ import os
32
+ import sys
33
+ import urllib.parse
34
+
35
+ import requests
36
+
37
+ ADDITIONAL_ARGS = False
38
+
39
+
40
+ def get_parser_arguments(parser):
41
+ parser.add_argument(
42
+ "--api-key",
43
+ help="API key for the destination environment (used to add workspace members). "
44
+ "Falls back to COMET_API_KEY environment variable if not provided.",
45
+ type=str,
46
+ default=None,
47
+ )
48
+ parser.add_argument(
49
+ "--url",
50
+ help="Base URL of the destination Comet environment (e.g. https://comet.example.com). "
51
+ "Required for self-hosted instances when the API key does not encode the server URL.",
52
+ type=str,
53
+ default=None,
54
+ )
55
+ parser.add_argument(
56
+ "--source-api-key",
57
+ help="API key for the source environment (used to fetch the chargeback report). "
58
+ "Required unless --chargeback-report is given.",
59
+ type=str,
60
+ default=None,
61
+ )
62
+ parser.add_argument(
63
+ "--source-url",
64
+ help="Base URL of the source Comet environment. "
65
+ "Required for self-hosted instances when the source API key does not encode the server URL.",
66
+ type=str,
67
+ default=None,
68
+ )
69
+ parser.add_argument(
70
+ "--chargeback-report",
71
+ help="Path to a local chargeback report JSON file (skips fetching from source environment)",
72
+ type=str,
73
+ default=None,
74
+ )
75
+ parser.add_argument(
76
+ "--create-workspaces",
77
+ help="Create workspaces on the destination environment if they don't already exist",
78
+ default=False,
79
+ action="store_true",
80
+ )
81
+ parser.add_argument(
82
+ "--dry-run",
83
+ help="Print what would happen without making any changes",
84
+ default=False,
85
+ action="store_true",
86
+ )
87
+ parser.add_argument(
88
+ "--failures-output",
89
+ help="Path to write failed operations JSON (default: bulk_add_failures_by_email.json)",
90
+ type=str,
91
+ default="bulk_add_failures_by_email.json",
92
+ )
93
+
94
+
95
+ def _resolve_server_url(api_key, explicit_url=None):
96
+ """Return the server base URL.
97
+
98
+ Priority:
99
+ 1. ``explicit_url`` (from --url / --source-url), if provided.
100
+ 2. URL encoded inside the API key (new-style keys contain ``*<base64>``).
101
+ 3. Error — no silent fallback to cloud URL.
102
+ """
103
+ if explicit_url:
104
+ parsed = urllib.parse.urlparse(explicit_url)
105
+ if parsed.scheme != "https":
106
+ print("[ERROR] --url/--source-url must use https://.")
107
+ sys.exit(1)
108
+ return explicit_url.rstrip("/")
109
+
110
+ if "*" in api_key:
111
+ try:
112
+ _, encoded = api_key.split("*", 1)
113
+ # base64 padding may be missing
114
+ padding = (4 - len(encoded) % 4) % 4
115
+ payload = json.loads(base64.b64decode(encoded + "=" * padding))
116
+ return payload["baseUrl"].rstrip("/")
117
+ except Exception:
118
+ pass # Fall through to error below
119
+
120
+ print(
121
+ "[ERROR] Cannot determine the server URL from the API key. "
122
+ "Pass --url (destination) or --source-url (source) explicitly, "
123
+ "e.g. --url https://comet.example.com"
124
+ )
125
+ sys.exit(1)
126
+
127
+
128
+ def _fetch_chargeback_report(server_url, source_api_key):
129
+ url = f"{server_url}/api/admin/chargeback/report"
130
+ print(f"Fetching chargeback report from {url}...")
131
+ resp = requests.get(
132
+ url,
133
+ headers={"Authorization": source_api_key},
134
+ timeout=30,
135
+ )
136
+ resp.raise_for_status()
137
+ return resp.json()
138
+
139
+
140
+ def _load_chargeback_report(path):
141
+ print(f"Loading chargeback report from {path}...")
142
+ with open(path, "r") as f:
143
+ return json.load(f)
144
+
145
+
146
+ def _get_existing_workspaces(dest_url, headers):
147
+ url = f"{dest_url}/api/rest/v2/workspaces"
148
+ resp = requests.get(url, headers=headers, timeout=15)
149
+ resp.raise_for_status()
150
+ data = resp.json()
151
+ # Handle {"workspaceNames": [...]} dict shape
152
+ if isinstance(data, dict):
153
+ names = data.get("workspaceNames", [])
154
+ if names and isinstance(names[0], dict):
155
+ return {ws["name"] for ws in names}
156
+ return set(names)
157
+ # Handle list of dicts with a "name" key
158
+ if data and isinstance(data[0], dict):
159
+ return {ws["name"] for ws in data}
160
+ # Handle flat list of strings
161
+ return set(data)
162
+
163
+
164
+ def _create_workspace(dest_url, headers, workspace_name):
165
+ url = f"{dest_url}/api/rest/v2/write/workspace/new"
166
+ resp = requests.post(
167
+ url,
168
+ headers=headers,
169
+ json={"name": workspace_name},
170
+ timeout=15,
171
+ )
172
+ resp.raise_for_status()
173
+
174
+
175
+ def _add_member(url, headers, email, workspace_name):
176
+ """Attempt to add a single member. Returns (status, error_info_or_None)."""
177
+ payload = {
178
+ "userEmail": email,
179
+ "workspaceName": workspace_name,
180
+ "admin": True,
181
+ }
182
+ try:
183
+ response = requests.post(
184
+ url,
185
+ headers=headers,
186
+ json=payload,
187
+ timeout=15,
188
+ )
189
+
190
+ if response.status_code == 200:
191
+ return "added", None
192
+
193
+ try:
194
+ error_msg = response.json()
195
+ except ValueError:
196
+ error_msg = response.text
197
+
198
+ is_already_member = (
199
+ isinstance(error_msg, dict)
200
+ and "already member of" in error_msg.get("msg", "")
201
+ )
202
+ if is_already_member:
203
+ return "already_member", None
204
+
205
+ return "failed", {
206
+ "status": response.status_code,
207
+ "error": error_msg,
208
+ }
209
+
210
+ except requests.exceptions.RequestException as e:
211
+ return "failed", {
212
+ "status": "exception",
213
+ "error": str(e),
214
+ }
215
+
216
+
217
+ def migrate_users(parsed_args):
218
+ api_key = parsed_args.api_key or os.environ.get("COMET_API_KEY")
219
+ if not api_key:
220
+ print("[ERROR] No API key found. Set COMET_API_KEY or pass --api-key.")
221
+ sys.exit(1)
222
+
223
+ source_api_key = parsed_args.source_api_key
224
+ create_workspaces = parsed_args.create_workspaces
225
+ dry_run = parsed_args.dry_run
226
+ failures_output = parsed_args.failures_output
227
+
228
+ if not parsed_args.chargeback_report and not source_api_key:
229
+ print(
230
+ "[ERROR] --source-api-key is required when --chargeback-report is not provided."
231
+ )
232
+ sys.exit(1)
233
+
234
+ dest_url = _resolve_server_url(api_key, parsed_args.url)
235
+ print(f"Destination URL: {dest_url}")
236
+
237
+ if parsed_args.chargeback_report:
238
+ data = _load_chargeback_report(parsed_args.chargeback_report)
239
+ else:
240
+ source_url = _resolve_server_url(source_api_key, parsed_args.source_url)
241
+ print(f"Source URL: {source_url}")
242
+ if source_url == dest_url and source_api_key == api_key:
243
+ print(
244
+ "[WARNING] Source and destination URL and API key are identical. "
245
+ "Are you sure you want to migrate users to the same environment?"
246
+ )
247
+ try:
248
+ data = _fetch_chargeback_report(source_url, source_api_key)
249
+ except requests.exceptions.RequestException as e:
250
+ print(f"[ERROR] Failed to fetch chargeback report: {e}")
251
+ sys.exit(1)
252
+
253
+ workspaces = data.get("workspaces", [])
254
+ print(f"Found {len(workspaces)} workspaces\n")
255
+
256
+ headers = {
257
+ "Authorization": api_key,
258
+ "Content-Type": "application/json",
259
+ }
260
+
261
+ if create_workspaces and not dry_run:
262
+ print("Checking existing workspaces on destination...")
263
+ try:
264
+ existing = _get_existing_workspaces(dest_url, headers)
265
+ except requests.exceptions.RequestException as e:
266
+ print(f"[ERROR] Failed to list destination workspaces: {e}")
267
+ sys.exit(1)
268
+
269
+ ws_created = 0
270
+ for workspace in workspaces:
271
+ ws_name = workspace["name"]
272
+ if ws_name not in existing:
273
+ try:
274
+ _create_workspace(dest_url, headers, ws_name)
275
+ print(f" [CREATED] Workspace '{ws_name}'")
276
+ ws_created += 1
277
+ except requests.exceptions.RequestException as e:
278
+ print(f" [ERROR] Failed to create workspace '{ws_name}': {e}")
279
+ print(f"Created {ws_created} new workspace(s)\n")
280
+ elif create_workspaces and dry_run:
281
+ print("[DRY RUN] Would check/create workspaces on destination\n")
282
+
283
+ add_member_url = f"{dest_url}/api/rest/v2/write/add-workspace-member"
284
+
285
+ total_added = 0
286
+ total_skipped = 0
287
+ total_no_email = 0
288
+ total_already_member = 0
289
+ total_failed = 0
290
+ failures = []
291
+
292
+ for workspace in workspaces:
293
+ ws_name = workspace["name"]
294
+ members = workspace.get("members", [])
295
+
296
+ if not members:
297
+ print(f"[SKIP] Workspace '{ws_name}' — no members, skipping")
298
+ total_skipped += 1
299
+ continue
300
+
301
+ print(f"\n{'=' * 60}")
302
+ print(f"Workspace: {ws_name} ({len(members)} members)")
303
+ print(f"{'=' * 60}")
304
+
305
+ ws_added = 0
306
+ ws_no_email = 0
307
+ ws_already_member = 0
308
+ ws_failed = 0
309
+
310
+ for member in members:
311
+ email = member.get("email")
312
+
313
+ if not email:
314
+ ws_no_email += 1
315
+ total_no_email += 1
316
+ print(f" [SKIP] '{member['userName']}' has no email, skipping")
317
+ continue
318
+
319
+ if dry_run:
320
+ print(f" [DRY RUN] Would add '{email}' to '{ws_name}'")
321
+ ws_added += 1
322
+ total_added += 1
323
+ continue
324
+
325
+ status, error_info = _add_member(add_member_url, headers, email, ws_name)
326
+
327
+ if status == "added":
328
+ ws_added += 1
329
+ total_added += 1
330
+ elif status == "already_member":
331
+ ws_already_member += 1
332
+ total_already_member += 1
333
+ else:
334
+ print(
335
+ f" [FAIL] '{email}' -> '{ws_name}' "
336
+ f"— {error_info['status']}: {error_info['error']}"
337
+ )
338
+ ws_failed += 1
339
+ total_failed += 1
340
+ failures.append(
341
+ {
342
+ "workspace": ws_name,
343
+ "email": email,
344
+ **error_info,
345
+ }
346
+ )
347
+
348
+ if dry_run:
349
+ print(f" [DRY RUN] Total: would add {ws_added}/{len(members)} users")
350
+ else:
351
+ parts = [f" Added {ws_added}/{len(members)} users successfully"]
352
+ if ws_already_member:
353
+ parts.append(f"{ws_already_member} already a member")
354
+ if ws_no_email:
355
+ parts.append(f"{ws_no_email} skipped (no email)")
356
+ if ws_failed:
357
+ parts.append(f"{ws_failed} failed")
358
+ print(
359
+ parts[0]
360
+ + (" (" + ", ".join(parts[1:]) + ")" if len(parts) > 1 else "")
361
+ )
362
+
363
+ print(f"\n{'=' * 60}")
364
+ print("SUMMARY")
365
+ print(f"{'=' * 60}")
366
+ print(f" Workspaces processed : {len(workspaces)}")
367
+ print(f" Workspaces skipped : {total_skipped} (no members)")
368
+ print(f" Users added : {total_added}")
369
+ print(f" Already a member : {total_already_member}")
370
+ print(f" Skipped (no email) : {total_no_email}")
371
+ print(f" Failures : {total_failed}")
372
+
373
+ if dry_run:
374
+ print(f"\n *** DRY RUN MODE — no API calls were made ***")
375
+ print(f" *** Remove --dry-run to execute for real ***")
376
+
377
+ if failures:
378
+ with open(failures_output, "w") as f:
379
+ json.dump(failures, f, indent=2)
380
+ print(f"\n Failed operations saved to {failures_output}")
381
+
382
+
383
+ def main(args):
384
+ parser = argparse.ArgumentParser(
385
+ description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
386
+ )
387
+ get_parser_arguments(parser)
388
+ parsed_args = parser.parse_args(args)
389
+ migrate_users(parsed_args)
390
+
391
+
392
+ if __name__ == "__main__":
393
+ main(sys.argv[1:])
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: cometx
3
- Version: 3.6.3
3
+ Version: 3.6.5
4
4
  Summary: Python tools for Comet
5
5
  Home-page: https://github.com/comet-ml/cometx/
6
6
  Author: cometx development team
@@ -102,6 +102,7 @@ for ML management.
102
102
  * [cometx download](#cometx-download)
103
103
  * [cometx list](#cometx-list)
104
104
  * [cometx log](#cometx-log)
105
+ * [cometx migrate-users](#cometx-migrate-users)
105
106
  * [cometx rename-duplicates](#cometx-rename-duplicates)
106
107
  * [cometx reproduce](#cometx-reproduce)
107
108
  * [cometx smoke-test](#cometx-smoke-test)
@@ -324,6 +325,7 @@ Not all combinations are possible:
324
325
  * `--symlink` - Instead of copying, create a link to an experiment in a project
325
326
  * `--sync` - Check to see if experiment name has been created first; if so, skip
326
327
  * `--path PATH` - Path to prepend to workspace_src when accessing files (supports ~ for home directory)
328
+ * `--create-workspaces` - Attempt to create the destination workspace if it does not exist (requires appropriate API permissions)
327
329
 
328
330
  ### Using --path
329
331
 
@@ -385,6 +387,47 @@ Where TYPE is one of the following names:
385
387
 
386
388
  For more information, `cometx log --help`
387
389
 
390
+ ## cometx migrate-users
391
+
392
+ This command migrates users into workspaces from a source Comet environment to a destination environment. It reads workspace membership from a chargeback report fetched from the source environment's admin API (or a pre-downloaded local file) and invites each user to the corresponding workspace in the destination environment by email. If a user with that email does not yet exist in the destination, they will be provisioned access once they sign up.
393
+
394
+ ```
395
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY [FLAGS ...]
396
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /path/to/report.json [FLAGS ...]
397
+ ```
398
+
399
+ **Arguments:**
400
+ * `--api-key API_KEY` - API key for the destination environment. Falls back to `COMET_API_KEY` if not provided.
401
+ * `--url URL` - Base URL of the destination Comet environment (e.g. `https://comet.example.com`). Required for self-hosted instances when the API key does not encode the server URL.
402
+ * `--source-api-key SOURCE_API_KEY` - API key for the source environment. Used to fetch the chargeback report. Required unless `--chargeback-report` is given.
403
+ * `--source-url SOURCE_URL` - Base URL of the source Comet environment. Required for self-hosted source instances when the source API key does not encode the server URL.
404
+ * `--chargeback-report PATH` - Path to a local chargeback report JSON file. When provided, `--source-api-key` is not required.
405
+
406
+ ### Flags
407
+
408
+ * `--create-workspaces` - Create workspaces on the destination if they don't already exist (default: off)
409
+ * `--dry-run` - Print what would happen without making any changes
410
+ * `--failures-output PATH` - Path to write failed operations JSON (default: `bulk_add_failures_by_email.json`)
411
+
412
+ ### Examples
413
+
414
+ ```bash
415
+ # Dry run — preview what would happen
416
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY --dry-run
417
+
418
+ # Execute the migration
419
+ cometx migrate-users --api-key DEST_KEY --source-api-key SOURCE_KEY
420
+
421
+ # Use a local chargeback report file
422
+ cometx migrate-users --api-key DEST_KEY --chargeback-report /tmp/chargeback_reports.json
423
+
424
+ # Self-hosted environments — provide explicit URLs
425
+ cometx migrate-users --api-key DEST_KEY --url https://comet.dest.example.com \
426
+ --source-api-key SOURCE_KEY --source-url https://comet.src.example.com
427
+ ```
428
+
429
+ For more information, `cometx migrate-users --help`
430
+
388
431
  ## cometx rename-duplicates
389
432
 
390
433
  This command is used to rename duplicate experiments within projects. When multiple experiments share the same name in a project, this command renames the duplicates to NAME-1, NAME-2, etc. while avoiding conflicts with existing names.
@@ -535,12 +578,8 @@ cometx admin chargeback-report [YEAR-MONTH]
535
578
  **Examples:**
536
579
  ```
537
580
  cometx admin chargeback-report
538
- <<<<<<< Updated upstream
539
- cometx admin usage-report
540
- =======
541
581
  cometx admin chargeback-report 2024-09
542
582
  ```
543
- >>>>>>> Stashed changes
544
583
 
545
584
  #### usage-report
546
585
 
@@ -35,6 +35,7 @@ cometx/cli/delete_assets.py
35
35
  cometx/cli/download.py
36
36
  cometx/cli/list_command.py
37
37
  cometx/cli/log.py
38
+ cometx/cli/migrate_users.py
38
39
  cometx/cli/rename_duplicates.py
39
40
  cometx/cli/reproduce.py
40
41
  cometx/cli/smoke_test.py
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes