cloudos-cli 2.27.0__tar.gz → 2.30.0__tar.gz

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.27.0
+Version: 2.30.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -535,6 +535,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -632,8 +695,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -649,7 +712,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -674,7 +737,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -771,6 +834,36 @@ If you require more information on the files and folder listed, you can use the
 - Filepath (the file or folder name)
 - S3 Path
 
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/README.md

@@ -500,6 +500,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -597,8 +660,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -614,7 +677,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -639,7 +702,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -736,6 +799,36 @@ If you require more information on the files and folder listed, you can use the
 - Filepath (the file or folder name)
 - S3 Path
 
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/cloudos_cli/__main__.py

@@ -3,17 +3,18 @@
 import rich_click as click
 import cloudos_cli.jobs.job as jb
 from cloudos_cli.clos import Cloudos
-from cloudos_cli.import_wf.import_wf import ImportGitlab, ImportGithub
+from cloudos_cli.import_wf.import_wf import ImportWorflow
 from cloudos_cli.queue.queue import Queue
+from cloudos_cli.utils.errors import BadRequestException
 import json
 import time
 import sys
 from ._version import __version__
 from cloudos_cli.configure.configure import ConfigurationProfile
-from cloudos_cli.datasets import Datasets
-from cloudos_cli.utils.resources import ssl_selector, format_bytes
 from rich.console import Console
 from rich.table import Table
+from cloudos_cli.datasets import Datasets
+from cloudos_cli.utils.resources import ssl_selector, format_bytes
 from rich.style import Style
 
 
@@ -59,6 +60,7 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'details': shared_config
             },
             'workflow': {
                 'list': shared_config,
@@ -79,7 +81,8 @@ def run_cloudos_cli(ctx):
                 'job': shared_config
             },
             'datasets': {
-                'ls': shared_config
+                'ls': shared_config,
+                'mv': shared_config
             }
         })
     else:
@@ -100,6 +103,7 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'details': shared_config
             },
             'workflow': {
                 'list': shared_config,
@@ -120,7 +124,8 @@ def run_cloudos_cli(ctx):
                 'job': shared_config
             },
             'datasets': {
-                'ls': shared_config
+                'ls': shared_config,
+                'mv': shared_config
             }
         })
 
@@ -284,7 +289,7 @@ def configure(ctx, profile, make_default):
               '--cromwell-token',
               help=('Specific Cromwell server authentication token. Currently, not necessary ' +
                     'as apikey can be used instead, but maintained for backwards compatibility.'))
-@click.option('--repository-platform',
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
               help='Name of the repository platform of the workflow. Default=github.',
               default='github')
 @click.option('--execution-platform',
@@ -696,6 +701,222 @@ def job_status(ctx,
           'or repeat the command you just used.')
 
 
+@job.command('details')
+@click.option('-k',
+              '--apikey',
+              help='Your CloudOS API key',
+              required=True)
+@click.option('-c',
+              '--cloudos-url',
+              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              default=CLOUDOS_URL)
+@click.option('--job-id',
+              help='The job id in CloudOS to search for.',
+              required=True)
+@click.option('--output-format',
+              help='The desired display for the output, either directly in standard output or saved as file. Default=stdout.',
+              type=click.Choice(['stdout', 'json'], case_sensitive=False),
+              default='stdout')
+@click.option('--output-basename',
+              help=('Output file base name to save jobs details. ' +
+                    'Default=job_details'),
+              default='job_details',
+              required=False)
+@click.option('--parameters',
+              help=('Whether to generate a ".config" file that can be used as input for --job-config parameter. ' +
+                    'It will have the same basename as defined in "--output-basename". '),
+              is_flag=True)
+@click.option('--verbose',
+              help='Whether to print information messages or not.',
+              is_flag=True)
+@click.option('--disable-ssl-verification',
+              help=('Disable SSL certificate verification. Please, remember that this option is ' +
+                    'not generally recommended for security reasons.'),
+              is_flag=True)
+@click.option('--ssl-cert',
+              help='Path to your SSL certificate file.')
+@click.option('--profile', help='Profile to use from the config file', default=None)
+@click.pass_context
+def job_details(ctx,
+               apikey,
+               cloudos_url,
+               job_id,
+               output_format,
+               output_basename,
+               parameters,
+               verbose,
+               disable_ssl_verification,
+               ssl_cert,
+               profile):
+    """Retrieve job details in CloudOS."""
+    profile = profile or ctx.default_map['job']['details']['profile']
+    # Create a dictionary with required and non-required params
+    required_dict = {
+        'apikey': True,
+        'workspace_id': False,
+        'workflow_name': False,
+        'project_name': False
+    }
+    # determine if the user provided all required parameters
+    config_manager = ConfigurationProfile()
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url
+        )
+    )
+
+    print('Executing details...')
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    if verbose:
+        print('\t...Preparing objects')
+    cl = Cloudos(cloudos_url, apikey, None)
+    if verbose:
+        print('\tThe following Cloudos object was created:')
+        print('\t' + str(cl) + '\n')
+        print(f'\tSearching for job id: {job_id}')
+
+    # check if the API gives a 403 error/forbidden error
+    try:
+        j_details = cl.get_job_status(job_id, verify_ssl)
+    except BadRequestException as e:
+        if '403' in str(e) or 'Forbidden' in str(e):
+            print("[Error] API can only show job details of your own jobs, cannot see other user's job details.")
+            sys.exit(1)
+    j_details_h = json.loads(j_details.content)
+
+    # Check if the job details contain parameters
+    if j_details_h["parameters"] != []:
+        param_kind_map = {
+            'textValue': 'textValue',
+            'arrayFileColumn': 'columnName',
+            'globPattern': 'globPattern',
+            'lustreFileSystem': 'fileSystem',
+        }
+        # there are different types of parameters, arrayFileColumn, globPattern, lustreFileSystem
+        # get first the type of parameter, then the value based on the parameter kind
+        concats = []
+        for param in j_details_h["parameters"]:
+            if param['parameterKind'] == 'dataItem':
+                # For dataItem, we need to use specific nested keys
+                concats.append(f"{param['prefix']}{param['name']}={param['dataItem']['item']['name']}")
+            else:
+                # For other parameter kinds, we use the appropriate key from param_kind_map
+                concats.append(f"{param['prefix']}{param['name']}={param[param_kind_map[param['parameterKind']]]}")
+        concat_string = '\n'.join(concats)
+        # If the user requested to save the parameters in a config file
+        if parameters:
+            # Create a config file with the parameters
+            config_filename = f"{output_basename}.config"
+            with open(config_filename, 'w') as config_file:
+                config_file.write("params {\n")
+                for param in j_details_h["parameters"]:
+                    config_file.write(f"\t{param['name']} = {param['textValue']}\n")
+                config_file.write("}\n")
+            print(f"\tJob parameters have been saved to '{config_filename}'")
+    else:
+        concat_string = 'No parameters provided'
+        if parameters:
+            print("\tNo parameters found in the job details, no config file will be created.")
+
+    # Determine the execution platform based on jobType
+    executors = {
+        'nextflowAWS':'Batch AWS',
+        'nextflowAzure': 'Batch Azure',
+        'nextflowGcp': 'GCP',
+        'nextflowHpc': 'HPC',
+        'nextflowKubernetes': 'Kubernetes',
+        'dockerAWS': 'Batch AWS',
+        'cromwellAWS': 'Batch AWS'
+    }
+    execution_platform = executors.get(j_details_h["jobType"], "None")
+
+    # revision
+    if j_details_h["jobType"] == "dockerAWS":
+        revision = j_details_h["revision"]["digest"]
+    else:
+        revision = j_details_h["revision"]["commit"]
+
+    # Output the job details
+    if output_format == 'stdout':
+        console = Console()
+        table = Table(title="Job Details")
+
+        table.add_column("Field", style="cyan", no_wrap=True)
+        table.add_column("Value", style="magenta", overflow="fold")
+
+        table.add_row("Job Status", str(j_details_h["status"]))
+        table.add_row("Parameters", concat_string)
+        if j_details_h["jobType"] == "dockerAWS":
+            table.add_row("Command", str(j_details_h["command"]))
+        table.add_row("Revision", str(revision))
+        table.add_row("Nextflow Version", str(j_details_h.get("nextflowVersion", "None")))
+        table.add_row("Execution Platform", execution_platform)
+        table.add_row("Profile", str(j_details_h.get("profile", "None")))
+        table.add_row("Master Instance", str(j_details_h["masterInstance"]["usedInstance"]["type"]))
+        if j_details_h["jobType"] == "nextflowAzure":
+            try:
+                table.add_row("Worker Node", str(j_details_h["azureBatch"]["vmType"]))
+            except KeyError:
+                table.add_row("Worker Node", "Not Specified")
+        table.add_row("Storage", str(j_details_h["storageSizeInGb"]) + " GB")
+        if j_details_h["jobType"] != "nextflowAzure":
+            try:
+                table.add_row("Job Queue ID", str(j_details_h["batch"]["jobQueue"]["name"]))
+                table.add_row("Job Queue Name", str(j_details_h["batch"]["jobQueue"]["label"]))
+            except KeyError:
+                table.add_row("Job Queue", "Master Node")
+        table.add_row("Accelerated File Staging", str(j_details_h.get("usesFusionFileSystem", "None")))
+        table.add_row("Task Resources", f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " +
+                                        f"{str(j_details_h['resourceRequirements']['ram'])} GB RAM")
+
+        console.print(table)
+    else:
+        # Create a JSON object with the key-value pairs
+        job_details_json = {
+            "Job Status": str(j_details_h["status"]),
+            "Parameters": ','.join(concat_string.split()),
+            "Revision": str(revision),
+            "Nextflow Version": str(j_details_h.get("nextflowVersion", "None")),
+            "Execution Platform": execution_platform,
+            "Profile": str(j_details_h.get("profile", "None")),
+            "Master Instance": str(j_details_h["masterInstance"]["usedInstance"]["type"]),
+            "Storage": str(j_details_h["storageSizeInGb"]) + " GB",
+            "Accelerated File Staging": str(j_details_h.get("usesFusionFileSystem", "None")),
+            "Task Resources": f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " + \
+                              f"{str(j_details_h['resourceRequirements']['ram'])} GB RAM"
+
+        }
+
+        # Conditionally add the "Command" key if the jobType is "dockerAWS"
+        if j_details_h["jobType"] == "dockerAWS":
+            job_details_json["Command"] = str(j_details_h["command"])
+
+        # Conditionally add the "Job Queue" key if the jobType is not "nextflowAzure"
+        if j_details_h["jobType"] != "nextflowAzure":
+            try:
+                job_details_json["Job Queue ID"] = str(j_details_h["batch"]["jobQueue"]["name"])
+                job_details_json["Job Queue Name"] = str(j_details_h["batch"]["jobQueue"]["label"])
+            except KeyError:
+                job_details_json["Job Queue"] = "Master Node"
+
+        if j_details_h["jobType"] == "nextflowAzure":
+            try:
+                job_details_json["Worker Node"] = str(j_details_h["azureBatch"]["vmType"])
+            except KeyError:
+                job_details_json["Worker Node"] = "Not Specified"
+
+        # Write the JSON object to a file
+        with open(f"{output_basename}.json", "w") as json_file:
+            json.dump(job_details_json, json_file, indent=4, ensure_ascii=False)
+        print(f"\tJob details have been saved to '{output_basename}.json'")
+
+
 @job.command('list')
 @click.option('-k',
               '--apikey',
@@ -1029,10 +1250,9 @@ def list_workflows(ctx,
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
-@click.option("--platform", type=click.Choice(["github", "gitlab"]),
-              help=('Repository service where the workflow is located. Valid choices: github, gitlab. ' +
-                    'Default=github'),
-              default="github")
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
+              help='Name of the repository platform of the workflow. Default=github.',
+              default='github')
 @click.option("--workflow-name", help="The name that the workflow will have in CloudOS.", required=True)
 @click.option("-w", "--workflow-url", help="URL of the workflow repository.", required=True)
 @click.option("-d", "--workflow-docs-link", help="URL to the documentation of the workflow.", default='')
@@ -1055,7 +1275,7 @@ def import_wf(ctx,
               workflow_docs_link,
               cost_limit,
               workflow_description,
-              platform,
+              repository_platform,
               disable_ssl_verification,
               ssl_cert,
               profile):
@@ -1082,15 +1302,14 @@ def import_wf(ctx,
             apikey=apikey,
             cloudos_url=cloudos_url,
             workspace_id=workspace_id,
-            workflow_name=workflow_name
+            workflow_name=workflow_name,
+            repository_platform=repository_platform
         )
     )
 
     verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
-    repo_services = {"gitlab": ImportGitlab, "github": ImportGithub}
-    repo_cls = repo_services[platform]
-    repo_import = repo_cls(cloudos_url=cloudos_url, cloudos_apikey=apikey, workspace_id=workspace_id,
-                             platform=platform, workflow_name=workflow_name, workflow_url=workflow_url,
+    repo_import = ImportWorflow(cloudos_url=cloudos_url, cloudos_apikey=apikey, workspace_id=workspace_id,
+                             platform=repository_platform, workflow_name=workflow_name, workflow_url=workflow_url,
                              workflow_docs_link=workflow_docs_link, cost_limit=cost_limit, workflow_description=workflow_description, verify=verify_ssl)
     workflow_id = repo_import.import_workflow()
     print(f'\tWorkflow {workflow_name} was imported successfully with the ' +
@@ -1644,7 +1863,7 @@ def remove_profile(ctx, profile):
               help=('Max time to wait (in seconds) to job completion. ' +
                     'Default=3600.'),
               default=3600)
-@click.option('--repository-platform',
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
               help='Name of the repository platform of the workflow. Default=github.',
               default='github')
 @click.option('--execution-platform',
@@ -1942,7 +2161,8 @@ def list_files(ctx,
             console = Console()
             for item in contents:
                 name = item.get("name", "")
-                if item.get("isDir"):
+                is_folder = item.get("folderType") or item.get("isDir")
+                if is_folder:
                     console.print(f"[blue underline]{name}[/]")
                 else:
                     console.print(name)
@@ -1950,5 +2170,154 @@ def list_files(ctx,
     except Exception as e:
         click.echo(f"[ERROR] {str(e)}", err=True)
 
+
+@datasets.command(name="mv")
+@click.argument("source_path", required=True)
+@click.argument("destination_path", required=True)
+@click.option('-k', '--apikey', required=True, help='Your CloudOS API key.')
+@click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=False, help='The CloudOS URL.')
+@click.option('--workspace-id', required=True, help='The CloudOS workspace ID.')
+@click.option('--project-name', required=True, help='The source project name.')
+@click.option('--destination-project-name', required=False, help='The destination project name. Defaults to the source project.')
+@click.option('--disable-ssl-verification', is_flag=True, help='Disable SSL certificate verification.')
+@click.option('--ssl-cert', help='Path to your SSL certificate file.')
+@click.option('--profile', default=None, help='Profile to use from the config file.')
+@click.pass_context
+def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspace_id,
+               project_name, destination_project_name,
+               disable_ssl_verification, ssl_cert, profile):
+    """
+    Move a file or folder from a source path to a destination path within or across CloudOS projects.
+
+    SOURCE_PATH [path] : the full path to the file or folder to move. It must be a 'Data' folder path. E.g.: 'Data/folderA/file.txt'\n
+    DESTINATION_PATH [path]: the full path to the destination folder. It must be a 'Data' folder path. E.g.: 'Data/folderB'
+    """
+
+    profile = profile or ctx.default_map['datasets']['move'].get('profile')
+    destination_project_name = destination_project_name or project_name
+
+    # Validate destination constraint
+    if not destination_path.strip("/").startswith("Data/") and destination_path.strip("/") != "Data":
+        click.echo("[ERROR] Destination path must begin with 'Data/' or be 'Data'.", err=True)
+        sys.exit(1)
+    if not source_path.strip("/").startswith("Data/") and source_path.strip("/") != "Data":
+        click.echo("[ERROR] SOURCE_PATH must start with  'Data/' or be 'Data'.", err=True)
+        sys.exit(1)
+    click.echo('Loading configuration profile')
+    # Load configuration profile
+    config_manager = ConfigurationProfile()
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': True
+    }
+
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id,
+            workflow_name=None,
+            repository_platform=None,
+            execution_platform=None,
+            project_name=project_name
+        )
+    )
+
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    # Initialize Datasets clients
+    source_client = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+
+    dest_client = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=destination_project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+    click.echo('Checking source path')
+    # === Resolve Source Item ===
+    source_parts = source_path.strip("/").split("/")
+    source_parent_path = "/".join(source_parts[:-1]) if len(source_parts) > 1 else None
+    source_item_name = source_parts[-1]
+
+    try:
+        source_contents = source_client.list_folder_content(source_parent_path)
+    except Exception as e:
+        click.echo(f"[ERROR] Could not resolve source path '{source_path}': {str(e)}", err=True)
+        sys.exit(1)
+
+    found_source = None
+    for collection in ["files", "folders"]:
+        for item in source_contents.get(collection, []):
+            if item.get("name") == source_item_name:
+                found_source = item
+                break
+        if found_source:
+            break
+    if not found_source:
+        click.echo(f"[ERROR] Item '{source_item_name}' not found in '{source_parent_path or '[project root]'}'", err=True)
+        sys.exit(1)
+
+    source_id = found_source["_id"]
+    source_kind = "Folder" if "folderType" in found_source else "File"
+    click.echo("Checking destination path")
+    # === Resolve Destination Folder ===
+    dest_parts = destination_path.strip("/").split("/")
+    dest_folder_name = dest_parts[-1]
+    dest_parent_path = "/".join(dest_parts[:-1]) if len(dest_parts) > 1 else None
+
+    try:
+        dest_contents = dest_client.list_folder_content(dest_parent_path)
+        match = next((f for f in dest_contents.get("folders", []) if f.get("name") == dest_folder_name), None)
+        if not match:
+            raise ValueError(f"Could not resolve destination folder '{destination_path}'")
+
+        target_id = match["_id"]
+        folder_type = match.get("folderType")
+        # Normalize kind: top-level datasets are kind=Dataset, all other folders are kind=Folder
+        if folder_type in ("VirtualFolder", "S3Folder", "Folder"):
+            target_kind = "Folder"
+        elif isinstance(folder_type, bool) and folder_type:  # legacy dataset structure
+            target_kind = "Dataset"
+        else:
+            raise ValueError(f"Unrecognized folderType '{folder_type}' for destination '{destination_path}'")
+
+    except Exception as e:
+        click.echo(f"[ERROR] Could not resolve destination path '{destination_path}': {str(e)}", err=True)
+        sys.exit(1)
+    click.echo(f"Moving {source_kind} '{source_item_name}' to '{destination_path}' in project '{destination_project_name} ...")
+    # === Perform Move ===
+    try:
+        response = source_client.move_files_and_folders(
+            source_id=source_id,
+            source_kind=source_kind,
+            target_id=target_id,
+            target_kind=target_kind
+        )
+        if response.ok:
+           click.secho(f"[SUCCESS] {source_kind} '{source_item_name}' moved to '{destination_path}' in project '{destination_project_name}'.", fg="green", bold=True)
+        else:
+            click.echo(f"[ERROR] Move failed: {response.status_code} - {response.text}", err=True)
+            sys.exit(1)
+    except Exception as e:
+        click.echo(f"[ERROR] Move operation failed: {str(e)}", err=True)
+        sys.exit(1)
+
+
 if __name__ == "__main__":
     run_cloudos_cli()

cloudos_cli-2.30.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.30.0'

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/cloudos_cli/datasets/datasets.py

@@ -5,8 +5,8 @@ This is the main class for file explorer (datasets).
 from dataclasses import dataclass
 from typing import Union
 from cloudos_cli.clos import Cloudos
-from cloudos_cli.utils.requests import retry_requests_get
-
+from cloudos_cli.utils.requests import retry_requests_get, retry_requests_put
+import json
 
 @dataclass
 class Datasets(Cloudos):
@@ -151,11 +151,15 @@ class Datasets(Cloudos):
                                                                                   self.workspace_id),
                                headers=headers, verify=self.verify)
         raw = r.json()
-
+        datasets = raw.get("datasets", [])
         #  Normalize response
-        for item in raw.get("datasets", []):
+        for item in datasets:
             item["folderType"] = True
-        return raw
+        response ={
+                "folders": datasets,
+                "files": []
+            }
+        return response
 
     def list_datasets_content(self, folder_name):
         """Uses
@@ -182,7 +186,7 @@ class Datasets(Cloudos):
         if folder_name == 'AnalysesResults':
             folder_name = 'Analyses Results'
 
-        for folder in pro_fol.get("datasets", []):
+        for folder in pro_fol.get("folders", []):
             if folder['name'] == folder_name:
                 folder_id = folder['_id']
         if not folder_id:
@@ -262,6 +266,7 @@ class Datasets(Cloudos):
                                                                                      self.workspace_id),
                                 headers=headers, verify=self.verify)
         return r.json()
+    
     def list_folder_content(self, path=None):
         """
         Wrapper to list contents of a CloudOS folder.
@@ -325,4 +330,40 @@ class Datasets(Cloudos):
             if not found:
                 raise ValueError(f"Folder '{job_name}' not found under dataset '{dataset_name}'")
 
-        return folder_content
+        return folder_content
+    
+    def move_files_and_folders(self, source_id: str, source_kind: str, target_id: str, target_kind: str):
+        """
+        Move a file to another dataset in CloudOS.
+
+        Parameters
+        ----------
+        file_id : str
+            The ID of the file to move.
+
+        target_dataset_id : str
+            The ID of the target dataset to move the file into.
+
+        Returns
+        -------
+        response : requests.Response
+            The response object from the CloudOS API.
+        """
+        url = f"{self.cloudos_url}/api/v1/dataItems/move?teamId={self.workspace_id}"
+        headers = {
+            "accept": "application/json",
+            "content-type": "application/json",
+            "ApiKey": self.apikey
+        }
+        payload = {
+            "dataItemToMove": {
+                "kind": source_kind,
+                "item": source_id
+            },
+            "toDataItemParent": {
+                "kind": target_kind,
+                "item": target_id
+            }
+        }
+        response = retry_requests_put(url, headers=headers, data=json.dumps(payload), verify=self.verify)
+        return response

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/cloudos_cli/import_wf/__init__.py

@@ -2,7 +2,7 @@
 Functions and classes related to importing workflows into CloudOS.
 """
 
-from .import_wf import WFImport, ImportGitlab, ImportGithub
+from .import_wf import WFImport, ImportWorflow
 
 
 __all__ = ['import_wf']

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/cloudos_cli/import_wf/import_wf.py

@@ -3,6 +3,8 @@ from urllib.parse import urlsplit
 from cloudos_cli.utils.errors import BadRequestException, AccountNotLinkedException
 from cloudos_cli.utils.requests import retry_requests_post, retry_requests_get
 import json
+from requests.exceptions import RetryError
+import sys
 
 
 class WFImport(ABC):
@@ -96,44 +98,46 @@ class WFImport(ABC):
         return content["_id"]
 
 
-# There are some duplicated lines here and on the github subclass. I did not put them in the abstract class because we
-# still don't know if the bitbucket data will come the same. If it does, then I will put as much as possible as part
-# of the abstract class
-class ImportGitlab(WFImport):
+class ImportWorflow(WFImport):
     def get_repo(self):
-        get_repo_url = f"{self.cloudos_url}/api/v1/git/gitlab/getPublicRepo"
-        self.repo_name = self.parsed_url.path.split("/")[-1]
-        self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
+        get_repo_url = f"{self.cloudos_url}/api/v1/git/{self.platform}/getPublicRepo"
+        if self.platform == "bitbucketServer":
+            # platform allows to add paths like /browse, so we need to check if the path ends with it
+            if self.parsed_url.path.endswith("browse"):
+                self.repo_name = self.parsed_url.path.split("/")[-2]
+            else:
+                self.repo_name = self.parsed_url.path.split("/")[-1]
+            self.repo_owner = self.parsed_url.path.split("/")[2]
+        else:
+            self.repo_name = self.parsed_url.path.split("/")[-1]
+            self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
         self.repo_host = f"{self.parsed_url.scheme}://{self.parsed_url.netloc}"
         get_repo_params = dict(repoName=self.repo_name, repoOwner=self.repo_owner, host=self.repo_host, teamId=self.workspace_id)
-        r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
-        if r.status_code == 404:
+        try:
+            r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
+        except RetryError as e:
+            # RetryError getting from missing BitBucket Server credentials
             raise AccountNotLinkedException(self.workflow_url)
-        elif r.status_code >= 400:
-            raise BadRequestException(r)
-        r_data = r.json()
-        self.payload["repository"]["repositoryId"] = r_data["id"]
-        self.payload["repository"]["name"] = r_data["name"]
-        self.payload["repository"]["owner"]["id"] = r_data["namespace"]["id"]
-        self.payload["repository"]["owner"]["login"] = r_data["namespace"]["full_path"]
-        self.payload["mainFile"] = self.main_file or self.get_repo_main_file()
 
-
-class ImportGithub(WFImport):
-    def get_repo(self):
-        get_repo_url = f"{self.cloudos_url}/api/v1/git/github/getPublicRepo"
-        self.repo_name = self.parsed_url.path.split("/")[-1]
-        self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
-        self.repo_host = f"{self.parsed_url.scheme}://{self.parsed_url.netloc}"
-        get_repo_params = dict(repoName=self.repo_name, repoOwner=self.repo_owner, host=self.repo_host, teamId=self.workspace_id)
-        r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
+        # for Github and Gitlab the API gives very general errors on missing credentials
+        # therefore we only have these at the moment
         if r.status_code == 404:
             raise AccountNotLinkedException(self.workflow_url)
         elif r.status_code >= 400:
             raise BadRequestException(r)
+
         r_data = r.json()
-        self.payload["repository"]["repositoryId"] = r_data["id"]
+        if self.platform == "bitbucketServer":
+            self.payload["repository"]["repositoryId"] = r_data["name"]
+        else:
+            self.payload["repository"]["repositoryId"] = r_data["id"]
         self.payload["repository"]["name"] = r_data["name"]
-        self.payload["repository"]["owner"]["id"] = r_data["owner"]["id"]
-        self.payload["repository"]["owner"]["login"] = r_data["owner"]["login"]
+        owner_data = {
+            "bitbucketServer": ("project", "id", "key"),
+            "gitlab": ("namespace", "id", "full_path"),
+            "github": ("owner", "id", "login")
+        }
+        key, id_field, login_field = owner_data[self.platform]
+        self.payload["repository"]["owner"]["id"] = r_data[key][id_field]
+        self.payload["repository"]["owner"]["login"] = r_data[key][login_field]
         self.payload["mainFile"] = self.main_file or self.get_repo_main_file()

{cloudos_cli-2.27.0 → cloudos_cli-2.30.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.27.0
+Version: 2.30.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -535,6 +535,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -632,8 +695,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -649,7 +712,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -674,7 +737,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -771,6 +834,36 @@ If you require more information on the files and folder listed, you can use the
 - Filepath (the file or folder name)
 - S3 Path
 
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

cloudos_cli-2.27.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.27.0'