cloudos-cli 2.26.1__tar.gz → 2.29.0__tar.gz

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.26.1
+Version: 2.29.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -535,6 +535,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -758,9 +821,49 @@ cloudos datasets ls <path> \
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+
 The output of this command is a list of files and folders present in the specified project.
 If the `<path>` is left empty, the command will return the list of folders present in the selected project.
 
+If you require more information on the files and folder listed, you can use the `--details` flag that will output a table containing the following columns:
+- Type (folder or file)
+- Owner
+- Size in human readable format
+- Last updated
+- Filepath (the file or folder name)
+- S3 Path
+
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/README.md

@@ -500,6 +500,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -723,9 +786,49 @@ cloudos datasets ls <path> \
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+
 The output of this command is a list of files and folders present in the specified project.
 If the `<path>` is left empty, the command will return the list of folders present in the selected project.
 
+If you require more information on the files and folder listed, you can use the `--details` flag that will output a table containing the following columns:
+- Type (folder or file)
+- Owner
+- Size in human readable format
+- Last updated
+- Filepath (the file or folder name)
+- S3 Path
+
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/cloudos_cli/__main__.py

@@ -5,14 +5,18 @@ import cloudos_cli.jobs.job as jb
 from cloudos_cli.clos import Cloudos
 from cloudos_cli.import_wf.import_wf import ImportGitlab, ImportGithub
 from cloudos_cli.queue.queue import Queue
+from cloudos_cli.utils.errors import BadRequestException
 import json
 import time
 import sys
-import os
-import urllib3
 from ._version import __version__
 from cloudos_cli.configure.configure import ConfigurationProfile
+from rich.console import Console
+from rich.table import Table
 from cloudos_cli.datasets import Datasets
+from cloudos_cli.utils.resources import ssl_selector, format_bytes
+from rich.style import Style
+
 
 # GLOBAL VARS
 JOB_COMPLETED = 'completed'
@@ -27,39 +31,6 @@ ABORT_JOB_STATES = ['running', 'initializing']
 CLOUDOS_URL = 'https://cloudos.lifebit.ai'
 INIT_PROFILE = 'initialisingProfile'
 
-
-def ssl_selector(disable_ssl_verification, ssl_cert):
-    """Verify value selector.
-
-    This function stablish the value that will be passed to requests.verify
-    variable.
-
-    Parameters
-    ----------
-    disable_ssl_verification : bool
-        Whether to disable SSL verification.
-    ssl_cert : string
-        String indicating the path to the SSL certificate file to use.
-
-    Returns
-    -------
-    verify_ssl : [bool | string]
-        Either a bool or a path string to be passed to requests.verify to control
-        SSL verification.
-    """
-    if disable_ssl_verification:
-        verify_ssl = False
-        print('[WARNING] Disabling SSL verification')
-        urllib3.disable_warnings()
-    elif ssl_cert is None:
-        verify_ssl = True
-    elif os.path.isfile(ssl_cert):
-        verify_ssl = ssl_cert
-    else:
-        raise FileNotFoundError(f"The specified file '{ssl_cert}' was not found")
-    return verify_ssl
-
-
 @click.group()
 @click.version_option(__version__)
 @click.pass_context
@@ -89,6 +60,7 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'details': shared_config
             },
             'workflow': {
                 'list': shared_config,
@@ -109,7 +81,8 @@ def run_cloudos_cli(ctx):
                 'job': shared_config
             },
             'datasets': {
-                'ls': shared_config
+                'ls': shared_config,
+                'mv': shared_config
             }
         })
     else:
@@ -130,6 +103,7 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'details': shared_config
             },
             'workflow': {
                 'list': shared_config,
@@ -150,7 +124,8 @@ def run_cloudos_cli(ctx):
                 'job': shared_config
             },
             'datasets': {
-                'ls': shared_config
+                'ls': shared_config,
+                'mv': shared_config
             }
         })
 
@@ -726,6 +701,222 @@ def job_status(ctx,
           'or repeat the command you just used.')
 
 
+@job.command('details')
+@click.option('-k',
+              '--apikey',
+              help='Your CloudOS API key',
+              required=True)
+@click.option('-c',
+              '--cloudos-url',
+              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              default=CLOUDOS_URL)
+@click.option('--job-id',
+              help='The job id in CloudOS to search for.',
+              required=True)
+@click.option('--output-format',
+              help='The desired display for the output, either directly in standard output or saved as file. Default=stdout.',
+              type=click.Choice(['stdout', 'json'], case_sensitive=False),
+              default='stdout')
+@click.option('--output-basename',
+              help=('Output file base name to save jobs details. ' +
+                    'Default=job_details'),
+              default='job_details',
+              required=False)
+@click.option('--parameters',
+              help=('Whether to generate a ".config" file that can be used as input for --job-config parameter. ' +
+                    'It will have the same basename as defined in "--output-basename". '),
+              is_flag=True)
+@click.option('--verbose',
+              help='Whether to print information messages or not.',
+              is_flag=True)
+@click.option('--disable-ssl-verification',
+              help=('Disable SSL certificate verification. Please, remember that this option is ' +
+                    'not generally recommended for security reasons.'),
+              is_flag=True)
+@click.option('--ssl-cert',
+              help='Path to your SSL certificate file.')
+@click.option('--profile', help='Profile to use from the config file', default=None)
+@click.pass_context
+def job_details(ctx,
+               apikey,
+               cloudos_url,
+               job_id,
+               output_format,
+               output_basename,
+               parameters,
+               verbose,
+               disable_ssl_verification,
+               ssl_cert,
+               profile):
+    """Retrieve job details in CloudOS."""
+    profile = profile or ctx.default_map['job']['details']['profile']
+    # Create a dictionary with required and non-required params
+    required_dict = {
+        'apikey': True,
+        'workspace_id': False,
+        'workflow_name': False,
+        'project_name': False
+    }
+    # determine if the user provided all required parameters
+    config_manager = ConfigurationProfile()
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url
+        )
+    )
+
+    print('Executing details...')
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    if verbose:
+        print('\t...Preparing objects')
+    cl = Cloudos(cloudos_url, apikey, None)
+    if verbose:
+        print('\tThe following Cloudos object was created:')
+        print('\t' + str(cl) + '\n')
+        print(f'\tSearching for job id: {job_id}')
+
+    # check if the API gives a 403 error/forbidden error
+    try:
+        j_details = cl.get_job_status(job_id, verify_ssl)
+    except BadRequestException as e:
+        if '403' in str(e) or 'Forbidden' in str(e):
+            print("[Error] API can only show job details of your own jobs, cannot see other user's job details.")
+            sys.exit(1)
+    j_details_h = json.loads(j_details.content)
+
+    # Check if the job details contain parameters
+    if j_details_h["parameters"] != []:
+        param_kind_map = {
+            'textValue': 'textValue',
+            'arrayFileColumn': 'columnName',
+            'globPattern': 'globPattern',
+            'lustreFileSystem': 'fileSystem',
+        }
+        # there are different types of parameters, arrayFileColumn, globPattern, lustreFileSystem
+        # get first the type of parameter, then the value based on the parameter kind
+        concats = []
+        for param in j_details_h["parameters"]:
+            if param['parameterKind'] == 'dataItem':
+                # For dataItem, we need to use specific nested keys
+                concats.append(f"{param['prefix']}{param['name']}={param['dataItem']['item']['name']}")
+            else:
+                # For other parameter kinds, we use the appropriate key from param_kind_map
+                concats.append(f"{param['prefix']}{param['name']}={param[param_kind_map[param['parameterKind']]]}")
+        concat_string = '\n'.join(concats)
+        # If the user requested to save the parameters in a config file
+        if parameters:
+            # Create a config file with the parameters
+            config_filename = f"{output_basename}.config"
+            with open(config_filename, 'w') as config_file:
+                config_file.write("params {\n")
+                for param in j_details_h["parameters"]:
+                    config_file.write(f"\t{param['name']} = {param['textValue']}\n")
+                config_file.write("}\n")
+            print(f"\tJob parameters have been saved to '{config_filename}'")
+    else:
+        concat_string = 'No parameters provided'
+        if parameters:
+            print("\tNo parameters found in the job details, no config file will be created.")
+
+    # Determine the execution platform based on jobType
+    executors = {
+        'nextflowAWS':'Batch AWS',
+        'nextflowAzure': 'Batch Azure',
+        'nextflowGcp': 'GCP',
+        'nextflowHpc': 'HPC',
+        'nextflowKubernetes': 'Kubernetes',
+        'dockerAWS': 'Batch AWS',
+        'cromwellAWS': 'Batch AWS'
+    }
+    execution_platform = executors.get(j_details_h["jobType"], "None")
+
+    # revision
+    if j_details_h["jobType"] == "dockerAWS":
+        revision = j_details_h["revision"]["digest"]
+    else:
+        revision = j_details_h["revision"]["commit"]
+
+    # Output the job details
+    if output_format == 'stdout':
+        console = Console()
+        table = Table(title="Job Details")
+
+        table.add_column("Field", style="cyan", no_wrap=True)
+        table.add_column("Value", style="magenta", overflow="fold")
+
+        table.add_row("Job Status", str(j_details_h["status"]))
+        table.add_row("Parameters", concat_string)
+        if j_details_h["jobType"] == "dockerAWS":
+            table.add_row("Command", str(j_details_h["command"]))
+        table.add_row("Revision", str(revision))
+        table.add_row("Nextflow Version", str(j_details_h.get("nextflowVersion", "None")))
+        table.add_row("Execution Platform", execution_platform)
+        table.add_row("Profile", str(j_details_h.get("profile", "None")))
+        table.add_row("Master Instance", str(j_details_h["masterInstance"]["usedInstance"]["type"]))
+        if j_details_h["jobType"] == "nextflowAzure":
+            try:
+                table.add_row("Worker Node", str(j_details_h["azureBatch"]["vmType"]))
+            except KeyError:
+                table.add_row("Worker Node", "Not Specified")
+        table.add_row("Storage", str(j_details_h["storageSizeInGb"]) + " GB")
+        if j_details_h["jobType"] != "nextflowAzure":
+            try:
+                table.add_row("Job Queue ID", str(j_details_h["batch"]["jobQueue"]["name"]))
+                table.add_row("Job Queue Name", str(j_details_h["batch"]["jobQueue"]["label"]))
+            except KeyError:
+                table.add_row("Job Queue", "Master Node")
+        table.add_row("Accelerated File Staging", str(j_details_h.get("usesFusionFileSystem", "None")))
+        table.add_row("Task Resources", f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " +
+                                        f"{str(j_details_h['resourceRequirements']['ram'])} GB RAM")
+
+        console.print(table)
+    else:
+        # Create a JSON object with the key-value pairs
+        job_details_json = {
+            "Job Status": str(j_details_h["status"]),
+            "Parameters": ','.join(concat_string.split()),
+            "Revision": str(revision),
+            "Nextflow Version": str(j_details_h.get("nextflowVersion", "None")),
+            "Execution Platform": execution_platform,
+            "Profile": str(j_details_h.get("profile", "None")),
+            "Master Instance": str(j_details_h["masterInstance"]["usedInstance"]["type"]),
+            "Storage": str(j_details_h["storageSizeInGb"]) + " GB",
+            "Accelerated File Staging": str(j_details_h.get("usesFusionFileSystem", "None")),
+            "Task Resources": f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " + \
+                              f"{str(j_details_h['resourceRequirements']['ram'])} GB RAM"
+
+        }
+
+        # Conditionally add the "Command" key if the jobType is "dockerAWS"
+        if j_details_h["jobType"] == "dockerAWS":
+            job_details_json["Command"] = str(j_details_h["command"])
+
+        # Conditionally add the "Job Queue" key if the jobType is not "nextflowAzure"
+        if j_details_h["jobType"] != "nextflowAzure":
+            try:
+                job_details_json["Job Queue ID"] = str(j_details_h["batch"]["jobQueue"]["name"])
+                job_details_json["Job Queue Name"] = str(j_details_h["batch"]["jobQueue"]["label"])
+            except KeyError:
+                job_details_json["Job Queue"] = "Master Node"
+
+        if j_details_h["jobType"] == "nextflowAzure":
+            try:
+                job_details_json["Worker Node"] = str(j_details_h["azureBatch"]["vmType"])
+            except KeyError:
+                job_details_json["Worker Node"] = "Not Specified"
+
+        # Write the JSON object to a file
+        with open(f"{output_basename}.json", "w") as json_file:
+            json.dump(job_details_json, json_file, indent=4, ensure_ascii=False)
+        print(f"\tJob details have been saved to '{output_basename}.json'")
+
+
 @job.command('list')
 @click.option('-k',
               '--apikey',
@@ -1855,6 +2046,11 @@ def run_bash_job(ctx,
 @click.option('--project-name',
               help='The name of a CloudOS project.')
 @click.option('--profile', help='Profile to use from the config file', default=None)
+@click.option('--details',
+              help=('When selected, it prints the details of the listed files. ' +
+                    'Details contains "Type", "Owner", "Size", "Last Updated", ' +
+                    '"Filepath", "S3 Path".'),
+              is_flag=True)
 @click.pass_context
 def list_files(ctx,
                apikey,
@@ -1864,14 +2060,14 @@ def list_files(ctx,
                ssl_cert,
                project_name,
                profile,
-               path):
+               path,
+               details):
     """List contents of a path within a CloudOS workspace dataset."""
 
     # fallback to ctx default if profile not specified
     profile = profile or ctx.default_map['datasets']['list'].get('profile')
 
     config_manager = ConfigurationProfile()
-
     required_dict = {
         'apikey': True,
         'workspace_id': True,
@@ -1879,7 +2075,6 @@ def list_files(ctx,
         'project_name': False
     }
 
-    # Unpack profile values first
     apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
         config_manager.load_profile_and_validate_data(
             ctx,
@@ -1912,21 +2107,219 @@ def list_files(ctx,
         result = datasets.list_folder_content(path)
         contents = result.get("contents") or result.get("datasets", [])
         if not contents:
-            files = result.get("files", [])
-            folders = result.get("folders", [])
-            contents = [{"name": f["name"], "isDir": False} for f in files] + \
-                       [{"name": f["name"], "isDir": True} for f in folders]
+            contents = result.get("files", []) + result.get("folders", [])
+
+        if details:
+            console = Console(width=None)  # Avoid terminal width truncation
+
+            table = Table(show_header=True, header_style="bold white")
+            table.add_column("Type", style="cyan", no_wrap=True)
+            table.add_column("Owner", style="white")
+            table.add_column("Size", style="magenta")
+            table.add_column("Last Updated", style="green")
+            table.add_column("Filepath", style="bold", overflow="fold")
+            table.add_column("S3 Path", style="dim", no_wrap=False, overflow="fold", ratio=2)
+
+            for item in contents:
+                is_folder = "folderType" in item or item.get("isDir", False)
+                type_ = "folder" if is_folder else "file"
+
+                user = item.get("user")
+                if isinstance(user, dict):
+                    name = user.get("name", "").strip()
+                    surname = user.get("surname", "").strip()
+                    if name and surname:
+                        owner = f"{name} {surname}"
+                    elif name:
+                        owner = name
+                    elif surname:
+                        owner = surname
+                    else:
+                        owner = "-"
+                else:
+                    owner = "-"
+
+                raw_size = item.get("sizeInBytes", item.get("size"))
+                size = format_bytes(raw_size) if not is_folder and raw_size is not None else "-"
+
+                updated = item.get("updatedAt") or item.get("lastModified", "-")
+                filepath = item.get("name", "-")
+
+                if is_folder:
+                    s3_bucket = item.get("s3BucketName")
+                    s3_key = item.get("s3Prefix")
+                    s3_path = f"s3://{s3_bucket}/{s3_key}" if s3_bucket and s3_key else "-"
+                else:
+                    s3_bucket = item.get("s3BucketName")
+                    s3_key = item.get("s3ObjectKey") or item.get("s3Prefix")
+                    s3_path = f"s3://{s3_bucket}/{s3_key}" if s3_bucket and s3_key else "-"
+
+                style = Style(color="blue", underline=True) if is_folder else None
+                table.add_row(type_, owner, size, updated, filepath, s3_path, style=style)
+
+            console.print(table)
 
-        for item in contents:
-            name = item.get("name", "")
-            if item.get("isDir"):
-                name = click.style(name, fg="blue", underline=True)
-            click.echo(name)
+        else:
+            console = Console()
+            for item in contents:
+                name = item.get("name", "")
+                is_folder = item.get("folderType") or item.get("isDir")
+                if is_folder:
+                    console.print(f"[blue underline]{name}[/]")
+                else:
+                    console.print(name)
 
     except Exception as e:
         click.echo(f"[ERROR] {str(e)}", err=True)
 
 
+@datasets.command(name="mv")
+@click.argument("source_path", required=True)
+@click.argument("destination_path", required=True)
+@click.option('-k', '--apikey', required=True, help='Your CloudOS API key.')
+@click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=False, help='The CloudOS URL.')
+@click.option('--workspace-id', required=True, help='The CloudOS workspace ID.')
+@click.option('--project-name', required=True, help='The source project name.')
+@click.option('--destination-project-name', required=False, help='The destination project name. Defaults to the source project.')
+@click.option('--disable-ssl-verification', is_flag=True, help='Disable SSL certificate verification.')
+@click.option('--ssl-cert', help='Path to your SSL certificate file.')
+@click.option('--profile', default=None, help='Profile to use from the config file.')
+@click.pass_context
+def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspace_id,
+               project_name, destination_project_name,
+               disable_ssl_verification, ssl_cert, profile):
+    """
+    Move a file or folder from a source path to a destination path within or across CloudOS projects.
+
+    SOURCE_PATH [path] : the full path to the file or folder to move. It must be a 'Data' folder path. E.g.: 'Data/folderA/file.txt'\n
+    DESTINATION_PATH [path]: the full path to the destination folder. It must be a 'Data' folder path. E.g.: 'Data/folderB'
+    """
+
+    profile = profile or ctx.default_map['datasets']['move'].get('profile')
+    destination_project_name = destination_project_name or project_name
+
+    # Validate destination constraint
+    if not destination_path.strip("/").startswith("Data/") and destination_path.strip("/") != "Data":
+        click.echo("[ERROR] Destination path must begin with 'Data/' or be 'Data'.", err=True)
+        sys.exit(1)
+    if not source_path.strip("/").startswith("Data/") and source_path.strip("/") != "Data":
+        click.echo("[ERROR] SOURCE_PATH must start with  'Data/' or be 'Data'.", err=True)
+        sys.exit(1)
+    click.echo('Loading configuration profile')
+    # Load configuration profile
+    config_manager = ConfigurationProfile()
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': True
+    }
+
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id,
+            workflow_name=None,
+            repository_platform=None,
+            execution_platform=None,
+            project_name=project_name
+        )
+    )
+
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    # Initialize Datasets clients
+    source_client = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+
+    dest_client = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=destination_project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+    click.echo('Checking source path')
+    # === Resolve Source Item ===
+    source_parts = source_path.strip("/").split("/")
+    source_parent_path = "/".join(source_parts[:-1]) if len(source_parts) > 1 else None
+    source_item_name = source_parts[-1]
+
+    try:
+        source_contents = source_client.list_folder_content(source_parent_path)
+    except Exception as e:
+        click.echo(f"[ERROR] Could not resolve source path '{source_path}': {str(e)}", err=True)
+        sys.exit(1)
+
+    found_source = None
+    for collection in ["files", "folders"]:
+        for item in source_contents.get(collection, []):
+            if item.get("name") == source_item_name:
+                found_source = item
+                break
+        if found_source:
+            break
+    if not found_source:
+        click.echo(f"[ERROR] Item '{source_item_name}' not found in '{source_parent_path or '[project root]'}'", err=True)
+        sys.exit(1)
+
+    source_id = found_source["_id"]
+    source_kind = "Folder" if "folderType" in found_source else "File"
+    click.echo("Checking destination path")
+    # === Resolve Destination Folder ===
+    dest_parts = destination_path.strip("/").split("/")
+    dest_folder_name = dest_parts[-1]
+    dest_parent_path = "/".join(dest_parts[:-1]) if len(dest_parts) > 1 else None
+
+    try:
+        dest_contents = dest_client.list_folder_content(dest_parent_path)
+        match = next((f for f in dest_contents.get("folders", []) if f.get("name") == dest_folder_name), None)
+        if not match:
+            raise ValueError(f"Could not resolve destination folder '{destination_path}'")
+
+        target_id = match["_id"]
+        folder_type = match.get("folderType")
+        # Normalize kind: top-level datasets are kind=Dataset, all other folders are kind=Folder
+        if folder_type in ("VirtualFolder", "S3Folder", "Folder"):
+            target_kind = "Folder"
+        elif isinstance(folder_type, bool) and folder_type:  # legacy dataset structure
+            target_kind = "Dataset"
+        else:
+            raise ValueError(f"Unrecognized folderType '{folder_type}' for destination '{destination_path}'")
+
+    except Exception as e:
+        click.echo(f"[ERROR] Could not resolve destination path '{destination_path}': {str(e)}", err=True)
+        sys.exit(1)
+    click.echo(f"Moving {source_kind} '{source_item_name}' to '{destination_path}' in project '{destination_project_name} ...")
+    # === Perform Move ===
+    try:
+        response = source_client.move_files_and_folders(
+            source_id=source_id,
+            source_kind=source_kind,
+            target_id=target_id,
+            target_kind=target_kind
+        )
+        if response.ok:
+           click.secho(f"[SUCCESS] {source_kind} '{source_item_name}' moved to '{destination_path}' in project '{destination_project_name}'.", fg="green", bold=True)
+        else:
+            click.echo(f"[ERROR] Move failed: {response.status_code} - {response.text}", err=True)
+            sys.exit(1)
+    except Exception as e:
+        click.echo(f"[ERROR] Move operation failed: {str(e)}", err=True)
+        sys.exit(1)
+
+
 if __name__ == "__main__":
     run_cloudos_cli()
-

cloudos_cli-2.29.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.29.0'

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/cloudos_cli/datasets/datasets.py

@@ -5,8 +5,8 @@ This is the main class for file explorer (datasets).
 from dataclasses import dataclass
 from typing import Union
 from cloudos_cli.clos import Cloudos
-from cloudos_cli.utils.requests import retry_requests_get
-
+from cloudos_cli.utils.requests import retry_requests_get, retry_requests_put
+import json
 
 @dataclass
 class Datasets(Cloudos):
@@ -150,7 +150,16 @@ class Datasets(Cloudos):
                                                                                   self.project_id,
                                                                                   self.workspace_id),
                                headers=headers, verify=self.verify)
-        return r.json()
+        raw = r.json()
+        datasets = raw.get("datasets", [])
+        #  Normalize response
+        for item in datasets:
+            item["folderType"] = True
+        response ={
+                "folders": datasets,
+                "files": []
+            }
+        return response
 
     def list_datasets_content(self, folder_name):
         """Uses
@@ -177,7 +186,7 @@ class Datasets(Cloudos):
         if folder_name == 'AnalysesResults':
             folder_name = 'Analyses Results'
 
-        for folder in pro_fol.get("datasets", []):
+        for folder in pro_fol.get("folders", []):
             if folder['name'] == folder_name:
                 folder_id = folder['_id']
         if not folder_id:
@@ -187,6 +196,7 @@ class Datasets(Cloudos):
                                                                               self.workspace_id),
                                 headers=headers, verify=self.verify)
         return r.json()
+
     def list_s3_folder_content(self, s3_bucket_name, s3_relative_path):
         """Uses
         ----------
@@ -256,6 +266,7 @@ class Datasets(Cloudos):
                                                                                      self.workspace_id),
                                 headers=headers, verify=self.verify)
         return r.json()
+    
     def list_folder_content(self, path=None):
         """
         Wrapper to list contents of a CloudOS folder.
@@ -319,4 +330,40 @@ class Datasets(Cloudos):
             if not found:
                 raise ValueError(f"Folder '{job_name}' not found under dataset '{dataset_name}'")
 
-        return folder_content
+        return folder_content
+    
+    def move_files_and_folders(self, source_id: str, source_kind: str, target_id: str, target_kind: str):
+        """
+        Move a file to another dataset in CloudOS.
+
+        Parameters
+        ----------
+        file_id : str
+            The ID of the file to move.
+
+        target_dataset_id : str
+            The ID of the target dataset to move the file into.
+
+        Returns
+        -------
+        response : requests.Response
+            The response object from the CloudOS API.
+        """
+        url = f"{self.cloudos_url}/api/v1/dataItems/move?teamId={self.workspace_id}"
+        headers = {
+            "accept": "application/json",
+            "content-type": "application/json",
+            "ApiKey": self.apikey
+        }
+        payload = {
+            "dataItemToMove": {
+                "kind": source_kind,
+                "item": source_id
+            },
+            "toDataItemParent": {
+                "kind": target_kind,
+                "item": target_id
+            }
+        }
+        response = retry_requests_put(url, headers=headers, data=json.dumps(payload), verify=self.verify)
+        return response

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/cloudos_cli/utils/__init__.py

@@ -4,6 +4,6 @@ Utility functions and classes to use across the package.
 
 from .errors import BadRequestException, TimeOutException
 from .requests import retry_requests_get,  retry_requests_post
+from .resources import format_bytes, ssl_selector
 
-
-__all__ = ['errors', 'requests']
+__all__ = ['errors', 'requests', 'resources']

cloudos_cli-2.29.0/cloudos_cli/utils/resources.py

@@ -0,0 +1,46 @@
+import os
+import urllib3
+
+def format_bytes(size):
+    """Convert bytes to human-readable format (e.g., 1.2 MB)."""
+    if size is None:
+        return "-"
+    power = 1024
+    n = 0
+    labels = ['B', 'KB', 'MB', 'GB', 'TB', 'PB']
+    while size >= power and n < len(labels) - 1:
+        size /= power
+        n += 1
+    return f"{size:.1f} {labels[n]}"
+
+
+def ssl_selector(disable_ssl_verification, ssl_cert):
+    """Verify value selector.
+
+    This function stablish the value that will be passed to requests.verify
+    variable.
+
+    Parameters
+    ----------
+    disable_ssl_verification : bool
+        Whether to disable SSL verification.
+    ssl_cert : string
+        String indicating the path to the SSL certificate file to use.
+
+    Returns
+    -------
+    verify_ssl : [bool | string]
+        Either a bool or a path string to be passed to requests.verify to control
+        SSL verification.
+    """
+    if disable_ssl_verification:
+        verify_ssl = False
+        print('[WARNING] Disabling SSL verification')
+        urllib3.disable_warnings()
+    elif ssl_cert is None:
+        verify_ssl = True
+    elif os.path.isfile(ssl_cert):
+        verify_ssl = ssl_cert
+    else:
+        raise FileNotFoundError(f"The specified file '{ssl_cert}' was not found")
+    return verify_ssl

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.26.1
+Version: 2.29.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -535,6 +535,69 @@ Executing status...
 	To further check your job status you can either go to https://cloudos.lifebit.ai/app/advanced-analytics/analyses/62c83a1191fe06013b7ef355 or repeat the command you just used.
 ```
 
+#### Check job details
+
+To check the details of a submitted job, the subcommand `details` of `job` can be used.
+
+For example, with explicit variable for required parameters:
+
+```bash
+cloudos job details \
+    --apikey $MY_API_KEY \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+Or with a defined profile:
+
+```bash
+cloudos job details \
+    --profile job-details \
+    --job-id 62c83a1191fe06013b7ef355
+```
+
+The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
+
+```console
+Executing details...
+                                             Job Details                                              
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Parameters               │ -test=value                                                             │
+│                          │ --gaq=test                                                              │
+│                          │ cryo=yes                                                                │
+│ Command                  │ echo 'test' > new_file.txt                                              │
+│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
+│ Nextflow Version         │ None                                                                    │
+│ Execution Platform       │ Batch AWS                                                               │
+│ Profile                  │ None                                                                    │
+│ Master Instance          │ c5.xlarge                                                               │
+│ Storage                  │ 500                                                                     │
+│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
+│ Accelerated File Staging │ None                                                                    │
+│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
+└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
+
+By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+
+The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
+
+```console
+params {
+    parameter_one = value_one
+    parameter_two = value_two
+    parameter_three = value_three
+}
+```
+
+This file can later be used when running a job with `cloudos job run --job-config job_details.config ...`.
+
+> [!NOTE]
+> Job details can only be retrieved for a single user, cannot see other user's job details.
+
 #### Get a list of your jobs from a CloudOS workspace
 
 You can get a summary of your last 30 submitted jobs (or your selected number of last jobs using `--last-n-jobs n`
@@ -758,9 +821,49 @@ cloudos datasets ls <path> \
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+
 The output of this command is a list of files and folders present in the specified project.
 If the `<path>` is left empty, the command will return the list of folders present in the selected project.
 
+If you require more information on the files and folder listed, you can use the `--details` flag that will output a table containing the following columns:
+- Type (folder or file)
+- Owner
+- Size in human readable format
+- Last updated
+- Filepath (the file or folder name)
+- S3 Path
+
+##### Moving files
+
+Files and folders can be moved **from** `Data` or any of its subfolders (i.e `Data`, `Data/folder/file.txt`) **to** `Data` or any of its subfolders programmatically.
+
+1. The move can happen **within the same project** running the following command:
+```
+cloudos datasets mv <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets mv <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path, starting from the `Data` datasets and its folder; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets mv Data/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.26.1 → cloudos_cli-2.29.0}/cloudos_cli.egg-info/SOURCES.txt

@@ -24,5 +24,6 @@ cloudos_cli/queue/queue.py
 cloudos_cli/utils/__init__.py
 cloudos_cli/utils/errors.py
 cloudos_cli/utils/requests.py
+cloudos_cli/utils/resources.py
 tests/__init__.py
 tests/functions_for_pytest.py

cloudos_cli-2.26.1/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.26.1'