cloudos-cli 2.35.0__tar.gz → 2.37.0__tar.gz

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.35.0
+Version: 2.37.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -512,6 +512,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
@@ -1013,6 +1058,38 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+#### Copying files and folders
+
+Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+
+1. The copy can happen **within the same project** running the following command:
+```
+cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
+
 #### Create a (virtual) folder
 
 New folders can be created within the `Data` dataset and its subfolders using the following command
@@ -1032,30 +1109,21 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+#### Removing files and folders
 
-#### Copying files and folders
-
-Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+Files and folders can be removed from file explorer (in the `Data` datasets and its subfolders) using the following command
 
-1. The copy can happen **within the same project** running the following command:
 ```
-cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+cloudos datasets rm <path> 
 ```
-where the source project as well as the destination one is the one defined in the profile.
+where `path` is the full path to the file/folder to be removed.
 
-2. The move can also happen **across different projects**  within the same workspace by running the following command
-```
-cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
-```
-In this case, only the source project is the one specified in the profile.
-
-Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+Please, be aware that removing files and folders will only remove them from the file explorer and not from the corresponding cloud storage.
 
-```
-cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
-```
+Please, keep in mind that you are only allowed to remove files or folders in `Data` or its subfolders.
 
 Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+
 ```bash
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/README.md

@@ -477,6 +477,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
@@ -978,6 +1023,38 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+#### Copying files and folders
+
+Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+
+1. The copy can happen **within the same project** running the following command:
+```
+cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
+
 #### Create a (virtual) folder
 
 New folders can be created within the `Data` dataset and its subfolders using the following command
@@ -997,30 +1074,21 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+#### Removing files and folders
 
-#### Copying files and folders
-
-Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+Files and folders can be removed from file explorer (in the `Data` datasets and its subfolders) using the following command
 
-1. The copy can happen **within the same project** running the following command:
 ```
-cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+cloudos datasets rm <path> 
 ```
-where the source project as well as the destination one is the one defined in the profile.
+where `path` is the full path to the file/folder to be removed.
 
-2. The move can also happen **across different projects**  within the same workspace by running the following command
-```
-cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
-```
-In this case, only the source project is the one specified in the profile.
-
-Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+Please, be aware that removing files and folders will only remove them from the file explorer and not from the corresponding cloud storage.
 
-```
-cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
-```
+Please, keep in mind that you are only allowed to remove files or folders in `Data` or its subfolders.
 
 Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+
 ```bash
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli/__main__.py

@@ -16,6 +16,7 @@ from rich.table import Table
 from cloudos_cli.datasets import Datasets
 from cloudos_cli.utils.resources import ssl_selector, format_bytes
 from rich.style import Style
+from cloudos_cli.utils.array_job import generate_datasets_for_project
 from cloudos_cli.utils.details import get_path
 
 
@@ -90,7 +91,8 @@ def run_cloudos_cli(ctx):
                 'mv': shared_config,
                 'rename': shared_config,
                 'cp': shared_config,
-                'mkdir': shared_config
+                'mkdir': shared_config,
+                'rm': shared_config
             }
         })
     else:
@@ -139,7 +141,8 @@ def run_cloudos_cli(ctx):
                 'mv': shared_config,
                 'rename': shared_config,
                 'cp': shared_config,
-                'mkdir': shared_config
+                'mkdir': shared_config,
+                'rm': shared_config
             }
         })
 
@@ -2157,7 +2160,7 @@ def run_bash_job(ctx,
                       hpc_id=None,
                       cost_limit=cost_limit,
                       verify=verify_ssl,
-                      command=command,
+                      command={"command": command},
                       cpus=cpus,
                       memory=memory)
 
@@ -2217,7 +2220,12 @@ def run_bash_job(ctx,
               help=('A single parameter to pass to the job call. It should be in the ' +
                     'following form: parameter_name=parameter_value. E.g.: ' +
                     '-p --test=value or -p -test=value or -p test=value. You can use this option as many ' +
-                    'times as parameters you want to include.'))
+                    'times as parameters you want to include. ' +
+                    'For parameters pointing to a file, the format expected is ' +
+                    'parameter_name=<project>/Data/parameter_value. The parameter value must be a ' +
+                    'file located in the `Data` subfolder. If no <project> is specified, it defaults to ' +
+                    'the project specified by the profile or --project-name parameter. ' +
+                    'E.g.: -p "--file=Data/file.txt" or "--file=<project>/Data/folder/file.txt"'))
 @click.option('--job-name',
               help='The name of the job. Default=new_job.',
               default='new_job')
@@ -2403,35 +2411,6 @@ def run_bash_array_job(ctx,
         "|": { "api": "%7C", "file": "|" }
     }
 
-    # Setup datasets
-    try:
-        ds = Datasets(
-            cloudos_url=cloudos_url,
-            apikey=apikey,
-            workspace_id=workspace_id,
-            project_name=array_file_project,
-            verify=verify_ssl,
-            cromwell_token=None
-        )
-        if custom_script_project is not None:
-            # If a custom script project is specified, create a new Datasets object for it
-            # This allows the user to run custom scripts in a different project
-            ds_custom = Datasets(
-                cloudos_url=cloudos_url,
-                apikey=apikey,
-                workspace_id=workspace_id,
-                project_name=custom_script_project,
-                verify=verify_ssl,
-                cromwell_token=None
-            )
-    except BadRequestException as e:
-        if 'Forbidden' in str(e):
-            print('[Error] It seems your call is not authorised. Please check if ' +
-                  'your workspace is restricted by Airlock and if your API key is valid.')
-            sys.exit(1)
-        else:
-            raise e
-
     # setup important options for the job
     if do_not_save_logs:
         save_logs = False
@@ -2451,7 +2430,12 @@ def run_bash_array_job(ctx,
                repository_platform=repository_platform, verify=verify_ssl)
 
     # retrieve columns
-    r = j.retrieve_cols_from_array_file(array_file, ds, separators[separator]['api'], verify_ssl)
+    r = j.retrieve_cols_from_array_file(
+        array_file,
+        generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl),
+        separators[separator]['api'],
+        verify_ssl
+    )
 
     if not disable_column_check:
         columns = json.loads(r.content).get("headers", None)
@@ -2468,7 +2452,12 @@ def run_bash_array_job(ctx,
         columns = []
 
     # setup parameters for the job
-    cmd = j.setup_params_array_file(custom_script_path, ds_custom, command, separators[separator]['file'])
+    cmd = j.setup_params_array_file(
+        custom_script_path,
+        generate_datasets_for_project(cloudos_url, apikey, workspace_id, custom_script_project, verify_ssl),
+        command,
+        separators[separator]['file']
+    )
 
     # check columns in the array file vs parameters added
     if not disable_column_check and array_parameter:
@@ -3044,7 +3033,7 @@ def copy_item_cli(ctx, source_path, destination_path, apikey, cloudos_url,
         sys.exit(1)
     # Find the source item
     source_item = None
-    for item in source_content.get('files' or 'folders', {}):
+    for item in source_content.get('files', []) + source_content.get('folders', []):
         if item.get("name") == source_name:
             source_item = item
             break
@@ -3206,5 +3195,104 @@ def mkdir_item(ctx, new_folder_path, apikey, cloudos_url,
         sys.exit(1)
 
 
+@datasets.command(name="rm")
+@click.argument("target_path", required=True)
+@click.option('-k', '--apikey', required=True, help='Your CloudOS API key.')
+@click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=True, help='The CloudOS URL.')
+@click.option('--workspace-id', required=True, help='The CloudOS workspace ID.')
+@click.option('--project-name', required=True, help='The project name.')
+@click.option('--disable-ssl-verification', is_flag=True, help='Disable SSL certificate verification.')
+@click.option('--ssl-cert', help='Path to your SSL certificate file.')
+@click.option('--profile', default=None, help='Profile to use from the config file.')
+@click.pass_context
+def rm_item(ctx, target_path, apikey, cloudos_url,
+            workspace_id, project_name,
+            disable_ssl_verification, ssl_cert, profile):
+    """
+    Delete a file or folder in a CloudOS project.
+
+    TARGET_PATH [path]: the full path to the file or folder to delete. Must start with 'Data'. \n
+    E.g.: 'Data/folderA/file.txt' or 'Data/my_analysis/results/folderB'
+    """
+    if not target_path.strip("/").startswith("Data/"):
+        click.echo("[ERROR] TARGET_PATH must start with 'Data/', pointing to a file or folder.", err=True)
+        sys.exit(1)
+    click.echo("Loading configuration profile...")
+    config_manager = ConfigurationProfile()
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': True
+    }
+
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id,
+            workflow_name=None,
+            repository_platform=None,
+            execution_platform=None,
+            project_name=project_name
+        )
+    )
+
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+
+    client = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+
+    parts = target_path.strip("/").split("/")
+    parent_path = "/".join(parts[:-1])
+    item_name = parts[-1]
+
+    try:
+        contents = client.list_folder_content(parent_path)
+    except Exception as e:
+        click.echo(f"[ERROR] Could not list contents at '{parent_path or '[project root]'}': {str(e)}", err=True)
+        sys.exit(1)
+
+    found_item = None
+    for item in contents.get('files', []) + contents.get('folders', []):
+        if item.get("name") == item_name:
+            found_item = item
+            break
+
+    if not found_item:
+        click.echo(f"[ERROR] Item '{item_name}' not found in '{parent_path or '[project root]'}'", err=True)
+        sys.exit(1)
+
+    item_id = found_item["_id"]
+    kind = "Folder" if "folderType" in found_item else "File"
+
+    click.echo(f"Deleting {kind} '{item_name}' from '{parent_path or '[root]'}'...")
+    try:
+        response = client.delete_item(item_id=item_id, kind=kind)
+        if response.ok:
+            click.secho(
+                f"[SUCCESS] {kind} '{item_name}' was deleted from '{parent_path or '[root]'}'.",
+                fg="green", bold=True
+            )
+            click.secho("This item will still be available on your Cloud Provider.", fg="yellow")
+        else:
+            click.echo(f"[ERROR] Deletion failed: {response.status_code} - {response.text}", err=True)
+            sys.exit(1)
+    except Exception as e:
+        click.echo(f"[ERROR] Delete operation failed: {str(e)}", err=True)
+        sys.exit(1)
+
+
 if __name__ == "__main__":
     run_cloudos_cli()

cloudos_cli-2.37.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.37.0'

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli/datasets/datasets.py

@@ -5,7 +5,7 @@ This is the main class for file explorer (datasets).
 from dataclasses import dataclass
 from typing import Union
 from cloudos_cli.clos import Cloudos
-from cloudos_cli.utils.requests import retry_requests_get, retry_requests_put, retry_requests_post
+from cloudos_cli.utils.requests import retry_requests_get, retry_requests_put, retry_requests_post, retry_requests_delete
 import json
 
 @dataclass
@@ -237,7 +237,7 @@ class Datasets(Cloudos):
             else:
                 item["s3Prefix"] = item['path']
                 item["s3BucketName"] = s3_bucket_name
-
+                item["fileType"] = "S3File"
                 normalized["files"].append(item)
 
         return normalized
@@ -436,7 +436,7 @@ class Datasets(Cloudos):
         elif item.get("fileType") == "S3File":
             payload = {
                 "s3BucketName": item["s3BucketName"],
-                "s3ObjectKey": item["s3ObjectKey"],
+                "s3ObjectKey": item.get("s3ObjectKey") or item.get("s3Prefix"),
                 "name": item["name"],
                 "parent": parent,
                 "isManagedByLifebit": item.get("isManagedByLifebit", False),
@@ -487,4 +487,34 @@ class Datasets(Cloudos):
         }
 
         response = retry_requests_post(url, headers=headers, json=payload, verify=self.verify)
+        return response
+    
+    def delete_item(self, item_id: str, kind: str):
+        """
+        Delete a file or folder in CloudOS.
+
+        Parameters
+        ----------
+        item_id : str
+            The ID of the file or folder to delete.
+        kind : str
+            Must be either "File" or "Folder".
+
+        Returns
+        -------
+        response : requests.Response
+            The response object from the CloudOS API.
+        """
+        if kind not in ("File", "Folder"):
+            raise ValueError("Invalid kind provided. Must be 'File' or 'Folder'.")
+
+        endpoint = "files" if kind == "File" else "folders"
+        url = f"{self.cloudos_url}/api/v1/{endpoint}/{item_id}?teamId={self.workspace_id}"
+
+        headers = {
+            "accept": "application/json",
+            "ApiKey": self.apikey
+        }
+
+        response = retry_requests_delete(url, headers=headers, verify=self.verify)
         return response

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli/jobs/job.py

@@ -10,6 +10,8 @@ from cloudos_cli.utils.errors import BadRequestException
 from cloudos_cli.utils.requests import retry_requests_post, retry_requests_get
 from pathlib import Path
 import base64
+from cloudos_cli.utils.array_job import classify_pattern, get_file_or_folder_id, extract_project
+import os
 
 
 @dataclass
@@ -382,14 +384,8 @@ class Job(Cloudos):
                 p_name = p_split[0]
                 p_value = '='.join(p_split[1:])
                 if workflow_type == 'docker':
-                    prefix = "--" if p_name.startswith('--') else ("-" if p_name.startswith('-') else '')
-                    # leave defined for adding files later
-                    parameter_kind = "textValue"
-                    param = {"prefix": prefix,
-                             "name": p_name.lstrip('-'),
-                             "parameterKind": parameter_kind,
-                             "textValue": p_value}
-                    workflow_params.append(param)
+                    # will differentiate between text, data items and glob patterns
+                    workflow_params.append(self.docker_workflow_param_processing(p, self.project_name))
                 elif workflow_type == 'wdl':
                     param = {"prefix": "",
                              "name": p_name,
@@ -834,3 +830,97 @@ class Job(Cloudos):
                 }
 
         return ap_param
+
+    def docker_workflow_param_processing(self, param, project_name):
+        """
+        Processes a Docker workflow parameter and determines its type and associated metadata.
+
+        Parameters
+        ----------
+        param : str
+            The parameter string in the format '--param_name=value'.
+            It can represent a file path, a glob pattern, or a simple text value.
+        project_name : str
+            The name of the current project to use if no specific project is extracted from the parameter.
+
+        Returns:
+            dict: A dictionary containing the processed parameter details. The structure of the dictionary depends on the type of the parameter:
+            - For glob patterns:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "globPattern": str,   # The glob pattern extracted from the parameter.
+                "parameterKind": str, # Always "globPattern".
+                "folder": str         # Folder ID associated with the glob pattern.
+            - For file paths:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "parameterKind": str, # Always "dataItem".
+                "dataItem": {
+                    "kind": str,      # Always "File".
+                    "item": str       # File ID associated with the file path.
+            - For text values:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "parameterKind": str, # Always "textValue".
+                "textValue": str      # The text value extracted from the parameter.
+
+        Notes
+        -----
+        - The function uses helper methods `extract_project`, `classify_pattern`, and `get_file_or_folder_id` to process the parameter.
+        - If the parameter represents a file path or glob pattern, the function retrieves the corresponding file or folder ID from the cloud workspace.
+        - If the parameter does not match any specific pattern or file extension, it is treated as a simple text value.
+        """
+
+        # split '--param_name=example_test'
+        # name -> '--param_name'
+        # rest -> 'example_test'
+        name, rest = param.split('=', 1)
+
+        # e.g. "/Project/Subproject/file.csv", project is "Project"
+        # e.g "Data/input.csv", project is '', leaving the global project name
+        # e.g "-p --test=value", project is ''
+        project, file_path = extract_project(rest)
+        current_project = project if project != '' else project_name
+
+        # e.g. "/Project/Subproject/file.csv"
+        command_path = Path(file_path)
+        command_dir = str(command_path.parent)
+        command_name = command_path.name
+        _, ext = os.path.splitext(command_name)
+        prefix = "--" if name.startswith('--') else ("-" if name.startswith('-') else "")
+        if classify_pattern(rest) in ["regex", "glob"]:
+            if not (file_path.startswith('/Data') or file_path.startswith('Data')):
+                raise ValueError("[ERROR] The file path inside the project must start with '/Data' or 'Data'. ")
+
+            folder = get_file_or_folder_id(self.cloudos_url, self.apikey, self.workspace_id, current_project, self.verify, command_dir, command_name, is_file=False)
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                'globPattern': command_name,
+                "parameterKind": "globPattern",
+                "folder": f"{folder}"
+            }
+        elif ext:
+            if not (file_path.startswith('/Data') or file_path.startswith('Data')):
+                raise ValueError("[ERROR] The file path inside the project must start with '/Data' or 'Data'. ")
+
+            file = get_file_or_folder_id(self.cloudos_url, self.apikey, self.workspace_id, current_project, self.verify, command_dir, command_name, is_file=True)
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                "parameterKind": "dataItem",
+                "dataItem": {
+                    "kind": "File",
+                    "item": f"{file}"
+                }
+            }
+        else:
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                "parameterKind": "textValue",
+                "textValue": f"{rest}"
+            }

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli/utils/__init__.py

@@ -3,10 +3,11 @@ Utility functions and classes to use across the package.
 """
 
 from .errors import BadRequestException, TimeOutException, AccountNotLinkedException, JoBNotCompletedException, NotAuthorisedException, NoCloudForWorkspaceException
-from .requests import retry_requests_get, retry_requests_post, retry_requests_put
+from .requests import retry_requests_get, retry_requests_post, retry_requests_put, retry_requests_delete
 from .resources import format_bytes, ssl_selector
 from .cloud import find_cloud
 from .cloud import find_cloud
+from .array_job import is_valid_regex, is_glob_pattern, is_probably_regex, classify_pattern, generate_datasets_for_project, get_file_or_folder_id
 from .details import get_path
 
-__all__ = ['errors', 'requests', 'resources', 'cloud', 'details']
+__all__ = ['errors', 'requests', 'resources', 'cloud', 'details', 'array_job']

cloudos_cli-2.37.0/cloudos_cli/utils/array_job.py

@@ -0,0 +1,254 @@
+import re
+import sys
+from cloudos_cli.utils.errors import BadRequestException
+
+
+def is_valid_regex(s):
+    """
+    Validates whether the given string is a valid regular expression.
+
+    Parameters
+    ----------
+    s : str
+        The string to be checked as a regular expression.
+
+    Returns
+    -------
+    bool
+        True if the string is a valid regular expression, False otherwise.
+    """
+    try:
+        re.compile(s)
+        return True
+    except re.error:
+        return False
+
+def is_glob_pattern(s):
+    """
+    Check if a given string contains glob pattern characters.
+
+    Glob patterns are commonly used for filename matching and include
+    special characters such as '*', '?', and '['.
+
+    Parameters
+    ----------
+    s : str
+        The string to check for glob pattern characters.
+
+    Returns
+    -------
+    bool
+        True if the string contains any glob pattern characters, otherwise False.
+    """
+    return any(char in s for char in "*?[")
+
+def is_probably_regex(s):
+    """
+    Determines if a given string is likely a regular expression.
+
+    This function checks whether the input string matches common patterns
+    that are indicative of regular expressions. It first validates the
+    string using `is_valid_regex(s)` and then searches for specific regex
+    indicators such as quantifiers, character classes, anchors, and
+    alternation.
+
+    Parameters
+    ----------
+    s : str
+        The string to evaluate.
+
+    Returns
+    -------
+    bool
+        True if the string is likely a regular expression, False otherwise.
+
+    Notes
+    -----
+    The function assumes the existence of `is_valid_regex(s)` which
+        validates whether the input string is a valid regex.
+    """
+    if not is_valid_regex(s):
+        return False
+
+    # Patterns that usually indicate actual regex use (not just file names)
+    regex_indicators = [
+        r"\.\*", r"\.\+", r"\\[dws]", r"\[[^\]]+\]", r"\([^\)]+\)",
+        r"\{\d+(,\d*)?\}", r"\^", r"\$", r"\|"
+    ]
+    return any(re.search(pat, s) for pat in regex_indicators)
+
+def classify_pattern(s):
+    """
+    Classifies a given string pattern into one of three categories: "regex", "glob", or "exact".
+
+    Parameters
+    ----------
+    s : str
+        The string pattern to classify.
+
+    Returns
+    -------
+    str: A string indicating the type of pattern:
+        - "regex" if the pattern is likely a regular expression.
+        - "glob" if the pattern matches glob-style syntax.
+        - "exact" if the pattern does not match regex or glob syntax.
+    """
+    if is_probably_regex(s):
+        return "regex"
+    elif is_glob_pattern(s):
+        return "glob"
+    else:
+        return "exact"
+
+def generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl):
+    """
+    Generate datasets for a specified project in a CloudOS workspace.
+
+    This function initializes a `Datasets` object for the given project and handles
+    potential errors such as missing project elements or unauthorized API calls.
+
+    Parameters
+    ----------
+    cloudos_url : str
+        The URL of the CloudOS instance.
+    apikey : str
+        The API key for authentication.
+    workspace_id : str
+        The ID of the workspace where the project resides.
+    project_name : str
+        The name of the project for which datasets are generated.
+    verify_ssl : bool
+        Whether to verify SSL certificates during API calls.
+
+    Returns
+    -------
+    Datasets
+        An instance of the `Datasets` class initialized for the specified project.
+
+    Raises
+    ------
+    ValueError
+        If the specified project is not found in the workspace.
+    BadRequestException
+        If the API call is unauthorized or encounters other issues.
+    """
+
+    # this avoids circular import error if import is added at the top
+    from cloudos_cli.datasets import Datasets
+    try:
+        ds = Datasets(
+            cloudos_url=cloudos_url,
+            apikey=apikey,
+            workspace_id=workspace_id,
+            project_name=project_name,
+            verify=verify_ssl,
+            cromwell_token=None
+        )
+    except ValueError:
+        print(f"[ERROR] No {project_name} element in projects was found")
+        sys.exit(1)
+
+    except BadRequestException as e:
+        if 'Forbidden' in str(e):
+            print('[Error] It seems your call is not authorised. Please check if ' +
+                  'your workspace is restricted by Airlock and if your API key is valid.')
+            sys.exit(1)
+        else:
+            raise e
+
+    return ds
+
+def get_file_or_folder_id(cloudos_url, apikey, workspace_id, project_name, verify_ssl, command_dir, command_name, is_file=True):
+    """Retrieve the ID of a specific file or folder within a CloudOS workspace.
+
+    Parameters
+    ----------
+    cloudos_url : str
+        The base URL of the CloudOS API.
+    apikey : str
+        The API key for authenticating requests to the CloudOS API.
+    workspace_id : str
+        The ID of the workspace containing the project.
+    project_name : str
+        The name of the project within the workspace.
+    verify_ssl : bool
+        Whether to verify SSL certificates for the API requests.
+    name : str
+        The name of the file or folder whose ID is to be retrieved.
+    is_file : bool, optional
+        Whether to retrieve a file ID (True) or folder ID (False). Default is True.
+
+    Returns
+    -------
+    str: The ID of the specified file or folder.
+
+    Raises
+    ------
+    ValueError
+        If the specified file or folder is not found.
+    Exception
+        If there is an error during the API interaction or data retrieval.
+
+    Notes
+    -----
+    - This function uses the `generate_datasets_for_project` function to create a Datasets object for the specified project.
+    - The `list_folder_content` method is used for files, and `list_project_content` is used for folders.
+    - The function assumes that the IDs are stored in the `"_id"` field of the metadata.
+    """
+    # create a Datasets() class
+    ds = generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl)
+
+    if is_file:
+        # get all files from a folder
+        content = ds.list_folder_content(command_dir)
+        for file in content['files']:
+            if file.get("name") == command_name:
+                return file.get("_id", '')
+        raise ValueError(f"File '{command_name}' not found in directory '{command_dir}'.")
+    else:
+        # get all folders from the project
+        # check if the command_dir has a sub-folder
+        if len(command_dir.split("/")) > 1:
+            # get the first folder which is just below the project
+            folders = ds.list_folder_content(command_dir.split("/")[0])
+            # use the last folder as is listed in the first folder
+            folder_to_search = command_dir.split("/")[-1]
+        else:
+            folders = ds.list_project_content()
+            folder_to_search = command_dir
+
+        for folder in folders['folders']:
+            if folder.get("name") == folder_to_search:
+                return folder.get("_id", '')
+        raise ValueError(f"Folder '{folder_to_search}' not found in project.")
+
+def extract_project(path):
+    """
+    Extracts the project name and the remaining path from a given file path.
+
+    The function assumes that a "project" exists if the path contains at least three parts
+    when split by slashes. If the path has fewer than three parts, the project name is
+    considered empty, and the entire path is returned as the remaining path.
+
+    Parameters
+    ----------
+    path : str
+        The file path to process.
+
+    Returns
+    -------
+    tuple: A tuple containing:
+        - str: The project name (empty string if no project exists).
+        - str: The remaining path after the project name.
+    """
+    # Strip slashes and split the path
+    parts = path.strip("/").split("/")
+    # A "project" exists only if there are at least 3 parts
+    # globs needs more than 3 parts i.e. PROJECT/Data/Downloads/*.csv
+    if (len(parts) >= 3 and not is_glob_pattern(path)) or \
+       (len(parts) > 3 and is_glob_pattern(path)):
+        # Return the first part as project name and the rest as remaining path
+        return parts[0], "/".join(parts[1:])
+    else:
+        # project is empty, use the project_name of the function
+        return "", "/".join(parts)

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli/utils/requests.py

@@ -107,3 +107,38 @@ def retry_requests_put(url, total=5, status_forcelist=[429, 500, 502, 503, 504],
     # Make a request using the session object
     response = session.put(url, **kwargs)
     return response
+
+
+def retry_requests_delete(url, total=5, status_forcelist=[429, 500, 502, 503, 504], **kwargs):
+    """
+    Wrap normal requests DELETE with an error retry strategy.
+
+    Parameters
+    ----------
+    url : str
+        The request URL.
+    total : int
+        Total number of retry attempts.
+    status_forcelist : list of int
+        HTTP status codes that should trigger a retry.
+    **kwargs :
+        Additional keyword arguments passed to `requests.delete`.
+
+    Returns
+    -------
+    requests.Response
+        The Response object returned by the API server.
+    """
+    retry_strategy = Retry(
+        total=total,
+        status_forcelist=status_forcelist,
+        allowed_methods=["DELETE"]
+    )
+    adapter = HTTPAdapter(max_retries=retry_strategy)
+
+    session = requests.Session()
+    session.mount("http://", adapter)
+    session.mount("https://", adapter)
+
+    response = session.delete(url, **kwargs)
+    return response

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.35.0
+Version: 2.37.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -512,6 +512,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
@@ -1013,6 +1058,38 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+
+#### Copying files and folders
+
+Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+
+1. The copy can happen **within the same project** running the following command:
+```
+cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+```
+where the source project as well as the destination one is the one defined in the profile.
+
+2. The move can also happen **across different projects**  within the same workspace by running the following command
+```
+cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
+```
+In this case, only the source project is the one specified in the profile.
+
+Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+
+```
+cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
+```
+
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+```bash
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+
+
 #### Create a (virtual) folder
 
 New folders can be created within the `Data` dataset and its subfolders using the following command
@@ -1032,30 +1109,21 @@ Please, note that in the above example a preconfigured profile has been used. If
     --workspace-id $WORKSPACE_ID \
     --project-name $PROJEC_NAME
 ```
+#### Removing files and folders
 
-#### Copying files and folders
-
-Files and folders can be copied **from** anywhere in the project **to** `Data` or any of its subfolders programmatically (i.e `Data`, `Data/folder/file.txt`).
+Files and folders can be removed from file explorer (in the `Data` datasets and its subfolders) using the following command
 
-1. The copy can happen **within the same project** running the following command:
 ```
-cloudos datasets cp <souce_path> <destination_path> --profile <profile name>
+cloudos datasets rm <path> 
 ```
-where the source project as well as the destination one is the one defined in the profile.
+where `path` is the full path to the file/folder to be removed.
 
-2. The move can also happen **across different projects**  within the same workspace by running the following command
-```
-cloudos datasets cp <source_path> <destiantion_path> --profile <profile_name> --destination-project-name <project_name>
-```
-In this case, only the source project is the one specified in the profile.
-
-Any of the `source_path` must be a full path; any `destination_path` must be a path starting with `Data` and finishing with the folder where to move the file/folder. An example of such command is:
+Please, be aware that removing files and folders will only remove them from the file explorer and not from the corresponding cloud storage.
 
-```
-cloudos datasets cp AnalysesResults/my_analysis/results/my_plot.png Data/plots 
-```
+Please, keep in mind that you are only allowed to remove files or folders in `Data` or its subfolders.
 
 Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to also provide the following flags
+
 ```bash
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \

{cloudos_cli-2.35.0 → cloudos_cli-2.37.0}/cloudos_cli.egg-info/SOURCES.txt

@@ -22,6 +22,7 @@ cloudos_cli/jobs/job.py
 cloudos_cli/queue/__init__.py
 cloudos_cli/queue/queue.py
 cloudos_cli/utils/__init__.py
+cloudos_cli/utils/array_job.py
 cloudos_cli/utils/cloud.py
 cloudos_cli/utils/details.py
 cloudos_cli/utils/errors.py

cloudos_cli-2.35.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.35.0'