cloudos-cli 2.36.0__tar.gz → 2.37.1__tar.gz

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.36.0
+Version: 2.37.1
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -512,6 +512,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/README.md

@@ -477,6 +477,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli/__main__.py

@@ -16,6 +16,7 @@ from rich.table import Table
 from cloudos_cli.datasets import Datasets
 from cloudos_cli.utils.resources import ssl_selector, format_bytes
 from rich.style import Style
+from cloudos_cli.utils.array_job import generate_datasets_for_project
 from cloudos_cli.utils.details import get_path
 
 
@@ -45,7 +46,10 @@ def run_cloudos_cli(ctx):
     config_manager = ConfigurationProfile()
     profile_to_use = config_manager.determine_default_profile()
     if profile_to_use is None:
-        print('[Warning] No profile found. Please create one with "cloudos configure".\n')
+        console = Console()
+        console.print(
+            "[bold yellow][Warning] No profile found. Please create one with \"cloudos configure\"."
+        )
         shared_config = dict({
             'apikey': '',
             'cloudos_url': CLOUDOS_URL,
@@ -210,7 +214,6 @@ def configure(ctx, profile, make_default):
     if make_default:
         config_manager.make_default_profile(profile_name=profile)
 
-
 @job.command('run')
 @click.option('-k',
               '--apikey',
@@ -219,7 +222,8 @@ def configure(ctx, profile, make_default):
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -654,7 +658,8 @@ def run(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--job-id',
               help='The job id in CloudOS to search for.',
               required=True)
@@ -725,7 +730,8 @@ def job_status(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -798,7 +804,8 @@ def job_logs(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -871,7 +878,8 @@ def job_results(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--job-id',
               help='The job id in CloudOS to search for.',
               required=True)
@@ -1095,7 +1103,8 @@ def job_details(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1225,7 +1234,8 @@ def list_jobs(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1318,7 +1328,8 @@ def abort_jobs(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1416,7 +1427,8 @@ def list_workflows(ctx,
               '--cloudos-url',
               help=('The CloudOS url you are trying to access to. ' +
                     f'Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1496,7 +1508,8 @@ def import_wf(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1618,7 +1631,8 @@ def list_projects(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1693,7 +1707,8 @@ def cromwell_status(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1795,7 +1810,8 @@ def cromwell_restart(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1868,7 +1884,8 @@ def cromwell_stop(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -1976,7 +1993,8 @@ def remove_profile(ctx, profile):
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -2159,7 +2177,7 @@ def run_bash_job(ctx,
                       hpc_id=None,
                       cost_limit=cost_limit,
                       verify=verify_ssl,
-                      command=command,
+                      command={"command": command},
                       cpus=cpus,
                       memory=memory)
 
@@ -2203,7 +2221,8 @@ def run_bash_job(ctx,
 @click.option('-c',
               '--cloudos-url',
               help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
-              default=CLOUDOS_URL)
+              default=CLOUDOS_URL,
+              required=True)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
@@ -2219,7 +2238,12 @@ def run_bash_job(ctx,
               help=('A single parameter to pass to the job call. It should be in the ' +
                     'following form: parameter_name=parameter_value. E.g.: ' +
                     '-p --test=value or -p -test=value or -p test=value. You can use this option as many ' +
-                    'times as parameters you want to include.'))
+                    'times as parameters you want to include. ' +
+                    'For parameters pointing to a file, the format expected is ' +
+                    'parameter_name=<project>/Data/parameter_value. The parameter value must be a ' +
+                    'file located in the `Data` subfolder. If no <project> is specified, it defaults to ' +
+                    'the project specified by the profile or --project-name parameter. ' +
+                    'E.g.: -p "--file=Data/file.txt" or "--file=<project>/Data/folder/file.txt"'))
 @click.option('--job-name',
               help='The name of the job. Default=new_job.',
               default='new_job')
@@ -2405,35 +2429,6 @@ def run_bash_array_job(ctx,
         "|": { "api": "%7C", "file": "|" }
     }
 
-    # Setup datasets
-    try:
-        ds = Datasets(
-            cloudos_url=cloudos_url,
-            apikey=apikey,
-            workspace_id=workspace_id,
-            project_name=array_file_project,
-            verify=verify_ssl,
-            cromwell_token=None
-        )
-        if custom_script_project is not None:
-            # If a custom script project is specified, create a new Datasets object for it
-            # This allows the user to run custom scripts in a different project
-            ds_custom = Datasets(
-                cloudos_url=cloudos_url,
-                apikey=apikey,
-                workspace_id=workspace_id,
-                project_name=custom_script_project,
-                verify=verify_ssl,
-                cromwell_token=None
-            )
-    except BadRequestException as e:
-        if 'Forbidden' in str(e):
-            print('[Error] It seems your call is not authorised. Please check if ' +
-                  'your workspace is restricted by Airlock and if your API key is valid.')
-            sys.exit(1)
-        else:
-            raise e
-
     # setup important options for the job
     if do_not_save_logs:
         save_logs = False
@@ -2453,7 +2448,12 @@ def run_bash_array_job(ctx,
                repository_platform=repository_platform, verify=verify_ssl)
 
     # retrieve columns
-    r = j.retrieve_cols_from_array_file(array_file, ds, separators[separator]['api'], verify_ssl)
+    r = j.retrieve_cols_from_array_file(
+        array_file,
+        generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl),
+        separators[separator]['api'],
+        verify_ssl
+    )
 
     if not disable_column_check:
         columns = json.loads(r.content).get("headers", None)
@@ -2470,7 +2470,12 @@ def run_bash_array_job(ctx,
         columns = []
 
     # setup parameters for the job
-    cmd = j.setup_params_array_file(custom_script_path, ds_custom, command, separators[separator]['file'])
+    cmd = j.setup_params_array_file(
+        custom_script_path,
+        generate_datasets_for_project(cloudos_url, apikey, workspace_id, custom_script_project, verify_ssl),
+        command,
+        separators[separator]['file']
+    )
 
     # check columns in the array file vs parameters added
     if not disable_column_check and array_parameter:
@@ -2709,7 +2714,7 @@ def list_files(ctx,
 @click.argument("source_path", required=True)
 @click.argument("destination_path", required=True)
 @click.option('-k', '--apikey', required=True, help='Your CloudOS API key.')
-@click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=False, help='The CloudOS URL.')
+@click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=True, help='The CloudOS URL.')
 @click.option('--workspace-id', required=True, help='The CloudOS workspace ID.')
 @click.option('--project-name', required=True, help='The source project name.')
 @click.option('--destination-project-name', required=False,

cloudos_cli-2.37.1/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.37.1'

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli/configure/configure.py

@@ -457,8 +457,7 @@ class ConfigurationProfile:
             profile_data = self.load_profile(profile_name=profile)
             apikey = self.get_param_value(ctx, apikey, 'apikey', profile_data['apikey'],
                                           required=required_dict['apikey'], missing_required_params=missing)
-            cloudos_url = self.get_param_value(ctx, cloudos_url, 'cloudos_url',
-                                               profile_data['cloudos_url']) or cloudos_url_default
+            resolved_cloudos_url = self.get_param_value(ctx, cloudos_url, 'cloudos_url', profile_data['cloudos_url'])
             workspace_id = self.get_param_value(ctx, workspace_id, 'workspace_id', profile_data['workspace_id'],
                                                 required=required_dict['workspace_id'], missing_required_params=missing)
             workflow_name = self.get_param_value(ctx, workflow_name, 'workflow_name', profile_data['workflow_name'],
@@ -473,7 +472,8 @@ class ConfigurationProfile:
             # when no profile is used, we need to check if the user provided all required parameters
             apikey = self.get_param_value(ctx, apikey, 'apikey', apikey, required=required_dict['apikey'],
                                           missing_required_params=missing)
-            cloudos_url = self.get_param_value(ctx, cloudos_url, 'cloudos_url', cloudos_url) or cloudos_url_default
+            resolved_cloudos_url = self.get_param_value(ctx, cloudos_url, 'cloudos_url', cloudos_url,
+                                          missing_required_params=missing) 
             workspace_id = self.get_param_value(ctx, workspace_id, 'workspace_id', workspace_id,
                                                 required=required_dict['workspace_id'],
                                                 missing_required_params=missing)
@@ -486,10 +486,19 @@ class ConfigurationProfile:
             project_name = self.get_param_value(ctx, project_name, 'project_name', project_name,
                                                 required=required_dict['project_name'],
                                                 missing_required_params=missing)
+        if not resolved_cloudos_url:
+            click.secho(
+                f"[Warning] No CloudOS URL provided via CLI or profile. Falling back to default: {cloudos_url_default}",
+                fg="yellow",
+                bold=True
+            )
+            cloudos_url = cloudos_url_default
+        else:
+            cloudos_url = resolved_cloudos_url
         cloudos_url = cloudos_url.rstrip('/')
 
         # Raise once, after all checks
         if missing:
             formatted = ', '.join(p for p in missing)
-            raise click.UsageError(f"Missing required option/s: {formatted}")
+            raise click.UsageError(f"Missing required option/s: {formatted} \nYou can configure the following parameters persistently by running cloudos configure:\n  --apikey,\n  --cloudos-url,\n  --workspace-id,\n  --workflow-name,\n  --repository-platform,\n  --execution-platform,\n  --project-name\n For more information on the usage of the command, please run cloudos configure --help")
         return apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli/jobs/job.py

@@ -10,6 +10,8 @@ from cloudos_cli.utils.errors import BadRequestException
 from cloudos_cli.utils.requests import retry_requests_post, retry_requests_get
 from pathlib import Path
 import base64
+from cloudos_cli.utils.array_job import classify_pattern, get_file_or_folder_id, extract_project
+import os
 
 
 @dataclass
@@ -382,14 +384,8 @@ class Job(Cloudos):
                 p_name = p_split[0]
                 p_value = '='.join(p_split[1:])
                 if workflow_type == 'docker':
-                    prefix = "--" if p_name.startswith('--') else ("-" if p_name.startswith('-') else '')
-                    # leave defined for adding files later
-                    parameter_kind = "textValue"
-                    param = {"prefix": prefix,
-                             "name": p_name.lstrip('-'),
-                             "parameterKind": parameter_kind,
-                             "textValue": p_value}
-                    workflow_params.append(param)
+                    # will differentiate between text, data items and glob patterns
+                    workflow_params.append(self.docker_workflow_param_processing(p, self.project_name))
                 elif workflow_type == 'wdl':
                     param = {"prefix": "",
                              "name": p_name,
@@ -834,3 +830,97 @@ class Job(Cloudos):
                 }
 
         return ap_param
+
+    def docker_workflow_param_processing(self, param, project_name):
+        """
+        Processes a Docker workflow parameter and determines its type and associated metadata.
+
+        Parameters
+        ----------
+        param : str
+            The parameter string in the format '--param_name=value'.
+            It can represent a file path, a glob pattern, or a simple text value.
+        project_name : str
+            The name of the current project to use if no specific project is extracted from the parameter.
+
+        Returns:
+            dict: A dictionary containing the processed parameter details. The structure of the dictionary depends on the type of the parameter:
+            - For glob patterns:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "globPattern": str,   # The glob pattern extracted from the parameter.
+                "parameterKind": str, # Always "globPattern".
+                "folder": str         # Folder ID associated with the glob pattern.
+            - For file paths:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "parameterKind": str, # Always "dataItem".
+                "dataItem": {
+                    "kind": str,      # Always "File".
+                    "item": str       # File ID associated with the file path.
+            - For text values:
+                {
+                "name": str,          # Parameter name without leading dashes.
+                "prefix": str,        # Prefix ('--' or '-') based on the parameter format.
+                "parameterKind": str, # Always "textValue".
+                "textValue": str      # The text value extracted from the parameter.
+
+        Notes
+        -----
+        - The function uses helper methods `extract_project`, `classify_pattern`, and `get_file_or_folder_id` to process the parameter.
+        - If the parameter represents a file path or glob pattern, the function retrieves the corresponding file or folder ID from the cloud workspace.
+        - If the parameter does not match any specific pattern or file extension, it is treated as a simple text value.
+        """
+
+        # split '--param_name=example_test'
+        # name -> '--param_name'
+        # rest -> 'example_test'
+        name, rest = param.split('=', 1)
+
+        # e.g. "/Project/Subproject/file.csv", project is "Project"
+        # e.g "Data/input.csv", project is '', leaving the global project name
+        # e.g "-p --test=value", project is ''
+        project, file_path = extract_project(rest)
+        current_project = project if project != '' else project_name
+
+        # e.g. "/Project/Subproject/file.csv"
+        command_path = Path(file_path)
+        command_dir = str(command_path.parent)
+        command_name = command_path.name
+        _, ext = os.path.splitext(command_name)
+        prefix = "--" if name.startswith('--') else ("-" if name.startswith('-') else "")
+        if classify_pattern(rest) in ["regex", "glob"]:
+            if not (file_path.startswith('/Data') or file_path.startswith('Data')):
+                raise ValueError("[ERROR] The file path inside the project must start with '/Data' or 'Data'. ")
+
+            folder = get_file_or_folder_id(self.cloudos_url, self.apikey, self.workspace_id, current_project, self.verify, command_dir, command_name, is_file=False)
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                'globPattern': command_name,
+                "parameterKind": "globPattern",
+                "folder": f"{folder}"
+            }
+        elif ext:
+            if not (file_path.startswith('/Data') or file_path.startswith('Data')):
+                raise ValueError("[ERROR] The file path inside the project must start with '/Data' or 'Data'. ")
+
+            file = get_file_or_folder_id(self.cloudos_url, self.apikey, self.workspace_id, current_project, self.verify, command_dir, command_name, is_file=True)
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                "parameterKind": "dataItem",
+                "dataItem": {
+                    "kind": "File",
+                    "item": f"{file}"
+                }
+            }
+        else:
+            return {
+                "name": f"{name.lstrip('-')}",
+                "prefix": f"{prefix}",
+                "parameterKind": "textValue",
+                "textValue": f"{rest}"
+            }

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli/utils/__init__.py

@@ -7,6 +7,7 @@ from .requests import retry_requests_get, retry_requests_post, retry_requests_pu
 from .resources import format_bytes, ssl_selector
 from .cloud import find_cloud
 from .cloud import find_cloud
+from .array_job import is_valid_regex, is_glob_pattern, is_probably_regex, classify_pattern, generate_datasets_for_project, get_file_or_folder_id
 from .details import get_path
 
-__all__ = ['errors', 'requests', 'resources', 'cloud', 'details']
+__all__ = ['errors', 'requests', 'resources', 'cloud', 'details', 'array_job']

cloudos_cli-2.37.1/cloudos_cli/utils/array_job.py

@@ -0,0 +1,254 @@
+import re
+import sys
+from cloudos_cli.utils.errors import BadRequestException
+
+
+def is_valid_regex(s):
+    """
+    Validates whether the given string is a valid regular expression.
+
+    Parameters
+    ----------
+    s : str
+        The string to be checked as a regular expression.
+
+    Returns
+    -------
+    bool
+        True if the string is a valid regular expression, False otherwise.
+    """
+    try:
+        re.compile(s)
+        return True
+    except re.error:
+        return False
+
+def is_glob_pattern(s):
+    """
+    Check if a given string contains glob pattern characters.
+
+    Glob patterns are commonly used for filename matching and include
+    special characters such as '*', '?', and '['.
+
+    Parameters
+    ----------
+    s : str
+        The string to check for glob pattern characters.
+
+    Returns
+    -------
+    bool
+        True if the string contains any glob pattern characters, otherwise False.
+    """
+    return any(char in s for char in "*?[")
+
+def is_probably_regex(s):
+    """
+    Determines if a given string is likely a regular expression.
+
+    This function checks whether the input string matches common patterns
+    that are indicative of regular expressions. It first validates the
+    string using `is_valid_regex(s)` and then searches for specific regex
+    indicators such as quantifiers, character classes, anchors, and
+    alternation.
+
+    Parameters
+    ----------
+    s : str
+        The string to evaluate.
+
+    Returns
+    -------
+    bool
+        True if the string is likely a regular expression, False otherwise.
+
+    Notes
+    -----
+    The function assumes the existence of `is_valid_regex(s)` which
+        validates whether the input string is a valid regex.
+    """
+    if not is_valid_regex(s):
+        return False
+
+    # Patterns that usually indicate actual regex use (not just file names)
+    regex_indicators = [
+        r"\.\*", r"\.\+", r"\\[dws]", r"\[[^\]]+\]", r"\([^\)]+\)",
+        r"\{\d+(,\d*)?\}", r"\^", r"\$", r"\|"
+    ]
+    return any(re.search(pat, s) for pat in regex_indicators)
+
+def classify_pattern(s):
+    """
+    Classifies a given string pattern into one of three categories: "regex", "glob", or "exact".
+
+    Parameters
+    ----------
+    s : str
+        The string pattern to classify.
+
+    Returns
+    -------
+    str: A string indicating the type of pattern:
+        - "regex" if the pattern is likely a regular expression.
+        - "glob" if the pattern matches glob-style syntax.
+        - "exact" if the pattern does not match regex or glob syntax.
+    """
+    if is_probably_regex(s):
+        return "regex"
+    elif is_glob_pattern(s):
+        return "glob"
+    else:
+        return "exact"
+
+def generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl):
+    """
+    Generate datasets for a specified project in a CloudOS workspace.
+
+    This function initializes a `Datasets` object for the given project and handles
+    potential errors such as missing project elements or unauthorized API calls.
+
+    Parameters
+    ----------
+    cloudos_url : str
+        The URL of the CloudOS instance.
+    apikey : str
+        The API key for authentication.
+    workspace_id : str
+        The ID of the workspace where the project resides.
+    project_name : str
+        The name of the project for which datasets are generated.
+    verify_ssl : bool
+        Whether to verify SSL certificates during API calls.
+
+    Returns
+    -------
+    Datasets
+        An instance of the `Datasets` class initialized for the specified project.
+
+    Raises
+    ------
+    ValueError
+        If the specified project is not found in the workspace.
+    BadRequestException
+        If the API call is unauthorized or encounters other issues.
+    """
+
+    # this avoids circular import error if import is added at the top
+    from cloudos_cli.datasets import Datasets
+    try:
+        ds = Datasets(
+            cloudos_url=cloudos_url,
+            apikey=apikey,
+            workspace_id=workspace_id,
+            project_name=project_name,
+            verify=verify_ssl,
+            cromwell_token=None
+        )
+    except ValueError:
+        print(f"[ERROR] No {project_name} element in projects was found")
+        sys.exit(1)
+
+    except BadRequestException as e:
+        if 'Forbidden' in str(e):
+            print('[Error] It seems your call is not authorised. Please check if ' +
+                  'your workspace is restricted by Airlock and if your API key is valid.')
+            sys.exit(1)
+        else:
+            raise e
+
+    return ds
+
+def get_file_or_folder_id(cloudos_url, apikey, workspace_id, project_name, verify_ssl, command_dir, command_name, is_file=True):
+    """Retrieve the ID of a specific file or folder within a CloudOS workspace.
+
+    Parameters
+    ----------
+    cloudos_url : str
+        The base URL of the CloudOS API.
+    apikey : str
+        The API key for authenticating requests to the CloudOS API.
+    workspace_id : str
+        The ID of the workspace containing the project.
+    project_name : str
+        The name of the project within the workspace.
+    verify_ssl : bool
+        Whether to verify SSL certificates for the API requests.
+    name : str
+        The name of the file or folder whose ID is to be retrieved.
+    is_file : bool, optional
+        Whether to retrieve a file ID (True) or folder ID (False). Default is True.
+
+    Returns
+    -------
+    str: The ID of the specified file or folder.
+
+    Raises
+    ------
+    ValueError
+        If the specified file or folder is not found.
+    Exception
+        If there is an error during the API interaction or data retrieval.
+
+    Notes
+    -----
+    - This function uses the `generate_datasets_for_project` function to create a Datasets object for the specified project.
+    - The `list_folder_content` method is used for files, and `list_project_content` is used for folders.
+    - The function assumes that the IDs are stored in the `"_id"` field of the metadata.
+    """
+    # create a Datasets() class
+    ds = generate_datasets_for_project(cloudos_url, apikey, workspace_id, project_name, verify_ssl)
+
+    if is_file:
+        # get all files from a folder
+        content = ds.list_folder_content(command_dir)
+        for file in content['files']:
+            if file.get("name") == command_name:
+                return file.get("_id", '')
+        raise ValueError(f"File '{command_name}' not found in directory '{command_dir}'.")
+    else:
+        # get all folders from the project
+        # check if the command_dir has a sub-folder
+        if len(command_dir.split("/")) > 1:
+            # get the first folder which is just below the project
+            folders = ds.list_folder_content(command_dir.split("/")[0])
+            # use the last folder as is listed in the first folder
+            folder_to_search = command_dir.split("/")[-1]
+        else:
+            folders = ds.list_project_content()
+            folder_to_search = command_dir
+
+        for folder in folders['folders']:
+            if folder.get("name") == folder_to_search:
+                return folder.get("_id", '')
+        raise ValueError(f"Folder '{folder_to_search}' not found in project.")
+
+def extract_project(path):
+    """
+    Extracts the project name and the remaining path from a given file path.
+
+    The function assumes that a "project" exists if the path contains at least three parts
+    when split by slashes. If the path has fewer than three parts, the project name is
+    considered empty, and the entire path is returned as the remaining path.
+
+    Parameters
+    ----------
+    path : str
+        The file path to process.
+
+    Returns
+    -------
+    tuple: A tuple containing:
+        - str: The project name (empty string if no project exists).
+        - str: The remaining path after the project name.
+    """
+    # Strip slashes and split the path
+    parts = path.strip("/").split("/")
+    # A "project" exists only if there are at least 3 parts
+    # globs needs more than 3 parts i.e. PROJECT/Data/Downloads/*.csv
+    if (len(parts) >= 3 and not is_glob_pattern(path)) or \
+       (len(parts) > 3 and is_glob_pattern(path)):
+        # Return the first part as project name and the rest as remaining path
+        return parts[0], "/".join(parts[1:])
+    else:
+        # project is empty, use the project_name of the function
+        return "", "/".join(parts)

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.36.0
+Version: 2.37.1
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -512,6 +512,51 @@ This assumes the interpreter is available on the container’s $PATH. If not, yo
 
 These options provide flexibility for configuring and running bash array jobs, allowing to tailor the execution for specific requirements.
 
+#### Use multiple projects for files in `--parameter` option
+
+The option `--parameter`, could specify a file input located in a different project than option `--project-name`. The files can only be located inside the project's `Data` subfolder, not `Cohorts` or `Analyses Results`. The accepted structures for different parameter projects are:
+- `-p/--parameter "--file=<project>/Data/file.txt"`
+- `-p/--parameter "--file=<project>/Data/subfolder/file.txt"`
+- `-p/--parameter "--file=Data/subfolder/file.txt"` (the same project as `--project-name`)
+- `-p/--parameter "--file=<project>/Data/subfolder/*.txt"`
+- `-p/--parameter "--file=<project>/Data/*.txt"`
+- `-p/--parameter "--file=Data/*.txt"` (the same project as `--project-name`)
+
+The project, should be specified at the beginning of the file path. For example:
+
+```console
+cloudos bash array-job \
+    -p file=Data/input.csv
+...
+```
+This will point to the global project, specified with `--project-name`. In contrast:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp=PROJECT_EXPRESSION/Data/input.csv \
+    --project-name "ADIPOSE"
+...
+```
+for parameter `exp` it will point to a project named `PROJECT_EXPRESSION` in the File Explorer, and `data` parameter will be found in the global project `ADIPOSE`.
+
+Apart from files, the parameter can also take glob patterns, for example:
+
+```console
+cloudos bash array-job \
+    -p data=Data/input.csv
+    -p exp="PROJECT_EXPRESSION/Data/*.csv" \
+    --project-name "ADIPOSE"
+...
+```
+will take all `csv` file extensions in the specified folder.
+
+> [!NOTE]
+> When specifying glob patterns, depending on the terminal is best to add it in double quotes to avoid the terminal searching for the glob pattern locally, e.g. `-p exp="PROJECT_EXPRESSION/Data/*.csv"`.
+
+> [!NOTE]
+> Project names in the `--parameter` option can start with either forward slash `/` or without. The following are the same `-p data=/PROJECT1/Data/input.csv` and `-p data=PROJECT1/Data/input.csv`.
+
 #### Get path to logs of job from CloudOS
 
 Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.

{cloudos_cli-2.36.0 → cloudos_cli-2.37.1}/cloudos_cli.egg-info/SOURCES.txt

@@ -22,6 +22,7 @@ cloudos_cli/jobs/job.py
 cloudos_cli/queue/__init__.py
 cloudos_cli/queue/queue.py
 cloudos_cli/utils/__init__.py
+cloudos_cli/utils/array_job.py
 cloudos_cli/utils/cloud.py
 cloudos_cli/utils/details.py
 cloudos_cli/utils/errors.py

cloudos_cli-2.36.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.36.0'