cloudos-cli 2.29.0__tar.gz → 2.31.0__tar.gz

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.29.0
+Version: 2.31.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -420,6 +420,44 @@ command.
 Other options like `--wait-completion` are also available and work in the same way as for the `cloudos job run` command.
 Check `cloudos bash job --help` for more details.
 
+#### Get path to logs of job from CloudOS
+
+Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing logs...
+Logs URI: s3://path/to/location/of/logs
+
+Nextflow log: s3://path/to/location/of/logs/.nextflow.log
+
+Nextflow standard output: s3://path/to/location/of/logs/stdout.txt
+
+Trace file: s3://path/to/location/of/logs/trace.txt
+```
+
+#### Get path to result files of jobs from CloudOS
+
+Get the path where CloudOS stores the output files for a job. This can be used only on your user's jobs and for jobs with "completed" status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing results...
+results: s3://path/to/location/of/results/results/
+```
+
 #### Abort single or multiple jobs from CloudOS
 
 Aborts jobs in the CloudOS workspace that are either running or initialising. It can be used with one or more job IDs provided as a comma separated string using the `--job-ids` parameter.
@@ -695,8 +733,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -712,7 +750,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -737,7 +775,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/README.md

@@ -385,6 +385,44 @@ command.
 Other options like `--wait-completion` are also available and work in the same way as for the `cloudos job run` command.
 Check `cloudos bash job --help` for more details.
 
+#### Get path to logs of job from CloudOS
+
+Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing logs...
+Logs URI: s3://path/to/location/of/logs
+
+Nextflow log: s3://path/to/location/of/logs/.nextflow.log
+
+Nextflow standard output: s3://path/to/location/of/logs/stdout.txt
+
+Trace file: s3://path/to/location/of/logs/trace.txt
+```
+
+#### Get path to result files of jobs from CloudOS
+
+Get the path where CloudOS stores the output files for a job. This can be used only on your user's jobs and for jobs with "completed" status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing results...
+results: s3://path/to/location/of/results/results/
+```
+
 #### Abort single or multiple jobs from CloudOS
 
 Aborts jobs in the CloudOS workspace that are either running or initialising. It can be used with one or more job IDs provided as a comma separated string using the `--job-ids` parameter.
@@ -660,8 +698,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -677,7 +715,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -702,7 +740,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/__init__.py

@@ -8,4 +8,4 @@ Python package for interacting with Cloud OS (https://cloudos.lifebit.ai/)
 from .clos import Cloudos
 from ._version import __version__
 
-__all__ = ['jobs', 'utils', 'clos', 'queue']
+__all__ = ['jobs', 'utils', 'clos', 'queue', 'configure', 'datasets', 'import_wf']

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/__main__.py

@@ -3,7 +3,7 @@
 import rich_click as click
 import cloudos_cli.jobs.job as jb
 from cloudos_cli.clos import Cloudos
-from cloudos_cli.import_wf.import_wf import ImportGitlab, ImportGithub
+from cloudos_cli.import_wf.import_wf import ImportWorflow
 from cloudos_cli.queue.queue import Queue
 from cloudos_cli.utils.errors import BadRequestException
 import json
@@ -31,13 +31,14 @@ ABORT_JOB_STATES = ['running', 'initializing']
 CLOUDOS_URL = 'https://cloudos.lifebit.ai'
 INIT_PROFILE = 'initialisingProfile'
 
+
 @click.group()
 @click.version_option(__version__)
 @click.pass_context
 def run_cloudos_cli(ctx):
     """CloudOS python package: a package for interacting with CloudOS."""
     ctx.ensure_object(dict)
-    if ctx.invoked_subcommand not in ['datasets'] and ctx.args and ctx.args[0] == 'ls':
+    if ctx.invoked_subcommand not in ['datasets']:
         print(run_cloudos_cli.__doc__ + '\n')
         print('Version: ' + __version__ + '\n')
     config_manager = ConfigurationProfile()
@@ -60,6 +61,8 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'logs': shared_config,
+                'results': shared_config,
                 'details': shared_config
             },
             'workflow': {
@@ -103,6 +106,8 @@ def run_cloudos_cli(ctx):
                 'abort': shared_config,
                 'status': shared_config,
                 'list': shared_config,
+                'logs': shared_config,
+                'results': shared_config,
                 'details': shared_config
             },
             'workflow': {
@@ -289,7 +294,7 @@ def configure(ctx, profile, make_default):
               '--cromwell-token',
               help=('Specific Cromwell server authentication token. Currently, not necessary ' +
                     'as apikey can be used instead, but maintained for backwards compatibility.'))
-@click.option('--repository-platform',
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
               help='Name of the repository platform of the workflow. Default=github.',
               default='github')
 @click.option('--execution-platform',
@@ -515,9 +520,9 @@ def run(ctx,
         nextflow_version = '22.10.8'
         if execution_platform == 'azure':
             print(f'[Message] The selected worflow \'{workflow_name}\' ' +
-                  'is a CloudOS module. For these workflows, worker nodes '+
-                  'are managed internally. For this reason, the options '+
-                  'azure-worker-instance-type, azure-worker-instance-disk and '+
+                  'is a CloudOS module. For these workflows, worker nodes ' +
+                  'are managed internally. For this reason, the options ' +
+                  'azure-worker-instance-type, azure-worker-instance-disk and ' +
                   'azure-worker-instance-spot are not taking effect.')
     else:
         queue = Queue(cloudos_url=cloudos_url, apikey=apikey, cromwell_token=cromwell_token,
@@ -574,7 +579,7 @@ def run(ctx,
         print(f'\tNextflow version: {nextflow_version}')
     j_id = j.send_job(job_config=job_config,
                       parameter=parameter,
-                      is_module =is_module,
+                      is_module=is_module,
                       git_commit=git_commit,
                       git_tag=git_tag,
                       git_branch=git_branch,
@@ -701,6 +706,152 @@ def job_status(ctx,
           'or repeat the command you just used.')
 
 
+@job.command('logs')
+@click.option('-k',
+              '--apikey',
+              help='Your CloudOS API key',
+              required=True)
+@click.option('-c',
+              '--cloudos-url',
+              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              default=CLOUDOS_URL)
+@click.option('--workspace-id',
+              help='The specific CloudOS workspace id.',
+              required=True)
+@click.option('--job-id',
+              help='The job id in CloudOS to search for.',
+              required=True)
+@click.option('--verbose',
+              help='Whether to print information messages or not.',
+              is_flag=True)
+@click.option('--disable-ssl-verification',
+              help=('Disable SSL certificate verification. Please, remember that this option is ' +
+                    'not generally recommended for security reasons.'),
+              is_flag=True)
+@click.option('--ssl-cert',
+              help='Path to your SSL certificate file.')
+@click.option('--profile', help='Profile to use from the config file', default=None)
+@click.pass_context
+def job_logs(ctx,
+             apikey,
+             cloudos_url,
+             workspace_id,
+             job_id,
+             verbose,
+             disable_ssl_verification,
+             ssl_cert,
+             profile):
+    """Get the path to the logs of a specified job."""
+    profile = profile or ctx.default_map['job']['logs']['profile']
+    # Create a dictionary with required and non-required params
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': False
+    }
+    # determine if the user provided all required parameters
+    config_manager = ConfigurationProfile()
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id
+        )
+    )
+
+    print('Executing logs...')
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    if verbose:
+        print('\t...Preparing objects')
+    cl = Cloudos(cloudos_url, apikey, None)
+    if verbose:
+        print('\tThe following Cloudos object was created:')
+        print('\t' + str(cl) + '\n')
+        print(f'\tSearching for job id: {job_id}')
+    logs = cl.get_job_logs(job_id, workspace_id, verify_ssl)
+    for name, path in logs.items():
+        print(f"{name}: {path}\n")
+
+
+@job.command('results')
+@click.option('-k',
+              '--apikey',
+              help='Your CloudOS API key',
+              required=True)
+@click.option('-c',
+              '--cloudos-url',
+              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              default=CLOUDOS_URL)
+@click.option('--workspace-id',
+              help='The specific CloudOS workspace id.',
+              required=True)
+@click.option('--job-id',
+              help='The job id in CloudOS to search for.',
+              required=True)
+@click.option('--verbose',
+              help='Whether to print information messages or not.',
+              is_flag=True)
+@click.option('--disable-ssl-verification',
+              help=('Disable SSL certificate verification. Please, remember that this option is ' +
+                    'not generally recommended for security reasons.'),
+              is_flag=True)
+@click.option('--ssl-cert',
+              help='Path to your SSL certificate file.')
+@click.option('--profile', help='Profile to use from the config file', default=None)
+@click.pass_context
+def job_results(ctx,
+                apikey,
+                cloudos_url,
+                workspace_id,
+                job_id,
+                verbose,
+                disable_ssl_verification,
+                ssl_cert,
+                profile):
+    """Get the path to the results of a specified job."""
+    profile = profile or ctx.default_map['job']['results']['profile']
+    # Create a dictionary with required and non-required params
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': False
+    }
+    # determine if the user provided all required parameters
+    config_manager = ConfigurationProfile()
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id
+        )
+    )
+
+    print('Executing results...')
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+    if verbose:
+        print('\t...Preparing objects')
+    cl = Cloudos(cloudos_url, apikey, None)
+    if verbose:
+        print('\tThe following Cloudos object was created:')
+        print('\t' + str(cl) + '\n')
+        print(f'\tSearching for job id: {job_id}')
+    logs = cl.get_job_results(job_id, workspace_id, verify_ssl)
+    for name, path in logs.items():
+        print(f"{name}: {path}\n")
+
+
 @job.command('details')
 @click.option('-k',
               '--apikey',
@@ -714,7 +865,8 @@ def job_status(ctx,
               help='The job id in CloudOS to search for.',
               required=True)
 @click.option('--output-format',
-              help='The desired display for the output, either directly in standard output or saved as file. Default=stdout.',
+              help=('The desired display for the output, either directly in standard output or saved as file. ' +
+                    'Default=stdout.'),
               type=click.Choice(['stdout', 'json'], case_sensitive=False),
               default='stdout')
 @click.option('--output-basename',
@@ -738,16 +890,16 @@ def job_status(ctx,
 @click.option('--profile', help='Profile to use from the config file', default=None)
 @click.pass_context
 def job_details(ctx,
-               apikey,
-               cloudos_url,
-               job_id,
-               output_format,
-               output_basename,
-               parameters,
-               verbose,
-               disable_ssl_verification,
-               ssl_cert,
-               profile):
+                apikey,
+                cloudos_url,
+                job_id,
+                output_format,
+                output_basename,
+                parameters,
+                verbose,
+                disable_ssl_verification,
+                ssl_cert,
+                profile):
     """Retrieve job details in CloudOS."""
     profile = profile or ctx.default_map['job']['details']['profile']
     # Create a dictionary with required and non-required params
@@ -826,7 +978,7 @@ def job_details(ctx,
 
     # Determine the execution platform based on jobType
     executors = {
-        'nextflowAWS':'Batch AWS',
+        'nextflowAWS': 'Batch AWS',
         'nextflowAzure': 'Batch Azure',
         'nextflowGcp': 'GCP',
         'nextflowHpc': 'HPC',
@@ -888,7 +1040,7 @@ def job_details(ctx,
             "Master Instance": str(j_details_h["masterInstance"]["usedInstance"]["type"]),
             "Storage": str(j_details_h["storageSizeInGb"]) + " GB",
             "Accelerated File Staging": str(j_details_h.get("usesFusionFileSystem", "None")),
-            "Task Resources": f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " + \
+            "Task Resources": f"{str(j_details_h['resourceRequirements']['cpu'])} CPUs, " +
                               f"{str(j_details_h['resourceRequirements']['ram'])} GB RAM"
 
         }
@@ -1250,10 +1402,9 @@ def list_workflows(ctx,
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
-@click.option("--platform", type=click.Choice(["github", "gitlab"]),
-              help=('Repository service where the workflow is located. Valid choices: github, gitlab. ' +
-                    'Default=github'),
-              default="github")
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
+              help='Name of the repository platform of the workflow. Default=github.',
+              default='github')
 @click.option("--workflow-name", help="The name that the workflow will have in CloudOS.", required=True)
 @click.option("-w", "--workflow-url", help="URL of the workflow repository.", required=True)
 @click.option("-d", "--workflow-docs-link", help="URL to the documentation of the workflow.", default='')
@@ -1276,7 +1427,7 @@ def import_wf(ctx,
               workflow_docs_link,
               cost_limit,
               workflow_description,
-              platform,
+              repository_platform,
               disable_ssl_verification,
               ssl_cert,
               profile):
@@ -1303,16 +1454,17 @@ def import_wf(ctx,
             apikey=apikey,
             cloudos_url=cloudos_url,
             workspace_id=workspace_id,
-            workflow_name=workflow_name
+            workflow_name=workflow_name,
+            repository_platform=repository_platform
         )
     )
 
     verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
-    repo_services = {"gitlab": ImportGitlab, "github": ImportGithub}
-    repo_cls = repo_services[platform]
-    repo_import = repo_cls(cloudos_url=cloudos_url, cloudos_apikey=apikey, workspace_id=workspace_id,
-                             platform=platform, workflow_name=workflow_name, workflow_url=workflow_url,
-                             workflow_docs_link=workflow_docs_link, cost_limit=cost_limit, workflow_description=workflow_description, verify=verify_ssl)
+    repo_import = ImportWorflow(
+        cloudos_url=cloudos_url, cloudos_apikey=apikey, workspace_id=workspace_id, platform=repository_platform,
+        workflow_name=workflow_name, workflow_url=workflow_url, workflow_docs_link=workflow_docs_link,
+        cost_limit=cost_limit, workflow_description=workflow_description, verify=verify_ssl
+    )
     workflow_id = repo_import.import_workflow()
     print(f'\tWorkflow {workflow_name} was imported successfully with the ' +
           f'following ID: {workflow_id}')
@@ -1865,7 +2017,7 @@ def remove_profile(ctx, profile):
               help=('Max time to wait (in seconds) to job completion. ' +
                     'Default=3600.'),
               default=3600)
-@click.option('--repository-platform',
+@click.option('--repository-platform', type=click.Choice(["github", "gitlab", "bitbucketServer"]),
               help='Name of the repository platform of the workflow. Default=github.',
               default='github')
 @click.option('--execution-platform',
@@ -2180,7 +2332,8 @@ def list_files(ctx,
 @click.option('-c', '--cloudos-url', default=CLOUDOS_URL, required=False, help='The CloudOS URL.')
 @click.option('--workspace-id', required=True, help='The CloudOS workspace ID.')
 @click.option('--project-name', required=True, help='The source project name.')
-@click.option('--destination-project-name', required=False, help='The destination project name. Defaults to the source project.')
+@click.option('--destination-project-name', required=False,
+              help='The destination project name. Defaults to the source project.')
 @click.option('--disable-ssl-verification', is_flag=True, help='Disable SSL certificate verification.')
 @click.option('--ssl-cert', help='Path to your SSL certificate file.')
 @click.option('--profile', default=None, help='Profile to use from the config file.')
@@ -2191,8 +2344,10 @@ def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspac
     """
     Move a file or folder from a source path to a destination path within or across CloudOS projects.
 
-    SOURCE_PATH [path] : the full path to the file or folder to move. It must be a 'Data' folder path. E.g.: 'Data/folderA/file.txt'\n
-    DESTINATION_PATH [path]: the full path to the destination folder. It must be a 'Data' folder path. E.g.: 'Data/folderB'
+    SOURCE_PATH [path]: the full path to the file or folder to move. It must be a 'Data' folder path.
+     E.g.: 'Data/folderA/file.txt'\n
+    DESTINATION_PATH [path]: the full path to the destination folder. It must be a 'Data' folder path.
+     E.g.: 'Data/folderB'
     """
 
     profile = profile or ctx.default_map['datasets']['move'].get('profile')
@@ -2272,7 +2427,8 @@ def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspac
         if found_source:
             break
     if not found_source:
-        click.echo(f"[ERROR] Item '{source_item_name}' not found in '{source_parent_path or '[project root]'}'", err=True)
+        click.echo(f"[ERROR] Item '{source_item_name}' not found in '{source_parent_path or '[project root]'}'",
+                   err=True)
         sys.exit(1)
 
     source_id = found_source["_id"]
@@ -2302,7 +2458,8 @@ def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspac
     except Exception as e:
         click.echo(f"[ERROR] Could not resolve destination path '{destination_path}': {str(e)}", err=True)
         sys.exit(1)
-    click.echo(f"Moving {source_kind} '{source_item_name}' to '{destination_path}' in project '{destination_project_name} ...")
+    click.echo(f"Moving {source_kind} '{source_item_name}' to '{destination_path}' " +
+               f"in project '{destination_project_name} ...")
     # === Perform Move ===
     try:
         response = source_client.move_files_and_folders(
@@ -2312,7 +2469,8 @@ def move_files(ctx, source_path, destination_path, apikey, cloudos_url, workspac
             target_kind=target_kind
         )
         if response.ok:
-           click.secho(f"[SUCCESS] {source_kind} '{source_item_name}' moved to '{destination_path}' in project '{destination_project_name}'.", fg="green", bold=True)
+            click.secho(f"[SUCCESS] {source_kind} '{source_item_name}' moved to '{destination_path}' " +
+                        f"in project '{destination_project_name}'.", fg="green", bold=True)
         else:
             click.echo(f"[ERROR] Move failed: {response.status_code} - {response.text}", err=True)
             sys.exit(1)

cloudos_cli-2.31.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.31.0'

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/clos.py

@@ -6,7 +6,8 @@ import requests
 import time
 import json
 from dataclasses import dataclass
-from cloudos_cli.utils.errors import BadRequestException
+from cloudos_cli.utils.cloud import find_cloud
+from cloudos_cli.utils.errors import BadRequestException, JoBNotCompletedException, NotAuthorisedException
 from cloudos_cli.utils.requests import retry_requests_get, retry_requests_post, retry_requests_put
 import pandas as pd
 
@@ -134,6 +135,145 @@ class Cloudos:
                   f'\t\t--job-id {job_id}\n')
         return {'name': j_name, 'id': job_id, 'status': j_status_h}
 
+    def get_storage_contents(self,  cloud_name, cloud_meta, container, path, workspace_id, verify):
+        """
+        Retrieves the contents of a storage container from the specified cloud service.
+
+        This method fetches the contents of a specified path within a storage container
+        on a cloud service (e.g., AWS S3 or Azure Blob). The request is authenticated
+        using an API key and requires valid parameters such as the workspace ID and path.
+
+        Parameters:
+            cloud_name (str): The name of the cloud service (e.g., 'aws' or 'azure').
+            container (str): The name of the storage container or bucket.
+            path (str): The file path or directory within the storage container.
+            workspace_id (str): The identifier of the workspace or team.
+            verify (bool): Whether to verify SSL certificates for the request.
+
+        Returns:
+            list: A list of contents retrieved from the specified cloud storage.
+
+        Raises:
+            BadRequestException: If the request to retrieve the contents fails with a
+            status code indicating an error.
+        """
+        headers = {
+            "Content-type": "application/json",
+            "apikey": self.apikey
+        }
+        cloud_data = {
+            "aws": {
+                "url": f"{self.cloudos_url}/api/v1/data-access/s3/bucket-contents",
+                "params": {
+                    "bucket": container,
+                    "path": path,
+                    "teamId": workspace_id
+                }
+            },
+            "azure": {
+                "url": f"{self.cloudos_url}/api/v1/data-access/azure/container-contents",
+                "container": "containerName",
+                "params": {
+                    "containerName": container,
+                    "path": path + "/",
+                    "storageAccountName": "",
+                    "teamId": workspace_id
+                }
+            }
+        }
+        if cloud_name == "azure":
+            cloud_data[cloud_name]["params"]["storageAccountName"] = cloud_meta["storage"]["storageAccount"]
+        params = cloud_data[cloud_name]["params"]
+        contents_req = retry_requests_get(cloud_data[cloud_name]["url"], params=params, headers=headers, verify=verify)
+        if contents_req.status_code >= 400:
+            raise BadRequestException(contents_req)
+        return contents_req.json()["contents"]
+
+    def get_job_logs(self, j_id, workspace_id, verify=True):
+        """
+        Get the location of the logs for the specified job
+        """
+        cloudos_url = self.cloudos_url
+        apikey = self.apikey
+        headers = {
+            "Content-type": "application/json",
+            "apikey": apikey
+        }
+        r = retry_requests_get(f"{cloudos_url}/api/v1/jobs/{j_id}", headers=headers, verify=verify)
+        if r.status_code == 401:
+            raise NotAuthorisedException
+        elif r.status_code >= 400:
+            raise BadRequestException(r)
+        r_json = r.json()
+        logs_obj = r_json["logs"]
+        job_workspace = r_json["team"]
+        if job_workspace != workspace_id:
+            raise ValueError("Workspace provided or configured is different from workspace where the job was executed")
+        cloud_name, cloud_meta, cloud_storage = find_cloud(self.cloudos_url, self.apikey, workspace_id, logs_obj)
+        container_name = cloud_storage["container"]
+        prefix_name = cloud_storage["prefix"]
+        logs_bucket = logs_obj[container_name]
+        logs_path = logs_obj[prefix_name]
+        contents_obj = self.get_storage_contents(cloud_name, cloud_meta, logs_bucket, logs_path, workspace_id, verify)
+        logs = {}
+        cloude_scheme = cloud_storage["scheme"]
+        storage_account_prefix = ''
+        if cloude_scheme == 'az':
+            storage_account_prefix = f'{workspace_id}.blob.core.windows.net/'
+        for item in contents_obj:
+            if not item["isDir"]:
+                filename = item["name"]
+                if filename == "stdout.txt":
+                    filename = "Nextflow standard output"
+                if filename == ".nextflow.log":
+                    filename = "Nextflow log"
+                if filename == "trace.txt":
+                    filename = "Trace file"
+                logs[filename] = f"{cloude_scheme}://{storage_account_prefix}{logs_bucket}/{item['path']}"
+        return logs
+
+    def get_job_results(self, j_id, workspace_id, verify=True):
+        """
+        Get the location of the results for the specified job
+        """
+        cloudos_url = self.cloudos_url
+        apikey = self.apikey
+        headers = {
+            "Content-type": "application/json",
+            "apikey": apikey
+        }
+        status = self.get_job_status(j_id, verify).json()["status"]
+        if status != JOB_COMPLETED:
+            raise JoBNotCompletedException(j_id, status)
+
+        r = retry_requests_get(f"{cloudos_url}/api/v1/jobs/{j_id}",
+                               headers=headers, verify=verify)
+        if r.status_code == 401:
+            raise NotAuthorisedException
+        if r.status_code >= 400:
+            raise BadRequestException(r)
+        req_obj = r.json()
+        job_workspace = req_obj["team"]
+        if job_workspace != workspace_id:
+            raise ValueError("Workspace provided or configured is different from workspace where the job was executed")
+        cloud_name, meta, cloud_storage = find_cloud(self.cloudos_url, self.apikey, workspace_id, req_obj["logs"])
+        # cont_name
+        results_obj = req_obj["results"]
+        results_container = results_obj[cloud_storage["container"]]
+        results_path = results_obj[cloud_storage["prefix"]]
+        scheme = cloud_storage["scheme"]
+        contents_obj = self.get_storage_contents(cloud_name, meta, results_container,
+                                                 results_path, workspace_id, verify)
+        storage_account_prefix = ''
+        if scheme == 'az':
+            storage_account_prefix = f'{workspace_id}.blob.core.windows.net/'
+        results = dict()
+        for item in contents_obj:
+            if item["isDir"]:
+                filename = item["name"]
+                results[filename] = f"{scheme}://{storage_account_prefix}{results_container}/{item['path']}"
+        return results
+
     def _create_cromwell_header(self):
         """Generates cromwell header.
 

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/import_wf/__init__.py

@@ -2,7 +2,7 @@
 Functions and classes related to importing workflows into CloudOS.
 """
 
-from .import_wf import WFImport, ImportGitlab, ImportGithub
+from .import_wf import WFImport, ImportWorflow
 
 
 __all__ = ['import_wf']

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/import_wf/import_wf.py

@@ -3,6 +3,8 @@ from urllib.parse import urlsplit
 from cloudos_cli.utils.errors import BadRequestException, AccountNotLinkedException
 from cloudos_cli.utils.requests import retry_requests_post, retry_requests_get
 import json
+from requests.exceptions import RetryError
+import sys
 
 
 class WFImport(ABC):
@@ -96,44 +98,46 @@ class WFImport(ABC):
         return content["_id"]
 
 
-# There are some duplicated lines here and on the github subclass. I did not put them in the abstract class because we
-# still don't know if the bitbucket data will come the same. If it does, then I will put as much as possible as part
-# of the abstract class
-class ImportGitlab(WFImport):
+class ImportWorflow(WFImport):
     def get_repo(self):
-        get_repo_url = f"{self.cloudos_url}/api/v1/git/gitlab/getPublicRepo"
-        self.repo_name = self.parsed_url.path.split("/")[-1]
-        self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
+        get_repo_url = f"{self.cloudos_url}/api/v1/git/{self.platform}/getPublicRepo"
+        if self.platform == "bitbucketServer":
+            # platform allows to add paths like /browse, so we need to check if the path ends with it
+            if self.parsed_url.path.endswith("browse"):
+                self.repo_name = self.parsed_url.path.split("/")[-2]
+            else:
+                self.repo_name = self.parsed_url.path.split("/")[-1]
+            self.repo_owner = self.parsed_url.path.split("/")[2]
+        else:
+            self.repo_name = self.parsed_url.path.split("/")[-1]
+            self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
         self.repo_host = f"{self.parsed_url.scheme}://{self.parsed_url.netloc}"
         get_repo_params = dict(repoName=self.repo_name, repoOwner=self.repo_owner, host=self.repo_host, teamId=self.workspace_id)
-        r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
-        if r.status_code == 404:
+        try:
+            r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
+        except RetryError as e:
+            # RetryError getting from missing BitBucket Server credentials
             raise AccountNotLinkedException(self.workflow_url)
-        elif r.status_code >= 400:
-            raise BadRequestException(r)
-        r_data = r.json()
-        self.payload["repository"]["repositoryId"] = r_data["id"]
-        self.payload["repository"]["name"] = r_data["name"]
-        self.payload["repository"]["owner"]["id"] = r_data["namespace"]["id"]
-        self.payload["repository"]["owner"]["login"] = r_data["namespace"]["full_path"]
-        self.payload["mainFile"] = self.main_file or self.get_repo_main_file()
 
-
-class ImportGithub(WFImport):
-    def get_repo(self):
-        get_repo_url = f"{self.cloudos_url}/api/v1/git/github/getPublicRepo"
-        self.repo_name = self.parsed_url.path.split("/")[-1]
-        self.repo_owner = "/".join(self.parsed_url.path.split("/")[1:-1])
-        self.repo_host = f"{self.parsed_url.scheme}://{self.parsed_url.netloc}"
-        get_repo_params = dict(repoName=self.repo_name, repoOwner=self.repo_owner, host=self.repo_host, teamId=self.workspace_id)
-        r = retry_requests_get(get_repo_url, params=get_repo_params, headers=self.headers)
+        # for Github and Gitlab the API gives very general errors on missing credentials
+        # therefore we only have these at the moment
         if r.status_code == 404:
             raise AccountNotLinkedException(self.workflow_url)
         elif r.status_code >= 400:
             raise BadRequestException(r)
+
         r_data = r.json()
-        self.payload["repository"]["repositoryId"] = r_data["id"]
+        if self.platform == "bitbucketServer":
+            self.payload["repository"]["repositoryId"] = r_data["name"]
+        else:
+            self.payload["repository"]["repositoryId"] = r_data["id"]
         self.payload["repository"]["name"] = r_data["name"]
-        self.payload["repository"]["owner"]["id"] = r_data["owner"]["id"]
-        self.payload["repository"]["owner"]["login"] = r_data["owner"]["login"]
+        owner_data = {
+            "bitbucketServer": ("project", "id", "key"),
+            "gitlab": ("namespace", "id", "full_path"),
+            "github": ("owner", "id", "login")
+        }
+        key, id_field, login_field = owner_data[self.platform]
+        self.payload["repository"]["owner"]["id"] = r_data[key][id_field]
+        self.payload["repository"]["owner"]["login"] = r_data[key][login_field]
         self.payload["mainFile"] = self.main_file or self.get_repo_main_file()

cloudos_cli-2.31.0/cloudos_cli/utils/__init__.py

@@ -0,0 +1,11 @@
+"""
+Utility functions and classes to use across the package.
+"""
+
+from .errors import BadRequestException, TimeOutException, AccountNotLinkedException, JoBNotCompletedException, NotAuthorisedException, NoCloudForWorkspaceException
+from .requests import retry_requests_get, retry_requests_post, retry_requests_put
+from .resources import format_bytes, ssl_selector
+from .cloud import find_cloud
+from .cloud import find_cloud
+
+__all__ = ['errors', 'requests', 'resources', 'cloud']

cloudos_cli-2.31.0/cloudos_cli/utils/cloud.py

@@ -0,0 +1,37 @@
+from cloudos_cli.utils.requests import retry_requests_get
+from cloudos_cli.utils import BadRequestException
+from cloudos_cli.utils.errors import NoCloudForWorkspaceException
+
+
+def find_cloud(cloudos_url, apikey, workspace_id, logs):
+    if "s3BucketName" in logs:
+        cloud_name = "aws"
+        meta = {}
+        storage = {
+                "container": "s3BucketName",
+                "prefix": "s3Prefix",
+                "scheme": "s3"
+            }
+        return cloud_name, meta, storage
+    else:
+        headers = {
+            "Content-Type": "application/json",
+            "Accept": "application/json",
+            "apikey": apikey
+        }
+        params = dict(teamId=workspace_id)
+        url = f"{cloudos_url}/api/v1/cloud/azure"
+        r = retry_requests_get(url, headers=headers, params=params)
+        if r.status_code >= 400:
+            raise BadRequestException(r)
+        if r.json() and r.text != "null":
+            cloud_data = r.json()
+            cloud_name = "azure"
+            storage = {
+                "container": "blobContainerName",
+                "prefix": "blobPrefix",
+                "scheme": "az"
+            }
+            return cloud_name, cloud_data, storage
+
+    raise NoCloudForWorkspaceException(workspace_id)

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli/utils/errors.py

@@ -41,4 +41,26 @@ class AccountNotLinkedException(Exception):
         msg = (f"The pipeline at the URL {wf_url} cannot be imported. Check that you repository account " +
                "has been linked in your cloudOS workspace")
         super(AccountNotLinkedException, self).__init__(msg)
-        self.wf_url = wf_url
+        self.wf_url = wf_url
+
+
+class JoBNotCompletedException(Exception):
+    def __init__(self, job, status):
+        msg = f"Job {job} has status {status}. Results are only available for jobs with status \"completed\""
+        super(JoBNotCompletedException, self).__init__(msg)
+        self.job = job
+        self.status = status
+
+
+class NotAuthorisedException(Exception):
+    def __init__(self):
+        msg = ("Not authorised to run this operation. Check your API key, and that the resource you request is "
+               "in the same workspace as the workspace specified in the cloudOS cli")
+        super(NotAuthorisedException, self).__init__(msg)
+
+
+class NoCloudForWorkspaceException(Exception):
+    def __init__(self, workspace_id):
+        msg = f"Workspace ID {workspace_id} is not associated with supported cloud providers. Check the workspace ID"
+        super(NoCloudForWorkspaceException, self).__init__(msg)
+        self.workspace_id = workspace_id

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.29.0
+Version: 2.31.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -420,6 +420,44 @@ command.
 Other options like `--wait-completion` are also available and work in the same way as for the `cloudos job run` command.
 Check `cloudos bash job --help` for more details.
 
+#### Get path to logs of job from CloudOS
+
+Get the path to "Nextflow logs", "Nextflow standard output", and "trace" files. It can be used only on your user's jobs, with any status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing logs...
+Logs URI: s3://path/to/location/of/logs
+
+Nextflow log: s3://path/to/location/of/logs/.nextflow.log
+
+Nextflow standard output: s3://path/to/location/of/logs/stdout.txt
+
+Trace file: s3://path/to/location/of/logs/trace.txt
+```
+
+#### Get path to result files of jobs from CloudOS
+
+Get the path where CloudOS stores the output files for a job. This can be used only on your user's jobs and for jobs with "completed" status.
+
+Example
+```console
+cloudos job logs --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --job-id "12345678910"
+    
+
+Executing results...
+results: s3://path/to/location/of/results/results/
+```
+
 #### Abort single or multiple jobs from CloudOS
 
 Aborts jobs in the CloudOS workspace that are either running or initialising. It can be used with one or more job IDs provided as a comma separated string using the `--job-ids` parameter.
@@ -695,8 +733,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
-- If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
+- The workflow repository is located at GitHub, GitLab or BitBucket Server (specified by the option `--repository-platform`. Available options: `github`, `gitlab` and `bitbucketServer`)
+- If your repository is private, you have access to the repository and to have linked your GitHub, Gitlab or Bitbucket server accounts to CloudOS.
 
 #### Usage of the workflow import command
 
@@ -712,7 +750,7 @@ cloudos workflow import \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --platform github 
+    --repository-platform github
 ```
 
 The expected output will be:
@@ -737,7 +775,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --platform github
+    --repository-platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.

{cloudos_cli-2.29.0 → cloudos_cli-2.31.0}/cloudos_cli.egg-info/SOURCES.txt

@@ -22,6 +22,7 @@ cloudos_cli/jobs/job.py
 cloudos_cli/queue/__init__.py
 cloudos_cli/queue/queue.py
 cloudos_cli/utils/__init__.py
+cloudos_cli/utils/cloud.py
 cloudos_cli/utils/errors.py
 cloudos_cli/utils/requests.py
 cloudos_cli/utils/resources.py

cloudos_cli-2.29.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.29.0'

cloudos_cli-2.29.0/cloudos_cli/utils/__init__.py

@@ -1,9 +0,0 @@
-"""
-Utility functions and classes to use across the package.
-"""
-
-from .errors import BadRequestException, TimeOutException
-from .requests import retry_requests_get,  retry_requests_post
-from .resources import format_bytes, ssl_selector
-
-__all__ = ['errors', 'requests', 'resources']