cloudos-cli 2.24.0__tar.gz → 2.26.0__tar.gz

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.24.0
+Version: 2.26.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -632,91 +632,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or Bitbucket server.
+- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
 - If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
-- You have got the `repository_id` and the `repository_project_id`.
-
-**How to get `repository_id` and `repository_project_id` from a GitHub repository**
-
-**Option 1: searching in the page source code**
-
-1. Go to the repository URL. Click on the right button of your mouse to get the following menu and click on "View Page Source".
-
-![Github Repo right click](docs/github_right_click.png)
-
-2. For collecting the `repository_project_id`, search for `octolytics-dimension-user_id` string in the source code. The `content` value is your `repository_project_id` (`30871219` in the example image).
-
-![Github Repo owner id](docs/github_user_id.png)
-
-3. For collecting the `repository_id`, search for `octolytics-dimension-repository_id` string in the source code. The `content` value is your `repository_id` (`122059362` in the example image).
-
-![Github Repo id](docs/github_repository_id.png)
-
-**Option 2: using github CLI**
-
-If you have access to the repository, you can use the following tools to collect the required values:
-
-- [gh](https://cli.github.com/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_project_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .owner.id)
-echo $repository_project_id
-30871219
-```
-
-For collecting the `repository_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .id)
-echo $repository_id
-122059362
-```
-
-**How to get `repository_project_id` from a Bitbucket server repository**
-
-For Bitbucket server repositories, only `repository_project_id` is required. To collect it:
-
-**Option 1: using the REST API from your browser**
-
-1. Create a REST API URL from your repo URL by adding `/rest/api/latest` to the URL: 
-
-```
-Original URL: https://bitbucket.com/projects/MYPROJECT/repos/my-repo
-REST API URL: https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo
-```
-
-> IMPORTANT NOTE: Please, as your repository original URL, do not use the "clone" URL provided by Bitbucket (the one with `.git` extension), use the actual browser URL, removing the terminal `/browse`.
-
-2. Use the REST API URL in a browser and it will generate a JSON output.
-
-3. Your `repository_project_id` is the value of the `project.id` field.
-
-![bitbucket project id](docs/bitbucket_project_id.png)
-
-**Option 2: using cURL**
-
-If you have access to the repository, you can use the following tools to collect the required value:
-
-- [cURL](https://curl.se/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-BITBUCKET_TOKEN="xxx"
-repository_project_id=$(curl https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo -H "Authorization: Bearer $BITBUCKET_TOKEN" | jq .project.id)
-echo $repository_project_id
-1234
-```
 
 #### Usage of the workflow import command
 
@@ -726,18 +643,13 @@ To import GitHub workflows to CloudOS, you can use the following command:
 # Example workflow to import: https://github.com/lifebit-ai/DeepVariant
 WORKFLOW_URL="https://github.com/lifebit-ai/DeepVariant"
 
-# You will need the repository_project_id and repository_id values explained above
-REPOSITORY_PROJECT_ID=30871219
-REPOSITORY_ID=122059362
-
 cloudos workflow import \
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
+    --platform github 
 ```
 
 The expected output will be:
@@ -762,25 +674,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
-```
-
-To import bitbucket server workflows, `--repository-id` parameter is not required:
-
-```bash
-WORKFLOW_URL="https://bitbucket.com/projects/MYPROJECT/repos/my-repo"
-
-# You will need only the repository_project_id
-REPOSITORY_PROJECT_ID=1234
-
-cloudos workflow import \
-    --cloudos-url $CLOUDOS \
-    --apikey $MY_API_KEY \
-    --workspace-id $WORKSPACE_ID \
-    --workflow-url $WORKFLOW_URL \
-    --workflow-name "new_name_for_the_bitbucket_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID
+    --platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -848,6 +742,25 @@ Platform workflows, i.e., those provided by CloudOS in your workspace as modules
 Therefore, CloudOS will automatically assign the valid queue and the user should not specify any queue using the `--job-queue` paramater.
 Any attempt of using this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
+#### Explore files programmatically
+
+##### Listing files
+
+To list files present in File Explorer in a given project (whether they are analysis results, cohorts etc.), the user can run the following command:
+```
+cloudos datasets ls <path> --profile <profile name>
+```
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to provide the following commands:
+```bash
+cloudos datasets ls <path> \
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+The output of this command is a list of files and folders present in the specified project.
+If the `<path>` is left empty, the command will return the list of folders present in the selected project.
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/README.md

@@ -597,91 +597,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or Bitbucket server.
+- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
 - If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
-- You have got the `repository_id` and the `repository_project_id`.
-
-**How to get `repository_id` and `repository_project_id` from a GitHub repository**
-
-**Option 1: searching in the page source code**
-
-1. Go to the repository URL. Click on the right button of your mouse to get the following menu and click on "View Page Source".
-
-![Github Repo right click](docs/github_right_click.png)
-
-2. For collecting the `repository_project_id`, search for `octolytics-dimension-user_id` string in the source code. The `content` value is your `repository_project_id` (`30871219` in the example image).
-
-![Github Repo owner id](docs/github_user_id.png)
-
-3. For collecting the `repository_id`, search for `octolytics-dimension-repository_id` string in the source code. The `content` value is your `repository_id` (`122059362` in the example image).
-
-![Github Repo id](docs/github_repository_id.png)
-
-**Option 2: using github CLI**
-
-If you have access to the repository, you can use the following tools to collect the required values:
-
-- [gh](https://cli.github.com/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_project_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .owner.id)
-echo $repository_project_id
-30871219
-```
-
-For collecting the `repository_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .id)
-echo $repository_id
-122059362
-```
-
-**How to get `repository_project_id` from a Bitbucket server repository**
-
-For Bitbucket server repositories, only `repository_project_id` is required. To collect it:
-
-**Option 1: using the REST API from your browser**
-
-1. Create a REST API URL from your repo URL by adding `/rest/api/latest` to the URL: 
-
-```
-Original URL: https://bitbucket.com/projects/MYPROJECT/repos/my-repo
-REST API URL: https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo
-```
-
-> IMPORTANT NOTE: Please, as your repository original URL, do not use the "clone" URL provided by Bitbucket (the one with `.git` extension), use the actual browser URL, removing the terminal `/browse`.
-
-2. Use the REST API URL in a browser and it will generate a JSON output.
-
-3. Your `repository_project_id` is the value of the `project.id` field.
-
-![bitbucket project id](docs/bitbucket_project_id.png)
-
-**Option 2: using cURL**
-
-If you have access to the repository, you can use the following tools to collect the required value:
-
-- [cURL](https://curl.se/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-BITBUCKET_TOKEN="xxx"
-repository_project_id=$(curl https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo -H "Authorization: Bearer $BITBUCKET_TOKEN" | jq .project.id)
-echo $repository_project_id
-1234
-```
 
 #### Usage of the workflow import command
 
@@ -691,18 +608,13 @@ To import GitHub workflows to CloudOS, you can use the following command:
 # Example workflow to import: https://github.com/lifebit-ai/DeepVariant
 WORKFLOW_URL="https://github.com/lifebit-ai/DeepVariant"
 
-# You will need the repository_project_id and repository_id values explained above
-REPOSITORY_PROJECT_ID=30871219
-REPOSITORY_ID=122059362
-
 cloudos workflow import \
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
+    --platform github 
 ```
 
 The expected output will be:
@@ -727,25 +639,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
-```
-
-To import bitbucket server workflows, `--repository-id` parameter is not required:
-
-```bash
-WORKFLOW_URL="https://bitbucket.com/projects/MYPROJECT/repos/my-repo"
-
-# You will need only the repository_project_id
-REPOSITORY_PROJECT_ID=1234
-
-cloudos workflow import \
-    --cloudos-url $CLOUDOS \
-    --apikey $MY_API_KEY \
-    --workspace-id $WORKSPACE_ID \
-    --workflow-url $WORKFLOW_URL \
-    --workflow-name "new_name_for_the_bitbucket_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID
+    --platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -813,6 +707,25 @@ Platform workflows, i.e., those provided by CloudOS in your workspace as modules
 Therefore, CloudOS will automatically assign the valid queue and the user should not specify any queue using the `--job-queue` paramater.
 Any attempt of using this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
+#### Explore files programmatically
+
+##### Listing files
+
+To list files present in File Explorer in a given project (whether they are analysis results, cohorts etc.), the user can run the following command:
+```
+cloudos datasets ls <path> --profile <profile name>
+```
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to provide the following commands:
+```bash
+cloudos datasets ls <path> \
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+The output of this command is a list of files and folders present in the specified project.
+If the `<path>` is left empty, the command will return the list of folders present in the selected project.
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/cloudos_cli/__main__.py

@@ -3,6 +3,7 @@
 import rich_click as click
 import cloudos_cli.jobs.job as jb
 from cloudos_cli.clos import Cloudos
+from cloudos_cli.import_wf.import_wf import ImportGitlab, ImportGithub
 from cloudos_cli.queue.queue import Queue
 import json
 import time
@@ -11,7 +12,7 @@ import os
 import urllib3
 from ._version import __version__
 from cloudos_cli.configure.configure import ConfigurationProfile
-
+from cloudos_cli.datasets import Datasets
 
 # GLOBAL VARS
 JOB_COMPLETED = 'completed'
@@ -64,9 +65,10 @@ def ssl_selector(disable_ssl_verification, ssl_cert):
 @click.pass_context
 def run_cloudos_cli(ctx):
     """CloudOS python package: a package for interacting with CloudOS."""
-    print(run_cloudos_cli.__doc__ + '\n')
-    print('Version: ' + __version__ + '\n')
     ctx.ensure_object(dict)
+    if ctx.invoked_subcommand not in ['datasets'] and ctx.args and ctx.args[0] == 'ls':
+        print(run_cloudos_cli.__doc__ + '\n')
+        print('Version: ' + __version__ + '\n')
     config_manager = ConfigurationProfile()
     profile_to_use = config_manager.determine_default_profile()
     if profile_to_use is None:
@@ -105,6 +107,9 @@ def run_cloudos_cli(ctx):
             },
             'bash': {
                 'job': shared_config
+            },
+            'datasets': {
+                'ls': shared_config
             }
         })
     else:
@@ -143,6 +148,9 @@ def run_cloudos_cli(ctx):
             },
             'bash': {
                 'job': shared_config
+            },
+            'datasets': {
+                'ls': shared_config
             }
         })
 
@@ -183,6 +191,14 @@ def bash():
     print(bash.__doc__ + '\n')
 
 
+@run_cloudos_cli.group()
+@click.pass_context
+def datasets(ctx):
+    """CloudOS datasets functionality."""
+    if ctx.args and ctx.args[0] != 'ls':
+        print(datasets.__doc__ + '\n')
+
+
 @run_cloudos_cli.group(invoke_without_command=True)
 @click.option('--profile', help='Profile to use from the config file', default='default')
 @click.option('--make-default',
@@ -1037,29 +1053,21 @@ def list_workflows(ctx,
               required=True)
 @click.option('-c',
               '--cloudos-url',
-              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              help=('The CloudOS url you are trying to access to. ' +
+                    f'Default={CLOUDOS_URL}.'),
               default=CLOUDOS_URL)
 @click.option('--workspace-id',
               help='The specific CloudOS workspace id.',
               required=True)
-@click.option('--workflow-url',
-              help=('URL of the workflow to import. Please, note that it should ' +
-                    'be the URL shown in the browser, and it should come without ' +
-                    'any of the .git or /browse extensions.'),
-              required=True)
-@click.option('--workflow-name',
-              help="The name that the workflow will have in CloudOS",
-              required=True)
-@click.option('--workflow-docs-link',
-              help="Workflow documentation URL.",
-              default='')
-@click.option('--repository-project-id',
-              type=int,
-              help="The ID of your repository project",
-              required=True)
-@click.option('--repository-id',
-              type=int,
-              help="The ID of your repository. Only required for GitHub repositories")
+@click.option("--platform", type=click.Choice(["github", "gitlab"]),
+              help=('Repository service where the workflow is located. Valid choices: github, gitlab. ' +
+                    'Default=github'),
+              default="github")
+@click.option("--workflow-name", help="The name that the workflow will have in CloudOS.", required=True)
+@click.option("-w", "--workflow-url", help="URL of the workflow repository.", required=True)
+@click.option("-d", "--workflow-docs-link", help="URL to the documentation of the workflow.", default='')
+@click.option("--cost-limit", help="Cost limit for the workflow. Default: $30 USD.", default=30)
+@click.option("--workflow-description", help="Workflow description", default="")
 @click.option('--disable-ssl-verification',
               help=('Disable SSL certificate verification. Please, remember that this option is ' +
                     'not generally recommended for security reasons.'),
@@ -1068,19 +1076,22 @@ def list_workflows(ctx,
               help='Path to your SSL certificate file.')
 @click.option('--profile', help='Profile to use from the config file', default=None)
 @click.pass_context
-def import_workflows(ctx,
-                     apikey,
-                     cloudos_url,
-                     workspace_id,
-                     workflow_url,
-                     workflow_name,
-                     repository_project_id,
-                     workflow_docs_link,
-                     repository_id,
-                     disable_ssl_verification,
-                     ssl_cert,
-                     profile):
-    """Imports workflows to CloudOS."""
+def import_wf(ctx,
+              apikey,
+              cloudos_url,
+              workspace_id,
+              workflow_name,
+              workflow_url,
+              workflow_docs_link,
+              cost_limit,
+              workflow_description,
+              platform,
+              disable_ssl_verification,
+              ssl_cert,
+              profile):
+    """
+    Import workflows from supported repository providers.
+    """
     profile = profile or ctx.default_map['workflow']['import']['profile']
     # Create a dictionary with required and non-required params
     required_dict = {
@@ -1106,16 +1117,12 @@ def import_workflows(ctx,
     )
 
     verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
-    print('Executing workflow import...\n')
-    print('\t[Message] Only Nextflow workflows are currently supported.\n')
-    cl = Cloudos(cloudos_url, apikey, None)
-    workflow_id = cl.workflow_import(workspace_id,
-                                     workflow_url,
-                                     workflow_name,
-                                     repository_project_id,
-                                     workflow_docs_link,
-                                     repository_id,
-                                     verify=verify_ssl)
+    repo_services = {"gitlab": ImportGitlab, "github": ImportGithub}
+    repo_cls = repo_services[platform]
+    repo_import = repo_cls(cloudos_url=cloudos_url, cloudos_apikey=apikey, workspace_id=workspace_id,
+                             platform=platform, workflow_name=workflow_name, workflow_url=workflow_url,
+                             workflow_docs_link=workflow_docs_link, cost_limit=cost_limit, workflow_description=workflow_description, verify=verify_ssl)
+    workflow_id = repo_import.import_workflow()
     print(f'\tWorkflow {workflow_name} was imported successfully with the ' +
           f'following ID: {workflow_id}')
 
@@ -1825,5 +1832,101 @@ def run_bash_job(ctx,
               f'\t\t--job-id {j_id}\n')
 
 
+@datasets.command(name="ls")
+@click.argument("path", required=False, nargs=1)
+@click.option('-k',
+              '--apikey',
+              help='Your CloudOS API key.',
+              required=True)
+@click.option('-c',
+              '--cloudos-url',
+              help=(f'The CloudOS url you are trying to access to. Default={CLOUDOS_URL}.'),
+              default=CLOUDOS_URL,
+              required=True)
+@click.option('--workspace-id',
+              help='The specific CloudOS workspace id.',
+              required=True)
+@click.option('--disable-ssl-verification',
+              help=('Disable SSL certificate verification. Please, remember that this option is ' +
+                    'not generally recommended for security reasons.'),
+              is_flag=True)
+@click.option('--ssl-cert',
+              help='Path to your SSL certificate file.')
+@click.option('--project-name',
+              help='The name of a CloudOS project.')
+@click.option('--profile', help='Profile to use from the config file', default=None)
+@click.pass_context
+def list_files(ctx,
+               apikey,
+               cloudos_url,
+               workspace_id,
+               disable_ssl_verification,
+               ssl_cert,
+               project_name,
+               profile,
+               path):
+    """List contents of a path within a CloudOS workspace dataset."""
+
+    # fallback to ctx default if profile not specified
+    profile = profile or ctx.default_map['datasets']['list'].get('profile')
+
+    config_manager = ConfigurationProfile()
+
+    required_dict = {
+        'apikey': True,
+        'workspace_id': True,
+        'workflow_name': False,
+        'project_name': False
+    }
+
+    # Unpack profile values first
+    apikey, cloudos_url, workspace_id, workflow_name, repository_platform, execution_platform, project_name = (
+        config_manager.load_profile_and_validate_data(
+            ctx,
+            INIT_PROFILE,
+            CLOUDOS_URL,
+            profile=profile,
+            required_dict=required_dict,
+            apikey=apikey,
+            cloudos_url=cloudos_url,
+            workspace_id=workspace_id,
+            workflow_name=None,
+            repository_platform=None,
+            execution_platform=None,
+            project_name=project_name
+        )
+    )
+
+    verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
+
+    datasets = Datasets(
+        cloudos_url=cloudos_url,
+        apikey=apikey,
+        workspace_id=workspace_id,
+        project_name=project_name,
+        verify=verify_ssl,
+        cromwell_token=None
+    )
+
+    try:
+        result = datasets.list_folder_content(path)
+        contents = result.get("contents") or result.get("datasets", [])
+        if not contents:
+            files = result.get("files", [])
+            folders = result.get("folders", [])
+            contents = [{"name": f["name"], "isDir": False} for f in files] + \
+                       [{"name": f["name"], "isDir": True} for f in folders]
+
+        for item in contents:
+            name = item.get("name", "")
+            if item.get("isDir"):
+                name = click.style(name, fg="blue", underline=True)
+            click.echo(name)
+
+    except Exception as e:
+        click.echo(f"[ERROR] {str(e)}", err=True)
+
+
 if __name__ == "__main__":
     run_cloudos_cli()
+

cloudos_cli-2.26.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.26.0'

cloudos_cli-2.26.0/cloudos_cli/datasets/__init__.py

@@ -0,0 +1,8 @@
+"""
+Functions and classes related to datasets.
+"""
+
+from .datasets import Datasets
+
+
+__all__ = ['datasets']

cloudos_cli-2.26.0/cloudos_cli/datasets/datasets.py

@@ -0,0 +1,322 @@
+"""
+This is the main class for file explorer (datasets).
+"""
+
+from dataclasses import dataclass
+from typing import Union
+from cloudos_cli.clos import Cloudos
+from cloudos_cli.utils.requests import retry_requests_get
+
+
+@dataclass
+class Datasets(Cloudos):
+    """Class for file explorer.
+
+    Parameters
+    ----------
+    cloudos_url : string
+        The CloudOS service url.
+    apikey : string
+        Your CloudOS API key.
+    workspace_id : string
+        The specific Cloudos workspace id.
+    project_name : string
+        The name of a CloudOS project.
+    verify: [bool|string]
+        Whether to use SSL verification or not. Alternatively, if
+        a string is passed, it will be interpreted as the path to
+        the SSL certificate file.
+    project_id : string
+        The CloudOS project id for a given project name.
+    """
+    workspace_id: str
+    project_name: str
+    verify: Union[bool, str] = True
+    project_id: str = None
+
+    @property
+    def project_id(self) -> str:
+        return self._project_id
+
+    @project_id.setter
+    def project_id(self, v) -> None:
+        if isinstance(v, property):
+            # Fetch the value as not defined by user.
+            self._project_id = self.fetch_cloudos_id(
+                self.apikey,
+                self.cloudos_url,
+                'projects',
+                self.workspace_id,
+                self.project_name,
+                verify=self.verify)
+        else:
+            # Let the user define the value.
+            self._project_id = v
+
+    def fetch_cloudos_id(self,
+                         apikey,
+                         cloudos_url,
+                         resource,
+                         workspace_id,
+                         name,
+                         mainfile=None,
+                         importsfile=None,
+                         repository_platform='github',
+                         verify=True):
+        """Fetch the cloudos id for a given name.
+
+        Parameters
+        ----------
+        apikey : string
+            Your CloudOS API key
+        cloudos_url : string
+            The CloudOS service url.
+        resource : string
+            The resource you want to fetch from. E.g.: projects.
+        workspace_id : string
+            The specific Cloudos workspace id.
+        name : string
+            The name of a CloudOS resource element.
+        mainfile : string
+            The name of the mainFile used by the workflow. Only used when resource == 'workflows'.
+            Required for WDL pipelines as different mainFiles could be loaded for a single
+            pipeline.
+        importsfile : string
+            The name of the importsFile used by the workflow. Optional and only used for WDL pipelines
+            as different importsFiles could be loaded for a single pipeline.
+        repository_platform : string
+            The name of the repository platform of the workflow resides.
+        verify: [bool|string]
+            Whether to use SSL verification or not. Alternatively, if
+            a string is passed, it will be interpreted as the path to
+            the SSL certificate file.
+
+        Returns
+        -------
+        project_id : string
+            The CloudOS project id for a given project name.
+        """
+        allowed_resources = ['projects', 'workflows']
+        if resource not in allowed_resources:
+            raise ValueError('Your specified resource is not supported. ' +
+                             f'Use one of the following: {allowed_resources}')
+        if resource == 'workflows':
+            content = self.get_workflow_list(workspace_id, verify=verify)
+            for element in content:
+                if (element["name"] == name and element["workflowType"] == "docker" and
+                        not element["archived"]["status"]):
+                    return element["_id"]  # no mainfile or importsfile
+                if (element["name"] == name and
+                        element["repository"]["platform"] == repository_platform and
+                        not element["archived"]["status"]):
+                    if mainfile is None:
+                        return element["_id"]
+                    elif element["mainFile"] == mainfile:
+                        if importsfile is None and "importsFile" not in element.keys():
+                            return element["_id"]
+                        elif "importsFile" in element.keys() and element["importsFile"] == importsfile:
+                            return element["_id"]
+        elif resource == 'projects':
+            content = self.get_project_list(workspace_id, verify=verify)
+            # New API projects endpoint spec
+            for element in content:
+                if element["name"] == name:
+                    return element["_id"]
+        if mainfile is not None:
+            raise ValueError(f'[ERROR] A workflow named \'{name}\' with a mainFile \'{mainfile}\'' +
+                             f' and an importsFile \'{importsfile}\' was not found')
+        else:
+            raise ValueError(f'[ERROR] No {name} element in {resource} was found')
+    def list_project_content(self):
+        """
+        Fetch the information of the directories present in the projects.
+
+        Uses
+        ----------
+        apikey : string
+            Your CloudOS API key
+        cloudos_url : string
+            The CloudOS service url.
+        workspace_id : string
+            The specific Cloudos workspace id.
+        project_id
+            The specific project id
+        """
+        headers = {
+            "Content-type": "application/json",
+            "apikey": self.apikey
+        }
+        r = retry_requests_get("{}/api/v2/datasets?projectId={}&teamId={}".format(self.cloudos_url,
+                                                                                  self.project_id,
+                                                                                  self.workspace_id),
+                               headers=headers, verify=self.verify)
+        return r.json()
+
+    def list_datasets_content(self, folder_name):
+        """Uses
+        ----------
+        apikey : string
+            Your CloudOS API key
+        cloudos_url : string
+            The CloudOS service url.
+        workspace_id : string
+            The specific Cloudos workspace id.
+        project_id : string
+            The specific project id
+        folder_name : string
+            The requested folder name
+        """
+        # Prepare api request for CloudOS to fetch dataset info
+        headers = {
+            "Content-type": "application/json",
+            "apikey": self.apikey
+        }
+        pro_fol = self.list_project_content()
+        folder_id = None
+
+        if folder_name == 'AnalysesResults':
+            folder_name = 'Analyses Results'
+
+        for folder in pro_fol.get("datasets", []):
+            if folder['name'] == folder_name:
+                folder_id = folder['_id']
+        if not folder_id:
+            raise ValueError(f"Folder '{folder_name}' not found in project '{self.project_name}'.")
+        r = retry_requests_get("{}/api/v1/datasets/{}/items?teamId={}".format(self.cloudos_url,
+                                                                              folder_id,
+                                                                              self.workspace_id),
+                                headers=headers, verify=self.verify)
+        return r.json()
+    def list_s3_folder_content(self, s3_bucket_name, s3_relative_path):
+        """Uses
+        ----------
+        apikey : string
+            Your CloudOS API key
+        cloudos_url : string
+            The CloudOS service url.
+        workspace_id : string
+            The specific Cloudos workspace id.
+        project_id : string
+            The specific project id
+        s3_bucket_name : string
+            The s3 bucket name
+        s3_relative_path: string
+            The relative path in the s3 bucket
+        """
+        # Prepare api request for CloudOS to fetch dataset info
+        headers = {
+            "Content-type": "application/json",
+            "apikey": self.apikey
+        }
+
+        r = retry_requests_get("{}/api/v1/data-access/s3/bucket-contents?bucket={}&path={}&teamId={}".format(self.cloudos_url,
+                                                                                                             s3_bucket_name,
+                                                                                                             s3_relative_path,
+                                                                                                             self.workspace_id),
+                                headers=headers, verify=self.verify)
+        raw = r.json()
+
+        #  Normalize response
+        normalized = {"folders": [], "files": []}
+        for item in raw.get("contents", []):
+            if item.get("isDir"):
+                item["folderType"] = "S3Folder"  # 👈 inject folderType
+                item["s3BucketName"] = s3_bucket_name
+                item["s3Prefix"] = item['path']
+                normalized["folders"].append(item)
+            else:
+                item["s3Prefix"] = item['path']
+                item["s3BucketName"] = s3_bucket_name
+
+                normalized["files"].append(item)
+
+        return normalized
+
+    def list_virtual_folder_content(self, folder_id):
+        """Uses
+        ----------
+        apikey : string
+            Your CloudOS API key
+        cloudos_url : string
+            The CloudOS service url.
+        workspace_id : string
+            The specific Cloudos workspace id.
+        project_id : string
+            The specific project id
+        folder_id : string
+            The folder id of the folder whose content are to be listed
+        """
+        headers = {
+            "Content-type": "application/json",
+            "apikey": self.apikey
+        }
+
+        r = retry_requests_get("{}/api/v1/folders/virtual/{}/items?teamId={}".format(self.cloudos_url,
+                                                                                     folder_id,
+                                                                                     self.workspace_id),
+                                headers=headers, verify=self.verify)
+        return r.json()
+    def list_folder_content(self, path=None):
+        """
+        Wrapper to list contents of a CloudOS folder.
+
+        Parameters
+        ----------
+        path : str, optional
+            A path like 'TopFolder', 'TopFolder/Subfolder', or deeper.
+            If None, lists all top-level datasets in the project.
+
+        Returns
+        -------
+        dict
+            JSON response from the appropriate CloudOS endpoint.
+        """
+        if not path:
+            return self.list_project_content()
+
+        parts = path.strip('/').split('/')
+
+        if len(parts) == 1:
+            return self.list_datasets_content(parts[0])
+
+        dataset_name = parts[0]
+        folder_content = self.list_datasets_content(dataset_name)
+
+        path_depth = 1
+        while path_depth < len(parts):
+            job_name = parts[path_depth]
+            found = False
+
+            for job_folder in folder_content.get("folders", []):
+                if job_folder["name"] == job_name:
+                    found = True
+                    folder_type = job_folder.get("folderType")
+
+                    if folder_type == "S3Folder":
+                        s3_bucket_name = job_folder['s3BucketName']
+                        s3_relative_path = job_folder['s3Prefix']
+                        if path_depth == len(parts) - 1:
+                            return self.list_s3_folder_content(s3_bucket_name, s3_relative_path)
+                        else:
+                            sub_path = '/'.join(parts[0:path_depth+1])
+                            folder_content = self.list_folder_content(sub_path)
+                            path_depth += 1
+                            break
+
+                    elif folder_type == "VirtualFolder":
+                        folder_id = job_folder['_id']
+                        if path_depth == len(parts) - 1:
+                            return self.list_virtual_folder_content(folder_id)
+                        else:
+                            sub_path = '/'.join(parts[0:path_depth+1])
+                            folder_content = self.list_folder_content(sub_path)
+                            path_depth += 1
+                            break
+
+                    else:
+                        raise ValueError(f"Unsupported folder type '{folder_type}' for path '{path}'")
+
+            if not found:
+                raise ValueError(f"Folder '{job_name}' not found under dataset '{dataset_name}'")
+
+        return folder_content

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/cloudos_cli/utils/errors.py

@@ -30,3 +30,15 @@ class TimeOutException(Exception):
                "Status: {}; Reason: {}".format(rv.status_code, rv.reason))
         super(TimeOutException, self).__init__(msg)
         self.rv = rv
+
+
+class AccountNotLinkedException(Exception):
+    """
+    Displays a meaningful message when the user tries to import a repository from an account that is not linked
+    with their cloudOS account
+    """
+    def __init__(self, wf_url):
+        msg = (f"The pipeline at the URL {wf_url} cannot be imported. Check that you repository account " +
+               "has been linked in your cloudOS workspace")
+        super(AccountNotLinkedException, self).__init__(msg)
+        self.wf_url = wf_url

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.24.0
+Version: 2.26.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -632,91 +632,8 @@ The collected workflows are those that can be found in "WORKSPACE TOOLS" section
 You can import new workflows to your CloudOS workspaces. The only requirements are:
 
 - The workflow is a Nextflow pipeline.
-- The workflow repository is located at GitHub or Bitbucket server.
+- The workflow repository is located at GitHub or GitLab (specified by the option `--platform`. Available options: `github`, `gitlab`)
 - If your repository is private, you have access to the repository and you have linked your GitHub or Bitbucket server accounts to CloudOS.
-- You have got the `repository_id` and the `repository_project_id`.
-
-**How to get `repository_id` and `repository_project_id` from a GitHub repository**
-
-**Option 1: searching in the page source code**
-
-1. Go to the repository URL. Click on the right button of your mouse to get the following menu and click on "View Page Source".
-
-![Github Repo right click](docs/github_right_click.png)
-
-2. For collecting the `repository_project_id`, search for `octolytics-dimension-user_id` string in the source code. The `content` value is your `repository_project_id` (`30871219` in the example image).
-
-![Github Repo owner id](docs/github_user_id.png)
-
-3. For collecting the `repository_id`, search for `octolytics-dimension-repository_id` string in the source code. The `content` value is your `repository_id` (`122059362` in the example image).
-
-![Github Repo id](docs/github_repository_id.png)
-
-**Option 2: using github CLI**
-
-If you have access to the repository, you can use the following tools to collect the required values:
-
-- [gh](https://cli.github.com/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_project_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .owner.id)
-echo $repository_project_id
-30871219
-```
-
-For collecting the `repository_id`:
-
-```
-# If your repo URL is https://github.com/lifebit-ai/DeepVariant
-OWNER="lifebit-ai"
-REPO="DeepVariant"
-repository_id=$(gh api -H "Accept: application/vnd.github+json" repos/$OWNER/$REPO | jq .id)
-echo $repository_id
-122059362
-```
-
-**How to get `repository_project_id` from a Bitbucket server repository**
-
-For Bitbucket server repositories, only `repository_project_id` is required. To collect it:
-
-**Option 1: using the REST API from your browser**
-
-1. Create a REST API URL from your repo URL by adding `/rest/api/latest` to the URL: 
-
-```
-Original URL: https://bitbucket.com/projects/MYPROJECT/repos/my-repo
-REST API URL: https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo
-```
-
-> IMPORTANT NOTE: Please, as your repository original URL, do not use the "clone" URL provided by Bitbucket (the one with `.git` extension), use the actual browser URL, removing the terminal `/browse`.
-
-2. Use the REST API URL in a browser and it will generate a JSON output.
-
-3. Your `repository_project_id` is the value of the `project.id` field.
-
-![bitbucket project id](docs/bitbucket_project_id.png)
-
-**Option 2: using cURL**
-
-If you have access to the repository, you can use the following tools to collect the required value:
-
-- [cURL](https://curl.se/)
-- [jq](https://jqlang.github.io/jq/download/)
-
-For collecting the `repository_project_id`:
-
-```
-BITBUCKET_TOKEN="xxx"
-repository_project_id=$(curl https://bitbucket.com/rest/api/latest/projects/MYPROJECT/repos/my-repo -H "Authorization: Bearer $BITBUCKET_TOKEN" | jq .project.id)
-echo $repository_project_id
-1234
-```
 
 #### Usage of the workflow import command
 
@@ -726,18 +643,13 @@ To import GitHub workflows to CloudOS, you can use the following command:
 # Example workflow to import: https://github.com/lifebit-ai/DeepVariant
 WORKFLOW_URL="https://github.com/lifebit-ai/DeepVariant"
 
-# You will need the repository_project_id and repository_id values explained above
-REPOSITORY_PROJECT_ID=30871219
-REPOSITORY_ID=122059362
-
 cloudos workflow import \
     --cloudos-url $CLOUDOS \
     --apikey $MY_API_KEY \
     --workspace-id $WORKSPACE_ID \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
+    --platform github 
 ```
 
 The expected output will be:
@@ -762,25 +674,7 @@ cloudos workflow import \
     --workflow-url $WORKFLOW_URL \
     --workflow-name "new_name_for_the_github_workflow" \
     --workflow-docs-link "https://github.com/lifebit-ai/DeepVariant/blob/master/README.md" \
-    --repository-project-id $REPOSITORY_PROJECT_ID \
-    --repository-id $REPOSITORY_ID
-```
-
-To import bitbucket server workflows, `--repository-id` parameter is not required:
-
-```bash
-WORKFLOW_URL="https://bitbucket.com/projects/MYPROJECT/repos/my-repo"
-
-# You will need only the repository_project_id
-REPOSITORY_PROJECT_ID=1234
-
-cloudos workflow import \
-    --cloudos-url $CLOUDOS \
-    --apikey $MY_API_KEY \
-    --workspace-id $WORKSPACE_ID \
-    --workflow-url $WORKFLOW_URL \
-    --workflow-name "new_name_for_the_bitbucket_workflow" \
-    --repository-project-id $REPOSITORY_PROJECT_ID
+    --platform github
 ```
 
 > NOTE: please, take into account that importing workflows using cloudos-cli is not yet available in all the CloudOS workspaces. If you try to use this feature in a non-prepared workspace you will get the following error message: `It seems your API key is not authorised. Please check if your workspace has support for importing workflows using cloudos-cli`.
@@ -848,6 +742,25 @@ Platform workflows, i.e., those provided by CloudOS in your workspace as modules
 Therefore, CloudOS will automatically assign the valid queue and the user should not specify any queue using the `--job-queue` paramater.
 Any attempt of using this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
+#### Explore files programmatically
+
+##### Listing files
+
+To list files present in File Explorer in a given project (whether they are analysis results, cohorts etc.), the user can run the following command:
+```
+cloudos datasets ls <path> --profile <profile name>
+```
+Please, note that in the above example a preconfigured profile has been used. If no profile is provided and there is no default profile, the user will need to provide the following commands:
+```bash
+cloudos datasets ls <path> \
+    --cloudos-url $CLOUDOS \
+    --apikey $MY_API_KEY \
+    --workspace-id $WORKSPACE_ID \
+    --project-name $PROJEC_NAME
+```
+The output of this command is a list of files and folders present in the specified project.
+If the `<path>` is left empty, the command will return the list of folders present in the selected project.
+
 ### WDL pipeline support
 
 #### Cromwell server managing

{cloudos_cli-2.24.0 → cloudos_cli-2.26.0}/cloudos_cli.egg-info/SOURCES.txt

@@ -13,6 +13,8 @@ cloudos_cli.egg-info/requires.txt
 cloudos_cli.egg-info/top_level.txt
 cloudos_cli/configure/__init__.py
 cloudos_cli/configure/configure.py
+cloudos_cli/datasets/__init__.py
+cloudos_cli/datasets/datasets.py
 cloudos_cli/jobs/__init__.py
 cloudos_cli/jobs/job.py
 cloudos_cli/queue/__init__.py

cloudos_cli-2.24.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.24.0'