cloudos-cli 2.60.2__tar.gz → 2.60.4__tar.gz

{cloudos_cli-2.60.2 → cloudos_cli-2.60.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.60.2
+Version: 2.60.4
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -43,60 +43,73 @@ Python package for interacting with CloudOS
 
 ## Table of Contents
 
-- [Requirements](#requirements)
-- [Installation](#installation)
-  - [From PyPI](#from-pypi)
-  - [Docker Image](#docker-image)
-  - [From Github](#from-github)
-- [Usage](#usage)
-- [Configuration](#configuration)
-  - [Configure Default Profile](#configure-default-profile)
-  - [Configure Named Profile](#configure-named-profile)
-  - [Change the Default Profile](#change-the-default-profile)
-  - [List Profiles](#list-profiles)
-  - [Remove Profile](#remove-profile)
-- [Commands](#commands)
-  - [Configure](#configure)
-  - [Project](#project)
-    - [List Projects](#list-projects)
-    - [Create Projects](#create-projects)
-  - [Queue](#queue)
-    - [List Queues](#list-queues)
-  - [Workflow](#workflow)
-    - [List All Available Workflows](#list-all-available-workflows)
-    - [Import a Nextflow Workflow](#import-a-nextflow-workflow)
-  - [Nextflow Jobs](#nextflow-jobs)
-    - [Submit a Job](#submit-a-job)
-    - [Get Job Logs](#get-job-logs)
-    - [Get Job Results](#get-job-results)
-    - [Clone or Resume Job](#clone-or-resume-job)
-    - [Abort Jobs](#abort-jobs)
-    - [Check Job Status](#check-job-status)
-    - [Get Job Details](#get-job-details)
-    - [Get Job Workdir](#get-job-workdir)
-    - [List Jobs](#list-jobs)
-  - [Bash Jobs](#bash-jobs)
-    - [Send Array Job](#send-array-job)
-    - [Submit a Bash Array Job](#submit-a-bash-array-job)
-    - [Use multiple projects for files in --parameter option](#use-multiple-projects-for-files-in---parameter-option)
-  - [Datasets](#datasets)
-    - [List Files](#list-files)
-    - [Move Files](#move-files)
-    - [Rename Files](#rename-files)
-    - [Copy Files](#copy-files)
-    - [Link S3 Folders to Interactive Analysis](#link-s3-folders-to-interactive-analysis)
-    - [Create Folder](#create-folder)
-    - [Remove Files or Folders](#remove-files-or-folders)
-  - [Procurement](#procurement)
-    - [List Procurement Images](#list-procurement-images)
-    - [Set Procurement Organization Image](#set-procurement-organization-image)
-    - [Reset Procurement Organization Image](#reset-procurement-organization-image)
-  - [Cromwell and WDL Pipeline Support](#cromwell-and-wdl-pipeline-support)
-    - [Manage Cromwell Server](#manage-cromwell-server)
-    - [Run WDL Workflows](#run-wdl-workflows)
-- [Python API Usage](#python-api-usage)
-  - [Running WDL pipelines using your own scripts](#running-wdl-pipelines-using-your-own-scripts)
-- [Unit Testing](#unit-testing)
+- [cloudos-cli](#cloudos-cli)
+  - [Table of Contents](#table-of-contents)
+  - [Requirements](#requirements)
+  - [Installation](#installation)
+    - [From PyPI](#from-pypi)
+    - [Docker Image](#docker-image)
+    - [From Github](#from-github)
+  - [Usage](#usage)
+  - [Configuration](#configuration)
+    - [Configure Default Profile](#configure-default-profile)
+    - [Configure Named Profile](#configure-named-profile)
+    - [Change the Default Profile](#change-the-default-profile)
+    - [List Profiles](#list-profiles)
+    - [Remove Profile](#remove-profile)
+  - [Commands](#commands)
+    - [Configure](#configure)
+    - [Project](#project)
+      - [List Projects](#list-projects)
+      - [Create Projects](#create-projects)
+    - [Queue](#queue)
+      - [List Queues](#list-queues)
+    - [Workflow](#workflow)
+      - [List All Available Workflows](#list-all-available-workflows)
+      - [Import a Nextflow Workflow](#import-a-nextflow-workflow)
+    - [Nextflow Jobs](#nextflow-jobs)
+      - [Submit a Job](#submit-a-job)
+      - [Get Job Logs](#get-job-logs)
+      - [Get Job Results](#get-job-results)
+      - [Clone or Resume job](#clone-or-resume-job)
+      - [Query working directory of job](#query-working-directory-of-job)
+      - [Abort single or multiple jobs from CloudOS](#abort-single-or-multiple-jobs-from-cloudos)
+      - [Abort Jobs](#abort-jobs)
+      - [Check Job Status](#check-job-status)
+      - [Get Job Details](#get-job-details)
+      - [Get Job Workdir](#get-job-workdir)
+      - [List Jobs](#list-jobs)
+    - [Bash Jobs](#bash-jobs)
+      - [Send Array Job](#send-array-job)
+      - [Submit a Bash Array Job](#submit-a-bash-array-job)
+        - [Options](#options)
+          - [Array File](#array-file)
+          - [Separator](#separator)
+          - [List Columns](#list-columns)
+          - [Array File Project](#array-file-project)
+          - [Disable Column Check](#disable-column-check)
+          - [Array Parameter](#array-parameter)
+          - [Custom Script Path](#custom-script-path)
+          - [Custom Script Project](#custom-script-project)
+      - [Use multiple projects for files in `--parameter` option](#use-multiple-projects-for-files-in---parameter-option)
+    - [Datasets](#datasets)
+      - [List Files](#list-files)
+      - [Move Files](#move-files)
+      - [Rename Files](#rename-files)
+      - [Copy Files](#copy-files)
+      - [Link S3 Folders to Interactive Analysis](#link-s3-folders-to-interactive-analysis)
+      - [Create Folder](#create-folder)
+      - [Remove Files or Folders](#remove-files-or-folders)
+    - [Procurement](#procurement)
+      - [List Procurement Images](#list-procurement-images)
+      - [Set Procurement Organization Image](#set-procurement-organization-image)
+      - [Reset Procurement Organization Image](#reset-procurement-organization-image)
+    - [Cromwell and WDL Pipeline Support](#cromwell-and-wdl-pipeline-support)
+      - [Manage Cromwell Server](#manage-cromwell-server)
+      - [Run WDL Workflows](#run-wdl-workflows)
+  - [Python API Usage](#python-api-usage)
+      - [Running WDL pipelines using your own scripts](#running-wdl-pipelines-using-your-own-scripts)
+  - [Unit Testing](#unit-testing)
 
 ---
 
@@ -805,36 +818,57 @@ cloudos job details --profile my_profile --job-id 62c83a1191fe06013b7ef355
 ```bash
 cloudos job details \
     --profile job-details \
-    --job-id 62c83a1191fe06013b7ef355
+    --job-id 68bf2178b4ae9f283ea8a0bf
 ```
 
 The expected output should be something similar to when using the defaults and the details are displayed in the standard output console:
 
 ```console
 Executing details...
-                                             Job Details                                              
-┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
-┃ Field                    ┃ Value                                                                   ┃
-┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
-│ Parameters               │ -test=value                                                             │
-│                          │ --gaq=test                                                              │
-│                          │ cryo=yes                                                                │
-│ Command                  │ echo 'test' > new_file.txt                                              │
-│ Revision                 │ sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 │
-│ Nextflow Version         │ None                                                                    │
-│ Execution Platform       │ Batch AWS                                                               │
-│ Profile                  │ None                                                                    │
-│ Master Instance          │ c5.xlarge                                                               │
-│ Storage                  │ 500                                                                     │
-│ Job Queue                │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-feee                        │
-│ Accelerated File Staging │ None                                                                    │
-│ Task Resources           │ 1 CPUs, 4 GB RAM                                                        │
-└──────────────────────────┴─────────────────────────────────────────────────────────────────────────┘
-```
-
-To change this behaviour and save the details into a local JSON, the parameter `--output-format` needs to be set as `--output-format=json`.
-
-By default, all details are saved in a file with the basename as `job_details`, for example `job_details.json` or `job_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
+                                                    Job Details
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+┃ Field                    ┃ Value                                                                                 ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
+│ Status                   │ completed                                                                             │
+│ Name                     │ mutSign_vcf_prep_genome_chart_non_resumable                                           │
+│ Project                  │ DB-PR-testing                                                                         │
+│ Owner                    │ Name Surname                                                                          │
+│ Pipeline                 │ MutSign                                                                               │
+│ ID                       │ 68bf2178b4ae9f283ea8a0bf                                                              │
+│ Submit time              │ 2025-09-08 18:34:26                                                                   │
+│ End time                 │ 2025-09-08 18:38:26                                                                   │
+│ Run time                 │ 4m 0s                                                                                 │
+│ Commit                   │ 11fea740366b92b2858349b764879792272f2996                                              │
+│ Cost                     │ $0.2515                                                                               │
+│ Master Instance          │ c5.xlarge                                                                             │
+│ Storage                  │ 500 GB                                                                                │
+│ Job Queue ID             │ nextflow-job-queue-5c6d3e9bd954e800b23f8c62-f7386cc5-cbdf-40e7-b379                   │
+│ Job Queue Name           │ v41                                                                                   │
+│ Task Resources           │ 1 CPUs, 4 GB RAM'                                                                 │
+│ Pipeline url             │ https://github.com/lifebit-ai/mutational-signature-nf                                 │
+│ Nextflow Version         │ 22.10.8                                                                               │
+│ Execution Platform       │ Batch AWS                                                                             │
+│ Accelerated File Staging │ False                                                                                 │
+│ Parameters               │ --snv_indel=lifebit-user-data-106c12d2-cf8f-446c-b77e-661d697c833c/deploit/teams/5c6d │
+│                          │ 3e9bd954e800b23f8c62/users/6329e3bd3c0e00014641eeea/projects/655cc29778391a7e1901a5b7 │
+│                          │ /jobs/68810fda9b301f037d38c4be/results/results/W0000664B01_W0000665B01/W0000664B01_W0 │
+│                          │ 000665B01_filtered.vcf.gz                                                             │
+│                          │ --sv=lifebit-user-data-f60fc49a-9081-417e-a4fc-03c52de4c820/deploit/teams/5c6d3e9bd95 │
+│                          │ 4e800b23f8c62/users/669a9dfe474e319e71685421/projects/66d864817afbdfa1d4515d56/jobs/6 │
+│                          │ 8763aac9e7fe38ec6e9236d/results/results/manta/W0000665B01-W0000664B01/somaticSV_annot │
+│                          │ ation.vcf.gz                                                                          │
+│                          │ --cnv=lifebit-user-data-106c12d2-cf8f-446c-b77e-661d697c833c/deploit/teams/5c6d3e9bd9 │
+│                          │ 54e800b23f8c62/users/6329e3bd3c0e00014641eeea/projects/655cc29778391a7e1901a5b7/jobs/ │
+│                          │ 687f5b383e05673d74aab1b9/results/W0000664B01_vs_W0000665B01/W0000664B01.copynumber.ca │
+│                          │ veman.csv                                                                             │
+│                          │ --sample_name=W0000664B01                                                             │
+│ Profile                  │ None                                                                                  │
+└──────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+To change this behaviour and save the details into a local CSV or JSON, the parameter `--output-format` needs to be set as `--output-format=json` or `--output-format=csv`.
+
+By default, all details are saved in a file with the basename as `{job_id}_details`, for example `68bf2178b4ae9f283ea8a0bf_details.json` or `68bf2178b4ae9f283ea8a0bf_details.config.`. This can be changed with the parameter `--output-basename=new_filename`.
 
 The `details` subcommand, can also take `--parameters` as an argument flag, which will create a new file `*.config` that holds all parameters as a Nexflow configuration file, example:
 
@@ -872,36 +906,59 @@ Working directory for job 68747bac9e7fe38ec6e022ad: az://123456789000.blob.core.
 
 #### List Jobs
 
-You can get a summary of the workspace's last 30 submitted jobs (or a selected number of last jobs using `--last-n-jobs n` parameter) in two different formats:
+You can get a summary of workspace jobs in two different formats:
 
 - CSV: this is a table with a minimum predefined set of columns by default, or all the available columns using the `--all-fields` argument.
 - JSON: all the available information from the workspace jobs, in JSON format (`--all-fields` is always enabled for this format).
 
-To get a list with the workspace's last 30 submitted jobs, in CSV format, use:
+**Job Listing Control Options**
+
+CloudOS CLI provides two ways to control the number of jobs retrieved:
+
+1. **Pagination Control (Default)**: Use `--page` and `--page-size` for precise pagination
+2. **Last N Jobs**: Use `--last-n-jobs` for retrieving the most recent jobs
+
+> [!IMPORTANT]
+> **These options are mutually exclusive**. When `--last-n-jobs` is specified, it takes precedence and `--page`/`--page-size` parameters are ignored. A warning message will be displayed if both are provided.
+
+**Default Behavior**
+
+By default, the command retrieves the first page with 10 jobs (equivalent to `--page 1 --page-size 10`):
 
 ```bash
-cloudos job list --profile my_profile--output-format csv --all-fields
+cloudos job list --profile my_profile --output-format csv --all-fields
 ```
 
 The expected output is something similar to:
 
 ```console
 Executing list...
-	Job list collected with a total of 30 jobs.
+	Job list collected with a total of 10 jobs.
 	Job list saved to joblist.csv
 ```
 
-In addition, a file named `joblist.csv` is created.
+**Pagination Examples**
 
-To get the same information, but for all the workspace's jobs and in JSON format, use the following command:
+Retrieve specific pages using `--page` and `--page-size`:
 
 ```bash
-cloudos job list \
-    --cloudos-url $CLOUDOS \
-    --apikey $MY_API_KEY \
-    --workspace-id $WORKSPACE_ID \
-    --last-n-jobs all \
-    --output-format json
+# Get page 2 with 15 jobs per page
+cloudos job list --profile my_profile --page 2 --page-size 15
+
+# Get page 5 with maximum 100 jobs per page
+cloudos job list --profile my_profile --page 5 --page-size 100
+```
+
+**Last N Jobs Examples**
+
+Use `--last-n-jobs` to get the most recent jobs:
+
+```bash
+# Get the last 50 jobs
+cloudos job list --profile my_profile --last-n-jobs 50
+
+# Get all workspace jobs
+cloudos job list --profile my_profile --last-n-jobs all --output-format json
 ```
 ```console
 Executing list...
@@ -909,6 +966,11 @@ Executing list...
 	Job list saved to joblist.json
 ```
 
+> [!NOTE]
+> - `--page-size` has a maximum limit of 100 jobs per page
+> - When filters or `--archived` flag are applied, pagination is applied to the filtered results
+> - If both `--last-n-jobs` and pagination parameters are provided, only `--last-n-jobs` will be used 
+
 You find specific jobs within your workspace using the listing filtering options. Filters can be combined to narrow down results and all filtering is performed after retrieving jobs from the server.
 
 **Available filters:**
@@ -922,21 +984,41 @@ You find specific jobs within your workspace using the listing filtering options
 - **`--filter-owner`**: Show only job for the specified owner (exact match required, i.e needs to be in quotes and be "Name Surname")
 - **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
 
-Here following are some examples:
+**Filtering Examples**
 
-Get all completed jobs from the last 50 jobs:
+You can find specific jobs within your workspace using the listing filtering options. Filters can be combined to narrow down results and all filtering is performed after retrieving jobs from the server.
+
+Using pagination approach (default):
 ```bash
+# Get completed jobs from page 1 (default 20 jobs)
+cloudos job list --profile my_profile --filter-status completed
+
+# Get completed jobs from page 2 with 15 jobs per page
+cloudos job list --profile my_profile --page 2 --page-size 15 --filter-status completed
+```
+
+Using last-n-jobs approach:
+```bash
+# Get all completed jobs from the last 50 jobs
 cloudos job list --profile my_profile --last-n-jobs 50 --filter-status completed
 ```
 
 Find jobs with "analysis" in the name from a specific project:
 ```bash
+# Using pagination (gets first 20 matching jobs)
 cloudos job list --profile my_profile --filter-job-name analysis --filter-project "My Research Project"
+
+# Using last-n-jobs
+cloudos job list --profile my_profile --last-n-jobs 100 --filter-job-name analysis --filter-project "My Research Project"
 ```
 
 Get all jobs using a specific workflow and queue:
 ```bash
-cloudos job list --profile my_profile --filter-workflow rnatoy --filter-queue high-priority-queue
+# Using pagination with larger page size
+cloudos job list --profile my_profile --page-size 50 --filter-workflow rnatoy --filter-queue high-priority-queue
+
+# Using last-n-jobs to search all jobs
+cloudos job list --profile my_profile --last-n-jobs 'all' --filter-workflow rnatoy --filter-queue high-priority-queue
 ```
 
 > [!NOTE]
@@ -944,8 +1026,6 @@ cloudos job list --profile my_profile --filter-workflow rnatoy --filter-queue hi
 > - Job name filtering is case insensitive and supports partial matches
 > - The `--last` flag can be used with `--filter-workflow` when multiple workflows have the same name
 
-
-
 ### Bash Jobs
 Execute bash scripts on CloudOS for custom processing workflows. Bash jobs allow you to run shell commands with custom parameters and are ideal for data preprocessing or simple computational tasks.