cloudos-cli 2.86.0__tar.gz → 2.87.1__tar.gz

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.86.0
+Version: 2.87.1
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -436,7 +436,7 @@ Job queues are required for running jobs using AWS batch executor. The available
 
 #### List Queues
 
-This command allows you to view available computational queues and their configurations. You can get a summary of all available workspace job queues in three different output formats using the `--output-format` option:
+This command allows you to view available computational queues and their configurations. By default, both regular workspace queues and system queues are displayed. You can get a summary of all available job queues in three different output formats using the `--output-format` option:
 
 - **stdout** (default): Displays a rich formatted table directly in the terminal with pagination and visual formatting
 - **csv**: Saves queue data to a CSV file with a selection of available queue information, or all information using the `--all-fields` flag
@@ -450,6 +450,12 @@ cloudos queue list --profile my_profile
 cloudos queue list --profile my_profile --output-format stdout
 ```
 
+To exclude system queues and show only workspace queues:
+
+```bash
+cloudos queue list --profile my_profile --exclude-system-queues
+```
+
 To save all available job queues in JSON format:
 
 ```bash
@@ -472,7 +478,7 @@ cloudos queue list --profile my_profile --output-format csv
 
 **Job queues for platform workflows**
 
-Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues. Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
+Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
 
 ### Workflow
@@ -886,7 +892,7 @@ You can find specific jobs within your workspace using the filtering options. Fi
 - **`--filter-job-id`**: Filter jobs by specific job ID (exact match required)
 - **`--filter-only-mine`**: Show only jobs belonging to the current user
 - **`--filter-owner`**: Show only jobs for the specified owner (exact match required, e.g., "John Doe")
-- **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
+- **`--filter-queue`**: Filter jobs by queue name (works with both regular and system queues; only applies to batch jobs)
 
 **Filtering Examples**
 

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/README.md

@@ -401,7 +401,7 @@ Job queues are required for running jobs using AWS batch executor. The available
 
 #### List Queues
 
-This command allows you to view available computational queues and their configurations. You can get a summary of all available workspace job queues in three different output formats using the `--output-format` option:
+This command allows you to view available computational queues and their configurations. By default, both regular workspace queues and system queues are displayed. You can get a summary of all available job queues in three different output formats using the `--output-format` option:
 
 - **stdout** (default): Displays a rich formatted table directly in the terminal with pagination and visual formatting
 - **csv**: Saves queue data to a CSV file with a selection of available queue information, or all information using the `--all-fields` flag
@@ -415,6 +415,12 @@ cloudos queue list --profile my_profile
 cloudos queue list --profile my_profile --output-format stdout
 ```
 
+To exclude system queues and show only workspace queues:
+
+```bash
+cloudos queue list --profile my_profile --exclude-system-queues
+```
+
 To save all available job queues in JSON format:
 
 ```bash
@@ -437,7 +443,7 @@ cloudos queue list --profile my_profile --output-format csv
 
 **Job queues for platform workflows**
 
-Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues. Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
+Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
 
 ### Workflow
@@ -851,7 +857,7 @@ You can find specific jobs within your workspace using the filtering options. Fi
 - **`--filter-job-id`**: Filter jobs by specific job ID (exact match required)
 - **`--filter-only-mine`**: Show only jobs belonging to the current user
 - **`--filter-owner`**: Show only jobs for the specified owner (exact match required, e.g., "John Doe")
-- **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
+- **`--filter-queue`**: Filter jobs by queue name (works with both regular and system queues; only applies to batch jobs)
 
 **Filtering Examples**
 

cloudos_cli-2.87.1/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.87.1'

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/clos.py

@@ -1239,9 +1239,7 @@ class Cloudos:
                 if target_job_count != 'all' and len(all_jobs) >= target_job_count:
                     break
             else:
-                # In direct mode (page/page_size), only get one page unless filter_queue is used
-                # When filter_queue is used, continue fetching pages until we have enough filtered results
-                if not filter_queue or len(all_jobs) >= current_page_size:
+                if not filter_queue and len(all_jobs) >= current_page_size:
                     break
 
             # Check if we reached the last page (fewer jobs than requested page size)
@@ -1257,6 +1255,18 @@ class Cloudos:
         if use_pagination_mode and target_job_count != 'all' and isinstance(target_job_count, int) and target_job_count > 0:
             all_jobs = all_jobs[:target_job_count]
 
+        # --- Adjust pagination metadata for client-side filtering ---
+        # When filter_queue is applied, we've fetched multiple API pages and filtered them.
+        # We need to return all filtered jobs so the CLI can handle pagination client-side.
+        if filter_queue and last_pagination_metadata:
+            # Mark this as client-filtered so the CLI knows to handle pagination differently
+            last_pagination_metadata = {
+                'Pagination-Count': len(all_jobs),  # Total filtered jobs collected  
+                'Pagination-Page': page,  # The initial page requested
+                'Pagination-Limit': current_page_size,  # Page size
+                '_client_filtered': True  # Flag indicating client-side pagination needed
+            }
+
         return {'jobs': all_jobs, 'pagination_metadata': last_pagination_metadata}
 
     @staticmethod
@@ -2321,8 +2331,8 @@ class Cloudos:
             # The API uses various names: 'ready' and 'aborted' but we display them as 'running' and 'stopped' to users
             status_mapping = {
                 'setup': 'setup',
-                'initialising': 'initialising',
-                'initializing': 'initialising',  # Accept both spellings
+                'initialising': 'initializing',
+                'initializing': 'initializing',  # Accept both spellings
                 'running': 'ready',  # API uses 'ready' for running sessions
                 'scheduled': 'scheduled',
                 'stopped': 'aborted',

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/interactive_session/cli.py

@@ -58,8 +58,8 @@ def interactive_session():
               required=True)
 @click.option('--filter-status',
               multiple=True,
-              type=click.Choice(['setup', 'initialising', 'running', 'scheduled', 'paused'], case_sensitive=False),
-              help='Filter sessions by status. Can be specified multiple times to filter by multiple statuses.')
+              type=click.Choice(['setup', 'initialising', 'initializing', 'running', 'scheduled', 'paused'], case_sensitive=False),
+              help='Filter sessions by status. Can be specified multiple times to filter by multiple statuses. (Supports both initialising and initializing spellings)')
 @click.option('--limit',
               type=int,
               default=10,

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/interactive_session/interactive_session.py

@@ -1296,7 +1296,8 @@ API_STATUS_MAPPING = {
 }
 
 # Pre-running statuses (watch mode only valid for these)
-PRE_RUNNING_STATUSES = {'setup', 'initialising', 'scheduled'}
+# Includes both British (initialising) and American (initializing) spellings
+PRE_RUNNING_STATUSES = {'setup', 'initialising', 'initializing', 'scheduled'}
 
 
 def format_duration(seconds: int) -> str:

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/jobs/cli.py

@@ -2,6 +2,11 @@
 
 import rich_click as click
 import cloudos_cli.jobs.job as jb
+from cloudos_cli.jobs.job import (
+    fetch_job_page,
+    create_api_pagination_callback,
+    create_client_pagination_callback
+)
 from cloudos_cli.clos import Cloudos
 from cloudos_cli.utils.errors import BadRequestException
 from cloudos_cli.utils.resources import ssl_selector
@@ -1335,7 +1340,13 @@ def list_jobs(ctx,
         ])
         if output_format == 'stdout':
             # For stdout, always show a user-friendly message
-            create_job_list_table([], cloudos_url, pagination_metadata, selected_columns)
+            # Create callback for interactive pagination using helper function
+            fetch_page = create_api_pagination_callback(
+                cl, workspace_id, page_size, archived, verify_ssl,
+                filter_status, filter_job_name, filter_project, filter_workflow,
+                filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+            )
+            create_job_list_table([], cloudos_url, pagination_metadata, selected_columns, fetch_page_callback=fetch_page)
         else:
             if filters_used:
                 print('A total of 0 jobs collected.')
@@ -1347,8 +1358,32 @@ def list_jobs(ctx,
                       'does not exist. Please, try a smaller number for --page or collect all the jobs by not ' +
                       'using --page parameter.')
     elif output_format == 'stdout':
-        # Display as table
-        create_job_list_table(my_jobs_r, cloudos_url, pagination_metadata, selected_columns)
+        # Display as table with interactive pagination
+        
+        # Check if results are client-side filtered (e.g., by queue)
+        is_client_filtered = pagination_metadata and pagination_metadata.get('_client_filtered', False)
+        
+        if is_client_filtered:
+            # For client-filtered results, we have all jobs already
+            # Create a callback that paginates them client-side using helper function
+            fetch_page = create_client_pagination_callback(my_jobs_r, page_size)
+            
+            # Show first page of filtered results
+            first_page_jobs = my_jobs_r[:page_size]
+            first_page_metadata = {
+                'Pagination-Count': len(my_jobs_r),
+                'Pagination-Page': 1,
+                'Pagination-Limit': page_size
+            }
+            create_job_list_table(first_page_jobs, cloudos_url, first_page_metadata, selected_columns, fetch_page_callback=fetch_page)
+        else:
+            # For normal (non-filtered) results, use API pagination with helper function
+            fetch_page = create_api_pagination_callback(
+                cl, workspace_id, page_size, archived, verify_ssl,
+                filter_status, filter_job_name, filter_project, filter_workflow,
+                filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+            )
+            create_job_list_table(my_jobs_r, cloudos_url, pagination_metadata, selected_columns, fetch_page_callback=fetch_page)
     elif output_format == 'csv':
         my_jobs = cl.process_job_list(my_jobs_r, all_fields)
         cl.save_job_list_to_csv(my_jobs, outfile)

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/jobs/job.py

@@ -1727,3 +1727,154 @@ class Job(Cloudos):
 
         return {"branches": all_branches, "total": total or len(all_branches)}
 
+
+def fetch_job_page(cl, workspace_id, page_num, page_size, last_n_jobs, archived, verify_ssl,
+                   filter_status, filter_job_name, filter_project, filter_workflow,
+                   filter_job_id, filter_only_mine, filter_owner, filter_queue, last):
+    """Helper function to fetch a specific page of jobs.
+    Parameters
+    ----------
+    cl : Cloudos
+        CloudOS API client instance
+    workspace_id : str
+        The CloudOS workspace ID
+    page_num : int
+        Page number to fetch (1-indexed)
+    page_size : int
+        Number of jobs per page
+    last_n_jobs : int or None
+        Last N jobs parameter (should be None for pagination mode)
+    archived : bool
+        Whether to include archived jobs
+    verify_ssl : bool or str
+        SSL verification setting
+    filter_status : str or None
+        Status filter
+    filter_job_name : str or None
+        Job name filter
+    filter_project : str or None
+        Project filter
+    filter_workflow : str or None
+        Workflow filter
+    filter_job_id : str or None
+        Job ID filter
+    filter_only_mine : bool
+        Filter for user's own jobs
+    filter_owner : str or None
+        Owner filter
+    filter_queue : str or None
+        Queue filter
+    last : bool
+        Use latest workflow for duplicates
+    Returns
+    -------
+    dict
+        Dictionary with 'jobs' list and 'pagination_metadata' dict
+    """
+    result = cl.get_job_list(
+        workspace_id,
+        last_n_jobs=last_n_jobs,
+        page=page_num,
+        page_size=page_size,
+        archived=archived,
+        verify=verify_ssl,
+        filter_status=filter_status,
+        filter_job_name=filter_job_name,
+        filter_project=filter_project,
+        filter_workflow=filter_workflow,
+        filter_job_id=filter_job_id,
+        filter_only_mine=filter_only_mine,
+        filter_owner=filter_owner,
+        filter_queue=filter_queue,
+        last=last
+    )
+    return result
+
+
+def create_api_pagination_callback(cl, workspace_id, page_size, archived, verify_ssl,
+                                    filter_status, filter_job_name, filter_project, filter_workflow,
+                                    filter_job_id, filter_only_mine, filter_owner, filter_queue, last):
+    """Create a pagination callback that fetches pages from the API.
+    
+    Parameters
+    ----------
+    cl : Cloudos
+        CloudOS API client instance
+    workspace_id : str
+        The CloudOS workspace ID
+    page_size : int
+        Number of jobs per page
+    archived : bool
+        Whether to include archived jobs
+    verify_ssl : bool or str
+        SSL verification setting
+    filter_status : str or None
+        Status filter
+    filter_job_name : str or None
+        Job name filter
+    filter_project : str or None
+        Project filter
+    filter_workflow : str or None
+        Workflow filter
+    filter_job_id : str or None
+        Job ID filter
+    filter_only_mine : bool
+        Filter for user's own jobs
+    filter_owner : str or None
+        Owner filter
+    filter_queue : str or None
+        Queue filter
+    last : bool
+        Use latest workflow for duplicates
+        
+    Returns
+    -------
+    callable
+        Callback function that takes page_num and returns job page data
+    """
+    def api_fetch_callback(page_num):
+        return fetch_job_page(
+            cl, workspace_id, page_num, page_size, None, archived, verify_ssl,
+            filter_status, filter_job_name, filter_project, filter_workflow,
+            filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+        )
+    return api_fetch_callback
+
+
+def create_client_pagination_callback(all_jobs, page_size):
+    """Create a pagination callback that paginates already-fetched jobs client-side.
+    
+    Used when jobs have been pre-filtered client-side (e.g., by queue) and we
+    want to paginate through the cached results.
+    
+    Parameters
+    ----------
+    all_jobs : list
+        List of all jobs to paginate through
+    page_size : int
+        Number of jobs per page
+        
+    Returns
+    -------
+    callable
+        Callback function that takes page_num and returns job page data
+    """
+    def client_fetch_callback(page_num):
+        """Paginate client-side filtered results"""
+        start_idx = (page_num - 1) * page_size
+        end_idx = start_idx + page_size
+        page_jobs = all_jobs[start_idx:end_idx]
+        
+        # Return with updated pagination metadata
+        return {
+            'jobs': page_jobs,
+            'pagination_metadata': {
+                'Pagination-Count': len(all_jobs),
+                'Pagination-Page': page_num,
+                'Pagination-Limit': page_size,
+                '_client_filtered': True
+            }
+        }
+    return client_fetch_callback
+
+

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/queue/cli.py

@@ -47,6 +47,9 @@ def queue():
                     'just the preconfigured selected fields. Only applicable ' +
                     'when --output-format=csv'),
               is_flag=True)
+@click.option('--exclude-system-queues',
+              help='Exclude system job queues from the list.',
+              is_flag=True)
 @click.option('--disable-ssl-verification',
               help=('Disable SSL certificate verification. Please, remember that this option is ' +
                     'not generally recommended for security reasons.'),
@@ -63,6 +66,7 @@ def list_queues(ctx,
                 output_basename,
                 output_format,
                 all_fields,
+                exclude_system_queues,
                 disable_ssl_verification,
                 ssl_cert,
                 profile):
@@ -72,7 +76,7 @@ def list_queues(ctx,
     verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
     print('Executing list...')
     j_queue = Queue(cloudos_url, apikey, None, workspace_id, verify=verify_ssl)
-    my_queues = j_queue.get_job_queues()
+    my_queues = j_queue.get_job_queues(exclude_system_queues=exclude_system_queues)
     if len(my_queues) == 0:
         raise ValueError('No AWS batch queues found. Please, make sure that your CloudOS supports AWS batch queues')
     if output_format == 'stdout':

{cloudos_cli-2.86.0 → cloudos_cli-2.87.1}/cloudos_cli/queue/queue.py

@@ -33,9 +33,14 @@ class Queue(Cloudos):
     workspace_id: str
     verify: Union[bool, str] = True
 
-    def get_job_queues(self):
+    def get_job_queues(self, exclude_system_queues=False):
         """Get all the job queues from a CloudOS workspace.
 
+        Parameters
+        ----------
+        exclude_system_queues : bool, default=False
+            Whether to exclude system job queues from the result.
+
         Returns
         -------
         r : list
@@ -47,6 +52,28 @@ class Queue(Cloudos):
                          headers=headers, verify=self.verify)
         if r.status_code >= 400:
             raise BadRequestException(r)
+        queues = json.loads(r.content)
+        # By default, include system queues unless excluded
+        if not exclude_system_queues:
+            system_queues = self.get_system_job_queues()
+            queues.extend(system_queues)
+        
+        return queues
+
+    def get_system_job_queues(self):
+        """Get all the system job queues from CloudOS.
+
+        Returns
+        -------
+        r : list
+            A list of dicts, each corresponding to a system job queue.
+        """
+        headers = {"apikey": self.apikey}
+        r = requests.get("{}/api/v1/teams/aws/v2/system-job-queues?teamId={}".format(self.cloudos_url,
+                                                                                       self.workspace_id),
+                         headers=headers, verify=self.verify)
+        if r.status_code >= 400:
+            raise BadRequestException(r)
         return json.loads(r.content)
 
     @staticmethod
@@ -118,7 +145,7 @@ class Queue(Cloudos):
         if len(available_queues) == 0:
             raise Exception(f'There are no available job queues for {workflow_type} ' +
                             'workflows. Consider creating one using CloudOS UI.')
-        default_queue = [q for q in available_queues if q['isDefault']]
+        default_queue = [q for q in available_queues if q.get('isDefault', False)]
         if len(default_queue) > 0:
             default_queue_id = default_queue[0]['id']
             default_queue_name = default_queue[0]['label']