cloudos-cli 2.86.0__tar.gz → 2.87.0__tar.gz

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.86.0
+Version: 2.87.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -436,7 +436,7 @@ Job queues are required for running jobs using AWS batch executor. The available
 
 #### List Queues
 
-This command allows you to view available computational queues and their configurations. You can get a summary of all available workspace job queues in three different output formats using the `--output-format` option:
+This command allows you to view available computational queues and their configurations. By default, both regular workspace queues and system queues are displayed. You can get a summary of all available job queues in three different output formats using the `--output-format` option:
 
 - **stdout** (default): Displays a rich formatted table directly in the terminal with pagination and visual formatting
 - **csv**: Saves queue data to a CSV file with a selection of available queue information, or all information using the `--all-fields` flag
@@ -450,6 +450,12 @@ cloudos queue list --profile my_profile
 cloudos queue list --profile my_profile --output-format stdout
 ```
 
+To exclude system queues and show only workspace queues:
+
+```bash
+cloudos queue list --profile my_profile --exclude-system-queues
+```
+
 To save all available job queues in JSON format:
 
 ```bash
@@ -472,7 +478,7 @@ cloudos queue list --profile my_profile --output-format csv
 
 **Job queues for platform workflows**
 
-Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues. Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
+Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
 
 ### Workflow
@@ -886,7 +892,7 @@ You can find specific jobs within your workspace using the filtering options. Fi
 - **`--filter-job-id`**: Filter jobs by specific job ID (exact match required)
 - **`--filter-only-mine`**: Show only jobs belonging to the current user
 - **`--filter-owner`**: Show only jobs for the specified owner (exact match required, e.g., "John Doe")
-- **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
+- **`--filter-queue`**: Filter jobs by queue name (works with both regular and system queues; only applies to batch jobs)
 
 **Filtering Examples**
 

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/README.md

@@ -401,7 +401,7 @@ Job queues are required for running jobs using AWS batch executor. The available
 
 #### List Queues
 
-This command allows you to view available computational queues and their configurations. You can get a summary of all available workspace job queues in three different output formats using the `--output-format` option:
+This command allows you to view available computational queues and their configurations. By default, both regular workspace queues and system queues are displayed. You can get a summary of all available job queues in three different output formats using the `--output-format` option:
 
 - **stdout** (default): Displays a rich formatted table directly in the terminal with pagination and visual formatting
 - **csv**: Saves queue data to a CSV file with a selection of available queue information, or all information using the `--all-fields` flag
@@ -415,6 +415,12 @@ cloudos queue list --profile my_profile
 cloudos queue list --profile my_profile --output-format stdout
 ```
 
+To exclude system queues and show only workspace queues:
+
+```bash
+cloudos queue list --profile my_profile --exclude-system-queues
+```
+
 To save all available job queues in JSON format:
 
 ```bash
@@ -437,7 +443,7 @@ cloudos queue list --profile my_profile --output-format csv
 
 **Job queues for platform workflows**
 
-Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues. Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
+Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
 
 ### Workflow
@@ -851,7 +857,7 @@ You can find specific jobs within your workspace using the filtering options. Fi
 - **`--filter-job-id`**: Filter jobs by specific job ID (exact match required)
 - **`--filter-only-mine`**: Show only jobs belonging to the current user
 - **`--filter-owner`**: Show only jobs for the specified owner (exact match required, e.g., "John Doe")
-- **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
+- **`--filter-queue`**: Filter jobs by queue name (works with both regular and system queues; only applies to batch jobs)
 
 **Filtering Examples**
 

cloudos_cli-2.87.0/cloudos_cli/_version.py

@@ -0,0 +1 @@
+__version__ = '2.87.0'

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/clos.py

@@ -1239,9 +1239,7 @@ class Cloudos:
                 if target_job_count != 'all' and len(all_jobs) >= target_job_count:
                     break
             else:
-                # In direct mode (page/page_size), only get one page unless filter_queue is used
-                # When filter_queue is used, continue fetching pages until we have enough filtered results
-                if not filter_queue or len(all_jobs) >= current_page_size:
+                if not filter_queue and len(all_jobs) >= current_page_size:
                     break
 
             # Check if we reached the last page (fewer jobs than requested page size)
@@ -1257,6 +1255,18 @@ class Cloudos:
         if use_pagination_mode and target_job_count != 'all' and isinstance(target_job_count, int) and target_job_count > 0:
             all_jobs = all_jobs[:target_job_count]
 
+        # --- Adjust pagination metadata for client-side filtering ---
+        # When filter_queue is applied, we've fetched multiple API pages and filtered them.
+        # We need to return all filtered jobs so the CLI can handle pagination client-side.
+        if filter_queue and last_pagination_metadata:
+            # Mark this as client-filtered so the CLI knows to handle pagination differently
+            last_pagination_metadata = {
+                'Pagination-Count': len(all_jobs),  # Total filtered jobs collected  
+                'Pagination-Page': page,  # The initial page requested
+                'Pagination-Limit': current_page_size,  # Page size
+                '_client_filtered': True  # Flag indicating client-side pagination needed
+            }
+
         return {'jobs': all_jobs, 'pagination_metadata': last_pagination_metadata}
 
     @staticmethod

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/jobs/cli.py

@@ -2,6 +2,11 @@
 
 import rich_click as click
 import cloudos_cli.jobs.job as jb
+from cloudos_cli.jobs.job import (
+    fetch_job_page,
+    create_api_pagination_callback,
+    create_client_pagination_callback
+)
 from cloudos_cli.clos import Cloudos
 from cloudos_cli.utils.errors import BadRequestException
 from cloudos_cli.utils.resources import ssl_selector
@@ -1335,7 +1340,13 @@ def list_jobs(ctx,
         ])
         if output_format == 'stdout':
             # For stdout, always show a user-friendly message
-            create_job_list_table([], cloudos_url, pagination_metadata, selected_columns)
+            # Create callback for interactive pagination using helper function
+            fetch_page = create_api_pagination_callback(
+                cl, workspace_id, page_size, archived, verify_ssl,
+                filter_status, filter_job_name, filter_project, filter_workflow,
+                filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+            )
+            create_job_list_table([], cloudos_url, pagination_metadata, selected_columns, fetch_page_callback=fetch_page)
         else:
             if filters_used:
                 print('A total of 0 jobs collected.')
@@ -1347,8 +1358,32 @@ def list_jobs(ctx,
                       'does not exist. Please, try a smaller number for --page or collect all the jobs by not ' +
                       'using --page parameter.')
     elif output_format == 'stdout':
-        # Display as table
-        create_job_list_table(my_jobs_r, cloudos_url, pagination_metadata, selected_columns)
+        # Display as table with interactive pagination
+        
+        # Check if results are client-side filtered (e.g., by queue)
+        is_client_filtered = pagination_metadata and pagination_metadata.get('_client_filtered', False)
+        
+        if is_client_filtered:
+            # For client-filtered results, we have all jobs already
+            # Create a callback that paginates them client-side using helper function
+            fetch_page = create_client_pagination_callback(my_jobs_r, page_size)
+            
+            # Show first page of filtered results
+            first_page_jobs = my_jobs_r[:page_size]
+            first_page_metadata = {
+                'Pagination-Count': len(my_jobs_r),
+                'Pagination-Page': 1,
+                'Pagination-Limit': page_size
+            }
+            create_job_list_table(first_page_jobs, cloudos_url, first_page_metadata, selected_columns, fetch_page_callback=fetch_page)
+        else:
+            # For normal (non-filtered) results, use API pagination with helper function
+            fetch_page = create_api_pagination_callback(
+                cl, workspace_id, page_size, archived, verify_ssl,
+                filter_status, filter_job_name, filter_project, filter_workflow,
+                filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+            )
+            create_job_list_table(my_jobs_r, cloudos_url, pagination_metadata, selected_columns, fetch_page_callback=fetch_page)
     elif output_format == 'csv':
         my_jobs = cl.process_job_list(my_jobs_r, all_fields)
         cl.save_job_list_to_csv(my_jobs, outfile)

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/jobs/job.py

@@ -1727,3 +1727,154 @@ class Job(Cloudos):
 
         return {"branches": all_branches, "total": total or len(all_branches)}
 
+
+def fetch_job_page(cl, workspace_id, page_num, page_size, last_n_jobs, archived, verify_ssl,
+                   filter_status, filter_job_name, filter_project, filter_workflow,
+                   filter_job_id, filter_only_mine, filter_owner, filter_queue, last):
+    """Helper function to fetch a specific page of jobs.
+    Parameters
+    ----------
+    cl : Cloudos
+        CloudOS API client instance
+    workspace_id : str
+        The CloudOS workspace ID
+    page_num : int
+        Page number to fetch (1-indexed)
+    page_size : int
+        Number of jobs per page
+    last_n_jobs : int or None
+        Last N jobs parameter (should be None for pagination mode)
+    archived : bool
+        Whether to include archived jobs
+    verify_ssl : bool or str
+        SSL verification setting
+    filter_status : str or None
+        Status filter
+    filter_job_name : str or None
+        Job name filter
+    filter_project : str or None
+        Project filter
+    filter_workflow : str or None
+        Workflow filter
+    filter_job_id : str or None
+        Job ID filter
+    filter_only_mine : bool
+        Filter for user's own jobs
+    filter_owner : str or None
+        Owner filter
+    filter_queue : str or None
+        Queue filter
+    last : bool
+        Use latest workflow for duplicates
+    Returns
+    -------
+    dict
+        Dictionary with 'jobs' list and 'pagination_metadata' dict
+    """
+    result = cl.get_job_list(
+        workspace_id,
+        last_n_jobs=last_n_jobs,
+        page=page_num,
+        page_size=page_size,
+        archived=archived,
+        verify=verify_ssl,
+        filter_status=filter_status,
+        filter_job_name=filter_job_name,
+        filter_project=filter_project,
+        filter_workflow=filter_workflow,
+        filter_job_id=filter_job_id,
+        filter_only_mine=filter_only_mine,
+        filter_owner=filter_owner,
+        filter_queue=filter_queue,
+        last=last
+    )
+    return result
+
+
+def create_api_pagination_callback(cl, workspace_id, page_size, archived, verify_ssl,
+                                    filter_status, filter_job_name, filter_project, filter_workflow,
+                                    filter_job_id, filter_only_mine, filter_owner, filter_queue, last):
+    """Create a pagination callback that fetches pages from the API.
+    
+    Parameters
+    ----------
+    cl : Cloudos
+        CloudOS API client instance
+    workspace_id : str
+        The CloudOS workspace ID
+    page_size : int
+        Number of jobs per page
+    archived : bool
+        Whether to include archived jobs
+    verify_ssl : bool or str
+        SSL verification setting
+    filter_status : str or None
+        Status filter
+    filter_job_name : str or None
+        Job name filter
+    filter_project : str or None
+        Project filter
+    filter_workflow : str or None
+        Workflow filter
+    filter_job_id : str or None
+        Job ID filter
+    filter_only_mine : bool
+        Filter for user's own jobs
+    filter_owner : str or None
+        Owner filter
+    filter_queue : str or None
+        Queue filter
+    last : bool
+        Use latest workflow for duplicates
+        
+    Returns
+    -------
+    callable
+        Callback function that takes page_num and returns job page data
+    """
+    def api_fetch_callback(page_num):
+        return fetch_job_page(
+            cl, workspace_id, page_num, page_size, None, archived, verify_ssl,
+            filter_status, filter_job_name, filter_project, filter_workflow,
+            filter_job_id, filter_only_mine, filter_owner, filter_queue, last
+        )
+    return api_fetch_callback
+
+
+def create_client_pagination_callback(all_jobs, page_size):
+    """Create a pagination callback that paginates already-fetched jobs client-side.
+    
+    Used when jobs have been pre-filtered client-side (e.g., by queue) and we
+    want to paginate through the cached results.
+    
+    Parameters
+    ----------
+    all_jobs : list
+        List of all jobs to paginate through
+    page_size : int
+        Number of jobs per page
+        
+    Returns
+    -------
+    callable
+        Callback function that takes page_num and returns job page data
+    """
+    def client_fetch_callback(page_num):
+        """Paginate client-side filtered results"""
+        start_idx = (page_num - 1) * page_size
+        end_idx = start_idx + page_size
+        page_jobs = all_jobs[start_idx:end_idx]
+        
+        # Return with updated pagination metadata
+        return {
+            'jobs': page_jobs,
+            'pagination_metadata': {
+                'Pagination-Count': len(all_jobs),
+                'Pagination-Page': page_num,
+                'Pagination-Limit': page_size,
+                '_client_filtered': True
+            }
+        }
+    return client_fetch_callback
+
+

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/queue/cli.py

@@ -47,6 +47,9 @@ def queue():
                     'just the preconfigured selected fields. Only applicable ' +
                     'when --output-format=csv'),
               is_flag=True)
+@click.option('--exclude-system-queues',
+              help='Exclude system job queues from the list.',
+              is_flag=True)
 @click.option('--disable-ssl-verification',
               help=('Disable SSL certificate verification. Please, remember that this option is ' +
                     'not generally recommended for security reasons.'),
@@ -63,6 +66,7 @@ def list_queues(ctx,
                 output_basename,
                 output_format,
                 all_fields,
+                exclude_system_queues,
                 disable_ssl_verification,
                 ssl_cert,
                 profile):
@@ -72,7 +76,7 @@ def list_queues(ctx,
     verify_ssl = ssl_selector(disable_ssl_verification, ssl_cert)
     print('Executing list...')
     j_queue = Queue(cloudos_url, apikey, None, workspace_id, verify=verify_ssl)
-    my_queues = j_queue.get_job_queues()
+    my_queues = j_queue.get_job_queues(exclude_system_queues=exclude_system_queues)
     if len(my_queues) == 0:
         raise ValueError('No AWS batch queues found. Please, make sure that your CloudOS supports AWS batch queues')
     if output_format == 'stdout':

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/queue/queue.py

@@ -33,9 +33,14 @@ class Queue(Cloudos):
     workspace_id: str
     verify: Union[bool, str] = True
 
-    def get_job_queues(self):
+    def get_job_queues(self, exclude_system_queues=False):
         """Get all the job queues from a CloudOS workspace.
 
+        Parameters
+        ----------
+        exclude_system_queues : bool, default=False
+            Whether to exclude system job queues from the result.
+
         Returns
         -------
         r : list
@@ -47,6 +52,28 @@ class Queue(Cloudos):
                          headers=headers, verify=self.verify)
         if r.status_code >= 400:
             raise BadRequestException(r)
+        queues = json.loads(r.content)
+        # By default, include system queues unless excluded
+        if not exclude_system_queues:
+            system_queues = self.get_system_job_queues()
+            queues.extend(system_queues)
+        
+        return queues
+
+    def get_system_job_queues(self):
+        """Get all the system job queues from CloudOS.
+
+        Returns
+        -------
+        r : list
+            A list of dicts, each corresponding to a system job queue.
+        """
+        headers = {"apikey": self.apikey}
+        r = requests.get("{}/api/v1/teams/aws/v2/system-job-queues?teamId={}".format(self.cloudos_url,
+                                                                                       self.workspace_id),
+                         headers=headers, verify=self.verify)
+        if r.status_code >= 400:
+            raise BadRequestException(r)
         return json.loads(r.content)
 
     @staticmethod
@@ -118,7 +145,7 @@ class Queue(Cloudos):
         if len(available_queues) == 0:
             raise Exception(f'There are no available job queues for {workflow_type} ' +
                             'workflows. Consider creating one using CloudOS UI.')
-        default_queue = [q for q in available_queues if q['isDefault']]
+        default_queue = [q for q in available_queues if q.get('isDefault', False)]
         if len(default_queue) > 0:
             default_queue_id = default_queue[0]['id']
             default_queue_name = default_queue[0]['label']

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli/utils/details.py

@@ -337,9 +337,217 @@ def create_job_details(j_details_h, job_id, output_format, output_basename, para
         print(f"\tJob details have been saved to '{output_basename}.csv'")
 
 
-def create_job_list_table(jobs, cloudos_url, pagination_metadata=None, selected_columns=None):
+def _build_job_row_values(job, cloudos_url, terminal_width, columns_to_show):
+    """Helper function to build row values for a single job.
+    
+    Parameters
+    ----------
+    job : dict
+        Job dictionary from CloudOS API
+    cloudos_url : str
+        CloudOS service URL for generating job links
+    terminal_width : int
+        Current terminal width for responsive formatting
+    columns_to_show : list
+        List of column keys to include
+        
+    Returns
+    -------
+    list
+        Row values in the order of columns_to_show
+    """
+    # Status with colored and bold ANSI symbols
+    status_raw = str(job.get("status", "N/A"))
+    status_symbol_map = {
+        "completed": "[bold green]✓[/bold green]",
+        "running": "[bold bright_black]◐[/bold bright_black]",
+        "failed": "[bold red]✗[/bold red]",
+        "aborted": "[bold orange3]■[/bold orange3]",
+        "initialising": "[bold bright_black]○[/bold bright_black]",
+        "N/A": "[bold bright_black]?[/bold bright_black]"
+    }
+    status = status_symbol_map.get(status_raw.lower(), status_raw)
+
+    # Name
+    name = str(job.get("name", "N/A"))
+
+    # Project
+    project = str(job.get("project", {}).get("name", "N/A"))
+
+    # Owner (compact format for small terminals)
+    user_info = job.get("user", {})
+    name_part = user_info.get('name', '')
+    surname_part = user_info.get('surname', '')
+    if terminal_width < 90:
+        if name_part and surname_part:
+            owner = f"{name_part[0]}.{surname_part[0]}."
+        elif name_part or surname_part:
+            owner = (name_part or surname_part)[:8]
+        else:
+            owner = "N/A"
+    else:
+        if name_part and surname_part:
+            owner = f"{name_part}\n{surname_part}"
+        elif name_part or surname_part:
+            owner = name_part or surname_part
+        else:
+            owner = "N/A"
+
+    # Pipeline
+    pipeline = str(job.get("workflow", {}).get("name", "N/A")).split('\n')[0].strip()
+    if len(pipeline) > 25:
+        pipeline = pipeline[:22] + "..."
+
+    # ID with hyperlink
+    job_id = str(job.get("_id", "N/A"))
+    job_url = f"{cloudos_url}/app/advanced-analytics/analyses/{job_id}"
+    job_id_with_link = f"[link={job_url}]{job_id}[/link]"
+
+    # Submit time (compact format for small terminals)
+    created_at = job.get("createdAt")
+    if created_at:
+        try:
+            dt = datetime.fromisoformat(created_at.replace('Z', '+00:00'))
+            submit_time = dt.strftime('%m-%d\n%H:%M') if terminal_width < 90 else dt.strftime('%Y-%m-%d\n%H:%M:%S')
+        except (ValueError, TypeError):
+            submit_time = "N/A"
+    else:
+        submit_time = "N/A"
+
+    # End time (compact format for small terminals)
+    end_time_raw = job.get("endTime")
+    if end_time_raw:
+        try:
+            dt = datetime.fromisoformat(end_time_raw.replace('Z', '+00:00'))
+            end_time = dt.strftime('%m-%d\n%H:%M') if terminal_width < 90 else dt.strftime('%Y-%m-%d\n%H:%M:%S')
+        except (ValueError, TypeError):
+            end_time = "N/A"
+    else:
+        end_time = "N/A"
+
+    # Run time (calculate from startTime and endTime)
+    start_time_raw = job.get("startTime")
+    if start_time_raw and end_time_raw:
+        try:
+            start_dt = datetime.fromisoformat(start_time_raw.replace('Z', '+00:00'))
+            end_dt = datetime.fromisoformat(end_time_raw.replace('Z', '+00:00'))
+            duration = end_dt - start_dt
+            total_seconds = int(duration.total_seconds())
+            hours = total_seconds // 3600
+            minutes = (total_seconds % 3600) // 60
+            seconds = total_seconds % 60
+            if hours > 0:
+                run_time = f"{hours}h {minutes}m {seconds}s"
+            elif minutes > 0:
+                run_time = f"{minutes}m {seconds}s"
+            else:
+                run_time = f"{seconds}s"
+        except (ValueError, TypeError):
+            run_time = "N/A"
+    else:
+        run_time = "N/A"
+
+    # Commit
+    revision = job.get("revision", {})
+    if job.get("jobType") == "dockerAWS":
+        commit = str(revision.get("digest", "N/A"))
+    else:
+        commit = str(revision.get("commit", "N/A"))
+    if commit != "N/A" and len(commit) > 7:
+        commit = commit[:7]
+
+    # Cost
+    cost_raw = job.get("computeCostSpent") or job.get("realInstancesExecutionCost")
+    if cost_raw is not None:
+        try:
+            cost = f"${float(cost_raw) / 100:.4f}"
+        except (ValueError, TypeError):
+            cost = "N/A"
+    else:
+        cost = "N/A"
+
+    # Resources (instance type only)
+    master_instance = job.get("masterInstance", {})
+    used_instance = master_instance.get("usedInstance", {})
+    instance_type = used_instance.get("type", "N/A")
+    resources = instance_type if instance_type else "N/A"
+
+    # Storage type
+    storage_mode = job.get("storageMode", "N/A")
+    if storage_mode == "regular":
+        storage_type = "Regular"
+    elif storage_mode == "lustre":
+        storage_type = "Lustre"
+    else:
+        storage_type = str(storage_mode).capitalize() if storage_mode != "N/A" else "N/A"
+
+    # Map column keys to their values
+    column_values = {
+        'status': status,
+        'name': name,
+        'project': project,
+        'owner': owner,
+        'pipeline': pipeline,
+        'id': job_id_with_link,
+        'submit_time': submit_time,
+        'end_time': end_time,
+        'run_time': run_time,
+        'commit': commit,
+        'cost': cost,
+        'resources': resources,
+        'storage_type': storage_type
+    }
+
+    # Return row values in the order of columns_to_show
+    return [column_values[col] for col in columns_to_show]
+
+
+def _build_job_table(jobs, cloudos_url, terminal_width, columns_to_show, all_columns):
+    """Helper function to build a complete job table.
+    
+    Parameters
+    ----------
+    jobs : list
+        List of job dictionaries from CloudOS API
+    cloudos_url : str
+        CloudOS service URL for generating job links
+    terminal_width : int
+        Current terminal width for responsive formatting
+    columns_to_show : list
+        List of column keys to include
+    all_columns : dict
+        Dictionary of all column configurations
+        
+    Returns
+    -------
+    Table
+        Rich Table object populated with job rows
     """
-    Creates a formatted job list table for stdout output with responsive design.
+    table = Table(title="Job List")
+
+    # Add columns to table
+    for col_key in columns_to_show:
+        col_config = all_columns[col_key]
+        table.add_column(
+            col_config["header"],
+            style=col_config.get("style"),
+            no_wrap=col_config.get("no_wrap", False),
+            overflow=col_config.get("overflow"),
+            min_width=col_config.get("min_width"),
+            max_width=col_config.get("max_width")
+        )
+
+    # Add rows for each job
+    for job in jobs:
+        row_values = _build_job_row_values(job, cloudos_url, terminal_width, columns_to_show)
+        table.add_row(*row_values)
+
+    return table
+
+
+def create_job_list_table(jobs, cloudos_url, pagination_metadata=None, selected_columns=None, fetch_page_callback=None):
+    """
+    Creates a formatted job list table for stdout output with responsive design and interactive pagination.
 
     The table automatically adapts to terminal width by showing different column sets:
     - Very narrow (<60 chars): Essential columns only (status, name, pipeline, id)
@@ -373,6 +581,10 @@ def create_job_list_table(jobs, cloudos_url, pagination_metadata=None, selected_
         - List: List of column names
         Valid columns: 'status', 'name', 'project', 'owner', 'pipeline', 'id',
         'submit_time', 'end_time', 'run_time', 'commit', 'cost', 'resources', 'storage_type'
+    fetch_page_callback : callable, optional
+        Callback function to fetch a specific page of results for interactive pagination.
+        Should accept page number (1-indexed) and return dict with 'jobs' and 'pagination_metadata' keys.
+        If provided, enables interactive navigation (n=next, p=previous, q=quit).
 
     Returns
     -------
@@ -446,207 +658,109 @@ def create_job_list_table(jobs, cloudos_url, pagination_metadata=None, selected_
 
     if not jobs:
         console.print("\n[yellow]No jobs found matching the criteria.[/yellow]")
-        # Still show pagination info even when no jobs
+        return
+
+    # Create table using helper function
+    table = _build_job_table(jobs, cloudos_url, terminal_width, columns_to_show, all_columns)
+
+    # If no fetch_page_callback, display static table
+    if not fetch_page_callback or not pagination_metadata:
+        console.print(table)
+        
+        # Display pagination info at the bottom
         if pagination_metadata:
             total_jobs = pagination_metadata.get('Pagination-Count', 0)
             current_page = pagination_metadata.get('Pagination-Page', 1)
             page_size = pagination_metadata.get('Pagination-Limit', 10)
             total_pages = (total_jobs + page_size - 1) // page_size if total_jobs > 0 else 1
 
-            console.print(f"\n[cyan]Total jobs matching filter:[/cyan] {total_jobs}")
-            console.print(f"[cyan]Page:[/cyan] {current_page} of {total_pages}")
-            console.print(f"[cyan]Jobs on this page:[/cyan] {len(jobs)}")
+            console.print(f"\n[cyan]Showing {len(jobs)} of {total_jobs} total jobs | Page {current_page} of {total_pages}[/cyan]")
         return
 
-    # Create table
-    table = Table(title="Job List")
-
-    # Add columns to table
-    for col_key in columns_to_show:
-        col_config = all_columns[col_key]
-        table.add_column(
-            col_config["header"],
-            style=col_config.get("style"),
-            no_wrap=col_config.get("no_wrap", False),
-            overflow=col_config.get("overflow"),
-            min_width=col_config.get("min_width"),
-            max_width=col_config.get("max_width")
-        )
-
-    # Process each job
-    for job in jobs:
-        # Status with colored and bold ANSI symbols
-        status_raw = str(job.get("status", "N/A"))
-        status_symbol_map = {
-            "completed": "[bold green]✓[/bold green]",      # Green check mark
-            "running": "[bold bright_black]◐[/bold bright_black]",       # Grey half-filled circle
-            "failed": "[bold red]✗[/bold red]",             # Red X mark
-            "aborted": "[bold orange3]■[/bold orange3]",    # Orange square
-            "initialising": "[bold bright_black]○[/bold bright_black]",  # Grey circle
-            "N/A": "[bold bright_black]?[/bold bright_black]"            # Grey question mark
-        }
-        status = status_symbol_map.get(status_raw.lower(), status_raw)
-
-        # Name
-        name = str(job.get("name", "N/A"))
-
-        # Project
-        project = str(job.get("project", {}).get("name", "N/A"))
-
-        # Owner (compact format for small terminals)
-        user_info = job.get("user", {})
-        name_part = user_info.get('name', '')
-        surname_part = user_info.get('surname', '')
-        if terminal_width < 90:
-            # Compact format: just first name or first letter of each
-            if name_part and surname_part:
-                owner = f"{name_part[0]}.{surname_part[0]}."
-            elif name_part or surname_part:
-                owner = (name_part or surname_part)[:8]
-            else:
-                owner = "N/A"
-        else:
-            # Full format for wider terminals
-            if name_part and surname_part:
-                owner = f"{name_part}\n{surname_part}"
-            elif name_part or surname_part:
-                owner = name_part or surname_part
-            else:
-                owner = "N/A"
-
-        # Pipeline
-        pipeline = str(job.get("workflow", {}).get("name", "N/A"))
-        # Only show the first line if pipeline name contains newlines
-        pipeline = pipeline.split('\n')[0].strip()
-        # Truncate to 25 chars with ellipsis if longer
-        if len(pipeline) > 25:
-            pipeline = pipeline[:22] + "..."
-
-        # ID with hyperlink
-        job_id = str(job.get("_id", "N/A"))
-        job_url = f"{cloudos_url}/app/advanced-analytics/analyses/{job_id}"
-        job_id_with_link = f"[link={job_url}]{job_id}[/link]"
-
-        # Submit time (compact format for small terminals)
-        created_at = job.get("createdAt")
-        if created_at:
-            try:
-                dt = datetime.fromisoformat(created_at.replace('Z', '+00:00'))
-                if terminal_width < 90:
-                    # Compact format: MM-DD HH:MM
-                    submit_time = dt.strftime('%m-%d\n%H:%M')
-                else:
-                    # Full format
-                    submit_time = dt.strftime('%Y-%m-%d\n%H:%M:%S')
-            except (ValueError, TypeError):
-                submit_time = "N/A"
-        else:
-            submit_time = "N/A"
-
-        # End time (compact format for small terminals)
-        end_time_raw = job.get("endTime")
-        if end_time_raw:
+    # Interactive pagination mode
+    current_page = pagination_metadata.get('Pagination-Page', 1) or 1  # Ensure never None
+    total_jobs = pagination_metadata.get('Pagination-Count', 0)
+    page_size_value = pagination_metadata.get('Pagination-Limit', 10)
+    total_pages = (total_jobs + page_size_value - 1) // page_size_value if total_jobs > 0 else 1
+    
+    show_error = None
+    
+    while True:
+        # Clear console and display table
+        console.clear()
+        console.print(table)
+        
+        # Display pagination info
+        console.print(f"\n[cyan]Total jobs:[/cyan] {total_jobs}")
+        if total_pages > 1:
+            console.print(f"[cyan]Page:[/cyan] {current_page} of {total_pages}")
+            console.print(f"[cyan]Jobs on this page:[/cyan] {len(jobs)}")
+        
+        # Show error message if any
+        if show_error:
+            console.print(show_error)
+            show_error = None
+        
+        # Show pagination controls only if there are multiple pages
+        if total_pages > 1:
+            # Check if we're in an interactive environment
+            if not sys.stdin.isatty():
+                console.print("\n[yellow]Note: Pagination not available in non-interactive mode. Showing page 1 of {0}.[/yellow]".format(total_pages))
+                console.print("[yellow]Run in an interactive terminal to navigate through all pages.[/yellow]")
+                break
+            
+            console.print(f"\n[bold cyan]n[/] = next, [bold cyan]p[/] = prev, [bold cyan]q[/] = quit")
+            
+            # Get user input for navigation
             try:
-                dt = datetime.fromisoformat(end_time_raw.replace('Z', '+00:00'))
-                if terminal_width < 90:
-                    # Compact format: MM-DD HH:MM
-                    end_time = dt.strftime('%m-%d\n%H:%M')
+                choice = input(">>> ").strip().lower()
+            except (EOFError, KeyboardInterrupt):
+                console.print("\n[yellow]Pagination interrupted.[/yellow]")
+                break
+            
+            if choice in ("q", "quit"):
+                break
+            elif choice in ("n", "next"):
+                if current_page < total_pages:
+                    try:
+                        result = fetch_page_callback(current_page + 1)
+                        jobs = result.get('jobs', [])
+                        pagination_metadata = result.get('pagination_metadata', {})
+                        current_page = pagination_metadata.get('Pagination-Page', current_page + 1)
+                        total_pages = pagination_metadata.get('totalPages',
+                                                             (pagination_metadata.get('Pagination-Count', 0) + page_size_value - 1) // page_size_value
+                                                             if pagination_metadata.get('Pagination-Count', 0) > 0 else 1)
+                        
+                        # Rebuild table with new jobs using helper function
+                        table = _build_job_table(jobs, cloudos_url, terminal_width, columns_to_show, all_columns)
+                            
+                    except Exception as e:
+                        show_error = f"[red]Error fetching page: {str(e)}[/red]"
                 else:
-                    # Full format
-                    end_time = dt.strftime('%Y-%m-%d\n%H:%M:%S')
-            except (ValueError, TypeError):
-                end_time = "N/A"
-        else:
-            end_time = "N/A"
-
-        # Run time (calculate from startTime and endTime)
-        start_time_raw = job.get("startTime")
-        if start_time_raw and end_time_raw:
-            try:
-                start_dt = datetime.fromisoformat(start_time_raw.replace('Z', '+00:00'))
-                end_dt = datetime.fromisoformat(end_time_raw.replace('Z', '+00:00'))
-                duration = end_dt - start_dt
-                total_seconds = int(duration.total_seconds())
-                hours = total_seconds // 3600
-                minutes = (total_seconds % 3600) // 60
-                seconds = total_seconds % 60
-                if hours > 0:
-                    run_time = f"{hours}h {minutes}m {seconds}s"
-                elif minutes > 0:
-                    run_time = f"{minutes}m {seconds}s"
+                    show_error = "[yellow]Already on last page[/yellow]"
+            elif choice in ("p", "prev", "previous"):
+                if current_page > 1:
+                    try:
+                        result = fetch_page_callback(current_page - 1)
+                        jobs = result.get('jobs', [])
+                        pagination_metadata = result.get('pagination_metadata', {})
+                        current_page = pagination_metadata.get('Pagination-Page', current_page - 1)
+                        total_pages = pagination_metadata.get('totalPages',
+                                                             (pagination_metadata.get('Pagination-Count', 0) + page_size_value - 1) // page_size_value
+                                                             if pagination_metadata.get('Pagination-Count', 0) > 0 else 1)
+                        
+                        # Rebuild table with new jobs using helper function
+                        table = _build_job_table(jobs, cloudos_url, terminal_width, columns_to_show, all_columns)
+                            
+                    except Exception as e:
+                        show_error = f"[red]Error fetching page: {str(e)}[/red]"
                 else:
-                    run_time = f"{seconds}s"
-            except (ValueError, TypeError):
-                run_time = "N/A"
-        else:
-            run_time = "N/A"
-
-        # Commit
-        revision = job.get("revision", {})
-        if job.get("jobType") == "dockerAWS":
-            commit = str(revision.get("digest", "N/A"))
-        else:
-            commit = str(revision.get("commit", "N/A"))
-        # Truncate commit to 7 characters if it's longer
-        if commit != "N/A" and len(commit) > 7:
-            commit = commit[:7]
-
-        # Cost
-        cost_raw = job.get("computeCostSpent") or job.get("realInstancesExecutionCost")
-        if cost_raw is not None:
-            try:
-                cost = f"${float(cost_raw) / 100:.4f}"
-            except (ValueError, TypeError):
-                cost = "N/A"
-        else:
-            cost = "N/A"
-
-        # Resources (instance type only)
-        master_instance = job.get("masterInstance", {})
-        used_instance = master_instance.get("usedInstance", {})
-        instance_type = used_instance.get("type", "N/A")
-        resources = instance_type if instance_type else "N/A"
-
-        # Storage type
-        storage_mode = job.get("storageMode", "N/A")
-        if storage_mode == "regular":
-            storage_type = "Regular"
-        elif storage_mode == "lustre":
-            storage_type = "Lustre"
+                    show_error = "[yellow]Already on first page[/yellow]"
+            else:
+                show_error = "[yellow]Invalid choice. Use 'n' (next), 'p' (previous), or 'q' (quit)[/yellow]"
         else:
-            storage_type = str(storage_mode).capitalize() if storage_mode != "N/A" else "N/A"
-
-        # Map column keys to their values
-        column_values = {
-            'status': status,
-            'name': name,
-            'project': project,
-            'owner': owner,
-            'pipeline': pipeline,
-            'id': job_id_with_link,
-            'submit_time': submit_time,
-            'end_time': end_time,
-            'run_time': run_time,
-            'commit': commit,
-            'cost': cost,
-            'resources': resources,
-            'storage_type': storage_type
-        }
-
-        # Add row to table with only selected columns
-        row_values = [column_values[col] for col in columns_to_show]
-        table.add_row(*row_values)
-
-    console.print(table)
-
-    # Display pagination info at the bottom
-    if pagination_metadata:
-        total_jobs = pagination_metadata.get('Pagination-Count', 0)
-        current_page = pagination_metadata.get('Pagination-Page', 1)
-        page_size = pagination_metadata.get('Pagination-Limit', 10)
-        total_pages = (total_jobs + page_size - 1) // page_size if total_jobs > 0 else 1
-
-        console.print(f"\n[cyan]Showing {len(jobs)} of {total_jobs} total jobs | Page {current_page} of {total_pages}[/cyan]")
+            # Only one page, exit after displaying
+            break
 
 
 def create_workflow_list_table(workflows, cloudos_url="https://cloudos.lifebit.ai", page_size=10):
@@ -845,6 +959,8 @@ def create_queue_list_table(queues, cloudos_url="https://cloudos.lifebit.ai"):
             resource_type = "N/A"
         elif resource_type == "teamBatchJobQueue":
             resource_type = "Batch Queues"
+        elif resource_type == "systemBatchJobQueue":
+            resource_type = "System Queue"
 
         # Status with checkmark/X icons
         status_raw = str(queue.get("status", "N/A"))

{cloudos_cli-2.86.0 → cloudos_cli-2.87.0}/cloudos_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cloudos_cli
-Version: 2.86.0
+Version: 2.87.0
 Summary: Python package for interacting with CloudOS
 Home-page: https://github.com/lifebit-ai/cloudos-cli
 Author: David Piñeyro
@@ -436,7 +436,7 @@ Job queues are required for running jobs using AWS batch executor. The available
 
 #### List Queues
 
-This command allows you to view available computational queues and their configurations. You can get a summary of all available workspace job queues in three different output formats using the `--output-format` option:
+This command allows you to view available computational queues and their configurations. By default, both regular workspace queues and system queues are displayed. You can get a summary of all available job queues in three different output formats using the `--output-format` option:
 
 - **stdout** (default): Displays a rich formatted table directly in the terminal with pagination and visual formatting
 - **csv**: Saves queue data to a CSV file with a selection of available queue information, or all information using the `--all-fields` flag
@@ -450,6 +450,12 @@ cloudos queue list --profile my_profile
 cloudos queue list --profile my_profile --output-format stdout
 ```
 
+To exclude system queues and show only workspace queues:
+
+```bash
+cloudos queue list --profile my_profile --exclude-system-queues
+```
+
 To save all available job queues in JSON format:
 
 ```bash
@@ -472,7 +478,7 @@ cloudos queue list --profile my_profile --output-format csv
 
 **Job queues for platform workflows**
 
-Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues. Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
+Platform workflows (those provided by CloudOS in your workspace as modules) run on separate and specific AWS batch queues (system queues). Therefore, CloudOS will automatically assign the valid queue and you should not specify any queue using the `--job-queue` parameter. Any attempt to use this parameter will be ignored. Examples of such platform workflows are "System Tools" and "Data Factory" workflows.
 
 
 ### Workflow
@@ -886,7 +892,7 @@ You can find specific jobs within your workspace using the filtering options. Fi
 - **`--filter-job-id`**: Filter jobs by specific job ID (exact match required)
 - **`--filter-only-mine`**: Show only jobs belonging to the current user
 - **`--filter-owner`**: Show only jobs for the specified owner (exact match required, e.g., "John Doe")
-- **`--filter-queue`**: Filter jobs by queue name (only applies to batch jobs)
+- **`--filter-queue`**: Filter jobs by queue name (works with both regular and system queues; only applies to batch jobs)
 
 **Filtering Examples**
 

cloudos_cli-2.86.0/cloudos_cli/_version.py

@@ -1 +0,0 @@
-__version__ = '2.86.0'