skypilot-nightly 1.0.0.dev20250306__tar.gz → 1.0.0.dev20250308__tar.gz

{skypilot_nightly-1.0.0.dev20250306/skypilot_nightly.egg-info → skypilot_nightly-1.0.0.dev20250308}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: skypilot-nightly
-Version: 1.0.0.dev20250306
+Version: 1.0.0.dev20250308
 Summary: SkyPilot: An intercloud broker for the clouds
 Author: SkyPilot Team
 License: Apache 2.0
@@ -208,12 +208,12 @@ Dynamic: summary
 
 SkyPilot is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.
 
-SkyPilot **abstracts away infra burdens**:
+SkyPilot **abstracts away AI infra burden**:
 - Launch [clusters](https://docs.skypilot.co/en/latest/examples/interactive-development.html), [jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html), and [serving](https://docs.skypilot.co/en/latest/serving/sky-serve.html) on any infra
 - Easy job management: queue, run, and auto-recover many jobs
 
-SkyPilot **supports multiple clusters, clouds, and hardware** ([the Sky](https://arxiv.org/abs/2205.07147)):
-- Bring your reserved GPUs, Kubernetes clusters, or 12+ clouds
+SkyPilot **supports multiple clusters, clouds, and hardware**:
+- Bring reserved GPUs, Kubernetes clusters, or 15+ clouds
 - [Flexible provisioning](https://docs.skypilot.co/en/latest/examples/auto-failover.html) of GPUs, TPUs, CPUs, with auto-retry
 
 SkyPilot **cuts your cloud costs & maximizes GPU availability**:
@@ -235,7 +235,9 @@ pip install "skypilot-nightly[kubernetes,aws,gcp,azure,oci,lambda,runpod,fluidst
 ```
 
 
-[Current supported infra](https://docs.skypilot.co/en/latest/getting-started/installation.html) (Kubernetes; AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack, RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Nebius):
+Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack,
+RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
+VMware vSphere, Nebius.
 <p align="center">
   <img alt="SkyPilot" src="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-light.png" width=85%>
 </p>

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/README.md

@@ -48,12 +48,12 @@
 
 SkyPilot is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.
 
-SkyPilot **abstracts away infra burdens**:
+SkyPilot **abstracts away AI infra burden**:
 - Launch [clusters](https://docs.skypilot.co/en/latest/examples/interactive-development.html), [jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html), and [serving](https://docs.skypilot.co/en/latest/serving/sky-serve.html) on any infra
 - Easy job management: queue, run, and auto-recover many jobs
 
-SkyPilot **supports multiple clusters, clouds, and hardware** ([the Sky](https://arxiv.org/abs/2205.07147)):
-- Bring your reserved GPUs, Kubernetes clusters, or 12+ clouds
+SkyPilot **supports multiple clusters, clouds, and hardware**:
+- Bring reserved GPUs, Kubernetes clusters, or 15+ clouds
 - [Flexible provisioning](https://docs.skypilot.co/en/latest/examples/auto-failover.html) of GPUs, TPUs, CPUs, with auto-retry
 
 SkyPilot **cuts your cloud costs & maximizes GPU availability**:
@@ -75,7 +75,9 @@ pip install "skypilot-nightly[kubernetes,aws,gcp,azure,oci,lambda,runpod,fluidst
 ```
 
 
-[Current supported infra](https://docs.skypilot.co/en/latest/getting-started/installation.html) (Kubernetes; AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack, RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Nebius):
+Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack,
+RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
+VMware vSphere, Nebius.
 <p align="center">
   <picture>
     <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-dark.png">

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/__init__.py

@@ -5,7 +5,7 @@ from typing import Optional
 import urllib.request
 
 # Replaced with the current commit when building the wheels.
-_SKYPILOT_COMMIT_SHA = 'db832e82e19faa3e428e946c74825a752ebb7402'
+_SKYPILOT_COMMIT_SHA = 'f2c2092f3fc343d59348bf209b3ad0d4d76cea7b'
 
 
 def _get_git_commit():
@@ -35,7 +35,7 @@ def _get_git_commit():
 
 
 __commit__ = _get_git_commit()
-__version__ = '1.0.0.dev20250306'
+__version__ = '1.0.0.dev20250308'
 __root_dir__ = os.path.dirname(os.path.abspath(__file__))
 
 

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/backends/wheel_utils.py

@@ -17,7 +17,7 @@ import re
 import shutil
 import subprocess
 import tempfile
-from typing import Tuple
+from typing import Optional, Tuple
 
 import filelock
 from packaging import version
@@ -149,16 +149,29 @@ def build_sky_wheel() -> Tuple[pathlib.Path, str]:
         - wheel_hash: The wheel content hash.
     """
 
-    def _get_latest_modification_time(path: pathlib.Path) -> float:
+    def _get_latest_modification_time(path: pathlib.Path) -> Optional[float]:
+        max_time = -1.
         if not path.exists():
-            return -1.
-        try:
-            return max(
-                os.path.getmtime(os.path.join(root, f))
-                for root, dirs, files in os.walk(path)
-                for f in (*dirs, *files))
-        except ValueError:
-            return -1.
+            return max_time
+        for root, dirs, files in os.walk(path):
+            # Prune __pycache__ directories to prevent walking into them and
+            # exclude them from processing
+            if '__pycache__' in dirs:
+                dirs.remove('__pycache__')
+            # Filter out .pyc files
+            filtered_files = [f for f in files if not f.endswith('.pyc')]
+            # Process remaining directories and files
+            for entry in (*dirs, *filtered_files):
+                entry_path = os.path.join(root, entry)
+                try:
+                    mtime = os.path.getmtime(entry_path)
+                    if mtime > max_time:
+                        max_time = mtime
+                except OSError:
+                    # Handle cases where file might have been deleted after
+                    # listing
+                    return None
+        return max_time
 
     # This lock prevents that the wheel is updated while being copied.
     # Although the current caller already uses a lock, we still lock it here
@@ -170,10 +183,12 @@ def build_sky_wheel() -> Tuple[pathlib.Path, str]:
         last_modification_time = _get_latest_modification_time(SKY_PACKAGE_PATH)
         last_wheel_modification_time = _get_latest_modification_time(WHEEL_DIR)
 
-        # Only build wheels if the wheel is outdated or wheel does not exist
-        # for the requested version.
-        if (last_wheel_modification_time < last_modification_time) or not any(
-                WHEEL_DIR.glob(f'**/{_WHEEL_PATTERN}')):
+        # Only build wheels if the wheel is outdated, wheel does not exist
+        # for the requested version, or files were deleted during checking.
+        if ((last_modification_time is None or
+             last_wheel_modification_time is None) or
+            (last_wheel_modification_time < last_modification_time) or
+                not any(WHEEL_DIR.glob(f'**/{_WHEEL_PATTERN}'))):
             if not WHEEL_DIR.exists():
                 WHEEL_DIR.mkdir(parents=True, exist_ok=True)
             latest_wheel = _build_sky_wheel()

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/jobs/server/core.py

@@ -157,6 +157,7 @@ def launch(
             'modified_catalogs':
                 service_catalog_common.get_modified_catalog_file_mounts(),
             'dashboard_setup_cmd': managed_job_constants.DASHBOARD_SETUP_CMD,
+            'dashboard_user_id': common.SERVER_ID,
             **controller_utils.shared_controller_vars_to_fill(
                 controller_utils.Controllers.JOBS_CONTROLLER,
                 remote_user_config_path=remote_user_config_path,
@@ -305,10 +306,9 @@ def _maybe_restart_controller(
     with rich_utils.safe_status(
             ux_utils.spinner_message('Starting dashboard...')):
         runner = handle.get_command_runners()[0]
-        user_hash = common_utils.get_user_hash()
         runner.run(
             f'export '
-            f'{skylet_constants.USER_ID_ENV_VAR}={user_hash!r}; '
+            f'{skylet_constants.USER_ID_ENV_VAR}={common.SERVER_ID!r}; '
             f'{managed_job_constants.DASHBOARD_SETUP_CMD}',
             stream_logs=True,
         )

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/jobs/server/server.py

@@ -159,6 +159,9 @@ async def dashboard(request: fastapi.Request,
             except Exception as e:  # pylint: disable=broad-except
                 # We catch all exceptions to gracefully handle unknown
                 # errors and retry or raise an HTTPException to the client.
+                # Assume an exception indicates that the dashboard connection
+                # is stale - remove it so that a new one is created.
+                dashboard_utils.remove_dashboard_session(user_hash)
                 msg = (
                     f'Dashboard connection attempt {attempt + 1} failed with '
                     f'{common_utils.format_exception(e, use_bracket=True)}')

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/skylet/constants.py

@@ -268,6 +268,12 @@ USER_ID_ENV_VAR = f'{SKYPILOT_ENV_VAR_PREFIX}USER_ID'
 # runs on a VM launched by SkyPilot will be recognized as the same user.
 USER_ENV_VAR = f'{SKYPILOT_ENV_VAR_PREFIX}USER'
 
+# Internal: Env var indicating the system is running with a remote API server.
+# It is used for internal purposes, including the jobs controller to mark
+# clusters as launched with a remote API server.
+USING_REMOTE_API_SERVER_ENV_VAR = (
+    f'{SKYPILOT_ENV_VAR_PREFIX}USING_REMOTE_API_SERVER')
+
 # In most clouds, cluster names can only contain lowercase letters, numbers
 # and hyphens. We use this regex to validate the cluster name.
 CLUSTER_NAME_VALID_REGEX = '[a-zA-Z]([-_.a-zA-Z0-9]*[a-zA-Z0-9])?'

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/templates/jobs-controller.yaml.j2

@@ -39,9 +39,7 @@ setup: |
   After=network.target
 
   [Service]
-  Environment="PATH={{ sky_python_env_path }}:\$PATH"
-  Environment="SKYPILOT_USER_ID={{controller_envs.SKYPILOT_USER_ID}}"
-  Environment="SKYPILOT_USER={{controller_envs.SKYPILOT_USER}}"
+  Environment="SKYPILOT_USER_ID={{ dashboard_user_id }}"
   Restart=always
   StandardOutput=append:/home/$USER/.sky/job-dashboard.log
   StandardError=append:/home/$USER/.sky/job-dashboard.log
@@ -51,6 +49,7 @@ setup: |
   WantedBy=default.target
   EOF
 
+  export SKYPILOT_USER_ID="{{ dashboard_user_id }}"
   {{ dashboard_setup_cmd }}
 
 run: |

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/utils/command_runner.py

@@ -531,8 +531,8 @@ class SSHCommandRunner(CommandRunner):
         if port_forward is not None:
             for local, remote in port_forward:
                 logger.info(
-                    f'Forwarding port {local} to port {remote} on localhost.')
-                ssh += ['-NL', f'{remote}:localhost:{local}']
+                    f'Forwarding local port {local} to remote port {remote}.')
+                ssh += ['-NL', f'{local}:localhost:{remote}']
         if self._docker_ssh_proxy_command is not None:
             docker_ssh_proxy_command = self._docker_ssh_proxy_command(ssh)
         else:

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/utils/common_utils.py

@@ -291,6 +291,9 @@ def get_current_client_entrypoint(server_entrypoint: str) -> str:
 
 def get_using_remote_api_server() -> bool:
     """Returns whether the API server is remote."""
+    if os.getenv(constants.USING_REMOTE_API_SERVER_ENV_VAR) is not None:
+        return os.getenv(constants.USING_REMOTE_API_SERVER_ENV_VAR,
+                         '').lower() in ('true', '1')
     if _using_remote_api_server is not None:
         return _using_remote_api_server
     # This gets the right status for the local client.

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/utils/controller_utils.py

@@ -431,6 +431,11 @@ def shared_controller_vars_to_fill(
         env_options.Options.SKIP_CLOUD_IDENTITY_CHECK.env_key: '1',
         # Disable minimize logging to get more details on the controller.
         env_options.Options.MINIMIZE_LOGGING.env_key: '0',
+        # Make sure the clusters launched by the controller are marked as
+        # launched with a remote API server if the controller is launched
+        # with a remote API server.
+        constants.USING_REMOTE_API_SERVER_ENV_VAR: str(
+            common_utils.get_using_remote_api_server()),
     })
     if skypilot_config.loaded():
         # Only set the SKYPILOT_CONFIG env var if the user has a config file.

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308}/sky/utils/rich_utils.py

@@ -230,15 +230,53 @@ def decode_rich_status(
     decoding_status = None
     try:
         last_line = ''
+        # Buffer to store incomplete UTF-8 bytes between chunks
+        undecoded_buffer = b''
+
         # Iterate over the response content in chunks. We do not use iter_lines
         # because it will strip the trailing newline characters, causing the
         # progress bar ending with `\r` becomes a pyramid.
-        for encoded_msg in response.iter_content(chunk_size=None):
-            if encoded_msg is None:
+        for chunk in response.iter_content(chunk_size=None):
+            if chunk is None:
                 return
-            encoded_msg = encoded_msg.decode('utf-8')
+
+            # Append the new chunk to any leftover bytes from previous iteration
+            current_bytes = undecoded_buffer + chunk
+            undecoded_buffer = b''
+
+            # Try to decode the combined bytes
+            try:
+                encoded_msg = current_bytes.decode('utf-8')
+            except UnicodeDecodeError as e:
+                # Check if this is potentially an incomplete sequence at the end
+                if e.start > 0:
+                    # Decode the valid part
+                    encoded_msg = current_bytes[:e.start].decode('utf-8')
+
+                    # Check if the remaining bytes are likely a partial char
+                    # or actually invalid UTF-8
+                    remaining_bytes = current_bytes[e.start:]
+                    if len(remaining_bytes) < 4:  # Max UTF-8 char is 4 bytes
+                        # Likely incomplete - save for next chunk
+                        undecoded_buffer = remaining_bytes
+                    else:
+                        # Likely invalid - replace with replacement character
+                        encoded_msg += remaining_bytes.decode('utf-8',
+                                                              errors='replace')
+                        undecoded_buffer = b''
+                else:
+                    # Error at the very beginning of the buffer - invalid UTF-8
+                    encoded_msg = current_bytes.decode('utf-8',
+                                                       errors='replace')
+                    undecoded_buffer = b''
+
             lines = encoded_msg.splitlines(keepends=True)
 
+            # Skip processing if lines is empty to avoid IndexError
+            if not lines:
+                continue
+
+            # Append any leftover text from previous chunk to first line
             lines[0] = last_line + lines[0]
             last_line = lines[-1]
             # If the last line is not ended with `\r` or `\n` (with ending

{skypilot_nightly-1.0.0.dev20250306 → skypilot_nightly-1.0.0.dev20250308/skypilot_nightly.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: skypilot-nightly
-Version: 1.0.0.dev20250306
+Version: 1.0.0.dev20250308
 Summary: SkyPilot: An intercloud broker for the clouds
 Author: SkyPilot Team
 License: Apache 2.0
@@ -208,12 +208,12 @@ Dynamic: summary
 
 SkyPilot is a framework for running AI and batch workloads on any infra, offering unified execution, high cost savings, and high GPU availability.
 
-SkyPilot **abstracts away infra burdens**:
+SkyPilot **abstracts away AI infra burden**:
 - Launch [clusters](https://docs.skypilot.co/en/latest/examples/interactive-development.html), [jobs](https://docs.skypilot.co/en/latest/examples/managed-jobs.html), and [serving](https://docs.skypilot.co/en/latest/serving/sky-serve.html) on any infra
 - Easy job management: queue, run, and auto-recover many jobs
 
-SkyPilot **supports multiple clusters, clouds, and hardware** ([the Sky](https://arxiv.org/abs/2205.07147)):
-- Bring your reserved GPUs, Kubernetes clusters, or 12+ clouds
+SkyPilot **supports multiple clusters, clouds, and hardware**:
+- Bring reserved GPUs, Kubernetes clusters, or 15+ clouds
 - [Flexible provisioning](https://docs.skypilot.co/en/latest/examples/auto-failover.html) of GPUs, TPUs, CPUs, with auto-retry
 
 SkyPilot **cuts your cloud costs & maximizes GPU availability**:
@@ -235,7 +235,9 @@ pip install "skypilot-nightly[kubernetes,aws,gcp,azure,oci,lambda,runpod,fluidst
 ```
 
 
-[Current supported infra](https://docs.skypilot.co/en/latest/getting-started/installation.html) (Kubernetes; AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack, RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Nebius):
+Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Lambda Cloud, Fluidstack,
+RunPod, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
+VMware vSphere, Nebius.
 <p align="center">
   <img alt="SkyPilot" src="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-light.png" width=85%>
 </p>