apache-airflow-providers-cncf-kubernetes 10.4.3rc1__tar.gz → 10.5.0rc1__tar.gz

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-cncf-kubernetes
-Version: 10.4.3rc1
+Version: 10.5.0rc1
 Summary: Provider package apache-airflow-providers-cncf-kubernetes for Apache Airflow
 Keywords: airflow-provider,cncf.kubernetes,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -21,14 +21,14 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: aiofiles>=23.2.0
-Requires-Dist: apache-airflow>=2.9.0rc0
+Requires-Dist: apache-airflow>=2.10.0rc1
 Requires-Dist: asgiref>=3.5.2
 Requires-Dist: cryptography>=41.0.0
 Requires-Dist: kubernetes>=29.0.0,<=31.0.0
 Requires-Dist: kubernetes_asyncio>=29.0.0,<=31.0.0
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -59,7 +59,7 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 
 Package ``apache-airflow-providers-cncf-kubernetes``
 
-Release: ``10.4.3``
+Release: ``10.5.0``
 
 
 `Kubernetes <https://kubernetes.io/>`__
@@ -72,7 +72,7 @@ This is a provider package for ``cncf.kubernetes`` provider. All classes for thi
 are in ``airflow.providers.cncf.kubernetes`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/>`_.
 
 Installation
 ------------
@@ -90,7 +90,7 @@ Requirements
 PIP package             Version required
 ======================  =====================
 ``aiofiles``            ``>=23.2.0``
-``apache-airflow``      ``>=2.9.0``
+``apache-airflow``      ``>=2.10.0``
 ``asgiref``             ``>=3.5.2``
 ``cryptography``        ``>=41.0.0``
 ``kubernetes``          ``>=29.0.0,<=31.0.0``
@@ -98,5 +98,5 @@ PIP package             Version required
 ======================  =====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html>`_.
 

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-cncf-kubernetes``
 
-Release: ``10.4.3``
+Release: ``10.5.0``
 
 
 `Kubernetes <https://kubernetes.io/>`__
@@ -36,7 +36,7 @@ This is a provider package for ``cncf.kubernetes`` provider. All classes for thi
 are in ``airflow.providers.cncf.kubernetes`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/>`_.
 
 Installation
 ------------
@@ -54,7 +54,7 @@ Requirements
 PIP package             Version required
 ======================  =====================
 ``aiofiles``            ``>=23.2.0``
-``apache-airflow``      ``>=2.9.0``
+``apache-airflow``      ``>=2.10.0``
 ``asgiref``             ``>=3.5.2``
 ``cryptography``        ``>=41.0.0``
 ``kubernetes``          ``>=29.0.0,<=31.0.0``
@@ -62,4 +62,4 @@ PIP package             Version required
 ======================  =====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html>`_.

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/docs/changelog.rst

@@ -28,6 +28,35 @@ Changelog
 ---------
 
 
+10.5.0
+......
+
+.. note::
+    This release of provider is only available for Airflow 2.10+ as explained in the
+    Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
+
+Features
+~~~~~~~~
+
+* ``Add fast fail for ErrImagePull and InvalidImageName for KubernetesPodOperator (#49867)``
+* ``KubernetesPodOperator uses different timeouts to check for schedule timeout and startup timeout (#49784)``
+* ``Add configurable automountServiceAccountToken for the KubernetesPodOperator (#50223)``
+* ``Add 'test_connection' method to 'KubernetesHook' (#47881)``
+* ``Add '@task.kuberenetes_cmd'  (#46913)``
+
+Misc
+~~~~
+
+* ``Move SQS message queue to Amazon provider (#50057)``
+* ``Remove AIRFLOW_2_10_PLUS conditions (#49877)``
+* ``Bump min Airflow version in providers to 2.10 (#49843)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Update description of provider.yaml dependencies (#50231)``
+   * ``Avoid committing history for providers (#49907)``
+   * ``capitalize the term airflow (#49450)``
+
 10.4.3
 ......
 

apache_airflow_providers_cncf_kubernetes-10.5.0rc1/docs/commits.rst

@@ -0,0 +1,35 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+ .. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+    `PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+
+ .. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN!
+
+Package apache-airflow-providers-cncf-kubernetes
+------------------------------------------------------
+
+`Kubernetes <https://kubernetes.io/>`__
+
+
+This is detailed commit list of changes for versions provider package: ``cncf.kubernetes``.
+For high-level changelog, see :doc:`package information including changelog <index>`.
+
+.. airflow-providers-commits::

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/docs/index.rst

@@ -87,7 +87,7 @@ apache-airflow-providers-cncf-kubernetes package
 `Kubernetes <https://kubernetes.io/>`__
 
 
-Release: 10.4.3
+Release: 10.5.0
 
 Provider package
 ----------------
@@ -105,13 +105,13 @@ For the minimum Airflow version supported, see ``Requirements`` below.
 Requirements
 ------------
 
-The minimum Apache Airflow version supported by this provider distribution is ``2.9.0``.
+The minimum Apache Airflow version supported by this provider distribution is ``2.10.0``.
 
 ======================  =====================
 PIP package             Version required
 ======================  =====================
 ``aiofiles``            ``>=23.2.0``
-``apache-airflow``      ``>=2.9.0``
+``apache-airflow``      ``>=2.10.0``
 ``asgiref``             ``>=3.5.2``
 ``cryptography``        ``>=41.0.0``
 ``kubernetes``          ``>=29.0.0,<=31.0.0``

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/docs/operators.rst

@@ -182,6 +182,40 @@ Also for this action you can use operator in the deferrable mode:
     :start-after: [START howto_operator_k8s_write_xcom_async]
     :end-before: [END howto_operator_k8s_write_xcom_async]
 
+
+Run command in KubernetesPodOperator from TaskFlow
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+With the usage of the ``@task.kubernetes_cmd`` decorator, you can run a command returned by a function
+in a ``KubernetesPodOperator`` simplifying it's connection to the TaskFlow.
+
+Difference between ``@task.kubernetes`` and ``@task.kubernetes_cmd``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``@task.kubernetes`` decorator is designed to run a Python function inside a Kubernetes pod using KPO.
+It does this by serializing the function into a temporary Python script that is executed inside the container.
+This is well-suited for cases where you want to isolate Python code execution and manage complex dependencies,
+as described in the :doc:`TaskFlow documentation <apache-airflow:tutorial/taskflow>`.
+
+In contrast, ``@task.kubernetes_cmd`` decorator allows the decorated function to return
+a shell command (as a list of strings), which is then passed as cmds or arguments to
+``KubernetesPodOperator``.
+This enables executing arbitrary commands available inside a Kubernetes pod --
+without needing to wrap it in Python code.
+
+A key benefit here is that Python excels at composing and templating these commands.
+Shell commands can be dynamically generated using Python's string formatting, templating,
+extra function calls and logic. This makes it a flexible tool for orchestrating complex pipelines
+where the task is to invoke CLI-based operations in containers without the need to leave
+a TaskFlow context.
+
+How does this decorator work?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+See the following examples on how the decorator works:
+
+.. exampleinclude:: /../tests/system/cncf/kubernetes/example_kubernetes_cmd_decorator.py
+    :language: python
+    :start-after: [START howto_decorator_kubernetes_cmd]
+    :end-before: [END howto_decorator_kubernetes_cmd]
+
 Include error message in email alert
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -285,7 +319,7 @@ Never use environment variables to pass secrets (for example connection authenti
 Kubernetes Pod Operator. Such environment variables will be visible to anyone who has access
 to see and describe PODs in Kubernetes. Instead, pass your secrets via native Kubernetes ``Secrets`` or
 use Connections and Variables from Airflow. For the latter, you need to have ``apache-airflow`` package
-installed in your image in the same version as airflow you run your Kubernetes Pod Operator from).
+installed in your image in the same version as Airflow you run your Kubernetes Pod Operator from).
 
 Reference
 ^^^^^^^^^

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/provider.yaml

@@ -22,9 +22,13 @@ description: |
     `Kubernetes <https://kubernetes.io/>`__
 
 state: ready
-source-date-epoch: 1744788926
-# note that those versions are maintained by release manager - do not update them manually
+source-date-epoch: 1747132113
+# Note that those versions are maintained by release manager - do not update them manually
+# with the exception of case where other provider in sources has >= new provider version.
+# In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
+# to be done in the same PR
 versions:
+  - 10.5.0
   - 10.4.3
   - 10.4.2
   - 10.4.1
@@ -147,6 +151,8 @@ connection-types:
 task-decorators:
   - class-name: airflow.providers.cncf.kubernetes.decorators.kubernetes.kubernetes_task
     name: kubernetes
+  - class-name: airflow.providers.cncf.kubernetes.decorators.kubernetes_cmd.kubernetes_cmd_task
+    name: kubernetes_cmd
 
 config:
   local_kubernetes_executor:
@@ -370,4 +376,4 @@ config:
         default: "0"
 
 executors:
-  - airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor
+  - airflow.providers.cncf.kubernetes.executors.kubernetes_executor.KubernetesExecutor

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-cncf-kubernetes"
-version = "10.4.3.rc1"
+version = "10.5.0rc1"
 description = "Provider package apache-airflow-providers-cncf-kubernetes for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -58,7 +58,7 @@ requires-python = "~=3.9"
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
     "aiofiles>=23.2.0",
-    "apache-airflow>=2.9.0rc0",
+    "apache-airflow>=2.10.0rc1",
     "asgiref>=3.5.2",
     "cryptography>=41.0.0",
     # The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
@@ -112,8 +112,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.4.3/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/src/airflow/providers/cncf/kubernetes/__init__.py

@@ -29,11 +29,11 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "10.4.3"
+__version__ = "10.5.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
-    "2.9.0"
+    "2.10.0"
 ):
     raise RuntimeError(
-        f"The package `apache-airflow-providers-cncf-kubernetes:{__version__}` needs Apache Airflow 2.9.0+"
+        f"The package `apache-airflow-providers-cncf-kubernetes:{__version__}` needs Apache Airflow 2.10.0+"
     )

apache_airflow_providers_cncf_kubernetes-10.5.0rc1/src/airflow/providers/cncf/kubernetes/decorators/kubernetes_cmd.py

@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import warnings
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Callable
+
+from airflow.providers.cncf.kubernetes.version_compat import AIRFLOW_V_3_0_PLUS
+
+if AIRFLOW_V_3_0_PLUS:
+    from airflow.sdk.bases.decorator import DecoratedOperator, TaskDecorator, task_decorator_factory
+else:
+    from airflow.decorators.base import (  # type: ignore[no-redef]
+        DecoratedOperator,
+        TaskDecorator,
+        task_decorator_factory,
+    )
+from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
+from airflow.utils.context import context_merge
+from airflow.utils.operator_helpers import determine_kwargs
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class _KubernetesCmdDecoratedOperator(DecoratedOperator, KubernetesPodOperator):
+    custom_operator_name = "@task.kubernetes_cmd"
+
+    template_fields: Sequence[str] = KubernetesPodOperator.template_fields
+    overwrite_rtif_after_execution: bool = True
+
+    def __init__(self, *, python_callable: Callable, args_only: bool = False, **kwargs) -> None:
+        self.args_only = args_only
+
+        cmds = kwargs.pop("cmds", None)
+        arguments = kwargs.pop("arguments", None)
+
+        if cmds is not None or arguments is not None:
+            warnings.warn(
+                f"The `cmds` and `arguments` are unused in {self.custom_operator_name} decorator. "
+                "You should return a list of commands or image entrypoint arguments with "
+                "args_only=True from the python_callable.",
+                UserWarning,
+                stacklevel=3,
+            )
+
+        # If the name was not provided, we generate operator name from the python_callable
+        # we also instruct operator to add a random suffix to avoid collisions by default
+        op_name = kwargs.pop("name", f"k8s-airflow-pod-{python_callable.__name__}")
+        random_name_suffix = kwargs.pop("random_name_suffix", True)
+
+        super().__init__(
+            python_callable=python_callable,
+            name=op_name,
+            random_name_suffix=random_name_suffix,
+            cmds=None,
+            arguments=None,
+            **kwargs,
+        )
+
+    def execute(self, context: Context):
+        generated = self._generate_cmds(context)
+        if self.args_only:
+            self.cmds = []
+            self.arguments = generated
+        else:
+            self.cmds = generated
+            self.arguments = []
+        context["ti"].render_templates()  # type: ignore[attr-defined]
+        return super().execute(context)
+
+    def _generate_cmds(self, context: Context) -> list[str]:
+        context_merge(context, self.op_kwargs)
+        kwargs = determine_kwargs(self.python_callable, self.op_args, context)
+        generated_cmds = self.python_callable(*self.op_args, **kwargs)
+        func_name = self.python_callable.__name__
+        if not isinstance(generated_cmds, list):
+            raise TypeError(
+                f"Expected python_callable to return a list of strings, but got {type(generated_cmds)}"
+            )
+        if not all(isinstance(cmd, str) for cmd in generated_cmds):
+            raise TypeError(f"Expected {func_name} to return a list of strings, but got {generated_cmds}")
+        if not generated_cmds:
+            raise ValueError(f"The {func_name} returned an empty list of commands")
+
+        return generated_cmds
+
+
+def kubernetes_cmd_task(
+    python_callable: Callable | None = None,
+    **kwargs,
+) -> TaskDecorator:
+    """
+    Kubernetes cmd operator decorator.
+
+    This wraps a function which should return command to be executed
+    in K8s using KubernetesPodOperator. The function should return a list of strings.
+    If args_only is set to True, the function should return a list of arguments for
+    container default command. Also accepts any argument that KubernetesPodOperator
+    will via ``kwargs``. Can be reused in a single DAG.
+
+    :param python_callable: Function to decorate
+    """
+    return task_decorator_factory(
+        python_callable=python_callable,
+        decorated_operator_class=_KubernetesCmdDecoratedOperator,
+        **kwargs,
+    )

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/src/airflow/providers/cncf/kubernetes/get_provider_info.py

@@ -85,7 +85,11 @@ def get_provider_info():
             {
                 "class-name": "airflow.providers.cncf.kubernetes.decorators.kubernetes.kubernetes_task",
                 "name": "kubernetes",
-            }
+            },
+            {
+                "class-name": "airflow.providers.cncf.kubernetes.decorators.kubernetes_cmd.kubernetes_cmd_task",
+                "name": "kubernetes_cmd",
+            },
         ],
         "config": {
             "local_kubernetes_executor": {
@@ -275,5 +279,5 @@ def get_provider_info():
                 },
             },
         },
-        "executors": ["airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor"],
+        "executors": ["airflow.providers.cncf.kubernetes.executors.kubernetes_executor.KubernetesExecutor"],
     }

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py

@@ -140,7 +140,8 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
 
     def __init__(
         self,
-        conn_id: str | None = default_conn_name,
+        conn_id: str | None = None,
+        kubernetes_conn_id: str | None = default_conn_name,
         client_configuration: client.Configuration | None = None,
         cluster_context: str | None = None,
         config_file: str | None = None,
@@ -149,7 +150,7 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
         disable_tcp_keepalive: bool | None = None,
     ) -> None:
         super().__init__()
-        self.conn_id = conn_id
+        self.conn_id = conn_id or kubernetes_conn_id
         self.client_configuration = client_configuration
         self.cluster_context = cluster_context
         self.config_file = config_file
@@ -706,6 +707,14 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
 
         return list(yaml.safe_load_all(response.text))
 
+    def test_connection(self):
+        try:
+            conn = self.get_conn()
+            version: client.VersionInfo = client.VersionApi(conn).get_code()
+            return True, f"Connection successful. Version Info: {version.to_dict()}"
+        except Exception as e:
+            return False, str(e)
+
 
 def _get_bool(val) -> bool | None:
     """Convert val to bool if can be done with certainty; if we cannot infer intention we return None."""

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/src/airflow/providers/cncf/kubernetes/operators/pod.py

@@ -157,8 +157,9 @@ class KubernetesPodOperator(BaseOperator):
     :param reattach_on_restart: if the worker dies while the pod is running, reattach and monitor
         during the next try. If False, always create a new pod for each try.
     :param labels: labels to apply to the Pod. (templated)
-    :param startup_timeout_seconds: timeout in seconds to startup the pod.
+    :param startup_timeout_seconds: timeout in seconds to startup the pod after pod was scheduled.
     :param startup_check_interval_seconds: interval in seconds to check if the pod has already started
+    :param schedule_timeout_seconds: timeout in seconds to schedule pod in cluster.
     :param get_logs: get the stdout of the base container as logs of the tasks.
     :param init_container_logs: list of init containers whose logs will be published to stdout
         Takes a sequence of containers, a single container name or True. If True,
@@ -180,6 +181,7 @@ class KubernetesPodOperator(BaseOperator):
         If more than one secret is required, provide a
         comma separated list: secret_a,secret_b
     :param service_account_name: Name of the service account
+    :param automount_service_account_token: indicates whether pods running as this service account should have an API token automatically mounted
     :param hostnetwork: If True enable host networking on the pod.
     :param host_aliases: A list of host aliases to apply to the containers in the pod.
     :param tolerations: A list of kubernetes tolerations.
@@ -289,6 +291,7 @@ class KubernetesPodOperator(BaseOperator):
         reattach_on_restart: bool = True,
         startup_timeout_seconds: int = 120,
         startup_check_interval_seconds: int = 5,
+        schedule_timeout_seconds: int | None = None,
         get_logs: bool = True,
         base_container_name: str | None = None,
         base_container_status_polling_interval: float = 1,
@@ -302,6 +305,7 @@ class KubernetesPodOperator(BaseOperator):
         node_selector: dict | None = None,
         image_pull_secrets: list[k8s.V1LocalObjectReference] | None = None,
         service_account_name: str | None = None,
+        automount_service_account_token: bool = True,
         hostnetwork: bool = False,
         host_aliases: list[k8s.V1HostAlias] | None = None,
         tolerations: list[k8s.V1Toleration] | None = None,
@@ -347,6 +351,8 @@ class KubernetesPodOperator(BaseOperator):
         self.labels = labels or {}
         self.startup_timeout_seconds = startup_timeout_seconds
         self.startup_check_interval_seconds = startup_check_interval_seconds
+        # New parameter startup_timeout_seconds adds breaking change, to handle this as smooth as possible just reuse startup time
+        self.schedule_timeout_seconds = schedule_timeout_seconds or startup_timeout_seconds
         env_vars = convert_env_vars(env_vars) if env_vars else []
         self.env_vars = env_vars
         pod_runtime_info_envs = (
@@ -380,6 +386,7 @@ class KubernetesPodOperator(BaseOperator):
         self.config_file = config_file
         self.image_pull_secrets = convert_image_pull_secrets(image_pull_secrets) if image_pull_secrets else []
         self.service_account_name = service_account_name
+        self.automount_service_account_token = automount_service_account_token
         self.hostnetwork = hostnetwork
         self.host_aliases = host_aliases
         self.tolerations = (
@@ -574,8 +581,9 @@ class KubernetesPodOperator(BaseOperator):
         try:
             self.pod_manager.await_pod_start(
                 pod=pod,
+                schedule_timeout=self.schedule_timeout_seconds,
                 startup_timeout=self.startup_timeout_seconds,
-                startup_check_interval=self.startup_check_interval_seconds,
+                check_interval=self.startup_check_interval_seconds,
             )
         except PodLaunchFailedException:
             if self.log_events_on_failure:
@@ -1175,6 +1183,7 @@ class KubernetesPodOperator(BaseOperator):
                 ],
                 image_pull_secrets=self.image_pull_secrets,
                 service_account_name=self.service_account_name,
+                automount_service_account_token=self.automount_service_account_token,
                 host_network=self.hostnetwork,
                 hostname=self.hostname,
                 subdomain=self.subdomain,

{apache_airflow_providers_cncf_kubernetes-10.4.3rc1 → apache_airflow_providers_cncf_kubernetes-10.5.0rc1}/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py

@@ -46,8 +46,11 @@ from airflow.utils.timezone import utcnow
 
 if TYPE_CHECKING:
     from kubernetes.client.models.core_v1_event_list import CoreV1EventList
+    from kubernetes.client.models.v1_container_state import V1ContainerState
+    from kubernetes.client.models.v1_container_state_waiting import V1ContainerStateWaiting
     from kubernetes.client.models.v1_container_status import V1ContainerStatus
     from kubernetes.client.models.v1_pod import V1Pod
+    from kubernetes.client.models.v1_pod_condition import V1PodCondition
     from urllib3.response import HTTPResponse
 
 
@@ -375,30 +378,68 @@ class PodManager(LoggingMixin):
         return self.run_pod_async(pod)
 
     def await_pod_start(
-        self, pod: V1Pod, startup_timeout: int = 120, startup_check_interval: int = 1
+        self, pod: V1Pod, schedule_timeout: int = 120, startup_timeout: int = 120, check_interval: int = 1
     ) -> None:
         """
         Wait for the pod to reach phase other than ``Pending``.
 
         :param pod:
+        :param schedule_timeout: Timeout (in seconds) for pod stay in schedule state
+            (if pod is taking to long in schedule state, fails task)
         :param startup_timeout: Timeout (in seconds) for startup of the pod
-            (if pod is pending for too long, fails task)
-        :param startup_check_interval: Interval (in seconds) between checks
+            (if pod is pending for too long after being scheduled, fails task)
+        :param check_interval: Interval (in seconds) between checks
         :return:
         """
-        curr_time = time.time()
+        self.log.info("::group::Waiting until %ss to get the POD scheduled...", schedule_timeout)
+        pod_was_scheduled = False
+        start_check_time = time.time()
         while True:
             remote_pod = self.read_pod(pod)
-            if remote_pod.status.phase != PodPhase.PENDING:
+            pod_status = remote_pod.status
+            if pod_status.phase != PodPhase.PENDING:
+                self.keep_watching_for_events = False
+                self.log.info("::endgroup::")
                 break
-            self.log.warning("Pod not yet started: %s", pod.metadata.name)
-            if time.time() - curr_time >= startup_timeout:
-                msg = (
-                    f"Pod took longer than {startup_timeout} seconds to start. "
-                    "Check the pod events in kubernetes to determine why."
-                )
-                raise PodLaunchFailedException(msg)
-            time.sleep(startup_check_interval)
+
+            # Check for timeout
+            pod_conditions: list[V1PodCondition] = pod_status.conditions
+            if pod_conditions and any(
+                (condition.type == "PodScheduled" and condition.status == "True")
+                for condition in pod_conditions
+            ):
+                if not pod_was_scheduled:
+                    # POD was initially scheduled update timeout for getting POD launched
+                    pod_was_scheduled = True
+                    start_check_time = time.time()
+                    self.log.info("Waiting %ss to get the POD running...", startup_timeout)
+
+                if time.time() - start_check_time >= startup_timeout:
+                    self.log.info("::endgroup::")
+                    raise PodLaunchFailedException(
+                        f"Pod took too long to start. More than {startup_timeout}s. Check the pod events in kubernetes."
+                    )
+            else:
+                if time.time() - start_check_time >= schedule_timeout:
+                    self.log.info("::endgroup::")
+                    raise PodLaunchFailedException(
+                        f"Pod took too long to be scheduled on the cluster, giving up. More than {schedule_timeout}s. Check the pod events in kubernetes."
+                    )
+
+            # Check for general problems to terminate early - ErrImagePull
+            if pod_status.container_statuses:
+                for container_status in pod_status.container_statuses:
+                    container_state: V1ContainerState = container_status.state
+                    container_waiting: V1ContainerStateWaiting | None = container_state.waiting
+                    if container_waiting:
+                        if container_waiting.reason in ["ErrImagePull", "InvalidImageName"]:
+                            self.log.info("::endgroup::")
+                            raise PodLaunchFailedException(
+                                f"Pod docker image cannot be pulled, unable to start: {container_waiting.reason}"
+                                f"\n{container_waiting.message}"
+                            )
+
+            time.sleep(check_interval)
 
     def fetch_container_logs(
         self,