apache-airflow-providers-cncf-kubernetes 10.7.0__tar.gz → 10.8.0rc1__tar.gz

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-cncf-kubernetes
-Version: 10.7.0
+Version: 10.8.0rc1
 Summary: Provider package apache-airflow-providers-cncf-kubernetes for Apache Airflow
 Keywords: airflow-provider,cncf.kubernetes,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -21,14 +21,14 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: aiofiles>=23.2.0
-Requires-Dist: apache-airflow>=2.10.0
+Requires-Dist: apache-airflow>=2.10.0rc1
 Requires-Dist: asgiref>=3.5.2
 Requires-Dist: cryptography>=41.0.0
 Requires-Dist: kubernetes>=32.0.0,<33.0.0
 Requires-Dist: kubernetes_asyncio>=32.0.0,<33.0.0
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -59,9 +59,8 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 
 Package ``apache-airflow-providers-cncf-kubernetes``
 
-Release: ``10.7.0``
+Release: ``10.8.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `Kubernetes <https://kubernetes.io/>`__
 
@@ -73,12 +72,12 @@ This is a provider package for ``cncf.kubernetes`` provider. All classes for thi
 are in ``airflow.providers.cncf.kubernetes`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-cncf-kubernetes``
 
@@ -99,5 +98,5 @@ PIP package             Version required
 ======================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/changelog.html>`_.
 

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/README.rst

@@ -23,9 +23,8 @@
 
 Package ``apache-airflow-providers-cncf-kubernetes``
 
-Release: ``10.7.0``
+Release: ``10.8.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `Kubernetes <https://kubernetes.io/>`__
 
@@ -37,12 +36,12 @@ This is a provider package for ``cncf.kubernetes`` provider. All classes for thi
 are in ``airflow.providers.cncf.kubernetes`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-cncf-kubernetes``
 
@@ -63,4 +62,4 @@ PIP package             Version required
 ======================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/changelog.html>`_.

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/docs/changelog.rst

@@ -28,6 +28,51 @@ Changelog
 ---------
 
 
+10.8.0
+......
+
+
+Release Date: ``|PypiReleaseDate|``
+
+Features
+~~~~~~~~
+
+* ``Add customizable log prefix and formatter to KubernetesPodOperator (#53598)``
+* ``Enhanced Pod Failure Logging in KubernetesExecutor  (#54115)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``fix: Handle Kubernetes API responses with non-JSON bodies (#55107)``
+* ``Fix KubernetesExecutor retry on 409 conflict (#54777)``
+* ``KubernetesPodOperator logs pod and container state after error occurred (#54828)``
+* ``Throttle HTTPError during consume pod logs (#54761)``
+
+Misc
+~~~~
+
+* ``Remove airflow.models.DAG (#54383)``
+* ``Refactor Kubernetes Executor Types to NamedTuples (#54858)``
+* ``Move some items in 'airflow.utils.context' to appropriate places (#53600)``
+
+Doc-only
+~~~~~~~~
+
+* ``Make term Dag consistent in providers docs (#55101)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Remove Legacy Query Object Usage from airflow-models (#52325)``
+   * ``Fix k8s pod yaml test to expect redacted sensitive data (#54883)``
+   * ``Move secrets_masker over to airflow_shared distribution (#54449)``
+   * ``Replace API server's direct Connection access workaround in BaseHook (#54083)``
+   * ``Switch pre-commit to prek (#54258)``
+   * ``make bundle_name not nullable (#47592)``
+   * ``Bump pytest-asyncio to 1.1.0 (#54123)``
+
+.. Review and move the new changes to one of the sections above:
+   * ``Fix Airflow 2 reference in README/index of providers (#55240)``
+
 10.7.0
 ......
 

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/docs/index.rst

@@ -66,7 +66,7 @@
     :maxdepth: 1
     :caption: Resources
 
-    Example DAGs <https://github.com/apache/airflow/tree/providers-cncf-kubernetes/|version|/providers/cncf/kubernetes/tests/system/cncf/kubernetes>
+    Example Dags <https://github.com/apache/airflow/tree/providers-cncf-kubernetes/|version|/providers/cncf/kubernetes/tests/system/cncf/kubernetes>
     PyPI Repository <https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/>
     Installing from sources <installing-providers-from-sources>
 
@@ -87,9 +87,7 @@ apache-airflow-providers-cncf-kubernetes package
 `Kubernetes <https://kubernetes.io/>`__
 
 
-Release: 10.7.0
-
-Release Date: ``|PypiReleaseDate|``
+Release: 10.8.0
 
 Provider package
 ----------------
@@ -100,7 +98,7 @@ All classes for this package are included in the ``airflow.providers.cncf.kubern
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation via
+You can install this package on top of an existing Airflow installation via
 ``pip install apache-airflow-providers-cncf-kubernetes``.
 For the minimum Airflow version supported, see ``Requirements`` below.
 

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/docs/kubernetes_executor.rst

@@ -36,7 +36,7 @@ not necessarily need to be running on Kubernetes, but does need access to a Kube
 
 KubernetesExecutor requires a non-sqlite database in the backend.
 
-When a DAG submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. The worker pod then runs the task, reports the result, and terminates.
+When a Dag submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. The worker pod then runs the task, reports the result, and terminates.
 
 .. image:: img/arch-diag-kubernetes.png
 
@@ -45,7 +45,7 @@ One example of an Airflow deployment running on a distributed set of five nodes
 
 .. image:: img/arch-diag-kubernetes2.png
 
-Consistent with the regular Airflow architecture, the Workers need access to the DAG files to execute the tasks within those DAGs and interact with the Metadata repository. Also, configuration information specific to the Kubernetes Executor, such as the worker namespace and image information, needs to be specified in the Airflow Configuration file.
+Consistent with the regular Airflow architecture, the Workers need access to the Dag files to execute the tasks within those Dags and interact with the Metadata repository. Also, configuration information specific to the Kubernetes Executor, such as the worker namespace and image information, needs to be specified in the Airflow Configuration file.
 
 Additionally, the Kubernetes Executor enables specification of additional features on a per-task basis using the Executor config.
 
@@ -103,24 +103,24 @@ With these requirements in mind, here are some examples of basic ``pod_template_
 
     The examples below should work when using default Airflow configuration values. However, many custom
     configuration values need to be explicitly passed to the pod via this template too. This includes,
-    but is not limited to, sql configuration, required Airflow connections, DAGs folder path and
+    but is not limited to, sql configuration, required Airflow connections, Dags folder path and
     logging settings. See :doc:`../../configurations-ref` for details.
 
-Storing DAGs in the image:
+Storing Dags in the image:
 
 .. literalinclude:: /../src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
     :language: yaml
     :start-after: [START template_with_dags_in_image]
     :end-before: [END template_with_dags_in_image]
 
-Storing DAGs in a ``persistentVolume``:
+Storing Dags in a ``persistentVolume``:
 
 .. literalinclude:: /../src/airflow/providers/cncf/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
     :language: yaml
     :start-after: [START template_with_dags_in_volume]
     :end-before: [END template_with_dags_in_volume]
 
-Pulling DAGs from ``git``:
+Pulling Dags from ``git``:
 
 .. literalinclude:: /../src/airflow/providers/cncf/kubernetes/pod_template_file_examples/git_sync_template.yaml
     :language: yaml
@@ -165,14 +165,14 @@ Here is an example of a task with both features:
 
     import pendulum
 
-    from airflow import DAG
+    from airflow import Dag
     from airflow.decorators import task
     from airflow.example_dags.libs.helper import print_stuff
     from airflow.settings import AIRFLOW_HOME
 
     from kubernetes.client import models as k8s
 
-    with DAG(
+    with Dag(
         dag_id="example_pod_template_file",
         schedule=None,
         start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
@@ -189,18 +189,18 @@ Here is an example of a task with both features:
             print_stuff()
 
 
-Managing DAGs and logs
+Managing Dags and logs
 ~~~~~~~~~~~~~~~~~~~~~~
 
 Use of persistent volumes is optional and depends on your configuration.
 
 - **Dags**:
 
-To get the DAGs into the workers, you can:
+To get the Dags into the workers, you can:
 
-  - Include DAGs in the image.
-  - Use ``git-sync`` which, before starting the worker container, will run a ``git pull`` of the DAGs repository.
-  - Storing DAGs on a persistent volume, which can be mounted on all workers.
+  - Include Dags in the image.
+  - Use ``git-sync`` which, before starting the worker container, will run a ``git pull`` of the Dags repository.
+  - Storing Dags on a persistent volume, which can be mounted on all workers.
 
 - **Logs**:
 

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/docs/operators.rst

@@ -111,7 +111,7 @@ Difference between ``KubernetesPodOperator`` and Kubernetes object spec
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 The :class:`~airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator` can be considered
 a substitute for a Kubernetes object spec definition that is able
-to be run in the Airflow scheduler in the DAG context. If using the operator, there is no need to create the
+to be run in the Airflow scheduler in the Dag context. If using the operator, there is no need to create the
 equivalent YAML/JSON object spec for the Pod you would like to run.
 The YAML file can still be provided with the ``pod_template_file`` or even the Pod Spec constructed in Python via
 the ``full_pod_spec`` parameter which requires a Kubernetes ``V1Pod``.

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/provider.yaml

@@ -22,12 +22,13 @@ description: |
     `Kubernetes <https://kubernetes.io/>`__
 
 state: ready
-source-date-epoch: 1754503046
+source-date-epoch: 1756876671
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 10.8.0
   - 10.7.0
   - 10.6.2
   - 10.6.1

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-cncf-kubernetes"
-version = "10.7.0"
+version = "10.8.0rc1"
 description = "Provider package apache-airflow-providers-cncf-kubernetes for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -54,11 +54,11 @@ requires-python = ">=3.10"
 
 # The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
-# Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
+# Make sure to run ``prek update-providers-dependencies --all-files``
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
     "aiofiles>=23.2.0",
-    "apache-airflow>=2.10.0",
+    "apache-airflow>=2.10.0rc1",
     "asgiref>=3.5.2",
     "cryptography>=41.0.0",
     # The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
@@ -108,8 +108,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.7.0/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.8.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/src/airflow/providers/cncf/kubernetes/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "10.7.0"
+__version__ = "10.8.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/src/airflow/providers/cncf/kubernetes/cli/kubernetes_command.py

@@ -32,12 +32,16 @@ from airflow.providers.cncf.kubernetes.executors.kubernetes_executor import Kube
 from airflow.providers.cncf.kubernetes.kube_client import get_kube_client
 from airflow.providers.cncf.kubernetes.kubernetes_helper_functions import create_unique_id
 from airflow.providers.cncf.kubernetes.pod_generator import PodGenerator, generate_pod_command_args
-from airflow.providers.cncf.kubernetes.version_compat import AIRFLOW_V_3_0_PLUS
+from airflow.providers.cncf.kubernetes.version_compat import AIRFLOW_V_3_0_PLUS, AIRFLOW_V_3_1_PLUS
 from airflow.utils import cli as cli_utils, yaml
-from airflow.utils.cli import get_dag
 from airflow.utils.providers_configuration_loader import providers_configuration_loaded
 from airflow.utils.types import DagRunType
 
+if AIRFLOW_V_3_1_PLUS:
+    from airflow.utils.cli import get_bagged_dag
+else:
+    from airflow.utils.cli import get_dag as get_bagged_dag  # type: ignore[attr-defined,no-redef]
+
 
 @cli_utils.action_cli
 @providers_configuration_loaded
@@ -45,9 +49,9 @@ def generate_pod_yaml(args):
     """Generate yaml files for each task in the DAG. Used for testing output of KubernetesExecutor."""
     logical_date = args.logical_date if AIRFLOW_V_3_0_PLUS else args.execution_date
     if AIRFLOW_V_3_0_PLUS:
-        dag = get_dag(bundle_names=args.bundle_name, dag_id=args.dag_id)
+        dag = get_bagged_dag(bundle_names=args.bundle_name, dag_id=args.dag_id)
     else:
-        dag = get_dag(subdir=args.subdir, dag_id=args.dag_id)
+        dag = get_bagged_dag(subdir=args.subdir, dag_id=args.dag_id)
     yaml_output_path = args.output_path
 
     dm = DagModel(dag_id=dag.dag_id)

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/src/airflow/providers/cncf/kubernetes/decorators/kubernetes_cmd.py

@@ -24,9 +24,9 @@ from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperato
 from airflow.providers.cncf.kubernetes.version_compat import (
     DecoratedOperator,
     TaskDecorator,
+    context_merge,
     task_decorator_factory,
 )
-from airflow.utils.context import context_merge
 from airflow.utils.operator_helpers import determine_kwargs
 
 if TYPE_CHECKING:

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py

@@ -66,6 +66,8 @@ from airflow.providers.cncf.kubernetes.exceptions import PodMutationHookExceptio
 from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import (
     ADOPTED,
     POD_EXECUTOR_DONE_KEY,
+    KubernetesJob,
+    KubernetesResults,
 )
 from airflow.providers.cncf.kubernetes.kube_config import KubeConfig
 from airflow.providers.cncf.kubernetes.kubernetes_helper_functions import annotations_to_key
@@ -85,10 +87,6 @@ if TYPE_CHECKING:
     from airflow.executors import workloads
     from airflow.models.taskinstance import TaskInstance
     from airflow.models.taskinstancekey import TaskInstanceKey
-    from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_types import (
-        KubernetesJobType,
-        KubernetesResultsType,
-    )
     from airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils import (
         AirflowKubernetesScheduler,
     )
@@ -156,8 +154,8 @@ class KubernetesExecutor(BaseExecutor):
     def __init__(self):
         self.kube_config = KubeConfig()
         self._manager = multiprocessing.Manager()
-        self.task_queue: Queue[KubernetesJobType] = self._manager.Queue()
-        self.result_queue: Queue[KubernetesResultsType] = self._manager.Queue()
+        self.task_queue: Queue[KubernetesJob] = self._manager.Queue()
+        self.result_queue: Queue[KubernetesResults] = self._manager.Queue()
         self.kube_scheduler: AirflowKubernetesScheduler | None = None
         self.kube_client: client.CoreV1Api | None = None
         self.scheduler_job_id: str | None = None
@@ -279,7 +277,7 @@ class KubernetesExecutor(BaseExecutor):
         else:
             pod_template_file = None
         self.event_buffer[key] = (TaskInstanceState.QUEUED, self.scheduler_job_id)
-        self.task_queue.put((key, command, kube_executor_config, pod_template_file))
+        self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))
         # We keep a temporary local record that we've handled this so we don't
         # try and remove it from the QUEUED state while we process it
         self.last_handled[key] = time.time()
@@ -330,17 +328,16 @@ class KubernetesExecutor(BaseExecutor):
             while True:
                 results = self.result_queue.get_nowait()
                 try:
-                    key, state, pod_name, namespace, resource_version = results
-                    last_resource_version[namespace] = resource_version
-                    self.log.info("Changing state of %s to %s", results, state)
+                    last_resource_version[results.namespace] = results.resource_version
+                    self.log.info("Changing state of %s to %s", results, results.state)
                     try:
-                        self._change_state(key, state, pod_name, namespace)
+                        self._change_state(results)
                     except Exception as e:
                         self.log.exception(
                             "Exception: %s when attempting to change state of %s to %s, re-queueing.",
                             e,
                             results,
-                            state,
+                            results.state,
                         )
                         self.result_queue.put(results)
                 finally:
@@ -361,7 +358,7 @@ class KubernetesExecutor(BaseExecutor):
                 task = self.task_queue.get_nowait()
 
                 try:
-                    key, command, kube_executor_config, pod_template_file = task
+                    key = task.key
                     self.kube_scheduler.run_next(task)
                     self.task_publish_retries.pop(key, None)
                 except PodReconciliationError as e:
@@ -371,31 +368,41 @@ class KubernetesExecutor(BaseExecutor):
                     )
                     self.fail(task[0], e)
                 except ApiException as e:
-                    body = json.loads(e.body)
+                    try:
+                        if e.body:
+                            body = json.loads(e.body)
+                        else:
+                            # If no body content, use reason as the message
+                            body = {"message": e.reason}
+                    except (json.JSONDecodeError, ValueError, TypeError):
+                        # If the body is a string (e.g., in a 429 error), it can't be parsed as JSON.
+                        # Use the body directly as the message instead.
+                        body = {"message": e.body}
+
                     retries = self.task_publish_retries[key]
-                    # In case of exceeded quota errors, requeue the task as per the task_publish_max_retries
+                    # In case of exceeded quota or conflict errors, requeue the task as per the task_publish_max_retries
+                    message = body.get("message", "")
                     if (
-                        str(e.status) == "403"
-                        and "exceeded quota" in body["message"]
-                        and (self.task_publish_max_retries == -1 or retries < self.task_publish_max_retries)
-                    ):
+                        (str(e.status) == "403" and "exceeded quota" in message)
+                        or (str(e.status) == "409" and "object has been modified" in message)
+                    ) and (self.task_publish_max_retries == -1 or retries < self.task_publish_max_retries):
                         self.log.warning(
                             "[Try %s of %s] Kube ApiException for Task: (%s). Reason: %r. Message: %s",
                             self.task_publish_retries[key] + 1,
                             self.task_publish_max_retries,
                             key,
                             e.reason,
-                            body["message"],
+                            message,
                         )
                         self.task_queue.put(task)
                         self.task_publish_retries[key] = retries + 1
                     else:
                         self.log.error("Pod creation failed with reason %r. Failing task", e.reason)
-                        key, _, _, _ = task
+                        key = task.key
                         self.fail(key, e)
                         self.task_publish_retries.pop(key, None)
                 except PodMutationHookException as e:
-                    key, _, _, _ = task
+                    key = task.key
                     self.log.error(
                         "Pod Mutation Hook failed for the task %s. Failing task. Details: %s",
                         key,
@@ -408,15 +415,56 @@ class KubernetesExecutor(BaseExecutor):
     @provide_session
     def _change_state(
         self,
-        key: TaskInstanceKey,
-        state: TaskInstanceState | str | None,
-        pod_name: str,
-        namespace: str,
+        results: KubernetesResults,
         session: Session = NEW_SESSION,
     ) -> None:
+        """Change state of the task based on KubernetesResults."""
         if TYPE_CHECKING:
             assert self.kube_scheduler
 
+        key = results.key
+        state = results.state
+        pod_name = results.pod_name
+        namespace = results.namespace
+        failure_details = results.failure_details
+
+        if state == TaskInstanceState.FAILED:
+            # Use pre-collected failure details from the watcher to avoid additional API calls
+            if failure_details:
+                pod_status = failure_details.get("pod_status")
+                pod_reason = failure_details.get("pod_reason")
+                pod_message = failure_details.get("pod_message")
+                container_state = failure_details.get("container_state")
+                container_reason = failure_details.get("container_reason")
+                container_message = failure_details.get("container_message")
+                exit_code = failure_details.get("exit_code")
+                container_type = failure_details.get("container_type")
+                container_name = failure_details.get("container_name")
+
+                task_key_str = f"{key.dag_id}.{key.task_id}.{key.try_number}"
+                self.log.warning(
+                    "Task %s failed in pod %s/%s. Pod phase: %s, reason: %s, message: %s, "
+                    "container_type: %s, container_name: %s, container_state: %s, container_reason: %s, "
+                    "container_message: %s, exit_code: %s",
+                    task_key_str,
+                    namespace,
+                    pod_name,
+                    pod_status,
+                    pod_reason,
+                    pod_message,
+                    container_type,
+                    container_name,
+                    container_state,
+                    container_reason,
+                    container_message,
+                    exit_code,
+                )
+            else:
+                task_key_str = f"{key.dag_id}.{key.task_id}.{key.try_number}"
+                self.log.warning(
+                    "Task %s failed in pod %s/%s (no details available)", task_key_str, namespace, pod_name
+                )
+
         if state == ADOPTED:
             # When the task pod is adopted by another executor,
             # then remove the task from the current executor running queue.
@@ -696,18 +744,20 @@ class KubernetesExecutor(BaseExecutor):
                 results = self.result_queue.get_nowait()
                 self.log.warning("Executor shutting down, flushing results=%s", results)
                 try:
-                    key, state, pod_name, namespace, resource_version = results
                     self.log.info(
-                        "Changing state of %s to %s : resource_version=%d", results, state, resource_version
+                        "Changing state of %s to %s : resource_version=%s",
+                        results,
+                        results.state,
+                        results.resource_version,
                     )
                     try:
-                        self._change_state(key, state, pod_name, namespace)
+                        self._change_state(results)
                     except Exception as e:
                         self.log.exception(
                             "Ignoring exception: %s when attempting to change state of %s to %s.",
                             e,
                             results,
-                            state,
+                            results.state,
                         )
                 finally:
                     self.result_queue.task_done()

{apache_airflow_providers_cncf_kubernetes-10.7.0 → apache_airflow_providers_cncf_kubernetes-10.8.0rc1}/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_types.py

@@ -16,26 +16,66 @@
 # under the License.
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Any, Literal, NamedTuple, TypedDict
 
-ADOPTED = "adopted"
 if TYPE_CHECKING:
     from collections.abc import Sequence
 
     from airflow.models.taskinstance import TaskInstanceKey
     from airflow.utils.state import TaskInstanceState
 
-    # TODO: Remove after Airflow 2 support is removed
-    CommandType = Sequence[str]
 
-    # TaskInstance key, command, configuration, pod_template_file
-    KubernetesJobType = tuple[TaskInstanceKey, CommandType, Any, str | None]
+ADOPTED = "adopted"
+
+
+class FailureDetails(TypedDict, total=False):
+    """Detailed information about pod/container failure."""
+
+    pod_status: str | None
+    pod_reason: str | None
+    pod_message: str | None
+    container_state: str | None
+    container_reason: str | None
+    container_message: str | None
+    exit_code: int | None
+    container_type: Literal["init", "main"] | None
+    container_name: str | None
+
+
+class KubernetesResults(NamedTuple):
+    """Results from Kubernetes task execution."""
+
+    key: TaskInstanceKey
+    state: TaskInstanceState | str | None
+    pod_name: str
+    namespace: str
+    resource_version: str
+    failure_details: FailureDetails | None
+
+
+class KubernetesWatch(NamedTuple):
+    """Watch event data from Kubernetes pods."""
+
+    pod_name: str
+    namespace: str
+    state: TaskInstanceState | str | None
+    annotations: dict[str, str]
+    resource_version: str
+    failure_details: FailureDetails | None
+
+
+# TODO: Remove after Airflow 2 support is removed
+CommandType = "Sequence[str]"
+
+
+class KubernetesJob(NamedTuple):
+    """Job definition for Kubernetes execution."""
 
-    # key, pod state, pod_name, namespace, resource_version
-    KubernetesResultsType = tuple[TaskInstanceKey, TaskInstanceState | str | None, str, str, str]
+    key: TaskInstanceKey
+    command: Sequence[str]
+    kube_executor_config: Any
+    pod_template_file: str | None
 
-    # pod_name, namespace, pod state, annotations, resource_version
-    KubernetesWatchType = tuple[str, str, TaskInstanceState | str | None, dict[str, str], str]
 
 ALL_NAMESPACES = "ALL_NAMESPACES"
 POD_EXECUTOR_DONE_KEY = "airflow_executor_done"