apache-airflow-providers-elasticsearch 6.0.0rc2__tar.gz → 6.2.0rc1__tar.gz

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: apache-airflow-providers-elasticsearch
-Version: 6.0.0rc2
+Version: 6.2.0rc1
 Summary: Provider package apache-airflow-providers-elasticsearch for Apache Airflow
 Keywords: airflow-provider,elasticsearch,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -20,35 +20,18 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
-Requires-Dist: apache-airflow-providers-common-sql>=1.20.0rc0
 Requires-Dist: apache-airflow>=2.9.0rc0
+Requires-Dist: apache-airflow-providers-common-sql>=1.20.0rc0
 Requires-Dist: elasticsearch>=8.10,<9
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://x.com/ApacheAirflow
 Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 
 
-.. Licensed to the Apache Software Foundation (ASF) under one
-   or more contributor license agreements.  See the NOTICE file
-   distributed with this work for additional information
-   regarding copyright ownership.  The ASF licenses this file
-   to you under the Apache License, Version 2.0 (the
-   "License"); you may not use this file except in compliance
-   with the License.  You may obtain a copy of the License at
-
-..   http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-   software distributed under the License is distributed on an
-   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-   KIND, either express or implied.  See the License for the
-   specific language governing permissions and limitations
-   under the License.
-
  .. Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
@@ -66,8 +49,7 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
     specific language governing permissions and limitations
     under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-    OVERWRITTEN WHEN PREPARING PACKAGES.
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
  .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
     `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -75,7 +57,7 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 
 Package ``apache-airflow-providers-elasticsearch``
 
-Release: ``6.0.0.rc2``
+Release: ``6.2.0``
 
 
 `Elasticsearch <https://www.elastic.co/elasticsearch>`__
@@ -88,7 +70,7 @@ This is a provider package for ``elasticsearch`` provider. All classes for this
 are in ``airflow.providers.elasticsearch`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/>`_.
 
 Installation
 ------------
@@ -130,4 +112,5 @@ Dependent package
 ============================================================================================================  ==============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/changelog.html>`_.
+

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1}/README.rst

@@ -1,21 +1,4 @@
 
-.. Licensed to the Apache Software Foundation (ASF) under one
-   or more contributor license agreements.  See the NOTICE file
-   distributed with this work for additional information
-   regarding copyright ownership.  The ASF licenses this file
-   to you under the Apache License, Version 2.0 (the
-   "License"); you may not use this file except in compliance
-   with the License.  You may obtain a copy of the License at
-
-..   http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-   software distributed under the License is distributed on an
-   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-   KIND, either express or implied.  See the License for the
-   specific language governing permissions and limitations
-   under the License.
-
  .. Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
@@ -33,8 +16,7 @@
     specific language governing permissions and limitations
     under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-    OVERWRITTEN WHEN PREPARING PACKAGES.
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
  .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
     `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -42,7 +24,7 @@
 
 Package ``apache-airflow-providers-elasticsearch``
 
-Release: ``6.0.0.rc2``
+Release: ``6.2.0``
 
 
 `Elasticsearch <https://www.elastic.co/elasticsearch>`__
@@ -55,7 +37,7 @@ This is a provider package for ``elasticsearch`` provider. All classes for this
 are in ``airflow.providers.elasticsearch`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/>`_.
 
 Installation
 ------------
@@ -97,4 +79,4 @@ Dependent package
 ============================================================================================================  ==============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/changelog.html>`_.

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1}/pyproject.toml

@@ -1,4 +1,3 @@
-
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -16,10 +15,9 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-# OVERWRITTEN WHEN PREPARING PACKAGES.
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
-# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# IF YOU WANT TO MODIFY THIS FILE EXCEPT DEPENDENCIES, YOU SHOULD MODIFY THE TEMPLATE
 # `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
 [build-system]
 requires = ["flit_core==3.10.1"]
@@ -27,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-elasticsearch"
-version = "6.0.0.rc2"
+version = "6.2.0.rc1"
 description = "Provider package apache-airflow-providers-elasticsearch for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -53,15 +51,18 @@ classifiers = [
     "Topic :: System :: Monitoring",
 ]
 requires-python = "~=3.9"
+
+# The dependencies should be modified in place in the generated file
+# Any change in the dependencies is preserved when the file is regenerated
 dependencies = [
-    "apache-airflow-providers-common-sql>=1.20.0rc0",
     "apache-airflow>=2.9.0rc0",
+    "apache-airflow-providers-common-sql>=1.20.0rc0",
     "elasticsearch>=8.10,<9",
 ]
 
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.0.0/changelog.html"
+"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0"
+"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/6.2.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"
@@ -73,3 +74,6 @@ provider_info = "airflow.providers.elasticsearch.get_provider_info:get_provider_
 
 [tool.flit.module]
 name = "airflow.providers.elasticsearch"
+
+[tool.pytest.ini_options]
+ignore = "tests/system/"

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1/src}/airflow/providers/elasticsearch/LICENSE

@@ -199,55 +199,3 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-
-============================================================================
-   APACHE AIRFLOW SUBCOMPONENTS:
-
-   The Apache Airflow project contains subcomponents with separate copyright
-   notices and license terms. Your use of the source code for the these
-   subcomponents is subject to the terms and conditions of the following
-   licenses.
-
-
-========================================================================
-Third party Apache 2.0 licenses
-========================================================================
-
-The following components are provided under the Apache 2.0 License.
-See project link for details. The text of each license is also included
-at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (ALv2 License) hue v4.3.0 (https://github.com/cloudera/hue/)
-    (ALv2 License) jqclock v2.3.0 (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
-    (ALv2 License) bootstrap3-typeahead v4.0.2 (https://github.com/bassjobsen/Bootstrap-3-Typeahead)
-    (ALv2 License) connexion v2.7.0 (https://github.com/zalando/connexion)
-
-========================================================================
-MIT licenses
-========================================================================
-
-The following components are provided under the MIT License. See project link for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (MIT License) jquery v3.5.1 (https://jquery.org/license/)
-    (MIT License) dagre-d3 v0.6.4 (https://github.com/cpettitt/dagre-d3)
-    (MIT License) bootstrap v3.4.1 (https://github.com/twbs/bootstrap/)
-    (MIT License) d3-tip v0.9.1 (https://github.com/Caged/d3-tip)
-    (MIT License) dataTables v1.10.25 (https://datatables.net)
-    (MIT License) normalize.css v3.0.2 (http://necolas.github.io/normalize.css/)
-    (MIT License) ElasticMock v1.3.2 (https://github.com/vrcmarcos/elasticmock)
-    (MIT License) MomentJS v2.24.0 (http://momentjs.com/)
-    (MIT License) eonasdan-bootstrap-datetimepicker v4.17.49 (https://github.com/eonasdan/bootstrap-datetimepicker/)
-
-========================================================================
-BSD 3-Clause licenses
-========================================================================
-The following components are provided under the BSD 3-Clause license. See project links for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (BSD 3 License) d3 v5.16.0 (https://d3js.org)
-    (BSD 3 License) d3-shape v2.1.0 (https://github.com/d3/d3-shape)
-    (BSD 3 License) cgroupspy 0.2.1 (https://github.com/cloudsigma/cgroupspy)
-
-========================================================================
-See 3rd-party-licenses/LICENSES-ui.txt for packages used in `/airflow/www`

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1/src}/airflow/providers/elasticsearch/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "6.0.0"
+__version__ = "6.2.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.9.0"

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1/src}/airflow/providers/elasticsearch/get_provider_info.py

@@ -15,8 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-# OVERWRITTEN WHEN PREPARING PACKAGES.
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 #
 # IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
 # `get_provider_info_TEMPLATE.py.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -28,8 +27,9 @@ def get_provider_info():
         "name": "Elasticsearch",
         "description": "`Elasticsearch <https://www.elastic.co/elasticsearch>`__\n",
         "state": "ready",
-        "source-date-epoch": 1734533734,
+        "source-date-epoch": 1739959618,
         "versions": [
+            "6.2.0",
             "6.0.0",
             "5.5.3",
             "5.5.2",
@@ -75,16 +75,11 @@ def get_provider_info():
             "1.0.1",
             "1.0.0",
         ],
-        "dependencies": [
-            "apache-airflow>=2.9.0",
-            "apache-airflow-providers-common-sql>=1.20.0",
-            "elasticsearch>=8.10,<9",
-        ],
         "integrations": [
             {
                 "integration-name": "Elasticsearch",
                 "external-doc-url": "https://www.elastic.co/elasticsearch",
-                "logo": "/integration-logos/elasticsearch/Elasticsearch.png",
+                "logo": "/docs/integration-logos/Elasticsearch.png",
                 "tags": ["software"],
             }
         ],
@@ -141,6 +136,20 @@ def get_provider_info():
                         "example": None,
                         "default": "False",
                     },
+                    "write_to_es": {
+                        "description": "Write the task logs to the ElasticSearch\n",
+                        "version_added": "5.5.4",
+                        "type": "string",
+                        "example": None,
+                        "default": "False",
+                    },
+                    "target_index": {
+                        "description": "Name of the index to write to, when enabling writing the task logs to the ElasticSearch\n",
+                        "version_added": "5.5.4",
+                        "type": "string",
+                        "example": None,
+                        "default": "airflow-logs",
+                    },
                     "json_format": {
                         "description": "Instead of the default log formatter, write the log lines as JSON\n",
                         "version_added": "1.10.4",
@@ -205,4 +214,9 @@ def get_provider_info():
                 },
             },
         },
+        "dependencies": [
+            "apache-airflow>=2.9.0",
+            "apache-airflow-providers-common-sql>=1.20.0",
+            "elasticsearch>=8.10,<9",
+        ],
     }

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1/src}/airflow/providers/elasticsearch/hooks/elasticsearch.py

@@ -17,14 +17,14 @@
 # under the License.
 from __future__ import annotations
 
+from collections.abc import Iterable, Mapping
 from functools import cached_property
 from typing import TYPE_CHECKING, Any
 from urllib import parse
 
-from elasticsearch import Elasticsearch
-
 from airflow.hooks.base import BaseHook
 from airflow.providers.common.sql.hooks.sql import DbApiHook
+from elasticsearch import Elasticsearch
 
 if TYPE_CHECKING:
     from elastic_transport import ObjectApiResponse
@@ -43,6 +43,73 @@ def connect(
     return ESConnection(host, port, user, password, scheme, **kwargs)
 
 
+class ElasticsearchSQLCursor:
+    """A PEP 249-like Cursor class for Elasticsearch SQL API."""
+
+    def __init__(self, es: Elasticsearch, **kwargs):
+        self.es = es
+        self.body = {
+            "fetch_size": kwargs.get("fetch_size", 1000),
+            "field_multi_value_leniency": kwargs.get("field_multi_value_leniency", False),
+        }
+        self._response: ObjectApiResponse | None = None
+
+    @property
+    def response(self) -> ObjectApiResponse:
+        return self._response or {}  # type: ignore
+
+    @response.setter
+    def response(self, value):
+        self._response = value
+
+    @property
+    def cursor(self):
+        return self.response.get("cursor")
+
+    @property
+    def rows(self):
+        return self.response.get("rows", [])
+
+    @property
+    def rowcount(self) -> int:
+        return len(self.rows)
+
+    @property
+    def description(self) -> list[tuple]:
+        return [(column["name"], column["type"]) for column in self.response.get("columns", [])]
+
+    def execute(
+        self, statement: str, params: Iterable | Mapping[str, Any] | None = None
+    ) -> ObjectApiResponse:
+        self.body["query"] = statement
+        if params:
+            self.body["params"] = params
+        self.response = self.es.sql.query(body=self.body)
+        if self.cursor:
+            self.body["cursor"] = self.cursor
+        else:
+            self.body.pop("cursor", None)
+        return self.response
+
+    def fetchone(self):
+        if self.rows:
+            return self.rows[0]
+        return None
+
+    def fetchmany(self, size: int | None = None):
+        raise NotImplementedError()
+
+    def fetchall(self):
+        results = self.rows
+        while self.cursor:
+            self.execute(statement=self.body["query"])
+            results.extend(self.rows)
+        return results
+
+    def close(self):
+        self._response = None
+
+
 class ESConnection:
     """wrapper class for elasticsearch.Elasticsearch."""
 
@@ -68,9 +135,19 @@ class ESConnection:
         else:
             self.es = Elasticsearch(self.url, **self.kwargs)
 
-    def execute_sql(self, query: str) -> ObjectApiResponse:
-        sql_query = {"query": query}
-        return self.es.sql.query(body=sql_query)
+    def cursor(self) -> ElasticsearchSQLCursor:
+        return ElasticsearchSQLCursor(self.es, **self.kwargs)
+
+    def close(self):
+        self.es.close()
+
+    def commit(self):
+        pass
+
+    def execute_sql(
+        self, query: str, params: Iterable | Mapping[str, Any] | None = None
+    ) -> ObjectApiResponse:
+        return self.cursor().execute(query, params)
 
 
 class ElasticsearchSQLHook(DbApiHook):
@@ -85,13 +162,13 @@ class ElasticsearchSQLHook(DbApiHook):
 
     conn_name_attr = "elasticsearch_conn_id"
     default_conn_name = "elasticsearch_default"
+    connector = ESConnection
     conn_type = "elasticsearch"
     hook_name = "Elasticsearch"
 
     def __init__(self, schema: str = "http", connection: AirflowConnection | None = None, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.schema = schema
-        self._connection = connection
 
     def get_conn(self) -> ESConnection:
         """Return an elasticsearch connection object."""
@@ -105,11 +182,10 @@ class ElasticsearchSQLHook(DbApiHook):
             "scheme": conn.schema or "http",
         }
 
-        if conn.extra_dejson.get("http_compress", False):
-            conn_args["http_compress"] = bool(["http_compress"])
+        conn_args.update(conn.extra_dejson)
 
-        if conn.extra_dejson.get("timeout", False):
-            conn_args["timeout"] = conn.extra_dejson["timeout"]
+        if conn_args.get("http_compress", False):
+            conn_args["http_compress"] = bool(conn_args["http_compress"])
 
         return connect(**conn_args)
 

{apache_airflow_providers_elasticsearch-6.0.0rc2 → apache_airflow_providers_elasticsearch-6.2.0rc1/src}/airflow/providers/elasticsearch/log/es_task_handler.py

@@ -19,7 +19,11 @@ from __future__ import annotations
 
 import contextlib
 import inspect
+import json
 import logging
+import os
+import pathlib
+import shutil
 import sys
 import time
 from collections import defaultdict
@@ -27,11 +31,10 @@ from operator import attrgetter
 from typing import TYPE_CHECKING, Any, Callable, Literal
 from urllib.parse import quote, urlparse
 
-# Using `from elasticsearch import *` would break elasticsearch mocking used in unit test.
-import elasticsearch
 import pendulum
-from elasticsearch.exceptions import NotFoundError
 
+# Using `from elasticsearch import *` would break elasticsearch mocking used in unit test.
+import elasticsearch
 from airflow.configuration import conf
 from airflow.exceptions import AirflowException
 from airflow.models.dagrun import DagRun
@@ -43,6 +46,8 @@ from airflow.utils.log.file_task_handler import FileTaskHandler
 from airflow.utils.log.logging_mixin import ExternalLoggingMixin, LoggingMixin
 from airflow.utils.module_loading import import_string
 from airflow.utils.session import create_session
+from elasticsearch import helpers
+from elasticsearch.exceptions import NotFoundError
 
 if TYPE_CHECKING:
     from datetime import datetime
@@ -106,10 +111,13 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
     """
     ElasticsearchTaskHandler is a python log handler that reads logs from Elasticsearch.
 
-    Note that Airflow does not handle the indexing of logs into Elasticsearch. Instead,
+    Note that Airflow by default does not handle the indexing of logs into Elasticsearch. Instead,
     Airflow flushes logs into local files. Additional software setup is required to index
     the logs into Elasticsearch, such as using Filebeat and Logstash.
 
+    Airflow can be configured to support directly writing logging to Elasticsearch. To enable this feature,
+    set `json_format` and `write_to_es` to `True`.
+
     To efficiently query and sort Elasticsearch results, this handler assumes each
     log message has a field `log_id` consists of ti primary keys:
     `log_id = {dag_id}-{task_id}-{logical_date}-{try_number}`
@@ -136,6 +144,8 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
         write_stdout: bool,
         json_format: bool,
         json_fields: str,
+        write_to_es: bool = False,
+        target_index: str = "airflow-logs",
         host_field: str = "host",
         offset_field: str = "offset",
         host: str = "http://localhost:9200",
@@ -166,6 +176,11 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
         self.index_patterns = index_patterns
         self.index_patterns_callable = index_patterns_callable
         self.context_set = False
+        self.write_to_es = write_to_es
+        self.target_index = target_index
+        self.delete_local_copy = kwargs.get(
+            "delete_local_copy", conf.getboolean("logging", "delete_local_logs")
+        )
 
         self.formatter: logging.Formatter
         self.handler: logging.FileHandler | logging.StreamHandler  # type: ignore[assignment]
@@ -226,7 +241,8 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
         else:
             if TYPE_CHECKING:
                 assert dag is not None
-            data_interval = dag.get_run_data_interval(dag_run)
+            # TODO: Task-SDK: Where should this function be?
+            data_interval = dag.get_run_data_interval(dag_run)  # type: ignore[attr-defined]
 
         if self.json_format:
             data_interval_start = self._clean_date(data_interval[0])
@@ -428,9 +444,11 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
                 extras={
                     "dag_id": str(ti.dag_id),
                     "task_id": str(ti.task_id),
-                    date_key: self._clean_date(ti.logical_date)
-                    if AIRFLOW_V_3_0_PLUS
-                    else self._clean_date(ti.execution_date),
+                    date_key: (
+                        self._clean_date(ti.logical_date)
+                        if AIRFLOW_V_3_0_PLUS
+                        else self._clean_date(ti.execution_date)
+                    ),
                     "try_number": str(ti.try_number),
                     "log_id": self._render_log_id(ti, ti.try_number),
                 },
@@ -480,6 +498,18 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
             self.handler.close()
             sys.stdout = sys.__stdout__
 
+        if self.write_to_es and not self.write_stdout:
+            full_path = self.handler.baseFilename  # type: ignore[union-attr]
+            log_relative_path = pathlib.Path(full_path).relative_to(self.local_base).as_posix()
+            local_loc = os.path.join(self.local_base, log_relative_path)
+            if os.path.exists(local_loc):
+                # read log and remove old logs to get just the latest additions
+                log = pathlib.Path(local_loc).read_text()
+                log_lines = self._parse_raw_log(log)
+                success = self._write_to_es(log_lines)
+                if success and self.delete_local_copy:
+                    shutil.rmtree(os.path.dirname(local_loc))
+
         super().close()
 
         self.closed = True
@@ -599,6 +629,31 @@ class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMix
         callback: type[Hit] | Callable[..., Any] = getattr(doc_class, "from_es", doc_class)
         return callback(hit)
 
+    def _parse_raw_log(self, log: str) -> list[dict[str, Any]]:
+        logs = log.split("\n")
+        parsed_logs = []
+        for line in logs:
+            # Make sure line is not empty
+            if line.strip():
+                parsed_logs.append(json.loads(line))
+
+        return parsed_logs
+
+    def _write_to_es(self, log_lines: list[dict[str, Any]]) -> bool:
+        """
+        Write the log to ElasticSearch; return `True` or fails silently and return `False`.
+
+        :param log_lines: the log_lines to write to the ElasticSearch.
+        """
+        # Prepare the bulk request for Elasticsearch
+        bulk_actions = [{"_index": self.target_index, "_source": log} for log in log_lines]
+        try:
+            _ = helpers.bulk(self.client, bulk_actions)
+            return True
+        except Exception as e:
+            self.log.exception("Unable to insert logs into Elasticsearch. Reason: %s", str(e))
+            return False
+
 
 def getattr_nested(obj, item, default):
     """