apache-airflow-providers-common-compat 1.3.0rc1__tar.gz → 1.5.0rc1__tar.gz

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: apache-airflow-providers-common-compat
-Version: 1.3.0rc1
+Version: 1.5.0rc1
 Summary: Provider package apache-airflow-providers-common-compat for Apache Airflow
 Keywords: airflow-provider,common.compat,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -24,8 +24,8 @@ Requires-Dist: apache-airflow>=2.9.0rc0
 Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
 Requires-Dist: apache-airflow-providers-standard ; extra == "standard"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://x.com/ApacheAirflow
@@ -34,23 +34,6 @@ Provides-Extra: openlineage
 Provides-Extra: standard
 
 
-.. Licensed to the Apache Software Foundation (ASF) under one
-   or more contributor license agreements.  See the NOTICE file
-   distributed with this work for additional information
-   regarding copyright ownership.  The ASF licenses this file
-   to you under the Apache License, Version 2.0 (the
-   "License"); you may not use this file except in compliance
-   with the License.  You may obtain a copy of the License at
-
-..   http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-   software distributed under the License is distributed on an
-   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-   KIND, either express or implied.  See the License for the
-   specific language governing permissions and limitations
-   under the License.
-
  .. Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
@@ -68,8 +51,7 @@ Provides-Extra: standard
     specific language governing permissions and limitations
     under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-    OVERWRITTEN WHEN PREPARING PACKAGES.
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
  .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
     `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -77,7 +59,7 @@ Provides-Extra: standard
 
 Package ``apache-airflow-providers-common-compat``
 
-Release: ``1.3.0.rc1``
+Release: ``1.5.0``
 
 
 ``Common Compatibility Provider - providing compatibility code for previous Airflow versions.``
@@ -90,7 +72,7 @@ This is a provider package for ``common.compat`` provider. All classes for this
 are in ``airflow.providers.common.compat`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/>`_.
 
 Installation
 ------------
@@ -131,4 +113,5 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/changelog.html>`_.
+

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1}/README.rst

@@ -1,21 +1,4 @@
 
-.. Licensed to the Apache Software Foundation (ASF) under one
-   or more contributor license agreements.  See the NOTICE file
-   distributed with this work for additional information
-   regarding copyright ownership.  The ASF licenses this file
-   to you under the Apache License, Version 2.0 (the
-   "License"); you may not use this file except in compliance
-   with the License.  You may obtain a copy of the License at
-
-..   http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-   software distributed under the License is distributed on an
-   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-   KIND, either express or implied.  See the License for the
-   specific language governing permissions and limitations
-   under the License.
-
  .. Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
@@ -33,8 +16,7 @@
     specific language governing permissions and limitations
     under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-    OVERWRITTEN WHEN PREPARING PACKAGES.
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
  .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
     `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -42,7 +24,7 @@
 
 Package ``apache-airflow-providers-common-compat``
 
-Release: ``1.3.0.rc1``
+Release: ``1.5.0``
 
 
 ``Common Compatibility Provider - providing compatibility code for previous Airflow versions.``
@@ -55,7 +37,7 @@ This is a provider package for ``common.compat`` provider. All classes for this
 are in ``airflow.providers.common.compat`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/>`_.
 
 Installation
 ------------
@@ -96,4 +78,4 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/changelog.html>`_.

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1}/pyproject.toml

@@ -1,4 +1,3 @@
-
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -16,10 +15,9 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-# OVERWRITTEN WHEN PREPARING PACKAGES.
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
-# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# IF YOU WANT TO MODIFY THIS FILE EXCEPT DEPENDENCIES, YOU SHOULD MODIFY THE TEMPLATE
 # `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
 [build-system]
 requires = ["flit_core==3.10.1"]
@@ -27,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-common-compat"
-version = "1.3.0.rc1"
+version = "1.5.0.rc1"
 description = "Provider package apache-airflow-providers-common-compat for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -53,13 +51,26 @@ classifiers = [
     "Topic :: System :: Monitoring",
 ]
 requires-python = "~=3.9"
+
+# The dependencies should be modified in place in the generated file
+# Any change in the dependencies is preserved when the file is regenerated
 dependencies = [
     "apache-airflow>=2.9.0rc0",
 ]
 
+# The optional dependencies should be modified in place in the generated file
+# Any change in the dependencies is preserved when the file is regenerated
+[project.optional-dependencies]
+"openlineage" = [
+    "apache-airflow-providers-openlineage"
+]
+"standard" = [
+    "apache-airflow-providers-standard"
+]
+
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.3.0/changelog.html"
+"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0"
+"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-compat/1.5.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"
@@ -68,13 +79,9 @@ dependencies = [
 
 [project.entry-points."apache_airflow_provider"]
 provider_info = "airflow.providers.common.compat.get_provider_info:get_provider_info"
-[project.optional-dependencies]
-"openlineage" = [
-    "apache-airflow-providers-openlineage",
-]
-"standard" = [
-    "apache-airflow-providers-standard",
-]
 
 [tool.flit.module]
 name = "airflow.providers.common.compat"
+
+[tool.pytest.ini_options]
+ignore = "tests/system/"

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1/src}/airflow/providers/common/compat/LICENSE

@@ -199,55 +199,3 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-
-============================================================================
-   APACHE AIRFLOW SUBCOMPONENTS:
-
-   The Apache Airflow project contains subcomponents with separate copyright
-   notices and license terms. Your use of the source code for the these
-   subcomponents is subject to the terms and conditions of the following
-   licenses.
-
-
-========================================================================
-Third party Apache 2.0 licenses
-========================================================================
-
-The following components are provided under the Apache 2.0 License.
-See project link for details. The text of each license is also included
-at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (ALv2 License) hue v4.3.0 (https://github.com/cloudera/hue/)
-    (ALv2 License) jqclock v2.3.0 (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
-    (ALv2 License) bootstrap3-typeahead v4.0.2 (https://github.com/bassjobsen/Bootstrap-3-Typeahead)
-    (ALv2 License) connexion v2.7.0 (https://github.com/zalando/connexion)
-
-========================================================================
-MIT licenses
-========================================================================
-
-The following components are provided under the MIT License. See project link for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (MIT License) jquery v3.5.1 (https://jquery.org/license/)
-    (MIT License) dagre-d3 v0.6.4 (https://github.com/cpettitt/dagre-d3)
-    (MIT License) bootstrap v3.4.1 (https://github.com/twbs/bootstrap/)
-    (MIT License) d3-tip v0.9.1 (https://github.com/Caged/d3-tip)
-    (MIT License) dataTables v1.10.25 (https://datatables.net)
-    (MIT License) normalize.css v3.0.2 (http://necolas.github.io/normalize.css/)
-    (MIT License) ElasticMock v1.3.2 (https://github.com/vrcmarcos/elasticmock)
-    (MIT License) MomentJS v2.24.0 (http://momentjs.com/)
-    (MIT License) eonasdan-bootstrap-datetimepicker v4.17.49 (https://github.com/eonasdan/bootstrap-datetimepicker/)
-
-========================================================================
-BSD 3-Clause licenses
-========================================================================
-The following components are provided under the BSD 3-Clause license. See project links for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
-
-    (BSD 3 License) d3 v5.16.0 (https://d3js.org)
-    (BSD 3 License) d3-shape v2.1.0 (https://github.com/d3/d3-shape)
-    (BSD 3 License) cgroupspy 0.2.1 (https://github.com/cloudsigma/cgroupspy)
-
-========================================================================
-See 3rd-party-licenses/LICENSES-ui.txt for packages used in `/airflow/www`

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1/src}/airflow/providers/common/compat/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "1.3.0"
+__version__ = "1.5.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.9.0"

{apache_airflow_providers_common_compat-1.3.0rc1 → apache_airflow_providers_common_compat-1.5.0rc1/src}/airflow/providers/common/compat/get_provider_info.py

@@ -15,8 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
-# OVERWRITTEN WHEN PREPARING PACKAGES.
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 #
 # IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
 # `get_provider_info_TEMPLATE.py.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
@@ -29,8 +28,7 @@ def get_provider_info():
         "description": "``Common Compatibility Provider - providing compatibility code for previous Airflow versions.``\n",
         "state": "ready",
         "source-date-epoch": 1731569875,
-        "versions": ["1.3.0", "1.2.2", "1.2.1", "1.2.0", "1.1.0", "1.0.0"],
-        "dependencies": ["apache-airflow>=2.9.0"],
+        "versions": ["1.5.0", "1.3.0", "1.2.2", "1.2.1", "1.2.0", "1.1.0", "1.0.0"],
         "integrations": [
             {
                 "integration-name": "Common Compat",
@@ -38,4 +36,9 @@ def get_provider_info():
                 "tags": ["software"],
             }
         ],
+        "dependencies": ["apache-airflow>=2.9.0"],
+        "optional-dependencies": {
+            "openlineage": ["apache-airflow-providers-openlineage"],
+            "standard": ["apache-airflow-providers-standard"],
+        },
     }

apache_airflow_providers_common_compat-1.5.0rc1/src/airflow/providers/common/compat/lineage/entities.py

@@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Defines base entities used for providing lineage information."""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+import attr
+
+
+@attr.s(auto_attribs=True)
+class File:
+    """File entity. Refers to a file."""
+
+    template_fields: ClassVar = ("url",)
+
+    url: str = attr.ib()
+    type_hint: str | None = None
+
+
+@attr.s(auto_attribs=True, kw_only=True)
+class User:
+    """User entity. Identifies a user."""
+
+    email: str = attr.ib()
+    first_name: str | None = None
+    last_name: str | None = None
+
+    template_fields: ClassVar = ("email", "first_name", "last_name")
+
+
+@attr.s(auto_attribs=True, kw_only=True)
+class Tag:
+    """Tag or classification entity."""
+
+    tag_name: str = attr.ib()
+
+    template_fields: ClassVar = ("tag_name",)
+
+
+@attr.s(auto_attribs=True, kw_only=True)
+class Column:
+    """Column of a Table."""
+
+    name: str = attr.ib()
+    description: str | None = None
+    data_type: str = attr.ib()
+    tags: list[Tag] = []
+
+    template_fields: ClassVar = ("name", "description", "data_type", "tags")
+
+
+# this is a temporary hack to satisfy mypy. Once
+# https://github.com/python/mypy/issues/6136 is resolved, use
+# `attr.converters.default_if_none(default=False)`
+
+
+def default_if_none(arg: bool | None) -> bool:
+    """Get default value when None."""
+    return arg or False
+
+
+@attr.s(auto_attribs=True, kw_only=True)
+class Table:
+    """Table entity."""
+
+    database: str = attr.ib()
+    cluster: str = attr.ib()
+    name: str = attr.ib()
+    tags: list[Tag] = []
+    description: str | None = None
+    columns: list[Column] = []
+    owners: list[User] = []
+    extra: dict[str, Any] = {}
+    type_hint: str | None = None
+
+    template_fields: ClassVar = (
+        "database",
+        "cluster",
+        "name",
+        "tags",
+        "description",
+        "columns",
+        "owners",
+        "extra",
+    )

apache_airflow_providers_common_compat-1.5.0rc1/src/airflow/providers/common/compat/openlineage/utils/spark.py

@@ -0,0 +1,130 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+log = logging.getLogger(__name__)
+
+if TYPE_CHECKING:
+    from airflow.providers.openlineage.utils.spark import (
+        inject_parent_job_information_into_spark_properties,
+        inject_transport_information_into_spark_properties,
+    )
+else:
+    try:
+        from airflow.providers.openlineage.utils.spark import (
+            inject_parent_job_information_into_spark_properties,
+            inject_transport_information_into_spark_properties,
+        )
+    except ImportError:
+        try:
+            from airflow.providers.openlineage.plugins.macros import (
+                lineage_job_name,
+                lineage_job_namespace,
+                lineage_run_id,
+            )
+        except ImportError:
+
+            def inject_parent_job_information_into_spark_properties(properties: dict, context) -> dict:
+                log.warning(
+                    "Could not import `airflow.providers.openlineage.plugins.macros`."
+                    "Skipping the injection of OpenLineage parent job information into Spark properties."
+                )
+                return properties
+
+        else:
+
+            def inject_parent_job_information_into_spark_properties(properties: dict, context) -> dict:
+                if any(str(key).startswith("spark.openlineage.parent") for key in properties):
+                    log.info(
+                        "Some OpenLineage properties with parent job information are already present "
+                        "in Spark properties. Skipping the injection of OpenLineage "
+                        "parent job information into Spark properties."
+                    )
+                    return properties
+
+                ti = context["ti"]
+                ol_parent_job_properties = {
+                    "spark.openlineage.parentJobNamespace": lineage_job_namespace(),
+                    "spark.openlineage.parentJobName": lineage_job_name(ti),
+                    "spark.openlineage.parentRunId": lineage_run_id(ti),
+                }
+                return {**properties, **ol_parent_job_properties}
+
+        try:
+            from airflow.providers.openlineage.plugins.listener import get_openlineage_listener
+        except ImportError:
+
+            def inject_transport_information_into_spark_properties(properties: dict, context) -> dict:
+                log.warning(
+                    "Could not import `airflow.providers.openlineage.plugins.listener`."
+                    "Skipping the injection of OpenLineage transport information into Spark properties."
+                )
+                return properties
+
+        else:
+
+            def inject_transport_information_into_spark_properties(properties: dict, context) -> dict:
+                if any(str(key).startswith("spark.openlineage.transport") for key in properties):
+                    log.info(
+                        "Some OpenLineage properties with transport information are already present "
+                        "in Spark properties. Skipping the injection of OpenLineage "
+                        "transport information into Spark properties."
+                    )
+                    return properties
+
+                transport = get_openlineage_listener().adapter.get_or_create_openlineage_client().transport
+                if transport.kind != "http":
+                    log.info(
+                        "OpenLineage transport type `%s` does not support automatic "
+                        "injection of OpenLineage transport information into Spark properties.",
+                        transport.kind,
+                    )
+                    return {}
+
+                transport_properties = {
+                    "spark.openlineage.transport.type": "http",
+                    "spark.openlineage.transport.url": transport.url,
+                    "spark.openlineage.transport.endpoint": transport.endpoint,
+                    # Timeout is converted to milliseconds, as required by Spark integration,
+                    "spark.openlineage.transport.timeoutInMillis": str(int(transport.timeout * 1000)),
+                }
+                if transport.compression:
+                    transport_properties["spark.openlineage.transport.compression"] = str(
+                        transport.compression
+                    )
+
+                if hasattr(transport.config.auth, "api_key") and transport.config.auth.get_bearer():
+                    transport_properties["spark.openlineage.transport.auth.type"] = "api_key"
+                    transport_properties["spark.openlineage.transport.auth.apiKey"] = (
+                        transport.config.auth.get_bearer()
+                    )
+
+                if hasattr(transport.config, "custom_headers") and transport.config.custom_headers:
+                    for key, value in transport.config.custom_headers.items():
+                        transport_properties[f"spark.openlineage.transport.headers.{key}"] = value
+
+                return {**properties, **transport_properties}
+
+
+__all__ = [
+    "inject_parent_job_information_into_spark_properties",
+    "inject_transport_information_into_spark_properties",
+]

apache_airflow_providers_common_compat-1.5.0rc1/src/airflow/providers/common/compat/openlineage/utils/sql.py

@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+log = logging.getLogger(__name__)
+
+if TYPE_CHECKING:
+    from airflow.providers.openlineage.sqlparser import get_openlineage_facets_with_sql
+
+else:
+    try:
+        from airflow.providers.openlineage.sqlparser import get_openlineage_facets_with_sql
+    except ImportError:
+
+        def get_openlineage_facets_with_sql(
+            hook,
+            sql: str | list[str],
+            conn_id: str,
+            database: str | None,
+        ):
+            try:
+                from airflow.providers.openlineage.sqlparser import SQLParser
+            except ImportError:
+                log.debug("SQLParser could not be imported from OpenLineage provider.")
+                return None
+
+            try:
+                from airflow.providers.openlineage.utils.utils import should_use_external_connection
+
+                use_external_connection = should_use_external_connection(hook)
+            except ImportError:
+                # OpenLineage provider release < 1.8.0 - we always use connection
+                use_external_connection = True
+
+            connection = hook.get_connection(conn_id)
+            try:
+                database_info = hook.get_openlineage_database_info(connection)
+            except AttributeError:
+                log.debug("%s has no database info provided", hook)
+                database_info = None
+
+            if database_info is None:
+                return None
+
+            try:
+                sql_parser = SQLParser(
+                    dialect=hook.get_openlineage_database_dialect(connection),
+                    default_schema=hook.get_openlineage_default_schema(),
+                )
+            except AttributeError:
+                log.debug("%s failed to get database dialect", hook)
+                return None
+
+            try:
+                sqlalchemy_engine = hook.get_sqlalchemy_engine()
+            except Exception as e:
+                log.debug("Failed to get sql alchemy engine: %s", e)
+                sqlalchemy_engine = None
+
+            operator_lineage = sql_parser.generate_openlineage_metadata_from_sql(
+                sql=sql,
+                hook=hook,
+                database_info=database_info,
+                database=database,
+                sqlalchemy_engine=sqlalchemy_engine,
+                use_connection=use_external_connection,
+            )
+
+            return operator_lineage
+
+
+__all__ = ["get_openlineage_facets_with_sql"]

apache_airflow_providers_common_compat-1.3.0rc1/airflow/providers/common/compat/openlineage/utils/spark.py

@@ -1,68 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from __future__ import annotations
-
-import logging
-from typing import TYPE_CHECKING
-
-log = logging.getLogger(__name__)
-
-if TYPE_CHECKING:
-    from airflow.providers.openlineage.utils.spark import inject_parent_job_information_into_spark_properties
-else:
-    try:
-        from airflow.providers.openlineage.utils.spark import (
-            inject_parent_job_information_into_spark_properties,
-        )
-    except ImportError:
-        try:
-            from airflow.providers.openlineage.plugins.macros import (
-                lineage_job_name,
-                lineage_job_namespace,
-                lineage_run_id,
-            )
-        except ImportError:
-
-            def inject_parent_job_information_into_spark_properties(properties: dict, context) -> dict:
-                log.warning(
-                    "Could not import `airflow.providers.openlineage.plugins.macros`."
-                    "Skipping the injection of OpenLineage parent job information into Spark properties."
-                )
-                return properties
-
-        else:
-
-            def inject_parent_job_information_into_spark_properties(properties: dict, context) -> dict:
-                if any(str(key).startswith("spark.openlineage.parent") for key in properties):
-                    log.info(
-                        "Some OpenLineage properties with parent job information are already present "
-                        "in Spark properties. Skipping the injection of OpenLineage "
-                        "parent job information into Spark properties."
-                    )
-                    return properties
-
-                ti = context["ti"]
-                ol_parent_job_properties = {
-                    "spark.openlineage.parentJobNamespace": lineage_job_namespace(),
-                    "spark.openlineage.parentJobName": lineage_job_name(ti),
-                    "spark.openlineage.parentRunId": lineage_run_id(ti),
-                }
-                return {**properties, **ol_parent_job_properties}
-
-
-__all__ = ["inject_parent_job_information_into_spark_properties"]