apache-airflow-providers-openlineage 2.6.1rc1__tar.gz → 2.7.0rc1__tar.gz

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-openlineage
-Version: 2.6.1rc1
+Version: 2.7.0rc1
 Summary: Provider package apache-airflow-providers-openlineage for Apache Airflow
 Keywords: airflow-provider,openlineage,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -27,8 +27,8 @@ Requires-Dist: attrs>=22.2
 Requires-Dist: openlineage-integration-common>=1.36.0
 Requires-Dist: openlineage-python>=1.36.0
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.6.1
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.7.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -59,9 +59,8 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 
 Package ``apache-airflow-providers-openlineage``
 
-Release: ``2.6.1``
+Release: ``2.7.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `OpenLineage <https://openlineage.io/>`__
 
@@ -73,12 +72,12 @@ This is a provider package for ``openlineage`` provider. All classes for this pr
 are in ``airflow.providers.openlineage`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-openlineage``
 
@@ -119,5 +118,5 @@ Dependent package
 ==================================================================================================================  =================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/changelog.html>`_.
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/README.rst

@@ -23,9 +23,8 @@
 
 Package ``apache-airflow-providers-openlineage``
 
-Release: ``2.6.1``
+Release: ``2.7.0``
 
-Release Date: ``|PypiReleaseDate|``
 
 `OpenLineage <https://openlineage.io/>`__
 
@@ -37,12 +36,12 @@ This is a provider package for ``openlineage`` provider. All classes for this pr
 are in ``airflow.providers.openlineage`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/>`_.
 
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
+You can install this package on top of an existing Airflow installation (see ``Requirements`` below
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-openlineage``
 
@@ -83,4 +82,4 @@ Dependent package
 ==================================================================================================================  =================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/changelog.html>`_.

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/docs/changelog.rst

@@ -26,6 +26,50 @@
 Changelog
 ---------
 
+2.7.0
+.....
+
+
+Release Date: ``|PypiReleaseDate|``
+
+Features
+~~~~~~~~
+
+* ``add tasksDuration to state run facet (#53644)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix setproctitle usage on macos (#53122)``
+* ``Fix mypy no-redef errors for timeout imports in providers (#54471)``
+
+Misc
+~~~~
+
+* ``Decouple serialization and deserialization code for tasks (#54569)``
+* ``Move secrets_masker over to airflow_shared distribution (#54449)``
+* ``Remove MappedOperator inheritance (#53696)``
+* ``Update usage of timeout contextmanager from SDK where possible (#54183)``
+
+Doc-only
+~~~~~~~~
+
+* ``Make term Dag consistent in providers docs (#55101)``
+* ``docs: fix broken link on OpenLineage developer section (#54356)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Remove airflow.models.DAG (#54383)``
+   * ``chore: add more test cases to OpenLineage system tests (#55138)``
+   * ``chore: update ol system tests after bug fix in AF3 (#54629)``
+   * ``Switch pre-commit to prek (#54258)``
+   * ``Remove SDK BaseOperator in TaskInstance (#53223)``
+   * ``chore: Adjust OL system tests to latest changes (#54352)``
+
+.. Review and move the new changes to one of the sections above:
+   * ``Move email notifications from scheduler to DAG processor (#55238)``
+   * ``Fix Airflow 2 reference in README/index of providers (#55240)``
+
 2.6.1
 .....
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/docs/guides/developer.rst

@@ -180,7 +180,7 @@ Custom Extractors
 This approach is recommended when dealing with Operators that you can not modify (f.e. third party providers), but still want the lineage to be extracted from them.
 If you want to extract lineage from your own Operators, you may prefer directly implementing OpenLineage methods as described in :ref:`openlineage_methods:openlineage`.
 
-This approach works by detecting which Airflow Operators your DAG is using, and extracting lineage data from them using corresponding Extractors class.
+This approach works by detecting which Airflow Operators your Dag is using, and extracting lineage data from them using corresponding Extractors class.
 
 Interface
 ^^^^^^^^^
@@ -288,7 +288,7 @@ To learn more about how Operators and Extractors work together under the hood, c
 When testing an Extractor, we want to firstly verify if ``OperatorLineage`` object is being created,
 specifically verifying that the object is being built with the correct input and output datasets and relevant facets.
 This is done in OpenLineage via pytest, with appropriate mocking and patching for connections and objects.
-Check out `example tests <https://github.com/apache/airflow/blob/main/providers/openlineage/tests/openlineage/extractors/test_base.py>`_.
+Check out `example tests <https://github.com/apache/airflow/blob/main/providers/openlineage/tests/unit/openlineage/extractors/test_base.py>`_.
 
 Testing each facet is also important, as data or graphs in the UI can render incorrectly if the facets are wrong.
 For example, if the facet name is created incorrectly in the Extractor, then the Operator's task will not show up in the lineage graph,
@@ -370,8 +370,8 @@ like extracting column level lineage and inputs/outputs from SQL query with SQL
             return lineage_metadata
 
 For more examples of OpenLineage Extractors, check out the source code of
-`BashExtractor <https://github.com/apache/airflow/blob/main/providers/amazon/aws/src/airflow/providers/openlineage/extractors/bash.py>`_ or
-`PythonExtractor <https://github.com/apache/airflow/blob/main/providers/amazon/aws/src/airflow/providers/openlineage/extractors/python.py>`_.
+`BashExtractor <https://github.com/apache/airflow/blob/main/providers/openlineage/src/airflow/providers/openlineage/extractors/bash.py>`_ or
+`PythonExtractor <https://github.com/apache/airflow/blob/main/providers/openlineage/src/airflow/providers/openlineage/extractors/python.py>`_.
 
 .. _custom_facets:openlineage:
 
@@ -476,19 +476,19 @@ a string of semicolon separated full import path to the functions.
 Job Hierarchy
 =============
 
-Apache Airflow features an inherent job hierarchy: DAGs, large and independently schedulable units, comprise smaller, executable tasks.
+Apache Airflow features an inherent job hierarchy: Dags, large and independently schedulable units, comprise smaller, executable tasks.
 
 OpenLineage reflects this structure in its Job Hierarchy model.
 
-- Upon DAG scheduling, a START event is emitted.
+- Upon Dag scheduling, a START event is emitted.
 - Subsequently, following Airflow's task order, each task triggers:
 
   - START events at TaskInstance start.
   - COMPLETE/FAILED events upon completion.
 
-- Finally, upon DAG termination, a completion event (COMPLETE or FAILED) is emitted.
+- Finally, upon Dag termination, a completion event (COMPLETE or FAILED) is emitted.
 
-TaskInstance events' ParentRunFacet references the originating DAG run.
+TaskInstance events' ParentRunFacet references the originating Dag run.
 
 .. _troubleshooting:openlineage:
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/docs/guides/structure.rst

@@ -47,9 +47,9 @@ The metadata collected can answer questions like:
 - Are there redundant data processes that can be optimized or removed?
 - What data dependencies exist for this critical report?
 
-Understanding complex inter-DAG dependencies and providing up-to-date runtime visibility into DAG execution can be challenging.
-OpenLineage integrates with Airflow to collect DAG lineage metadata so that inter-DAG dependencies are easily maintained
-and viewable via a lineage graph, while also keeping a catalog of historical runs of DAGs.
+Understanding complex inter-Dag dependencies and providing up-to-date runtime visibility into Dag execution can be challenging.
+OpenLineage integrates with Airflow to collect Dag lineage metadata so that inter-Dag dependencies are easily maintained
+and viewable via a lineage graph, while also keeping a catalog of historical runs of Dags.
 
 For OpenLineage backend that will receive events, you can use `Marquez <https://marquezproject.ai/>`_
 
@@ -60,8 +60,8 @@ OpenLineage integration implements `AirflowPlugin <https://airflow.apache.org/do
 This allows it to be discovered on Airflow start and register
 `Airflow Listener <https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html>`_.
 
-The ``OpenLineageListener`` is then called by Airflow when certain events happen - when DAGs or TaskInstances start, complete or fail.
-For DAGs, the listener runs in Airflow Scheduler. For TaskInstances, the listener runs on Airflow Worker.
+The ``OpenLineageListener`` is then called by Airflow when certain events happen - when Dags or TaskInstances start, complete or fail.
+For Dags, the listener runs in Airflow Scheduler. For TaskInstances, the listener runs on Airflow Worker.
 
 When TaskInstance listener method gets called, the ``OpenLineageListener`` constructs metadata like event's unique ``run_id`` and event time.
 Then, it tries to extract metadata from Airflow Operators as described in :ref:`extraction_precedence:openlineage`.

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/docs/guides/user.rst

@@ -24,7 +24,7 @@ Using OpenLineage integration
 OpenLineage is an open framework for data lineage collection and analysis. At its core is an extensible specification that systems can use to interoperate with lineage metadata.
 `Check out OpenLineage docs <https://openlineage.io/docs/>`_.
 
-**No change to user DAG files is required to use OpenLineage**. Basic configuration is needed so that OpenLineage knows where to send events.
+**No change to user Dag files is required to use OpenLineage**. Basic configuration is needed so that OpenLineage knows where to send events.
 
 Quickstart
 ==========
@@ -57,7 +57,7 @@ This example is a basic demonstration of OpenLineage setup.
 
       AIRFLOW__OPENLINEAGE__TRANSPORT='{"type": "http", "url": "http://example.com:5000", "endpoint": "api/v1/lineage"}'
 
-3. **That's it !**  OpenLineage events should be sent to the configured backend when DAGs are run.
+3. **That's it !**  OpenLineage events should be sent to the configured backend when Dags are run.
 
 Usage
 =====
@@ -352,10 +352,10 @@ reproducing your environment setup by setting ``debug_mode`` option to ``true``
   By setting this variable to true, OpenLineage integration may log and emit extensive details. It should only be enabled temporary for debugging purposes.
 
 
-Enabling OpenLineage on DAG/task level
+Enabling OpenLineage on Dag/task level
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-One can selectively enable OpenLineage for specific DAGs and tasks by using the ``selective_enable`` policy.
+One can selectively enable OpenLineage for specific Dags and tasks by using the ``selective_enable`` policy.
 To enable this policy, set the ``selective_enable`` option to True in the [openlineage] section of your Airflow configuration file:
 
 .. code-block:: ini
@@ -371,47 +371,47 @@ To enable this policy, set the ``selective_enable`` option to True in the [openl
 
 
 While ``selective_enable`` enables selective control, the ``disabled`` :ref:`option <options:disable>` still has precedence.
-If you set ``disabled`` to True in the configuration, OpenLineage will be disabled for all DAGs and tasks regardless of the ``selective_enable`` setting.
+If you set ``disabled`` to True in the configuration, OpenLineage will be disabled for all Dags and tasks regardless of the ``selective_enable`` setting.
 
 Once the ``selective_enable`` policy is enabled, you can choose to enable OpenLineage
-for individual DAGs and tasks using the ``enable_lineage`` and ``disable_lineage`` functions.
+for individual Dags and tasks using the ``enable_lineage`` and ``disable_lineage`` functions.
 
-1. Enabling Lineage on a DAG:
+1. Enabling Lineage on a Dag:
 
 .. code-block:: python
 
     from airflow.providers.openlineage.utils.selective_enable import disable_lineage, enable_lineage
 
-    with enable_lineage(DAG(...)):
-        # Tasks within this DAG will have lineage tracking enabled
+    with enable_lineage(Dag(...)):
+        # Tasks within this Dag will have lineage tracking enabled
         MyOperator(...)
 
         AnotherOperator(...)
 
 2. Enabling Lineage on a Task:
 
-While enabling lineage on a DAG implicitly enables it for all tasks within that DAG, you can still selectively disable it for specific tasks:
+While enabling lineage on a Dag implicitly enables it for all tasks within that Dag, you can still selectively disable it for specific tasks:
 
 .. code-block:: python
 
     from airflow.providers.openlineage.utils.selective_enable import disable_lineage, enable_lineage
 
-    with DAG(...) as dag:
+    with Dag(...) as dag:
         t1 = MyOperator(...)
         t2 = AnotherOperator(...)
 
-    # Enable lineage for the entire DAG
+    # Enable lineage for the entire Dag
     enable_lineage(dag)
 
     # Disable lineage for task t1
     disable_lineage(t1)
 
-Enabling lineage on the DAG level automatically enables it for all tasks within that DAG unless explicitly disabled per task.
+Enabling lineage on the Dag level automatically enables it for all tasks within that Dag unless explicitly disabled per task.
 
-Enabling lineage on the task level implicitly enables lineage on its DAG.
+Enabling lineage on the task level implicitly enables lineage on its Dag.
 This is because each emitting task sends a `ParentRunFacet <https://openlineage.io/docs/spec/facets/run-facets/parent_run>`_,
-which requires the DAG-level lineage to be enabled in some OpenLineage backend systems.
-Disabling DAG-level lineage while enabling task-level lineage might cause errors or inconsistencies.
+which requires the Dag-level lineage to be enabled in some OpenLineage backend systems.
+Disabling Dag-level lineage while enabling task-level lineage might cause errors or inconsistencies.
 
 .. _options:spark_inject_parent_job_info:
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/docs/index.rst

@@ -81,9 +81,7 @@ apache-airflow-providers-openlineage package
 `OpenLineage <https://openlineage.io/>`__
 
 
-Release: 2.6.1
-
-Release Date: ``|PypiReleaseDate|``
+Release: 2.7.0
 
 Provider package
 ----------------
@@ -94,7 +92,7 @@ All classes for this package are included in the ``airflow.providers.openlineage
 Installation
 ------------
 
-You can install this package on top of an existing Airflow 2 installation via
+You can install this package on top of an existing Airflow installation via
 ``pip install apache-airflow-providers-openlineage``.
 For the minimum Airflow version supported, see ``Requirements`` below.
 
@@ -140,5 +138,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-openlineage 2.6.1 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1.tar.gz.sha512>`__)
-* `The apache-airflow-providers-openlineage 2.6.1 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.6.1-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-openlineage 2.7.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-openlineage 2.7.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_openlineage-2.7.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/provider.yaml

@@ -22,12 +22,13 @@ description: |
   `OpenLineage <https://openlineage.io/>`__
 
 state: ready
-source-date-epoch: 1754503416
+source-date-epoch: 1756877213
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 2.7.0
   - 2.6.1
   - 2.6.0
   - 2.5.0

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-openlineage"
-version = "2.6.1rc1"
+version = "2.7.0rc1"
 description = "Provider package apache-airflow-providers-openlineage for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -54,7 +54,7 @@ requires-python = ">=3.10"
 
 # The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
-# Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
+# Make sure to run ``prek update-providers-dependencies --all-files``
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
     "apache-airflow>=2.10.0rc1",
@@ -105,8 +105,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.6.1"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.6.1/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.7.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-openlineage/2.7.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "2.6.1"
+__version__ = "2.7.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/plugins/adapter.py

@@ -55,9 +55,12 @@ if TYPE_CHECKING:
     from airflow.utils.state import DagRunState
 else:
     try:
-        from airflow.sdk.execution_time.secrets_masker import SecretsMasker, _secrets_masker
+        from airflow.sdk._shared.secrets_masker import SecretsMasker, _secrets_masker
     except ImportError:
-        from airflow.utils.log.secrets_masker import SecretsMasker, _secrets_masker
+        try:
+            from airflow.sdk.execution_time.secrets_masker import SecretsMasker, _secrets_masker
+        except ImportError:
+            from airflow.utils.log.secrets_masker import SecretsMasker, _secrets_masker
 
 _PRODUCER = f"https://github.com/apache/airflow/tree/providers-openlineage/{OPENLINEAGE_PROVIDER_VERSION}"
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/plugins/facets.py

@@ -76,10 +76,12 @@ class AirflowStateRunFacet(RunFacet):
     Attributes:
         dagRunState: This indicates the final status of the entire DAG run (e.g., "success", "failed").
         tasksState: A dictionary mapping task IDs to their respective states. (e.g., "failed", "skipped").
+        tasksDuration: A dictionary mapping task IDs to it's duration in seconds.
     """
 
     dagRunState: str
     tasksState: dict[str, str]
+    tasksDuration: dict[str, float]
 
 
 @define

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/plugins/listener.py

@@ -18,13 +18,13 @@ from __future__ import annotations
 
 import logging
 import os
+import sys
 from concurrent.futures import ProcessPoolExecutor
 from datetime import datetime
 from typing import TYPE_CHECKING
 
 import psutil
 from openlineage.client.serde import Serde
-from setproctitle import getproctitle, setproctitle
 
 from airflow import settings
 from airflow.listeners import hookimpl
@@ -48,16 +48,22 @@ from airflow.providers.openlineage.utils.utils import (
     is_selective_lineage_enabled,
     print_warning,
 )
+from airflow.providers.openlineage.version_compat import timeout, timezone
 from airflow.settings import configure_orm
 from airflow.stats import Stats
-from airflow.utils import timezone
 from airflow.utils.state import TaskInstanceState
-from airflow.utils.timeout import timeout
 
 if TYPE_CHECKING:
     from airflow.sdk.execution_time.task_runner import RuntimeTaskInstance
     from airflow.settings import Session
 
+if sys.platform == "darwin":
+    from setproctitle import getproctitle
+
+    setproctitle = lambda title: logging.getLogger(__name__).debug("Mac OS detected, skipping setproctitle")
+else:
+    from setproctitle import getproctitle, setproctitle
+
 _openlineage_listener: OpenLineageListener | None = None
 
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/utils/utils.py

@@ -97,19 +97,27 @@ else:
             from airflow.datasets import Dataset as Asset
 
     try:
-        from airflow.sdk.execution_time.secrets_masker import (
+        from airflow.sdk._shared.secrets_masker import (
             Redactable,
             Redacted,
             SecretsMasker,
             should_hide_value_for_key,
         )
     except ImportError:
-        from airflow.utils.log.secrets_masker import (
-            Redactable,
-            Redacted,
-            SecretsMasker,
-            should_hide_value_for_key,
-        )
+        try:
+            from airflow.sdk.execution_time.secrets_masker import (
+                Redactable,
+                Redacted,
+                SecretsMasker,
+                should_hide_value_for_key,
+            )
+        except ImportError:
+            from airflow.utils.log.secrets_masker import (
+                Redactable,
+                Redacted,
+                SecretsMasker,
+                should_hide_value_for_key,
+            )
 
 log = logging.getLogger(__name__)
 _NOMINAL_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ"
@@ -308,7 +316,7 @@ def get_user_provided_run_facets(ti: TaskInstance, ti_state: TaskInstanceState)
 def get_fully_qualified_class_name(operator: BaseOperator | MappedOperator) -> str:
     if isinstance(operator, (MappedOperator, SerializedBaseOperator)):
         # as in airflow.api_connexion.schemas.common_schema.ClassReferenceSchema
-        return operator._task_module + "." + operator._task_type
+        return operator._task_module + "." + operator.task_type
     op_class = get_operator_class(operator)
     return op_class.__module__ + "." + op_class.__name__
 
@@ -732,6 +740,12 @@ def get_airflow_state_run_facet(
         "airflowState": AirflowStateRunFacet(
             dagRunState=dag_run_state,
             tasksState={ti.task_id: ti.state for ti in tis},
+            tasksDuration={
+                ti.task_id: ti.duration
+                if ti.duration is not None
+                else (ti.end_date - ti.start_date).total_seconds()
+                for ti in tis
+            },
         )
     }
 

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/src/airflow/providers/openlineage/version_compat.py

@@ -39,7 +39,11 @@ if AIRFLOW_V_3_0_PLUS:
 else:
     from airflow.models import BaseOperator
 
-__all__ = [
-    "AIRFLOW_V_3_0_PLUS",
-    "BaseOperator",
-]
+try:
+    from airflow.sdk import timezone
+    from airflow.sdk.execution_time.timeout import timeout
+except ImportError:
+    from airflow.utils import timezone  # type: ignore[attr-defined,no-redef]
+    from airflow.utils.timeout import timeout  # type: ignore[assignment,attr-defined,no-redef]
+
+__all__ = ["AIRFLOW_V_3_0_PLUS", "BaseOperator", "timeout", "timezone"]

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/tests/system/openlineage/example_openlineage_base_complex_dag.py

@@ -106,7 +106,7 @@ with DAG(
     task_2 = PythonOperator(
         task_id="task_2",
         python_callable=do_nothing,
-        inlets=[Asset(uri="s3://bucket2/dir2/file2.txt")],
+        inlets=[Asset(uri="s3://bucket2/dir2/file2.txt"), Asset(uri="s3://bucket2/dir2/file3.txt")],
         max_retry_delay=42,
         doc="text doc",
         doc_md="should be skipped",

{apache_airflow_providers_openlineage-2.6.1rc1 → apache_airflow_providers_openlineage-2.7.0rc1}/tests/system/openlineage/example_openlineage_base_simple_dag.py

@@ -33,6 +33,11 @@ from airflow.providers.standard.operators.python import PythonOperator
 from system.openlineage.expected_events import get_expected_event_file_path
 from system.openlineage.operator import OpenLineageTestOperator
 
+try:
+    from airflow.sdk import chain
+except ImportError:
+    from airflow.models.baseoperator import chain  # type: ignore[no-redef]
+
 
 def do_nothing():
     pass
@@ -53,7 +58,7 @@ with DAG(
         task_id="check_events", file_path=get_expected_event_file_path(DAG_ID)
     )
 
-    do_nothing_task >> check_events
+    chain(do_nothing_task, check_events)
 
 
 from tests_common.test_utils.system_tests import get_test_run  # noqa: E402

apache_airflow_providers_openlineage-2.7.0rc1/tests/system/openlineage/example_openlineage_setup_teardown_dag.py

@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Simple DAG with setup and teardown operators.
+
+It checks:
+    - if setup and teardown information is included in DAG event for all tasks
+    - if setup and teardown information is included in task events for AF2
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.standard.operators.python import PythonOperator
+
+from system.openlineage.expected_events import get_expected_event_file_path
+from system.openlineage.operator import OpenLineageTestOperator
+
+
+def do_nothing():
+    pass
+
+
+DAG_ID = "openlineage_setup_teardown_dag"
+
+with DAG(
+    dag_id=DAG_ID,
+    start_date=datetime(2021, 1, 1),
+    schedule=None,
+    catchup=False,
+    default_args={"retries": 0},
+) as dag:
+    create_cluster = PythonOperator(task_id="create_cluster", python_callable=do_nothing)
+    run_query = PythonOperator(task_id="run_query", python_callable=do_nothing)
+    run_query2 = PythonOperator(task_id="run_query2", python_callable=do_nothing)
+    delete_cluster = PythonOperator(task_id="delete_cluster", python_callable=do_nothing)
+
+    check_events = OpenLineageTestOperator(
+        task_id="check_events", file_path=get_expected_event_file_path(DAG_ID)
+    )
+
+    (
+        create_cluster
+        >> run_query
+        >> run_query2
+        >> delete_cluster.as_teardown(setups=create_cluster)
+        >> check_events
+    )
+
+
+from tests_common.test_utils.system_tests import get_test_run  # noqa: E402
+
+# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
+test_run = get_test_run(dag)