apache-airflow-providers-openlineage 1.8.0rc1__tar.gz → 1.9.0__tar.gz

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apache-airflow-providers-openlineage
-Version: 1.8.0rc1
+Version: 1.9.0
 Summary: Provider package apache-airflow-providers-openlineage for Apache Airflow
 Keywords: airflow-provider,openlineage,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -21,15 +21,15 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
-Requires-Dist: apache-airflow-providers-common-sql>=1.6.0rc0
-Requires-Dist: apache-airflow>=2.7.0rc0
+Requires-Dist: apache-airflow-providers-common-sql>=1.6.0
+Requires-Dist: apache-airflow>=2.7.0
 Requires-Dist: attrs>=22.2
-Requires-Dist: openlineage-integration-common>=0.28.0
-Requires-Dist: openlineage-python>=0.28.0
+Requires-Dist: openlineage-integration-common>=1.16.0
+Requires-Dist: openlineage-python>=1.16.0
 Requires-Dist: apache-airflow-providers-common-sql ; extra == "common.sql"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://twitter.com/ApacheAirflow
@@ -80,7 +80,7 @@ Provides-Extra: common.sql
 
 Package ``apache-airflow-providers-openlineage``
 
-Release: ``1.8.0.rc1``
+Release: ``1.9.0``
 
 
 `OpenLineage <https://openlineage.io/>`__
@@ -93,7 +93,7 @@ This is a provider package for ``openlineage`` provider. All classes for this pr
 are in ``airflow.providers.openlineage`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0/>`_.
 
 Installation
 ------------
@@ -113,8 +113,8 @@ PIP package                              Version required
 ``apache-airflow``                       ``>=2.7.0``
 ``apache-airflow-providers-common-sql``  ``>=1.6.0``
 ``attrs``                                ``>=22.2``
-``openlineage-integration-common``       ``>=0.28.0``
-``openlineage-python``                   ``>=0.28.0``
+``openlineage-integration-common``       ``>=1.16.0``
+``openlineage-python``                   ``>=1.16.0``
 =======================================  ==================
 
 Cross provider package dependencies
@@ -137,4 +137,4 @@ Dependent package
 ============================================================================================================  ==============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0/changelog.html>`_.

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/README.rst

@@ -42,7 +42,7 @@
 
 Package ``apache-airflow-providers-openlineage``
 
-Release: ``1.8.0.rc1``
+Release: ``1.9.0``
 
 
 `OpenLineage <https://openlineage.io/>`__
@@ -55,7 +55,7 @@ This is a provider package for ``openlineage`` provider. All classes for this pr
 are in ``airflow.providers.openlineage`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0/>`_.
 
 Installation
 ------------
@@ -75,8 +75,8 @@ PIP package                              Version required
 ``apache-airflow``                       ``>=2.7.0``
 ``apache-airflow-providers-common-sql``  ``>=1.6.0``
 ``attrs``                                ``>=22.2``
-``openlineage-integration-common``       ``>=0.28.0``
-``openlineage-python``                   ``>=0.28.0``
+``openlineage-integration-common``       ``>=1.16.0``
+``openlineage-python``                   ``>=1.16.0``
 =======================================  ==================
 
 Cross provider package dependencies
@@ -99,4 +99,4 @@ Dependent package
 ============================================================================================================  ==============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.8.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/1.9.0/changelog.html>`_.

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/airflow/providers/openlineage/LICENSE

@@ -215,7 +215,7 @@ Third party Apache 2.0 licenses
 
 The following components are provided under the Apache 2.0 License.
 See project link for details. The text of each license is also included
-at licenses/LICENSE-[project].txt.
+at 3rd-party-licenses/LICENSE-[project].txt.
 
     (ALv2 License) hue v4.3.0 (https://github.com/cloudera/hue/)
     (ALv2 License) jqclock v2.3.0 (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
@@ -227,7 +227,7 @@ MIT licenses
 ========================================================================
 
 The following components are provided under the MIT License. See project link for details.
-The text of each license is also included at licenses/LICENSE-[project].txt.
+The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
 
     (MIT License) jquery v3.5.1 (https://jquery.org/license/)
     (MIT License) dagre-d3 v0.6.4 (https://github.com/cpettitt/dagre-d3)
@@ -243,11 +243,11 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
 BSD 3-Clause licenses
 ========================================================================
 The following components are provided under the BSD 3-Clause license. See project links for details.
-The text of each license is also included at licenses/LICENSE-[project].txt.
+The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
 
     (BSD 3 License) d3 v5.16.0 (https://d3js.org)
     (BSD 3 License) d3-shape v2.1.0 (https://github.com/d3/d3-shape)
     (BSD 3 License) cgroupspy 0.2.1 (https://github.com/cloudsigma/cgroupspy)
 
 ========================================================================
-See licenses/LICENSES-ui.txt for packages used in `/airflow/www`
+See 3rd-party-licenses/LICENSES-ui.txt for packages used in `/airflow/www`

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/airflow/providers/openlineage/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "1.8.0"
+__version__ = "1.9.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.7.0"

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/airflow/providers/openlineage/conf.py

@@ -33,7 +33,15 @@ from __future__ import annotations
 import os
 from typing import Any
 
-from airflow.compat.functools import cache
+# Disable caching if we're inside tests - this makes config easier to mock.
+if os.getenv("PYTEST_VERSION"):
+
+    def decorator(func):
+        return func
+
+    cache = decorator
+else:
+    from airflow.compat.functools import cache
 from airflow.configuration import conf
 
 _CONFIG_SECTION = "openlineage"
@@ -130,3 +138,10 @@ def dag_state_change_process_pool_size() -> int:
     """[openlineage] dag_state_change_process_pool_size."""
     option = conf.get(_CONFIG_SECTION, "dag_state_change_process_pool_size", fallback="")
     return _safe_int_convert(str(option).strip(), default=1)
+
+
+@cache
+def execution_timeout() -> int:
+    """[openlineage] execution_timeout."""
+    option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
+    return _safe_int_convert(str(option).strip(), default=10)

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/airflow/providers/openlineage/extractors/base.py

@@ -41,7 +41,8 @@ class OperatorLineage:
 
 
 class BaseExtractor(ABC, LoggingMixin):
-    """Abstract base extractor class.
+    """
+    Abstract base extractor class.
 
     This is used mostly to maintain support for custom extractors.
     """
@@ -55,7 +56,8 @@ class BaseExtractor(ABC, LoggingMixin):
     @classmethod
     @abstractmethod
     def get_operator_classnames(cls) -> list[str]:
-        """Get a list of operators that extractor works for.
+        """
+        Get a list of operators that extractor works for.
 
         This is an abstract method that subclasses should implement. There are
         operators that work very similarly and one extractor can cover.
@@ -77,7 +79,8 @@ class DefaultExtractor(BaseExtractor):
 
     @classmethod
     def get_operator_classnames(cls) -> list[str]:
-        """Assign this extractor to *no* operators.
+        """
+        Assign this extractor to *no* operators.
 
         Default extractor is chosen not on the classname basis, but
         by existence of get_openlineage_facets method on operator.

apache_airflow_providers_openlineage-1.9.0/airflow/providers/openlineage/facets/AirflowJobFacet.json

@@ -0,0 +1,40 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "$defs": {
+      "AirflowJobFacet": {
+        "allOf": [
+          {
+            "$ref": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet"
+          },
+          {
+            "type": "object",
+            "properties": {
+              "taskTree": {
+                "description": "The hierarchical structure of tasks in the DAG.",
+                "type": "object",
+                "additionalProperties": true
+              },
+              "taskGroups": {
+                "description": "Information about all task groups within the DAG.",
+                "type": "object",
+                "additionalProperties": true
+              },
+              "tasks": {
+                "description": "Details of all individual tasks within the DAG.",
+                "type": "object",
+                "additionalProperties": true
+              }
+            },
+            "required": ["taskTree", "taskGroups", "tasks"]
+          }
+        ],
+        "type": "object"
+      }
+    },
+    "type": "object",
+    "properties": {
+      "airflow": {
+        "$ref": "#/$defs/AirflowJobFacet"
+      }
+    }
+  }

apache_airflow_providers_openlineage-1.9.0/airflow/providers/openlineage/facets/AirflowRunFacet.json

@@ -0,0 +1,261 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$defs": {
+    "AirflowRunFacet": {
+      "allOf": [
+        {
+          "$ref": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet"
+        },
+        {
+          "type": "object",
+          "properties": {
+            "dag": {
+              "$ref": "#/$defs/DAG"
+            },
+            "dagRun": {
+              "$ref": "#/$defs/DagRun"
+            },
+            "taskInstance": {
+              "$ref": "#/$defs/TaskInstance"
+            },
+            "task": {
+              "$ref": "#/$defs/Task"
+            },
+            "taskUuid": {
+              "type": "string"
+            }
+          },
+          "required": [
+            "dag",
+            "dagRun",
+            "taskInstance",
+            "task",
+            "taskUuid"
+          ]
+        }
+      ]
+    },
+    "Task": {
+      "type": "object",
+      "properties": {
+        "depends_on_past": {
+          "type": "boolean"
+        },
+        "downstream_task_ids": {
+          "type": "string"
+        },
+        "execution_timeout": {
+          "type": "string"
+        },
+        "executor_config": {
+          "type": "object",
+          "additionalProperties": true
+        },
+        "ignore_first_depends_on_past": {
+          "type": "boolean"
+        },
+        "is_setup": {
+          "type": "boolean"
+        },
+        "is_teardown": {
+          "type": "boolean"
+        },
+        "mapped": {
+          "type": "boolean"
+        },
+        "max_active_tis_per_dag": {
+          "type": "integer"
+        },
+        "max_active_tis_per_dagrun": {
+          "type": "integer"
+        },
+        "max_retry_delay": {
+          "type": "string"
+        },
+        "multiple_outputs": {
+          "type": "boolean"
+        },
+        "operator_class": {
+          "description": "Module + class name of the operator",
+          "type": "string"
+        },
+        "owner": {
+          "type": "string"
+        },
+        "priority_weight": {
+          "type": "integer"
+        },
+        "queue": {
+          "type": "string"
+        },
+        "retries": {
+          "type": "integer"
+        },
+        "retry_exponential_backoff": {
+          "type": "boolean"
+        },
+        "run_as_user": {
+          "type": "string"
+        },
+        "sla": {
+          "type": "number"
+        },
+        "task_id": {
+          "type": "string"
+        },
+        "trigger_rule": {
+          "type": "string"
+        },
+        "upstream_task_ids": {
+          "type": "string"
+        },
+        "wait_for_downstream": {
+          "type": "boolean"
+        },
+        "wait_for_past_depends_before_skipping": {
+          "type": "boolean"
+        },
+        "weight_rule": {
+          "type": "string"
+        },
+        "task_group": {
+          "description": "Task group related information",
+          "type": "object",
+          "properties": {
+            "group_id": {
+              "type": "string"
+            },
+            "downstream_group_ids": {
+              "type": "string"
+            },
+            "downstream_task_ids": {
+              "type": "string"
+            },
+            "prefix_group_id": {
+              "type": "boolean"
+            },
+            "tooltip": {
+              "type": "string"
+            },
+            "upstream_group_ids": {
+              "type": "string"
+            },
+            "upstream_task_ids": {
+              "type": "string"
+            }
+          },
+          "additionalProperties": true,
+          "required": ["group_id"]
+        }
+      },
+      "additionalProperties": true,
+      "required": [
+        "task_id"
+      ]
+    },
+    "DAG": {
+      "type": "object",
+      "properties": {
+        "dag_id": {
+          "type": "string"
+        },
+        "description": {
+          "type": "string"
+        },
+        "owner": {
+          "type": "string"
+        },
+        "schedule_interval": {
+          "type": "string"
+        },
+        "start_date": {
+          "type": "string",
+          "format": "date-time"
+        },
+        "tags": {
+          "type": "string"
+        },
+        "timetable": {
+          "description": "Describes timetable (successor of schedule_interval)",
+          "type": "object",
+          "additionalProperties": true
+        }
+      },
+      "additionalProperties": true,
+      "required": [
+        "dag_id",
+        "start_date"
+      ]
+    },
+    "TaskInstance": {
+      "type": "object",
+      "properties": {
+        "duration": {
+          "type": "number"
+        },
+        "map_index": {
+          "type": "integer"
+        },
+        "pool": {
+          "type": "string"
+        },
+        "try_number": {
+          "type": "integer"
+        },
+        "queued_dttm": {
+          "type": "string",
+          "format": "date-time"
+        }
+      },
+      "additionalProperties": true,
+      "required": [
+        "pool",
+        "try_number"
+      ]
+    },
+    "DagRun": {
+      "type": "object",
+      "properties": {
+        "conf": {
+          "type": "object",
+          "additionalProperties": true
+        },
+        "dag_id": {
+          "type": "string"
+        },
+        "data_interval_start": {
+          "type": "string",
+          "format": "date-time"
+        },
+        "data_interval_end": {
+          "type": "string",
+          "format": "date-time"
+        },
+        "external_trigger": {
+          "type": "boolean"
+        },
+        "run_id": {
+          "type": "string"
+        },
+        "run_type": {
+          "type": "string"
+        },
+        "start_date": {
+          "type": "string",
+          "format": "date-time"
+        }
+      },
+      "additionalProperties": true,
+      "required": [
+        "dag_id",
+        "run_id"
+      ]
+    }
+  },
+  "type": "object",
+  "properties": {
+    "airflow": {
+      "$ref": "#/$defs/AirflowRunFacet"
+    }
+  }
+}

apache_airflow_providers_openlineage-1.9.0/airflow/providers/openlineage/facets/AirflowStateRunFacet.json

@@ -0,0 +1,34 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "$defs": {
+      "AirflowStateRunFacet": {
+        "allOf": [
+          {
+            "$ref": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunFacet"
+          },
+          {
+            "type": "object",
+            "properties": {
+              "dagRunState": {
+                "description": "The final status of the entire DagRun",
+                "type": "string"
+              },
+              "tasksState": {
+                "description": "Mapping of task IDs to their respective states",
+                "type": "object",
+                "additionalProperties": true
+              }
+            },
+            "required": ["dagRunState", "tasksState"]
+          }
+        ],
+        "type": "object"
+      }
+    },
+    "type": "object",
+    "properties": {
+      "airflowState": {
+        "$ref": "#/$defs/AirflowStateRunFacet"
+      }
+    }
+  }

{apache_airflow_providers_openlineage-1.8.0rc1 → apache_airflow_providers_openlineage-1.9.0}/airflow/providers/openlineage/get_provider_info.py

@@ -28,8 +28,9 @@ def get_provider_info():
         "name": "OpenLineage Airflow",
         "description": "`OpenLineage <https://openlineage.io/>`__\n",
         "state": "ready",
-        "source-date-epoch": 1715684338,
+        "source-date-epoch": 1718605195,
         "versions": [
+            "1.9.0",
             "1.8.0",
             "1.7.1",
             "1.7.0",
@@ -50,8 +51,8 @@ def get_provider_info():
             "apache-airflow>=2.7.0",
             "apache-airflow-providers-common-sql>=1.6.0",
             "attrs>=22.2",
-            "openlineage-integration-common>=0.28.0",
-            "openlineage-python>=0.28.0",
+            "openlineage-integration-common>=1.16.0",
+            "openlineage-python>=1.16.0",
         ],
         "integrations": [
             {
@@ -134,6 +135,13 @@ def get_provider_info():
                         "type": "integer",
                         "version_added": "1.8.0",
                     },
+                    "execution_timeout": {
+                        "description": "Maximum amount of time (in seconds) that OpenLineage can spend executing metadata extraction.\n",
+                        "default": "10",
+                        "example": None,
+                        "type": "integer",
+                        "version_added": "1.9.0",
+                    },
                 },
             }
         },