openlineage-python 0.25.0__tar.gz → 0.26.0__tar.gz

openlineage_python-0.26.0/.gitignore

@@ -0,0 +1,3 @@
+__pycache__
+/.tox
+/dist

openlineage_python-0.26.0/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.1
+Name: openlineage-python
+Version: 0.26.0
+Summary: OpenLineage Python Client
+Author-email: OpenLineage <info@openlineage.io>
+Keywords: openlineage
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.7
+Requires-Dist: attrs>=23.1
+Requires-Dist: python-dateutil>=2.8.2
+Requires-Dist: pyyaml>=6
+Requires-Dist: requests>=2.30
+Provides-Extra: kafka
+Requires-Dist: confluent-kafka>=2.1.1; extra == 'kafka'
+Provides-Extra: test
+Requires-Dist: covdefaults>=2.3; extra == 'test'
+Requires-Dist: pytest-cov>=4; extra == 'test'
+Requires-Dist: pytest>=7.3.1; extra == 'test'
+Requires-Dist: pyyaml>=6; extra == 'test'
+Description-Content-Type: text/markdown
+
+# OpenLineage-python
+
+To install from source, run:
+
+```bash
+$ python -m pip install .
+```
+
+## Configuration
+### Config file
+
+The most common way to configure the OpenLineage Client is by `.yaml` file, which contains all the
+details of how to connect to your OpenLineage backend.
+
+The config file is located by:
+1) looking at the `OPENLINEAGE_CONFIG` environment variable
+2) looking for the `openlineage.yml` file in the current working directory
+3) looking for the `openlineage.yml` file in the `$HOME/.openlineage` directory.
+
+Different ways of connecting to OpenLineage backend are supported
+by the standarized `Transport` interface.
+This is an example config for specifying `http` transport:
+
+```yaml
+transport:
+  type: "http"
+  url: "https://backend:5000"
+  auth:
+    type: "api_key"
+    api_key: "f048521b-dfe8-47cd-9c65-0cb07d57591e"
+```
+
+The `type` property is required. It can be one of the built-in transports or a custom one.
+There are three built-in transports, `http`, `kafka` and `console`.
+Custom transports `type` is a fully qualified class name that can be imported.
+
+The rest of the properties are defined by the particular transport.
+Specifications for the built-in options are below.
+
+#### HTTP
+
+* `url` - required string parameter.
+* `endpoint` - optional string parameter specifying the endpoint to which events are sent. By default `api/v1/lineage`.
+* `timeout` - optional float parameter specifying the timeout when sending events. By default 5 seconds.
+* `verify` optional boolean attribute specifying if the client should verify TLS certificates of the backend. By default true.
+* `auth` - optional dictionary specifying authentication options. The type property is required.
+    * `type`: required property if an auth dictionary is set. Set to `api_key` or to the fully qualified class name of your TokenProvider.
+    * `api_key`: if `api_key` type is set, it sets value at `Authentication` HTTP header as `Bearer`.
+
+#### Kafka
+
+For KafkaTransport, `confluent-kafka` needs to be installed.
+You can also install `pip install openlineage-python[kafka]`
+
+* `config` - required string parameter. A dictionary that contains a Kafka producer config as in [Kafka producer config](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#kafka-client-configuration).
+* `topic` - required string parameter. The topic on what events will be sent.
+* `flush` - optional boolean parameter. If set to True, Kafka will flush after each event. By default true.
+
+
+### Custom transport
+
+To implement a custom transport, follow the instructions in the `openlineage/client/transport/transport.py` file.
+
+### Config as env vars
+
+If there is no config file found, the OpenLineage client looks at environment variables.
+This way of configuring the client supports only `http` transport, and only a subset of its config.
+
+* `OPENLINEAGE_URL` - point to the service that will consume OpenLineage events.
+* `OPENLINEAGE_API_KEY` - set if the consumer of OpenLineage events requires a `Bearer` authentication key.
+
+`OPENLINEAGE_URL` and `OPENLINEAGE_API_KEY` can also be set up manually when creating a client instance.

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/README.md

@@ -3,22 +3,22 @@
 To install from source, run:
 
 ```bash
-$ python setup.py install
+$ python -m pip install .
 ```
 
 ## Configuration
 ### Config file
 
 The most common way to configure the OpenLineage Client is by `.yaml` file, which contains all the
-details of how to connect to your OpenLineage backend. 
+details of how to connect to your OpenLineage backend.
 
 The config file is located by:
 1) looking at the `OPENLINEAGE_CONFIG` environment variable
 2) looking for the `openlineage.yml` file in the current working directory
 3) looking for the `openlineage.yml` file in the `$HOME/.openlineage` directory.
 
-Different ways of connecting to OpenLineage backend are supported 
-by the standarized `Transport` interface.  
+Different ways of connecting to OpenLineage backend are supported
+by the standarized `Transport` interface.
 This is an example config for specifying `http` transport:
 
 ```yaml
@@ -31,10 +31,10 @@ transport:
 ```
 
 The `type` property is required. It can be one of the built-in transports or a custom one.
-There are three built-in transports, `http`, `kafka` and `console`. 
+There are three built-in transports, `http`, `kafka` and `console`.
 Custom transports `type` is a fully qualified class name that can be imported.
 
-The rest of the properties are defined by the particular transport.  
+The rest of the properties are defined by the particular transport.
 Specifications for the built-in options are below.
 
 #### HTTP
@@ -45,7 +45,7 @@ Specifications for the built-in options are below.
 * `verify` optional boolean attribute specifying if the client should verify TLS certificates of the backend. By default true.
 * `auth` - optional dictionary specifying authentication options. The type property is required.
     * `type`: required property if an auth dictionary is set. Set to `api_key` or to the fully qualified class name of your TokenProvider.
-    * `api_key`: if `api_key` type is set, it sets value at `Authentication` HTTP header as `Bearer`. 
+    * `api_key`: if `api_key` type is set, it sets value at `Authentication` HTTP header as `Bearer`.
 
 #### Kafka
 
@@ -63,10 +63,10 @@ To implement a custom transport, follow the instructions in the `openlineage/cli
 
 ### Config as env vars
 
-If there is no config file found, the OpenLineage client looks at environment variables.  
+If there is no config file found, the OpenLineage client looks at environment variables.
 This way of configuring the client supports only `http` transport, and only a subset of its config.
 
 * `OPENLINEAGE_URL` - point to the service that will consume OpenLineage events.
 * `OPENLINEAGE_API_KEY` - set if the consumer of OpenLineage events requires a `Bearer` authentication key.
 
-`OPENLINEAGE_URL` and `OPENLINEAGE_API_KEY` can also be set up manually when creating a client instance.
+`OPENLINEAGE_URL` and `OPENLINEAGE_API_KEY` can also be set up manually when creating a client instance.

openlineage_python-0.26.0/openlineage/__init__.py

@@ -0,0 +1,2 @@
+# Copyright 2018-2023 contributors to the OpenLineage project
+# SPDX-License-Identifier: Apache-2.0

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/__init__.py

@@ -4,4 +4,8 @@
 from openlineage.client.client import OpenLineageClient, OpenLineageClientOptions
 from openlineage.client.facet import set_producer
 
-__all__ = [OpenLineageClient, OpenLineageClientOptions, set_producer]
+__all__ = [
+    "OpenLineageClient",
+    "OpenLineageClientOptions",
+    "set_producer",
+]

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/client.py

@@ -6,6 +6,7 @@ import typing
 from typing import Optional
 
 import attr
+
 from openlineage.client.serde import Serde
 
 if typing.TYPE_CHECKING:

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/constants.py

@@ -1,7 +1,7 @@
 # Copyright 2018-2023 contributors to the OpenLineage project
 # SPDX-License-Identifier: Apache-2.0
 
-__version__ = "0.25.0"
+__version__ = "0.26.0"
 
 DEFAULT_TIMEOUT_MS = 5000
 DEFAULT_NAMESPACE_NAME = 'default'

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/facet.py

@@ -5,6 +5,7 @@ from enum import Enum
 from typing import Dict, List, Optional
 
 import attr
+
 from openlineage.client.constants import DEFAULT_PRODUCER
 from openlineage.client.utils import RedactMixin
 

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/run.py

@@ -7,6 +7,7 @@ from typing import Dict, List, Optional
 
 import attr
 from dateutil import parser
+
 from openlineage.client.facet import NominalTimeRunFacet, ParentRunFacet
 from openlineage.client.utils import RedactMixin
 

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/transport/__init__.py

@@ -23,17 +23,18 @@ def get_default_factory():
 
 # decorator to wrap transports with
 def register_transport(clazz: Type[Transport]):
+    assert clazz.kind is not None
     _factory.register_transport(clazz.kind, clazz)
     return clazz
 
 
 __all__ = [
-    Config,
-    TransportFactory,
-    HttpConfig,
-    HttpTransport,
-    KafkaConfig,
-    KafkaTransport,
-    ConsoleTransport,
-    NoopTransport
+    "Config",
+    "TransportFactory",
+    "HttpConfig",
+    "HttpTransport",
+    "KafkaConfig",
+    "KafkaTransport",
+    "ConsoleTransport",
+    "NoopTransport"
 ]

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/transport/factory.py

@@ -72,7 +72,7 @@ class DefaultTransportFactory(TransportFactory):
         if not inspect.isclass(config_class) or not issubclass(config_class, Config):
             raise TypeError(f"Config {config_class} has to be class, and subclass of Config")
 
-        return transport_class(config_class.from_dict(config))
+        return transport_class(config_class.from_dict(config))  # type: ignore[call-arg]
 
     def _try_config_from_yaml(self) -> Optional[dict]:
         file = self._find_yaml()

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/transport/http.py

@@ -127,7 +127,7 @@ class HttpTransport(Transport):
         event = Serde.to_json(event)
         resp = self.session.post(
             urljoin(self.url, self.endpoint),
-            event,
+            event,  # type: ignore[arg-type]
             timeout=self.timeout,
             verify=self.verify
         )

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/transport/kafka.py

@@ -5,6 +5,7 @@ import logging
 from typing import Dict
 
 import attr
+
 from openlineage.client.run import RunEvent
 from openlineage.client.serde import Serde
 from openlineage.client.transport.transport import Config, Transport

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/openlineage/client/transport/transport.py

@@ -1,5 +1,6 @@
 # Copyright 2018-2023 contributors to the OpenLineage project
 # SPDX-License-Identifier: Apache-2.0
+from typing import Optional, Type
 
 from openlineage.client.run import RunEvent
 
@@ -27,8 +28,8 @@ class Config:
 
 
 class Transport:
-    kind = None
-    config = Config
+    kind: Optional[str] = None
+    config: Type[Config] = Config
 
     def emit(self, event: RunEvent):
         raise NotImplementedError()

openlineage_python-0.26.0/pyproject.toml

@@ -0,0 +1,72 @@
+[build-system]
+build-backend = "hatchling.build"
+requires = [
+  "hatchling>=1.15",
+]
+
+[project]
+name = "openlineage-python"
+version = "0.26.0"
+description = "OpenLineage Python Client"
+readme = "README.md"
+keywords = [
+  "openlineage",
+]
+authors = [{ name = "OpenLineage", email = "info@openlineage.io" }]
+requires-python = ">=3.7"
+classifiers = [
+  "Operating System :: OS Independent",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.7",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+  "attrs>=23.1",
+  "python-dateutil>=2.8.2",
+  "pyyaml>=6",
+  "requests>=2.30",
+]
+optional-dependencies.kafka = [
+  "confluent-kafka>=2.1.1",
+]
+optional-dependencies.test = [
+  "covdefaults>=2.3",
+  "pytest>=7.3.1",
+  "pytest-cov>=4",
+  "pyyaml>=6",
+]
+
+[tool.hatch.build]
+build.targets.wheel = ["openlineage"]
+
+[tool.coverage]
+html.show_contexts = true
+html.skip_covered = false
+report.fail_under = 92
+run.parallel = true
+run.plugins = ["covdefaults"]
+
+[tool.mypy]
+show_error_codes = true
+overrides = [
+  { ignore_missing_imports = true, module = [
+  "confluent_kafka.*",
+] } ]
+
+[tool.ruff]
+line-length = 100
+select = [
+    "F",
+    "E",
+    "W",
+    "I001"
+]
+target-version = "py37"
+ignore-init-module-imports = true
+src = ["openlineage", "tests"]
+namespace-packages = ["openlineage/client"]

{openlineage-python-0.25.0 → openlineage_python-0.26.0}/setup.cfg

@@ -1,24 +1,16 @@
 [bumpversion]
-current_version = 0.25.0
+current_version = 0.26.0
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(?P<rc>.*)
-serialize = 
+serialize =
 	{major}.{minor}.{patch}{rc}
 	{major}.{minor}.{patch}
 
-[bumpversion:file:setup.py]
-search = version="{current_version}"
-replace = version="{new_version}"
+[bumpversion:file:pyproject.toml]
+search = version = "{current_version}"
+replace = version = "{new_version}"
 
 [bumpversion:file:openlineage/client/constants.py]
 search = __version__ = "{current_version}"
 replace = __version__ = "{new_version}"
-
-[mypy]
-ignore_missing_imports = True
-
-[egg_info]
-tag_build = 
-tag_date = 0
-

openlineage_python-0.26.0/tests/__init__.py

@@ -0,0 +1,2 @@
+# Copyright 2018-2023 contributors to the OpenLineage project
+# SPDX-License-Identifier: Apache-2.0

openlineage_python-0.26.0/tests/config/config.yml

@@ -0,0 +1,3 @@
+transport:
+  type: "tests.transport.FakeTransport"
+  something: "true"

openlineage_python-0.26.0/tests/config/http.yml

@@ -0,0 +1,7 @@
+transport:
+  type: http
+  url: http://localhost:5050
+  endpoint: api/v1/lineage
+  auth:
+    type: api_key
+    apiKey: random_token

openlineage_python-0.26.0/tests/nominal_time_without_end.json

@@ -0,0 +1,5 @@
+{
+  "_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.0.1/client/python",
+  "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/NominalTimeRunFacet",
+  "nominalStartTime": "2020-01-01"
+}

openlineage_python-0.26.0/tests/openlineage.yml

@@ -0,0 +1,3 @@
+transport:
+  type: "tests.transport.AccumulatingTransport"
+  something: "false"

openlineage_python-0.26.0/tests/serde_example.json

@@ -0,0 +1,23 @@
+{
+	"eventTime": "2021-11-03T10:53:52.427343",
+	"eventType": "START",
+	"run": {
+		"facets": {
+			"nominalTime": {
+				"_producer": "https://github.com/OpenLineage/OpenLineage/tree/0.0.1/client/python",
+				"_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/NominalTimeRunFacet",
+				"nominalEndTime": "2020-01-02",
+				"nominalStartTime": "2020-01-01"
+			}
+		},
+		"runId": "69f4acab-b87d-4fc0-b27b-8ea950370ff3"
+	},
+	"job": {
+		"namespace":  "openlineage",
+		"name":  "name",
+		"facets": {}
+	},
+	"inputs": [],
+	"outputs": [],
+	"producer": "https://github.com/OpenLineage/OpenLineage/tree/0.0.1/client/python"
+}

openlineage_python-0.26.0/tests/test_client.py

@@ -0,0 +1,89 @@
+# Copyright 2018-2023 contributors to the OpenLineage project
+# SPDX-License-Identifier: Apache-2.0
+
+from unittest.mock import MagicMock
+
+import pytest
+from openlineage.client.client import OpenLineageClient
+from openlineage.client.run import Job, Run, RunEvent, RunState
+
+
+def test_client_fails_with_wrong_event_type():
+    session = MagicMock()
+    client = OpenLineageClient(url="http://example.com", session=session)
+
+    with pytest.raises(ValueError):
+        client.emit("event")
+
+
+def test_client_fails_to_create_with_wrong_url():
+    session = MagicMock()
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="notanurl", session=session)
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="http://", session=session)
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="example.com", session=session)
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="http:example.com", session=session)
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="http:/example.com", session=session)
+    with pytest.raises(ValueError):
+        OpenLineageClient(url="196.168.0.1", session=session)
+
+
+def test_client_passes_to_create_with_valid_url():
+    session = MagicMock()
+    assert OpenLineageClient(url="http://196.168.0.1", session=session).transport.url == \
+           "http://196.168.0.1"
+    assert OpenLineageClient(url="http://196.168.0.1", session=session).transport.url == \
+           "http://196.168.0.1"
+    assert OpenLineageClient(url="http://example.com  ", session=session).transport.url == \
+           "http://example.com"
+    assert OpenLineageClient(url=" http://example.com", session=session).transport.url == \
+           "http://example.com"
+    assert OpenLineageClient(url="  http://marquez:5000  ", session=session).transport.url == \
+           "http://marquez:5000"
+    assert OpenLineageClient(url="  https://marquez  ", session=session).transport.url == \
+           "https://marquez"
+
+
+def test_client_sends_proper_json_with_minimal_event():
+    session = MagicMock()
+    client = OpenLineageClient(url="http://example.com", session=session)
+
+    client.emit(
+        RunEvent(
+            RunState.START,
+            "2021-11-03T10:53:52.427343",
+            Run("69f4acab-b87d-4fc0-b27b-8ea950370ff3"),
+            Job("openlineage", "job"),
+            "producer"
+        )
+    )
+
+    session.post.assert_called_with(
+        "http://example.com/api/v1/lineage",
+        '{"eventTime": "2021-11-03T10:53:52.427343", "eventType": "START", "inputs": [], "job": '
+        '{"facets": {}, "name": "job", "namespace": "openlineage"}, "outputs": [], '
+        '"producer": "producer", "run": {"facets": {}, "runId": '
+        '"69f4acab-b87d-4fc0-b27b-8ea950370ff3"}}',
+        timeout=5.0,
+        verify=True
+    )
+
+
+def test_client_uses_passed_transport():
+    transport = MagicMock()
+    client = OpenLineageClient(transport=transport)
+    assert client.transport == transport
+
+    client.emit(event=RunEvent(
+            RunState.START,
+            "2021-11-03T10:53:52.427343",
+            Run("69f4acab-b87d-4fc0-b27b-8ea950370ff3"),
+            Job("openlineage", "job"),
+            "producer"
+        )
+    )
+    client.transport.emit.assert_called_once()

openlineage_python-0.26.0/tests/test_events.py

@@ -0,0 +1,132 @@
+# Copyright 2018-2023 contributors to the OpenLineage project
+# SPDX-License-Identifier: Apache-2.0
+
+import json
+import os
+from typing import List, Optional
+
+import attr
+import pytest
+from openlineage.client import facet, run, set_producer
+from openlineage.client.run import RunState
+from openlineage.client.serde import Serde
+
+
+@pytest.fixture(scope='session', autouse=True)
+def setup_producer():
+    set_producer('https://github.com/OpenLineage/OpenLineage/tree/0.0.1/client/python')
+
+
+def get_sorted_json(file_name: str) -> str:
+    dirpath = os.path.dirname(os.path.realpath(__file__))
+    with open(os.path.join(dirpath, file_name), 'r') as f:
+        loaded = json.load(f)
+        return json.dumps(loaded, sort_keys=True)
+
+
+def test_full_core_event_serializes_properly():
+    runEvent = run.RunEvent(
+        eventType=run.RunState.START,
+        eventTime='2021-11-03T10:53:52.427343',
+        run=run.Run(
+            runId='69f4acab-b87d-4fc0-b27b-8ea950370ff3',
+            facets={
+                "nominalTime": facet.NominalTimeRunFacet(
+                    nominalStartTime='2020-01-01',
+                    nominalEndTime='2020-01-02'
+                )
+            }
+        ),
+        job=run.Job(
+            namespace="openlineage",
+            name="name",
+            facets={}
+        ),
+        inputs=[],
+        outputs=[],
+        producer="https://github.com/OpenLineage/OpenLineage/tree/0.0.1/client/python"
+    )
+
+    assert Serde.to_json(runEvent) == get_sorted_json('serde_example.json')
+
+
+def test_run_id_uuid_check():
+    # does not throw when passed uuid
+    run.Run(runId='69f4acab-b87d-4fc0-b27b-8ea950370ff3')
+
+    with pytest.raises(ValueError):
+        run.Run(
+            runId='1500100900',
+            facets={}
+        )
+
+
+def test_run_event_type_validated():
+    valid_event = run.RunEvent(
+        RunState.START,
+        "2021-11-03T10:53:52.427343",
+        run.Run("69f4acab-b87d-4fc0-b27b-8ea950370ff3", {}),
+        run.Job("default", "name"),
+        "producer"
+    )
+    with pytest.raises(ValueError):
+        run.RunEvent(
+            "asdf",
+            valid_event.eventTime,
+            valid_event.run,
+            valid_event.job,
+            valid_event.producer
+        )
+
+    with pytest.raises(ValueError):
+        run.RunEvent(
+            valid_event.eventType,
+            "2021-11-03",
+            valid_event.run,
+            valid_event.job,
+            valid_event.producer
+        )
+
+
+def test_nominal_time_facet_does_not_require_end_time():
+    assert Serde.to_json(facet.NominalTimeRunFacet(
+        nominalStartTime='2020-01-01',
+    )) == get_sorted_json("nominal_time_without_end.json")
+
+
+def test_schema_field_default():
+    assert Serde.to_json(facet.SchemaField(name='asdf', type='int4')) == \
+           '{"name": "asdf", "type": "int4"}'
+
+    assert Serde.to_json(facet.SchemaField(
+        name='asdf',
+        type='int4',
+        description='primary key')
+    ) == '{"description": "primary key", "name": "asdf", "type": "int4"}'
+
+
+@attr.s
+class NestedObject:
+    value: Optional[int] = attr.ib(default=None)
+
+
+@attr.s
+class NestingObject:
+    nested: List[NestedObject] = attr.ib()
+    optional: Optional[int] = attr.ib(default=None)
+
+
+def test_serde_nested_nulls():
+    assert Serde.to_json(NestingObject(
+        nested=[
+            NestedObject(),
+            NestedObject(41)
+        ],
+        optional=3
+    )) == '{"nested": [{"value": 41}], "optional": 3}'
+
+    assert Serde.to_json(NestingObject(
+        nested=[
+            NestedObject()
+        ]
+    )) == '{"nested": []}'