csvpath 0.0.498__tar.gz → 0.0.499__tar.gz

{csvpath-0.0.498 → csvpath-0.0.499}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: csvpath
-Version: 0.0.498
+Version: 0.0.499
 Summary: A declarative language for validating CSV, Excel, and other tabular data files
 Author: David Kershaw
 Author-email: dk107dk@hotmail.com
@@ -43,15 +43,16 @@ Project-URL: Github, https://github.com/csvpath/csvpath.git
 Description-Content-Type: text/markdown
 
 
-# <a href='https://www.csvpath.org/'><img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/></a> About CsvPath
+# <a href='https://www.csvpath.org/'><img src='https://github.com/csvpath/csvpath/blob/main/docs/images/csvpath-icon-sm.png'/></a> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
+The CsvPath language defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
 
-CsvPath's goal is to make it easy to setup a Collect, Store, Validate-pattern flat-file landing zone that:
-- Analyzes the content and structure of flat files
-- Validates that files match expectations
-- Reports on content validity
-- Creates new derived files using copy-on-write
+The CsvPath library's goal is to make it easy to setup a <a href='https://www.atestaanalytics.com/s/The-Collect-Store-Validate-Pattern-Atesta-Analytics.pdf'>Collect, Store, Validate Pattern</a> flat-file landing zone that:
+- Registers files
+- Validates that the data matches expectations
+- Reports on content validity and other metadata
+- Shapes files for consistency using copy-on-write
+- And stages the results for loading into a data lake
 
 And does it all in an automation-friendly way.
 
@@ -59,9 +60,15 @@ CsvPath's validation is inspired by:
 - XPath for XML files
 - The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
-CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create.
+CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create. CsvPath can stream events to an OpenLineage server, such as the open source Marquez server.
 
-Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+<a href='https://openlineage.io' >
+<img target='_blank' src="https://github.com/csvpath/csvpath/blob/main/docs/images/openlineage-logo-sm.png" alt="OpenLineage"/></a>
+<a href='https://peppy-sprite-186812.netlify.app/' >
+<img target='_blank' src="https://github.com/csvpath/csvpath/blob/main/docs/images/marquez-logo-sm.png" alt="Marquez Server"/></a>
+
+
+Read more about CsvPath and see CSV, Excel, and Data Frames validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
 
 If you need help, use the <a href='https://www.csvpath.org/getting-started/get-help'>contact form</a> or the <a href='https://github.com/csvpath/csvpath/issues'>issue tracker</a> or talk to one of our [sponsors](#sponsors).
 
@@ -99,11 +106,13 @@ If you need help, use the <a href='https://www.csvpath.org/getting-started/get-h
 
 CSV files are everywhere!
 
-A surprisingly large number of companies depend on CSV processing for significant amounts of revenue. Research organizations are awash in CSV. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use day to day uses CSV for sharing. CSV is the lowest of common dominators. Many CSVs are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
+The majority of companies depend on file processing for significant revenue operations. Research organizations and archives are awash in CSVs. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use spits out CSV or Excel files for sharing. Delimited and tabular files are the lowest of common dominators. Many are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
+
+CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract and shape data and create reports.
 
-CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract data, create reports, and do other useful things.
+The CsvPath library implements the CsvPath language, but goes far beyond it to provide a full <a href='https://www.atestaanalytics.com/s/The-Collect-Store-Validate-Pattern-Atesta-Analytics.pdf'>Collect, Store, Validate Pattern</a> framework for landing flat files, registering them, validating them, shaping them to a consistent and comparable form, and staging them for a data lake. In that way, CsvPath fills the gap commonly found between an organization's MFT (managed file transfer) and a typical data lake architecture.
 
-CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library, not a system, so it relies on being easy to integrate with other DataOps tools.
+CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library and framework, not a system, so it relies on being easy to integrate with other DataOps tools.
 
 
 <a name="install"></a>
@@ -217,8 +226,7 @@ The simplest way to get started is using the CLI. <a href='https://www.csvpath.o
 When you're ready to think about automation, you'll want to start with a simple driver. This is a very basic programmatic use of CsvPath.
 
 ```python
-    path = CsvPath()
-    path.parse("""
+    path = CsvPath().parse("""
             $test.csv[5-25][
                 #firstname == "Frog"
                 @lastname.onmatch = "Bat"

{csvpath-0.0.498 → csvpath-0.0.499}/README.md

@@ -1,13 +1,14 @@
 
-# <a href='https://www.csvpath.org/'><img src='https://www.csvpath.org/~gitbook/image?url=https%3A%2F%2F3739708663-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FMXTJeGvaEsqwNG39F37h%252Fsites%252Fsite_SPBqJ%252Ficon%252FMCSxo7k6rXWnqoPE204u%252Fcsvpath-icon.png%3Falt%3Dmedia%26token%3D28869fdd-d54e-400e-8917-b8097f935f42&width=32&dpr=2&quality=100&sign=71ca9f3e&sv=1'/></a> About CsvPath
+# <a href='https://www.csvpath.org/'><img src='https://github.com/csvpath/csvpath/blob/main/docs/images/csvpath-icon-sm.png'/></a> About CsvPath
 
-CsvPath defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
+The CsvPath language defines a declarative syntax for inspecting and validating CSV and Excel files, and other tabular data.
 
-CsvPath's goal is to make it easy to setup a Collect, Store, Validate-pattern flat-file landing zone that:
-- Analyzes the content and structure of flat files
-- Validates that files match expectations
-- Reports on content validity
-- Creates new derived files using copy-on-write
+The CsvPath library's goal is to make it easy to setup a <a href='https://www.atestaanalytics.com/s/The-Collect-Store-Validate-Pattern-Atesta-Analytics.pdf'>Collect, Store, Validate Pattern</a> flat-file landing zone that:
+- Registers files
+- Validates that the data matches expectations
+- Reports on content validity and other metadata
+- Shapes files for consistency using copy-on-write
+- And stages the results for loading into a data lake
 
 And does it all in an automation-friendly way.
 
@@ -15,9 +16,15 @@ CsvPath's validation is inspired by:
 - XPath for XML files
 - The ISO standard <a href='https://schematron.com/'>Schematron validation</a>
 
-CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create.
+CsvPath is intended to fit with other DataOps and data quality tools. Files are streamed. The interface is simple. Metadata is plentiful. New functions are easy to create. CsvPath can stream events to an OpenLineage server, such as the open source Marquez server.
 
-Read more about CsvPath and see realistic CSV and Excel validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
+<a href='https://openlineage.io' >
+<img target='_blank' src="https://github.com/csvpath/csvpath/blob/main/docs/images/openlineage-logo-sm.png" alt="OpenLineage"/></a>
+<a href='https://peppy-sprite-186812.netlify.app/' >
+<img target='_blank' src="https://github.com/csvpath/csvpath/blob/main/docs/images/marquez-logo-sm.png" alt="Marquez Server"/></a>
+
+
+Read more about CsvPath and see CSV, Excel, and Data Frames validation examples at <a href='https://www.csvpath.org'>https://www.csvpath.org</a>.
 
 If you need help, use the <a href='https://www.csvpath.org/getting-started/get-help'>contact form</a> or the <a href='https://github.com/csvpath/csvpath/issues'>issue tracker</a> or talk to one of our [sponsors](#sponsors).
 
@@ -55,11 +62,13 @@ If you need help, use the <a href='https://www.csvpath.org/getting-started/get-h
 
 CSV files are everywhere!
 
-A surprisingly large number of companies depend on CSV processing for significant amounts of revenue. Research organizations are awash in CSV. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use day to day uses CSV for sharing. CSV is the lowest of common dominators. Many CSVs are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
+The majority of companies depend on file processing for significant revenue operations. Research organizations and archives are awash in CSVs. And everyone's favorite issue tracker, database GUI, spreadsheet, APM platform, and most any other type of tool we use spits out CSV or Excel files for sharing. Delimited and tabular files are the lowest of common dominators. Many are invalid or broken in some way. Often times a lot of manual effort goes into finding problems and fixing them.
+
+CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract and shape data and create reports.
 
-CsvPath is first and foremost a validation language. It describes tabular data in simple declarative rules that define what valid means for that data. CsvPath can also extract data, create reports, and do other useful things.
+The CsvPath library implements the CsvPath language, but goes far beyond it to provide a full <a href='https://www.atestaanalytics.com/s/The-Collect-Store-Validate-Pattern-Atesta-Analytics.pdf'>Collect, Store, Validate Pattern</a> framework for landing flat files, registering them, validating them, shaping them to a consistent and comparable form, and staging them for a data lake. In that way, CsvPath fills the gap commonly found between an organization's MFT (managed file transfer) and a typical data lake architecture.
 
-CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library, not a system, so it relies on being easy to integrate with other DataOps tools.
+CsvPath's goal is to make simple validations almost trivial and more complex situations more manageable. It is a library and framework, not a system, so it relies on being easy to integrate with other DataOps tools.
 
 
 <a name="install"></a>
@@ -173,8 +182,7 @@ The simplest way to get started is using the CLI. <a href='https://www.csvpath.o
 When you're ready to think about automation, you'll want to start with a simple driver. This is a very basic programmatic use of CsvPath.
 
 ```python
-    path = CsvPath()
-    path.parse("""
+    path = CsvPath().parse("""
             $test.csv[5-25][
                 #firstname == "Frog"
                 @lastname.onmatch = "Bat"

{csvpath-0.0.498 → csvpath-0.0.499}/config/config.ini

@@ -25,13 +25,21 @@ path = cache
 imports = config/functions.imports
 
 [listeners]
+#uncomment for OpenLineage events to a local Marquez
+#file = from csvpath.managers.files.file_listener_ol import OpenLineageFileListener
+#paths = from csvpath.managers.paths.paths_listener_ol import OpenLineagePathsListener
+#result = from csvpath.managers.results.result_listener_ol import OpenLineageResultListener
+#results = from csvpath.managers.results.results_listener_ol import OpenLineageResultsListener
 
 [marquez]
 base_url = http://localhost:5000
+endpoint = api/v1/lineage
+api_key = "none"
+timeout = 5
+verify = False
 
 [results]
 archive = archive
-transfers = transfers
 
 [inputs]
 files = inputs/named_files

csvpath-0.0.499/csvpath/managers/files/file_listener_ol.py

@@ -0,0 +1,6 @@
+from ..ol.ol_listener import OpenLineageListener
+
+
+class OpenLineageFileListener(OpenLineageListener):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

{csvpath-0.0.498 → csvpath-0.0.499}/csvpath/managers/listener.py

@@ -1,10 +1,25 @@
 from abc import ABC, abstractmethod
 from .metadata import Metadata
+from ..util.config import Config
 
 
 class Listener(ABC):
     def __init__(self, config=None) -> None:
-        self.config = config
+        super().__init__()
+        self._config = config
+
+    @property
+    def config(self):
+        if not self._config:
+            #
+            # this really should never happen. but perhaps in testing?
+            #
+            self._config = Config()
+        return self._config
+
+    @config.setter
+    def config(self, c):
+        self._config = c
 
     @abstractmethod
     def metadata_update(self, mdata: Metadata) -> None:

{csvpath-0.0.498 → csvpath-0.0.499}/csvpath/managers/ol/event.py

@@ -42,53 +42,10 @@ class EventBuilder:
             # do we want to support this one, if it comes?
             return None
 
-    """
-    def get_identities_facet(self, mdata):
-        mp = f"{mdata.base_path}{os.sep}{mdata.named_paths_root}{os.sep}{mdata.named_paths_name}/manifest.json"
-        j = []
-        with open(mp, "r", encoding="utf-8") as file:
-            j = json.load(file)
-        d = j[len(j)-1]
-        ps = d["named_paths"]
-        fields=[]
-        for p in d:
-            f = schema_dataset.SchemaDatasetFacetFields(
-                name=p, type="CsvPath", description=""
-            )
-            fields.append(f)
-        csvpaths = self.dataset(
-            f"{mdata.archive_name}/{mdata.named_paths_name}",
-            schema_dataset.SchemaDatasetFacet(fields=fields),
-            mdata.archive_name
-        )
-        return csvpaths
-
-    def dummy_facets(self):
-        print(">>> creating dataset data #{i}" )
-        user_history = self.dataset(
-            "archive",
-            schema_dataset.SchemaDatasetFacet(
-                fields=[
-                    schema_dataset.SchemaDatasetFacetFields(
-                        name="id", type="BIGINT", description="the user id"
-                    ),
-                    schema_dataset.SchemaDatasetFacetFields(
-                        name="email_domain", type="VARCHAR", description="the user id"
-                    ),
-                    schema_dataset.SchemaDatasetFacetFields(
-                        name="status", type="BIGINT", description="the user id"
-                    ),
-                ]
-            ),
-            "archive"
-        )
-        return user_history
-    """
-
     def _build_results_event(self, mdata: Metadata, job, run, facets, inputs):
         file = InputDataset(
             namespace=mdata.archive_name, name=f"{mdata.named_file_name}"
-        )  # , inputFacets=inputfacets
+        )
         path = InputDataset(
             namespace=mdata.archive_name, name=f"{mdata.named_paths_name}"
         )

csvpath-0.0.499/csvpath/managers/ol/ol_listener.py

@@ -0,0 +1,7 @@
+from ..metadata import Metadata
+from .sender import Sender
+
+
+class OpenLineageListener(Sender):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

csvpath-0.0.499/csvpath/managers/ol/sender.py

@@ -0,0 +1,38 @@
+from abc import ABC
+from openlineage.client import OpenLineageClient
+from openlineage.client.transport.http import (
+    ApiKeyTokenProvider,
+    HttpConfig,
+    HttpCompression,
+    HttpTransport,
+)
+from ..metadata import Metadata
+from ..ol.event import EventBuilder
+from ..listener import Listener
+
+
+class Sender(Listener):
+    def __init__(self, *, config=None, client=None):
+        super().__init__(config)
+        self._client = client
+
+    @property
+    def client(self):
+        if self._client is None:
+            h = HttpConfig(
+                url=self.config._get("marquez", "base_url", "https://backend:5000"),
+                endpoint=self.config._get("marquez", "endpoint", "api/v1/lineage"),
+                timeout=int(self.config._get("marquez", "timeout", 5)),
+                verify=bool(self.config._get("marquez", "verify", False)) is True,
+                auth=ApiKeyTokenProvider(
+                    {"apiKey": self.config._get("marquez", "api_key", "none")}
+                ),
+                compression=HttpCompression.GZIP,
+            )
+            self._client = OpenLineageClient(transport=HttpTransport(h))
+        return self._client
+
+    def metadata_update(self, mdata: Metadata) -> None:
+        es = EventBuilder().build(mdata)
+        for e in es:
+            self.client.emit(e)

csvpath-0.0.499/csvpath/managers/paths/paths_listener_ol.py

@@ -0,0 +1,11 @@
+from openlineage.client import OpenLineageClient
+
+from ..metadata import Metadata
+from ..ol.event import EventBuilder
+from ..ol.sender import Sender
+from ..ol.ol_listener import OpenLineageListener
+
+
+class OpenLineagePathsListener(OpenLineageListener):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

csvpath-0.0.499/csvpath/managers/results/result_listener_ol.py

@@ -0,0 +1,6 @@
+from ..ol.ol_listener import OpenLineageListener
+
+
+class OpenLineageResultListener(OpenLineageListener):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

csvpath-0.0.499/csvpath/managers/results/results_listener_ol.py

@@ -0,0 +1,6 @@
+from ..ol.ol_listener import OpenLineageListener
+
+
+class OpenLineageResultsListener(OpenLineageListener):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

csvpath-0.0.499/csvpath/managers/run/run_listener_ol.py

@@ -0,0 +1,6 @@
+from ..ol.ol_listener import OpenLineageListener
+
+
+class OpenLineageResultsListener(OpenLineageListener):
+    def __init__(self, config=None, client=None):
+        super().__init__(config=config, client=client)

{csvpath-0.0.498 → csvpath-0.0.499}/csvpath/util/config.py

@@ -107,7 +107,7 @@ class Config:
     def config_path(self) -> str:
         return self._configpath
 
-    def _get(self, section: str, name: str):
+    def _get(self, section: str, name: str, default=None):
         if self._config is None:
             raise ConfigurationException("No config object available")
         try:
@@ -121,7 +121,7 @@ class Config:
         except KeyError:
             if self.csvpath_log_level == LogLevels.DEBUG:
                 print(f"Check config at {self.config_path} for [{section}][{name}]")
-            return None
+            return default
 
     def add_to_config(self, section, key, value) -> None:
         if not self._config.has_section(section):
@@ -176,6 +176,10 @@ path =
 #results = from csvpath.managers.results.results_listener_ol import OpenLineageResultsListener
 #[marquez]
 #base_url = http://localhost:5000
+#endpoint = api/v1/lineage
+#api_key = "none"
+#timeout = 5
+#verify = False
 #
 [results]
 archive = archive

csvpath-0.0.499/docs/config.md

@@ -0,0 +1,109 @@
+
+# Config
+
+CsvPaths has a few config options. By default, the config options are in `./config/config.ini`. You can change the location of your .ini file in two ways:
+- Set a `CSVPATH_CONFIG_FILE` env var pointing to your file
+- Create an instance of CsvPathConfig, set its CONFIG property, and call the `reload()` method
+
+The config options, at this time, are about:
+- File system locations
+- File extensions
+- Error handling
+- Logging
+- Event listeners
+- Custom functions
+
+## File System Locations
+
+CsvPath stores files in three places:
+- The data staging location
+- The csvpath files location
+- An archive or namespace of results
+
+The first two are in the `[inputs]` section as `files` and `csvpaths`. The default location for data files and csvpath files is under the `./inputs` directory. Each has its own folder. You can move these two locations anywhere you like.
+
+The archive is set in the `[results]` section as `archive`. By default it is a directory named `archive`. You can name the archive anything you like. Keep in mind that as well as simply storing files, the archive is also a namespacing tool. If you have many data partners or separate data operations you may want to have separate archives. If you do use separate archives and you are running OpenLineage events you will see your events namespaced by archive name. See below for configuring OpenLineage event listeners.
+
+In addition, there are cache, config, and log file locations. They have sensible defaults but can be moved, if needed.
+
+## File Extensions
+
+There are two types of files you can set extensions for:
+- CSV files
+- CsvPath files
+
+The defaults for these are:
+
+```ini
+    [csvpath_files]
+    extensions = txt, csvpath
+
+    [csv_files]
+    extensions = txt, csv, tsv, dat, tab, psv, ssv
+```
+
+## Error Handling
+
+The error settings are for when CsvPath or CsvPaths instances encounter problems. The options are:
+- `stop` - Halt processing; the CsvPath stopped property is set to True
+- `fail` - Mark the currently running CsvPath as having failed
+- `raise` - Raise the exception in as noisy a way as possible
+- `quiet` - Do nothing that affects the system out; this protects command line redirection of `print()` output. Logging is also minimized such that errors that would release a lot of metadata are slimmed down.
+- `collect` - Collect the errors in the error results for the CsvPath. This option is available with and without a CsvPaths instance.
+- `print` - Prints the errors using the Printer interface to whatever printers are available. By default this goes to standard out.
+
+Multiple of these settings can be configured together.`quiet` and `raise` do not coexist well; likewise `quiet` and `print`. `raise` will win over `quiet` because seeing problems lets you fix them. `print` is most useful in getting simple inline error messages when `raise` is off.
+
+## Logging
+
+Logging levels are set at the major-component level. The components are:
+- `csvpath`
+- `csvpaths`
+- `matcher`
+- `scanner`
+
+Four levels are available:
+- `error`
+- `warning`
+- `debug`
+- `info`
+
+The levels are intended for the same functionality as their Python equivalents.
+
+CsvPath logs are directed to a file. The log file settings are:
+- `log_file` - a path to the log
+- `log_files_to_keep` - a number of logs, 1 to 100, kept in rotation before being deleted
+- `log_file_size` - an indication of roughly when a log file will be rotated
+
+As an example:
+```ini
+    log_file = logs/csvpath.log
+    log_files_to_keep = 100
+    log_file_size = 52428800
+```
+
+## Listeners
+
+CsvPath generates events that it converts to manifest files full of asset and runtime metadata. You can add OpenLineage listeners that will send results to an OpenLineage server like Marquez. In principle any OpenLineage API could receive CsvPath events, but only Marquez is tested and supported.
+
+Be aware, OpenLineage events are currently handled in line, not out of band, asynchronously. That means there is a small performance hit. Typically this would not be noticeable, but in certain instances it could be a factor. For example, CsvPath's hundreds of unit tests run slower when OpenLineage events are fired. This small performance hit may be remediated in the future if it becomes an issue.
+
+The settings are:
+```ini
+    [listeners]
+    #uncomment for OpenLineage events to a Marquez server
+    #file = from csvpath.managers.files.file_listener_ol import OpenLineageFileListener
+    #paths = from csvpath.managers.paths.paths_listener_ol import OpenLineagePathsListener
+    #result = from csvpath.managers.results.result_listener_ol import OpenLineageResultListener
+    #results = from csvpath.managers.results.results_listener_ol import OpenLineageResultsListener
+
+    [marquez]
+    base_url = http://localhost:5000
+```
+
+## Custom Functions
+
+<a href='https://github.com/csvpath/csvpath/blob/main/docs/functions/implementing_functions.md'>See this page for how to create and run custom functions</a>
+
+
+

{csvpath-0.0.498 → csvpath-0.0.499}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "csvpath"
-version = "0.0.498"
+version = "0.0.499"
 description = "A declarative language for validating CSV, Excel, and other tabular data files"
 authors = ["David Kershaw <dk107dk@hotmail.com>"]
 readme = "README.md"

csvpath-0.0.498/csvpath/managers/files/file_listener_ol.py

@@ -1,27 +0,0 @@
-from openlineage.client import OpenLineageClient
-
-from ..metadata import Metadata
-from ..listener import Listener
-from ..ol.event import EventBuilder
-from ..ol.job import JobBuilder
-
-
-class OpenLineageFileListener(Listener):
-    def __init__(self, config=None):
-        super().__init__(config)
-        self.ol_client = None
-
-    def metadata_update(self, mdata: Metadata) -> None:
-        if self.ol_client is None:
-            client_url = self.config._get("marquez", "base_url")
-            if client_url is None:
-                print(
-                    "WARNING: OpenLineage listeners are live but there is no Marquez API URL"
-                )
-                return
-                # client_url = "http://localhost:5000"
-            self.ol_client = OpenLineageClient(url=client_url)
-
-        es = EventBuilder().build(mdata)
-        for e in es:
-            self.ol_client.emit(e)

csvpath-0.0.498/csvpath/managers/paths/paths_listener_ol.py

@@ -1,27 +0,0 @@
-from openlineage.client.client import OpenLineageClient
-
-from ..metadata import Metadata
-from ..listener import Listener
-from ..ol.event import EventBuilder
-from ..ol.job import JobBuilder
-
-
-class OpenLineagePathsListener(Listener):
-    def __init__(self, config=None):
-        super().__init__(config)
-        self.ol_client = None
-
-    def metadata_update(self, mdata: Metadata) -> None:
-        if self.ol_client is None:
-            client_url = self.config._get("marquez", "base_url")
-            if client_url is None:
-                print(
-                    "WARNING: OpenLineage listeners are live but there is no Marquez API URL"
-                )
-                return
-                # client_url = "http://localhost:5000"
-            self.ol_client = OpenLineageClient(url=client_url)
-
-        es = EventBuilder().build(mdata)
-        for e in es:
-            self.ol_client.emit(e)

csvpath-0.0.498/csvpath/managers/results/result_listener_ol.py

@@ -1,27 +0,0 @@
-from openlineage.client.client import OpenLineageClient
-
-from ..metadata import Metadata
-from ..listener import Listener
-from ..ol.event import EventBuilder
-from ..ol.job import JobBuilder
-
-
-class OpenLineageResultListener(Listener):
-    def __init__(self, config=None):
-        super().__init__(config)
-        self.ol_client = None
-
-    def metadata_update(self, mdata: Metadata) -> None:
-        if self.ol_client is None:
-            client_url = self.config._get("marquez", "base_url")
-            if client_url is None:
-                print(
-                    "WARNING: OpenLineage listeners are live but there is no Marquez API URL"
-                )
-                return
-                # client_url = "http://localhost:5000"
-            self.ol_client = OpenLineageClient(url=client_url)
-
-        es = EventBuilder().build(mdata)
-        for e in es:
-            self.ol_client.emit(e)

csvpath-0.0.498/csvpath/managers/results/results_listener_ol.py

@@ -1,28 +0,0 @@
-from openlineage.client.client import OpenLineageClient
-
-
-from ..metadata import Metadata
-from ..listener import Listener
-from ..ol.event import EventBuilder
-from ..ol.job import JobBuilder
-
-
-class OpenLineageResultsListener(Listener):
-    def __init__(self, config=None):
-        super().__init__(config)
-        self.ol_client = None
-
-    def metadata_update(self, mdata: Metadata) -> None:
-        if self.ol_client is None:
-            client_url = self.config._get("marquez", "base_url")
-            if client_url is None:
-                print(
-                    "WARNING: OpenLineage listeners are live but there is no Marquez API URL"
-                )
-                return
-                # client_url = "http://localhost:5000"
-            self.ol_client = OpenLineageClient(url=client_url)
-
-        es = EventBuilder().build(mdata)
-        for e in es:
-            self.ol_client.emit(e)

csvpath-0.0.498/csvpath/managers/run/run_listener_ol.py

@@ -1,30 +0,0 @@
-from openlineage.client.client import OpenLineageClient
-from marquez_client import MarquezClient
-
-from ..metadata import Metadata
-from ..listener import Listener
-from ..ol.event import EventBuilder
-from ..ol.job import JobBuilder
-from ..ol.run import RunBuilder
-
-
-class OpenLineageRunListener(Listener):
-    def __init__(self, config=None):
-        super().__init__(config)
-        self.client = None
-        self.ol_client = None
-
-    def metadata_update(self, mdata: Metadata) -> None:
-        if self.client is None:
-            client_url = self.config._get("marquez", "base_url")
-            if client_url is None:
-                print(
-                    "WARNING: OpenLineage listeners are live but there is no Marquez API URL"
-                )
-                return
-                # client_url = "http://localhost:5000"
-            self.client = MarquezClient(url=client_url)
-            self.ol_client = OpenLineageClient(url=client_url)
-        es = EventBuilder().build(mdata)
-        for e in es:
-            self.ol_client.emit(e)