data-validation-engine 0.6.2__py3-none-any.whl

data_validation_engine-0.6.2.dist-info/METADATA

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: data-validation-engine
+Version: 0.6.2
+Summary: `nhs data validation engine` is a framework used to validate data
+License-File: LICENSE
+Author: NHS England
+Author-email: england.contactus@nhs.net
+Requires-Python: >=3.10,<3.12
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Dist: Jinja2 (==3.1.*)
+Requires-Dist: boto3 (>=1.34.162,<1.36)
+Requires-Dist: botocore (>=1.34.162,<1.36)
+Requires-Dist: delta-spark (==2.4.*)
+Requires-Dist: duckdb (==1.1.*)
+Requires-Dist: lxml (>=4.9.1,<5.0.0)
+Requires-Dist: openpyxl (>=3.1,<4.0)
+Requires-Dist: pandas (>=2.2.2,<3.0.0)
+Requires-Dist: polars (==0.20.*)
+Requires-Dist: pyarrow (>=17.0.0,<18.0.0)
+Requires-Dist: pydantic (==1.10.15)
+Requires-Dist: pyspark (==3.4.*)
+Requires-Dist: typing_extensions (>=4.6.2,<5.0.0)
+Description-Content-Type: text/markdown
+
+<h1 style="display: flex; align-items: center; gap: 10px;">
+    <img src="overrides/.icons/nhseng.svg" alt="NHS Logo" width="5%" height="100%" align="left">
+    Data Validation Engine
+</h1>
+
+![License](https://img.shields.io/github/license/NHSDigital/data-validation-engine)
+![Version](https://img.shields.io/github/v/release/NHSDigital/data-validation-engine)
+[![CI Unit Tests](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml)
+[![CI Formatting & Linting](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml)
+
+The Data Validation Engine (DVE) is a configuration driven data validation library built and utilised by NHS England. Currently the package has been reverted from v1.0.0 release to a 0.x as we feel the package is not yet mature enough to be considered a 1.0.0 release. So please bear this in mind if reading through the commits and references to a v1+ release when on v0.x.
+
+As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` file and example files can be accessed [here](./tests/testdata/). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here](./docs/).
+
+Once a dischema file has been defined, you are ready to use the DVE. The DVE is typically orchestrated based on four key "services". These are...
+
+|    | Service | Purpose |
+| -- | ------- | ------- |
+| 1. | File Transformation | This service will take submitted files and turn them into stringified parquet file(s) to ensure that a consistent data structure can be passed through the other services. |
+| 2. | Data Contract | This service will validate and perform type casting against a stringified parquet file using [pydantic models](https://docs.pydantic.dev/1.10/). |
+| 3. | Business Rules | The business rules service will perform more complex validations such as comparisons between fields and tables, aggregations, filters etc to generate new entities. |
+| 4. | Error Reports | The error reports service will take all the errors raised in previous services and surface them into a readable format for a downstream users/service. Currently, this implemented to be an excel spreadsheet but could be reconfigured to meet other requirements/use cases. |
+
+If you'd like more detailed documentation around these services the please read the extended documentation [here](./docs/).
+
+The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
+
+Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing][#Contributing] section.
+
+## Installation and usage
+
+The DVE is a Python package and can be installed using `pip`. As of release v0.6.1 we currently support Python 3.10 & 3.11, with Spark version 3.4 and DuckDB version of 1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
+
+If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
+
+Python dependencies are listed in `pyproject.toml`.
+
+To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
+
+```
+pip install git+https://github.com/NHSDigital/data-validation-engine.git@v0.6.1
+```
+
+Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
+
+Please note - The long term aim is to make the DVE available via PyPi and Conda but we are not quite there yet. Once available this documentation will be updated to contain the new installation options.
+
+Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
+
+## Requesting new features and raising bug reports
+**Before creating new issues, please check to see if the same bug/feature has been created already. Where a duplicate is created, the ticket will be closed and referenced to an existing issue.**
+
+If you have spotted a bug with the DVE then please raise an issue [here](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
+
+If you have feature request then please follow the same process whilst using the "Feature request template".
+
+## Upcoming features
+Below is a list of features that we would like to implement or have been requested.
+| Feature | Release Version | Released? |
+| ------- | --------------- | --------- |
+| Open source release | 0.1.0 | Yes |
+| Uplift to Python 3.11 | 0.2.0 | Yes |
+| Upgrade to Pydantic 2.0 | Not yet confirmed | No |
+| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
+
+Beyond the Python upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
+
+## Contributing
+Please see guidance [here](./CONTRIBUTE.md).
+
+## Legal
+This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.
+
+Any HTML or Markdown documentation is [© Crown copyright](https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/) and available under the terms of the [Open Government 3.0 licence](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).
+

data_validation_engine-0.6.2.dist-info/RECORD

@@ -0,0 +1,105 @@
+dve/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dve/common/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dve/common/error_utils.py,sha256=uy4kl3U9ZxturTXDkZ0gegSQIf36AgwQsgY4oe14fP8,6545
+dve/core_engine/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dve/core_engine/backends/__init__.py,sha256=Mb0VB9fDUw2wkncbHUwAegwyAEh1XI30SA2sJkz4oHk,51
+dve/core_engine/backends/base/__init__.py,sha256=a7reDmK2Jwn57iW1tcUc6VWsCQWt2Q8QffNXVas9bc0,39
+dve/core_engine/backends/base/auditing.py,sha256=k9bqgnYLH2_ReuRsBGomidaOBs3iBuC7Cp4lAx_SmPQ,22913
+dve/core_engine/backends/base/backend.py,sha256=gZiyFCrOCy-GWmMyeMqHnwXJTO4su55wsQleUWmA9nM,9992
+dve/core_engine/backends/base/contract.py,sha256=1lpBb335ABOSPrtUFPXo66r8_uMTzd-L2IZKjKvupqs,17842
+dve/core_engine/backends/base/core.py,sha256=YKLkalATdJ_z1c5y2qmJjU-193Km09tjQDlv_3YDO00,4766
+dve/core_engine/backends/base/reader.py,sha256=Iv3D8J_nXvGOquf0pJhiiOCcB1zUyvEcLL1tN4Y3m9A,6529
+dve/core_engine/backends/base/reference_data.py,sha256=M3HnzbC0r3V6qVNUz94-5Rod3I6WmO-xlMsp0Q5heJQ,8102
+dve/core_engine/backends/base/rules.py,sha256=3V3mxHAEj8A-LlS6_dpaRqgzPfBSubPw2oA5iyPV_5g,28734
+dve/core_engine/backends/base/utilities.py,sha256=nY0z-UOFqyrAO7jwu0j2nOZyQ5kuMVGQtNCi2JjpvnI,5148
+dve/core_engine/backends/exceptions.py,sha256=1YEzFaGMTkJTUwmoyzownByrgPCvBSQDG3DtYLswRJA,11143
+dve/core_engine/backends/implementations/__init__.py,sha256=7dB6WqLni3JHk3zcQdlECZkzpFLfrzhhoiqiWn_ruAg,62
+dve/core_engine/backends/implementations/duckdb/__init__.py,sha256=IjGIYTGOSSKqbz5C3yE2fvcaq7l5fjYu41fW8hIdp_M,761
+dve/core_engine/backends/implementations/duckdb/auditing.py,sha256=-d9VaqDF9hcopRoD_UXzxkjQoNuPa9s3H4vt1Oa8-4M,8684
+dve/core_engine/backends/implementations/duckdb/contract.py,sha256=UeJ4xdNwE1EmomzfRtF5oQvPsPkMoQ1app_4nW64_pY,9063
+dve/core_engine/backends/implementations/duckdb/duckdb_helpers.py,sha256=1H7MyiRI_RIYSfL5I0gJ_37ugx3Va5CYyS1xn84oKdc,11013
+dve/core_engine/backends/implementations/duckdb/readers/__init__.py,sha256=qyyFQX64LQGlpR25Ofud1YvADP3VWRVz8oS_P40jjTQ,367
+dve/core_engine/backends/implementations/duckdb/readers/csv.py,sha256=0yPO-Xb-c5gMHz6pZqoIXyCycyTh-eOoCYAuo-pkuzk,9451
+dve/core_engine/backends/implementations/duckdb/readers/json.py,sha256=lC7xEprsoKOGxP4IgGR9jQqeOUfu5URP7lj_uGzrwCg,1719
+dve/core_engine/backends/implementations/duckdb/readers/xml.py,sha256=HXpY59oNLgJEC-hKudqbvL-CjzbnBRgP9td9eHh_hMM,1869
+dve/core_engine/backends/implementations/duckdb/reference_data.py,sha256=r6pXB4Q0ti2NRLCxbiVEUu53FnSfwR7PAVX5aiYfF60,1774
+dve/core_engine/backends/implementations/duckdb/rules.py,sha256=s9W_bxzRekUIXyQEmZaqOB3Q_g4IEf-ksNj7b41-Znk,23168
+dve/core_engine/backends/implementations/duckdb/types.py,sha256=EP-CioAT5fEPCKOjIFA8mHbWZlD27RHtz_NkDS_3Kqo,1061
+dve/core_engine/backends/implementations/duckdb/utilities.py,sha256=0ZybsiYUu0tOqvzUHxebGZJ-GXf0pl48haLSEgMt85M,1365
+dve/core_engine/backends/implementations/spark/__init__.py,sha256=o5GWNbAXQ3PWryLaBpRBrhyBf9qkXypayEQAzFBp53g,624
+dve/core_engine/backends/implementations/spark/auditing.py,sha256=9lc7CvPi146eqGBZ7wlcOIQLgJbSBeukW2M_FHA3eDQ,8753
+dve/core_engine/backends/implementations/spark/backend.py,sha256=V5qL6T0FDl6n-VMQIJtY9qwg-kzxqPYTbzMOfw_H1Hc,3471
+dve/core_engine/backends/implementations/spark/contract.py,sha256=TB66HKHdZaL-vxiarf7N56XUMEoYeBdjhsdzYiW3gNg,9565
+dve/core_engine/backends/implementations/spark/readers/__init__.py,sha256=bQYNIPd1OSlvCp3bW3bS_ren3qFxiafM4JBRrmwk_Lg,436
+dve/core_engine/backends/implementations/spark/readers/csv.py,sha256=rA5dhqLy6J3zs0Y7p0mw2YefGJsgz7e_zk8UKdfHqNs,2585
+dve/core_engine/backends/implementations/spark/readers/json.py,sha256=zhVAYRovLJLEMIWoMQeH0pMFjW0Uu1-dPT8iN4XKJb4,2233
+dve/core_engine/backends/implementations/spark/readers/xml.py,sha256=WnSY6Qet6Rn4HzdAJK3HMinEDYK943V3WikbkVU3Z2E,7808
+dve/core_engine/backends/implementations/spark/reference_data.py,sha256=rD6PHckGGNw9-3FIQKueep-ID82am4Q28zLEoOfjiv8,1392
+dve/core_engine/backends/implementations/spark/rules.py,sha256=NjY3CjDwb-FjBbEpY0w0ecLDJlHojz_D9bhjeyN3MFc,18336
+dve/core_engine/backends/implementations/spark/spark_helpers.py,sha256=OoddXPTGEb0UHLkT4V_tDxMAIad_gQ3aBbxPyZfl5DQ,15216
+dve/core_engine/backends/implementations/spark/types.py,sha256=IkJd6-wjFiaUdcOyQUM7XFS9olWJ3YYqLlY9yzy6u58,623
+dve/core_engine/backends/implementations/spark/utilities.py,sha256=DRdDaQvIB5lgzCJLvxrOuLU7bkmqXk4gQGOMyf8FgVQ,4505
+dve/core_engine/backends/metadata/__init__.py,sha256=LrJQinyvt3QK8o-NpMmY7nV0E6ncaUQNd9ibqyFj1Y4,1171
+dve/core_engine/backends/metadata/contract.py,sha256=Pk53xxrMXNbTSHrGRDvJqpwFOYnz5MYtPEnKHWNQT7g,2859
+dve/core_engine/backends/metadata/reporting.py,sha256=N3O0Ve9I1NKnn40wwl1dS58C5OjanqLo2V9Fc8WLtyU,14143
+dve/core_engine/backends/metadata/rules.py,sha256=b1N5m2kNiUoXNTxb6GwU-M4jIaudhMPOKSh2l0K9W3g,25463
+dve/core_engine/backends/readers/__init__.py,sha256=Gpq3xAiXOk8KRHmOt5pgu6P-v08LflaJiPW51h8wrEo,1366
+dve/core_engine/backends/readers/csv.py,sha256=wKZbJxXbwuc_Xdj1PDDhX_qxekZiXsjFj4B7iahLN3o,8901
+dve/core_engine/backends/readers/utilities.py,sha256=yUfSMeRfaBcYQph1RF0GqNrCF_3C6Ffy8w4NHIeUfpk,677
+dve/core_engine/backends/readers/xml.py,sha256=d5-YQDKaFPTIh3_h_K1ogJTS0stmmm7cwaeUAR5XeDA,17160
+dve/core_engine/backends/readers/xml_linting.py,sha256=a5c4FFjFn3rowjCg3V793zLOaOgmp1W3qPUnIknQhug,5422
+dve/core_engine/backends/types.py,sha256=jn1knqFrxMMzchjpkDyDI3w2tR39N6XFk4oJS-LeQeQ,680
+dve/core_engine/backends/utilities.py,sha256=mJspa4p2fJXcR8WPe24AsC_DWG9ZOqTe6Wwf1LChsQQ,7515
+dve/core_engine/configuration/__init__.py,sha256=CLlpunGvqJcavVYRU-4KL7whI4_rbFlWfIbKNLmf7MQ,35
+dve/core_engine/configuration/base.py,sha256=YUqag6m-xl8BVVRKXjwz0phznnPt5lvLGZX6Fn7ZnCE,1881
+dve/core_engine/configuration/v1/__init__.py,sha256=qMFNKX_uTfLh5SFKcyyeZA0OS0Jz7jOdfF3XXYqxP2E,14088
+dve/core_engine/configuration/v1/filters.py,sha256=ULPvGHjBvIqlUivbW1AoNRjINFE8WjlwXroIqcQVp-0,2076
+dve/core_engine/configuration/v1/rule_stores/__init__.py,sha256=z6j1fyYvRr55uX5AyL4Hdl1FRfFmHM9PDHyECyR7bMo,45
+dve/core_engine/configuration/v1/rule_stores/models.py,sha256=WDszhc9b2BGOfkusZ9XLF9ICkwCNeLhPZPmO4xQ5IRM,1855
+dve/core_engine/configuration/v1/steps.py,sha256=grvlmdfZWtOpoki2ocySxkq4iCfbFem6nVgI82VLDag,10273
+dve/core_engine/constants.py,sha256=_rOMpoo2eeL95DgaCijUKqK-BTazsGpLRt4MCZyaR98,323
+dve/core_engine/engine.py,sha256=TiXyxqkB87dA88xye_cPKwNzBnueEc2V2euil2Qsax0,10428
+dve/core_engine/exceptions.py,sha256=P9ISgn4TULQa_lAjmZjneCPkHh8L_FNduHOCR4BPR7Q,954
+dve/core_engine/functions/__init__.py,sha256=CORD5jh6SQ-VOLQg8KveY7x0CDQ6ExXcN_dKqqk6kjw,168
+dve/core_engine/functions/implementations.py,sha256=tafTfCvbrQAiiEFoUSvuPFosaq8rXU_7GAQV03l714U,4893
+dve/core_engine/loggers.py,sha256=Kfn66ZRrnSOGCHtIgtFG3n4NzEjx_SZCa3eUkZzh1rc,1743
+dve/core_engine/message.py,sha256=YKhKsn7-9-nUw5el9lIDxDGk_HY7BfRQsp8HQ4AX__Y,17677
+dve/core_engine/models.py,sha256=xS0Py7BcoNei0aReiPEprZdOEyhhO7maiBILPld6Gc4,7573
+dve/core_engine/templating.py,sha256=WZJxxD9FgtbkFHXVvqgEI1mNT092v_Ql_HHWHGlaTZM,3854
+dve/core_engine/type_hints.py,sha256=2ciOVkYS10n-BfjziOf0Ulrp7v53EcSgufhh1Fk1j2Y,8422
+dve/core_engine/validation.py,sha256=T0OSoc8lY-ibqxQ7ubBUs0Pb3eMsy0Rn_9CRA_RX_vY,6127
+dve/metadata_parser/__init__.py,sha256=aW5V8AaufpFiB_UVrazDBDb3DI4ac8_LJz82EnV2hd4,107
+dve/metadata_parser/domain_types.py,sha256=xZ-aWVEdv3a8ZUWY2IR82SeQufLvqoWaPtMbC0-l-Rk,24194
+dve/metadata_parser/exc.py,sha256=ngP2JVRPEfuZ8Pk7q4LPuEd6AbtY9LNzIV8VponWitA,1108
+dve/metadata_parser/function_library.py,sha256=h7WQjH3fV17IoPwvDnvCtRCH1_xIcnXThqgNdrb91yk,1640
+dve/metadata_parser/function_wrapper.py,sha256=G4kY-AMJzrcS6yvI2PmPngUWCFEuj1aGeSI4CaBbtY4,6666
+dve/metadata_parser/model_generator.py,sha256=wErxpnpTPgFRwZIAve8-MDU3hqRS5XdoKd-uzjqhujg,4142
+dve/metadata_parser/models.py,sha256=ylqct8_PKHztd7rcueXrfjooGYRd_qScNytsBD-WAkc,15930
+dve/metadata_parser/utilities.py,sha256=BtpbuVfgty7dcGFZggKIqj0XCBQV6a9bK0CA0BeB4-E,1588
+dve/parser/__init__.py,sha256=8uVxu0C5drBMQI3ssFgq4-KpdUCr2bK5Wph7XFs1JgU,45
+dve/parser/exceptions.py,sha256=zCe634MpBlBq-bTX8F2s98tnnoXbsAuFc_eR7B_zj6o,1319
+dve/parser/file_handling/__init__.py,sha256=57VgwNocd3r5CVBlvNFJO5yKdH_Nv3ZjX-vz0kdQSc8,788
+dve/parser/file_handling/helpers.py,sha256=_vTDA5jQvfRs66FNcZncBc-lDUhTVPYeB470V_sPnNc,868
+dve/parser/file_handling/implementations/__init__.py,sha256=xI43lMBceGVjK7VhEosblO5iooquT3zoztUXT0C9CRI,261
+dve/parser/file_handling/implementations/base.py,sha256=IvXnqKWb6te2ERY-_-W-QOS3BA-ZfS34SitEPDQQN2o,3525
+dve/parser/file_handling/implementations/dbfs.py,sha256=lF-C1zjw_irAiXHKc5RFRwayAZwApijBLM-wS-qgBog,2967
+dve/parser/file_handling/implementations/file.py,sha256=Up32oANER-nNmFhIiF22f9PZ2NlaFbCHWU0LiFXAe4w,8150
+dve/parser/file_handling/implementations/s3.py,sha256=AAV02n2uBddIqvk8_C7b1gcWC6L1XSlqlH7dNwOj8cs,15084
+dve/parser/file_handling/log_handler.py,sha256=wZcs0OFxWarYyB9aPPSLtZ4RiLf2AN9yoRxm69ya9u4,7492
+dve/parser/file_handling/service.py,sha256=AdIniT1TYaPzM5mYW-_lxrfK7SGvf1KFcY34V9_7i8c,15942
+dve/parser/file_handling/utilities.py,sha256=5BrQ8-ip9nZvAym1fSNvDZT5rcJ7l8dZcHrG_ny2XxI,1775
+dve/parser/type_hints.py,sha256=S82emxVeOrZ4K9ei4A2bEHu2OmjjVCNQ5_HCZR80P4Y,1477
+dve/parser/utilities.py,sha256=QoffR54nCdu_S5Na_Feq5tpFWAIt1xogPzlr4B8Bopo,3809
+dve/pipeline/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dve/pipeline/duckdb_pipeline.py,sha256=bl33gWglqqP_YaZV7gOQcfta6aJRPZSNj4FIWBPfILM,2073
+dve/pipeline/foundry_ddb_pipeline.py,sha256=2JWsEnWZbBrqsObvcrbGnCiV1pZblTT6WRV7GJnu-fs,8395
+dve/pipeline/pipeline.py,sha256=OCZHkkBrg0uoKdx_mqYgiXcA0wt5938rN2Ab-89Y8vk,40637
+dve/pipeline/spark_pipeline.py,sha256=jgDW8FcLuihQocnOVIzSWKM63EIlUL-C8ez3QTcbjVw,2573
+dve/pipeline/utils.py,sha256=KvRr43OEku9Kl1oJIc94JEsDgly7DoK_i68HLZ5LDuk,3410
+dve/reporting/__init__.py,sha256=mWauBREIYl8OBUsmrxIpLqEuDTBWyRSimB8p7_GYZE0,28
+dve/reporting/error_report.py,sha256=kxTcSNbmgDCBCVUGVcrdn10xjZq77zN7cu7uf1i0kDw,5365
+dve/reporting/excel_report.py,sha256=Oipi7gmd0ijmX0VTsfdEfzfpus_a1NpIlNQ87kKRoeU,12062
+data_validation_engine-0.6.2.dist-info/METADATA,sha256=DSBlLMjhDV6o6606T2T24Ga8FVj9ngHPUtfptoMdYZY,7798
+data_validation_engine-0.6.2.dist-info/WHEEL,sha256=zp0Cn7JsFoX2ATtOhtaFYIiE2rmFAD4OcMhtUki8W3U,88
+data_validation_engine-0.6.2.dist-info/licenses/LICENSE,sha256=weoppAS71QMtLuFT0rITA0GAQPos-b7-_PGlL0Bsqvc,1085
+data_validation_engine-0.6.2.dist-info/RECORD,,

data_validation_engine-0.6.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.2.1
+Root-Is-Purelib: true
+Tag: py3-none-any

data_validation_engine-0.6.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Crown Copyright NHS England.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dve/common/error_utils.py

@@ -0,0 +1,189 @@
+"""Utilities to support reporting"""
+
+import datetime as dt
+import json
+import logging
+from collections.abc import Iterable
+from itertools import chain
+from multiprocessing import Queue
+from threading import Thread
+from typing import Optional, Union
+
+import dve.parser.file_handling as fh
+from dve.core_engine.exceptions import CriticalProcessingError
+from dve.core_engine.loggers import get_logger
+from dve.core_engine.message import UserMessage
+from dve.core_engine.type_hints import URI, DVEStageName, Messages
+
+
+def get_feedback_errors_uri(working_folder: URI, step_name: DVEStageName) -> URI:
+    """Determine the location of json lines file containing all errors generated in a step"""
+    return fh.joinuri(working_folder, "errors", f"{step_name}_errors.jsonl")
+
+
+def get_processing_errors_uri(working_folder: URI) -> URI:
+    """Determine the location of json lines file containing all processing
+    errors generated from DVE run"""
+    return fh.joinuri(working_folder, "processing_errors", "processing_errors.jsonl")
+
+
+def dump_feedback_errors(
+    working_folder: URI,
+    step_name: DVEStageName,
+    messages: Messages,
+    key_fields: Optional[dict[str, list[str]]] = None,
+) -> URI:
+    """Write out captured feedback error messages."""
+    if not working_folder:
+        raise AttributeError("processed files path not passed")
+
+    if not key_fields:
+        key_fields = {}
+
+    error_file = get_feedback_errors_uri(working_folder, step_name)
+    processed = []
+
+    for message in messages:
+        if message.original_entity is not None:
+            primary_keys = key_fields.get(message.original_entity, [])
+        elif message.entity is not None:
+            primary_keys = key_fields.get(message.entity, [])
+        else:
+            primary_keys = []
+
+        error = message.to_dict(
+            key_field=primary_keys,
+            value_separator=" -- ",
+            max_number_of_values=10,
+            record_converter=None,
+        )
+        error["Key"] = conditional_cast(error["Key"], primary_keys, value_separator=" -- ")
+        processed.append(error)
+
+    with fh.open_stream(error_file, "a") as f:
+        f.write("\n".join([json.dumps(rec, default=str) for rec in processed]) + "\n")
+    return error_file
+
+
+def dump_processing_errors(
+    working_folder: URI, step_name: str, errors: list[CriticalProcessingError]
+):
+    """Write out critical processing errors"""
+    if not working_folder:
+        raise AttributeError("processed files path not passed")
+    if not step_name:
+        raise AttributeError("step name not passed")
+    if not errors:
+        raise AttributeError("errors list not passed")
+
+    error_file: URI = get_processing_errors_uri(working_folder)
+    processed = []
+
+    for error in errors:
+        processed.append(
+            {
+                "step_name": step_name,
+                "error_location": "processing",
+                "error_level": "integrity",
+                "error_message": error.error_message,
+                "error_traceback": error.messages,
+            }
+        )
+
+    with fh.open_stream(error_file, "a") as f:
+        f.write("\n".join([json.dumps(rec, default=str) for rec in processed]) + "\n")
+
+    return error_file
+
+
+def load_feedback_messages(feedback_messages_uri: URI) -> Iterable[UserMessage]:
+    """Load user messages from jsonl file"""
+    if not fh.get_resource_exists(feedback_messages_uri):
+        return
+    with fh.open_stream(feedback_messages_uri) as errs:
+        yield from (UserMessage(**json.loads(err)) for err in errs.readlines())
+
+
+def load_all_error_messages(error_directory_uri: URI) -> Iterable[UserMessage]:
+    "Load user messages from all jsonl files"
+    return chain.from_iterable(
+        [
+            load_feedback_messages(err_file)
+            for err_file, _ in fh.iter_prefix(error_directory_uri)
+            if err_file.endswith(".jsonl")
+        ]
+    )
+
+
+class BackgroundMessageWriter:
+    """Controls batch writes to error jsonl files"""
+
+    def __init__(
+        self,
+        working_directory: URI,
+        dve_stage: DVEStageName,
+        key_fields: Optional[dict[str, list[str]]] = None,
+        logger: Optional[logging.Logger] = None,
+    ):
+        self._working_directory = working_directory
+        self._dve_stage = dve_stage
+        self._feedback_message_uri = get_feedback_errors_uri(
+            self._working_directory, self._dve_stage
+        )
+        self._key_fields = key_fields
+        self.logger = logger or get_logger(type(self).__name__)
+        self._write_thread: Optional[Thread] = None
+        self._queue: Queue = Queue()
+
+    @property
+    def write_queue(self) -> Queue:  # type: ignore
+        """Queue for storing batches of messages to be written"""
+        return self._queue
+
+    @property
+    def write_thread(self) -> Thread:  # type: ignore
+        """Thread to write batches of messages to jsonl file"""
+        if not self._write_thread:
+            self._write_thread = Thread(target=self._write_process_wrapper)
+        return self._write_thread
+
+    def _write_process_wrapper(self):
+        """Wrapper for dump feedback errors to run in background process"""
+        # writing thread will block if nothing in queue
+        while True:
+            if msgs := self.write_queue.get():
+                dump_feedback_errors(
+                    self._working_directory, self._dve_stage, msgs, self._key_fields
+                )
+            else:
+                break
+
+    def __enter__(self) -> "BackgroundMessageWriter":
+        self.write_thread.start()
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        if exc_type:
+            self.logger.exception(
+                "Issue occured during background write process:",
+                exc_info=(exc_type, exc_value, traceback),
+            )
+        # None value in queue will trigger break in target
+        self.write_queue.put(None)
+        self.write_thread.join()
+
+
+def conditional_cast(value, primary_keys: list[str], value_separator: str) -> Union[list[str], str]:
+    """Determines what to do with a value coming back from the error list"""
+    if isinstance(value, list):
+        casts = [
+            conditional_cast(val, primary_keys, value_separator) for val in value
+        ]  # type: ignore
+        return value_separator.join(
+            [f"{pk}: {id}" if pk else "" for pk, id in zip(primary_keys, casts)]
+        )
+    if isinstance(value, dt.date):
+        return value.isoformat()
+    if isinstance(value, dict):
+        return ""
+    return str(value)

dve/core_engine/backends/__init__.py

@@ -0,0 +1 @@
+"""Backend implementations for the core engine."""

dve/core_engine/backends/base/__init__.py

@@ -0,0 +1 @@
+"""The base backend implementation."""