data-validation-engine 0.8.4__tar.gz → 0.9.0__tar.gz

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-validation-engine
-Version: 0.8.4
+Version: 0.9.0
 Summary: `nhs data validation engine` is a framework used to validate data
 License-Expression: MIT
 License-File: LICENSE
@@ -11,7 +11,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Classifier: Operating System :: OS Independent
+Classifier: Operating System :: POSIX :: Linux
 Classifier: Topic :: Software Development :: Libraries
 Classifier: Typing :: Typed
 Requires-Dist: Jinja2 (==3.1.6)
@@ -25,7 +25,7 @@ Requires-Dist: openpyxl (==3.1.5)
 Requires-Dist: pandas (==2.3.3)
 Requires-Dist: polars (==0.20.31)
 Requires-Dist: pyarrow (==23.0.1)
-Requires-Dist: pydantic (==1.10.19)
+Requires-Dist: pydantic (==2.13.4)
 Requires-Dist: pyspark (>=3.5.0,<=3.5.5)
 Requires-Dist: typing_extensions (==4.15.0)
 Requires-Dist: urllib3 (==2.7.0)
@@ -37,7 +37,7 @@ Project-URL: Repository, https://github.com/NHSDigital/data-validation-engine.gi
 Description-Content-Type: text/markdown
 
 <h1 style="display: flex; align-items: center; gap: 10px;">
-    <img src="https://github.com/NHSDigital/data-validation-engine/blob/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
+    <img src="https://raw.githubusercontent.com/NHSDigital/data-validation-engine/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
     Data Validation Engine
 </h1>
 
@@ -47,49 +47,27 @@ Description-Content-Type: text/markdown
 [![CI Unit Tests](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml)
 [![CI Formatting & Linting](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml)
 
-The Data Validation Engine (DVE) is a configuration driven data validation library built and utilised by NHS England. Currently the package has been reverted from v1.0.0 release to a 0.x as we feel the package is not yet mature enough to be considered a 1.0.0 release. So please bear this in mind if reading through the commits and references to a v1+ release when on v0.x.
+The Data Validation Engine (DVE) is a configuration-driven data validation library created and used by NHS England. It lets users define validation rules once and apply them across multiple dataset collections - supporting consistent, accurate data checks.
 
-As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` file and example files can be accessed [here](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here](https://nhsdigital.github.io/data-validation-engine/).
+__The DVE offers__:
 
-Once a dischema file has been defined, you are ready to use the DVE. The DVE is typically orchestrated based on four key "services". These are...
+- SQL configuration-based validations
+- Format normalization to Parquet for a unified data representation
+- Data modelling and typecasting
+- Business-rule validations executed on supported backends such as Spark and DuckDB, with the option to add custom backends
+- Deriving new fields and entities
+- Clear validation reporting, including summary insights and record-level error messages
 
-|    | Service | Purpose |
-| -- | ------- | ------- |
-| 1. | File Transformation | This service will take submitted files and turn them into stringified parquet file(s) to ensure that a consistent data structure can be passed through the other services. |
-| 2. | Data Contract | This service will validate and perform type casting against a stringified parquet file using [pydantic models](https://docs.pydantic.dev/1.10/). |
-| 3. | Business Rules | The business rules service will perform more complex validations such as comparisons between fields and tables, aggregations, filters etc to generate new entities. |
-| 4. | Error Reports | The error reports service will take all the errors raised in previous services and surface them into a readable format for a downstream users/service. Currently, this implemented to be an excel spreadsheet but could be reconfigured to meet other requirements/use cases. |
-
-If you'd like more detailed documentation around these services the please read the extended documentation [here](https://nhsdigital.github.io/data-validation-engine/).
-
-The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
-
-Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing](#Contributing) section.
+As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` (data ingest schema) file and example files can be accessed [here ↗️](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/).
 
 ## Installation and usage
 
-The DVE is a Python package and can be installed using package managers such as [pip](https://pypi.org/project/pip/). As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
-
-If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
-
-Python dependencies are listed in `pyproject.toml`.
-
-To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
-
-```
-pip install data-validation-engine
-```
-
-*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
-
-Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](https://nhsdigital.github.io/data-validation-engine/).
-
-Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
+Please see the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/user_guidance/install/).
 
 ## Requesting new features and raising bug reports
 **Before creating new issues, please check to see if the same bug/feature has been created already. Where a duplicate is created, the ticket will be closed and referenced to an existing issue.**
 
-If you have spotted a bug with the DVE then please raise an issue [here](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
+If you have spotted a bug with the DVE then please raise an issue [here ↗️](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
 
 If you have feature request then please follow the same process whilst using the "Feature request template".
 
@@ -101,7 +79,7 @@ Below is a list of features that we would like to implement or have been request
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
 | Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
 | Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
-| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | Yes        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
 | Polars upgrade to v1+ | TBA | No |
 | DuckDB upgrade to v1.5+ | TBA | No |
@@ -111,7 +89,7 @@ Below is a list of features that we would like to implement or have been request
 If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
-Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
+Please see guidance [here ↗️](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
 
 ## Legal
 This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

data_validation_engine-0.9.0/README.md

@@ -0,0 +1,59 @@
+<h1 style="display: flex; align-items: center; gap: 10px;">
+    <img src="https://raw.githubusercontent.com/NHSDigital/data-validation-engine/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
+    Data Validation Engine
+</h1>
+
+![License](https://img.shields.io/github/license/NHSDigital/data-validation-engine)
+![PyPi](https://img.shields.io/pypi/v/data-validation-engine)
+![Conda](https://anaconda.org/nhs/data-validation-engine/badges/version.svg)
+[![CI Unit Tests](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_testing.yml)
+[![CI Formatting & Linting](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml/badge.svg)](https://github.com/NHSDigital/data-validation-engine/actions/workflows/ci_linting.yml)
+
+The Data Validation Engine (DVE) is a configuration-driven data validation library created and used by NHS England. It lets users define validation rules once and apply them across multiple dataset collections - supporting consistent, accurate data checks.
+
+__The DVE offers__:
+
+- SQL configuration-based validations
+- Format normalization to Parquet for a unified data representation
+- Data modelling and typecasting
+- Business-rule validations executed on supported backends such as Spark and DuckDB, with the option to add custom backends
+- Deriving new fields and entities
+- Clear validation reporting, including summary insights and record-level error messages
+
+As mentioned above, the DVE is "configuration driven" which means the majority of development for you as a user will be building a JSON document to describe how the data will be validated. The JSON document is known as a `dischema` (data ingest schema) file and example files can be accessed [here ↗️](https://github.com/NHSDigital/data-validation-engine/tree/main/tests/testdata). If you'd like to learn more about JSON document and how to build one from scratch, then please read the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/).
+
+## Installation and usage
+
+Please see the documentation [here ↗️](https://nhsdigital.github.io/data-validation-engine/user_guidance/install/).
+
+## Requesting new features and raising bug reports
+**Before creating new issues, please check to see if the same bug/feature has been created already. Where a duplicate is created, the ticket will be closed and referenced to an existing issue.**
+
+If you have spotted a bug with the DVE then please raise an issue [here ↗️](https://github.com/nhsengland/Data-Validation-Engine/issues) using the "bug template". 
+
+If you have feature request then please follow the same process whilst using the "Feature request template".
+
+## Upcoming features
+Below is a list of features that we would like to implement or have been requested.
+| Feature                                                                         | Release Version   | Released? |
+| ------------------------------------------------------------------------------- | ----------------- | --------- |
+| Open source release                                                             | 0.1.0             | Yes       |
+| Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
+| Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
+| Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | Yes        |
+| Uplift Pyspark to 4.0+                                                          | TBA               | No        |
+| Polars upgrade to v1+ | TBA | No |
+| DuckDB upgrade to v1.5+ | TBA | No |
+| Python 3.13 & 3.14 upgrade | TBA | No |
+| Create a more user friendly interface for building and modifying dischema files | TBA | No        |
+
+If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
+
+## Contributing
+Please see guidance [here ↗️](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).
+
+## Legal
+This codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.
+
+Any HTML or Markdown documentation is [© Crown copyright](https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/) and available under the terms of the [Open Government 3.0 licence](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/pyproject.toml

@@ -11,7 +11,7 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
-    "Operating System :: OS Independent",
+    "Operating System :: POSIX :: Linux",
     "Topic :: Software Development :: Libraries",
     "Typing :: Typed",
 ]
@@ -25,7 +25,7 @@ Issues = "https://github.com/NHSDigital/data-validation-engine/issues"
 Changelog = "https://github.com/NHSDigital/data-validation-engine/blob/main/CHANGELOG.md"
 
 [tool.poetry]
-version = "0.8.4"
+version = "0.9.0"
 packages = [
     { include = "dve", from = "src" },
 ]
@@ -43,7 +43,7 @@ openpyxl = "3.1.5"
 pandas = "2.3.3"
 polars = "0.20.31"
 pyarrow = "23.0.1"
-pydantic = "1.10.19"
+pydantic = "2.13.4"
 pyspark = ">=3.5.0,<=3.5.5"
 typing_extensions = "4.15.0"
 urllib3 = "2.7.0"  # dependency of boto3 & botocore
@@ -84,7 +84,8 @@ black = "26.3.1"
 astroid = "3.3.9"
 isort = "5.13.2"
 pylint = "3.3.9"
-mypy = "1.11.2"
+mypy = "1.20.2"
+librt = "0.11.0"  # mypy dependency
 boto3-stubs = {extras = ["essential"], version = "1.26.72"}
 botocore-stubs = "1.29.72"
 pandas-stubs = "1.2.0.62"
@@ -291,7 +292,7 @@ redefining-builtins-modules = [
 
 [tool.commitizen]
 name = "cz_conventional_commits"
-tag_format = "v$major.$minor.$patch"
+tag_format = "v$major.$minor.${patch}$prerelease"
 version_scheme = "pep440"
 version_provider = "poetry"
 update_changelog_on_bump = true

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/base/auditing.py

@@ -15,7 +15,7 @@ from queue import Queue as ThreadQueue
 from types import TracebackType
 from typing import Any, ClassVar, Generic, Optional, TypeVar, Union
 
-from pydantic import ValidationError, validate_arguments
+from pydantic import ValidationError, validate_call
 from typing_extensions import Literal, get_origin
 
 from dve.core_engine.models import (
@@ -99,8 +99,8 @@ class BaseAuditor(Generic[AuditReturnType]):
     def schema(self) -> dict[str, type]:
         """Determine python schema of auditor"""
         return {
-            fld: str if get_origin(mdl.type_) == Literal else mdl.type_
-            for fld, mdl in self._record_type.__fields__.items()
+            fld: str if get_origin(mdl.annotation) == Literal else mdl.annotation  # type: ignore
+            for fld, mdl in self._record_type.model_fields.items()
         }
 
     @staticmethod
@@ -196,7 +196,7 @@ class BaseAuditingManager(
         """Convert AuditReturnType to iterable of dictionaries"""
         raise NotImplementedError()
 
-    @validate_arguments
+    @validate_call
     def add_processing_records(self, processing_records: list[ProcessingStatusRecord]):
         """Add an entry to the processing_status auditor."""
         if self.pool:
@@ -208,7 +208,7 @@ class BaseAuditingManager(
             records=[dict(rec) for rec in processing_records]
         )
 
-    @validate_arguments
+    @validate_call
     def add_submission_statistics_records(self, sub_stats: list[SubmissionStatisticsRecord]):
         """Add an entry to the submission statistics auditor."""
         if self.pool:
@@ -218,7 +218,7 @@ class BaseAuditingManager(
             )
         return self._submission_statistics.add_records(records=[dict(rec) for rec in sub_stats])
 
-    @validate_arguments
+    @validate_call
     def add_transfer_records(self, transfer_records: list[TransferRecord]):
         """Add an entry to the transfers auditor"""
         if self.pool:
@@ -227,7 +227,7 @@ class BaseAuditingManager(
             )
         return self._transfers.add_records(records=[dict(rec) for rec in transfer_records])
 
-    @validate_arguments
+    @validate_call
     def add_new_submissions(
         self,
         submissions: list[SubmissionMetadata],
@@ -250,7 +250,7 @@ class BaseAuditingManager(
                     processing_status="received",
                     job_run_id=job_run_id,
                     **ts_info,
-                ).dict(),
+                ).model_dump(),
             }
             processing_status_recs.append(processing_rec)
             if sub_info:

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/auditing.py

@@ -14,10 +14,10 @@ from dve.core_engine.backends.base.auditing import (
     OrderCriteria,
 )
 from dve.core_engine.backends.implementations.duckdb.duckdb_helpers import (
-    PYTHON_TYPE_TO_DUCKDB_TYPE,
+    get_duckdb_type_from_annotation,
     table_exists,
 )
-from dve.core_engine.backends.utilities import PYTHON_TYPE_TO_POLARS_TYPE
+from dve.core_engine.backends.utilities import get_polars_type_from_annotation
 from dve.core_engine.models import (
     AuditRecord,
     ProcessingStatusRecord,
@@ -62,7 +62,10 @@ class DDBAuditor(BaseAuditor[DuckDBPyRelation]):
         """Generate create table sql script for auditor"""
         _sql_expression = f"CREATE TABLE {self._name} ("
         _sql_expression += ", ".join(
-            [f"{fld} {PYTHON_TYPE_TO_DUCKDB_TYPE.get(dtype)}" for fld, dtype in self.schema.items()]
+            [
+                f"{fld} {get_duckdb_type_from_annotation(dtype)}"
+                for fld, dtype in self.schema.items()
+            ]
         )
         _sql_expression += ")"
         return _sql_expression
@@ -70,10 +73,7 @@ class DDBAuditor(BaseAuditor[DuckDBPyRelation]):
     @property
     def polars_schema(self) -> dict[str, PolarsType]:
         """Get polars dataframe schema for auditor"""
-        return {
-            fld: PYTHON_TYPE_TO_POLARS_TYPE.get(dtype, pl.Utf8)  # type: ignore
-            for fld, dtype in self.schema.items()
-        }
+        return {fld: get_polars_type_from_annotation(dtype) for fld, dtype in self.schema.items()}
 
     def get_relation(self) -> DuckDBPyRelation:
         """Get a relation to interact with the auditor duckdb table"""
@@ -106,7 +106,7 @@ class DDBAuditor(BaseAuditor[DuckDBPyRelation]):
         """Convert a list of audit records to a relation"""
         # pylint: disable=W0612
         rec_df = pl.DataFrame(  # type: ignore
-            [rec.dict() for rec in recs],
+            [rec.model_dump() for rec in recs],
             schema=self.polars_schema,
         )
         return self._connection.sql("select * from rec_df")

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/contract.py

@@ -14,7 +14,6 @@ from duckdb import DuckDBPyConnection, DuckDBPyRelation
 from duckdb.typing import DuckDBPyType
 from polars.datatypes.classes import DataTypeClass as PolarsType
 from pydantic import BaseModel
-from pydantic.fields import ModelField
 
 import dve.parser.file_handling as fh
 from dve.common.error_utils import (
@@ -96,8 +95,8 @@ class DuckDBDataContract(BaseDataContract[DuckDBPyRelation]):
     ) -> DuckDBPyRelation:
         """Create DuckDB Relation from iterator of records"""
         polars_schema: dict[str, PolarsType] = {
-            fld.name: get_polars_type_from_annotation(fld.type_)
-            for fld in stringify_model(schema).__fields__.values()
+            name: get_polars_type_from_annotation(fld.annotation)
+            for name, fld in stringify_model(schema).model_fields.items()
         }
         _lazy_df = pl.LazyFrame(records, polars_schema)  # type: ignore # pylint: disable=unused-variable
         return self._connection.sql("select * from _lazy_df")
@@ -130,17 +129,15 @@ class DuckDBDataContract(BaseDataContract[DuckDBPyRelation]):
         ) as msg_writer:
             for entity_name, relation in entities.items():
                 # get dtypes for all fields -> python data types or use with relation
-                entity_fields: dict[str, ModelField] = contract_metadata.schemas[
-                    entity_name
-                ].__fields__
+                entity_fields = contract_metadata.schemas[entity_name].model_fields
                 ddb_schema: dict[str, DuckDBPyType] = {
-                    fld.name: get_duckdb_type_from_annotation(fld.annotation)
-                    for fld in entity_fields.values()
+                    name: get_duckdb_type_from_annotation(fld.annotation)
+                    for name, fld in entity_fields.items()
                 }
                 ddb_schema[RECORD_INDEX_COLUMN_NAME] = get_duckdb_type_from_annotation(int)
                 polars_schema: dict[str, PolarsType] = {
-                    fld.name: get_polars_type_from_annotation(fld.annotation)
-                    for fld in entity_fields.values()
+                    name: get_polars_type_from_annotation(fld.annotation)
+                    for name, fld in entity_fields.items()
                 }
                 polars_schema[RECORD_INDEX_COLUMN_NAME] = get_polars_type_from_annotation(int)
                 if relation_is_empty(relation):

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/duckdb_helpers.py

@@ -8,7 +8,7 @@ from dataclasses import is_dataclass
 from datetime import date, datetime, time
 from decimal import Decimal
 from pathlib import Path
-from typing import Any, ClassVar, Union
+from typing import Any, ClassVar, Literal, Union
 from urllib.parse import urlparse
 
 import duckdb.typing as ddbtyp
@@ -129,8 +129,8 @@ def get_duckdb_type_from_annotation(type_annotation: Any) -> DuckDBPyType:
       'optional' wrapper and return the inner type
     - A subclass of `typing.TypedDict` with values typed using supported types. This
       will parse the value types as Polars types and return a duckdb STRUCT.
-    - A dataclass or `pydantic.main.ModelMetaClass` with values typed using supported types.
-      This will parse the field types as Polars types and return a duckdb STRUCT.
+    - A dataclass or `pydantic.BaseModel` with values typed using supported types.
+      This will parse the field types as duckdb types and return a duckdb STRUCT.
     - Any supported type, with a `typing_extensions.Annotated` wrapper.
 
     Any `ClassVar` types within `TypedDict`s, dataclasses, or `pydantic` models will be
@@ -139,6 +139,14 @@ def get_duckdb_type_from_annotation(type_annotation: Any) -> DuckDBPyType:
     """
     type_origin = get_origin(type_annotation)
 
+    if type_origin is Literal:
+        ddb_types = [get_duckdb_type_from_annotation(type(t)) for t in get_args(type_annotation)]
+        if not ddb_types or not all(t == ddb_types[0] for t in ddb_types):
+            raise ValueError(
+                f"Unable to determine a single concrete type for Literal. Got {type_annotation!r}"
+            )
+        return ddb_types[0]
+
     # An `Optional` or `Union` type, check to ensure non-heterogenity.
     if type_origin is Union:
         python_type = _get_non_heterogenous_type(get_args(type_annotation))

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/readers/csv.py

@@ -111,8 +111,8 @@ class DuckDBCSVReader(CSVFileReader):
         }
 
         ddb_schema: dict[str, SQLType] = {
-            fld.name: str(get_duckdb_type_from_annotation(fld.annotation))  # type: ignore
-            for fld in schema.__fields__.values()
+            name: str(get_duckdb_type_from_annotation(fld.annotation))  # type: ignore
+            for name, fld in schema.model_fields.items()
         }
 
         reader_options["columns"] = ddb_schema
@@ -166,8 +166,8 @@ class PolarsToDuckDBCSVReader(DuckDBCSVReader):
         }
 
         polars_types = {
-            fld.name: get_polars_type_from_annotation(fld.annotation)  # type: ignore
-            for fld in schema.__fields__.values()
+            name: get_polars_type_from_annotation(fld.annotation)  # type: ignore
+            for name, fld in schema.model_fields.items()
         }
         reader_options["dtypes"] = polars_types
 

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/readers/json.py

@@ -56,8 +56,8 @@ class DuckDBJSONReader(BaseFileReader):
         """Returns a relation object from the source json"""
 
         ddb_schema: dict[str, SQLType] = {
-            fld.name: str(get_duckdb_type_from_annotation(fld.annotation))  # type: ignore
-            for fld in schema.__fields__.values()
+            name: str(get_duckdb_type_from_annotation(fld.annotation))  # type: ignore
+            for name, fld in schema.model_fields.items()
         }
 
         return self.add_record_index(

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/duckdb/readers/xml.py

@@ -47,8 +47,8 @@ class DuckDBXMLStreamReader(XMLStreamReader):
                 )
 
         polars_schema: dict[str, pl.DataType] = {  # type: ignore
-            fld.name: get_polars_type_from_annotation(fld.annotation)
-            for fld in stringify_model(schema).__fields__.values()
+            name: get_polars_type_from_annotation(fld.annotation)
+            for name, fld in stringify_model(schema).model_fields.items()
         }
 
         _lazy_frame = self.add_record_index(

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/spark/auditing.py

@@ -116,7 +116,7 @@ class SparkAuditor(BaseAuditor[DataFrame]):
     def conv_to_entity(self, recs: list[AuditRecord]) -> DataFrame:
         """Convert the dataframe to an iterable of the related audit record"""
         return self._spark.createDataFrame(  # type: ignore
-            [rec.dict() for rec in recs], schema=self.spark_schema
+            [rec.model_dump() for rec in recs], schema=self.spark_schema
         )
 
     def add_records(self, records: Iterable[dict[str, Any]]):

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/spark/backend.py

@@ -92,5 +92,5 @@ class SparkBackend(BaseBackend[DataFrame]):
 
     def convert_submission_info(self, submission_info: SubmissionInfo) -> DataFrame:
         return self.spark_session.createDataFrame(  # type: ignore
-            [submission_info.dict()], schema=get_type_from_annotation(type(submission_info))
+            [submission_info.model_dump()], schema=get_type_from_annotation(type(submission_info))
         )

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/spark/contract.py

@@ -7,7 +7,7 @@ from typing import Any, Optional
 from uuid import uuid4
 
 from pydantic import BaseModel
-from pydantic.fields import ModelField
+from pydantic.fields import FieldInfo
 from pyspark.sql import DataFrame, SparkSession
 from pyspark.sql import functions as sf
 from pyspark.sql.functions import col, lit
@@ -104,7 +104,9 @@ class SparkDataContract(BaseDataContract[DataFrame]):
 
         successful = True
         for entity_name, record_df in entities.items():
-            entity_fields: dict[str, ModelField] = contract_metadata.schemas[entity_name].__fields__
+            entity_fields: dict[str, FieldInfo] = contract_metadata.schemas[
+                entity_name
+            ].model_fields
             spark_schema = get_type_from_annotation(contract_metadata.schemas[entity_name])
             spark_schema.add(StructField(RECORD_INDEX_COLUMN_NAME, LongType()))
             if df_is_empty(record_df):
@@ -147,19 +149,17 @@ class SparkDataContract(BaseDataContract[DataFrame]):
                 self.logger.info(f"Data contract found {msg_count} issues in {entity_name}")
 
             try:
-                # TODO: will need to revisit in pydantic v2 bump as model field no longer available
                 record_df = record_df.select(
                     *[
                         (
                             get_spark_cast_statement_from_annotation(
-                                fld, mdl_field.annotation
+                                fld, fld_info.annotation
                             ).alias(fld)
                             if fld in record_df.columns
                             else lit(None).cast(
-                                get_type_from_annotation(mdl_field.annotation).alias(fld)
-                            )
+                                get_type_from_annotation(fld_info.annotation)).alias(fld)
                         )
-                        for fld, mdl_field in entity_fields.items()
+                        for fld, fld_info in entity_fields.items()
                     ],
                     col(RECORD_INDEX_COLUMN_NAME).cast(LongType()).alias(RECORD_INDEX_COLUMN_NAME),
                 )

{data_validation_engine-0.8.4 → data_validation_engine-0.9.0}/src/dve/core_engine/backends/implementations/spark/spark_helpers.py

@@ -13,11 +13,10 @@ from dataclasses import dataclass, is_dataclass
 from decimal import Decimal
 from functools import wraps
 from pathlib import Path
-from typing import Any, ClassVar, Optional, TypeVar, Union, overload
+from typing import Any, ClassVar, Literal, Optional, TypeVar, Union, overload
 
 from delta.exceptions import ConcurrentAppendException, DeltaConcurrentModificationException
 from pydantic import BaseModel
-from pydantic.types import ConstrainedDecimal
 from pyspark.sql import DataFrame, Row, SparkSession
 from pyspark.sql import functions as sf
 from pyspark.sql import types as st
@@ -52,6 +51,7 @@ FourArgWrapped = Callable[[Column, Column, Column, Column], Column]
 """A wrapped function (Spark UDF) taking four args."""
 
 
+# TODO - lets see if we can bin this off as it's a bit overkill
 @dataclass(frozen=True)
 class DecimalConfig:
     """Configuration for a Python decimal to enable it to be mapped to a
@@ -64,13 +64,13 @@ class DecimalConfig:
 
     """
 
-    precision: int = 38
+    max_digits: int = 38
     """
     The precision of the decimal. This is the total number of digits in the
     decimal.
 
     """
-    scale: int = 18
+    decimal_places: int = 18
     """
     The scale of the decimal. This is the number of digits to the right of the
     decimal point.
@@ -78,10 +78,12 @@ class DecimalConfig:
     """
 
     def __post_init__(self):
-        if not 0 < self.precision <= 38:
-            raise ValueError("Precision must be between 1 and 38 (inclusive)")
-        if not 0 <= self.scale <= self.precision:
-            raise ValueError("Scale must be between 0 and the precision (inclusive)")
+        if not 0 < self.max_digits <= 38:
+            raise ValueError("Max digits must be between 1 and 38 (inclusive)")
+        if not 0 <= self.decimal_places <= self.max_digits:
+            raise ValueError(
+                "Decimal Places must be between 0 and the specified number of digits (inclusive)"
+            )
 
 
 DEFAULT_DECIMAL_CONFIG = DecimalConfig()
@@ -96,7 +98,9 @@ PYTHON_TYPE_TO_SPARK_TYPE: dict[type, st.DataType] = {
     bytes: st.BinaryType(),
     dt.date: st.DateType(),
     dt.datetime: st.TimestampType(),
-    Decimal: st.DecimalType(DEFAULT_DECIMAL_CONFIG.precision, DEFAULT_DECIMAL_CONFIG.scale),
+    Decimal: st.DecimalType(
+        DEFAULT_DECIMAL_CONFIG.max_digits, DEFAULT_DECIMAL_CONFIG.decimal_places
+    ),
 }
 """A mapping of Python types to the equivalent Spark types."""
 
@@ -164,7 +168,7 @@ def get_type_from_annotation(type_annotation: Any) -> st.DataType:
       'optional' wrapper and return the inner type (Spark types are all nullable).
     - A subclass of `typing.TypedDict` with values typed using supported types. This
       will parse the value types as Spark types and return a Spark `StructType`.
-    - A dataclass or `pydantic.main.ModelMetaClass` with values typed using supported types.
+    - A dataclass or `pydantic.BaseModel` with values typed using supported types.
       This will parse the field types as Spark types and return a Spark `StructType`.
     - Any supported type, with a `typing_extensions.Annotated` wrapper.
     - A `decimal.Decimal` wrapped with `typing_extensions.Annotated` with a `DecimalConfig`
@@ -178,6 +182,14 @@ def get_type_from_annotation(type_annotation: Any) -> st.DataType:
     """
     type_origin = get_origin(type_annotation)
 
+    if type_origin is Literal:
+        types = [get_type_from_annotation(type(t)) for t in get_args(type_annotation)]
+        if not types or not all(t == types[0] for t in types):
+            raise ValueError(
+                f"Unable to determine a single concrete type for Literal. Got {type_annotation!r}"
+            )
+        return types[0]
+
     # An `Optional` or `Union` type, check to ensure non-heterogenity.
     if type_origin is Union:
         python_type = _get_non_heterogenous_type(get_args(type_annotation))
@@ -194,13 +206,13 @@ def get_type_from_annotation(type_annotation: Any) -> st.DataType:
         if python_type is not Decimal:
             return get_type_from_annotation(python_type)
 
-        try:  # Grab the decimal configuration from the list of other args.
-            configuration: DecimalConfig = next(
-                filter(lambda config: isinstance(config, DecimalConfig), other_args)
-            )
-        except StopIteration:
+        config_options = [arg for arg in other_args if hasattr(arg, "max_digits")]
+        if config_options:
+            configuration = config_options[0]
+        else:
             configuration = DEFAULT_DECIMAL_CONFIG
-        return st.DecimalType(configuration.precision, configuration.scale)
+
+        return st.DecimalType(configuration.max_digits, configuration.decimal_places)
 
     # Ensure that we have a concrete type at this point.
     if not isinstance(type_annotation, type):
@@ -234,11 +246,6 @@ def get_type_from_annotation(type_annotation: Any) -> st.DataType:
 
         return st.StructType(fields)
 
-    if issubclass(type_annotation, ConstrainedDecimal):
-        precision = int(type_annotation.max_digits or 38)
-        scale = int(type_annotation.decimal_places or precision)
-        return st.DecimalType(precision, scale)
-
     if type_annotation is list:
         raise ValueError(
             f"list must have type annotation (e.g. `list[str]`), got {type_annotation!r}"