data-validation-engine 0.7.5__tar.gz → 0.8.0__tar.gz

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/PKG-INFO

@@ -1,32 +1,33 @@
 Metadata-Version: 2.4
 Name: data-validation-engine
-Version: 0.7.5
+Version: 0.8.0
 Summary: `nhs data validation engine` is a framework used to validate data
 License-Expression: MIT
 License-File: LICENSE
 Author: NHS England
 Author-email: england.contactus@nhs.net
-Requires-Python: >=3.10,<3.12
+Requires-Python: >=3.10,<3.13
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Operating System :: OS Independent
 Classifier: Topic :: Software Development :: Libraries
 Classifier: Typing :: Typed
-Requires-Dist: Jinja2 (==3.1.*)
+Requires-Dist: Jinja2 (==3.1.6)
 Requires-Dist: boto3 (>=1.34.162,<1.36)
 Requires-Dist: botocore (>=1.34.162,<1.36)
-Requires-Dist: delta-spark (==2.4.*)
-Requires-Dist: duckdb (==1.1.*)
-Requires-Dist: lxml (>=4.9.1,<5.0.0)
+Requires-Dist: delta-spark (==3.2.0)
+Requires-Dist: duckdb (==1.1.3)
+Requires-Dist: lxml (==4.9.4)
 Requires-Dist: numpy (==1.26.4)
-Requires-Dist: openpyxl (>=3.1,<4.0)
-Requires-Dist: pandas (>=2.2.2,<3.0.0)
-Requires-Dist: polars (==0.20.*)
-Requires-Dist: pyarrow (>=17.0.0,<18.0.0)
-Requires-Dist: pydantic (==1.10.16)
-Requires-Dist: pyspark (==3.4.*)
-Requires-Dist: typing_extensions (>=4.6.2,<5.0.0)
+Requires-Dist: openpyxl (==3.1.5)
+Requires-Dist: pandas (==2.3.3)
+Requires-Dist: polars (==0.20.31)
+Requires-Dist: pyarrow (==17.0.0)
+Requires-Dist: pydantic (==1.10.19)
+Requires-Dist: pyspark (==3.5.2)
+Requires-Dist: typing_extensions (==4.15.0)
+Requires-Dist: urllib3 (==2.7.0)
 Project-URL: Changelog, https://github.com/NHSDigital/data-validation-engine/blob/main/CHANGELOG.md
 Project-URL: Documentation, https://nhsdigital.github.io/data-validation-engine/
 Project-URL: Homepage, https://github.com/NHSDigital/data-validation-engine
@@ -96,13 +97,16 @@ Below is a list of features that we would like to implement or have been request
 | ------------------------------------------------------------------------------- | ----------------- | --------- |
 | Open source release                                                             | 0.1.0             | Yes       |
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
-| Uplift Pyspark to 3.5                                                           | TBA               | No        |
-| Allow DVE to run on Python 3.12+                                                | TBA               | No        |
-| Upgrade to Pydantic 2.0                                                         | TBA               | No        |
+| Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
+| Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
-| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No        |
+| Polars upgrade to v1+ | TBA | No |
+| DuckDB upgrade to v1.5+ | TBA | No |
+| Python 3.13 & 3.14 upgrade | TBA | No |
+| Create a more user friendly interface for building and modifying dischema files | TBA | No        |
 
-Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#Contributing) section and get involved.
+If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
 Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/README.md

@@ -60,13 +60,16 @@ Below is a list of features that we would like to implement or have been request
 | ------------------------------------------------------------------------------- | ----------------- | --------- |
 | Open source release                                                             | 0.1.0             | Yes       |
 | Uplift to Python 3.11                                                           | 0.2.0             | Yes       |
-| Uplift Pyspark to 3.5                                                           | TBA               | No        |
-| Allow DVE to run on Python 3.12+                                                | TBA               | No        |
-| Upgrade to Pydantic 2.0                                                         | TBA               | No        |
+| Uplift Pyspark to 3.5                                                           | 0.8.0             | Yes       |
+| Allow DVE to run on Python 3.12+                                                | 0.8.0             | Yes       |
+| Upgrade to Pydantic 2.0                                                         | 0.9.0             | No        |
 | Uplift Pyspark to 4.0+                                                          | TBA               | No        |
-| Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No        |
+| Polars upgrade to v1+ | TBA | No |
+| DuckDB upgrade to v1.5+ | TBA | No |
+| Python 3.13 & 3.14 upgrade | TBA | No |
+| Create a more user friendly interface for building and modifying dischema files | TBA | No        |
 
-Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#Contributing) section and get involved.
+If you are interested in getting any of the unreleased features listed above available, then please read the [Contributing](#Contributing) section and then submit us a pull request.
 
 ## Contributing
 Please see guidance [here](https://github.com/NHSDigital/data-validation-engine/blob/main/CONTRIBUTE.md).

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/pyproject.toml

@@ -24,27 +24,28 @@ Issues = "https://github.com/NHSDigital/data-validation-engine/issues"
 Changelog = "https://github.com/NHSDigital/data-validation-engine/blob/main/CHANGELOG.md"
 
 [tool.poetry]
-version = "0.7.5"
+version = "0.8.0"
 packages = [
     { include = "dve", from = "src" },
 ]
 
 [tool.poetry.dependencies]
-python = ">=3.10,<3.12"
+python = ">=3.10,<3.13"  # breaking changes beyond 3.12
 boto3 = ">=1.34.162,<1.36"  # breaking change beyond 1.36
 botocore = ">=1.34.162,<1.36"  # breaking change beyond 1.36
-delta-spark = "2.4.*"
-duckdb = "1.1.*"  # breaking changes beyond 1.1
-Jinja2 = "3.1.*"
-lxml = "^4.9.1"
+delta-spark = "3.2.0"
+duckdb = "1.1.3"  # breaking changes beyond 1.1
+Jinja2 = "3.1.6"
+lxml = "4.9.4"
 numpy = "1.26.4"
-openpyxl = "^3.1"
-pandas = "^2.2.2"
-polars = "0.20.*"
-pyarrow = "^17.0.0"
-pydantic = "1.10.16"
-pyspark = "3.4.*"
-typing_extensions = "^4.6.2"
+openpyxl = "3.1.5"
+pandas = "2.3.3"
+polars = "0.20.31"
+pyarrow = "17.0.0"
+pydantic = "1.10.19"
+pyspark = "3.5.2"
+typing_extensions = "4.15.0"
+urllib3 = "2.7.0"  # dependency of boto3 & botocore
 
 [tool.poetry.group.dev]
 optional = true
@@ -58,30 +59,30 @@ commitizen = "4.9.1"
 pre-commit = "4.3.0"
 charset-normalizer = "3.4.6"
 python-discovery = "1.2.0"
-requests = "2.33.0"
+idna = "3.15"  # dependency of requests
 
 [tool.poetry.group.test]
 optional = true
 
 [tool.poetry.group.test.dependencies]
-faker = "18.11.1"
 behave = "1.3.3"
 coverage = "7.11.0"
-moto = {extras = ["s3"], version = "4.0.13"}
+moto = {extras = ["s3"], version = "4.2.14"}
+requests = "2.33.0"  # dependency of `moto`
 Werkzeug = "3.1.6"
-pytest = "8.4.2"
+pytest = "9.0.3"
 pytest-lazy-fixtures = "1.4.0"  # switched from https://github.com/TvoroG/pytest-lazy-fixture as it's no longer supported
-xlsx2csv = "0.8.2"
+xlsx2csv = "0.8.4"  # polars requirement
 
 [tool.poetry.group.lint]
 optional = true
 
 [tool.poetry.group.lint.dependencies]
 black = "24.3.0"
-astroid = "2.14.2"
-isort = "5.11.5"
-pylint = "2.16.4"
-mypy = "0.991"
+astroid = "3.3.9"
+isort = "5.13.2"
+pylint = "3.3.9"
+mypy = "1.11.2"
 boto3-stubs = {extras = ["essential"], version = "1.26.72"}
 botocore-stubs = "1.29.72"
 pandas-stubs = "1.2.0.62"
@@ -100,10 +101,10 @@ optional = true
 
 [tool.poetry.group.docs.dependencies]
 click = "8.2.1"
-mkdocs = "^1.6.1"
+mkdocs = "1.6.1"
 mkdocstrings = { version = "1.0.3", extras = ["python"] }
 griffelib = "2.0.1"
-pymdown-extensions = "10.21.2"
+pymdown-extensions = "10.21.3"
 zensical = "0.0.31"
 
 [tool.ruff]
@@ -142,10 +143,6 @@ source_pkgs = [
 [tool.coverage.report]
 show_missing = true
 
-[tool.pylint]
-init-hook = "import sys; sys.path.append('./pylint_checkers')"
-load-plugins = "check_typing_imports"
-
 [tool.pylint.main]
 extension-pkg-allow-list = ["pyspark", "lxml", "pydantic"]
 fail-under = 10.0
@@ -194,7 +191,7 @@ max-statements = 50
 min-public-methods = 2
 
 [tool.pylint.exceptions]
-overgeneral-exceptions = ["BaseException", "Exception"]
+overgeneral-exceptions = ["builtins.BaseException", "builtins.Exception"]
 
 [tool.pylint.format]
 ignore-long-lines = "^\\s*(# )?<?https?://\\S+>?$"
@@ -229,6 +226,8 @@ disable = [
     "use-symbolic-message-instead",
     "logging-fstring-interpolation",
     "fixme",
+    "too-many-positional-arguments",
+    "too-many-arguments",
 ]
 enable = ["c-extension-no-member"]
 

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/base/reader.py

@@ -90,6 +90,7 @@ class BaseFileReader(ABC):
         resource: URI,
         entity_name: EntityName,
         schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> Iterator[dict[str, Any]]:
         """Iterate through the contents of the resource, yielding dicts
         representing each record.
@@ -107,6 +108,7 @@ class BaseFileReader(ABC):
         resource: URI,
         entity_name: EntityName,
         schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> EntityType:
         """Read to the specified entity type, if supported.
 
@@ -116,7 +118,12 @@ class BaseFileReader(ABC):
 
         """
         if entity_name == Iterator[dict[str, Any]]:
-            return self.read_to_py_iterator(resource, entity_name, schema)  # type: ignore
+            return self.read_to_py_iterator(
+                resource,
+                entity_name,
+                schema, # type: ignore
+                all_model_fields
+            )
 
         self.raise_if_not_sensible_file(resource, entity_name)
 
@@ -125,7 +132,13 @@ class BaseFileReader(ABC):
         except KeyError as err:
             raise ReaderLacksEntityTypeSupport(entity_type=entity_type) from err
 
-        return reader_func(self, resource, entity_name, schema)
+        return reader_func(
+            self,
+            resource,
+            entity_name,
+            schema,
+            all_model_fields=all_model_fields  # type: ignore
+        )
 
     def add_record_index(self, entity: EntityType, **kwargs) -> EntityType:
         """Add a record index to the entity"""

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/exceptions.py

@@ -33,6 +33,33 @@ class MessageBearingError(BackendError):
         """The messages to be returned as part of the error."""
 
 
+class UnableToParseCSVError(MessageBearingError):
+    """An error raised when unable to parse a CSV file"""
+
+    def __init__(
+        self,
+        entity_name: str,
+        field_check_error_message: str,
+        field_check_error_code: str
+    ):
+        super().__init__(
+            messages=[
+                FeedbackMessage(
+                    entity="csv_structure",
+                    record={
+                        entity_name: "Unable to parse file. Please check the structure of the file."
+                    },
+                    failure_type="submission",
+                    is_informational=False,
+                    error_type="csv read",
+                    error_location=entity_name,
+                    error_message=field_check_error_message,
+                    error_code=field_check_error_code,
+                )
+            ]
+        )
+
+
 class BackendErrorMixin(ABC, BackendError):
     """A mixin used to create backend error type."""
 

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/implementations/duckdb/readers/csv.py

@@ -9,21 +9,25 @@ import polars as pl
 from duckdb import (
     DuckDBPyConnection,
     DuckDBPyRelation,
+    InvalidInputException,
     StarExpression,
-    default_connection,
     read_csv,
 )
 from pydantic import BaseModel
 
-from dve.core_engine.backends.base.reader import BaseFileReader, read_function
-from dve.core_engine.backends.exceptions import EmptyFileError, MessageBearingError
+from dve.core_engine.backends.base.reader import read_function
+from dve.core_engine.backends.exceptions import (
+    EmptyFileError,
+    MessageBearingError,
+    UnableToParseCSVError,
+)
 from dve.core_engine.backends.implementations.duckdb.duckdb_helpers import (
     duckdb_record_index,
     duckdb_write_parquet,
     get_duckdb_type_from_annotation,
 )
 from dve.core_engine.backends.implementations.duckdb.types import SQLType
-from dve.core_engine.backends.readers.utilities import check_csv_header_expected
+from dve.core_engine.backends.readers.csv import CSVFileReader
 from dve.core_engine.backends.utilities import get_polars_type_from_annotation, polars_record_index
 from dve.core_engine.constants import RECORD_INDEX_COLUMN_NAME
 from dve.core_engine.message import FeedbackMessage
@@ -33,7 +37,7 @@ from dve.parser.file_handling import get_content_length
 
 @duckdb_record_index
 @duckdb_write_parquet
-class DuckDBCSVReader(BaseFileReader):
+class DuckDBCSVReader(CSVFileReader):
     """A reader for CSV files including the ability to compare the passed model
     to the file header, if it exists.
 
@@ -53,66 +57,57 @@ class DuckDBCSVReader(BaseFileReader):
         quotechar: str = '"',
         connection: Optional[DuckDBPyConnection] = None,
         field_check: bool = False,
-        field_check_error_code: Optional[str] = "ExpectedVsActualFieldMismatch",
-        field_check_error_message: Optional[str] = "The submitted header is missing fields",
+        field_check_error_code: str = "ExpectedVsActualFieldMismatch",
+        field_check_error_message: str = "The submitted header is missing fields",
         null_empty_strings: bool = False,
         **_,
     ):
-        self.header = header
-        self.delim = delim
-        self.quotechar = quotechar
-        self._connection = connection if connection else default_connection
-        self.field_check = field_check
-        self.field_check_error_code = field_check_error_code
-        self.field_check_error_message = field_check_error_message
+        self._connection = connection if connection else ddb.connect(":memory:")
         self.null_empty_strings = null_empty_strings
 
-        super().__init__()
-
-    def perform_field_check(
-        self, resource: URI, entity_name: str, expected_schema: type[BaseModel]
-    ):
-        """Check that the header of the CSV aligns with the provided model"""
-        if not self.header:
-            raise ValueError("Cannot perform field check without a CSV header")
-
-        if missing := check_csv_header_expected(resource, expected_schema, self.delim):
-            raise MessageBearingError(
-                "The CSV header doesn't match what is expected",
-                messages=[
-                    FeedbackMessage(
-                        entity=entity_name,
-                        record={"missing_fields": missing},
-                        failure_type="submission",
-                        error_location="Whole File",
-                        reporting_field="missing_fields",
-                        error_code=self.field_check_error_code,
-                        error_message=f"{self.field_check_error_message}",  # pylint: disable=line-too-long
-                    )
-                ],
-            )
+        super().__init__(
+            header=header,
+            delimiter=delim,
+            quote_char=quotechar,
+            field_check=field_check,
+            field_check_error_code=field_check_error_code,
+            field_check_error_message=field_check_error_message
+        )
 
     def read_to_py_iterator(
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> Iterator[dict[str, Any]]:
         """Creates an iterable object of rows as dictionaries"""
-        yield from self.read_to_relation(resource, entity_name, schema).pl().iter_rows(named=True)
+        yield from self.read_to_relation(
+            resource,
+            entity_name,
+            schema,
+            all_model_fields,
+        ).pl().iter_rows(named=True)
 
     @read_function(DuckDBPyRelation)
     def read_to_relation(  # pylint: disable=unused-argument
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> DuckDBPyRelation:
         """Returns a relation object from the source csv"""
         if get_content_length(resource) == 0:
             raise EmptyFileError(f"File at {resource} is empty.")
 
         if self.field_check:
-            self.perform_field_check(resource, entity_name, schema)
+            self.perform_field_check(resource, entity_name, schema, all_model_fields)
 
         reader_options: dict[str, Any] = {
             "header": self.header,
-            "delimiter": self.delim,
-            "quotechar": self.quotechar,
+            "delimiter": self.delimiter,
+            "quotechar": self.quote_char,
         }
 
         ddb_schema: dict[str, SQLType] = {
@@ -122,7 +117,14 @@ class DuckDBCSVReader(BaseFileReader):
 
         reader_options["columns"] = ddb_schema
 
-        rel = self.add_record_index(read_csv(resource, **reader_options, parallel=False))
+        try:
+            rel = self.add_record_index(read_csv(resource, **reader_options, parallel=False))
+        except InvalidInputException as exc:
+            raise UnableToParseCSVError(
+                entity_name="csv_structure",
+                field_check_error_message=self.field_check_error_message,
+                field_check_error_code=self.field_check_error_code,
+            ) from exc
 
         if self.null_empty_strings:
             cleaned_cols = ",".join(
@@ -144,19 +146,23 @@ class PolarsToDuckDBCSVReader(DuckDBCSVReader):
 
     @read_function(DuckDBPyRelation)
     def read_to_relation(  # pylint: disable=unused-argument
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> DuckDBPyRelation:
         """Returns a relation object from the source csv"""
         if get_content_length(resource) == 0:
             raise EmptyFileError(f"File at {resource} is empty.")
 
         if self.field_check:
-            self.perform_field_check(resource, entity_name, schema)
+            self.perform_field_check(resource, entity_name, schema, all_model_fields)
 
         reader_options: dict[str, Any] = {
             "has_header": self.header,
-            "separator": self.delim,
-            "quote_char": self.quotechar,
+            "separator": self.delimiter,
+            "quote_char": self.quote_char,
         }
 
         polars_types = {
@@ -167,11 +173,18 @@ class PolarsToDuckDBCSVReader(DuckDBCSVReader):
 
         # there is a raise_if_empty arg for 0.18+. Future reference when upgrading. Makes L85
         # redundant
-        df = self.add_record_index(  # pylint: disable=W0612
-            pl.scan_csv(resource, **reader_options).select(  # type: ignore
-                list(polars_types.keys())
+        try:
+            df = self.add_record_index(  # pylint: disable=W0612
+                pl.scan_csv(resource, **reader_options).select(  # type: ignore
+                    list(polars_types.keys())
+                )
             )
-        )
+        except pl.exceptions.PolarsError as exc:
+            raise UnableToParseCSVError(
+                entity_name="csv_structure",
+                field_check_error_message=self.field_check_error_message,
+                field_check_error_code=self.field_check_error_code,
+            ) from exc
 
         if self.null_empty_strings:
             pl_exprs = [
@@ -181,7 +194,16 @@ class PolarsToDuckDBCSVReader(DuckDBCSVReader):
             ] + [pl.col(RECORD_INDEX_COLUMN_NAME)]
             df = df.select(pl_exprs)
 
-        return ddb.sql("SELECT * FROM df")
+        entity = self._connection.sql("SELECT * FROM df")
+
+        if entity.pl().shape[0] == 0:
+            raise UnableToParseCSVError(
+                entity_name="csv_structure",
+                field_check_error_message=self.field_check_error_message,
+                field_check_error_code=self.field_check_error_code,
+            )
+
+        return entity
 
 
 class DuckDBCSVRepeatingHeaderReader(PolarsToDuckDBCSVReader):
@@ -222,10 +244,17 @@ class DuckDBCSVRepeatingHeaderReader(PolarsToDuckDBCSVReader):
 
     @read_function(DuckDBPyRelation)
     def read_to_relation(  # pylint: disable=unused-argument
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> DuckDBPyRelation:
         entity: DuckDBPyRelation = super().read_to_relation(
-            resource=resource, entity_name=entity_name, schema=schema
+            resource=resource,
+            entity_name=entity_name,
+            schema=schema,
+            all_model_fields=all_model_fields
         )
         entity = entity.select(StarExpression(exclude=[RECORD_INDEX_COLUMN_NAME])).distinct()
         no_records = entity.shape[0]

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/implementations/duckdb/readers/json.py

@@ -4,7 +4,8 @@
 from collections.abc import Iterator
 from typing import Any, Optional
 
-from duckdb import DuckDBPyRelation, read_json
+import duckdb
+from duckdb import DuckDBPyConnection, DuckDBPyRelation
 from pydantic import BaseModel
 
 from dve.core_engine.backends.base.reader import BaseFileReader, read_function
@@ -26,21 +27,31 @@ class DuckDBJSONReader(BaseFileReader):
         self,
         *,
         json_format: Optional[str] = "array",
+        connection: Optional[DuckDBPyConnection] = None,
         **_,
     ):
         self._json_format = json_format
+        self._connection = duckdb.connect(":memory:") if not connection else connection
 
         super().__init__()
 
     def read_to_py_iterator(
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        all_model_fields: Optional[set[str]] = None,
     ) -> Iterator[dict[str, Any]]:
         """Creates an iterable object of rows as dictionaries"""
         return self.read_to_relation(resource, entity_name, schema).pl().iter_rows(named=True)
 
     @read_function(DuckDBPyRelation)
     def read_to_relation(  # pylint: disable=unused-argument
-        self, resource: URI, entity_name: EntityName, schema: type[BaseModel]
+        self,
+        resource: URI,
+        entity_name: EntityName,
+        schema: type[BaseModel],
+        **_,
     ) -> DuckDBPyRelation:
         """Returns a relation object from the source json"""
 
@@ -50,5 +61,7 @@ class DuckDBJSONReader(BaseFileReader):
         }
 
         return self.add_record_index(
-            read_json(resource, columns=ddb_schema, format=self._json_format)  # type: ignore
+            self._connection.read_json(
+                resource, columns=ddb_schema, format=self._json_format  # type: ignore
+            )
         )

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/implementations/duckdb/readers/xml.py

@@ -3,8 +3,9 @@
 
 from typing import Optional
 
+import duckdb
 import polars as pl
-from duckdb import DuckDBPyConnection, DuckDBPyRelation, default_connection
+from duckdb import DuckDBPyConnection, DuckDBPyRelation
 from pydantic import BaseModel
 
 from dve.core_engine.backends.base.reader import read_function
@@ -24,12 +25,18 @@ from dve.core_engine.type_hints import URI
 class DuckDBXMLStreamReader(XMLStreamReader):
     """A reader for XML files"""
 
-    def __init__(self, *, ddb_connection: Optional[DuckDBPyConnection] = None, **kwargs):
-        self.ddb_connection = ddb_connection if ddb_connection else default_connection
+    def __init__(self, *, connection: Optional[DuckDBPyConnection] = None, **kwargs):
+        self._connection = connection if connection else duckdb.connect(":memory:")
         super().__init__(**kwargs)
 
     @read_function(DuckDBPyRelation)
-    def read_to_relation(self, resource: URI, entity_name: str, schema: type[BaseModel]):
+    def read_to_relation(
+        self,
+        resource: URI,
+        entity_name: str,
+        schema: type[BaseModel],
+        **_,
+    ):
         """Returns a relation object from the source xml"""
         if self.xsd_location:
             msg = self._run_xmllint(file_uri=resource)
@@ -49,4 +56,4 @@ class DuckDBXMLStreamReader(XMLStreamReader):
                 data=self.read_to_py_iterator(resource, entity_name, schema), schema=polars_schema
             )
         )
-        return self.ddb_connection.sql("select * from _lazy_frame")
+        return self._connection.sql("select * from _lazy_frame")

{data_validation_engine-0.7.5 → data_validation_engine-0.8.0}/src/dve/core_engine/backends/implementations/duckdb/rules.py

@@ -152,7 +152,7 @@ class DuckDBStepImplementations(BaseStepImplementations[DuckDBPyRelation]):
             group_pl = entity.pl().pivot(
                 columns=[config.pivot_column],
                 values=agg_cols,
-                index=(group_cols or [const_fld]),
+                index=(group_cols or [const_fld]),  # pylint: disable=E0606
                 aggregate_function=config.agg_function,
             )
             if const_fld in group_pl.columns: