data-validation-engine 0.6.2__tar.gz → 0.7.0__tar.gz

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data-validation-engine
-Version: 0.6.2
+Version: 0.7.0
 Summary: `nhs data validation engine` is a framework used to validate data
 License-File: LICENSE
 Author: NHS England
@@ -28,7 +28,7 @@ Requires-Dist: typing_extensions (>=4.6.2,<5.0.0)
 Description-Content-Type: text/markdown
 
 <h1 style="display: flex; align-items: center; gap: 10px;">
-    <img src="overrides/.icons/nhseng.svg" alt="NHS Logo" width="5%" height="100%" align="left">
+    <img src="https://github.com/NHSDigital/data-validation-engine/blob/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
     Data Validation Engine
 </h1>
 
@@ -54,11 +54,11 @@ If you'd like more detailed documentation around these services the please read
 
 The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
 
-Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing][#Contributing] section.
+Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing](#Contributing) section.
 
 ## Installation and usage
 
-The DVE is a Python package and can be installed using `pip`. As of release v0.6.1 we currently support Python 3.10 & 3.11, with Spark version 3.4 and DuckDB version of 1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
+The DVE is a Python package and can be installed using package managers such as [pip](https://pypi.org/project/pip/). As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
 
 If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
 
@@ -67,12 +67,12 @@ Python dependencies are listed in `pyproject.toml`.
 To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
 
 ```
-pip install git+https://github.com/NHSDigital/data-validation-engine.git@v0.6.1
+pip install data-validation-engine
 ```
 
-Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
+*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
 
-Please note - The long term aim is to make the DVE available via PyPi and Conda but we are not quite there yet. Once available this documentation will be updated to contain the new installation options.
+Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
 
 Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
 
@@ -89,10 +89,10 @@ Below is a list of features that we would like to implement or have been request
 | ------- | --------------- | --------- |
 | Open source release | 0.1.0 | Yes |
 | Uplift to Python 3.11 | 0.2.0 | Yes |
-| Upgrade to Pydantic 2.0 | Not yet confirmed | No |
+| Upgrade to Pydantic 2.0 | Before 1.0 release | No |
 | Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
 
-Beyond the Python upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
+Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
 
 ## Contributing
 Please see guidance [here](./CONTRIBUTE.md).

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/README.md

@@ -1,5 +1,5 @@
 <h1 style="display: flex; align-items: center; gap: 10px;">
-    <img src="overrides/.icons/nhseng.svg" alt="NHS Logo" width="5%" height="100%" align="left">
+    <img src="https://github.com/NHSDigital/data-validation-engine/blob/616b55890306db4546177f7effac48ca241857ec/overrides/.icons/nhseng.svg" alt="" width="5%" height="100%" align="left">
     Data Validation Engine
 </h1>
 
@@ -25,11 +25,11 @@ If you'd like more detailed documentation around these services the please read
 
 The DVE has been designed in a way that's modular and can support users who just want to utilise specific "services" from the DVE (i.e. just the file transformation + data contract). Additionally, the DVE is designed to support different backend implementations. As part of the base installation of DVE, you will find backend support for `Spark` and `DuckDB`. So, if you need a `MySQL` backend implementation, you can implement this yourself. Given our organisations requirements, it will be unlikely that we add anymore specific backend implementations into the base package beyond Spark and DuckDB. So, if you are unable to implement this yourself, I would recommend reading the guidance on [requesting new features and raising bug reports here](#requesting-new-features-and-raising-bug-reports).
 
-Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing][#Contributing] section.
+Additionally, if you'd like to contribute a new backend implementation into the base DVE package, then please look at the [Contributing](#Contributing) section.
 
 ## Installation and usage
 
-The DVE is a Python package and can be installed using `pip`. As of release v0.6.1 we currently support Python 3.10 & 3.11, with Spark version 3.4 and DuckDB version of 1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
+The DVE is a Python package and can be installed using package managers such as [pip](https://pypi.org/project/pip/). As of the latest release we support Python 3.10 & 3.11, with Spark v3.4 and DuckDB v1.1. In the future we will be looking to upgrade the DVE to working on a higher versions of Python, DuckDB and Spark.
 
 If you're planning to use the Spark backend implementation, you will also need OpenJDK 11 installed.
 
@@ -38,12 +38,12 @@ Python dependencies are listed in `pyproject.toml`.
 To install the DVE package you can simply install using a package manager such as [pip](https://pypi.org/project/pip/).
 
 ```
-pip install git+https://github.com/NHSDigital/data-validation-engine.git@v0.6.1
+pip install data-validation-engine
 ```
 
-Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
+*Note - Only versions >=0.6.2 are available on PyPi. For older versions please install directly from the git repo or build from source.*
 
-Please note - The long term aim is to make the DVE available via PyPi and Conda but we are not quite there yet. Once available this documentation will be updated to contain the new installation options.
+Once you have installed the DVE you are ready to use it. For guidance on how to create your dischema JSON document (configuration), please read the [documentation](./docs/).
 
 Version 0.0.1 does support a working Python 3.7 installation. However, we will not be supporting any issues with that version of the DVE if you choose to use it. __Use at your own risk__.
 
@@ -60,10 +60,10 @@ Below is a list of features that we would like to implement or have been request
 | ------- | --------------- | --------- |
 | Open source release | 0.1.0 | Yes |
 | Uplift to Python 3.11 | 0.2.0 | Yes |
-| Upgrade to Pydantic 2.0 | Not yet confirmed | No |
+| Upgrade to Pydantic 2.0 | Before 1.0 release | No |
 | Create a more user friendly interface for building and modifying dischema files | Not yet confirmed | No |
 
-Beyond the Python upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
+Beyond the Python and Pydantic upgrade, we cannot confirm the other features will be made available anytime soon. Therefore, if you have the interest and desire to make these features available, then please read the [Contributing](#contributing) section and get involved.
 
 ## Contributing
 Please see guidance [here](./CONTRIBUTE.md).

data_validation_engine-0.7.0/pyproject.toml

@@ -0,0 +1,272 @@
+[tool.poetry]
+name = "data-validation-engine"
+version = "0.7.0"
+description = "`nhs data validation engine` is a framework used to validate data"
+authors = ["NHS England <england.contactus@nhs.net>"]
+readme = "README.md"
+packages = [
+    { include = "dve", from = "src" },
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Operating System :: OS Independent",
+    "Topic :: Software Development :: Libraries",
+    "Typing :: Typed",
+]
+
+[tool.poetry.dependencies]
+python = ">=3.10,<3.12"
+boto3 = ">=1.34.162,<1.36"  # breaking change beyond 1.36
+botocore = ">=1.34.162,<1.36"  # breaking change beyond 1.36
+delta-spark = "2.4.*"
+duckdb = "1.1.*"  # breaking changes beyond 1.1
+Jinja2 = "3.1.*"
+lxml = "^4.9.1"
+openpyxl = "^3.1"
+pandas = "^2.2.2"
+polars = "0.20.*"
+pyarrow = "^17.0.0"
+pydantic = "1.10.15"
+pyspark = "3.4.*"
+typing_extensions = "^4.6.2"
+
+[tool.poetry.group.dev]
+optional = true
+include-groups = [
+    "test",
+    "lint"
+]
+
+[tool.poetry.group.dev.dependencies]
+commitizen = "4.9.1"
+pre-commit = "4.3.0"
+
+[tool.poetry.group.test]
+optional = true
+
+[tool.poetry.group.test.dependencies]
+faker = "18.11.1"
+behave = "1.3.3"
+coverage = "7.11.0"
+moto = {extras = ["s3"], version = "4.0.13"}
+Werkzeug = "3.1.6"
+pytest = "8.4.2"
+pytest-lazy-fixtures = "1.4.0"  # switched from https://github.com/TvoroG/pytest-lazy-fixture as it's no longer supported
+xlsx2csv = "0.8.2"
+
+[tool.poetry.group.lint]
+optional = true
+
+[tool.poetry.group.lint.dependencies]
+black = "24.3.0"
+astroid = "2.14.2"
+isort = "5.11.5"
+pylint = "2.16.4"
+mypy = "0.991"
+boto3-stubs = {extras = ["essential"], version = "1.26.72"}
+botocore-stubs = "1.29.72"
+pandas-stubs = "1.2.0.62"
+types-awscrt = "0.19.1"
+types-openpyxl = "3.1.0.19"
+types-pytz = "2023.3.0.1"
+types-PyYAML = "6.0.12.1"
+types-requests = "2.31.0.2"
+types-s3transfer = "0.6.2"
+types-setuptools = "68.2.0.0"
+types-urllib3 = "1.26.25.14"
+types-xmltodict = "0.13.0.3"
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "B", "D1", "D2", "I"]
+ignore = ["B028", "D213", "D203", "D205", "D107", "D105"]
+
+[tool.mypy]
+plugins = ["pydantic.mypy"]
+
+[[tool.mypy.overrides]]
+module = "polars.*"
+follow_imports = "skip"
+# ^language server knows what's going on, but mypy can't find attributes on Self? type
+
+[tool.black]
+line-length = 100
+
+[tool.isort]
+profile = "black"
+line_length = 100
+
+[tool.pytest]
+testpaths = ["tests"]
+
+[tool.coverage]
+[tool.coverage.run]
+command_line = "-m pytest"
+concurrency = ["multiprocessing"]
+source_pkgs = [
+    "dve",
+]
+
+[tool.coverage.report]
+show_missing = true
+
+[tool.pylint]
+init-hook = "import sys; sys.path.append('./pylint_checkers')"
+load-plugins = "check_typing_imports"
+
+[tool.pylint.main]
+extension-pkg-allow-list = ["pyspark", "lxml", "pydantic"]
+fail-under = 10.0
+ignore = ["CVS"]
+ignore-patterns = ["^\\.#"]
+jobs = 1
+limit-inference-results = 100
+persistent = true
+py-version = "3.10"
+suggestion-mode = true
+
+[tool.pylint.basic]
+argument-naming-style = "snake_case"
+attr-naming-style = "snake_case"
+bad-names = ["foo", "bar", "baz", "toto", "tutu", "tata"]
+class-attribute-naming-style = "any"
+class-const-naming-style = "UPPER_CASE"
+class-naming-style = "PascalCase"
+const-naming-style = "UPPER_CASE"
+docstring-min-length = -1
+function-naming-style = "snake_case"
+good-names = ["i", "j", "k", "ex", "Run", "df", "_", "f"]
+inlinevar-naming-style = "any"
+method-naming-style = "snake_case"
+module-naming-style = "snake_case"
+no-docstring-rgx = "^_"
+property-classes = ["abc.abstractproperty"]
+variable-naming-style = "snake_case"
+
+[tool.pylint.classes]
+defining-attr-methods = ["__init__", "__new__", "setUp", "__post_init__"]
+exclude-protected = ["_asdict", "_fields", "_replace", "_source", "_make"]
+valid-classmethod-first-arg = ["cls"]
+valid-metaclass-classmethod-first-arg = ["cls"]
+
+[tool.pylint.design]
+max-args = 8
+max-attributes = 10
+max-bool-expr = 5
+max-branches = 15
+max-locals = 20
+max-parents = 7
+max-public-methods = 20
+max-returns = 6
+max-statements = 50
+min-public-methods = 2
+
+[tool.pylint.exceptions]
+overgeneral-exceptions = ["BaseException", "Exception"]
+
+[tool.pylint.format]
+ignore-long-lines = "^\\s*(# )?<?https?://\\S+>?$"
+indent-after-paren = 4
+indent-string = "    "
+max-line-length = 100
+max-module-lines = 1000
+
+[tool.pylint.imports]
+known-third-party = ["enchant"]
+
+[tool.pylint.logging]
+logging-format-style = "old"
+logging-modules = ["logging"]
+
+[tool.pylint."messages control"]
+confidence = [
+    "HIGH",
+    "CONTROL_FLOW",
+    "INFERENCE",
+    "INFERENCE_FAILURE",
+    "UNDEFINED",
+]
+disable = [
+    "raw-checker-failed",
+    "bad-inline-option",
+    "locally-disabled",
+    "file-ignored",
+    "suppressed-message",
+    "useless-suppression",
+    "deprecated-pragma",
+    "use-symbolic-message-instead",
+    "logging-fstring-interpolation",
+    "fixme",
+]
+enable = ["c-extension-no-member"]
+
+[tool.pylint.miscellaneous]
+notes = ["FIXME", "XXX", "TODO"]
+
+[tool.pylint.refactoring]
+max-nested-blocks = 5
+never-returning-functions = ["sys.exit", "argparse.parse_error"]
+
+[tool.pylint.reports]
+evaluation = "max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10))"
+score = true
+
+[tool.pylint.similarities]
+ignore-comments = true
+ignore-docstrings = true
+ignore-imports = true
+ignore-signatures = true
+min-similarity-lines = 10
+
+[tool.pylint.spelling]
+max-spelling-suggestions = 4
+spelling-ignore-comment-directives = "fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy:"
+
+[tool.pylint.typecheck]
+contextmanager-decorators = ["contextlib.contextmanager"]
+ignore-none = true
+ignore-on-opaque-inference = true
+ignored-checks-for-mixins = [
+    "no-member",
+    "not-async-context-manager",
+    "not-context-manager",
+    "attribute-defined-outside-init",
+]
+ignored-classes = [
+    "optparse.Values",
+    "thread._local",
+    "_thread._local",
+    "argparse.Namespace",
+]
+missing-member-hint = true
+missing-member-hint-distance = 1
+missing-member-max-choices = 1
+mixin-class-rgx = ".*[Mm]ixin"
+
+[tool.pylint.variables]
+allow-global-unused-variables = true
+callbacks = ["cb_", "_cb"]
+dummy-variables-rgx = "_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_"
+ignored-argument-names = "_.*|^ignored_|^unused_"
+redefining-builtins-modules = [
+    "six.moves",
+    "past.builtins",
+    "future.builtins",
+    "builtins",
+    "io",
+]
+
+[tool.commitizen]
+name = "cz_conventional_commits"
+tag_format = "v$major.$minor.$patch"
+version_scheme = "pep440"
+version_provider = "poetry"
+update_changelog_on_bump = true
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/base/backend.py

@@ -163,7 +163,7 @@ class BaseBackend(Generic[EntityType], ABC):
             return entities, get_parent(processing_errors_uri), successful
 
         for entity_name, entity in entities.items():
-            entities[entity_name] = self.step_implementations.add_row_id(entity)
+            entities[entity_name] = self.step_implementations.add_record_index(entity)
 
         # TODO: Handle entity manager creation errors.
         entity_manager = EntityManager(entities, reference_data)
@@ -172,9 +172,6 @@ class BaseBackend(Generic[EntityType], ABC):
         # TODO: and return uri to errors
         _ = self.step_implementations.apply_rules(working_dir, entity_manager, rule_metadata)
 
-        for entity_name, entity in entity_manager.entities.items():
-            entity_manager.entities[entity_name] = self.step_implementations.drop_row_id(entity)
-
         return entity_manager.entities, get_parent(dc_feedback_errors_uri), True
 
     def process(

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/base/contract.py

@@ -337,9 +337,9 @@ class BaseDataContract(Generic[EntityType], ABC):
         successful = True
         for entity_name, resource in entity_locations.items():
             reader_metadata = contract_metadata.reader_metadata[entity_name]
-            extension = "." + (
-                get_file_suffix(resource) or ""
-            ).lower()  # Already checked that extension supported.
+            extension = (
+                "." + (get_file_suffix(resource) or "").lower()
+            )  # Already checked that extension supported.
 
             reader_config = reader_metadata[extension]
             reader_type = get_reader(reader_config.reader)
@@ -369,6 +369,14 @@ class BaseDataContract(Generic[EntityType], ABC):
 
         return entities, dedup_messages(messages), successful
 
+    def add_record_index(self, entity: EntityType, **kwargs) -> EntityType:
+        """Add a record index to the entity"""
+        raise NotImplementedError(f"add_record_index not implemented in {self.__class__}")
+
+    def drop_record_index(self, entity: EntityType, **kwargs) -> EntityType:
+        """Drop a record index from the entity"""
+        raise NotImplementedError(f"drop_record_index not implemented in {self.__class__}")
+
     @abstractmethod
     def apply_data_contract(
         self,

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/base/reader.py

@@ -127,6 +127,14 @@ class BaseFileReader(ABC):
 
         return reader_func(self, resource, entity_name, schema)
 
+    def add_record_index(self, entity: EntityType, **kwargs) -> EntityType:
+        """Add a record index to the entity"""
+        raise NotImplementedError(f"add_record_index not implemented in {self.__class__}")
+
+    def drop_record_index(self, entity: EntityType, **kwargs) -> EntityType:
+        """Drop a record index to the entity"""
+        raise NotImplementedError(f"drop_record_index not implemented in {self.__class__}")
+
     def write_parquet(
         self,
         entity: EntityType,

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/base/rules.py

@@ -135,15 +135,13 @@ class BaseStepImplementations(Generic[EntityType], ABC):  # pylint: disable=too-
         """Method to register all custom dve functions for use during business rules application"""
         raise NotImplementedError()
 
-    @staticmethod
-    def add_row_id(entity: EntityType) -> EntityType:
-        """Add a unique row id field to an entity"""
-        raise NotImplementedError()
+    def add_record_index(self, entity: EntityType, **kwargs) -> EntityType:
+        """Add a record index to the entity"""
+        raise NotImplementedError(f"add_record_index not implemented in {self.__class__}")
 
-    @staticmethod
-    def drop_row_id(entity: EntityType) -> EntityType:
-        """Add a unique row id field to an entity"""
-        raise NotImplementedError()
+    def drop_record_index(self, entity: EntityType) -> EntityType:
+        """Drop a unique row id field to an entity"""
+        raise NotImplementedError(f"drop_record_index not implemented in {self.__class__}")
 
     @classmethod
     def _raise_notimplemented_error(

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/implementations/duckdb/contract.py

@@ -29,6 +29,7 @@ from dve.core_engine.backends.base.utilities import (
 )
 from dve.core_engine.backends.implementations.duckdb.duckdb_helpers import (
     duckdb_read_parquet,
+    duckdb_record_index,
     duckdb_write_parquet,
     get_duckdb_type_from_annotation,
     relation_is_empty,
@@ -37,6 +38,7 @@ from dve.core_engine.backends.implementations.duckdb.types import DuckDBEntities
 from dve.core_engine.backends.metadata.contract import DataContractMetadata
 from dve.core_engine.backends.types import StageSuccessful
 from dve.core_engine.backends.utilities import get_polars_type_from_annotation, stringify_model
+from dve.core_engine.constants import RECORD_INDEX_COLUMN_NAME
 from dve.core_engine.message import FeedbackMessage
 from dve.core_engine.type_hints import URI, EntityLocations
 from dve.core_engine.validation import RowValidator, apply_row_validator_helper
@@ -54,6 +56,7 @@ class PandasApplyHelper:
         return row  # no op
 
 
+@duckdb_record_index
 @duckdb_write_parquet
 @duckdb_read_parquet
 class DuckDBDataContract(BaseDataContract[DuckDBPyRelation]):
@@ -144,10 +147,12 @@ class DuckDBDataContract(BaseDataContract[DuckDBPyRelation]):
                     fld.name: get_duckdb_type_from_annotation(fld.annotation)
                     for fld in entity_fields.values()
                 }
+                ddb_schema[RECORD_INDEX_COLUMN_NAME] = get_duckdb_type_from_annotation(int)
                 polars_schema: dict[str, PolarsType] = {
                     fld.name: get_polars_type_from_annotation(fld.annotation)
                     for fld in entity_fields.values()
                 }
+                polars_schema[RECORD_INDEX_COLUMN_NAME] = get_polars_type_from_annotation(int)
                 if relation_is_empty(relation):
                     self.logger.warning(f"+ Empty relation for {entity_name}")
                     empty_df = pl.DataFrame([], schema=polars_schema)  # type: ignore # pylint: disable=W0612
@@ -170,6 +175,9 @@ class DuckDBDataContract(BaseDataContract[DuckDBPyRelation]):
 
                 self.logger.info(f"Data contract found {msg_count} issues in {entity_name}")
 
+                if RECORD_INDEX_COLUMN_NAME not in relation.columns:
+                    relation = self.add_record_index(relation)
+
                 casting_statements = [
                     (
                         self.generate_ddb_cast_statement(column, dtype)

{data_validation_engine-0.6.2 → data_validation_engine-0.7.0}/src/dve/core_engine/backends/implementations/duckdb/duckdb_helpers.py

@@ -12,13 +12,14 @@ from urllib.parse import urlparse
 
 import duckdb.typing as ddbtyp
 import numpy as np
-from duckdb import DuckDBPyConnection, DuckDBPyRelation
+from duckdb import DuckDBPyConnection, DuckDBPyRelation, StarExpression
 from duckdb.typing import DuckDBPyType
 from pandas import DataFrame
 from pydantic import BaseModel
 from typing_extensions import Annotated, get_args, get_origin, get_type_hints
 
 from dve.core_engine.backends.base.utilities import _get_non_heterogenous_type
+from dve.core_engine.constants import RECORD_INDEX_COLUMN_NAME
 from dve.core_engine.type_hints import URI
 from dve.parser.file_handling.service import LocalFilesystemImplementation, _get_implementation
 
@@ -286,3 +287,29 @@ def duckdb_rel_to_dictionaries(
     cols: tuple[str] = tuple(entity.columns)  # type: ignore
     while rows := entity.fetchmany(batch_size):
         yield from (dict(zip(cols, rw)) for rw in rows)
+
+
+def _add_duckdb_record_index(
+    self, entity: DuckDBPyRelation  # pylint: disable=W0613
+) -> DuckDBPyRelation:
+    """Add record index to duckdb relation"""
+    if RECORD_INDEX_COLUMN_NAME in entity.columns:
+        return entity
+
+    return entity.select(f"*, row_number() OVER () as {RECORD_INDEX_COLUMN_NAME}")
+
+
+def _drop_duckdb_record_index(
+    self, entity: DuckDBPyRelation  # pylint: disable=W0613
+) -> DuckDBPyRelation:
+    """Drop record index from duckdb relation"""
+    if RECORD_INDEX_COLUMN_NAME not in entity.columns:
+        return entity
+    return entity.select(StarExpression(exclude=[RECORD_INDEX_COLUMN_NAME]))
+
+
+def duckdb_record_index(cls):
+    """Class decorator to add record index methods for duckdb implementations"""
+    setattr(cls, "add_record_index", _add_duckdb_record_index)
+    setattr(cls, "drop_record_index", _drop_duckdb_record_index)
+    return cls