tgedr-dataops 0.0.36__tar.gz → 1.0.1__tar.gz

tgedr_dataops-1.0.1/PKG-INFO

@@ -0,0 +1,72 @@
+Metadata-Version: 2.4
+Name: tgedr-dataops
+Version: 1.0.1
+Summary: data operations related code
+Author-email: joao tiago viegas <3536754+jtviegas@users.noreply.github.com>
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tgedr-dataops-abs==0.0.3
+Requires-Dist: s3fs==2024.5.0
+Requires-Dist: boto3==1.34.106
+Requires-Dist: openpyxl==3.1.2
+Requires-Dist: pyarrow>=23.0.0
+Requires-Dist: moto>=5.1.20
+Dynamic: license-file
+
+# data-ops
+
+![Coverage](./coverage.svg)
+[![PyPI](https://img.shields.io/pypi/v/tgedr-dataops)](https://pypi.org/project/tgedr-dataops/)
+
+
+
+data operations related code
+
+## motivation
+*data-ops* is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel
+
+## installation
+        `pip install tgedr-dataops`
+
+## package namespaces and its contents
+
+#### commons
+- __S3Connector__: base class to be extended, providing a connection session with aws s3 resources
+- __utils_fs__: utility module with file system related functions ([example](tests/tgedr_dataops/commons/test_utils_fs.py))
+
+#### quality
+- __PandasValidation__ : __GreatExpectationsValidation__ implementation to validate pandas dataframes with Great Expectations library ([example](tests/tgedr_dataops/quality/test_pandas_validation.py))
+
+
+#### sink
+- __LocalFsFileSink__: __Sink__ implementation class used to save/persist an object/file to a local fs location ([example](tests/tgedr_dataops/sink/test_localfs_file_sink.py))
+- __S3FileSink__: __Sink__ implementation class used to save/persist a local object/file to an s3 bucket ([example](tests/tgedr_dataops/sink/test_s3_file_sink.py))
+
+#### source
+- __AbstractS3FileSource__: abstract __Source__ class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
+- __LocalFsFileSource__: __Source__ implementation class used to retrieve local objects/files to another local fs location ([example](tests/tgedr_dataops/source/test_localfs_file_source.py))
+- __PdDfS3Source__: __Source__ implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file ([example csv](tests/tgedr_dataops/source/test_pd_df_s3_source_csv.py), [example excel](tests/tgedr_dataops/source/test_pd_df_s3_source_excel.py))
+- __S3FileCopy__: __Source__ implementation class used to copy objects/files from an s3 bucket to another s3 bucket ([example](tests/tgedr_dataops/source/test_s3_copy.py))
+- __S3FileExtendedSource__: __Source__ implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method `get_metadata` providing sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")([example](tests/tgedr_dataops/source/test_s3_file_extended_source.py))
+- __S3FileSource__: __Source__ implementation class used to retrieve objects/files from s3 bucket to local fs location ([example](tests/tgedr_dataops/source/test_s3_file_source.py))
+
+#### store
+- __FsSinglePartitionParquetStore__ : abstract __Store__ implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
+- __LocalFsSinglePartitionParquetStore__ : __FsSinglePartitionParquetStore__ implementation using local file system ([example](tests/tgedr_dataops/store/test_local_fs_single_partition_parquet.py))
+- __S3FsSinglePartitionParquetStore__ : __FsSinglePartitionParquetStore__ implementation using aws s3 file system ([example](tests/tgedr_dataops/store/MANUAL_test_s3_single_partition_parquet.py))
+
+
+## development
+- main requirements:
+  - _uv_  
+  - _bash_
+- Clone the repository like this:
+
+  ``` bash
+  git clone git@github.com:tgedr/pycommons
+  ```
+- cd into the folder: `cd pycommons`
+- install requirements: `./helper.sh reqs`
+
+

tgedr_dataops-1.0.1/README.md

@@ -0,0 +1,56 @@
+# data-ops
+
+![Coverage](./coverage.svg)
+[![PyPI](https://img.shields.io/pypi/v/tgedr-dataops)](https://pypi.org/project/tgedr-dataops/)
+
+
+
+data operations related code
+
+## motivation
+*data-ops* is a library with tested and used code aligning on some standards regarding code structure and quality and to avoid reinventing the wheel
+
+## installation
+        `pip install tgedr-dataops`
+
+## package namespaces and its contents
+
+#### commons
+- __S3Connector__: base class to be extended, providing a connection session with aws s3 resources
+- __utils_fs__: utility module with file system related functions ([example](tests/tgedr_dataops/commons/test_utils_fs.py))
+
+#### quality
+- __PandasValidation__ : __GreatExpectationsValidation__ implementation to validate pandas dataframes with Great Expectations library ([example](tests/tgedr_dataops/quality/test_pandas_validation.py))
+
+
+#### sink
+- __LocalFsFileSink__: __Sink__ implementation class used to save/persist an object/file to a local fs location ([example](tests/tgedr_dataops/sink/test_localfs_file_sink.py))
+- __S3FileSink__: __Sink__ implementation class used to save/persist a local object/file to an s3 bucket ([example](tests/tgedr_dataops/sink/test_s3_file_sink.py))
+
+#### source
+- __AbstractS3FileSource__: abstract __Source__ class used to retrieve objects/files from s3 bucket to local fs location circumventing some formats download limitation
+- __LocalFsFileSource__: __Source__ implementation class used to retrieve local objects/files to another local fs location ([example](tests/tgedr_dataops/source/test_localfs_file_source.py))
+- __PdDfS3Source__: __Source__ implementation class used to read a pandas dataframe from s3, whether a csv or an excel (xslx) file ([example csv](tests/tgedr_dataops/source/test_pd_df_s3_source_csv.py), [example excel](tests/tgedr_dataops/source/test_pd_df_s3_source_excel.py))
+- __S3FileCopy__: __Source__ implementation class used to copy objects/files from an s3 bucket to another s3 bucket ([example](tests/tgedr_dataops/source/test_s3_copy.py))
+- __S3FileExtendedSource__: __Source__ implementation class used to retrieve objects/files from s3 bucket to local fs location with the extra method `get_metadata` providing sile metadata ("LastModified", "ContentLength", "ETag", "VersionId", "ContentType")([example](tests/tgedr_dataops/source/test_s3_file_extended_source.py))
+- __S3FileSource__: __Source__ implementation class used to retrieve objects/files from s3 bucket to local fs location ([example](tests/tgedr_dataops/source/test_s3_file_source.py))
+
+#### store
+- __FsSinglePartitionParquetStore__ : abstract __Store__ implementation defining persistence on parquet files with an optional single partition, regardless of the location it should persist
+- __LocalFsSinglePartitionParquetStore__ : __FsSinglePartitionParquetStore__ implementation using local file system ([example](tests/tgedr_dataops/store/test_local_fs_single_partition_parquet.py))
+- __S3FsSinglePartitionParquetStore__ : __FsSinglePartitionParquetStore__ implementation using aws s3 file system ([example](tests/tgedr_dataops/store/MANUAL_test_s3_single_partition_parquet.py))
+
+
+## development
+- main requirements:
+  - _uv_  
+  - _bash_
+- Clone the repository like this:
+
+  ``` bash
+  git clone git@github.com:tgedr/pycommons
+  ```
+- cd into the folder: `cd pycommons`
+- install requirements: `./helper.sh reqs`
+
+

tgedr_dataops-1.0.1/pyproject.toml

@@ -0,0 +1,135 @@
+[project]
+name = "tgedr-dataops"
+version = "1.0.1"
+description = "data operations related code"
+authors = [
+    {name = "joao tiago viegas",email = "3536754+jtviegas@users.noreply.github.com"}
+]
+readme = "README.md"
+requires-python = ">=3.11"
+
+dependencies = [
+    "tgedr-dataops-abs==0.0.3",
+    "s3fs==2024.5.0",
+    "boto3==1.34.106",
+    "openpyxl==3.1.2",
+    "pyarrow>=23.0.0",
+    "moto>=5.1.20",
+]
+[dependency-groups]
+dev = [
+    "pre-commit~=4.2.0",
+    "pytest~=8.3.5",
+    "pytest-bdd~=8.1.0",
+    "pytest-cov~=4.1.0",
+    "pytest-mock~=3.15.0",
+    "ruff==0.9.10",
+    "bandit==1.8.3",
+    "safety==3.5.1",
+    "typer<0.17.0",
+    "genbadge[coverage]>=1.1.3",
+]
+
+# [project.scripts]
+# run = "tgedr.pycommons.entrypoint:entrypoint" 
+
+[build-system]
+requires = ["setuptools>=78.1.0", "wheel>=0.45.1"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.package-data]
+"*" = ["CHANGELOG"]
+
+[tool.coverage.paths]
+source = ["src/"]
+
+[tool.coverage.run]
+source = ["src/"]
+include = ["src/*"]
+omit = [
+    "*/tests/*",
+    "*/test_*",
+    "*/__pycache__/*",
+    "*/migrations/*",
+    "*/venv/*",
+    "*/.venv/*"
+]
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "def __repr__",
+    "raise AssertionError",
+    "raise NotImplementedError",
+    "if __name__ == .__main__.:",
+    "if TYPE_CHECKING:",
+]
+show_missing = true
+skip_covered = false
+skip_empty = false
+
+[tool.pytest.ini_options]
+# bdd_features_base_dir = "documentation/features"
+pythonpath = "."
+
+[tool.ruff]
+exclude = [
+    ".bzr",
+    ".direnv",
+    ".eggs",
+    ".git",
+    ".git-rewrite",
+    ".hg",
+    ".ipynb_checkpoints",
+    ".mypy_cache",
+    ".nox",
+    ".pants.d",
+    ".pyenv",
+    ".pytest_cache",
+    ".pytype",
+    ".ruff_cache",
+    ".svn",
+    ".tox",
+    ".venv",
+    ".vscode",
+    "__pypackages__",
+    "_build",
+    "buck-out",
+    "build",
+    "dist",
+    "node_modules",
+    "site-packages",
+    "venv",
+    "tests/",
+    "typings/"
+]
+
+line-length = 120
+indent-width = 4
+
+[tool.ruff.lint]
+select = ["ALL"]
+ignore = ["D203", "S101", "D104", "INP001", "D213", "COM812", "I001", 
+    "D401", "D407", "RET504", "PLR2004", "FA102", "E501", "EXE002", "PLR0913", 
+    "PLR0912", "C901", "PLR0911", "D413", "N818", "B024", "ANN401", "SIM300",
+    "FBT001", "FBT002", "G004", "TRY003", "EM102", "EM101", "PD015", "PD901"]
+fixable = ["ALL"]
+unfixable = []
+# Allow unused variables when underscore-prefixed.
+dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
+
+[tool.ruff.format]
+# Like Black, use double quotes for strings.
+quote-style = "double"
+# Like Black, indent with spaces, rather than tabs.
+indent-style = "space"
+# Like Black, respect magic trailing commas.
+skip-magic-trailing-comma = false
+# Like Black, automatically detect the appropriate line ending.
+line-ending = "auto"

{tgedr_dataops-0.0.36/src/tgedr/dataops → tgedr_dataops-1.0.1/src/tgedr_dataops}/commons/s3_connector.py

@@ -1,17 +1,30 @@
+"""AWS S3 connector module.
+
+This module provides the S3Connector class for establishing and managing
+connections to AWS S3 resources.
+"""
 import os
 import boto3
 
 
 class S3Connector:
-    """utility base class to be extended, providing a connection session with aws s3 resources"""
+    """utility base class to be extended, providing a connection session with aws s3 resources."""
 
-    def __init__(self):
+    def __init__(self) -> None:
+        """Initialize the S3Connector with empty session, resource, and client attributes."""
         self.__resource = None
         self.__session = None
         self.__client = None
 
     @property
-    def _session(self):  # pragma: no cover
+    def _session(self) -> boto3.Session:
+        """Get or create a boto3 session.
+
+        Returns
+        -------
+        boto3.Session
+            The boto3 session instance, using credentials if configured.
+        """
         if self.__session is None:
             if "1" == os.getenv("S3_CONNECTOR_USE_CREDENTIALS", default="0"):
                 self.__session = boto3.Session(
@@ -26,13 +39,27 @@ class S3Connector:
         return self.__session
 
     @property
-    def _resource(self):
+    def _resource(self) -> boto3.resources.base.ServiceResource:
+        """Get or create an S3 resource.
+
+        Returns
+        -------
+        boto3.resources.base.ServiceResource
+            The boto3 S3 resource instance.
+        """
         if self.__resource is None:
             self.__resource = self._session.resource("s3")
         return self.__resource
 
     @property
-    def _client(self):
+    def _client(self) -> boto3.client:
+        """Get or create an S3 client.
+
+        Returns
+        -------
+        boto3.client
+            The boto3 S3 client instance.
+        """
         if self.__client is None:
             self.__client = self._session.client("s3")
         return self.__client

tgedr_dataops-1.0.1/src/tgedr_dataops/commons/utils_fs.py

@@ -0,0 +1,187 @@
+"""Filesystem utility functions for file and path operations.
+
+This module provides utilities for:
+- Creating temporary files and directories
+- Parsing and processing S3 URLs and paths
+- Extracting URL protocols
+- Generating file hashes
+"""
+import tempfile
+import re
+import hashlib
+from pathlib import Path
+
+
+def temp_dir(root: str | None = None, suffix: str | None = None, prefix: str | None = None) -> str:
+    """Create a temporary directory and return its path.
+
+    Parameters
+    ----------
+    root : str | None
+        Directory where the temporary directory will be created.
+    suffix : str | None
+        Suffix for the temporary directory name.
+    prefix : str | None
+        Prefix for the temporary directory name.
+
+    Returns
+    -------
+    str
+        Path to the created temporary directory.
+    """
+    return tempfile.mkdtemp(suffix=suffix, prefix=prefix, dir=root)
+
+
+def temp_file(
+    root: str | None = None, suffix: str | None = None, prefix: str | None = None,
+    discard_handle: bool = True
+) -> str | tuple[int, str]:
+    """Create a temporary file and return its path or handle and path.
+
+    Parameters
+    ----------
+    root : str | None
+        Directory where the temporary file will be created.
+    suffix : str | None
+        Suffix for the temporary file name.
+    prefix : str | None
+        Prefix for the temporary file name.
+    discard_handle : bool
+        If True, return only the file path. If False, return tuple of (handle, path).
+
+    Returns
+    -------
+    str | tuple[int, str]
+        File path if discard_handle is True, otherwise tuple of (file handle, file path).
+    """
+    h, f = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=root)
+    if discard_handle:
+        return f
+    return (h, f)
+
+
+def resolve_url_protocol(url: str) -> str:
+    """Extract the protocol from a URL.
+
+    Parameters
+    ----------
+    url : str
+        The URL to extract the protocol from.
+
+    Returns
+    -------
+    str
+        The protocol (e.g., 'http://', 'https://') or None if no protocol is found.
+    """
+    result = None
+    group_match = re.search("(.*://).*", url)
+    if group_match is not None:
+        result = group_match.group(1)
+    return result
+
+
+def resolve_s3_protocol(url: str) -> str:
+    """Extract the S3 protocol from a URL.
+
+    Parameters
+    ----------
+    url : str
+        The URL to extract the S3 protocol from.
+
+    Returns
+    -------
+    str
+        The S3 protocol (e.g., 's3://', 's3a://') or None if no S3 protocol is found.
+    """
+    result = None
+    group_match = re.search("(s3[a]?://).*", url)
+    if group_match is not None:
+        result = group_match.group(1)
+    return result
+
+
+def remove_s3_protocol(url: str) -> str:
+    """Remove the S3 protocol prefix from a URL.
+
+    Parameters
+    ----------
+    url : str
+        The S3 URL to remove the protocol from.
+
+    Returns
+    -------
+    str
+        The URL without the S3 protocol prefix (s3:// or s3a://).
+    """
+    if url.startswith("s3://"):
+        result = url[5:]
+    elif url.startswith("s3a://"):
+        result = url[6:]
+    else:
+        result = url
+    return result
+
+
+def process_s3_path(path: str) -> tuple[str, str]:
+    """Extract bucket and key from an S3 path.
+
+    Parameters
+    ----------
+    path : str
+        The S3 path to process (with or without protocol).
+
+    Returns
+    -------
+    tuple[str, str]
+        A tuple containing (bucket, key).
+    """
+    no_protocol_path = remove_s3_protocol(path)
+    path_elements = no_protocol_path.split("/")
+    bucket = path_elements[0]
+    key = "/".join(path_elements[1:])
+    return (bucket, key)
+
+
+def process_s3_url(url: str) -> tuple[str, str, str]:
+    """Extract protocol, bucket, and key from an S3 URL.
+
+    Parameters
+    ----------
+    url : str
+        The S3 URL to process.
+
+    Returns
+    -------
+    tuple[str, str, str]
+        A tuple containing (protocol, bucket, key).
+    """
+    protocol = resolve_s3_protocol(url)
+    no_protocol_url = remove_s3_protocol(url)
+    path_elements = no_protocol_url.split("/")
+    bucket = path_elements[0]
+    key = "/".join(path_elements[1:])
+    return ("" if protocol is None else protocol, bucket, key)
+
+
+def hash_file(filepath: str, hash_func=hashlib.sha256) -> str:  # noqa: ANN001
+    """Generate a hash for a file.
+
+    Args:
+        filepath (str): The path to the file.
+        hash_func: A hashlib hash function, e.g., hashlib.md5().
+
+    Returns:
+        str: The hexadecimal hash string of the file.
+    """
+    # Initialize the hash object
+    hasher = hash_func()
+
+    # Open the file in binary read mode
+    with Path(filepath).open("rb") as file:
+        # Read the file in chunks to avoid using too much memory
+        chunk_size = 8192
+        while chunk := file.read(chunk_size):
+            hasher.update(chunk)
+
+    # Return the hexadecimal digest of the hash
+    return hasher.hexdigest()

tgedr_dataops-1.0.1/src/tgedr_dataops/quality/pandas_validation.py

@@ -0,0 +1,21 @@
+"""Pandas DataFrame validation implementation module.
+
+This module provides the Pandas-specific implementation of Great Expectations validation.
+"""
+from great_expectations.execution_engine import ExecutionEngine
+from great_expectations.execution_engine import PandasExecutionEngine
+from tgedr_dataops_abs.great_expectations_validation import GreatExpectationsValidation
+
+
+class PandasValidation(GreatExpectationsValidation):
+    """Pandas DataFrame validation implementation."""
+
+    def _get_execution_engine(self, batch_data_dict: dict) -> ExecutionEngine:
+        """Get the execution engine used by the validation implementation.
+
+        Returns
+        -------
+        ExecutionEngine
+            The execution engine instance.
+        """
+        return PandasExecutionEngine(batch_data_dict=batch_data_dict)

tgedr_dataops-1.0.1/src/tgedr_dataops/sink/local_fs_file_sink.py

@@ -0,0 +1,77 @@
+"""Local filesystem sink implementation for persisting files.
+
+This module provides:
+- LocalFsFileSink: sink class for saving/persisting files to local filesystem.
+"""
+
+import logging
+import shutil
+from pathlib import Path
+from typing import Any
+
+from tgedr_dataops_abs.sink import Sink, SinkException
+
+
+logger = logging.getLogger(__name__)
+
+
+class LocalFsFileSink(Sink):
+    """sink class used to save/persist an object/file to a local fs location."""
+
+    CONTEXT_SOURCE_PATH = "source"
+    CONTEXT_TARGET_PATH = "target"
+
+    def put(self, context: dict[str, Any] | None = None) -> Any:
+        """Copy a file from source to target on local filesystem.
+
+        Parameters
+        ----------
+        context : dict[str, Any], optional
+            Context dictionary containing 'source' and 'target' paths.
+
+        Raises
+        ------
+        SinkException
+            If source or target context is missing.
+        """
+        logger.info(f"[put|in] ({context})")
+
+        if self.CONTEXT_SOURCE_PATH not in context:
+            raise SinkException(f"you must provide context for {self.CONTEXT_SOURCE_PATH}")
+        if self.CONTEXT_TARGET_PATH not in context:
+            raise SinkException(f"you must provide context for {self.CONTEXT_TARGET_PATH}")
+
+        source = context[self.CONTEXT_SOURCE_PATH]
+        target = context[self.CONTEXT_TARGET_PATH]
+
+        shutil.copy(source, target)
+        logger.info("[put|out]")
+
+    def delete(self, context: dict[str, Any] | None = None) -> None:
+        """Delete a file or directory from local filesystem.
+
+        Parameters
+        ----------
+        context : dict[str, Any], optional
+            Context dictionary containing 'target' path.
+
+        Raises
+        ------
+        SinkException
+            If target context is missing or target is neither file nor directory.
+        """
+        logger.info(f"[delete|in] ({context})")
+
+        if self.CONTEXT_TARGET_PATH not in context:
+            raise SinkException(f"you must provide context for {self.CONTEXT_TARGET_PATH}")
+
+        target = Path(context[self.CONTEXT_TARGET_PATH])
+
+        if target.is_file():
+            target.unlink()
+        elif target.is_dir():
+            shutil.rmtree(target)
+        else:
+            raise SinkException(f"[delete] is it a dir or a folder? {target}")
+
+        logger.info("[delete|out]")