data-engineering-exp 0.1.0__tar.gz

data_engineering_exp-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tu Nombre Completo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_engineering_exp-0.1.0/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.3
+Name: data-engineering-exp
+Version: 0.1.0
+Summary: A minimalist, agnostic Python framework to standardize data engineering pipelines.
+License: MIT
+Keywords: data-engineering,pyspark,pandas,data-catalog
+Author: idperez720
+Author-email: ivandavidperez4@gmail.com
+Requires-Python: >=3.11,<4.0.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: click (>=8.4.1,<9.0.0)
+Requires-Dist: pandas (>=3.0.3,<4.0.0)
+Requires-Dist: pyspark (>=4.1.2,<5.0.0)
+Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
+Project-URL: Repository, https://github.com/idperez720/data-engineering-exp
+Description-Content-Type: text/markdown
+
+# Data Engineering Experience 🚀
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Python Version](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.14-blue)](https://www.python.org)
+
+**dex** (*Data Engineering Experience*) is a minimalist, agnostic Python framework designed to streamline and standardize data engineering pipelines. By embracing **Convention over Configuration**, `dex` eliminates environment friction, absolute path hardcoding, and complex PySpark session management.
+
+---
+
+## ✨ Key Features
+
+* **Zero-Config File Discovery:** Automatic tree-walking directory resolution anchors your data catalog using your local `pyproject.toml` file.
+* **Decentralized Catalog:** Declare your metadata layouts inside modular, self-contained mini-YAML files.
+* **Elastic Processing Runtimes:** Switch dynamically between **Pandas** and **PySpark** execution engines using exactly the same unified interface.
+* **Interactive CLI Scaffolding:** Spin up a new production-ready data directory structure instantly with `dex init`.
+
+---
+
+## 📦 Installation
+
+*(Once published to PyPI)*
+```bash
+pip install dex
+```
+Or install it directly from the source repository using Poetry:
+```bash
+poetry add git+[https://github.com/idperez720/data-engineering-exp.git](https://github.com/idperez720/data-engineering-exp.git)
+```
+
+---
+
+## 🏁 Quick Start
+
+### 1. Initialize your workspace
+
+Navigate to an empty directory and let the interactive wizard scaffold the workspace conventions:
+
+```bash
+dex init
+
+```
+
+### 2. Declare a dataset
+
+Add a specification block inside `conf/catalog/sample_dataset.yaml`:
+
+```yaml
+customers:
+  description: "Main production customer data"
+  format: "csv"
+  engine: "pandas"
+  storage_path: "data/sample_table.csv"
+
+```
+
+### 3. Load data anywhere
+
+Create a Python script or open a Jupyter Notebook inside `src/notebooks/` and fetch your data instantly:
+
+```python
+from data_engineering_exp.core.io import DataLoader
+
+# Autodiscovers your project root boundaries and settings
+loader = DataLoader()
+
+# Loads the dataset securely as a Pandas DataFrame
+df = loader.load("customers")
+df.head()
+
+```
+
+---
+
+## 📖 Complete Documentation
+
+For comprehensive guides, testing architecture deep-dives, and complete API references, visit our documentation site:
+👉 **[http://127.0.0.1:8000/](https://www.google.com/search?q=http://127.0.0.1:8000/)** *(Replace with your deployed docs URL, e.g., GitHub Pages)*
+
+---
+
+## ⚖️ License
+
+Distributed under the **MIT License**. Any modification or distribution (including forks) must include the original copyright notice and liability waiver. See `LICENSE` for more information.
+
+```

data_engineering_exp-0.1.0/README.md

@@ -0,0 +1,85 @@
+# Data Engineering Experience 🚀
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Python Version](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.14-blue)](https://www.python.org)
+
+**dex** (*Data Engineering Experience*) is a minimalist, agnostic Python framework designed to streamline and standardize data engineering pipelines. By embracing **Convention over Configuration**, `dex` eliminates environment friction, absolute path hardcoding, and complex PySpark session management.
+
+---
+
+## ✨ Key Features
+
+* **Zero-Config File Discovery:** Automatic tree-walking directory resolution anchors your data catalog using your local `pyproject.toml` file.
+* **Decentralized Catalog:** Declare your metadata layouts inside modular, self-contained mini-YAML files.
+* **Elastic Processing Runtimes:** Switch dynamically between **Pandas** and **PySpark** execution engines using exactly the same unified interface.
+* **Interactive CLI Scaffolding:** Spin up a new production-ready data directory structure instantly with `dex init`.
+
+---
+
+## 📦 Installation
+
+*(Once published to PyPI)*
+```bash
+pip install dex
+```
+Or install it directly from the source repository using Poetry:
+```bash
+poetry add git+[https://github.com/idperez720/data-engineering-exp.git](https://github.com/idperez720/data-engineering-exp.git)
+```
+
+---
+
+## 🏁 Quick Start
+
+### 1. Initialize your workspace
+
+Navigate to an empty directory and let the interactive wizard scaffold the workspace conventions:
+
+```bash
+dex init
+
+```
+
+### 2. Declare a dataset
+
+Add a specification block inside `conf/catalog/sample_dataset.yaml`:
+
+```yaml
+customers:
+  description: "Main production customer data"
+  format: "csv"
+  engine: "pandas"
+  storage_path: "data/sample_table.csv"
+
+```
+
+### 3. Load data anywhere
+
+Create a Python script or open a Jupyter Notebook inside `src/notebooks/` and fetch your data instantly:
+
+```python
+from data_engineering_exp.core.io import DataLoader
+
+# Autodiscovers your project root boundaries and settings
+loader = DataLoader()
+
+# Loads the dataset securely as a Pandas DataFrame
+df = loader.load("customers")
+df.head()
+
+```
+
+---
+
+## 📖 Complete Documentation
+
+For comprehensive guides, testing architecture deep-dives, and complete API references, visit our documentation site:
+👉 **[http://127.0.0.1:8000/](https://www.google.com/search?q=http://127.0.0.1:8000/)** *(Replace with your deployed docs URL, e.g., GitHub Pages)*
+
+---
+
+## ⚖️ License
+
+Distributed under the **MIT License**. Any modification or distribution (including forks) must include the original copyright notice and liability waiver. See `LICENSE` for more information.
+
+```

data_engineering_exp-0.1.0/data_engineering_exp/cli.py

@@ -0,0 +1,63 @@
+"""Command Line Interface (CLI) entry point for dex using Click."""
+
+import os
+
+import click
+
+from data_engineering_exp.core.initializer import ProjectInitializer
+
+
+@click.group()
+def entry_point() -> None:
+    """dex - Data Engineering Experience CLI utilities.
+
+    Args:
+
+    Returns:
+        None: Main CLI entry group.
+    """
+    pass
+
+
+@entry_point.command()
+@click.option(
+    "--path",
+    type=click.Path(file_okay=False, dir_okay=True),
+    default=".",
+    help="Root directory where scaffolding will be built. Defaults to '.'",
+)
+def init(path: str) -> None:
+    """Scaffold a new data engineering project structure interactively.
+
+    Args:
+        path(str): Target directory string path for project structure.
+
+    Returns:
+        None: Side-effect creating folders and configurations.
+    """
+    click.echo("🚀 Welcome to the dex project initialization wizard!\n")
+
+    # Derives default project name dynamically from the target directory name
+    default_name = os.path.basename(os.path.abspath(path))
+    if not default_name or default_name in (".", ""):
+        default_name = "my-dex-project"
+
+    # Sequential metadata prompts with default fallbacks
+    name = click.prompt("Project name", default=default_name, type=str)
+    version = click.prompt("Version", default="0.1.0", type=str)
+    description = click.prompt(
+        "Description",
+        default="Data engineering project scaffolded by dex",
+        type=str,
+    )
+    author = click.prompt("Author", default="Anonymous", type=str)
+
+    initializer = ProjectInitializer(base_path=path)
+    initializer.init_project(
+        name=name,
+        version=version,
+        description=description,
+        author=author,
+    )
+
+    click.echo(f"\n✨ Project successfully initialized at '{path}'!")

data_engineering_exp-0.1.0/data_engineering_exp/core/catalog.py

@@ -0,0 +1,197 @@
+"""This module implements decentralized data catalog engines for dex."""
+
+import os
+from typing import Any, Dict, List, Optional, cast
+
+import yaml
+
+
+class DataCatalog:
+    """Parses and consolidates declarative configurations from files or directories.
+
+    Scans catalog definitions recursively, allowing user architectures to be
+    split into mini-YAML files or kept within a single configuration layout.
+    """
+
+    def __init__(self, catalog_path: Optional[str] = None) -> None:
+        """Initializes the DataCatalog by scanning files or configuration paths.
+
+        If no path is provided, it automatically seeks upwards from the current
+        working directory to find the standard 'conf/catalog' directory based on
+        the pyproject.toml root anchor file.
+
+        Args:
+            catalog_path(Optional[str]): System directory path or file string
+                pointing to catalog definitions. Defaults to None.
+
+        Returns:
+            None: Initializes the class instance.
+        """
+        self.project_root: str = ""
+
+        if catalog_path is None:
+            catalog_path = self._discover_catalog_path()
+        else:
+            # Derive the root directory context if an explicit path is supplied
+            abs_path = os.path.abspath(catalog_path)
+            if os.path.isfile(abs_path):
+                self.project_root = os.path.dirname(
+                    os.path.dirname(os.path.dirname(abs_path))
+                )
+            else:
+                self.project_root = os.path.dirname(os.path.dirname(abs_path))
+
+        if not os.path.exists(catalog_path):
+            raise FileNotFoundError(f"Catalog source path not found at: {catalog_path}")
+
+        self._datasets: Dict[str, Dict[str, Any]] = {}
+        self._load_catalog_sources(catalog_path)
+
+    @property
+    def dataset_names(self) -> List[str]:
+        """Exposes the list of all dataset identifiers present in the catalog.
+
+        Args:
+
+        Returns:
+            List[str]: List of string names for all loaded datasets.
+        """
+        return list(self._datasets.keys())
+
+    def get_dataset_metadata(self, dataset_name: str) -> Dict[str, Any]:
+        """Retrieves core metadata mapping configurations for a target dataset.
+
+        Args:
+            dataset_name(str): Unique identifier string of the dataset.
+
+        Returns:
+            Dict[str, Any]: Dictionary containing properties like description,
+                format, primary_keys, and storage_path.
+
+        Raises:
+            KeyError: If the target dataset name is missing from the catalog.
+        """
+        if dataset_name not in self._datasets:
+            raise KeyError(f"Dataset '{dataset_name}' missing from catalog.")
+
+        meta = self._datasets[dataset_name].copy()
+        meta.pop("columns", None)
+        return meta
+
+    def get_column_names(self, dataset_name: str) -> List[str]:
+        """Extracts the expected list of column names for a target dataset.
+
+        Args:
+            dataset_name(str): Unique identifier string of the dataset.
+
+        Returns:
+            List[str]: Ordered list of strings representing expected columns.
+
+        Raises:
+            KeyError: If the target dataset name is missing from the catalog.
+        """
+        if dataset_name not in self._datasets:
+            raise KeyError(f"Dataset '{dataset_name}' missing from catalog.")
+
+        columns_spec = self._datasets[dataset_name].get("columns", [])
+        return [col["name"] for col in columns_spec]
+
+    def validate_schema_presence(self, df: Any, dataset_name: str) -> bool:
+        """Validates that all columns defined in the catalog exist in the dataframe.
+
+        Args:
+            df(Any): Active input Pandas or PySpark DataFrame object.
+            dataset_name(str): Catalog identifier string of the dataset target.
+
+        Returns:
+            bool: True if all expected columns are present, raises ValueError
+                otherwise.
+
+        Raises:
+            TypeError: If the input dataframe structure is not supported.
+            ValueError: If a column mismatch is discovered against the catalog.
+        """
+        expected_cols = set(self.get_column_names(dataset_name))
+
+        df_type = type(df).__name__
+        if df_type == "DataFrame" and "pandas" in type(df).__module__:
+            actual_cols = set(df.columns.tolist())
+        elif df_type == "DataFrame" and "pyspark" in type(df).__module__:
+            actual_cols = set(df.columns)
+        else:
+            raise TypeError("Unsupported DataFrame type for validation.")
+
+        missing_cols = expected_cols - actual_cols
+        if missing_cols:
+            raise ValueError(
+                f"Schema mismatch for '{dataset_name}'. "
+                f"Missing expected catalog columns: {list(missing_cols)}"
+            )
+
+        return True
+
+    def _discover_catalog_path(self) -> str:
+        """Traverses parent directories upwards to locate the pyproject.toml file.
+
+        Once the layout anchor file is found, it automatically appends the standard
+        'conf/catalog' structural workspace mapping.
+
+        Returns:
+            str: Absolute path to the discovered catalog directory.
+
+        Raises:
+            FileNotFoundError: If the pyproject.toml configuration cannot be found.
+        """
+        current_dir = os.getcwd()
+
+        while True:
+            potential_toml = os.path.join(current_dir, "pyproject.toml")
+            if os.path.exists(potential_toml):
+                self.project_root = current_dir
+                return os.path.join(current_dir, "conf", "catalog")
+
+            parent_dir = os.path.dirname(current_dir)
+            if parent_dir == current_dir:
+                break
+            current_dir = parent_dir
+
+        raise FileNotFoundError(
+            "Configuration file (pyproject.toml) could not be located. "
+            "Please run 'dex init' to establish a valid project layout."
+        )
+
+    def _load_catalog_sources(self, path: str) -> None:
+        """Internal worker to process and parse path targets recursively.
+
+        Args:
+            path(str): Targeting file path or folder configuration directory.
+
+        Returns:
+            None: Populates internal dictionary instances.
+        """
+        if os.path.isfile(path):
+            if path.endswith((".yml", ".yaml")):
+                self._parse_file(path)
+            return
+
+        for root, _, files in os.walk(path):
+            for file in files:
+                if file.endswith((".yml", ".yaml")):
+                    full_path = os.path.join(root, file)
+                    self._parse_file(full_path)
+
+    def _parse_file(self, file_path: str) -> None:
+        """Parses an individual YAML catalog file and updates target states.
+
+        Args:
+            file_path(str): Exact system string location pointing to file.
+
+        Returns:
+            None: Updates the core datasets dictionary data states.
+        """
+        with open(file_path, "r", encoding="utf-8") as stream:
+            content = yaml.safe_load(stream)
+
+        if content and isinstance(content, dict):
+            typed_content = cast(Dict[str, Dict[str, Any]], content)
+            self._datasets.update(typed_content)

data_engineering_exp-0.1.0/data_engineering_exp/core/deduplication.py

@@ -0,0 +1,192 @@
+"""This module implements versatile data deduplication utilities for dex."""
+
+from typing import Any, List, Optional, Union
+
+# Handle optional dependencies at the top of the file
+try:
+    import pyspark.sql.functions as F
+    from pyspark.sql import Window
+
+    HAS_SPARK = True
+except ImportError:
+    Window: Any = None
+    F: Any = None
+    HAS_SPARK = False
+
+
+class Deduplicator:
+    """Utility class providing multiple strategies for data deduplication.
+
+    Supports Pandas and PySpark DataFrames dynamically, allowing extraction of
+    first, latest, multi-sorted, consolidated, or strictly unique records.
+    """
+
+    def __init__(self):
+        """Initializes the Deduplicator instance.
+
+        Args:
+
+        Returns:
+            None: Initializes the class instance.
+        """
+        pass
+
+    def latest(self, df: Any, keys: List[str], order_by_col: str) -> Any:
+        """Extracts the most recent record per business key group.
+
+        Args:
+            df(Any): Input Pandas or PySpark DataFrame.
+            keys(List[str]): Columns that identify a business entity.
+            order_by_col(str): Column used to determine chronological order.
+
+        Returns:
+            Any: Deduplicated DataFrame containing only the latest events.
+        """
+        framework = self._detect_framework(df)
+        if framework == "pandas":
+            return (
+                df.sort_values(by=order_by_col, ascending=True)
+                .groupby(keys)
+                .last()
+                .reset_index()
+            )
+
+        window_spec = Window.partitionBy(keys).orderBy(F.col(order_by_col).desc())
+        return (
+            df.withColumn("_row_num", F.row_number().over(window_spec))
+            .filter(F.col("_row_num") == 1)
+            .drop("_row_num")
+        )
+
+    def first(self, df: Any, keys: List[str], order_by_col: str) -> Any:
+        """Extracts the earliest chronological record per business key group.
+
+        Args:
+            df(Any): Input Pandas or PySpark DataFrame.
+            keys(List[str]): Columns that identify a business entity.
+            order_by_col(str): Column used to determine chronological order.
+
+        Returns:
+            Any: Deduplicated DataFrame containing only the first events.
+        """
+        framework = self._detect_framework(df)
+        if framework == "pandas":
+            return (
+                df.sort_values(by=order_by_col, ascending=True)
+                .groupby(keys)
+                .first()
+                .reset_index()
+            )
+
+        window_spec = Window.partitionBy(keys).orderBy(F.col(order_by_col).asc())
+        return (
+            df.withColumn("_row_num", F.row_number().over(window_spec))
+            .filter(F.col("_row_num") == 1)
+            .drop("_row_num")
+        )
+
+    def distinct(self, df: Any, subset: Optional[List[str]] = None) -> Any:
+        """Removes strict duplicate rows from the DataFrame.
+
+        Args:
+            df(Any): Input Pandas or PySpark DataFrame.
+            subset(Optional[List[str]]): Specific columns to consider when
+                identifying duplicates. Defaults to None (all columns).
+
+        Returns:
+            Any: Clean DataFrame with unique records only.
+        """
+        framework = self._detect_framework(df)
+        if framework == "pandas":
+            return df.drop_duplicates(subset=subset).reset_index(drop=True)
+
+        return df.dropDuplicates(subset=subset)
+
+    def by_order(
+        self,
+        df: Any,
+        keys: List[str],
+        order_by_cols: List[str],
+        ascending: Union[bool, List[bool]] = True,
+    ) -> Any:
+        """Deduplicates rows by sorting through multiple custom column criteria.
+
+        Args:
+            df(Any): Input Pandas or PySpark DataFrame.
+            keys(List[str]): Columns that identify a business entity.
+            order_by_cols(List[str]): Ordered list of columns to sort by.
+            ascending(Union[bool, List[bool]]): Type of sorting direction for
+                each column. Defaults to True.
+
+        Returns:
+            Any: Deduplicated DataFrame containing the top-ranked records.
+        """
+        framework = self._detect_framework(df)
+        if framework == "pandas":
+            return (
+                df.sort_values(by=order_by_cols, ascending=ascending)
+                .groupby(keys)
+                .first()
+                .reset_index()
+            )
+
+        if isinstance(ascending, bool):
+            asc_list = [ascending] * len(order_by_cols)
+        else:
+            asc_list = ascending
+
+        order_exprs = []
+        for col, asc_dir in zip(order_by_cols, asc_list):
+            expr = F.col(col).asc() if asc_dir else F.col(col).desc()
+            order_exprs.append(expr)
+
+        window_spec = Window.partitionBy(keys).orderBy(*order_exprs)
+        return (
+            df.withColumn("_row_num", F.row_number().over(window_spec))
+            .filter(F.col("_row_num") == 1)
+            .drop("_row_num")
+        )
+
+    def combined(self, df: Any, keys: List[str]) -> Any:
+        """Stitches rows together by compacting nulls into a golden record.
+
+        Args:
+            df(Any): Input Pandas or PySpark DataFrame.
+            keys(List[str]): Columns that identify a business entity.
+
+        Returns:
+            Any: Consolidated DataFrame with filled attributes per group.
+        """
+        framework = self._detect_framework(df)
+        if framework == "pandas":
+            return df.groupby(keys).first().reset_index()
+
+        agg_cols = [
+            F.first(c, ignorenulls=True).alias(c) for c in df.columns if c not in keys
+        ]
+        return df.groupBy(keys).agg(*agg_cols)
+
+    def _detect_framework(self, df: Any) -> str:
+        """Internal helper to identify the dataframe framework type safely.
+
+        Args:
+            df(Any): Input dataframe object.
+
+        Returns:
+            str: String identifier ("pandas" or "spark").
+        """
+        df_type = type(df).__name__
+        module_type = type(df).__module__
+
+        if "pandas" in module_type and df_type == "DataFrame":
+            return "pandas"
+
+        if "pyspark" in module_type and df_type == "DataFrame":
+            if not HAS_SPARK:
+                raise ImportError("PySpark is required but could not be imported.")
+            return "spark"
+
+        raise TypeError(
+            f"Unsupported type: {type(df)}. "
+            "Only Pandas and PySpark DataFrames are supported."
+        )