datadock 0.1.2__tar.gz

datadock-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Otávio Oliveira
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datadock-0.1.2/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.1
+Name: datadock
+Version: 0.1.2
+Summary: Datadock is a PySpark-based data interoperability library. It automatically detects schemas from heterogeneous files (CSV, JSON, Parquet), groups them by structural similarity, and performs standardized batch reads. Designed for pipelines handling non-uniform large-scale data, enabling robust integration and reuse in distributed environments.
+Author: Otavio Oliveira
+Author-email: datadock.sup@gmail.com
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: loguru (>=0.7.3,<0.8.0)
+Requires-Dist: pyarrow (>=20.0.0,<21.0.0)
+Requires-Dist: pyspark (>=3.5.5,<4.0.0)
+Description-Content-Type: text/markdown
+
+# Datadock
+
+**Datadock** is a Python library built on top of PySpark, designed to simplify **data interoperability** between files of different formats and schemas in modern data engineering pipelines.
+
+It automatically detects schemas from CSV, JSON and Parquet files, groups structurally similar files, and allows standardized reading of all grouped files into a single Spark DataFrame — even in highly heterogeneous datasets.
+
+
+## ✨ Key Features
+
+- 🚀 **Automatic parsing** of multiple file formats: `.csv`, `.json`, `.parquet`
+- 🧠 **Schema-based file grouping** by structural similarity
+- 📊 **Auto-selection of dominant schemas**
+- 🛠️ **Unified read** across similar files into a single PySpark DataFrame
+- 🔍 **Schema insight** for diagnostics and inspection
+
+
+## 🔧 Installation
+
+```bash
+pip install datadock
+```
+
+
+## 🗂️ Expected Input Structure
+
+Place your data files (CSV, JSON or Parquet) inside a single folder. The library will automatically detect supported files and organize them by schema similarity.
+
+```bash
+/data/input/
+├── sales_2020.csv
+├── sales_2021.csv
+├── products.json
+├── archive.parquet
+├── log.parquet
+```
+
+
+## 🧪 Usage Example
+
+```python
+from datadock import scan_schema, get_schema_info, read_data
+
+path = "/path/to/your/data"
+
+# Logs schema groups detected
+scan_schema(path)
+
+# Retrieves schema metadata
+info = get_schema_info(path)
+print(info)
+
+# Loads all files from schema group 1
+df = read_data(path, schema_id=1, logs=True)
+df.show()
+```
+
+
+## 📌 Public API
+
+### `scan_schema`
+Logs the identified schema groups found in the specified folder.
+
+
+### `get_schema_info`
+Returns a list of dictionaries containing:
+- `schema_id`: ID of the schema group
+- `file_count`: number of files in the group
+- `column_count`: number of columns in the schema
+- `files`: list of file names in the group
+
+
+### `read_data`
+Reads and merges all files that share the same schema.  
+If `schema_id` is not specified, the group with the most columns will be selected.
+
+
+## ✅ Requirements
+
+- Python 3.10+
+- PySpark
+
+
+## 📚 Motivation
+
+In real-world data engineering workflows, it's common to deal with files that represent the same data domain but have slight structural variations — such as missing columns, different orders, or evolving schemas.  
+**Datadock** automates the process of grouping, inspecting, and reading these files reliably, allowing you to build pipelines that are schema-aware, scalable, and format-agnostic.
+
+
+## 📄 License
+
+This project is licensed under the **MIT License**.

datadock-0.1.2/README.md

@@ -0,0 +1,91 @@
+# Datadock
+
+**Datadock** is a Python library built on top of PySpark, designed to simplify **data interoperability** between files of different formats and schemas in modern data engineering pipelines.
+
+It automatically detects schemas from CSV, JSON and Parquet files, groups structurally similar files, and allows standardized reading of all grouped files into a single Spark DataFrame — even in highly heterogeneous datasets.
+
+
+## ✨ Key Features
+
+- 🚀 **Automatic parsing** of multiple file formats: `.csv`, `.json`, `.parquet`
+- 🧠 **Schema-based file grouping** by structural similarity
+- 📊 **Auto-selection of dominant schemas**
+- 🛠️ **Unified read** across similar files into a single PySpark DataFrame
+- 🔍 **Schema insight** for diagnostics and inspection
+
+
+## 🔧 Installation
+
+```bash
+pip install datadock
+```
+
+
+## 🗂️ Expected Input Structure
+
+Place your data files (CSV, JSON or Parquet) inside a single folder. The library will automatically detect supported files and organize them by schema similarity.
+
+```bash
+/data/input/
+├── sales_2020.csv
+├── sales_2021.csv
+├── products.json
+├── archive.parquet
+├── log.parquet
+```
+
+
+## 🧪 Usage Example
+
+```python
+from datadock import scan_schema, get_schema_info, read_data
+
+path = "/path/to/your/data"
+
+# Logs schema groups detected
+scan_schema(path)
+
+# Retrieves schema metadata
+info = get_schema_info(path)
+print(info)
+
+# Loads all files from schema group 1
+df = read_data(path, schema_id=1, logs=True)
+df.show()
+```
+
+
+## 📌 Public API
+
+### `scan_schema`
+Logs the identified schema groups found in the specified folder.
+
+
+### `get_schema_info`
+Returns a list of dictionaries containing:
+- `schema_id`: ID of the schema group
+- `file_count`: number of files in the group
+- `column_count`: number of columns in the schema
+- `files`: list of file names in the group
+
+
+### `read_data`
+Reads and merges all files that share the same schema.  
+If `schema_id` is not specified, the group with the most columns will be selected.
+
+
+## ✅ Requirements
+
+- Python 3.10+
+- PySpark
+
+
+## 📚 Motivation
+
+In real-world data engineering workflows, it's common to deal with files that represent the same data domain but have slight structural variations — such as missing columns, different orders, or evolving schemas.  
+**Datadock** automates the process of grouping, inspecting, and reading these files reliably, allowing you to build pipelines that are schema-aware, scalable, and format-agnostic.
+
+
+## 📄 License
+
+This project is licensed under the **MIT License**.

datadock-0.1.2/pyproject.toml

@@ -0,0 +1,24 @@
+[tool.poetry]
+name = "datadock"
+version = "0.1.2"
+description = "Datadock is a PySpark-based data interoperability library. It automatically detects schemas from heterogeneous files (CSV, JSON, Parquet), groups them by structural similarity, and performs standardized batch reads. Designed for pipelines handling non-uniform large-scale data, enabling robust integration and reuse in distributed environments."
+authors = ["Otavio Oliveira <datadock.sup@gmail.com>"]
+readme = "README.md"
+packages = [{include = "datadock", from = "src"}]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+pyspark = "^3.5.5"
+loguru = "^0.7.3"
+pyarrow = "^20.0.0"
+
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.3.5"
+black = "^25.1.0"
+mypy = "^1.15.0"
+flake8 = "^7.2.0"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

datadock-0.1.2/src/datadock/__init__.py

@@ -0,0 +1,3 @@
+from .api import scan_schema, read_data, get_schema_info
+
+__all__ = ["scan_schema", "read_data", "get_schema_info"]

datadock-0.1.2/src/datadock/_reader.py

@@ -0,0 +1,69 @@
+from pyspark.sql import SparkSession, DataFrame
+import csv
+import json
+from pathlib import Path
+from typing import Optional, List, Tuple
+import pyarrow.parquet as pq
+from datadock._utils import logger
+
+def _read_schema_only(path: str) -> Optional[List[Tuple[str, str]]]:
+    """
+    Reads only the schema (field names and types) from a file without loading data.
+    """
+    ext = Path(path).suffix.lower()
+
+    try:
+        if ext == ".csv":
+            with open(path, newline='', encoding='utf-8') as f:
+                reader = csv.DictReader(f)
+                columns = reader.fieldnames
+                if columns:
+                    return [(col, "string") for col in columns]
+
+        elif ext == ".json":
+            with open(path, encoding='utf-8') as f:
+                line = json.loads(f.readline())
+                if isinstance(line, dict):
+                    return [(key, "string") for key in line.keys()]
+
+        elif ext == ".parquet":
+            schema = pq.read_schema(path)
+            return [(col.name, str(col.type)) for col in schema]
+
+        else:
+            logger.warning(f"[WARNING] Unsupported file extension: {ext}")
+            return None
+
+    except Exception as e:
+        logger.error(f"Failed to read schema for {path}: {e}")
+        return None
+    
+
+def _load_file(spark: SparkSession, file: str) -> Optional[DataFrame]:
+    """
+    Loads a file in the appropriate format (CSV, JSON, Parquet, Avro, TXT).
+    
+    Args:
+        spark (SparkSession): The active Spark session.
+        file (str): The path to the file.
+
+    Returns:
+        Optional[DataFrame]: A Spark DataFrame if successful, None otherwise.
+    """
+    ext = Path(file).suffix.lower()
+
+    try:
+        if ext == ".csv":
+            return spark.read.option("header", True).csv(file)
+        elif ext == ".json":
+            return spark.read.option("multiline", True).json(file)
+        elif ext == ".parquet":
+            return spark.read.parquet(file)
+        elif ext == ".txt":
+            return spark.read.text(file)
+        else:
+            logger.warning(f"Unsupported file format: {ext}")
+            return None
+    except Exception as e:
+        logger.error(f"Error loading file {file}: {e}")
+        return None

datadock-0.1.2/src/datadock/_schema_manager.py

@@ -0,0 +1,115 @@
+from pyspark.sql import SparkSession, DataFrame
+from pyspark.sql.types import StructType
+from datadock._reader import _read_schema_only
+from datadock._utils import logger
+from typing import List, Dict, Optional, Tuple
+from collections import defaultdict
+from pathlib import Path
+
+
+def _extract_schema_signature(schema: List[Tuple[str, str]]) -> Tuple[str, ...]:
+    """
+    Create a normalized schema signature: tuple of (column_name, type), sorted by column name.
+    """
+    return tuple(sorted((col, dtype) for col, dtype in schema))
+
+def _structtype_to_list(schema: StructType) -> List[Tuple[str, str]]:
+    """
+    Convert a Spark StructType schema to a list of (column_name, data_type) tuples.
+    """
+    return [(field.name, field.dataType.simpleString()) for field in schema]
+
+
+def _group_by_schema(paths: List[str], min_similarity: float = 0.8) -> Dict[int, List[Tuple[str, List[Tuple[str, str]]]]]:
+    """
+    Groups files based on schema similarity using only column names.
+    Files are assigned to the first group where the similarity exceeds the given threshold.
+    If no group is similar enough, a new group is created.
+
+    Args:
+        paths (List[str]): 
+            A list of file paths to be analyzed.
+        min_similarity (float, optional): 
+            Minimum similarity ratio (0–1) required to group files together. 
+            Defaults to 0.8.
+
+    Returns:
+        Dict[int, List[Tuple[str, List[Tuple[str, str]]]]]: 
+            A dictionary mapping each schema ID to a list of tuples, 
+            each containing a file path and its inferred schema.
+    """
+    schema_groups = {}
+    schema_signatures = {}
+    next_id = 1
+
+    for path in paths:
+        schema = _read_schema_only(path)
+        if schema is None:
+            logger.warning(f"Could not read schema for file: {path}")
+            continue
+
+        matched = False
+        for schema_id, ref_schema in schema_signatures.items():
+            sim = _schema_similarity(schema, ref_schema)
+            if sim >= min_similarity:
+                schema_groups[schema_id].append((path, schema))
+                matched = True
+                break
+
+        if not matched:
+            schema_signatures[next_id] = schema
+            schema_groups[next_id] = [(path, schema)]
+            next_id += 1
+
+    return schema_groups
+
+
+def _read_schema_group(
+    grouped_by_id: Dict[int, List[Tuple[str, List[Tuple[str, str]]]]],
+    schema_id: Optional[int] = None
+) -> Optional[List[str]]:
+    """
+    Returns the file paths that share the same schema.
+    If schema_id is not provided, returns the group with the most columns (rank 1).
+    
+    Args:
+        grouped_by_id (Dict[int, List[Tuple[str, List[Tuple[str, str]]]]]): 
+            A dictionary where the key is the schema ID and the value is a list of tuples 
+            containing file paths and their associated schema.
+        schema_id (Optional[int]): 
+            The schema ID to retrieve. If not specified, defaults to the group with the most columns.
+
+    Returns:
+        Optional[List[str]]: 
+            A list of file paths that share the specified schema. 
+            Returns None if the group is not found or if there are no datasets.
+    """
+    if not grouped_by_id:
+        logger.error("No datasets available to read.")
+        return None
+
+    selected_id = schema_id or sorted(grouped_by_id.keys())[0]
+    logger.info(f"Selected dataset from schema group {selected_id}.")
+
+    return [path for path, _ in grouped_by_id[selected_id]]
+
+def _schema_similarity(a: List[Tuple[str, str]], b: List[Tuple[str, str]]) -> float:
+    """
+    Calculates the similarity between two schemas based on column names only,
+    ignoring data types. Uses Jaccard similarity.
+
+    Args:
+        a (List[Tuple[str, str]]): 
+            Schema A as a list of (column name, data type) tuples.
+        b (List[Tuple[str, str]]): 
+            Schema B as a list of (column name, data type) tuples.
+
+    Returns:
+        float: 
+            A value between 0.0 and 1.0 representing the schema similarity.
+    """
+    set_a = set(col_name for col_name, _ in a)
+    set_b = set(col_name for col_name, _ in b)
+    intersection = set_a & set_b
+    union = set_a | set_b
+    return len(intersection) / len(union) if union else 0

datadock-0.1.2/src/datadock/_utils.py

@@ -0,0 +1,6 @@
+from loguru import logger
+import sys
+
+logger.remove()
+logger.add(sys.stderr, level="INFO", colorize=True,
+           format="<green>{time:HH:mm:ss}</green> | <level>{level}</level> | <cyan>{message}</cyan>")

datadock-0.1.2/src/datadock/api.py

@@ -0,0 +1,136 @@
+from pyspark.sql import SparkSession, DataFrame
+from datadock._schema_manager import _group_by_schema, _read_schema_group
+from datadock._reader import _load_file
+from datadock._utils import logger
+from pathlib import Path
+from typing import Optional, List, Dict, Any
+
+SUPPORTED_EXTENSIONS = {".csv", ".json", ".parquet"}
+
+def scan_schema(path: str):
+    """
+    Scans and prints schema groupings for all supported files in the specified path.
+    This is the public entry point for schema inspection.
+    """
+    data_dir = Path(path)
+    paths = [str(p) for p in data_dir.glob("*") if p.suffix.lower() in SUPPORTED_EXTENSIONS]
+
+    if not paths:
+        logger.warning("No supported data files found.")
+        return
+
+    logger.info(f"Found {len(paths)} files to process.")
+    grouped = _group_by_schema(paths)
+
+    if len(grouped) > 1:
+        logger.info(f"Found {len(grouped)} different schemas.")
+    else:
+        logger.info("Found 1 schema.")
+
+    for sid, group in grouped.items():
+        col_count = len(group[0][1])
+        logger.info(f"Schema {sid}: {col_count} columns – {len(group)} file(s):")
+        for file_path, _ in group:
+            logger.info(f"  • {file_path}")
+
+def read_data(path: str, schema_id: Optional[int] = None, logs: bool = False) -> Optional[DataFrame]:
+    """
+    Reads and merges all files that belong to a schema group.
+
+    :param path: Folder containing data files
+    :param schema_id: ID of the schema group to read (defaults to most complex)
+    :param logs: Whether to print detailed logs during loading
+    :return: Spark DataFrame or None if load failed
+    """
+    spark = SparkSession.builder.appName("Databridge").getOrCreate()
+
+    data_dir = Path(path)
+    paths = [str(p) for p in data_dir.glob("*") if p.suffix.lower() in SUPPORTED_EXTENSIONS]
+    if not paths:
+        if logs:
+            logger.warning("No supported data files found.")
+        return None
+
+    if logs:
+        logger.info(f"Found {len(paths)} files to process.")
+    elif logs == False:
+        logger.info(f"For more details about the reading process, set `logs=True`.")
+    grouped = _group_by_schema(paths)
+
+    if len(grouped) > 1 and logs:
+        logger.warning(f"Multiple schemas found in path '{path}'. Total: {len(grouped)}")
+
+    if schema_id is None:
+        if logs:
+            logger.warning("Schema ID not specified. Defaulting to schema 1.")
+        schema_id = 1
+
+    if schema_id not in grouped:
+        logger.error(f"Schema ID {schema_id} not found among detected schema groups. Available IDs: {list(grouped.keys())}")
+        return None
+
+    selected_files = _read_schema_group(grouped, schema_id=schema_id)
+    if not selected_files:
+        if logs:
+            logger.warning(f"No dataset found for schema {schema_id}.")
+        return None
+
+    if logs:
+        logger.info(f"Reading data from schema group {schema_id}")
+
+    dfs = []
+    for file in selected_files:
+        df = _load_file(spark, file)
+        if df:
+            dfs.append(df)
+            if logs:
+                logger.info(f"Loaded file: {file}")
+
+    if not dfs:
+        if logs:
+            logger.warning("No DataFrames were loaded.")
+        return None
+
+    final_df = dfs[0]
+    for df in dfs[1:]:
+        try:
+            final_df = final_df.unionByName(df)
+        except Exception as e:
+            if logs:
+                logger.error(f"Error merging DataFrames: {e}")
+
+    if logs:
+        logger.info("Dataset successfully loaded.")
+
+    return final_df
+
+
+def get_schema_info(path: str) -> List[Dict[str, Any]]:
+    """
+    Returns detailed information about schema groups detected in the given directory.
+
+    :param path: Path to the folder containing raw data files.
+    :return: A list of dictionaries with schema_id, file count, column count, and list of files.
+    """
+    data_dir = Path(path)
+    paths = [str(p) for p in data_dir.glob("*") if p.suffix.lower() in SUPPORTED_EXTENSIONS]
+
+    if not paths:
+        logger.warning("No supported data files found in the provided directory.")
+        return []
+
+    grouped = _group_by_schema(paths)
+    schema_info = []
+
+    for schema_id, group in grouped.items():
+        file_list = [Path(file_path).name for file_path, _ in group]
+        column_count = len(group[0][1]) if group else 0
+
+        schema_info.append({
+            "schema_id": schema_id,
+            "file_count": len(file_list),
+            "column_count": column_count,
+            "files": file_list
+        })
+
+    return schema_info