etlplus 0.11.12__tar.gz → 0.12.9__tar.gz

{etlplus-0.11.12/etlplus.egg-info → etlplus-0.12.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: etlplus
-Version: 0.11.12
+Version: 0.12.9
 Summary: A Swiss Army knife for simple ETL operations
 Home-page: https://github.com/Dagitali/ETLPlus
 Author: ETLPlus Team
@@ -17,8 +17,11 @@ Classifier: Programming Language :: Python :: 3.14
 Requires-Python: >=3.13,<3.15
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: fastavro>=1.12.1
 Requires-Dist: jinja2>=3.1.6
+Requires-Dist: openpyxl>=3.1.5
 Requires-Dist: pyodbc>=5.3.0
+Requires-Dist: pyarrow>=22.0.0
 Requires-Dist: python-dotenv>=1.2.1
 Requires-Dist: pandas>=2.3.3
 Requires-Dist: pydantic>=2.12.5
@@ -26,6 +29,8 @@ Requires-Dist: PyYAML>=6.0.3
 Requires-Dist: requests>=2.32.5
 Requires-Dist: SQLAlchemy>=2.0.45
 Requires-Dist: typer>=0.21.0
+Requires-Dist: xlrd>=2.0.2
+Requires-Dist: xlwt>=1.3.0
 Provides-Extra: dev
 Requires-Dist: black>=25.9.0; extra == "dev"
 Requires-Dist: build>=1.2.2; extra == "dev"
@@ -63,6 +68,17 @@ package and command-line interface for data extraction, validation, transformati
   - [Features](#features)
   - [Installation](#installation)
   - [Quickstart](#quickstart)
+  - [Data Connectors](#data-connectors)
+    - [REST APIs (`api`)](#rest-apis-api)
+    - [Databases (`database`)](#databases-database)
+    - [Files (`file`)](#files-file)
+      - [Stubbed / Placeholder](#stubbed--placeholder)
+      - [Tabular \& Delimited Text](#tabular--delimited-text)
+      - [Semi-Structured Text](#semi-structured-text)
+      - [Columnar / Analytics-Friendly](#columnar--analytics-friendly)
+      - [Binary Serialization and Interchange](#binary-serialization-and-interchange)
+      - [Spreadsheets](#spreadsheets)
+      - [Data Archives](#data-archives)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
       - [Argument Order and Required Options](#argument-order-and-required-options)
@@ -186,6 +202,83 @@ assert validate(filtered, rules)["valid"]
 load(filtered, "file", "temp/sample_output.json", file_format="json")
 ```
 
+## Data Connectors
+
+Data connectors abstract sources from which to extract data and targets to which to load data. They
+are differentiated by their types, each of which is represented in the subsections below.
+
+### REST APIs (`api`)
+
+ETLPlus can extract from REST APIs and load results via common HTTP methods. Supported operations
+include GET for extract and PATCH/POST/PUT for load.
+
+### Databases (`database`)
+
+Database connectors use connection strings for extraction and loading, and
+DDL can be rendered from table specs for migrations or schema checks.
+
+### Files (`file`)
+
+File formats are grouped as in `FileFormat`. Support is marked as:
+
+- **Y**: implemented (may require optional dependencies)
+- **N**: stubbed or not yet implemented
+
+#### Stubbed / Placeholder
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `stub` | N | Placeholder format for tests and future connectors. |
+
+#### Tabular & Delimited Text
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `csv` | Y | Comma-Separated Values |
+| `fwf` | N | Fixed-Width Fields |
+| `dat` | N | Generic data file, often delimited or fixed-width |
+| `psv` | N | Pipe-Separated Values |
+| `tab` | N | Often synonymous with TSV |
+| `tsv` | Y | Tab-Separated Values |
+| `txt` | Y | Plain text, often delimited or fixed-width |
+
+#### Semi-Structured Text
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `json` | Y | JavaScript Object Notation |
+| `ndjson` | Y | Newline-Delimited JSON |
+| `xml` | Y | Extensible Markup Language |
+| `yaml` | Y | YAML Ain't Markup Language |
+
+#### Columnar / Analytics-Friendly
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `feather` | Y | Apache Arrow Feather |
+| `orc` | Y | Optimized Row Columnar; common in Hadoop |
+| `parquet` | Y | Apache Parquet; common in Big Data |
+
+#### Binary Serialization and Interchange
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `avro` | Y | Apache Avro |
+
+#### Spreadsheets
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `xls` | Y | Microsoft Excel (BIFF); read-only |
+| `xlsx` | Y | Microsoft Excel (Open XML) |
+
+#### Data Archives
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `gz` | Y | Gzip-compressed file |
+| `zip` | Y | ZIP archive |
+
 ## Usage
 
 ### Command Line Interface

{etlplus-0.11.12 → etlplus-0.12.9}/README.md

@@ -18,6 +18,17 @@ package and command-line interface for data extraction, validation, transformati
   - [Features](#features)
   - [Installation](#installation)
   - [Quickstart](#quickstart)
+  - [Data Connectors](#data-connectors)
+    - [REST APIs (`api`)](#rest-apis-api)
+    - [Databases (`database`)](#databases-database)
+    - [Files (`file`)](#files-file)
+      - [Stubbed / Placeholder](#stubbed--placeholder)
+      - [Tabular \& Delimited Text](#tabular--delimited-text)
+      - [Semi-Structured Text](#semi-structured-text)
+      - [Columnar / Analytics-Friendly](#columnar--analytics-friendly)
+      - [Binary Serialization and Interchange](#binary-serialization-and-interchange)
+      - [Spreadsheets](#spreadsheets)
+      - [Data Archives](#data-archives)
   - [Usage](#usage)
     - [Command Line Interface](#command-line-interface)
       - [Argument Order and Required Options](#argument-order-and-required-options)
@@ -141,6 +152,83 @@ assert validate(filtered, rules)["valid"]
 load(filtered, "file", "temp/sample_output.json", file_format="json")
 ```
 
+## Data Connectors
+
+Data connectors abstract sources from which to extract data and targets to which to load data. They
+are differentiated by their types, each of which is represented in the subsections below.
+
+### REST APIs (`api`)
+
+ETLPlus can extract from REST APIs and load results via common HTTP methods. Supported operations
+include GET for extract and PATCH/POST/PUT for load.
+
+### Databases (`database`)
+
+Database connectors use connection strings for extraction and loading, and
+DDL can be rendered from table specs for migrations or schema checks.
+
+### Files (`file`)
+
+File formats are grouped as in `FileFormat`. Support is marked as:
+
+- **Y**: implemented (may require optional dependencies)
+- **N**: stubbed or not yet implemented
+
+#### Stubbed / Placeholder
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `stub` | N | Placeholder format for tests and future connectors. |
+
+#### Tabular & Delimited Text
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `csv` | Y | Comma-Separated Values |
+| `fwf` | N | Fixed-Width Fields |
+| `dat` | N | Generic data file, often delimited or fixed-width |
+| `psv` | N | Pipe-Separated Values |
+| `tab` | N | Often synonymous with TSV |
+| `tsv` | Y | Tab-Separated Values |
+| `txt` | Y | Plain text, often delimited or fixed-width |
+
+#### Semi-Structured Text
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `json` | Y | JavaScript Object Notation |
+| `ndjson` | Y | Newline-Delimited JSON |
+| `xml` | Y | Extensible Markup Language |
+| `yaml` | Y | YAML Ain't Markup Language |
+
+#### Columnar / Analytics-Friendly
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `feather` | Y | Apache Arrow Feather |
+| `orc` | Y | Optimized Row Columnar; common in Hadoop |
+| `parquet` | Y | Apache Parquet; common in Big Data |
+
+#### Binary Serialization and Interchange
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `avro` | Y | Apache Avro |
+
+#### Spreadsheets
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `xls` | Y | Microsoft Excel (BIFF); read-only |
+| `xlsx` | Y | Microsoft Excel (Open XML) |
+
+#### Data Archives
+
+| Format | Supported | Description |
+| --- | --- | --- |
+| `gz` | Y | Gzip-compressed file |
+| `zip` | Y | ZIP archive |
+
 ## Usage
 
 ### Command Line Interface

etlplus-0.12.9/etlplus/file/_imports.py

@@ -0,0 +1,141 @@
+"""
+:mod:`etlplus.file._imports` module.
+
+Shared helpers for optional dependency imports.
+"""
+
+from __future__ import annotations
+
+from importlib import import_module
+from typing import Any
+
+# SECTION: INTERNAL CONSTANTS =============================================== #
+
+
+_MODULE_CACHE: dict[str, Any] = {}
+
+
+# SECTION: INTERNAL FUNCTIONS =============================================== #
+
+
+def _error_message(
+    module_name: str,
+    format_name: str,
+) -> str:
+    """
+    Build an import error message for an optional dependency.
+
+    Parameters
+    ----------
+    module_name : str
+        Module name to look up.
+    format_name : str
+        Human-readable format name for templated messages.
+
+    Returns
+    -------
+    str
+        Formatted error message.
+    """
+    return (
+        f'{format_name} support requires '
+        f'optional dependency "{module_name}".\n'
+        f'Install with: pip install {module_name}'
+    )
+
+
+# SECTION: FUNCTIONS ======================================================== #
+
+
+def get_optional_module(
+    module_name: str,
+    *,
+    error_message: str,
+) -> Any:
+    """
+    Return an optional dependency module, caching on first import.
+
+    Parameters
+    ----------
+    module_name : str
+        Name of the module to import.
+    error_message : str
+        Error message to surface when the module is missing.
+
+    Returns
+    -------
+    Any
+        The imported module.
+
+    Raises
+    ------
+    ImportError
+        If the optional dependency is missing.
+    """
+    cached = _MODULE_CACHE.get(module_name)
+    if cached is not None:  # pragma: no cover - tiny branch
+        return cached
+    try:
+        module = import_module(module_name)
+    except ImportError as e:  # pragma: no cover
+        raise ImportError(error_message) from e
+    _MODULE_CACHE[module_name] = module
+    return module
+
+
+def get_fastavro() -> Any:
+    """
+    Return the fastavro module, importing it on first use.
+
+    Raises an informative ImportError if the optional dependency is missing.
+
+    Notes
+    -----
+    Prefer :func:`get_optional_module` for new call sites.
+    """
+    return get_optional_module(
+        'fastavro',
+        error_message=_error_message('fastavro', format_name='AVRO'),
+    )
+
+
+def get_pandas(
+    format_name: str,
+) -> Any:
+    """
+    Return the pandas module, importing it on first use.
+
+    Parameters
+    ----------
+    format_name : str
+        Human-readable format name for error messages.
+
+    Returns
+    -------
+    Any
+        The pandas module.
+
+    Notes
+    -----
+    Prefer :func:`get_optional_module` for new call sites.
+    """
+    return get_optional_module(
+        'pandas',
+        error_message=_error_message('pandas', format_name=format_name),
+    )
+
+
+def get_yaml() -> Any:
+    """
+    Return the PyYAML module, importing it on first use.
+
+    Raises an informative ImportError if the optional dependency is missing.
+
+    Notes
+    -----
+    Prefer :func:`get_optional_module` for new call sites.
+    """
+    return get_optional_module(
+        'yaml',
+        error_message=_error_message('PyYAML', format_name='YAML'),
+    )

etlplus-0.12.9/etlplus/file/_io.py

@@ -0,0 +1,121 @@
+"""
+:mod:`etlplus.file._io` module.
+
+Shared helpers for record normalization and delimited text formats.
+"""
+
+from __future__ import annotations
+
+import csv
+from pathlib import Path
+from typing import cast
+
+from ..types import JSONData
+from ..types import JSONDict
+from ..types import JSONList
+
+# SECTION: FUNCTIONS ======================================================== #
+
+
+def normalize_records(
+    data: JSONData,
+    format_name: str,
+) -> JSONList:
+    """
+    Normalize payloads into a list of dictionaries.
+
+    Parameters
+    ----------
+    data : JSONData
+        Input payload to normalize.
+    format_name : str
+        Human-readable format name for error messages.
+
+    Returns
+    -------
+    JSONList
+        Normalized list of dictionaries.
+
+    Raises
+    ------
+    TypeError
+        If a list payload contains non-dict items.
+    """
+    if isinstance(data, list):
+        if not all(isinstance(item, dict) for item in data):
+            raise TypeError(
+                f'{format_name} payloads must contain only objects (dicts)',
+            )
+        return cast(JSONList, data)
+    return [cast(JSONDict, data)]
+
+
+def read_delimited(path: Path, *, delimiter: str) -> JSONList:
+    """
+    Read delimited content from ``path``.
+
+    Parameters
+    ----------
+    path : Path
+        Path to the delimited file on disk.
+    delimiter : str
+        Delimiter character for parsing.
+
+    Returns
+    -------
+    JSONList
+        The list of dictionaries read from the delimited file.
+    """
+    with path.open('r', encoding='utf-8', newline='') as handle:
+        reader: csv.DictReader[str] = csv.DictReader(
+            handle,
+            delimiter=delimiter,
+        )
+        rows: JSONList = []
+        for row in reader:
+            if not any(row.values()):
+                continue
+            rows.append(cast(JSONDict, dict(row)))
+    return rows
+
+
+def write_delimited(path: Path, data: JSONData, *, delimiter: str) -> int:
+    """
+    Write ``data`` to a delimited file and return record count.
+
+    Parameters
+    ----------
+    path : Path
+        Path to the delimited file on disk.
+    data : JSONData
+        Data to write as delimited rows.
+    delimiter : str
+        Delimiter character for writing.
+
+    Returns
+    -------
+    int
+        The number of rows written.
+    """
+    rows: list[JSONDict]
+    if isinstance(data, list):
+        rows = [row for row in data if isinstance(row, dict)]
+    else:
+        rows = [data]
+
+    if not rows:
+        return 0
+
+    fieldnames = sorted({key for row in rows for key in row})
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open('w', encoding='utf-8', newline='') as handle:
+        writer = csv.DictWriter(
+            handle,
+            fieldnames=fieldnames,
+            delimiter=delimiter,
+        )
+        writer.writeheader()
+        for row in rows:
+            writer.writerow({field: row.get(field) for field in fieldnames})
+
+    return len(rows)

etlplus-0.12.9/etlplus/file/avro.py

@@ -0,0 +1,164 @@
+"""
+:mod:`etlplus.file.avro` module.
+
+Helpers for reading/writing Avro files.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+from typing import cast
+
+from etlplus.file._imports import get_fastavro
+
+from ..types import JSONData
+from ..types import JSONDict
+from ..types import JSONList
+from ._io import normalize_records
+
+# SECTION: EXPORTS ========================================================== #
+
+
+__all__ = [
+    'read',
+    'write',
+]
+
+
+# SECTION: INTERNAL CONSTANTS =============================================== #
+
+
+_PRIMITIVE_TYPES: tuple[type, ...] = (
+    bool,
+    int,
+    float,
+    str,
+    bytes,
+    bytearray,
+)
+
+
+# SECTION: INTERNAL FUNCTIONS =============================================== #
+
+
+def _infer_schema(records: JSONList) -> dict[str, Any]:
+    """
+    Infer a basic Avro schema from record payloads.
+
+    Only primitive field values are supported; complex values raise TypeError.
+    """
+    field_names = sorted({key for record in records for key in record})
+    fields: list[dict[str, Any]] = []
+    for name in field_names:
+        types: list[str] = []
+        for record in records:
+            value = record.get(name)
+            if value is None:
+                types.append('null')
+                continue
+            if isinstance(value, dict | list):
+                raise TypeError(
+                    'AVRO payloads must contain only primitive values',
+                )
+            if not isinstance(value, _PRIMITIVE_TYPES):
+                raise TypeError(
+                    'AVRO payloads must contain only primitive values',
+                )
+            types.append(cast(str, _infer_value_type(value)))
+        fields.append({'name': name, 'type': _merge_types(types)})
+
+    return {
+        'name': 'etlplus_record',
+        'type': 'record',
+        'fields': fields,
+    }
+
+
+def _infer_value_type(value: object) -> str | list[str]:
+    """
+    Infer the Avro type for a primitive value.
+
+    Raises TypeError for unsupported types.
+    """
+    if value is None:
+        return 'null'
+    if isinstance(value, bool):
+        return 'boolean'
+    if isinstance(value, int):
+        return 'long'
+    if isinstance(value, float):
+        return 'double'
+    if isinstance(value, str):
+        return 'string'
+    if isinstance(value, (bytes, bytearray)):
+        return 'bytes'
+    raise TypeError('AVRO payloads must contain only primitive values')
+
+
+def _merge_types(types: list[str]) -> str | list[str]:
+    """Return a stable Avro type union for a list of types."""
+    unique = list(dict.fromkeys(types))
+    if len(unique) == 1:
+        return unique[0]
+    ordered = ['null'] + sorted(t for t in unique if t != 'null')
+    return ordered
+
+
+# SECTION: FUNCTIONS ======================================================== #
+
+
+def read(
+    path: Path,
+) -> JSONList:
+    """
+    Read AVRO content from ``path``.
+
+    Parameters
+    ----------
+    path : Path
+        Path to the AVRO file on disk.
+
+    Returns
+    -------
+    JSONList
+        The list of dictionaries read from the AVRO file.
+    """
+    fastavro = get_fastavro()
+    with path.open('rb') as handle:
+        reader = fastavro.reader(handle)
+        return [cast(JSONDict, record) for record in reader]
+
+
+def write(
+    path: Path,
+    data: JSONData,
+) -> int:
+    """
+    Write ``data`` to AVRO at ``path`` and return record count.
+
+    Parameters
+    ----------
+    path : Path
+        Path to the AVRO file on disk.
+    data : JSONData
+        Data to write.
+
+    Returns
+    -------
+    int
+        Number of records written.
+    """
+    records = normalize_records(data, 'AVRO')
+    if not records:
+        return 0
+
+    fastavro = get_fastavro()
+    schema = _infer_schema(records)
+    parsed_schema = fastavro.parse_schema(schema)
+
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open('wb') as handle:
+        fastavro.writer(handle, parsed_schema, records)
+
+    return len(records)