etlplus 0.11.11__py3-none-any.whl → 0.12.1__py3-none-any.whl

etlplus/README.md

@@ -0,0 +1,37 @@
+# etlplus package
+
+The `etlplus` package provides a unified Python API and CLI for ETL operations: extraction,
+validation, transformation, and loading of data from files, APIs, and databases.
+
+- Top-level entry points for extract, validate, transform, and load
+- Utilities for pipeline orchestration and helpers
+- Exposes all subpackages for advanced usage
+
+Back to project overview: see the top-level [README](../README.md).
+
+## Subpackages
+
+- [etlplus.api](api/README.md): Lightweight HTTP client and paginated REST helpers
+- [etlplus.file](file/README.md): Unified file format support and helpers
+- [etlplus.config](config/README.md): Configuration helpers for connectors, pipelines, jobs, and
+  profiles
+- [etlplus.cli](cli/README.md): Command-line interface for ETLPlus workflows
+- [etlplus.database](database/README.md): Database engine, schema, and ORM helpers
+- [etlplus.templates](templates/README.md): SQL and DDL template helpers
+- [etlplus.validation](validation/README.md): Data validation utilities and helpers
+
+## Quickstart
+
+```python
+from etlplus import extract, validate, transform, load
+
+data = extract("file", "input.csv")
+filtered = transform(data, {"filter": {"field": "age", "op": "gt", "value": 25}})
+assert validate(filtered, {"age": {"type": "number", "min": 0}})["valid"]
+load(filtered, "file", "output.json", file_format="json")
+```
+
+## See Also
+
+- [Top-level project README](../README.md)
+- [API reference](../docs/README.md)

etlplus/api/README.md

@@ -1,7 +1,7 @@
-# etlplus.api module.
+# etlplus.api subpackage
 
-Focused documentation for the `etlplus.api` subpackage: a lightweight HTTP client and helpers for
-paginated REST endpoints.
+Documentation for the `etlplus.api` subpackage: a lightweight HTTP client and helpers for paginated
+REST endpoints.
 
 - Provides a small `EndpointClient` for calling JSON APIs
 - Supports page-, offset-, and cursor-based pagination via `PaginationConfig`
@@ -12,6 +12,20 @@ paginated REST endpoints.
 
 Back to project overview: see the top-level [README](../../README.md).
 
+- [etlplus.api subpackage](#etlplusapi-subpackage)
+  - [Installation](#installation)
+  - [Quickstart](#quickstart)
+    - [Overriding Rate Limits Per Call](#overriding-rate-limits-per-call)
+  - [Choosing `records_path` and `cursor_path`](#choosing-records_path-and-cursor_path)
+  - [Cursor-Based Pagination Example](#cursor-based-pagination-example)
+  - [Offset-based pagination example](#offset-based-pagination-example)
+  - [Authentication](#authentication)
+  - [Errors and Rate Limiting](#errors-and-rate-limiting)
+  - [Types and Transport](#types-and-transport)
+  - [Supporting Modules](#supporting-modules)
+  - [Minimal Contract](#minimal-contract)
+  - [See also](#see-also)
+
 ## Installation
 
 `etlplus.api` ships as part of the `etlplus` package. Install the package as usual:
@@ -233,3 +247,6 @@ providers can fall back to their own defaults. If you already possess a static t
 ## See also
 
 - Top-level CLI and library usage in the main [README](../../README.md)
+
+
+[def]: #installation

etlplus/cli/README.md

@@ -0,0 +1,40 @@
+# etlplus.cli subpackage
+
+Documentation for the `etlplus.cli` subpackage: command-line interface for ETLPlus workflows.
+
+- Provides a CLI for running ETL pipelines, jobs, and utilities
+- Supports commands for running, validating, and inspecting pipelines
+- Includes options for configuration, state, and output control
+- Exposes handlers for custom command integration
+
+Back to project overview: see the top-level [README](../../README.md).
+
+- [etlplus.cli subpackage](#etlpluscli-subpackage)
+  - [Available Commands](#available-commands)
+  - [Command Options](#command-options)
+  - [Example: Running a Pipeline](#example-running-a-pipeline)
+  - [See Also](#see-also)
+
+## Available Commands
+
+- **run**: Execute a pipeline or job
+- **validate**: Validate pipeline or config files
+- **inspect**: Show pipeline/job details
+
+## Command Options
+
+- `--config`: Path to config file
+- `--state`: Path to state file
+- `--output`: Output file or format
+
+## Example: Running a Pipeline
+
+```bash
+etlplus run --config configs/pipeline.yml --output results.json
+```
+
+## See Also
+
+- Top-level CLI and library usage in the main [README](../../README.md)
+- Command handlers in [handlers.py](handlers.py)
+- Command options in [options.py](options.py)

etlplus/config/README.md

@@ -0,0 +1,52 @@
+# etlplus.config subpackage
+
+Documentation for the `etlplus.config` subpackage: configuration helpers for connectors, pipelines,
+jobs, and profiles.
+
+- Provides classes and utilities for managing ETL pipeline configuration
+- Supports YAML/JSON config loading and validation
+- Includes helpers for connectors, jobs, pipelines, and profiles
+- Exposes type definitions for config schemas
+
+Back to project overview: see the top-level [README](../../README.md).
+
+- [etlplus.config subpackage](#etlplusconfig-subpackage)
+  - [Supported Configuration Types](#supported-configuration-types)
+  - [Loading and Validating Configs](#loading-and-validating-configs)
+  - [Example: Loading a Pipeline Config](#example-loading-a-pipeline-config)
+  - [See Also](#see-also)
+
+## Supported Configuration Types
+
+- **Connector**: Connection details for databases, files, or APIs
+- **Job**: ETL job definitions and scheduling
+- **Pipeline**: End-to-end pipeline configuration
+- **Profile**: User or environment-specific settings
+
+## Loading and Validating Configs
+
+Use the provided classes to load and validate configuration files:
+
+```python
+from etlplus.config import PipelineConfig
+
+cfg = PipelineConfig.from_yaml("pipeline.yml")
+```
+
+- Supports YAML and JSON formats
+- Validates against expected schema
+
+## Example: Loading a Pipeline Config
+
+```python
+from etlplus.config import PipelineConfig
+
+pipeline = PipelineConfig.from_yaml("configs/pipeline.yml")
+print(pipeline)
+```
+
+## See Also
+
+- Top-level CLI and library usage in the main [README](../../README.md)
+- Config type definitions in [types.py](types.py)
+- Config utilities in [utils.py](utils.py)

etlplus/database/README.md

@@ -0,0 +1,48 @@
+# etlplus.database subpackage
+
+Documentation for the `etlplus.database` subpackage: database engine, schema, and ORM helpers.
+
+- Provides database engine and connection management
+- Supports schema definition and DDL generation
+- Includes lightweight ORM utilities for tabular data
+- Exposes type definitions for database objects
+
+Back to project overview: see the top-level [README](../../README.md).
+
+- [etlplus.database subpackage](#etlplusdatabase-subpackage)
+  - [Database Engine and Connections](#database-engine-and-connections)
+  - [Schema and DDL Helpers](#schema-and-ddl-helpers)
+  - [ORM Utilities](#orm-utilities)
+  - [Example: Creating a Table](#example-creating-a-table)
+  - [See Also](#see-also)
+
+## Database Engine and Connections
+
+- Manage connections to supported databases
+- Configure engines for different backends
+
+## Schema and DDL Helpers
+
+- Define table schemas and columns
+- Generate DDL statements for supported databases
+
+## ORM Utilities
+
+- Map rows to Python objects
+- Simple CRUD helpers for tabular data
+
+## Example: Creating a Table
+
+```python
+from etlplus.database import Schema, Engine
+
+engine = Engine.connect("sqlite:///example.db")
+schema = Schema.from_dict({"name": "users", "columns": [ ... ]})
+engine.create_table(schema)
+```
+
+## See Also
+
+- Top-level CLI and library usage in the main [README](../../README.md)
+- Schema helpers in [schema.py](schema.py)
+- ORM utilities in [orm.py](orm.py)

etlplus/file/README.md

@@ -0,0 +1,105 @@
+# etlplus.file subpackage
+
+Documentation for the `etlplus.file` subpackage: unified file format support and helpers for reading
+and writing data files.
+
+- Provides a consistent interface for reading and writing files in various formats
+- Supports all formats defined in `FileFormat` (see below)
+- Includes helpers for inferring file format and compression from filenames, extensions, or MIME
+  types
+- Exposes a `File` class with instance methods for reading and writing data
+
+Back to project overview: see the top-level [README](../../README.md).
+
+- [etlplus.file subpackage](#etlplusfile-subpackage)
+  - [Supported File Formats](#supported-file-formats)
+  - [Inferring File Format and Compression](#inferring-file-format-and-compression)
+  - [Reading and Writing Files](#reading-and-writing-files)
+    - [Reading a File](#reading-a-file)
+    - [Writing a File](#writing-a-file)
+  - [File Instance Methods](#file-instance-methods)
+  - [Example: Reading and Writing](#example-reading-and-writing)
+  - [See Also](#see-also)
+
+## Supported File Formats
+
+The following formats are defined in `FileFormat` and supported for reading and writing:
+
+| Format    | Description                                 |
+|-----------|---------------------------------------------|
+| avro      | Apache Avro binary serialization            |
+| csv       | Comma-separated values text files           |
+| feather   | Apache Arrow Feather columnar format        |
+| gz        | Gzip-compressed files (see Compression)     |
+| json      | Standard JSON files                         |
+| ndjson    | Newline-delimited JSON (JSON Lines)         |
+| orc       | Apache ORC columnar format                  |
+| parquet   | Apache Parquet columnar format              |
+| tsv       | Tab-separated values text files             |
+| txt       | Plain text files                            |
+| xls       | Microsoft Excel (legacy .xls)               |
+| xlsx      | Microsoft Excel (modern .xlsx)              |
+| zip       | ZIP-compressed files (see Compression)      |
+| xml       | XML files                                   |
+| yaml      | YAML files                                  |
+
+Compression formats (gz, zip) are also supported as wrappers for other formats.
+
+## Inferring File Format and Compression
+
+Use `infer_file_format_and_compression(value, filename=None)` to infer the file format and
+compression from a filename, extension, or MIME type. Returns a tuple `(file_format,
+compression_format)`.
+
+## Reading and Writing Files
+
+The main entry point for file operations is the `File` class. To read or write files:
+
+### Reading a File
+
+```python
+from etlplus.file import File
+
+f = File("data/sample.csv")
+data = f.read()
+```
+
+- The `read()` method automatically detects the format and compression.
+- Returns parsed data (e.g., list of dicts for tabular formats).
+
+### Writing a File
+
+```python
+from etlplus.file import File
+
+f = File("output.json")
+f.write(data)
+```
+
+- The `write()` method serializes and writes data in the appropriate format.
+- Supports all formats listed above.
+
+## File Instance Methods
+
+- `read()`: Reads and parses the file, returning structured data.
+- `write(data)`: Writes structured data to the file in the detected format.
+
+## Example: Reading and Writing
+
+```python
+from etlplus.file import File
+
+# Read CSV
+csv_file = File("data.csv")
+rows = csv_file.read()
+
+# Write JSON
+json_file = File("output.json")
+json_file.write(rows)
+```
+
+## See Also
+
+- Top-level CLI and library usage in the main [README](../../README.md)
+- File format enums in [enums.py](enums.py)
+- Compression format enums in [enums.py](enums.py)

etlplus/file/avro.py

@@ -1,19 +1,150 @@
 """
 :mod:`etlplus.file.avro` module.
 
-Stub helpers for AVRO read/write.
+Helpers for reading/writing Avro files.
 """
 
 from __future__ import annotations
 
 from pathlib import Path
+from typing import Any
+from typing import cast
 
 from ..types import JSONData
+from ..types import JSONDict
+from ..types import JSONList
 
 # SECTION: EXPORTS ========================================================== #
 
 
-def read(path: Path) -> JSONData:
+__all__ = [
+    'read',
+    'write',
+]
+
+
+# SECTION: INTERNAL CONSTANTS =============================================== #
+
+
+_FASTAVRO_CACHE: dict[str, Any] = {}
+
+
+_PRIMITIVE_TYPES: tuple[type, ...] = (
+    bool,
+    int,
+    float,
+    str,
+    bytes,
+    bytearray,
+)
+
+
+# SECTION: INTERNAL FUNCTIONS =============================================== #
+
+
+def _get_fastavro() -> Any:
+    """
+    Return the fastavro module, importing it on first use.
+
+    Raises an informative ImportError if the optional dependency is missing.
+    """
+    mod = _FASTAVRO_CACHE.get('mod')
+    if mod is not None:  # pragma: no cover - tiny branch
+        return mod
+    try:
+        _fastavro = __import__('fastavro')  # type: ignore[assignment]
+    except ImportError as e:  # pragma: no cover
+        raise ImportError(
+            'AVRO support requires optional dependency "fastavro".\n'
+            'Install with: pip install fastavro',
+        ) from e
+    _FASTAVRO_CACHE['mod'] = _fastavro
+
+    return _fastavro
+
+
+def _normalize_records(data: JSONData) -> JSONList:
+    """
+    Normalize JSON payloads into a list of dictionaries.
+
+    Raises TypeError when payloads contain non-dict items.
+    """
+    if isinstance(data, list):
+        if not all(isinstance(item, dict) for item in data):
+            raise TypeError('AVRO payloads must contain only objects (dicts)')
+        return cast(JSONList, data)
+    return [cast(JSONDict, data)]
+
+
+def _infer_value_type(value: object) -> str | list[str]:
+    """
+    Infer the Avro type for a primitive value.
+
+    Raises TypeError for unsupported types.
+    """
+    if value is None:
+        return 'null'
+    if isinstance(value, bool):
+        return 'boolean'
+    if isinstance(value, int):
+        return 'long'
+    if isinstance(value, float):
+        return 'double'
+    if isinstance(value, str):
+        return 'string'
+    if isinstance(value, (bytes, bytearray)):
+        return 'bytes'
+    raise TypeError('AVRO payloads must contain only primitive values')
+
+
+def _merge_types(types: list[str]) -> str | list[str]:
+    """Return a stable Avro type union for a list of types."""
+    unique = list(dict.fromkeys(types))
+    if len(unique) == 1:
+        return unique[0]
+    ordered = ['null'] + sorted(t for t in unique if t != 'null')
+    return ordered
+
+
+def _infer_schema(records: JSONList) -> dict[str, Any]:
+    """
+    Infer a basic Avro schema from record payloads.
+
+    Only primitive field values are supported; complex values raise TypeError.
+    """
+    field_names = sorted({key for record in records for key in record})
+    fields: list[dict[str, Any]] = []
+    for name in field_names:
+        types: list[str] = []
+        for record in records:
+            value = record.get(name)
+            if value is None:
+                types.append('null')
+                continue
+            if isinstance(value, dict | list):
+                raise TypeError(
+                    'AVRO payloads must contain only primitive values',
+                )
+            if not isinstance(value, _PRIMITIVE_TYPES):
+                raise TypeError(
+                    'AVRO payloads must contain only primitive values',
+                )
+            types.append(cast(str, _infer_value_type(value)))
+        fields.append({'name': name, 'type': _merge_types(types)})
+
+    return {
+        'name': 'etlplus_record',
+        'type': 'record',
+        'fields': fields,
+    }
+
+
+# SECTION: FUNCTIONS ======================================================== #
+
+
+def read(
+    path: Path,
+) -> JSONList:
     """
     Read AVRO content from ``path``.
 
@@ -24,20 +155,21 @@ def read(path: Path) -> JSONData:
 
     Returns
     -------
-    JSONData
-        Parsed payload.
-
-    Raises
-    ------
-    NotImplementedError
-        AVRO :func:`read` is not implemented yet.
+    JSONList
+        The list of dictionaries read from the AVRO file.
     """
-    raise NotImplementedError('AVRO read is not implemented yet')
+    fastavro = _get_fastavro()
+    with path.open('rb') as handle:
+        reader = fastavro.reader(handle)
+        return [cast(JSONDict, record) for record in reader]
 
 
-def write(path: Path, data: JSONData) -> int:
+def write(
+    path: Path,
+    data: JSONData,
+) -> int:
     """
-    Write ``data`` to AVRO at ``path``.
+    Write ``data`` to AVRO at ``path`` and return record count.
 
     Parameters
     ----------
@@ -50,10 +182,17 @@ def write(path: Path, data: JSONData) -> int:
     -------
     int
         Number of records written.
-
-    Raises
-    ------
-    NotImplementedError
-        AVRO :func:`write` is not implemented yet.
     """
-    raise NotImplementedError('AVRO write is not implemented yet')
+    records = _normalize_records(data)
+    if not records:
+        return 0
+
+    fastavro = _get_fastavro()
+    schema = _infer_schema(records)
+    parsed_schema = fastavro.parse_schema(schema)
+
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open('wb') as handle:
+        fastavro.writer(handle, parsed_schema, records)
+
+    return len(records)

etlplus/file/csv.py

@@ -1,7 +1,7 @@
 """
 :mod:`etlplus.file.csv` module.
 
-CSV read/write helpers.
+Helpers for reading/writing CSV files.
 """
 
 from __future__ import annotations
@@ -14,6 +14,15 @@ from ..types import JSONData
 from ..types import JSONDict
 from ..types import JSONList
 
+# SECTION: EXPORTS ========================================================== #
+
+
+__all__ = [
+    'read',
+    'write',
+]
+
+
 # SECTION: FUNCTIONS ======================================================== #
 
 
@@ -21,7 +30,7 @@ def read(
     path: Path,
 ) -> JSONList:
     """
-    Load CSV content as a list of dictionaries.
+    Read CSV content from ``path``.
 
     Parameters
     ----------
@@ -48,7 +57,7 @@ def write(
     data: JSONData,
 ) -> int:
     """
-    Write CSV rows to ``path`` and return the number of rows.
+    Write ``data`` to CSV at ``path`` and return record count.
 
     Parameters
     ----------