etlplus 0.9.1__py3-none-any.whl → 0.9.2__py3-none-any.whl

etlplus/README.md

@@ -0,0 +1,37 @@
+# `etlplus` Package
+
+The `etlplus` package provides a unified Python API and CLI for ETL operations: extraction,
+validation, transformation, and loading of data from files, APIs, and databases.
+
+- Top-level entry points for extract, validate, transform, and load
+- Utilities for pipeline orchestration and helpers
+- Exposes all subpackages for advanced usage
+
+Back to project overview: see the top-level [README](../README.md).
+
+## Subpackages
+
+- [etlplus.api](api/README.md): Lightweight HTTP client and paginated REST helpers
+- [etlplus.file](file/README.md): Unified file format support and helpers
+- [etlplus.cli](cli/README.md): Command-line interface definitions for `etlplus`
+- [etlplus.database](database/README.md): Database engine, schema, and ORM helpers
+- [etlplus.templates](templates/README.md): SQL and DDL template helpers
+- [etlplus.validation](validation/README.md): Data validation utilities and helpers
+- [etlplus.workflow](etlplus/workflow/README.md): Helpers for data connectors, pipelines, jobs, and
+  profiles
+
+## Quickstart
+
+```python
+from etlplus.ops import extract, validate, transform, load
+
+data = extract("file", "input.csv")
+filtered = transform(data, {"filter": {"field": "age", "op": "gt", "value": 25}})
+assert validate(filtered, {"age": {"type": "number", "min": 0}})["valid"]
+load(filtered, "file", "output.json", file_format="json")
+```
+
+## See Also
+
+- [Top-level project README](../README.md)
+- [API reference](../docs/README.md)

etlplus/__init__.py

@@ -2,42 +2,17 @@
 :mod:`etlplus` package.
 
 Top-level facade for the ETLPlus toolkit.
-
-Importing :mod:`etlplus` exposes the handful of coarse-grained helpers most
-users care about: ``extract``, ``transform``, ``load``, ``validate``, and
-``run``. Each helper delegates to the richer modules under ``etlplus.*`` while
-presenting a compact public API surface.
-
-Examples
---------
->>> from etlplus import extract, transform
->>> raw = extract('file', 'input.json')
->>> curated = transform(raw, {'select': ['id', 'name']})
-
-See Also
---------
-- :mod:`etlplus.cli` for the command-line interface
-- :mod:`etlplus.run` for orchestrating pipeline jobs
 """
 
 from .__version__ import __version__
 
 __author__ = 'ETLPlus Team'
 
-from .extract import extract
-from .load import load
-from .run import run
-from .transform import transform
-from .validate import validate
 
 # SECTION: EXPORTS ========================================================== #
 
 
 __all__ = [
+    '__author__',
     '__version__',
-    'extract',
-    'load',
-    'run',
-    'transform',
-    'validate',
 ]

etlplus/api/README.md

@@ -1,7 +1,7 @@
-# etlplus.api module.
+# `etlplus.api` Subpackage
 
-Focused documentation for the `etlplus.api` subpackage: a lightweight HTTP client and helpers for
-paginated REST endpoints.
+Documentation for the `etlplus.api` subpackage: a lightweight HTTP client and helpers for paginated
+REST endpoints.
 
 - Provides a small `EndpointClient` for calling JSON APIs
 - Supports page-, offset-, and cursor-based pagination via `PaginationConfig`
@@ -12,6 +12,21 @@ paginated REST endpoints.
 
 Back to project overview: see the top-level [README](../../README.md).
 
+- [`etlplus.api` Subpackage](#etlplusapi-subpackage)
+  - [Installation](#installation)
+  - [Quickstart](#quickstart)
+    - [Overriding Rate Limits Per Call](#overriding-rate-limits-per-call)
+  - [Choosing `records_path` and `cursor_path`](#choosing-records_path-and-cursor_path)
+  - [Cursor-Based Pagination Example](#cursor-based-pagination-example)
+  - [Offset-based pagination example](#offset-based-pagination-example)
+  - [Authentication](#authentication)
+  - [Errors and Rate Limiting](#errors-and-rate-limiting)
+  - [Types and Transport](#types-and-transport)
+  - [Config Schemas](#config-schemas)
+  - [Supporting Modules](#supporting-modules)
+  - [Minimal Contract](#minimal-contract)
+  - [See also](#see-also)
+
 ## Installation
 
 `etlplus.api` ships as part of the `etlplus` package. Install the package as usual:
@@ -211,6 +226,36 @@ providers can fall back to their own defaults. If you already possess a static t
   `etlplus/api/request_manager.py` wraps `requests` sessions plus retry orchestration. Advanced
   users may consult those modules to adapt behavior.
 
+## Config Schemas
+
+`etlplus.api.types` defines TypedDict-based configuration shapes for API profiles and endpoints.
+Runtime parsing remains permissive in `etlplus.api.config`, but these types improve IDE
+autocomplete and static analysis.
+
+Exported types:
+
+- `ApiConfigMap`: top-level API config shape
+- `ApiProfileConfigMap`: per-profile API config shape
+- `ApiProfileDefaultsMap`: defaults block within a profile
+- `EndpointMap`: endpoint config shape
+
+Example:
+
+```python
+from etlplus.api import ApiConfigMap
+
+api_cfg: ApiConfigMap = {
+    "base_url": "https://example.test",
+    "headers": {"Authorization": "Bearer token"},
+    "endpoints": {
+        "users": {
+            "path": "/users",
+            "method": "GET",
+        },
+    },
+}
+```
+
 ## Supporting Modules
 
 - `etlplus.api.types` collects friendly aliases such as `Headers`, `Params`, `Url`, and
@@ -233,3 +278,6 @@ providers can fall back to their own defaults. If you already possess a static t
 ## See also
 
 - Top-level CLI and library usage in the main [README](../../README.md)
+
+
+[def]: #installation

etlplus/api/__init__.py

@@ -78,6 +78,7 @@ from .config import ApiConfig
 from .config import ApiProfileConfig
 from .config import EndpointConfig
 from .endpoint_client import EndpointClient
+from .enums import HttpMethod
 from .pagination import CursorPaginationConfigMap
 from .pagination import PagePaginationConfigMap
 from .pagination import PaginationClient
@@ -98,6 +99,10 @@ from .types import Headers
 from .types import Params
 from .types import RequestOptions
 from .types import Url
+from .utils import compose_api_request_env
+from .utils import compose_api_target_env
+from .utils import paginate_with_client
+from .utils import resolve_request
 
 # SECTION: EXPORTS ========================================================== #
 
@@ -119,9 +124,14 @@ __all__ = [
     'RequestOptions',
     'RetryStrategy',
     # Enums
+    'HttpMethod',
     'PaginationType',
     # Functions
     'build_http_adapter',
+    'compose_api_request_env',
+    'compose_api_target_env',
+    'paginate_with_client',
+    'resolve_request',
     # Type Aliases
     'CursorPaginationConfigMap',
     'Headers',

etlplus/api/config.py

@@ -3,11 +3,6 @@
 
 Configuration dataclasses for REST API services, profiles, and endpoints.
 
-These models used to live under :mod:`etlplus.config`, but they belong in the
-API layer because they compose runtime types such as
-:class:`etlplus.api.EndpointClient`, :class:`etlplus.api.PaginationConfig`, and
-:class:`etlplus.api.RateLimitConfig`.
-
 Notes
 -----
 - TypedDict references remain editor hints only; :meth:`from_obj` accepts
@@ -18,6 +13,7 @@ Notes
 
 from __future__ import annotations
 
+from collections.abc import Callable
 from collections.abc import Mapping
 from dataclasses import dataclass
 from dataclasses import field
@@ -29,20 +25,20 @@ from typing import overload
 from urllib.parse import urlsplit
 from urllib.parse import urlunsplit
 
-from ..enums import HttpMethod
 from ..types import StrAnyMap
 from ..types import StrStrMap
 from ..utils import cast_str_dict
 from ..utils import coerce_dict
 from ..utils import maybe_mapping
 from .endpoint_client import EndpointClient
+from .enums import HttpMethod
 from .pagination import PaginationConfig
 from .rate_limiting import RateLimitConfig
 
 if TYPE_CHECKING:
-    from ..config.types import ApiConfigMap
-    from ..config.types import ApiProfileConfigMap
-    from ..config.types import EndpointMap
+    from .types import ApiConfigMap
+    from .types import ApiProfileConfigMap
+    from .types import EndpointMap
 
 
 # SECTION: EXPORTS ========================================================== #
@@ -106,6 +102,33 @@ def _effective_service_defaults(
     return fallback_base, fallback_headers
 
 
+def _freeze_mapping(
+    mapping: Mapping[Any, Any],
+    *,
+    key_cast: Callable[[Any], Any] | None = None,
+) -> MappingProxyType:
+    """
+    Return an immutable copy of a mapping, optionally normalizing keys.
+
+    Parameters
+    ----------
+    mapping : Mapping[Any, Any]
+        Source mapping to freeze.
+    key_cast : Callable[[Any], Any] | None, optional
+        Optional key coercion applied to each key.
+
+    Returns
+    -------
+    MappingProxyType
+        Read-only mapping proxy with normalized keys.
+    """
+    if key_cast is None:
+        data = dict(mapping)
+    else:
+        data = {key_cast(key): value for key, value in mapping.items()}
+    return MappingProxyType(data)
+
+
 def _normalize_method(
     value: Any,
 ) -> Any | None:
@@ -232,16 +255,8 @@ class ApiProfileConfig:
     # -- Magic Methods (Object Lifecycle) -- #
 
     def __post_init__(self) -> None:
-        object.__setattr__(
-            self,
-            'headers',
-            MappingProxyType(dict(self.headers)),
-        )
-        object.__setattr__(
-            self,
-            'auth',
-            MappingProxyType(dict(self.auth)),
-        )
+        object.__setattr__(self, 'headers', _freeze_mapping(self.headers))
+        object.__setattr__(self, 'auth', _freeze_mapping(self.auth))
 
     # -- Class Methods -- #
 
@@ -340,20 +355,16 @@ class ApiConfig:
     # -- Magic Methods (Object Lifecycle) -- #
 
     def __post_init__(self) -> None:
-        object.__setattr__(
-            self,
-            'headers',
-            MappingProxyType(dict(self.headers)),
-        )
+        object.__setattr__(self, 'headers', _freeze_mapping(self.headers))
         object.__setattr__(
             self,
             'endpoints',
-            MappingProxyType({str(k): v for k, v in self.endpoints.items()}),
+            _freeze_mapping(self.endpoints, key_cast=str),
         )
         object.__setattr__(
             self,
             'profiles',
-            MappingProxyType({str(k): v for k, v in self.profiles.items()}),
+            _freeze_mapping(self.profiles, key_cast=str),
         )
 
     # -- Internal Instance Methods -- #
@@ -545,12 +556,12 @@ class EndpointConfig:
         object.__setattr__(
             self,
             'path_params',
-            MappingProxyType(dict(self.path_params)),
+            _freeze_mapping(self.path_params),
         )
         object.__setattr__(
             self,
             'query_params',
-            MappingProxyType(dict(self.query_params)),
+            _freeze_mapping(self.query_params),
         )
 
     # -- Class Methods -- #

etlplus/api/endpoint_client.py

@@ -455,7 +455,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.get(url, **kwargs)
 
@@ -479,7 +479,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.post(url, **kwargs)
 
@@ -506,7 +506,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.request(method, url, **kwargs)
 

etlplus/api/enums.py

@@ -0,0 +1,51 @@
+"""
+:mod:`etlplus.api.enums` module.
+
+File-specific REST API-aligned enums and helpers.
+"""
+
+from __future__ import annotations
+
+from ..enums import CoercibleStrEnum
+
+# SECTION: EXPORTS ========================================================= #
+
+
+__all__ = [
+    # Enums
+    'HttpMethod',
+]
+
+
+# SECTION: ENUMS ============================================================ #
+
+
+class HttpMethod(CoercibleStrEnum):
+    """Supported HTTP verbs that accept JSON payloads."""
+
+    # -- Constants -- #
+
+    CONNECT = 'connect'
+    DELETE = 'delete'
+    GET = 'get'
+    HEAD = 'head'
+    OPTIONS = 'options'
+    PATCH = 'patch'
+    POST = 'post'
+    PUT = 'put'
+    TRACE = 'trace'
+
+    # -- Getters -- #
+
+    @property
+    def allows_body(self) -> bool:
+        """
+        Whether the method typically allows a request body.
+
+        Notes
+        -----
+        - RFCs do not strictly forbid bodies on some other methods (e.g.,
+            ``DELETE``), but many servers/clients do not expect them. We mark
+            ``POST``, ``PUT``, and ``PATCH`` as True.
+        """
+        return self in {HttpMethod.POST, HttpMethod.PUT, HttpMethod.PATCH}

etlplus/api/pagination/client.py

@@ -1,5 +1,5 @@
 """
-:mod:`etlplus.api.client` module.
+:mod:`etlplus.api.pagination.client` module.
 
 Client-facing pagination driver for REST API responses.
 

etlplus/api/rate_limiting/config.py

@@ -1,5 +1,5 @@
 """
-:mod:`etlplus.api.rate_limiting.rate_limiter` module.
+:mod:`etlplus.api.rate_limiting.config` module.
 
 Rate limiting configuration primitives.
 
@@ -268,6 +268,18 @@ class RateLimitConfig(BoundsWarningsMixin):
     ) -> Self:
         """
         Normalize rate-limit config and overrides into a single instance.
+
+        Parameters
+        ----------
+        rate_limit : StrAnyMap | RateLimitConfig | None, optional
+            Base rate-limit configuration to normalize.
+        overrides : RateLimitOverrides, optional
+            Override values that take precedence over ``rate_limit``.
+
+        Returns
+        -------
+        Self
+            Normalized rate-limit configuration.
         """
         normalized = _coerce_rate_limit_map(rate_limit)
         cfg = _merge_rate_limit(normalized, overrides)

etlplus/api/rate_limiting/rate_limiter.py

@@ -20,6 +20,7 @@ from __future__ import annotations
 
 import time
 from dataclasses import dataclass
+from typing import Self
 
 from ...utils import to_float
 from ...utils import to_positive_float
@@ -143,13 +144,13 @@ class RateLimiter:
     # -- Class Methods -- #
 
     @classmethod
-    def disabled(cls) -> RateLimiter:
+    def disabled(cls) -> Self:
         """
         Create a limiter that never sleeps.
 
         Returns
         -------
-        RateLimiter
+        Self
             Instance with rate limiting disabled.
         """
         return cls(sleep_seconds=0.0)
@@ -158,7 +159,7 @@ class RateLimiter:
     def fixed(
         cls,
         seconds: float,
-    ) -> RateLimiter:
+    ) -> Self:
         """
         Create a limiter with a fixed non-negative delay.
 
@@ -170,7 +171,7 @@ class RateLimiter:
 
         Returns
         -------
-        RateLimiter
+        Self
             Instance with the specified delay.
         """
         value = to_float(seconds, 0.0, minimum=0.0) or 0.0
@@ -181,7 +182,7 @@ class RateLimiter:
     def from_config(
         cls,
         cfg: RateLimitInput,
-    ) -> RateLimiter:
+    ) -> Self:
         """
         Build a :class:`RateLimiter` from a configuration mapping.
 
@@ -201,12 +202,10 @@ class RateLimiter:
 
         Returns
         -------
-        RateLimiter
+        Self
             Instance with normalized ``sleep_seconds`` and ``max_per_sec``.
         """
         config = RateLimitConfig.from_inputs(rate_limit=cfg)
-        if config is None:
-            return cls.disabled()
 
         # RateLimiter.__post_init__ will normalize and enforce invariants.
         return cls(**config.as_mapping())
@@ -261,6 +260,4 @@ class RateLimiter:
             rate_limit=rate_limit,
             overrides=overrides,
         )
-        if config is None or not config.sleep_seconds:
-            return 0.0
-        return float(config.sleep_seconds)
+        return float(config.sleep_seconds) if config.sleep_seconds else 0.0

etlplus/api/request_manager.py

@@ -14,6 +14,7 @@ from collections.abc import Sequence
 from dataclasses import dataclass
 from dataclasses import field
 from functools import partial
+from types import TracebackType
 from typing import Any
 from typing import cast
 
@@ -137,7 +138,7 @@ class RequestManager:
         self,
         exc_type: type[BaseException] | None,
         exc: BaseException | None,
-        tb: Any,
+        tb: TracebackType | None,
     ) -> None:
         """
         Exit the runtime context and close owned sessions.
@@ -148,7 +149,7 @@ class RequestManager:
             Exception type if raised, else ``None``.
         exc : BaseException | None
             Exception instance if raised, else ``None``.
-        tb : Any
+        tb : TracebackType | None
             Traceback if an exception was raised, else ``None``.
         """
         if self._ctx_session is None:
@@ -275,7 +276,7 @@ class RequestManager:
 
         try:
             policy = self.retry
-            if not policy:
+            if policy is None:
                 try:
                     return fetch(url, **call_kwargs)
                 except requests.RequestException as exc:  # pragma: no cover
@@ -438,9 +439,13 @@ class RequestManager:
             if isinstance(payload, dict):
                 return cast(JSONDict, payload)
             if isinstance(payload, list):
-                if all(isinstance(item, dict) for item in payload):
-                    return cast(JSONData, payload)
-                return [{'value': item} for item in payload]
+                out: list[JSONDict] = []
+                for item in payload:
+                    if isinstance(item, dict):
+                        out.append(cast(JSONDict, item))
+                    else:
+                        out.append({'value': item})
+                return cast(JSONData, out)
             return {'value': payload}
         return {
             'content': response.text,

etlplus/api/transport.py

@@ -191,7 +191,19 @@ def _build_retry_value(
 def _normalize_retry_kwargs(
     retries_cfg: Mapping[str, Any],
 ) -> dict[str, Any]:
-    """Filter and normalize urllib3 ``Retry`` kwargs from a mapping."""
+    """
+    Filter and normalize urllib3 ``Retry`` kwargs from a mapping.
+
+    Parameters
+    ----------
+    retries_cfg : Mapping[str, Any]
+        Raw retry configuration mapping.
+
+    Returns
+    -------
+    dict[str, Any]
+        Filtered and normalized keyword arguments for ``Retry``.
+    """
     allowed_keys = {
         'total',
         'connect',
@@ -239,7 +251,7 @@ def _resolve_max_retries(
     """
     match retries_cfg:
         case int():
-            return retries_cfg
+            return to_maximum_int(retries_cfg, 0)
         case Mapping():
             try:
                 return _build_retry_value(retries_cfg)