etlplus 0.13.0__tar.gz → 0.14.1__tar.gz

{etlplus-0.13.0 → etlplus-0.14.1}/CONTRIBUTING.md

@@ -119,7 +119,7 @@ Use these guidelines to decide whether a test belongs in the unit or integration
   - Can use temporary files/directories, and stub network with fakes/mocks.
   - Examples in this repo: CLI end-to-end, pipeline smoke tests, pagination strategy, runner defaults for pagination/rate limits, target URL composition.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()`, it is integration by default.
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()`, it is integration by default.
 
 ### Where to put tests
 

{etlplus-0.13.0 → etlplus-0.14.1}/DEMO.md

@@ -196,7 +196,7 @@ $ etlplus load transformed.json file final_output.csv
 ## Demo 6: Using Python API
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract
 data = extract("file", "data.csv", format="csv")

{etlplus-0.13.0/etlplus.egg-info → etlplus-0.14.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: etlplus
-Version: 0.13.0
+Version: 0.14.1
 Summary: A Swiss Army knife for simple ETL operations
 Home-page: https://github.com/Dagitali/ETLPlus
 Author: ETLPlus Team
@@ -196,7 +196,7 @@ etlplus extract file examples/data/sample.csv \
 [Python API](#python-api):
 
 ```python
-from etlplus import extract, transform, validate, load
+from etlplus.ops import extract, transform, validate, load
 
 data = extract("file", "input.csv")
 ops = {"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}
@@ -531,7 +531,7 @@ cat examples/data/sample.json \
 Use ETLPlus as a Python library:
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract data
 data = extract("file", "data.json")
@@ -726,7 +726,7 @@ We split tests into two layers:
   pagination + rate limit defaults, file/API connector interactions) may touch temp files and use
   fake clients.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()` it’s integration by default.  Full
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()` it’s integration by default. Full
 criteria: [`CONTRIBUTING.md#testing`](CONTRIBUTING.md#testing).
 
 ### Code Coverage

{etlplus-0.13.0 → etlplus-0.14.1}/README.md

@@ -146,7 +146,7 @@ etlplus extract file examples/data/sample.csv \
 [Python API](#python-api):
 
 ```python
-from etlplus import extract, transform, validate, load
+from etlplus.ops import extract, transform, validate, load
 
 data = extract("file", "input.csv")
 ops = {"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}
@@ -481,7 +481,7 @@ cat examples/data/sample.json \
 Use ETLPlus as a Python library:
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 # Extract data
 data = extract("file", "data.json")
@@ -676,7 +676,7 @@ We split tests into two layers:
   pagination + rate limit defaults, file/API connector interactions) may touch temp files and use
   fake clients.
 
-If a test calls `etlplus.cli.main()` or `etlplus.run.run()` it’s integration by default.  Full
+If a test calls `etlplus.cli.main()` or `etlplus.ops.run.run()` it’s integration by default. Full
 criteria: [`CONTRIBUTING.md#testing`](CONTRIBUTING.md#testing).
 
 ### Code Coverage

{etlplus-0.13.0 → etlplus-0.14.1}/docs/pipeline-guide.md

@@ -281,7 +281,7 @@ section.
 
 ## Validations
 
-Validation rule sets map field names to rules, mirroring `etlplus.validate.FieldRules`:
+Validation rule sets map field names to rules, mirroring `etlplus.ops.validate.FieldRules`:
 
 ```yaml
 validations:
@@ -297,7 +297,7 @@ validations:
 
 ## Transforms
 
-Transformation pipelines follow `etlplus.transform` shapes exactly:
+Transformation pipelines follow `etlplus.ops.transform` shapes exactly:
 
 ```yaml
 transforms:
@@ -431,14 +431,14 @@ Notes:
 - Environment-variable substitution (e.g. `${GITHUB_TOKEN}`) is applied the same way as when loading
   configs via the Python API.
 - For more details on the orchestration implementation, see
-  [Runner internals: etlplus.run](run-module.md).
+  [Runner internals: etlplus.ops.run](run-module.md).
 
-### Python: `etlplus.run.run`
+### Python: `etlplus.ops.run.run`
 
 To trigger a job programmatically, use the high-level runner function exposed by the package:
 
 ```python
-from etlplus.run import run as run_job
+from etlplus.ops.run import run as run_job
 
 result = run_job(
     job="file_to_file_customers",

{etlplus-0.13.0 → etlplus-0.14.1}/etlplus/README.md

@@ -23,7 +23,7 @@ Back to project overview: see the top-level [README](../README.md).
 ## Quickstart
 
 ```python
-from etlplus import extract, validate, transform, load
+from etlplus.ops import extract, validate, transform, load
 
 data = extract("file", "input.csv")
 filtered = transform(data, {"filter": {"field": "age", "op": "gt", "value": 25}})

etlplus-0.14.1/etlplus/__init__.py

@@ -0,0 +1,18 @@
+"""
+:mod:`etlplus` package.
+
+Top-level facade for the ETLPlus toolkit.
+"""
+
+from .__version__ import __version__
+
+__author__ = 'ETLPlus Team'
+
+
+# SECTION: EXPORTS ========================================================== #
+
+
+__all__ = [
+    '__author__',
+    '__version__',
+]

{etlplus-0.13.0 → etlplus-0.14.1}/etlplus/api/__init__.py

@@ -98,6 +98,10 @@ from .types import Headers
 from .types import Params
 from .types import RequestOptions
 from .types import Url
+from .utils import compose_api_request_env
+from .utils import compose_api_target_env
+from .utils import paginate_with_client
+from .utils import resolve_request
 
 # SECTION: EXPORTS ========================================================== #
 
@@ -122,6 +126,10 @@ __all__ = [
     'PaginationType',
     # Functions
     'build_http_adapter',
+    'compose_api_request_env',
+    'compose_api_target_env',
+    'paginate_with_client',
+    'resolve_request',
     # Type Aliases
     'CursorPaginationConfigMap',
     'Headers',

{etlplus-0.13.0 → etlplus-0.14.1}/etlplus/api/endpoint_client.py

@@ -455,7 +455,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.get(url, **kwargs)
 
@@ -479,7 +479,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.post(url, **kwargs)
 
@@ -506,7 +506,7 @@ class EndpointClient:
         -------
         JSONData
             Parsed JSON payload or fallback structure matching
-            :func:`etlplus.extract.extract_from_api` semantics.
+            :func:`etlplus.ops.extract.extract_from_api` semantics.
         """
         return self._request_manager.request(method, url, **kwargs)
 

etlplus-0.13.0/etlplus/run_helpers.py → etlplus-0.14.1/etlplus/api/utils.py

@@ -1,30 +1,13 @@
 """
-:mod:`etlplus.run_helpers` module.
-
-Helper functions and small utilities used by ``etlplus.run`` to compose API
-request/load environments, pagination configs, session objects, and endpoint
-clients. Extracted to keep ``run.py`` focused on orchestration while enabling
-reuse and testability.
-
-Public (re-export safe) helpers:
-- build_pagination_cfg(pagination, overrides)
-- build_session(cfg)
-- compose_api_request_env(cfg, source_obj, extract_opts)
-- compose_api_target_env(cfg, target_obj, overrides)
-- build_endpoint_client(base_url, base_path, endpoints, env)
-- compute_rl_sleep_seconds(rate_limit, overrides)
-- paginate_with_client(client, endpoint_key, params, headers,
-    timeout, pagination, sleep_seconds)
-
-Notes
------
-These helpers intentionally accept permissive ``Any``/``Mapping`` inputs to
-avoid tight coupling with config dataclasses while keeping runtime flexible.
+:mod:`etlplus.api.utils` module.
+
+Shared HTTP helpers for API clients that communicate with REST endpoints.
 """
 
 from __future__ import annotations
 
 import inspect
+from collections.abc import Callable
 from collections.abc import Mapping
 from typing import Any
 from typing import TypedDict
@@ -32,24 +15,33 @@ from typing import cast
 
 import requests  # type: ignore[import]
 
-from .api import ApiConfig
-from .api import EndpointClient
-from .api import EndpointConfig
-from .api import Headers
-from .api import PaginationConfig
-from .api import PaginationConfigMap
-from .api import Params
-from .api import RateLimitConfig
-from .api import RateLimitConfigMap
-from .api import RateLimiter
-from .api import RetryPolicy
-from .api import Url
-from .types import Timeout
+from ..enums import HttpMethod
+from ..types import Timeout
+from .config import ApiConfig
+from .config import EndpointConfig
+from .endpoint_client import EndpointClient
+from .pagination import PaginationConfig
+from .pagination import PaginationConfigMap
+from .rate_limiting import RateLimitConfig
+from .rate_limiting import RateLimitConfigMap
+from .rate_limiting import RateLimiter
+from .retry_manager import RetryPolicy
+from .types import Headers
+from .types import Params
+from .types import Url
+
+# SECTION: CONSTANTS ======================================================== #
+
+
+DEFAULT_TIMEOUT: float = 10.0
+
 
 # SECTION: EXPORTS ========================================================== #
 
 
 __all__ = [
+    # Constants
+    'DEFAULT_TIMEOUT',
     # Functions
     'build_endpoint_client',
     'build_pagination_cfg',
@@ -58,6 +50,7 @@ __all__ = [
     'compose_api_target_env',
     'compute_rl_sleep_seconds',
     'paginate_with_client',
+    'resolve_request',
     # Typed Dicts
     'ApiRequestEnv',
     'ApiTargetEnv',
@@ -68,43 +61,83 @@ __all__ = [
 # SECTION: TYPED DICTS ====================================================== #
 
 
-class ApiRequestEnv(TypedDict, total=False):
-    """API request environment configuration."""
+class BaseApiHttpEnv(TypedDict, total=False):
+    """
+    Common HTTP request environment for API interactions.
+
+    Fields shared by both source-side and target-side API operations.
+    """
 
+    # Request details
     url: Url | None
     headers: dict[str, str]
     timeout: Timeout
+
+    # Session
     session: requests.Session | None
+
+
+class ApiRequestEnv(BaseApiHttpEnv, total=False):
+    """
+    Composed HTTP request environment configuration for REST API sources.
+
+    Returned by :func:`compose_api_request_env` and consumed by the API extract
+    branch. Values are fully merged with endpoint/API defaults and job-level
+    overrides, preserving the original precedence and behavior.
+    """
+
+    # Client
     use_endpoints: bool
     base_url: str | None
     base_path: str | None
     endpoints_map: dict[str, str] | None
     endpoint_key: str | None
+
+    # Request
     params: dict[str, Any]
     pagination: PaginationConfigMap | None
     sleep_seconds: float
+
+    # Reliability
     retry: RetryPolicy | None
     retry_network_errors: bool
 
 
-class ApiTargetEnv(TypedDict, total=False):
-    """API target environment configuration."""
+class ApiTargetEnv(BaseApiHttpEnv, total=False):
+    """
+    Composed HTTP request environment configuration for REST API targets.
+
+    Returned by :func:`compose_api_target_env` and consumed by the API load
+    branch. Values are merged from the target object, optional API/endpoint
+    reference, and job-level overrides, preserving original precedence and
+    behavior.
+
+    Notes
+    -----
+    - Precedence for inherited values matches original logic:
+        overrides -> target -> API profile defaults.
+    - Target composition does not include pagination/rate-limit/retry since
+        loads are single-request operations; only headers/timeout/session
+        apply.
+    """
 
-    url: Url | None
-    headers: dict[str, str]
-    timeout: Timeout
-    session: requests.Session | None
+    # Request
     method: str | None
 
 
 class SessionConfig(TypedDict, total=False):
-    """Configuration for requests.Session."""
+    """
+    Minimal session configuration schema accepted by the
+    :class:`requests.Session` runner.
+
+    Keys mirror common :class:`requests.Session` options; all are optional.
+    """
 
     headers: Mapping[str, Any]
     params: Mapping[str, Any]
-    auth: Any
+    auth: Any  # (user, pass) tuple or requests-compatible auth object
     verify: bool | str
-    cert: Any
+    cert: Any  # str or (cert, key)
     proxies: Mapping[str, Any]
     cookies: Mapping[str, Any]
     trust_env: bool
@@ -113,9 +146,6 @@ class SessionConfig(TypedDict, total=False):
 # SECTION: INTERNAL FUNCTIONS ============================================== #
 
 
-# -- API Environment Composition -- #
-
-
 def _get_api_cfg_and_endpoint(
     cfg: Any,
     api_name: str,
@@ -226,9 +256,6 @@ def _merge_session_cfg_three(
     return cast(SessionConfig | None, (merged or None))
 
 
-# -- Mapping Helpers -- #
-
-
 def _copy_mapping(
     mapping: Mapping[str, Any] | None,
 ) -> dict[str, Any]:
@@ -266,9 +293,6 @@ def _update_mapping(
         target.update(extra)
 
 
-# -- Session -- #
-
-
 def _build_session_optional(
     cfg: SessionConfig | None,
 ) -> requests.Session | None:
@@ -285,7 +309,6 @@ def _build_session_optional(
     requests.Session | None
         Configured session or ``None``.
     """
-
     if isinstance(cfg, dict):
         return build_session(cfg)
     return None
@@ -294,9 +317,6 @@ def _build_session_optional(
 # SECTION: FUNCTIONS ======================================================== #
 
 
-# -- API Environment Composition -- #
-
-
 def build_endpoint_client(
     *,
     base_url: str,
@@ -323,15 +343,7 @@ def build_endpoint_client(
     EndpointClient
         The constructed endpoint client.
     """
-    # Allow tests to monkeypatch etlplus.run.EndpointClient and have it
-    # propagate here by preferring the class on the run module if present.
-    try:
-        from . import run as run_mod  # local import to avoid cycles
-
-        ClientClass = getattr(run_mod, 'EndpointClient', EndpointClient)
-    except (ImportError, AttributeError):  # pragma: no cover - fallback path
-        ClientClass = EndpointClient
-    return ClientClass(
+    return EndpointClient(
         base_url=base_url,
         base_path=base_path,
         endpoints=endpoints,
@@ -558,9 +570,6 @@ def compose_api_target_env(
     }
 
 
-# -- Pagination -- #
-
-
 def build_pagination_cfg(
     pagination: PaginationConfig | None,
     overrides: Mapping[str, Any] | None,
@@ -667,9 +676,6 @@ def build_pagination_cfg(
     return cast(PaginationConfigMap, cfg)
 
 
-# -- Pagination Invocation -- #
-
-
 def paginate_with_client(
     client: Any,
     endpoint_key: str,
@@ -727,9 +733,6 @@ def paginate_with_client(
     return client.paginate(endpoint_key, **kw_pag)
 
 
-# -- Rate Limit -- #
-
-
 def compute_rl_sleep_seconds(
     rate_limit: RateLimitConfig | Mapping[str, Any] | None,
     overrides: Mapping[str, Any] | None,
@@ -782,9 +785,6 @@ def compute_rl_sleep_seconds(
     )
 
 
-# -- Session -- #
-
-
 def build_session(
     cfg: SessionConfig | None,
 ) -> requests.Session:
@@ -841,3 +841,45 @@ def build_session(
             pass
 
     return s
+
+
+def resolve_request(
+    method: HttpMethod | str,
+    *,
+    session: Any | None = None,
+    timeout: Timeout = None,
+) -> tuple[Callable[..., requests.Response], float, HttpMethod]:
+    """
+    Resolve a request callable and effective timeout for an HTTP method.
+
+    Parameters
+    ----------
+    method : HttpMethod | str
+        HTTP method to execute.
+    session : Any | None, optional
+        Requests-compatible session object. Defaults to module-level
+        ``requests``.
+    timeout : Timeout, optional
+        Timeout in seconds for the request. Uses ``DEFAULT_TIMEOUT`` when
+        omitted.
+
+    Returns
+    -------
+    tuple[Callable[..., requests.Response], float, HttpMethod]
+        Tuple of (callable, timeout_seconds, resolved_method).
+
+    Raises
+    ------
+    TypeError
+        If the session object does not expose the requested HTTP method.
+    """
+    http_method = HttpMethod.coerce(method)
+    request_timeout = DEFAULT_TIMEOUT if timeout is None else timeout
+    requester = session or requests
+    request_callable = getattr(requester, http_method.value, None)
+    if not callable(request_callable):
+        raise TypeError(
+            'Session object must supply a callable '
+            f'"{http_method.value}" method',
+        )
+    return request_callable, request_timeout, http_method

{etlplus-0.13.0 → etlplus-0.14.1}/etlplus/cli/handlers.py

@@ -18,15 +18,16 @@ from ..config import PipelineConfig
 from ..config import load_pipeline_config
 from ..database import load_table_spec
 from ..database import render_tables
-from ..extract import extract
 from ..file import File
-from ..load import load
-from ..run import run
-from ..transform import transform
+from ..file import FileFormat
+from ..ops import extract
+from ..ops import load
+from ..ops import run
+from ..ops import transform
+from ..ops import validate
+from ..ops.validate import FieldRules
 from ..types import JSONData
 from ..types import TemplateKey
-from ..validate import FieldRules
-from ..validate import validate
 from . import io as cli_io
 
 # SECTION: EXPORTS ========================================================== #
@@ -569,8 +570,17 @@ def transform_handler(
 
     data = transform(payload, cast(TransformOperations, operations_payload))
 
+    # TODO: Generalize to handle non-file targets.
     if target and target != '-':
-        File(target, file_format=target_format).write(data)
+        # Convert target to Path and target_format to FileFormat if needed
+        file_path = Path(target)
+        file_format = None
+        if target_format is not None:
+            try:
+                file_format = FileFormat(target_format)
+            except ValueError:
+                file_format = None  # or handle error as appropriate
+        File(file_path, file_format=file_format).write(data)
         print(f'Data transformed and saved to {target}')
         return 0
 

{etlplus-0.13.0/etlplus/validation → etlplus-0.14.1/etlplus/ops}/README.md

@@ -1,4 +1,4 @@
-# etlplus.validation subpackage
+# etlplus.ops subpackage
 
 Documentation for the `etlplus.validation` subpackage: data validation utilities and helpers.
 
@@ -8,7 +8,7 @@ Documentation for the `etlplus.validation` subpackage: data validation utilities
 
 Back to project overview: see the top-level [README](../../README.md).
 
-- [etlplus.validation subpackage](#etlplusvalidation-subpackage)
+- [etlplus.ops subpackage](#etlplusops-subpackage)
   - [Validation Features](#validation-features)
   - [Defining Validation Rules](#defining-validation-rules)
   - [Example: Validating Data](#example-validating-data)

etlplus-0.14.1/etlplus/ops/__init__.py

@@ -0,0 +1,61 @@
+"""
+:mod:`etlplus.ops` package.
+
+Data operations helpers.
+
+Importing :mod:`etlplus.ops` exposes the coarse-grained helpers most users care
+about: ``extract``, ``transform``, ``load``, ``validate``, ``run``, and
+``run_pipeline``. Each helper delegates to the richer modules under
+``etlplus.ops.*`` while presenting a compact public API surface. Conditional
+validation orchestration is available via
+:func:`etlplus.ops.utils.maybe_validate`. The legacy compatibility module
+:mod:`etlplus.ops.__init__validation` is deprecated in favor of this package.
+
+Examples
+--------
+>>> from etlplus.ops import extract, transform
+>>> raw = extract('file', 'input.json')
+>>> curated = transform(raw, {'select': ['id', 'name']})
+
+>>> from etlplus.ops.utils import maybe_validate
+>>> payload = {'name': 'Alice'}
+>>> rules = {'required': ['name']}
+>>> def validator(data, config):
+...     missing = [field for field in config['required'] if field not in data]
+...     return {'valid': not missing, 'errors': missing, 'data': data}
+>>> maybe_validate(
+...     payload,
+...     when='both',
+...     enabled=True,
+...     rules=rules,
+...     phase='before_transform',
+...     severity='warn',
+...     validate_fn=validator,
+...     print_json_fn=lambda message: message,
+... )
+{'name': 'Alice'}
+
+See Also
+--------
+:mod:`etlplus.ops.run`
+:mod:`etlplus.ops.utils`
+"""
+
+from .extract import extract
+from .load import load
+from .run import run
+from .run import run_pipeline
+from .transform import transform
+from .validate import validate
+
+# SECTION: EXPORTS ========================================================== #
+
+
+__all__ = [
+    'extract',
+    'load',
+    'run',
+    'run_pipeline',
+    'transform',
+    'validate',
+]